uncompressed 14-bit RAW video recording

Started by g3gg0, April 27, 2013, 12:07:12 AM

Previous topic - Next topic

0 Members and 4 Guests are viewing this topic.

Yomommassis

Is there a link we can go to for donating to the developers?
Lets not forget to support the developers, they are really doing some amazing things here

and with the recent increase in attention and the current project there is no reason not to donate
EOS 5D MkIII (ML alpha 3)| EOS 5D MkII (ML v2.3) | EOS 7D (ML alpha 2) | EOS T2i (ML v2.3)
f/2.8 70-200MM L USM | f/1.4 50mm USM | f/1.4 35mm L USM | T/1.5 24mm
Zoom H4n | Rode NTG2

mixmastermike

Quote from: jasonp on May 16, 2013, 03:54:33 AM
Komputer Bay 1000x 64Gb
The latest raw module posted previously by Andy in this thread works perfectly in 1920x1080 with a 5D3, drops 1 frame in x1152 and several in x1180. Let me know if you need more tests with different resolutions.

Nice test, I just ran the same test on a 32Gb x1000 Lexar, 1920 x 1180 now works without dropping frames. 1920 x 1280 drops a few frames.

Today's raw_rec file really improves the reliability. Much respect to all the coders!

The work over the last few days has given me a lot more confidence in filming 1920 x 1080

RenatoPhoto

5D3 with 1000x 32 gig Lexar .  New Compile TEST

New compile gives me problems with producing cropped digital video.  If I record at 5X or 10X zoom, the RAW video can be separated by raw2dng.exe converter but the dng files cannot be opened in ACR 7.1 photoshop.

Also sometimes the raw2dng.exe converter does not put out any frames.

Also there is always a new RAW video on the card at 0Bites.

Maybe it is my compile or maybe not!

The regular video (not zoomed) is ok and I can now record to 1920x1152 without frame drops so the module clears the buffer a bit faster.

The new start - stop method works fine!

Anyway you guys are awesome!!
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

mlrocks

Is there a way to record the upstream raw frames right after the sensor acquiring data? That will be 7 k.

KMA_WWC

Tried to install new build and I always get "please copy al ML files" even though I did copied everything. Also all the menus are blured and look weird (small blocks). Anyone knows why? I will try to install an old build for now.

UPD: Just to confirm, old build works perfectly. Not sure why new one freaks out.

UPD 2: Looks like in new build missing some fonts etc. will try to fix it now and test it out. Am i the only one who has missing fonts in the new build?
Canon 5D Mark III

jpgentry

Thoughtful reply, thanks.

OK, so regarding the issue of 5d3 skipping and KomputerBay cards being junk as some have said...  It seems from what I'm gathering that the guys who have been successful at 1920x1080 were using the KomputerBay cards of 64 and 32GB size.  I have the 128GB and myself and you (platu) we are not getting 1920x1080 without almost immediate line skipping.

Let's all put our collective heads together to find out if this is the card size that is the issue.  How many of you 5d3 guys get no line skipping with 1920x1080 and what size card do you have?

Quote from: platu on May 16, 2013, 12:42:55 AM

I just ordered and received the same CF card as you... the KomputerBay 128gb 1000x CF card and have been running tests with each of the various Raw builds over the last 3 days.  The latest build for 5D3 someone just posted from today seems to be the best in terms of speed... see http://www.magiclantern.fm/forum/index.php?topic=5413.25 (Reply #30).  This includes some small memory tweaks that definitely helped me get a little bit longer runtimes as well as moved me up to the next resolution without dropped frames.

Here has been my experience with the KomputerBay 128 GB 1000x CF card...

For my tests, I tried Global draw turned ON and OFF and there was no significant difference except maybe a second or two extra of recording time but the extra time was not consistent.  So I decided to leave Global Draw = ON (with just peaking and crop marks enabled).


My tests using the ML benchmark utility shows read speeds that range anywhere from 52MB/s to 72MB/s.  Using CrystalDiskMark and ATTO Disk Benchmark, I get similar write speeds so my particular card seems to have a write speed that varies a bit.  I would say the average write speed that I get most of the time is about 65MB/s which is completely in line with the ML benchmarks and the listed speed needed by each resolution listed under the RAW video section of ML.  So I would trust the ML benchmarks and recommended write speeds for each resolution.  I have also confirmed that at least one other person is getting similar write speed using this card.

That said, the buffer/memory tweak Alex made today did have a significant impact when using my card. 

Here are my tests Before today's build...
1920 x 720 or less resolution (no skipped frames for complete 4gb file)
1920 x 840 (no skipped frames for complete 4 gb file)... before today's build, this was the best I could do.
1920 x 900 (no skipped frames for complete 4 gb file)... tweaks added to today's build allowed me to use resolution for first time
1920 x 960 (maybe 10 seconds before frame skipping begins.. unusable beyond that)
1920 x 1080 (maybe 1-2 seconds before frame skipping begins... unusabe beyond that)

After using today's build...
1920 x 900 (no skipped frames for complete 4 gb file) before today's build, I could not use this resolution
1920 x 960 (went from 10 seconds to 20 seconds before frame skipping begins.. unusable beyond that)
1920 x 1080 (went from 1-2 seconds to 5 seconds before frame skipping begins... unusabe beyond that)

So the tweaks made by Alex today have definitely helped so I encourage further memory/buffer optimizations... they are making a difference.

I don't know how EOSHD, Nuemann Films, and Cinema 5D are getting 1080P and greater without dropped frames. They say they are using the same card.  Maybe some of these cards are getting faster write times.  Or maybe they are just getting that resolution for a limited time but a bit longer than me before frame skipping appears.  If any of them can chime in here to clarify, it would helpful for those considering the Komputerbay cards.  This card is by far the most realistic in terms of pricing for the vast majority of people who want to take advantage of this RAW update and plan on using it beyond test videos and personal work.  None of the other cards comes close in terms of price/gb.  The other solutions by Lexar and Toshiba (soon) are more like $650 for 128gb vs $178 or KomputerBay. I'm sure this will eventually change, but that could take a year or longer before cheaper alternatives are available.  But there is no getting around the fact that the less than ideal write speed of these cards may prevent continuous 1080p for some (depending on their particular card?).  But continued memory optimization and other techniques by the ML team may be able to extend 1080p recording from 5 seconds to 30 seconds or possibly more when using this card.  If that can be attained, the vast majority of folks will be able use this resolution in many shooting scenarios, short of documentary or event work.  I do realize that smaller resolutions can be scaled up in post nicely, but I tend to avoid that as do many others I'm sure.

Lastly, on the topic of workflow mentioned above... it's completely worth the time as far as I'm concerned.  The difference is night and day.  There are a lot of test videos coming out now, some of which don't fully capture the quality improvement gained here.  EOSHD, Nuemann Films, and Cinema 5D did great job of showing what's possible.  Just wait until some shorts and features by other skillful DOPs start to appear online.  While I don't love the added work required in post, I find it impossible to go back to H.264 after getting used to the look of my footage now...it's improved that much.

Thank you Alex and team for this... also to g3gg0 who apparently had a huge role in making this particular breakthrough happen.

kaoshotbeatz

I got the may 12th version working fine on my 5dmk3  i see more detail and inside i see less noise at the higher iso im using  promasters 64 gig highspeed udma 1000 card the files write super big would be great to just get a 14 bit or even a 12 bit prores  that would be fine i havent been able to get past the 2 gig limit but thats ok out side i see a big color difference . still no sound and hope they figure out a 1.2.1 version as i have th ninja 2 and want to use that . great alex i get paid on friday i will most def donate some money again

jpgentry

OK Platu.  I just downloaded the build you referenced and got the same results as you.  1920x960 went about 650 frames and then started skipping.  I think that numann and cinema5d must be using the 32 and 64gb cards where you and I are using the 128gb.  I don't know why that would have an effect, but I get no love from the 1920x1080 (full 1080p) setting on the 5d3 with the 128gb 1000x KomputerBay card. 

Quote from: platu on May 16, 2013, 12:42:55 AM

Here has been my experience with the KomputerBay 128 GB 1000x CF card...

After using today's build...
1920 x 900 (no skipped frames for complete 4 gb file) before today's build, I could not use this resolution
1920 x 960 (went from 10 seconds to 20 seconds before frame skipping begins.. unusable beyond that)
1920 x 1080 (went from 1-2 seconds to 5 seconds before frame skipping begins... unusabe beyond that)

So the tweaks made by Alex today have definitely helped so I encourage further memory/buffer optimizations... they are making a difference.

I don't know how EOSHD, Nuemann Films, and Cinema 5D are getting 1080P and greater without dropped frames. They say they are using the same card.  Maybe some of these cards are getting faster write times.  Or maybe they are just getting that resolution for a limited time but a bit longer than me before frame skipping appears.  If any of them can chime in here to clarify, it would helpful for those considering the Komputerbay cards.  This card is by far the most realistic in terms of pricing for the vast majority of people who want to take advantage of this RAW update and plan on using it beyond test videos and personal work.  None of the other cards comes close in terms of price/gb.  The other solutions by Lexar and Toshiba (soon) are more like $650 for 128gb vs $178 or KomputerBay. I'm sure this will eventually change, but that could take a year or longer before cheaper alternatives are available.  But there is no getting around the fact that the less than ideal write speed of these cards may prevent continuous 1080p for some (depending on their particular card?).  But continued memory optimization and other techniques by the ML team may be able to extend 1080p recording from 5 seconds to 30 seconds or possibly more when using this card.  If that can be attained, the vast majority of folks will be able use this resolution in many shooting scenarios, short of documentary or event work.  I do realize that smaller resolutions can be scaled up in post nicely, but I tend to avoid that as do many others I'm sure.

mattmvpmedia

Here's a sample from the 600D at 960x540 resolution with the latest build:


There is still some tearing and magenta blocks showing up, but it's definitely getting better and moving in the right direction. Great job ML team.

bumkicho

Someone stated that the 4GB file limit is a constraint the Magic Lantern people put on it while its in experimental stage. They will remove it at some point.

Is this true or not?

platu

jpgentry,

It seems like the 64GB KomputerBay card version may be faster than the 128GB.  Someone else mentioned this possibility either here or on another forum but I had discounted it until now.  We'll have to see what other folks experience to know for sure.

platu

Quote from: jpgentry on May 16, 2013, 05:17:14 AM
OK Platu.  I just downloaded the build you referenced and got the same results as you.  1920x960 went about 650 frames and then started skipping.  I think that numann and cinema5d must be using the 32 and 64gb cards where you and I are using the 128gb.  I don't know why that would have an effect, but I get no love from the 1920x1080 (full 1080p) setting on the 5d3 with the 128gb 1000x KomputerBay card.

It actually makes a lot of sense now.  There are many examples of different speeds from the same series of card but just different capacity.


Audionut

Consistent err 70 when half pressing shutter (silent pic) after RAW video.

Latest changeset doesn't work with old RAW2DNG, API version mismatch 29030400

jpgentry

In the meantime 1920x900 is my friend.  I kind of like the wider format, though I'm not sure if it fits into any standard.  I guess we can always uprez a bit.

It's good to see that they made such progress today.  That holds out great hope for this making it to full 1080P even with our 128GB card.  I would like to see them raise the limit to 8GB for a single clip in the build for tomorrow.

Quote from: platu on May 16, 2013, 05:23:12 AM
jpgentry,

It seems like the 64GB KomputerBay card version may be faster than the 128GB.  Someone else mentioned this possibility either here or on another forum but I had discounted it until now.  We'll have to see what other folks experience to know for sure.

Colemar

just a guess, but it may be that the 128GB cards are formatted as exFAT, while the <64GB cards are FAT32 and the Canon's firmware is optimized for FAT32..somewhat analogous to the possible 32bit v. 64bit problem with raw2dng.

jpgentry

If we are formatting the cards in the camera before using them wouldn't the camera dictate the format of the card? 

Quote from: Colemar on May 16, 2013, 05:34:58 AM
just a guess, but it may be that the 128GB cards are formatted as exFAT, while the <64GB cards are FAT32 and the Canon's firmware is optimized for FAT32..somewhat analogous to the possible 32bit v. 64bit problem with raw2dng.

Colemar

Quote from: jpgentry on May 16, 2013, 05:44:03 AM
If we are formatting the cards in the camera before using them wouldn't the camera dictate the format of the card?

I haven't found that to be the case, rather the firmware of the card dictates its native formatting schema and size allotment, etc.

also, 1080p23.98 14bit raw footage: https://vimeo.com/66268193  with more to follow

platu

Quote from: jpgentry on May 16, 2013, 05:30:26 AM
In the meantime 1920x900 is my friend.  I kind of like the wider format, though I'm not sure if it fits into any standard.  I guess we can always uprez a bit.

It's good to see that they made such progress today.  That holds out great hope for this making it to full 1080P even with our 128GB card.  I would like to see them raise the limit to 8GB for a single clip in the build for tomorrow.

1920x900 is a pretty good resolution if that's the best we can get.... you can crop down in post to 1920x817 for perfect 2:35:1 Cinemascope.  As a bonus, you'll have 83 pixels of extra vertical space for recomposing.  I'm sure that further optimizations will get us closer to 1080p, but I'm not sure if it will get us all the way there.  The reality is that 1080p requires 80MB/s+ of sustained writing, and our cards just don't do that... not even in short bursts have I ever measured  80+ on my card.  But given how clever the ML folks have been thus far, it wouldn' t surprise me if they figured something out.

Digital Corpus

Just consolidating some information here from what I've read as I find this thread highly intoxicating, despite not owning a 5D3.

Correct me if I'm wrong here, but a RAW frame will be composed of R, G, & B channels. However since the data is not debayered, R & B are 1/2 the horizontal and vertical resolution of a debayered frame, thus 1/4 of the frame's resolution is color data for either R or B. And since G is twice as pixels than either R or B, it'd consume 1/2 of the frame's resolution is color data. Overall, this add's us to just the pixel count of what a frame is. Now, this is 14-bits per pixel data, 8-bits in a byte, 23.976 fps (simplified to 24 fps for the sake of simplicity), frame_res*14/8*24= datarate in bytes/sec.

If my understanding of the data format is sound then this is what follows so far...

For the 5D3 with the KomputerBay 1000x cards being used, we've seen:
1920x900 @ 24 fps --> 1,728,000 pixels, 14 bpp @ 24 fps --> ~69.2 MB/sec write speed requirement
1920x1080 @ 24 fps --> 2,073,600 pixels, 14 bpp @ 24 fps --> ~83.1 MB/sec write speed requirement
1920x1152 @ 24 fps --> 2,211,840 pixels, 14 bpp @ 24 fps --> ~88.6 MB/sec write speed requirement

For the 600D, the reported card I happened to miss, we've seen:
1280x400 @ 24 fps --> 512,000 pixels, 14 bpp @ 24 fps --> ~20.5 MB/sec write speed
960x540 @ 24 fps --> 518,400 pixels, 14 bpp @ 24 fps --> ~20.76 MB/sec write speed

CF and SD cards are Flash memory. There is a bit of a principle of Flash memory that *will* affect  your write speeds, and though these memory cards are not what we consider SSD's, they have to abide by the same rules.

In order to provide a proper test, also to ensure your CF/SD card wears evenly, which not all manufacturers use or advertise if they have wear leveling, you should format your memory card in a computer with a "Full" format to make sure you get real world results. Initial writes may be faster, but that won't be how the rest of the card performs during the rest of it's life.
7D w/ ML | Tokina ATX 11-16 | Canon 24 mm pancake | Canon 40 mm pancake | Canon 17-55 f/2.8 IS | Sigma 150-600 Sports

platu

Quote from: Colemar on May 16, 2013, 05:34:58 AM
just a guess, but it may be that the 128GB cards are formatted as exFAT, while the <64GB cards are FAT32 and the Canon's firmware is optimized for FAT32..somewhat analogous to the possible 32bit v. 64bit problem with raw2dng.

I already tried that.  The card came originally formatted Fat32 and when all my attempts to get faster speed failed, I did a low level reformat to exFAT to see if it faired any better... but there was no significant change in speed (sort of what I expected).

eyeland

Quote from: squig on May 16, 2013, 01:34:18 AM
I'm working on a Canon standard replicant sLUT for adobe camera raw.
Please do share :)
Daybreak broke me loose and brought me back...

platu

Quote from: Colemar on May 16, 2013, 05:46:35 AM
I haven't found that to be the case, rather the firmware of the card dictates its native formatting schema and size allotment, etc.

also, 1080p23.98 14bit raw footage: https://vimeo.com/66268193  with more to follow

That's correct... the camera will respect whatever compatible format the card is using.  So formatting my exFAT card in-camera maintains exFAT.  FYI... the benefit of exFAT will be when the ML folks enable >4GB raw files.  Fat32 only supports files up to 4GB...  exFAT has no such restriction.  This may be a moot point if the ML folks decide to create separate 4GB files instead of one larger file >4GB for long takes.  I see the pros and cons of either approach so ideally it would be a configurable option.

platu

Quote from: Digital Corpus on May 16, 2013, 05:48:49 AM
Just consolidating some information here from what I've read as I find this thread highly intoxicating, despite not owning a 5D3.

Correct me if I'm wrong here, but a RAW frame will be composed of R, G, & B channels. However since the data is not debayered, R & B are 1/2 the horizontal and vertical resolution of a debayered frame, thus 1/4 of the frame's resolution is color data for either R or B. And since G is twice as pixels than either R or B, it'd consume 1/2 of the frame's resolution is color data. Overall, this add's us to just the pixel count of what a frame is. Now, this is 14-bits per pixel data, 8-bits in a byte, 23.976 fps (simplified to 24 fps for the sake of simplicity), frame_res*14/8*24= datarate in bytes/sec.

If my understanding of the data format is sound then this is what follows so far...

For the 5D3 with the KomputerBay 1000x cards being used, we've seen:
1920x900 @ 24 fps --> 1,728,000 pixels, 14 bpp @ 24 fps --> ~69.2 MB/sec write speed requirement
1920x1080 @ 24 fps --> 2,073,600 pixels, 14 bpp @ 24 fps --> ~83.1 MB/sec write speed requirement
1920x1152 @ 24 fps --> 2,211,840 pixels, 14 bpp @ 24 fps --> ~88.6 MB/sec write speed requirement

For the 600D, the reported card I happened to miss, we've seen:
1280x400 @ 24 fps --> 512,000 pixels, 14 bpp @ 24 fps --> ~20.5 MB/sec write speed
960x540 @ 24 fps --> 518,400 pixels, 14 bpp @ 24 fps --> ~20.76 MB/sec write speed

CF and SD cards are Flash memory. There is a bit of a principle of Flash memory that *will* affect  your write speeds, and though these memory cards are not what we consider SSD's, they have to abide by the same rules.

In order to provide a proper test, also to ensure your CF/SD card wears evenly, which not all manufacturers use or advertise if they have wear leveling, you should format your memory card in a computer with a "Full" format to make sure you get real world results. Initial writes may be faster, but that won't be how the rest of the card performs during the rest of it's life.

Interesting analysis... thank you.

1%

Silent pic with raw_rec armed will kill silent pics... all memory is allocated.

4gb limit already off in my modules but files won't convert.

If you format your whatever GB cf card to exfat then the 4gb limit doesn't apply to you either. You wouldn't gain speed... just lose the limit.