60D RAW video - it's working !!!

[Can]

Yes, that is better. Thanks marekk, for that and all your other work.

[tin2tin]

If you downloaded EFC within the least few days the latest raw2dng is included in EyeFrame.

[jayhas]

I also made short video three weeks ago. I like the details and richness of color raw footage delivers.

Very nice! Makes it hard to go back shooting H264!!

[Can]

If you downloaded EFC within the least few days the latest raw2dng is included in EyeFrame.

Even better - thanks!

[tin2tin]

Do you mean the the latest version of EFC is working better? If not you can exchange the raw2dng file in the EyeFrame Converter folder.
(Sorry for the thread jack)

[JusT92]

where can i get the latest version ?

[tin2tin]

http://eyeframe.tk/

[JusT92]

i mean where can i get the latest magic lantern 60D with raw video

[Can]

i mean where can i get the latest magic lantern 60D with raw video

Hint: read this thread. 

[Can]

i mean where can i get the latest magic lantern 60D with raw video

But seriously; check the first post in this thread, there's also a recent raw_rec module link in post #534, and there are a couple dropbox links for the latest autoexec/raw2dng and modules for 60D as well...I usually install the nightly build, then combine that with the most current stuff for the 60D. It's all here.

[Haliburton]

Thanks

I just received gift of a Nikon F2, motor drive, and Nikkor 50mm f1.4 and 80-200 zoom lenses.

So, off to pick up Cinevate adapter Nikon AI-S to Canon EOS - smooth pul-focus here we come!

I used

* Voigtlander Color-Ultron 50mm f1.8 (much scene. It's my favorite lens )
* SIGMA 17-70mm f2.8-4 DC OS HSM Macro (Tokyo Tower)
* SIGMA APO 70-300mm F4-5.6 DG MACRO (Tokyo Tower)

[Haliburton]

My raw2dng is working now. I think before I turned of the camera too soon after completing the shot and the .raw file was corrupted. I'm getting around 125 frames now too at 1728 x 972 (was getting around 109 before). Very nice!

My add to Shield's kind post on recovering corrupted RAW files:
http://www.magiclantern.fm/forum/index.php?topic=5732.msg47898#msg47898

[Abstrak]

Sorry if this has been answered I just read this thread front to back. How do these aspect ratios correlate to resolution once rendering to a deliverable extension. Like I know some are 2.35:1, 16:9, but even recording at raw 960x540 (16:9) would seem pointless. Is there a sweet spot that we can get some detail better than h.264 but with record times of 30-40 secs for raw? I know there is limitations with the hardware. I'm new to the film world but a tech guy just trying to get educated thanks. I have the 2.3 build of ML other than training and learning how to shoot raw any real world applications on the 60D outside of testing?

[8beeeaaat]

Are anyone using the raw2dng.app in Mac ??
I can't convert to ProRes no hot/dead pixels by raw2dng without using ACR.
Of course, I can convert noiseless ProRes with AFX via ACR.
However, it is the painful work!

Please teach me the workflow.
Thanks.

[OrionnebelGalaxie17]

New "1080p" RAW video...

[cthornhill]

PS (Yeah I edited it once it posted) - I know this is 'experimental' code. I know what development is like - I have managed commercial imaging products before. I am not trying to be unreasonable...but I just feel like either I am missing something, or there is WAY more variability in the 60D performance than I expected...Any diagnostic or settings advice is welcome - I WANT to find a way to use raw on this camera if I can. - CT

PPS - more data - I just added a test at 3:1 )1280 x 426. Once again, pink blocks and a couple mostly pink frames, at least 18 in 10 seconds (again, ISO 100, 180 degree shutter 24FPS defaults, Global Draw off, audio off (may not mater in raw), latest build, SanDisk Extreme Pro).  I did a test at 1280X536 as well, 3 frames in 10 seconds. As far as I can see now it is just pretty random...I can take a shot and hope for the best, but there does not appear to be any combination that wont get messed up at random right now...I just can't appear to find any 'safe' settings for 7-12 seconds that are consistent shot to shot (no single shot is actually free of bad data in some frames). I am continuing to try, but so far I can't find a way to get consistent footage in raw.

BTW - 2X data rate in H.264 works GREAT on this card and camera...I can go all day and not loose a shot...which is nice, but not quite as flexible as raw...but may be all I can get for now.

I downloaded the latest build from the hourly compiles by Rocket, and gave it a try on my new SanDisk Extreme Pro to see if the combination of this card and using a 2.39:1 lower res might get me past the random pink frames I got before.

I tried 1600x670 and 1472x616, and while I did get pretty good long recordings, I also got some (not as many as in older builds or 16:9 aspect ratio) pink blocks in my frames (sprinkled in the clips). Major bummer...

I have my audio turned off, I have Global Draw turned off, I have my still style set to S2, and every other setting I know about 'optimized' for RAW. My ISO was set to 100, speed to 24 (well you know it's really 23.976) FPS, shutter default to 180 (again, I am rounding the GUI off)...in short I did all I know to do with the fastest card I could buy (way faster than the interface on the 60D).

I know this sounds ungrateful, but I am a bit frustrated. I would really like to evaluate raw as a tool, but if I can't get consistent useful shots even at very short burst (7-14 seconds) how can I really use it as a tool at all? If I could find an aspect ratio and resolution that worked I would be at least able to do serious shots and decide if I can use the footage.

If I just get random messed up frames, is there any real way to easily extract them (automagically or not)? I suppose I can mark and delete them by hand, but frankly, at the amount I see in a 9-11 second clip I can't be sure the result will be worth looking at.

I am really just looking for even one resolution that actually works consistently (say more than 8 out of 10 times)...and yeah, I know, maybe I should just come back in 6 months when more code has been written and tested...I am an image processing specialist myself so I appreciate the code issues and hard work...but I am also a guy who wants to get good image quality out of his Canon camera gear for HD...and a happy Magic Lantern user before all this.

I can accept it if the 60D just won't ever work consistently for raw, but I think other folks may be getting MUCH better results, and I just want to figure out how.

I love the sharpness and DR of raw (just like stills), and I am not expecting full HD or any other impossible goal - I just can't for the life of me figure out what to change to get the sort of results others are posting from the 60D with apparently (from the correspondences I have had) no more data corruption from 'pink blocks'. 

Is there any sort of diagnostic test or settings 'bible' I can use to figure out a way around the darn pink menace? Or is everyone else getting it and just removing those frames and I should expect to do the same?

I understand it may boil down to cards too (or serial numbers of 60D), but if there was some way to test in advance and know what your results would be (or a source for cards that was consistent) that might help...like I said, if it is just my camera, well, I guess I will deal with it the old fashioned way (pay for another) if that works (same for cards),  but I really can't just keep playing 'pot luck' with every shot or every purchase (at the costs of cameras and high end cards).

Again - if this sort of thing is really just normal now, I can accept that, and I guess I can just wait and see, but it there is something I can actually do to get solid shots (or mostly solid shots) I really want to do it.

[Haliburton]

New "1080p" RAW video...

Lovely work, great color, OrionnebelGalaxie17!

Annoying that “the pinks” slip in from time to time:



I have been making quickie fill-in frames to replace these, with the subsequent frame at 50% over the prior frame and flattened.

If someone knows a lot more about eg. After Effects, could you show us how to generate real tween frames, with motion interpolation?

[Haliburton]

@cthornhill: I can’t address all that you’ve said, though it’s been my experience that SanDisk ExtremePro offer no advantage over regular “Extreme” branded SD cards (45 MB/s), due to the 60D’s hardware limitation of about 20 MB/s write speed.

Can’t imagine that the ExtremePro should be ant DISadvantage, but I have been shooting with the same tips you mention (GlobalDraw Off, larger buffer with still quality set to JPG only), and I rarely see pink frames any more, eg. numerous 1728x972 1 fps time lapse, 1800 frames (1/2 hour) with no pinks.

Again, I would hate to think it could be a problem, but perhaps try another (perhaps not-so-) fast card?

[cthornhill]

Haliburton - Thanks, I agree other cards are well worth a try. I have been using regular 'extreme' (45MB) cards before and was hoping the 'Pro' might be better ( not so much that the 60D interface can use it, since it can't) but hoping that maybe the card might deliver smoother write performance at the speed the 60D can do...voodo really, so the engineer in me says 

Just not sure what to try next...hence the extended whine feast above.

Oh - also wasn't to say how great the last couple video's are...I reall got pumped by them too, I just need to get s set of code and settings so I can give ita whirl. I hate to wast limited location time if I can't use the footage - better to stick with X2 mode and do my best to grade it...not as flexible as raw, hence my casting about for a debug/solution...guess I just have to keep trying new builds and cards and settings for now.

[benllben]

@cthornhill: That is quite unusual, i am also using a Sandisk Extreme 45MB/s card, in fact two of the 32gb flavor and i get pretty decent recording times. As Hailburton said, the ExtremePro shouldn't show any differences, however perhaps there is a problem with your current card.

Besides changing firmware settings, what other things have you tried?
Here are few suggestions:

• Use a fresh DNG convertor. Newer ones seem to have less color and pink frame issues.
• Scrap the nightly build and go back to an older, stable version. I believe im using update6 on the first post.
• Format the card and start fresh. Make sure to restore camera settings to default (instructions in ML Install guide)

Now when recording, several stars are drawn on screen representing the buffer as well as a number in Mb/s. How many stars do you have and what is the speed the camera says your card is working at?  If the speed is fine, the number should be between 20-21 (maximum of camera) and there should be plenty of stars (unsure of top of head, but its either around 4 or around 8-10). It should be the bigger number of stars.

[Bioskop.Inc]

Out of the lesser cameras, we seem to be getting the best results & A1ex has already said that he thinks we have spare memory, which might give us a max of about 178 frames at full resolution (1728x972 @ 16:9). We probably won't get much more than this.

Also to be considered is that there seem to be 2 types of 60D & the first models (first year of release?) do produce more frames.
Faster cards do seem to make a bit of difference, but we're talking frames & not seconds.
Try with & without FPS Override.
The first 2 recordings always seem to be a bit dodgy, almost like the camera is warming up.
Try lower resolutions and/or crop sizes - 1600 or 1472 / 2.35:1 or 2.39:1 etc...

The new recording technique that A1ex has implemented works a lot better - from 118 to 137 frames (1728x972 @ 16:9). Yes, you'll get some pink frames every now & then, but this is barely at Alpha stage.

This is the build i'm using:
https://www.dropbox.com/sh/1kitzbmanifb6sf/K3xfBKkVN0/ML-RAW-60D-hourly_2013-06-22_07-40.zip

Just be patient, on the 60D we've gone from getting 72 Silent Pics to actually recording a few seconds RAW video - excellent achievement all considering!

[cthornhill]

PS - Benllben -I was using the latest raw2dng (from the hourly builds by Rocket...many thanks to him too!) Bioskop, my build was ML-RAW-60D-hourly_2013-06-22_15-30 and as noted I used this raw2dng as well (I replace the version in BACHelor2.0 when I update versions and keep the old ones renames in case). In general with a code team I trust like the ML team, and an active dev project, I sort of prefer to stay up to date on code releases (for most things I have stuck with the release build but can't do that on active projects like raw). I am being lazy and not compiling myself (too much of that at work), but I can grab an interim if one is preferred and stick with it for a while...if that helps. I try to read the check in comments each day to see what is going on, and I got super excited by A1ex's work on buffers, and other's additons/refactors too so I updated to this newer build. I will try some of the tests suggested and see what I get...also - should I follow the FPS override notes on the 5DM3 page as the test? For frames I guess...I use that a lot in the stable release for undercrank and love it, but never tried it for normal 24FPS...Thanks for any ideas, I am hoping to get a viable 1600 res mode for me (viable for me) in at least 2.39:1...or when it all finalizes maybe even 16:9, but whatever works for now...and yeah, I know it will only be for as long as the buffer holds out (limited frame count) on the 60D!

Bioskop, Benllben, et. al, Thanks for the good advice! I will also be patient, I really do appreciate all the work and progress the Dev team is making. I used to write assembler and C to support video processing for early HD systems, and I have a great respect for anyone doing that sort do fiddly, high speed data transfer...heck, I had my work cutout for me and I had the darn manufacturer's direct support... !

Pulling the sensor data from the buffer direct to storage was a brilliant move...I only wish the team had access to ASIC logic/processing to run light compression to get rid of some redundant data for a more efficient raw stream...Canon really ought to consider doing that themselves or helping ML do it...even 2:1 or 4:1 (visual lossless, non temporal compression) on the bitstream would make such a huge difference in write speeds possible ( in terms of frames at lower bandwidth to storage)...but anyway what we are getting is great as it is...the optimization and resource juggling the Devs are doing is fantastic.

On a positive note, I am seeing 1.5+ stops of DR head room in my tests. You really can ETTR (I just do it manually via my meter and EVF waveform/zebra). I can recover blown out image elements and hold subject detail in high contrast really well...it is no Alexa, but it blows away 8bit 4:2:0 H.264! Of course the added sharpness is great too, and we get radical color depth, so it rocks all round as a concept. The post stream is going to keep getting better too...and it is nice now (great free tools). I really am a fan!

I will test some of the options suggested, and try some alternate cards...I can live with some 'pink', I just have to get it way lower than 10% of my frames. Is there a range of serial numbers to identify the versions of 60D? I would be happy to check that too.

Glad to get some positive actions to try! That is all I can needed, just was not sure what was working for others. Also  - does anyone find frame skipping helps? I will go back to the older version and report buffers and to the new...I liked the buffer chnanges in the latest builds, as I got a lot more frames, but whatever works is fine with me...if I can get 240 - 340 frames or close to it I expect it will do for a lot of shots! Also, yeah 1600 or even 1440 or so is pretty darn good at 2.39:1 for me, I am not trying to be greedy...much as I would love a 5DM3's rates! I know I could insert money and fix it, I just like Super35 fine, and want to get the most I can from my 60D (and that leaves more money for glass anyway) !

[cthornhill]

@cthornhill: That is quite unusual, i am also using a Sandisk Extreme 45MB/s card, in fact two of the 32gb flavor and i get pretty decent recording times. As Hailburton said, the ExtremePro shouldn't show any differences, however perhaps there is a problem with your current card.

Benllben - could not resist a short 'rant' - I am SO GLAD to hear this! One of my biggest gripes about all raw (not just ML - Red and BM, etc.) is the storage pain...space and costs are REAL issues for commercial shooters I speak with. I know there is nothing much the ML team can do right now (they have no support from Canon to find compression options, so they are working with their hands tied), but uncompressed raw is not such a great thing for those of us from image processing...I mean it is an easy fix (gives you great access to sensor data at native bit depth/res), but if you ever worked with sensor development or the processing pipeline, you must know that everything the sensor sends out is not so great on CMOS (the sensor data typically needs lots of post processing in most cases to provide optimal response curves - I think the ML raw data is post some amplifier options and maybe cleaner, but have not gotten deep into it...believe it is before profiles in the pipeline, but think it is after amps/linierizers), and in any image like that there is a HUGE amount of data that can be dropped with no large impact to image fidelity. I worked on lots of broadcast and medical devices over the years (motion and still) and it makes a HUGE difference to performance if you can just get even a little compression from the bitstream...Now I know that dealing with a Bayer sensor makes all this a huge pain compared to the mostly clean streams I typically got back in the day, but I know this can be done to (did it just a little)...but I guess this is not going to be an option here, as the ML team just don't have either the processor  power or access to it if Canon has it hidden away in the chips.

Anyway, even without compression, there is at least the option to look at storage costs...the guys on the 5DM3 are sort of at the whim of the market...the good news is they have a fast storage path, the bad news it costs a bundle to buy certified devices at that speed right now, so everyone is going for the 'seconds' market (which is cool if it works for you)...in our case, the bad news is we have a slow path to storage on the 60D, but I keep hoping that the silver lining there is lower costs.

I hope that the 45MBs cards are really all we need...if so (if they keep write speeds that exceed the chip limit of just about 20MBs) we have a vast cost effective storage option...if there is really no need (from a shooting perspective) to use 95MBs cards...sure transfer is an issue that is helped on the faster cards...but the cost difference is HUGE...they are basically giving away the 45MBs cards right now at my local outlets (less than $20 US for a 16GB card)...and I hope that means all prices drop for all cards (when they flush the pipeline and have new options to sell us).

Anyway I am really hoping that the results I get are about the same on my 45MBs and 95MBs cards (on record frames per shot)...if so, at least that is a HUGE win on storage costs. I guess I will see what I get when I try some more config and storage tests.

Notes: yes, Red does offer (and everyone I know shooting them uses) compressed raw...I can't say it is cheap on their media (considering you will need at least two copies for safety since it is likely a movie you are shooting and can't afford any losses of critical takes), but compression helps. Why BM has not done this yet I don't know, but they keep threatening to give people a compression option...hope they do. Raw when you get to 10bit is a funny thing... I know folks who shoot log on 10bit, but I have not seen anyone excited about raw on top of the line systems as it gums up the post chain a lot...I suspect there is a lot of flux in this end, as storage costs is not so critical in the budget but time costs is...it may change a lot but not my direct issues... .

[cthornhill]

Per the suggestions I got, I tried a new set of tests. I decided to stick with my current build (ML-RAW-60D-hourly_2013-06-22_15-30) since it was essentially about as close to BioskopInc's build...mine should have the last commits from A1ex and Giovanni C, since I don't see any updates to Rocket's builds since this point in the commit log. This effectivly should be a current build. My Raw2dng is from this build too (automated with BATCHelor 2.0. My card was a freshly formatted (in camera) 16GB SanDisk Extreme 45MBs card (pretty new).

I tried setting FPS override on and off - it looks like I got more frames (a couple more) with it on and set to 23.976 and a few shots with it set to flat 24. ISO was 100, Cannon Audio off, Cannon picture quality to S2, Canon video to 1080, Gobal Draw Off, aspect ratio 2.39:1 1600 pixel width. I shot a couple sessions first to let the camera 'warm up'. BTW - the new memory use and bandwidth reports are really much nicer than the old stars. I notice that it takes a good while to clean up all the buffers after a shot is ended, but I just waited and then shot again. The camera reported a data rate of 20.1 MBs or so, count gets to 244MB before stopping, never more than 2ms idle, and yes it says it skiped 1 frame every darn shot. I got about 202 - 280 frames per attempt, so pretty good for this resolution.

Alas, no good news to report. ,  27 frames in whole or in part ruined by pink blocks out of 284 is about average...in fairly random burst of one or two (lots of two at a time pink lego blocks). I get these spread across a shot like ugly seasoning... So about 10% of the footage is corrupted in some way, and it is spread randomly and in about every shot. I am not trying to be a downer, but to me that is really bad news.

I look through about 16GB of clips of this length, and none are defect free, they all have about the same random pattern of 20-28 pink block runined frames and one frame skip (1 skip frame reported, not too bad unless there are people or animals or other clear and continuous motion in the frame).

The images that are there are wonderful and clear, with great dynamic range, but alas for me, not really something I can use just yet. ML FPS override in the release build does a great job at undercrank/timelapse for me, and maybe raw might do for that, but even there, if the frames are not all good I would just have to stick with the release build or my intervelometer and Canon raw.

I am glad to see so much progress in raw, and it is wonderful if it works for others. I really hope this continues to get better and I will test some other options, but so far, I am just not getting the sort of results other appear to have. I keep hoping it is just me and I can find out what I am messing up, but I have a sad feeling that not all 60D's are going to behave the same way...and that will be a shame so I hope I am wrong. Anyway, at least I have high data rate H.264 to work with for now - all thanks to ML! So I at least have a fall back until I can get some raw joy .

Maybe I will find out what I need to do, or maybe there really will be a code breakthrough or some other good thing...but I guess I better keep saving up for that next camera until I find some luck.

Again - not giving up, and LOTS of thanks to the community and the Dev team, I will just be patient and try some other options. If I have any joy with raw using other setting or cards I will report it, and will try experiments or report other values if anyone needs them.

Lots of thanks again for all the help and all the great code!

[QUATRO]

@cthornhill - REALLY long posts are hard to read. Just my two cents
Anyway, I personally have not had any issues with pink frames or dead pixels and I am totally confused as by how that even happens. I've been using the update #6.1 from the first page (-without LV_AF_RAW) as the last one that got me no problems. I've recently updated to an hourly build "ML-RAW-60D-hourly_2013-06-20_17-40" and will test that out with it's native RAW2DNG and the older one that I've been using and see whether it will give me pink stuff.
I do all my editing on Adobe After Effects CS5, by the way.