Fix for pink frames

[duey101]

[EDIT: Sorry a1ex, I added it to my profile.] 

I tested today also with the Feb. 12th build [EDIT - used the wrong version] and now I get pink corrupted frames on all takes (usually within the first 5 frames).  MLV

[a1ex]

Always specify your camera.

Do you get pink frames with raw_rec?

Also, my calendar might be broken, but it shows Feb 14. I've only announced the fix yesterday, and only for 5D3 and partially for 60D.

- pink frame fix for 5D3 (I've only tried on 123, let me know if it works on 113)
- partial pink frame fix for 60D (seems OK in crop mode at 2.5K, but still got pink frames at 60fps)

[duey101]

Okay, I just ran another set of shots using this build (magiclantern-v2.3.NEXT.2014Feb14.60D111) ...



RAW takes 1 & 5 I got pink frames, but only on frame 1.
MLV takes 2, 3, & 5 I got pink frames, again, only on the first frame. 

I also noticed the glitchy frames are ONLY in the first frame, and all look very similar, like a pattern!    a1ex, you may also want to reference the pixel bug issues being discussed on this thread as well.  http://www.magiclantern.fm/forum/index.php?topic=5618.new#new

I don't know if that will be useful for you but I have been doing extensive testing with raw2Cdng and have these weird buggy pixel issues, but only in RAW/MLV converted files.

[a1ex]

If it's only in the first frame, it may be from some initialization and may give a hint about what's still missing; will check. The EDMAC is not yet fully understood (for example I couldn't get this working well on 5D2).

The cold pixels are fixed in raw2dng, and this one is the reference converter maintained by me; raw2cdng does not even appear to be open source, so there's nothing I can do. I'm not even sure if these closed-source converters are respecting our copyright on raw2dng/mlv2dng (we have released them as GPL, not as public domain code).

[duey101]

Okay, thank you very much a1ex for clarifying, and I think now that I only have them at the first frame does indicate a pattern which may be helpful for you.  Before I was getting these pink frames in the middle and end of clips so you are on the right track, now it is only at the first frame and all other frames are clean. 

I'll shoot a set of 10 each MLV/RAW tomorrow and see what percentage of pinkies I get.  If you need my help with more 60D specific testing let me know, I'm glad to help.  Right now I can get sustained 7 second clips on my 60D and only first frame corruption in 1280x720 which scales up just fine.  And of course with frame skipping on I get continuous recording (I shot for 90 seconds before turning record off, but of course I had lots of skipped frames.

So if I convert with raw2dng I won't see these cold pixels?  Hmm... I thought I tried that.  Let me check.  But those files don't work in PPro either (dng) only the Cdng files work if I try to import them.

[a1ex]

Okay, make sure you try this one: http://www.magiclantern.fm/forum/index.php?topic=9700.msg98302#msg98302

I've almost got the same fix working on 550D, and indeed, the first frame is corrupted. But if all the others are OK, it's still an improvement.

g3gg0 is also doing some related research here: https://bitbucket.org/hudson/magic-lantern/branch/mlv_rec_edmac_tests

[duey101]

I did indeed try that one just now and I can open the dng files in After Effects (with ACR popup first) but they don't open in PPro, which of course is what is needed.  In AE the files are clean, no funky pixels!  Sadly, they don't work in PPro.

[1%]

(for example I couldn't get this working well on 5D2).

So I guess 7D/50D are going to be out... I haven't looked for a place on 50D where I can get the buffer addy.. did you find one on 5DII? Was it also only recording in crop mode? EOSM/650D seems like a good candidate to try this as well.

[a1ex]

Nope, I've malloced one.

550D is weird: when lucky it gets perfect sync, when unlucky it has continuous pink frames.

Also, it looks like you must set the preferred raw type for every frame (without it you may not get any raw data at all). So this time it's no longer preferred raw type, it's mandatory.

[1%]

Maybe that is why 6D didn't work and got garbage data. When the canon test routine is on the buffer never changes so in theory it could be hardcoded like the LV buffers were.

[Thejungle]

So... what about 600d?

[1%]

Its like 60D so it might be worth a try... but remember when we were using raw type and got all of those tiny lines everywhere? Its probably going to be like that.

On the plus side nanomad found another sync function that might fix the MZ better.

BTW setting the raw type on 6D didn't help.


Ok, did this for 600D... also used the 1100D sync for MZ, it now works in 60p. Commits are at the top: https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all

Much fewer pink frames, seems to write as fast as it did (try more edmacs if you want)... but all of the raw type squiggles are everywhere. No DEFC at ALL. 60P + gd off only pinked on the first frame, 24P is pretty decent.

[duey101]

Good news!!! 

Okay, I ran a huge battery of tests today using this recent build magiclantern-v2.3.NEXT.2014Feb14.60D111 (The Valentine's Day build). My card is a Sandisk Extreme Pro 95 MB/s (SD card).  Pink frames are listed in their frame number(s).

I started in sRAW...

Canon menu: sRaw, 1920x1080 24

(raw_rec.mo) Canon menu: mRAW
24 fps override
Shot 1  PinkFrame 36            (141 frames)
Shot 2  PinkFrame 11,71            (169 frames)
Shot 3  clean                        (169 frames)
Shot 4  clean                       (170 frames)
Shot 5  clean                       (173 frames)
Shot 6  PinkFrame 47,137,167         (170 frames)
Shot 7  clean                       (169 frames)
Shot 8  clean                       (168 frames)
Shot 9  PinkFrame 1,5         (170 frames)
Shot 10 PinkFrame 1         (173 frames)

This was giving me a headache seeing all these pink frames still    so I switched into mRAW in the Canon menu and things got a little better.

(raw_rec.mo) Canon menu: mRAW
FPS override - 23.976 FPS
Shot 1  clean                    (197 frames)
Shot 2  clean               (211 frames)
Shot 3  PinkFrame 117      (214 frames)
Shot 4  clean               (212 frames)
Shot 5  clean               (211 frames)
Shot 6  PinkFrame 45,105   (213 frames)
Shot 7  PinkFrame 71      (214 frames)
Shot 8  clean               (212 frames)
Shot 9  PinkFrame 193      (210 frames)
Shot 10 clean             (211 frames)

(raw_rec.mo) Canon menu: mRAW
FPS override - 24 FPS
Shot 1  clean            (197 frames)
Shot 2  PinkFrame 2,139         (211 frames)
Shot 3  clean               (212 frames)
Shot 4  clean               (211 frames)
Shot 5  clean               (214 frames)
Shot 6  clean               (213 frames)
Shot 7  clean               (213 frames)
Shot 8  clean               (214 frames)
Shot 9  clean               (213 frames)
Shot 10 clean               (179 frames)

With mRAW and MLV mode the results are dramatically improved!  Only one pink frame out of 2114!
MLV/ (mlv_rec.mo) Canon menu: mRAW
FPS override - 23.976 FPS
Shot 1  clean            (207 frames)
Shot 2  clean               (212 frames)
Shot 3  clean            (212 frames)
Shot 4  clean               (211 frames)
Shot 5  clean               (212 frames)
Shot 6  clean            (213 frames)
Shot 7  PinkFrame 1         (212 frames)
Shot 8  clean               (212 frames)
Shot 9  clean            (212 frames)
Shot 10 clean            (211 frames)

MLV/ (mlv_rec.mo) Canon menu: mRAW...  The sweet spot!  A perfect score! 2113 frames and no pink frames!  Is it the 60D Holy Grail?
FPS override - 24 FPS
Shot 1  clean                    (213 frames)
Shot 2  clean               (212 frames)
Shot 3  clean            (212 frames)
Shot 4  clean               (211 frames)
Shot 5  clean               (211 frames)
Shot 6  clean            (212 frames)
Shot 7  clean            (212 frames)
Shot 8  clean               (212 frames)
Shot 9  clean            (206 frames)
Shot 10 clean            (212 frames)

In full RAW 5184x3456 with mlv_rec.mo (MLV mode) it got even better and I got full takes without any issues.
24 FPS override NO PINK FRAMES!
23.976 FPS override NO PINK FRAMES!
I can get sustained 210-217 frames per shot. (8-9 seconds)

I think three key settings I changed from previous attempts were...
Auto power off - off (was on)
Auto preview - off (was set at 2 seconds)
sRaw 2592x1728 (changed to mRaw 3888x2592 and RAW 5184x3456))

MLV MODE in mRAW and RAW is what works on my 60D right now, 10 out of 10 shots and no pink frames or any other kind of glitches.  I can shoot in RAW!!! 

Here are all my pertinent settings both in ML and the Canon camera menus in a text document...

www.dueymoore.com/raw_settings.txt

[andy kh]

tried today's nightly on 550d. i get lots of pink frames

[duey101]

@ andy kh 

Oh... unfortunate.  Did you check all my Canon/ML settings?  I think these three things had something to do with it.

Auto power off - off (was on)
Auto preview - off (was set at 2 seconds)
sRaw 2592x1728 (changed to mRaw 3888x2592 and RAW 5184x3456)

[duey101]

One more note, on the newest build "magiclantern-v2.3.NEXT.2014Feb16.60D111" I now get an additional 40+ frames in 1280x720 mlv_rec.mo in RAW (Canon menu: Quality - RAW 5184x3456)  That is a bonus as now I get 10.20 -10.45 second clips and that is very usable.  Thanks a1ex!  Whatever you did seems to be working on the 60D!  I'm still running further tests now on crop 5X-10X/60FPS.

[andy kh]

@ andy kh 

Oh... unfortunate.  Did you check all my Canon/ML settings?  I think these three things had something to do with it.

Auto power off - off (was on)
Auto preview - off (was set at 2 seconds)
sRaw 2592x1728 (changed to mRaw 3888x2592 and RAW 5184x3456)
[/

it doesnt work with 550D

[a1ex]

Nobody announced pink frame fix in nightly builds for 550D. Let me repeat:

- pink frame fix for 5D3 (I've only tried on 123, let me know if it works on 113)
- partial pink frame fix for 60D (seems OK in crop mode at 2.5K, but still got pink frames at 60fps)

[duey101]

@ a1ex  After further extensive testing I can report that I can shoot in mlv_rec.mo/raw_rec.mo with mRaw/RAW (Canon menu) with no pink frames and also the same in Magic Zoom 5X and 10X, no pink frames zoomed in.  All output is clean and perfect with raw2dng_escho with no sensels/cold/hot/dead pixels.  Perfect raw and output on my 60D.  Looks good.  Any changes you want me to test for 60D?

Same format as before 10 shots per setting, no pink frames in 10/10.

[a1ex]

Nice!

Can you do some tests in 50/60fps, just like the ones you did before, so I could estimate a rough probability for pink frames in this mode?

[duey101]

Okay, will do.  If only the raw2dng_escho output would go straight into Premiere Pro CC... it's a dream come true.   

[a1ex]

Sure, just convince chmee and Thomas Worth to share their source code.

[duey101]

After lots of more testing in 60i here are the results...

mlv_rec.mo
Canon menu: Quality: RAW > 1280x720 60i / ML menu:
FPS Override > 60fps
Clean > average 220 frames 10/10 shots no pink frames.

FPS Override > 60fps 5X Magic Zoom
Clean > average 205 frames 10/10 shots no pink frames.

FPS Override > 60fps  10X Magic Zoom
Clean > average 215 frames 10/10 shots no pink frames.


raw_rec.mo
Canon menu: Quality: RAW > 1280x720 60i / ML menu:
FPS Override > 60fps
Clean > average 227 frames 10/10 shots no pink frames.

FPS Override > 60fps 5X Magic Zoom
Clean > average 203 frames 10/10 shots no pink frames.

FPS Override > 60fps  10X Magic Zoom
Clean > average 205 frames 10/10 shots no pink frames.

Total Pink Frame Count = 0

Code: [Select]

#include <iostream>

using namespace std;

int main ()

cout << "Those are really great results for the 60D"  ;

cout << "Now all I need is a way to go straight to PPro!  ;

return 0;
}
I just took my first C++ lessons today!

[a1ex]

Hm... then why I've got a bunch of pink frames in all my tests at 60fps?!

=> will check again.

[duey101]

I don't know but right now this is the FW I'm staying on for now because it works!  Any more ideas?  Let me know if you need more tests.