RAW/MLV black level issues fix needs testers

[g3gg0]

more here:
https://bitbucket.org/hudson/magic-lantern/pull-request/484/black-level-fix/diff
for those who know how to compile ML, please test it so we can merge that.

[Audionut]

I can compile builds on request if you would like to help with development feedback, but are unable to compile yourself.

[vertigopix]

I can try this evening with my 60D. (i can't compile, need a built)
Just need to know what to do and what results/informations to provide.

Cheers

[Stedda]

If no one has done the 5D III I can test but I'll need a build. No matter how many times I try I can't get it setup right.

[a1ex]

Just need to know what to do and what results/informations to provide.

First link, scroll to "Relevant tests".

[Audionut]

So, @ early adopters, please test it out before including it in the nightly builds.

Relevant tests:

•     open the console; it prints the black level whenever it's refreshed
•     sane values for black level: around 1024, around 2048, around 1700-1800... others?
•     if there are black warnings, a frame named bad.dng will be saved on the card; upload it
•     record raw/mlv in crop mode with memory hack on (this was likely to give bad frames)
•     press record really quick after going to liveview or after toggling the zoom mode or Canon's video mode
•     playback test videos in mlv_play (they should not have green or pink cast)
•     check if dual iso images passes the black tests

Here is a build for 60D.  Note, I do not own this camera, and cannot test it.  The build output looks like the nightly build, so it should be perfectly fine.  However, this build comes without any warranty, implied or otherwise.

https://dl.dropboxusercontent.com/u/34113196/ML/Builds/magiclantern-Nightly.2014May20.60D111.zip

Looks like a few modules are failing to build on 60D.  Including mlv_play.  So 60D testers will be unable to confirm playback in camera.

If no one has done the 5D III I can test but I'll need a build.

1.1.3 or 1.2.3?

[Stedda]

1.2.3

[Audionut]

This comes with the same warranty as above. 

https://dl.dropboxusercontent.com/u/34113196/ML/Builds/magiclantern-Nightly.2014May20.5D3123.zip

These links will remain active until such time as sufficient feedback is generated.

[Stedda]

Thanks, I'll take care of it this afternoon (it's 8AM here) and post results.

[vertigopix]

Thank you,

Will try this evening (i'm in Switzerland) and post results tomorrow.

[ayshih]

I just did some super-quick testing on the 50D before I had to run out the door this morning.  So far, with toggling crop mode on and off, the black levels appear to be detecting fine (with some occasional failures of consistency checks).  I hadn't realized that the 50D's black level changed with crop mode:

• No crop: ~1791
• Crop: ~1023 (same as normal photos)

[a1ex]

A good stress test was using ETTR in Always On mode, in LiveView (did that yesterday). If you push the zoom button often, you will notice some "raw error" messages, when raw_update_params returns 0.

To keep the user code simple, I think it's best if the raw backend retries a few times if needed, if the black level (or resolution or whatever else) can't be autodetected from the first try. So, when it returns 0, user code would simply print an error and give up, rather than using custom retry loops in every raw-related module (not just raw recorders, but also overlays, ETTR and whatever else).

[Stedda]

Just like Alex I'm getting the RAW Error but in both Live View modes... maybe I'm missing some thing obvious but I'm not seeing a printed black level. This is 5D III 1.2.3 Audionuts build.

I did get a bad.dng... I'll put it on dropbox in a min...

https://drive.google.com/file/d/0B4JH0INldMfBejlSS2JSWWdST2c/edit?usp=sharing

[a1ex]

The above builds have a nasty bug that affects the bad.dng files (they are corrupted when saving). I've fixed it in the source code, so Audionut may update the builds.

I've moved the retry mechanism from mlv_rec.c to raw.c (raw_update_params_retry_lv), so all other raw-based code can use retrying as needed (and be explicit about it). This fixes the ETTR issue (raw error when hitting the zoom button like crazy), and the UI remains responsive even if black level is broken for a longer period (because overlays or things that just print something in the menu do not have to retry, since they are cyclic processes anyway).

To test the UI responsiveness, you should first disable the bad.dng code (raw.c:839), because it's quite slow. Then, you can set the camera to ISO 100/6400 in movie mode, 5 fps, and overexpose completely. This will create bad pixels and nonuniform black level, which will fail the tests and will trigger the retry mechanism. Then, enable some raw-intensive stuff (like raw/mlv recording, or ettr always on in LV, or some raw overlays), make sure the black level calculation keeps failing (watch the debug console) and try to navigate the menu (should work at normal speed, without lockups or slowdowns). Or watch the raw overlays giving up.

So the issue is quite hard to solve in the presence of random bad frames, and it starts to rely on some tricky error paths that are not exercised often in practice. Before, it simply reported "success" with wrong black level values, and it appeared to work. Need to think some more about it.

[Audionut]

New builds contain the same date stamp, so they have been silently updated, and can be download from the links above,

To test UI responsiveness.

https://dl.dropboxusercontent.com/u/34113196/ML/Builds/magiclantern-Nightly.2014May20.60D111_UI.zip
https://dl.dropboxusercontent.com/u/34113196/ML/Builds/magiclantern-Nightly.2014May20.5D3123_UI.zip

[Audionut]

Random bad frames aside, which I assume are another issue (unrelated to BL detection).  How often does the BL change?

What about, is_mode_safe, detect BL.  is_mode_unsafe, printf "you're on your own".  Not exactly elegant!

Or some sort of discard routine?  Detect BL over 10 frames.  Discard the random BL, assume the other n frames (with same BL) aren't bad, and use that BL.

[Stedda]

Alex, doing as you stated with those settings renders the menu system totally useless. I'm unable to do anything with those setting while recording RAW.The ML Menu comes up but thats abut it. The black levels printed in the Module Debug Console range between 2043-2055 (F2 1/30 bright spot on the floor) however when I point directly at a window (same settings) I get  Black 1/5 or Black 5/5 mean too different and recording RAW same result in the menu system, no action can be performed.

[a1ex]

Can't reproduce; menu works just fine here, even if the console is only printing black warnings without being able to settle to some value. I can enable raw recording, navigate the submenu...

Black level may not change often now (some cameras have different values in different modes, others have round values like 1024 or 2048), but it will change more once the iso research stuff will be released.

[Stedda]

Here's whats enabled from my Modified page...

Dual ISO 100/6400  (which I see now you probably meant HDR Video) sorry not something I use
Exposure Override ON
5FPS
RAW Video ON
Memory Hack ON
LV Digic Peaking Slightly Sharper
Warning for Bad Settings

I'll try again...

[vertigopix]

My results :

EOS 60D (magiclantern-Nightly.2014May20.60D111.zip)

open the console; it prints the black level whenever it's refreshed
    sane values for black level: around 1024, around 2048, around 1700-1800... others?

2047-2050

if there are black warnings, a frame named bad.dng will be saved on the card; upload it

NO bad.dng

record raw/mlv in crop mode with memory hack on (this was likely to give bad frames)
    press record really quick after going to liveview or after toggling the zoom mode or Canon's video mode
    playback test videos in mlv_play (they should not have green or pink cast)

NO CAST

check if dual iso images passes the black tests

OK (Black Level : 2053-2059)

A good stress test was using ETTR in Always On mode, in LiveView (did that yesterday). If you push the zoom button often, you will notice some "raw error" messages, when raw_update_params returns 0.

No "RAW Error"

To test the UI responsiveness, you should first disable the bad.dng code (raw.c:839), because it's quite slow. Then, you can set the camera to ISO 100/6400 in movie mode, 5 fps, and overexpose completely. This will create bad pixels and nonuniform black level, which will fail the tests and will trigger the retry mechanism. Then, enable some raw-intensive stuff (like raw/mlv recording, or ettr always on in LV, or some raw overlays), make sure the black level calculation keeps failing (watch the debug console) and try to navigate the menu (should work at normal speed, without lockups or slowdowns). Or watch the raw overlays giving up.

NO Slowdowns (tested with magiclantern-Nightly.2014May20.5D3123_UI.zip)


Hope it helps !

[Stedda]

NO Slowdowns (tested with magiclantern-Nightly.2014May20.5D3123_UI.zip)

Are you using HDR Video for the 100/6400? Because I tried the test again and it happens for me under those conditions too...

[vertigopix]

Are you using HDR Video

Yes (Dual ISO video)

[Stedda]

I will record a video for you then Alex because mine locks up big time...I will try to get one up ASAP.

[ayshih]

I hadn't realized that the 50D's black level changed with crop mode:

• No crop: ~1791
• Crop: ~1023 (same as normal photos)

Actually, I was wrong!  Despite the initial indications of the console output, the actual black level for the 50D should be ~1791 in both modes.  I confirmed this by grabbing DNG frames from MLVs recorded in both in crop and non-crop modes, and verifying that 1023 produced a magenta cast and that 2047 produced a green cast.  Curiously, the MLVs recorded in that mode somehow had the correct black level of ~1791, so I dove in a little further.

When in crop mode, a black level of ~1023 is being reported in the console.  However, with a half-shutter press or the start of recording (this time paying closer to the console output), it immediately switches to reading ~1791.  After release of the shutter or termination of recording, it goes back to reading ~1023.  Any suggestions for what is going on?  Perhaps these black level changes are why the preview turns pink with a half-shutter press?

The upshot is that, although the console output seems to be misleading, the black-level detection appears to work fine.

[a1ex]

Keyword: PINK_FIX_TEST

There are more raw streams available, and in old cameras, one of them is suitable for preview (doesn't look pink) and another is suitable for recording (doesn't have autofocus info burned into the frame). So, ML switches them when you start recording.

That's why you get a grayscale preview while recording. Try commenting out the workaround from raw_rec.c/mlv_rec.c, comment "try to fix pink preview in zoom mode".