Author Topic: RAW/MLV black level issues fix needs testers  (Read 35811 times)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3144
RAW/MLV black level issues fix needs testers
« on: May 20, 2014, 01:22:23 AM »
more here:
https://bitbucket.org/hudson/magic-lantern/pull-request/484/black-level-fix/diff
for those who know how to compile ML, please test it so we can merge that.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3535
  • Blunt and to the point
Re: RAW/MLV black level issues fix needs testers
« Reply #1 on: May 20, 2014, 05:18:41 AM »
I can compile builds on request if you would like to help with development feedback, but are unable to compile yourself.

vertigopix

  • Senior
  • ****
  • Posts: 281
  • EOS M - 60D - 6D
Re: RAW/MLV black level issues fix needs testers
« Reply #2 on: May 20, 2014, 08:46:47 AM »
I can try this evening with my 60D. (i can't compile, need a built)
Just need to know what to do and what results/informations to provide.

Cheers

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: RAW/MLV black level issues fix needs testers
« Reply #3 on: May 20, 2014, 01:34:40 PM »
If no one has done the 5D III I can test but I'll need a build. No matter how many times I try I can't get it setup right.
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11902
  • 5D Mark Free
Re: RAW/MLV black level issues fix needs testers
« Reply #4 on: May 20, 2014, 01:47:55 PM »
Just need to know what to do and what results/informations to provide.

First link, scroll to "Relevant tests".

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3535
  • Blunt and to the point
Re: RAW/MLV black level issues fix needs testers
« Reply #5 on: May 20, 2014, 01:50:33 PM »
Quote
So, @ early adopters, please test it out before including it in the nightly builds.

Relevant tests:

  •     open the console; it prints the black level whenever it's refreshed
  •     sane values for black level: around 1024, around 2048, around 1700-1800... others?
  •     if there are black warnings, a frame named bad.dng will be saved on the card; upload it
  •     record raw/mlv in crop mode with memory hack on (this was likely to give bad frames)
  •     press record really quick after going to liveview or after toggling the zoom mode or Canon's video mode
  •     playback test videos in mlv_play (they should not have green or pink cast)
  •     check if dual iso images passes the black tests


Here is a build for 60D.  Note, I do not own this camera, and cannot test it.  The build output looks like the nightly build, so it should be perfectly fine.  However, this build comes without any warranty, implied or otherwise.

https://dl.dropboxusercontent.com/u/34113196/ML/Builds/magiclantern-Nightly.2014May20.60D111.zip

Looks like a few modules are failing to build on 60D.  Including mlv_play.  So 60D testers will be unable to confirm playback in camera.

If no one has done the 5D III I can test but I'll need a build.

1.1.3 or 1.2.3?

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: RAW/MLV black level issues fix needs testers
« Reply #6 on: May 20, 2014, 01:53:03 PM »
1.2.3
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3535
  • Blunt and to the point
Re: RAW/MLV black level issues fix needs testers
« Reply #7 on: May 20, 2014, 01:56:41 PM »
This comes with the same warranty as above. 

https://dl.dropboxusercontent.com/u/34113196/ML/Builds/magiclantern-Nightly.2014May20.5D3123.zip

These links will remain active until such time as sufficient feedback is generated.

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: RAW/MLV black level issues fix needs testers
« Reply #8 on: May 20, 2014, 02:04:36 PM »
Thanks, I'll take care of it this afternoon (it's 8AM here) and post results.
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

vertigopix

  • Senior
  • ****
  • Posts: 281
  • EOS M - 60D - 6D
Re: RAW/MLV black level issues fix needs testers
« Reply #9 on: May 20, 2014, 02:51:09 PM »
Thank you,

Will try this evening (i'm in Switzerland) and post results tomorrow.

ayshih

  • Senior
  • ****
  • Posts: 266
Re: RAW/MLV black level issues fix needs testers
« Reply #10 on: May 20, 2014, 04:48:41 PM »
I just did some super-quick testing on the 50D before I had to run out the door this morning.  So far, with toggling crop mode on and off, the black levels appear to be detecting fine (with some occasional failures of consistency checks).  I hadn't realized that the 50D's black level changed with crop mode:
  • No crop: ~1791
  • Crop: ~1023 (same as normal photos)
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11902
  • 5D Mark Free
Re: RAW/MLV black level issues fix needs testers
« Reply #11 on: May 20, 2014, 04:58:36 PM »
A good stress test was using ETTR in Always On mode, in LiveView (did that yesterday). If you push the zoom button often, you will notice some "raw error" messages, when raw_update_params returns 0.

To keep the user code simple, I think it's best if the raw backend retries a few times if needed, if the black level (or resolution or whatever else) can't be autodetected from the first try. So, when it returns 0, user code would simply print an error and give up, rather than using custom retry loops in every raw-related module (not just raw recorders, but also overlays, ETTR and whatever else).

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: RAW/MLV black level issues fix needs testers
« Reply #12 on: May 20, 2014, 05:19:42 PM »
Just like Alex I'm getting the RAW Error but in both Live View modes... maybe I'm missing some thing obvious but I'm not seeing a printed black level. This is 5D III 1.2.3 Audionuts build.

I did get a bad.dng... I'll put it on dropbox in a min...

https://drive.google.com/file/d/0B4JH0INldMfBejlSS2JSWWdST2c/edit?usp=sharing
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11902
  • 5D Mark Free
Re: RAW/MLV black level issues fix needs testers
« Reply #13 on: May 20, 2014, 08:45:33 PM »
The above builds have a nasty bug that affects the bad.dng files (they are corrupted when saving). I've fixed it in the source code, so Audionut may update the builds.

I've moved the retry mechanism from mlv_rec.c to raw.c (raw_update_params_retry_lv), so all other raw-based code can use retrying as needed (and be explicit about it). This fixes the ETTR issue (raw error when hitting the zoom button like crazy), and the UI remains responsive even if black level is broken for a longer period (because overlays or things that just print something in the menu do not have to retry, since they are cyclic processes anyway).

To test the UI responsiveness, you should first disable the bad.dng code (raw.c:839), because it's quite slow. Then, you can set the camera to ISO 100/6400 in movie mode, 5 fps, and overexpose completely. This will create bad pixels and nonuniform black level, which will fail the tests and will trigger the retry mechanism. Then, enable some raw-intensive stuff (like raw/mlv recording, or ettr always on in LV, or some raw overlays), make sure the black level calculation keeps failing (watch the debug console) and try to navigate the menu (should work at normal speed, without lockups or slowdowns). Or watch the raw overlays giving up.

So the issue is quite hard to solve in the presence of random bad frames, and it starts to rely on some tricky error paths that are not exercised often in practice. Before, it simply reported "success" with wrong black level values, and it appeared to work. Need to think some more about it.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3535
  • Blunt and to the point
Re: RAW/MLV black level issues fix needs testers
« Reply #14 on: May 20, 2014, 09:06:47 PM »
New builds contain the same date stamp, so they have been silently updated, and can be download from the links above,

To test UI responsiveness.

https://dl.dropboxusercontent.com/u/34113196/ML/Builds/magiclantern-Nightly.2014May20.60D111_UI.zip
https://dl.dropboxusercontent.com/u/34113196/ML/Builds/magiclantern-Nightly.2014May20.5D3123_UI.zip

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3535
  • Blunt and to the point
Re: RAW/MLV black level issues fix needs testers
« Reply #15 on: May 20, 2014, 09:41:30 PM »
Random bad frames aside, which I assume are another issue (unrelated to BL detection).  How often does the BL change?

What about, is_mode_safe, detect BL.  is_mode_unsafe, printf "you're on your own".  Not exactly elegant!

Or some sort of discard routine?  Detect BL over 10 frames.  Discard the random BL, assume the other n frames (with same BL) aren't bad, and use that BL.

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: RAW/MLV black level issues fix needs testers
« Reply #16 on: May 20, 2014, 09:49:59 PM »
Alex, doing as you stated with those settings renders the menu system totally useless. I'm unable to do anything with those setting while recording RAW.The ML Menu comes up but thats abut it. The black levels printed in the Module Debug Console range between 2043-2055 (F2 1/30 bright spot on the floor) however when I point directly at a window (same settings) I get  Black 1/5 or Black 5/5 mean too different and recording RAW same result in the menu system, no action can be performed.
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11902
  • 5D Mark Free
Re: RAW/MLV black level issues fix needs testers
« Reply #17 on: May 20, 2014, 10:19:53 PM »
Can't reproduce; menu works just fine here, even if the console is only printing black warnings without being able to settle to some value. I can enable raw recording, navigate the submenu...

Black level may not change often now (some cameras have different values in different modes, others have round values like 1024 or 2048), but it will change more once the iso research stuff will be released.

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: RAW/MLV black level issues fix needs testers
« Reply #18 on: May 20, 2014, 10:30:12 PM »
Here's whats enabled from my Modified page...

Dual ISO 100/6400  (which I see now you probably meant HDR Video) sorry not something I use
Exposure Override ON
5FPS
RAW Video ON
Memory Hack ON
LV Digic Peaking Slightly Sharper
Warning for Bad Settings

I'll try again...
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

vertigopix

  • Senior
  • ****
  • Posts: 281
  • EOS M - 60D - 6D
Re: RAW/MLV black level issues fix needs testers
« Reply #19 on: May 21, 2014, 12:32:46 PM »
My results :

EOS 60D (magiclantern-Nightly.2014May20.60D111.zip)

Quote
    open the console; it prints the black level whenever it's refreshed
    sane values for black level: around 1024, around 2048, around 1700-1800... others?

2047-2050

Quote
if there are black warnings, a frame named bad.dng will be saved on the card; upload it

NO bad.dng

Quote
record raw/mlv in crop mode with memory hack on (this was likely to give bad frames)
    press record really quick after going to liveview or after toggling the zoom mode or Canon's video mode
    playback test videos in mlv_play (they should not have green or pink cast)

NO CAST

Quote
check if dual iso images passes the black tests

OK (Black Level : 2053-2059)

Quote
A good stress test was using ETTR in Always On mode, in LiveView (did that yesterday). If you push the zoom button often, you will notice some "raw error" messages, when raw_update_params returns 0.

No "RAW Error"

Quote
To test the UI responsiveness, you should first disable the bad.dng code (raw.c:839), because it's quite slow. Then, you can set the camera to ISO 100/6400 in movie mode, 5 fps, and overexpose completely. This will create bad pixels and nonuniform black level, which will fail the tests and will trigger the retry mechanism. Then, enable some raw-intensive stuff (like raw/mlv recording, or ettr always on in LV, or some raw overlays), make sure the black level calculation keeps failing (watch the debug console) and try to navigate the menu (should work at normal speed, without lockups or slowdowns). Or watch the raw overlays giving up.

NO Slowdowns (tested with magiclantern-Nightly.2014May20.5D3123_UI.zip)


Hope it helps !

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: RAW/MLV black level issues fix needs testers
« Reply #20 on: May 21, 2014, 02:26:35 PM »
NO Slowdowns (tested with magiclantern-Nightly.2014May20.5D3123_UI.zip)

Are you using HDR Video for the 100/6400? Because I tried the test again and it happens for me under those conditions too...
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

vertigopix

  • Senior
  • ****
  • Posts: 281
  • EOS M - 60D - 6D
Re: RAW/MLV black level issues fix needs testers
« Reply #21 on: May 21, 2014, 03:06:17 PM »
Quote
Are you using HDR Video

Yes (Dual ISO video)

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: RAW/MLV black level issues fix needs testers
« Reply #22 on: May 22, 2014, 01:37:12 PM »
I will record a video for you then Alex because mine locks up big time...I will try to get one up ASAP.
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

ayshih

  • Senior
  • ****
  • Posts: 266
Re: RAW/MLV black level issues fix needs testers
« Reply #23 on: May 26, 2014, 06:06:52 AM »
I hadn't realized that the 50D's black level changed with crop mode:
  • No crop: ~1791
  • Crop: ~1023 (same as normal photos)

Actually, I was wrong!  Despite the initial indications of the console output, the actual black level for the 50D should be ~1791 in both modes.  I confirmed this by grabbing DNG frames from MLVs recorded in both in crop and non-crop modes, and verifying that 1023 produced a magenta cast and that 2047 produced a green cast.  Curiously, the MLVs recorded in that mode somehow had the correct black level of ~1791, so I dove in a little further.

When in crop mode, a black level of ~1023 is being reported in the console.  However, with a half-shutter press or the start of recording (this time paying closer to the console output), it immediately switches to reading ~1791.  After release of the shutter or termination of recording, it goes back to reading ~1023.  Any suggestions for what is going on?  Perhaps these black level changes are why the preview turns pink with a half-shutter press?

The upshot is that, although the console output seems to be misleading, the black-level detection appears to work fine.
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11902
  • 5D Mark Free
Re: RAW/MLV black level issues fix needs testers
« Reply #24 on: May 26, 2014, 06:38:17 AM »
Keyword: PINK_FIX_TEST

There are more raw streams available, and in old cameras, one of them is suitable for preview (doesn't look pink) and another is suitable for recording (doesn't have autofocus info burned into the frame). So, ML switches them when you start recording.

That's why you get a grayscale preview while recording. Try commenting out the workaround from raw_rec.c/mlv_rec.c, comment "try to fix pink preview in zoom mode".