How to view RAW histograms after taking the image?

[heyjoe]

In your LiveView screenshots, the histogram is raw; just double-check "Use RAW zebras" is really "Always". If that still doesn't give raw zebras, it's a bug, but I'm unable to emulate LiveView in QEMU yet. A video might be useful to figure out what's going on.

Ok, I tested again (yes, RAW zebras is set to always). It seems zebras don't change on WB change in LiveView, only in Play+Light view. So if you can make Play+Light display raw histograms and zebras/clip warnings (or at least as an option) it would be great.

Safety-wise, they are about the same. These systems don't have a MMU, any task can write anywhere in the RAM, and Canon code saves their settings at shutdown by... reflashing the ROM.

Thanks for explaining.

The crop_rec_4k branch
... is *probably* a little safer.

You are doing a great job.

In any case, the strongest safety net we have is the ability to emulate the firmware in QEMU

I still haven't had the time to install that and try it out but I read that not only LiveView doesn't work in it but also Image capture and review without which I guess there is really nothing to test in QEMU, right?

(with the user's ROM),

What is that?

so if something goes really wrong (such as camera not booting or acting weird), I should be able to look into it.

How? What if my camera gets bricked and there is no way to diagnose?

And, of course, the ROM dumper from bootloader.

Is that something I *must* do before installing the crop_rec_4k build? Please explain as for a layman as this whole thing with so many links to different long threads is a little overwhelming for an ML newbie

Well, somebody *has* to test this before it goes into mainline

I would be happy to but of course I am cautious not to cause a damage.

Do I simply download and unzip  magiclantern-crop_rec_4k.2017Sep26.5D3123.zip onto the card? Do I need to run "Firmware update" after that (as when installing ML for the first time)? Please provide the steps.

Thanks.

[DeafEyeJedi]

I can do this test for you if you still insist on it @heyjoe? The probability of bricking is extremely low and almost impractical to not give it a shot either way.

[heyjoe]

I can do this test for you if you still insist on it @heyjoe?

That would be very nice of you. Could you please also explain how you do it? I.e. are you doing it in QEMU or in camera and how is the crop_rec_4k version to be installed?

The probability of bricking is extremely low and almost impractical to not give it a shot either way.

I suppose it is so but because I am very very new to ML I am extra careful (maybe extra paranoid too).

[DeafEyeJedi]

That would be very nice of you. Could you please also explain how you do it? I.e. are you doing it in QEMU or in camera and how is the crop_rec_4k version to be installed?

Actually I do everything in Cameras. Wish I had gotten into QEMU sooner. Will do one day not only just for out of respect to @a1ex and crew but to help push things forward at a quicker pace. Anyway off I go to my test...

Just downloaded the latest experimental (9/26/17) for 5D3.123 and was able to turn on 'Always On' under 'Use RAW zebras' within the Overlay tab in ML menu. Took a photo and while in preview (mine's set to 4 seconds) there I can see the lovely ML Histogram but then once I press the 'Playback' button to preview the photo once more which doesn't show ML Histogram. Was I expecting (or at least I was hoping) to see this while in playback mode?

I suppose it is so but because I am very very new to ML I am extra careful (maybe extra paranoid too).

I don't blame you. Regardless of how many days you have been using ML I am impressed with your investigation so far and quite intriguing especially coming from a newbie if that's what you call yourself. I would say a heck of a newbie that's capable of pushing things forward overall as a whole.

'Don't just risk anything, take risk with everything' is one of my favorite motto's. 

Also in case you were still curious on how I was able to download and install the experimental build. Very similar to the nightlies, if not the same to be exact.

[heyjoe]

Thanks @DeafEyeJedi!

Here is the result of my test: CR2 files.

All images are @f/2.8. The screenshots are arranged left to right LiveView (LV), Image review (IR), RawDigger (RD).

ISO 160

0.6s


0.8s


1.0s


1.3s


ISO 100

1s:


1.3s:



1.6s:


2s:


The closest to perfect ETTR shots seem to be:

ISO 160, 0.8s: RD shows overexposure of about 0.02EV, IR shows R=33, G=20 overexposure
ISO 100, 1.6s: RD shows overexposure of about 0.02EV, IR shows R=45, G=33 overexposure

Does this prove the new heuristic to be more accurate?

Disclaimer: the light in the room where I shoot may not be perfectly constant.

[a1ex]

Nice test.

I'm unable to reproduce the black zebras - are they from Canon?

To capture the LiveView images, you may use Debug -> Dump image buffers (a 14-bit DNG and two 8-bit YUV422).

There's a screenshot option as well (no need to photograph the camera screen). It won't capture the fast (YUV-based) zebras well (as these are computed by the display controller), but should work fine with all other kinds of zebras (including the RAW-based ones). Your LiveView screenshots show YUV zebras.

[heyjoe]

I'm unable to reproduce the black zebras - are they from Canon?

Yes.

To capture the LiveView images, you may use Debug -> Dump image buffers (a 14-bit DNG and two 8-bit YUV422).

There's a screenshot option as well (no need to photograph the camera screen). It won't capture the fast (YUV-based) zebras well (as these are computed by the display controller), but should work fine with all other kinds of zebras (including the RAW-based ones).

Thanks for the tip. However sometimes it is easier to use my phone and just send the image via bluetooth instead of moving the card back and between the camera and card reader.

Your LiveView screenshots show YUV zebras.

My settings are (taken using the screenshot option):



So what did we learn from this test? Is there anything more that can be done regarding the accuracy of ML for ETTR? Or anything else to test?

It would be really great if the LV histograms are really raw and to have raw histogram/zebras in Play button too (not only in image review).

[a1ex]

With the same zebra settings, here's how they look like here (LiveView vs QR):



Note the LiveView RAW zebras have horizontal lines (not diagonal), they show the same color as the clipped channel(s) (or black if all channels are clipped) and they have "square" edges for speed reasons (they operate on a very low-res image). In QR (after taking a picture), speed is no longer an issue, so they are computed for every displayed pixel.

The Luma zebras (YUV-based) are diagonal red. The YUV RGB zebras also have horizontal lines, but thicker, and fully overexposed areas are solid black.

However, your LiveView histograms are RAW-based.

Can you upload your ML/SETTINGS directory so I can try to reproduce the issue?

[heyjoe]

Note the LiveView RAW zebras have horizontal lines (not diagonal), they show the same color as the clipped channel(s) (or black if all channels are clipped) and they have "square" edges for speed reasons (they operate on a very low-res image).

Thanks for explaining. It is possible that during the test I have had the settings to "Photo only". Now I tested again with "Always" and they are horizontal and unaffected by WB setting (histograms too).
ETA: But still LV and QR show different overexposure warnings.

In QR (after taking a picture), speed is no longer an issue, so they are computed for every displayed pixel.

Does that mean you can also make it to work in Play-button mode?

BTW there is an issue in QR mode. If previously (before taking the current shot) a picture was Played and with a few presses of the Info button it was set in a mode to display Canon's JPG histograms, ML's stuff overlays all that and the view becomes a mess. Example:



The rectangle on top left is the reduced image which Canon shows and right and below of which Canon's histogram and info display.
ETA: same with Play->Light.

The Luma zebras (YUV-based) are diagonal red. The YUV RGB zebras also have horizontal lines, but thicker, and fully overexposed areas are solid black.

However, your LiveView histograms are RAW-based.

Can you upload your ML/SETTINGS directory so I can try to reproduce the issue?

https://drive.google.com/open?id=0B2Mb7hSVSnnFVjV4RjFIMEZUb1U

Please explain what is needed to test further to improve the accuracy as it obviously still cannot be used for correct ETTR.

Also why the accuracy in ISO 160 is worse than that for 100?

[heyjoe]

Here is a proof that LV (RAW) zebras and histograms don't match those of QR (supposedly also RAW):

[a1ex]

Please explain what is needed to test further to improve the accuracy as it obviously still cannot be used for correct ETTR.

For me it's not obvious at all. Except for the lack of clip warnings at ISO 160 1" (which I'm looking into), I don't see anything wrong with the QR zebras.

On the affected image, clipping is not harsh, but looks like this:

Code: [Select]

octave:1> a = read_raw('_MG_6049.CR2', 1);
octave:2> prctile(a(:), [10 50 90 99 99.99 100])'
ans =
    3638   10539   13473   13474   13474   13668
octave:3> imshow(a(1:8:end,1:8:end), [13460,13480])


and that means I should imagine some other heuristic for detecting the peaks (ML used white=13664, above 13474).

Code: [Select]

octave:4>sum(a(:)==13473)
ans =  6856086
octave:5> sum(a(:)==13474)
ans =  1916444
octave:6> sum(a(:)==13475)
ans = 0
octave:7> sum(a(:)>13475)
ans =  987

The issue is those few pixels above the clipping point (in your case, "only" 987). I might have to build a real histogram for evaluating the white level.

The LiveView RAW overlays are not very exact, but not trivial to fix - let's figure out the QR ones first.

[heyjoe]

For me it's not obvious at all. Except for the lack of clip warnings at ISO 160 1" (which I'm looking into), I don't see anything wrong with the QR zebras.

I say obvious because:

ISO 160, 0.6s is quite underexposed. The maximum value in RD is far below saturation (~11400), perhaps about -0.55EV. QR shows R=4 and G=2 clipping for this shot (6047).

ISO 160, 0.8s is just 0.02EV overexposed in RD (R=12%, G=6.8%, B=0%, G2=6.7%). QR shows R=33, G=20 overexposure. Assuming that ML's values are % too, the difference is about 3 times.

etc.

and that means I should imagine some other heuristic for detecting the peaks

You can probably compare the size of the rightmost values in the histogram to those just left of it and if it is above a certain threshold that should indicate a clipping. That may work for the "OVER" indication in the histogram but how will you propagate it back to the image (zebras)? Sounds like computation consuming.

The LiveView RAW overlay are not very exact, but not trivial to fix - let's figure out the QR ones first.

Sure. QR is more important for photo.

[DeafEyeJedi]

The LiveView RAW overlays are not very exact, but not trivial to fix - let's figure out the QR ones first.

Sure. QR is more important for photo.

Agreed.

[a1ex]

Okay, found a histogram-based method that appears to work fine for the above samples (didn't check all of them, as conversion to QEMU format is still a bit of a hack).

The raw_diag module has a slower version of the same algorithm (it checks more pixels and prints some details on the console).

Also found the issue with histogram clipping warnings (it was considering one additional column from the histogram - about 0.1 stops - as overexposed).

Builds coming soon.

[heyjoe]

Great. Looking forward to it.

[heyjoe]

Please write when it is ready to test.

[a1ex]

Take a look at the change log -> it is.

[heyjoe]

Thanks! I will test a little later and post results.

[heyjoe]

CR2 files

ISO 160, 0.8s



ISO 160, 1s



ISO 160, 1.3s





ISO 100, 1.6s



ISO 100, 2s



ISO 100, 2.5s




The result in QR is almost perfect! Great work. The difference seems biggest in the G channel in ISO 160 shots.

There is some discrepancy in LiveView, perhaps due to different calculation.

Do you think you can make the raw ETTR warnings to work in Play mode?

[a1ex]

ISO 100 1.6" is the only one obviously different - to me - from RawDigger. Taking a closer look:

Code: [Select]

octave:1> a = read_raw('_MG_6086.CR2');
octave:2> prctile(a(:), [0 1 10 50 90 99 99.9 99.99 99.999 100])'
ans =
    2212    2865    3963   10080   14306   15215   15449   15464   15465   15471
octave:3> sum(a(:) == 15462)
ans =  332
octave:4> sum(a(:) == 15463)
ans =  381
octave:5> sum(a(:) == 15464)
ans =  14765
octave:6> sum(a(:) == 15465)
ans =  2072
octave:7> sum(a(:) == 15466)
ans = 0
octave:8> sum(a(:) >= 15467)
ans =  1
octave:9> imwrite(a(1:2:end,1:2:end) > 15463, 'clip-red.pgm')



=> ML was right

Do you think you can make the raw ETTR warnings to work in Play mode?

Asked and answered.

Additionally, if anyone does the "boring" part of parsing a CR2 to get the lossless JPEG payload (StripOffsets tag retrieved in C rather than with exiftool), I can do the rest (figuring out the current image in playback mode, decoding it...).

This is a very easy coding task, does not require any camera programming, just reading existing docs and source codes around the net (and time to debug it).

[heyjoe]

ISO 100 1.6" is the only one obviously different - to me - from RawDigger. Taking a closer look:

What does this code do? Looks like some statistical analysis but I can't grasp the details.

I see that ML shows significant overexposure G=29% for ISO 160 1" and G=58% for ISO 160 1.3" but in RD green is not overexposed. What is the reason for this difference? 29% and 58% is quite big. For ISO 100 the values are with not more than 1% difference from RD, so just like in the older test ISO 100 seems to work more accurately.

This is a very easy coding task, does not require any camera programming, just reading existing docs and source codes around the net (and time to debug it).

Can you provide links to the necessary docs? I am curious to see if I have completely forgotten to code in C (it's been many years)

[a1ex]

Why RawDigger shows the numbers it shows, I have no idea.

Code: [Select]

octave:1> a = read_raw('_MG_6084.CR2');
octave:2> prctile(a(:), [0 1 10 50 90 99 99.9 99.99 99.999 100])'
ans =
    2208    2856    3929   10107   13473   13474   13474   13474   13581   13683
octave:3> sum(a(:) == 13472)
ans =  2709
octave:4> sum(a(:) == 13473)
ans =  4543731
octave:5> sum(a(:) == 13474)
ans =  1371767
octave:6> sum(a(:) >= 13475)
ans =  663
octave:7> a_red  = a(1:2:end, 1:2:end);
octave:8> a_g1   = a(1:2:end, 2:2:end);
octave:9> a_g2   = a(2:2:end, 1:2:end);
octave:10> a_blue = a(2:2:end, 2:2:end);
octave:11> sum(a_g1(:) > 13472) * 100 / prod(size(a_g1))
ans =  30.257
octave:12> sum(a_g2(:) > 13472) * 100 / prod(size(a_g2))
ans =  30.213

Hopefully the code is clear enough this time.

Don't be afraid to read the docs.

[heyjoe]

Thanks.

Don't be afraid to read the docs.

I am not and that's why I asked for a link. However all I find is the user guide and the Lua api doc which I suppose is not what you mean?

[a1ex]

Click!

[heyjoe]

Click!

Obviously some misunderstanding. I thought you were talking about docs of ML. Nevermind. I am quite happy with what this thread has lead to. Hopefully after all the testing this code is ready to go into the regular versions