[DONE] RAW overexposure warning

[garry23]

I think many ML supporter are eagerly waiting a raw histogram in LV.

One, most probably silly, thought is could ML be used to provide the user a overexposure warning in the view finder?

My thought was that ML could high jack part of the canon display, for example could ML be used to flash the exposure indicator? Or one of the other canon display figure to give an unambiguous warning to the user that the raw histogram is clipping?

As I say, most probably a silly thought.

[a1ex]

ML can't detect overexposure unless you are looking at the image review screen.

[garry23]

Thanks for the clarification, I should have worked that out for myself!

[Audionut]

After some quick tests, RAW histogram and the zebras are matching exactly what I see from RawDigger

Sweet!

[engardeknave]

This is absolutely fantastic. You guys are amazing.

[ouuzi]

Hi guys,
First post.i have installed the nightly build to my 5D MKIII and I want to know how to enable the raw histogram.From what I have read you pick from canon menus to shoot raw,and then you pickup rgb histogram from ML menus.Is this correct?Also how you get an indication of a clipped channel?I see some numbers in the RGB histogram.
Thanks

[a1ex]

There's no nightly for the 5D3 yet...

[ouuzi]

I followed these instructions :
So, I haven't seen any clear instructions for how people can try out the (unstable) development release yet (instead of the older Alpha 3), so here's my guide. Because it is an unstable development release, it could wreck your camera, so do this at your own risk. Please correct me if I get any of this wrong.

1. Start by updating your camera's firmware to the latest Canon version (1.1.3).

2. Because this isn't a final release, your camera needs a little modification made to it so that it can load Magic Lantern from the SD card. This is called turning on the camera's bootflag. When the bootflag is turned on, and a bootable SD card is inserted, the camera will load and run the file "autoexec.bin" from the SD card. Here, autoexec.bin will be the Magic Lantern software.

To turn on the bootflag, format your SD card in your camera, then copy this file to the root of your card:

http://a1ex.magiclantern.fm/bleeding-edge/5D3/5D3-113-bootflag.fir

Put it back into your camera, then run the firmware update option from the menus. The update will turn on your camera's bootflag.

3. Download the nightly Magic Lantern zip here. Unpack the zip file and copy just the "ML" folder to your SD card. Delete any .fir files that are on the SD card. You've now copied just the fonts and other data that ML needs to operate to your SD card.

4. Now you can copy a Magic Lantern development build (autoexec.bin) to the root of the SD card. Building this file is tricky, but NerveGas has kindly built a copy for us and posted it here. Unpack that zip file and copy the autoexec.bin to the root of your SD card.

5. You should now have an "ML" folder and an "autoexec.bin" file in the root of your SD card. That's the data/fonts and the actual Magic Lantern software. All that is left now is to set the "bootable" flag on the SD card, which tells the camera that it should look for and run the autoexec.bin file for us.

On Windows, you can use the EOSCard utility to do this:

http://pel.hu/eoscard/

Select your SD card, tick the EOS_DEVELOP and BOOTDISK options, install no FIR files, and Save.

On Linux / OSX, you can use the make_bootable.sh script instead. Plug in your SD card, then run the script from the terminal with sudo ./make_bootable.sh . Hopefully you know how to use the terminal.

6. Okay, so now you have the bootflag turned on in your camera (so it looks for bootable SD cards), you've marked your SD card as bootable, and you've copied the development Magic Lantern build "autoexec.bin" and the Magic Lantern data files "ML" to your SD card, you're ready to go! Turn off your camera, put the card in, and turn the camera back on. It should boot and run Magic Lantern.

Note: The nightly builds available on the main site do NOT support the 5D Mark III yet. You need to build autoexec.bin yourself or use the version provided by NerveGas instead (as these instructions already mentioned .


It works.Did i do something wrong?

[Francis]

It works.Did i do something wrong?

Apparently not. You still are probably not running the newest version even with that autoexec.bin supplied by the other user. To get up to date code you have to compile for yourself. Look for help with that under the stickied post 'Getting Started with Development' under General Development Discussion.

[l_d_allan]

Alright, now you have true RAW histogram and zebras. 5D2 only.
To use, simply choose RAW in Canon menu and set histo/zebra to RGB. No new menus - it should just work.

Several comments (sorry for the possible tl:dr ... and the SMF forum s/w "list" doesn't seem to be working):

• ** I installed the 2013-April-24 nightly build ... bit of a glitchy install ... had to pull battery / restart 5dm2 two or three times to get it to come back alive ... so this ML newbie has been hesitant to test beyond checking the version number to confirm installation.
• ** I'm unclear just how to get the RAW histogram in effect. I infer from the above that selecting RAW in the Canon menus and also RGB in the Canon menu for histograms would make this happen? This ML newbie is confused by the mention of zebras (which I haven't used yet).
• ** Perhaps the default "it just works" shouldn't be to have RAW histograms? Based on the "principal of least astonishment", having RAW histograms is significantly enough different ... imo ... to the Canon default that perhaps it should be an intentional choice? Rather than a silent behind-the-back choice made by ML? Plus I think it gives greater visibility to a what I feel is an impressive plus of ML for people considering using ML?  Especially if they've wrestled with the hassle of UniWB?
• ** Or not so much astonishment? I may be like most DLSR users that assumed that blinkies/histograms are based on JPEG data when JPEG selected, and RAW data when RAW selected.  I supposed I was astonished  (unpleasantly) when originally trying to use ETTR that the histogram/blinkies were always based on the JPEG engine.
• ** Which brings up the question/complication: are "blinkies" based on RAW, or remain based on JPEG even when shooting RAW? If JPEG, you almost might want to disable "blinkies" when RAW histograms are in effect to avoid confusion/astonishment. Especially if RAW histograms are displayed when shooting in RAW, but the blinkies remain JPEG based.
• ** In any case, my impression is that it is a terrific accomplishment to get RAW histograms working. Well done!
• ** But in perspective, it would perhaps seem not that complicated for Canon to implement this, with their in-depth knowledge of the "inner black box". But I suppose that applies to a great deal of ML?  I wonder if the "out of my budget" pro-level 1DX has RAW histogram/blinkies?

[l_d_allan]

After some quick tests, RAW histogram and the zebras are matching exactly what I see from RawDigger

I infer that this would be the "gold standard test" that the RAW histograms were working?

[l_d_allan]

After some quick tests, RAW histogram and the zebras are matching exactly what I see from RawDigger

My understanding is that using the desktop RawDigger utility would be the "gold standard test" ... it's indication of blown highlights and ETTR should correspond closely to what shows on the camera LCD? And the blinkies (if they are also based on RAW data)?

My further understanding is that use of the ACR histograms would be less accurate, as so much is going on that adjusts the pixel values (contrast, noise, sharpening, tone curves, etc). It gets more complicated because ProcessVersion 2003 is pretty good at Recovery, PV2010 may be better, and PV2012 is even better at Recovery, as long as all RGBG channels aren't saturated.

As noted in another post, this overwhelmed ML newbie is unclear just how to get the camera set so that RAW histograms are in effect. Sorry.

I suppose you could tell RAW histograms were operational by setting up a static, high contrast situation (using tripod?) and switching back and forth between having ML on, and having ML off. Or switching between RAW and JPEG in the Canon menus while ML was active? Correct?

My impression is that when in JPEG shooting mode and the histogram curve is just before the right-hand-side, a switch to RAW in the Canon menus would show the curve before the right-hand-side to a greater or lesser extent, but definitely a noticeable difference. Correct? Or do I again have a flawed understanding?

And a even more newbie question from someone unfamiliar with "zebras" ... can "zebras" be thought of as similar to "blinkies"? Or perhaps "smarter blinkies" that give more indication of "how much blown"? Rather than just "on / off" like blinkies?

[Francis]

Several comments (sorry for the possible tl:dr ... and the SMF forum s/w "list" doesn't seem to be working):

I installed the 2013-April-24 nightly build ... bit of a glitchy install ... had to pull battery / restart 5dm2 two or three times to get it to come back alive ... so this ML newbie has been hesitant to test beyond checking the version number to confirm installation.

You don't have to "install" anything. Just copy the new autoexec.bin onto the card replacing the old one.

[Audionut]

My understanding is that using the desktop RawDigger utility would be the "gold standard test" ... it's indication of blown highlights and ETTR should correspond closely to what shows on the camera LCD? And the blinkies (if they are also based on RAW data)?

RawDigger is analyzing The captured RAW file on a PC.  It has nothing to do with the camera LCD or anything else.  Since it's accessing the RAW data directly (and not applying any post processing, tone curves etc), it's the most accurate.

My further understanding is that use of the ACR histograms would be less accurate, as so much is going on that adjusts the pixel values (contrast, noise, sharpening, tone curves, etc). It gets more complicated because ProcessVersion 2003 is pretty good at Recovery, PV2010 may be better, and PV2012 is even better at Recovery, as long as all RGBG channels aren't saturated.

Correct, ACR shows the histogram with all its processing applied.  You can take a completely blown image and reduce exposure by 5EV and it will stop showing overexposure warning.  Clearly, that highlight data is still gone though.  What I have found though, is that ACR 2012 will not show the overexposure warning if it thinks it can recover the highlight data.

So RawDigger can show overexposure in the green channel (correctly), ACR will not, if it knows it can recover the detail from the other channels.

As noted in another post, this overwhelmed ML newbie is unclear just how to get the camera set so that RAW histograms are in effect. Sorry.

I suppose you could tell RAW histograms were operational by setting up a static, high contrast situation (using tripod?) and switching back and forth between having ML on, and having ML off. Or switching between RAW and JPEG in the Canon menus while ML was active? Correct?

My impression is that when in JPEG shooting mode and the histogram curve is just before the right-hand-side, a switch to RAW in the Canon menus would show the curve before the right-hand-side to a greater or lesser extent, but definitely a noticeable difference. Correct? Or do I again have a flawed understanding?

Canons histograms are based on the JPG preview that is generated for the LCD.  This happens regardless of whether you are recording to JPG or RAW.  Hence why there is UniWB and people using neutral profiles etc.  Trying to get the JPG preview as close to the actual RAW data as possible.  Same goes for Canon blinkies.

ML is now capable of accessing the RAW data directly.  Hence, it's histogram and zebras are very accurate.
You have to be running the latest nightly and have RAW histogram and zebras enabled in the overlay menu.

And a even more newbie question from someone unfamiliar with "zebras" ... can "zebras" be thought of as similar to "blinkies"? Or perhaps "smarter blinkies" that give more indication of "how much blown"? Rather than just "on / off" like blinkies?

Zebras are MLs implementation of Canons blinkies.  Same thing.  Canons is based on it's JPG preview.  ML can now base it's off RAW data.

[ouuzi]

So With that autoexec.bin file the RAW histogram is not enabled ?If i have to compile on my own is beyond my knowledge at the moment.Is a simple way to get raw histogram in the 5D III at the moment or i have to wait for a stable release?

[chris_overseas]

The new RAW histogram is fantastic, it cuts a lot of guesswork out of getting the correct exposure and I will be using this feature heavily. Thank you! So far I've played around with it on my 5D3 and it seems to be working well. I especially like that it appears over the preview even if I'm viewing it fullsize. Up until now I've always found it frustrating that I couldn't see the (standard) histogram unless the image was just a small thumbnail in the top left. I would have to cycle through the 4 different display modes to review both the composition and the exposure.

Prior to now I've always used an "Image Review" setting of 2 seconds but now I'm finding 4 secs isn't enough either and I might need to set it to "hold" instead so I can take a good look at the raw histogram - it's much smaller than the standard histogram and takes my brain (and eyes!) a while to register what it says. Perhaps I'll get used to that in time though...

Something I noticed while reviewing a just-taken image. If I cycle to a different display mode via the "info" button, the RAW histogram immediately disappears Would it be difficult to keep it there for as long as the image was being reviewed, regardless of how many times info was pressed?

Further to that, I know it's a long shot (and non-trivial) but would it be possible to save the underlying histogram data that goes with each image (perhaps in a sidecar file, or a central DB)? Being able to see the raw histograms when reviewing older photos too would be incredible. If such a thing was possible then the code could be extended to store other Magic Lantern metadata too, opening up all sorts of interesting possibilities.

The one thing I can't quite figure out is that it's not clear to me what the numbers inside the R/G/B circles mean. I know the circles indicate a blown channel, is it a case of the higher the number the larger the area that is blown out?

Lastly, something that has been in the back of my mind for a while is for a way to get the camera to automatically calculate the best ETTR settings for you and take a shot based on that. There's an interesting article about exactly that here too: http://www.luminous-landscape.com/tutorials/optimizing_exposure.shtml (read the section titled "Welcome to the 21st Century"). Would it be possible for ML to take a test shot then calculate from the RAW histogram what the optimal ETTR exposure would be then immediately take another shot using the adjusted values? I can see this isn't always going to be easy, eg presumably there would need to be a threshold of sensels that were allowed to over-expose, and if the test shot was already beyond that then it might take a couple of iterations to find the correct lower exposure.

[engardeknave]

Would it be possible to write exif data or (even partial exif data) to the DNG files?

[a1ex]

Yes, see chdk-dng.c

[engardeknave]

It looks like RAW histograms are not available when review time is set to 'Hold' due to the Ken Rockwell zoom trick.

[Audionut]

It's been working fine here with it set to hold.

[engardeknave]

I'm using a 5d2 and when I have it set to 'Hold' the histogram doesn't say 'RAW' so I'm assuming it's jpeg based.

[a1ex]

Yea, disable the zoom tricks. It only works in QuickReview, not in regular playback mode.

[1%]

If you zoom at all histogram goes away... I think the raw data is lost? Ken rockwell zoom I think just takes you back to play mode when you zoom out.

[engardeknave]

Just pointing this out for anyone who might not be seeing it.

[a1ex]

It's not lost, but as soon as you move to another picture, the data is no longer valid. So, the simplest check was to see whether we are in QR_MODE or not (since in QR you can't move to another picture).