How to use Histogram? bug?

Started by Yomommassis, June 03, 2013, 04:49:50 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Yomommassis

Maybe I'm doing something wrong but I feel like my histogram readings are always way off..like..WAY off

I usually expose most of my well lit scenes by pushing the exposure as high as it can go without clipping the highlights but this doesn't always seem to have consistent results

for example:
there was a pretty mean fire north of LA that went right past my house yesterday and I busted out the cameras to take some shots
testing out my new viewfinder I shot some stills using the onscreen histogram
the histogram was saying that i could really push the exposure before clipping but I could tell I was clipping..sure enough when I got the footage on the computer the highlights were clipped

and then I noticed that the histogram wasn't analyzing the entire scene but just the top half of the screen
this was on the T2i with the latest RAW test build as of yesterday

the other time this happened was on a shoot with he 5D Mark 3, getting proper exposure I was looking at the histogram to see how badly a window in the background was clipping, as the sun went down the lighting changed so I bumped up the ISO to notice a weird thing happen..as I went up in ISO the window stopped clipping...

ie
160 clip
200 no clip
250 no clip
320 clip

i believe that was the 5d3 may 24th raw build


I use manual lenses and tried with exposure override on and off and switching raw histogram on and off but I just cant help but feel like im doing something wrong  :(
EOS 5D MkIII (ML alpha 3)| EOS 5D MkII (ML v2.3) | EOS 7D (ML alpha 2) | EOS T2i (ML v2.3)
f/2.8 70-200MM L USM | f/1.4 50mm USM | f/1.4 35mm L USM | T/1.5 24mm
Zoom H4n | Rode NTG2

Audionut

I can assure you that having done considerable testing on RAW histograms (on the 5D3 at least), they are very accurate with regards to the RAW data (CR2).

With regards to the results you are seeing at various ISO's, it's important to have an understanding of what the camera is doing when using these intermediate ISO's (125, 160, 250, 320, 500, 640, etc etc) as compared to what happens when using base ISO's (100, 200, 400, 800 etc etc).

Base ISO's are a result of direct analog amplification of the senor.  Intermediate ISO's are that same analog amplification (base ISO's) with digital manipulation to produce the correct exposure.

ISO 160 and ISO 250 are both produced by digital manipulation of base ISO 200. 

Take these 3 results from RawDigger showing the differences in RAW files from ISO 160, ISO 200 and ISO 250.

Base ISO of 200.
-----------------------------------------------------------------------------


ISO 160 with a 1/3rd reduction in shutter speed to compensate for the 1/3rd reduction in ISO.
-----------------------------------------------------------------------------


ISO 250 with a 1/3rd increase in shutter speed to compensate.
-----------------------------------------------------------------------------




Notice how in the ISO 160 shot, we have a significant increase in overexposed pixels.  And yet when looking at the histogram, we can see that the maximum pixel saturation is only 11260 compared to ISO 200 which is giving us a maximum pixel saturation of 13236 (more dynamic range).  The reason being that ISO 160 is actually ISO 200 with a 1/3rd stop over exposure in camera with the RAW data reduced digitally by 1/3rd on file save.  By using ISO 160 (and 320, 640 etc etc), we've basically reduced the cameras dynamic range by 1/3rd of a stop.  We've gained nothing that can't be done in a post processing* program while still maintaining the full dynamic range of ISO 200.

ISO 250, while looking good on initial glance at just the histogram seems to be ok, but it's actually (arguably) worse.  ISO 250 is simply ISO 200 with a 1/3rd stop underexposure in camera, with a 1/3rd stop digital increase on file save.  To understand the detrimental effects of underexposing in camera, an understanding of how the camera captures light is needed.  You can find some information on the subject in this thread.  But basically, it just results in increased noise in the capture (an increase of under exposed pixels can be seen in ISO 250, but not shown in my screenshots), and again, it's something that can be done in a post processor* with much better results.

So in summary, we can see how ISO 160 results in over exposure warning in the histogram, while ISO's 200 and 250 show no over exposure warning.

*Post processing
When we understand that ISO 160 and ISO 250 are both actually ISO 200 with some digital manipulation, it makes sense to use a base ISO in camera and perform the digital manipulation in post process.  This is especially true when using ISO's 160, 320, 640 et all, as we can over expose our shot @base ISO's and reduce in post.  The benefit being that we don't simply throw away 1/3rd of a stop of highlights (which is what the camera does and results in the reduced maximum pixel saturation), and RAW processors are getting better and better at recovering actual detail out of these over exposed pixels (that the camera otherwise throws away).

And again, instead of using ISO 125, 250, 500, 1000 et all, we can under expose ourselves @ base ISO's and selectively increase exposure in post.  Here we have the benefit of only increasing mid tones and/or highlights that have a better Signal To Noise Ratio (read: less noise) then the shadows, while leaving the shadows in tact.  While the camera at these ISO is just blindly increasing the entire exposure by 1/3rd of a stop, including the shadows where noise is greatest.

RenatoPhoto

Quote from: Yomommassis on June 03, 2013, 04:49:50 AM
Maybe I'm doing something wrong but I feel like my histogram readings are always way off..like..WAY off

On ETTR I use the option Always on and take a couple of pics to find the right solution.  I know this works very well but on earlier builds I was not getting the right exposure using the Press Set function.  So test it both ways and see if you are getting the same results.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Yomommassis

Btw I tried changing it to "Always ON" and now autoETTR seems to give me better results :] ^^

@Audionut
man you're really blowing my mind with all this ISO stuff, that was a decent rabbit hole of information

if you don't mind I think I understand:
1. lower ISO = higher SNR (good), more dynamic range (good), more banding (bad)
2. higher ISO = lower SNR (bad), less dynamic range (bad), less banding (good)

3. increasing exposure in post tends to lead to more noise in shadows (plus banding if lower ISO)
4. ideal to ETTR and decrease exposure in post

4. base ISO's have less noise due to no digital manipulation
5. intermediate ISO's have more noise due to digital manipulation

6. 160 ISO Increments (160,320,640...) have lower dynamic range compared to their base ISO counterpart
7. 125 ISO Increments (125,250,500...) have more noise compared to their base ISO counterpart

8. shooting a scene @ ISO 100 and increasing exposure in post is the same as shooting ISO 125
9. shooting a scene @ ISO 200 and decreasing exposure in post is the same as shooting ISO 160
but its always better to shoot brighter and decrease rather than shoot darker and increase

but how does this all translate for video?
is base ISO strictly better for RAW or does it also apply to h.264?

I was always told that for video 160 ISO increments (160,320,640...) had the lowest noise and were the best ISO's to use
but now I know they actually decrease the overall dynamic range? and are less ideal over the base ISO?

my life is a lie!  :-\

Wouldn't it be best to set the camera for full ISO increments then?
and suddenly the T2i only having full ISO increments natively doesn't seem so bad?
EOS 5D MkIII (ML alpha 3)| EOS 5D MkII (ML v2.3) | EOS 7D (ML alpha 2) | EOS T2i (ML v2.3)
f/2.8 70-200MM L USM | f/1.4 50mm USM | f/1.4 35mm L USM | T/1.5 24mm
Zoom H4n | Rode NTG2

Audionut

That all looks good to me Yomommassis  :)

A couple of points I feel deserve some more attention.

1.  While the lower ISO's have more banding, IMO, that banding is so far down in the shadows (the read noise), that it doesn't really play a part.  Of course, I always ETTR, and I'm not one for raising the shadows 36 :P stops in post.

3.  Correct.  Cameras have a defined Signal to Noise Ratio that decreases exponentially in the shadows.  When increasing exposure in post, the noise comes along for the ride.

5.  The reduced ISO's (160, 320 et all) will have less noise.  But it doesn't come for free, it comes with reduced dynamic range also.  And it's not doing anything that a competent shooter can't do with base ISO's and ETTR better.  The increased ISO's (125, 250 et al) are a complete and utter waste of time IMO, and will have more noise.

As a further explanation on increased ISO's and their noise.  The benefit of ETTR comes from more photons (light) hitting the sensor (better SNR) (see: Exposure Triangle).  A user shooting at an increased ISO is going to be shooting with either 1/3rd faster shutter, or 1/3rd stopped down aperture to compensate for the 1/3rd increased ISO.  Less photos hitting the sensor, more noise.

8 and 9.  Correct, without the disadvantages that come from letting the camera do it.


I would still recommend the reduced ISO's (160,320 et al) for h.264 encoding as the ISO noise is more prominent IMO (in h.264), probably from all the resizing that happens with the conversion from Raw to h.264.  And IMO, that 1/3rd of a stop over-exposure and reduction in camera (noise reduction) (especially in the shadows), outweighs the 1/3rd of a stop of reduced dynamic range.

With Raw video however, base ISO's will always be best as we have access to each Raw frame with non destructive editing.

My camera is set to 1/1 (base) ISO's.  There's less you have to scroll through  ;)

P.s.  Glad you got it working with always on.  Have fun, it works really well.  It's the single best feature of ML for a photo shooter IMO.

a1ex

There was indeed a small accuracy problem for LV photo mode histogram, caused by ExpSim (the actual LV exposure settings were not the ones you have set for taking the picture, but some others that resulted in equivalent brightness). Since analog ISOs are in full stops, and digital ISOs are not applied to LV RAW data, this resulted in roundoff errors that caused auto ETTR to hunt sometimes.

Now, auto ETTR is spot on, at least on my 5D3.
https://bitbucket.org/hudson/magic-lantern/commits/ad41a4a97c5c
https://bitbucket.org/hudson/magic-lantern/commits/39727af61c67

Audionut

I'm not sure why digital ISO's are effecting AETTR when it's only using full stops.

In (movie mode) LV with raw_rec enabled (AETTR working in LV), it's only using full stops.
But if you don't have AETTR enabled in (movie mode) LV and you're using digital ISO's, the histogram is now incorrect.  It's not reflecting the digital pushes and pulls when shutter and aperture do not change.

Also, any thoughts on the dynamic range indicator reflecting the 1/3rd less dynamic range with digital pulled ISO's (yes I'm being picky).

ninja edit:  Ok, I only just realized that there is photo mode LV, and movie mode LV.  ::)
Photo mode LV is showing correct histogram changes with digital ISO's, movie mode LV is not.

The dynamic range indicator doesn't work in photo LV.

Both LV mode histograms are around 1/3rd (edit:  probably closer to 2/3rds actually) under exposed now.  Movie mode LV was accurate before iirc.

I can't test the set button (I use that for ISO), but double click half shutter works well except for the slight exposure error as noted above.

a1ex

In movie mode, digital ISOs do not affect the raw image at all. So, the movie raw histogram is now corect and 100% accurate (for raw videos, of course).

In photo mode, I want the photo LV histogram to reflect exposure changes accurately, not rounded to 1 stop. The accuracy issues appear because photo LV has different settings, not the ones you have dialed with scrollwheels. So, it's just an approximation. But I want this approximation to be as close as possible to the real thing.

Between LV ExpSim histogram and the CR2 raw histogram I notice a difference less than 0.3 EV (5D3: mostly 0.2, 5D2, mostly 0.1 EV). I think this is normal; LV and photo use different sensor scaning methods, different exposure settings, less pixels in LV and so on.

Another accuracy fix, this time for photo LV highlight warnings:
https://bitbucket.org/hudson/magic-lantern/commits/c6c797e43699

Audionut

It's much closer now.  My suggestion would be to make it err on the side over overexposure rather then underexposure.  It's better to get your resulting file 0.2 EV or 0.3 EV under maximum pixel saturation, rather then slightly over.   I realize I'm splitting hairs though!

Audionut

Sorry, I was being lazy.

+        raw_info.white_level = raw_info.white_level * 3413 / shad_gain;

My maths says that's 1/3rd of a stop.   I'm sure I'm doing it the long way round (if it's even correct anyway).

4096/2 = 2048/3 = 682.6

That's closer, but still a little on the under side.  I'd play with numbers until I found something I like and throw it at you, but you seem to prefer some maths behind it.  Did I mention I was splitting hairs.