Full-screen histogram WIP

[a1ex]

Something like this?







Topic split from here.

[Lars Steenhoff]

Looks nice! 
I would think if you make it bigger why not use the whole horizontal space. then you know left of the screen is under and right of the screen is over, as long as we are in the middle we are good.
What I mean by this is why the histogram is aligned on the right side if the screen instead of centred.

Also the underexposed grey fill looks less distracting of its on the left side of the screen instead of in the middle right?

[a1ex]

To keep things simple (integer multipliers), I'd have to resize the small histogram from 128px to either 120 (720/6) or 144 (720/5) or 180 (720/4). No big deal - with the first choice, the histograms look like this:



A bit cluttered for my taste after enabling the zebras, but I'm not sure what to change. The exposure indication is harder to find (low contrast); it's obvious in the small version (being white on black also helps). Maybe it should be moved at the bottom and displayed with black background in the large version as well.

edit: updated second screenshot to show both versions.

edit: the top bar somehow vanished, but to me it looks cleaner without it; what about making the EV grid lines smaller as well?

[Audionut]

That last example looks schmick.

[garry23]

Looking forward to this feature being in the experimental builds, ie to try out.

A thought I had, as a stills photographer, is: Once composition is fixed, then looking at the scene becomes secondary, hence can we make use of toggling through INFO to provide a view of the large histogram on a white background?

Trying to read the histogram over a scene is not ideal.

In my focus bar script I try and do this, but I'm afraid, using 'just' Lua, I do get conflicts in writing to the screen. I've tried tracking GUI states, with camera.state, but not very well.

Just a thought.

Cheers

Garry

[heyjoe]

Something like this?

Yes!
I am really happy to see how things are improving. Great work. I also like the DR indication.

A bit cluttered for my taste after enabling the zebras, but I'm not sure what to change.

Perhaps add options in menu, so the user can choose what elements to have displayed to their own taste. FWIW colored zebras don't work really well while shooting. Example: shoot something green (leaves) and see green zebra on top of it is very difficult to see. Pretty much the same happens with other channels, so maybe another form of zebra display could be good to, e.g. inverted colors: magenta for G, green for R, yellow for B. Or maybe blinking.

edit: the top bar somehow vanished, but to me it looks cleaner without it; what about making the EV grid lines smaller as well?

Yes, looks better.

Can you also make an option to display only the top and/or bottom 1EV of the histogram but zoomed at full screen? While shooting we don't actually need full statistical analysis of the image (full histogram). What we need is to check exposure at the extreme highlights and shadows.

Do you think it would be possible (worth it) to have (an option) for separate display of the two G channels as histograms? Sometimes they are not the same and clip differently so I've been asking myself what the current G channel histogram actually displays.

[heyjoe]

BTW thinking further: for visual evaluation of histogram clipping in highlights it may be good to have (an option for) linear horizontal scale (like RawDigger) because with logarithmic it may be difficult to evaluate a hard peak vs smooth (non clipped) on the LCD. Example to illustrate what I mean:

Log horizontal and vertical (like ML currently) shows fairly smooth, may be mistaken for non-clipped:



But with linear horizontal see how well the clipping of green shows:



And if that is combined with a display of only the top 1EV I think there is simply no way to make a mistake (vertical grid lines spaced at 1/3EV may be helpful here in order to know how much to change exposure controls):



For evaluation of shadows it may be necessary to display log/log histogram of several EV. This is image is not a good example as it doesn't have much shadows but here is another example of an image which as both highlights and shadows clipping:




So I suppose ideally on the LCD we would need only the last 2 graphics combined in one, e.g. sth like this (rough ugly sketch, text is just for descriptive purpose):



Perhaps the shadows can be smaller, so that there is more room for highlights histogram, for example 25% width for shadows 75% for highlights or something like that.

[a1ex]

May I have the second image for tests?

[DeafEyeJedi]

That last example looks schmick.

+1 and very well said!

[Audionut]

Log horizontal and vertical (like ML currently) shows fairly smooth, may be mistaken for non-clipped:

That's not exactly an accurate example as the display shows 17 stops.  The point still stands regarding the advantage of increased resolution in the highlights.

What is the usefulness of the shadow area being presented in the manner that rawdigger does?
Personally I prefer the grey marking that ML currently displays.  I don't need to see the lack of bits being represented the way that rawdigger shows, only that there is simply not enough bits there, and that there is data being contained within.

I would like to see the current histogram shadow area being user adjustable along the lines of the zebra underexposure option.  So as a user I can decide, I don't want any data contained within the bottom three stops, and the grey overlay on the histogram will represent these bottom three stops, and I can then move the data in the histogram as needed with exposure.

[a1ex]

The linear zoom on the Y axis in the last stop makes sense - it's easier to judge the clipping status in tricky cases.

However, linear scaling on the X axis does not - at least to me. Why? With logarihtmic scaling on X:
- you can always read the offset in EV
- when exposure changes, the entire histogram translates horizontally.

Good luck doing that with a linear-X histogram

BTW, there is already a linear Y scaling option in ML histogram menu. It's off by default because whenever there's a large peak (strong highlight, large flat area, whatever), it will make all the other things look tiny. Try evaluating the exposure from it on the examples below.

While we are at linear scaling on Y, why not consider another "view" - the CDF (integral of histogram) ? In our case, it shows everything a linear histogram does, it requires no scaling (range is always 0 - 100%) and reveals a few other things that are not easy to read from the histogram:

- where the median value is on each channel (just look at the middle point)
- what percentage of shadows are under the noise floor (easy - vertical scale is linear)

The above points are exactly where AutoETTR evaluates the exposure (median, shadow percentiles).

The CDF doesn't do a very good job showing the overexposure (it's not obvious if only a small percentage is clipped), but a zoomed view on e.g. the last two stops (EV on X, linear on Y) shows exactly that.

The CDF is also a LOT smoother out of the box

Examples (left: log-Y histogram, middle: linear-Y histogram, right: linear-Y CDF).
All of them are log-X. The zoomed view shows linear-Y histogram for the last two stops:













Note: in all the above examples, the histogram was heavily smoothed, while the CDF is totally raw.


Wait, what? Smoothing?

RawDigger displays thin bars. ML spreads them to nearby bins, to get a continuous graph (though it's still jagged). To draw the lines nicely, I've applied some additional smoothing in shadows, and also wherever the number of data points was very small. You may want to know how much I'm smoothing, so here are some examples.

Top: log histogram. Bottom: CDF.
Left: raw (no smoothing, no spreading). Middle: shadow spreading only. Right: spreading + smoothing.
Test image: ISO 3200, 2-minute exposure (AutoETTR was a bit overzealous when picking this ISO).




In the CDF images, there's barely any difference! They are already smooth out of the box!

On the other hand, the histogram requires aggressive smoothing to look acceptable.

Let's try an extreme example - one where the raw CDF looks bad:




CDF already looks very good after spreading - there's no need for additional trickery.

Histogram:
- left side resembles RawDigger display (this one is totally raw)
- after spreading, there's still one gap left (that gap is actually in the raw data, so it's showing the truth)
- after smoothing, I still don't like the result (too many waves...)


P.S. It's refreshing to be able to program all this without one single shutter actuation

(everything was done in QEMU, with existing CR2 images as reference data)

[heyjoe]

May I have the second image for tests?

Unfortunately I can't send the raw file for which this histogram is shown but I can try to shoot another one with slight clipping of highlights and shadows if that is what you need for testing. Please let me know. Or it may be faster to use one of the studio samples from dpreview or imaging-resource.com.  This one seems to have slight clipping of shadows and highlights.

That's not exactly an accurate example as the display shows 17 stops.

Yes, because as the screenshot shows it is using Auto setting for EV scale on X-axis. With 14bit raw files we obviously can't have values above 2^14.

What is the usefulness of the shadow area being presented in the manner that rawdigger does?

Evaluation of shadow clipping based on the actual raw data. It is not necessary to have it identical (jagged). I just mean to have a "zoom" of the shadows too.

Personally I prefer the grey marking that ML currently displays.  I don't need to see the lack of bits being represented the way that rawdigger shows, only that there is simply not enough bits there, and that there is data being contained within.

I would like to see the current histogram shadow area being user adjustable along the lines of the zebra underexposure option.  So as a user I can decide, I don't want any data contained within the bottom three stops, and the grey overlay on the histogram will represent these bottom three stops, and I can then move the data in the histogram as needed with exposure.

Of course we don't want data in the lowest bits but sometimes it is not possible to avoid it. Example: in an outdoors shoot we can't control the fill light in the shadows of the far away trees like we can in a studio. So being able to evaluate that visually is a plus. Don't you think?

The linear zoom on the Y axis in the last stop makes sense - it's easier to judge the clipping status in tricky cases.

What cases? Generally log Y is better. If Y is linear it makes small clippings unnoticeable. Example (using this CR2):

Left - log X/ linear Y; Middle - log X/ log Y; Right - linear X / log Y:

However, linear scaling on the X axis does not - at least to me. Why?

Just look at the examples. It shows much better the peaks in clipping. Log X smooths it because logarithm naturally compresses the levels horizontally.

With logarihtmic scaling on X:
- you can always read the offset in EV
- when exposure changes, the entire histogram translates horizontally.

But the whole idea of having bigger histogram of highlights is the visual (human intelligence) evaluation of the shape of the highlights. While shooting we are not really interested in what is in the middle of the histogram and how it translates left or right. We want to maximize the use of the sensor where we have most levels (ETTR), right?

<CDF, smoothing etc..>

I agree that the raw without smoothing creates cluttering, so it looks better with spreading + smoothing (as long as that doesn't affect accuracy).
CDF... I don't know. Personally I am not used to it (perhaps most people too). It looks like a lot of wasted space is wasted just to display a smooth S-like line which doesn't really say what adjustment of exposure might be needed for ETTR. Perhaps it may be useful for shadows only.

Is it possible to see an example using the CR2 from above showing (similar to my ugly sketch):
Left 1/2 of the graphic: LogX/LogY of shadows (with smoothing)
Right 1/2 of the graphic: LinearX/LogY of highlights (the top 1EV, no smoothing)

I am really curious if this will work.

[a1ex]

But the whole idea of having bigger histogram of highlights is the visual (human intelligence) evaluation of the shape of the highlights. While shooting we are not really interested in what is in the middle of the histogram and how it translates left or right. We want to maximize the use of the sensor where we have most levels (ETTR), right?

Yes, but there are scenes where you have to sacrifice some highlights (think something with strong lights, or any scene where you may want to use Dual ISO). In this case, guess where you have to look - at shadows and midtones!

Look up previous discussions on Auto ETTR, SNR limits and related stuff.

Left and right translation is essential - you'll know how the histogram will look like after adjusting the exposure. At a glance.

Personally I am not used to it (perhaps most people too)

I'm not used to it either - I've read about it today

... smooth S-like line which doesn't really say what adjustment of exposure might be needed for ETTR ...

It does! Have you read this?

Is it possible to see an example using the CR2 from above showing (similar to my ugly sketch):
Left 1/2 of the graphic: LogX/LogY of shadows (with smoothing)
Right 1/2 of the graphic: LinearX/LogY of highlights (the top 1EV, no smoothing)

Linear X is not trivial in the current codebase (the histogram is built with log assumption from the beginning); will leave it as an exercise after committing the code

[Audionut]

Thumbs up here for CDF.  Would be handy to have markers.  Perhaps on the right hand side.  Median and 5th percentile?  If you can draw a line from those markers to their position on the graph that would help to have an accurate determination.  Perhaps even allow user adjustment (think SNR limits in ETTR).

It would probably be useful to have a fixed x axis also, to ensure the 'useful' data is displayed with the highest resolution, and we don't just end up with a graph that uses all of the available resolution to display an outlier.  By useful I mean the tail, and being able to see any valley's between clusters*.

Markers on the graph for infinity values would help to make it obvious that infinity has occurred.

You would need to rework the markers on the bottom of the display for CFD mode also, correct?

Thanks for the display top right a1ex (what the initial bugging you was all about).  It is nice to be able to see these things with eyes.

P.S.  If you send through some images when the code is commited, I'll be sure to update the raw based exposure guide asap.

*edit:  hmmm, I'm not sure now.  Being able to visually see that an outlier is present has it's uses also.  Perhaps that outlier is a specular highlight, so then we could adjust exposure as necessary.  And removing the outlier also means that without careful consideration, one may mistake the visual representation of the data, and assume that useful data has not been quashed by specular highlight.

[a1ex]

Markers on the bottom? Both CDF and histogram have the same X axis (you can superimpose them if you like). In all the above examples, I'm showing 12 stops below white level.

The zoomed view also shares its X axis with the rest of the graph (that's why it's placed above). Another argument for log X: you can see the specular highlight clipping (in this case you have to clip it, otherwise you'll end up with a really dark image), and you can also evaluate how many stops there are until the non-specular highlights (that you probably don't want to clip).

In this particular image, you can probably push the exposure to the right by one extra stop, without much loss in that specular highlight. I've read the amount from the zoomed view - I couldn't have done that with a linear X.



(simulated a digital ISO push by 1 stop)

[Audionut]

The histogram on the left appears to have data extend to 7.5EV below saturation, the histogram on the right appears to have data that extends to 6.5EV below saturation.

These next two highlight the difference better.




When I look at all of the comparison images you linked, you can clearly see how the data matches between the different histograms.

[a1ex]

Yeah, that's the difference between log and linear histograms. CDF is linear - maybe it's worth "compressing" it somehow?

Code: [Select]

octave:1> f = @(x) asin(2 * (x - 0.5)) / pi + 0.5
octave:2> p = [0.01 0.1 1 10 50 90 99 99.9 99.99]'
octave:3> printf("%5.2f -> %5.2f\n", [p, f(p / 100) * 100]')
 0.01 ->  0.64
 0.10 ->  2.01
 1.00 ->  6.38
10.00 -> 20.48
50.00 -> 50.00
90.00 -> 79.52
99.00 -> 93.62
99.90 -> 97.99
99.99 -> 99.36

That would be a nonlinear scale on Y that reveals the small "details" in a way similar to log. I'm not sure how to come up with a well-defined mapping based on pure log (or whether that even makes sense).

It does a much better job at revealing highlights and shadows, but really needs grid lines. You can even "compress" the CDF twice, for "zooming" in the small percentiles even more:

Code: [Select]

octave:4> printf("%5.2f -> %5.2f\n", [p, f(f(p / 100)) * 100]')
 0.01 ->  5.08
 0.10 ->  9.06
 1.00 -> 16.25
10.00 -> 29.90
50.00 -> 50.00
90.00 -> 70.10
99.00 -> 83.75
99.90 -> 90.94
99.99 -> 94.92

Left to right: log histogram, linear CDF, "log" CDF compressed once, "log" CDF compressed twice:







To me, the last one provides at least the same information as the zoomed view! In addition:
- it' not affected by autoscaling (the zoomed view is)
- I can see at a glance the percentage of clipped highlights and noisy shadows (at any noise threshold for the latter)
- it's obvious where the curve starts, and matches the log histogram at that
- it shows tiny changes even better than linear zoom (look closely at first image, top right)

[Audionut]

Hmm, so how does this fit in?



I'm thinking to use markers like in the above image, with the median and minimum representing something similar to ETTR SNR.
This way we could see what data falls below, or lies above the ranges.

But I didn't realise CDF was linear, and it would be nice (I guess) to see how the data falls regarding EV scale, like all of the other ML exposure feedback tools.

Transforming linear to log is significantly above my pay grade. 
I can google search.  But still no idea if it helps the current situation.

[a1ex]

Here's a picture:



X scaling stays the same (EV), Y gets compressed to reveal the smaller percentiles.

[Audionut]

Thanks a1ex, now it makes sense.  The spike at the right hand side of the clipped compressed CDF graph was throwing me off.  And for some reason I was expecting the shadow area to look different also.

I like it.

[heyjoe]

Yes, but there are scenes where you have to sacrifice some highlights (think something with strong lights, or any scene where you may want to use Dual ISO). In this case, guess where you have to look - at shadows and midtones!

But if you have to overexpose highlights, then midtones become highlights, so you still evaluate highlights and control how much you overexpose. It is still highlights control, is it not?

Left and right translation is essential - you'll know how the histogram will look like after adjusting the exposure. At a glance.

That's true. The problem is that we don't have a 30" LCD on the camera to see everything So I am just thinking about using the available space for the most critical things. BTW if CDF is used - the exposure movement becomes more complex, not just left/right.

I'm not used to it either - I've read about it today

It does! Have you read this?

Yes, we also studied that in the university. I just mean that from usability viewpoint it may need special education for the end user Also from which picture is it really simpler to evaluate clipping peaks:



Yes, the CDF shows a flat area at the end and for non-clipped ETTR one should aim for not having any flat area on the right but I suppose it needs testing in real conditions. So maybe the best thing to do is to have an option in the menu what to display: histogram or CDF.

Another thing which comes to mind: perhaps with CDF it will be difficult to evaluate shadows due to the log X axis and the linear nature of raw data which always has less values in shadows. Which again comes to screen space - 80% of the CDF (or histogram) displays midtone data which is not critical for exposure control. Won't it be better to have a smaller full CDF/histo and a zoom of the critical areas?

Linear X is not trivial in the current codebase (the histogram is built with log assumption from the beginning); will leave it as an exercise after committing the code

I am sure you know how to make it

Markers on the bottom?

Or maybe a thin diagonal straight line?

The zoomed view also shares its X axis with the rest of the graph (that's why it's placed above).

Hm. But what is it zooming then? My initial idea was to have it stretched on both X and Y (damn.. I am repeating myself too much ) Another thing: having the vertical clipping line right at the edge of the screen makes it difficult to read. Maybe it needs some spacing. Just imagine shooting in a bright sunny day and having to read all that visual info.

Yeah, that's the difference between log and linear histograms. CDF is linear - maybe it's worth "compressing" it somehow?

I think that compressing CDF in a way which reduces the critical areas (highlights and shadows) is contrary to what we need. It is also contrary to the advantage of CDF to display outliers much better than a histogram.

BTW: How about a new heuristics based on CDF?

[a1ex]

BTW if CDF is used - the exposure movement becomes more complex, not just left/right.

False, see #14. With log scaling on X, the movement is always a translation (one notch on the grid = 1 EV).

My initial idea was to have it stretched on both X and Y

The current scaling already lets you evaluate the exposure with a resolution of 1/60 EV (if you count pixels), or ~0.1 EV (if you look at it). I don't see the need for further zooming on X.

BTW: How about a new heuristics based on CDF?

Already done... 1 month ago

The exposure hints from Auto ETTR are also CDF-based.

BTW - please try cutting down the "splitting hairs".

[Audionut]

Moved over to mobile, trying to quote specific parts of a large post is retarded.  Hopefully viewers can follow.

The flat areas of the CDF graph indicate areas of no signal.  The reason why there is that large flat area in the highlights in that graph is because of the specular highlight.  There is clipped data, then there is no other data until -1EV.  a1ex posted an example earlier with a simulated 1EV gain of that test image, and the resulting CDF graph.  The principle of the flatness of the CDF graph applies throughout the entire graph.  If the graph tends towards flatness, it translates to limited data in that area.

Compression of the CDF graph does the exact opposite.  It expands the highlights and shadows.

ML is provided free as in beer.

Speaking of the flatness of CDF, perhaps zoomed histogram on top left, with the CDF expanded to fill the vertical space also.  Possibly beginging to be cluttered.  Garry made a good point above, but I guess that needs to wait until that image processing pipeline is fully understood.  I have mentioned how exciting all that backend work you guys have been doing is, right.  qemu has made you even quicker a fancy images 

[heyjoe]

False, see #86. With log scaling on X, the movement is always a translation (one notch on the grid = 1 EV).

Hm. You are right. I didn't think about the X-log.

The current scaling already lets you evaluate the exposure with a resolution of 1/60 EV (if you count pixels), or ~0.1 EV (if you look at it). I don't see the need for further zooming on X.

Sounds great. Could you at least add a little spacing (few pixels) left and right of the graphics to avoid having vertical lines being placed right at the edge of the LCD?

Already done... 1 month ago

The exposure hints from Auto ETTR are also CDF-based.

That's great! I guess I was confused because of that:

I'm not used to it either - I've read about it today

BTW - please try cutting down the "splitting hairs".

Please don't look at it this way. The usability suggestions are based on the actual ETTR tests. BTW what is the function of the small vertical colored ticks?

Compression of the CDF graph does the exact opposite.  It expands the highlights and shadows.

Are you sure? The compressed versions show shorter flat areas for clipped shadows and highlights.

[a1ex]

Well, CDF is simply the sum of each bin on the histogram. So, wherever the exposure algorithm sums the bins in either direction, we can say it looks at the CDF.

So, even if I wasn't used with CDF as a visualization tool (until reading the linked article), that didn't stop me from using it in the algorithms (even if I didn't know it was called that way). That's why, back then, I said Auto ETTR computes the exposure by looking at the histogram (which is not wrong either - as the CDF is simply the "integral" of the histogram).

The tiny markers are the medians. Maybe I should find some other representation.

BTW, for your side by side comparison, try this picture:



(reason: both of them have log scaling vertically; we are comparing the two horizontal scaling methods)

From the left picture, from the CDF graph, I can tell:
- there is a specular highlight, less than 0.1% of the image area (this will be more obvious after adding markers)
- there's about 1 stop until the non-specular highlight data on the green channel (so if I decide to clip that highlight, the exposure can be pushed by 1 stop to the right)
- if I decide to capture that specular highlight, I'd have to underexpose by a couple of stops (look at the slope in the flat area)
- a small percentage of the shadows (about 1%) is buried in the noise floor
- midtones are about 5 stops above the noise floor (this lets you evaluate the overall noise levels in the image)
- if I decide to clip that highlight and also sacrifice the green channel (thus relying on highlight recovery), there are about 2/3 additional EV until the non-specular highlights of R and B (you may want to do that if midtones are too low).


From the right picture, I can tell there's a specular highlight clipped.

I can tell nothing about its area (well, I can after doing the "integral" of that peak mentally - is that 2 or 3 pixels wide?)

It's hard to tell how many stops I can move in either direction (to either capture that highlight, or discard it and maximize the SNR in the rest of the image) - I need to transform from linear to log mentally, and as Audionut said, it's not straightforward.