What does the Canon Luminance Histogram actually show?

[reox]

Might be a bit OT but I think maybe someone here knows this from reverse engineering... This is actually a cross post from here: https://photo.stackexchange.com/questions/132339/what-does-canons-brightness-histogram-actually-show I posted also some images there.

I wondered what the luminance/brightness histogram of Canon cameras actually show (in my case a Canon RP).
I tried to replicate the histogram from the embedded JPG of RAW files and it matches the RGB histograms quite nicely but the luminance histogram looks different.
I know that the histogram is some kind of weighted RGB sum, but what exactly Canon does I have no idea.
What I find interesting is, that the histogram on the camera has these spikes - which I think are from integer calculations and binning artefacts. I could see similar things if I only used 6 significant bits for the calculation.

[Walter Schulz]

Before we go there: Please add ISO information. Intermediate ISO (like 160, 240) are calculated from full ISO and will not give proper data.
And avoid noise reduction settings.

Unfortunately RP is not yet supporting "full-scale" ML and ML's data analysis capabilities and our knowledge about Digic 8 designs is quite limited.
Maybe asking Bclaff (runs Photons to Photos site and active Dpreview forum member) will shed some light. Sorry for the pun!

[reox]

I see, so it is indeed dependent on the camera firmware how the brightness/luminance is calculated from the RGB values?

Is it know what the 5D mkii does? I have that camera as well and see the same patterns of the peaks and so on. So maybe the implementation is similar.

[names_are_hard]

If you really want to know, you'll probably have to dig around in the rom code to find out.

It makes sense to me that it could be an artifact of integer maths; these cams don't have hardware for floating point.

[Walter Schulz]

I see, so it is indeed dependent on the camera firmware how the brightness/luminance is calculated from the RGB values?

Sorry? How you jumped to this conclusion?

Is it know what the 5D mkii does? I have that camera as well and see the same patterns of the peaks and so on. So maybe the implementation is similar.

AFAIK it is pretty well known for 5DII and other early cams. Some digging around in the web will show. Cambridge in Colour, Photons To Photos, RAWdigger, ...
Just occupied with some cam disassembly (70D, argh!).

[reox]

Sorry? How you jumped to this conclusion?

Oh sorry, I think I misunderstood you. Someone at stackexchange mentioned that it is different per vendor and camera.

AFAIK it is pretty well known for 5DII and other early cams. Some digging around in the web will show. Cambridge in Colour, Photons To Photos, RAWdigger, ...

Thanks for the hints, I'll have a look! So far, I only found the information that it is "luminance" or "brightness" but that is a pretty broad term...

If you really want to know, you'll probably have to dig around in the rom code to find out.

It makes sense to me that it could be an artifact of integer maths; these cams don't have hardware for floating point.

I did some disassembly years ago and might give Ghidra a go.

By any chance, is there maybe some documentation of integer based luminance caculations (except the (2R + 3G + B) / 6 and co)? There are some optimizations using shifts (for example these: https://stackoverflow.com/questions/596216/formula-to-determine-perceived-brightness-of-rgb-color/596241#comment46218947_596241) but I guess there are even more efficient ones?

[names_are_hard]

Documentation of how Canon implements it?  None that I know of, but I haven't looked at all.

[reox]

Documentation of how Canon implements it?  None that I know of, but I haven't looked at all.

Maybe not by Canon itself, but for example for computer graphics, there are pages describing efficient algorithms for various things using only integer arithmetic.
I thought maybe someone has published something like that. You never know if Canon Devs have not found the same source

[Walter Schulz]

Quick search online (not in ML code) tells cam should use NTSC (= ITU BT.601 Luminance = 0.299*R + 0.587*G + 0.114*B). Haven't tried it. I think you did and it failed, right?

[a.sintes]

there's this in ML code:

Y = COERCE(((217) * R + (732) * G + (73) * B) / 1024, 0, 255); // rgb2yuv422_rec709 function
Y = COERCE(((306) * R + (601) * G + (116) * B) / 1024, 0, 255); // rgb2yuv422_rec601 function

Y being the the luminance channel in YUV422

don't know if it's helping...

[names_are_hard]

Maybe not by Canon itself, but for example for computer graphics, there are pages describing efficient algorithms for various things using only integer arithmetic.

Why do you want this?

In fact, can I get some more detail on what you want to find out, and why?  You don't actually ask a question in your first post.  The closest you get is "I wondered what the luminance/brightness histogram of Canon cameras actually show" - for this, you'll need to check the rom code.

Do you want any somewhat fast alg?  Do you want Canon's exact alg?  Why?

[reox]

Why?

I'm sorry, maybe the initial post was a bit confusing.

I was reading the manual of my camera the other day, because I was looking for some settings. I stumbled across the description of the histogram and the manual simply says "brightness". I found that a bit peculiar, because "brightness" is not something that is defined by a law of nature. Furthermore, the term is ambiguous and can mean different things. I was curious and took some pictures and checked the histogram that was shown in the preview. Then I took the same image, extracted the emebedded JPG from the RAW file and looked at the histogram for the RGB values as well as different types of luminance calculations.
The RGB histograms looked very similar to those shown by the camera - which I took as (weak) evidence that the histogram shown on the camera's preview are from the same source (the embedded JPG). Somewhere I read, that Canon cameras will always use this embedded JPG and never touch the RAW file. However, to my surprise, I could not replicate the "brightness" histogram. While the overall shape of the histogram is the same, when I use the ITU BT.601 luminance equation (or variations), I cannot replicate the details of the histogram.

So I started searching what kind of "brightness" Canon could mean. But I found always the same equations, basically the BT.709 and BT.601 ones.
The Canon histogram always looks a bit spikey - which I could not reproduce when using floats. But, when using integers and messing with the bin size, I can create something similar to Canon, but not exactly.
My hypothesis is, that they multiply the values such that they maybe stay within a unit16 but then do not apply the division and simply use 256 bins on the 0..65535 range (or something like that).
Another variant might be to take only 5 significant bits for the calculation, then you could do it inside an uint8 - this creates massive amount of spikes but the histogram also looks then very different. Thus I doubt that they do it like that.

Do you want any somewhat fast alg?  Do you want Canon's exact alg?

My thinking was, that there might be a published fast integer based calculation for "brightness" which might be as well used by Canon. I know from other fields, that there are certain algorithms that are published, which are used in products.

All in all, I'm just curious to know what Canon's "brightness" is. But you are right, I'll need to check the ROM.

Quick search online (not in ML code) tells cam should use NTSC (= ITU BT.601 Luminance = 0.299*R + 0.587*G + 0.114*B). Haven't tried it. I think you did and it failed, right?

yes, both BT.601 and BT.709 look somewhat similar but I cannot reproduce the histogram exactly. Maybe they use just different coefficients than these ones:

there's this in ML code:

Y = COERCE(((217) * R + (732) * G + (73) * B) / 1024, 0, 255); // rgb2yuv422_rec709 function
Y = COERCE(((306) * R + (601) * G + (116) * B) / 1024, 0, 255); // rgb2yuv422_rec601 function

Y being the the luminance channel in YUV422

don't know if it's helping...

With my python script, these two variants look very close to the result I get when using the float numbers - so these equations are a very good approximation (too good to be the Canon histogram )

As from the camera:


From the embedded JPG:

https://i.imgur.com/R5Rosjf.png

[names_are_hard]

The initial post wasn't confusing, it just doesn't ask anything.  Hence, I was trying to confirm what you wanted to know.

More important was *why* you wanted to know this, because that informs *what you actually want to know*.  People are on average quite bad at asking questions and often don't ask in the right way to get the answer they want.

If the real reason truly is just because you're curious what Canon use, then there's only one way to answer the question: check the code to find out what they do.  You don't want to do anything with this information once you have it?  It's hard to imagine that you really care that much about this particular histogram.  If they're approximating it in order to be fast, they won't use the approximation for anything important, so it doesn't matter.

[reox]

You don't want to do anything with this information once you have it?  It's hard to imagine that you really care that much about this particular histogram.  If they're approximating it in order to be fast, they won't use the approximation for anything important, so it doesn't matter.

The answer "it is some sort of approximation of a weighted sum of the RGB components (possibly using the coefficients defined in ITU BT.601) of the processed JPG" is already something I wanted to know and satisfies me for the moment. It is more information than "it is brightness".
However, I'm still curious why it looks how it looks. These spikes you can observe in most of the histograms did not made sense to me. Is that a display issue? From binning? Integer arithmetic? ...
But yes, this is a purely academical question and I do not want to use it for anything specific, just to quench my thirst for knowledge

So far, thanks for all the help and suggestions here! Looking into the ROM code sounds like a fun activity for a rainy weekend, and I'll definitely try it.

[a.sintes]

Note also this kind of luminance histogram is generally displayed on a Log scale, it can explain the shape

[names_are_hard]

Trying to find the specific code for rendering that one graph is likely to take more than a weekend   But it is fun if you're in the mood for it.

Human perception of brightness is a weighted sum of red, green and blue components, pretty much.  So that makes sense.

[reox]

Trying to find the specific code for rendering that one graph is likely to take more than a weekend   But it is fun if you're in the mood for it.

haha, true... Yesterday evening I read the threads how to do ROM dumping to get started but I was not brave enough to try it on my RP yet