5D iii CDNG Lossless vs Uncompressed file sizes

[belalkhansocial]

Been exploring the differences between various resolutions and bit depths using Magic Lantern video files converted to Cinema dng Raw lossless vs uncompressed.

I feel like sticking to lossless is the best option, especially when using the workflow showcased in this video.

Let me know your thoughts and considerations you take when working with these files.

[vastunghia]

Thanks for this. Not sure tho, are you suggesting that lossless Raw is losing some information when compared to uncompressed?

[belalkhansocial]

The opposite actually. I don't think there's any difference between lossless and uncompressed except for file size really. If there is, then my skills are simply not there to notice the difference.

The thing I'm curious about is genuinely finding the difference at least within my skill set limitations between 10 bit and 14 bit color.

[DarkTrapStudio]

Been exploring the differences between various resolutions and bit depths using Magic Lantern video files converted to Cinema dng Raw lossless vs uncompressed.

I feel like sticking to lossless is the best option, especially when using the workflow showcased in this video.

Let me know your thoughts and considerations you take when working with these files.

If you messed up too much the white balance, to be sure to still have the Raw data it has to be done into the MLV app (cause if you transcode to uncompressed, is it gonna read camera sensor the same way that MLV ?), and Im not sure because if Im not mistaken ML RAW is compressed (lossless?) Raw so there a lot of considerations to be taken here, differents Raw Codec will not behave and recover the same things.

And to do A/B test youll have to do indoors controlled conditions test cause any differences would make subjectives consideration.

But Its cool to see the files sizes differences between all theses parameters, and I will look more into the details between Lossless and Uncompressed, but it can be how Final Cut read/manage the data, maybe it has to be set manually, like Resolve that doin it in auto, Eos M raw sensor not being listed in the camera color science/sensor decoding is where it fails maybe.

[DarkTrapStudio]

Here is some practical explanations with test that objective what I said :
https://youtu.be/t6-1XXJSPQw?si=LQk1zM3o5pCZYlO_&t=71

Edit : Sorry I though It was the correct video but he don't show just explain it briefly, here is another video that shows it with real world tests.
https://www.youtube.com/watch?v=qPB6wrVKHiw&t=248s

[vastunghia]

The opposite actually. I don't think there's any difference between lossless and uncompressed except for file size really. If there is, then my skills are simply not there to notice the difference.

The thing I'm curious about is genuinely finding the difference at least within my skill set limitations between 10 bit and 14 bit color.

Aha, I see thanks. And FWIW, I ran some tests on this topic as well:

My humble contribution to the discussion [...]

Based on which, I find that 12 bit can be a compromise in some cases, while 10 bit is not viable.

Personally, though, if I have to choose between, say, 3K @14 bit and 3.5K @12 bit, I will go with the former. After all, if all I want is 4K, I can use my iPhone.

Ciao

[DarkTrapStudio]

The opposite actually. I don't think there's any difference between lossless and uncompressed except for file size really. If there is, then my skills are simply not there to notice the difference.

The thing I'm curious about is genuinely finding the difference at least within my skill set limitations between 10 bit and 14 bit color.

You may have to need for a 100%sRGB monitor/P3 to be able to see it without grading, being able to see the color shift is hard if not experienced, I only began to really see it/experience it when understanding the meters in the color page of Davinci to decode what my brain was seeing, and to be sure about what is wrong, then you are quicker and quicker like any skill.

[vastunghia]

You may have to need for a 100%sRGB monitor/P3 to be able to see it without grading, being able to see the color shift is hard if not experienced, I only began to really see it/experience it when understanding the meters in the color page of Davinci to decode what my brain was seeing, and to be sure about what is wrong, then you are quicker and quicker like any skill.

Mmh... I think that uncompressed and lossless are the *same* thing. They contain the same information. Hence, if you spot any difference... that's your brains

As for 14 vs 12 vs 10 bit, please refer to the test I was providing in my previous post. You can definitively see the difference, even on a $100 sub-sRGB monitor. It is called banding, and it is the result of (metaphorically speaking) trying to use a much smaller vocabulary to describe tiny nuances of meanings in a language: you will end up with much lower richness in transitioning from one nuance to another (banding, that is), no matter what.

[Skinny]

Yeah banding and all kind of noisy shadows in 10bit, but 12 is just a little bit worse than 14, not as much though.
Subjectively if you place bit depth at the scale, it looks like:

10........................................12..14

Where 12 is much much closer to 14 than to 10. It is really like that.
It is just my opinion though

Someone even said 11 bit is a good balance between the image quality and file size, but I can't compare - don't have a camera that supports 11 bit.

[DarkTrapStudio]

Mmh... I think that uncompressed and lossless are the *same* thing. They contain the same information. Hence, if you spot any difference... that's your brains

Lossless is compressed, Uncompressed is not compressed. Just google it next time you are not sure of something don't spread it out please.

As for 14 vs 12 vs 10 bit, please refer to the test I was providing in my previous post. You can definitively see the difference, even on a $100 sub-sRGB monitor. It is called banding

Banding happens when you downscale, if Im wrong please prove me wrong.

And I was refering to belalkhansocial comments, which has nothing to do with banding, but being able to see the difference between a 10 bit frequency spectrum vs 14 bit, for a blue for example, so even with banding, you wouldn't be able to see the actual color with a crappy monitor, you would just see the banding. (I said crappy cause theses day they are awesome monitors for 100%)

Edit : 100$ not 100%

[vastunghia]

Yeah banding and all kind of noisy shadows in 10bit, but 12 is just a little bit worse than 14, not as much though.
Subjectively if you place bit depth at the scale, it looks like:

10........................................12..14

Totally agree. Not 100% sure, but I guess this must be somehow related to actual sensor DR, which is ~12 stops for most of the ML-enabled cams.

[vastunghia]

Lossless is compressed, Uncompressed is not compressed. Just google it next time you are not sure of something don't spread it out please.

Yup of course lossless = compressed and uncompressed = not compressed.

However, compression can be lossy (e.g. JPEG, meaning that you are losing information in the process, which cannot be recovered anymore at a later stage) or -- you know -- lossless (e.g. PNG is a lossless compressed image format; or think of ZIP archives when you compress your documents: you are not losing information of course, otherwise your Word document would not be recoverable when you deflate your compressed archive).

Lossless means exactly this, it is not losing (altering) information, it is just achieving a smarter way to store it in a smaller space.

So lossless = uncompressed in terms of recoverable information. I.e. if you see a difference, that's your brains.

Banding happens when you downscale, if Im wrong please prove me wrong.

Negative.

Banding happens, for instance, when you have a small palette of different blues (say, 2^10 different levels of blue instead of 2^14) and need to paint a sky with a very uniform tone of blue, showing only a very small and slow gradient between very similar tones of blue.

Let's say that your blues can go from 0 (black) to 1 (full brightness blue) and somewhere in your still / video frame your sky passes from 0.5 to 0.5006 over 11 pixels. With 14 bit, you are storing enough details to go like this:

0.50000 0.50006 0.50012 0.50018 0.50024 0.50031 0.50037 0.50043 0.50049 0.50055 0.50060

To realize this, just multiply the above by 2^14 = 16384, and you will see that this corresponds to different integers <= 2^14, that thus can be stored using 14 bit:

8192 8193 8194 8195 8196 8197 8198 8198 8199 8200 8201

This also means that every pixel, in that row of 11 pixels, will display a slightly different blue, providing for a very smooth and pleasant color gradient from 0.5 to 0.5006.

On the contrary, if you had only 10 bit, you could only go like

0.50000 0.50000 0.50000 0.50000 0.50000 0.50000 0.50000 0.50000 0.50098 0.50098 0.50098

Once again, multiply the above values by 2^10 = 1024 and you will realize that these become

512 512 512 512 512 512 512 512 513 513 513

So in the 10-bit case, you see that you are not representing the color gradient as nicely as in the 14-bit case: basically, you have only 2 levels of blue available (one of which is even brighter than the brightest level you are willing to represent). The rendered image will thus show a harsh change between the first 8 pixels being represented at a level of blue of 0.50000 and the last 3 pixels being represented @0.50098.

This is called banding.

HTH

Mr Google

[DarkTrapStudio]

Please be pretentious, not arrogant. Thanks for being clear on that point.

[Skinny]

I wonder if some kind of dithering could be implemented and if it will look better.
Probably LSB could be used as almost random source..
You know, in theory you have to dither every time you go to lower bit depth, not just truncate least significant bits..

[ilia3101]

Much more effective than dithering would be log encoding. I've dreamed about adding it to the MLV format for years.

Of course it would have to be done in transcoding, not possible to record that way in camera.

10-bit would be indistingiushable from 14-bit linear (maybe even 9-bit would be enough with how noisy these cameras are). Would be great for storage combined with lossless compression.

Edit: actually, I think custom encoding tables are already possible in DNG. Might be possible to add it to MLV App.

[a.sintes]

log encoding is actually a very good idea (BM & Sony cameras are doing it nowadays), actually what is so complicated to replace the actual 14bits to 12/10bits decimation with a log (e.g.: Cineon Log) profile conversion? sounds like a look-up-table to me? or is it more complicated?

[names_are_hard]

log encoding is actually a very good idea (BM & Sony cameras are doing it nowadays), actually what is so complicated to replace the actual 14bits to 12/10bits decimation with a log (e.g.: Cineon Log) profile conversion? sounds like a look-up-table to me? or is it more complicated?

It's not very complicated, but it may be too slow to be practical.  Might work.  Have to try it to see.  The current 14 -> 12 step apparently loses very little information, the low two bits are very noisy (on most supported cams?  More modern cams have better dynamic range...).

See an earlier thread about this: https://www.magiclantern.fm/forum/index.php?topic=5601.0
I don't have the time right now to read 75 pages to get all the info out.

Also see this recent interest in a fast log approximation routine: https://www.magiclantern.fm/forum/index.php?topic=26976.0

[Skinny]

What about picture profiles? They can do log or anything, if there is a place where the information could be read after picture profile processing it would be an ideal solution...


By the way LUT (I mean the real look-up table, an array) is probably the fastest way possible, if done correctly it can take only a few processor cycles

P.s. I may be wrong about look-up table, it seems like it depends on the system.. For example, on modern computers the access time to the ram is too slow.
My experience is mostly with 8 bit MCUs and a little bit of 32 bit MCUs but the memory there is integrated, so the access time is instant. I don't know how Canon cameras will behave
But I did a synth project in the past and eventually replaced almost every part of the code by luts, it was faster. Instead of one voice I could implement four.

[a.sintes]

thought the same about the picture profiles like Cinestyle but I don't know exactly how it's applied internally on the camera (H.264 encoding stage?) and I'm pretty sure it's designed to be a 14bits to 8bits conversion only.

LUT usage seems to be a good way to accelerate things, at least on the 5D3: I'm currently working on framed preview speed increase with very promising results using this technique (almost realtime grayscale preview in 3.5K), the main limitation being the memory size we can allocate (e.g.: need to downscale the preview resolution by two horizontally in color mode to avoid allocation failure).

[Danne]

Some good reading on picture styles.
https://www.magiclantern.fm/forum/index.php?topic=16299.0

[Skinny]

Interesting, but there is no information about whether it is possible to read the information after picture style processing, and save it to the card, without h.264 compression... But anyway

I used CineStyle on 550D but with any luts including official lut which comes with it, colors were just dead
It is not that obvious when you use it for video, but once I forgot to change the picture style and shot some JPGs with it, and then switched back to some of my usual profiles, later I couldn't make those JPGs look even somewhat close, even with luts and all photoshop corrections. The difference was so big..

After that I decided to get the look I wait in-camera first, and adjust it further I in post. Maybe I lack some skills or something but even youtube demos with CineStyle to my eye look somewhat flat in color, missing some punch and variations. I guess you just can't fit 14 bit range in 8 bit file and preserve all gradients at the same time.. But maybe I am just too noob to do it correctly, who knows

[a.sintes]

As you said reducing from 14bits to 8bits (plus lossy h.264 compression) cannot be magical, even when using the best profile :-)
Used Cinestyle a lot in the past and even if it's at least better for shadows & highlights recovery it can be really hurtful for medium & skin tones, which is understandable regarding the mathematics behind it.
Make way more sense when used in camera like BM to just downscale from 14 to 12/10bits, using a proper high-end ProRes 444 encoding.

That said, I checked the crop_rec code after discussing with Danne in order to try to get more clues about how the 14 to 12/10/8bits decimation actually works on the 5D3 and if I've understood well there's no obvious "image processing" stage before the MLV file recording that can be usable to insert some kind of LOG LUT conversion over each RGB components, the actual decimation trick relying over specific registers usage set directly to the camera internal DNG recording :-/

[names_are_hard]

I wouldn't be at all surprised if there's a log <-> linear hardware unit somewhere on the cam.  It's a pretty common conversion in processing images, and it's unlikely the official CLog upgrade works via software.