UGLY clipping samples

[ilia3101]

Does anyone here ever have raw images with ugly clipping? Be that simple highlight clipping, blue light clipping in weird ways, or red lights going yellow, anything...

I'm looking for examples of these kind of images, they can be from any camera (from any brand) and in any raw format. I want them all. Let's make a collection of raw files with difficult colours. It would be really helpful for developing and testing raw conversion software (MLV App in my case)

Here's a particularly extreme example of what I want:



Would love more like this!


Fun fact: Using 3x3 matrices for raw conversion (which almost all software currently does) produces negative Luminance (Y) values for pure blue light. So why doesn't blue show up as black? ... per-channel clipping! (most often in ProPhoto RGB)




This sums up what I am looking for very well:

In this thread, we are looking specifically for sample images with extreme out-of-gamut clipping issues (extremely saturated colors that are rendered obviously wrong). Such issues can be found when using highly saturated light sources such as colored LEDs, or with concert photos/videos (with unusual lighting), or highly saturated scenes/subjects, such as flowers.

[a1ex]

Possibly related: https://twitter.com/CT_Bergstrom/status/1333614982647353347

The topic is of interest to me as well, btw - just not actively researching it right now.

My earlier hackish attempt - back then, I didn't know what I was doing:
https://www.magiclantern.fm/forum/index.php?topic=5197.0

Found two sample images (I can dig for more, if needed):

IMG_0307.CR2
IMG_5207.CR2

Some stuff I came across, but didn't try yet:
https://bottosson.github.io/posts/gamutclipping/
http://www.brucelindbloom.com/index.html?UPLab.html

Maybe it helps.

[2blackbar]

Yes i had that especially at night when filming traffic lights or cop lights, i dont have dngs now tho.

[Skinny]

Sometimes I get this ugly pink area if the highlights clipped really hard (like on the sun, for example) even when "highlight reconstruction" is on..

This is how it looks (click for full size):


One-frame mlv and the preset I used:
https://gofile.io/d/q6sagD

It is clearly visible even with the default MLV App settings, and if you bring down the highlights and turn on reconstruction it becomes really obvious.
I used 5D2 and 12 bits.

[a1ex]

That's the "black sun" effect - see e.g.:
- pco.de/.../20140412_FOM_CameraTutorial_PCO_fin_pdf.pdf p.31
- imagesensors.org/.../7-02__geurts.pdf p.3
- https://www.physicsforums.com/threads/black-sun-effect-in-cmos-sensors.799334/

For this particular image, the easiest way to render it would be to reduce the white level. If your raw processing software has a slider for that, use it. In mlv_dump, the relevant option is --white-fix. For this particular file, 3300 looks about right to me (from the default of 4050).


In this thread, we are looking specifically for sample images with extreme out-of-gamut clipping issues (extremely saturated colors that are rendered obviously wrong). Such issues can be found when using highly saturated light sources such as colored LEDs, or with concert photos/videos (with unusual lighting), or highly saturated scenes/subjects, such as flowers.

These colors are more colorful than the color space is able to encode.

Here's an example of an image that's clearly not clipped in RAW, but rendered incorrectly (clipped) in sRGB. I couldn't find the original raw file, but didn't look too hard.

https://www.dpreview.com/forums/post/62681348

The best way to deal with these issues, in my opinion, would be with some kind of gamut compression - but you'd have to carefully choose a suitable color space for that. In particular, I can tell you for sure that CIELAB is not the right color space for this purpose (long answer in the links shared earlier).

[Walter Schulz]

I'm looking for examples of these kind of images, they can be from any camera and in any raw format.

Any camera = any brand?

[Kharak]

Try the same clips in Resolve, yes they will have pinkish highlights, but not as saturated/“artifacty” like your example.

I usually mitigate it by turning luma mix off and pushing green in to the highlights, easy fix. All in a RCM workflow ofcourse.

I cant find the thread by baldavenger, but he made a post about altering the dng’s in order to get perfect white and a full signal straight in to Resolve. In his workflow highlight reconstruction wont work anymore, so pulling highlight slider wont have any effect. Maybe it was a combination of one of his DCTL’s and dng manipulation. I can’t recall and I never tried it.

EDIT: How do your clipped blues look on the waveform? I assume clipped, try adding Green and Red, subtract blue.

RE-EDIT: Found the baldavenger post: https://www.magiclantern.fm/forum/index.php?topic=15801.100 Very good read.

RE RE Edit: The clipping is also heavily influenced by your color space e.g. Squeezing 14 bit linear in to a Rec709 color space will cause clipping and out of gamut solarization that has to be adjusted for. And the other way around setting Linear - to ACES color space which is a huge color space will cause pink highlights because the RGB Pixels are not clipping at the same level. Like majority of all sensors, the RGGB pattern will cause Pink highlights in the sun or Blacksun in BM's case(not sure about BRAW).

[ilia3101]

@a1ex Thanks for the samples, exactly the kind I'm looking for. Did your highlight patch get merged to UFRaw?

Possibly related: https://twitter.com/CT_Bergstrom/status/1333614982647353347

This one is amazing Must be related. He provides raw files yay.

@Walter Yes, I would like to see pictures from any camera brand if it has interesting artifacts.

@Skinny I had forgotten that the black sun effect was a thing. Not sure of the best way to fix it other than lowering white level. It might also become less noticable with some MLV App changes I plan to make

The best way to deal with these issues, in my opinion, would be with some kind of gamut compression - but you'd have to carefully choose a suitable color space for that. In particular, I can tell you for sure that CIELAB is not the right color space for this purpose (long answer in the links shared earlier).

This post by the Oklab guy shows how much CIELAB skews purple between blue and white, definitely an outdated colour space that shouldn't be used anymore.

I have been experimenting with highlight desaturation/roll-off recently in Jzazbz, Oklab and xyY, all work quite well, but xyY is a little bit worse for blue. I want this to replace MLV App's "tonemapping" curve stuff.

[Audionut]

The best way to deal with these issues, in my opinion, would be with some kind of gamut compression - but you'd have to carefully choose a suitable color space for that. In particular, I can tell you for sure that CIELAB is not the right color space for this purpose (long answer in the links shared earlier).

There must be some form of mapping, to map the out of gamut colors inside of the expected display gamut. But how do you map that?

Do you simply drag all of the out of gamut colors into the end color space!
Do you apply some sort of perceptual mapping, where you not only map the out of gamut colors to the end color space, but also apply some mapping to the rest of the color to retain some form of perceptual linearity!

What about correcting for brightness as the saturation changes.

https://www.sciencedirect.com/topics/engineering/perceptual-colour-space

Color science is way, way above this plebes head.

I only very rarely follow along with madVR development, but madshi has to tackle a similar problem for his HDR > SDR conversion.

2) Let's say you have a highly saturated red color which the HDR Blu-Ray has encoded with 4,000 Nits. And your display actually *can* do 4,000 Nits. No problems, right? Actually yes, BIG problem, because the display peak Nits capability is for white, not for red. So what should a tone mapping algorithm do now? Should it make the pixel white? It could achieve the wanted 4,000 Nits, but the pixel's color/saturation would be completely lost. Or should the tone mapping maintain the full saturation/color, and lose all the Nits it can't handle? Then a significant amout of highlight punch & detail would get lost. So what should we do? In madVR you can choose. See option "fix too bright & saturated pixels by".

madVR's tone mapping works like this: If you actually tell madVR the proper peak Nits value that you measure your display as, all the pixels in the lower Nits range (ideally from 0-100 Nits) are displayed absolutely perfectly, in the same way a true 10,000 Nits display would show them. Tone mapping only starts somewhere above this lower Nits range. However, we can't simply jump abruptly from 0 compression to strong compression, so the tone mapping curve needs to start smoothly, otherwise the image would get a somewhat unnatural clipped look.

[Skinny]

Thanks guys!

That's the "black sun" effect - see e.g.:
- pco.de/.../20140412_FOM_CameraTutorial_PCO_fin_pdf.pdf p.31
- imagesensors.org/.../7-02__geurts.pdf p.3
- https://www.physicsforums.com/threads/black-sun-effect-in-cmos-sensors.799334/

Wow, interesting links. I kind of knew about this phenomenon, but never thought it could happen so easily on a "pro" camera. And it isn't black on my clips..

Reducing the white level works, but it seems like I'm loosing some highlight details this way. The easiest way for me (for now) is to mask it away in post, since it is always surrounded by some blank area, masking is relatively easy.
I haven't tried Resolve yet, I'm using MLVApp -> Premiere, for now..

For some reason it is very easy to get this blown-out sun effect, this example was shot on relatively cloudy day, it was already evening and sunset, not a lot of light... In a day time, I get this artefact even not on the sun, but in the bright sky. Unfortunately I already deleted MLVs with bright sky example...

It would be nice if "highlight reconstruction" in MLV App could fix this, somehow.. but I think it's a little bit offtopic.


We have this new RGB LED spotlights in a park, I'm gonna take a walk and shoot some clips today, will try to catch this negative-blue effect.

[ilia3101]

We have this new RGB LED spotlights in a park, I'm gonna take a walk and shoot some clips today, will try to catch this negative-blue effect.

Sounds great!

[a1ex]

Did your highlight patch get merged to UFRaw?

No, it's still over here. But now I realize the approach was way too much of a hack, so... no wonder they weren't impressed

On the other hand, the soft-film curve works really well in my opinion (OK, I'm biased). Sample images:
https://www.magiclantern.fm/forum/index.php?topic=5197.msg91513#msg91513

This post by the Oklab guy shows how much CIELAB skews purple between blue and white, definitely an outdated colour space that shouldn't be used anymore.

Excellent! You must be a few light years ahead of me on this topic

Do you apply some sort of perceptual mapping, where you not only map the out of gamut colors to the end color space, but also apply some mapping to the rest of the color to retain some form of perceptual linearity!

Right - the user shouldn't notice any sharp transitions in the colors. You could keep the "normal" (not highly-saturated) colors unchanged, and compress only the extreme ones - both the ones near the gamut limits, and the ones outside the gamut. The soft-film curve could actually be useful for that - but I'd have to experiment.

Thanks guys!
Wow, interesting links. I kind of knew about this phenomenon, but never thought it could happen so easily on a "pro" camera. And it isn't black on my clips..

It doesn't have to go all the way to black - but once the analog side of the sensor is saturated, the reference dark value will continue to grow (it's slightly influenced by the incoming light, as the electronic shutter is not perfect). If I understand well, that's a side effect of CDS (a technique commonly used in image sensors).

This shouldn't happen with a mechanical shutter - but need to double-check.

The visible outcome is that, above a certain threshold (when saturation happens), the apparent response curve of the image sensor becomes non-monotonic (negative slope). In the sample response curve below, imagine the response goes downwards at some point after reaching saturation.



(image source)

In Canon cameras, there's also some harsh clipping caused by saturation of downstream electronics (ADC, amplifiers and whatever else Canon uses in their pipeline). Some theories are buried over here:
https://www.magiclantern.fm/forum/index.php?topic=10111.

[Skinny]

Sounds great!

I failed They have light-blue LEDs in rgb matrix, and they seems to look OK under any condition... I haven't discovered anything suspicious.

This shouldn't happen with a mechanical shutter - but need to double-check.

I'll check it next time, interesting though. I know how ADCs works in general (SAR ADCs and so on), and a lot about electronics... But photo and video world is kind of new to me. If only I had more time to study everything.. probably need another life

[Levas]

@Ilia3101

I've got many concert photos with blue/purple led light...Blue/Purple is the worst

Here's one example of Example :
https://drive.google.com/file/d/1vzKa3SS-ATFOof-zM1oqxwofxscwoIKR/view?usp=sharing

How it shows in google drive is not bad, but that's because I used a modified picture style preset on my camera with desaturation, apparently google drive reads these custom settings from the file and actually uses them, not bad.
If I open the file in Finder on MacOs, a standard picture style preset is used and you'll see your bad clipping.

I will look for more...

[Levas]

Found some more examples, placed them in this folder on google drive.
https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing

Most of the time you can fix those shots by adjusting white balance and tint, many times there is a sweet spot where the colors don't fight each other with hard clipping.
A little desaturation will help too.

Best example (image_3110.CR2) of how a straight out of camera jpg conversion would look:

[Levas]

Diving some more into this, the above example is actually how the CR2 file is displayed in preview in MacOs.
As you can see, weird highlight clipping, where the reds and blues become suddenly darker intowards the lights. (The negative luma describe by Ilia in the first post)
Now, the file looks normal on Google drive.
Here's a side by side, Google drive on the left, MacOs on the right.



Opening the file in most raw editors, it looks perfectly fine...
Opening it up in Lightroom (in this case LR 5.7), on the camera calibration tab -> with "adobe standard" profile, it looks not that good.
Changing the profile to the Canon profiles, for example, the profile "Camera Neutral" it looks much better.
Going back "adobe standard" and changing the processing version from 2012 to 2010...you start crying 

So most software get's it right, some don't, like Adobe and MacOs.

[Danne]

Testing your file in Mlv App:

Rec709 + use camera matrix + disable cyan highlight fix
use camera matrix + cyan highlight fix


Rec2020 + use camera matrix + cyan highlight fix


Don´t use camera matrix

[wib]

IMO "Rec709 + use camera matrix + disable cyan highlight fix" looks fine with vibrant colors, second one seems more natural.

[ilia3101]

@Levas thank you so much, thosee examples are great for testing highlight handling

Do you have any especially disastrous images ruined by blue/violet light?

[Levas]

Uploaded some more examples in the same folder:
https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing

About the negative luma and hard clipping, it also has to do with highlight reconstruction in software (reconstructions by using values from non clipped channels)
In my examples you'll find blown out highlights in the red/blue channel, while the green channel is almost black.
So...if software is gonna reconstruct highlight detail from the green channel(s), it has a real hard time doing that 

For highlight reconstruction, you should need some sort of minimum value in other channels to perform a nice detail reconstruction. If the value is below that threshold, it should not even try to reconstruct detail from other channels.

[Levas]

Found one photo that's completely done in blue/violet lighting.
It's in the same folder as the other images:
https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing

You can get some sort of natural colors when using crazy white balance values in your photo editor (in this case RawTherapee 5.8 ).
White balance = 60000
Tint = 10.000
Blue/red balance = 0.8


If you use standard white balance, as shot, it looks like this:

[ilia3101]

More great samples! thanks again @Levas

[ilia3101]

@Levas thanks again for the samples. They are extremely helpful for designing highlight reconstruction algorithms, and improving difficult colour reproduction.

Haven't had much time yet, but I finally got a new highlight reconstruction algorithm working... I would describe my highlight reconstruction algorithm as "ratio inpainting". Still glitchy and nowhere near finished though.

For highlight reconstruction, you should need some sort of minimum value in other channels to perform a nice detail reconstruction. If the value is below that threshold, it should not even try to reconstruct detail from other channels.

I thought about this, but it seems if one channel is bright enough to be clipped, the other two always have enough information, even with very saturated light, so no minimum is even needed, at least I haven't found an example.

Rawtherapee with "colour propagation" highlight reconstruction (left) vs my algorithm (right):


This image shows the worst artifacts of I've seen, very valuable.

Another image for comparison:



The red skews a little too yellow imo, and there's a weird cyan artifact, but an improvement over old mlv app

[elenhil]

Are more samples still wanted?

[ilia3101]

Haven't actively been working on this but I always appreciate more samples.

Edit:

Just ran the images from Levas through the latest version of my little program:



"camera matrices" and everything