Magic Lantern Forum

Using Magic Lantern => Raw Video => Raw Video Postprocessing => Topic started by: ilia3101 on February 01, 2021, 06:35:29 PM

Title: UGLY clipping samples
Post by: ilia3101 on February 01, 2021, 06:35:29 PM
Does anyone here ever have raw images with ugly clipping? Be that simple highlight clipping, blue light clipping in weird ways, or red lights going yellow, anything...

I'm looking for examples of these kind of images, they can be from any camera (from any brand) and in any raw format. I want them all. Let's make a collection of raw files with difficult colours. It would be really helpful for developing and testing raw conversion software (MLV App in my case)

Here's a particularly extreme example of what I want:

(https://i.ibb.co/2tPdtJP/5D3-9975.jpg)

Would love more like this!


Fun fact: Using 3x3 matrices for raw conversion (which almost all software currently does) produces negative Luminance (Y) values for pure blue light. So why doesn't blue show up as black? ... per-channel clipping! (most often in ProPhoto RGB)




This sums up what I am looking for very well:

Quote from: a1ex on February 02, 2021, 08:53:12 AM
In this thread, we are looking specifically for sample images with extreme out-of-gamut clipping issues (extremely saturated colors that are rendered obviously wrong). Such issues can be found when using highly saturated light sources such as colored LEDs, or with concert photos/videos (with unusual lighting), or highly saturated scenes/subjects, such as flowers.
Title: Re: UGLY clipping samples
Post by: a1ex on February 01, 2021, 09:57:02 PM
Possibly related: https://twitter.com/CT_Bergstrom/status/1333614982647353347

The topic is of interest to me as well, btw - just not actively researching it right now.

My earlier hackish attempt - back then, I didn't know what I was doing:
https://www.magiclantern.fm/forum/index.php?topic=5197.0

Found two sample images (I can dig for more, if needed):

IMG_0307.CR2 (http://a1ex.magiclantern.fm/hdr/ufraw%20bug/IMG_0307.CR2)
IMG_5207.CR2 (http://a1ex.magiclantern.fm/hdr/ufraw%20bug/IMG_5207.CR2)

Some stuff I came across, but didn't try yet:
https://bottosson.github.io/posts/gamutclipping/
http://www.brucelindbloom.com/index.html?UPLab.html

Maybe it helps.
Title: Re: UGLY clipping samples
Post by: 2blackbar on February 02, 2021, 12:19:02 AM
Yes i had that especially at night when filming traffic lights or cop lights, i dont have dngs now tho.
Title: Re: UGLY clipping samples
Post by: Skinny on February 02, 2021, 08:00:39 AM
Sometimes I get this ugly pink area if the highlights clipped really hard (like on the sun, for example) even when "highlight reconstruction" is on..

This is how it looks (click for full size):
(https://i.ibb.co/b6MG0rZ/M26-1635-frame-55.png) (https://ibb.co/b6MG0rZ)

One-frame mlv and the preset I used:
https://gofile.io/d/q6sagD

It is clearly visible even with the default MLV App settings, and if you bring down the highlights and turn on reconstruction it becomes really obvious.
I used 5D2 and 12 bits.
Title: Re: UGLY clipping samples
Post by: a1ex on February 02, 2021, 08:53:12 AM
That's the "black sun" effect - see e.g.:
- pco.de/.../20140412_FOM_CameraTutorial_PCO_fin_pdf.pdf (https://www.pco.de/fileadmin/user_upload/pco-knowledge_base/20140412_FOM_CameraTutorial_PCO_fin_pdf.pdf) p.31
- imagesensors.org/.../7-02__geurts.pdf (https://www.imagesensors.org/Past%20Workshops/2015%20Workshop/2015%20Papers/Sessions/Session_7-Posters2/7-02__geurts.pdf) p.3
- https://www.physicsforums.com/threads/black-sun-effect-in-cmos-sensors.799334/

For this particular image, the easiest way to render it would be to reduce the white level. If your raw processing software has a slider for that, use it. In mlv_dump, the relevant option is --white-fix. For this particular file, 3300 looks about right to me (from the default of 4050).




In this thread, we are looking specifically for sample images with extreme out-of-gamut clipping issues (extremely saturated colors that are rendered obviously wrong). Such issues can be found when using highly saturated light sources such as colored LEDs, or with concert photos/videos (with unusual lighting), or highly saturated scenes/subjects, such as flowers.

Quote from: https://bottosson.github.io/posts/gamutclipping/These colors are more colorful than the color space is able to encode.

Here's an example of an image that's clearly not clipped in RAW, but rendered incorrectly (clipped) in sRGB. I couldn't find the original raw file, but didn't look too hard.

https://www.dpreview.com/forums/post/62681348

The best way to deal with these issues, in my opinion, would be with some kind of gamut compression - but you'd have to carefully choose a suitable color space for that. In particular, I can tell you for sure that CIELAB is not the right color space for this purpose (long answer in the links shared earlier).
Title: Re: UGLY clipping samples
Post by: Walter Schulz on February 02, 2021, 08:58:31 AM
Quote from: ilia3101 on February 01, 2021, 06:35:29 PM
I'm looking for examples of these kind of images, they can be from any camera and in any raw format.

Any camera = any brand?
Title: Re: UGLY clipping samples
Post by: Kharak on February 02, 2021, 09:13:30 AM
Try the same clips in Resolve, yes they will have pinkish highlights, but not as saturated/"artifacty" like your example.

I usually mitigate it by turning luma mix off and pushing green in to the highlights, easy fix. All in a RCM workflow ofcourse.

I cant find the thread by baldavenger, but he made a post about altering the dng's in order to get perfect white and a full signal straight in to Resolve. In his workflow highlight reconstruction wont work anymore, so pulling highlight slider wont have any effect. Maybe it was a combination of one of his DCTL's and dng manipulation. I can't recall and I never tried it.

EDIT: How do your clipped blues look on the waveform? I assume clipped, try adding Green and Red, subtract blue.

RE-EDIT: Found the baldavenger post: https://www.magiclantern.fm/forum/index.php?topic=15801.100 Very good read.

RE RE Edit: The clipping is also heavily influenced by your color space e.g. Squeezing 14 bit linear in to a Rec709 color space will cause clipping and out of gamut solarization that has to be adjusted for. And the other way around setting Linear - to ACES color space which is a huge color space will cause pink highlights because the RGB Pixels are not clipping at the same level. Like majority of all sensors, the RGGB pattern will cause Pink highlights in the sun or Blacksun in BM's case(not sure about BRAW).
Title: Re: UGLY clipping samples
Post by: ilia3101 on February 02, 2021, 12:19:41 PM
@a1ex Thanks for the samples, exactly the kind I'm looking for. Did your highlight patch get merged to UFRaw?

Quote from: a1ex on February 01, 2021, 09:57:02 PM
Possibly related: https://twitter.com/CT_Bergstrom/status/1333614982647353347

This one is amazing :D Must be related. He provides raw files yay.

@Walter Yes, I would like to see pictures from any camera brand if it has interesting artifacts.

@Skinny I had forgotten that the black sun effect was a thing. Not sure of the best way to fix it other than lowering white level. It might also become less noticable with some MLV App changes I plan to make


Quote from: a1ex on February 02, 2021, 08:53:12 AM
The best way to deal with these issues, in my opinion, would be with some kind of gamut compression - but you'd have to carefully choose a suitable color space for that. In particular, I can tell you for sure that CIELAB is not the right color space for this purpose (long answer in the links shared earlier).

This post (https://bottosson.github.io/posts/oklab/#blending-colors) by the Oklab guy shows how much CIELAB skews purple between blue and white, definitely an outdated colour space that shouldn't be used anymore.

I have been experimenting with highlight desaturation/roll-off recently in Jzazbz, Oklab and xyY, all work quite well, but xyY is a little bit worse for blue. I want this to replace MLV App's "tonemapping" curve stuff.
Title: Re: UGLY clipping samples
Post by: Audionut on February 02, 2021, 01:08:13 PM
Quote from: a1ex on February 02, 2021, 08:53:12 AM
The best way to deal with these issues, in my opinion, would be with some kind of gamut compression - but you'd have to carefully choose a suitable color space for that. In particular, I can tell you for sure that CIELAB is not the right color space for this purpose (long answer in the links shared earlier).

There must be some form of mapping, to map the out of gamut colors inside of the expected display gamut. But how do you map that?

Do you simply drag all of the out of gamut colors into the end color space!
Do you apply some sort of perceptual mapping, where you not only map the out of gamut colors to the end color space, but also apply some mapping to the rest of the color to retain some form of perceptual linearity!

What about correcting for brightness as the saturation changes.

https://www.sciencedirect.com/topics/engineering/perceptual-colour-space

Color science is way, way above this plebes head.

I only very rarely follow along with madVR development, but madshi has to tackle a similar problem for his HDR > SDR conversion (https://www.avsforum.com/threads/improving-madvr-hdr-to-sdr-mapping-for-projector.2954506/).
Quote2) Let's say you have a highly saturated red color which the HDR Blu-Ray has encoded with 4,000 Nits. And your display actually *can* do 4,000 Nits. No problems, right? Actually yes, BIG problem, because the display peak Nits capability is for white, not for red. So what should a tone mapping algorithm do now? Should it make the pixel white? It could achieve the wanted 4,000 Nits, but the pixel's color/saturation would be completely lost. Or should the tone mapping maintain the full saturation/color, and lose all the Nits it can't handle? Then a significant amout of highlight punch & detail would get lost. So what should we do? In madVR you can choose. See option "fix too bright & saturated pixels by".

QuotemadVR's tone mapping works like this: If you actually tell madVR the proper peak Nits value that you measure your display as, all the pixels in the lower Nits range (ideally from 0-100 Nits) are displayed absolutely perfectly, in the same way a true 10,000 Nits display would show them. Tone mapping only starts somewhere above this lower Nits range. However, we can't simply jump abruptly from 0 compression to strong compression, so the tone mapping curve needs to start smoothly, otherwise the image would get a somewhat unnatural clipped look.
Title: Re: UGLY clipping samples
Post by: Skinny on February 02, 2021, 01:17:44 PM
Thanks guys!

Quote from: a1ex on February 02, 2021, 08:53:12 AM
That's the "black sun" effect - see e.g.:
- pco.de/.../20140412_FOM_CameraTutorial_PCO_fin_pdf.pdf (https://www.pco.de/fileadmin/user_upload/pco-knowledge_base/20140412_FOM_CameraTutorial_PCO_fin_pdf.pdf) p.31
- imagesensors.org/.../7-02__geurts.pdf (https://www.imagesensors.org/Past%20Workshops/2015%20Workshop/2015%20Papers/Sessions/Session_7-Posters2/7-02__geurts.pdf) p.3
- https://www.physicsforums.com/threads/black-sun-effect-in-cmos-sensors.799334/
Wow, interesting links. I kind of knew about this phenomenon, but never thought it could happen so easily on a "pro" camera. And it isn't black on my clips..

Reducing the white level works, but it seems like I'm loosing some highlight details this way. The easiest way for me (for now) is to mask it away in post, since it is always surrounded by some blank area, masking is relatively easy.
I haven't tried Resolve yet, I'm using MLVApp -> Premiere, for now..

For some reason it is very easy to get this blown-out sun effect, this example was shot on relatively cloudy day, it was already evening and sunset, not a lot of light... In a day time, I get this artefact even not on the sun, but in the bright sky. Unfortunately I already deleted MLVs with bright sky example...

It would be nice if "highlight reconstruction" in MLV App could fix this, somehow.. but I think it's a little bit offtopic.


We have this new RGB LED spotlights in a park, I'm gonna take a walk and shoot some clips today, will try to catch this negative-blue effect. :)
Title: Re: UGLY clipping samples
Post by: ilia3101 on February 02, 2021, 03:11:00 PM
Quote from: Skinny on February 02, 2021, 01:17:44 PM
We have this new RGB LED spotlights in a park, I'm gonna take a walk and shoot some clips today, will try to catch this negative-blue effect. :)

Sounds great!
Title: Re: UGLY clipping samples
Post by: a1ex on February 02, 2021, 03:59:54 PM
Quote from: ilia3101 on February 02, 2021, 12:19:41 PM
Did your highlight patch get merged to UFRaw?

No, it's still over here (https://sourceforge.net/p/ufraw/bugs/362/). But now I realize the approach was way too much of a hack, so... no wonder they weren't impressed :D

On the other hand, the soft-film curve works really well in my opinion (OK, I'm biased). Sample images:
https://www.magiclantern.fm/forum/index.php?topic=5197.msg91513#msg91513

Quote
This post (https://bottosson.github.io/posts/oklab/#blending-colors) by the Oklab guy shows how much CIELAB skews purple between blue and white, definitely an outdated colour space that shouldn't be used anymore.

Excellent! You must be a few light years ahead of me on this topic :D

Quote from: Audionut on February 02, 2021, 01:08:13 PM
Do you apply some sort of perceptual mapping, where you not only map the out of gamut colors to the end color space, but also apply some mapping to the rest of the color to retain some form of perceptual linearity!

Right - the user shouldn't notice any sharp transitions in the colors. You could keep the "normal" (not highly-saturated) colors unchanged, and compress only the extreme ones - both the ones near the gamut limits, and the ones outside the gamut. The soft-film curve could actually be useful for that - but I'd have to experiment.

Quote from: Skinny on February 02, 2021, 01:17:44 PM
Thanks guys!
Wow, interesting links. I kind of knew about this phenomenon, but never thought it could happen so easily on a "pro" camera. And it isn't black on my clips..

It doesn't have to go all the way to black - but once the analog side of the sensor is saturated, the reference dark value will continue to grow (it's slightly influenced by the incoming light, as the electronic shutter is not perfect). If I understand well, that's a side effect of CDS (https://en.wikipedia.org/wiki/Correlated_double_sampling) (a technique commonly used in image sensors).

This shouldn't happen with a mechanical shutter - but need to double-check.

The visible outcome is that, above a certain threshold (when saturation happens), the apparent response curve of the image sensor becomes non-monotonic (negative slope). In the sample response curve below, imagine the response goes downwards at some point after reaching saturation.

(https://www.baslerweb.com/fp-1485687434/media/editorial/content_images/faqs/faq_sensitivity_1_397x269.gif)

(image source) (https://www.baslerweb.com/en/sales-support/knowledge-base/frequently-asked-questions/what-is-sensitivity-and-why-are-sensitivity-statements-often-misleading/14987/)

In Canon cameras, there's also some harsh clipping caused by saturation of downstream electronics (ADC, amplifiers and whatever else Canon uses in their pipeline). Some theories are buried over here:
https://www.magiclantern.fm/forum/index.php?topic=10111.
Title: Re: UGLY clipping samples
Post by: Skinny on February 02, 2021, 05:26:10 PM
Quote from: ilia3101 on February 02, 2021, 03:11:00 PM
Sounds great!
I failed :) They have light-blue LEDs in rgb matrix, and they seems to look OK under any condition... I haven't discovered anything suspicious.

Quote from: a1ex on February 02, 2021, 03:59:54 PM
This shouldn't happen with a mechanical shutter - but need to double-check.
I'll check it next time, interesting though. I know how ADCs works in general (SAR ADCs and so on), and a lot about electronics... But photo and video world is kind of new to me. If only I had more time to study everything.. probably need another life :D
Title: Re: UGLY clipping samples
Post by: Levas on February 03, 2021, 10:39:33 AM
@Ilia3101

I've got many concert photos with blue/purple led light...Blue/Purple is the worst :P

Here's one example of Example ;) :
https://drive.google.com/file/d/1vzKa3SS-ATFOof-zM1oqxwofxscwoIKR/view?usp=sharing  (https://drive.google.com/file/d/1vzKa3SS-ATFOof-zM1oqxwofxscwoIKR/view?usp=sharing)

How it shows in google drive is not bad, but that's because I used a modified picture style preset on my camera with desaturation, apparently google drive reads these custom settings from the file and actually uses them, not bad.
If I open the file in Finder on MacOs, a standard picture style preset is used and you'll see your bad clipping.

I will look for more...



Title: Re: UGLY clipping samples
Post by: Levas on February 03, 2021, 12:12:03 PM
Found some more examples, placed them in this folder on google drive.
https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing (https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing)

Most of the time you can fix those shots by adjusting white balance and tint, many times there is a sweet spot where the colors don't fight each other with hard clipping.
A little desaturation will help too.

Best example (image_3110.CR2) of how a straight out of camera jpg conversion would look:
(https://live.staticflickr.com/65535/50903847078_59fe484266_o.png)
Title: Re: UGLY clipping samples
Post by: Levas on February 03, 2021, 03:07:01 PM
Diving some more into this, the above example is actually how the CR2 file is displayed in preview in MacOs.
As you can see, weird highlight clipping, where the reds and blues become suddenly darker intowards the lights. (The negative luma describe by Ilia in the first post)
Now, the file looks normal on Google drive.
Here's a side by side, Google drive on the left, MacOs on the right.

(https://live.staticflickr.com/65535/50904203188_834ce0b62b_o.png)

Opening the file in most raw editors, it looks perfectly fine...
Opening it up in Lightroom (in this case LR 5.7), on the camera calibration tab -> with "adobe standard" profile, it looks not that good.
Changing the profile to the Canon profiles, for example, the profile "Camera Neutral" it looks much better.
Going back "adobe standard" and changing the processing version from 2012 to 2010...you start crying  :P

So most software get's it right, some don't, like Adobe and MacOs.
Title: Re: UGLY clipping samples
Post by: Danne on February 03, 2021, 04:02:04 PM
Testing your file in Mlv App:

Rec709 + use camera matrix + disable cyan highlight fix
use camera matrix + cyan highlight fix
(https://i.postimg.cc/CKc3FMWh/Ska-rmavbild-2021-02-03-kl-15-55-28-png-scaled.png)

Rec2020 + use camera matrix + cyan highlight fix
(https://i.postimg.cc/25jpRL99/Ska-rmavbild-2021-02-03-kl-15-55-58-png-scaled.png)

DonĀ“t use camera matrix
(https://i.postimg.cc/K8DXHDfg/Ska-rmavbild-2021-02-03-kl-15-55-15-png-scaled.png)
Title: Re: UGLY clipping samples
Post by: wib on February 04, 2021, 01:01:44 PM
IMO "Rec709 + use camera matrix + disable cyan highlight fix" looks fine with vibrant colors, second one seems more natural.
Title: Re: UGLY clipping samples
Post by: ilia3101 on February 05, 2021, 04:03:54 PM
@Levas thank you so much, thosee examples are great for testing highlight handling

Do you have any especially disastrous images ruined by blue/violet light?
Title: Re: UGLY clipping samples
Post by: Levas on February 05, 2021, 11:02:59 PM
Uploaded some more examples in the same folder:
https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing (https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing)

About the negative luma and hard clipping, it also has to do with highlight reconstruction in software (reconstructions by using values from non clipped channels)
In my examples you'll find blown out highlights in the red/blue channel, while the green channel is almost black.
So...if software is gonna reconstruct highlight detail from the green channel(s), it has a real hard time doing that  :P

For highlight reconstruction, you should need some sort of minimum value in other channels to perform a nice detail reconstruction. If the value is below that threshold, it should not even try to reconstruct detail from other channels.
Title: Re: UGLY clipping samples
Post by: Levas on February 06, 2021, 01:39:25 PM
Found one photo that's completely done in blue/violet lighting.
It's in the same folder as the other images:
https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing (https://drive.google.com/drive/folders/1XMrMDNt_-dy4K6cXUHxhU3w4MPedG8dL?usp=sharing)

You can get some sort of natural colors when using crazy white balance values in your photo editor (in this case RawTherapee 5.8 ).
White balance = 60000
Tint = 10.000
Blue/red balance = 0.8
(https://live.staticflickr.com/65535/50914560841_71575a076b_o.png)

If you use standard white balance, as shot, it looks like this:
(https://live.staticflickr.com/65535/50914689322_ccdefef401_o.png)
Title: Re: UGLY clipping samples
Post by: ilia3101 on February 09, 2021, 03:50:51 PM
More great samples! thanks again @Levas
Title: Re: UGLY clipping samples
Post by: ilia3101 on March 17, 2021, 04:39:42 PM
@Levas thanks again for the samples. They are extremely helpful for designing highlight reconstruction algorithms, and improving difficult colour reproduction.

Haven't had much time yet, but I finally got a new highlight reconstruction algorithm working... I would describe my highlight reconstruction algorithm as "ratio inpainting". Still glitchy and nowhere near finished though.

QuoteFor highlight reconstruction, you should need some sort of minimum value in other channels to perform a nice detail reconstruction. If the value is below that threshold, it should not even try to reconstruct detail from other channels.

I thought about this, but it seems if one channel is bright enough to be clipped, the other two always have enough information, even with very saturated light, so no minimum is even needed, at least I haven't found an example.

Rawtherapee with "colour propagation" highlight reconstruction (left) vs my algorithm (right):
(https://i.ibb.co/JHhYx0r/rawtherapee.jpg) (https://i.ibb.co/DD6s53g/new.png)

This image shows the worst artifacts of I've seen, very valuable.

Another image for comparison:

(https://i.ibb.co/CBY8f0n/90s.png)

The red skews a little too yellow imo, and there's a weird cyan artifact, but an improvement over old mlv app ::)
Title: Re: UGLY clipping samples
Post by: elenhil on August 28, 2021, 04:47:02 PM
Are more samples still wanted?
Title: Re: UGLY clipping samples
Post by: ilia3101 on August 30, 2021, 08:27:03 PM
Haven't actively been working on this but I always appreciate more samples.

Edit:

Just ran the images from Levas through the latest version of my little program:

(https://i.ibb.co/1XFhgVM/4196.jpg) (https://i.ibb.co/gtSBqw3/3110.jpg)

"camera matrices" and everything
Title: Re: UGLY clipping samples
Post by: theBilalFakhouri on August 30, 2021, 09:39:35 PM
Really nice results, amazing!
Title: Re: UGLY clipping samples
Post by: masc on August 31, 2021, 04:54:57 PM
Quote from: ilia3101 on August 30, 2021, 08:27:03 PM
Just ran the images from Levas through the latest version of my little program:
...
"camera matrices" and everything
Nice! Do you still use the "NewProcessing" branch on github for that, or did you start something completely new? In the github issue I asked some questions on how to integrate into MLVApp...
Title: Re: UGLY clipping samples
Post by: ilia3101 on September 01, 2021, 04:18:19 PM
Yea I've written a separate program(i have to recompile to change any parameter)

It takes 1 minute to process a 20mp photo, and still has some issues with colour