UFRAW bug fix for overexposed channels

[a1ex]

Just fixed a bug in UFRAW in highlight handling (seems to be some sort of overflow and it's present in 0.18 and current CVS). Before reporting it to the UFRAW bug tracker, I'd like you to try it and help me find a test case where the bug is really obvious.

matrix.patch (updated)

Here's my first try:

bug (posterize effects on hands; blown-out magenta):


fixed:


bug (magenta halo; harsh transition from overexposed highlight to background):


fixed:


bug (magenta cast; the LED was blue):


fixed:


If you have some really blown out RAW shot that you'd like to try recovering, let me know.

[Audionut]

Not sure how related this is to the color artifacts your showing above.  But based on the example you posted in the RAW histogram thread with the blown out white bottom left that was not recovered.

https://dl.dropboxusercontent.com/u/34113196/Camera%20stuff/_46A4448.CR2


Rawdigger is showing a decent amount of blow out in the green channel, and just a touch in the blue (the extra saturated area top middle).  Easily recovered in ACR 2012.



UFRaw is struggling to recover the details imo.



Adobe Camera Raw 2012 does a much better job.



UFRaw is also generating artifacts.


Not seen in ACR.  You can also really see how ACR has recovered much more highlight detail.  There is blue sky in that white mess


I'm running short on HDD space and tend to delete test shots like these (so don't have any other ones).  But in my experience with white detail, as long as you have 1 channel with no blown areas, ACR can recover enough detail.  I can post a more extreme example later if your interested.

[a1ex]

Yes, the bugfix applies here too. Still not as good as ACR.

[Audionut]

That's a vast improvement a1ex.

[a1ex]

Also, the sensor saturation level for 5D Mark III seems to be 15200 (not 15488). This probably applies to dcraw too. I didn't notice any improvement by adjusting that, other than moving the histogram peak to the far right, where it should be.

[Audionut]

Hmm, RawDigger seems to vary a little.  Always around 13300-14300 though. 

As far as I know, it's detecting the maximum luminance values of the pixels, rather then importing with a specific fixed value.



I don't have any other way to inspect the RAW files.

[a1ex]

Some more tweaks:

1) Enhancement: softer "soft film" highlight handling
Before/after (--clip=film):


2) Small highlight fix for negative EV with restore==clip
Before/after (--clip=digital --restore=clip):


Before/after with both patches applied (--clip=digital --restore=clip):








Raw files:
ettr.dng, no blown channels, but tight ETTR.
_46A4448.CR2 from Audionut, same link as above, some channels overexposed. Convert to Adobe DNG before loading into ufraw (wrong white level in the CR2).

These 2 patches do not replace the first one (they just apply in different cases). You can apply all 3 of them if you like.

Need to test them on more than 2 images, but if there are no surprises, I think they are ready to submit to ufraw tracker.

[Audionut]

I'll find a bunch more samples and upload them after sleep.

[a1ex]

Another test case for the patch from the first post.



1) without the patch, color profile disabled (highlights ok, colors a little faded)
2) without the patch, color profile enabled (pink highlights)
3) with the patch, color profile enabled (colors ok, highlights ok)

Results are the same with LCMS 1.x and 2.x (not sure if it matters).

[Audionut]

Here is a bunch of overexposed shots, including of a colorchecker passport from slightly overexposed, through to OMG I forgot I was in manual mode, in 1/3rd increments.

xyY, Lab and RGB values in various color spaces of the color checker can be found in this PDF (page 5 ColorChecker 2005).  I've done my best to limit the UV exposure to the color checker, but it is around 2 years old now, so the values may not be 100% accurate.  You can WB from the neutral 5 patch (top row 3rd from left).

I've also included a LR template* I created based on the colorchecker.  It provides HSL adjustments only, WB/Luminance adjustments are left for each exposure.  It was created some time ago and won't be 100% accurate for the shots I've uploaded.  Note also that LR only provides RGB in percentage values, ACR is better for accurately checking level values.

To get an idea of the changes it makes.



I've got a Canon picture style floating around here somewhere based on the colorchecker also if you're interested.

*In development pane, in preset section on left, right click, import.

[glubber]

1) without the patch, color profile disabled (highlights ok, colors a little faded)

While trying to fix the pink highlights (esp. with Sunset scenes like Audionuts D46A9441.CR2) in UFRAW i came across the solution to disable color profile by T&E.
But if i look real close i still can see some artifacts or banding. Along with causing faded colors it's no proper solution to the problem.

As compiling ufraw for Windows is way beyond my skills, i couldn't test your patches.
So I would really appreciate if you submitted your patches to the tracker. 

[a1ex]

Yes, I've already submitted them: http://sourceforge.net/p/ufraw/bugs/362/

update: I've been able to compile UFRaw under Windows: http://www.magiclantern.fm/forum/index.php?topic=9597

[glubber]

Thank you a1ex for compiling your patches for Windows!

Convert to Adobe DNG before loading into ufraw (wrong white level in the CR2).

Wow.... this solved my pink highlight issue!! 

Is it UFRAW messing up with White levels or is that a Canon bug?

I read about UFRAW/ dcraw having troubles with pink highlights in Canon-raws.
But I couldn't imagine ADOBE fixing that problem. LOL

[a1ex]

Yes, it's ufraw using a hardcoded white level (taken fron dcraw), which is not always correct. I might port the autodetection routine from cr2hdr, since I couldn't find a way to figure it out from EXIF settings (it's influenced by both digital ISO and exposure time, and each camera has its own levels).

[IliasG]

Yes, it's ufraw using a hardcoded white level (taken fron dcraw), which is not always correct. I might port the autodetection routine from cr2hdr, since I couldn't find a way to figure it out from EXIF settings (it's influenced by both digital ISO and exposure time, and each camera has its own levels).

* In the raw histograms one can see that WL is influenced from wide apertures also, there is digital scaling which starts as negligible (1/200) at about f/3.2 and gradually increases as we go wider (1/10 at f/1.4) for FF models and more for APS-C.

Alex, with latest exitool you can find in -Canon tags
0x01fc NormalWhiteLevel
0x01fd SpecularWhiteLevel
0x01fe LinearityUpperMargin

I think "SpecularWhiteLevel" is what dcraw and derivatives need (although it's sometimes a bit conservative related to the raw histogram)

I do not know more than what Iliah Borg wrote at RT issues discussions http://code.google.com/p/rawtherapee/issues/detail?id=2125#c10
For more info you could contact with libraw or rawdigger developers as they are who decoded those tags.

Hopefully LinearityUpperMargin will be useful to you for DualISO etc.