Dual ISO: back into CR2

[Marsblessed]

Hi everyone!

If you ever been an owner of DxO Software, you would agree with me that there is no way of doing anything in the software with dual_iso-dng. That is a real problem especially when you do not want to have such a heavy peace of software as Adobe Photoshop is. Thus, there is only one option after making dual_iso shots: BACK into cr2!

Well, at first, it looks completely impossible - private Canon RAW file format and the result of the cr2hdr, which is not only in dng format, but also has 16 bits per sample instead of 14 bps. However, is it really unfeasible?

The good news: No, it is NOT unfeasible. I did it. There is no trouble in replacing the RAW data in a CR2-file. Not at all. It works fine in the DxO Optics Pro now!

The bad news: I need help. It turns out that the raw values in the dng-result are not that simple to convert them back into 14 bps. SHR by 2 creates something really dark with a strange tint of green although the bright areas look perfectly normal. It just means that I need a proper transformation from the cr2hdr resulting color space into Canon original color space. Therefore, I would appreciate it if someone could help me to write the proper transformation in C. The problem is the black level which does not correspond to the original one from cr2. I will have to dig into the cr2hdr sources.

[a1ex]

There's no change in color space; the two exposures are simply scaled to match in the original color space (whatever that may be, as long as it's close enough to RGB).

My best guess is that you have simply ran out of bits. There are situations where not even 16 bits are enough, and the 20-bit branch of cr2hdr includes an option to compress the highlights and allow more bits for shadows: http://www.magiclantern.fm/forum/index.php?topic=7139.msg105120#msg105120

How did you replace the data in the original CR2? did you use libcraw2 or you wrote your own?

[Marsblessed]

Thank you for your answer, A1ex, I was waiting for it first.

Concerning my results - you see, my transformation is simple - just shr by 2. I have made an assumption that if the original color space is linear than it is perfectly justified that in order to achieve 14 bps I should only make a simple linear transformation. Nonetheless, the result is unexpectedly green in the dark areas and the picture is much darker than the original (shadows transformed into deep shadows or even black), which proof one of the things: there is a transformation of original cr2-color space in cr2hdr and it is not linear OR the original Canon color space is itself not linear. The latter, however, should not be true if the cr2-values are raw levels - voltage, in fact (correct me if I am wrong with that assumption), which again bring us back to the idea that there is some color transformation in cr2hdr and it is not linear. It is difficult for me to check the code of cr2hdr that fast and that is why I started this topic.

you have simply ran out of bits.

Well, I would agree with you if it were not possible to have normal dynamic range using just 8 bps. The problem is deeper, and it does not matter whether it is 14 bps or as much as 20. If it were just shadows I would not say a word about it - I still have a lot to learn about appropriate ISO's in different scenes. This is the green tint which makes it all wrong. It means that the original and the resulting color spaces are not correspond to each other.

How did you replace the data in the original CR2? did you use libcraw2 or you wrote your own?

Well, that is quite a story and probably if I were acquainted with libcraw, I would not spend that much time. I did it using a result of comprehension of the dcraw.c, ITU-T.81, and ppmtoljpg example (which is based on the libjpeg). My tool is actually a mix of code excerpts from dcraw.c and ppmtoljpg.c and hard-coded values from 650d version of cr2-files (I have no idea whether the values are different in the other models cr2-files but I guess they could be). Actually, it was really a surprise for me that cr2-files (as well as dng) are not special files but just tiff-files with some private tags known only to Canon, and the raw data in such a file is just another lossless jpeg file inside tiff. Moreover, the ljpg-file is in the end of cr2-file and in order to replace the data you should only compress new data into ljpgf-file, replace the old one with it and modify the size of the "compressed data" in the "header" (tiff tag 279 of the corresponding ifd). The problem, however, is lossless jpeg compression, but I managed to find its free implementation, which suits good to the purpose.

[a1ex]

Again, with default option, cr2hdr outputs *linear* images in the *exact* *same* color space as Canon's. The only situation when the output is nonlinear is with the --soft-film option (this is why it requires you to specify the WB in advance).

Without sample images, it's hard to diagnose from tons of text. Another guess would be black level (which is not simply multiplied by 4 in cr2hdr).

[Marsblessed]

I have perfect samples of cr2-files, but I have no idea of how to add these files to the message. I would be grateful if someone could help me with that.

[engardeknave]

dropbox, wetransfer

[Audionut]

You can use any number of free file hosts.

Upload your sample to one of these, and copy the link.

I personally prefer Dropbox.

[Marsblessed]

Meanwhile, I have checked carefully my results and it turns out that it is the DxO which makes it all green by amplifying shadows. The picture itself (without DxO modification) is much much darker than I expected. Sure it should be done by someone else to check whether my dual_iso-shot were done "properly" in the first place. Will add the sample a little later.

[dmilligan]

The picture itself (without DxO modification) is much much darker than I expected.

That's because you are viewing linear data.

Again, with default option, cr2hdr outputs *linear* images in the *exact* *same* color space as Canon's.

[dubzeebass]

That's because you are viewing linear data.

This linear data thing... Is that why dual iso pictures are typically very dark until one increases the shadow brightness? What is the root cause of that?

[a1ex]

Yes. If you want to get normal brightness, use --soft-film to burn some (nonlinear) exposure compensation in the output DNG.

[Marsblessed]

Sample to analyze:
DUAL3174.CR2

that is the original cr2-file. After applying cr2hdr and recompress the result into a new cr2-file, the DxO shows it very green or too dark.

the result file:
DUAL3174.forged.cr2

==============

Here is another example which is not that bad with dual-ISO. The same green effect is clearly visible even on the pure cr2hdr-result! Take a look:

original CR2
hdr CR2

@a1ex
Could you please confirm that the data sample in the 16bps-DNG-file is two bytes long without any shift which means that every two bytes in the data represent single color value? Is the order of the colors the same as in CR2 = one line is rg rg rg rg ... and the next line is gb gb gb gb ...? My result looks as if I have changed the order of the colors in one line of every pair.

[a1ex]

I'm not sure where to look. The output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.

My DxO demo license expired a few months ago.

The byte order in the DNG is according to Adobe spec (but different from Canon's internal buffer described in raw.h), and the order of colors is the same. I force it to RGGB while processing, but I switch it back before saving the file.

Besides black level, another hypothesis could be the noise distribution in the optical black areas. After processing with cr2hdr, it's no longer Gaussian.

[Marsblessed]

The output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.

UFRaw is based on the dcraw.c which shows the same Filter pattern for the resulting DNG-file as it does for the original CR2-file. That is why I supposed that the data in the DNG-file is completely identical to CR2-file except bps and compression. May be this is the key to green color (which appears not only in deep shadows as it could be seen in DUAL3175.forged.cr2)

[IliasG]

Hi everyone!

If you ever been an owner of DxO Software, you would agree with me that there is no way of doing anything in the software with dual_iso-dng. That is a real problem especially when you do not want to have such a heavy peace of software as Adobe Photoshop is. Thus, there is only one option after making dual_iso shots: BACK into cr2!

I don't have DxO converter but
What does DxO displays when opening dualISO dngs ?.

UFRaw is based on the dcraw.c which shows the same Filter pattern for the resulting DNG-file as it does for the original CR2-file. That is why I supposed that the data in the DNG-file is completely identical to CR2-file except bps and compression. May be this is the key to green color (which appears not only in deep shadows as it could be seen in DUAL3175.forged.cr2)

As Alex wrote, the usual reason for dark greenish picture is wrong black Level.

For CR2s, Dcraw and derivatives calculate it from the side optically black area and the picture comes out correct (at least with RawTherapee that I tried). Looks like DxO does not use this method. It either has a hardcoded value as BlackLevel or reads it from exif  -Canon data where the info is invalid after cr2hdr conversions.

For DNGs, Dcraw and derivatives read the BL and WL values from exif (not exif.Canon ) ..and it is again displayed correctly.

To check if DxO uses the -Canon data for BL/WL try the corrected dual DNG and ..forged.cr2 . I copied the values from DNG's exif data to -Canon data with exiftool.
http://filebin.net/6lagewyuhl/DUAL3174-BLWLcorrected.DNG
http://filebin.net/6lagewyuhl/DUAL3174.forged-BLcorrected.cr2

[Marsblessed]

What does DxO displays when opening dualISO dngs ?

Well, I would not call the DxO software a converter. It is rather a simplified version of Photoshop, which can do some post. The problem is it can not open DNG-files at all! If it were not the case I would not need to do anything with the CR2-files.

To check if DxO uses the -Canon data for BL/WL try the corrected dual DNG and ..forged.cr2 . I copied the values from DNG's exif data to -Canon data with exiftool.
http://filebin.net/6lagewyuhl/DUAL3174-BLWLcorrected.DNG
http://filebin.net/6lagewyuhl/DUAL3174.forged-BLcorrected.cr2

Tried this file - it looks even darker now but it looks absolutly identical with the same green tint everywhere. Which means no, the black level correction does nothing. Again, the problem is not only in the dark areas. Every point in the compressed result of cr2hdr has more green than it should be. As A1ex stated, it is even seen in the UFRaw (which has nothing in common with DxO).

[Marsblessed]

I'm not sure where to look. The output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.
...
The byte order in the DNG is according to Adobe spec (but different from Canon's internal buffer described in raw.h)

I have an idea and I will check it later. The dcraw.c use the byte order that is specified in the first two bytes of any tiff. Therefore, if Adobe has a special byte order for its data in dng-file (which is tiff too) and it is differ from the one that is specified in the tiff header than that could be the reason of green tint in UFRaw and after my conversion which is also based on dcraw.

[a1ex]

Byte ordering mismatch will result in complete gibberish (not the case here).

Can you post some JPEGs to see what exactly you are talking about?

If DxO expects a black level of 2048, simply add a constant offset to your raw values to force your black level to that value. Usually, the black level from cr2hdr ends up a little lower than 2048*4.

[IliasG]

Well, I would not call the DxO software a converter. It is rather a simplified version of Photoshop, which can do some post. The problem is it can not open DNG-files at all! If it were not the case I would not need to do anything with a CR2-files.

We are researching the raw conversion part here only, I think ..

Tried this file - it looks even darker now but with the same green tint everywhere. Which means no, the black level correction does nothing. Again, the problem is not only in the dark areas. Every point in the compressed result of cr2hdr has more green than it should be. As A1lex stated, it is even seen in the UFRaw (which has nothing in common with DxO).

So if it's darker this means that DxO uses these tags that I changed ..

Look what exactly Alex wrote below .. the DNG is greener than your .forged.cr2, same as with Rawtherapee. And the reason is that the Black level tag in the DNG is higher that the calculated black level (7964/4 = 1991 vs 1988 calculated) ..

... The output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.

Please upload some converted samples ..

[a1ex]

Forged vs DNG:

Code: [Select]

ufraw-batch --exposure=5 --out-type=jpg --shrink=2 --temperature=3500 --green=0.8 DUAL3174.forged.cr2
ufraw-batch --exposure=5.8 --out-type=jpg --shrink=2 --temperature=3500 --green=0.8 DUAL3174.DNG


[Marsblessed]

Tried this file - it looks even darker now

well, again - thorough checking shows that my first conclusion was incorrect - there are no differences between DUAL3174.forged-BLcorrected.cr2 and DUAL3174.forged.cr2 in DxO

another couple of samples, produced by DxO with every feature switched off:
DUAL3185_DxO.jpg
DUAL3185.forged_DxO.jpg

[a1ex]

Black level issue.

[Marsblessed]

well, but notice that the sky is darker too despite the fact that it is done by interpolating between blue and white!

Besides, I have tried to add 2048 to every raw value of the picture - the result is corrupted data (can not be opend in the DxO)

At the same time, it means that I have to apply some more sophisticated logic to 16 bps raw values instead of just SHR by 2. (Again, that means that the color spaces are not identical)

[Marsblessed]

I think I understand now. I have to compress 16 bps raw values back to the range between 2048*4 and 2^16-1 and only after that I should apply SHR by 2. A1ex, please, correct me if I am wrong.

[Shizuka]

Please don't actually do a SHR 2 because it introduces a small bias in values.

In the case of [aaaaaaaaaaaaaabb], you are forcing bb to be always 00 when it was 01, 10, or 11 before the shift.
These might not matter much when you're much higher than the black level, but rounding errors can throw out information for the darkest stop of data.

Two other ways to do it are:

Flat rounding - no bias nearest neighbor dithering
if bit15 is 1, add 1 to bit14

Random rounding - random dither
01 - 25% chance of adding 1 to bit14
10 - 50%
11 - 75%