Dual ISO: back into CR2

[a1ex]

That's a nice trick, I should actually use it too in cr2hdr.

[IliasG]

I think I understand now. I have to compress 16 bps raw values back to the range between 2048*4 and 2^16-1 and only after that I should apply SHR by 2. A1ex, please, correct me if I am wrong.

As I understand it, you have to read from DNG exif the black level value (it was 7964 in dual3174.dng and become 7964/4 = 1991 in forged.cr2) then add  2048-1911=139 to all the values of forged.cr2 to normalize for the suspected DxO's default BL for 650D .

[Marsblessed]

It is even more complicated. I can not just add something - the result is always corrupted data. As far as I see it now, I have to extract two ranges (black and white levels) from dng and cr2 and then compress or expand the dng-range into (multiplied by 4) cr2-range. After that I have to "SHR by 2" (using some rounding mechanism) and add cr2-black_level to the result before put it into cr2-file. I will test it later.

[Marsblessed]

One question though:
which white level should I use from CR2 - SpecularWhiteLevel or NormalWhiteLevel (according to this description)?

[IliasG]

I would bet on the specular .. this is closer to the peak white level one can see in the raw histograms. But one has to test first. Canon's DPP use all of them ..

Have you checked if DxO uses these tags at all .. by making extreme changes there an see the effect (if any) on DxO's conversion ?.

Related to the offset on values, now that you checked that by offsetting the values by the difference 2048-dng_Black_Level is a bad idea, you could try scaling by 2048/dng_Black_Level .. sounds also logical .

The suspicion that DxO uses a default value (i.e. 2048) as "black level" was purely a guess based on that you wrote that nothing happens by changing the -Canon tags ..

If you can work with Rawtherapee you can change the black level used by RT in tab raw - black levels to see  when RT's conversion matches DxO's. There i put +139.0 and took a dark greenish result like you described it ..     

[Marsblessed]

I have finally managed to show a picture here in a small size. So here is my original problem as I got it:

[Marsblessed]

@IliasG
Google search on LinearityUpperMargin shows me that you have just discussed all these parameters with A1ex in another thread of the forum. However, I have not find the conclusion yet. I mean the way of how it should be done (let alone the fact that you were talking there about cr2 into dng conversion).

By the way, DNG-result of cr2hdr contains the original value for the parameter in Exif data - 10000, although the data is 16 bps and this parameter should be at least 40000.

[IliasG]

@Marsblessed,

Can you upload in www.filebin.net the dual3185.cr2 and dual3185.forged.cr2 ?.

We have no knowledge about these -Canon tags as they are not used by ufraw/dcraw/rawthwrapee. If you want more info you could ask Iliah Borg/Alex Tutubalin at www.rawdigger.com. It's the Rawdigger team that discovered and decoded them.

The cr2hdr process don't change the -Canon:xxx data tags. It only calculates the 16 bit black/white levels and writes them at -exif:Blacklevel   -exif:WhiteLevel and this is enough for most converters. 
Now if you will understand what is used by DXO regarding Black/White levels when converting *.CR2s ( -Canon:xxx tags or some kind of calculation or a predefined default value ..)  then you will be able to translate the exif:BlackLevel and exif:WhiteLevel to the needed by DxO info by either updating the -Canon.xxx tags (if they are used by DxO) or offsetting/scaling the raw data to match with DxO's calculations or predefined values ..

It's the BlackLevel you should care about as WL looks more or less correct ..

[Marsblessed]

Can you upload in www.filebin.net the dual3185.cr2 and dual3185.forged.cr2 ?.

here they are

It's the BlackLevel you should care about as WL looks more or less correct ...

As it appears to me now, the problem is not just the black level. You see, normalized color space is one and the same for both DNG and CR2 files - it is sRGB. However, the problem is that in CR2 file the range of absolute values in 14 bps form is narrower than the one from dng when SHR'ed by 2, which is the reason why a simple addition of 139 to each dng-value corrupts the data in CR2 - the highest values translates into the values which are beyond cr2-white level (predefined or calculated by DxO). That is my assumption. The effect is visible on the JPGs above - even the clouds are slightly darker than in the original - the resulting rgb colors were shifted down a little.

In fact, in order to achieve the correct result, I have to convert the dng-values using normalization. That is why I need to know the correct black and white levels for both files.

By the way, I have seen in the other threads that A1ex mentioned that he wrote an automatic white level detector which determine the unique white level for each individual CR2-file. It seems to me that I will have to use his procedure to do the same job or I will never find the proper backward transformation for a particular CR2-file.


UPDATE:
I was right about normalization. Test confirmed my assumption. Here is the result:
origin

normalized forged


it was done by applying ranges built on black and white levels from exif. There is no green tint anymore, though the overall brightness is lower (do not know why).

[IliasG]

Please, don't mixup things. Raw files have no color space. There is no color space involved nor any relation to color space in the calculations of cr2hdr nor in your division by 4 of the DNG raw values nor in setting the black and white levels. In fact the BL & WL are the clipping points for a converter .. every value lower than BL is assigned to the BL and every value higher than the WL is assigned as equal to WL. In a 0-1 range any RawValue <=BL becomes 0 and any >= WL becomes 1 ..

By adding an offset of 139/14bit you take a very small increase in clipped highlights. The WL in the dualDNG is 50104 and should be in the forged.cr2 50104/4 =  12526. This after subtracting the BL (7964/4 = 1991) becomes 12526-1991= 10535. Adding 139 = 10674 .. You loose log2(10535) - log2(10674) = 0.02 stops .. it's negligible ..

Of course the normalization of all the range is better than the offset but you have to know the values that DxO uses as BL & WL ..

Is the ..forged.jpeg sample from the forged.cr2 that you uploaded or from a later normalized forged.cr2 ?. Looks good to me ..
Can you describe how exactly do you normalize the values ?

You shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..

[Marsblessed]

@IliasG
Why should I always prove that my understanding is correct? Though, I try to be patient. As everyone could see at the wiki-page, the definition of a color space is:
a color space is a color model with a certain mapping function between the color model and a certain reference color space where a color model is an abstract mathematical model describing the way colors can be represented as tuples of numbers.
Hence, my reference to a color space in the context of the trouble is perfectly justified. We have one color space for CR2-files and another color space for DNG-files. They ARE color spaces by the definition - they have both mathematical representations of color components AND mapping functions to an absolute color space - the sRGB. Therefore, when I found out that simple SHR by 2 is not a proper transformation between the two, I asked someone to help me to define the correct transformation. Why then A1ex started to point me out that the color space is one and the same? No, IliasG and A1ex, they are not. CR2HDR transforms one color space of CR2-files into another color space which by the definition will never be the same color space and the reason for that is the fact that the numbers in the resulting DNG which represent certain colors are different from those in the original CR2-file. Does it sound convincing now?

Of course the normalization of all the range is better than the offset but you have to know the values that DxO uses as BL & WL ..

I have checked WL&BL for this file only - they are exactly the values which are defined by Canon in Exif. Any attempt to exceed the WL by applying different ranges to multiply after normalization results in a corrupted data error in the DxO. Though, I still have another couple of files where the WL is different - it is 15092 (at ISO200) instead of 12277 (ISO100) but I will be very surprised if the DxO have another WL (no meter why).

Is the ..forged.jpeg sample from the forged.cr2 that you uploaded or from a later normalized forged.cr2 ?. Looks good to me ..

No, it is a new file with the same name - my tool uses this name-template for the resulting files! Here is the new one if you need it.

Can you describe how exactly do you normalize the values ?

Easily but using C and do not forget that it was just a test - all the values are hard-coded for the particular file. Take a look:

Code: [Select]

      val = ((val - 7964) * 40920) / 42140;
      mcu[j] = (ushort) (2047 + (val >> 2));where the "val" is the original value from a DNG-data and "mcu[j]" is the resulting value to be compressed and saved in the forged CR2-file. 42140 is the subtraction of BL from WL for DNG's WL&BL, 40920 is the same for CR2 but multiplied by 4, and 7964 and 2047 are the BLs. As you can also see, I did not do any special roundings but that will be done in the final version (sooner or later).

You shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..

Ok, it was just a thought.

[IliasG]

Not convinced  .. you need a big stretch of definitions to say there is a color space encapsulating raw data. No primaries but spectral responses, no gamut, no way to take color fidelity by linear transformations (see the good raw input color profiles use types of non linear ( not mathematically defined ) mappings ), 
http://www.dpreview.com/forums/post/53023856 and other posts of Iliah on the subject

Thanks for the sample and the normalization description .. I'll have to download a demo Dxo to try the difference because with RT the result is the same

So the -Canon:xx tags are not used (as nothing changes by changing them) but somehow are the correct values to use with DxO !!.

[IliasG]

I think you'll not need normalization any more ..

Inspecting the jpeg samples .. both render the green tree as almost black ..

Inspecting the raw data in both .. forged.cr2 and ..normed.forged.cr2 you have wrong green values in the first column (column 0, channel G2), it's average is around double than the other pixel's avg in the optically black area. This could give a wrong BL if DxO includes this column in BL calculations and calculates a pure average. Dcraw and derivatives don't use the first 2 or 4 columns (nor the last ..) so their calculation is correct.

[IliasG]

@Alex,

Do the cr2hdr calculate the BL from the top OB area ?.. The usual calculation area is the left OB and there the avg is 7852-7956 while the BL (read in the DNG's exif) is 7964 and matches the avg of the top OB area (2 first lines excepted ..)

The rendering at the darks looks better with BL calculated from the side OB area I think, or perhaps the avg of both OB areas is better .. didn't pixel peeped much

[a1ex]

Yes, before processing (function black_subtract) it groups the OB area in 8 column groups IIRC (the difference was most obvious in 7D). After processing, I do a fine-tuning step from the left OB only (black_subtract_simple). The code from the 20-bit branch should be a bit better.

Do you have an example of black level being incorrect after processing with cr2hdr?

[IliasG]

No just the above samples where you also saw in the dual3174.DNG has a bit green tint at the darks. The BL in DNG tag is 1991/14bit while dcraw calculates 1988 from the side OB ..

[Marsblessed]

You shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..

By the way we have a chance to do that!
normal vs dual without any post:


normal vs dual with some post (same settings):


DxO has done almost magic! Black areas of the normal shot are just a little more brown than the same areas of the dual. Though, it could be my monitor which makes the difference barely visible.
the CR2-files are here

[IliasG]

Looks like the "first column error" remains although it is faded down (by the normalization step ?).

In 3187.forged.cr2 I measure in the first two columns. 1st column Ravg=2045.8 G2avg=2247.1 - 2nd col. G1avg=2045.7 Bavg=2046.2
Averaging all 0-72 columns of the left OB area we take R=2046.3, G1=2046.4, B=2046.9, G2=2051.5 (this +5 can make the slight difference at the darks we see )
Averaging 2-70 we take a correct measure R=2046.3, G1=2045.9, B=2046.2, G2=2045.9

This gets more significant in the not normalized forged cr2s.

It was much more before with 3185 (1988 vs 3740 in forged, 2044 vs 3748 in normed.forged)
http://www.magiclantern.fm/forum/index.php?topic=10895.msg106484#msg106484

Does the rendering remains buggy on a forged.cr2 without the "first column error" if you don't apply the normalization step ?

The jpegs looks almost same with slightly darker-greener darks in the forget.jpeg.  Nice !. I think it will be better after correcting the first column ..

[a1ex]

Please don't actually do a SHR 2 because it introduces a small bias in values.

In the case of [aaaaaaaaaaaaaabb], you are forcing bb to be always 00 when it was 01, 10, or 11 before the shift.
These might not matter much when you're much higher than the black level, but rounding errors can throw out information for the darkest stop of data.

Two other ways to do it are:

Flat rounding - no bias nearest neighbor dithering
if bit15 is 1, add 1 to bit14

Random rounding - random dither
01 - 25% chance of adding 1 to bit14
10 - 50%
11 - 75%

Quick posterization experiment.

Code: [Select]

x = linspace(0,10,500);             # simulate an analog signal from 0 to 10
im = ones(50,1) * x;                # extend to get an image
rn = randn(size(im));               # Gaussian noise of stdev = 1
ru = rand(size(im)) - 0.5;          # uniform noise between -0.5 and 0.5

# add Gaussian noise before quantization
for stdev = [0.1 0.2 0.3 0.4 0.5 1]
    imrn = round(im + rn*stdev);
    imwrite(imrn/10, sprintf("randn%g.png", stdev));
end

# add uniform noise before quantization
for factor = [1 1.5 2 3]
    imrn = round(im + ru*factor);
    imwrite(imrn/10, sprintf("rand_x%g.png", factor));
end

no noise before quantization
stdev=0.1
stdev=0.2
stdev=0.3
stdev=0.4
stdev=0.5
stdev=1

factor=1
factor=1.5
factor=2
factor=3

To me, the sweet spot seems to be with Gaussian noise of stdev=0.4 added be before quantization.

Definitely worth trying this in cr2hdr and ufraw-mod.

[IliasG]

I would say that stdev=0.5 is better to my eyes but is there a real value to find which dithering is better on noiseless values ?.

Is this experiment to find an optimal 20to16 bit conversion in cr2hdr-20bit ?

I recall John sheehy writing many times in Dpreview forums that according to his experiments if the stdev in the exported raw file is more than 1 ADU (I think he says around 1.3 ADUs is optimal) there is no danger for posterization even under heavy processing.

BTW nice improvement with the latest cr2hdr-20bit !!.

[Audionut]

For me, stdev=0.5 is slightly better also.  It has a more uniform appearance.

Can you make the noise, uniform over the entire image, rather then just on the level boundaries?

[Marsblessed]

May I have some details, A1ex?

1) Are you sure that it is a correct Guassian distribution after being multiplied by another stdev? It is impossible that G(stdev=0.5) is "equal" to 0.5*G(stdev=1). At least according to this definition.

2) as far as I understand, it has nothing to do with the mentioned "Random rounding - random dither", which means it does not take into account the reminder (which is 00, 01, 10 or 11 in my case). Is my assumption correct?

[Shri]

Hi everyone,
I’ve read a lot of theoretical post above and understood nothing! I am simply a photographer and I need simple way to post process my photos. I used to convert my Cr2 files in Canon’s DPP. And for me it works perfect. So, can somebody tell me, is there a way to convert Dual ISO file in normal Canon Cr2 file? Can somebody write a converter that will convert dual ISO files in ordinary Cr2 file? I don’t like to use DNG’s and Photoshop.  Photoshop I use only to print my photos.
Regards,

[Walter Schulz]

http://www.magiclantern.fm/forum/index.php?topic=9102.0
http://www.magiclantern.fm/forum/index.php?topic=8520.0
http://www.magiclantern.fm/forum/index.php?topic=8738.0

[Marsblessed]

is there a way to convert Dual ISO file in normal Canon Cr2 file? Can somebody write a converter that will convert dual ISO files in ordinary Cr2 file?

That is exactly what I am working on. I have made a version of the converter, which I should tune for each dng-file separately. Though, I still have some problems to be solved. The most important one is how the values in a DNG correlates to those in a CR2. That is the question to A1ex but as far as I see, nor had he time neither had he desire to explain his conversion in details. On the other hand, I still have not had time to investigate it in the sources of the cr2hdr which are far too heavy to just sit and have it done. It is almost miracle that I have made such progress - I HAVE the cr2-file in the end which could be used in any supporting software but without the proper transformation it could result in an improper picture. However, as you can see, the differences are barely visible (at least in the examples above). Therefore, there is only one thing for you to do - wait for my final version which will appear sooner or later. Or you could go along my way in order to create your own tool.