Hi everyone!
If you ever been an owner of DxO Software, you would agree with me that there is no way of doing anything in the software with dual_iso-dng. That is a real problem especially when you do not want to have such a heavy peace of software as Adobe Photoshop is. Thus, there is only one option after making dual_iso shots: BACK into cr2!
Well, at first, it looks completely impossible - private Canon RAW file format and the result of the cr2hdr, which is not only in dng format, but also has 16 bits per sample instead of 14 bps. However, is it really unfeasible?
The good news: No, it is NOT unfeasible. I did it. There is no trouble in replacing the RAW data in a CR2-file. Not at all. It works fine in the DxO Optics Pro now!
The bad news: I need help. It turns out that the raw values in the dng-result are not that simple to convert them back into 14 bps. SHR by 2 creates something really dark with a strange tint of green although the bright areas look perfectly normal. It just means that I need a proper transformation from the cr2hdr resulting color space into Canon original color space. Therefore, I would appreciate it if someone could help me to write the proper transformation in C. The problem is the black level which does not correspond to the original one from cr2. I will have to dig into the cr2hdr sources.
There's no change in color space; the two exposures are simply scaled to match in the original color space (whatever that may be, as long as it's close enough to RGB).
My best guess is that you have simply ran out of bits. There are situations where not even 16 bits are enough, and the 20-bit branch of cr2hdr includes an option to compress the highlights and allow more bits for shadows: http://www.magiclantern.fm/forum/index.php?topic=7139.msg105120#msg105120
How did you replace the data in the original CR2? did you use libcraw2 or you wrote your own?
Thank you for your answer, A1ex, I was waiting for it first.
Concerning my results - you see, my transformation is simple - just shr by 2. I have made an assumption that if the original color space is linear than it is perfectly justified that in order to achieve 14 bps I should only make a simple linear transformation. Nonetheless, the result is unexpectedly green in the dark areas and the picture is much darker than the original (shadows transformed into deep shadows or even black), which proof one of the things: there is a transformation of original cr2-color space in cr2hdr and it is
not linear OR the original Canon color space is itself
not linear. The latter, however, should not be true if the cr2-values are raw levels - voltage, in fact (correct me if I am wrong with that assumption), which again bring us back to the idea that there is some color transformation in cr2hdr and it is not linear. It is difficult for me to check the code of cr2hdr that fast and that is why I started this topic.
Quote from: a1ex on March 09, 2014, 07:08:25 PMyou have simply ran out of bits.
Well, I would agree with you if it were not possible to have normal dynamic range using just 8 bps. The problem is deeper, and it does not matter whether it is 14 bps or as much as 20. If it were just shadows I would not say a word about it - I still have a lot to learn about appropriate ISO's in different scenes. This is the green tint which makes it all wrong. It means that the original and the resulting color spaces are not correspond to each other.
Quote from: a1ex on March 09, 2014, 07:08:25 PMHow did you replace the data in the original CR2? did you use libcraw2 or you wrote your own?
Well, that is quite a story and probably if I were acquainted with libcraw, I would not spend that much time. I did it using a result of comprehension of the dcraw.c, ITU-T.81, and ppmtoljpg example (which is based on the libjpeg). My tool is actually a mix of code excerpts from dcraw.c and ppmtoljpg.c and hard-coded values from 650d version of cr2-files (I have no idea whether the values are different in the other models cr2-files but I guess they could be). Actually, it was really a surprise for me that cr2-files (as well as dng) are not special files but just tiff-files with some private tags known only to Canon, and the raw data in such a file is just another lossless jpeg file inside tiff. Moreover, the ljpg-file is in the end of cr2-file and in order to replace the data you should only compress new data into ljpgf-file, replace the old one with it and modify the size of the "compressed data" in the "header" (tiff tag 279 of the corresponding ifd). The problem, however, is lossless jpeg compression, but I managed to find its free implementation, which suits good to the purpose.
Again, with default option, cr2hdr outputs *linear* images in the *exact* *same* color space as Canon's. The only situation when the output is nonlinear is with the --soft-film option (this is why it requires you to specify the WB in advance).
Without sample images, it's hard to diagnose from tons of text. Another guess would be black level (which is not simply multiplied by 4 in cr2hdr).
I have perfect samples of cr2-files, but I have no idea of how to add these files to the message. I would be grateful if someone could help me with that.
dropbox, wetransfer
You can use any number of free file hosts (https://www.google.com.au/search?q=file+host&ie=utf-8&oe=utf-8&rls=org.mozilla:en-US:official&client=firefox-a&channel=sb&gws_rd=cr&ei=-1cdU6e9HInwkAWpjoGIAg).
Upload your sample to one of these, and copy the link.
I personally prefer Dropbox (https://www.dropbox.com/).
Meanwhile, I have checked carefully my results and it turns out that it is the DxO which makes it all green by amplifying shadows. The picture itself (without DxO modification) is much much darker than I expected. Sure it should be done by someone else to check whether my dual_iso-shot were done "properly" in the first place. Will add the sample a little later.
Quote from: Marsblessed on March 10, 2014, 07:28:50 AM
The picture itself (without DxO modification) is much much darker than I expected.
That's because you are viewing linear data.
Quote from: a1ex on March 09, 2014, 09:11:43 PM
Again, with default option, cr2hdr outputs *linear* images in the *exact* *same* color space as Canon's.
Quote from: dmilligan on March 10, 2014, 12:17:35 PM
That's because you are viewing linear data.
This linear data thing... Is that why dual iso pictures are typically very dark until one increases the shadow brightness? What is the root cause of that?
Yes. If you want to get normal brightness, use --soft-film to burn some (nonlinear) exposure compensation in the output DNG.
Sample to analyze:
DUAL3174.CR2 (https://www.dropbox.com/s/v6n3fxo1o3gojpr/DUAL3174.CR2)
that is the original cr2-file. After applying cr2hdr and recompress the result into a new cr2-file, the DxO shows it very green or too dark.
the result file:
DUAL3174.forged.cr2 (https://www.dropbox.com/s/1zxneektq8n6ha6/DUAL3174.forged.cr2)
==============
Here is another example which is not that bad with dual-ISO. The same green effect is clearly visible even on the pure cr2hdr-result! Take a look:
original CR2 (https://www.dropbox.com/s/576mggm9jymal9w/DUAL3175.CR2)
hdr CR2 (https://www.dropbox.com/s/tgy72fx02l3r1x5/DUAL3175.forged.CR2)
@a1ex
Could you please confirm that the data sample in the 16bps-DNG-file is two bytes long without any shift which means that every two bytes in the data represent single color value? Is the order of the colors the same as in CR2 = one line is rg rg rg rg ... and the next line is gb gb gb gb ...? My result looks as if I have changed the order of the colors in one line of every pair.
I'm not sure where to look. The output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.
My DxO demo license expired a few months ago.
The byte order in the DNG is according to Adobe spec (but different from Canon's internal buffer described in raw.h), and the order of colors is the same. I force it to RGGB while processing, but I switch it back before saving the file.
Besides black level, another hypothesis could be the noise distribution in the optical black areas. After processing with cr2hdr, it's no longer Gaussian.
Quote from: a1ex on March 11, 2014, 01:07:49 AMThe output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.
UFRaw is based on the dcraw.c which shows the same Filter pattern for the resulting DNG-file as it does for the original CR2-file. That is why I supposed that the data in the DNG-file is completely identical to CR2-file except bps and compression. May be this is the key to green color (which appears not only in deep shadows as it could be seen in DUAL3175.forged.cr2)
Quote from: Marsblessed on March 09, 2014, 03:31:23 PM
Hi everyone!
If you ever been an owner of DxO Software, you would agree with me that there is no way of doing anything in the software with dual_iso-dng. That is a real problem especially when you do not want to have such a heavy peace of software as Adobe Photoshop is. Thus, there is only one option after making dual_iso shots: BACK into cr2!
I don't have DxO converter but
What does DxO displays when opening dualISO dngs ?.
Quote from: Marsblessed on March 11, 2014, 10:05:26 AM
UFRaw is based on the dcraw.c which shows the same Filter pattern for the resulting DNG-file as it does for the original CR2-file. That is why I supposed that the data in the DNG-file is completely identical to CR2-file except bps and compression. May be this is the key to green color (which appears not only in deep shadows as it could be seen in DUAL3175.forged.cr2)
As Alex wrote, the usual reason for dark greenish picture is wrong black Level.
For CR2s, Dcraw and derivatives calculate it from the side optically black area and the picture comes out correct (at least with RawTherapee that I tried). Looks like DxO does not use this method. It either has a hardcoded value as BlackLevel or reads it from exif -Canon data where the info is invalid after cr2hdr conversions.
For DNGs, Dcraw and derivatives read the BL and WL values from exif (not exif.Canon ) ..and it is again displayed correctly.
To check if DxO uses the -Canon data for BL/WL try the corrected dual DNG and ..forged.cr2 . I copied the values from DNG's exif data to -Canon data with exiftool.
http://filebin.net/6lagewyuhl/DUAL3174-BLWLcorrected.DNG
http://filebin.net/6lagewyuhl/DUAL3174.forged-BLcorrected.cr2
Quote from: IliasG on March 11, 2014, 01:09:14 PM
What does DxO displays when opening dualISO dngs ?
Well, I would not call the DxO software a converter. It is rather a simplified version of Photoshop, which can do some post. The problem is it
can not open DNG-files at all! If it were not the case I would not need to do anything with the CR2-files.
Quote from: IliasG on March 11, 2014, 01:09:14 PM
To check if DxO uses the -Canon data for BL/WL try the corrected dual DNG and ..forged.cr2 . I copied the values from DNG's exif data to -Canon data with exiftool.
http://filebin.net/6lagewyuhl/DUAL3174-BLWLcorrected.DNG
http://filebin.net/6lagewyuhl/DUAL3174.forged-BLcorrected.cr2
Tried this file -
it looks even darker now but it looks absolutly identical with the same green tint everywhere. Which means no, the black level correction does nothing. Again, the problem is not only in the dark areas. Every point in the compressed result of cr2hdr has more green than it should be. As A1ex stated, it is even seen in the UFRaw (which has nothing in common with DxO).
Quote from: a1ex on March 11, 2014, 01:07:49 AM
I'm not sure where to look. The output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.
...
The byte order in the DNG is according to Adobe spec (but different from Canon's internal buffer described in raw.h)
I have an idea and I will check it later. The dcraw.c use the byte order that is specified in the first two bytes of any tiff. Therefore, if Adobe has a special byte order for its data in dng-file (which is tiff too) and it is differ from the one that is specified in the tiff header than that could be the reason of green tint in UFRaw and after my conversion which is also based on dcraw.
Byte ordering mismatch will result in complete gibberish (not the case here).
Can you post some JPEGs to see what exactly you are talking about?
If DxO expects a black level of 2048, simply add a constant offset to your raw values to force your black level to that value. Usually, the black level from cr2hdr ends up a little lower than 2048*4.
Quote from: Marsblessed on March 11, 2014, 02:22:23 PM
Well, I would not call the DxO software a converter. It is rather a simplified version of Photoshop, which can do some post. The problem is it can not open DNG-files at all! If it were not the case I would not need to do anything with a CR2-files.
We are researching the raw conversion part here only, I think ..
Quote from: Marsblessed on March 11, 2014, 02:22:23 PMTried this file - it looks even darker now but with the same green tint everywhere. Which means no, the black level correction does nothing. Again, the problem is not only in the dark areas. Every point in the compressed result of cr2hdr has more green than it should be. As A1lex stated, it is even seen in the UFRaw (which has nothing in common with DxO).
So if it's darker this means that DxO uses these tags that I changed ..
Look what exactly Alex wrote below ..
the DNG is greener than your .forged.cr2, same as with Rawtherapee. And the reason is that the Black level tag in the DNG is higher that the calculated black level (7964/4 = 1991 vs 1988 calculated) ..
Quote from: a1ex on March 11, 2014, 01:07:49 AM
... The output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.
Please upload some converted samples ..
Forged vs DNG:
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/backtocr2/DUAL3174.forged.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/isoless/backtocr2/DUAL3174.jpg)
ufraw-batch --exposure=5 --out-type=jpg --shrink=2 --temperature=3500 --green=0.8 DUAL3174.forged.cr2
ufraw-batch --exposure=5.8 --out-type=jpg --shrink=2 --temperature=3500 --green=0.8 DUAL3174.DNG
Quote from: Marsblessed on March 11, 2014, 02:22:23 PM
Tried this file - it looks even darker now
well, again - thorough checking shows that my first conclusion was incorrect - there are no differences between DUAL3174.forged-BLcorrected.cr2 and DUAL3174.forged.cr2 in DxO
another couple of samples, produced by DxO with every feature switched off:
DUAL3185_DxO.jpg (https://www.dropbox.com/s/a464txxnwmmuk9u/DUAL3185_DxO.jpg)
DUAL3185.forged_DxO.jpg (https://www.dropbox.com/s/fpca5q6tarh309u/DUAL3185.forged_DxO.jpg)
Black level issue.
well, but notice that the sky is darker too despite the fact that it is done by interpolating between blue and white!
Besides, I have tried to add 2048 to every raw value of the picture - the result is corrupted data (can not be opend in the DxO)
At the same time, it means that I have to apply some more sophisticated logic to 16 bps raw values instead of just SHR by 2. (Again, that means that the color spaces are not identical)
I think I understand now. I have to compress 16 bps raw values back to the range between 2048*4 and 2^16-1 and only after that I should apply SHR by 2. A1ex, please, correct me if I am wrong.
Please don't actually do a SHR 2 because it introduces a small bias in values.
In the case of [aaaaaaaaaaaaaabb], you are forcing bb to be always 00 when it was 01, 10, or 11 before the shift.
These might not matter much when you're much higher than the black level, but rounding errors can throw out information for the darkest stop of data.
Two other ways to do it are:
Flat rounding - no bias nearest neighbor dithering
if bit15 is 1, add 1 to bit14
Random rounding - random dither
01 - 25% chance of adding 1 to bit14
10 - 50%
11 - 75%
That's a nice trick, I should actually use it too in cr2hdr.
Quote from: Marsblessed on March 11, 2014, 07:02:17 PM
I think I understand now. I have to compress 16 bps raw values back to the range between 2048*4 and 2^16-1 and only after that I should apply SHR by 2. A1ex, please, correct me if I am wrong.
As I understand it, you have to read from DNG exif the black level value (it was 7964 in dual3174.dng and become 7964/4 = 1991 in forged.cr2) then add 2048-1911=139 to all the values of forged.cr2 to normalize for the suspected DxO's default BL for 650D .
It is even more complicated. I can not just add something - the result is always corrupted data. As far as I see it now, I have to extract two ranges (black and white levels) from dng and cr2 and then compress or expand the dng-range into (multiplied by 4) cr2-range. After that I have to "SHR by 2" (using some rounding mechanism) and add cr2-black_level to the result before put it into cr2-file. I will test it later.
One question though:
which white level should I use from CR2 - SpecularWhiteLevel or NormalWhiteLevel (according to this (http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#ColorData7) description)?
I would bet on the specular .. this is closer to the peak white level one can see in the raw histograms. But one has to test first. Canon's DPP use all of them ..
Have you checked if DxO uses these tags at all .. by making extreme changes there an see the effect (if any) on DxO's conversion ?.
Related to the offset on values, now that you checked that by offsetting the values by the difference 2048-dng_Black_Level is a bad idea, you could try scaling by 2048/dng_Black_Level .. sounds also logical :).
The suspicion that DxO uses a default value (i.e. 2048) as "black level" was purely a guess based on that you wrote that nothing happens by changing the -Canon tags ..
If you can work with Rawtherapee you can change the black level used by RT in tab raw - black levels to see when RT's conversion matches DxO's. There i put +139.0 and took a dark greenish result like you described it ..
I have finally managed to show a picture here in a small size. So here is my original problem as I got it:
(https://dl.dropboxusercontent.com/s/a464txxnwmmuk9u/DUAL3185_DxO.jpg)
(https://dl.dropboxusercontent.com/s/fpca5q6tarh309u/DUAL3185.forged_DxO.jpg)
@IliasG
Google search on LinearityUpperMargin shows me that you have just discussed all these parameters with A1ex in another thread of the forum. However, I have not find the conclusion yet. I mean the way of how it should be done (let alone the fact that you were talking there about cr2 into dng conversion).
By the way, DNG-result of cr2hdr contains the original value for the parameter in Exif data - 10000, although the data is 16 bps and this parameter should be at least 40000.
@Marsblessed,
Can you upload in www.filebin.net the dual3185.cr2 and dual3185.forged.cr2 ?.
We have no knowledge about these -Canon tags as they are not used by ufraw/dcraw/rawthwrapee. If you want more info you could ask Iliah Borg/Alex Tutubalin at www.rawdigger.com. It's the Rawdigger team that discovered and decoded them.
The cr2hdr process don't change the -Canon:xxx data tags. It only calculates the 16 bit black/white levels and writes them at -exif:Blacklevel -exif:WhiteLevel and this is enough for most converters.
Now if you will understand what is used by DXO regarding Black/White levels when converting *.CR2s ( -Canon:xxx tags or some kind of calculation or a predefined default value ..) then you will be able to translate the exif:BlackLevel and exif:WhiteLevel to the needed by DxO info by either updating the -Canon.xxx tags (if they are used by DxO) or offsetting/scaling the raw data to match with DxO's calculations or predefined values ..
It's the BlackLevel you should care about as WL looks more or less correct ..
Quote from: IliasG on March 13, 2014, 12:55:52 PM
Can you upload in www.filebin.net the dual3185.cr2 and dual3185.forged.cr2 ?.
here they are (http://filebin.net/qlh1lwzjwt)
Quote from: IliasG on March 13, 2014, 12:55:52 PM
It's the BlackLevel you should care about as WL looks more or less correct ...
As it appears to me now, the problem is not just the black level. You see, normalized color space is one and the same for both DNG and CR2 files - it is sRGB. However, the problem is that in CR2 file the range of absolute values in 14 bps form is narrower than the one from dng when SHR'ed by 2, which is the reason why a simple addition of 139 to each dng-value corrupts the data in CR2 - the highest values translates into the values which are beyond cr2-white level (predefined or calculated by DxO). That is my assumption. The effect is visible on the JPGs above - even the clouds are slightly darker than in the original - the resulting rgb colors were shifted down a little.
In fact, in order to achieve the correct result, I have to convert the dng-values using normalization. That is why I need to know the correct black and white levels for both files.
By the way, I have seen in the other threads that A1ex mentioned that he wrote an automatic white level detector which determine the unique white level for each individual CR2-file. It seems to me that I will have to use his procedure to do the same job or I will never find the proper backward transformation for a particular CR2-file.
UPDATE:
I was right about normalization. Test confirmed my assumption. Here is the result:
origin
(https://dl.dropboxusercontent.com/s/zex4uliz0b82vzc/DUAL3185.2.jpg)
normalized forged
(https://dl.dropboxusercontent.com/s/56sbrg7e2bwr82c/DUAL3185.forged.2.jpg)
it was done by applying ranges built on black and white levels from exif. There is no green tint anymore, though the overall brightness is lower (do not know why).
Please, don't mixup things. Raw files have no color space. There is no color space involved nor any relation to color space in the calculations of cr2hdr nor in your division by 4 of the DNG raw values nor in setting the black and white levels. In fact the BL & WL are the clipping points for a converter .. every value lower than BL is assigned to the BL and every value higher than the WL is assigned as equal to WL. In a 0-1 range any RawValue <=BL becomes 0 and any >= WL becomes 1 ..
By adding an offset of 139/14bit you take a very small increase in clipped highlights. The WL in the dualDNG is 50104 and should be in the forged.cr2 50104/4 = 12526. This after subtracting the BL (7964/4 = 1991) becomes 12526-1991= 10535. Adding 139 = 10674 .. You loose log2(10535) - log2(10674) = 0.02 stops .. it's negligible ..
Of course the normalization of all the range is better than the offset but you have to know the values that DxO uses as BL & WL ..
Is the ..forged.jpeg sample from the forged.cr2 that you uploaded or from a later normalized forged.cr2 ?. Looks good to me ..
Can you describe how exactly do you normalize the values ?
You shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..
@IliasGWhy should I always prove that my understanding is correct? Though, I try to be patient. As everyone could see at the wiki-page (http://en.wikipedia.org/wiki/Color_space), the definition of a color space is:
a color space is a color model with a certain mapping function between the color model and a certain reference color space where a color model is
an abstract mathematical model describing the way colors can be represented as tuples of numbers.Hence,
my reference to a color space in the context of the trouble is perfectly justified. We have one color space for CR2-files and another color space for DNG-files. They ARE color spaces by the definition - they have both mathematical representations of color components AND mapping functions to an absolute color space - the sRGB. Therefore, when I found out that simple SHR by 2 is not a proper transformation between the two, I asked someone to help me to define the correct transformation. Why then A1ex started to point me out that the color space is one and the same? No, IliasG and A1ex, they are not. CR2HDR transforms one color space of CR2-files into another color space which
by the definition will
never be the same color space and the reason for that is the fact that the numbers in the resulting DNG which represent certain colors are different from those in the original CR2-file. Does it sound convincing now?
Quote from: IliasG on March 14, 2014, 05:07:14 PM
Of course the normalization of all the range is better than the offset but you have to know the values that DxO uses as BL & WL ..
I have checked WL&BL for this file only - they are exactly the values which are defined by Canon in Exif. Any attempt to exceed the WL by applying different ranges to multiply after normalization results in a corrupted data error in the DxO. Though, I still have another couple of files where the WL is different - it is 15092 (at ISO200) instead of 12277 (ISO100) but I will be very surprised if the DxO have another WL (no meter why).
Quote from: IliasG on March 14, 2014, 05:07:14 PM
Is the ..forged.jpeg sample from the forged.cr2 that you uploaded or from a later normalized forged.cr2 ?. Looks good to me ..
No, it is a new file with the same name - my tool uses this name-template for the resulting files! :P Here is the new one if you need it. (http://filebin.net/c1nz310a0u)
Quote from: IliasG on March 14, 2014, 05:07:14 PM
Can you describe how exactly do you normalize the values ?
Easily but using C and do not forget that it was just a test - all the values are hard-coded for the particular file. Take a look:
val = ((val - 7964) * 40920) / 42140;
mcu[j] = (ushort) (2047 + (val >> 2));where the "val" is the original value from a DNG-data and "mcu[j]" is the resulting value to be compressed and saved in the forged CR2-file. 42140 is the subtraction of BL from WL for DNG's WL&BL, 40920 is the same for CR2 but multiplied by 4, and 7964 and 2047 are the BLs. As you can also see, I did not do any special roundings but that will be done in the final version (sooner or later).
Quote from: IliasG on March 14, 2014, 05:07:14 PMYou shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..
Ok, it was just a thought.
Not convinced .. you need a big stretch of definitions to say there is a color space encapsulating raw data. No primaries but spectral responses, no gamut, no way to take color fidelity by linear transformations (see the good raw input color profiles use types of non linear ( not mathematically defined ) mappings ),
http://www.dpreview.com/forums/post/53023856 and other posts of Iliah on the subject
Thanks for the sample and the normalization description .. I'll have to download a demo Dxo to try the difference because with RT the result is the same :)
So the -Canon:xx tags are not used (as nothing changes by changing them) but somehow are the correct values to use with DxO !!.
I think you'll not need normalization any more ..
Inspecting the jpeg samples .. both render the green tree as almost black ..
Inspecting the raw data in both .. forged.cr2 and ..normed.forged.cr2 you have wrong green values in the first column (column 0, channel G2), it's average is around double than the other pixel's avg in the optically black area. This could give a wrong BL if DxO includes this column in BL calculations and calculates a pure average. Dcraw and derivatives don't use the first 2 or 4 columns (nor the last ..) so their calculation is correct.
@Alex,
Do the cr2hdr calculate the BL from the top OB area ?.. The usual calculation area is the left OB and there the avg is 7852-7956 while the BL (read in the DNG's exif) is 7964 and matches the avg of the top OB area (2 first lines excepted ..)
The rendering at the darks looks better with BL calculated from the side OB area I think, or perhaps the avg of both OB areas is better .. didn't pixel peeped much :)
Yes, before processing (function black_subtract) it groups the OB area in 8 column groups IIRC (the difference was most obvious in 7D). After processing, I do a fine-tuning step from the left OB only (black_subtract_simple). The code from the 20-bit branch should be a bit better.
Do you have an example of black level being incorrect after processing with cr2hdr?
No just the above samples where you also saw in the dual3174.DNG has a bit green tint at the darks. The BL in DNG tag is 1991/14bit while dcraw calculates 1988 from the side OB ..
Quote from: IliasG on March 14, 2014, 05:07:14 PM
You shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..
By the way we have a chance to do that! :)
normal vs dual without any post:
(https://dl.dropboxusercontent.com/s/298b4slcbb9tfdf/IMG_3186_DxO.jpg)(https://dl.dropboxusercontent.com/s/jfbp5ehrd8muh57/DUAL3187.forged_DxO.jpg)
normal vs dual with some post (same settings):
(https://dl.dropboxusercontent.com/s/7wh5a1c6dldjlbm/IMG_3186_DxO_with_post.jpg)(https://dl.dropboxusercontent.com/s/gj1a24nwsuh360i/DUAL3187.forged_DxO_with_post.jpg)
DxO has done almost magic! Black areas of the normal shot are just a little more brown than the same areas of the dual. Though, it could be my monitor which makes the difference barely visible.
the CR2-files are here (http://filebin.net/w4xw3zqd2n)
Looks like the "first column error" remains ;) although it is faded down (by the normalization step ?).
In 3187.forged.cr2 I measure in the first two columns. 1st column Ravg=2045.8 G2avg=2247.1 - 2nd col. G1avg=2045.7 Bavg=2046.2
Averaging all 0-72 columns of the left OB area we take R=2046.3, G1=2046.4, B=2046.9, G2=2051.5 (this +5 can make the slight difference at the darks we see )
Averaging 2-70 we take a correct measure R=2046.3, G1=2045.9, B=2046.2, G2=2045.9
This gets more significant in the not normalized forged cr2s.
It was much more before with 3185 (1988 vs 3740 in forged, 2044 vs 3748 in normed.forged)
http://www.magiclantern.fm/forum/index.php?topic=10895.msg106484#msg106484
Does the rendering remains buggy on a forged.cr2 without the "first column error" if you don't apply the normalization step ?
The jpegs looks almost same with slightly darker-greener darks in the forget.jpeg. Nice !. I think it will be better after correcting the first column .. :)
Quote from: Shizuka on March 11, 2014, 09:55:12 PM
Please don't actually do a SHR 2 because it introduces a small bias in values.
In the case of [aaaaaaaaaaaaaabb], you are forcing bb to be always 00 when it was 01, 10, or 11 before the shift.
These might not matter much when you're much higher than the black level, but rounding errors can throw out information for the darkest stop of data.
Two other ways to do it are:
Flat rounding - no bias nearest neighbor dithering
if bit15 is 1, add 1 to bit14
Random rounding - random dither
01 - 25% chance of adding 1 to bit14
10 - 50%
11 - 75%
Quick posterization experiment.
x = linspace(0,10,500); # simulate an analog signal from 0 to 10
im = ones(50,1) * x; # extend to get an image
rn = randn(size(im)); # Gaussian noise of stdev = 1
ru = rand(size(im)) - 0.5; # uniform noise between -0.5 and 0.5
# add Gaussian noise before quantization
for stdev = [0.1 0.2 0.3 0.4 0.5 1]
imrn = round(im + rn*stdev);
imwrite(imrn/10, sprintf("randn%g.png", stdev));
end
# add uniform noise before quantization
for factor = [1 1.5 2 3]
imrn = round(im + ru*factor);
imwrite(imrn/10, sprintf("rand_x%g.png", factor));
end
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/zeronoise.png) no noise before quantization
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/randn0.1.png) stdev=0.1
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/randn0.2.png) stdev=0.2
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/randn0.3.png) stdev=0.3
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/randn0.4.png) stdev=0.4
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/randn0.5.png) stdev=0.5
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/randn1.png) stdev=1
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/rand_x1.png) factor=1
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/rand_x1.5.png) factor=1.5
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/rand_x2.png) factor=2
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/posterize/rand_x3.png) factor=3
To me, the sweet spot seems to be with Gaussian noise of stdev=0.4 added be before quantization.
Definitely worth trying this in cr2hdr and ufraw-mod.
I would say that stdev=0.5 is better to my eyes but is there a real value to find which dithering is better on noiseless values ?.
Is this experiment to find an optimal 20to16 bit conversion in cr2hdr-20bit ?
I recall John sheehy writing many times in Dpreview forums that according to his experiments if the stdev in the exported raw file is more than 1 ADU (I think he says around 1.3 ADUs is optimal) there is no danger for posterization even under heavy processing.
BTW nice improvement with the latest cr2hdr-20bit !!.
For me, stdev=0.5 is slightly better also. It has a more uniform appearance.
Can you make the noise, uniform over the entire image, rather then just on the level boundaries?
May I have some details, A1ex?
1) Are you sure that it is a correct Guassian distribution after being multiplied by another stdev? It is impossible that G(stdev=0.5) is "equal" to 0.5*G(stdev=1). At least according to this definition (http://en.wikipedia.org/wiki/Gaussian_smoothing).
2) as far as I understand, it has nothing to do with the mentioned "Random rounding - random dither", which means it does not take into account the reminder (which is 00, 01, 10 or 11 in my case). Is my assumption correct?
Hi everyone,
I've read a lot of theoretical post above and understood nothing! I am simply a photographer and I need simple way to post process my photos. I used to convert my Cr2 files in Canon's DPP. And for me it works perfect. So, can somebody tell me, is there a way to convert Dual ISO file in normal Canon Cr2 file? Can somebody write a converter that will convert dual ISO files in ordinary Cr2 file? I don't like to use DNG's and Photoshop. Photoshop I use only to print my photos.
Regards,
http://www.magiclantern.fm/forum/index.php?topic=9102.0
http://www.magiclantern.fm/forum/index.php?topic=8520.0
http://www.magiclantern.fm/forum/index.php?topic=8738.0
Quote from: Shri on March 22, 2014, 11:42:48 AM
is there a way to convert Dual ISO file in normal Canon Cr2 file? Can somebody write a converter that will convert dual ISO files in ordinary Cr2 file?
That is exactly what I am working on. I have made a version of the converter, which I should tune for each dng-file separately. Though, I still have some problems to be solved. The most important one is how the values in a DNG correlates to those in a CR2. That is the question to A1ex but as far as I see, nor had he time neither had he desire to explain his conversion in details. On the other hand, I still have not had time to investigate it in the sources of the cr2hdr which are far too heavy to just sit and have it done. It is almost miracle that I have made such progress - I HAVE the cr2-file in the end which could be used in any supporting software but without the proper transformation it could result in an improper picture. However, as you can see, the differences are barely visible (at least in the examples above). Therefore, there is only one thing for you to do - wait for my final version which will appear sooner or later. Or you could go along my way in order to create your own tool.
@ Marsblessed and A1ex
As I understand it .. (correct me if I am wrong)
When on a homogenous sampling the values of a sample are multiplied we have avg=16 stdev=4 then on the 0.25X multiplied values we have avg= 4 stdev=1
I think gaussian stdevs are added in quadrature i.e. a stdev2 0.5 upon a stdev1 0.5 is sqrt(stdev1^2+stdev2^2) = 0.71
Now if the stdev in 20bit is 16 then after converting to 16bit the stdev will be 1.0 .. less than 1.3 (John Sheehy's safe limit) so there is a danger for posterization ..
we have to add in quadrature gaussian noise of stdev=13 so that we take total stdev=21 in 20bits and so 21/16 = 1.3 in 16bits :)
In case we use either gaussian noise with stdev 0.5*div_factor or uniform noise -0.5 to +0.5*div_factor we don't need the trick to add 0.5*div_factor to correct the bias that a simple shift will insert in the result,
The difference in quantization error after pure shift vs dithering or (maybe) the 0.5*div_factor offset is what is described as truncating model vs rounding model
http://en.wikipedia.org/wiki/Quantization_%28signal_processing%29#Quantization_error_models
@ Marsblessed
have you corrected the first column values ?. In your forged.CR2s the green pixels in the first column of the optically black area are not divided by 4 related to the same pixels in the dual.DNGs. It is very possible this is the reason for wrong black level's calculation by DxO.
@ IliasG
at this time nothing has been done to the dng-values excluding the normalisation. As I stated before the code is simple:
val = ((val - 7964) * 40920) / 42140;
mcu[j] = (ushort) (2047 + (val >> 2));
If you spotted the difference then that could be the difference of the cr2hdr versions which are used by both of us.
Mine is:
d0ac769 on 2014-01-23 10:13:39
Marsblessed,
I hope you succeed soon to create properly working converter and it will be great!
Best regards,
Quote from: Marsblessed on March 22, 2014, 03:39:47 PM
If you spotted the difference then that could be the difference of the cr2hdr versions which are used by both of us.
Mine is:
d0ac769 on 2014-01-23 10:13:39
Mine was the same and there are no first column bright pixels in the DNGs produced by this version. Maybe the libjpeg compression makes them ..
I will check it using the dcraw decompressor
@ IliasG
Finally! I have found the bug! 8) It was not the compression (which is not done by the libjpeg by the way). It was the division into a couple of strips which made the dirty thing in the data (which was strange ignorance of some values during processing). In the end it turns out that I forgot just a small thing - to add the equality case while dividing a row into the strips. That was all. Now pre- and post-compression results are perfectly identical. Will show some remade examples later.
Sorry that I don't have anything to contribute technically to this discussion. I just recently obtained a nightly build for the first time, a couple days ago (I've been using version 2.3 on my Canon 60D), and I love what Dual ISO does! It is a really great achievement. However, I, too, am a DXO Optics Pro user. I also use Lightroom (version 4.4). My reason for jumping into this discussion is to point out that DXO Optics Pro is not just a simplified version of Photoshop. DXO Optics Pro is almost without a doubt the best software for both optical corrections AND noise reduction. DXO Optic Pro's "PRIME" noise reduction is arguably the best noise reduction technology (and maybe the slowest as well!) available, and for a photographer working with higher ISO photos, PRIME would be a great complement to the wonders of dual ISO, bringing the best quality, lowest noise shadow detail possible. However, as noted, in order to use the automatic optical corrections and the PRIME noise reductions in DXO, the photographer must have a CR2 file.
So, I was happy to read that someone has taken up the torch on this, and I just wanted to provide my two cents that having a way to create cr2 files from dual iso output, either directly by the cr2hdr program, or through a reasonably simple post-process, would be welcomed by me and probably many other photographers. Thanks to everyone who is contributing to this development.
@
whyukonI am glad there is not just a couple of us who needs it to be in the Canon CR2 format. ;)
Quote from: whyukon on March 27, 2014, 07:20:18 PMor through a reasonably simple post-process
well, at this point it is as simple as the process with cr2hdr - just another console executable file to work with (and yes, in the end it could be integrated into a special version of cr2hdr but probably not by me)
Quote from: Marsblessed on March 27, 2014, 03:56:54 PM
@ IliasG
Finally! I have found the bug! 8) It was not the compression (which is not done by the libjpeg by the way). It was the division into a couple of strips which made the dirty thing in the data (which was strange ignorance of some values during processing). In the end it turns out that I forgot just a small thing - to add the equality case while dividing a row into the strips. That was all. Now pre- and post-compression results are perfectly identical. Will show some remade examples later.
:) :)
And now comes the dithering I suppose :) .. although as a first step you can just add +2 on the 16 bit values to remove the truncation effect 8)
@IliasG
No, not so fast. Nothing really changed after the debugging. The black level is far from the correct one. There is only one way left - to dig into CR2HDR sources... but I am ready for it now! :)
here are two examples (dual vs forged with post):
(https://dl.dropboxusercontent.com/s/c5uogaedtg3xx21/DUAL3104_DxO_with_post.jpg)(https://dl.dropboxusercontent.com/s/212qqrez73td8cc/DUAL3104.forged_DxO_with_post.jpg)
The CR2-files are here (http://filebin.net/sxi6yf18zb)
Make sure you use the 20-bit branch; I've just fixed a black level problem for 650D a few days ago.
Also, do try adding a constant offset to all raw pixels and observe the effect.
@A1ex
ok, I will try.
By the way, what would you say if I try to make a special version of CR2HDR based on 20-bit branch? I think it could be much much easier to me to adopt your solution (by simply adding few more modules) than implement a lot of the same things in my own way (using your code as an example).
Well, a option like --output-cr2 should be enough, right?
yep, this way it may work too! :)
Hey all. I'm VERY illiterate when it comes to Terminal and commands, scripts etc. I've been trying to figure out how to post process my MLV files. Thus far I am unsuccessful, have been doing a lot of reading, research, trial and error. Nothing will work for me. I have mlv_dump in a folder on my desktop along with my original .mlv files I need to process. I've tried the commands such as
cd desktop
cd mlv
./mlv_dump --dng M01-0705.MLV
this does NOTHING for me, only gives me an error message
" `_` is not recognized as an internal or external command, operable program or batch file"
I'm at my wits end over this, just want to understand what EXACTLY I need to do in order to successfully process these .mlv files. Any suggestions or hints would be GREATLY APPRECIATED!!!!
Not the right thread but you could try these links
http://www.magiclantern.fm/forum/index.php?topic=9731.0
http://www.magiclantern.fm/forum/index.php?topic=9560.msg108465#msg108465
A1ex, IliasG,
I have made a strange discovery. The data in the result of CR2HDR contains a LOT of zeros and 0xFFFF's. That is why a simple addition always lead to the corrupted data errors! Neither can I subtract something from 0 nor can I add something to 0xFFFF! Is it correct that COERCE-macro was not applied to the data using the black to white level's limits?
ADDITION:
I have also found a lot of zeros in the original CR2-data (at least in the data that was decompressed by the dcraw). There are lots of extremely high values too (far beyond white level). It looks to me now that it is a kind of normal situation. Thus, the question is how I should deal with all these extreme values.
Have an example?
As an example take a look at the CR2-files from the above link (http://filebin.net/sxi6yf18zb). Do not look at the forged one. Just use the original CR2-file which has a lot of zeros itself. Then check the result of CR2HDR (20 bits one). There will be a lot of zeros too.
Could not find any zeros (except for the first two top OB lines, which are gibberish anyway).
I can see zeros (and the values up to 0x0007) in the first four two lines - up to byte offset of 0x5280. However, 0x0008 values are almost everywhere. Which means not exactly zeros but near zeros values which in any way much lower than the black level. - it is a mistake. I will try to build a distribution of low and high values for better analyzing.
The darkest pixel, ignoring the first 4 lines, is 1584 in input and 8090 in output.
The black levels are 2046 for input (autodetected by cr2hdr) and 8184 for output (exiftool). There are 3 cold pixels found by cr2hdr.
Finally, I have got enough time to do some experiments with CR2HDR. There was a lot of struggle with hg but I mastered the skill and it is not an obstacle anymore (though, bitbucket still has not wanted to authorize me in order to push my changes into ml).
So far the results of my experiments are not that encouraging. I wish I could have a universal solution to the problem but right now it seems that such a solution does not exist. As a case in point, here are a couple of photos.
(dual vs forged with some post)
(https://dl.dropboxusercontent.com/s/c5uogaedtg3xx21/DUAL3104_DxO_with_post.jpg)(https://dl.dropboxusercontent.com/s/70m4uvh0zty6klq/DUAL3104.forged_4_DxO_with_post.jpg)
(https://dl.dropboxusercontent.com/s/iuhivg5jwo69vod/DUAL3107_DxO_with_post.jpg?dl=1&token_hash=AAGEqXxmMEyuoSZXcUIR4cILr6GmgWlbqO-X7Xudj94hhw)(https://dl.dropboxusercontent.com/s/9oc87jh9f293gao/DUAL3107.forged_7_DxO_with_post.jpg?dl=1&token_hash=AAGbXqH6uCu-9voUdkQjUceyD_UHFCAYesyMgocyug1JZA)
the CR2-files are here (http://filebin.net/2r9jfsuabv)
At first glance my results look perfect. The problem is the fact that I had had to add different offsets to the values of the original DNG-files in order to achieve adequate black levels. Moreover, as it could be seen on the lower photos, the resulting black level is not perfect at all - the radiator does look more pink than it should be and if I decrease the offset for just 1, the radiator will look ok but the floor will become really green.
The actual offsets were 4 for the top picture and 7 for the bottom plus the difference between the CR2HDR BL and Canon BL for each channel.
A1ex, there is also an issue with CR2HDR which could be seen on the bottom pictures - the suitcase (which is barely visible in the corner of the room) lost its color completely. It seems to me that it could be improved.
Quote from: Marsblessed on April 20, 2014, 09:58:42 PM
(though, bitbucket still has not wanted to authorize me in order to push my changes into ml).
It would be a little silly, if anyone and everyone, could push code changes to ML, don't you think.
http://www.magiclantern.fm/forum/index.php?topic=10774.msg104838#msg104838
Quote from: Audionut on April 21, 2014, 04:25:55 AM
It would be a little silly, if anyone and everyone, could push code changes to ML, don't you think.
http://www.magiclantern.fm/forum/index.php?topic=10774.msg104838#msg104838
Do you really think I have not read it?
You see, in order to create a pull request, I have to push my changesets into my bitbucket's account and that is exactly the point where I got "Authorization failed" despite using correct bitbucket login. It looks to me that I am forced to use ssh-keys instead of simple password and that "technique" I have not tried yet.
Quote from: Marsblessed on April 21, 2014, 06:58:34 AM
Do you really think I have not read it?
I don't know. I cannot read minds, that's probably why I linked it ;)
Your statement sounded like you were trying to push your changes directly to the ML repository. As opposed to forking, and creating a pull-request.
You don't need ssh keys.
Bitbucket 101 (https://confluence.atlassian.com/display/BITBUCKET/Bitbucket+101;jsessionid=F35FC6940780E12910048C2957979247).
Bitbucket documentation (https://confluence.atlassian.com/display/BITBUCKET/Bitbucket+Documentation+Home;jsessionid=FD594F4A778F343E8BF74F4C3F54B3EB).
Bitbucket support (https://bitbucket.org/support).
Quote from: Audionut on April 21, 2014, 07:40:22 AM
I don't know. I cannot read minds, that's probably why I linked it ;)
Your statement sounded like you were trying to push your changes directly to the ML repository. As opposed to forking, and creating a pull-request.
You don't need ssh keys.
Thank you. Actually, you were right - I was trying to push changes directly from my local rep-clone into the main ML-rep. Reviewing the documentation one more time helped me to find the appropriate way of doing things (https://confluence.atlassian.com/display/BITBUCKET/Fork+a+Repo%2C+Compare+Code%2C+and+Create+a+Pull+Request).
Quote from: Marsblessed on April 20, 2014, 09:58:42 PM
At first glance my results look perfect. The problem is the fact that I had had to add different offsets to the values of the original DNG-files in order to achieve adequate black levels. Moreover, as it could be seen on the lower photos, the resulting black level is not perfect at all - the radiator does look more pink than it should be and if I decrease the offset for just 1, the radiator will look ok but the floor will become really green.
If the results change by adjusting the offset by just 1, you need more than 14 bits. Try using --soft-film=4 to compress the highlights (maybe more, maybe less) - this should reserve more bits for the shadows and let you do a finer adjustment.
Quote from: Marsblessed on April 20, 2014, 09:58:42 PM
(dual vs forged with some post)
(https://dl.dropboxusercontent.com/s/iuhivg5jwo69vod/DUAL3107_DxO_with_post.jpg?dl=1&token_hash=AAGEqXxmMEyuoSZXcUIR4cILr6GmgWlbqO-X7Xudj94hhw)(https://dl.dropboxusercontent.com/s/9oc87jh9f293gao/DUAL3107.forged_7_DxO_with_post.jpg?dl=1&token_hash=AAGbXqH6uCu-9voUdkQjUceyD_UHFCAYesyMgocyug1JZA)
I'm trying to reproduce the noise pattern from the first image, but I'm not able to. Here's what I've got from ufraw+enfuse, without noise reduction:
(http://a1ex.magiclantern.fm/bleeding-edge/isoless/DUAL3107-ufraw-enfuse.jpg)
Do you have a conversion log for this image?
Do you mean the noise pattern of the left one of the two pictures? If so, then it seems that you regard it as a result of cr2hdr but it is not. It is the result of direct conversion of the original dual-CR2-file made by DxO. Due to reduction in size, the difference between dark and bright stripes has almost disappeared, although it is clearly visible (for example, on the wall).
I am creating a tutorial for RE:Vision Effects. I am their training and content manager. We have a new plug-in called DE:Flicker and I am working on a new tutorial for them to show an alternative use for DE:Flicker which would be would be producing a different workflow. DE:Flicker Auto levels can pick odd or even frames and drop the other so it cuts out all those other steps and also gives some other options inside auto levels.
My question is, if any of you have dual ISO HDR footage I could use for a tutorial to demonstrate this workflow? If I use your footage, we could give you a DE:Flicker plugin http://www.revisionfx.com/products/deflicker/
in addition to a credit and link to your website on the tutorial. Message me if any of you have footage. Thank you, Lori
Has there been any progress on this utility? I'm very interested in getting CR2 files out of the Dual ISO stills procedure as a) I use Canon's DPP as the first stage of my normal image processing and b) I don't seem to be able to load the .DNG files currently produced into Photoshop CS3.
PaulB - are you using DPP to only perform lens corrections and then exporting the CR2 to another editor? If so, I believe you can still do that, then export as a CR2 from DPP and then pass the file(s) over cr2hdr and then work with the DNGs...
No, I have not yet found a need to do lens corrections at all. What I tend to do in DPP is play with the colour saturation, brightness, contrast and shadow/highlight levels, do any cropping or rotations needed and then save the changes back to the CR2 file before saving a TIFF format copy for further processing in Photoshop CS3. I use CS3 for sharpening, fine adjustments and size reduction before saving as a JPG for upload to Flickr if the end result is up to par. I like the fact that the original image data is unchanged in the CR2 file, (which may also be true for a DNG file, I don't know), but for financial reasons I'm restricted to the tools I currently have, and I simply don't have anything that plays nice with the DNG files that cr2hdr produces. I've only had my 600D for a couple of months and didn't have any DSLR before that, so I'm still on a bit of a learning curve with it all.