Dual ISO: back into CR2

Started by Marsblessed, March 09, 2014, 03:31:23 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

IliasG

@ Marsblessed and A1ex

As I understand it .. (correct me if I am wrong)

When on a homogenous sampling the values of a sample are multiplied we have avg=16 stdev=4 then on the 0.25X multiplied values we have avg= 4 stdev=1

I think gaussian stdevs are added in quadrature i.e. a stdev2 0.5 upon a stdev1 0.5 is sqrt(stdev1^2+stdev2^2) = 0.71

Now if the stdev in 20bit is 16 then after converting to 16bit the stdev will be 1.0 .. less than 1.3 (John Sheehy's safe limit) so there is a danger for posterization ..
we have to add in quadrature gaussian noise of stdev=13 so that we take total stdev=21 in 20bits and so 21/16 = 1.3 in 16bits :)

In case we use either gaussian noise with stdev 0.5*div_factor or uniform noise -0.5 to +0.5*div_factor we don't need the trick to add 0.5*div_factor to correct the bias that a simple shift will insert in the result,

The difference in quantization error after pure shift vs dithering or (maybe) the 0.5*div_factor offset is what is described as truncating model vs rounding model
http://en.wikipedia.org/wiki/Quantization_%28signal_processing%29#Quantization_error_models


IliasG

@ Marsblessed

have you corrected the first column values ?. In your forged.CR2s the green pixels in the first column of the optically black area are not divided by 4 related to the same pixels in the dual.DNGs. It is very possible this is the reason for wrong black level's calculation by DxO.

Marsblessed

@ IliasG
at this time nothing has been done to the dng-values excluding the normalisation. As I stated before the code is simple:

      val = ((val - 7964) * 40920) / 42140;
      mcu[j] = (ushort) (2047 + (val >> 2));

If you spotted the difference then that could be the difference of the cr2hdr versions which are used by both of us.
Mine is:
d0ac769 on 2014-01-23 10:13:39
Canon EOS 650D with ML

Shri

Marsblessed,

I hope you succeed soon to create properly working converter and it will be great!

Best regards,

IliasG

Quote from: Marsblessed on March 22, 2014, 03:39:47 PM

If you spotted the difference then that could be the difference of the cr2hdr versions which are used by both of us.
Mine is:
d0ac769 on 2014-01-23 10:13:39

Mine was the same and there are no first column bright pixels in the DNGs produced by this version. Maybe the libjpeg compression makes them ..

Marsblessed

I will check it using the dcraw decompressor
Canon EOS 650D with ML

Marsblessed

@ IliasG
Finally! I have found the bug! 8) It was not the compression (which is not done by the libjpeg by the way). It was the division into a couple of strips which made the dirty thing in the data (which was strange ignorance of some values during processing). In the end it turns out that I forgot just a small thing - to add the equality case while dividing a row into the strips. That was all. Now pre- and post-compression results are perfectly identical. Will show some remade examples later.
Canon EOS 650D with ML

whyukon

Sorry that I don't have anything to contribute technically to this discussion.  I just recently obtained a nightly build for the first time, a couple days ago (I've been using version 2.3 on my Canon 60D), and I love what Dual ISO does!  It is a really great achievement.  However, I, too, am a DXO Optics Pro user.  I also use Lightroom (version 4.4).  My reason for jumping into this discussion is to point out that DXO Optics Pro is not just a simplified version of Photoshop. DXO Optics Pro is almost without a doubt the best software for both optical corrections AND noise reduction.  DXO Optic Pro's "PRIME" noise reduction is arguably the best noise reduction technology (and maybe the slowest as well!) available, and for a photographer working with higher ISO photos, PRIME would be a great complement to the wonders of dual ISO, bringing the best quality, lowest noise shadow detail possible.  However, as noted, in order to use the automatic optical corrections and the PRIME noise reductions in DXO, the photographer must have a CR2 file.

So, I was happy to read that someone has taken up the torch on this, and I just wanted to provide my two cents that having a way to create cr2 files from dual iso output, either directly by the cr2hdr program, or through a reasonably simple post-process, would be welcomed by me and probably many other photographers.  Thanks to everyone who is contributing to this development.

Marsblessed

@whyukon
I am glad there is not just a couple of us who needs it to be in the Canon CR2 format. ;)

Quote from: whyukon on March 27, 2014, 07:20:18 PMor through a reasonably simple post-process
well, at this point it is as simple as the process with cr2hdr - just another console executable file to work with (and yes, in the end it could be integrated into a special version of cr2hdr but probably not by me)
Canon EOS 650D with ML

IliasG

Quote from: Marsblessed on March 27, 2014, 03:56:54 PM
@ IliasG
Finally! I have found the bug! 8) It was not the compression (which is not done by the libjpeg by the way). It was the division into a couple of strips which made the dirty thing in the data (which was strange ignorance of some values during processing). In the end it turns out that I forgot just a small thing - to add the equality case while dividing a row into the strips. That was all. Now pre- and post-compression results are perfectly identical. Will show some remade examples later.

:)  :)

And now comes the dithering I suppose :) .. although as a first step you can just add +2 on the 16 bit values to remove the truncation effect  8)

Marsblessed

@IliasG
No, not so fast. Nothing really changed after the debugging. The black level is far from the correct one. There is only one way left - to dig into CR2HDR sources... but I am ready for it now! :)

here are two examples (dual vs forged with post):


The CR2-files are here
Canon EOS 650D with ML

a1ex

Make sure you use the 20-bit branch; I've just fixed a black level problem for 650D a few days ago.

Also, do try adding a constant offset to all raw pixels and observe the effect.

Marsblessed

@A1ex
ok, I will try.
By the way, what would you say if I try to make a special version of CR2HDR based on 20-bit branch? I think it could be much much easier to me to adopt your solution (by simply adding few more modules)  than implement a lot of the same things in my own way (using your code as an example).
Canon EOS 650D with ML

a1ex

Well, a option like --output-cr2 should be enough, right?

Marsblessed

yep, this way it may work too! :)
Canon EOS 650D with ML

Stevenbaril

Hey all. I'm VERY illiterate when it comes to Terminal and commands, scripts etc. I've been trying to figure out how to post process my MLV files. Thus far I am unsuccessful, have been doing a lot of reading, research, trial and error. Nothing will work for me. I have mlv_dump in a folder on my desktop along with my original .mlv files I need to process. I've tried the commands such as

cd desktop
cd mlv
./mlv_dump --dng M01-0705.MLV

this does NOTHING for me, only gives me an error message

" `_` is not recognized as an internal or external command, operable program or batch file"

I'm at my wits end over this, just want to understand what EXACTLY I need to do in order to successfully process these .mlv files. Any suggestions or hints would be GREATLY APPRECIATED!!!!


Marsblessed

A1ex, IliasG,
I have made a strange discovery. The data in the result of CR2HDR contains a LOT of zeros and 0xFFFF's. That is why a simple addition always lead to the corrupted data errors! Neither can I subtract something from 0 nor can I add something to 0xFFFF! Is it correct that COERCE-macro was not applied to the data using the black to white level's limits?

ADDITION:
I have also found a lot of zeros in the original CR2-data (at least in the data that was decompressed by the dcraw). There are lots of extremely high values too (far beyond white level). It looks to me now that it is a kind of normal situation. Thus, the question is how I should deal with all these extreme values.
Canon EOS 650D with ML

a1ex


Marsblessed

As an example take a look at the CR2-files from the above link. Do not look at the forged one. Just use the original CR2-file which has a lot of zeros itself. Then check the result of CR2HDR (20 bits one). There will be a lot of zeros too.
Canon EOS 650D with ML

a1ex

Could not find any zeros (except for the first two top OB lines, which are gibberish anyway).

Marsblessed

I can see zeros (and the values up to 0x0007) in the first four two lines - up to byte offset of 0x5280. However, 0x0008 values are almost everywhere. Which means not exactly zeros but near zeros values which in any way much lower than the black level. - it is a mistake. I will try to build a distribution of low and high values for better analyzing.
Canon EOS 650D with ML

a1ex

The darkest pixel, ignoring the first 4 lines, is 1584 in input and 8090 in output.

The black levels are 2046 for input (autodetected by cr2hdr) and 8184 for output (exiftool). There are 3 cold pixels found by cr2hdr.

Marsblessed

Finally, I have got enough time to do some experiments with CR2HDR. There was a lot of struggle with hg but I mastered the skill and it is not an obstacle anymore (though, bitbucket still has not wanted to authorize me in order to push my changes into ml).

So far the results of my experiments are not that encouraging. I wish I could have a universal solution to the problem but right now it seems that such a solution does not exist. As a case in point, here are a couple of photos.

(dual vs forged with some post)



the CR2-files are here

At first glance my results look perfect. The problem is the fact that I had had to add different offsets to the values of the original DNG-files in order to achieve adequate black levels. Moreover, as it could be seen on the lower photos, the resulting black level is not perfect at all - the radiator does look more pink than it should be and if I decrease the offset for just 1, the radiator will look ok but the floor will become really green.

The actual offsets were 4 for the top picture and 7 for the bottom plus the difference between the CR2HDR BL and Canon BL for each channel.

A1ex, there is also an issue with CR2HDR which could be seen on the bottom pictures - the suitcase (which is barely visible in the corner of the room) lost its color completely. It seems to me that it could be improved.
Canon EOS 650D with ML

Audionut

Quote from: Marsblessed on April 20, 2014, 09:58:42 PM
(though, bitbucket still has not wanted to authorize me in order to push my changes into ml).

It would be a little silly, if anyone and everyone, could push code changes to ML, don't you think.

http://www.magiclantern.fm/forum/index.php?topic=10774.msg104838#msg104838