Messages

Do you mean the noise pattern of the left one of the two pictures? If so, then it seems that you regard it as a result of cr2hdr but it is not. It is the result of direct conversion of the original dual-CR2-file made by DxO. Due to reduction in size, the difference between dark and bright stripes has almost disappeared, although it is clearly visible (for example, on the wall).

I don't know.  I cannot read minds, that's probably why I linked it 
Your statement sounded like you were trying to push your changes directly to the ML repository.  As opposed to forking, and creating a pull-request.

You don't need ssh keys.

Thank you. Actually, you were right - I was trying to push changes directly from my local rep-clone into the main ML-rep. Reviewing the documentation one more time helped me to find the appropriate way of doing things.

It would be a little silly, if anyone and everyone, could push code changes to ML, don't you think.

http://www.magiclantern.fm/forum/index.php?topic=10774.msg104838#msg104838

Do you really think I have not read it?
You see, in order to create a pull request, I have to push my changesets into my bitbucket's account and that is exactly the point where I got "Authorization failed" despite using correct bitbucket login. It looks to me that I am forced to use ssh-keys instead of simple password and that "technique" I have not tried yet.

[A1ex]

Finally, I have got enough time to do some experiments with CR2HDR. There was a lot of struggle with hg but I mastered the skill and it is not an obstacle anymore (though, bitbucket still has not wanted to authorize me in order to push my changes into ml).

So far the results of my experiments are not that encouraging. I wish I could have a universal solution to the problem but right now it seems that such a solution does not exist. As a case in point, here are a couple of photos.

(dual vs forged with some post)



the CR2-files are here

At first glance my results look perfect. The problem is the fact that I had had to add different offsets to the values of the original DNG-files in order to achieve adequate black levels. Moreover, as it could be seen on the lower photos, the resulting black level is not perfect at all - the radiator does look more pink than it should be and if I decrease the offset for just 1, the radiator will look ok but the floor will become really green.

The actual offsets were 4 for the top picture and 7 for the bottom plus the difference between the CR2HDR BL and Canon BL for each channel.

A1ex, there is also an issue with CR2HDR which could be seen on the bottom pictures - the suitcase (which is barely visible in the corner of the room) lost its color completely. It seems to me that it could be improved.

I can see zeros (and the values up to 0x0007) in the first four two lines - up to byte offset of 0x5280. However, 0x0008 values are almost everywhere. Which means not exactly zeros but near zeros values which in any way much lower than the black level. - it is a mistake. I will try to build a distribution of low and high values for better analyzing.

As an example take a look at the CR2-files from the above link. Do not look at the forged one. Just use the original CR2-file which has a lot of zeros itself. Then check the result of CR2HDR (20 bits one). There will be a lot of zeros too.

[A1ex]

A1ex, IliasG,
I have made a strange discovery. The data in the result of CR2HDR contains a LOT of zeros and 0xFFFF's. That is why a simple addition always lead to the corrupted data errors! Neither can I subtract something from 0 nor can I add something to 0xFFFF! Is it correct that COERCE-macro was not applied to the data using the black to white level's limits?

ADDITION:
I have also found a lot of zeros in the original CR2-data (at least in the data that was decompressed by the dcraw). There are lots of extremely high values too (far beyond white level). It looks to me now that it is a kind of normal situation. Thus, the question is how I should deal with all these extreme values.

yep, this way it may work too!

[A1ex]

@A1ex
ok, I will try.
By the way, what would you say if I try to make a special version of CR2HDR based on 20-bit branch? I think it could be much much easier to me to adopt your solution (by simply adding few more modules)  than implement a lot of the same things in my own way (using your code as an example).

[IliasG]

@IliasG
No, not so fast. Nothing really changed after the debugging. The black level is far from the correct one. There is only one way left - to dig into CR2HDR sources... but I am ready for it now!

here are two examples (dual vs forged with post):


The CR2-files are here

[whyukon]

@whyukon
I am glad there is not just a couple of us who needs it to be in the Canon CR2 format.

or through a reasonably simple post-process

well, at this point it is as simple as the process with cr2hdr - just another console executable file to work with (and yes, in the end it could be integrated into a special version of cr2hdr but probably not by me)

[IliasG]

@ IliasG
Finally! I have found the bug! It was not the compression (which is not done by the libjpeg by the way). It was the division into a couple of strips which made the dirty thing in the data (which was strange ignorance of some values during processing). In the end it turns out that I forgot just a small thing - to add the equality case while dividing a row into the strips. That was all. Now pre- and post-compression results are perfectly identical. Will show some remade examples later.

I will check it using the dcraw decompressor

[IliasG]

@ IliasG
at this time nothing has been done to the dng-values excluding the normalisation. As I stated before the code is simple:

Code Select Expand


      val = ((val - 7964) * 40920) / 42140;
      mcu[j] = (ushort) (2047 + (val >> 2));
If you spotted the difference then that could be the difference of the cr2hdr versions which are used by both of us.
Mine is:
d0ac769 on 2014-01-23 10:13:39

is there a way to convert Dual ISO file in normal Canon Cr2 file? Can somebody write a converter that will convert dual ISO files in ordinary Cr2 file?

That is exactly what I am working on. I have made a version of the converter, which I should tune for each dng-file separately. Though, I still have some problems to be solved. The most important one is how the values in a DNG correlates to those in a CR2. That is the question to A1ex but as far as I see, nor had he time neither had he desire to explain his conversion in details. On the other hand, I still have not had time to investigate it in the sources of the cr2hdr which are far too heavy to just sit and have it done. It is almost miracle that I have made such progress - I HAVE the cr2-file in the end which could be used in any supporting software but without the proper transformation it could result in an improper picture. However, as you can see, the differences are barely visible (at least in the examples above). Therefore, there is only one thing for you to do - wait for my final version which will appear sooner or later. Or you could go along my way in order to create your own tool.

May I have some details, A1ex?

1) Are you sure that it is a correct Guassian distribution after being multiplied by another stdev? It is impossible that G(stdev=0.5) is "equal" to 0.5*G(stdev=1). At least according to this definition.

2) as far as I understand, it has nothing to do with the mentioned "Random rounding - random dither", which means it does not take into account the reminder (which is 00, 01, 10 or 11 in my case). Is my assumption correct?

You shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..

By the way we have a chance to do that!
normal vs dual without any post:


normal vs dual with some post (same settings):


DxO has done almost magic! Black areas of the normal shot are just a little more brown than the same areas of the dual. Though, it could be my monitor which makes the difference barely visible.
the CR2-files are here

[@IliasG]

@IliasG
Why should I always prove that my understanding is correct? Though, I try to be patient. As everyone could see at the wiki-page, the definition of a color space is:
a color space is a color model with a certain mapping function between the color model and a certain reference color space where a color model is an abstract mathematical model describing the way colors can be represented as tuples of numbers.
Hence, my reference to a color space in the context of the trouble is perfectly justified. We have one color space for CR2-files and another color space for DNG-files. They ARE color spaces by the definition - they have both mathematical representations of color components AND mapping functions to an absolute color space - the sRGB. Therefore, when I found out that simple SHR by 2 is not a proper transformation between the two, I asked someone to help me to define the correct transformation. Why then A1ex started to point me out that the color space is one and the same? No, IliasG and A1ex, they are not. CR2HDR transforms one color space of CR2-files into another color space which by the definition will never be the same color space and the reason for that is the fact that the numbers in the resulting DNG which represent certain colors are different from those in the original CR2-file. Does it sound convincing now?

Of course the normalization of all the range is better than the offset but you have to know the values that DxO uses as BL & WL ..

I have checked WL&BL for this file only - they are exactly the values which are defined by Canon in Exif. Any attempt to exceed the WL by applying different ranges to multiply after normalization results in a corrupted data error in the DxO. Though, I still have another couple of files where the WL is different - it is 15092 (at ISO200) instead of 12277 (ISO100) but I will be very surprised if the DxO have another WL (no meter why).

Is the ..forged.jpeg sample from the forged.cr2 that you uploaded or from a later normalized forged.cr2 ?. Looks good to me ..

No, it is a new file with the same name - my tool uses this name-template for the resulting files! Here is the new one if you need it.

Can you describe how exactly do you normalize the values ?

Easily but using C and do not forget that it was just a test - all the values are hard-coded for the particular file. Take a look:

Code Select Expand


      val = ((val - 7964) * 40920) / 42140;
      mcu[j] = (ushort) (2047 + (val >> 2));
where the "val" is the original value from a DNG-data and "mcu[j]" is the resulting value to be compressed and saved in the forged CR2-file. 42140 is the subtraction of BL from WL for DNG's WL&BL, 40920 is the same for CR2 but multiplied by 4, and 7964 and 2047 are the BLs. As you can also see, I did not do any special roundings but that will be done in the final version (sooner or later).

You shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..

Ok, it was just a thought.

Can you upload in www.filebin.net the dual3185.cr2 and dual3185.forged.cr2 ?.

here they are

It's the BlackLevel you should care about as WL looks more or less correct ...

As it appears to me now, the problem is not just the black level. You see, normalized color space is one and the same for both DNG and CR2 files - it is sRGB. However, the problem is that in CR2 file the range of absolute values in 14 bps form is narrower than the one from dng when SHR'ed by 2, which is the reason why a simple addition of 139 to each dng-value corrupts the data in CR2 - the highest values translates into the values which are beyond cr2-white level (predefined or calculated by DxO). That is my assumption. The effect is visible on the JPGs above - even the clouds are slightly darker than in the original - the resulting rgb colors were shifted down a little.

In fact, in order to achieve the correct result, I have to convert the dng-values using normalization. That is why I need to know the correct black and white levels for both files.

By the way, I have seen in the other threads that A1ex mentioned that he wrote an automatic white level detector which determine the unique white level for each individual CR2-file. It seems to me that I will have to use his procedure to do the same job or I will never find the proper backward transformation for a particular CR2-file.


UPDATE:
I was right about normalization. Test confirmed my assumption. Here is the result:
origin

normalized forged


it was done by applying ranges built on black and white levels from exif. There is no green tint anymore, though the overall brightness is lower (do not know why).

[@IliasG]

@IliasG
Google search on LinearityUpperMargin shows me that you have just discussed all these parameters with A1ex in another thread of the forum. However, I have not find the conclusion yet. I mean the way of how it should be done (let alone the fact that you were talking there about cr2 into dng conversion).

By the way, DNG-result of cr2hdr contains the original value for the parameter in Exif data - 10000, although the data is 16 bps and this parameter should be at least 40000.

I have finally managed to show a picture here in a small size. So here is my original problem as I got it:

One question though:
which white level should I use from CR2 - SpecularWhiteLevel or NormalWhiteLevel (according to this description)?

It is even more complicated. I can not just add something - the result is always corrupted data. As far as I see it now, I have to extract two ranges (black and white levels) from dng and cr2 and then compress or expand the dng-range into (multiplied by 4) cr2-range. After that I have to "SHR by 2" (using some rounding mechanism) and add cr2-black_level to the result before put it into cr2-file. I will test it later.

[A1ex]

I think I understand now. I have to compress 16 bps raw values back to the range between 2048*4 and 2^16-1 and only after that I should apply SHR by 2. A1ex, please, correct me if I am wrong.

well, but notice that the sky is darker too despite the fact that it is done by interpolating between blue and white!

Besides, I have tried to add 2048 to every raw value of the picture - the result is corrupted data (can not be opend in the DxO)

At the same time, it means that I have to apply some more sophisticated logic to 16 bps raw values instead of just SHR by 2. (Again, that means that the color spaces are not identical)