Messages

1

General Chat / Re: Lens effects on photons

« on: November 04, 2014, 03:02:26 AM »

Focal length (and thus f-ratio) has absolutely no effect on the number of photons collected and delivered.

Lets say you have a FF sensor and 70-200/f2.8

You shoot at a small target in the sky (say one that you would need 1000mm to cover all the frame) ..
Shoot at 100mm 1sec f/2.8 then 200mm 1sec f/2.8 .. by the summary of
   1. Object flux (photons/second/square-meter)
   2. Aperture size (square-meters) and efficiency
   3. Exposure time

You will have recorded 4X more photons with 200mm f/2.8 because the "2. aperture size (sq-meters)" will be 4X larger 

2

Reverse Engineering / Re: CMOS/ADTG/Digic register investigation on ISO

« on: July 29, 2014, 05:33:47 PM »

And that level of luminance is proportional to the number of photons.

I'm not sure where you are trying to get.

Forgive me .. while my eyes were reading "photons" my brain translated it to "photoelectrons" ..

My point is that QE cannot affect the full saturation level .. 

I will reread the thread and come back if I have something useful to say ..

3

Reverse Engineering / Re: CMOS/ADTG/Digic register investigation on ISO

« on: July 29, 2014, 02:03:18 PM »

Alex, as I understand it, DxO does not measure the amplification but the output sensitivity. i.e. for an A level of illuminance how high in the 0-1 scale (0=black - 1=full saturation) is the output in raw levels and compares it to the nominal value.

Say for the same illumination
5D4 has QE=50% and saturation at 10000 electrons/square micron. Gives as output 0.5 in the 0-1 range (white clipping level - Black level) which corresponds to 5000 electrons
6D2 has QE=55% and saturation at 11000 electrons/ square micron. Gives as output 0.5 in the 0-1 range which corresponds to 5500 electrons

the two cameras will have the same dxo-measured ISO

4

Reverse Engineering / Re: CMOS/ADTG/Digic register investigation on ISO

« on: July 29, 2014, 10:59:15 AM »

....
- 6D's sensor has a higher quantum efficiency
...
Theory:
- photon to electron ratio = quantum efficiency (assumming it's a sensor constant)
- equal ISOs = equal number of photons per area (light required to saturate the sensor)
...

I think that equal ISOs = equal relative saturation (electrons counted/full saturation level ).

At least, Dxo's measures are using this 
http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity
"... relates sensitivity to the exposure necessary to saturate the camera"

Higher QE gives higher "Dxo measured ISOs" for the same electron capacity ..

5

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 27, 2014, 11:51:17 PM »

@ IliasG
Finally! I have found the bug! It was not the compression (which is not done by the libjpeg by the way). It was the division into a couple of strips which made the dirty thing in the data (which was strange ignorance of some values during processing). In the end it turns out that I forgot just a small thing - to add the equality case while dividing a row into the strips. That was all. Now pre- and post-compression results are perfectly identical. Will show some remade examples later.

  

And now comes the dithering I suppose .. although as a first step you can just add +2 on the 16 bit values to remove the truncation effect 

6

Modules Development / Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)

« on: March 26, 2014, 09:30:31 PM »

I  am on win-vista32 bit, is your result on linux ?. did you checked the dng that I uploaded ?.

7

Modules Development / Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)

« on: March 26, 2014, 09:00:44 PM »

Could not reproduce this one, can you show a crop where I should look?

The crop, used exe , and two dngs one with the guilty cr2hdr-20bit (did not try the last one) and one with the official cr2hdr from the first post (no problem there).
Edit: black pixels exist with the last version also, not sure if they are the same ..

http://filebin.net/3379r86b43

http://filebin.net/3379r86b43/dual3185-patched.dng-crop00-738620-300.2.png
crop is 0,0 - 738,620 .. there are at least four black pixels visible in this crop ..

With the last build ..
- the crappy line at the bottomn disappeared
- (after excluding the 2 top lines) there are 122-125-126-130 black pixels in R-G1-B-G2 same as with the previous 20bit versions

8

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 23, 2014, 12:44:09 AM »

If you spotted the difference then that could be the difference of the cr2hdr versions which are used by both of us.
Mine is:
d0ac769 on 2014-01-23 10:13:39

Mine was the same and there are no first column bright pixels in the DNGs produced by this version. Maybe the libjpeg compression makes them ..

9

Modules Development / Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)

« on: March 23, 2014, 12:39:58 AM »

Updated binary: cr2hdr-20bit.exe

Did not run any tests, so watch out for regressions.

edit: I think I've just created a regression, bug #1 in stevefal's sample.
edit2: fixed and reuploaded.

These last versions give many totally dark pixels (I mean absolute 0 without subtracting BL) spread over the frame and the OB areas,
- and the third line up from the bottom is corrupted with most pixels again at level 0 and some at 65535 ..

Sample is http://filebin.net/qlh1lwzjwt/DUAL3185.CR2

10

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 22, 2014, 01:43:58 PM »

@ Marsblessed

have you corrected the first column values ?. In your forged.CR2s the green pixels in the first column of the optically black area are not divided by 4 related to the same pixels in the dual.DNGs. It is very possible this is the reason for wrong black level's calculation by DxO.

11

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 22, 2014, 01:34:51 PM »

@ Marsblessed and A1ex

As I understand it .. (correct me if I am wrong)

When on a homogenous sampling the values of a sample are multiplied we have avg=16 stdev=4 then on the 0.25X multiplied values we have avg= 4 stdev=1

I think gaussian stdevs are added in quadrature i.e. a stdev2 0.5 upon a stdev1 0.5 is sqrt(stdev1^2+stdev2^2) = 0.71

Now if the stdev in 20bit is 16 then after converting to 16bit the stdev will be 1.0 .. less than 1.3 (John Sheehy's safe limit) so there is a danger for posterization ..
we have to add in quadrature gaussian noise of stdev=13 so that we take total stdev=21 in 20bits and so 21/16 = 1.3 in 16bits

In case we use either gaussian noise with stdev 0.5*div_factor or uniform noise -0.5 to +0.5*div_factor we don't need the trick to add 0.5*div_factor to correct the bias that a simple shift will insert in the result,

The difference in quantization error after pure shift vs dithering or (maybe) the 0.5*div_factor offset is what is described as truncating model vs rounding model
http://en.wikipedia.org/wiki/Quantization_%28signal_processing%29#Quantization_error_models

12

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 22, 2014, 02:28:40 AM »

I would say that stdev=0.5 is better to my eyes but is there a real value to find which dithering is better on noiseless values ?.

Is this experiment to find an optimal 20to16 bit conversion in cr2hdr-20bit ?

I recall John sheehy writing many times in Dpreview forums that according to his experiments if the stdev in the exported raw file is more than 1 ADU (I think he says around 1.3 ADUs is optimal) there is no danger for posterization even under heavy processing.

BTW nice improvement with the latest cr2hdr-20bit !!.

13

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 15, 2014, 06:05:13 PM »

Looks like the "first column error" remains although it is faded down (by the normalization step ?).

In 3187.forged.cr2 I measure in the first two columns. 1st column Ravg=2045.8 G2avg=2247.1 - 2nd col. G1avg=2045.7 Bavg=2046.2
Averaging all 0-72 columns of the left OB area we take R=2046.3, G1=2046.4, B=2046.9, G2=2051.5 (this +5 can make the slight difference at the darks we see )
Averaging 2-70 we take a correct measure R=2046.3, G1=2045.9, B=2046.2, G2=2045.9

This gets more significant in the not normalized forged cr2s.

It was much more before with 3185 (1988 vs 3740 in forged, 2044 vs 3748 in normed.forged)
http://www.magiclantern.fm/forum/index.php?topic=10895.msg106484#msg106484

Does the rendering remains buggy on a forged.cr2 without the "first column error" if you don't apply the normalization step ?

The jpegs looks almost same with slightly darker-greener darks in the forget.jpeg.  Nice !. I think it will be better after correcting the first column ..

14

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 15, 2014, 04:29:02 AM »

No just the above samples where you also saw in the dual3174.DNG has a bit green tint at the darks. The BL in DNG tag is 1991/14bit while dcraw calculates 1988 from the side OB ..

15

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 15, 2014, 02:27:45 AM »

@Alex,

Do the cr2hdr calculate the BL from the top OB area ?.. The usual calculation area is the left OB and there the avg is 7852-7956 while the BL (read in the DNG's exif) is 7964 and matches the avg of the top OB area (2 first lines excepted ..)

The rendering at the darks looks better with BL calculated from the side OB area I think, or perhaps the avg of both OB areas is better .. didn't pixel peeped much

16

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 15, 2014, 02:10:49 AM »

I think you'll not need normalization any more ..

Inspecting the jpeg samples .. both render the green tree as almost black ..

Inspecting the raw data in both .. forged.cr2 and ..normed.forged.cr2 you have wrong green values in the first column (column 0, channel G2), it's average is around double than the other pixel's avg in the optically black area. This could give a wrong BL if DxO includes this column in BL calculations and calculates a pure average. Dcraw and derivatives don't use the first 2 or 4 columns (nor the last ..) so their calculation is correct.

17

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 15, 2014, 12:30:03 AM »

Not convinced  .. you need a big stretch of definitions to say there is a color space encapsulating raw data. No primaries but spectral responses, no gamut, no way to take color fidelity by linear transformations (see the good raw input color profiles use types of non linear ( not mathematically defined ) mappings ), 
http://www.dpreview.com/forums/post/53023856 and other posts of Iliah on the subject

Thanks for the sample and the normalization description .. I'll have to download a demo Dxo to try the difference because with RT the result is the same

So the -Canon:xx tags are not used (as nothing changes by changing them) but somehow are the correct values to use with DxO !!.

18

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 14, 2014, 05:07:14 PM »

Please, don't mixup things. Raw files have no color space. There is no color space involved nor any relation to color space in the calculations of cr2hdr nor in your division by 4 of the DNG raw values nor in setting the black and white levels. In fact the BL & WL are the clipping points for a converter .. every value lower than BL is assigned to the BL and every value higher than the WL is assigned as equal to WL. In a 0-1 range any RawValue <=BL becomes 0 and any >= WL becomes 1 ..

By adding an offset of 139/14bit you take a very small increase in clipped highlights. The WL in the dualDNG is 50104 and should be in the forged.cr2 50104/4 =  12526. This after subtracting the BL (7964/4 = 1991) becomes 12526-1991= 10535. Adding 139 = 10674 .. You loose log2(10535) - log2(10674) = 0.02 stops .. it's negligible ..

Of course the normalization of all the range is better than the offset but you have to know the values that DxO uses as BL & WL ..

Is the ..forged.jpeg sample from the forged.cr2 that you uploaded or from a later normalized forged.cr2 ?. Looks good to me ..
Can you describe how exactly do you normalize the values ?

You shouldn't compare the brightness of the forged.cr2 with the dual.cr2 but with a normal (single ISO) cr2 shot with the same settings and lighting ..

19

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 13, 2014, 12:55:52 PM »

@Marsblessed,

Can you upload in www.filebin.net the dual3185.cr2 and dual3185.forged.cr2 ?.

We have no knowledge about these -Canon tags as they are not used by ufraw/dcraw/rawthwrapee. If you want more info you could ask Iliah Borg/Alex Tutubalin at www.rawdigger.com. It's the Rawdigger team that discovered and decoded them.

The cr2hdr process don't change the -Canon:xxx data tags. It only calculates the 16 bit black/white levels and writes them at -exif:Blacklevel   -exif:WhiteLevel and this is enough for most converters. 
Now if you will understand what is used by DXO regarding Black/White levels when converting *.CR2s ( -Canon:xxx tags or some kind of calculation or a predefined default value ..)  then you will be able to translate the exif:BlackLevel and exif:WhiteLevel to the needed by DxO info by either updating the -Canon.xxx tags (if they are used by DxO) or offsetting/scaling the raw data to match with DxO's calculations or predefined values ..

It's the BlackLevel you should care about as WL looks more or less correct ..

20

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 12, 2014, 08:49:11 PM »

I would bet on the specular .. this is closer to the peak white level one can see in the raw histograms. But one has to test first. Canon's DPP use all of them ..

Have you checked if DxO uses these tags at all .. by making extreme changes there an see the effect (if any) on DxO's conversion ?.

Related to the offset on values, now that you checked that by offsetting the values by the difference 2048-dng_Black_Level is a bad idea, you could try scaling by 2048/dng_Black_Level .. sounds also logical .

The suspicion that DxO uses a default value (i.e. 2048) as "black level" was purely a guess based on that you wrote that nothing happens by changing the -Canon tags ..

If you can work with Rawtherapee you can change the black level used by RT in tab raw - black levels to see  when RT's conversion matches DxO's. There i put +139.0 and took a dark greenish result like you described it ..     

21

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 11, 2014, 11:24:43 PM »

I think I understand now. I have to compress 16 bps raw values back to the range between 2048*4 and 2^16-1 and only after that I should apply SHR by 2. A1ex, please, correct me if I am wrong.

As I understand it, you have to read from DNG exif the black level value (it was 7964 in dual3174.dng and become 7964/4 = 1991 in forged.cr2) then add  2048-1911=139 to all the values of forged.cr2 to normalize for the suspected DxO's default BL for 650D .

22

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 11, 2014, 04:17:46 PM »

Well, I would not call the DxO software a converter. It is rather a simplified version of Photoshop, which can do some post. The problem is it can not open DNG-files at all! If it were not the case I would not need to do anything with a CR2-files.

We are researching the raw conversion part here only, I think ..

Tried this file - it looks even darker now but with the same green tint everywhere. Which means no, the black level correction does nothing. Again, the problem is not only in the dark areas. Every point in the compressed result of cr2hdr has more green than it should be. As A1lex stated, it is even seen in the UFRaw (which has nothing in common with DxO).

So if it's darker this means that DxO uses these tags that I changed ..

Look what exactly Alex wrote below .. the DNG is greener than your .forged.cr2, same as with Rawtherapee. And the reason is that the Black level tag in the DNG is higher that the calculated black level (7964/4 = 1991 vs 1988 calculated) ..

... The output from cr2hdr DUAL3174.CR2 has greener shadows than DUAL3174.forged.cr2, at least in ufraw.

Please upload some converted samples ..

23

HDR and Dual ISO Postprocessing / Re: Dual ISO: back into CR2

« on: March 11, 2014, 01:09:14 PM »

Hi everyone!

If you ever been an owner of DxO Software, you would agree with me that there is no way of doing anything in the software with dual_iso-dng. That is a real problem especially when you do not want to have such a heavy peace of software as Adobe Photoshop is. Thus, there is only one option after making dual_iso shots: BACK into cr2!

I don't have DxO converter but
What does DxO displays when opening dualISO dngs ?.

UFRaw is based on the dcraw.c which shows the same Filter pattern for the resulting DNG-file as it does for the original CR2-file. That is why I supposed that the data in the DNG-file is completely identical to CR2-file except bps and compression. May be this is the key to green color (which appears not only in deep shadows as it could be seen in DUAL3175.forged.cr2)

As Alex wrote, the usual reason for dark greenish picture is wrong black Level.

For CR2s, Dcraw and derivatives calculate it from the side optically black area and the picture comes out correct (at least with RawTherapee that I tried). Looks like DxO does not use this method. It either has a hardcoded value as BlackLevel or reads it from exif  -Canon data where the info is invalid after cr2hdr conversions.

For DNGs, Dcraw and derivatives read the BL and WL values from exif (not exif.Canon ) ..and it is again displayed correctly.

To check if DxO uses the -Canon data for BL/WL try the corrected dual DNG and ..forged.cr2 . I copied the values from DNG's exif data to -Canon data with exiftool.
http://filebin.net/6lagewyuhl/DUAL3174-BLWLcorrected.DNG
http://filebin.net/6lagewyuhl/DUAL3174.forged-BLcorrected.cr2

24

Modules Development / Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)

« on: February 20, 2014, 09:57:16 PM »

Thanks for the link A1ex, very interesting !!.

I have to clarify that what I wrote is about using mixted ISO shots and the properties of the data in the DualISO CR2s. The final result (the DualISO DNGs) additionally include some kind of software denoise and some excellent algos for Black Level normalization + line/banding elimination  + .. so a part of DR is regained

I really hope that we could use the same on normal CR2s ..

25

Modules Development / Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)

« on: February 20, 2014, 09:18:23 PM »

I thing that all these calculations on DR increase by using Dual-ISO are overestimating the real world visual effect because they are based on per pixel data while the correct is to normalize for displayed at equal size and equal apparent detail. It's "screen" vs "print" DR in DxO's terms .. we use print DR in our (based on DxO data) comparisons don't we ?).

As soon as at the dark areas only half the pixels are used then the print-DR gain is 0.5 stops less than if all pixels were used.

So in the case of 5D3 if the per pixel "screen-DR" gain of dual ISO 100/3200 is 3.5 stops the "print-Dr" gain will be 3.0 stops.