Messages

On forums answers sometimes go out... 
I have this problem with ALL Dngs...
Try the original from Bolex site:

http://www.digitalbolex.com/wp-content/uploads/2013/07/BLX00001-20110317-0000436-15mm-Elitar.dng

What happen when you import in AE or DaVinci?

Once again, you need to make a custom profile, buy http://www.amazon.com/X-Rite-MSCCPP-ColorChecker-Passport/dp/B002NU5UW8
give it a shot under daylight and build your profile. Or upload the colorchecker shot if you need help ..

If the above is impossible you have to change the colormatrix in exif. As it is now seems it's in a wrong format for DNG specs but works OK as camera to sRGB conversion. Unfortunately I don't know how to translate it to DNG format ..

This is how it looks in RT after copying the colomatrix multipliers in channel mixer ..

First of all, wow, wow, wow!

A few questions. I may have missed this in reading through the doc, so apologies if so. Is it correct that when in this mode, that highlights will always be half-res regardless of the DR of the scene? If this is so, it sounds like we need to be careful when shooting a flat scene (low DR) and exposing to the right. You could end up with a substantial portion of the frame in half-res.

And if that's true, is it possible to perform the dual-ISO trick only as much as the DR of the frame calls for? For lower-DR scenes it seems like maintaining highlight fidelity should be the first priority. As DR increases, I'd want to sacrifice shadow resolution first, and then highlight resolution last. Or maybe the other way. Or maybe symmetrically.

Also, is there something special about 4 stops? I mean how about the option to run 100/6400 for a full 16 stops, understanding that the extra 2 bits will also be at half-res?.

I can't wait to start shooting with this. The few videos I've seen posted so far are like miracles. Soon we'll all be lifting shadows bravely and chuckling over "noise reduction". Which reminds me, care to SHARPEN? Don't mind if I do.

I hope the below data will answer your questions. 

Based on DXO screen-DR measures

Code Select Expand


5DIII
       screen  per ISO  Total Gain
  ISO    DR     Gain    from base ISO
0100   10.97    0.00    0.00
0200   10.87    0.90    0.90
0400   10.69    0.82    1.72
0800   10.41    0.72    2.44
1600    9.94    0.53    2.97
3200    9.23    0.29    3.26
6400    8.30    0.07    3.33
12600    7.48    0.18    3.51

7D
       screen  per ISO  Total Gain
  ISO    DR     Gain    from base ISO
0100   11.12    0.00    0.00
0200   11.08    0.96    0.96
0400   10.76    0.68    1.64
0800   10.10    0.34    1.98
1600    9.02   -0.08    1.90
3200    8.26    0.24    2.14
6400    7.09   -0.17    1.97
12600    6.22    0.13    2.10   


As we can see there is no reason to go over ISO 800 for 7D or ISO 3200 for 5DIII. Because we gain negligible DR while we loose a 1 stop range of the "perfect midtones" (where all pixels are used so no resolution loss no artifacts) to the problematic hi and low ranges where only half the pixels are in use.

IMHO for 5DIII I would not sacrifice the range of 1 stop of full sensor use for 1/3 stop DR increase at ISO 3200 not even for the 1/2 stop gain from ISO 1600.

Lets look at it from another side .. for example the 100-1600 case .. the calculated 3 stop gain is valid if we use all the pixels.
In theory, by using only half of the sensor at darks we should have 0.5stop more noise there than a full 1600 ISO shot, so the net gain should be 2.5 stops.

What we see in the samples is not only the better sensor behavior at ISO1600 but also a kind of simplistic denoise (by interpolation - averaging) and the excellent holistic approach of ML team regarding black point regulation, channel imbalance and FPN elimination.

It will be interesting to see the differences between using 100-800 vs 100-1600 ranges with 5DIII ... especially after using a mild denoise in ACR or Rawtherapee ..

I tried to do the conversion in exif data but I had limited success. While the colors seem more correct, the t-shirt turns from green tone to blue tone but the colors are very saturated. Something is missing from my method ...

If you can make a shot of colorchecker passport under sunlight then we can make a dcp profile and copy the color matrix in the exif data. You will need to batch convert the exifs of your files with exiftool - exiftoolgui.

If you cannot provide a colorchecker passport shot then we can ask from color experts to provide instructions .. I am thinking of color management category at Luminous Landscape forums ..

[Magical]

Magical !!!!!!

Although the bad side effects (low resolution moire aliasing) are serious defects.

Just thinking loudly ..
What if you try with alternating ISO line by line and use no interpolation just scale the not-burned hiISO pixels by low/hi (zero point should be the Black level). Then a good highlight recovery algo can make the job it knows best ...
This could decrease the DR expansion a bit (1 stop ??) but I think denoise algos can make a good job there also .. I am thinking of denoise at raw level like those at the "RAW" tab of RawTherapee

Line by line interlacing will give the opportunity to use hi ISO on the weak line depending on the WB (red for daylight, blue for tungsten) ..   

Really magic !!!!!!

I haven't got my head around how the logarithmic SNR ratio plays into the equation.

It's simple when working in stops (log2). ISO 1600 should have 4 stops less DR than ISO100. DXO measures it at only 1stop lower so the gain is 3 stops.

Judging from the histogram (and others I've been playing with) I posted above, the tonal quality of a single ISO 100 shot falls to crap around -10.5EV.  The histogram of the enfused shots, show all but 100% tonal quality over the entire 16EV.  That's more then 3 stops improvement   I haven't done any testing by eye.

At -10.5EV SNR is 1.0 .. totally crap .. but I would call it simply crap even at -9EV where SNR is around 4.0 ..



Look at how many values have population ..  8500 from 50000 available at best. On 0R0A0640-fullres-soft.DNG I count 12800 from 52000 available.
Raw levels density starts at 1:1 at the darks and goes up to 1:9 at highlights. Looks like a 14bit log file linearized at 16bit

Indeed.  Just like everything in life, there are compromises.  From what I've seen of a1ex's code so far, this little trick doesn't come for free

Can you clarify .. what's the cost as you see it ?.

The same in DaVinci...
Any help?

Something goes wrong in the way the color matrix is written in exif.

If you use no color matrix (with Rawtherapee choose "no profile" at color management) and then you copy the color matrix 1 coefficients (multiplied by 100 so 1.89 becomes 189..) in RT's "channel mixer" you take the colors you describe as "perfect".

In case you have difficulty finding those multipiers here they are ..  as written in exif
1.80477 -0.54011 -0.26466 -0.19744 1.36156 -0.16413 0.21195 -0.83605 1.6241
and after X100 .... 180 -54 -26 -20 136 -16 21 -84 163.

We have to dig in DNG specs to find the correct writing of color matrices and correct it in exif data..

Edit. Looks like the multipliers are at the inverse order .. 163 -84 21 -16 136 -20 -26 -54 180.

3 stops should be the expected increase in DR.

As Alex wrote, 3 stops is expected for 5DIII. It depends on the model .. for 50D using ISOs range 100-1600 DR increase is expected at a bit more than 2 stops.
It also depends on the ISOs used .. On 5DIII ISOs range 200 - 1600 will give 2 stops 100-3200 will give a bit more than 3 stops ..

Easy, with that theory that almost nobody believed: ISO 1600 is less noisy than ISO 100.



Something along these lines: http://www.guillermoluijk.com/article/nonoise/index_en.htm

Well, that theory is pretty established, even Dr Emil Martinec made an experiment-presentation using the same idea you use
http://www.dpreview.com/forums/post/28749589
http://www.dpreview.com/forums/post/28750076

The miraculous is the way you gather all this info from a Canon boby in a single file !!!.

It's the best Canon users could hope .. a clear increase of at least 2-3 stops in DR. And this makes Canons Isoless cameras !!.

Can't keep waiting to learn what is this "small ADTG trick".

[With resolve 9.15 support of 14bit dng's i dont see any need paying more attention to my app.]

Hey there, back from holiday (picture?) So.. where to start. With the important thing: With resolve 9.15 support of 14bit dng's i dont see any need paying more attention to my app. (i'm a bit puzzled, blackmagic did that - thought, they have some more important things to code and to distant their own camera - but finally they get more users with this decision.) if there's no real pro argument, v1.2.1 will stay the last one

@pascal - no infos, no help.

@all - thanks to all, especially them used my tool.

Well come back.

I was expecting you back on July 17 so I didn't upload a pack of LUTs (I added 12bit log also) which are ready. I just want to do some refining and maybe include linearization to 14bit now that Resolve supports it.  If nothing goes wrong I 'll upload in the next 12 hours.

Do the new Resolve support losslessly compressed DNGs as those we take after running through Adobe DNG converter ?.

I think 10bit log (and 12bit) losslesly compressed is a great space saver. Everyone needs it.

ACR can't recognize camera model because chdk-dng.c creates DNG files with name "Canikon" .. You should to change it in the code for example to "Canon EOS 60D" and compile raw2dng.

Or change the "unique camera model" tag from canicon to "your model" using exiftool (it's easy with exitoolgui as frontend) ..

Strange !!!

The effect is the exact inverse from what we know about HTP in photo-raws. Are you sure you named correctly then ON/OFF ??.

Can you write the camera settings you used ?

brilliant. thats a huge step forward. but we have to wake up 1%, a1ex or g3gg0 for implementing the measured data

regards chmee

I didn't read the DNG specs but as I understand it the "As shot neutral" tag contains the as shot multipliers and not the measured ones. If the camera WB setting is at auto those two are the same, but if we have manually set WB there can be a great difference. I would prefer both but if the available space is only enough for one I vote for the measured WB.

i think i've set this two illuminants for the color matrices in my cdng's.. but if we have to recalculate the color matrices  holy boly.. i'm not ready yet, but if its only numbers, we can handle that

No need to recalculate, you can find them ready in exif data of DNG converted CR2s or in Adobe's standard dcp profiles for each model.
In fact Dcraw matrices is a copy of Adobe's D65 matrices.

[raw2cdng v1.1.9 is online]

http://www.phreekz.de/wordpress/2013/06/magiclantern-raw2cdng-cinema-dng/

raw2cdng v1.1.9 is online
- 10bit log is in (but experimental) - no linearizationtable (thx IliasG)
- naming conventions slightly changed
- fixed freezing gui

(!) 10bit log only works for bodies with blacklevel higher 2000 (!) next patch will have a lut for blacklevels around 1024 (!)

@IliasG
is your excel-calc that stable to calculate this lut by changing start values?

(lol. short test in resolve - 10bit/log is not looking that bad and hotpixels are gone in this one sample i have - if the red tint is too much, lower red gain to ~0.86 - but now its time to test the ability to grade and work with them - we squeezed all information into the 10bit - as a side advantage/drawback theres nearly no more data in blownout or dark parts of the picture - i dont really know if 10bit/log is a good grading base - possibly good for directly working in premiere/fcp/avid, after transcoding into a useable codec)

urgent on todoList for next days
- support of splitted files
- 10bit|log for blacklevel ~1024-bodies.

regards chmee

chmee, thanks for your work !!.

Sorry for the late responce, I was offline for a week.

Excel-calc should be stable (not 100% sure about this) for calculating on different black/white levels, only needs to pay attention on the details to have the smallest possible error at Black Level. It's easy (because the rec.709 gamma curve is linear there) by just increasing/decreasing the clipping point by 1-2 steps. I can give a hand on this as I have plenty of free time these days ..

About wrong colors .. don't bother with dirty corrections. Both WB and color matrices need to be applied on linear data or else we take strong color shifts. So we have to wait for the linearization table to be injected in exif data before we can make valuable tests. If this "injection" is a hard work there is a possibility we can surpass this need by a dedicated icc profile and use a converter which can make use of icc profiles (RawTherapee).

Talking about tests .. can you upload some RAW files which include dark and blown areas, made with the latest ML code ?. 

so it seems i have to compute the rec709-log on the fly also. ok. thanks.

regards chmee

Or use predefined "caseA", caseB" ..

A small detail which might help if searching for "perfection" ..
While I started with BL-32, I used BL-31 to keep the BL=2048 at the center when mapping to 10bit (2047, 2048, 2049 all mapped to 10) so as to have the smallest error. With radically different range (WL-BL) this could change.   

right before implementing the 10Bit-log-Curve i'm in doubt, if some bodies have a blacklevel lower than 2017..?!

Someone any info for me?

Yes, models older than  7D (5DII, 50D, 40D) have BL at around 1024. This in photo-raws, video-raws can be a bit different.

In any case for a correct calculation we need the full range limits

Except BL which is computed in camera by ML and accessible in metadata, there is the White Level which is set at a predefined value depending on model and based on photo-raw WL as used in CR2 to DNG conversions. This is not accurate enough. For example metadata for 5DIII say WL=15000 while histograms show 16382 ..

I think that samples for every model with burned areas are necessary to declare the WL based on real video-raw data ..   

Yes, 1024, 1500...

It's computed on the fly; I have no idea what the absolute range is.

FYI: http://lclevy.free.fr/cr2/#wb

I didn't knew a case of BL at 1500 !!.

Alex, as I saw on your 5DIII black frames the BL calculated by ML is truncated to the next lower integer so for values 2047.8-2047.9 it becomes 2047. Not a serious error for 14bit DNG, but after truncating to 12bit (as raw2cdng does for the moment) it can be a bit harmful ..

@IliasG
Just to be on the right track.. Your ConversionLUT 10<->14bit you linked.
on left side is the lin->log, on the right side the inversion log->lin?!

Exactly.
14bit linear (A) to 10bit log (H) is on the left.
Just fill in columnA the clipped values (<2017) all mapped to zero in columnH.

As you can see I forgot to adapt columnA to a variable Black Level .. just used the typical for 5DIII.
Normally I should start from the full 14bit range, clip 31 values lower than Black Level (which is variable) for columnB ...

10bit log (J) to 16bit linear (Q) is on the right. With red color (141) is the 16bit black Level.

It was very dirty calculation sorry.

..
For outliers, median rejects them very well, but it's harder to compute than mean. But yeah, a hot pixel in that area may affect the results.

Do you have any samples with black band and no noise?

Whats the current state with hot/dead pixels in video raws ?. are they corrected in camera ?.

I have some samples from Canon40D photo-raw black frames with black band  .. I can post when I 'll be back home (weekend) but I don't think they are useful.

Running through the raw samples for vertical banding, collected from the forum, I've noticed the white level is 16382 or 16383. The 16383 one may be because some samples were already corrected for banding, and I've reversed the raw data.

If this is true in LiveView for all ISOs, that's interesting. Will check on the other cameras.
edit: on 5D2 it varies between 15850 and 15882.

I didn't check the 4channel, because the conclusion is clear for me: digital ISO has no effect on LiveView raw data.

I can compute 4 black levels, no big deal. Though it's unlikely to make a difference, can you find some examples?

You are correct, whith the up to date samples there is no difference between ISO 160, 200, 250. May be because of a different LV mode vs squig's (now ancient) samples.

Does White Level goes up to 14bit limit with all ISOs ?. If so 15000 is over conservative, you can set it at 16000 leaving 382 levels clipped just in case a setting makes something different.
Hi ISO NR and long exposures comes in my mind which (in photo mode) results in a different clipping .. instead of a single level it becomes a bell distribution and to prevent magenta casts we have to clip at the lower value of this distribution.
Same as Gulliermo's article .. http://translate.google.com/translate?u=http%3A%2F%2Fwww.guillermoluijk.com%2Ftutorial%2Fsatlevel%2Findex.htm&langpair=es|en&hl=EN&ie=UTF-8

But first we have to check the behavior near saturation (magenta casts).   

I also think that we have somehow corrected data. The sad thinks is that this correction can be erroneous .. sometimes in black frames we see a black band with no noise and I think this is exactly erroneous over-correction.

I could find samples with heavy BL imbalance at high ISOs but only for photo mode which is not useful as we see that LV data are different.

PS. When calculating BL sometimes there are ouliers which disturb the result .. if it's possible to remove them the result would be more accurate.

Wait a minute, we were talking about video, not photo. They do not behave in the same way.

My samples: 160  200  250

dcraw -h does not do demosaicing, it just groups a RGGB cell in a RGB pixel.

The difference in squig's files comes from different white levels, because I thought raw in movie mode behaves like in photo mode when you change the ISO.

Hello Alex, thanks for the samples.

As I wrote for Dcraw's -h the interpolation is minimal, just combines the two green channels of a 2X2 square into one. So you loose a little info regarding channel imbalance ..

BTW I think it would be useful for raw2dng and the pattern correction it does if it have available the per channel Black level. Just run 4 loops with N/4 samples instead of 1 with N samples. I think Adobe's converters make use of those four independent values and others will follow .. It can make a difference at high ISOs.

I can change the metadata in squig's samples so this wrong WL was not a problem.

Did you checked Libraw's 4channel code ?.

Here are 6 samples.  ISO's 160, 200 and 250.  Both with lens cap on, and of a white image as close as I could get to saturation @ ISO 200.

.....

I couldn't really understand the results shown by RawDigger and a1ex's observation that these ISO's are the identical raw data.
So I got my test bunny, and again shot @ ISO 200 as close to saturation as possible.  Then with the same shutter/aperture, I shot again @ ISO's 160 and 250.

I don't really use dcraw so my process might be flawed, but to me, it's clear that ISO 250 has the raw data pushed into saturation.  Based on that observation, ISO 160 simply has all of it's data pushed further south into the noise floor.

So the questions for me, am I simply not using the correct process to recover the detail, or has it indeed been pushed into saturation for the digital pushed ISO's, never to be recovered*?  If I am using the correct process, when in the pipeline is the digital manipulation occurring?  Have the digitally pulled ISO's been pulled into the noise floor of the camera?  Or is it done after the effects of noise etc etc in the circuitry of the camera?
...

*While I can understand that it is (was), the same initial raw data, if it's being pushed into saturation with a resulting loss of detail, it's open to interpretation on whether it's still the same raw data from a usability stand point.

Hello Audionut,

As Alex said we need video-raw files not photo raw. Although your samples will be useful to compare histograms between the two modes.

For a start we need video-raws with burned areas (totally saturated) for every ISO to declare the White clipping point. Then we can continue investigating what happens near saturation.

We will have answers for your questions when we will have proper samples. For the moment just keep in mind that digital push-pull takes place "after the effects of noise etc etc in the circuitry of the camera". By pushing by 1.25 (for ISOs 125-250 ..) it depends on where is the "integer" ISOs real White point, if it's at 0.80 of the max 14bit level then nothing changes ..

Not sure what is happening here but normally if there is no skipping we should take less noise after bining pixels which is not the case according to measures on Black frames ..

http://www.magiclantern.fm/forum/index.php?topic=5609.msg39410#msg39410

@ a1ex,

I used squig's (outdayted) samples for my histograms. Can you share your samples ?.

In Dcraw command line you use -h which (at least in my system) gives demosaiced data .. not a big problem as there is minimal interpolation. I really don't understand how you take gappless histogram. Rawanalyze show "gappfull" histo also.

Can you export histograms per channel ?. If not try http://www.libraw.org/download 4channels.exe

so far i understand the pink highlights problem - its no problem of data handling but of managing the "beyond whitelevel sensels". the checkbutton is a fast built in tester if its changing something. if you search after "canon raw pink highlights" you (we) see, its a problem since years - manually managed by the raw-converters - and the dcraw-community is the only one trying to solve that by explaining the problem.

example links:

may 2012 - http://sourceforge.net/p/ufraw/discussion/434060/thread/b43582e3/
march 2009 - http://www.luminous-landscape.com/forum/index.php?topic=32942.0
feb 2010 - http://www.dpreview.com/forums/thread/2748822
oct 2011 - http://code.google.com/p/rawtherapee/issues/detail?id=1037
dec 2009 - http://www.flickr.com/groups/photomatix/discuss/72157623070526860/

regards chmee

In fact the problem comes from Dcraw which uses a single (many times wrong) value for White Level while WL floats depending on ISO settings (different for "integer" ISOs and "intermediate" ISOs) and on len's f-stop (there is digital amplification for larger than f/2.8 settings pushing the WP higher).

With DNGs the solution could be easy if ISO & f-stop was reported because there is an exif tag for WL, raw2dng/raw2cdng can take care of it.

For native raws (*.CR2) there is a need from the decoders side for a database as will be implemented by RT team .. http://code.google.com/p/rawtherapee/issues/detail?id=1752
I am sure Adobe uses a database also with rather conservative settings and as far as I know in DNG conversions they do not increase WL for open f-stops, not a big issue .. just a slight loss of highlights.

As it is now, ML uses the conservative settings that we see in Adobe's DNG conversions and I think they are safe, but we need a decoder to cooperate and maybe it's Resolve's error interpreting the exif tags wrongly .. like RT did a while ago https://code.google.com/p/rawtherapee/issues/detail?id=1695