Messages

Are you referring to the register values tagged as WhiteLevel in my dropbox graphs, or to values detected by raw_diag?

In any case, feel free to use them for RawTherapee. I took the AMaZE algorithm from them, and I'm thinking to send them a patch for floating-point DNG support (my version doesn't open CeroNoice DNGs, but didn't check the bleeding edge).

If you collect a table with white level values in a nice format, I'd be interested too (just to have them organized).

I think raw_diag white levels is what I need .. basically the values that are same as if I would inspect  the raw histogram for white clipping. This for normal Canon cr2s not ML modified .. and I think ML should also target to the same WLevels as the native Canon cr2s.

I would strongly prefer to use the exif data (specularWL) so the best would be a DCraw patched to use the tags, but I have to recheck as I remember a not insignificant deviation from the histogram derived WL in some rare cases ..

floatDNG is not supported ... waiting for your patch ..

The WL info will be freely available in RT code just search for the "camconst.json" file. I don't know if you'll like the format but as I will use spreadsheet to get the data ready I can sent it when it's ready ..

Ufraw doesn't care about spec(ta)cular white level either.

I mean that Dcraw and derivatives only care for the topmost value of data. This is hardcoded in Dcraw and Ufraw (single value for all cases .. you know how good this is ..) or a lookup table.
Rawtherapee uses an external file named camconst.json (camera constants) where among others one can put per ISO WL and a multiplier for each Fnumber ..
DNG looks like also using a table for each ISO but has nothing to fine tune WL depending on apperture ..

In fact I came again in ML forums searching for data regarding the per ISO and per aperture WhiteLevels .. so if it's possible to use the already collected data I ask your permition to use them for RT .. and a bit help how to mine them from the log files .. or even better if the are readily available as numbers ..

@ IliasG what is that program showing all those metadata entries?

exiftool with exiftoolGUI as frontend

Yes, DPP should use this info but I don't have it to test.

The Normal white level was 14000 before I changed it to 4321 .. so the test with DPP should be between 14000 and a changed one ..

I don't know the exact meaning of these tags. Adobe DNG/Dcraw and derivatives only care about SpecularWhiteLevel as I understand it. Although it does not always match the white clipping level as we can see it in raw histograms.

Although I think it's interesting that there are three tags about WL which DPP probably uses.
We see the linearity upper margin being 1/2 stop under normal WL . does this mean that the raw highlight data are non linear ? or were  they non linear at the start (ADC output) and then linearized before storage in *.CR2 ?.

WL values seem to be steady after gain manipulations, so I don't know  if they are reliable as of now. But I hope ML can push Digic to write the correct ones if needed .. .

BL values look to be measured in the optically black zones so they are reliable and could help for a faster BL identification ..

Works for me on win vista32 with exiftool 9.51

Does not exist ? for which model do you take this ?. Any sample available ?.

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#ColorData7

Convert them to DNG and use exiftool if needed. There are no white level tags on the CR2.

I believe Canon autodetects it somewhat like raw_diag does.

With exitool on CR2s you can try
-Canon:PerChannelBlackLevel  (needs four values)
-Canon:NormalWhiteLevel
-Canon:SpecularWhiteLevel
-Canon:LinearityUpperMargin

Binary ??. I don't know how .. a simple text file with input and output columns is the best I can ..

I'll have to reopen and remember what the refinement is .. hopefully tomorrow

Did you managed to insert the linearization table in DNG exif data  ?.

Yes, it's ufraw using a hardcoded white level (taken fron dcraw), which is not always correct. I might port the autodetection routine from cr2hdr, since I couldn't find a way to figure it out from EXIF settings (it's influenced by both digital ISO and exposure time, and each camera has its own levels).

* In the raw histograms one can see that WL is influenced from wide apertures also, there is digital scaling which starts as negligible (1/200) at about f/3.2 and gradually increases as we go wider (1/10 at f/1.4) for FF models and more for APS-C.

Alex, with latest exitool you can find in -Canon tags
0x01fc NormalWhiteLevel
0x01fd SpecularWhiteLevel
0x01fe LinearityUpperMargin

I think "SpecularWhiteLevel" is what dcraw and derivatives need (although it's sometimes a bit conservative related to the raw histogram)

I do not know more than what Iliah Borg wrote at RT issues discussions http://code.google.com/p/rawtherapee/issues/detail?id=2125#c10
For more info you could contact with libraw or rawdigger developers as they are who decoded those tags.

Hopefully LinearityUpperMargin will be useful to you for DualISO etc.

https://infogr.am/bmcc-delinearization-curve?src=web
(some brainfood) (now both curves lin>log and log>lin inside)
full csv-data for log>lin on http://dl.phreekz.de/raw2cdng/bmccLog.csv

regards chmee

Chmee hi,

Do you mean that you (will) use Bmcc curves ?. At which bitdepth ?. If you need something just ask. I think the old gamma curve I provided needs a bit refinement. Of course it was proposed for 14 bit linear to 10 bit log so we will need something different if our intention is to use gamma encoding for 14lin to 12log or 16 to 12. 

Regarding magenta highlights, because the clipping White level for each model depends on ISO and apperture,  in latest RT there is now an external file (camconst.json)  where we can set per ISO White Level and a multiplier depending on apperture. I think a database is the only way to be safe while not loose highlights.

Or you can use exif (maker data) info in CR2 files, latest exiftool decodes more Canon tags  and White Levels are among them although they are a bit conservative sometimes ...

Currently, for RT, there is a need of Canon raw samples with at least a "burned" area shot with intermediate ISOs (125, 160, 250, 320 ...) to fill the database ..
5D2 and 5D3 are the only models whith complete data, all others (and 6D) are missing.

I'm also on the way to fixing the black banding from 7D at ISO 100 (like this). How many of you did you notice it? Do you have more samples? Did you notice similar problems on other cameras?

I hope this helps a bit .. http://www.dpreview.com/forums/post/51945370

Small update for cr2hdr_exp.exe: better black level handling. This is only noticeable in extremely underexposed shots (e.g. when you need to push the shadows at +10 EV).

Before and after (CR2 from AdamTheBoy):
http://acoutts.com/a1ex/7O4C8804_before.jpg http://acoutts.com/a1ex/7O4C8804_after.jpg

For grading: I've developed the DNG at +4 EV ... +12 EV in 1 EV increments and ran the shots through enfuse. The image was quite underexposed, next time try using ETTR

Just for fun: here's how it would have looked at ISO 100. Good luck denoising that.
http://acoutts.com/a1ex/7O4C8804_dark.jpg

For higher-res crops, download the CR2 and grade it yourself. This update was about color cast in shadows, not about fine detail.

I'll update it on the first page and in raw2dng when I'll solve the remaining issues in highlights.

@IliasG: you around? can you check this and see if you get better results?

Yep, around and just checked. Much better about color cast at darks, but at a cost. The noise is harsh and it's not points (easy to denoise) but like candles. And I think that there is too much black clipping.

As I see here the Black Level in the CR2 file is around 2032-2034 for the low ISO lines and 2062-2066 for the hi ISO lines. How do you normalize for this ?. (In the meantime I have to read again your pdf I suppose ..).
This shows clearly that Canon tries to normalize at the desired average of 2048 as BL but how exactly ??. Is it just a data sliding up or down or a multiplication ?.

It happens this shot is with f/1.8 so there should be data scaling (due to wide aperture) and the White Level must be around 16000 ... the the DNG's exif says 49600 !! ?? even lower than 4*13000 ...

In the case of white level I still vote for a Lookup table.

dcraw uses 13584 (hardcoded). And yes, since the white points changes with ISOs, it makes things difficult; for now, I just use the smallest value from burned areas, if it's above 10000. Probably a bit overkill, maybe 13000 should be a better starting point?

Dcraw is ineffective regarding WP, it uses a single value for every ISO and runs into the known problem, either pink highlights (ISO 100-160-320-...) or 1/3 stop burned highlights.

The best solution is to have a lookup table with the correct clipping point for every case. Keep in mind that we have data scaling (and higher WP) even when shooting at wide apertures (wider than f/2. with Canon glass ..

Adobe DNG converter uses 13000 as WP for 100-160-320 .. and 15000 for 200-400-800 .. For 250-500... I think it goes up to 16000.
But takes no care for wide apertures so in this case up to 1/3 stop highlights get lost ..

I do think that if you use a single WP then 13000 is a safe bet. Although you loose 1/3 stops of highlights for ISO 200-400-.. and maybe more for 250-500.. and when using wide apertures.

13584 is on the limits for ISOs 100-160-320-.. because when there is raw denoise (like "black frame subtraction") then we take not a single clipping point but a distribution extending -/+ 100 raw levels from the center value .. it's the same case as in Guillermo's  study .. http://www.guillermoluijk.com/tutorial/satlevel/index.htm

In these files, the white level is close to 13000; in my 7D test sample it's 15382. Normally you get these values with intermediate ISOs (160, 320).

If you change the white level in cr2hdr.c to 13000 instead of 15000, these files will be converted without pink highlights. Now trying to figure out how to get this value from exif or what settings to use to get raw values higher than 15000.

Some Canon models (I think all APS-C with 18Mp) have lower white levels for ISO 100 than the rest ISOs. For 7D it's around 13500 for ISO 100 and 15300 for the rest "integer ISOs".

It would be very useful if you find a way to declare this from *.CR2 and it's exif

The sad thing is that some channels (B and G2 in my samples) have raw values higher that this white clipping point making the detection difficult even if we have burned areas ..

..
By the way Ilias, did you look at the noisetest dng,s?

Yes, and in this comparison the single ISO shot matches better the Dual ISO one. The difference is smaller than with the first pair, no weird casts etc.

@ Alex
Do you consider instead of calculating "black data" in camera just save the side optically black pixels for better calculations later with raw2dng ?. For these black data 12bit are enough as it is now and if there is a possibility to subtract the black offset 8bit will be OK (for BL at 2048 subtract 2048-128= 1920 so 256 will cover the range +/- 128).

Try to change the model name in exif from "Canikon" to your model "Canon EOS 5D Mark III". This way ACR will use the color profiles for your camera and will behave better for warm temperatures because it will use the dedicated for this case "Illuminant A" color profile.

Thanks for the samples.

check out my picture of a 100-6400 some posts up. Worked rather well.

It did worked.
But the point is that the difference between 100-3200 vs 100-6400 is very small  in noise quality (0.09 stops in favor of 100-6400) while the difference in artifacts and resolution - detail is significant in favor of 100-3200.

Keep in mind that the result you see using 100-6400 is not only due to the cleaner ISO 6400 but also due to better processing by cr2hdr (black point calibration, channel matching ..) and a kind of denoise ( the data averaging by median filters) which get activated even when the ISO range is small.

You will need tests 100-6400 vs 100-3200 vs  100-1600 vs 100-800 (or 1600-3200 vs 1600-6400) combined with the appropriate for each case denoise by your raw converter to have useful data for a correct strategy for dual-ISO shots.

IliasG could you elaborate the concept in some more detail.

Alex explains it in http://acoutts.com/a1ex/dual_iso.pdf page 1-2 and gives some links ..

http://www.guillermoluijk.com/article/nonoise/index_en.htm (it's a dead link for the moment, let's hope for recovery)
http://www.dpreview.com/forums/post/28749589
http://www.dpreview.com/forums/post/28750076

The explanation for the decreased of noise at the dark areas at hi ISOs is that there is electronic noise at the late stages (transfer lines, DAC) after ISO amplification which noise is constant. If we feed the DAC with low signal (ISO100) lets say the noise is equal to this signal (SNR 1.0). If we feed it with amplified signal (ISO 1600) then the signal is 16 times stronger while the noise remains constant so SNR is 16.0 .. This with an ideal sensor and first stage electronics with no noise. In practice there is first stage noise also (but lower than late stage) so the SNR becomes around 8.0 and we gain 3 stops DR instead of 4 ..   

How can you whitebalance rawmovies. Rawmovie converted with latest raw2dng.exe. Dngs provided in the link. The test was aimed for noisecomparison. Thanks for watching

To make fair noise comparisons we have to compare shots at the same exposure (light inconsistency can give faulty conclusions) and the compared images been at the same contrast, saturation, WB and detail levels ...

Your uploaded dual ISO DNG is named 6400-100 .. wasn't your shot at 6400-1600 ??

Can you declare what lighting you used ??.
And upload the first frames of each session if possible, I would like to have them because the Black Level data is calculated on the start of each session.

[and adjusted the exposure accordingly.]

Hi everyone,

somewhere here in this thread I read that the 5D3 produces less noise in dark areas with a higher ISO, therefore the dark areas are recovered from high ISO while light areas are recovered from low ISO.

I did some tests today and filmed the same scene (without Dual ISO) in ISO steps from 100 to 3200 and adjusted the exposure accordingly.

In other words: I don't understand the Dual ISO. What did I miss?
As I see it lowest ISO is best for highlights and for shadows.

(Faulty step in bold).

You missed the fact that to take the effect of cleaner darks with hi ISO you have to keep the exposure constant between shots .. and exactly is what dual-ISO does ..

The movie raws are about 350mb each, to big to upload from where I sit atm. You can play around with two extracted dng-files if you like . Link below.
https://docs.google.com/file/d/0B4tCJMlOYfirTTRETlpaQnZVMG8/edit?usp=sharing

@Ilias G. I don,t see anything wrong with the single iso-file?

Thanks for the samples.

The problem I see in your 200% crop of the single ISO file is a magenta cast at the darks of the sculpture and some clipping. Which is much less present with my conversions using Rawtherapee.

What was the lighting during the shots ? I have a hard time finding a correct WB.
Was it consistent in color and intensity ?. Because if it was a candle we cannot come at a valid conclusion comparing only two single frames.

@ Danne,

Can you upload the RAWs of this last comparison please.

I think something is badly wrong with the single-ISO sample.

.... Samuel mentioned that 100/400 still gives you 13.5 stops and only the top and bottom 2 stops have aliasing vs top and bottom 4 with 100/1600. With only a half a stop DR penalty to substantially reduce the aliasing 100/400 looks like the best option so far.

What you describe is for 100/800 not 100/400 .. http://www.magiclantern.fm/forum/index.php?topic=7139.msg59971#msg59971
And I also think 100/800 is optimal.

At 100/400 the DR increase is 1.25 less than 100/1600.

I can remove the shadow aliasing completely, but right now I'm only cleaning 50% of it, because the noise gets pretty bad. Still lots of room for tweaking (but you need to compile yourself, there are no user parameters yet).

Here's the spot you need to change:

Code Select Expand


            if (b0 < white && d < white)
                f = MAX(f, c * FIXP_ONE / 2);


change to f = MAX(f, c * FIXP_ONE); to remove all the shadow aliasing, and maybe tweak CONTRAST_MAX to fine-tune stuff.

I think that as soon as FPN is suppressed (and cr2dng does a beautiful job on this) then the optimal mix depends on the workflow. I suppose that with ACR (and it's very effective denoise in use) a user would be better to have a DNG with less aliasing at the cost of more noise and then denoise in ACR.
The same with Rawtherapee which has some nice tools for demosaic, denoise, false color suppression ...
With Resolve with it's primitive demosaic and denoise the needs are different.

User parameters would be a nice option.

 

@ neofg,

give a try at the modified DNGs where I just changed the colormatrix with the one from Pentax645, it's closer to what you want, please try and report if it's close enough.

http://rapidshare.com/files/2796423499/BLX00001-20110317-0000436-15mm-Elitar%20-%20Pentax645.dng
http://rapidshare.com/files/3303040773/BLX00001-20110317-0000436-15mm-Elitar%20-%20Pentax645_dual_illuminant.dng