Messages

Marten, which Dcraw build do you use ?.

Did not try RAWanizer yet but I have to say that the faster Dcraw for my system (win vista32, C2D 3.0 Ghz) is https://sites.google.com/site/manuelllorens/

and it includes Lcms - libjpeg - jasper

[can any expert give an opinion on this]

...

I hope it is clear now. But, look-up table might be indeed a better alternative, unless referencing memory via ptr is expensive, I am not familiar with efficiency of DIGIC5+.

BTW why it was so important to round it so well? Perhaps for 10 bits?

In case it is a question for me, yes with 10bits raw bayer we need very careful rounding and even with careful rounding 10bits depth can be inadequate. While with RGB 10bits can be (and are) OK with raw Bayer "master" there is the need to apply some heavy manipulations not needed for the ready to use RGB file. Demosaic, WB (which is a 1+ stop digital amplification for the weak channel), Color transformation from Bayer R-G1-B-G2 TO RGB (once again about 1 stop amp for the weak channel)

But with a lookup table we can have untouched the low values (keep the sampling at the same density as the 14bit starting data) where the problem is significant and gamma encode the rest.

We can even develop a curve imitating the S-log ..

Say we go for 14 to 10bit,
- we can clip at 32 (or even less) 14bit levels below "Black Level" instead of 2048 as it is now. Just map any 14bit level lower than 2016 (14bit) to zero. The remaining levels are 16384-2016 = 14368 which will be mapped (with any
- the first (say 128) 14bit values stay untouched by using a linear part with a slope of 16 y= 16*x
- then continue with a gamma .. say y= x^1/2
- if we want the S-curve then at the last 1/2 or 1/3 stop we invert the gamma y=x^2

Try to dump Nikon's D5200 raw curve with Rawdigger .. it's as described before .. linear-gamma-inverted gamma. Same proposal as AXIOM's Smart Dynamic Range

Let's hope the lookup table can be managed fastly by Digic .. can any expert give an opinion on this ??.
I think for a PC (X86 architecture) a lookup table is almost always preferable/faster.

One more compression scheme crosses my mind... Can we, during writing the raw data, use variable bit depth adapted to the real needs (values up to 256 really need 8bits). Then the same lookup table can be expanded with one more field giving this info ... .

[how fast can this lookup be ??.]

Of course the best is to get a ready 12bit raw or even 10 bit if it is gamma/log encoded. In fact I believe a 10bit log is better than 12bit linear.

But if this hack is difficult-impossible then the compression must be made in CPU.

I think that the way to go is 10bit log or rec.709 gamma encoding by a lookup table. No floats, no calculations .. all precalculated in at table.
Then the reverse (linearization table) can be in raw2dng or in the dedicated exif tag.

Something similar do BlackMagick and Sony and Nikon's etc.
BlackMagic encodes a 16bit linear raw in a 12bit gamma rec.709 (or log) file. The 12 log to 16 linear bitdepth sounds hyperbolic but we have to remember that BM uses only digital amplification (in fact it is just a exif tag .."Baseline exposure") for ISOs higher that the base so they need a bit more bitdepth.

ML needs 14bit linear to 12-10bit gamma or log and together with the analog ISO (up to 3200-6400) the expected file quality is at the same level as BM for mid ISO and higher at high ISO.

The question is how fast can this lookup be ??.

The lookup table with 16384 indexes and one column data with 1024 distinct values (10bit) is OK I think.

To make things better we can safely clip the 14bit data at about 1984 ( Black point is around  2048 - 64 = 1984) so any pixel with 14bit value <= 1984 will be mapped to 0 10bit. This way we can have a denser sampling of the useful range of values.

Looking at Blackmagick's and Nikon's linearisation tables ("Dump raw curve" with RawDigger) one can see that Nikon use an extended linear area at the darks and the exp area follows for midtones and highlights. It is usefull to keep data linear at the darks and without any manipulation .. usefull to avoid posterization and color shifts at the darks.

The 64 levels "below black" I choose is because so does BM (256 levels 16bit) .. and to leave footroom if something goes wrong (Black Level). It can be half (32 levels) with no problem.

Ideally we need a gamma with 16X multiplier at the linear part (so that no change needed there going from 14bit to 10) which extends around 20-40 levels (10bit) over "Black Level" (Nikon uses 400 linear levels at 14bits) .. and a relatively mild exp (^2.0) so that the highlight data stay relatively dense. Sadly my background in Maths is weak and i cannot match the linear with the exp part .. help needed ..

16X linear multiplier has the Prophoto gamma but it is linear area is very small, all falls in the "below black" area ..

are these photo's?  Cus when i did this test with video it was undenyably clearer on 320,640,1250,2500, and 5000

Those ISOs 160 - 320 ... were darker than the rest isn't it ?.

If we change the Black Level and White Level tags at the DNG's exif with the correct values they will look cleaner also ...

These are the Spydercheckr colors:
http://spyder.datacolor.com/wp-content/uploads/datacolorproductliterature/SpyderCheckr%20Color%20Data.pdf

It is very different from Macbeth cc24. So no way to build dcp profiles with free software.

But I think we can make icc profiles with Agryllcms (if needed).

@Samuel,

Thanks for the offer.

I am interested in shots with 5DIII of the X-rite passport color checker because I only have the "Color Checker Passport" profiler available.

Do Datacolor's Spyder checker and X-rite passport have the exactly same color patches ??. To make dcp profile with Datacolor's  I need their dedicated profiler .. No ?.

I shot that test with the 21 May build so it's post magenta cast fix, the MK3 is known for its magenta cast in raw stills. As you can see from the DNGs I uploaded I haven't messed with the setting in ACR except to 0 the tint and set a daylight white balance. I think 640 will look noisier than 800 in a real world test, I'll get around to one soon. When the weather clears up I'll get on to the white clip test.

Ohh .. I missed to scroll down the exif so I didn't see your settings.
Good that you used a better than the "as shot" WB but in the case of black frames we need around 4000-4500K to balance R vs B channel multipliers and take a more representative visual effect of the noise.

Regarding ISO 640 vs 800.
I just discovered that at ISO 640 the exif.White Level is 8918 !!. This looks like a bug, the WL of ISO 640, ... cannot be less than 0.80*ISO800_WL. and in order to look 1/3 ev darker it should have the same WL as the ISO 800 ...

The exif Black Level is 2050 and the measured avg. = 4047.15 = 2.85 avg. clipped levels.

At ISO 800 exif.WL = 15000 exif.BL = 2051 measured avg = 2046.8 = 4.2 avg clipped levels.

So in the end the ISO 640 black frame takes a seamless +2/3 EV compensation due to the wrong very low WL and 1.35 avg. levels less clipping at the blacks .. No wonder that it looks noisier.

I am waiting for the white clip test to clarify the actual White Levels.
Even better would be to additionally have 1-2 raw shots (both photo and video) under direct sunlight of a x-rite colorchecker passport,  in order to build a dcp color profile ..
Just for the case that the video-raw needs a different profile than the photo-raw... and I think this different profiling is possibly needed due to the scaling of the video-raw values.

Wow, simply lowered black and white (*0.25f) in raw2dng before it exports

Code Select Expand

raw_info.bits_per_pixel=12;
raw_info.white_level *= 0.25f;
    raw_info.black_level *= 0.25f;

And now the file opens ACR 



I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Nice, thanks.

Is this posterisation down there ??.

Can you upload your dng file ??.

These samples look a bit magenta to me (or maybe I'm color blind?). If it would have been too much, these things would be green.

I'd say it's a bit too little, or very close to just right.

You are not color blind for sure.

But we cannot judge color casts from the black frames. Most times the dominant color there is red-magenta because (in the total absence of light) it depends on the WB multipliers and R & B are always bigger than Green. Try to change WB to "tungsten" and the dominant color is blue.

BTW the prefixed WB multipliers by ML are now G=1, R=2.477, B=1.462 which gives a very unnatural WB as if it is (in Lightroom4.4) 6200K/+66 tint (strong shift to magenta ) which makes the "as shot" conversions .. you know what non color blinds see ...

As a reference Canon gives (in exif. maker data) the Daylight multipliers as G=1.0, R=1.980, B=1.5977

The conversation about BL and magenta cast started because early ML builds calculated BL very low, less than 2040. When the BL was correct there was no objection.
Now (after some improvements of the code) looks like ML gives correct BL by calculating the average of optically black pixels and then adds this stdev/2 to increase the correct result at a wrong level.

A useful (and easy) improvement for the BL is the per channel Black Levels ..

The thing about higher than the correct Black Level is not only color casts and difficulty to build a faithful WB and color profile. It is about clipping data also.

I 'll have to experiment a bit changing exif data to provide samples supporting my position .. wait until tomorrow.

Right, I found it, had to overwrite raw_info.bits_per_pixel=12; in raw2dng.

Now the image is good, only the colors are messed up.

Looks like this in irfan view (doesn't open in ACR):

http://i.imgur.com/lsMeEFO.jpg

(It's the bottom of a (very dirty) window)

You also have to adapt the Black Level & White Level to 12 bit ..

[Please anyone who can provide video-raw samples (made by the last build) at all ISO settings, which include burned areas (and not only), .. do upload]

Here's how the fixed pattern noise looks +5 stops no noise reduction applied.
Downloadable DNGs for your viewing pleasure https://drive.google.com/folderview?id=0BzJ3L6nv6Fn0TmF0N0xLSFBaalk&usp=sharing

Cropped
http://i.imgur.com/OSJ6Wip.jpg

Scaled
http://i.imgur.com/22e4cHf.jpg

Squig, many thanks for the DNG black frames.

Most samples have a Black Level of 2047-2048 but some of the high ISO have different BL for any channel (ISO 3200: R,G1=2046, B,G2=2048).
ML code adds half the stdev at the measured BL giving difference of +3 up to +14 (ISO 6400 measured BL 2047, exif tag = 2061 .. this looks too much .. )

Looks like the video-raw samples follow the normal (photo-raw) Canon pattern ..

ISO 100, 200, 400 ... derived mostly* by analog amplification
Read noise is ISO100, 200 - 6400, ... 7.55, 7.9,  8.1, 8.65, 10.0, 12.6, 18.5

*Looks like there is a part of digital amp (by 1.07 ?) as there are some gaps and some "half gaps" in the histogram. I believe that there is a histogram stretching at the "video raws" so that it covers all values up to 16382 instead of the 15283 limit for the "Photo raws"

ISO 125, 250, 500 ... derived from ISO 100, 200, 400 ... by digital amplification by 1.25
Read noise is ISO125, 250 - 4000, ... 9.9, 10.2, 10.5, 11.2, 12.9, 16.4,

ISO 160, 320, 640 ... derived from ISO 200, 400, 800 ... by digital amplification by 0.80
Read noise is ISO160, 320 - 5000, ... 6.1,   6.3,   6.7,   7.7,   9.7, 14.2

I had the hope that those video-raw pixels come of binning 8 photo-raw pixels and this would decrease the noise floor by 2 stops ... but NO .. the noise floor is almost the same (a bit worse) as with the photo-raw !!. So DR is not improved !!.
Reasons for not having improved noise floor can be either pixel skipping or the system (sensor-signal transfer-DAC) working at higher speed resulting in more noise. .. or both.
Anyway, without detailed measures over all the luminance scale, we cannot have a correct conclusion.

The impression (at another thread) that ISOs 160, 320 .. are noisier is faulty.
If the samples was derived by converting with Dcraw the reason they look as noisier is the (by default) histogram stretching giving in fact a seamless +1-2 stop EV.

This together with a wrongly lower than the correct White Point that ML sets .. The exif White Level tag for ISOs 160, 320 .. is 11566 while it should be 16382*0.8=13105 if no histogram stretching happens for those ISOs and 16382 if histogram stretching exists.

It is obvious that to fully characterize the video-raws we need video-raw samples with a burned (white clipped) area.

Please anyone who can provide video-raw samples (made by the last build) at all ISO settings, which include burned areas (and not only),  .. do upload

Pattern noise removal would be dark frame subtraction?

Any noise reduction for raw video would be best done temporal and possibly with motion compensation.  So basic spatial and or dark frame in raw then temporal noise reduction, luma sharpening etc either in image sequence stage or during video edit / grading stage if chosen raw wrangling apps don't support temporal noise reduction.

I think the first that can help to fight any "Channel imbalance" (which is the the main reason for this banding or pattern noise) is measuring the Black Levels per Channel and write them at the related exif tag.  Then use this info to normalize ...

I find useful to

- run all DNGs through exiftool to change exif tags like model name, WB as shot,  Black and White Levels, dates, .. go on.

- run all DNGs through Adobe DNG Converter and resave them as losslessly compressed. Size goes from 3.6MB to 2.1-2.2MB on average when it includes a medium jpeg preview (1024X576) and a bit less with the "minimal" preview.
This "medium" preview can be a substitute of the requested mp4 preview when extracted as jpeg sequence ..

If one choose to compress to Lossy DNG, size goes to less than 1MB but the result is a 8bit-log dng file .. someone has to check if it is robust enough for heavy gradation ..

If the above could be automated in this "rawtoDNGb" utility it would be the nice ..

A related GUI utility is Olli's TurboDNGimporter .. http://www.visualbakery.com/Tools/DNGImporter.aspx

[ῢ]

What strange character?!

This one .. ῢ as I see it in exif with exiftool ..

As you say it's correct in silent pic DNGs.

Please include any info is possible (and easy for the moment) in exif tags.

Model name, ISO, Illumination A matrix ... any ..

Especially for the model name please change this strange character ( shows as greek ypsilon polytonic  in my system ). It is very distracting for me because except that it is strange and with no meaning,  I am a RawTherapee user and this character triggers RT to freeze when I try to open a folder with ML video-raw files for the second time (first time is OK but at the second something goes wrong with RT's cash).