Author Topic: Dual ISO - massive dynamic range improvement (dual_iso.mo) (Read 2156705 times)

a1ex · « **Reply #2400 on:** March 02, 2015, 12:32:02 AM »

Updated cr2hdr-20bit (first post).

Fixes: this, these, this, this and a few other minor things.

DeafEyeJedi · « **Reply #2401 on:** March 02, 2015, 07:03:45 AM »

Thanks @a1ex for the kind update!

Audionut · « **Reply #2402 on:** March 02, 2015, 10:41:37 AM »

Quote from: a1ex on March 02, 2015, 12:32:02 AM

The second one should be marked as blown out, right? It's a white level problem (not detected correctly if you decrease the gain too much).

Noticed this issue today without any ADTG changes.

Is this something I might be able to diagnose with a little nudge in the right direction?

a1ex · « **Reply #2403 on:** March 02, 2015, 10:59:11 AM »

If you upload a CR2, I'll be able to load it in QEMU and see what happens. Here we need a pretty good heuristic, because once the gain drops too much, white is no longer clipping to a constant value - there's noise instead. The white level should be right below this noise in the clipped areas.

Something like this:

Our heuristic should figure out the white level from a single image (from histogram data), not from a nice graph like this. Not exactly an easy problem.

Audionut · « **Reply #2404 on:** March 02, 2015, 11:18:04 AM »

https://dl.dropboxusercontent.com/u/34113196/ML/dual_iso/_MG_2788.CR2

https://dl.dropboxusercontent.com/u/34113196/ML/dual_iso/_MG_2789.CR2

Just to confirm, no ADTG changes (module not even loaded). Base ISO 100 shot above, and then dual ISO image below (no other changes).

On closer inspection, there are some subtle changes in the histogram data also.

mothaibaphoto · « **Reply #2405 on:** March 05, 2015, 07:05:20 AM »

Great thanks for fixing this "Hot" (cold, actually) pixels issue. I did some pixel peeping and did not find any IQ degradation. Really impressive done, bravo !!!

Original:

Fixed:

keepersdungeon · « **Reply #2406 on:** March 05, 2015, 12:08:02 PM »

Gd job on fixing the Hot pixels.

I have a stupid question though, I don't seem to be able to subscribe to this discussion, is there a way to do it ? or i just don't have the permissions?

Danne · « **Reply #2407 on:** March 05, 2015, 12:10:47 PM »

That pixel correction handling is mind blowing.

keepersdungeon · « **Reply #2408 on:** March 05, 2015, 12:37:59 PM »

Do you usually use ETTR with the dualISO?

mothaibaphoto · « **Reply #2409 on:** March 05, 2015, 03:43:46 PM »

Quote from: keepersdungeon on March 05, 2015, 12:37:59 PM

Do you usually use ETTR with the dualISO?

If you ask me, than answer is no. Why - described here

keepersdungeon · « **Reply #2410 on:** March 05, 2015, 04:07:39 PM »

Alright, thank you!

N/A · « **Reply #2411 on:** March 07, 2015, 02:26:46 AM »

New cr2hdr compiled for Mac

https://www.dropbox.com/s/0yqh0baqur4wlcf/cr2hdr.zip?dl=0

DeafEyeJedi · « **Reply #2412 on:** March 08, 2015, 07:08:47 AM »

Thanks for sharing @ N/A!

N/A · « **Reply #2413 on:** March 08, 2015, 08:25:26 AM »

Anything to help.

chris_1975 · « **Reply #2414 on:** March 10, 2015, 11:02:31 AM »

Thx for hotpixel reduction .....
50sec long exposure ....

https://www.facebook.com/122338811269922/photos/a.122339334603203.1073741826.122338811269922/281093542061114/?type=1&theater

keepersdungeon · « **Reply #2415 on:** March 10, 2015, 11:34:31 AM »

I got confused a bit with all the posts so I apologize in advance for my stupid question. But is the hot pixel fix due to cr2hdr 20bit update or is the dualISO module has been updated?

rpt · « **Reply #2416 on:** March 10, 2015, 01:21:42 PM »

Chris_1975, lovely picture.

Danne · « **Reply #2417 on:** March 11, 2015, 03:11:51 PM »

Hi!
There,s one thing puzzling me with dual iso RAW and MLV. I compared the formats recording with a recent build, (about two weeks old, don,t have my camera here)

WHen shooting RAW all is fine giving me a file a little darker as expected. WHen shooting with exact same settings with MLV the file is brighter after postprocessing. It,s like it,s inverted? Am I doing an obvious mistake here?
examples shot in 100/1600 3x crop mode. Same result when shooting non crop mode. Will upload RAW/MLV examples if needed. (Short ones).

Converted with latest cr2hdr20bit build Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:.
The same goes for the former build. Shot with a 5d mark 3, firmware 1.1.3

DNG files
https://drive.google.com/file/d/0B4tCJMlOYfirYWl6WmF2ME9laWM/view?usp=sharing
3x crop RAW 100/1600

3x crop MLV 100/1600

a1ex · « **Reply #2418 on:** March 11, 2015, 03:53:33 PM »

Can you upload the source DNGs created by raw2dng / mlv_dump?

Danne · « **Reply #2419 on:** March 11, 2015, 04:11:17 PM »

Dual_iso dng files, unprocessed from raw2dng and mlv_dump
https://drive.google.com/file/d/0B4tCJMlOYfirdXdmaktRUG1NRXM/view?usp=sharing

dual_iso RAW original(75mb)
https://drive.google.com/file/d/0B4tCJMlOYfirV0ktd2w3c0UtVjg/view?usp=sharing

dual_iso MLV originals (161mb)
https://drive.google.com/file/d/0B4tCJMlOYfirdWhMWXNJRWxNMW8/view?usp=sharing

Thanks
/D

a1ex · « **Reply #2420 on:** March 11, 2015, 04:27:27 PM »

I've got two DNGs with similar brightness.

Can you paste the conversion log?

Danne · « **Reply #2421 on:** March 11, 2015, 05:19:04 PM »

It seems the problem is coming when I,m inside folders and letting c2hdr-r working in different levels. Therefore the conversion process works from parent folder. I have to check further. A little busy atm.

*Actually looks like it is only the first file that is giving strange results.

Here is a log from dual iso conversion. Also noticed some strange intermediate issue before it turns into a transcoded DNG (picture).

Log from faulty conversion dng files

Last login: Wed Mar 11 17:23:10 on ttys000
itgusercbook241:~ itguser$ /Users/itguser/Desktop/testMLV\:RAW/cr2hdr-r\ 5.3k_s.command ; exit;
ORIGINALS

MLV Dumper v1.0
-----------------

Mode of operation:
- Input MLV file: 'M11-1138.MLV'
- Convert to DNG frames
- Output into 'M11-1138_1_2015-03-11_0001_C0000_'
File M11-1138.MLV opened
File M11-1138.M00 not existing.
Processing...

Vertical stripes correction:
1.000 1.000 1.000 1.002 0.999 0.996 0.997 0.999
Reached end of chunk 1/1 after 56 blocks
Processed 39 video frames
Done

mv: rename M11-1138_1_2015-03-11_0001_C0000_.wav to M11-1138_1_2015-03-11_0001_C0000.wav: No such file or directory
mv: rename M11-1138.M* to ../ORIGINALS/M11-1138.M*: No such file or directory
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2 : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file : ./M11-1138_1_2015-03-11_0001_C0000_000000.dng
Camera : Canon EOS 5D Mark III
Camera model : Canon EOS 5D Mark III
Full size : 2048 x 1152
Active area : 2048 x 1152
Black borders : N/A
ISO pattern : BddB RGGB
White levels : 10000 14882
Noise levels : 8.00 8.00 8.00 8.00 (14-bit)
ISO difference : 3.89 EV (1487)
Black delta : 5.26
Dynamic range : 9.96 (+) 10.65 => 14.54 EV (in theory)
Looking for hot/cold pixels...
Hot pixels : 34
AMaZE interpolation ...
Amaze took 0.30 s
Edge-directed interpolation...
Semi-overexposed: 55.57%
Deep shadows : 4.55%
Horizontal stripe fix...
Full-res reconstruction...
ISO overlap : 3.1 EV (approx)
Half-res blending...
Chroma smoothing...
Building alias map...
Filtering alias map...
Smoothing alias map...
Final blending...
Noise level : 8.00 (20-bit), ideally 8.00
Dynamic range : 15.96 EV (cooked)
AsShotNeutral : 0.53 1 0.59, 4793K/g=1.05 (gray max)
Output file : ./M11-1138_1_2015-03-11_0001_C0000_000000.DNG
./M11-1138_1_2015-03-11_0001_C0000_000000.DNG: copying EXIF from ./M11-1138_1_2015-03-11_0001_C0000_000000.dn6
grep: tee: No such file or directory
1 image files updated
find: *.dng: No such file or directory
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2 : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file : .
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2 : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file : ./M11-1138_1_2015-03-11_0001_C0000_000000.DNG
Cannot decode file .
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2 : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file : ./M11-1138_1_2015-03-11_0001_C0000_000001.dng
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2 : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file : ./M11-1138_1_2015-03-11_0001_C0000_000002.dng
Camera : Canon EOS 5D Mark III
Camera model : Canon EOS 5D Mark III
Full size : 2048 x 1152
Active area : 2048 x 1152
Camera : Canon EOS 5D Mark III
Camera model : Canon EOS 5D Mark III
Full size : 2048 x 1152
Active area : 2048 x 1152
Camera : Canon EOS 5D Mark III
Camera model : Canon EOS 5D Mark III
Full size : 2048 x 1152
Active area : 2048 x 1152
Black borders : N/A
Black borders : N/A
ISO pattern : BddB RGGB
ISO pattern : BddB RGGB
White levels : 10000 14882
White levels : 10000 14882
Doesn't look like interlaced ISO
cr2hdr: a post processing tool for Dual ISO images

Danne · « **Reply #2422 on:** March 11, 2015, 05:35:30 PM »

Could be exiftool related. I apply a copy of the xmp from orignal dng files in the first file which I use after conversion to get applied to the rest of the files. This information clearly must be corrupt.
Thanks for listening. I,ll check on it when I get the time.

exiftool ${BASE}_1_$(date +%Y-%m-%d)_0001_C0000_000000.dng -X > /tmp/magic_l_MLV/all.xml -overwrite_original ;

find . -name ${BASE}_1_$(date +%Y-%m-%d)_0001_C0000_000000.dng -print0 | xargs -0 /usr/bin/cr2hdr20bit | tee /tmp/magic_l_MLV/cr2hdr20bit.log ;

if grep -q "Edge-directed interpolation..." tee /tmp/magic_l_MLV/cr2hdr20bit.log
then

Files are here converted

I apply the information below from first file to the rest of the files.

exiftool -tagsfromfile /tmp/magic_l_MLV/all.xml -all:all ${BASE}_1_$(date +%Y-%m-%d)_0001_C0000_000000.DNG -exif:whitelevel -overwrite_original

Danne · « **Reply #2423 on:** March 11, 2015, 08:07:50 PM »

Hi!
I don,t know what I,ve been up to but is cr2hdr20bit keeping the metadata information from mlv dual iso movie dng even after conversion?? Has this always been? For RAW it,s hardcoded to 25fps after conversion. When I started messing with exiftool I,m certain also MLV files where stripped from metadata?
Is white/blacklevel same for movie files as well when filming MLV?

Tried an older build and sure it was hardcoding to 25fps. This is new stuff!! Great! I can erase a lot of stuff from the script.

DeafEyeJedi · « **Reply #2424 on:** March 11, 2015, 09:00:51 PM »

REALLY Interesting information... Thanks for pointing these out @Danne!

News:

Author Topic: Dual ISO - massive dynamic range improvement (dual_iso.mo) (Read 2156705 times)