Dual ISO - massive dynamic range improvement (dual_iso.mo)

Started by a1ex, July 16, 2013, 06:33:50 PM

Previous topic - Next topic

0 Members and 3 Guests are viewing this topic.

a1ex


DeafEyeJedi

5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Audionut

Quote from: a1ex on March 02, 2015, 12:32:02 AM
The second one should be marked as blown out, right? It's a white level problem (not detected correctly if you decrease the gain too much).

Noticed this issue today without any ADTG changes. 

Is this something I might be able to diagnose with a little nudge in the right direction?

a1ex

If you upload a CR2, I'll be able to load it in QEMU and see what happens. Here we need a pretty good heuristic, because once the gain drops too much, white is no longer clipping to a constant value - there's noise instead. The white level should be right below this noise in the clipped areas.

Something like this:


Our heuristic should figure out the white level from a single image (from histogram data), not from a nice graph like this. Not exactly an easy problem.

Audionut

https://dl.dropboxusercontent.com/u/34113196/ML/dual_iso/_MG_2788.CR2



https://dl.dropboxusercontent.com/u/34113196/ML/dual_iso/_MG_2789.CR2



Just to confirm, no ADTG changes (module not even loaded).  Base ISO 100 shot above, and then dual ISO image below (no other changes).

On closer inspection, there are some subtle changes in the histogram data also.

mothaibaphoto

Great thanks for fixing this "Hot" (cold, actually) pixels issue. I did some pixel peeping and did not find any IQ degradation. Really impressive done, bravo !!!

Original:


Fixed:


keepersdungeon

Gd job on fixing the Hot pixels.

I have a stupid question though, I don't seem to be able to subscribe to this discussion, is there a way to do it ? or i just don't have the permissions?


Danne


keepersdungeon




N/A

7D. 600D. Rokinon 35 cine. Sigma 30 1.4
Audio and video recording/production, Random Photography
Want to help with the latest development but don't know how to compile?

DeafEyeJedi

5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

N/A

7D. 600D. Rokinon 35 cine. Sigma 30 1.4
Audio and video recording/production, Random Photography
Want to help with the latest development but don't know how to compile?


keepersdungeon

I got confused a bit with all the posts so I apologize in advance for my stupid question. But is the hot pixel fix due to cr2hdr 20bit update or is the dualISO module has been updated?

rpt


Danne

Hi!
There,s one thing puzzling me with dual iso RAW and MLV. I compared the formats recording with a recent build, (about two weeks old, don,t have my camera here)

WHen shooting RAW all is fine giving me a file a little darker as expected. WHen shooting with exact same settings with MLV the file is brighter after postprocessing. It,s like it,s inverted? Am I doing an obvious mistake here?
examples shot in 100/1600 3x crop mode. Same result when shooting non crop mode. Will upload RAW/MLV examples if needed. (Short ones).

Converted with latest cr2hdr20bit build Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:.
The same goes for the former build. Shot with a 5d mark 3, firmware 1.1.3


DNG files
https://drive.google.com/file/d/0B4tCJMlOYfirYWl6WmF2ME9laWM/view?usp=sharing
3x crop RAW 100/1600


3x crop MLV 100/1600

a1ex

Can you upload the source DNGs created by raw2dng / mlv_dump?


a1ex

I've got two DNGs with similar brightness.

Can you paste the conversion log?

Danne

It seems the problem is coming when I,m inside folders and letting c2hdr-r working in different levels. Therefore the conversion process works from parent folder. I have to check further. A little busy atm.

*Actually looks like it is only the first file that is giving strange results.

Here is a log from dual iso conversion. Also noticed some strange intermediate issue before it turns into a transcoded DNG (picture).




Log from faulty conversion dng files

Last login: Wed Mar 11 17:23:10 on ttys000
itgusercbook241:~ itguser$ /Users/itguser/Desktop/testMLV\:RAW/cr2hdr-r\ 5.3k_s.command ; exit;
ORIGINALS

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M11-1138.MLV'
   - Convert to DNG frames
   - Output into 'M11-1138_1_2015-03-11_0001_C0000_'
File M11-1138.MLV opened
File M11-1138.M00 not existing.
Processing...

Vertical stripes correction:
  1.000  1.000  1.000  1.002  0.999  0.996  0.997  0.999
Reached end of chunk 1/1 after 56 blocks
Processed 39 video frames
Done

mv: rename M11-1138_1_2015-03-11_0001_C0000_.wav to M11-1138_1_2015-03-11_0001_C0000.wav: No such file or directory
mv: rename M11-1138.M* to ../ORIGINALS/M11-1138.M*: No such file or directory
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge    : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2         : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax    : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file      : ./M11-1138_1_2015-03-11_0001_C0000_000000.dng
Camera          : Canon EOS 5D Mark III
Camera model    : Canon EOS 5D Mark III
Full size       : 2048 x 1152
Active area     : 2048 x 1152
Black borders   : N/A
ISO pattern     : BddB RGGB
White levels    : 10000 14882
Noise levels    : 8.00 8.00 8.00 8.00 (14-bit)
ISO difference  : 3.89 EV (1487)
Black delta     : 5.26
Dynamic range   : 9.96 (+) 10.65 => 14.54 EV (in theory)
Looking for hot/cold pixels...
Hot pixels      : 34
AMaZE interpolation ...
Amaze took 0.30 s
Edge-directed interpolation...
Semi-overexposed: 55.57%
Deep shadows    : 4.55%
Horizontal stripe fix...
Full-res reconstruction...
ISO overlap     : 3.1 EV (approx)
Half-res blending...
Chroma smoothing...
Building alias map...
Filtering alias map...
Smoothing alias map...
Final blending...
Noise level     : 8.00 (20-bit), ideally 8.00
Dynamic range   : 15.96 EV (cooked)
AsShotNeutral   : 0.53 1 0.59, 4793K/g=1.05 (gray max)
Output file     : ./M11-1138_1_2015-03-11_0001_C0000_000000.DNG
./M11-1138_1_2015-03-11_0001_C0000_000000.DNG: copying EXIF from ./M11-1138_1_2015-03-11_0001_C0000_000000.dn6
grep: tee: No such file or directory
    1 image files updated
find: *.dng: No such file or directory
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge    : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2         : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax    : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file      : .
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge    : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2         : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax    : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file      : ./M11-1138_1_2015-03-11_0001_C0000_000000.DNG
Cannot decode file .
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge    : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2         : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax    : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file      : ./M11-1138_1_2015-03-11_0001_C0000_000001.dng
cr2hdr: a post processing tool for Dual ISO images

Last update: cc4340b on 2015-03-01 23:16:03 UTC by a1ex:
cr2hdr: fix parsing of --wb options

Active options:
--amaze-edge    : use a temporary demosaic step (AMaZE) followed by edge-directed interpolation (default)
--cs2x2         : apply 2x2 chroma smoothing in noisy and aliased areas (default)
--wb=graymax    : set AsShotNeutral by maximizing the number of gray pixels (default)

Input file      : ./M11-1138_1_2015-03-11_0001_C0000_000002.dng
Camera          : Canon EOS 5D Mark III
Camera model    : Canon EOS 5D Mark III
Full size       : 2048 x 1152
Active area     : 2048 x 1152
Camera          : Canon EOS 5D Mark III
Camera model    : Canon EOS 5D Mark III
Full size       : 2048 x 1152
Active area     : 2048 x 1152
Camera          : Canon EOS 5D Mark III
Camera model    : Canon EOS 5D Mark III
Full size       : 2048 x 1152
Active area     : 2048 x 1152
Black borders   : N/A
Black borders   : N/A
ISO pattern     : BddB RGGB
ISO pattern     : BddB RGGB
White levels    : 10000 14882
White levels    : 10000 14882
Doesn't look like interlaced ISO
cr2hdr: a post processing tool for Dual ISO images

Danne


Could be exiftool related. I apply a copy of the xmp from orignal dng files in the first file which I use after conversion to get applied to the rest of the files. This information clearly must be corrupt.
Thanks for listening. I,ll check on it when I get the time.


exiftool ${BASE}_1_$(date +%Y-%m-%d)_0001_C0000_000000.dng -X > /tmp/magic_l_MLV/all.xml -overwrite_original ;


find . -name ${BASE}_1_$(date +%Y-%m-%d)_0001_C0000_000000.dng -print0 | xargs -0 /usr/bin/cr2hdr20bit | tee /tmp/magic_l_MLV/cr2hdr20bit.log ;


    if grep -q "Edge-directed interpolation..." tee /tmp/magic_l_MLV/cr2hdr20bit.log
    then
   
Files are here converted

I apply the information below from first file to the rest of the files.

exiftool -tagsfromfile /tmp/magic_l_MLV/all.xml -all:all ${BASE}_1_$(date +%Y-%m-%d)_0001_C0000_000000.DNG -exif:whitelevel -overwrite_original

Danne

Hi!
I don,t know what I,ve been up to but is cr2hdr20bit keeping the metadata information from mlv dual iso movie dng even after conversion?? Has this always been? For RAW it,s hardcoded to 25fps after conversion. When I started messing with exiftool I,m certain also MLV files where stripped from metadata?
Is white/blacklevel same for movie files as well when filming MLV?

:o Tried an older build and sure it was hardcoding to 25fps. This is new stuff!! Great! I can erase a lot of stuff from the script.

DeafEyeJedi

REALLY Interesting information... Thanks for pointing these out @Danne!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109