Dual ISO - massive dynamic range improvement (dual_iso.mo)

Started by a1ex, July 16, 2013, 06:33:50 PM

Previous topic - Next topic

0 Members and 4 Guests are viewing this topic.

Marsu42

Quote from: a1ex on June 03, 2014, 05:24:25 PMSo, if you set the camera on a tripod and shoot the same test scene at different exposure settings, you get different auto WB results?

I didn't do systematic tests, just noticed borked greymax awb with undexposed shots. If you didn't look into that much yourself I'll try to find the time and do a proper test series with greymax and greymed.

ShaunWoo

A1ex

bright dark detection error, iso blending didnt work

do you have any advice on that, i saw loads of pages back that you fixed it, but for your new version of cr2hdr 20 the error comes up

a1ex

Not yet, but confirmed the bug. Seems to be caused by extreme overexposure in the high ISO (that is, very little overlap to match them properly).

Marsu42

Quote from: Marsu42 on June 03, 2014, 05:26:34 PM
I didn't do systematic tests, just noticed borked greymax awb with undexposed shots. If you didn't look into that much yourself I'll try to find the time and do a proper test series with greymax and greymed.

I ran a test series with the dual_iso samples I uploaded earlier (https://bitbucket.org/Marsu42/ml-pull/downloads/dual_iso_samples_6d_60d_FIXED.zip) and graymax works ok with underexposure on these. Obviously it's really the algorithm failing on non-studio shots when a plain color area cannot be found. Knowing this, I'll use greymed in the future on my shots...

... but as an idea, maybe an smart auto-detection greymax/greymed would be possible if the tint is off the scale with greymax.

ShaunWoo

Quote from: a1ex on June 03, 2014, 07:06:49 PM
Not yet, but confirmed the bug. Seems to be caused by extreme overexposure in the high ISO (that is, very little overlap to match them properly).

ok thank you, may i ask if a fix is to be expected? i made the mistake on shooting a production in dual iso, even thought i tested it for a week still a new problem came about, and now i cannot continue it until this works, anyway i can help by any chance lol? researching seems to get me nowhere thought as im not a hacker or coder and cannot find any info on this either

but the thing that confuses me, shots, 0 and 1, compared to 2 and 3, are very similar, yet 0 and 1 dont process but 2 and 3 do

Audionut

Quote from: a1ex on June 03, 2014, 05:08:32 PM
I didn't notice major issues in my samples,

Another sample for your list.  --wb=graymed gets closer, but still a few thousand K wrong.  Somewhere around 7000-7500K is about right IMO.

https://dl.dropboxusercontent.com/u/34113196/ML/dual_iso/_46A5433.CR2

ShaunWoo

Hey guys, hope i aint being a nuisance, but any update on the "bright dark detection error, iso blending didnt work" bug?
im really desperate for a fix

or would it be possible if the developers let us know if a fix is to be expected anytime soon, because if not we can move on from waiting and continue with an alternate method



update and notes and tutorial for the un educated people on the following problems:
"bright dark detection error, iso blending didnt work"
flickering when filmed with dual iso

using the older version:
2014.04.08\cr2hdr-20bit.exe, doesnt give the "bright dark detection error, iso blending didnt work" error
whilst
2014.05.06\cr2hdr-20bit.exe, DOES
so apply same levels in post yourself, as cr2hdr same levels parameter uses exiftool to apply same levels anyway
processes the 0 and 1 files, and to apply same levels i used, ExifToolGUI, run the program, navigate to first file of the sequence, on the right find BlackLevel and WhiteLevel, note down the values, select all the files (i found that if i select around 2000 files it doesnt process the command so theres a file ammount limitation, 1430 worked, so maybe 2k is the limit?) at top right of the gui program, click on exiftool direct, copy and paste:

-Exif:BlackLevel=2046

NOTE, the 2046 is the variable, enter the BlackLevel you noted down from first file, click on ok, itll process the files, and create a backup of the original files, if you dont want them backed up, go to options at top left > dont backup files when writing.

now repeat the procedure for the whitelevel:

-Exif:WhiteLevel=2046

again the 2046 is the variable, click on ok, once both the whitelevel and black level have been changed to match the first file of the sequence, there should be no flickering what so ever, i dont believe ACR caused the flickering in the first place, as when i applied ACR adjustments, even to extreme cases, like -100 highlights, +100 shadows, etc, no flickering was caused what so ever

tron

Give Alex a break I bet he is already on that (judging from previous DUAL_ISO bugs).

In the meantime may I suggest to shoot alternating dual_iso and normal shots so at least to have a working raw file?

rpt

Quote from: tron on June 07, 2014, 02:31:40 AM
Give Alex a break I bet he is already on that (judging from previous DUAL_ISO bugs).

In the meantime may I suggest to shoot alternating dual_iso and normal shots so at least to have a working raw file?
+1
If you are unsure then do that. That is how I started shooting dual ISO. However, now in just shoot in dual ISO mode - no alternate files.

djronbxs

Hi,

I cannot convert files in DNG format. They are remaining as if they are not converted.

rpt

Quote from: djronbxs on June 08, 2014, 09:24:56 AM
Hi,

I cannot convert files in DNG format. They are remaining as if they are not converted.
Walk us through the steps. Make sure you list out each step as a bullet point and I will try to help.

However, before you do that, have you looked in the folder where the dualISO CR2 file was? If you exported right, the .DNG should be there. Go into windows explorer, navigate to that folder and look.

djronbxs

what I meant is that if I convert the file to dng with another software then i try to convert it from dual iso the photo is not being processed

Walter Schulz

Looks like a cr2hdr problem. @Mod: Please transfer offtopic to http://www.magiclantern.fm/forum/index.php?topic=7139.2150

We're no wizards here or paranormals. Give information to reproduce your problem.
- Program used to generate DNGs
- cr2hdr version used
- A link to a DNG having this problem
- Have you tried to run this file through cr2hdr?

And why do you have to create DNGs through this program?

Ciao
Walter


rpt

Quote from: djronbxs on June 09, 2014, 09:26:17 PM
what I meant is that if I convert the file to dng with another software then i try to convert it from dual iso the photo is not being processed
Why do you need to convert your dual ISO cr2 to dng using another program?

LB7D

Hi, I'm new to this forum, so please forgive me if this is not posted in the correct thread.

I just installed ML on my 7D, mainly to take advantage of the dual ISO feature.  Under relatively normal lighting conditions, it works fantastic; however last night, I was taking longer exposures on a tripod and noticed that all of my pictures taken with dual ISO have hundreds of random black marks scattered throughout the frame.  I thought this was related to the sensor overheating, so I ran some benchmarks with and without dual ISO and with and without Long Exposure Noise Reduction.  I figured the noise reduction would do the trick, but I was surprised to see that every photo taken in Dual ISO, regardless of whether or not noise reduction was activated, had these spots, all in the same places.  Exposures without Dual ISO turned on were fine at both the low and the high ISO, with and without LENR.

Is dual ISO just overwhelming my sensor?  Has anybody else had this issue?

Very interested to hear your thoughts.  Thanks.

a1ex

From the previous page:

Quote from: a1ex on May 31, 2014, 10:56:37 PM
Side effects noticed by me: there seem to be more hot pixels than usual during long exposures, and the ISO alternation confuses the feedback loop a bit.

About temperature: anybody can run an experiment and record it during a timelapse, for example, with and without dual iso; if there are differences, such a test should reveal them. I didn't run any temperature comparison, so I can't tell.

philmoz

Hope this is the right place to post this :)

I've been playing with dual-iso (5D3-123 branch) and noticed that the white balance Temp & Tint values in Lightroom (5.3) on the converted DNG file from cr2hdr did not match the original CR2 values.

After a bit of digging through the code, and playing with values I've found the following:
- the AsShotNeutral calculated using the RawMeasuredRGGB tag appears wrong, this tag gets used if the camera is set to Auto WB
- using the RedBalance & BlueBalance tags from the CR2 give the correct AsShotNeutral (as compared to a DNG converted using the Adobe DNG Convertor)
- using the WB_RGGBLevelsMeasured tag also gives the correct AsShotNeutral

The comments in exiftool-bridge.c state the RedBalance and BlueBalance values are not trustworthy - does anyone know why? They seem to work ok on my 5D3.

After fixing the AsShotNeutral value, the Temp and Tint were now closer to the original CR2 values in Lightroom; but still not exactly the same.

Looking more closely at the cr2hdr generated DNG file I noticed that only one color matrix is being saved - time to get the latest DNG code from CHDK ;)

After adding the second color matrix array, both calibration arrays, both forward matrix arrays and setting the correct value for illuminant1, I'm now getting consistent WB Temp and Tint values in Lightroom.

So far, the code changes to do this are in:
- src/raw.h (updated raw_info struct)
- src/chdk-dng.c (additional DNG tags)
- modules/dual-iso/cr2hdr.c (init raw_info)
- modules/dual-iso/dcraw-bridge.c (populate raw_info)
- modules/dual-iso/exiftool-bridge.c (get white balance)

Since there are changes to the core files I'm not sure what other impacts this may have - need to do some more testing.

Lastly, what's the preferred place to post patches for ML?

Phil.

ayshih

Quote from: philmoz on June 13, 2014, 01:27:46 PM
Hope this is the right place to post this :)
For relevant discussion and testing, go to http://www.magiclantern.fm/forum/index.php?topic=10265.0.

Quote from: philmoz on June 13, 2014, 01:27:46 PM
...
The comments in exiftool-bridge.c state the RedBalance and BlueBalance values are not trustworthy - does anyone know why? They seem to work ok on my 5D3.
...
I wrote the metadata-based white-balance estimation.    It's surprising to me that WB_RGGBLevelsMeasured is giving you a reasonable white balance; that did not appear to be true in general with the test shots (see the thread).  I don't understand how changing the matrices managed to make your white balance consistent, so I'm curious about your changes.

a1ex coded a raw-data-based white-balance approach (http://www.magiclantern.fm/forum/index.php?topic=10265.msg111203#msg111203), but it's only in the cr2hdr-20bit branch on Bitbucket, not in the unified branch.

Quote from: philmoz on June 13, 2014, 01:27:46 PM
Lastly, what's the preferred place to post patches for ML?
Submit a pull request to the repository on Bitbucket: https://bitbucket.org/hudson/magic-lantern
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

a1ex

Hehe, ayshih was faster :)

Indeed, you will want to try the cr2hdr-20bit branch, which includes some workarounds for this problem (both the exif approach from ayshih, and the custom WB algorithm from me).

To post patches, you can follow this tutorial: http://www.magiclantern.fm/forum/index.php?topic=7940.0
(in this case, you will probably want to submit your changes to the cr2hdr-20bit branch)

The problem with EXIF WB is that Canon's auto WB algorithm gets confused by dual iso. It's possible that some channels may be sampled from bright lines, and others from dark lines, and the exact mix may be camera-specific. However, if you can copy the manual WB from the CR2 to the DNG (which does not use Canon's auto WB algorithm), that would be a big improvement.

Modifying the raw_info structure will break the .RAW file format, which is not exactly desirable (there are still people who prefer it instead of MLV).

FOTOTO64

Hello
My coworker wants to 7D also installed Magic Lantern, but something we fail to install
I have a 550D and I have no problem with me
Of the instrument it is version 2.5 Firmware
Everything is done in the footsteps of the site
thank you
Sorry for the spelling I am not very good with English

philmoz

Quote from: ayshih on June 13, 2014, 02:25:27 PM
For relevant discussion and testing, go to http://www.magiclantern.fm/forum/index.php?topic=10265.0.

Many thanks, will continue the discussion there.
Ignore most of what I posted previously - seems I was chasing the wrong problem :)

Phil.

Canon Amateur

Quote from: FOTOTO64 on June 13, 2014, 03:06:08 PM
Hello
My coworker wants to 7D also installed Magic Lantern, but something we fail to install
I have a 550D and I have no problem with me
Of the instrument it is version 2.5 Firmware
Everything is done in the footsteps of the site
thank you
Sorry for the spelling I am not very good with English

A 7D with firmware 2.0.5 will not install ML.
Get the 7D back to firmware 2.0.3 and try again.
Canon EOS 7D / Canon 1D Mark III / Lytro illum

FOTOTO64


ShaunWoo


rpt

Quote from: djronbxs on June 09, 2014, 09:26:17 PM
what I meant is that if I convert the file to dng with another software then i try to convert it from dual iso the photo is not being processed
Why would you want to do that? The plugin does both the steps for you...