Dual ISO - massive dynamic range improvement (dual_iso.mo)

[dfort]

A much longer explanation of what is going on is on Reply #2968. Looks like this is affecting 700D, 650D and 6D cameras depending (we suspect) on when and where they were manufactured.

[a1ex]

... depending on what's stored in their settings area (properties) and where that stuff happens to be allocated at startup. It can happen on any camera model, but some of them were just lucky.

Conspiracy theories do not apply here; one has to find where that data structure ends up in the memory without guesswork.

[dfort]

Conspiracy theories do not apply here...

The fix @theBilalFakhouri made on his 700D didn't work on Walter's 650D so yeah, we need a no guesswork solution.

Found where it's allocated, now to figure out where the result is stored. Or, maybe better, as I prefer to avoid messing with the property data structures - refactor with the crop_rec method for overriding these registers. That would require finding the cmos_write stub for all camera models.

Is this the same as the CMOS_WRITE_FUNC stub used in the adtg_gui module?

[a1ex]

Yeah, that's the one, but don't have any proof of concept ready yet.

[Levas]

Disabled all modules on my 6d (Made in Japan, it says on the bottom).
Only activated dual iso module.

Got the 'ISOless PH err(3)' error both in video mode and photo mode...
So it doesn't work in any mode.

Will try some other builds of magic lantern.

[Walter Schulz]

My 650D does ISOless PH err(2) and I have to restore ML defaults to get rid of it. Will appear again after using Dual-ISO.
Try before switching to other builds.
My 2 cents, that is.

[dfort]

@Levas - have you tried what @theBilalFakhouri suggested? It might be a "fix" until we can resolve this conspiracy--I mean find where that data structure ends up in the memory.

[Levas]

@Dfort, Will try it out when I have some time for it, doesn’t sound that complicated to try out.
Probably first will check ISO CMOS values with adtg_gui and what they are set to in sourcecode.

[nzfritz]

Hi all,

Awhile back I had downloaded what I need to take the .mov files and seperate the high and low ISO frames into seperate DNG files.  Now for the life of me I cant find the download link of the software I need to do that.  Possible for someone to point me in the right direction?

Thanks much!

[Walter Schulz]

You posted a question about HDR video (alternating ISO for odd/even frame number) into Dual-ISO (alternating ISO setting for every 2nd pixel line) thread. You're not the first one mixing up both techniques.
HDR described in User Guide is outdated and contains dead links mostly. Sorry! You may want to look into
https://www.magiclantern.fm/forum/index.php?topic=20025.msg201314#msg201314
https://www.magiclantern.fm/forum/index.php?topic=15271.0

[Sapporo]

Yes, I think it's the same way I did you can do, it's about CMOS 0 Registers at ISO 100, Some cameras have different registers than others for
the same model, (I don't know yet if it will be more than two Registers for same model, waiting 700d users to feedback)

So you have to see your registers on your camera by adtg_gui.mo (Assuming adtg_gui.mo is working on canon 650d)

1-Set ISO at 100
2-Set camera on movie mode 1080/24
3-After loading the modules, turn on adtg_gui in Debug submenu
4-Turn on dual_iso
5-Go to PLAY mode then go back to liveview
6-Check your CMOS 0 Registers data and write it down (In the white box)


7-Now go to photo mode
8-Take a picture and Check your CMOS 0 Registers data and write it down again.

 --> Now in dual_iso.c
This value for movie mode

        FRAME_CMOS_ISO_START = 0x********;

and this for photo mode:

        PHOTO_CMOS_ISO_START = 0x********;

Leave the (0x) and change the others to the values you wrote, Compile and test it in the camera!

Tried this with my 6D which I bought in Japan. Frame CMOS I changed to 40e6190 and photo CMOS I changed to 404e4e06. Still ISOless PH 3.  Only Magic-Lantern-4k crop mode has the issue. Not the Nightly build.

EDIT: It works! It was only me who didn't read everything. New values in CMOS 3 and not CMOS 0:

404e61b6
and
404e4e08

[fortymileswest]

I'm a new to Magic Lantern, but have been getting on well with it, so I thought I would try out the duel ISO.
I have just done some tests on my Canon 5D MK3. Recording at either 11-14bit lossless, 16:9, 1920x1080. I'm having issues though, it stops recording after a few frames and buffers. I have a Komputerbay 64GB 1000x card, which normally works fine at higher resolutions. Does duel ISO have a heavy hit on the CPU, or am I doing something wrong?

[nzfritz]

@Walter Schulz  Thank you so much !!

[Walter Schulz]

Thanks to dfort I'm now able to use ADTG_GUI with 650D.105!

Regarding a1ex's and dfort's post above in this page (and a1ex's posts some pages ago (quoted below)) I need some clarification: As far as I understood there is a structural deficit in dual_iso code. Tweaking value for PHOTO_CMOS_ISO_START will not be sufficient to get rid of ISOless error once and for all. It *might* work for a particuliar cam and not work for another.
Did I grab the concept, a1ex?

If I'm - somehow - correct (and if I'm not, too):
So, where to go from here? Anything we can do to support you?

Wait, what?! All these years, we've been writing in the middle of Canon's property data structures without even knowing about it?! Guess we were lucky, as I believe the FIX settings block is probably not written back to the permanent memory at shutdown... (one small piece added to this puzzle).

[...]

Found where it's allocated, now to figure out where the result is stored. Or, maybe better, as I prefer to avoid messing with the property data structures - refactor with the crop_rec method for overriding these registers. That would require finding the cmos_write stub for all camera models.

[QUAKE]

I wonder why is not possible to move more than x files over the cr2hdr-denoice.exe for processing (where x > 495) or you get a windows "The filename or extension is too long" error... someone have a solution?

Anyway around this issue?

[Audionut]

Anyway around this issue?

https://www.magiclantern.fm/forum/index.php?board=56.0

[zachnfine]

I made the mistake of processing a directory full of MLV files (using the excellent Switch.app on macOS) and then deleting the MLV files before checking the resulting DNG sequences. 4 of the DNG sequences were my first attempt at using dual ISO, and I did not have Switch set to enable dual ISO processing. So I have several DNG sequences that have the interlaced dark and light line look of unprocessed dual-ISO. I've checked my most recent backups, and unfortunately they were made before I copied these new MLV files to my RAID. It's a perfect storm of absent-mindedness on my part.

Do any of the available dual-ISO tools have the ability to process DNG sequences rather than MLV files? I figure the same information is right there in the DNG waiting to be utilized.

Thanks for any help or information.

[Danne]

You can drag a folder containing dng files onto Switch then open this in then main menu:
(d)  cr2hdr dualiso processing(CR2)
then select:
(r)  run Switch
Move on to the next dng folder when done.

Note that your dng files will be renamed and erased the in the process. Extension .dng will simply become .DNG so if you want to keep anything, make copies.

[a1ex]

Yet another academic paper mentioning Dual ISO:

Reconstructing Interlaced High-Dynamic-Range Video using Joint Learning

[theBilalFakhouri]

Experience enhancement:
Automatically disabling dual_iso in x10 zoom for better focusing in video mode ?

[a1ex]

Makes sense, yet another easy coding task

[a1ex]

Workflow tip: how to get a quick'n'dirty preview in the CR2 before the (time-and-space-consuming) cr2hdr step?

Method 1:

Code: [Select]

exiftool IMG_6706.CR2 -PreviewImage -b | convert - -resize 100%x25% -resize 100%x400% preview.jpg
exiftool IMG_6706.CR2 '-PreviewImage<=preview.jpg' # -overwrite_original
rm preview.jpg

Question for exiftool and/or bash experts: can this be done in one command, without a temporary file?

How it works: resizing on the Y axis gets rid of the stripes; resizing it back gives a nicer preview (could be useful when selecting a few "keepers" from a large number of files).

Run time: about 4 seconds, single thread on Core i7-7700HQ.

Method 2: use cr2hdr --fast (or --fast --fullres)

Code: [Select]

cr2hdr IMG_6706.CR2 --fast # or --fast --fullres
dcraw -b 16 -c IMG_6706.DNG | convert - preview.jpg
rm IMG_6706.DNG
exiftool IMG_6706.CR2 '-PreviewImage<=preview.jpg' # -overwrite_original
rm preview.jpg

Run time: about 9 seconds on the same system (or 20 seconds with full processing).

Of course, the above can be wrapped into scripts to process all files in one directory, tweaked to run in parallel and so on.

Pixel peeping:

- original preview from the CR2:


- with method 1, using the following resize factors:
  a) -resize 100%x50% -resize 100%x200%
  b) -resize 100%x33x33% -resize 100%x300%
  c) -resize 100%x25% -resize 100%x400%


- with method 2, using the following options for cr2hdr
   a) --fast
   b) --fast --fullres
   c) default (full processing)



For my own use, I've ended up with this:

Code: [Select]

#!/bin/bash

# Dual ISO preview
# To process all files from a directory, with GNU parallel:
#
#   ls *.CR2 | parallel dual_iso_preview.sh

# exit on error
set -e

# base file name, without extension
fn=$(basename $1 .CR2)

# backup old previews
dcraw -e $1 && mkdir -p original_previews && mv $fn.thumb.jpg original_previews/

# quick processing, just good enough for a half-res preview
cr2hdr $1 --fast --fullres
#dcraw -w -b 16 -c $fn.DNG | convert - $fn.jpg
ufraw-batch $fn.DNG --shrink=2 --exposure=auto --wb=camera --out-type=jpg --compression=70

# replace the preview with our own (caveat: it will delete the old preview!)
exiftool $fn.CR2 "-PreviewImage<=$fn.jpg" -overwrite_original

# cleanup
rm $fn.jpg
rm $fn.DNG

Make sure you've got backups before running this! (just in case)

[Danne]

Cool. Will check your latest tomorrow.

One fast way to extract previews is by using exiv2:

Code: [Select]

exiv2 -ep3 -l . *.{cr2,CR2}

Script below not exactly fast but will apply auto white balance(white balance not correct when dual iso) by roundtripping creating a hald clut and build a fake dng which dcraw happily takes to grab multipliers.

When I think of it we could easily get the exact white balance from the cr2 and apply it back to the preview jpg instead of using auto wb. Will check into this later.

Code: [Select]

#dependencies
#dcraw
#ffmpeg
#exiftool
#exiv2

exiv2 -ep3 -l . IMG_6706.CR2 && convert IMG_6706-preview3.jpg -resize 100%x25% -resize 100%x400% preview.jpg

ffmpeg -f lavfi -i haldclutsrc=8 -compression_algo raw -pix_fmt rgb24 -vframes 1 clut.tif
#create the reference tif
ffmpeg -i preview.jpg -compression_algo raw -pix_fmt rgb24 -vframes 1 reference.tif

#add needed dng tag. Will open up in dcraw but not in acr. Doesn´t matter, good old dcraw is all we need
exiftool -DNGVersion=1.4.0.0 -PhotometricInterpretation='Linear Raw' clut.tif reference.tif -overwrite_original

#now you have dcraw compatible dng /raw files
mv clut.tif clut.dng
mv reference.tif reference.dng

#add auto white balance
multiplier=$(dcraw -T -a -v c reference.dng 2>&1 | awk '/multipliers/ { print $2,$3,$4,$5; exit }')

#apply white balance and -H 2 to your hald clut
dcraw -v -T -H 2 -r $multiplier clut.dng

#here we go. Let´s apply auto white balance
ffmpeg -i preview.jpg -i clut.tiff -filter_complex '[0][1] haldclut' wb_preview.jpg
#done

exiftool IMG_6706.CR2 '-PreviewImage<=wb_preview.jpg' # -overwrite_original

#trash files
rm preview.jpg wb_preview.jpg IMG_6706-preview3.jpg clut.tiff clut.dng reference.dng

[ibrahim]

12bit MLV file  used(13mb) cut in Mlv app:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M18-1354.MLV

Hi. Since recording rates are getting better with lossless but also with using the sd_uhs it seems even more interesting to get dual iso movie recording working also with at least 12bit files. While 14bit really looks awesome filming with 3xzoom mode 12bits are still problematic(check stripes after conversion):


Checking dual iso code sources and it´s really some of the most impressive stuff I´ve seen. I don´t know where to start even looking for tweaks in there so I thought I´d post an example MLV if maybe Alex or somenone else wants to have a look.

Hi Danne,

Which crop-build do you use to record video in 14-bit 3x + dual iso? I have both 5d3 1.1.3 and 1.2.3.
I still haven't moved to the experimental builds and want to try 3x and dual iso.

[dfort]

EDIT: It works! It was only me who didn't read everything. New values in CMOS 3 and not CMOS 0:

404e61b6
and
404e4e08

Sorry to resurrect an old post but are you sure about this?

I'm bringing this up because I took a look at @Levas module and the changes for his camera were:

Code: [Select]

diff -r f4213dd747cd modules/dual_iso/dual_iso.c
--- a/modules/dual_iso/dual_iso.c
+++ b/modules/dual_iso/dual_iso.c
@@ -733,11 +733,11 @@
     {
         is_6d = 1;
 
-        FRAME_CMOS_ISO_START = 0x40452196; // CMOS register 0003 - for LiveView, ISO 100 (check in movie mode, not photo!)
+        FRAME_CMOS_ISO_START = 0x404e6196; // CMOS register 0003 - for LiveView, ISO 100 (check in movie mode, not photo!)
         FRAME_CMOS_ISO_COUNT =          7; // from ISO 100 to 6400
         FRAME_CMOS_ISO_SIZE  =         32; // distance between ISO 100 and ISO 200 addresses, in bytes
 
-        PHOTO_CMOS_ISO_START = 0x40450E08; // CMOS register 0003 - for photo mode, ISO 100
+        PHOTO_CMOS_ISO_START = 0x404e4e08; // CMOS register 0003 - for photo mode, ISO 100
         PHOTO_CMOS_ISO_COUNT =          7; // from ISO 100 to 6400 (last real iso!)
         PHOTO_CMOS_ISO_SIZE  =         18; // distance between ISO 100 and ISO 200 addresses, in bytes

The difference here is +0x94000 for both. On yours it is also +0x94000 for PHOTO_CMOS_ISO_START but it is +0x94020 for FRAME_CMOS_ISO_START.

Assuming this is an error in the post and the values changed by the same amount on both cameras we've got only two know possible values for the 6D. On the 700D we also have only two known possible values and the difference is +0x400 for both registers.

@Walter_Schulz -- A while back we were trying to get this working on the 650D but it seemed that camera would switch values. In addition, I assumed the offset was the same as the 700D, now I'm thinking it is probably different. Could you run the adtg gui module on your 650D following the instructions from @theBilalFakhouri? I'd like to see if we could figure out if CMOS register 3 switches between just two values.

I'd do it in QEMU but it doesn't show the CMOS registers.