Author Topic: Dual ISO - massive dynamic range improvement (dual_iso.mo)  (Read 1584982 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12324
  • Emergencies only
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3025 on: January 27, 2019, 04:18:23 PM »
The solution is to avoid making assumptions about the addresses of these registers, i.e. to get rid of these hardcoded addresses. See reply #2968 and this topic. I find it hard to believe there are only 2 possible values on a given camera model. FRAME_SHUTTER_BLANKING constants should be probably handled in the same way (i.e. hardcoded addresses should be deleted).

The CMOS data is in the middle of Canon's property data structure, so playing around with prop_diag, or with Canon's property functions in QEMU, might reveal something. One possible step is identifying these properties (for example, by parsing a RAM dump around the known addresses with prop_diag).

Sapporo

  • Freshman
  • **
  • Posts: 83
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3026 on: January 27, 2019, 04:22:00 PM »
Sorry to resurrect an old post but are you sure about this?

I'm bringing this up because I took a look at @Levas module and the changes for his camera were:

Code: [Select]
diff -r f4213dd747cd modules/dual_iso/dual_iso.c
--- a/modules/dual_iso/dual_iso.c
+++ b/modules/dual_iso/dual_iso.c
@@ -733,11 +733,11 @@
     {
         is_6d = 1;
 
-        FRAME_CMOS_ISO_START = 0x40452196; // CMOS register 0003 - for LiveView, ISO 100 (check in movie mode, not photo!)
+        FRAME_CMOS_ISO_START = 0x404e6196; // CMOS register 0003 - for LiveView, ISO 100 (check in movie mode, not photo!)
         FRAME_CMOS_ISO_COUNT =          7; // from ISO 100 to 6400
         FRAME_CMOS_ISO_SIZE  =         32; // distance between ISO 100 and ISO 200 addresses, in bytes
 
-        PHOTO_CMOS_ISO_START = 0x40450E08; // CMOS register 0003 - for photo mode, ISO 100
+        PHOTO_CMOS_ISO_START = 0x404e4e08; // CMOS register 0003 - for photo mode, ISO 100
         PHOTO_CMOS_ISO_COUNT =          7; // from ISO 100 to 6400 (last real iso!)
         PHOTO_CMOS_ISO_SIZE  =         18; // distance between ISO 100 and ISO 200 addresses, in bytes

The difference here is +0x94000 for both. On yours it is also +0x94000 for PHOTO_CMOS_ISO_START but it is +0x94020 for FRAME_CMOS_ISO_START.

Assuming this is an error in the post and the values changed by the same amount on both cameras we've got only two know possible values for the 6D. On the 700D we also have only two known possible values and the difference is +0x400 for both registers.

@Walter_Schulz -- A while back we were trying to get this working on the 650D but it seemed that camera would switch values. In addition, I assumed the offset was the same as the 700D, now I'm thinking it is probably different. Could you run the adtg gui module on your 650D following the instructions from @theBilalFakhouri? I'd like to see if we could figure out if CMOS register 3 switches between just two values.

I'd do it in QEMU but it doesn't show the CMOS registers.



Code: [Select]
else if (is_camera("5D2", "2.1.2"))
    {
        is_5d2 = 1;
       
        PHOTO_CMOS_ISO_START = 0x404b3b5c; // CMOS register 0000 - for photo mode, ISO 100
        PHOTO_CMOS_ISO_COUNT =          5; // from ISO 100 to 1600
        PHOTO_CMOS_ISO_SIZE  =         14; // distance between ISO 100 and ISO 200 addresses, in bytes

        CMOS_ISO_BITS = 3;
        CMOS_FLAG_BITS = 2;
        CMOS_EXPECTED_FLAG = 3;
    }
    else if (is_camera("6D", "1.1.6"))
    {
        is_6d = 1;

        FRAME_CMOS_ISO_START = 0x404e61b6; // CMOS register 0003 - for LiveView, ISO 100 (check in movie mode, not photo!)
        FRAME_CMOS_ISO_COUNT =          7; // from ISO 100 to 6400
        FRAME_CMOS_ISO_SIZE  =         32; // distance between ISO 100 and ISO 200 addresses, in bytes

        PHOTO_CMOS_ISO_START = 0x404e4e08; // CMOS register 0003 - for photo mode, ISO 100
        PHOTO_CMOS_ISO_COUNT =          7; // from ISO 100 to 6400 (last real iso!)
        PHOTO_CMOS_ISO_SIZE  =         18; // distance between ISO 100 and ISO 200 addresses, in bytes

        CMOS_ISO_BITS = 4;
        CMOS_FLAG_BITS = 0;
        CMOS_EXPECTED_FLAG = 0;
    }
    else if (is_camera("50D", "1.0.9"))

dual_iso.mo https://drive.google.com/open?id=1IHa1ofE0ZdSWEj15Y6Xz4QB5fZrj3i1Z

I am using this setting with my old Japanese Canon 6D together with experimental build. I haven't updated the experimental build since I wrote that.
I am using the original setting with my new Swedish sold 6D together with nightly build.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3740
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3027 on: January 28, 2019, 01:08:40 AM »
@a1ex -- Understood but let me play in the shallow end a little longer before diving into the deep end.

@Sapporo -- Your module indeed has the values you posted but I'm still skeptical about your FRAME_CMOS_ISO_START address. Have you tried recording dual_iso raw video? I know this is something that not very many people do because of the aliasing issues--that is unless you're using the new experimental 1x3 crop_rec setting.

What I did was to change the FRAME_CMOS_ISO_START to a random value on the 700D and try it out. No error message. Then I put it back and did the same with the PHOTO_CMOS_ISO_START and got an 'ISOless PH err(2)' so it turns out that only a wrong photo mode value will trigger an error message.

EDIT: And here we go again. Switched Picture Style to UserDef1 (Marvels Advanced 3.4) and err(2) appears.
EDIT2: Not that easy to reproduce. With former builds I had to reset ML settings to get rid of it. Now I have troubles to provoke the error. Not persistent anymore.

Seems logical that something like activating/deactivating third party picture styles would cause things to move around in memory. It also seems reasonable to assume that the LiveView and Photo CMOS 3 register would change by the same amount. I'm not getting into conspiracy theories here--the right thing to do is to get rid of these hardcoded addresses.

By the way, the reason I looked into this again was because one of the show stoppers on the EOSM2 port is not being able to find the value for PHOTO_CMOS_ISO_START using the tools available in the adtg_gui module.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Sapporo

  • Freshman
  • **
  • Posts: 83
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3028 on: January 28, 2019, 06:47:56 PM »

@Sapporo -- Your module indeed has the values you posted but I'm still skeptical about your FRAME_CMOS_ISO_START address. Have you tried recording dual_iso raw video? I know this is something that not very many people do because of the aliasing issues--that is unless you're using the new experimental 1x3 crop_rec setting.

Failed with video and Dual ISO.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3740
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3029 on: January 28, 2019, 07:13:29 PM »
Failed with video and Dual ISO.

That what I suspected. Now try the module Levas posted:

I've had the same error, and made the changes in dual_iso.c, compiled the file and it works on my 6d, this is the dual_iso.mo file:
https://drive.google.com/open?id=1l_DgNZH6Gh_rqQeXOme-kV6XGrvMcMC1
Big chance it works on your 6d, but also a small chance that you still get the same error, since it appears to differ per model 6d  ???

Or -- use the value he found and compile it yourself:

Code: [Select]
FRAME_CMOS_ISO_START = 0x404e6196;
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Walter Schulz

  • Contributor
  • Hero Member
  • *****
  • Posts: 6904
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3030 on: January 28, 2019, 10:36:31 PM »
@Walter_Schulz -- A while back we were trying to get this working on the 650D but it seemed that camera would switch values. In addition, I assumed the offset was the same as the 700D, now I'm thinking it is probably different. Could you run the adtg gui module on your 650D following the instructions from @theBilalFakhouri? I'd like to see if we could figure out if CMOS register 3 switches between just two values.

Rusty I am. Last time I compiled ... boy, feels like another life ... had to check some VMs to find a proper configuration ... and failed kicking Err2 out of the system. Will try to redo that after some sleep!
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3740
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3031 on: January 29, 2019, 06:04:31 AM »
Low priority. We know that the values can change on the same camera. The 650D is a camera that it seems to happen more frequently than the others.

The right fix is getting rid of the hardcoded addresses. Haven't figured out how yet but we got some clues.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Sapporo

  • Freshman
  • **
  • Posts: 83
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3032 on: February 03, 2019, 09:53:31 PM »
That what I suspected. Now try the module Levas posted:

Or -- use the value he found and compile it yourself:

Code: [Select]
FRAME_CMOS_ISO_START = 0x404e6196;
Works. Thanks!

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3740
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3033 on: February 04, 2019, 01:57:52 AM »
Great--another piece of the puzzle solved. The LiveView and photo mode values change by the same amount.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

70MM13

  • Senior
  • ****
  • Posts: 292
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3034 on: February 19, 2019, 08:00:01 PM »
Using Danne's latest stable 5d3 build, I'm getting these vertical stripes on mid tones.
The lower the exposure, the more prominent the effect.
Provided is a well exposed night scene, and you can see the effect on the wall.  I exaggerated the processing in mlvapp to showcase the issue, the receipt is included along with the MLV in the zip.
It's not a fatal flaw, especially at higher exposures, but it really shows at lower exposures, the kind I typically use.
I provided this well exposed example to show it isn't just a problem in the darkest case scenario.  This is a definite issue when exposing for highlights.

It looks the same in resolve also...

https://drive.google.com/open?id=13N3-9q8o9GmfuUuEI5-9GkKrOhfdfLFc




histor

  • New to the forum
  • *
  • Posts: 49
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3035 on: February 19, 2019, 09:06:25 PM »
I've just checked another approach: MLVProducer with DualISO processing (high-iso option). The result looks perfect.

70MM13

  • Senior
  • ****
  • Posts: 292
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3036 on: February 20, 2019, 01:18:35 AM »
Unless I'm mistaken, choosing that option negates dual iso and simply uses the high iso half, cutting the vertical resolution in half...

histor

  • New to the forum
  • *
  • Posts: 49
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3037 on: February 20, 2019, 10:04:55 AM »
Absolutely (sorry for the messy post).
So the stripes come from the underexposed low ISO layer. That's just a usual noise pattern (like in pushed up darkframe) exaggerated while scaling. I've blended them manually, using low-iso layer for extreme highlights only.

70MM13

  • Senior
  • ****
  • Posts: 292
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3038 on: February 20, 2019, 02:11:42 PM »
right...

so i need to find a way to get this look without the stripes.  i'm playing with different ISOs now to see what improvement can be made in that regard, but it would be nice to have a simple way to adjust the blending to minimize this also.

it would be beneficial to be able to do this interactively, and not via a command prompt with a pile of numbers, rendering blindly just to see what the settings do.  is there such an option out there?

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3039 on: April 17, 2019, 08:18:43 AM »
Seems that cr2hdr don't work on old versions of osx.
I've tried it in 10.7.5 and gives error:
Code: [Select]
Illegal instruction: 4

Is compiled on 10.13.6 and here it works ok.
Maybe is because of clang?
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1412
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3040 on: April 17, 2019, 01:12:54 PM »
@scrax: Compatibility from a newer to an older OSX is often very bad. Better you compile on your old(est) system.
5D2.212 | EOSM.202

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3041 on: April 17, 2019, 03:11:48 PM »
@scrax: Compatibility from a newer to an older OSX is often very bad. Better you compile on your old(est) system.
Yes, and it's really very old version :P
It's a core2duo with two Osx versions, the oldest is a backup of like 6 years ago and is not important to have cr2hdr working, was just a curiosity.
Problem is that even the last supported Osx can't install all deps for compiling ML (problems with python iirc) I think it is 10.10.something

But it works ok on High Sierra so I'm good for now.

Good news is that I've added it to MLTools and also update it with other updated utils. and ML files like XMP, UFR and so on. Only thing left to convert so are MLV files (but not sure if is needed, there are better option and not being a video isn't helping).

(as a reminder, had to modify vid2hdr.sh for make it work ok with mplayer).
This the new line (a couple of options seem to have changed name):
Code: [Select]
opts="vbitrate=12150000:mbd=2:keyint=132:v4mv:vqmin=3:o=luma_elim_threshold=-4:o=chroma_elim_threshold=7:lumi_mask=0.07:dark_mask=0.10:naq:vqcomp=0.7:vqblur=0.2:mpeg_quant"
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2102
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3042 on: August 22, 2019, 06:22:15 AM »
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dual_iso_video-updates_8-21-2019.zip
updated 5D2 with dual-iso video , works in 3x3 (FHD 1856x1248) & 5xZoom (3xcrop_rec 2144x1074) @ 10bit raw

myown

  • New to the forum
  • *
  • Posts: 23
  • 5D Mark IV / 60D
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3043 on: August 28, 2019, 08:10:14 AM »
Very interesting Canon patent :D

https://www.canonrumors.com/patent-expanded-dynamic-range-using-dpaf-sensors/

"It should be noted that the patent does mention in-vehicle sensors, but can also be applied to any other type of sensor as well"

...pity that they only want to use it for cars



Igor_Braun

  • Freshman
  • **
  • Posts: 97
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3044 on: August 29, 2019, 05:52:16 PM »
@Reddeercity. Thanks. But could you tell how you process the video? In my case, i got the dual iso video more noisy in shadows the regular video using mlv app.
But when i take photos and use cr2hdr the result is stunning.

In the first page i found:

mlv_dump clip.mlv --dng --no-fixcp --no-stripes
cr2hdr --same-levels *.DNG

So, how to do it using windows? I have cr2hdr, but don't know how to open the terminal.

Walter Schulz

  • Contributor
  • Hero Member
  • *****
  • Posts: 6904
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3045 on: August 29, 2019, 06:10:45 PM »
AFAIK Windows version does not support wildcards, at least ATM. You won't be able to use it this way and command line length is limited in Windows. 2047 or 8191 characters, haven't tested cr2hdr limit yet.
For "terminal": In Windows terms it may be called "Command prompt", "command line", "CLI" (command line interface), "cmd" ... Startup search engine of your choice ...
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

Igor_Braun

  • Freshman
  • **
  • Posts: 97
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #3046 on: August 30, 2019, 10:49:30 PM »
Understood, thanks