Dual ISO - massive dynamic range improvement (dual_iso.mo)

[theBilalFakhouri]

@dfort I really don't know what makes these registers different. Maybe Canon did some tiny edits between the models or the country of manufacture?
Assuming if it true, my camera was made in Taiwan, what is yours? crazy theory.

When I adjusted the photo register I wrote it wrong by mistake, I got ISOless err(3), it was after correcting the video register and only dual_iso was working in video mode.

Okay, can we add these both registers in the code? If yes, this will be cool! , if not, so why? What is the problem exactly?

[a1ex]

1) the cause for this variation is unknown; are these two really the only possible addresses?
2) Canon firmware startup is non-deterministic; maybe it's that; maybe the startup mode or Canon settings will make a difference, no idea
3) if outside LV, the error can be probably reproduced in QEMU; if so, that can be used to solve the mystery.

[dfort]

Maybe Canon did some tiny edits between the models or the country of manufacture?
Assuming if it true, my camera was made in Taiwan, what is yours? crazy theory.

Not so crazy, my Canon EOS Rebel T5i (a.k.a. 700D/Kiss X7i) was made in Japan. I've got an MLV file you shared and see this in mlv_dump -v:

Code: [Select]

Block: IDNT
  Offset: 0x00000108
  Number: 3
    Size: 84
    Time: 3.970000 ms
     Camera Name:   'Canon EOS 700D'
     Camera Serial: '59310CBFB6' (383075008438)
     Camera Model:  0x80000326

While on my camera the same block looks like this:

Code: [Select]

Block: IDNT
  Offset: 0x00000108
  Number: 3
    Size: 84
    Time: 0.797000 ms
     Camera Name:   'Canon EOS REBEL T5i'
     Camera Serial: '775328882' (32031017090)
     Camera Model:  0x80000326

I'm pretty sure that ML is identifying cameras using the "Camera Model" hex code (in propvalues.h?) but maybe in this case a different identifier needs to be used?

[dfort]

3) if outside LV, the error can be probably reproduced in QEMU; if so, that can be used to solve the mystery.

Ran @theBilalFakhouri build on my 700D ROM in QEMU and got this:



Here is the QEMU log.

[theBilalFakhouri]

I got 0x4045368e on my 700D, just like you did:

@dfort
Sorry but it's different, look at the picture again you missed it.
It's 0x4045328e the same as dual_iso.c not 0x4045368e.

1) the cause for this variation is unknown; are these two really the only possible addresses?

Maybe yes, we are waiting for more users to make sure.

[dfort]

Oops--sorry. Looks like mine is the same as what is currently in modules/dual_iso.c and both of your addresses are offset by 0x400. Wonder if that's affecting anything else.

[dfort]

This is the theBilalFakhouri ROM dump from his Taiwanese 700D running the official current Nightly build from the Jenkins server in QEMU:

Issue related to ISOless message on 650D? Any advice how to corner this?

I believe that the 650D ROM I have is yours. It was saved with the language set to German. Are you seeing this using the current Nightly build?

[Walter Schulz]

I believe that the 650D ROM I have is yours. It was saved with the language set to German. Are you seeing this using the current Nightly build?

Tried my best to make this error appear but no avail with latest build (2018Feb04). Retried with the one before (2017Dec07) and got it again after power off and powering on again.
Issue solved, I suppose?

EDIT: And here we go again. Switched Picture Style to UserDef1 (Marvels Advanced 3.4) and err(2) appears.
EDIT2: Not that easy to reproduce. With former builds I had to reset ML settings to get rid of it. Now I have troubles to provoke the error. Not persistent anymore.

[dfort]

Ok, it looks like I can see the error in QEMU but can't get adtg_gui working in QEMU so if you could run the test theBilalFakhouri posted on your 650D maybe we can track this down. On the 700D the addresses for both FRAME_CMOS_ISO_START and PHOTO_CMOS_ISO_START were offset by 0x400 but when I tried it on the 650D it didn't work. Looks like once we get the CMOS[0] value from your camera using adtg_gui (on the modules downloads page) we should be able to "fix" it--at least for your camera. Need some help from a1ex for this to work on all cameras. So far it looks like it only affects the 700D and 650D but maybe there are others?

[Walter Schulz]

Unable to perform test.

Code: [Select]

tcc: error: undefined symbol 'unpatch_memory'
tcc: error: undefined symbol 'patch_hook_function'
  [E] failed to link modules

[dfort]

Are you using the crop_rec_4k build? I think the lua_fix branch also works. Just in case I'll post an iso-research branch build for you on my my downloads page.

[Walter Schulz]

@theBilalFakhouri & dfort:

Thank you for support and build but numbers shown with Adtg_gui.mo are identical to those in dual_iso.c.

Additional info: Error shows up with ISO200/800, too.

[theBilalFakhouri]

@Walter Schulz

Okay, you have followed all the steps as it is, now can you do it again with dual_iso turned off?

A question What is working with dual_iso on your 650D and what is not working?
e.g. The module is not working at all? or in some specific settings like in video mode only ? or at 100/400 is working fine but at 200/800 isn't working?

And also try this build based on crop_rec_4k_mlv_lite_snd branch and feedback please.

[dfort]

Wonder if the addresses change depending on which picture profile is selected.

Made another 650D QEMU run, this time on the qemu branch with some of the debugging goodies turned on. The "ISOless PH err (2)" is showing up consistantly here. Maybe this QEMU log will help?

https://pastebin.com/WPP9V90x

[dfort]

@theBilalFakhouri - I tried your build in QEMU and am still getting the "ISOless PH err (2)"

[Walter Schulz]

@Walter Schulz

Okay, you have followed all the steps as it is, now can you do it again with dual_iso turned off?

Dual ISO module loaded, not activated in Expo tab: Same registers.

A question What is working with dual_iso on your 650D and what is not working?
e.g. The module is not working at all? or in some specific settings like in video mode only ? or at 100/400 is working fine but at 200/800 isn't working?

Everything is working fine until err(2) occurs. With this error on screen cam will do "normal" stills without interlaced lines.
Error is not exclusive to 100/xxx. Happens in 200/xxx, too.

And also try this build based on crop_rec_4k_mlv_lite_snd branch and feedback please.

Unable to run adtg_gui.mo together with dual_iso.mo. Tried adtg_gui from Daniel's repository and a1ex/hudson. Later gives undefined symbol 'get_ms_clock_value' as you mentioned before.

Error happens but it's err(8) now!

[theBilalFakhouri]

Dual ISO module loaded, not activated in Expo tab: Same registers.

Sorry my fault I meant to activate it when I said "enable" this is important step. So try again and if you have tried it already and it didn't work:


Looks like the problem only happened in photo mode. In video mode it's working correctly with no problem right?

Error happens but it's err( 8 ) now!

I changed the two registers in the build to the registers on this page, maybe it's the same problem with you?

@dfort Can you add the same codes on the page to dual_iso.c for 650d?

Code: [Select]

//is_address_right must use after all other value setted
//despite the FRAME_CMOS_START and PHOTO_CMOS_START
//return 1  means true, 0 meas false
static uint8_t is_address_right(uint32_t address,uint16_t count,uint16_t size,uint16_t expected,char *model)
{
    if(streq(model,"650D"))
    {
        for(uint i=0;i< count;i++)
        {
            uint16_t raw=*(uint16_t *)(address+i*size);
            uint32_t flag= raw & 0x3;
        //if start_addr is right,all flag will be 11
        //else raw is 0x0 0x4046,flag is 0x0 or 0x10 bad address
            if(flag != expected)
                return 0;
        }
        return 1;

    }
    else
    {
        return 0;
    }
}

Code: [Select]

         *100 0x803 0x4049f144 or 244   
         *200 0x827 0x4049f154          0x24
         *400 0x84b 0x4049f164          0x24
         *800 0x86f 0x4049f174          0x24
         *1600 0x893 0x4049f184         0x24
         */
        is_650d = 1;   

        uint32_t photo_cmos_start_set[]={0x4049f144,0x4049f244};
        uint32_t frame_cmos_start_set[]={0x404a038e,0x404a048e};
        FRAME_CMOS_ISO_COUNT =          6;
        FRAME_CMOS_ISO_SIZE  =       0x22;

        PHOTO_CMOS_ISO_COUNT =          6;
        PHOTO_CMOS_ISO_SIZE  =       0x10;

        CMOS_ISO_BITS = 3;
        CMOS_FLAG_BITS = 2;
        CMOS_EXPECTED_FLAG = 3;

        //determin the PHOTO_CMOS_ISO_START VALUE
        if(is_address_right(photo_cmos_start_set[0],PHOTO_CMOS_ISO_COUNT,PHOTO_CMOS_ISO_SIZE,CMOS_EXPECTED_FLAG,camera_model_short))
        {
            PHOTO_CMOS_ISO_START = photo_cmos_start_set[0];
        }
        else
        {
            PHOTO_CMOS_ISO_START = photo_cmos_start_set[1];
        }
        //determin the FRAME_CMOS_ISO_START VALUE
        if(is_address_right(frame_cmos_start_set[0],FRAME_CMOS_ISO_COUNT,FRAME_CMOS_ISO_SIZE,CMOS_EXPECTED_FLAG,camera_model_short))
        {
            FRAME_CMOS_ISO_START=frame_cmos_start_set[0];
        }
        else
        {
            FRAME_CMOS_ISO_START=frame_cmos_start_set[1];
        }
I tried but I got errors when I compiled it because I don't have any coding skills yet.
Maybe it will solve the problem and if not:

I think the problem is while you are trying to get the register from photo mode because when I tried for the first time I got different register and it didn't solve the problem on my 700d, after the second try I got another register for photo mode at the same settings and it worked! (Maybe I changed something between the tries).

Try again: Disable all the modules from loading except adtg_gui & dual_iso
1-Go to photo mode without live view.
2-Enable dual_iso (ISO: 100 in canon menu 800 in dual_iso).
3-Enable adtg_gui.
4-Take normal .cr2 picture
5-If the photo still holding press half shutter then 6-Press trash to get back to ML and check CMOS 0

Try the same thing in other ways like take JPEG instead of .cr2, if nothing worked I really don't know what the problem exactly.

[Walter Schulz]

Sorry my fault I meant to activate it when I said "enable" this is important step. So try again and if you have tried it already and it didn't work:

Sorry, I'm confused. You wrote "dual_iso turned off" above. Now I have no idea what modules to load and (if module dual iso loaded) intended cam status.
1) Turning module dual_iso.mo off and redo register check with/without restart?
2) Loading module dual_iso.mo (ON), restart cam and activate or not activate or activate and deactivate Dual-ISO before checking registers?

Looks like the problem only happened in photo mode. In video mode it's working correctly with no problem right?

Can't say because I'm not that much into video and actually did Dual-ISO movie tests maybe once or twice.

I changed the two registers in the build to the registers on this page, maybe it's the same problem with you?

You want me to compile this, right?

[a1ex]

The issue can be reproduced with theBilalFakhouri's ROM in QEMU.

First step: what Canon code writes to this address range? (0x40452044 on the "good" firmware).

Code: [Select]

# this goes into 700D/debugmsg.gdb
# note: 0x40000000 is the uncacheable memory flag, so both addresses refer to the same physical memory location
watch *0x40452044
watch *0x452044

# run with:
./run_canon_fw.sh 700D,firmware="boot=1" -s -S & arm-none-eabi-gdb -x 700D/debugmsg.gdb

No results that way.

Second try: use our own logging framework (-d ram). The logs obtained that way would be huge, so add this into logging.c, eos_log_mem, right after the call to should_log_memory_region_verbosely:

Code: [Select]

    if ((addr & 0x0FFFFFF0) != 0x00452040)
    {
        return;
    }


Code: [Select]

./run_canon_fw.sh 700D,firmware="boot=1" -d ram |& grep -a -C 5 ram
...
[EEPROM-DMA]! [0x340000] -> [0x40450E00] (0xBB860 bytes)
[ram]   at Startup2:FF139C24:FF139C24 [0x40452040] <- 0xFF      : was 0x0; 8-bit
...

A-ha! This data is copied from the serial flash.

On the bad firmware:

Code: [Select]

[EEPROM-DMA]! [0x340000] -> [0x40451200] (0xBB860 bytes)

Alright, so next step would be to find out where 0x40450E00 or 0x40451200 is allocated, and how to find a pointer to it. Grep is usually good for finding needles in the haystack, so...

Code: [Select]

./run_canon_fw.sh 700D,firmware="boot=1" -d ram,io,calls,tail,sflash,debugmsg |& grep -a -C 5 -i 40450E00
...
   call 0xFF13AB98(340000, 40450e00, bb6dc, 100000)                              at [Startup2:ff12087c:ff123344]
    call 0xFF139FFC(340000, 40450e00, bb6dc, c0820404)                           at [Startup2:ff13abfc:ff120880]
[    Startup2:ff13a028 ] (00:01) [SF] readSerialFlashWithQuad Dest 0x40450e00 Src 0x340000 Size 767708
     call 0xFF139AE8(bd000, 40450e00, bc, ffffff3e)                              at [Startup2:ff13a3f4:ff13ac00]
      [SFDMA] at Start:FF139BE8:FF139BE8 [0xC0530060] <- 0x40450E00: Transfer memory address
[EEPROM-DMA]! [0x340000] -> [0x40450E00] (0xBB860 bytes)

All of the above addresses are good candidates to be looked up in the disassembly.

Stack trace for the call to readSerialFlashWithQuad:

Code: [Select]

./run_canon_fw.sh 700D,firmware="boot=1" -d callstack -s -S & arm-none-eabi-gdb -x 700D/debugmsg.gdb
...
(gdb) b *0xff13a028
...
Breakpoint 4, 0xff13a028 in ?? ()
(gdb) print_current_location_with_callstack

Current stack: [14bf60-14bb60] sp=14be88                                         at [Startup2:ff13a028:ff13ac00]
0xFF0C35B0(0, ff0c35b0, 19980218, 19980218)                                      at [Startup2:bba0:14bf58] (pc:sp)
 0xFF1232FC(14bf44, 1, 340000, 1000)                                             at [Startup2:ff0c3634:14bf28] (pc:sp)
  0xFF1206E8(450d3c, 450d64, 23b24, 0)                                           at [Startup2:ff123340:14bf10] (pc:sp)
   0xFF13AB98(340000, 40450e00, bb6dc, 100000)                                   at [Startup2:ff12087c:14bed0] (pc:sp)
    0xFF139FFC(340000, 40450e00, bb6dc, c0820404)                                at [Startup2:ff13abfc:14beb0] (pc:sp)

In particular, 0xFF1232FC is... PROPAD_CreateFROMPropertyHandle : FIX.

Wait, what?! All these years, we've been writing in the middle of Canon's property data structures without even knowing about it?! Guess we were lucky, as I believe the FIX settings block is probably not written back to the permanent memory at shutdown... (one small piece added to this puzzle).

Back to our quest: who allocates memory for this? There are some calls to alloc_dma_memory around, so let's trace them with gdb:

Code: [Select]

b *0x16e8c
commands
  silent
  print_current_location
  KRED
  printf "alloc_dma_memory(0x%x)\n", $r0
  log_result
  KRESET
  c
end

b *0xFF1232FC
commands
  silent
  print_current_location
  KRED
  printf "PROPAD_CreateFROMPropertyHandle(0x%x, 0x%x, 0x%x, 0x%x, 0x%x, 0x%x, 0x%x)\n", $r0, $r1, $r2, $r3, MEM($sp), MEM($sp+4), MEM($sp+8)
  KRESET
  c
end


Code: [Select]

(./run_canon_fw.sh 700D,firmware="boot=1" -d debugmsg -s -S & arm-none-eabi-gdb -x 700D/debugmsg.gdb) |& grep -a -C 50 -i 40450E00
...
[    Startup2:ff0c3634 ] PROPAD_CreateFROMPropertyHandle(0x14bf44, 0x1, 0x340000, 0x1000, 0x100, 0x100000, 0x1000000)
[    Startup2:ff1230a8 ] alloc_dma_memory(0x1000ff)
[    Startup2:ff1230ac ]  => 0x40450dc4
...
[    Startup2:ff13a028 ] (00:01) [SF] readSerialFlashWithQuad Dest 0x40450e00 Src 0x340000 Size 767708
...

Found where it's allocated, now to figure out where the result is stored. Or, maybe better, as I prefer to avoid messing with the property data structures - refactor with the crop_rec method for overriding these registers. That would require finding the cmos_write stub for all camera models.

Reference for the above commands: README.rst - Debugging with GDB.

[dfort]

Found where it's allocated, now to figure out where the result is stored. Or, maybe better, as I prefer to avoid messing with the property data structures - refactor with the crop_rec method for overriding these registers. That would require finding the cmos_write stub for all camera models.

Wouldn't that be the same as CMOS_WRITE_FUNC that is used in adtg_gui? Seems like adtg_gui worked for "fixing" the 700D belonging to theBilalFakhouri but we might be trying to hit a moving target with Walter's 650D.

[theBilalFakhouri]

@Walter Schulz

Okay sorry my fault again :

After loading the two modules which are dual_iso and adtg_gui only
In the first try, do it (the steps) with dual iso turned on in Expo menu.
In the second try, do it with dual iso turned off in Expo menu.

If the registers didn't changed, so then try to compile the codes from that page which I put it in the last post. And maybe your problem will be resolved!

@a1ex
Wow it's huge.
If I can help with my cam, just tell me that.

[a1ex]

@dfort: yes, it's the same function. If these functions become widely used, it would make sense to add them to stubs.S, along with some generic callbacks that other modules (dual_iso, crop_rec etc) could use to override these register values. Need to think a bit about it, as it's not a straightforward change (at least not for me).

Something like this would be also welcome for the ISO tweaks (to implement them in a way that doesn't conflict with existing features - something which was holding me back, besides the complexity of that topic).

[dfort]

The issue can be reproduced with theBilalFakhouri's ROM in QEMU.

Same with Walter's 650D ROM in QEMU.

[BeneM]

I have a 700D.115, purchased in Germany (as 700D.113). On my camera the two addresses also have an offset of 0x400.
With the nightly builds I always got "ISOless PH err(15)".
I changed the addresses accordingly => now dual iso works for me 

Maybe this helps...

[vicshift]

Tried in my 5d Mark II and I get these strong bands in photos
https://www.dropbox.com/s/2br0b39s5125lsv/Image%20139.png?dl=0