CMOS/ADTG/Digic register investigation on ISO

[a1ex]

Two minor bug fixes for adtg_gui (actually in the menu backend):
- registers can now be edited while recording H.264
- no more crash when editing large values (such as 0x80000000)

HTH.

[timbytheriver]

here's an excellent example of the benefits of amplifier gain optimisation:
...
this REALLY needs to become mainstream.now.please!

@70MM13 With results like these I totally agree!  But perhaps uptake is slow because people (like me  ) are seriously baffled as to how to even begin testing this – and I've read through the entire thread **EDIT And I'm currently re-reading it** – and consider myself a seasoned ML user!

Could a user of these modules maybe make a tutorial post/video on: which modules to install, how to safely tweak the registers, (lol) etc. ? It would be tremendously helpful and might encourage more people to test, feedback, and develop these proof of concepts into a mainstream module.

Rock this!

[scrax]

Started today to take a look at this on 600D, for now I'm looking for differeces in changing ISO and got this so far:

Changed Registers	ISO100	ISO200	ISO400	ISO800	ISO1600	ISO3200	ISO6400
CMOS[0]	0	24	48	6c	90	B4	B4
CMOS[1]	423	423	423	423	423	443	443
ADTG2[8830]	1	1	1	1	1	3	3
ADTG2[8882]	417	413	427	41A	42E	453	8A6
ADTG2[8884]	417	413	428	41C	42F	454	8A8
ADTG2[8886]	41e	419	42a	41D	42D	44D	89A
ADTG2[8888]	41c	417	429	41B	42B	44A	894

Other notes:
CMOS[0] confirmed chaging ISO (will resut in horizontal stripes if set wrong)
CMOS[1] is changing exp, but can't understand what is, is set at 423 normally and 443 at highest ISO, if I set to 123 got a dark image like 4 less stop, at 223 or more than 200 i can't see any difference
CMOS[2] is set to 0x18 if I change it from 0x00 to 0xBF can't see any difference, then after 0xC0 I got totally black pics (with noise)
CMOS[5] (is at 140 by canon) set under 40 gives black image, but from 40 going up I have some funky color in shadows going green and lights going pinkish, so seems not related to vertical offset on 600D
CMOS[6] shifts images up if set to more than 800 (810 is already half pic ok, bottom half black) less than 800 is totally black image , so this could be vertical offset on 600D?
CMOS[7] not sure it starts at 1 and at 21 shows only a little lines of pixels captured on left side of image, other values gives black pic if it don't finish in 1

[scrax]

@70MM13 With results like these I totally agree!  But perhaps uptake is slow because people (like me  ) are seriously baffled as to how to even begin testing this – and I've read through the entire thread **EDIT And I'm currently re-reading it** – and consider myself a seasoned ML user!

Could a user of these modules maybe make a tutorial post/video on: which modules to install, how to safely tweak the registers, (lol) etc. ? It would be tremendously helpful and might encourage more people to test, feedback, and develop these proof of concepts into a mainstream module.

Rock this!

I think I'm in your same situation, so far to find the code for my 600D I've put ML iso research branch on a card and started it up, then i did those steps:
- enabled adtg_gui and raw_diag in modules,
- set iso to 100 and restarted camera by opening and closing card door
- in Debug->ADTG Register set the show option to "Modified since ..."
- take a pic at 100 ISO
- change ISO to 200 and take another pic
- go in ML->Debug->ADTG Registers and you will see the registers changed from 100 to 200 iso (for example i got for CMOS[0]: 24 (was 0) so 24 is the actual value at 200ISO and 0 the value that was set at 100ISO
- note down the values and proced to the next: iso 400, set it take pic and return to ADTG registes to see the new values and go on like that for all the ISOs

Just to be more clear If you start camera with 100ISO, take a pic and look in ADTG Registers you will not see nothing changed (since values started from boot with 100 ISO, that's why at first you should take two consecutive pic at 100ISO and 200ISO to see the differences from those two...

Now I'm gonna try to change something to see if makes differences and maybe this wil hel me also in understanding what all this topic is talking about.
I'm thinking that a lot of stuff is more hard to understand by only reading the forum and instead by making some test while following the topic helps a lot in understand it better

Now to start play I've set 8882 to 0 take a pic and got like banding on the pic set also 8884 to 0 and the banding gone away, that explained me what it mean the help with: "ISO ADTG gain (per column, mod 4 or mod 8 )"
If I override also 8886 and 8888 to 0 I get black pics and if I set 8886 and 8888 to 0 and 8884 8882 leave untouched I got one pic pic with only red channel

so next step will be set up a scene with high DR take a couple of pic with original values and canonc values and try to see if the overrides gave better results or not

[timbytheriver]

Ok, so after another read I've made a start following @70MM13 hints and @a1ex's info and screenshots. So far I think I'm missing a piece of the puzzle:

I'm documenting this as a trail for others attempting to follow in my intrepid footsteps into the dark wood ahead… 

1) I've downloaded the ISO Research tools from here: https://builds.magiclantern.fm/modules.html

2) I've added the iso_regs.mo into my modules folder of @Danne's crop_rec_4k_mlv_snd_isogain_1x3_presets_2019Apr02.5D3123.zip  build here: https://bitbucket.org/Dannephoto/magic-lantern/downloads/  (I understand I should load *either* iso_regs *or* adtg_gui.mo but not both! They provide similar functionality?)

3) I've enabled the iso_regs.mo

4) Gone into ML menu Debug > ISO Registers >

5) Now I've followed @70MM13's following instructions [brackets mine]:

"I have a very simple method based on my own experiments following a quick reading of a1ex's information contained within this thread:

1: clear the custom iso settings [I take this to mean if there are already changed values here from a previous shot?]
2: set desired base iso (canon menu) [Easy – I tried ISO200]
3: take a silent photo [Having trouble making this work, option is greyed out even when turned ON. I took a conventional 'shutter' pic instead]
4: "copy canon settings" [Pressed this and some values filled the page]
5: set 0xfe to 3, preamp to 2, and reduce adtg until the "iso" is half of the base value [I set CMOS Gain (0xfe?) to 3, ADTG preamp to 2,  ADTG Gain to 100 (half of ISO200)]

That's it!"

6) I come out of that menu and shoot a short mv1080 clip of an HDR scene making sure to avoid clipped highlights but to include darkest shadows.

7) The clip records – but the liveview has smeary wide vertical green bands on it. These look very scary – but fade after a couple of shutter presses – and totally clear after I disable the Debug > ISO Registers > to OFF [BTW Do these values reset to default when I do this?]

I examine the [dark] clip in MLVApp1.6 and adjust the dark strength slide to recover the shadows. I adjust the RAW White value to tame the pink highlights. Overall result: Ugly coloured noise in the shadows! 

Am I on the right track here?

Thanks!

@scrax Just seen your post! Looks like you have loaded different ISO Research modules to mine? 

[70MM13]

i'd be overjoyed to see more users play with this!

i don't really have anything new to offer in terms of tips, everything i'm doing is still pretty much the same as what i wrote before on this thread, but in terms of experience, i have another useful tidbit:

for still photography this is also quite useful.  i have been doing some long exposures for macro and night sky photographs, and the noise floor is so nice with the gain reduction that it is a joy to work with.

this same principle would be very good for dual iso as well, reducing the noise on the high iso portion.

plus, for extremely bright highlights, we could have iso 50 or even 25.  no ND filter required!

there's a lot that can be explored here, it just needs enthusiastic explorers!

these days i am using this mode alongside dual iso with the excellent binning.  between the two modes, everything i need to do is attainable!

this is the best video i have done so far using only this iso "109" mode, every single shot with only iso "109" on my 5d3 at 3k...

[70MM13]

@timbytheriver:

if you turn it off, it no longer works.  you're looking at native canon iso.

the nastiness you are seeing fades away after 30 seconds and then you're ready to record.

BTW: lately i have been lazy with the settings here, and still getting great results.

here's what i do these days:

start with clear settings
set iso to 200
take a regular still photo
copy settings
set ADTG preamp to 2
set ADTG 0xFE to 0

done!

[timbytheriver]

@70MM13 Hey!

Thanks for the response. Hope that green liveview isn't a fried sensor! 

The only module I have loaded is iso_regs.mo. Is this and my subsequent actions correct? 

Will try another test shortly with your updated settings.

[scrax]

@scrax Just seen your post! Looks like you have loaded different ISO Research modules to mine? 

I've got the iso research branch 2019mar23 for 600D and I didn't added the module from Danne because iso-regs works only for 5D if I understand right, so I'm using adtg_gui and not iso_reg as module to edit adtg that's why you have different stuff in your module

[70MM13]

@timbytheriver
yes, for this simple "use it only" mode, iso_regs is all you need to load.

here's an example image i took using liveview silent capture using those settings, and i intentionally snapped it before the settling was finished, giving interesting stripes in the shadows as an effect!

fun stuff...

https://drive.google.com/open?id=1UVLHp0_qq1t_xfsCmkp47A0QfBL4H4uU

[scrax]

by the way... what is:
badcaffe

[timbytheriver]

@scrax I understand - thanks for the clarification! Good luck!

@70MM13 That smearing used as an effect looks edgy! So far with those settings starting out at ISO200 > 100 (ML) I get these 100% crops from MLVApp:

Spot before and after – not getting much improvement yet!  Is the 'take a silent picture' a necessary step, or can it be just a regular RAW/Jpeg shot in liveview mode?

[70MM13]

any capture should work, i think...  i use it in stills mode also...

the difference shows up when you push the shadows.  play around and see what you can come up with.

i look forward to people making progress with this.  i am lazy and just do what works for me to get cleaner shadows.  there's probably a lot more just waiting to be discovered!

[timbytheriver]

@70MM13 Thanks, but I'm wondering whether it's even working on my cam! 

When you take the 'reference' pic, are you ETTR or ETTL ? or in the middle?

[70MM13]

you'll know it's working when you see the liveview get messed up and slowly clean up again...

if you look at the final shot of the video i linked above, you can see it happening.  i used it intentionally as an effect.  i added the "alien" writing scrolling up through part of the actual amplifier settling process!

as for the reference photo, i don't know if it matters?  i have never been careful about it.  maybe it matters, but i have no idea!

are you doing this all in video mode?  i do...

[70MM13]

i don't know if this makes a difference, but i do this in 5x zoom mode for 3k recording...

part of this seems to involve stumbling around to get the right order of operations it seems.

also, i'm using a version from december 2017.  somewhere earlier in this thread i posted a copy of it for anyone to grab...  maybe it would help?

again, i have no idea!

[timbytheriver]

@70MM13 Yes, video mode. Non-crop so far!

I love your suck-it-and-see style! Will keep playing… 

[timbytheriver]

by the way... what is:
badcaffe

I don't know! What is badcaffe? 

[timbytheriver]

ISO 109 tested for shadow noise improvement:

Three images, all settings identical except iso.  Exposed for highlights in a low light scene to really showcase shadow noise.  This time I adjusted register settings to compensate for highlights (as I should have before, but was too eager just to see results!)

No image processing except exposure compensation.  No noise reduction at all.

109 rocks.

5D3 1.13

@70MM13 ISO 109? Are you setting this 109 as a ML ISO ? Not sure what your process was in this post – but it looks great! 

[scrax]

I don't know! What is badcaffe? 

I don't know either 

I got it set to all value after a fresh boot:

Code: [Select]

Canon EOS 600D 1.0.2
00f00000:       0 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3910 addr=4069464c Analog ISO (most cameras)
00f00001:     423 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3910 addr=4069464e Vertical offset
00f00002:      18 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3138 addr=40694636 Horizontal offset / column skipping
00f00003:     1d0 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3138 addr=40694638 Analog ISO on 6D
00f00007:       1 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3910 addr=40694650 5D3: image fading out; 6D, 700D: vertical offset
000200c7:    3201 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c2e9c addr=40694288
00020002:       1 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c2e9c addr=4069428c
00020006:       1 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c2e9c addr=40694290
00020007:       0 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c2e9c addr=40694294 ADTG preamp (4 2-bit column gains; clean image at 00, 55, AA and FF)
...

I suppose is a magic code like deadbeef, waiting the answer of the experts

from wikipedia:
"("bad cafe") is used by Libumem to indicate uninitialized memory area. "

so could be same here I suppose

[timbytheriver]

I don't know either 
...
so could be same here I suppose

New street word of the day: "You're soooo badcaffe!" 

[a1ex]

Hehe - badcaffe is a magic value for registers not yet discovered.

So, when comparing two video modes, "was badcaffe" actually means the register was discovered while switching to the second mode, but its value was unknown in the first mode.

Solution: just retry the procedure, as now adtg_gui will be aware of all registers. That's become most registers are configured only during a mode switch, not continuously on every LiveView frame.

[70MM13]

@timbytheriver

the process i used for 109 is the one i outlined back then for getting 109 from 200, pretty much exactly what a1ex taught us at the start of this thread!
just look at posts i made on here around that time and you'll see the exact values i used.

i think it will be similar to what i'm getting now from the simplified approach i posted earlier today, but that's just a guess...  who knows... not me!

the values i use now result in something like 111, i think.  i don't pay much attention, i just rock and roll

i'm the wrong guy to do serious analysis.  i just want to play, mainly in low light, and i know what i like!

[timbytheriver]

@70MM13 Thanks once again. You've given me confidence to play!

@a1ex badcaffe = so many possibilities… 

[timbytheriver]

After further tests I have some more questions, as I'm still not seeing the dramatic improvements that others have demonstrated:

Could someone confirm that I'm not missing something important here as I hope this will be useful for those to come!

I'm using iso_regs.mo *only* on 5DMK3 shooting in 3K 1:1 mode, base ISO400.

Take the following shot of HDR shot Desk lamp and shadowy books.








1) I've exposed for the lampshade at the point where the raw histogram 'OVER' appears, and then backed off a hair. Is this correct procedure?

2) I take a still photo while in Liveview mode (CR2 and jpeg which save to my SD card). I don't shoot in 'silent picture' mode as this seems not be working here. Does this make a difference?

3) I take a reference movie clip at these settings.

4) I go into ISO registers and make sure default values are present.

5) I click 'Copy Canon settings'. Sometimes ALL the values populate – other times only some values (say CMOS, ADTG, Digi GAIN) Is this expected behaviour?

6) I tweak the registers as per @70MM13 instructions: I tried set 0xfe to 3, preamp to 2, and reduce adtg until the "iso" is half of the base value = I exit that menu. Is this correct?

7) I shoot the clip at these settings. Now, I'm confused about the instructions to 'match the liveview exposure' What does this mean?

8 ) I load the clips into MLVApp, adjust WB, Exp, Dark strength, and get disappointed! 

Any thoughts / suggestions?

Many thanks.