Cleaner ISO presets

[timbytheriver]

This thread is continued from here: https://www.magiclantern.fm/forum/index.php?topic=10111.msg223371#msg223371 CMOS/ADTG/Digic register investigation on ISO

The objective here is to add an ML menu or sub-menu, that allows the user to quickly adjust ISO related Preamp et al registers on-the-fly.

[2blackbar]

Are You showing example pics straight from MLVApp? If yes then its tonemapped gamma is lifted thats why you see a lot more noise in shadows than in lets say adobe software if you would load the same frames as DNG files in photoshop.
With that being said i prefere the look of footage from MLVApp, in other programs shadows are darker which hides the noise.
From what i understand you can get cleaner shadows if they arent recorded at the bottom but more like mid-bottom with ETTR and then go down in post to make them sit on bottom but that could hurt dynamic range in some scenes ( not every scene requires full 11 stops so that might not be so bad if you want better dark areas).

[timbytheriver]

@2blackbar Thanks for the info. I edit and grade in Davinci Resolve. You will see the same changes in noise behaviour in many apps. MLVApp is great for checking files as they come out of the camera!

If possible I'd like to keep this thread focussed on the objective as stated in the first post.

[ghost0cnc]

I'm still experimenting with the settings, but so far I modified the engio_write_log, adtg_log and cmos_log functions to automatically adjust the settings depending on the current lens_info.raw_iso value.
Caution: This is probably racy! I haven't looked into the code much further.

I added an "autoconfig" option to the iso_regs module and hard-coded some settings which will be applied if the autoconfig-flag is set.

Camera: 5D Mark III

cmos_log: Shifts each selectable ISO-value by 1 stop.

Code: [Select]

switch (lens_info.raw_iso) {
case 64: cmos_gain = 0x003; break; // ISO 50 -> ISO 100 Gain
case 72: cmos_gain = 0x113; break; // ISO 100 -> ISO 200 Gain
case 80: cmos_gain = 0x223; break; // ISO 200 -> ISO 400 Gain
case 88: cmos_gain = 0x333; break; // ISO 400 -> ISO 800 Gain
...
}When I select ISO 100 in the Canon settings (lens_info.raw_iso == 72), the CMOS-Gain is set to 0x113 which normally would be the correct value for Canon's ISO 200.
ISO 200 gets the CMOS-Gain of Canon's ISO 400 and so on.

So far, the images taken with ISO 50 to 6400 just get brighter by 1 stop.

To compensate for this I also adjusted the ADTG-Settings and stretched the usable range of the 14-bit format:

adtg_log:

Code: [Select]

- adtg_preamp = 2;
- adtg_fe = 4;
- adtg_gain = 1050; (except for lens_info.raw_iso == 64: adtg_gain = 1220;)
- black_reference = 2048;adtg_fe=3 gives a little more highlight headroom but causes more banding in the shadows compared to adtg_fe=4.

engio_write_log:

Code: [Select]

- black_white_offset = 3058;
- saturate_offset = 118;
- digital_gain = 463;
The resulting images have about the same brightness as the original Canon settings but with less shadow noise (and ISO 50 is actually useful).

I haven't fully understood the black_reference, saturate_offset and black_white_offset settings, which causes issues in live-view mode.

Let's find some "optimal" settings and write a config that loads the best settings for each selected ISO-value.

[timbytheriver]

@ghost0cnc Hi! This looks interesting work. Thanks for sharing.

The problem we have with iso_regs.mo is that it doesn't work if you choose an extended resolution (3K, UHD et al). The current state of my preset allows the regs to be set for those resolutions.

From my reading of the original thread (see first post) – which I heartily encourage you to dive into – CMOS gain simply alters ISO by a whole stop increment. That being said, I believe a1ex had some success getting a clean ISO 60 (from 100) with CMOS 4 = 0x318 along with other settings.

The overall conclusion I reached from reading the entire thread is that the money registers for fine ISO tweaks are:
ADTG2/4 8,9,A,B
ADTG2/4 0xFE

There may of course be  hundreds of others yet to be discovered…

[70MM13]

here's a new "how to" video i just made to follow up on the first one, with the latest enhancements...

hopefully soon tim will have this working perfectly as a preset!

[timbytheriver]

Whilst I can now hard-code a preset, we really need to be able to dial-in the correct settings on-the-fly.

So, I'm making some progress on the crop mode > Q-sub menu preset working on 5D3. But it's only half-working at the moment! I could use some help!

I can get the Preamps ADTG2/4 8,9,A,B controllable with the following code (snippets):

Code: [Select]


if (is_5D3)
    {
        adtg_new[20] = (struct adtg_new) {6, 0x8, 0x60 + reg_8};
        adtg_new[21] = (struct adtg_new) {6, 0x9, 0x61 + reg_9};
        adtg_new[22] = (struct adtg_new) {6, 0xA, 0x62 + reg_a};
        adtg_new[23] = (struct adtg_new) {6, 0xB, 0x60 + reg_b};
       
        adtg_new[24] = (struct adtg_new) {2, 0xFE, 0x4 + reg_fe};
       
        /* all modes may want to override shutter speed */
        /* ADTG[0x8060]: shutter blanking for 3x3 mode  */
        /* ADTG[0x805E]: shutter blanking for zoom mode  */

        adtg_new[0] = (struct adtg_new) {6, 0x8060, shutter_blanking};
        adtg_new[1] = (struct adtg_new) {6, 0x805E, shutter_blanking};

        switch (crop_preset)
        {




But for some reason I can't control the ADTG reg 0xFE (fe) with the code above. It stays the same no matter what I set in the UI.

Also, I want to be able to set the 0xFE at '0' but the UI will only let me set 0 as OFF, rather than as a selection itself.

Code: [Select]


{
    {
        .name       = "Crop mode",
        .priv       = &crop_preset_index,
        .update     = crop_update,
        .depends_on = DEP_LIVEVIEW,
        .children =  (struct menu_entry[]) {
            {
                .name   = "reg_8",
                .priv   = &reg_8,
                .min    = -1,
                .max    = 0x100,
                .unit   = UNIT_DEC,
                .help  = "reg 8",
                //.advanced = 1,
            },
            {
                .name   = "reg_9",
                .priv   = &reg_9,
                .min    = -1,
                .max    = 0x100,
                .unit   = UNIT_DEC,
                .help  = "reg 9",
                //.advanced = 1,
            },
            {
                .name   = "reg_a",
                .priv   = &reg_a,
                .min    = -1,
                .max    = 0x100,
                .unit   = UNIT_DEC,
                .help  = "reg a",
                //.advanced = 1,
            },
            {
                .name   = "reg_b",
                .priv   = &reg_b,
                .min    = -1,
                .max    = 0x100,
                .unit   = UNIT_DEC,
                .help  = "reg b",
                //.advanced = 1,
            },
            {
                .name   = "reg_fe",
                .priv   = &reg_fe,
                .min    = -1,
                .max    = 7,
                .unit   = UNIT_DEC,
                .help  = "reg fe",
                //.advanced = 1,
            },





and…

Code: [Select]


static int32_t  reg_8 = 0;
static int32_t  reg_9 = 0;
static int32_t  reg_a = 0;
static int32_t  reg_b = 0;
static int32_t  reg_fe = 0;


Can someone spot anything please?

Many thanks!

[names_are_hard]

Need more context.  Where do these lines come from?  Now you have a repo, can you link to the commit in question?  I cloned yours but can't find this code, so I suspect it's local only?

If you don't want to commit broken code to your repo, that's an understandable concern - if so it might be time for you to learn about branches.  That's the Git term, I think in Mercurial the equivalent might be bookmarks?  I find the Mercurial branch model confusing.  Anyway, you can use your repo to keep multiple versions of your files safe.  You can play around in one branch and when the code works merge it to another.

[timbytheriver]

Sure. Some of this code isn't in my remote repo yet. Getting a bit confused when to start a new repo Vs branches, so this is timely advice. Thank you!

I notice in Sourcetree (that I'm using) there is an option to 'Branch' so maybe that will work in Mecurial also? I will check it out.

I'm making some good progress on the Q-menu preset. So some of the above I've now solved. But this one eludes me:

How might I approach targeting all these registers:

Code: [Select]

adtg_new[20] = (struct adtg_new) {6, 0x8, 0x60 + reg_8};
adtg_new[21] = (struct adtg_new) {6, 0x9, 0x61 + reg_9};
adtg_new[22] = (struct adtg_new) {6, 0xA, 0x62 + reg_a};
adtg_new[23] = (struct adtg_new) {6, 0xB, 0x60 + reg_b};

Into one adjustment:

i.e. : 0x8, 0x9, 0xA, 0xB = a single value (and ui slot)

Can't be as simple as this below I'd expect: 

Code: [Select]


adtg_new[24] = (struct adtg_new) {6, 0x8,0x9,0xA,0xB, 0x60 + reg_all-in-one};


I've seen this snippet below in iso_regs but I don't know how relevant it is to this request!

Code: [Select]

((reg == 0x8 || reg == 0x9 || reg == 0xA || reg == 0xB) && (dst == 2 || dst == 4))

Many thanks!

PS I have scoured crop_rec.c already to see whether there might be something similar, but I can't see it [yet]!

[Danne]

Just call all regs reg_8 or whatever that you want to change multiple values.
Try and publish full source code so we can follow context.

[timbytheriver]

Ah-ha! Simples.  Thanks!

I will try to publish this as a branch later, if I can make it work.

Working. Published!

[timbytheriver]

I now have a Q-menu build available for testing. https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

Added:
Crop > Q-submenu
reg_8 = ADTG Preamp
reg_fe = ADTG 0xFE

They load as default values (96,and 4 respectively) so you'll want to lower your gains by a negative value. My Preamp is -48 for my camera. The 0xfe is best set to -4 (which is 0) at present. If you want to check how the 'calibration' works check out @70MM13's how-to video here https://www.magiclantern.fm/forum/index.php?topic=24706.msg223409#msg223409 But now you don't need to use iso_regs.mo – brilliant though that module is!

Thanks to @Danne & @names_are_hard for your coding help!

[DeafEyeJedi]

Good morning Fellas! I've been out of town traveling the past week. Thanks, Tim for the PM and leading me to this thread. I will definitely get my hands on a 5D3.123 to test this build of yours re: clean ISO's @ 200.

I'll definitely share and post my experience this weekend for sure. Thanks, @Danne for your insights as always!

[timbytheriver]

Hey!

Try the new improved version with the adjustments in the Q-menu. 1.2.3 version now up:

https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

 

[70MM13]

hey everyone,
i reshot the opening scene of the "biorobots" music video using the q menu and it was such a JOY to work with!

check out the new version with the opening scene shot with even less light!

https://www.magiclantern.fm/forum/index.php?topic=24690.msg223243#msg223243


here's a new tutorial on using this "TNT" build, including sample DNGs for you all to play with!

https://www.magiclantern.fm/forum/index.php?topic=24713.msg223515;topicseen#msg223515

[timbytheriver]

I'm looking for some pointers for making my iso-regs Q-menu adjustments available in NON-crop modes (i.e. mv1080p). 

This would mean using a different Q-menu, say in Movie > and either > Image Fine-tuning or Movie Tweaks.

I can see the Image Fine-tuning functions in lv-img-engio.c https://bitbucket.org/hudson/magic-lantern/src/crop_rec_4k/src/lv-img-engio.c

Is this the right place to add these reg-adjust functions - or am I on the wrong track?

Thanks!

[Danne]

I keep a mv1080p preset in crop_rec code in my build so you should be able to use the same place as before. It´s the bottom preset, at least in later builds(mv1080p_mv720p). I kept it for experimenting reasons as yours for instance.

[timbytheriver]

@Danne Many thanks! That's great news.

lv-img-engio.c looks scary… 

[names_are_hard]

Some point soon you may have to sit down and learn C

[timbytheriver]

Some point soon you may have to sit down and learn C

Well, I've made a start with some tutorials… but my mind just isn't at its best thinking in code.

[names_are_hard]

It takes time, but it gets easier.  It's a complex set of skills to get good at, everyone feels lost at first.

[timbytheriver]

Here are some results from my most recent ISO test (Stock ISO vs Tweaked ISO) using my custom ML build for 5D3 and lua script to load the presets.

Branch: https://bitbucket.org/rivertim/magic-lantern-danneclone/commits/branch/iso-regs-Q-submenu-access

iso-dngs.zip contains the test files. All available on my repo downloads page:

https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

[Danne]

Not being able to view with audio on. If possible, could you share exactly the rehmgs tweaked both stock iso and after applied?

[timbytheriver]

Strange. Audio fine here. Does it play correctly on YouTube?

[Danne]

I have audio only can't turn it on were I am atm .