Author Topic: Cleaner ISO presets  (Read 6076 times)

timbytheriver

  • Senior
  • ****
  • Posts: 459
Cleaner ISO presets
« on: December 11, 2019, 01:33:11 PM »
This thread is continued from here: https://www.magiclantern.fm/forum/index.php?topic=10111.msg223371#msg223371 CMOS/ADTG/Digic register investigation on ISO

The objective here is to add an ML menu or sub-menu, that allows the user to quickly adjust ISO related Preamp et al registers on-the-fly.
5D3 1.1.3
5D2 2.1.2

2blackbar

  • Senior
  • ****
  • Posts: 470
Re: Cleaner ISO presets
« Reply #1 on: December 11, 2019, 01:53:09 PM »
Are You showing example pics straight from MLVApp? If yes then its tonemapped gamma is lifted thats why you see a lot more noise in shadows than in lets say adobe software if you would load the same frames as DNG files in photoshop.
With that being said i prefere the look of footage from MLVApp, in other programs shadows are darker which hides the noise.
From what i understand you can get cleaner shadows if they arent recorded at the bottom but more like mid-bottom with ETTR and then go down in post to make them sit on bottom but that could hurt dynamic range in some scenes ( not every scene requires full 11 stops so that might not be so bad if you want better dark areas).

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #2 on: December 11, 2019, 02:12:35 PM »
@2blackbar Thanks for the info. :) I edit and grade in Davinci Resolve. You will see the same changes in noise behaviour in many apps. MLVApp is great for checking files as they come out of the camera!

If possible I'd like to keep this thread focussed on the objective as stated in the first post.
5D3 1.1.3
5D2 2.1.2

ghost0cnc

  • New to the forum
  • *
  • Posts: 3
Re: Cleaner ISO presets
« Reply #3 on: December 11, 2019, 02:48:54 PM »
I'm still experimenting with the settings, but so far I modified the engio_write_log, adtg_log and cmos_log functions to automatically adjust the settings depending on the current lens_info.raw_iso value.
Caution: This is probably racy! I haven't looked into the code much further.

I added an "autoconfig" option to the iso_regs module and hard-coded some settings which will be applied if the autoconfig-flag is set.

Camera: 5D Mark III

cmos_log: Shifts each selectable ISO-value by 1 stop.
Code: [Select]
switch (lens_info.raw_iso) {
case 64: cmos_gain = 0x003; break; // ISO 50 -> ISO 100 Gain
case 72: cmos_gain = 0x113; break; // ISO 100 -> ISO 200 Gain
case 80: cmos_gain = 0x223; break; // ISO 200 -> ISO 400 Gain
case 88: cmos_gain = 0x333; break; // ISO 400 -> ISO 800 Gain
...
}
When I select ISO 100 in the Canon settings (lens_info.raw_iso == 72), the CMOS-Gain is set to 0x113 which normally would be the correct value for Canon's ISO 200.
ISO 200 gets the CMOS-Gain of Canon's ISO 400 and so on.

So far, the images taken with ISO 50 to 6400 just get brighter by 1 stop.

To compensate for this I also adjusted the ADTG-Settings and stretched the usable range of the 14-bit format:

adtg_log:
Code: [Select]
- adtg_preamp = 2;
- adtg_fe = 4;
- adtg_gain = 1050; (except for lens_info.raw_iso == 64: adtg_gain = 1220;)
- black_reference = 2048;
adtg_fe=3 gives a little more highlight headroom but causes more banding in the shadows compared to adtg_fe=4.

engio_write_log:
Code: [Select]
- black_white_offset = 3058;
- saturate_offset = 118;
- digital_gain = 463;

The resulting images have about the same brightness as the original Canon settings but with less shadow noise (and ISO 50 is actually useful).

I haven't fully understood the black_reference, saturate_offset and black_white_offset settings, which causes issues in live-view mode.

Let's find some "optimal" settings and write a config that loads the best settings for each selected ISO-value. :)

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #4 on: December 11, 2019, 03:21:36 PM »
@ghost0cnc Hi! This looks interesting work. Thanks for sharing.

The problem we have with iso_regs.mo is that it doesn't work if you choose an extended resolution (3K, UHD et al). The current state of my preset allows the regs to be set for those resolutions.

From my reading of the original thread (see first post) – which I heartily encourage you to dive into – CMOS gain simply alters ISO by a whole stop increment. That being said, I believe a1ex had some success getting a clean ISO 60 (from 100) with CMOS 4 = 0x318 along with other settings.

The overall conclusion I reached from reading the entire thread is that the money registers for fine ISO tweaks are:
ADTG2/4 8,9,A,B
ADTG2/4 0xFE

There may of course be  hundreds of others yet to be discovered… :)
5D3 1.1.3
5D2 2.1.2

70MM13

  • Senior
  • ****
  • Posts: 494
Re: Cleaner ISO presets
« Reply #5 on: December 11, 2019, 11:05:50 PM »
here's a new "how to" video i just made to follow up on the first one, with the latest enhancements...

hopefully soon tim will have this working perfectly as a preset!


timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #6 on: December 12, 2019, 06:25:27 PM »
Whilst I can now hard-code a preset, we really need to be able to dial-in the correct settings on-the-fly.

So, I'm making some progress on the crop mode > Q-sub menu preset working on 5D3. But it's only half-working at the moment! I could use some help! :)

I can get the Preamps ADTG2/4 8,9,A,B controllable with the following code (snippets):

Code: [Select]

if (is_5D3)
    {
        adtg_new[20] = (struct adtg_new) {6, 0x8, 0x60 + reg_8};
        adtg_new[21] = (struct adtg_new) {6, 0x9, 0x61 + reg_9};
        adtg_new[22] = (struct adtg_new) {6, 0xA, 0x62 + reg_a};
        adtg_new[23] = (struct adtg_new) {6, 0xB, 0x60 + reg_b};
       
        adtg_new[24] = (struct adtg_new) {2, 0xFE, 0x4 + reg_fe};
       
        /* all modes may want to override shutter speed */
        /* ADTG[0x8060]: shutter blanking for 3x3 mode  */
        /* ADTG[0x805E]: shutter blanking for zoom mode  */

        adtg_new[0] = (struct adtg_new) {6, 0x8060, shutter_blanking};
        adtg_new[1] = (struct adtg_new) {6, 0x805E, shutter_blanking};

        switch (crop_preset)
        {




But for some reason I can't control the ADTG reg 0xFE (fe) with the code above. It stays the same no matter what I set in the UI.

Also, I want to be able to set the 0xFE at '0' but the UI will only let me set 0 as OFF, rather than as a selection itself.

Code: [Select]

{
    {
        .name       = "Crop mode",
        .priv       = &crop_preset_index,
        .update     = crop_update,
        .depends_on = DEP_LIVEVIEW,
        .children =  (struct menu_entry[]) {
            {
                .name   = "reg_8",
                .priv   = &reg_8,
                .min    = -1,
                .max    = 0x100,
                .unit   = UNIT_DEC,
                .help  = "reg 8",
                //.advanced = 1,
            },
            {
                .name   = "reg_9",
                .priv   = &reg_9,
                .min    = -1,
                .max    = 0x100,
                .unit   = UNIT_DEC,
                .help  = "reg 9",
                //.advanced = 1,
            },
            {
                .name   = "reg_a",
                .priv   = &reg_a,
                .min    = -1,
                .max    = 0x100,
                .unit   = UNIT_DEC,
                .help  = "reg a",
                //.advanced = 1,
            },
            {
                .name   = "reg_b",
                .priv   = &reg_b,
                .min    = -1,
                .max    = 0x100,
                .unit   = UNIT_DEC,
                .help  = "reg b",
                //.advanced = 1,
            },
            {
                .name   = "reg_fe",
                .priv   = &reg_fe,
                .min    = -1,
                .max    = 7,
                .unit   = UNIT_DEC,
                .help  = "reg fe",
                //.advanced = 1,
            },





and…

Code: [Select]

static int32_t  reg_8 = 0;
static int32_t  reg_9 = 0;
static int32_t  reg_a = 0;
static int32_t  reg_b = 0;
static int32_t  reg_fe = 0;


Can someone spot anything please?

Many thanks!

5D3 1.1.3
5D2 2.1.2

names_are_hard

  • Developer
  • Senior
  • *****
  • Posts: 347
  • 200D idiot
Re: Cleaner ISO presets
« Reply #7 on: December 14, 2019, 06:30:07 AM »
Need more context.  Where do these lines come from?  Now you have a repo, can you link to the commit in question?  I cloned yours but can't find this code, so I suspect it's local only?

If you don't want to commit broken code to your repo, that's an understandable concern - if so it might be time for you to learn about branches.  That's the Git term, I think in Mercurial the equivalent might be bookmarks?  I find the Mercurial branch model confusing.  Anyway, you can use your repo to keep multiple versions of your files safe.  You can play around in one branch and when the code works merge it to another.

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #8 on: December 14, 2019, 09:36:20 AM »
Sure. Some of this code isn't in my remote repo yet. Getting a bit confused when to start a new repo Vs branches, so this is timely advice. Thank you!

I notice in Sourcetree (that I'm using) there is an option to 'Branch' so maybe that will work in Mecurial also? I will check it out.

I'm making some good progress on the Q-menu preset. So some of the above I've now solved. But this one eludes me:

How might I approach targeting all these registers:

Code: [Select]
adtg_new[20] = (struct adtg_new) {6, 0x8, 0x60 + reg_8};
adtg_new[21] = (struct adtg_new) {6, 0x9, 0x61 + reg_9};
adtg_new[22] = (struct adtg_new) {6, 0xA, 0x62 + reg_a};
adtg_new[23] = (struct adtg_new) {6, 0xB, 0x60 + reg_b};

Into one adjustment:

i.e. : 0x8, 0x9, 0xA, 0xB = a single value (and ui slot)

Can't be as simple as this below I'd expect:  :-\

Code: [Select]

adtg_new[24] = (struct adtg_new) {6, 0x8,0x9,0xA,0xB, 0x60 + reg_all-in-one};


I've seen this snippet below in iso_regs but I don't know how relevant it is to this request!

Code: [Select]
((reg == 0x8 || reg == 0x9 || reg == 0xA || reg == 0xB) && (dst == 2 || dst == 4))

Many thanks!

PS I have scoured crop_rec.c already to see whether there might be something similar, but I can't see it [yet]!
5D3 1.1.3
5D2 2.1.2

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #9 on: December 14, 2019, 11:38:17 AM »
Just call all regs reg_8 or whatever that you want to change multiple values.
Try and publish full source code so we can follow context.

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #10 on: December 14, 2019, 12:27:45 PM »
Ah-ha! Simples.  8) Thanks!

I will try to publish this as a branch later, if I can make it work.

Working. Published!
5D3 1.1.3
5D2 2.1.2

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #11 on: December 14, 2019, 04:11:21 PM »
I now have a Q-menu build available for testing. https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

Added:
Crop > Q-submenu
reg_8 = ADTG Preamp
reg_fe = ADTG 0xFE

They load as default values (96,and 4 respectively) so you'll want to lower your gains by a negative value. My Preamp is -48 for my camera. The 0xfe is best set to -4 (which is 0) at present. If you want to check how the 'calibration' works check out @70MM13's how-to video here https://www.magiclantern.fm/forum/index.php?topic=24706.msg223409#msg223409 But now you don't need to use iso_regs.mo – brilliant though that module is! ;)

Thanks to @Danne & @names_are_hard for your coding help!
5D3 1.1.3
5D2 2.1.2

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3401
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: Cleaner ISO presets
« Reply #12 on: December 14, 2019, 08:00:19 PM »
Good morning Fellas! I've been out of town traveling the past week. Thanks, Tim for the PM and leading me to this thread. I will definitely get my hands on a 5D3.123 to test this build of yours re: clean ISO's @ 200.

I'll definitely share and post my experience this weekend for sure. Thanks, @Danne for your insights as always!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #13 on: December 14, 2019, 08:51:17 PM »
Hey!

Try the new improved version with the adjustments in the Q-menu. 1.2.3 version now up:

https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

 :)
5D3 1.1.3
5D2 2.1.2

70MM13

  • Senior
  • ****
  • Posts: 494
Re: Cleaner ISO presets
« Reply #14 on: December 15, 2019, 07:43:55 PM »
hey everyone,
i reshot the opening scene of the "biorobots" music video using the q menu and it was such a JOY to work with!

check out the new version with the opening scene shot with even less light!

https://www.magiclantern.fm/forum/index.php?topic=24690.msg223243#msg223243



here's a new tutorial on using this "TNT" build, including sample DNGs for you all to play with!

https://www.magiclantern.fm/forum/index.php?topic=24713.msg223515;topicseen#msg223515

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #15 on: December 30, 2019, 06:20:53 PM »
I'm looking for some pointers for making my iso-regs Q-menu adjustments available in NON-crop modes (i.e. mv1080p).  :)

This would mean using a different Q-menu, say in Movie > and either > Image Fine-tuning or Movie Tweaks.

I can see the Image Fine-tuning functions in lv-img-engio.c https://bitbucket.org/hudson/magic-lantern/src/crop_rec_4k/src/lv-img-engio.c

Is this the right place to add these reg-adjust functions - or am I on the wrong track?

Thanks!
5D3 1.1.3
5D2 2.1.2

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #16 on: December 30, 2019, 06:23:10 PM »
I keep a mv1080p preset in crop_rec code in my build so you should be able to use the same place as before. It´s the bottom preset, at least in later builds(mv1080p_mv720p). I kept it for experimenting reasons as yours for instance.


timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #17 on: December 30, 2019, 06:41:30 PM »
@Danne Many thanks! That's great news. :)

lv-img-engio.c looks scary…  :o
5D3 1.1.3
5D2 2.1.2

names_are_hard

  • Developer
  • Senior
  • *****
  • Posts: 347
  • 200D idiot
Re: Cleaner ISO presets
« Reply #18 on: December 30, 2019, 06:48:39 PM »
Some point soon you may have to sit down and learn C :P

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #19 on: December 30, 2019, 07:01:59 PM »
Quote
Some point soon you may have to sit down and learn C :P

Well, I've made a start with some tutorials… but my mind just isn't at its best thinking in code. :(
5D3 1.1.3
5D2 2.1.2

names_are_hard

  • Developer
  • Senior
  • *****
  • Posts: 347
  • 200D idiot
Re: Cleaner ISO presets
« Reply #20 on: December 30, 2019, 07:14:27 PM »
It takes time, but it gets easier.  It's a complex set of skills to get good at, everyone feels lost at first.

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #21 on: January 18, 2020, 12:10:28 PM »
Here are some results from my most recent ISO test (Stock ISO vs Tweaked ISO) using my custom ML build for 5D3 and lua script to load the presets.

Branch: https://bitbucket.org/rivertim/magic-lantern-danneclone/commits/branch/iso-regs-Q-submenu-access

iso-dngs.zip contains the test files. All available on my repo downloads page:

https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

5D3 1.1.3
5D2 2.1.2

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #22 on: January 18, 2020, 12:20:57 PM »
Not being able to view with audio on. If possible, could you share exactly the rehmgs tweaked both stock iso and after applied?

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #23 on: January 18, 2020, 12:25:32 PM »
Strange. Audio fine here. Does it play correctly on YouTube?
5D3 1.1.3
5D2 2.1.2

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #24 on: January 18, 2020, 12:26:39 PM »
I have audio only can't turn it on were I am atm ;).

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #25 on: January 18, 2020, 12:29:23 PM »
Oh right. There are annotations on the film for the important bits! ;)

Regs and values are clearly visible in the submenu when the presets load! The stock ISOs are – stock! No reg tweaks.
5D3 1.1.3
5D2 2.1.2

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #26 on: January 18, 2020, 01:15:42 PM »
I'll have a look.
By the way. You should post sources, (or branch in this case), relevant to your builds shared. Not clear what code was used.
Disregard. Found it.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #27 on: January 18, 2020, 01:50:27 PM »
Checked your movie. It looks like a valid refinement. Still you do not reveal exactly what registry used, preset used etc. I check your branch and in crop_rec.c only place I could find modifying iso reg are in this preset CROP_PRESET_UHD:
Code: [Select]
                    /* ADTG2 8,9,a,b adjusts blue and green channel 1 (Audionut?) */
                    /* ADTG4 8,9,a,b  adjusts red and green channel 2 (Audionut?) */
                   
                    adtg_new[30] = (struct adtg_new) {6, 0x8, 46}; /* these control each sensor column: should be 0x2 at lowest Value can be in HEX or DEC */
                    adtg_new[31] = (struct adtg_new) {6, 0x9, 46};
                    adtg_new[32] = (struct adtg_new) {6, 0xA, 46}; /* 0x2E = 46 for iso160, 0x31 = 48 for iso200 */
                    adtg_new[33] = (struct adtg_new) {6, 0xB, 46};
                   
                   
                    //adtg_new[34] = (struct adtg_new) {4, 0x8, 0x31};
                    //adtg_new[35] = (struct adtg_new) {4, 0x9, 0x31};
                    //adtg_new[36] = (struct adtg_new) {4, 0xA, 0x31};  /* set all @ 0x2 for basic clean iso. Calibrate on a per-camera basis for the 70MM13 'OVER' tweak. See comment below. */
                    //adtg_new[37] = (struct adtg_new) {4, 0xB, 0x31};
                   
                    /* ADTG4 0xFE master gain? */
                    adtg_new[35] = (struct adtg_new) {6, 0xFE, 0}; /* ADTG2/4 0xFE = 0 is clean. Causing yellow & blue artefacts if not set to same value */
                    //adtg_new[39] = (struct adtg_new) {2, 0xFE, 0x0}; /* ADTG2 0xFE = 0 is clean. Causing yellow & blue artefacts if not set to same value */

Is this the only place you are tweaking registry in? Are you at any point modifying analog gain registry? Please be transparent.

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #28 on: January 18, 2020, 01:58:05 PM »
For sure! :) That's the wrong branch! I updated my op video post awhile back with the branch I'm now using. https://bitbucket.org/rivertim/magic-lantern-danneclone/src/iso-regs-Q-submenu-access/ All the reg values are there to see – or on the camera in Q-menu. Nothing hidden!
5D3 1.1.3
5D2 2.1.2

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #29 on: January 18, 2020, 02:09:59 PM »
Ok, checked updated branch.
Still. From examples you are providing in your movie you do not tell how you tweaked? I can´t see any hard coded presets so I assume you have your own set up with tweaks? Please share those.
I do see analog gain registry added in here. To me, a big warning sign ;):
Code: [Select]
/* timby edit 'clean-iso' stuff. */
static int32_t  preamp_89ab = 0;
//static int32_t  reg_9 = 0;
//static int32_t  reg_a = 0;
//static int32_t  reg_b = 0;
static int32_t  gain_0xfe = 0;
static int32_t  gain_888x = 0;
/* timby: more iso_regs imports */
static int32_t  reg_8880 = 0;
//static int32_t  reg_8000 = 0;
//static int32_t  reg_8024 = 0;
//static int32_t  reg_8038 = 0;
//static int32_t  reg_8050 = 0;
//static int32_t  reg_814c = 0;
//static int32_t  reg_x14 = 0;
//static int32_t  reg_x15 = 0;

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #30 on: January 18, 2020, 02:24:38 PM »
The values for the presets are all in the lua script! :) 5d3iso-tweaks.lua.zip https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

The preamp values have to be calibrated for each camera to the max white level. So your camera maybe slightly different.
5D3 1.1.3
5D2 2.1.2

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #31 on: January 18, 2020, 02:36:33 PM »
This is balony. Still the same confusing non revealing pseudotweaks that sends these iso regs tests back to pseudouniverse. Sorry, I will not support this, too many question marks. Share tweaks where you published your tests so we clearly see what is going on.
And lastly. Analog gain certainly tweaked here. Not?
Code: [Select]
--menu.set("Presets", "reg_8xx8", "-31")Have fun.

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #32 on: January 18, 2020, 05:14:56 PM »
@Danne

From your rather alarmist tone one might think I was trying to force the community to "drink the Kool-Aid!" Let's get some perspective. This is simply my contribution to an open-source project. I'm not seeking endorsement, or support, or asking that my code be merged into the main branch.

Other users are free to test it and see if their results match mine – or like the proverbial TV, are welcome to turn it off at any time if they don't like it. :)
5D3 1.1.3
5D2 2.1.2

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7129
Re: Cleaner ISO presets
« Reply #33 on: January 18, 2020, 05:26:59 PM »
I shared and clearly showed you how to work out the registry in crop rec. Others helped too. Is it too much to ask for returning basic understandable results?
Pseudosmoke will take this project nowhere. Too bad. Thought you were on to something.

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #34 on: January 20, 2020, 01:22:22 PM »
Here's part two of my videos exploring the results of register gain adjustments. Using this tool: https://www.fastrawviewer.com/ the video examines the extracted dng files in much greater detail, comparing percentage of over and under-exposed pixels in each pair of comparisons.  Surely we all want less underexposed and overexposed pixels counts…?

5D3 1.1.3
5D2 2.1.2

Luther

  • Senior
  • ****
  • Posts: 322
Re: Cleaner ISO presets
« Reply #35 on: January 21, 2020, 07:35:04 AM »
Nice results.
Quote
Surely we all want less underexposed and overexposed pixels counts…?
I think the disagreement is just on the accuracy of the measurement method. In your video, how much of this better DR was caused by the ADTG tweak versus changes in exposure? If you're changing the ISO between stock 100 to 108, this will also increase exposure and could account for the changes in DR.
I doubt that's the case though (too big of a noise reduction in ISO 800) and I think these tweaked values should be of easy access to everyone (as simple as enabling a module).
Nice job @timbytheriver.

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #36 on: January 21, 2020, 10:01:40 AM »
Thanks @Luther!  :)

how much of this better DR was cause by the ADTG tweak versus changes in exposure? If you're changing the ISO between stock 100 to 108, this will also increase exposure and could account for the changes in DR.

It's my present understanding that 'exposure' is: an amount of photons hitting the sensor. This amount can only be altered by a) Lighting. b) Aperture. c) Shutter speed. The sensor only has a sensitivity that is factory-fixed – it cannot be made more or less sensitive by settings in the camera. The micro-voltages from the sensor (photo-electrons?) are then only amplified by the analogue gains and then converted into digital.

Since I didn't change a), b), or c) between the pairs of shots, and in the light of the above – can this be described as an exposure increase? I'm sure I'll be corrected if I've got this wrong! ;)

But irrespective of the photo-mechanics and the terms we use – and now these presets are easy to load (for 5D3 anyway…), anyone is free to try out the presets to compare results!

Cheers! :)

**PS** What is crucial to maximising these 'gains', is nailing your 'exposure' to maximum White Level –  absolutely accurate ETTR –which is a challenge! and requires some attention to the tools ML provides us with.
5D3 1.1.3
5D2 2.1.2

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3654
  • Blunt and to the point
Re: Cleaner ISO presets
« Reply #37 on: January 21, 2020, 10:50:21 AM »
The underexposure percentage value in fastrawviewer only indicates how many pixels are below a defined saturation point. It DOES NOT describe pixel quality, noise in pixels, DR or any other value of interest to the discussion.

So what register is it that you are tweaking to increase exposure (reduce underexposure percentage in fastrawviewer)? And why?

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #38 on: January 21, 2020, 11:33:56 AM »
…fastrawviewer only indicates how many pixels are below a defined saturation point.

Useful to know! But I'm describing the difference that it reports between pairs of figures; I'm not changing the saturation point in the prefs – and both images reported 16281 as a white level in the raw_diag readout. So both are at their clipping point I presume.

…So what register is it that you are tweaking to increase exposure (reduce underexposure percentage in fastrawviewer)?

Whatever the tweaks are doing behind the scenes I don't pretend to know, but surely no [register] tweak can increase the light hitting the sensor, or increase its sensitivity.  So, for my money they're not increasing exposure.

The register tweaks are no different to the discoveries made by yourself, a1ex and others way back in the original thread, and all available to check on my repo in the source code and Lua preset that defines the values. https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

As for the increase in dynamic range, I'm getting a difference in almost 2/3 stop as reported by raw_diag.

And why?

Because it's there. :)

5D3 1.1.3
5D2 2.1.2

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3654
  • Blunt and to the point
Re: Cleaner ISO presets
« Reply #39 on: January 21, 2020, 11:49:30 AM »
But I'm describing the difference that it reports between pairs of figures

Yes, but you don't seem to understand exactly what it is reporting!

At some point in the cameras processing chain, the register values you are tweaking have increased (brightened) the pixel values in the raw output file.  That is why the fastrawviewer underexposure percentage has dropped, because less pixels in the raw file are below the defined saturation point compared to Canon settings. But this value does not define the quality of those pixels or the amount of noise in those pixels, only that they are brighter!

Whatever the tweaks are doing behind the scenes I don't pretend to know, but surely no [register] tweak can increase the light hitting the sensor, or increase its sensitivity.  So, for my money they're not increasing exposure.

Nonsense.  ISO is exactly a sensitivity setting.  Let's not get hung up on the semantics of definition of exposure.
What we want is quality of pixels. The registers directly affect the quality of pixels.


Because it's there.

I honestly think this topic (register tweaking) is well past the point of blindly throwing shit at a wall and seeing what sticks.

timbytheriver

  • Senior
  • ****
  • Posts: 459
Re: Cleaner ISO presets
« Reply #40 on: January 21, 2020, 12:05:03 PM »
Not my understanding of exposure. From the people who bring us RawDigger – a tool I believe you have used in the past in your research into this subject. Source of quote: https://www.fastrawviewer.com/blog/mystic-exposure-triangle

Code: [Select]
Definitions and Facts
To start with, let’s lay out a few basic definitions and empirically verifiable (and indeed verified) facts that have to do with exposure and ISO:

Exposure time: this is the time during which light hits the photosensitive material in your camera; when the shutter closes, exposure ends.
Photographic exposure is only about two things: the light intensity and the exposure time: the light, coming from the scene, hitting the front element of the lens, reduced by the aperture opening, and acting during the exposure time.
By definition, photographic exposure is calculated as the product of the illuminance of the photosensitive material and the exposure time, and is measured in lux-seconds; photographic exposure is the product of luminous flux per unit area by time.
The elements that have direct control over exposure in the camera are the shutter and the aperture (not counting built-in neutral or other filters).

As such, ISO cannot be an element of exposure; it’s determined through the exposure (see standard ISO 12232:2019, "Photography — Digital still cameras — Determination of exposure index, ISO speed ratings, standard output sensitivity, and recommended exposure index" and the wiki), and is “applied” after exposure.

ISO 12232:2019(E) - ISO speed definition
Standard ISO 12232:2019, section 3, p. 3.7 (page.2)

At the same time, ISO is not the sensitivity of the sensor (see the datasheets on sensors): the sensitivity of the sensor doesn’t change when altering ISO, the only thing that is changed is the scaling coefficient applied to the signal / data obtained as a result of exposure.


Quote
blindly throwing shit at a wall and seeing what sticks.

This is totally unnecessary, lowers the tone of debate, and contributes nothing of value to the discussion. Not appreciated.
5D3 1.1.3
5D2 2.1.2

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3654
  • Blunt and to the point
Re: Cleaner ISO presets
« Reply #41 on: January 21, 2020, 01:00:09 PM »
Ok, lets play semantics.

What makes the author of that software a definitive authority on this subject?

https://www.iso.org/obp/ui/#iso:std:iso:12232:ed-3:v1:en
Quote
3.7
ISO speed
numerical value calculated from the exposure provided at the focal plane of a DSC to produce specified DSC image signal characteristics
Note 1 to entry: The ISO speed is usually the highest exposure index value that still provides peak image quality for normal scenes. However, a DSC does not necessarily use the ISO speed value as the exposure index value when capturing images.

Quote
3.3
exposure index
EI
numerical value that is inversely proportional to the exposure provided to an image sensor to obtain an image
Note 1 to entry: Images obtained from a DSC using a range of exposure index values will normally provide a range of image quality levels.

Quote
3.9
photographic sensitivity
general term used for numerical values calculated based on the exposure at the focal plane of a DSC which produces a specified DSC image signal level, such as the standard output sensitivity or recommended exposure index
Note 1 to entry: In practise, the photographic sensitivity is often called the "sensitivity" or the "camera sensitivity". It is sometimes called the “ISO sensitivity”, for historical reasons that date from ISO standards for photographic film cameras.

Quote
3.13
sensitivity setting
numerical value of the photographic sensitivity used by a DSC when capturing images
Note 1 to entry: In some cases the sensitivity setting is set by the user. In other cases it is set automatically by the DSC.
Note 2 to entry: In DSCs employing an automatic exposure control system, the difference between the EI value used to capture an image and the sensitivity setting is called the “exposure bias”. The value of the exposure bias is typically indicated using EV (exposure value) units.
Note 3 to entry: For historical reasons, the sensitivity setting of a DSC is often labelled the "ISO".

This is totally unnecessary, lowers the tone of debate, and contributes nothing of value to the discussion. Not appreciated.

You, yourself, acknowledge that you do not understand what is going on behind the scenes, so to speak. What you are doing is throwing shit at a wall, and seeing what sticks.
You are somewhat randomly (based on the time and effort of others) creating a preset of values for registers that you do not understand, applying no scientific method at all, and presenting the shit that's left stuck to the wall as useful data.

Then when someone who has spent the time finding those registers, and trying to define what they do, comes along and questions the shit that's flung all over the walls around the place, you deflect from the topics of discussion which hold value (how is the quality of the pixels being captured improved, and how is that being scientifically verified) with semantics and hurt feelings.

http://www.catb.org/esr/faqs/smart-questions.html#not_losing
Quote
Exaggeratedly “friendly” (in that fashion) or useful: Pick one.


I absolutely and unequivocally respect you getting your hands dirty (playing with code).

histor

  • Freshman
  • **
  • Posts: 60
Re: Cleaner ISO presets
« Reply #42 on: January 21, 2020, 01:56:03 PM »
Not to interfere with the discussion, just to be a mediator here.

This standard is barely applicable to RAW files in terms of sensitivity. Camera producers use JPEGs for that case.
Quote
"ISO speed and ISO speed latitude values shall not be reported for raw images, however, because with raw images processing that affects the values has not been performed" (ISO 12232:2019). https://www.fastrawviewer.com/blog/fujifilm-x-series-iso-bracketing

The author of the software - Alex Tutubalin is one of the leading developers of LibRaw (not just the commercial part of the project). Fastrawviewer blog is very informative.
Looks like these two Alexes are the best experts on the question in the open-source world.
But both usually say, they know nothing. As Socrates did.

________
As Luther noted, there was a canon trick for intermediate ISOs (like 160) – exposure time was changed behind the scenes. Silently. But really it's not the case now. You can always use some moving (rotating) object to prove the same exposure for two frames.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3654
  • Blunt and to the point
Re: Cleaner ISO presets
« Reply #43 on: January 22, 2020, 02:46:19 AM »
In a film camera you have film.
In a digital camera you have a processing chain (sensor, amplifiers, digital manipulation).

In a film camera, if you change from ISO 100 film to ISO 400 film you have increased the sensitivity (for the same amount of light hitting the camera = brighter output) of the camera.

If you adjust the ISO setting on a digital camera, you also adjust the sensitivity of the camera. Did you adjust the mechanical sensitivity of one specific part (sensor) of the processing chain, no, but (in Canon cameras especially) you gain an increase in output (brightness) of the camera with a corresponding increase in the quality of pixels (output).

ISO is exactly a sensitivity setting.

The sensitivity of the downstream processing chain from the sensor. It is a gain control (input sensitivity) for further downstream processing components.

In a film camera you are stuck with a preset sensitivity of the camera based on the film. To change the sensitivity of the camera you need to change the film. With a DSLR you can simply change a few knobs.

70MM13

  • Senior
  • ****
  • Posts: 494
Re: Cleaner ISO presets
« Reply #44 on: March 03, 2020, 01:48:43 PM »
tim and i have made excellent progress on TNT, and you can see the results in my latest videos, the candlelight music video, ttt episode 23, and zen episode 5.

we've basically got the 5d3 iso invariant from 100-400, with a vast improvement in the noise of 100.  it's actually cleaner in the shadows than 200 now!  peer into the shadows of the episodes listed above...

the candlelight video was shot at 400.  zero lighting in the scene except the candles.  it isn't the best representation of the video from youtube, but i will eventually upload a very high data rate 4k copy.

i know that there are some rather hostile closed minds trying to suppress our work, but there's no reason for 5d3 users who are actually interested in using this camera to produce excellent quality video not to use TNT!

PS: i think this thread has been polluted enough at this point.  can the hostile people please focus their negativity elsewhere?


After 6 weeks......you probably could have just stayed focused on the topic, rather then taking a pot shot because you obviously haven't been practicing that zen stuff enough, and you're still holding a grudge.  //Audionut

garry23

  • Contributor
  • Hero Member
  • *****
  • Posts: 2208
Re: Cleaner ISO presets
« Reply #45 on: March 03, 2020, 04:15:53 PM »
I’ve kept out of this discussion for obvious reasons.

However, I’m drawn to statements that claim the 5D3 can be made ISO invariant down to ISO100.

Looking at the current DR vs ISO, eg http://www.photonstophotos.net/Charts/PDR.htm tells me I would be expecting the DR at ISO100 to be some 13 units on the PDR chart, ie a straight line to ISO100.

Looking forward to seeing more info on this ‘claim’.


Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3654
  • Blunt and to the point
Re: Cleaner ISO presets
« Reply #46 on: March 05, 2020, 04:00:29 PM »
Negative attitude = / = Strong criticism.

we've basically got the 5d3 iso invariant from 100-400

No, you have not.

with a vast improvement in the noise of 100.  it's actually cleaner in the shadows than 200 now!

Everything else being equal, no, it is not.

CMOS sensor > downstream electronics.

CMOS sensor in 5D3 capable of 14.7 stops of dynamic range.
Quote
Sensor Full Well Capacity at lowest ISO: 68,900 electrons.
Sensor dynamic range = 68900/2.5 = 27,560 = 14.7 stops.
(note: limit read noise to ISO that give at least 8 stops dynamic range)

The 5D3 doesn't have 14.7 stops of DR at ISO 100.  Why?  Because downstream electronics are to noisy, limits DR otherwise available from the CMOS sensor.  About 3.7 stops of the sensors DR is lost in the noise of the downstream electronics.

When you adjust ISO (sensitivity :P), you say to camera, clip (throw away) 1 stop of highlight data so that you may better capture 1 stop of shadow data.  The full stop of shadow data from the sensor that would otherwise be lost in the noise of the downstream electronics is boosted so that it is no longer lost in the noise of the downstream electronics. Of course, the drawback is that you MUST sacrifice one full stop of highlight data to capture that shadow data.

So......

Saying that you've suddenly made the camera ISO invariant from 100-400, is the equivalent of saying you increased the DR of ISO 100 by 3 stops.  You have not!
Saying that the noise in the shadows of ISO 100 is cleaner then the noise of the shadows at ISO 200, is the equivalent of saying you increased the DR of ISO 100 by over 1 full stop, but haven't been able to extract any improvement at ISO 200.  This doesn't make any sense what so ever, since the improvement has to come from the downstream electronics, and adjusting sensitivity (I mean ISO) gives the shadows a boost by default.