Author Topic: CMOS/ADTG/Digic register investigation on ISO  (Read 548121 times)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3707
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #975 on: July 25, 2018, 12:59:53 AM »
Got a note that I've been sitting on for a while and am not sure which branch I should be working on.

The EOSM (and some others) require ENGIO_WRITE_FUNC and ENG_DRV_OUT_FUNC in order to compile the adtg_gui module in the lua_fix, crop_rec_4k and other branches.

modules/adtg_gui/adtg_gui.c
Code: [Select]
    else if (is_camera("EOSM", "2.0.2")) // from 1%
    {
        ADTG_WRITE_FUNC = 0x2986C;
        CMOS_WRITE_FUNC = 0x2998C;
        ENGIO_WRITE_FUNC = 0xff2c19ac;
        ENG_DRV_OUT_FUNC = 0xff2c1694;
    }

It would make sense to put this into the iso-research branch but surprise, it isn't needed in that branch. Where should it go so we don't have to keep looking up these stubs when compiling the adtg_gui module these other branches?
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

theBilalFakhouri

  • Contributor
  • Senior
  • *****
  • Posts: 323
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #976 on: July 25, 2018, 01:39:23 AM »
@dfort

It would be nice if you made a PR for these addresses in iso-research branch. Here it is for 700D:
Code: [Select]
else if (is_camera("700D", "1.1.5"))
    {
        ADTG_WRITE_FUNC = 0x178FC; //"[REG] @@@@@@@@@@@@ Start ADTG[CS:%lx]"
        CMOS_WRITE_FUNC = 0x17A1C; //"[REG] ############ Start CMOS"
ENGIO_WRITE_FUNC = 0xFF2C2D00;  // from stubs
        ENG_DRV_OUT_FUNC = 0xFF2C29E8;
    }
700D 1.1.5 | no more ISOless LV err 8

theBilalFakhouri

  • Contributor
  • Senior
  • *****
  • Posts: 323
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #977 on: July 25, 2018, 10:07:51 AM »
Found some ADTG gain registers in 700D:

Negative gain     Original value     Overridden
ADTG2[21] =        0x4626              0x1036
ADTG2[22] =        0x4626              0x1036
ADTG2[23] =        0x4626              0x1036
ADTG2[24] =        0x4626              0x1036

It darken the image about 1.5 stop with some vertical artifacts and increased noise in the shadows and it maybe affects the satrution.

Positive gain     Original value     Overridden
ADTG2[0] =        0x2062              0x2762
ADTG2[2] =        0x2066              0x2766
ADTG2[4] =        0x2062              0x2762
ADTG2[6] =        0x206c              0x276c

It bright the image about 2 stops? with shifting highlights of course.
700D 1.1.5 | no more ISOless LV err 8

ghost0cnc

  • New to the forum
  • *
  • Posts: 2
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #978 on: July 25, 2018, 05:25:37 PM »
Very nice clean shadows with little small clipping of highlights in contrasty scenes only (there are no clipping highlights in the example above).

From 800 ISO to 100 (Using ADTG Gain) the shadows is more cleaner but with losing dynamic range in highlights of course. Now I am trying to restore that dynamic range (highlights) at ISO 800 with the registers that related to the gain and I am not really sure if that possible.

The biggest increase in dynamic range I could get on my 5D3 was about one stop at ISO 200->100.

Let's assume you can also get about 1 EV of extra dynamic range by reducing the analog gain (like on the 5D3).

At ISO 400->100 or even 800->100 you will get about 1 EV of extra dynamic range (that was clipped before). The remaining 1 or 2 EV will just darken the photo.

Let's have a look at 400->100:

Basically you dropped the analog gain by 2 EV, which is 1 EV below the saturation point, and pushed that 1 EV back up in post-processing, effectively replacing 1 EV of analog gain with 1 EV of digital gain in post.

When using ISO 400->200, you can possibly get better noise levels [1] and more dynamic range than at 400->100 (at the same shutterspeed+aperture).

Just reducing the analog gain will not help reducing noise. After passing the sweet spot that allows you to capture more light and/or use higher "base"/CMOS ISOs (which will improve SNR), you just replace analog with digital amplification (which is a problem due to ADC noise and limited precision, as you only use a fraction of the 14 bit resolution, but can be better, depending on the analog amplifier noise).

Test on the 5D3:
Same subject and lighting, shutterspeed+aperture fixed. WB and all other settings except "Exposure" (in LR) are the same.
(Sharpening 33/1.0/25/25, Luminance NR disabled, Color NR 25/50/75)
a) ISO 800->400 (-1 EV, sweet spot)
b) ISO 800->200 (-2 EV, 1 EV below sweet spot)
Pushed both images in Lightroom to about the same brightness (a +4EV, b +5EV). I know the exposures are not 100% identical, but the difference should be small enough.
Noise levels are pretty similar (hard to compare as the color-rendering is a bit off), but photo "b" has much stronger banding.



Fine-tuning the ADTG parameters is another story. There are many combinations that result in the same amplification but differ in image quality. Sadly I haven't had time to analyze which settings work best.

[1] Better ADC SNR, as you have a stronger analog signal; depends on the noise generated by the analog amplifier vs ADC noise

70MM13

  • Member
  • ***
  • Posts: 235
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #979 on: July 28, 2018, 02:30:19 PM »
Inspired by this discussion, I have been testing ISO 320 pulled down to 100.

I tried a few other Isos as well and found that 320 is excellent for the cleanest shadows and only a very small loss of highlights in actual usage.

The highlights clip straight to pink, making it very easy to expose exactly at the threshold, and the beautiful shadows are a joy to work with. In the example frame, I exposed for the direct sunlight on the wall. The room was dark otherwise.



I'm using 320 a lot now, except those times I need as much highlight detail as possible at the expense of slightly noisier shadows.  Then it's back to 200!

Excellent stuff.

theBilalFakhouri

  • Contributor
  • Senior
  • *****
  • Posts: 323
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #980 on: July 28, 2018, 03:05:44 PM »
Let's have a look at 400->100:

Basically you dropped the analog gain by 2 EV, which is 1 EV below the saturation point, and pushed that 1 EV back up in post-processing, effectively replacing 1 EV of analog gain with 1 EV of digital gain in post.

When using ISO 400->200, you can possibly get better noise levels [1] and more dynamic range than at 400->100 (at the same shutterspeed+aperture).

Just reducing the analog gain will not help reducing noise. After passing the sweet spot that allows you to capture more light and/or use higher "base"/CMOS ISOs (which will improve SNR), you just replace analog with digital amplification (which is a problem due to ADC noise and limited precision, as you only use a fraction of the 14 bit resolution, but can be better, depending on the analog amplifier noise).

Yes you are right I noticed that the whole story about setting ISO 400 as the base ISO from Canon which will help for reducing noise in the shadows (without tweaking anything) and with losing some of dynamic range (clipped highlights) and we can restoring already some of this highlights (some of that dynamic range)  by tweaking down ADTG 8882 to 8888 the analog gains registers .

I did 400 to 100 ISO only for matching up the two exposures in normal 100 ISO and with the trick but then it was not okay to do that because of pink highlights I couldn't fix it in some cases. So it's better to find the sweet spot as you mentioned and stop there :D .

But I am confusing how Preamps registers are working ? what it does in 5D3 for enhancing the shadows ?
700D 1.1.5 | no more ISOless LV err 8

jpegmasterjesse

  • Freshman
  • **
  • Posts: 87
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #981 on: July 28, 2018, 06:32:54 PM »
I've been enjoying keeping up with this thread, but I'll admit that changing registers is a bit above my skill level. I'd really appreciate a step-by-step to try some of this out on my 5d2.


50mm1200s

  • Member
  • ***
  • Posts: 177
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #982 on: July 29, 2018, 06:18:05 AM »
Best settings from @Audionut and @a1ex, for 5D3:
https://www.magiclantern.fm/forum/index.php?topic=10111.msg97743#msg97743
https://www.magiclantern.fm/forum/index.php?topic=10111.msg104278#msg104278

"Rules":
Quote
ADTG gains and SaturateOffset can be used to recover some more highlight detail.
changing only ADTG gain is enough (because the other one runs out of range much quicker).
At some point, there's no more highlight detail to recover. When this happens, the white level will begin to decrease. Let's call this the sweet spot, and it can be found easily with binary search, for example.
Quote
Once you've decreased the gain enough that your white level is dropping, you've already gone too far.

For 5D2:
Quote
SaturateOffset (0xc0f0819c) from 0x66f to 0x66f + 32 - 1024 - 624 and B/W offset (0xC0F08034) from 0x1991 to 0x1991 + 624. Also fix the digital gain (0xc0f08030) to 0x1000.
The effect of SaturateOffset is that it expands the recorded range while bringing in more highlight detail. So, if your default range is say 1024...15760 and you increase this range to say 32-16383 without changing digital gain (these are the 5D2 values), this would be equivalent to scaling ADTG gain to (15760-1024) / (16383-32) = 0.9 (that is, -0.15 stops)

jpegmasterjesse

  • Freshman
  • **
  • Posts: 87
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #983 on: July 31, 2018, 01:45:38 AM »
Appreciate it! However, I am getting some errors when I load raw_diag.mo and adtg_gui.mo

https://drive.google.com/file/d/1k9bWM2Wz3gorjws2lC36iam9-ngTwFau/preview

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #984 on: July 31, 2018, 04:27:02 AM »
Found more ISO-related gain

Couldn't seem to get anything useful here.  Another that reduces DR by 0.3 is ADTG2[6] 0x0 > 0x1
Back when, I found a few that darkened the image considerably, but I don't recall if I ever mentioned them here.


For those following along with a 6D.


Code: [Select]
ADTG2[8]  0x56 > 0x36
ADTG2[9]  0x55 > 0x35
ADTG2[a]  0x57 > 0x37
ADTG2[b]  0x57 > 0x37

CMOS[6]  0x0 > 0x2

ADTG2[8882] 0x400 > 0x330
ADTG2[8884] 0x400 > 0x330
ADTG2[8886] 0x400 > 0x330
ADTG2[8888] 0x400 > 0x330

ISO digital gain 0x200 > 0x229

I seem to recall CMOS[0] netting about 0.1 DR, but I'll have to dig through this thread to find what value was the winner.

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1466
  • 6d - Nightly build user
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #985 on: July 31, 2018, 12:35:08 PM »
@Audionut
Interesting, the values for improving DR on the 6d.
Fiddled a little with it, but results were all rather small,   :P
Are you sure you need to change CMOS 6 form 0x0 to 0x2 ?
CMOS 6 is used for vertical position in crop record...

The values for ADTG2 8,9,a & b and 8882 to 8888 are interesting, I guess this could be used on all iso until iso 6400, right ?
That is, if you want maximum DR and not exact proper corresponding ISO/exposure values.

EDIT,

Weird, my standard values for 8,9,a and b registers are much lower than yours  ???
I have values of about '4B' and not '56'.
Put an extra tab in the 6d sheet and write out standard values of the registers for all iso's.
https://docs.google.com/spreadsheets/d/1iapLI7UrgfCJGwPSFsyhYeKl8fSTY4RVXvDR7MiaOp4/edit#gid=837197681

Kharak

  • Hero Member
  • *****
  • Posts: 888
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #986 on: July 31, 2018, 09:09:48 PM »
Anyone have experience with ISO 66 on 5d3. How is it?

I remember a1ex got a small increase of DR about 0.2 in video and I think 0.5 in photo on ISO 66. I’d also like it for going under ISO 100, to possibly avoid having to put ND on. I cant seem to find that post.

Just read this article from imaging-recource.com about how the biggest pride of the Nikon sensor development team was that they managed to get the D850 sensor to go to ISO 64 at its lowest. Its a really good read for anyone interested in Sensor development and production. https://www.imaging-resource.com/news/2018/07/17/pixels-for-geeks-a-peek-inside-nikons-super-secret-sensor-design-lab
once you go raw you never go back

70MM13

  • Member
  • ***
  • Posts: 235
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #987 on: August 01, 2018, 12:59:15 AM »
ISO 74 is the lowest I get while keeping the full range without losing highlights.

It's a pleasure to work with when there's lots of light.

My preference is for darker scenes, but the more I'm tinkering with the experimental ISO, the more comfortable I'm getting with a wider variety of conditions...

This frame was saved from mlvapp, in standard mode, no lut, no filters, no noise reduction, etc.  Simple exposure adjustment and low strength to raise the shadows a touch.

Looks good to me!


Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #988 on: August 01, 2018, 02:38:31 AM »
Are you sure you need to change CMOS 6 form 0x0 to 0x2 ?

I seem to recall a1ex mentioned this one worked in the shadows, vs other registers that affect highlights.  But I can't find the post.
Here is CMOS[6] 0x0 > 0x2



The values for ADTG2 8,9,a & b and 8882 to 8888 are interesting, I guess this could be used on all iso until iso 6400, right ?
That is, if you want maximum DR and not exact proper corresponding ISO/exposure values.

From my quick experiments, you should be able to hit a sweet spot with 8,9,a,b that works at all ISO's.  The sweet spot for 888? needs adjusting for all ISO's if you want maximum results.  Generally higher ISO's require higher values.  With 888?, you can simply find a value that works at all ISO's, with slightly (emphasis on slightly) less then optimal results.

There's some initial work by a1ex to determine what amps are first in the processing chain, berried (follow the links) in the link below.

0xFE should already be at 0x0 on 6D.

Don't forget the hidden ISO's.

edit:  Corresponding ISO values are of no concern to me.  I like saving those photons and not letting them get clipped.

50mm1200s

  • Member
  • ***
  • Posts: 177
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #989 on: August 02, 2018, 07:24:28 PM »
I need some help here (disclaimer: newbie here). After reading all this thread and making some notes, I flashed a @dfort's build for 50D with the iso_research branch. Basically, from what I understood:
- Take a picture of a light bulb with normal settings. Use raw_diag.
- Enter menu, expand the BL and WL
- Use ADTG gains and SaturateOffset until the WL starts to decrease
- Increase DFE gain a bit
- Take another picture with these settings. Use raw_diag. Probably able to gain >0.5 on DR.

It seems easy, but on the "ADTG Registers" menu, I don't see how to change BL, WL and SaturateOffset you guys talk on this thread. How can I do that? Also, you folks write using EV values (like "decrease adtg to -0.37EV"), but I can only see registers (e.g, 0x344). I have to calculate these values manually? Here is what's on the menu:

Code: [Select]

ADTG1[105f]N
ADTG1[1061]
ADTG1[1172]
ADTG1[1173]
ADTG1[1178]
ADTG1[1179]N
DFE[1d02]
DFE[1d04]
DFE[1d06]
DFE[1d08]
CMOS[0]
CMOS[1]
CMOS[2]
CMOS[3]
CMOS[4]
CMOS[5]
CMOS[6]
ADTG1[9]
ADTG2[9]
ADTG1[b]
ADTG2[b]
ADTG1[13]
ADTG2[13]



I'm sure many people already suggested that, but, wouldn't be possible to create a script to test all the possible values (brutal force)?

I see there's some pressure here about making this feature into mainstream ML, but it's kinda difficult to help the research on the current state. Even people willing to spend hours reading can't easily get into it.
Based on the practical examples from this thread, this feature would have a high impact on the final result.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #990 on: August 03, 2018, 05:09:35 AM »
I need some help here.........It seems easy,

Looks can be deceiving.

What's the current state?
Research. We are trying to optimize the parameters that influence ISO, understand their effects (did we really gain 0.5 stops of DR or are we just daydreaming?) and port the results on other cameras..........


Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!

Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?"

But if you have some basic coding/math skills - enough so you can follow the entire discussion without getting dizzy - I have some nifty research tools for you:

Don't assume that "we" know what we are doing, just because "we" post about the subject.


but on the "ADTG Registers" menu, I don't see how to change BL, WL and SaturateOffset you guys talk on this thread. How can I do that?

These registers should be labeled in the camera menu, as per this code snippet.
https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/adtg_gui/adtg_gui.c

Code: [Select]
{0xC0F0,   0x8034, 0, "Black level in LiveView / BW offset in photo mode (SHAD_PRESETUP)"},
    //~ {0xC0F0,   0x8038, 0, "ISO related?"},
    //~ {0xC0F0,   0x8050, 0, "ISO related?"},
    //~ {0xC0F0,   0x814c, 0, "ISO related?"},
    {0xC0F0,   0x819c, 0, "Saturate Offset (photo mode) (HIV_POST_SETUP)"},
    {0xC0F1,   0x2054, 0, "White level?"},


Also, you folks write using EV values (like "decrease adtg to -0.37EV"), but I can only see registers (e.g, 0x344). I have to calculate these values manually?

At this stage, yes.  If you understand the maths really well (I do not), I think it can be calculated.  Otherwise, use raw_diag to monitor the DR as you change the registers.

I'm sure many people already suggested that, but, wouldn't be possible to create a script to test all the possible values (brutal force)?

This would make life so much easier.  Sounds like a coding nightmare.


I see there's some pressure here about making this feature into mainstream ML, but it's kinda difficult to help the research on the current state. Even people willing to spend hours reading can't easily get into it.
Based on the practical examples from this thread, this feature would have a high impact on the final result.

See the first quote at the top of this post from a1ex. 

"Pressure" to put this feature into mainstream, isn't the same as doing what is needed to put this into mainstream.  It is very frustrating seeing such a wonderful feature not reaching it's peak value to the community, and being unable to help.  We should be mindful of not putting pressure on others to perform functions that we are otherwise unable too.

I am quietly confident that people such as a1ex can accomplish this task within a reasonably short timeframe.  I am extremely confident that a1ex is his own person, with his own desires and expectations, who has a keen understanding on keeping this development project (ML) moving in the best possible direction.  ML is more then just this single feature.  So for me, personally, I sit patiently, helping where I can, and respect the choices made by others within this project, especially when those choices made by others do not seem to fit within my own expectations of where the project should be heading.

Everything about this feature needs reverse engineering.  It would be extremely easier if Canon sent us a copy of their documentation on these registers.  But..........such is life.

50mm1200s

  • Member
  • ***
  • Posts: 177
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #991 on: August 03, 2018, 05:58:47 AM »
I know this is an ongoing research and I'm familiar with the open source model. However:

"Pressure" to put this feature into mainstream, isn't the same as doing what is needed to put this into mainstream.

I'm not putting pressure. I said "I see there's some pressure here".
My point being: if more collaboration from the community is needed to make things faster, then it should be somewhat more accessible for people without programming/math skills to do the tests and share them.

Quote
Otherwise, use raw_diag to monitor the DR as you change the registers.

So, currently, you need to change each register, press HalfShutter to activate raw_diag and just then see the changes? This would take too much time, even for a simple test.
Since the brutal force idea seems to be too complicated, can't this be printed inside the menu (e.g "Current WL is xxxx", and so on), using a automatic low resolution silent pic as base for raw_diag calculation everytime a register is changed? Like a daemon in a operating system. DIGIC is not that powerful, but if the pic is lowres enough this might be possible, no?

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #992 on: August 03, 2018, 10:59:00 AM »
I'm not putting pressure. I said "I see there's some pressure here".



My point being: if more collaboration from the community is needed to make things faster, then it should be somewhat more accessible for people without programming/math skills to do the tests and share them.

I'm pretty sure you missed the message I was trying to hammer home with the below quote.

My bold for emphasis.
I am quietly confident that people such as a1ex can accomplish this task within a reasonably short timeframe.  I am extremely confident that a1ex is his own person, with his own desires and expectations, who has a keen understanding on keeping this development project (ML) moving in the best possible direction.  ML is more then just this single feature.  So for me, personally, I sit patiently, helping where I can, and respect the choices made by others within this project, especially when those choices made by others do not seem to fit within my own expectations of where the project should be heading.

The point isn't to ask for assistance, so that you can assist.  There are a bunch of other tasks that this entire development project has outstanding.  Almost everybody can assist with some outstanding project tasks, now, which will free spare time for specific tasks, such as this feature, by those people with the skills to make it happen.  There is plenty of information available, describing how people can help the Magic Lantern development project (my signature should get the reader started).

Programming and maths are subjects that people spend a significant amount of money (and time) on, in educating themselves, on a daily basis.  People should not expect a bunch of guys playing around in their spare time, on a hobby project, without financial assistance, to educate others in these subjects.  That takes time!  The time being spent on this project by it's collaborators is already spare, not plentiful.


This would take too much time, even for a simple test.

Yes, we know it does.  I personally have a couple thousand shutter actuation's on my hardware, from testing this feature.

What does this register do?
Tweak it.
Hit shutter.
Quickly observe raw zebras before the overlay is removed.
Observe raw_diag feedback.
Observe image taken.
Tweak register again.
Hit shutter.
Quickly observe raw zebras before the overlay is removed.
Observe raw_diag feedback.
Observe image taken.

Oh shit, I think I found a register that does something cool!

Repeat the above process some countless number of times.
Turns out to be nothing.............or maybe it turns out to be something, document as much as possible, present the data to people with more knowledge.

Move to the next register address.
Rinse and repeat.

I don't even bother to track the amount of time spent, I would feel guilty for my family.

Since the brutal force idea seems to be too complicated, can't this be.....

Be careful, if you make it sound to easy, it will start smelling like an "easy coding task"*.

*otherwise known as low hanging fruit.  A fairly simple coding task that does not have a significant effect on the development project, and is thus left as a "low hanging fruit" to encourage those with less skills to get involved.

70MM13

  • Member
  • ***
  • Posts: 235
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #993 on: August 03, 2018, 01:19:16 PM »
I find this functionality essential to my work, even though I only use the parameters on the ISO registers GUI.  I'm getting great results from using those, even if it could be better (what couldn't?)

That being said, it's extremely tedious and sometimes very frustrating to change the base ISO and redo the whole process just to be able to get a shot under much different lighting conditions.

All I ask is for information on any possible way to store and retrieve settings for faster setup in use.

Is it possible?

The ideal under current (and seemingly for years to come) circumstances would be "ISO presets" from the GUI, perhaps with the ability to provide a name for each to avoid confusion in the field.

I'm just asking for information.

Please keep it friendly.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #994 on: August 03, 2018, 05:49:04 PM »
That being said, it's extremely tedious and sometimes very frustrating to change the base ISO and redo the whole process just to be able to get a shot under much different lighting conditions.

All I ask is for information on any possible way to store and retrieve settings for faster setup in use.

Is it possible?

Take the iso_regs module....
iso_regs.mo - 5D3 only, requires either the crop_rec_4k build, or a custom ML build from the iso-research branch:

A research tool (or hacker's tool if you prefer) that lets you change most ISO-related parameters on 5D Mark III only and study their effect. Details here.


Adapt it for your camera.  Add in preset values to the menu entries.
The LUA scripting module may be able to be used also, but I'm not entirely sure.

I like answering these questions, because I can provide answers.  I can't solve lack of skill with a swift hand movement (because I lack skills also), but I can provide assistance.  Perhaps the assistance is a little to blunt and to the point, but we're all fallible..............so.........

I do have a problem with being able to ignore the not so smart questions.  See below.

The ideal under current (and seemingly for years to come) circumstances would be "ISO presets" from the GUI, perhaps with the ability to provide a name for each to avoid confusion in the field.

I'm just asking for information.

This sounds like a thinly veiled "when will it be released" question.

Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!

Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?"



When it will be released?

When it's ready. I also want to summarize the findings in a small paper (like the Dual ISO PDF), so I need a little time.

https://wiki.magiclantern.fm/faq#any_progress_on_xyz
https://wiki.magiclantern.fm/faq#when_will_you_implement_xyz
https://wiki.magiclantern.fm/faq#when_will_you_release_the_next_version
http://www.catb.org/esr/faqs/smart-questions.html

Please keep it friendly.

http://www.catb.org/esr/faqs/smart-questions.html#not_losing
Quote
Odds are you'll screw up a few times on hacker community forums — in ways detailed in this article, or similar. And you'll be told exactly how you screwed up, possibly with colourful asides. In public.

When this happens, the worst thing you can do is whine about the experience, claim to have been verbally assaulted, demand apologies, scream, hold your breath, threaten lawsuits, complain to people's employers, leave the toilet seat up, etc. Instead, here's what you do:.............

Community standards do not maintain themselves: They're maintained by people actively applying them, visibly, in public. Don't whine that all criticism should have been conveyed via private e-mail: That's not how it works. Nor is it useful to insist you've been personally insulted when someone comments that one of your claims was wrong, or that his views differ. Those are loser attitudes..............

There have been hacker forums where, out of some misguided sense of hyper-courtesy, participants are banned from posting any fault-finding with another's posts, and told “Don't say anything if you're unwilling to help the user.” The resulting departure of clueful participants to elsewhere causes them to descend into meaningless babble and become useless as technical forums.

Exaggeratedly “friendly” (in that fashion) or useful: Pick one.............

Remember: When that hacker tells you that you've screwed up, and (no matter how gruffly) tells you not to do it again, he's acting out of concern for (1) you and (2) his community. It would be much easier for him to ignore you and filter you out of his life. If you can't manage to be grateful, at least have a little dignity, don't whine, and don't expect to be treated like a fragile doll just because you're a newcomer with a theatrically hypersensitive soul and delusions of entitlement.

Ron100

  • New to the forum
  • *
  • Posts: 7
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #995 on: August 08, 2018, 12:11:03 PM »
Dear friends,are there ADTG values (8, 9, a, b and 8882 - 8888)for maximum highlights saving in photo mode??
on 6D

ghost0cnc

  • New to the forum
  • *
  • Posts: 2
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #996 on: August 21, 2018, 11:28:19 PM »
Yesterday I encountered serious vertical fixed-pattern banding on some images. On other images there was no banding at all.

5D Mark III 1.2.3 + ML crop_rec_4k.2018Mar10.5D3123
Canon EF80-200mm f/2.8L (very old lens)

Same exposure settings and iso_regs configuration on all images, iso_regs modified to enable the "CMOS-Patch".
AI Servo, 1/1000s, f/2.8, Canon ISO 3200, tweaked with the following overrides:

Code: [Select]
CMOS[3] = 2372 (0x944)
CMOS[4] = 792 (0x318)

ADTG-0xFE = 4
ADTG-Preamp (8/9/A/B) = 2 (2 is the lowest value that does not cause banding due to an underflow of one of the other parameters)
ADTG-Gain (8888) = 950
SaturateOffset = 1200 (caused some green tint issues on a few older images)


Top 800px of each image at +2EV and the same white-balance applied, images 1 + 2 with banding, 3 without:


Top "optical black" bars of the three images (dcraw -T -E -W -b 1 -g 50 500)
The fixed-pattern noise is pretty visible. The last ob-image (without banding) is far brighter - if these dcraw settings are not useful for a comparison, please tell me better settings and I will update the image.

Download the RAWs (90 MB, zip)



Do you have an idea what could have caused the fixed-pattern banding?

Brainstorming:
- Bad iso_regs settings
  -> critical value for SaturateOffset (?), but this should change the tint
  -> variations of ADTG-Preamp, if the offsets change and I fix the first vaule to 2, it could possibly cause an underflow of another value

- Electromagnetic interferences from the lens
  -> would cause horizontal banding (?)
  -> no fixed pattern

- Mobile-phone interferences
  -> also horizontal (?), non-fixed-pattern noise

- Voltage fluctuations due to AF-motor power-consumption

- Aliens



In this case a normal dark-frame subtraction will not work, as the intensity of the FPN exceeds the amount of FPN I could get from a "normal" dark-frame.

As the approach of correcting banding using data from the optical-black bars was mentioned earlier in this thread - is there a banding correction tool for CR2 files available yet?
If not, I will have a look at it. Just need to get into RAW de- and encoding.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #997 on: August 22, 2018, 07:47:22 AM »
Is the FPN only in the shadows?  Or only most notable in the shadows?
One of the effects of reducing read noise, the fixed pattern noise becomes even more distinguishable (the percentage of FPN vs other noise sources increases).

http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html#patternnoise
Quote
This gives an indication of how visually disruptive pattern noise can be -- even though the fixed pattern noise is only about 20% of the overall noise, it is quite apparent because our perception is adapted to picking out patterns, finding edges, etc.


70MM13

  • Member
  • ***
  • Posts: 235
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #998 on: August 24, 2018, 07:45:18 PM »
Here's a sample frame of the results I'm getting with my current settings, emphasis on highlights and maximum dynamic range.

Based on ISO 200.

It's working well in every scenario so far, from screaming sunlight to dark scenes.


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12223
  • Maintenance mode
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #999 on: August 25, 2018, 07:55:30 PM »
Started to write a generic register overriding interface. Goals:
- to be used for other features as well: FPS override, crop_rec (for now)
- allow any modules or core features to override any registers (ADTG, CMOS, ENGIO, DFE, whatever else we'll find in the future)
- allow more than one module to override the same registers without "fighting", e.g.:
  - if dual_iso changes CMOS[0], the ISO tweaks module should be able to override CMOS[7], for example [easy]
  - if dual_iso changes CMOS[0] and the ISO tweaks module decides to use some custom gains in CMOS[0], that should work as well
  - FPS override and crop_rec should work well together (for some definition of "well"... TBD)
- allow custom overriding logic
- do that fast (LiveView doesn't like slow overrides; searching in a big list for every single ADTG or ENGIO register is likely overkill)
- minimally invasive (install/uninstall the hooks only when actually used)
- compatible with old and new models (DIGIC 4 and 5 for now, keeping in mind DIGIC 2, 3, 6, 7 and 8 )

Initial draft, for review:

Code: [Select]
/* Register overrides */

/* Core APIs for overriding various registers:
 * - ADTG
 * - CMOS
 * - DFE
 * - ENGIO
 * - ...
 */

/* Each register type has an "add" function, which can be used to set up certain overrides,
 * and a "del" function, which "uninstalls" previous overrides (identified by what "add" returned).
 *
 * Optional features: check for expected values or call back some CBR.
 */

/* Each "add" function has the following parameters:
 * - register "target" (optional, if there are more chips of the same type, see e.g. ADTG "destination")
 * - register address (each register has some unique identifier)
 * - register mask (which address bits should be ignored, e.g. to override more than one register)
 * - new value (after overriding)
 * - expected value (with a boolean flag to enable this check)
 * - optional CBR for custom overriding logic (if not NULL, the "new value" is ignored)
 * and returns a positive ID if successful, or negative error code if not.
 *
 * Each "del" function takes one parameter (the ID returned by "add")
 * and returns 0 on success, or negative error code on failure.
 */

#ifndef _regs_h_
#define _regs_h_

typedef uint16_t (*adtg_override_func)(
    uint8_t dest,       /* ADTG chip "destination" (bit field, e.g. 6 will override both ADTG2 and ADTG4) */
    uint16_t reg,       /* ADTG register ID (0x800C, 0xFE etc) */
    uint16_t old_val    /* value before overriding */
);                      /* returns: updated value (or old_val if nothing changes) */

typedef uint16_t (*cmos_override_func)(
    uint8_t reg,        /* CMOS register ID (0, 1, 2 etc) */
    uint16_t old_val    /* value before overriding */
);                      /* returns: updated value (or old_val if nothing changes) */

typedef uint32_t (*engio_override_func)(
    uint32_t reg,       /* ENGIO register address (0xC0F06014 etc) */
    uint32_t old_val    /* value before overriding */
);                      /* returns: updated value (or old_val if nothing changes) */

typedef uint32_t (*dfe_override_func)(
    uint16_t reg,       /* DFE register address (0x1D02 etc) */
    uint16_t old_val    /* value before overriding */
);                      /* returns: updated value (or old_val if nothing changes) */

int regs_ovr_adtg_add(uint8_t dest, uint16_t reg, uint16_t mask,
                      uint16_t new_val,
                      adtg_override_func cbr, void * opaque);

int regs_ovr_cmos_add(uint8_t reg, uint8_t mask,
                      uint16_t new_val, cmos_override_func cbr, void * opaque);

int regs_ovr_engio_add(uint32_t reg, uint32_t mask,
                       uint32_t new_val, engio_override_func cbr, void * opaque);

int regs_ovr_dfe_add(uint16_t reg, uint16_t mask,
                     uint16_t new_val, dfe_override_func cbr, void * opaque);

int regs_ovr_adtg_del(int id);
int regs_ovr_cmos_del(int id);
int regs_ovr_engio_del(int id);
int regs_ovr_dfe_del(int id);

/* TODO (if the above will work well enough):
 * - TFT SIO registers?
 * - HDMI chip registers?
 * - Sound registers? (integrate with new-sound-system?)
 * - Other peripheral registers?
 */

#endif /* _regs_h_ */