CMOS/ADTG/Digic register investigation on ISO

[timbytheriver]

@names_are_hard Thank you for your comprehensive answer. I will have to reflect awhile on the information to understand it fully.

@Audionut Thanks! I will explore the 'SET' button option. I am conscious that there are thousands of registers as yet unknown! So maybe the values I am trying to find are amongst those.

**UPDATE**
Used adtg_gui to spy registers: > 'Changed since now' and 'ISO related'
If anyone is looking for these values they report as follows:

Code: [Select]

ADTG24[7] = 0x0
ADTG24[8] = 0x5a
ADTG24[9] = 0x5a
ADTG24[a] = 0x5a
ADTG24[b] = 0x5a
ADTG24[1b] = 0x320

They've no doubt been discussed before – but possibly way back in the thread.

[Audionut]

In adtg_gui.c there are no hardcoded values for registers.  adtg_gui.c is looking up the values (as currently set by Canon) and displaying them to the user.  From what I can gather, however adtg_gui does that is your answer.

iso_regs is an old attempt to bring register tweaking to a more user friendly level.  It's 5D3 only and contained hard coded values because it was useful for the specific task.  Those values were determined from looking at what adtg_gui displayed.

They've no doubt been discussed before – but possibly way back in the thread.

You haven't read the entire thread.....shame. 

Might find something useful here too: https://www.magiclantern.fm/forum/index.php?topic=6751.0

[timbytheriver]

Thanks! More reading! lol

For the record: I have indeed read this entire thread through – twice. It just takes a long time to filter through me!

[timbytheriver]

Looking at your code, I think you have an integer overflow bug when you're setting registers using 0x400 + reg_x pattern.  Because the addresses you're assigning to are later considered to be 2 bytes wide, and menu input + the constant could make this higher than uint16_t max.  This could lead to menu choices giving the user a radically different result than they expect.  You should probably clamp the value to between 0 and uint16_t max.  Also since your variable is signed, underflow will lead to sign extension and really mess your values up.

@names_are_hard So far, with the code as-is I'm seeing the kinds of changes I'd expect to the regs targeted. With the integer overflow bug, are we talking values that the user thinks they're inputing at 10 are actually [say] 100? Is that the sort of discrepancy you outline?

[names_are_hard]

Tim - read around "integer overflow" if you want to understand the problem, it's too off topic to go into here.  The short version is if you do user_value + 0x401 and the user puts in 0xfc00, the result is not 0x10001 as you'd expect, but 0x1.  I guess there's some risk when assigning unexpected values to control registers.

[timbytheriver]

Audionut on March 02, 2014, 03:20:11 PM
Does iso_regs scale [8,9,A,B], keeping the deltas between them?

a1ex
Yes. It considers the first one as reference (that one will get the value from menu) and all the others as delta (so, when you set it to 0 or 1 or maybe even 2 or 3, it may underflow and cause strong banding). I preferred this underflow instead of clamping to 0 because in this way you see when you went too far. However, ADTG gains 888x are not scaled in iso_regs (these are simply set all of them to the same value).

I'm reviewing whether the decimal adjustments in my Q iso-regs menus are using the correct math. Line 1205 > https://bitbucket.org/rivertim/magic-lantern-danneclone/src/ddebb19831d7ce7472bb80db9926bc69b3bc2d6d/modules/crop_rec/crop_rec.c

Is the "deltas between them" described by @a1ex and @Audionut in this context meaning to keep the integer relationship constant? e.g.

Say for: ADTG2 8,9,a,b values = 0x60,0x61,0x62,0x60 meaning x, +1, +1, -2 Do the relationships scale as they change?

Sorry for math duh-ness! 

**EDIT**

a1ex
However, ADTG gains 888x are not scaled in iso_regs (these are simply set all of them to the same value).

I notice in adtg_gui my values for these (888x) registers are different:

Code: [Select]

ADTG2[8882] = 0x436
ADTG2[8884] = 0x438
ADTG2[8886] = 0x439
ADTG2[8888] = 0x43a

ADTG4[8882] = 0x437
ADTG4[8884] = 0x437
ADTG4[8886] = 0x437
ADTG4[8888] = 0x43a


Should they in fact all be set to the same value?

Thanks!

[a1ex]

Short answer: everything from this thread needs to be taken with a grain of salt (as none of us is expert in sensor calibration), so it's probably best to verify each hypothesis from scratch.

The 5D3 reads out 8 columns at a time (that is, 8 analog circuits operating in parallel), and the 888x gains (4x2 = 8 values) are for each group of columns. So, one gain affects columns 1, 9, 17 etc, another gain affects columns 2, 10, 18 etc, and so on. The 8,9,a,b are IIRC similar. These values should be tweaked so all these 8 groups of columns have consistent output; otherwise, you get vertical stripes / banding. Other cameras may read out a different number of columns at a time (usually 4 or 2), so you'll see fewer registers there.

The 888x registers are linear gains (i.e. if you set them to 1000 vs 2000, that's a difference of 1 EV), so it's probably best to scale the factory values (camera-specific) by the same constant. The 8/9/A/B gains are adjusted in fractions of EV (about 0.0059 EV/step?), so it's probably best to keep the deltas constant.

Now, since vertical stripes also appear at default configurations at least on 5D3, it's probably best to re-calibrate all these gains from scratch somehow. Related work.

[timbytheriver]

@a1ex Many thanks for clarification and links.

These values should be tweaked so all these 8 groups of columns have consistent output; otherwise, you get vertical stripes / banding…

I've found this also true of the ADTG2/4 0xFE register. Uneven values between the banks give a similar result.

[timbytheriver]

I'm trying to target Digital Gain regs with an adjustment. I find this in iso_regs:

Code: [Select]

if (reg == 0xc0f37ae4 || reg == 0xc0f37af0 || reg == 0xc0f37afc || reg == 0xc0f37b08)
        {
            default_digital_gain = val;
            if (digital_gain) *copy_ptr = digital_gain;
        }

I've tried:

Code: [Select]

adtg_new[40] = (struct adtg_new) {6, 0xc0f37ae4, 0x200 + gain_dgain};
adtg_new[41] = (struct adtg_new) {6, 0xc0f37af0, 0x200 + gain_dgain};
adtg_new[42] = (struct adtg_new) {6, 0xc0f37afc, 0x200 + gain_dgain};
adtg_new[43] = (struct adtg_new) {6, 0xc0f37b08, 0x200 + gain_dgain};

or

Code: [Select]

adtg_new[44] = (struct adtg_new) {6, 0x8030, 0x200 + gain_dgain};
Do any of these reg addresses look correct?

Cheers!

[Levas]

Was messing with ADTG register override module and found some interesting registers for video mode (Not sure if these were mentioned before  ).
2 registers bump the gain, without dynamic range penalty!
Edit: False alarm, the registers seemed to drop frames per second, from 25 to about 8 fps, so while ML was reporting 25fps based on timer registers, the recordings were in 8fps, so more light hitting the sensor...
combined they give about 1.66 stop of extra light, without sacrificing dynamic range, works on all iso's.
Not sure what the trickery is done in the background, but rolling shutter becomes huge with these 2 registers override.



Now comes the weirdness, the registers only become visible in ADTG register override module, after a photo or video is taken.
To further investigate, I made a crop_rec preset with these two registers overriden, but they don't get activated the first time.
Crop_rec preset is loaded, nothing happens, record one video and after that the settings kick in, why doesn't it work the first time without having to record a video first ?

Also tried to take photos with these settings, but photos are messed up, purple or green color, sometimes even messed up frame, timing issues or something.

I don't know how values look on other cams, but I lowered the most left digit of these values until live view becomes brighter.



Side by side comparison, same ISO, same Shuttertime and same Diaphragm Aperture  . Left with ADTG2 registers 23 and 24 overrides, on the right, basic values (with 1.66 exposure correction in post).

Here are the two dng frames on google drive:
https://drive.google.com/open?id=1mwtcX7FVyrJTth-Xjqpr0-cdIJxJj9XE
https://drive.google.com/open?id=11MEW5mOHFBXrBlrssz7uTj0FJx5-Xzj6

[a1ex]

2 registers bump the gain, without dynamic range penalty!
combined they give about 1.66 stop of extra light, without sacrificing dynamic range, works on all iso's.
Not sure what the trickery is done in the background, but rolling shutter becomes huge with these 2 registers override.

Confirmed on 5D3 - really interesting. It appears to reduce the frame rate as well.

ADTG[23]: default 0xF00, setting to 0x200 or some other magic values will cut the frame rate in half*, doubling the exposure time.
ADTG[24]: default 0x2d1e, setting to 0x1d1e will cut the frame rate in half*, doubling the exposure time.
* changing both registers together will result in 1/3 of the original frame rate (e.g. 24 -> 8 fps), and 3x exposure time.

This could explain the results, as the dynamic range does not change significantly with shutter speed.

In addition, changing any of those 2 registers in photo mode appears to have no effect (possibly because of the mechanical shutter), but changing both of them together results in garbled image (possibly timing issues because of slower readout). With FRSP, changing any or both of those 2 registers results in garbled image (?!)

Definitely worth a closer look, in particular at higher ISOs sorry, false alarm

[Danne]

Tried some on eosm but couldn't get these results.
Reducing framerate, hmm...

[Levas]

changing both registers together will result in 1/3 of the original frame rate (e.g. 24 -> 8 fps), and 3x exposure time.

You're right, only checked individual frames yesterday, didn't process them to a video file.
Now I did test some more and processed to a video file, and it's indeed about 3 times slower fps...
That explains why the exposure is so much better...damn was hoping to have found some magic register which could gave me ultimate low light performance on iso6400 

But it's interesting to know that there are other registers then A-timer and B-timer that influence fps...now if it only could speed up instead of slow down fps...

[names_are_hard]

ADTG[24]: default 0x2d1e, setting to 0x1d1e will cut the frame rate in half*, doubling the exposure time.

The obvious first test for increasing frame rate would be ADTG[24] = 0x4d1e.  But don't blame me if it makes your cam explode   0x3d1e would be my second choice.

[Levas]

Obviously tried, but didn't work, couldn't increas fps speed with these two registers 

[Icaab]

Hello. Has anyone tried this on camera 650d?

[ro xl]

How can i find the readme for this module? I'm trying to use it on my 550D to inspect noise values at diffrent settings, and i get weird values like -2.11 eV read noise, 278051 e full well capacity, or 1006e read noise. How am i supposed to use it, there in no guide and in camera hints aren't very useful. I can't access the readme because bitbucket isn't working. Any ideas?

[Audionut]

The readme's are here, but they won't give you what you want.

You're going to need to read the recommendations in the OP, and ask questions after.

Any recommended reading?
http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html
http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html (main site: http://www.clarkvision.com/articles/ )
http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity

Happy reading. 

[dream951]

Hi guys!
Is it possible in any build to take pictures from liveview with isoregs with an intervalometer.
This does not work for me, it turns out only a raw frame without processing by isoregs in build crop_rec_4k.2018Jul22 and I get only one frame at a resolution of 1920x1248 with isoregs, but then liveview resets the effect and all subsequent frames I get with the iso I chose for setting isoregs in build crop_rec_4k_mlv_snd_isogain_1x3_presets_2021Apr01.
It would be very cool for time lapses.
Thanks

[dream951]

Yesterday I encountered serious vertical fixed-pattern banding on some images. On other images there was no banding at all.

5D Mark III 1.2.3 + ML crop_rec_4k.2018Mar10.5D3123
Canon EF80-200mm f/2.8L (very old lens)

Same exposure settings and iso_regs configuration on all images, iso_regs modified to enable the "CMOS-Patch".
AI Servo, 1/1000s, f/2.8, Canon ISO 3200, tweaked with the following overrides:

Code: [Select]

CMOS[3] = 2372 (0x944)
CMOS[4] = 792 (0x318)

ADTG-0xFE = 4
ADTG-Preamp (8/9/A/B) = 2 (2 is the lowest value that does not cause banding due to an underflow of one of the other parameters)
ADTG-Gain (8888) = 950
SaturateOffset = 1200 (caused some green tint issues on a few older images)

Top 800px of each image at +2EV and the same white-balance applied, images 1 + 2 with banding, 3 without:


Top "optical black" bars of the three images (dcraw -T -E -W -b 1 -g 50 500)
The fixed-pattern noise is pretty visible. The last ob-image (without banding) is far brighter - if these dcraw settings are not useful for a comparison, please tell me better settings and I will update the image.

Download the RAWs (90 MB, zip)



Do you have an idea what could have caused the fixed-pattern banding?

Brainstorming:
- Bad iso_regs settings
  -> critical value for SaturateOffset (?), but this should change the tint
  -> variations of ADTG-Preamp, if the offsets change and I fix the first vaule to 2, it could possibly cause an underflow of another value

- Electromagnetic interferences from the lens
  -> would cause horizontal banding (?)
  -> no fixed pattern

- Mobile-phone interferences
  -> also horizontal (?), non-fixed-pattern noise

- Voltage fluctuations due to AF-motor power-consumption

- Aliens



In this case a normal dark-frame subtraction will not work, as the intensity of the FPN exceeds the amount of FPN I could get from a "normal" dark-frame.

As the approach of correcting banding using data from the optical-black bars was mentioned earlier in this thread - is there a banding correction tool for CR2 files available yet?
If not, I will have a look at it. Just need to get into RAW de- and encoding.

Hi, how you modify isoregs to enable cmos patch? can you give me link for this module please

[JogaDann]

Hi! I have a 5D2 and the ISO experimental build have an annoying issue.;

It just stucks the camera at a blue + red LED on when I turn it off. When I delete ML files and put another version (eg. the lastest Lua Dec/2020) the problem is gone.

But I need the ISO experiments branch to run a module (hdmi_out.mo), so I really have to solve this. Is there a workarount do deal with it?

Thanks in advance 

[lea84]

I have the same problem with 5D II version 2.1.2 and the iso-research version. After I install "iso-research" Latest Build (2019-12-21 10:24) version on Canon 5D ii and restart the camera, the screen stays black only the red LED light is on. I have to disconnect power so that the camera goes off again. Does anyone have a solution for this?

[Markelian]

Can confirm as well, the 5D2 build is bricked, you get both leds on (red and blue) indefinitely when turning off, also it sometimes crashes during idle (?).
Tried to scrape the thread but red's old builds are down, and the only other build you can load these modules on is the 4k raw one, which 5d2 doesnt have, so basically this project doesnt exist for 5d2 on public internet.

Im not sure if it's that, but i think the both leds on thing might correspond to the state of the "ARM chip being locked in a loop, heating up very rapidly" that's on the wiki. Most i waited for 20s but there's another user on the forum who got the 2 leds on issue and left it a couple of minutes without taking the battery out, and now his camera works on and off (this thread https://www.magiclantern.fm/forum/index.php?topic=3358.0)
Anyway maybe delete it off the list since its not working anyway (and everyone moved on from the project i guess?) , so it doesnt destroy people's cameras in case it does represent the locked ARM processor state.

[names_are_hard]

I've never seen anyone call a build "bricked" before.  Hard to imagine what that would mean.

The post you reference is from 2012, with the user not using this build at all, so I don't think it's relevant.  No, having both LEDs lit doesn't mean the processor is locked up and heating rapidly.

I see no evidence that this is damaging cameras, do you have something specific?  It's explained that ML can crash and temporarily lock up your cam, and that you should remove the battery if this happens.

[Walter Schulz]

Basically it is just as Markelian wrote:
ISO_research build loads fine but fails to shutdown properly with both LEDs on indefinitely. Restart attempt requires battery pull and after that ML will tell you about shutdown issue.
Makes working with this build quite complicated.