Author Topic: CMOS/ADTG/Digic register investigation on ISO (Read 845758 times)

Ron100 · « **Reply #950 on:** February 15, 2018, 03:11:12 PM »

RAW_DIAG - video instruction

Ron100 · « **Reply #951 on:** February 18, 2018, 08:51:28 AM »

how to make iso 6D reduction in Cmos[3] iso100 to iso77?

70MM13 · « **Reply #952 on:** April 14, 2018, 04:18:55 PM »

here's an excellent example of the benefits of amplifier gain optimisation:

the two images are frame grabs from MLV video. no noise reduction applied.

one is using "straight" iso 200, while the other is using 200 dropped to 100 in the usual manner.

check out the difference in the chroma noise on the pant legs and the jacket sleeve. there's none in the optimised shot!

this REALLY needs to become mainstream.

now.

please!

michael84 · « **Reply #953 on:** June 13, 2018, 12:02:06 PM »

I have a problem with the current iso_regs module in the 4k crop_rec build. I can only adjust the ADTG gain in very coarse steps (100 or 200), an the ADTG preamp in steps of five until i am below 30 or so. Is this a bug, or do i do something wrong? I did not have these problems with older builds.

70MM13 · « **Reply #954 on:** June 15, 2018, 05:10:17 PM »

Confirmed here. Same behaviour.
5dmkiii 1.13

I installed it to check, and immediately switched back to the December 2017 build. It works correctly.

70MM13 · « **Reply #955 on:** July 10, 2018, 01:17:03 PM »

It's a real curiosity that this amplifier optimization is being ignored.

Those using dual iso would benefit from this also, because even at canon iso 100, shadows are just plain ugly. Not so after gain optimization!

AHH, the joys of bias and prejudice. Why, I wonder, should it apply to this?

"Cornelius..."

theBilalFakhouri · « **Reply #956 on:** July 10, 2018, 04:41:07 PM »

@70MM13

Can you compare please 100 from 200 ISO (Your technique) vs Dual ISO 100/800 vs normal Canon ISO at 100. And upload DNG sample please for each one.

Thanks!

Edit: Can you even get 100 from 400 ISO,, its there better result ?

70MM13 · « **Reply #957 on:** July 10, 2018, 05:18:58 PM »

Will do.

Yes, you could drop 400 to 100, but 200 is the sweet spot for dynamic range...

Quote from: theBilalFakhouri on July 10, 2018, 04:41:07 PM

@70MM13

Can you compare please 100 from 200 ISO (Your technique) vs Dual ISO 100/800 vs normal Canon ISO at 100. And upload DNG sample please for each one.

Thanks!

Edit: Can you even get 100 from 400 ISO,, its there better result ?

Danne · « **Reply #958 on:** July 10, 2018, 05:34:34 PM »

Yes some comparison would be nice which shows tge clear benefits. No time atm to dig into anything but checking is always possible

70MM13 · « **Reply #959 on:** July 10, 2018, 07:43:49 PM »

ok, here it is!

these images were created by loading the DNGs into mlv app, and dropping the dark strength to zero.

note the aliasing on the dual iso shot. no surprise there...

also note that the optimised "experiments" shot is beautiful. no hassle, no slow post processing, no unexpected behaviours and side effects, etc.

the canon 100 iso... sucks.

CAN WE FINALLY DEVELOP THIS?

??
what the.
i'm done pushing for it. if this isn't enough proof, the world has gone mad.

Don't forget, these results are just from me arbitrarily reducing ISO 200 to 100. Who knows how much better it can get!

No one will know unless this gets some serious attention.

here's the DNGs, zipped: http://drive.google.com/open?id=145YfRJB9lbauqHGJpHTzbD__iBYj4WOt

funny music poems

theBilalFakhouri · « **Reply #960 on:** July 10, 2018, 08:52:10 PM »

Wow! It's Nikon quality in Canon cameras! Dual ISO without dual_iso.mo at least for clean shadows!
I still confusing how I can do this in 700D,, maybe I should read from the first post.

@a1ex.. COME ON

very BIG DEAL.

How we can do this ? In a module or something like "Optimized ISO" in ISO submenu ON/OFF. Is the values should be hardcoded for each camera?

ilia3101 · « **Reply #961 on:** July 10, 2018, 09:52:56 PM »

Nice shadows. How's the highlights doing?

70MM13 · « **Reply #962 on:** July 10, 2018, 10:19:17 PM »

Grab the DNGs and play, or better still, play with the iso experiments tools and help get the ball rolling!

Quote from: Ilia3101 on July 10, 2018, 09:52:56 PM

Nice shadows. How's the highlights doing?

ArcziPL · « **Reply #963 on:** July 10, 2018, 10:34:51 PM »

Quote from: 70MM13 on July 10, 2018, 07:43:49 PM

these images were created by loading the DNGs into mlv app, and dropping the dark strength to zero.

note the aliasing on the dual iso shot. no surprise there...

also note that the optimised "experiments" shot is beautiful. no hassle, no slow post processing, no unexpected behaviours and side effects, etc.

the canon 100 iso... sucks.

Absolutely stunning results!

EDIT: but... after closer look the "optimized" file seems to be amplified by about +0.85EV, i.e. the overall brightness is matched only after pushing the normal ISO100 by +0.85EV from RAW. One can also already notice loss of data in the highlights on the metal cover of the floppy disk on the left. So, unfortunately, it's not comparing apples with apples. I don't say it doesn't have a potential but a better comparison is needed to quantize the difference in DR.

70MM13 · « **Reply #964 on:** July 10, 2018, 10:53:19 PM »

Exactly. But as I've said all along, I'm not doing anything special or analytical. I'm just happy with the results I'm getting, and want to see this taken seriously so we can all benefit from this!

Quote from: ArcziPL on July 10, 2018, 10:34:51 PM

Absolutely stunning results!

EDIT: but... after closer look the "optimized" file seems to be amplified by about +0.85EV, i.e. the overall brightness is matched only after pushing the normal ISO100 by +0.85EV from RAW. One can also already notice loss of data in the highlights on the metal cover of the floppy disk on the left. So, unfortunately, it's not comparing apples with apples. I don't say it doesn't have a potential but a better comparison is needed to quantize the difference in DR.

70MM13 · « **Reply #965 on:** July 11, 2018, 12:46:46 PM »

Before this thread once again fades to oblivion, I'll just provide this final thought:

I'm not looking for reasons NOT to take advantage of this excellent functionality. I'm not shooting pictures of charts or cats. I'm making beautiful video using this experiment ISO, without knowing what I'm doing. I'm just loving the clean shadows and actually using this.

Watch this video, and then you'll see for yourself.

See any aliasing?

I had to use canon ISO for the opening shot because it was filmed at 60fps, and I couldn't get the right results with the iso experiments build. I had to apply extreme denoising to that shot to get a barely acceptable result, but you can still see how bad it is compared to the rest of the video.

Amplifier optimization is a great tool!

I'll just keep enjoying this incredible capability and scratching my head as to why no one else is.

http://drive.google.com/open?id=1RXZ5vrQggaX1EHi1Yeg0hA9gokJmnquc

Audionut · « **Reply #966 on:** July 11, 2018, 01:56:38 PM »

A point has been made.
Further progress will result from development of the code, not beating a dead horse.

Regards,

a1ex · « **Reply #967 on:** July 20, 2018, 07:28:52 PM »

Found yet another ISO-related gain: register ADTG[7]. Works for both stills and LiveView. There are 4 gains, 2-bit each, applied to columns. 5D3 appears to have two ADTG chips (ADTG2 and ADTG4); same values is written to both. Most other models have only one ADTG chip.

Values that give clean image:
- 0x00 (default, all gains set to 0b00)
- 0x55 (all gains set to 0b01)
- 0xAA (all gains set to 0b10)
- 0xFF (all gains set to 0b11)

Didn't evaluate gain values and noise performance yet. The column offsets have to be recomputed after changing this gain (takes a few seconds in LiveView).

And another one at ADTG[0x1B]. This one darkens the image a lot.

Also found a way to override registers on the fly, even if Canon firmware sets them only once during video mode switches. Previously, you had to get out of LiveView and back to see the effect of changing many registers. Now you can see the effect right away.

Didn't update builds yet; will do after looking into the issue reported earlier.

Danne · « **Reply #968 on:** July 20, 2018, 07:45:51 PM »

Cool, on the fly

Levas · « **Reply #969 on:** July 20, 2018, 07:49:55 PM »

Alex, do you know if these new found gain registers are digital or analog ?

theBilalFakhouri · « **Reply #970 on:** July 23, 2018, 07:39:41 PM »

Porting iso_regs.mo to 700D in this commit with some problems:

"There is problem with ADTG Preamp registers it's not giving the write value? Is there something specific for 5D3 I did not notice it in the code?
And also what the effect of these registers in 5D3 ? in 700D these registers doesn't change between the ISOs.

ADTG[FE] is missing in 700D how I can find the right register for that (for same effect of this register). Or how did you find it in 5D3?"

Okay but I can still make the trick only with ADTG 8882 to 8888 (ADTG Gain in iso_regs.mo) :

How the scene looks like without editing:

How the scene looks like in the camera" border="0

Normal ISO 100 (Pushed 4.00 ev in MLVApp) :

ISO 400 from Canon to 100 (Pushed 4.00 ev in MLVApp):

Dual ISO 100/800 (Pushed 4.00 ev in MLVApp):

Very nice clean shadows with little small clipping of highlights in contrasty scenes only (there are no clipping highlights in the example above).

From 800 ISO to 100 (Using ADTG Gain) the shadows is more cleaner but with losing dynamic range in highlights of course. Now I am trying to restore that dynamic range (highlights) at ISO 800 with the registers that related to the gain and I am not really sure if that possible.

Danne · « **Reply #971 on:** July 24, 2018, 10:42:18 AM »

I can confirm register ADTG Preamp not working and ADTG[FE] missing for eosm and 100D too same as Bilal:
https://bitbucket.org/Dannephoto/magic-lantern/commits/bcad4c9e1f86f5ce01bd9001dbd8f781a0cae457?at=iso-research_eosm_100D_and_more

On another note I´ve been corresponding with Bilal getting hints and reading conversations from Levas and a1ex realizing we can use analog gain registers to change 14bit lossless to lower bits. Working with higher resolutions:
https://bitbucket.org/Dannephoto/magic-lantern/src/ad649aff48a02c942a4ce6ecbe825af1da5097ea/modules/crop_rec/crop_rec.c

code excerpt:

Code: [Select]

    if (is_100D)
    {
        switch (crop_preset)
        {
            /* 3x3 binning in 720p (in 1080p it's already 3x3) */
            case CROP_PRESET_3x3_1X:
                /* ADTG2/4[0x800C] = 2: vertical binning factor = 3 */
                adtg_new[0] = (struct adtg_new) {6, 0x800C, 2};
                break;
        }
		
		switch (crop_preset)
		{
			case CROP_PRESET_2K_100D:		
			case CROP_PRESET_3K_100D:
			case CROP_PRESET_4K_100D:
			{
				 /* assuming FPS timer B was overridden before this */
                int fps_timer_b = (shamem_read(0xC0F06014) & 0xFFFF) + 1;
                int readout_end = shamem_read(0xC0F06804) >> 16;    /* fixme: D5 only */

                /* PowerSaveTiming registers */
                /* after readout is finished, we can turn off the sensor until the next frame */
                /* we could also set these to 0; it will work, but the sensor will run a bit hotter */
                /* to be tested to find out exactly how much */

	/* will add 10bit through analog gain */
	adtg_new[0] = (struct adtg_new) {2, 0x8882, 0x46};
	adtg_new[1] = (struct adtg_new) {2, 0x8884, 0x47};
	adtg_new[2] = (struct adtg_new) {2, 0x8886, 0x46};
	adtg_new[3] = (struct adtg_new) {2, 0x8888, 0x45};

                adtg_new[4]  = (struct adtg_new) {6, 0x8172, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (6D/700D) */
                adtg_new[5]  = (struct adtg_new) {6, 0x8178, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3/6D/700D) */
                adtg_new[6]  = (struct adtg_new) {6, 0x8196, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3) */

                adtg_new[7]  = (struct adtg_new) {6, 0x8173, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (6D/700D) */
                adtg_new[8]  = (struct adtg_new) {6, 0x8179, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (5D3/6D/700D) */
                adtg_new[9]  = (struct adtg_new) {6, 0x8197, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (5D3) */

                adtg_new[10] = (struct adtg_new) {6, 0x82B6, nrzi_encode(readout_end - 1) }; /* PowerSaveTiming ON? (700D); 2 units below the "ON" timing from above */

                /* ReadOutTiming registers */
                /* these shouldn't be 0, as they affect the image */
                adtg_new[11] = (struct adtg_new) {6, 0x82F8, nrzi_encode(readout_end + 1) }; /* ReadOutTiming */
                adtg_new[12] = (struct adtg_new) {6, 0x82F9, nrzi_encode(fps_timer_b - 1) }; /* ReadOutTiming end? */
			}
                break;		
		}
    }

.......

Code: [Select]

/* Values for 100D */
static inline uint32_t reg_override_2K_100d(uint32_t reg, uint32_t old_val)
{
    switch (reg)
    {
        /* raw resolution (end line/column) */
        /* X: (3072+140)/8 + 0x17, adjusted for 3072 in raw_rec */
        case 0xC0F06804: return 0x53902a1; // 2520x1304  x5 Mode;
        case 0xC0F06014: return 0x71c;
        case 0xC0F0713c: return 0x535;

	/* correct liveview brightness */
	case 0xC0F42744: return 0x4040404;
    }

    return 0;
}

Will darken image lowering causing pinks(footage can easily be restored in Mlv App working the white level slider):

Code: [Select]

	/* will add 10bit through analog gain */
	adtg_new[0] = (struct adtg_new) {2, 0x8882, 0x46};
	adtg_new[1] = (struct adtg_new) {2, 0x8884, 0x47};
	adtg_new[2] = (struct adtg_new) {2, 0x8886, 0x46};
	adtg_new[3] = (struct adtg_new) {2, 0x8888, 0x45};

Preserving the liveview brightness:

Code: [Select]

	/* correct liveview brightness */
	case 0xC0F42744: return 0x4040404;

Only caveat right now is that running "framing" preview(slow but correct preview) we still get the darkened image since white level preview is reported wrongly. Seems fixable? But where? raw.c, mlv_lite.c or by registry change in camera? Is it fixable? If so we could have working 10/12bit lossless for higher resolution probably without needing to fix white levels in post.

Major thanks to Bilal for handing me registers and what and where to look for answers. And Levas for pointing at the analog gain registers in conversations with a1ex.

builds for 100D available. Code runs in crop_rec settings:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2018Jul23.100D101_2K-3K-4K_10bit_patch_g3gg0.zip
https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2018Jun30.100D101_2K-3K-4K_10bit_patch.zip

Back to vacation obligations....

bouncyball · « **Reply #972 on:** July 24, 2018, 03:08:31 PM »

"ISO 400 from Canon to 100 (Pushed 4.00 ev in MLVApp)" vs "Dual ISO 100/800 (Pushed 4.00 ev in MLVApp)" IS amazing!
It looks so organic and nice in comparison to dual iso aliased image!!! And low noise levels are the same.

Danne · « **Reply #973 on:** July 24, 2018, 05:17:16 PM »

Quote from: bouncyball on July 24, 2018, 03:08:31 PM

"ISO 400 from Canon to 100 (Pushed 4.00 ev in MLVApp)" vs "Dual ISO 100/800 (Pushed 4.00 ev in MLVApp)" IS amazing!
It looks so organic and nice in comparison to dual iso aliased image!!! And low noise levels are the same.

Just checked on my computer. It looks amazingly clean.

IDA_ML · « **Reply #974 on:** July 24, 2018, 05:27:59 PM »

Yeah, hard coding a few of these magic numbers to each of our cameras would be God sent ! Sometimes, even a fraction of a stop improvement in dynamic range can make a HUGE difference in the signal to noise ratio in the Shadows after proper exposing to the right.

News:

Author Topic: CMOS/ADTG/Digic register investigation on ISO (Read 845758 times)