Cleaner ISO presets

[timbytheriver]

Oh right. There are annotations on the film for the important bits!

Regs and values are clearly visible in the submenu when the presets load! The stock ISOs are – stock! No reg tweaks.

[Danne]

I'll have a look.
By the way. You should post sources, (or branch in this case), relevant to your builds shared. Not clear what code was used.
Disregard. Found it.

[Danne]

Checked your movie. It looks like a valid refinement. Still you do not reveal exactly what registry used, preset used etc. I check your branch and in crop_rec.c only place I could find modifying iso reg are in this preset CROP_PRESET_UHD:

Code: [Select]

                    /* ADTG2 8,9,a,b adjusts blue and green channel 1 (Audionut?) */
                    /* ADTG4 8,9,a,b  adjusts red and green channel 2 (Audionut?) */
                   
                    adtg_new[30] = (struct adtg_new) {6, 0x8, 46}; /* these control each sensor column: should be 0x2 at lowest Value can be in HEX or DEC */
                    adtg_new[31] = (struct adtg_new) {6, 0x9, 46};
                    adtg_new[32] = (struct adtg_new) {6, 0xA, 46}; /* 0x2E = 46 for iso160, 0x31 = 48 for iso200 */
                    adtg_new[33] = (struct adtg_new) {6, 0xB, 46};
                   
                   
                    //adtg_new[34] = (struct adtg_new) {4, 0x8, 0x31};
                    //adtg_new[35] = (struct adtg_new) {4, 0x9, 0x31};
                    //adtg_new[36] = (struct adtg_new) {4, 0xA, 0x31};  /* set all @ 0x2 for basic clean iso. Calibrate on a per-camera basis for the 70MM13 'OVER' tweak. See comment below. */
                    //adtg_new[37] = (struct adtg_new) {4, 0xB, 0x31};
                   
                    /* ADTG4 0xFE master gain? */
                    adtg_new[35] = (struct adtg_new) {6, 0xFE, 0}; /* ADTG2/4 0xFE = 0 is clean. Causing yellow & blue artefacts if not set to same value */
                    //adtg_new[39] = (struct adtg_new) {2, 0xFE, 0x0}; /* ADTG2 0xFE = 0 is clean. Causing yellow & blue artefacts if not set to same value */
Is this the only place you are tweaking registry in? Are you at any point modifying analog gain registry? Please be transparent.

[timbytheriver]

For sure! That's the wrong branch! I updated my op video post awhile back with the branch I'm now using. https://bitbucket.org/rivertim/magic-lantern-danneclone/src/iso-regs-Q-submenu-access/ All the reg values are there to see – or on the camera in Q-menu. Nothing hidden!

[Danne]

Ok, checked updated branch.
Still. From examples you are providing in your movie you do not tell how you tweaked? I can´t see any hard coded presets so I assume you have your own set up with tweaks? Please share those.
I do see analog gain registry added in here. To me, a big warning sign :

Code: [Select]

/* timby edit 'clean-iso' stuff. */
static int32_t  preamp_89ab = 0;
//static int32_t  reg_9 = 0;
//static int32_t  reg_a = 0;
//static int32_t  reg_b = 0;
static int32_t  gain_0xfe = 0;
static int32_t  gain_888x = 0;
/* timby: more iso_regs imports */
static int32_t  reg_8880 = 0;
//static int32_t  reg_8000 = 0;
//static int32_t  reg_8024 = 0;
//static int32_t  reg_8038 = 0;
//static int32_t  reg_8050 = 0;
//static int32_t  reg_814c = 0;
//static int32_t  reg_x14 = 0;
//static int32_t  reg_x15 = 0;

[timbytheriver]

The values for the presets are all in the lua script! 5d3iso-tweaks.lua.zip https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

The preamp values have to be calibrated for each camera to the max white level. So your camera maybe slightly different.

[Danne]

This is balony. Still the same confusing non revealing pseudotweaks that sends these iso regs tests back to pseudouniverse. Sorry, I will not support this, too many question marks. Share tweaks where you published your tests so we clearly see what is going on.
And lastly. Analog gain certainly tweaked here. Not?

Code: [Select]

--menu.set("Presets", "reg_8xx8", "-31")Have fun.

[timbytheriver]

@Danne

From your rather alarmist tone one might think I was trying to force the community to "drink the Kool-Aid!" Let's get some perspective. This is simply my contribution to an open-source project. I'm not seeking endorsement, or support, or asking that my code be merged into the main branch.

Other users are free to test it and see if their results match mine – or like the proverbial TV, are welcome to turn it off at any time if they don't like it.

[Danne]

I shared and clearly showed you how to work out the registry in crop rec. Others helped too. Is it too much to ask for returning basic understandable results?
Pseudosmoke will take this project nowhere. Too bad. Thought you were on to something.

[timbytheriver]

Here's part two of my videos exploring the results of register gain adjustments. Using this tool: https://www.fastrawviewer.com/ the video examines the extracted dng files in much greater detail, comparing percentage of over and under-exposed pixels in each pair of comparisons.  Surely we all want less underexposed and overexposed pixels counts…?

[Luther]

Nice results.

Surely we all want less underexposed and overexposed pixels counts…?

I think the disagreement is just on the accuracy of the measurement method. In your video, how much of this better DR was caused by the ADTG tweak versus changes in exposure? If you're changing the ISO between stock 100 to 108, this will also increase exposure and could account for the changes in DR.
I doubt that's the case though (too big of a noise reduction in ISO 800) and I think these tweaked values should be of easy access to everyone (as simple as enabling a module).
Nice job @timbytheriver.

[timbytheriver]

Thanks @Luther! 

how much of this better DR was cause by the ADTG tweak versus changes in exposure? If you're changing the ISO between stock 100 to 108, this will also increase exposure and could account for the changes in DR.

It's my present understanding that 'exposure' is: an amount of photons hitting the sensor. This amount can only be altered by a) Lighting. b) Aperture. c) Shutter speed. The sensor only has a sensitivity that is factory-fixed – it cannot be made more or less sensitive by settings in the camera. The micro-voltages from the sensor (photo-electrons?) are then only amplified by the analogue gains and then converted into digital.

Since I didn't change a), b), or c) between the pairs of shots, and in the light of the above – can this be described as an exposure increase? I'm sure I'll be corrected if I've got this wrong!

But irrespective of the photo-mechanics and the terms we use – and now these presets are easy to load (for 5D3 anyway…), anyone is free to try out the presets to compare results!

Cheers!

**PS** What is crucial to maximising these 'gains', is nailing your 'exposure' to maximum White Level –  absolutely accurate ETTR –which is a challenge! and requires some attention to the tools ML provides us with.

[Audionut]

The underexposure percentage value in fastrawviewer only indicates how many pixels are below a defined saturation point. It DOES NOT describe pixel quality, noise in pixels, DR or any other value of interest to the discussion.

So what register is it that you are tweaking to increase exposure (reduce underexposure percentage in fastrawviewer)? And why?

[timbytheriver]

…fastrawviewer only indicates how many pixels are below a defined saturation point.

Useful to know! But I'm describing the difference that it reports between pairs of figures; I'm not changing the saturation point in the prefs – and both images reported 16281 as a white level in the raw_diag readout. So both are at their clipping point I presume.

…So what register is it that you are tweaking to increase exposure (reduce underexposure percentage in fastrawviewer)?

Whatever the tweaks are doing behind the scenes I don't pretend to know, but surely no [register] tweak can increase the light hitting the sensor, or increase its sensitivity.  So, for my money they're not increasing exposure.

The register tweaks are no different to the discoveries made by yourself, a1ex and others way back in the original thread, and all available to check on my repo in the source code and Lua preset that defines the values. https://bitbucket.org/rivertim/magic-lantern-danneclone/downloads/

As for the increase in dynamic range, I'm getting a difference in almost 2/3 stop as reported by raw_diag.

And why?

Because it's there.

[Audionut]

But I'm describing the difference that it reports between pairs of figures

Yes, but you don't seem to understand exactly what it is reporting!

At some point in the cameras processing chain, the register values you are tweaking have increased (brightened) the pixel values in the raw output file.  That is why the fastrawviewer underexposure percentage has dropped, because less pixels in the raw file are below the defined saturation point compared to Canon settings. But this value does not define the quality of those pixels or the amount of noise in those pixels, only that they are brighter!

Whatever the tweaks are doing behind the scenes I don't pretend to know, but surely no [register] tweak can increase the light hitting the sensor, or increase its sensitivity.  So, for my money they're not increasing exposure.

Nonsense.  ISO is exactly a sensitivity setting.  Let's not get hung up on the semantics of definition of exposure.
What we want is quality of pixels. The registers directly affect the quality of pixels.

Because it's there.

I honestly think this topic (register tweaking) is well past the point of blindly throwing shit at a wall and seeing what sticks.

[timbytheriver]

Not my understanding of exposure. From the people who bring us RawDigger – a tool I believe you have used in the past in your research into this subject. Source of quote: https://www.fastrawviewer.com/blog/mystic-exposure-triangle

Code: [Select]

Definitions and Facts
To start with, let’s lay out a few basic definitions and empirically verifiable (and indeed verified) facts that have to do with exposure and ISO:

Exposure time: this is the time during which light hits the photosensitive material in your camera; when the shutter closes, exposure ends.
Photographic exposure is only about two things: the light intensity and the exposure time: the light, coming from the scene, hitting the front element of the lens, reduced by the aperture opening, and acting during the exposure time.
By definition, photographic exposure is calculated as the product of the illuminance of the photosensitive material and the exposure time, and is measured in lux-seconds; photographic exposure is the product of luminous flux per unit area by time.
The elements that have direct control over exposure in the camera are the shutter and the aperture (not counting built-in neutral or other filters).

As such, ISO cannot be an element of exposure; it’s determined through the exposure (see standard ISO 12232:2019, "Photography — Digital still cameras — Determination of exposure index, ISO speed ratings, standard output sensitivity, and recommended exposure index" and the wiki), and is “applied” after exposure.

ISO 12232:2019(E) - ISO speed definition
Standard ISO 12232:2019, section 3, p. 3.7 (page.2)

At the same time, ISO is not the sensitivity of the sensor (see the datasheets on sensors): the sensitivity of the sensor doesn’t change when altering ISO, the only thing that is changed is the scaling coefficient applied to the signal / data obtained as a result of exposure.



blindly throwing shit at a wall and seeing what sticks.

This is totally unnecessary, lowers the tone of debate, and contributes nothing of value to the discussion. Not appreciated.

[Audionut]

Ok, lets play semantics.

What makes the author of that software a definitive authority on this subject?

https://www.iso.org/obp/ui/#iso:std:iso:12232:ed-3:v1:en

3.7
ISO speed
numerical value calculated from the exposure provided at the focal plane of a DSC to produce specified DSC image signal characteristics
Note 1 to entry: The ISO speed is usually the highest exposure index value that still provides peak image quality for normal scenes. However, a DSC does not necessarily use the ISO speed value as the exposure index value when capturing images.

3.3
exposure index
EI
numerical value that is inversely proportional to the exposure provided to an image sensor to obtain an image
Note 1 to entry: Images obtained from a DSC using a range of exposure index values will normally provide a range of image quality levels.

3.9
photographic sensitivity
general term used for numerical values calculated based on the exposure at the focal plane of a DSC which produces a specified DSC image signal level, such as the standard output sensitivity or recommended exposure index
Note 1 to entry: In practise, the photographic sensitivity is often called the "sensitivity" or the "camera sensitivity". It is sometimes called the “ISO sensitivity”, for historical reasons that date from ISO standards for photographic film cameras.

3.13
sensitivity setting
numerical value of the photographic sensitivity used by a DSC when capturing images
Note 1 to entry: In some cases the sensitivity setting is set by the user. In other cases it is set automatically by the DSC.
Note 2 to entry: In DSCs employing an automatic exposure control system, the difference between the EI value used to capture an image and the sensitivity setting is called the “exposure bias”. The value of the exposure bias is typically indicated using EV (exposure value) units.
Note 3 to entry: For historical reasons, the sensitivity setting of a DSC is often labelled the "ISO".

This is totally unnecessary, lowers the tone of debate, and contributes nothing of value to the discussion. Not appreciated.

You, yourself, acknowledge that you do not understand what is going on behind the scenes, so to speak. What you are doing is throwing shit at a wall, and seeing what sticks.
You are somewhat randomly (based on the time and effort of others) creating a preset of values for registers that you do not understand, applying no scientific method at all, and presenting the shit that's left stuck to the wall as useful data.

Then when someone who has spent the time finding those registers, and trying to define what they do, comes along and questions the shit that's flung all over the walls around the place, you deflect from the topics of discussion which hold value (how is the quality of the pixels being captured improved, and how is that being scientifically verified) with semantics and hurt feelings.

http://www.catb.org/esr/faqs/smart-questions.html#not_losing

Exaggeratedly “friendly” (in that fashion) or useful: Pick one.

I absolutely and unequivocally respect you getting your hands dirty (playing with code).

[histor]

Not to interfere with the discussion, just to be a mediator here.

This standard is barely applicable to RAW files in terms of sensitivity. Camera producers use JPEGs for that case.

"ISO speed and ISO speed latitude values shall not be reported for raw images, however, because with raw images processing that affects the values has not been performed" (ISO 12232:2019). https://www.fastrawviewer.com/blog/fujifilm-x-series-iso-bracketing

The author of the software - Alex Tutubalin is one of the leading developers of LibRaw (not just the commercial part of the project). Fastrawviewer blog is very informative.
Looks like these two Alexes are the best experts on the question in the open-source world.
But both usually say, they know nothing. As Socrates did.

________
As Luther noted, there was a canon trick for intermediate ISOs (like 160) – exposure time was changed behind the scenes. Silently. But really it's not the case now. You can always use some moving (rotating) object to prove the same exposure for two frames.

[Audionut]

In a film camera you have film.
In a digital camera you have a processing chain (sensor, amplifiers, digital manipulation).

In a film camera, if you change from ISO 100 film to ISO 400 film you have increased the sensitivity (for the same amount of light hitting the camera = brighter output) of the camera.

If you adjust the ISO setting on a digital camera, you also adjust the sensitivity of the camera. Did you adjust the mechanical sensitivity of one specific part (sensor) of the processing chain, no, but (in Canon cameras especially) you gain an increase in output (brightness) of the camera with a corresponding increase in the quality of pixels (output).

ISO is exactly a sensitivity setting.

The sensitivity of the downstream processing chain from the sensor. It is a gain control (input sensitivity) for further downstream processing components.

In a film camera you are stuck with a preset sensitivity of the camera based on the film. To change the sensitivity of the camera you need to change the film. With a DSLR you can simply change a few knobs.

[70MM13]

tim and i have made excellent progress on TNT, and you can see the results in my latest videos, the candlelight music video, ttt episode 23, and zen episode 5.

we've basically got the 5d3 iso invariant from 100-400, with a vast improvement in the noise of 100.  it's actually cleaner in the shadows than 200 now!  peer into the shadows of the episodes listed above...

the candlelight video was shot at 400.  zero lighting in the scene except the candles.  it isn't the best representation of the video from youtube, but i will eventually upload a very high data rate 4k copy.

i know that there are some rather hostile closed minds trying to suppress our work, but there's no reason for 5d3 users who are actually interested in using this camera to produce excellent quality video not to use TNT!

PS: i think this thread has been polluted enough at this point.  can the hostile people please focus their negativity elsewhere?

After 6 weeks......you probably could have just stayed focused on the topic, rather then taking a pot shot because you obviously haven't been practicing that zen stuff enough, and you're still holding a grudge.  //Audionut

[garry23]

I’ve kept out of this discussion for obvious reasons.

However, I’m drawn to statements that claim the 5D3 can be made ISO invariant down to ISO100.

Looking at the current DR vs ISO, eg http://www.photonstophotos.net/Charts/PDR.htm tells me I would be expecting the DR at ISO100 to be some 13 units on the PDR chart, ie a straight line to ISO100.

Looking forward to seeing more info on this ‘claim’.

[Audionut]

Negative attitude = / = Strong criticism.

we've basically got the 5d3 iso invariant from 100-400

No, you have not.

with a vast improvement in the noise of 100.  it's actually cleaner in the shadows than 200 now!

Everything else being equal, no, it is not.

CMOS sensor > downstream electronics.

CMOS sensor in 5D3 capable of 14.7 stops of dynamic range.

Sensor Full Well Capacity at lowest ISO: 68,900 electrons.
Sensor dynamic range = 68900/2.5 = 27,560 = 14.7 stops.
(note: limit read noise to ISO that give at least 8 stops dynamic range)

The 5D3 doesn't have 14.7 stops of DR at ISO 100.  Why?  Because downstream electronics are to noisy, limits DR otherwise available from the CMOS sensor.  About 3.7 stops of the sensors DR is lost in the noise of the downstream electronics.

When you adjust ISO (sensitivity ), you say to camera, clip (throw away) 1 stop of highlight data so that you may better capture 1 stop of shadow data.  The full stop of shadow data from the sensor that would otherwise be lost in the noise of the downstream electronics is boosted so that it is no longer lost in the noise of the downstream electronics. Of course, the drawback is that you MUST sacrifice one full stop of highlight data to capture that shadow data.

So......

Saying that you've suddenly made the camera ISO invariant from 100-400, is the equivalent of saying you increased the DR of ISO 100 by 3 stops.  You have not!
Saying that the noise in the shadows of ISO 100 is cleaner then the noise of the shadows at ISO 200, is the equivalent of saying you increased the DR of ISO 100 by over 1 full stop, but haven't been able to extract any improvement at ISO 200.  This doesn't make any sense what so ever, since the improvement has to come from the downstream electronics, and adjusting sensitivity (I mean ISO) gives the shadows a boost by default.