Magic Lantern Forum
Developing Magic Lantern => Reverse Engineering => Topic started by: a1ex on January 10, 2014, 12:11:01 PM
-
So, what's all this stuff about "sensor update"?
Just a small improvement in dynamic range in photo mode (around 0.3...0.5 0.8 stops). We were able to fine-tune the amplifier gains in order to squeeze a little more highlight detail.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-cmos.png&hash=f0131ad5729d99c9b02da7cc5f7793de) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d2-cmos.png&hash=3cb29a07063b19c300da62566389f85b) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F6d.png&hash=20433025b67889809e94918894ee99f9)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FkXrbKTq.png&hash=1e2cd0032b0c9723bb4fe34c981261b5) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F5d3-movie.png&hash=8b5a51e7004494587c74a2ab32b8a405)
Graphs for the other cameras can be found here: http://www.magiclantern.fm/forum/index.php?topic=10111.msg97780#msg97780
[February 23] 5D3 reaches nearly 0.8 stops of improvement: 11.77 EV at ISO 115 (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103465#msg103465), and also a new ISO 66 (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103447#msg103447).
[January 17] 5D2 reaches 11.92 EV of dynamic range at ISO 81 (http://www.magiclantern.fm/forum/index.php?topic=10111.msg98281#msg98281).
Only 0.5 EV? That's way too small!
Yes. You may take a look at Dual ISO (http://www.magiclantern.fm/forum/index.php?topic=7139) (which will get an additional 0.3...0.5 stop boost with this "sensor update"; also resolution issues were largely solved), or you may consider switching to Nikon.
Wait a minute, that means less noise, right?
Well, it means you get a little more detail in highlights. This doesn't mean less noise per se (the new ISOs will be just as noisy in shadows as the old ones), but it will let you shift the exposure to the right by 1/3 ... 2/3 EV and collect more photons. This will result in lower noise.
For example, on 5D Mark III I could lower the ISO by 0.37 stops from 100, resulting a new ISO 77. After some more tweaking, I've got 0.6 stops below ISO 100 => ISO 66.
Sample images?
No relevant samples yet, sorry.
How exactly are you getting more highlight detail compared to Canon firmware?
The signal from the sensor seems to be amplified in 2 stages (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p2.html#read_vs_iso): a CMOS amplifier (which operates in full stops - powers of 2 - and we have tweaked it when implementing Dual ISO (http://www.magiclantern.fm/forum/index.php?topic=7139)) and an ADTG (http://magiclantern.wikia.com/wiki/ADTG) amplifier which can be configured in finer increments. After these two stages, the signal is digitized (with an ADC), probably tweaked digitally, and saved to CR2. We have noticed the ADTG amplifier tends to run a little "hot" (that means, it gets saturated a little too early - nothing to do with temperature).
To get the extra highlight detail, one has to reduce the gain for the ADTG amplifier until the ADC will no longer be saturated. At this point, the white level (maximum recorded level in the raw file) will begin to decrease and no more detail will be recovered (since now the CMOS itself or the CMOS amplifiers will get saturated instead).
To play with these gains, scroll down to the research tools section.
Does this mean Canon did not fully optimize their sensor for low noise?
I'd say they simply left a safety margin in their code to make sure the ADC is always saturated (that is, to make sure white is always recorded as white).
How are intermediate ISOs implemented in Canon cameras?
I will try to answer this question in a detailed paper (including how exactly I've reached the conclusion).
For now, you may take a look at these graphs: https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-iso-100-1600%2F0.png&hash=4b181eca6eab6421266e251fc1142fba) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-iso-100-1600%2F1.png&hash=cdee4e1c1fdea3a0e4cf423038f6ab8d) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-iso-100-1600%2F2.png&hash=e946417866b827c1c6ff23dd273be3cf) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-iso-100-1600%2F5.png&hash=108165c369a0ed493a55af03c0c588f5) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-iso-100-1600%2F7.png&hash=88cbaf8f05e46a4d4cb9593fa3794529) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-iso-100-1600%2F8.png&hash=0d9574a2668eb0de2ac468015fe148cd)
What's with that message about f1.2 lenses?
According to DxO, Canon applies a digital ISO boost at wide apertures (http://www.luminous-landscape.com/essays/an_open_letter_to_the_major_camera_manufacturers.shtml) (about 0.5 stops on APS-C cameras, and 0.2 stops on FF from my findings (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96952#msg96952)) to compensate for light loss. Since the digital gain is burned into the CR2 files, at wide apertures you will lose a small amount of highlight details (under 0.1 stops on FF (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96952#msg96952), did not check on APS-C). Also you may get some extra noise from round-off errors, and if your raw editor does not handle white level properly, you may lose even more highlights (details here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96850#msg96850)).
We are going to fix this by turning off the digital gain.
Any side effects?
None yet.
What about Dual ISO?
You will be able to use both tweaks (it will just work).
What about video?
I've got only 0.1 stops of improvement on 5D3, but didn't try too hard.
What's the current state?
Research. We are trying to optimize the parameters that influence ISO, understand their effects (did we really gain 0.5 stops of DR or are we just daydreaming?) and port the results on other cameras. Scroll down for some research tools.
Will it work on my camera?
- If you have a 550D or newer camera, it will most likely work.
- If you have a 7D, no idea yet.
- If you have a 5D Mark II, 50D or 500D, don't get too excited. I've barely got 0.15 stops of improvement on 5D2. Mystery solved!
- Confirmed working on 5D3, 550D, 600D, 650D, 700D, 60D, 6D, 5D2, 50D and 500D.
How can I help?
- Play with the research tools below and report your findings.
- If you have access to laboratory equipment and you can measure the real ISO, you can help us check the validity of our ISO theory.
Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!
Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?" (http://quotationsbook.com/quote/34064/)
But if you have some basic coding/math skills - enough so you can follow the entire discussion without getting dizzy - I have some nifty research tools for you:
raw_diag.mo (https://builds.magiclantern.fm/modules.html#raw_diag) (source (https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/raw_diag/))- cross-platform, anyone can run it
This tool does technical analysis on the raw image data (black/white levels, noise, dynamic range, SNR curve):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr-60d.png&hash=5e0e2ad092cee22112c1262f7bb7150e)
Alternative: RawDigger (proprietary, nonfree).
iso_regs.mo (https://builds.magiclantern.fm/modules.html#iso-research) - 5D3 only, requires either the crop_rec_4k build (http://www.magiclantern.fm/forum/index.php?topic=19300), or a custom ML build from the iso-research branch:
A research tool (or hacker's tool if you prefer) that lets you change most ISO-related parameters on 5D Mark III only and study their effect. Details here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96715#msg96715).
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso_regs.png&hash=5de17337d977b6a988333d07be240ad5)
adtg_gui.mo (https://builds.magiclantern.fm/modules.html#iso-research) - cross-platform, requires CONFIG_GDB=y a custom ML build from the iso-research branch (on 5D3, the crop_rec_4k build works fine as well):
The good old ADTG/CMOS tool (http://www.magiclantern.fm/forum/index.php?topic=6751.msg71720;topicseen#msg71720) updated to also intercept DIGIC (ENGIO) registers.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2F3p02r92st%2FVRAM0.png&hash=07972bf3d7b7eaaa68370f35e5964a08)
For sources, please check the branch iso-research on the main repository.
When it will be released?
When it's ready. I also want to summarize the findings in a small paper (like the Dual ISO PDF (http://acoutts.com/a1ex/dual_iso.pdf)), so I need a little time.
Any recommended reading?
http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html
http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html (main site: http://www.clarkvision.com/articles/ )
http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity
Original message:
Now some fun stuff:
Compile ML with CONFIG_GDB=y and load the adtg_gui module. Override the column gain registers (0x8882-0x8888) to roughly half the original value.
My results (with 978bf96 and these changes (http://a1ex.magiclantern.fm/bleeding-edge/dr.patch); shutter 1/30):
Unaltered ISOs:
ISO ADTG default white level noise stdev DR
100 0x41b 15282 6.62 10.96
160 0x435 13306 5.53 10.99
200 0x54c 15282 6.83 10.92
250 0x435 15282 8.55 10.60
1600 0x454 15282 11.87 10.12
ISOs altered with ADTG gain overriden to 0x250:
ISO ADTG overriden white level noise stdev DR
100 0x250 11911 3.68 11.38
160 0x250 9935 3.25 11.24
200 0x250 11924 3.88 11.31
250 0x250 11917 3.84 11.33
1600 0x250 11915 6.46 10.58
So... we can get almost half-stop of DR just by tweaking some registers?! Looking forward to see the RawDigger results.
-
Keen to try the results when I get home. I would be happy with an indicator to notify for true overexposure vs guessed. That's probably a bit ugly though.
-
Well, I think I found a possible bug. If the image is not overexposed, and max stops at say 13000, and there aren't 10 pixels of the same value to confirm that maximum, my algorithm will assume a white level of 12000 (or whatever the initial guess was).
If the heuristic will take the unconfirmed max if there's no confirmed one, it might be a little better. Also, the safety margin probably can be made smaller (100? need to experiment).
The main problem is that I don't know how to tell for sure whether the image is clipped or not (only through heuristics).
-
Can you discard for hot pixels and average the next bunch? Maybe some weighting towards the max values.
Or discard as hot pixels if their value is above 16000.
-
Yes, something like averaging between 5th max and 50th max might work pretty well. Or maybe just the 5th max minus a small amount.
If you play with the algorithm, be sure to test it at 30-second exposure. This was the main reason for a big safety margin (if you over-estimate the white level by just 1 unit, auto ETTR will no longer work properly).
-
Quick test before bed.
Default
ISO Saturation S.Dev SNR DR
100 16383 7.14 2294 11.21
1/4th shutter. ML reported the same saturation level, or there abouts. To tired to be testing properly.
ADTG tweak
ISO Saturation S.Dev SNR DR
100 13045 4.04 3329 11.73
I also noticed that if you only adjust 1 bank of registers, the saturation level differs (lower) for the G2 channel. My first thought, highlight recovery. Why blow both green channels and rely on post to pull detail from RB, when you can keep actual detail in one of the G channels.
edit: Of course, that would only work if the ADC clips before the pixel wells saturate.
11.73 stops before dualISO. You'll be spanking Nikon soon!
-
I think there are 8 columns where you can tweak the gains.
I also think ADC does clip off a little earlier before the pixel wells saturate. With this tweak, notice that DR at 160/200/250 is now almost identical, with slight advantage towards 250 (differences being probably from quantization error in the noise).
-
I think there are 8 columns where you can tweak the gains.
Yeah, they're in 2 batches of 4.
I also think ADC does clip off a little earlier before the pixel wells saturate.
Well, it actually looks to be around 0.5EV. 0.5EV increased exposure before saturation (tweaked vs normal). Needs further testing, but shows that not only do we gain 0.5EV extended range in the shadows, we also gain 0.5EV increased SNR through the entire exposure. It's going to need your magic on the white level, there's pink highlights in saturated tweaked ISOs. I should probably look at raw2dng, but I'm much to sleep deprived for that atm!
Looks like you nailed that register too. Going lower continues to drop the noise, but the white level drops at a rate where DR remains equal (or there abouts). Not to mention that you get to a point where even with full sensor saturation, there isn't enough gain to deliver any signal. ::) I'd be interested to find out what the measured ISO is with 0x250. DxO suggests measured ISO is 80 for camera ISO 100, which suggests Canon choose the register value as a compromise between noise and mapping the maximum value to 14bits.
-
With this tweak, notice that DR at 160/200/250 is now almost identical, with slight advantage towards 250 (differences being probably from quantization error in the noise).
I need to brush up on quantization.
As expected, intermediate ISOs like 160 or 250 do not cause any changes in ADTG/CMOS con-figuration. These ISOs are obtained by applying some digital gain to the raw data acquired at the nearest full-stop ISO, and this gain is configured from the DIGIC register 0xC0F08030 (SHAD_GAIN).
A different register again? Digital and (0x8882-0x8888) being analog?
Makes you wonder what they were thinking with ISO 50. They could have made that below 100 ISO useful ffs. dmilligan seems to think decisions like this is par for the course (http://www.magiclantern.fm/forum/index.php?topic=9867.msg95040#msg95040). :(
-
For 500D:
ADTG3[0] = 0x149 (default) -> 0x0 (trick)
ADTG3[3] = 0x14a (default) -> 0x1 (trick)
-
A different register again? Digital and (0x8882-0x8888) being analog?
Yes. I'm just not sure how to override it in photo mode on 5D3. On 5D2 it works (add raw_blinkies.o in the Makefile) but there's little or no change to DR numbers. It could be used to remap the values back to 14 bits, but we need to check the histogram for gaps.
So, the ADTG gain may be yet another amplifier stage?
In LiveView (raw video + silent pics in movie mode), the behavior is different. ISO 160/200/250 result in no difference at all in the raw data (so the raw histogram doesn't change at all). Instead, the JPEG is developed with different gains, and these gains are adjusted from 0xC0F08030. I read this register in photo mode to figure out how to adjust the white point for correct clipping warnings, and this register is also used for gradual exposure, for example.
-
Looks like the ADC is clipping 0.7EV before sensor saturation...................
edit: deleted graphs and conclusion since it was way off the mark.
-
F1.4 T1/4s ISO 100 (-3EV in ACR) :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs9.postimg.org%2Frucnfntpb%2F100.jpg&hash=6a2e6e504d7054479926fb5426fc9eb7)
F1.4 T1/4s ISO 200 HTP + ADTG trick (-3EV in ACR) :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs9.postimg.org%2Fp1jfvmtcv%2F200htp.jpg&hash=a9183d117f6b2c7f465f27fc9d4a9e52)
-
Signal on the left of the image is throwing the ML calculations out of whack.
Images such as this.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2F6GTZLy4.jpg&hash=b472c2591a4f2d0b094a159abf8a41d8)
https://dl.dropboxusercontent.com/u/34113196/ML/Noise%20samples/_46A5167.CR2
ML reports,
Mean: 2048
Dev: 8.30
Max: 15463
RawDigger shows the Std.Dev as 6.52, same Mean and Max values.
If I measure right to the image edge in RawDigger, I still see a low S.D value. If I measure from the very top left of the optical black, RawDigger reports S.D of 7.34.
If I make sure the left side of the image contains near black, ML doesn't have the issue.
edit: I'm being lazy, I haven't even looked at your code. I'll head off and do that now, and test some other means if I understand how it's being coded.
if (raw_info.active_area.x1 > 10) /* use the left black bar for black calibration */
{
autodetect_black_level_calc(
16, raw_info.active_area.x1 - 16,
raw_info.active_area.y1 + 20, raw_info.active_area.y2 - 20,
3, 16,
&mean1, &stdev1
);
autodetect_black_level_calc(
16, raw_info.active_area.x1 - 16,
raw_info.active_area.y1 + 22, raw_info.active_area.y2 - 20,
3, 16,
&mean2, &stdev2
);
}
I don't understand what values are shifting the scanning area, and what values are setting the scanning size. I'd like it to scan almost the entire optical black area leaving a few pixels off left and right, and more pixels top and bottom.
-
@a1es: Sorry for late reply; as dmilligan said, RON is Read On Noise.
I pointed you this because in conventional photography (I mean short exposure at 'temperate' temperature), when we speak about SNR, N (noise) is quasi resumed to RON. So where we get the smaller RON we get the best SNR. Of course I suppose S (signal) is the same to compare.
About DR, I'm surprised what you use an offset at 500 (?). Unit is not ADU?
Typical offset is 2048 ADU for 5D3, 1024 ADU for 5D2 and 128 ADU for 5D1.
I presume you know it, but in case; these values are to ensure sigma from RON + FPN will not produce clipped pixels because of low values. Of course, most the offset is high, most you can expect high RON + FPN levels, and less DR.
I find your studies very interesting because it's sure all of that could be optimised, it should be possible to get a DR gain (not 5EV of course) and a better RON, FPN, FWC and detectability.
-
I think there are 8 columns where you can tweak the gains.
ADTG2 adjusts blue and green channel 1
ADTG4 adjusts red and green channel 2
Would have been nice to dial the green sensitivity back, bring it more into line with the sensitivity of the RB channels.I'm playing with adjustments to each column, but it doesn't look like it will bring any benefit to the table.
I pointed you this because in conventional photography (I mean short exposure at 'temperate' temperature), when we speak about SNR, N (noise) is quasi resumed to RON. So where we get the smaller RON we get the best SNR. Of course I suppose S (signal) is the same to compare.
Increased SNR from reduced noise assumes the signal stayed the same. This ADTG tweak is reducing the noise more then the signal for instance.
About DR, I'm surprised what you use an offset at 500 (?). Unit is not ADU?
That was for the white level that ML uses. As a1ex mentioned, if ML uses a white level value above the actual peak white level in the image, things get broken.
-
edit: fixed graphs. All test images taken with 1/13s shutter and within 2 minutes, so temperature influence should be minimal.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FZzvJNg0.png&hash=74c883202ffd289996a17681b24da21c)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FeE7zlBT.png&hash=cf187b68d0207f5b9d24f8e6e19fadfd)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FoiRVFuH.png&hash=01cdc747aeba5603493865ea4db5f475)
For dynamic range I averaged the white level of each ISO set. With normal ISO 3200 and Canon pulled ISO 3200, the white level was a little higher then normal so these were not averaged and were calculated separately. I'll paste the data in numbers a little later.
Looks like the ADC is clipping 0.7EV before sensor saturation.
I was a little overzealous. 0.3EV is closer to the truth.
-
autodetect_black_level_calc(
16, raw_info.active_area.x1 - 16,
raw_info.active_area.y1 + 20, raw_info.active_area.y2 - 20,
3, 16,
&mean1, &stdev1
);
I don't understand what values are shifting the scanning area, and what values are setting the scanning size. I'd like it to scan almost the entire optical black area leaving a few pixels off left and right, and more pixels top and bottom.
Look at the definition of autodetect_black_level_calc:
static void autodetect_black_level_calc(int x1, int x2, int y1, int y2, int dx, int dy, int* out_mean, int* out_stdev)
and read it as "scan from x1 to x2, from y1 to y2, with downsampling factors dx and dy".
So, it has a safety margin on 16 pixels horizontally, on both sides.
However, the optical black size according to dcraw is 124, in ML it's assummed to be 126+20. This is where the bug comes from.
I changed skip_left and skip_right to 120+16 and it seems fine now (stdev the same with both dark frame and fully blown-out image, and no black bars on zebras). Need to run the tests again.
(just as with white level, the skip offsets should be overestimated a tiny bit, but not too much)
-
Some more findings:
1) 0x8880 is black level. Changing it doesn't seem to bring any improvement (DR decreases a little, not sure why).
2) The sweet spot seems to be where the white level is still the same as with the unmodified ISOs. Here it's around 0x320 => the real ISO under 100 might be ISO 80.
3) As you know from the dual ISO PDF, you can override the ISO gains from CMOS[0]. Now check the dynamic range of ISO 100 overriden to 1600, and ISO 1600 overriden to 100 (with tweak disabled). Here, the DR is roughy 0.1 EV higher whenever you select 100 in Canon menu.
This means dual ISO should work better as 100/1600 than as 1600/100. Anyone confirmed this experimentally? (though it's quite unlikely to notice a difference of 0.1 stops)
At ISO 1600, the difference vanishes when enabling the tweak. At ISO 100, the difference is still present, and it gets up to 1 stop when you select 25600 in Canon menu and 100 as override. So, there are some other settings used for these higher ISOs (and finding them may result in some more parameters that can be tweaked).
-
(...)
3) As you know from the dual ISO PDF, you can override the ISO gains from CMOS[0]. Now check the dynamic range of ISO 100 overriden to 1600, and ISO 1600 overriden to 100 (with tweak disabled). Here, the DR is roughy 0.1 EV higher whenever you select 100 in Canon menu.
This means dual ISO should work better as 100/1600 than as 1600/100. Anyone confirmed this experimentally? (though it's quite unlikely to notice a difference of 0.1 stops)
At ISO 1600, the difference vanishes when enabling the tweak. At ISO 100, the difference is still present, and it gets up to 1 stop when you select 25600 in Canon menu and 100 as override. So, there are some other settings used for these higher ISOs (and finding them may result in some more parameters that can be tweaked).
Some month ago (http://www.magiclantern.fm/forum/index.php?topic=5533.msg87217#msg87217) I noticed some strange things about DR, black & white levels; it was when using high ISO recovered by low ISO in Dual ISO. But it is with a 5D2, not a 5D3.
-
White level in LiveView is set to 15000 as a one-size-fits-all code. If the true white level is higher than that, it can't be wrong by more than 0.14 stops. If the true white is 15600, the error is roughly 0.07 stops.
You can always squeeze this extra DR with exiftool.
-
1) 0x8880 is black level. Changing it doesn't seem to bring any improvement (DR decreases a little, not sure why).
Raising it looks to improve things here. Raising it with Canon ISO seems to clip the RGBG channels earlier, but you can offset it with (0x8882-0x8888) ;)
All values below RGBG.
Normal ISO
(0x8882-0x8888)
(0x8882-0x8888)+(0x8880)
Optical black average/SD
2048.0/7.27 2047.8/7.22 2048.0/7.10 2048.17.30
2047.9/5.63 2047.9/5.54 2048.1/5.49 2048.05.62
3932.5/5.59 3932.3/5.52 3923.8/20.4 3932.5/5.55 //Blue channel borked in OB
Here, I measured the levels/SD in the RGB patches in a colorchecker passport. R value from R patch and so on.
2297/28.9 4374/45.0 1741/25.7 4344/44.6
1778/22.8 3339/33.7 1330/20.1 3354/34.8
1894/23.3 3454/31.0 1424/19.9 3469/32.0
Here I measured the RGBG levels/SD from the yellow colorchecker patch. (Brighter patch)
7962/50.1 10992/70.6 3724/24.5 10943/71.9
6633/39.3 8922/55.3 3336/19.0 8946/55.0
8624/39.1 10905/49.8 5313/18.8 10931/50.7
With the blue channel giving strange reading in the OB, I measured the black colorchecker patch.
2413/12.7 2683/20.8 2361/11.9 2885/20.0
2309/9.89 2610/15.9 2286/9.0 2548/15.8
4109/10.6 4541/14.5 4260/9.45 4585/15.4
Biggest difference looks to be in the green channel. Wonder if there are other registers that perform a similar action?
This register is adjusting OB level, so would appear to be digital.
-
From what I've read in the ADTG datasheet (look in the dual ISO PDF at page 15), it's done by a feedback loop (there is a DAC somewhere inside). So I think black level is analog.
So... raising the black level could reduce the noise in midtones?
-
It looks to increase the SNR through the entire exposure. I guess we can exploit it by reducing the gain (highlight biased?) with (0x8882-0x8888) and preventing saturation.
These maybe of interest:
http://documents.stsci.edu/hst/acs/documents/handbooks/DataHandbookv6/acs_Ch42.html
http://documents.stsci.edu/hst/acs/documents/handbooks/DataHandbookv6/acs_Ch43.html
http://www.stsci.edu/hst/wfc3/documents/handbooks/currentDHB/wfc3_Ch52.html
Rodger Clark has Gain e/DN values. Here for the 5D3: http://www.clarkvision.com/articles/evaluation-canon-5diii/
Also, the standard (0x8882-0x8888) values seem to vary slightly on startup. Haven't nailed down what causes the changes.
When adjusting, I've been keeping the slight offsets in each column rather then setting all values the same.
-
I've been trying to figure out how increasing the black level via that register is keeping a similar/smaller Standard Deviation.
It must be clamping the black value up to 2048, and hence by clamping it up further (stronger feedback), it reduces noise further.
Negative feedback loops
When the fed-back output signal is out of phase with the input signal. This occurs when the fed-back signal is anywhere from 90° to 270° with respect to the input signal. Negative feedback is generally used to correct output errors or to lower device output gain to a pre-determined level. In feedback amplifiers, this correction is generally for waveform distortion reduction or to establish a specified gain level. A general expression for the gain of a negative feedback amplifier is the asymptotic gain model.
Positive feedback loops
When the fed-back signal is in phase with the input signal. Under certain gain conditions, positive feedback reinforces the input signal to the point where the output of the device oscillates between its maximum and minimum possible states. Positive feedback may also introduce hysteresis into a circuit. This can cause the circuit to ignore small signals and respond only to large ones. It is sometimes used to eliminate noise from a digital signal. Under some circumstances, positive feedback may cause a device to latch, i.e., to reach a condition in which the output is locked to its maximum or minimum state.
-
Do you have a tool that can plot a signal-noise graph like this?
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fprofile-5D3-iso100.png&hash=f9b0a60292732d1b58d32d01d79e0d8a)
I did this one a while ago by taking many pictures of a blank wall at different shutter speeds. I know darktable guys did such graphs from a single image (http://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/), but they plotted it on linear axes, which is complete nonsense (they buried the midtones and shadows into a tiny corner and their graph covers well only the highlights).
If not, I can write a module that plots this kind of graphs directly in the camera, from a defocused test picture.
-
White level in LiveView is set to 15000 as a one-size-fits-all code. If the true white level is higher than that, it can't be wrong by more than 0.14 stops. If the true white is 15600, the error is roughly 0.07 stops.
1792 -> 15000 / 1024 -> 15600 => 1368 ADU reduction in the range; so more than 8% of reduction.
You can always squeeze this extra DR with exiftool.
Thanks for the tips, I'll see how to do that.
Actually I'm trying to produce a 'very' RAW for scientific imaging, astro-photography or other special imaging by record a RAW with the minimal operations done on. It means the best bias/offset matching for the RON + FPN sigma at the optimal analogic gain etc. (no multiple ISO choice in this case so no way for regular photography). I don't fully understand how to use datas in ADTG datasheet yet, but it looks we are just at the beginning of what we can do with ML!
If you want I can do test on 5D2, just ask.
-
1792 -> 15000 / 1024 -> 15600 => 1368 ADU reduction in the range; so more than 8% of reduction.
1792 is Canon's choice.
-
I've been trying to figure out how increasing the black level via that register is keeping a similar/smaller Standard Deviation.
(...)
Over all sigma ADU values on a short exposure is ≈ RON + FPN + offset.
So if offset > sigma/2 ; sigma will always be the same. You can change as you want the pedestal, it will not affect sigma.
And if offset <= sigma/2 ; sigma could be better but with information lost (low values clipping).
(I supposed a symmetric deviation for sigma here, to be simple)
-
1792 is Canon's choice.
I confirm not.
Canon offset is 1024 ADU on 5D2.
Here is a RON + FPN profil:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_ron-profil-nominal-inv-gain.jpg&hash=36479e1c2521d654429e006d2245d3f5)
-
And if offset <= sigma/2 ; sigma could be better but with information lost (low values clipping).
I suppose you are talking about something else. We are changing the reference value for the black feedback loop (which is an analog bias), we are not changing black level in exiftool.
-
I confirm not.
Canon offset is 1024 ADU on 5D2.
LiveView is not the same as photo mode.
-
I suppose you are talking about something else. We are changing the reference value for the black feedback loop (which is an analog bias), we are not changing black level in exiftool.
Indeed, I speak about analogic bias. We are OK.
LiveView is not the same as photo mode.
Ah, I did not know there's specific pedestal config in LiveView. Au temps pour moi.
-
Okay, but the bias is around 1024...2048 and sigma is around 8.
I guess a larger bias could mean stronger signal for the amplifier that comes after the DAC that clamps black level. But if you raise the bias too much, you are forced to lower the ADTG gain, and this will introduce quantization noise. Probably there's a sweet spot somewhere.
-
interesting!
-
(...)
I guess a larger bias could mean stronger signal for the amplifier that comes after the DAC that clamps black level. But if you raise the bias too much, you are forced to lower the ADTG gain, and this will introduce quantization noise. Probably there's a sweet spot somewhere.
Indeed again, as I said there's a relation between all these parameters.
The optimal config is easy to gess, on the paper, but hard to implement. I'm stuck at this point right now.
In a scientific view of all of this, the best choice will be:
_. the lower pedestal value where RON + FPN sigma is contained therein
[current original Canon bias is too high, because have to fit with all ISO; for example with 5D2 it could be N-times lower for ISO where gain <1 (in my graphic just up you can see it could be ~10x lower)]
_. the more closer analogic gain than 1
[but in real world and for usual photography, gain has to be <1 to equiv low ISO sensibility, so we should use a matrix for all ISO/gain/bias optimal values]
_. the lower digital gain
[in fact if we could have a matrix for all ISO up to gain <1 where DG is null it will be optimal, as it done for audio signal]
Please not that I ever do most of these optimizations in post for my work, I could demonstrate/detail/explain parts if need.
-
Do you have a tool that can plot a signal-noise graph like this?
I've been manually entering the numbers in a spreadsheet and using it to generate the graphs.
-
gnuplot (http://www.gnuplot.info/) is useful for that.
-
Made some progress in understanding how ISO 160x are implemented on 5D3 photo mode. This does not apply to LiveView / raw video.
I've looked at what DIGIC registers get changed between ISO 160 and 200 (you remember them from the good old digic poke, FPS override and all sorts of funky effects), and found this one: 0xc0f0819c = 0xC4C for full-stop ISOs and 250 multiples, and 0x8F0 for 160 multiples.
black stdev white DR
ADTG gains default:
ISO 160: 2047 5.42 13307 11.02
ISO 200: 2047 6.86 15283 10.91
ISO 200 with 0xc0f0819c = 0x8F0: 1187 6.80 15283 11.02
ISO 160 with 0xc0f0819c = 0xC4C: 2734 5.50 13307 10.91
ADTG gains changed to 0.85 x default (right above the point where white level begins to decrease):
ISO 160: 2047 4.56 13307 11.27
ISO 200: 2047 5.71 15283 11.18
ISO 200 with 0xc0f0819c = 0x8F0: 1187 5.71 15283 11.27
ISO 160 with 0xc0f0819c = 0xC4C: 2734 4.61 13307 11.16
Changed 8880 to 0xb04 to bring the black level back to 2047:
ISO 200 with 0xc0f0819c = 0x8F0: 2047 5.88 15283 11.14
You draw the conclusions.
-
Is it possible to set custom value for 0xc0f0819c ?
Ex.
ISO 200 with 0xc0f0819c = 0xAA4 => offset = 1024 ADU
-
Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.
When changing this register, the image looks just as bright as the original ISO (unlike with ADTG gain, where overall brightness changes). Also, both dcraw/ufraw and Adobe DNG converter handle the modified black level properly.
-
Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.
Nice!
When changing this register, the image looks just as bright as the original ISO (unlike with ADTG gain, where overall brightness changes).
Agree, the overall brightness should not change.
In the same time pseudo-blacks should be less noisy because of lower RON levels and DR should be increased because of larger range between RON and max level.
Also, both dcraw/ufraw and Adobe DNG converter handle the modified black level properly.
Yes, but it's better to set that by analogic, at the source. Numerical post optimization with a single source frame is equivalent to stretch DR etc. Some informations are not record in the source so can't be restored after.
-
Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.
Changing 8880 to 0x0 produces this.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FZPc5RFZ.png&hash=a71cd622cf07cfc94ab10bf8512bdade)
Clearly (http://www.magiclantern.fm/forum/index.php?topic=9867.msg95703#msg95703), it's having a bigger effect through the exposure on RB then G.
Agree, the overall brightness should not change.
Most software has enough controls over exposure, I don't see it would matter.
-
That's why I need a tool to plot the entire SNR curve, not just the black/noise/white levels.
-
I couldn't find anything with a google search. The only one I know of is imatest (http://www.imatest.com/docs/multicharts-noise/), but it isn't cheap.
The trial version might be enough for what you need.
-
I got my 6D and digic poke, if you need some data i can try and provide you some figures.
Just let me know the procedure.
-
Yes, the last graph from their page is what I'm looking for. If it runs on the camera from a defocused test image (without test charts or other special requirements), that's even better, since I can just tweak parameters and see the result right away.
So, I started to write my own module. @SpcCb, I bet you will like these graphs a lot (ISO 200 vs 160):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fob-histo-200.png&hash=3e70581815f5c1db39ba888797c48830) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fob-histo-160.png&hash=1b79918810c986fbc50c2cd76b0ef19a)
The histogram pattern does not change with either ADTG gains or SaturateOffset.
@ilguercio: for the beginning, you can start here (http://www.magiclantern.fm/forum/index.php?topic=9867.msg95354#msg95354) and see if you can follow the discussion and arrive at similar numbers. So far, we are just tweaking parameters kinda blindly.
"If we knew what it was we were doing, it would not be called research, would it?"
-
Damn, i haven't had a compiling environment since i bought this new laptop.
I'll try to get it running tomorrow and meanwhile i can try to understand what is exactly involved.
I need to learn all this stuff, so far what seems to be the trick?
-
http://www.microscopyu.com/tutorials/java/digitalimaging/signaltonoise/
Because photon noise is an inherent property of CCD signal detection, which cannot be reduced by camera design factors, it essentially represents a "noise floor" that is the minimum achievable noise level, diminishing in relative effect as photon flux increases. Consequently, it is desirable to operate an imaging system under conditions that are limited by photon noise, with other noise components being reduced to relative insignificance. Under low illumination level conditions (assuming dark noise is essentially eliminated by CCD cooling), read noise is greater than photon noise and the image signal is said to be read-noise limited. The camera exposure time (integration time) can be increased to collect more photons and increase SNR, until a point is reached at which photon noise exceeds both read noise and dark noise. Above this exposure time, the image is said to be photon-noise limited.
-
(...) If it runs on the camera from a defocused test image (without test charts or other special requirements), that's even better, since I can just tweak parameters and see the result right away.
Yes, it's better to run tests on a plain surface without shape because all variations will be include in sigma.
So, I started to write my own module. @SpcCb, I bet you will like these graphs a lot (ISO 200 vs 160):
(...)
Useful tool! (I dream to make tools like that!)
We can well see how is the population around the 2048 ADU pedestal. Is the vertical scale is the same?
If yes, for ISO 160 it clearly shows less population. Some vergences too; RON harmonics. I'll see to publish a RON map, sometimes it's more clear with grfx.
http://www.microscopyu.com/tutorials/java/digitalimaging/signaltonoise/
It confirms what I explained posts ago. ;)
-
Vertical scale is "to fit", obviously, sum(all these bars) = 100% ;)
If you take some gaussian random noise, round it, then scale with something like 0.9, its histogram will have the same kind of spikes. If you scale it by 1.1, it will have gaps. So, I believe these histograms show the results of some digital manipulation.
-
Also, the standard (0x8882-0x8888) values seem to vary slightly on startup. Haven't nailed down what causes the changes.
Photo mode shows
ADTG2-8882 0x41c
ADTG2-8884 0x41a
ADTG2-8886 0x41a
ADTG2-8888 0x41c
ADTG4-8882 0x41a
ADTG4-8884 0x41a
ADTG4-8886 0x41c
ADTG4-8888 0x419
Exposure does not have any effect. Register values always remain the same. Whether the values change during exposure and reset after exposure is beyond my capabilities. ISO does have an effect however, and I've plotted the values and put it into pastebin cause I suck at doing it with the forum code tag.
http://pastebin.com/jtND2EJd
In photo LV, there is a pattern, and the crux point seems to be EV 5 (http://www.fredparker.com/ultexp1.htm#EXPOSURE%20FACTOR%20RELATIONSHIP%20CHART%20B) (ISO 100). ie: 1/20s f/1.4 produces one set of register values and 1/15s f/1.4 (EV 5) produces another. This is reproducible by changing aperture. So above EV 5, the register values retain their values, below EV 5 the register values change in a 1 stop pattern. ISO produces the same effect as a 1 stop change in shutter/aperture.
I've plotted that here (http://pastebin.com/hJDzxZHb). ISO 100 results = ISO 100 - 1/30s - f/1.4
I've labelled the changes with ISO, but as mentioned 1 stop with shutter/aperture produces the same. So for instance, you get the same stated ISO 100 results with ISO 100 - 4s - f/16.0 and with ISO 12800 - 1/3000s - f/1.4
This is with 1/2 stop exposure settings in Canon menu. Interestingly, the crux point shifts with 1/3 stop exposure settings. Here, 1/20s f/1.4 produces the change, and it sticks to a 1 stop pattern from there.
Notice how ISO 800 produces lower register values then ISO 400 in both photo mode and LV. Wonder if this has something to do with unity gain?
I've been of the opinion that the register values translate to gain. ie: lower values = lower gain. And this makes sense when you consider the values rise with increasing ISO. This wouldn't appear to be the case though considering the values also rise with increasing exposure.
ISO 12800 is the last ISO to make a change. In the previous pages I should have labelled ISO 3200 as the last useful analog ISO.
So, I started to write my own module.
Nice work. I can plot the same with RawDigger and output CSV files containing the values like so,
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FC43rNq3.png&hash=c322de88cc8a0d81807dc851c7757397)
https://dl.dropboxusercontent.com/u/34113196/ML/Noise%20samples/_46A5329-rawdata.csv
Obviously, this way I have to remove the card, load it into Rawdigger, etc, etc. Note: That's my dodgy graph!
-
http://pastebin.com/jtND2EJd
Same values here, +/- 3 units.
In photo LV, there is a pattern
This is called ExpSim. LiveView normally is limited to 1/30 exposure time.
Since the ADTG gain does not change the histogram pattern, and the histogram is mostly without gaps even with this gain at 3x the original value, I believe this gain is analog amplification. However, bumping the gain at 5x does not result in lower noise:
stdev at 1x: 6.71
stdev at 5x: 34.02
-
This is called ExpSim.
Well that's disappointing. For a moment there I thought I was smart!
-
My current theory about 5D3 ISOs:
I believe these parameters are relevant (there may be others):
- ADTG gains (0x8882 ... 0x8888)
- ADTG black (0x8880)
- SaturateOffset (0xc0f0819c), related to ADTG gains
- 0xc0f37ae4, af0, afc, b08 (digital gain - this one changes the shape of the histogram)
- c0f37ae0/aec/af8/b04 (offset for black and white levels)
The last two registers can be used to configure the range of recorded data (Canon default is from 2048 to a little over 15000 for full-stop ISOs). Changing them will only influence the round-off errors. No big deal.
ADTG black is uncertain, I'll leave it unchanged for now.
ADTG gains and SaturateOffset can be used to recover some more highlight detail. They are tightly coupled (you can recover the same highlight detail from one, from another, or split between them). So, changing only ADTG gain is enough (because the other one runs out of range much quicker).
At some point, there's no more highlight detail to recover. When this happens, the white level will begin to decrease. Let's call this the sweet spot, and it can be found easily with binary search, for example.
Now, remember the ISO definition from DxO (http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity): ISO is all about the exposure necessary to saturate the sensor. Since we could change the clipping point to capture more highlights, we can effectively got some lower ISOs. They are just as noisy in shadows as their full-stop counterparts, but they include a little more highlight detail.
So, assumming ISO 100 is really ISO 100, and setting ADTG gain to the sweet spot (found by binary search), I believe the new ISOs are, at least on my 5D3:
ISO 77, 150, 295, 595, 1175, 2267, 4360 and 8520 (based on ISO 100 ... 12800).
To confirm the formula, I've checked Canon ISOs (starting from ISO 50):
ISO 100, 100, 125, 185, 200, 250, 375, 400, 502, 750, 800, 1005, 1495, 1595, 2008, 2993, 3188, 4015, 5988, 6378, 8035, 12008, 12790, 16110, 20295.
After this, the digital gain register gets changed from 0x200 to 0x10200 and my formula shows 12790, so... 25580.
Take it with a grain of salt, I need to double-check the math and confirm the results with some test shots. I plan to post some code to play with, and some more details.
-
My current theory about 5D3 ISOs
It would be terrific if ML could result better dr/noise than Canon, this would be the ultimate killer feature - of course only if it makes its way to the 6d :-> ...
... I very much suspect that something can be done about this since the 1Dx has a nearly linear dr/iso curve, and Canon most likely doesn't use a different sensor tech on that model - instead the theory on CR is that they applied a high amount of fine-tuning to get this difference vs. non-1d cameras.
-
500D default values :
ISO 100 200 400 800 1600 3200 6400 12800
ADTG3[0] 149 149 148 148 142 142 142 142
ADTG3[3] 14a 14a 14a 14a 145 145 145 145
-
of course only if it makes its way to the 6d
Its very similar to 5DIII so it should. Hopefully it makes it to all cameras.
-
Very, very interesting. Nice work guys!! As I said before you guys should be paid by Canon R&D department! Maybe in Bitcoins? :)
-
So is there any truth in this http://www.clarkvision.com/articles/iso/index.html
-
Contrary to popular belief, ISO is post exposure gain. Exposure is the light hitting the sensor which is a result of lens diameter, shutter and aperture.
So by having a lower gain from the electronics,
So, assumming ISO 100 is really ISO 100, and setting ADTG gain to the sweet spot (found by binary search), I believe the new ISOs are, at least on my 5D3:
ISO 77, 150, 295, 595, 1175, 2267, 4360 and 8520 (based on ISO 100 ... 12800).
Means that we can now increase the exposure (photons hitting the sensor) before saturation (overexposure).
So even though the new ISOs,
are just as noisy in shadows as their full-stop counterparts
We can ETTR a little more. Which means we can increase the SNR of the shot noise (http://www.magiclantern.fm/forum/index.php?topic=9285.msg93716#msg93716).
-
(...)
If you take some gaussian random noise, round it, then scale with something like 0.9, its histogram will have the same kind of spikes. If you scale it by 1.1, it will have gaps. So, I believe these histograms show the results of some digital manipulation.
This histograms show the RON.
It's why you don't see a gaussian random noise distribution.
I'm a bit busy right now but in a couple of hours I could publish some materials.
-
This histograms show the RON.
Yes, but the interest is in the distribution. Specifically, the effects by digital manipulation.
The histogram pattern does not change with either ADTG gains or SaturateOffset.
a1ex called it pattern. The point being, as the distribution does not change with ADTG or SaturateOffset, we can safely assume that the register is making changes in the analog domain. If a register changes the histogram distribution, we can safely assume it's making changes in the digital domain, in which case, it's probably pointless.
-
octave:1> noise = round(randn(100000,1) * 7); # Gaussian noise of similar magnitude as Canon's ISO 100
octave:2> noise_positive_gain = round(noise * 1.1); # apply a small positive digital gain
octave:3> noise_negative_gain = round(noise * 0.9); # apply a small negative digital gain
octave:4> hist(noise_positive_gain, -30:30)
octave:5> hist(noise_negative_gain, -30:30)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fnoise-pos.png&hash=ddb199c828892ded022256a2b87058df) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fnoise-neg.png&hash=779ef3c04279d3eb467c8816f27eddba)
-
Means that we can now increase the exposure (photons hitting the sensor) before saturation (overexposure)........................We can ETTR a little more.
a1ex showed me how to calculate exactly how far we can ETTR with these new ISOs vs Canon base ISOs. Which is obviously better then random guesses from histograms.
100 0.377 EV
200 0.415 EV
400 0.439 EV
800 0.427 EV
1600 0.445 EV
3200 0.497 EV
6400 0.554 EV
12800 0.587 EV
Not only can we safely increase the exposure by 1/3, there's also some headroom.
edit: This is over and above the DR gains from the tweak.
-
Half a stop at 12800? That's goooooood.
-
Half a stop at 12800? That's goooooood.
It's bound to provide a little leeway, but if you're using ISOs above 100, you should (probably) only be doing so because you're already at the limits of exposure.
-
It's bound to provide a little leeway, but if you're using ISOs above 100, you should (probably) only be doing so because you're already at the limits of exposure.
I guess everybody knows that if you have a pretty static scene and a tripod, iso 100 is the way to go.
At least, they should :)
-
I guess everybody knows that if you have a pretty static scene and a tripod, iso 100 is the way to go.
At least, they should :)
I dunno, given the iq of the 6d using iso ~200 is very good esp. if ML adds a little more dynamic range than to iso 100, up to iso 400 the read noise halves which more or less drowns the dynamic range. With longer exposure time (larger dof for ~macro or bad lighting) in the seconds my observation you're getting another noise source, so raising the iso a little might not be a bad idea.
Half a stop at 12800? That's goooooood.
Well, yes, but that's digital push anyway - my understanding is for raw shooting you're equal or better off with iso 6400 and underexposing, or is there something to be gained with 12800? The dynamic range 6400->12800 drops by 1 ev :-p
-
Shot noise (http://en.wikipedia.org/wiki/Shot_noise), a significant contributor to total noise, is the square root of the number of photons hitting the sensor.
So noise actually increases with increased photons, but it's the SNR that's more important, and this also increases with increased photons hitting the sensor.
ISO doesn't increase the number of photons hitting the sensor.
ISO is useful in Canon cameras, because the ADC is noisy. So where you can't saturate the pixel wells with exposure settings (shutter/aperture) and generate a large voltage from the sensor, you can use ISO to apply gain to the signal to boost it to the saturation point of the ADC, and hence, increase the SNR through the ADC.
-
Well, yes, but that's digital push anyway - my understanding is for raw shooting you're equal or better off with iso 6400 and underexposing, or is there something to be gained with 12800?
It's analog push, not digital. On 5D3, from 1600 to 12800 you gain 0.22 EV in shadow noise (http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html) and for this you pay 3 stops of highlight detail.
On 60D, ISO 6400 is not digital push either. It's analog push from the ADTG gain (which we found to be analog from histogram analysis).
Some quick results from 60D, assumming full-stop ISOs are real:
Base ISO With low ADTG gain Highlights gained
100 88 0.18 EV
200 158 0.34 EV
400 311 0.36 EV
800 633 0.34 EV
1600 1260 0.34 EV
3200 2515 0.35 EV
6400 2515 1.35 EV
(note that ISO 100 on 60D is implemented more like 160 than like 200 multiples; I didn't check saturation level yet)
-
Have anyone checked the 5D Mark II?
Or how would I do it?
-
In theory, simply follow the discussion from the beginning (you need code compiling skils).
In practice, I could not find the equivalent registers on 5D2, but I've only tried for around 2 hours.
-
On 60D, ISO 6400 is not digital push either. It's analog push from the ADTG gain (which we found to be analog from histogram analysis).
You're assuming that Canon is not doing something delibrate to hide the fact that it is actually a digital gain. I do not give Canon that much credit. If you scale by 4x from ISO1600, then dither (http://en.wikipedia.org/wiki/Dither) (i.e. apply gaussian noise to) the two LSBs, your histogram will still look the same, it will not have gaps.
-
At ISO 160 multiples, the histogram does have spikes (including ISO 1250). So if they were trying to hide it, I guess they would have done it at these ISOs too.
-
I thought 160 was pulled, not pushed?
-
In LV you should be able to auto adjust ADTG for maximum white point, correct? So in manual mode you can choose your exposure settings, and ML applies ADTG gain as needed.
An analog auto ISO only limited by the accuracy of the white point detection.
-
LV ADTG settings are only relevant for raw video (and I didn't study them yet). In photo mode you get ExpSim, so the only interesting thing here is to match the clipping points for ETTR.
I thought 160 was pulled, not pushed?
Yes, but I was talking about the attempts to hide digital trickery.
-
What would a development thread be without shots of boring stuff!
Canon ISO 100 (1,055 KB)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FSm3cCHQ.jpg&hash=4f2e663c73e744488e55a1adc89a65e1)
ADTG tweak (853 KB)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FtInNviT.jpg&hash=597e198cc26914e94515b4ea36f20986)
ADTG tweak + Dual_ISO 100/1600 (437 KB)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FNiuTETo.jpg&hash=1eaa9f95a504689d0ea7f8000f15ba03)
-
What would a development thread be without shots of boring stuff
Where does this vertical banding after raising shadows come from - is this standard on 5d3? I've never seen this on the 6d, probably the latter is really an improvement in this category, but of course hard to say w/o taking identical shots.
-
ML iso is much better!!!! Cant wait! I will always have ML iso on if it makes it to the nightly builds. (i sure it will be a little more complex then turn ml iso on but i hope its that simple) You guys rock!!!
-
Where does this vertical banding after raising shadows come from - is this standard on 5d3? I've never seen this on the 6d, probably the latter is really an improvement in this category, but of course hard to say w/o taking identical shots.
I was going to ask the same question. I have a 5D3 and have never seen that. Also, it is not there on the dual ISO shot for some reason.
-
Have you ever tried to shoot a light bulb outside at night, without clipping the highlights, then recover the shadows around it? If there are not a lot of details in the shadows you will see the sensor noise (not banding). I can assure you if you shot this exact situation with your 5d3 or 6d, recovering the shadows will give you this result. Its not in the dual iso image because thats what dual iso was designed to do. To give you better highlight and shadow detail.
These ML iso options will be a great advancement in the mysteries surrounding canon iso and cleaner, better iso.
-
That's deep into the shadows.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FrJ8jVQt.jpg&hash=c18f9f0dd30c1d8e5833b77253664f60)
http://graphics.stanford.edu/courses/cs448a-10/sensors-noise-14jan10-opt.pdf <--- Page 43
http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html
http://www.clarkvision.com/articles/evaluation-canon-6d/index.html
-
WOW.... that tweak and Dual ISO shot looks great!
-
Especially considering where it came from, that is incredible.
-
If there are not a lot of details in the shadows you will see the sensor noise (not banding).
I'd consider this noise pattern banding, I know that very well from the 60d where it appears very soon. At some point you might see it on any camera, that's why I was asking if this is usual or how much ev the shadows were raised to make it appear... and the areas were near black, so it's not surprising and a little dr gain from ml is unable to fix such a worst case situation.
Btw & ot: it would be interesting to see if DxO's new PRIME algorithm is able to do better than ACR (or whatever oss) which fail at removing patterns - DxO afaik uses a noise pattern detection to clean up something like this w/o too much plastic look.
-
and the areas were near black, so it's not surprising and a little dr gain from ml is unable to fix such a worst case situation.
Dual-ISO looks like it did fine :P
-
Dual-ISO looks like it did fine :P
At first glance Dual-ISO looks softer but there is a focus difference between the shots. This can be seen by examining the "MA" letters near the top left.
-
This is the problem with showing off any sort of impressive capture of a wide DR scene. Nobody really gets how bad it was before recovery. I may have to create a before/after section for my real estate photography.
-
At first glance Dual-ISO looks softer but there is a focus difference between the shots. This can be seen by examining the "MA" letters near the top left.
Indeed there is. I'll post some other comparisons and updated SNR graphs after sleep.
-
Dual-ISO looks like it did fine :P
No doubt, it's great and but this is really the type of shot made for it - but once you get details in the shadows and highlights it's a tough decision how much resolution to lose to gain a certain amount of dynamic range.
That's why I'm very excited that now there's every possibility that ML might conjure up 1/2ev of dynamic range out of thin air ... makes you really, really wonder if there's a hidden catch and why Canon doesn't include this, maybe they don't want to confuse users with too much iso wizardry?
-
i don't understand 90% of this thread, but seeing so many "DR" talk makes me so excited about the progress of the topic haha.
-
Yea that banding is on almost everything at higher ISOs, 6D, 7D, etc. After ACR or other post processing its mostly gone. Its hardest to remove it from LV dngs.
-
Btw & ot: it would be interesting to see if DxO's new PRIME algorithm is able to do better than ACR (or whatever oss) which fail at removing patterns - DxO afaik uses a noise pattern detection to clean up something like this w/o too much plastic look.
On my test shot with Greg's car, it didn't remove the pattern noise and actually interpreted it as detail.
I have found a math model for this pattern noise though, and I hope to have a clean fix for it in the near future.
-
Yes, but the interest is in the distribution. Specifically, the effects by digital manipulation.
a1ex called it pattern. The point being, as the distribution does not change with ADTG or SaturateOffset, we can safely assume that the register is making changes in the analog domain. If a register changes the histogram distribution, we can safely assume it's making changes in the digital domain, in which case, it's probably pointless.
When you get this (5D2 distribution, but I think it's close to the 5D3):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_sngl-frame_histo.jpg&hash=88d17d37cc67361f89a24553888b472c)
It is because of this (average of 300 frames with winsorized sigma clipping to reject random noises + bin2 + pix value x16):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_mo_hi-ctrst.jpg&hash=613046990d07a28bea3e847e95686079)
With a 2D FFT we can better see how is the pattern:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_sngl-frame_fft.jpg&hash=148dcf63ce883132bff24ce514381519)
-
The banding pattern accounts for only a small part of these histogram distortions.
Proof:
ISO 100 dark frame (5D3), original and corrected
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fiso100.png&hash=62a723ff61bb21ed89c0de8a1bf8b359) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fiso100-nobanding.png&hash=c1c95e05784439c190ac93a95cb6c23a)
Histograms of horizontal and vertical banding correction (per-line and per-column)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fhband.png&hash=7ac9c797360649acd520e5f765a810e7) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fvband.png&hash=8ed906f6aecef04a37f191ed58383067)
Image histogram before and after correction:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fhist0.png&hash=dde8eb3fda72d4d84df709aab77e6ef2) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fhist1.png&hash=325235ff026961e4e906b746df43178c)
How I've estimated the correction:
# estimate horizontal and vertical banding
fh = mean(im'); fh = fh - mean(fh);
fv = mean(im); fv = fv - mean(fv);
# banding histograms
hist(fh, -3:0.1:3)
print -dpng -r60 hband.png
hist(fv, -3:0.1:3)
print -dpng -r60 vband.png
# compute the flat field
[m,n] = size(im);
ffh = fh(1:m)' * ones(1,n);
ffv = ones(m,1) * fv(1:n);
ff = ffh + ffv;
# image histogram before and after correction
hist(im(:), 2048 + (-50:50))
print -dpng -r60 hist0.png
hist(im(:) - ff(:), 2048 + (-50:50))
print -dpng -r60 hist1.png
-
When I do:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_sngl-frame_histo.jpg&hash=88d17d37cc67361f89a24553888b472c) * sub * (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_mo_hi-ctrst.jpg&hash=613046990d07a28bea3e847e95686079)
I get this:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_mo-sub_histo.jpg&hash=b0f9e808c76edb63394d27e97a18f609)
Of course it's not a perfect gaussian yet, but it's close. :)
-
May I see a single dark frame minus the averaged one?
(I know, I could probably take 300 frames myself, get a PixInsight license and try it)
-
Ah, sorry, I don't have it right now. I just have the histograms of those frames (here in my last post).
But I could do it later. I'v planed to publish the same operation on a video RAW frame.
About Pixinsith, mh... You can use Iris (http://www.astrosurf.com/buil/iris/iris.htm) too, it's free.
-
Some new graphs with calibrated ML ISOs and fixed DR calculation.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FCxHnY5LB%2FSTD.png&hash=ab6d625e68716864da3e3b2edc947f26)
ISO Canon ML
100 7.18 5.48
200 7.58 5.58
400 8.25 6.01
800 9.45 6.98
1600 12.8 9.3
3200 19.4 13.9
6400 37.2 25.1
12800 63.4 42.6
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FFsSJWGMx%2Fimage.png&hash=4dd66083d1aac2f81bd2c71c899f6f89)
ISO Canon ML
100 11.16 11.55
200 11.08 11.52
400 10.96 11.41
800 10.76 11.2
1600 10.32 10.78
3200 9.72 10.2
6400 8.78 9.35
12800 8.01 8.59
ML ISO 800 has greater DR then Canon ISO 100 8) As a1ex points out on the next page though, this doesn't consider noise through the exposure (ie: your midtones).
Removed the inaccurate SNR graph.
-
wow wow wow wow wow wow wow wow!
-
I would plot all these graphs on logarithmic scale (convert the units to EV). In linear plots, the differences appear much larger than they really are in practice.
How did you measure the SNR?
-
I would plot all these graphs on logarithmic scale (convert the units to EV). In linear plots, the differences appear much larger than they really are in practice.
Do you have an easy equation for doing that?
How did you measure the SNR?
saturation/std.dev
-
1) log2
2) that's DR
-
For DR I was
log2(saturation) - log2(std.dev)
-
...which is identical to log2(saturation/stdev)
-
So SNR would be the sqaure root of saturation/std.dev?
And I convert saturation and std.dev to EV?
-
SNR can be computed at any input (signal) level. Take a look at DxO's Full SNR graphs.
DR is about deep shadows, but it doesn't tell much about the midtones. The reason is that noise stdev also varies with signal level (it usually increases). Here's a SNR curve for 5D3 at ISO 100 (notice how SNR goes below the straight line, because noise gets higher):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fprofile-5D3-iso100.png&hash=f9b0a60292732d1b58d32d01d79e0d8a)
If you compare 60D and 5D3 on DxO for example, you will see the DR is identical at ISO 100, but the SNR curves are not. So, in shadows, both cameras are comparable, but in midtones you can see a difference.
In ETTR, I measure SNR at median signal value from the entire image (midtone SNR) and at 5th percentile (shadow SNR). This is not quite correct for bright images, but at low signal values (say under 7 stops, where ETTR normally works), the approximation is very good, as you can see on my graph.
-
I'll leave that for you (or someone else) to plot. All this maths is making my head hurt!
-
Here's how to read the SNR from DxO:
1) Go to Full SNR graph, e.g. http://www.dxomark.com/Cameras/Canon/EOS-5D-Mark-III#measuretabs-6
2) Choose a reference level (say 18%) and read the SNR (in this example, 39.7 dB)
3) Remember the modified ISOs will let you expose a little more to the right, let's say by 0.4 stops. That means, your reference level on DxO graph will now be 18 * 2^0.4 = 23.75. Note that in your image, the reference level will be still 18% relative to your new white point (but this new white point just got higher than Canon default with 0.4 stops). New SNR: 40.9 dB.
4) SNR improvement at 18% level will be 1.2 dB = 0.2 stops if you shift the exposure to the right (so you keep the same amount of highlight detail). If you keep the same shutter speed and aperture, same scene, and develop at the same brightness in post, noise will not change (except for maybe some round-off errors).
So, the improvement in midtones is pretty small, but these midtones have little noise to begin with.
And here's a module that does a *rough* estimation of the SNR curve from a defocused image, and also computes the dynamic range:
raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (source: raw_diag.c (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.c))
The module does 3 types of analysis:
- optical black (it draws the noise histograms as the ones I've posted earlier in this thread)
- dark frame (same, but analyzes the entire image - very slow)
- SNR curve (take a defocused picture and it plots an approximation of SNR as a function of signal level)
I'd like you to run some tests with the raw_diag module and check the repeatability of the measurements (maybe compare it with DxO, RawDigger or whatever, or just review the math). For dynamic range, make sure you have something overexposed in the frame (e.g. point the camera to a light bulb). You may also post tables with dynamic range at each ISO from your camera (which should be comparable, since they will be computed with the same algorithm).
-
A quick play.
Canon (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A6530.CR2) vs ML (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A6533.CR2) - Last stop before Canon saturation.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F8PZ7c9gw%2FVRAM0.png&hash=3ff56066c825ed77f61a0caa5ea96020) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FnhQnxPFX%2FVRAM3.png&hash=0ed352c5b0a680c2ac601d1371e20052)
Rawdigger results.
Canon ML
White 14907 11223 <---Highest recorded white pixel
Std.Dev 9.04 6.78 <---Worse case measuring entire left optical black area
Std.Dev 7.32 5.49 <---Worse case cropping pixels either side of measuring area
DR 10.687 10.693 <---Measuring entire left OB
DR 10.992 10.997 <---Measuring cropped OB
Canon saturated.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FJ02WVbS3%2FVRAM1.png&hash=3e8aa1148350dce239089f3d6d797d92)
Canon vs ML - Last stop before ML saturation. I could have reduced ML negative gain here to push the white point.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FQtNLF43k%2FVRAM2.png&hash=247d854ec3345de56e4090f1a887a96a) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fk5Q96Rt9%2FVRAM4.png&hash=7fd6eb76144cd3f207cd35e7d7ef66d4)
ML Saturated.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FBv94nBbM%2FVRAM5.png&hash=41d5116ed90b5e032255d3b887b00d79)
Canon vs ML.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F9M1h13pn%2FVRAM6.png&hash=6d18a14d177f4fb7d911631f5ab6265a) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fj5RTzqzf%2FVRAM11.png&hash=268259218defad6b4ee2b0d2c069c3c4)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F65PBDXXs%2FVRAM7.png&hash=15a675c0e14c86e17835304ac58a105e) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F9FsjLYdC%2FVRAM12.png&hash=e882e132a1be9b930d05a41d6f39008f)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F7h6yWBtY%2FVRAM9.png&hash=26a57016edaa4322d459476e494df8ac) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FKv127Hg0%2FVRAM13.png&hash=d41b88772f45aa046684e82c545cf7db)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FHWZmygjJ%2FVRAM10.png&hash=0feb13c030be1650ecae28b704ec56b1) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FrpNXF6vw%2FVRAM14.png&hash=d9b2e149662a56814c3b2dfc8691b296)
-
wow! when will this be released for 5D3?
-
To measure DR, you should have something overexposed in the image.
Also, white level of 16382 is likely a Canon bug (overflow in digital gain) and it will prevent a fair side-by-side comparison.
-
May I see a single dark frame minus the averaged one?
(...)
Here is the frames, with respective distributions and same screen range (960 - 1088 ADU):
Single frame @ 1250 ISO
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_sngl-frame.jpg&hash=704029d3d8912148787e1db4bf50d696) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_sngl-frame_histo.jpg&hash=88d17d37cc67361f89a24553888b472c)
Master offset @ 1250 ISO (average of 101 offset frames)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_mo.jpg&hash=cd933a7826dafe861c70dd08a4bbd40d) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_mo_histo.jpg&hash=9889f48d1af9dd107c18811ac95c8ef8)
(Single frame @ 1250 ISO) - (Master offset @ 1250 ISO)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_mo-sub.jpg&hash=5e5c994cba836067b7324bcd29444960) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_1250iso_mo-sub_histo.jpg&hash=b0f9e808c76edb63394d27e97a18f609)
-
The DR measurement was just to compare ML vs RawDigger. Nothing serious.
Reducing negative gain (yes I was past calibrated result) fixed the WL, but here is a standard Canon ISO where ML reports a different WL to RawDigger.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A6566.CR2
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FVk0V5ybc%2Fcanon.png&hash=f168512ab116897229e56c73dcb01d64)
Rawdigger
White 13956
Std.Dev 7.2
-
This image is not overexposed, so I don't consider the white level relevant. As long as there's nothing overexposed in the frame, the DR reported is plain wrong.
ML doesn't just take the max value because there are hot pixels.
-
ML doesn't just take the max value because there are hot pixels.
Of course. You explained that back on the first page :(
-
Indeed there is. I'll post some other comparisons and updated SNR graphs after sleep.
When you do this test Audionut it would be beneficial if the camera were locked down on a tripod. That way we can watch the noise in the blacks disappear as we go from Canon DR, to ML DR, to ML DR + Dual ISO.
-
ML vs Canon / 0EV settings
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FmZy6JN77%2FVRAM0.png&hash=c51eebca16182e662cfb755bb52d4f18) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FGmVMGM1n%2FVRAM1.png&hash=66b92da9fb923cfba7cdd64be02b71ef)
Imported into ACR, exposure levels for the 0EV frames were measured from the white patch, and all remaining frames were adjusted from these 0EV frames. White balance set from 0EV ML frame. dual ISO had it's white balance set separately and there was slight movement forward of the camera between dual ISO frames and all other frames.
ML spotmeter reads the grey patches at top of each image as being -4.7EV, -3.2EV, -2.2EV, -1.3EV, -0.6EV, 0EV, left to right. Which puts the far left black patch in the -6EV shots @ -10.7EV below sensor saturation.
ML vs Canon / -1EV
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fm2SVpLfj%2F46_A6732-as-_Smart-_Object-1.jpg&hash=2b9c7569cfdb01a5b878f29f02bcae4e) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FfRrc1kHc%2F46_A6739-as-_Smart-_Object-1.jpg&hash=9d27f1e32bbf1d376253598931c2ca6d)
ML vs Canon vs Dual_ISO 100/1600 / -2EV
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FbwWn3WJL%2F46_A6733-as-_Smart-_Object-1.jpg&hash=8a056e008b8ab1350450b54bd42247cb) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FrwBdS4nP%2F46_A6740-as-_Smart-_Object-1.jpg&hash=cb89e0275399dd4ccfc0a7d19e8cbda8) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FHsWPgctR%2F46_A6748-as-_Smart-_Object-1.jpg&hash=d7d2fa899282048c285c873f553fb5af)
-3EV
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FvTK8GRCN%2F46_A6734-as-_Smart-_Object-1.jpg&hash=280df6b583236c4c2d181a5335f4a73a) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F5ysYvkTB%2F46_A6741-as-_Smart-_Object-1.jpg&hash=70c7d427014779d4f7457b036ded5f9f) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FMp2x5YwV%2F46_A6749-as-_Smart-_Object-1.jpg&hash=f6b7dcd2cbb12c7c2f790bde79bc3dec)
-4EV
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F44PhrmyY%2F46_A6735-as-_Smart-_Object-1.jpg&hash=d0e9b557545de4c6b56044e571ad7af6) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FSQgJg318%2F46_A6742-as-_Smart-_Object-1.jpg&hash=f1d20d2d68c1b18d7608c0120b9a6b76) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F3xxMnf6y%2F46_A6750-as-_Smart-_Object-1.jpg&hash=3d1c8b005ced1e7e6a2d48288be196c8)
-5EV
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F8zJJJ4Hv%2F46_A6736-as-_Smart-_Object-1.jpg&hash=5cbb1cca657e47b9f03a7a50e99d9cab) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FXJp7m7zC%2F46_A6743-as-_Smart-_Object-1.jpg&hash=341a63d985e5b90579c58826724fecc6) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FsgNLKj78%2F46_A6751-as-_Smart-_Object-1.jpg&hash=bf0c07e7680022387e86404807d2bc56)
-6EV - Images slightly underexposed due to lack of latitude in ACR.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FFRQLQS0L%2F46_A6737-as-_Smart-_Object-1.jpg&hash=d8e270f73133701df11db5737017edf7)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fgk6m3TCG%2F46_A6744-as-_Smart-_Object-1.jpg&hash=c571849a95f116fd3d49796d59e44631)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FFsPCsrWR%2F46_A6752-as-_Smart-_Object-1.jpg&hash=10924aef743dba639048756c6befb5fb)
-
The DR measurement was just to compare ML vs RawDigger. Nothing serious.
Reducing negative gain (yes I was past calibrated result) fixed the WL, but here is a standard Canon ISO where ML reports a different WL to RawDigger.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A6566.CR2
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fdl.dropboxusercontent.com%2Fu%2F34113196%2FML%2FADTG%2FCanon.BMP&hash=214974030a67fa3cf0f5b0a55d8ce285)
Rawdigger
White 13956
Std.Dev 7.2
Found this thread after a lot of searching after the Tweet "Psst..." etc.
Looks promising, 1/3 or 1/2 stop of extra dynamic range (and less noise).
But this material raises a lot of questions to me ?
How is the dynamic range calculated with the black level, white level and st.dev (I know what St.dev means, but can it in this context literally be seen as a noise indicator ?)
In the example above the black level is 2049 and the white level is 13919 and a Stdev. of 9,3.
Does this mean that the sensor at that moment can roughly measure from values 2049=black and 13919= white which results in 11870 usable values (and are these number literally counted photons?)
What is the formula for the dynamic range with these numbers ?
Does anyone know already if this stuff is related to the temperature of the sensor ?
-
Looking at the latest pictures of Audionut.
Looks to me ??? that there's more chroma noise in the ML pictures, but there should be less noise in the ML pictures according to the numbers in this topic ?
Sorry, the first ones are the ML picture and the second ones are the canons ::)
Clearly less chroma noise in the ML pictures ;D
-
How is the dynamic range calculated with the black level, white level and st.dev (I know what St.dev means, but can it in this context literally be seen as a noise indicator ?)........................What is the formula for the dynamic range with these numbers ?
For DR I was
log2(saturation) - log2(std.dev)
...which is identical to log2(saturation/stdev)
(and are these number literally counted photons?)
http://en.wikipedia.org/wiki/Color_depth
Looks to me ??? that there's more chroma noise in the ML pictures, but there should be less noise in the ML pictures according to the numbers in this topic ?
This tweak reduces read noise only.
http://www.cambridgeincolour.com/tutorials/image-noise.htm
-
Also, white level of 16382 is likely a Canon bug (overflow in digital gain) and it will prevent a fair side-by-side comparison.
Why is this? 14bits should give 16384 with a black level set at 2048?
-
http://en.wikipedia.org/wiki/Color_depth
Of course photons are measured through analoge signal... :-[ ::) stupid me, these are numbers after AD conversion, 14 bits so maximum of 16384.
-
Why is this? 14bits should give 16384 with a black level set at 2048?
The 14-bit limit is 16383. Since our white level is way too close to this limit, it's quite likely that the real white level may be above that.
I can't get this white level on my camera though (it varies with the aperture, but not that much).
-
Why is this? 14bits should give 16384 with a black level set at 2048?
16384 - 2048 does not give usable 14 bits.
And in fact the ADU range is (16383 - ?*) - (2048 - RON) with 5D3.
Like a1ex I never get 16383 ADU with 5D2 too. The most high value I can reach is 15600 ADU.
-
The most high value I can reach is 15600 ADU.
Aperture effects white level.
f/1.4 16382
f/1.6 16316
f/1.8 16182
f/2.0 15950
f/2.2 15769
f/2.5 15655
f/2.8 15463
f/3.2 15440
f/3.5 15383
f/4.0 15330
After this point, WL remains the same.
-
???
Maybe a 5D3 special feature (?).
I'll try with f/0 to see that...
-
Maybe a 5D3 special feature (?).
I can use larger negative gain values for ADTG then a1ex before WL changes.
I'll try with f/0 to see that...
On my 5D3, detaching the lens didn't effect the register that adjusts the gain. It needs an electronic link from the lens.
-
I think it's a trick to adjust the brightness in the jpeg previeew. It's done by adjusting the digital gain registers, and as a side effect it changes the recorded range and the roundoff errors.
If it's only at f1.4, and has a safety margin at 1.6 and below, it's probably not overflowing (or if it is, it's only a tiny amount).
So, it should not affect our highlight recovery trick in any way other than round-off errors.
-
500D (raw_diag)
F1.4 16382
F1.6 15788
F1.8 14918
F2.0 14173
F22 13431
-
(...)
On my 5D3, detaching the lens didn't effect the register that adjusts the gain. It needs an electronic link from the lens.
Indeed, I use a custom chip where I can define f/whatIwant.
I can get on the 5D2:
f/5.6 -> 15764 ADU
f/1 -> 16383 ADU
f/0 -> 16383 ADU
Update: It depends of the ISO setting too, not only the f/stop.
-
Now, if my theory is right, the sensor saturation point (where detail gets clipped) should be the same in all these cases. Can you confirm or infirm?
-
500D (raw_diag)
Make sure you are getting sensor saturation Greg.
Update: It depends of the ISO setting too, not only the f/stop.
Not here. If you are changing exposure settings with ISO, make sure you are getting sensor saturation.
Now, if my theory is right, the sensor saturation point (where detail gets clipped) should be the same in all these cases. Can you confirm or infirm?
I'll check tomorrow.
-
Make sure you are getting sensor saturation Greg.
Do you have it in mind?
500D (RawDigger) :
F1.4 (Black Level 1018, Max value 15365) 15365 + 1018 = 16383
F1.6 (Black Level 1019, Max value 14770) 14770 + 1019 = 15789
ISO :
F2.2 ISO100 (Black Level 1020, Max value 12782) 12782 + 1020 = 13802
F2.2 ISO200 (Black Level 1018, Max value 15181) 15181 + 1018 = 16199
-
(...)
Not here. If you are changing exposure settings with ISO, make sure you are getting sensor saturation.
Of course I'm sure to get sensor saturation. ;)
But with 5D2 and with just some quick tests I see it only depends of the ISO, f/stop doesn't affect RAW max value for a equivalent light intensity and same ISO.
I need to investigate about this to confirm relations..
I'll check tomorrow.
+1
-
Do you have it in mind?
Point it at the sun :o
If you think you have saturation @whatever aperture, increase exposure by 1 stop with shutter to confirm.
-
Point it at the sun :o
If you think you have saturation @whatever aperture, increase exposure by 1 stop with shutter to confirm.
Pictures are OvExp. In the background I put the flashlight, the exposure time 1s ;)
-
ML vs Canon
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FW3QT2j8R%2FML100.png&hash=5f099d768bde041c0d4252cc17212ba8) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F8Cj2s9jT%2FVRAM0.png&hash=1d09f045295251898e996ace16137fd3)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FRCXLbFgb%2FML200.png&hash=8dc17bfec27e0a84452d5e0109e44e0c) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FHsv3jx57%2FVRAM1.png&hash=9b6c1aa084ee2330f64634affcf6ce26)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FMT0dGmbW%2FML400.png&hash=d5a9cbfac4905ebc7ee04da0759d6f6e) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F3JNFMT6r%2FVRAM2.png&hash=b0540fdf19f852da457428f24f16351e)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FRVZXMf3B%2FML800.png&hash=a375a61443bd3e171ecdd9f1df328fb8) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FBn8pF0c7%2FVRAM3.png&hash=44e0790abb3be21ec4e2d1cf7e961f2f)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FMHKbp2pD%2FML1600.png&hash=c5e9be337b7fe5ea8abb5731bd0b0aca) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F1tKrS5sX%2FVRAM4.png&hash=66b2f07c4ea4ad8f542fbc9dab5ef5ac)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F0yqRWcWf%2FML3200.png&hash=3e3472b71d95fb4e4e0ea04888cf7455) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F8CMbymYh%2FVRAM5.png&hash=7138eab4e75f321cf1050cfb1bea00bb)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fhvq6yGMG%2FML6400.png&hash=eafc362b2245dd2e86a7406c1172f9c7) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FPxHQ08G0%2FVRAM6.png&hash=5e10199f140c85652e7f303bd02275f9)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FtCdpHKVf%2FML12800.png&hash=67b00eaa21160b4a49f3cb8bbae95feb) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fj2g6WgWr%2FVRAM7.png&hash=600ab1ce4d4809b6c4d6330204471307)
edit: I performed the Canon exposures first, so any temperature influence should be in Canons favour.
-
Nice!
Do you see a stdev difference between 1/12s and 1/8000s @3200 ISO?
-
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F0yqRWcWf%2FML3200.png&hash=3e3472b71d95fb4e4e0ea04888cf7455) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FCdTt2xN1%2FVRAM1.png&hash=e732b61adcefb5468267728de697b406) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F3rsxhjfy%2FVRAM2.png&hash=e85943cf6d59d92e26acf4fde2bb0a0c)
-
Now, if my theory is right, the sensor saturation point (where detail gets clipped) should be the same in all these cases. Can you confirm or infirm?
Well this was a lesson in frustration. Between waiting for the light to warm up (60 mins) and trying to account for the inaccurate (not saturated) WL reading from ML (which I keep forgetting is not accurate when it's not saturated), I think I finally got some decent test images.
4 images @ f/1.4
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FtTW1FrYx%2FVRAM0.png&hash=2787f50de489f135af40c3343f706351) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F65yXS8fH%2FVRAM0.png&hash=b37383187e095675eaae180e9055b82b)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fj502qzZd%2FVRAM1.png&hash=d8cdbdf19ca4a0bf4b332a91b2802d64) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FBvDZzj4W%2FVRAM1.png&hash=1d7cff812b047889a818bf53f09f2f16)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FMTSZJzpn%2FVRAM2.png&hash=6422b4ed2826c5b9cccc2cd4b87f9e65) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F3JJRwrCp%2FVRAM2.png&hash=5e400394505d6ef9c2de79c325b45bcf)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FfysVCx9g%2FVRAM3.png&hash=533b25909f9303a9d1c7cb05c9ef4d87) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FpXGd5NqS%2FVRAM3.png&hash=f07165c3d88dbfbcc42a3cc637d404e4)
4 images @ f/4.0
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fg06rPMyT%2FVRAM4.png&hash=21506630c61d8b185263d3112ad7bf1d) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fd14bNFkP%2FVRAM4.png&hash=f5d92f9d3687df312431ece0c999c3bc)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FHkXfDXct%2FVRAM5.png&hash=5322fc173f341a2f147ab9c362a1dd1d) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FtJcLyzNB%2FVRAM5.png&hash=98a262b39ab0e5365bf356b99495feac)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FzXLs79q6%2FVRAM6.png&hash=cead908cd44589764a6128ceb3a07dd0) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FNfScwQNQ%2FVRAM6.png&hash=2149d73c1915b51265385d656fd72c4d)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F4dL0b1qX%2FVRAM7.png&hash=5e7066b7d84814713032be9e4be04428) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F5yXRPCWL%2FVRAM7.png&hash=e9fcaae32ac4db1403f75f6f89680eb9)
Taking a mean of all my previous saturated ML ISO 100 results, looks like DR is being reduced with increased aperture. stdev is not being reduced enough to compensate for the lowered WL. Also looks like f/4.0 is clipping more highlights. Whether this falls into round off errors, I'll leave that for you to decide a1ex.
Of course, it's entirely possible that the aperture blades weren't closing a full 3 stops either. Or they were closing slightly more then 3 stops. I left aperture in electronic mode to increase deviation.
-
I'd say it's because of different DOF, vignette at wide aperture, and the difference in how much light gets captured at f1.4 and f4.0 may not be exactly 3 stops.
You can try something like a gradient (e.g. point a lamp to a white wall) and see where the clipping point is (like you would read it on a ruler). Take the test shots exposed at -0.5, 0 (both f1.4 and f4.0) and +0.5, to see the scale of our clipping point ruler. I expect the difference between f4 and f1.4 to be around 0.1 stops.
-
I'd say it's because of different DOF, vignette at wide aperture, and the difference in how much light gets captured at f1.4 and f4.0 may not be exactly 3 stops.
Starting with f2.8, Canon (and other manufacturers) secretly raise the sensor iso behind your back because unlike film, digital sensors cannot capture light from large angles - this effect gets very strong with fast lenses so that for example a f1.2 lens isn't able to gather much more light than f1.4. This effect might be one of the reason why Canon starts releasing slower primes with IS.
-
From looking at the registers, they only raise the ISO in the JPEG previews, not in RAW. This is what we are trying to confirm experimentally.
* note that by ISO I mean DxO definition, not overall brightness. They are trying to change overall brightness in RAW by faking the white level.
-
From looking at the registers, they only raise the ISO in the JPEG previews, not in RAW. This is what we are trying to confirm experimentally. * note that by ISO I mean DxO definition, not overall brightness. They are trying to change overall brightness in RAW by faking the white level.
Ok, thanks for the explanation - it would be interesting to know how they actually do this, the current theory at least from all people I've read from on CR is that they raise iso, but then again few photogs have the in-depth tech knowledge to really tell.
-
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FvT0NhtKQ%2F1-30.png&hash=b195a607853c6cf194d49359d6d564c9) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FL8Ybkf3p%2F1-4.png&hash=6d6ae22d892baad4233b536b54b14bd7)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FxTFZbgH5%2F1-20.png&hash=fec6338a878f05dd17b80fe8df84c5d1) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FG3jw9kqg%2F0.3.png&hash=503108b45056c259c29e379bda6f256e)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FQxMPtvBV%2F1-15.png&hash=9d08bbe4f483e8f838502f6bd69f0632) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.imgur.com%2FHMDA9Do.png&hash=1f37c1d0a058c9560a9e743eb37536c0)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FpLpg03x3%2F0.5-f1.4.png&hash=ad41a02d7ba4ed88598817e42bb0839a) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FHkRFF3kn%2Fimage.png&hash=10a9968d78182411a2fcdea4f2ba0769)
Pretty sure I got it all right, been a while since I've slept. I've got all the files uploading now if you want to take a better look.
-
Just fixed a typo in raw_diag (the sampling area for optical black was too small). Black level, noise stdev and dynamic range should be more accurate now.
Also added aperture on the graphs, since it affects the white level.
raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (source: raw_diag.c (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.c))
-
Yeah, that's better. I'll post some updated graphs tomorrow if someone doesn't beat me to it.
Is there a specific reason why the screenshot is limited to 10s? PITA when trying to pump out heaps of them quickly.
-
Good idea, added a screenshot feature to raw_diag. Should be easier to get the screenshots in sync with raw files.
-
Hopefully this solves this riddle.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FtJnqG7HG%2FVRAM6.png&hash=8c4dcf2f728ec9b401b8e34f3ffcf5c4) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F8CwPhqb9%2FVRAM7.png&hash=ad39819e7d71e10414325009b59b6ce3)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F3rzp4vNp%2FVRAM2.png&hash=333010c6862bc0e88405fee49c0183b2) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FG223zq72%2FVRAM3.png&hash=29690e191dd074c9215e860f09a20e9b)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FfRcbb3Sw%2FVRAM0.png&hash=b8bcbe7cd442bce3c60797b4044392c8) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F63mWPngh%2FVRAM1.png&hash=1d1bb7bfa76eb6c31c70bef16307d510)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FpyN2Wx2h%2FVRAM4.png&hash=99b9ec5dc139920c1cf8ccdc8ec7957f) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FSQnCxrSk%2FVRAM5.png&hash=f0c8e156ea05100d3c26ab64a1a1d118)
* note that by ISO I mean DxO definition, not overall brightness. They are trying to change overall brightness in RAW by faking the white level.
Boosting the WL of low aperture values?
That last histogram is getting close to Gaussian!
-
Most raw processors use some hardcoded white level for CR2 files (dcraw, ufraw, Adobe DNG converter, so probably all Adobe apps). So, with this trick, Canon effectively added a small boost in image brightness at default settings for large apertures.
For example, Adobe DNG converter simply sets the white level to 15000 for 5D3 images taken at full-stop ISOs. DCRAW uses a hard-coded white level for each camera (which is plain wrong, and the cause of probably all pink highlight issues in dcraw-based raw processors).
With a program that reads the white level correctly (e.g. by looking at brightest white values or by figuring it out from EXIF), the effect of this ISO boost disappears.
Now, I don't know how Adobe handles the data above the white level. I know for sure that ufraw clips it (so here it's important to convert the files to DNG and set the correct white level manually). If Adobe doesn't clip the data above white level, and uses it for highlight recovery, all is fine (you did not lose any highlight detail because of Canon's tricks). But if Adobe behaves like ufraw, you'll have to use exiftool to fix the white level.
My conclusion is simple: Canon's aperture trick does not remove any highlight detail from CR2 files at f1.6 and smaller apertures. BUT raw processors that do not handle the white level properly may ignore the detail above what they consider to be white level, and clip it.
At f1.4 there seems to be indeed an overflow of 0.03 stops (which I would consider a bug in Canon code). From f1.6 there should be no differences in dynamic range.
-
Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic? I still feel like 1.2 would get more light to the sensor even if its not an exact amount. right? Canon treats us like babies sometimes. Id rather know exactly how much light my sensor is getting rather then this fake boost thing. ehhh but who care cuz ML iso is so much better anyways =)
-
A lot of the information gained with these registers should help with cr2hdr and such correct? With histograms for instance, there's no more guessing of WL. Discard some hot pixels and you're good to go!
-
Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic?
When I unsrew the lens on my 5D3 it uses the non-boosted white level. Now that I know what's happening, I'll post a comparison tomorrow of f/1.4 boosted vs non-boosted.
-
When I unsrew the lens on my 5D3 it uses the non-boosted white level. Now that I know what's happening, I'll post a comparison tomorrow of f/1.4 boosted vs non-boosted.
Maybe I'm just suffering from a placebo effect, but when i do this, take one shot with the lens detected and one shot with it screwed half off, I see a considerable jump in brightness when the lens is properly detected at 1.8/f on a 50mm using the neutral picture style and shooting in jpeg.
-
Yes, this confirms our theory.
-
Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic? I still feel like 1.2 would get more light to the sensor even if its not an exact amount. right? Canon treats us like babies sometimes. Id rather know exactly how much light my sensor is getting rather then this fake boost thing. ehhh but who care cuz ML iso is so much better anyways =)
The only reason I can think of canon is doing this (and probably all other camera manufacturers) is because most photographers expect exactly 1 stop extra light when they're going from f2.8 to f2.0.
If it's really true that the digital sensors don't get all the light with apertures bigger than 2.8, you must do this trick to get that extra stop of ligt on you're histogram.
But when you're after the best image quality, dynamic range etc. you're (a tiny bit) screwed.
-
I just found something very interesting with my camera (which I think is the basis of this whole discussion).
So people can replicate;
550D -> Video mode -> Expo Menu -> Iso Menu.
I set the canon analog to 100, canon digital to 0, then ML digital to +7.0 EV (giving an equivalent of 12800).
Aperture set to something below 2.0/f
Now, when one scrolls through the canon digital 0 EV has about the same amount of noise than 0.5 EV
But when one sets the aperture to 1.8/f
Now, when one scrolls through the canon digital 0 EV has ALOT more noise than 0.5 EV
Wait a minute. That could be my own stupidity.
When it's set so low it has nothing to pull up but noise, hence why when the aperture is wider it has more image to pull up.
Or is it, I don't know now.
-
Here's a research tool (or hacker's tool if you prefer) that lets you change most ISO-related parameters on 5D Mark III only and study their effect:
* CMOS gain, described in the Dual ISO PDF
* ADTG gains (0x8882 ... 0x8888) - same value for all of them
* Saturate Offset (0xc0f0819c)
* Digital Gain (0xc0f37ae4/af0/afc/b08)
* Black/White Offset (c0f37ae0/aec/af8/b04))
iso_regs.mo (http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.mo) (source: iso_regs.c (http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.c))
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso_regs.png&hash=5de17337d977b6a988333d07be240ad5)
Notes:
- Do not mix photo mode settings with LiveView ones. Hit "disable overrides" before switching between photo and video.
- It requires a custom-compiled ML with CONFIG_GDB = y. As I said, it's not for end-users, it's a research tool.
- The parameters are explained in the help text. More info here (http://www.magiclantern.fm/forum/index.php?topic=9867.msg96106;topicseen#msg96106) and in this thread.
- This has a TON of parameters hardcoded for 5D3. Porting on other cameras is very hard. Assuming the registers behave in the same way, there are only 25 camera-specific parameters. On 60D for example, only ADTG gains are the same, so you will have to figure out all the other registers with ADTG_GUI and do the math from scratch. Don't even think about trying on 5D2/500D/50D.
- To disable Canon's aperture trick, simply fix digital gain to some value.
Be careful not to fry your sensor :D
-
there are a few 6d users here, like 1%.
Is there anything for the 6d at this moment?
-
there are a few 6d users here, like 1%.
Is there anything for the 6d at this moment?
For the simple ADTG tweak, see the response below.
For the above.
Porting on other cameras is very hard.
a1ex opinion of easy, is suitably hard for most users. Here he describes very hard with bold!
-
Besides, for 6D somebody has to fix the raw backend in photo mode first: www.magiclantern.fm/forum/index.php?topic=10028
-
I got the same results with the holding down the DOF button while unscrewing canon lens. I saw the jumps in the raws too in adobe LR 4.4. Didnt try recovering highlights but i shot 1.4 1.6 1.8 2.0 and 2.8 on canon 50mm with there unscrewed twins. It didnt really start to level off till about 2.8 5d3 I also compared it to 1.8 on an old 50mm nikon. Good news is canons 50mm 1.8 unscrewed with DOF button held is brighter then the 1.8 on nikon 50. 1.4 unscrewed was still brighter then all the apertures so other then it being soft its still good to use if you need that light. I notice a slight difference in color to even though it was all the same temp. So i guess that might have to do with the black and white levels
Will this jump occur with ML RAW video? Ill test that next, I assume yes since it seems like its in the RAW concoction duringr atdc.
When this is done this will be an amazing mod to have! I dought it but i hope this helps a lil.....il shut up now
-
Can you compare the clipped highlights at f1.6 normal vs unscrewed?
(I want to see how Adobe handles the white level)
-
500D (default) :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Ffrs15f6fx%2FVRAM0.png&hash=4d2eec5c0690fee9d50c978255b56433)
ADTG3[0] = 0x149 (default) -> 0xb9 (trick)
ADTG3[3] = 0x14a (default) -> 0xba (trick)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Fhkuxtqrml%2FVRAM5.png&hash=16033982516e7907ba884524df9695a4)
500D - T1/200s, F1.4
ISO 100 160 200 320 400 640 800 1250 1600 3200 6400 12800
ADTG3[0] 149 149 149 148 148 148 148 142 142 142 142 142
ADTG3[3] 14a 14a 14a 14a 14a 14a 14a 145 145 145 145 145
CMOS[0] 0 90 90 1b0 1b0 1f8 1f8 168 168 168 168 168
C0F0[819c] f87 f87 66f 66f 66f 66f 66f 66f 66f 66f 66f 700
C0F0[8030] 1570 1126 1570 1126 1570 1126 1570 1126 1570 1570 1570 1570
C0F0[8034] 1079 1079 1991 1991 1991 1991 1991 1991 1991 1991 1991 1900
-
Can you compare the clipped highlights at f1.6 normal vs unscrewed?
(I want to see how Adobe handles the white level)
Sure. Sorry Just crashed for a while. My shots did include blown highlights. Il post them and there recovered twins in a bit
Sent a pm a1ex with raw files included
-
At f1.4 there seems to be indeed an overflow of 0.03 stops (which I would consider a bug in Canon code). From f1.6 there should be no differences in dynamic range.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FZK030LJJ%2Ff1.4.png&hash=616b11fa1fb8b574a88c8d8f6f888566) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F9MWw2dx5%2Ff1.6.png&hash=c648a3a66e3f59fb18ffc8cd75e2380f)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F1tJn38Tg%2Ff4.0.png&hash=f9987c24cc011555812064e19549969c)
0.003EV. I did the f/4.0 shot first, so the camera was a little warmer in the other shots.
-
Alright, so the side effects of Canon's ISO/aperture trick are:
- small overflow at f1.4 (0.03 stops), larger at f1.2 (I guess 0.25 stops from DxO chart, link below). This results in highlight loss in the CR2.
- histogram gaps indicating digital scaling (not quite good for denoising algorithms)
- possible highlight loss in some raw editors (ufraw for sure; I'll fix it in ufraw-mod). To check your favorite editor, take two test shots, f1.6 normal and with lens unscrewed by holding the DOF button, and compare the highlights.
http://www.luminous-landscape.com/essays/an_open_letter_to_the_major_camera_manufacturers.shtml
The good news is that I can disable this trick fairly easy on all cameras.
-
Good read. I'll do a heap of tests later showing Canon vs ML with the WL capped and brightness fixed in post.
This will be a good test also to see just how much light loss there is at each aperture value below f/4.0 for my lenses.
-
500D - T1s, ISO 100
aperture - F 1.4 1.6 1.8 2.0 2.2 2.5 2.8 4.0 8.0 22
digital gain - C0F0[8030] 1570 130a 11eb 10f5 107a 107a 1000 1000 1000 1000
F1.4 lens unmount, the value is 1000
unmount lens (So that the contacts do not touch.):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs10.postimg.org%2Fq8x1x57ex%2FVRAM2.png&hash=503686df1351c7a807b002af7d614ca4)
mount lens :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs10.postimg.org%2Fvydahgdl5%2FVRAM3.png&hash=09a755b115948d11daf0da79e8815bab)
No lens affects the colors:
unmount lens :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Fk4tbw3gj1%2Funmount.jpg&hash=98330252fefa14ff588dd473eaf7782d)
mount lens :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Fq3756bzhp%2Fmount.jpg&hash=5af8dd5b4945e8e3e7602edcbac9d348)
mount lens + trick (digital gain 1000):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Fs91g0u2y5%2Fmounttrick.jpg&hash=0f9d38fad672ee21d67e9ad3da4572fc)
-
Shutter speed.
Changes are 1s and longer.
1s :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2F3p02r92st%2FVRAM0.png&hash=07972bf3d7b7eaaa68370f35e5964a08)
10s :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Fcy291dbot%2FVRAM1.png&hash=f9c23ab7a45c596cc90313008ef3bba2)
I found something interesting. Turn on the camera. ADTG registers -> show set Modified since.
Take a picture.
ADTG3[0] = 0x149 (was 0x111)
ADTG3[3] = 0x14a (was 0x111)
-
No lens affects the colors:
For me (5D3), with consistent white level, the as shot temperature doesn't vary, and the tint only varies by +/- 2
The RGB values, of the RGB patches, of an x-rite colorchecker only vary by +/-1.
My light source was a led light, so it should have fairly consistent temperature output.
-
For me (5D3), with consistent white level, the as shot temperature doesn't vary, and the tint only varies by +/- 2
The RGB values, of the RGB patches, of an x-rite colorchecker only vary by +/-1.
My light source was a led light, so it should have fairly consistent temperature output.
I set the same color temperature in ACR.
-
2 images, 1 with lens attached and 1 with lens unscrewed. Both at f/4.0 to ensure no secret gain changes.
Both developed with the same settings in ACR. Camera faithful profile, no noise reduction, no lens correction, as shot color temperature.
Layered in PS with a difference layer blend on the top layer.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2Fcojf1vu.png&hash=ab92b082d9966f34244820e07e30eaea)
Only difference is the slight shot positioning error.
unmount lens (So that the contacts do not touch.):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs10.postimg.org%2Fq8x1x57ex%2FVRAM2.png&hash=503686df1351c7a807b002af7d614ca4)
mount lens :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs10.postimg.org%2Fvydahgdl5%2FVRAM3.png&hash=09a755b115948d11daf0da79e8815bab)
Ouch.
5D3.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FQCQz7rQW%2FVRAM1.png&hash=3cf5f42677dc791264fe77724dde75e8)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F6qJSTQvX%2FVRAM0.png&hash=99b00c97a523e817407f17219a470e83)
-
500D: log2(10.71)-log2(8.62) = 0.31 stops of ISO boost at f1.4, apparent ISO 100->124
DR 10.492-10.486 = 0.006 => real ISO 100->100.5
5D3: log2(7.24)-log2(6.57) = 0.14 stops, apparent ISO 100->110
DR 10.976-10.952 = 0.024 => real ISO 100->101.6
Apparent ISO: image brightness in JPEG preview and maybe at default settings in a raw editor.
Real ISO (by DxO definition, that is, from clipping point) assummes correct white level handling by the raw editing software.
If the raw editor assumes some fixed white level and clips everything above it, apparent ISO will be closer to reality.
According to DxO, the difference can be noticeable at f1.2. Any volunteer to check it?
You don't need ML to run the test; simply take a test picture at f1.2 with the lens normal and then with the lens unscrewed from the body, compare the highlights and measure noise stdev from the left optical black area. Or just upload the two CR2's.
-
Any particular settings? ISO etc as its night time here.
I have an 85mm 1.2 here, and also have a canon cinema lens T1.3 if that helps any.
-
Yes, M mode, ISO 100, aperture wide open, and the image should include clipped highlights (e.g. a light bulb). Same settings in both (only change screwed vs unscrewed).
-
4 images uploaded to dropbox as I don't seem to be able to upload to forum.
First 2 are from the 85mm 1.2 1st with data 2nd unscrewed
Second lot are using the canon cinema 50mm T1.3 not sure if they will help with any comparisons.
If these are not what you require let me know and I can try other shots.
https://dl.dropboxusercontent.com/u/49168208/85mmf1.2.CR2
https://dl.dropboxusercontent.com/u/49168208/85mmunlockedf1.2.CR2
https://dl.dropboxusercontent.com/u/49168208/50mm-t1.3.CR2
https://dl.dropboxusercontent.com/u/49168208/50mmunlockedt1.3.CR2
-
From the first two:
Noise 7.0886 vs 6.1815 => ISO was boosted by 0.2 stops
DR 10.98 vs 11.06 => you lost 0.08 stops of highlights because of Canon's trick. Not really noticeable in practice.
If you have an APS-C body, the difference may be a higher (DxO suggests 0.5 stops).
-
Maybe the image wasn't the best, would you rather have something say black then a candle beside it to make sure the whole dr is covered?
DXO says that the real light for that particular lens is T1.4
Will be interesting to hear regarding the cinema lens, as there should not be any "trickery" needed by canon.
-
For black, each CR2 has an optical black area (that's where I'm measuring noise). For a visible difference, some gradient pattern (gradual transition to clipped white) would be better.
The shots with the cinema lens were not taken at the same shutter speed.
-
Here's a couple more from the cine lens T1.3 with same shutter speed.
Not the sharpest, as hand holding and trying to get it to clip at a low shutter speed, hope that will be OK, can take better ones during the day tomorrow with a tripod if needed.
https://dl.dropboxusercontent.com/u/49168208/50mmt1.3.CR2
https://dl.dropboxusercontent.com/u/49168208/50mmt1.3locked.CR2
-
I also have the 85 1.2 on a 5D2 if that would be useful.
-
50mm t1.3:
Noise 7.11 vs 6.16 => iso boost 0.2 stops
white 16383 vs 15283
DR 10.98 vs 11.07 => highlights lost 0.09 stops
Would be great if you could take this shot on tripod, so I could do a side-by-side comparison of the images.
-
7D 50mm f1.2 pentax, can I help?
-
7D 50mm f1.2 pentax, can I help?
I don't think so. The camera needs to know that it has connected the lens and the current aperture.
-
50mm t1.3:
Noise 7.11 vs 6.16 => iso boost 0.2 stops
white 16383 vs 15283
DR 10.98 vs 11.07 => highlights lost 0.09 stops
Would be great if you could take this shot on tripod, so I could do a side-by-side comparison of the images.
Not sure you are experimenting with the right things.
2 images below set all in manual and using the latest firmware 1.2.3
I can tell from the histogram before shooting that whenever you unscrew the lens slightly that its clipping sooner.
I had to move the clipped light from 1/40th to 1/20th to get it to clip in histogram when it was clipping at 1/40th when locked
Both shots taken exact same settings.
50mm T1.3
https://dl.dropboxusercontent.com/u/49168208/locked.CR2
https://dl.dropboxusercontent.com/u/49168208/unscrewed.CR2
-
I can tell from the histogram before shooting that whenever you unscrew the lens slightly that its clipping sooner.
Histogram before shooting is not exact (even ML raw histogram in LV it's an approximation). Histogram *after* shooting (e.g. in RawDigger) is the one you should look at.
-
I have some good news for 5D2 (credits go to Greg, he found it on 500D):
- SaturateOffset register works and I have moved the black level from 1024 to 64.
- According to my 5D3 theory, this would be equivalent to reducing ADTG gain by -0.091 EV (ISO 100->94).
- However, the dynamic range in raw_diag is showing an improvement of over 0.7 stops at ISO 100 (!)
What's going on?!
edit: diagnosed. In some Canon cameras, bad pixels are marked as 0 in the raw buffer (and then interpolated). In raw_diag, I was not doing any checks of the raw pixel values (so, a bunch of zeros compared to 1024 were bumping stdev a lot more than the same bunch of zeros compared to 64).
The real improvement was close to 0.1 stops, so the theory about Saturate Offset is still valid.
Updated raw_diag to skip hot pixels:
raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (source: raw_diag.c (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.c))
(the bad pixel modification should be backported to ML raw.c)
-
I have this sick obsession with tweaking things. Hence the nut at the end of audio ;)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FrmKc5zsR%2Ftweak.png&hash=5f9a201cbbd997893f2aaaa351dde008)
vs Canon vs mini_ISO
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FqvxkT9Wq%2Fcanon.png&hash=28f88f353e485193f05a5304907065c1) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fd0QJt5KB%2Fmini_ISO.png&hash=94702095c5b089835c6cfa00d11dd342)
Off now to take some sample images and see what the change is visually.
-
I have sighted bias, so I would like some opinion on these 2 test images before I give my own.
1
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F4NGNTyT3%2Fimage.jpg&hash=d17a053c3c399d5091e2a2de3b928751)
2
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FZnSZZ72Q%2Fimage.jpg&hash=040ac89cf7a23da6980967b74efd331b)
edit: I can still see, what I think I can see, in these resized and JPG (minimum) compressed images. I can upload sample CR2s at a later date if requested.
-
I'm no expert and its hard to tell online but picture 2 for me seems noisier than 1 and the colours are flatter, or maybe I'm just seeing things because I'm expecting a difference.
-
I see a clear difference in colours but not in terms of noise, may be the contrast of picture 1 is weaker which leads to less noise. I prefer colours of shot 2, shot 1 seems to be more red.
-
Nikon 500D :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs17.postimg.org%2F9ncdyl2y7%2FVRAM21.png&hash=2b9819f6c19a31a7e5ff659feb064052) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs17.postimg.org%2Fi4bw9i7n3%2FVRAM20.png&hash=7c40c75a08f88bad00fc3066640817ef)
-
PM response is in favour of image 1. In my opinion, this image has less banding noise. Check the blurred background left and right of the colorchecker. It's easier to see in full res CR2.
This image (1) is with these settings.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FFs0djwhY%2Fsettings.png&hash=268ff98b65cabb8be681bee7a0d31f0c)
Further testing favours a digital gain value of 462.
Note: This is only with ISO 100, and only tested with a 5D3 (Greg would be interested to see results with 500D).
-
PM response is in favour of image 1. In my opinion, this image has less banding noise. Check the blurred background left and right of the colorchecker. It's easier to see in full res CR2.
TBH both images were pretty noisey, just no2 was more so.
I have only seen noise like that at much higher ISOs or where they have been considerably pushed in post processing.
-
In practice, the difference is not so big. :(
Canon 500D (-3EV ACR):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs23.postimg.org%2Fv5egtnyu3%2Fdefault.jpg&hash=8ede21d4a7826e0875cc0d1549a4508a)
Nikon 500D (-3EV ACR):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs23.postimg.org%2F3w33f5xqz%2Ftrick.jpg&hash=757e0d001e45f7f163465978b9e1da3b)
Crop 1:1 (+2,5EV ACR):
Canon 500D :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs11.postimg.org%2F67wwbutzn%2Fimage.jpg&hash=f7eeed8a5b56f650120fed72d3265aad)
Nikon 500D :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs11.postimg.org%2Fx4gvk6csz%2Fimage.jpg&hash=de7109e9e759c026cb6b5041a15aa720)
-
I have only seen noise like that at much higher ISOs or where they have been considerably pushed in post processing.
I expose so that the white patch (in the colorchecker) is exposed as high as possible before over exposure. The images in the above test are exposed 5EV lower with shutter, and increased 5EV in post. Note: This should put the black patch around -9.7EV (http://www.magiclantern.fm/forum/index.php?topic=9867.msg96563#msg96563). I WB and brightness correct from a correctly exposed image. This removes any error due to noise, and obviously, ensure lighting remains consistent.
To be clear, image 2 was an ISO calibrated mini_iso result. And the difference was always going to be minor, because I've already shown (http://www.magiclantern.fm/forum/index.php?topic=9867.msg96563#msg96563) the gains being made with just ADTG tweaks.
The quickest way to see any differences, is with images down near the noise floor IMHO.
When I get time to sort out a 32bit (ACR limited to +/-5EV) raw based workflow, I'll test further into the noise floor ;)
-
In practice, the difference is not so big. :(
The Nikon 500D result looks slightly better, but it's biased by a lower rendered brightness.
-
I expose so that the white patch (in the colorchecker) is exposed as high as possible before over exposure. The images in the above test are exposed 5EV lower with shutter, and increased 5EV in post. Note: This should put the black patch around -9.7EV (http://www.magiclantern.fm/forum/index.php?topic=9867.msg96563#msg96563). I WB and brightness correct from a correctly exposed image. This removes any error due to noise, and obviously, ensure lighting remains consistent.
Sorry new about here.
Thanks for the explanation, makes much more sense now ;)
-
Would it be possible to option a change of color of the SNR curve, or an option to only screenshot the curve without the background data. This should make it easy to overlay data in post.
I can get this, but it misses the fine detail in one of the curves.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FmkZYLrb9%2Foverlay.png&hash=492ec046efb7706bd7ce9f427bfc37fd)
With my tweaks on the last page. Changing only ISO in canon menu and CMOS gain in ML.
https://www.dropbox.com/sh/dwobqot98fdm8k3/AAC3WYZhGtwzuHtK_99_trKRa?dl=0
-
The SNR curve needs some offset tweaking first. Since we are tweaking the ISO (getting a different clipping), this results in shifting the curve horizontally (so, 100% in Canon ISO should not be the same as 100% in ML ISO, but right now on the graph they appear to be the same).
Instead, I'd like to consider the test scene as reference (that is, when you double the shutter speed and the shadow noise gets twice as small, you should see a 1-stop difference on the graph). That means we should shift the curve horizontally with the Bv value (from expo menu) and the ISO correction computed by the iso_regs module.
-
The benefit oft the mini_iso module is automatic, but I still think adding it directly to auto_iso in M mode would be a good idea. In Av and Tv, you cannot do anything about it, the user has to add the +ec gained by ml (since we have no way of intercepting the shot to modify the ec, then reset it afterwards).
But if auto_iso & autoexpo set iso, aperture, shutter and ec in m anyway, wouldn't be a good idea to modify the ec in m value with the exact mini_iso.mo value to always gain optimum exposure? Should this be done in the modules or could mini_iso please deliver a weak function to return a modified Canon 8-system value according to the requested iso?
-
I assume,
static CONFIG_INT("black.ref", black_reference, 0);
is ADTG black (0x8880)?
-
Yes, not implemented yet.
-
Yes, not implemented yet.
I was going to have a go at doing it myself :)
Instead, I'd like to consider the test scene as reference (that is, when you double the shutter speed and the shadow noise gets twice as small, you should see a 1-stop difference on the graph). That means we should shift the curve horizontally with the Bv value (from expo menu) and the ISO correction computed by the iso_regs module.
I'm not sure I understand. A 1 stop shutter difference shows this.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FL4JGR4bf%2FVRAM0.png&hash=2865fa01ab5f5c07ad07a5307e4281a3) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FRVf86s9Z%2FVRAM1.png&hash=6403ee2c5aa889e8145fdec2dcaecfb1)
The entire curve has shifted 1EV. 1/3s shutter shot had clipped highlights which is why the WL didn't shift a full stop. But the rest of the SNR curve looks like it did.
I know I don't understand what needs to happen for the true ISO adjustments. But the above data looks accurate.
-
Yes, but they overlap perfectly. Think at one single color patch from your test image: one of them will be rendered with twice as much signal, and roughly the same amount of noise (so the SNR should be roughly 1 stop higher on one of them). And it shows up correctly.
However, the two points on the curve corresponding to our patch are not at the same X position. I want them to be in the same position, so from two overlapped curves I should be able tell how big the improvement was for our patch (and all others). Now I can tell the exposure was shifted to the right, the curve did not really change with shutter speed, but I can't tell how better our patch was rendered.
So, there's nothing wrong with the curve, but to compare things it may be useful to change the X origin (link it to the test scene, not to the recorded data).
-
I like how it shows where the data is rendered in the 14bits also.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F66h4Ww4W%2F1.3.png&hash=4f9b44142a26db77c9a9efb432f116ff) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fc1T6qmP6%2F1.6.png&hash=8f09e17a36203e50f766dd90075194cd)
My highlight (red dot) got shifted up both scales (as it should IMO). I should have gained a little extra SNR, but it got clipped at 13.5EV, but I did gain bit depth precision.
My shadow detail (green dot) got shifted down 1EV in both SNR and bitdepth.
edit: I see your point now, it would be helpful to see how the SNR changed in the image without the bitdepth interference.
-
Are there more tests needed? Anything I can help with? Not good at programming but in the middle of a snow storm so just sitting here refreshing this page ha ha JK.
5d3 running the nighters
-
Updated some research tools:
iso_regs.mo (http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.mo) (source: iso_regs.c (http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.c)) - 5D3 only, requires CONFIG_GDB=y:
- added register 8880 (halfway done by Audionut)
- increased range of B/W offset
adtg_gui.mo (http://a1ex.magiclantern.fm/bleeding-edge/adtg_gui.mo) (source: adtg_gui.c (http://a1ex.magiclantern.fm/bleeding-edge/adtg_gui.c) avl.c (http://a1ex.magiclantern.fm/bleeding-edge/avl.c) avl.h (http://a1ex.magiclantern.fm/bleeding-edge/avl.h)) - cross-platform, requires CONFIG_GDB=y:
- also intercepts DIGIC registers (experimental, there are LOTS of them)
- it adds no less than 4096 menu entries (2048 were not enough on 5D3) => it will slow down boot and menu navigation.
- if you get ERR70 out of memory, change the max region heuristic in memory.c from free_space / 2 to free_space / 4 (or add a wrapper to GetSizeOfMaxRegion)
- it's not usable in LiveView, but if you really want to try, FPS override is your friend
raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (source: raw_diag.c (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.c)) - cross-platform, anyone can run it
- LiveView support (only relevant in movie mode, not in photo LiveView)
- updated README (including a small TODO list, if anyone feels like coding or doing some math)
- to read it, press Q in ML modules menu
-
I'm seeing some ADTG/CMOS registers being doubled/tripled in the menu. I'm seeing ADTG6[8000] 4 times.
-
SOLVED! thanks Nanomad for help
-
edit: Of course you need to subtract the mean to calculate DR.
-
Regarding raw_diag.mo, this is a very minor quibble, but you may want to tweak the definition of the optical-black region for the left black bar. You start at y = 20 instead of y = raw_info.active_area.y1 + 20 as is done in raw.c. The 50D, at least, has a structured pattern for the first ~thirty lines of the top black bar, so starting the optical-black region that early picks up that pattern. It's so few pixels that it probably doesn't really affect any histograms, but you might as well play it safe.
-
Solved.
Also updated the first post with a little FAQ and links to research tools. If you still have questions, just let me know.
-
Great work Alex and everyone! Appreciate all that you do!
-
As discussed on a recent pull request (https://bitbucket.org/hudson/magic-lantern/pull-request/344/50d-fixed-starting-location-of-raw-photo/diff), a massively overexposed frame affects the left optical-black region for the 50D (but does not in the 5D Mark III). This effect complicates determination of the black level and the read noise, and consequently can throw off the measurement of dynamic range.
The first plot shows the histogram for raw_diag.mo with only minor overexposure. The measured read noise is comparable to that of a dark frame with the same exposure settings. The second plot shows the histogram when the image is massively overexposed. You can see that the read noise appears to be larger, which appears to reduce the dynamic range by ~0.2 EV.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2F4pa2t3ft3hah2xs%2Fraw_diag_0250.png&hash=ec17dd9e76d0df947c14f728e3320d15)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fq945sbuhw9d4bw3%2Fraw_diag_0247.png&hash=327c0c1e6cb61176e5d6dfac5baa6896)
Diving into the images themselves, here are the histograms for pairs of columns, starting from the left edge of the sensor (the bottom-most line) and going to just before the image itself (the top-most line). The solid lines (as opposed to the dashed lines) are the columns in the inner optical-black region used by raw_diag.mo to determine the black level and the read noise. The first plot, corresponding to the minor overexposure, shows that all of the columns in the inner optical-black region have about the same mean and standard deviation. The second plot, corresponding to the massive overexposure, shows that all of the columns have about the same standard deviation, but there is significant drift in the mean, and averaging over this drift is what is producing the appearance of a larger read noise and a slight decrease in the black level.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Ffld1glcryfcxkki%2Fvariation_0250.png&hash=1238f3eee558e7488c87623a6e464a3a)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fs9y77vl6h1roluk%2Fvariation_0247.png&hash=756471272162b0b3935356384a856c6c)
Thus, the read noise isn't actually changing, which is good, but the apparent drift of the black level is not good. I don't know whether this points to some form of sensor bleed or a deficiency of the black-level clamp. Either way, on cameras with sensors like the 50D's, take care not to overexpose images by too much.
-
Have you tried making small adjustments to 0x8880?
-
My investigative work so far.
Old settings vs New
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FTY56ftk7%2Fsettings.png&hash=06f50a1cf834eb40d032e369b89ef26b) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FXqTtr022%2Fsettings2.png&hash=c27232d4c5cfaaf9da0c7f06235347b7)
mini_iso vs old settings vs new
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fk5PpvxJv%2Fmini_iso.png&hash=471300fb170b3b6d7d74a98cc9fbca10) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FvTKPgn29%2FVRAM0.png&hash=46eecf9c8fa341043f597d55371dabd0) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FNMNcfSqV%2FISO100.png&hash=302eb2173b327a62951adb798fe87c36)
With the new settings, I've brought the WL back to 15281 and cleaned the histogram further. However, a close look at the histogram suggests there is probably still some room for (slight) improvement.
@a1ex: Notice the Digital Gain default changes. I have no idea what causes this. I've tried retracing my steps but can't narrow it down.
@all: No sample images as yet, but my last samples (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97073#msg97073) showed that the old settings vs simple ADTG gain adjustment delivered an improvement in image quality.
Digital Gain shifts the entire exposure. This appears to be digital as it has a large impact on the histogram. Try adjusting this one first.
Black/White Offset and Saturate Offset are closely aligned. Adjust either one by N steps and you can adjust the other one by the same N step (in the opposite direction) to bring the histogram back to original. Here, you can use the BW offset to compensate for digital gain reduction (bring the WL back) and Saturate offset to adjust the black mean.
So for instance, you can adjust the WL higher then 15281 with BW offset (up to 16383), and use Saturate offset to bring the black mean back to 2048. Note: I have tried this and didn't see any improvement (slight negative effect actually), but I've been looking at optical black histograms and adjusting registers for the last 5 days or so, so YMMV.
After I adjust all these, I reduce the ADTG gain to the sweet spot (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96106#msg96106).
I haven't been able to find a pattern to black reference, but I also haven't spent much time with it. This requires further investigation.
The effect of my new settings on all ISOs. Note that stdev can vary from shot to shot.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FNMNcfSqV%2FISO100.png&hash=302eb2173b327a62951adb798fe87c36) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FGpdwPdbV%2FISO200.png&hash=c6af90b28aae6f84dda48a895a541562) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FqRDPdK2H%2FISO400.png&hash=67f13ab427727f4d4cb31b207759c855)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fyd8M5jgT%2FISO800.png&hash=b381ca19b567eb961da5abaeb7cdd30d) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FtJ1LRgzZ%2FISO1600.png&hash=6043424e77c4822228c089bc61099f5f) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FSQ5BTWyW%2FISO3200.png&hash=3b8b121ff0e0c7dcaeed54627d1cb42a)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FdVMK1Gpq%2FISO6400.png&hash=2bf359f7fdf23d79d8d869811215cdfd) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FfypQYdbf%2FISO12800.png&hash=03ff0be52fb1ea4b78ef7bfcc41852b3)
-
@ayshih: for these cameras I would point the camera to a light bulb in the middle of the image, and expose until the edge pixels become dark. If you have like 1% of the image overexposed, it's enough for DR calculation.
@Audionut: default digital gain changes with aperture (it shows Canon is trying to boost the ISO).
Tip: you don't really need to copy Canon settings; you can override only one or two settings at a time, and leave the others to Canon defaults. Might be useful when doing tests at more than one ISO.
I'm not yet sure about 8880 (at first sight it looks it was already tuned to sweet spot by Canon).
-
Have you tried making small adjustments to 0x8880?
I'm still getting the hang of looking at ADTG registers, but I don't believe the 50D uses that particular register, so I don't know how to change the black level. (As an aside, I also haven't figured out how to change the ADTG gains on the 50D. Inspired by Greg's testing with the 500D, I tried overriding the ADTG3[0] and ADTG3[3] registers, but they don't appear to affect anything.)
Thinking more about the effect I've described, it looks like the black-level clamp just has a characteristic time for clamping to the reference voltage (presumably generated by the register value and a DAC), and actually oscillates a bit about the desired level as it gets there. The amplitude of that oscillation then corresponds to how much the clamp needs to shift the signal in voltage. If you consider the signal on a given line in an AC-coupled sense, a massive overexposure have a high average charge per pixel, which means that pixels in the optical-black region will be very far from the reference voltage, and the resulting amplitude of the oscillation to be large. The 5D Mark III presumably has a different, more effective black-level clamp.
If that's the right description, my guess is that there'd be no benefit from small changes to the black level. Changing the black level by a significant amount could help to stabilize the optical-black region for images that are significantly overexposed, but would compromise normally exposed images, so that doesn't really make sense as a change.
Edit: I got a little mixed up about the direction of the needed change.
@ayshih: for these cameras I would point the camera to a light bulb in the middle of the image, and expose until the edge pixels become dark. If you have like 1% of the image overexposed, it's enough for DR calculation.
Yup, agreed.
-
If the damage is being done in the CMOS stage, you could presumably fix it there with the digital registers and correct the black level with ADTG. Just guessing though!
-
Yes, old-generation cameras do not use 888x registers from ADTG (there may be some other registers).
For 5D2, I've got a tiny improvement (0.15 stops) by setting SaturateOffset (0xc0f0819c) from 0x66f to 0x66f + 32 - 1024 - 624 and B/W offset (0xC0F08034) from 0x1991 to 0x1991 + 624. Also fix the digital gain (0xc0f08030) to 0x1000.
The effect of SaturateOffset is that it expands the recorded range while bringing in more highlight detail. So, if your default range is say 1024...15760 and you increase this range to say 32-16383 without changing digital gain (these are the 5D2 values), this would be equivalent to scaling ADTG gain to (15760-1024) / (16383-32) = 0.9 (that is, -0.15 stops). The theory is explained in get_resulting_iso (iso_regs.c).
Basically, ADTG gains and SaturateOffset adjust the same thing internally. So, to get say ISO 77, you can:
a) simply scale factory ADTG gains by 0.77
b) increase the range with SaturateOffset (e.g. with equivalent gain of 0.9) and scale ADTG gains by 0.85 (so the total gain is 0.77). The total gain is limited by the sweet spot (under this point, white level begins to decrease at least in some columns => you get artifacts).
So, on 5D2, the mystery is what register is used for ADTG gain. I believe this gain is used to implement ISO 125 from 100, 250 from 200 and 3200 from 1600 (because of lack of gaps in histogram), but the ADTG tool doesn't report any ADTG register changes when switching between these ISOs.
-
Out of curiosity, I tried two dual ISO shots to see whether the theory applies. Since the new trick only improves the highlights side, the theory says you will only benefit from the DR improvement at ISO 100 (because in shadows, the modified ISOs are just as noisy as before).
I had to update the white level detection routine in cr2hdr to consider two separate levels for bright and dark exposure. I'll only post the relevant lines from the conversion log. The subject was boring (a light bulb).
ISO 100/1600:
White levels : 15094 15109
Noise levels : 11.05 6.48 6.52 11.08 (14-bit)
ISO difference : 3.97 EV (1572)
Semi-overexposed: 9.47%
Deep shadows : 83.80%
ISO overlap : 4.0 EV (approx)
Dynamic range : 10.98 (+) 10.21 => 14.19 EV (in theory)
Dynamic range : 14.35 EV (cooked)
ISO 77/1230:
White levels : 15096, 15108
Noise levels : 8.60 5.13 5.03 8.54 (14-bit)
ISO difference : 3.96 EV (1559)
Semi-overexposed: 6.06%
Deep shadows : 84.38%
ISO overlap : 4.4 EV (approx)
Dynamic range : 11.35 (+) 10.58 => 14.54 EV (in theory)
Dynamic range : 14.68 EV (cooked)
Here, "in theory" means from analyzing noise level in input exposure, and "cooked" means right before saving the output file (might include some noise reduction, interpolation artifacts and whatever other tricks I'm using).
So, the DR improvement in this test was 0.33 stops (on the highlight side).
What about the ~0.5-stop improvement at the ISO 1600? I believe it translates into larger overlap between the two ISOs => less aliasing.
-
Audionut has sent me the corrected 5d3 stats, I changed them on my CR post (I didn't think this would make such a large splash, no ML development I have ever crossposted there has :-o)...
... one question that puzzles me: The DR for iso 800 is the same ML vs. Canon, what's this about? Also, did you do any recent changes that also invalidate the 6d calibration done by 1% this effect is also there?
-
After changing the reference for the SNR graph to be able to compare the curves as in ejm's article, I hope to find out. Right now, I expect something like 1/100 ISO 100 might be as clean as 1/640 ISO 640 pulled from 800, but I did not run any tests to confirm this.
The calibration is still valid. You may want to fine-tune it for your camera, but I guess it will stay within +/- 0.05 stops from the defaults.
-
just for my curisoty, (notting else, I can wait a long time for this coming to the 6d, take your time ;D)
Since the 5d3 and the 6d are the same generation, sensor and digic wise.
How different are they for magic lantern, is it totally different meaning trial and error or do they share most of the stuff and can this relatively easy be ported to 6d ?
-
How different are they for magic lantern, is it totally different meaning trial and error or do they share most of the stuff and can this relatively easy be ported to 6d ?
Wrong thread, but short answer: They seem to be very similar, but concerning ML the 6d is unmaintained and in an earlier state than 5d3, so features might make it to the 6d or not.
-
I correctly exposed @ ISO 100 and then boosted ISO to 1600. Loaded all images with same ACR settings (WB from ISO 100 shot, no noise reduction etc & highlight slider -100) and brightness matched from the black patch top left.
My ADTG settings vs Canon vs Canon -1/3EV with shutter.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fcw615aj4b%2Fmine.jpg&hash=542cc6de2be59ed6d7bf97e4eb92d2b9) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fok030bdjf%2FCanon.jpg&hash=5ad7374b86de2febe2910cc2c92fa18e) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fu9gbk4jkb%2FCanon-0.3.jpg&hash=a69eeddbad9aa9303eec656324f038be)
What about the ~0.5-stop improvement at the ISO 1600? I believe it translates into larger overlap between the two ISOs => less aliasing.
Indeed, on top of the 0.3-0.5EV less shot noise. Because it would be pointless to pull the highlights back if you don't then expose for them. :D
If you're DR limited then you gain 0.3-0.5EV of latitude. You can either gain that advantage in the highlights or the shadows.
-
I'm very much excited for this news, is it possible to port this module to 6D. And how much DR increase should i expect
.
-
For adtg_gui.mo, I've tested adding the DIGIC hooks for the 50D by grabbing the addresses from stubs.S, and they appear to work fine. I can see registers such as SaturateOffset (0xc0f0819c) , and I can still see ADTG registers (which is mentioned in a comment as an issue for the 5D Mark II).
else if (streq(camera_model_short, "50D")) // http://www.magiclantern.fm/forum/index.php?topic=6751.msg63322#msg63322
{
ADTG_WRITE_FUNC = 0xFFA11FDC;
CMOS_WRITE_FUNC = 0xFFA12190;
ENGIO_WRITE_FUNC = 0xFF97D904; // from stubs
ENG_DRV_OUT_FUNC = 0xff97d794;
}
-
I'm very much excited for this news, is it possible to port this module to 6D. And how much DR increase should i expect
Not reading the last page of a thread and the OP, is just bad form. Surprisingly enough, both of these places have the answers to your questions!
-
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fmffpytr7v%2FBell.png&hash=ed1d0826c78d271bd9bb1e89e8693329)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Ftinleg4d7%2FSettings3.png&hash=8abab32526fae028718707fb45694742)
Adobe doesn't have a problem with the black level, which is only offset from 0 to account for stdev (https://www.micro-manager.org/wiki/Measuring_camera_specifications#Measure_Offset_and_readout_noise_in_digital_numbers).
edit: This one doesn't scale well with ISO.
-
Added another analysis type to raw_diag: fixed-pattern noise (banding) from a dark frame (image taken with lens cap on). FPN is estimated by averaging each line (or each column), and the graph shows a plot of this correction "signal" and its stdev. It may be useful for testing the custom configurations from Audionut (since one of them attempted to reduce exactly this FPN).
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Ffpn.png&hash=e542ec0da3d3fbad7cd814ca2816217b)
(I've showed earlier in this thread (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96465;topicseen#msg96465) how to use this info to subtract the FPN a dark frame)
Source and binary in first post.
Some autocorrelation analysis would be nice, if anyone feels like doing the math.
-
Nice a1ex!
(...)
Some autocorrelation analysis would be nice, if anyone feels like doing the math.
I have some pretty nice FPN reduction algorithms, maybe it could help (maybe it is too gourmand for the DIGIC...).
-
I'm interested. Especially if you can estimate the FPN from an arbitrary image without extra dark frames.
-
Indeed, without master offset.
It's too big to post the code here, I send you a PM.
-
Canon vs My tweaks (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/my_tweaks/settings2.BMP)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Frpkoqgja3%2FFPN_100_Canon.png&hash=779c60fef668c7d56b8625c15d6d83c7) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F75furzb8r%2FFPN_100_Mine.png&hash=c46cb2a6d5af5e692dfe106fc8609f7a)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F3ltx26g8r%2FFPN_1600_Canon.png&hash=a310257f727656dd04049309d62b7bb9) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fxqidnjt1n%2FFPN_1600_Mine.png&hash=f9e0a1ab22aa813a851e71620db7bd18)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F9mrlz9iaj%2FFPN_6400_Canon.png&hash=0e9ecb788e587376fee6131bc96f43d0) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fovhjd1jor%2FFPN_6400_Mine.png&hash=96158f771288bed504d0076df6eefd4b)
FPN should be reducing with increasing ISO.
-
Yes, old-generation cameras do not use 888x registers from ADTG (there may be some other registers).
For 5D2, I've got a tiny improvement (0.15 stops) by setting SaturateOffset (0xc0f0819c) from 0x66f to 0x66f + 32 - 1024 - 624 and B/W offset (0xC0F08034) from 0x1991 to 0x1991 + 624.
On my 50D, the white level at ISO 100 is only 13432, so I can adjust those same registers to expand the range from 1023..13432 to 31..16383, which gives an apparent DR increase of 0.4 EV.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fi1cr5ff2fkawgkq%2Fraw_diag_0279.png&hash=a985e15203cacda476c0f7c66e542789)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fe8asngxvqsfspdt%2Fraw_diag_0280.png&hash=0e0903a71436026cbd85e6127a94c412)
I increased B/W Offset (0xc0f08034) from 0x1079 to 0x1c00 to max out the white level, and decreased Saturate Offset (0xc0f0819c) from 0xf87 to 0x20. (I just now realized that these are essentially the same settings you cited for the 5D2.) I find it curious that not only is this setting for B/W Offset "nice" as a multiple of 256, it also shifts the meaning of the Saturate Offset to correspond to the actual empirical black level (0x20 ~= 31).
-
Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.
(I probably need to include Canon's digital gain in my formula; can you check the default value of c0f08030 at iso 100?)
-
Just noticed that on 5D2, at ISO 3200 vs 6400, raw_diag reports exactly the same values for noise and DR (and raw zebras agree). However, the CR2 at ISO 6400 contains clearly less highlight detail, is brighter, has larger file size, and its histogram has gaps: http://www.guillermoluijk.com/article/isos5dmkii/index.htm
Could it be a multiplication of the raw values directly at CR2 compression stage?! (in any case, after ML intercepts the raw buffer)
-
500D (default) :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs22.postimg.org%2Fjg91w114h%2FVRAM0.png&hash=aa208d3f74a3de96ddd5f7e5a76a1d63)
500D (adtg 149 -> 100) :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs22.postimg.org%2Fyq8x37wmp%2FVRAM2.png&hash=8a41a8669e037d4817bf199ea31e51b2)
500D (adtg 149 -> 0, bad white level) :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs22.postimg.org%2F6eryjx241%2FVRAM3.png&hash=2c040a0751e199e8ff762b36fc6c5c50)
-
5D2 mistery solved!!!
Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.
Results:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d2.png&hash=f2b14c70fa5906ebeb268b9c987f6480)
Canon ISO ML ISO Improvement
100 82 0.29 EV
200 165 0.27 EV
400 332 0.27 EV
800 692 0.21 EV
1600 1432 0.16 EV
3200 1432 1.16 EV
Forgot to mention that new ADTG GUI also has an option to group registers by task or program counter where the register was changed. With this trick, you should be able to identify the ADTG registers on EOS M (look for those in ShootCapture task or something similar, not for the LiveView ones).
-
Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.
The green channels are solid white at 16383; the red channels are at 16065, and the blue channels are at 16001. Here are the CR2 files for default settings (https://dl.dropboxusercontent.com/s/52au0i32lu62vrb/IMG_0279.CR2) and tweaked settings (https://dl.dropboxusercontent.com/s/z98wnby7c1bstbe/IMG_0280.CR2).
I haven't looked too closely at the CR2 files, but here's a quick comparison of their histograms (with the channels lumped together). What needs more investigation is the "rise" before each channel reaches its max with the tweaked settings.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2F4zyej10q9tar4v5%2Fhistogram_0279_0280.png&hash=94a848ba1a9ad22629f80f35bc3cfd01)
(I probably need to include Canon's digital gain in my formula; can you check the default value of c0f08030 at iso 100?)
At ISO 100, the default value of digital gain (0xc0f08030) is 0x1000, so I did not change it.
-
On 5D3:
Digital Gain is white level control. Does not effect black. Does not effect stdev. Sweet spot is 463 with pushed black/white levels. Relax the contrast (black/white levels), and values down to 461 will still deliver a Gaussian bell curve.
ADTG gain. Not sure. Sweet spot (for ISO 100 and modified Saturate offset) is 1002. You can lower this value much further before affecting white level, but values lower then 1002 effect the bell curve. This one effects stdev.
Black Reference. No Idea. Agreed a1ex, Canon seems to have set this at sweet spot. I haven't been able to find any other values that deliver a bell curve.
B/W offset. Gain, this one shifts the entire signal. Tied to Saturate offset.
Saturate offset is black level control. Does not effect white. Tied to B/W offset.
Both offsets can deliver a Gaussian bell curve at different levels. Note here where I've changed black level with Saturate offset (fixed white level).
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fcfapovqij%2F2048.png&hash=bc6e4afd85364116aff026f705211462) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fducadl4gb%2F999.png&hash=a3404b7ba6fff806860415defe3d669c) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fi3h0fqkkb%2F574.png&hash=645fb8d07d6a78589ecf6ec985bee2c3)
And here where I have shifted the signal with B/W offset.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fo4epcup6z%2F15924.png&hash=f9c1fa5cc49dfc5fc4ee6db921767f90) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fpw7o7rg9n%2F16132.png&hash=3e355c4ce11038ad0ecf3e0e9ac4aa00) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Ffm598iy3v%2F16349.png&hash=7e080a1fc574c68b929480bdd616c92e)
There are sure to be other points for both controls where the histogram retains it's bell curve.
Note how stdev remains within shot to shot variation for all examples.
With Saturate offset at the minimum value (118) that still produces a bell curve (black level 574), I can push B/W offset to produce black levels 999 and 2048, matching the levels where saturate offset produced a bell curve with fixed white point, even with white point pushed way past 16383.
CMOS/ADTG gains are clearly analog, they effect noise.
Digital gain, B/W offset and Saturate offset seem to be digital level controls. Probably only subject to quantisation errors. They also clearly have defined points where they work well together (produce bell curve), and Digital Gain seems to only have 1 sweet spot (subject to contrast).
Suggested settings.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fqyhuqbm8b%2FSettings.png&hash=e2f23748d26d7fb164e748151dd8182c) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F3k9vedwl7%2FDefault.png&hash=db5529c7d40fce5be67d04bd0718ce56)
These produce maximum contrast and Dynamic range, with white level recorded at nearly 14bits. White level does not change with ISO. At higher white levels, high ISOs will push the WL even further.
Left to confirm.
What programs, if any, respect high white levels. They should, what's the point in having 14 bits when you don't use them all :o
ADTG gain needs to be adjusted on a per ISO basis, find correct values.
-
Adobe respects the white level.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fidjxl7k1n%2FAdobe.png&hash=4866507cd78ceae87bdb82ce9b8fa295)
Also, to high ADTG gain value.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fw8i83wbyj%2Fhigh_gain.png&hash=e83df0623e630de5be12ce7893728534)
And to low ADTG gain value.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fofrkbxt4r%2Flow_gain.png&hash=c0bd40fec63d27285b61c218a63c2183)
Note the change in histogram.
-
5D2 mistery solved!!!
Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.
(...)
Canon ISO ML ISO Improvement
100 82 0.29 EV
200 165 0.27 EV
400 332 0.27 EV
800 692 0.21 EV
1600 1432 0.16 EV
3200 1432 1.16 EV
(...)
Well done! :D
By seeing this, with an optimized pedestal, max WL and a good FPN reduction I think we can reach 12 stops, maybe 12.5 stops DR on the 5D2 for daylight (short exposure) photography.
-
(moved to next page because forum can't handle so many graphs)
-
Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.
The green channels are solid white at 16383; the red channels are at 16065, and the blue channels are at 16001. ... What needs more investigation is the "rise" before each channel reaches its max with the tweaked settings.
I take that back. It looks like the overexposed regions are not solid white, and the pixel-to-pixel differences in saturation is what is producing that "rise" in the histogram. Here are plots of where the pixels have values shy of the overall white level (in one of the green channels), for the default settings (13200..13400) and the tweaked settings (16100..16350). I'd expect to see just the border of the overexposed region.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fr4ij8shlyq38gh5%2Fpre-saturation_0279.png&hash=570503f6cdfc805da4be57dc0b80847a)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fp59qhwveft4ojpi%2Fpre-saturation_0280.png&hash=cf4af0fab77dae938b8896fb87307252)
There does not appear to be any noticeable banding in the per-column black level when looking at the top black bar.
-
I choose the sweet spot as the minimum gain where a fairly large solid area is really solid white and stays at the original white level (except for isolated bad pixels). I don't go below this level (basically a binary search that looks at the output of this function and chooses the lowest gain that still results in the same white level as Canon):
/* calibration */
static int raw_center_white()
{
int xc = (raw_info.active_area.x1 + raw_info.active_area.x2) / 2;
int yc = (raw_info.active_area.y1 + raw_info.active_area.y2) / 2;
int w = lv ? 100 : 500;
int h = lv ? 100 : 500;
int min = INT_MAX;
for (int x = xc-w; x < xc+w; x++)
{
for (int y = yc-h; y < yc+h; y++)
{
/* a little bad pixel filter */
int p1 = raw_get_pixel(x, y);
int p2 = raw_get_pixel(x+1, y);
int p3 = raw_get_pixel(x+1, y+1);
int p4 = raw_get_pixel(x+1, y+1);
int p = MAX(MAX(p1,p2), MAX(p3,p4));
if (p > raw_info.black_level)
{
min = MIN(min, p);
}
}
}
bmp_draw_rect(COLOR_BLACK, RAW2BM_X(xc-w), RAW2BM_Y(yc-h), RAW2BM_DX(2*w), RAW2BM_DX(2*h));
return min;
}
-
This is amazing!
-
Some new graphs:
5D3 and 5D2: I've measured them myself with latest raw_diag. Camera pointed to light bulb, left edge dark to avoid surprises, manual lens to exclude aperture trickery, shutter 1/30. I have simply read the DR number, with and without ADTG trick tuned to sweet-spot (didn't write down noise & other numbers).
Notice that ADTG trick reduces the actual ISO (by definition, it captures more highlights), so it makes sense to plot these new ISOs at the correct X position. This means there's no 0.5 stops improvement, but only 0.35 (what you get at ISO 100).
For 5D3, I've included 160 ISO multiples too. Notice these are not really ISO 160 according to clipping point, so I've estimated the real ones with iso_regs.mo (you can check the formula in the source code).
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3.png&hash=928fe7eca0fabd74f31a5951ddac0db5) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F6d.png&hash=20433025b67889809e94918894ee99f9)
For the others, I took the DR numbers from DxO (screen) and the calibration values from early testers. Take them with a grain of salt (I did not measure anything here).
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F550d.png&hash=7487faad5934d7bb015538d59d927d98) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F60d.png&hash=ad767900096d8df02901b3adc2580da0)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F650d.png&hash=2e03617eb430f6efd82877f56672eaa6) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F700d.png&hash=1a3dec5b5fbfa97dc4e9e60b9f63ada2)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d2.png&hash=f2b14c70fa5906ebeb268b9c987f6480) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F50d.png&hash=23541e20a3eaccb550ddb672d24e3100)
The script I used for these graphs, including test data (run it with Octave, might work in Matlab too): dr.m (http://a1ex.magiclantern.fm/bleeding-edge/iso50/dr.m). You are welcome to grab raw_diag and confirm (or infirm) the measurements.
Note the DR in these graphs is per-pixel (so they can't be compared directly if the cameras have different resolutions). If anyone would like me to normalize them (at say 8 megapixels or whatever), let me know.
I don't have calibration data for 600D (need a volunteer). Also I am 99% sure it will work on 1100D and EOS M (need volunteers who can read hexdumps). For 50D and 500D, I will have to take a closer look. 7D might require dual DIGIC trickery (didn't look at it yet).
-
Notice that ADTG trick reduces the actual ISO (by definition, it captures more highlights), so it makes sense to plot these new ISOs at the correct X position. This means there's no 0.5 stops improvement, but only 0.35 (what you get at ISO 100).
Thanks for the clarification, actually I was already wondering about that, but felt too much intellectually challenged looking at all the data to ask :-o ... do you have a table with numbers from the grap with the actual dynamic range gained for each full stop iso?
-
Yes, there's a link to the script right below the graphs.
-
Yes, there's a link to the script right below the graphs.
Thanks, I didn't realize I can just open this with notepad and copy/past the data.
-
So, as far as iso is concerned, why did I buy a 5d3 over 5d2? ;)
-
So, as far as iso is concerned, why did I buy a 5d3 over 5d2? ;)
The 5d3 (and more so: 6d) have less banding than 5d2, and this is what kills your dynamic range after lifting shadows much faster than just random noise. But of course as all reviews stated, the 5d3 does just about everything better than the 5d2 - but the sensor generation is the same. If you want high dr high res, get a d800.
-
I know I know but its funny that the charts are showing better DR results with the 5d2. Believe me aside from native DR, I love my 5d3 + ML. I was about to sell and get a D800 but then ML saved me! 5D3 +ML is amazing. With these new ISOs, Im soooo excited.
So as this module evolves, are we going to need to input these settings and fine tune it for our specific cam or will it be more basic like "awesome ML ISO ON or OFF" ha ha either way Im happy!
-
disregard
-
That's why I'm very excited that now there's every possibility that ML might conjure up 1/2ev of dynamic range out of thin air ... makes you really, really wonder if there's a hidden catch and why Canon doesn't include this, maybe they don't want to confuse users with too much iso wizardry?
This is great stuff ... 0.3 to 0.5 DR improvement is nothing to sneeze at. This thread reminds me of the classic book from the early 80's, "Soul of a New Machine", except on Internet time at 3x to 10x to 100x faster pace.
Much of this thread is over my head, but I do have a concern that parameters are being optimally tweaked to specific cameras ... A1ex's 5d3 and AudioNut's 6d. Especially with analog, what works on AudioNut's 6d might be sub-optimal compared to the default on my 6d.
Or will the end result at release be tunable to a specific camera, like Auto-Dot-Tune? It might be like AudioNut using Auto-Dot-Tune with his 6d+50mm f1.8 and indicating the appropriate value was -7. Well, yes, on his camera, but unlikely to be optimal on my camera. And maybe even worse than the default of zero.
Or does that not apply?
FWIW: I think in an earlier post on this thread, someone mentioned that an incorrect value for some ATDG register (don't recall the specific) might mess up A-ETTR. Am I the only one concerned about such an issue?
-
At first glance Dual-ISO looks softer but there is a focus difference between the shots. This can be seen by examining the "MA" letters near the top left.
I am also impressed by the image of the outdoor porch light. However, the left tube seems to be missing some of the yellow crud at the bottom. Or is that accounted for by the focus difference? The right side yellow crud seems mostly intact.
And would there be a difference between
* ATDG trick + Dual-ISO and
* "just" Dual-ISO?
That would seem to be a fourth combination.
Or do I have a flawed understanding of what is going on?
-
So as this module evolves, are we going to need to input these settings and fine tune it for our specific cam or will it be more basic like "awesome ML ISO ON or OF" ha ha either way Im happy!
Right now you enter the low-level settings from iso_regs or adtg_gui, but I'm working on a user-friendly frontend.
Or will the end result at release be tunable to a specific camera, like Auto-Dot-Tune? It might be like AudioNut using Auto-Dot-Tune with his 6d+50mm f1.8 and indicating the appropriate value was -7. Well, yes, on his camera, but unlikely to be optimal on my camera.
It seems there's a small variation between cameras of the same model (I believe it's within +/- 0.1 stops). I'll include a calibration routines (so default settings would be fairly good - just turn it on - but you'll be able to find the sweet spot for your camera if you wish, or dial a custom ISO).
Also, if you shoot JPEG, you may be able to dial the ISO even lower, because with most picture styles, the white point seems to be a little lower than white level in raw.
-
5D2 mistery solved!!!
Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.
For the 50D:
SEND_DATA_TO_DFE_FUNC = 0xffa71598;
I confirmed that each of the four 1d0x registers changes the gain of a fourth of the columns, but I haven't yet done anything more (e.g., generating the analogue of your 5D2 graph).
-
Nice. On 50D, from your older results, I believe you can recover all the highlights only via Saturate Offset (and you might even benefit from *increasing* the DFE gain, though it's probably not noticeable).
However, I remember reading 50D has its ISO 100 somehow pulled from 200 (not sure exactly how), so things may be different at ISO 200 and above.
Can you tell me some default values for these DFE registers and send me a RAM dump via PM? (Debug -> Dump ROM and RAM -> RAM4.bin). You may also look them up yourself in the hexdump and only show me the relevant section (e.g. if register 1d02 is 0x456 and you search for 0x1d020456, you will find something that looks like an array of values for these registers; it should be fairly obvious where this array starts and stops).
-
However, I remember reading 50D has its ISO 100 somehow pulled from 200 (not sure exactly how), so things may be different at ISO 200 and above.
http://www.magiclantern.fm/forum/index.php?topic=7102.0
-
@ayshih, can you write down a table with register values at all the ISOs? (photo mode, of course)
I'm interested especially in CMOS[0] (CMOS gain), C0F08030 (digital gain aka SHAD_GAIN), 0xC0F08034 (BW offset aka SHAD_PRESETUP), C0F0819C (Saturate Offset), and the DFE/ADTG gains. If you notice other relevant registers that are changed with ISO, you may include these too.
(actually I'm interested in a table with these values for all other cameras; simply write down the numbers from ADTG GUI)
-
Nice. On 50D, from your older results, I believe you can recover all the highlights only via Saturate Offset (and you might even benefit from *increasing* the DFE gain, though it's probably not noticeable).
However, I remember reading 50D has its ISO 100 somehow pulled from 200 (not sure exactly how), so things may be different at ISO 200 and above.
The low white level of 13432 is the case at only ISO 100. At higher ISOs, the white level is 15760, so just like with the 5D2, decreasing the DFE gain could be useful. (This does suggest that ISO 100 is handled differently in some way...)
Can you tell me some default values for these DFE registers and send me a RAM dump via PM? (Debug -> Dump ROM and RAM -> RAM4.bin). You may also look them up yourself in the hexdump and only show me the relevant section (e.g. if register 1d02 is 0x456 and you search for 0x1d020456, you will find something that looks like an array of values for these registers; it should be fairly obvious where this array starts and stops).
As an example, the default values for ISO 100 are:
1d02 = 0x419
1d04 = 0x41a
1d06 = 0x41e
1d08 = 0x41c
I'll send you the array by PM.
@ayshih, can you write down a table with register values at all the ISOs? (photo mode, of course)
Okay, I've already have some partial tables, but I might as well put together a comprehensive table tomorrow.
-
hmm it looks like the DFE is the fpga in 5D2..
-
hmm it looks like the DFE is the fpga in 5D2..
What would that imply?
Sorry if it comes off as being ignorant :P
-
trying to get adtg_gui to run on the 1100D, but I'm getting memory errors with CONFIG_GDB=y
underflow/overflow[31/0] malloc(0) at rbf_font.c:32, task ml_init
Allocated RAW 694kB, peak 716kB
malloc 157kB, 187kB used
AllocateMemory 946kB, 320 B used
shoot_malloc 30MB, 506kB used
stack_space 505kB
any ideas?
-
The binary got bigger than 512K (MemSiz in the compile log). You can disable some debug stuff from features.h.
(there is a check for this, but only on cache-hacked boot, which I'm not quite convinced it's the way to go)
-
Dunno what your total free is but you can try boosting the bin size to 640K or getting more free memory by putting the bvram mirror in another memory area. esp. just to get GDB working.
-
Okay I got GDB working, but adtg_gui fails to load, 'Module Init Failed' is all I can tell you, the console text is so small it's unreadable. I already tried the free_space / 4 thing you mentioned. Free memory is 272K + 894K
-
You need the stub addresses for ADTG/CMOS functions for 1100D (in adtg_gui_init). If you trust my automatic tools, they should be:
MakeName(0xFF2DBB28, "reg_Start_ADTG");
MakeName(0xFF2DBD1C, "reg_Start_CMOS");
For the dropbox version (which requires some more RAM, but also catches more registers), you may also want the engio functions (from stubs).
-
don't anybody kill me but is the latest nightly build the 15bit build or is that still a work in progress?
-
What does 15bit have to do with anything?
-
Added 50D graph, thanks to @ayshih for measurements. Looks kinda tricky, the only real improvement seems to be at ISO 100.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F50d.png&hash=23541e20a3eaccb550ddb672d24e3100)
From 5D2, I've combined the DFE gain with the previous tweak (SaturateOffset) and squeezed a tiny bit more DR (0.31 vs 0.29). Graph updated.
-
Here's the table of all the data I've collected for the 50D. The highlights:
- Most of the x160 ISOs are digital pulls from the next-higher ISO, so have the same dynamic range as that full ISO.
- Normal ISOs greater than 1600 are analog pushes using only DFE gains, and there is no reduction in read noise as measured in electrons (the read noise in ADUs scales directly with ISO). Thus, using ISOs greater than 1600 throws away highlights without any improvement in the shadows.
- The two expanded ISOs are digital pushes that multiply the ISO 3200 data by 2 and 4. The discreteness can be seen in the CR2 files.
Link to Google Docs (https://docs.google.com/spreadsheet/ccc?key=0Aqd_Zo2KQfZHdHBsVE5KZ3VsY3Vsd3FBVW1CeUtmSFE&usp=sharing)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fp38s6hra7ne2ilc%2F50D_ISO.png&hash=738be8485d9c7595c3936ef6d1a166d3)
-
Great work!
Can we look two images taking with Canon Iso and ML Iso?
-
http://www.magiclantern.fm/forum/index.php?topic=10111.msg96563#msg96563
-
Is it possible to make the camera use the actual iso values of mini_iso like ISO 77, or is Canon hardwired to the usual 1/3 ev stops?
The reason for the request is that in plain Av/Tv (and probably P) mode, flash users cannot easily benefit from the dr gain as a permanent +ec also modifies the ettl exposure - you also have to juggle with the flash ec, it's the same as with dual_iso but in the other direction.
Edit: In case you non-flash shooters don't know, the ettl flash ec isn't linear, at least on 60d/6d the effect from 0->1/3 is much stronger than the change from 1/3->2/3->... so to gain a "neutral" fill flash you really want to use 0 flash ec.
-
I can't imagine a clean solution, but I also don't understand why a +EC wouldn't do the trick.
-
I can't imagine a clean solution, but I also don't understand why a +EC wouldn't do the trick.
I suspect so as you don't do the flash thing. In Av/Tv with flash on, ec doesn't change the exposure but the *ratio* between flash and ambient illumination - the general exposure stays always "neutral" and the same as metered by the camera unless you change *both* camera and flash ec.
Catch as written above is that flash ec for some reason isn't (always) linear, +-1/3 often has a very large effect while further flash ec changes are more or less like you'd expect them to be. That's why changing flash and camera ec to 1/3 to compensate for the gained dynamic range isn't as easy as it sounds.
-
This is really amazing.
I work mostly on ads and editorial portraits. So I always use strobe light and I almost always end up setting up a strobe as a fill to take down the contrast a bit.
Really curious how these new iso's will compare to the fill light with the canon iso's
I know it isn't really the same, but instead of flattening my light and adding contrast in post I might be able to shoot more stuff the way I actually like it.
Reminds me of shooting on film. Slightly overexposing and under developing to get nice shadow detail.
I was also wondering about something.
The ISO 50 you can choose. Is it iso 100 on the sensor and then adjusted in the conversion to digital?
If so you probably also lose DR with ISO 50.
I didn't see any mention of it on the charts and they go down towards the higher ISO's put probably also lower towards lower Canon Iso's because the only good one is 100?
Curious because I often use 50 to get a wider aperture.
Thanks
-
ISO 50 and ISO 100 are identical.
-
I suspect so as you don't do the flash thing. In Av/Tv with flash on, ec doesn't change the exposure but the *ratio* between flash and ambient illumination - the general exposure stays always "neutral" and the same as metered by the camera unless you change *both* camera and flash ec.
Catch as written above is that flash ec for some reason isn't (always) linear, +-1/3 often has a very large effect while further flash ec changes are more or less like you'd expect them to be. That's why changing flash and camera ec to 1/3 to compensate for the gained dynamic range isn't as easy as it sounds.
I still don't get it, can you show this with some images? (include the raw files too)
-
MakeName(0xFF2DBB28, "reg_Start_ADTG");
MakeName(0xFF2DBD1C, "reg_Start_CMOS");
These appear to be correct.
ISO 100 200 400 800 1600 3200 6400
CMOS[0] 0x0 0x120 0x240 0x360 0x480 0x5a0 0x5a0
ADTG2[8882] 0x417 0x41c 0x421 0x430 0x427 0x428 0x850
ADTG2[8884] 0x424 0x428 0x426 0x437 0x429 0x42a 0x854
ADTG2[8886] 0x425 0x429 0x428 0x438 0x42b 0x42d 0x85a
ADTG2[8888] 0x419 0x41b 0x421 0x430 0x426 0x428 0x850
ADTG2[8839] 0x1 0x1 0x2 0x2 0x3 0x3 0x3
ADTG2[88bc] 0x0 0x0 0x0 0x0 0x0 0x0 0xff
ADTG2[88be] 0x1e 0x1e 0x1e 0x1e 0x1e 0x1e 0x708
ADTG2[88c0] 0x8 0x8 0x8 0x8 0x8 0x8 0x180
ADTG2[88eb] 0x850 0x850 0x850 0x850 0x850 0x850 0x900
Interestingly enough, ISO3200 appears to be analog on the 1100D. There are no intermediate ISOs. This was done at f/1.8, I'll compare that to a manual lens shortly...
edit: well thats dumb, the forum seems to be unaligning my columns, they're all lined up in preview mode
-
There are no intermediate ISOs.
Not even from ML menu?
About the dump, you need to lookup 0x88820417 in the RAM dump (or just send it to me).
-
Yes ML can, I meant for Canon, I was just testing the actual Canon ISOs.
-
On 550D they were simply hidden from menu.
For 1100D I'm very curious about DR measurements, mostly because of your astro pics. I guess you use the 1100D for a reason.
-
For 1100D I'm very curious about DR measurements, mostly because of your astro pics. I guess you use the 1100D for a reason.
Yeah, I'm working on that, I have to do it basically manually, 1100D doesn't have raw stuff working so raw_diag won't load.
It does have larger pixel pitch due to the smaller pixel count. That's also probably why ISO3200 is analog.
-
Photo raw should have no trouble in running on 1100D (though somebody needs to find the hooks; most likely similar to 600D). LiveView is troublesome, I know, but it's not needed for these tests.
-
B/W offset. Gain, this one shifts the entire signal. Tied to Saturate offset.
Saturate offset is black level control. Does not effect white. Tied to B/W offset.
Digital gain, B/W offset and Saturate offset seem to be digital level controls. Probably only subject to quantisation errors. They also clearly have defined points where they work well together (produce bell curve), and Digital Gain seems to only have 1 sweet spot (subject to contrast).
Saturate offset is not adjusting the black level per se, it's only adjusting the offset*.
B/W offset isn't any form of gain, it's digitally shifting the analog signal into the recorded bit depth.
When they produce a negative effect on histogram, simply they have not been optimised for the incoming analog signal, and the round off errors have become large.
Digital gain would seem to be the gain of the digital stage. Judging by the effect on the histogram, this would appear to be gaining the digital signal. ie: 1 in, 1.5 out. Rather then using gain to boost the digital signal, it's preferable IMO to use B/W offset to map the bit depth of the recorded signal.
I can gain an extra hundredth or two of DR, by not adjusting the black level offset lower (as I was in my earlier tests), and leaving more room for ADTG gain reduction. Such a shame that CMOS gain cannot be reduced further, Canon is probably (over)saturating that gain also.
Does this mean Canon did not fully optimize their sensor for low noise?
I'd say they simply left a safety margin in their code to make sure the ADC is always saturated (that is, to make sure white is always recorded as white).
*The offset is so far down into the noise floor that it does not effect black in the recorded image. The offset can be adjusted higher then the default value, but there doesn't appear to be any gain to DR.
http://www.cool.id.au/astronomy/Neb_tut/Scientifically%20determining%20CCD%20Gain%20and%20Offset.pdf
-
Such a shame that CMOS gain cannot be reduced further, Canon is probably (over)saturating that gain also.
ADTG2[8839] 0x1 0x1 0x2 0x2 0x3 0x3 0x3
ADTG2[88bc] 0x0 0x0 0x0 0x0 0x0 0x0 0xff
ADTG2[88be] 0x1e 0x1e 0x1e 0x1e 0x1e 0x1e 0x708
ADTG2[88c0] 0x8 0x8 0x8 0x8 0x8 0x8 0x180
ADTG2[88eb] 0x850 0x850 0x850 0x850 0x850 0x850 0x900
I smell something here, though I can't tell yet exactly what. I did get an extremely low ISO (something like ISO 1) on 5D2, but very noisy.
-
I smell something here, though I can't tell yet exactly what. I did get an extremely low ISO (something like ISO 1) on 5D2, but very noisy.
Really low iso, like 1, how was the dynamic range, normal like 11 stops ?
This way there's less need for ND filters...
-
More like 1-2 stops.
-
More like 1-2 stops.
1 -2 stops :'( , can't see something useful for me in that, maybe for people creating modern art...
-
Based on my calculations for the 1100D at ISO100, -1 EV of the ADTG gains resulted in DR increase from 10.7 -> 10.9, std dev went from 8.1 to 4.4
I'll do more analysis after work...
-
5D2:
CMOS[4] = 0x242 at iso 100...800 and 0x244 at iso 1600. Setting it to 0x[46ce]4[24] (e.g. 0x442 at low ISO or 0x444 at high ISO) seems to squeeze 0.2 more stops of DR. Not yet sure if in highlights or shadows (I believe it's in shadows, since clipping point didn't change); feel free to pixel-peep and tell me.
5D3:
CMOS[3] = 0x944 at lower ISO and 0x144 at higher ISO. Doesn't seem to make a difference.
CMOS[4] = 0x718. Set it to 0x300...0x37f to squeeze 0.1 stops of DR. Setting he 0x80 bit results in white image. Other values like 0x100, 0x200, 0x400 result in different noise/DR levels.
CMOS[6] and [7] seem to do something about white level clipping and darkening the image. No clear conclusion yet.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d2-cmos.png&hash=3cb29a07063b19c300da62566389f85b) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-cmos.png&hash=f0131ad5729d99c9b02da7cc5f7793de)
-
I haven't read this entire thread, so I don't know if this has been discussed: but the raw converter should be updated to be able to handle the new type of saturation. I asked the libraw author what needs to be done and this was his answer:
http://www.dpreview.com/forums/post/52994258
-
Short answer: current raw converters will work unmodified.
However, for best results, I believe raw converters should use an autodetected white level, instead of assumming a fixed value. This is especially important *without* this tweak.
With plain Canon firmware, white level varies wildly with intermediate ISOs and also with aperture. With the tweak, I'm disabling these variations.
There are also white level variations at long exposures. I can't do much about these.
There are no nonlinear issues in Canon sensors as far as I could tell. When it clips, it clips abruptly. There are, however, bad pixels with values above the clipping point, and one should be careful about this when autodetecting the white level.
I'm planning to submit a patch to dcraw and ufraw, with white level autodetection. Dcraw assummes fixed values around 13000...15600, depending on the camera model, which is plain wrong (sorry). I believe Dave simply took the white level from one of the test shots he received (because I could get his exact white level by adjusting the aperture). I believe this is the cause of most (if not all) pink highlight problems with dcraw-based converters.
Things might be different with Dual ISO, but that's another story (that's something I'm handling in the converter).
-
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fki4amiyln%2Fcanon.png&hash=45cbbdf770aa7cc2f026cc0d5d79938f) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fuffbfldx7%2Fimage.png&hash=8fc9502988629d3708413c8730f84260)
I had a screenshot with DR @ 11.494 with a little less ADTG gain, but I lost it somewhere. Still, this one is 0.13EV better then my previous best (so close to 11.5).
I should be in bed, not tweaking registers :o
I'll post some more information tomorrow.
-
With a little less ADTG gain than calibrated value, the DR reported by raw_diag may be false (overestimated), because you'll get variations in clipping point.
If in doubt, recalibrate the gain. This will make sure everything clips at the same point.
-
Sorry, it's late.
A little less negative gain. So a higher register value. :)
The histogram wasn't as clean, but DR was higher.
-
I believe DR is also influenced by roundoff errors, so I'm not sure what to say yet.
Maybe using a more robust indicator instead of stdev (e.g. MAD (http://en.wikipedia.org/wiki/Median_absolute_deviation)) makes more sense?
-
There are no nonlinear issues in Canon sensors as far as I could tell. When it clips, it clips abruptly.
This seems unlikely to me, electronics are by nature non-linear, esp. near saturation. What was your methodology?
http://photo.stackexchange.com/questions/33984/how-linear-are-dslr-sensors/33985#33985
However when a photosite is nearing saturation various electrical effects reduce the charge generated by incident photons leading to a roll-off of the response curve (i.e. it becomes shallower).
Rodger Clark describes a methodology for determining the linearity of the sensor reponse (among other things) in this article:
http://www.clarkvision.com/articles/evaluation-1d2/
-
What was your methodology?
cr2hdr --iso-curve
However when a photosite is nearing saturation various electrical effects reduce the charge generated by incident photons leading to a roll-off of the response curve (i.e. it becomes shallower).
I wish it was like that, but could not find any sign of it. A graph from last night, with ADTG gain lowered much under the sweet spot:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fnonlinear.png&hash=8df6f23bfb73ce451a6a13700c0a12cc)
(I could not find any correlation between these variations and the actual signal data, so I believe they are just noise)
Also, I could not see any nonlinearity on the graph from Roger Clark. So, for practical purposes, the data is linear and clips abruptly.
-
Short answer: current raw converters will work unmodified. However, for best results, I believe raw converters should use an autodetected white level, instead of assumming a fixed value. This is especially important *without* this tweak.
Is this also true for ACR, or only for the oss ufraw/dcraw, i.e. when using Adobe apps, will using mini_iso result in artifacts or other problems? From what I've tested for myself everything looks fine, but then again I didn't know what problems to look for.
-
Saturate offset is not adjusting the black level per se, it's only adjusting the offset*.
B/W offset isn't any form of gain, it's digitally shifting the analog signal into the recorded bit depth.
...
*The offset is so far down into the noise floor that it does not effect black in the recorded image. The offset can be adjusted higher then the default value, but there doesn't appear to be any gain to DR.
It's a bit of an aside, but I object to your characterizing SaturateOffset as only an "offset", and the implication that changing SaturateOffset is not useful for improving DR. However, perhaps I am reading too much into your choice of words. SaturateOffset changes the digitized value of the black level without changing the digitized value of the white level, which has DR consequences.
In my mental picture of the electronics, there are two relevant DACs:
- The first DAC produces the reference voltage for the black-level clamp, and this clamping occurs before the ADTG/DFE amplification stage. The corresponding register is SaturateOffset (0xc0f0819c).
- The second DAC produces an analog offset that is added to the signal after both amplification stages, before it is fed into the ADC. The corresponding register is B/W offset (0xC0F08034).
Thus, changing B/W offset just shifts the digitized values because it just shifts the signal up or down in analog space. There is no change in the dynamic range because the black level and white level move together and the read noise stays the same.
On the other hand, SaturateOffset is more than just an offset because it comes before at least one stage of amplification. At the output of the amplifier, it becomes directly related to the separation between the black level and white level (which is the saturation at the rail of the amplifier). Increasing SaturateOffset decreases this separation, meaning that saturation occurs at a lower analog signal, so your DR must decrease. Conversely, decreasing SaturateOffset must increase your DR. (The changes in DR may be negligible if the register change is small).
For example, at ISO 100 on the 50D, you can achieve most of the DR improvement by just changing these two registers, without touching any amplifier gains. (While changing B/W offset itself doesn't change DR, increasing it provides more room to decrease SaturateOffset.)
-
Is this also true for ACR, or only for the oss ufraw/dcraw, i.e. when using Adobe apps, will using mini_iso result in artifacts or other problems? From what I've tested for myself everything looks fine, but then again I didn't know what problems to look for.
The only problems that may appear are either pink highlights or too many clipped highlights. Adobe DNG Converter seems to use a white level fixed to 15000, but it's important how the data above this level is actually handled (wheter it's used for highlight recovery or not).
You can check with "exiftool IMG_1234.dng -WhiteLevel=3000"; if you can recover the highlights, all is fine. With ufraw you can't.
-
a1ex:
Have you looked at saturation vs which of the parallel read channels is being used? I am wondering if the 2 apparent clipping points in your plot are from different read channels.
cr2hdr --iso-curve
I wish it was like that, but could not find any sign of it. A graph from last night, with ADTG gain lowered much under the sweet spot:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fnonlinear.png&hash=8df6f23bfb73ce451a6a13700c0a12cc)
(I could not find any correlation between these variations and the actual signal data, so I believe they are just noise)
Also, I could not see any nonlinearity on the graph from Roger Clark. So, for practical purposes, the data is linear and clips abruptly.
-
Yes, they are. Each column has its own clipping point (but there are two groups, indeed). The pattern repeats roughly every 4 columns, but there are clipping point variations on top of that.
I've removed the constant offsets and looked in the clipped area, but could not find any image details. I will do a controlled experiment and post a detailed analysis if there are people interested; this one was just from troubleshooting a dual ISO shot from 6D.
But this is not what you are getting normally in a CR2 file. In a CR2, things are simply clipped sharply (at the level where the mouse cursor is in my graph). This graph could be relevant for recovering even more DR with some clever algorithm, but from my early tests it seems quite unlikely.
-
Have you done any studies to see how stable the clipping point is? I am wondering if the fixed level chosen for .CR2 files might be too high for some circumstances. Some major things that come to mind that might change the clipping point are: battery voltage, temperature, and camera to camera variation.
-
At short exposures I get the same white level every day (checked with raw_diag, with these tweaks enabled and ADTG gain at sweet spot, not below). On 5D2 I've configured it at 16380, but I can choose any other value.
At long exposures, white level may become lower, but this also happens with Canon firmware.
-
Also, I could not see any nonlinearity on the graph from Roger Clark. So, for practical purposes, the data is linear and clips abruptly.
Right, but he was not using your ADTG trick, nor the same camera or even the same generation of camera. I simply posted that article as a demonstration of methodology, the idea being that you or someone else carry out that same methodology with the ADTG trick to determine if we are reaching into the non-linear part of the sensor response.
The basic idea is that you take pairs of exposures and subtract them, this residual is simply the square of the noise of one of the exposures, since photon noise is simply the sqrt of the number of photons, we can determine the true reponse in photons based on the std dev of the residual frame.
-
The basic idea is that you take pairs of exposures and subtract them, this residual is simply the square of the noise of one of the exposures, since photon noise is simply the sqrt of the number of photons, we can determine the true reponse in photons based on the std dev of the residual frame.
I'm actually using this trick to get an estimation of the noise from bracketed shots on a sturdy tripod (download the samples from engardeknave from the CeroNoice thread), and I found the noise estimated in this way to be very good regarding Gaussian shape and lack of autocorrelation.
-
You can check with "exiftool IMG_1234.dng -WhiteLevel=3000"; if you can recover the highlights, all is fine. With ufraw you can't.
Tried this briefly with one DNG image file in Adobe Lightroom 5 and, indeed, the highlights were lost. After setting the white level to 3000 in the EXIF metadata, one cannot recover the lost highlights using the conventional Exposure, Highlights, White or Tone Curve adjustments.
In fact, my attempts to restore the original white level back by executing: exiftool IMG_1234.dng -WhiteLevel=15000
on the command line did not succeed completely even after reading the metadata back from the file. Changing DNG camera profile from Adobe Standard to Camera Standard on the Camera Calibration tab fixed the issue temporarily but after returning from the Library module back to Develop module, the highlights seemed to be lost again. I should better comprehend myself with the Lightroom's metadata management since I don't quite understand how reverting the white level setting (at EXIF level) should be done in Lightroom. Anyway, it seems to me that the most recent release of this software does not resolve the white level from the actual image data.
-
I should better comprehend myself with the Lightroom's metadata management since I don't quite understand how reverting the white level setting (at EXIF level) should be done in Lightroom. Anyway, it seems to me that the most recent release of this software does not resolve the white level from the actual image data.
Thanks for trying this - so if I understand correctly, acr (or at least lr) cannot harvest the dynamic range gained from using mini_iso?
Btw I wouldn't have expected that simply changing one exif tag is enough for acr, I guess they're using a bunch of maker notes and dng tags for the technical stuff and the exif data is only there for a limited subset of image property descriptions.
Last not least, if saving/restoring metadata doesn't do it, maybe it's worth a try to remove the dng from the catalog and then re-add it, maybe not all "internal" tags Adobe doesn't expect to change are updated when reading an exif-modified dng (just a guess).
-
I'm asking you to check that, so I know what level I should choose in mini_iso (and whether it makes sense to make it configurable).
@jkaura: I think this explains why I've got better highlight recovery (compared to ACR) with a really dumb algorithm: http://www.magiclantern.fm/forum/index.php?topic=9458.msg91590#msg91590
and from your findings, I believe that - on top of the DR graphs I'm plotting - you will also get back some more highlights at wide apertures, that might be lost because of digital amplification of the raw data. Check the first post for details.
-
I'm asking you to check that, so I know what level I should choose in mini_iso (and whether it makes sense to make it configurable).
Make what exactly configurable - the cr2 tags aren't modified by mini_iso, or are they?
-
The CR2 doesn't have any tag for white level, but I can change the range of recorded data (black and white level).
-
Last not least, if saving/restoring metadata doesn't do it, maybe it's worth a try to remove the dng from the catalog and then re-add it, maybe not all "internal" tags Adobe doesn't expect to change are updated when reading an exif-modified dng (just a guess).
Thanks for the tip, that worked for me! After removing the photo from the catalog and reimporting it, the white level seems to be again at a normal level.
-
Thanks for the tip, that worked for me! After removing the photo from the catalog and reimporting it, the white level seems to be again at a normal level.
Yippee, I'm contributing something useful again for a change :-p
The CR2 doesn't have any tag for white level, but I can change the range of recorded data (black and white level).
Meaning as in "keeping the increased dr, but just recording it with different 14bit (16bit?) values", or as in "dimming down the effect of mini_iso because acr currently cannot cope"?
Edit: In any case, an alpha mini_iso.mo with advanced configuration for this to debug probably is a good idea, esp. if the source code isn't available yet.
-
Remapping the recorded range has nothing to do with DR, they are decoupled.
... except maybe for some round-off errors :P
-
Remapping the recorded range has nothing to do with DR, they are decoupled.
Ok, thanks for explaining, in this case why not stuff the data in the original range that is used w/o mini_iso to avoid any compatibility problems ... or am I still mis-understanding what is going on? If I want to try to debug any regressions with acr, I should at least have a vague idea :-p
-
1) At larger apertures, Canon amplifies the raw data digitally (with the digital gain register), which works by changing the recorded range. That means, a raw processor that autodetects white level will not have any side effects (except for f1.4 and f1.2 where the white level overflows above 16383), but a raw processor that assummes a constant white level (most common ones I think) will effectively interpret this as a higher ISO (and throw away whatever gets above the "declared" or "assummed" white level). For Adobe DNG Converter (and probably other Adobe apps), this level seems to be 15000; for dcraw, it varies with camera model.
2) Right now I'm fixing the white level at the same level Canon uses with a manual lens (or at f8). However, if this level (camera-specific) is around 15600, and ACR clips at 15000, this may not be the best choice.
So, it looks like it's probably a good idea to choose the white level from menu at values used by common raw processors (15000, 15600, 16380 and so on).
-
So, it looks like it's probably a good idea to choose the white level from menu at values used by common raw processors (15000, 15600, 16380 and so on).
Yes, that does sound like a good idea - maybe the oss converters might accept patches to have a more flexible white level, but acr most likely won't ... but it would be nice to be able to use DxO for their prime noise reduction then and again with an acr-cr2 if that's possible, maybe they happen to have the same clipping point.
If you've got an acr-calibrated mini_iso.mo, I'd certainly like to test drive that :->
-
However, if this level (camera-specific) is around 15600, and ACR clips at 15000, this may not be the best choice.
Adobe respects the white level.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fidjxl7k1n%2FAdobe.png&hash=4866507cd78ceae87bdb82ce9b8fa295)
Note the maximum pixel value of the sample patch in RawDigger.
MAD does seem like a good idea. It sounds like it will help narrow down DR increases to the signal itself without the precision errors having an effect.
It's a bit of an aside, but I object to your characterizing SaturateOffset as only an "offset", and the implication that changing SaturateOffset is not useful for improving DR. However, perhaps I am reading too much into your choice of words.
It was not my intent to imply that it wasn't useful. However, I went on to describe that more gains can be made by not reducing saturate offset, and hence, leaving more room for ADTG gain reduction.
I can gain an extra hundredth or two of DR, by not adjusting the black level offset lower (as I was in my earlier tests), and leaving more room for ADTG gain reduction.
Lowering the offset does increase DR. However, it does not reduce noise (stdev) and the DR increase comes from the simple fact that DR = log2(saturation-offset/stdev).
SaturateOffset changes the digitized value of the black level without changing the digitized value of the white level..........................Increasing SaturateOffset decreases this separation, meaning that saturation occurs at a lower analog signal, so your DR must decrease. Conversely, decreasing SaturateOffset must increase your DR.
The key here IMHO is your description (and my observation) of it just being a digitized value. Did we really increase the DR of the signal (the photons that deliver our image)? I would guess not, we just digitally cheated. This black level that is being changed is so far down into the noise floor to be negligible. Perhaps it does shift the entire signal (black weighted) down and increases the DR through the signal. But IMO, by the time we move up to a signal level that is outside of the noise floor, the gains would be minimal at best.
With ADTG gain reduction, we do decrease the noise. So IMHO, it's much more preferable (in increasing DR) then saturate offset.
In my mental picture of the electronics, there are two relevant DACs:
- The first DAC produces the reference voltage for the black-level clamp, and this clamping occurs before the ADTG/DFE amplification stage. The corresponding register is SaturateOffset (0xc0f0819c).
- The second DAC produces an analog offset that is added to the signal after both amplification stages, before it is fed into the ADC. The corresponding register is B/W offset (0xC0F08034).
Thus, changing B/W offset just shifts the digitized values because it just shifts the signal up or down in analog space. There is no change in the dynamic range because the black level and white level move together and the read noise stays the same.
If this is true (and I am not saying otherwise), then it would appear that the analog reference voltage is only having an effect in the digital domain. Noise (stdev) does not increase or decrease. I think of it (reference voltage) as saying to the devices later in the chain, map black to this voltage. Also, it seems clear that the reference voltage has a fixed level of noise. The voltage per se hasn't changed the photons we are interested in, it's simply an analog instruction to a digital device.
B/W offset (which acts in the same manner) is more useful IMHO, because of bit depth. B/W offset doesn't increase or decrease the DR of the signal, simply that it pushes the signal into a higher recorded bitdepth. Here, the gain is bit precision for the entire signal.
Take it with a grain of salt, I don't have a deep understanding of the electronics involved. I have only got observational skills and common sense going for me ;)
-
Lowering the offset does increase DR. However, it does not reduce noise (stdev) and the DR increase comes from the simple fact that DR = log2(saturation-offset/stdev).
...
With ADTG gain reduction, we do decrease the noise. So IMHO, it's much more preferable (in increasing DR) then saturate offset.
If you take a look at my 50D numbers (http://www.magiclantern.fm/forum/index.php?topic=10111.msg98056#msg98056"), you can see the following cases starting at Canon's ISO 100:
- changing just DFE gains: 1023..13432, stdev = 4.79 => 11.34 DR
- changing primarily offsets: 1024..15759, stdev = 5.76 => 11.32 DR
(Note that the 0.02 DR difference is below the uncertainty of the DR calculation and the reproducibility of the mini_iso calibration.) It's true that changing the offsets doesn't decrease the noise as measured in ADUs, while the width in ADUs of the noise does decrease when reducing gain. However, in both cases, the SNR at typical brightness levels hasn't actually changed. Instead, what you get in both cases is increased range in the highlights. Getting better SNR in the shadows requires you to ETTR to make use of this increased range.
B/W offset (which acts in the same manner) is more useful IMHO, because of bit depth. B/W offset doesn't increase or decrease the DR of the signal, simply that it pushes the signal into a higher recorded bitdepth. Here, the gain is bit precision for the entire signal.
This is not correct: B/W offset has no effect on precision. If you shift the range of digitization from 1024..15000 to 2048..16024 by increasing the offset by 1024, you are using larger numbers, but the width of each bin doesn't change.
-
Getting better SNR in the shadows requires you to ETTR to make use of this increased range.
I make a point of ETTR in my tests for this reason. Of course, the increased SNR is through the entire exposure.
This is not correct: B/W offset has no effect on precision. If you shift the range of digitization from 1024..15000 to 2048..16024 by increasing the offset by 1024, you are using larger numbers, but the width of each bin doesn't change.
Indeed, thanks.
-
Now that we have raw_diag working properly, here's more data for the 1100D
Canon ADTG[888x] /= 2
ISO σ White DR σ White DR
100 6.17 13583 10.869 3.1 9714 11.273
200 6.54 15303 10.985 3.25 10956 11.421
400 7.55 15303 10.778 3.81 10998 11.196
800 10.4 15303 10.315 5.26 11033 10.737
1600 16.7 15303 9.632 8.64 11092 10.031
3200 29.7 15303 8.802 14.97 11111 9.242
6400 59.89 15303 7.79 15.55 11110 9.187
-
The white point looks a little low. Are you applying to much ADTG gain reduction?
With no changes to digital gain, I can keep reducing ADTG gain and increase DR. After tweaking the digital gain though, further reduction to ADTG gain (after the sweet spot) simply reduces WL.
-
Once you've decreased the gain enough that your white level is dropping, you've already gone too far. Note that even if the white level is unchanged as measured by raw_diag, it's still possible for the gain to be too low, because you may no longer have solid white highlights due to non-uniformity of saturation. See these images (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97761#msg97761) to see what can happen in the highlights (there, from changing digital offsets rather than gain, but it's functionally equivalent).
-
I have no idea what I'm doing ;)
You're right, I can raise the gain a little bit and keep the white level the same without really impacting the DR. The sweet spot seems to be around 0x330. Here's some new numbers.
Canon ADTG[888x] = 0x330
ISO σ White DR σ White DR
100 6.17 13583 10.869 4.65 13583 11.277
200 6.54 15303 10.985 4.92 15303 11.397
400 7.55 15303 10.778 5.76 15303 11.168
800 10.4 15303 10.315 7.83 15303 10.726
1600 16.7 15303 9.632 12.6 15303 10.038
3200 29.7 15303 8.802 22.11 15303 9.227
6400 59.89 15303 7.79 22.10 15303 9.228
-
Got some small updates to research tools:
raw_diag:
- JPEG curve analysis, see http://www.magiclantern.fm/forum/index.php?topic=10038.msg99049#msg99049
- cross-covariance analysis of FPN between two dark frames (it gives a hint about how much FPN you can fix from a dark frame)
- option to dump the raw buffer as DNG
adtg_gui:
- updated with 6D engio stubs
-
Trying to use the adtg_gui module on 6d.
But I got some linking errors with GDB.
I read in the first post the following:
"adtg_gui.mo (source: adtg_gui.c avl.c avl.h) - cross-platform, requires CONFIG_GDB=y:"
So it requires CONFIG_GDB=y, but what does that mean, what is that config_GDB ? Is it something I have to download or is it a setting in the ML menu ?
-
It's a safety check (this is how you tell it "hey, I know what I'm doing, now let me fry the sensor")
https://bitbucket.org/hudson/magic-lantern/src/tip/modules/adtg_gui/README.rst
-
Reading some stuff in the link you send, but I still don't get it.
Makefile.user ?
Does it mean I have to compile stuff myself ? (which I never did and is probably the reason why I don't understand how to set config_GDB = Y)
-
If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download ;D
-
So it requires CONFIG_GDB=y, but what does that mean, what is that config_GDB ? Is it something I have to download or is it a setting in the ML menu ?
You need an autoexec bin/symbols compiled with GDB=Y
-
If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download ;D
Unless you really have a good understanding on the effects and can help development, it's probably best just to wait until it has been polished and released.
Start learning how to compile (http://www.magiclantern.fm/forum/index.php?topic=9517.0). This way you will be up to speed on the basics with later development.
-
The link to the "how to compile" stuff is handy, sure will read into it (Lot's to read and learn, probably will take a month or more before I can compile :D).
But if I can, I will be one of the first people to use the next impressive magic lantern future!
-
If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download ;D
Changing the code & compiling isn't the only way to help, and in all honesty it needs quite a lot of time to get started, it's only easy in hindsight - but I can remember that setting up the whole thing and getting the bearings on the core code isn't self-explanatory.
At least imho as valuable are suggestions / requests on how to improve usability (menus, options, button layout, ...), helping others with ML questions here on the forum & bug/enhancement reports because even with stable code, a couple of people cannot test it enough to iron out every quirk.
The very same mini_iso.mo module about to be released in this thread needs testing, too - does it work with your postprocessing workflow, is the calibration the same on your camera & lenses...
-
6d TL users see here to help out with mini_iso - I'm trying not to spam the ML thread with TL (and that's what I'm currently still using): http://www.magiclantern.fm/forum/index.php?topic=10292
-
Got some funky graphs (5D3). These cover all the ISO registers I could find.
https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW
-
Updated adtg_gui: I've added a routine that dumps all the registers into a log file after taking a picture (this is what I've used to make the funky graphs). I'd like to get these graphs for all cameras, so I need your help:
0) check if adtg_gui.c has engio stubs for your camera (if not, copy them from platform stubs, compile it and post the diff)
1) enable adtg_gui in debug menu
2) enable digic registers
3) take a picture at every single ISO, including intermediates, starting to lowest (50 or 100) until the highest, and back. After each picture, make sure it saves a large number of registers (I expect around 1000 or 2000).
4) send me iso.log from the root of card (on 5D3 this log file had 4 MB, so if you get only 10K or something, double-check the settings and the digic register stubs)
In addition, I'd also like a second data set at some fixed ISO, but with variable aperture.
0) rename or move iso.log from the previous experiment in order to start fresh
1) enable the same stuff as before
2) take pictures at iso 200, with variable aperture from f8 to f1.2 or whatever you have, and back
3) rename this iso.log to av.log and upload it
Optionally, you can do the same with shutter speed.
Troubleshooting: if you get overflow errors after loading adtg_gui, add a small quantity to extra_ram in tcc/tccrun.c (I added 1024 on 60D). If you get ERR70, change max_region heuristic to free_space / 4 in mem.c.
On 5D3, adtg_gui now now works on both 5D3 113 and 123. Also updated iso_regs but didn't try it.
edit: please repeat the tests by changing the tested parameter in both directions.
-
WOW that module is BIG
Here is the first iso.log.. I went through all the ISOs in the menu, I think its the only module I can load. Even with bvram mirror in shoot mem.
6D:
http://ge.tt/6e5CtgI1/v/0?c
I have a chipped ring that lets you change the aperture from 1.2 to whatever... would that be better than a real electronic lens?
-
5D3.123
f/1.4.....f/16
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Aperture.log
1/8000s.......30seconds in full stops.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Shutter.log
-
Moved all the graphs here: https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW
-
http://optipng.sourceforge.net/
This is supposed to be better, but single threaded and slower: https://en.wikipedia.org/wiki/PNGOUT
-
Since there is a large difference between my logs and Audionut's logs, I'd like to ask you to repeat the tests by changing the tested parameter in both directions (incrementing and decrementing). This will help identifying which of these graphs are just noise and which of them are relevant.
The graphs now show all the data points from the log file, so if there are different values obtained in different experiments, that graphs may be just noise (or may be not).
-
Aperture reverse direction
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Aperture_reverse.log
Shutter reverse direction
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Shutter_reverse.log
Shutter 30s....1/8000s in full stops with lens untwisted
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/30-8000_lens_disconnect.log
Aperture in half stops
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Aperture_0.5_stop.log
-
Updated the graphs and dropped some irrelevant ones. Interesting that after merging the logs I've got much fewer results (will double-check to make sure it's not a bug).
Also moved all the graphs here: https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW
-
ISO up and down.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/iso.log
-
CMOS[1] 0x663 ISO 600
CMOS[1] 0xee3 ISO 1200
I couldn't find any other intermediate ISO steps. Noise level changes for the better compared to next higher Canon ISOs (800/1600), but needs more consistent workflow for accurate results.
On my test samples, I changed ISO in Canon menu for Canon ISOs, so these have been tainted by other CMOS/ADTG/Digic register changes. I'll attempt a more thorough workflow tomorrow for DR results.
I tested these CMOS[1] values.
3
113
223
333
443
553
663
773
883
993
aa3
bb3
cc3
dd3
ee3
ff3
1003
1113
1223
1333
1443
1553
1663
1773
1883
1993
1aa3
1bb3
1cc3
1dd3
1ee3
1ff3
2003
There is a defined pattern. Here are the average raw values for a patch on the color checker.
101.8
195.9
392.1
777.6
1555.8
2984.6
567.4
5598
194.3
377.8
750.5
1501.4
3006.7
5814.9
1095.1
11196.7
100.6
194.8
387.2
772.5
1539.4
2969.5
561.4
5684.2
192.4
373.9
745.4
1489.3
2992.3
5792.2
1087.8
11130.1
99.6
Notice how the equivalent ISO steps have slightly lower raw values.
For instance, ISO 100 gives raw value 101.8 and other register values produce 100.6 and 99.6.
I can't test noise differences since I messed the workflow up. I'll have another attempt tomorrow.
If you want to test yourself, the money registers for equivalent ISO 100
0x3
1003
2003
3003
4003....etc, etc.
It's the same pattern for other ISOs.
663
1663
2663......etc, etc.
-
I believe 0x773 has a little less noise than 0xDD3 (need to double-check). At first sight, this combined with CMOS[3] = 0x944 seems to give a 0.25 stop improvement in shadows at ISO 4370 (derived from 6400).
BTW, all research tools are now working on both 5D3 113 and 123.
-
0x223 - ISO 400
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Ff45if33rf%2F0x223.png&hash=9d715b7610e5fb75b436208e3705c631)
0x663 - ISO 600
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fupmtz1v57%2F0x663.png&hash=9d2fba336acb92453bb5c9813f1109e9)
0x333 - ISO 800
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fsl2gxz18b%2F0x333.png&hash=a723bc97cf2a3bca8ccb6c7d68dc1cb1)
0xee3 - ISO 1200
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fgx8f2w1nf%2F0xee3.png&hash=900abfe2bd32cbfc9271b23b36925d89)
0x443 - ISO 1600
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Flulzoiymz%2F0x443.png&hash=da7114bcd3485c0d7473e876371fc087)
Some other interests.
ISO 200 - 0x113 vs 0x883
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Feemq2pvi3%2F0x113.png&hash=9ee620d4ed4c6297131001208aba4609) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fkfkezu04r%2F0x883.png&hash=7726bc26b552f34ae9ff105e7ef36b33)
ISO 400 - 0x223 vs 0x993
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Ff45if33rf%2F0x223.png&hash=9d715b7610e5fb75b436208e3705c631) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fmww673rqz%2F0x993.png&hash=1ca0a0df4d5a33947259bc255c772de7)
This trend continues for ISO 800 and ISO 1600.
Only changing CMOS[1] with a starting point of Canon ISO 100, so not sure why the WL changes between 15283 and 15331.
I'd like to find ISOs 150 and 300.
I've made a dump here (https://www.dropbox.com/sh/u8bgqownig8s3og/D1A5E8P5vy). Up to 0xff3 for the moment.
-
For the 50D, here's the register dump for the ISO sweep, from 100 to 12800 and back down:
https://www.dropbox.com/s/7ftxovt2oqu2njn/50D-ISO.log
I'm not going to do the aperture sweep on my lenses because they're not particularly fast, but I can borrow a lens next week if someone else hasn't provided that register dump before then.
-
Uploaded 50D graphs.
Also modified iso_regs to handle all the CMOS gains.
Also updated the first post with links to these graphs.
-
Hypothesis about CMOS gains:
- each ISO is described by a number from 0 to 15. That's 4 bits, from b3 (MSB) to b0 (LSB).
- these 4 bits might be toggling 4 amplifiers on and off
- if this is true, the individual gains of these amplifiers would be approx. b0=1.92, b1=3.85, b2=15.28 and b3=1.91 (based on Audionut's estimation).
- all gains except 6, 7, 14 and 15 follow this logic
- gains 6 and 14 seem to be attenuated by 10.5 (the attenuator is triggered by logical function b2 & b1 & !b0 )
- gains 7 and 15 seem to be attenuated by 2 (the attenuator is triggered by logical function b2 & b1 & b0 )
CMOS gain Measured Derived
0 1.00 1.00
1 1.92 1.92
2 3.85 3.85
3 7.64 7.41
4 15.28 15.28
5 29.32 29.41
6 5.57 58.86 ?
7 54.99 113.28 ?
8 1.91 1.91
9 3.71 3.67
10 7.37 7.35
11 14.75 14.15
12 29.54 29.17
13 57.12 56.13
14 10.76 112.35 ?
15 109.99 216.21 ?
from pylab import *
gains = array([101.8,195.9,392.1,777.6,1555.8,2984.6,567.4,5598,194.3,377.8,750.5,1501.4,3006.7,5814.9,1095.1,11196.7]);
gains = gains / gains[0];
print "CMOS gain Measured Derived"
for g in range(16):
measured_gain = gains[g]
derived_gain = 1
for b in range(8):
if g & (1 << b):
derived_gain *= gains[1 << b];
status = " " if abs((derived_gain - measured_gain) / measured_gain) < 0.1 else "?"
print "%8d %8.2f %8.2f %s" % (g, measured_gain, derived_gain, status)
-
5D3: CMOS[6] = 0xa00 results in an extremely low gain (very dark image). It gets a little better if you increase the CMOS gain. It has more highlight detail than ISO 100 with ADTG gain at -2 EV, and I've estimated the clipping point at around ISO 67 (by comparing with another image underexposed with shutter).
It's too noisy for real use, but at least it shows that our CMOS amplifier at ISO 100 is running too hot for my taste.
-
It's too noisy for real use, but at least it shows that our CMOS amplifier at ISO 100 is running too hot for my taste.
Your last discoveries sound great, I'm confident you'll squeeze every last bit of dynamic range out of Canon ... and as usual I hope all cameras will benefit from it. I didn't stop wondering why Canon didn't harvest these possibilities, probably they rather concentrate on (non-raw) video and jpeg?
-
I forgot to mention that the raw values I listed above are black subtracted. Not sure how that effects your gain calculations.
I'll take a look at CMOS[6] and [7] today.
I didn't stop wondering why Canon didn't harvest these possibilities, probably they rather concentrate on (non-raw) video and jpeg?
At the very least, they could have had an almost real ISO 50.
-
CMOS[6] with saturated pixel wells
0x100 is the last register value to give White Level 15283 (0x1ff actually).
0x200-300 - WL 3100
0x400-500-600-700 - WL 2570
0x800 - WL 2081
0x900 - BL 15283 WL 15331 - Full white screen
0xa00-b00 - WL 2140
0xc00 and above WL 2100
This is a repeating pattern like CMOS[0]. At 0x1100 the pattern repeats.
While typing this, I thought I had better dig down further.
0x100 WL 15283
0x101 WL 8680
0x102 WL 12370
0x103-104 WL 15283
0x105 WL 10250
0x106 WL 12440
0x107 WL 12570
0x108-109-10a-10b WL 15284
0x10c WL 14610
0x10d-10e-10f WL 15283
From here, something is happening in the blacks. The histogram reminds me of high ISO.
0x110 WL 15283 - DR reduces 1 EV (vs no register change)
0x111 WL 8640
0x112 WL 12290
0x113-114 WL 15283 - DR reduced 1EV
0x115 WL 10200 - Histogram looks normal
0x116 WL 12410
0x117 WL 12540
0x118-119-11a-11b WL 15283 - DR reduced 1EV
0x11c WL 14520 BL 2046
0x11d-11e WL 15282 BL 2044
0x11f WL 15282 BL 2045
Now we are back to having normal looking histograms with a little more noise (deviation)
0x120 WL 15283 - DR reduced 0.2 EV
0x121 WL 8650
0x122 WL 12360
0x123-124 WL 15283 - DR reduced 0.2 EV
0x125 WL 10220
0x126 WL 12420
0x127 WL 12550
0x128-129-12a-12b WL 15283 - DR reduced 0.2 EV
0x12c WL 14540
0x12d-12e-12f WL 15283 BL 2047
First thing to notice, repeating patterns.
Same thing happens at 0x13? except here there is extreme deviation and the black level also changes wildly. I'm not testing each register as I have some respect for my shutter.
I'll test each 0x140 as the first few appear to mimick 0x10? (useful) with different WL's.
0x140 WL 15283
0x141 WL 10480
0x142 WL 13800
0x143 -114 WL 15283
0x145 WL 9912
0x146 WL 11140
0x147 WL 9420
0x148-149-14a-14b WL 15283
0x14c WL 12530
0x14d WL 14240
0x14e-14f WL 15283
0x15? - 0x16? - 0x17? - 0x18? - 0x19? - 0x1a? - 0x1b? - 0x1c? - 0x1d? - 0x1f? All appear to have the same pattern as 0x10?
A quick look at 0x2?? shows no sign of any difference. ie all the 0x2?? values match 0x200
There's some funky stuff happening at 0x9?? Have a look at 0x909 for instance.
All of these are like ADTG gain. Make sure you saturate the pixel wells!
edit1: Noise level doesn't reduce with reducing WL in the batches where I haven't specifically mentioned changes.
edit2: All of these tests where conducted at CMOS[0] 0x3. At CMOS[4] 0x2?? and above where the WL drops drastically, you can bring everything back with higher CMOS[0] values (which a1ex mentioned above). So for instance CMOS[0] 0x443 and CMOS[4] 0x200 brings back a normal image with WL 15283. Initial observation suggests that increasing CMOS[0] simply negates the effect of CMOS[4]. ie: No difference in produced image. Needs further testing.
edit3: CMOS[0] 0x10 through 0x18 produce WL 2060. 0x19 through x01f produce dual_iso pattern. Same thing happens at 0x20 through 0x2f and again at 0x30 through 0x3f, so I'll assume this continues through the other 0x?? values.
What interesting though, the ISO difference reported by cr2hdr seems to match the register normal ISOs. For instance, 0x1f = 0.96 EV ISO difference, 0x2f = 1.97 EV ISO difference etc. Including those half stop ISOs.
a1ex, can you create an option to disable iso.log creation? Also with the latest raw_diag the information overlay disappears after a short time. This appears to be happening when the iso.log info gets cleared.
edit4: The same pattern is repeating at CMOS[6] 0x? and 0x??
Have a look at CMOS[7] 0xc0
-
I have a chipped ring that lets you change the aperture from 1.2 to whatever... would that be better than a real electronic lens?
See if the digital gain register is changing with aperture. Can you set the aperture value lower then 1.2? That could be interesting.
-
Will do; until then, just comment out the prop handler and recompile.
Didn't digest all of the stuff yet, just some small remarks:
- you mean CMOS[0] instead of CMOS[1]
- the CMOS register values are 12-bit, so the fourth hex digit does nothing
- ADTG values are 16-bit and ENGIO (DIGIC) registers are 32-bit
-
Will do, thanks. edit: I might even be able to add the menu entry myself.
Yes I did.
The highlighted one 0x100?
-
I finally was able to borrow that faster lens. For the 50D, here's the register dump for the aperture sweep, from f/8 to f/1.4 and back:
https://www.dropbox.com/s/ihry3dq9drv2a05/50D-aperture.log
I can immediately see that the digital gain (0xC0F08030) is 0x1000 down to f/2.8, and then starts increasing, up to 0x147A at f/1.4.
-
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F50d-digital-gain-aperture.png&hash=83d3a9928c42960a461bd5f485f0f6a4)
So... 0.35 stops of digital gain? It matches the DxO value quite well.
-
Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534
(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)
Also, while digging in history, I've found a log from 5D2 that shows all the debug messages from capturing and processing a LiveView frame: http://pastebin.com/J21HuvLX
(you may now recognize the registers mentioned in this thread, and that's one small step towards understanding this process)
-
Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534
(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)
Also, while digging in history, I've found a log from 5D2 that shows all the debug messages from capturing and processing a LiveView frame: http://pastebin.com/J21HuvLX
(you may now recognize the registers mentioned in this thread, and that's one small step towards understanding this process)
I can do 550D, I am just unsure as to what "check if adtg_gui.c has engio stubs for your camera" means.
-
Open the source code for the modified adtg_gui (not the repo one, but the one from this thread) and look for "engio", you'll find them right away (some cameras have them, others don't).
-
The ones that don't, they just need to be found?
The ones missing the stub,
500D, 550D, EOSM, 600D, 650D, 700D
-
Open the source code for the modified adtg_gui (not the repo one, but the one from this thread) and look for "engio", you'll find them right away (some cameras have them, others don't).
Ah!
And maybe it's drop box or my internet, but I go to the link for "adtg_gui.c" and the page just hangs loading. So I might have to wait til tomorrow.
(Not an excuse, that actually happens. So I will do it tomorrow, as I would like to see this tstop issue dead.)
-
Try right click, save as.
-
The ones that don't, they just need to be found?
The ones missing the stub,
500D, 550D, EOSM, 600D, 650D, 700D
Well I'm on 550D. But it says:
check if adtg_gui.c has engio stubs for your camera (if not, copy them from platform stubs, compile it and post the diff)
-
Try right click, save as.
Oh that worked.
-
Here is the stub for 550D.
NSTUB(0xff1c15cc, _engio_write)
I'll sort the other cameras out and submit a patch to a1ex.
-
Here is the stub for 550D.
NSTUB(0xff1c15cc, _engio_write)
I concur.
Thanks for looking it up though.
-
Just do like ayshih did here:
For adtg_gui.mo, I've tested adding the DIGIC hooks for the 50D by grabbing the addresses from stubs.S, and they appear to work fine. I can see registers such as SaturateOffset (0xc0f0819c) , and I can still see ADTG registers (which is mentioned in a comment as an issue for the 5D Mark II).
else if (streq(camera_model_short, "50D")) // http://www.magiclantern.fm/forum/index.php?topic=6751.msg63322#msg63322
{
ADTG_WRITE_FUNC = 0xFFA11FDC;
CMOS_WRITE_FUNC = 0xFFA12190;
ENGIO_WRITE_FUNC = 0xFF97D904; // from stubs
ENG_DRV_OUT_FUNC = 0xff97d794;
}
-
Added missing stubs:
https://bitbucket.org/Audionut/zebras/commits/8779a5fa91ed4750268087d104bbf80b646a5a5c
Where do you find the CMOS/ADTG registers?
-
I have compiled ML before with no problems.
And I have just gone to compile it and I get this error.
mem.o: In function `GetMaxRegionForAllocateMemory':
mem.c:(.text+0x168): undefined reference to `GetSizeOfMaxRegion'
collect2: error: ld returned 1 exit status
make: *** [magiclantern] Error 1
But also!!!
what is: requires CONFIG_GDB=y
-
Need to find the stub for GetSizeOfMaxRegion and add it to stubs.S
I have no idea how to do that.
Enable CONFIG_GDB in the makefile (https://bitbucket.org/hudson/magic-lantern/src/a7b8f4853167bc8763e22fc14f5cab966a58bd9c/Makefile.user.default?at=unified).
-
Oh and, compiling the module, (as I wasn't actually doing that), spits out this:
/hudson-magic-lantern-1d65c42c0448/modules/adtg_gui$ makeabort: no repository found in '/hudson-magic-lantern-1d65c42c0448/modules/adtg_gui' (.hg not found)!
REBUILDING
[ README ] module_strings.h
Traceback (most recent call last):
File "../readme2modulestrings.py", line 111, in <module>
last_change_date, last_changeset, author, commit_msg = last_change_info.split("\n")
ValueError: need more than 2 values to unpack
make: *** [module_strings.h] Error 1
PS. I think I give up
-
Can you compile other modules?
The module build system extracts some info from the mercurial repository.
-
5D3.113
ISO: https://www.dropbox.com/s/ltxk0117up6eb9h/iso.log
Aperture(f/2.8 ... f/8): https://www.dropbox.com/s/c59d874jibzw7bw/av.log
(Is shutter log needed?)
Will do same for 550D when stub for GetSizeOfMaxRegion is added.
-
I already have the 5D3 graphs, but being able to cross-check the data never hurts.
Added 550D GetSizeOfMaxRegion from mk11174.
-
Added 550D GetSizeOfMaxRegion from mk11174.
550D.109
ISO: https://www.dropbox.com/s/sifq319w4533bpu/ISO.LOG
Aperture(f/2.8 ... f/32): https://www.dropbox.com/s/p5f508ds249l3cq/Aperture.LOG
Shutter(30" ... 1/4000): https://www.dropbox.com/s/5dtntvhsahv7wqr/Shutter.LOG
-
Dear Mr. Developer Sir, any chance of getting "re-cr2.exe" to change the [edit: white point] in a cr2 (from whatever it was) to a standard acr, or dcraw?
OSS raw converters might be modified to work with mini_iso unorthodox [edit: white point] values, but acr surely won't so not having to decide for one way or another while shooting but being able to convert would lift a huge burden from simply photogs' minds :-o
Next to that, forcing any Canon-changed [edit: white point] to the fixed acr value might be nice anyway, because since I know about this I better understand how acr deals with highlight recovery (or, more to the point, doesn't).
-
What are you talking about?
None of these tweaks can be used to change WB.
-
What are you talking about? None of these tweaks can be used to change WB.
Sorry, just came back from a day crawling around photographing wildlife :-p ... I'm of course talking of the white level, the thingy you can set to various values by newer versions of your mini_iso module.
-
Convert them to DNG and use exiftool if needed. There are no white level tags on the CR2.
I believe Canon autodetects it somewhat like raw_diag does.
-
Convert them to DNG and use exiftool if needed. There are no white level tags on the CR2.
Well, I was thinking of converting the *data* (that's what you do with mini_iso?) so the cr2 can be used with the new white point. Is tagging the dng with the acr white point the same thing, what's the exact tag? Thanks for any info.
-
exiftool foobar.dng -BlackLevel=64 -WhiteLevel=15000
-
Convert them to DNG and use exiftool if needed. There are no white level tags on the CR2.
I believe Canon autodetects it somewhat like raw_diag does.
With exitool on CR2s you can try
-Canon:PerChannelBlackLevel (needs four values)
-Canon:NormalWhiteLevel
-Canon:SpecularWhiteLevel
-Canon:LinearityUpperMargin
-
Thanks, but...
exiftool foobar.CR2 -Canon:NormalWhiteLevel=1234
Warning: Tag 'NormalWhiteLevel' does not exist
-
Does not exist ? for which model do you take this ?. Any sample available ?.
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#ColorData7
-
5D2 and 5D3. There are many samples on the forum (e.g. in the dual ISO thread).
-
Works for me on win vista32 with exiftool 9.51
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Ffilebin.net%2Fxpt6c2c5wh%2FCanon50Dexif.png&hash=233896d00e9ec5901379874c79c16a91)
-
Thanks, I had 9.39, downloaded a few months ago.
Is there any software that uses this tag, given that exiftool didn't know about it until recently?
(ufraw doesn't, so I assume dcraw doesn't either, and Adobe DNG converter simply sets the white level to 15000 in the DNGs)
-
ACR/LR ignores it.
-
Digital Photo Professional is showing a difrence in the highlights when I put values between 0 and 900 in exiftool.
-edit: it changes the color of the highlight somehow
Hmm the lower values desaturate indeed. No headroom to change colors afterwards.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.alisultan.nl%2Fmagicbug%2Fhighlightcolor.JPG&hash=cc7b90b0ceb2e10483480b51ed815fb9)
-
Looks like it's changing the desaturate threshold. Cool find.
-
DPP confirmed as respecting the tags.
-
Yes, DPP should use this info but I don't have it to test.
The Normal white level was 14000 before I changed it to 4321 .. so the test with DPP should be between 14000 and a changed one ..
I don't know the exact meaning of these tags. Adobe DNG/Dcraw and derivatives only care about SpecularWhiteLevel as I understand it. Although it does not always match the white clipping level as we can see it in raw histograms.
Although I think it's interesting that there are three tags about WL which DPP probably uses.
We see the linearity upper margin being 1/2 stop under normal WL . does this mean that the raw highlight data are non linear ? or were they non linear at the start (ADC output) and then linearized before storage in *.CR2 ?.
WL values seem to be steady after gain manipulations, so I don't know if they are reliable as of now. But I hope ML can push Digic to write the correct ones if needed .. :).
BL values look to be measured in the optically black zones so they are reliable and could help for a faster BL identification ..
-
It's not being desaturated per se, the white levels are being crushed.
You can create the same effect with luminance tone curves.
-
@ IliasG what is that program showing all those metadata entries?
-
I don't know the exact meaning of these tags. Adobe DNG/Dcraw and derivatives only care about SpecularWhiteLevel as I understand it.
Ufraw doesn't care about spec(ta)cular white level either.
-
Can you update your ufraw-mod to respect the tags?
-
I think so (and dcraw too).
Now, is there a connection between these exif levels and the register named "white level?" from these graphs? https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW
(I tried to override it and did not notice any visible effect, but didn't check exif)
WL values seem to be steady after gain manipulations, so I don't know if they are reliable as of now. But I hope ML can push Digic to write the correct ones if needed .. :).
I can re-scale the data analogically so I match the actual white level with any value you want. Just tell me what value I should use.
So far I've included 15000 (Adobe), 16380 (full range), the dcraw value (camera-specific) and Canon's default setting with manual lens (that is, simply canceling the aperture gain).
-
I just cycled through all of the WL options in mini_iso. All exposures resulted in these values regardless of saturation.
NominalWhiteLevel: 16228
SpecularWhiteLevel: 16383
LinearityUpeerMargin: 11129
The RawMeasuredRGGB tag is weird. With mini_iso on a batch of exposures, saturation in the center of the image cause tag values 0,0,0,0. Without saturation, this tag contained large values.
Going back through a bunch of shots without mini_iso, this tag looks very random.
-
@ IliasG what is that program showing all those metadata entries?
exiftool with exiftoolGUI as frontend
-
Ufraw doesn't care about spec(ta)cular white level either.
I mean that Dcraw and derivatives only care for the topmost value of data. This is hardcoded in Dcraw and Ufraw (single value for all cases .. you know how good this is ..) or a lookup table.
Rawtherapee uses an external file named camconst.json (camera constants) where among others one can put per ISO WL and a multiplier for each Fnumber ..
DNG looks like also using a table for each ISO but has nothing to fine tune WL depending on apperture ..
In fact I came again in ML forums searching for data regarding the per ISO and per aperture WhiteLevels .. so if it's possible to use the already collected data I ask your permition to use them for RT .. and a bit help how to mine them from the log files .. or even better if the are readily available as numbers ..
-
Are you referring to the register values tagged as WhiteLevel in my dropbox graphs, or to values detected by raw_diag?
In any case, feel free to use them for RawTherapee. I took the AMaZE algorithm from them, and I'm thinking to send them a patch for floating-point DNG support (my version doesn't open CeroNoice DNGs, but didn't check the bleeding edge).
If you collect a table with white level values in a nice format, I'd be interested too (just to have them organized).
-
Are you referring to the register values tagged as WhiteLevel in my dropbox graphs, or to values detected by raw_diag?
In any case, feel free to use them for RawTherapee. I took the AMaZE algorithm from them, and I'm thinking to send them a patch for floating-point DNG support (my version doesn't open CeroNoice DNGs, but didn't check the bleeding edge).
If you collect a table with white level values in a nice format, I'd be interested too (just to have them organized).
I think raw_diag white levels is what I need .. basically the values that are same as if I would inspect the raw histogram for white clipping. This for normal Canon cr2s not ML modified .. and I think ML should also target to the same WLevels as the native Canon cr2s.
I would strongly prefer to use the exif data (specularWL) so the best would be a DCraw patched to use the tags, but I have to recheck as I remember a not insignificant deviation from the histogram derived WL in some rare cases ..
floatDNG is not supported ... waiting for your patch .. :)
The WL info will be freely available in RT code just search for the "camconst.json" file. I don't know if you'll like the format but as I will use spreadsheet to get the data ready I can sent it when it's ready ..
-
Hi everybody,
I am following this thread from the beginning, being excited about the possible improvements of your amazing find.
Is my following short conclusion correct?
The DR improvement on 5D MKIII is confirmed and the registers are identified.
Nevertheless there is no tool that could automatically tweak all those registers for a specific camera (without the possibility of damage). This tweaking may be required because every sensor is different.
Furthermore some RAW-converters will not benefit from the improvements because their whitelevel is fixed. But ACR seems to work.
In case there already is a module for 5DMKIII that can be used for testing and does not involve setting all those registers manually, please tell me where to find it :-) I would love to help testing.
-
This tool is not yet published.
Furthermore some RAW-converters will not benefit from the improvements because their whitelevel is fixed
Quite the opposite. It's exactly these raw converters that will benefit (because the CR2 can be adjusted to match what these converters expect). I've included presets for dcraw and Adobe.
-
Great!
Thanks for clarification :-)
-
Are there more tests needed? Anything I can help with?
5d3 running the nighters
Are there useful tests that would be helpful from a 6d owned by a risk-adverse non-dev?
I notice there is a module that seems related to ADTG, but I am ignorant about what it does. Much of this thread is over my head.
-
5D3, ADTG2 and ADTG4[0x8, 9, A, B]:
- at first sight, these seem to be some black biases: in LiveView, they will cause vertical lines in shadows, but there is a filter that will adapt to the changes fairly quickly (seems to be a moving average).
- at a closer look, changing this register also affects the gain, and we can lower the ISO a little bit more (noise gets lower and there is some more highlight detail captured). Initial guess: 0.15 ... 0.25 stops (maybe more if there is nonlinearity, I don't know yet).
- I believe these are the registers Greg was playing with on 500D.
Will try to update the research tools to the latest codebase today.
-
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fmk4s0m48r%2F0x8.png&hash=c3badb2adcf43b3ca6c0ba7c605d9397)
Previously 11.44
A separate button to enter top of the ADTG menu would be handy. This way when you are in the middle of a 600 (or more) item menu, you can get back to the top of the menu quickly. Or listing the actual registers in a separate menu with the settings contained in a top menu?
Digic registers do not turn off.
edit: To be more precise, they do not get removed from the menu when turning the digic option off.
-
Is this only for Raw? Or will it be able to be applied to in camera jpegs and h264s?
-
We are doing the analysis only for RAW.
If you are interested in JPEG, you may be lucky, since the JPEG is created from RAW. But you'll have to do your own analysis ;)
If you can find a digital gain that affects JPEG but does not get burned in the CR2 file, it may be interesting.
-
Updated research tools to the latest codebase (unified) and added them to a branch on the main repository.
https://bitbucket.org/hudson/magic-lantern/pull-request/412/iso-research-tools
-
Looks like the column gain registers need to be adjusted for each ISO when tweaking ADTG2 and ADTG4[0x8, 9, A, B].
Here is with the column gain registers at 0x32?
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F5ibxyss17%2F100.png&hash=bb5a025de1a5a2423ae3a2efc8d75b9b) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Flgknoxtyz%2F200.png&hash=bff063fec242813037e99674f4f877d0)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F8p6hifrwr%2F400.png&hash=8a5c8951b631fc06be86bb4ce3815168) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fimhibi78b%2F800.png&hash=23deb646adaaca9f417bb95af6cafc5c)
Very close to 1 full stop improvement over Canon gain at ISOs 200, 400, 800 :D
And here with the column gain registers adjusted to bring the WL back to sweet spot.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F6xdinjnzf%2F1001.png&hash=98bad367222113b0f8702fc201084ec1)
Still have lots of observing to conduct, but so far I cannot raise ISO 100 any further.
Thanks for the menu tweaks.
-
We are doing the analysis only for RAW.
If you are interested in JPEG, you may be lucky, since the JPEG is created from RAW. But you'll have to do your own analysis ;)
If you can find a digital gain that affects JPEG but does not get burned in the CR2 file, it may be interesting.
Ah.
I didn't mean a separate thing, I mean the same thing that gets applied to Raw then renders the jpeg and video in camera, to get rid of the tstop at f/1.8.
-
Hi again,
I am sorry if I am again posting something stupid, but Lenny reminded me of some fact I did read about the 5D III sensor.
There was a company that used an electron microscope to examine the sensor of Canons 5D MK III.
They revealed transistors that could be used for electronic pixel binning of three adjacent image pixels. The furthermore assumed this could be used in video mode to reduce aliasing. Unfortunately I could not find a proof of this speculation. Even worse is the fact that this paper was only accessible for some hours or days. I can not find it any more.
Why do I post it here?
Wouldn't binning of three pixels improve the resulting full well capacity by a factor of three? That COULD significantly improve DR in video mode.
On the other hand it would also increase the shot noise compared to one single channel. So if three binned pixels are read with the same settings (Gain & ADC) as one single this would result in WORSE DR compared to photo mode. (Which is what you seemed to see in the beginning?) If one could now somehow adjust the settings to respect the greater full well capacity it could finally increase the video DR, wouldn't it?
Again: I assume that somebody has already investigated / tested this and did not publish his results because it did not work or the information about those transistors is wrong. But since there is a tiny chance that this has not yet been tested I thought I should give it a try.
-
We didn't get this far (we focused mostly on photo mode), but the question does make sense.
-
Is the FPN filter mentioned earlier something that would run in-camera, or post? If in-camera, is it possible that it could be used in video mode, even if the ISO tweaks couldn't?
Making that deep shadow noise prettier would be big plus for video, especially given that the bands crawl, in my experience.
-
It's Canon's auto-calibration algorithm (you are already using it).
-
Maybe I've described it wrong. I'm talking about these mentions:
On my test shot with Greg's car, it didn't remove the pattern noise and actually interpreted it as detail.
I have found a math model for this pattern noise though, and I hope to have a clean fix for it in the near future.
The banding pattern accounts for only a small part of these histogram distortions.
Proof:
ISO 100 dark frame (5D3), original and corrected
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fiso100.png&hash=62a723ff61bb21ed89c0de8a1bf8b359) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fiso100-nobanding.png&hash=c1c95e05784439c190ac93a95cb6c23a)
Histograms of horizontal and vertical banding correction (per-line and per-column)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fhband.png&hash=7ac9c797360649acd520e5f765a810e7) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fvband.png&hash=8ed906f6aecef04a37f191ed58383067)
Image histogram before and after correction:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fhist0.png&hash=dde8eb3fda72d4d84df709aab77e6ef2) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fhist1.png&hash=325235ff026961e4e906b746df43178c)
How I've estimated the correction:
# estimate horizontal and vertical banding
fh = mean(im'); fh = fh - mean(fh);
fv = mean(im); fv = fv - mean(fv);
# banding histograms
hist(fh, -3:0.1:3)
print -dpng -r60 hband.png
hist(fv, -3:0.1:3)
print -dpng -r60 vband.png
# compute the flat field
[m,n] = size(im);
ffh = fh(1:m)' * ones(1,n);
ffv = ones(m,1) * fv(1:n);
ff = ffh + ffv;
# image histogram before and after correction
hist(im(:), 2048 + (-50:50))
print -dpng -r60 hist0.png
hist(im(:) - ff(:), 2048 + (-50:50))
print -dpng -r60 hist1.png
I took that as an idea for new software to de-pattern shadow noise.
-
Ah, you should have linked it from the beginning. I thought it's about today's findings.
It can run in camera, but expect around 10 seconds (maybe more) for one picture.
-
An old personal goal of mine was 11.5 :)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fcljte6ctn%2FISO200.png&hash=08f1848e387199a94d252027ae773fc0)
http://lifeinmegapixels.com/blog/2012/01/pixel-binning-does-it-work/
So while you can do pixel binning in post if you want, it doesn’t particularly gain much. So is it still worth doing? Yes, when done during capture.
The key issue is overcoming the noise floor, and once the noise is present, no amount of post work with remove it. Read noise is one of the problems, which is the noise generated when the pixels are read from the sensor. If pixel binning is done on sensor, i.e the sets of pixels are combined before being read, then the read noise will be greatly reduced (one portion of noise per final pixel instead of 4/9/etc). This means that additive binning really can provide some benefit in this case by reducing that source of noise. This is performed in some point and shoot cameras where high ISO images come out with reduced resolution, but also on high end cameras such as the Phase One medium format backs with the Sensor+ setting.
-
Ah, you should have linked it from the beginning. I thought it's about today's findings.
It can run in camera, but expect around 10 seconds (maybe more) for one picture.
Yeah sorry, I just read the whole thread for the first time today. I thought 'FPN' captured it.
So I assume your idea was for an post tool addition. I'm pitching the case for video. Banding is bad, but it's really bad when the bands crawl - the motion catches your eye, and you really see them.
-
Banding is bad, but it's really bad when the bands crawl - the motion catches your eye, and you really see them.
Well FPN, is fixed by nature.
"Crawling" pattern noise, will be an entirely different beast to fix, afaik.
-
(...)
Wouldn't binning of three pixels improve the resulting full well capacity by a factor of three? That COULD significantly improve DR in video mode.
(...)
This is possible if the binning is done on-ship (accumulated charge from each pixel involved in the binning is brought together and summed before the process of ADC occurs) and if the output register pixels can hold the extend charge.
If not, binning reduces the detectable amount of light per pixel, hence the FWC.
Beside, the ADC quantification have to be calibrate for this special configuration.
But in the worse case binning significantly improves the SNR, witch is very interesting by the way.
-
If not, binning reduces the detectable amount of light per pixel, hence the FWC.
White level. It would have to be a special post processor that reduces the capacity of the pixel wells :)
-
Got some graphs.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewreg%2Fcanon.png&hash=09cb9a85ab04da413dd70799ff2147ec) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewreg%2Fadtg-2.png&hash=4c00c6240a223a1061af16e3c79e2754) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewreg%2Fnew-reg.png&hash=a3744ffa149cb224d28f2a4a807a7d97)
On X, there is the raw data from ISO 100 1/200, manual lens, with no tweaks applied. Black subtracted and data converted to EV.
On Y, we have:
1) raw data from ISO 100 1/50, no tweaks applied. It's 2EV brighter (obvious).
2) ADTG gain at -2 EV, along with some other tweaks (black forced to 64, white forced to 15000, CMOS[4] patched to 0x318).
3) the tweaks from 2, plus the latest tweak discovered today (all those registers set to 0).
Source: https://bitbucket.org/hudson/magic-lantern/commits/32ef83a5ee41
Updated raw_diag binary on first post.
-
for #3 > sigma looks higher, isn't? But transition before clipping looks smoother.
-
Take it with a grain of salt; the white level detection algorithm doesn't quite agree with the graphs. Time to figure out why...
edit: graph autoscaling was broken, white level was OK, re-uploaded the graphs.
Here's another one, with what I believe to be the sweet spot for both gains. First I've tuned the ADTG gain to sweet spot (which can be done with 2 test pictures, since the response curve is linear), then I've reduced the new gain until the white level started to decrease (but this one is trickier because of the nonlinearity).
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewreg%2Fsweet-spot.png&hash=aa2a958a9396ab910f49f4a233d3fa50)
-
Disregard this one (poor repeatability, probably because of flicker from the light source).
Repeated the experiment with sweet spots only, no black/white level changes (so the graphs should be easier to compare) and no camera movement (the entire test sequence was ran from a script (https://bitbucket.org/hudson/magic-lantern/commits/efc750013a69)).
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv2%2Fcanon.png&hash=eccff8020f8404576f9ad98814f0dbe5) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv2%2Fadtg-sweetspot-nocmos.png&hash=a5856123d528abe5149a0f1ccc7a998b) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv2%2Fadtg-sweetspot-cmos4.png&hash=4b35d8bf64121ca102ba5547a9af2379) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv2%2Fnew-reg-sweetspot-nocmos.png&hash=ffbeaadf069d45a06d61bb572c2c740a) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv2%2Fnew-reg-sweetspot-cmos4.png&hash=3d2b4ba3ded25442fe385180bea0629d)
On X, there is the raw data from ISO 100 1/200, manual lens, with no tweaks applied (vanilla Canon settings).
On Y, we have:
1) raw data from ISO 100 1/50, no tweaks applied. It's 2EV brighter (obvious).
2) ADTG gain at sweet spot (-0.37 EV).
3) ADTG gain at sweet spot (-0.37 EV) and CMOS[4] patched to 0x318.
4) ADTG gain at sweet spot (-0.37 EV), new gain register set at 36 (default was 59, units are not yet known).
5) ADTG gain at sweet spot (-0.37 EV), new gain register set at 36, CMOS[4] patched to 0x318.
Ran the test again to check the repeatability of the results:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv3%2Fcanon.png&hash=bc2e23cf7a6c7065fbbc48017a62be00) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv3%2Fadtg-sweetspot-nocmos.png&hash=c9929badc26215c674897be1c4d64244) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv3%2Fadtg-sweetspot-cmos4.png&hash=89d3c4f2575fd2d28b4bbb6913ac2410) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv3%2Fnew-reg-sweetspot-nocmos.png&hash=e0bc69ab151098698371e41433c8d071) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv3%2Fnew-reg-sweetspot-cmos4.png&hash=427a21c8638266751f2c7e4c1d729eac)
ISO differences, computed as median(real(log2(b)-log2(a))) from the second data set:
2.07, 1.71, 1.57, 1.51, 1.54 EV.
=> the modified ISOs (estimated from median brightness, not from clipping point) were, assumming Canon ISO 100 is really ISO 100:
ISO 100, 77.9, 70.7, 67.8 and 69.2.
The ISO 77.9 matches the theory (I've lowered ADTG gain by 0.36 stops, I've measured 0.36 stops of difference between the two pictures).
ISO differences, computed from clipping point: 2.07, 1.68, 1.44, 1.34, 1.49.
=> the modified ISOs (estimated from clipping point) were, assumming Canon ISO 100 is really ISO 100:
ISO 100, 76.3, 64.6, 60.2 and 66.8.
ISO estimation code: https://bitbucket.org/hudson/magic-lantern/commits/0c18a31891b6
-
new gain register set at 36 (default was 59, units are not yet known).
My defaults.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F4rj7sscqj%2Fregister.png&hash=021c40f03620cad34b64f82728b95eff)
They don't appear to vary, at least not with the lower ISOs.
-
Yes, they don't vary with ISO. I've set all of them to 36 = 0x24 from iso_regs, and the first one (the one printed in iso_regs) was 59 = 0x3b.
Looks like the repeatability is much better at 1/50 vs 1/25 (probably because of artificial light flicker and maybe also mechanical flicker) => will repeat the experiment with these values.
-
Did the the experiment again with 1/25 (tested image) vs 1/50 (reference image) because this improves the repeatability. ISO 100 in Canon menu.
Repeatability series 1: Canon 1/25 vs 1/50, ISO tweaking modules not loaded, camera not moved during the entire experiment.
Expo difference (EV)
median clip
1.01 1.03
1.01 1.01
1.01 1.00
1.01 1.00
1.01 1.01
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-canon-1.png&hash=4fad2eff5e8238f0e17471279ff8e0d2) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-canon-2.png&hash=84daec09bdc849e71a2c85b0a3695eec) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-canon-3.png&hash=aee6be695076c55c24b085b5ee91e7f5) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-canon-4.png&hash=be618f0ca08a6b3ac9bc17f59d4748c1) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-canon-5.png&hash=0ad1e367aaf7147aa2adf44a89a7d89c)
Checking register settings:
Expo difference (EV)
median clip
1.00 0.99 # Canon 1/25 vs 1/50
0.63 0.59 # 1/25 with ADTG gain at -0.37 EV vs Canon 1/50
0.64 0.60 # 1/25 with ADTG gain at -0.37 EV and CMOS patched vs Canon 1/50
0.49 0.40 # 1/25 with ADTG gain at -0.37 EV and new gain at 36 vs Canon 1/50
0.50 0.42 # 1/25 with ADTG gain at -0.37 EV, new gain at 36 and CMOS patched vs Canon 1/50
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Fcanon.png&hash=f88f873c03ac3d21d430095582db248d) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Fadtg-sweetspot-nocmos.png&hash=842f16863bbb4611c105b299146b83f4) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Fadtg-sweetspot-cmos4.png&hash=749457b1455742065f21087a4cfc73e6) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Fnew-reg-sweetspot-nocmos.png&hash=394c18acf0a45483d79ef2a9940d857c) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Fnew-reg-sweetspot-cmos4.png&hash=55adfaeb5dd9aa846e09f84d229d5656)
Repeatability series 2: settings from the third tweaked set (1/25, ADTG gain -0.37 EV, new gain 36, no CMOS patch).
Camera moved intentionally between each bracket, but not moved during a bracket.
Expo difference (EV)
median clip
0.51 0.43
0.49 0.38
0.50 0.40
0.47 0.38
0.49 0.40
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-tweak-1.png&hash=d3744b7c4e6cab590e0a32ef752d3085) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-tweak-2.png&hash=bd3e44dbae54a8eee24796ad6bf32bc6) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-tweak-3.png&hash=35d570c1e3f2f105ba0c47f62c6573c0) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-tweak-4.png&hash=b3163492760ac7c26f77d157a3fb8938) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fnewregv4%2Frepeat-tweak-5.png&hash=1e5076ff2651d4b63d820d9ee7438b29)
Clarifications:
- CMOS patched: changed CMOS[4] from 0x718 to 0x318.
- ADTG at -0.37 EV: all ADTG gains were scaled by this value from factory gains, and this value was found to be the sweet spot (lower this gain more => white level starts to decrease).
- new gain at 36: ADTG2/4[8,9,A,B] all set to 36 from iso_regs (one of them was 59 by default). Below this level, highlights are no longer clean (they start to get vertical banding).
- ISO estimation code: https://bitbucket.org/hudson/magic-lantern/commits/0c18a31891b6
- the script for running the experiment https://bitbucket.org/hudson/magic-lantern/commits/5df6093487a6 (note: you need mini_iso, which is not yet published, but I can send you a copy on request via PM).
Conclusions:
- on 5D3 we now have a really nice ISO 66 (-0.6 stops below ISO 100).
- we also have two methods of estimating the relative ISO in raw_diag + octave.
-
ISO 200 in Canon menu:
Expo difference (EV)
median clip
1.00 1.03 # Canon 1/25 vs 1/50, ISO 200
0.58 0.59 # 1/25 with ADTG gain at -0.41 EV vs Canon 1/50
0.60 0.59 # 1/25 with ADTG gain at -0.41 EV and CMOS patched vs Canon 1/50
0.25 0.21 # 1/25 with ADTG gain at -0.41 EV and new gain at 0 vs Canon 1/50
0.24 0.20 # 1/25 with ADTG gain at -0.41 EV, new gain at 0 and CMOS patched vs Canon 1/50
=> the new ISO pulled from 200 was actually 115.
From ISO 400:
Expo difference (EV)
median clip
1.00 1.04 # Canon 1/25 vs 1/50, ISO 400
0.57 0.58 # 1/25 with ADTG gain at -0.43 EV vs Canon 1/50
0.56 0.57 # 1/25 with ADTG gain at -0.43 EV and CMOS patched vs Canon 1/50
0.22 0.20 # 1/25 with ADTG gain at -0.43 EV and new gain at 0 vs Canon 1/50
0.22 0.21 # 1/25 with ADTG gain at -0.43 EV, new gain at 0 and CMOS patched vs Canon 1/50
=> the new ISO pulled from 400 was actually 230.
From ISO 800:
Expo difference (EV)
median clip
0.97 1.02 # Canon 1/25 vs 1/50, ISO 800
0.55 0.56 # 1/25 with ADTG gain at -0.43 EV vs Canon 1/50
0.57 0.60 # 1/25 with ADTG gain at -0.43 EV and CMOS patched vs Canon 1/50
0.21 0.20 # 1/25 with ADTG gain at -0.43 EV and new gain at 0 vs Canon 1/50
0.20 0.21 # 1/25 with ADTG gain at -0.43 EV, new gain at 0 and CMOS patched vs Canon 1/50
=> the new ISO pulled from 800 was actually 460.
From ISO 1600:
Expo difference (EV)
median clip
0.93 1.00 # Canon 1/25 vs 1/50, ISO 1600
0.53 0.58 # 1/25 with ADTG gain at -0.44 EV vs Canon 1/50
0.52 0.56 # 1/25 with ADTG gain at -0.44 EV and CMOS patched vs Canon 1/50
0.18 0.19 # 1/25 with ADTG gain at -0.44 EV and new gain at 0 vs Canon 1/50
0.18 0.19 # 1/25 with ADTG gain at -0.44 EV, new gain at 0 and CMOS patched vs Canon 1/50
=> the new ISO pulled from 1600 was actually 912.
From ISO 3200:
Expo difference (EV)
median clip
0.97 0.99 # Canon 1/25 vs 1/50, ISO 3200
0.49 0.50 # 1/25 with ADTG gain at -0.49 EV vs Canon 1/50
0.50 0.51 # 1/25 with ADTG gain at -0.49 EV and CMOS patched vs Canon 1/50
0.14 0.14 # 1/25 with ADTG gain at -0.49 EV and new gain at 0 vs Canon 1/50
0.13 0.13 # 1/25 with ADTG gain at -0.49 EV, new gain at 0 and CMOS patched vs Canon 1/50
=> the new ISO pulled from 3200 was actually 1750.
From ISO 6400:
Expo difference (EV)
median clip
0.95 1.00 # Canon 1/25 vs 1/50, ISO 6400
0.41 0.43 # 1/25 with ADTG gain at -0.55 EV vs Canon 1/50
0.43 0.49 # 1/25 with ADTG gain at -0.55 EV and CMOS patched* vs Canon 1/50
0.06 0.06 # 1/25 with ADTG gain at -0.55 EV and new gain at 0 vs Canon 1/50
0.10 0.11 # 1/25 with ADTG gain at -0.55 EV, new gain at 0 and CMOS patched* vs Canon 1/50
* CMOS[4] patched from 0x718 to 0x318 and CMOS[3] patched from 0x144 to 0x944.
=> the new ISO pulled from 6400 was actually 3333.
From 12800:
Expo difference (EV)
median clip
0.91 1.05 # Canon 1/25 vs 1/50, ISO 12800
0.35 0.39 # 1/25 with ADTG gain at -0.58 EV vs Canon 1/50
0.36 0.40 # 1/25 with ADTG gain at -0.58 EV and CMOS patched* vs Canon 1/50
0.01 0.03 # 1/25 with ADTG gain at -0.58 EV and new gain at 0 vs Canon 1/50
0.00 0.00 # 1/25 with ADTG gain at -0.58 EV, new gain at 0 and CMOS patched* vs Canon 1/50
* CMOS[4] patched from 0x718 to 0x318 and CMOS[3] patched from 0x144 to 0x944.
=> the new ISO pulled from 12800 was actually 6400.
Graphs for these sets uploaded here: https://www.dropbox.com/sh/onppbwy44fqomxa/VlGTJBoSr4/5d3-iso-experiments
(note: the metadata from the graph itself may be incomplete; also look at the file name to know which is which)
-
I'd like to understand them first :o
Haven't had a chance to play with updated raw_diag.
What do the blue/green/red lines represent? I find it hard to tell what is what. I think I know what I am looking at, but could you please explain it a little more.
-
Blue is blue channel, red is red channel, green is green channel. They are compared to the values from a reference shot taken with Canon settings (unmodified ISOs) and underexposed by 1 stop via shutter.
So, in the first graph (Canon 1/25 vs Canon 1/50), you see the graph exactly 1 stop above the diagonal. That's because the exposure difference between these two is exactly 1 stop.
In the subsequent graphs, you see the graph getting closer to the diagonal (that's because ISO is getting lower).
If you define ISO as "overall brightness", I estimate it from the median vertical distance between the graph and the diagonal.
If you define ISO as "clipping point" (DxO definition), I estimate the clipping point and then take the horizontal distance between the two clipping points. If that distance is 0.6 stops, it means our ISO is 1-0.6 = 0.4 stops lower. Note the overall brightness might difer a little because of nonlinearity.
When you want to know how much you can expose to the right with the new ISOs, you need to know the clipping point.
-
I enjoy just sort of watching this thread utterly without comprehension. Like a cat watching TV.
-
Blue is blue channel, red is red channel, green is green channel.
Yeah, that makes sense!
I assume the 2 banks of column gain registers (ADTG2/ADTG4) are for the dual amps that dual_iso uses, correct? And ADTG6 looks to be the feedback amp?
I only ask because cr2hdr reports a low DR (about 0.2-0.3 EV to low) for the recovery ISO. Looks like we are getting 0.6 EV of overlap too :)
-
Dual ISO only changes the CMOS[0] register. But, of course, it benefits from the other tweaks too.
ADTG6... I don't know yet what it is, did you notice any effect by changing it?
-
Looks like ADTG6[8880] 0x750 gives 0.02 EV. Every little bit counts right?
Otherwise, I haven't found anything useful yet.
-
Dynamic range series (ISO 100-12800 in Canon menu, ADTG gain at sweet spot, ADTG2/4[8,9,A,B] at 36 for ISO 100 and 0 for all other ISOs, CMOS patches enabled.
ISO DR
66 11.589 + 0.1
115 11.772
230 11.649
460 11.427
912 10.968
1750 10.369
3333 9.610
6400 8.834
I've added 0.1 to the measurement at ISO 100 because of the nonlinear highlight rolloff (approximate). The other ISOs seem to clip harshly, the difference between median and clip ISO is smaller, so I think we can consider the response linear.
Comparing with my older measurements:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-cmos.png&hash=f0131ad5729d99c9b02da7cc5f7793de)
Note: the ISOs may be approximate. As I said in the dual ISO thread: I'm an engineer, for me, Pi is 3.
-
Most of this is way over my head but from the looks of that chart you guys have hit on another incredible ML feature!!!!
-
11.77 EV is remarkable. I don't imagine it is going to be possible to modify the ADTG registers for dual_iso.
Based on a1ex's ISO data above, and the measured ISOs from DxO (http://www.dxomark.com/Cameras/Canon/EOS-5D-Mark-III---Measurements), the true ISOs should be.
Canon Menu ML
100 53
200 92
400 186
800 369
1600 729
3200 1377
6400 2697
12800 5042
-
With new ADTG gain at 0x0, I haven't noticed any highlight banding.
Consider these histograms.
Canon
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/Canon.png
ML with ADTG gain at -0.37 EV, new gain at 36 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML36.png
ML with ADTG gain at -0.37 EV, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.png
We still gained highlight detail with new gain at 0.
-
So, we have ISO even lower than 66?
Will take a closer look.
-
It looks like it from here. I trust your closer look more :)
If I am reading the histogram right, here is ISO 200. Here we lose some highlight detail compared to ISO 100.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0ISO200.png
Then we start to gain some more highlight detail through every higher ISO. Here at ISO 6400.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0ISO6400.png
Both shots, ML with calibrated ADTG gain, new gain at 0 and CMOS patched
Look at ISO 100 with calibrated ADTG gain, new gain at 36 and CMOS patched
vs
ISO 200 with calibrated ADTG gain, new gain at 0 and CMOS patched
We see a gain in highlight detail that matches ISO 200 through ISO 12800.
ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched seems to be applying a large amount of highlight roll off. Need to check if there is actual detail right near saturation, or Rawdigger is struggling to accurately report details.
edit: I'm probably just seeing increased stdev with increasing ISO, not extra highlight detail.
-
Here's the banding I'm talking about:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fhighlight-banding.jpg&hash=9a6cfea2b89393b0999e34ae820cf39b)
(ISO 100, new gain = 25, white level forced to 15000 when developing)
So, it looks there is some more detail, but I think it would require some extra postprocessing (this signal is not exactly clean, has strong trend and banding noise). My settings were conservative, with clipped white being clean white.
By trial and error, I changed the white level for this shot from 15282 to 14682, and at that point the highlights were clean (well, just a tiny bit dirty). On the reference shot (the one underexposed by 1 stop), I changed the white level until the clipping area was the same (that was 12282). So, the input signal required to saturate the sensor at 1/25 was log2(15282-2048) - log2(12282-2048) = 0.371 stops weaker than the one required to saturate it at 1/50.
This means our ISO got lower by 0.629 stops, that is, ISO 64.7.
QED
-
Whites are fine here, but there is a very faint pattern noise through the exposure, with new gain at 0x0 ISO 100. White level - manual lens (15283).
The pattern is that faint, these images needed to be full resolution to show the effect.
ML ISO 100 with calibrated ADTG gain, new gain at default and CMOS patched
2.23 MB
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML_no_new_gain.jpg
1.77MB
ML ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.jpg
Looks to me (my eyes), that there is actually less read noise with new gain 0.
-
1.77MB
ML ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.jpg
It has vertical stripes.
-
Nevermind. One day I will learn maths.
-
I enjoy just sort of watching this thread utterly without comprehension. Like a cat watching TV.
I'm in a similar state of non-comprehension. Over my head. But fascinating.
But ... is the following something of a "bottom line" and "rubber meets the road" ????
- If this ADTG tweaking works, ML equipped cameras would have the capability of using a "for real" lower ISO such as ISO 65 that would have lower noise and greater dynamic range than the "native" ISO 100 without ML.
- The increase in DR could be on the order of 0.25 to 0.50 EV?
- I'm not clear is this works with Dual-ISO
- I'm unclear if other ISO's are potentially improved, like something between 100 & 200?
- It wouldn't be like the existing ISO 50 that is kind of the equivalent of a neutral density filter, with reduced DR and no real improvement in IQ.
- Or am I confused and "unclear on the concept"
- My other less-than-informed impression is that there would be a conservative ADTG default value that would improve DR, and a in-camera tool that would allow tweaking of an individual camera to optimize DR (sort of like Auto-Dot-Tune)
- I'm fuzzy on whether ISO 100 improves, or remains the same?
-
Look at the graph from this page, and at the FAQ from first post.
Generally speaking, by applying some of these tweaks you get some more highlight detail (which means a lower ISO, usually with some more DR).
With the latest findings, the improvement is nearly 0.8 stops on 5D3, ISO 115 pulled from 200. So it's not exactly ISO 100, because it will clip the highlights a little earlier, but I expect it to have 0.8 stops of less shadow noise. Didn't pixel-peep anything, only did the math with open source algorithms that you can try and review.
Dual ISO will be improved mostly on the highlight side (from ISO 66) and on the overlap amount (less aliasing - minor improvement). On the shadow side, at high ISOs most of the noise is photon shot noise rather than electronics noise, so there isn't much room to improve (we are close to the physics limit if I understand well (http://www.dpreview.com/forums/post/53043775), feel free to point me to a better explanation).
I've tried to exploit the nonlinearity by cleaning up the vertical banding and have barely reached ISO 60, so I don't think it's worth the hassle.
The registers on 6D are pretty much identical to 5D3, so you can fire up ADTG GUI and try these tweaks (you will need a custom compilation, it won't load on stock nightly).
-
The registers on 6D are pretty much identical to 5D3
Thank heaven for that :-p ... looking forward to a mini_iso with working white point casting to try :->
-
Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534
(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)
;)
-
;)
Ah, right, will do - but you've already got 6d iso.log from 1%, so you're only lacking av.log? Would you prefer having it with aperture f8 to f2.8 (my fastest lens) or with shutter that covers the ev range f8 to f1.2?
-
It's OK (better than nothing). For white level, the aperture changes are important (not shutter).
Meanwhile I've got logs from 550D from LebedevRI (uploaded to https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW ) and also Nanomad tried to get them from 650D/1100D with only partial success (not yet sure why, probably wrong stubs, the birthday paradox issue from cache hacks, or simply Canon code does no longer calls what I expect it to call).
-
It's OK (better than nothing). For white level, the aperture changes are important (not shutter).
Well, if someone 6d-ish has to do it again in any case like 1% with his famed aperture gadget, I don't need to do half of it unnecessarily - or will the f8-f2.8 log be sufficient make the mini_mo usable for the 6d? As written before, the white point option (like acr 15k) doesn't work atm.
-
I hope it's sufficient. The time required to compile the iso-research branch and take the test pictures should be less than the time required to write all these forum posts :D
-
I hope it's sufficient. The time required to compile the iso-research branch and take the test pictures should be less than the time required to write all these forum posts :D
Yes, but I'm still idling at work so writing is easy, when back at home and doing braindead test series it's my spare time I rather value and only invest if there's some roi (like a working 6d mini_iso) :-o
-
I'm glad a1ex places ML as a high value of his spare time. In fact, I am highly appreciative.
-
I'm glad a1ex places ML as a high value of his spare time.
Obviously we all are as we all are using his free work, and he knows I cannot thank him enough for doing ML and not ending up as a complete nerd but very nice to communicate with.
Personally, I also appreciate a sound work-life balance, whatever that may be... I'm neither applauding self-exploitation (*not* directed towards alex) nor am I entering a contest for the most altruistic man alive, been there, done that, found it to be very unhealthy and being evicted from your flat for missing rent also doesn't help coding efforts.
-
The new gain seems to be expressed in EV. At first sight, 1 unit seems to be around 0.006 EV.
Quick data set (only changed these gains, nothing else; started from ISO 100; the diffs are the median exposure difference printed by latest raw_diag, in EV):
gains = [0 10 20 30 40 50 60 70 80 90 100 150 200 256 ]
diffs = [0.66 0.70 0.79 0.84 0.89 0.94 1.01 1.06 1.13 1.20 1.25 1.56 1.84 2.17]
This means, in order to keep the factory calibration, simply translate the register values by some constant amount (just make sure you don't go below zero). This should fix the vertical banding (didn't try, it's just theory, let me know if it works).
-
Playback filter?
Image :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.org%2Fn56blno6v%2Fimage.jpg&hash=c8e3e8b0375c339f4e54602248d69e46)
It looks like a dual iso :D
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.org%2F4e4ebhtmf%2Fimage2.jpg&hash=cdcfe72fd5f71ad6a689987160cff2d7)(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.org%2F69r6mnign%2Fimage3.jpg&hash=f2c6904582a649f40d7e6f601c713c02)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.org%2Fw2r1j0gmv%2Fimage4.jpg&hash=fe67a91af7cacc64078fefcc6c4c647d)(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.org%2Fylcqjp2d3%2Fimage5.jpg&hash=ffc2014b5bb30c75bdf7ee1ad3dd322a)
-
Lol? Vertical dual Iso. :P
-
Lol? Vertical dual Iso. :P
Maybe it is a constant vertical banding.
-
- removed -
I misunderstood Greg's post.
-
What about breaking the register lists up further?
List only these types,
ADTG2
ADTG4
ADTG6
COF0
COF1
COF2
etc.
When you are deep into a bunch of registers, its rather slow to navigate all the way back to the top of the menu to change editing step.
Menu caret would be extremely useful and remove the need for editing step.
I normally edit in steps of 16 first, and then if there is a register that tickles my fancy, I like to then edit by 1.
-
Canon 5D3 - ISO 100 - 1/100s - Lens unscrewed.
ADTG pre-gain only (otherwise Canon defaults), from 0-200 in steps of 10.
Slightly blurred color checker, 100x100 pixel sample patch, and noting the average raw levels (black subtracted), in one of the green channels, together with patch stdev, and optical black stdev.
WL stdev OB
2739 30.0 5.29
2852 30.5 5.50
2967 32.3 5.71
3092 33.4 5.96
3229 34.7 6.23
3361 35.5 6.46
3497 38.2 6.74
3644 39.2 7.00
3789 40.6 7.26
3958 43.0 7.61
4118 43.8 7.87
4281 46.1 8.28
4473 47.7 8.60
4658 50.0 8.96
4855 52.7 9.33
5063 53.7 9.74
5272 56.8 10.2
5497 57.7 10.5
5722 61.2 11.0
5969 63.6 11.4
6198 66.3 11.9
6468 69.2 12.4
6737 71.5 12.9
7028 75.5 13.5
7326 79.3 14.0
7611 82.6 14.7
ADTG gain, 0.0 EV through -1.00 EV in 0.05 EV steps.
WL sdtev OB
10035 83.4 6.49
9925 80.2 6.29
9362 76.8 6.14
9032 75.9 5.92
8733 71.9 5.75
8427 68.9 5.49
8133 67.4 5.31
7850 63.8 5.18
7584 62.3 5.00
7319 60.2 4.83
7263 58.3 4.60
6828 55.2 4.46
6600 54.1 4.32
6365 51.8 4.15
6158 51.0 4.03
5942 48.0 3.87
5739 47.5 3.76
5543 45.0 3.57
5358 43.1 3.49
5174 42.6 3.33
4989 41.8 3.23
ADTG gain, -1.10 EV through -2.30 EV in 0.10 EV steps.
WL sdtev OB
4655 38.2 3.02
4345 35.9 2.83
4047 33.2 2.57
3782 30.8 2.38
3519 28.6 2.20
3285 27.0 2.08
3065 25.3 1.95
2865 23.7 1.80
2666 21.8 1.68
2429 20.4 1.59
2321 19.2 1.49
2168 18.1 1.41
2022 16.9 1.31
-
Dynamic range series (ISO 100-12800 in Canon menu, ADTG gain at sweet spot, ADTG2/4[8,9,A,B] at 36 for ISO 100 and 0 for all other ISOs, CMOS patches enabled.
ISO DR
66 11.589 + 0.1
115 11.772
230 11.649
460 11.427
912 10.968
1750 10.369
3333 9.610
6400 8.834
I made this DR chart. Is it correct?
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FkXrbKTq.png&hash=1e2cd0032b0c9723bb4fe34c981261b5)
-
The values saturate offset and black/white, such as ISO 200
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs8.postimg.org%2Fral6r7z39%2FVRAM1.png&hash=ba22d0f0910912191e5f275f43c2f773)
-
I made this DR chart. Is it correct?
Looks fine to me.
On the shadow end, one may argue that because of half resolution, you lose log2(sqrt(1/2)) = 0.5 stops (and it makes sense if you look at this test (http://www.magiclantern.fm/forum/index.php?topic=7139.msg99437#msg99437)). I didn't do the math to see if it's really 0.5, but it's probably close.
I should probably update the formulas from the menu => some settings like 800/1600 might result in worse shadows than either of them according to this theory. Didn't try to see if it's true or not, and the blending algorithm wasn't exactly optimized for situations like this.
-
c0F0[819c] 0xf87 -> 0x20
c0F0[8034] 0x1079 -> 0x1e70 (less value to improve DR, but cause vertical stripes blue channel, wrong white point blue channel - http://www.magiclantern.fm/forum/index.php?topic=10111.msg103483#msg103483)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs22.postimg.org%2Fqgjv1vmep%2FVRAM1.png&hash=2580e3fa2e11726f638497b8cf6eaca0)
-
(...)
On the shadow end, one may argue that because of half resolution, you lose log2(sqrt(1/2)) = 0.5 stops (and it makes sense if you look at this test (http://www.magiclantern.fm/forum/index.php?topic=7139.msg99437#msg99437)). I didn't do the math to see if it's really 0.5, but it's probably close.
(...)
I did the math for dual_iso* to compare with some tests; according to the sigma and Nyquist matching algorithm of the Aliasing Minimization and Zipper Elimination demosaicing, with multi-pass >4 and considering the matrix intrication at the source, the lose of DR should be less than 1/2 stops for all the spectral range in daylight photography.
*did for some special wavelengths, not relevant here.
-
fe - default 0x4, lower values reduce stdev, higher values increase stdev
This one is interesting, try setting it to 0 in both ADTG2 and ADTG4. With raw_diag I can measure 11.83 stops, but my intuition says it might be a little more.
@SpcCb: can you share some more details about how you did the math?
(I tried to understand the AMaZE algorithm, but only figured out how to call it; if you can shed some light, I'm interested)
-
With raw_diag I can measure 11.83 stops,
Same here.
It seems to do this to the highlights.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Flqs681ny3%2Fhighlights.png&hash=3d826b5193dd2b7c25bbc61db1b316e1)
I was only using ADTG pre-gain/gain, CMOS tweak and yet another gain. Naturally, the WL was pretty low. I bumped the lens back into electronic mode and hit 11.91 EV.
So then I unscrewed the lens and changed Digital Gain from 0x200 to 0x220, and again 11.91 EV. This seemed to clean the highlight zebras also.
This configuration also looks to clean the highlights a little @ ISO 100 with pre-gain 0
@vroem
I really like that graph. It visualises in a simple manner. I like how it shows the linear loss in highlights, and the non-linear gain in the shadows. Of course, using ML ISO, the loss in the shadows isn't as great as with Canon ISO.
-
500D has the same problem when trying to gain more than 0.33EV.
Up to 0.4EV occur on the blue channel.
-
c0F0[819c] 0xf87 -> 0x20
This one is saturate offset on 5D3.
c0F0[8034] 0x1079 -> 0x1e70 (less value to improve DR, but cause vertical stripes blue channel, wrong white point blue channel - http://www.magiclantern.fm/forum/index.php?topic=10111.msg103483#msg103483)
I can't find this one on 5D3. So many registers :o
I did find c0F3[8034] 0x0
Looks like the faint banding is happening at other ISOs also, here is ISO 200
Canon
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F3nz3gl7gb%2FCanon_ISO200.jpg&hash=13bedde1f29bd4d18c0830381dfe3e95)
ML
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F77l16ehvv%2FML_ISO200.jpg&hash=a8025cea24cd524b2ca418e543178aa7)
-
When you get these artifacts in highlights, one of the gains is too low (below its sweet spot).
0xFE seems to control 3 amplifiers triggered by bits (valid values from 0 to 7):
gains = [0 1 2 3 4 5 6 7 ]
diffs = [0 0.08 0.11 0.17 0.54 0.65 0.72 0.86]
so changing it from 4 to 3 might be already too much (you may need to compensate by increasing some other gain).
@Audionut: can you do the noise tests (WL + stdev + OB) for this register too?
-
For clarity, this sample set was taken at 1/60s. Same setup as previous test. Also, my OB measurement is 10 pixels from left, 500 px from top, 90px wide and 2000px high.
ADTG[0xFE] - 0 through 7.
WL stdev OB
1144 14.9 4.70
1205 16.3 4.91
1237 16.4 5.05
1314 17.2 5.29
1673 21.9 6.54
1809 24.0 6.96
1881 24.7 7.28
2087 27.4 7.99
-
Based on the data from Audionut, I've computed how much noise we gain (or lose) by increasing each of these 3 gains.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fadtg-fe.png&hash=769fbf7d94881a2d4774a9747b38927e) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fadtg-888x.png&hash=8aad1ccc0eb2469889d005700faad80e) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fadtg-89ab.png&hash=84589a89438dc58ab2036bff1435d077)
Think at ISO 100 vs 200: you increase the gain and you get lower noise (cleaner shadows). There is a point in keeping this kind of gains at large values (you may clip highlight detail, but you get cleaner shadows in return). So, the 0xFE gain is one of these gains that help reducing noise.
Also think at ISO 3200 vs 6400: you increase the gain, but there's little or no improvement in noise. You want to keep this kind of gains at smaller values (because you may clip highlight detail and you get nothing in return). The other two gains have only a minor impact on the measured noise, so we want to keep them as low as possible.
Now, from these numbers, can we draw a conclusion regarding the position of each amp in the processing chain? Can we say the order is CMOS[0] -> 0xFE -> 888x + SaturateOffset -> 8/9/A/B -> ADC?
(note that I've renamed them back to their register address, because I'm no longer sure whether 8/9/A/B is before 888x or viceversa)
-
Would it be useful to measure saturateOffset? I can do that when I get home later tonight.
-
Yes, not a bad idea.
-
Hoping this is what was needed for 700D Iso log?
https://bitbucket.org/mk11174/t5i-backup/downloads/700D_ISO.LOG (https://bitbucket.org/mk11174/t5i-backup/downloads/700D_ISO.LOG)
-
That's it. Can you also do the aperture and shutter series?
-
That's it. Can you also do the aperture and shutter series?
will try and post if works.
-
700D PHOTO MODE ADTG Logs
Aperture from F2.8 to F22
https://bitbucket.org/mk11174/t5i-backup/downloads/700D_APERTURE_F2.8_TO_F22.LOG (https://bitbucket.org/mk11174/t5i-backup/downloads/700D_APERTURE_F2.8_TO_F22.LOG)
Shutter from 30second to 1/4000
https://bitbucket.org/mk11174/t5i-backup/downloads/700D_SHUTTER_30sec_TO_4000.LOG (https://bitbucket.org/mk11174/t5i-backup/downloads/700D_SHUTTER_30sec_TO_4000.LOG)
-
edit: I neglected to unscrew the lens
ADTG[0xFE] looks a little bumpy, here it is in reverse.
WL stdev OB
1856 25.9 7.97
1676 23.9 7.21
1625 22.5 6.93
1499 20.9 6.47
1164 16.4 5.25
1111 15.4 5.00
1068 15.5 4.88
1012 14.4 4.66
SaturateOffset - 2000 through 100, in steps of 100. BL is the reported black level in the file.
iso_regs reported default for this register 3148. iso_regs only allowed adjustment up to 2000.
WL stdev OB BL
4633 41.8 6.53 883
4658 42.9 6.49 782
4619 41.4 6.80 681
4659 43.4 6.50 580
4696 42.5 6.49 478
4677 41.5 6.86 377
4646 41.9 6.54 275
4677 41.7 6.57 174
4668 42.2 6.63 73
4573 40.5 0.0
4508 41.0 0.0
4416 41.3 0.0
4330 41.8 0.0
4185 41.2 0.0
4156 42.2 0.0
3999 41.3 0.0
3897 41.8 0.0
3812 41.3 0.0
3729 42.6 0.0
3644 42.7 0.0
Here is the same test images as above, but measuring from a darker grey patch.
WL stdev
770 14.4
776 14.7
717 14.7
726 14.8
730 15.0
727 14.9
724 14.5
727 14.6
726 15.1
687 14.7
590 15.3
490 14.6
391 14.4
282 14.6
193 14.9
82 15.2
0.6 2.83
0.0 0.0
0.0 0.0
0.0 0.0
-
My shutter unit says, give me a break.
Another 500 images... :P
-
I forgot to unscrew the lens for the above results.
Patching iso_regs to allow adjustments up to 3148, just by checking that my code changes worked, I can see that SaturateOffset is not reducing the noise. It fluctuates around the default Canon measured stdev.
The DR increase comes from lowering of the black level (crushing the blacks).
My shutter unit says, give me a break.
Another 500 images... :P
Heh. Mine too :)
# HG changeset patch
# User Audionut
# Date 1393431415 -7200
# Branch unified
# Node ID 4ab57c1f286e8b7338feb162e9998ba37ede3431
# Parent 1ca7d872bdb2f18dc5813d69db474a7f25b74951
Increase SaturateOffset adjustment range
diff -r 1ca7d872bdb2 -r 4ab57c1f286e modules/iso_regs/iso_regs.c
--- a/modules/iso_regs/iso_regs.c Wed Feb 26 06:49:12 2014 +0200
+++ b/modules/iso_regs/iso_regs.c Wed Feb 26 18:16:55 2014 +0200
@@ -513,7 +513,7 @@
.priv = &saturate_offset,
.update = saturate_offset_update,
.min = 0,
- .max = 2000,
+ .max = 3148,
.unit = UNIT_DEC,
.help = "Alters black level and stretches the range, keeping white fixed.",
.help2 = "Decrease to get more highlight details, but watch out RAW zebras.",
-
500D ISO 200
ADTG[7] 0x4046 -> 0x4036
c0F0[819c] 0x66f -> 0x40
c0F0[8034] 0x1991 -> 0x1f11
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs10.postimg.org%2Fbufirr7g9%2FVRAM2.png&hash=9a5934b413c0c6d88eae650e5ba9dd19) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs10.postimg.org%2Fbrvn4x3sp%2FVRAM1.png&hash=8847301b5d1d9725fe09c882acd5efb8)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs10.postimg.org%2Fmij9qlhfd%2FVRAM4.png&hash=0081aba4327a15247047bd28eb552da4)
Vertical banding in the shadows (default vs trick). ACR +5EV
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs2.postimg.org%2F4g28lp9g9%2Fdefault.jpg&hash=40a261b8cdcb1b060e00893c76cfa006) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs2.postimg.org%2Finrxacm55%2Ftrick.jpg&hash=d7eb7b35decdd29a2e7dc63e82e9ec50)
-
Here's a hint about where each amplifier might be in the processing chain. I've tried to estimate how much noise is at the input of each stage and how much is introduced after it.
Theory:
In complete darkness (zero input signal), let's consider this simplified noise behavior for any amplification stage:
output = K * input_noise + output_noise
Assumming the input and output noises are not correlated and Gaussian, assumming 100% linear behavior, and also assumming we can modify the linear gain K somehow (either by turning the amplifier on/off or by adjusting its gain with an integer parameter), we can find out the two noise components from the following linear system by measuring the output noise and the value of the gains (see this (http://www.statlect.com/normal_distribution_linear_combinations.htm)):
K1^2 * var(input_noise) + var(output_noise) = var(output1)
K2^2 * var(input_noise) + var(output_noise) = var(output2)
I will start from ISO 100, consider this setting the reference configuration (gain = 1.0, or 0.0 EV), and will change one amplifier at a time from iso_regs.
Measurements and results:
measured gain (EV) measured OB noise (ADU) input noise (ADU) output noise (ADU)
normalized to ISO 100 noise level
Canon ISO 100 0.00 6.54 NaN NaN
CMOS[0] = 0x773 5.66 26.26 0.50 6.52
CMOS[0] = 0xDD3 5.73 28.00 0.51 6.52
CMOS[0] = 0xFF3 6.47 48.44 0.54 6.52
CMOS[0] = 0xCC3 4.81 16.93 0.56 6.52
CMOS[0] = 0x553 4.79 16.76 0.56 6.52
CMOS[0] = 0x443 3.96 11.15 0.58 6.51
CMOS[0] = 0xEE3 3.52 9.33 0.58 6.51
CMOS[0] = 0xBB3 3.90 10.90 0.59 6.51
CMOS[0] = 0xAA3 2.95 8.15 0.63 6.51
CMOS[0] = 0x333 2.99 8.33 0.65 6.51
CMOS[0] = 0x663 2.52 7.65 0.70 6.50
CMOS[0] = 0x993 1.93 7.04 0.71 6.50
CMOS[0] = 0x883 0.97 6.65 0.72 6.50
CMOS[0] = 0x223 1.95 7.14 0.77 6.49
CMOS[0] = 0x113 0.99 6.70 0.85 6.48
ADTG 0xFE = 3 -0.35 5.30 6.18 2.14
ADTG 0xFE = 0 -0.55 4.70 6.23 2.00
ADTG 0xFE = 7 0.31 8.03 6.36 1.53
ADTG 8/9/A/B 0 -0.36 5.17 6.39 1.39
ADTG 888x +1EV 1.00 13.04 6.51 0.59
ADTG 8/9/A/B 30 -0.17 5.81 6.55 0.00
ADTG 888x -1EV -1.00 3.22 6.57 0.00
ADTG 8/9/A/B 90 0.17 7.41 6.76 0.00
k = 2 .^ gains(i);
A = [1 1 ;
k*k 1];
b = [noise(1); noise(i)] .^ 2;
noises = sqrt(A\b);
input_noises(i) = noises(1);
output_noises(i) = noises(2);
So, it seems clear that ADTG 0xFE is before the other two, but it's not that clear which one from the other two ADTG amps is the first in the image processing chain.
However, ADTG 8/9/A/B is able to pull more highlight detail than ADTG 888x. Since turning 888x all the way down did not bring the other one out of saturation, my conclusion is that ADTG 8/9/A/B is before 888x.
Something like this:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fiso-chain.png&hash=d382fcb33bfd741702c83ed4e2a96c12)
-
Applying that theory @ ISO 100
I can drop [0xFE] to 1 before WL drops giving 11.463 EV.
From here, I can drop ADTG [888x] to -0.04 EV before WL drop, giving 11.545 EV
Or, drop ADTG 8/9/A/B to 46 before WL drop, giving 11.514 EV.
In either case, once 1 of the gains has been optimised, the other can not be reduced without dropping WL.
Using ADTG[0xFE] 1, and ADTG [888x] -0.04, applying Digital Gain 540 increases DR to 11.593
edit: ISO 200 should provide more useful results since here we can be sure of plenty of gain into the ADTG stage.
Here, I can reduce ADTG[0xFE] to 0 before WL drop, giving 11.460 EV
From here, I can drop ADTG [888x] -0.32 EV before WL drop, giving 11.776 EV
Or, drop ADTG 8/9/A/B to 0 before WL drop, giving 11.772 EV.
Again, in either case, I cannot adjust the other without reducing WL.
@ ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.34 EV = 11.679 EV. Cannot reduce ADTG 8/9/A/B without WL drop.
[0xFE] 0 = 11.335 + ADTG 8/9/A/B 0 = 11.641 + ADTG [888x] -0.03 EV = 11.673 EV
[0xFE] 0 = 11.335 + SaturateOffset 1150 = 11.528 + ADTG [888x] -0.34 EV = 11.671 EV. Cannot reduce ADTG 8/9/A/B without WL drop.
[0xFE] 0 = 11.335 + SaturateOffset 1150 = 11.528 + ADTG 8/9/A/B 29 = 11.662. Cannot reduce ADTG [888x] without WL drop.
Looks to me, ADTG [888x] is before ADTG 8/9/A/B
A little digital gain (530) always gives a little DR increase here.
ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.34 EV = 11.679 + digital gain 530 = 11.732 EV.
It would be nice to have more adjustment in ADTG[0xFE], since we can reduce this to 0 (at higher ISOs) and still have adjustment in the other gains.
-
500D ISO 200
I lengthen exposure of 1/2EV with a trick.
Trick vs default :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs3.postimg.org%2F5tw95mu6r%2Ftrick1.jpg&hash=898563add8bc3b8a8e54ab9d0e5e9881) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs3.postimg.org%2F533izurtf%2Fdefault1.jpg&hash=c5048bfad636cab8499eae82282f4267)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs3.postimg.org%2Fca57vq2qb%2Ftrick2.jpg&hash=5bb4cbd47a8f76a566f7d070a54aff2c) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs3.postimg.org%2Fswms4svo3%2Fdefault2.jpg&hash=4f6b16888550907f26bfac1c5f9829ec)
-
A little digital gain (530) always gives a little DR increase here.
ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.35 EV = 11.679 + digital gain 530 = 11.732 EV.
I have an idea about this. Could you estimate the gain by comparing both whitelevels?
-
I have an idea about this. Could you estimate the gain by comparing both whitelevels?
ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV. WL 15283, stdev 4.07
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV. Digital gain 520, WL 15490, stdev 4.09
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV. Digital gain 525, WL 15619, stdev 4.07
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV. Digital gain 530, WL 15748, stdev 4.06
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV. Digital gain 540, WL 16007, stdev 4.11
-
To cut a long story short: This subtle amount of digital gain alters the shape of the noise probability distribution and thus alters the standard deviation.
Long version:
As the data is already quantized for any amount of digital gain there will be gaps (at value v_gap in the histogram (as seen in your shown histograms a few pages earlier) because there is no value v_old to fulfill
Round[gain*v_old]==v_gap
In the following I assume an histogram centered at 2024 at unity digital gain.
Depending on the gap position the retrieved standard deviation may rise. This occurs the number of gaps in the histogram increases. The initial histogram has no gaps (because of unity gain). Increasing the gain increases the number of gaps inside the data. But those gaps lie outside the noise distribution. So the will not affect the computed stdev. Nevertheless the white level rises, so the retrieved dynamic range rises.
For your gain ([0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV. Digital gain 540, WL 16007, stdev 4.11) the first gap is produced within the distribution (at initial value of v_old=2023 -> v_gap=2090). Thus the histogram gets wider and the stdev rises. This bigger stdev is now divided by the white level which did only rise one incremental step, resulting in a decreased Dynamic range.
I think this explanation holds true also for initial histograms that already have gaps, because the additional gaps will appear anyway. I will try to add some graphics.
UPDATE: I am sorry, this only affects the standard deviation of the noise probability distribution. Depending on your calculation of the Dynamic range this may not change the dynamic range at all, if the calculation takes into account the median of the noise's probability distribution.
Maybe these finding at least help to figure out the exact amount of digital gain, as the positions of gaps in a full scale histogram are sharp indicators for digital gain. Maybe those "half gaps" seen in some histograms originate from digital smoothing algorithms or correction matrices?
UPDATE2: This is too obvious. I'm sure it is already stated somewhere in those 21 pages :-[
-
Digital gain, from low values to higher values.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fpxd0nkaqz%2FVRAM0.png&hash=ff8dac481110c57059536d74a7b8706a) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F7ujxwcmm3%2FVRAM1.png&hash=25b2dd863c201daef1e3d128092569db)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fcgg24pfuz%2FVRAM2.png&hash=0760d0476fe52ceb7f5aac193f1a8208) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fhruypf9nf%2FVRAM3.png&hash=b2766b1d4e927d87fefa5371c62367fa)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fttqcjkqln%2FVRAM4.png&hash=6301799d5e0b5fe00d50a76fc418b804) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F6fid7nge3%2FVRAM5.png&hash=4236d81f8ed7ecdfea518e784de2eaed)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fgcte0q3ff%2FVRAM6.png&hash=2af0c822c1a75c83a30644c80eacec1f) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fm0zorm023%2FVRAM7.png&hash=700f81165cf41c0a3a2bfe94082e6e52)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F6fid7o3jf%2FVRAM8.png&hash=c73c401a03fb3a795d962ce9e1d5e4b5)
When it comes to log2(white/stdev), there is a sweet spot. This seems to be around value (530), for lower ISOs.
BTW, 14.5 EV with dual ISO and 11 EV @ ISO 1600 is remarkable. And is better then all other consumer camera afaik.
Heck, ISO 3200 still gives 10.5 EV. :)
-
500D
CMOS[3] 0x810 -> 0xfff
ISO 100 0.015EV
ISO 200 0.062EV
Also cause vertical stripes... :-\
So 500D can be changed safely only in black / white and saturate offset.
Gain 0.33 EV ISO 100
Other changes cause banding.
-
ADTG 8/9/A/B defaults to 85 in movie mode. Reducing it looks to pull back some highlight detail.
-
Finally something that has effect in movie mode :D
My current settings for photo mode (based on the above theory, which basically says bump the gains with low input noise and lower the ones with high input noise):
- ISO 66 from 100: unchanged (see my previous posts).
- ISO 200 - 12800 in Canon menu: ADTG 0xFE = 3 instead of 4, ADTG 8/9/A/B as close to 0 as possible (keeping the deltas between them), ADTG gains -0.16, -0.18, -0.18, -0.19, -0.25, -0.31 and -0.35 EV. Resulting ISOs: 100, 200, 400, 800, 1530, 3000, 5700 (that is, I can effectively pull down all the important ISOs from the next full-stop ISO).
This time I did not measure the ISOs; I've simply used the theoretical model from iso_regs and rounded the ISOs. Feel free to compile raw_diag and iso_regs if you think you can get better measurements.
dr = [ 10.989 10.928 10.799 10.594 10.153 9.521 8.584 7.778 ] ; # measured with raw_diag
isos = [ 100 200 400 800 1600 3200 6400 12800 ] ; # ISOs from Canon menu
dr_tweak = [ 11.589 11.782 11.687 11.463 11.028 10.447 9.704 8.930 ] + ... # measured with raw_diag
[ 0.1 0.1 0.1 0.1 0.1 0 0 0 ] ; # correction for nonlinearity
iso_tweak = [ 66 100 200 400 800 1530 3000 5700 ] ; # estimated with iso_regs, adjusted for nonlinearity
# and rounded (e.g. 218->203->200)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-cmosfe.png&hash=d050c94a405c852bc7c1e10740dbb346) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fiso200from400.png&hash=19c29ef8d7060bfeee6db12537363753)
Note: in the screenshot, the ISO equivalence model assumes linear response; however, the response near highlights is slightly nonlinear. I estimate this nonlinearity helps capturing another 0.1 EV of highlights. Therefore, with these tweaks applied, the equivalent ISO according to DxO definition (clipping point) is nearly 1 stop lower (so the screenshot shows ISO 200 pulled from 400). For the same reason, I've added 0.1 stops to DR measurements where the nonlinear response was noticeable.
This graph does not include the white level optimization for the raw converters or the digital gain correction from the aperture. On 5D3, I estimate this improvement from 0.03 EV (manual lens) to 0.183 EV (f1.4 lens), from the white levels and aperture logs:
log2(15282-2048) - log2(15000-2048) => 0.031
log2(569) - log2(512) => 0.152
Of course, each of these gains is tiny, but when all of them add up to almost 1 full stop, the total improvement should be noticeable.
P.S. Be careful when you say you get 10.5 stops of DR at ISO 3200. After tweaking these parameters, the ISO from Canon menu is no longer relevant, but the equivalent ISO estimated with raw_diag should be good. The linear model from iso_regs should be pretty close too.
-
(...)
@SpcCb: can you share some more details about how you did the math?
(I tried to understand the AMaZE algorithm, but only figured out how to call it; if you can shed some light, I'm interested)
I will try to be simple, I mean I hope I will be.. :)
We can refer to what we discussed in the dual_iso thread (http://www.magiclantern.fm/forum/index.php?topic=7139.msg102924#msg102924) about SVE images to explain the DR in this case:
We can see how is the gain with dual_iso in a perfect world (calculations done @10E-15 & results round @10E-3);
With a single frame:
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB
With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB
-> Wooh, on the paper it is monstrous!
However I suppose you prefer a more physical approach to compare with samples, then from DRSVE we get:
DRdi = log2[(( WL - offset ) / (σSmaxx̄ σSmin) ) (smax / smin)]
Note: we should use the same WL and offset in both frames; because of course _for example_ with a different offset it will generate noise (!).
In my case I used a 5D2;
With a single frame, in the best case and with thermal considerations (short exposure time, up to CMOS @40°C):
DRsf = log2(( 15760 - 1024 ) / 6.47 )
DRsf = 11.153 stops
With dual_iso 100/800:
DRdi = log2[(( 15760 - 1024 ) / √{( 8² + 6.47² ) / 2 }) 64 ]
DRdi = 16.984 stops
Then, now we have basic maths to do projection of our SVE, but be careful it does not take consideration of the PSF + MTF, so the wavelength of the source, etc.
To do that we have to introduce light diffraction model in the equation.
I will not demonstrate the maths here, it's a bit complex and I think not relevant to make projections with samples we use: It requires calibrated sources, lens modelisation, etc.
BTW, in my study with a off-continuum mono-spectral source (λ = 0.656μm, Hα (http://en.wikipedia.org/wiki/H-alpha) source) I recorded an average difference of 3.9% and a sigma difference of 5.0% (both ±10E-1) between single frame image and dual_iso image by using AMaZE both for CR2->DNG & DNG demosaicing.
This mainly because I use 1 of 4px in the Bayer Matrix in this case and there's an intrication between the dual_iso SVE and the Bayer Matrix.
Beside, a similar effect should be observed in regular photography.
About AMaZE, what is very interesting is this algorithm do multiple analyses to proceed to the reconstruction.
There's a (spatial) vector analyse of structures (macro-blocks & special sophisticate diagonal), color/chromic analyse and _in the lasts versions_ a CA analyse, and there's a Nyquist texture (pattern) analyse to make better area interpolations.
IMHO, it is very smart from Emil Martinec _and certainly not foreign to the fact what Emil works in Astrophysics_ because this is well know in our domain what without taking consideration of the light nature it is hard to be close to the reality (even we never match her).
With this algorithm there's a significant SNR improvement face to other reconstruction algorithms; I ever measured +25% on hard cases face to VNG HAD HFFE etc. So compared to the simple mathematical projection where just an average is considered, we should see a gain, even if it's not +2 stops or -50% of sigma.
When we see SNR gain, we see DR gain. So, voilà. :)
Don't hesitate to point me what you don't find clear, I will see how to make it more simple or from a different point of view.
-
P.S. Be careful when you say you get 10.5 stops of DR at ISO 3200. After tweaking these parameters, the ISO from Canon menu is no longer relevant, but the equivalent ISO estimated with raw_diag should be good. The linear model from iso_regs should be pretty close too.
Indeed. The graph clearly shows how there is very little DR gain in extreme ISOs. Simply shifting the ISO point.
Using the settings a1ex listed above, comparing the highlights @ ISO 200.
Images removed.
-
Indeed. The graph clearly shows how there is very little DR gain in extreme ISOs.
This probably just proves that Canon did this right by themselves, they already have superior dr vs. Nikon on higher iso ... and every little gain is a blessing in these situations. Plus since the base dr is lower, a small absolute gain here is a larger relative one :->
-
Using the settings a1ex listed above, comparing the highlights @ ISO 200.
Can you share the EXIF info? Are you sure you compared 200 pulled from 400 vs Canon ISO 200 at the same shutter speed and aperture?
-
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB
With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB
-> Wooh, on the paper it is monstrous!
I'm not familiar with these notations, can you define them first?
In particular, from "ISO 100/800, so e1 = 64e0", I guess e should be the noise variance, assumming equal DR for 100 and 800, and the ISO 800 image darkened to match the ISO 100 one. However, when you say DRSVE = 20log10[16384×64], you are using this 64 as if it were the standard deviation of the noise, so you've got a theoretical 6-stop improvement for 100/800 (which means the noise stdev for the darkened ISO 800 would be 64x lower than the one for ISO 100). Under ideal conditions (equal DR for 800 and 100), the stdev ratio would be 1/8, not 1/64.
From here I'm lost, because the checksum doesn't match.
-
Are you sure you compared 200 pulled from 400 vs Canon ISO 200 at the same shutter speed and aperture?
I was still thinking in terms of relatively small ISO differences. So Canon ISO 200 vs ML ISO 100, they were at the same exposure though.
In essence, the gain is all in the shadows now?
-
If you just turn down some amps starting from some base ISO from Canon menu, the gain will be in highlights.
Since the gain is almost 1 stop, you can now bump Canon ISO one stop and move some of this gain to the shadows (you know the story, you always lose 1 EV of highlights, but you gain back a little less than 1 EV in shadows, and this shadow improvement vanishes at high ISOs).
So, now that we can re-create the full-stop ISOs with different settings (e.g. pull them down from the next higher ISO), we can compare them to the unmodified Canon ISOs. There are two things you should look for:
1) clipping point (should be the same, because this is how ISO is defined - the input signal required to saturate the sensor)
2) noise (our modified ISOs should be cleaner: higher DR => higher SNR)
-
Can you confirm recommended settings for ISO 100 please?
And is it ok to use the CMOS tweak and WL settings inside mini_iso? With ADTG gain set to 0.0?
I have a set of data, but this has the above mentioned tweaks, and I need to redo the test with correct ISO 100 settings.
Canon shutter file WL stdev
100 6 6880 13027 194.8
200 13 6896 13074 204.7
400 25 6909 13221 221.0
800 50 6922 13116 238.1
1600 100 6938 13115 271.1
3200 200 6951 13091 326.0
6400 400 6965 13318 433.8
ML shutter file WL stdev
100 6 6984 14016 180.9
200 13 6995 14106 193.4
400 25 7007 14060 203.5
800 50 7019 13964 223.5
1600 100 7033 13502 247.0
3200 200 7047 13180 203.5
6400 400 7062 12965 383.8
note: that ML ISO is reported as Canon ISO -1 EV. Also, reported WL has not been black subtracted.
-
If you use both modules, there might be some conflicts over these settings; so, to be safe, just use iso_regs and a manual lens.
White level from raw_diag should be 15282 in all cases. There might be exceptions at high ISOs because of hot pixels, where WL might be estimated above 15282 (ignore them).
So:
- unmodified ISO 100
- ISO 200 in Canon menu, ADTG 0xFE = 3, ADTG 8/9/A/B (preamp) = as close to 0 as you can (usually 1 or 2), ADTG gain -0.16 EV (do the math) and ISO estimated in iso_regs around 109. The difference until 100 should be covered by nonlinearity.
- unmodified ISO 200
- ISO 400 in Canon menu, ADTG 0xFE = 3, ADTG 8/9/A/B (preamp) = as close to 0 as you can (usually 1 or 2), ADTG gain -0.18 EV (do the math) and ISO estimated in iso_regs around 218. The difference until 200 should be covered by nonlinearity.
and so on.
The CMOS and black/white tweaks are separate and do not have a noticeable impact over the measured ISOs. You can analyze these ones separately (to make sure their impact is indeed minor) or combined (to check the overall effect of all these tweaks).
-
Lens unscrewed, last stop before full saturation of the green channel.
WL is the average WL measured in the sample patch of one of the green channels. stdev is the standard deviation measured from the sample patch, in the green channel.
Canon shutter file WL stdev
100 15 7316 13255 158.2
200 30 7323 13330 170.6
400 60 7330 13513 185.8
800 125 7337 13414 208.9
1600 250 7344 13471 247.3
3200 500 7351 13607 308.5
6400 1000 7358 13963 419.6
1280 2000 7365 14489 543.4*
1280 2500 7366 12010 523.9
* Max value in channel pushed to saturation.
ML shutter file WL stdev
100 10 7380 14305 158.9*
100 13 7381 12028 144.9
100 15 7382 10071 121.5
200 15 7386 14169 148.7
400 30 7393 14295 157.3
800 60 7400 14208 169.7
1600 125 7407 14176 195.9
3200 250 7414 13702 227.3
6400 500 7421 13627 285.8
1280 1000 7428 13366 370.1
* Last stop before total saturation in channel.
The ISO rating displayed is that in Canon menu, ie: the pulled ISO. The gains used are as above, and for ISO 100 here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103447#msg103447).
CMOS tweak was applied.
If I have my maths right, The noise difference between ML ISO 100 and Canon ISO 100 in the highlights.
log2(14169/13255) - log2(158.2/148.7) = 0.0068 EV
And the exposure difference is log2(14169/13255) = 0.096 EV. Which puts Canon ISO 200 @ ISO 107, subject to any nonlinear behaviour, not captured.
Canon ISO 100 vs Canon pulled ISO 100
log2(15/10) - log2(14305/13255) = 0.4750 EV
Which puts ISO 100 @ ISO 77, again, subject to any nonlinear behaviour, not captured.
Is the noise level difference log2(15/10) - log2(14305/13255) - (158.9/158.2) = 0.529 EV?
Having a quick look at the shadows.
Click thumbnails for full resolution JPGs. These are around 10mb.
Canon ISO 100
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F3lf7tobhn%2FCanon_thumb.jpg&hash=b3aa9627e55088b33a6ec238c218b581) (https://s15.postimg.cc/oibfyc1sr/Canon.jpg)
Canon pulled ISO 200
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fmdr2x9fln%2FML_thumb.jpg&hash=fc7046b0222b2a3d3f3c5f11c71da41c) (https://s15.postimg.cc/tgyyd2fy3/image.jpg)
We can see the read noise is reduced in the Canon pulled ISO, however, the banding noise (probably from the sensor/CMOS amplifiers) has not been reduced.
-
(...)
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB
With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB
(...)
I'm not familiar with these notations, can you define them first?
In particular, from "ISO 100/800, so e1 = 64e0", I guess e should be the noise variance, assumming equal DR for 100 and 800, and the ISO 800 image darkened to match the ISO 100 one. However, when you say DRSVE = 20log10[16384×64], you are using this 64 as if it were the standard deviation of the noise, so you've got a theoretical 6-stop improvement for 100/800 (which means the noise stdev for the darkened ISO 800 would be 64x lower than the one for ISO 100). Under ideal conditions (equal DR for 800 and 100), the stdev ratio would be 1/8, not 1/64.
From here I'm lost, because the checksum doesn't match.
OK, let's see this like that:
emax and emin are the maximum and minimum energy scale (not a correct naming, just to get a picture of this) present in the exposure pattern of the SVE.
We took as example to do maths ISO 100/800 in dual_iso, so we get:
ISO 100 - 200 - 400 - 800 range
Corresponding to:
e3 = 4e2 = 16e1 = 64e0
Remember we are in power of 2 world here, to be in accord with the ADU range (16384). Intercourse between ADU and ISO numbers, this is weird. :)
There's no standard deviation here as we are in the 'perfect world' example; it is just to give us the absolute viewing and to know what are the system limits. I always do it to be sure we will not go outside these limits after, when we add 'more physical' parameters, and to give the most 'simple' figure at the beginning (like an introduction).
-
Lens unscrewed, last stop before full saturation of the green channel.
WL: from your description, this is not white level (you have useful data above it).
Gains: you linked the settings before finding the 0xFE register. Once you have changed this one from 4 to 3, the sweet spot for ADTG gains is now the one from here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104170#msg104170).
Shutter: please use SI units.
Noise: you are probably trying to compute the upper limit of the SNR curve (best-case SNR). Signal is that thing you named WL (assumming you subtracted the black level), noise is stdev. SNR from some more signal levels would be more interesting (the highlights are already clean; most of the noise is in the shadows).
Exposure difference: yes, overall brightness is higher by 0.1 stops. If you measure patches at 1 stop below clipping point, you will not see the nonlinear behavior. If you compare the exact clipping point, you will see it.
You quoted Canon ISO in modified ISOs, which is irrelevant and makes it hard to choose what to compare with what. Moreover, since the equivalent ISOs above 800 are no longer full-stop ISOs, these lines can't be compared directly (you can either plot a graph showing the equivalent ISO vs measured SNR, or you can pull exactly one stop to get ISOs that can be compared directly).
@SpcCb: I'm lost. Where did you get these numbers from? Why ISO 800 has 64x more "energy" than 100? What kind of "energy" is this?
I use the engineering definition of DR: log2(full well capacity / noise RMS), which is log2(white - black) - log2(dark frame stdev), approximated to log2(white - black) - log2(optical black stdev). How does this "energy" fit here?
-
WL: from your description, this is not white level (you have useful data above it).
Correct, it is the average ADU value. This has not been black subtracted.
Gains: you linked the settings before finding the 0xFE register. Once you have changed this one from 4 to 3, the sweet spot for ADTG gains is now the one from here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104170#msg104170).
The gains for Canon Pulled ISO 100 were the ones that I linked (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103447#msg103447). The gains for Canon Pulled ISOs 200-12800 are the ones that you linked (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104170#msg104170).
Because you said,
- ISO 66 from 100: unchanged (see my previous posts).
Noise: you are probably trying to compute the upper limit of the SNR curve (best-case SNR). Signal is that thing you named WL (assumming you subtracted the black level), noise is stdev. SNR from some more signal levels would be more interesting (the highlights are already clean; most of the noise is in the shadows).
No. I am reporting the standard deviation of the measured patch. Simple. If the data is not useful, well, it's better then leaving it out and finding out later, that it would have been useful.
In the images I linked, this is simple. Same shutter, same claimed ISO rating, boost exposure +5 EV in post. I can use my eyes to see this difference, I don't have to rely on my lack of maths skills.
Exposure difference: yes, overall brightness is higher by 0.1 stops. If you measure patches at 1 stop below clipping point, you will not see the nonlinear behavior. If you compare the exact clipping point, you will see it.
I specifically mentioned that any nonlinear behaviour probably wasn't captured.
You quoted Canon ISO in modified ISOs, which is irrelevant and makes it hard to choose what to compare with what. Moreover, since the equivalent ISOs above 800 are no longer full-stop ISOs, these lines can't be compared directly (you can either plot a graph showing the equivalent ISO vs measured SNR, or you can pull exactly one stop to get ISOs that can be compared directly).
Or I could report the Canon ISO that the data was obtained from. Which is what I did, because at the time, it seemed like to most easiest way to describe the results. Certainly, it left no room for error in the maths.
I enjoy helping out in anyway I can, you just need to remember that I am (maths) stupid. I like to visualise.
-
Canon ISO is simply the amplifier configuration you are starting from. You could as well start from ISO 6400 in Canon menu, override CMOS[0] and ADTG gains to the values from ISO 200, an apply some other tweaks until you get the same highlight detail as with ISO 100.
The end result is ISO 100 and you should compare it with Canon ISO 100.
If you don't do this, you end up saying you get e.g. 0.5 stops more DR at ISO 6400, which is simply not true.
-
(...)
@SpcCb: I'm lost. Where did you get these numbers from? Why ISO 800 has 64x more "energy" than 100? What kind of "energy" is this?
Oh my.. I should not used 'energy' to get a picture.
Well, clear your mind. :)
Let's see this path;
emax / emin represent the delta between the maximum and the minimum.
We have taken as example ISO 100/800, so the range:
100 - 200 - 400 - 800
Witch means a delta between each:
1 - 2 - 4 - 6
-> ISO 200 is 2x ISO 100, ISO 400 is 4x ISO 100, etc.
As we are in a power of 2 world, we have the scaling:
21 - 22 - 24 - 26
Then:
1 - 4 - 16 - 64
Hence:
e3 = 4e2 = 16e1 = 64e0
I can't imagine you can't figure that, this is simple boolean maths. ;)
I use the engineering definition of DR: log2(full well capacity / noise RMS), which is log2(white - black) - log2(dark frame stdev), approximated to log2(white - black) - log2(optical black stdev). How does this "energy" fit here?
Note that you can keep the condensate figure:
DR = log2((white - black) / optical black standard deviation)
We do log2 one time, it's faster.
This is close to the equation what I used here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104208#msg104208) to make calculation for a single frame:
DRsf = log2(( 15760 - 1024 ) / 6.47 )
In this equation, extend DR given by dual_iso appears nowhere.
EDIT: Maybe we should discuss of that in the dual_iso thread?
-
As we are in a power of 2 world, we have the scaling:
21 - 22 - 24 - 28
Then:
1 - 4 - 16 - 64
Hence:
e3 = 4e2 = 16e1 = 64e0
Why do we have this scaling?
and 28 is not quite 64 ;)
-
Why do we have this scaling?
and 28 is not quite 64 ;)
Indeed! This is an error; I said I do sometimes... :D
I made the correction.
-
Okay. What's the reason you use these scalings?
Minor: where is that corrected 6 coming from?
-
Okay. What's the reason you use these scalings?
Minor: where is that corrected 6 coming from?
An imaging system with a spatially varying exposure (note that I replace energy by exposure, maybe a better word choice) pattern simultaneously measures local scene radiance I using different exposures. In our example, four exposures range are used such that the maximum exposure e3 measures low scene radiance with high fidelity, while the minimum exposure e0 can measure very high radiance values without saturation.
When information from e0 and e3 are used together, a non-linear quantization of scene radiance is obtained: 1 -> 6.
(my last error coming from here, I was thinking linear; sometimes it appends, I switch :) )
-
Why 6 and why 64?
-
Why 6 and why 64?
100 - 200 - 400 - 800,
1 - 2 - 4 - 6,
21 - 22 - 24 - 26
Then:
1 - 4 - 16 - 64
???
-
Why not 2^7 or not 2^3.14 or any other number? And then, why 2^x?
Remember you are talking about ISO 100/800. Where is the 6 coming from?
Arguments like "we are in a power of 2 world" are not accepted.
If anyone else understands this theory, please enlighten me.
-
We can see the read noise is reduced in the Canon pulled ISO, however, the banding noise (probably from the sensor/CMOS amplifiers) has not been reduced.
However the results are great. Maybe your subject forces the viewers attention towards the banding because the wood on the left side has a structure that is very similar to the banding period. I had a look at the cardbord (former battery package) in 1:1 and could clearly read the whole text - In contrast to the Canon standard photo, where I could barely read the bigger letters.
I think for efficient banding reduction you have to subtract a darkframe. (The question is, how often one has to take a darkframe to compensate for possible changes of the banding)
-
The banding is not really correlated from one shot to another (which means a dark frame subtraction will not help much).
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-fpn-iso100%2Ffpn-canon.png&hash=f72444bdb2a4da61e89466115d9a989e) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-fpn-iso100%2Ffpn-tweak.png&hash=9e3e532279c903043741d390b83903a5)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-fpn-iso100%2Ffpn-xcov-canon.png&hash=f696e8b72b71077fbf9c91882d475bfe) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-fpn-iso100%2Ffpn-xcov-tweak.png&hash=44bfd0a509a7495eb9a19fca015a583a) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F5d3-fpn-iso100%2Ffpn-xcov-tweakvscanon.png&hash=e3a8770b73430a0e6451bc16083c0cc0)
- FPN at ISO 100 (from dark frame, averaged lines/columns)
- FPN at ISO 100 pulled from 200
- FPN cross-covariance between 2 dark frames at ISO 100
- FPN cross-covariance between 2 dark frames at ISO 100 pulled from 200
- FPN cross-covariance between 1 dark frame at ISO 100 (X) and 1 dark frame at ISO 100 pulled from 200 (Y)
So, the absolute amount of FPN is lower with the tweaks applied, but it doesn't decrease as much as the random noise => it becomes more noticeable.
The FPN has some autocorrelation at lags multiple of 16; this hint might help with correcting it in post.
-
Why not 2^7 or not 2^3.14 or any other number? And then, why 2^x?
Remember you are talking about ISO 100/800. Where is the 6 coming from?
(...)
So, if I understand you suppose that there's no spatial cumulative energy in the Bayer Matrix from each ISO (100 & 800) so a linear quantization of scene radiance?
And that emax / emin = 8 for dual_iso 100/800?
If it is, we should get in this case (always with same 5D2 with dual_iso 100/800 to compare):
DRdi = log2[(( 15760 - 1024 ) / √{( 8² + 6.47² ) / 2 }) 8 ]
DRdi = 13.984 stops
Looks a low gain, no? ???
Face to:
DRsf = 11.153 stops
Can you make the maths for your 5D3 and run a physical test to confirm it?
If it is, it will be an interesting theory. BTW maybe it is, we have to stay open (I'm always open for that! :) ).
-
I've asked you to define the terms you are using. What is "spatial cumulative energy" and how does it apply here?
(tried searching and found 2 pdf's, a paper that assumes I know what that thing is, and a thesis about kinetic energy spectrum in the upper ocean)
-
I've asked you to define the terms you are using. What is "spatial cumulative energy" and how does it apply here?
(tried searching and found 2 pdf's, a paper that assumes I know what that thing is, and a thesis about kinetic energy spectrum in the upper ocean)
I mean that we have a SVE image composed by two frames 'interlaced' (I hope this word is correct) in the Bayer Matrix, like the matrix is half cut; in the RGGB pattern the first line RG will be @ISO 100 and the second GB @ISO 800.
So when the demosaicing algorithm will reconstruct the RGB image some of the second line values will be used by the first, first used by the second, etc.
In final, levels should be higher than taking lines discretely.
However, be careful; this is what I read in scientific papers (from Univs, Nikon & Canon) about SVE. I found this relevant but I don't find this myself.
Maybe with dual_iso it is quite different.
Edit: I tried to find papers on the web about this, without great success like you; only some bizarre PDF not in accord. Interesting papers I have was found with the search engine of an University.
-
Canon ISO is simply the amplifier configuration you are starting from. You could as well start from ISO 6400 in Canon menu, override CMOS[0] and ADTG gains to the values from ISO 200, an apply some other tweaks until you get the same highlight detail as with ISO 100.
The end result is ISO 100 and you should compare it with Canon ISO 100.
Personally, I think most people would read it as having been reduced from the Canon ISO. Stock standard, everyday ISO, set via Canon menu.
The ISO rating displayed is that in Canon menu, ie: the pulled ISO.
Rather then some convoluted gain configuration that reaches the same value.
However, your point still stands.
If you don't do this, you end up saying you get e.g. 0.5 stops more DR at ISO 6400, which is simply not true.
I believe they call this, an honest mistake.
SNR from some more signal levels would be more interesting (the highlights are already clean; most of the noise is in the shadows).
If I compare these 2 exposures.
ISO 200 - 1/60s - f/8.0 (lens unscrewed) - ADU 13519 - stdev 187.6
ISO 400 - 1/60s - f/8.0 (lens unscrewed) - ADU 7682 - stdev 92.7
We can see that the pure SNR (http://en.wikipedia.org/wiki/Signal-to-noise_ratio_%28imaging%29) is,
13519/187.6 = 72.063
7682/92.7 = 82.869
To me, this shows that we are not entirely shot noise limited. The same exposure resulted in different noise levels. Whether both exposures are considered clean or not is irrelevant. Clearly, even at strong signal levels, there is a measurable difference. And furthermore, since the 2 exposures were identical, the noise difference is entirely electronic induced. ie: Shot noise is not a contributor.
If we compare to the original example I described.
If I have my maths right, The noise difference between ML ISO 100 and Canon ISO 100 in the highlights.
log2(14169/13255) - log2(158.2/148.7) = 0.0068 EV
We can confirm that the exposures are very similar. At least, if my maths is correct!
14169/148.7 = 95.286
13255/158.2 = 83.786
Even though the noise levels are different.
If I had a full blown lab setup where I could control light output precisely, to capture the nonlinear behaviour near saturation, then life would be all good. In the real world, I'm trying to do the best I can, with what I have. Both physical equipment and mental aptitude.
If my results are far removed from useful, then I am more then happy to stop clicking the shutter count higher and higher.
So, the absolute amount of FPN is lower with the tweaks applied, but it doesn't decrease as much as the random noise => it becomes more noticeable.
Indeed. This is an important distinction. The FPN is not being induced by these gain tweaks. Simply, other sources of noise are being reduced sufficiently, to make this noise more apparent.
Using the 2 images on the previous page, we can conclude that the ADTG stage is a significant contributor to total noise. CMOS amplification was increased, ADTG amplification was decreased, total apparent noise also decreased. We can conclude that the noise is not (significantly) related to components further down the signal chain, because the signal level delivered to these components was very similar.
log2(14169/13255) = 0.096 EV
Here, the nonlinear behaviour in the highlights might effect the noise level in the highlights, but since the rest of the signal is linear, the signal difference between the point I measured, and the shadows, should also be linear. In fact, if I compare a darker region of the 2 exposures.
log2(2722/2651) = 0.038 EV
This shows that as we move further towards the noise floor, the signal level between exposures becomes closer. Further emphasising that the ADTG stage is a significant contributor to total noise.
If we compare the noise in the shadows from 2 identical exposures, 1 @ ISO 100 and 1 @ ISO 1600, we know from plenty of past examples that the apparent noise level reduces.
Now, if we compare the noise from 2 different exposures, ML ISO 100 - 1/15s and ML ISO 1600 - 1/250s, where we haven't used ISO to increase the rendered brightness, but instead, have used ISO to brightness match 2 different exposures, the result is this.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Compare/ML1600.jpg
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Compare/ML.jpg
Here we can see 2 things.
The apparent noise level increased.
The gain configuration for the ADTG was identical. The signal level delivered to components further down the signal chain was very similar.
log2(14169/13702) = 0.048 EV
So we can conclude that the increased noise was from poisson statistics, and CMOS amplification.
We can also see that the FPN has not been (significantly) reduced. The increased shot/CMOS noise has reduced the apparent effect of the FPN, but the FPN level is very similar. I can attempt to use multiple exposures to average out the random noise, leaving the FPN component intact, however, if the FPN noise is not "really correlated from one shot to another", then the level of FPN will also be reduced.
Since the FPN is reduced in identical exposures with CMOS gain, where the signal through the ADTG stage is higher, and the level of FPN does not appear to differ (significantly), with reduced photons on sensor, or, directly with CMOS gain (ie: CMOS doesn't reduce the noise, it simply produces a knock on effect), it is my conclusion that the FPN noise is directly related to the ADTG stage.
This FPN noise may be a result of the scales of the individual ADTG gains. Or it may simply be a fixed component of the ADTG stage at it's noise floor.
Since we reduced the total noise of the ADTG stage with register adjustments, the FPN component of the ADTG stage is now more apparent.
https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html#patternnoise
This gives an indication of how visually disruptive pattern noise can be -- even though the fixed pattern noise is only about 20% of the overall noise, it is quite apparent because our perception is adapted to picking out patterns, finding edges, etc.
If the FPN component can be reduced with register adjustments also, it may be beneficial to sacrifice total noise reduction (my DR is better then your DR), in order to reduce the FPN component. In other words, we reduce (the engineering definition of) DR, but we increase the visual aspect of the output images.
This is what I was trying to explain in the dual_iso thread. Engineering DR has it's uses, but it is useless for describing what is happening between (white) and (stdev).
My first guess (for reducing FPN), would be increasing the gain of a single component in the ADTG stage, ie: 888x or 0xFE or 8/9/A/B, and observing the effects. Then expanding on that by changing the gain of the individual registers in each ADTG gain stage.
Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?" (http://quotationsbook.com/quote/34064/)
;) :P
-
I mean that we have a SVE image composed by two frames 'interlaced' (I hope this word is correct) in the Bayer Matrix, like the matrix is half cut; in the RGGB pattern the first line RG will be @ISO 100 and the second GB @ISO 800.
The RGGB pattern is scanned intact with dual ISO. That is to say, 1 ISO scans an entire RGGB pattern, and the other ISO scans an entire RGGB pattern.
So the interlaced fields are not a single pixel width, but dual pixel width to capture the entire RGGB pattern. a1ex makes specific mention of this in his paper on dual_iso. Page 5 (http://acoutts.com/a1ex/dual_iso.pdf).
I had a look at the cardbord (former battery package) in 1:1 and could clearly read the whole text - In contrast to the Canon standard photo, where I could barely read the bigger letters.
Yes, the reason why I choose low contrast text for the shadows. :)
This area of the image averages -10 EV in the red channel, -9.5 EV in the green channels, and -11 EV in the blue channel, referenced to ISO 100 - 1/15s.
-
The RGGB pattern is scanned intact with dual ISO. That is to say, 1 ISO scans an entire RGGB pattern, and the other ISO scans an entire RGGB pattern.
So the interlaced fields are not a single pixel width, but dual pixel width to capture the entire RGGB pattern. a1ex makes specific mention of this in his paper on dual_iso. Page 5 (http://acoutts.com/a1ex/dual_iso.pdf).
(...)
Haa, OK. I thought it was interlaced every lines. Thank you Audionut to see that. ;)
So for the emax / emin gain, maths I used have to be adapted to this. As I said very first, I presumed that it will not be relevant here (I used 1/4px of the matrix).
Then maybe the emax / emin follow the ISOmax / ISOmin linearly with a minimum gap.
-
Indeed. This is an important distinction. The FPN is not being induced by these gain tweaks. Simply, other sources of noise are being reduced sufficiently, to make this noise more apparent.
Exactly.
And, since it's not correlated from picture to picture, stacking a few pictures (with averaging) will reduce it (because it is very close to a Gaussian random noise applied to each column), but a single dark frame will not reduce it (and might even increase it).
In LiveView, this FPN is corrected on the fly (you can see something like a moving average algorithm that recalibrates: toggle "ISO registers" on and off in movie mode, with LiveView visible behind ML menu, and you'll notice the effect in the first few seconds).
In photo mode, there is probably a similar auto-tuning. Might be worth trying some sensor cleaning routines?
(edit: tried, but didn't help; FPN stdev stays the same)
Personally, I think most people would read it as having been reduced from the Canon ISO. Stock standard, everyday ISO, set via Canon menu.
We are not yet talking about the user interface of this thing (we may as well map ISO 66 -> ISO 50 in Canon menu, ISO 100 -> ISO 100 in Canon menu and so on, but we are not there yet). Right now, at this research stage, the ISO in Canon menu is simply a descriptor for the amplifier configuration (consider it on the same level as any other register value). That is, set Canon ISO to 200, set register X to A, set register Y to B, and the result is ISO 100.
To avoid confusion, I think you can say "ISO 100 pulled from 200". From this, I understand ISO 200 in Canon menu and gains configured in such a way that you recover 1 stop of highlight detail.
Can you make the maths for your 5D3 and run a physical test to confirm it?
Any processing log from cr2hdr is such an experiment: it measures noise levels and DR for the two input exposures from OB area, estimates te output DR based on ISO difference, and at the end it measures the noise level in OB area after all the processing. However, this has two problems:
- the output image also uses some small noise reduction (well, chroma smoothing) => that's why I call it "cooked"
- the noise pattern after processing is no longer a plain old uncorrelated Gaussian
To consider the DR loss caused by the half-resolution in shadows, it may be interesting to downsample the image by 2x2 before comparing the noise levels.
-
Right now, at this research stage, the ISO in Canon menu is simply a descriptor for the amplifier configuration (consider it on the same level as any other register value). That is, set Canon ISO to 200, set register X to A, set register Y to B, and the result is ISO 100.
To avoid confusion, I think you can say "ISO 100 pulled from 200". From this, I understand ISO 200 in Canon menu and gains configured in such a way that you recover 1 stop of highlight detail.
Agreed.
I have a tenancy to call the same thing by multiple names also, which obviously doesn't help. ::)
The next time I have some free time, I will attempt to remove the variable component of the pattern noise (post process), and see if I can use the registers to remove the fixed component.
-
On raw_diag's FPN display it may be worth printing the OB or dark frame stdev too (on the same screen).
So, besides maximizing DR, you may want to minimize the ratio between FPN stdev and overall noise stdev. This ratio would show effectively how much of our noise is FPN.
-
For further analysis yes.
Initially, I think it will be easiest to start at Canon settings, and concentrate on minimum FPN.
Once (if) I have a confirmed list of settings that minimise FPN, I'll leave it for you to correlate the data and draw the conclusions. You're accuracy trumps mine!
-
(...)
To avoid confusion, I think you can say "ISO 100 pulled from 200". From this, I understand ISO 200 in Canon menu and gains configured in such a way that you recover 1 stop of highlight detail.
Any processing log from cr2hdr is such an experiment: it measures noise levels and DR for the two input exposures from OB area, estimates te output DR based on ISO difference, and at the end it measures the noise level in OB area after all the processing. However, this has two problems:
- the output image also uses some small noise reduction (well, chroma smoothing) => that's why I call it "cooked"
- the noise pattern after processing is no longer a plain old uncorrelated Gaussian
To consider the DR loss caused by the half-resolution in shadows, it may be interesting to downsample the image by 2x2 before comparing the noise levels.
I figured that chroma smoothing can be deactivated by cr2hdr command line? Maybe you are speaking about an other chroma smoothing in the process?
I agree for the 2x2 down-sampling, it should help to compare signal levels.
-
These details are incorrect. See correct analysis on the next page.
ADTG2[fe] - This affects RBG channels. What Rawdigger calls Green 2. This scales all channels equally, except for a smaller effect on one of the green channels.
ADTG4[fe] - This affects RGB channels. This scales all channels equally, with a smaller effect on the other green channel.
ADTG2[8882] - Blue channel - pixels 5,6,7
ADTG2[8884] - Blue channel - pixels 1,2,3
ADTG2[8886] - Blue channel - pixels 7,8,9
ADTG2[8888] - Blue channel - pixels 3,4,5
This register predominately effects the blue channel and one of the green channels.
The pixel values listed, show the pixel count from OB on left side of image. That is to say, ADTG2[8882] - Blue channel - pixels 5,6,7, predominately effects the 6th pixel from OB, with a smaller effect on the 5th and 7th pixels. This pattern repeats every 8th pixel. So the same effect happens at the 14th, 15th and 16th pixels from OB.
Also, vertically, the strength varies every other pixel.
edit: I am describing the effect in the RGB render. Consider the raw composites on the next page.
Consider this image.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fu%2F34113196%2FML%2FADTG%2Fblue.PNG&hash=4f6d9a25444172ca419d4cb889f451f9)
Each of the registers also seems to vary with strength. Although I scaled all registers by the bolded digit 0x419. So the varying apparent strength of the register, is probably related to the differences between the stock values.
The same pattern is repeated with registers ADTG2[8,9,A,B]. However, the gain reduction is less.
The same occurs at ADTG4[888?] and ADTG4[8,9,A,B]. However, these registers predominately effect the red channel, with a smaller effect on the green channels.
Also, the pixels effected, follow a slightly different pattern.
ADTG4[8880] - Red channel - 6, 7, 8
ADTG4[8884] - Red channel - 2, 3, 4
ADTG4[8886] - Red channel - 4, 5 ,6
ADTG4[8888] - Red channel - 1, 2
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fu%2F34113196%2FML%2FADTG%2Fred.PNG&hash=711d83679bb2915365b437ed863924de)
raw composites on next page.
-
wouldnt it be better visible in raw composite mode?
-
Yes.
The raw composites of this image.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Ffld8evocr%2Fblue.png&hash=739952e334128a940d862016eb2edf27)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F8i5czadsb%2Fred1.png&hash=837a967d8dccdb014419e7b31e4bc351) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fyqghonq63%2Fgreen1.png&hash=294d41de170629e0b6fc74dcb7bf74ce) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fk7eejs36j%2Fblue1.png&hash=cc22238931b8e9212fcacfc780a33e2f) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F97o5bmywb%2Fgreen2.png&hash=ec116b5008b9b2f524c443957307142c) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fc1wclmmnf%2FComposite.png&hash=512515e8ad381ad8abbb0d1699350e47)
-
500D with ADTG on live view also has a constant vertical banding.
Maybe 500D need new fabric calibration :-\
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs11.postimg.org%2F9q3h8udtf%2F500d.jpg&hash=d6803422fa0298a8fc5f7af667e24f4a)
-
Some further analysis following the bayer pattern (http://acoutts.com/a1ex/dual_iso.pdf).
ab cd ef gh
0 RG RG RG RG
1 gb gb gb gb
2 rg rg rg rg
3 GB GB GB GB
4 RG RG RG RG
5 gb gb gb gb
6 rg rg rg rg
7 GB GB GB GB
8 RG RG RG RG
Column a - row 0 starts at pixel position 122 from left, and 80 from top.
Optical black finishes at 121 pixels from left, and 79 pixels from top. However, pixel row 79 contains some amount of signal.
Column a is ADTG4[8888]
Column b is ADTG2[8884]
Column c is ADTG4[8884]
Column d is ADTG2[8888]
Column e is ADTG4[8886]
Column f is ADTG2[8882]
Column g is ADTG4[8882]
Column h is ADTG2[8886]
8882 looks to only effect a single column. 8884/8886/8888 look to have a predominate effect on the listed column, and a smaller effect on all other columns.
Further details to follow.
ADTG2/4[8,9,A,B] is also more complex.
-
Maybe it can help: OB zone contains multiple Reference Areas.
Here is crops, top-left, at zoom 4x with color marks.
White mark corresponding to the standard output image.
Orange responding like the standard output image but does not get light.
5D2:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_ob_ref-zones.png&hash=d3be6b8e843da74c8672a5445ef14b3d)
5D3:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d3_ob_ref-zones.png&hash=5028ee7f4bce241d831c3a5b9e8a5692)
Note: It needs to be confirmed, however OBRA of the 5D2 should be similar on the 50D and 550D (maybe other from the same generation). And OBRA of the 5D3 should be similar on the 6D.
-
These zones are definitely helpful (I need this data for cr2hdr too). Thinking to add a display mode in raw_diag to show exactly this cropped view, and some statistics for each zone.
Anyone can help me counting the pixels of each zone and write down their sizes?
888x: didn't check offline, but if you try to bump each one in LiveView, you get a brighter vertical line in the Magic Zoom window (and all of them behave in the same way). I believe the order is ADTG2[8882], ADTG4[8882], ADTG2[8884], ADTG4[8884], ADTG2[8886], ADTG4[8886], ADTG2[8888], ADTG4[8888].
At least, if you bump the first 2, you get a thicker vertical, and if you bump the first 4, you get 50% dark and 50% bright stripes.
When checking these gains, it can be useful to turn off demosaicing. For offline analysis, I prefer dcraw -4 -E folllowed by octave.
-
Anyone can help me counting the pixels of each zone and write down their sizes?
I'm not entirely sure what you mean by "zone".
Would it be best to sample the first 8 columns of the image?
Some other useful things, WL for each color channel.
Also, histogram of the top stop, or top two stops of exposure.
-
according to analog devices' manuals for CCD ADTGs, they have a more or less fixed use case.
the CCD readout happens in a serial way - you scan pixel for pixel in every line from top to bottom.
top optical black lines:
usually more than 10 lines to calibrate black level before sensor readout happens.
result is used for optical black clamping (= moving analog "zero" to a specific voltage)
left optical black pixels:
as they come right before the pixels in the current line, they are used to update current line black level for this line.
so now a question raises...
if we use the output of the AD converter which is in those OB areas the output of a closed loop filter,
can we be sure to get something useful from it?
i say "no, not without paying attention and knowing what we do"
why?
check the datasheet of e.-g. AD9992 (http://www.autex.spb.ru/ad/AD9992.pdf) from ADI, which is the manufacturer canon uses for their customized ADCs/ADTGs
there you see that the analog value from the CCD goes into a correlated double sampling path which will measure
sensel's "empty" and "exposed" values and gives out the delta for further processing.
this value then is added to a variable voltage to push "black" a bit higher that 0V. (=black clamp)
thats the black level we know from raw, just in analog domain.
this variable voltage is estimated by averaging OB areas and adjusting it to keep black level at some target value.
in OB digital data we get not just the charge of the sensels, but also the correction value that updates.
as closed loop filters usually oscillate and converge to the target value slowly, the first few DIGITAL OB lines would be trash and only reflect the filter's learning phase.
same for the first few OB pixels in every line.
conclusion:
throw away the first n lines/pixels of OB areas.
how much is n? i dont know, it depends on the loop parameters in the chip.
ADI recommends 20 OB pixels and 10 OB lines so i would say:
-> skip top 10 lines and the leftmost 40 pixels
-
additional note:
check the topmost lines in raw data of above images. it shows the filter learning its black clamp :)
randomly oscillating line brightness, so clamp is updated per-line.
seems to match my theories.
can someone average all pixels in the OB lines? (all pixels, no matter which color)
these values plotted should show a converging oscillation.
(i have to watch cartoons right atm....)
-
can someone average all pixels in the OB lines? (all pixels, no matter which color)
these values plotted should show a converging oscillation.
Dark frames at ISO 100 and 100/1600, 1/30, without any other tweaks applied (I had them saved from the first experiments on dual ISO).
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fob%2Ftop_ob_100.png&hash=d55324a2353fda80b552989593a1e7d2) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fob%2Ftop_ob_100_1600.png&hash=9f4c731fcbe7818c7797c76ba974ba8e)
Octave script:
function top_ob(im, name, outfile)
top = im(1:80,:);
top_ex = im(1:120,:);
figure
plot(mean(top_ex'), 'r'), hold on
plot(mean(top'))
legend('Active area', 'Top OB', 'location', 'southeast')
title(name)
xlabel('Line number')
ylabel('Raw average (per-line)')
set(gca, 'position', [0.18 0.18 0.7 0.7])
grid on
print("-dpng", "-r70", outfile)
end
octave:31> top_ob(im100_1600, "ISO 100/1600", "top_ob_100_1600.png")
octave:32> top_ob(im100, "ISO 100", "top_ob_100.png")
To load the raw data, you need to run "dcraw -4 -E foobar.cr2" first, then load the pgm in octave with imread.
Also note that with dual ISO, in the OB area, the higher ISO always ends up with the lower OB values (more graphs in the dual ISO PDF). I used this trick to autodetect which lines are dark and which are bright in the dual ISO preview experiment (not in cr2hdr); it worked very well, except for 100/200 where the difference was too weak and the algorithm got confused by noise. This behavior should give a hint about how the feedback loop works.
I'm not entirely sure what you mean by "zone".
Those rectangles highlighted by SpcCb. Inside one zone, the noise pattern is the same, but each zone has its own noise pattern (see g3gg0's theory about different clamping modes).
(i have to watch cartoons right atm....)
Good idea :D
-
Some interesting autocorrelations regarding vertical FPN (5D3, ISO 100 pulled from 200):
function plot_xcov(x)
x = xcov(x, 49, 'coeff');
stem(x)
end
# average each column
octave:2> b = mean(im);
# look for autocorrelations
octave:11> plot_xcov(b), print -dpng -r70 0.png
octave:12> plot_xcov(b(1:8:end)), print -dpng -r70 1.png
octave:13> plot_xcov(b(2:8:end)), print -dpng -r70 2.png
octave:14> plot_xcov(b(3:8:end)), print -dpng -r70 3.png
octave:15> plot_xcov(b(4:8:end)), print -dpng -r70 4.png
octave:16> plot_xcov(b(5:8:end)), print -dpng -r70 5.png
octave:17> plot_xcov(b(6:8:end)), print -dpng -r70 6.png
octave:18> plot_xcov(b(7:8:end)), print -dpng -r70 7.png
octave:19> plot_xcov(b(8:8:end)), print -dpng -r70 8.png
# standard deviation of the vertical FPN
octave:22> std(b)
ans = 0.73383
octave:23> std(b(1:8:end))
ans = 0.75365
octave:24> std(b(2:8:end))
ans = 0.75451
octave:25> std(b(3:8:end))
ans = 0.63608
octave:26> std(b(4:8:end))
ans = 0.65589
octave:27> std(b(5:8:end))
ans = 0.73094
octave:28> std(b(6:8:end))
ans = 0.79068
octave:29> std(b(7:8:end))
ans = 0.75369
octave:30> std(b(8:8:end))
ans = 0.69111
# standard deviation of the entire dark frame
octave:35> std(im(:))
ans = 3.1371
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F0.png&hash=19b0ba7d7062e5652e02a842902a54b7) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F1.png&hash=40f980f067ab900a83e3707c21466ad3) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F2.png&hash=f594c0a16026edea373d143f2440da92) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F3.png&hash=a3270003ec564c40cabae5ef12d5b977) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F4.png&hash=0a9cac42280468e43a4db746cadabddc) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F5.png&hash=30cffaf345451640ccdd9ecfa1707c40) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F6.png&hash=b48fed0297aa94fe119719f24ea17aa6) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F7.png&hash=57e4a2b8053f879bd56563e934bfdfd7) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F8.png&hash=12644b953d717339feba55409a2c2afa)
First graph: notice the autocorrelations at lags multiple of 16.
Subsequent graphs: notice the autocorrelations in columns 0, 4, 5 and 6 (modulo 8 ).
Also notice the lower stdev in columns 2, 3 and 7 (modulo 8 ). Not sure if relevant.
Why modulo 8? because there seem to be 8 parallel processing channels (there are 8 gains in the 888x set) and each of these gains affects one column in a group of 8.
You draw the conclusions.
-
My first guess would be that 8882 are masters of some description, but that doesn't explain columns 5 and 6.
Can you compare with Canon default? Or with one of mine (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A7467.CR2)
Canon pulled ISO 200
ADTG gain -16
[fe] - 3
[8,9,A,B] - 2
Mine also contains the faint banding type noise. Does iso_regs scale [8,9,A,B], keeping the deltas between them? If not, then I guess that is the reason why I got the banding noise. Else, a value of 2 is obviously to low for my 5D3.
-
This register seems to associated with constant banding.
500D C0f0[4908]
-
left optical black pixels:
as they come right before the pixels in the current line, they are used to update current line black level for this line.
...
in OB digital data we get not just the charge of the sensels, but also the correction value that updates.
as closed loop filters usually oscillate and converge to the target value slowly, the first few DIGITAL OB lines would be trash and only reflect the filter's learning phase.
same for the first few OB pixels in every line.
Here are some plots that I posted much earlier in this thread here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97510#msg97510), with accompanying explanation here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97529#msg97529), that show the effectiveness of the per-line black-level clamp for a normal exposure and a massive overexposure in the 50D. Each curve on the plot is the histogram of a pair of columns in the left optical-black region, with the top curve adjacent to the actual image region.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Ffld1glcryfcxkki%2Fvariation_0250.png&hash=1238f3eee558e7488c87623a6e464a3a)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fs9y77vl6h1roluk%2Fvariation_0247.png&hash=756471272162b0b3935356384a856c6c)
You can see both oscillation and convergence, even for the normally exposed image.
-
These zones are definitely helpful (I need this data for cr2hdr too). Thinking to add a display mode in raw_diag to show exactly this cropped view, and some statistics for each zone.
Anyone can help me counting the pixels of each zone and write down their sizes?
(...)
For 5D2:
blue: 0;4 5792;26
green: 0;31 158;33
yellow: 158;31 5792;32
red: 0;34 158;37
cyan: 0;37 78;3804
orange: 78;37 158;3804
For 5D3:
blue: 0;5 5918;31
green: 0;32 122;41
yellow: 122;32 5918;41
red: 0;42 122;47
cyan: 0;47 122;56
orange: 0;56 122;3950
Edit: Add 5D3, but for other cameras I don't have RAW files.
Note: Coordinates origin is top-left(0;0).
-
Does iso_regs scale [8,9,A,B], keeping the deltas between them?
Yes. It considers the first one as reference (that one will get the value from menu) and all the others as delta (so, when you set it to 0 or 1 or maybe even 2 or 3, it may underflow and cause strong banding). I preferred this underflow instead of clamping to 0 because in this way you see when you went too far.
However, ADTG gains 888x are not scaled in iso_regs (these are simply set all of them to the same value).
Can you compare with Canon default?
With my ISO 100 dark frame from early dual ISO experiments:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F0.png&hash=a4175d825cea2c27e871b55ff200f8cb) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F1.png&hash=d20b86a08cb9b5330905016a17f9bef0) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F2.png&hash=d857e8c3ba4cee3c407ae92f9571e96f) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F3.png&hash=41c05967578e6fe438d490d2a080634c) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F4.png&hash=14f24b4c94784801056a4a29fc0dcb9a) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F5.png&hash=c374d2b1a24ba8f0a1aff447aaace9eb) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F6.png&hash=cf97f3a5a953d7ee67abbdaa42ae7dbf) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F7.png&hash=31e79086b561802942684f76b501ce6b) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn100%2F8.png&hash=52f3e893c348e4e021c7b7b1618eee37)
Didn't expect this change; need to check a few more dark frames to be sure.
Or with one of mine (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A7467.CR2)
The autocorrelation analysis was performed on a dark frame, not on a regular image.
-
Yes. It considers the first one as reference (that one will get the value from menu) and all the others as delta (so, when you set it to 0 or 1 or maybe even 2 or 3, it may underflow and cause strong banding). I preferred this underflow instead of clamping to 0 because in this way you see when you went too far.
However, ADTG gains 888x are not scaled in iso_regs (these are simply set all of them to the same value).
In this case, I should be safe with a value of 2, since the minimum value for the other registers, is within 2 of the first one.
Also, I was scaling 888x with mini-iso.
I will search for the cause of the problem.
-
Didn't expect this change; need to check a few more dark frames to be sure.
Canon ISO 100 (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A7706.CR2)
Canon pulled ISO 200 (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A7709.CR2)
ADTG gain -16
[fe] 3
[8,9,A,B] 4 (to be safe)
The autocorrelation analysis was performed on a dark frame, not on a regular image.
Sorry, I thought OB would be sufficient.
-
Anyone can help me counting the pixels of each zone and write down their sizes?
For 5D2:
blue: 0;4 5792;26
green: 0;31 158;33
yellow: 158;31 5792;32
red: 0;34 158;37
cyan: 0;37 78;3804
orange: 78;37 158;3804
For 5D3:
blue: 0;5 5918;31
green: 0;32 122;41
yellow: 122;32 5918;41
red: 0;42 122;47
cyan: 0;47 122;56
orange: 0;56 122;3950
Edit: Add 5D3, but for other cameras I don't have RAW files.
Note: Coordinates origin is top-left(0;0).
Don't forget that there can be indexing differences between the CR2 files and the buffer that ML uses (that can be dumped with RAW_DEBUG_DUMP). For example, ML intentionally skips the buffer ahead one line for both the 5D2 and the 50D to put the red pixel on even-parity lines, but there could be other discrepancies too.
-
The DNG buffer can be also dumped with raw_diag (no need to edit the source code; it's in the menu).
I'd say it's a good idea to align the internal DNG buffer to the output CR2 (+/- 1 line/column for forcing the active area starting with a red pixel); should make it easier if one wants to repeat some OB analysis with PC-based tools and compare it to raw_diag.
-
Don't forget that there can be indexing differences between the CR2 files and the buffer that ML uses (that can be dumped with RAW_DEBUG_DUMP). For example, ML intentionally skips the buffer ahead one line for both the 5D2 and the 50D to put the red pixel on even-parity lines, but there could be other discrepancies too.
Of course area coordinates I provided are from full original Canon CR2. ML features like dual_iso or mini_iso was not used.
-
Updated raw_diag to show OB areas on all 4 sides, like this (no zones defined yet):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fob%2Fob-zones-5d3-6400.png&hash=0683d0b209c59d00a52d8c006b69a686)
Interesting thing: the in-camera raw buffer is 16 pixels larger than the CR2 one (as reported by dcraw, exiftool and Adobe DNG), so there seems to be a small part of OB which is not saved (or at least I couldn't find it in the CR2). Note the right margin has 18 pixels, and to get a tight fit I had to use skip_left = 122 (dcraw uses 124), so my best guess is that 2 pixels from the right OB will get shifted back to the left side before saving the CR2, and the remaining 16 are discarded.
Now I'd like to ask you to help with something really easy:
1) grab raw_diag.mo from first post, enable "Optical Black zones", take a picture and post the screenshot (you'll find it under raw_diag directory on your card).
2) if you have some coding skills (or, to be more precise, text editing and command-line skills), fine-tune the photo offsets for your camera in raw.c, like I did right now for 5D3 (https://bitbucket.org/hudson/magic-lantern/commits/aa362087a397); the active area in raw_diag should look (more or less) like in my screenshot.
Exact matching with CR2 is hard because a struct raw_pixblock should be aligned in memory at 16 bits, which means the raw buffer can be shifted horizontally only in multiples of 16 pixels. So, don't strive for that (approximate match is OK).
-
(...)
Now I'd like to ask you to help with something really easy:
1) grab raw_diag.mo from first post, enable "Optical Black zones", take a picture and post the screenshot (you'll find it under raw_diag directory on your card).
2) if you have some coding skills (or, to be more precise, text editing and command-line skills), fine-tune the photo offsets for your camera in raw.c, like I did right now for 5D3 (https://bitbucket.org/hudson/magic-lantern/commits/aa362087a397); the active area in raw_diag should look (more or less) like in my screenshot.
(...)
Here is for 5D2:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2F5d2_ob-zones_raw_diag.png&hash=212bd1d81ac00cc16bae82a80b578b23)
Note: I had a problem with the last raw_diag.mo, get a error with previous ML build (2014-02-04_ad):
Linking..
tcc: error: undefined symbol 'camera_model_short'
[E] failed to link modulesBut it works with the last nighty (2014-03-03).
Beside mini_iso (2014-01-31) no longer works with these last builds, I get the same error.
Looks like it's since 'camera_model_short' became private (?).
-
Here's the optical-black screenshot for the 50D:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fs%2Fn1psoa9p0i7nmz8%2Fob-zones_50D.png&hash=cbef7ed69a13a55658d166b94b67a4b3)
I've double-checked the skip offsets, and skip_top and skip_left are as small as they can be before a gap appears in the raw zebras.
-
@ayshih: it's fine; there may be roundoff errors from raw_get_gray_pixel or the code that uses it. You probably tried to decrease skip_top by 2 and got a top bar in zebras => just leave it like this.
@SpcCb: that colorful palette is likely a bug in the screenshot code (another reason to rewrite it) or in the file copying code. Reproduced it.
Regarding linking errors:
camera_model_short became private because most modules were checking only the camera model name, but not the firmware version. This could cause trouble if somebody loads a module expecting a different firmware version (the upgrades in progress are 5D3 113->123, 7D 203->205 and 700D 111->113). So, to force all modules do proper checking, I made the old way obsolete.
mini_iso is undergoing major rewrite (one of the reasons it's not yet published, besides the research part being in progress and new findings being made). You can apply all these tweaks via ADTG GUI (but with more clicks) and from iso_regs (but that one is 5D3-only) and I want the new module to just work rather than having to fiddle with it (it will be no longer a research tool). I'll refactor 70% of it as a reusable library for patching things around in Canon code/data areas, and publish this one first.
Once I'll publish mini_iso, I expect people trying to port it on the other cameras. I want this to happen after the theory is fully confirmed, documented and working at least on two cameras from different generations. Reasons:
- to make sure the implementation is indeed portable (unlike iso_regs which is hardcoded for 5D3)
- to have some instructions for porting (there are much more addresses and constants to be found, compared to say dual_iso, and some of them can't be found by pattern matching without understanding how things work)
- I don't want to throw away the porting work every time the theory gets changed.
The current research tools from first post should work on current nightly (binaries are up to date).
-
and I want the new module to just work rather than having to fiddle with it (it will be no longer a research tool).
Depending on the complexity with different camera models, a simple on/off, sounds like a good idea.
-
thanks for the plots.
it really looks like i expected it to be. values in the first OB line are either "overexposed" or "underexposed", the whole line too bright or dark.
thats because before reading out the sensor, the black clamp is at a not-perfect value.
then the first line is sampled. all values that go into the ADC get the black clamp voltage added.
as it is too high or low, the output will be too dark or bright.
the ADC will adjust the black clamp voltage (from an DAC) (*) to a value that makes the black region looking the best.
(we dont know the real target values yet)
this process happens in the first few lines, thus they are crap. totally white or black or just b/w noise.
lets call this the OB calibration.
for every line now the ADC uses the OB pixels in front of the real image to re-adjust the OB clamp to match the target value.
there should only be a small drift (i guess only V_adcref swings should need compensation)
so with dual_iso (if i understood alex correct) the ADC will have a lot of to compensate. it might or might not be finished with that adjustment when image data in this line starts.
@alex:
you wrote that with high iso difference the value of the OB areas to the left varies much more.
did you check the variation of the leftmost OB pixels compared to the rightmost OB pixels (on the left side of the image) in dual_iso images?
the rightmost OB pixels in front of the image data could give you a good hint how good the black level was compensated.
all lines should have about the same value there. (*) ...if the values are updated after every pixel though...
if these theories are correct then you might confuse the ADC with dual_iso as it doesnt have enough time to compensate OB.
this maybe just happens under extreme conditons, like a hot sensor or bad battery.
still i think we should know about this fact.
(*)=what i still dont know is if the black clamp is updated after every pixel or after the OB area ends (every line in top OB and on the left OB area when its over)
-
Yes, so the top few lines are the raw (uncorrected) output from the amplifiers. Notice how it repeats every 8 columns.
This number being small, a possible hypothesis is that it's simply not enough to correct the banding because the noise could not be averaged well enough (so finding the register that increases the number of these lines should help). To check this hypothesis, I'd like to see such a screenshot from 6D (which doesn't suffer from FPN).
Regarding dual ISO, here's a screenshot that shows both OB zones (100/6400):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fob%2Fob-zones-5d3-100-6400.png&hash=43619897c2893c07fe3e18af358d116a)
A while ago you found a register named "line count that gets darker" (ADTG2[82F3]). For some reason I've tagged it as NRZI, though I didn't double-check it. So, now I've changed this register from 0x30 NRZI (0x20 decoded) to 0x50 NRZI (0x60 decoded) and the result is:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fob%2Fob-zones-5d3-100-6400-82F3-0x60.png&hash=84170b00fc124fc9f94120e6cc574bf8)
(that is, the OB area is now larger by ~70 pixels)
-
From the pdf g3gg0 linked last page.
Optical Black Clamp
The optical black clamp loop is used to remove residual offsets in the signal chain and to track low frequency variations in the CCD’s black level. During the optical black (shielded) pixel interval on each line, the ADC output is compared with a fixed black level reference, selected by the user in the clamp level register. The value can be programmed between 0 LSB and 255 LSB in 1023 steps. The resulting error signal is filtered to reduce noise; the correction value is applied to the ADC input through a DAC. Normally, the optical black clamp loop is turned on once per horizontal line, but this loop can be updated more slowly to suit a particular application. If external digital clamping is used during postprocessing, the AD9992 optical black clamping can be disabled using Bit D2 in the AFE Register Address 0x00. When the loop is disabled, the clamp level register can still be used to provide fixed offset adjustment.
The CLPOB pulse should be aligned with the CCD’s optical black pixels. It is recommended that the CLPOB pulse duration be at least 20 pixels wide. Shorter pulse widths can be used, but the ability for the loop to track low frequency variations in the black level will be reduced. See the Horizontal Clamping and Blankingsection for timing examples.
-
(...)
@SpcCb: that colorful palette is likely a bug in the screenshot code (another reason to rewrite it) or in the file copying code. Reproduced it.
When I saw that I retried a couple of times, but I thought it was not a problem because we want to check zone sizes.
Regarding linking errors:
camera_model_short became private because most modules were checking only the camera model name, but not the firmware version. This could cause trouble if somebody loads a module expecting a different firmware version (the upgrades in progress are 5D3 113->123, 7D 203->205 and 700D 111->113). So, to force all modules do proper checking, I made the old way obsolete.
(...)
I understood, it is not adequate to only check the camera model, firmware version also need to be check.
So to port mini_iso tool on last nightly builds I thought to update:
if (streq(camera_model_short, "target_model"))with:
if (IS_CAMERA("target_model", "target_firmware"))Should work like that isn't?
I don't use ATDG_GUI actually; I tried to compile ML with adequate config to run it but for the moment I only get wrong compilation and my camera crashes all times on load, I don't see why for the moment and it's a bit freaky to crash camera every times. :P
-
Again: do not rush, be patient.
-
camera_model_short became private because most modules were checking only the camera model name, but not the firmware version.
To "fix" it and continue using the legacy binary, simply un-private the symbol in your local build - you can then be more patient to wait for the result of the current in-depth investigation.
-
Yes, so the top few lines are the raw (uncorrected) output from the amplifiers. Notice how it repeats every 8 columns.
hmm not sure. might even be corrected, but with the wrong black clamp value.
dont remember that the datasheets said to output dummy values during OB areas. have to doublecheck.
i thought these are the *corrected* values, but with a *wrong* black level untuil the internal calibration has no need to update black clamp anymore.
might be updated after every pixel, might also be updated after every line (when the OB state input is toggled) dont know.
the images you posted show in the top OB area how the b/w noise gets a gray noise smoothly. (3rd pixel line?)
(that is, the OB area is now larger by ~70 pixels)
hmmm 0x40 (=64) lines more?
after the original OB area, the brightness goes down a bit?
during OB area we see the same pattern that causes vertical banding on most models?
ever 8 pixels a bit brighter/darker? i see some connection between the two things...
-
The top OB has some correlation with the vertical FPN, but it's fairly weak. I could use that info to reduce its stdev from ~0.72 to ~0.5, but didn't check with a validation data set.
-
weak = not the same amount, but the same pattern?
then its an offset ;)
edit: i mean the same offset that was scaled in these pixels due to black level being wrong.
-
weak =
b = mean(im(80:end,:));
top = mean(im(1:80,:));
t1 = mean(im(5:40,:));
t2 = mean(im(45:80,:));
xcov(top,b) => 0.31
xcov(t1,b) => 0.35
xcov(t2,b) => 0.64
Will add some graphs later.
-
Again: do not rush, be patient.
Don't worry, I'm very.
Au contraire, I find that ML evolutes too fast! (many times I oar to assimilate new possibility :D )
But here it is to get a tool to help: Since last month I run a mod min_iso to observe impact of TV/ISO/ADTG/etc. settings on OBRA.
-
what if:
we have two/four ADC converters with 8 channels each.
every ADC converter has 8 channels, just like the 4 channel one here (http://www.analog.com/en/audiovideo-products/cameracamcorder-analog-front-ends/addi7004/products/product.html)
4x ADTG with 8 channels each: explains the 8 pixel pattern.
-
Do you think this architecture concerns 5D3 only or _some_ other cameras too?
Thought is if we could drive CDS it could help to reduce FPN.
-
In addition to this (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104514#msg104514), maybe it also could help:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.bgimage.fr%2Fpublis_ext%2Fml%2Fcmos_typical-ob-zones.png&hash=43df8cabfc31277a3683ca64524befcd)
Area delimitations are not accurate because it depend of the architecture, but this is the typical setup for CMOS, specially those made for high speed video applications.
The setup looks to what we get on our cameras, apparently.
-
500D
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs24.postimg.org%2Fb18agnscl%2Fob_zones.png&hash=62af217c069c2f436785b8256874a441)
-
In addition to this (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104514#msg104514), maybe it also could help:
my current theory is that these top zones are no digital data.
its just the output of the ADC as it is in its black clamp learning phase.
-
If you refer to the very first rows (4~5) I agree; the device maybe uses these rows to acquire the proper black level (apparently the first rows to be read out).
-
ADTG6[8880] - Lower values
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F96e7i426j%2Flower.jpg&hash=d9e6f42245ebc09d4a4ac713b54dc305)
Higher values
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fmnb60zk7v%2Fhigher.jpg&hash=1fda2dedb70b25e2da497bb1822d54e8)
ADTG6[886a] 0x10
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fj3p8b74nf%2F886a.jpg&hash=fde389abfffea493242281d9c3909659)
A while ago you found a register named "line count that gets darker" (ADTG2[82F3]). For some reason I've tagged it as NRZI, though I didn't double-check it. So, now I've changed this register from 0x30 NRZI (0x20 decoded) to 0x50 NRZI (0x60 decoded) and the result is:
0x6f (decoded)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F6p2gauxpn%2F0x6f.jpg&hash=a10942580eaf9266db979a8e3686cd6e)
0x0
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F5zjnyha0r%2F0x0.jpg&hash=8d3f3739aecd0312578a8892f49db243)
As far as I know, the optical black area is labelled as such, because it doesn't receive photons.
So the image must be digitally shifted?
-
ADTG6[8880] - Lower values
-> target black value the black clamp should be calibrated for?
ADTG6[886a] 0x10
-> number of top lines the ADTG has to interpret as OB area?
-
Latest raw_diag has a tool that renders dark frames as grayscale, without demosaicing, downsampled by averaging from full size to 720x480, which reveals FPN. Gray levels from +/- ob_stdev/4, which maps to roughly +/- 2 averaged_stdev after downsampling.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F100a.png&hash=3354c48e428ee439af7acffffbabb912) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F100b.png&hash=fdeff7771515765ad9e5efcd8e6822bc) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F1600.png&hash=d868dc2a9029cea6ccf1eeef41aa7b79) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F6400.png&hash=05b00d4e154f0b83265e9f2d210bf217) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F100from200.png&hash=1320e06400c52bc344fbe2100808ac0a)
Samples: ISO 100, 100 (second try), 1600, 6400, 100 pulled from 200, all 1/8000. Notice the lack of correlation between the two ISO100 dark frames, and notice how clean ISO 6400 sems to be. However, don't forget these are not absolute noise levels; they simply show the ratio between FPN and Gaussian noise.
Quick theory about why FPN gets revealed by downsampling:
Assume the total noise is a Gaussian noise applied to each pixel + Gaussian FPN applied to each column.
If you downsample the image by 0.125 x 0.125 (that is, 1/8 in each direction), the noise stdevs will be, compared to a 1:1 crop:
sqrt(8x8) = 8 times smaller for the Gaussian noise
sqrt( 8 ) = 2.8 times smaller for the Gaussian FPN
=> Gaussian noise gets averaged out faster than FPN.
Roger Clark has similar samples here: http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html
but I believe they are downsampled by nearest neighbour (Audionut, since you already have contact with him, maybe you can ask?)
In this case (no averaging involved), the stdevs for noise and FPN would be similar to the ones from a 1:1 crop (which could explain why his 1600 sample is cleaner than mine).
-
Roger Clark has similar samples here: http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html
but I believe they are downsampled by nearest neighbour (Audionut, since you already have contact with him, maybe you can ask?)
I thought they were crops.
Table 2a. Apparent Read Noise, Central Image
Central 500 x 300 pixel statistics:
min= 132 electrons
max= 198 electrons
mean= 162 electrons
standard deviation= 2.99 electrons
But I'll ask.
The high frequency components of grain are generally visually pleasing, because they help to increase apparent detail. Averaging removes the high frequency components of the image, leaving only the low to mid frequency components. In this case it's even worse, because all those fine details (noise or otherwise), combine to become lower frequency components.
Averaging increases apparent edge detail. This is excellent on wanted (edge) detail, not so much on detail that isn't wanted. Averaging also increases aliasing, but this isn't a problem with FPN, because the edges are straight.
The maths might say the noise level (of the FPN) was decreased 2.8 times when it was averaged, but it doesn't tell you exactly what happened with the remaining noise. In this case, while the total noise dropped, the remaining noise became more visually destructive.
https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html#patternnoise
This gives an indication of how visually disruptive pattern noise can be -- even though the fixed pattern noise is only about 20% of the overall noise, it is quite apparent because our perception is adapted to picking out patterns, finding edges, etc.
There are a number of algorithms being designed to measure this subjectivity, but afaik, our eyes are still the best determinator.
http://en.wikipedia.org/wiki/Human_Visual_System_Model
For a visual representation of the FPN, the averaging works extremely well. Because it emphasis the FPN (visually).
Since the FPN remains consistent (mainly vertical in this case), do you think you could measure the average brightness in a column (1 pixel wide, or maybe even 2 or 3 pixels wide will be sufficient), and then analyse the data in a row?
By measuring the data vertically, you should end up with a data set like so.
1, 3, 5, 1, 1, 5, 1, 9, 3, 6, 7
Since the Gaussian component of the noise should hover around a median (equal in each vertical measurement set), the data set should accurately represent the FPN component?
-
The maths might say the noise level (of the FPN) was decreased 2.8 times when it was averaged, but it doesn't tell you exactly what happened with the remaining noise. In this case, while the total noise dropped, the remaining noise became more visually destructive.
It does - the random component decreased 8 times. That's why the FPN became more destructive.
The ratio between FPN stdev and overall noise stdev should be a good indicator of how badly the FPN affects the image.
Since the FPN remains consistent (mainly vertical in this case), do you think you could measure the average brightness in a column (1 pixel wide, or maybe even 2 or 3 pixels wide will be sufficient), and then look at this data horizontally?
If I understand the question well, the FPN analysis from raw_diag does exactly this.
If you subtract the two FPN components, estimated by averaging, you end up with this (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96465#msg96465).
Problem: you can't do that on a dark frame and then use that data to correct a useful image, because the FPN is not correlated between images (well, the autocorrelation is extremely weak, as you can see from the "xcov" analysis in raw_diag).
-
i believe your theory about the missing corellation of FPN between shots.the shots show that.
but. how do we get a FPN that is horizontally and vertically repeating in the single image?
if you average the top OB area, you will get the vertical FPN, right?
now subtract that from the whole image and do the same for horizontal FPN using the left OB area.
also subtract that noise frome very column and the low frequency part of the noise should be gone?
-
if you average the top OB area, you will get the vertical FPN, right?
Nope, answered here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104758#msg104758).
You do get an improvement, but it's small.
-
Latest raw_diag has a tool that renders dark frames as grayscale, without demosaicing, downsampled by averaging from full size to 720x480, which reveals FPN. Gray levels from +/- ob_stdev/4, which maps to roughly +/- 2 averaged_stdev after downsampling.
Samples: ISO 100, 100 (second try), 1600, 6400, 100 pulled from 200, all 1/8000. Notice the lack of correlation between the two ISO100 dark frames, and notice how clean ISO 6400 sems to be. However, don't forget these are not absolute noise levels; they simply show the ratio between FPN and Gaussian noise.
(...)
Wohh, you get a strong corners illumination @ISO 1600 & 6400!
'Hope this is not a electromagnetic field effect as we got on old camera models (350D etc.).
(...)
The high frequency components of grain are generally visually pleasing, because they help to increase apparent detail.
(...)
I agree. It is an interesting psycho visual effect.
Plus, it help to dither level banding effects in image with large gradients or pseudo flat areas.
-
Nope, answered here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104758#msg104758).
You do get an improvement, but it's small.
A low covariance is nothing special if there is a lot of Gaussian noise evident. The fact that there is some covariance tempted me to try a FPN removal.
The improvement in terms of dynamic range my be small. Nevertheless the improvement in visual quality can be significant.
But judge by yourself:
I took your sample ISO 200 pulled to 100 and split it in the top 80 and the lower 400 rows. Below you see the lower 400 without processing:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fnothing_removed.png&hash=45cad77a09d7fa93d0542149861479f5)
Then I did what g3gg0 just suggested. An vertical average (let's call it top) of the top 80. Below you see the result of a (floating point) subtraction of this median each line of the image pixels. After the subtraction I added Mean(top) to each pixel value to not shift the blacklevel and rounded the result to yield the new pixel values:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fjust_horizontal_removed.png&hash=6af20071581d85ee00b77209da2a4e1e)
Now the same done in the vertical direction:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fvertical%26amp%3Bhorizontal_removed.png&hash=d62d7e6dc05fbb8be4528e37689c6266)
My subjective opinion is that removing the FPN can improve the visual quality significantly.
Please note that this test is just a quick&dirty one so there my be some errors induced during re-export of the data to .png format. Plus the input data is not optimal quantized as the histogram shows strong clipping at the borders:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fhist.png&hash=17e609c92b522a33363ecc4b340112d7)
(some typos corrected)
-
top 80 and the lower 400 rows
=> you have assumed a OB as large as 658 pixels ;)
-
You're right, in my euphoria I missed the fact that your averaging provided me with lower Gaussian noise component / virtually increased OB.
I just took the values you used in your Reply #584.
I repeated my test with 10pixels (which should be about 80 pixels of OB), resulting in slight (but to my eye pleasing) vertical FPN reduction.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fjust_horizontal_removed%28OB%3D80%29.png&hash=9db36d6ee28534e2a4cfcc8cc0712b72)
The effect on horizontal FPN is negligible.
Below the reduction in mean horizontal pixel values distribution for different (virtual) OB sizes:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fhist%28OB%3D40%29.png&hash=3f8c21bb1bc0baa2e8991bd7e5f8d995)(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fhist%28OB%3D80%29.png&hash=8d00328ff7cb9b6e4220dc1b793b349a)(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fhist%28OB%3D160%29.png&hash=4a17e37da749dd85a43531d47fb7a884)
It is apparent that for OB=40 the FPN may even get worse.
I hope with this post my tests contribute at least a little for those, who are not such familiar with FPN...
-
function ob_corr_plot(file)
src_image=imread(file);
%{
average some top areas:
1st: black clamp learning area, 4 through 28
2nd: OB area that is just black, which is 42 to 79 on 5D3
3rd: the top 500 pixels ofthe image data (dark frame) starting at 80
4th: the whole dark frame
%}
vert_noise_1=avg_top(src_image, 4, 28);
vert_noise_2=avg_top(src_image, 44, 38);
vert_noise_3=avg_top(src_image, 80, 500);
vert_noise_4=avg_top(src_image, 80, size(src_image)-80);
%{
now cut the left OB area and smooth over 256 pixels.
should be gaussian, but for some tests it might be enough.
%}
smooth_1=smooth(vert_noise_1(122:1:end), 256);
smooth_2=smooth(vert_noise_2(122:1:end), 256);
smooth_3=smooth(vert_noise_3(122:1:end), 256);
smooth_4=smooth(vert_noise_4(122:1:end), 256);
% show correlation %
corr(smooth_2, smooth_3)
figure('name', 'Black clamp area');
subplot(3,1,1);
plot(smooth_1);
title('Black clamp area profile');
subplot(3,1,2);
plot(smooth_2);
title('Optical black area profile');
subplot(3,1,3);
plot(smooth_1, smooth_2);
title('X/Y plot');
figure('name', 'OB vs. Dark frame - top 500');
subplot(3,1,1);
plot(smooth_2);
title('Optical black area profile');
subplot(3,1,2);
plot(smooth_3);
title('Dark frame - top 500');
subplot(3,1,3);
plot(smooth_2, smooth_3);
title('X/Y plot');
figure('name', 'OB vs. Dark frame - full');
subplot(3,1,1);
plot(smooth_2);
title('Optical black area profile');
subplot(3,1,2);
plot(smooth_4);
title('Dark frame - top 500');
subplot(3,1,3);
plot(smooth_2, smooth_4);
title('X/Y plot');
end
function ret = smooth(data, fact)
for pos = 1:size(data)-fact
ret(pos) = mean(data(pos : 1 : pos + fact));
end
end
function ret = downsample(data, fact)
for pos = 1:size(data)/fact
ret(pos) = mean(data(((pos-1)*fact) + 1 : 1 : ((pos)*fact)));
end
end
function ret = avg_top(src_image, start_idx, lines)
part = src_image(start_idx:1:start_idx+lines, :);
ret = sum(part) ./ size(part, 1);
ret = transpose(ret);
end
% draw plots for a dark frame %
ob_corr_plot('D:\Users\g3gg0\Pictures\Kamera\2014_03_05\NO8A3083.pgm');
pause;
which produced this plot and a corr() of 0.95883 for the noise and some data from the top OB area.
dont use the topmost 44 lines. use line 44 to 79 and average them.
does it work better?
http://upload.g3gg0.de/pub_files/16261ae822934c9175945a5e34cba116/plot.png
-
My subjective opinion is that removing the FPN can improvement in visual quality significantly.
For raw video in less than lit conditions its the difference between usable shots and garbage...
-
It does - the random component decreased 8 times. That's why the FPN became more destructive.
The ratio between FPN stdev and overall noise stdev should be a good indicator of how badly the FPN affects the image.
I was attempting to discuss the visual aspect of the FPN itself. So we can say, the Gaussian noise reduced 8x, and the FPN reduced 2x, the destructive nature of the FPN became more apparent, because it's percentage in the image, increased as a result.
For large changes in the percentages, this should be an accurate subjective measurement.
But we don't know exactly what happened to the FPN. Since stdev is basically an average, it doesn't (accurately) describe what happened on the edges of the FPN. Did the edge contrast of the FPN increase? See bottom.
If I understand the question well, the FPN analysis from raw_diag does exactly this.
Excellent. I was misunderstanding how to determine its results earlier.
If you subtract the two FPN components, estimated by averaging, you end up with this (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96465#msg96465).
Problem: you can't do that on a dark frame and then use that data to correct a useful image, because the FPN is not correlated between images (well, the autocorrelation is extremely weak, as you can see from the "xcov" analysis in raw_diag).
For post processing, the correlation is important. In terms of, adjust this register, did FPN reduce or increase? It's not so important.
I repeated my test with 10pixels (which should be about 80 pixels of OB), resulting in slight (but to my eye pleasing) vertical FPN reduction.
The effect on horizontal FPN is negligible.
It softens the edge detail of the FPN. This is important, because contrast is a determining factor in subjectivity.
http://en.wikipedia.org/wiki/Contrast-to-noise_ratio
It is the contrast of this FPN, that makes it a noise component of images.
Let's consider the images at reduced size to emphasise the edge detail.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn%2F100from200.png&hash=1320e06400c52bc344fbe2100808ac0a) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naturalsound.de%2FMagicLantern%2FCMOS%2Fjust_horizontal_removed%2528OB%3D80%2529.png&hash=5e7929e961e2cb0827b56a2783ec970f)
Because these images are made up of only 3 components, FPN, Gaussian, and black, the differences in the images might not be considered overly great.
However, with the forth component present in the images, wanted detail, the difference would be subjectively greater. Where we reduce edge detail/contrast of the noise, the edge detail/contrast of the wanted detail becomes more predominant.
The contrast ratio of wanted detail becomes greater then the contrast ratio of the noise.
-
I've been following this thread and please forgive me if the answer is present somewhere but I was curious if this increase in dynamic range will be applicable to h264 videos. If so, this development combined with the recent bitrate hack being investigated here http://www.magiclantern.fm/forum/index.php?topic=4124.0 is really exciting for 5d3 owners who want to shoot h264 for a project.
-
I have had a quick play with it here (http://www.magiclantern.fm/forum/index.php?topic=4124.msg104972#msg104972), and I could see extra highlight detail when adjusting the registers.
-
Just a question about calculating capability inside the camera (DIGIC I presume):
Will it be too much to imagine calculate the vertical 2D FFT harmonics of 'some' (dozen?) rows just top of the active area but in the OB, done the same process with OB area on left to find the horizontal 2D FFT harmonics, then do the invert FFT from this on the whole active area?
a1ex, you spoke about seconds for the FPN algorithm reduction; 2D FFT operations look faster to do, what do you think by doing this?
-
Just take a look at how fast raw_diag is when processing the entire image (try the dark frame analyses).
-
It is apparent that for OB=40 the FPN may even get worse.
Even if you take a single line from top OB, as long as there is some weak correlation between that line and the FPN, you can still use it to reduce the FPN. The key is not to subtract it blindly, just give it a lower weight.
How much?
Let's look at the Kalman filter theory: http://robocup.mi.fu-berlin.de/buch/kalman.pdf
From page 3, if we know how noisy our estimations are, the optimal weights are inversely proportional with the noise variances. For averaging 2 random variables x1 and x2, the optimal weights can be simplified to var(x2) and var(x1):
x_optimal = (x1 * var(x2) + x2 * var(x1)) / (var(x1) + var(x2))
where var(x) = std(x)^2.
So, if we average a single line from the top OB, we need to know how noisy is that line and how bad our FPN is. Assumming Gaussian noise with variance vg = sg^2 + Gaussian FPN with variance vf = sf^2, our OB line estimates the FPN with a variance equal to vg. If we simply assume the FPN is zero, the error is a Gaussian with variance vf. Combining these two estimations optimally means simply downweighting the top OB line:
fpn_estimated_from_one_line = vf / (vf + vg) * that line + vg / (vf + vg) * 0
If we average n lines to estimate the FPN better, its error will have a variance equal to vg/n; that is, the stdev of our estimation will be sg/sqrt(n). To get minimal variance in the corrected FPN, the optimal weighting is now:
weight_avg = vf / (vf + vg/n)
weight_fpn = (vg/n) / (vf + vg/n).
fpn_estimated_from_n_lines = weight_avg * mean(n_lines) + weight_fpn * 0
What will be the variance of our corrected FPN?
var_combined = var_avg * weight_avg^2 + var_fpn * weight_fpn^2
where var_avg = vg/n and var_fpn = vf.
=> var_combined = vg/n * vf^2 / (vf + vg/n)^2 + vf * (vg/n)^2 / (vf + vg/n)^2
=> var_combined = (vg/n * vf^2 + vf * (vg/n)^2) / (vf + vg/n)^2
=> std_combined = sqrt(var_combined)
Now let's check the theory.
Let's say we have a 720x480 dark frame that contains a random noise of stdev sg=0.6 and a vertical FPN of stdev sf=0.25 (values typical for a downsampled ISO 100 pulled from 200).
sg = 0.6; # Noise stdev
sf = 0.25; # FPN stdev
vg = sg^2; vf = sf^2; # Variances
G = randn(480,720) * sg; # generate Gaussian noise
f = randn(1,720) * sf; F = ones(480,1) * f; # generate Gaussian FPN
GF = G + F; # combine both noises => ideal dark frame
imshow(GF, [])
print -dpng -r60 ideal-dark.png
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn-ideal%2Fideal-dark.png&hash=7afb6d6e2e2909421e12592d19cfd807)
# try to average n lines
for n = 1:80
fe = mean(GF(1:n,:)) * vf / (vf + vg/n); # FPN estimated and weighted optimally
S(n) = std(f-fe); # experimental stdev of corrected FPN
SI(n) = sqrt((vg/n * vf^2 + vf * (vg/n)^2) / (vf+vg/n)^2); # theoretical stdev of corrected FPN
SR(n) = std(f - mean(GF(1:n,:) * (rand*2))); # try some other random weights between 0 and 2
end
plot(SI/sf, 'r-o', 'markersize', 2), hold on
plot(S/sf, 'b-o', 'markersize', 2)
plot(SR/sf, 'kx', 'markersize', 2)
axis([0 80 0 1]), grid on
set(gca, 'ytick', 0:0.25:1);
set(gca, 'position', [0.2 0.15 0.7 0.7])
legend('Theoretical improvement', 'Simulated improvement', 'Random weighting', 'location', 'southwest')
title('FPN improvement in ideal conditions')
xlabel('Number of lines averaged')
ylabel('stdev(corrected FPN) / stdev(original FPN)')
print -dpng -r70 ideal-fpn-improvement.png
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn-ideal%2Fideal-fpn-improvement.png&hash=dcaf2e6c98e4e466c94c120b97fa1b3c)
So, this graph shows:
1) this weighting is indeed optimal (all the random weights resulted in worse improvement)
2) we have an upper bound for the improvement that can be obtained by averaging some lines
=> to reduce the FPN component by 1 stop, we need to average 17 lines out of 480 for this numerical example.
How much is 1 stop of FPN improvement? (note the Gaussian noise is not touched)
fe = mean(GF(1:17,:)) * vf / (vf + vg/n); # FPN estimated from 17 lines and weighted optimally
std(f-fe) / std(f) # experimental improvement for FPN
ans = 0.54244
FE = ones(480,1) * fe; # extend the correction to the entire image
clf, imshow(GF-FE, []) # subtract the estimated FPN
print -dpng -r60 ideal-corrected-1ev.png
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Ffpn-ideal%2Fideal-corrected-1ev.png&hash=a9b31a99e1d36cbd3d5dc88832f68056)
To be continued; feel free to try to apply this theory to a practical example.
-
@alex:
today i sat in front of matlab with a colleague and we tried to get some more information from the top OB area.
he had the idea to do a singular value decomposition for pattern recognition. we implemented it based on the example image generator and built a script for it.
the result was visually pleasing - could you check if it is improving things? thanks :)
sg = 0.6;
sf = 0.25;
vg = sg^2; vf = sf^2;
G = randn(480,720) * sg;
f = randn(1,720) * sf; F = ones(480,1) * f;
GF = G + F;
% the error seems to be with main energy in 1st order %
max_order = 1;
num_plots = 1+max_order;
% plot original image %
figure;
subplot(num_plots,1,1);
imagesc(GF);
% only use top 40 pixels for correction %
top = GF(1:40,:);
top_mean_free = top-ones(size(top))*mean(mean(top));
% run a singular value decomposition (major component analysis) %
[A,B,C] = svd(top_mean_free');
% start correction %
GF2=GF';
for order=1:max_order
% get deviation pattern of n-th order %
dev = A(:,order);
% scale pattern with stddev of this pattern %
corr_vec = sqrt(B(order,order)) * dev;
% remove mean to make it mean-free %
corr_vec_mean_free = corr_vec - ones(size(corr_vec)) * mean(corr_vec);
% to detect the pattern polarity we do a projection of our pattern onto our top OB area and check its mean value %
dir = sign(mean(corr_vec_mean_free' * top_mean_free'));
% now we can add/subtract that pattern from image data %
GF2 = bsxfun(@minus,GF2,corr_vec_mean_free * dir);
% and show the result %
subplot(num_plots,1,1+order);
imagesc(GF2');
end
-
It simply finds the same optimal solution with a different method.
Result from your program:
std(mean(GF2')) / sf
ans =
0.3525
My theoretical result for n=40:
n=40;
sqrt((vg/n * vf^2 + vf * (vg/n)^2) / (vf+vg/n)^2) / sf
ans =
0.3548
In this ideal example, your principal component analysis summarized the top 40 lines into a single one (which is to be expected, because the signal we are looking for in these lines - the FPN - is the same in all these lines, and anything else besides this FPN is just Gaussian noise). What would be interesting is to try this one on a real OB. There I expect surprises (probably pleasant ones).
-
Updated adtg_gui: I've added a routine that dumps all the registers into a log file after taking a picture (this is what I've used to make the funky graphs). I'd like to get these graphs for all cameras, so I need your help
Ok, here you go for 6d: https://bitbucket.org/Marsu42/ml-aiso/downloads/adtd-logs-6d.zip
On another note: The above research is absolutely impressive and all and no doubt this will result in an optimal usage of the Canon sensor, even though the complicated re may stall the work from time to time. But, speaking with my simple user's hat on...
... I would be most grateful if we'd get a working, but not optimal or final mini_iso.mo for latest core and with working white point adjustment for 6d. This is because every day w/o this, I'm losing some dr in my shots, and this is sad to know :-o
-
Well, the next step towards mini_iso is a library for patching things around in Canon code. Around 70% of it is mini_iso code refactored to be generic, and the rest is a small GUI frontend to see what exactly got patched.
That's because mini_iso needs to patch a LOT of memory addresses, and I want to do it cleanly, with sanity checks before patching, integrity checks at runtime, and, very important to me, undo all patches when you disable the menu. This doesn't happen with iso_regs.
This could be reused for a cleaner implementation of things like bitrate hacks from Tragic Lantern, since this library would help with all the dirty stuff like how to undo the patches, when to enable the cache hacks, and will also let the user what hacks are enabled at any given time. When shooting something a little more serious, I want to know for sure what patches are applied, and that stuff turned off from menu is indeed turned off.
In my local copy I already refactored dual ISO and raw_rec/mlv_rec patches to be applied via this library.
-
In my local copy I already refactored dual ISO and raw_rec/mlv_rec patches to be applied via this library.
Sounds great and I'm a big fan of safe and clean code (because even as a module coder, I sometimes have to look at it :-)).
I just wanted to add my 2ct not to let perfectionism stand in the way of pragmatism for a working mini_iso, just like Linus Torvalds always says about Linux I feel ML is not a research project, but made to be actually used :-o
-
And exactly for this reason I want mini_iso to just work.
-
Im not a coder by any means but I liken the finely tuned modules to a song and artist is writing. He or she wont release it until all of it is perfect and flows just right. These modules/code are the coders babies, with all the work they put in to them Im sure they want them to be flawless before they are public or beta ect. But its hard for us non coders to be patient with such incredible features. Cant wait for this mod!!!
-
Btw when this code goes public, it would be very nice to have an external function to get the optimal overexposure ec from the mini_iso calibration. This way autoexposers like autoexpo or auto_iso that work in m mode can automatically modify the exposure accordingly.
int get_mini_iso_calibration(int iso_canon8)
{
int ec_canon8;
...
return ec_canon8; // ec value in canon steps (0/3/4/5/8)
}
-
ADTG[7] 0x4046 -> 0x4036
It is darken the image by 0.5 EV 500D.
White level and black level remains unchanged.
Other records ADTG change the white level or black level.
-
I havent been here for a while, what is current status of the video mode on the 5d3? Is it still 0,1 EV (much less than in stills mode) like in the description on the first page or maybe there was some improvement in this matter?
-
I'm pretty sure I was seeing some good changes with the latest iso_regs module. But there didn't appear to be an easy way to see the OB area of raw video, and I don't shoot video, so I gave up very easily. ;)
If someone points out how to see the OB area of raw video, I'll take another look.
-
I was pondering something today.
Canon ISO 100/1600
Input file : _46a0505.cr2
Camera : Canon EOS 5D Mark III
White levels : 15179 13789
Noise levels : 11.05 6.48 6.54 10.98 (14-bit)
ISO difference : 3.95 EV (1550)
Dynamic range : 10.99 (+) 10.05 => 14.01 EV (in theory)
Semi-overexposed: 0.89%
Deep shadows : 96.88%
ISO overlap : 4.0 EV (approx)
Noise level : 37.21 (20-bit), ideally 37.22
Dynamic range : 14.46 EV (cooked)
ML ISO 100/800
Input file : _46a0504.cr2
Camera : Canon EOS 5D Mark III
White levels : 15645 14250
Noise levels : 6.49 3.95 3.96 6.40 (14-bit)
ISO difference : 3.00 EV (799)
Dynamic range : 11.75 (+) 10.88 => 13.87 EV (in theory)
Semi-overexposed: 0.82%
Deep shadows : 95.91%
ISO overlap : 5.8 EV (approx)
Noise level : 42.41 (20-bit), ideally 42.41
Dynamic range : 14.32 EV (cooked)
ML ISO 100, is actually Canon ISO 200, with the highlight saturation point adjusted. In the midtones/shadows, it behaves exactly like Canon ISO 200, but the rated ISO gets shifted down to ISO 100, as per DxO's definition (http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity). This is how the DR increases with ML ISO. The midtones/shadows stay the same, but it increases highlight capturing ability.
ML ISO 800, is actually Canon ISO 1600, with the highlight saturation point adjusted. So, the shadow point of ML ISO 800, is actually equal to Canon ISO 1600.
The DR of ML ISO 100, is 0.8 EV greater then Canon ISO 100.
So, we gain 0.8 EV of midtone overlap, from the cleaner ISO, and we gain another 1 EV of midtone overlap, because we are only 3 stops apart (Canon ISO 200/1600), instead of 4 stops apart (Canon ISO 100/1600).
-
Well, the next step towards mini_iso is a library for patching things around in Canon code.
The first step was made: https://bitbucket.org/hudson/magic-lantern/pull-request/476/experimental-library-for-managing-memory
As a second pre-requisite, I would also like a tool for pre-processing raw files - in this context, I need it to reduce the FPN. Without it, the noise improvement will have little practical value if the FPN will stay at the old levels. And since I wasn't able to find a way to fix the FPN in camera, the quickest solution would be a software tool similar to cr2hdr (which will most likely output a DNG file).
Advantage: the FPN fix is also needed for some regular ISOs, for example here (http://www.magiclantern.fm/forum/index.php?topic=9564), so the same tool could do the job in both cases.
-
The first step was made: https://bitbucket.org/hudson/magic-lantern/pull-request/476/experimental-library-for-managing-memory
Great there's progress on this!
As a second pre-requisite, I would also like a tool for pre-processing raw files - in this context, I need it to reduce the FPN. Without it, the noise improvement will have little practical value if the FPN will stay at the old levels. And since I wasn't able to find a way to fix the FPN in camera, the quickest solution would be a software tool similar to cr2hdr (which will most likely output a DNG file).
Doh, I believe you this cannot be done in camera, but another cr2hdr-like stop adds a lot of overhead - it would be important that this at least a lossless conversion so the original cr2 can be deleted instead of dual_iso double storage space requirements. I'd also like to note that ending up with dng instead of cr2 makes use of dxo's prime noise reduction impossible, that's why I recently switched from dng to preserving cr2 again. If it's possible, an option to patch the original cr2 would be welcome.
-
Mars did some good progress with faking the CR2 data, so CR2 output is not excluded.
I can try to implement the FPN fix in the camera too, but speed will be a problem.
-
Mars did some good progress with faking the CR2 data, so CR2 output is not excluded.
Interesting to know, thanks for the information - btw keeping cr2 would improve the general acceptance of the ml dr improvement, still lots of folks out there who aren't comfortable with dng.
Last not least, another reason for cr2 is that except ~+4mb/shot it has no drawbacks in LR/ACR unless you want to use dng-only features (embedded color profile, lossy compresssion) - but saving metadata is much faster since it only writes the sidecar xmp.
-
but another cr2hdr-like stop adds a lot of overhead
Only if you want the FPN correction. Otherwise, it's just like any other Canon file, with reduced read noise.
-
Only if you want the FPN correction. Otherwise, it's just like any other Canon file, with reduced read noise.
Btw this doesn't concern 6d users, I've never ever seen any fpn on this model (unlike my good ol' 60d).
-
Yes, but sadly I can't target just the 6D :P
BTW, can you post the raw_diag dark frame graphs from 6D? I'm curious to see how the numbers compare with 5D3. Try a few ISOs that you consider relevant.
-
BTW, can you post the raw_diag dark frame graphs from 6D?
... here you go (how does the 6d do?): https://bitbucket.org/Marsu42/ml-pull/downloads/6d_fpn.zip
-
6D - 5D3 (ISO 100)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F5ms9s6hff%2FDARKGRAY.jpg&hash=8ce03e640e93b6cc7860dbb6e25c66ab) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fmajruojwr%2Fdarkgray1.jpg&hash=47f14baca90b54c5ed07f05bc77d42b0)
http://www.clarkvision.com/articles/evaluation-canon-6d/index.html
But even more impressive than the high signal and low read noise, is the far better control of pattern noise. The 6D sensor performs well ahead of the 1DX at low ISO, similar to that for the 1DX at moderate to high ISOs, but the 1DX pulls slightly ahead at very high ISOs. The pattern noise of the 6D is overall the lowest I have measured for any Canon camera.
edit: dual_iso images removed.
-
6D - 5D3 (ISO 100)
I don't quite know what to make of these greyscale images - does it mean that the 6d has nearly no vertical fpn, but some horizontal?
Edit: dual_iso images deleted because they're useless.
-
Comparing the 6D/5D3 images above, you can see how the 5D3 has large amounts of vertical FPN. The 6D is doing a remarkable job of removing the vertical FPN component, but there is still some horizontal FPN remaining.
I believe I read that Canon was doing some FPN correction on the 6D. Looks like they concentrated on the vertical component. It would be interesting to find out if the work Canon is doing in the 6D firmware, could be duplicated in other cameras. I'm not sure where to start looking though, other then poking registers. Maybe a1ex has some better ideas, if it's possible, and if you or another 6D user has the time.
It's important to remember that the pattern noise is generally a fixed component (the noise level stays the same, for each individual camera). And also, that human eyes/brain, are adapt at detecting patterns. So when it looks like the pattern noise is getting stronger or weaker (on an individual camera), actually, the random noise is increasing, or decreasing.
-
If you look at the DARKFPN graphs, the 6D shows:
- Vertical FPN: stdev 0.30 with ratio 0.06
- Horizontal FPN: stdev=0.22 with ratio 0.05
- Overall stdev: 4.59 (from DARKHIST)
At ISO 100, 5D3 has:
- Vertical FPN: stdev 0.85 with ratio 0.13
- Horizontal FPN: stdev=0.21 with ratio 0.03
- Overall stdev: 6.81
These ratios are the FPN stdev divided by overall stdev, that is, how much of the total noise is FPN. I believe it's measuring the subjective effect of the FPN (that is, a low ratio, let's say below 0.05, tells the FPN gets buried into noise, and a high ratio, say above 0.1, tells the FPN is becoming obvious).
Side note: on a 4000x5000 ideal Gaussian, the estimated vertical FPN ratio is 0.015.
octave:16> G = randn(4000,5000);
octave:17> std(mean(G)) / std(G( : ))
ans = 0.015700
Overall stdev:
From raw_diag data, 6D has around 0.57 stops of extra dynamic range at ISO 100, compared to 5D3. To compare these numbers, you need to normalize by white-black, but from the data I have, the white levels are 15331 and 15283, which does not really make a difference.
From Roger Clark, shadow noise levels at ISO 100 are 34.9 vs 29.0 electrons, so 6D is ahead 5D3 by only 0.26 stops. The extra difference in DR may be explained because 6D's ISO 100 captures 79600 electrons, compared to 68900 for 5D3. That's 0.2 stops of extra highlight detail for 6D (so 0.46 stops difference in dynamic range).
FPN:
The FPN ratios are explaining the first two screenshots posted by Audionut above (http://www.magiclantern.fm/forum/index.php?topic=10111.msg112945#msg112945). 5D3 has obvious vertical FPN (ratio=0.13); 6D improves it by one stop (ratio=0.6), so vertical FPN gets buried into noise. But the horizontal FPN is slightly better on 5D3 (ratio=0.03 vs 0.05).
So, considering the FPN improvement, I'd say that, in practice, out of the box, at ISO 100, the 6D is ahead of 5D3 by around one stop.
Side note: the raw_diag grayscale analysis on dual ISO images is not relevant, because it will simply average the two components. In deep shadows, the final output from dual iso only contains data from the higher ISO.
5D3 ISO 100 reimplemented from 200 (without black/white level tweaks)
- Vertical FPN: stdev 0.71 with ratios 0.20
- Horizontal FPN: stdev=0.14 with ratio 0.04
- Overall stdev: 3.70
So, the vertical FPN is reduced only a little (0.26 stops), but the Gaussian noise is reduced a little more (0.88 stops), which makes the FPN even more obvious (the ratio gets degraded by 0.62 stops).
And this is why we need a FPN correction tool: we have almost 0.9 stops of noise improvement on 5D3, but in practice it's not very useful because of the FPN. You have already seen the effect earlier in this thread: here on a test sample from Audionut (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104298#msg104298), and here on a downsampled dark frame (http://www.magiclantern.fm/forum/index.php?topic=10111.msg105001#msg105001).
-
And now here's a little experiment that reveals the feedback loop from the left OB area, and its time constant. Set ADTG 82F3 (the one that sets top optical bar size) to 0, then run the OB zones analysis, then average the data from the left OB area.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fob-zones-5d3-iso100-82F3-0.png&hash=9a6e5cf3f6ea070c38e250669e610fa5) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fleft-feedback.png&hash=6bd8e18d26f529b91dec408a9688e4f8)
You may cross-check it with this graph (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104546#msg104546) and with g3gg0's theory: [1] (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104531#msg104531) and [2] (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104682#msg104682).
This info should explain the low frequency look of the horizontal FPN and should be very useful for correcting it.
-
Side note: the raw_diag grayscale analysis on dual ISO images is not relevant, because it will simply average the two components. In deep shadows, the final output from dual iso only contains data from the higher ISO.
Which explains why I wasn't seeing the same FPN reduction that I do in real life images. In practice, dual_iso does an excellent job of reducing FPN.
ML ISO / ML ISO + dual_iso
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fyp6juuyij%2F46_A1297.jpg&hash=ff0b97fca362e46343b421ca8ea1f688) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fg9m2xgztn%2F46_A1301.jpg&hash=3229440aa1ef34caf0f633f0ed976a55)
-
I believe I read that Canon was doing some FPN correction on the 6D. Looks like they concentrated on the vertical component. It would be interesting to find out if the work Canon is doing in the 6D firmware, could be duplicated in other cameras.
Interesting, reminds me of test shots showing the 6d is slightly softer than the 5d3 - I though this would be due to some forced chroma noise reduction, but maybe fpn correction is (also) the reason. If it's really done in camera by software, this is indeed remarkable as the softening is minor.
Question is: If so, why did Canon concentrate on vertical fpn, maybe because their speed-tuned algorithm only works this way while the image is read from the sensor?
-
Question is: If so, why did Canon concentrate on vertical fpn,
That was a poor choice of words on my behalf, sorry.
I have no true understanding why the vertical component is reduced, but the horizontal FPN remains similar to the 5D3. I assume that there is some relationship to the column amplifiers.
-
Offtopic:
Are you guys talking about this? Noticed it in h264 recording and on liveview: fixed noise. On monday Im going to the canon service site in the netherlands and on the phone they told me: It has to be re alligned ( :S :S) or something nothing to be concerned of.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.alisultan.nl%2Fmagicbug%2Fverticalbanding.JPG&hash=bb31c6eecfe9c2efe179f9a76541e100)
-
Offtopic:
This.
-
Pulled Canon ISO 200.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F3tpd3ne6j%2Fob-dr.png&hash=2091dfa4447608e86cb8e67e26c97699)
I suspect there may even be a little more then 0.1 EV of nonlinear highlights. But I need to confirm.
8/9/A/B is defiantly biased towards the highlights, and a strong cause of the nonlinear roll off in the highlights. This register bank is not really noisy (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103911#msg103911), so if I have over estimated the non-linearity, adjusting these registers back towards default, won't have a large effect on noise (printed DR results).
Adjusting saturate offset on 5D3.123, blanks the rear display. The camera appears to function correctly, except no display.
+ {0xC0F0, 0x819c, 0, "Saturate Offset (photo mode) (HIV_POST_SETUP)"},
-
Looks like there is some room for improvement, by adjusting the dual 0xFE registers, individually.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fku89cebt7%2Fob-dr1.png&hash=8dd4f4e44d417a39c9b40ef85e1f9a1f)
-
So the gain is more than 1EV? :o
-
Well, there are issues with large individual differences to 0xFE.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F8scvicpqz%2F46_A3294.jpg&hash=f49f48818e3680eb6de9fecc8bf0e6b2)
There is a very faint pattern noise. But here, I was using a difference of 3, between the register values.
Note: This is a horizontal pattern noise, as opposed to the vertical pattern noise from other registers.
-
Hi,
could somebody maybe send upload me one darkframe and one or two pictures with severe vertical FPN (ISO100 pulled from Best would be a linear TIFF 16bit (maybe even including the OB zones)?
CR2 would be OK also, although I will not be able to get the optical blacks with my current software.
Thanks in advance.
-
@A1ex any news about the paper? The community can help you with something, like tests and so on?
-
Not really; need to find the time and the mood to sit down and figure out stuff. Pretty math intensive...
But you can try to follow the thread and check our findings for other cameras; that would be nice. Raw_diag and adtg_gui are portable (they run on all cameras, first without modification, second may require minor tweaks) and should be a very good time killer during long train journeys, for example.
-
Looks like Canon increased the ISO Digital Gain in 1.2.3. The histogram has 1 pixel gaps with 1.2.3, and these are not present in 1.1.3.
-
Losing the mentality of maximum DR, whites be damned. I started looking more closely at the histogram with RawDigger.
@ISO 100, the maximum gain reductions I can do while still maintaining saturation in all color channels.
0xFE - 0x3
888x - 0x3ax
And with these tweaks.
CMOS4 - 0x308
ISO digital gain - 0x200
One thing became clear, the blue channel had less gain then the other channels. On a fully saturated image, the stdevs.
black white
R 4.50 24.6
G 4.46 32.1
G 4.50 27.3
B 4.42 95.8
There is a lot more deviation at white in the blue channel, due to it only just hitting saturation. From my research here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104511#msg104511), it's clear that ADTG2 is affecting G1 and B channels, and ADTG4 is affecting G2 and R channels. So, reducing ADTG4(888x) to 0x38x resulted in these numbers.
black white
R 4.33 89.1
G 4.50 32.3
G 4.34 89.5
B 4.41 95.8
Dynamic range in the R and G2 channels increased by 0.05 EV. All channels still saturated.
-
Here are 2 histograms showing the description from my last post.
ADTG 2/4 equal values.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fz0o07fk1n%2Fhistogram1.png&hash=96830218248c933e165eedb0910322f3)
ADTG 4 reduced.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2F9549o8ai3%2Fhistogram.png&hash=55381a0163737a7300c8c9aad483c9dd)
With ADTG 4 reduced, the histograms look remarkably similar. For reference, without ADTG tweaks, the histogram is a single peak of pixels at saturation, with a small amount of pixels above this saturation point.
Looking closely at the data, it appears as though having only some amount of saturation, is not sufficient for correctly rendered images. You'll notice on the histograms that there is some data above the saturation point, and reducing the gains sufficiently, this data is no longer there (presumably it has been pulled back into a useable range). However this results in artifacts in the image.
This is the image of the second histogram above.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fcoq7e1snf%2Fimperfections.png&hash=97867479a15fe87059444e9d57a15128)
It should be pure white. Observing the color channels individually, the B, R and G2 channels show this pattern. The G channel, which still has data above the saturation point, still retains an even appearance.
I would say this data above the saturation point is simply noise, and where the gains are reduced to much, this noise gets pulled back into the rendered image.
I thinking about WL detection in ML. Rather then guessing the hot pixels (based on some percentage), it may be useful to determine the saturation level based on the amount of pixels, at some level based on the histogram, with pixels above the saturation level being discarded. I can't think of any situations in nature where this would occur, so it should be safe to assume that any pixels above the spike at white, are simply noise.
-
Right, detecting a sharp peak makes sense.
The difficulties appear when the peak isn't very sharp (amplifier gains set way too low), or when the image doesn't have a peak (the image is not overexposed, but it's close - so, in this case, white level detection via percentile would cause useful detail to be marked as "clipped").
-
Personally, I wouldn't consider a situation where the gains have been set to low. This is a user problem.
@ISO 200, ADTG 4 gains can only be reduced a little further to match the gain of ADTG 2. For instance, ADTG2 - 888x - 0x37x, ADTG4 - 888x - 0x36x
-
Changing the topic a bit.
I was trying to understand how Roger Clark figured out the full well capacity in electrons (http://www.clarkvision.com/articles/evaluation-1d2/), so I've implemented a fairly similar method in raw_diag.
Click for his analysis of 5D3 (http://www.clarkvision.com/photoinfo/evaluation-canon-5diii/index.html), 6D (http://www.clarkvision.com/photoinfo/evaluation-canon-6d/index.html), 5D2 (http://www.clarkvision.com/photoinfo/evaluation-canon-5dii/index.html) and 7D (http://www.clarkvision.com/photoinfo/evaluation-canon-7d/index.html).
Basic theory:
- SNR curve can be estimated from the difference of two test images, taken with identical settings, from a static scene.
- Roger recommends shooting a out-of-focus white wall and take a bunch of images. I went for something much easier (but less accurate): simply shoot a out-of-focus HDR scene (with both deep shadows and clipped highlights) and take two test pictures. I think it's enough for our purposes.
- The difference will contain noise multiplied by sqrt(2), because noise from two subsequent images is not correlated (well, mostly).
- If you take small patches of the same color, we can say the patch average (taken from one of the test images) is the input signal, and the patch stdev (taken from the difference image) is the noise level at that input signal. So, we have all we need to plot the SNR curve.
Curve fitting:
- Now, we can fit an ideal model for the SNR curve. I'll use one that combines read noise (assumed constant additive Gaussian) and shot noise (which follows Poisson statistics). Something like this:
dn = patch_average - black_level;
electrons = dn * gain;
shot_snr = sqrt(electrons);
shot_noise = dn / shot_snr;
combined_noise = sqrt(read_noise*read_noise + shot_noise*shot_noise);
model_snr = log2(dn) - log2(combined_noise);
- So, we have a model with two parameters (read_noise and gain), that gives an ideal SNR. We can compare that SNR with measured values (from the SNR curve) and minimize the difference between our model and our measurements.
- Robust statistics tricks / speedup: we have up to 20000 data points; a few of them are outliers, but most of them are OK. I've grouped them into bins, every 0.5 EV, and took the median from each bin. So, now I have up to 28 data points, without outliers. Clean and fast :)
- Since I don't know how to find a closed-form solution for the best fit (if you know how, please enlighten me), I went for a very simple minimization routine: at each step, choose a random point nearby and evaluate the function there; if it's better, keep it, if not, discard it. It does the job.
My results:
I've only tested ISO 100 and 1600 in photo mode.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr-fit-100.png&hash=3f312f00989f243f74b44f3f9ba687a7) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr-fit-1600.png&hash=36354acafd4f2d8834e97c765413ff57)
Wanted: your results
1. grab raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (no need to compile anything, just place it in your MODULES folder).
2. use 1/50 for PAL, 1/60 for NTSC, or any shutter speed if you don't use artificial lights.
3. point the camera to a out-of-focus HDR scene, and adjust ISO and aperture to get a good picture (both clipped highlights and fairly deep shadows).
4. enable Debug->RAW Diagnostics, and from there, select "SNR curve (2 shots)". Disable the other analyses types (unless you want them). Make sure "Auto screenshot" is enabled (it is by default).
5. take the two test pictures from Debug -> RAW Diagnostics -> Dummy bracket. Simply click it once and wait until it's finished.
6. post the screenshot (it's called SNR2.PPM and it's saved under the RAW_DIAG folder; convert it to PNG first to make it smaller, and give it a meaningful name).
I'm looking for results covering all ISOs, for all cameras, both photo and movie mode.
If you wish to review the code or compile yorself, look here (https://bitbucket.org/hudson/magic-lantern/commits/c4b81743843f).
Have fun!
-
5D3 - Photo mode. Canon ISOs @ f/1.4 and f/4.0.
ML ISO shots have ADTG 2/4 0xFE 0x3, CMOS4 0x308 and ISO digital gain 0x200 (Canon f/4.0 default) in common.
ISO 100, ADTG2 888x 0x3cx, ADTG4 888x 0x3ax.
ISO 200-25600, ADTG2 888x 0x38x, ADTG4 888x 0x37x.
Making sure not to have gains reduced to much.
https://www.dropbox.com/sh/ktc2kyabmozjofq/AAC7kSBnzX3E5cwoLn9nrsbYa
If you aren't aware, you can click the blue download button and save the results directly to your own dropbox. :)
It would be handy if the screenshot only captured useful data. That is, it didn't capture the first image in this dual image sequence.
For those of you who fancy "unity gain", you may be interested in ML ISO 800.
-
50D photo mode. ISO 100-3200
https://www.dropbox.com/sh/d333un8o8lwmtgy/AACHL0HAY2bjUNqzhxCbtsWga (https://www.dropbox.com/sh/d333un8o8lwmtgy/AACHL0HAY2bjUNqzhxCbtsWga)
50D movie mode. ISO 100-3200
https://www.dropbox.com/sh/spiepm7sbts0t7t/AACu3FU3iB6GfDKlkLvwTidva (https://www.dropbox.com/sh/spiepm7sbts0t7t/AACu3FU3iB6GfDKlkLvwTidva)
-
Care to share your thoughts on why the SNR reduces around 0.5 EV in the highlights, with an ISO bump?
-
Cool, so the tweaks are confirming the ISO changes with a totally different method (Poisson statistics, instead of linear matching of raw data).
ISO 100, f4->f1.8:
OB noise increase: 6.48 -> 7.14 (1.1x)
read noise increase: 6.71 -> 7.38 (1.1x)
gain decrease: 5.18 -> 4.74 (1.09x)
=> the digital gain applied at f1.4 is about 1.1x. From the digital gain register, previously we have estimated it at 569/512 = 1.11x. Not bad.
LiveView results are interesting, but need to figure out the math first. They might show how many pixels are binned.
@dsManning: can you retry the LiveView tests at ISO 400 and above? They have obvious outliers.
Care to share your thoughts on why the SNR reduces around 0.5 EV in the highlights, with an ISO bump?
Not sure I understand the question, but I assume it's the difference between ISO 100 f4.0 and ISO 200 f4.0, for example. ISO 200 captures half of the photons, compared to ISO 100, so the shot noise is 1.41x higher. In shadows, the read noise is dominant, and it changes very little from 100 to 200. In highlights, the Poisson noise is dominant, so the difference is roughly half-stop.
For exact figures, you can take the equation of the ideal SNR curve (pasted above), plug the two parameters (estimated read noise & gain) and do the math for any signal level.
This SNR model can be interesting for profiled denoising, since you can model the noise curve with only two parameters (cc @hanatos from Darktable).
-
ISO 200 captures half of the photons, compared to ISO 100, so the shot noise is 1.41x higher. In shadows, the read noise is dominant, and it changes very little from 100 to 200. In highlights, the Poisson noise is dominant, so the difference is roughly half-stop.
Of course, now it makes sense, thanks. :)
-
Do I have to use a tripod? Found it - yes.
How long should it take to sample data? Mine's already working a couple of minutes - maybe it halted?
(600D)
Well, tried it for a couple of times. One time it always said "RAW Error", second time it halted again.
-
Well, it just has to stay fixed while it's taking the two exposures. In LiveView it's not a problem (no mechanical movement), but outside LV it might be (since the mirror vibrations can be strong).
However, if the image is defocused, minor vibrations are unlikely to affect the results.
Computations take a few seconds, maybe 1 minute... didn't time it. If in doubt, record a video of what happens and upload it.
-
@dsManning: can you retry the LiveView tests at ISO 400 and above? They have obvious outliers.
Fixed. Same link as above.
https://www.dropbox.com/sh/spiepm7sbts0t7t/AACu3FU3iB6GfDKlkLvwTidva (https://www.dropbox.com/sh/spiepm7sbts0t7t/AACu3FU3iB6GfDKlkLvwTidva)
Do I have to use a tripod? Found it - yes.
How long should it take to sample data? Mine's already working a couple of minutes - maybe it halted?
(600D)
Well, tried it for a couple of times. One time it always said "RAW Error", second time it halted again.
Make sure image review from the Canon menu is set to hold. I had some issues with the different image layers (plot.mo grid, graphed data, curve line) all ended up in different .ppm files. Seemed like it didn't have the time to process when it jumped back to liveview. MAY be your trouble.
-
This image shows very good coverage (which is important, since this fitting problem is a bit ill-conditioned).
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fsh%2Fspiepm7sbts0t7t%2FAACfwIVfjTbgLD6k8-ZN2KZPa%2F50Dmovie100.png%3Ftoken_hash%3DAAFKHjqcr1PqI_mdu4NTTs6Vz_wMzCAG7-nZqaUhsEhSuQ%26amp%3Bexpiry%3D1401980306&hash=7702fe28e133d1e4aeb585df1ff2b212)
Are there any conflicts with the plot module? I didn't use it yet, but this refactoring is on my todo list.
-
Are there any conflicts with the plot module? I didn't use it yet, but this refactoring is on my todo list.
No it seemed to work fine once I got the settings correct. Before I had image review set to hold in the Canon menu I was getting the data, but I was also getting separate .ppm files in different folders under RAW_DIAG folder (IMG_0001, IMG_0002, etc). Shown below.
https://www.dropbox.com/sh/l4dlb32q75pqwkn/AAClqaIXy6vsCF6pGyFEcQt9a (https://www.dropbox.com/sh/l4dlb32q75pqwkn/AAClqaIXy6vsCF6pGyFEcQt9a)
-
Well, it still hangs. With Review Hold, without hold (Off), with 2sec hold. It just draws a dozen of dots and then becomes unresponsive. It hanged with LED on, hanged with LED off.
I used 600D may 08 nightly, then cleared the card and installed clean may 13 - same thing. Btw, 600D has all builds Failed starting from May 13.
Here's the vid:
And here's what's in PPMs:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.naumik.com%2Ftemp%2Fscr4%2F20140605-u82-73kb.jpg&hash=2ef167855b412ed4fd9d3243e1ef2ea7)
Btw, I also get this SNR curve (noice profile) table overlay slightly blinking in every menu (including canons own) and LV - even after restart with battery out.
-
Can you try outside LiveView?
-
One thing I noticed while in LV was that I had to hold the half shutter till a box popped up top left of screen saying raw_diag. Dummy Bracket didn't work for the 50D in LV. Uploading a quick video (got a terrible curve, not on a tripod, no audio, just showing how to make it work) to show that it works in LV, just have to not use Dummy Bracket.
-
500D Photo Mode :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs14.postimg.org%2Fm5pl178qp%2F100.png&hash=5e981ff65c47698af773272046a3b23b) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs14.postimg.org%2Frslxyob9d%2F1600.png&hash=a02d36eeb291170c50fcd4adfdd8b554)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs14.postimg.org%2F4gxu9kwzl%2F16002.png&hash=05f5990e6442f51aec5619570a8b7b87)
-
Dummy Bracket didn't work for the 50D in LV.
Were you in movie mode LiveView, or photo mode LiveView? It matters here.
-
Were you in movie mode LiveView, or photo mode LiveView? It matters here.
Movie mode. Get mixed results with LV on and Dummy Bracket. Here is another one. This one seemed to process, but the overlays are strange if I don't do halfshutter.
https://www.dropbox.com/sh/l4dlb32q75pqwkn/AAClqaIXy6vsCF6pGyFEcQt9a (https://www.dropbox.com/sh/l4dlb32q75pqwkn/AAClqaIXy6vsCF6pGyFEcQt9a)
-
Wanted: your results
You keep updating the module in the repo - should we wait some more, or is this the one you want the tests to be run with? If I take the time to cover the 6d, I really don't want to do it twice :-p
-
The changes were not essential, so you can just grab a copy and run the tests. All the previous graphs are still valid, maybe with less fancy displays.
-
Reproduced the lockup in 5D3 at 50fps and hopefully fixed it.
I made some core changes for a clean fix, but I've applied some workarounds (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag_workarounds.patch) in the precompiled binary, to allow it to run on older nightlies out of the box. So, download the binary raw_diag.mo from this post (http://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955) (or the first post, it's the same) and it should just work.
-
Ran on 5D2 with 2014-05-30 NB, 2 shots pass but get the message: "You may need to solder some RAM chips :(".
edit:
Debug shows a memory error -> shoot_malloc(29MB) failed at raw_diag.c:445, raw_diag_task.
-
Solved (well, workaround; will still show error in the Debug menu, but it will work).
-
OK, works fine now -.^
Here is results for ISO 100-1600 with -1/3 steps (could be useful):
-> http://www.bgimage.fr/publis_ext/ml/2014-06-05_snr2_5d2.zip (bad samples)
Note: For me there's 'something somewhere'; in regular conditions 5D2 has a FWC of ~65700e- (around 5ke- more with some tweaks) and here I get ~55600e-.
Maybe an OB' stuff who false the analyse.
-
OB stdev is only used as an initial guess (but should have no effect on the final result).
However, in your ISO 100 sample, you don't have any data points above 8 EV. This is very important for a good fit.
Here's how it should look like (notice the red dots covering the entire range) : http://www.magiclantern.fm/forum/index.php?topic=10111.msg117999#msg117999
My quick test on 5D2 at ISO 100:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fsnr-fit-5d2-100.png&hash=b2c92709e638db4c2ed06625b27ce792)
-
(...)
However, in your ISO 100 sample, you don't have any data points above 8 EV. This is very important for a good fit.
(...)
Ah, it is the why.
Though a part of the surface was overexposed (burned in fact) and another well dark. As usual..
I will see later to remake a run if needs, because you look to already have 5D2 data (?).
Beside, it will be interesting to run this kind of analyse with a special mini_iso conf (to see the FWC gain) and dual_iso feature (the FWC gain should be [...!] ). -.^
note: I don't know if it's a UI restriction, but the unit for FWC is e- (e is negative actually).
-
I only ran a test shot to check that memory error. I'd prefer not to sit down and take a bunch test pictures that other people can do very easily.
FWC gain with dual ISO... should be like ISO 100 for half of the rows, and like ISO 1600 for the other half (for example). The exact SNR curve will depend on the blending algorithm (so you will need to run this experiment offline) and also on the scene contents (where it tries to balance between noise and aliasing in shadows). Running it without --alias-map should give consistent results though (since the blending factors are chosen based on input signal and OB noise levels).
What would be interesting is to run it in LiveView and figure out how many pixels are averaged when downsampling. Can you help me figure out the math for this? (that is, how the noise should behave if N sensels are averaged on the CMOS sensor, before amplification - or wherever you think Canon might do this step)
-
I only ran a test shot to check that memory error. I'd prefer not to sit down and take a bunch test pictures that other people can do very easily.
OK -.^
(...)
What would be interesting is to run it in LiveView and figure out how many pixels are averaged when downsampling. Can you help me figure out the math for this? (that is, how the noise should behave if N sensels are averaged on the CMOS sensor, before amplification - or wherever you think Canon might do this step)
Of course, if I can help.
I thought resizing was done by pixel groups exclusion (?). Witch is causing, with VAF (working as a median filter), aliasing/moiré problems (because of Nyquist limit) except in x5 video mode (where CMOS surface is recorded at full resolution). So in this case, pixel informations (levels, etc.) should be the same before/after downsampling resizing.
Or are you speaking about the video feed directly on the camera back screen?
Beside, I well know about noise, FWC, etc. in case of binning downsampling (hardware or not BTW) but obviously this is not the case here (?). And I'v never seen hardware CMOS average capability (looks technically improbable), even it is _maybe_ possible.
-
Hello!
thought i would try this on the Eos M. Got stuck with some message on the screen.
https://drive.google.com/file/d/0B04NFrlfFHNvUi15dy1Cb19ONUU/edit?usp=sharing
-
I've translated the raw_diag code for SNR curves into octave, so you can run the same analysis offline (including on dual iso files).
snr_curve.m (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/snr_curve.m) (for two test images of the same scene, taken with identical settings)
run_tests.m (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/run_tests.m) (for all my test images, see below)
The curve fitting results might differ a little from raw_diag (nothing to worry about).
ISO 100 and 1600:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F100.png&hash=0f2dc7776d549a522bf79bc9e2571ab3) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F1600.png&hash=dbed63a140a70bbfe6d0a39895d4ee3f)
Dual ISO 100/1600, developed with different options:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F100-1600-default.png&hash=616f921c50d896d671d93275273f526c) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F100-1600-no-alias-map.png&hash=ed2c1bc71584406e3eb2059306728423)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F100-1600-fast.png&hash=92943308dc0bed909b8286a517a3ea7b) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F100-1600-fast-fullres.png&hash=a262ed40ee82d9080dea9ca6e7766f6d)
The graphs were scaled to show 15 EV below white level, and the SNR axis is from -2 to +10 EV. Therefore, you can overlap any two images and compare them directly (with alt+tab, for example).
edit: also added ISO 1600 darkened by 4 EV, to match the brightness of ISO 100. Ideally, the SNR of a dual ISO image should be roughly the maximum of the two single-iso SNRs (maybe better in overlapped areas) if you don't consider the resolution loss.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F1600-darkened.png&hash=522f50f4604f9421a4e202f9a1db0d1a)
-
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.org%2Fd5c3yrulz%2Fimage.png&hash=668ac1e558fe2087421841a0641f6e1c)
Trick :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.org%2Fuh7v6gypj%2Fsnr2.png&hash=2de0b46bc0079268b8d0d7f693e702a8)
-
Some more results, this time with the Read noise and Poisson noise plots, and trying to fill the entire graph with data, like so.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fgjtlh39dn%2FML_ISO_100.png&hash=b47210c62e86f3e1c558a3971facc4d4)
Canon ISO (https://www.dropbox.com/sh/suf1wchza37d9ig/AABVRktQl1usGQ9fuAgGZrYta)
ML ISO (https://www.dropbox.com/sh/vz309qys8zsv7t4/AAAEk48Y87S4abAd1K26vp6Ca)
-
Nice!
Can you use the same test scene (since it gives great coverage) for some movie mode tests? I'm interested in:
- 1080p (I expect identical results at all 3 FPS settings)
- 720p
- 5x zoom (tip: use raw video grayscale preview to frame the image)
The movie mode tests may run a little slower, but they don't waste any shutter actuations, so everybody should do them. All cameras, please!
-
Well, I just completed a shit load of sample images, to find the screenshot function only captured the first image, and then went on strike. Sure enough, the first image was setting up the testing procedure and doesn't contain useful information. Anyway, I can share my thoughts.
Thoughts on the data:
At first it appeared like movie mode was entirely Read noise limited at lower ISOs. I suspect that movie mode is somewhat rather Read noise limited (compared to photo mode), however, the curve was extending above the top of the graph, so I can't be certain. It needs some automatic fitting of the curve, or at least, different scales on the graph for movie mode and photo mode.
The DR is greater in movie mode then photo mode. Now, we know the ADTG registers don't change, so if we also consider that the the images are more Read noise dominated, then I believe that this shows pixel binning.
If memory serves me correctly, the stdev was similar to photo mode, but the FWC was less. I didn't check the read noise measurement during the capture phase. I know, this doesn't make sense.
Around ISO 800, the curve gets clipped at 0 EV on the vertical scale. From here, it increases back towards -2 EV with increasing ISO. Confirmed in both 1080p and 720p modes. Speaking of which, I assume you meant Canon resolution modes, since cropping with ML resolution changes seems pointless?
In crop mode, somewhat expected, the figures were remarkably similar to photo mode.
ADTG tweaks work! 11.5 EV @ ISO 200 with some quick tweaking. CMOS 5 is 0x501 and CMOS 6 is 0x370 in movie mode. ADTG6[8000] is 0x6.
Swapping in and out of crop mode, adjusting ADTG registers, taking SNR curve measurements, eventually the camera had enough, and displayed really weird images like the ones I shared in the ISO research pull request.
Thoughts on functionality:
It says "sampling data" in the top right section, but at first I thought it wasn't doing anything, I'm pretty sure for whatever reason, the graph wasn't updating on the first try.. Maybe you could change that to, "Sit tight, I'm counting your photons"! Or something else really obvious to us common folks.
Since the data is sampled in real time, the first image in the two shot sequence is rather redundant, right?
The framing of the image on the LCD screen changes, when the SNR curve is processing. However, the curve still reflects the original framing.
Did I mention the screenshot function only works on the first image :P
And whatever else I forgot, since having a notepad and pen with me, makes to much sense ::)
Off to bed. I look forward to raw_diag_movie_mode version 2 when I wake up ;D
-
I think I know why the screenshot works only for one image. The PPM files are named after the last movie filename...
(so, it was the last screenshot that got saved, not the first one)
Solved.
"the curve was extending above the top of the graph" => there is a second view of the same thing, called "Noise curve". This one is a little harder to overflow. Still, the overflow is only on the display, and won't affect the fitting results.
-
(...)
Thoughts on the data:
At first it appeared like movie mode was entirely Read noise limited at lower ISOs. I suspect that movie mode is somewhat rather Read noise limited (compared to photo mode), however, the curve was extending above the top of the graph, so I can't be certain. It needs some automatic fitting of the curve, or at least, different scales on the graph for movie mode and photo mode.
The DR is greater in movie mode then photo mode. Now, we know the ADTG registers don't change, so if we also consider that the the images are more Read noise dominated, then I believe that this shows pixel binning.
(...)
To confirm if pixel binning is done it's quite simple: RON should be the same before/after binning (only the signal value change), or with a normalized signal value the SNR should be up in function of the number of pixel binned.
PS. I re-run some test with raw_diag but I don't find a way to get good samples with artificial lights. Audionut > Is your scene is by day?
-
If you know the RON and the FWC values, one set from 1:1 crop mode (5x zoom), and the other from 1080p (or 720p), can you find out how many pixels were binned? How?
(with my "pi=3" math skills, I get around 10 pixels binned in 5D3 1080p and around 15 in 720p, which doesn't look quite right)
-
Lets take SNR as reference, because overall values could be hard to compare, in a simple and not so realistic example:
If in crop mode (5x) you get 3 and in 1080p you get 12 it means that you should do a bin2: 4 pixels group, hence 4x3 and so 12.
SNR increasing by power of 2 of the pixel matrix size; bin2 -> 2x2 pixels -> 4 pixels -> 4x SNR, bin4 -> 4x4 pixels -> 16x SNR...
In the real world it could have some variances, but it should be close to that.
Note: Could you anticipate the binning value by comparing the full resolution to the 1080p? I was thinking about 5760px -> 1920px, so a bin3 BTW.
-
Two 9 watt LED spot lights. I believe that LED doesn't flicker, so this way I can use any shutter value.
One LED is around 7cm (3in) from a wall, pointed almost directly at it, the other LED is around 13cm (5in) from the wall, angled slightly away to give a nice gradient. The angled light has a small hint of saturation, the point blank light is complete saturation (useful when ADTG tweaking). Exposure time = 1/100s @ f/4.0.
The test setup is in a dimly lit room (I would say around 2 EV (http://www.fredparker.com/ultexp1.htm#Light%20Intensity%20Chart)). I use objects in the scene to control light spill, and in the dark area, I have a window to outside, that for all intents and purposes, is pitch black (-6 EV). So I estimate the test scene to have around 17 EV of DR, with reasonably good gradients throughout.
The part I found most difficult, is getting a nice gradient from the light, for very good coverage in the area around 0 EV through -4 EV from sensor saturation. I have ideas for the next test batch :) Daytime here so I cannot recreate the scene, but I will take a bracket later tonight when I run the tests again.
One day I will learn how to light a portrait with the same care I place onto test scenes. ::)
-
Ouch, looks like you didn't update the .mo link, and I didn't check functionality.
Third time is a charm right! Good news is, I've become really proficient at changing ISO in movie mode, without triggering the SNR curve.
This one (https://bitbucket.org/hudson/magic-lantern/commits/1e1e18ba5acef8a7bf649229f1c770de3afd74df?at=unified) causes conflicts with raw_rec and cr2hdr.
-
Whoops, updated now.
No conflicts here when merging unified and iso-research. I usually compile the module directly from the branch, without merging anything.
edit: ah, you mean compile error.
-
Yes I did mean compile errors :P
Finally finished stuffing around, and here is a good bunch of results. Enjoy!
https://www.dropbox.com/sh/ktc2kyabmozjofq/AAC7kSBnzX3E5cwoLn9nrsbYa
One with a little ADTG tweaking :D
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs15.postimg.cc%2Fn7ubxme23%2F5_D3-1080p-_ADTG-_FE-0x3_Pre-amp-0x1x.png&hash=01e78b81b714ff9d1638e0d469b3ff77)
Edit, I did this series as, 1080p, crop, 720p, ADTG tweaks. By the end, the camera was getting a little warm, and probably affecting results, I guess. They all had a resolution width of 1080p in ML.
Crop series doesn't have the coverage that I hoped, of course, I didn't remember that I could increase resolution until after I completed the captures. ::)
Is there anyway to determine the CMOS/ADTG settings in crop mode?
-
I expect temperature to cause differences in the read noise (in this model), but I'm interested mostly in the shot noise. So it should be fine.
CMOS/ADTG settings are printed by both iso_regs and adtg_gui (if you don't change any settings). Not all of them have effect in both modes though.
On 5D3, the ADTG preamp (8/9/A/B) is 91 in 1080p, 102 in 720p and 59 in 5x/photo. My previous tests showed the unit for this one is 0.006 EV, so the differences (compared to 5x/photo) are +0.19 EV for 1080p and +0.26 EV for 720p. I don't know if these differences result in different amounts of captured highlights.
Anyway, these differences should be OK if you can work with the "pi=3" hypothesis.
From the above data, I'll try to guess the pixel binning factors from LiveView.
Pixel binning discussion continued at http://magiclantern.fm/forum/index.php?topic=16516.
-
On 5D3, the ADTG preamp (8/9/A/B) is 91 in 1080p, 102 in 720p and 59 in 5x/photo.
Here, they are 59, 63, 39. Highest values of the set.
If I reduce ADTG preamp, the curve in the highlights has a slight bend towards lower EV values on the graph. With ADTG 888x, I can create a huge bend in the curve at the highlights. I seem to have deleted the graph, I'll post one tomorrow.
Notice with this capture, "5D3-1080p-ADTG-FE-0x3+888x-0x37x.png", ADTG 888x has been reduced to much.
What do you suspect is happening with the clipped shadows @ ISO 800?
-
Clipped shadows? Not sure, can you take some silent pictures in that mode? (a good one and a bad one should be enough).
-
I haven't inspected any images (yet). Look at the curve data.
It only happens in LV.
-
Dynamic range plot from Audionut's data (5D3, movie mode, no tweaks):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F5d3-movie.png&hash=8b5a51e7004494587c74a2ab32b8a405)
(noise graphs here: https://www.dropbox.com/sh/ktc2kyabmozjofq/AAC7kSBnzX3E5cwoLn9nrsbYa )
Therefore:
- 5D3 video shooters: do not be afraid of ISO 6400 in non-crop modes (the improvement is a consequence of pixel binning, see my previous post (http://magiclantern.fm/forum/index.php?topic=16516))
- DR estimated as log2(white-black) - log2(DN where SNR = 1)
- the difference beween photo and LV crop ISO appears to be 0.4 EV (LV being darker), therefore I've shifted all the LV plots by 0.4 EV to the left (needs to be confirmed by checking the clipping point; I've got the difference from the number of electrons in the noise model)
- this graph is not directly comparable to the previous ones (I believe this method is more accurate, but need to double-check the theory to be sure)
isos = [ 100 200 400 800 1600 3200 6400 12800 ];
dr_720 = [ 11.02 10.96 10.93 10.85 10.67 10.36 9.89 8.89 ];
dr_1080 = [ 11.10 11.05 11.02 10.93 10.73 10.39 9.88 8.88 ];
dr_crop = [ 11.22 11.13 11.00 10.82 10.32 9.68 8.92 7.70 ];
dr_photo = [ 10.92 10.83 10.69 10.46 10.02 9.40 8.49 ];
-
I expect temperature to cause differences in the read noise (in this model), but I'm interested mostly in the shot noise. So it should be fine.
Indeed, analysis I made show that temperature does not significantly affects RON. Values are slightly under 1% per a 50°C variation, in the range -50°C -> +60°C, temperature recorded directly on the back side of the CMOS with a special device (the internal thermometer is not relevant because placed on the mainboard).
Beside, the thermal signal well follow the CMOS temperature, causing major noise issues when we use our cameras with a short exposure time when the temperature rises (thermal signal is _mainly_ function of the temperature and the exposure duration).
So when we made noise analysis it is important to use a process where CMOS temperature is included.
(...)
From the above data, I'll try to guess the binning factors from LiveView (and I'll ask SpcCb to double-check what follows):
(...)
Very interesting data.
But I don't figure yet why RON blow up @5x with high ISO face to photo mode. Maybe a level intercorrelation inside the logic pixel intrication area because of the .8n reduction factor.
BTW I did not know there was a reduction factor in 5x video mode; I cant make tests with 5D3 here, 'does not have Bayer matrix anymore, but I'm interested to make test to see if it's the same thing with a 5D2. What kind of resolution chart should we use?
-
For temperature, as long as the noise does not depend on the image signal (if it's additive), the current model will include it in the read noise. So, the FWC should not be affected by temperature variations. An easy way to tell would be to retry the first test image at the end of the experiment (so you have a screenshot with camera hot, and another with the camera cold, from which you can check the repeatability).
Possible explanations for the RON difference in 5x vs photo:
- the ISO ends up higher in photo mode (why? the sensor registers are identical though, at least those identified in this thread)
- photo exposure uses mechanical shutter, while LV exposure uses electronic rolling shutter; as a result, the photo exposure (electronically) may be longer than the actual exposure time
- the readout speeds may be different (could this cause noise differences?)
For resolution charts: you can use any of them (even some fine print that gets progressively smaller). This one is well-known: http://www.techradar.com/news/photography-video-capture/cameras/camera-testing-resolution-charts-explained-1027585
-
(...)
- the readout speeds may be different (could this cause noise differences?)
(...)
Yes, RON is function of the read speed.
It could be an explanation _or a part of_ for the RON value differences.
For example, 2~5e- RMS corresponding to a read speed about some 10E+2 MHz and 6~9e- RMS corresponding to a read speed about a couple of 10E+1 MHz.
Note; There's some other hardware characteristics who defined RON, not only the read speed (photodiode current, etc.)
Beside we have to not forget that Canon CMOS are CDS (Correlated Double Sampling), so the extra low RON that we get is 'hardware optimized' and it could be adjusted to fit on demands (specific video operations etc.).
-
Wanted: your results
1. grab raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (no need to compile anything, just place it in your MODULES folder).
2. use 1/50 for PAL, 1/60 for NTSC, or any shutter speed if you don't use artificial lights.
3. point the camera to a out-of-focus HDR scene, and adjust ISO and aperture to get a good picture (both clipped highlights and fairly deep shadows).
4. enable Debug->RAW Diagnostics, and from there, select "SNR curve (2 shots)". Disable the other analyses types (unless you want them). Make sure "Auto screenshot" is enabled (it is by default).
5. take the two test pictures from Debug -> RAW Diagnostics -> Dummy bracket. Simply click it once and wait until it's finished.
6. post the screenshot (it's called SNR2.PPM and it's saved under the RAW_DIAG folder; convert it to PNG first to make it smaller, and give it a meaningful name).
I'm looking for results covering all ISOs, for all cameras, both photo and movie mode.
If you wish to review the code or compile yorself, look here (https://bitbucket.org/hudson/magic-lantern/commits/c4b81743843f).
Have fun!
Canon 6d - all (native) iso's both photo and video mode(1080P), (filename of the png's are describing which mode and which iso):
https://drive.google.com/folderview?id=0B1BxGc3dfMDaUjlnaENVZ2drYUU&usp=sharing
:D
-
See my results in the link posted before this post:
In the lower iso's the full well capacity is about 3,4 times larger in video mode(1080p) VS photo mode on the 6d
In the higher iso's the full well capacity is about 4 times larger in video mode(1080p) VS photo mode on the 6d
Don't know if my understanding about full well capacity is correct, but could that mean that the 6d in video mode uses a 4 pixel binning method ?
List of the results posted in the link before this post
Canon 6D
Full Well Capacity Dyn.Range Full Well Capacity Dyn.Range
ISO Photo mode Video Mode(1080p)
100 79546e 11,48EV 274301e 11,84EV
200 42220e 11,49EV 142616e 11,82EV
400 20643e 11,28EV 75858e 11,75EV
800 10482e 11,00EV 39219e 11,62EV
1600 5215e 10,55EV 19868e 11,38EV
3200 2568e 9,87EV 10092e 10,95EV
6400 1271e 9,11EV 4931e 10,47EV
12800 556e 8,04EV 2408e 9,44EV
25600 371e 7,10EV X X
-
Don't know if my understanding about full well capacity is correct, but could that mean that the 6d in video mode uses a 4 pixel binning method ?
5472 / 4 = 1368
5472 / 3 = 1824
3 is more likely
-
5472 / 4 = 1368
5472 / 3 = 1824
3 is more likely
Ah, forgot about the real world logical numbers... ::) ML raw video live view feed is 1808 pixels wide.
3 pixel wide binning method (and lineskipping) is the most likely indeed.
-
Can you do the tests in all video modes? (I'm interested in 5x zoom, 1080p and 720p - does the 6D have other video modes?) I guess you should get a ratio closer to 3 if you compare 1080p with 5x zoom (which is also a 1:1 crop).
Tip: turn off global draw to get clean screenshots in LiveView.
-
Can you do the tests in all video modes? (I'm interested in 5x zoom, 1080p and 720p - does the 6D have other video modes?) I guess you should get a ratio closer to 3 if you compare 1080p with 5x zoom (which is also a 1:1 crop).
Tip: turn off global draw to get clean screenshots in LiveView.
Ah I already was wondering how I could get rid of the time and histogram... ::) on those screenshots, Will turn global draw off.
I think I can do the rest of the video modes tomorrow (It's 10 in the evening over here).
But just to be sure which modes you need:
In live view in 1080p mode I have 5x zoom and 10x zoom.
In 720p mode there's also 5x zoom and 10 x zoom.
The 5x zoom turns automatic to ML gray scale preview, 10x zoom is in full color preview.
If I do the 720P (without zoom) and 1080p mode with 5x zoom, would those 2 complete the numbers you'd like to see ?
Oh, forget to mention, there's also a 640x480 25fps mode in the canon menu on the 6D, when I choose it, the framing doesn't change.
So or canon is writing/saving it to a 640x480 format on card or it uses another binning modes, maybe this video mode is interesting too perform the RAW_DIAG test on too...
-
5x zoom is the same in all modes (at least on my cameras). So, adding 720p and 5x to the existing graphs will do the trick.
-
5x zoom is the same in all modes (at least on my cameras). So, adding 720p and 5x to the existing graphs will do the trick.
Oh, forget to mention, there's also a 640x480 25fps mode in the canon menu on the 6D, when I choose it, the framing doesn't change.
So or canon is writing/saving it to a 640x480 format on card or it uses another binning mode, maybe this video mode is interesting too, to perform the RAW_DIAG test on...
-
Can you do the tests in all video modes? (I'm interested in 5x zoom, 1080p and 720p - does the 6D have other video modes?) I guess you should get a ratio closer to 3 if you compare 1080p with 5x zoom (which is also a 1:1 crop).
Tip: turn off global draw to get clean screenshots in LiveView.
Got the numbers for 5xzoom mode on canon 6d
will do 720 mode soon, couldn't find a good spot with HDR, very cloudy over here
(got not so nice RAW-DIAG graphs, dots in graphs where not nice linear, random dots in highlights)
Full Well Capacity Dyn.Range Full Well Capacity Dyn.Range Full Well Capacity Dyn.Range
ISO Photo mode Video Mode(1080p) Video Mode(5xzoom)
100 79546e 11,48EV 274301e 11,84EV 97890e 11,78EV
200 42220e 11,49EV 142616e 11,82EV 48335e 11,72EV
400 20643e 11,28EV 75858e 11,75EV 25171e 11,59EV
800 10482e 11,00EV 39219e 11,62EV 12958e 11,29EV
1600 5215e 10,55EV 19868e 11,38EV 6508e 10,85EV
3200 2568e 9,87EV 10092e 10,95EV 3354e 10,25EV
6400 1271e 9,11EV 4931e 10,47EV 1663e 9,65EV
12800 556e 8,04EV 2408e 9,44EV 831e 8,51EV
25600 371e 7,10EV X X X X
Full well capacity of 1080p divided by 5x zoom results in the following numbers (looks like 3 pixels are binned)
ISO - 1080p/5xzoom
100 - 2,8
200 - 3,0
400 - 3,0
800 - 3,0
1600 - 3,1
3200 - 3,0
6400 - 3,0
12800 - 2,9
Will try to find a good HDR spot for 720 mode
-
:)
Updated this folderlink with the 5xzoom RAW-DIAG images.
https://drive.google.com/folderview?id=0B1BxGc3dfMDaUjlnaENVZ2drYUU&usp=sharing
-
I noticed in the data above that the dynamic range in 5x zoom video mode is still 0.5 stop better then photo mode (almost equal to 1080p video mode).
There should be no pixelbinning in the 5x zoom mode, how come the dynamic range is still better compared to photo mode (Full well capacity is also higher) ?
Is this difference caused by mechanical shutter VS electronic shutter ?
-
5x zoom is the same in all modes (at least on my cameras). So, adding 720p and 5x to the existing graphs will do the trick.
Hi Alex,
Tried various setups, but I can't get a decent blue line plotted with RAW_DIAG in 720p video mode.
The blue dots are scattered all over the place in the graph, does that mean that the images aren't alike ?
I use the dummy bracket function, but I use available light from outdoors on a cloudy windy day, maybe that doesn't work too well... :-\
-
Ideally the scene should be extremely static. A cloudy windy day, probably doesn't satisfy those requirements.
-
I don't have a printer, I might be able to access (a decent) one over the weekend, and printout a target to capture.
Haven't been able to find anything around the house that seems suitable. ::)
-
Hi all,
This is astronomer from NikonHacker. I had been investigating the Sony sensor for months. Could someone explain to me what's the difference between white offset and black level offset in this thread? I'd expect the black level and gain are the only 2 values to fix a straight line. Some tests done by others suggested Canon did offset to correct dark current and bring it back to 2048. I thought this is done digitally as the saturation point for dark frame is not 16383. But it seems contradict some of the posts here where you guys suggest an additional feedback loop during optical black pixel read out.
Sony sensor has all the analog chain integrated so it's much simpler. A register to set black level and 4 gain registers for each color channels. The sensor output spans from 0 to 16382. All the rest are done in digital domain, such as color channel scaling, black level clip (dark current correction), large aperture light loss scaling, etc.
Best,
astronomer
-
The black level is set to some value above 0, to allow the stdev to be easily captured. This is controlled by a voltage feedback loop at the ADC, ADTG[8880].
The saturation point is controlled by a number of registers. On a 5D3, I'll describe the ones I understand.
Analog domain.
ADTG[8,9,a,b] + ADTG[888x] - These control the (individual) column gains. There are two banks of 4, ADTG2 + ADTG4.
ADTG[fe] - This is like some master gain control.
The native (Canon default) saturation level is 15283. Increasing analog gains does not increase this value, only decreases dynamic range. We can decrease these analog gains to recover highlight detail, without affecting the saturation level.
And the input to this ADTG is CMOS[0].
In the digital domain.
Saturate offset - I think a1ex mentioned this may be analog. edit see here: http://www.magiclantern.fm/forum/index.php?topic=10111.msg103971#msg103971
Black/White offset
Digital gain - Canon uses this one to offset the light loss at wide apertures (http://www.luminous-landscape.com/essays/an_open_letter_to_the_major_camera_manufacturers.shtml).
-
This is astronomer from NikonHacker.
Hi, I've been following your thread for a while. For the record, the link is https://nikonhacker.com/viewtopic.php?t=1840
what's the difference between white offset and black level offset in this thread?
If you are asking about the registers that can be tweaked, you can find them documented in the iso-regs module:
https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/iso_regs/iso_regs.c#cl-583
(we simply changed them, written down the observed effects, and then guessed what they might do)
If you are asking about how to interpret black level and white level, it's as simple as:
normalized = (raw - black_level) / (white_level - black_level)
-
ADTG 82F3 (the one that sets top optical bar size)
For 5D2, the equivalent register is 0x1179.
(my theory is that increasing top bar size could help with correcting vertical FPN, but need to sit down and figure out the details)
-
5x zoom is the same in all modes (at least on my cameras). So, adding 720p and 5x to the existing graphs will do the trick.
Hi Alex,
Do you still have interest in the graphs for 720p mode on 6D ?
Can give it another try if it's useful
-
Sure.
-
Sure.
Nice bright sunny day over here, so here are the numbers, now also added 720P video mode:
all the graphs can be found on my google drive link:
https://drive.google.com/folderview?id=0B1BxGc3dfMDaUjlnaENVZ2drYUU&usp=sharing
CANON 6D
Full Well Capacity Dyn.Range Full Well Capacity Dyn.Range Full Well Capacity Dyn.Range Full Well Capacity Dyn.Range
ISO Photo mode Video Mode(1080p) Video Mode(5xzoom) Video Mode(720p)
100 79546e 11,48EV 274301e 11,84EV 97890e 11,78EV 242833e 11,87EV
200 42220e 11,49EV 142616e 11,82EV 48335e 11,72EV 110182e 11,85EV
400 20643e 11,28EV 75858e 11,75EV 25171e 11,59EV 66885e 11,81EV
800 10482e 11,00EV 39219e 11,62EV 12958e 11,29EV 36324e 11,67EV
1600 5215e 10,55EV 19868e 11,38EV 6508e 10,85EV 18721e 11,40EV
3200 2568e 9,87EV 10092e 10,95EV 3354e 10,25EV 9748e 10,96EV
6400 1271e 9,11EV 4931e 10,47EV 1663e 9,65EV 4874e 10,48EV
12800 556e 8,04EV 2408e 9,44EV 831e 8,51EV 2380e 9,35EV
25600 371e 7,10EV X X X X X X
The numbers of the 720p mode are almost identical as the ones in 1080P, so no extra binning, just extra row skipping I guess ?
-
If you are asking about the registers that can be tweaked, you can find them documented in the iso-regs module:
https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/iso_regs/iso_regs.c#cl-583
(we simply changed them, written down the observed effects, and then guessed what they might do)
OK, correct me if my understanding is wrong.
1. CMOS gain is the analog gain inside the sensor itself. So amplified signal in high iso will avoid noise contamination when the signal wire run across the PCB until it hits PGC/ADC chip?
2. All other ADTG gain are to fine tune the voltage range input to ADC?
3. Digital Gain is used when a larger aperture is selected?
4. From that image, it appear the saturation point is analog adjust but black level is offset in post processing?
This really contradicts what I understand in a Sony Exmor sensor.
In Sony sensor, the OFFSET 0x1F register sets the input number to bias voltage. It calibrated the ADC bias voltage so that the dummy pixels would be read out as your OFFSET value. During the first 50 rows, sensor will continue refine this bias level to within 1 ADU using dummy readout. Regardless of OFFSET and gain setting, the saturation point always reach 16382. It's like adjusting the intercept of a linear function.
Dark current will be registered on optical black pixels and all the active pixels as well. So it will show an elevated bias. This is where it differs from Canon. Somehow it clamped the dark current to bias level.
The stock Nikon firmware adjust the dark current to 0 using those optical black pixels in post processing. Thus saturation level will be pulled down. I'm curios to know if it's possible for Canon to disable dark normalization as well?
This is interesting since some Nikon DSLRs also uses external ADCs. D700/D3 uses AD9974.
-
Testing ADTG module outdoors :D
500D, the same exposure
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2F6bh5yhfa5%2Fimage.jpg&hash=4c28ecc532fe9dbba4b55fe5aa3ca846)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Fgmtir56zh%2Fimage.jpg&hash=43e9442b43b8c900d34fcf5c095029d8)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Fn1x51k331%2Fimage.jpg&hash=12efce31212faf5bf7a6daad36f61dc6)
-
Trying to combine the results from 5D3 and 6D (data from Audionut (http://www.magiclantern.fm/forum/index.php?topic=10111.msg118220#msg118220) and Levas (http://www.magiclantern.fm/forum/index.php?topic=10111.msg120656#msg120656), no ISO tweaks applied).
Notice the 6D captures quite a bit more electrons than 5D3. You may say it's normal, because the 6D has lower resolution, so it captures more photons per pixel. However, the pixel count ratio is (5796*3870)/(5496*3670) = 1.1121, while the electron count ratio is about 1.1988. There are 0.11 EV not explained by the resolution difference. Let's find out what's going on.
Possible causes:
- systematic errors in my measurement method
- 6D uses lower ISOs than 5D3 (even if they both print the same value)
- 6D's sensor has a higher quantum efficiency
- a combination of the above
Theory:
- photon to electron ratio = quantum efficiency (assumming it's a sensor constant)
- equal ISOs = equal number of photons per area (light required to saturate the sensor)
- ISO doubles => half as many photons (or electrons) captured
- N pixels binned => number of photons (or electrons) per pixel increase N times
Assumming Canon full-stop ISOs in photo mode match the ones measured by some other method (let's say DxO), we can find a scaling factor for converting our electron counts into ISO, while matching the reference values in the least squares sense. I'll minimize the ISO difference, in EV, for ISO 100..6400:
function err = crit_electrons(ref_electrons, measured_electrons, reference_isos)
isos_estimated = 100 ./ (measured_electrons / ref_electrons);
err = norm(log2(isos_estimated) - log2(reference_isos));
end
function ref_electrons = fit_electrons(measured_electrons, isos_dxo)
ref_electrons = fminsearch(@(e) crit_electrons(e, measured_electrons, isos_dxo), 50000);
ref_electrons = round(ref_electrons);
end
Assuming Canon's full-stop ISOs are correct, the scaling factors are:
isos = [ 100 200 400 800 1600 3200 6400 ];
electrons_photo_5d3 = [ 67684 34146 16835 8735 4288 2197 1085 ];
electrons_photo_6d = [ 79546 42220 20643 10482 5215 2568 1271 ];
ref_electrons_5d3 = fit_electrons(electrons_photo_5d3, isos);
ref_electrons_6d = fit_electrons(electrons_photo_6d, isos);
=>
ref_electrons_5d3 = 68787
ref_electrons_6d = 82465
ref_electrons_6d / ref_electrons_5d3 = 1.1988
Now I'm going to assume Canon's full-stop ISOs are pretty much correct, but I'll keep the relative differences measured by DxO between the two cameras. That difference is...
octave:1> isos_dxo_5d3 = [ 80 160 323 641 1280 2518 5179 ];
octave:2> isos_dxo_6d = [ 80 153 311 616 1210 2400 4991 ];
octave:3> isos_dxo_5d3 ./ isos_dxo_6d
ans =
1.0000 1.0458 1.0386 1.0406 1.0579 1.0492 1.0377
octave:4> log2(mean(isos_dxo_5d3 ./ isos_dxo_6d))
ans = 0.054522
=> 6D ISOs are 0.055 EV lower than 5D3's.
Let's distribute the ISO difference equally between the two cameras, and redo the fit:
iso_difference = log2(mean(isos_dxo_5d3 ./ isos_dxo_6d));
ref_electrons_5d3 = fit_electrons(electrons_photo_5d3, isos * 2.^(iso_difference/2));
ref_electrons_6d = fit_electrons(electrons_photo_6d, isos * 2.^(-iso_difference/2));
=>
ref_electrons_5d3 = 70099
ref_electrons_6d = 80927
ref_electrons_6d / ref_electrons_5d3 = 1.1545
Getting closer. The ratio is still a bit higher than what I expect from the resolution difference, and since I don't have any other hypothesis to explain it, I'll say the 6D's quantum efficiency is a little higher:
qe_ratio = log2(ref_electrons_6d / ref_electrons_5d3 * (5496*3670) / (5796*3870))
0.053998
So, 6D's quantum efficiency seems to be 0.054 EV better than 5D3's, and its ISOs seem to be calibrated at 0.055 EV lower.
With these numbers, the ISOs estimated from electron counts become:
5D3:
isos_720 = [ 100 163 326 644 1266 2519 5211 10277 ]
isos_1080 = [ 82 160 316 622 1243 2429 5059 10049 ]
isos_crop = [ 80 157 318 630 1241 2495 5094 9777 ]
isos_photo = [ 104 205 416 803 1635 3191 6461 ]
And 6D:
isos_720 = [ 100 220 363 668 1297 2491 4981 10201 ]
isos_1080 = [ 89 170 320 619 1222 2406 4924 10082 ]
isos_crop = [ 83 167 322 625 1244 2413 4866 9739 ]
isos_photo = [ 102 192 392 772 1552 3151 6367 14555 ]
Graphs:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F5d3-movie.png&hash=8b5a51e7004494587c74a2ab32b8a405) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F6d-movie.png&hash=74f9834f9afcd8dd635a920a52abfdf0)
To compare the two graphs, I think it's better to normalize the results to some reference output size. I'll normalize movie 1080p results to 1920x1080 and photo mode results to 8 megapixels.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F5d3-vs-6d-movie.png&hash=4dfab048cc508edcede334719b4e438d) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2Fsnr%2F5d3-vs-6d-photo-8mp.png&hash=74c3ab9ad1f051e2bcd8a06d6c9313b6)
Normaling the 6D results from 1792x1008 to 1920x1080 gives the 5D3 0.1 EV advantage.
Normalizing to 8 megapixels gives the 5D3 a 0.075 EV advantage (compare with DxO results):
norm_5d3 = log2(sqrt((5796*3870))) - log2(sqrt(8e6))
norm_6d = log2(sqrt(5496*3670)) - log2(sqrt(8e6))
norm_5d3 = 0.74370
norm_6d = 0.66708
5D3 normalized DR: 11.6637 11.5737 11.4337 11.2037 10.7637 10.1437 9.2337
6D normalized DR: 12.1471 12.1571 11.9471 11.6671 11.2171 10.5371 9.7771 8.7071
Complete octave script: dr.m (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/dr.m)
Please double-check my math, as I might be a little beyond Ballmer peak :)
-
Possible causes:
- systematic errors in my measurement method
- 6D uses lower ISOs than 5D3 (even if they both print the same value)
- 6D's sensor has a higher quantum efficiency
- a combination of the above
add:
- 6D's CFA lets more light pass (e.g. wider spectrum for the sensel filters)
-
I think we can include this one in the quantum efficiency.
Updated the graphs to include the shot noise limit (maximum dynamic range, if the read noise would be zero). At high ISO, you can see that, while in photo mode, the maximum improvement that can be obtained by reducing the read noise is around 1.2 stops for 6D and 1.6 stops for 5D3, in movie mode, with 3x3 pixel binning, the maximum theoretical improvement would be over 4 stops.
So... who said we are limited by shot noise at high ISOs? :P
-
How far are we from implementation, even as a pre-alpha?
-
I'll tell you after receiving the DR test results (http://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955) from at least 3 more cameras, and LV binning results (http://www.magiclantern.fm/forum/index.php?topic=10111.msg118232#msg118232) from at least 2 cameras, one of them being 5D3.
None of the tests require you to compile anything; you only need to take some test pictures in controlled environments.
-
I can help, have a 5Dmk3. What do I need to do?
-
I can help for 5D III and have a High Resolution chart and color checkers if that helps. I'll pop them out today.
Are you looking for detail cause I can use a nice macro lens and get close or does it not matter?
-
Doesn't matter; as long as you get the resolution chart details above the Nyquist frequency, it's fine.
-
...Nyquist frequency...
Even after looking it up I'm still lost, so yeah ummm... I'll take some pics/video. I'll assume for 720/1080 I should use Canon .mov or RAW?
-
Silent pictures, with "Silent zoom bracket" from raw_diag. That way, there will be no camera movement at all.
-
After looking at it I get it now.
-
Hope they help...
-
Possible causes:
- systematic errors in my measurement method
- 6D uses lower ISOs than 5D3 (even if they both print the same value)
- 6D's sensor has a higher quantum efficiency
- a combination of the above
Hi Alex,
Some of my data, shot on the 6d, is done with aperture f2.0
Later on I remembered reading somewhere on the forum here that canon automatic adds some gain by apertures below f2.8
Can this explain some of the differences measured between 6d and 5dIII
-
Also...
The 6d is known to have less banding noise then 5d3.
Is banding noise caused by "wiring" on the cmos ?
So does less banding noise mean, less "wiring" on the cmos chip, which results in more room to capture electrons... ???
-
I don't expect that gain to influence the results, unless it's clipping useful data (and that happens beyond f1.4 IIRC). And in that case, the reported DR would be a little lower.
If in doubt, you can double-check a few of the numbers.
I don't know the cause of the FPN, but I'm making progress in understanding how Canon corrects it.
-
5D3: Pixel pitch= 6.25 microns.
6D: Pixel pitch= 6.58 microns.
Bigger pixels are like bigger buckets, take longer to fill, but hold more water. Data from Rodger Clark.
I wasn't ware of resolution playing a role, per se. Since resolution is a count of the number of pixels, not how much data each pixel captures. What am I missing?
-
Yes, but still, the 6D holds even more water than you would expect from different bucket sizes.
-
Well if we assume that the reported pixel sizes are correct, and that Canon hasn't developed a way to squeeze photons, then it must be electrical efficiencies.
The only other possible causes I can think of.
Shielding/copper efficiency: less electrons lost during transfer.
A/D improvements. They both have the same 14bits of accuracy, but for all we know, Canon may be simply clipping some data in the 5D3.
Need to loan DxO' ISO rig. :D
add:
- 6D's CFA lets more light pass (e.g. wider spectrum for the sensel filters)
I'm not sure how this one can play a role, since this doesn't effect what happens after the photons are in the wells.
-
In the wells you get electrons, not photons.
-
Where do the photons become electrons?
I always thought the wells captured photons, like buckets capture water, and then in simple terms, those photons are counted and converted into electrons.
-
Somewhere before the electronic amplifiers. These amplifiers gets saturated, from too many electrons.
(it might be different at the lowest ISO, not 100% sure, but at higher ISOs, the highlights are clipped in by the amplifiers)
There is a ratio between the number of captured photons, and the number of electrons that reach the amplifiers; that one is called quantum efficiency, is usually less than 1:1, and I don't have a way to measure it. My method only counts the number of captured electrons.
http://www.clarkvision.com/articles/digital.sensor.performance.summary/
-
You can determine accurate exposure time with the FPS registers now?
So you need a way to accurately measure the photons emitted per time period for your light?
-
In photo mode, I can determine the time when shutter is "open" electronically, which is usually longer than the actual exposure time (limited by mechanical shutter). So, I can determine the exposure time with better accuracy by recording the sound of the shutter :P
Measuring photons... is way beyond my knowledge.
-
Counting photons may be something SpcCb can share advice on.
-
5d3 has a CMOS chip which is capable of vertical pixelbinning, so it doesn't need vertical lineskipping like the 6d does in video mode.
I assume, that vertical pixelbinning is only possible by more copper-wiring/infrastructure within/on the cmos chip, which results in a lower full well capacity ???
I wasn't ware of resolution playing a role, per se. Since resolution is a count of the number of pixels, not how much data each pixel captures. What am I missing?
Bigger pixels on the same area (full-frame), means bigger wells, higher full well capacity.
I think it's the same like the 3x3 pixelbinning that the 5d3 does, full well capacity raises in videomode by a factor 9.
So if you had a 2 megapixel full-frame chip from this generation, you're full well capacity would be enormous, like 900000.
-
Yeah I'm pretty sure I'm just being pedantic, since resolution doesn't care about well depth.
Now I'm going to assume Canon's full-stop ISOs are pretty much correct, but I'll keep the relative differences measured by DxO between the two cameras. That difference is...
octave:1> isos_dxo_5d3 = [ 80 160 323 641 1280 2518 5179 ];
octave:2> isos_dxo_6d = [ 80 153 311 616 1210 2400 4991 ];
octave:3> isos_dxo_5d3 ./ isos_dxo_6d
ans =
1.0000 1.0458 1.0386 1.0406 1.0579 1.0492 1.0377
octave:4> log2(mean(isos_dxo_5d3 ./ isos_dxo_6d))
ans = 0.054522
=> 6D ISOs are 0.055 EV lower than 5D3's.
But only at gains higher then base? Since the base gain is generating the same ISO on both cameras, this should mean that they capture the same amount of photons. Or rather, that since the 6D has an higher rated well capacity, that it generates more electrons for the same number of photons. Since ISO based on saturation, should be representing a fixed photon count, right?
edit: Nevermind, I reread your post.
-
Don't know if this is correct, but the way I think about it is:
A full frame CMOS chip has han area of 36mmx24mm= 864mm2 to catch light.
Since CMOS has circuitry inside the chip, this circuitry is blocking light (that's why backside illuminated CMOS sensors are slightly better in catching light, backside illumination prevents the circuitry from blocking light).
By normal CMOS, the pixel wells are below the circuitry.
So due to circuitry the active area of CMOS full-frame is a little lower then 864mm2
Now my assumption is, that the 5d3 CMOS chip has more/bigger circuitry, (it can do some more tricks than the 6D CMOS, like 3x3 pixelbinning).
-
Yes, but a fixed ISO rating from DxO should be saying, for a constant N of photons, the 5D3 generates N electrons, and the 6D generates N electrons, at full well capacity. By measuring the ISO at saturation, I believe this negates any influences by sensor filters. With the sensor filter removed, this should lower the ISO rating, for the same quantum efficiency.
Since we know that even the base gains have some (ISO) headroom between the default gain, and actual well saturation (and seems to vary with camera), this should probably be taken into account. And should drive the accuracy of the quantum efficiency results higher, since it's less affected by downstream influences.
If you're looking at sensor efficiency, rather then camera efficiency, I guess.
-
Ah I get what you mean, the circuitry has the same effect as any difference in the color filter array.
But since things are measured with saturated pixels, the filters/circuitry don't have effect on the end results.
So the difference in full well capacity between the 5d3 and 6d is not due to difference in color filter array or circuitry.
The way Alex measures the full well capacity, what does the result mean:
-Is the full well capacity result the best possible capacity a pixel can have.
or
-Is it an average amount, like all electrons divided by the number of pixels.
?
Since I don't believe every pixel has exact the same full well capacity, or has it ::)
Now I think about it, maybe that's the reason why canon is staying on the save side with ISO/dynamic range.
-
So the difference in full well capacity between the 5d3 and 6d is not due to difference in color filter array or circuitry.
No, because full well capacity is a measure of the maximum number of electrons generated. And the photon to electron witch craft happens after the filter array.
The way Alex measures the full well capacity, what does the result mean:
-Is the full well capacity result the best possible capacity a pixel can have.
or
-Is it an average amount, like all electrons divided by the number of pixels.
?
Since the maths doesn't account for the number of pixels, I assume it's a measure of the entire sensor.
edit: That doesn't make sense since there are significantly more pixels then the electron count.
http://www.clarkvision.com/articles/digital.sensor.performance.summary/#full_well
Since I don't believe every pixel has exact the same full well capacity ::)
I believe flat fields (http://en.wikipedia.org/wiki/Flat-field_correction) will correct variations in pixel output.
-
....
- 6D's sensor has a higher quantum efficiency
...
Theory:
- photon to electron ratio = quantum efficiency (assumming it's a sensor constant)
- equal ISOs = equal number of photons per area (light required to saturate the sensor)
...
I think that equal ISOs = equal relative saturation (electrons counted/full saturation level ).
At least, Dxo's measures are using this
http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity
"... relates sensitivity to the exposure necessary to saturate the camera"
Higher QE gives higher "Dxo measured ISOs" for the same electron capacity ..
-
Exposure, in this context, is proportional to the number of captured photons per area (in the sensor's plane). On the same sensor, this translates, indeed, to equal relative saturation (electrons counted / full saturation level), but on different cameras, at the same ISO, both sensors would saturate at the same level of incoming light (photons), if the shutter speeds are equal.
A sensor with higher QE can be used with lower internal amplification, so the ISO value measured by DxO can be lower. If QE's were equal, I expect DxO ISOs to be even lower on 6D, because the difference in electron capacity is quite high (higher than what you would get from different pixel sizes).
-
Alex, as I understand it, DxO does not measure the amplification but the output sensitivity. i.e. for an A level of illuminance how high in the 0-1 scale (0=black - 1=full saturation) is the output in raw levels and compares it to the nominal value.
Say for the same illumination
5D4 has QE=50% and saturation at 10000 electrons/square micron. Gives as output 0.5 in the 0-1 range (white clipping level - Black level) which corresponds to 5000 electrons
6D2 has QE=55% and saturation at 11000 electrons/ square micron. Gives as output 0.5 in the 0-1 range which corresponds to 5500 electrons
the two cameras will have the same dxo-measured ISO
-
And that level of luminance is proportional to the number of photons.
I'm not sure where you are trying to get.
-
I'll tell you after receiving the DR test results (http://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955) from at least 3 more cameras,
Finally sunny after 3 days of rain and overcast/100% humidity good day for HDR... if my last attempt to help with LV Binning wasn't a dud I'll give this one a shot today. Should I bother? :)
-
Last attempt to help with LV binning? Where?
I don't see much else, other than...
After looking at it I get it now.
Hope they help...
-
I PMd you a link with a Dropbox .zip 100 megs of shots that day...
-
Dope.. sent it to alex not a1ex
Pm comming from dummy now...
-
Thanks, received now.
-
Pixel binning test results moved here: http://magiclantern.fm/forum/index.php?topic=16516.msg160832#msg160832
-
Glad it worked out... I'll do the other test and get you results for tomorrow...
-
And that level of luminance is proportional to the number of photons.
I'm not sure where you are trying to get.
Forgive me :) .. while my eyes were reading "photons" my brain translated it to "photoelectrons" ..
My point is that QE cannot affect the full saturation level ..
I will reread the thread and come back if I have something useful to say ..
-
Yes, QE alone can't affect the saturation level, but if you increase it and you want to keep the same ISO, you will have to turn down the amplifier gains - and this increases the full well capacity.
-
500D, trying to increase the resolution of the video raw (crop mode).
Default :
http://s29.postimg.cc/ugjr8rxet/res1x.jpg
I changed (FPS timer A, C0F0[6084], C0F0[6088])
http://s29.postimg.cc/isuaryxnp/res2x.jpg
Add 512px width :
http://s4.postimg.cc/s4sbor8iz/res4x.jpg
Add 2048px and shift :
http://s16.postimg.cc/mpui9r4sz/res5x.jpg
Full width :
http://s27.postimg.cc/be7g4qaz5/full_width.jpg
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.cc%2Fdtk387c7b%2FVRAM0.png&hash=6e5a2d8c1d591d4b082ec824b2cb7579) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs29.postimg.cc%2Fmpzx99953%2FVRAM1.png&hash=38efe08a350fcc6623f710aac3f9c66a)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.cc%2F6uvo2lhrh%2FVRAM0.png&hash=581f71c1514c7d53916ac91afb30d80b)
Trying to change the height (photo mode):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs18.postimg.cc%2F4tp8g2zyh%2Fob_zones.png&hash=5ab84dc4b16a91b7e9ae19733af5d256) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs18.postimg.cc%2Fursum45ft%2Fob_zones2.png&hash=674fe9a254c279807dff57f569ca3979)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs18.postimg.cc%2Fgzefqheo9%2FVRAM0.png&hash=7bf99b05ca3eee81a9af7eb2ed0bbc6f) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs18.postimg.cc%2Fibw4sdc3t%2FVRAM1.png&hash=63c4a986338fc0d176cd2b104dbf7af5)
-
Well it took me forever to realise that i had to put "image review" to "HOLD" :-\
But finally:
SNR for 550D ISO 100 Photomode:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fu%2F90483500%2FMAGICLANTERN%2F550D-ISO100-50-Photo.png&hash=aefad8fdd49fb5569199aa89a972218e)
SNR ISO 100 Moviemode
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fu%2F90483500%2FMAGICLANTERN%2F550D-ISO100-50-Movie.png&hash=2facb3e6fdda824b29a5ee80c8c113c5)
SNR ISO 1600 Photomode
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fu%2F90483500%2FMAGICLANTERN%2F550D-ISO1600-50-Photo.png&hash=d9fa28b6cc1087fc2870d79f84148e9e)
I hope those above make sense
-
The first image is almost perfect, the second is incorrect (very poor coverage in highlights), the third is fine, but could have better coverage in shadows.
If you can do a complete set (photo, movie 1080, movie 720, and 5x zoom), would be great.
Side note: I will also use this data for optimizing exposure choices in ETTR.
-
A1ex, if I shot this out in the sun with a full set for 5D III would it suffice?
Spyder Cube
https://search.yahoo.com/search;_ylt=AwrBTzgKP9lTAhIA.x5XNyoA?ei=UTF-8&type=599486&fr=chr-greentree_ff&ilc=12&p=spyder+cube&SpellState=&fr2=sp-qrw-corr-top
Specular Highlights, White, Black Trap, 18% Grey
-
I'm looking for continuous coverage from deep shadows to clipped highlights, and static scene (lack of motion or light changes is very important, so natural light variations may cause issues).
Shooting a light bulb with something dark near it, and really out of focus, should do the trick. Also Audionut explained his setup a while ago (a picture would be nice).
-
OK, I'll put that cube next to a light bulb and lock everything down like the last test.
Any time I've used that cube I've gotten blown highlights and pure black.
-
I setup a nice little scene to shoot yesterday but when I powered up the camera I got an API warning for RAW_DIAG version 5 expected 6... is there one online I can use?
-
Just download it again.
-
Thank you Sir... :)
-
OK, I'll put that cube next to a light bulb and lock everything down like the last test.
Any time I've used that cube I've gotten blown highlights and pure black.
The Datacolor SpyderCube is one of the neatest little devices I've come across. Overpriced, however.
BTW, it isn't difficult to make your own:
- small box, like what smaller Canon lens come in
- cut out hole, and paint interior black for the "black trap"
- maybe put in black unmatched sock
- put stainless steel spoon next to it for spectral highlights.
- gray gaffers tape is decent for gray card
http://berean.zenfolio.com/dual_iso/h101028e1#h101028e1
http://berean.zenfolio.com/dual_iso/h101028e1#h3e0c323b
-
On the 6D I'm only seeing ADTG2, CMOS and Digic, no ADTG4/6. Do you think this is 6D specific, or simply wrong/missing stub?
-
Right, I see the same thing in the logs. Probably the 6D has only one chip, while most other cameras have two?
-
Did some testing with the "adtg_gui.mo" on 6d.
Changing the iso in canon menu gives the following numbers for CMOS[3]
Weird thing is, why doesn't use iso 6400 0x66 ? (checked it twice, it really uses 0x77)
And do you know what will happen if I override it to 0x88 or 0x99 ??? Will my precious cmos chip catch fire :D
And how do you get the iso's lower than 100, it's not done in the CMOS register is it, how to approach ?
ISO CMOS[3]
100 0x0
200 0x11
400 0x22
800 0x33
1600 0x44
3200 0x55
6400 0x77
12800 0x77
25600 0x77
Hi1 0x77
EDIT: when I override it to 0x66 it shows ISO6400 in the exif, but I get exact same exposure results as ISO 800, weird :o
-
These are some "hidden" ISOs - 5D3 has them too (http://www.magiclantern.fm/forum/index.php?topic=10111.msg99665#msg99665), and a few of them seem cleaner then Canon's (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103971;topicseen#msg103971).
Can you do a dynamic range test for all of them? (just like you did before). Set ISO 100 in Canon menu, use a manual lens, then set CMOS[3] to 0x00, 0x11 ... 0xFF => 16 ISOs (all based on Canon's ISO 100, and we'll only change one parameter).
I'd also like the same test on 5D3. Here you would set CMOS[0] to 0x003, 0x113 ... 0xFF3. Don't change any other settings, just ISO 100, manual lens, and this testing procedure (http://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955). If you can't compile ML yourself, ask for a build here (http://www.magiclantern.fm/forum/index.php?topic=12608).
These ISOs are applicable in both photo and movie; you may test any of these modes, but if you can do all of them, that's even better.
-
Did test some cmos[3] overrides, from 0x00 to 0xff (CMOS did not catch fire :D).
CMOS[3] override on 6d
Setting ISO according to full well capacity numbers.
0x00 100
0x11 200
0x22 400
0x33 800
0x44 1600
0x55 3200
0x66 640 :o Weird one, full well capacity is between iso 800 and iso 400, looks like a "native" iso 640.
0x77 6400
0x88 100
0x99 200
0xaa 400
0xbb 800
0xcc 1600
0xdd 3200
0xee 1250 :o Weird one, full well capacity is between iso 800 and iso 1600, looks like a "native" iso 1250.
0xff 6400
The ISO settings that aren't used by Canon(the bold ones) don't seem much better/cleaner/different (But I did the test "Quick and dirty" so my measurement may not be accurate enough)
SNR curves are available on my google drive:
https://drive.google.com/folderview?id=0B1BxGc3dfMDabkZybDB6YldXR3M&usp=sharing
-
Ha, that was fast.
0xFF even seems to be a bit cleaner than 0x77, but it would be nice to have some more data points in the dark areas on the graph.
The dynamic range is from the intersection between SNR curve (red line) and the X axis at 0 EV (the bold one) - that is, from the point where SNR = 1, until the far right on the X axis (look for the last red pixel on the curve), where you have the white level (clipping point).
-
I remeasured the 0x77 and 0xff with more shadow points, now the difference is even smaller then before.
See the "remeasured" files on google drive:
https://drive.google.com/folderview?id=0B1BxGc3dfMDabkZybDB6YldXR3M&usp=sharing
-
Some other stuff for 6d
CMOS[0] Default value is 0x593, changing it seems to do nothing.
CMOS[1] and CMOS[2] seems to be used for vertical and horizontal offset (didn't test it though)
CMOS[3] controls the analog iso (see posts above)
CMOS[4] on the 6d seems to give control over (digitally?) pulling back a few stops.
CMOS[4] default value = 0x4f0
CMOS[4] set to value 0x8f0, for the first digit, 4 and 8 can be used and it seems that 8 pulls back the whole histogram by 2 stops. (Is this Canon's Auto lighting optimizer ?)
CMOS[4] set to value 0x800 seems to only pull back the highlights, so the second digit can give use some highlight pulling/crushing
CMOS[5] didn't test it yet, but probably does control vertical squeeze
CMOS[6] gives control over horizontal squeeze, not interesting.
Although it could be used to squeeze an 16:9 to a 4:3 ratio and record in 4:3 ratio and stretch it back to 16:9 in post, sort of fake anamorphic format(means lower bitrate for 16:9 aspect ratio and 1.33x stretch).
EDIT (The squeeze is probably used for smaller raw types, sRaw ?)
CMOS[7] and CMOS[8] gives weird dark frames/blackframes/readoutframes stuff ?
CMOS[8] default value 0xa
CMOS[8] value 0xf and 0x7 seems to give some blackframe or dark frame or something
-
Some other interesting registers, don't know what normal values are for them and what values I should try, they do however change with iso setting, so maybe some fine-tuning for iso ?
EDIT: these four registers, seems to be some amplifiers and are per color channel, the first two are Red and Blue(or vice versa), the last two seems to be the two Green channels
EDIT 2 :Looking at the pictures now in lightroom, every 4th pixel is missing, so it's not used per color channel.
ISO ADTG2[8882] ADTG2[8884] ADTG2[8886] ADTG2[8888]
L1 0x416 0x416 0x416 0x417
100 0x416 0x416 0x416 0x417
200 0x400 0x400 0x400 0x400
400 0x422 0x422 0x423 0x423
800 0x42d 0x42d 0x42e 0x42e
1600 0x440 0x440 0x441 0x442
3200 0x485 0x484 0x484 0x484
6400 0x4a9 0x4a8 0x4a7 0x4a7
12800 0x952 0x950 0x94e 0x94e
25600 0x12a4 0x12a0 0x129c 0x129c
Hi1 0x12a4 0x12a0 0x129c 0x129c
-
Those are the registers this entire discussion is about. They control some amplifier stage that is between CMOS and ADC. We can get about 0.5 EV more DR by turning them down. You might want to read the first couple of pages of this thread at least.
-
Read about the first ten pages of this thread, but didn't know it was about exact these registers.
Since the 5d3 has a nice module "iso_regs.mo" - 5D3 only
A lot of talk is only about one ADTG register, not 4 of them. So a bit confusing for me :-[
But now I start to get the idea a little, the trick is to pull these numbers down.
They're in the 400 range, 250 seems to be a good starting point...
The sweet spot has to be found by trial and error with the raw-diag tool I Guess ?
-
EDIT: these four registers, seems to be some amplifiers and are per color channel, the first two are Red and Blue(or vice versa), the last two seems to be the two Green channels
EDIT 2 :Looking at the pictures now in lightroom, every 4th pixel is missing, so it's not used per color channel.
You might be interested in this description (http://www.magiclantern.fm/forum/index.php?topic=12789.msg123345#msg123345) that I wrote in another thread. Because of how the colors are distributed between the vertical lines, you are somewhat correct in that the red pixels are affected by only 2 of the 4 registers (and the blue pixels are affected only by the other 2), but all 4 registers will affect green pixels.
For the purposes of the investigation in this thread, we've been setting all 4 registers to the same value, but Canon has some fine-tuning differences (as you can see in your table), which were presumably/hopefully determined from their calibration of the electronics.
-
Juicing up the battery now, so will try some more later on :D
But theoreticaly , some very low iso's could be achieved by lowering the 400 number in these registers.
Has anyone tried using the number 50 in these registers ? Did it look like an ISO 12,5 ?
-
That won't accomplish what you're thinking because the ADTG/DFE amplification comes after the CMOS amplification. The brightness at which you saturate in the first stage will still be saturated in the second stage, so you really can't reduce the gain of the second stage all that far before it becomes pointless (the effective ISO stops decreasing).
-
Hmmm.... :(, shall try later on, and see the results...
Other suggestions to get low ISO ?
-
Here are some curves from fake iso's below 100 (with lower ADTG gain).
https://drive.google.com/folderview?id=0B1BxGc3dfMDaU1NJd0dHUWNsNGc&usp=sharing
Used iso 100 as base iso and set registers to 300, 250, 200, 100 and 25.
Creating ISO's of about 75, 63, 50, 25 and 6.
Did get pink highlights, and it seems that highlights are cut off(by about using 250(ISO 63) or something, have to try some real photo's with it, but I think you could get away with ISO 25 if you want.
Seems like 275 is the number to go with if you care about dynamic range.
https://drive.google.com/folderview?id=0B1BxGc3dfMDaNGszU2g5aHBfTUE&usp=sharing
Did curves for ISO 200, 800 and 6400.
Dynamic range with ISO 200 -> 11,80 eV
Dynamic range with ISO 800 -> 11,36 eV
Dynamic range with ISO 6400 -> 9,60 eV
EDIT: Didn't alter any black or white levels, should I do that :-\ and where do I do that ?
-
This "adtg_gui.mo" is very addicting.
Is there some overview page where known registers and the effects they have are reported ?
-
Hit page 1 of this thread and start reading :P
My quick observations show that we will probably only get 0.4EV on the 6D. ADTG [FE] is at 0 by default.
However, I'm seeing 11.4 EV of DR at stock, so 11.8 EV is decent enough I guess.
-
Hit page 1 of this thread and start reading :P
Was already afraid that this topic is the way all findings in these registers are organized...
The 0.4 eV extra is what I get too, fits perfectly with the theoretical values graph A1ex posted before:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fiso50%2F6d.png&hash=20433025b67889809e94918894ee99f9)
I'm also messing around a lot with the 8 CMOS registers.
CMOS[0], I have no idea what it does.
CMOS[6], is set on 400 in video mode (3x pixelbinning horizontal ?), setting it to 800, gives a stretched image and shows real pixels horizontal, if you make the image 3 times smaller horizontal in post, the aspect ratio is normal.
CMOS[7] seems to be used by switching from 1080p to 720p...
Have you done some other register testing ?
-
After finding enough time to read most of this long (and quite intimidating ;)) thread I have done my first tests in my 500D using the iso-research branch and adtg_gui module.
I can play modifying C0F0[819c] (SaturateOffset) and also CMOS[0] (to change analog ISO), and everything seems to work.
But if I try to modify C0F0[8034] (BlackLevel/BW offset) my camera hangs. As soon as I press "Set" button to modify it, the menu screen gets frozen, I cannot move with arrow keys, or exit with LV button. The shutter button works, but only to get a blank screen, and I cannot shutdown the camera, I need to remove the battery.
I am missing something? ( do I need a specific camera setup?). It happens equally with manual or EF lenses.
The procedure I have followed is:
1 - "hg clone" the iso-research branch.
2 - modify "Makefile_user" with CONFIG_GDB=y and Makefile.modules.default to include adtg_gui and raw_diag in the build.
3 - modify "adtg_gui.c" to include missing values for 500D, "ENGIO_WRITE_FUNC" and "ENG_DRV_OUT_FUNC"
else if (is_camera("500D", "1.1.1")) // http://www.magiclantern.fm/forum/index.php?topic=6751.msg70325#msg70325
{
ADTG_WRITE_FUNC = 0xFF22F8F4; //"[REG] @@@@@@@@@@@@ Start ADTG[CS:%lx]"
CMOS_WRITE_FUNC = 0xFF22F9DC; //"[REG] ############ Start CMOS"
ENGIO_WRITE_FUNC = 0xFF190CF4;
ENG_DRV_OUT_FUNC = 0xFF190B84;
}
I have copied these values from stubs.S, and double checked them, comparing the stubs with ROM dumps from other cameras, the stubs are Ok.
With these values adtg_gui is able to find DIGIC registers. Everything seems to be working fine ... but I cannot edit C0F0[8034] :(
And according to Greg's posts (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104136#msg104136) in this thread, modifying C0F0[819c] and C0F0[8034] is the only way so far to get some DR gain on 500D (0,33EV).
Any thoughts? I am doing something wrong?
-
I know this problem. Number of known regs is too large.
The easiest solution to remove unnecessary in adtg_gui.c
-
The easiest solution to remove unnecessary in adtg_gui.c
Where? Are you refering to this struct list, at the beginning of "adtg_gui.c" ?
/* todo: check for duplicates */
static struct known_reg known_regs[] = {
{DST_CMOS, 0, 0, "Analog ISO (most cameras)"},
{DST_CMOS, 1, 0, "Vertical offset"},
{DST_CMOS, 2, 0, "Horizontal offset / column skipping"},
{DST_CMOS, 3, 0, "Analog ISO on 6D"},
{DST_CMOS, 4, 0, "ISO-related?"},
{DST_CMOS, 5, 0, "Fine vertical offset, black area maybe"},
{DST_CMOS, 6, 0, "ISO 50 or timing related: FFF => darker image"},
{DST_CMOS, 7, 0, "Looks like the cmos is dieing (g3gg0)"},
{DST_ADTG, 0x8000, 0, "Causes interlacing (g3gg0)"},
...
-
Yes, this list is too long.
-
Yes, this list is too long.
Done, and it works now. Thanks, Greg ! ;)
-
@Levas
I'm pretty sure a1ex' graph is only based on the ADTG [888x] bank of registers, which fits in with my quick test. Based on my past experience, 0x34x seems about right for these registers @ ISO 100, but I haven't confirmed the results. I've been looking for other registers that stand out as useful, but so far no luck.
-
The preamp registers on 6d
ADTG2 8 ADTG2 9 ADTG2 a ADTG2 b
Normal value about 0x4b (75 decimal)
If you set them to 0x190 (400decimal) you get 2 stops extra (noisier ofcourse, but nonetheless 2 stops extra)
Now for video, where the ISO limit is 25600, you can set these 4 preamps to 190, and you have ISO 102400 video mode 8)
And a lot of vertical lines across the frame(CMOS pattern), so blackframe extraction in post must be done to get usable results.
Will try it if it get's dark tonight :D
EDIT:
If you use the other amplifier, ADTG[fe] (standardvalue=0, maxvalue=f)
And set it on max value f, gives another extra stop.
So ISO 204800 videomode :D
Tried it out, video looks like crap, noise is all over the place ;D
-
500D
1080p ADTG2[100c] = 0x2
720p ADTG2[100c] = 0x4
vertical skipping lines (video mode)
0x2 0x4 0x5
RG+ RG+ RG+
GB GB GB
RG RG RG
GB+ GB GB
RG RG RG
GB GB+ GB
RG+ RG RG+
So 0x1 and 0x5 bad colors (confirmed preview live view)
0x0 give an image of 1920x1080 (distorted) -> scaled post process 1920x360 without moire (less noise).
To work 0x0, need tune CMOS[5]
-
@Greg, what do you mean with "To work 0x0, need tune CMOS[5]"
What values are normal in CMOS[5] on 500d ?
And where do you set them too, to get a value of 0x0 to work in video mode ?
-
See in crop mode when change the height of the focus box (change the sensor area).
CMOS[5] value will be different/new.
ADTG2 [100c] = 0x0 // no skipping vertical lines, so we need to set which part of the sensor has to work
-----------------------------------------------------------------------------------------------------------------------------------
In this way we can get crop H264 video on all cameras :
ADTG2 [100c] = 0x0 // no skipping vertical lines
CMOS [4] = value of the crop mode (center) // horizontal lines?
CMOS [5] = value of the crop mode (center)
The image is not good, horizontal pixels (3) still averaged or something :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs3.postimg.org%2Ffnvaiv0ar%2Ftest_crop.jpg&hash=455e22be43a25a3df282698dc43b4ef0)
-
For the 6d
ADTG2[800c] = vertical lineskipping, 0(crop), 2 (1080p) and 4(720p)
CMOS[6]
CMOS[7]
Overriding ADTG[800c] = 0
CMOS[6] = d0
CMOS[7] = 208
Gives me correct live view preview for crop mode on 6d, so it gives 3x zoom (only white border in the bottom visible... :-\)
-
Gives me correct live view preview for crop mode on 6d, so it gives 3x zoom (only white border in the bottom visible... :-\)
On the 500D also, maybe it is caused by raw size
1080p = 1590x1060
crop = about 2000x838?
So lack height.
ADTG2[1000] horizontal (column) pixels averaging
1080p ADTG2[1000] = 0x6
crop ADTG2[1000] = 0x5
Now look good:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2Fqvaqvvvp9%2Fcrop_test2.jpg&hash=8c1801906c8b80f2ef8ada44c75a5037)
-
Does anyone know where the horizontal pixelbinning is done ?
I'm curious if it's a baked in setting in the CMOS chips (then it's probably controlled via CMOS[6] or CMOS[7] on the 6d.)
Or could it be a setting which tells to average 3 pixels of the same color ?
These registers are changed if you change the view from 10x zoom to standard view (1080p25)
1080p25 10x zoom(no pixelbinning, probably stretching pixels for live view)
ADTG2-805f 2b0 13c Edit: These values seems to vary within days, so not hard values.
ADTG2-8061 2b0 13c Edit: These values seems to vary within days, so not hard values.
CMOS[6] 400 d0
CMOS[7] 0 208
ADTG2-802c 0 110
ADTG2-8047 550 110
ADTG2-82f3 c 319
ADTG2-8820 303 301
ADTG2-8826 e04 6
ADTG2-8827 42 4
ADTG2-8828 3010 301
ADTG2-8860 f e
ADTG2-8866 7 f
-
Canon 6d Video mode Video mode Photomode Photomode
1080p25 10xzoom Live view off Live view on
ADTG2-802c 0 110 110 0
ADTG2-8047 550* 110 110 550*
ADTG2-8860 f e d f
ADTG2-82f3 c 319 211 8
Looks like ADTG2-802c and ADTG2-8047 do sort of control if every pixel horizontal is shown or that it does 3x pixelbinning
ADTG2-8860 has a unique settings for 1080p/10x zoom and live view-off, has anyone figured out what f, e and d setting does ?
Edit: Also ADTG2-82f3 has unique values for all modes
EDIT: *Could 550 mean, take 5 pixels wide for the first color, for example red(rgrgr) and 5 pixels wide for green (grgrg) and bin them together, this would give 3x pixelbinning... When my battery is recharged I'll try a value of 330 (Could mean pixelbinning of 2 pixels horizontal, gives a readout of 2736 pixels horizontal instead of 1824(binning of 3pixels)
Nope, doesn't work,gives funky live view colors and bottom half of the frame is placed at the top, nothing usuable
-
I add 0x60 to CMOS[5]
Now there is no white border in the bottom :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs23.postimg.org%2Ft9y9ccsgb%2FVRAM0.png&hash=e34ac485ca2ee69ae3060c4aa1babdc7)
Now we need to understand DIGIC scaling.
Raw (crop mode) vs H264 (1080p - ADTG/CMOS crop mode):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs14.postimg.org%2F6oupf7oj1%2Frawvs.jpg&hash=ed37c0d6450b6da6a7a0027c9656ae4e)
H264 is upscaled.
Now I can record crop raw 1590x1060, height increase :o
Default crop mode is 2000x838
-
Crop video with a good preview/live view? Possible! ;D
Is not perfect, but near. We need tune DIGIC scaling regs.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs28.postimg.org%2F40djk12wt%2FVRAM0.png&hash=8a3e9c39fe409340c5a9c68459157981)
mlv_rec frame - http://s28.postimg.org/w1rkxq871/testcropp.jpg
-
Ale zajebiscie :)!
Did you actually found out how to get a centered 3x zoom mode with a good preview?
-
Amazing find!
-
The camera work in 1080p. Sensor work in crop mode.
So the preview is almost good, you can compare with my previous post.
This solved the problem of pink preview on old cameras.
Raw buffer is 1080p mode, so we have a different resolution, only 5D3 will give you 1920x1280.
I think it can be changed, but it requires a lot of research (when you change the size of the raw, preview Canon freeze).
-
I'm not sure if I understood correctly. Are you saying that 5d3 crop mode will display realtime liveview at 1920x1280 but not 1920x1080? Is that right?
-
@Greg, can you share the data of the values and registers you changed ?
ADTG2 [100c] = standard value = 2, override value = 0 (lineskipping register)
CMOS [4] = standard value = ?, Override value = ? (value of the crop mode (center) // horizontal lines?)
CMOS [5] = Standard value = ? Override value = ? (value of the crop mode (center))
ADTG2[1000] = Standard value = 6, Override value = 5 (horizontal (column) pixels averaging)
Can you share the values for the CMOS[4] and CMOS[5] ?
You said you'd also change some digic registers for scaling, can you share those registers and values ?
-
@MA Visuals,
500D in 1080p mode has 1590x1060, crop trick 1590x1060 (same buffer). The camera still thinks that it is in 1080p, but the sensor give the crop area.
So this way only 5D3 can 1920x1280. I mean raw sizes.
@Greg, can you share the data of the values and registers you changed ?
I discharged all batteries 500D, I'll try tomorrow.
You said you'd also change some digic registers for scaling, can you share those registers and values ?
I'm still looking for those registers. This is only needed for more accurate preview or H264 video crop.
-
500D
1080p crop-mode(5x) 1080p-trick-crop
ADTG2[105f]N 0x0 0x38f 0x38f //shutter blanking for 5x/10x zoom
ADTG2[1061] 0x67f 0x0 0x0 //shutter blanking for 1080/720p mode?
CMOS[4] 0xc0d 0xc85 0xc85
CMOS[5] 0x400 0x609 0x689
ADTG2[100c] 0x2 0x4 0x0 //vertical skipping lines, not needed?
ADTG2[1000] 0x6 0x5 0x5 //vertical skipping, horizontal binning?
-
Thanks Greg,
I can get 3xzoom/crop mode on 6d by overriding the following registers:
1080p 10xzoom 3xzoom(1:1 crop)
ADTG2-805f 2b0 13c No change needed
ADTG2-8061 2b0 13c No change needed
CMOS[6] 400 d0 d0 (=Changing the "d" moves horizontal position on the sensor)
CMOS[7] 0 208 206 (Lowering the 8 to 6 moves the white bar outside 16:9 view)
ADTG2-800c 2 0 0 (vertical lines to skip)
ADTG2-8000 6 5 5 (Don't know what it does, if it is 6 you get some vertical interlaced looking footage...)
With these settings I get 3xzoom/crop mode, correct preview in live view on the 6d
Raw recording works, gives correct framing.
I can even record canon "native" h.264 with it, and it works too!
-
wow hats off for taking so much time and efforts to help.
-
With these settings I get 3xzoom/crop mode, correct preview in live view on the 6d
Raw recording works, gives correct framing.
I can even record canon "native" h.264 with it, and it works too!
Curious. Do you get a centered 3x zoom mode or is it on the left or right side?
-
Still wonder how to increase edmac buffer or something to increase the size of the crop raw.
A few pages previously, I found how to get the full width of the sensor - http://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909
I would get a crop raw 1500x1500px (so it will be less FPS). Here need to find something else than ADTG/CMOS. I do not know what.
Sensor say "I'm hot! Give me a break!" :o
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs9.postimg.org%2Fna44d5jj3%2FVRAM1.jpg&hash=6e55385230878e645942c0cb2b047bbc)
-
Curious. Do you get a centered 3x zoom mode or is it on the left or right side?
I can move it horizontal in 16 steps (0 to f hexadecimal) with the first "digit" value in CMOS[6]
CMOS[6] d0 (=Changing the "d" moves horizontal position on the sensor)
-
Still wonder how to increase edmac buffer or something to increase the size of the crop raw.
Have you tried setting ADTG2-100c on 0 in 1080p mode ? (override it and switch between photo and videomode to "reset" video live view to do 0 line skipping.
On 6d I have to use CMOS[7] to center the image on the live view screen (white borders above and below it), on your 500d it probably is CMOS[4] or CMOS[5].
I can get my 6d to do zero vertical lineskipping in 1080p mode, only it does 1/3 of the sensor height, probably has to do with the edmac buffer size, like you say, increase the edmac to increase the frame height. So I get a stretched image with real vertical pixels and horizontal pixelbinning of 3 pixels.
For those of you with a 6d
Go to 1080p25 video mode
Override registers ADTG2-800c to value "0"
Override registers CMOS[7] to value "14"
Press magnifying button to go to 5x zoom and 10x zoom and back to normal view (this somehow resets live view to the new override settings)
Or
Switch to photo mode and switch back to video mode (this resets live view to the new override settings)
Now you have full sensor width view with 1/3 vertical sensor height(with no lineskipping)
-
To refresh the parameters use (being in 5x not need to change to another mode) :
static void run_test()
{
msleep(200);
set_lv_zoom(5);
}
Yes, I tried it ADTG2-100c 0 in 1080p mode.
0x0 give an image of 1920x1080 (distorted) -> scaled post process 1920x360 without moire (less noise).
This is just an example for the 500D to give it to 1590x1060 -> scaled post process 1590x353.
-
If the image buffer size can be made bigger, this may be the way to go for 4k, as a 1:1 center crop... Edit: And we need some memory card and SD write speed upgrade to get 360Mb/s write speed ;D
Or 1920(full sensor width) x 2160(2/3 sensor height) distorted -> Scaled post proces, 1920 x 720 moire free.
Couldn't find any mode which bins vertical, the 5d3 probably has such a mode in one of the CMOS[] registers...
Wouldn't surprise me if the 6d has such a mode too...same generation CMOS...
Does someone know if the 5d3 has the ADTG-x00c register to set the vertical lineskipping ?
It is using lineskipping in 50p and 60p modes... can someone confirm this register stays at zero in 1080p mode ?
-
If the image buffer size can be made bigger, this may be the way to go for 4k, as a 1:1 center crop...
Yes, but it will be less FPS on the 5D3 should be 15-20FPS (4096x1714).
FPS timer A is width
FPS timer B is height
You can take a photo normal mode, lock timer A and B and go to live view. FPS will be continuous photo mode (500D 3.554)
I wonder if there is somewhere a clock sensor (50D sensor is faster than the 500D).
Maybe we can do ultra fast camera 320x240 crop mode, but it will not be easy. It will take a very long time. And on the way may be other limitations.
Couldn't find any mode which bins vertical, the 5d3 probably has such a mode in one of the CMOS[] registers...
Wouldn't surprise me if the 6d has such a mode too...same generation CMOS...
Yes, this is interesting. I only have a 500D so I can not help.
-
FPS timer A is width
FPS timer B is height
Can these FPS registers be found in the "adtg_gui.mo" tool ?
How are these registers named on you're 500d ?
-
ADTG Registers -> Show -> FPS timres only
(requires DIGIC registers)
0xC0F0, 0x6008, 0, "FPS register A"
0xC0F0, 0x6014, 0, "FPS register B"
-
Oh boy :D, can't wait to try some settings with those registers when the battery is recharged.
Do you know what values the 6008 and 6014 stand for ? Can't make any resolution out of it ?
And do you know if there's some place (bitbucket maybe :-\) where this information is documented ?
-
Photo mode :
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs30.postimg.org%2F8whjq8uap%2FVRAM0.png&hash=df22ce92a3c6ddec5636017fe4937a3f)
Movie mode (with values photo mode):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs30.postimg.org%2F63oc67ty9%2FVRAM1.png&hash=86f94814316eb15c0e159f71635b0421)
This does not affect the image. But the sensor work in full resolution (this is confirmed by previous research the full width video).
(focus box should be on the left, I do not know whether the top or bottom)
-
Got it, same registers on the 6d.
Locking photo mode values gives 4.7 FPS in video mode :D
But how did you record the full image width ?
I can't get any higher resolutions in MLV_record then in standard video mode, still max of 1792pixels wide ?
-
It is complicated because the buffer is still the same.
We get the full width by reducing the height.
This requires a modification in raw.c and few registers.
When lock all registers ADTG/CMOS 1080p. Go to 720p, buffer edmac is 720p. So this is not it.
Maybe it will help?
static void run_test()
{
msleep(2000);
debug_intercept();
call("lv_save_raw", 1);
msleep(1000);
call("lv_save_raw", 0);
debug_intercept();
}
-
Can someone tell me the values of digic registers:
C0F0[6008]
C0F0[6014]
On the 5d3 in video mode ?
-
FPS timer A is width
FPS timer B is height
How are those values used, I can't make any sense out of the number which is used in timer A
6d
Photomode Video(1080)
Timer A 5970597 2210221
Timer B e8d 63f
Value for timer B e8d = 3725 in decimals, could make sense for height...
Value for timer B 63f = 1599 in decimals, could make sense for height...
But I can't make anything out of the numbers in timer A :-\
-
Timer You can change in the FPS menu.
I received a the log, there are a lot of registers. We need someone experienced.
So we need wait for A1ex
And now we can go on vacation ;D
-
How are those values used, I can't make any sense out of the number which is used in timer A
6d
Photomode Video(1080)
Timer A 5970597 2210221
Timer B e8d 63f
...
But I can't make anything out of the numbers in timer A :-\
Timer A, which are 0x597 and 0x221 in your two examples because it is doubled, is the number of clock ticks to wait before each successive row is read out. Thus, it should not be set shorter than the time it takes to read out a single row of the image (i.e., the width), but it can be set longer depending on the needs of a rolling shutter or an exact frame rate.
-
Ah, that explains why the real low values didn't gave any live view at all...
Is it double because of 2 column, rows that are read out or something ?
Anyone tried combining them, for example:
Timer A value on 5970221 ?
-
Yeah, I imagine Timer A is doubled for the same reason that the CMOS gain is doubled: each row in a row pair will use one of the halves. However, I can't think of a clear benefit from mismatching the two halves of Timer A, unlike with CMOS gain and dual ISO.
-
@A1ex and @Nikfreak,
Would be nice if we can get the 3xzoom/1:1 crop preview in live view eventually in the nightly builds.
A1ex, any suggestions how to implement it, since it differs per camera model, would you put it as an option in the raw and mlv recording modules or is it better to place the code somewhere else ?
Maybe we could put an option in the raw/mlv recording menu, which put the camera in crop mode. For example with an option "Crop mode" on or off.
To get 1:1 crop / 3x zoom preview in live view on 6d, these registers need to be changed to the following values:
CMOS[6] d0 (=Changing the "d" moves horizontal position on the sensor)
CMOS[7] 206 (Vertical centering the image on the screen)
ADTG2-800c 0 (vertical lines to skip)
ADTG2-8000 5 (Don't know what it does, if it is 6(as in 1080p25mode) you get some vertical interlaced looking footage... so must be 5)
-
Levas thx for all that awesome findings. We will need to wait until a1ex is back. Actually I think I am not capable of implementing what you requested. Don't want to reinvent the wheel so it's better to hope that a1ex will do that job...
-
And thanks to Greg :), he did the crop findings on his 500d, I translated it to the registers on the 6d.
A1ex probably has a clear view on how he would like to add options like these to the existing code.
-
I can read the values, but do not know how to overwrite.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs1.postimg.org%2Fmnuf4687z%2FVRAM0.png&hash=d60e11c5c27dd6310085c2650f89588b)
crop_rec ;D
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs1.postimg.org%2F7pc09qt5r%2FVRAM1.png&hash=e69360cd2d4ae0917dab9c9f55a2e642)
-
Aha, a Crop_rec module!
Only programming language I know is Quickbasic ;D and some HTML... :P
But the answer in how to overwrite registers probably can be found in the source code of the adtg_gui module
https://bitbucket.org/hudson/magic-lantern/src/3288e152d8ae0dcb2cdaf1f9afc1a17611eea2ff/modules/adtg_gui/adtg_gui.c?at=unified
-
adtg_gui is not the solution, it requires 15FPS live view and GDB.
We need a module that will let 30FPS ;)
Not easy :o
-
adtg_gui is slow because it intercepts everything. it's more appropriate to look at iso_regs for guidance, although you'll need the camera-specific addresses found in adtg_gui.
-
I don't mean using the adtg_gui module to modify or rebuild.
I expect there to be some command/syntax which shows how to replace values in the registers... :o <- my face looking at the source code ;D
-
I modified ADTG_GUI (remove some REG_ENTRY). Now can I get more FPS.
500D in crop mode can get 24FPS (default is 22FPS).
The 1080p (crop trick) can get 21FPS when try more, the camera will freeze.
So maybe somewhere is to limit FPS for each mode.
Currently, it does not make sense for a 500D.
-
ah, some unforseen problem... nog getting a minimum of 24fps...
-
On the other cameras should be 30FPS, 500D has a limit in full HD.
-
That reminds me, the 500d is limited to 20 FPS, as factory default, isn't it ?
-
Yes 500D without ML in 1080p has 20FPS. With ML can go to 21FPS.
1080p crop trick, the camera think that it is still 1080p (not crop mode). So you can not go over 21FPS.
-
500D H264 default vs crop (crop mode is blinking :o)
Default - http://s22.postimg.org/mklz071wf/deafult.jpg
Crop - http://s22.postimg.org/sbc5dx9wf/crop.jpg
-
Maybe you should try some different values for ADTG2-1000 registers, maybe 5 isn't good either ?
I got weird vertical interlacing with the value 6, 5 seems good for me, but I didn't check that much, I shall record some video to see if a value of 5 for 6d is good or not.
And I don't really get where the value in the ADTG-1000 register stands for, have no idea what the value exactly does...
Edit: The value is set to zero in photo mode.
-
Does this work on actual image?or it is just playing with graph?
I tried to install the module on my 5dmark3 but it appears to be fail to load.
Some please advice,I am eager to finds out.
-
Please read the OP
Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!
Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?" (http://quotationsbook.com/quote/34064/)
requires a custom ML build with CONFIG_GDB=y
-
CONFIG_GDB=y is put to makefile.user, right?
I'm compiling silentpics branch with this line, but loading iso_regs gives me an error. Please help to find out what I'm doing wrong.
Thx!
-
Try merging the fullres-silent-pic branch with the iso-research branch.
If you don't want to merge, compile the fullres-silent-pics branch as usual, with CONFIG_GDB=y, then compile only the extra modules from iso-research.
-
Great! Thank you!
-
My test scene looks like this. Underexposure zebras are set to 0.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FG2FwvQB0%2FScene.png&hash=b0bed4efeeb67eb3083f40069fb625e4)
I ETTR the second brightest object (which is angled for a good gradient) and gives a good response near the highlights. Which produces results like so.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FprPNCF8m%2FCanon-6-D-Photo-ISO-100.png&hash=05013b396c128989db3e2e35228b97e0)
Some further detail about this test setup is back here (https://www.magiclantern.fm/forum/index.php?topic=10111.msg118194;topicseen#msg118194).
My results for 6D photo mode and no aperture gain.
https://www.dropbox.com/sh/rrn7chasohg36hk/AAD7fngMRzvGNfrz6IzOgX3pa?dl=0
-
Moved my off-topic discussion regarding lenses over here (http://www.magiclantern.fm/forum/index.php?topic=13770).
Dynamic range and Equivalence discussion moved here (http://www.magiclantern.fm/forum/index.php?topic=13787.0).
-
Is there a reason why the 7D never got included?
Found the easy stubs
ENG_DRV_OUT_FUNC = 0xFF1F675C; // from stubs
ENGIO_WRITE_FUNC = 0xFF1F6B20; // from stubs
But I guess I have to do some stub checking for CMOS_WRITE_FUNC and ADTG_WRITE_FUNC, unless someone wants to use his automatic stub tool (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97953#msg97953). :)
-
work on 50D ? Who can make this module ? I think 66 ISO it's best)
-
Is there a reason why the 7D never got included?
The whole dual processor thing I think. All the interesting stuff happens on the master. ML can only debug on the slave.
work on 50D ? Who can make this module ? I think 66 ISO it's best)
What module? There are several different modules associated with this thread and they are all for reverse engineering and research purposes. They also require compiling ML in 'debug mode' (Config_GDB=y). This is something you must do yourself.
-
The whole dual processor thing I think. All the interesting stuff happens on the master. ML can only debug on the slave.
Yes, that sounds familiar. Thanks for the memory jog.
-
500D
1080p crop-mode(5x) 1080p-trick-crop
ADTG2[105f]N 0x0 0x38f 0x38f //shutter blanking for 5x/10x zoom
ADTG2[1061] 0x67f 0x0 0x0 //shutter blanking for 1080/720p mode?
CMOS[4] 0xc0d 0xc85 0xc85
CMOS[5] 0x400 0x609 0x689
ADTG2[100c] 0x2 0x4 0x0 //vertical skipping lines, not needed?
ADTG2[1000] 0x6 0x5 0x5 //vertical skipping, horizontal binning?
To get 1:1 crop / 3x zoom preview in live view on 6d, these registers need to be changed to the following values:
CMOS[6] d0 (=Changing the "d" moves horizontal position on the sensor)
CMOS[7] 206 (Vertical centering the image on the screen)
ADTG2-800c 0 (vertical lines to skip)
ADTG2-8000 5 (Don't know what it does, if it is 6(as in 1080p25mode) you get some vertical interlaced looking footage... so must be 5)
For 5D3, a configuration that appears to work is:
CMOS[1] 0xB8B vertical position and size
CMOS[2] 0x10E horizontal position and downsizing factor
CMOS[6] 0x170 ISO related? pink highlights without it
ADTG2[0x8000] 0x5 it's 5 in zoom mode and 6 in 1080p
ADTG4[0x8000] 0x5 same
ADTG2[0x8806] 0x6088 without this, you get some weird artifacts
note: ADTG4[0x8806] should not be changed (default 0x6048)
How to apply it:
- compile ML for adtg_gui (see first post)
- load adtg_gui and enable it in Debug menu
- go to movie mode 1080p, 24 or 25 fps
- go to play mode and back
- configure the registers in adtg_gui submenu (latest adtg_gui has a preset for crop mode registers)
- go to play mode and back (again)
Minor bug: it has a few black lines at the top.
-
Updated adtg_gui: it now works even at 60fps*, and you no longer need CONFIG_GDB to run it (just compile ML from iso-research normally).
https://bitbucket.org/hudson/magic-lantern/commits/14776f860522
* if you also log DIGIC registers, those are not fast enough for 60fps.
-
So it should work on all cameras, just find only registers.
On old cameras we need read shutter table and overwrite ADTG2[105f]N (if it's possible).
At 5D3 probably image is ok.
On the other cameras will be extended. Native (raw) 1080p mode -> H264 (1920x1080).
-
Hi! Is the changes to register settings reverted to default on restart or do I have to manually reset reg changes?
Got the adtg_gui loaded and planning on trying the 3d crop mode settings.
-
They are all reverted to default on power off. Also registers that have not been manually changed, will also change with camera settings. So if you manually change some register, in essence you lock the register to that value. If changing from photo mode to movie mode normally changes that register, this will not happen.
-
Hi! Is the changes to register settings reverted to default on restart or do I have to manually reset reg changes?
Got the adtg_gui loaded and planning on trying the 3d crop mode settings.
Great question as I was just thinking the same earlier...
They are all reverted to default on power off. Also registers that have not been manually changed, will also change with camera settings. So if you manually change some register, in essence you lock the register to that value. If changing from photo mode to movie mode normally changes that register, this will not happen.
valuable info to know... Thanks @Audionut!
-
I'd like to use this in combination with dual ISO.
Is there a nightly build version of this so I dont have to attempt to try to build this myself?
Thanks
-
Nope. Read the OP.
-
I have a 5DMKII to use for the summer, and if there is a build to check out, would love to check out the 3X screen.
It seems to me that this is one of the biggest hurdles to solve to make 2K shooting more practical.
BTW love the 5DMKII with ML.
-
It seems that crop mode (http://www.magiclantern.fm/forum/index.php?topic=10111.msg145036#msg145036) has some practical value. I shot some videos without issues.
Advantages:
1. Centered (at lest at 24 FPS) - minimal distortion with wide lenses.
At 30 FPS slightly down center(CMOS[1] - 0xb8E), but centered horizontally.
Maybe it's possible to get it centered and with 30 FPS, I just not so skilled in brute forcing.
2. Works via HDMI.
3. Same as "Canon" preview - fast updating and color.
Disadvantages:
1. Recording times about 30%-50% less.
2. Cumbersome activating process. Of course, that adtg_gui tool was not intended for shoot.
3. No way to exit back to non-crop view without restarting - liveview gets corrupted.
4. Just 1920x1080.
5. Dual ISO doesn't work. Its's looks like it simply commented out.
Notes:
1. I don't have "a few black lines at the top" a1ex was told about.
2. I didn't notice any artifacts, it looks like "regular" crop-mode video.
So main question - is it possible to get this without such a great impact on performance?
-
Thanks for the detailed explanation Motbaiphoto.
-
Now I want to try that ISO 66 from first post.
First, its unclear, do I need to tweak with iso_regs, or I can do the same with adtg_gui, or I need both? I decided to start with iso_regs, as it seems more relevant.
I compiled with CONFIG_GDB=y, iso_regs from iso-research branch(as it looks newer version than posted in OP) start camera with module loaded, ISO 100, take picture - everything is OK. But, as soon as I activate ISO registers in Debug menu every image I take gets corrupted intill I deactivate ISO registers and restart camera. iso_regs from OP behaves the same :(
What am I doing wrong?
-
On 5D3, you can just use iso_regs.
Will double-check the issue; I suspect a problem in the patching library.
Meanwhile, I suggest trying to compile 497b816 from the iso-research branch, instead of the latest one. Or, if it still doesn't work, I think just the unified branch with CONFIG_GDB=y should do the trick (so you can load iso_regs). It will probably not play nice in LiveView at higher FPS, but should work fine in photo mode.
-
unified branch with CONFIG_GDB=y should do the trick (so you can load iso_regs)
Yes, this works, thanks.
-
Just curious ... How is ISO 66 different from ISO 50 that we can get from Canon's settings (better or worse)?
-
If you define ISO based on the clipping point, your camera's ISO 50 is actually just ISO 100. This ISO 66 is a true ISO 66. It allows capturing more photons without clipping, than does ISO 100. Canon ISO 50 and ISO 100 clip at the same number of photons captured.
-
Brilliantly well said, David and Thanks for the clarifications!
-
1. I don't have "a few black lines at the top" a1ex was told about.
I wasn't told about it, I've seen it myself, and I'm seeing it right now.
At 30 FPS slightly down center(CMOS[1] - 0xb8E), but centered horizontally.
Didn't really check how centered it is, but at 30 fps I see a big dark bar at the top (not black, but darker than the entire image), plus a lot of artifacts in the image (somewhat like vertical banding).
edit: 30fps works much better with 0xb8E, no more artifacts, nice find :D
-
I wasn't told about it, I've seen it myself, and I'm seeing it right now.
Maybe some camera settings affect this - i.e PAL/NTSC(i'm on NTSC)
Didn't really check how centered it is, but at 30 fps I see a big dark bar at the top (not black, but darker than the entire image), plus a lot of artifacts in the image (somewhat like vertical banding).
This probably means you didn't follow exactly your own manual regarding entering play mode twice or changed camera FPS after enabling registers hook. It works even with 60 FPS - 0xB95.
-
You can't get 30fps in PAL.
I did follow the instructions (without pressing PLAY twice, you can't even see the last registers in adtg_gui). Didn't touch FPS (override is off).
At 60 FPS I get green image, even with 0xB95.
-
Didn't touch FPS (override is off).
I didn't mean override. I mean canon menu changes. Need restart camera, set "Movie rec size" in canon menu and just after that hooks and so on.
-
Is there a nightly build for the iso research functionality? I need the special branch in order to use ISO 66 correct?
Thanks
-
Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!
Take it easy, the current state is research.
-
Updated adtg_gui to allow non-destructive overriding of registers (currently only implemented for CMOS/ADTG/DFE registers, and only tested for CMOS ones). Trick borrowed from crop_rec.mo (http://www.magiclantern.fm/forum/index.php?topic=17021).
That means, as soon as you stop overriding a register, it gets back to its original value.
Previously, if a register value was not updated continuously by Canon code, disabling our overrides did not restore the old value.
-
Good catch @a1ex ... I was fiddling with the adtg_gui last night but had forced myself to take a screenshot of defaults before making changes.
Now this eliminates that process ... Nice work once again! [emoji106]
-
Hi,
Is anyone getting ISO 35, 70, 140 on my 600d and 5d3?
I set the ml digital iso to -1,5 EV and HTP off.
When I turn HTP on I only get ISO 70 140, ... (no longer 35)
Which one results in less noise?
-
I spent the last days learning how to compile ML to use these tweaks on my 5D MK III. On the weekend i decided to give the ISO REGS module a go and used ISO 860 (pulled from 1600) and ISO 3400 (pulled from 6400) for some scenes which are typically troublesome because of the extreme dynamic range involved like harsh stage lighting and a subject sitting in the sun with a dark wooden wall behind him.
Exposure was set at +2/3 EV from the correct value for unmodified ISO. Shadow/midtone noise is clearly better. I need to post some crops tomorrow, i also shot some test images of real world scenes.
Just one question - i do not really get what "CMOS patched" means? As far as i understand, this is some tweak which has to be applied with ADTG GUI, and cannot be done with ISO REGS, right? Can i use ADTG GUI to apply the CMOS patch, and then use ISO REGS to set the rest?
-
Indeed, it's not implemented in iso_regs, but it's easy to do so (in cmos_log).
Unfortunately, you can't run both adtg_gui and iso_regs at the same time.
-
A quick test with a high DR scene that would require dual iso, or exposure bracketing HDR, or a flash. One Picture was taken with Canon ISO 100, 1/125 s shutter and f8. Second exposure was taken with ML ISO 100 (actually 119 reported by iso_regs), ADTG gain at 962 (-0.16 EV as reported by iso_regs), preamp at 1, 0xfe at 3, same exposure as with Canon ISO 100.
Both raws loaded in Lightroom 6.6, exposure pushed by almost 4 stops (+3.74 EV on ML ISO one, +3.84 on Canon ISO) to bring out the shadows, highlights at -100, WB as shot. ML ISO visibly reduces color noise and the purple shadow color shift i get when i push them too much from Canon ISO. Luma noise is also reduced and looks finer, which helps to preserve details. Only the banding noise seems to remain. Color noise reduction in Lightroom was left at standard (25), even bringing this one to 100 on the Canon ISO picture could not bring it to a comüparable level to the ML one.
Now if there is a fix for the banding noise....
Full Size JPEG, Canon ISO 100: https://www.dropbox.com/s/4jjnvt9y9h1p3m4/5D3_ISO%20100.jpg?dl=0 (https://www.dropbox.com/s/4jjnvt9y9h1p3m4/5D3_ISO%20100.jpg?dl=0)
Full Size JPEG, ML ISO: https://www.dropbox.com/s/kh16i53nb1zw4iz/5D3_ML_ISO-1.jpg?dl=0 (https://www.dropbox.com/s/kh16i53nb1zw4iz/5D3_ML_ISO-1.jpg?dl=0)
Crop 1, Canon ISO: https://www.dropbox.com/s/c779b25yn4sgali/5D3_ISO%20100_Crop%201.jpg?dl=0 (https://www.dropbox.com/s/c779b25yn4sgali/5D3_ISO%20100_Crop%201.jpg?dl=0)
Crop 1, ML ISO: https://www.dropbox.com/s/b3ahzvatyjf3fg4/5D3_ML%20ISO_Crop%201.jpg?dl=0 (https://www.dropbox.com/s/b3ahzvatyjf3fg4/5D3_ML%20ISO_Crop%201.jpg?dl=0)
Crop 2, Canon ISO: https://www.dropbox.com/s/f5e4fuh09cbz2uj/5D3_ISO%20100_Crop%202.jpg?dl=0 (https://www.dropbox.com/s/f5e4fuh09cbz2uj/5D3_ISO%20100_Crop%202.jpg?dl=0)
Crop 2, ML ISO https://www.dropbox.com/s/cukp18kvkdaiclh/5D3_ML%20ISO_Crop%202.jpg?dl=0 (https://www.dropbox.com/s/cukp18kvkdaiclh/5D3_ML%20ISO_Crop%202.jpg?dl=0)
Crop 3, Canon ISO: https://www.dropbox.com/s/0wnhr84g4am458b/5D3_ISO%20100_Crop%203.jpg?dl=0 (https://www.dropbox.com/s/0wnhr84g4am458b/5D3_ISO%20100_Crop%203.jpg?dl=0)
Crop 3, ML ISO: https://www.dropbox.com/s/97luk8kwiptdlhw/5D3_MIL%20ISO_Crop%203.jpg?dl=0 (https://www.dropbox.com/s/97luk8kwiptdlhw/5D3_MIL%20ISO_Crop%203.jpg?dl=0)
Crop 4, Canon ISO: https://www.dropbox.com/s/olers9xsnux0d2l/5D3_ISO%20100_Crop%204.jpg?dl=0 (https://www.dropbox.com/s/olers9xsnux0d2l/5D3_ISO%20100_Crop%204.jpg?dl=0)
Crop 4, ML ISO: https://www.dropbox.com/s/miauq2zk6xw11sc/5D3_ML%20ISO_Crop%204.jpg?dl=0 (https://www.dropbox.com/s/miauq2zk6xw11sc/5D3_ML%20ISO_Crop%204.jpg?dl=0)
Crop 5, Canon ISO: https://www.dropbox.com/s/l31k4xfjuuzhme3/5D3_ISO%20100_Crop%205.jpg?dl=0 (https://www.dropbox.com/s/l31k4xfjuuzhme3/5D3_ISO%20100_Crop%205.jpg?dl=0)
Crop 5, ML ISO: https://www.dropbox.com/s/ncpg1hzxcc7361h/5D3_ML%20ISO_Crop%205.jpg?dl=0 (https://www.dropbox.com/s/ncpg1hzxcc7361h/5D3_ML%20ISO_Crop%205.jpg?dl=0)
Now i need to learn how to use cmos_log to apply the CMOS patch.
-
Looks better than I've expected. Was the ISO 119 pulled down from Canon ISO 200?
Can you share the screenshot with iso_regs settings?
-
Yes, this was pulled from Canon ISO 200.
Screenshot