Author Topic: Sensor Factory Calibration (Read 16135 times)

Greg · « **on:** November 28, 2018, 02:19:44 PM »

Sensor Factory Calibration
(500D)

RAM (ISO100) 0x30A10C - 0x30EC8C
ROM (ISO100) 0xFFA0590C - 0xFFA0A48C

RAM

ISO 100 0x30A10C
ISO 200 0x30EC94
ISO 400 0x31381C
ISO 800 0x3183A4
ISO 1600 0x31CF2C
ISO 3200 0x321AB4
ISO 6400 0x32663C
ISO 12800 0x32B1C4

I found the adress in RAM using QEMU FRSP-log:

Code: [Select]

[EDMAC#9] Starting transfer from 0x30A10C to <15>, 4832x4, flags=0x10000
[EDMAC#9] 19328 bytes read from 30A10C-30EC8C.

How to find in RAM?

Code: [Select]

Find hex values :
4D 00 01 01 88 4B 00 00
4D -> ISO 100?
88 4B -> 0x4b88 = ‭4834‬ // 4834 - 2 = 4832 500D width

Now let's try to find in 5D3 rom
‭5936‬ + 2 = 5938
5938*4 = 0x‭5CC8‬

4D 00 01 01 C8 5C 00 00
The address is after these hex values

What is this data?

The data is packed in 16 bits of 2 values for each column.

Left optical black [512, 0] (these are approximate values, different for each column):

After left optical black [512, 512] (these are approximate values, different for each column):

Changing the first value affects the black level column by -1.
Black level 1024
value -1 -> 1025
value +1 -> 1023

Dark frames (+5EV)

(Canon Default | Whole buffer filled [0,0] | Whole buffer filled [512,512] | Buffer after left optical black filled [512,512]) :

Greg · « **Reply #1 on:** December 10, 2018, 05:33:35 PM »

The first attempts of automatic calibration :

The first image is a dark frame with ADTG-CMOS amplifier reduced. The second image, after automatic calibration.

70MM13 · « **Reply #2 on:** December 10, 2018, 11:35:45 PM »

Wow, exciting!
I can't wait to see where this leads...

Greg · « **Reply #3 on:** December 12, 2018, 04:29:27 PM »

Averaged 16 frames to make the effect more visible.

ISO 1600
(Canon default | automatic calibration)

70MM13 · « **Reply #4 on:** December 12, 2018, 05:13:51 PM »

Nice reduction of those nightmarish vertical stripes!

Kharak · « **Reply #5 on:** December 12, 2018, 10:17:55 PM »

That is huge!

Would be nice with some Real life examples on how the noise and FPN has improved.

70MM13 · « **Reply #6 on:** December 12, 2018, 10:49:22 PM »

Those stripes are the only issue that stands in the way of complete satisfaction for me with this camera.

Here's a typical "before" for me, and I can't wait for the after!!!

g3gg0 · « **Reply #7 on:** December 13, 2018, 01:08:02 AM »

1) so you found the location where the vertical ADC offset tables for FPN correction are?
2) these probably were calibrated in factory and written into the flash only once?
3) you updated values in RAM to achieve better FPN reduction - for CR2? or did it even affect raw video?

would be interesting to use these tables into a MLV and creating the DARK frame from it

g3gg0 · « **Reply #8 on:** December 13, 2018, 01:28:55 AM »

doing a quick analysis on 5D3.113, it is the TUNE property table, starting at 0xF8B20000

you can dump it like that:
PropertyEditor.exe 0xF8000000-0xF8FFFFFF.BIN 0xB20000 0x2E0000

and it will spit out a property dump like this:

where you can see that this memory location is for properties 0x0101004D-0x0101007C with 5936 uint32 values.
before them there are properties 0x0101007E-0x0101008F with 376 uint32 values also looking like a offset table.

getting their value is as simple as reading a property. no need for memory locations.
modifying them, well, i would stick to patching it in-memory

good find!

would really be interesting if this is the FPN compensation table we could simply pass to MLV files.
(well not simply, a dark frame must be calculated from these values)

Greg · « **Reply #9 on:** December 13, 2018, 03:46:03 AM »

@g3gg0,
1) yes
2) 500D yes, I do not know how other cameras. This could be it - https://www.spt.info/index.php/service-adjustment-software?id=21
3) I have a module that takes 64 FRSP dark frames, it takes 9 minutes to process this data (in camera), then the calculated values are saved in RAM.

Calibration in this way:

Code: [Select]

int new_value = current_value - (int)roundf(allframes_mean - column_allframes_mean[x]);It's possible that raw_diag does some more advanced things. I just skip cold/hot pixels.

CR2 and FRSP are cleaner (FPN), 500D is not good for video so I do not know, I did not check it.
The module requires code cleanup and some work, currently it is only a tool that is not good for official publication.

We can also use something like two_add engine to speed up the processing. But is it necessary?

My main goal was to calibrate FPN for 500D with ADTG modifications. The effect is good, now it's a new camera.

Current limitation - I can change the black level of the column by 1. So some of the columns still have 1024.5 or 1023.5.

70MM13 · « **Reply #10 on:** December 14, 2018, 04:40:59 PM »

Can someone bring this magic to us 5D3 users?

I'm sure I'm not the only one who wants to test this!

g3gg0 · « **Reply #11 on:** December 14, 2018, 09:36:31 PM »

on 5D3 the properties with size 0x5E0 are being read by function "GetHivshdIRparameter(Po=%ld[%ld])"
it is later written to HIV by first writing this engio list: (0xBC * 8 = 0x5E0)

and then copied there:

and the 0x5CC0 sized ones are used for
stru_45294.edmac_0x18_address_HivShd = get_dark_offset(param_iso, v4);

g3gg0 · « **Reply #12 on:** December 14, 2018, 09:53:19 PM »

Quote from: Greg on December 13, 2018, 03:46:03 AM

The module requires code cleanup and some work, currently it is only a tool that is not good for official publication.

i would head towards reading and writing the properties if i were you.
this makes the code more robust and then would be an equivalent to the SPT tool.
but that requires thorough testing of course.

Quote from: Greg on December 13, 2018, 03:46:03 AM

We can also use something like two_add engine to speed up the processing. But is it necessary?

nah, if you update properties you wont run it every day.

Quote from: Greg on December 13, 2018, 03:46:03 AM

Current limitation - I can change the black level of the column by 1. So some of the columns still have 1024.5 or 1023.5.

as the offsets are integer, odd numbers aren't possible. at least in-camera

would have to use float-files which are a bit overkill.

Greg · « **Reply #13 on:** December 14, 2018, 10:26:14 PM »

Quote from: g3gg0 on December 14, 2018, 09:53:19 PM

as the offsets are integer, odd numbers aren't possible. at least in-camera
would have to use float-files which are a bit overkill.

This should be possible in the camera.
The "B" value can change column black level below 1.

Code: [Select]

B = 0 -> column black level = 1233.88
B = 1023 ->  column black level = 1259.98
diff = 26.1

But this is an offset from the black level of the entire frame...

Code: [Select]

B = 0 -> column black level = 1129.34
B = 1023 ->  column black level = 1142.50
diff = 13.6

B = 0 -> column black level = 1037.44
B = 1023 ->  column black level = 1038.96
diff = 1.52

(column_black_level - frame_black_level)*12.5%

So 1023.5 we can change to 1023.56
It is possible that we need to change some engio register, some offset.

Greg · « **Reply #14 on:** December 15, 2018, 01:17:56 AM »

cmos_cal.mo
https://bitbucket.org/Gr3g01/ml-cmos-cal/src/iso-research/modules/cmos_cal/

To run on another camera:
- SENSOR_VFPN_ADDR_ISO,
- SENSOR_FULL_WIDTH,
- line 325: "if(p < 512 || p > 1536) // digic IV cameras black level 1024 - fix it",

The code is still not clean. But I think someone can try results on 5D2 (is similar to 500D).

TODO :
- I think that the whole module should be rewritten

- use silent.mo to take frsp with mirror down,
- save/load data into a file,
- improve the algorithm of calculation to reduce the number of frames and less heat,
- increase the accuracy of calibration below 1,
- validation - comparison of effects before and after calibration,

Greg · « **Reply #15 on:** December 15, 2018, 09:27:55 PM »

An easy way to find RAM address.
edmac log :

Code: [Select]

EDMAC#9: addr=0x30a10c conn=15 cbr=0 name='''''' size='''4832x4'''       <------------------------
EDMAC#0: addr=0x79df1a4 conn=2 cbr=0 name='''''' size='''480x16'''
EDMAC#1: addr=0x79e13a4 conn=28 cbr=0 name='''''' size='''(476, skip 36) x 176'''
EDMAC#2: addr=0x600637c conn=0 cbr=0 name='''''' size='''8456x3203'''
EDMAC#6: addr=0x79f9524 conn=18 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#9: addr=0x39a3f4 conn=15 cbr=0 name='''''' size='''8x3203'''
EDMAC#8: addr=0x3a48f4 conn=1 cbr=0 name='''''' size='''2000x224'''

Horizontal correction
The address is different for FRSP and CR2
ISO 100 regular picture CR2 (0x39a3f4)
In 500D ROM and RAM is empty, canon does not use it, but if you enter some values manually, it changes the black level of the horizontal line.

It probably will not be useful, we must remember that fixed pattern noise is not random pattern noise.
We can not correct something that is random in this way.

Danne · « **Reply #16 on:** December 15, 2018, 10:30:07 PM »

Hi Greg. Followed your advice and logged edmac channels on my 100D:

Code: [Select]

EDMAC#9: addr=0x15aff00 conn=15 cbr=0 name='''''' size='''8x727'''
EDMAC#0: addr=0x1fb37a4 conn=46 cbr=0 name='''''' size='''248x2'''
EDMAC#12: addr=0xdb88c00 conn=14 cbr=0 name='''''' size='''4096x4096'''
EDMAC#2: addr=0xc1f7800 conn=4 cbr=0 name='''''' size='''1440x480'''
EDMAC#6: addr=0xdb76000 conn=18 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#9: addr=0x15aff00 conn=15 cbr=0 name='''''' size='''8x727'''
EDMAC#8: addr=0x14f12bc conn=0 cbr=0 name='''''' size='''2000x224'''

LOG:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/EDMAC000_100D.LOG

A little lost I am but I got your fine cmos calibration tool running on my 100D but it fails to do what it is inteneded to do. I probably should get some sleep and reread your post again but if you feel like it It would be great to kickstart my 100D with some values suitable to this:
- SENSOR_VFPN_ADDR_ISO,
- SENSOR_FULL_WIDTH,
- line 325: "if(p < 512 || p > 1536) // digic IV cameras black level 1024 - fix it",

I logged while in photo mode. Would the same logs appear in movie mode?

Greg · « **Reply #17 on:** December 16, 2018, 12:19:00 AM »

@Danne,
This log looks like a live view.
We need a regular pic log.

Live View
I looked at the live view in 500D :

Code: [Select]

EDMAC#9: addr=0x442784 conn=15 cbr=0 name='''''' size='''32x208''' //vertical column
EDMAC#9: addr=0x485fd4 conn=15 cbr=0 name='''''' size='''32x165''' //horizontal line

This looks like a 4x1664 buffer and 4x1320
Looking at hexeditor and RAM, the data in this buffer looks very similar to those in photo mode.

Changing these registers does not affect ML raw. ML takes raw before FPN correction.

Change the value of several columns:

Several lines:

I do not plan further research in live view/video mode. I do not have time for it.

Danne · « **Reply #18 on:** December 16, 2018, 08:08:32 AM »

Files here:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/EDMAC000_fullres_silent_100D.LOG
https://bitbucket.org/Dannephoto/magic-lantern/downloads/EDMAC000_CR2_100D.LOG

Ok, these should be better(CR2):

Code: [Select]

EDMAC#0: addr=0x80090c4 conn=35 cbr=0 name='''''' size='''9240x3529'''
EDMAC#8: addr=0x13a0000 conn=0 cbr=0 name='''''' size='''2000x256'''
EDMAC#9: addr=0x144f9e4 conn=15 cbr=0 name='''''' size='''8x3529'''
EDMAC#6: addr=0xccc000 conn=18 cbr=0 name='''''' size='''4096x13, 752'''
EDMAC#12: addr=0xf5a884 conn=14 cbr=0 name='''''' size='''4096x41, 128'''
EDMAC#16: addr=0xf5a884 conn=23 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#17: addr=0xf40504 conn=2 cbr=0 name='''''' size='''480x16'''
EDMAC#18: addr=0xf42704 conn=28 cbr=0 name='''''' size='''(434, skip 78) x 192'''

Log from fullres silent DNG:

Code: [Select]

EDMAC#1: addr=0x63cc100 conn=27 cbr=0 name='''''' size='''1920x640'''
EDMAC#2: addr=0xc1f7800 conn=4 cbr=0 name='''''' size='''1440x480'''
EDMAC#5: addr=0xd9c5a00 conn=2 cbr=0 name='''''' size='''32x240'''
EDMAC#8: addr=0x14f12bc conn=0 cbr=0 name='''''' size='''2000x224'''
EDMAC#9: addr=0x15aff00 conn=15 cbr=0 name='''''' size='''8x727'''
EDMAC#16: addr=0xdb98000 conn=40 cbr=0 name='''''' size='''864x288'''
EDMAC#17: addr=0x6df100 conn=15 cbr=0 name='''''' size='''20'''
EDMAC#18: addr=0x6798100 conn=0 cbr=0 name='''''' size='''3164x726'''
EDMAC#24: addr=0x20c80a0 conn=54 cbr=0 name='''''' size='''12594x16'''

and more:

Code: [Select]

EDMAC#0: addr=0x80090c4 conn=35 cbr=0 name='''''' size='''9240x3529'''
EDMAC#8: addr=0x13a0000 conn=0 cbr=0 name='''''' size='''2000x256'''
EDMAC#9: addr=0x1441d44 conn=15 cbr=0 name='''''' size='''8x3529'''
EDMAC#16: addr=0xf5a884 conn=23 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#17: addr=0xf40504 conn=2 cbr=0 name='''''' size='''480x16'''

and these:

Code: [Select]

EDMAC#6: addr=0xdb76000 conn=18 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#12: addr=0xdc28000 conn=14 cbr=0 name='''''' size='''864x288'''
EDMAC#6: addr=0xdb88c00 conn=18 cbr=0 name='''''' size='''4096x13, 752'''
EDMAC#12: addr=0xdb76000 conn=14 cbr=0 name='''''' size='''4096x41, 128'''

aprofiti · « **Reply #19 on:** December 16, 2018, 02:29:57 PM »

I'm trying to replicate on 50D using EDMAC Log, but I'm a bit stuck because it seems it's missing some infos...

Here is the CR2 ISO 100 log

Code: [Select]

EDMAC#0: addr=0x5a10a64 conn=2 cbr=0 name='''''' size='''480x16'''
EDMAC#1: addr=0x5a12c64 conn=28 cbr=0 name='''''' size='''(476, skip 36) x 176'''
EDMAC#2: addr=0x400637c conn=0 cbr=0 name='''''' size='''8456x3228'''
EDMAC#6: addr=0x5a2ade4 conn=18 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#8: addr=0x3de078 conn=0 cbr=0 name='''''' size='''2000x224'''
EDMAC#9: addr=0x3d3a9c conn=15 cbr=0 name='''''' size='''8x3228'''
EDMAC#12: addr=0x5a2ade4 conn=14 cbr=0 name='''''' size='''4096x41, 128'''
EDMAC#16: addr=0x1f50000 conn=23 cbr=0 name='''''' size='''4096x13, 752'''
EDMAC#3: addr=0x2014088 conn=5 cbr=0 name='''''' size='''7400x4096'''
EDMAC#4: addr=0x143ec070 conn=22 cbr=0 name='''''' size='''((1212, skip 2412) x 395, 1212, skip -1431480) x 2,   (1200, skip 2424) x 396'''
EDMAC#10: addr=0x4000048 conn=0 cbr=0 name='''''' size='''((2884, skip 5572) x 3231, 2884, skip -27321392) x 2,   (2828, skip 5628) x 3232'''
EDMAC#3: addr=0x1454c07c conn=5 cbr=0 name='''''' size='''3260x4096'''
EDMAC#4: addr=0x1f12a30 conn=3 cbr=0 name='''''' size='''48x9x396'''
EDMAC#10: addr=0x4071962 conn=0 cbr=0 name='''''' size='''(((1036, skip 7420) x 13, 1036, skip -110040) x 8,   (1036, skip 7420) x 13, 1036, skip -50708) x 396'''
EDMAC#11: addr=0x5a12c64 conn=33 cbr=0 name='''''' size='''512x176'''
EDMAC#13: addr=0x1f10000 conn=3 cbr=0 name='''''' size='''432x446'''
EDMAC#3: addr=0x1b00f00 conn=6 cbr=0 name='''''' size='''3840x1079'''
EDMAC#11: addr=0x1b00000 conn=6 cbr=0 name='''''' size='''(3840, skip -3840) x 1079'''
EDMAC#3: addr=0xd00000 conn=3 cbr=0 name='''''' size='''(((120, skip 3444) x 2, 120, skip -7128) x 29,   (84, skip 3480) x 2, 84) x 396'''
EDMAC#11: addr=0x1b00000 conn=3 cbr=0 name='''''' size='''1440x252'''

This is what I've done:
- Compiled ML from iso-research branch
- Enabled modules edmac.mo, adtg_gui.mo and raw_diag.mo
- In EDMAC Tool menu I have set Log very 2000us
- Start logging EDMAC by pressing half shutter
- Quickly press half-shutter again to take a cr2 photo

Also how to compile qemu-frsp? if I use the make file from it's folder it tells me that it's missing Camera MODEL.
Do I need to copy minimal.c to /src directory?

Greg · « **Reply #20 on:** December 16, 2018, 04:31:41 PM »

These logs are incomplete.

Quote from: aprofiti on December 16, 2018, 02:29:57 PM

- In EDMAC Tool menu I have set Log very 2000us

Try 100us or 50us.

dfort · « **Reply #21 on:** December 16, 2018, 05:20:52 PM »

Quote from: aprofiti on December 16, 2018, 02:29:57 PM

Also how to compile qemu-frsp? if I use the make file from it's folder it tells me that it's missing Camera MODEL.

Code: [Select]

make MODEL=50D

Danne · « **Reply #22 on:** December 16, 2018, 08:54:43 PM »

Quote from: Greg on December 16, 2018, 04:31:41 PM

These logs are incomplete.
Try 100us or 50us.

Are mine incomplete too you mean Greg?
Also, how to obtain the full logs? Press full shutter and actually fire off a CR2 and wait for the log?
While taking the full resolution dng and wait for the log?
Sorry but in need of guidance if we could get any. Would be really cool to be able to use and test out this module on other cameras but would certainly speed up things if steps and outlines could be provided.
Thanks.

g3gg0 · « **Reply #23 on:** December 17, 2018, 12:10:47 AM »

Quote from: Greg on December 16, 2018, 12:19:00 AM

Several lines:

which table did you modify for these horizontal lines?

aprofiti · « **Reply #24 on:** December 17, 2018, 01:22:26 AM »

Quote from: Greg on December 16, 2018, 04:31:41 PM

These logs are incomplete.
Try 100us or 50us.

Tried again with 50us and got infos for iso 200 and iso 400 but wasn't working with iso 100/800/1600/3200 (neither 100us or 200us).

Iso 200:

Code: [Select]

EDMAC#9: addr=0x31832c conn=15 cbr=0 name='''''' size='''4832x4'''
EDMAC#0: addr=0x5a10a64 conn=2 cbr=0 name='''''' size='''480x16'''
EDMAC#1: addr=0x5a12c64 conn=28 cbr=0 name='''''' size='''(476, skip 36) x 176'''
EDMAC#2: addr=0x400637c conn=0 cbr=0 name='''''' size='''8456x3228'''
EDMAC#6: addr=0x5a2ade4 conn=18 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#8: addr=0x3de078 conn=0 cbr=0 name='''''' size='''2000x224'''
EDMAC#9: addr=0x3d3a9c conn=15 cbr=0 name='''''' size='''8x3228'''

Before and after 1\25 +5EV (mean values and calibration changes on different runs)

Iso 400:

Code: [Select]

EDMAC#9: addr=0x3265c4 conn=15 cbr=0 name='''''' size='''4832x4'''
EDMAC#0: addr=0x5a10a64 conn=2 cbr=0 name='''''' size='''480x16'''
EDMAC#1: addr=0x5a12c64 conn=28 cbr=0 name='''''' size='''(476, skip 36) x 176'''
EDMAC#2: addr=0x400637c conn=0 cbr=0 name='''''' size='''8456x3228'''
EDMAC#6: addr=0x5a2ade4 conn=18 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#9: addr=0x3d3a9c conn=15 cbr=0 name='''''' size='''8x3228'''
EDMAC#8: addr=0x3de078 conn=1 cbr=0 name='''''' size='''2000x224'''

Quote from: dfort on December 16, 2018, 05:20:52 PM

Code: [Select]
make MODEL=50D

Thank you! Had to make small changes to minimal.c and prop_values.c to solve some errors of missing stubs (sync_caches() and display_idle())

This is what I in console when running in Qemu:

Code: [Select]

[EDMAC#9] Starting transfer from 0x31832C to <15>, 4832x4, flags=0x10000
[EDMAC#9] 19328 bytes read from 31832C-31CEAC.

How do I change iso from there?

News:

Author Topic: Sensor Factory Calibration (Read 16135 times)