Messages

[Horizontal correction]

An easy way to find RAM address.
edmac log :

Code Select Expand

EDMAC#9: addr=0x30a10c conn=15 cbr=0 name='''''' size='''4832x4'''       <------------------------
EDMAC#0: addr=0x79df1a4 conn=2 cbr=0 name='''''' size='''480x16'''
EDMAC#1: addr=0x79e13a4 conn=28 cbr=0 name='''''' size='''(476, skip 36) x 176'''
EDMAC#2: addr=0x600637c conn=0 cbr=0 name='''''' size='''8456x3203'''
EDMAC#6: addr=0x79f9524 conn=18 cbr=0 name='''''' size='''4096x18, 3072'''
EDMAC#9: addr=0x39a3f4 conn=15 cbr=0 name='''''' size='''8x3203'''
EDMAC#8: addr=0x3a48f4 conn=1 cbr=0 name='''''' size='''2000x224'''

Horizontal correction
The address is different for FRSP and CR2
ISO 100 regular picture CR2 (0x39a3f4)
In 500D ROM and RAM is empty, canon does not use it, but if you enter some values manually, it changes the black level of the horizontal line.

It probably will not be useful, we must remember that fixed pattern noise is not random pattern noise.
We can not correct something that is random in this way.

[cmos_cal.mo]

cmos_cal.mo
https://bitbucket.org/Gr3g01/ml-cmos-cal/src/iso-research/modules/cmos_cal/



To run on another camera:
- SENSOR_VFPN_ADDR_ISO,
- SENSOR_FULL_WIDTH,
- line 325: "if(p < 512 || p > 1536) // digic IV cameras black level 1024 - fix it",

The code is still not clean. But I think someone can try results on 5D2 (is similar to 500D).

TODO :
- I think that the whole module should be rewritten
- use silent.mo to take frsp with mirror down,
- save/load data into a file,
- improve the algorithm of calculation to reduce the number of frames and less heat,
- increase the accuracy of calibration below 1,
- validation - comparison of effects before and after calibration,

as the offsets are integer, odd numbers aren't possible. at least in-camera
would have to use float-files which are a bit overkill.

This should be possible in the camera.
The "B" value can change column black level below 1.

Code Select Expand

B = 0 -> column black level = 1233.88
B = 1023 ->  column black level = 1259.98
diff = 26.1

But this is an offset from the black level of the entire frame...

Code Select Expand

B = 0 -> column black level = 1129.34
B = 1023 ->  column black level = 1142.50
diff = 13.6

B = 0 -> column black level = 1037.44
B = 1023 ->  column black level = 1038.96
diff = 1.52

(column_black_level - frame_black_level)*12.5%

So 1023.5 we can change to 1023.56
It is possible that we need to change some engio register, some offset.

@g3gg0,
1) yes
2) 500D yes, I do not know how other cameras. This could be it - https://www.spt.info/index.php/service-adjustment-software?id=21
3) I have a module that takes 64 FRSP dark frames, it takes 9 minutes to process this data (in camera), then the calculated values are saved in RAM.

Calibration in this way:

Code Select Expand

int new_value = current_value - (int)roundf(allframes_mean - column_allframes_mean[x]);
It's possible that raw_diag does some more advanced things. I just skip cold/hot pixels.

CR2 and FRSP are cleaner (FPN), 500D is not good for video so I do not know, I did not check it.
The module requires code cleanup and some work, currently it is only a tool that is not good for official publication.

We can also use something like two_add engine to speed up the processing. But is it necessary?

My main goal was to calibrate FPN for 500D with ADTG modifications. The effect is good, now it's a new camera. 
Current limitation - I can change the black level of the column by 1. So some of the columns still have 1024.5 or 1023.5.

Averaged 16 frames to make the effect more visible.

ISO 1600
(Canon default | automatic calibration)

The first attempts of automatic calibration :


The first image is a dark frame with ADTG-CMOS amplifier reduced. The second image, after automatic calibration.

[Sensor Factory Calibration]

Sensor Factory Calibration
(500D)

RAM (ISO100) 0x30A10C - 0x30EC8C
ROM (ISO100) 0xFFA0590C - 0xFFA0A48C

RAM

Code Select Expand

ISO 100 0x30A10C
ISO 200 0x30EC94
ISO 400 0x31381C
ISO 800 0x3183A4
ISO 1600 0x31CF2C
ISO 3200 0x321AB4
ISO 6400 0x32663C
ISO 12800 0x32B1C4

I found the adress in RAM using QEMU FRSP-log:

Code Select Expand

[EDMAC#9] Starting transfer from 0x30A10C to <15>, 4832x4, flags=0x10000
[EDMAC#9] 19328 bytes read from 30A10C-30EC8C.

How to find in RAM?

Code Select Expand

Find hex values :
4D 00 01 01 88 4B 00 00
4D -> ISO 100?
88 4B -> 0x4b88 = ‭4834‬ // 4834 - 2 = 4832 500D width

Now let's try to find in 5D3 rom
‭5936‬ + 2 = 5938
5938*4 = 0x‭5CC8‬

4D 00 01 01 C8 5C 00 00
The address is after these hex values


What is this data?

The data is packed in 16 bits of 2 values for each column.

Left optical black [512, 0] (these are approximate values, different for each column):


After left optical black [512, 512] (these are approximate values, different for each column):


Changing the first value affects the black level column by -1.
Black level 1024
value -1 -> 1025
value +1 -> 1023

Dark frames (+5EV)

(Canon Default | Whole buffer filled [0,0] | Whole buffer filled [512,512] | Buffer after left optical black filled [512,512]) :

No worries, I prefer to buy the D3300 (it's cheaper than 1300D) than get a free 750D.

qemu job rom      - Firmware version: 1.0.0 / 8.5.1 B4 (52)
felipemazieri rom - Firmware version: 1.0.0 / 8.7.1 B5 (53)



So 750D has several firmware versions something like 70D?

It looks like no one wants a sensor in technology from 15 years ago. 

I found this in dm-spy log and my interpretation. Edmac configuration also explains this.
I needed this to configure the image processor to live view/video mode.

Looking at your link - http://lclevy.free.fr/cr2/#app
Look at EOS Kiss X3 (500D) :

Code Select Expand

sensor width = 4832
slices[0] = 2
slices[1] = 1616
slices[2] = 1600

1616*2 + 1600 = 4832

Of course, the sizes needed for decoding may be slightly different.

Any plans with digic 6/7?

It works with live view (not decoded) but we have a problem.

When I take a silent simple picture (LV) it will wait with processing until I get out of LV (just go to play mode).

Conflict with live view reslock :

Code Select Expand

0x5000d,
0x5000f,
0x5001a,
0x80000,
0x130006,
0x130008,
0x130012,
0x130014,
0x130018,
0x13001c,

Some notes :

Code Select Expand

int slice1 = (width + 80 + 32) / 3;
int slice2 = (width + 80 + 32) / 3;
int slice3 = (width + 80 + 32) / 3 - 32;

int sliceerr = slice1 + slice2 + slice3 - width;
   
if (sliceerr != 80)
{
    printf("ERROR slice size margin %d\n", sliceerr);
    return -2;
}

EngDrvOut(0xc0f08548, PACK32(slice3 - 2, height + 3));
EngDrvOut(0xc0f08554, PACK32(width + 16, height + 4));
EngDrvOut(0xc0f0858c, slice1 - 32);
EngDrvOut(0xc0f08594, slice1 - 2);
EngDrvOut(0xc0f0d014, PACK32(slice3, height + 3));
EngDrvOut(0xc0f0d018, slice3 + 11);
EngDrvOut(0xc0f0d01c, slice2);
EngDrvOut(0xc0f0d020, slice2 + 3);
EngDrvOut(0xc0f0d024, slice1);
EngDrvOut(0xc0f0d054, PACK32(slice3 + 7, height + 4));
EngDrvOut(0xc0f0d05c, slice2 - 9);
EngDrvOut(0xc0f0d080, PACK32(slice3 + 7, height - 1));
EngDrvOut(0xc0f0d088, slice2 - 9);
EngDrvOut(0xC0F13068, PACK32(width - 1, height - 1));
EngDrvOut(0xc0f12010, PACK32(slice1 - 33, height - 37));
EngDrvOut(0xc0f12014, PACK32(slice3 - 17, height - 37));
EngDrvOut(0xc0f1201c, PACK32((slice1 - 33) / 8, (height  - 37) / 8));
EngDrvOut(0xc0f12020, PACK32((slice3 - 17) / 8, (height  - 37) / 8));

struct edmac_info RD1_info = {
    .xa     = slice1 * 14/8,
    .xb     = slice3 * 14/8,
    .yb     = height + 4 - 1,
    .xn     = 2,
    .off1a  = width * 14/8 - slice1 * 14/8,
    .off1b  = width * 14/8 - slice3 * 14/8,
    .off2b  = (((width * 14/8) * (height + 4 - 1) + 56) * -1 & 0xFFFFFFE),  //digic 3/4
};

500D ML TTJ_Start :

Code Select Expand

*** StartEDmac(0x3, 0x0), from ff28f6e4
    addr c000064, ptr c000064, size
StartEDmac(^^)/=== (3)(0)(0xc000064) ===
[ENGIO](Addr:0xc0f04304, Data:0x20000001) EngDrvOut  //WR1
[ENGIO](Addr:0xc0f04300, Data:0x       1) EngDrvOut  //WR1
[ENGIO](Addr:0xc0f12000, Data:0x       1)[0]
[ENGIO](Addr:0xc0f13048, Data:0x       3)[1]
[ENGIO](Addr:0xc0f13060, Data:0x       0)[2]
[ENGIO](Addr:0xc0f13000, Data:0x       0) EngDrvOut
[ENGIO](Addr:0xc0f13048, Data:0x       3) EngDrvOut
[ENGIO](Addr:0xc0f1304c, Data:0x       0) EngDrvOut
[ENGIO](Addr:0xc0f13004, Data:0x       1) EngDrvOut
[ENGIO](Addr:0xc0f0d000, Data:0x       1)[0]         
[ENGIO](Addr:0xc0f08540, Data:0x       1)[1]         //RSHD_ENB
[ENGIO](Addr:0xc0f08070, Data:0x       1)[2]         //UNPACK24
*** StartEDmac(0xa, 0x2), from ff251bc0
    addr 8008484, ptr 8008484, size
StartEDmac(^^)/=== (10)(2)(0x8008484) ===
[ENGIO](Addr:0xc0f04a04, Data:0x60020000) EngDrvOut  //RD1
[ENGIO](Addr:0xc0f04a00, Data:0x       1) EngDrvOut  //RD1


[https://www.diffnow.com/?report=92yqa]

It works with FRSP but the image is not decoded.

Code Select Expand

if (is_camera("500D", "1.1.1"))
{
    /* uses TwoInTwoOutJpegPath, but functions are mostly compatible */
    TTL_SetArgs     = (void *) 0xFF1AD47C;
    TTL_Prepare     = (void *) 0xFF252748;
    TTL_RegisterCBR = (void *) 0xFF2517D0;
    TTL_Start       = (void *) 0xFF2527B8;
    TTL_Stop        = (void *) 0xFF2527F0;
    TTL_Finish      = (void *) 0xFF252828;

    TTL_ResLock     = (void *) MEM(0x39D0);
}

500D edmac RD1 default configuration :

Code Select Expand

struct edmac_info RD1_info = {
    .xa     = 1648 * 14/8,
    .xb     = 1616 * 14/8,
    .yb     = 3208 - 1,
    .xn     = 2,
    .off1a  = 4832 * 14/8 - 1648 * 14/8,
    .off1b  = 4832 * 14/8 - 1616 * 14/8,
    .off2b  = 0xe623490,
};

struct edmac_info RD1_info = {
    .xa     = slice1 * 14/8,
    .xb     = slice3 * 14/8,
    .yb     = height + 4 - 1,
    .xn     = 2,
    .off1a  = width * 14/8 - slice1 * 14/8,
    .off1b  = width * 14/8 - slice3 * 14/8,
    .off2b  = (((width * 14/8) * (height + 4 - 1) + 56) * -1 & 0xFFFFFFE),  //digic 3/4
};


7D edmac RD1 default configuration :

Code Select Expand

struct edmac_info RD1_info = {
    .xa     = 1760 * 14/8,
    .xb     = 1920 * 14/8,
    .yb     = 3520 - 1,
    .xn     = 2,
    .off1a  = 5360 * 14/8 - 1760 * 14/8,
    .off1b  = 5360 * 14/8 - 1920 * 14/8,
    .off2b  = 0xe08556c,
};

Comparison of configuration 500D vs 7D - https://www.diffnow.com/?report=92yqa

Current interpretation - http://s24.postimg.org/uu3knpc0z/ttj-config3.png/

500D :

Code Select Expand

634AA> FrontShtDe:ff085618:96:05: sdfExecuteMem1ToRawPath(84538)
6352E> FrontShtDe:ff0859a8:96:05: sdfExecuteMem1ToRawPath(84538)©(SemOK)
6356D> FrontShtDe:ff0859e0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(84538)
645F9> FrontShtDe:ff252004:16:03: [TTJ][150,84538,0] RAW(4832,3204,0,14)
64634> FrontShtDe:ff191590:15:02: [JPCORE] SetEncodeLosslessType 1
64669> FrontShtDe:ff1929d0:15:02: [JPCORE] SetEncodeLosslessParam 1208,3204              <- note: 4 slices
84993> **INT-64h*:ff251de0:16:03: [TTJ][150,84538,0] JpCoreCompleteCBR( 0xd01f99 )
8647D> FrontShtDe:ff085a10:96:05: ProcessTwoInTwoOutJpegPath(R) End(84538)

ISO 100
dark frame - 50%
tree leaves - 70-75%

UnLockEngineResources :

Code Select Expand

851E4> FrontShtDe:00092990:00:00: *** UnLockEngineResources(869d98) x31 from ff2519bc:
8521F> FrontShtDe:00092a08:00:00:      1)    10002 (read channel 0xa)
85244> FrontShtDe:00092a08:00:00:      2)        3 (write channel 0x3)
85264> FrontShtDe:00092a08:00:00:      3)        4 (write channel 0x4)
85284> FrontShtDe:00092a08:00:00:      4)    30000 (read connection 0x0)
852A8> FrontShtDe:00092a08:00:00:      5)    20005 (write connection 0x5)
852CC> FrontShtDe:00092a08:00:00:      6)    20016 (write connection 0x16)
852EA> FrontShtDe:00092a08:00:00:      7)    50003 (?)
85307> FrontShtDe:00092a08:00:00:      8)    5000d (?)
85324> FrontShtDe:00092a08:00:00:      9)    5000f (?)
85341> FrontShtDe:00092a08:00:00:     10)    5001a (?)
8535E> FrontShtDe:00092a08:00:00:     11)    80000 (?)
8537B> FrontShtDe:00092a08:00:00:     12)    90000 (?)
85399> FrontShtDe:00092a08:00:00:     13)    a0000 (?)
853B6> FrontShtDe:00092a08:00:00:     14)   160000 (?)
853D5> FrontShtDe:00092a08:00:00:     15)   130003 (?)
853F3> FrontShtDe:00092a08:00:00:     16)   130004 (?)
85412> FrontShtDe:00092a08:00:00:     17)   130005 (?)
85430> FrontShtDe:00092a08:00:00:     18)   130006 (?)
8544F> FrontShtDe:00092a08:00:00:     19)   130007 (?)
8546D> FrontShtDe:00092a08:00:00:     20)   130008 (?)
8548C> FrontShtDe:00092a08:00:00:     21)   13000c (?)
854AB> FrontShtDe:00092a08:00:00:     22)   130010 (?)
854C9> FrontShtDe:00092a08:00:00:     23)   130011 (?)
854E8> FrontShtDe:00092a08:00:00:     24)   130012 (?)
85507> FrontShtDe:00092a08:00:00:     25)   130013 (?)
85526> FrontShtDe:00092a08:00:00:     26)   130014 (?)
85544> FrontShtDe:00092a08:00:00:     27)   130015 (?)
85563> FrontShtDe:00092a08:00:00:     28)   130018 (?)
85582> FrontShtDe:00092a08:00:00:     29)   13001a (?)
855A0> FrontShtDe:00092a08:00:00:     30)   13001b (?)
855BF> FrontShtDe:00092a08:00:00:     31)   13001c (?)

Resolution :

Code Select Expand


6583E> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f08548, Data:0x c87064e)[29]
6589B> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f08554, Data:0x c8812f0)[31]
660BD> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f13068, Data:0x c8312df)[78]
661F2> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12010, Data:0x c5f064f)[85]
66224> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12014, Data:0x c5f063f)[86]
66281> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f1201c, Data:0x 18b00c9)[88]
662B2> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12020, Data:0x 18b00c7)[89]

[2040x1268]

In 1:1 a surprising 2040x1268  and in 3x crop (5x Zoom) 2312x1128 , Really !
I was under the impression that full Liveview was only 1880x1250  and 3x crop was 2152x1078.

This is probably a resolution with black borders.

raw.c

Code Select Expand

#ifdef CONFIG_5D2
skip_top        = zoom ?   52 : 18;
skip_left       = 160;
#endif

I spent with ML code several thousand hours for free.
Do you think it's fair?
If I worked for money. Today I should buy a house, a car and a camera. But I can not because I worked for free.

Alex should be a millionaire. But he also works for free.

The only person here who collects private donations is g3gg0.
Of course, he spent with ML much more time than me. But it's a little unclear.

Will this thread lead to the end of ML?

It was in live view 5x zoom. Both buffer patch and edmac size.

The buffer should not be needed :

silent.c

Code Select Expand

/* Reprogram the raw EDMAC to output the data in our buffer (ptr) */
raw_lv_redirect_edmac(sp_frames[next_slot % sp_buffer_count]);
sp_num_frames++;

raw.c

Code Select Expand

/* to be called from vsync hooks */
void FAST raw_lv_redirect_edmac(void* ptr)
{
    #ifdef CONFIG_EDMAC_RAW_SLURP
    redirected_raw_buffer = (void*) CACHEABLE(ptr);
    #else
    MEM(RAW_LV_EDMAC) = (intptr_t) CACHEABLE(ptr);
    #endif
}

So ML uses its own buffer, which should have the correct size after patching EDMAC.

If you do not solve this problem, we will announce the end of digic IV support.

Buffer patched :

Code Select Expand

void* buf = malloc(2064 * 2000 / 8 * 14);
MEM(0x16604 + 808) = buf;



It still works a bit randomly, some frames have standard height...

So if someone finds a solution, it will work.
I do not plan further research.

I increased the height by 140px (841px to 981px). More does not work.
I do not know why, some problem with memory or edmac.

To do this you need to change (500D):

Code Select Expand

CMOS[5] //vertical offset
ADTG[2]1172 -> old_val + YRES_DELTA (nrzi)
ADTG[2]1173 -> old_val + YRES_DELTA (nrzi)
ADTG[2]1178 -> old_val + YRES_DELTA (nrzi)
ADTG[2]1179 -> old_val + YRES_DELTA (nrzi)
ADTG[2]1214 -> old_val + YRES_DELTA (nrzi)
ADTG[2]120A -> old_val + YRES_DELTA (nrzi)
C0F0713C -> 0x370 + YRES_DELTA
C0F07150 -> 0x388 + YRES_DELTA
C0F06088 //RAW last line|column
C0F06008 -> 0xaf90af9   //photo mode value
C0F0600C -> 0xaf90af9  //photo mode value
C0F06010 -> 0xaf9    //photo mode value
C0F06014 -> 0xc83   //photo mode value
patch EDMAC size
patch buffer

500D  LV zoom 5x - "StartPass_x5 CrawAddr:%lx / Kind:%d"

Code Select Expand

SetEDmac(0x10, *(0x16604 + 808), 0xFF44F0B0, 0x20000000);

ROM:FF44F0B0
ROM:FF44F0C8        DCD 0xE1C, 0      //‭3612‬
ROM:FF44F0D0        DCD 0x35D, 0, 0   //861

5D2  LV zoom 5x - "StartImagePass_x5 CrawAddr : %lx / KindOfCraw : %d"

Code Select Expand

SetEDmac(0x5, *(0x13ECC + 804), 0xFFCAA1D4, 0x20000000);

ROM:FFCAA1D4
ROM:FFCAA1EC        DCD 0xFCE, 0       //‭4046
ROM:FFCAA1F4        DCD 0x467, 0, 0  ‭  //1127‬


This code will increase EDMAC size, but the buffer will be overflowed - it's dangerous.

Code Select Expand

static void run_test()
{
    static uint32_t craw_5x_edmac_h = 0xFF44F0D0;
    patch_memory(craw_5x_edmac_h, 861, 861 + YRES_DELTA, "lv_edmac_h"); // add height
}

Now we should use our own buffer.

4046/14*8 = 2312
So the size is 2312 x 1127

500D "LV_Initialize %s" :

Code Select Expand

int a1 = 0x1D78 + 0x2600;
*(a1 + 208) = 1664;
*(a1 + 210) = 660;
It looks like width and height raw, but who knows.

This will not work on old cameras (500D, 5D2 etc) until you find a way to change the EDMAC size.

Old cameras do not support #ifdef CONFIG_EDMAC_RAW_SLURP
https://bitbucket.org/hudson/magic-lantern/src/11f405b62b312294d705e8e7bb8688b820bb9eea/src/raw.c?at=crop_rec_4k&fileviewer=file-view-default#raw.c-429

Maybe patch here :

Code Select Expand

LiveViewMg:000929e0:00:00: *** ConnectWriteEDmac(0x10, 0x0), from ff1f7134
LiveViewMg:000929e0:00:00: *** SetEDmac(0x10, 0x463390a4, 0xff44e5c8, 0x20000000), from ff1f7164

Or try "#define CONFIG_ALLOCATE_RAW_LV_BUFFER"
https://bitbucket.org/hudson/magic-lantern/src/11f405b62b312294d705e8e7bb8688b820bb9eea/src/raw.c?at=crop_rec_4k&fileviewer=file-view-default#raw.c-126

I'm afraid there may be a problem with raw buffer. Something like 10/12bit raw video with old cameras.

Sorry it will not be possible.
It's just a hint for someone like reddeercity but with 500D.