EekoAddRawPath

[a1ex]

Starting from the latest discovery from g3gg0, I've looked into some routines that appeared to do raw additions and subtractions (EekoAddRawPath). They seem to work, you can also do some scaling, compute min/max, and they are fairly fast (about 60 ms for a full-size raw buffer, enough for stacking some photos in the camera).

Problem: these functions are only available on DIGIC 5 cameras. However, g3gg0 found out they use the TWOADD module (present on all cameras), so there might be some hope.

5D3 stubs:

Code: [Select]

/* 1.2.3 */
void (*EekoAddRawPath)(void *a, void *b, void *out, int op, int off_a, int gain_a, int black_a, int gain_b, int black_b, int div8, int out_off, void (*cbr)(void*), void* cbr_arg) = 0xFF32C538;
void (*EekoAddRawPath_cleanup_engio)() = 0xFF5127F0;
void (*EekoAddRawPath_cleanup_reslock)() = 0xFF512698;

/* 1.1.3 */
FF327A54 (called after "%s Addsub Count:%d")
FF507CE8 (called after "stsCompleteMultiExp", after BEQ)
FF507B90 (called next)

Basic call to add two images:

Code: [Select]

    EekoAddRawPath(image_a, image_b, image_out, 0, 0, 4096, 2048, 4096, 2048, 0, 0, (void(*)(void*))give_semaphore, eeko_sem);
    take_semaphore(eeko_sem, 5000);
    EekoAddRawPath_cleanup_engio();
    EekoAddRawPath_cleanup_reslock();

Operations:

To figure them out, I've created two gradients (raw image buffers with fake data), horizontal and vertical, with values from 0 to 16383, and used them as operands for the Eeko routine. The third image is the result.

0: a + b


1: a - b


2: max(a, b)


3: min(a, b)


Octave code of what it does (incomplete, doesn't model all overflows):

Code: [Select]

function out = eeko(a, b, op, off_a, gain_a, black_a, gain_b, black_b, div8, out_off)

    % valid range for the parameters
    op = bitand(op, 3);
    gain_a = bitand(gain_a, 8191);
    gain_b = bitand(gain_b, 8191);
   
    % offset image A
    a = a + off_a;
    a = max(a, -2048);

    if gain_a ~= 4096,
        % scale image A, relative to black level
        a = round(max((a - black_a) * gain_a / 4096, -2048) + black_a);

        % note: without the "if", clamping to 16383 here will fail some tests
        a = min(a, 16383);
    end
   
    if black_a > 4096
        % some strange overflow
        a = min(a + 8192, 16383);
    end
   
    if gain_b ~= 4096,
        % scale image B, relative to black level
        b = round(max((b - black_b) * gain_b / 4096, -2048) + black_b);

        % note: without the "if", clamping to 16383 here will fail some tests
        b = min(b, 16383);
    end
   
    % perform some operation between A and B
    switch op
        case 0
            out = a + b;
        case 1
            out = a - b;
        case 2
            out = max(a, b);
        case 3
            out = min(a, b);
    end

    out = coerce(out, 0, 16383);

    if div8
        % darken image by 3 stops (not adjustable)
        % optionally offset by image; for some reason, the offset is multiplied by 7/8 (why?)
        out = round((out + out_off*7) / 8);
    end
   
    out = coerce(out, 0, 16383);
end

function y = coerce(x, lo, hi)
    y = min(max(x, lo), hi);
end

Notes:
- image_out buffer can be reused (you may use either image_a or image_b)
- to change resolution, one has to call some lower-level routines, with uglier syntax

To figure out the meaning of the parameters, I've started from the basic call and changed the values (one or two at a time). Then, by trial and error, I've adjusted the octave code (eeko.m) until it matched the images saved from camera.

Test data, camera code and octave scripts can be found here: https://www.dropbox.com/sh/k037ulitl3kkf0t/AABXZYcTQYDqvqMQ-bNrHRtpa?dl=0

[1%]

The twoadd might convert 14 to 12?  TWOADD_SETUP_14_12

[a1ex]

Yeah, that's why I said we may hope for 12-bit conversion. But so far I have no idea how to use it.

[Levas]

Ah, maybe 14 to 12 conversion

Was already wondering what was so interesting about these set of routines.
Raw image blending can be perfectly done in post, outside the camera...
And why put functions in ML which can be easily done in post.

14 to 12 conversion can be done in post too 

[a1ex]

Yes, but I may want to take a long exposure (say 1 minute) made up of silent pictures (say 5fps, because it's daylight and can't use longer exposures), and that would require 1.2 GB. For a 5-minute exposure, I would get a 6 GB MLV file. In this case I'd prefer to have all these frames merged in the camera, and only save a 40MB DNG. Or maybe two, if the 14 bits will not be enough.

[Levas]

Ah, that's useful indeed.
Sort of neutral density filter replacement for photos.
I can see myself using it for photo's of moving water or crowded places in the city(and get a photo with no people on it)

[a1ex]

Speaking of crowded places: since we have add, sub, min, max and scaling, can you figure out how to compute the median, just from these operations?

(the median is good at removing the people from stacked images)

[nikfreak]

...
14 to 12 conversion can be done in post too 

Dunno if you guys are talking bout raw but corect me if i am wrong (we had a long thread which is closed now):
12 is less data so needs less time to write to ram/sdcard.

[1%]

It could be fast enough for real time if a smaller frame takes less than 41.66ms to convert.

[Levas]

"12 is less data so needs less time to write to ram/sdcard"
                     
That's why I put the  behind my words...


Add, sub, min, max, and scaling...median
Edit: only way to do median if you have the data of all the pictures, the whole set...
Only two pictures can be processed at a time I guess ?

Did one of you already succeed in doing this functions on smaller frames ?
No doubt this function is used by canon to do in camera double exposures and noise reduction on long exposures.
But can it also handle smaller frames ?

[chris_overseas]

I think this is about the best that can be done for finding (an approximation of) the median without having to store all the frames:

http://web.ipac.caltech.edu/staff/fmasci/home/statistics_refs/Remedian.pdf

Multiple frames of storage are still required though. With three frames the median is easy to compute, max(A, min(B, C)). Three frames may not be enough but it might be worth a try?

[a1ex]

Great find.

BTW, max(5,min(4,3)) is not median.

[chris_overseas]

Argh good point - that'll teach me for trying to use my brain too late at night. How about this one:

median = max(min(A,B), min(max(A,B),C)) Though that will require another temporary buffer (unless there's a way to set A=min(A,B), B=max(A,B) without each computation affecting the other? Presumably that would require trickery like row-by-row access and a row sized buffer).

Alternatively:

median = (A + B + C) - min(A, min(B,C)) - max(A, max(B,C)) But that has the same problem as above, plus overflow issues.

[a1ex]

On 5D3 I think you can allocate 5 buffers at once: http://www.magiclantern.fm/forum/index.php?topic=12528.msg120670#msg120670

so the first formula looks doable.

This may also provide some hints: http://ndevilla.free.fr/median/median/src/optmed.c

(I guess PIX_SORT(a,b) can be implemented as [min,max] = MINMAX(a, b) with a temporary buffer)

[chris_overseas]

(I guess PIX_SORT(a,b) can be implemented as [min,max] = MINMAX(a, b) with a temporary buffer)

Indeed, the opt_med3() formula is exactly equivalent to max(min(A,B), min(max(A,B),C))

[g3gg0]

i did some research.

from what i understand, there are three pixel modes:

 0 - 10 bit per pixel
 1 - 12 bit per pixel
 2 - 14 bit per pixel
 3 - 16 bit per pixel

the function you found
  void EekoAddRawPath(void *img_a, void *img_b, void *img_out, int op, int minus_off_a, int gain_a, int black_a, int gain_b, int black_b, int div8, int out_off, void (__cdecl *cbr)(void *), void *cbr_arg)

it fills these structs with important stuff

struct struc_eeko_opts_shad // for image [a]
{
    uint16_t SHAD_PRESETUP_14_12; // set to -black_a
    uint16_t SHAD_POSTSETUP_14_12; // set to black_a
    uint32_t SHAD_MODE; // set to 0
    uint32_t SHAD_GAIN; // amplify by (gain / 1024)
    uint32_t SHAD_RMODE; // set to 1
} // probably sets up: out = (in - black) * gain + black

struct struc_eeko_opts // one for image [b, out]
{
    uint16_t presetup;
    uint16_t gain; // amplify by (gain / 128)
    uint16_t zero; // what does it do?
    uint16_t postsetup;
    uint16_t mask; // probably saturated maximum value
    uint16_t unused;
}

struct struc_eeko_img_ptr // one per image [a, b, out]
{
    void *ptr;
    uint32_t pixel_mode; // hardcoded to 2 - 14 bit per pixel
}



the opts_out.gain is being hardcoded to 0x80, which results in a gain by 0.5 - this means the addition result is divided by two. (averaging)


then it calls:

int EekoAddRawPath_Start(int a1, struct struc_eeko_img_ptr *img_a, struct struc_eeko_img_ptr *img_b, struct struc_eeko_img_ptr *img_out, int xsize, int ysize, int out_mode, struc_eeko_opts_shad *opts_shad, struc_eeko_opts *opts_img_b, int minus_off_a, struc_eeko_opts *opts_img_out, void (__cdecl *cbr)(void *), int cbr_ctx)

with out_mode==2, so probably hardcoded to 14 bit per pixel output here *too*.
xsize and ysize are read from pDsDefine and are the MEM1 resolution.

EekoAddRawPath_Start now copies all onto stack again, sets the callback and calls the real processing function.
it checks which bit mode per in/out image is set and sets up the engines properly.

i would say: can you cache hack/manually reimplement the calling function and replace 2 by 1 for 12 bit?



edit: updated as suggested by a1ex

[nikfreak]

simply wow! 

[ItsMeLenny]

Just adding a comment so I can follow this post.

And I'm available to test on 550D if it makes it to it. (550D is not Digic 5).

[a1ex]

simply wow! 

Can't wait to try

[g3gg0]

updated some details about the struc_eeko_opts struct.
gain is relative to 0x80 (128) for the image_b and image_out "engine part", whereas the shad parameter which is used for image_a is relative to 0x400 (1024)

[1%]

Which path constructs LV raw? maybe cache hacking there would be the simplest way to achieve 12 or 10 bit for video without re inventing the wheel.

[g3gg0]

as soon we understand the things, we can do that
yet this is finding which of the 480 small wheels on the car do what.

[g3gg0]

by the way, the LV_RAW_type_C0F37014 register is also involved when converting.
its set to 0x500 or 0x1C00. on multiple exposures its only 0x1C00 for the last exposure.

[g3gg0]

here are the functions how i reversed them in IDA, just for reference.

edit: http://pastebin.com/sCxBeF87

these enums make the code a bit more readable:

Code: [Select]

; enum enum_pixelmode
enum_pixelmode_10bpp  EQU 0
enum_pixelmode_12bpp  EQU 1
enum_pixelmode_14bpp  EQU 2
enum_pixelmode_16bpp  EQU 3
enum_pixelmode_4  EQU 4
enum_pixelmode_5  EQU 5
enum_pixelmode_6  EQU 6
enum_pixelmode_7  EQU 7

; enum enum_eeko_operation
enum_eeko_op_ADD  EQU 0
enum_eeko_op_SUB  EQU 1
enum_eeko_op_MAX  EQU 2
enum_eeko_op_MIN  EQU 3

; enum enum_eeko_chain
enum_eeko_chain_DSUNPACK  EQU 0
enum_eeko_chain_UNPACK32  EQU 1
enum_eeko_chain_ADUNPACK  EQU 2
enum_eeko_chain_UNK2  EQU 3
enum_eeko_chain_UNK3  EQU 4

; enum enum_eeko_outchain
enum_eeko_outchain_PACK32  EQU 0
enum_eeko_outchain_PACK16  EQU 1
enum_eeko_outchain_UNK  EQU 2


[a1ex]

Minor correction:

Code: [Select]

struct struc_eeko_opts
{
  __int16 minus_black_b;
  __int16 gain;
  __int16 zero;
  __int16 black;
  __int16 mask;
  __int16 field_A;
};