Author Topic: ProcessTwoInTwoOutLosslessPath  (Read 129904 times)

littlebobbytables

  • New to the forum
  • *
  • Posts: 36
Re: ProcessTwoInTwoOutLosslessPath
« Reply #125 on: April 05, 2017, 02:22:43 PM »
I too would be very interested in a build for my camera. I foolishly tried to build for the 50D in the compressed raw branch but the mlvlite module crashes the camera. I guess some work needs to be done for other (Digic IV) camera's?
50D, 300, BMPCC4K

domasa

  • Freshman
  • **
  • Posts: 93
Re: ProcessTwoInTwoOutLosslessPath
« Reply #126 on: April 08, 2017, 04:04:11 PM »
Quote
...processing options such as bad pixel or vertical stripe fixes are not working yet
Would it be possible delete some rows and columns during processing (or color black for better compression)?

It will be great for record some resolution unsupported by ML.
E.g.: use supported resolution 5796x3870 25p for record UHD (3840x2160) 25p.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #127 on: April 12, 2017, 06:59:50 AM »
Some people are playing around with adtg_gui in order to get other cameras working with the crop_rec on steroids. Of course another part of the puzzle is getting lossless compression working so I thought I'd try my luck with porting ProcessTwoInTwoOutLosslessPath to the the 700D and EOSM. First lesson, it is called ProcessTwoInTwoOutJpegath on the 700D and EOSM.

I started out with the crop_rec_4k branch working on the silent module. First problem, Lossless DNG is broken on that branch. Switch over to the compressed_raw branch and it looks like the camera specific part is limited to this:

Code: [Select]
    if (is_camera("700D", "1.1.4"))
    {
        /* ProcessTwoInTwoOutJpegath, 700D 1.1.4 */
        TTL_SetArgs     = (void *) 0xFF35F510;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF423650;  /* called right after ProcessTwoInTwoOutJpegath(R) Start; */
                                                /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF423B88;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF36B2D8;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF423960;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF424C84;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF424CBC;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

    if (is_camera("EOSM", "2.0.2"))
    {
        /* ProcessTwoInTwoOutJpegath, EOSM 2.0.2 */
        TTL_SetArgs     = (void *) 0xFF361498;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF427CBC;  /* called right after ProcessTwoInTwoOutJpegath(R) Start; */
                                                /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF4281F4;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF36D124;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF427FCC;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF4292F0;  /* called right after StopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF429328;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

Don't get too excited, this isn't working yet. I'm posting so others that might be working on it can compare notes and point out errors. I'm pretty sure I nailed TTL_SetArgs and TTL_Prepare the the others are rather iffy. They "look" good but neither camera can create Lossless DNG files.

Looking through lossless.c there seem to be some hardcoded values that are fine for the 5D3 but might need to be adjusted for other cameras?

By the way, for those of you who haven't combed through a disassembly there are some pretty interesting things in there. For example:
  • ./ShootVfx/ShootPotato/ShootPotatoState.c
  • ->FindObject LUCKY
  • LUCKY_ANGEL
and the very profound:
  • **'this'

5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Teamsleepkid

  • Member
  • ***
  • Posts: 229
Re: ProcessTwoInTwoOutLosslessPath
« Reply #128 on: April 12, 2017, 07:54:47 AM »
i thought even the 5dii wasn't making lossless dng files...isnt it still coming out as an mlv file? and looks like you're definitely making progress:)
EOS M

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #129 on: April 12, 2017, 08:21:27 AM »
The 5D2 has ProcessTwoInTwoOutJpegPath as do most other cameras. Note that the 700D and EOSM have ProcessTwoInTwoOutJpegath (typo?) while the 5D3 has both ProcessTwoInTwoOutJpegPath and ProcessTwoInTwoOutLosslessPath.

So is it possible to do lossless compression on all cameras? I don't know but probably. The 700D and EOSM compresses the CR2 files.

isnt it still coming out as an mlv file?

As far as I know lossless compression was first tried out on DNG files so that's where I'm starting.

5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12317
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #130 on: April 12, 2017, 09:54:36 AM »
My older attempt at 700D in QEMU (I've used it to understand how 5D3 TTJ works):

Code: [Select]
    if (is_camera("700D", "1.1.4"))
    {
        /* uses TwoInTwoOutJpegPath, but functions are mostly compatible */
        TTL_SetArgs     = (void *) 0xFF35F510;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF424BA4;      /* called right after StartTwoInTwoOutJpegPath Raw(R) Start(%d) */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF423B88;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) set_flags_700D;  /* this function is inline on 700D */
        TTL_Start       = (void *) 0xFF424C4C;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF423DD4;      /* called after sdfStopYuvToJpegPath(%d); unsure, need a dm-spy log */
        TTL_Finish      = (void *) 0xFF424CBC;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */

        TTL_ResLock     = (void *) MEM(0x25A60);    /* this should work outside LiveView (e.g. full-res silent pics) */
    }

static void set_flags_700D()
{
    EngDrvOut(0xC0F37010, (shamem_read(0xC0F37010) & 0xFFE3FFFF) | 0xC0000);
    EngDrvOut(0xC0F3704C, (shamem_read(0xC0F3704C) & 0xFFFFFFC0) | 0x20);
    EngDrvOut(0xC0F37074, (shamem_read(0xC0F37074) & 0xFFC0FFFF) | 0x180000);
    EngDrvOut(0xC0F37078, (shamem_read(0xC0F37078) & 0xFFC0FFFF) | 0x180000);
}

As dfort guessed, all of this is found by following Canon's CR2 compression routines (codenamed Mem1ToRaw). Unfortunately, there are differences between models, so it's not always just a pattern matching.

You've noticed ResLock requires some tweaking. What's up with it?

ResLock is a way to lock certain image processing modules (or "resources") that we are going to use (in order to prevent two tasks from using that image processing module at the same time). Each image processing "resource" is identified by a 32-bit resource ID (aka resID). From those, the most significant 16 bits are the class ID (category of resource), and the least significant bits are the resource ID within that class. Here you can find the known resource identifiers.

Each ResLock ID has an (internal) semaphore associated. Some resources (such as EDMAC channels / connections) may work without locking, as long as only one task uses them (otherwise you get corrupted data), but the compression ones refuse to work that way. Apparently, LockEngineResources also enables the clocks required by certain resources (it probably powers up the hardware required by those operations).

The first step is to try Canon's ResLock configuration outside LiveView (where all image processing resources are free, so their default reslock setup is likely to work).

What about LiveView?

In LiveView, many EDMAC channels are already assigned and locked (by Canon code); they will probably conflict with the choice used for CR2 compression (which is done outside LiveView). That means, we'll have to re-assign the EDMAC channels (and a good choice is to reuse the channels from edmac-memcpy.c, since they won't be used at the same time anyway).

On 5D3, I only had to redirect the write channel, but for a portable solution, the read channel might have to be redirected as well.

On this camera, the list of resources is found in TTL_SetArgs, for PictureSize == 0:
Code: [Select]
0x10000,             // RD1: first read channel (EDMAC channel 8) - didn't have to change it
0xD,                 // WR1: write channel #13 (look it up in edmac_chanlist - it's channel 22; changed to channel 17 for LiveView)
2,                   // WR2: write channel #2 (unused, deleted; didn't check what data it outputs)
0x30001,             // connection 1; don't remember why I've deleted it (possibly by mistake)
0x2002d ... 0x260003 // unknown, copied verbatim

On 700D, the list is at 0x334E4 (e.g. in a RAM dump).

For reference, here's the TTJ version for 5D3 1.2.3 (works only outside LiveView; code based on 700D):
Code: [Select]
    if (is_camera("5D3", "1.2.3"))
    {
        /* ProcessTwoInTwoOutJpegPath, 5D3 1.2.3 */
        TTL_SetArgs     = (void *) 0xFF327DE8;  /* using TTL version */
        TTL_Prepare     = (void *) 0xFF3D00A0;  /* called between sdfExecuteYuvToJpegPath and StartYuvToJpegPath */
        TTL_RegisterCBR = (void *) 0xFF3CF084;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) set_flags_700D;   /* using 700D version */
        TTL_Start       = (void *) 0xFF3D0148;  /* called after StartYuvToJpegPath; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF3D0180;  /* called right after sssStopMem1ToJpegPath */
        TTL_Finish      = (void *) 0xFF3D01B8;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

    uint32_t resources[] = {
        0x10000,                        /* read channel 0x8 */
        edmac_channel_to_index(0x11),   /* write channel 0x11 */
        0x30002,                        /* read conn 2 */
        0x20005,                        /* write conn 5 */
        0x50034,
        0x5002D,
        0x50010,
        0x90001,
        0x90000,
        0xA0000,
        0x160000,
        0x260000,
        0x260001,
        0x260002,
        0x260003,
    };

    /* copy this one from 0x33528 until 0x33918 (700D 1.1.4) */
    TTL_Args.engio_cmd_1_prepare = prepare_700D;

Overriding resolution is a different story; let's first get it working in full-res mode (Canon hard-codes resolution parameters for a full-res raw buffer). You may have to comment out the EngDrvOut calls for overriding the resolution, and maybe also the EDMAC setup for RD1.

OK, what's up with those funny names like RD1, WR1 etc?

From the name: TwoInTwoOut means this routine accepts two inputs (RD1, RD2) and two outputs (WR1, WR2). All of them are optional (see TTL_Start and TTL_Stop - these channels are only configured if the corresponding address or memsuite from TTL_Args is nonzero).

For lossless compression, we are interested in RD1 and WR1. What the others do, I don't know.

The default configuration (CR2 with quality set to RAW) uses RD1, WR1 and WR2. On 5D3, they use EDMAC channels 8, 22 and 2 (look at this picture).

RD1 reads from a contiguous buffer (that's the 14-bit raw buffer in the default configuration). In the picture, you can see it appears to scan the image twice, and its size looks a bit unusual. Here's how to decipher it and why we are overriding it.

WR1 uses a memSuite (unlike the other channels) - this is a data structure in Canon firmware that can contain one or more contiguous memChunk's (in other words, the WR1 channel can write to fragmented memory, possibly allocated with *_malloc_suite routines from exmem.c). The output size is not known in advance - it is set by TTL_Finish (actually JpCoreCompleteCBR). If the size is larger, it will not overflow the memSuite, but apparently there are weird alignment constraints on its size.

WR2 outputs something of size 1776*2 x 394 bytes, but didn't try to decode it (probably some sort of preview). Unused in our routine.

Here's a debug log from 5D3, showing the CR2 compression (among many others). Some relevant lines:
Code: [Select]
ShootSsDev:ff162e10:a7:03: sssExecuteMem1ToRawPath(79319)
ShootSsDev:ff163230:a7:03: ProcessTwoInTwoOutLosslessPath(R) Start(79319)
ShootSsDev:ff3d3f34:16:03: [TTL][167,79319,0] RAW(5920,3950,0,14)
ShootSsDev:000afe98:00:00: *** StartEDmac(0x16, 0x0), from ff576498
ShootSsDev:000afe98:00:00: *** StartEDmac(0x2, 0x0), from ff3d3af0
ShootSsDev:000afe98:00:00: *** StartEDmac(0x8, 0x2), from ff3d3b50
**INT-A3h*:ff3d3d14:16:03: [TTL][167,79319,0] JpCoreCompleteCBR( 0x146a56b )

To be continued.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #131 on: April 13, 2017, 07:39:44 AM »
More coming?

Good to verify some of the 700D addresses. Still can't save lossless full res DNG's but the warning messages and crashes are gone.

Looking through RAM4 and doing a lot of head scratching.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12317
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #132 on: April 13, 2017, 08:10:48 AM »
Some logs would be helpful (set verbose=2, or merge dm-spy-experiments and call debug_intercept before/after the compression routine).

Also, some dm-spy logs covering a still image capture (with quality set to RAW).

goldenchild9to5

  • Hero Member
  • *****
  • Posts: 534
Re: ProcessTwoInTwoOutLosslessPath
« Reply #133 on: April 13, 2017, 02:53:18 PM »
@a1ex WOW.. very promising.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12317
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #134 on: April 14, 2017, 12:51:42 PM »
More coming?

Some logs would be helpful (set verbose=2, or merge dm-spy-experiments and call debug_intercept before/after the compression routine).

Episode #2 is up - on the dm-spy-experiments thread.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #135 on: April 16, 2017, 08:32:41 AM »
Also, some dm-spy logs covering a still image capture (with quality set to RAW).

I'm having problems saving logs when attempting to shoot lossless compressed full resolution silent pictures on the 700D. However, I did record a dm-spy log shooting a single CR2 and it looks like there are some hints at how lossless compression works on this camera:

Full log file:
https://www.dropbox.com/sh/d7jyhbfk86vbxcu/AACYm04k7cTZvy_j3HzuFVaoa?dl=0

Some interesting snippets:
Code: [Select]
EEF51> ShootSsDev:ff168580:a7:03: sssExecuteMem1ToJpegPath(11296)
EEF96> ShootSsDev:ff4242a0:16:03: [TTJ][0,0,0] JPEG(5184,3456,0,0)
...
EF442> ShootSsDev:ff42463c:16:03: [TTJ] START WR1:0xa000064 WR2:0xd3e5f8
EF473> ShootSsDev:ff424668:16:03: [TTJ] START RD1:0x608e2a0 RD2:0x1082704
...
15488> ShootSsDev:ff42463c:16:03: [TTJ] START WR1:0xa06c07c WR2:0x23f8070
154BA> ShootSsDev:ff424668:16:03: [TTJ] START RD1:0x6004894 RD2:0x7f21f9c
...
15699> ShootSsDev:ff168ac8:a7:03: StartTwoInTwoOutJpegPath Raw(R) End(11296)
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #136 on: April 22, 2017, 04:15:05 AM »
@lostfeliz

You'll probably get a better response if you post that question on the crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView topic. (Just don't double post  :) )

Lots of work went into getting this working in MLV Lite though it looks like much of it can be applied to regular mlv_rec which is required for mlv_snd.

[EDIT: Original question moved to: http://www.magiclantern.fm/forum/index.php?topic=19300.msg183504#msg183504 ]
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12317
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #137 on: April 23, 2017, 05:59:46 PM »
I did record a dm-spy log shooting a single CR2

Can you take the same log, with interrupts and engio messages enabled? They are both disabled by default (first in dm-spy-extra.c, second in dm-spy.c).

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #138 on: April 23, 2017, 08:01:26 PM »
Found where to enable interrupts logging:

dm-spy-extra.c
Code: [Select]
/* this needs pre_isr_hook/post_isr_hook stubs */
#define LOG_INTERRUPTS

but don't understand how to enable engio messages. Seems to me that it is triggered if a certain condition is met:

dm-spy.c
Code: [Select]
static void my_DebugMsg(int class, int level, char* fmt, ...)
{
    if (!buf) return;
       
    if (class == 21) // engio, lots of messages
        return;
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

nikfreak

  • Developer
  • Hero Member
  • *****
  • Posts: 1139
Re: ProcessTwoInTwoOutLosslessPath
« Reply #139 on: April 23, 2017, 08:20:01 PM »
http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168

follow the guide "Want more details" and remove

Code: [Select]
if (class == 21) // engio, lots of messages
        return;
70D.112 & 100D.101

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #140 on: April 24, 2017, 12:37:19 AM »
Thanks @nikfreak -- here you go @alex

https://www.dropbox.com/sh/d7jyhbfk86vbxcu/AACYm04k7cTZvy_j3HzuFVaoa?dl=0

700D_shoot_CR2_2017-04-23

Lots of information logged for a single shutter click.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12317
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #141 on: April 24, 2017, 12:49:53 AM »
Looks like the 700D has the same problem as 5D3 - engio_write and others are silent. Can you include them as well? (they are in the same guide)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #142 on: April 24, 2017, 02:19:40 AM »
Ahh--I see the error in my thinking. I thought that /* on 5D3, Canon's engio_write is silent */ only applied to the 5D3. Should have just run the patch instead typing in the changes.

Replaced the log file from earlier today with an even longer log file that has a bunch of [ENGIO] in it.

Just to make sure we're on the same page, this is a log for just a single CR2 shot in order to see where the lossless compression is taking place. I still can't save a lossless DNG on the 700D with the settings I made on this commit. Never mind the EOSM settings, I'll get back to that once the 700D is working.

I'll be away from my development setup for a few days--hope I don't get ML separation anxiety!
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12317
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #143 on: April 24, 2017, 10:28:06 AM »
At the time of writing, it only applied to 5D3, from the models I've looked at.

Our logging routines are not printing [ENGIO], so the reason for larger log might have been a few more seconds allocated for the test routine. The new log doesn't appear to have any new information though.

EngDrvOut and engio_write can be taken from stubs; the others are easy to identify from strings (as they only log a debug message in case of error). Most DIGIC 4 models also log the register and the data written to it, when the call is successful.

festr

  • New to the forum
  • *
  • Posts: 31
Re: ProcessTwoInTwoOutLosslessPath
« Reply #144 on: April 24, 2017, 02:53:37 PM »
Hello, is there already tool which can decompress and fix vertical stripes?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12317
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #145 on: April 24, 2017, 03:23:10 PM »
There is a closed-source tool that can decode them, to my knowledge.

The decompressor is based on Baldand's LJ92 codec (from MlRawViewer), but it needs some changes. Some of these changes were shared here, so I assume one should be able to integrate them in other converters.

So, all that's required is some manpower to do the integration. No reverse engineering is required.

Greg

  • Contributor
  • Hero Member
  • *****
  • Posts: 591
Re: ProcessTwoInTwoOutLosslessPath
« Reply #146 on: April 24, 2017, 06:14:36 PM »
500D :
Code: [Select]
634AA> FrontShtDe:ff085618:96:05: sdfExecuteMem1ToRawPath(84538)
6352E> FrontShtDe:ff0859a8:96:05: sdfExecuteMem1ToRawPath(84538)©(SemOK)
6356D> FrontShtDe:ff0859e0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(84538)
645F9> FrontShtDe:ff252004:16:03: [TTJ][150,84538,0] RAW(4832,3204,0,14)
64634> FrontShtDe:ff191590:15:02: [JPCORE] SetEncodeLosslessType 1
64669> FrontShtDe:ff1929d0:15:02: [JPCORE] SetEncodeLosslessParam 1208,3204              <- note: 4 slices
84993> **INT-64h*:ff251de0:16:03: [TTJ][150,84538,0] JpCoreCompleteCBR( 0xd01f99 )
8647D> FrontShtDe:ff085a10:96:05: ProcessTwoInTwoOutJpegPath(R) End(84538)

ISO 100
dark frame - 50%
tree leaves - 70-75%

UnLockEngineResources :
Code: [Select]
851E4> FrontShtDe:00092990:00:00: *** UnLockEngineResources(869d98) x31 from ff2519bc:
8521F> FrontShtDe:00092a08:00:00:      1)    10002 (read channel 0xa)
85244> FrontShtDe:00092a08:00:00:      2)        3 (write channel 0x3)
85264> FrontShtDe:00092a08:00:00:      3)        4 (write channel 0x4)
85284> FrontShtDe:00092a08:00:00:      4)    30000 (read connection 0x0)
852A8> FrontShtDe:00092a08:00:00:      5)    20005 (write connection 0x5)
852CC> FrontShtDe:00092a08:00:00:      6)    20016 (write connection 0x16)
852EA> FrontShtDe:00092a08:00:00:      7)    50003 (?)
85307> FrontShtDe:00092a08:00:00:      8)    5000d (?)
85324> FrontShtDe:00092a08:00:00:      9)    5000f (?)
85341> FrontShtDe:00092a08:00:00:     10)    5001a (?)
8535E> FrontShtDe:00092a08:00:00:     11)    80000 (?)
8537B> FrontShtDe:00092a08:00:00:     12)    90000 (?)
85399> FrontShtDe:00092a08:00:00:     13)    a0000 (?)
853B6> FrontShtDe:00092a08:00:00:     14)   160000 (?)
853D5> FrontShtDe:00092a08:00:00:     15)   130003 (?)
853F3> FrontShtDe:00092a08:00:00:     16)   130004 (?)
85412> FrontShtDe:00092a08:00:00:     17)   130005 (?)
85430> FrontShtDe:00092a08:00:00:     18)   130006 (?)
8544F> FrontShtDe:00092a08:00:00:     19)   130007 (?)
8546D> FrontShtDe:00092a08:00:00:     20)   130008 (?)
8548C> FrontShtDe:00092a08:00:00:     21)   13000c (?)
854AB> FrontShtDe:00092a08:00:00:     22)   130010 (?)
854C9> FrontShtDe:00092a08:00:00:     23)   130011 (?)
854E8> FrontShtDe:00092a08:00:00:     24)   130012 (?)
85507> FrontShtDe:00092a08:00:00:     25)   130013 (?)
85526> FrontShtDe:00092a08:00:00:     26)   130014 (?)
85544> FrontShtDe:00092a08:00:00:     27)   130015 (?)
85563> FrontShtDe:00092a08:00:00:     28)   130018 (?)
85582> FrontShtDe:00092a08:00:00:     29)   13001a (?)
855A0> FrontShtDe:00092a08:00:00:     30)   13001b (?)
855BF> FrontShtDe:00092a08:00:00:     31)   13001c (?)

Resolution :
Code: [Select]
6583E> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f08548, Data:0x c87064e)[29]
6589B> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f08554, Data:0x c8812f0)[31]
660BD> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f13068, Data:0x c8312df)[78]
661F2> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12010, Data:0x c5f064f)[85]
66224> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12014, Data:0x c5f063f)[86]
66281> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f1201c, Data:0x 18b00c9)[88]
662B2> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12020, Data:0x 18b00c7)[89]

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: ProcessTwoInTwoOutLosslessPath
« Reply #147 on: April 25, 2017, 01:38:19 AM »
There is a closed-source tool that can decode them, to my knowledge.

The decompressor is based on Baldand's LJ92 codec (from MlRawViewer), but it needs some changes. Some of these changes were shared here, so I assume one should be able to integrate them in other converters.

So, all that's required is some manpower to do the integration. No reverse engineering is required.

can you please try this patch?
https://pastebin.com/Ggm0nkw4
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

nikfreak

  • Developer
  • Hero Member
  • *****
  • Posts: 1139
Re: ProcessTwoInTwoOutLosslessPath
« Reply #148 on: April 25, 2017, 06:06:47 PM »
EOS 7D.203
Attached the full dm.log while taking CR2 in photo mode:
https://drive.google.com/file/d/0B9Mu66yg5QzRV3RzZ0I2Q3dMZWM/view?usp=sharing

Log snippet:
Code: [Select]
9AA01> FrontShtDe:ff09c558:96:05: sdfExecuteMem1ToRawPath(673)
9AA78> FrontShtDe:ff09c8f4:96:05: sdfExecuteMem1ToRawPath(673)i(SemOK)
9AAAE> FrontShtDe:ff09c92c:96:05: ProcessTwoInTwoOutJpegPath(R) Start(673)
9AAF2> FrontShtDe:00ca6e80:00:00: *** LockEngineResources(89dd38) x17 from ff2de6e4:
9AB33> FrontShtDe:00ca6dd8:00:00:      1)    10002 (read channel 0xa)
9AB5A> FrontShtDe:00ca6dd8:00:00:      2)        3 (write channel 0x3)
9AB7A> FrontShtDe:00ca6dd8:00:00:      3)        4 (write channel 0x4)
9AB9B> FrontShtDe:00ca6dd8:00:00:      4)    30000 (read connection 0x0)
9ABBF> FrontShtDe:00ca6dd8:00:00:      5)    20005 (write connection 0x5)
9ABE4> FrontShtDe:00ca6dd8:00:00:      6)    20016 (write connection 0x16)
9AC04> FrontShtDe:00ca6dd8:00:00:      7)    50003 (?)
9AC23> FrontShtDe:00ca6dd8:00:00:      8)    5000d (?)
9AC42> FrontShtDe:00ca6dd8:00:00:      9)    5000f (?)
9AC60> FrontShtDe:00ca6dd8:00:00:     10)    5001a (?)
9AC7F> FrontShtDe:00ca6dd8:00:00:     11)    80000 (?)
9AC9E> FrontShtDe:00ca6dd8:00:00:     12)    90000 (?)
9ACBD> FrontShtDe:00ca6dd8:00:00:     13)    a0000 (?)
9ACDC> FrontShtDe:00ca6dd8:00:00:     14)   160000 (?)
9ACFC> FrontShtDe:00ca6dd8:00:00:     15)   130003 (?)
9AD1C> FrontShtDe:00ca6dd8:00:00:     16)   130004 (?)
9AD3D> FrontShtDe:00ca6dd8:00:00:     17)   130005 (?)
9ADBE> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0400008, Data:0x 36b110f) EngDrvOut
9AE07> FrontShtDe:ff1f8cf4:00:01: [CLKSAVER] ZClockSave OutZ
9AE28> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0400008, Data:0x 36b110f) EngDrvOut
9AE5C> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0400048, Data:0x       0) EngDrvOut
9AE86> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f01000, Data:0x    8000) EngDrvOut
9AEB4> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f01004, Data:0x       0) EngDrvOut
9AEE8> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0400008, Data:0x 36b110f) EngDrvOut
....
....
....
C20A7> **INT-5Bh*:ff1f4de8:15:01: WriteDMACInterrupt(3)(0x4)(2)
C20CD> **INT-5Bh*:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04304, Data:0x       2) EngDrvOut
C20FB> **INT-5Bh*:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04300, Data:0x       4) EngDrvOut
C2127> **INT-5Bh*:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04304, Data:0x       0) EngDrvOut
C2156> **INT-5Bh*:ff3d2424:15:02: [PackMem] PopInterrupt
C217F> **INT-5Bh*:ff2ddf74:16:03: [TTJ][150,673,0] Write1Complete(POP)CBR
C21AA> **INT-5Bh*:00000588:00:00: <<< INT-5Bh done
C21C1> **INT-5Bh*:ff3d268c:15:02: [PackMem] PopEDmacForMemorySuite 3
C21E2> **INT-5Bh*:00000588:00:00: <<< INT-64h done
C222E> FrontShtDe:ff2ddc80:16:03: [TTJ] STOP WR1:0x3095978 WR2:0x12420030
C2270> FrontShtDe:ff2ddcac:16:03: [TTJ] STOP RD1:0x5f7cf64 RD2:0x1ca25c84
C22AA> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f0300c, Data:0x       0)[0]
C22D9> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f03010, Data:0x       0)[1]
C2304> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f03014, Data:0x       0)[2]
C232E> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f03090, Data:0x       0)[3]
C2358> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f08070, Data:0x80000000)[4]
C238B> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f08540, Data:0x80000000)[5]
C23BE> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f0d000, Data:0x80000000)[6]
C23F1> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f13060, Data:0x80000000)[7]
C2424> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f12000, Data:0x80000000)[8]
C2464> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04a04, Data:0x       0) EngDrvOut
C249C> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04304, Data:0x       0) EngDrvOut
C24D1> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04404, Data:0x      80) EngDrvOut
C251C> FrontShtDe:00ca6e80:00:00: *** UnLockEngineResources(89dd38) x17 from ff2ddc08:
C255C> FrontShtDe:00ca6dd8:00:00:      1)    10002 (read channel 0xa)
C2582> FrontShtDe:00ca6dd8:00:00:      2)        3 (write channel 0x3)
C25A2> FrontShtDe:00ca6dd8:00:00:      3)        4 (write channel 0x4)
C25C3> FrontShtDe:00ca6dd8:00:00:      4)    30000 (read connection 0x0)
C25E7> FrontShtDe:00ca6dd8:00:00:      5)    20005 (write connection 0x5)
C260D> FrontShtDe:00ca6dd8:00:00:      6)    20016 (write connection 0x16)
C262D> FrontShtDe:00ca6dd8:00:00:      7)    50003 (?)
C264B> FrontShtDe:00ca6dd8:00:00:      8)    5000d (?)
C266A> FrontShtDe:00ca6dd8:00:00:      9)    5000f (?)
C2689> FrontShtDe:00ca6dd8:00:00:     10)    5001a (?)
C26A7> FrontShtDe:00ca6dd8:00:00:     11)    80000 (?)
C26C6> FrontShtDe:00ca6dd8:00:00:     12)    90000 (?)
C26E5> FrontShtDe:00ca6dd8:00:00:     13)    a0000 (?)
C2704> FrontShtDe:00ca6dd8:00:00:     14)   160000 (?)
C2724> FrontShtDe:00ca6dd8:00:00:     15)   130003 (?)
C2745> FrontShtDe:00ca6dd8:00:00:     16)   130004 (?)
C2765> FrontShtDe:00ca6dd8:00:00:     17)   130005 (?)




70D.112 & 100D.101

festr

  • New to the forum
  • *
  • Posts: 31
Re: ProcessTwoInTwoOutLosslessPath
« Reply #149 on: April 26, 2017, 01:03:38 PM »
Hello again, I have tried the mlv dump for mac - it decodes compressed mlv into dng files which is very nice! But there is one major problem - it does not remove vertical strips. Is this known issue? Is there any way how to remove them currently?

edit: oh I realised that the mlv_dump is not able to handle compressed image