ProcessTwoInTwoOutLosslessPath

[reddeercity]

@reddeercity -- Getting it to compile is rather easy, getting it to work is much harder. 

yes that for sure

@reddeercity --  Did you put in the 5D2 specific changes in lossless.c? One thing that is still missing for the 5D2 is the "resources[]" which should be in your dm logs but they aren't.

Yes  I added the lossless.c changes , but messed up on the resources , more then likely that's the lockup for me I think .
To get it to compile with the crop_rec_4k branch had to comment out this in raw.c

Code Select Expand

//if (gain)
    //{
        //lv_raw_gain = gain;
       // raw_info.white_level = get_default_white_level();
        //raw_info.black_level = BLACK_LEVEL;
    //}
        //else
and in lv-img-engio.c

Code Select Expand


//total_movie_gain *= _raw_lv_get_iso_post_gain();

I do see other thing I missed , Thanks -- but I didn't have to comment out anything in the "compressed_raw" branch .
it seemed that all the problems with compiling was with the "digital gain" stuff . Mainly I was interested if it even would compile at all .
I'll check some other LOG's I have and see if I can resolve/found the "resources[]" , if not I'll make some new  Log's

I'm looking for right now is a silent module that can save a simple lossless compressed DNG

Agreed , I've been focusing on trying to get a Lossless silent dng
Yea it feels close 

[dfort]

Studied the code some more. There is this section that only applies to the 5D3 that I can't figure out:

modules/silent/lossless.c

Code Select Expand

    if (is_camera("5D3", "*"))
    {
        /* resolution is hardcoded in some places; patch them */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F13068, PACK32(width*2  - 1,  height/2  - 1));  /* 0xF6D171F */
        EngDrvOut(0xC0F12010,        width    - 1                 );  /* 0xB8F     */
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F1201C,        width/10 - 1                 );  /* 0x127     */
        EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  /* 0x18A0127 */
    }


It has been in compressed_raw branch from the beginning. None of the other cameras need it but maybe something similar is necessary for the Digic 4 cameras?

Searched the forum and we've come across it before on the 6D. In any case, this isn't causing the 7D lockup issues when shooting a lossless DNG so I did this to try and track it down:

modules/silent/silent.c

Code Select Expand

    /* fixme: not all models are able to allocate such a large contiguous chunk */
    int max_compressed_size = ((uint64_t) raw_info->frame_size * 80 / 100) & ~0xFFF;
   
    bmp_printf( FONT_MED, 0, 83, "max_compressed_size = %d", max_compressed_size);


This is what it printed in mv1080 mode:



and in mv720 mode:



The reason I was interested in mv720 mode was so I could compare it with the EOSM which is the other camera I've got with me at the moment.



I repeated this at several different settings and it didn't change so it is probably a fixed value. Why are they different on the two cameras? IDK.

Whatever the issue, it is probably somewhere in save_lossless_dng. Redirecting to save_dng is working fine.

@reddeercity, still need to find resources[]  for the 5D2. Here's what I did on the 7D--not sure if this is correct but it seems to make sense according to the dm log file.

modules/silent/lossless.c

Code Select Expand

        else if (is_camera("7D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10002 | edmac_channel_to_index(edmac_read_chan),
            0x30000,    /* read  connection  0 */
            0x20005,    /* write connection  5 */
            0x20016,    /* write connection 22 */
            0x50003,
            0x5000d,
            0x5000f,
            0x5001a,
            0x80000,
            0x90000,
            0xa0000,
            0x160000,
            0x130003,
            0x130004,
            0x130005,
        };
/*       
1)    10002 (read channel 0xa)
2)        3 (write channel 0x3)
3)        4 (write channel 0x4)
4)    30000 (read connection 0x0)
5)    20005 (write connection 0x5)
6)    20016 (write connection 0x16)
7)    50003 (?)
8)    5000d (?)
9)    5000f (?)
10)    5001a (?)
11)    80000 (?)
12)    90000 (?)
13)    a0000 (?)
14)   160000 (?)
15)   130003 (?)
16)   130004 (?)
17)   130005 (?)
*/


I added the 5D2 code in my branch in case anyone wants to check out what max_compressed_size is on that camera.

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/18/loss-compression-for-digic-4-cameras/diff

[a1ex]

The resolution section covers registers that seem to be related to width and height (possibly by some scaling factor). They are pretty much guesswork (tweaked them until they happened to work), but there are hints that not all of them might be needed. However, the exact meaning of these registers is still unknown (possibly configuring resolution for various image processing modules in the compression path).

Before, I've got these register values from QEMU, but the procedure is not very straightforward (it can emulate only a part of the lossless compression process). Now you can get the full conversation with the hardware, by using io_trace. Some of the registers (but maybe not all) are also present in the regular logs.

Look up the above registers in the photo capture log - you'll find the values from the comments.

Capture similar logs for other camera models (possibly with different sensor resolutions) to identify which registers might have to be changed.

I have some (incomplete) logs from QEMU with register diffs, but with the current tools (which evolved a lot during last year), I think it's best to capture them from real hardware using the latest io_trace_full branch. If it crashes, try reducing the logged range (to capture only the registers we are actually interested in).

Recommended models for comparison:
- 5D3 vs 6D vs 70D (all of them using ProcessTwoInTwoOutLosslessPath)
- all other D5 models (using ProcessTwoInTwoOutJpegPath)
- all other D4 models (caveat: io_trace is more likely to crash)

To get these logs in QEMU, the basic procedure is to call the compression routine with fake data. It won't get emulated completely anyway, so you can just plug some random address (0x12345678 will do), and give valid width and height to the raw_info structure passed to lossless_compress_raw. You don't have to capture any image. The test code can be executed from silent.c's init function, for example:

Code Select Expand


static unsigned int silent_init()
{
    /* keep the old stuff */

    /* code for running only in QEMU - no error checking done, no valid input, no valid results expected */
    struct memSuite * out_suite = shoot_malloc_suite_contig(16 * 1024 * 1024);
    struct raw_info out_raw_info = {
       .buffer = 0x12345678,
       .width = 1920,
       .height = 1080,
    };
    lossless_compress_raw(&out_raw_info, out_suite);

    return 0;
}


Then, run with -d io:

Code Select Expand


# from platform/CAM.FW
make install_qemu ML_MODULES=silent

# from qemu-eos
./run_canon_fw.sh CAM,firmware="boot=1" -d debugmsg,io [...]


The emulation will fail at the semaphore, but you will see the registers set by the lossless compression routine (and you can compare them with the ones obtained from io_trace). If they are identical, that's a very good sign.

TODO: automate this task, run it in QEMU for all single-DIGIC models that have lossless routines identified, then diff all these logs. Difficulty: easy, just time-consuming (edit: done for DIGIC 5, took only 2 hours; I've expected it would take a lot more).

lossless-1080-qemu-5D3.log
lossless-1080-qemu-6D.log
lossless-1080-qemu-70D.log
lossless-1080-qemu-700D.log
lossless-1080-qemu-650D.log
lossless-1080-qemu-EOSM.log
lossless-1080-qemu-100D.log
lossless-1080-qemu-EOSM2.log
lossless-1080-qemu-5D3-vs-6D.html
lossless-1080-qemu-5D3-vs-70D.html
lossless-1080-qemu-6D-vs-70D.html
lossless-1080-qemu-5D3-vs-700D.html
lossless-1080-qemu-700D-vs-650D.html
lossless-1080-qemu-700D-vs-EOSM.html
lossless-1080-qemu-700D-vs-100D.html
lossless-1080-qemu-100D-vs-EOSM2.html

In particular, the differences between 700D (working) and 650D (having artifacts) are quite small. No idea what they are, but... try overriding the highlighted registers and see if it helps.

The differences between 700D and EOSM (both working) are not relevant (one is some clock enabled for some unknown device, the other is from a malloc'ed address). The differences between 700D and 100D look a little more interesting, but they are probably related to how powersaving is implemented in hardware (TLDR: not relevant). The 100D and EOSM2 are also nearly identical (as expected).

For 70D, registers 0xC0F37300 and 0xC0F373E8 seem to be resolution-related, but are not used on 5D3/6D; @esas - please try. The subsequent registers smell like some possible Bayer configuration...

Emulating the full photo capture can be also useful, as it will use the unaltered Canon configuration, but it's a lot harder (it's much easier to get this one with io_trace).

One bit of mystery: on 5D3, capturing a still photo from movie mode's LiveView breaks the lossless compressor from mlv_lite. Investigating this one could give even more clues for figuring out what happens on the other models.

Why are they different on the two cameras? IDK.

You have pasted the answer a few lines above that

[reddeercity]

@dfort , I'll try the resource you posted , they seem right for 5d2 from what I see in my new dm-log .

Code Select Expand

**INT-5Ch*
not sure if this mean it's using channel #5 , I'll compile and see what happens

[reddeercity]

Some success  I can load the silent.mo and no lockups on D4 5D2



It gets better , I can take a photo with DNG Lossless compression



But for some reason it doesn't save a DNG as the screen shot shows , just locks up the camera
everything is unresponsive , had a solid blue led on the Liveview button and a frozen Liveview image
needed to pull the battery . After a reboot , every thing is working again -- If I take a full res lossless photo
locks up again so need to find out why it's locking up after a half shutter press . On a side note
the mlv_lite module didn't compile , but I never look at it so until I can save a lossless image that is last on the list .

Edit: I didn't add the raw.c changes that @dfort had in the https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/18/loss-compression-for-digic-4-cameras/diff
I just left the changes I made in post https://www.magiclantern.fm/forum/index.php?topic=18443.msg197970#msg197970 for the digital gain stuff .
But everything else I add to my source.
 

[a1ex]

For solving the 650D and 70D lossless compression puzzle:

[...] what lossless compression rates are you getting on 700D, EOSM and 100D? Do they depend on horizontal resolution?

I've tried to trick it on 5D3, by commenting out the resolution patching block, but could not notice any significant change in compression rate.

However, I did some more experiments. I took the following changes from the 5D3 vs 6D log:

Code Select Expand


        EngDrvOut(0xC0F37634, 0x72000);
        EngDrvOut(0xC0F3763C, 0x1000000);
        EngDrvOut(0xC0F37640, 0x2000000);
        EngDrvOut(0xC0F37644, 0x4000000);
        EngDrvOut(0xC0F37648, 0x8000000);


and got the same artifacts as on 6D - both this (only partly fixable by overriding 0xC0F375B4) and the per-channel black level variations!

Also got these changes from the 700D vs 650D log, and tried them on 5D3:

Code Select Expand


        EngDrvOut(0xC0F37610, 0x11);
        EngDrvOut(0xC0F37628, 0x72000);
        EngDrvOut(0xC0F3762c, 0x72000);
        EngDrvOut(0xC0F37630, 0x72000);
        EngDrvOut(0xC0F37634, 0x72000);
        EngDrvOut(0xC0F3763C, 0x1000000);
        EngDrvOut(0xC0F37640, 0x2000000);
        EngDrvOut(0xC0F37644, 0x4000000);
        EngDrvOut(0xC0F37648, 0x8000000);


and the artifacts were a lot like on 650D (but also had the per-channel black levels).

=> we've got stuff to try on 6D, 650D and 70D

It's probably going to be a bit of trial and error, but I'm pretty sure this is the right track.

[a1ex]

Possible patch that might handle all D5 models (to be tested; can be applied with "hg import"):

d5-check.patch
lossless-res-d5.patch

To avoid conflicts, the easiest way is to undo any (half-working) changes to the resolution patching block from the pull requests (for example, on 6D there is one such change). The other changes should be taken from the PRs.

[dfort]

Ha ha--I was posting this:

So if you are able to reproduce the 650D artifacts on the 5D by patching the differences between those log files would it be reasonable to try and patch the 650D  to the 700D?

modules/lossless.c

Code Select Expand

    if (is_camera("650D", "*"))
    {
        /* patch 650D to 700D values */
        EngDrvOut(0xC0F37610, 0x0);
        EngDrvOut(0xC0F37628, 0x71000);
        EngDrvOut(0xC0F3762c, 0x71000);
        EngDrvOut(0xC0F37630, 0x71000);
        EngDrvOut(0xC0F37634, 0x71000);
        EngDrvOut(0xC0F3763C, 0x0);
        EngDrvOut(0xC0F37640, 0x0);
        EngDrvOut(0xC0F37644, 0x0);
        EngDrvOut(0xC0F37648, 0x0);
    }


Of course you took this to a whole new level. Thanks for the patches. I'll test on the 700D and EOSM. Need volunteers for the 6D and 650D -- @Levas and @saulbass ?

[saulbass]

happy to help

[dfort]

Tried it on the EOSM before posting test builds--found a problem with simple silent lossless compressed DNG's:



No DNG or log saved. However, Full-res silent lossless compressed DNG's are working fine.

[dfort]

Similar issues on the 700D. In addition, it is not saving FRSP lossless DNG's.

[dfort]

mlv_lite lossless compression seems to be working fine with all bit depths on both EOSM and 700D. I believe most people have been testing video recording so I'll put up the test builds for 650D and 6D. It will be interesting to see if we can get some good lossless MLV's out of these cameras. As usual, look for the test builds on my downloads page.

[dfort]

Also wondering: what lossless compression rates are you getting on 700D, EOSM and 100D? Do they depend on horizontal resolution? (a bit off-topic, but pretty sure dfort is reading). I'd expect bad compression rates if the declared width does not match the actual image width (but that won't affect decoding).

This is probably the right topic. What test should we run to answer this question?

[Levas]

and got the same artifacts as on 6D - both this (only partly fixable by overriding 0xC0F375B4) and the per-channel black level variations!

Wow, new insights 

Tried the test build, for 6d, made by Dfort, but unfortunately it doesn't work for lossless recording and silent lossless...
I can't get no simple and full-ress lossless silent pics.
Lossless recording with mlv_lite gives data corruption error at slot 24 
error log:

Code Select Expand


ML ASSERT:
0
at mlv_lite.c:2497 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 1df738 [1df7c8-1de7c8]
0x0044C93C @ c056bc:1df768
0x0044C478 @ 44c998:1df738

Magic Lantern version : Nightly.2018Mar03.6D116
Mercurial changeset   : e8dba6222cd4 (crop_rec_4k_Digic5_patches) tip
Built on 2018-03-03 17:07:06 UTC by rosiefort@RosieFoComputer.
Free Memory  : 379K + 2320K


[dfort]

Sorry, my mistake. I applied the patches to a clean crop_rec_4k branch to prevent conflicts. That should be fine for the 650D but not for the 6D because the pull request hasn't been merged yet. That PR is getting a bit messy so I'll make a new one once we figure this out. Posting a new build and going offline for a while. If you want to keep tweaking the code it is in my crop_rec_4k_Digic5_patches branch.

[Levas]

Thanks, will try it out and download your source 

[Levas]

The patch WORKS

Did a quick test with latest build from Dfort.
I have recorded a lossless 14 bit MLV file in 1824x1016 1026 resolution and the resulting DNG's are normal!

Thanks Alex 

[a1ex]

Previously, the 700D and EOSM worked without having to tweak any registers. That means, one or more patches are probably interfering somehow with the encoder setup.

Checked the 700D in QEMU, from changeset e8dba6222cd4 - can't see anything wrong with it: lossless-1080-qemu-700D-before-vs-after.html.

Which one(s) are causing the issue? No idea, but can be found by trial and error.

Got another hypothesis - the blocks with width/10, width/12 etc might be for the second path from the TTL, which we don't use - likely used for http://lclevy.free.fr/cr2/ section 2.6. If true, that would remove a large amount of code from the patch (all the 0xC0F12* registers). The numbers seem to match very well on 5D3:

Code Select Expand


exiftool TEST_5D3.CR2 -IFD2:ImageWidth -IFD2:ImageHeight -IFD2:StripByteCounts -IFD2:StripOffsets
Image Width                     : 592
Image Height                    : 395
Strip Byte Counts               : 1403040
Strip Offsets                   : 981116


Code Select Expand


dd if=TEST_5D3.CR2 bs=1 count=1403040 skip=981116 of=uncompressed.bin
convert -size 592x395 -depth 16 RGB:uncompressed.bin -auto-level uncompressed.png


Code Select Expand


[0xC0F12020] <- 0x18A0127  -> 0x18B = 395, 0x128 * 2 = 592


WR2 outputs something of size 1776*2 x 394 bytes

Still unsure what's up with 0xC0F13068.

[a1ex]

Simplified patch (to be applied from scratch):

d5-check.patch (unchanged)
lossless-res-d5-v2.patch

Please try on 6D, 700D, 650D, 70D, 100D, EOSM and (if possible) EOSM2. If it works, I'll commit this one.

[reddeercity]

I'm trying to figure out how the silent Full res lossless DNG is saved or in my case not being save .
I step thought the lossless.c and from what I can tell it doing it thing . The thing I'm not sure about  is the

Code Select Expand

EngDrvOut in lossless.c362 or is the saving routines some where else like "raw.c"? I'll try a uncompressed raw full res and see it it save a dng and doesn't lockup .

[reddeercity]

I can save a uncompressed Full Res silent photo what out problems , so the saving routines and not a issue must be in lossless.c or silent.c
I did try another lossless FRSP after taking a uncompressed and it's trying to save , gives the resolution and max compression size plus say it's saving but locks up/frozen Liveview
needs a battery pull  .



funny it started to save a dng etc. ... only after I took a uncompressed frsp  very puzzling .

[dfort]

@reddeercity - That's pretty much the same on the 7D. Putting aside Digic 4 for now though this new Digic 5 code will most likely help get lossless compression working on these older cameras.

The latest changes are working on the EOSM and 700D. No problem saving lossless silent stills and mlv_lite lossless compression is working at all bit depths. Posted test builds on my downloads page for 6D, 700D, 650D, 70D, 100D, EOSM. Took the 70D code from nikfreak's repository. Also created a new pull request that supersedes the 6D PR for crop_rec_4k.

The EOSM2 is also saving lossless simple silent stills. Lossless with FRSP isn't working yet and mlv_lite displays "Compression error -3" the frames are corrupted and it saves assert logs.

Code Select Expand

ML ASSERT:
0
at mlv_lite.c:2499 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 216780 [216810-215810]
0x0044C938 @ adc3e8:2167b0
0x0044C478 @ 44c994:216780

Magic Lantern version : Nightly.2018Mar03.EOSM2103
Mercurial changeset   : 60bb31cb0539+4b3cf949679e+ (EOSM2.103_crop_rec_4k_wip) tip
Built on 2018-03-04 05:48:50 UTC by rosiefort@RosieFoComputer.
Free Memory  : 384K + 1982K

Of course there are lots of other stability issues with the EOSM2 but hey, the simple silent feature is working.

[Danne]

Simplified patch (to be applied from scratch):

d5-check.patch (unchanged)
lossless-res-d5-v2.patch

Please try on 6D, 700D, 650D, 70D, 100D, EOSM and (if possible) EOSM2. If it works, I'll commit this one.

Added above patch to crop_rec_4k branch(hg import) and changes were added according to patches. Tested on a 100D. All lossless workflows works which I tested:
mv1080p 8-11, 12, 14bit check
3xzoom 8-11, 12, 14bit check
crop_rec(mv720p) 8-11, 12, 14bit check
Silent dng
Fullres lossless silent dng check
lossless silent dng mv1080p check

Now how do one revert a patch when done testing?

Anyway. Great to see progress on 6D and also eosm2 and simply following the discussion is very exciting.

[Levas]

Checked out Dfort latest build for 6d (made 5 hours ago, so I'm guessing this one is with the simplified patch)

Lossless recording in 14, 12 and 11-8 bit depth works in 1080 mode, 720 mode and zoom mode 
Simple silent lossless also works.
But, full-res silent lossless doesn't work, it doesn't create a DNG file 
Normal full-res silent, without lossless compression, does work in this build.

The previous build, Dfort made for 6d, with the first patches from Alex, does have a working full-res silent lossless mode.
So with first patches, 6d has working lossless full-res silent pictures.
With simplified patches, 6d has non working lossless full-res silent pictures (simple lossless does work and normal-nonlossless full-res also works)

[a1ex]

First guess: does it have to do with 0xC0F13068? However, I have a feeling this one caused the issues on 700D and EOSM.

What happens if you move it out of the if (0) ?

If it doesn't help, you could try the remaining ones (although I believe they are only for RD2/WR2, which we don't use). The 6D and 70D use width/12 and height/24; the small cameras use width/8 and height/16 (see the initial patch).



I've tried to override it on Canon's CR2 compression routine (dm-spy-experiments, dm-spy-extra.c):

Code Select Expand


#ifdef CONFIG_5D3_113
    { 0xFF3D46F0, "TTL_Start", 1, ttl_start_log },
#endif

static void ttl_start_log(uint32_t* regs, uint32_t* stack, uint32_t pc)
{
    /* log the original call as usual */
    generic_log(regs, stack, pc);

    EngDrvOut(0xC0F13068, 0x0x1edb0b8f);  /* 0xF6D171F */
    EngDrvOut(0xC0F12010, 0xB8F);  /* 0xB8F     */
    EngDrvOut(0xC0F12014, 0xF6D0B8F);  /* 0xF6D0B8F */
    EngDrvOut(0xC0F1201C, 0x127);  /* 0x127     */
    EngDrvOut(0xC0F12020, 0x18A0127);  /* 0x18A0127 */

    DryosDebugMsg(0, 0, "*** TTL regs overridden");
}


If I change it to something that has the same area (in the above example: 5920x3950 -> 2960x7900), it still works (without any obvious changes in the output). If I change it by one or two pixels in either direction, the CR2 no longer gets saved. Same change in lossless.c, with RD2/WR2 enabled, would result in error -2 (failed at semaphore).

Fun stuff: the image processing block that uses 0xC0F13068 is called RABBIT. No idea what it actually does.