ProcessTwoInTwoOutLosslessPath

[a1ex]

I think we may have a encoder size/resolution  miss match on the 5d2/d4 ,

Encoder is always configured from full raw buffer size. Active area is just how these pixels are interpreted by ML or post-processing software; the lossless encoder does not know which pixels are active and which ones are not; it simply doesn't care about that. The lossless routine does not use the active area parameters in any way.

[reddeercity]

Ok , understood -- Is there any way to see or log what the encoder is doing ?
Is it that the expected data is not where the encoder needs it to be ?
other words the address mem1 is not right ? 

[ArcziPL]

lossless-1080-qemu-5D3-vs-6D.html
lossless-1080-qemu-5D3-vs-70D.html

Hi there. There's some progress on a lossless RAW compression on 70D. Not as far, as I'd wish but still.

The starting point was a recent revision of crop_rec_4k_mlv_snd, where the stubs and two image size registers were already set. However, the compressed image shown heavy deviations from a reference non-compressed one. It was similar to the initial distortions seen on 6D and 650D.

Uncompressed FRSP vs. compressed FRSP:
   


I've found two solutions leading to a working losslessly compressed FRSP ("Full-res" option in Silent Picture sub-menu) and losslessly compressed LV buffer ("Simple" option).


1. Overriding register 0xC0F373B4

Whetever it really does, a value of 0 in register 0xC0F373B4 results in a valid compressed DNG.

Code: [Select]

EngDrvOut(0xC0F373B4, 0);
Interesting fact: originally the camera sets this register to 0x11. A similar change of another register, also from a value 0x11 to 0 is needed for 6D and 650D.

Code: [Select]

    if (is_camera("6D", "*") ||
        is_camera("650D", "*"))
    {
        /* not sure what exactly these do, but they seem to be required to get correct image
         * taken from register log diffs: http://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987 */
        EngDrvOut(0xC0F37610, 0);       /* 0x11 on 650D, 0 on all other D5 (not sure if needed) */

2. Second, independent solution is an overriding of register 0xC0F373F4

This one is really interesting.

Original log from the 70D:

Code: [Select]

[0xC0F373F4] <- 0x7FFF7FFF: ???
[0xC0F373F8] <- 0x1111BBBB: ???
The first register contains two 16-bit words, which seem to define horizontal and vertical boundaries of a slice, which should get somehow processed.

By lowering the least significant word ([0xC0F373F4] <- 0x7FFF1111) we get such a picture:


By lowering also the most significant word ([0xC0F373F4] <- 0x01000A00) it looks like this:


What it really is used for, I've no idea.

Changing any (or both) of the words to 0x0000 will give us a nice, valid, losslessly compressed DNG. E.g:

Code: [Select]

EngDrvOut(0xC0F373F4, 0x7FFF0000);
I wasn't able to find the meaning of the second mentioned register. There was no visible difference when changing it. Maybe it refers to another processing mode of the compressor module or maybe to a different write channel?

Interesting fact: similar values can be seen other cameras; here 5D3:

Code: [Select]

177 [0xC0F375B8] <- 0x7FFF7FFF: ???
178 [0xC0F375BC] <- 0x7FFF7FFF: ???
179 [0xC0F375C0] <- 0x7FFF7FFF: ???
180 [0xC0F375C4] <- 0x7FFF7FFF: ???
181 [0xC0F375C8] <- 0x7FFF7FFF: ???
182 [0xC0F375CC] <- 0x7FFF7FFF: ???
183 [0xC0F375D0] <- 0x7FFF0000: ???
184 [0xC0F375D4] <- 0x7FFF0000: ???
185 [0xC0F375D8] <- 0x7FFF7FFF: ???
186 [0xC0F375DC] <- 0x7FFF7FFF: ???
187 [0xC0F375E0] <- 0x7FFF7FFF: ???

The miniature picture embedded in DNG is unfortunately all black in the compressed files (despite being fine on the uncompressed FRSP). But I don't see it as an issue now, as it's same on 700D.


Now the sad part. Despite silent pictures are working very well with these changes, the compressed MLV still doesn't. The symptoms are still like in the past:

#dfort  tried installing your latest crop rec4k for 70D. after installing  i turn On crop rec and mlv lite. restart my camera. no buttons would work after that. i tried reinstalling your build 2 times but no luck

with your latest build i get this FRSP https://files.fm/u/3kc5sbt if i activate mlv lite, recording button nad trast button dont response

Describing it a bit more detailed:

If mlv_lite is activated and configured to 14-bit uncompressed, the camera will normally record MLV.

If being already in the movie mode, opening ML menu, changing bit-depth to 14-bit lossless: the ML menu will freeze immediately just after highlighting this option (without clicking SET button). No reaction to menu navigation, ML menu can't be closed. However, the cam still works, it can be switched to photo mode and will work normally showing "Raw detect error" for a while.

When being in photo mode, you can select 14-bit lossless in RAW video menu. Apparently there is no initialization of buffers/compressor executed at that point. When switching the camera to movie mode the buttons themself are working properly. ML console shows even "REC key pressed" (when pressed) and "Starting raw recording..." but the recording doesn't start. You can enter ML menu but it will freeze as soon as you change the tab to Movie. To unfreeze the menu: change mode to photo.


I have a feeling it will unfortunately not be enough to play with some more registers in lossless.c to fix the RAW video. I've "simulated the 650D" behaviour with wrongly configured compressor by applying wrong values on 700D.

Result: silent picuteres had distorted colors but the lossless MLV were still working (with distorted colors as well).

Could it rather be related to some buffer allocation, conflicting resources or waiting for flags/semaphores? How can it be further debugged?

1. What is exactly executed as soon as the 14bit lossless option is only highlighted in the ML menu when being in the video mode?
2. What and how to debug? I have already compiled with #define RAW_DEBUG but I don't know how to store the content of the console to a file O_o
3. How does compression differ in photo mode ("simple" option should be the LV buffer?) from movie mode?
4. Can these be conflicticting/wrong EDMAC resources in video mode? I am reluctant to play around with it on my own, as I have no idea how it works but it's clear for me, that a wrong change might results in "random" writes into any memory area.
5. Any other ideas...?

[a1ex]

Very nice!

1. ML is attempting to compress the LiveView frame every 2 seconds, to find out the compression ratio. This happens in measure_compression_ratio -> forwarded to compress_task -> it compresses one frame as usual.

2. Hard to answer... I'd probably get some MMIO logs (io_trace_full branch), and... from there I'll probably end up staring at them.

3. In full-res mode, ML is operating outside LiveView (only the mirror is still up). In other words, all the image processing resources are available. In LiveView, some of them may be used. In movie mode, the configuration performed by Canon might be slightly different.

Need to read more carefully, but if the simple silent picture, saved as lossless DNG, works in movie mode, I see no reason why MLV compression wouldn't work. The former would be a lot easier to debug, too.

I'm starting to think we were really lucky on most DIGIC 5 models, where the resources used by lossless compression weren't in conflict with resources used by LiveView (well, there were a few conflicts that we knew how to solve). I had many attempts to get it working on DIGIC 4, but didn't document them, because... it just didn't work in LiveView.

4. EDMAC resources: will check. You could experiment with edmac.mo and see what it prints - screenshots, free channels, logs. These should help refreshing my memory. General idea: you can pick any unused channels (not active in the overview screen, and reported as "free" when you run the "Find free EDMAC channels" test). By default, the ones used for edmac_memcpy (i.e. uncompressed raw video) are reused, so I don't expect any trouble here.

Other ideas... not sure, maybe near the end of the week.

"The miniature picture embedded in DNG is unfortunately all black in the compressed files" -> expected; it's not implemented.

[ArcziPL]

Need to read more carefully, but if the simple silent picture, saved as lossless DNG, works in movie mode, I see no reason why MLV compression wouldn't work

Hi Alex, that's a great point! I tried silent pictures in photo mode only, as the hint in the menu was showing that full-res silent pics are not working in movie mode. But you're fully right, simple silent pictures can be also done in movie mode.

So, here comes more:

1. Only silent.mo loaded: simple compressed DNG in movie mode works
2. Only silent.mo + mlv_rec.mo loaded, RAW video set to off: simple compressed DNG in movie mode works
3. Only silent.mo + mlv_rec.mo loaded, RAW video set to on: simple compressed DNG in movie mode works, raw video recording is also working (uncompressed of course)
4. Only silent.mo + mlv_lite.mo loaded, RAW video set to off: simple compressed DNG in movie mode works
5. Only silent.mo + mlv_lite.mo loaded, RAW video set to on, bit depth to 14bit uncompressed: simple compressed DNG in movie mode doesn't work!

In case 5 it stucks with "Preparing..." on the screen. The camera is still responsive. Just the text stays there until mode is changed to photo.


For a start I made some screenshots of EDMAC channels.

Photo mode:



Video mode:


I'm a total noob here but they seem for me nearly the same.

BTW: no in-cam screenshots for the last case, as trying to take them in movie mode results in a memory error. Just checked on 700D and it's same.

[a1ex]

With mlv_lite's RAW Video enabled, the entire memory is preallocated during stanby. I'm seriously thinking to revert this change, just haven't managed to code anything in this direction. Been focusing on new ports + QEMU lately (as my main camera is defective anyway).

I forgot one important detail - in crop_rec_4k, the silent picture module will go out of LiveView to save images, even simple ones. That change was done for being able to show a preview, but... the image processing resources are also free in this "paused LV" mode; no wonder it worked

Try undoing out this change; will it still work? The goal is compressing one frame while LiveView is still running.

[ArcziPL]

With mlv_lite's RAW Video enabled, the entire memory is preallocated during stanby.

Sounds not that "lite" at all.

I forgot one important detail - in crop_rec_4k, the silent picture module will go out of LiveView to save images, even simple ones. That change was done for being able to show a preview, but... the image processing resources are also free in this "paused LV" mode; no wonder it worked

Try undoing out this change; will it still work? The goal is compressing one frame while LiveView is still running.

Changes in silent.c undone. Result:

silent.mo + mlv_rec loaded, RAW video on: simple silent pictures can be taken (both compressed and uncompressed)

silent.mo + mlv_lite loaded, RAW video on (14-bit selected): simple silent pictures can't be taken (both compressed and uncompressed) -> stuck at "Preparing..."

EDIT:
I've just double checked. Before reverting the change in silent.c, I was also not able to save uncompressed simple silent pictures with mlv_lite loaded and activated.
I also don't see any perceptual difference in behaviour between these two versions. Shall there be any? (to be sure it worked as expected, i.e. simple pictures are taken without pausing LV...)

EDIT #2:
700D behaves the same. The simple silent pictures don't work in movie mode with mlv_lite activated. However, lossless MLV videos do work... Looks like a non-related (?) issue.

[a1ex]

You don't have to try saving pictures with mlv_lite active. It just won't work, because the silent picture module will not be able to allocate memory.

The goal is compressing one frame while Canon's LiveView is running (in both photo and movie mode).

You can tell whether the picture is saved outside LiveView or not, by looking at the screen. Does the image pause for a moment (with a short flicker), or does it stay real-time while the image is saved?

You could call the following from 'don't click me' (run_test in debug.c):

Code: [Select]

msleep(2000);
PauseLiveView();
ResumeLiveView();

You will notice a difference. The same "flicker" (or stuttering or whatever) happens on crop_rec_4k with simple silent picture - it pauses LiveView for a moment. On the "unified" codebase, a simple silent picture is saved right away, without pausing LiveView, so there will be no "flicker" or stuttering.

[ArcziPL]

Thank you a1ex!

My previous post was garbage. The silent.mo didn't compile after my manually entered changes and I didn't notice immediately... There were also leftovers of the previous build on the card...

Code: [Select]

silent.c: In function 'silent_pic_take_lv':
silent.c:1250:14: error: too many arguments to function 'silent_pic_save_file'
         ok = silent_pic_save_file(&local_raw_info, 0);
              ^
silent.c:560:12: note: declared here
 static int silent_pic_save_file(struct raw_info * raw_info)
            ^
../../Makefile.filerules:31: recipe for target 'silent.o' failed
make[5]: *** [silent.o] Error 1
Now it's fixed, I can confirm there is no visible "flicker" when taking a simple silent photo anymore and no preview is shown afterwards, so we can be sure LV is not paused anymore.

Result with silent.mo + mlv_rec loaded, RAW video on (just in case it matters):

Simple silent pictures do work, both compressed and uncompressed, in photo and in video mode! The compressed ones look identical to uncompressed and are 48% smaller, so all as it should be. 

You could call the following from 'don't click me' (run_test in debug.c):

Thanks for reminding where to find it. I will definitely need it sooner or later.

[a1ex]

Sounds good!

In this case, debugging should be "just" a matter of adding printf's in mlv_lite, possibly in compress_task, to find out where it gets stuck. Just a random thought - does that task even start? Most of ML code does not check whether a task could be successfully started; need to fix it.

Otherwise, lossless.c has its own share of debug messages - enable them with verbose=1 or verbose=2 (just hardcode the value). These are also useful, to see whether the encoder is called with the right arguments, whether it starts, whether it finishes, how long it takes and so on. These are very useful hints if things don't work.

I was afraid of something worse, i.e. one of these cases where only a MMIO trace could help - assuming it could get saved to card before the camera locked up completely. That's harder to debug, but usually not impossible.

[ArcziPL]

Let me say it again: my previous post was garbage.

Implementing a lot of printf() in
mlv_lite.c -> compress_task()
lossless.c -> lossless_compress_raw_rectangle()

let me find out, that the camera stucks at

Code: [Select]

lossless_compress_raw_rectangle():
    ...
    /* configure the processing modules */
    TTL_Prepare(TTL_ResLock, &TTL_Args);
I checked it closer to find out, that it stucks also in photo mode (sic!) when saving simple silent photos with L-DNG option (with the modified simple.c, so LV is not paused) 

Previously I reported that it was working, because the file indeed got saved. Unfortunately this is what happens:
- entering LV
- pressing half-shutter
- lossless_compress_raw_rectangle() is started but stucks at the above mentioned line
- I did exit LV at that point...
- and only then the execution continued and file was finally saved.

If LV is not exited by user, you can't trigger second silent picture as the first one is still hanging and waiting.

Shit, so we're worse than thought.

I don't believe this log helps but posting anyway:
Camera start in photo mode -> LV entry -> simple silent photo (which stucks) -> exit LV -> card door open

Code: [Select]

Scanning modules...
Load modules...
  [i] load: SILENT.MO
Linking..
.text: 0
.data: 0
.config_vars: 0
.module_strings: 0
.text: 0
.data: 0
.config_vars: 0
.module_strings: 0
Register modules...
Load configs...
Init modules...
  [i] Init: 'silent'
  [i] prop 0x80000007
updating Movie Tweaks -> Movie Logging
  [i] prop 0x80000005
  [i] cbr 'CBR_CUSTOM_PICTURE_TAKING' -> 000BE738C
  [i] cbr 'CBR_SHOOT_TASK' -> 000BE8380
  [i] cbr 'CBR_VSYNC' -> 000BE685C
  [i] cbr 'CBR_DISPLAY_FILTER' -> 000BE66FC
Updating symbols...
  [i] 404: edmac_format_size 44d810
  [i] 404: edmac_format_size 44fba0
  [i] 404: edmac_format_size 459780
  [i] 404: edmac_format_size 45b3c0
  [i] 404: dual_iso_get_recovery_iso 4648c0
  [i] 404: dual_iso_is_active 4648c0
  [i] 404: auto_ettr_intervalometer_wait 46f7b0
  [i] 404: auto_ettr_intervalometer_warning 46f7b0
  [i] 404: auto_ettr_export_correction 47b880
  [i] 404: dual_iso_get_dr_improvement 488530
  [i] 404: dual_iso_get_recovery_iso 488530
  [i] 404: edmac_format_size 48d140
Modules loaded
Default raw buffer OK for 1984x761 (2.5MB).
raw update from shoot_task
LV raw buffer: b328000 (1984x761)
Skip left:144 right:8 top:28 bottom:0
Resolution changed: 0x0 -> 1984x761
Subsampling mode: 3B0Sx1B2S (5472x3648 1.62)
LV raw dimensions changed
raw update from shoot_task
LV raw buffer: b328000 (1984x761)
Skip left:144 right:8 top:28 bottom:0
Subsampling mode: 3B0Sx1B2S (5472x3648 1.62)
active area: x=144..1976, y=28..761
lv2raw sx:2565 sy:1529 tx:158 ty:36
raw2lv test: (-6,-6) - (725,485)
  should be: (0,0) - (720,480)
raw2bm test: (-6,-6) - (725,485)
  should be: (0,0) - (720,480)
bm2raw test: (158,36) - (1961,752)
  should be: (144,28) - (1976,761)
Black check 1/5: 2047…5.37, ref 2047…5.55, delta=0
Black check 2/5: 2047…5.21, ref 2047…5.55, delta=0
Black check 3/5: 2048…5.39, ref 2047…5.55, delta=1
Black check 4/5: 2047…5.86, ref 2047…5.55, delta=0
Black check 5/5: 2047…5.36, ref 2047…5.55, delta=0
Black check 1/5: 2047…5.78, ref 2047…5.55, delta=0
Black check 2/5: 2047…5.56, ref 2047…5.55, delta=0
Black check 3/5: 2047…5.79, ref 2047…5.55, delta=0
Black check 4/5: 2047…5.44, ref 2047…5.55, delta=0
Black check 5/5: 2047…5.68, ref 2047…5.55, delta=0
Black level: 2047
dynamic range: 10.70 EV (iso=200)
black=2047 white=10461
raw update from shoot_task
LV raw buffer: b328000 (1984x761)
Skip left:144 right:8 top:28 bottom:0
Subsampling mode: 3B0Sx1B2S (5472x3648 1.62)
Black check 1/5: 2047…5.80, ref 2047…5.64, delta=0
Black check 2/5: 2047…5.71, ref 2047…5.64, delta=0
Black check 3/5: 2047…5.40, ref 2047…5.64, delta=0
Black check 4/5: 2047…5.98, ref 2047…5.64, delta=0
Black check 5/5: 2047…5.56, ref 2047…5.64, delta=0
Black check 1/5: 2047…5.87, ref 2047…5.64, delta=0
Black check 2/5: 2047…5.36, ref 2047…5.64, delta=0
Black check 3/5: 2047…5.55, ref 2047…5.64, delta=0
Black check 4/5: 2047…5.44, ref 2047…5.64, delta=0
Black check 5/5: 2047…5.14, ref 2047…5.64, delta=0
dynamic range: 10.70 EV (iso=200)
black=2047 white=10461
srm_malloc_suite(1)...
[SRM] alloc all buffers
UILock: 00000000 -> 41000001 => 41000001
[SRM] buffer 5ca00064
[SRM] buffer 42000064
[SRM] buffer 44400064
UILock: 41000001 -> 41000000 => 41000000
[SRM] not using 5ca00064
srm_malloc_suite => ff5e8
Freeing LV raw buffer 4b328000.
ArcziPL: starting lossless_compress_raw_rectangle()
ArcziPL: lossless_compress_raw_rectangle, stage 1.
ArcziPL: lossless_compress_raw_rectangle, stage 2.
ArcziPL: lossless_compress_raw_rectangle, stage 3.
ArcziPL: lossless_compress_raw_rectangle, stage 4.
ArcziPL: lossless_compress_raw_rectangle, stage 5.
ArcziPL: lossless_compress_raw_rectangle, stage 6.
ArcziPL: lossless_compress_raw_rectangle, stage 7. <- printed just before TTL_Prepare(TTL_ResLock, &TTL_Args); the camera hangs here until I quit LV
ArcziPL: lossless_compress_raw_rectangle, stage 8. <- printed just after TTL_Prepare(TTL_ResLock, &TTL_Args);
ArcziPL: lossless_compress_raw_rectangle, stage 9.
ArcziPL: lossless_compress_raw_rectangle, stage 10.
ArcziPL: lossless_compress_raw_rectangle, stage 11.
ArcziPL: lossless_compress_raw_rectangle, stage 12.
[TTL] 1984x761 14bpp
 WR1: 46800064 EDMAC#17<45> (113428 2.0MB)
 WR2: 0 EDMAC#2<22>
 RD1: 4400080 EDMAC#25<1>
 RD2: 0 EDMAC#25<0>
ArcziPL: lossless_compress_raw_rectangle, stage 13.
ArcziPL: lossless_compress_raw_rectangle, stage 14.
ArcziPL: lossless_compress_raw_rectangle, stage 15.
ArcziPL: lossless_compress_raw_rectangle, stage 16.
ArcziPL: lossless_compress_raw_rectangle, stage 17.
[TTL] Elapsed time: 30483 us
ArcziPL: lossless_compress_raw_rectangle, stage 18.
ArcziPL: lossless_compress_raw_rectangle, stage 19.
[TTL] Output size : 1.3MB (51.25%)
ArcziPL: lossless_compress_raw_rectangle, stage 20.
srm_free_suite(ff5e8)
[SRM] free all buffers
UILock: 41000000 -> 41000000 => 41000000
Save configs...


[a1ex]

Alright - it locks up while trying to get exclusive access to some image processing resources. This is because something else from Canon's LiveView is using some of them.

I'd say let's find out what resources are causing the lockup (by trial and error, commenting out some of them to narrow down). Once that is done, we can attempt to find what other code attempts to lock the same resources, and maybe - if lucky - we can find a way to disable it somehow.

[ArcziPL]

I'd say let's find out what resources are causing the lockup (by trial and error, commenting out some of them to narrow down)

I guess this would mean removing some lines from

Code: [Select]

    else if (is_camera("5D3", "*") || is_camera("6D", "*") || is_camera("70D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x30001,    /* Read connection 1 (uncompressed input) */
            0x2002d,    /* Write connection 45 (compressed output) */
          //0x20016,    /* Write connection 22 (for WR2 - not used) */
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x230000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };
Is it right?

[a1ex]

Yes, that's right. If you comment out all of them, it should go past TTL_Prepare, but... encoding is not going to work.

Besides, if you test this outside LiveView, it may work even without locking some of the resources. For example, IIRC EDMAC channels/connections don't *have* to be locked in order to use them; the locks are just a way to make sure each resource will be used by one task at a time. However, other resources seem to require locking; these routines will also perform some changes to clock enable bits (messages about ClockSave in the log file).

If lucky, some resources might be always locked during LiveView, but they might be used only during mode changes or autofocus or something like that. In this case, encoding might work with these entries commented out.

[ArcziPL]

Hell yeah, first 14-bit lossless MLV from 70D!



It's amazing what you know about and can do with closed internals of these cams!

The one to be commented out is

Code: [Select]

//0x5002d,
If too much was commented out (all but first four lines: EDMAC channels and read/write connections) it was still stucking at the same point and additionally locking the camera. LV couldn't be exited. After changing mode between photo/video it thrown Err80.

With only this one line removed I haven't noticed any side effects yet but wasn't focusing on it. I'll test it in the next days/weeks, report back and submit a pull request if all seems to be fine.

[theBilalFakhouri]

This is the right time to switch to 70D 
Great work!

[Danne]

Thumbs up! Great job.

[buro341]

awesome, is there much difference between ML for 70D and 77D?

[Walter Schulz]

Yes, there is a working ML for 70D but not for 77D.

[2blackbar]

Great work ! ArcziPL Do You think 5D2 needs some code to be commented out to prevent camera lockup like 70D ?

Hell yeah, first 14-bit lossless MLV from 70D!



It's amazing what you know about and can do with closed internals of these cams!

The one to be commented out is

Code: [Select]

//0x5002d,
If too much was commented out (all but first four lines: EDMAC channels and read/write connections) it was still stucking at the same point and additionally locking the camera. LV couldn't be exited. After changing mode between photo/video it thrown Err80.

With only this one line removed I haven't noticed any side effects yet but wasn't focusing on it. I'll test it in the next days/weeks, report back and submit a pull request if all seems to be fine.

[reddeercity]

Starting to dig in this again for d4 cam's , found something very interesting

Code: [Select]

        EosCompressLevel :
         Unknown = 0xF,
        JpegUncompressed = 0,
        JpegCompression1 = 1,
        Normal = 2,
        Fine = 3,
        Lossless = 4,
        SuperFine = 5,
        JpegCompression6 = 6,
        JpegCompression7 = 7,
        JpegCompression8 = 8,
        JpegCompression9 = 9,
        JpegCompression10 = 10,
"Lossless = 4" , seems there more compression available then just Lossless 

Edit:
found this string

Code: [Select]

"Primary Image: Size <{0}>, Format <{1}>, CompressLevel <{2}>\n"
+ "Secondary Image: Size <{3}>, Format <{4}>, CompressLevel <{5}>"
So this is what Canon is doing in the background 

1: PrimaryImageSize
2: PrimaryImageFormat
3: PrimaryCompressLevel
4: SecondaryImageSize
5: SecondaryImageFormat
6: SecondaryCompressLevel

There's 2 levels of compression it looks like , input compression ? (from sensor ?) and then export compression ,Lossless ,Uncompressed , etc. ...
Interesting !

[names_are_hard]

Primary vs secondary could be full size image vs preview?  I'm not sure what to make of Lossless vs JpegUncompressed, unless only the first is Raw?

Definitely looks interesting and worth some experiments.

[reddeercity]

Primary vs secondary could be full size image vs preview?

Yes i thinks so the Log files from my 5d2 supports this plus it looks like the JP62 chip (to my knowledge the d4 encoder chip for lossless/cr2/h264/etc. ...)
is producing 4 different image streams from the sensor .
5616x3744
4080x2720
3861x2574
2784x1856

Code: [Select]


9.707.568     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(5616) keta(4)
9.708.482     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(3744) keta(4)
.......
9.700.451      CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(4080) keta(4)
9.701.108     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(2720) keta(4)
........
9.711.244     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(3861) keta(4)
9.712.166     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(2574) keta(4)
......
9.703.646     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(2784) keta(4)
9.704.562     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(1856) keta(4)

Log for HD resizing

Code: [Select]

7.264.373  ImgPlayDrv:ffa3d77c:1a:02: RSZ Input  Add:0x40D00000 Basew:2106 x:0 y:0 w:2106 h:1404
 7.264.406  ImgPlayDrv:ffa3d7a8:1a:02: RSZ Output Add:0x41B07800 Basew:720 x:0 y:52 w:720 h:428
 7.264.426  ImgPlayDrv:ffa3d7d4:1a:02: RSZ pfil:2 pres:2 xxa:117 xxb:80 xya:351 xyb:214
 7.264.433  ImgPlayDrv:ffa3e9e4:1a:02: RSZ ColMode:1
 7.264.449  ImgPlayDrv:ffa3e624:1a:02: RSZ SetFixedEngineParameter(0,2106)
 7.264.460  ImgPlayDrv:ffa3e0e0:1a:02: RSZ ConvertParameterOfInputAndOutput
 7.264.495  ImgPlayDrv:ffa3e550:1a:02: RSZ ConvertParameterOfInputAndOutput (Err:0x0)
 7.264.512  ImgPlayDrv:ffa3e570:1a:02: RSZ x:0 y:0 w:2106 h:1404
 7.264.534  ImgPlayDrv:ffa3e5a4:1a:02: RSZ pfil:2 xx:80/117 pres:2 xy:214/351
 7.264.554  ImgPlayDrv:ffa3e5cc:1a:02: RSZ In  xa:0 xb:2106 xn:0 ya:0 yb:1404 yn:0
 7.264.573  ImgPlayDrv:ffa3e5f4:1a:02: RSZ Out xa:0 xb:720 xn:0 ya:0 yb:428 yn:0
 7.264.597  ImgPlayDrv:ffa3dd80:1a:02: RSZ CorrectPositionAndSize(0,2106)
preview & position i think "RSZ pfil:2 xx:80/117 pres:2 xy:214/351"
I know this is lcd liveview  "RSZ Output Add:0x41B07800 Basew:720 x:0 y:52 w:720 h:428"

I'm not sure what to make of Lossless vs JpegUncompressed, unless only the first is Raw?

Being there are using a Jpeg JP62 chip for encoding image compression , Jpeg must have a uncompressed setting .
from this chart below link , Jpeg2000 has uncompressed & Lossless/Lossy: Wavelet @ 12ibt (38 bit)
http://preservationtutorial.library.cornell.edu/presentation/table7-1.html
looks like its Jpeg2000
https://en.wikipedia.org/wiki/Lossless_JPEG

All this makes sense , since the h264 is Motion Jpeg (Mjpeg)
Log from 5D2 playing a H264.mov file on the LCD screen on camera .
And It's Motion Jpeg ! 
My dream is still to have MJpeg @ native sensor size (5.6k)
All ml cam would benefit from that , specially the one with slow card write speed.

Code: [Select]

4.447.781     CtrlSrv:ff84a7b0:2c:03: StartResizeMotionJpeg(0,0x40d00000,0,0x5c007800)
 4.447.797     CtrlSrv:ff98ddf4:1a:03: -------------------------------------501
 4.447.823     CtrlSrv:ffa4e818:1a:03: [H264Resize] pInputAddress  = 0x40D00000
 4.447.844     CtrlSrv:ffa4e82c:1a:03: [H264Resize] pOutputAddress = 0x5C007800
 4.447.849     CtrlSrv:ffa4e83c:1a:03: [H264Resize Req]
  4.447.883     CtrlSrv:ffa3f6cc:1a:02: RSZ RequestResizeYuv RdAdr:0x40d00000, WrAdr:0x5c007800
 4.447.902     CtrlSrv:ffa3f6f0:00:00: *** LockEngineResources(861d20) x20:
 4.447.915     CtrlSrv:000970ec:00:00:      1)    6000f (?)
 4.447.926     CtrlSrv:000970ec:00:00:      2)    60009 (?)
 4.447.937     CtrlSrv:000970ec:00:00:      3)    6000e (?)
 4.447.948     CtrlSrv:000970ec:00:00:      4)    60006 (?)
 4.447.960     CtrlSrv:000970ec:00:00:      5)    6000d (?)
 4.447.971     CtrlSrv:000970ec:00:00:      6)    6000a (?)
 4.447.982     CtrlSrv:000970ec:00:00:      7)    60002 (?)
 4.447.994     CtrlSrv:000970ec:00:00:      8)    60001 (?)
 4.448.006     CtrlSrv:000970ec:00:00:      9)   130001 (?)
 4.448.019     CtrlSrv:000970ec:00:00:     10)   130002 (?)
 4.448.032     CtrlSrv:000970ec:00:00:     11)   130010 (?)
 4.448.045     CtrlSrv:000970ec:00:00:     12)   130011 (?)
 4.448.058     CtrlSrv:000970ec:00:00:     13)   130017 (?)
 4.448.071     CtrlSrv:000970ec:00:00:     14)   130018 (?)
 4.448.084     CtrlSrv:000970ec:00:00:     15)   130019 (?)
 4.448.097     CtrlSrv:000970ec:00:00:     16)   13001a (?)
 4.448.110     CtrlSrv:000970ec:00:00:     17)   13001b (?)
 4.448.123     CtrlSrv:000970ec:00:00:     18)   13001c (?)
 4.448.142     CtrlSrv:000970ec:00:00:     19)    10003 (read channel 0xb)
 4.448.160     CtrlSrv:000970ec:00:00:     20)        3 (write channel 0x3)
Edit:
Did a little google searching and found out that digital still camera's (not sure if all )are using JPEG-LS Standard compression
 http://dicom.nema.org/medical/dicom/2017a/output/chtml/part05/sect_F.2.html

The lossless and lossy (near-lossless) coding is based on a predictive scheme with statistical modeling,
in which differences between pixels and their surround are computed and their context modeled prior to coding,
with a run-length escape mechanism.
This scheme achieves consistently better compression in lossless mode than the lossless processes of JPEG defined in ISO 10918-1, with less complexity.
..........
A single JPEG-LS process is used for bit depths up to 16 bits.

[reddeercity]

Taking a another run at Lossless on 5d2 --
One thing i have notice between Lossless dng & Uncompressed dng is when uncompressed dng is being save there's a pause & flicker in liveview
before it save the dng , but in lossless on 5d2 it just freezes liveview , no pause or flicker .

I see in a previous post by a1ex  https://www.magiclantern.fm/forum/index.php?topic=18443.msg212992#msg212992 about this .
So tried to comment out part of the ttl resources to see if this made a differences , No still don't save , just freeze liveview .

At one point i disable all the resources & had a "ERROR 70" flash at bootup , needed a quick battery pull
Here part of the log file with the error

Code: [Select]

1269:  3309.178 [STARTUP] ERROR ASSERT : Memory\Memory.c, Task = module_task
1270:  3309.268 [STARTUP] ERROR ASSERT : Line 184
1271:  3309.299 [STARTUP] ERROR ASSERT : FALSE
1274:  3309.909 [ENG] CreateResLockEntry cannot allocate pLockEntry->pResource
1277:  3317.781 [STARTUP] ERROR ASSERT : KernelDry\KerSem.c, Task = audio_common_task
1278:  3317.859 [STARTUP] ERROR ASSERT : Line 297
1279:  3317.890 [STARTUP] ERROR ASSERT : 0
1280:  3318.028 [STARTUP] startupErrorRequestChangeCBR : OverWrite (0x1d => 0x1d)
1298:  3398.663 [MC] Disp Error : 70
This line

Code: [Select]

1274:  3309.909 [ENG] CreateResLockEntry cannot allocate pLockEntry->pResourceSeems to me before i commend out all the resource i must have had Res Lock but I think it couldn't find the path where to save it  ?
Here the log file 5d2_photo-mode-error70-lossless-log000.log

I need to get my head around the ResLock stuff , there's more going on here then i can see .
But the good thing there it saved a total Log file in Photo mode , gave me some nice info
Here is those 4 raw streams being produce again . Be nice to get inside the JP62 chip and see the gears work

Code: [Select]

1003:  1092.749 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(5616) keta(4)
1004:  1093.519 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(3744) keta(4)
1005:  1094.247 [GUI] DlgShootOlcInfo.c CreateCompoBmpForImageSize()
1006:  1095.904 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(4080) keta(4)
1007:  1096.543 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(2720) keta(4)
1008:  1097.312 [GUI] DlgShootOlcInfo.c CreateCompoBmpForImageSize()
1009:  1098.747 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(2784) keta(4)
1010:  1099.563 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(1856) keta(4)
1011:  1100.323 [GUI] DlgShootOlcInfo.c CreateCompoBmpForImageSize()
1015:  1105.096 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(3861) keta(4)
1016:  1105.905 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(2574) keta(4)
1017:  1106.825 [GUI] DlgShootOlcInfo.c CreateCompoBmpForImageSize()

   

[reddeercity]

Found a very good document about Lossless compression from a canon cr2 file ,
breaks down how lossless data is aranged on a image .
Canon_CR2_file_format_ raw bit coder/decode.txt

The forth IFD is the sweet spot of the CR2 file.  It is a lossless JPEG
* compressed image of dimensions the size of the camera's photo sensor.  Canon
* adds three proprietary TIFF tags

the compressed data is organized as a concatenation of a Huffmann
* code (i.e. HufCode or codeword) followed by a variable length difference code
* (i.e. Diff) bit stream

Code: [Select]

The RAW image is divided into 4 equal sized
* The finished image inteleaves quadrants row segments 0/1 and interleaves
* quandrant rows segments 2/3 while combined quandrants 0/1 is located to the
* left and combined quadrants 2/3 are concatenated to the right:
*     +------------------------+------------------------+ Finished Image
*     |      Quad0.Row0        |      Quad2.Row0 |
*     |      Quad1.Row0        |      Quad3.Row0 |
*     |      Quad0.Row1        |      Quad2.Row1 |
*     |      Quad1.Row1        |      Quad3.Row1 |
*     | etc        |   etc |
*     |      Quad0.RowN        |      Quad2.RowN |
*     |      Quad1.RowN        |      Quad3.RowN |
*     +------------------------+------------------------+
* So it can be said that Quads 0 and 2 contain the same two BAYER grid colors
* and Quads 1 and 3 contain the other two BAYER grid colors.
etc. ....
Interesting to see how lossless data is organized , since this is what we use for lossless  .