ProcessTwoInTwoOutLosslessPath

[a1ex]

when i try to review in the camera "no image"

capture image, then use file_man & pic_view

However there is still this behaviour that the brightness of the realtime live-view image decreases as I step from 14 > 12 > 11bit lossless. This doesn’t happen without lossless compression engaged in the mlv-lite menu. This characteristic also appears to be encoded into the compressed mlv recordings so they actually appear to get brighter and brighter as the compression ratio increases.

Recordings shouldn't get brighter and brighter - they should look the same at all bit depths (more or less - it doesn't match perfectly yet).

However, LiveView should be brightened back - this step doesn't seem to work. Try Display -> LV display gain - is this feature working (in photo mode) ? Also try Movie -> Image fine-tuning -> ML digital ISO (in movie mode).

If it doesn't work, I'm afraid one would have to follow this procedure to diagnose it:

https://www.magiclantern.fm/forum/index.php?topic=10111.msg191218#msg191218

The other issue here is that the highlights become pinker and pinker and even using the highlight reconstruction of the mlv014 app it won’t go away.

Please run this test: http://www.magiclantern.fm/forum/index.php?topic=16054.msg195395#msg195395

(also applies to all other cameras; only 100D done so far)

[dfort]

lossless dng has something to do with black level.

Well the lossless DNG is smaller than the uncompressed one and it isn't totally black so that's a good sign. That bottle with the skull and crossbones on the label next to your computer, that's a bad sign!

One thing to try is to switch back and forth between regular DNG and lossless DNG. I had an issue where if the camera was started with lossless active it wouldn't work. That has since been resolved but maybe there's something similar going on with the 70D? Sorry, couldn't find the post where this issue was discussed.

Are you able to compile? That would help speed up the process because there are several things we can try but it is best to take them like your poison--just a little drop.

Candidates: C0F37074 (try 0x180000), 0xC0F37290/94 (maybe 0? no idea)

Seems that 0xC0F37078 could also use 0x180000 so we can try doing this with the 70D adjustments:

Code: [Select]

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37290, 0);       /* 0x400 on 70D, not set on all other D5 */
        EngDrvOut(0xC0F37294, 0);       /* 0x800 on 70D, not set on all other D5 */
    }

I'll post a new 70D test build with these changes.

However there is still this behaviour that the brightness of the realtime live-view image decreases as I step from 14 > 12 > 11bit lossless.

For sure you don't have Auto ISO set? I noticed that in certain situations when ISO is set to Auto, switching bit depths affects the brightness of the LiveView.

As far as the pink highlights, that is also happening on the 700D. Looks like there's a test to run that might eventually resolve this issue.

No, the idea is to feed larger images into the encoder (not necessarily with valid data - it will compress whatever it finds in memory), and keep height-1 for 0xC0F13068. Click on the links.

Understand I did not. Ok--put back the height-1 and will try to figure out what we were talking about last month on this discussion.

[saulbass]

@ dfort & a1ex: I can confirm I had idiotmode activated.

with auto iso off(!)

Again using dfort’s recent crop_rec_4k.2018Mar04.650D104. 650D in 1080p:
14, 12, 10, 14LL, 12LL, 10LL - all working - no differences in exposure level & no pink highlights.

Fabulous. Thank you so much - this is amazing!!

[Danne]

Understand I did not

@saulbass
Great stuff going on!

[reddeercity]

Found the correct resources[] for 5d2  but I little confuse on how to implement as it's similar to 7d/D4 & 5d3/D5

Code: [Select]

ShootCapture:ff87ffec:93:03: GetCaptureResources start.
     ShootCapture:00096480:00:00: *** LockEngineResources(68007c) x38 from ffa384f8:
     ShootCapture:00096534:00:00:      1)    50001 (?)
     ShootCapture:00096534:00:00:      2)    50005 (?)
     ShootCapture:00096534:00:00:      3)    50007 (?)
     ShootCapture:00096534:00:00:      4)    50009 (?)
     ShootCapture:00096534:00:00:      5)    5000e (?)
     ShootCapture:00096534:00:00:      6)    50011 (?)
     ShootCapture:00096534:00:00:      7)    50019 (?)
     ShootCapture:00096534:00:00:      8)    5001b (?)
     ShootCapture:00096534:00:00:      9)    5001e (?)
     ShootCapture:00096534:00:00:     10)    50013 (?)
     ShootCapture:00096534:00:00:     11)    50020 (?)
     ShootCapture:00096534:00:00:     12)    50021 (?)
     ShootCapture:00096534:00:00:     13)    70000 (?)
     ShootCapture:00096534:00:00:     14)    f0000 (?)
     ShootCapture:00096534:00:00:     15)    b0000 (?)
     ShootCapture:00096534:00:00:     16)   200002 (?)
     ShootCapture:00096534:00:00:     17)   140000 (?)
     ShootCapture:00096534:00:00:     18)   120000 (?)
     ShootCapture:00096534:00:00:     19)   130009 (?)
     ShootCapture:00096534:00:00:     20)   13000a (?)
     ShootCapture:00096534:00:00:     21)   13000b (?)
     ShootCapture:00096534:00:00:     22)   13000d (?)
     ShootCapture:00096534:00:00:     23)   13000e (?)
     ShootCapture:00096534:00:00:     24)   13000f (?)
     ShootCapture:00096534:00:00:     25)   130016 (?)
     ShootCapture:00096534:00:00:     26)   22001b (?)
     ShootCapture:00096534:00:00:     27)        0 (write channel 0x0)
     ShootCapture:00096534:00:00:     28)        1 (write channel 0x1)
     ShootCapture:00096534:00:00:     29)        2 (write channel 0x2)
     ShootCapture:00096534:00:00:     30)        6 (write channel 0x6)
     ShootCapture:00096534:00:00:     31)    10000 (read channel 0x8)
     ShootCapture:00096534:00:00:     32)    10001 (read channel 0x9)
     ShootCapture:00096534:00:00:     33)    20000 (write connection 0x0)
     ShootCapture:00096534:00:00:     34)    20002 (write connection 0x2)
     ShootCapture:00096534:00:00:     35)    20012 (write connection 0x12)
     ShootCapture:00096534:00:00:     36)    2001c (write connection 0x1c)
     ShootCapture:00096534:00:00:     37)    30008 (read connection 0x8)
     ShootCapture:00096534:00:00:     38)    3000f (read connection 0xf)
This is the 7D resources[] which 5d2 was just add to see if it would work

Code: [Select]

  uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10002 | edmac_channel_to_index(edmac_read_chan),
            0x30000,    /* read  connection  0 */
            0x20005,    /* write connection  5 */
            0x20016,    /* write connection 22 */
            0x50003,
            0x5000d,
            0x5000f,
            0x5001a,
            0x80000,
            0x90000,
            0xa0000,
            0x160000,
            0x130003,
            0x130004,
            0x130005,
        };
 /*       
 1)    10002 (read channel 0xa)
 2)        3 (write channel 0x3)
 3)        4 (write channel 0x4)
 4)    30000 (read connection 0x0)
 5)    20005 (write connection 0x5)
 6)    20016 (write connection 0x16)
 7)    50003 (?)
 8)    5000d (?)
 9)    5000f (?)
10)    5001a (?)
11)    80000 (?)
12)    90000 (?)
13)    a0000 (?)
14)   160000 (?)
15)   130003 (?)
16)   130004 (?)
17)   130005 (?)
*/I didn't run the right dm-spy test etc. ... Thank to a1ex for so very old work on the 5d2
If I would have listen to a1ex mouths ago , he told me to run a FA_Capture Test but couldn't understand at the time why .
Now I do there all the info I need for compressed raw & even very good info for 3.5k/UHD for D4's cam's
I use this FA_CaptureTestImage.log for the first page on FRSP -- should have known , I didn't search enough.
so do I use all the 32 entries ? or stop at 17 ?
if anyone can  help that would great 

Edit: seem incomplete , doesn't show the resources[] for compression but the over all resources etc. ... need to run a fa-capture test myself see if I can extract the info needed .

[dfort]

@reddeercity -- Looks like you're getting close but wow, that log file is very different from the one @nickfreak posted from his 7D. What's going on with the Digic 5 lossless code is likely to apply to Digic 4. Of course getting any Digic 4 camera working with lossless compression is still quite a challenge. I would have thought that the 60D would be the first to work. Someone we know who is really good with ML code owns one.

One tip, the 5D2 should compile in the crop_rec_4k branch with this change:

src/raw.c

Code: [Select]

-#if defined(EVF_STATE)  /* 60D and newer */
+#if defined(EVF_STATE) || defined(CONFIG_7D)  /* 60D and newer */
 #define BLACK_LEVEL 2047
 #endif

So never mind my previous comment about juggling babies, we might as well jump onto this branch because it is where much of the current development is happening.

In the meantime I've been trying to run some tests in the hopes that the Lossless Compression updates for all Digic 5 cameras pull request gets accepted. (a1ex did all the heavy lifting so...)

No, the idea is to feed larger images into the encoder (not necessarily with valid data - it will compress whatever it finds in memory), and keep height-1 for 0xC0F13068. Click on the links.

Thought I figured it out so I set up this elaborate test on the 700D and half-way through realized I was doing it all wrong because of this:

I meant, can you save full-res lossless DNG with height greater than 3528 on 700D/M?

(the answer will be useful as documentation, regarding what this register does - what we know about it; I don't care about that one pixel difference, I just want to know where that limit comes from)

Still, the test I ran was interesting so here it is.

What I did was to add this to see what was going on:

modules/silent/silent.c

Code: [Select]

    ASSERT(out_suite->size == max_compressed_size);
       
+   printf("out_raw_info.width = %d\n", out_raw_info.width);
+   printf("out_raw_info.height = %d\n", out_raw_info.height);
+   printf("out_raw_info.active_area.y2 = %d\n", out_raw_info.active_area.y2);

    out_raw_info.frame_size = lossless_compress_raw(&out_raw_info, out_suite);

With the console open it prints out the frame size being fed into lossless compression.

So, how to increase the image height? What we were working on a month ago culminated in this:

src/raw.c

Code: [Select]

#if defined(CONFIG_700D) || defined(CONFIG_650D)
    (*height)++;
#endif

So, I decided to play around here:

Code: [Select]

(*height)--;
(*height)+=2;
etc...
Yeah--that wasn't the right thing to do. I probably should have done this right before the printf statements:

modules/silent/silent.c

Code: [Select]

(out_raw_info.height)+=3;
(out_raw_info.active_area.y2)+=3;
@a1ex - Is this what you want me to do with FRSP lossless?

In any case, the results I got were interesting, at least to me. Testing on the 700D.

(*height)--;

Code: [Select]

Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  725
out_raw_info.active_area.y2 =  721
image = 1736x693

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1188
out_raw_info.active_area.y2 = 1184
image = 1736x1156

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1058
out_raw_info.active_area.y2 = 1056
image = 1800x1028

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1106
out_raw_info.active_area.y2 = 1106
image = 2520x1080

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  725
out_raw_info.active_area.y2 =  721
image = 1736x693

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476
No height adjustment

Code: [Select]

Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  726
out_raw_info.active_area.y2 =  722
image = 1736x694

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1189
out_raw_info.active_area.y2 = 1185
image = 1736x1157

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1059
out_raw_info.active_area.y2 = 1057
image = 1800x1029

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1107
out_raw_info.active_area.y2 = 1107
image = 2520x1081

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  726
out_raw_info.active_area.y2 =  722
image = 1736x694

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476
(*height)++; - This is the current code

Code: [Select]

Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  727
out_raw_info.active_area.y2 =  723
image = 1736x695

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1190
out_raw_info.active_area.y2 = 1186
image = 1736x1158

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1060
out_raw_info.active_area.y2 = 1058
image = 1800x1030

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1108
out_raw_info.active_area.y2 = 1108
image = 2520x1082


Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  727
out_raw_info.active_area.y2 =  723
image = 1736x695

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476
(*height)+=2;

Code: [Select]

Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  728
out_raw_info.active_area.y2 =  724
image = 1736x696 - ng

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1191
out_raw_info.active_area.y2 = 1187
image = 1736x1159

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1061
out_raw_info.active_area.y2 = 1059
image = 1800x1031 - ng

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1109
out_raw_info.active_area.y2 = 1109
image = 2520x1083 - ng

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  728
out_raw_info.active_area.y2 =  724
image = 1736x696 - ng

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476
(*height)+=3;

Code: [Select]

Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  729
out_raw_info.active_area.y2 =  725
image = 1736x697 - white line bottom of frame

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1192
out_raw_info.active_area.y2 = 1188
image = 1736x1160 - ng

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1062
out_raw_info.active_area.y2 = 1060
image = 1800x1032

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1110
out_raw_info.active_area.y2 = 1110
image = 2520x1084 - ng

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  729
out_raw_info.active_area.y2 =  725
image = 1736x697 - ng

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476
Ok--you get the idea. Things are starting to break and FRSP isn't budging so let's really push it:

(*height)+=9;

Code: [Select]

Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  735
out_raw_info.active_area.y2 =  731
image = 1736x703 - ng

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1198
out_raw_info.active_area.y2 = 1194
image = 1736x1166 - ng

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1068
out_raw_info.active_area.y2 = 1066
image = 1800x1038 - ng

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1116
out_raw_info.active_area.y2 = 1116
image = 2520x1090 - ng

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  735
out_raw_info.active_area.y2 =  731
image = 1736x703 - ng

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476
The raw buffer height and image size kept changing but it wasn't creating valid simple silent lossless compressed DNG files while FRSP wasn't affected in the least.

I'll try doing this test the right way tomorrow.

[andy kh]

capture image, then use file_man & pic_view

lossless dng is completely black

[reddeercity]

@reddeercity -- Looks like you're getting close but wow, that log file is very different from the one @nickfreak posted from his 7D. What's going on with the Digic 5 lossless code is likely to apply to Digic 4. Of course getting any Digic 4 camera working with lossless compression is still quite a challenge. I would have thought that the 60D would be the first to work. Someone we know who is really good with ML code owns one.

Yup , closer -- I bet @nickfreak did a FA_Capture_Test LOG , I'm still trying to find more info on that I think it only done in QEMU , I was hoping to do it in cam .
I may finally have to get QEMU running if I can't do it in cam.

One tip, the 5D2 should compile in the crop_rec_4k branch with this change:
src/raw.c

Code: [Select]

-#if defined(EVF_STATE)  /* 60D and newer */
+#if defined(EVF_STATE) || defined(CONFIG_7D)  /* 60D and newer */
 #define BLACK_LEVEL 2047
 #endif


Yes , I'm working off "crop_rec_4k branch" , I  comment out the "digital gain" stuff for lower bit lossless raw (8-12)
in raw.c

Code: [Select]

//if (gain)
    //{
        //lv_raw_gain = gain;
       // raw_info.white_level = get_default_white_level();
        //raw_info.black_level = BLACK_LEVEL;
    //}
        //elseand in lv-img-engio.c

Code: [Select]

//total_movie_gain *= _raw_lv_get_iso_post_gain();
Added all the changes that you had in pull-requests/18/loss-compression-for-digic-4-cameras/diff except I didn't add

Code: [Select]

]-#if defined(EVF_STATE)  /* 60D and newer */
+#if defined(EVF_STATE) || defined(CONFIG_7D)  /* 60D and newer */
 #define BLACK_LEVEL 2047
 #endifComplied OK , thou the mlv_lite didn't make but silent.mo did so I been working & testing off it .

Magic Lantern Nightly.2018Mar03.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 27345b3c1ad6+ (crop_rec_4k)
Built on : 2018-03-03 06:58:42 by ml@ml-pc

diff -r 27345b3c1ad6 src/gdb.c
--- a/src/gdb.c   Sat Feb 24 17:06:06 2018 +0100
+++ b/src/gdb.c   Sat Mar 03 07:58:42 2018

Maybe , that why the mlv_lite didn't make , I'll add that also and see if it make a difference .

[reddeercity]

For the 5D2 the black level is 1792 , 2047 if I remember right that's the black level for 5D3 & all D5 cams.

Code: [Select]

-#if defined(EVF_STATE)  /* 60D and newer */
+#if defined(EVF_STATE) || defined(CONFIG_7D)  /* 60D and newer */
 #define BLACK_LEVEL 2047
 #endifI did run full mlv , & had no black level problems (green or pink cast)
And silent photo are also ok

[dfort]

For the 5D2 the black level is 1792 , 2047 if I remember right that's the black level for 5D3 & all D5 cams.

Then do this:

Code: [Select]

#if defined(EVF_STATE)  /* 60D and newer */
#define BLACK_LEVEL 2047
#endif

#if defined(LV_STATE)  /* older Digic 4 */
#define BLACK_LEVEL 1792
#endif

The 60D, 600D, 1100D are Digic 4 EVF_STATE cameras which are newer and thus closer to the Digic 5 cameras than the 50D, 5D2, 500D, 550D, 7D which are LV_STATE cameras.

Note that defining BLACK_LEVEL was done on commit 239c022. At that time LV_STATE cameras were not working with the crop_rec_4k branch so it didn't matter if these cameras wouldn't compile.

I did run full mlv , & had no black level problems (green or pink cast)
And silent photo are also ok

You mean mlv_rec? It should work as long as you don't use the reduced bit depth settings. Same with silent uncompressed. What we're trying to do here is to get lossless compression working and we might be close--or not!

lossless dng is completely black

Uncompressed FRSP DNG working? Upload samples please, even if they are completely black.

[andy kh]

Uncompressed FRSP DNG working? Upload samples please, even if they are completely black.

yes FRSP works in my quick test. here is a sample https://files.fm/u/wp7qhjnj

sample for lossless dng completely black https://files.fm/u/wsmmdjx5

[dfort]

@andy kh

Your black DNG isn't a Full-res silent lossless, it is a simple silent. Still black though.

According to exiftool your uncompressed FRSP is 5648x3720 and the lossless DNG you uploaded is 1984x1251.

I'm not saying you did anything wrong--it might be what the code is doing but please make sure you are in photo mode, LiveView on and double check your settings:



@a1ex -- Instead of editing my post so it looks like I know what I'm doing I decided to show my unedited notes. I think this was a good lesson that others might learn from.

This isn't the way to do it, is it?

modules/silent/silent.c

Code: [Select]

(out_raw_info.height)+=3;
(out_raw_info.active_area.y2)+=3;
It has no affect on either FRSP or silent.

So how am I supposed to do this?

...can you save full-res lossless DNG with height greater than 3528 on 700D/M?

Or is the answer -- no. Even though I manipulated out_raw_info.height and out_raw_info.active_area.y2 all the way to +9, the image was always saved as:

5208x3476 - according to Adobe Camera Raw (Photoshop)
5280x3537 - according to exiftool

No, I didn't go dyslexic. I checked those numbers multiple times.

Wait a minute -- Whack on the side of he head moment.

This is the first time that I double checked with exiftool -- so something is changing.

Without doing any height manipulation:

5208x3476 - according to Adobe Camera Raw (Photoshop)
5280x3528 - according to exiftool

So it looks like the answer is Yes! A FRSP lossless DNG can be saved with a height greater than 3528 but ACR will always crop it to 3476.

How about dcraw? The DNG that exiftool reported as 5280x3528 (normal - no height manipulation) will convert to a TIFF that is 5208x3476 (both ACR and exiftool agree on that). The DNG that was manipulated to +9 and reported as 5280x3537 in exiftool will convert to a TIFF that is 5208x3485. However, there's no free lunch here, the extra height is just noise on the bottom of the frame:

[a1ex]

@dfort: yes, exactly that was the result I was expecting. There's no additional image data (I didn't expect any); I just wanted to see whether the "rabbit" register was somehow related to the 3528 limit we have reached a while ago.

In other words, changing this register on all models is likely a good move.

The funny thing about coding for ML is that no one really knows what they are doing.

For 70D, it might be worth trying just 0xC0F37074/78, 0xC0F37300/3e8 and also the rabbit (without 0xC0F37290/94). If it still doesn't work, we may have to call this procedure in its original environment (during CR2 capture, without overriding any parameters) and start tweaking from there until things break. That's at least one order of magnitude more difficult than the current register tinkering (but I'm pretty sure it's solvable, simply because... it worked for Canon )

Edit: seem incomplete , doesn't show the resources[] for compression but the over all resources etc. ... need to run a fa-capture test myself see if I can extract the info needed .

FA_CaptureTestImage does not compress anything, so the resources it uses are not relevant for lossless compression. You need to log a CR2 still image - that includes the capture process (the analog stuff followed by getting uncompressed image data into memory), lossless encoding (the one you are interested in), JPEG preview (also interesting for MJPEG), quick review (unsure if that's done by decoding the JPEG or otherwise, but this suggests it might be the former), creating the CR2 (exif info, secondary images it may contain etc), saving the file, updating the file catalog and who knows what else.

However, FA_CaptureTestImage is a lot easier for studying (it's a lot simpler than a complete photo capture process, which I don't fully understand yet) and even works in QEMU on a few models. The full capture process doesn't work yet, but I've got the data necessary to implement it ("only" a 12MB log to interpret and match the emulation against). It's the second log from io_trace_full.

For the 5D2 the black level is 1792

I remember it was 1024 at some point, maybe at certain settings (either x1 or x5), or was that just in photo mode? The only MLV sample with black=1024 I could find was a FRSP one from 5D2. Tried some old 500D builds, including the oldest nightly available, but couldn't re-create.

Looking through the old code, noticed the 5D2/50D had a workaround to prevent pink previews in x5 zoom, but the 500D requires it too (and works out of the box - with grayscale preview while recording in x5 mode, just like 5D2 used to be in the old days).

[dfort]

For 70D, it might be worth trying just 0xC0F37074/78, 0xC0F37300/3e8 and also the rabbit (without 0xC0F37290/94).

So this is our next 70D test:

modules/silent/lossless.c

Code: [Select]

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
    }


0xC0F13068 is called RABBIT.

Test build on my downloads page.

@andy kh - The ball is on your side of the court.

[andy kh]

@andy kh

Your black DNG isn't a Full-res silent lossless, it is a simple silent. Still black though.

According to exiftool your uncompressed FRSP is 5648x3720 and the lossless DNG you uploaded is 1984x1251.

I'm not saying you did anything wrong--it might be what the code is doing but please make sure you are in photo mode, LiveView on and double check your settings:

i was wrong. FRSL dont work. it doesnt save the dng in the camera. im gona do the test of your latest build now

[andy kh]

with your latest build i get this FRSP https://files.fm/u/3kc5sbt if i activate mlv lite, recording button nad trast button dont response

[dfort]

That link does not contain any files.

Don't turn on mlv_lite yet. Let's concentrate on just the silent module. Turn all other modules off. Well, you could try to see if you can view the files in camera like a1ex suggested but let's try to keep the variables down to a minimum.

Are you able to get simple silent lossless DNG's?

[dfort]

Took another look at the 7D and figured out where it is getting stuck:

modules/silent/silent.c

Code: [Select]

    ASSERT(out_suite->size == max_compressed_size);

    out_raw_info.frame_size = lossless_compress_raw(&out_raw_info, out_suite); // This is where the 7D is getting stuck

    if (out_raw_info.frame_size > out_suite->size)

Of course I haven't figured out how to get it unstuck.

Had some problems building the crop_rec_4k branch on the 7D. At first I thought it was a Digic 4 thing but it turned out to be an issue with the 7D.MASTER.203 platform. Made a comment on the Patch manager pull request because that's where I was able to track down the issue.

Opening the console shows some issues but I doubt this is related to lossless compression because it also shows up if it is set to uncompressed DNG. Still, something seems to be out of whack here.



This is in photo mode with just the silent module active and set to simple silent lossless DNG.

[EDIT] Tried various values here but kept getting those error messages:

src/raw.c

Code: [Select]

#if defined(LV_STATE)  /* older Digic 4 */
#define BLACK_LEVEL 1024
#endif

[EDIT 2] Did another test using the unified build. It also gave a similar error message but it was just a single message as I shot multiple simple silent DNG's.

[andy kh]

i can view in the camera like i can view in my pc. lossless FRSP https://files.fm/u/66fxm8ny
now
simple silent losssles pic https://files.fm/u/cywkwa4

edit : they are all  lossless. DNG FRSP works

[reddeercity]

Oh Boy lot of stuff going on , Thanks @a1ex for info & direction -- there so much going on that it's very easy to get off track and go in circles .

You mean mlv_rec? It should work as long as you don't use the reduced bit depth settings.

Yes , Full mlv +audio , yes I can record in 10-12 bit in 3xcrop_mode & 1:1 with some frame sync issue, but not corrupted
I was going to say as good as the old 2016/dec. build but better , Yes 3xcrop is flawless @ 10bit/24p 2144x1076
1:1 is much better and 14bit works as expected -- I'm not worried about that right now of course Lossless FRSP is the priority right now .

You need to log a CR2 still image - that includes the capture process (the analog stuff followed by getting uncompressed image data into memory), lossless encoding (the one you are interested in), JPEG preview (also interesting for MJPEG), quick review (unsure if that's done by decoding the JPEG or otherwise, but this suggests it might be the former), creating the CR2 (exif info, secondary images it may contain etc), saving the file, updating the file catalog and who knows what else.

I did do a CR2 still with the startup log(the one's I posted a few pages back) but it doesn't print out the resoures[] like that FA_Capture_Test_log
It just give the Clock out

Code: [Select]

02E5E> ShootCaptu:ff87ffec:93:03: GetCaptureResources start.
02ED4> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
02F11> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
.......... 27 more clocksave out
0338D> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
033BA> ShootCaptu:ff880004:93:03: GetCaptureResources end.


However, FA_CaptureTestImage is a lot easier for studying (it's a lot simpler than a complete photo capture process, which I don't fully understand yet) and even works in QEMU on a few models. The full capture process doesn't work yet, but I've got the data necessary to implement it ("only" a 12MB log to interpret and match the emulation against). It's the second log from io_trace_full.

That's where I got part of the resources[] , I did run & compile the very simple io_trace for the 5D2 -- thou it did save any logs , I must of messed up some where .

I remember it was 1024 at some point, maybe at certain settings (either x1 or x5), or was that just in photo mode? The only MLV sample with black=1024 I could find was a FRSP one from 5D2. Tried some old 500D builds, including the oldest nightly available, but couldn't re-create.

Yes the original old raw from 2013  "1023" -- from the summer of 2013 3x Crop mode 2048x1024 extracted with the old raw dump of the time (raw2dng) I think  . (I have lots of old .raw file from2013 if needed , let me know. I also have the 2013 raw video build for 5d2 archive if needed .)
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/000001.dng
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/ExifTool_metadata_000001.dng.txt

Code: [Select]

Black Level                     : 1023
White Level                     : 15000

[a1ex]

it doesn't print out the resoures[] like that FA_Capture_Test_log

If you capture the two logs with the same code, it should list the resources in both cases. In any case, you will need LockEngineResources_log enabled (it is, by default, for 5D2, but not at startup). You could also enable CreateResLockEntry_log at startup, as the "lossless" resources are initialized when the camera starts.

Yes the original old raw from 2013  "1023" -- from the summer of 2013 3x Crop mode 2048x1024 extracted with the old raw dump of the time (raw2dng) I think

Right; mind running a "hg bisect" to find out when the black level changed? This trick is very useful to track all sorts of bugs, where you know it worked in some older version, but some reason is no longer working in recent ones and you have no idea where to start looking.

You could start with something like:

Code: [Select]

cd platform/5D2.212
hg bisect --reset
hg up fa84643 -C # some old changeset from 2013; still compiles cleanly here
make clean; make
# make sure the black level is about 1024 with this build
hg bisect --good

hg up unified -C # current unified
make clean; make
# make sure the black level is about 1728 in the same mode
hg bisect --bad

# test intermediate builds suggested by hg, and answer with --good or --bad
# you will need to test ~ 12 builds to find the changeset that resulted in different black level


[reddeercity]

thanks , sure I'll do that tomorrow .

[andy kh]

there is a problem with the previous website so im sharing some more simple lossless again
https://expirebox.com/filesgroup/6faf08bb7e162083a8345f5e06a34781.html

[dfort]

DNG FRSP works

First lossless DNG from the 70D--Yay!



Well, it is a start.

It is Full-res, compressed and it looks like what we were getting out of the 6D just a few days ago. Simple silent lossless DNG's are still coming out black but the size of the file looks about right and the metadata is intact.

Code: [Select]

Image Size                      : 1984x1251

Really? Did you shoot the simple silent with the camera set in photo mode? Try switching the camera to movie mode and shoot simple silent DNG's with the Canon menu set to 1920x1080 and 1280x720. Zoom mode also works on that camera, right? Just want to check if you are getting black DNG's at all settings because on the 700D some settings would continue working even though we were pushing the height beyond what it could handle.

[a1ex]

The current samples do not answer one important question: is there any difference between all these 70D builds?

Same test scene and same exposure settings for all the builds you have tested, please (including the build from esas). Both simple and full-res, uncompressed and compressed => 4 samples for each build. Error message (screenshot if possible) if some particular setting with some particular build doesn't work.

Full-res silent pictures only work in photo mode, btw.