ProcessTwoInTwoOutLosslessPath

[David_Hugh]

Hey you wizards! Happy to serve with my limited knowledge and a 70D at hand. With the latest crop_rec_4k.2018Mar10.70D112.zip build which I downloaded from @dfort 's bitbucket page, I get an Error message when I try to load either one of the Raw modules. Camera in Manual Photo Mode, iso 1250, shutter 1/50, Picture saving style Raw only (i.e. NOT raw +jpeg), Lens was a Sigma art 18-35,if that makes any difference.
Here's the screenshot

[dfort]

@David_Hugh - the only module that we are testing at this time is the silent module. Turn everything else off--we know that the modules you loaded don't work yet.

What we're looking for at this time is regular and lossless DNG's shot in every video mode this camera supports, mv720, mv1080 and 5x zoom and also Full-resolution silent pictures.

There is an earlier build with an incomplete set of DNG's. That build is located here:

https://bitbucket.org/es_as/magic-lantern/downloads/magiclantern-crop_rec_4k_70D.2018Jan11.70D112.zip

[David_Hugh]

Alright! Sorry, I thought you guys were already talking full-on video mode. Let's see if I can be a little more helpful this time around (Although, looking at the Dngs - all broken - I'm sure I probably f***** up something...) . Here are the 720/1080/5xZoom/FRSP Dngs, normal and lossless. Wasn't able to save a compressed FRSP, the error message is attached instead.

https://we.tl/OYDFhjVE3O

[David_Hugh]

Oh, and this was for the march 10th build by the way. If this is any help at all (in other words, if I'm not one of the inviduals too stupid to test because I dont get all the settings right - see previous post    ) I'll test the January build tomorrow, it's late where I'm at.

[a1ex]

Results are, indeed, a bit unexpected, but the test was done correctly (exactly what I asked for). Will double-check tomorrow in QEMU.

In particular, the "card full" error is quite surprising. I'd expect this on the build from esas (before this was fixed), but I'd expect either success or some other error message on the March 10 build. Mind double-checking the build version? (Help menu, and also on silent.mo after pressing Q).

[dfort]

Yes, very unexpected. I've been working with andy kh trying to get a full set of DNG files and his files from the March 10 build are very different. I've only got FRSP for that build and simple silent for the esas version but I'll keep updating this as he continues testing.

https://www.dropbox.com/sh/ba278pyafw9hlze/AACUrAyC9pWv8CnpOzXnlpqFa?dl=0

[David_Hugh]

Yes, you were both right. Build info in the help menu said it's esas build from January (although I'm not exactly sure how that happenend because I definitely updated the version, why else would the raw modules be "broken"). Anyway. Reinstalled the March 10 version (the right one this time - 100% sure, looked it up in help pre + post test). So now here's the Dng Set for this test, but they're not really looking any different and I still cant open them in Lightroom. On the plus side, as you predicted, I was able to save a compressed FRSP this time.

https://we.tl/GGVBUAU8kU

[a1ex]

Thanks, this time it matches my expectations.

This was the same issue we had on 6D, and was solved by this block, but on 70D the registers are different.

I'd like the same test with the other register tweaks:

Code Select Expand


    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
    }


The March 10 build includes just the first 3.

An easier task that can be done with any recent build: full set of VRAM dumps.

[dfort]

New build posted - Mar13

@David_Hugh - Could you please double check this:

https://www.magiclantern.fm/forum/index.php?topic=5601.msg198375#msg198375

The instructions on how to run the test is on Reply #1579 of the 12-bit (and 10-bit) RAW video development discussion.

[David_Hugh]

Here's the DNG set for March 13th build.

https://we.tl/AyWvpX82yu


As for the memory test, am I correct in assuming that I need to run the allocate-raw-lv-buffer build from your download page? If that's all there is to it (plus the instructions on the 10/12 bit thread) I'll do this later in the evening (got some stuff to do now..)

[dfort]

As for the memory test, am I correct in assuming that I need to run the allocate-raw-lv-buffer build from your download page?

Yes, that's right. The build is still posted. Thanks!

[a1ex]

Sorry, cannot compare with previous results, as the exposure is way different.

Please repeat with both March builds on the same test scene, if possible.

[David_Hugh]

Ok! Will do! Here are the screenshots from the alloc-raw-lv-buffer test in the meantime!
https://we.tl/tGatqZnFOh

I did:
photo mode, right after startup
720 mode, right after startup
1080 mode, right after startup
Zoom mode (with digital zoom in the menu selected), right after startup
Zoom mode  - the "normal" way by pressing the magnyfing button, but that's obviously not how the cam starts up

[a1ex]

- open the console (Debug menu) and enable something that uses LiveView RAW features (raw video, raw histogram etc)

Maybe I should have written these on two different lines. Your histogram was YUV-based in all the screenshots; no RAW-based features were enabled.

RAW-based histogram looks like this, with vertical bars at each EV.

[David_Hugh]

Right... Just for future dummy proofing this test: you can select histogram RAW based without having loaded the mlv_rec or other raw modules.

What follows: Histogram is visible, Dumbest Assumable User (in this case me) thinks - hey! histogram raw based selected, everything fine, lets do the test!

Here's the same test with raw stuff enabled:
https://we.tl/2q0izHvPBv

0=photo mode normal startup
1=photo mode live view
2=720p selected at startup
3=1080 selected at startup
4= digital zoom enabled at startup via canon menu

[a1ex]

Right; there is a PR that addresses this issue, need to revisit it.

Just tried running this test in QEMU, after a very recent breakthrough - couple of hours ago - and got 4B328000-4D7FFFF0. First address matches

[David_Hugh]

That's great news! Here's the repeated DNG test by the way. Same exposure settings shooting the same "target" for March1oth and 13th builds.

https://we.tl/0Qklp88OxJ

[a1ex]

Looks identical to me, so probably 0xC0F37074/78 don't have any effect.

Long shot: what about enabling the 6D/650D code block for 70D? After all, today seems to be our lucky day...

[dfort]

I made three new test builds for the 70D and put them on my downloads page. I believe these are all combinations that we haven't tried yet. Let's see if this really is a lucky day.

modules/silent/lossless.c

test A

Code Select Expand

    /* configure the processing modules */
    /* configure the processing modules */
    TTL_Prepare(TTL_ResLock, &TTL_Args);

    if (is_camera("6D", "*") ||
        is_camera("650D", "*") ||
        is_camera("70D", "*"))
    {
        /* not sure what exactly these do, but they seem to be required to get correct image
         * taken from register log diffs: http://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987 */
        EngDrvOut(0xC0F37610, 0);       /* 0x11 on 650D, 0 on all other D5 (not sure if needed) */
        EngDrvOut(0xC0F37628, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3762C, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37630, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37634, 0x71000); /* 72000 on 6D and 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3763C, 0);       /* 0x1000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37640, 0);       /* 0x2000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37644, 0);       /* 0x4000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37648, 0);       /* 0x8000000 on 6D and 650D, 0 on all other D5 */
    }

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
    }
    else if (is_camera("DIGIC", "5"))
    {
        /* all other D5 models use this register instead */
        /* the hardware encoder (and other image processing modules that might be used)
         * expect slice width and slice height for these registers
         * Canon configuration: slice width = image width / 2, slice height = image height
         * our configuration:   slice width = image width, slice height = image height / 2 */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F on 5D3 */
    }

    if (is_camera("DIGIC", "5"))
    {
        /* required for full-res silent pictures? */
        /* this one doesn't seem to refer to slice size, but to total image size */
        /* sht_mirror shows 0xC0F13000 - 0xC0F13064 as RABBIT */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
    }


test B

Code Select Expand

    /* configure the processing modules */
    TTL_Prepare(TTL_ResLock, &TTL_Args);

    if (is_camera("6D", "*") ||
        is_camera("650D", "*") ||
        is_camera("70D", "*"))
    {
        /* not sure what exactly these do, but they seem to be required to get correct image
         * taken from register log diffs: http://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987 */
        EngDrvOut(0xC0F37610, 0);       /* 0x11 on 650D, 0 on all other D5 (not sure if needed) */
        EngDrvOut(0xC0F37628, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3762C, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37630, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37634, 0x71000); /* 72000 on 6D and 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3763C, 0);       /* 0x1000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37640, 0);       /* 0x2000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37644, 0);       /* 0x4000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37648, 0);       /* 0x8000000 on 6D and 650D, 0 on all other D5 */
    }

    else if (is_camera("DIGIC", "5"))
    {
        /* all other D5 models use this register instead */
        /* the hardware encoder (and other image processing modules that might be used)
         * expect slice width and slice height for these registers
         * Canon configuration: slice width = image width / 2, slice height = image height
         * our configuration:   slice width = image width, slice height = image height / 2 */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F on 5D3 */
    }

    if (is_camera("DIGIC", "5"))
    {
        /* required for full-res silent pictures? */
        /* this one doesn't seem to refer to slice size, but to total image size */
        /* sht_mirror shows 0xC0F13000 - 0xC0F13064 as RABBIT */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
    }


test C

Code Select Expand

    /* configure the processing modules */
    TTL_Prepare(TTL_ResLock, &TTL_Args);

    if (is_camera("6D", "*") ||
        is_camera("650D", "*") ||
        is_camera("70D", "*"))
    {
        /* not sure what exactly these do, but they seem to be required to get correct image
         * taken from register log diffs: http://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987 */
        EngDrvOut(0xC0F37610, 0);       /* 0x11 on 650D, 0 on all other D5 (not sure if needed) */
        EngDrvOut(0xC0F37628, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3762C, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37630, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37634, 0x71000); /* 72000 on 6D and 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3763C, 0);       /* 0x1000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37640, 0);       /* 0x2000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37644, 0);       /* 0x4000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37648, 0);       /* 0x8000000 on 6D and 650D, 0 on all other D5 */
    }

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
    }
    else if (is_camera("DIGIC", "5"))
    {
        /* all other D5 models use this register instead */
        /* the hardware encoder (and other image processing modules that might be used)
         * expect slice width and slice height for these registers
         * Canon configuration: slice width = image width / 2, slice height = image height
         * our configuration:   slice width = image width, slice height = image height / 2 */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F on 5D3 */
    }

    if (is_camera("DIGIC", "5"))
    {
        /* required for full-res silent pictures? */
        /* this one doesn't seem to refer to slice size, but to total image size */
        /* sht_mirror shows 0xC0F13000 - 0xC0F13064 as RABBIT */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
    }


[David_Hugh]

Out of luck for the day as it seems... none of the 3 tests shows any sign of improvment as far as I can tell (which hopefully doesnt mean much since I know 0 about how all of this works )

Moreover, they dont seem any different from each other, all 3 tests, or rather all 5 tests look pretty much the same.


https://we.tl/1uKL17PkrO

[dfort]

Actually, there are significant differences between them. Now it is a matter of sorting them out and figuring out what to try out next.

[reddeercity]

Not sure if this is going to fly

Code Select Expand

if (is_camera("5D2", "*"))
    {
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2    - 1)); 
        EngDrvOut(0xC0F12020, PACK32(width    - 1,  height/10 - 1));  /* 0x18A0127 */
    }


I'm finding the 5D3 113 is very similar to 5D2 , most of the 0xC0f***** below match up in the 5d2 disassembly with the same strings
I haven't try it yet , looking for some feedback -- It's late right now I'll try it tomorrow

Code Select Expand

{
        /* values for 5D3 - just for future reference */
        EngDrvOut(0xC0F12010,        width    - 1                 );  /* 0xB8F     */
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F1201C,        width/10 - 1                 );  /* 0x127     */
        EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  /* 0x18A0127 */
    }

[dfort]

Not sure if this is going to fly

Well there's only one way to find out, right?

Those 0xC0f***** addresses aren't necessarily something that can be determined from the disassembly. Of course we have to use whatever tools we have available whether it be IDA, ARMu, QEMU, just a simple text editor or whatever else you have in your toolbox.

[a1ex]

A and C look identical to me, B is worse (has diagonal artifacts in addition to black level issues).

B is missing 0xC0F37300/4 compared to A, so these are important. The test just confirms our previous hypothesis, but doesn't give any hints on how to solve the black level issues.

To summarize:
- the artifact we have solved is this or this (using 0xC0F37300/4)
- the artifacts we have yet to solve are this (vertical split, possibly pointing to slice configuration) and this (black level issue, solved on 6D by that big block we don't know what exactly it does)
- the relevant log diff (6D vs 70D) is this

[dfort]

That was an interesting experiment that didn't pan out so I took down the ABC builds and put up a build from the current crop_rec_4k in the main repository. As usual it is on my downloads page but this time I'm adding the commit number in the file name.

This is the build that has been tested rather extensively by two users, David_Hugh and andy kh. I'm seeing some significant differences between the two sets of tests.

David_Hugh FRSP


David_Hugh FRSP lossless


andy kh FRSP


andy kh FRSP lossless


I examined the metadata with exiftool and the notable difference that I found was that David had his camera set on ISO 640 while Andy used ISO 3200.

The difference with simple silent is even more dramatic:

David_Hugh 5x zoom


David_Hugh 5x zoom lossless


andy kh 5x zoom


andy kh 5x zoom lossless


Looking at the log diff there seems to be other things we can try. Any suggestions?