Canon 6D

[Levas]

Dfort's bitbucket download place, he posted a link to it some posts ago.

Bitbucket downloads page.

[Levas]

Did some testing with the edited mlv_lite... but it actually got worse , some resolutions give MLV's without dng's and other resolutions give frames which are sort of allright. None half corrupted frames though.

But I did follow Dfort's tutorial and looks I can compile myself now 

I downloaded the crop_rec_4k_6D.116 (last updated 18-Oct) branch from Dfort's bitbucket and compiled a 6d build myself.
Did some more investigation on the frames I get.

The frames who have the bottom half corrupted, seem to have a normal upper half. Once black levels are adjusted, the upper half is perfect, with all debayering  options.

The frames that did appear perfect in the first place, aren't that perfect at all, they look reasonable with AMAZE debayering, but they look plain wrong with LMMSE or EAHD

'normal' frame with AMAZE:


'Normal' frame with LMMSE:


'Normal' frame with EAHD:


Corrupted frame with LMMSE:


Corrupted frame with EAHD:


So there's more going on then only wrong black levels...

[ilguercio]

Dfort's bitbucket download place, he posted a link to it some posts ago.

Thank you kindly.
I haven't really followed the ML scene for a long time so i am catching up on all the news.

[Levas]

ah catching up.
In summary, for the 6d, lossless raw recording isn't working yet.
10/12 bit recording on the 6d works perfect, see downloads page and go to experiments.

[ilguercio]

ah catching up.
In summary, for the 6d, lossless raw recording isn't working yet.
10/12 bit recording on the 6d works perfect, see downloads page and go to experiments.

Thanks, i really follow cause i am still quite a geek but my 6D has been spending too much time in a cupboard rather than shooting.
I think i tried 10bit raw and appreciated it.

[dfort]

Removed the mlv_lite-edited version from my downloads page because it only made things worse.

@ilguercio - there are several test builds in my Bitbucket downloads page: https://bitbucket.org/daniel_fort/magic-lantern/downloads/

I'll leave them up for a while so others can experiment but will remove them when I see we've moved past those tests.

@Levas - good to hear you're able to compile. Now that wasn't so hard, was it?

[Danne]

@dfort - Grandfather of compilers. Could you have a go with this if you have the time?
http://www.magiclantern.fm/forum/index.php?topic=9560.msg192172#msg192172

[dfort]

I see you and a few others been working on reviving ML Raw Viewer. Nice! Anyway, different topic so we'll continue the conversation on that thread.

[Levas]

The getting ready to compile, wasn't that hard. Getting the frames right on 6d with lossless is

I'm  a bit stuck here, not sure where to look.
As far as I can tell there are 2 different things going wrong, not sure if they're related to the same source thing ?

1 - The black level values per channel.
2 - Frames that are corrupt or not completely right after correcting for the black levels.

Should I test some more preferred raw type values ?
Other things to try ?

[Audionut]

Hmmm, sf_dump stuff is completed for 6D?

edit:  pm'd a copy to a1ex.

Have new-dryos-task-hooks.2017Oct23.6D116.zip on the card and will test drive as time permits.

[dfort]

Hmmm, sf_dump stuff is completed for 6D?

Could you also comment on the pull request?

[Audionut]

Could you also comment on the pull request?

Threw my two cents in.

[dfort]

Thanks @Audionut -- the serial flash file is needed to run the 6D in QEMU.

[Levas]

@alex and other smart people, I'm out of idea's.

Tried many, many things in lossless.c, still can't get simple silent lossless picture with normal black levels.
Tried different read and write connections, altered or deleted other addresses in the piece of code below, doesn't help.
Or I get a picture with wrong black levels, or no picture at all (camera locks up or doesn't save dng etc.)
 

else if (is_camera("5D3", "*") || is_camera("6D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x30001,    /* Read connection 1 (uncompressed input) */
            0x2002d,    /* Write connection 45 (compressed output) */
          //0x20016,    /* Write connection 22 (for WR2 - not used) */
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x230000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };

Didn't touch this part though, could it be in here ?

    if (is_camera("5D3", "*") || is_camera("6D", "*"))
    {
        /* resolution is hardcoded in some places; patch them */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F13068, PACK32(width*2  - 1,  height/2  - 1));  /* 0xF6D171F */
        EngDrvOut(0xC0F12010,        width    - 1                 );  /* 0xB8F     */
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F1201C,        width/10 - 1                 );  /* 0x127     */
        EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  /* 0x18A0127 */
    }

    /* n

The other specific 6d parts in lossless.c are found in the firmware disassembly by others I've heard, so these parts should't be the problem I guess 

if (is_camera("6D", "1.1.6"))
    {
        /* ProcessTwoInTwoOutLosslessPath, 6D 1.1.6 */
        TTL_SetArgs     = (void *) 0xFF3491C8;
        TTL_Prepare     = (void *) 0xFF4129BC;

        TTL_RegisterCBR = (void *) 0xFF411A44;
        TTL_SetFlags    = (void *) 0xFF359C78;
        TTL_Start       = (void *) 0xFF412A2C;
        TTL_Stop        = (void *) 0xFF412A64;
        TTL_Finish      = (void *) 0xFF412A9C;
    }

if (is_camera("6D", "1.1.6"))
    {
        Setup_DecodeLosslessRawPath = (void *) 0xFF409218;
        Start_DecodeLosslessPath    = (void *) 0xFF4092E0;
        Cleanup_DecodeLosslessPath  = (void *) 0xFF409444;
    }

So to summarise the above: Help please

[dfort]

@Levas - I was going to suggest reading the ProcessTwoInTwoOutLosslessPath topic but I see you were the first person to reply on that discussion so obviously you've been studying this for a while. Still, it would be good to review it now that you have something that is almost working. Make sure to follow those links to the EDMAC internals topic.

...The other specific 6d parts in lossless.c are found in the firmware disassembly by others I've heard, so these parts should't be the problem I guess 

Don't assume we got it right, do a disassembly of your ROM and check it out. It is easy using disassemble.pl posted in the Tutorial: finding stubs topic.

Another thing to do is to compile the dm-spy-experiments branch and shoot a CR2 while logging. This will create a lot of information that you can sift through to see what functions are being called that you can look up in your disassembly. You can also merge branches to see what is going on when you shoot a lossless compressed simple silent DNG. That I can help you with.

It might be possible to run some tests on QEMU. The 6D is one of those cameras that doesn't display the GUI but a1ex posted some hints on how it can be used even without the GUI in the ML on EOS-M2 topic.

This stuff can get overwhelming, believe me I know. It is sort of like that scene from the movie, "The Matrix" :

"You take the blue pill, the story ends. You wake up in your bed and believe whatever you want to believe. You take the red pill, you stay in Wonderland, and I show you how deep the rabbit hole goes."

Ok, kidding aside--

Code: [Select]

    if (is_camera("5D3", "*") || is_camera("6D", "*"))
    {
        /* resolution is hardcoded in some places; patch them */
 ...
You could remove the || is_camera("6D", "*") and see what happens. The other cameras don't need it.

[Levas]

Hi everyone, still alive and kicking, still no results 

Tried some more things.
Removed the 6d part of this piece of code, got weird striped frame:

if (is_camera("5D3", "*") || is_camera("6D", "*"))
{
        /* resolution is hardcoded in some places; patch them */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F13068, PACK32(width*2  - 1,  height/2  - 1));  /* 0xF6D171F */
        EngDrvOut(0xC0F12010,        width    - 1                 );  /* 0xB8F     */
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F1201C,        width/10 - 1                 );  /* 0x127     */
        EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  /* 0x18A0127 */
    }   

Resulting image


After that, tried disabling lines in that piece of code, after many combinations, seems that 6d can do with only the first line, this will fix the striped pattern|(not the black level issue  )
When disabling the first line, and not the others, still stripe pattern as above:

if (is_camera("5D3", "*") || is_camera("6D", "*"))
    {
        /* resolution is hardcoded in some places; patch them */
        EngDrvOut(0xC0F375B4, PACK32(width     - 1,  height/2 - 1 )); /* 0xF6D0B8F */
       
   /* EngDrvOut(0xC0F13068, PACK32(width*2  - 1,  height/2  - 1));  0xF6D171F */
        /* EngDrvOut(0xC0F12010,        width    - 1                 );  0xB8F     */
        /* EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2  - 1));  0xF6D0B8F */
        /* EngDrvOut(0xC0F1201C,        width/10 - 1                 );  0x127     */
        /* EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  0x18A0127 */
    }

Read trough the suggested topics, can't get the DM-spy-experiments to work. Still looking into the code, did find out that raw.c also has very new changes made for detecting the black levels and stuff.
See below, not tested changes to the code yet, trying to not brick my camera this week because of one of my kids birthday is coming 

/* integer gain used to fix the image darkening caused by lv_raw_gain */
/* this gain must not (!) change the raw data */
int _raw_lv_get_iso_post_gain()
{
    if (lv_raw_gain)
    {
        return 4096 / lv_raw_gain;
    }

    return 1;
}

[Danne]

Why not just hardcode black level? It´s only metadata anyway?

[Levas]

As far as I know, the black level is already hardcoded, it's always 2047 (in 14 bit)
The problem the 6d has with lossless compression is black level offset differs per color channel.
The lossless dng's we get from the 6d must be adjusted for black levels as follow:
Red channel = -128
Green channel = -256
Green channel = -512
Blue channel = -1024

Which is rather weird because there's only one black level value in the exif, 2047, which should apply to all channels.
So where do these black level offset differences per channel come from 
I recently find out about the pieces of black level determinations in raw.c, maybe there's going something wrong in there.

[a1ex]

Why not just hardcode black level? It´s only metadata anyway?

+1. Easy to do with exiftool, see DNG spec 1.4 p.26-27 (sorry for the spoiler).

Unfortunately, dcraw and darktable interpret the per-channel black level in some order, while RawTherapee interprets it in some different order. YMMV with other programs. Adobe's dng_validate 1.3 agrees with the former, so RawTherapee folks will have to prepare for a bug report

So where do these black level offset differences per channel come from 

Asked and answered.

[Levas]

Easy to do with exiftool, see DNG spec 1.4 p.26-27 (sorry for the spoiler).

In my case that would be
Easy to do It's possible with exiftool, see DNG spec 1.4 p.26-27 (sorry for the spoiler).

I didn't no there where black level options for row and column, I assumed there's was only one black level in the exif.
Now trying to figure out how to fix this with exif tool

Asked and answered.

The above is very common for clueless people, it means they where not able to get any further with the previous answer they got 


But thanks for the answers, again, much appreciated 

[Danne]

This should work(although I couldn´t get it to work on a file I just tested on):
e.g

Code: [Select]

exiftool "-BlackLevel=2048" *.dng -overwrite_original
Naturally black level numbers will vary. Think a1ex posted them a few posts up...

[Levas]

As expected, I'm already spending 2 hours with this so called "easy to do task" in exiftool and still no luck.
Ofcourse I can change the 'ordinary' black level, but that isn't the problem here.

I'm trying to change the BlackLevelRepeatDim values, but there's none present in the original dng ?
So trying to add BlackLevelRepeatDim values -> error message:
Warning: Not enough values specified (2 required) for IFD0:BlackLevelRepeatDim

I know it needs 2 values, I'm giving it 2 values.
Typed the following in terminal, all giving the above warning that it needs 2 values:

exiftool -IFD0:BlackLevelRepeatDim=1,1 /Volumes/AppleHDD/Lossless-Testing/14bit_2496_Lossless_frame_000000.dng

exiftool -IFD0:BlackLevelRepeatDim=1 1 /Volumes/AppleHDD/Lossless-Testing/14bit_2496_Lossless_frame_000000.dng

exiftool -IFD0:BlackLevelRepeatDim=1 =1 /Volumes/AppleHDD/Lossless-Testing/14bit_2496_Lossless_frame_000000.dng

exiftool -IFD0:BlackLevelRepeatDim=11 /Volumes/AppleHDD/Lossless-Testing/14bit_2496_Lossless_frame_000000.dng

exiftool -IFD0:BlackLevelRepeatDim=1:1 /Volumes/AppleHDD/Lossless-Testing/14bit_2496_Lossless_frame_000000.dng

[dfort]

Wasn't it working on Darktable back on this post?

So the latest is that we should add this for the 6D and possibly 650D?

Code: [Select]

if (is_camera("6D", "*"))
{
        /* resolution is hardcoded in some places; patch them */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));
}

[Levas]

So the latest is that we should add this for the 6D and possibly 650D?

Code: [Select]

if (is_camera("6D", "*"))
{
        /* resolution is hardcoded in some places; patch them */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));
}

For as far as I have tested it, this should be sufficient for the 6d

[Levas]

So about the black level and black level offset, I'm not sure we're all talking about the same here and I'm confused.

The black level value in the lossless dng's from the 6d is fine, and what I've seen always has a value of 2047 for the 14 bit lossless files on the 6d.
But, as we know, there's however a problem with the black level offset per color channel.
Now Alex and Danne talk about hardcoding the black level, are they talking about the normal black level, which is about 2047, or are they talking about hardcoding these black level offset per color channel.
Is it even possible to change the info in a dng to rebalance the black level offset per channel ?

Another problem is, that this black level offset per channel problem, isn't always solvable.
See this post:
http://www.magiclantern.fm/forum/index.php?topic=15088.msg192226#msg192226

There are times that even when compensating for black level offset per channel, there are still yellow and blue pixels in the frame, mostly in the shadows.
I'm not sure if this is, because the blue channel has the biggest offset difference ?
Also, it doesn't work with 10 and 12 bit, when compensating for the black level offset per channel, there's still a raster on the frame, probably because 10/12 bit is lossy and not lossless ?
And if 10/12 bit can't be repaired with compensating for the black level offset per channel, does this mean that the data stream that is put into ProcessTwoInTwoOutLosslessPath
is already wrong, black level offset per channel wrong I mean.