Canon 6D

[Danne]

I am talking about the black level tag only. Not the offset tags.  Don't know if those can be changed in post?

[bad-and-ugly]

Hi everyone
My 6D wouldn't turn on and I got it to work again by following these instructions

bad shutdown
Switch camera Off, remove: battery, card and lens.
Reattach lens, close card slot, reinsert battery and switch camera On again

It won't turn on with the ML memory card though, so I had to fix it a second time.

My question is: do I have to give up ML then? Has anyone been through this and found a way of using it? What could I be doing wrong?

Thank you for your time [=

[Walter Schulz]

Make sure to wait some time after "power-down" and after opening card-slot before removing card. ML writes data to card *after* screens went dark.

For defective ML card:
Format in cardreader, then format in cam and reinstall ML from scratch.

Do you have troubles with this card only? Try another one.

[jhayes2118]

Hi,

I'm having a problem after installing and uninstalling ML. the rotating dial on the back of the camera won't respond to left turns. It still works for right turns. I.E. If I try to change the aperture to a lower number, it doesn't work. If I change it to a higher number it works. Very Frustrating. Anyone have any ideas on how that might have happened (other than you know using experimental firmware that is not approved by the manufacturer (: )

Any ideas on how to fix that?

Thanks
jhayes2118

[Levas]

Are there people here who can check if the addresses below, for lossless compression, are right for the 6d ?
I've searched the rom for the addresses, they are are certainly in there, but are they the right ones ?
Changed a few off them, but all changes I tried, made the camera freeze before saving a silent lossless dng 

if (is_camera("6D", "1.1.6"))
    {
        /* ProcessTwoInTwoOutLosslessPath, 6D 1.1.6 */
        TTL_SetArgs     = (void *) 0xFF3491C8;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF4129BC;      /* called right after ProcessTwoInTwoOutJpegPath(R) Start(%d); */
                                             /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF411A44;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF359C78;      /* called next; starts the EDmac transfers */
        TTL_Start       = (void *) 0xFF412A2C;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF412A64;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF412A9C;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */

[dfort]

Those addresses should work. The way they were found was by checking disassemblies of various similar cameras. (Digic V)

[Levas]

Found some debug stuff in raw.c and enabled it:

And there are a few things that are wrong
raw2lv -> (-6, -3) - (726,483)
should be (0,0) - (720,480)
raw2bm -> (-6, -3) - (726,483)
should be (0,0) - (720,480)
bm2raw -> (94,36) - (1895,1243)
should be (80,28) - (1910,1252)

And of course, the black check is horribly wrong.

Anybody here who can help to get this right in raw.c for the 6d ?

[Levas]

Got one more screenshot, now with 5x zoom mode.


Doesn't look good to me 

[a1ex]

The initial hypothesis (when I wrote that test) was that the (newly found at that time) raw buffer should match the LiveView buffer, including its corners. Later, it turned out not to be true, so that test is no longer exact (but if you ignore the "roundoff" errors, it's still useful).

For 1x, look in raw_set_geometry - I believe there is an offset of 14 Bayer pixels skipped by Canon code when creating the preview. The test should be updated to account for this offset (e.g. raw_info.active_area.x1 would become raw_info.active_area.x1 + 14), after confirming that offset is correct for other cameras (I've only tested 5D3, and keep in mind my pixel peeping skills aren't the best).

For 5x, the raw buffer contains a lot more than what's displayed (guess why you can record 2...3K in this mode), so that test will display huge errors. In this case, the test would be more difficult to write (somebody has to sit down and do the math); easiest way - it should probably not be interpreted in this mode.

A better check would be to look at raw zebras - are they aligned with Canon's preview? (in all modes)

Outside LiveView, they are computed for every displayed pixel (on the BMP overlay). There, you should see pixel-perfect alignment with the analyzed image.

However, the raw zebras are quite low-res in LiveView - computed every 8 BMP pixels horizontally, so they won't align very well because of this. You could change the code in draw_zebras_raw_lv to operate at byte level (uint8_t instead of uint64_t, plus other adjustments) - it will be slower, but better for checking alignment. There is also zebra_highlight_raw_advanced (used by Auto ETTR when selecting Show metered areas). That one is computed for every BMP pixel, so it might be a better way to check the alignment between raw and LV buffers.

If you like easy coding tasks: what about adding some sort of alignment test in raw_diag? You could reuse RAW_ZEBRA_TEST or the existing zebra drawing functions from zebra.c, and display things in a way suitable for judging alignment (maybe a checkerboard pattern displaying either Canon or ML rendering, alternating between them to make any misalignment obvious).


Regarding your earlier issue - have you tried typing the error message, or the tag you are trying to change, in a search engine?
( advice from http://xkcd.com/627/ )

Code: [Select]

exiftool "Warning: Not enough values specified (2 required)"
BlackLevelRepeatDim site:magiclantern.fm
per channel black level


[Levas]

Regarding your earlier issue - have you tried typing the error message, or the tag you are trying to change, in a search engine?
( advice from http://xkcd.com/627/ )

Code: [Select]

exiftool "Warning: Not enough values specified (2 required)"
BlackLevelRepeatDim site:magiclantern.fm
per channel black level


Already tried googling exiftool and the tag and 2 values, multiple values, altering the tag etc.
But I didn't google the exact error message...now I see, it's a string.

So I tried some other values for BlackLevelRepeatDim...can't crack the math to get normal dng 
All 4 color channels have a different offset and I can only change 2 values in BlackLevelRepeatDim.

Furthermore, I can't get DM-SPY stuff working, too complicated.
@DFort, maybe you can help make a build, with DM-SPY and lossless silent pic according to these instructions from Alex?
http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168

[a1ex]

All 4 color channels have a different offset and I can only change 2 values in BlackLevelRepeatDim.

Of course, you have a 2x2 Bayer pattern, not a 4D matrix.

Hint: you should read two pages from the DNG spec, not just one

[Levas]

I hate it when you give just about enough information for hours of googling and trial and error and not enough information to solve the problem at once 

Aaarggghhh !!! 

But I’ll try cracking the code 

Just wondering, do you already know where the problem is, with the 6d.
Only thing left where I think is the problem is in this piece of code in lossless.

else if (is_camera("5D3", "*") || is_camera("6D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x30001,    /* Read connection 1 (uncompressed input) */
            0x2002d,    /* Write connection 45 (compressed output) */
          //0x20016,    /* Write connection 22 (for WR2 - not used) */
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x230000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };

What’s up with all the addresses without commends, do you know where they are for ?

[a1ex]

That information is enough to solve the problem, if you actually read those two pages from the DNG spec and put in practice (in exiftool's command line) what you have read.

What we know about ResLock is documented at http://www.magiclantern.fm/forum/index.php?topic=6740.0 and http://www.magiclantern.fm/forum/index.php?topic=18443.msg182989#msg182989

The others are unknown; they can be found by understanding other image processing paths and cross-checking the numbers (that's how the known ones were found). It's what Canon code uses for CR2, and they are not the same in all cameras. Why there are differences - complete mystery.

Short answer: I have no idea where the black level difference comes from. It's likely from encoder configuration, but where exactly... no idea. It's not from our raw backend, for sure. I also doubt it's from these resources (if these are wrong, it either doesn't work, or locks up, or locks out other stuff from LiveView, or locks out unused stuff with no obvious side effects).

However, the defect is easy to work around it - hence that exiftool puzzle. Per-channel black level in DNG is a trick I know from this forum (others described it some years ago).

The defect on 650D is even harder to understand, but that one is likely from resolution-related registers (maybe the ones I'm overriding, maybe others). Other models were just lucky (they happened to work without much tinkering).

[Levas]

Read the information on page 26 and 27 of the adobe dng 1.4 spec over and over again...can't see the light.
Time to sleep


But as I understand correct, you're saying:
Be happy, the 6d has working lossless already, all the data is there, it just need some fiddling with exiftool for the moment, but all the correct data is in the file.
 

[dfort]

@DFort, maybe you can help make a build, with DM-SPY and lossless silent pic according to these instructions from Alex?
http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168

It can get tricky merging the dm-spy-experiments branch. I tried a few different ways but couldn't get it to compile a working silent module--much less with the 6D changes. There have been lots of updates since those instructions were posted back in April using the compressed_raw branch.

I'll update you if I can get something to work but I've also got a question--are you able to save full-res silent picture with lossless compression without requiring my hack pull request? Yeah, got to figure out a better method because this will never be accepted into the main repository.

[Audionut]

I hate it when you give just about enough information for hours of googling and trial and error and not enough information to solve the problem at once 

I feel your pain.  There's something to be said for not being spoon fed though.
If only there were 25 or maybe 35 hours in a day.   

I followed the instructions and got a working build with working silent.mo, but no lossless compression. 

[Danne]

Checking this post:
http://www.magiclantern.fm/forum/index.php?topic=15088.msg191860#msg191860

I see 4 black level numbers to fix the image:
1919
1791
1536
1024

In the dng file I ony see one black level tag stored:

Alright, so the defect appears to be a per-channel offset. Let's check the two images that are supposed to be identical - will look at the 4 Bayer channel differences (R, G1, G2, B):

http://www.magiclantern.fm/forum/index.php?topic=15088.msg191858#msg191858

And from:

Code: [Select]

However, the defect is easy to work around it - hence that exiftool puzzle. Per-channel black level in DNG is a trick I know from this forum (others described it some years ago).http://www.magiclantern.fm/forum/index.php?topic=15088.msg193240#msg193240

Ok, so alterning black level tag goes like this:
Example:

Code: [Select]

exiftool "-IFD0:BlackLevel=1024" Input.dngTesting this gives various results but seems there is more to it.

adding the BlackLevelRepeatDim:

Code: [Select]

exiftool "-IFD0:BlackLevelRepeatDim=1 1" Input.dngDoesn't appear to change anything.

So what i boils down to is "Per-channel black level in DNG"?

Some additional info:
http://www.openphotographyforums.com/forums/showthread.php?t=12280

[Levas]

@Danne, ah I see you're also trying to solve the puzzle, that's good, because I can't solve it 
Others are free to join to solve this puzzle 

I'm not sure, but I think there are 4 tags at play here:
BlackLevel
BlackLevelRepeatDim
BlackLevelDeltaV
BlackLevelDeltaH

All described in on page 26 to 28 in adobe dng 1.4 spec:
https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf

I've read the pages over and over again, but I'm still not sure how black levels are calculated here.
But I think, you can set a global base black level with BlackLevel
And a matrix that's repeated with BlackLevelRepeatDim
With BlackLevelDeltaH you can put an offset per column, in the matrix described with BlackLevelRepeatDim
With BlackLevelDeltaV you can put an offset per row, in the matrix described with BlackLevelRepeatDim

We have 4 channels, Red, Green1, Green2, Blue.
But how to setup the matrix ?
4 by 1, so we get the 4 channels in one row with 4 columns.
R | G1 | G2 | B
Or do we have to set it up as 2 by 2 , So we get two rows with two columns.
R   | G1
G2 | B
Or maybe 1 by 4 will work too ?
R
G1
G2
B

A lossless dng straight out off the 6d has a global black level of 2047:
The per channel offset has to be altered in a raw editor as follows:
R = -128 -> Effective black level of 2047 -128 = 1919
G1 = -256 -> Effective black level of 2047 -256 = 1791
G2 = -512 -> Effective black level of 2047 -512 = 1535
B = -1024 -> Effective black level of 2047 -1024 = 1024

Now we need to setup the 4 tags so that the above black levels per channel are in the dng file.
 
Maybe things get easier if we use a global black level of 1024, that case, the blue offset black level is 0
black level offset would become:
R = 896
G1 = 768
G2 = 512
B = 0

Anybody, ideas for the correct values for the following tags?
BlackLevel
BlackLevelRepeatDim
BlackLevelDeltaV
BlackLevelDeltaH

[Levas]

Got a great idea, took an original CR2 file and put it through adobe DNG converter.
And now let's check the EXIF data in the DNG 

Some new hints:

  | | 16) BlackLevelRepeatDim = 2 2
  | | 17) BlackLevel = 2047 2047 2048 2048 (524032/256 524032/256 524288/256 524288/256)

Already tried something with the numbers 524032 and 256 with BlackLevelDeltaH and BlackLevelDeltV but no luck in getting the lossless DNG right 

[Levas]

Yeah cracked the code 
 

[Levas]

BlackLevel can be one value, but it can also be a string of values 

So in order to make the lossless dng's from the 6d normal, use the following exiftool commands:

-exiftool -IFD0:BlackLevelRepeatDim='2 2' FILE.DNG
-exiftool -IFD0:BlackLevel='1919 1791 1535 1024' FILE.DNG

[Danne]

Cool. Was thinking along these lines but didn't really bother to test(thought it was a no go).
Great cracking @Levas.

[Levas]

Fiddling now with 11 bit and 12 bit lossless.
Seems to work with the same black levels as 14bit lossless, although white level needs to be changed to 16300 too.
Seems that 11bit/12bit files have 14bit black level in the dng header, but 11/12bit whitelevel in the dng header ?
Maybe it's caused by the MLV_dump I'm using ?

But 11/12bit files look really good, with this command in exiftool
exiftool -IFD0:BlackLevel='1920 1792 1536 1025' -IFD0:BlackLevelRepeatdim='2 2' -IFD0:WhiteLevel=16200 FILE.DNG

If you pixel peep to 200/300% there seems to be a small pattern in the 11/12bit files, maybe because it's not really lossless, but lossy ?

[dfort]

Be happy, the 6d has working lossless already, all the data is there, it just need some fiddling with exiftool for the moment, but all the correct data is in the file.

Is there anything on the pull request that needs to be changed?

I followed the instructions and got a working build with working silent.mo, but no lossless compression. 

Right, following the instructions--I didn't try that yet. Maybe that same pull request has what's needed to get lossless compression working on your modified compressed_raw/dm-spy-experiments build?

It doesn't seem right that the DNG files need to be post processed. If we could get a log file out of that special build would it show where the black levels are shifting around?

[Levas]

I'd like to combine this build of Dfort for 6d lossless raw recording with the MLV_Lite with audio recording:
https://www.magiclantern.fm/forum/index.php?topic=16650.msg192065#msg192065

Trying to combine these two, not successful yet.
I've got  the build of ErwinH, the crop_rec_4k_mlv_lite_snd, from his bitbucket:
https://bitbucket.org/ehoutsma/magic-lantern/src/5f4ed213aaeac50b85ae2578d15e9306c54f9db5/?at=crop_rec_4k_mlv_lite_snd

First, I tried to compile it, as it is, for 6d. But it gives some errors:

In file included from ../../src/exmem.c:3:0:
../../src/exmem.c: In function 'srm_malloc_cbr':
../../src/exmem.c:380:12: error: 'SRM_BUFFER_SIZE' undeclared (first use in this function)
     ASSERT(SRM_BUFFER_SIZE == raw_buffer_size);
            ^
../../src/dryos.h:142:27: note: in definition of macro 'ASSERT'
 #define ASSERT(x) { if (!(x)) { ml_assert_handler(#x, __FILE__, __LINE__, __func__); }}
                           ^
../../src/exmem.c:380:12: note: each undeclared identifier is reported only once for each function it appears in
     ASSERT(SRM_BUFFER_SIZE == raw_buffer_size);
            ^
../../src/dryos.h:142:27: note: in definition of macro 'ASSERT'
 #define ASSERT(x) { if (!(x)) { ml_assert_handler(#x, __FILE__, __LINE__, __func__); }}
                           ^
../../src/exmem.c: In function '_srm_malloc_suite':
../../src/exmem.c:535:66: error: 'SRM_BUFFER_SIZE' undeclared (first use in this function)
                 suite = CreateMemorySuite(srm_buffers.buffer, SRM_BUFFER_SIZE, 0);
                                                                  ^
In file included from ../../src/exmem.c:3:0:
../../src/exmem.c: In function '_srm_free_suite':
../../src/exmem.c:568:24: error: 'SRM_BUFFER_SIZE' undeclared (first use in this function)
         ASSERT(size == SRM_BUFFER_SIZE);
                        ^
../../src/dryos.h:142:27: note: in definition of macro 'ASSERT'
 #define ASSERT(x) { if (!(x)) { ml_assert_handler(#x, __FILE__, __LINE__, __func__); }}
                           ^
../../src/exmem.c: In function '_srm_malloc':
../../src/exmem.c:616:9: error: 'SRM_BUFFER_SIZE' undeclared (first use in this function)
     if (SRM_BUFFER_SIZE < size + 4)
         ^
../../src/exmem.c: In function '_srm_get_max_region':
../../src/exmem.c:650:16: error: 'SRM_BUFFER_SIZE' undeclared (first use in this function)
         return SRM_BUFFER_SIZE;
                ^
../../src/exmem.c: In function '_srm_get_free_space':
../../src/exmem.c:670:16: error: 'SRM_BUFFER_SIZE' undeclared (first use in this function)
         return SRM_BUFFER_SIZE;
                ^
make: *** [exmem.o] Error 1

So I took the MLV_lite.c and the MLV_sound.c from his build, put it in the build I'm using from Dfort.
https://bitbucket.org/daniel_fort/magic-lantern/src/57dc2077089bb18bd4a5d4172cf37c0bc1347909/?at=crop_rec_4k_6D_wip
It will compile, MLV_Lite and MLV_SND modules will load.
But there's no audio menu, to activate it. But maybe it's on by default ?
I'v got a lossless MLV file from MLV_Lite, but mlv_dump doesn't extract a wav file out of it