Author Topic: Canon 6D  (Read 475508 times)

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #650 on: November 09, 2017, 09:50:36 PM »
So about the black level and black level offset, I'm not sure we're all talking about the same here and I'm confused.

The black level value in the lossless dng's from the 6d is fine, and what I've seen always has a value of 2047 for the 14 bit lossless files on the 6d.
But, as we know, there's however a problem with the black level offset per color channel.
Now Alex and Danne talk about hardcoding the black level, are they talking about the normal black level, which is about 2047, or are they talking about hardcoding these black level offset per color channel.
Is it even possible to change the info in a dng to rebalance the black level offset per channel ?

Another problem is, that this black level offset per channel problem, isn't always solvable.
See this post:
http://www.magiclantern.fm/forum/index.php?topic=15088.msg192226#msg192226

There are times that even when compensating for black level offset per channel, there are still yellow and blue pixels in the frame, mostly in the shadows.
I'm not sure if this is, because the blue channel has the biggest offset difference ?
Also, it doesn't work with 10 and 12 bit, when compensating for the black level offset per channel, there's still a raster on the frame, probably because 10/12 bit is lossy and not lossless ?
And if 10/12 bit can't be repaired with compensating for the black level offset per channel, does this mean that the data stream that is put into ProcessTwoInTwoOutLosslessPath
is already wrong, black level offset per channel wrong I mean.


Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6262
Re: Canon 6D
« Reply #651 on: November 09, 2017, 09:56:26 PM »
I am talking about the black level tag only. Not the offset tags.  Don't know if those can be changed in post?

bad-and-ugly

  • Just arrived
  • *
  • Posts: 1
Re: Canon 6D
« Reply #652 on: November 10, 2017, 03:00:21 PM »
Hi everyone
My 6D wouldn't turn on and I got it to work again by following these instructions
Quote
bad shutdown
Switch camera Off, remove: battery, card and lens.
Reattach lens, close card slot, reinsert battery and switch camera On again
It won't turn on with the ML memory card though, so I had to fix it a second time.

My question is: do I have to give up ML then? Has anyone been through this and found a way of using it? What could I be doing wrong?

Thank you for your time [=

Walter Schulz

  • Contributor
  • Hero Member
  • *****
  • Posts: 7041
Re: Canon 6D
« Reply #653 on: November 11, 2017, 10:36:29 AM »
Make sure to wait some time after "power-down" and after opening card-slot before removing card. ML writes data to card *after* screens went dark.

For defective ML card:
Format in cardreader, then format in cam and reinstall ML from scratch.

Do you have troubles with this card only? Try another one.
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

jhayes2118

  • Just arrived
  • *
  • Posts: 1
Re: Canon 6D
« Reply #654 on: November 12, 2017, 07:27:27 PM »
Hi,

I'm having a problem after installing and uninstalling ML. the rotating dial on the back of the camera won't respond to left turns. It still works for right turns. I.E. If I try to change the aperture to a lower number, it doesn't work. If I change it to a higher number it works. Very Frustrating. Anyone have any ideas on how that might have happened (other than you know using experimental firmware that is not approved by the manufacturer (: )

Any ideas on how to fix that?

Thanks
jhayes2118

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #655 on: November 12, 2017, 07:30:49 PM »
Are there people here who can check if the addresses below, for lossless compression, are right for the 6d ?
I've searched the rom for the addresses, they are are certainly in there, but are they the right ones ?
Changed a few off them, but all changes I tried, made the camera freeze before saving a silent lossless dng  :P

Quote
if (is_camera("6D", "1.1.6"))
    {
        /* ProcessTwoInTwoOutLosslessPath, 6D 1.1.6 */
        TTL_SetArgs     = (void *) 0xFF3491C8;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF4129BC;      /* called right after ProcessTwoInTwoOutJpegPath(R) Start(%d); */
                                             /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF411A44;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF359C78;      /* called next; starts the EDmac transfers */
        TTL_Start       = (void *) 0xFF412A2C;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF412A64;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF412A9C;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */


dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: Canon 6D
« Reply #656 on: November 12, 2017, 07:44:02 PM »
Those addresses should work. The way they were found was by checking disassemblies of various similar cameras. (Digic V)
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #657 on: November 14, 2017, 06:52:53 PM »
Found some debug stuff in raw.c and enabled it:

And there are a few things that are wrong
raw2lv -> (-6, -3) - (726,483)
should be (0,0) - (720,480)
raw2bm -> (-6, -3) - (726,483)
should be (0,0) - (720,480)
bm2raw -> (94,36) - (1895,1243)
should be (80,28) - (1910,1252)

And of course, the black check is horribly wrong.

Anybody here who can help to get this right in raw.c for the 6d ?


Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #658 on: November 14, 2017, 09:57:35 PM »
Got one more screenshot, now with 5x zoom mode.


Doesn't look good to me  :P

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12357
  • Quarantined
Re: Canon 6D
« Reply #659 on: November 14, 2017, 10:27:11 PM »
The initial hypothesis (when I wrote that test) was that the (newly found at that time) raw buffer should match the LiveView buffer, including its corners. Later, it turned out not to be true, so that test is no longer exact (but if you ignore the "roundoff" errors, it's still useful).

For 1x, look in raw_set_geometry - I believe there is an offset of 14 Bayer pixels skipped by Canon code when creating the preview. The test should be updated to account for this offset (e.g. raw_info.active_area.x1 would become raw_info.active_area.x1 + 14), after confirming that offset is correct for other cameras (I've only tested 5D3, and keep in mind my pixel peeping skills aren't the best).

For 5x, the raw buffer contains a lot more than what's displayed (guess why you can record 2...3K in this mode), so that test will display huge errors. In this case, the test would be more difficult to write (somebody has to sit down and do the math); easiest way - it should probably not be interpreted in this mode.

A better check would be to look at raw zebras - are they aligned with Canon's preview? (in all modes)

Outside LiveView, they are computed for every displayed pixel (on the BMP overlay). There, you should see pixel-perfect alignment with the analyzed image.

However, the raw zebras are quite low-res in LiveView - computed every 8 BMP pixels horizontally, so they won't align very well because of this. You could change the code in draw_zebras_raw_lv to operate at byte level (uint8_t instead of uint64_t, plus other adjustments) - it will be slower, but better for checking alignment. There is also zebra_highlight_raw_advanced (used by Auto ETTR when selecting Show metered areas). That one is computed for every BMP pixel, so it might be a better way to check the alignment between raw and LV buffers.

If you like easy coding tasks: what about adding some sort of alignment test in raw_diag? You could reuse RAW_ZEBRA_TEST or the existing zebra drawing functions from zebra.c, and display things in a way suitable for judging alignment (maybe a checkerboard pattern displaying either Canon or ML rendering, alternating between them to make any misalignment obvious).



Regarding your earlier issue - have you tried typing the error message, or the tag you are trying to change, in a search engine?
( advice from http://xkcd.com/627/ )

Code: [Select]
exiftool "Warning: Not enough values specified (2 required)"
BlackLevelRepeatDim site:magiclantern.fm
per channel black level

;)

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #660 on: November 15, 2017, 07:06:31 PM »

Regarding your earlier issue - have you tried typing the error message, or the tag you are trying to change, in a search engine?
( advice from http://xkcd.com/627/ )

Code: [Select]
exiftool "Warning: Not enough values specified (2 required)"
BlackLevelRepeatDim site:magiclantern.fm
per channel black level

;)

Already tried googling exiftool and the tag and 2 values, multiple values, altering the tag etc.
But I didn't google the exact error message...now I see, it's a string.

So I tried some other values for BlackLevelRepeatDim...can't crack the math to get normal dng  :P
All 4 color channels have a different offset and I can only change 2 values in BlackLevelRepeatDim.

Furthermore, I can't get DM-SPY stuff working, too complicated.
@DFort, maybe you can help make a build, with DM-SPY and lossless silent pic according to these instructions from Alex?
http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168



a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12357
  • Quarantined
Re: Canon 6D
« Reply #661 on: November 15, 2017, 07:42:38 PM »
All 4 color channels have a different offset and I can only change 2 values in BlackLevelRepeatDim.

Of course, you have a 2x2 Bayer pattern, not a 4D matrix.

Hint: you should read two pages from the DNG spec, not just one :D

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #662 on: November 15, 2017, 09:12:51 PM »
I hate it when you give just about enough information for hours of googling and trial and error and not enough information to solve the problem at once  :P

Aaarggghhh !!!  ;D

But I’ll try cracking the code  8)

Just wondering, do you already know where the problem is, with the 6d.
Only thing left where I think is the problem is in this piece of code in lossless.
Quote
else if (is_camera("5D3", "*") || is_camera("6D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x30001,    /* Read connection 1 (uncompressed input) */
            0x2002d,    /* Write connection 45 (compressed output) */
          //0x20016,    /* Write connection 22 (for WR2 - not used) */
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x230000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };

What’s up with all the addresses without commends, do you know where they are for ?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12357
  • Quarantined
Re: Canon 6D
« Reply #663 on: November 15, 2017, 09:29:09 PM »
That information is enough to solve the problem, if you actually read those two pages from the DNG spec and put in practice (in exiftool's command line) what you have read.

What we know about ResLock is documented at http://www.magiclantern.fm/forum/index.php?topic=6740.0 and http://www.magiclantern.fm/forum/index.php?topic=18443.msg182989#msg182989

The others are unknown; they can be found by understanding other image processing paths and cross-checking the numbers (that's how the known ones were found). It's what Canon code uses for CR2, and they are not the same in all cameras. Why there are differences - complete mystery.

Short answer: I have no idea where the black level difference comes from. It's likely from encoder configuration, but where exactly... no idea. It's not from our raw backend, for sure. I also doubt it's from these resources (if these are wrong, it either doesn't work, or locks up, or locks out other stuff from LiveView, or locks out unused stuff with no obvious side effects).

However, the defect is easy to work around it - hence that exiftool puzzle. Per-channel black level in DNG is a trick I know from this forum (others described it some years ago).

The defect on 650D is even harder to understand, but that one is likely from resolution-related registers (maybe the ones I'm overriding, maybe others). Other models were just lucky (they happened to work without much tinkering).

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #664 on: November 15, 2017, 11:13:19 PM »
Read the information on page 26 and 27 of the adobe dng 1.4 spec over and over again...can't see the light.
Time to sleep


But as I understand correct, you're saying:
Be happy, the 6d has working lossless already, all the data is there, it just need some fiddling with exiftool for the moment, but all the correct data is in the file.
 :D


dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: Canon 6D
« Reply #665 on: November 16, 2017, 02:37:40 AM »
@DFort, maybe you can help make a build, with DM-SPY and lossless silent pic according to these instructions from Alex?
http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168

It can get tricky merging the dm-spy-experiments branch. I tried a few different ways but couldn't get it to compile a working silent module--much less with the 6D changes. There have been lots of updates since those instructions were posted back in April using the compressed_raw branch.

I'll update you if I can get something to work but I've also got a question--are you able to save full-res silent picture with lossless compression without requiring my hack pull request? Yeah, got to figure out a better method because this will never be accepted into the main repository.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3584
  • Blunt and to the point
Re: Canon 6D
« Reply #666 on: November 16, 2017, 07:59:39 AM »
I hate it when you give just about enough information for hours of googling and trial and error and not enough information to solve the problem at once  :P

I feel your pain.  There's something to be said for not being spoon fed though.
If only there were 25 or maybe 35 hours in a day.   :D

I followed the instructions and got a working build with working silent.mo, but no lossless compression.  ::)

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6262
Re: Canon 6D
« Reply #667 on: November 16, 2017, 12:56:29 PM »
Checking this post:
http://www.magiclantern.fm/forum/index.php?topic=15088.msg191860#msg191860

I see 4 black level numbers to fix the image:
1919
1791
1536
1024

In the dng file I ony see one black level tag stored:
Quote
Alright, so the defect appears to be a per-channel offset. Let's check the two images that are supposed to be identical - will look at the 4 Bayer channel differences (R, G1, G2, B):
http://www.magiclantern.fm/forum/index.php?topic=15088.msg191858#msg191858

And from:
Code: [Select]
However, the defect is easy to work around it - hence that exiftool puzzle. Per-channel black level in DNG is a trick I know from this forum (others described it some years ago).http://www.magiclantern.fm/forum/index.php?topic=15088.msg193240#msg193240

Ok, so alterning black level tag goes like this:
Example:
Code: [Select]
exiftool "-IFD0:BlackLevel=1024" Input.dngTesting this gives various results but seems there is more to it.

adding the BlackLevelRepeatDim:
Code: [Select]
exiftool "-IFD0:BlackLevelRepeatDim=1 1" Input.dngDoesn't appear to change anything.

So what i boils down to is "Per-channel black level in DNG"?

Some additional info:
http://www.openphotographyforums.com/forums/showthread.php?t=12280

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #668 on: November 16, 2017, 02:03:32 PM »
@Danne, ah I see you're also trying to solve the puzzle, that's good, because I can't solve it  :-\
Others are free to join to solve this puzzle  :D

I'm not sure, but I think there are 4 tags at play here:
BlackLevel
BlackLevelRepeatDim
BlackLevelDeltaV
BlackLevelDeltaH


All described in on page 26 to 28 in adobe dng 1.4 spec:
https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf

I've read the pages over and over again, but I'm still not sure how black levels are calculated here.
But I think, you can set a global base black level with BlackLevel
And a matrix that's repeated with BlackLevelRepeatDim
With BlackLevelDeltaH you can put an offset per column, in the matrix described with BlackLevelRepeatDim
With BlackLevelDeltaV you can put an offset per row, in the matrix described with BlackLevelRepeatDim

We have 4 channels, Red, Green1, Green2, Blue.
But how to setup the matrix ?
4 by 1, so we get the 4 channels in one row with 4 columns.
R | G1 | G2 | B
Or do we have to set it up as 2 by 2 , So we get two rows with two columns.
R   | G1
G2 | B
Or maybe 1 by 4 will work too ?
R
G1
G2
B

A lossless dng straight out off the 6d has a global black level of 2047:
The per channel offset has to be altered in a raw editor as follows:
R = -128 -> Effective black level of 2047 -128 = 1919
G1 = -256 -> Effective black level of 2047 -256 = 1791
G2 = -512 -> Effective black level of 2047 -512 = 1535
B = -1024 -> Effective black level of 2047 -1024 = 1024

Now we need to setup the 4 tags so that the above black levels per channel are in the dng file.
 
Maybe things get easier if we use a global black level of 1024, that case, the blue offset black level is 0
black level offset would become:
R = 896
G1 = 768
G2 = 512
B = 0

Anybody, ideas for the correct values for the following tags?
BlackLevel
BlackLevelRepeatDim
BlackLevelDeltaV
BlackLevelDeltaH

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #669 on: November 16, 2017, 02:43:28 PM »
Got a great idea, took an original CR2 file and put it through adobe DNG converter.
And now let's check the EXIF data in the DNG  :D

Some new hints:

  | | 16) BlackLevelRepeatDim = 2 2
  | | 17) BlackLevel = 2047 2047 2048 2048 (524032/256 524032/256 524288/256 524288/256)

Already tried something with the numbers 524032 and 256 with BlackLevelDeltaH and BlackLevelDeltV but no luck in getting the lossless DNG right  :-\

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #670 on: November 16, 2017, 04:14:11 PM »
Yeah cracked the code  8)
 :D :D :D :D :D :D :D :D

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #671 on: November 16, 2017, 04:18:25 PM »
BlackLevel can be one value, but it can also be a string of values  ;D

So in order to make the lossless dng's from the 6d normal, use the following exiftool commands:

-exiftool -IFD0:BlackLevelRepeatDim='2 2' FILE.DNG
-exiftool -IFD0:BlackLevel='1919 1791 1535 1024' FILE.DNG





Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6262
Re: Canon 6D
« Reply #672 on: November 16, 2017, 04:45:37 PM »
Cool. Was thinking along these lines but didn't really bother to test(thought it was a no go).
Great cracking @Levas.

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1487
  • 6d - Nightly build user
Re: Canon 6D
« Reply #673 on: November 16, 2017, 06:45:45 PM »
Fiddling now with 11 bit and 12 bit lossless.
Seems to work with the same black levels as 14bit lossless, although white level needs to be changed to 16300 too.
Seems that 11bit/12bit files have 14bit black level in the dng header, but 11/12bit whitelevel in the dng header ?
Maybe it's caused by the MLV_dump I'm using ?

But 11/12bit files look really good, with this command in exiftool
exiftool -IFD0:BlackLevel='1920 1792 1536 1025' -IFD0:BlackLevelRepeatdim='2 2' -IFD0:WhiteLevel=16200 FILE.DNG

If you pixel peep to 200/300% there seems to be a small pattern in the 11/12bit files, maybe because it's not really lossless, but lossy ?

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: Canon 6D
« Reply #674 on: November 16, 2017, 07:44:38 PM »
Be happy, the 6d has working lossless already, all the data is there, it just need some fiddling with exiftool for the moment, but all the correct data is in the file.

Is there anything on the pull request that needs to be changed?

I followed the instructions and got a working build with working silent.mo, but no lossless compression.  ::)

Right, following the instructions--I didn't try that yet. Maybe that same pull request has what's needed to get lossless compression working on your modified compressed_raw/dm-spy-experiments build?

It doesn't seem right that the DNG files need to be post processed. If we could get a log file out of that special build would it show where the black levels are shifting around?
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102