Messages

Data IDs:

0x0000 - KEY
0x1001 - Camera Model
0x1002 - Caption
0x1003 - Remain
0x1004 - Gnz Gamma
0x1005 - Gnz LUT
0x1006 - Lucky Data
0x1007 - Angel Regs
0x1008 - Wk Data
0x1009 - PC set Selected ID
0x100a - Copyright
0x100b - Lucky Enable
0x100c - Lucky Valid
0x100d - Site
0x1010 - Partial Info
0x1011 - Made In Canon
0x1012 - Partial Info Base
0x1018 - RGB Gamma
0x1019 - LAB Gamma
0x1020 - ICC (sRGB)
0x1021 - 3x20 Matrix (sRGB)
0x1022 - Lucky Table (sRGB)
0x1023 - Partial Info (sRGB)
0x1028 - ICC (AdobeRGB)
0x1029 - 3x20 Matrix (AdobeRGB)
0x102a - Lucky Table (AdobeRGB)
0x102b - Partial Info (AdobeRGB)
0x1030 - Sharpness
0x1031 - Contrast
0x1032 - Saturation
0x1033 - Color Tone
0x1050 - LUT Long (sRGB)
0x1051 - LUT Long (AdobeRGB)
0x1052 - LUT Double (sRGB)
0x1053 - LUT Double (AdobeRGB)
0x1054 - User RGB Gamma
0x1055 - RGB Gamma From Camera
0x1060 - Base PS Gamma ID
0x1061 - Defined Color Partial Info (sRGB)
0x1062 - Defined Color Partial Info (AdobeRGB)
0x1063 - Canon Internal Partial Info (sRGB)
0x1064 - Canon Internal Partial Info (AdobeRGB)
0x1065 - User RGB Gamma After Canon
0x1070 - LUT 3D (sRGB)
0x1071 - LUT 3D (AdobeRGB)
0x1080 - Disable Edit
0x1f00 - Camera Lucky Table (sRGB)
0x1f01 - Camera Lucky Table (AdobeRGB)
0x2001 - Sub Copyright
0x210 - Picture Style LUT Param
0x2fff - Sub Terminator
0x8000 - Backup Flag
0x8001 - PF Version
0xa001 - Work Data
0xa002 - Copyright
0xb001 - Select Object
0xfffe - Finalize
0xffff - Terminator


Data Type:
1 - Byte
2 - ASCII
3 - Short
4 - Long
0xffff - Unknown


Type:
0x2e4d524b - MRK
0x2e504632 - PF2
0x2e504644 - PFD
0x2e505345 - PSE
0x2e544344 - TCD
0x2e574244 - WBD


Picture Styles:
0x21 - User1
0x22 - User2
0x23 - User3
0x41 - PC1
0x42 - PC2
0x43 - PC3
0x81 - Standard
0x82 - Portrait
0x83 - Landscape
0x84 - Neutral
0x85 - Faithful
0x86 - Monochrome
0x87 - Auto
0x88 - FineDetail

Amazing progress and what a helpful tool. Works great over here. Now what to make of the hex info left in there, hehe 

Great to hear it's working for you. I'd suggest starting out by taking a pf3, load it in the Style Editor, make a single change, save it with a new name, then compare decoded versions of each to see what changed. Rinse and repeat, documenting any interesting changes you find. Once you start getting an understanding of the file format, try editing the decoded version appropriately then re-encode it and see if the change takes effect as expected in the Style Editor. I wrote the decoder with this workflow in mind so you can decode a file, edit it, then re-encode it quickly/easily. If you have any suggestions for improvements though, let me know.

One thing to try - toggle the byte that is about 20 bytes from the end of the file (just before the FF FE pair) from a 00 to a 01. I suspect that one enables/disables the edit protection.

The first firmware update is available for 5D4:

http://www.canon.de/support/consumer_products/products/cameras/digital_slr/eos-5d-mark-iv.aspx?type=firmware&firmwaredetailid=tcm:83-1475064

Here's a little Java app that'll covert a pf3 file between its encoded and unencoded forms:

https://www.dropbox.com/s/4x8x9epalybm6aw/Pf3Decoder.jar?dl=0

To use, just run:

java -jar Pf3Decoder.jar input-file [output-file]

Note that a jar file is basically a .zip file. The source code is included inside the jar.

If the input file is encoded, Pf3Decoder will create an unencoded copy. If the input file is decoded, Pf3Decoder will create an encoded copy.
If you don't specify an output file, Pf3Decoder will create an output file with the same name but with a ".decoded" extension. If the input file already has a ".decoded" extension however, it'll create a file without the ".decoded" extension (ie it'll write out a file with the original encoded filename). Be aware that any existing output file with the same name will be overwritten without warning.

@chris_overseas !big! kudos!
Do you have a encoding key extracted? Pieces? maybe its worth looking inside a canon-firmware for this key. a binary search. the longer the better.

I don't have the key but it should be trivial to get by XORing the encoded vs decoded files (ignoring the first 0x0C bytes). No need to search the binaries or firmware - it's not stored in its raw value in there anyway.

[Edit:]

I've made a bit of progress on solving the XOR encoding. Here's a zip containing the three log picture styles, plus another random one I made. For each, I've also included a copy that (hopefully!) has the XOR encoding removed but should otherwise be identical. Currently it's a fairly manual process to do this but I'll try to automate it when I have time (probably won't be for a few days at least). I'll also try to figure out what the layout of the files are in some detail. In the meantime, maybe the rest of you can figure out some of the file contents from what I have here?

https://www.dropbox.com/s/1xt92z62fosrsri/Decoded-Styles.zip?dl=0

Notes:

• The entire pf3 file is XORed apart from the first 12 bytes which are left as-is.
• I think the XOR operation has a period of 0x200 (512 bytes), though haven't confirmed that 100% yet.
  Edit: turns out this is not true, we'll either need to apply a formula based on two separate keys, or have one huge key (512*513 = 260k)
• The decoded files won't open in Picture Style Editor, they're just for examining with a hex editor.

I've only just installed Canon's Picture Style Editor a few minutes ago but may have figured out how to disable the protection on pf3 files. On the handful of files I've tried at least, changing byte 0x0006A122 from a 0x31 to a 0x30 seems to allow it to be edited, without any obvious side effects. Based on that and some of the other behaviour I've seen too, I don't think there's a checksum on the file.

Can someone try changing that byte on a pf3 file of theirs and let me know if it enables editing? Also, are pf3 files always 434,487 bytes in size?

@Danne, @DeafEyeJedi: The XOR operator (usually represented by the ^ symbol) is its own inverse. That is, if X ^ Y = Z, then it is also true that X = Y ^ Z and Y = X ^ Z. If you know any two of the values you can always figure out the third, but if you only have one value you can't determine either of the other two. This property means it can be used as a sort of "poor man's encryption", where e.g. X = unencrypted data, Y = encryption key, Z = encrypted data.

Taking @jkaura's example, we know all of Z (the encrypted file) and some of X (the original data), so we can compute some of Y (the key) using Z ^ X = Y:

580F 2FC2 552F 2E76 D23D ^ 4141 4141 4141 4141 4141 = 194E 6E83 146E 6F37 937C
5B0C 2CC1 562C 2D75 D13E ^ 4242 4242 4242 4242 4242 = 194E 6E83 146E 6F37 937C

Because the results (key fragments) are the same in both cases, it implies the key is a constant. So now the puzzle is to figure out the rest of the key. It might be hard-coded in the encoding/decoding software, or it might be generated using a formula. If it's hard-coded it should be relatively easy to find by scanning the binaries, if it's a formula then some reverse-engineering of the encoding/decoding software (or extracting more fragments of the key using @jkaura's approach) is probably required.

You don't need Magic Lantern for this, you can copy files via "Image Copy" in the (blue) Canon menus.

Looks like stereoscopic images are possible, with quite good results: https://www.dpreview.com/forums/post/58326573

5D4 teardown: https://www.lensrentals.com/blog/2016/09/lensrentals-canon-5d-mk-iv-teardown/

Dual Pixel RAW is certainly looking intriguing. I'm curious to know if the hardware can control the ISO of the A and B pixels separately, opening up the possibility of an improved type of dual ISO. [Edit: I guess this is unlikely, given dual ISO works by alternating ISO every second row rather than by column]

There's a few new pieces of info about the 5D4 in general in this interview:

http://www.birdsasart-blog.com/2016/09/03/canon-5d-mark-iv-eos-1dx-mark-ii-and-5ds-r-top-canon-tech-rep-rudy-winston-answers-my-questions-and-yours/

Interesting to see that using UHS-II SD cards might give worse performance than UHS-I.

Please note the build is untested, so you should probably wait for feedback from early testers before upgrading.

Thanks a1ex. So far so good with this build. ML menus all work fine, and I haven't noticed any problems with what I've tried (briefly) so far: dual-ISO, auto ETTR, advanced bracketing, silent pictures, various global draw features. I haven't tried any video features.

5D4 specs are here: http://www.canonrumors.com/full-specifications-images-of-eos-5d-mark-iv/

Note that it's possible the SD slot (with SDXC) is going to be the faster than the CF slot. We don't know the full spec of that SD slot yet but SDXC v4 with UHS-II is rated at up to 312MB/s.

I'll probably be picking one of these up, but not for another couple of months once there's good availability and the prices drop slightly.

The 5D3 resolution is 5760x3840 and RAW files are (very roughly) 27MB or so each.

A 5D4 at 30.4MP would mean a resolution something like 6750x4500, with RAW files around say 37MB each.

The 5D3 can shoot photos at 6fps, which requires 6 x 27 = ~160MB/s. The 5D3 CF card slot can only write at around 100MB/s though, which is why the onboard RAW buffer fills up (and the fps slows down) after a few seconds of continuous shooting.

Even assuming the 5D4 can write at the maximum theoretical speed of 160MB/s to CF card, at 7fps x 37MB= ~260MB/s the RAW buffer will still fill up (and fps will slow down) quite quickly. Presumably the buffer will be big enough that the 5D4 is capable of 7fps continuous for at least as long as the 5D3, but I wouldn't expect too much beyond that.

As far as continuous RAW video goes, 1920x1080x25fps requires ~87MB/s which is close to the upper limit of the 5D3. Even in the best case scenario on the 5D4 (160MB/s writes), it doesn't look like it would be capable of 1920x1080x50fps continuous but would still have fairly noticeable headroom over the 5D3. Only time will tell what the real numbers are.

Without following this too closely, it sounds to me like your camera runs the old 2014Aug02 build just fine but hangs/crashes when booting the newer 2016May16 version. If so, try this (might take a while...):

• Get 2014Aug02 installed and working again.
• Turn off camera, remove battery and card.
• Put card in PC. Delete ML directory and autoexec.bin from card.
• Copy 2016May16 ML directory and autoexec.bin onto card (don't format or anything else).
• Replace battery, insert card, turn on camera. Presumably it will fail to start again.
• Now go back and repeat from step 2, except try a build from e.g. mid 2015 at step 3.
• If the build works, go back to step 2 and try a newer build (e.g. from late 2015). If it fails, try an older build (e.g. from early 2015).

Once you narrow down exactly which build introduces the problem it should be easier for a developer to figure out what's wrong.

Where am I going wrong?  I need the data to appear with the image.

I'm guessing you installed the stable version of the plugin? Try using the beta version of the LR plugin, it works well and has a lot of improvements over the stable version. The stable version is quite a bit older and likely has an old version of cr2hdr that doesn't transfer the EXIF across for you. Alternatively, you can just replace the version of the cr2hdr executable that is bundled with a newer version.

Stubs for 5D3 1.2.3 (untested):

Code Select Expand

NSTUB(0xFF3CCC14, ProcessPathForFurikake)
NSTUB(0xFF57CA1C, PACK16_Setup)

Edit: I just Googled Furikake. Interesting, though not sure what it has to do with a camera! https://en.wikipedia.org/wiki/Furikake

It's probably more to do with stupid import duty laws than anything: http://www.tested.com/tech/photography/44445-why-digital-cameras-have-a-30-minute-video-recording-limit/

This sounds really promising, an up to date and comprehensive user guide will be a hugely useful contribution. One thing I'm curious about - am I right in thinking you're taking actual photos of the back of the camera for each menu, rather than using the built in screenshot functionality?

I read somewhere that 1.3.3 deliberately attempts to block ML? I wondering if that has any implications on a 1.3.3 version ML?

That was just unfounded speculation from people without much knowledge of the situation, based on the new check in 1.3.3 to prevent a straightforward downgrade to lower firmware versions. There is no deliberate attempt to block ML.

I made an initial port to 1.3.3 just over a year ago. It was fairly straightforward to get up and running initially and for the most part seemed to work fine. There was some flickering in the menus that was beyond my skill to resolve though so it's waiting on someone else to pick up where I left off before it's ready for a wider audience. Given the tiny differences between 1.2.3 and 1.3.3 I can't imagine finishing the 1.3.3 port is very high on the priority list however.

Here's my repo with the relevant changes:

https://bitbucket.org/chris_miller/magic-lantern/branch/5D3-133

My pull request (though I've since broken this when I recreated my own bitbucket repo above):

https://bitbucket.org/hudson/magic-lantern/pull-requests/605/support-for-5d3-firmware-133/diff

Note that the Android version has much more functionality than the iOS version (so perhaps consider picking up a cheap compatible Android phone or tablet - just be careful which one you get), but that version hasn't been updated in a long time either. I suspect it's because Chainfire (the developer) wasn't seeing much return on investment, especially given the tricky and ongoing compatibility issues he was facing with APIs, USB ports and cables, and various other limitations he was hitting with the phone hardware and software. He has several other more mainstream mobile apps that are most likely much more lucrative, so if he has indeed lost interest in developing DSLR Controller much further I can't say I blame him, sad as it is. Having said that, the Android version does just about everything you'd want it to anyway.

Thanks a bunch for this! A Cygwin/MinGW environment something I've been meaning to try to get set up for a while now because developing/building from an old 32 bit Linux VM (my current setup) is a pain at times. When I get a chance I'll follow your instructions and let you know if I hit any problems along the way. In the meantime though, it looks like your commands to run setup-x86.exe and setup-x86_64.exe should be prefixed with the user's home directory, eg:

Code Select Expand

%USERPROFILE%\Downloads\setup-x86.exe -K http://cygwinports.org/ports.gpg

Thanks again, hopefully I can give some better feedback soon.

It sounds like the maximum theoretical values are +/-127 (as per http://www.magiclantern.fm/forum/index.php?topic=4648.msg27254#msg27254), though I doubt that's ever been tested. As far as I know you're the first person that's ever needed to go beyond 100.

Yes the 14 bits is per channel. The thing to bear in mind is that a RAW file is not really an image in the same sense as a JPG or TIFF file that has RGB pixel values that you can view directly on screen. The RAW file contains Bayer patterned data straight from the sensor where the intensity of each point corresponds to what was read from each photosensor. The sensor has very different characteristics to the human eye (linear vs logarithmic being one of them), so it's not feasible to map directly from the 14 bits RAW to 16 bits TIFF and expect results that will look sane.

To translate the data in a RAW file into something that you can view on screen a lot of processing has to be done. Choice of white and black points, demosaicing, gamma correction, white balance, colourspace conversion, noise reduction, sharpening, tone curve adjustments and so on. Virtually all of these operations lose some information along the way (eg clipping shadows or highlights), and/or suffer from rounding errors due to the conversion from one bit depth to another. Of course once the information is lost there's no getting it back, hence why the choice of profile can make such a big difference to the amount of data retained in the initial RGB image data you get after the conversion.