Messages

@dcw1979, see this thread where we investigated pretty thoroughly: https://www.magiclantern.fm/forum/index.php?topic=16299

The answer is that it's theoretically possible, and we did reverse engineer and start documenting the .pf2/.pf3 file format, but that development stalled when it came to developing tools to make use of the info. It's still on my back burner, so if you feel technically inclined it'd be great for someone to dig in with a fresh pair of eyes!

Interesting video, @deafeyejedi. Is there a particular reason you were changing the input space around so much? Was it just to experiment with random settings?

I saved a basic file with no adjustments from Canon Picture Style Editor and made my changes in a hex-editor.

That could be due to the "Finalize" tag, which acts as a checksum. It's possible that some cameras can ignore it. If you change data in a hex editor without modifying the Finalize block, then the checksum is incorrect and PSE will tell you there's something wrong with the file if you try to load it. But, I found that Canon's downloadable styles don't even include Finalize, and if you remove it with a hex editor and load the file into PSE it will both load correctly and when saved PSE will handle generating a correct tag for you.

I think it's easiest to just remove it, but the latest .JAR version of the encoding script posted earlier in this thread includes the correct formula for generating a valid Finalize tag.

As for the 3D LUTs, it's certain to be very powerful but I don't know anything about how to properly edit them yet.

I'm not sure if it's a problem or intentional, but I can't find any way to be notified of new posts in threads I've participated in. (I only use the forum through a web browser, not the Tapatalk app)

I can see the notifications tab in the modify profile page, and I've turned notifications on, but the section for "Current Topic Notifications" is blank. The SMF help pages suggest that I can click a Notify button on any thread I want, but it doesn't appear to be visible anywhere on the Magic Lantern forums.

Second 'problem' - are profile pictures limited to Hero users? I've tried adding a link through modify profile before but it never actually saves.

The PF3-format has an adavantage here. There you can define a additional RGB-curve that you can use for this purpose. Although I can create "valid files" that are accepted by EOS Utility and the camera, I can't see the effect of the applied curve (5D Mark III) in the live view/resulting image. If I use the exact same file in Canon Digital Professionel or my other cam EOS 5Ds (s!) it works. This is very strange!

That is strange - did you make those files by hex edit or through the Canon Picture Style Editor software?

Does anyone has some information about the PF3-format and can tell me why these files are so huge compared to a PF2-file?

I haven't dug all the way in but from what I've seen it's just down to the existence of many more tags available in the pf3 format. For instance, in PSE, the extra Sharpness settings, Tone curve (RGB), and the entire Six-color axes tab are not saved in pf2 format. I believe the minimum size for pf3 comes in around 450kb. For instance, I saved a custom pf3 out of PSE with default settings, and the resulting file had 3D LUTs inside for both sRGB and AdobeRGB that each took up 216kb of data.

I though the base picture style is neutral.

But also, is this still all done on top of canons film curve, or is that actually hidden away in the picture style?

The Base Picture Style is determined by the 0x1009 tag ("Base Picture Style ID") in the file, with at least nine known styles available. If you open up the PSE program it's the first dropdown option in the Basic tab. I think @andy600 would know better about whether the Canon film curve is involved, as I'm sure he would have dealt with that in his Cinelog work.

I can at least answer Q2 - You don't need to add the OCIO effect if you're rendering out Cinelog-C masters. You can find a description of the correct render output settings in the Cinelog Quick Start guide.

@agentirons

If you have some time you can update the specification of PF2.

In the "10 18 00 04   RGB Gamma [long]" block you can enter 12 points for each RGB-channel. The channel order is definitely R, G, B. I have tested this and reproduced curves from Photoshop. Works like charme. The values are altered independently of each channel.

Sure thing, sorry I didn't copy that from the Cinestyle tab into the main tab. It's the same for L*a*b* Gamma too.

For everyone's reference, the way it works is that for each channel, the first set of 4 bytes (ie 00 00 00 02) indicates how many points there are in the curve, and then each pair of 8 bytes following corresponds to an input and output for that point, used to calculate transformations from incoming data. So for instance if there are only 2 curve points, then the first set of I/O corresponds to black level and the next set corresponds to white level, and the remaining values in that channel are left blank. If there are three points, then first is black, second is your custom point, third is white, and the rest are blank. Strangely, the max seems to be twelve even though there's enough space in each data block for 15 points. So, each channel within the data block will always end with at least 24 bytes of zeros.

I think the biggest thing to figure out right now is how much manipulation is possible using the picture styles, and whether the base picture style (ie Landscape) can be negated somehow so that we can effectively apply curves directly to the raw data.

My project that I've been poking away at is a Javascript-based style editor, using Electron as a cross-platform app container. You can drop in a file, see all the data as hex values with tag names as you see in the spreadsheet, and edit individual values for saving back out. At this point I have to get the edit and save parts working, then I'll post a link. With something like this it will at least be much easier for people to see what they're editing while we test custom values out.

@350D that's my end goal. I think a handful of us are working separately on teasing apart the code in various ways and building some tools.

Okay, thank you very much for the additional clarifications. I've edited the image again based on how I'm understanding it now, although I've got additional questions:

1) Is the image data white balanced during the demosaic, or is it a separate process?

2) Is the demosaic what transforms the data into Canon's 6000 colorspace?

3) In order to apply the Picture Style LUTs in L*a*b* colorspace, is it necessary to transform the data from either 6000 or XYZ into L*a*b* and then do the Picture Style, or does the LUT handle this as a single step?

4) Is there a final conversion from either XYZ or 6000 into sRGB or Adobe1998, or does the initial 3x20 matrix do this? It seems like there must a final conversion since the latter two are their own unique colorspaces, and if it's the initial matrix then it would be like having the image be both XYZ and sRGB simultaneously, right?

Thanks, @eduperez, I forgot about the jpg thumbnails, so I added that step.

You're right that it'd seem like you'd want to downsample near the very end of the process, but our investigation in the 'Reverse Engineering Picture Styles' thread has revealed that the picture style process is 8-bit as far as we can see (pending some further testing). The downsample might be misplaced in this graphic by a block or two but should be in the right area.

Fascinating little detail I picked up by examining Canon's Downloadable Picture Styles - You can actually delete the Edit Lock, Finalize (Checksum), and Base Picture Style settings (sharpness, contrast, saturation, color tone) blocks entirely!

For some reason Canon omitted all three of these sections in the Nostalgia, Clear, Twilight, Emerald, and Autumn Hues styles.

For Studio Portrait, Snapshot Portrait, and Video Camera X Series Look, they kept Finalize and the Base Settings but omitted the edit lock.

	EDIT LOCK		FINALIZE		BASE SETTINGS
NOSTALGIA
CLEAR
TWILIGHT
EMERALD
AUTUMN_HUES
P-STUDIO			X		X
P-SNAPSHOT			X		X
VIDEO-X			X		X

[(EDIT 10/7: Fixed step order)]

Picture Styles are applied to the demosaiced, color balanced, normalized image i.e. Picture Styles are added very late in the chain of processing - too late for an effective log conversion.

Can you tell me if the following flowchart is correct? You can click on the image to go to the Google Drawing and edit it if you'd like, I've got a backup. I tried to read back through this thread and the Reverse Engineering thread to assemble the actual order of operations, but I'm sure I'm missing a few steps or I've got some things wrong:

(EDIT 10/7: Fixed step order)
Google Drawing

Right, so, .pf2 has an RGB Gamma curve and a L*a*b* Gamma curve that both work, but only the L*a*b* Gamma curve is directly editable inside Canon's Picture Style Editor software (it's the curve box in the 'Specific Colors' tab). The RGB Gamma curve has to be edited manually using the hex editor, and I'm guessing only one is applied to the image if you have data in both curves.

PSE can edit the RGB Gamma curve if you save as .pf3. (Assuming here, I haven't actually tried it to confirm that's what's happening.)

The separate and much larger 'User' RGB Gamma block that is present in the Cinestyle .pf2 has seemingly no effect, and @Andy600 is trying to figure out it's purpose.

Wow, great info! Thank you for all that research. I made it clear in the Google Sheet that the three parts of the LAB block are L*, a*, and b*, I don't know why that didn't occur to me. Have you been able to confirm if the same three parts in the RGB Gamma block are R, G, and B? Cinestyle uses the same curve for all three so it wasn't immediately obvious. Same with the Matrices, I'd love to know if they are actually broken up the way I colored them in the sheet.

As far as increasing the bit depth of the gamma curves, increasing the file length shouldn't be an issue (esp once chris adds that to the decoder, for now you can do it by hand), so I suppose it's just a matter of trying it out. You'd also need to change the block length tag to match your edits, and there's no obvious tag between components in the gamma blocks so it may of course be a hard limit.

I can also recommend Hex Fiend for OSX as a free and simple editor, with a very useful comparison tool: http://ridiculousfish.com/hexfiend/

[EDIT: Specifically the curve in the 'Specific Colors' tab, not the Tone Curve (RGB) box in the 'Basic' tab.]

The lower left corner point and upper right corner point (black and white levels) in PSE are points 1 and 12, with a max of ten additional points in between.

If you take a look at the spreadsheet in the LAB Gamma block, thats what gets adjusted when you edit points in PSE. EDIT: Specifically the curve in the 'Specific Colors' tab, not the Tone Curve (RGB) box in the 'Basic' tab. You can see by the notes that it goes input level then output level (in hex) for each point in order, followed by zeroes if you have less than 12 points. So if you know the numbers you want for each point, just convert them to hex and edit that block, then use the latest Java coder to convert back to pf2 with a correct checksum.

Okay, that's really interesting. Did you make this by swapping in your own tone curve into the Cinestyle pf2? I noticed it's still got the Cinestyle User RGB Gamma block. I'm also curious about the Edit Lock and Site tags - they got changed from 0x10800004 and 0x100d0002 to 0x0000FFFE and 0x93B3FFFF, with seemingly no ill effects? Maybe PSE simply ignored those blocks since the one before Edit Lock specifies a length of 4 bytes, and then the next block has a proper tag.

I've updated the Pf3Decoder app so it now automatically corrects the checksum when processing a pf2/pf3 file with the ".decoded" extension. This means you should be able to decode a pf2/pf3 file, make any changes you wish with a hex editor, then re-encode it without worrying about the checksum being invalidated.

https://www.dropbox.com/s/4x8x9epalybm6aw/Pf3Decoder.jar?dl=0

This hasn't had very thorough testing so please let me know if you hit any problems.

Thank you! I'm on my phone so I haven't checked it out yet, but does it also modify the file length value at the beginning of the file?

The RGB gamma blocks seem to provide an underlying s-curve which is likely to be the inverse of the s-curve used by the Neutral profile i.e. the curve you are always fighting in PSE.

Applying the inverse = Neutral linear (don't confuse linear in this context with scene-referred linear).

So if I'm understanding that right, then the DIGIC processor as well as DPP are first applying the "RGB Gamma" (0x1018) curve, which inverses the curve of the Base Picture Style, "Neutral", and then the "User RGB Gamma" curve is applied after to turn the incoming data into more or less Cineon Log?

The User RGB Gamma block is curious to me because it doesn't exist in any other pf2 files I've looked at from Canon or those made in PSE, and it clearly offers a much finer level of detail than the twelve point curve you get in the RGB or LAB gamma blocks.

Assuming that order of operations is right, and also that we could create our own User Gamma block, then it would appear the key to adding any truly custom picture style is to use some of the data blocks to cancel out the base Canon style and then use the other blocks to apply the desired curves.

Of course at the moment nothing is testable until someone can figure out how the final checksum is generated. I think I was slightly off when I said it was an LRC, it seems maybe more likely that it's CRC-32 since it's a 32-bit hex string, and I've seen at least 24-bits of it used. I'm not very familiar with the topic, but hopefully @chmee or @chris_overseas could shed some light on it. It's gotta just be a matter of figuring out how much of the file is treated as the message to encode by feeding different sections into a CRC-32 algorithm until the output matches the checksum in the file.

Once that's done I feel like we've got pf2 successfully reverse engineered and can easily create software to read/write. Then onto pf3, I guess!

Alright everyone, here's the pf2 format guide in Google Sheets format, including the annotated Technicolor Cinestyle:

https://docs.google.com/spreadsheets/d/1ePeeBpraX7AlM_pbBOTILNGQ9q0G_YQZGSEJYYFA1w8/edit?usp=sharing

I've set it to view only while I investigate on my own but feel free to duplicate and make changes. There are two tabs at the bottom, first is a breakdown of the basic Canon styles, second tab is the full Cinestyle hex data.

Probably most interesting is the Cinestyle "User RGB Gamma" block (0x1054 0004), which is a 4096 byte(!) block that I can only assume is the actual log curve in all it's glory.

Technicolor rearranged their blocks compared to Canon and nixed a few like "Partial Info (sRGB)" and "WK Data", whatever those are. I'd love more info from @chris_overseas about what some of these tags mean, like "Site" or "Remain".

[EDIT: The 4 bytes after the 0x50535000 tag are actually the file length in bytes, not including the 12byte header.]

Spent a lot of time poring through the data last night, just for pf2 files. I think I can confirm that those bytes near the end are some kind of checksum, with the property ID being 0xfffe - Finalize (type 0x0004 ~ Long) according to @chris_overseas's list. There's also a similar 32bit checksum just for copyright field data that appears at the beginning of the file after the 0x50535000 tag.
EDIT: The 4 bytes after the 0x50535000 tag are actually the file length in bytes, not including the 12byte header.

@jkaura noticed that 0x1009 is specifically Base Picture Style - Technicolor Cinestyle lists 0x0084 as it's value, because it's based on Neutral. I've tried putting in values higher than 86, since my latest OSX PSE software doesn't even list Monochrome as an option, but of course that checksum prevents me from re-encoding the file successfully without knowing how those values are generated.

EDIT: Oh, it's called a Longitudinal redundancy check. Working on the formula.

In charting out the files, it seems that the tags go:
0x1009 0003 = Tag name followed by data type
0x0000 0004 = The actual length of the following data block (ie 04 = 4 bytes. The 3x20 color matrices are F0 = 240 bytes)
0x0000 0001 = Actual data, like 32bit Saturation Value

The caption field under propID 0x1002 is always a fixed 32 byte length, consisting of 31 character bytes plus a 00 ending byte.

The copyright field under propID 0x100A is a variable byte length, up to a max of 64 bytes, consisting of n character bytes plus a 00 ending byte.

Cinestyle actually matches the Canon structure closer than I thought, it's just that some of the tables and matrices are in a different order than Canon's.

Slowly going through Canon's built-in styles and comparing setting changes to breakdown the binary data.

It seems that the 1 or 2 bytes that are 9 bytes in from the end of the pf2 files (just before FF FF FF FF 00 00 00 00) are some kind of integrity check. If you save the same settings twice you'll get identical data, but if you change anything you'll see a flag change near the top but also those integrity bytes change, and not in a 1:1 relationship. It's like the aggregate of your changes are somehow summed into those two bytes, like "Change two settings by 1 = A4", "Change three settings by 1 = A2", but also "Change three completely different settings by 1 also = A2". That includes incrementing a number in your caption field.

Thanks, @chris_overseas, I'm not as familiar with Java or C so it took me a little while to translate but I finally got it down.

Here's the working python script version of @chris_overseas's Java app for anybody who finds it useful:
http://pastebin.com/2xhBcgkT
(Make sure to copy the raw script into your own text file. If you hit the download button and try to run the file directly on OSX, you'll get errors because pastebin added '/r' newline characters to the script for some reason.)

Hey, @chmee, I'm trying to translate your php script into python, but I seem to have a bug somewhere as the file output doesn't match chris_overseas's java script. Would you be able to take a look and tell me if I'm doing the byteArray wrong or something?

If I can get it to work I can probably whip up a better OSX app for decoding/encoding, then we'll have a good start later on for quick and easy LUT->Picture Style conversion.

http://pastebin.com/ft2LkvYB

Another interesting quirk in PSE - If you open a Canon style and then save out as a new file, the subsequent pf2 contains a copy of both the original style (with tiny modifications) and your adjustments. This doesn't happen if you open your own file and save a new one.

If you decode a double-saved Canon style and compare the two blocks of data inside, the only difference is that the second byte in that start tag becomes 90 in the original block, and stays as 10 in the new block. You can also see that this flag is repeated four times throughout each data block, because the only difference is 90 vs 10. The second block of data will also begin with your new comment/copyright fields.

I made a quick and dirty OSX Automator app that works with Chris's java file, so if you put pfcoder.app and Pf3Decoder.jar in ~/Desktop/PSE, you can add the app to your dock and drag/drop pf2/3 files onto the icon and it will auto run the shell script

Code Select Expand

java -jar Pf3Decoder.jar input-file (no output specified, so it will overwrite stuff). Lot faster than having to revise the terminal command every time you want to decode or encode.

https://drive.google.com/file/d/0B2q7De8nh2q7SzJ4OXp5ZXlKejA/view?usp=sharing

I believe I've found some kind of start tag in .pf2 files for the LUT data - "00 10 21 00 04 00 00 00 F0 00", which is always followed by "00" or "01" and then arbitrary data. This holds true for Canon's styles available through their website, Cinestyle, and styles created through PSE.

In PSE styles, that tag starts immediately after the copyright field no matter how long that field is, and the last byte is 01. In Canon's styles it appears after a middle block of mostly empty bytes, and ends with 00. In Cinestyle it's near the copyright field and ends in 01 like PSE, but the data doesn't start immediately like PSE and Canon's.