Reverse Engineering Picture Styles

Started by dfort, December 07, 2015, 05:50:39 AM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

agentirons

Okay, that's really interesting. Did you make this by swapping in your own tone curve into the Cinestyle pf2? I noticed it's still got the Cinestyle User RGB Gamma block. I'm also curious about the Edit Lock and Site tags - they got changed from 0x10800004 and 0x100d0002 to 0x0000FFFE and 0x93B3FFFF, with seemingly no ill effects? Maybe PSE simply ignored those blocks since the one before Edit Lock specifies a length of 4 bytes, and then the next block has a proper tag.

Andy600

Yes, pretty much. I built a new 12point curve, converted to hex then replaced the RGB gamma blocks. I thought I'd uploaded the one with a linear User RGB Gamma block but here it is: https://s3-us-west-2.amazonaws.com/ofxpublic/Cineon_Log_plus2EV_lin_tone_curve.pf2

You'll see the resulting output is no different than the profile with a log curve in the User RGB gamma block - I'm still trying to work out what that log curve is for? if anything ??? Might it be something exclusively to do with HDMI output? (though I don't know if that's even possible) or is it just a dummy curve? (that would be a shocker :o) because as far as I can tell it does nothing for H.264 or Jpeg.

No ill effects that I can see although the unlock was done before decoding the profile for the curve replacement.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Danne

QuoteI built a new 12point curve, converted to hex then replaced the RGB gamma blocks.
Very good indeed. 12 points? As opposed to the locked 10 points in PSE?

Any additional/simplifying info about how to build your own curve points and inject are most welcome whenever there is time :P

agentirons

The lower left corner point and upper right corner point (black and white levels) in PSE are points 1 and 12, with a max of ten additional points in between.

If you take a look at the spreadsheet in the LAB Gamma block, thats what gets adjusted when you edit points in PSE. EDIT: Specifically the curve in the 'Specific Colors' tab, not the Tone Curve (RGB) box in the 'Basic' tab. You can see by the notes that it goes input level then output level (in hex) for each point in order, followed by zeroes if you have less than 12 points. So if you know the numbers you want for each point, just convert them to hex and edit that block, then use the latest Java coder to convert back to pf2 with a correct checksum.

Andy600

I replaced the RGB gamma blocks but the points will not show in PSE because there is/was no editable Tone Curve in a pf2. Cinestyle is a pf2 and has a tone curve so it's just a PSE omission. The editable Tone curve was added later (presumably under a different tag?) as part of the pf3 format.

If you change the LAB gamma block remember it is in L*a*b* colorspace so changing the first set only will produce a change in luminosity. Changing all three will cause color issues - this is why PSE used to be a complete pain because you couldn't alter each curve independently.

As for building the curves, I did originally try mathematically plotting a known log curve (in this case a Cineon curve plus 2 EV), by converting the expression to an 8bit lut (256 CV) and then took 10 of those values as 'y values' to plot against a 0-255 linear set ('x values'). I then entered the x and y values in PSE and applied the PS to the CR2 in DPP and saved a jpeg. Then I just compared it to the same raw image (a Stouffer transmission wedge) debayered to linear and converted to Cineon plus 2EV (the +2EV was purely to remind me that I was looking at a specific test image). This was all assuming that Neutral -4 contrast was somewhat linear.

Even accounting for some small offsets there were differences caused by the contrast curve (neutral -4 because it's not strictly linear) so I then did it the hard way by roughly setting the black and white levels before saving a jpeg then set up an A/B i.e. the 2 images with a layer mask to show in split screen. Had to desaturate the images because the As Shot WB rendered slightly differently between DCRaw and DPP.

You need to gain the images up and down as you work to ensure you're matching/fine-tuning the shadow values properly. As you plot the additional points you need to move the points accordingly to produce the smoothest curve (i.e. if one set doesn't work because the interpolation causes a kink then try the next set - it can be like herding cats) but even so, 10 points isn't enough - it's very tedious and laborious work ::)

note: I made the curve in PSE before the decoder became available so I already had the curve points ready for conversion to hex.

I'm wondering if we can alter the file size to increase the RGB and/or LAB gamma blocks to 10bits or if it will only work in 8bits with 12 points. Also, I'm still trying to work out these 20x3 matrices which appear to be used for a third-order polynomial regression to XYZ space (a fairly common practice in calibration).

Cinestyle is definitely in RGB space (the sRGB matrix is all zeros), so assuming the Adobe matrix is correct it should be possible through derivation to configure a matrix to output RGB in another colorspace either by default by re-tasking the sRGB matrix or as an option by changing the Adobe1998 matrix and selecting Adobe1998 in the camera (it's a better idea to be able to select IMO).

I'm using Matlab but for hex2dec conversion you can use a spreadsheet or something like: http://www.asciitohex.com/ and to edit the decoded file, this is good http://www.sweetscape.com/010editor/


There is a remote possibility that pf2, pf3 files are close enough to ICC standards that we might be able to add float expressions instead of hard-coded luts i.e. y=x^2.2 etc etc and maybe substitute the 3x20 matrices for simpler 3x3 chromatically adapted RGB matrices - there is lots to play with and discover :)

To do any of this more accurately/more easily the contrast function needs to be modeled to enable linearization and I think it's probably a SINE function!?
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

agentirons

Wow, great info! Thank you for all that research. I made it clear in the Google Sheet that the three parts of the LAB block are L*, a*, and b*, I don't know why that didn't occur to me. Have you been able to confirm if the same three parts in the RGB Gamma block are R, G, and B? Cinestyle uses the same curve for all three so it wasn't immediately obvious. Same with the Matrices, I'd love to know if they are actually broken up the way I colored them in the sheet.

As far as increasing the bit depth of the gamma curves, increasing the file length shouldn't be an issue (esp once chris adds that to the decoder, for now you can do it by hand), so I suppose it's just a matter of trying it out. You'd also need to change the block length tag to match your edits, and there's no obvious tag between components in the gamma blocks so it may of course be a hard limit.

I can also recommend Hex Fiend for OSX as a free and simple editor, with a very useful comparison tool: http://ridiculousfish.com/hexfiend/

Danne

Yes, very greatful for thorough answers and quality digging along with sheets and ingenious tools. Wish I had a vacation now :P.
Hex fiend does the work great. I use it all the time. Gonna check those blocks hopefully soon.

chris_overseas

Quote from: agentirons on October 03, 2016, 06:14:26 PM
As far as increasing the bit depth of the gamma curves, increasing the file length shouldn't be an issue (esp once chris adds that to the decoder, for now you can do it by hand)

I added support for that yesterday. Just redownload the jar and it should then update the data length entry seamlessly during the encoding step, same as it does for the checksum. I didn't test it very thoroughly though so any problems with it, let me know!

For hex editing on Windows I can recommend HxD (https://mh-nexus.de/en/hxd/).
EOS R5 1.1.0 | Canon 16-35mm f4.0L | Tamron SP 24-70mm f/2.8 Di VC USD G2 | Canon 70-200mm f2.8L IS II | Canon 100-400mm f4.5-5.6L II | Canon 800mm f5.6L | Canon 100mm f2.8L macro | Sigma 14mm f/1.8 DG HSM Art | Yongnuo YN600EX-RT II

Andy600

@agentirons - I didn't try independent RGB curves yet but it's easy to verify the order by making 2 of the 3 curves all 0s and see what color you're left with. I would assume it's RGB in that order but could also be BGR.

If you convert each byte of the matrix you get a value between 0 and 255 so they are 8bit values but I don't know if the order is correct yet - it probably is. 20x3 is tricky math ??? There are toolkits in Matlab (and probably Octave, Scilab etc) but I need to do some research and learn it - I doubt I've got time to do that this week unfortunately.


Thanks @chris_overseas - your decoder is a super useful tool!! :)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

jkaura

Quote from: Andy600 on October 03, 2016, 04:17:47 AM
You'll see the resulting output is no different than the profile with a log curve in the User RGB gamma block - I'm still trying to work out what that log curve is for?

Quote from: Andy600 on October 03, 2016, 03:36:59 PM
I replaced the RGB gamma blocks but the points will not show in PSE because there is/was no editable Tone Curve in a pf2. Cinestyle is a pf2 and has a tone curve so it's just a PSE omission. The editable Tone curve was added later (presumably under a different tag?) as part of the pf3 format.

You've been doing inspiring research! So you came to a conclusion that User RGB gamma block is specific to pf3 format only and is not supported/conveyed by pf2 format?

Andy600

No. I'm saying the User RGB Gamma block in the CineStyle.pf2 (i.e. it's log curve) doesn't seem to do anything. You get the exact same image out of DPP or an in-camera Jpeg or H.264 whatever data is in that block and I've changed the curve data multiple times and even removed it to prove this - you still get the same Cinestyle image.

The .p3f format introduced an editable RGB tone curve and a couple of other things to the PSE app where previously you had only a L*a*b curve.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

agentirons

Right, so, .pf2 has an RGB Gamma curve and a L*a*b* Gamma curve that both work, but only the L*a*b* Gamma curve is directly editable inside Canon's Picture Style Editor software (it's the curve box in the 'Specific Colors' tab). The RGB Gamma curve has to be edited manually using the hex editor, and I'm guessing only one is applied to the image if you have data in both curves.

PSE can edit the RGB Gamma curve if you save as .pf3. (Assuming here, I haven't actually tried it to confirm that's what's happening.)

The separate and much larger 'User' RGB Gamma block that is present in the Cinestyle .pf2 has seemingly no effect, and @Andy600 is trying to figure out it's purpose.

Andy600

Quote from: agentirons on October 04, 2016, 01:31:15 AM
Right, so, .pf2 has an RGB Gamma curve and a L*a*b* Gamma curve that both work, both only the L*a*b* Gamma curve is directly editable inside Canon's Picture Style Editor software (it's the curve box in the 'Specific Colors' tab). The RGB Gamma curve has to be edited manually using the hex editor

Correct

Quote from: agentirons on October 04, 2016, 01:31:15 AM
..and I'm guessing only one is applied to the image if you have data in both curves.

No, they can both be applied

Quote from: agentirons on October 04, 2016, 01:31:15 AM
PSE can edit the RGB Gamma curve if you save as .pf3. (Assuming here, I haven't actually tried it to confirm that's what's happening.)

You can edit the pf3 tone curve in addition to the RGB gamma curve data but any hex data you enter into the RGB gamma block does not show as control points in PSE - I really don't get what is going on with all these curves!

Quote from: agentirons on October 04, 2016, 01:31:15 AM
The separate and much larger 'User' RGB Gamma block that is present in the Cinestyle .pf2 has seemingly no effect, and @Andy600 is trying to figure out it's purpose.

Yes and still trying to figure it out ::). I did think that maybe it was applied only to H.264 and that I might not be seeing the true result of the curve due to the colorspace I was in but I've checked tweaked Cinestyle H.264s in Rec709, sRGB, Linear and across multiple color apps and a couple of NLEs - still the same result. The curve does nothing here. The last hope is HDMI output but I doubt that's going to be anything different.

PSE and DPP don't seem to like independent R,G & B Gamma curves either ???
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

chmee

very early stage. PictureStyler as Online-Tool. Im at a job in L.A., hadnt the time to code more :)

v0.2 - PF3 only, uploading, viewing 0x10-Tags

regards chmee
[size=2]phreekz * blog * twitter[/size]

Danne

Uploaded mine and dfort,s logNeutral.pf3
Looking good.

DeafEyeJedi

Looking really good @chmee re: PictureStyler however when I go ahead and upload a pf3 file it just stays on loading for a long time.

Is this normal?
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

chmee

by now, yes. its the moment, the filereader-api reads the file into memory. (look into a inspector [F12]). later i can try to speed up.
[size=2]phreekz * blog * twitter[/size]

agentirons

Fascinating little detail I picked up by examining Canon's Downloadable Picture Styles - You can actually delete the Edit Lock, Finalize (Checksum), and Base Picture Style settings (sharpness, contrast, saturation, color tone) blocks entirely!

For some reason Canon omitted all three of these sections in the Nostalgia, Clear, Twilight, Emerald, and Autumn Hues styles.

For Studio Portrait, Snapshot Portrait, and Video Camera X Series Look, they kept Finalize and the Base Settings but omitted the edit lock.











EDIT LOCKFINALIZEBASE SETTINGS
NOSTALGIA
CLEAR
TWILIGHT
EMERALD
AUTUMN_HUES
P-STUDIOXX
P-SNAPSHOTXX
VIDEO-XXX

chmee

@agentirons
great find. so it behaves quite like tiff, where some Tags are mandatory, some are recommended, some are free to use.
[size=2]phreekz * blog * twitter[/size]

dfort

I want to bring back reply 137 from a1ex:

QuoteFYI, properties PROP_PC_FLAVOR[123]_PARAM 0x401000[135] might contain user picture style data. In ML, they are only used to get the picture style names.

Sizes: 16704 on 5D3, 600D, 16680 on 5D2, 550D.

Example:

0x4010001  16704   0x824140 'Flaat_2'             0xffff00fe        0x0        0x0        0x0        0x1   0x219c60 0xfffffffc        0x2 0xfffffffe        0x0 0xdeadbeef 0xdeadbeef   0xfc0040 0xffff2fff        0x0 0xf64ad173 0x7e60894d 0xf452b0c0 ...


HTH

Here's a peek at one of the custom picture styles embedded in the ROM1.BIN file that ML creates in the Logs directory on the memory card:



Looks like the picture style data is stored unencrypted in the camera's non-volatile memory.

Note that the address is way off because it needs an offset of 0xFF000000 to match what a1ex is pointing out. Uh--I think, maybe? More information can be found in the finding stubs tutorial.

hjfilmspeed

So I most likely missed this but did anyone figure out why RAW video is so much more detailed then the h.264? Is it the the 444 vs the 420 or is it compression algorithms or do we think that canon is trying to blur those hit pixels. I noticed the 5Ds and 5DsR  have a fine detail picture style and I didn't know if this is something that could help the older cameras have a sharper h.264 option. I would think the all I codecs could provide more detail but maybe its not possible in camera at 24fps. What is the consensus on this?

reddeercity

Quote from: hjfilmspeed on October 09, 2016, 04:07:32 AM
did anyone figure out why RAW video is so much more detailed then the h.264? Is it the the 444 vs the 420 or is it compression algorithms or do we think that canon is trying to blur those hit pixels.
It's not 444 that's for sure.
Raw Video is 14bit RGGB  , no color space no processing at all , well there's a little from sensor data to ISO RGB then linearize more or less
( there a few more things done but that's another thread) saved in .mlv container then extracted as 14bit dng or 16bit Cdng format.
H264 all "I" or not it's 8bit 4.2.0 color space with burnt in Picture Style (not much latitude , very little) which is very inferior .

I guess you can ask your self Why are CR2/Raw photo cleaner and sharper then the Jpeg photo ?  14 vs 8 Bit .

                                                                                                                                                 

hjfilmspeed

True. Maybe this should fall under the pipeline topic. But I do believe there is something else going on. I mean there is a fairly substantial detail loss there.

dfort

The 5Ds and 5DsR also allows for Fineness and Threshold settings under Sharpness. Note that if you load a CR2 file from any other camera you can still adjust Sharpness from 0-10 but you can't move the Fineness and Threshold sliders.



I have not seen video from 5Ds and 5DsR with the "Fine Detail" picture style. Considering everything that's going on H.264 video from these Canon DSLR cameras is really pretty good--in my opinion.

Note that the APS-C cameras actually line skips the full sensor and shoots 1728x972 then uprezes in camera (someone can fact check me on this) while the Canon C100 actually has a 4K sensor and downscales internally so the H.264 files from the professional line is much better. The 5D3 shoots full 1920x1080 and uses pixel binning which is superior to line skipping and the resulting H.264 movie files are almost as good as the "real" cine cameras.

Of course H.264 is a lossy codec so that will account for some loss of detail over a raw image.

ddelreal

Quote from: dfort...Note that the APS-C cameras actually line skips the full sensor and shoots 1728x584 then uprezes in camera (someone can fact check me on this) while the Canon C100 actually has a 4K sensor and downscales internally so the H.264 files from the professional line is much better. The 5D3 shoots full 1920x1080 and uses pixel binning which is superior to line skipping and the resulting H.264 movie files are almost as good as the "real" cine cameras.

The 7D2 does this also? It sure looks like an APS-C version of the 5D3.