Author Topic: Reverse Engineering Picture Styles  (Read 198189 times)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: Reverse Engineering Picture Styles
« Reply #75 on: September 05, 2016, 11:38:48 PM »
how does the EOSHD C-Log compare to the free Clog? ;D

The free C-Log is similar to CineStyle while the EOSHD C-Log looks a lot like the "real" Canon Log from the 1DC. Yeah, I know it is confusing that there is more than one C-Log out there.

This brings up an interesting issue. Is the Canon cinema line significantly different from the still cameras that can use Magic Lantern so that Canon Log doesn't translate properly? Log in the 1DC is located in the firmware while these other log picture styles are custom picture styles. A few posts back I asked about base ISO and it seems that it doesn't apply to the DSLR's yet the 1DC does have a base ISO of 400. As a comparison the C100 (CP Cinema Locked) has a base ISO of 850. Shooting below this setting results in increased noise (vertical stripes and banding) and reduced dynamic range when shooting in Canon Log mode. It is ok to use a higher ISO but the dynamic range is reduced. So do these reverse engineered C-Log picture styles really work? Do we need to shoot at a minimum ISO setting? Again, more questions than answers.

In my last post I may have given the impression that we were done with this project--not the case. There's still lots that can be done.

One experiment that is showing promise is to take a technical lut, invert the curve and create a reverse engineered picture style.

Danne has been experimenting with dcptool which is a command line tool for editing DCP, Digital Camera Profile, files that can be embedded in DNG files and works with Adobe Camera Raw. Adobe has an editing tool called DNG Profile Editor that creates DCP files and is erringly similar to Canon's Picture Style Editor. In addition Canon's Digital Photo Professional application uses .pf2 and .pf3 files much like Adobe Camera Raw and Lightroom uses .dcp files. The difference being that Adobe has an SDK for their DNG file standard while all of Canon's file formats are closely guarded. What would be possible if we could get into the Canon file formats? It would be possible to build an MLV to DNG application that translates the picture style into a dcp that can be tagged into the DNG files. Of course the DNG files need to be debayered by a program that understands how to decode the embedded dcp information and it seems that only Adobe Camera Raw and Lightroom can do it.

5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: Reverse Engineering Picture Styles
« Reply #76 on: September 06, 2016, 02:25:20 AM »
Holy Shizzle this Project has finally taken off and it is even more surprising to see how much has come along in so little time. All the thanks goes to @dfort and @Danne for pushing this forward in a much quicker pace. Actually I am starting to regret not following this closely while I was out on Vacation in Chicago because I shot a bunch of things in H.264 (HDR, 1:1 crop, etc) all with Technicolor Cinestyle PP (I sometimes do use Prolost Flat settings within Neutral PP) and now I've decided to jump on the REPS wagon (even though I may be a little late) for this journey.

I completely agree with @dfort that we should all figure out a way to set the native base ISO (I think it's ISO 400 for 5D3 but not sure of others?) in order to maximize not only the DR but to keep the noise as free (or real) as possible rather than introducing useless artifacts into the footage.

So I downloaded the v1.3 (Thanks again for that!) and installed them into my SL1 (100D) just because I have it here with me at work (left the 5D3 at home) and decided to shoot them in ISO 100 (even tho it has more noise than ISO 200) but it's day time here for me and of course I am just too anxious to see the results.

Shot in H.264 on SL1 in 24p with 1/120th closed down to f5.6 @ ISO 100 with the PP in order enlisted below.

1) Neutral (Prolost Flat settings)
2) LogNeutral (0, -4, -2, 0)
3) C_LOG_htp (0, -4, -2, 0) <-- I got this from here
4) Technicolor CineStyle (0, -4, 0, 0) <-- I usually have Saturation down to -2 but for this test I just left it at default.


I couldn't find a LUT that would help better show the differences (left my MBP plus external drives at home) so I just impatiently chose the crappy Kodak2393_Rec709 that I downloaded from the internet. I know it's not the most ideal.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: Reverse Engineering Picture Styles
« Reply #77 on: September 06, 2016, 02:53:01 AM »
Hey @DeafEyeJedi I'm next for a vacation, will be gone for a month.

In the meantime if you really want to go crazy add EOSHD C-LOG and all of the variations Danne and I made, logNeutral, logStandard, logPortrait, logLandscape and logFaithful. Some of these are very subtle. You don't have to go searching for a lut, a Technicolor CineStyle lut is built into DaVinci Resolve. It also has Canon Log luts so you can try them on the C-Log picture profiles. Yeah, I know--wish you could load in more than three custom picture styles. Note that if you can see a difference between CineStyle and logNeutral we didn't do a good job.

5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: Reverse Engineering Picture Styles
« Reply #78 on: September 06, 2016, 09:45:58 PM »
I created a very crude 10 point 1D lut and an inverse with so-rose fine tool openlut and turned those into 3D_luts luts to play around with. While cinestyle s-curve seems a somehow boosted inverse curve I think these 3D luts will catch the resemblance to the canon neutral setting more correctly.
The main issue creating an inverse is the fact that cinestyle contrast setting is set to -4. This setting can,t be inversed with a 1D lut as it probably affects clarity and other built in overall settings not represented in a 1D lut gamma curve. In order to get a correct inverse canon neutral picture back you have to either filmed with logNeutral with the cam settings set to 0 both contrast and color(same as canon neutral) or set canon neutral setting to the same in cam settings as logNeutral. Happy testing!

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: Reverse Engineering Picture Styles
« Reply #79 on: September 07, 2016, 04:43:54 PM »
Interesting experiments going on. Danne tweaked the tone curve and got more highlight detail. That's the beauty of opening up CineStyle, well let's just say a CineStyle look alike.

@reddeercity did a massive test comparing logNeutral and CineStyle with other picture styles:
http://magiclantern.fm/forum/index.php?topic=17828

I'm about to leave on a month long trip but before going I packaged up a new release. This one has Danne's highlight recovery. It looks like there's not much interest in the logStandard and logFaithful picture styles so I'm dropping those but leaving in logPortrait and logLandscape. We might play with saturation and contrast levels on those in the future but testing three distinct picture styles will be easier than five subtle ones. Besides, you can only load three custom picture styles at a time.

Version 1.4 link on first post.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: Reverse Engineering Picture Styles
« Reply #80 on: September 11, 2016, 03:23:10 AM »
Thanks @Danne & @dfort for these rapid updates. However not sure if this is to be expected but I noticed that if shoot in LogNeutral (v1.4) which then Zebra's aren't showing on the LiveView LCD whereas it does if I were to use CineStyle or any other PP.

LogNeutral PP:


CineStyle PP:


Also note in the histogram which shows the highlights being completely clipped in the LogNeutral PP -- could this be the issue related?
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: Reverse Engineering Picture Styles
« Reply #81 on: September 11, 2016, 06:25:56 AM »
...not sure if this is to be expected but I noticed that if shoot in LogNeutral (v1.4) which then Zebra's aren't showing on the LiveView LCD whereas it does if I were to use CineStyle or any other PP...

That's expected. If you open up logNeutral v1.4 in Canon's Picture Style Editor you'll notice that the highlights are clipped. This is something that CineStyle does in the shadows and we were experimenting to see what it does in the highlights. Check out EOSHD C-Log if you can, it will probably exhibit similar behavior.

I'm running around Europe at the moment and am operating on my wife's laptop so no development while I'm on vacation!

In any case, note that v1.4 is an experimental version. Guess I should have marked it beta or something. So far v1.3 is the closest we've gotten to CineStyle.

[EDIT] Ok--so I'm cheating on my vacation. Here's a screenshot showing the highlight clipping on v1.4. Like I said, this is an experiment not necessarily a "final" version.



5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Reverse Engineering Picture Styles
« Reply #82 on: September 11, 2016, 09:28:52 AM »
Well I kinda like it ... It's like a cheat sheet for the highlights. Definitely worth the risk. Lovely 10-point curve designed by @Danne. I wonder if there's a way to add more points into it? Love the fact that you don't just risk anything but do risk everything!

Looking forward to testing with some more this weekend.

btw @dfort I went ahead and purchased EOSHD C-log just for shits and giggles. Will eventually let you all know how it goes internally and externally.

I also am confident that once this LogNeutral hits by 1.6 or 1.8 it'll be smoking hot like no other!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: Reverse Engineering Picture Styles
« Reply #83 on: September 11, 2016, 11:05:23 AM »
C-log clips highlight even more. I don,t know how well that responds to H.264 in non C-log cameras.
I like technicolor more and more I must say. When I tried using it when it first came out I was under the wrong impression that color was wasted and I did my share of skin/skylight color complaining and then settled for neutral with in cam contrast/sharpening set to zero. Guess I was wrong all along.
Clipping the whites is interesting. It seems there is some more information in highlights which are clipped. Why not try to save it?

Crops

version 1.4


version 1.3


version 1.4


version 1.3



Quote
Lovely 10-point curve designed by @Danne
Well, let,s not underestimate what these two dots actually do to refienements coming from dfort himself.


gdims

  • Just arrived
  • *
  • Posts: 1
Re: Reverse Engineering Picture Styles
« Reply #84 on: September 11, 2016, 12:28:29 PM »
I can also confirm that C-Log clips highlights. It clips everything above the brightness level of 235. Wouldn't it theoretically have more latitude if it didn't clip?

I'm getting some brilliantly clean footage with it, however. I still believe it is bett than cinestyle or I just failed to properly grade the latter.

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: Reverse Engineering Picture Styles
« Reply #85 on: September 13, 2016, 04:29:42 AM »
Neutral 1.4 has the same problem as Cinestyle. Some kind of grey paste in the skin.
I made the shot with actors, with neutral -2 -4 and and the neutral. I applied the same color post and 1.4 it's not usefull yet.

For my perception, and opinion, it's something in the high luminance curve that is not working fine.

I don't know how to add an image here, I tried the shortcut but dont work.

https://www.dropbox.com/sh/2q3x73jf66wmj0d/AADWJ7hID3ny2Qay8Zeoy2Ioa?dl=0


Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: Reverse Engineering Picture Styles
« Reply #86 on: September 13, 2016, 06:27:24 AM »
Could you upload the originals (jpg or cr2 if they exist too) without anything applied? I would like to test some. If you have the same shot in regular neutral and in logNeutral we could compare the results.

Here are some tests you could do
1 - Try the same pic style 1.4 but this time don,t dial down the the color slider in cam at all. keep it at 0. Then apply the inverse lut from here. Keep all in camera settings the same(especially the contrast slider) for both neutral and logNeutral.
http://www.magiclantern.fm/forum/index.php?topic=16299.msg171880#msg171880

2 - Try one of the 1.3 versions with different color. I prefer standard or portrait, not sure if dfort added those but you could do that easily in picture style editor and resave it yourself. You can add the inverse lut the same as in above example to compare.

On a sidenote I don,t know how you white balanced your shots, seems a little cold. If the tone curve is added back correctly there should be no difference in skintone from neutral and logNeutral as long as you set the in cam sliders the same in both logNeutral and neutral.
Did you by the way compare 1.4 and 1.3 since you say it affects luminance? I thought luminance would mostly be noticable in brightest whites, lesser so in midtones and colortones.

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: Reverse Engineering Picture Styles
« Reply #87 on: September 13, 2016, 09:17:25 AM »
@Danne, I made video. It's h264. And I exported the frames as JPG.
The light was fluorescente. And I made WB from rate button, with ML, pressing set.

After that I remembered your work in the reherseal and I just change the preset with the same WB, ISO, F and I took one of each.

I added to the same link two frames without post.

There's something in the highlight, in the skin tone on the face. Like a grey. But maybe you perceive something in the setting. I think, for my little expierence, - this is not a fact, please -  the luminance without any saturation make some of this grey and even adding in post some saturation, that information is not usefull.

I think could be usefull investigate the EOS log as @DeafEyeJedi propoused.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: Reverse Engineering Picture Styles
« Reply #88 on: September 13, 2016, 10:23:15 AM »
I appreciate when there is some testing around this. However, it will help this project forward better if you upload a test sample file. Simply saying "There,s something in the highlight, in the skin tone on the face" will only add to the misconception on how not to work with cinestyle/logNeutral I,m afraid.

*update
Will download your unedited files. Thanks
*update
Problematic to use the provided files for testing. If we are doing a comparison test we would need two exposures of both logNeutral and neutral with the same scene.


Here is one H.264 shot with version 1.4 and with a lut applied.




chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: Reverse Engineering Picture Styles
« Reply #89 on: September 13, 2016, 10:36:22 PM »
I did my thoughts on canon-profiles (aka styles) as well. i couldnt believe, these files arent hacked yet. there naturally is more than just contrast, saturation and so on. my thoughts by now ( i started yesterday a little bit analyzing files)

* a "simple and fast processing" inside the body is needed, therefore it must be a simple encoding.
* the pf3-files are ~double the size of the icc files found in the preset-style icc-path
Quote
example neutral with no adjustments
* icc file size - NS.ICC - 212kb
* pf3 file size - 425kb
* maybe for safety purposes the lut is written two times.
* maybe there s a inverted lut inside as well

* i saved a pf3-file with no changes three times with ~5secs difference. it seems, theres a timestamp in it, but the timestamp does not affect the encoding. nearly the end of the file a byte changes as well. a checksum? integrity check?
* the pf3-file must be rockstable to avoid crashing or deadlock the processing inside the body.
* the title and copyright arent saved plain, this leads me to the idea of a simple encoding. rle? difference? xor? not?

example links:
Canon Neutral SRGB 6091 ICC as XML (parts of it)
pf3 neutral saved three times

Because i dont know much about ICCs, i have to read more about ICC-LUTS to understand and recognize lut-data inside the hexview :) maybe i find a byte/word (simply the clue) to decode the pf3-file. after this it should be much more simple to create own luts and pf3-files. Working only with the luma-curve is unsatisfactory.

by the way. did you tried combining your lumacurve with low-contrast-setting? is this affecting the max-luma from 235 to maybe 255?

simple toDos:
* decode title and copyright (we know the plaintext)
* decode the simple things (contrast, saturation hue aso)
* decode lumacurve
* decode 6axes-data
- i assume there are all in the beginning, then the lut comes.

regards chmee

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: Reverse Engineering Picture Styles
« Reply #90 on: September 14, 2016, 07:46:54 AM »
 
Quote
after this it should be much more simple to create own luts and pf3-files. Working only with the luma-curve is unsatisfactory.
word.

Quote
by the way. did you tried combining your lumacurve with low-contrast-setting? is this affecting the max-luma from 235 to maybe 255?
You mean low contrast setting in cam or altering the luma curve in picture style editor? If in cam it,s set to -4.

Quote
is this affecting the max-luma from 235 to maybe 255?
Probably :)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: Reverse Engineering Picture Styles
« Reply #91 on: September 14, 2016, 08:18:58 AM »
Interesting findings @chmee -- I'm on a vacation at the moment and don't have the time to look into this but it seems similar to what I was doing with reading header and footer information in my focus pixel bash script.

http://magiclantern.fm/forum/index.php?topic=16054.msg163309#msg163309

I guess we can start out with baby steps by creating a very simple picture profile saved with and without caption and copyright information and search the pf2 or pf3 file in order to find where it is stored. Then keep doing this changing the base picture style, add just one coordinate in the RGB tone curve, etc.

Sounds tedious but that is "real" reverse engineering and not simply trying to create something that "looks" like a picture style we're trying to mimic.

Note that when creating logNeutral I was checking it on a waveform monitor and it was frustrating that sometimes making a change in one area of the tone curve would cause changes to other areas. I'm guessing this is because of the "smoothing" that the Canon Picture Style Editor does between the points (maximum of 10 points) on the tone curve. I'll do an illustration of this when I get a chance. It would be much more accurate if we can define every coordinate on the tone curve. I'll bet that the EOS Utility is translating the information stored in the pf2 or pf3 files into a simpler binary format (meaning no smoothing between coordinates) that is stored in the camera's firmware. It would make sense that Canon is using ICC profiles that are also saved to the CR2 files but so far it seems that Canon has not released this information. Of course now I'm just rambling about things I don't really understand.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2295
Re: Reverse Engineering Picture Styles
« Reply #92 on: September 15, 2016, 07:39:28 AM »
@chmee thanks for the info , and with that I decided to look inside the Picture Style Editor as the base picture styles are there.
After a little time I do believe I have found the start of the Picture Styles with help of "HxD" Hex editor  , there are labeled as ".icc"
and can be found  on Windows at C:\Program Files (x86)\Canon\Picture Style Editor\DPP4Lib\icc

FA.ICC is = "6191 Faithful AdobeRGB"

This just the first few lines the rest I really don't understand

Quote
M€UCCM   scnrRGB Lab Ø           acspMSFT    CANO6191              öÕ     Ó+CANO                                                cprt     8desc  L   xwtpt  Ä   rTRC  Ø   gTRC  è   bTRC  ø   rXYZ     gXYZ     bXYZ  0   A2B0  D K<A2B1  D K<A2B2  D K<text    Copyright CANON INC. 2008 All Rights Reserved   desc       6191 Faithful AdobeRGB         6 1 9 1   F a i t h f u l   A d o b e R G B    6191 Faithful AdobeRGB      XYZ       öÕ     Ó+curv        c.  curv        c.  curv        c.  XYZ                 XYZ                 XYZ                 mft2    !                                      ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  €

LA.ICC is =  "6091 Landscape AdobeRGB"       

Quote
M€UCCM   scnrRGB Lab Ö
     acspMSFT    CANO6091              öÕ     Ó+CANO                                                cprt     8desc  L   xwtpt  Ä   rTRC  Ø   gTRC  è   bTRC  ø   rXYZ     gXYZ     bXYZ  0   A2B0  D K<A2B1  D K<A2B2  D K<text    Copyright CANON INC. 2003 All Rights Reserved   desc       6091 Landscape AdobeRGB         6 0 9 1   L a n d s c a p e   A d o b e R G B    6091 Landscape AdobeRGB  XYZ       öÕ     Ó+curv        c.  curv        c.  curv        c.  XYZ                 XYZ                 XYZ                 mft2    !                                      ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  € €  L€»}O ~z„‚¯v&Ñ„mpцái¡/Š3cFúŽ“]K”5Vç 8š6QÝŸIKbV£<F!¦NA#Ï©i<
#;¬–6Ô(ǯÔ1.o³$,47¶†&

NA.ICC is = "6091 Neutral AdobeRGB"       

Quote
MxUCCM   scnrRGB Lab Ö 
       acspMSFT    CANO6091              öÕ     Ó+CANO                                                cprt     8desc  L   pwtpt  ¼   rTRC  Р  gTRC  à   bTRC  ð   rXYZ      gXYZ     bXYZ  (   A2B0  < K<A2B1  < K<A2B2  < K<text    Copyright CANON INC. 2005 All Rights Reserved   desc       6091 Neutral AdobeRGB         6 0 9 1   N e u t r a l   A d o b e R G B    6091 Neutral AdobeRGB  XYZ       öÕ     Ó+curv        c.  curv        c.  curv        c.  XYZ                 XYZ                 XYZ                 mft2    !                                      ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  € €  ”|.Z‚ow"3„op9G‡Hi]þ‹2b‡   % \, Á•¼VC⚆Pԍž^KÛ»¡OGQÿ¤BéC¦È>#‡©r:G'Ǭ6   , ®ª1Ù0S±9-´4—³Á)–8ܶE%€="¸¿!sAi»7lEµ½©nIÿÀtNQ„R•ÄÝ
¦VÕÇ/   Ú[É|_^ËÄ`cªÎ  gûÐC  lRÒ|  p³Ô°  uÖÝ   |²‚²Ñ~&}bzZy^s/Ì


PA.ICC is = "6091 Portrait AdobeRGB"       

Quote
M€UCCM   scnrRGB Lab Ö 
       acspMSFT    CANO6091              öÕ     Ó+CANO                                                cprt     8desc  L   xwtpt  Ä   rTRC  Ø   gTRC  è   bTRC  ø   rXYZ     gXYZ     bXYZ  0   A2B0  D K<A2B1  D K<A2B2  D K<text    Copyright CANON INC. 2005 All Rights Reserved   desc       6091 Portrait AdobeRGB         6 0 9 1   P o r t r a i t   A d o b e R G B    6091 Portrait AdobeRGB      XYZ       öÕ     Ó+curv        c.  curv        c.  curv        c.  XYZ                 XYZ                 XYZ                 mft2    !                                      ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  € €   € |5  €
w  ‚éo=  ‡6fÓ šŒW^çB‘ºW·†˜†PòvŸáJž '¥ãDÄ°ªr?o$­j:­ø¯Å6 ¼²J1q&z´æ,°,3·™'Ú1ñº["ó7°½0ú=sÀñC=ÃÖI Æ«NÎÉ
   ƒT‰ÌYZHÏ   `   Ò8  eÎÕY  k™Ø‚  qiÛ´  w<Þí  }â.  ‚øåv  ˆßèÇ  ŽËì  xӁY‹z=}É`| x’~™pD4åhŠY†4a-
 

SA.ICC is =  "6091 Standard AdobeRGB"       

Quote
M€UCCM   scnrRGB Lab Ö 
       acspMSFT    CANO6091              öÕ     Ó+CANO                                                cprt     8desc  L   xwtpt  Ä   rTRC  Ø   gTRC  è   bTRC  ø   rXYZ     gXYZ     bXYZ  0   A2B0  D K<A2B1  D K<A2B2  D K<text    Copyright CANON INC. 2005 All Rights Reserved   desc       6091 Standard AdobeRGB         6 0 9 1   S t a n d a r d   A d o b e R G B    6091 Standard AdobeRGB      XYZ       öÕ     Ó+curv        c.  curv        c.  curv        c.  XYZ                 XYZ                 XYZ                 mft2    !                                      ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  € €  |%H‚’wÁ…oÚ'ˆ¬hˆ
ŒdaÓ‘[µ O–XVv›P¾ žÐKäå¡ßGmÓ¤¶C"©§Œ>ë!kªW:Ê%+­6Ê) ¯a3,Þ±«/W0œ³ï+À4K¶!(D7ì¸A$ä;wºZ!›>ï¼fmB1¾˜BE^ÀÁ2H¬Â™cKèÄe±OÆ$ R(ÇÏ
ÁU(Ék ‡XÊû   v[Ì€]éÍõÒ`ÃÏbEk{Ü‚Nü|ø~†§~SyŸ€vrVìƒrk   ˆ7dŽ  ^<
’€X^~—7Rý×›NŽžKI…H¡UE%ÿ¤A@ç"¶§ <Ì&f©¹8Ò*¬L4ö-©®Ì1518±9

I also noticed that with every Style/ICC was either sRGB or AdobeRGB , I guess it's depended on if you set your cam to sRGB or Adobe . Personally my 5d2 is set to AdobeRGB .
 
This line seem to be a constant "cprt     8desc  L   xwtpt  Ä   rTRC  Ø   gTRC  è   bTRC  ø   rXYZ     gXYZ     bXYZ  0   A2B0  D K"  Could this be curve information ?

and this one here also  "XYZ       öÕ     Ó+curv        c.  curv        c.  curv        c.  XYZ                 XYZ                 XYZ                 mft2    !                                      ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ  ""33DDUUffwwˆˆ™™"
the trade mark "TM" is interesting .

I think if we understand the XYZ  & the "c. curv" we should be able to customize the profile/style without having to relaying on the 10 point curve in Pic.Style Editor.
Or I could be Totally off base here ! :D

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: Reverse Engineering Picture Styles
« Reply #93 on: September 15, 2016, 08:39:36 AM »
Nice find. On Mac those files are in:

Code: [Select]
/Applications/Canon\ Utilities/Picture\ Style\ Editor/PictureStyleEditor.app/Contents/Resources/icc
Wolfcrow has a good introduction to what XYZ colorspace is and how it relates to RGB.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: Reverse Engineering Picture Styles
« Reply #94 on: September 15, 2016, 11:15:23 AM »
iccXML is able to convert the icc files to XML-human readable data. just for analyzing purposes. As i did that, i ve seen, icc files are not simple luts as i thought, its not a converting list 0->0, 1->1.2 and so on. its much more complex. (maybe there are 32bit-floatinpoints saved instead of 16bit-integer?)

later i will post some more findings on decoding pf-files. it seems something like a simple bitshift over the whole file or a list of bitshifts/adds/subs. its kind of a puzzle we can solve together :)

the main icc's are not in the cr2-files - theres only a integer/string to name the needed picture-style. Thats the reason, Adobe isnt able to use the real canon-styles - only dpp. for us, its only important for mp4/mov-recordings.

regards chmee

Andy600

  • Contributor
  • Hero Member
  • *****
  • Posts: 1863
  • Have you tried turning it off and on again?
Re: Reverse Engineering Picture Styles
« Reply #95 on: September 15, 2016, 11:38:54 AM »
@reddeercity - You won't get far looking at encoded files  ;D

There are several tools available to view, dissect and edit ICC/ICM profiles and the first place to look is The International Color Consortium (ICC themselves). A good tool to start with is wxProfileDump (http://www.color.org/profdump.xalter) but you'll need a something like Matlab, ICCedit or ICCxml to delve deeper and edit things.

The Picture Style ICC files are all pretty much Linear (Y=X) and work in conjunction with a colorspace (either sRGB or Adobe 1999) - as I mentioned earlier, if there is an S (NS, FS, LS, SS, PS) it's the sRGB version and if there's an A (NA, FA, LA, SA, PA) it's the Adobe 1998 version.

There is an XYZ 3x3 matrix but it's neutral because everything is done with luts. The profile connecting space (PCS) is Lab.

The luts are the AToB0, AToB1 and AToB2 tags and specify the color adjustments depending on rendering intent. The luts are in Lab colorspace and have an RGB shaper that is Linear.


With that being said I think ICC is probably a red herring regarding Picture Styles. They are very similar but not interchangeable. You can dissect a profile in Matlab and copy parts to/from other files by finding the 3x1 struct array and headers but you can't change an ICC to a pf2 or pf3 for editing in PSE.


I think Chmee's idea of embedding look luts in a PS is the most interesting but I think there will be some restrictions because it has to be in Lab colorspace.

What is really needed is a 1D lut or function that works on the code values coming from the sensor before the picture style and encoder stages and I'm not sure that's even possible (a1ex?). If it were then Canon log would become a possibility. At present a Picture Style made in PSE is working with a truncated, display range signal.



Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: Reverse Engineering Picture Styles
« Reply #96 on: September 15, 2016, 12:04:07 PM »
Red herring :D

Very interesting information and digging and some useful tools too shared in these later posts. Great.

Quote
the main icc's are not in the cr2-files
Wonder if that is really true here. How can one explain CR2 files shot with a selfmade picture style having all information applied in dpp without needing to install the actual picture style in there? An encoded icc profile only read/known by canon?

Andy600

  • Contributor
  • Hero Member
  • *****
  • Posts: 1863
  • Have you tried turning it off and on again?
Re: Reverse Engineering Picture Styles
« Reply #97 on: September 15, 2016, 12:18:11 PM »
cr2 doesn't store ICC's but there will be a flag for DPP to call one and if the PS is custom it is embedded in some way (probably as pf2 or pf3) else DPP wouldn't know how to produce a Jpeg that matches the one shot in the camera.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: Reverse Engineering Picture Styles
« Reply #98 on: September 15, 2016, 12:26:52 PM »
point taken, @Danne :) you re right. But, if so.. we could try to extract the cinestyle-Preset from a cr2 and then compare with the pf3.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: Reverse Engineering Picture Styles
« Reply #99 on: September 15, 2016, 12:55:27 PM »
Yes that could be useful. As for creating icc profiles one can convert 3D luts to icc profiles with for instance OpenColorIO. Tip from Andy600 a while ago. That fact doesn,t really solve the core issue here for know but might be useful later.