Reverse Engineering Picture Styles

[ImaXineria]

@Kharak: Thanks for the answer. How do you get the number?
or how did you get the number (base ISO)?

[DeafEyeJedi]

When it comes to MLV RAW -- I've always liked the results when the 5D3's at ISO 400 give or take. Used to think it was ISO 800 until @quickhitrecord & @baldavenger's each separate experiments which proves otherwise respectfully.

But in here we are talking about log styles being recorded into H264 format (not RAW) which is a whole different ball game and we normally (or at least should) shoot them in increments of ISO 160 such as 160, 320, 640, etc...

I've also noticed that 5D3 likes to sit at ISO 640 (either close down or use ND) or drop down to ISO 320 which seem to have slightly better shadow noise than at ISO 160.

FYI -- when we use C100II out on the fields at where I work -- it's 'sweet spot' is actually ISO 850 which is also being recorded into Wide DR custom profile (rather than a flat log) due to quicker turnaround workflows in post for broadcasting contents per Editors/Producers requests.

Guess it all comes down to how much time do we have in post to determine whether to shoot in a customized profile look or a flat log, right?

[dfort]

Thanks for this @DeafEyeJedi it always is enlightening to hear what is going on in the "reel world" 

I wonder how to scientifically determine the base ISO for various cameras using the available log profiles and also with the flat picture styles.

That reminds me, one of these days when your work load lightens up we should do that EOS-M vs. C100 Mark II test we talked about. If nothing else it would be interesting to see what real Canon Log looks like next to the various log picture styles. It would be fun too.

[reddeercity]

I hope this is on topic but I have been working with VisionLog DCP in A.E. with an added color/tone curve I'm getting very good results in the mids .
To the point , I found a open source pc app called "dcp2icc" which makes a icc profile from a dcp .
Since the picture profiles are really icc profiles for your camera , I thought if some how this information could be useful to get a neutral flat high DR. profile
optimized for magic lantern enabled camera's . Below are some screen shots from DNG Profile Editor and one from the command line with dcp2icc


VisionLogBaseToneCurve by RedDeerCityTV, on Flickr

With Vision log tone curve looks like there a lot more work on shadow detail.


Embeded_Camera_Profile-Base-Tone-Curve by RedDeerCityTV, on Flickr



Camera_Netrual_5D2_Profile-Base-Tone-Curve by RedDeerCityTV, on Flickr



dcp2icc by RedDeerCityTV, on Flickr

Usage: dcp2icc.exe <dcp> <color temperature>
make sure the xx.dcp is in the same root as dcp2icc.exe , just like the screen shot.
download at https://sourceforge.net/projects/dcp2icc/

[dfort]

Very interesting discoveries. There's also an open source tool to work with dcp profiles, dcptool.

This is all very much on topic because we're pretty sure that picture styles are saved as icc profiles in the camera. We just have to figure out exactly where the icc information is saved.

[Danne]

Yes, conversion tools are great to share. One thing can lead to the other. So we have dcptool for analysing and manually build dcp profiles.
http://dcptool.sourceforge.net/Introduction.html
Hopefully all logs are based on what adobe calls linear.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg172443#msg172443
Too build a mathematical log rather than a log "look" we would need the dcp to act as a 1D lut. It would be interesting to see a conversion from cineon or logC to dcp using dcp tool since dng profile editor itself only allows for 96 curve points. This however would be a totally a different reverse engineering project and could benefit from being discussed in a new thread.

[DeafEyeJedi]

Absolutely wonderful stuff @reddeercity and definitely agree with @Danne on the fact that we should focus more on mathematical logs rather than just a log "look" -- so for us to be able to spit out our very own linearized DCP profiles is rather remarkable and works flawlessly in MLP.

Re: Conversion from Cineon/logC to DCP for the 96 curve points is totally a new thread worthy already!

[SpcCb]

Excellent discoveries guys.
Since we know now that PS are somewhere ICC, or we can inject ICC in PS, we can imaging many things!
So sure a good math log would be very nice (I always found that filming things of life in linear was inhuman), and I saw we also could modified the color balance if I'm not wrong (?); it will be pretty nice to generate a couple of PFs calibrated for some special situations, we could switch fast between then while we are filming. But I think I've a lack of information about the internal process; I thought after readings here and imaging that RAW/PS->H264 4:2:0 but if we can modify the color balance with PS do you how this is interfere with the internal color balance?
When I will found where to download a recent version of EOSU to load PF3 in my 5D2 I will try to make some tests..

[dfort]

...if we can modify the color balance with PS do you how this is interfere with the internal color balance?
When I will found where to download a recent version of EOSU to load PF3 in my 5D2 I will try to make some tests..

You bring up a couple of interesting points.

Can we modify the color balance with the picture style? I would think that if the picture style has a bias towards warm or cool colors then sure, it will have an influence over the color balance but you can still adjust the color balance and create a custom white balance. So what affect does adjusting the color balance have on the "look" of the picture style? In other words, is white balance applied in the camera before or after the picture style is applied? This is something that I haven't tested.

You can go to any of the official Canon sites for the various regions and there should be a software download section for your camera. On the USA site it requires that you enter your camera serial number before downloading. Now what I just discovered is that the EOS Utility for a newer camera like the 5D4 has version 3.5.0 while your 5D2 has an older 2.14.20a. This shouldn't make any difference to you because 2.14.20a can handle the .pf3 picture styles.

Mac Sierra users like me will have to wait for an update to the EOS Utility from Canon that works with this new operating system.

[SpcCb]

(...) is white balance applied in the camera before or after the picture style is applied? This is something that I haven't tested. (...)

I had the same thought. And also that internal WB is apply directly from RAW (during debayering process), and PS WB *after* in RGB mode (ICC RGB/LAB way supposition). I hope not, because it should mean range will be short so reserved for 'film effects' stuff.

Indeed, I saw EOSU v3.5.0 but even in Europe we need 5D4 serial number to download it.
I'll try with EOSU 2.14.20a, I will see if we can play with color balance with this version.

[350D]

Sorry for offtopic, but, is there any theoretical options to create a tool to convert LUT files to Canon Picture Style files?

[dfort]

That is very much on topic. We haven't quite figured out how to make a picture style file that works with EOS Utility from scratch but some people are figuring out the pieces. It might also be possible to insert picture styles directly into the camera without using EOS Utility once the location and format that it is stored in is found. It seems that picture styles are stored in using icc profiles so is there a tool that can convert a lut to an icc? This is all theoretical at this point.

[agentirons]

@350D that's my end goal. I think a handful of us are working separately on teasing apart the code in various ways and building some tools.

[markanini]

Keep in mind ICC is more like a container format which can hold both LUT and matrix/shaper data. Should we discover Canon uses a standard-compliant implementation conversion between formats would be trivial with open source tools. First we need to figure out how to reliably extract ICC profiles from a pf2. Opening such a file in ICC Profile Inspector could also give us some clues.

[Lars Steenhoff]

http://nikonpc.com

I know this is for nikon, but its so awesome

[dfort]

http://nikonpc.com

I know this is for nikon, but its so awesome

Wow, that is awesome. A web based picture style editor.

[ImaXineria]

I have no idea where is the right place to post this
I got this results
Does somebody could confirm are acurate?
Please?

Using the waveform of the atomos ninja blade
Datacolor greycard
Arri fresnel 650 with CTB
5dmk3 with 70-200/2.8

   f   t   EI (ISO)   STOPS   
PP Neutral
   4   8     100         +3   
   4   60     100            0  TOTAL 6
   4   500     100   -3   
   4   100     1250.     +3   
   4   800     1250   0          TOTAL 6.5
   5   8000   1250      -3.5   
Eos Log   
       4   6       100.        +3.3   
   4   60       100        0        TOTAL 8.6
   4   2500     100      -5.3   
   4   80         1250.    +3.3   
   4   800       1250      0   TOTAL 8.9
   9   8000       1250.     -5.6   

[dfort]

@ImaXineria your table is rather difficult to read so if you don't mind I cleaned it up a bit.

   f      t    IE  STOPS   
Neutral
   4      8   100       -3   
   4     60   100  TOTAL 6
   4    500   100       +3   
   4    100  1250       -3   
   4    800  1250  TOTAL 6.5
   5   8000  1250       +3.5   
Eos Log   
   4      6   100       -3.3   
   4     60   100  TOTAL 8.6
   4   2500   100       +5.3   
   4     80  1250       -3.3   
   4    800  1250  TOTAL 8.9
   9   8000  1250       +5.6

Now what are we looking at? I assume f = f-stop and t = time but what is IE? Do you mean EI = exposure index? What about STOPS?

This looks interesting and very much on topic but could you please give a clearer explanation of this table?

[ImaXineria]

@ImaXineria your table is rather difficult to read so if you don't mind I cleaned it up a bit.

   f      t    EI  STOPS   
Neutral
   4      8   100       +3   
   4     60   100        0.   TOTAL 6
   4    500   100       -3   
   4    100  1250       +3   
   4    800  1250       0.    TOTAL 6.5
   5   8000  1250       -3.5   
Eos Log   
   4      6   100       +3.3   
   4     60   100         0.     TOTAL 8.6
   4   2500   100       -5.3   
   4     80  1250       +3.3   
   4    800  1250       0.      TOTAL 8.9
   9   8000  1250       -5.6

Now what are we looking at? I assume f = f-stop and t = time but what is IE? Do you mean EI = exposure index? What about STOPS?

This looks interesting and very much on topic but could you please give a clearer explanation of this table?

Thanks!
EI means ISO
Stops - underexposed or + overexposed  from base exposure

the that i am trying to discover is the dynamic range of sensor with different pictures profiles and EI (ISO) so could learn if picture profile really increases the dynamic range or only allows visually the color correction

and know if the chart  for the sensors of the series c cameras also works with the 5dmk3 camera in where it says that -contrary to the logic- the  high EI increase the  dynamic range towards the high lights and the low EI increase the dynamic range towards the low lights

[dfort]

Hum--the table still looks a bit wonky on tapatalk. Maybe I should have put it in code tags instead of just using courier font.

In any case, there seems to be a couple of typos. The 2500 in the EOS Log section should probably be 500 and f9 should probably be f5? (5.6?) That's just being nit picky--the real issue is that you are seeing a big increase in dynamic range with the log picture style and a slight increase at higher ISO settings. It looks like your neutral grey target is a bit too high in the Log test. Note that you have more stops below middle grey than above it. When looking at log footage before being processed it is generally quite dark. Your test seems to confirm the articles that I pointed out in previous posts. I don't have a Ninja 2 but looking at the manual it seems that it doesn't have a waveform monitor so you can't set your grey value to various IRE values.

Anyone confirm this or am I way off base with my observations? It would be interesting to determine the base ISO of your setup. Some people say it is around 650 but doesn't it depend on which log picture style you're using?

[EDIT] Now I sort of get what you are doing with the aperture but how are you getting those t values? 1/6th sec. to 1/8000 sec. exposures recorded on a Ninja 2. How?

[DeafEyeJedi]

Can everyone check on their Mac's whether or not EOS Utility now all of a sudden works on their OS X systems?



Recently Apple released an update (10.12.1) and what's even more funny is the fact that I haven't done this update yet. 

[dfort]

Yep started working here too. Strange--I thought we'd have to wait for Canon to come up with a fix.

[ImaXineria]

Re @dfort "Now I sort of get what you are doing with the aperture but how are you getting those t values? 1/6th sec. to 1/8000 sec. exposures recorded on a Ninja 2. How?"

my setup:
datacolor graycard
arri fresnel 650 CTB 216 
5dmk3
hdmi
atomos ninja blade waveform

base 
f4     t: 1/60s   EI(ISO) 100 Profile NEUTRAL and EOS HD LOG
f4    t:1/800s EI(ISO) 1250  Profile NEUTRAL EOS HD LOG

process:
base exposure 50 IRE for NEUTRAL 32IRE for EOShdLog
overexpose using time exposure as possible until reach 99IRE
underpose using time exposure as possible until reach about 5IRE

[ItsMeLenny]

Has anybody ever captured the USB data being transferred from eos utility to the camera?
I did a couple of years ago when trying to reverse engineer picture styles.
I figured why bother trying to reverse engineer the picture style, I'll just reverse engineer the data and then write it to the camera in my own program.
Anyway, the data that gets transferred is nothing like the picture style file.
I'm pretty sure it double checks the data as well against itself after transfer.
In addition to that, when you delete the picture style it writes a tonne of zeros.
If you reverse engineer the picture styles to use them in eos utility or dpp you're still limited to how canon programs read those files to create whatever stream of data is sent to the camera.

[dfort]

Has anybody ever captured the USB data being transferred from eos utility to the camera?

How did you do it?

From the beginning I assumed that the picture style files had information that was only useful for the Picture Style Editor and what was stored in the camera was basically boiled down, compiled, flattened or whatever metaphor works for you, into a simplified form that works with the camera. Right now most of us are assuming what is stored in the camera is an icc profile.

Writing a tonne of zeros is a clue that the picture style is possibly stored as a block of data that is a fixed size. When ML creates the ROM1.BIN file it also dumps the picture style data. I would supposed that what you are seeing in that data stream is the same information that's in that in that data dump.

So if you're saying that if you delete a custom picture style it fills up that block with zeros then that's a great clue to help figure out where the picture styles are stored by comparing a ROM1.BIN with and without custom picture styles installed.