Ok--here we go:
2016-09-07 v1.4 - Test version to improve highlights detail. (Don't use if you're using zebras to judge exposure.) Dropped logStandard and logFaithful. (https://bitbucket.org/daniel_fort/magic-lantern/downloads/log_picture_styles_v1.4.zip)
2016-09-02 v1.3 - Overall better match (https://bitbucket.org/daniel_fort/magic-lantern/downloads/log_picture_styles_v1.3.zip)
2016-09-01 v1.2 - Combined highlights from v1.0 with shadows from v1.1 and Saturation now defaults to -2 (https://bitbucket.org/daniel_fort/magic-lantern/downloads/log_picture_styles_v1.2.zip)
2016-08-31 v1.1 - Small adjustments to correct color shifts in shadows (https://bitbucket.org/daniel_fort/magic-lantern/downloads/log_picture_styles_v1.1.zip)
2016-08-31 v1.0 - Initial release (https://bitbucket.org/daniel_fort/magic-lantern/downloads/log_picture_styles_v1.0.zip)
This was a joint effort between Danne and myself, dfort. For the first release we did the best we could at matching Technicolor CineStyle. The reason we picked CineStyle is because it is in wide use, hasn't been updated since it was initially released in 2011 and it is a .pf2 that cannot be editing using Canon's Picture Style Editor. Ours is in the newer .pf3 file format and is editable. We didn't even put a copyright notice on it so feel free to do with it as you please.
We did a lot of testing and have a fairly high degree of confidence that this can be a direct replacement for CineStyle. This means that all of the luts available for CineStyle can be used. Is it a perfect copy? No, but it is the best we could do using the tools we had at our disposal. Visually it should be impossible to see any difference between CineStyle and logNeutral. Why call it logNeutral? Because it turns out that CineStyle uses the standard Canon Neutral picture style as its base. Only the luma gamma curve has been modified. How do we know this? Let's just say we're pretty sure.
So if there is a logNeutral could there be variations using other standard Canon picture styles? Yes. We found out that the luma gamma curve works with other picture styles. Sure, you could probably adjust the look of the Neutral style in post to match the other picture styles but one of most often mentioned issues with CineStyle is that it is hard to get pleasing skin tones. So why not experiment with the Portrait style as a base? We included all of standard Canon picture styles with this release.
Now with all the excitement over raw video why is a copy of a 5-year old picture style of any significance? It turns out that many people are using CineStyle not only for H.264 (8-bit 4:2:0) but also for HDMI out to an external recorder. In the future perhaps someone can do the same with Log-C or something that will take better advantage of the 8-bit 4:2:2 HDMI output.
Consider using MLP (http://magiclantern.fm/forum/index.php?topic=13512.0) in your workflow for this picture style. Danne continues to update this program and it can handle luts, denoising and conversion to Prores. MLP can also work with H.264 HDR which is a natural for this picture style.
Most of all, try it out and report back. If you don't like it, hey at least you didn't pay for it.
********************
Original post starts here:
********************
Not sure which category to post this under because it is part general chat, part shooting, part post production, part raw and part not...
I've been thinking about picture styles because of a conversation I had with Danne about CineStyle. I call that conversation the "CineStyle Conspiracy Theory." Basically it goes like this:
Ok, let's start out with who developed Cinestyle. Technicolor--not exactly a little fly by night company but one of the most respected companies in the motion picture industry. Interestingly it is a French multinational company, not just a film lab in Hollywood.
When. 2011, that's a year before the 5D Mark III but more importantly also a year before the C100 which has the so-called "log" profile built in. Why did Technicolor create Cinestyle then stopped supporting it? The luts and tools that Technicolor developed are no longer available directly from the company though the webpage is still live and the profile is still available for download.
Looking at the sample footage shot on "Canon Log" from the C100 compared with Cinestyle they look very similar. Same with the GoPro "Protune" but here's a little known fact--Protune was developed by Technicolor. I'm not saying that GoPro footage is great but Protune does allow more flexibility in post in order to more closely match footage shot on other cameras so there is some merit to this.
I have read a lot of pros and cons (mostly cons) on the boards about Cinestyle and mostly praises for the "log" profiles like S-Log, Log C, Canon Log, etc. Log does have advantages and disadvantages. The old saying that you can't get something from nothing certainly applies here. Technically, it may even be inferior to getting everything "right" in the camera but in practice things are a little different.
I can't look into what they did with CineStyle because that profile is locked out of the Canon PictureStyleEditor but I did read up on how to make a picture style and you have to start with a CR2 file. That got me thinking, all this talk about picture styles not affecting raw images--is that really true? I shot some still and MLV files with monochrome and CineStyle then took a look at them in various programs. Guess what, it does make a difference depending on what program you're using. Try it yourself, set the profile to monochrome and shoot something then open it up in Canon's Digital Photo Professional app--that's a free one that you can download from Canon. The raw image is in black and white. Now open it in Photoinfo--I'm not sure where that came from but I think it comes with OS-X. Also in black and white but it is apparently showing the jpeg preview. How about Adobe Camera Raw or RawTherapee? Color! Run exiftool on it and it will show:
Picture Style : Monochrome
Running exiftool with the -v option will give even more information:
| | | 20) ProcessingInfo (SubDirectory) -->
| | | + [BinaryData directory, 28 bytes]
| | | | ToneCurve = 0
| | | | Sharpness = 3
| | | | SharpnessFrequency = 0
| | | | SensorRedLevel = 0
| | | | SensorBlueLevel = 0
| | | | WhiteBalanceRed = 0
| | | | WhiteBalanceBlue = 0
| | | | WhiteBalance = -1
| | | | ColorTemperature = 7300
| | | | PictureStyle = 134
| | | | DigitalGain = 0
| | | | WBShiftAB = 0
| | | | WBShiftGM = 0
What is interesting is running an MLV shot with the monochrome picture profile through mlv_dump with the -mv option will also show some picture style information:
Block: STYL
Offset: 0x0000022c
Size: 52
Time: 0.006000 ms
picStyle: 134
contrast: 0
sharpness: 3
saturation: -559038737
colortone: -559038737
There's that same "134" picture style which I guess is the monochrome profile. How about an MLV shot on CineStyle?
Block: STYL
Offset: 0x0000022c
Size: 52
Time: 0.006000 ms
picStyle: 33
contrast: -4
sharpness: 0
saturation: -2
colortone: 0
So what is the significance of this? Well it might be possible to make a "real" log picture profile that will create proper log H.264 files. I looked into this and yes it is possible thought it might not be practical. DeafEyeJedi mentioned that the editors where he works prefer wide gamut over log and I'm thinking that CineStyle might just be a hybrid log/flat/wide gamut picture style because there's only so much you can do with 8-bit 4.2.0 files. However, if the picture style can also be used on the raw files then you can actually create a true log like Arri Log C picture style.
Now for the bad news. Picture styles are binary files and the PictureStyleEditor is a rather elementary tool that isn't really suitable for creating a proper log curve. Then of course there's the job of figuring out how to interpret the picture style that is embedded in the raw file. I looked at the CineStyle picture style in a HEX editor and can't make heads or tails of it but there is a lot more information in there than what exiftool is reporting.
The picture profile "Cinestyle" was a great effort by Technicolor but the Cost of Killing Skin tone was & is unacceptable.
Been using it with H264 since it came out on my 5D2 & being fighting with Skin tone since that day,(skin was gray-no mid tones) .
ML raw was a godsend for skin tones , and from what I understand about Raw video there is no Picture profile being applied at all.
Will not totally thou because the HDMI & Liveview still relies on the Picture Profile , because of the processing of the Jpeg chip where the Profiles are applied ,
that where the HDMI feed came from and it is truly uncompressed (1.5Gb/s 4:2:2 60i)(check with my AJA Kona Lhi card) so in respect to that
the profile is important ,as from time to time I capture the HDMI to my Ninja for backup of Raw Video I record internally .
With Raw/MLV Video I use "standard picture profile" for exposure , give me a look that is close to a fully graded image.
If a "Log" Picture Profile like "Cinestyle" could be developed that respected Skin Tones , that would be bonus for H264 & HDMI Capture.
Not raw video as its not being applied , at least not to my knowledge.
My 2 cents :D
+1 for starting this thread! [emoji108]
Quote from: dfort on December 07, 2015, 05:50:39 AM
However, if the picture style can also be used on the raw files then you can actually create a true log like Arri Log C picture style.
As far as I understand, all flavors of in-camera Log is a lossy compression of debayered image (and even Adobe lossy DNG too). So, it's not applicable to "real" undebayered Raw.
I've done a fair bit of work with regards to understanding Cinestyle and Picture Styles in general. I do see the point that many users make regarding black level, skin tones and other negatives but like all intermediate log profiles the end result is really about how you handle correction in post.
I'm not going to comment on Technicolor's sudden discontinuing of Color Assist (their own grading app) as I think it was purely a business decision based on numbers but there are a lot of clues about Cinestyle in interviews with it's creator - Josh Pines (find the videos on Youtube).
My findings regarding Cinestyle are pretty much this:
Cinestyle and Picture Styles are essential .ICC profiles. You can unlock Cinestyle using Matlab by modifying the header structure but that will not get you far or make the PS editable in PSE because the tone curve is affected by something further upstream than occurs with other picture styles.
Cinestyle is the only 'correctly' coded log Picture Style profile for Canon DSLRs. That is to say that each stop has the same amount of code values and are mapped using a formula, not some manually applied, arbitrary control points made in PSE. I, and I guess many others, have written to Josh Pines and Technicolor to ask for the exact formula used but Technicolor will not release the info - i.e. it had to be a reverse engineering job.
H.264 is always encoded with legal levels but Cinestyle is full range data.
The log curve of Cinestyle is close to Cineon log but is more a derivative of JPLog in that it is based on pivoted middle grey. JPLog is also from Josh Pines. A full-to-legal pre-adjustment of IRE (as used in the formula for Canon Log) is also 'likely' to be used in the Cinestyle formula.
The delogging lut that Technicolor provide is often confused as being 'Cinestyle to Rec709' and there have been several apps that base Cinestyle support on this. This assumption is wrong because it does not account for the s-curve. It is Rec709 in as much as it fits Cinestyle into a rec709 display space but it's not the actual BT.709 transfer function.
The inverse of the Cinestyle lut is not Cinestyle.
Cinestyle was apparently developed using 'Canon Standard' color but the size of the PS file is very small and leads me to believe that it omits any color profile and is simply a straight forward log transform of white balanced raw data with sRGB primaries.
With H.264 being 8bit, correct exposure for Cinestyle is critical but there is very little in the way of instruction available. Going by the cryptic clues in interview videos with JP, it seems as though ISO320 is a sweet spot but, as DSLRs do not have a high base ISO, it's likely that ISO160 will capture the best DR vs noise ratio.
The recorded 'raised' black level of Cinestyle is one of the main points of user/reviewer criticism but it is there for a reason. H.264 encoder pays less attention to low code values which typically contain signal with the most noise so Cinestyle raises the noise floor just above the point where H.264 encoding will be detrimental to shadows.
Cinestyle was developed by Technicolor for the 5D Mark II BUT (importantly) internal H.264 encoding was never used in development. If you watch any of the show videos where JP is being interviewed about Cinestyle, he points out that the signal is taken from the HDMI port - but afaik the HDMI out on the 5D2 is interlaced?
The captured dynamic range of Cinestyle is actually very impressive (I would say slightly better than 8bit Canon log) but 8bit H.264 encoding destroys a lot of information in the process.
Some tips for shooting Cinestyle:
Most ML users will select Cinestyle as the record PS and set exposure with Neutral but the exposure tools in ML (except the raw histogram and zebras) will be affected by the settings of the exposure PS. In the absence of a better raw waveform monitor it's probably best to expose with raw zebras to ensure highlights are not clipped while using the exposure PS (i.e. Neutral, Standard etc) for checking the exposure isn't being lowered too much BUT you can't currently do this in ML as you would need to enable raw shooting, expose then change back to H.264 - hence my request a while back for this ability.
Nail your white balance before shooting using a white balance card and be sure to set the custom white balance to use the setting or you'll regret it later!
Use the recommended settings for Cinestyle i.e. Contrast -4, sat -2 and NO SHARPENING! Sharpening at this stage will only add artifacts. It's much better to add it in post. High pass or unsharpen masking are a good choice over normal sharpen plugins as they will only enhance the luminance edge contrast leading to less color artifacts and more natural images but don't over do it!
I'm personally a little skeptical of the recommended saturation setting as desaturating before encoding could actually be losing much color information. If the -2 setting is to emulate the appearance of a wider gamut it's not really a good idea because pixels with RGB Values that already have close to no color will become grey and this cannot be recovered.
Editing H.264 can be a pain, especially in DaVinci Resolve so it's a good idea to transcode to ProRes. There is little point using the high-end 444 or XQ options as there is no extra color information to take advantage of so ProRes HQ is fine. http://rarevision.com/5dtorgb/ (http://rarevision.com/5dtorgb/) is still a great option for this.
Contouring or banding artifacts go hand-in-hand with 8bit H.264 so you will see them in gradients. If you have Neat video or another noise reduction plugin that offers Temporal Filtering you can remove or at least reduce banding by setting this option to use more than 1 frame. With Temporal Filtering set to 4 or 5 in Neat Video it usually removes all but the most stubborn banding.
If there is anything else I can think of I'll add it later.
Incidentally, GoPro Protune wasn't a Technicolor invention. It was developed (co-developed) by David Newman of GoPro and was derived from SI-2K log (also David Newman). He's written several articles on it over the years.
The Picture Style number in .cr2 files is just a flag to tell DPE which PS was active (and used for Jpegs). Custom Picture Styles are embedded in the .cr2 and read by DPE.
Great read, Andy, as always and I am actually glad that I still use Rarevision's 5DtoRGB for my H264 recordings (which is a great option) especially with CineStyle as the PS since the results are quite decent even with the Blacks especially coming from only 8-bit in H264.
So is it best not to use this particular PS when it comes to shooting Raw? I usually get better exposure results by eye because of using it. But you have a valid point that Standard or even Neutral will give closer to what it'll look like at the end in Post, right?
Thanks again, Daniel, for starting this wonderful thread and great to hear from you again, Andy!
Your choice of Picture Style doesn't matter when shooting raw. It's only used for Jpeg and H.264. You shouldn't be eyeballing exposure but you should use the raw histo and zebras.
The point I was making is that these raw exposure helpers are not available when shooting Cinestyle H.264. Metering is affected by settings in the picture style. Ideally you would exposure for Cinestyle in the same way as for raw, taking care to not clip highlights but you have to enable raw recording to do that.
Wow, what fantastic insight! Where shall I start? Feel like I dove into the deep end of the pool before knowing how to swim so I'll start at the shallow end with a waterproof camera.
Quote from: Andy600 on December 07, 2015, 12:15:05 PM
Incidentally, GoPro Protune wasn't a Technicolor invention. It was developed (co-developed) by David Newman of GoPro and was derived from SI-2K log (also David Newman). He's written several articles on it over the years.
Interesting, that just adds to my CineStyle conspiracy theory. Here's a link to a Technicolor news release: GOPRO PARTNERS WITH TECHNICOLOR TO INCORPORATE CINESTYLE INTO GOPRO'S NEW PROTUNE FIRMWARE FOR HD HERO2 AT NAB 2012--April 20, 2012 (http://www.technicolor.com/en/who-we-are/press-news-center/press-releases/gopro-partners-technicolor-incorporate-cinestyle-gopro's-new-protune-firmware-hd-hero2-nab-2012) Guess you can't believe everything that's posted on the Internet.
Interesting that Josh Pines developed CineStyle, he is also one of the developers of CDL, Color Decision List, which was awarded a Technical Achievement Award by the Academy of Motion Picture Arts and Sciences in 2012. I haven't worked on a show that used CDL but from what I've heard it is more than just a standardized method for defining Slope, Offset and Power. The CDL is embedded into the metadata of the original file and the CDL travels all the way through post production so when it gets to color grading the DP's original settings when the scene was shot can be recalled. Well, at least that's what I was told at an Editors Guild workshop. The editors were grumbling because they are doing more and more color adjustments before the cut goes to color grading and the first thing the DP does is ask that the colors be set to what he created on set with the DIT, Digital Image Technician. (I'm trying to write this for those who want to follow this discussion without having to look up every acronym I throw out there.)
Of course there's theory and there's practice. From what I heard new technology like CDL and ACES, Academy Color Encoding System, isn't widely used and when it is only certain parts are implemented. In addition, the transition to these technologies don't happen all at once. For example, I'm doing a job at a studio that has recently switched over from using DPX to EXR in their production pipeline as they adapt to the ACES workflow but they have not yet adapted all of the ACES components. (EXR, extended range, files are part of the ACES specification which uses a color gamut greater the visible spectrum and can handle HDR images in a file format that can be losslessly compressed for archival purposes.)
Gee, hope I'm not sounding pretentious--there is a whole lot that I don't know, like deciding a look ahead of time and create a picture style for it.
Quote from: Andy600 on December 07, 2015, 12:15:05 PM
The Picture Style number in .cr2 files is just a flag to tell DPE which PS was active (and used for Jpegs). Custom Picture Styles are embedded in the .cr2 and read by DPE.
Yay! So the proxies for editorial will have the right look and when we get to final grading we can start with the DP's original grade and get creative from there.
But wait, I'm not sure about this:
Quote from: Andy600 on December 07, 2015, 05:47:19 PM
Your choice of Picture Style doesn't matter when shooting raw. It's only used for Jpeg and H.264.
Of course you are right, at least at this point in time because only Canon's software can seem to work with picture styles. Since the metadata is being embedded in not only the CR2 files but also the MLV and the extracted DNG's maybe we can make some good use of this information, if we can get to it and decipher what it means--one of the reasons I posted in the Reverse Engineering section of the forum.
Picture styles can also be useful for recording off the HDMI port:
Quote from: reddeercity on December 07, 2015, 07:12:27 AM
... HDMI & Liveview still relies on the Picture Profile , because of the processing of the Jpeg chip where the Profiles are applied ,
that where the HDMI feed came from and it is truly uncompressed (1.5Gb/s 4:2:2 60i)(check with my AJA Kona Lhi card) so in respect to that
the profile is important ,as from time to time I capture the HDMI to my Ninja for backup of Raw Video I record internally . ...
I have no experience whatsoever with HDMI recorders but I do know all about the confusion between raw as in capturing the unadulterated data straight from the sensor and raw as in the uncompressed video streaming through the HDMI, High-Definition Multimedia Interface. The video coming through the HDMI has a baked in look so if it looks good on the monitor chances are you'll have a hard time if you decide to change the look in post. It would probably be best to put one of the standard camera log gamma curves on the HDMI stream before recording it to ProRes or DNxHD. If you have a log gamma video going into a professional monitor you can apply an output lut so people on the set won't think that the camera is broken.
Ok, so there's yet another reason to figure out how to create picture styles that work.
As a first step I think it would be great to deconstruct the CineStyle profile for those of us who still shoot mostly H.264--what can I say, I'm working on some documentaries. It looks like Andy600 has a good idea of how it works but it would be great to figure out how to reconstruct it from scratch as an open source project.
Quote from: reddeercity on December 07, 2015, 07:12:27 AM
Been using it with H264 since it came out on my 5D2 & being fighting with Skin tone since that day,(skin was gray-no mid tones) .
From what I read about log gamma curves it is all about evening out the distribution of bits to luminescence and not HSL (hue, saturation, and lightness) and HSV ( hue, saturation, and value). I'm going to take Andy600's advice next time I go out to shoot with CineStyle:
Quote from: Andy600 on December 07, 2015, 12:15:05 PM
I'm personally a little skeptical of the recommended saturation setting as desaturating before encoding could actually be losing much color information. If the -2 setting is to emulate the appearance of a wider gamut it's not really a good idea because pixels with RGB Values that already have close to no color will become grey and this cannot be recovered.
This should improve the grey skin tone issue.
Thanks Andy600, not for just this but for all the great tips that you are sharing with the ML community.
So it looks like my todo list just got a lot more ambitious.
So I download the "Cinestyle" Picture Profile from Technicolor Web site again to Check it out and
I never new there was s-curve lut that's going with that profile to de-log it in the download package.
If anyone interested I'll put the whole LUT package (including the Pf2 file & PDF user guide) it in my Dropbox Link below
https://www.dropbox.com/sh/pboi1nziu9rus9f/AABv1_-iJ49ox5W9TfKwEJdka?dl=0
There a .txt file plus a .mga lut file not to sure about that format maybe a 'cube file"
@dfort, Thank you for taking the time to explain the acronyms, sometimes posts on here get so full of acronyms which are impossible to google, that I feel like I am reading hieroglyphs at times.
I only record RAW since ML Raw was unleashed on the world, but I use custom Picture Profiles all the time. Cinestyle has a very good emulation of how the "final" RAW Log image will look like, In my case Cinelog-C to Log-C.
I was very excited when I first learned about Cinestyle those many years ago, so I was doing a lot of reading. I can understand that they don't want to share the ins and outs of it. They spent over a year developing the Profile, so there is some magic in there.
I've tried Marvels flat and Flaaat and VisionTech and whatever flat profile that I could find out there, but none of them come even close to delivering the "dynamic range" of Cinestyle.
So my setup now is always having Cinestyle and two Picture Profiles with a look to it. Because we can not use in-camera LUT's, instead we have the picture profiles and for giving a scene a better cinematic look or feel, its nice to have Custom Profiles. And I can highly recommend the 'Cinema' Picture Profile from Cineplus. It has a very nice and Natural Film look to it, but what I really like about it, is that you can easily "Grade" the shot, in camera by adjusting the WB respectively, the Picture Profile doesn't fall apart like the standard profiles and many other custom profiles I've used, it retains the filmic look but just grades nicely towards the temperature you set. And then changing the G/M towards either Green or Magenta, you can get more or less the exact look you want in camera.
I just wish we could have more than 3 Custom Profiles or atleast be able to change the standard ones.
Just beware of skintones with Cinestyle.
and sorry for straying so much off topic
Googling around I found this interesting blog post by Jorgen Escher (https://colorbyjorg.wordpress.com/2011/01/07/developing-a-new-picture-profile-editor-for-canon-vdslr-cameras/) who worked on one of the other Canon picture profiles out there, Marvels cinestyle (https://marvelsfilm.wordpress.com/marvels-cine-canon/). Yeah, I know--confusing because it has the same name as Technicolor's CineStyle but it is obviously something else. What is interesting is that Marvels has updated their picture style (https://marvelsfilm.wordpress.com/2015/04/22/marvels-cinestyle-for-canon-cameras-page-has-been-updated/) as recently as April 2015.
Another interesting piece of this puzzle is that Jorgen apparently asked for help reverse engineering the Canon .pf2 file format.
QuoteI have real trouble finding reliable information about the color tone curves and HLS mapping. The luminance curve is relatively obvious and "easy", for the rest i need someone to help me with the file format, like Tramel Hudson or someone of that stature.
Jorgen
Tramel Hudson (https://trmm.net/About) started Magic Lantern. I'm not a Magic Lantern historian and I don't know if he is still involved but the source code is still hosted on his bitbucket account. I wonder if he knows what's in the Canon picture style files.
My two cents - I juggle between Technicolor and Marvels. Marvels mostly when the production is DSLR only, but I use Technicolor when I have to mix with other pro video cameras, Somehow it is easier for me to get them look the same. Tough yes - skin tones is problem, I get them first with vectroscope, and technicolor is rather unforgiving g if the white balance is not exactly right.
With all due respect to Jorgen Escher who does know a thing or two, I have to say I'm of the exact same opinion as Mike Seymour of FXPHD in that Jorgen is wrong when it comes to Cinestyle and he wrote a good article to explain why:
https://www.fxphd.com/blog/squeezing-the-most-out-of-a-5d-mk-ii-or-alexa-signal (https://www.fxphd.com/blog/squeezing-the-most-out-of-a-5d-mk-ii-or-alexa-signal)
The argument that is always made is that it is best to make use of 256 code values (8bits) for the tone curve as any less will cause banding (true - but that will happen anyway with this much linear data being compressed into 8bit log). Keeping shadow information in the lowest code values means important shadow detail is in the range where the H.264 encoder will do the most damage and, coupled with sensor noise, it becomes a mess. It is better in terms of noise and blocking artifacts (caused by the encoder) to raise the shadows to just above the noise floor, capture then drop them back down in post.
It's true that a lot of users of Marvel's Cine find it easier to grade than Cinestyle simply because Marvels pegs black at 0 while Cinestyle needs a little more work in post BUT ultimately you will get cleaner images and maximum DR with Cinestyle.
Another important aspect is the gamut. If you record a DSC CamAlign chart and analyze the color patches on a vectorscope (2x), Cinestyle puts primaries and secondaries right where they should be. Picture styles that have been developed in PSE will show hue rotations caused by the base color profile (Neutral, Standard etc etc) - This is where Cinestyle has the edge. It's very accurate to Rec709 primaries BUT we are so used to seeing the prettier colors and warmer skin tones of Picture Styles that we tend to prefer the 'Canon' look.
The base profiles that most Picture Styles are based on are basically 16bit linear 3D luts in XYZ colorspace inside an ICC profile. The XYZ lut is processed in a Canon colorspace they named '6000'. This 6000 colorspace has Adobe 1998 primaries but with a D50 white point instead of D65 (simply because the processing all happens in XYZ which also uses D50 white). After the 'look' is added the inverse of 6000 is applied before the output colorspace is assigned (either sRGB or Adobe 1998). The lut part is crafted to produce a nice 'look', control any out-of-gamut colors and help keep saturation under control in the highlights and shadows - many of the most important decisions about color processing are taken out of your hands and, because the 'look' is not a simple matrix, it is pretty much impossible to undo.
You can see now that with Picture Styles, what comes out is far removed from what goes in - Cinestyle is much much closer to what the sensor captures and is corrected/calibrated for accurate rec709 primaries - this is how most video cameras work.
Quote from: Andy600 on December 09, 2015, 02:30:13 PMKeeping shadow information in the lowest code values means important shadow detail is in the range where the H.264 encoder will do the most damage and, coupled with sensor noise, it becomes a mess. It is better in terms of noise and blocking artifacts (caused by the encoder) to raise the shadows to just above the noise floor, capture then drop them back down in post.
Agreed (somewhat). If you search the doom9 forums from around 04' you will see all the complaining regarding shadow detail. Back then it was mostly people with shitty display devices that were not calibrated, and tended to have the brightness to high. So they could only just spot some
issues, then went and raised the brightness on encoded content to exacerbate the issue.
To be fair, it's not really a problem
per se. To compress content, you must throw some away, and the shadows are an excellent source of content that can be discarded without perception.
The problems arise where people raise the brightness of these shadow areas
after encoding. The encoder isn't a mind reader, it doesn't know that the user wants to raise brightness
after it's compressed the content.
The important part to understand in determining whether raising the brightness before encoding (with cinestyle or whatever) is useful, is to understand your final rendered image. If you want to raise the shadows in post, then raise the shadows before encoding to maintain maximum detail. If you're just going to crush the shadows in post, then raising the shadows before encoding is pointless, and means that the overall quality of your encode will be less, since the compression efficiency in the shadows will be lost.
Mind expanding stuff. Now I'm sorting out LUTs and ICCs and Picture Styles and how they are related.
The Canon PictureStyleEditor was updated to version 1.15.30 on October 28 and it looks like the same Picture Styles file format (.pf2 and .pf3) are being used in their latest cameras. I took a peak into the contents of the Mac version and found several ICC files, probably defining their default picture styles.
(https://farm1.staticflickr.com/593/23273955439_267be972f8.jpg)
Quote from: reddeercity on December 08, 2015, 01:00:28 AM
I never new there was s-curve lut that's going with that profile to de-log it in the download package.
A while back I searched for those LUT's and someone converted the ones provided by Technicolor in .mga and .txt format to other formats including, .3dl, .a3d, .cms, .csp, .cube, .icc and .vf. I looked at the .icc file with Apple's ColorSync Utility and found that the file was created with LUT Buddy. It also had a "Copyright 2007 The Orphanage" in the file so who knows where that file has been!
Anyway the point of this isn't to steal copyrighted material but to figure out the intricacies of .pf2 and .pf3 files. Looks like they could be useful for more than just shooting JPEG stills and H.264 video. I must admit that my original intent was to see if it would be possible to come up with an open source log gamma Picture Style even though it looks like Technicolor CineStyle has that pretty well covered.
@Andy600 -- I thought that a Log Gamma curve needs to be tailored to each individual camera, how is it that Technicolor CineStyle apparently works for all models of Canon cameras?
The Picture Style ICC profiles are the looks and are available for both sRGB and Adobe 1998 colorspaces i.e. FA = Faithful (Adobe 1998), FS = Faithful (sRGB), SA = Standard (Adobe 1998), SS = Standard (sRGB) etc etc
The camera too has an ICC (several actually) which come as binary files denoted by a number, look and colorspace i.e. 6211_ntla.BIN = camera_model/Neutral/Adobe1998. The number denoting the camera model can be found by using a live monitoring app which shows which files are called by an app while it is running. The binary files are included with Canon's SDK but you can't use them without DPP.
re: the S-Curve_for_Cinestyle.mga lut included with Cinestyle. It's actually quite poor in terms of resolution even for 8bit.
The other formats that can be found online (.3dl, .cube etc) are all converted from this low res lut too and won't improve things. The copyright notice indicates that the lut was likely converted with LUTBuddy which was developed by the Orphanage.
re: Cameras and log profiles - yes, ideally each camera model would have a tailored log profile but the difference in sensitivity between Canon DSLRs isn't that great (less than 1/2 a stop). The lin2log transfer function in Cinestyle was developed for the 5D Mark II so we can assume it is for the 5D2's DR and, because it gets applied after white balancing in XYZ space it works with all the cameras. There would be some gain if Cinestyle were developed for each camera independently but it would be very small and not amount to much - it's 8bit anyway so we're splitting hairs here.
@Audionut - I get what you are saying and while that method might achieve better detail and less noise (visually at one end of the signal), it is not really the intention of a log conversion which is merely to compress the linear signal into a smaller space so that it can later be expanded back to something resembling what the camera sensor captured without the influence of curves, contrast etc and with minimal loss. The camera will only capture what is within it's dynamic range capability so it's more about maximizing this linear range so that the log conversion can compress the most information without wasting bits but 14 into 8bits is going to mean high granulation and big gaps in the delogged signal.
I had the idea to add noise to the recorded signal which fills the gaps resulting in a solid histogram and no banding when linearized - it seems to work well and can be done with a lut.
Quote from: Andy600 on December 10, 2015, 02:28:17 PM
The Picture Style ICC profiles are the looks and are available for both sRGB and Adobe 1998 colorspaces i.e. FA = Faithful (Adobe 1998), FS = Faithful (sRGB), SA = Standard (Adobe 1998), SS = Standard (sRGB) etc etc
The camera too has an ICC (several actually) which come as binary files denoted by a number, look and colorspace i.e. 6211_ntla.BIN = camera_model/Neutral/Adobe1998. The number denoting the camera model can be found by using a live monitoring app which shows which files are called by an app while it is running. The binary files are included with Canon's SDK but you can't use them without DPP.
This is good to know...
Quote from: Andy600 on December 10, 2015, 02:28:17 PM
re: the S-Curve_for_Cinestyle.mga lut included with Cinestyle. It's actually quite poor in terms of resolution even for 8bit.
The other formats that can be found online (.3dl, .cube etc) are all converted from this low res lut too and won't improve things. The copyright notice indicates that the lut was likely converted with LUTBuddy which was developed by the Orphanage.
That actually makes sense and hence the reason why I didn't really like it or at least wasn't too fond of the results when combining the S-Curve Lut that came with Cinestyle from their site. Haven't used it again since. I like your products better, Andy!
Quote from: Andy600 on December 10, 2015, 02:28:17 PM
@Audionut - I get what you are saying and while that method might achieve better detail and less noise (visually at one end of the signal), it is not really the intention of a log conversion which is merely to compress the linear signal into a smaller space so that it can later be expanded back to something resembling what the camera sensor captured without the influence of curves, contrast etc and with minimal loss. The camera will only capture what is within it's dynamic range capability so it's more about maximizing this linear range so that the log conversion can compress the most information without wasting bits but 14 into 8bits is going to mean high granulation and big gaps in the delogged signal.
I had the idea to add noise to the recorded signal which fills the gaps resulting in a solid histogram and no banding when linearized - it seems to work well and can be done with a lut.
Point taken. Hopefully this idea of yours will be implemented into the next update for Cinelog-C and hopefully soon by the Holidays! 8)
Quote from: Andy600 on December 10, 2015, 02:28:17 PM
The Picture Style ICC profiles are the looks and are available for both sRGB and Adobe 1998 colorspaces i.e. FA = Faithful (Adobe 1998), FS = Faithful (sRGB), SA = Standard (Adobe 1998), SS = Standard (sRGB) etc etc
Interesting, now we're getting somewhere in this reverse engineering thing-a-ma-jig. So it seems that a custom picture style is based on one of the standard picture styles? Maybe with the exception of Auto and Monochrome? The Canon apps use the icc to get into the ballpark then apply the custom parts of the picture style? That's probably why other apps don't display custom picture styles properly on raw files.
Quote from: Andy600 on December 10, 2015, 02:28:17 PM
re: the S-Curve_for_Cinestyle.mga lut included with Cinestyle. It's actually quite poor in terms of resolution even for 8bit.
You know this better than I but rumor has it that colorists are removing the S-Curve_for_Cinestyle lut before applying their own adjustments. Maybe if the lut were higher quality that wouldn't be happening?
Quote from: Andy600 on December 10, 2015, 02:28:17 PM
I had the idea to add noise to the recorded signal which fills the gaps resulting in a solid histogram and no banding when linearized - it seems to work well and can be done with a lut.
Some people are transcoding their H.254 footage before editing. On one of my shoots I removed the objectionable noise with Neat Video then added in a little film grain before exporting to ProRes 444 HQ and wow, what a difference. Of course it took a long time to process but it was worth it.
Anyway, I'm typing this at work and need to wrap up. I'll dive into this deeper later but it seems that maybe creating an open source log camera picture style and associated luts might not be such a bad idea after all?
QuoteThe Picture Style ICC profiles are the looks and are available for both sRGB and Adobe 1998 colorspaces i.e. FA = Faithful (Adobe 1998), FS = Faithful (sRGB), SA = Standard (Adobe 1998), SS = Standard (sRGB) etc etc
On a related note, could these ICC profiles be used to create a Digital Camera Profile for use in Adobe Camera Raw that would let me match a .cr2 to the color look of the in-camera JPG?
Interesting question. If compiled with "gcc -o dcraw -O4 dcraw.c -lm -ljasper -ljpeg -llcms2" dcraw has these options:
-p camera.icm [ -o output.icm ]
Use ICC profiles to define the camera's raw colorspace and the desired output colorspace (sRGB by default).
-p embed
Use the ICC profile embedded in the raw photo.
Don't know how it works. My understanding is that there is an ICC profile embedded inside of CR2 files but only Canon's software seems to be able to make any use of it.
Somewhere in one of the ML threads I think I found this link - or was it somewhere else? http://lclevy.free.fr/cr2/
At any rate it confirms that ICC data are embedded in CR2 files.
Ah I knew it ... This is definitely good news and should point us into the right direction. Thanks for sharing @axelcine!
Quote from: andy600
The Picture Style number in .cr2 files is just a flag to tell DPE which PS was active (and used for Jpegs). Custom Picture Styles are embedded in the .cr2 and read by DPE.
Andy also confirmed this, and I saw it myself by looking at a cr2 shot with Cinestyle and loading it into DPP - it correctly extracts the profile for further processing, and that software also allows you to select a new custom picture style for after-the-fact applying to cr2s.
I would love to know why Adobe wasn't able to use Canon's ICC profiles for creating their Digital Camera Profiles in ACR. I remember reading that they had to reverse engineer the look of each one in the standard set (ie Portrait, Landscape, Neutral, etc.), and the results never match Canon's in-camera or DPP processing of the same image in terms of color. Since ACR's raw processing is so much better in all other aspects, I wish there was a way for it to match Canon's color for cases where their picture style is the exact look desired.
Quote from: Andy600 on December 10, 2015, 02:28:17 PM
re: the S-Curve_for_Cinestyle.mga lut included with Cinestyle. It's actually quite poor in terms of resolution even for 8bit.
Any suggestions on what to use instead?
Edit: I forgot about your cinelog-c, I assume that's what you refer to?
Quote from: eyeland on January 27, 2016, 02:52:11 AM
Any suggestions on what to use instead?
Edit: I forgot about your cinelog-c, I assume that's what you refer to?
Picture Styles are loaded into the camera to give custom looks. Cinestyle is a log Picture Style. Think of it like the log profiles available for Black Magic, Panasonic and Sony cameras. Canon also has log Picture Styles built into their professional cine cameras.
LUTs can be applied to footage that was shot in a log Picture Style to change it to a standard color space like say Rec709 or it can apply various "looks" to the footage.
The S-Curve_for_Cinestyle.mga changes Cinestyle shot footage to Rec709. What Andy600 was talking about (I believe) is that the LUT could be of higher resolution. Note that Cinestyle was released in 2011 and there was no further development since. Meanwhile technology in this area has progressed by leaps and bounds. Well, you know what I mean--we should be able to create a better log profile for our cameras. Thing is, Cinestyle is closed source, locked up and somebody seems to have swallowed the key.
Quote from: dfort on January 27, 2016, 03:40:15 AM
Note that Cinestyle was released in 2011 and there was no further development since. Meanwhile technology in this area has progressed by leaps and bounds. Well, you know what I mean--we should be able to create a better log profile for our cameras. Thing is, Cinestyle is closed source, locked up and somebody seems to have swallowed the key.
Well said, Dan and yes I also agree w the fact that we should all come up w some kind of an original ML Log for our Cameras and perhaps as well our own set of ML Color LUT's to apply them on top together in post ... Man imagine what that would be like for us?
@Dfort
I might have gotten slightly confused as it was late night when I posted :)
I understand the difference between "flatting" the image in-camera vs ACR (or similar).
I still use h264 with cinestyle on occasion as I have been doing some super-low-budget stuff for friends recently.
To word my question differently, is there a better S-curve lut for cinestyle than the low resolution one mentioned by Andy?
Or perhaps also a better flat picture style all together? (I know that most of the effort concerning h264 died out after ML raw appeared, but I suspect that many people still use canons h264 regularly)
@eyeland
I would not say that we need a better flat Picture Style though we could use an open source log gamma curve Picture Style. Although a log gamma Picture Style will look flat, a flat Picture Style doesn't necessarily mean that it has a log gamma curve.
Although there has been some debate about this it does seem that Cinestyle was designed as a log Picture Style. Sure, there are users that prefer other picture styles because they are easier to grade. This also happens in professional environments where camera operators for news and sports programs are told to use a "Wide Dynamic Range Gamma" setting instead of a log setting because it is quicker to grade--if it gets graded at all before going to air.
As far as a better S-curve lut for Cinestyle, I believe most people aren't even using the lut originally provided by Technicolor and instead are creating their own custom settings. Andy600 also adds some noise in his setting and I agree that it helps smooth out the notorious 8-bit banding issue.
For the most part the latest DSLR's still don't feature recording video to raw, H.265 or even ProRes. H.264 is still very much alive. (Slightly off topic the newly announced Nikon D5 records up to 3 minutes of 8-bit H.264 video--wow. :o)
@dfort Yea, that's more or less what I have been doing, guess I'll stick to tweaking curves as needed until I can convince my clients to go the extra mile with ML Raw :)
Is it possible to convert ICC to PF2?
Quote from: markanini on June 02, 2016, 07:31:20 PM
Is it possible to convert ICC to PF2?
I can't find a published document on the Canon Picture Style file format so for now I'd say that isn't possible. I'd love to be proved wrong on this.
Updated first post (http://magiclantern.fm/forum/index.php?topic=16299.0) with release announcement of log picture styles.
In addition to the posted picture styles above I would like to post a few comparison examples.
logNeutral
(https://s15.postimg.org/a6doxw9cb/01_log_N_noodle.jpg)
cinestyle
(https://s15.postimg.org/f8zm6f80b/02_cine_noodle.jpg)
logStandard
(https://s13.postimg.org/q9xxfeg9z/log_Standard_noodle.jpg)
logNeutral
(https://s15.postimg.org/3reo182mj/01_log_N_house.jpg)
cinestyle
(https://s15.postimg.org/jth77m2bv/02_cine_house.jpg)
logStandard
(https://s13.postimg.org/6e1xzuz8n/log_Standard_house.jpg)
logNeutral
(https://s17.postimg.org/6h4byn6jz/image.jpg)
cinestyle
(https://s17.postimg.org/vof7yw9nz/image.jpg)
I will most definitely try this out on my 7D2. I record using the HDMI out to the Atomos Ninja 2 so this should be fun to experiment with. Thanks guys!
Danne and I have been doing a lot of testing with these and we think they are very close--almost perfect. If anyone can do better, please share.
CineStyle
(https://c4.staticflickr.com/9/8302/29294092091_8e0e66c21a_z.jpg)
logNeutral_v1.1
(https://c8.staticflickr.com/9/8168/28751407143_577c6e587f_z.jpg)
There is lots of information available from Canon, Sony, Panasonic, Arri, (any others?) about shooting log but the documentation for the original Technicolor CineStyle is very sparse.
QuoteBased on Technicolor's usage and testing of its CineStyle™ Profile, we
recommend the following camera settings to optimize the image quality of your
Canon EOS camera:
Sharpness: 0
Contrast: -4
Saturation: -2
Color Tone: 0
ISO: a multiple of the camera's native ISO
(i.e. a multiple of 100 or 160 depending on the camera)
CineStyle loads up with the recommended Sharpness, Contrast and Color Tone as defaults but the Saturation is set to 0. Since our first goal was to make as close a copy as possible these picture styles also load up the same way as the original. According to some users who have lots more experience that I do about these things turning down the Saturation may not be the best recommendation.
In addition, the shooting on multiples of ISO settings recommendation is rather vague. According to some users who tested their cameras there is a multiple that is so called "native" another that is pushed or gain up while another is pulled or gain down. Pushing increases noise, pulling reduces noise while the "native" setting has the most dynamic range. It seems that if you're using a log profile you're probably more interested in dynamic range.
Some professional cameras limit white balance choices when shooting log and technical specifications lists a limited range of recommended ISO settings. Take for example this chart from Arri to illustrate why setting the ISO to 800 on the Alexa is recommended:
(https://www.arri.com/uploads/RTEmagicC_faq_s_3_greyscale_03.jpg.jpg)
Does this apply to our Canon DSLR's? I've got more questions than answers after starting this project.
Quote from: dfort on September 01, 2016, 03:36:17 AM
Take for example this chart from Arri to illustrate why setting the ISO to 800 on the Alexa is recommended:
[...]
Does this apply to our Canon DSLR's?
This chart appears to show an ISO-less sensor (not sure how to read it, but the increments at both top and bottom ends match the ISO increments, +/- 0.1 stops). Usually, with such a sensor you can just shoot at the lowest ISO and increase the brightness in post as you need - there is little reason to use a higher ISO, as the shadow details will not get significantly better (they will get a little better, but not much).
On high-end cameras, I remember reading ISO is just metadata; if this is the case, it really doesn't matter which one you use. Or maybe it affects the exposure indicators and that's it (just a guess, as I have zero experience with these cameras).
Canon sensor is different - if you look at a dynamic range chart, you will see that, until about ISO 400, the DR does not drop much. Remember ISO is defined from the clipping point. Therefore, increasing ISO helps capturing more shadow detail. However, at higher ISOs (around 1600), Canon sensor becomes nearly ISO-less - there's little to gain by pushing ISO beyond that.
So, on an ISO-less sensor, you can just lower the ISO to protect your highlights without worrying much about the shadows, while on Canon sensor (especially on full-frame), you need to find a balance between how many highlights you clip (when increasing the ISO) and how many shadow details you capture.
Side note: with dual ISO, Canon sensor behaves a lot like an ISO-less sensor (so, when the light changes a lot, I just set it to 100/1600 and forget about it).
I found the FAQ for Technicolor CineStyle cinestyle-faq (http://www.technicolor.com/en/solutions-services/cinestyle-download/cinestyle-faq)
So as far as ISO is concerned it's recommended to use ISO of multiple of 160, so that would be a push @ 100 but at 360 is pulled down 400 .
So digital ISO's have more D.R. then Analog ISO's e.g. 100,200 etc..... . in the Compressed internal pipe line ?
But in Raw Video this is not the case as we know that the Analog ISO 100,200,400 etc.... have more D.R. & cleaner a much lower noise floor.
We know the Video stream for HDMI & H264 comes from the Jpeg chip , which I understand is YUV full range 0-255 .
The reason I mention it because I notice with the log picture style in the PSE (Picture Style Editor) I see the range is clipped at 16-235 ,
not sure if that will cause any limited D.R. Just a thought.
I also in may search for info on this thread topic I came across this alternative Cinestyle Look up tables (http://www.magiclantern.fm/forum/index.php?topic=9782.msg94220#msg94220) (January 03, 2014) on the ML forum ,
look like you're not the only one @dfort that's looking for a Cinestyle workflow for compressed video. :D
I downloaded the lut pack , the image below shows all the formats (ICC , Cube , etc... )
(http://movies.online-arts.de/Files/aftertest.png)
says he or she took Cinestyle .mga and converted to Cube , ICC etc.. also say "Please keep in mind, that I take no responsibility for 100% correct Look up tables".
Hope there some useful information in there to work with .
Creating a simple 3D lut upon the S-curve isn,t hard. Is there an inverse somewhere in cube format? Would it even work. People complaining the S-curve being too simple.
From my understanding the .mga file is from Technicolor and they use a s-curve to transform back to Rec709.
So it may be that easy , don't know unless you try.
I'm still trying to found more information , on all this thou.
Could you inverse the S-curve numbers for me Reddeercity? My mathbrain is not working.
is there anyone tried the clog someone posted here not long ago?
just got my ass out yesterday to test the clog on 6D,
its very close to cinestyle, just slightly boost on saturation and slightly underexpose then cinestyle.
clog:
(http://ph-production.com/tmp/clog1.jpg)
cinestyle:
(http://ph-production.com/tmp/cine1.jpg)
(grade just apply default premiere Alexa_Default_LogC2Rec709 lut)
clog graded:
(http://ph-production.com/tmp/clog1_g.jpg)
cinestyle graded::
(http://ph-production.com/tmp/cine1_g.jpg)
clog:
(http://ph-production.com/tmp/clog3.jpg)
cinestyle:
(http://ph-production.com/tmp/cine3.jpg)
(grade just apply default premiere Alexa_Default_LogC2Rec709 lut)
clog graded:
(http://ph-production.com/tmp/clog3_g.jpg)
cinestyle graded::
(http://ph-production.com/tmp/cine3_g.jpg)
@Danne - here you go: https://s3-us-west-2.amazonaws.com/ofxpublic/Cinestyle_S-curve_1D_fwd_and_rev.zip (https://s3-us-west-2.amazonaws.com/ofxpublic/Cinestyle_S-curve_1D_fwd_and_rev.zip)
I've got the s-curve function somewhere that can reproduce the curve exactly but I need to find it (on an old drive).
Cinestyle is not the inverse of the S-curve though the inverse could be used to create a flat PS by multiplying the float values by 256 - though you only get 10 control points in the PSE.
Great work with your PS. It looks bang-on in DPP on my monitor. I'll have a look on a step wedge to see if the shadows are holding up - that's where things can get tricky but it looks very good.
BTW I don't think .pf3 is compatible with older cameras (50D, 5D2 etc)
Thanks Andy600. It,s been quite the challenge to obtain closest match especially in shadows as you mention. Dfort had test charts to verify a lot of the ground work. For subtle luma/color changes I had to create some extreme lighting conditions, bright lamps etc. I think it turned out alright. Of course with a 10 point curve tool moving only one step will often alter the overall look so getting the last subtle matching points was a lot of work. I welcome any closer inspections and refinements possible on the subject. Thanks for the 1D luts. Will check those later.
I bet dfort will put up pf2 versions soon :). Meanhwile it,s possible to open up the pf3 and resave them to pf2 in picture style editor.
Wow, this topic came back to life. Ironically we introduced the new picture styles on the three year anniversary when the Technicolor CineStyle Facebook page (https://www.facebook.com/TechnicolorCineStyle/) was last updated.
One of the reasons why I got back to this project was the announcement of the 5D Mark IV. Although some people are underwhelmed by the specs it is still destined to become a popular camera. If or when Magic Lantern will be ported to it is something we won't even discuss at this point but having a log picture style that can be tweaked for 4K mjpeg video is certainly within our reach. Note that early reviewers have reported that C-Log is not available for the 5D4. Slightly off topic the 1.74x crop factor when shooting 4K is close enough to the Super 35 sensor used in the C100, C300, C500 that 5D4 users might consider doing hardware hacks on EF-S lenses like the popular Canon EF-S 17-55mm f/2.8 IS USM Lens.
Ok, back on topic. I'll use .pf2 for the next release so we don't leave out owners of some older camera models. The files are much smaller and we aren't using any of the advanced .pf3 features anyway. First I want to revisit the highlights. The 1.1 release improved on the shadows but it looks like 1.0 might have been a slightly better match to CineStyle in the highlights. These are minute details that can't be seen by eye but show up on a waveform monitor.
@ph2007 - We looked at that new C-Log picture style (http://magiclantern.fm/forum/index.php?topic=17730) that @Chungdha posted. We would certainly like to know more about it. It is locked for editing and he hasn't answered questions that users posted so we don't know if he wants to play with us.
@a1ex - Thanks for the technical insights. You gave us a lot to think about.
@reddeercity @ddelreal - It will be interesting to see if we can eventually make a log picture style that is optimized for external recorders off the HDMI output.
And most of all--thanks @Danne, you did all the hard work on this.
Quote from: a1ex on September 01, 2016, 08:03:36 AM
This chart appears to show an ISO-less sensor (not sure how to read it, but the increments at both top and bottom ends match the ISO increments, +/- 0.1 stops). Usually, with such a sensor you can just shoot at the lowest ISO and increase the brightness in post as you need - there is little reason to use a higher ISO, as the shadow details will not get significantly better (they will get a little better, but not much).
On high-end cameras, I remember reading ISO is just metadata; if this is the case, it really doesn't matter which one you use. Or maybe it affects the exposure indicators and that's it (just a guess, as I have zero experience with these cameras).
Canon sensor is different - if you look at a dynamic range chart, you will see that, until about ISO 400, the DR does not drop much. Remember ISO is defined from the clipping point. Therefore, increasing ISO helps capturing more shadow detail. However, at higher ISOs (around 1600), Canon sensor becomes nearly ISO-less - there's little to gain by pushing ISO beyond that.
So, on an ISO-less sensor, you can just lower the ISO to protect your highlights without worrying much about the shadows, while on Canon sensor (especially on full-frame), you need to find a balance between how many highlights you clip (when increasing the ISO) and how many shadow details you capture.
Side note: with dual ISO, Canon sensor behaves a lot like an ISO-less sensor (so, when the light changes a lot, I just set it to 100/1600 and forget about it).
I believe the link between high ISO and extra highlight latitude is based on choice of exposure for a given scene. It's certainly not intuitive (especially from a stills perspective) but there is a logic there. In the previously shown chart for Alexa, the premise is that the scene exposure level is the same for every ISO rating, and every change in ISO is offset by changing the amount of light that hits the sensor (via aperture, shutter speed, shutter angle, ND filters, external light levels, etc.). Therefore, to match the correct exposure level with a low ISO you need a lot of light entering the camera which is great for shadows but also means the sensor hits saturation sooner, therefore highlight latitude is reduced. On the other hand, exposing with a high ISO means less actual light entering the camera, so less latitude in the shadows but more in the highlights. This is also based on a Raw workflow, where the ISO is indeed just metadata that can be adhered to, altered, or removed in post. 800 ISO is the base for Alexa for practical reasons.
Dual-ISO is in deed quite similar. Shooting 100/1600 and exposing for 1600 (to gain an extra few stops highlight latitude in post) is much like a DP choosing an ISO of 1600 (with a 1 stop ND filter attached to the lens) in order to gain an extra stop of highlight latitude.
Quote from: Danne on September 01, 2016, 02:26:04 PM
I bet dfort will put up pf2 versions soon :). Meanhwile it,s possible to open up the pf3 and resave them to pf2 in picture style editor.
So saving the pf3 as a pf2 will show this warning:
(https://c2.staticflickr.com/8/7505/29392076945_02e5885e75_z.jpg)
And poof! There goes our tone curve. I'm going to move the nodes over to the tone curve that's on the "Specific Colors" tab in order to save a pf2 version. Let's see if that doesn't mess up all the work we've done on our curve.
Moving to the pf2 curve will likely produce a different result than you get with the pf3 tone curve. The pf2 curve is applied after white balance and will probably upset the color balance. Canon could easily enable pf3 support in older cameras if they wanted to :(
Edit: Just tried a pf3 in a 50D and the tone curve works - so it should work in the 5D2 too :)
Quote from: Andy600 on September 02, 2016, 12:36:02 AM
Edit: Just tried a pf3 in a 50D and the tone curve works - so it should work in the 5D2 too :)
Great because copying the nodes from the Tone curve (RGB) didn't work. Using the pf2 curve was like starting over, only more difficult.
@Andy600 -- A while back you made a comment about the -2 saturation setting recommended by Technicolor:
QuoteI'm personally a little skeptical of the recommended saturation setting as desaturating before encoding could actually be losing much color information. If the -2 setting is to emulate the appearance of a wider gamut it's not really a good idea because pixels with RGB Values that already have close to no color will become grey and this cannot be recovered.
I compared saturation settings 0, -1 and -2 and applied the Technicolor CineStyle lut that ships with DaVinci Resolve. Looking at the RGB Parade it seems to me that the -2 saturation is a much closer match to a reference shot with the Neutral picture style. Perhaps the -2 setting accounts for the increased saturation when the gamma is adjusted in post?
I was thinking of saving the default -2 saturation setting on the next version.
The reason I'm bringing this up is because I've often loaded up CineStyle on a new camera and have forgotten to set the saturation to -2. The CineStyle picture style defaults to 0. Strange, because you would think that the recommended settings would be saved as the default. When I throw on a lut for a rough idea of what I shot it often seems a bit too saturated on footage shot on the 0 setting.
[Edit] New release, v1.2 (http://magiclantern.fm/forum/index.php?topic=16299.msg158364#msg158364). Managed to combine the highlights from v1.0 with the shadows of v1.1 to make an even better match. Saturation now defaults to -2 so users won't have to remember to change that setting if they want to use Technicolor's recommendations. Note that Sharpness 0, Contrast -4, Saturation -2 and Color tone 0 is also the way to adjust the Neutral picture style to Prolost Flat (http://prolost.com/blog/2012/4/10/prolost-flat.html), a style that has been around about as long as CineStyle.
This was posted 11 hours ago on Instagram by Phillip Bloom very interesting James Miller C-logV1.2 (https://www.instagram.com/p/BJ0o3unD7qk/)
Here's James Miller's Instagram (https://www.instagram.com/p/BJ0tJzcjk-S/?taken-by=jamesmiller72) . Can't find it anywhere yet , I wonder if that C-log Pic Style
is from him? I'm trying to get my hands on it .
Great work on this one dfort. Not sure if I would dial down the saturation though since technicolor doesn,t even if they recommend it. It,s 8bit color. We cannot afford it. Better to loose it in post?
I still have one CR2 which shows a slight, slight shift so I guess we have one more version to try and achieve. But it,s the closest I,ve seen.
Comprison done with saturation put back to original state
cinestyle
(https://s10.postimg.org/dw7axod21/Screen_Shot_2016_09_02_at_07_05_19.png)
v.1.2
(https://s10.postimg.org/ox2g2p5ax/Screen_Shot_2016_09_02_at_07_05_40.png)
v.1.1
(https://s10.postimg.org/q0mkenpy1/Screen_Shot_2016_09_02_at_07_06_39.png)
some more information , http://www.similaar.com/foto/flaat-picture-styles/technical-tests-1.html
download link http://www.similaar.com/foto/flaat-picture-styles/download.html
This looks very interesting I like the tests results , I'll have to give it a try.
edit:
Flaat Picture Styles: the long explanation , http://www.similaar.com/foto/flaat-picture-styles/long-1.html
Why not test the reverse engineered logs we created here? Especially those versions with other canon color settings baked in like portrait, faithful, standard etc? As we keep all information open with pf3 format unlike most of the other flat pic styles we can all follow what,s going on and build upon the process.
Think I got a version even closer to cinestyle here. Have look.
https://drive.google.com/file/d/0B4tCJMlOYfirbzhEOVhCTUJIQ2M/view?usp=sharing
Cinestyle
(https://s13.postimg.org/m4wrfwnwn/01_bcine.png)
new build
(https://s13.postimg.org/pahd646iv/01_blog.png)
v.1.2
(https://s13.postimg.org/88oj40rnr/01_blog3.png)
cinestyle
(https://s13.postimg.org/xxukk4kcn/01_acine.png)
new build
(https://s13.postimg.org/3rw65cdfr/01_alog.png)
v.1.2
(https://s13.postimg.org/g590c9347/01_alog_3.png)
I just play with clog more today with the 6D.
is possible the picture style affect camera codec?
I just find out the clog is more better to hold up details than any other picture styles
I have been used 6D quite some time, I know in this scenes the h264 in camera codec would not hold up the details no matter what I do.
please have a look here is the normal result I normally get
cinestyle graded no post sharpness:
(http://ph-production.com/tmp/cine_s.jpg)
(http://ph-production.com/tmp/cine_s2.jpg)
but now have a look for the clog no post sharpness!
(http://ph-production.com/tmp/clog_s.jpg)
(http://ph-production.com/tmp/clog_s2.jpg)
here is the full pic if anyone want check the differences.
http://ph-production.com/tmp/cine.jpg
http://ph-production.com/tmp/clog.jpg
it look more more better for me.
I shoot both these shoot is using all same setting and the camera on tripod. just changing the picture style between shoot.
I will do more testing next week to see how the clog behave.
btw the clog default is 1 sharpness in camera setting and the cine is at 0, will the 1 step made the a such differences?
If the clog profile really affect the details, then thats why canon holding up so long and not easy giving it to any of thier camera including the 5d4 and 1dx2? :o
Turn off sharpness. Post results.
In camera sharpening not usable for filming.
Quote from: Danne on September 02, 2016, 09:30:34 AM
Think I got a version even closer to cinestyle here. Have look.
I checked it out on the DaVinci Resolve scopes and it looks like a closer match to CineStyle.
Packaged it up and posted v1.3 (http://magiclantern.fm/forum/index.php?topic=16299.msg158364#msg158364).
Quote from: ph2007 on September 02, 2016, 01:24:16 PM
If the clog profile really affect the details, then thats why canon holding up so long and not easy giving it to any of thier camera including the 5d4 and 1dx2? :o
Log is included in Canon's cinema line of cameras. The 1D X Mark II is very similar to the 1D C but the 1D X is for still photographers while the 1D C is part of the cinema line. The 1D C has a log picture style built in but the 1D X Mark II does not. Note that the log settings in professional cinema cameras are built into the firmware so you can't just copy Canon Log from a 1D C and install it on your camera.
(https://c7.staticflickr.com/9/8528/29298349462_e9a3c94efb_z.jpg)
I don't want to get into a long discussion about a camera that isn't supported by Magic Lantern but something interesting that I found in the manual is this showing the default settings for Canon Log on the 1D C.
(https://c1.staticflickr.com/9/8394/29372601856_517ef82f58_n.jpg)
Note that Saturation is set to 0, not the -2 recommended by Technicolor for CineStyle and used on several of the popular flat picture styles. I set the Saturation to -2 on the latest releases because I keep forgetting to switch it from 0 when installing CineStyle. Maybe we shouldn't be using -2? I'd like to see some comments on Saturation settings.
Danne and I are pleased with how well our picture style matches CineStyle. It isn't a perfect match but it is very close. Here is a shot of logNeutral with the DaVinci Technicolor CineStyle lut:
(https://c3.staticflickr.com/9/8326/29118992410_3c9fa41451_z.jpg)
You can also create custom luts in Resolve. If you have a one of the standard color charts it can automagically create a fairly accurate lut.
(https://c2.staticflickr.com/9/8149/28716795993_aecb4d2978_z.jpg)
Here is an interesting article on how to create a viewing lut for your on set monitor using this feature in Resolve:
http://www.matchstixstudios.com/blog/2015/12/10/how-to-create-your-own-custom-luts-in-under-5-minutes
The problem is that CineStyle isn't one of the "Source Gamma" settings so I just used "Auto." I applied it to images shot at Saturation 0 and -2 and it brought the colors closer together, though there is still a slight difference that is easily visible on the video scopes.
Of course all this is rather technical and the goal of shooting in log is to allow you more room to manipulate the image creatively in post.
We're using CineStyle as our gold standard for these picture styles because, hey it was developed by Technicolor, it supported by DaVinci Resolve, Cinelog-C (http://www.cinelogdcp.com/) and there are lots of luts for it that can be freely shared. At one point Technicolor had big plans for CineStyle including a special software for it like Color Assist (http://blog.vincentlaforet.com/2012/11/27/technicolor-cinestyle-color-assist/) but all this has since been abandoned. Yet CineStyle is still in wide use.
I'm sure a lot of work went into the new Canon DSLR C-Log (http://magiclantern.fm/forum/index.php?topic=17730) as well as the flat picture styles like Marvel (https://marvelsfilm.wordpress.com/marvels-cine-canon/), Flatt (http://www.similaar.com/foto/flaat-picture-styles/), VisionColor (http://www.vision-color.com/) and even Prolost Flat (http://prolost.com/blog/2012/4/10/prolost-flat.html). Use whatever works best for you.
https://m.facebook.com/story.php?story_fbid=10153668741410426&substory_index=0&id=177852000425 (https://m.facebook.com/story.php?story_fbid=10153668741410426&substory_index=0&id=177852000425)
philip bloom also writes about clog curve in a picture style, coming from miller, you can follow the discussion on facebook
Quote from: Lars Steenhoff on September 02, 2016, 08:18:31 PM
philip bloom also writes about clog curve in a picture style, coming from miller, you can follow the discussion on facebook
Yeah, I've seen that announcement but where can you get it? I can't find any information on where to download it, is it free, is it locked, etc.
(https://c1.staticflickr.com/9/8664/28785710944_c43917031a.jpg)
Jumping off topic, can anyone verify that the Sigma 17-50 f2.8 has an EF and
not an EF-S mount? This might make it a good choice for the 5D4 when shooting 4K (cropped).
[EDIT] Looks like only Canon makes EF-S mount lenses so that Sigma lens should fit on a 5D4.
Ok , Tried to load up the picture profile "pf3" on 5D2 and all I get is errors "can't be installed" but the "pf2" I have no problem at all.
If you try to just save it as a "pf2" you lose the Tone Map , so I spend a few hours trying to match the "logNeutral" I come close , looks not bad .
I used as CR2 file for calibration . I have the custom picture style in my Drop Box if anyone what try it out logNeutral_5DMrk2_v1.4 (https://www.dropbox.com/s/rde427jwcavkjxg/logNeutral_5DMrk2_v1.4.pf2?dl=0)
This all Started so I could do a HDMI Capture test with different picture style profiles install , first I used Canon Neutral then Canon Land Scape capture the uncompressed
60i HDMI stream to my AJA kona LHi first with ProRes 422HQ , then DPX 10BE RGB and of course internal ML H264 with constant bite rate of 2.3 over factory default .
The neutral PS was about 60Mb/s & the Landscape was 88 Mb/s @ 23.976p .
I'll also test Technicolor CineStyle , Flaat11, and C-log HTP
The other problem I had was you can not install picture style profile with ML running , had to use a non ML CF card to go forward , FYI
I post the result later on in a New thread , as this will take some time to complete .
:)
So pf3 files work with the 50D but not on the 5D2? Darn it. How are you loading the picture styles? I thought the only way was using the EOS Utility via a usb cable. It works even with no card in the camera.
I see you copied over the nodes to the "other" graph. I tried that too but it didn't match as close as we would like and it takes a long time to fiddle with the nodes to get things to line up.
I'm going on a vacation in a few days and will be AFK (away from keyboard) for about a month. I try to check up on this topic's progress but I'm not promising anything. In the meantime it seems that this is a pretty good CineStyle "like" picture style that works for most cameras except the one that CineStyle was originally developed on. Yeah, according the Technicolor website it was developed on the 5D2.
Quote from: dfort on September 03, 2016, 02:39:22 AM
So pf3 files work with the 50D but not on the 5D2? Darn it. How are you loading the picture styles? I thought the only way was using the EOS Utility via a usb cable. It works even with no card in the camera.
yes I used the EOS Utility with the USB cable , didn't know you could load picture style without a card never tried that way before.
If the cam is booted up with magic lantern running the "Register User Defined Style" tab is grayed out.
Just tried to load pic. style without CF card and yes it works , retried to load up your PF3 and no Joy ! say it's the wrong version . :-\
Quote from: dfort on September 03, 2016, 02:39:22 AM
I see you copied over the nodes to the "other" graph. I tried that too but it didn't match as close as we would like and it takes a long time to fiddle with the nodes to get things to line up.
Yea , it's not perfect I could spend more time on it . I seem to be off in the mid's a bite It seems I can't lower the saturation or it's too contrasty , I'll have to do more work on it.
Quote from: dfort on September 03, 2016, 02:39:22 AM
I'm going on a vacation in a few days and will be AFK (away from keyboard) for about a month. I try to check up on this topic's progress but I'm not promising anything.
Have a great Vacation !
Quote from: reddeercity on September 03, 2016, 06:29:51 AM
Just tried to load pic. style without CF card and yes it works , retried to load up your PF3 and no Joy ! say it's the wrong version . :-\
Maybe it is the EOS Utility and not the camera?
Changes for EOS Utility 2.12.3 Updater :
Supports EOS-6D camera
Supports read-in of Picture Style files (.pf3) created in Picture Style Editor 1.12.2 and laterSent from my iPad using Tapatalk
Quote from: dfort on September 02, 2016, 08:50:24 PM
Yeah, I've seen that announcement but where can you get it? I can't find any information on where to download it, is it free, is it locked, etc
download link has not been provided by james miller yet. as of now nobody knows whether its going to be free or not
More important. How was it created and if done in primitive picture style editor it should be considered a mimic clog which of course could be more or less accurate depending on how much effort was put into it.
@reddeercity
This is the EOS Utility version that I'm using. (2.14.20)
(https://c1.staticflickr.com/9/8258/28802613624_536b533337_n.jpg)
The reason I bring this up is because when I was working on the picture styles I switched to from an EOSM to a T5i (700D) as my test camera and I couldn't open up the CR2 files in Canon's Digital Picture Professional. It turns out that I had an older version that didn't support that camera.
I searched all over and can't find anything that specifically states that the 5D2 can't use pf3 files so I'm thinking it might be the EOS Utility--or the camera firmware.
@reddeercity.
Not sure, maybe helps. Didn,t check.
http://magiclantern.fm/forum/index.php?topic=17808.msg171753#msg171753
Not sure if i remember correctly, but a stupid mistake I've made lots of times with EOS utility is that the liveview must be off or something like that, in order for eos utility to work properly with PP.
Sure most of you know this, its just something I've been stuck with a couple of times when installing pp's
Found the problem I'm having
(https://c1.staticflickr.com/9/8334/29435811535_cbb3212055_o.png) (https://flic.kr/p/LR9eeZ)
EOS_Untiliy_Ver_2.6.0.0 (https://flic.kr/p/LR9eeZ) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
I have the old EOS Utility software ver. 2.6.0.0
I updated to EOS Utility 2 ver. 2.14.20.0
(https://c1.staticflickr.com/9/8355/29401619646_04f10d6773_o.png) (https://flic.kr/p/LN7ZbE)
EOS_Utility_2_Ver_2.14.20.0 (https://flic.kr/p/LN7ZbE) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Now I can install PF3 picture styles on the 5D2.
Looks like we aren't the only ones reverse engineering log picture styles.
http://www.eoshd.com/2016/09/now-available-eoshd-picture-profiles-brings-c-log-canon-dslrs-including-1d-x-mark-ii-5d-mark-iv/
QuoteThis is not an official Canon LOG profile but the difference in performance is minimal. I have for months been fine tuning the EOSHD C-LOG Picture Profile on my 1D C, so that all other Canon DSLRs can benefit from it as well. The only difference is that the official Canon 1D C LOG mode has a bit more latitude in the highlights, but they are VERY similar!
Log mode on the Canon 1D C is built into the firmware so you can't just copy it and install it on your Magic Lantern enabled camera.
Is isn't free and you can't open the .pf3 files in Canon's Picture Style Editor (editing is disabled) but it does come from Andrew Reid, author of the popular website, EOSHD (http://www.eoshd.com/). He's also including some interesting creative picture styles and luts in the package.
i dont like the fact that he is asking money for it, it would be more beneficial for the community to start working with an open profile and not a locked one that is payed.
otherwise why not just use cinestyle wich is locked and free.
Every picture style around except cinestyle seems have been originally baked in picture style editor. The editor is way to crude creating any math based logs. If anybody knows any other way to build a curve than in this editor it would be interesting to know how.
On a side topic there is a dng profile editor from adobe which turn a dcp profile into 96 points. You can check this with open source program dcptools. Now this 96 points can be overriden with for instance dcptool and suddenly you can build a dcp profile with full log math it seems. Looking at tc maybe this is the deal here.
FYI: on the technicolor website, when cinestyle was first released, it said that their technicians spent one year developing the Profile in collaboration with Canon. I can not find this information anymore on their website.
Quote from: Lars Steenhoff on September 04, 2016, 09:37:16 AM
i dont like the fact that he is asking money for it, it would be more beneficial for the community to start working with an open profile and not a locked one that is payed.
otherwise why not just use cinestyle wich is locked and free.
You make an interesting point. Some of us do need to make a living and what better than to make an income doing what you like to do. However, the EOSHD C-Log is reverse engineered from the log setting of the EOS 1D C so is it right to sell a copy of it? Artists have been copying works by the masters and selling them for ages but copyright laws are supposed to protect the original authors.
When Danne and I were deciding what to name our picture style we also debated whether we should put our names in the copyright field. We decided not to.
Quote from: Kharak on September 04, 2016, 12:55:23 PM
...their technicians spent one year developing the Profile in collaboration with Canon...
Another reason why we aren't copyrighting or selling our reverse engineered version of CineStyle.
So putting that discussion aside the EOSHD C-LOG picture profile is very interesting and hey, it is only $9.99 USD and it does come with a bunch of other picture styles and luts to play around with.
I'll be going on a month long trip and will be taking an EOSM which can double as a great little point and shoot camera or an almost fully featured Magic Lantern enabled monster. Here are the picture styles on my camera:
(https://c5.staticflickr.com/9/8308/29168470540_cd79e74589_z.jpg)
Note that the blue letters on the left column indicates the active picture style and the blue letters on right column indicates changes from the default settings. Neutral is set to the Prolost Flat settings, easy because you don't need to load the picture style with the EOS Utility. CineStyle has the default 0 Saturation changed to the recommended -2. Note an issue with the EOSHD C-LOG name on the Canon menu. Not a problem with the Magic Lantern menu.
(https://c5.staticflickr.com/9/8470/29168470380_d4040ce5af_z.jpg)
Ok, we're at a good spot with reverse engineered CineStyle picture style--logNeutral. There's also the variations based on Canon standard picture styles, logStandard, logPortrait, logLandscape and logFaithful so there's plenty to play around with.
What's the next project? Probably not reverse engineering EOSHD C-LOG because it is already a reverse engineered picture style. Maybe make an original log picture style? Sounds like a new topic.
how does the EOSHD C-Log compare to the free Clog? ;D
Quote from: ph2007 on September 05, 2016, 10:11:56 PM
how does the EOSHD C-Log compare to the free Clog? ;D
The free C-Log (http://clog) is similar to CineStyle while the EOSHD C-Log (http://www.eoshd.com/2016/09/now-available-eoshd-picture-profiles-brings-c-log-canon-dslrs-including-1d-x-mark-ii-5d-mark-iv/) looks a lot like the "real" Canon Log from the 1DC. Yeah, I know it is confusing that there is more than one C-Log out there.
This brings up an interesting issue. Is the Canon cinema line significantly different from the still cameras that can use Magic Lantern so that Canon Log doesn't translate properly? Log in the 1DC is located in the firmware while these other log picture styles are custom picture styles. A few posts back I asked about base ISO and it seems that it doesn't apply to the DSLR's yet the 1DC does have a base ISO of 400. As a comparison the C100 (CP Cinema Locked) has a base ISO of 850 (https://www.cinema5d.com/exposing-with-native-iso/). Shooting below this setting results in increased noise (vertical stripes and banding) and reduced dynamic range when shooting in Canon Log mode. It is ok to use a higher ISO but the dynamic range is reduced. So do these reverse engineered C-Log picture styles really work? Do we need to shoot at a minimum ISO setting? Again, more questions than answers.
In my last post I may have given the impression that we were done with this project--not the case. There's still lots that can be done.
One experiment that is showing promise is to take a technical lut, invert the curve and create a reverse engineered picture style.
Danne has been experimenting with dcptool (http://dcptool.sourceforge.net/Introduction.html) which is a command line tool for editing DCP, Digital Camera Profile, files that can be embedded in DNG files and works with Adobe Camera Raw. Adobe has an editing tool called DNG Profile Editor that creates DCP files and is erringly similar to Canon's Picture Style Editor. In addition Canon's Digital Photo Professional application uses .pf2 and .pf3 files much like Adobe Camera Raw and Lightroom uses .dcp files. The difference being that Adobe has an SDK for their DNG file standard while all of Canon's file formats are closely guarded. What would be possible if we could get into the Canon file formats? It would be possible to build an MLV to DNG application that translates the picture style into a dcp that can be tagged into the DNG files. Of course the DNG files need to be debayered by a program that understands how to decode the embedded dcp information and it seems that only Adobe Camera Raw and Lightroom can do it.
Holy Shizzle this Project has finally taken off and it is even more surprising to see how much has come along in so little time. All the thanks goes to @dfort and @Danne for pushing this forward in a much quicker pace. Actually I am starting to regret not following this closely while I was out on Vacation in Chicago because I shot a bunch of things in H.264 (HDR, 1:1 crop, etc) all with Technicolor Cinestyle PP (I sometimes do use Prolost Flat settings within Neutral PP) and now I've decided to jump on the REPS wagon (even though I may be a little late) for this journey.
I completely agree with @dfort that we should all figure out a way to set the native base ISO (I think it's ISO 400 for 5D3 but not sure of others?) in order to maximize not only the DR but to keep the noise as free (or real) as possible rather than introducing useless artifacts into the footage.
So I downloaded the v1.3 (Thanks again for that!) and installed them into my SL1 (100D) just because I have it here with me at work (left the 5D3 at home) and decided to shoot them in ISO 100 (even tho it has more noise than ISO 200) but it's day time here for me and of course I am just too anxious to see the results.
Shot in H.264 on SL1 in 24p with 1/120th closed down to f5.6 @ ISO 100 with the PP in order enlisted below.
1) Neutral (Prolost Flat settings)
2) LogNeutral (0, -4, -2, 0)
3) C_LOG_htp (0, -4, -2, 0) <-- I got this from here (http://magiclantern.fm/forum/index.php?topic=17730.msg171136#msg171136here)
4) Technicolor CineStyle (0, -4, 0, 0) <-- I usually have Saturation down to -2 but for this test I just left it at default.
https://vimeo.com/181561894
I couldn't find a LUT that would help better show the differences (left my MBP plus external drives at home) so I just impatiently chose the crappy Kodak2393_Rec709 that I downloaded from the internet. I know it's not the most ideal.
Hey @DeafEyeJedi I'm next for a vacation, will be gone for a month.
In the meantime if you really want to go crazy add EOSHD C-LOG and all of the variations Danne and I made, logNeutral, logStandard, logPortrait, logLandscape and logFaithful. Some of these are very subtle. You don't have to go searching for a lut, a Technicolor CineStyle lut is built into DaVinci Resolve. It also has Canon Log luts so you can try them on the C-Log picture profiles. Yeah, I know--wish you could load in more than three custom picture styles. Note that if you can see a difference between CineStyle and logNeutral we didn't do a good job.
(https://c3.staticflickr.com/9/8326/29118992410_3c9fa41451_z.jpg)
I created a very crude 10 point 1D lut and an inverse with so-rose fine tool openlut (http://www.magiclantern.fm/forum/index.php?topic=17820.msg171830;topicseen#msg171830) and turned those into 3D_luts (https://drive.google.com/file/d/0B4tCJMlOYfirUWdJb2ZqRjJzUEU/view?usp=sharing) luts to play around with. While cinestyle s-curve seems a somehow boosted inverse curve I think these 3D luts will catch the resemblance to the canon neutral setting more correctly.
The main issue creating an inverse is the fact that cinestyle contrast setting is set to -4. This setting can,t be inversed with a 1D lut as it probably affects clarity and other built in overall settings not represented in a 1D lut gamma curve. In order to get a correct inverse canon neutral picture back you have to either filmed with logNeutral with the cam settings set to 0 both contrast and color(same as canon neutral) or set canon neutral setting to the same in cam settings as logNeutral. Happy testing!
Interesting experiments going on. Danne tweaked the tone curve and got more highlight detail. That's the beauty of opening up CineStyle, well let's just say a CineStyle look alike.
@reddeercity did a massive test comparing logNeutral and CineStyle with other picture styles:
http://magiclantern.fm/forum/index.php?topic=17828
I'm about to leave on a month long trip but before going I packaged up a new release. This one has Danne's highlight recovery. It looks like there's not much interest in the logStandard and logFaithful picture styles so I'm dropping those but leaving in logPortrait and logLandscape. We might play with saturation and contrast levels on those in the future but testing three distinct picture styles will be easier than five subtle ones. Besides, you can only load three custom picture styles at a time.
Version 1.4 link on first post. (http://magiclantern.fm/forum/index.php?topic=16299.msg158364#msg158364)
Thanks @Danne & @dfort for these rapid updates. However not sure if this is to be expected but I noticed that if shoot in LogNeutral (v1.4) which then Zebra's aren't showing on the LiveView LCD whereas it does if I were to use CineStyle or any other PP.
LogNeutral PP:
(https://c3.staticflickr.com/9/8680/29484252442_58e04f8e49.jpg) (https://flic.kr/p/LVqv33)
CineStyle PP:
(https://c1.staticflickr.com/8/7696/29484251152_327224640b.jpg) (https://flic.kr/p/LVquDN)
Also note in the histogram which shows the highlights being completely clipped in the LogNeutral PP -- could this be the issue related?
Quote from: DeafEyeJedi on September 11, 2016, 03:23:10 AM
...not sure if this is to be expected but I noticed that if shoot in LogNeutral (v1.4) which then Zebra's aren't showing on the LiveView LCD whereas it does if I were to use CineStyle or any other PP...
That's expected. If you open up logNeutral v1.4 in Canon's Picture Style Editor you'll notice that the highlights are clipped. This is something that CineStyle does in the shadows and we were experimenting to see what it does in the highlights. Check out EOSHD C-Log if you can, it will probably exhibit similar behavior.
I'm running around Europe at the moment and am operating on my wife's laptop so no development while I'm on vacation!
In any case, note that v1.4 is an experimental version. Guess I should have marked it beta or something. So far v1.3 is the closest we've gotten to CineStyle.
[EDIT] Ok--so I'm cheating on my vacation. Here's a screenshot showing the highlight clipping on v1.4. Like I said, this is an experiment not necessarily a "final" version.
(https://c7.staticflickr.com/9/8140/28972478854_a965378035.jpg)
Well I kinda like it ... It's like a cheat sheet for the highlights. Definitely worth the risk. Lovely 10-point curve designed by @Danne. I wonder if there's a way to add more points into it? Love the fact that you don't just risk anything but do risk everything!
Looking forward to testing with some more this weekend.
btw @dfort I went ahead and purchased EOSHD C-log just for shits and giggles. Will eventually let you all know how it goes internally and externally.
I also am confident that once this LogNeutral hits by 1.6 or 1.8 it'll be smoking hot like no other!
C-log clips highlight even more. I don,t know how well that responds to H.264 in non C-log cameras.
I like technicolor more and more I must say. When I tried using it when it first came out I was under the wrong impression that color was wasted and I did my share of skin/skylight color complaining and then settled for neutral with in cam contrast/sharpening set to zero. Guess I was wrong all along.
Clipping the whites is interesting. It seems there is some more information in highlights which are clipped. Why not try to save it?
Crops
version 1.4
(https://s21.postimg.org/eichmct1z/Screen_Shot_2016_09_11_at_10_46_52.png)
version 1.3
(https://s21.postimg.org/4z7e6muxj/Screen_Shot_2016_09_11_at_10_46_31.png)
version 1.4
(https://s21.postimg.org/l7j12dedz/Screen_Shot_2016_09_11_at_10_50_08.png)
version 1.3
(https://s21.postimg.org/6n1y7jjfb/Screen_Shot_2016_09_11_at_10_50_30.png)
QuoteLovely 10-point curve designed by @Danne
Well, let,s not underestimate what these two dots actually do to refienements coming from dfort himself.
(https://s22.postimg.org/v8oz0caz5/Screen_Shot_2016_09_11_at_11_25_12.png)
I can also confirm that C-Log clips highlights. It clips everything above the brightness level of 235. Wouldn't it theoretically have more latitude if it didn't clip?
I'm getting some brilliantly clean footage with it, however. I still believe it is bett than cinestyle or I just failed to properly grade the latter.
Neutral 1.4 has the same problem as Cinestyle. Some kind of grey paste in the skin.
I made the shot with actors, with neutral -2 -4 and and the neutral. I applied the same color post and 1.4 it's not usefull yet.
For my perception, and opinion, it's something in the high luminance curve that is not working fine.
I don't know how to add an image here, I tried the shortcut but dont work.
https://www.dropbox.com/sh/2q3x73jf66wmj0d/AADWJ7hID3ny2Qay8Zeoy2Ioa?dl=0
Could you upload the originals (jpg or cr2 if they exist too) without anything applied? I would like to test some. If you have the same shot in regular neutral and in logNeutral we could compare the results.
Here are some tests you could do
1 - Try the same pic style 1.4 but this time don,t dial down the the color slider in cam at all. keep it at 0. Then apply the inverse lut from here. Keep all in camera settings the same(especially the contrast slider) for both neutral and logNeutral.
http://www.magiclantern.fm/forum/index.php?topic=16299.msg171880#msg171880
2 - Try one of the 1.3 versions with different color. I prefer standard or portrait, not sure if dfort added those but you could do that easily in picture style editor and resave it yourself. You can add the inverse lut the same as in above example to compare.
On a sidenote I don,t know how you white balanced your shots, seems a little cold. If the tone curve is added back correctly there should be no difference in skintone from neutral and logNeutral as long as you set the in cam sliders the same in both logNeutral and neutral.
Did you by the way compare 1.4 and 1.3 since you say it affects luminance? I thought luminance would mostly be noticable in brightest whites, lesser so in midtones and colortones.
@Danne, I made video. It's h264. And I exported the frames as JPG.
The light was fluorescente. And I made WB from rate button, with ML, pressing set.
After that I remembered your work in the reherseal and I just change the preset with the same WB, ISO, F and I took one of each.
I added to the same link two frames without post.
There's something in the highlight, in the skin tone on the face. Like a grey. But maybe you perceive something in the setting. I think, for my little expierence, - this is not a fact, please - the luminance without any saturation make some of this grey and even adding in post some saturation, that information is not usefull.
I think could be usefull investigate the EOS log as @DeafEyeJedi propoused.
I appreciate when there is some testing around this. However, it will help this project forward better if you upload a test sample file. Simply saying "There,s something in the highlight, in the skin tone on the face" will only add to the misconception on how not to work with cinestyle/logNeutral I,m afraid.
*update
Will download your unedited files. Thanks
*update
Problematic to use the provided files for testing. If we are doing a comparison test we would need two exposures of both logNeutral and neutral with the same scene.
Here is one H.264 shot with version 1.4 and with a lut applied.
(https://s15.postimg.org/ltptszbmz/Screen_Shot_2016_09_13_at_11_34_27.png)
(https://s15.postimg.org/ep80jy4dn/Screen_Shot_2016_09_13_at_11_34_42.png)
I did my thoughts on canon-profiles (aka styles) as well. i couldnt believe, these files arent hacked yet. there naturally is more than just contrast, saturation and so on. my thoughts by now ( i started yesterday a little bit analyzing files)
* a "simple and fast processing" inside the body is needed, therefore it must be a simple encoding.
* the pf3-files are ~double the size of the icc files found in the preset-style icc-path
Quoteexample neutral with no adjustments
* icc file size - NS.ICC - 212kb
* pf3 file size - 425kb
* maybe for safety purposes the lut is written two times.
* maybe there s a inverted lut inside as well
* i saved a pf3-file with no changes three times with ~5secs difference. it seems, theres a timestamp in it, but the timestamp does not affect the encoding. nearly the end of the file a byte changes as well. a checksum? integrity check?
* the pf3-file must be rockstable to avoid crashing or deadlock the processing inside the body.
* the title and copyright arent saved plain, this leads me to the idea of a simple encoding. rle? difference? xor? not?
example links:
Canon Neutral SRGB 6091 ICC as XML (parts of it) (http://pastebin.com/YHdY91Z2)
pf3 neutral saved three times (https://www.flickr.com/photos/chmee/29580475931/in/dateposted-public/)
Because i dont know much about ICCs, i have to read more about ICC-LUTS to understand and recognize lut-data inside the hexview :) maybe i find a byte/word (simply the clue) to decode the pf3-file. after this it should be much more simple to create own luts and pf3-files. Working only with the luma-curve is unsatisfactory.
by the way. did you tried combining your lumacurve with low-contrast-setting? is this affecting the max-luma from 235 to maybe 255?
simple toDos:
* decode title and copyright (we know the plaintext)
* decode the simple things (contrast, saturation hue aso)
* decode lumacurve
* decode 6axes-data
- i assume there are all in the beginning, then the lut comes.
regards chmee
Quoteafter this it should be much more simple to create own luts and pf3-files. Working only with the luma-curve is unsatisfactory.
word.
Quoteby the way. did you tried combining your lumacurve with low-contrast-setting? is this affecting the max-luma from 235 to maybe 255?
You mean low contrast setting in cam or altering the luma curve in picture style editor? If in cam it,s set to -4.
Quoteis this affecting the max-luma from 235 to maybe 255?
Probably :)
Interesting findings @chmee -- I'm on a vacation at the moment and don't have the time to look into this but it seems similar to what I was doing with reading header and footer information in my focus pixel bash script.
http://magiclantern.fm/forum/index.php?topic=16054.msg163309#msg163309
I guess we can start out with baby steps by creating a very simple picture profile saved with and without caption and copyright information and search the pf2 or pf3 file in order to find where it is stored. Then keep doing this changing the base picture style, add just one coordinate in the RGB tone curve, etc.
Sounds tedious but that is "real" reverse engineering and not simply trying to create something that "looks" like a picture style we're trying to mimic.
Note that when creating logNeutral I was checking it on a waveform monitor and it was frustrating that sometimes making a change in one area of the tone curve would cause changes to other areas. I'm guessing this is because of the "smoothing" that the Canon Picture Style Editor does between the points (maximum of 10 points) on the tone curve. I'll do an illustration of this when I get a chance. It would be much more accurate if we can define every coordinate on the tone curve. I'll bet that the EOS Utility is translating the information stored in the pf2 or pf3 files into a simpler binary format (meaning no smoothing between coordinates) that is stored in the camera's firmware. It would make sense that Canon is using ICC profiles that are also saved to the CR2 files but so far it seems that Canon has not released this information. Of course now I'm just rambling about things I don't really understand.
@chmee thanks for the info , and with that I decided to look inside the Picture Style Editor as the base picture styles are there.
After a little time I do believe I have found the start of the Picture Styles with help of "HxD" Hex editor , there are labeled as ".icc"
and can be found on Windows at C:\Program Files (x86)\Canon\Picture Style Editor\DPP4Lib\icc
FA.ICC is = "6191 Faithful AdobeRGB"
This just the first few lines the rest I really don't understand
QuoteM€UCCM scnrRGB Lab Ø acspMSFT CANO6191 öÕ Ó+CANO cprt 8desc L xwtpt Ä rTRC Ø gTRC è bTRC ø rXYZ gXYZ bXYZ 0 A2B0 D K<A2B1 D K<A2B2 D K<text Copyright CANON INC. 2008 All Rights Reserved desc 6191 Faithful AdobeRGB 6 1 9 1 F a i t h f u l A d o b e R G B 6191 Faithful AdobeRGB XYZ öÕ Ó+curv c. curv c. curv c. XYZ XYZ XYZ mft2 ! ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ €
LA.ICC is = "6091 Landscape AdobeRGB"
QuoteM€UCCM scnrRGB Lab Ö
acspMSFT CANO6091 öÕ Ó+CANO cprt 8desc L xwtpt Ä rTRC Ø gTRC è bTRC ø rXYZ gXYZ bXYZ 0 A2B0 D K<A2B1 D K<A2B2 D K<text Copyright CANON INC. 2003 All Rights Reserved desc 6091 Landscape AdobeRGB 6 0 9 1 L a n d s c a p e A d o b e R G B 6091 Landscape AdobeRGB XYZ öÕ Ó+curv c. curv c. curv c. XYZ XYZ XYZ mft2 ! ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ € € L€»}O ~z,,,¯v&Ñ,,mpцái¡/Š3cFúŽ"]K"5Vç 8š6QÝŸIKbV£<F!¦NA# Ï©i<
#;¬–6Ô(ǯÔ1.o³$,47¶†&
NA.ICC is = "6091 Neutral AdobeRGB"
QuoteMxUCCM scnrRGB Lab Ö
acspMSFT CANO6091 öÕ Ó+CANO cprt 8desc L pwtpt ¼ rTRC Ð gTRC à bTRC ð rXYZ gXYZ bXYZ ( A2B0 < K<A2B1 < K<A2B2 < K<text Copyright CANON INC. 2005 All Rights Reserved desc 6091 Neutral AdobeRGB 6 0 9 1 N e u t r a l A d o b e R G B 6091 Neutral AdobeRGB XYZ öÕ Ó+curv c. curv c. curv c. XYZ XYZ XYZ mft2 ! ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ € € "|.Z,ow"3,,op9G‡Hi]þ‹2b‡ % \, Á•¼VC⚆PÔž^KÛ»¡OGQÿ¤Bé C¦È>#‡©r:G'Ǭ6 , ®ª1Ù0S±9-´4—³Á)–8ܶE%€="¸¿!sAi»7 lEµ½©nIÿÀtNQÂ,,R•ÄÝ
¦VÕÇ/ Ú[É|_^ËÄ`cªÎ gûÐC lRÒ| p³Ô° u ÖÝ |²,²Ñ~& }bzZy^s/Ì
PA.ICC is = "6091 Portrait AdobeRGB"
QuoteM€UCCM scnrRGB Lab Ö
acspMSFT CANO6091 öÕ Ó+CANO cprt 8desc L xwtpt Ä rTRC Ø gTRC è bTRC ø rXYZ gXYZ bXYZ 0 A2B0 D K<A2B1 D K<A2B2 D K<text Copyright CANON INC. 2005 All Rights Reserved desc 6091 Portrait AdobeRGB 6 0 9 1 P o r t r a i t A d o b e R G B 6091 Portrait AdobeRGB XYZ öÕ Ó+curv c. curv c. curv c. XYZ XYZ XYZ mft2 ! ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ € € € |5 €
w ,éo= ‡6fÓ šŒW^çB'ºW·†˜†PòvŸáJž '¥ãDÄ°ªr?o$j:ø¯Å6 ¼²J1q&z´æ,°,3·™'Ú1ñº["ó7°½0 ú=sÀñC=ÃÖI Æ«NÎÉ
ƒT‰ÌYZHÏ ` Ò8 eÎÕY k™Ø, qiÛ´ w<Þí }â. ,øåv ˆßèÇ ŽËì xÓY‹z=}É`| x'~™pD4åhŠY†4a-
SA.ICC is = "6091 Standard AdobeRGB"
QuoteM€UCCM scnrRGB Lab Ö
acspMSFT CANO6091 öÕ Ó+CANO cprt 8desc L xwtpt Ä rTRC Ø gTRC è bTRC ø rXYZ gXYZ bXYZ 0 A2B0 D K<A2B1 D K<A2B2 D K<text Copyright CANON INC. 2005 All Rights Reserved desc 6091 Standard AdobeRGB 6 0 9 1 S t a n d a r d A d o b e R G B 6091 Standard AdobeRGB XYZ öÕ Ó+curv c. curv c. curv c. XYZ XYZ XYZ mft2 ! ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ € € |%H,'wÁ...oÚ'ˆ¬hˆ
ŒdaÓ'[µ O–XVv›P¾ žÐKäå¡ßGmÓ¤¶C" ©§Œ>ë!kªW:Ê%+6Ê) ¯a3,Þ±«/W0œ³ï+À4K¶!(D7ì¸A$ä;wºZ!›>ï¼f mB1¾˜BE^ÀÁ2H¬Â™cKèÄe±OÆ$ R(ÇÏ
ÁU(Ék ‡X Êû v[Ì€]éÍõÒ`ÃÏbEk{Ü,Nü|ø~†§~SyŸ€vrVìƒrk ˆ7dŽ ^<
'€X^~—7Rý×›NŽžKI...H¡UE% ÿ¤A@ç"¶§ <Ì&f©¹8Ò*¬L4ö-©®Ì1518±9
I also noticed that with every Style/ICC was either sRGB or AdobeRGB , I guess it's depended on if you set your cam to sRGB or Adobe . Personally my 5d2 is set to AdobeRGB .
This line seem to be a constant "cprt 8desc L xwtpt Ä rTRC Ø gTRC è bTRC ø rXYZ gXYZ bXYZ 0 A2B0 D K" Could this be curve information ?
and this one here also "XYZ öÕ Ó+curv c. curv c. curv c. XYZ XYZ XYZ mft2 ! ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™ªª»»ÌÌÝÝîîÿÿ ""33DDUUffwwˆˆ™™"
the trade mark "TM" is interesting .
I think if we understand the XYZ & the "c. curv" we should be able to customize the profile/style without having to relaying on the 10 point curve in Pic.Style Editor.
Or I could be Totally off base here ! :D
Nice find. On Mac those files are in:
/Applications/Canon\ Utilities/Picture\ Style\ Editor/PictureStyleEditor.app/Contents/Resources/icc
Wolfcrow (http://wolfcrow.com/) has a good introduction to what XYZ colorspace (http://wolfcrow.com/blog/what-is-the-difference-between-cie-lab-cie-rgb-cie-xyy-and-cie-xyz/) is and how it relates to RGB.
iccXML (https://sourceforge.net/projects/iccxml/) is able to convert the icc files to XML-human readable data. just for analyzing purposes. As i did that, i ve seen, icc files are not simple luts as i thought, its not a converting list 0->0, 1->1.2 and so on. its much more complex. (maybe there are 32bit-floatinpoints saved instead of 16bit-integer?)
later i will post some more findings on decoding pf-files. it seems something like a simple bitshift over the whole file or a list of bitshifts/adds/subs. its kind of a puzzle we can solve together :)
the main icc's are not in the cr2-files - theres only a integer/string to name the needed picture-style. Thats the reason, Adobe isnt able to use the real canon-styles - only dpp. for us, its only important for mp4/mov-recordings.
regards chmee
@reddeercity - You won't get far looking at encoded files ;D
There are several tools available to view, dissect and edit ICC/ICM profiles and the first place to look is The International Color Consortium (ICC themselves). A good tool to start with is wxProfileDump (http://www.color.org/profdump.xalter) but you'll need a something like Matlab, ICCedit or ICCxml to delve deeper and edit things.
The Picture Style ICC files are all pretty much Linear (Y=X) and work in conjunction with a colorspace (either sRGB or Adobe 1999) - as I mentioned earlier, if there is an S (NS, FS, LS, SS, PS) it's the sRGB version and if there's an A (NA, FA, LA, SA, PA) it's the Adobe 1998 version.
There is an XYZ 3x3 matrix but it's neutral because everything is done with luts. The profile connecting space (PCS) is Lab.
The luts are the AToB0, AToB1 and AToB2 tags and specify the color adjustments depending on rendering intent. The luts are in Lab colorspace and have an RGB shaper that is Linear.
With that being said I think ICC is probably a red herring regarding Picture Styles. They are very similar but not interchangeable. You can dissect a profile in Matlab and copy parts to/from other files by finding the 3x1 struct array and headers but you can't change an ICC to a pf2 or pf3 for editing in PSE.
I think Chmee's idea of embedding look luts in a PS is the most interesting but I think there will be some restrictions because it has to be in Lab colorspace.
What is really needed is a 1D lut or function that works on the code values coming from the sensor before the picture style and encoder stages and I'm not sure that's even possible (a1ex?). If it were then Canon log would become a possibility. At present a Picture Style made in PSE is working with a truncated, display range signal.
Red herring :D
Very interesting information and digging and some useful tools too shared in these later posts. Great.
Quotethe main icc's are not in the cr2-files
Wonder if that is really true here. How can one explain CR2 files shot with a selfmade picture style having all information applied in dpp without needing to install the actual picture style in there? An encoded icc profile only read/known by canon?
cr2 doesn't store ICC's but there will be a flag for DPP to call one and if the PS is custom it is embedded in some way (probably as pf2 or pf3) else DPP wouldn't know how to produce a Jpeg that matches the one shot in the camera.
point taken, @Danne :) you re right. But, if so.. we could try to extract the cinestyle-Preset from a cr2 and then compare with the pf3.
Yes that could be useful. As for creating icc profiles one can convert 3D luts to icc profiles with for instance OpenColorIO. Tip from Andy600 a while ago. That fact doesn,t really solve the core issue here for know but might be useful later.
here two screenshots about the plaintext-riddle :) i assume, if we take this hurdle, we can read the whole file.
(https://c1.staticflickr.com/9/8422/29069984784_f414799a19_c.jpg) (https://flic.kr/p/LhPgGd)
pf3_plaintext_decode_01 (https://flic.kr/p/LhPgGd)
the difference is hex20 as in ASCII, but sometimes -20, sometimes +20
(https://c5.staticflickr.com/9/8321/29069984444_9835501aa8_c.jpg) (https://flic.kr/p/LhPgAm)
pf3_plaintext_decode_03 (https://flic.kr/p/LhPgAm)
the same characters are encoded differently. maybe some of you see a pattern :) But: Do not forget, after decoding we ve got still a binary file, we have to find offsets for the entries.
It looks like some kind of Alien Technology to me.
but what do I know.
Quote from: Danne on September 15, 2016, 12:55:27 PM
Yes that could be useful. As for creating icc profiles one can convert 3D luts to icc profiles with for instance OpenColorIO. Tip from Andy600 a while ago. That fact doesn,t really solve the core issue here for know but might be useful later.
I might be wrong but argyllcms does this too.
ICCs contain can contain LUTs but also matrix data. No one will be surprised though if the data in a pf2 is represented in a proprietary way can't interpret like existing standards like ICC and DCP. I hope one day we will able to convert an ICC or Adobe DCP to pf2. That would allow for capturing colors with high accuracy for documentary footage or slide transfers using DCamProf for profiling, or film simulations straight out of the camera.
Found something interesting at https://searchcode.com/codesearch/view/20202630/
perl-Image-ExifTool /Image-ExifTool-8.90/lib/Image/ExifTool/Canon.pm
I guess this is Perl Language , anyways at line 4569 there some tag info on picture style
from line 4703 to line 4709
# base picture style names:
0xd8 => {
Name => 'UserDef1PictureStyle',
Format => 'int16u',
SeparateTable => 'UserDefStyle',
PrintConv => \%userDefStyles,
},
But I can't find the "SeparateTable" info
If nothing else some good info , but not sure if any of this really helps.
Quote from: Danne on September 15, 2016, 12:55:27 PM
Yes that could be useful. As for creating icc profiles one can convert 3D luts to icc profiles with for instance OpenColorIO. Tip from Andy600 a while ago. That fact doesn,t really solve the core issue here for know but might be useful later.
Agreed.
There's a discussion about the Canon Emerald picture style in another topic:
http://magiclantern.fm/forum/index.php?topic=17896.msg172219#msg172219
I checked it out in the Canon Picture Style Editor and surprisingly it isn't locked for editing. However, in the editor it only shows the default settings with the "Base Picture Style" listed as Emerald. I just thought this was interesting and it might give us some clues into how picture styles work. The Emerald pf2 file is only 1KB in size.
I managed to unlock Cinestyle a couple of years ago and just found the file again (not sure I can post here!? :-\).
I'm 99% certain that Cinestyle is all done with a lut in Lab space, similar to some ICC profiles.
The profile will open in PSE but reveals no user-editable parts. There is no tone curve data, no HSL adjustments etc and everything turns yellow. The tone curve has no effect except when moving the top and bottom control points. Removing the yellow saturation (6 color-axes) and pulling brightness -2 stops (preliminary adjustments) reveals more detail in the image.
I'm guessing the introduction of the PF3 format with an RGB tone curve basically gave users much the same level of control that Technicolor had when developing Cinestyle (pf2 format) which is why you can get very close to the Cinestyle tone curve - but not exact because of the limited, interpolated control points.
Sadly, this all seems to take place after Jpeg compression. For a proper log profile like Canon Log there needs to be a way to add the function before compression.
Standard
(http://i.cubeupload.com/PcRKTN.png) (http://i.cubeupload.com/PcRKTN.png)
Cinestyle (unlocked)
(http://i.cubeupload.com/QK37HN.png) (http://i.cubeupload.com/QK37HN.png)
Cinestyle (unlocked) -2 brightness 0 yellow sat
(http://i.cubeupload.com/JlOy1x.png) (http://i.cubeupload.com/JlOy1x.png)
In DPP
(http://i.cubeupload.com/VFzTen.jpg) (http://i.cubeupload.com/VFzTen.jpg)
Not posting any unlocked profiles is probably the better idea :).
QuoteSadly, this all seems to take place after Jpeg compression.
So practically applying log after it,s filmed with neutral will yield the same result as with technicolor in cam? Havn,t tested this.
Not quite. Once that contrast is baked into the image the detail is gone.
There are only 256 values to play with but you can gain a little more wiggle room with Cinestyle and with your's and @dforts remake (which is very good BTW) but doing so compromises other parts of the image which can manifest as more visible banding or noise. Raising the black level is probably the most useful thing to do because the H.264 encoder doesn't handle noise well.
wiggle, wiggle :D
QuoteOnce that contrast is baked into the image the detail is gone.
Confirmed then. Thought so over here too.
uuh. i was (theoretically) sure, the styles are used after demosaicing and before/while rescaling to 8bit/jpg.
How to test this? One thing is that picture style editor doesn,t seem to get to linear. Or is it? I played around in dng profile editor and created a dcp profile around logNeutral curvepoints(not exact translation, the dng profile editor curve tool has its own life when adding points). Base tone curve set to linear and whoops suddenly scene referred? highlights are retained even more or am I having a placebo? I turned down sharpness settings in acr.
logNeutral 1.4
(http://s15.postimg.org/vrhm350jv/Screen_Shot_2016_09_20_at_21_14_20.png)
logNeutral dcp linear opened in acr
(http://s18.postimg.org/uforr69s9/Screen_Shot_2016_09_20_at_21_23_36.png)
Does this experiment (http://www.magiclantern.fm/forum/index.php?topic=10038.msg99049#msg99049) help you with checking the linearity?
I'm pretty sure ACR has some auto - Highlight reconstruction, which might be what you see in ACR. Try setting the process to 2010 and 2003. To confirm if you are getting better results.
Also the green on the left side of that balcony, could indicate that ACR is using highlight reconstruction from the Green channel which is more present in the Bayer pattern.
That,s the thing @Kharak. You can,t grab the linear extra information from within acr. You have to get it through a dcp in dng profile editor. I think it,s true linear. Then again you might be right Kharak. ACR is a hidden box of tricks. Tried setting to process2010. Looked like crap.
Since acr is a still image editor most people doesn,t care about this and start working with highlight sliders etc which will give you roughly the same info back but will cause flicker if used in movie sequences.
For sequenced movie dng files a simple linear dcp will cause no flicker but get you some extra highlights.
Thanks for the link A1ex. Missed that one. Gonna check that some more after a good nights sleep :)
Quote from: Danne on September 20, 2016, 09:27:36 PM
...highlights are retained even more or am I having a placebo?...
I took a closer look at the images you posted using the Mac Digital Color Meter app and it looks like they both have the same amount of highlight detail. The only difference is that the second one doesn't reach the same brightness as the first example. It just looks like there's more detail in it.
@Andy600 -- that's some strange behavior on your unlocked CineStyle. Interesting--so your experiment looks like CineSyle is based on the Standard picture style, not Neutral? Maybe our reverse engineered CineStyle named logStandard is closer to the original than logNeutral?
I found that locked picture styles can be used in DPP (Canon Digital Picture Professional) but of course can't be edited in Canon Picture Style Editor. A CR2 exported to JPEG using CineStyle in DPP doesn't quite match a JPEG shot in camera using CineStyle though they are very close.
So are picture styles applied before or after JPEG compression? I've been thinking about that too. If it is after JPEG compression we'll never be able to create a picture style with a dynamic range greater than one of the standard built-in picture styles set at the -4 contrast setting, right?
Quote from: dfortWhat would be possible if we could get into the Canon file formats? It would be possible to build an MLV to DNG application that translates the picture style into a dcp that can be tagged into the DNG files. Of course the DNG files need to be debayered by a program that understands how to decode the embedded dcp information and it seems that only Adobe Camera Raw and Lightroom can do it.
You guys are my heroes! I just started looking into this exact pipeline again at work, hoping to figure more out. This is what we've wanted for years at our stop-motion studio - monitor on stage using Canon Landscape jpgs, use the CR2s (converted to linear EXRs using ACR/After Effects) in VFX, then apply the Canon Landscape look as a LUT in post so that our DP can start coloring based on exactly what he saw on stage. @Andy600 has done some fantastic work with Cinelog (thanks again, Andy!) but this is still the missing piece of the puzzle. I PM'ed Eric Chan from Adobe about the ACR side of things but haven't heard back yet. I'm still learning with regards to LUT formats, and maybe this was already doable with some other method, but being able to turn a picture style into a dcp would be a god-send.
Wow, quite a workflow. I have worked in animation myself for years and am somewhat familiar with using EXR's in the production pipeline. Picture style to ICC to DCP (which of course gives you a picture style applied to a CR2 on ACR) is what we are trying to figure out.
Yet another log picture style. This is the one Philip Bloom (http://philipbloom.net/blog/clog/) commented on about a while back (https://www.instagram.com/p/BJ0o3unD7qk/), CLOG NEUTRAL (https://sellfy.com/p/kjzE/).
According to the developer, James Miller:
QuoteCLOG Neutral comes in two options, one with a highlight roll off knee that stays below clipping (v1) and one where its a little more pronounced (v2).
CLOG 3 gives you a more Canon style modern LOG image.
(Note that I found out about this while Googling "logNeutral" just to see if anyone was talking about our reverse engineered CineStyle picture style.)
I wonder if another way of doing this would be to create a computer-generated CR2 file that contained known values in a color chart (like a faked, exact Macbeth chart instead of a photographed one), load it into Canon's Digital Photo Professional for converting to 16-bit TIFF using a specific picture style, and then analyzing the RGB differences for each swatch to generate an ICC and a DCP.
It wouldn't let you mathematically generate a pf2/pf3 from a known colorspace like CLog or ArriLog, but it would let you get from existing pf2/pf3 files like Cinestyle into a mathematically generated ICC/DCP file.
Here's the breakdown of the CR2 file format - http://lclevy.free.fr/cr2/
Well creating the cinestyle log into a dcp profile seems not too hard does it? We recreated the curve somewhat primitive 10 points but you can add the points into dng profile editor and come close. If you really want to work your way into a dcp profile you can use dcptool both to decompile and compile anything you want in floating point.
http://dcptool.sourceforge.net/Introduction.html
If you want to convert a 3D lut into an icc profile that can be done with opencolorIO
http://opencolorio.org/
Not that easy though to get coherent results with icc profiles. There seems to be a lot of complexity with icc profiles and various converters seems to produce different results. Then again I might have tested it wrongly. I know that dcraw exports raw_color when applying an icc profile.
I provide 3D luts of cinestyle some posts up, try converting those to icc profiles to start with maybe.
Producing and finetune matrix conversions between formats will probably take time but there is probably a lot of experience here that can be used in a lot of different workflows.
Hi!
I think there seems to be XOR operation taking place in encoding of picture style files. Take a look at the latter pair of chmee's example files having title and copyright fields consisting of series of single letters: A and B, respectively.
(1) In the first file the title consists of 10 x 'A' and it is encoded in hexadecimal as: 580F 2FC2 552F 2E76 D23D. When XOR'ed with 10 x 0x41h ("AAAAAAAAAA" in hexadecimal) we get: 194E 6E83 146E 6F37 937C
(2) In the second file the title consists of 10 x 'B' and it is encoded in hexadecimal as: 5B0C 2CC1 562C 2D75 D13E. When XOR'ed with 10 x 0x42h ("BBBBBBBBBB" in hexadecimal) we get: 194E 6E83 146E 6F37 937C. The same as with the first file!
Interesting. Thanks for sharing. I tried googling this info but I didn,t understand much. Can you make out any numbers or other understandable patterns we could use?
Quote from: Danne on September 28, 2016, 09:36:55 PM
Interesting. Thanks for sharing. I tried googling this info but I didn,t understand much. Can you make out any numbers or other understandable patterns we could use?
+1
I think in order to fully reverse-engineer .pf2/.pf3 format we should probably start with the lowest hanging fruits and then make progress to the harder parts (e.g. encoding of 6 color-axes data, possibly in form of ICC profile). One of the easiest things to do (still probably not very easy), as chmee presented in a previous post, is to reverse-engineer encoding of caption and copyright since by focusing on those two fields we can apply a so-called "chosen plaintext attack" as the exact input is known and can be freely selected. As I mentioned in my previous post, I think one component of encoding caption/copyright is applying a mathematical bitwise "Exclusive or" (XOR) operation on plaintext (https://en.wikipedia.org/wiki/Exclusive_or). XOR takes two operands; in this case one operand seems to be the plaintext string given by the user while saving a file in Canon Picture Style Editor and the other one is an opaque bunch of bytes the semantic of which is unknown. Resolving/understanding that unknown is IMO one step towards reverse-engineering this file format.
@Danne, @DeafEyeJedi: The XOR operator (usually represented by the ^ symbol) is its own inverse. That is, if X ^ Y = Z, then it is also true that X = Y ^ Z and Y = X ^ Z. If you know any two of the values you can always figure out the third, but if you only have one value you can't determine either of the other two. This property means it can be used as a sort of "poor man's encryption", where e.g. X = unencrypted data, Y = encryption key, Z = encrypted data.
Taking @jkaura's example, we know all of Z (the encrypted file) and some of X (the original data), so we can compute some of Y (the key) using Z ^ X = Y:
580F 2FC2 552F 2E76 D23D ^ 4141 4141 4141 4141 4141 = 194E 6E83 146E 6F37 937C
5B0C 2CC1 562C 2D75 D13E ^ 4242 4242 4242 4242 4242 = 194E 6E83 146E 6F37 937C
Because the results (key fragments) are the same in both cases, it implies the key is a constant. So now the puzzle is to figure out the rest of the key. It might be hard-coded in the encoding/decoding software, or it might be generated using a formula. If it's hard-coded it should be relatively easy to find by scanning the binaries, if it's a formula then some reverse-engineering of the encoding/decoding software (or extracting more fragments of the key using @jkaura's approach) is probably required.
@chris_overseas: Thanks for the clear explanation! You have clever ideas on further steps to resolve the key behind picture style files!
@Danne, @DeafEyeJedi: Btw, on Windows 10 machine, these example XOR calculations can be quite easily repeated with basic Calculator application in programmer/hex mode. There one can type in hexadecimal characters (number digits and letters between A-F) and perform XOR operations between two hexadecimal numbers.
I've only just installed Canon's Picture Style Editor a few minutes ago but may have figured out how to disable the protection on pf3 files. On the handful of files I've tried at least, changing byte 0x0006A122 from a 0x31 to a 0x30 seems to allow it to be edited, without any obvious side effects. Based on that and some of the other behaviour I've seen too, I don't think there's a checksum on the file.
Can someone try changing that byte on a pf3 file of theirs and let me know if it enables editing? Also, are pf3 files always 434,487 bytes in size?
Wow, cool find. The only pf3 files I've seen online are the ones in post 1 here, which are all either 434,490 or 434,466. I just tested saving out a default (zero adjustments) pf3 with PSE and it seems the file size fluctuates slightly with the number of characters in the Comment/Copyright fields when saving. The comment field can have 31 characters max, and the copyright field can have 63. Filling both fields with all 'z' characters gets a filesize of 434,529 bytes, and putting only a single '0' in the comment field gives me 434,466.
From looking at pf2 files by Canon and this collection of user styles: http://cinescopophilia.com/wp-content/uploads/2012/03/158%20VW%20Collection%20CPS.zip - I can see a wide variety of file sizes from 1243 bytes up to 20623 bytes.
Quote from: chris_overseas on September 28, 2016, 10:57:20 PM
...changing byte 0x0006A122 from a 0x31 to a 0x30 seems to allow it to be edited, without any obvious side effects...
Hex to ascii 0x30 = 0 and 0x31 = 1. Seems to be a boolean operator. Have you found the address for pf2 files?
So far all of the pf3 picture files I have looked at appear to have been created with the Canon Picture Style Editor. However, the Canon and Technicolor pf2 files were apparently created by some other method. Try downloading and opening Canon's Emerald picture style (http://www.canon.co.jp/imaging/picturestyle/file/emerald.html) and you'll see what I mean. The file is very small and it is possible to open it up in the PSE but all of the controls are at their default positions.
The Canon EOS Utility probably converts pf2 and pf3 files into icc profiles and loads them into the camera's non-volatile memory. The Canon and Technicolor pf2 files might already be in icc format only somehow encrypted? Wonder if the icc profile is stored in the camera encrypted? Maybe I'm just spitting into the wind?
(EDIT: Had to go read through the thread again to realize that the newest version of PSE lets you pick pf2 or pf3)
Found PSE 1.3 for Windows at Canon Europe's site: http://www.canon-europe.com/support/consumer_products/products/cameras/digital_slr/eos_1d.aspx?page=3&type=download&language=en&os=all
Still uses pf2 files, in case that's useful for anybody. Saving out a default file with 0 in the comment field gave a file size of only 1431 bytes, so this may be a much simpler format to decode.
jepp. the technicolor.pf2 is only about 5kb. if we can decode pf2, it would be a big milestone afor understanding the plain format - after that pf3 should be not that difficult.
@agentirons
a "nearly empty file" has no information for us to get the knowledge, we have decoded accordingly. i assume its still binary not readable (like xml).
- the copyright/title fields can be max 32/20 chars long. have to check if it s maybe as long as the encoding key. (dont believe it :) )
- if the known fields are long enough and xor-decoded, the key should start repeating. win.win.win.
- furthermore, if this decoded binary file is comparable to tiff, there should be unique tag-numbers (fixed length) and maybe a hex00 for data-end.
- maybe there are no tagnumbers but only hex00 (or other static bytevalues) to describe beginning and end of datafields.
- in terms of statistics this separation-value should be the val with the most occurence and its only one or two times successive.
maybe [tag][datavalue][separator][tag][datavalue][separator][tag][datavalue][separator][tag][datavalue][separator]..
or
maybe [datavalue][separator][datavalue][separator][datavalue][separator]..
or
maybe [separator][datavalue][separator][separator][datavalue][separator][separator][datavalue][separator][separator][datavalue][separator]...
regexlike: /[separator]{1:2}[^separator]+/
ah, and datetime could be a value we can work with.
regards chmee
This is all more than just beautiful. Simply stunning @chmee and thanks for the heads up @jkaura and much thanks to @chris_overseas for your clear pointers. Keep up the great work @agentirons!
I've made a bit of progress on solving the XOR encoding. Here's a zip containing the three log picture styles, plus another random one I made. For each, I've also included a copy that (hopefully!) has the XOR encoding removed but should otherwise be identical. Currently it's a fairly manual process to do this but I'll try to automate it when I have time (probably won't be for a few days at least). I'll also try to figure out what the layout of the files are in some detail. In the meantime, maybe the rest of you can figure out some of the file contents from what I have here?
https://www.dropbox.com/s/1xt92z62fosrsri/Decoded-Styles.zip?dl=0 (https://www.dropbox.com/s/1xt92z62fosrsri/Decoded-Styles.zip?dl=0)
Notes:
- The entire pf3 file is XORed apart from the first 12 bytes which are left as-is.
- I think the XOR operation has a period of 0x200 (512 bytes), though haven't confirmed that 100% yet.
Edit: turns out this is not true, we'll either need to apply a formula based on two separate keys, or have one huge key (512*513 = 260k) - The decoded files won't open in Picture Style Editor, they're just for examining with a hex editor.
@chris_overseas !big! kudos!
Do you have a encoding key extracted? Pieces? maybe its worth looking inside a canon-firmware for this key. a binary search. the longer the better.
Thanks a lot for digging into this. Will check your files chris_overseas.
Quote from: chmee on September 29, 2016, 10:03:23 AM
@chris_overseas !big! kudos!
Do you have a encoding key extracted? Pieces? maybe its worth looking inside a canon-firmware for this key. a binary search. the longer the better.
I don't have the key but it should be trivial to get by XORing the encoded vs decoded files (ignoring the first 0x0C bytes). No need to search the binaries or firmware - it's not stored in its raw value in there anyway.
FYI, properties PROP_PC_FLAVOR[123]_PARAM 0x401000[135] might contain user picture style data. In ML, they are only used to get the picture style names.
Sizes: 16704 on 5D3, 600D, 16680 on 5D2, 550D.
Example:
0x4010001 16704 0x824140 'Flaat_2' 0xffff00fe 0x0 0x0 0x0 0x1 0x219c60 0xfffffffc 0x2 0xfffffffe 0x0 0xdeadbeef 0xdeadbeef 0xfc0040 0xffff2fff 0x0 0xf64ad173 0x7e60894d 0xf452b0c0 ...
HTH
@chris_overseas
the key (if its static for all bodies, and it should) then will be important for encoding created files. nonetheless, i'm watching now your decoded files.
edit: do i see that right? the encoding key is 256kb long. (for pf3) holy sh*t.
Here's a little Java app that'll covert a pf3 file between its encoded and unencoded forms:
https://www.dropbox.com/s/4x8x9epalybm6aw/Pf3Decoder.jar?dl=0
To use, just run:
java -jar Pf3Decoder.jar input-file [output-file]
Note that a jar file is basically a .zip file. The source code is included inside the jar.
If the input file is encoded, Pf3Decoder will create an unencoded copy. If the input file is decoded, Pf3Decoder will create an encoded copy.
If you don't specify an output file, Pf3Decoder will create an output file with the same name but with a ".decoded" extension. If the input file already has a ".decoded" extension however, it'll create a file without the ".decoded" extension (ie it'll write out a file with the original encoded filename). Be aware that any existing output file with the same name will be overwritten without warning.
Amazing progress and what a helpful tool. Works great over here. Now what to make of the hex info left in there, hehe :P
Successfully toggled the encoding on...
Quote from: Danne on September 29, 2016, 03:13:23 PM
Amazing progress and what a helpful tool. Works great over here. Now what to make of the hex info left in there, hehe :P
Great to hear it's working for you. I'd suggest starting out by taking a pf3, load it in the Style Editor, make a single change, save it with a new name, then compare decoded versions of each to see what changed. Rinse and repeat, documenting any interesting changes you find. Once you start getting an understanding of the file format, try editing the decoded version appropriately then re-encode it and see if the change takes effect as expected in the Style Editor. I wrote the decoder with this workflow in mind so you can decode a file, edit it, then re-encode it quickly/easily. If you have any suggestions for improvements though, let me know.
One thing to try - toggle the byte that is about 20 bytes from the end of the file (just before the FF FE pair) from a 00 to a 01. I suspect that one enables/disables the edit protection.
Thanks for the Java tool, chris_overseas! Ran the test like you suggested, by opening PSE and making two pf3 files where the only difference was checking the 'disable subsequent editing' box on save. Compared in a hex editor, turns out that one option changes two bytes - the one you noticed, but also the byte that's 9 bytes in from the end.
(https://s13.postimg.org/wz00gpdtz/Screen_Shot_2016_09_29_at_10_21_53.png) (https://postimg.org/image/c23sc1fsz/)
(Left side is the locked file, right side is the unlocked file.)
I tried changing only the byte at 0x6a113 from 00 to 01, and when trying to open in PSE it no longer shows the caption/copyright info and gives an error when loading. If you change both bytes to match then the file is successfully unlocked and editable.
And here's the same test in pf2 format:
(https://s11.postimg.org/6jmu4wuf7/Screen_Shot_2016_09_29_at_11_20_40.png) (https://postimg.org/image/z99q1jyf3/)
(Locked on the left, unlocked on the right)
Technicolor Cinestyle on the other hand has a completely different end of file, so not sure what to make of that. It also has two sections that according to my hex editor simply run through every keyboard character.
I believe I've found some kind of start tag in .pf2 files for the LUT data - "00 10 21 00 04 00 00 00 F0 00", which is always followed by "00" or "01" and then arbitrary data. This holds true for Canon's styles available through their website, Cinestyle, and styles created through PSE.
In PSE styles, that tag starts immediately after the copyright field no matter how long that field is, and the last byte is 01. In Canon's styles it appears after a middle block of mostly empty bytes, and ends with 00. In Cinestyle it's near the copyright field and ends in 01 like PSE, but the data doesn't start immediately like PSE and Canon's.
looking on the decoded file i have the "feeling" there are kind of tags as in tiff. double bytes, starting with 0x10. btw. for the players and coders, this is the key-file (http://video.phreekz.de/PictureStyles/canon_ps.key). as @chris_overseas stated, this key is static and equal for all bodies/picturestyles (513*512bytes)
edit: Here s an php-example of the usage - how to decode (http://pastebin.com/UApwdjNF)
regards chmee
Another interesting quirk in PSE - If you open a Canon style and then save out as a new file, the subsequent pf2 contains a copy of both the original style (with tiny modifications) and your adjustments. This doesn't happen if you open your own file and save a new one.
If you decode a double-saved Canon style and compare the two blocks of data inside, the only difference is that the second byte in that start tag becomes 90 in the original block, and stays as 10 in the new block. You can also see that this flag is repeated four times throughout each data block, because the only difference is 90 vs 10. The second block of data will also begin with your new comment/copyright fields.
I made a quick and dirty OSX Automator app that works with Chris's java file, so if you put pfcoder.app and Pf3Decoder.jar in ~/Desktop/PSE, you can add the app to your dock and drag/drop pf2/3 files onto the icon and it will auto run the shell script java -jar Pf3Decoder.jar input-file (no output specified, so it will overwrite stuff). Lot faster than having to revise the terminal command every time you want to decode or encode.
https://drive.google.com/file/d/0B2q7De8nh2q7SzJ4OXp5ZXlKejA/view?usp=sharing
Here (as php-example (http://pastebin.com/56Vneuud)) the split by tag 0x10. starting piece as decimal text (http://pastebin.com/Fp7PZva6)
16 48-51 seems to be the contrast/saturation and so on values. and yes the dataformat seems to be tiff-like
16 9 0 3 0 0 0 2 0 129
tagstart and type 16 09 [2byte - int]
valuetype 00 03 [2byte - int] (3 means SHORT 2byte)
length 0 0 0 2 [4byte - long]
value 0 129 [value SHORT]
tiff specification 6.0 page 15-16 for valuetype (http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf)
regards chmee
Yep, makes sense. By testing with PSE, the tags affected by making adjustments on PSE's Basic tab and saving a file are the following (Tone Curve not included):
- 0x1002: Caption (type 0x0002 ~ ASCII)
- 0x1009: Base Picture Style (type 0x0003 ~ Short)
- 0x100a: Copyright (type 0x0002 ~ ASCII)
- 0x1030: Strength (type 0x0004 ~ Long)
- 0x1031: Contrast (type 0x0004 ~ Long)
- 0x1032: Saturation (type 0x0004 ~ Long)
- 0x1033: Color tone (type 0x0004 ~ Long)
These are not verified, but I guess semantics for the following tags that seem to be of Long type (0x0004) as well:
- 0x1034: Fineness
- 0x1035: Threshold
Here are some actual tested hexadecimal responses for changing basic adjustments in PSE.
Base Picture Style:
1009 0003 0000 0002 0081 # Standard
1009 0003 0000 0002 0082 # Portrait
1009 0003 0000 0002 0083 # Landscape
1009 0003 0000 0002 0084 # Neutral
1009 0003 0000 0002 0085 # Faithful
Strength:
1030 0004 0000 0004 0000 0000 # Strength: 0
1030 0004 0000 0004 0000 000a # Strength: 10
Contrast:
1031 0004 0000 0004 ffff fffc # Contrast: -4
1031 0004 0000 0004 0000 0000 # Contrast: 0
1031 0004 0000 0004 0000 0004 # Contrast: 4
Saturation:
1032 0004 0000 0004 ffff fffc # Saturation: -4
1032 0004 0000 0004 0000 0000 # Saturation: 0
1032 0004 0000 0004 0000 0004 # Saturation: 4
Color tone:
1033 0004 0000 0004 ffff fffd # Color tone: -3
1033 0004 0000 0004 0000 0000 # Color tone: 0
1033 0004 0000 0004 0000 0003 # Color tone: 3
Here is the caption field from Example-Decoded.pf3 (provided by @chris_overseas in a previous message) and guesses for the structure:
1002 # tag?
0002 # type: ASCII?
0000 0020 # length?
4341 5054 494f 4e5f 4341 5054 494f 4e5f # "CAPTION_CAPTION_"
4341 5054 494f 4e5f 4341 5054 494f 4e00 # "CAPTION_CAPTION<NUL>"
Data IDs:
0x0000 - KEY
0x1001 - Camera Model
0x1002 - Caption
0x1003 - Remain
0x1004 - Gnz Gamma
0x1005 - Gnz LUT
0x1006 - Lucky Data
0x1007 - Angel Regs
0x1008 - Wk Data
0x1009 - PC set Selected ID
0x100a - Copyright
0x100b - Lucky Enable
0x100c - Lucky Valid
0x100d - Site
0x1010 - Partial Info
0x1011 - Made In Canon
0x1012 - Partial Info Base
0x1018 - RGB Gamma
0x1019 - LAB Gamma
0x1020 - ICC (sRGB)
0x1021 - 3x20 Matrix (sRGB)
0x1022 - Lucky Table (sRGB)
0x1023 - Partial Info (sRGB)
0x1028 - ICC (AdobeRGB)
0x1029 - 3x20 Matrix (AdobeRGB)
0x102a - Lucky Table (AdobeRGB)
0x102b - Partial Info (AdobeRGB)
0x1030 - Sharpness
0x1031 - Contrast
0x1032 - Saturation
0x1033 - Color Tone
0x1050 - LUT Long (sRGB)
0x1051 - LUT Long (AdobeRGB)
0x1052 - LUT Double (sRGB)
0x1053 - LUT Double (AdobeRGB)
0x1054 - User RGB Gamma
0x1055 - RGB Gamma From Camera
0x1060 - Base PS Gamma ID
0x1061 - Defined Color Partial Info (sRGB)
0x1062 - Defined Color Partial Info (AdobeRGB)
0x1063 - Canon Internal Partial Info (sRGB)
0x1064 - Canon Internal Partial Info (AdobeRGB)
0x1065 - User RGB Gamma After Canon
0x1070 - LUT 3D (sRGB)
0x1071 - LUT 3D (AdobeRGB)
0x1080 - Disable Edit
0x1f00 - Camera Lucky Table (sRGB)
0x1f01 - Camera Lucky Table (AdobeRGB)
0x2001 - Sub Copyright
0x210 - Picture Style LUT Param
0x2fff - Sub Terminator
0x8000 - Backup Flag
0x8001 - PF Version
0xa001 - Work Data
0xa002 - Copyright
0xb001 - Select Object
0xfffe - Finalize
0xffff - Terminator
Data Type:
1 - Byte
2 - ASCII
3 - Short
4 - Long
0xffff - Unknown
Type:
0x2e4d524b - MRK
0x2e504632 - PF2
0x2e504644 - PFD
0x2e505345 - PSE
0x2e544344 - TCD
0x2e574244 - WBD
Picture Styles:
0x21 - User1
0x22 - User2
0x23 - User3
0x41 - PC1
0x42 - PC2
0x43 - PC3
0x81 - Standard
0x82 - Portrait
0x83 - Landscape
0x84 - Neutral
0x85 - Faithful
0x86 - Monochrome
0x87 - Auto
0x88 - FineDetail
Wow! I don't know where did you get that list but that really covers a whole lot of things. Awesome!! :)
Hey, @chmee, I'm trying to translate your php script into python, but I seem to have a bug somewhere as the file output doesn't match chris_overseas's java script. Would you be able to take a look and tell me if I'm doing the byteArray wrong or something?
If I can get it to work I can probably whip up a better OSX app for decoding/encoding, then we'll have a good start later on for quick and easy LUT->Picture Style conversion.
http://pastebin.com/ft2LkvYB
Quote from: agentirons on September 30, 2016, 12:26:18 AM
Hey, @chmee, I'm trying to translate your php script into python, but I seem to have a bug somewhere as the file output doesn't match chris_overseas's java script. Would you be able to take a look and tell me if I'm doing the byteArray wrong or something?
If I can get it to work I can probably whip up a better OSX app for decoding/encoding, then we'll have a good start later on for quick and easy LUT->Picture Style conversion.
http://pastebin.com/ft2LkvYB
Take a look at my Java source (rename Pf3Decoder.jar to Pf3Decoder.zip and have a look at Pf3Decoder.java within), it should be very easy to port to any other language of your choice. Note that it uses two key arrays (one 512 bytes, one 513 bytes) so you don't need the full 260KB array.
Thanks, @chris_overseas, I'm not as familiar with Java or C so it took me a little while to translate but I finally got it down.
Here's the working python script version of @chris_overseas's Java app for anybody who finds it useful:
http://pastebin.com/2xhBcgkT
(Make sure to copy the raw script into your own text file. If you hit the download button and try to run the file directly on OSX, you'll get errors because pastebin added '/r' newline characters to the script for some reason.)
Wow, fantastic progress on the pf2 pf3 files.
I'm thinking that these picture style files are processed by the EOS Utility and written to the camera's memory. According to a1ex:
QuoteFYI, properties PROP_PC_FLAVOR[123]_PARAM 0x401000[135] might contain user picture style data. In ML, they are only used to get the picture style names.
Maybe the rest of the data is the icc profile?
Slowly going through Canon's built-in styles and comparing setting changes to breakdown the binary data.
It seems that the 1 or 2 bytes that are 9 bytes in from the end of the pf2 files (just before FF FF FF FF 00 00 00 00) are some kind of integrity check. If you save the same settings twice you'll get identical data, but if you change anything you'll see a flag change near the top but also those integrity bytes change, and not in a 1:1 relationship. It's like the aggregate of your changes are somehow summed into those two bytes, like "Change two settings by 1 = A4", "Change three settings by 1 = A2", but also "Change three completely different settings by 1 also = A2". That includes incrementing a number in your caption field.
Spent a lot of time poring through the data last night, just for pf2 files. I think I can confirm that those bytes near the end are some kind of checksum, with the property ID being 0xfffe - Finalize (type 0x0004 ~ Long) according to @chris_overseas's list. There's also a similar 32bit checksum just for copyright field data that appears at the beginning of the file after the 0x50535000 tag.
EDIT: The 4 bytes after the 0x50535000 tag are actually the file length in bytes, not including the 12byte header.
@jkaura noticed that 0x1009 is specifically Base Picture Style - Technicolor Cinestyle lists 0x0084 as it's value, because it's based on Neutral. I've tried putting in values higher than 86, since my latest OSX PSE software doesn't even list Monochrome as an option, but of course that checksum prevents me from re-encoding the file successfully without knowing how those values are generated.
EDIT: Oh, it's called a Longitudinal redundancy check (http://"https://en.wikipedia.org/wiki/Longitudinal_redundancy_check"). Working on the formula.
In charting out the files, it seems that the tags go:
0x1009 0003 = Tag name followed by data type
0x0000 0004 = The actual length of the following data block (ie 04 = 4 bytes. The 3x20 color matrices are F0 = 240 bytes)
0x0000 0001 = Actual data, like 32bit Saturation Value
The caption field under propID 0x1002 is always a fixed 32 byte length, consisting of 31 character bytes plus a 00 ending byte.
The copyright field under propID 0x100A is a variable byte length, up to a max of 64 bytes, consisting of n character bytes plus a 00 ending byte.
Cinestyle actually matches the Canon structure closer than I thought, it's just that some of the tables and matrices are in a different order than Canon's.
Alright everyone, here's the pf2 format guide in Google Sheets format, including the annotated Technicolor Cinestyle:
https://docs.google.com/spreadsheets/d/1ePeeBpraX7AlM_pbBOTILNGQ9q0G_YQZGSEJYYFA1w8/edit?usp=sharing (https://docs.google.com/spreadsheets/d/1ePeeBpraX7AlM_pbBOTILNGQ9q0G_YQZGSEJYYFA1w8/edit?usp=sharing)
I've set it to view only while I investigate on my own but feel free to duplicate and make changes. There are two tabs at the bottom, first is a breakdown of the basic Canon styles, second tab is the full Cinestyle hex data.
Probably most interesting is the Cinestyle "User RGB Gamma" block (0x1054 0004), which is a 4096 byte(!) block that I can only assume is the actual log curve in all it's glory.
Technicolor rearranged their blocks compared to Canon and nixed a few like "Partial Info (sRGB)" and "WK Data", whatever those are. I'd love more info from @chris_overseas about what some of these tags mean, like "Site" or "Remain".
Quote from: agentirons on September 30, 2016, 09:33:30 PM
Alright everyone, here's the pf2 format guide in Google Sheets format, including the annotated Technicolor Cinestyle:
Holy Moly, John! This is all really happening and it is actually unfortunate that @dfort is still out on Vacation. Meanwhile hopefully @chris_overseas could fill us in some more on these tags. So unreal! 8)
Seriously it seems more like @Danne's & @dfort's "CineStyle Conspiracy Theory" is starting to become realistic or at least shredding some light for this project to move forward. Keep it up w the great work everyone!
May i ask the reason for reddercity's posts being removed / deleted (or whatever is happening to 'em)
He posted code from EDSDK, which is covered by an NDA.
(FYI, his ports were reported by a few users, so it's not just me)
Quote from: agentirons on September 30, 2016, 09:33:30 PM
Probably most interesting is the Cinestyle "User RGB Gamma" block (0x1054 0004), which is a 4096 byte(!) block that I can only assume is the actual log curve in all it's glory.
Converting that block to decimal reveals a 10bit log curve and it's pretty much Cineon math (0.6 gamma). The black level is offset to legal level (cv 64/1023).
spreadsheet (excel format) https://s3-us-west-2.amazonaws.com/ofxpublic/cinestyle_hex2dec.xlsx (https://s3-us-west-2.amazonaws.com/ofxpublic/cinestyle_hex2dec.xlsx)
(http://9.t.imgbox.com/BmZTlbCF.jpg) (http://imgbox.com/BmZTlbCF)
Update:
The RGB gamma blocks seem to provide an underlying s-curve which is likely to be the inverse of the s-curve used by the Neutral profile i.e. the curve you are always fighting in PSE.
Applying the inverse = Neutral linear (don't confuse linear in this context with scene-referred linear).
(http://1.t.imgbox.com/ieNOxSN8.jpg) (http://imgbox.com/ieNOxSN8)
https://s3-us-west-2.amazonaws.com/ofxpublic/rgb_gamma_curve.xlsx (https://s3-us-west-2.amazonaws.com/ofxpublic/rgb_gamma_curve.xlsx)
I've also put the RGB gamma curve into a .pf3
https://s3-us-west-2.amazonaws.com/ofxpublic/RGB_Gamma_curve.pf3 (https://s3-us-west-2.amazonaws.com/ofxpublic/RGB_Gamma_curve.pf3)
I'm not sure yet what interpolation is used (if any) but it should be possible to create alternative log profiles by first offsetting the RGB gamma curve - however, 8bits isn't going to be nearly enough for most types.
Protune might be a good alternative but that is very close to Cinestyle anyway. Original Canon log (done properly) could be viable!
Wow
can we make the fuji flog for the canon with this information?
and whats interesting for me it that they use ITU-R BT.2020 for the log encoding
http://www.fujifilm.com/support/digital_cameras/software/lut/pdf/F-Log_DataSheet_E_Ver.1.0.pdf
http://www.fujifilm.com/support/digital_cameras/software/lut/
Quote from: Andy600 on October 01, 2016, 11:58:54 AM
I'm not sure yet what interpolation is used (if any) but it should be possible to create alternative log profiles by first offsetting the RGB gamma curve - however, 8bits isn't going to be nearly enough for most types.
Perhaps Graeme Gill (http://argyllcms.com/) can provide further insight on effective curves for 8-bit capture.
Here are some 8-bit curves that minimize the quantization error, relative to the noise levels.
https://files.apertus.org/AXIOM-Beta/optimal_curve.html
TLDR: they depend on the noise profile, and work best at higher ISOs (because there are more noise bits that can be discarded without impacting the image quality). I also believe these curves are optimal for reducing the bit depth from 14 bits to 10 bits (or even 8 bits at higher ISOs) with minimal loss, if we ever *) find a way to apply curves to the raw data.
*) there is an easy coding task in the 12-bit research thread (reminder)
The big question: is the input data for these curves limited to 10 bits or not?
@a1ex so you think there may be some downsampling before the data gets anywhere near a Picture Style? I thought this too but wouldn't that be over complicating things in terms of in-camera processing because there is then another conversion to 8bit when it gets encoded for output?
Your 8bit curve research is very useful as is your comment about better results at higher ISOs. I have a hunch view that for a 'static' log profile like Cinestyle we should be shooting at higher ISOs (probably 640) and using NDs to maintain a consistent noise profile. I don't think there is any real difference between a DLSR and a dedicated cinema camera in this respect. It's the same math but we tend to shoot at the lowest ISO possible because it's cleaner. If this carried through to a cinema camera everyone would be shooting at the lowest ISOs all the time but that's not the case and the base is typically fixed between 400-800 ISO.
re: curves in raw data. @cpc's Slimraw can log encode raw data at 10bit. It's very good - but I suspect you mean a way to do it in-camera!?
Quote from: Andy600 on October 01, 2016, 11:58:54 AM
The RGB gamma blocks seem to provide an underlying s-curve which is likely to be the inverse of the s-curve used by the Neutral profile i.e. the curve you are always fighting in PSE.
Applying the inverse = Neutral linear (don't confuse linear in this context with scene-referred linear).
So if I'm understanding that right, then the DIGIC processor as well as DPP are first applying the "RGB Gamma" (0x1018) curve, which inverses the curve of the Base Picture Style, "Neutral", and then the "User RGB Gamma" curve is applied after to turn the incoming data into more or less Cineon Log?
The User RGB Gamma block is curious to me because it doesn't exist in any other pf2 files I've looked at from Canon or those made in PSE, and it clearly offers a much finer level of detail than the twelve point curve you get in the RGB or LAB gamma blocks.
Assuming that order of operations is right, and also that we could create our own User Gamma block, then it would appear the key to adding any truly custom picture style is to use some of the data blocks to cancel out the base Canon style and then use the other blocks to apply the desired curves.
Of course at the moment nothing is testable until someone can figure out how the final checksum is generated. I think I was slightly off when I said it was an LRC, it seems maybe more likely that it's CRC-32 since it's a 32-bit hex string, and I've seen at least 24-bits of it used. I'm not very familiar with the topic, but hopefully @chmee or @chris_overseas could shed some light on it. It's gotta just be a matter of figuring out how much of the file is treated as the message to encode by feeding different sections into a CRC-32 algorithm until the output matches the checksum in the file.
Once that's done I feel like we've got pf2 successfully reverse engineered and can easily create software to read/write. Then onto pf3, I guess!
I've updated the Pf3Decoder app so it now automatically corrects the checksum when processing a pf2/pf3 file with the ".decoded" extension. This means you should be able to decode a pf2/pf3 file, make any changes you wish with a hex editor, then re-encode it without worrying about the checksum being invalidated.
https://www.dropbox.com/s/4x8x9epalybm6aw/Pf3Decoder.jar?dl=0
This hasn't had very thorough testing so please let me know if you hit any problems.
I could be wrong but the 'User RGB Gamma' block (the log curve) doesn't seem to have any affect at all ???
I just filled it with linear data and the linear profile looks exactly like Cinestyle in the cam and in DPP - zero change! If I alter the other R,G & B gamma blocks i.e. by setting the first curve point output from 64 to 0, it does have the expected effect with the black level dropping.
If I'm right then it's the RGB gamma blocks (9x curve points) that are the Cinestyle curve and that's not great news*. The inverse s-curve may or may not linearize the neutral profile AND convert to log but something doesn't quite add up when looking at the actual curve - I've always assumed (until this experiment) that the base Picture Styles were linear because the ICC versions are and there are dynamic controls for altering contrast in the camera and DPP - I'm back to thinking that.
* having independent control over R,G&B gamma curves might be useful for making rough print emulation looks similar to what you could do with ASC CDL i.e. not as good as a 3D lut but better than a single 1D RGB tone curve.
Edit: one thing did come to mind regarding the log curve - it may be applied but twice (forward and reverse) and that's why I don't see a change and there may be some color processing (either the base PS or a matrix, possibly saturation) taking place in log gamma. This can be very useful when it comes to highlight saturation rolloff similar to the Alexa Film Matrix. Just a thought! forget that, if that were the case then I would still see some change in color.
So much nice development coming through lately. Not neing able to check into it all atm I just want to pass along an exiftool command which might be of interest. Or not.
Specifying exiftool -U my.CR2 file will reveal "unknown tags" and it seems there are color and picture scale tags from inside the CR2 file here which might be useful. An excerpt here which seems to be the tone curve numbers(not really sure yet).
Canon PS Info 2 0x01d0 : 188
Canon PS Info 2 0x01d1 : 86
Canon PS Info 2 0x01d2 : 192
Canon PS Info 2 0x01d3 : 87
Canon PS Info 2 0x01d4 : 219
Canon PS Info 2 0x01d5 : 0
Quote from: chris_overseas on October 02, 2016, 04:35:51 PM
I've updated the Pf3Decoder app so it now automatically corrects the checksum when processing a pf2/pf3 file with the ".decoded" extension. This means you should be able to decode a pf2/pf3 file, make any changes you wish with a hex editor, then re-encode it without worrying about the checksum being invalidated.
https://www.dropbox.com/s/4x8x9epalybm6aw/Pf3Decoder.jar?dl=0
This hasn't had very thorough testing so please let me know if you hit any problems.
Thank you! I'm on my phone so I haven't checked it out yet, but does it also modify the file length value at the beginning of the file?
Quote from: Danne on October 02, 2016, 08:13:52 PM
Specifying exiftool -U my.CR2 file will reveal "unknown tags" ...
Tip: add
>unknown.txt to save to a text file i.e.
exiftool -U my.CR2>unknown.txt
Just ran that on a CR2 that was set to Cinestyle - some parts of the
Canon Color Data 4 entries look very much like the RGB and equivalent Lab values for a color chart (not a Macbeth ,might be ITU8? dunno) - it's a lut - this 'could be' the look of Neutral. Need to compare with the ICC.
Quote from: agentirons on October 02, 2016, 08:21:18 PM
Thank you! I'm on my phone so I haven't checked it out yet, but does it also modify the file length value at the beginning of the file?
It doesn't yet, but I'll add that later today.Edit: I've now updated Pf3Decoder.jar so it updates the header with the correct data length when re-encoding files.
This is just a proof of concept but I've written a different RGB curve into a pf2 with a linear tone curve. It's based on (but not completely accurate to) Cineon + 2EV so probably best not to use it for anything. It works in camera and DPP. It also opens in PSE but there is no user accessible data there - re-saving it in PSE either as pf2 or pf3 nulls the RGB curve.
https://s3-us-west-2.amazonaws.com/ofxpublic/Cineon_Log_plus2EV.pf2 (https://s3-us-west-2.amazonaws.com/ofxpublic/Cineon_Log_plus2EV.pf2)
Okay, that's really interesting. Did you make this by swapping in your own tone curve into the Cinestyle pf2? I noticed it's still got the Cinestyle User RGB Gamma block. I'm also curious about the Edit Lock and Site tags - they got changed from 0x10800004 and 0x100d0002 to 0x0000FFFE and 0x93B3FFFF, with seemingly no ill effects? Maybe PSE simply ignored those blocks since the one before Edit Lock specifies a length of 4 bytes, and then the next block has a proper tag.
Yes, pretty much. I built a new 12point curve, converted to hex then replaced the RGB gamma blocks. I thought I'd uploaded the one with a linear User RGB Gamma block but here it is: https://s3-us-west-2.amazonaws.com/ofxpublic/Cineon_Log_plus2EV_lin_tone_curve.pf2 (https://s3-us-west-2.amazonaws.com/ofxpublic/Cineon_Log_plus2EV_lin_tone_curve.pf2)
You'll see the resulting output is no different than the profile with a log curve in the User RGB gamma block - I'm still trying to work out what that log curve is for? if anything ??? Might it be something exclusively to do with HDMI output? (though I don't know if that's even possible) or is it just a dummy curve? (that would be a shocker :o) because as far as I can tell it does nothing for H.264 or Jpeg.
No ill effects that I can see although the unlock was done before decoding the profile for the curve replacement.
QuoteI built a new 12point curve, converted to hex then replaced the RGB gamma blocks.
Very good indeed. 12 points? As opposed to the locked 10 points in PSE?
Any additional/simplifying info about how to build your own curve points and inject are most welcome whenever there is time :P
The lower left corner point and upper right corner point (black and white levels) in PSE are points 1 and 12, with a max of ten additional points in between.
If you take a look at the spreadsheet in the LAB Gamma block, thats what gets adjusted when you edit points in PSE. EDIT: Specifically the curve in the 'Specific Colors' tab, not the Tone Curve (RGB) box in the 'Basic' tab. You can see by the notes that it goes input level then output level (in hex) for each point in order, followed by zeroes if you have less than 12 points. So if you know the numbers you want for each point, just convert them to hex and edit that block, then use the latest Java coder to convert back to pf2 with a correct checksum.
I replaced the RGB gamma blocks but the points will not show in PSE because there is/was no editable Tone Curve in a pf2. Cinestyle is a pf2 and has a tone curve so it's just a PSE omission. The editable Tone curve was added later (presumably under a different tag?) as part of the pf3 format.
If you change the LAB gamma block remember it is in L*a*b* colorspace so changing the first set only will produce a change in luminosity. Changing all three will cause color issues - this is why PSE used to be a complete pain because you couldn't alter each curve independently.
As for building the curves, I did originally try mathematically plotting a known log curve (in this case a Cineon curve plus 2 EV), by converting the expression to an 8bit lut (256 CV) and then took 10 of those values as 'y values' to plot against a 0-255 linear set ('x values'). I then entered the x and y values in PSE and applied the PS to the CR2 in DPP and saved a jpeg. Then I just compared it to the same raw image (a Stouffer transmission wedge) debayered to linear and converted to Cineon plus 2EV (the +2EV was purely to remind me that I was looking at a specific test image). This was all assuming that Neutral -4 contrast was somewhat linear.
Even accounting for some small offsets there were differences caused by the contrast curve (neutral -4 because it's not strictly linear) so I then did it the hard way by roughly setting the black and white levels before saving a jpeg then set up an A/B i.e. the 2 images with a layer mask to show in split screen. Had to desaturate the images because the As Shot WB rendered slightly differently between DCRaw and DPP.
You need to gain the images up and down as you work to ensure you're matching/fine-tuning the shadow values properly. As you plot the additional points you need to move the points accordingly to produce the smoothest curve (i.e. if one set doesn't work because the interpolation causes a kink then try the next set - it can be like herding cats) but even so, 10 points isn't enough - it's very tedious and laborious work ::)
note: I made the curve in PSE before the decoder became available so I already had the curve points ready for conversion to hex.
I'm wondering if we can alter the file size to increase the RGB and/or LAB gamma blocks to 10bits or if it will only work in 8bits with 12 points. Also, I'm still trying to work out these 20x3 matrices which appear to be used for a third-order polynomial regression to XYZ space (a fairly common practice in calibration).
Cinestyle is definitely in RGB space (the sRGB matrix is all zeros), so assuming the Adobe matrix is correct it should be possible through derivation to configure a matrix to output RGB in another colorspace either by default by re-tasking the sRGB matrix or as an option by changing the Adobe1998 matrix and selecting Adobe1998 in the camera (it's a better idea to be able to select IMO).
I'm using Matlab but for hex2dec conversion you can use a spreadsheet or something like: http://www.asciitohex.com/ (http://www.asciitohex.com/) and to edit the decoded file, this is good http://www.sweetscape.com/010editor/ (http://www.sweetscape.com/010editor/)
There is a remote possibility that pf2, pf3 files are close enough to ICC standards that we might be able to add float expressions instead of hard-coded luts i.e. y=x^2.2 etc etc and maybe substitute the 3x20 matrices for simpler 3x3 chromatically adapted RGB matrices - there is lots to play with and discover :)
To do any of this more accurately/more easily the contrast function needs to be modeled to enable linearization and I think it's probably a SINE function!?
Wow, great info! Thank you for all that research. I made it clear in the Google Sheet that the three parts of the LAB block are L*, a*, and b*, I don't know why that didn't occur to me. Have you been able to confirm if the same three parts in the RGB Gamma block are R, G, and B? Cinestyle uses the same curve for all three so it wasn't immediately obvious. Same with the Matrices, I'd love to know if they are actually broken up the way I colored them in the sheet.
As far as increasing the bit depth of the gamma curves, increasing the file length shouldn't be an issue (esp once chris adds that to the decoder, for now you can do it by hand), so I suppose it's just a matter of trying it out. You'd also need to change the block length tag to match your edits, and there's no obvious tag between components in the gamma blocks so it may of course be a hard limit.
I can also recommend Hex Fiend for OSX as a free and simple editor, with a very useful comparison tool: http://ridiculousfish.com/hexfiend/
Yes, very greatful for thorough answers and quality digging along with sheets and ingenious tools. Wish I had a vacation now :P.
Hex fiend does the work great. I use it all the time. Gonna check those blocks hopefully soon.
Quote from: agentirons on October 03, 2016, 06:14:26 PM
As far as increasing the bit depth of the gamma curves, increasing the file length shouldn't be an issue (esp once chris adds that to the decoder, for now you can do it by hand)
I added support for that yesterday. Just redownload the jar and it should then update the data length entry seamlessly during the encoding step, same as it does for the checksum. I didn't test it very thoroughly though so any problems with it, let me know!
For hex editing on Windows I can recommend HxD (https://mh-nexus.de/en/hxd/ (https://mh-nexus.de/en/hxd/)).
@agentirons - I didn't try independent RGB curves yet but it's easy to verify the order by making 2 of the 3 curves all 0s and see what color you're left with. I would assume it's RGB in that order but could also be BGR.
If you convert each byte of the matrix you get a value between 0 and 255 so they are 8bit values but I don't know if the order is correct yet - it probably is. 20x3 is tricky math ??? There are toolkits in Matlab (and probably Octave, Scilab etc) but I need to do some research and learn it - I doubt I've got time to do that this week unfortunately.
Thanks @chris_overseas - your decoder is a super useful tool!! :)
Quote from: Andy600 on October 03, 2016, 04:17:47 AM
You'll see the resulting output is no different than the profile with a log curve in the User RGB gamma block - I'm still trying to work out what that log curve is for?
Quote from: Andy600 on October 03, 2016, 03:36:59 PM
I replaced the RGB gamma blocks but the points will not show in PSE because there is/was no editable Tone Curve in a pf2. Cinestyle is a pf2 and has a tone curve so it's just a PSE omission. The editable Tone curve was added later (presumably under a different tag?) as part of the pf3 format.
You've been doing inspiring research! So you came to a conclusion that User RGB gamma block is specific to pf3 format only and is not supported/conveyed by pf2 format?
No. I'm saying the User RGB Gamma block in the CineStyle.pf2 (i.e. it's log curve) doesn't seem to do anything. You get the exact same image out of DPP or an in-camera Jpeg or H.264 whatever data is in that block and I've changed the curve data multiple times and even removed it to prove this - you still get the same Cinestyle image.
The .p3f format introduced an editable RGB tone curve and a couple of other things to the PSE app where previously you had only a L*a*b curve.
Right, so, .pf2 has an RGB Gamma curve and a L*a*b* Gamma curve that both work, but only the L*a*b* Gamma curve is directly editable inside Canon's Picture Style Editor software (it's the curve box in the 'Specific Colors' tab). The RGB Gamma curve has to be edited manually using the hex editor, and I'm guessing only one is applied to the image if you have data in both curves.
PSE can edit the RGB Gamma curve if you save as .pf3. (Assuming here, I haven't actually tried it to confirm that's what's happening.)
The separate and much larger 'User' RGB Gamma block that is present in the Cinestyle .pf2 has seemingly no effect, and @Andy600 is trying to figure out it's purpose.
Quote from: agentirons on October 04, 2016, 01:31:15 AM
Right, so, .pf2 has an RGB Gamma curve and a L*a*b* Gamma curve that both work, both only the L*a*b* Gamma curve is directly editable inside Canon's Picture Style Editor software (it's the curve box in the 'Specific Colors' tab). The RGB Gamma curve has to be edited manually using the hex editor
Correct
Quote from: agentirons on October 04, 2016, 01:31:15 AM
..and I'm guessing only one is applied to the image if you have data in both curves.
No, they can both be applied
Quote from: agentirons on October 04, 2016, 01:31:15 AM
PSE can edit the RGB Gamma curve if you save as .pf3. (Assuming here, I haven't actually tried it to confirm that's what's happening.)
You can edit the pf3 tone curve in addition to the RGB gamma curve data but any hex data you enter into the RGB gamma block does not show as control points in PSE - I really don't get what is going on with all these curves!
Quote from: agentirons on October 04, 2016, 01:31:15 AM
The separate and much larger 'User' RGB Gamma block that is present in the Cinestyle .pf2 has seemingly no effect, and @Andy600 is trying to figure out it's purpose.
Yes and still trying to figure it out ::). I did think that maybe it was applied only to H.264 and that I might not be seeing the true result of the curve due to the colorspace I was in but I've checked tweaked Cinestyle H.264s in Rec709, sRGB, Linear and across multiple color apps and a couple of NLEs - still the same result. The curve does nothing here. The last hope is HDMI output but I doubt that's going to be anything different.
PSE and DPP don't seem to like independent R,G & B Gamma curves either ???
very early stage. PictureStyler (http://video.phreekz.de/pictureStyles/) as Online-Tool. Im at a job in L.A., hadnt the time to code more :)
v0.2 - PF3 only, uploading, viewing 0x10-Tags
regards chmee
Uploaded mine and dfort,s logNeutral.pf3
Looking good.
(https://s14.postimg.org/w0hho8hlt/Screen_Shot_2016_10_04_at_17_48_56.png)
Looking really good @chmee re: PictureStyler however when I go ahead and upload a pf3 file it just stays on loading for a long time.
Is this normal?
by now, yes. its the moment, the filereader-api reads the file into memory. (look into a inspector [F12]). later i can try to speed up.
Fascinating little detail I picked up by examining Canon's Downloadable Picture Styles (http://web.canon.jp/imaging/picturestyle/file/download.html) - You can actually delete the Edit Lock, Finalize (Checksum), and Base Picture Style settings (sharpness, contrast, saturation, color tone) blocks entirely!
For some reason Canon omitted all three of these sections in the Nostalgia, Clear, Twilight, Emerald, and Autumn Hues styles.
For Studio Portrait, Snapshot Portrait, and Video Camera X Series Look, they kept Finalize and the Base Settings but omitted the edit lock.
| EDIT LOCK | | FINALIZE | | BASE SETTINGS |
NOSTALGIA |
CLEAR |
TWILIGHT |
EMERALD |
AUTUMN_HUES |
P-STUDIO | | | X | | X |
P-SNAPSHOT | | | X | | X |
VIDEO-X | | | X | | X |
@agentirons
great find. so it behaves quite like tiff, where some Tags are mandatory, some are recommended, some are free to use.
I want to bring back reply 137 from a1ex (http://magiclantern.fm/forum/index.php?topic=16299.msg172641#msg172641):
QuoteFYI, properties PROP_PC_FLAVOR[123]_PARAM 0x401000[135] might contain user picture style data. In ML, they are only used to get the picture style names.
Sizes: 16704 on 5D3, 600D, 16680 on 5D2, 550D.
Example:
0x4010001 16704 0x824140 'Flaat_2' 0xffff00fe 0x0 0x0 0x0 0x1 0x219c60 0xfffffffc 0x2 0xfffffffe 0x0 0xdeadbeef 0xdeadbeef 0xfc0040 0xffff2fff 0x0 0xf64ad173 0x7e60894d 0xf452b0c0 ...
HTH
Here's a peek at one of the custom picture styles embedded in the ROM1.BIN file that ML creates in the Logs directory on the memory card:
(https://c8.staticflickr.com/6/5560/30182460615_1439b67071_c.jpg)
Looks like the picture style data is stored unencrypted in the camera's non-volatile memory.
Note that the address is way off because it needs an offset of 0xFF000000 to match what a1ex is pointing out. Uh--I think, maybe? More information can be found in the finding stubs tutorial (http://magiclantern.fm/forum/index.php?topic=12177.msg117735#msg117735).
So I most likely missed this but did anyone figure out why RAW video is so much more detailed then the h.264? Is it the the 444 vs the 420 or is it compression algorithms or do we think that canon is trying to blur those hit pixels. I noticed the 5Ds and 5DsR have a fine detail picture style and I didn't know if this is something that could help the older cameras have a sharper h.264 option. I would think the all I codecs could provide more detail but maybe its not possible in camera at 24fps. What is the consensus on this?
Quote from: hjfilmspeed on October 09, 2016, 04:07:32 AM
did anyone figure out why RAW video is so much more detailed then the h.264? Is it the the 444 vs the 420 or is it compression algorithms or do we think that canon is trying to blur those hit pixels.
It's not 444 that's for sure.
Raw Video is 14bit RGGB , no color space no processing at all , well there's a little from sensor data to ISO RGB then linearize more or less
( there a few more things done but that's another thread) saved in .mlv container then extracted as 14bit dng or 16bit Cdng format.
H264 all "I" or not it's 8bit 4.2.0 color space with burnt in Picture Style (not much latitude , very little) which is very inferior .
I guess you can ask your self Why are CR2/Raw photo cleaner and sharper then the Jpeg photo ? 14 vs 8 Bit .
True. Maybe this should fall under the pipeline topic. But I do believe there is something else going on. I mean there is a fairly substantial detail loss there.
The 5Ds and 5DsR also allows for Fineness and Threshold settings under Sharpness. Note that if you load a CR2 file from any other camera you can still adjust Sharpness from 0-10 but you can't move the Fineness and Threshold sliders.
(https://c6.staticflickr.com/9/8414/30130897781_87b2dbea3b_z.jpg)
I have not seen video from 5Ds and 5DsR with the "Fine Detail" picture style. Considering everything that's going on H.264 video from these Canon DSLR cameras is really pretty good--in my opinion.
Note that the APS-C cameras actually line skips the full sensor and shoots 1728x972 then uprezes in camera (someone can fact check me on this) while the Canon C100 actually has a 4K sensor and downscales internally so the H.264 files from the professional line is much better. The 5D3 shoots full 1920x1080 and uses pixel binning which is superior to line skipping and the resulting H.264 movie files are almost as good as the "real" cine cameras.
Of course H.264 is a lossy codec so that will account for some loss of detail over a raw image.
Quote from: dfort...Note that the APS-C cameras actually line skips the full sensor and shoots 1728x584 then uprezes in camera (someone can fact check me on this) while the Canon C100 actually has a 4K sensor and downscales internally so the H.264 files from the professional line is much better. The 5D3 shoots full 1920x1080 and uses pixel binning which is superior to line skipping and the resulting H.264 movie files are almost as good as the "real" cine cameras.
The 7D2 does this also? It sure looks like an APS-C version of the 5D3.
It has to be verified 7D2 indeed does binning. But unscaling has to take place. 1920 x 3 = 5760 pixels. 7D2 -> 5472 pixels horizontal. Close ...
Quote from: Walter SchulzIt has to be verified 7D2 indeed does binning. But unscaling has to take place. 1920 x 3 = 5760 pixels. 7D2 -> 5472 pixels horizontal. Close ...
Ah, that's what I thought. I used to have a couple of 6D's and the line skipping was horrible. Binning, although not great, is much better IMHO to line skipping.
P.S. I shot an entire 26 episode season of a children's TV show using the Neutral profile flattened out. I've recently tried James Miller's CLog3 and Andrew Reid's EOSHD CLog. I have a hard time pulling shadow detail out of those "Log" profiles. So I recently started using "Faithful" flattened out. If I expose protecting highlights, I get much better shadow detail. Very interesting. I wish I would've used "Faithful" on those episodes.
Quote from: ddelreal on October 09, 2016, 07:40:42 PM
...I shot an entire 26 episode season of a children's TV show using the Neutral profile flattened out. I've recently tried James Miller's CLog3 and Andrew Reid's EOSHD CLog. I have a hard time pulling shadow detail out of those "Log" profiles. So I recently started using "Faithful" flattened out. If I expose protecting highlights, I get much better shadow detail. Very interesting. I wish I would've used "Faithful" on those episodes.
You bring up a couple of interesting points.
- What differences have you observed between using "Neutral" and "Faithful" flattened out? The reason I'm interested is because on our CineStyle like picture styles I dropped logFaithful on the latest test version, maybe I should bring it back?
- Do you have an issue pulling shadow detail with CineStyle? I find that I'm getting more shadow detail with CineStyle but haven't done a direct comparison with those other log picture styles yet.
QuoteI have a hard time pulling shadow detail out of those "Log" profiles.
The little I tried of various mimic C-logs they clip shadows too much. I compared with cinestyle(logNeutral) and shadow details are much better preserved.
I tried to refine the shadow part in cinestyle(logNeutral) but it,s just too good as it is for me to bring out anything else from it.
Have you tried logNeutral? Or log logFaithful? @ddreal
Faithful also seems to help skin tones, but Cinestyle is good with shadow detail too. I think I'll be using those two styles for now until something changes. I haven't used logFaithful yet, I may give it a try.
Quote from: dfort...What differences have you observed between using "Neutral" and "Faithful" flattened out? The reason I'm interested is because on our CineStyle like picture styles I dropped logFaithful on the latest test version, maybe I should bring it back?...
You have to be extra careful with Neutral when exposing, it can get muddy and difficult to recover in post. Faithful, even flattened out seems to give more color information and shadow detail - that is if you're exposing to protect highlights.
I like Cinestyle too, I'll be using both styles for now depending on my shooting situation.
I,d say neutral is the most rec709-ish colorspace of them all and I cant, really say what the other styles are but they can of course work as you are the creative mastermind here.
Would be nice to have any second opinions about neutral(rec709) colorspace thoughts.
Funny you say Neutral is the most rec709'ish, to me it seems like good ol' faithful is the closest to Rec709. But I dont have any technical analysis to back this up, but it looks to me like Faithful is the closest to the colours and DR of Rec709 broadcast cameras. Not saying the look of Faithful, because it definitely has a style to it, but saturation wise and shadow to highlight ratio seems close to rec709. The look of Faithful is this suttle teal/green/blueish cast, which I really like, and if you nail the WB I find it easy to grade in to a very filmic look. I'll post a video later
Also converting the video levels to log yields more information in the highlights and shadows than first perceived. I guess because the look is so contrasty, the compression algorithm attacks the parts of the image where no information is and leaves more bits for real information contrary to cinestyle where the compression has to deal with the entire image because nothing is crushed. I am not trying to bash the cinestyles or logneutral. I've shot with cinestyle since its release, but i was digging up old footage for a project from before cinestyle was born, it was shot in Faithful and I was amazed at how good the images were and how easily gradable they were(good wb). And i believe that if i shot that project in cinestyle i would not get such good results as i did. Especially my latest project where i used all this old footage from that shoot and graded it with all the knowledge I've learned since then. Again, I'll post it later. If you wanna download 5 gigs.
Yeah, pretty much my same findings. I can't technically explain it but I can get more filmic looks from Faithful than any other style.
Thanks Kharak and ddreal. Maybe logFaithful would be worth a try then reading about opinions/conclusions.
Indeed logFaithful is definitely worth another shot to take a dive with and will share my findings.
Will test again thoroughly between Faithful, Neutral, logFaithful v1.3, logNeutral v1.3 and maybe few other 3rd party log's that I haven't tried yet.
The one PP has me deeply curious is Canon's very own 'Emerald' version in which a few have reported supposedly nice results earlier in this thread -- will probably give that one a test run as well.
The Faithful Video.
Some friends of mine were producing an Opera/show, I was asked by them to produce a video for the Opera part of the show, they wanted something rough and preferably from the sea, luckily I have a bunch of footage like that. The video was to be projected behind the opera singer, it was quite hard and fun at the same time for me, because they were doing this show in another country then I was in and they didn't have an overall idea of what they wanted, "rough and wild, but slow but it follows the tempo". So I was very free. I got the opera song they were going to sing via a youtube link, the youtube song was from a big concert from a famous German female opera singer (forgot her name) so I had to match the tempo to another Opera.. So I just followed my gut and cut it to the youtube version.. The tempo matched perfectly they said. I never got to the see the show.
:link removed:
This is the rough cut I already have uploaded, the final version is very much similar.. So I don't wanna upload another 5gigs. This was the final Color grading also. The youtube version song is in the video for reference only
Ignore the unfinished black border and quirky fast forwarding in the one scene.
I shot this with a Cinemascope in mind, this was before ML or atleast before I knew of ML, so I had no Cropmarks to follow, I framed by centering everything a bit more.
5D MK II
PP-Faithful.
Canon 24-70 2.8
Polarizing filter
All handheld on a rolling ship..
Neat Video, Dark Energy, Video Levels to Log LUT - Conversion, CC, log2hd Luts, Graded in AE, Synthetic Aperture and Colorista.
Cut in Premiere.
Don't get sea sick. I'll leave the link up for a few days.
Yeah, that got me a little queasy. Add some dirt and scratches and you've got a film look.
Just kidding, nice moody piece. Thanks for sharing.
So this was shot on Faithful but on the default settings, not flattened, right?
I don't recall, but I don't think I've touched any of the settings. That was my first piece shot on DSLR, I had just purchased the 5D MK II and it was my first venture in to the DSLR video recording. I just remember that I only shot on Faithful before Cinestyle was introduced.
I bought the MK II for video recording, I think by the time I upgraded to MK III, my MK II had taken just over a 1000 Photos and about 15000 Shutter activations from switching to Liveview for video.. heheh
[btw] is it necessary discussing usage of picture-styles in a "REVERSE ENGINEERING"-Thread?
No you are right. It all stemmed from what PP is closest to rec709 to me babling too much about faithful. I'm a talker and sometimes lose track of my surroundings.
LogFaithful sounds interesting though, but not sure if I would ever use it other than reference. Read my signature.
Quote from: chmee[btw] is it necessary discussing usage of picture-styles in a "REVERSE ENGINEERING"-Thread?
True to a point, I think application of said REVERSED ENGINEERED style is important as well.
Quote from: KharakLogFaithful sounds interesting though, but not sure if I would ever use it other than reference. Read my signature.
I just downloaded LogFaithful 1.3 and will try soon.
Guys-
I know this may be expected but Canon/Apple has done it again. Upgraded to 10.12 Sierra on OS X and now EOS Utility (all three versions) don't seem to work.
Version 2 just hangs with spinning ball.
Version 3 just tells me that the camera is unsupported (when I know it is) as I've previously used it before the upgrade.
Also found this forum (http://www.canonrumors.com/forum/index.php?topic=31020.0forum) which shows similar problems of others reporting 10.12 Sierra breaking things up -- What gives?
LogFaithful is my new friend (until Raw becomes a dream come true). Magenta seems more correct than Cinestyle, and less yellow. Really like it on my 7D2's.
Interesting, I thought logPortrait might be the hero because people were complaining about skin tones on CineStyle but it looks like logFaithful is the underdog.
I'll have to wait to load it on my camera out since I'm on Sierra now:
(https://c8.staticflickr.com/6/5723/30330434375_e57b661294_z.jpg)
Does anybody knows the base/native ISO of the 5mk3?
Im not sure this is the right place for these noobie question but:
http://s1119.photobucket.com/user/hingsberg/media/canon_c-log_iso.png.html
I saw this for the sensor on C canon cameras, this works for 5dmk3 too?
Seams the native/base ISO of cinema cameras are 640. As ask before does anybody knows the one for 5dmk3?
and finally... in the same papers said the 18% should be at 32.79% IRE for Cinema Cameras. Does anybody knows if the same goes for all this news C-log picture profiles?
@ImaXineria - If you are referring to the base ISO on the C100 using log mode the Canon documentation rates it at ISO 850 and not 650 as you commented in your post. You can go higher but shouldn't go lower. This may seem counterintuitive to still photographers because we're used to setting the camera at the lowest ISO to get the best image. Here's an article (https://www.cinema5d.com/exposing-with-native-iso/) that explains this better than I can.
Take another look at the graph you referenced:
(https://c2.staticflickr.com/6/5759/30099716790_a5854ae639_o.png)
Note that ISO 850 is where the "Canon Log Base Sensitivity" is pointed out.
There was a discussion between @a1ex and @Andy600 that we should also be using higher ISO values when using log profiles in this post (http://www.magiclantern.fm/forum/index.php?topic=16299.msg172787#msg172787):
QuoteI have a view that for a 'static' log profile like Cinestyle we should be shooting at higher ISOs (probably 640) and using NDs to maintain a consistent noise profile. I don't think there is any real difference between a DLSR and a dedicated cinema camera in this respect. It's the same math but we tend to shoot at the lowest ISO possible because it's cleaner. If this carried through to a cinema camera everyone would be shooting at the lowest ISOs all the time but that's not the case and the base is typically fixed between 400-800 ISO.
As far as exposure, once again we find ourselves at odds over what we normally do as still photographers. It has become common practice to expose as far to the right of the histogram without blowing out the highlights. However, that's not the way to properly expose a log profile. You are probably referring to this AbleCine blog post (http://blog.abelcine.com/2012/10/05/working-with-canon-log/) about working with Canon Log--it has the graph that you pointed out. There is also this post (http://www.hdvideopro.com/columns/help-desk/the-rules-of-log-exposure/#) showing that each log profile has an optimum IRE value for 18% grey and 90% white.
I'm not sure how to work this out for the cameras and profiles that we're using here but if anyone has ideas how the "Base ISO" for each log picture style can be calculated and how it should be exposed, it would certainly add to this discussion.
5d mk iii base iso is 200, same for mk ii. But mkiii has greater DR at lower iso than base e.g. iso 100 and the dark and ominous iso 66.
Cinestyle's base iso for 5d mk ii is 160.
@dfort: wonderful. great info. Really really thanks a lot
@Kharak: Thanks for the answer. How do you get the number?
or how did you get the number (base ISO)?
When it comes to MLV RAW -- I've always liked the results when the 5D3's at ISO 400 give or take. Used to think it was ISO 800 until @quickhitrecord (http://www.magiclantern.fm/forum/index.php?topic=13152.msg163666#msg163666@quickhitrecord) & @baldavenger's (http://www.magiclantern.fm/forum/index.php?topic=15801.msg164620#msg164620@baldavenger's) each separate experiments which proves otherwise respectfully.
But in here we are talking about log styles being recorded into H264 format (not RAW) which is a whole different ball game and we normally (or at least should) shoot them in increments of ISO 160 such as 160, 320, 640, etc...
I've also noticed that 5D3 likes to sit at ISO 640 (either close down or use ND) or drop down to ISO 320 which seem to have slightly better shadow noise than at ISO 160.
FYI -- when we use C100II out on the fields at where I work -- it's 'sweet spot' is actually ISO 850 which is also being recorded into Wide DR custom profile (rather than a flat log) due to quicker turnaround workflows in post for broadcasting contents per Editors/Producers requests.
Guess it all comes down to how much time do we have in post to determine whether to shoot in a customized profile look or a flat log, right?
Thanks for this @DeafEyeJedi it always is enlightening to hear what is going on in the "reel world" ;D
I wonder how to scientifically determine the base ISO for various cameras using the available log profiles and also with the flat picture styles.
That reminds me, one of these days when your work load lightens up we should do that EOS-M vs. C100 Mark II test we talked about. If nothing else it would be interesting to see what real Canon Log looks like next to the various log picture styles. It would be fun too.
I hope this is on topic but I have been working with VisionLog DCP in A.E. with an added color/tone curve I'm getting very good results in the mids .
To the point , I found a open source pc app called "dcp2icc" which makes a icc profile from a dcp .
Since the picture profiles are really icc profiles for your camera , I thought if some how this information could be useful to get a neutral flat high DR. profile
optimized for magic lantern enabled camera's . Below are some screen shots from DNG Profile Editor and one from the command line with dcp2icc
(https://c4.staticflickr.com/9/8621/30371814091_9c8b51180f_n.jpg) (https://flic.kr/p/NgRuc6)
VisionLogBaseToneCurve (https://flic.kr/p/NgRuc6) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
With Vision log tone curve looks like there a lot more work on shadow detail.
(https://c1.staticflickr.com/9/8616/30160671720_15b7308e63_n.jpg) (https://flic.kr/p/MXcjVo)
Embeded_Camera_Profile-Base-Tone-Curve (https://flic.kr/p/MXcjVo) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
(https://c1.staticflickr.com/6/5666/30422030136_abe97f59ff_n.jpg) (https://flic.kr/p/NmhREY)
Camera_Netrual_5D2_Profile-Base-Tone-Curve (https://flic.kr/p/NmhREY) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
(https://c5.staticflickr.com/6/5337/30160672060_55fa551029_m.jpg) (https://flic.kr/p/MXck2f)
dcp2icc (https://flic.kr/p/MXck2f) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Usage: dcp2icc.exe <dcp> <color temperature>
make sure the xx.dcp is in the same root as dcp2icc.exe , just like the screen shot.
download at https://sourceforge.net/projects/dcp2icc/
Very interesting discoveries. There's also an open source tool to work with dcp profiles, dcptool (http://dcptool.sourceforge.net/Introduction.html).
This is all very much on topic because we're pretty sure that picture styles are saved as icc profiles in the camera. We just have to figure out exactly where the icc information is saved.
Yes, conversion tools are great to share. One thing can lead to the other. So we have dcptool for analysing and manually build dcp profiles.
http://dcptool.sourceforge.net/Introduction.html
Hopefully all logs are based on what adobe calls linear.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg172443#msg172443
Too build a mathematical log rather than a log "look" we would need the dcp to act as a 1D lut. It would be interesting to see a conversion from cineon or logC to dcp using dcp tool since dng profile editor itself only allows for 96 curve points. This however would be a totally a different reverse engineering project and could benefit from being discussed in a new thread.
Absolutely wonderful stuff @reddeercity and definitely agree with @Danne on the fact that we should focus more on mathematical logs rather than just a log "look" -- so for us to be able to spit out our very own linearized DCP profiles is rather remarkable and works flawlessly in MLP.
Re: Conversion from Cineon/logC to DCP for the 96 curve points is totally a new thread worthy already!
Excellent discoveries guys.
Since we know now that PS are somewhere ICC, or we can inject ICC in PS, we can imaging many things!
So sure a good math log would be very nice (I always found that filming things of life in linear was inhuman), and I saw we also could modified the color balance if I'm not wrong (?); it will be pretty nice to generate a couple of PFs calibrated for some special situations, we could switch fast between then while we are filming. But I think I've a lack of information about the internal process; I thought after readings here and imaging that RAW/PS->H264 4:2:0 but if we can modify the color balance with PS do you how this is interfere with the internal color balance?
When I will found where to download a recent version of EOSU to load PF3 in my 5D2 I will try to make some tests..
Quote from: SpcCb on October 22, 2016, 04:17:25 AM
...if we can modify the color balance with PS do you how this is interfere with the internal color balance?
When I will found where to download a recent version of EOSU to load PF3 in my 5D2 I will try to make some tests..
You bring up a couple of interesting points.
Can we modify the color balance with the picture style? I would think that if the picture style has a bias towards warm or cool colors then sure, it will have an influence over the color balance but you can still adjust the color balance and create a custom white balance. So what affect does adjusting the color balance have on the "look" of the picture style? In other words, is white balance applied in the camera before or after the picture style is applied? This is something that I haven't tested.
You can go to any of the official Canon sites for the various regions and there should be a software download section for your camera. On the USA site it requires that you enter your camera serial number before downloading. Now what I just discovered is that the EOS Utility for a newer camera like the 5D4 has version 3.5.0 while your 5D2 has an older 2.14.20a. This shouldn't make any difference to you because 2.14.20a can handle the .pf3 picture styles.
Mac Sierra users like me will have to wait for an update to the EOS Utility from Canon that works with this new operating system.
Quote from: dfort on October 22, 2016, 04:53:18 PM
(...) is white balance applied in the camera before or after the picture style is applied? This is something that I haven't tested. (...)
I had the same thought. And also that internal WB is apply directly from RAW (during debayering process), and PS WB *after* in RGB mode (ICC RGB/LAB way supposition). I hope not, because it should mean range will be short so reserved for 'film effects' stuff.
Indeed, I saw EOSU v3.5.0 but even in Europe we need 5D4 serial number to download it.
I'll try with EOSU 2.14.20a, I will see if we can play with color balance with this version.
Sorry for offtopic, but, is there any theoretical options to create a tool to convert LUT files to Canon Picture Style files?
That is very much on topic. We haven't quite figured out how to make a picture style file that works with EOS Utility from scratch but some people are figuring out the pieces. It might also be possible to insert picture styles directly into the camera without using EOS Utility once the location and format that it is stored in is found. It seems that picture styles are stored in using icc profiles so is there a tool that can convert a lut to an icc? This is all theoretical at this point.
@350D that's my end goal. I think a handful of us are working separately on teasing apart the code in various ways and building some tools.
Keep in mind ICC is more like a container format which can hold both LUT and matrix/shaper data. Should we discover Canon uses a standard-compliant implementation conversion between formats would be trivial with open source tools (http://www.color.org/opensource.xalter). First we need to figure out how to reliably extract ICC profiles from a pf2. Opening such a file in ICC Profile Inspector could also give us some clues.
http://nikonpc.com (http://nikonpc.com)
I know this is for nikon, but its so awesome :)
Quote from: Lars Steenhoff on October 26, 2016, 02:46:36 AM
http://nikonpc.com (http://nikonpc.com)
I know this is for nikon, but its so awesome :)
Wow, that is awesome. A web based picture style editor.
I have no idea where is the right place to post this
I got this results
Does somebody could confirm are acurate?
Please?
Using the waveform of the atomos ninja blade
Datacolor greycard
Arri fresnel 650 with CTB
5dmk3 with 70-200/2.8
f t EI (ISO) STOPS
PP Neutral
4 8 100 +3
4 60 100 0 TOTAL 6
4 500 100 -3
4 100 1250. +3
4 800 1250 0 TOTAL 6.5
5 8000 1250 -3.5
Eos Log
4 6 100. +3.3
4 60 100 0 TOTAL 8.6
4 2500 100 -5.3
4 80 1250. +3.3
4 800 1250 0 TOTAL 8.9
9 8000 1250. -5.6
@ImaXineria your table is rather difficult to read so if you don't mind I cleaned it up a bit.
f t IE STOPS
Neutral
4 8 100 -3
4 60 100 TOTAL 6
4 500 100 +3
4 100 1250 -3
4 800 1250 TOTAL 6.5
5 8000 1250 +3.5
Eos Log
4 6 100 -3.3
4 60 100 TOTAL 8.6
4 2500 100 +5.3
4 80 1250 -3.3
4 800 1250 TOTAL 8.9
9 8000 1250 +5.6
Now what are we looking at? I assume f = f-stop and t = time but what is IE? Do you mean EI = exposure index? What about STOPS?
This looks interesting and very much on topic but could you please give a clearer explanation of this table?
Quote from: dfort on October 27, 2016, 08:21:54 AM
@ImaXineria your table is rather difficult to read so if you don't mind I cleaned it up a bit.
f t EI STOPS
Neutral
4 8 100 +3
4 60 100 0. TOTAL 6
4 500 100 -3
4 100 1250 +3
4 800 1250 0. TOTAL 6.5
5 8000 1250 -3.5
Eos Log
4 6 100 +3.3
4 60 100 0. TOTAL 8.6
4 2500 100 -5.3
4 80 1250 +3.3
4 800 1250 0. TOTAL 8.9
9 8000 1250 -5.6
Now what are we looking at? I assume f = f-stop and t = time but what is IE? Do you mean EI = exposure index? What about STOPS?
This looks interesting and very much on topic but could you please give a clearer explanation of this table?
Thanks!
EI means ISO
Stops - underexposed or + overexposed from base exposure
the that i am trying to discover is the dynamic range of sensor with different pictures profiles and EI (ISO) so could learn if picture profile really increases the dynamic range or only allows visually the color correction
and know if the chart for the sensors of the series c cameras also works with the 5dmk3 camera in where it says that -contrary to the logic- the high EI increase the dynamic range towards the high lights and the low EI increase the dynamic range towards the low lights
Hum--the table still looks a bit wonky on tapatalk. Maybe I should have put it in code tags instead of just using courier font.
In any case, there seems to be a couple of typos. The 2500 in the EOS Log section should probably be 500 and f9 should probably be f5? (5.6?) That's just being nit picky--the real issue is that you are seeing a big increase in dynamic range with the log picture style and a slight increase at higher ISO settings. It looks like your neutral grey target is a bit too high in the Log test. Note that you have more stops below middle grey than above it. When looking at log footage before being processed it is generally quite dark. Your test seems to confirm the articles that I pointed out in previous posts. I don't have a Ninja 2 but looking at the manual it seems that it doesn't have a waveform monitor so you can't set your grey value to various IRE values.
Anyone confirm this or am I way off base with my observations? It would be interesting to determine the base ISO of your setup. Some people say it is around 650 but doesn't it depend on which log picture style you're using?
[EDIT] Now I sort of get what you are doing with the aperture but how are you getting those t values? 1/6th sec. to 1/8000 sec. exposures recorded on a Ninja 2. How?
Can everyone check on their Mac's whether or not EOS Utility now all of a sudden works on their OS X systems?
(https://c3.staticflickr.com/6/5671/30575768386_e7a2aeaa38.jpg) (https://flic.kr/p/NzSNGs)
Recently Apple released an update (10.12.1) and what's even more funny is the fact that I haven't done this update yet. :o
Yep started working here too. Strange--I thought we'd have to wait for Canon to come up with a fix.
Re @dfort "Now I sort of get what you are doing with the aperture but how are you getting those t values? 1/6th sec. to 1/8000 sec. exposures recorded on a Ninja 2. How?"
my setup:
datacolor graycard
arri fresnel 650 CTB 216
5dmk3
hdmi
atomos ninja blade waveform
base
f4 t: 1/60s EI(ISO) 100 Profile NEUTRAL and EOS HD LOG
f4 t:1/800s EI(ISO) 1250 Profile NEUTRAL EOS HD LOG
process:
base exposure 50 IRE for NEUTRAL 32IRE for EOShdLog
overexpose using time exposure as possible until reach 99IRE
underpose using time exposure as possible until reach about 5IRE
Has anybody ever captured the USB data being transferred from eos utility to the camera?
I did a couple of years ago when trying to reverse engineer picture styles.
I figured why bother trying to reverse engineer the picture style, I'll just reverse engineer the data and then write it to the camera in my own program.
Anyway, the data that gets transferred is nothing like the picture style file.
I'm pretty sure it double checks the data as well against itself after transfer.
In addition to that, when you delete the picture style it writes a tonne of zeros.
If you reverse engineer the picture styles to use them in eos utility or dpp you're still limited to how canon programs read those files to create whatever stream of data is sent to the camera.
Quote from: ItsMeLenny on November 03, 2016, 12:12:19 PM
Has anybody ever captured the USB data being transferred from eos utility to the camera?
How did you do it?
From the beginning I assumed that the picture style files had information that was only useful for the Picture Style Editor and what was stored in the camera was basically boiled down, compiled, flattened or whatever metaphor works for you, into a simplified form that works with the camera. Right now most of us are assuming what is stored in the camera is an icc profile.
Writing a tonne of zeros is a clue that the picture style is possibly stored as a block of data that is a fixed size. When ML creates the ROM1.BIN file it also dumps the picture style data (http://www.magiclantern.fm/forum/index.php?topic=16299.msg173050#msg173050). I would supposed that what you are seeing in that data stream is the same information that's in that in that data dump.
So if you're saying that if you delete a custom picture style it fills up that block with zeros then that's a great clue to help figure out where the picture styles are stored by comparing a ROM1.BIN with and without custom picture styles installed.
Quote from: dfort on November 03, 2016, 03:27:41 PM
So if you're saying that if you delete a custom picture style it fills up that block with zeros then that's a great clue to help figure out where the picture styles are stored by comparing a ROM1.BIN with and without custom picture styles installed.
This smells legit to me... Thanks for the wake up call @ItsMeLenny!
So, this is in between the camera and host sending little bits of data to each other continuously in EOS utility, probably to make sure it's still connected.
The computer sends to the camera data with a length of 576 which contains the picture style name.
The computer then sends a mass amount of data with the length of 16284, regardless if it's writing a picture style or blanking it. (Although whether this is a result of blanking the same picture style, but I doubt it.)
The camera then sends back a different data with a length of 576 which also contains the picture style name.
It then sends data of a length of 19520, then a length of 13376, then data with a length that changes (probably between 400 to 600), then sends data with a length of 440 (which is almost the same every time, bits of hex change at the beginning and end).
In between all that data of less than a length of 120 is constantly being sent back and forth.
(Lengths are in bytes and I'm not sure if they're 100% correct)
(In fact, minusing 64 seems to correct them 576-64=512, i dunno)
Quote from: DeafEyeJedi on November 03, 2016, 05:51:50 PM
This smells legit to me... Thanks for the wake up call @ItsMeLenny!
I've been watching this post for a while and have been tempted to say something for a while.
There'd be a post somewhere in the forums that I made at least a couple of years ago when I first wanted to "reverse engineer picture styles".
I am not a fan of canons ugly "film curve", I am most interested in getting a perfect gamma curve of somewhere between 1.8 and 2.2 (or 0.65 to 0.45 (or whatever) if you work it out backwards).
I think gamma curves are a good place to start, or even better creating a completely linear picture style, it would be absolutely useless to use, but good for the theory of it.
Quote from: ItsMeLenny on November 04, 2016, 08:07:28 AM
...it would be absolutely useless to use, but good for the theory of it.
Great quote.
The pf2 format custom picture styles on the official Canon website and CineStyle (when unlocked for editing) will open up in Picture Style Editor with all the settings in their default positions. This seems to indicate that these picture styles were not created with the Canon Picture Style Editor but with something that has perhaps more precise control.
Our logNeutral, logStandard, etc. picture styles were an attempt at matching CineStyle using the Canon Picture Style editor. These styles had to be in pf3 format in order to save the RGB curve. These "reverse engineered" picture styles allow users to modify it if they want or to just look at what the curve looks like in the editor.
Creating picture styles that can be loaded on the camera without using Canon's EOS Utility is something else that would be interesting. This should allow switching out the 3 picture styles loaded into memory with others that are saved on the card. Why more than 3 custom picture styles? It might be absolutely useless but good for the theory of it!
Quote from: ItsMeLenny on November 04, 2016, 08:07:28 AM
I think gamma curves are a good place to start, or even better creating a completely linear picture style, it would be absolutely useless to use, but good for the theory of it.
Agreed -- very well said!
Quote from: dfort on November 04, 2016, 03:59:35 PM
Creating picture styles that can be loaded on the camera without using Canon's EOS Utility is something else that would be interesting. This should allow switching out the 3 picture styles loaded into memory with others that are saved on the card. Why more than 3 custom picture styles? It might be absolutely useless but good for the theory of it!
Excellent theory actually.
Now this is interesting ??? https://youtu.be/u7RjJNLnWF8?t=41m41s watch at 41min 41 seconds i hope the link save that time
I didn't catch the software they are using but the Cam is a 1DX , came from a post on the Research exemption for consumer devices (DMCA) (http://www.magiclantern.fm/forum/index.php?topic=18120.msg174250#msg174250)
thread that nikfreak posted . As you see the cam setting are being access and presumably hackable .
So there some merit to look at the connection ptp/ip at least with the newer cams 5D3 and up
Quote from: reddeercity on November 04, 2016, 10:32:28 PM
I didn't catch the software they are using but the Cam is a 1DX
They're using their own python program on linux, however the underlying library is gphoto or libgphoto2 (whatever it is referred to nowadays). It's the go to for all cameras on linux (and pulling pictures from the camera as well as some other functions, controlling the camera, triggering the shutter, etc).
At the same time I don't think it's necessary in this instance, so long as you know how the usb device communicates with the computer for a certain function, one can just rewrite that function in a program, basically a single function ignoring the rest.
Quote from: dfort on November 04, 2016, 03:59:35 PM
Great quote.
I try... I am camera confucious.
Quote from: dfort on November 04, 2016, 03:59:35 PM
Creating picture styles that can be loaded on the camera without using Canon's EOS Utility is something else that would be interesting. This should allow switching out the 3 picture styles loaded into memory with others that are saved on the card. Why more than 3 custom picture styles? It might be absolutely useless but good for the theory of it!
In this case it would be absolutely useful! But bad in the theory of it :P
(There's been plenty of these "more than 3 picture styles" posts in the past, but I think the wrong question was always being asked "can we add more slots for picture styles").
Yeah, basically one wouldn't store a pf2 or pf3 on the camera and try to translate it in the camera, although a module could probably be made for that (absolutely useless to use, but good for the theory).
One would store the decoded data in a simple file and have it write to the area of the camera from the card. This would require the picture styles undergoing a translation (I'm sure a program can be made for that).
HOWEVER! Here's the nub. Back when I was exploring this in 2013 (none of the images of the comparison of curves of picture styles works anymore http://www.magiclantern.fm/forum/index.php?topic=4895.0 ) In discussion with someone (alex or geggo (maybe I'm just naming names here)) I was told that the registers for the picture styles are unsafe to write to, they are ok to read from though.
Ultimately, going back to writing a picture style from the camera, and the whole linear curve, if that's possible, a module could be made in the camera where one could customise a curve, how much gamma and what not, and write it straight in.
Also in that video, "the camera drops my connection if I don't ping it all the time" I assume that's all the small data between camera and computer.
Did I kill this conversation? Has anyone had any further progress?
Quote from: ItsMeLenny on November 15, 2016, 10:03:39 AM
Did I kill this conversation?
No, not at all. I've been spread out a little thin lately trying to figure out focus pixel map files for a new crop_rec video mode for the EOSM, working on some lens information puzzles and helping out users that are having trouble compiling. Add to that it is the start of awards season so there are lots of screenings and I love watching movies. Oh yeah, and there's that work thing going on too.
It looks like we were onto the same idea about copying pictures style from memory and writing them back into memory. It seems possible, every time you boot up a fresh version of Magic Lantern it writes log files onto the card that we're told to save in case anything goes wrong. These log files, named ROM0.BIN and ROM1.BIN, are the camera's firmware saved at the state things were when Magic Lantern first loaded up. Apparently there's a way to restore this data to the camera if something goes horribly wrong. If that's so then there might be a way to restore just a segment of that data. It should be possible to build up a library of precompiled picture styles--but:
Quote from: ItsMeLenny on November 05, 2016, 12:09:20 AM
In discussion with someone (alex or geggo (maybe I'm just naming names here)) I was told that the registers for the picture styles are unsafe to write to, they are ok to read from though.
That's a bummer, wonder what the consequences are.
Another way to load up precompiled picture styles is through the USB port and it looks like that should also be possible. I've got to take another look at that video on Paparazzi Over IP (https://youtu.be/u7RjJNLnWF8).
In the meantime more and more "log" picture styles are popping up, especially for cameras that have yet to get the ML treatment like the 5D4, 7D2 and this one for the 80D:
Quote from: LesterL on November 04, 2016, 03:07:41 PM
I own an 80D and have done some tests with low-iso dynamic range. I made a picture profile that expands the dynamic range in video and I've been very impressed by the clean results in the shadows. It looks very similar to the 13-stops that come off of the BMPCC. I think this camera would be amazing in RAW video, as the stills captured with this camera can be pushed a lot and still look clean. Plenty of youtube videos showing this.
I'm open for helping with the testing of a ML firmware. Also, if anyone would like the Custom Picture profile I created, I'd be glad to share it for use in the meantime. Just PM me.
Thanks to everyone working on this.
A few of us have been checking out these home cooked picture styles and comparing them with CineStyle. It seems to me that Technicolor's CineStyle is still the gold standard of log picture styles when recording H.264 video on both the old and the newest Canon DSLR's.
What's the point of all this if we already have something that hasn't been improved upon? Because we're curious and there's a lot more to picture styles than I first realized. For example when shooting a feature project is is common for the director or photography and the digital image technician to come up with "looks" that are applied to the raw or log footage when reviewing takes on the set for the director. These "looks" are burned into the files that are sent to editorial and when the raw or log files are used in color grading the colorist can pull up these "looks" as a reference. There are tools like ASC CDL (American Society of Cinematographers Color Decision List) and Academy Color Encoding Specification (ACES) that sets standards over how these "looks" can be assigned to the footage. I'm not very familiar exactly how these systems work but it seems to me that custom picture styles can serve a similar role in a Magic Lantern raw video workflow--review footage on set with the look (picture style) applied and the picture style can travel through the post process as metadata all the way to color grading.
So I spent a long time trying to find the quote of it being bad to write to, going back through all my past posts, here it is: http://www.magiclantern.fm/forum/index.php?topic=3896.msg21124#msg21124
From a user named "Francis": Not possible. Picture styles are stored in a part of memory that is very dangerous to write to.
One of the first things I noticed when I got my camera was the awful canon curve, which between that and the rolling shutter actually made me regret buying it to an extent, I started shooting neutral straight away, but it still doesn't get rid of their "film style" curve.
There is a huge amount of user made picture styles, however I believe the majority of them are made in the picture style editor, as a result they don't interest me so much.
Noteable picture styles are probably, Video-x from canon, technicolors cinestyle, marvels cinestyle, and flaat (which has multiple versions of dynamic range).
Another post of mine http://www.magiclantern.fm/forum/index.php?topic=10008 "Where is that long list of registers?" All of them are linked, probably most importantly though this one: http://magiclantern.wikia.com/wiki/Register_Map/Brute_Force/C0F0Fxxx -- "C0F0Fxxx - picture style parameters maybe"
Rest of the links thanks to audionut at the time:
http://magiclantern.wikia.com/wiki/ADTG
http://magiclantern.wikia.com/wiki/Register_Map
http://magiclantern.wikia.com/wiki/Memory_map
http://magiclantern.wikia.com/wiki/Register_Map/Brute_Force
http://magiclantern.wikia.com/wiki/Register_Map/550D
Also: http://www.magiclantern.fm/forum/index.php?topic=4830 "Contrast and picture styles, the real scoop"
Hey ItsMeLenny,
Just curious, is there a reason Visiontech isn't mentioned as a noteable picture style? I tried C-LOG in low light and it was really noisy and had banding. But Visiontech seems to work well in any situation I throw at it. I feel more people should try this picture style since it retains dynamic range without trying to make a log image in an 8 bit compressed color space. I realize Visionlog is a no go compared to Cinelog-C, but Visiontech seems to hold up well when color grading.
- Paul Harwood
yes I currently use cinestyle, visiontech, and flaat
Quote from: PaulHarwood856 on November 17, 2016, 08:05:45 PM
Hey ItsMeLenny,
Just curious, is there a reason Visiontech isn't mentioned as a noteable picture style?
Because I've never heard of it and never used it. I haven't looked at picture styles for a long time, I shoot photos on neutral and video on videoX.
I'm not any kind of superior in terns of picture styles so anybody is welcome to mention their own notable picture styles. However probably not in this topic unless its mentioned along with what the topic is about.
I'll have a brief look at it when I get the time. I was just mentioning a few that attempt to get the most out of an image without being some kind of gimmick, as in one that changes colour or desaturates things or what not.
Hey ItsMeLenny,
Ok I understand. Thanks for your response.
- Paul Harwood
Part of what we're trying to find out in this discussion is which log picture style works the best with these cameras. That's rather subjective but if we just look at the math it appears that CineStyle is still the best, at least technically. It appears to have a 10-bit log curve. (http://www.magiclantern.fm/forum/index.php?topic=16299.msg172770#msg172770) We can argue over which picture style gives the better "look" or is easier to grade and never come to an agreement.
@ItsMeLenny -- thanks so much for those links. We're building up quite a reference library on this topic.
Hey dfort,
Ok I understand. I just think people don't realize Visiontech's advantages, and whenever I film log in low light there's noise and it's difficult to grade. Is it better to just use a log picture style in good lighting? I apologize if I came off strong I just was surprised Visiontech hasn't been discussed, like if there was something people know that I don't about it. I didn't mean to deviate the discussion, sorry.
- Paul Harwood
Quote from: PaulHarwood856 on November 18, 2016, 02:06:38 AM
I didn't mean to deviate the discussion, sorry.
Not at all and no need to apologize.
If you have a picture style that you like it would be great if you could open it up and see how it works and what makes it different from another picture style that doesn't quite do it for you. Unfortunately the people making these custom picture styles seem to be hiding something. We're not hiding anything here.
Quote from: PaulHarwood856 on November 18, 2016, 02:06:38 AM
I apologize if I came off strong I just was surprised Visiontech hasn't been discussed, like if there was something people know that I don't about it. I didn't mean to deviate the discussion, sorry.
I don't read emotion on peoples posts because it's impossible to tell, I just read what is said, that is also how my posts should be read, emotionless because there is noway to convey emotion unless you want to put it in brackets after every statement (thought invoking :P ). Also I was typing on my phone so it may have read more abrupt than usual.
Quote from: dfort on November 18, 2016, 02:48:34 AM
If you have a picture style that you like it would be great if you could open it up and see how it works and what makes it different from another picture style that doesn't quite do it for you. Unfortunately the people making these custom picture styles seem to be hiding something. We're not hiding anything here.
This x 3 for each statement.
Quote from: dfort on November 17, 2016, 11:36:30 PM
@ItsMeLenny -- thanks so much for those links. We're building up quite a reference library on this topic.
Yeah there's other posts from years ago that have already tried to explore this (I think I started at least 3). But I think those are all the relevant links.
I ultimately think a module could be made that can adjust the various curves inside the picture styles and write it to the camera from the camera, so long as those registers are safe to write to.
Back to the canon film curve that I'm always complaining about, whether that is somewhere in the registers as well and being mistaken as the register that is unsafe to write to, as the registers are unknown and partly undocumented for picture styles.
It'd be great to somehow pass half of the processing rather than equalizing it out, say if that film curve is somewhere in the pipeline if that step can just be jumped.
Now, this is another theory which I have not tested, but! I am unsure if the camera utility that uploads the picture styles detects what camera it is and writes it accordingly, if it applies extra modifications between cameras for the different sensors between cameras. So the picture styles are universal in the picture style format, but if you were to compare them on the cameras if, say for instance, one of the colour channels has a slight different curve applied to it to accommodate for that sensor. If that all makes sense.
Hey dfort,
Ok great. Thanks for letting me know. I might need to dive into this Visiontech picture style at some point. Could you let me know which tool to start with? And what to post first?
Hey ItsMeLenny,
Ok good point. Thanks for that.
- Paul Harwood
Could you shoot a comparison between cinestyle and visiontech?
Hey Danne,
Sure! A well lit shot and a shot in low light? Is there a chart I should have in the shot?
- Paul Harwood
you can also test this one if you want
https://marvelsfilm.wordpress.com/marvels-cine-canon/
Something to check shadows. Just the same settings, same motif, just for fun.
Hey Danne,
Alright awesome! I have a 7D and T3i which are both the same sensor, so I'll try up to six picture profiles. Visiontech, Cinestyle, Marvels, EOS HD CLOG, and Flaat 10 and 11. If there are any others please let me know. I'll get this done when I can.
- Paul Harwood
http://web.canon.jp/imaging/picturestyle/file/videocamera.html (http://web.canon.jp/imaging/picturestyle/file/videocamera.html)
perhaps the canon video x? I have never used it but im curious how it compares
Honestly I'm not the one to call this, but I feel as if this is getting slightly off topic, maybe a new thread should be started for comparisons of picture styles.
On top of that, I guess what's valuable, at least in this thread, is how the picture styles are made, not how they compare visually. It would be valuable if there were two very visually similar picture styles but one did something worse (had bad banding or something) and that was a result of how the picture style is put together under the hood.
In terms of comparing picture styles, I'm just assuming here that you're going to take a photo in each picture style. Even if all settings are kept the same I don't think this is the best or easiest way to go about it. All you'd need to do is take one photo in RAW and then put it into canons DPP software (digital photo professional) and apply the different picture styles.
In saying that it's still a visual comparison (which I guess what the photo looks like at the end of the day is what counts). A better process would be to generate a CR2 image that can fake itself as being taken with a canon that has linear gradients of 0% to 100% black to white and red green blue channels, feed this into DPP and see what the curves actually are, which would also allow finding the canon film curve, also the difference in colours between neutral faithful etc. In addition to faking these raw images to have them look as though they come from different canon models, which would allow to see if the curves do change between camera sensors (assuming this is all in the software, which I'm going to assume it is, if it exists in the first place).
I will actually make one of these CR2 images, hopefully this weekend, which it is the weekend already. Anybody who would like a copy it will only cost $12.99 :P
didn't follow the whole thread.
but if you want to know where the custom picture styles are located, i'd say they are stored in the properties.
you can dump the properties using PropertyEditor.exe which i just updated
see http://www.magiclantern.fm/forum/index.php?topic=4729.msg27865#msg27865
it will operate on a ROM-dump and save all properties as viewable XML.
some models have a CUST property region, like 6D or 60D. would expect it there.
Quote from: g3gg0 on November 20, 2016, 12:07:06 PM
you can dump the properties using PropertyEditor.exe which i just updated
Going to have to dust off the old Windows Laptop to try it out. Is the code available so I could compile a Mac binary?
Quote from: g3gg0 on November 20, 2016, 12:07:06 PM
it will operate on a ROM-dump and save all properties as viewable XML.
some models have a CUST property region, like 6D or 60D. would expect it there.
Just tried on my 5D2 Rom0.BIN got a lot of "<Property Id=" string but I don't know what they are and not sure where to look to figure out what there mean.
But I got this string with text I under stand ,
Quote
</Property>
<Property Id="02000001" BaseAddress="00310024" Length="00000010">
<String>2.1.2_A1(00)____</String>
<Data>322E312E320041312830302900000000</Data>
</Property>
<Property Id="02000005" BaseAddress="0031003C" Length="00000010">
<String>6.9.8 A1(00)____</String>
<Data>362E392E382041312830302900000000</Data>
</Property>
<Property Id="02000002" BaseAddress="00310054" Length="00000004">
<String>____</String>
<Data>00000000</Data>
</Property>
<Property Id="02000003" BaseAddress="00310060" Length="00000004">
<String>____</String>
<Data>FFFFFFFF</Data>
</Property>
<Property Id="02000004" BaseAddress="0031006C" Length="00000020">
<String>David Miazga_,__________________</String>
<Data>4461766964204D69617A6761002C000000000000000000000000000000000000</Data>
</Property>
In red , the first text is my firmware "2.1.2" , next not sure "6.9.8 A1" and then my name .
this repeat itself thought out the xml file from the Rom0.BIN file , I could not get the programs to dump the Rom1.BIN file just
gave a error "Autodetect property offset failed"
Below is the my XML file , if anyone wishes to have a look and maybe can understand it more.
https://www.dropbox.com/s/cn4fiog894b8hd1/ROM0.BIN_00310000.propxml?dl=0
On my search for info on Property Id string definitions I came upon this old firmware map from 5D2 , I thing it's from 2008 or 2009
below is the link to my Dropbox , just use a text editor to view the *.map
https://www.dropbox.com/s/cwwu89uldl3loql/5D21070a.map?dl=0
what I found very interesting about where Picture Style is in the blocks
0001:001B06E0 GUI_SetPictureStyleData
0001:001B082C GUI_GetPictureStyleData
0001:001B094C GUI_SetPictureStyleOfUserSetting
0001:001B09B4 GUI_GetPictureStyleOfUserSetting
0001:001B0A00 GUI_GetPsUserSetting_PCsetValid
0001:001B0A4C GUI_GetPsUserSetting_PCsetParam
0001:001B0BCC GUI_GetPsUserSetting_PCsetValid_0
Maybe this is the Luma Curve ?
0001:000898F8 FA_EnableLinerEShutCurve
0001:00089908 FA_DisableLinerEShutCurve
Something to do when connected thought USB cable ?
0001:002990A0 StopLiveViewPictureStyleApp
0001:00299508 StartLiveViewPictureStyleApp
0001:002BE868 StartShootOlcPictureStyleApp
0001:002BEA08 UpdateShootOlcPictureStyleApp
0001:0034030C StopMnPictureStyleDetailApp
0001:003407BC StartMnPictureStyleDetailApp
0001:00340F1C StopMnPictureUserDetailApp
0001:003413C0 StartMnPictureUserDetail
Hope this helps :)
Quote from: g3gg0 on November 20, 2016, 12:07:06 PM
you can dump the properties using PropertyEditor.exe which i just updated
see http://www.magiclantern.fm/forum/index.php?topic=4729.msg27865#msg27865
it will operate on a ROM-dump and save all properties as viewable XML.
some models have a CUST property region, like 6D or 60D. would expect it there.
Thanks for making progress on this project @g3gg0!
Quote from: reddeercity on November 21, 2016, 08:38:38 AM
Maybe this is the Luma Curve ?
0001:000898F8 FA_EnableLinerEShutCurve
0001:00089908 FA_DisableLinerEShutCurve
Holy
SHIT @reddeercity that's some serious digging you have done there...
This definitely smells good. Thanks for sharing these!!! :)
Hey @reddeercity - this is cool:
0001:002990A0 StopLiveViewPictureStyleApp
0001:00299508 StartLiveViewPictureStyleApp
0001:002BE868 StartShootOlcPictureStyleApp
0001:002BEA08 UpdateShootOlcPictureStyleApp
0001:0034030C StopMnPictureStyleDetailApp
0001:003407BC StartMnPictureStyleDetailApp
0001:00340F1C StopMnPictureUserDetailApp
0001:003413C0 StartMnPictureUserDetail
Wonder if it would be possible to display one picture style on the live view while recording another picture style. This would be like an external monitor that can use luts so that it can preview what a log image will look like after it has a look applied to it.
but dont confuse memory addresses with property IDs
properties are "containers" that are managed by a property manager.
routines can allocate a property and can place any data in it.
the property manager takes care of storing it into flash (which is not a too simple task if you want to make it robust)
that stuff you see in the .propxml is a human readable representation of the content in the flash.
all properties found in the container you specified on the command line are dumped into that .propxml.
there are a few property containers. check the post that i linked in the post before.
@dfort. this is what i already kind of do with raw recording, I put the picture style to high sharpness and good contrast to get a nice image while recording in the liveview. and when i open the raw the picture style is not applied. It could be nice if the picture style that is used for recodring is stored in the metadata of the mlv, then during raw conversion the same picture style lut can be applied. in this way we can have the look consistent between recoding and post.
@g3gg0 thanks for the help , I'll do more reading in fact being reading a lot of old reverse engineering thread from 2013 etc... being given me a few ideas for this thread .
@DeafEyeJedi not really just got lucky to find that .map file , but that's what you get from reading obsolete threads :D
Quote from: dfort on November 21, 2016, 07:32:33 PM
Hey @reddeercity - this is cool:
Wonder if it would be possible to display one picture style on the live view while recording another picture style.
This would be like an external monitor that can use luts so that it can preview what a log image will look like after it has a look applied to it.
Thanks, It may be possible at some point , not by me thou .
After reading thought many old threads about h264 and 422 mjpeg reverse engineering ,
It seems there is a lut for HDMI and or Liveview made reference to in a some documentation I read not too sure if 2 different styles/LUTS can be applied
line 55 // cached LUTs for BM2LV-like macros line 370 vram_update_luts(); those two lines came from bitbucket.org/hudson/magic-lantern/src/tip/src/vram.c (https://bitbucket.org/hudson/magic-lantern/src/tip/src/vram.c?fileviewer=file-view-default#vram.c-55) lot of info about hdmi , this is way be on my ability right now , but I'm learning :D
Ok back on topic
Quote from: Lars Steenhoff on November 21, 2016, 08:22:14 PM
It could be nice if the picture style that is used for recording is stored in the metadata of the mlv,
then during raw conversion the same picture style lut can be applied. in this way we can have the look consistent between recoding and post.
Will if the info that I'm working on lead me to the source data of picture style then you will have a exact copy of your PS in a LUT or ICC
which picture style are (ICC Profiles), then that can be converted to DCP for ACR etc.... .
hi
There i find this blog, he make a file CINEMAEX.cpf
http://ntown.at/2013/11/20/canon-cinema-custom-picture-styles/ (http://ntown.at/2013/11/20/canon-cinema-custom-picture-styles/)
http://ntown.at/2015/01/08/canon-c100-cpp-noise-test/ (http://ntown.at/2015/01/08/canon-c100-cpp-noise-test/)
Very low noise level :)
Some one think it's possible to adapt (Reverse Engineering ) this picture style to .pf2 for the canon dsrl ? ;D
ps: sorry for my english :-[
kyusu
@kyusu this thread is for Reverse Engineering picture styles (pf2 , pf3) for eos dslr's !
Apples & Oranges one fruit here only , if you want to know more about c100's profile start reverse engineering the cpf file
and let us know.
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#PictureStyle
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#PSInfo
Looks like we have some "0x" Values in the tags here , may be useful
Quote from: reddeercity on January 01, 2017, 07:02:48 AM
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#PictureStyle
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#PSInfo
Looks like we have some "0x" Values in the tags here , may be useful
Oh hell yeah @reddeercity!
Quote from: reddeercity on January 01, 2017, 07:02:48 AM
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#PictureStyle
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#PSInfo
Looks like we have some "0x" Values in the tags here , may be useful
Just came across this a couple weeks back.
This is the EXIF data in the CR2 files, however the "Canon PictureStyle Values" does make me wonder if those extra ones are available in cameras that don't have them.
I'm still working on my little project to do with this thread which I have to finish before I can get back into looking at the stuff in the camera itself.
Quote from: ItsMeLenny on January 01, 2017, 01:14:45 PM
I'm still working on my little project to do with this thread...
Provocative.
Quote from: kyusu on December 19, 2016, 11:33:14 AM
...make a file CINEMAEX.cpf
...
Some one think it's possible to adapt (Reverse Engineering ) this picture style to .pf2 for the canon dsrl ? ;D
.
Although as @reddeercity says it is a different "fruit" look at the way custom picture styles are installed on the Canon cinema cameras:
QuoteUSAGE:
Just copy the files on a SD Card in the directory structure PRIVATE/C_PICT
(create it if it's not on the sdcard)
import them via Custom Picture Menu:
Transfer File -> Load from SD -> (A or B)
or just set the CP directly from the SD-Card.
Wouldn't that be cool? For still cameras you get three custom picture styles maximum and you need to use EOS Utility to load them.
Thanks for the stunning progress investigating the file structure of PF2 and PF3! So amazing!
In the excel sheet I recognized a block called "3x20 Matrix sRGB [long]". What can you do with this block? Is it a transformation of RGB-pixel values from one to another value? If so, can someone give me a short example how the transformation is being performed?
@agentirons
If you have some time you can update the specification of PF2.
In the "10 18 00 04 RGB Gamma [long]" block you can enter 12 points for each RGB-channel. The channel order is definitely R, G, B. I have tested this and reproduced curves from Photoshop. Works like charme. The values are altered independently of each channel.
Quote from: michael.huber.2932 on January 24, 2017, 01:19:11 PM
@agentirons
If you have some time you can update the specification of PF2.
In the "10 18 00 04 RGB Gamma [long]" block you can enter 12 points for each RGB-channel. The channel order is definitely R, G, B. I have tested this and reproduced curves from Photoshop. Works like charme. The values are altered independently of each channel.
Sure thing, sorry I didn't copy that from the Cinestyle tab into the main tab. It's the same for L*a*b* Gamma too.
For everyone's reference, the way it works is that for each channel, the first set of 4 bytes (ie 00 00 00 02) indicates how many points there are in the curve, and then each pair of 8 bytes following corresponds to an input and output for that point, used to calculate transformations from incoming data. So for instance if there are only 2 curve points, then the first set of I/O corresponds to black level and the next set corresponds to white level, and the remaining values in that channel are left blank. If there are three points, then first is black, second is your custom point, third is white, and the rest are blank. Strangely, the max seems to be twelve even though there's enough space in each data block for 15 points. So, each channel within the data block will always end with at least 24 bytes of zeros.
I think the biggest thing to figure out right now is how much manipulation is possible using the picture styles, and whether the base picture style (ie Landscape) can be negated somehow so that we can effectively apply curves directly to the raw data.
My project that I've been poking away at is a Javascript-based style editor, using Electron as a cross-platform app container. You can drop in a file, see all the data as hex values with tag names as you see in the spreadsheet, and edit individual values for saving back out. At this point I have to get the edit and save parts working, then I'll post a link. With something like this it will at least be much easier for people to see what they're editing while we test custom values out.
Unfortunately the pf2-format only allows to define 3 channels. But often you want to add another RGB-curve that affects all channels at the same time. For example you want to add some contrast with the help of curves. In Photoshop you usually do this with a s-curve in the "RGB-channel". There are 2 solutions here:
1) Replicate this "RGB-channel" to each channel separately. So 3x the same curve in the red-, green- and blue-channel. This is exactly what the "RGB-channel" does.
2) But sometimes the red-channel has its own curve. Or maybe the green- or blue-channel. So in this case you have to add two curves to "simulate" the non existing rgb-curve in pf2-format. All you have to do is to overlap the control-points of the separate channel and the rgb-channel.
The PF3-format has an adavantage here. There you can define a additional RGB-curve that you can use for this purpose. Although I can create "valid files" that are accepted by EOS Utility and the camera, I can't see the effect of the applied curve (5D Mark III) in the live view/resulting image. If I use the exact same file in Canon Digital Professionel or my other cam EOS 5Ds (s!) it works. This is very strange!
Note: I made a lot of tests and sometimes the curves in Photoshop does not exactly look like the curves in PSE - even if use the exact control points. So I think Canon uses a different interpolation method than Adobe. To fit the curve I sometimes have to add another control points for the exact desired replication of the curves.
Does anyone has some information about the PF3-format and can tell me why these files are so huge compared to a PF2-file?
Quote from: agentirons on February 03, 2017, 06:56:44 PM
I think the biggest thing to figure out right now is how much manipulation is possible using the picture styles, and whether the base picture style (ie Landscape) can be negated somehow so that we can effectively apply curves directly to the raw data.
I though the base picture style is neutral.
But also, is this still all done on top of canons film curve, or is that actually hidden away in the picture style?
Quote from: michael.huber.2932 on February 04, 2017, 06:31:49 PM
The PF3-format has an adavantage here. There you can define a additional RGB-curve that you can use for this purpose. Although I can create "valid files" that are accepted by EOS Utility and the camera, I can't see the effect of the applied curve (5D Mark III) in the live view/resulting image. If I use the exact same file in Canon Digital Professionel or my other cam EOS 5Ds (s!) it works. This is very strange!
That is strange - did you make those files by hex edit or through the Canon Picture Style Editor software?
Quote from: michael.huber.2932 on February 04, 2017, 06:31:49 PM
Does anyone has some information about the PF3-format and can tell me why these files are so huge compared to a PF2-file?
I haven't dug all the way in but from what I've seen it's just down to the existence of many more tags available in the pf3 format. For instance, in PSE, the extra Sharpness settings, Tone curve (RGB), and the entire Six-color axes tab are not saved in pf2 format. I believe the minimum size for pf3 comes in around 450kb. For instance, I saved a custom pf3 out of PSE with default settings, and the resulting file had 3D LUTs inside for both sRGB and AdobeRGB that each took up 216kb of data.
Quote from: ItsMeLenny on February 05, 2017, 12:34:31 AM
I though the base picture style is neutral.
But also, is this still all done on top of canons film curve, or is that actually hidden away in the picture style?
The Base Picture Style is determined by the 0x1009 tag ("Base Picture Style ID") in the file, with at least nine known styles available. If you open up the PSE program it's the first dropdown option in the Basic tab. I think @andy600 would know better about whether the Canon film curve is involved, as I'm sure he would have dealt with that in his Cinelog work.
Quote from: agentirons on February 06, 2017, 06:22:41 PM
That is strange - did you make those files by hex edit or through the Canon Picture Style Editor software?
I saved a basic file with no adjustments from Canon Picture Style Editor and made my changes in a hex-editor. It's very strange that my pf3-file works in the 5Ds, but not in the 7D Mark I / 5D Mark III.
Quote from: agentirons on February 06, 2017, 06:22:41 PM
I haven't dug all the way in but from what I've seen it's just down to the existence of many more tags available in the pf3 format. For instance, in PSE, the extra Sharpness settings, Tone curve (RGB), and the entire Six-color axes tab are not saved in pf2 format. I believe the minimum size for pf3 comes in around 450kb. For instance, I saved a custom pf3 out of PSE with default settings, and the resulting file had 3D LUTs inside for both sRGB and AdobeRGB that each took up 216kb of data.
Ok, that makes sense, that the pf3-files are much larger than the pf2-files if there are 3d-luts. Do you have more information about the 3d-luts? Do you know the offsets of the two tables and any idea how to use these informations? This would be real cool if we can have customized 3D LUTs out of Photoshop for example. We could nearly reproduce any effect/filter made in Photoshop.
Just a guess: maybe Canon PSE creates a 3D-LUT in the case of the PF3-format out of the seetings made with the Canon Tool and uses this one to make the adjustments in the camera. This would be an explanation why my changes in the PF3-files are ignored in the 7D I and 5D Mark III.
Quote from: michael.huber.2932 on February 07, 2017, 06:01:22 PM
I saved a basic file with no adjustments from Canon Picture Style Editor and made my changes in a hex-editor.
That could be due to the "Finalize" tag, which acts as a checksum. It's possible that some cameras can ignore it. If you change data in a hex editor without modifying the Finalize block, then the checksum is incorrect and PSE will tell you there's something wrong with the file if you try to load it. But, I found that Canon's downloadable styles don't even include Finalize, and if you remove it with a hex editor and load the file into PSE it will both load correctly and when saved PSE will handle generating a correct tag for you.
I think it's easiest to just remove it, but the latest .JAR version of the encoding script posted earlier in this thread includes the correct formula for generating a valid Finalize tag.
As for the 3D LUTs, it's certain to be very powerful but I don't know anything about how to properly edit them yet.
The announcement of Canon Log for the 5D4 (https://www.usa.canon.com/internet/portal/us/home/explore/see-legendary/canon-log) is certainly grabbing a lot of attention and the comments from users range from delight to outrage--mostly outrage that the camera needs to be sent in and it cost $99. It won't be available until July so we'll have to wait a few months to hear from users' experiences with it.
One thing does seem to be clear, this is partially a hardware modification. If you check out log settings on professional video cameras you'll notice that it is more than just a picture style. In order to use log, automatic exposure is not possible and the recommended ISO settings center around a sweet spot called the base or native ISO. In the case of the 5D4 it will be ISO 400.
This will be the second Canon DSLR to receive the log option, the first was the EOS-1D C. Several comments are floating around as to why the EOS-1D X Mark II isn't getting this log upgrade. Perhaps the question should be why Canon doesn't combine those two cameras but it would have to be a merge that doesn't compromise the video capabilities of the 1D C or the still photo capabilities of the 1D X.
Back to Canon Log on the 5D4, will we be able to come up with a picture style that can do Canon Log on the other camera models? Though we have had log picture styles since Technicolor released CineStyle in 2011, are log picture styles the same as the hardware log profiles? Probably not. The EOSHD C-LOG picture style (http://www.eoshd.com/2016/09/now-available-eoshd-picture-profiles-brings-c-log-canon-dslrs-including-1d-x-mark-ii-5d-mark-iv/) was developed by matching the EOS-1D C log setting as closely as possible. It will be interesting comparing this picture style with the real thing when it comes out in July.
The Canon announcement includes reference to a new "View Assist" setting that will show a "normal" looking image on the camera while recording log internally or outputting log video through the HDMI port. That is something that can be approximated with the display settings in Magic Lantern on certain cameras.
(https://c1.staticflickr.com/5/4179/33817249840_ff4b8720d0.jpg)
Note that on the 5D4 the HDMI output is still at best 1920x1080 8-bit 4:2:2 so 4k will still need to be recorded internally with the 8-bit Motion JPEG 4:2:2 at 500Mbit/s codec. Ideally the HDMI output would be 4k 10-bit (or higher) 4:4:4 so you can use an external recorder to capture broadcast ready 4k video.
If you want log media another option is to shoot MLV and apply a log DNG Camera Profile when creating your movie files. @Andy600 is the expert when it comes to that workflow.
I just got back from NAB and saw the 5D Mark IV with Canon Log being demonstrated. Right next to it was a 1 DX Mark II without Canon Log. OK, we knew about that but missing from the camera lineup was the 1 DC which of course was the first to have Canon Log. Will it be discontinued? They were tight lipped about that.
Interesting was that they were still offering to send you information about the 5D Mark III when they scanned your badge so either it hasn't been discontinued yet or they must still have a stack of 5D3's in the warehouse.
Quote from: dfort on April 28, 2017, 04:13:06 AM
I just got back from NAB and saw the 5D Mark IV with Canon Log being demonstrated. Right next to it was a 1 DX Mark II without Canon Log. OK, we knew about that but missing from the camera lineup was the 1 DC which of course was the first to have Canon Log. Will it be discontinued? They were tight lipped about that.
Uh-oh... smells like a hint to me. Shall we bait on it? :-X
Quote from: dfort on April 28, 2017, 04:13:06 AM
Interesting was that they were still offering to send you information about the 5D Mark III when they scanned your badge so either it hasn't been discontinued yet or they must still have a stack of 5D3's in the warehouse.
Heck by the end of this year before the Holidays kicks in we might as well just grab another brand new 5D3 or two just to have as back-up's down the road. These beasts are still going to be kicking ass come 2020 and beyond. Just not sure whether I should invest into a 5D4 just yet. Maybe a used one by the time the Mark V comes out.
Just thought I'd post my obligatory shot of Canon Log at NAB.
(https://c1.staticflickr.com/5/4178/34323151545_3fcd5c6033_z.jpg)
Although Magic Lantern wasn't represented at NAB I did find a booth where an exhibitor had it installed on one of their demo cameras.
(https://c1.staticflickr.com/5/4167/33513002433_c2d215e293_z.jpg)
We now return to our regularly scheduled topic on reverse engineering picture styles.
Is that an early Canon from the Paleolithic era?
;)
More like from the Flintstone's era!
Don't you get the joke? Hint--that's a brick.
Quote from: dfort on April 28, 2017, 08:35:58 PM
Don't you get the joke? Hint--that's a brick.
Ah, you got me. Just shat a brick! :P
Just wait for a1ex to set up a fir for that brick guys.
Can you elaborate on the brick? :)
So what are the hardware changes that have to be made?
I guess the thing as we still dont know where the canon curve lies as the cinestyle picture style reversed it within the picture style.
Quote from: agentirons on February 03, 2017, 06:56:44 PM
Sure thing, sorry I didn't copy that from the Cinestyle tab into the main tab. It's the same for L*a*b* Gamma too.
For everyone's reference, the way it works is that for each channel, the first set of 4 bytes (ie 00 00 00 02) indicates how many points there are in the curve, and then each pair of 8 bytes following corresponds to an input and output for that point, used to calculate transformations from incoming data. So for instance if there are only 2 curve points, then the first set of I/O corresponds to black level and the next set corresponds to white level, and the remaining values in that channel are left blank. If there are three points, then first is black, second is your custom point, third is white, and the rest are blank. Strangely, the max seems to be twelve even though there's enough space in each data block for 15 points. So, each channel within the data block will always end with at least 24 bytes of zeros.
Resurrection from the dead. I've been digging into this for two days straight....
I'm unable to have success with this....but I've been trying it with .pf3 files.
Does anybody have a .pf2/3 file with individually adjusted R, G, & B curves? I'd like to open it up and see what I'm doing wrong.
Thanks!
you could make one pretty quickly in the canon picture style editor.
i'd do it and send a copy but I'm on linux so it's a bit of a process.
Since I started developing my RAW photos with an awesome Canon 450D derived tone curve the prospect of bringing OOC previews closer to my final curve sounds too sweet to pass. My hacking skills are below the pay grade of most contributors in this thread, but I'm ready to do some testing so let me know if this sounds feasible. Perhaps this will be useful for for multi-camera setups too. Roughly what I want to do:
- Start with a general purpose PS like Standard, Neutral or Faithful.
- Using dcptool or dcp2json I extract my target curve from "Canon EOS 450D Camera Standard.dcp" stored in decimal form. (A large number of unique tone curves can be sourced from DCP files by Adobe, free to download.)
- Scale the target curve by 50% to in order to apply it equally between L and RGB blocks. This should provide a good balance between saturation and hue accuracy.
- The linearizing reverse RGB S-curve(discovered in Cinestyle) is converted to decimal and added to the curve made in the previous step.
- Curves created in step 3 & 4 are interpolated and converted into hex and finally written into L and RGB blocks respectively.
Not following exactly what you are trying to do but hey, we're here to learn.
Did you know that the custom profiles are accessible from a firmware dump? Not exactly sure how to do it but if you search a disassembly on a camera that had Cinestyle installed you'll see something like this:
"CineStyle":
fff01b44: 656e6943 strbvs r6, [lr, #-2371]! ; 0x943
fff01b48: 6c797453 cfldrdvs mvd7, [r9], #-332 ; 0xfffffeb4
fff01b4c: 00000065 andeq r0, r0, r5, rrx
...
Quote from: dfort on October 16, 2017, 01:14:54 AM
Not following exactly what you are trying to do but hey, we're here to learn.
Basically, since we know how to create a linear tone picture style(see RGB_Gamma_curve.pf3 http://www.magiclantern.fm/forum/index.php?topic=16299.msg172770#msg172770 (http://www.magiclantern.fm/forum/index.php?topic=16299.msg172770#msg172770)) I'm playing with the idea of creating picture styles with my custom tone curve for photographic purposes. You might think this would be negated by shooting raw. While I agree with that, this would improve visual feedback while shooting. Here's the tone curve I want to try:
(https://i.imgur.com/CvhsaSo.png)
https://www.dropbox.com/s/3t8sq71bm3ku200/450D%20TC_Adobe.CSV?dl=0
Quote from: dfort on October 16, 2017, 01:14:54 AM
Did you know that the custom profiles are accessible from a firmware dump? Not exactly sure how to do it but if you search a disassembly on a camera that had Cinestyle installed you'll see something like this:
"CineStyle":
fff01b44: 656e6943 strbvs r6, [lr, #-2371]! ; 0x943
fff01b48: 6c797453 cfldrdvs mvd7, [r9], #-332 ; 0xfffffeb4
fff01b4c: 00000065 andeq r0, r0, r5, rrx
...
Cool find. Do you think the values correspond to sharpness contrast etc settings or do they provide further insight on the picturestyle format?
EDIT:It's shouldn't be understated that separate RGB and L curve adjustment that we have now is really powerful. To illustrate this I developed a raw photo with the tone curve applied in RGB space, L space and 50% RGB and L.
Tone curve in RGB / Tone curve in L / Tone curve in 50% RGB, 50% L
(https://i.imgur.com/Utt2BHwl.jpg)(https://i.imgur.com/hWX1doSl.jpg)(https://i.imgur.com/YGY802cl.jpg)
RGB Reminds me of Neutral and Standard. Gray-ish skintones, overall yellow-leaning. / L Reminds me of Faithful. Pastell-leaning hues. Washed out skintones. / 50-50 RGB-L Has skintones with a natural definition, smoother tonal transitions and truer hues.
Would it be possible to convert a LUT from for example the ektar chrome to a canon picture style?
https://www.magiclantern.fm/forum/index.php?topic=19338.0
You can mimic luts but it will take effect very late in the chain. Also, when using picture style editor you only have twelwe gamma/tone curve points to work with.
There are a few initiatives that digged out a whole lot of infomation about how it all works but one reason it all stopped is probably we are too far from sensor level to make good use of it all. Please correct me if I'm wrong here.
If its only to display on the back of the screen and for the h264 proxy to look a certain way the shot is intended to look, would it really matter if its too late in the chain?
With too late I assume you mean after some conversions already happenend to the signal and therefore losing dynamic range.
A lut(3D) has so much information so to manually recreate it with picture style editor will at its best be a "similarly" looking look. If that´s ok then go for it. Converting the lut into a .pf3 with some existing tool. Havn´t heard of it
Quote from: Danne on October 16, 2017, 01:20:15 PM
...we are too far from sensor level to make good use of it all...
I don't know exactly why the 5D4 needs to be sent in to a Canon service center to get the Log Feature Upgrade (https://www.usa.canon.com/internet/portal/us/home/about/newsroom/press-releases/press-release-details/2017/20170420-C-Log/20170420-C-Log) installed but lately I've been fiddling around with some higher end cine cameras and there are some quirks to working in log that makes me think that it needs to be done in hardware. Some cameras lock you into a "Native ISO" others recommend not to go
below a certain ISO when using log and the new Panasonic EVA1 has Dual Native ISO which gives you a choice of 800 or 2500 but nothing else.
The following only applies to h264 4:2:0 & uncompressed 8bit HDMI 4:2:2 Color Space not Raw Video .
Yes , when recording in Log Space there needs to be a certain parameters for the camera to capture the maximum dynamic range in it's native color space . That kind of what A.C.E.S is about , that's a whole different thread .
Lets take Technicolor CineStyle Profile for example , as it was originally developed for 5D mark ii but can be used by all canon dslr's , the optimal image quality (http://www.technicolor.com/en/solutions-services/cinestyle-download/cinestyle-faq#what-are-the-recommended-canon-eos-5d-mark-ii-camera-settings-to-obtain-optimal-image-quality) to record Log is
•Sharpness: 0
•Contrast: -4
•Saturation: -2
•Color Tone: 0
•ISO: a multiple of 160
The only 2 things I would add to this is
Always record @
160 ISO (if possible but not over 320 ISO) and drop Saturation to -4 .
What does cinestyle profile do to the image (http://www.technicolor.com/en/solutions-services/cinestyle-download/cinestyle-faq#what-does-cinestyle-profile-do-to-the-image)
QuoteWhen the Technicolor CineStyle Profile is selected in the camera, video images are recorded in log space.
.... though the image will appear flat and de-saturated, there is actually more detail retained in the shadows and mid-tones .
So by apply a modified Picture Style (Not Technicolor CineStyle as this is the Low Level chain of image processing) with
LOG Type tone curve or thought software after the fact you are really not recording and or saving in LOG and thus are only imitating LOG space which is not maximuming dynamic range to it's fullest extent .
Once you study the parameters required to record true LOG Space with Technicolor CineStyle it's clear the camera's picture style had been calibrated to the Sensor at a low level along the lines of the IDT (Input Device Transform)is in A.C.E.S (https://en.wikipedia.org/wiki/Academy_Color_Encoding_System)
If anyone is interested in A.C.E.S here a link http://acescentral.com/ I've been a member since 2014 , it's free to join and even if you don't use A.C.E.S , there is lot's of great info on color grading and color space's etc. .... I get a new letter once a mouth with updates & or discussions .
What is Academy ACES (http://acescentral.com/t/what-is-academy-aces/8)
Quote
WHAT IS ACES?
The Academy Color Encoding System (ACES) is becoming the industry standard for managing color throughout the life cycle of a motion picture or television production. From image capture through editing, VFX, mastering, public presentation, archiving and future remastering, ACES ensures a consistent color experience that preserves the filmmaker's creative vision. In addition to the creative benefits, ACES addresses and solves a number of significant production, post-production and archiving problems that have arisen with the increasing variety of digital cameras and formats in use, as well as the surge in the number of productions that rely on worldwide collaboration using shared digital image files.
ACES is a free, open, device-independent color management and image interchange system that can be applied to almost any current or future workflow. It was developed by hundreds of the industry's top scientists, engineers and end users, working together under the auspices of the Academy of Motion Picture Arts and Sciences.
ACES 1.0 is the first production-ready release of the system, the result of over 10 years of research, testing and field trials. It includes support for a wide variety of digital and film-based production workflows, visual effects, animation and archiving.
Edit:
What Technicolor Color up to now ? check out the YouTube link
https://www.youtube.com/watch?v=FFCoFlOaUKo
I don't know about that recommendation of dropping the saturation to -4.
QuoteWHAT ARE THE RECOMMENDED CANON EOS-5D MARK II CAMERA SETTINGS TO OBTAIN OPTIMAL IMAGE QUALITY? (http://www.technicolor.com/en/solutions-services/cinestyle-download/cinestyle-faq#what-are-the-recommended-canon-eos-5d-mark-ii-camera-settings-to-obtain-optimal-image-quality)
We recommend the following settings:
Sharpness: 0
Contrast: -4
Saturation: -2
Color Tone: 0
ISO: a multiple of 160
Another popular picture style that has worked well for me is the Prolost Flat (https://prolost.com/blog/2012/4/10/prolost-flat.html) which doesn't require loading a custom picture style through the Canon EOS Utility. Simply start with Neutral and:
(https://farm5.staticflickr.com/4477/37489991840_c5133c36b7.jpg)
Now why would professional cinematographers and a company with a reputation like Technicolor recommend taking the saturation down to only -2 and not to -4? There must be a reason for that.
As far as the ISO, on the Canon C100 and C300 when shooting C-Log the recommendation is to shot at ISO 850. Going below this and you will lose some dynamic range. This is clearly illustrated in this Cinema 5D article (https://www.cinema5d.com/exposing-with-native-iso/). This may not apply to picture styles because there is difference between a "real" log profile that is applied so the low level chain of image processing that it requires a hardware modification on the 5D4 and a picture style. Cinestyle is a picture style, you can clearly see if you disassemble the firmware that it lives in the same space as other picture styles so how can it possible be applied lower in the image processing chain than any other picture style?
I'm not trying to start an argument, just struggling understanding these concepts.
As far as the dropping the saturation down you lose colour depth. It's like over underexposing.
Underexposing is easier to fix than clipped highlights, but if you're so far underexposed then try to bring it up to a normal exposure you'll get banding.
Considering our eyes perceive contrast more than colour who knows how much it would affect visually, it's already 8 bit and 4:2:0 chroma sub sampled.
But try it yourself, in an extreme case, take any photo, de-saturate it to -95 (and apply it so its destructive if the program doesn't already), and then re-saturate repeatedly til it's roughly the same saturation. Enjoy colour banding for all.
Hi @reddeercity , can you answer this other thread (http://www.magiclantern.fm/forum/index.php?topic=20677.msg191538)? I'm very curious about ACES, but don't know much. Thanks.
I've seen a quite a few complaints about skin tones looking grey or plastic with different picture styles. Anyone here still facing that issue?
Quote from: bpv5P on October 17, 2017, 06:15:07 PM
Hi @reddeercity , can you answer this other thread (http://www.magiclantern.fm/forum/index.php?topic=20677.msg191538)? I'm very curious about ACES, but don't know much. Thanks.
Sure I'll try to explain it without confusing you more I hope
Quote from: markanini on October 17, 2017, 10:08:09 PM
I've seen a quite a few complaints about skin tones looking grey or plastic with different picture styles. Anyone here still facing that issue?
Yes, I had this issue with VisionLog (old beta profile from vision-color), but not with LogNeutral. You have to be really hard on color grading, though.
The way to go is MLV if you want quality. If you need to record something that doesn't need so much quality, go with LogNeutral + 3DLUTs, I would say.
Quote from: reddeercity on October 18, 2017, 04:41:21 AM
Sure I'll try to explain it without confusing you more I hope
Ok, thanks!
Quote from: dfort on January 02, 2017, 08:30:28 AM
Although as @reddeercity says it is a different "fruit" look at the way custom picture styles are installed on the Canon cinema cameras:
Wouldn't that be cool? For still cameras you get three custom picture styles maximum and you need to use EOS Utility to load them.
Thank you for your answer.
Ok i find the time to try understand.
The idea of picture style for the both is the same, different "fruit" but not soo different, it's a fruit.
The idea it's, if he can do that (reduce some color artefacts or noise) maybe it's possible to do the same by custom picture styles for DSRL.
Maybe for Log imitation picture styles too, like OESHD C-log.
I do some test in this way here (Half Log imitation picture styles too).
https://www.magiclantern.fm/forum/index.php?topic=20991.msg193295#msg193295 (https://www.magiclantern.fm/forum/index.php?topic=20991.msg193295#msg193295)
Hello! Today I tried out some picture styles on my Canon, and downloaded a few what I have found for free. I am interested in having bigger dynamic range without too much noise.
I was interested in checking the curves and settings of some, but the Picture Style Editor 1.22.20(the actual one) said that some of them is protected.
I searched for all my .pf2 and pf3 files on my PC, and found that Canon left a japanese tool called NakedPSE in the installer.
You can find them here:
C:\Program Files (x86)\Common Files\Canon_Inc_IC\UniversalInstaller\Install\Define\ja\NakedPSE
This program can open and edit any kind of Picture Style, does not matter if it is protected or not. You can save your modifications and import it into Picture Style Editor.
It has a long usage documentation, but all in japanese, so that is all I can say about it.
-obsolete-
Great timing. I read through this thread a year or so ago while looking for a way to access and hopefully edit the R, G, and B curves separately. Tried via hex as brought up earlier in this thread but couldn't get reliable results; probably more my limitation than the method itself, but anyway. Randomly got the itch to search again today and I'm glad I did. This NakedPSE gives quick access to the R, G and B curves separately. Signed up to the forum say thanks for bringing this to people's attention, at the very least.
What I've found so far is though profiles with edited curves work fine in-camera and in DPP, regular PSE doesn't reflect the alterations, and trying to edit the curves or most other 'hidden' things in NakedPSE and then editing the profile further in PSE causes the NakedRGB edits to be lost. Seems the only thing NakedPSE can change that PSE also maintains is the option to prevent a saved profile from being edited further or not. (And the basic sharpness, contrast, saturation and tone settings, but you don't need NakedPSE to change those.) Maybe I've missed something but it seems opening a .PF2 in PSE will essentially strip it of anything that wasn't edited within PSE itself. So you need to use NakedPSE to 'unlock' a profile, then edit it as much as possible in PSE, then open it up again in NakedPSE to mess with the 'hidden' parameters, or do it all in NakedPSE.
Additionally I've found that entering in any kind of negative or inverted curve, either in the RGB curves or the LAB curve, will cause DPP to crash when loading the profile; I don't know if a camera could take it but I'm not willing to risk it in a camera if DPP can't handle it. Seems every curve point has to be higher than the one preceding it. Whatever algorithms Canon uses to smooth out the curve can't handle it, I'm guessing. So no in-camera correction for scanning film negatives, I'm afraid. (Shame, since Nikon's editor can do that.)
Will be doing a lot of trial-and-error (since I don't speak Japanese) with every function for the next few days. Hope some of the more clued-in posters here can return to this thread and help 'decode' how much NakedPSE can do.
In any case, thanks again for finding that and bringing it to peoples' attention, this is potentially huge. Never thought I'd be able to get split-toned colour in-camera, but there it is.
Thanks! Keep us posted on what you uncover :)
Way back I unlocked those protected files. Actually it was a simple comparing locked and unlocked file and then patch some value at the end. So long ago but shouldn't take long to fix.
On the other hand. Picture styles seems rather meaningless as they are applied to 8bit h264. Better just dial in some visibly wider range straight in cam imo.
Oh boy!!! I've been trying to edit/understand picture styles for about 4 years now! You just made me get back to this puzzle, Dobfek. Thank you!
By taking screenshots and then uploading to an online image translator called "yandex" I was able to understand a little and I think that this software needs more stuff to work properly.
Anyway, by messing around I saw that you can Export .icc files, .luc files but it gives a message of error that I don't understand...
You can Import a 3x20 Matrix by importing a file (.mat) but I have no idea on how to build a Matrix... Maybe with Matlab like Andy600 said years ago
The LabCurve values have to be in RGB and not LAB values. The values can be both in 8bit or 10bit, in both curves.
You can disable the Edit Lock by changing the value of the DisableEdit tab from 1 to 0. Some Picture profiles show the L curve in PSE after resaving.
The RGB curve doesn't show on PSE and I think its because in NakedPSE you can actually set the R,G,B values separetly, while in PSE its a merged RGB curve.
You can't open .pf3 files in NakedPse, but if you rename the file extension to .pf2 you can.
I'm not sure if by renaming you can really access all the information.. When I opened a famous CLOG profile, there where no information in both RGB and LAB curves.
I'm looking foward to see what the experts will say about this.
btw, I checked several paid Picture Profiles and realize that there are some which are very poor in terms of quality...
Oh, the long documentation is about the PSE and not the NakedPSE. But there are way more information there than in other places like Canon website(I uploaded to an .Doc translator as well.)
Sorry if theres anything wrong in the text. English is not my main language, and I'm far from being anything in this Software/Color World
Quote from: Danne on September 14, 2020, 08:21:24 PM
On the other hand. Picture styles seems rather meaningless as they are applied to 8bit h264. Better just dial in some visibly wider range straight in cam imo.
If your goal is only to push the latitude a little further for video and your camera doesn't have log, sure.
But for creating specific looks, this is a big deal. For anyone who needs to run-and-gun and use either footage or .jpgs straight out of camera, being able to now easily edit R, G and B curve values in addition to all the other adjustments that can be made in PSE could be quite significant. I have to keep a Fuji system on me alongside Canon as although I have a better lens selection with Canon, sometimes I'm asked to take some behind-the-scenes snaps to post right away and Fuji's Classic Chrome and Classic Negative profiles have so far been my most-requested looks for that. They're also the two colour profiles that are really unique to Fuji and PSE couldn't replicate before due to the lack of shadow tinting. With this NakedPSE and a bit of time (and ideally some way to get .PF3s working), that could well no longer be the case.
We're basically talking about bypassing any need to shoot flat for later grading and just getting the finished, graded look done in-camera instead. It may not be 100% of the quality on pixel-peeping inspection, but for times when speed matters more than quality, this is (potentially) extremely useful.
Quote from: aceflibble on September 14, 2020, 09:05:30 PM
We're basically talking about bypassing any need to shoot flat for later grading and just getting the finished, graded look done in-camera instead. It may not be 100% of the quality on pixel-peeping inspection, but for times when speed matters more than quality, this is (potentially) extremely useful.
Thats what made me excited about this again. The idea to make pretty accurate looks, like: Accurate REC709 colors straight outta camera(while Faithful is the closest one to the REC709, the colors are still a little bit off), or, in the other hand an specific look as you mentioned with your FUJI system, that in my case would be a specifc colors from a Film Emulation that I use as a starting point in my gradings.
As I said before, I am far from being able to understand this Sofware/Color/Math thing, but I have a feeling that it might be possible to get accurate results by importing a Matrix (.mat) files.
Quote from: thiagoteh on September 14, 2020, 08:24:45 PM
You can Import a 3x20 Matrix by importing a file (.mat)
I was wrong. You can actually
export a 3x20 Matrix. Not import.
Yeah most of the options in NakedPSE are for exporting in various formats, which doesn't help us really decontruct the colour profiles.
FWIW I've found that, like with the R, G and B curves, any changes to the 3x20 matrices in NakedPSE will be lost when opening the profile in PSE and some severe changes in either matrix results in my test EOS R crashing. Still trying to find out the exact limits but it's a lot of trial-and-error. For now it seems like editing the matrices is not advised.
The good news there is I've gone through many premium profiles—about thirty now—and none of them have used custom matrices anyway. It does seem like every profile people have been paying money for over the last ~14 years has in fact been made entirely using regular PSE with nothing truly custom or out of the ordinary. Good news for those of us who'd like to know what changes were being made and how, since it's all much simpler than it used to seem, but bad news for those who paid money for a profile thinking they were getting some actually new functionality.
Can I make a request? I wish to use a Neutral picture style with a Standard tone curve.
So a kind of reverse of what the Fine Detail profile does? (Which is the Standard hues but with the Neutral/Faithful contrast.) I'll see if I can cook it up. Might be a bit tricky because the Neutral profile is truly unique among all the profiles as its matrix is literally the most basic it can be; out of the entire 3x20 matrix grid, the only values are simply 1 in each of the initial red, green and blue channels, and nothing more. (See way below for an example table.) All the other profiles (update: except Standard, see following edit) have entries in every single position, accurate to six decimal places and don't actually use the tone curve inputs at all. Getting 60 values to work with the hue produced by just 3 might not be possible precisely, but I'll try to get as close as I can.
EDIT: I just checked. I don't think it'll be possible to copy the tone of Standard, and I've hit something very interesting. It seems the Standard and Monochrome profiles do not work the same as other profiles. There's almost nothing in them—the Standard matrix is the same basic one used by Neutral, shown way below—and instead it causes the software or camera to load up some kind of hidden 'default' rendering based on whatever profile is named at the top of the file. If you try to open up Standard you're basically just working with a drop-down list of the camera defaults. These are probably tailor-made for each camera model, compensating for the different sensitivity of different sensor models. This does not happen for other profiles; the drop-down list and named profile are ignored if anything other than Standard or BW (Monochrome) are chosen.
For example, if I change Standard's listed profile to Landscape, using that new profile just looks 100% like the Landscape profile even though I didn't touch the the matrix or curves; the look of Standard has been lost entirely. If I load up that new profile the matrix still is untouched, even though it's now producing the Landscape colour. But if I do the opposite, changing the Landscape profile's setting to Standard, it continues looking like Landscape and never gains the look of Standard.
In other words, Standard doesn't exist like the other profiles do. Standard is some kind of real firmware-level default, not something we can inspect.
Monochrome is the same deal, there's no way to extract it and it only exists in the same drop-down menu as Standard. Strangely, though selecting Standard in other profiles does
not turn them into Standard, choosing 'BW'
does turn any other profile into Monochrome but without any of the Monochrome-specific controls.
I expect there are many values still hidden to us that are controlling these things. I shall try to cook up a version of Neutral with contrast similar to Standard for you, but it looks like being able to copy Standard's contrast 1:1 is off the table.
Funnily enough it's those sorts of 'simple' adjustments that are turning out to be the hardest.
In the last month I've been messing with the matrix a lot, trying to 'decode' what each entry point refers to (given the matrix isn't labelled and the limited Japanese documentation with the software seems to only refer to the standard version of PSE), and I've found some ways NakedPSE can push beyond what regular PSE offers in order to create a variety of effects; some very useful, some... not so much.
Effects I've got mostly nailed now include negative/inverted colour and exposure, selective colour, single- and two-tone colourisation (separate from split toning), and posterisation. I've also mostly copied over Fujifilm's 'Classic Chrome' profile (about 90% accurate, stills needs some tweaking) and I'm going to start working on their 'Classic Negative'. (I did start copying Astia and Pro Neg S as well, but it turned out those were already 99% like Canon's own Portrait and Faithful profiles, so there wasn't much point bothering.)
... Then there's this thing (https://twitter.com/AceFlibble/status/1320000253286883331), which I'm tentatively naming 'in-camera LSD'. That's what happens when you use NakedPSE to push regular PSE's 'specific colour' tool beyond its normal 30-180° limitations... in this case, typing in a hue range of 6000° and a hue shift of 4000°. Totally useless, but quite amusing to see working in live view/video and it goes to show just how free our in-camera colour choices are now. That one was an accident of mucking about, but I am now trying to work on a profile that keeps that has that sky gradient on purpose, without affecting the rest of the image. Just thought it'd be fun to have rainbow-coloured skies. Probably won't work, but we'll see.
Anyhoo, some more practical things I've picked up:
The Fine Detail profile that Canon added with the 5DS and the introduction of the .pf3 format can be converted to .pf2 and works perfectly fine in older cameras. There's nothing in its colour or contrast that the older cameras can't do. The only thing is you do of course lose the improved sharpening controls (old cameras still only get the one sharpening control, not the strength, fineness and threshold controls of newer cameras), but that's to be expected as that's not a system that is saved in any of the colour profiles. Or, in other words, Canon claiming Fine Detail was only made possible by the new tech and limiting it to .pf3 was always a lie. It's just a 3x20 matrix, same as all the others.
Canon's Video-X profile is the only one of Canon's own profiles that uses a non-linear RGB curve. The Video-X's curve lifts the shadows quite a lot and very slightly lowers mid tones. In fact Video-X is the only Canon profile that touches the curve inputs at all; every other profile simply has the curve set at 0-0, 255-255, with no other points between. The 'LAB' curve (which as noted earlier in this thread doesn't seem to actually be a LAB curve as it uses RGB values and the A and B axis are entirely ignored) is also plain linear in all profiles. (Video-X included.) All the contrast changes between the profiles other than Video-X is done entirely within the 3x20 matrix. Makes you wonder why Canon even bother having the curves at all.
Common wisdom for the last ~12 years, especially for video, is to create a faux-'log' look by selecting the Neutral profile and then setting both Contrast and Saturation to -4. Many books and sites will tell you that even the Neutral profile adds some slight contrast. You may also have heard that the Contrast and Saturation controls are applied before the picture style. Turns out, that's all gibberish. By doing a lot of back-and-forth with different custom matrices and seeing how each control behaves in response, I've deduced that Contrast and Saturation are applied
after the profile's matrix and gamma but
before the specific colour adjustments, then Tone and Sharpening are applied and finally the LAB curve is added. (Yes, after sharpening, bizarrely.) Neutral
doesn't add any contrast to anything, it truly is as pure as a colour profile can be, as I detailed above. So setting Contrast and Saturation to -4 doesn't actually record more detail in anything with the stock profiles; if a stock profile's matrix is going to blow out a colour or crush a shadow, it will have already done so
before those controls are applied. All that is happening with the Contrast control is it's applying a post-matrix, post-gamma brightness curve. Anything it appears to have 'saved' was in fact always there, just perhaps not entirely clear to you. (After all, how many people can tell the difference between a 100% white pixel and a 99% white pixel?) It also does definitively reduce detail as whatever space you make in the highlights and shadows is taken away from the mid tones. The Saturation control works the same way; it can't actually recover anything that has been genuinely blown, it's just enabling you to see more of the nuance in the most saturated parts of the image at the expense of losing nuance in the less-saturated parts.
In practical terms this shouldn't really affect how anyone uses either the Contrast or Saturation controls, or the Neutral profile—after all, as long as your results look good to you that's all that matters—but it does mean anyone who previously said the Contrast control could actually expand dynamic range was simply, categorically wrong. Technically Neutral already has the fullest range possible, and changing Contrast or Saturation is just trading one part of the range for another.
Somewhat related,
a question of my own: A lot of people used to rely on CineStyle, especially with with Contrast and Saturation turned down. That was, for years, 'the' way to film with a 5D2. CineStyle and many premium 'flat' profiles boasted about being able to give you more dynamic range and more colour nuance than any stock profile. Well, here's the CineStyle matrix:
1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Look familiar? Yup, it's just the Canon Neutral profile. No change to initial recorded colour or contrast. Just like I outlined with the Contrast and Saturation controls, if something was being blown out (either a tone or colour) then that's it gone for good, nothing following it could save it; anything that seems 'saved' by CineStyle would also be equally saved by Neutral. It's not doing anything to save more data than the stock Neutral profile already does.
But what about the gamma curve? That certainly is different, but it's not doing quite what you might expect and this is where I want someone else to weigh in:
0 - 64
128 - 227
256 - 350
384 - 441
512 - 533
640 - 625
768 - 722
896 - 832
1023 - 1023
Given the curves in normal PSE work on just an 8-bit 0-255 scale, and Canon's own profiles all use an 8-bit 0-255 range, that sure is weird. It seems Technicolor found out (or Canon tipped them off) that they didn't have to just stick to the standard scale and used a 10-bit 0-1023 scale, even though 10-bit recording wasn't available at the time the profile was made, and then they... remap 90% of it to fit in a 64-940 video range/reference monitor's range? But then not the very top of the range? It's like they thought they were mixing HDR to SDR... before HDR existed... and thought the black point had to meet video standards but
not the white point. It's
bizarre.
This is, like I said, where I bow out and am asking for someone else to explain this one 'cause it's the one part of all my digging that has me stumped. I have been able to verify with lots of trial-and-error that it's not presenting any more 'recovery' than a simple 8-bit value, low-contrast curve can. It can't be for stills because .jpgs are 8-bit and this profile has no bearing on raw files. I know ML eventually got 10-bit video working on otherwise 8-bit SLRs, but that was well after CineStyle was made and released and neither Technicolor nor Canon would have been working on a profile on the assumption that a third party might make additional software to hack in extra functionality later.
So I just can't get my head around why they plotted the tone curve this way and what they hoped to achieve. Who would be working on an 8-bit system but use 10-bit values? And why would the cameras accept this when they can only apply it to 8-bit files? Anyone? Anyone? Bueller?
Cinestyle gamma was already reverse engineered. We called it logneutral. when it was developed a lot of other related questions was dicussed. Long story short. Applying such a cineon-like push in gamma on 8bit-content is not adviced. Picture style changes comes to late in the chain, it will mostly do worse than good.
My personal take. Much better just stick with some careful in camera setting changes and then do all other changes in post, grading in your nle. Easy does it.
Their values aren't the same as what you had earlier in the thread, though. It also doesn't explain why, let alone how, they put in 10-bit values for an 8-bit system, which is what I'm asking about. We know what the end results are; I want to know what drove them to come up with that method to begin with.
And no, picture style doesn't come in "late in the chain"; it's throughout the whole chain, it is the chain. The first thing that happens is the cameras is checking to see if Standard or BW was chosen and then after that, assuming neither was, the image is rendered using the matrix specified in the picture style. It's coming before the gamma curve, before all the in-camera controls, and before the LAB curve. The different in-camera controls are implemented at different points between. That is not insignificant.
Also, as I said the last time you replied with something along those lines, there are more purposes at hand than just trying to get a neutral video to grade. That is one use case that is interesting to dig into, but ultimately we're talking about something that affects in-camera .jpgs, histograms, and of course can be used for all manner of creative purposes, especially for those who possess cameras that are not able to shoot in more technically-accurate, edit-friendly styles such as real log or 14-bit raw photo. (Or simply for people who don't want to or can't spend their life sat at a computer editing their photos or grading their videos, which is an increasingly large market thanks to vlogging taking over the planet and peoples' desires to get photos from camera to online ASAP.)
I'm not trying to suss this out because it's the end-all best workflow for professional video, I'm trying to suss it out because it's useful for all manner of purposes and because it's plain interesting and fun to dig into.
What 10bit values. I can't follow.
Some of what you're saying might be true but bottom line is still. 8bit from cam or with a picture style applied there will not be any magic changes happening with a pic style. For that to happen we shoot raw ;).
I gladly review comparison tests backing up your info.
For shooting I find the picture profile makes a difference to an extent, because it affects my prefererad exposure and custom WB with other factors being equal. That's why part of my shooing routine involves setting the contrast and saturation controls as close as possible to how I will be processing it in post(mostly doing photos).
Standards colors can look a mannequin-like on some people subjects, sometimes it looks more flattering on female subjects. Neutral looks more life-like but looses saturation in highlights.
Standards tone curve raises the shadows a bit while neutral lowers them. DPP adds shadow and highlight controls to picture styles and I use them a lot. It's a buggy piece of software though, previews often don't match the output and I'm pretty sure it's wide-gamut mode is broken. I created custom camera profiles for Lightroom using a color checker and they look more accurate and generally more useful than the included ones, but the stock Canon colors are simply more pleasing and require less work.
If you think it's possible to make a neutral PS with standard tone curve I'll certainly give it a try. A "backported" fine details sounds interesting too.
Like I said, it won't be possible to copy the Standard curve over 100% onto the Neutral colours as you requested above—it's not a copy&paste job, unfortunately—but I see no reason why something
very close to the Standard tone curve could not be created on top of the Neutral colours. It'll just be a matter of trial-and-error, lot of tweaking, checking, tweaking a bit more, checking again, etc. Of course anybody else is welcome to have a stab at it as well, but in any case I'll give it a shot this week and let you know how I get on. At the very least it'll be interesting to try to replicate the curve without having the actual numbers on-hand to copy.
In the meantime, here is Fine Detail converted to .pf2 (https://we.tl/t-BXCx9xJh6s). By default it will have sharpening set to 1 and everything else at 0, but of course that can all be changed however you like. I really like Fine Detail for all-round use, it's got a bit more saturation and life to it than Neutral but it's more realistic than Standard, I find.
I've only tested that profile with an EOS R (to verify it was producing the same colour as .pf3 Fine Detail), a 7D mark II and a 5D mark II, though. I can't think of any reason why a custom profile would damage any other camera, but just to be on the safe side, I suggest you try it out by loading it into a user style slot that
isn't currently set as your default; take a raw photo with the camera set to one of the stock profiles, then in playback have the camera re-process the raw image into a .jpg with the custom picture style. That way you can safely check that the camera will load the profile up and use it correctly. As I said, I can't imagine there will be any problems, but y'know, no harm in being cautious when loading custom files onto an expensive piece of equipment.
And yeah, I'm with you on the ColorChecker profiles. I used to swear by them and they are definitely more accurate than anything Canon makes, but I got tired of the extra effort years ago and I've been working mostly with just Neutral as the base profile in Lightroom or Fine Detail in DPP and in-camera. That's part of why I shoot Fuji alongside Canon and why I'm so keen on digging into custom profiles; if I can finally get the colour I want right in-camera, nice and quick, then I don't mind giving up a bit of technical quality. The less time I have to spend jerking colour around in Lightroom, the better.
Quote from: Danne on October 26, 2020, 08:29:54 PM
What 10bit values. I can't follow.
... The ones I listed above and asked about. Y'know, the question which you responded to in your previous comment. The ones you already read. Technically, the ones that were talked about in this thread many years ago, 's just back then you were trying to dig it out of hex values and eyeballing with image analysis tools. (Page 7, Andy600's post about halfway down.) Well, now we've got the
actual, factual numbers for that curve (which turn out to not be the same as the earlier thread concluded), and they seem completely out of place with everything else we know about these files. So to reiterate my question, I'd like to know if anybody can think of a reason why CineStyle would have been made this way instead of with 8-bit values like every other profile is, or, alternatively, if someone can spot something in those numbers that suggests an actual unique purpose that I've overlooked which could help us further 'decode' these profiles.
Works perfectly on my 600D.
Cool. My assumption is there's nothing harmful in a 1-4kb colour profile, but I always like to be as safe as possible. You never know when technology is going to disappoint you.
Imo there´s no conscious 10bit decisicion behind building the cinestyle profile. Maybe a visual 10bit reference somewhere(maybe cineon) but I would assume more likely a "just for fun" marketing hype pic style. Still, I don´t really get the 10bit reference, or source, since all that was done creating the pic style is done with the 8 point(or 10, can´t remember), curve tool in the pic style editor. All in eight bit. Unfortunately no "blackbox magic" in the pic style.
I really like the Cinetech picture style for my shooting style from Visioncolor for shooting 8 bit .h264. More dynamic range compared to neutral (prolost settings). Maybe a little bit more noise but you can clean this up in post.
Quote from: Danne on October 27, 2020, 08:04:54 AM
Still, I don´t really get the 10bit reference, or source, since all that was done creating the pic style is done with the 8 point(or 10, can´t remember), curve tool in the pic style editor. All in eight bit. Unfortunately no "blackbox magic" in the pic style.
But that's the thing, we all assumed that for years but it turns out it
wasn't created that way. They could not have possibly put in these values in PSE. It's literally not possible for that software to do it. So either Canon clued them in to NakedPSE all those years back (unlikely given it has apparently only ever existed in Japanese and if it was being offered around you'd think we'd have heard of it sooner) or Technicolor had some other method
not using any version of PSE in order to set those values. So our previous assumption that they just knocked something together in PSE is wrong, they had something else going on back then and I'd love to track down one of those developers and find out what they used.
But
however they set the values that way still leaves me wondering
why they used the 10-bit values, when they could have created effectively the same curve shape using the 8-bit values every other profile uses, and of course it doesn't explain at all how it is the Canon systems can recognise and action those 10-bit values when they're supposed to be working with 8-bit values and only applying them to 8-bit files. The only thing I can think of is the slim possibility that the cameras actually do work in at least 10-bit (if not higher) before the gamma curve is applied. This tallies with ML being able to get 10-bit video out of older cameras (albeit usually with a crop or strict time limit) but the assumption
there has always been that that was some kind of extra-tough hacking wizardry that ML was creating and Canon had no clue about... but these profiles and the cameras supporting 10-bit values suggests Canon was working towards this themselves, but for some reason never completed the functionality. I mean, I just can't think of any other reason why Canon would make sure their system could work with 10-bit values on a post-capture processing ruleset unless they actually thought they might include some kind of 10-bit (or higher) recording. (Outside of raw photos, which of course aren't affected by these curves.)
As I said earlier, the whole thing is mind-bogglingly strange.
Quote from: Nigel95 on October 27, 2020, 09:15:11 PM
I really like the Cinetech picture style for my shooting style from Visioncolor for shooting 8 bit .h264. More dynamic range compared to neutral (prolost settings). Maybe a little bit more noise but you can clean this up in post.
I'm afraid as we've pulled apart the files, there isn't anything which can capture more dynamic range than the stock Neutral profile already does with its true bare-bones matrix. What you're seeing in those boosted styles is not
more dynamic range but simply a
reallocation of the range, typically brightening shadows up (hence why you're seeing more noise) at the expense of cramming the mid tones further together. There's actually a good example of this type of 'expansion' much earlier in the thread. If you look on page 5 you'll find Danne trying to work out a more linear DCP profile and thinking they may have seen more dynamic range (but to their credit, acknowledging they may have been imaging it) and dfort following up a few replies further down pointing out that after careful inspection there
wasn't more
range, it's just that the peak brightness was a little lower so it made the
same detail look slightly better-defined. What you're getting with your noisier Cinetech profile is the same thing at the opposite end of the scale, as I said before; the
same range, just gaining definition in one area at the cost of definition in another.
Canon's own plain ol' Neutral is the matrix that renders the widest range and the most detail. Not necessarily the most clearly-defined detail and range, but the most in a total sense. You simply can not get purer than that 1/1/1/0/0 matrix.
But, as I said as part of my much longer comment above, if what you're using is working for you and getting you the results you like then don't worry about whether or not some other method is or isn't technically better, or easier, or more popular, or whatever. If the final image you get is how you wanted it to be and you like the workflow then just keep doing what you're doing. 8)
Before anyone suggests it, I've already tried using 0.9 and 0.5 matrices and then boosting saturation back up later, and they don't improve anything. Dropping the initial strength of each channel does improve clarity in mid tones but takes it away from shadows and highlights, so again it's just a trade-off. There's no actual improvement on the simplest matrix, unsurprisingly.
If you want to shoot in black & white then I have found a totally even 0.3/0.59/0.11/0/0 matrix does seem to give more mid tone nuance at no cost to shadows or highlights compared to the stock monochrome profile, but personally I like my black & whites to be a bit more punchy than true-to-life.
Quote from: aceflibble on October 28, 2020, 12:05:18 AM
I'm afraid as we've pulled apart the files, there isn't anything which can capture more dynamic range than the stock Neutral profile already does with its true bare-bones matrix. What you're seeing in those boosted styles is not more dynamic range but simply a reallocation of the range, typically brightening shadows up (hence why you're seeing more noise) at the expense of cramming the mid tones further together. There's actually a good example of this type of 'expansion' much earlier in the thread. If you look on page 5 you'll find Danne trying to work out a more linear DCP profile and thinking they may have seen more dynamic range (but to their credit, acknowledging they may have been imaging it) and dfort following up a few replies further down pointing out that after careful inspection there wasn't more range, it's just that the peak brightness was a little lower so it made the same detail look slightly better-defined. What you're getting with your noisier Cinetech profile is the same thing at the opposite end of the scale, as I said before; the same range, just gaining definition in one area at the cost of definition in another.
Canon's own plain ol' Neutral is the matrix that renders the widest range and the most detail. Not necessarily the most clearly-defined detail and range, but the most in a total sense. You simply can not get purer than that 1/1/1/0/0 matrix.
But, as I said as part of my much longer comment above, if what you're using is working for you and getting you the results you like then don't worry about whether or not some other method is or isn't technically better, or easier, or more popular, or whatever. If the final image you get is how you wanted it to be and you like the workflow then just keep doing what you're doing. 8)
Interesting stuff. At least the Cinetech gives me a faster workflow with colour grading in post compared to Neutral. The noise doesn't bother me as it's very easy clean up with a plug-in like Neat video. Thanks for the explanation.
Just a vague update, I've been in pretty poor health this last week or so—on-going back problems—so I haven't been able to sit down at the computer and work on that neutral-standard profile or, well, anything really. I realise this is inane to most people but since I did say I'd work on it and be back in a week, I just wanted to drop in quickly and explain why I'm a bit slow.
While I'm here I'd like to again request for help, from anyone capable and that has the time, in investigating whether we can get NakedPSE to open and save .pf3 files properly (thus keeping the useful 'six axis' changes from regular PSE) or if we can get regular PSE to honour the matrix and curve changes that NakedPSE can make. There are some changes that are simply quicker and easier to make with the .pf3 six axis option than within the matrix or the specific colour adjustments, plus it's a bit suspicious how the .pf3s are 100x larger than .pf2 files even when no changes have been made and it'd be great to know what's happening in all that extra data.
Thanks for the update, hope you are doing better by now. I've considered doing the neutral-standard profile myself. Maybe using a shot of a Kodak grey scale chart and color picker windows app as a guide should work?
Hi
Where can I download logNeutral v1.3?
Thx in advance.
https://web.archive.org/web/20200622002229/https://bitbucket.org/daniel_fort/magic-lantern/downloads/
The link to aceflibbles fine detail picture style for older cameras died, hope he's fine with me re-uploading. It's served me well vs. natural which seems to desaturate near white colors excessively, causing hot spots on people subjects to look worse than they are.
https://www.mediafire.com/file/agpzrqxo7zdgdry/Fine_Detail_for_older_cameras.PF2/file
I played around with the picture style editor tonight, and I can't seem to find a good way to simply lower exposure by 1 stop across the board. My purpose for doing so is to create a picture style that will allow me to preview final exposure in realtime on the camera LCD and downstream when i plan to pull exposure by one stop in post. If anyone has any tips thanks in advance