Messages

And here's the same test in pf2 format:


(Locked on the left, unlocked on the right)

Technicolor Cinestyle on the other hand has a completely different end of file, so not sure what to make of that. It also has two sections that according to my hex editor simply run through every keyboard character.

Thanks for the Java tool, chris_overseas! Ran the test like you suggested, by opening PSE and making two pf3 files where the only difference was checking the 'disable subsequent editing' box on save. Compared in a hex editor, turns out that one option changes two bytes - the one you noticed, but also the byte that's 9 bytes in from the end.


(Left side is the locked file, right side is the unlocked file.)

I tried changing only the byte at 0x6a113 from 00 to 01, and when trying to open in PSE it no longer shows the caption/copyright info and gives an error when loading. If you change both bytes to match then the file is successfully unlocked and editable.

(EDIT: Had to go read through the thread again to realize that the newest version of PSE lets you pick pf2 or pf3)

Found PSE 1.3 for Windows at Canon Europe's site: http://www.canon-europe.com/support/consumer_products/products/cameras/digital_slr/eos_1d.aspx?page=3&type=download&language=en&os=all

Still uses pf2 files, in case that's useful for anybody. Saving out a default file with 0 in the comment field gave a file size of only 1431 bytes, so this may be a much simpler format to decode.

Wow, cool find. The only pf3 files I've seen online are the ones in post 1 here, which are all either 434,490 or 434,466. I just tested saving out a default (zero adjustments) pf3 with PSE and it seems the file size fluctuates slightly with the number of characters in the Comment/Copyright fields when saving. The comment field can have 31 characters max, and the copyright field can have 63. Filling both fields with all 'z' characters gets a filesize of 434,529 bytes, and putting only a single '0' in the comment field gives me 434,466.

From looking at pf2 files by Canon and this collection of user styles: http://cinescopophilia.com/wp-content/uploads/2012/03/158%20VW%20Collection%20CPS.zip - I can see a wide variety of file sizes from 1243 bytes up to 20623 bytes.

I wonder if another way of doing this would be to create a computer-generated CR2 file that contained known values in a color chart (like a faked, exact Macbeth chart instead of a photographed one), load it into Canon's Digital Photo Professional for converting to 16-bit TIFF using a specific picture style, and then analyzing the RGB differences for each swatch to generate an ICC and a DCP.

It wouldn't let you mathematically generate a pf2/pf3 from a known colorspace like CLog or ArriLog, but it would let you get from existing pf2/pf3 files like Cinestyle into a mathematically generated ICC/DCP file.

Here's the breakdown of the CR2 file format - http://lclevy.free.fr/cr2/

What would be possible if we could get into the Canon file formats? It would be possible to build an MLV to DNG application that translates the picture style into a dcp that can be tagged into the DNG files. Of course the DNG files need to be debayered by a program that understands how to decode the embedded dcp information and it seems that only Adobe Camera Raw and Lightroom can do it.

You guys are my heroes! I just started looking into this exact pipeline again at work, hoping to figure more out. This is what we've wanted for years at our stop-motion studio - monitor on stage using Canon Landscape jpgs, use the CR2s (converted to linear EXRs using ACR/After Effects) in VFX, then apply the Canon Landscape look as a LUT in post so that our DP can start coloring based on exactly what he saw on stage. @Andy600 has done some fantastic work with Cinelog (thanks again, Andy!) but this is still the missing piece of the puzzle. I PM'ed Eric Chan from Adobe about the ACR side of things but haven't heard back yet. I'm still learning with regards to LUT formats, and maybe this was already doable with some other method, but being able to turn a picture style into a dcp would be a god-send.

In case anyone missed it: http://www.cinelogdcp.com/news-updates/2016/5/31/cinelog-c-dcp-2016-coming-soon

Hey @Andy600, any updates about the new ACR/AE camera profiles and Cinelog plugin? Still really excited to try them out!

The Picture Style number in .cr2 files is just a flag to tell DPE which PS was active (and used for Jpegs). Custom Picture Styles are embedded in the .cr2 and read by DPE.

Andy also confirmed this, and I saw it myself by looking at a cr2 shot with Cinestyle and loading it into DPP - it correctly extracts the profile for further processing, and that software also allows you to select a new custom picture style for after-the-fact applying to cr2s.

I would love to know why Adobe wasn't able to use Canon's ICC profiles for creating their Digital Camera Profiles in ACR. I remember reading that they had to reverse engineer the look of each one in the standard set (ie Portrait, Landscape, Neutral, etc.), and the results never match Canon's in-camera or DPP processing of the same image in terms of color. Since ACR's raw processing is so much better in all other aspects, I wish there was a way for it to match Canon's color for cases where their picture style is the exact look desired.

The Picture Style ICC profiles are the looks and are available for both sRGB and Adobe 1998 colorspaces i.e. FA = Faithful (Adobe 1998), FS = Faithful (sRGB), SA = Standard (Adobe 1998), SS = Standard (sRGB) etc etc

On a related note, could these ICC profiles be used to create a Digital Camera Profile for use in Adobe Camera Raw that would let me match a .cr2 to the color look of the in-camera JPG?

You could use Premiere to export an image sequence of the whole movie in a format that supports higher than 10-bit (ie 16-bit DPX, TIFF, maybe EXR if you want to buy the ProEXR plugin), as well as a separate audio file. Keep those as your master, then render out compressed versions of whatever you need for deliveries.

ProEXR is for giving EXR functionality to Adobe products, and in After Effects' case some extra functionality above what's already there. AE's built-in EXR plugin is a free version of ProEXR, minus the ability to work with multiple channels.

Yeah, thank you again for your information Andy, I finally feel like I'm getting a solid grasp of all this!

And for anyone else reading this, I can definitely recommend the Cinelog package, very much worth it, especially for timelapse/stop-motion.

Ok, most of what you're saying makes sense, at least compared to what my eyes tell me, but I'm still struggling with the math.

If a Bayer patterned pixel array goes to a 14-bit analog-to-digital converter in the camera, then presumably for each pixel there's a certain amount of light that will result in a maximum value (ie 16383). My understanding of demosaicing is that at its most basic you are taking, say, a green sensor element with a certain value, and averaging it with the neighboring green elements, and then averaging the nearby red elements and the nearby blue elements to come up with an RGB value for that pixel in the resulting image file.

So, if there's a hard limit in the sensor to the amount of light it can record at one element, and that resulting value even when averaged with nearby values can't possibly go above the maximum value of a 14-bit integer, then shouldn't we be able to convert raw 14-bit data into a 16-bit image with nothing done to the values? If it's a linear sensor, then wouldn't it just give me a linear 16-bit image? I know that doesn't look 'sane' on a monitor, but it's trivial at this point to use a LUT to convert from linear values to something else for viewing or editing.

Have I just answered my own question? Is it simply that ACR is not built to go from linear to linear, and that the conversion from linear to vid (by which I mean display-referred values) is capable of pushing the max linear value of 16383 to a vid value of something much higher, resulting in clipping even in a 16-bit space?

So I've been playing around with Cinelog at work, it's a great tool for the toolbox! But now I'm trying to settle a debate with my coworkers:

If the Canon 60D has a 14-bit sensor (I'm assuming that means 14-bit per channel) and yet going through ACR with default settings and one of the Canon camera profiles such as Camera Neutral into a 16-bit per channel project somehow does not allow for retaining the full dynamic range, then what is the bit depth of the ACR importer? Is it less than 14-bit?

I have read that these sensors record 14-bit integer data to their .cr2 files, and After Effects can easily do 16-bit integer data, so unless the camera is unsigned and AE is signed, we're having a very hard time understanding why something like Cinelog is even necessary (even though clearly it is for what we want to do). Why isn't the brightest pixel in the 14-bit .cr2 file still the brightest pixel in the 16-bit integer project?

Perhaps 14-bits of un-debayered data is equivalent to much more than 16-bits of integer data?

Ah, that is very interesting. Are you referring to Cinelog?

Thank you, Andy! That simple description is what I've been looking for all week.

mothaibaphoto, sorry it was a poor choice of image host. Fixed now!

Yesterday several pieces finally came together that helped me understand the issue. Regarding Adobe Camera Raw, I found this Adobe support thread: https://forums.adobe.com/message/4519452 with this very important post from an Adobe staff member:

Hi Joe,

Our design of ACR/LR is to render images for display or print.  When starting with scene-referred images (e.g., raw), this means tone and color mapping the input linear light data (e.g., whether 12-bit integer or 32-bit float) to values suitable for reproduction (output-referred).  Since the resulting is always intended to be output-referred, the resulting images are always stored using 8-bit or 16-bit values.  We do not see any need to represent output-referred data using 32-bit values.

(ACR/LR is not intended to be a "pass-through" system that can, say, take 32-bit image data as input and also emit 32-bit image data at the end.  It can be made to do so, though clumsily.)

There is a ton of misinformation out there about ACR keeping all of the raw data, and it's simply not true, although whether that matters for your workflow is another issue. I wish I had known that awhile ago.

For some reason that simple string of directions that Andy gave has been very hard to come by, even though it's standard practice at other studios. So thank you again for helping spread that knowledge around!

Hello everyone,

I have been developing a VFX pipeline for a stop-motion production (essentially the same requirements as RAW timelapse) and am struggling to find an answer to something that seems very basic.

We are using Canon 60D cameras to capture 5K RAW .cr2 files, and our VFX team uses After Effects for compositing. However, our computers are not really up to the challenge of working directly with .cr2 image sequences using the Adobe Camera Raw importer, at least not when we need to crank out dozens of shots a week.

What I would like to do is convert the .cr2 sequences into 2.5K .exr sequences (linear/scene-referred). In doing so, I would love to keep ALL of the dynamic range that is present in the original .cr2. This seems like a pretty standard workflow for large productions - take raw camera footage, convert to 16-bit linear .exr for grading and VFX, then output to whatever. It's even the standard process for the new Academy ACES workflow.

However, I cannot find a method for transcoding that actually keeps all of the dynamic range (essentially the overbrights), and in the process of my research found that not even Adobe Camera Raw seems capable of doing so within After Effects.

Here is an example of what I mean, using After Effects CC 2015 and Adobe Camera Raw 9.1.1:



I feel like I've tried every format and program under the sun (including Resolve) and still always come back to this same issue - Somewhere there is information being clipped.

Am I simply misunderstanding how RAW processing works? From everything I've read, it really sounds like all major productions have somehow solved this issue. I know that I can adjust the exposure in ACR manually to recover the highlight information I want, but for our workflow with thousands of sequences, manual adjustment simply isn't an option.

Thanks for anyone who can help shed some light on this!