Cinelog - True logspace conversion for DNG and CinemaDNG footage

Started by Andy600, January 24, 2014, 06:05:11 PM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

DeafEyeJedi

I understand you are a busy bee, @Andy600 - it's actually a good sign that you are because then that means there's another update coming soon!

FYI... I am off for the next 4 days and would like to get myself more familiar with your products on Cinelog-C -- especially the REC709 FM 3D LUTS... So good that I don't even need to use NR with ACR nor Neat Video... Good job on that, Andy!!!

Tho, Sometimes I often find myself having to lower down the opacity by 10-20% to smooth out the contrast a bit more within LUT Utility in FCPX.

No big deal though because I am learning everyday with how useful FCPX can be especially coming from FCP7.

However, it seems more realistic to try and use Adobe Dynamic Link to skip the rendering part of Cinelog-C in PR4444XQ from AE and go straight to PP in order to color grade & edit right off the bat but not sure how to bypass this so called bug w OCIO plugin for PP since I asked you about it and haven't heard back yet.

But then again I often tell myself that perhaps it's even more ideal to already have a set of PR4444XQ files in Cinelog-C and it'll always be flat & ready for me to use again as the starting point later on. At this setting I feel it can be just as good as archiving the original MLV's right? lol

TBH, I'm just basically sending a friendly reminder that I'll be standing by and hopefully hear back from you sooner rather than later.

So excuse my chirping!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

swordinhand

Hi everyone! Wow awesome thread. Read basically the entire 26 pages. Lots of great info. Thanks Andy600 for so much technical and practical advice. I am very close to making a purchase of cinelog, but had a few questions:

1. You keep mentioning setting correct white balance while shooting. Just wondering what this means and what settings I should select in camera and then in Resolve to achieve proper WB?

2. I have an x rite color checker chart and Kodak 18% grey card and would like to know the best practices when filming a scene?. I have been filming the chart for every single shot, is this necessary? Also, it appears that the camera calibration panel with the chart throws the purpose of your Luts off, no?  Just trying to understand how or if a color checker and grey card should be used in a production to post (resolve) workflow, using cinelog.

3. I've read and understand all of the concepts / pros you've outlined, regarding a raw to log intermediate workflow. But in a perfect world, where CPU / GPU and HDD capacity / speed are non issues, wouldn't archiving and grading the raw files / dngs be ideal? IF I don't need to grade and create intermediates of every shot and rather just spit out super rough proxies, edit (I'm the editor) and then only grade my edited timeline, then wouldn't I be saving time, by not setting white balance and exposure for every shot?  Does that make sense? If I screwed up white balance / exposure somehow, it's not a necessarily simple fix, right? Also, what if some years down the road, new technology or techniques inevitably arrive, I'd probably be better with a digital negative, then log "print"' for archive, no?

Anyway thanks for all of your amazing thoughts and helping us all.

DeafEyeJedi

Great questions @swordinhand!!!

Even though I am not Andy but I will do my best sharing from experience and unfortunately I cannot run Resolve on my mac's as of yet (keeps crashing due to unsupported graphic cards) so I use all the other Adobe CC apps along with FCPX which works well for me.

1) I do believe there are more than one ways to achieve this basically for me -- I set WB & Exposure on Camera and then once am in the process of achieving flat log's with Cinelog-C (I do this in AE with Smart Import 2 along with OpenColoroIO) & ACR (to set WB to 'custom' as well as applying Cinelog-C .dcp for certain camera models) which works really well in exporting beautiful flat looking ProRes4444XQ files and then from there you will always have a whole set of footage ready to use as starting point prior to grading in NLE.

2) Often times, I do use a Kodak 18% grey card (if I have it with me on set) but I then double check and confirm WB in ACR (even if it is correct -- I would still need to enable WB to "Custom") in order to get Cinelog-C to respond properly to the color-matrices within DNG's during PR4444XQ export all from AE.

3) I do think it may be more ideal to have every shot to have at least both correct WB/Exposure in order to save time like you said but wouldn't that also be the reason why most of us try our best to set the correct WB/Exposure prior to pressing record on camera, right? I'm sure you thought of this too but I also understand the fact that years down the road from now new techs are going to keep coming and revolve with time.

But in my gut it is telling me that MLV is going to be around for a very long time with good reasons. I often times tell myself to use the log print w Cinelog-C (PR4444HQ) and use them for archival purpose but then again I am not too fond of deleting digital negatives (MLV/DNG's) so if I really love the shot/take then I'd keep both for safety purpose but that's jm2c.

Until then I'll let @Andy600 give you his infamous deep technical and practical advices in which I look forward to as well!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Andy600

Sorry for my absence but I have a very heavy workload at the moment. I'll get around to answering your questions sometime in the next day or so.

Anyway, for anyone that doesn't know, DaVinci Resolve 12 beta is out https://www.blackmagicdesign.com/products/davinciresolve.

In my initial tests, it's a little more GPU/memory hungry than previous versions but has some fantastic features. The key feature that I was looking forward to is Resolve Color Management and it works great! - plus it also proves the absolute accuracy and resolution of Cinelog-C luts.

Although Resolve 12 does somewhat obsolete 'some' of what Cinelog-C (Resolve version) does, there are still those who will struggle to run V12 (especially on laptops and low-mid spec PCs) and others who wish to use our additional transforms (Film matrix, ACES2065-1, Slog, REDColor variants etc etc) and Cinelog Rec709 and Film Look Luts, so we will keep the product in development alongside Cinelog's forthcoming updates for ACR/AE/Premier Pro, Speedgrade and others. We have also been working hard on Magic Lantern color specific issues, ACES IDTs for Magic Lantern shooting cameras and non-raw input transforms for picture styles.

Part of why Cinelog-C luts have proved so popular is in how it transforms raw DNG images (debayered to BMD Film colorspace) to Alexa Log-C RGB primaries - a favorite colorspace among a lot of colorists.

With Resolve 12's new color management you can now transcode MLV footage to Alexa Log-C - this produces an identical signal to what you get when using Cinelog's BMD Film to Cinelog-C shaper lut in previous versions of Resolve.

You need to enable Color management in the Resolve control panel then select:

Input colorspace: Linear
Timeline colorspace: (doesn't really matter but leave this on Rec709)
Output colorspace: Alexa Log-C

Correctly exposed MLV footage will usually require a push of ~1-F-stop but is wholly dependent on your own shooting practices. Check the exposure by temporarily selecting Rec709 (2.4 gamma) as output colorspace before changing back to Alexa Log-C before rendering - we recommend ProRes 444, ProRes XQ or DNxHR 444 for digital intermediates (DI).

For transcoding jobs it is important to NOT alter curves, saturation etc etc - only change white balance if needed using the Camera Raw panel.

Once you have your rendered DI's you can import them for editing/grading. Just remember to select Alexa Log-C as the input colorspace (if using Resolve 12 color management for Rec709 output).

For any footage where you previously transcoded using our BMD Film to Cinelog-C luts you can select Alexa Log-C as input colorspace and use Resolve 12 color management to set a non-clipping Rec709 output transform - 2.4 gamma is standard ;)

If you wish to use Cinelog-C film look luts you can either use a non-color managed timeline or set input, timeline and output colorspaces to Alexa Log-C (although I still need to double check this).

I have a lot more testing to do but I'll post any tips, tricks, gotchas etc as and when I find them and be back here to answer any previous questions. Our updates will be out shortly.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

DanHaag

Hi Andy, thanks for the update. The Color Management tools are greyed out in my project settings and I can't find the exact option to turn them on (as you mentioned in your post). Any hints?  :)

Andy600

@DanHaag - You need to select 'YRGB Color Managed' in the main settings panel - hint: it's where you would also select ACES
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

DanHaag


DanHaag

A bit off-topic: What input would you suggest for GoPro-Cineform (shot Protune / native white balance)? I shoot a lot of interviews with ML raw and GoPro. Just figured out one can select different color inputs for individual clips/timelines etc. so I could set a different one for each camera.

Andy600

@DanHaag - Resolve 12 input transforms are VERY specific to the colorspace of the footage being transformed so it's not usually a good idea to use one that is not specific to the log curve/gamut of the camera. For the moment I would suggest choosing Linear and use the curves to get the image gamma and levels to approximately the same as your MLV footage.

ProTune log is quite different to any of the input colorspaces available in V12 and will still need a precision lut to accurately transform to a scene-referred colorspace - there is a ProTune to Cinelog-C lut in the next Cinelog-C release that makes GoPro Protune footage compatible with a Cinelog-C/Log-C timeline.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

DanHaag

Right on, I've been looking forward to your LUT ever since you've announced working on it! Best thing I've found online so far is this ProTune -> BMD conversion LUT and even though it doesn't often quite work as intended it gives my GoPro footage an overall better look. Have high hopes for your take on it, Andy! Much needed addition!!!  ;)


SAVideoman

Hi Andy,

I'm considering purchasing Cinelog but I have a couple of concerns. I've tried using MLV to record raw but I find it too unreliable as I get bad frames once in a while. But when I use raw_rec, it is rock solid. So I'd like to use Cinelog with raw_rec but I understand you don't recommend that or it just won't work or come up on the ACR selections. Even though I'm using raw_rec, would Cinelog be better than VisionLog? Is it possible to get a version of Cinelog that will work with raw_rec on a 5D3?

Another issue is the master file export using AE. I've read through this entire post and I believe you recommend using DNxHD 444 to export the master file in AE on a Windows 7 PC. However there are issues when doing that because when you select Trillions of colors while selecting the DNxHD 444 codec, it exports a corrupted file. But if you leave it at Millions of colors, everything goes well but I'm not sure if it's actually exporting a 10 bit DNxHD 444 file. You can see another post about this issue here: http://www.magiclantern.fm/forum/index.php?topic=15098.0
Do you have any recommendations on this?

Thanks!
Gary

swinxx

Hello savideoman,
You can buy mine, cause i dont use them.. @andy600: is it ok if i sell mine? How dowd that work? Do i need your agreement?

Andy600

@SAVideoman - We recommend MLV because of the deeper metadata it embeds vs .raw but it's all down to how the DNG files are extracted from the container. Apps like raw2cdng and MLVFS will write new color related metadata which is important if you use a non-ACR based raw workflow (i.e. Resolve, Nuke, Fusion, Baselight etc etc).

If you are using ACR it will not matter as the profiles ignore embedded matrix data and use their own. At present the ACR version expects a camera specific tag but some converters embed a generic 'Canikon' camera model tag which means any camera specific profiles will be unavailable in ACR (including Cinelog) - The forthcoming update addresses this with an additional set of profiles for 'Canikon' tagged DNG files but you should only install the 'Canikon' profile(s) that are specific to your camera - if not you will have lots of profile options but only one with the correct color meta and lots of room for mistakes.

Incorrect or missing metadata can usually be re-written using Exiftool but this can be time consuming process. If the raw2dng app only embeds the default single matrix (5D mark III) it will only produce good color if the footage was shot on a 5D III and more specifically, in daylight. The additional matrices embedded by raw2cdng, MLVSF correctly map XYZ colorspace to your camera sensors native RGB (used for accurate white balance) and the forward matrices are there to 'calibrate' the white balanced raw color - but even with all 4 matrices you can still get out of gamut color resulting in clipping, hue twists and other nasties - this is where camera profiles or luts come into play.

re: DNxHD - DNxHR is better. I think you can now download the codec pack from Avid.

re: The Trillions of colors issue - yes, correct (I didn't update the post, sorry). You would select millions of colors - this will export a 10bit file with 444 chroma subsampling. You can check the file using Mediainfo or similar apps.

Feel free to contract me using the contact page of our website if you have any further questions - whatever the issue we will do our best to help.

@swinxx - Sorry, Cinelog is non-transferable for obvious reasons. Of course, there is nothing to stop you selling your pack but the buyer would not be registered or receive updates - I would urge you to hold onto your copy/registration as the next update is significant, plus we have software products in development where you will receive a customer discount plus some 3rd party discounts we are negotiating for Cinelog users.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

dubzeebass


Andy600

@dubzeebass - The short answer is yes. I'll be in touch shortly.

It's been a big part of our work over the past couple of months but we are still a little way off from being able to give you exactly what you need - I will need a couple more things from you for this (I'll email you).

Actually, your C100 request and our subsequent work has exposed some very useful things about Canon DSLR and C-series internal color science that we suspected but didn't fully understand and this will eventually benefit ML - we may even be able to produce a true Canon log picture style at some point, not some psuedo, log-like facsimile.

I'm going to get technical again (sorry) and start talking about ACES and matching to other C-series cameras because it may help you and others to understand what we have to do.

The shots you provided were unfortunately over/under exposed for the exacting requirements of shooting Canon Log. Correct exposure is critical to preserve dynamic range and white balancing must be performed to a proper reference as you are baking this into the video file and don't have the luxury we have with raw images.

FYI - There is a very specific procedure for obtaining accurate exposure when shooting Canon log on any C-series camera, as explained here by Jem Schofield - http://www.learn.usa.canon.com/galleries/galleries/tutorials/eos_c300_tutorials_gallery.shtml. If you follow this you will get perfect exposure every time, if you don't you will always have issues.

Unfortunately, we don't have a C100 MkII to obtain a controlled lab reference (I'm still trying to source a rental here in Latvia), I had to base all the calculations on the color and white balance coefficients already available for the Mk1 (as per the official Canon ACES IDT) and some correctly exposed and balanced C100 and C100 MkII original stock footage to fully understand the MKII's colorspace.

Fortunately, the difference in color appearance between the MkI (and C300 MkI) and MkII can be neutralized by scaling RGB channels (basically changing white balance offsets) to achieve visually identical color with low DeltaE -  we can assume the remaining difference is mainly down to noise (improved sensor in the MkII) but you will see pretty much identical color - this solved the first part of the problem and allowed us to build a MkII ACES IDT - basically we did Canon's job for them.

The next part is where things get really complex - matching color and levels to ML raw footage.

At present, MLV DNG files (from Raw2CDNG and MLVFS) have their color assigned by forward matrices which are linear and target an Adobe reference (I want to change this to a common, easily obtainable reference such as a Colorchecker) - the color in the C series cameras have (as far as we can tell) a non-linear color profile applied somewhere in the signal path before the IRE scaled lin-to-log transform happens (the IRE scaling is what makes Canon log and Cineon technically incompatible BTW so we had to factor this into our work).

This color profile stage essentially means an unknown amount of color values are mapped individually to remove/lessen any out-of-gamut issues or for aesthetic reasons (possibly to make it look more film like/cinematic) - this helps with things like spikes in the spectral emissions of certain light sources (i.e. LEDs) so already, color matching these non-linear to linear assigned sets of RGB values is not a simple procedure - it requires a lut somewhere in the pipeline.

Without the keys to Canon's lab, it would be impossible to exactly invert the color mapping that happens in the C100 MkII to obtain a true scene-referred linear signal (as we can do with raw footage) so we need to match MLV footage to the MkII (or vice versa) with some 'educated' guess work based on what we know about Canon color science and available data (i.e. the C100 MkI ACES IDT).

So, using our MkII ACES IDT we can be reasonably confident that we can transform C-Log MkII footage to scene-referred linear RGB values and because the DNG files can easily be debayered to scene referred values we can then color match them. The significant color matching part should ideally be handled by a trivial 3x3 matrix (as this will not introduce artifacts or errors) but to do any of this we need an very accurate chart reference shot from the C100 that is as close to perfect white balance and exposure as possible (and preferably shot with a very good lens) to enable us to calculate the required RGB offsets. Any remaining inconsistencies in reference color values can then be handled by a 3D color correction lut containing float values before a final transform to an output referred or log colorspace i.e. a signal that can be rendered or graded.

That covers a 'theoretical' color matching procedure for 2 cameras but we have to remember that even though we can obtain scene-referred values from the MLV DNG files it will not necessarily be reference-accurate or look nice when viewed in a display colorspace (i.e. sRGB/Rec709 etc if we only use the XYZ matrix and a 1D transfer function). The DNG files should also be color matched to a known reference. Ideally this would be handled by the the forward matrix tags but no sensor is perfect and a forward matrix transform can never fully match reference levels - infact it may make things worse plus forward matrices do not form part of the CinemaDNG spec so you can't guarantee all apps will render the same values.

Just as the in-camera profile of the C-Series cameras 'tweak' color, we need to apply the same logic to raw files if we are to match them and this means matching both cameras to the same reference values (i.e. a color chart) and not match one camera to another. In practice this approach brings actual color matching of 2 different cameras 'perceptually' close to all but the untrained eye. The downside is that you will lose some of the camera's color aesthetic so it is always a trade-off.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

dubzeebass

Got it thanks. I can re-send you the files with IRE 32 as mid grey if you wish when I get home. Will that help?

DeafEyeJedi

I can also shoot some test footage for you from C100II per your request.

Just let me know which lens you prefer that suits your needs.

Even if it has to be a Zeiss...

5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dubzeebass

Yeah I've got a Zeiss 100mm makro that is probably the sharpest lens I've ever used at f4.

Andy600

@dubzeebass - Yes, that would be great. It's best done in daylight if you don't have controlled lighting and must be white balanced - I suggest exposure > white balance > re-check exposure and adjust if necessary. If you also have a greyscale chart it would allow me to pull a good reference level but I'll need the colorchecker shot too. Middle grey on the colorchecker is the 4th patch along from white (directly under yellow).

@DeafEyeJedi - If you can also do this it will help - a Zeiss prime would be ideal.

The colorchecker needs to be fairly large in the frame and focused. Ideally it should be face-on i.e. no perspective distortion if you can do it without casting a shadow. Also, try not to use ND.

Thanks :)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

cmccullum

Andy,
100% honest question. If I don't understand ANY of the technical stuff you've been explaining, would cinelog still be beneficial to me?

Andy600

@cmccullum - Probably but it depends what you want to do and what app(s) you use (Resolve, After Effects etc etc).

Cinelog is basically a colorspace management toolset and beneficial to transcoding your DNG files to a smaller but very high quality log video file. It also comes with some look luts and basic guides and I'm available for advice on shooting and workflow practices. If you know what you want to achieve or have problems in your current workflow I'll give you an honest answer as to whether or not Cinelog will be of any use to you.

If you need anything explained in simpler terms just let me know and I'll try to help you understand everything as best I can.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

cmccullum

@Andy600 I've only just recently started exploring raw video with magic lantern, but I've done a few tests and been able to get some pretty cool results using only ACR. So far, I haven't introduced any flicker or other weird artifacts using the ACR sliders to adjust the images as I please. I guess where cinelog really interests me so far is color accuracy. I've only recently started learning about using scopes for correcting/grading, and I still don't REALLY understand what's going on there so most of the CC I've done has been by eye, and ends up looking different between displays (I'm assuming this is pretty much unavoidable).

I guess the short answer is that I don't really know what it is I'm looking for. Just looking to explore options, and the "proper" ways to work with these things, I from what I've read, I feel that using Cinelog might be a good way for me to do some hands on learning about a Proper raw/log workflow. I'm far from a professional colorist (I'm sure that goes without saying), but I think I do a pretty damn good job, and haven't really run into a situation in which I can't do something I'm trying to do. This is pretty all over the place, but I hope it sheds at least some light on where I'm coming from.

Also possibly worth noting, I haven't come across any "looks" (lut or otherwise) that I particularly care for, but I figure people use them, so there must be something to it and I'm trying to find the allure I suppose

Current Workflow:
mlv_rec
mlRawViewer -> DNG (I understand this will have to change)
Import DNG sequence into AE CS6
Correct/Grade/whatever with ACR
Export

Tumble

Hi Andy,

I'm having some issues regarding the histogram when trying to make Cinelog-C masters.

When adjusting project settings to:
3D Input Lookup Table: [TRANSFORM]_BMD_Film_to_Cinelog-C
3D Video Monitor Lookup Table: [REC709] Cinelog-C to Cinelog REC709 Full
1D Color Viewer Lookup Table: Use Video Monitor Selection
3D Color Viewer Lookup Table: Use Video Monitor Selection
1D Scopes Lookup Table: Use Video Monitor Selection
3D Scopes Lookup Table: Use Video Monitor Selection

The histogram signal is clamped between 128 and 896. What setting is doing this?

If I'm just using the [REC709] Cinelog-C to Cinelog REC709 Full in a node it works fine.(with 3D Video Monitor Lookup Table selection removed)



Andy600

@Tumble - have you got Resolve set to use Video levels? If so, change it to Data Levels

You can move the Cinelog-C to Cinelog Rec709 Full lut to a track node and bypass or delete it when you are ready to export the log masters. It will do the same job and is what you need to do if you use a monitor calibration lut in the Monitor Lut slot.

Bear in mind the Rec709 Full lut allows overbrights (values that extend beyond what Resolve can display) so it may look like highlights are being clipped on the scopes but once disabled the Cinelog-C log signal will fit nicely into 10bits - this particular Rec709 lut is for when you want to adjust highlight rolloff yourself. The other Rec709 luts have an s-curve to smooth rolloff and keep the signal within the display space - they are tone mapped to look something between Alexa 709 and Sony 709 curves.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com