Cinelog - True logspace conversion for DNG and CinemaDNG footage

[Andy600]

@bpv5P

I tried previously to answer your questions in an open and thoughtful manner (even offering you test conversions) yet you chose to take a very negative approach based only on your own assumptions with what I can only assume to be a mindset that is prejudiced against anything that is not open source. You also chose to question the integrity of 2 highly regarded members of this community with this: 

But... I have to ask: is @Danne and @hyalinejim doing some astroturfing for you?

Certainly not! I have never solicited, used or manipulated responses from users, deceptively or otherwise. @Danne, @hyalinjim and others can speak for themselves on this but no users have ever benefited, in any way, through endorsing what I do. Their comments and views are their own.

Over time, Cinelog has received numerous endorsements privately from users, some of whom are highly regarded industry professionals and known for being highly critical but I chose not to use these comments even though I could for marketing purposes for the reasons I stated in my previous reply to you.

I've noticed you're using some selling techniques (especially social proof and bandwagon bias).
There's nothing wrong trying to sell your products, but I don't like astroturfing and discourse manipulation, and as a open source community we should keep these things out of here.

I had to Google the marketing terms you used.

I agree with the last part of your statement (above) however, in the context of your post, there is very obvious implication of dishonesty directed towards myself, Cinelog and the users you mentioned previously that I strongly deny and take issue with. I am not knowingly using any selling techniques. I do not make spurious claims and I am very open with my answers.

Magic Lantern is an open source project but using it or contributing to these forums has never precluded anyone from discussing or recommending commercial applications or products here except where those products have violated Magic Lantern's licensing or the terms and conditions of this message board.  The fact that the vast majority of users do use commercial software is evident.

So, if you don't need to achieve the exact same colors between various cameras and don't need all the other stuff (luts and support), there's no advantages? Yes, it's our entire choise to feel we want/need it, but if there's no advantage, why would anyone waste money on it?

Sorry I don't quite understand your logic here. Why would you buy anything that you don't want? Aside from the things you mention (accuracy, luts, support etc i.e. some of the advantages) you should take another look at my previous reply and the responses of others as to why Cinelog is regarded as it is. In addition to that, it provides an effective scene-referred processing capability in an app (ACR) that is strictly display-referred and bypasses any requirement to use image-adaptive filters (that will cause flicker). That might sound like marketing spiel but it is factual.

Since your choise is to keep color conversion linear, your two points of improvement can me the luma curve and color precision. Most people here don't need that precision in color, so on the luma curve there's no better dynamic range preservation compared to alexa-log (the version implemented on MLVProducer, for example)?

What is your assumption about 'dynamic range preservation' and 'alexa-log' based on exactly?

Alexa Log-C is for encoding the 16bit DGA signal from the Alexa's sensor and not an efficient use of the space for transcoding 14bit MLV (and becomes increasingly detrimental to 12 or 10 bit MLV as it spreads code values too far apart and can increase the visibility of banding).

When it comes to transcoding, my choice (dictated by color science and best working practices) is to keep initial color rendering strict and color manipulation to an absolute minimum, deferring color decisions to later in the pipeline. Basically retaining the maximum latitude in a known colorspace.

You again mention 'alexa-log' but what is that exactly? MLVProducer is a great app that can be used for everything (and there are several others too) but there are quality differences and often issues between raw video debayering and encoding with such apps compared to their commercial counterparts else why would those exist and why would people in their millions purchase them? You might answer with another one of the marketing terms I looked up 'herd mentality' but I know quite a few artists, film makers, colorists and developers who might take offence at such a suggestion as they opt to use commercial tools simply because they get the job done without compromises. The free tools on offer often have short comings and, as I described in my last reply, the limitations in open source raw libraries can restrict or limit development.

I'm not detracting from any OS app developer because I know they can be as dedicated as commercial devs and, if you support them, and their tools are good enough for you then who is going to argue with that? Certainly not me.

Again, don't get me wrong, I'm not trying to prejudice you or anyone possibly working with/for you. It's just that, if your product has no advantage over a free/open implementation, I think no one should buy it. I can think in a recent example like this: corporations were selling certificate authority for many time; "Let's Encrypt" implement a free/open implementation doing the exact (or better) same thing. Now everyone is going to Let's Encrypt. The same should happen with any product that does not do it's job. Contrary to what marketers say, quality is very important. You can do money with basically anything, but not everything keep itself on top of others if it has no advantages over these other alternatives.

I have stated quite clearly what the advantages are and the added value that comes with Cinelog. If you don't value that then simply don't buy it. 

Regardless of your initial statement above, you seem to have a jaded view towards what I offer but Cinelog is not 'Let's Encrypt' and there is no 'fake news' mentality at work here. Regardless of your insinuations I am perfectly happy to respond here to your questions and will always answer as clearly as I can, within the limits of protecting Cinelog IP. However, if you again choose to imply dishonesty or question the integrity of myself or other users without foundation, I will simply ignore you.

[DanHaag]

Andy600 keeps contributing detailed input about color science, working with ML raw video in post production and shares his ideas for possible improvements of the MLV format. He doesn't only try to sell a product, he shares a hell lot of crucial technical input no one else figured out or at least shared to that degree and detail with the community. No matter if you bought or used any of his products or none, he's helped a lot of users in the past with color correction/grading workflow problems and understanding the concept of color spaces etc. - If you ask him "What's the best way to grade my MLV files without Cinelog", be sure Andy gives you the best answer possible no matter what. He has proven this time and time again on this forum.

[Danne]

it provides an effective scene-referred processing capability in an app (ACR) that is strictly display-referred and bypasses any requirement to use image-adaptive filters (that will cause flicker). That might sound like marketing spiel but it is factual.

Yup. Confirmed.

@bpv5P
My answer before was only meant as a quick tip, nothing else. You seemed to need one.
By the way. Here is an open source tip to get you towards scene referred in acr(check link below). You will still have to build the math based log(and put into the dcp file) and account for color but it is a start.
One reason I started poking around with this is actually based on amasement of dynamic range I spotted from cinelog dcp examples.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg172443#msg172443

[Andy600]

@Danne - Linear in DNGPE only removes the tone curve so it's still producing a display-referred image and can only work display-referred. Try it with a very high DR image and you'll get clipping.

A lot of calculations are needed to get ACR to produce Cinelog-C colorspace for any particular camera (every camera is different) and it simply can't be done with DNGPE because the matrix controls will always snap relative to the internal processing space (Cinelog-C is outside of this) and the curve points are very limited with relatively coarse interpolation so transformed RGB values are not precise enough for an accurate log conversion. The Cinelog curve still uses ACR's interpolation but applied to 4096 control points.

Pulling highlights down with the Parametric curve in DNGPE is really just faking highlight recovery with no real gains i.e. once you render with those settings you are just getting a flattened image with compromised highlights because you have stretched and/or compressed the spread of code values with no mathematical way to get them back. i.e. there will be large differences in the amount of information contained in one stop compared to the next. Some stops will have more information and some much less information than required - this ultimately leads to less latitude but more chance of banding and other artifacts.

Generally speaking, a log conversion will spread the code values more efficiently and equally between all stops and because you know the CVs input and transformed values you can put them back with the inverse, anti-log formula - thus scene-linear (relative to light).

With Cinelog profiles a known RGB value can be input, transformed with the profile and output in AE then reverted to the input value (relative to the CCT) using the math built into the inverse transforms. The gamut transform is calculated dynamically and relative to the user selected white balance settings i.e. it is recalculated continually as you move the white balance and tint sliders making it more accurate for the chosen white balance than a fixed CCT plus I also use ARRI's recommended chromatic adaptation (CAT02) not Bradford for the calculations. Beyond that I can't divulge anything else

[Danne]

Generally speaking, a log conversion will spread the code values more efficiently and equally between all stops and because you know the CVs input and transformed values

You can fit in 4096 points into the dcp profile which in itself is pretty neat. I never seen any dcp profiles from adobe using this. Was it ever intended for this? Now how to get hold of that conversion chart. Would be pretty neat to have lets say cineon dcp in acr 

[Andy600]

You can fit in 4096 points into the dcp profile which in itself is pretty neat. I never seen any dcp profiles from adobe using this. Was it ever intended for this?

No. Check out the DNG SDK for the basics. There are also some undocumented and often illogical things to ACR that dictate what you can and can't do with profiles.

Now how to get hold of that conversion chart.

There is no conversion chart as such only math. I started out with spread sheets then CTL then custom Python scripts but there is still a degree of manual intervention required for building and testing each profile.

Would be pretty neat to have lets say cineon dcp in acr 

Cinelog-C is Cineon in terms of gamma already but Cineon without the gamut mapping of Cinelog-C can still clip color. I know because I have extensively modeled, tried and tested each and every log curve and full log colorspaces in ACR.

[Danne]

Cinelog-C is Cineon in terms of gamma already but Cineon without the gamut mapping of Cinelog-C can still clip color. I know because I have extensively modeled, tried and tested each and every log curve and full log colorspaces in ACR.

Does it mean that cineon gamma acts different from applied as a 1D lut in a nle as opposed to applying the same gamma curve through acr? Color is affected differently/clipped?

[eyeland]

I Just bought the bundle but can't get started since the user guide is in the membership area of the site and I get some weird registration error.
Wrote a support ticket earlier today.
Getting slightly impatient as I can't get started on the project that I bought it for
Would anyone care to send me a link or a copy of the userguide for Resolve while I wait?
I know that it is a pretty straightforward workflow, just wanna make sure that I get everything right from the start.
Much obliged/ Rune

[Kharak]

Try PM'ing Andy600 here on ml forums, that should get his attention.

[eyeland]

Cheers

[allemyr]

Hi,

Bought Cinelog 4 years ago Never included it in my workflow, don't know why, didn't go that way.

Just started reading again about it a while back to see if this workflow could save me some space when archieve footage. Recently started to work with a UHD workflow with DNxHR HQ as an end product. Realised that there is no 10 bit codec from Avid only 8 or 12 bit. DNxHR HQX in UHD takes 83mb/s and it's 12-bit similar to what a 16-bit CDNG files takes up at 1080p.

"Transcoding your DNG or Cinema DNG footage to Cinelog-C in an intermediate 4:4:2 or 4:4:4 (recommended)
video format such as ProRes or DNxHD offers not only significant storage savings but enables you to archive your footage"
Above, from Cinelog

Is it any other advantages of working with Cinelog other then saving space while the footage is in computer?

Didn't think I will continue to use this terrific 5D3 with ML camera for 4 years when I started using it, will change when a a camera features of the FS7 (also released 4 years ago) is availible in a Canon/Sony/Panasonic "DSLR" house. GH5 is very close but the 60 fps in good resolution isn't 10 bit.

What archive footage codec do you all use with Cinelog? 8, 10 or 12 bit?

Best Regards

[Kharak]

I render to Cineform 12 bit in cinelog. You save a little space comparred to uncompressed CinemaDNG but i'd imagine you would save more space converting your mlv's to lossless compressed cinemadng or archive the lossless mlv's (if you recorded lossless). I have not looked at the numbers, but i think that if lets say you have 1GB Lossless MLV, uncompress it to CinemaDng (roughly 1.4 gb avg) and then render to cineform 12 bit, i think the cineform file would be more than 1GB or roughly around that. Maybe a little less than the lossless mlv.

There is also cineform 10 bit YUV, takes up a lot less space than 12 bit RGBA.

[masc]

... or archive the lossless mlv's (if you recorded lossless).

You can convert your normal MLV's to lossless MLV's via MLVApp - saves space and you don't loose anything! (okay... you loose some time for converting)

[tigerbengal2009]

I love Cinelog-c , but  the time I spent importing to AE then render every single shot is not fun....Can someone give me an idea how this process can be in some way improved, I don't mind to import one by one of the shots, but creating the timelines and  render them is the part   that is time consuming and boring. I have been looking around, found the great  smartimport2 script, but that does not work with the latest AE only with the original CC 12.2, which I don't have. How do you guys process your shots? workflow?....Andy600 , DeafEyeJedi any advice??? Thank you in advance.

[Kharak]

By Timeline, i suppose you mean Compositions. The AE Cinelog route requires some manual labour, but there are ways of speeding it up, with the help of tricks and familarizing yourself with Keyshortcuts. And turning yourself in to a robot for 15-30 mins.

First of, if you are Upscaling in AE or not, this method will speed up the process significantly, but especially if you are upscaling. To speed up the upscaling process significantly, it requires that you record mlv with audio, so that you have .wav files in the folder. If not you will have to go by each comp and ctrl+k for comp settings and setting the preferred resolution for every comp.

Step 1. Create a new empty composition from scratch with the resolution and framerate you want, just call it 'Settings', not from any file in the Project. If you create composition from a sequence. All subsequent compositions will be in the original resolution..

Step 2. Multi import all dng sequences and .wav files from sequence folders, i recommend you first import the .wav file then the sequence, set ACR setting, WB, Cinelog, exposure etc.

Step 3. You have imported all files. Select all .wav files, make sure you select from top to bottom to keep the shot order, right-click and click 'New comp from selection'. AE will make a composition for each .wav file with the setting  you made in the original composition e.g. 2560x1440 for 2k upscale and preferred framerate. This will also correctly name each comp as the shot name e.g. M20-1645

Step 4. Manual labour starts, all your sequences should be in their right order in the project window. Click 'type' if you want to group sequences together. Now drag and drop the sequence to the composition on top of the .wav, go through all compositions, repeating the drag and drop, i find it fastest that you only do the drag and drop and not start fitting the comp to frame or anything else, this, to keep the work repetitive and fast.

Step 5. All sequences and sound files are in their respected compositions. Go through each composition, select the sequence and hit Ctrl+alt+F to fit frame to comp size and then click the bicubic upscale next to the sequence in each comp select next composition and repeat. Make sure you upscale to the right Ratio, otherwise Ctrl+alt+F will stretch it to fit. If you are not sure of the Ratio, because your recorded in a weird resolution or for other reasons, you can hit ctrl+alt+shift+H to 'Stretch to Width' and you will see a black border or the frame extends over the Composition.

Step 6. Apply adjustment layers for NR, Opencolor IO etc.

Let me know how it goes.

[PaulHarwood856]

I love Cinelog-c , but  the time I spent importing to AE then render every single shot is not fun....Can someone give me an idea how this process can be in some way improved, I don't mind to import one by one of the shots, but creating the timelines and  render them is the part   that is time consuming and boring. I have been looking around, found the great  smartimport2 script, but that does not work with the latest AE only with the original CC 12.2, which I don't have. How do you guys process your shots? workflow?....Andy600 , DeafEyeJedi any advice??? Thank you in advance.

Use the Smart Import 2 script for Adobe After Effects. It saves me so much time, and I think you will enjoy. If you have any questions please let me know. Here is a link: https://www.magiclantern.fm/forum/index.php?topic=13354.0

- Paul

[allemyr]

I render to Cineform 12 bit in cinelog. You save a little space comparred to uncompressed CinemaDNG but i'd imagine you would save more space converting your mlv's to lossless compressed cinemadng or archive the lossless mlv's (if you recorded lossless). I have not looked at the numbers, but i think that if lets say you have 1GB Lossless MLV, uncompress it to CinemaDng (roughly 1.4 gb avg) and then render to cineform 12 bit, i think the cineform file would be more than 1GB or roughly around that. Maybe a little less than the lossless mlv.

There is also cineform 10 bit YUV, takes up a lot less space than 12 bit RGBA.

You can convert your normal MLV's to lossless MLV's via MLVApp - saves space and you don't loose anything! (okay... you loose some time for converting)

Hi, late reply here sry. Thanks for your answer. Yes a good one would be to compress MLV or the CDNG a bit so its still 14 bit, something like ProresRAW or something just for storage in computer.

I always upscale my footage to 4k, mostly because the bitrate is more then the resolution increase on Youtube/Vimeo, so the quality is much better in 4k/UHD. That's why Cinelog-C doesn't suite me at all.

My workflow will still continue that it is but I will take your advice and look at ways to compress it a bit, something like ProresRAW or more like Avid DNxHR equivalent would be terrific.
Since I think the connection  with the XML workflow between Premiere Pro and Davinci Resolve, I started to do everything in Davinci Resolve Studio with edit and grade, quite sweet actually.

[tigerbengal2009]

By Timeline, i suppose you mean Compositions. The AE Cinelog route requires some manual labour, but there are ways of speeding it up, with the help of tricks and familarizing yourself with Keyshortcuts. And turning yourself in to a robot for 15-30 mins.

First of, if you are Upscaling in AE or not, this method will speed up the process significantly, but especially if you are upscaling. To speed up the upscaling process significantly, it requires that you record mlv with audio, so that you have .wav files in the folder. If not you will have to go by each comp and ctrl+k for comp settings and setting the preferred resolution for every comp.

Step 1. Create a new empty composition from scratch with the resolution and framerate you want, just call it 'Settings', not from any file in the Project. If you create composition from a sequence. All subsequent compositions will be in the original resolution..

Step 2. Multi import all dng sequences and .wav files from sequence folders, i recommend you first import the .wav file then the sequence, set ACR setting, WB, Cinelog, exposure etc.

Step 3. You have imported all files. Select all .wav files, make sure you select from top to bottom to keep the shot order, right-click and click 'New comp from selection'. AE will make a composition for each .wav file with the setting  you made in the original composition e.g. 2560x1440 for 2k upscale and preferred framerate. This will also correctly name each comp as the shot name e.g. M20-1645

Step 4. Manual labour starts, all your sequences should be in their right order in the project window. Click 'type' if you want to group sequences together. Now drag and drop the sequence to the composition on top of the .wav, go through all compositions, repeating the drag and drop, i find it fastest that you only do the drag and drop and not start fitting the comp to frame or anything else, this, to keep the work repetitive and fast.

Step 5. All sequences and sound files are in their respected compositions. Go through each composition, select the sequence and hit Ctrl+alt+F to fit frame to comp size and then click the bicubic upscale next to the sequence in each comp select next composition and repeat. Make sure you upscale to the right Ratio, otherwise Ctrl+alt+F will stretch it to fit. If you are not sure of the Ratio, because your recorded in a weird resolution or for other reasons, you can hit ctrl+alt+shift+H to 'Stretch to Width' and you will see a black border or the frame extends over the Composition.

Step 6. Apply adjustment layers for NR, Opencolor IO etc.

Let me know how it goes.

Hi Kharak thank you for your detail explanation and for taking the time to help me, in my case I am using 60fps footage so I have no sound with them, and in your procedure I have to had sound. I am trying to simplify , automate the procedure to get the masters only, I will color correct them later either in fcpx or in davinci resolve. I learnt some tricks from your post, really appreciate that.

[tigerbengal2009]

Use the Smart Import 2 script for Adobe After Effects. It saves me so much time, and I think you will enjoy. If you have any questions please let me know. Here is a link: https://www.magiclantern.fm/forum/index.php?topic=13354.0

- Paul

Hi Paul, if you read again my post I said I would like to use smartimport2, but it does not work with adobe after effect 2018, I don't get the timeline for each video, it does nothing. There is a new version? seems that smartimport2 only works with Adobe After Effect cc, it might work with cc 2014 according to some guys.
There is something else I could try to make it to work Paul, otherwise I am looking around trying to find After effect cc, Thanks for your advice

Al

[PaulHarwood856]

Hi Paul, if you read again my post I said I would like to use smartimport2, but it does not work with adobe after effect 2018, I don't get the timeline for each video, it does nothing. There is a new version? seems that smartimport2 only works with Adobe After Effect cc, it might work with cc 2014 according to some guys.
There is something else I could try to make it to work Paul, otherwise I am looking around trying to find After effect cc, Thanks for your advice

Al

Hey tigerbengal2009,

   I'm currently using Adobe After Effects CC 2018 (Version 15.0.1 - Build 73) and Smart Import 2 is working. This is the version that works with Miraizon ProRes 4444 XQ (Version 2) before Adobe stopped supporting 32 bit Quicktime.

   This video by DeafEyeJedi might help you: https://vimeo.com/130834452

    He's very helpful on this forum.

    He's on Mac.

   I'm going to be away for a few days, but I'll be glad to help you pinpoint what the issue is. You can try looking where you put the script for Smart Import 2, if it's in the right location. I currently have it working on Windows 10 in: Local Disk (C): / Program Files / Adobe / Adobe After Effects CC 2018 / Support Files / Scripts

   Also, try placing Smart Import as well as Smart Import 2 both in that location.

   We'll figure this out, not to worry!

- Paul Harwood

[ibrahim]

Is there a way to use cinelog-C and then apply a rec.709 lut directly in an external field monitor/recorder?
I intend to do video capturing to PC for live streaming through vMix or some other live streaming software.

[Tyronetheterrible]

Does anyone know how the Cinelog-C Full Colorspace Transform LUTs compare to DaVinci Resolve's Color Space Transform OFX in terms of quality and accuracy of transforming the image's colorspace?

I watched a tutorial recently that stated the Color Space Transform OFX plugin in Resolve had the advantage of controls such as Luminance Mapping to better accurately handle things such as highlight rolloff more so than Colorspace Transform LUTs would. However, given the quality product that are the Cinelog-C LUTs, I was uncertain if this would apply to them and if they do something different than other standard Colorspace Transform LUTs would.

[Kharak]

I don’t know the ins and outs, but Cinelog is/was supposedly mathemathically correct where Resolve was leaving some parts of transforms in the ballpark.. but last i checked on this was in Resolve 12, this might have  changed since then. And i never personally used Cinelog in resolve, only in Adobe.

I personally go with what just works snd have switched completely to Resolve. I cant say i miss Cinelog, though it was very nice when I did use it in AE.

[DeafEyeJedi]

...I cant say i miss Cinelog, though it was very nice when I did use it in AE.

That's the key... Adobe allows for us to use OpenColorIO (OCIO) via AE and that is a critical piece to be able to use OCIO as a plug-in when required whether we like the time consuming route or not. Ha.

Just bumping this thread to show how vital Cinelog-C still is up to this day... Keep your support coming along for @Andy600 in order to keep this hidden gem afloat!

Shot on 7D in 2496 × 1134, 2496x1198 12-bit RAW w 11-20 Tokina @ 11mm in 2.8 and here are the comparisons for shits & giggles (including the original dng's):

Adobe Standard


Cinelog-C


Cinelog-C Rec709


7D 2.5K DNG's: https://mega.nz/#F!O4sShSxY!c8dbOfhhCJtJvmLY-htgxA