ACES for Magic Lantern Raw Video

[Andy600]

As the DNG files produced by Magic Lantern Raw Video are predominantly used for moving images I would like to propose that Magic Lantern adopt ACES (Academy Color Encoding System).

ACES is an rapidly becoming the standard color science for motion picture and VFX pipelines and is aimed to standardizing color reproduction across cameras, regardless of manufacturer. It also offers the benefits of full Dynamic Range reproduction in the widest possible color gamut and accurate conversion to any colorspace (with an ODT) without needing to re-grade.

Incorporating ACES into Magic Lantern Raw Video will mean some minor changes to color matrix calculations in .raw, .mlv and probably raw2dng but beyond that, users should not notice an difference in their current workflows. The big benefits will come when using ML Video in an ACES pipeline as color reproduction will become both predictable and standardized. To fully utilize ACES we will likely need Input Device Transforms (IDT) for the cameras but these are relatively simple and (I suspect) we can base them temporarily on current knowledge deemed from DNG files, Canon IDTs and maybe Canon .ICC profiles.

OpenEXR will not directly be needed but may be the best option for DNG app devs who want to incorporate the highest quality OpenEXR output.

ACES and OpenEXR are both open source.

For more information on ACES:-

Presentation: http://www.oscars.org/science-technology/council/projects/pdf/ACESOverview.pdf
ACES: http://www.oscars.org/science-technology/council/projects/aces.html

Ampas/ACES sourcecode: https://github.com/ampas/aces-dev/tree/v0.1.1

ACES discussion group: https://groups.google.com/forum/#!forum/academyaces

OpenEXR: http://www.openexr.com/
OpenEXR sourcecode: http://www.openexr.com/downloads.html


Thoughts?

[Midphase]

I second this. Even though most people still don't know about ACES, it's about to become the new standard and it would make sense for ML raw to get in on the standard as soon as possible.

[reddeercity]

Yes I agree, I have been following ACES since the beginning , i'm members a the
Group. My only problem is getting the (IDT) for canon . That stop me ,
If I had that I could use the ACES.

[Andy600]

Resolve has IDT's for the 5D (2/3?) and 7D (I don't think these were developed by Canon either) and you can also download new IDT's for Canon's C cameras. I'm studying them. To accurately develop an IDT means some serious scientific measurements but I think there is enough of a foundation using current IDT's plus what can be derived from .ICC and raw images to approximate a useable IDT for ML raw video. I'm currently getting to grips with CTL which will be very useful for developing this.

We're working with raw data so it's completely possible to bypass Canon's input if they are unwilling and I'm sure Blackmagic will include a Magic Lantern Raw Video IDT in Resolve if we can supply them with one.

I'm pretty sure MLV already has capacity to do this. It just means recalculating the matrices to D55 with linear gamma which is what ACES uses. I think we'll need g3gg0's input

[Midphase]

To accurately develop an IDT means some serious scientific measurements but I think there is enough of a foundation using current IDT's plus what can be derived from .ICC and raw images to approximate a useable IDT for ML raw video.

Andy, it seems to me that you probably have some pretty good measurements already from your CINELog ML raw work. What can we do to help?

[Midphase]

Anyone who isn't sure what this is about, should watch this video which explains some of the ideas behind it:

[Andy600]

Most of our measurements to date relate to color differences between cameras and to help with making uniform LUTs for our DCP profiles using standard matrices but we are now working on custom matrix transformations as it's a superior, non destructive method compared to Lookup tables. We still have a ton of new testing to do but I'm sure some of this will be useful for making IDTs and I'm sure we'll even have a go at it ourselves. I'll make the data (well, some of it lol) available when I can and certainly the new color matrix info.

I just hope people aren't too attached to Canon color science because it's totally irrelevant and would be discarded for ACES . I guess, as ML is now modular we could always have ACES compatible and a RGB/Rec709 compatible raw modules but I don't know enough about the backend of ML (yet) to be sure

Incidentally, we're having some early success using Cinelog with the ARRI Alexa and Sony Raw IDTs in Resolve (ACES Log color science) so we're no far off.

[reddeercity]

Great work  Andy600,  I will then get more active with ACES
And use my email connection with the group to get info
Related to Raw data etc...
I have been waiting for a chance to use ACES workflow !

[Andy600]

@Midphase - good introduction video!

I'll also add that ACES is a vastly wider color gamut than humans can actually see and some of the typical problems we currently experience like certain colors (i.e. reds and greens) over-saturating and clipping (caused by linear increase in saturation levels), color crosstalk caused by less than ideal matrices and primary imbalances are far less of a problem with ACES. You really can push colors around and the ODT will take care of things.

We all seem to forget that Canon DSLR's are stills cameras and were never really intended for video hence the photography methodology behind the color science. ML Raw with ACES would gift us the opportunity to use these cameras in a cinematic way and elevate this already great software even higher!

[Lars Steenhoff]

Very interesting to have this!!

[eundd]

Would be very interested in this! Is there any progress made yet? Maybe it should be posted as a proposal on Bitbucket? https://bitbucket.org/hudson/magic-lantern/issues?status=new&status=open

[baldavenger]

The new Cinelog updates have provisions for an ACES workflow via both ACR and Davinci Resolve.

[Andy600]

We've also built IDT's for all MLV capable cameras but need to have another look at them since ACES 1.0 (ACES 2065-1) was just released. Once we're 100% happy with the results we will submit an application to the Academy to see if we can get them into the ACES source code.

In the meantime, as @baldavenger said - The Cinelog-C release has transforms to ACES Colorspace, ACES Log and ACES Proxy colorspace. Setting the transform to ACES will work as a bridge to ACES 1.0 - If you do want to output ACES files they must be .exr format because the colorspace is very big and needs a floating point container.

The actual MLV IDT's we built are described for 2 Illuminants and are camera specific. Whether or not this actually gets into ACES is another thing because ACES is centric to high-end professional cameras - but we will see

[AgentJJ]

I recently constructed a series of IDTs for three input Illuminants (D56, Tungsten and F3) for a set of ISOs on the Canon 650D for a Web Series ( JackAndCarmen.com ) .  The IDTs are constructed as 3x3 Affine Transformations (Matrix Form).  Chromatic Adaptation transformations were constructed as well. 

The transformations were applied via color transformation language (ampas' CTL) and sony's OCIO configs.  OCIO was used to construct 3D LUTs converting ACES to our calibrated monitors.  ODTs were handled in Davinci Resolve.  We originally tried the rec709 ODT, but ended up constructing our own LUT for ODT output (being piped through Davinci's internal RRT) for final web distribution (output to D65). 

It was a great experience and I recommend anyone to construct their own IDTs per their own camera, because I've found (and it's been shown) that even between cameras of the same model that visibly detectable differences in sensors occur.

I plan on posting a detailed overview of IDT construction in the near future for those interested.

[DeafEyeJedi]

I second that -- good to know and thanks to @Andy600 & @baldavenger for refreshing this thread!

[zachnfine]

I don't suppose anyone has one of these devices lying around:
http://www.image-engineering.de/index.php?option=com_content&view=article&id=452&Itemid=198

I'd love to have IDTs for the Canon mkIII and to try the ACES workflow in Resolve with DNG sequences captured using MLV, but I have neither reflective targets nor neato illuminating boxes at which to aim the camera. But it'd be neat to have profiles specific to the sensor.

[g3gg0]

okay never noticed this thread.
to be honest, i am a bit confused...

MLV, as we have it today is an open container format for
a) raw video frames that
b) contain images in mosaic'ed (bayer) pattern, which are
c) in their shape specific to canon's sensors and
d) every mosaic tile filters the wavelength of the incoming light with properties specific to the sensor

okay now a) usually means that the frames have several flaws that have to be corrected, like stripes or dead pixels etc.
this is highly camera specific if you want a perfect image. just remember the FPN or AF pixels.
this also means, we cannot change that data during writing as we do not have any free CPU resources for that.

point b) means that we have to debayer it before the image is accessible as a X*Y pixelmap with pixels each representing a tristimulis value at that orthograpic coordinate.

about c)... canon usually uses the same bayer pattern for all models. other cameras have different patterns, some vendors even use CFAs with different shapes and colors.

and d) as the bayer pixels canon uses is not the same on all models (i even remember some modely having two different greens)
also the CFAs vary in their filter properties from camera model to another, this is highly model specific.
we are providing the dcraw sourced color matrices that provide a suitable sensor->XYZ colorspace transformation along with every MLV file.


if i understood ACES correctly, it describes how to separate the necessary steps during production from each other and having e.g. an
input transformator that delivers tristimulus values for a standardized color domain (extending xyY beyond its limits) on which all other modules continue their work.

just in case i understood it right, what is different to what we have now in MLV?
it provides raw data, encoded in canon style, to be stripe-fixed, to be debayered properly plus the transformation matrix to get standardized XYZ (which maps 1:1 to xyY)
to my understanding, colors are definitely the least problem with MLV

the only things to improve regarding colors, is
a) finding better sensor->XYZ matrices (no MLV needed, everyone could do it with CR2)
b) adding an output format that tools accept and which allows the MLV/RAW colorspace to develop how it deserves (or better: creating an IDT that directly reads MLV.....)

BR,
g3gg0

[Andy600]

Hi @g3gg0 - My original post was before MLV had matured to the level that it is now. Getting MLV into ACES colorspace is as simple as transforming from CIE XYZ to ACES 2065-1 chromaticity (D60 white/scene referred linear). The only major issue was the color matrices but I recently gleaned the dual illuminant Adobe Standard matrices for every MLRaw camera as the Adobe matrices do a much better job than the DCraw matrix (this also helps when using white balance defaults in most apps - ACR overrides these values with whatever is set in the camera profile so to keep it simple we use the same ones).

I think there is a degree of fine tuning possible when it comes to color matching but it's a huge and expensive technical job to do this for every camera in lab conditions. Adobe Standard is well known and close enough - even the main manufacturers don't stick rigidly to the ACES specs so there is always some variance between cameras (which kind of defeats the stated aim of ACES). 

Anyway, the Adobe Standard matrices are now embedded by MLVFS and raw2cdng but it would be useful if MLV can embed these matrices by default so that other converter apps can simply copy the default metadata (some apps are still embedding Canikon generic or fixed 5D Mark III matrices for any camera). It's possible to expand the meta content with more matrices for other illuminants but not many manufacturers are going beyond simple Tungsten and Daylight variants for their IDTs.

The MLVFS commit is here: https://bitbucket.org/dmilligan/mlvfs/src/d961f752f326856081a42f294e22ffca301df875/mlvfs/dng.c#cl-54

[DeafEyeJedi]

Great read and good to know!

[g3gg0]

ah okay, good to know.
ive read that you tried getting matrices, but never realized that it made its way to the DNG converters.
hoped that the converters use the matrix provided by the MLV files, and the matrices im ML core get their update.

dont you think its better to trust the embedded matris in the first place and make a pull request, updating ML's internal matrices?

the matrices you posted, into which colorspace do they convert? are these XYZ->bayer mtrices?

[chmee]

mlvfs dng.c -> https://bitbucket.org/dmilligan/mlvfs/src/078aca39af8c65f2825b69e2e12a6bbb0e926a86/mlvfs/dng.c?at=master
raw2cdng io.cs -> https://bitbucket.org/chmee/raw2cdng/src/e62afbaf5fc524f884b813e1f2d77510b38c4b39/csharp2010_code/raw2cdng_v2/io.cs?at=master

regards chmee

[Andy600]

@g3gg0 - The Adobe Standard matrices that I gleaned from Adobe's DNG converter are CIE XYZ to Camera RGB for 2x illuminants (A and D65). They produce fairly consistent color across all cameras.

I believe the bayer pattern order is described elsewhere in another tag so that the raw reader knows how to apply the transforms but the matrices are standard order for all DNG readers.

DNG readers can use a single color matrix and this can even be described with an unknown illuminant using the 'unknown 38168' tag but colors will be off and unless the white balance was spot on when shooting it takes a lot of work to find neutral white point - and sometimes the workspace/colorspace (and/or white balance controls) in certain apps are still too small to find the required offsets.

A good example is the Sony FS700 recorded as Sony FSRaw on the Convergent Design Odyssey 7Q. This embeds a single color matrix with an unknown illuminant tag and the result is usually VERY wrong reds. I built a camera profile for the FS700 DNGs with better matrices and calibrated everything to a Macbeth chart so FS users who may not have white balanced correctly when shooting can have a better starting point.

Some users (even professionals) think this is 'baked-in' white balance - this is of course nonsense as the pixel data is still raw and can be manipulated. It's just that the embedded metadata isn't described correctly for a raw reader to make any real sense of and not sufficient for the transforms plus, as there is only a single (unknown) illuminant there is no information for matrix interpolation to work correctly - and that's on a professional system  - Us ML users are more lucky than we realize

[g3gg0]

yeah and all of these can get embedded into ML core then
or would there be some issues with copyright etc?
(ML core has matrices for all models, depending for which model it was compiled)

this would help not only MLV or RAW users, but also DNG through the silent.mo

[Andy600]

Yes! I think we will also see some of the 'color' issues that users post about being resolved or improved. Incorrect matrices can manifest in colorcasts, weird green tints etc  - sound familiar? 

It's no more of a copyright issue than the current ones. DCRaw matrices came from Adobe but they're a little outdated now plus DNG is an open format that requires matrices. These are just the same as you get with the DNG converter and it's all part of the SDK AFAIK.

[DFM]

Sorry but the Adobe DNG SDK does not include matrices, just the C++ libraries required to interact with a DNG file object. All support files supplied with installations of Adobe Camera Raw or the Adobe DNG Converter utility are copyrighted and subject to Adobe's standard end user license agreement. They may not be extracted for standalone use or incorporated into any third party product.

[dmilligan]

That makes no sense. If the Adobe DNG converter adds some metadata to a DNG file (the camera matrices), you are saying that a DNG reader is not allowed to extract and use that information? I thought the whole point of putting metadata in a DNG file was so that a third party could extract and use said metadata.

Also, if I run an image through Adobe DNG converter, then suddenly part of that file becomes the IP of Adobe? That also make no sense.

[Andy600]

...and puts DC Raw and a host of other apps/libraries in breach of copyright  These matrices simply describe the transform from camera RGB values to CIE XYZ. So what you are effectively saying is that you can only convert to a DNG with no matrices and can only use the Adobe DNG Converter now  , and that any camera that Adobe labs have defined a set of matrices for can't be used, written, or read with anything other than Adobe apps? Really?  Ok, we'll rebuild them using Dx0 labs published data - it would be funny if the matrices compute to the same values.

[DeafEyeJedi]

Hmmm... Point taken guys!

Could you please help clarify what,re referring to earlier @DFM?

Otherwise it just doesn't make any sense...

[reddeercity]

Sounds like back to the "embedded" Camera matrices from ML.
Personal I always choose the embed matrices when working with raw . I feel the embedded matrices produces a better Image.
With less noise etc.... I don't think embedding all this extra info is really worth the effort , There is no Short cut in producing a IDT for A.C.E.S

[Andy600]

@reddeercity - according to DFM that too would breach copyright because the original matrices that  ML used for DNGs were based on DCRaw matrices which also came from Adobe.

As I've repeatedly said, these matrices are simply telling the DNG reader how to translate sensor RGB tristimulus values to scaled CIE XYZ which is the basic connecting colorspace for most (if not all) raw readers. Canon do it with ICC and Adobe do it with DNG tags. It's basic color science and is a requisite for the DNG format to work. The original embedded matrices are for a single illuminant (based on half of the the 5D Mark III's set). To make DNG's color and white balance offsets behave well in any app other than Adobe Camera Raw requires you to either nail WB every time when shooting or embed dual illuminant matrices which can give a hint to the DNG reader so that it can interpolate properly between light sources and enable use of rudimentary preset white balance settings. basically it's a fundamental requirement of the format and the only reason you won't notice this in Adobe Camera Raw is because any embedded camera matrices are overridden by the camera profile you choose and all of them use these matrices.

Re: IDT's if these matrices are embedded the basic CinemaDNG IDT will work fine - as i mentioned earlier in the thread, the aim of ACES is to define a universal, camera agnostic look and even the big boys are having trouble doing this because they all want their cameras to look good! ARRI seem to be the only ones taking it seriously. Making IDT's for something like Cinestyle is another matter - that is a world of pain!

[reddeercity]

@ Andy600 , I do see your Point to all this.
Just a thought have you look at the IDT for the Cinema EOS Cameras ? http://www.usa.canon.com/cusa/professional/products/professional_cameras/cinema_eos_cameras/eos_c500#DriversAndSoftware
I see they now have IDT for C100, C300, C500. with Test EXR & DPX images, would this be helpful ?

[a1ex]

Incorrect matrices can manifest in colorcasts, weird green tints etc  - sound familiar? 

Here, those "weird green tints" are usually caused by wrong black level, not wrong matrices.

[chmee]

@DFM
please leave us an "official" statement about using these colormatrices. they are taken from the dng's, so i see also no point of breaching copyright or similar.

[DFM]

There's a difference between editing a DNG file that already has matrix information embedded within it, and copying the Adobe Standard matrix supplied with the Camera Raw/DNG Converter software so you can use it to create other DNG files (or for that matter, mentioning the term "Adobe ###" in any third party product).

The EULA for the DNG SDK is different to that for Adobe's desktop software, it permits any use of the SDK's source code provided copyright notices are maintained - and that's why there are no matrices, lens profiles, etc. in the package. The EULAs for Camera Raw and DNG Converter are the standard versions which forbid any extraction, decompiling or reuse (clause 4.5b and 5.2).

I can't give you an exemption from any of this; you would need to negotiate terms with Adobe's legal department. In the first instance I would advise posting a detailed request on the DNG Community Forum (including explanations of the license terms for ML, where and how the matrices would be used, what you intend to call them, etc.) and I'll make sure it gets read by the right people.

[Andy600]

@DFM - there is no exemption required here because there is no decompiling, reuse, porting, modification, reverse engineering or any other process here relating to Adobe software that violates the terms of Adobe's EULA. The software is not touched. There are many apps that make extensive use of the DCRaw libraries and that library features older versions of Adobe Standard matrices. The matrices are simply telling the raw reader how to transform a specific camera's sensor RGB values (under 2 illuminants) to CIE XYZ colorspace with Bradford chromatic adaptation. If we remeasure/recalculate we will come up with the same numbers.

Also, (I may be totally wrong on this but) I think the recorded DNG files are produced in-camera. Then wrapped in an MLV container for convenience with some of the header/footer info stripped for speed reasons.

If you run your raw files through the DNG Converter it will embed these matrices in the DNG according to the camera model it identifies. Regarding copyright of Adobe trademarks, that's another matter and why some of us do follow the guidelines and not use Adobe logos plus add the registered trademark symbol when referring to Adobe products and post a full disclaimer.

@a1ex - Yes, in most cases it's mainly to do with incorrect black level but I have seen DNGs where the matrix and completely wrong white balance (when shooting) has caused a similar issue that may cause the user to start messing with the DNG black level. My point is that colorcast issues caused by white point offset will be less likely and point to DNG black level calibration being the problem. In something like DaVinci Resolve where there is no automatic method of white balance for raw images most users will find it easier to be able to use the white balance color temp presets as a neutral starting point.

@reddeercity - Thanks! I know about the Canon IDTs. I have them and quite a few others plus build routines, scripts and a ton of other documents/images etc for developing and testing ACES IDTs.

[dmilligan]

There's a difference between editing a DNG file that already has matrix information embedded within it, and copying the Adobe Standard matrix supplied with the Camera Raw/DNG Converter software so you can use it to create other DNG files

This is idiotic. By this logic I can't even use my operating system to make a copy of a DNG that has been through DNG converter, because I would be using it "to create other DNG files" with said matrix information embedded in it.

[DFM]

If we remeasure/recalculate we will come up with the same numbers.

And if you did create an independent set of values via your own measurements and called it "ML Standard", we wouldn't be having this conversation.

This is idiotic. By this logic I can't even use my operating system to make a copy of a DNG that has been through DNG converter, because I would be using it "to create other DNG files" with said matrix information embedded in it.

No, you're misunderstanding what I said. Adobe makes no ownership claim against any files you create with their products, but the algorithms used by Adobe software to create those files are copyrighted and/or patented, as are certain file formats (e.g. the structure of a PSD file is proprietary even though Adobe publishes the full specs). If you used Photoshop to create a panorama, you still own the rights to the stitched image - but you certainly cannot take the class library that does the stitching and compile it into your own program.

[Andy600]

@DFM - Ok, we will remeasure and recalculate to produce the same numbers and call it ML Standard (a pointless exercise). The matrices don't need the Adobe name attached to them. We're not trying to coat-tail on the Adobe name, if anything, ML has increased Adobe's user base. I only wrote Adobe Standard so that the devs would know the matrices are gleaned from the DNG converter.

[Andy600]

Somebody should inform X-rite that their colorchecker software violates Adobe EULA

[chmee]

please @all. discuss in a gentle, right manner.

@DFM i dont see any reverse-engineering and not breaking rules.
4.5b - its about the software, not about the calculated files (vulgo dng and tags)
5.1 misusing and kind of Rules of Conduct - we re not serving adobe services (nor parts) in any illegal kind.

[DeafEyeJedi]

We're not trying to coat-tail on the Adobe name, if anything, ML has increased Adobe's user base. I only wrote Adobe Standard so that the devs would know the matrices are gleaned from the DNG converter.

This one puts the nail on the wall. So true & Well said @Andy600!

Only because I slowly lost interest in Adobe softwares back in the hey days until @a1ex's ML forum became a huge hit (& still is) in which obviously brought me back!

[Andy600]

@Chmee - Yes, just to clarify the tone of my own comments in this thread because I know written text can easily be taken out of context. Everything I have written is not intended to be confrontational or derogatory to anyone concerned. My comment about X-rite was a little tongue-in-cheek. I totally respect the opinions of others even if I disagree them.