ACES for Magic Lantern Raw Video

Started by Andy600, February 22, 2014, 11:04:42 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Andy600

As the DNG files produced by Magic Lantern Raw Video are predominantly used for moving images I would like to propose that Magic Lantern adopt ACES (Academy Color Encoding System).

ACES is an rapidly becoming the standard color science for motion picture and VFX pipelines and is aimed to standardizing color reproduction across cameras, regardless of manufacturer. It also offers the benefits of full Dynamic Range reproduction in the widest possible color gamut and accurate conversion to any colorspace (with an ODT) without needing to re-grade.

Incorporating ACES into Magic Lantern Raw Video will mean some minor changes to color matrix calculations in .raw, .mlv and probably raw2dng but beyond that, users should not notice an difference in their current workflows. The big benefits will come when using ML Video in an ACES pipeline as color reproduction will become both predictable and standardized. To fully utilize ACES we will likely need Input Device Transforms (IDT) for the cameras but these are relatively simple and (I suspect) we can base them temporarily on current knowledge deemed from DNG files, Canon IDTs and maybe Canon .ICC profiles.

OpenEXR will not directly be needed but may be the best option for DNG app devs who want to incorporate the highest quality OpenEXR output.

ACES and OpenEXR are both open source.

For more information on ACES:-

Presentation: http://www.oscars.org/science-technology/council/projects/pdf/ACESOverview.pdf
ACES: http://www.oscars.org/science-technology/council/projects/aces.html

Ampas/ACES sourcecode: https://github.com/ampas/aces-dev/tree/v0.1.1

ACES discussion group: https://groups.google.com/forum/#!forum/academyaces

OpenEXR: http://www.openexr.com/
OpenEXR sourcecode: http://www.openexr.com/downloads.html


Thoughts?
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Midphase

I second this. Even though most people still don't know about ACES, it's about to become the new standard and it would make sense for ML raw to get in on the standard as soon as possible.

reddeercity

Yes I agree, I have been following ACES since the beginning , i'm members a the
Group. My only problem is getting the (IDT) for canon . That stop me ,
If I had that I could use the ACES.

Andy600

Resolve has IDT's for the 5D (2/3?) and 7D (I don't think these were developed by Canon either) and you can also download new IDT's for Canon's C cameras. I'm studying them. To accurately develop an IDT means some serious scientific measurements but I think there is enough of a foundation using current IDT's plus what can be derived from .ICC and raw images to approximate a useable IDT for ML raw video. I'm currently getting to grips with CTL which will be very useful for developing this.

We're working with raw data so it's completely possible to bypass Canon's input if they are unwilling and I'm sure Blackmagic will include a Magic Lantern Raw Video IDT in Resolve if we can supply them with one.

I'm pretty sure MLV already has capacity to do this. It just means recalculating the matrices to D55 with linear gamma which is what ACES uses. I think we'll need g3gg0's input ;)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Midphase

Quote from: Andy600 on February 22, 2014, 07:56:09 PM
To accurately develop an IDT means some serious scientific measurements but I think there is enough of a foundation using current IDT's plus what can be derived from .ICC and raw images to approximate a useable IDT for ML raw video.

Andy, it seems to me that you probably have some pretty good measurements already from your CINELog ML raw work. What can we do to help?

Midphase

Anyone who isn't sure what this is about, should watch this video which explains some of the ideas behind it:


Andy600

Most of our measurements to date relate to color differences between cameras and to help with making uniform LUTs for our DCP profiles using standard matrices but we are now working on custom matrix transformations as it's a superior, non destructive method compared to Lookup tables. We still have a ton of new testing to do but I'm sure some of this will be useful for making IDTs and I'm sure we'll even have a go at it ourselves. I'll make the data (well, some of it lol) available when I can and certainly the new color matrix info.

I just hope people aren't too attached to Canon color science because it's totally irrelevant and would be discarded for ACES . I guess, as ML is now modular we could always have ACES compatible and a RGB/Rec709 compatible raw modules but I don't know enough about the backend of ML (yet) to be sure ;)

Incidentally, we're having some early success using Cinelog with the ARRI Alexa and Sony Raw IDTs in Resolve (ACES Log color science) so we're no far off.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

reddeercity

Great work  Andy600,  I will then get more active with ACES
And use my email connection with the group to get info
Related to Raw data etc...
I have been waiting for a chance to use ACES workflow !

Andy600

@Midphase - good introduction video!

I'll also add that ACES is a vastly wider color gamut than humans can actually see and some of the typical problems we currently experience like certain colors (i.e. reds and greens) over-saturating and clipping (caused by linear increase in saturation levels), color crosstalk caused by less than ideal matrices and primary imbalances are far less of a problem with ACES. You really can push colors around and the ODT will take care of things.

We all seem to forget that Canon DSLR's are stills cameras and were never really intended for video hence the photography methodology behind the color science. ML Raw with ACES would gift us the opportunity to use these cameras in a cinematic way and elevate this already great software even higher!
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Lars Steenhoff


eundd

Would be very interested in this! Is there any progress made yet? Maybe it should be posted as a proposal on Bitbucket? https://bitbucket.org/hudson/magic-lantern/issues?status=new&status=open

baldavenger

The new Cinelog updates have provisions for an ACES workflow via both ACR and Davinci Resolve.
EOS 5D Mark III | EOS 600D | Canon 24-105mm f4L | Canon 70-200mm f2.8L IS II | Canon 50mm f1.4 | Samyang 14mm T3.1 | Opteka 8mm f3.5

Andy600

We've also built IDT's for all MLV capable cameras but need to have another look at them since ACES 1.0 (ACES 2065-1) was just released. Once we're 100% happy with the results we will submit an application to the Academy to see if we can get them into the ACES source code.

In the meantime, as @baldavenger said - The Cinelog-C release has transforms to ACES Colorspace, ACES Log and ACES Proxy colorspace. Setting the transform to ACES will work as a bridge to ACES 1.0 - If you do want to output ACES files they must be .exr format because the colorspace is very big and needs a floating point container.

The actual MLV IDT's we built are described for 2 Illuminants and are camera specific. Whether or not this actually gets into ACES is another thing because ACES is centric to high-end professional cameras - but we will see :)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

AgentJJ

I recently constructed a series of IDTs for three input Illuminants (D56, Tungsten and F3) for a set of ISOs on the Canon 650D for a Web Series ( JackAndCarmen.com ) .  The IDTs are constructed as 3x3 Affine Transformations (Matrix Form).  Chromatic Adaptation transformations were constructed as well. 

The transformations were applied via color transformation language (ampas' CTL) and sony's OCIO configs.  OCIO was used to construct 3D LUTs converting ACES to our calibrated monitors.  ODTs were handled in Davinci Resolve.  We originally tried the rec709 ODT, but ended up constructing our own LUT for ODT output (being piped through Davinci's internal RRT) for final web distribution (output to D65). 

It was a great experience and I recommend anyone to construct their own IDTs per their own camera, because I've found (and it's been shown) that even between cameras of the same model that visibly detectable differences in sensors occur.

I plan on posting a detailed overview of IDT construction in the near future for those interested.

DeafEyeJedi

I second that -- good to know and thanks to @Andy600 & @baldavenger for refreshing this thread!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

zachnfine

I don't suppose anyone has one of these devices lying around:
http://www.image-engineering.de/index.php?option=com_content&view=article&id=452&Itemid=198

I'd love to have IDTs for the Canon mkIII and to try the ACES workflow in Resolve with DNG sequences captured using MLV, but I have neither reflective targets nor neato illuminating boxes at which to aim the camera. But it'd be neat to have profiles specific to the sensor.

g3gg0

okay never noticed this thread.
to be honest, i am a bit confused...

MLV, as we have it today is an open container format for
a) raw video frames that
b) contain images in mosaic'ed (bayer) pattern, which are
c) in their shape specific to canon's sensors and
d) every mosaic tile filters the wavelength of the incoming light with properties specific to the sensor

okay now a) usually means that the frames have several flaws that have to be corrected, like stripes or dead pixels etc.
this is highly camera specific if you want a perfect image. just remember the FPN or AF pixels.
this also means, we cannot change that data during writing as we do not have any free CPU resources for that.

point b) means that we have to debayer it before the image is accessible as a X*Y pixelmap with pixels each representing a tristimulis value at that orthograpic coordinate.

about c)... canon usually uses the same bayer pattern for all models. other cameras have different patterns, some vendors even use CFAs with different shapes and colors.

and d) as the bayer pixels canon uses is not the same on all models (i even remember some modely having two different greens)
also the CFAs vary in their filter properties from camera model to another, this is highly model specific.
we are providing the dcraw sourced color matrices that provide a suitable sensor->XYZ colorspace transformation along with every MLV file.


if i understood ACES correctly, it describes how to separate the necessary steps during production from each other and having e.g. an
input transformator that delivers tristimulus values for a standardized color domain (extending xyY beyond its limits) on which all other modules continue their work.

just in case i understood it right, what is different to what we have now in MLV?
it provides raw data, encoded in canon style, to be stripe-fixed, to be debayered properly plus the transformation matrix to get standardized XYZ (which maps 1:1 to xyY)
to my understanding, colors are definitely the least problem with MLV :)

the only things to improve regarding colors, is
a) finding better sensor->XYZ matrices (no MLV needed, everyone could do it with CR2)
b) adding an output format that tools accept and which allows the MLV/RAW colorspace to develop how it deserves (or better: creating an IDT that directly reads MLV.....)

BR,
g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Andy600

Hi @g3gg0 - My original post was before MLV had matured to the level that it is now. Getting MLV into ACES colorspace is as simple as transforming from CIE XYZ to ACES 2065-1 chromaticity (D60 white/scene referred linear). The only major issue was the color matrices but I recently gleaned the dual illuminant Adobe Standard matrices for every MLRaw camera as the Adobe matrices do a much better job than the DCraw matrix (this also helps when using white balance defaults in most apps - ACR overrides these values with whatever is set in the camera profile so to keep it simple we use the same ones).

I think there is a degree of fine tuning possible when it comes to color matching but it's a huge and expensive technical job to do this for every camera in lab conditions. Adobe Standard is well known and close enough - even the main manufacturers don't stick rigidly to the ACES specs so there is always some variance between cameras (which kind of defeats the stated aim of ACES). 

Anyway, the Adobe Standard matrices are now embedded by MLVFS and raw2cdng but it would be useful if MLV can embed these matrices by default so that other converter apps can simply copy the default metadata (some apps are still embedding Canikon generic or fixed 5D Mark III matrices for any camera). It's possible to expand the meta content with more matrices for other illuminants but not many manufacturers are going beyond simple Tungsten and Daylight variants for their IDTs.

The MLVFS commit is here: https://bitbucket.org/dmilligan/mlvfs/src/d961f752f326856081a42f294e22ffca301df875/mlvfs/dng.c#cl-54
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

DeafEyeJedi

5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

g3gg0

ah okay, good to know.
ive read that you tried getting matrices, but never realized that it made its way to the DNG converters.
hoped that the converters use the matrix provided by the MLV files, and the matrices im ML core get their update.

dont you think its better to trust the embedded matris in the first place and make a pull request, updating ML's internal matrices?

the matrices you posted, into which colorspace do they convert? are these XYZ->bayer mtrices?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!


Andy600

@g3gg0 - The Adobe Standard matrices that I gleaned from Adobe's DNG converter are CIE XYZ to Camera RGB for 2x illuminants (A and D65). They produce fairly consistent color across all cameras.

I believe the bayer pattern order is described elsewhere in another tag so that the raw reader knows how to apply the transforms but the matrices are standard order for all DNG readers.

DNG readers can use a single color matrix and this can even be described with an unknown illuminant using the 'unknown 38168' tag but colors will be off and unless the white balance was spot on when shooting it takes a lot of work to find neutral white point - and sometimes the workspace/colorspace (and/or white balance controls) in certain apps are still too small to find the required offsets.

A good example is the Sony FS700 recorded as Sony FSRaw on the Convergent Design Odyssey 7Q. This embeds a single color matrix with an unknown illuminant tag and the result is usually VERY wrong reds. I built a camera profile for the FS700 DNGs with better matrices and calibrated everything to a Macbeth chart so FS users who may not have white balanced correctly when shooting can have a better starting point.

Some users (even professionals) think this is 'baked-in' white balance - this is of course nonsense as the pixel data is still raw and can be manipulated. It's just that the embedded metadata isn't described correctly for a raw reader to make any real sense of and not sufficient for the transforms plus, as there is only a single (unknown) illuminant there is no information for matrix interpolation to work correctly - and that's on a professional system  ::) - Us ML users are more lucky than we realize ;)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

g3gg0

yeah and all of these can get embedded into ML core then :)
or would there be some issues with copyright etc?
(ML core has matrices for all models, depending for which model it was compiled)

this would help not only MLV or RAW users, but also DNG through the silent.mo
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Andy600

Yes! I think we will also see some of the 'color' issues that users post about being resolved or improved. Incorrect matrices can manifest in colorcasts, weird green tints etc  - sound familiar?  :D

It's no more of a copyright issue than the current ones. DCRaw matrices came from Adobe but they're a little outdated now plus DNG is an open format that requires matrices. These are just the same as you get with the DNG converter and it's all part of the SDK AFAIK.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

DFM

Sorry but the Adobe DNG SDK does not include matrices, just the C++ libraries required to interact with a DNG file object. All support files supplied with installations of Adobe Camera Raw or the Adobe DNG Converter utility are copyrighted and subject to Adobe's standard end user license agreement. They may not be extracted for standalone use or incorporated into any third party product.