Magic Lantern Raw Video Default Color (plus a task for any willing ML users)

[Andy600]

This is a long post (yawn) but most of it is summed up in the first couple of paragraphs. In the first reply to this post is a simple exercise for any willing participants to provide normalized matrices using DNG metadata and Microsoft Excel.

In an effort to achieve a basic, uniform rendering of color from Magic Lantern Raw Video across many different applications there is first a need to standardize color related metadata in the DNG and CinemaDNG files output by Magic Lantern raw2dng apps.

Presently, there are multiple raw2dng apps that provide users with DNG or CinemaDNG files from original Magic Lantern Raw (.raw) and 2nd gen Magic Lantern Raw Video (.mlv) container formats with each app embedding color related metadata (matrices) according to whatever level of meta support the app developer chooses to include. The variations in default color rendering between post production apps proves the need to begin standardizing this important part of Magic Lantern Raw Video.

Enhanced color and look management using embedded DNG camera profiles and/or .ICC camera profiles would be an obvious next step and the (remote) possibility of any future ACES support for MLV will hinge on this small but important change to metadata.


Why?

It is important to understand the differences between DNG and CinemaDNG standards in relation to color in raw video workflows. CinemaDNG is a subset of the DNG Standard (both developed by Adobe Labs). CinemaDNG has several addition video related meta tags (timecode, shooting information, reel numbers etc) however some recommended DNG specification tags are omitted in CinemaDNG. While both standards 'can' be used/mixed it leads to actual color appearance being rendered differently depending on the particular standard that the raw reader (i.e. your NLE or color grading app) is based upon.


To simplify things we should look at the most basic purpose of .raw and .mlv and that is to produce moving images.
 
DNG standard is intended for stills photography with raw photography apps  like Adobe Camera Raw, Photoshop, Lightroom, RawTherapee etc making use of any/all DNG standard meta content.
 
CinemaDNG standard is for raw video. Color grading, compositing, DI and NLE apps (Resolve, Premier Pro, Scratch, Nuke etc) are not required to parse certain DNG tags (i.e. they can and will ignore some tags even if they are included in metadata) and therefore, as .raw and .mlv is primarily intended for video production and editing with video-centric post production software, magic Lantern Raw Video should ideally be conforming to CinemaDNG standard. Some raw2dng conversion apps already do this to a lesser or greater extent.


What this means in relation to color:

The key color related difference between these 2 standards is the inclusion or omission of Forward Matrices. A forward matrix (part of DNG standard but not part of CinemaDNG standard) acts as an additional raw level color calibration function. A forward matrix maps normalized sensor RGB values (after white balancing) to a connecting colorspace or PCS. In this case the PCS is CIE XYZ with its D50 white point - PCS is used when transforming from one RGB colorspace to another i.e. to translate your DSLR sensor's wide gamut primaries to fit a much smaller display colorspace like sRGB or Rec709.

The color matrix tags which are included in both standards map a sensors non-white balanced RGB values and white point to PCS using a simple 3x3 least squares matrix. In CinemaDNG, the color matrices perform both the white balance and initial color calibration however, color is only a secondary, fallback function and will usually require additional color calibration after debayering to a colorspace.

Presently there are 2 ML supported raw2dng apps that embed 2 sets of color matrices and 2 sets of forward matrices per camera. These apps are raw2cdng (for Windows) and MLVFS (for OSX but can be configured for use under Windows).

The CinemaDNG files produced by these apps will tend to produce a 'nicer look' by default (similar to Adobe Standard color) but only when the files are debayered in an application that is built to honor the DNG standard and/or forward matrix tags. However, most video-centric color grading applications (Resolve, Premier Pro's native DNG support, Nuke etc) do not support forward matrices i.e. the forward matrices are ignored -  only the color matrices will be used. Resolve may have supported forward matrices in the past but v12 does not.

CinemaDNG standard does not support embedded 'look management' - stating that this part should be deferred to post production to enable a very basic, low level default rendering of the color across multiple apps and workstations.

So what needs to change?

The first change is very simple and is dictated by both DNG and CinemaDNG standards - We need DNG files with normalized primary matrices.
 
As CinemaDNG is the more relevant standard for MLV to follow, and because CinemaDNG does not support forward matrices, the color matrices used should be normalized (hard-coded normalized). The idea behind this is where a pixel value is recorded as white it should remain white (or neutral) throughout the transformation process to a display space. The normalization of color matrices in photographic related apps like Adobe Camera Raw is typically an automated function i.e. the color matrix will always be scaled at input regardless of the hard-coded scale of the matrices in metadata. The forward matrices are always normalized and so the final output will always be normalized and scaled correctly ready for the PCS to transform to a display referred or scene referred colorspace.

In applications that are coded to support the CinemaDNG standard and using only the current color matrix sets in MLV (which are not normalized), this auto-scaling does not typically happen. The debayered primaries are therefore not correctly scaled/optimized for PCS mapping or translation to a colorspace and this may lead to minor inaccuracies in white balancing and/or color rendering.

The most likely issue is with default saturation levels being slightly higher than expected but there may also be other, more significant issues where the same white balance multipliers (color temp and tint offsets) may behave differently across different apps. This part is somewhat theoretical but obvious differences can be seen when comparing 2 copies of the same DNG (one containing normalized matrices and the other containing non-normalized color matrices) and then analyzing the resulting trace on the Vectorscope in DaVinci Resolve.

Possible issues with normalized color matrices:

Although I am not posting the full normalized matrix set here (see the first reply of this post for the reason why) I have checked most normalized matrices using DNG_Validate.exe (v1.4 found in the Adobe DNG SDK) and they should be ok for use as-is (without forward matrices) but it is understandable that developers may wish to continue with the inclusion of forward matrices for raw processing apps that can make use of the extra metadata. I have also tested the normalized color matrices with forward matrices (using Exiftool) and these too validate using this method but I strongly recommend app developers test privately with their own apps before release as there may be validation issues when compiling (if using the Adobe SDK) that I have not personally experienced when changing meta with Exiftool.



My personal view is that MLV should follow CinemaDNG standard and omit forward matrices completely. This approach 'should' then produce the same default color rendering of Magic Lantern Raw Video across raw post production apps with any remaining difference then being purely down to an individual application's color processing i.e. debayer algorithms, noise reduction, chromatic adaptation model etc.
 
This change will not perform any color matching between different Canon DSLR models. The differences due to color sensitivity and color reproduction between say a 5D mark III vs an EOS-M will remain. Color matching cameras is a whole other subject and in CinemaDNG is deferred to post production. Conversely we are not creating additional problems or imposing a look ahead of any such color matching or look application i.e. this proposal gives the user the most flexibility in post without anything degrading the image color at its most basic raw level.

If raw2dng developers agree with this proposal and want to continue with the inclusion of forward matrices (i.e. mix DNG/CinemaDNG standards) I would suggest making it an option, not a default and perhaps link the option to this post or inform users of the difference in color they may experience. I will add further content to this thread including some simple color comparisons for users who are less technically oriented or who may not speak English.




Disclaimer: I (Andy600) and my company (CinelogDCP) produce commercial color management products that can be used with DNG footage however the changes I propose here are not related to CinelogDCP and are intended to benefit the wider ML community regardless of whether or not CinelogDCP or other third party products are used.

[Andy600]

What ML users can do to help:
Level: very simple

Normalizing the color matrices is a very simple procedure.

I have uploaded an Excel spreadsheet (norm_cm_calc.xlsx) where you can enter a color matrix (in the blue row) to obtain a normalized color matrix (shown in the yellow row). The 2x DNG color matrices can be obtained from any Canon raw image (.cr2) which has been converted to a DNG with Adobe DNG converter. I recommend doing this rather than extracting metadata from MLV as I can't be sure that the app you used embeds the correct color matrices (with the exception of the latest versions of MLVFS and raw2cdng).


IMPORTANT: Enter the color matrix into the blue row as it is shown in Exif (reverse row scan order), do this only for the color matrices and not the forward matrix metadata. The forward matrices are already normalized.

Please post only the normalized color matrix 1 and color matrix 2 and your camera model below and I will check them off against a set I have already complied. Once we have the full set for ML supported cameras they can be used by raw2dng app developers.

The numbers you post should be as shown in the yellow row (i.e to 4 decimal places) therefore you cannot copy/paste the row. You should type the numbers and double check against what is shown in the yellow row.

download: https://s3-us-west-2.amazonaws.com/magiclanternrelated/norm_cm_calc.zip

[Andy600]

Thanks to everyone who participates

I'll compile your normalized matrices here after checking them against my set and add any cameras that I've missed off the list.


EOS 5D mark III

Normalized Color Matrix 1 : 0.7882  -0.1540  -0.0654  -0.3956  1.2149  0.3105  -0.0416  0.1455  0.7014
Normalized Color Matrix 2 : 0.7862  -0.0743  -0.1126  -0.5014  1.4573  0.2372  -0.1062  0.2529  0.6629


EOS 5D mark II

Normalized Color Matrix 1 : 0.7076 -0.0305 -0.0448 -0.8318 1.7680 0.4448 -0.1089 0.1619 0.8882
Normalized Color Matrix 2 : 0.7103 0.0908 -0.1250 -1.1745 2.3305 0.3735 -0.2253 0.2917 1.0017

EOS 7D

Normalized Color Matrix 1 : 1.1277 -0.6163 0.0005 -0.2483 0.9847 0.2730 0.0023 0.0833 0.6722
Normalized Color Matrix 2 : 0.7871 -0.1145 -0.0984 -0.4458 1.3526 0.2756 -0.0682 0.2038 0.7128

EOS 6D

Normalized Color Matrix 1 : 0.8335   -0.1585   -0.1026   -0.4248   1.2689   0.2973   -0.0367   0.1335   0.7036
Normalized Color Matrix 2 : 0.8174   -0.0934   -0.1178   -0.5136   1.4600   0.2392   -0.0989   0.2317   0.6691

EOS 70D

Normalized Color Matrix 1 : 0.8335 -0.1585 -0.1026 -0.4248 1.2689 0.2973 -0.0367 0.1335 0.7036
Normalized Color Matrix 2 : 0.8174 -0.0934 -0.1178 -0.5136 1.4600 0.2392 -0.0989 0.2317 0.6691

EOS 60D

Normalized Color Matrix 1 : 0.7991   -0.2041   -0.0528   -0.3771   1.1794   0.3151   -0.0363   0.1336   0.6899
Normalized Color Matrix 2 : 0.8073   -0.1194   -0.1111   -0.5297   1.4931   0.2657   -0.1066   0.2558   0.7271

EOS 50D

Normalized Color Matrix 1 : 0.7124 -0.0704 -0.0050 -0.5710 1.4238 0.4172 -0.1079 0.2828 0.8374
Normalized Color Matrix 2 : 0.6853 0.0858 -0.0826 -0.9044 1.9451 0.3878 -0.2471 0.4427 0.9757

EOS 550D (Rebel T2i)

Normalized Color Matrix 1 : 0.7844 -0.2477 -0.0353 -0.3142 1.0340 0.3401 -0.0158 0.0997 0.6483
Normalized Color Matrix 2 : 0.7879 -0.1321 -0.0973 -0.4342 1.3164 0.2876 -0.0472 0.1748 0.6855

EOS 600D (Rebel T3i)

Normalized Color Matrix 1 : 0.7635  -0.2042  -0.0459  -0.3582  1.1297  0.3410  -0.0290  0.1128  0.6866
Normalized Color Matrix 2 : 0.7734  -0.1086  -0.1056  -0.5147  1.4584  0.2846  -0.0980  0.2327  0.7099

EOS 650D (Rebel T4i)

Normalized Color Matrix 1 : 0.7528 -0.1736 -0.0428 -0.3876 1.1585 0.3551 -0.0376 0.1224 0.7019
Normalized Color Matrix 2 : 0.7950 -0.1013-0.1131 -0.5385 1.5001 0.2706 -0.1174 0.2455 0.7403

EOS 700D (Rebel T5i)

Normalized Color Matrix 1 : 0.7528 -0.1736 -0.0428 -0.3876 1.1585 0.3551 -0.0376 0.1224 0.7019
Normalized Color Matrix 2 : 0.7950 -0.1013 -0.1131 -0.5385 1.5001 0.2706 -0.1174 0.2455 0.7403

EOS 750D (rebel T6i)

Normalized Color Matrix 1 :
Normalized Color Matrix 2 :

EOS-M

Normalized Color Matrix 1 : 0.7524 -0.1735 -0.0428 -0.3871 1.1579 0.3549 -0.0376 0.1224 0.7015
Normalized Color Matrix 2 : 0.7950 -0.1013 -0.1131 -0.5385 1.5001 0.2706 -0.1174 0.2455 0.7403

[Andy600]

(reserved)

[Danne]

Hi Andy!

Are these normalized color matrices derived from you own calculations and measurements or where are they obtained from? Why are you taking the detour with the spreadsheet rather than just publishing the numbers? Very tedious to copy paste every number in the sheet I think .

I inserted the "normalized" numbers for the 5d mark 3 and also created two dng files for comparing purposes here.
https://drive.google.com/file/d/0B4tCJMlOYfirWVU2TDRaeThmclU/view?usp=sharing

When viewing in acr you can,t spot a visual difference. Would there be a visual difference in davinci resolve? Can,t test right now.
Thanks for sharing.

[Andy600]

I posted the spreadsheet so that ML users who are not technically minded but who may want to participate in something easy can do so.

If the list isn't complete in a couple of days I will post mine. I'm just using it to check numbers until then.

You will not see a change in ACR:

The normalization of color matrices in photographic related apps like Adobe Camera Raw is typically an automated function i.e. the color matrix will always be scaled at input regardless of the hard-coded scale of the matrices in metadata.

but there is a difference in Resolve

The matrices are from DNG converter. The reason for using them is because the default DNG matrices embedded by DNG converter are solved accurately for the 2 recommended illuminants (CIE-A and CIE D65) from very precise spectral sensitivity measurements of the sensors. There is another 'free' way to produce the 2 required matrices using Dx0 published data but one of them has to be chromatically adapted to D65 as Dx0 measured under a D50 lightsource and the end result doesn't appear to be as accurate - but I'll post a sample matrix set for the 5D3 later so you can compare.

[Danne]

This is interesting. Could you please share sources from where you found this?
Are you saying acr is scaling primary matrices creating normalized matrices "under the hood" while davinci resolve sticks to not normalized primary matrices therefore incorporating normalized primary matrices would be matching acr?
Whatever happened to the idea to create a ML original color matrice system which I thought was the first idea talked about?

[DeafEyeJedi]

This is Definitely interesting and great questions @Danne! 

... and I still believe in magic and it's true pure ML original color matrices system to be coming soon.

*Prays*

As always I have faith in you, Andy!

p.s. I'll be sending over the color matrix codes from my current cameras shortly, Thanks!

p.s.s @Danne comparing your both DNG files which clearly shows the differences -- look at the black jacket becoming more noisy/flat with the Norm version. Is that the way it supposed to 'look' normal or is the first one more closer to being 'normal' and I apologize for being a goof ball in here.

*edit*

EOSM:


70D:

[vertigopix]

My results :

EOS 6D :

Color Matrix 1 : 0.8335   -0.1585   -0.1026   -0.4248   1.2689   0.2973   -0.0367   0.1335   0.7036
Color Matrix 2 : 0.8174   -0.0934   -0.1178   -0.5136   1.4600   0.2392   -0.0989   0.2317   0.6691


EOS 60D :

Color Matrix 1 : 0.7991   -0.2041   -0.0528   -0.3771   1.1794   0.3151   -0.0363   0.1336   0.6899
Color Matrix 2 : 0.8073   -0.1194   -0.1111   -0.5297   1.4931   0.2657   -0.1066   0.2558   0.7271

[Andy600]

Thanks for helping guys! I'll check your numbers and compile the growing list in a reserved post above.

@Danne - It comes from multiple sources and I'll need to go through some Adobe forum posts and a few websites to find the official quotes (most of them are from @madmanchan a color scientist at Adobe labs) but normalizing the color matrices in the absence of forward matrices it is technically the correct thing to do for CinemaDNG.

Are you saying acr is scaling primary matrices creating normalized matrices "under the hood" while davinci resolve sticks to not normalized primary matrices therefore incorporating normalized primary matrices would be matching acr?

ACR scales the color matrices at input, Resolve does not scale the matrices at input. You can test this by scaling (up or down) a matrix set and viewing in ACR against a normalized set - there is no visual difference in output but in Resolve there is. Ideally you need to use scopes, not your eyes. I have TestGear in After Effects for scopes but you could use the Color Finesse plugins' Vectorscope.

ACR is display-referred and also applies camera profiles - this must be taken into account when comparing the output with other apps that are scene-referred linear. Linear output (i.e. no s-curves, gamma, HSV corrections etc) should 'technically' produce the same image as Resolve except where debayering algorithms or chromatic adaptation models differ. .ICC profiles in the ACR pipeline 'might' also affect things but I'll be looking into how to negate this.

The idea of 'ML color' is still the intention and this technical change is simply a first step.

If everyone used ACR I could provide a better forward matrix set now but there is no point in doing this because the FMs will be ignored by most of the other apps that people use - i.e. we will be back to square one with users getting different color depending on the app they are using. The idea is to remove or at least minimize any difference in default color rendering. The next step is fine-tune things (calibrate) so that ML supported cameras produce the same color in relation to a standard color model (within the limitations of each sensor/model) - then we can define some 'looks', ideally keeping all of this based on linear matrices so as to not introduce artifacts.

The color matrices are, technically speaking, only used for white balancing the image and any color processing, calibration and/or look management will happen downstream (it is deferred to post-production as per CinemaDNG specs). The color matrices can be 'tweaked' but it's not a good idea to do so. The color matrices applied in DNG converter are solved from actual spectral sensitivity measurements (i.e. the sensors native colorspace) and there is no reason to believe the Adobe Labs measurements are anything but accurate i.e. they are the best foundation to base any further color processing on.


@DeafEyeJedi - Thanks for your input and enthusiasm as always

I would caution against getting too excited at this stage but we will definitely move the color related part of ML raw video forward. There is a lot of scope for improvement.

[Andy600]

@DeafEyeJedi - I'm not sure how you are viewing Danne's test images but any difference in levels is not related to the matrices - if anything, there should be slightly less saturation with the normalized matrices. I just had a quick look at them in Rawdigger and they look pretty much identical (as expected) but for some reason Rawdigger it is not picking up the blacklevel offset tag and I had to manually change it from 0 to 2047.

[Danne]

Thanks for answers. Cdng seems to be the way to go working with film files. My 5 cents.

[Andy600]

@Danne - Yes. I can't really understand the reason for CinemaDNG not supporting forward matrices. Even Adobe (who created both standards) understand the benefits. FMs are very useful (non-destructive) and don't take up much space in meta. Alas, it doesn't look like CDNG will be evolving any time soon. The last update was, I think about 4 years ago.

I also can't see why they don't make it mandatory for raw decoders to parse these important tags. It would be a little more work for the coders but it's not like there isn't an SDK to use - DNG based workflows in general would benefit.

[baldavenger]

My results:

EOS 600D (Rebel T3i)


Normalized Color Matrix 1 : 0.7635  -0.2042  -0.0459  -0.3582  1.1297  0.3410  -0.0290  0.1128  0.6866
Normalized Color Matrix 2 : 0.7734  -0.1086  -0.1056  -0.5147  1.4584  0.2846  -0.0980  0.2327  0.7099



EOS 5D mark III


Normalized Color Matrix 1 : 0.7882  -0.1540  -0.0654  -0.3956  1.2149  0.3105  -0.0416  0.1455  0.7014
Normalized Color Matrix 2 : 0.7862  -0.0743  -0.1126  -0.5014  1.4573  0.2372  -0.1062  0.2529  0.6629

[DeafEyeJedi]

@DeafEyeJedi - I'm not sure how you are viewing Danne's test images but any difference in levels is not related to the matrices - if anything, there should be slightly less saturation with the normalized matrices. I just had a quick look at them in Rawdigger and they look pretty much identical (as expected) but for some reason Rawdigger it is not picking up the blacklevel offset tag and I had to manually change it from 0 to 2047.

I actually just tap the space bar and preview both frequently back n forth -- clearly the black levels (or at least I thought) were slightly off?

*edit*

Here's what I was seeing... The blacks on her jacket is different slightly a bit and also watch her hair it becomes somewhat a vertical type of a cut going right through and shifting to the left by a few resolutions if you look closely to it. It's there. I wonder what caused that to occur?

Non-Norm:

Post Norm:

Also FYI re: 70D from the second take per your request from earlier...

[Andy600]

@DeafEyeJedi - It's Photoshop and/or ACR causing the difference you are seeing. You need to look at the image without any processing to see they are the same in terms of levels. RawDigger or Resolve (set to RCM linear input) will show you this. If you have RawDigger and load the image without any auto-scaling and then switch on the over/under exposure indicators (zebras) there is no difference between the images and the zebra patterns do not change - they would if there was a difference in levels.

You may have uncovered something interesting relating to Photoshop. The Black Level tag which tells the decoder where absolute black is (and removes flare) is being used by Photoshop but it looks like it is also applying some sort of levels adjustment.

[baldavenger]

If you're using the Quick Preview function on a Mac (selecting an image in Finder or on the Desktop and pressing spacebar), then that might be at the core of the issue.  The same thing happens on my Mac (as does to @DeafEyeJedi)in Quick Preview, but not in RawDigger.

[Andy600]

Yes. What @baldavenger said. I saw the word Photoshop in your images but didn't notice it was Quick Preview. I'm not a day-to-day Mac user.

I'm still looking back through posts related to the EOS-M but MLVFS and raw2cdng source code definitely has a different set of matrices for the EOS-M than you get with Adobe DNG converter.

[JADURCA]

5D Mark III

0.7882 -0.1540 -0.0654 -0.3956 1.2149 0.3105 -0.0416 0.1455 0.7014

0.7862 -0.0743 -0.1126 -0.5014 1.4573 0.2372 -01062 0.2529 0.6629

[Andy600]

I've filled most of the remaining matrices (see reply#2) but there is still a few to add (750D, and the non-I rebel cameras)

It's interesting when you look at the numbers. You see that the 6D and 70D have the same coefficients, as do the 650D and 700D i.e. they probably have the same sensor. What is a little curious is the EOS-M which has the same D65 matrix as the 650D and 700D but a different CIE-A matrix. I'm not sure exactly how Adobe profile the cameras but it seems a bit unlikely for this to happen and may point to an error when calibrating or when entering code for a DNG converter update.

Each set of matrices is derived from measuring the sensitivity of a candidate camera and all other cameras of the same model or sensor type use the candidate cameras solved matrices as a baseline. The differences between sensors of the same batch should be small enough to allow this so we have to assume that where matrices match they are indeed the same sensor but I would still expect to see some very minor variation between candidate cameras which would affect the matrix coefficients - it's a bit odd that this doesn't happen with certain models.

[DeafEyeJedi]

I have the SL1 (100D.100A) if that could help?

[dfort]

What is a little curious is the EOS-M which has the same D65 matrix as the 650D and 700D but a different CIE-A matrix. I'm not sure exactly how Adobe profile the cameras but it seems a bit unlikely for this to happen and may point to an error when calibrating or when entering code for a DNG converter update.

I'm a little late to the party but finally figured out what you are doing. Here are the slightly different numbers that I got--differences in red.

EOS-M

Normalized Color Matrix 1 : 0.7528 -0.1736 -0.0428 -0.3876 1.1585 0.3551 -0.0376 0.1224 0.7019
Normalized Color Matrix 2 : 0.7950 -0.1013 -0.1131 -0.5385 1.5001 0.2706 -0.1174 0.2455 0.7403

So it looks like the EOS-M does match the 650D and 700D.

[vertigopix]

6D and 70D can't have the same sensor as the 6D is FF and 70D is APS-C !

[Andy600]

Thanks guys!

@dfort - which version of DNG converter did you use?

@vertigopix - yes! and so it's even more unlikely that they would have identical spectral sensitivity.

The color matrices from DNG converter may actually be more of a look than a calibration and/or may have been developed to only work in concert with Adobe's contrast-referred forward matrices. If this is the case we may need to do things the hard way (because the forward matrices will be ignored in Resolve, Premier and most other color/NLE apps) but I'll first see if I can get some time with a CamSpecs system to test the response of a couple of models for comparison.

As it stands, the Adobe color matrices seem to be the lesser of two evils and they work in terms of white balance. Further calibration can be done in post without causing any damage provided the pixels are processed in 32bit float and reliable methods for doing that is the next step.

[DeafEyeJedi]

Nice progress so far, @Andy600!