EOS DSLR Internal Color Pipe Line

[reddeercity]

Over the last mouth or so I been researching the internal color pipe line to understand what's really going and if there a way to manipulate it .
I guess I should start with the basic's (some already know this part but bare with me) Will most dslr to my knowledge follow a digital still cameras ISO standard which is called "ISO 17321"

The following came from this PDF  https://www.cs.sfu.ca/~mark/ftp/IsoWg18/holm.wd2.pdf

The DSC(digital still cameras) characterisations obtained using this International Standard are expressed as transformations. These transformations, when applied to raw DSC data, produce estimates of scene (or original) colorimetry. The most common transformation form is a set of tone reproduction curves (TRCs) for each DSC analysis channel, followed by a matrix, followed by a single TRC applied to each of the three channels produced by the matrix. The purpose of the first set of TRCs is to linearize the DSC data with respect to scene radiance, and normalize the data with respect to the adopted white point. These TRCs may vary from scene to scene because of DSC flare, even if the adopted white point remains constant. The matrix transforms the data from the linear DSC spectral space to estimates of scene colorimetry expressed in the linear original RGB colour space, which is also defined in this International Standard. Different matrices based on different scene spectral correlation assumptions, and obtained using each of the three methods described may be used with the same DSC. The final TRC converts the linear original RGB data to ISO original RGB data, which is a more perceptually uniform representation for encoding. This TRC is based on the EOCF(electro-optical conversion function**relationship between the digital code values provided to an output device and the equivalent neutral densities produced by the device**) of an IEC standard sRGB display without veiling glare, so that any scene dynamic range can be represented. In addition to being relatively uniform perceptually, data encoded in this manner has the advantage that if it is displayed on an ideal sRGB display, the colorimetry of the display when viewed in a dark room (with no veiling glare) will match the estimated colorimetry of the scene or original viewed under the capture illuminant, with the adopted white point transformed to that of the display (D65) in the manner specified in this International Standard..... .

The linear original RGB to XYZ conversion matrix is determined based on the CIE XYZ values for the reference white point. This is accomplished by pre-multiplying the equi-energy original RGB to XYZ matrix with a diagonal matrix containing the XYZ values of the reference white point, as shown in equations 8 and 9, which are for D65 and D50 respectively.

The recommendation for extended bit depth and gamut ISO original RGB is therefore to use the last bit for a sign, the second to last bit for gamut extension of the normalized linear values above unity, and any additional bits for extended bit depth. The minimumbit depth requirement for extended gamut is therefore 10-bits per channel with k = 255. If 16-bits per channel are available, it is possible to increase k to 16 383 and still encode a full perceptual gamut of colours.

So as I understand it sensor data -->ISO original RGB -->XYZ conversion matrix-->D65 or D50 white point in sRGB wide gamut matrix determent by scene original color with what there called
extended bit depth and gamut ISO original RGB.
It's interesting that there 4 Tone curve being applied , 1 per RGB channel = 3  and 1 curve over the whole RGB image .
So much more to understand.



So how dose this work with Picture Style , Will I believe that and my research seem to verify that Pic.Style are really ICC profiles that have acr like adjustments applied to the Base Matrix.
And all cameras base matrix are not all equal to each other.
Because if you examine what's happen after the sensor data before ISO RGB there is tone curve applied to each channel (which linearize them) normalize the data with respect to the adopted white point. Then there is a tone curve applied to all channel (RGB) etc... .
With respect to CineStyle pf2 file , I do firmly believe that you are really applying not only the base adjustments saturation , contrast , sharpness , etc... .
but a color matrix or in this case Log color space .
Since I have a 5D2 the cinestyle pf2 file was design for my camera so I have no need to reverse engineer it , but what I do see here is with raw video . Knowing ML dose I think do some
XYZ conversion to RGGB raw data . My thought was if with could apply some sort of Log space that's save to mlv , maybe it not possible not sure.

There more to come , hope so far this information is useful to someone . 

[DeafEyeJedi]

Another solid reason why some of us should have kept the good old 5D2, right?

Thanks for sharing yet another useful details @reddeercity as per usual!

[Andy600]

...I believe that and my research seem to verify that Pic.Style are really ICC profiles that have acr like adjustments applied to the Base Matrix.
And all cameras base matrix are not all equal to each other.

You are correct that most cameras have a different calibration matrix because they have different sensors with different spectral sensitivities. I say 'most' because some Canon DSLRs share the same sensor. This all happens before the chosen Picture Style is applied and is not something you have control over in camera. You can change white balance, tint offset and exposure but not the cam2xyz matrix except for raw images where these coefficients are stored as metadata.

Picture Styles are applied to the demosaiced, color balanced, normalized image i.e. Picture Styles are added very late in the chain of processing - too late for an effective log conversion.

With respect to CineStyle pf2 file , I do firmly believe that you are really applying not only the base adjustments saturation , contrast , sharpness , etc... .
but a color matrix or in this case Log color space .

Cinestyle does not alter the colorspace. It's sRGB or Adobe 1998 depending on your camera settings. The base profile is neutral so you can say it's non-linear because it has targeted adjustments made to colors and you don't have control over that either.

As for log, well there is definitely a 10bit log curve in Cinestyle but it looks like it does nothing (see the research in the other Picture Styles thread). An RGB tone curve performs the 'lin2log' transform but, after some research, I'm seriously doubting it's credibility. i.e. it can probably be improved upon but will still be limited because of when it is applied.

Since I have a 5D2 the cinestyle pf2 file was design for my camera so I have no need to reverse engineer it , but what I do see here is with raw video . Knowing ML dose I think do some
XYZ conversion to RGGB raw data . My thought was if with could apply some sort of Log space that's save to mlv , maybe it not possible not sure.

There more to come , hope so far this information is useful to someone . 

Cinestyle was developed using a 5D mark II but it does not contain anything specific to the 5D mark II i.e. the data that goes into Cinestyle should be more or less the same regardless of the camera if the cameras are setup the same and the pre-picture style calibration process is uniform across the Canon range.

re:raw log - You can post process a log conversion of the raw data to reduce it's bit depth (see SlimRaw) or compress the tonal range and gamut into a log colorspace to reduce it's footprint (and other benefits) but MLV itself doesn't do this kind of thing. The raw2dng converter adds metadata that is then read by the app that debayers the images (Resolve, After Effects, DCRaw etc) and it is then that the color calibration, white balance, normalization etc happens depending on the embedded meta and any overrides you may choose.

[agentirons]

Picture Styles are applied to the demosaiced, color balanced, normalized image i.e. Picture Styles are added very late in the chain of processing - too late for an effective log conversion.

Can you tell me if the following flowchart is correct? You can click on the image to go to the Google Drawing and edit it if you'd like, I've got a backup. I tried to read back through this thread and the Reverse Engineering thread to assemble the actual order of operations, but I'm sure I'm missing a few steps or I've got some things wrong:

(EDIT 10/7: Fixed step order)
Google Drawing

[eduperez]

My two cents:

• The CR2 file contains a small version of the JPEG file, with all the transformations applied.
• I do not think Canon would downsample to 8bit at the beginning of the pipeline.

[agentirons]

Thanks, @eduperez, I forgot about the jpg thumbnails, so I added that step.

You're right that it'd seem like you'd want to downsample near the very end of the process, but our investigation in the 'Reverse Engineering Picture Styles' thread has revealed that the picture style process is 8-bit as far as we can see (pending some further testing). The downsample might be misplaced in this graphic by a block or two but should be in the right area.

[dmilligan]

You're missing the block where Canon makes their sometimes dubious digital modifications to the actual captured raw data

[Andy600]

There is no down-sampling before demosaicing and 'if' there is down-sampling before the Picture Style it could actually be to 10bits, not 8. The logic behind this is the recently discovered CineStyle 10bit log curve i.e. why is it 10bit? (even though the CineStyle lut appears to do absolutely nothing this still might be a vital clue).

The 3x20 matrices seem to perform a camera to XYZ regression and are probably nothing to do with Picture Styles. They will only be used depending on the colorspace setting in the camera (i.e. sRGB or Adobe1998). I think the native Picture Styles are the same linear ICC profiles as used in DPP. The luts must be held in firmware and are applied in L*a*b* colorspace.

The '6000' lut is not a lut, it's a 3x3 RGB matrix that is the same as Adobe 1998 colorspace but chromatically adapted to D50 (Adobe 1998 is D65). With name like '6000' you would think that it might be 6000k or D60 but it's not. The 6000 matrix appears to be used twice in DPP, once to go from 6000 to XYZ and then in reverse from XYZ to 6000 with some color processing in between - I think this is likely where the Picture Style lut is applied.

The 6000toXYZ/XYZto6000 order may be the other way around i.e. the Picture Style lut may actually be added (in L*a*b* colorspace) after conversion to 6000 space then transformed back to XYZ. It's hard to tell using process monitoring.

A search of Canon Patents on Google might be a good place to look for schematics. I did look through a few but haven't found anything specifically related to DSLR Picture Styles yet.

[agentirons]

Okay, thank you very much for the additional clarifications. I've edited the image again based on how I'm understanding it now, although I've got additional questions:

1) Is the image data white balanced during the demosaic, or is it a separate process?

2) Is the demosaic what transforms the data into Canon's 6000 colorspace?

3) In order to apply the Picture Style LUTs in L*a*b* colorspace, is it necessary to transform the data from either 6000 or XYZ into L*a*b* and then do the Picture Style, or does the LUT handle this as a single step?

4) Is there a final conversion from either XYZ or 6000 into sRGB or Adobe1998, or does the initial 3x20 matrix do this? It seems like there must a final conversion since the latter two are their own unique colorspaces, and if it's the initial matrix then it would be like having the image be both XYZ and sRGB simultaneously, right?