Messages

@megapolis - I'm not sure where you got your Cinelog profile(s) from? - but Cinelog-C profiles don't contain HSV luts and they will only work correctly in Adobe Camera Raw.

@bpv5P

If you want the approximate same colors as you get with H.264 then convert Cinelog-C to one of the DSLR looks i.e. if you shot H.264 Neutral choose Cinelog Neutral Rec709 in the OCIO config.

re: Which Impulz luts to use? I would suggest watching some of Denver Riddle's Impulz videos.

As @hyalinejim suggests, Cinelog-C uses the same primaries as Alexa Log-C and conveniently the difference between Cinelog-C (when applied to a ~11stop raw image from a Canon DSLR) and a normalized Log-C shot from the Alexa are very close. When you add a log-c lut Cinelog-C will probably look a little over-exposed compared to an actual Alexa shot but you can usually use Log-C luts IF they were developed for the Alexa by tweaking the exposure offset before the output/look lut.

@hyalinejim - I'll check out your Ektar lut later - your images look nice! . We've actually got several rolls of that here and planning to profile it soon.

@NoCp_Albert

MLV Dump and MLVP should work fine as long as the correct camera model tags are written. If cam profiles (not just Cinelog) specific to the model are selectable in ACR then the tags are correct.

The older raw2dng with .raw format wrote a generic model tag and fixed 5D3 matrices which meant either using a custom profile or re-tagging and adding extra exif data so MLV is the better option.

MLRawViewer was one to avoid because it saved lossy DNGs.

I really need to do a full evaluation of all new/current apps to verify their DNG exports especially since the addition of 10/12/14bit recording but most 'seem' to do ok.

@budafilms

There is currently no 'stable' OFX solution for OCIO in Resolve but we're working on something

For colorspace management in Resolve Lite you have the option of Resolve's Colorspace Transform plugin, ACES workflows or Lut solutions (like Cinelog).

If you have Resolve Studio (the paid version) you also have DCTL. Paul Dore (@baldavenger) has built quite a few useful DCTL scripts: https://github.com/baldavenger/DCTLs

If you want a better insight into HDR SGO made some good introductory videos but unless you're grading for HDR on a HDR compliant monitor it will not be of any practical use to you. It is worth understanding the concept though

https://vimeo.com/184542836

https://vimeo.com/197910909

https://vimeo.com/200670359



There are a few more on their Vimeo page https://vimeo.com/user45728073/videos

I tried mlv_rec (and mlv_play + raw_twk) with the nightly experimental version for the 50D using non-crop, 5x crop and 10x crop at 10, 12 and 14bit.

10 and 12bit recording exhibit frame tearing with the lower portion of the image always being frame 1 but there were no colored/noisy corrupt frames. 14bit has no corrupt frames.

I don't remember the 50D ever recording in 10x but it does now  (though it might just be that I have never used it before  )

You can find references to hard-coded calibrations by searching ACR plugin in notepad or other basic reader. Obviously, being code, most of what you see is random characters but the camera models that are supported show up in English. Try looking for a localized tag to see if it's supported.


EDIT:

Interestingly, if you reconvert your de-tagged DNG (or any detagged DNG) with DNG converter it adds a UniqueCameraModel tag (UCM) called 'Digital Negative'. This means the local tag is not sufficient enough to identify the camera to the converter or ACR (as I thought - else it would have added the usual UCM tag) but it likely needs the tag filled for validation (need to recheck DNG specs).

ACR will usually read a DNG regardless of which model tags are used but will default to an 'embedded' calibration if no tags are included or recognized. This means only the embedded meta is used. The embedded meta might or might not be correct depending on how the calibration was obtained and the type of calibration that was performed. Think of this as tier 3 calibration - i.e. there are typically one or 2 color matrices allowing ACR to find a good auto white balance if needed. Some ML apps also embed forward matrices which offer some degree of color tweaking to the white balanced image.

If a model tag other than UniqueCameraModel is recognized by ACR (i.e. it is from a camera that has a calibration hardcoded into ACR) then ACR will use its internally coded calibration for that specific camera model and embedded color matrices will be overridden. ACR will show something like ACR V9.x for recent cameras and ACR v4.x for older cameras depending on when the last update was for that model. Think of this as tier 2 - it is calibrated (by Adobe) but doesn't allow selection of camera specific color profiles. Forward matrices are used only if embedded in the DNG.

If the UniqueCameraModel tag is filled then ACR will let you select alternate profiles (Adobe Standard, Camera Standard, Camera Neutral etc) where available. Think of this as tier 1 with the most options and typically the best, most up-to-date calibration. Embedded calibration tags are overridden by the profile.

The UniqueCameraModel tag is used by most, if not all, cam manufacturers that have cams that capture to the DNG/CinemaDNG format (BMD,Digital Bolex, Convergent Design, Zenmuse etc). It is the most commonly used identifier and may or may not be important for accessing camera-specific calibration and certain features in some NLE and color apps.

Regional tags are best translated to uniqueCameraModel tags if possible/required. In terms of calibration and color, there are no internal differences between the US, Japanese and Euro versions of a camera. The main difference is usually the front badge.

The only time the UniqueCameraModel tag should be purposely omitted (or better still, changed to something unique for the purpose of identifying a specific camera) is when a specific sensor has been purposefully lab calibrated and where it's images will usually only be developed in ACR i.e. to ensure this very specific calibration is not over ridden in any way by ACR - this is usually only done for industrial or research reasons.

@JeDail,

This looks like a workflow related issue.

Can you send me the dng and I'll check. 

@mdchev - I trust everything was sorted to your satisfaction on the 16th?

The enemy here is the h.264 compression.
h.264 is made to compress, it will look for areas with small color and brightness difference and compress it in a way that it gets the same color/brightness values and thus saving data(and losing detail).
So what do you think happens when you feed h.264 a flat, low contrast image...it will find lot's of areas with detail that has such small difference in color/brightness values, that it can compress the hell out of it 

Maybe you should experiment in filming the same scene with two different picture styles, one flat, desaturated profile and another profile, like 'landscape'.
I think the non flat picture profiles gives you bigger files, even with All-i compression.

That said, you can't color correct much in a non flat picture profile...

Makes sense. Maintaining contrast and saturation should produce a superior encoded H.264 image. I need to test the matching functions I tried between ProLost and log Pic Styles but substitute ProLost for default Picture Styles settings (with sharpening off) to see if perceivable latitude increases - I guess grayscale might take a hit but color will be better.

It's not so much the Picture Style but the 8bit internal H.264 encoding that makes log picture styles ultimately detrimental to latitude in the final image (when compared to simple flat settings aka ProLost settings).

The only potential benefit is through raising the black level to just above the point where the H.264 algorithm gets really bad i.e. deep shadow detail. You can actually get away with relatively fewer code values in deep shadows without there being much impact on the image - this is how most modern log functions are designed (Log-C, BMD Film etc). Raising the black level in a PicStyle emulates this.

Technically speaking you can compress any camera's DR into an 8bit space but anything over ~6stops gets extremely lossy. There are so few code values written that the image will just fall to pieces when you try expanding it i.e. the more grayscale range squashed into 8bits = the bigger that gaps/holes will appear between cv's when expanded = bad banding/contouring.

Cinestyle was initially developed solely to make inter-cutting with log footage easier, not to maintain greater DR (see the Picture Styles thread for some interesting findings). The same image can be achieved after the fact using flat settings and a lut so there really is little point to log picture styles. Picture styles that emulate Canon Log but recorded to H.264 is just asking for trouble - it will seriously impair highlight information. It might not be noticeable in well lit, normal DR shots but throw some big contrasts or gradients in there and it will soon show it's weakness. I predict it to be much worse than Cinestyle.

I actually did an experiment a while back (still need to write it up) where I produced perceptual log images that matched Cinestyle, Marvels etc from a ProLost source image outside of the camera using simple 1D luts. This maintained the most DR and latitude (latitude is important) and would still allow inter-cutting with log footage in a much less destructive way through inverting the lut i.e. Cinestyle to Rec709 = visible banding in Rec709 space whereas Rec709 (flat settings) to Cinestyle = little or no banding in Rec709 space - because it is already (perceptually) encoded as Rec709.

So to answer your question. Flat Picture Styles and Log Picture Styles are 2 different things. Relative to 8bit H.264 internal encoding, a Flat PS can be useful but a Log PS will typically throw away more information when there is only just enough to start with. Snake Oil - probably yes! BUT when there are more than 10bits to play with things get flipped and a 10bit log encoding is a much better way to store the information.

...from here it's harder

there should be a sub that actually loads the address of the raw buffer, you need to find that! how? well, read the lv_raw_dump function (the bl from the string you found) and check when the return pointer is used for reading or writing something

Ok, but I don't understand the last bit 

the bl is loc_ff064830 but what do I do with it? How do I check?

I've disassembled the 50D's FW and found 4 instances where 'lv_raw_dump' is mentioned but after that I'm lost  - yes I did read @dmilligan's post (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969) but I still don't know what I'm looking at.

I don't know about Adobe using DCRaw. They invented DNG and have access to developer SDKs for Canon, Nikon etc so why would they?

re: altering matrices: As, I said, you need to compensate if you remove the contrast-referred HSV luts used in most DCP profiles just to maintain a degree of calibration. The color matrices are intended for white balance - color is just a fallback function if no forward matrices are included. The HSV luts are contast-referred (tone curve dependent) so switching the tone curve off (i.e. creating a DNGPE linear profile) makes those HueSatDelta tables invalid because DNGPE doesn't dynamically adjust their values to account for the new 'linear' tone curve.

A 'linear' DCP created with DNGPE is linear only in terms of the tone curve. You could call this 'perceptually linear' but it is still display-referred and will clip values outside 0.0 - 1.0.

You can't get a full range unclamped scene-referred linear image directly from ACR. Exporting a linear profile from DNGPE also affects color (sometimes by a lot depending on the camera) because the 1 or 2 HSV luts in most DCPs are contrast-referred corrections i.e. they need the tone curve. If you just remove the HSV components you need to factor in a degree of additional calibration when calculating new forward matrices.

Can we assume that DryOS 2.3 is likely in Digic4 cams too because the copyright notice is dated pre-Digic5? Those strings look VERY interesting! 

There is no down-sampling before demosaicing and 'if' there is down-sampling before the Picture Style it could actually be to 10bits, not 8. The logic behind this is the recently discovered CineStyle 10bit log curve i.e. why is it 10bit? (even though the CineStyle lut appears to do absolutely nothing this still might be a vital clue).

The 3x20 matrices seem to perform a camera to XYZ regression and are probably nothing to do with Picture Styles. They will only be used depending on the colorspace setting in the camera (i.e. sRGB or Adobe1998). I think the native Picture Styles are the same linear ICC profiles as used in DPP. The luts must be held in firmware and are applied in L*a*b* colorspace.

The '6000' lut is not a lut, it's a 3x3 RGB matrix that is the same as Adobe 1998 colorspace but chromatically adapted to D50 (Adobe 1998 is D65). With name like '6000' you would think that it might be 6000k or D60 but it's not. The 6000 matrix appears to be used twice in DPP, once to go from 6000 to XYZ and then in reverse from XYZ to 6000 with some color processing in between - I think this is likely where the Picture Style lut is applied.

The 6000toXYZ/XYZto6000 order may be the other way around i.e. the Picture Style lut may actually be added (in L*a*b* colorspace) after conversion to 6000 space then transformed back to XYZ. It's hard to tell using process monitoring.

A search of Canon Patents on Google might be a good place to look for schematics. I did look through a few but haven't found anything specifically related to DSLR Picture Styles yet.

Try their Facebook or Instagram pages.

[if]

...I believe that and my research seem to verify that Pic.Style are really ICC profiles that have acr like adjustments applied to the Base Matrix.
And all cameras base matrix are not all equal to each other.

You are correct that most cameras have a different calibration matrix because they have different sensors with different spectral sensitivities. I say 'most' because some Canon DSLRs share the same sensor. This all happens before the chosen Picture Style is applied and is not something you have control over in camera. You can change white balance, tint offset and exposure but not the cam2xyz matrix except for raw images where these coefficients are stored as metadata.

Picture Styles are applied to the demosaiced, color balanced, normalized image i.e. Picture Styles are added very late in the chain of processing - too late for an effective log conversion.

With respect to CineStyle pf2 file , I do firmly believe that you are really applying not only the base adjustments saturation , contrast , sharpness , etc... .
but a color matrix or in this case Log color space .

Cinestyle does not alter the colorspace. It's sRGB or Adobe 1998 depending on your camera settings. The base profile is neutral so you can say it's non-linear because it has targeted adjustments made to colors and you don't have control over that either.

As for log, well there is definitely a 10bit log curve in Cinestyle but it looks like it does nothing (see the research in the other Picture Styles thread). An RGB tone curve performs the 'lin2log' transform but, after some research, I'm seriously doubting it's credibility. i.e. it can probably be improved upon but will still be limited because of when it is applied.

Since I have a 5D2 the cinestyle pf2 file was design for my camera so I have no need to reverse engineer it , but what I do see here is with raw video . Knowing ML dose I think do some
XYZ conversion to RGGB raw data . My thought was if with could apply some sort of Log space that's save to mlv , maybe it not possible not sure.

There more to come , hope so far this information is useful to someone . 

Cinestyle was developed using a 5D mark II but it does not contain anything specific to the 5D mark II i.e. the data that goes into Cinestyle should be more or less the same regardless of the camera if the cameras are setup the same and the pre-picture style calibration process is uniform across the Canon range.

re:raw log - You can post process a log conversion of the raw data to reduce it's bit depth (see SlimRaw) or compress the tonal range and gamut into a log colorspace to reduce it's footprint (and other benefits) but MLV itself doesn't do this kind of thing. The raw2dng converter adds metadata that is then read by the app that debayers the images (Resolve, After Effects, DCRaw etc) and it is then that the color calibration, white balance, normalization etc happens depending on the embedded meta and any overrides you may choose.

@Deadcode - The sharing of commercial products here is prohibited so don't ask.

Right, so, .pf2 has an RGB Gamma curve and a L*a*b* Gamma curve that both work, both only the L*a*b* Gamma curve is directly editable inside Canon's Picture Style Editor software (it's the curve box in the 'Specific Colors' tab). The RGB Gamma curve has to be edited manually using the hex editor

Correct

..and I'm guessing only one is applied to the image if you have data in both curves.

No, they can both be applied

PSE can edit the RGB Gamma curve if you save as .pf3. (Assuming here, I haven't actually tried it to confirm that's what's happening.)

You can edit the pf3 tone curve in addition to the RGB gamma curve data but any hex data you enter into the RGB gamma block does not show as control points in PSE - I really don't get what is going on with all these curves!

The separate and much larger 'User' RGB Gamma block that is present in the Cinestyle .pf2 has seemingly no effect, and @Andy600 is trying to figure out it's purpose.

Yes and still trying to figure it out . I did think that maybe it was applied only to H.264 and that I might not be seeing the true result of the curve due to the colorspace I was in but I've checked tweaked Cinestyle H.264s in Rec709, sRGB, Linear and across multiple color apps and a couple of NLEs - still the same result. The curve does nothing here. The last hope is HDMI output but I doubt that's going to be anything different.

PSE and DPP don't seem to like independent R,G & B Gamma curves either

No. I'm saying the User RGB Gamma block in the CineStyle.pf2 (i.e. it's log curve) doesn't seem to do anything. You get the exact same image out of DPP or an in-camera Jpeg or H.264 whatever data is in that block and I've changed the curve data multiple times and even removed it to prove this - you still get the same Cinestyle image.

The .p3f format introduced an editable RGB tone curve and a couple of other things to the PSE app where previously you had only a L*a*b curve.

@agentirons - I didn't try independent RGB curves yet but it's easy to verify the order by making 2 of the 3 curves all 0s and see what color you're left with. I would assume it's RGB in that order but could also be BGR.

If you convert each byte of the matrix you get a value between 0 and 255 so they are 8bit values but I don't know if the order is correct yet - it probably is. 20x3 is tricky math There are toolkits in Matlab (and probably Octave, Scilab etc) but I need to do some research and learn it - I doubt I've got time to do that this week unfortunately.


Thanks @chris_overseas - your decoder is a super useful tool!!

[L*a*b* colorspace]

I replaced the RGB gamma blocks but the points will not show in PSE because there is/was no editable Tone Curve in a pf2. Cinestyle is a pf2 and has a tone curve so it's just a PSE omission. The editable Tone curve was added later (presumably under a different tag?) as part of the pf3 format.

If you change the LAB gamma block remember it is in L*a*b* colorspace so changing the first set only will produce a change in luminosity. Changing all three will cause color issues - this is why PSE used to be a complete pain because you couldn't alter each curve independently.

As for building the curves, I did originally try mathematically plotting a known log curve (in this case a Cineon curve plus 2 EV), by converting the expression to an 8bit lut (256 CV) and then took 10 of those values as 'y values' to plot against a 0-255 linear set ('x values'). I then entered the x and y values in PSE and applied the PS to the CR2 in DPP and saved a jpeg. Then I just compared it to the same raw image (a Stouffer transmission wedge) debayered to linear and converted to Cineon plus 2EV (the +2EV was purely to remind me that I was looking at a specific test image). This was all assuming that Neutral -4 contrast was somewhat linear.

Even accounting for some small offsets there were differences caused by the contrast curve (neutral -4 because it's not strictly linear) so I then did it the hard way by roughly setting the black and white levels before saving a jpeg then set up an A/B i.e. the 2 images with a layer mask to show in split screen. Had to desaturate the images because the As Shot WB rendered slightly differently between DCRaw and DPP.

You need to gain the images up and down as you work to ensure you're matching/fine-tuning the shadow values properly. As you plot the additional points you need to move the points accordingly to produce the smoothest curve (i.e. if one set doesn't work because the interpolation causes a kink then try the next set - it can be like herding cats) but even so, 10 points isn't enough - it's very tedious and laborious work

note: I made the curve in PSE before the decoder became available so I already had the curve points ready for conversion to hex.

I'm wondering if we can alter the file size to increase the RGB and/or LAB gamma blocks to 10bits or if it will only work in 8bits with 12 points. Also, I'm still trying to work out these 20x3 matrices which appear to be used for a third-order polynomial regression to XYZ space (a fairly common practice in calibration).

Cinestyle is definitely in RGB space (the sRGB matrix is all zeros), so assuming the Adobe matrix is correct it should be possible through derivation to configure a matrix to output RGB in another colorspace either by default by re-tasking the sRGB matrix or as an option by changing the Adobe1998 matrix and selecting Adobe1998 in the camera (it's a better idea to be able to select IMO).

I'm using Matlab but for hex2dec conversion you can use a spreadsheet or something like: http://www.asciitohex.com/ and to edit the decoded file, this is good http://www.sweetscape.com/010editor/


There is a remote possibility that pf2, pf3 files are close enough to ICC standards that we might be able to add float expressions instead of hard-coded luts i.e. y=x^2.2 etc etc and maybe substitute the 3x20 matrices for simpler 3x3 chromatically adapted RGB matrices - there is lots to play with and discover

To do any of this more accurately/more easily the contrast function needs to be modeled to enable linearization and I think it's probably a SINE function!?

Yes, pretty much. I built a new 12point curve, converted to hex then replaced the RGB gamma blocks. I thought I'd uploaded the one with a linear User RGB Gamma block but here it is: https://s3-us-west-2.amazonaws.com/ofxpublic/Cineon_Log_plus2EV_lin_tone_curve.pf2

You'll see the resulting output is no different than the profile with a log curve in the User RGB gamma block - I'm still trying to work out what that log curve is for? if anything Might it be something exclusively to do with HDMI output? (though I don't know if that's even possible) or is it just a dummy curve? (that would be a shocker ) because as far as I can tell it does nothing for H.264 or Jpeg.

No ill effects that I can see although the unlock was done before decoding the profile for the curve replacement.