Messages

This type of noise is normal for underexposed footage. There simply isn't enough light fed to the sensor for proper clean darks. Furthermore, the noise is very visible here because your blacks are lifted with the log-ish profile; this also lifts the noise and makes it more apparent.

Neat Video will help a bit, but will soften the image. Better play with ACR's noise reduction first and see where it gets you. But the best you can do is pull black down to hide the noise. You will need to pull it down in correction anyway, because its too high for a proper image.

It is already too late, ofc, but as a general guideline: in low light scenes it is better to raise the ISO in camera in order to bring the image up. With 5dm3 raw iso 1600 is fine, 3200 is useable.

So the new version is out.
slimRAW can now recompress DNG/CinemaDNG files that are already losslessly compressed. For Canon ML raw footage compressed by other means that's 8-20% reduction of size.
Also, this version runs faster (140+ fps for fullHD on my stock quad i7) and compresses better. Plus, there is a Max compression option for people who like to squeeze as much as possible.

Full release notes here: http://www.slimraw.com/relnotes.html

I think Resolve actually honors the forward matrix. At least Resolve 10 did last time I checked in the beginning of 2014.

Reason for 16 bit is computational.. Or so I read on these forums quite some time ago.

It is more CPU intensive with 14 bit compared to 16 bit, because 14 bit is not on pair with 2-4-6-16-32 etc..

I believe it was in the The CinemaDNG Discussion thread that Chmee mentioned this and therefore 2 extra "empty" bits are added in RAW2CDNG conversion.

Well, there is a minor unpacking overhead with 14-bit uncompressed files, but that's probably offset by the fact that 1/7 more data needs to be read from storage with 16-bit uncompressed files; and storage is the slowest link in the processing chain.

With compressed files there is no overhead with 14-bit files compared to 16-bit.

@Kharak:
12 and 16 bit dng will work fine, but for normal Canon ML footage there is no reason to lose precision converting bitdepth down to 12-bit, nor is there a reason to increase size going 16-bit. 16-bit in particular has no advantage whatsoever, and the compressed 16-bit files will be marginally bigger compared to the 14-bit compressed files.

Dual ISO is another story, since it is 16-bit already, AFAIK.


@Markus:
I am not sure why would you want to do this? The compressed cDNG is already compressed (obviously), so it is good for both archiving and production as is since lossless CinemaDNG is standard and compatible with a wide range of software. And if you only want to quickly preview sequences, you can drop them in either Scratch Play or Resolve Lite.

The new version of slimRAW adds a trial mode. In trial mode a few frames of each dng/cinemadng sequence will be converted. This way expected compression ratios can be previewed and the compressed frames can be tested and pixel peeped in Resolve, Premiere, Speedgrade, Scratch, etc.

Thanks Andy for supporting slimRAW and thanks Tim for posting the link.

I believe Andy means he has converted the original uncompressed files with slimRAW and the compressed output works in Premiere/Speedgrade CC. I would be surprised myself if slimRAW actually makes DNGConverter files compatible with Premiere. It will in the future, but does not currently.

Uncompressed DNG and CinemaDNG should work fine in Premiere/Speedgrade CC after slimRAW compression. In fact, if a slimRAW output file does not work with Adobe CC I'd very much like to have a look at the original file.

@SamoMalo404: slimRAW will work on any x86-64 processor. Obviously, the more power you have available, the better. But it is fast enough that in the majority of cases storage bandwidth is the bottleneck, not the CPU.
FWIW, on my quad-core Intel i7-4770 3.4ghz running at stock core speed FullHD 5d mark3 14-bit DNG footage converts at around 95-100fps (off an SSD). Now obviously you need fast storage to achieve such speeds (SSD and/or RAID), an HDD simply can't do it physically. In any case, slimRAW will work with whatever is available on the system, at the bandwidth available.

You can read a bit more here:
http://www.shutterangle.com/2015/slimraw-cinemadng-raw-video-lossless-compressor/

ACR uses the UniqueCameraModel dng/tiff tag to identify cameras and enable and display the corresponding profiles. You can batch add it with exiftool to your files.

Most of you are probably familiar with losslessly compressing DNG raw video with DNG Converter. The compressed DNG sequences work fine in both Resolve and ACR.
I've written a small tool (DNGStrip) to strip these of useless stuff which is unconditionally added by DNGConverter to each frame: a thumbnail and some pointless tags. Mac and Windows versions, free to use. Pretty fast. Download link in this article: Lossless compression for DNG raw video + introducing DNGStrip. The text also gives more context on the whole process.

In short, compressed 16:9 frames will lose 114kb per frame, compressed 2:1 frames will lose 102kb per frame after going  through DNGStrip. Test footage (2:1 1920x960 Canon 5dm3 raw) was reduced from 710gb to 399gb with DNG Converter and then further down to 377gb with DNGStrip. That's 53.1% of the original size for full original image quality and average bitrate only 317 mbps. Not bad, methinks.

Embedded - what's this?

That's the profile calculated from the metadata tags in the DNG file.

Maybe it's different if working in ACES with input transformations? but Resolve only has them for the 5D (doesn't say if its Mark 2 or 3), 7D and 1D.

Canon cameras ACES IDTs in Resolve are for camera native codec footage (although unclear to me what picture profile is assumed).
As the DNG tags should technically describe the camera space fully, any DNG footage is imported with the CinemaDNG ACES IDT, no matter what camera it comes from. ACES workflow in Resolve is still a bit rough though.

Just a quick correction: Resolve 10.1 most certainly uses both the color matrices and both the forward matrices. I haven't bothered checking with previous versions.

if we only have one matrix, there should be only one illuminant, the D65 one, correct?

Technically, yes.
Resolve will just use the one matrix supplied over the entire illuminant range anyway. ACR appears to do something more, but the difference between the illuminants is practically imperceptible with the same matrix.
The sizeable difference will only show after the second matrix (and, eventually, the CameraCalibration matrix) is added.
I believe dcraw got their matrices from Adobe. Shouldn't be a problem, these are just numbers.

As per DNG spec, there are a few tags that define the conversion of the raw data from camera space to XYZ space.

Here they are:
CalibrationIlluminant1       
CalibrationIlluminant2       
ColorMatrix1                 
ColorMatrix2                 
CameraCalibration1     
CameraCalibration2     
ForwardMatrix1           
ForwardMatrix2           
ReductionMatrix1
ReductionMatrix2
AnalogBalance
ProfileHueSatMapDims
ProfileHueSatMapData1
ProfileHueSatMapData2

DNG Converter doesn't produce Reduction Matrices for canon raw files, so these can probably be ignored. AnalogBalance seems to always be 1 1 1.
Everything else is used by ACR when developing a Canon raw file turned into .dng (by Adobe DNG Converter)
Tags with 1 and 2 at the end correspond to the respective calibration illuminant. While using both the illuminants is optional, these allow for a more precise mapping at different white balance settings.

Here is what DNG Converter always outputs from a Canon 5d mark 3 .cr2 raw file:
Calibration Illuminant 1        : Standard Light A
Calibration Illuminant 2        : D65
Color Matrix 1                  : 0.7234 -0.1413 -0.06 -0.3631 1.115 0.285 -0.0382 0.1335 0.6437
Color Matrix 2                  : 0.6722 -0.0635 -0.0963 -0.4287 1.246 0.2028 -0.0908 0.2162 0.5668
Camera Calibration 1            : 0.9923 0 0 0 1 0 0 0 0.9843
Camera Calibration 2            : 0.9923 0 0 0 1 0 0 0 0.9843
Forward Matrix 1                : 0.7868 0.0092 0.1683 0.2291 0.8615 -0.0906 0.0027 -0.4752 1.2976
Forward Matrix 2                : 0.7637 0.0805 0.1201 0.2649 0.9179 -0.1828 0.0137 -0.2456 1.057

These are model based, except Camera Calibration, which in theory is individual camera specific. I guess the way to check if CameraCalibration is indeed individual camera specific is to compare with the same tag from other 5dm3 cameras. CameraCalibration has a noticeable effect on the resulting image (well, with these particular values, at least), slightly boosting reds and blues.

And here is what comes from raw2dng:
Calibration Illuminant 1        : Standard Light A
Calibration Illuminant 2        : D65
Color Matrix 1                  : 0.6722 -0.0635 -0.0963 -0.4287 1.246 0.2028 -0.0908 0.2162 0.5668

Note how the matrix used is actually the matrix for the D65 illuminant (as specified by DNG Converter) but here corresponding to Standard Light A (tungsten) instead. The second illuminant will be ignored by raw processors because there are no matching matrix tags. As a result the raw processor only uses this matrix instead of interpolating between the two based on chosen WB.

ForwardMatrix1 and ForwardMatrix2 are applied by both ACR and Resolve but appear to have a minor effect. I actually expected Resolve to ignore them, as they are absent in BMCC files, but they are applied.

ProfileHueSatMapDims, ProfileHueSatMapData1, ProfileHueSatMapData2 define hue/saturation/value mapping tables which are applied last, and actually have significant impact on the final image in some parts of the color space. These are read and applied by ACR, but ignored by Resolve. Note that these aren't exactly small and add around 64KB to the file. These are binary tags and can be copied from an existing DNG Converter created .dng still onto the video .dng files (by ACR users, anyway).

So I'd suggest adding to the EXIF:
CalibrationIlluminant1       
CalibrationIlluminant2       
ColorMatrix1                 
ColorMatrix2                 
ForwardMatrix1           
ForwardMatrix2           

CameraCalibration1 and CameraCalibration2 should be added only if they turn out to be model specific and not individual camera specific. In the latter case probably an option to specify them explicitly would be nice? Handling the HSV table tags is probably an overkill (and increases size significantly), and it is easy to add with ExifTool anyway.

[Note that the values listed above only apply to 5dm3.]

The above doesn't touch on the black/white levels based on white balance, which is another story.

Hope this helps.

I'm preping to start doing some post on a few months old raw project shot on 5dm3 and I was checking the DNG files I got. Off-the-camera files were converted with RAWanizer.
What strikes me as odd is that in the resulting .dng files the tag for ColorMatrix1 uses the daylight (D65) matrix, but the corresponding CalibrationIlluminant1 tag is Standard Light A (Tungsten).
There is also no ColorMatrix2 tag, although there is a CalibrationIlluminant2 tag (D65). I had a quick look at the source code and it appears this is coming from raw2dng and not from RAWanizer.

If you wonder what the visual effect of these matricies is in a raw processing software: there is subtle tint movement mostly over the green-magenta axis, but also skin moves subtly from yellow to pink. This can be verified in ACR or Resolve.

I can fix this easily with exiftool, and also add the second color matrix to my files, but before doing this I wonder if there is a specific reason for what looks like a mismatch in the values used in these tags? Thanks.
(I wasn't sure if this is the correct subforum for the question, pls move the post if appropriate)