SlimRaw – CDNG compression tool

[andicgn]

Where can i get Switch?

[Walter Schulz]

https://www.magiclantern.fm/forum/index.php?board=54.0

[cpc]

Hi, sorry if I re-open this old topic...
I've seen that also Swicth has an option for compressing DNGs (with LJ92) from MLVs, so which is the difference between compressing DNGs in Switch versus doing it with MLVFS+SlimRAW? (Believe me, as I wrote in another topic, I really don't want make a SW-vs-SW challenge: I'm just curious to know which are the differences between compressing DNG with Switch and with SlimRAW since both are in this forum and both are loved by users, and both have a similar function, so I suppose both have great PROs and perhaps are useful in different situations.)

And I confess I don't clearly understand if SlimRAW lossless compression methods is "visually lossless" or "data lossless", and if someone noticed any slowdown in Davinci playback with compressed DNGs (in comparison with regular DNGs).

Thanks a lot.

A bit late, but here are some answers for posterity.

First, your lossless DNG question is answered in slimraw's faq: http://www.slimraw.com/faq.html
Second, performance in Resolve with compressed DNG will depend on what's the bottleneck of your system when streaming the raw footage. If storage is the bottleneck, Resolve performance should improve. If the CPU is the bottleneck, performance may degrade.

As for the differences between slimRAW and Switch, Danne will correct me if I'm wrong, but my understanding is that Switch uses Adobe's DNG Converter. DNG Converter is no match for slimRAW neither in terms of file size achieved, nor in terms of speed (last time I checked, slimRAW was 32 times faster on a quad-core CPU). But hey, Switch is free and I am sure Danne is doing a great job with the whole Switch pipeline. If you aren't in a hurry, you may gain some additional space by passing DNG Converter's output through DNGStrip, a free tool that strips some of DNG Converter's useless chaff from the compressed files, which I've written some years ago: https://www.shutterangle.com/2014/lossless-compression-for-dng-raw-video-dngstrip/

[Danne]

Hi cpc! Long ago .
Switch is more or less a sleeping beauty nowadays. Mlv App is standard more or less. However compressed dng from mlv uses lj92 compression introduced by A.Baldwin in MlRawViewer: https://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html

Better than adobe dng and cdng but of course not as fast as slimraw .

[cpc]

Hi cpc! Long ago .
Switch is more or less a sleeping beauty nowadays. Mlv App is standard more or less. However compressed dng from mlv uses lj92 compression introduced by A.Baldwin in MlRawViewer: https://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html

Better than adobe dng and cdng but of course not as fast as slimraw .

Hi Danne. Good to know. Thank you for the heads up, I haven't been following developments closely for some time now. (Ok, I should probably say "at all" instead of "closely". )

I glanced through your link and it seems that he is describing Blackmagic's way of doing the prediction, which isn't particularly good for linear Canon raw. I would guess that DNG Converter + DNGStrip should produce smaller files (well, if one could live with the painfully slow DNG Converter, that is ).

[Danne]

Hi Danne. Good to know. Thank you for the heads up, I haven't been following developments closely for some time now. (Ok, I should probably say "at all" instead of "closely". )

I glanced through your link and it seems that he is describing Blackmagic's way of doing the prediction, which isn't particularly good for linear Canon raw. I would guess that DNG Converter + DNGStrip should produce smaller files (well, if one could live with the painfully slow DNG Converter, that is ).

Probably. You should know if anyone  .

[Levas]

I’m using adobe dng converter to compress ‘old’ uncompressed 14bit ML dng files.
Adobe dng converter has this option to only use lossless compression. I was always assuming adobe dng converter uses lj92 compression. But after last few posts in this topic, I’m in doubt? Isn’t adobe dng converter lossless compression option the same as lj92 compression used in mlv’s ?
And is there a difference when using MLVapp to extract lossless dng’s from a mlv file compared to use command line mlvdump to extract lossless dng’s?
(With mlvdump I always see adobe dng converter popup, so it’s probably used by mlvdump by command line)

[cpc]

I’m using adobe dng converter to compress ‘old’ uncompressed 14bit ML dng files.
Adobe dng converter has this option to only use lossless compression. I was always assuming adobe dng converter uses lj92 compression. But after last few posts in this topic, I’m in doubt? Isn’t adobe dng converter lossless compression option the same as lj92 compression used in mlv’s ?

Anything "lossless dng" uses the same spec when compressing the pixel data. But the spec allows for a lot of wiggle room and choices to make, which accounts for significant differences in compressed pixel data size. SlimRAW will produce files significantly smaller than DNG Converter and way smaller than anything that uses Blackmagic's setup for Canon linear raw (usually ~15% smaller, sometimes more). That's because slimraw tries a lot of stuff on the fly to get the smallest results (and even more stuff to try can be enabled through an option), while the others fix parameters in advance. Regardless, results will be truly lossless with any compressor, and compressed files will decompress to the same original file. (Well, DNG Converter will convert all input to nominal 16-bit regardless of true input bit depth, so the decompressed file will be nominally 16-bit, but that's mostly technical details.)

The thing is, DNG Converter is made for photos, and it has the peculiarity that it inserts thumbnails (previews) in each frame which certainly is pointless for video, and it adds a bunch of useless conjured metadata. All this bloats the output size. This is what gets stripped by DNGStrip. But the main issue that initially pushed me towards making slimraw back in the day was the excruciatingly slow processing of DNG Converter. Well, that and the fact that there was no log raw output in DNG Converter.

[Kharak]

I use Adobe DNG converter to convert CR2 to DNG, so I can grade them in Resolve. I then further compress them with Slimraw. and I believe it compresses a further 15% or so.

The lossless compressed DNG's from SlimRaw greatly improve the performance in Resolve, especially Magic Lantern DNG's. Now that prices on SSD's are plummeting, it is not so much a problem anymore, but I used to have huge projects on External HDD and the bottleneck would be the USB Read Speeds, especially after filling more than 50% of an external HDD, the performance drop gets clear, but with Slimraw keeping the Data Rate around 50 MB/s, I have real-time playback in Resolve (with heavy grading) because the USB speed is not a bottleneck any more. (Until you are down to 20% space left).

I've been overly happy with the software, its very fast! it has saved me tons of space on HDD's, especially old projects from before Lossless compression was introduced in camera and I had a bunch of 80-100 MB/s MLV's/DNG's stored. So I went back to archived drives that were more or less filled to the brim and compressed with Slimraw, gaining on average 50% more capacity on those drives. I don't put more footage on those drives, Instead I consider them full at 50%. Leaving them at 50%, I know I have best performance from those HDD's, should I ever use some of the footage again.

Quick tip. When compressing, if you are leaving your computer or spare laptop overnight, set it to Defragment the External HDD after compression, it made a huge difference in keeping footage Real time in resolve and not getting random freezes.