SlimRaw – CDNG compression tool

[eyeland]

Nice work!
Would someone care to make a size comparison between Slimraw lossless cDNG and log Pro Ress 4444 XQ ?

[hjfilmspeed]

@cpc  "(I am now getting 150+ fps when compressing 14-bit FullHD raw on an i7 4770), so updating is recommended even if you don't care for lossy compression."
I don't get this. Not even close! I would settle for real time. How are you getting this?

[Kharak]

Compress on the fastest drive you have, preferably SSD to get High speed Compression.

[cpc]

@eyeland:
Lossless CinemaDNG size varies with content, but if we assume an average compression ratio of 2:1 for ML raw, the calculation goes like this:
FullHD 14-bit lossless CinemaDNG: 1920*1080*14/8*24/2 ~= 43.5 MB/s (at 24fps)
FullHD 12-bit ProRess4444XQ: 396 mb/s = 49.5 MB/s (at 24fps)

@hjfilmspeed:
As noted by Kharak, you need a very fast storage setup to hit these speeds. For 150fps you'll need almost 550MB/s sustained storage reading speed and around half of this rate on the output end. This means very high end SSD, and preferably a RAID of SSDs. For real time processing, you need a sustained reading throughput of around 87 MB/s (at 24fps). If you are processing straight from the CF card (mounting raw through MLVFS), you are most likely limited by the throughput of your CF card + CF reader combination (many readers are way slower than the CF card itself).

[reddeercity]

MLV --> Cdng 16bit -->Log ProRes4444XQ+alpha=16bit 
Which would be the Very best format to keep your ML files archive as
Since it's Linear , there's really no advantage to keep MLV file's or compressing the Cdng's

Plus All Apps (Mac) (some on PC if you have the PC ProResXQ Ver.) , support ProResXQ now , and there no need to have expensive hardware .
Just standard SSD or fast spindle drive e.g. WD VelociRaptor  600GB 10K RPM (I have 2 in a raid 0 on MP5.1)
Just about any midrange Laptop/desktop will do.

[cpc]

MLV --> Cdng 16bit -->Log ProRes4444XQ+alpha=16bit

Color channels in ProRes4444XQ are 12-bit. Only alpha is 16-bit and it is optional, you will generally want to skip it, cause it only inflates size in this case.

[eyeland]

Lossless CinemaDNG size varies with content, but if we assume an average compression ratio of 2:1 for ML raw, the calculation goes like this:
FullHD 14-bit lossless CinemaDNG: 1920*1080*14/8*24/2 ~= 43.5 MB/s (at 24fps)
FullHD 12-bit ProRess4444XQ: 396 mb/s = 49.5 MB/s (at 24fps)

Thanks, Thats really nice I second the suggestion above about a container to avoid having to store them as individual files

[mothaibaphoto]

Comparing DNG with prores or any other delivery/intermediate format is just like comparing apples and oranges: DNG allows me to reprocess the raw data anytime later, while others - not. The result of processing the raw data very depends on a) software used b) personal skills. Both things tend to develop vastly. Debayer software evolves rapidly. Just look what a progress Resolve did. I just reprocessed in 4K a dozen of timilapses I shot a long time ago on 400D and I managed to get better results than in HD that time. And when new version of Resolve or ACR, or maybe MLVProducer will bring new debayer feature, I can reprocess my files again, to get better image.

[Danne]

@motbaiphoto +1.

[PeteAG]

Hi, I made a little quality comparison video yesterday to see how well SlimRAW behaves against the original CDNG and all ProRes flavors.

https://vimeo.com/153739453

I have a lot of Footage from my recent Iceland trip and really want to archive my RAW files as small in size but best quality as possible.
SlimRaw is a superb tool and the support is fast and professional.

The downside is that compressed CNDGs are only up to Resolve at the moment (don't know much about the Windows and Linux side).

[cpc]

New version 1.5 is now released with some improvements. Release notes here: http://www.slimraw.com/relnotes.html

[strikermed]

I have a few questions before I commit to purchasing SlimRAW.  The first and most important is about performance in Premiere and After Effects.  Currently I have an i7 3930K and have a server feeding a 10GbE pipe with 650MB/s of read and write performance.  As it stands I get choppy, and drop frame performance reading 16bit CDNG content on Premiere Pro.  I converted these the MLV files using RAW2CDNG (which I has proven to be the best performance and reliable out of all the options out there).  I want to know if this will actually improve performance, as resolve plays the files fine, and PPcc has hiccups.  This has lead me to believe that PPcc is the issue over file size and data rates.  So, can anyone give me a reasonable explanation about why slimraw would make playback and editing better in PPcc?

Second, I would like to know if anyone has had any issues with crashing or bad frames in the compression process.  I've had my fair share of issues finding the right software to do the conversions from MLV, and I finally found a version of raw2cdng that works pretty reliably.

Third, how does licensing work?  I generally shoot for myself, but I use a computer at work to process footage occasionally.  Do I need to purchase two separate licenses, or will it allow me to use the same license on each computer (like Adobe products) as long as I don't have both using the same license at the same time?

Thanks, looks like a cool product to save some expensive RAID storage.

[cpc]

I have a few questions before I commit to purchasing SlimRAW.  The first and most important is about performance in Premiere and After Effects.  Currently I have an i7 3930K and have a server feeding a 10GbE pipe with 650MB/s of read and write performance.  As it stands I get choppy, and drop frame performance reading 16bit CDNG content on Premiere Pro.  I converted these the MLV files using RAW2CDNG (which I has proven to be the best performance and reliable out of all the options out there).  I want to know if this will actually improve performance, as resolve plays the files fine, and PPcc has hiccups.  This has lead me to believe that PPcc is the issue over file size and data rates.  So, can anyone give me a reasonable explanation about why slimraw would make playback and editing better in PPcc?

Second, I would like to know if anyone has had any issues with crashing or bad frames in the compression process.  I've had my fair share of issues finding the right software to do the conversions from MLV, and I finally found a version of raw2cdng that works pretty reliably.

Third, how does licensing work?  I generally shoot for myself, but I use a computer at work to process footage occasionally.  Do I need to purchase two separate licenses, or will it allow me to use the same license on each computer (like Adobe products) as long as I don't have both using the same license at the same time?

Thanks, looks like a cool product to save some expensive RAID storage.

re: Premiere performance
Strange that you don't get realtime playback in Premiere with your specs. I haven't tried 16-bit uncompressed files, but I get smooth playback of 14-bit losslessly compressed 1080p DNG on my quad i7 4770. Have you tried processing from local storage? Could be Premiere doesn't like the network storage setup for whatever reason?
To answer your question, it is possible that using lossless DNG will improve the situation since Premiere would need to stream significantly less data, and the lossless compresson will be no problem for your hexcore cpu.

re: stability
FWIW, I've never had a crash report in the year and more that slimRAW's been out. Obviously, if you have a bad frame before compression, it will stay bad after compression. No way around this.

re: licensing
slimRAW can be used on all your computers with one license, and this includes both Mac and Windows platforms.

[cpc]

New version 1.6 is now out with 4 new compression modes.
Added 5:1 lossy compression and two variable bit rate lossy modes. But perhaps most interesting for ML users will be the new 10-bit log encoded lossless mode. The linear 14-bit signal is mapped to a 10-bit coding space using a log tonal curve, similarly to Canon C500 10-bit log raw. Log data is then losslessly compressed. This results in 20-30% smaller files than straight lossless compression, while preserving high image quality. The log tranform is transparent, it is reversed by the raw processor on import; no workflow changes required. The output is compatible with pretty much all DNG capable video software.

All lossy compression now also uses a non-linear transform, which allows higher levels of compression by keeping stress off dark tones.

More about the different compression modes here: http://www.slimraw.com/article-cdngmodes.html

[Danne]

And command line options updated as well. Great. Will test asap.

[DeafEyeJedi]

Finally this one last update finally pushed me over the edge ... Look forward to hear from me @cpc as I'll soon be purchasing this from you and @Danne will then be able to give me a script to include this within MLP, right? [emoji1]

[cpc]

Feedback is always welcome.

[jmanord]

After seeing Danne's post, I was curious as to where, or if, there is documentation for command line options for slimraw. The Lossless 10bit log compression is awesome!

[cpc]

After seeing Danne's post, I was curious as to where, or if, there is documentation for command line options for slimraw. The Lossless 10bit log compression is awesome!

Command line is not "officially" supported and is only available on Mac, you can list the available command line options (including a short description) with something like this (just put a dummy option after the program name, it will dump the full options list in the terminal):
path-to-the-app/slimraw.app/contents/macos/slimraw blabla

[PeteAG]

I made another little test using the newest version 1.6 and it's various compression nodes.

https://vimeo.com/178435652

[yobarry]

Currently I use MLVFS for MLV -> CDNG, I'm debating whether to add SlimRaw for compression to save on HDD space. The biggest advantage for me would be the ability to delete my original MLV files afterwards and save the compressed CDNGs for offline/online editing. But I read in this thread many users advising to keep the original MLV files, is this because later advances/improvements to programs such as MLVFS will increase the quality of the original CDNGs?

Can someone explain this to me? Thanks

[cpc]

And another update v1.7 is now released. Adds DNG downscale to half the horizontal and vertical resolution. Not particularly useful for ML raw footage, but it can be handy for reducing the excessive size of raw timelapses (will have to convert cr2 to dng first), particularly when going for hd/2k deliveries, as well as for raw proxies in general. Here is a post on using raw proxies: http://www.slimraw.com/article-proxies.html

Also, some optimizations here and there, so update anyway, even if you don't care for rescales.

[DeafEyeJedi]

Thanks for the update (v1.7) and you may have noticed that I've finally bit the $50 bullet to purchase this wonderful tool.

Thanks for all that you do @cpc and I am looking forward to using this magic together with @Danne's apps (MLP & cr2hdr.app).

[cpc]

New version 1.8 is now released with support for ML in-camera lossless dng stills. Also, a new 7:1 compression option and a small performance boost for all lossy compression.
Upgrade here: http://www.slimraw.com/download.html

[adrjork]

Hi, sorry if I re-open this old topic...
I've seen that also Swicth has an option for compressing DNGs (with LJ92) from MLVs, so which is the difference between compressing DNGs in Switch versus doing it with MLVFS+SlimRAW? (Believe me, as I wrote in another topic, I really don't want make a SW-vs-SW challenge: I'm just curious to know which are the differences between compressing DNG with Switch and with SlimRAW since both are in this forum and both are loved by users, and both have a similar function, so I suppose both have great PROs and perhaps are useful in different situations.)

And I confess I don't clearly understand if SlimRAW lossless compression methods is "visually lossless" or "data lossless", and if someone noticed any slowdown in Davinci playback with compressed DNGs (in comparison with regular DNGs).

Thanks a lot.