Anyone seen/used this yet?
http://slimraw.com/
Best
Tim
PS I'm nothing to do with this company – I just saw it on Twitter!
Seems it can create cdng files of regular dng,s. That,s good.
Yes for sure it's interesting but how much cpu usage does it need?
I have it and it's fast!
Had some issues with DNGs in Premier Pro but ran them through the adobe converter first and then Slimraw - they open and playback in Premier Pro CC2014 without a problem. The actual compression you get depends on the original file (i.e. user does not set the amount). I get an average 40-50% file compression. Still need to do some pixel peeping.
Why adobe dng converter? Don,t the raw/mlv produced dng files convert straight away?
@Danne - All DNGs seem to convert ok but I had an issue with some cDNG sequences not being read by Premier or Speedgrade - I ran these through the Adobe app then Slimraw and it fixed them. I still need to find out what the issue was.
With my very minimal testing this seems to be a very good app and gets a folder of DNGs down to a similar size as a ProRes 4444 xq transcode although I would still probably transcode for editing and convenience.
Thanks Andy for supporting slimRAW and thanks Tim for posting the link.
I believe Andy means he has converted the original uncompressed files with slimRAW and the compressed output works in Premiere/Speedgrade CC. I would be surprised myself if slimRAW actually makes DNGConverter files compatible with Premiere. :) It will in the future, but does not currently.
Uncompressed DNG and CinemaDNG should work fine in Premiere/Speedgrade CC after slimRAW compression. In fact, if a slimRAW output file does not work with Adobe CC I'd very much like to have a look at the original file. :)
@SamoMalo404: slimRAW will work on any x86-64 processor. Obviously, the more power you have available, the better. But it is fast enough that in the majority of cases storage bandwidth is the bottleneck, not the CPU.
FWIW, on my quad-core Intel i7-4770 3.4ghz running at stock core speed FullHD 5d mark3 14-bit DNG footage converts at around 95-100fps (off an SSD). Now obviously you need fast storage to achieve such speeds (SSD and/or RAID), an HDD simply can't do it physically. In any case, slimRAW will work with whatever is available on the system, at the bandwidth available.
You can read a bit more here:
http://www.shutterangle.com/2015/slimraw-cinemadng-raw-video-lossless-compressor/
Yes, it was a very old set of DNGs that I couldn't open in PP. I think they were converted with an early version of raw2cdng. Running them through SlimRaw didn't fix the issue but running them through the Adobe converter (set to uncompressed v1.3) first then SlimRaw did the trick.
It's a great app @cpc and I would certainly recommend it! :)
The new version of slimRAW adds a trial mode. In trial mode a few frames of each dng/cinemadng sequence will be converted. This way expected compression ratios can be previewed and the compressed frames can be tested and pixel peeped in Resolve, Premiere, Speedgrade, Scratch, etc. :)
Cool stuff!
Would be cool to have this integrated in to a hot folder.
to transfer files to a folder and they would be automatically converted to compressed DNG with this tool.
Or what would be nice too is to convert directly from MLV to Compressed DNG.
im with lars :-)
compress mlv would be awesome just like r3d ;-)
cheers janosch
Definitely following this thread from here on out...
Thanks @cpc!
Does anyone have any comparisons between CinemaDNG from raw2dng, preferably converted to 12 bit and 16 bit ? and then slimraw CinemaDNG of 12 bit and 16 bit conversion?
Would be nice if someone could post the files via dropbox or something alike so I could pixel peep.
Why don,t you test yourself? You get a few frames in demo mode to check. I have the application and it slims the files in no time.
Hmm.. I didn't catch that part!
Thank you. I go pixel peep!
A compression tool like this that could put the compressed DNGs back into a MLV container, directly playable with mlv viewer and mountable with mlvfs would be a fantastic way to save storage space in archiving.
You mean something like this (https://bitbucket.org/dmilligan/mlvfs/downloads/CompressMLV.dmg). See the OP of the MLVFS thread under the section "MLV Compression" for more info.
@Kharak:
12 and 16 bit dng will work fine, but for normal Canon ML footage there is no reason to lose precision converting bitdepth down to 12-bit, nor is there a reason to increase size going 16-bit. 16-bit in particular has no advantage whatsoever, and the compressed 16-bit files will be marginally bigger compared to the 14-bit compressed files.
Dual ISO is another story, since it is 16-bit already, AFAIK.
@Markus:
I am not sure why would you want to do this? The compressed cDNG is already compressed (obviously), so it is good for both archiving and production as is since lossless CinemaDNG is standard and compatible with a wide range of software. And if you only want to quickly preview sequences, you can drop them in either Scratch Play or Resolve Lite.
Quote from: cpc on May 13, 2015, 01:28:14 PM
I am not sure why would you want to do this?
There are certainly advantages to only needing to archive a single MLV file per clip, rather than thousands of individual CDNG files.
Yes! Thanks for posting this. Excited to try out slimraw
Quote from: dmilligan on May 13, 2015, 01:57:47 PM
There are certainly advantages to only needing to archive a single MLV file per clip, rather than thousands of individual CDNG files.
Exactly, there is an advantage to have the data in one big chunk. There is a storage difference when they are saved in small separate files on the harddrive vs mlv container and if you whant to move the data a big chunk of a file will get higher transfer speeds. Also you have hot pixel fix and stripe removal options still available when you can mount the files with mlvfs.
@dmilligan Would love to try out your compression option but I'm on Windows here.
Reason for 16 bit is computational.. Or so I read on these forums quite some time ago.
It is more CPU intensive with 14 bit compared to 16 bit, because 14 bit is not on pair with 2-4-6-16-32 etc..
I believe it was in the The CinemaDNG Discussion thread that Chmee mentioned this and therefore 2 extra "empty" bits are added in RAW2CDNG conversion.
Quote from: KharakReason for 16 bit is computational.. Or so I read on these forums quite some time ago.
It is more CPU intensive with 14 bit compared to 16 bit, because 14 bit is not on pair with 2-4-6-16-32 etc..
I believe it was in the The CinemaDNG Discussion thread that Chmee mentioned this and therefore 2 extra "empty" bits are added in RAW2CDNG conversion.
Well, there is a minor unpacking overhead with 14-bit uncompressed files, but that's probably offset by the fact that 1/7 more data needs to be read from storage with 16-bit uncompressed files; and storage is the slowest link in the processing chain.
With compressed files there is no overhead with 14-bit files compared to 16-bit.
So the new version is out.
slimRAW can now recompress DNG/CinemaDNG files that are already losslessly compressed. For Canon ML raw footage compressed by other means that's 8-20% reduction of size.
Also, this version runs faster (140+ fps for fullHD on my stock quad i7) and compresses better. Plus, there is a Max compression option for people who like to squeeze as much as possible. :)
Full release notes here: http://www.slimraw.com/relnotes.html
Looks seriously handy if you're making a feature and are dealing with massive amounts of data...
Those apps work great with nativa raw footage. What if I want to convert a CR2 file from a Canon Camera? It still is a Raw that I want to convert to a single cinemati DNG.
How would you go about that?
Thanks
PS: I just want to try and post produce still images in Resolve by applying some log curve to it.
You could use adobe dng converter. Should work.
Great app slimraw.
@Danne's correct on that -- use Adobe DNG converter which should do the trick.
@cpc: Thanks for the kind update on SlimRaw -- just downloaded and will be testing it out!
So far it looks like an incredible decent app to use... Great job!
if all goes well... I'll definitely purchase it to get rid of the trial limitations. ;D
I just wanted to give a recommendation for slimraw. I purchased it yesterday to process 377.37 GB of MLV videos. After using MLVFS (another awesome program) to mount and read the MLVs, I was disappointed to find that slimraw was unable to correctly process subdirectories with MLVFS. After sending an email to slimraw detailing the problem, I received a quick response from Mihail and, this afternoon, received an email with an updated build which works perfectly with MLVFS. The files now take up 213.44 GB of space.
yes this program is great and the support is superb.
but i have one question to all the slimraw users:
so what do you do with the not compressed files?
is it safe to delete them?
thank you.
I decided to keep the MLV files as my backup copy, rather than backing up the dngs created from slimraw. I use lightroom for all my color correction, and use the catalog to store the edits rather than within the individual dng files, so I can always restore the dngs from the original MLV files without loosing my edits. I'm sure this is true with Davince Resolve as well.
Bought it!
Love it!
I haven't used this on any of my personal projects, cause I have OCD for quality loss, so even though its lossless my OCD doesn't allow me to compress anything I personally stand by, but been filming a play these last days and my harddrive space was shredded and I've just had the Trial of SlimRaw on my desktop staring at me for a long time, but I gotta say, for projects like this, this is absolutely fantastic.
Converted the MLVs with Raw2Cdng to 12 bit and then "Slimmed" them, so all in all I'd say its a 60% reduction from Original MLV's.
And lightning fast compression too.
Thanks for the kind words, Kharak.
I usually suggest staying in 14 bits though. In most cases, if you simply convert to 14-bit dng and compress that with slimraw it will usually shrink the files to less than half the original size with no quality loss whatsoever. On the other hand, the quantization to 12-bit that you are doing in your initial dng conversion is losing you quality for (I presume) a tiny size gain.
@cpc: There is no 14 bit CDNG out of any ML tool :)
Does 16 bit brings really significant overhead?
I didn't try your slimmer yet, but Adobe's Converter compresses almost to the same degree 14 and 16 bit:
MLVProducer CDNG compressed Adobe DNG Converter - 4,83 MB (5 070 338 bytes) (16 bit)
MLV_dump DNG compressed Adobe DNG Converter - 4,82 MB (5 060 678 bytes) (14 bit)
slimRAW doesn't require cdng, it works fine with dng files so anything converting raw/mlv to straight 14-bit dng is fine (like, say, raw2dng or any of the GUI frontends that use it).
But yes, you can go 16-bit, and the overhead over 14-bit is negligible both in size and decompression speed. The reason for the similar compressed size is that the 16-bit uncompressed really only uses 14 meaningful bits and 2 redundant bits. And handling redundancy is exactly what lossless compression does. Still, 14-bit is optimal.
Also, unlike uncompressed 14-bit dng, lossless compressed 14-bit dng/cdng is widely compatible (for example, Premiere chokes on 14-bit uncompressed but works fine with 14-bit compressed).
Yes, but "anything converting raw/mlv to straight 14-bit dng" has no timecode, FPS and so on - this why the question about 16 bit CDNG rises.
As for me - I just bought a couple of 2TB discs and for some time storage problem is solved :) But later I definitely come back here, ... or buy another couple :)
Hi cpc, you are welcome.
I use Raw2cdng, so I can't convert to 14 bit. Only - 12 bit maximized, 16 bit and 16 bit maximized.
I converted to 12 bit to save space, I filled up 6 TB these last 6 months and I recorded 3 hours more of footage from the Play, so 3 hours in MLVs and converting them to 16 bit just don't fit on my drives. I am looking through the Compressed footage and so far no errors or missing frames, so I will delete the MLV's when I have checked all of them. First half of the footage I didn't have the Verification Pass checked, but can you tell me what the Verification Pass actually verifies. Is it doing the job for me, that I am doing right now?
The play doesn't require any colour grade, just colour correction and some contrast to make the look natural. So 12 bits is more than enough for that.
And another question, the Cdng's are not playable in MLRawViewer after compression, It plays a few frames and crashes. I also see that top half of the image is "broken", only in mlrawviewer. Not sure how Mlrawviewer works, might be it "expects" a certain uncompressed footage.. I don't know. With CC compatibility, the Compressed footage runs smooth in PP.
@Kharak:
Verification Pass will check for missing files and for checksum errors.
"Missing file" would be any dng file present in the input folders and missing in the output folders. This should never happen, if it happens then something is seriously messed up with the filesystem of the output device. Note that if the input device filesystem is messed and the OS/software doesn't see a particular input .dng file, then there is no way for the software to recognize that, so it can't report it missing in the output.
Checksum verification calculates the checksums of the written output (bypassing the OS file caches) and compares them to the checksums of the data in memory. A checksum error generally means the output storage device is unreliable, or RAM is faulty.
AFAIK, mlvrawviewer doesn't read all types of lossless dng correctly. It seemed to crash a lot on dng playback last time I checked some months ago.
Premiere has seen some serious performance improvements in DNG raw processing in CC 2015. I believe it is only second to Resolve in terms of performance at the moment.
@mothaibaphoto:
Well, one of the cool things with slimRAW is that half the size means half the storage throughput needed in post. This is especially nice with HDDs as it can be all the difference between choppy playback and smooth playback. :)
slimRAW 1.4 has now been released and it adds 3:1 and 4:1 lossy CinemaDNG options. Here are the release notes: http://www.slimraw.com/relnotes.html
Note that lossy CinemaDNG is only compatible with Davinci Resolve for now. Also note that all lossy CinemaDNG is 12-bit, so any 14- or 16-bit DNG/CinemaDNG will be converted to 12-bit before compression.
Also, MLVFS support (in the OS X version), previously available on demand, has now been rolled into the official release. And there are some minor performance improvements when handling 14-bit raw files (I am now getting 150+ fps when compressing 14-bit FullHD raw on an i7 4770), so updating is recommended even if you don't care for lossy compression.
Thanks for the much needed update and excellent work as always @cpc!
Wow nice! Thank you!
Great progress cpc.
Thanks, guys! This one took a long time of testing and tuning, but I am quite pleased with the results. :)
Hi,
I am about to start shooting my first documentary with 5D3+ML in Asia. I have been following this lovely forum for quite a while and learnt a lot but time is running out and theres still lot to learn and to understand. I would love to have idiot check from someone who has knowledge and experience is my workflow bulletproof (or at least rain proof) cause this is something I havent tried yet:
Shooting to MLV -> slimraw -> CDNGs to hardrive and backup, erase CF-cards and original MLV. files. I know it may not be the best idea to erase original DNG-files but as we are shooting non-budget doc (or lowlow budget), all the pennys we can save makes difference. Perhaps I should always go through CDNG to be sure that theres no erros? Btw is there a tool for that? Any suggestions?
My back up plan is to shoot with h.264 If everything will falls apart. But after working only a short moment with ML it would feel wrong :)
Slimraw is a great tool. Check out mlvfs as well and maybe reconsider erasing your mlv files.
@sams
i would not recommend deleting the mlv files.. hdds are not that expensive. just buy a second one and save the original files there.
when the developers make some progress with the conversion from mlv to cdng (mlvfs) its always good to keep the backup files, cause then you are on the safe side.
best wishes, sw
Not sure if this was mentioned yet but I use MLVFS Fuse for windows, quick mount MLVs to DNGs then use slimraw to lossless compress the mounted dngs. Then your left to grade the reduced size DNGs. This will save you a lot of time on big projects and some space.
Using MLVFS to feed MLV into slimRAW should probably be the fastest way, since this skips the original mlv->dng conversion and directly outputs compressed cdng.
While slimRAW is, I believe, extremely reliable and can also verify the written output for you, it obviously can't verify the MLV input for errors since it has no way of recognizing them. And if there are errors in the MLV, these errors will translate to the DNGs.
Now, of course, you shouldn't be getting errors in the MLV in the first place, but you never know with (some) CF cards and high rate recording. I am a bit behind on ML developments, but back when I was shooting ML raw (MLV didn't existed yet) I would occasionally get corruption in the ML .raw files (the precursor to .mlv) cause I liked using some of the "forbidden" ML tools (features requiring Global Draw ON which was supposed to be turned OFF for raw recording). Dunno, maybe this is not an issue anymore.
But nothing beats visual inspection of the material. I like Assimilate Scratch Play for this (cause it starts almost instantly), but you can use anything that plays DNG footage.
Nice work!
Would someone care to make a size comparison between Slimraw lossless cDNG and log Pro Ress 4444 XQ ?
@cpc "(I am now getting 150+ fps when compressing 14-bit FullHD raw on an i7 4770), so updating is recommended even if you don't care for lossy compression."
I don't get this. Not even close! I would settle for real time. How are you getting this?
Compress on the fastest drive you have, preferably SSD to get High speed Compression.
@eyeland:
Lossless CinemaDNG size varies with content, but if we assume an average compression ratio of 2:1 for ML raw, the calculation goes like this:
FullHD 14-bit lossless CinemaDNG: 1920*1080*14/8*24/2 ~= 43.5 MB/s (at 24fps)
FullHD 12-bit ProRess4444XQ: 396 mb/s = 49.5 MB/s (at 24fps)
@hjfilmspeed:
As noted by Kharak, you need a very fast storage setup to hit these speeds. For 150fps you'll need almost 550MB/s sustained storage reading speed and around half of this rate on the output end. This means very high end SSD, and preferably a RAID of SSDs. For real time processing, you need a sustained reading throughput of around 87 MB/s (at 24fps). If you are processing straight from the CF card (mounting raw through MLVFS), you are most likely limited by the throughput of your CF card + CF reader combination (many readers are way slower than the CF card itself).
MLV --> Cdng 16bit -->Log ProRes4444XQ+alpha=16bit ;)
Which would be the Very best format to keep your ML files archive as
Since it's Linear , there's really no advantage to keep MLV file's or compressing the Cdng's
Plus All Apps (Mac) (some on PC if you have the PC ProResXQ Ver.) , support ProResXQ now , and there no need to have expensive hardware .
Just standard SSD or fast spindle drive e.g. WD VelociRaptor 600GB 10K RPM (I have 2 in a raid 0 on MP5.1)
Just about any midrange Laptop/desktop will do.
QuoteMLV --> Cdng 16bit -->Log ProRes4444XQ+alpha=16bit
Color channels in ProRes4444XQ are 12-bit. Only alpha is 16-bit and it is optional, you will generally want to skip it, cause it only inflates size in this case. :)
Quote from: cpc on January 30, 2016, 08:36:19 PM
Lossless CinemaDNG size varies with content, but if we assume an average compression ratio of 2:1 for ML raw, the calculation goes like this:
FullHD 14-bit lossless CinemaDNG: 1920*1080*14/8*24/2 ~= 43.5 MB/s (at 24fps)
FullHD 12-bit ProRess4444XQ: 396 mb/s = 49.5 MB/s (at 24fps)
Thanks, Thats really nice :) I second the suggestion above about a container to avoid having to store them as individual files :)
Comparing DNG with prores or any other delivery/intermediate format is just like comparing apples and oranges: DNG allows me to reprocess the raw data anytime later, while others - not. The result of processing the raw data very depends on a) software used b) personal skills. Both things tend to develop vastly. Debayer software evolves rapidly. Just look what a progress Resolve did. I just reprocessed in 4K a dozen of timilapses I shot a long time ago on 400D and I managed to get better results than in HD that time. And when new version of Resolve or ACR, or maybe MLVProducer will bring new debayer feature, I can reprocess my files again, to get better image.
@motbaiphoto +1.
Hi, I made a little quality comparison video yesterday to see how well SlimRAW behaves against the original CDNG and all ProRes flavors.
https://vimeo.com/153739453
I have a lot of Footage from my recent Iceland trip and really want to archive my RAW files as small in size but best quality as possible.
SlimRaw is a superb tool and the support is fast and professional.
The downside is that compressed CNDGs are only up to Resolve at the moment (don't know much about the Windows and Linux side).
New version 1.5 is now released with some improvements. Release notes here: http://www.slimraw.com/relnotes.html
I have a few questions before I commit to purchasing SlimRAW. The first and most important is about performance in Premiere and After Effects. Currently I have an i7 3930K and have a server feeding a 10GbE pipe with 650MB/s of read and write performance. As it stands I get choppy, and drop frame performance reading 16bit CDNG content on Premiere Pro. I converted these the MLV files using RAW2CDNG (which I has proven to be the best performance and reliable out of all the options out there). I want to know if this will actually improve performance, as resolve plays the files fine, and PPcc has hiccups. This has lead me to believe that PPcc is the issue over file size and data rates. So, can anyone give me a reasonable explanation about why slimraw would make playback and editing better in PPcc?
Second, I would like to know if anyone has had any issues with crashing or bad frames in the compression process. I've had my fair share of issues finding the right software to do the conversions from MLV, and I finally found a version of raw2cdng that works pretty reliably.
Third, how does licensing work? I generally shoot for myself, but I use a computer at work to process footage occasionally. Do I need to purchase two separate licenses, or will it allow me to use the same license on each computer (like Adobe products) as long as I don't have both using the same license at the same time?
Thanks, looks like a cool product to save some expensive RAID storage.
QuoteI have a few questions before I commit to purchasing SlimRAW. The first and most important is about performance in Premiere and After Effects. Currently I have an i7 3930K and have a server feeding a 10GbE pipe with 650MB/s of read and write performance. As it stands I get choppy, and drop frame performance reading 16bit CDNG content on Premiere Pro. I converted these the MLV files using RAW2CDNG (which I has proven to be the best performance and reliable out of all the options out there). I want to know if this will actually improve performance, as resolve plays the files fine, and PPcc has hiccups. This has lead me to believe that PPcc is the issue over file size and data rates. So, can anyone give me a reasonable explanation about why slimraw would make playback and editing better in PPcc?
Second, I would like to know if anyone has had any issues with crashing or bad frames in the compression process. I've had my fair share of issues finding the right software to do the conversions from MLV, and I finally found a version of raw2cdng that works pretty reliably.
Third, how does licensing work? I generally shoot for myself, but I use a computer at work to process footage occasionally. Do I need to purchase two separate licenses, or will it allow me to use the same license on each computer (like Adobe products) as long as I don't have both using the same license at the same time?
Thanks, looks like a cool product to save some expensive RAID storage.
re: Premiere performance
Strange that you don't get realtime playback in Premiere with your specs. I haven't tried 16-bit uncompressed files, but I get smooth playback of 14-bit losslessly compressed 1080p DNG on my quad i7 4770. Have you tried processing from local storage? Could be Premiere doesn't like the network storage setup for whatever reason?
To answer your question, it is possible that using lossless DNG will improve the situation since Premiere would need to stream significantly less data, and the lossless compresson will be no problem for your hexcore cpu.
re: stability
FWIW, I've never had a crash report in the year and more that slimRAW's been out. Obviously, if you have a bad frame before compression, it will stay bad after compression. No way around this.
re: licensing
slimRAW can be used on all your computers with one license, and this includes both Mac and Windows platforms.
New version 1.6 is now out with 4 new compression modes.
Added 5:1 lossy compression and two variable bit rate lossy modes. But perhaps most interesting for ML users will be the new 10-bit log encoded lossless mode. The linear 14-bit signal is mapped to a 10-bit coding space using a log tonal curve, similarly to Canon C500 10-bit log raw. Log data is then losslessly compressed. This results in 20-30% smaller files than straight lossless compression, while preserving high image quality. The log tranform is transparent, it is reversed by the raw processor on import; no workflow changes required. The output is compatible with pretty much all DNG capable video software.
All lossy compression now also uses a non-linear transform, which allows higher levels of compression by keeping stress off dark tones.
More about the different compression modes here: http://www.slimraw.com/article-cdngmodes.html
And command line options updated as well. Great. Will test asap.
Finally this one last update finally pushed me over the edge ... Look forward to hear from me @cpc as I'll soon be purchasing this from you and @Danne will then be able to give me a script to include this within MLP, right? [emoji1]
Feedback is always welcome. :)
After seeing Danne's post, I was curious as to where, or if, there is documentation for command line options for slimraw. The Lossless 10bit log compression is awesome!
QuoteAfter seeing Danne's post, I was curious as to where, or if, there is documentation for command line options for slimraw. The Lossless 10bit log compression is awesome!
Command line is not "officially" supported and is only available on Mac, you can list the available command line options (including a short description) with something like this (just put a dummy option after the program name, it will dump the full options list in the terminal):
path-to-the-app/slimraw.app/contents/macos/slimraw blabla
I made another little test using the newest version 1.6 and it's various compression nodes.
https://vimeo.com/178435652
Currently I use MLVFS for MLV -> CDNG, I'm debating whether to add SlimRaw for compression to save on HDD space. The biggest advantage for me would be the ability to delete my original MLV files afterwards and save the compressed CDNGs for offline/online editing. But I read in this thread many users advising to keep the original MLV files, is this because later advances/improvements to programs such as MLVFS will increase the quality of the original CDNGs?
Can someone explain this to me? Thanks
And another update v1.7 is now released. Adds DNG downscale to half the horizontal and vertical resolution. Not particularly useful for ML raw footage, but it can be handy for reducing the excessive size of raw timelapses (will have to convert cr2 to dng first), particularly when going for hd/2k deliveries, as well as for raw proxies in general. Here is a post on using raw proxies: http://www.slimraw.com/article-proxies.html
Also, some optimizations here and there, so update anyway, even if you don't care for rescales.
Thanks for the update (v1.7) and you may have noticed that I've finally bit the $50 bullet to purchase this wonderful tool.
Thanks for all that you do @cpc and I am looking forward to using this magic together with @Danne's apps (MLP (http://www.magiclantern.fm/forum/index.php?topic=13512.0MLP) & cr2hdr.app (https://www.magiclantern.fm/forum/index.php?topic=15108.0cr2hdr.app)).
New version 1.8 is now released with support for ML in-camera lossless dng stills. Also, a new 7:1 compression option and a small performance boost for all lossy compression.
Upgrade here: http://www.slimraw.com/download.html
Hi, sorry if I re-open this old topic...
I've seen that also Swicth has an option for compressing DNGs (with LJ92) from MLVs, so which is the difference between compressing DNGs in Switch versus doing it with MLVFS+SlimRAW? (Believe me, as I wrote in another topic, I really don't want make a SW-vs-SW challenge: I'm just curious to know which are the differences between compressing DNG with Switch and with SlimRAW since both are in this forum and both are loved by users, and both have a similar function, so I suppose both have great PROs and perhaps are useful in different situations.)
And I confess I don't clearly understand if SlimRAW lossless compression methods is "visually lossless" or "data lossless", and if someone noticed any slowdown in Davinci playback with compressed DNGs (in comparison with regular DNGs).
Thanks a lot.
Where can i get Switch?
https://www.magiclantern.fm/forum/index.php?board=54.0
Quote from: adrjork on August 07, 2019, 08:48:43 AM
Hi, sorry if I re-open this old topic...
I've seen that also Swicth has an option for compressing DNGs (with LJ92) from MLVs, so which is the difference between compressing DNGs in Switch versus doing it with MLVFS+SlimRAW? (Believe me, as I wrote in another topic, I really don't want make a SW-vs-SW challenge: I'm just curious to know which are the differences between compressing DNG with Switch and with SlimRAW since both are in this forum and both are loved by users, and both have a similar function, so I suppose both have great PROs and perhaps are useful in different situations.)
And I confess I don't clearly understand if SlimRAW lossless compression methods is "visually lossless" or "data lossless", and if someone noticed any slowdown in Davinci playback with compressed DNGs (in comparison with regular DNGs).
Thanks a lot.
A bit late, but here are some answers for posterity.
First, your lossless DNG question is answered in slimraw's faq: http://www.slimraw.com/faq.html
Second, performance in Resolve with compressed DNG will depend on what's the bottleneck of your system when streaming the raw footage. If storage is the bottleneck, Resolve performance should improve. If the CPU is the bottleneck, performance may degrade.
As for the differences between slimRAW and Switch, Danne will correct me if I'm wrong, but my understanding is that Switch uses Adobe's DNG Converter. DNG Converter is no match for slimRAW neither in terms of file size achieved, nor in terms of speed (last time I checked, slimRAW was 32 times faster on a quad-core CPU). But hey, Switch is free and I am sure Danne is doing a great job with the whole Switch pipeline. If you aren't in a hurry, you may gain some additional space by passing DNG Converter's output through DNGStrip, a free tool that strips some of DNG Converter's useless chaff from the compressed files, which I've written some years ago: https://www.shutterangle.com/2014/lossless-compression-for-dng-raw-video-dngstrip/
Hi cpc! Long ago :).
Switch is more or less a sleeping beauty nowadays. Mlv App is standard more or less. However compressed dng from mlv uses lj92 compression introduced by A.Baldwin in MlRawViewer: https://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html
Better than adobe dng and cdng but of course not as fast as slimraw ;).
Quote from: Danne on July 16, 2020, 09:13:13 PM
Hi cpc! Long ago :).
Switch is more or less a sleeping beauty nowadays. Mlv App is standard more or less. However compressed dng from mlv uses lj92 compression introduced by A.Baldwin in MlRawViewer: https://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html
Better than adobe dng and cdng but of course not as fast as slimraw ;).
Hi Danne. Good to know. Thank you for the heads up, I haven't been following developments closely for some time now. (Ok, I should probably say "at all" instead of "closely". ;) )
I glanced through your link and it seems that he is describing Blackmagic's way of doing the prediction, which isn't particularly good for linear Canon raw. I would guess that DNG Converter + DNGStrip should produce smaller files (well, if one could live with the painfully slow DNG Converter, that is :) ).
Quote from: cpc on July 16, 2020, 09:25:49 PM
Hi Danne. Good to know. Thank you for the heads up, I haven't been following developments closely for some time now. (Ok, I should probably say "at all" instead of "closely". ;) )
I glanced through your link and it seems that he is describing Blackmagic's way of doing the prediction, which isn't particularly good for linear Canon raw. I would guess that DNG Converter + DNGStrip should produce smaller files (well, if one could live with the painfully slow DNG Converter, that is :) ).
Probably. You should know if anyone 8).
I'm using adobe dng converter to compress 'old' uncompressed 14bit ML dng files.
Adobe dng converter has this option to only use lossless compression. I was always assuming adobe dng converter uses lj92 compression. But after last few posts in this topic, I'm in doubt? Isn't adobe dng converter lossless compression option the same as lj92 compression used in mlv's ?
And is there a difference when using MLVapp to extract lossless dng's from a mlv file compared to use command line mlvdump to extract lossless dng's?
(With mlvdump I always see adobe dng converter popup, so it's probably used by mlvdump by command line)
Quote from: Levas on July 16, 2020, 11:52:49 PM
I'm using adobe dng converter to compress 'old' uncompressed 14bit ML dng files.
Adobe dng converter has this option to only use lossless compression. I was always assuming adobe dng converter uses lj92 compression. But after last few posts in this topic, I'm in doubt? Isn't adobe dng converter lossless compression option the same as lj92 compression used in mlv's ?
Anything "lossless dng" uses the same spec when compressing the pixel data. But the spec allows for a lot of wiggle room and choices to make, which accounts for significant differences in compressed pixel data size. SlimRAW will produce files significantly smaller than DNG Converter and way smaller than anything that uses Blackmagic's setup for Canon linear raw (usually ~15% smaller, sometimes more). That's because slimraw tries a lot of stuff on the fly to get the smallest results (and even more stuff to try can be enabled through an option), while the others fix parameters in advance. Regardless, results will be truly lossless with any compressor, and compressed files will decompress to the same original file. (Well, DNG Converter will convert all input to nominal 16-bit regardless of true input bit depth, so the decompressed file will be nominally 16-bit, but that's mostly technical details.)
The thing is, DNG Converter is made for photos, and it has the peculiarity that it inserts thumbnails (previews) in each frame which certainly is pointless for video, and it adds a bunch of useless conjured metadata. All this bloats the output size. This is what gets stripped by DNGStrip. But the main issue that initially pushed me towards making slimraw back in the day was the excruciatingly slow processing of DNG Converter. Well, that and the fact that there was no log raw output in DNG Converter. :)
I use Adobe DNG converter to convert CR2 to DNG, so I can grade them in Resolve. I then further compress them with Slimraw. and I believe it compresses a further 15% or so.
The lossless compressed DNG's from SlimRaw greatly improve the performance in Resolve, especially Magic Lantern DNG's. Now that prices on SSD's are plummeting, it is not so much a problem anymore, but I used to have huge projects on External HDD and the bottleneck would be the USB Read Speeds, especially after filling more than 50% of an external HDD, the performance drop gets clear, but with Slimraw keeping the Data Rate around 50 MB/s, I have real-time playback in Resolve (with heavy grading) because the USB speed is not a bottleneck any more. (Until you are down to 20% space left).
I've been overly happy with the software, its very fast! it has saved me tons of space on HDD's, especially old projects from before Lossless compression was introduced in camera and I had a bunch of 80-100 MB/s MLV's/DNG's stored. So I went back to archived drives that were more or less filled to the brim and compressed with Slimraw, gaining on average 50% more capacity on those drives. I don't put more footage on those drives, Instead I consider them full at 50%. Leaving them at 50%, I know I have best performance from those HDD's, should I ever use some of the footage again.
Quick tip. When compressing, if you are leaving your computer or spare laptop overnight, set it to Defragment the External HDD after compression, it made a huge difference in keeping footage Real time in resolve and not getting random freezes.