Author Topic: SlimRaw – CDNG compression tool (Read 50073 times)

cpc · « **Reply #25 on:** June 01, 2015, 04:37:09 PM »

So the new version is out.
slimRAW can now recompress DNG/CinemaDNG files that are already losslessly compressed. For Canon ML raw footage compressed by other means that's 8-20% reduction of size.
Also, this version runs faster (140+ fps for fullHD on my stock quad i7) and compresses better. Plus, there is a Max compression option for people who like to squeeze as much as possible.

Full release notes here: http://www.slimraw.com/relnotes.html

Terry Tibbs · « **Reply #26 on:** June 18, 2015, 02:33:55 PM »

Looks seriously handy if you're making a feature and are dealing with massive amounts of data...

theartofweb · « **Reply #27 on:** June 26, 2015, 06:50:51 PM »

Those apps work great with nativa raw footage. What if I want to convert a CR2 file from a Canon Camera? It still is a Raw that I want to convert to a single cinemati DNG.

How would you go about that?

Thanks

PS: I just want to try and post produce still images in Resolve by applying some log curve to it.

Danne · « **Reply #28 on:** June 26, 2015, 07:23:34 PM »

You could use adobe dng converter. Should work.
Great app slimraw.

DeafEyeJedi · « **Reply #29 on:** June 26, 2015, 07:33:37 PM »

@Danne's correct on that -- use Adobe DNG converter which should do the trick.

@cpc: Thanks for the kind update on SlimRaw -- just downloaded and will be testing it out!

So far it looks like an incredible decent app to use... Great job!

if all goes well... I'll definitely purchase it to get rid of the trial limitations.

jmanord · « **Reply #30 on:** July 31, 2015, 10:22:46 PM »

I just wanted to give a recommendation for slimraw. I purchased it yesterday to process 377.37 GB of MLV videos. After using MLVFS (another awesome program) to mount and read the MLVs, I was disappointed to find that slimraw was unable to correctly process subdirectories with MLVFS. After sending an email to slimraw detailing the problem, I received a quick response from Mihail and, this afternoon, received an email with an updated build which works perfectly with MLVFS. The files now take up 213.44 GB of space.

swinxx · « **Reply #31 on:** August 01, 2015, 01:53:44 PM »

yes this program is great and the support is superb.

but i have one question to all the slimraw users:

so what do you do with the not compressed files?
is it safe to delete them?

thank you.

jmanord · « **Reply #32 on:** August 01, 2015, 06:22:50 PM »

I decided to keep the MLV files as my backup copy, rather than backing up the dngs created from slimraw. I use lightroom for all my color correction, and use the catalog to store the edits rather than within the individual dng files, so I can always restore the dngs from the original MLV files without loosing my edits. I'm sure this is true with Davince Resolve as well.

Kharak · « **Reply #33 on:** November 15, 2015, 07:37:46 PM »

Bought it!

Love it!

I haven't used this on any of my personal projects, cause I have OCD for quality loss, so even though its lossless my OCD doesn't allow me to compress anything I personally stand by, but been filming a play these last days and my harddrive space was shredded and I've just had the Trial of SlimRaw on my desktop staring at me for a long time, but I gotta say, for projects like this, this is absolutely fantastic.

Converted the MLVs with Raw2Cdng to 12 bit and then "Slimmed" them, so all in all I'd say its a 60% reduction from Original MLV's.

And lightning fast compression too.

cpc · « **Reply #34 on:** November 16, 2015, 03:56:25 PM »

Thanks for the kind words, Kharak.

I usually suggest staying in 14 bits though. In most cases, if you simply convert to 14-bit dng and compress that with slimraw it will usually shrink the files to less than half the original size with no quality loss whatsoever. On the other hand, the quantization to 12-bit that you are doing in your initial dng conversion is losing you quality for (I presume) a tiny size gain.

mothaibaphoto · « **Reply #35 on:** November 16, 2015, 04:49:11 PM »

@cpc: There is no 14 bit CDNG out of any ML tool

Does 16 bit brings really significant overhead?
I didn't try your slimmer yet, but Adobe's Converter compresses almost to the same degree 14 and 16 bit:
MLVProducer CDNG compressed Adobe DNG Converter - 4,83 MB (5 070 338 bytes) (16 bit)
MLV_dump DNG compressed Adobe DNG Converter - 4,82 MB (5 060 678 bytes) (14 bit)

cpc · « **Reply #36 on:** November 16, 2015, 05:13:19 PM »

slimRAW doesn't require cdng, it works fine with dng files so anything converting raw/mlv to straight 14-bit dng is fine (like, say, raw2dng or any of the GUI frontends that use it).

But yes, you can go 16-bit, and the overhead over 14-bit is negligible both in size and decompression speed. The reason for the similar compressed size is that the 16-bit uncompressed really only uses 14 meaningful bits and 2 redundant bits. And handling redundancy is exactly what lossless compression does. Still, 14-bit is optimal.

Also, unlike uncompressed 14-bit dng, lossless compressed 14-bit dng/cdng is widely compatible (for example, Premiere chokes on 14-bit uncompressed but works fine with 14-bit compressed).

mothaibaphoto · « **Reply #37 on:** November 16, 2015, 05:58:36 PM »

Yes, but "anything converting raw/mlv to straight 14-bit dng" has no timecode, FPS and so on - this why the question about 16 bit CDNG rises.
As for me - I just bought a couple of 2TB discs and for some time storage problem is solved

But later I definitely come back here, ... or buy another couple

Kharak · « **Reply #38 on:** November 17, 2015, 10:37:26 AM »

Hi cpc, you are welcome.

I use Raw2cdng, so I can't convert to 14 bit. Only - 12 bit maximized, 16 bit and 16 bit maximized.

I converted to 12 bit to save space, I filled up 6 TB these last 6 months and I recorded 3 hours more of footage from the Play, so 3 hours in MLVs and converting them to 16 bit just don't fit on my drives. I am looking through the Compressed footage and so far no errors or missing frames, so I will delete the MLV's when I have checked all of them. First half of the footage I didn't have the Verification Pass checked, but can you tell me what the Verification Pass actually verifies. Is it doing the job for me, that I am doing right now?

The play doesn't require any colour grade, just colour correction and some contrast to make the look natural. So 12 bits is more than enough for that.

And another question, the Cdng's are not playable in MLRawViewer after compression, It plays a few frames and crashes. I also see that top half of the image is "broken", only in mlrawviewer. Not sure how Mlrawviewer works, might be it "expects" a certain uncompressed footage.. I don't know. With CC compatibility, the Compressed footage runs smooth in PP.

cpc · « **Reply #39 on:** November 17, 2015, 12:37:35 PM »

@Kharak:
Verification Pass will check for missing files and for checksum errors.
"Missing file" would be any dng file present in the input folders and missing in the output folders. This should never happen, if it happens then something is seriously messed up with the filesystem of the output device. Note that if the input device filesystem is messed and the OS/software doesn't see a particular input .dng file, then there is no way for the software to recognize that, so it can't report it missing in the output.

Checksum verification calculates the checksums of the written output (bypassing the OS file caches) and compares them to the checksums of the data in memory. A checksum error generally means the output storage device is unreliable, or RAM is faulty.

AFAIK, mlvrawviewer doesn't read all types of lossless dng correctly. It seemed to crash a lot on dng playback last time I checked some months ago.

Premiere has seen some serious performance improvements in DNG raw processing in CC 2015. I believe it is only second to Resolve in terms of performance at the moment.

@mothaibaphoto:
Well, one of the cool things with slimRAW is that half the size means half the storage throughput needed in post. This is especially nice with HDDs as it can be all the difference between choppy playback and smooth playback.

cpc · « **Reply #40 on:** January 11, 2016, 05:58:31 PM »

slimRAW 1.4 has now been released and it adds 3:1 and 4:1 lossy CinemaDNG options. Here are the release notes: http://www.slimraw.com/relnotes.html
Note that lossy CinemaDNG is only compatible with Davinci Resolve for now. Also note that all lossy CinemaDNG is 12-bit, so any 14- or 16-bit DNG/CinemaDNG will be converted to 12-bit before compression.

Also, MLVFS support (in the OS X version), previously available on demand, has now been rolled into the official release. And there are some minor performance improvements when handling 14-bit raw files (I am now getting 150+ fps when compressing 14-bit FullHD raw on an i7 4770), so updating is recommended even if you don't care for lossy compression.

DeafEyeJedi · « **Reply #41 on:** January 11, 2016, 06:02:07 PM »

Thanks for the much needed update and excellent work as always @cpc!

Kharak · « **Reply #42 on:** January 11, 2016, 07:15:35 PM »

Wow nice! Thank you!

Danne · « **Reply #43 on:** January 11, 2016, 08:10:24 PM »

Great progress cpc.

cpc · « **Reply #44 on:** January 12, 2016, 12:23:03 PM »

Thanks, guys! This one took a long time of testing and tuning, but I am quite pleased with the results.

SamS · « **Reply #45 on:** January 28, 2016, 03:14:13 AM »

Hi,

I am about to start shooting my first documentary with 5D3+ML in Asia. I have been following this lovely forum for quite a while and learnt a lot but time is running out and theres still lot to learn and to understand. I would love to have idiot check from someone who has knowledge and experience is my workflow bulletproof (or at least rain proof) cause this is something I havent tried yet:

Shooting to MLV -> slimraw -> CDNGs to hardrive and backup, erase CF-cards and original MLV. files. I know it may not be the best idea to erase original DNG-files but as we are shooting non-budget doc (or lowlow budget), all the pennys we can save makes difference. Perhaps I should always go through CDNG to be sure that theres no erros? Btw is there a tool for that? Any suggestions?

My back up plan is to shoot with h.264 If everything will falls apart. But after working only a short moment with ML it would feel wrong

Danne · « **Reply #46 on:** January 28, 2016, 11:16:54 PM »

Slimraw is a great tool. Check out mlvfs as well and maybe reconsider erasing your mlv files.

swinxx · « **Reply #47 on:** January 29, 2016, 05:36:46 PM »

@sams
i would not recommend deleting the mlv files.. hdds are not that expensive. just buy a second one and save the original files there.
when the developers make some progress with the conversion from mlv to cdng (mlvfs) its always good to keep the backup files, cause then you are on the safe side.
best wishes, sw

hjfilmspeed · « **Reply #48 on:** January 29, 2016, 10:00:05 PM »

Not sure if this was mentioned yet but I use MLVFS Fuse for windows, quick mount MLVs to DNGs then use slimraw to lossless compress the mounted dngs. Then your left to grade the reduced size DNGs. This will save you a lot of time on big projects and some space.

cpc · « **Reply #49 on:** January 29, 2016, 11:33:34 PM »

Using MLVFS to feed MLV into slimRAW should probably be the fastest way, since this skips the original mlv->dng conversion and directly outputs compressed cdng.
While slimRAW is, I believe, extremely reliable and can also verify the written output for you, it obviously can't verify the MLV input for errors since it has no way of recognizing them. And if there are errors in the MLV, these errors will translate to the DNGs.

Now, of course, you shouldn't be getting errors in the MLV in the first place, but you never know with (some) CF cards and high rate recording. I am a bit behind on ML developments, but back when I was shooting ML raw (MLV didn't existed yet) I would occasionally get corruption in the ML .raw files (the precursor to .mlv) cause I liked using some of the "forbidden" ML tools (features requiring Global Draw ON which was supposed to be turned OFF for raw recording). Dunno, maybe this is not an issue anymore.

But nothing beats visual inspection of the material. I like Assimilate Scratch Play for this (cause it starts almost instantly), but you can use anything that plays DNG footage.

News:

Author Topic: SlimRaw – CDNG compression tool (Read 50073 times)