SlimRaw – CDNG compression tool

Started by timbytheriver, April 23, 2015, 10:43:14 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

cpc

So the new version is out.
slimRAW can now recompress DNG/CinemaDNG files that are already losslessly compressed. For Canon ML raw footage compressed by other means that's 8-20% reduction of size.
Also, this version runs faster (140+ fps for fullHD on my stock quad i7) and compresses better. Plus, there is a Max compression option for people who like to squeeze as much as possible. :)

Full release notes here: http://www.slimraw.com/relnotes.html

Terry Tibbs

Looks seriously handy if you're making a feature and are dealing with massive amounts of data...

theartofweb

Those apps work great with nativa raw footage. What if I want to convert a CR2 file from a Canon Camera? It still is a Raw that I want to convert to a single cinemati DNG.

How would you go about that?

Thanks

PS: I just want to try and post produce still images in Resolve by applying some log curve to it.

Danne

You could use adobe dng converter. Should work.
Great app slimraw.

DeafEyeJedi

@Danne's correct on that -- use Adobe DNG converter which should do the trick.

@cpc: Thanks for the kind update on SlimRaw -- just downloaded and will be testing it out!

So far it looks like an incredible decent app to use... Great job!

if all goes well... I'll definitely purchase it to get rid of the trial limitations.  ;D
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

jmanord

I just wanted to give a recommendation for slimraw. I purchased it yesterday to process 377.37 GB of MLV videos. After using MLVFS (another awesome program) to mount and read the MLVs, I was disappointed to find that slimraw was unable to correctly process subdirectories with MLVFS. After sending an email to slimraw detailing the problem, I received a quick response from Mihail and, this afternoon, received an email with an updated build which works perfectly with MLVFS. The files now take up 213.44 GB of space.

swinxx

yes this program is great and the support is superb.

but i have one question to all the slimraw users:

so what do you do with the not compressed files?
is it safe to delete them?

thank you.

jmanord

I decided to keep the MLV files as my backup copy, rather than backing up the dngs created from slimraw. I use lightroom for all my color correction, and use the catalog to store the edits rather than within the individual dng files, so I can always restore the dngs from the original MLV files without loosing my edits. I'm sure this is true with Davince Resolve as well.

Kharak

Bought it!

Love it!

I haven't used this on any of my personal projects, cause I have OCD for quality loss, so even though its lossless my OCD doesn't allow me to compress anything I personally stand by, but been filming a play these last days and my harddrive space was shredded and I've just had the Trial of SlimRaw on my desktop staring at me for a long time, but I gotta say, for projects like this, this is absolutely fantastic.


Converted the MLVs with Raw2Cdng to 12 bit and then "Slimmed" them, so all in all I'd say its a 60% reduction from Original MLV's.

And lightning fast compression too.
once you go raw you never go back

cpc

Thanks for the kind words, Kharak.

I usually suggest staying in 14 bits though. In most cases, if you simply convert to 14-bit dng and compress that with slimraw it will usually shrink the files to less than half the original size with no quality loss whatsoever. On the other hand, the quantization to 12-bit that you are doing in your initial dng conversion is losing you quality for (I presume) a tiny size gain.

mothaibaphoto

@cpc: There is no 14 bit CDNG out of any ML tool :)
Does 16 bit brings really significant overhead?
I didn't try your slimmer yet, but Adobe's Converter compresses almost to the same degree 14 and 16 bit:
MLVProducer CDNG compressed Adobe DNG Converter - 4,83 MB (5 070 338 bytes) (16 bit)
MLV_dump DNG compressed Adobe DNG Converter - 4,82 MB (5 060 678 bytes) (14 bit)

cpc

slimRAW doesn't require cdng, it works fine with dng files so anything converting raw/mlv to straight 14-bit dng is fine (like, say, raw2dng or any of the GUI frontends that use it).

But yes, you can go 16-bit, and the overhead over 14-bit is negligible both in size and decompression speed. The reason for the similar compressed size is that the 16-bit uncompressed really only uses 14 meaningful bits and 2 redundant bits. And handling redundancy is exactly what lossless compression does. Still, 14-bit is optimal.

Also, unlike uncompressed 14-bit dng, lossless compressed 14-bit dng/cdng is widely compatible (for example, Premiere chokes on 14-bit uncompressed but works fine with 14-bit compressed).

mothaibaphoto

Yes, but "anything converting raw/mlv to straight 14-bit dng" has no timecode, FPS and so on - this why the question about 16 bit CDNG rises.
As for me - I just bought a couple of 2TB discs and for some time storage problem is solved :) But later I definitely come back here, ... or buy another couple :)

Kharak

Hi cpc, you are welcome.

I use Raw2cdng, so I can't convert to 14 bit. Only - 12 bit maximized, 16 bit and 16 bit maximized.

I converted to 12 bit to save space, I filled up 6 TB these last 6 months and I recorded 3 hours more of footage from the Play, so 3 hours in MLVs and converting them to 16 bit just don't fit on my drives. I am looking through the Compressed footage and so far no errors or missing frames, so I will delete the MLV's when I have checked all of them. First half of the footage I didn't have the Verification Pass checked, but can you tell me what the Verification Pass actually verifies. Is it doing the job for me, that I am doing right now?

The play doesn't require any colour grade, just colour correction and some contrast to make the look natural. So 12 bits is more than enough for that.

And another question, the Cdng's are not playable in MLRawViewer after compression, It plays a few frames and crashes. I also see that top half of the image is "broken", only in mlrawviewer. Not sure how Mlrawviewer works, might be it "expects" a certain uncompressed footage.. I don't know. With CC compatibility, the Compressed footage runs smooth in PP.
once you go raw you never go back

cpc

@Kharak:
Verification Pass will check for missing files and for checksum errors.
"Missing file" would be any dng file present in the input folders and missing in the output folders. This should never happen, if it happens then something is seriously messed up with the filesystem of the output device. Note that if the input device filesystem is messed and the OS/software doesn't see a particular input .dng file, then there is no way for the software to recognize that, so it can't report it missing in the output.

Checksum verification calculates the checksums of the written output (bypassing the OS file caches) and compares them to the checksums of the data in memory. A checksum error generally means the output storage device is unreliable, or RAM is faulty.

AFAIK, mlvrawviewer doesn't read all types of lossless dng correctly. It seemed to crash a lot on dng playback last time I checked some months ago.

Premiere has seen some serious performance improvements in DNG raw processing in CC 2015. I believe it is only second to Resolve in terms of performance at the moment.

@mothaibaphoto:
Well, one of the cool things with slimRAW is that half the size means half the storage throughput needed in post. This is especially nice with HDDs as it can be all the difference between choppy playback and smooth playback. :)

cpc

slimRAW 1.4 has now been released and it adds 3:1 and 4:1 lossy CinemaDNG options. Here are the release notes: http://www.slimraw.com/relnotes.html
Note that lossy CinemaDNG is only compatible with Davinci Resolve for now. Also note that all lossy CinemaDNG is 12-bit, so any 14- or 16-bit DNG/CinemaDNG will be converted to 12-bit before compression.

Also, MLVFS support (in the OS X version), previously available on demand, has now been rolled into the official release. And there are some minor performance improvements when handling 14-bit raw files (I am now getting 150+ fps when compressing 14-bit FullHD raw on an i7 4770), so updating is recommended even if you don't care for lossy compression.

DeafEyeJedi

Thanks for the much needed update and excellent work as always @cpc!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Kharak

once you go raw you never go back

Danne


cpc

Thanks, guys! This one took a long time of testing and tuning, but I am quite pleased with the results.  :)

SamS

Hi,

I am about to start shooting my first documentary with 5D3+ML in Asia. I have been following this lovely forum for quite a while and learnt a lot but time is running out and theres still lot to learn and to understand. I would love to have idiot check from someone who has knowledge and experience is my workflow bulletproof (or at least rain proof) cause this is something I havent tried yet:

Shooting to MLV -> slimraw -> CDNGs to hardrive and backup, erase CF-cards and original MLV. files. I know it may not be the best idea to erase original DNG-files but as we are shooting non-budget doc (or lowlow budget), all the pennys we can save makes difference. Perhaps I should always go through CDNG to be sure that theres no erros? Btw is there a tool for that? Any suggestions?

My back up plan is to shoot with h.264 If everything will falls apart. But after working only a short moment with ML it would feel wrong :)


Danne

Slimraw is a great tool. Check out mlvfs as well and maybe reconsider erasing your mlv files.

swinxx

@sams
i would not recommend deleting the mlv files.. hdds are not that expensive. just buy a second one and save the original files there.
when the developers make some progress with the conversion from mlv to cdng (mlvfs) its always good to keep the backup files, cause then you are on the safe side.
best wishes, sw

hjfilmspeed

Not sure if this was mentioned yet but I use MLVFS Fuse for windows, quick mount MLVs to DNGs then use slimraw to lossless compress the mounted dngs. Then your left to grade the reduced size DNGs. This will save you a lot of time on big projects and some space.

cpc

Using MLVFS to feed MLV into slimRAW should probably be the fastest way, since this skips the original mlv->dng conversion and directly outputs compressed cdng.
While slimRAW is, I believe, extremely reliable and can also verify the written output for you, it obviously can't verify the MLV input for errors since it has no way of recognizing them. And if there are errors in the MLV, these errors will translate to the DNGs.

Now, of course, you shouldn't be getting errors in the MLV in the first place, but you never know with (some) CF cards and high rate recording. I am a bit behind on ML developments, but back when I was shooting ML raw (MLV didn't existed yet) I would occasionally get corruption in the ML .raw files (the precursor to .mlv) cause I liked using some of the "forbidden" ML tools (features requiring Global Draw ON which was supposed to be turned OFF for raw recording). Dunno, maybe this is not an issue anymore.

But nothing beats visual inspection of the material. I like Assimilate Scratch Play for this (cause it starts almost instantly), but you can use anything that plays DNG footage.