Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - cpc

#1
Quote from: Levas on July 16, 2020, 11:52:49 PM
I'm using adobe dng converter to compress 'old' uncompressed 14bit ML dng files.
Adobe dng converter has this option to only use lossless compression. I was always assuming adobe dng converter uses lj92 compression. But after last few posts in this topic, I'm in doubt? Isn't adobe dng converter lossless compression option the same as lj92 compression used in mlv's ?

Anything "lossless dng" uses the same spec when compressing the pixel data. But the spec allows for a lot of wiggle room and choices to make, which accounts for significant differences in compressed pixel data size. SlimRAW will produce files significantly smaller than DNG Converter and way smaller than anything that uses Blackmagic's setup for Canon linear raw (usually ~15% smaller, sometimes more). That's because slimraw tries a lot of stuff on the fly to get the smallest results (and even more stuff to try can be enabled through an option), while the others fix parameters in advance. Regardless, results will be truly lossless with any compressor, and compressed files will decompress to the same original file. (Well, DNG Converter will convert all input to nominal 16-bit regardless of true input bit depth, so the decompressed file will be nominally 16-bit, but that's mostly technical details.)

The thing is, DNG Converter is made for photos, and it has the peculiarity that it inserts thumbnails (previews) in each frame which certainly is pointless for video, and it adds a bunch of useless conjured metadata. All this bloats the output size. This is what gets stripped by DNGStrip. But the main issue that initially pushed me towards making slimraw back in the day was the excruciatingly slow processing of DNG Converter. Well, that and the fact that there was no log raw output in DNG Converter. :)
#2
Quote from: Danne on July 16, 2020, 09:13:13 PM
Hi cpc! Long ago :).
Switch is more or less a sleeping beauty nowadays. Mlv App is standard more or less. However compressed dng from mlv uses lj92 compression introduced by A.Baldwin in MlRawViewer: https://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html

Better than adobe dng and cdng but of course not as fast as slimraw ;).

Hi Danne. Good to know. Thank you for the heads up, I haven't been following developments closely for some time now. (Ok, I should probably say "at all" instead of "closely". ;) )

I glanced through your link and it seems that he is describing Blackmagic's way of doing the prediction, which isn't particularly good for linear Canon raw. I would guess that DNG Converter + DNGStrip should produce smaller files (well, if one could live with the painfully slow DNG Converter, that is :) ).
#3
Quote from: adrjork on August 07, 2019, 08:48:43 AM
Hi, sorry if I re-open this old topic...
I've seen that also Swicth has an option for compressing DNGs (with LJ92) from MLVs, so which is the difference between compressing DNGs in Switch versus doing it with MLVFS+SlimRAW? (Believe me, as I wrote in another topic, I really don't want make a SW-vs-SW challenge: I'm just curious to know which are the differences between compressing DNG with Switch and with SlimRAW since both are in this forum and both are loved by users, and both have a similar function, so I suppose both have great PROs and perhaps are useful in different situations.)

And I confess I don't clearly understand if SlimRAW lossless compression methods is "visually lossless" or "data lossless", and if someone noticed any slowdown in Davinci playback with compressed DNGs (in comparison with regular DNGs).

Thanks a lot.

A bit late, but here are some answers for posterity.

First, your lossless DNG question is answered in slimraw's faq: http://www.slimraw.com/faq.html
Second, performance in Resolve with compressed DNG will depend on what's the bottleneck of your system when streaming the raw footage. If storage is the bottleneck, Resolve performance should improve. If the CPU is the bottleneck, performance may degrade.

As for the differences between slimRAW and Switch, Danne will correct me if I'm wrong, but my understanding is that Switch uses Adobe's DNG Converter. DNG Converter is no match for slimRAW neither in terms of file size achieved, nor in terms of speed (last time I checked, slimRAW was 32 times faster on a quad-core CPU). But hey, Switch is free and I am sure Danne is doing a great job with the whole Switch pipeline. If you aren't in a hurry, you may gain some additional space by passing DNG Converter's output through DNGStrip, a free tool that strips some of DNG Converter's useless chaff from the compressed files, which I've written some years ago: https://www.shutterangle.com/2014/lossless-compression-for-dng-raw-video-dngstrip/
#4
Cameras have nonlinearities in the signal above some level. This usually leads to tints and improperly reconstructed highlights when this part of the range is used for recovery. You'd need to restrict the upper limit and discard some of the signal. This is what the white level metadata is used for, it tells the raw processor where to cut. Ideally this is calibrated per camera, but you can usually get the approximately correct values by running some stills through exiftool and seeing what's supplied by the camera manufacturer.
#5
Looks like the white level is too high, you should be able to remedy this by pulling it down.
#6
New version 1.8 is now released with support for ML in-camera lossless dng stills. Also, a new 7:1 compression option and a small performance boost for all lossy compression.
Upgrade here: http://www.slimraw.com/download.html
#7
Raw Video / Re: Slimraw 3:1 compression with ML dngs?
December 06, 2016, 02:13:50 PM
Can you attach a sample frame?
#8
Raw Video / Re: Slimraw 3:1 compression with ML dngs?
December 06, 2016, 01:14:47 PM
Quote from: Markus on December 05, 2016, 11:44:52 PM
Mlvs mounted with pismo file mount. Compressed with slimraw to hard drive. Processing in resolve 12.5.
Latest version of Slimraw.
Any issues with the current versions of slimraw and mlvfs? I've tried 10, 12 and 14 bit files. Everything is fine here.
#9
Raw Video / Re: Slimraw 3:1 compression with ML dngs?
December 05, 2016, 03:26:36 PM
Quote from: Markus on December 03, 2016, 03:02:07 AM
Last I tryed I only got white frames in resolve tough.
This shouldn't be happening. Certainly not now, but also not in the past. How do you produce your dng files?
#10
And another update v1.7 is now released. Adds DNG downscale to half the horizontal and vertical resolution. Not particularly useful for ML raw footage, but it can be handy for reducing the excessive size of raw timelapses (will have to convert cr2 to dng first), particularly when going for hd/2k deliveries, as well as for raw proxies in general. Here is a post on using raw proxies: http://www.slimraw.com/article-proxies.html

Also, some optimizations here and there, so update anyway, even if you don't care for rescales.
#11
QuoteAfter seeing Danne's post, I was curious as to where, or if, there is documentation for command line options for slimraw. The Lossless 10bit log compression is awesome!

Command line is not "officially" supported and is only available on Mac, you can list the available command line options (including a short description) with something like this (just put a dummy option after the program name, it will dump the full options list in the terminal):
path-to-the-app/slimraw.app/contents/macos/slimraw blabla

#12
Feedback is always welcome. :)
#13
New version 1.6 is now out with 4 new compression modes.
Added 5:1 lossy compression and two variable bit rate lossy modes. But perhaps most interesting for ML users will be the new 10-bit log encoded lossless mode. The linear 14-bit signal is mapped to a 10-bit coding space using a log tonal curve, similarly to Canon C500 10-bit log raw. Log data is then losslessly compressed. This results in 20-30% smaller files than straight lossless compression, while preserving high image quality. The log tranform is transparent, it is reversed by the raw processor on import; no workflow changes required. The output is compatible with pretty much all DNG capable video software.

All lossy compression now also uses a non-linear transform, which allows higher levels of compression by keeping stress off dark tones.

More about the different compression modes here: http://www.slimraw.com/article-cdngmodes.html
#14
QuoteI have a few questions before I commit to purchasing SlimRAW.  The first and most important is about performance in Premiere and After Effects.  Currently I have an i7 3930K and have a server feeding a 10GbE pipe with 650MB/s of read and write performance.  As it stands I get choppy, and drop frame performance reading 16bit CDNG content on Premiere Pro.  I converted these the MLV files using RAW2CDNG (which I has proven to be the best performance and reliable out of all the options out there).  I want to know if this will actually improve performance, as resolve plays the files fine, and PPcc has hiccups.  This has lead me to believe that PPcc is the issue over file size and data rates.  So, can anyone give me a reasonable explanation about why slimraw would make playback and editing better in PPcc?

Second, I would like to know if anyone has had any issues with crashing or bad frames in the compression process.  I've had my fair share of issues finding the right software to do the conversions from MLV, and I finally found a version of raw2cdng that works pretty reliably.

Third, how does licensing work?  I generally shoot for myself, but I use a computer at work to process footage occasionally.  Do I need to purchase two separate licenses, or will it allow me to use the same license on each computer (like Adobe products) as long as I don't have both using the same license at the same time?

Thanks, looks like a cool product to save some expensive RAID storage.

re: Premiere performance
Strange that you don't get realtime playback in Premiere with your specs. I haven't tried 16-bit uncompressed files, but I get smooth playback of 14-bit losslessly compressed 1080p DNG on my quad i7 4770. Have you tried processing from local storage? Could be Premiere doesn't like the network storage setup for whatever reason?
To answer your question, it is possible that using lossless DNG will improve the situation since Premiere would need to stream significantly less data, and the lossless compresson will be no problem for your hexcore cpu.

re: stability
FWIW, I've never had a crash report in the year and more that slimRAW's been out. Obviously, if you have a bad frame before compression, it will stay bad after compression. No way around this.

re: licensing
slimRAW can be used on all your computers with one license, and this includes both Mac and Windows platforms.
#15
New version 1.5 is now released with some improvements. Release notes here: http://www.slimraw.com/relnotes.html
#16
QuoteMLV --> Cdng 16bit -->Log ProRes4444XQ+alpha=16bit

Color channels in ProRes4444XQ are 12-bit. Only alpha is 16-bit and it is optional, you will generally want to skip it, cause it only inflates size in this case. :)
#17
@eyeland:
Lossless CinemaDNG size varies with content, but if we assume an average compression ratio of 2:1 for ML raw, the calculation goes like this:
FullHD 14-bit lossless CinemaDNG: 1920*1080*14/8*24/2 ~= 43.5 MB/s (at 24fps)
FullHD 12-bit ProRess4444XQ: 396 mb/s = 49.5 MB/s (at 24fps)

@hjfilmspeed:
As noted by Kharak, you need a very fast storage setup to hit these speeds. For 150fps you'll need almost 550MB/s sustained storage reading speed and around half of this rate on the output end. This means very high end SSD, and preferably a RAID of SSDs. For real time processing, you need a sustained reading throughput of around 87 MB/s (at 24fps). If you are processing straight from the CF card (mounting raw through MLVFS), you are most likely limited by the throughput of your CF card + CF reader combination (many readers are way slower than the CF card itself).
#18
Using MLVFS to feed MLV into slimRAW should probably be the fastest way, since this skips the original mlv->dng conversion and directly outputs compressed cdng.
While slimRAW is, I believe, extremely reliable and can also verify the written output for you, it obviously can't verify the MLV input for errors since it has no way of recognizing them. And if there are errors in the MLV, these errors will translate to the DNGs.

Now, of course, you shouldn't be getting errors in the MLV in the first place, but you never know with (some) CF cards and high rate recording. I am a bit behind on ML developments, but back when I was shooting ML raw (MLV didn't existed yet) I would occasionally get corruption in the ML .raw files (the precursor to .mlv) cause I liked using some of the "forbidden" ML tools (features requiring Global Draw ON which was supposed to be turned OFF for raw recording). Dunno, maybe this is not an issue anymore.

But nothing beats visual inspection of the material. I like Assimilate Scratch Play for this (cause it starts almost instantly), but you can use anything that plays DNG footage.
#19
Thanks, guys! This one took a long time of testing and tuning, but I am quite pleased with the results.  :)
#20
slimRAW 1.4 has now been released and it adds 3:1 and 4:1 lossy CinemaDNG options. Here are the release notes: http://www.slimraw.com/relnotes.html
Note that lossy CinemaDNG is only compatible with Davinci Resolve for now. Also note that all lossy CinemaDNG is 12-bit, so any 14- or 16-bit DNG/CinemaDNG will be converted to 12-bit before compression.

Also, MLVFS support (in the OS X version), previously available on demand, has now been rolled into the official release. And there are some minor performance improvements when handling 14-bit raw files (I am now getting 150+ fps when compressing 14-bit FullHD raw on an i7 4770), so updating is recommended even if you don't care for lossy compression.
#21
Random image off the internet, showing AE default settings. Note the 25 under Color Noise Reduction (last slider):


Are you saying you explicitly set this to zero in the AE example?

#22
@Kharak:
Verification Pass will check for missing files and for checksum errors.
"Missing file" would be any dng file present in the input folders and missing in the output folders. This should never happen, if it happens then something is seriously messed up with the filesystem of the output device. Note that if the input device filesystem is messed and the OS/software doesn't see a particular input .dng file, then there is no way for the software to recognize that, so it can't report it missing in the output.

Checksum verification calculates the checksums of the written output (bypassing the OS file caches) and compares them to the checksums of the data in memory. A checksum error generally means the output storage device is unreliable, or RAM is faulty.

AFAIK, mlvrawviewer doesn't read all types of lossless dng correctly. It seemed to crash a lot on dng playback last time I checked some months ago.

Premiere has seen some serious performance improvements in DNG raw processing in CC 2015. I believe it is only second to Resolve in terms of performance at the moment.

@mothaibaphoto:
Well, one of the cool things with slimRAW is that half the size means half the storage throughput needed in post. This is especially nice with HDDs as it can be all the difference between choppy playback and smooth playback. :)
#23
slimRAW doesn't require cdng, it works fine with dng files so anything converting raw/mlv to straight 14-bit dng is fine (like, say, raw2dng or any of the GUI frontends that use it).

But yes, you can go 16-bit, and the overhead over 14-bit is negligible both in size and decompression speed. The reason for the similar compressed size is that the 16-bit uncompressed really only uses 14 meaningful bits and 2 redundant bits. And handling redundancy is exactly what lossless compression does. Still, 14-bit is optimal.

Also, unlike uncompressed 14-bit dng, lossless compressed 14-bit dng/cdng is widely compatible (for example, Premiere chokes on 14-bit uncompressed but works fine with 14-bit compressed).
#24
Thanks for the kind words, Kharak.

I usually suggest staying in 14 bits though. In most cases, if you simply convert to 14-bit dng and compress that with slimraw it will usually shrink the files to less than half the original size with no quality loss whatsoever. On the other hand, the quantization to 12-bit that you are doing in your initial dng conversion is losing you quality for (I presume) a tiny size gain.
#25
The problem with ETTR is that even if it will give the cleanest possible image, it is time consuming in post to match shots in a sequence if each shot is exposed using ETTR. I personally never use ETTR.
But what I often do with any camera is I rate the camera slower than the official rating; in other words, equally overexpose all shots. This trades highlights latitude for cleaner image. You get some of the benefits of ETTR but without sacrificing consistency. And image consistency is beneficial on many levels. Rating slower is especially useful for shooting raw or log transfer curves because manufacturers almost always specify ISOs which maximize DR but result in noisy shadows.

The 5dm3 can be beautiful at iso 1600 as long as it is properly exposed. Here is a couple of images from an old short I shot (I don't have a Canon camera anymore); both at 1600, with no noise reduction whatsoever:





The first image is fine as it is, and the second can use a little bit of NR in the shadows. Pretty clean where it counts though. In fact, passing it through the default color NR settings of ACR would probably deliver pretty clean shadows (I used Resolve on this short).