Smooth GPU RAID setup for MLVFS snappy smooth Premiere video preview playback?

Started by erikbaldwinson, April 07, 2017, 01:36:45 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

erikbaldwinson

Who has MLVFS flying in Premiere Pro and has great performance in timeline and playback?

What GPU and SSD HDD Raid 0 setup are you using?

I'm on windows 7 64bit using Premiere Pro CC 2015.

Cheers,
Erik

justinbacle

Hi,

I use MLVFS with both Premiere/AfterEffects and Resolve

Here is what I understood :
- MLVFS uses CPU for processing and transcoding on the fly you MLV files into dngs.
- Premiere (Lumetri) /Resolve  uses GPU for grading the footage
- Première just Looooves scratch disks, so a good SSD is required.

Here is what sucks :
Adobe Camera Raw (e.g. the processing algorithm you use in AfterEffects / Photoshop) still wanna use the one CPU thread for everything and is damn slow.

Here is my configuration :
CPU : i7 4790k
GPU : Nvidia GTX670
SSD : 240Gb Kingston

I manage to get :
- good performance with resolve (though I just work with 2k footage)
- acceptable performance in premiere (Performance drops when I use more than two lumetri layers)
- unusable performance in after effects -> If I wanna use after effects (because ACR is just damn good) I have to render intermediates and cannot work in real-time. Which is a pain as I prefer ACR's noise reduction and overall quality to resolve.

I'd say get a good CPU (i7 or Ryzen) and good GPU (AMD & Nvidia seems to works both fine now) and a large SSD for your OS/Scratch

bpv5P

The latest Nvidia Quadro should have a good performance.

Any good SSD will do the job, I would choose a M.2 input, though, since it has better performance.


Other systems may give you best performance. For example, generally open source software will perform better on linux, because generally the original port of this software was written on linux. So, test cr2hdr, for example, on Debian (not on Virtual Machine).

erikbaldwinson

I'm looking for specifics.

What write size speed benchmark should one be looking for with the SSD?
What Quadro? Why not just Geforce and than 2nd gpu cheap used Quadro 2000 for 10bitpercolor (30bitcolor) mercury transmit output in premiere?

bpv5P

QuoteWhat write size speed benchmark should one be looking for with the SSD?

>400MB/s is good. The M.2 I was talking about is like this (your motherboard need to support it):
https://www.newegg.com/Product/Product.aspx?Item=N82E16820147467

QuoteWhat Quadro?

The Quadro P1000 will do the job:
https://www.amazon.com/PNY-NVIDIA-Professional-Graphics-VCQP1000-PB/dp/B06X9SC9SD

QuoteWhy not just Geforce and than 2nd gpu cheap used Quadro 2000 for 10bitpercolor (30bitcolor) mercury transmit output in premiere?

Quadro is optimized for graphics work. It has all the API's uptodate. You can use a normal Geforce, but Quadro will be more optimized for what you need.

erikbaldwinson

My motherboard doesn't have m.2 - would an adapter offer the same speed? it just requires a pci slot

What about this SSD? The size is desirable - https://www.newegg.ca/Product/Product.aspx?Item=N82E16820226855&cm_re=ssd_1tb-_-20-226-855-_-Product

The quadro P1000 is very new and about $500CAD after tax.

Are you telling me that this P1000 card is going to crush my ($150CAD) GTX 750 Ti paired with a ($150CAD) Quadro 4000 (for 10bitpercolor mercury transmit playback in Premiere)?

How/Why?

bpv5P

Quotewould an adapter offer the same speed?

no. Just go with the SATA ones.

QuoteWhat about this SSD?

not too good. The benchmarks I've see got about 375MB/s.
The Samsung EVO would do the job better.

https://www.amazon.com/Samsung-2-5-Inch-Internal-MZ-75E250B-AM/dp/B00OAJ412U


Also, don't mind with the SSD capacity. You should not use it to store your info, it's not reliable as the old HDD.
SSD is mostly used for your system, so it can store caches at high speed. Also, NTFS (windows default file system) quite sucks on SSD's, hence what I have said before: the system is crucial for performance. For example, linux has support for F2FS and Btrfs that has much better performance and stability.
If you have to use windows, at least strip all the bloat from it, and use Home Basic instead of the other fancy versions:
http://installgentoo.wikia.com/wiki/Windows_7_De-Bloating

Quoteis going to crush my

no. What I'm saying is that Quadro, especially the new ones with Pascal architecture, is much more optimized. It will waste less energy, will have less heat, will perform better on most of the tasks in comparison with some non Quadro board with the same ram memory / clock speed.
I don't know if you can use two GPU, btw. I think you would need an SLI system and not all boards support that.

Walter Schulz

Quote from: bpv5P on April 08, 2017, 10:34:57 AM
no. Just go with the SATA ones.

Please tell why not using an m.2 device locked into an PCIe adapter.

erikbaldwinson

Quote from: bpv5P on April 08, 2017, 10:34:57 AM
Also, don't mind with the SSD capacity. You should not use it to store your info, it's not reliable as the old HDD.

So use HDD to store media and use seperate SSD for a premiere pro scratch disk? How significant of a role does the RAM play with MLVFS and premiere in general and what would you recommend size/speed?

In windows7 you can install two cards.  I just tested, installed seperate drivers for the NVS 295 and 750 Ti and they both installed and had normal functionality alongside each other, mercury transmit recognized the second monitor via display port from the NVS295. Unfortunately the NVS 295 doesn't do 10bitpercolor and the card is interrupting the fire-wire audio interface I have. I read others had this issue, hoping the quadro 4000 won't bother the firewire, found one for 150.

If this fails I'm wondering if the P600 or the K1200 standalone will perform? I'm really not excited to drop 500 on the P1000.. Thoughts?

All the NVIDIA promo material seems to suggest the low end quadros (like p1000) offer no significant improvements besides 10bit so I'm focused on working around achieving a 2 gpu setup (gtx + quadro) unless I can find damning evidence otherwise..

bpv5P

Quote from: Walter Schulz on April 08, 2017, 10:43:35 AM
Please tell why not using an m.2 device locked into an PCIe adapter.

Well, good point. I was wrong, a PCIe adapter would work too.

Quote from: erikbaldwinson on April 08, 2017, 11:09:43 AM
So use HDD to store media and use seperate SSD for a premiere pro scratch disk? How significant of a role does the RAM play with MLVFS and premiere in general and what would you recommend size/speed?

Yep. SSD for caches or files that need a high read speed, such as OS lib files.
RAM plays a important role, but more than 16GB is not necessary if you're doing normal work (read: not some massive Blender rendering). I think 8GB of DDR3 with standard speed will work good enough for what you seem to need.

Quote
In windows7 you can install two cards.

Well, good. But you have to test if it's really working. I don't think it will work together, you probably are using just one of them. I think you would need something to get then working together, an SLI, as I said.  You can ask someone on Nvidia related forums about this...

Quote
If this fails I'm wondering if the P600 or the K1200 standalone will perform? I'm really not excited to drop 500 on the P1000.. Thoughts?

P600 would work good too. The K1200 uses an old architecture, don't know if it will perform good with the newest softwares...


Also, you guys should take my advices with a grain of salt. I'm not an specialist, just a random guy trying to help here.

Walter Schulz

Quote from: bpv5P on April 09, 2017, 02:02:45 PMWell, good. But you have to test if it's really working. I don't think it will work together, you probably are using just one of them. I think you would need something to get then working together, an SLI, as I said.  You can ask someone on Nvidia related forums about this...

Problem may be performance drop. Most motherboards have a very limited amount of PCIe lanes and using two graphic cards will drop interface bandwidth by 50 percent. Using PCIe adapters for m.2 will not help either. Read specs with care.
Example:  If specs for PCIe are (x16/x0/x4*, x8/x8/x4*) you may have 1 PCIe card connected by 16 lanes and have the option to use an PCIe adapter for m.2 with 4 lanes. Or use two graphic cards with 8 lanes each and running PCIe adapter for m.2, too. Most entry level motherboards don't allow that. Motherboards optimized for dual graphic card configuration be able to offer x16/x16/x4 but will come with a much higher price tag.

reddeercity

Read over these posts and I was going to pass by but I through I would throw my 2 cents worth in .
1st I think you have the pipe line wrong , or not wrong but very inefficient .
I know this is a Win 7 platform the OP was talking about and I Love Win7 , it does the job
not like win 8 ,10  :( ( My opinion ) On Windows there to option for mlvfs
One is the current Web Gui version which is fine but can be slow (MAC Version is way faster  :D )
with all the extras post processing features , that's the way it is . and the other is ( which is no longer used)
Quick mount with "Prismo" file mount software and a windows mlvfs.dll .
Read about it here at the bottom of the page and pfm-171 is the version of mounting software  I use on Win7 .
I feel is very fast , I have both systems install on my PC and I will talk about hardware in a bit .
Once installed all you have to do is right click over the .mlv file and it will open up in the same folder/dir/drive you are in.
even on the CF card or SD card but without all the extra post processing options , thou if you expose correctly , etc.... and do mess up
there really no need for extra process (usually) . So this may be faster for those who feel web gui is slow , there are other way of course of
exporting the Cdng's from .mlv's like Raw2Cdng  (win platform) Very fast  and then import in to Adobe

About hardware , I use this in my pro work so it has to be fast & safe . I run a AMD FX8350 (don't laugh) overclock to 4.8GHz with 8GB 1600mhz Ram
the PCIe bus I bump up to 105Mhz from 100 gives a snapper response , faster GPU , faster data i/o . I run 2 GPU's GTX580 1.5GB vram each over unlocked version
I use MSI After burner to over clock more , I run my OS on Kingston SSD 240 GB and Mass on board store 8TB raid0 (4-2TB drives)
fyi mother board Old version1.0 Sabertooth 990FX , 3 PICe slots one at x16 & dual @x16 .
The point is that Older hardware still works and is fast enough for post video work , and no you don't have to have a SLI connector on dual gpu's actually it not recommended for video work
In Resolve 12.5 I have access to both gpu's one for GUI stuff and the other for GPU computing , and I monitor the timeline with a little USB 3.0 device for Blackmagic Design  UltraStudio SDI
with the HDMI @ 10bit . I get about 45-50 frames per second in resolve so twice my 24p timeline frame rate .
So I run all my file out of my raid0 R/W on average ( depending on how full the raid is anything over 50% full slows the raid down by 30to 40%) 700-800MB's
with bursts up to 2800MB/s R/W  :o , and raids are cheap to build these days and my raid is internal it's 4 striped drives.

Yes ACR A.E. is slow because it use just CPU and Frame store to process Very Slow but Very Good (Nothing better)
but PP does not have the same problem , thou I'm on CS6 so when I use Prismo quick mount I don't see any issue with performance.

There are work around of course , Proxy workflows (I use this on fat files) or better yet Log-C work flow  :) :) :) 
This is the much more the desirable pipeline workflow , just convert you files to Log-C , de-log in App and grade no
performance penalty ! Sorry when OT a little and on a bit  :D
There also many options in PP to help increase performance  e.g. timeline codec for one , here use a light codec (more compression)
when rendered  you still use the raw file .

Or Better Yet Since you have a Intel cpu i7 build your self a Hackintosh and go MAC
" Once to go MAC you Never to Back "  :P :P

OK Ok I'm done now  ;D

Edit :
just a note sorry any window app that can use dual or more GPU will !!
e.g. I use Microsoft expressions pro (H264) to render web streaming file for my HD Web Site and
it uses any many as I can install even on 8x & 16x PCIe slots  it doesn't care , one stream my
finish a few micro seconds later . It just cares how many gpu's core you have to use .

erikbaldwinson

Quote from: reddeercity on April 10, 2017, 12:26:28 AM
Read over these posts and I was going to pass by but I through I would throw my 2 cents worth in .
Thx

Quote from: reddeercity on April 10, 2017, 12:26:28 AM
One is the current Web Gui version which is fine but can be slow (MAC Version is way faster  :D )
with all the extras post processing features , that's the way it is . and the other is ( which is no longer used)
Quick mount with "Prismo" file mount software and a windows mlvfs.dll .
Read about it here at the bottom of the page and pfm-171 is the version of mounting software  I use on Win7 .

Why is it no longer used if it's faster?

Quote from: reddeercity on April 10, 2017, 12:26:28 AM
Once installed all you have to do is right click over the .mlv file and it will open up in the same folder/dir/drive you are in.
even on the CF card or SD card but without all the extra post processing options , thou if you expose correctly , etc.... and do mess up
there really no need for extra process (usually) .

How does this virtual drive work/behave? Does it negotiate with ram mostly or is it taxing the raid drive setup which hosts the original MLV files? Is it taxing both the read and write speeds? How fast read/write would you guess is required for smooth playback?******** This info is important because all the specs of the required hardware depend upon it.


Quote from: reddeercity on April 10, 2017, 12:26:28 AM
I use MSI After burner to over clock more , I run my OS on Kingston SSD 240 GB and Mass on board store 8TB raid0 (4-2TB drives)

So you have the MLVFS files on the Raid as well as the scratch disks and page file? I think its probably best to have a separate SSD for scratch and page. How big required tho?


Quote from: reddeercity on April 10, 2017, 12:26:28 AM
In Resolve 12.5 I have access to both gpu's one for GUI stuff and the other for GPU computing , and I monitor the timeline with a little USB 3.0 device for Blackmagic Design  UltraStudio SDI

Samples to YUV 422 otherwise I would spend the money on this or the mini-monitor despite being overpriced.

Quote from: reddeercity on April 10, 2017, 12:26:28 AM
with the HDMI @ 10bit . I get about 45-50 frames per second in resolve so twice my 24p timeline frame rate .

What about premiere pro? What are you using? How is the timeline?

Quote from: reddeercity on April 10, 2017, 12:26:28 AM
So I run all my file out of my raid0 R/W on average ( depending on how full the raid is anything over 50% full slows the raid down by 30to 40%) 700-800MB's
with bursts up to 2800MB/s R/W  :o , and raids are cheap to build these days and my raid is internal it's 4 striped drives.

That's mb/s not mbit/s? That's fast! What drives?

Quote from: reddeercity on April 10, 2017, 12:26:28 AM
thou I'm on CS6 so when I use Prismo quick mount I don't see any issue with performance.

95% thx to your raid setup I would guess? No?

Quote from: reddeercity on April 10, 2017, 12:26:28 AM
There are work around of course , Proxy workflows (I use this on fat files) or better yet Log-C work flow  :) :) :) 
This is the much more the desirable pipeline workflow , just convert you files to Log-C , de-log in App and grade no

Using Pro-res or something? This is what I have been doing - take a lot of time. I think the whole point of mlvfs is to retain full requality cDNGS straight from card into premiere from mlv files without any time wasted.

Quote from: reddeercity on April 10, 2017, 12:26:28 AM
just a note sorry any window app that can use dual or more GPU will !!

I have a 750ti as my main card and I tested out plugging in a nvs295 quadro to see if I could use it as a playback card with my 10bit monitor (i know now the nvs295 doesnt do 10bit). The driver installed and windows and nvidia reognized both cards and software funcationality. In premiere the mercury transmit recognized the 10bit monitor attached to the NVS295 via display port and sent the program feed to it. The only problem was my firewire sound device (which is known to be finiky, was not intilizing.) Someone is sending me a Quadro 600, hopefully that doesn't interfere with the firewire and I will have 10bit display port out to the 10bit monitor in premiere via mercury transmit. Unless I'm missing something here I think I'm saving a lot of money to achieve 10bit color..?

I will probably pick up a used 970 to replace the 750 and be flying. I can't find any Quadro GTX comparison material that suggests premiere performance will do better with quadro. I found one experiment that actually suggests the super affordable quadro K1200 was actually better in premiere than the pricey quadro K4000 just to make matters more confusing.. http://professional-workstation.com/news/technology/96-gpu/191-which-nvidia-quadro-for-adobe-video-2

Seems like quadro is for the 3d world but adobe is trying to appeal to the video editing world for revenue or branding purposes. I can't find any material on the brand new quadro P600 or P1000 that suggests these cards are any different in the regard. Any opposition?

erikbaldwinson

So the Quadro 600 which has OpenGL 10-bitpercolor is a bottleneck when plugged into my HP LP2480zx via display port but sent a laggy 1080p video preview in premire pro via mercury transmit. Overclocked it to absolute max and minimal improvement, dropping frames often. I also had a Geforce 750 Ti installed and tried turning off everything and putting as much load on the 750 as NVIDIA's user interface would allow, cuda etc. Minimal improvement, dropping frames often with both MLVFS and prores 422 files.

Disable the Quadro 600 and performance is basically "normal" altho it's dropping a frame or two every 15 seconds. The 750 Ti load runs at about 8% and the CPU is about 20% so I guess the dropped frames is a result of the SSD and the RAM negotiation? Any ideas?

Looks like I'll probably try the Quadro 4000 or slightly higher next.. it's similar to 750 Ti so should be good.

Cheers,
Erik

dmilligan

Quote
Why is it no longer used if it's faster?
It's not supported or maintained. There is no evidence it is faster (that I am aware of). I think maybe he's trying to say that that the newer version has more processing options and they may slow things down if you turn them on (which is true).


The virtual drives are in fact virtual, and they have no bearing on the physical drive they appear to be mounted on. You may mount them to their own drive letters if you like (in the dokany version).

erikbaldwinson

Quote from: dmilligan on April 12, 2017, 02:15:07 PM
The virtual drives are in fact virtual, and they have no bearing on the physical drive they appear to be mounted on. You may mount them to their own drive letters if you like (in the dokany version).

Well they must have some bearing to the physical drive they are on because that drive has to be accessed to mount them. Are you saying that once they are mounted there is no longer a relationship between the drive and the mounted files..?

What bears the load then? Memory? Or are you saying that the virtual files are just masks (for the purpose of navigating in the OS) and the real files that are being accessed still reside on the drive and so when you import the virtual dngs in your editing software what you are really doing is importing the mlv files disguised as virtual files and reading them from the original storage drive?

**I'm trying to understand which computer component is doing the work when editing using the MLVFS workflow..

dmilligan

QuoteOr are you saying that the virtual files are just masks (for the purpose of navigating in the OS) and the real files that are being accessed still reside on the drive and so when you import the virtual dngs in your editing software what you are really doing is importing the mlv files disguised as virtual files and reading them from the original storage drive?
Yes, basically.

MLVFS works by basically intercepting file I/O calls made by programs to the OS.

For example a program can call a function in the OS to get a list of files within a directory. For calls to paths that are inside the mount directory (or mount drive letter) MLVFS intercepts this call, looks for real MLV files in some real directory, perhaps find a matching MLV file, find the number of frames in that file and then return something back that says "there are x dng files in this directory", even though there is no such physical directory that exists, the program will think there is.

Then the program can call a function in the OS to read data from a particular file it has been lead to believe is actually there. MLVFS intercepts these calls in the same way, and then responds with the requested content (since there is no such physical file). To do so MLVFS will have to read some particular frame out of the MLV file, process it, apply any image processing the user has enabled and wrap it up with headers to look like a DNG.

erikbaldwinson

Thank you for explaining it in layman's terms as best as you could.