Author Topic: [deprecated] MLVFS windows client  (Read 94948 times)

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #50 on: September 18, 2014, 10:02:30 AM »
I'll give it a try later today. Thanks.
Everything that might have reduced performance was off, as far as I remember.

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #51 on: September 18, 2014, 07:56:00 PM »
Testing new update.
Antivir and other stuff - off
Power plan - High performance

10s MLV clip (almost static scene)
Resolve render cache off:
first play - 1 min 12s (approx 50% of black frames)
2nd - 30s (80% black frames)
3rd - 12s (only few frames visible, the rest is black)

With Smart render cache:
1st play - 24s (approx 10% of frames visible, rest is black)
2nd - 24s (few frames visible)
3rd - 24s (few frames visible)

When cursor is randomly placed on timeline, if the that frame was black during playback, remains black. Doesn't show preview.

So, in my case, too many bottlenecks to review the app properly. GTX 275 with 850MB VRAM and 7200rpm 1TB HD is OK for H264, but not in this case. Let me know if I can help in some way, but I think my results are not representative because of hardware limitations.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #52 on: September 18, 2014, 08:05:49 PM »
okay i would say anyway that this solution is not meant for realtime playback etc.
is it really necessary for post processing?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #53 on: September 18, 2014, 08:25:04 PM »
Hmm... I'd say that if it wouldn't be possible to have realtime playback regardless of computer parameters, then it becomes quite problematic.

If one can't see the edit or color corrected clips in realtime, usefulness of such solution is rather limited, in my opinion.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #54 on: September 18, 2014, 10:57:43 PM »
update: speeded up code to probably maximum possible speed using network sockets :)
now a GET request takes 27 msec, and 25 msec are "writing the DNG to the socket"

set prefetch images to 100 or more for your tests.

next optimization step: check why disk reads are somewhat slow
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #55 on: September 18, 2014, 11:22:24 PM »
I wish I had good news, but...

Prefetch set to 150.
It took 3 minutes 7 seconds to play the clip. A lot of frozen frames, jumps, and blacks. Practically no playback at all. This time also some pink artifacts like horizontal lines and parts of the frame in stylish "pink and white" color palette  ;)

At the end Microsoft .NET Framework has reported  unhandled exception.

************** Exception Text **************
System.ArgumentOutOfRangeException: Value of '1004' is not valid for 'Value'. 'Value' should be between 'minimum' and 'maximum'.
Parameter name: Value
   at System.Windows.Forms.ProgressBar.set_Value(Int32 value)
   at WebDAVServer.WebDAVServerForm.RefreshBar()
   at WebDAVServer.WebDAVServerForm.UpdateTimer_Tick(Object sender, EventArgs e)
   at System.Windows.Forms.Timer.OnTick(EventArgs e)
   at System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

After that Resolve refused to cooperate. Damn  :D

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #56 on: September 18, 2014, 11:42:04 PM »
confirmed, that the crash may happen.
fixed it.

can you open your browser, enable request logging and look for the GET ....dng requests, how long they take?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #57 on: September 19, 2014, 12:50:49 AM »
can you (...) enable request logging and look for the GET ....dng requests, how long they take?

If you tell me how to to it, of course.
I tried to figure out what am I supposed to do, but have no clue, sorry.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #58 on: September 19, 2014, 02:09:15 AM »
 - browser: http://localhost:8082/log
 - click [Start] at the bottom of the page
 - play some files
 - check WebDavServer window (or browser) for logged GET data

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #59 on: September 19, 2014, 10:37:29 AM »
Here are some values from the beginning of the log:
 412,0236 ms
 470,0268 ms
 1531,0876 ms
 1585,0907 ms
 2219,1269 ms
 2263,1295 ms
 451,0258 ms
 2981,1705 ms
 5494,3143 ms
 5596,3201 ms
 2016,1153 ms
 2953,1689 ms
 596,0341 ms
 424,0242 ms
 186,0106 ms
 553,0316 ms
 10284,5883 ms

Prefetch 150, no render cache in Resolve.
Smallest value i noticed is 52,003 ms. Biggest, the last one in log, is11354,6495 ms. The numbers seem quite high. Did I win something?  ;)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #60 on: September 19, 2014, 11:02:39 AM »
wow is that slow oO
if you view the files' properties using the explorer, what do the numbers look like?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #61 on: September 19, 2014, 11:12:03 AM »
If you mean MLV file i used for the test, it's 767 MB, 10 seconds 9 frames long. No color correction applied.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #62 on: September 19, 2014, 12:13:53 PM »
na, browse the virtual MLV directory where the DNGs are and do a rightclick, properties on a few DNGs one after another.
this should trigger a GET command and you will get again log entries.
tell me how long these GET requests took.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #63 on: September 19, 2014, 01:48:44 PM »
They look like this:

312,0006 ms
62,4001 ms
62,4001 m
62,4001 ms
62,4001 ms
78,0001 ms
78,0001 ms
78,0001 ms
78,0001 ms
15,6 ms
0 ms
31,2 ms
78,0001 ms

The file is in mapped folder on a hard drive. OS and apps are on separate disk in different partitions.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #64 on: September 19, 2014, 09:38:54 PM »
oh in this case...
your machine is way slower than mine.
i guess that is the problem :(
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #65 on: September 19, 2014, 10:36:05 PM »
Yep, that's what i suspected. So far, for h264 it was more than enough. But since I started experimenting with RAW recently, looks like it's time for serious upgrade.
Anyways, thanks a lot, g3gg0.

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #66 on: September 19, 2014, 11:29:20 PM »
I'm making some more tests, checking if location of mapped folder makes any difference. Just noticed, that times can go as low as 10ms or 20ms, and then for example 80,000 for several times, and then again 10. Will let you know if I reach any conclusions.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #67 on: September 19, 2014, 11:34:43 PM »
thanks that will help a lot.
dont have any tools that could play dng :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Canon eos m

  • Hero Member
  • *****
  • Posts: 562
  • 5D Mark III, Canon EOS M
Re: MLVFS windows client
« Reply #68 on: September 20, 2014, 05:08:04 PM »


getting this error. what can I do

Edit: don't know how to attach pic so directly typing the error:

SW: Adobe After Effect 2014
Error: After Effects error: File exists but couldn't open for reading "\\[email protected]\M12-1507_000022.jpg". (3::0)
File format: .jpg
Error after pressing import in Ae file menu
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

Canon eos m

  • Hero Member
  • *****
  • Posts: 562
  • 5D Mark III, Canon EOS M
Re: MLVFS windows client
« Reply #69 on: September 20, 2014, 05:19:10 PM »
Error 2: File 'H\5D3\20140912\M12-1558.m00' has an a invalid block at offset 0xB192BF34

Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #70 on: September 21, 2014, 02:51:42 AM »
ok will check this tomorrow.
meanwhile i've added raw FITS support (e.g. DeepSkyStacker reads such files)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #71 on: September 23, 2014, 01:30:15 AM »
major bugfixes:
 - reworked file accesses as sometimes it has thrown access exceptions on rapid accesses
 - changed buffering to butter at most 500MiB to prevent memory exceptions
 - tweaked network socket settings to squeeze out the last milisecond ;)

@Canon eos m:
can you try again?

@all:
overall performance should be better now. can you playback now in realtime?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

LRF

  • New to the forum
  • *
  • Posts: 34
Re: MLVFS windows client
« Reply #72 on: September 23, 2014, 10:22:09 AM »
I didn't finish previous tests, so still don't have precise data, but it might be insignificant since I didn't manage to get realtime playback with previous build anyways.

With this build still no luck with realtime, but numbers seem to be lower.

Here's data from "GET" properties in Explorer:
283,0162 ms
321,0184 ms
2,0001 ms
3,0002 ms
3,0001 ms
4,0002 ms
3,0002 ms
3,0002 ms
3,0002 ms
0 ms
0 ms
0 ms
0 ms
0 ms
15,6 ms
0 ms
0 ms
0 ms

Precache set to 150.
Where there is 0 ms, I was still getting preview and properties without problem.

In Resolve it took 36 seconds for first play. But there as a lot of skipping and black frames. Second half of the clip black. Some error and .NET crash at the end, so wasn't able to collect results.

I also benchmarked the storage hard drive, and more or less it keeps manufacturer's parameters: Average Seek Time 8.9ms (10.3 in my case), Average Latency 4.17ms, Interface  SATA 3.0Gb/s, 1TB 7200 RPM, 32MB Cache.
But since it's obvious that the whole system can't cope with such amount of data, my results might be totally misleading.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3061
Re: MLVFS windows client
« Reply #73 on: September 24, 2014, 12:09:50 AM »
@Canon eos m:
can you also provide the output of a mlv_dump -v for the file causing trouble?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Canon eos m

  • Hero Member
  • *****
  • Posts: 562
  • 5D Mark III, Canon EOS M
Re: MLVFS windows client
« Reply #74 on: September 24, 2014, 08:08:42 PM »
g3gg0 really sorry for not re-testing the mlvfs win version after you worked to refine it. It is already 2:06 am in Singapore and I am still pre-occupied with office work. I should be done by tonight and promise to test the refreshed version of mlvfs first thing tomorrow morning and report back to you soon after.

Apologies again.
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)