Recent posts

#1

Raw Video / Re: I am trying to come back t...

Last post by Walter Schulz - Today at 04:30:36 PM

So the 1080p 60fps output doesn't use 3x3 pixel binning?

See for yourself:



3x3 = 3x3 pixel binning

#2

Raw Video / Re: I am trying to come back t...

Last post by a.sintes - Today at 12:47:24 PM

The upscaling in the example above is from 1080p@60fps to 4K@120fps, so yes, you can actually record 1080p@60fps (RAW/cropped) continuously with the 5D3 (depending of your CF/SD cards capabilities, of course).

#3

Raw Video / Re: I am trying to come back t...

Last post by Onion - Today at 12:25:00 AM

I was curious about this as well.
It's confusing because a lot of sources refer to upscaling and frame generation, like the video above.

Does this mean that, with Danne's build, the 5D Mark III can record Raw 1920×1080p video at 60fps indefinitely?

#4

Raw Video / Canon 50D crop_rec 3.5k center...

Last post by zcream - Yesterday at 06:33:59 AM

Hi. I'm getting back to my Canon 50D after a few years. Are there any good videos/posts about using the latest builds for crop_rec 3.5k centered preset(2784x1160) 24fps with dual ISO 100/800 or 200/1600 with 12-bit RAW ?

Thanks!

#5

Raw Video Postprocessing / Re: MLVFS - a FUSE based, "on ...

Last post by vastunghia - May 25, 2024, 11:37:50 PM

Thanks for the hints on profiling. Will check them out. I'm on MacOS so kcachegrind won't do. But anyway, will find my way.

And yeah, because here you can treat each frame / file as a separate decompression job, you can trivially parallelise the overall task.  8 cores?  Use say, 6 threads, and decode whatever frame someone requests, and the next 5.

That's *exactly* my point, thank you. And the part on the cache, as well.

This would be the most obvious -- and I also think the most effective -- way to optimize MLVFS. Like, ok, maybe the decomp. algo can be a bit fine tuned; but if you can parallelize it, who cares?

Will look into this.

#6

Raw Video Postprocessing / Re: MLVFS - a FUSE based, "on ...

Last post by names_are_hard - May 25, 2024, 07:26:36 PM

You could log, and it can help, and I do it in some cases when optimising.  It can be a quick check in the early stages especially if you have a suspicion some particular thing is slow.

Generally though, use a real profiler.  I tend to use callgrind, from Valgrind suite of tools first, because it works on anything and doesn't need recompiling.  Use kcachegrind to view results.

Here, since it's open source and easy to build, and you're probably using gcc, gprof is another obvious tool to try.

Both of these will give you a lot of good info, without you needing to understand the code first.  You will need to learn the tools, but they're not hard and good tutorials exist.

So it looks like this could be a bottleneck after all, at least to some extent. And in particular as long as one keeps asking for single frames in sequence, one at a time (as DVR seems to do, naturally assuming that the bottleneck is hard drive read speed, and not CPU), instead of asking for a bunch of them, and then proceed to the next bunch.

The Huffman stuff is interesting, I haven't dealt with this alg.  I bet it's still faster to decode than encode.  Also, looks like you can speed it up fine using parallel techniques, there are a lot of easy to find papers on this, as well as repos with code they claim works.  Skim reading, it does sound like it presents some problems, so you won't get 8x speedup from 8 cores, but you only need 2x or so, right?

And yeah, because here you can treat each frame / file as a separate decompression job, you can trivially parallelise the overall task.  8 cores?  Use say, 6 threads, and decode whatever frame someone requests, and the next 5.

And / or, assign some amount of cache (1/4 of available ram or whatever), and decode frames you think will be useful until cache is full, then serve from cache.  Don't wait for something to request the file first - fill the cache speculatively.  As you say, it should be quite predictable.  If the whole MLV file fits in cache (a common case?), MLVFS can just decode it all as fast as possible when you mount the file, then all work is done, and keep using the cache.

How to make MLVFS do that?  Well, I dunno

#7

Raw Video Postprocessing / Re: MLVFS - a FUSE based, "on ...

Last post by vastunghia - May 25, 2024, 06:53:45 PM

But then you have to do the real step: profiling.

100% agree. That's exactly what I meant when I wrote

Could be helpful to make MLVFS log single frame read requests with their corresponding timestamp, to try and understand the time sequence with which DVR asks for frames. And check if frame requests are repeated during the third playback, when all seems to be cached already. May give it a try as soon as I have 10 minutes.

Not an expert here, not sure logging into a txt file with milliseconds timestamps is the way to go for profiling. But that's what I'm currently doing, adding a log entry every time a frame read request is acknowledged from MLVFS and then again when it is successfully completed.

Decompression is normally much faster than compression and parallelises well.  I would expect this not to be a bottleneck, and if it is, I'd expect it to be easy to fix.  Easy to measure with a profiler.

Mmh not so sure: according to FastCinemaDNG developers,

Performance of Lossless JPEG decoder is mostly limited by Huffman decoding. Actually, we need to read bitstream (bit after bit), to recover Huffman codes and data bits. Huffman decompression is essentially serial algorithm, so one can't implement it on GPU, that's why everything is done on CPU.

So it looks like this could be a bottleneck after all, at least to some extent. And in particular as long as one keeps asking for single frames in sequence, one at a time (as DVR seems to do, naturally assuming that the bottleneck is hard drive read speed, and not CPU), instead of asking for a bunch of them, and then proceed to the next bunch.

Anyway. Will dig into profiling. Which is simple in Python.. but I never did it in C.

#8

Raw Video Postprocessing / Re: MLVFS - a FUSE based, "on ...

Last post by names_are_hard - May 25, 2024, 05:03:23 PM

When it comes to optimising, a bit of speculation is good, try and think of the overall shape of the problem.  But then you have to do the real step: profiling.  Should be fairly easy to profile this code, it's open source.  Intuition is often very bad for guessing what performance is really doing.

Decompression is normally much faster than compression and parallelises well.  I would expect this not to be a bottleneck, and if it is, I'd expect it to be easy to fix.  Easy to measure with a profiler.

#9

Raw Video Postprocessing / Re: MLVFS - a FUSE based, "on ...

Last post by vastunghia - May 25, 2024, 04:14:27 PM

Can't say whether DVR is fast or not, I can only say that, after upgrading my iMac GPU with a powerful eGPU, it can now playback no problem in real-time hard DNG sequences like 3.5K @14-bit. Provided that MLV -> DNG conversion was performed before (i.e. not on the fly via MLVFS).

Yeah, planning to study how DVR is accessing DNG sequences as a first step towards understanding whether we can tweak somehow the interface between DVR and MLVFS. Once again, caching could be a key factor.

#10

Raw Video Postprocessing / Re: MLVFS - a FUSE based, "on ...

Last post by Danne - May 25, 2024, 04:01:43 PM

Probably.
However. Dvr is not even close when previewing dng sequences when I comes to speed. At least last I messed with this program.

"time, instead of leveraging on the parallelization potential of MLVFS."

How about looking into this  ?