Magic Lantern Forum

Using Magic Lantern => Raw Video => Topic started by: a1ex on June 20, 2013, 05:08:30 PM

Title: New writing strategy - variable buffering
Post by: a1ex on June 20, 2013, 05:08:30 PM
If you ever looked in the comments from raw_rec.c (https://bitbucket.org/hudson/magic-lantern/src/tip/modules/raw_rec/raw_rec.c), you have noticed that I've stated a little goal: 1920x1080 on 1000x cards (of course, 5D3 at 24p). Goal achieved and exceeded - even got reports of 1920x1280 continuous.

During the last few days I took a closer look at the buffering strategy. While it was near-optimal for continuous recording (large contiguous chunks = faster write speed), there was (and still is) room for improvement for those cases when you want to push the recording past the sustained write speed, and squeeze as many frames as possible.

So, I've designed a new buffering strategy (https://bitbucket.org/hudson/magic-lantern/commits/90c50ab005b7) (I'll call it variable buffering), with the following ideas in mind:

* Write speed varies with buffer size, like this (http://www.magiclantern.fm/forum/index.php?topic=5471) (thanks to testers who ran the benchmarks for hours on their cameras)

* Noticing the speed drop is small, it's almost always better to start writing as soon as we have one frame captured. Therefore, the new strategy aims for 100% duty cycle of the card writing task.

* Because large buffers are faster than small ones, these are preferred. If the card is fast enough, only the largest buffers will be touched, and therefore the method is still optimal for continuous recording. Even better - adding a bunch of small buffers will not slow it down at all.

* This algorithm will use every single memory buffer that can contain at least one frame (because small buffers are no longer slowing it down).

* Another cause of stopping: when the buffer is about to overflow, it's usually because the camera is trying to save a huge buffer (say a 32MB one), which will take a long time (say 1.5 seconds on slow SD cameras, 21MB/s). So, I've added a heuristic that limits buffer size - so, in this case, if we predict the buffer will overflow after only 1 second, we'll save only 20MB out of 32, which will finish at 0.95 seconds. At that moment, the capturing task will have a 20MB free chunk that can be used for capturing more frames.

* Buffering is now done at frame level, not at chunk level. This finer granularity allows me to split buffers on the fly, in whatever configuration I believe it's best for the situation.

* The algorithm is designed to adjust itself on the fly; for this, it does some predictions, such as when the buffer is likely to overflow. If it predicts well, it will squeeze a few more frames. If not... not :P

* More juicy details in the comments (https://bitbucket.org/hudson/magic-lantern/src/tip/modules/raw_rec/raw_rec.c).


This is experimental. I've ran a few tests, played back a few videos on the camera, but that was all. I didn't even check whether the frames are saved in the correct order.

Build notes

- This breaks bolt_rec. Buffering is now done at frame level, not at chunk level, so bolt_rec has to be adjusted.
- The current source code has debug mode enabled - it prints funky graphs. You'll find them on the card.
- The debug code will slow down the write speed.
- I'd like you to run some test recordings and paste the graphs - this will allow me to check if there's any difference between theory and practice (you know, in theory there isn't any).
- I did not run any comparison with the older method on the camera (I did only in simulation). Would be very nice if you can do this.
- It may achieve lower write speeds. This is normal, because it also uses smaller buffers. If you also consider the idle time, it should be better overall.
- For normal usage, disable the debug code (look at the top of raw_rec.c).

History

[2013-05-17] Experiment (http://www.magiclantern.fm/forum/index.php?topic=5471) about checking the optimal buffer sizes. People ran the benchmarks for hours on their cameras and posted a bunch of logs. They pretty much confirmed my previous theory, that any buffer size between 16MB and 32MB should result in highest speeds.

[2013-05-30] Noticed that file writes aligned at 512 bytes are a little faster (credits: CHDK (http://chdk.setepontos.com/index.php?topic=9970)). Rounded image size to  multiples of 64x32 or 128x16 (https://bitbucket.org/hudson/magic-lantern/commits/647d93aa34924eb936e89d58356c02ea217fe3b2) to ensure 512-byte rounding.

[2013-08-06] Figured out that I could just add some padding (https://bitbucket.org/hudson/magic-lantern/commits/1479df3a2322068663433a000d52749d6a93659a) to each frame to ensure 512-byte rounding and keep the high write speeds without breaking the converters too hard. Also aligned everything at 4096 bytes (https://bitbucket.org/hudson/magic-lantern/commits/150a9c5810834e60da6311282d61aabc267ad9a7), which solved some mysterious lockups from EDMAC and brought back the highest speed in benchmarks (over 700MB/s).

[2013-05-30] speedsim.py (https://bitbucket.org/hudson/magic-lantern/commits/42196e77c62b2a27e78a60b2aa6c70bfa5f8347a) - First attempt to get a mathematical model of the recording process. Input: resolution, fps and available buffers. Output: how many frames you will get, with detailed graphs. Also in-camera estimation (https://bitbucket.org/hudson/magic-lantern/commits/fce6d462d2bb9940d28a5de9d56a395ff542752f) of how many frames you will get with current settings.

[2013-06-18] Took a closer look (https://bitbucket.org/hudson/magic-lantern/commits/82900279ffdfe4fbb435eb89a9a45666124b6675) at these logs (http://www.magiclantern.fm/forum/index.php?topic=5471) and fitted a mathematical model for the speed drop at small buffer sizes.

[2013-06-18] Does buffer ordering/splitting matter? 1% experimented with it before, but there was no clear conclusion.
* is it better to take the highest one first or the smallest one first? there's no clear answer, each one is best for some cases and suboptimal for others.
* since some cameras had very few memory chunks (e.g. 550D: 32+32+8 MB), what if each of the 32MB buffer is divided in 2x16 or 4x8 MB? This brought a significant improvement for resolutions just above the continuous recording threshold, but lowered performance for continuous recording.
* optimization: updated speedsim.py (https://bitbucket.org/hudson/magic-lantern/commits/0db037b3b960f79dc259cdabfbf144f6eb164270) so it finds the best memory configuration for one particular situation. Xaint confirmed (http://www.magiclantern.fm/forum/index.php?topic=5582.msg52770#msg52770) the optimization results on 550D.
* problem: there was no one-size-fits-all solution.

[2013-06-19] Simulation now matches perfectly the real-world results (https://bitbucket.org/hudson/magic-lantern/commits/7eee493f6fc992ea87963b0ac0fd5992e6119582). So, the mathematical model is accurate!

[2013-06-19] Started to sketch the variable buffering algorithm and already got some simulation results (http://www.magiclantern.fm/forum/index.php?topic=5582.msg52817#msg52817). There was a clear improvement for borderline cases (settings that require just a little more write speed that your camera+card can deliver).

Example: 550D, 1280x426, 23.976fps, 21.16MB/s, simulation:

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2F8%252B2x32MB.png&hash=1bf8e41e6a2a9d765d934c8e030b7d37) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2F9x8MB.png&hash=693623282230cee037a63fc26cbb6a67) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fvariable-buffering.png&hash=04995f018cce32c13f963765490ecd9d)

- 8MB + 2x32MB (current method) - 317 frames
- 9x8MB - 1566 frames
- Variable buffering, starting from 8MB + 2x32MB - 1910 frames

[2013-06-20] Ran a few more tests and noticed that it meets or exceeds the performance of the old algorithm with sort/split optimization. There are still some cases where the sort/split method gives 2-3 more frames (no big deal).

Got rid of some spikes, which squeezed a few more frames (1925 or something).

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fvariable-buffering-1.png&hash=027fd16e88955c492b96c113ed86ea9c)

Implemented the algorithm in camera. It's a bit simplified, I didn't include all the optimizations from the simulated version, but at least it sems to work.

Took me two hours just to write this post. Whooo :D


Enjoy and let me know if the theory actually works!


Fixed broken link - Audionut
Title: Re: New writing strategy - variable buffering
Post by: Rewind on June 20, 2013, 06:30:39 PM
Yes, it works.
I get about 45-55% increase in number of frames before skipping compared to 1x8Mb + 4x16Mb buffering method.
(550d Sandisk extreme pro 95 mb/s).
Some actual test results (http://www.magiclantern.fm/forum/index.php?topic=5582.msg53189#msg53189)
Title: Re: New writing strategy - variable buffering
Post by: haemma on June 20, 2013, 06:31:54 PM
Very nice. Hopefully my Lexar 1000x 64gb Card arrives tomorrow so maybe I could test some few things. (on 5D3)

Thanks for your amazing work!
Title: Re: New writing strategy - variable buffering
Post by: Stedda on June 20, 2013, 06:32:45 PM
You sir are a wizard... can't wait to try it out.  ;D
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 20, 2013, 06:34:27 PM
@Rewind: did you find any screenshots on your card? can you upload them?

They are saved when recording stops (if the build was compiled with debug flags on, which is default right now).
Title: Re: New writing strategy - variable buffering
Post by: Rewind on June 20, 2013, 06:46:41 PM
Screenshots for this test (http://www.magiclantern.fm/forum/index.php?topic=5582.msg53189#msg53189)
[spoiler](https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fmoredo.ru%2Fraw%2FVRAM3.BMP&hash=a278dfb3add6c949b61f458f4dfd2bc9)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fmoredo.ru%2Fraw%2FVRAM4.BMP&hash=8a097dbe8afc1ef421eb19758a45ce2a)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fmoredo.ru%2Fraw%2FVRAM5.BMP&hash=fcc4d87b7913bf79f18533b685880b7a)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fmoredo.ru%2Fraw%2FVRAM6.BMP&hash=4f84d036d1dea0afe59e2bcb41c58684)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fmoredo.ru%2Fraw%2FVRAM7.BMP&hash=e0bb72430049a2cfa13a5b3d6f72a78b)[/spoiler]
Title: Re: New writing strategy - variable buffering
Post by: driftwood on June 20, 2013, 06:47:28 PM
@a1ex I want to test this on the 5DMKIII immediately - have you got a build link?
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 20, 2013, 06:51:56 PM
Straight from the card, not packaged:

raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/raw_rec.mo) - portable, should work on any camera

autoexec.bin (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/autoexec.bin) and 5D3_113.sym (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/5D3_113.sym) - only for 5D3 1.1.3 (optional)
Title: Re: New writing strategy - variable buffering
Post by: driftwood on June 20, 2013, 06:57:10 PM
Thanks. Testing now...
Title: Re: New writing strategy - variable buffering
Post by: Stedda on June 20, 2013, 07:18:00 PM
I'm not finding any screenshots... maybe it's me...
Title: Re: New writing strategy - variable buffering
Post by: Bioskop.Inc on June 20, 2013, 07:27:47 PM
Here's some from 60D @ 16:9:

https://dl.dropboxusercontent.com/u/90827273/60D.zip

This was the first build before you disabled Debug Code
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 20, 2013, 07:32:05 PM
So far, all the screenshots looked good, so we can disable the debug info and let it run at full speed.
Title: Re: New writing strategy - variable buffering
Post by: a.d. on June 20, 2013, 07:33:08 PM
@Stedda
You need to edit the code
from
Code: [Select]
#undef DEBUG_BUFFERING_GRAPH to
Code: [Select]
#define DEBUG_BUFFERING_GRAPH
Title: Re: New writing strategy - variable buffering
Post by: Stedda on June 20, 2013, 07:34:39 PM
ahhh... i thought I read it was by default. My bad. Thanks.
Title: Re: New writing strategy - variable buffering
Post by: Danne on June 20, 2013, 07:39:39 PM
Couldn,t find any screenshots but managed to record 1920x1152 25fps continuosly on my transcend 1000x 64gb card on my 5d mark 3. Before it worked on 1920x1080 24fps, occasionally I could get it to work with 25fps in 1920x1080 25fps depending on the mood of my camera ;).

This means 1 step up in aspect ratio fully working with 25fps. NICE!

Tried also some 50fps and got it working continuosly in 1856x492.

Awesome work Alex!
//D
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 20, 2013, 07:46:17 PM
I didn't expect any improvement in continuous recording TBH.

But if you were getting around 1000 frames before, now you can expect to get around 10000 (which is almost continuous).
Title: Re: New writing strategy - variable buffering
Post by: driftwood on June 20, 2013, 07:53:21 PM
Definite improvement a1ex, getting continuous at 24p/25p 1920x1152 now on the lexar 128Gb 1000x. Global Draw On, ETTR On

Quick tests:

24p - 1080 - 92.8MB/s - 10% idle continuous
24p - 1152 - 93MB/s - 5% idle continuous
24p - 1280 - 95.7MB/s - 1-7ms idle > 1000 frames stopped
30p - 1080  - 91.8MB/s - 1ms idle stopped at 502 frames
30p - 1.85:1 - 93.8MB/s - 1ms idle continuous
25p - 1080 - 91.9MB/s - 5% idle continuous
25p - 1152 - 92.2MB/s - 49ms climbing to 80ms idle continuous

Incred1ble work a1ex. :-)

Also getting around 25% improvement on slower 800x cards.

DNG files look good.
Title: Re: New writing strategy - variable buffering
Post by: NateVolk on June 20, 2013, 07:59:07 PM
I'm seeing an improvement as well.  5d3 Komputerbay 128gb card.  Was 1920 @24 for 150 frames before skipping.  Now 518 frames before skipping...  Oh, and global draw on, memory hack on, force left...
I'm still new, hope this helps!
Title: Re: New writing strategy - variable buffering
Post by: Stedda on June 20, 2013, 08:05:22 PM
I am seeing the same as Danne Reported.

I was able to record 1920x1080 continuous previously. Now I can record 1920x1152 to what seems to be continuous. I didn't let it run until it filled the card so I'll have to test more but I look like I may even be able to go to 1920x1280 without a problem based off the on screen graph.

This is on a Transcend 1000x 32GB GD on with ETTR only NO Memory Hack.
Nice !
Title: Re: New writing strategy - variable buffering
Post by: Yuppa on June 20, 2013, 08:10:55 PM
Model: 60D
Card: Sandisk Extreme 45 MB/s 16 GB
Global draw off.

1280 x 544 (2.35:1).

Before: 707 frames.
After: 861 frames.

Got an ERR 70 the 1st time, but not the 2nd, 3rd...

The buffer(s) indicator is cool, better than the stars by far.

Edit: at 1152 x 490 (2.35:1), before 1802 frames, after over 3,000 frames!

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.toxicsuds.com%2Fmagic_lantern%2FVRAM2.jpg&hash=f0a7fd16bd8427af22dcb8db163427aa)
Title: Re: New writing strategy - variable buffering
Post by: swinxx on June 20, 2013, 08:52:58 PM
that looks really promising. great alex
Title: Re: New writing strategy - variable buffering
Post by: KMA_WWC on June 20, 2013, 09:03:57 PM
As soon as I get back home Alex, I will run the test on all of my cards, and let you know the results and screenshots.
Title: Re: New writing strategy - variable buffering
Post by: Aborgh on June 20, 2013, 09:11:38 PM
Does it work with 650D?
Title: Re: New writing strategy - variable buffering
Post by: kgv5 on June 20, 2013, 09:55:57 PM
Could we test it on 6D?
Title: Re: New writing strategy - variable buffering
Post by: crash-film on June 20, 2013, 10:04:16 PM
thank you a1ex for that wonderful work!

has anybody found a reason why there´s always that mysterious 2 times so called  warm up recordings?
it can´t really be a thermal warm up.... then we would be way out of regular technical specifications.

what´s your strategy in optimizing the (probably) card specific writing strategy?
firmware check on the cards (brand, size, manufacturing date)?

or a new benchmarking tool in ML to determine the best settings?
is there a way to format the card from inside ML with optimized cluster settings and/or aligning?

i think the problem here will be the cards and their constantly changing inner structure. let there be a new controler IN the card and the writing strategy might be a nightmare.... a komputerbay x1000 might work for one user and for the next it might be completely "useless"....because everybody is looking for the highest resolutions possible. sick pixel x pixel marketing....

or am i wrong and everything you do is card independent?
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 20, 2013, 10:05:49 PM
Straight from the card, not packaged:

autoexec.bin (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/autoexec.bin), raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/raw_rec.mo) and 5D3_113.sym (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/5D3_113.sym)
I replaced the files on my sd with these 3 files and the camera wont boot.
Should I be doing something additional?
Title: Re: New writing strategy - variable buffering
Post by: dariSSight#1 on June 20, 2013, 10:15:49 PM
Correct me if I'm mistaking but it's the 5D Mark II & III the top DSLR from Canon and isn't Magic Lantern pulling the most out of these 2 DSLR? I'm just asking because I'm extremely grateful by the work you guys do and I'm hoping you don't stop pushing the 5D Mark II, because I read a lot on these forum about the 6D and other Canon cameras. I just want you guys to give me a straighten out, I would love to hear that 5D Mark II can get variable buffering before I start reading 60D is been tested. I know I know I know, I'm selfish. I'm working on it ;)
Title: Re: New writing strategy - variable buffering
Post by: g3gg0 on June 20, 2013, 10:16:06 PM
with my buggy CF card it gets now >1k instead of ~100 frames when recording in 1080
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 20, 2013, 10:22:03 PM
Cool :D

The strategy is universal, it's trying to use the available resources in the best possible way. It measures the card speed on the fly.

You can use the module from my previous post on any camera (the code is portable). The autoexec and sym are only for 5D3.
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 20, 2013, 10:27:39 PM
I figured it out.
SHIT Alex, this is amazing!!!


Recording 2560x1090 on a Sandisk 600x CF 16GB card

with the june 16th build
72 frames with memory hack off, and preview set to auto
120 frames with memory hack on, and preview set to 'HaCKed'



With the new 3 files you posted, i;m getting
142 frames with memory hack off, and preview set to auto
258 frames with memory hack on, and preview set to 'HaCKed'

as you can see, it doubled!

With the CF that I have, I've always been able to get continuous 1920x1080, so I thought I'd go with something a little more demanding.

BRAVO!!!
Title: Re: New writing strategy - variable buffering
Post by: dubarry on June 20, 2013, 10:38:14 PM
WOW!

60D 304 frames  1280x720  16x9   (previously 175 frames)

1728x692  185 frames 2.50

Very nice.
Title: Re: New writing strategy - variable buffering
Post by: kgv5 on June 20, 2013, 10:49:56 PM
I figured it out.

So could you please tell us how did you do that?
I replaced the old raw_rec.mo with the new one from the post #7 in my 6D but no difference. :-\
Title: Re: New writing strategy - variable buffering
Post by: Viente on June 20, 2013, 11:04:16 PM
Awesome job! Got 200 frames in 1920x on 50D in crop mode, was about 70 before.
Title: Re: New writing strategy - variable buffering
Post by: dariSSight#1 on June 20, 2013, 11:23:46 PM
Cool :D

The strategy is universal, it's trying to use the available resources in the best possible way. It measures the card speed on the fly.

You can use the module from my previous post on any camera (the code is portable). The autoexec and sym are only for 5D3.


Cool, I will install and get back to you. By the way does the AutoExec Bin activate the module?
Title: Re: New writing strategy - variable buffering
Post by: Mido on June 20, 2013, 11:24:37 PM
Very nice.
5D3, KomputerBay 128GB 1000x
1920x1080 @25
Before 300 frames
Now 500 frames
Thank you!
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 20, 2013, 11:28:13 PM
And I've just got a proof that this strategy is far from optimal :D

(read: some more improvements may be coming soon)
Title: Re: New writing strategy - variable buffering
Post by: Jakobmen on June 20, 2013, 11:58:39 PM
the Force is strong with you A1EX!  :o
Title: Re: New writing strategy - variable buffering
Post by: dariSSight#1 on June 21, 2013, 12:01:18 AM
A1ex are we losing some resolution with variable buffering because the highest resolution with 2.20.1 was 1880x854 but now it's 1872x850 and its not continuous on a Lexar 1000x, but 1856x844 is continuous.
Title: Re: New writing strategy - variable buffering
Post by: noisyboy on June 21, 2013, 12:13:49 AM
Damn A1ex! Congrats  8)
Title: Re: New writing strategy - variable buffering
Post by: Rush on June 21, 2013, 12:18:34 AM
Canon 600D 24p variable buffer improvements vs old method
(before max seconds -> now max seconds).

W/o line skipping:
1280x720 - 3 > 6 (crop-factor = 2.16)
1280x536 - 6 > 14 (crop-factor = 2.16)

16:9 line skipping (unsqueezed resolution in brackets)
1280x434(720) - 29 > 42 (crop-factor = 2.16)
1344x456(756) - 11 > 27 (crop-factor = 2.06)
1472x500(828) - 5 > 11 (crop-factor = 1.88)
1600x542(900) - 3 > 7 (crop-factor = 1.72)
1728x586(972) - 2 > 5 (crop-factor = 1.6)

2.39:1 line skipping (unsqueezed resolution in brackets)
1472x370(616) - 50 > very long (crop-factor = 1.88)
1600x404(670) - 8 > 19 (crop-factor = 1.72)
1728x436(724) - 5 > 10 (crop-factor = 1.6)

Thank you, a1ex!
Title: Re: New writing strategy - variable buffering
Post by: 1% on June 21, 2013, 01:20:00 AM
Promising... I wonder what it will do on 600D w/ dialog timers disabled.
Title: Re: New writing strategy - variable buffering
Post by: lourenco on June 21, 2013, 01:58:15 AM
2560 x 1072 23.976 fps over 700 frames before frame skipping.  104 MB/s.
Title: Re: New writing strategy - variable buffering
Post by: squig on June 21, 2013, 02:15:08 AM
1920x448 (720) 60p 309 frames pre variable buffer, 436 frames with variable buffer. That's about 18 seconds of 2.67:1 slo-mo, very usable. 1968 frames hacked. The pre variable build needed 4 or 5 recordings to warm up to the 309 frames.

2560x960 24p continuous in crop mode.  ;D

1920 x 508 (818) 48p continuous.

Lexar 1000x 64Gb GD off.

1920x448 60p 568 frames with the Toshiba 1066x 64Gb.
Title: Re: New writing strategy - variable buffering
Post by: Edgar Matos on June 21, 2013, 03:06:45 AM
Promising... I wonder what it will do on 600D w/ dialog timers disabled.

Magic! But I'm still fixing the wetting problem. I think a bib can do wonders.
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 21, 2013, 03:31:46 AM
Where Do you enable this? Do I need to format the card to have certain buffer size for this to work optimally?
Title: Re: New writing strategy - variable buffering
Post by: ilguercio on June 21, 2013, 03:52:09 AM
1280*720 on my 60D went from 335 to 379-381 frames before skipping at 23.976 fps.
Title: Re: New writing strategy - variable buffering
Post by: squig on June 21, 2013, 03:52:24 AM
You don't enable it, it's built into the raw_rec code.
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 21, 2013, 03:54:54 AM
You don't enable it, it's built into the raw_rec code.


But what about setting the buffer size on the card?
Title: Re: New writing strategy - variable buffering
Post by: ilguercio on June 21, 2013, 03:59:54 AM


But what about setting the buffer size on the card?
And what would this mean?
A card is a card, a buffer is a buffer.
The buffer is in the camera, the card can just do whatever possible to sustain the data rate if it's capable of doing it.
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 21, 2013, 04:00:48 AM
And what would this mean?
A card is a card, a buffer is a buffer.

I meant cluster size  :-\
Title: Re: New writing strategy - variable buffering
Post by: kgv5 on June 21, 2013, 04:12:59 AM
Great!!! Got it. Thank you 1%   :D

[6D] The hacks - 1792x752 before 280 farmes 11-12 sec, now 373 frames 16 sec.  I cannot believe this  :D
We went up two aspect ratios: from 2,39:1 to 2,20:1 - 1792x816 now records 282 frames  :D

additionally
1792x976 1,85:1 - now gives 153 frames
1792x896 2:1 - 190 frames

Everything Hack OFF, global draw off, sound  beep.
Title: Re: New writing strategy - variable buffering
Post by: 1% on June 21, 2013, 04:23:48 AM
With don't click me?
Title: Re: New writing strategy - variable buffering
Post by: crazyrunner33 on June 21, 2013, 04:28:05 AM
Great job, I was able to hit 2560x1024 for over 1000 frames and 2880x1154 for around 130 frames.  After a little bit I started getting the "nothing saved, card maybe full" error.
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 21, 2013, 04:43:38 AM
Great job, I was able to hit 2560x1024 for over 1000 frames and 2880x1154 for around 130 frames.  After a little bit I started getting the "nothing saved, card maybe full" error.
I got that error as well. but I've also gotten it before this newest hack
Title: Re: New writing strategy - variable buffering
Post by: budafilms on June 21, 2013, 04:47:19 AM
I read de explanation from A1ex, but my english is not so good.

I understand that when the buffer is going to colapsed, the size of the frame is reduced.
If this is like this, means that in some frames the quality is low? So in the image we can perceibe this automatically reduce of quality?

Any explanation (or in Italian, spanish) or english for dummies?

thanks
Title: Re: New writing strategy - variable buffering
Post by: 1% on June 21, 2013, 04:48:33 AM
Cluster size you pick when you format the card.
Title: Re: New writing strategy - variable buffering
Post by: kaoshotbeatz on June 21, 2013, 05:05:52 AM
@alex i intslled your 3 files and im using a promasters 1000x card 64 gigs on 5d3 still cant get continuous 1920 x 1080 only 320 frames  and 410 frames with hacked preview. but 1920x960 got 827 frames about 30 % more then before . With hacked preview I got 1725 frames, @  1920 x 872 is continuous  And I  was able to get about 120 frames @ 1920 x 1280 this is really good i still havent been able to convert files bigger then 4 gigs with raw magic. maybe my compiled version isnt right maybe you can email me one @kaoshotbeatz@gmail.com
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 21, 2013, 05:44:21 AM
Cluster size you pick when you format the card.

Yeah I know but what should it be? I think I read something on this thread about having to set your cluster size to a certain optimal value.

Holy Shit - I got 600 frames on 1792 x 672. Originally I had 450. Thats a 25 percent increase!
Title: Re: New writing strategy - variable buffering
Post by: ted ramasola on June 21, 2013, 07:58:20 AM
I just ran a comparison test with this and jun 19 build by a.d.

the higher resolutions have changed to slightly smaller so I see slight improvements.

but strangely at 1728 x 972

A1ex variable buffering--- stops at  approx. 1,738 or 1,625 frames.

Jun 19 build----------------- records until card almost full. 10,000 = frames.

Is there a setting i should use when running this variable buffer build to get optimum performance?
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 21, 2013, 08:04:33 AM
I just ran a comparison test with this and jun 19 build by a.d.

the higher resolutions have changed to slightly smaller so I see slight improvements.

but strangely at 1728 x 972

A1ex variable buffering--- stops at  approx. 1,738 or 1,625 frames.

Jun 19 build----------------- records until card almost full. 10,000 = frames.

Is there a setting i should use when running this variable buffer build to get optimum performance?

Maybe you didn't enable the memory hack... Global draw should be off. Make sure fps over ride is set to on (24fps), sometimes my camera changed it to 30. All I can think of right now, maybe someone else can find out what happened.
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 21, 2013, 08:09:11 AM
@ted: the builds from a.d. contain a number of extra changes, e.g. allows non-mod16 resolutions (which I prefer to stay away from, for DMA alignment reasons), or auto-disabling global draw (I think this should be up to the user, not automatic).

This one is vanilla, there are no changes that I didn't review.
Title: Re: New writing strategy - variable buffering
Post by: ted ramasola on June 21, 2013, 08:21:59 AM
@ted: the builds from a.d. contain a number of extra changes, e.g. allows non-mod16 resolutions (which I prefer to stay away from, for DMA alignment reasons), or auto-disabling global draw (I think this should be up to the user, not automatic).

This one is vanilla, there are no changes that I didn't review.

Ok. Thank you.

In that case when I run the rawrecmod that you posted, There are no special configurations or settings i should do aside from the usual? FPS over ride to=23.976
preview=hacked
global draw =OFF
??

I see no one is reporting results from the 5D2, is there a test I can do to help you in finding bug?
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 21, 2013, 08:43:08 AM
Well... hacked preview has little or no effect on 5D2 afaik. I'd say try normal settings, global draw off, and compare with what you already have. Just make sure you have the same FPS and resolution for both.
Title: Re: New writing strategy - variable buffering
Post by: mustardseed on June 21, 2013, 09:30:56 AM
I tried this out on my 5D2 this afternoon. Previously the highest continuous 16:9 video I could record was 1600x900, but with this new module I can get continuous 1724x970 on my SanDisk 90MBs. Upscales to 1080p beautifully.
Title: Re: New writing strategy - variable buffering
Post by: kgv5 on June 21, 2013, 09:32:09 AM
With don't click me?

No, without.
Title: Re: New writing strategy - variable buffering
Post by: budafilms on June 21, 2013, 10:07:33 AM
2560 x 1072 23.976 fps over 700 frames before frame skipping.  104 MB/s.

Near to 2K! A real Cinema camcorder!
Title: Re: New writing strategy - variable buffering
Post by: hammermina on June 21, 2013, 12:32:37 PM
what the hell alex ;D

i got countinius shooting at 1280*388 in 720p mode at 23,97fps
and at 1280*544 in 1080p 191 frames on my 550d

massive!!!
Title: Re: New writing strategy - variable buffering
Post by: Jackie on June 21, 2013, 12:36:45 PM
On my 5D3 1920X818 with SanDisk Extreme 500 frames before and now is it 700+ frames so it work realy great :)
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 21, 2013, 01:15:47 PM
Just added a small startup optimization (speculative start): https://bitbucket.org/hudson/magic-lantern/commits/50209fff8bbb

Before, it was writing to the card as soon as there was one captured frame in the buffer. Even if this meant it was writing a little slower.

Now, it starts writing as soon as there is one frame captured, but the write command is issued in advance, for the entire contiguous memory area. Of course, subsequent frames will be captured while writing. As long as the file writing task is slower than frame capturing task, this trick will work.

(Don't confuse this with pre-buffering.)

Obviously, this technique is inherently unsafe. If, for some reason, the card controller starts writing 120% faster than we have measured, the risk is that we may save a frame that does not contain complete image data.

In theory, I can detect this situation, and I've added quite a few consistency checks. I did not encounter any lost frames yet.

Don't expect miracles from it. On my cards, I did not get any noticeable improvement, other than prettier buffer graphs (with large steps from the very beginning) and the mathematical proof that both startup and cruise "segments" are saved at the maximum speed possible with standard file I/O.

I expect it to work a little better on Ted's card, where yesterday's method didn't work very well.

And of course, since the optimization is a little unsafe, it's turned off by default.

Binary (same link): raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/raw_rec.mo)
Title: Re: New writing strategy - variable buffering
Post by: N/A on June 21, 2013, 02:02:48 PM
600D, 1280x512 (640 mode), GD off, 410 frames.

:D
Title: Re: New writing strategy - variable buffering
Post by: Redrocks on June 21, 2013, 02:10:51 PM
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
comparing A1ex's new build to the normal build I use (thanks a.d.)


A1ex: 1872 x 1250 @ 1:2

200
208
201
190
205

5x 2144 x 1078 @ 1:2

201
201
206
201
208

a.d.: 1880 x 1250 @1:2

174
174
174
174
174

5x 2152 x 1078 @1:2

185
184
185
184
185
__________________

A1ex: 1872 x 936 @2:1 (first unreliable ratio)

2077
2016
2429
2678
2171

5 x 2144 x 1072 @2:1 (first unreliable ratio)

211
209
207
207
211

a.d.: 1880 x 940 @2:1 (first unreliable ratio)

2061
2432
2289
2660
Battery gave out during final clip

5x 2152 x 1076 @2:1 (first unreliable ratio)

59
126
177
185
184
___________________

A1ex: 1872 x 1012 @1.85:1

674
738
676
725
679

5x 2144 x 1078 @1.85:1

207
200
208
201
208

a.d.: 1880 x 1016 @1.85:1

541
661
551
532
617

5x 2152 x 1078 @1.85:1

185
184
185
184
185
___________________

A1ex: 1872 x 1054 @16:9

475
488
466
470
468

5x 2144 x 1078 @16:9

204
205
206
203
208

a.d.: 1880 x 1058 @16:9

408
417
408
417
416

5x 2152 x 1078 @16:9

185
184
185
184
185
Title: Re: New writing strategy - variable buffering
Post by: dariSSight#1 on June 21, 2013, 02:35:49 PM
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
with A1ex's module from first page

1870 x 1250

200
208
201
190
205

5x 2144 x 1078

201
201
206
201
208

with current a.d. module on another Komputerbay

1880 x 1250

174
174
174
174
174

5x 2152 x 1078

185
184
185
184
185


Redrock are you checking for full ability with the 5D Mark II and we're getting at 1x with new module average 200 at 1870x1250, 5x 2144x1078? I will check my Lexar limitation and post back

Title: Re: New writing strategy - variable buffering
Post by: Redrocks on June 21, 2013, 03:48:23 PM
Yes dariSSight, I aimed my first test at the maximum resolution and aspect ratio. I've since then updated the post with other aspect ratio tests.
Title: Re: New writing strategy - variable buffering
Post by: dariSSight#1 on June 21, 2013, 04:46:22 PM
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
comparing A1ex's new build to the normal build I use (thanks a.d.)


A1ex: 1872 x 1250 @ 1:2

200
208
201
190
205

5x 2144 x 1078 @ 1:2

201
201
206
201
208

a.d.: 1880 x 1250 @1:2

174
174
174
174
174

5x 2152 x 1078 @1:2

185
184
185
184
185
__________________

A1ex: 1872 x 936 @2:1 (first unreliable ratio)

2077
2016
2429
2678
2171

5 x 2144 x 1072 @2:1 (first unreliable ratio)

211
209
207
207
211

a.d.: 1880 x 940 @2:1 (first unreliable ratio)

2061
2432
2289
2660
Battery gave out during final clip

5x 2152 x 1076 @2:1 (first unreliable ratio)

59
126
177
185
184
___________________

A1ex: 1872 x 1012 @1.85:1

674
738
676
725
679

5x 2144 x 1078 @1.85:1

207
200
208
201
208

a.d.: 1880 x 1016 @1.85:1

541
661
551
532
617

5x 2152 x 1078 @1.85:1

185
184
185
184
185
___________________

A1ex: 1872 x 1054 @16:9

475
488
466
470
468

5x 2144 x 1078 @16:9

204
205
206
203
208

a.d.: 1880 x 1058 @16:9

408
417
408
417
416

5x 2152 x 1078 @16:9

185
184
185
184
185


Why has the resolution change from 1880 to 1872?
Title: Re: New writing strategy - variable buffering
Post by: Redrocks on June 21, 2013, 05:02:32 PM
It's just a quirk of A1ex's build, I'm sure he will have a reason for the slight differences between his and a.d's resolutions. I actually set the resolution to 3584 for all the tests, but the numbers quoted are what the camera actually puts out.
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 21, 2013, 05:16:13 PM
If you have read the previous posts and the source code, you would have noticed the quirk is in a.d.'s build (he has an exception for 1880, which I don't like). Bigger is not always better, mod16 resolutions are more important IMO, and the EDMAC is very sensitive to alignment.
Title: Re: New writing strategy - variable buffering
Post by: Redrocks on June 21, 2013, 05:22:24 PM
Sorry for the inference, it's quirky to me in the context of me using a.d's builds is what I meant. I just realised that 1872 is an exact 1/3 rd of the actual sensor, which you just explained is more suitable to the task.
Title: Re: New writing strategy - variable buffering
Post by: Redrocks on June 21, 2013, 08:59:20 PM
I did a test on my old 30MB/s Sandisk Ultras. Bold text is resolution, 1st numbers are ratio and 2nd numbers frames:

1872x

796 182
850 155
936 131
1012 119
1054 107
1124 99
1248 85

1856 x

790 186
844 165
928 143
1004 124
1044 116
1114 104
1238 86

1728 x

736 267
786 267
864 183
934 156
972 151
1036 123
1152 107

1600 x

680 388
728 318
800 250
864 218
900 198
1200 115

1472 x

796 312
828 259
884 220
982 187
1104 155

1344 x

572 3122
610 1456
672 780
726 573
756 447
806 412
896 305
1008 226
1250 150

1280 x

544 6518
582 3526
640 1628
692 908
720 717
768 539
854 399
960 281
1250 152
Title: Re: New writing strategy - variable buffering
Post by: noisyboy on June 22, 2013, 02:06:48 AM
Interesting developments over in the 6D Dev thread (http://www.magiclantern.fm/forum/index.php?topic=3904.2275)!

teo77 has made a great discovery that formatting to exfat dramatically increases recording times. Basically I was banging my head against a wall all day, wondering why he was getting 5:05 mins @ 1600x640 and I was getting 02:15 with latest hack. Anyway - he then figured that formatting his Sandisk Extreme Pro 95MB/s to exfat (4096 clusters), combined with latest module and don't click me from 02:15 mins all the way up to SEVEN MINUTES! If anyone else can try this out and share your results it could help :)

A1ex - do you think this is a problem caused by the files being split @ 4gb?
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 22, 2013, 02:20:08 AM
Interesting developments over in the 6D Dev thread (http://www.magiclantern.fm/forum/index.php?topic=3904.2275)!

teo77 has made a great discovery that formatting to exfat dramatically increases recording times. Basically I was banging my head against a wall all day, wondering why he was getting 5:05 mins @ 1600x640 and I was getting 02:15 with latest hack. Anyway - he then figured that formatting his Sandisk Extreme Pro 95MB/s to exfat (4096 clusters), combined with latest module and don't click me from 02:15 mins all the way up to SEVEN MINUTES! If anyone else can try this out and share your results it could help :)

A1ex - do you think this is a problem caused by the files being split @ 4gb?

What does dont click me do?
He clicks that AFTER he's started recording for 2 minutes(while still recording)?
Title: Re: New writing strategy - variable buffering
Post by: teo770 on June 22, 2013, 02:55:17 AM
The "procedure" is:

- Turn cam on in PHOTO mode
- load raw_rec.mo
- in Debug menu, click "Don't click me"
- switch LV to MOVIE mode

then record.

On Canon 6D, I have recorded 8.30 minutes (22GB) @1600x640 on a Sandisk Extreme PRO 32GB (95MB/s) card formated Exfat.
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 22, 2013, 03:45:18 AM
The "procedure" is:

- Turn cam on in PHOTO mode
- load raw_rec.mo
- in Debug menu, click "Don't click me"
- switch LV to MOVIE mode

then record.

On Canon 6D, I have recorded 8.30 minutes (22GB) @1600x640 on a Sandisk Extreme PRO 32GB (95MB/s) card formated Exfat.

Im on a 5d3, after clicking 'dont click me' and flipping the switch for movie mode, nothing happens. No matter what I do, I can't get any screen to come up
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 04:02:24 AM
Im on a 5d3, after clicking 'dont click me' and flipping the switch for movie mode, nothing happens. No matter what I do, I can't get any screen to come up

That's what it's supposed to do. It's supposed to cut CPU utilization which therefore allows a higher write speed.
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 22, 2013, 04:07:45 AM
That's what it's supposed to do. It's supposed to cut CPU utilization which therefore allows a higher write speed.
After going to video mode I hit record. Nothing recorded to camera
Title: Re: New writing strategy - variable buffering
Post by: kingbones on June 22, 2013, 04:13:57 AM
What Does Don't Click Me! Do???????????????
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 04:23:38 AM
After going to video mode I hit record. Nothing recorded to camera

You should probably reinstall ML. You may have acquired an incorrect version for your 5D3. I mean you are trying to record raw video right? Maybe it disables h264 video. Look in the tutorial section of this forum to see proper instructions.
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 04:25:42 AM
What Does Don't Click Me! Do???????????????

It disables canon menus for the sacrifice of higher write speed.
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 22, 2013, 04:28:23 AM
You should probably reinstall ML. You may have acquired an incorrect version for your 5D3. I mean you are trying to record raw video right? Maybe it disables h264 video. Look in the tutorial section of this forum to see proper instructions.
I can record raw video fine. I can also record the h264 video when I disable raw. But when hitting the don't click button and then switch from photo to video mode, nothing records.
Maybe it just works differently on the 5d3. You're using a 60/6d, right?
Title: Re: New writing strategy - variable buffering
Post by: kingbones on June 22, 2013, 04:30:18 AM
It disables canon menus for the sacrifice of higher write speed.

O..... Okay Koolz Thanks!  ;D
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 04:31:21 AM
I can record raw video fine. I can also record the h264 video when I disable raw. But when hitting the don't click button and then switch from photo to video mode, nothing records.
Maybe it just works differently on the 5d3. You're using a 60/6d, right?

I'm using a 6d, I think you should boot in video mode and hit the don't touch me button in video mode. Starting in photo mode, I think it is not necessary.

http://www.magiclantern.fm/forum/index.php?topic=6099.0 some good guides are poster here on this page.
Title: Re: New writing strategy - variable buffering
Post by: JuanIrache on June 22, 2013, 04:42:43 AM
on my 550D, when I click Don't click me! the LV starts panning left and right
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 22, 2013, 04:43:41 AM
I'm using a 6d, I think you should boot in video mode and hit the don't touch me button in video mode. Starting in photo mode, I think it is not necessary.

http://www.magiclantern.fm/forum/index.php?topic=6099.0 some good guides are poster here on this page.
tried it again starting in photo mode
hit the do not click me the tried to hurry and hit record. also tried hitting record then going into menu and hitting the do not click me

each time I got 'err 70' on the top lcd and camera moves weird then tells me to take out battery

don't think I'll be trying that again.
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 04:46:44 AM
tried it again starting in photo mode
hit the do not click me the tried to hurry and hit record. also tried hitting record then going into menu and hitting the do not click me

each time I got 'err 70' on the top lcd and camera moves weird then tells me to take out battery

don't think I'll be trying that again.

You never started and tried it in video mode. Just a guess...

If not I really feel like you have an old version of ML.  I really do. Look at the link I posted a couple posts ago. The videos are good tutorials.
Title: Re: New writing strategy - variable buffering
Post by: Can on June 22, 2013, 04:48:33 AM
My live view starts panning back and forth after don't click me, then I get an err80...???
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 04:50:53 AM
My live view starts panning back and forth after don't click me, then I get an err80...???

What camera are you using?
Title: Re: New writing strategy - variable buffering
Post by: Can on June 22, 2013, 04:52:21 AM
What camera are you using?

60D
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 05:03:18 AM
60D

It seems that the only camera that successfully can use this feature is the 6D.
Title: Re: New writing strategy - variable buffering
Post by: Can on June 22, 2013, 05:08:44 AM
It seems that the only camera that successfully can use this feature is the 6D.

Thanks - I will discontinue this practice.
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 05:12:18 AM
Thanks - I will discontinue this practice.

A1ex and 1% will sort out these issues. Try to describe your problems to them to starting mending this bug.
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 22, 2013, 05:24:05 AM
You never started and tried it in video mode. Just a guess...

If not I really feel like you have an old version of ML.  I really do. Look at the link I posted a couple posts ago. The videos are good tutorials.
I meant to say that I started in video mode NOT photo mode.

I have the newest version of ML, with Alex's new findings..
I update it almost every day.
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on June 22, 2013, 05:35:06 AM
I meant to say that I started in video mode NOT photo mode.

I have the newest version of ML, with Alex's new findings..
I update it almost every day.

Yeah It seems with these other users that they are experiencing similar problems. Wait for a fix. Sorry.
Title: Re: New writing strategy - variable buffering
Post by: budafilms on June 22, 2013, 06:46:28 AM
¨The kaos is an order to resolve¨.

(See you in a couple of months).
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 22, 2013, 08:28:05 AM
The don't click trick is only from Tragic Lantern and it's camera-specific. I don't consider it safe enough for normal builds, because it breaks Canon UI and you can't undo it without rebooting.
Title: Re: New writing strategy - variable buffering
Post by: aaphotog on June 22, 2013, 09:35:18 PM
It slowed down for me on the newest build.
I don't have the option to have the preview set to 'HaCKed'
The "preview option is no longer there.
FPS over ride set to 23.976
I got over 10 seconds with the 3 files you posted(258 frames)
with the newest build that Lourenco posted(last commit: 363a968), I'm only getting 202 frames.
Title: Re: New writing strategy - variable buffering
Post by: 1% on June 23, 2013, 12:41:33 AM
It can be undone now, just have to close lv after unhacking.

Also for 50D at least they can chose timers from 1024-32768 and disabled... so I have to port this part.. on SD cameras the less aggressive hack may be enough.

I too think it got *slightly* slower and the quick ramp up is hit or miss in terms of speed.


Title: Performance gains with variable buffer and speculative start.
Post by: ted ramasola on June 25, 2013, 07:15:50 AM
Performance gains with variable buffer and speculative start.
Using Lexar 1000x 32gb on 5d2. Each resolution was tested twice, results separated by ( / ).
Card is always formatted in camera before a new build is tested.
All tests done with: GD=off, preview=hacked, FPS over ride= 23.976

resolutions at 1x -- frames 1st take/2nd take

jun 19 by a.d.

1880 x 1058 -- 319/326
1880 x 1016 -- 532/425
1880 x 940  -- 779/988
1880 x 854  --continuous

alex variable buffer jun 21

1872 x 1054 -- 486/480
1872 x 1012 -- 755/732
1872 x 936  -- 2,391/2,272
1872 x 850  -- continuous
1856 x 1044 -- 521
1856 x 928  -- continuous

jun 22 by a.d. spec start off

1872 x 1054 -- 485/582
1872 x 1012 -- 932/876
1872 x 936  -- 4,085/3,626
1872 x 850  -- continuous
1856 x 1044 -- 644/563
1856 x 928  -- continuous

jun 22 by a.d. spec start ON

1872 x 1054 -- 504/531
1872 x 1012 -- 949/893
1872 x 936  -- 2,833/2,369
1872 x 850  -- continuous
1856 x 1044 -- 572/639
1856 x 928  -- continuous

jun 24 by a.d.

1872 x 1054 -- 572/506
1872 x 1012 -- 871/858
1872 x 936  -- 2,868/4,009
1872 x 850  -- continuous
1856 x 1044 -- 557/555
1856 x 928  -- continuous

CROP modes in the following builds;

jun 18-19-21-24 were all Unpredictable and always have pinkish cast in all frames.
I have the number of frames tabulated but since they're mostly pinkish I didn't bother posting.

last build with no pink cast in most resolutions was Jun 16 by a.d.
Title: Re: New writing strategy - variable buffering
Post by: a1ex on June 27, 2013, 01:21:57 PM
Maybe you already noticed that latest builds already include an estimation of how many frames you are going to get with current settings.

That estimation is the ideal mathematical model: if the buffer filling speed is capture_speed - write_speed (say in bytes per second), and we know the memory buffer size, you are going to get X frames, where x = buffer_size / (capture_speed - write_speed). The exact formula is in predict_frames in raw_rec.c.

Of course, when write_speed >= capture_speed, you have continuous recording.

I think it's clear that you are not going to get more frames unless (a) you increase the write speed, or (b) you increase the memory buffer.

In my experiments, the numbers predicted with this model were pretty close to reality, which means the buffering strategy is close to optimal. I'd like you to check this hypothesis - e.g. run some test and check whether predicted and actual frame numbers are close.

If you notice that predicted numbers are way higher than actual numbers, this means ML is overestimating the card speed, so it's trying to use buffers that are too large. I need to know whether this happens or not, so I can fine-tune the prediction algorithm.



Here's the latest binary (for all cameras). I've added the dialog slowdown tricks from 1% (enabled for 5D3/5D2/50D only for now, feel free to find the address for the other cameras), so you can expect a little speed boost if you enable "small hacks" in menu.

raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw/b77a85f7ac68/RAW_REC.MO)
Title: Re: New writing strategy - variable buffering
Post by: Bioskop.Inc on June 27, 2013, 03:08:51 PM
Seems accurate on 60D, maybe 1 frame out (but that's down to there always being a skipped frame @ max res) & it changes its prediction as recording goes along.
I only tested it on full resolution, so 1728x....
Title: Re: New writing strategy - variable buffering
Post by: 1% on June 27, 2013, 04:09:25 PM
50D can over/under estimate in the initial recordings.

How did it work on 5d3? Esp with timer changed before LV open?
Title: Re: New writing strategy - variable buffering
Post by: KSphoto on June 27, 2013, 05:52:09 PM
I'm getting close numbers with my 5D2 Lexar pro 600/32 card @ 1880x1058.

actual/predicted
259/263     GD on
244/244     GD off
344/352     GD on/Hack on
345/370     GD off/Hack on
Title: Re: New writing strategy - variable buffering
Post by: Redrocks on June 27, 2013, 06:50:02 PM
1872 x 1250, 5d2, Komputerbay 64GB, small hacks on:

Getting 9/10 frames less than expected consistently, although the number of frames captured has increased from 216 - 229 with the first variable buffer module on the 22nd to 260 - 282 with this module.
Title: Re: New writing strategy - variable buffering
Post by: xNiNELiVES on August 03, 2013, 12:13:12 AM
Has variable buffering been optimized to it's fullest potential yet?