If you ever looked in the comments from raw_rec.c (https://bitbucket.org/hudson/magic-lantern/src/tip/modules/raw_rec/raw_rec.c), you have noticed that I've stated a little goal: 1920x1080 on 1000x cards (of course, 5D3 at 24p). Goal achieved and exceeded - even got reports of 1920x1280 continuous.
During the last few days I took a closer look at the buffering strategy. While it was near-optimal for continuous recording (large contiguous chunks = faster write speed), there was (and still is) room for improvement for those cases when you want to push the recording past the sustained write speed, and squeeze as many frames as possible.
So, I've designed a new buffering strategy (https://bitbucket.org/hudson/magic-lantern/commits/90c50ab005b7) (I'll call it variable buffering), with the following ideas in mind:
* Write speed varies with buffer size, like this (http://www.magiclantern.fm/forum/index.php?topic=5471) (thanks to testers who ran the benchmarks for hours on their cameras)
* Noticing the speed drop is small, it's almost always better to start writing as soon as we have one frame captured. Therefore, the new strategy aims for 100% duty cycle of the card writing task.
* Because large buffers are faster than small ones, these are preferred. If the card is fast enough, only the largest buffers will be touched, and therefore the method is still optimal for continuous recording. Even better - adding a bunch of small buffers will not slow it down at all.
* This algorithm will use every single memory buffer that can contain at least one frame (because small buffers are no longer slowing it down).
* Another cause of stopping: when the buffer is about to overflow, it's usually because the camera is trying to save a huge buffer (say a 32MB one), which will take a long time (say 1.5 seconds on slow SD cameras, 21MB/s). So, I've added a heuristic that limits buffer size - so, in this case, if we predict the buffer will overflow after only 1 second, we'll save only 20MB out of 32, which will finish at 0.95 seconds. At that moment, the capturing task will have a 20MB free chunk that can be used for capturing more frames.
* Buffering is now done at frame level, not at chunk level. This finer granularity allows me to split buffers on the fly, in whatever configuration I believe it's best for the situation.
* The algorithm is designed to adjust itself on the fly; for this, it does some predictions, such as when the buffer is likely to overflow. If it predicts well, it will squeeze a few more frames. If not... not :P
* More juicy details in the comments (https://bitbucket.org/hudson/magic-lantern/src/tip/modules/raw_rec/raw_rec.c).
This is experimental. I've ran a few tests, played back a few videos on the camera, but that was all. I didn't even check whether the frames are saved in the correct order.
Build notes
- This breaks bolt_rec. Buffering is now done at frame level, not at chunk level, so bolt_rec has to be adjusted.
- The current source code has debug mode enabled - it prints funky graphs. You'll find them on the card.
- The debug code will slow down the write speed.
- I'd like you to run some test recordings and paste the graphs - this will allow me to check if there's any difference between theory and practice (you know, in theory there isn't any).
- I did not run any comparison with the older method on the camera (I did only in simulation). Would be very nice if you can do this.
- It may achieve lower write speeds. This is normal, because it also uses smaller buffers. If you also consider the idle time, it should be better overall.
- For normal usage, disable the debug code (look at the top of raw_rec.c).
History
[2013-05-17] Experiment (http://www.magiclantern.fm/forum/index.php?topic=5471) about checking the optimal buffer sizes. People ran the benchmarks for hours on their cameras and posted a bunch of logs. They pretty much confirmed my previous theory, that any buffer size between 16MB and 32MB should result in highest speeds.
[2013-05-30] Noticed that file writes aligned at 512 bytes are a little faster (credits: CHDK (http://chdk.setepontos.com/index.php?topic=9970)). Rounded image size to multiples of 64x32 or 128x16 (https://bitbucket.org/hudson/magic-lantern/commits/647d93aa34924eb936e89d58356c02ea217fe3b2) to ensure 512-byte rounding.
[2013-08-06] Figured out that I could just add some padding (https://bitbucket.org/hudson/magic-lantern/commits/1479df3a2322068663433a000d52749d6a93659a) to each frame to ensure 512-byte rounding and keep the high write speeds without breaking the converters too hard. Also aligned everything at 4096 bytes (https://bitbucket.org/hudson/magic-lantern/commits/150a9c5810834e60da6311282d61aabc267ad9a7), which solved some mysterious lockups from EDMAC and brought back the highest speed in benchmarks (over 700MB/s).
[2013-05-30] speedsim.py (https://bitbucket.org/hudson/magic-lantern/commits/42196e77c62b2a27e78a60b2aa6c70bfa5f8347a) - First attempt to get a mathematical model of the recording process. Input: resolution, fps and available buffers. Output: how many frames you will get, with detailed graphs. Also in-camera estimation (https://bitbucket.org/hudson/magic-lantern/commits/fce6d462d2bb9940d28a5de9d56a395ff542752f) of how many frames you will get with current settings.
[2013-06-18] Took a closer look (https://bitbucket.org/hudson/magic-lantern/commits/82900279ffdfe4fbb435eb89a9a45666124b6675) at these logs (http://www.magiclantern.fm/forum/index.php?topic=5471) and fitted a mathematical model for the speed drop at small buffer sizes.
[2013-06-18] Does buffer ordering/splitting matter? 1% experimented with it before, but there was no clear conclusion.
* is it better to take the highest one first or the smallest one first? there's no clear answer, each one is best for some cases and suboptimal for others.
* since some cameras had very few memory chunks (e.g. 550D: 32+32+8 MB), what if each of the 32MB buffer is divided in 2x16 or 4x8 MB? This brought a significant improvement for resolutions just above the continuous recording threshold, but lowered performance for continuous recording.
* optimization: updated speedsim.py (https://bitbucket.org/hudson/magic-lantern/commits/0db037b3b960f79dc259cdabfbf144f6eb164270) so it finds the best memory configuration for one particular situation. Xaint confirmed (http://www.magiclantern.fm/forum/index.php?topic=5582.msg52770#msg52770) the optimization results on 550D.
* problem: there was no one-size-fits-all solution.
[2013-06-19] Simulation now matches perfectly the real-world results (https://bitbucket.org/hudson/magic-lantern/commits/7eee493f6fc992ea87963b0ac0fd5992e6119582). So, the mathematical model is accurate!
[2013-06-19] Started to sketch the variable buffering algorithm and already got some simulation results (http://www.magiclantern.fm/forum/index.php?topic=5582.msg52817#msg52817). There was a clear improvement for borderline cases (settings that require just a little more write speed that your camera+card can deliver).
Example: 550D, 1280x426, 23.976fps, 21.16MB/s, simulation:
(http://a1ex.magiclantern.fm/bleeding-edge/8%2B2x32MB.png) (http://a1ex.magiclantern.fm/bleeding-edge/9x8MB.png) (http://a1ex.magiclantern.fm/bleeding-edge/variable-buffering.png)
- 8MB + 2x32MB (current method) - 317 frames
- 9x8MB - 1566 frames
- Variable buffering, starting from 8MB + 2x32MB - 1910 frames
[2013-06-20] Ran a few more tests and noticed that it meets or exceeds the performance of the old algorithm with sort/split optimization. There are still some cases where the sort/split method gives 2-3 more frames (no big deal).
Got rid of some spikes, which squeezed a few more frames (1925 or something).
(http://a1ex.magiclantern.fm/bleeding-edge/variable-buffering-1.png)
Implemented the algorithm in camera. It's a bit simplified, I didn't include all the optimizations from the simulated version, but at least it sems to work.
Took me two hours just to write this post. Whooo :D
Enjoy and let me know if the theory actually works!
Fixed broken link - Audionut
Yes, it works.
I get about 45-55% increase in number of frames before skipping compared to 1x8Mb + 4x16Mb buffering method.
(550d Sandisk extreme pro 95 mb/s).
Some actual test results (http://www.magiclantern.fm/forum/index.php?topic=5582.msg53189#msg53189)
Very nice. Hopefully my Lexar 1000x 64gb Card arrives tomorrow so maybe I could test some few things. (on 5D3)
Thanks for your amazing work!
You sir are a wizard... can't wait to try it out. ;D
@Rewind: did you find any screenshots on your card? can you upload them?
They are saved when recording stops (if the build was compiled with debug flags on, which is default right now).
Screenshots for this test (http://www.magiclantern.fm/forum/index.php?topic=5582.msg53189#msg53189)
[spoiler](http://moredo.ru/raw/VRAM3.BMP)
(http://moredo.ru/raw/VRAM4.BMP)
(http://moredo.ru/raw/VRAM5.BMP)
(http://moredo.ru/raw/VRAM6.BMP)
(http://moredo.ru/raw/VRAM7.BMP)[/spoiler]
@a1ex I want to test this on the 5DMKIII immediately - have you got a build link?
Straight from the card, not packaged:
raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/raw_rec.mo) - portable, should work on any camera
autoexec.bin (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/autoexec.bin) and 5D3_113.sym (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/5D3_113.sym) - only for 5D3 1.1.3 (optional)
Thanks. Testing now...
I'm not finding any screenshots... maybe it's me...
Here's some from 60D @ 16:9:
https://dl.dropboxusercontent.com/u/90827273/60D.zip
This was the first build before you disabled Debug Code
So far, all the screenshots looked good, so we can disable the debug info and let it run at full speed.
@Stedda
You need to edit the code
from
#undef DEBUG_BUFFERING_GRAPH
to
#define DEBUG_BUFFERING_GRAPH
ahhh... i thought I read it was by default. My bad. Thanks.
Couldn,t find any screenshots but managed to record 1920x1152 25fps continuosly on my transcend 1000x 64gb card on my 5d mark 3. Before it worked on 1920x1080 24fps, occasionally I could get it to work with 25fps in 1920x1080 25fps depending on the mood of my camera ;).
This means 1 step up in aspect ratio fully working with 25fps. NICE!
Tried also some 50fps and got it working continuosly in 1856x492.
Awesome work Alex!
//D
I didn't expect any improvement in continuous recording TBH.
But if you were getting around 1000 frames before, now you can expect to get around 10000 (which is almost continuous).
Definite improvement a1ex, getting continuous at 24p/25p 1920x1152 now on the lexar 128Gb 1000x. Global Draw On, ETTR On
Quick tests:
24p - 1080 - 92.8MB/s - 10% idle continuous
24p - 1152 - 93MB/s - 5% idle continuous
24p - 1280 - 95.7MB/s - 1-7ms idle > 1000 frames stopped
30p - 1080 - 91.8MB/s - 1ms idle stopped at 502 frames
30p - 1.85:1 - 93.8MB/s - 1ms idle continuous
25p - 1080 - 91.9MB/s - 5% idle continuous
25p - 1152 - 92.2MB/s - 49ms climbing to 80ms idle continuous
Incred1ble work a1ex. :-)
Also getting around 25% improvement on slower 800x cards.
DNG files look good.
I'm seeing an improvement as well. 5d3 Komputerbay 128gb card. Was 1920 @24 for 150 frames before skipping. Now 518 frames before skipping... Oh, and global draw on, memory hack on, force left...
I'm still new, hope this helps!
I am seeing the same as Danne Reported.
I was able to record 1920x1080 continuous previously. Now I can record 1920x1152 to what seems to be continuous. I didn't let it run until it filled the card so I'll have to test more but I look like I may even be able to go to 1920x1280 without a problem based off the on screen graph.
This is on a Transcend 1000x 32GB GD on with ETTR only NO Memory Hack.
Nice !
Model: 60D
Card: Sandisk Extreme 45 MB/s 16 GB
Global draw off.
1280 x 544 (2.35:1).
Before: 707 frames.
After: 861 frames.
Got an ERR 70 the 1st time, but not the 2nd, 3rd...
The buffer(s) indicator is cool, better than the stars by far.
Edit: at 1152 x 490 (2.35:1), before 1802 frames, after over 3,000 frames!
(http://www.toxicsuds.com/magic_lantern/VRAM2.jpg)
that looks really promising. great alex
As soon as I get back home Alex, I will run the test on all of my cards, and let you know the results and screenshots.
Does it work with 650D?
Could we test it on 6D?
thank you a1ex for that wonderful work!
has anybody found a reason why there´s always that mysterious 2 times so called warm up recordings?
it can´t really be a thermal warm up.... then we would be way out of regular technical specifications.
what´s your strategy in optimizing the (probably) card specific writing strategy?
firmware check on the cards (brand, size, manufacturing date)?
or a new benchmarking tool in ML to determine the best settings?
is there a way to format the card from inside ML with optimized cluster settings and/or aligning?
i think the problem here will be the cards and their constantly changing inner structure. let there be a new controler IN the card and the writing strategy might be a nightmare.... a komputerbay x1000 might work for one user and for the next it might be completely "useless"....because everybody is looking for the highest resolutions possible. sick pixel x pixel marketing....
or am i wrong and everything you do is card independent?
Quote from: a1ex on June 20, 2013, 06:51:56 PM
Straight from the card, not packaged:
autoexec.bin (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/autoexec.bin), raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/raw_rec.mo) and 5D3_113.sym (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/5D3_113.sym)
I replaced the files on my sd with these 3 files and the camera wont boot.
Should I be doing something additional?
Correct me if I'm mistaking but it's the 5D Mark II & III the top DSLR from Canon and isn't Magic Lantern pulling the most out of these 2 DSLR? I'm just asking because I'm extremely grateful by the work you guys do and I'm hoping you don't stop pushing the 5D Mark II, because I read a lot on these forum about the 6D and other Canon cameras. I just want you guys to give me a straighten out, I would love to hear that 5D Mark II can get variable buffering before I start reading 60D is been tested. I know I know I know, I'm selfish. I'm working on it ;)
with my buggy CF card it gets now >1k instead of ~100 frames when recording in 1080
Cool :D
The strategy is universal, it's trying to use the available resources in the best possible way. It measures the card speed on the fly.
You can use the module from my previous post on any camera (the code is portable). The autoexec and sym are only for 5D3.
I figured it out.
SHIT Alex, this is amazing!!!
Recording 2560x1090 on a Sandisk 600x CF 16GB card
with the june 16th build
72 frames with memory hack off, and preview set to auto
120 frames with memory hack on, and preview set to 'HaCKed'
With the new 3 files you posted, i;m getting
142 frames with memory hack off, and preview set to auto
258 frames with memory hack on, and preview set to 'HaCKed'
as you can see, it doubled!
With the CF that I have, I've always been able to get continuous 1920x1080, so I thought I'd go with something a little more demanding.
BRAVO!!!
WOW!
60D 304 frames 1280x720 16x9 (previously 175 frames)
1728x692 185 frames 2.50
Very nice.
Quote from: aaphotog on June 20, 2013, 10:27:39 PM
I figured it out.
So could you please tell us how did you do that?
I replaced the old raw_rec.mo with the new one from the post #7 in my 6D but no difference. :-\
Awesome job! Got 200 frames in 1920x on 50D in crop mode, was about 70 before.
Quote from: a1ex on June 20, 2013, 10:22:03 PM
Cool :D
The strategy is universal, it's trying to use the available resources in the best possible way. It measures the card speed on the fly.
You can use the module from my previous post on any camera (the code is portable). The autoexec and sym are only for 5D3.
Cool, I will install and get back to you. By the way does the AutoExec Bin activate the module?
Very nice.
5D3, KomputerBay 128GB 1000x
1920x1080 @25
Before 300 frames
Now 500 frames
Thank you!
And I've just got a proof that this strategy is far from optimal :D
(read: some more improvements may be coming soon)
the Force is strong with you A1EX! :o
A1ex are we losing some resolution with variable buffering because the highest resolution with 2.20.1 was 1880x854 but now it's 1872x850 and its not continuous on a Lexar 1000x, but 1856x844 is continuous.
Damn A1ex! Congrats 8)
Canon 600D 24p variable buffer improvements vs old method
(before max seconds -> now max seconds).
W/o line skipping:
1280x720 - 3 > 6 (crop-factor = 2.16)
1280x536 - 6 > 14 (crop-factor = 2.16)
16:9 line skipping (unsqueezed resolution in brackets)
1280x434(720) - 29 > 42 (crop-factor = 2.16)
1344x456(756) - 11 > 27 (crop-factor = 2.06)
1472x500(828) - 5 > 11 (crop-factor = 1.88)
1600x542(900) - 3 > 7 (crop-factor = 1.72)
1728x586(972) - 2 > 5 (crop-factor = 1.6)
2.39:1 line skipping (unsqueezed resolution in brackets)
1472x370(616) - 50 > very long (crop-factor = 1.88)
1600x404(670) - 8 > 19 (crop-factor = 1.72)
1728x436(724) - 5 > 10 (crop-factor = 1.6)
Thank you, a1ex!
Promising... I wonder what it will do on 600D w/ dialog timers disabled.
2560 x 1072 23.976 fps over 700 frames before frame skipping. 104 MB/s.
1920x448 (720) 60p 309 frames pre variable buffer, 436 frames with variable buffer. That's about 18 seconds of 2.67:1 slo-mo, very usable. 1968 frames hacked. The pre variable build needed 4 or 5 recordings to warm up to the 309 frames.
2560x960 24p continuous in crop mode. ;D
1920 x 508 (818) 48p continuous.
Lexar 1000x 64Gb GD off.
1920x448 60p 568 frames with the Toshiba 1066x 64Gb.
Quote from: 1% on June 21, 2013, 01:20:00 AM
Promising... I wonder what it will do on 600D w/ dialog timers disabled.
Magic! But I'm still fixing the wetting problem. I think a bib can do wonders.
Where Do you enable this? Do I need to format the card to have certain buffer size for this to work optimally?
1280*720 on my 60D went from 335 to 379-381 frames before skipping at 23.976 fps.
You don't enable it, it's built into the raw_rec code.
Quote from: squig on June 21, 2013, 03:52:24 AM
You don't enable it, it's built into the raw_rec code.
But what about setting the buffer size on the card?
Quote from: xNiNELiVES on June 21, 2013, 03:54:54 AM
But what about setting the buffer size on the card?
And what would this mean?
A card is a card, a buffer is a buffer.
The buffer is in the camera, the card can just do whatever possible to sustain the data rate if it's capable of doing it.
Quote from: ilguercio on June 21, 2013, 03:59:54 AM
And what would this mean?
A card is a card, a buffer is a buffer.
I meant cluster size :-\
Great!!! Got it. Thank you 1% :D
[6D] The hacks - 1792x752 before 280 farmes 11-12 sec, now 373 frames 16 sec. I cannot believe this :D
We went up two aspect ratios: from 2,39:1 to 2,20:1 - 1792x816 now records 282 frames :D
additionally
1792x976 1,85:1 - now gives 153 frames
1792x896 2:1 - 190 frames
Everything Hack OFF, global draw off, sound beep.
With don't click me?
Great job, I was able to hit 2560x1024 for over 1000 frames and 2880x1154 for around 130 frames. After a little bit I started getting the "nothing saved, card maybe full" error.
Quote from: crazyrunner33 on June 21, 2013, 04:28:05 AM
Great job, I was able to hit 2560x1024 for over 1000 frames and 2880x1154 for around 130 frames. After a little bit I started getting the "nothing saved, card maybe full" error.
I got that error as well. but I've also gotten it before this newest hack
I read de explanation from A1ex, but my english is not so good.
I understand that when the buffer is going to colapsed, the size of the frame is reduced.
If this is like this, means that in some frames the quality is low? So in the image we can perceibe this automatically reduce of quality?
Any explanation (or in Italian, spanish) or english for dummies?
thanks
Cluster size you pick when you format the card.
@alex i intslled your 3 files and im using a promasters 1000x card 64 gigs on 5d3 still cant get continuous 1920 x 1080 only 320 frames and 410 frames with hacked preview. but 1920x960 got 827 frames about 30 % more then before . With hacked preview I got 1725 frames, @ 1920 x 872 is continuous And I was able to get about 120 frames @ 1920 x 1280 this is really good i still havent been able to convert files bigger then 4 gigs with raw magic. maybe my compiled version isnt right maybe you can email me one @
[email protected]
Quote from: 1% on June 21, 2013, 04:48:33 AM
Cluster size you pick when you format the card.
Yeah I know but what should it be? I think I read something on this thread about having to set your cluster size to a certain optimal value.
Holy Shit - I got 600 frames on 1792 x 672. Originally I had 450. Thats a 25 percent increase!
I just ran a comparison test with this and jun 19 build by a.d.
the higher resolutions have changed to slightly smaller so I see slight improvements.
but strangely at 1728 x 972
A1ex variable buffering--- stops at approx. 1,738 or 1,625 frames.
Jun 19 build----------------- records until card almost full. 10,000 = frames.
Is there a setting i should use when running this variable buffer build to get optimum performance?
Quote from: ted ramasola on June 21, 2013, 07:58:20 AM
I just ran a comparison test with this and jun 19 build by a.d.
the higher resolutions have changed to slightly smaller so I see slight improvements.
but strangely at 1728 x 972
A1ex variable buffering--- stops at approx. 1,738 or 1,625 frames.
Jun 19 build----------------- records until card almost full. 10,000 = frames.
Is there a setting i should use when running this variable buffer build to get optimum performance?
Maybe you didn't enable the memory hack... Global draw should be off. Make sure fps over ride is set to on (24fps), sometimes my camera changed it to 30. All I can think of right now, maybe someone else can find out what happened.
@ted: the builds from a.d. contain a number of extra changes, e.g. allows non-mod16 resolutions (which I prefer to stay away from, for DMA alignment reasons), or auto-disabling global draw (I think this should be up to the user, not automatic).
This one is vanilla, there are no changes that I didn't review.
Quote from: a1ex on June 21, 2013, 08:09:11 AM
@ted: the builds from a.d. contain a number of extra changes, e.g. allows non-mod16 resolutions (which I prefer to stay away from, for DMA alignment reasons), or auto-disabling global draw (I think this should be up to the user, not automatic).
This one is vanilla, there are no changes that I didn't review.
Ok. Thank you.
In that case when I run the rawrecmod that you posted, There are no special configurations or settings i should do aside from the usual? FPS over ride to=23.976
preview=hacked
global draw =OFF
??
I see no one is reporting results from the 5D2, is there a test I can do to help you in finding bug?
Well... hacked preview has little or no effect on 5D2 afaik. I'd say try normal settings, global draw off, and compare with what you already have. Just make sure you have the same FPS and resolution for both.
I tried this out on my 5D2 this afternoon. Previously the highest continuous 16:9 video I could record was 1600x900, but with this new module I can get continuous 1724x970 on my SanDisk 90MBs. Upscales to 1080p beautifully.
Quote from: lourenco on June 21, 2013, 01:58:15 AM
2560 x 1072 23.976 fps over 700 frames before frame skipping. 104 MB/s.
Near to 2K! A real Cinema camcorder!
what the hell alex ;D
i got countinius shooting at 1280*388 in 720p mode at 23,97fps
and at 1280*544 in 1080p 191 frames on my 550d
massive!!!
On my 5D3 1920X818 with SanDisk Extreme 500 frames before and now is it 700+ frames so it work realy great :)
Just added a small startup optimization (speculative start): https://bitbucket.org/hudson/magic-lantern/commits/50209fff8bbb
Before, it was writing to the card as soon as there was one captured frame in the buffer. Even if this meant it was writing a little slower.
Now, it starts writing as soon as there is one frame captured, but the write command is issued in advance, for the entire contiguous memory area. Of course, subsequent frames will be captured while writing. As long as the file writing task is slower than frame capturing task, this trick will work.
(Don't confuse this with pre-buffering.)
Obviously, this technique is inherently unsafe. If, for some reason, the card controller starts writing 120% faster than we have measured, the risk is that we may save a frame that does not contain complete image data.
In theory, I can detect this situation, and I've added quite a few consistency checks. I did not encounter any lost frames yet.
Don't expect miracles from it. On my cards, I did not get any noticeable improvement, other than prettier buffer graphs (with large steps from the very beginning) and the mathematical proof that both startup and cruise "segments" are saved at the maximum speed possible with standard file I/O.
I expect it to work a little better on Ted's card, where yesterday's method didn't work very well.
And of course, since the optimization is a little unsafe, it's turned off by default.
Binary (same link): raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/5D3/variable-buffering/raw_rec.mo)
600D, 1280x512 (640 mode), GD off, 410 frames.
:D
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
comparing A1ex's new build to the normal build I use (thanks a.d.)
A1ex: 1872 x 1250 @ 1:2
200
208
201
190
205
5x 2144 x 1078 @ 1:2
201
201
206
201
208
a.d.: 1880 x 1250 @1:2
174
174
174
174
174
5x 2152 x 1078 @1:2
185
184
185
184
185
__________________
A1ex: 1872 x 936 @2:1 (first unreliable ratio)
2077
2016
2429
2678
2171
5 x 2144 x 1072 @2:1 (first unreliable ratio)
211
209
207
207
211
a.d.: 1880 x 940 @2:1 (first unreliable ratio)
2061
2432
2289
2660
Battery gave out during final clip
5x 2152 x 1076 @2:1 (first unreliable ratio)
59
126
177
185
184
___________________
A1ex: 1872 x 1012 @1.85:1
674
738
676
725
679
5x 2144 x 1078 @1.85:1
207
200
208
201
208
a.d.: 1880 x 1016 @1.85:1
541
661
551
532
617
5x 2152 x 1078 @1.85:1
185
184
185
184
185
___________________
A1ex: 1872 x 1054 @16:9
475
488
466
470
468
5x 2144 x 1078 @16:9
204
205
206
203
208
a.d.: 1880 x 1058 @16:9
408
417
408
417
416
5x 2152 x 1078 @16:9
185
184
185
184
185
Quote from: Redrocks on June 21, 2013, 02:10:51 PM
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
with A1ex's module from first page
1870 x 1250
200
208
201
190
205
5x 2144 x 1078
201
201
206
201
208
with current a.d. module on another Komputerbay
1880 x 1250
174
174
174
174
174
5x 2152 x 1078
185
184
185
184
185
Redrock are you checking for full ability with the 5D Mark II and we're getting at 1x with new module average 200 at 1870x1250, 5x 2144x1078? I will check my Lexar limitation and post back
Yes dariSSight, I aimed my first test at the maximum resolution and aspect ratio. I've since then updated the post with other aspect ratio tests.
Quote from: Redrocks on June 21, 2013, 02:10:51 PM
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
comparing A1ex's new build to the normal build I use (thanks a.d.)
A1ex: 1872 x 1250 @ 1:2
200
208
201
190
205
5x 2144 x 1078 @ 1:2
201
201
206
201
208
a.d.: 1880 x 1250 @1:2
174
174
174
174
174
5x 2152 x 1078 @1:2
185
184
185
184
185
__________________
A1ex: 1872 x 936 @2:1 (first unreliable ratio)
2077
2016
2429
2678
2171
5 x 2144 x 1072 @2:1 (first unreliable ratio)
211
209
207
207
211
a.d.: 1880 x 940 @2:1 (first unreliable ratio)
2061
2432
2289
2660
Battery gave out during final clip
5x 2152 x 1076 @2:1 (first unreliable ratio)
59
126
177
185
184
___________________
A1ex: 1872 x 1012 @1.85:1
674
738
676
725
679
5x 2144 x 1078 @1.85:1
207
200
208
201
208
a.d.: 1880 x 1016 @1.85:1
541
661
551
532
617
5x 2152 x 1078 @1.85:1
185
184
185
184
185
___________________
A1ex: 1872 x 1054 @16:9
475
488
466
470
468
5x 2144 x 1078 @16:9
204
205
206
203
208
a.d.: 1880 x 1058 @16:9
408
417
408
417
416
5x 2152 x 1078 @16:9
185
184
185
184
185
Why has the resolution change from 1880 to 1872?
It's just a quirk of A1ex's build, I'm sure he will have a reason for the slight differences between his and a.d's resolutions. I actually set the resolution to 3584 for all the tests, but the numbers quoted are what the camera actually puts out.
If you have read the previous posts and the source code, you would have noticed the quirk is in a.d.'s build (he has an exception for 1880, which I don't like). Bigger is not always better, mod16 resolutions are more important IMO, and the EDMAC is very sensitive to alignment.
Sorry for the inference, it's quirky to me in the context of me using a.d's builds is what I meant. I just realised that 1872 is an exact 1/3 rd of the actual sensor, which you just explained is more suitable to the task.
I did a test on my old 30MB/s Sandisk Ultras. Bold text is resolution, 1st numbers are ratio and 2nd numbers frames:
1872x
796 182
850 155
936 131
1012 119
1054 107
1124 99
1248 85
1856 x
790 186
844 165
928 143
1004 124
1044 116
1114 104
1238 86
1728 x
736 267
786 267
864 183
934 156
972 151
1036 123
1152 107
1600 x
680 388
728 318
800 250
864 218
900 198
1200 115
1472 x
796 312
828 259
884 220
982 187
1104 155
1344 x
572 3122
610 1456
672 780
726 573
756 447
806 412
896 305
1008 226
1250 150
1280 x
544 6518
582 3526
640 1628
692 908
720 717
768 539
854 399
960 281
1250 152
Interesting developments over in the 6D Dev thread (http://www.magiclantern.fm/forum/index.php?topic=3904.2275)!
teo77 has made a great discovery that formatting to exfat dramatically increases recording times. Basically I was banging my head against a wall all day, wondering why he was getting 5:05 mins @ 1600x640 and I was getting 02:15 with latest hack. Anyway - he then figured that formatting his Sandisk Extreme Pro 95MB/s to exfat (4096 clusters), combined with latest module and don't click me from 02:15 mins all the way up to SEVEN MINUTES! If anyone else can try this out and share your results it could help :)
A1ex - do you think this is a problem caused by the files being split @ 4gb?
Quote from: noisyboy on June 22, 2013, 02:06:48 AM
Interesting developments over in the 6D Dev thread (http://www.magiclantern.fm/forum/index.php?topic=3904.2275)!
teo77 has made a great discovery that formatting to exfat dramatically increases recording times. Basically I was banging my head against a wall all day, wondering why he was getting 5:05 mins @ 1600x640 and I was getting 02:15 with latest hack. Anyway - he then figured that formatting his Sandisk Extreme Pro 95MB/s to exfat (4096 clusters), combined with latest module and don't click me from 02:15 mins all the way up to SEVEN MINUTES! If anyone else can try this out and share your results it could help :)
A1ex - do you think this is a problem caused by the files being split @ 4gb?
What does dont click me do?
He clicks that AFTER he's started recording for 2 minutes(while still recording)?
The "procedure" is:
- Turn cam on in PHOTO mode
- load raw_rec.mo
- in Debug menu, click "Don't click me"
- switch LV to MOVIE mode
then record.
On Canon 6D, I have recorded 8.30 minutes (22GB) @1600x640 on a Sandisk Extreme PRO 32GB (95MB/s) card formated Exfat.
Quote from: teo770 on June 22, 2013, 02:55:17 AM
The "procedure" is:
- Turn cam on in PHOTO mode
- load raw_rec.mo
- in Debug menu, click "Don't click me"
- switch LV to MOVIE mode
then record.
On Canon 6D, I have recorded 8.30 minutes (22GB) @1600x640 on a Sandisk Extreme PRO 32GB (95MB/s) card formated Exfat.
Im on a 5d3, after clicking 'dont click me' and flipping the switch for movie mode, nothing happens. No matter what I do, I can't get any screen to come up
Quote from: aaphotog on June 22, 2013, 03:45:18 AM
Im on a 5d3, after clicking 'dont click me' and flipping the switch for movie mode, nothing happens. No matter what I do, I can't get any screen to come up
That's what it's supposed to do. It's supposed to cut CPU utilization which therefore allows a higher write speed.
Quote from: xNiNELiVES on June 22, 2013, 04:02:24 AM
That's what it's supposed to do. It's supposed to cut CPU utilization which therefore allows a higher write speed.
After going to video mode I hit record. Nothing recorded to camera
What Does Don't Click Me! Do???????????????
Quote from: aaphotog on June 22, 2013, 04:07:45 AM
After going to video mode I hit record. Nothing recorded to camera
You should probably reinstall ML. You may have acquired an incorrect version for your 5D3. I mean you are trying to record raw video right? Maybe it disables h264 video. Look in the tutorial section of this forum to see proper instructions.
Quote from: kingbones on June 22, 2013, 04:13:57 AM
What Does Don't Click Me! Do???????????????
It disables canon menus for the sacrifice of higher write speed.
Quote from: xNiNELiVES on June 22, 2013, 04:23:38 AM
You should probably reinstall ML. You may have acquired an incorrect version for your 5D3. I mean you are trying to record raw video right? Maybe it disables h264 video. Look in the tutorial section of this forum to see proper instructions.
I can record raw video fine. I can also record the h264 video when I disable raw. But when hitting the don't click button and then switch from photo to video mode, nothing records.
Maybe it just works differently on the 5d3. You're using a 60/6d, right?
Quote from: xNiNELiVES on June 22, 2013, 04:25:42 AM
It disables canon menus for the sacrifice of higher write speed.
O..... Okay Koolz Thanks! ;D
Quote from: aaphotog on June 22, 2013, 04:28:23 AM
I can record raw video fine. I can also record the h264 video when I disable raw. But when hitting the don't click button and then switch from photo to video mode, nothing records.
Maybe it just works differently on the 5d3. You're using a 60/6d, right?
I'm using a 6d, I think you should boot in video mode and hit the don't touch me button in video mode. Starting in photo mode, I think it is not necessary.
http://www.magiclantern.fm/forum/index.php?topic=6099.0 some good guides are poster here on this page.
on my 550D, when I click Don't click me! the LV starts panning left and right
Quote from: xNiNELiVES on June 22, 2013, 04:31:21 AM
I'm using a 6d, I think you should boot in video mode and hit the don't touch me button in video mode. Starting in photo mode, I think it is not necessary.
http://www.magiclantern.fm/forum/index.php?topic=6099.0 some good guides are poster here on this page.
tried it again starting in photo mode
hit the do not click me the tried to hurry and hit record. also tried hitting record then going into menu and hitting the do not click me
each time I got 'err 70' on the top lcd and camera moves weird then tells me to take out battery
don't think I'll be trying that again.
Quote from: aaphotog on June 22, 2013, 04:43:41 AM
tried it again starting in photo mode
hit the do not click me the tried to hurry and hit record. also tried hitting record then going into menu and hitting the do not click me
each time I got 'err 70' on the top lcd and camera moves weird then tells me to take out battery
don't think I'll be trying that again.
You never started and tried it in video mode. Just a guess...
If not I really feel like you have an old version of ML. I really do. Look at the link I posted a couple posts ago. The videos are good tutorials.
My live view starts panning back and forth after don't click me, then I get an err80...???
Quote from: Can on June 22, 2013, 04:48:33 AM
My live view starts panning back and forth after don't click me, then I get an err80...???
What camera are you using?
Quote from: Can on June 22, 2013, 04:52:21 AM
60D
It seems that the only camera that successfully can use this feature is the 6D.
Quote from: xNiNELiVES on June 22, 2013, 05:03:18 AM
It seems that the only camera that successfully can use this feature is the 6D.
Thanks - I will discontinue this practice.
Quote from: Can on June 22, 2013, 05:08:44 AM
Thanks - I will discontinue this practice.
A1ex and 1% will sort out these issues. Try to describe your problems to them to starting mending this bug.
Quote from: xNiNELiVES on June 22, 2013, 04:46:44 AM
You never started and tried it in video mode. Just a guess...
If not I really feel like you have an old version of ML. I really do. Look at the link I posted a couple posts ago. The videos are good tutorials.
I meant to say that I started in video mode NOT photo mode.
I have the newest version of ML, with Alex's new findings..
I update it almost every day.
Quote from: aaphotog on June 22, 2013, 05:24:05 AM
I meant to say that I started in video mode NOT photo mode.
I have the newest version of ML, with Alex's new findings..
I update it almost every day.
Yeah It seems with these other users that they are experiencing similar problems. Wait for a fix. Sorry.
¨The kaos is an order to resolve¨.
(See you in a couple of months).
The don't click trick is only from Tragic Lantern and it's camera-specific. I don't consider it safe enough for normal builds, because it breaks Canon UI and you can't undo it without rebooting.
It slowed down for me on the newest build.
I don't have the option to have the preview set to 'HaCKed'
The "preview option is no longer there.
FPS over ride set to 23.976
I got over 10 seconds with the 3 files you posted(258 frames)
with the newest build that Lourenco posted(last commit: 363a968), I'm only getting 202 frames.
It can be undone now, just have to close lv after unhacking.
Also for 50D at least they can chose timers from 1024-32768 and disabled... so I have to port this part.. on SD cameras the less aggressive hack may be enough.
I too think it got *slightly* slower and the quick ramp up is hit or miss in terms of speed.
Performance gains with variable buffer and speculative start.
Using Lexar 1000x 32gb on 5d2. Each resolution was tested twice, results separated by ( / ).
Card is always formatted in camera before a new build is tested.
All tests done with: GD=off, preview=hacked, FPS over ride= 23.976
resolutions at 1x -- frames 1st take/2nd take
jun 19 by a.d.
1880 x 1058 -- 319/326
1880 x 1016 -- 532/425
1880 x 940 -- 779/988
1880 x 854 --continuous
alex variable buffer jun 21
1872 x 1054 -- 486/480
1872 x 1012 -- 755/732
1872 x 936 -- 2,391/2,272
1872 x 850 -- continuous
1856 x 1044 -- 521
1856 x 928 -- continuous
jun 22 by a.d. spec start off
1872 x 1054 -- 485/582
1872 x 1012 -- 932/876
1872 x 936 -- 4,085/3,626
1872 x 850 -- continuous
1856 x 1044 -- 644/563
1856 x 928 -- continuous
jun 22 by a.d. spec start ON
1872 x 1054 -- 504/531
1872 x 1012 -- 949/893
1872 x 936 -- 2,833/2,369
1872 x 850 -- continuous
1856 x 1044 -- 572/639
1856 x 928 -- continuous
jun 24 by a.d.
1872 x 1054 -- 572/506
1872 x 1012 -- 871/858
1872 x 936 -- 2,868/4,009
1872 x 850 -- continuous
1856 x 1044 -- 557/555
1856 x 928 -- continuous
CROP modes in the following builds;
jun 18-19-21-24 were all Unpredictable and always have pinkish cast in all frames.
I have the number of frames tabulated but since they're mostly pinkish I didn't bother posting.
last build with no pink cast in most resolutions was Jun 16 by a.d.
Maybe you already noticed that latest builds already include an estimation of how many frames you are going to get with current settings.
That estimation is the ideal mathematical model: if the buffer filling speed is capture_speed - write_speed (say in bytes per second), and we know the memory buffer size, you are going to get X frames, where x = buffer_size / (capture_speed - write_speed). The exact formula is in predict_frames in raw_rec.c.
Of course, when write_speed >= capture_speed, you have continuous recording.
I think it's clear that you are not going to get more frames unless (a) you increase the write speed, or (b) you increase the memory buffer.
In my experiments, the numbers predicted with this model were pretty close to reality, which means the buffering strategy is close to optimal. I'd like you to check this hypothesis - e.g. run some test and check whether predicted and actual frame numbers are close.
If you notice that predicted numbers are way higher than actual numbers, this means ML is overestimating the card speed, so it's trying to use buffers that are too large. I need to know whether this happens or not, so I can fine-tune the prediction algorithm.
Here's the latest binary (for all cameras). I've added the dialog slowdown tricks from 1% (enabled for 5D3/5D2/50D only for now, feel free to find the address for the other cameras), so you can expect a little speed boost if you enable "small hacks" in menu.
raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw/b77a85f7ac68/RAW_REC.MO)
Seems accurate on 60D, maybe 1 frame out (but that's down to there always being a skipped frame @ max res) & it changes its prediction as recording goes along.
I only tested it on full resolution, so 1728x....
50D can over/under estimate in the initial recordings.
How did it work on 5d3? Esp with timer changed before LV open?
I'm getting close numbers with my 5D2 Lexar pro 600/32 card @ 1880x1058.
actual/predicted
259/263 GD on
244/244 GD off
344/352 GD on/Hack on
345/370 GD off/Hack on
1872 x 1250, 5d2, Komputerbay 64GB, small hacks on:
Getting 9/10 frames less than expected consistently, although the number of frames captured has increased from 216 - 229 with the first variable buffer module on the 22nd to 260 - 282 with this module.
Has variable buffering been optimized to it's fullest potential yet?