Author Topic: New writing strategy - variable buffering  (Read 59184 times)

kgv5

  • Hero Member
  • *****
  • Posts: 643
Re: New writing strategy - variable buffering
« Reply #50 on: June 21, 2013, 04:12:59 AM »
Great!!! Got it. Thank you 1%   :D

[6D] The hacks - 1792x752 before 280 farmes 11-12 sec, now 373 frames 16 sec.  I cannot believe this  :D
We went up two aspect ratios: from 2,39:1 to 2,20:1 - 1792x816 now records 282 frames  :D

additionally
1792x976 1,85:1 - now gives 153 frames
1792x896 2:1 - 190 frames

Everything Hack OFF, global draw off, sound  beep.
www.pilotmovies.pl   5D Mark III, 6D, 550D

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New writing strategy - variable buffering
« Reply #51 on: June 21, 2013, 04:23:48 AM »
With don't click me?

crazyrunner33

  • New to the forum
  • *
  • Posts: 42
Re: New writing strategy - variable buffering
« Reply #52 on: June 21, 2013, 04:28:05 AM »
Great job, I was able to hit 2560x1024 for over 1000 frames and 2880x1154 for around 130 frames.  After a little bit I started getting the "nothing saved, card maybe full" error.
5D Mark III, 7D

aaphotog

  • Senior
  • ****
  • Posts: 356
Re: New writing strategy - variable buffering
« Reply #53 on: June 21, 2013, 04:43:38 AM »
Great job, I was able to hit 2560x1024 for over 1000 frames and 2880x1154 for around 130 frames.  After a little bit I started getting the "nothing saved, card maybe full" error.
I got that error as well. but I've also gotten it before this newest hack

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: New writing strategy - variable buffering
« Reply #54 on: June 21, 2013, 04:47:19 AM »
I read de explanation from A1ex, but my english is not so good.

I understand that when the buffer is going to colapsed, the size of the frame is reduced.
If this is like this, means that in some frames the quality is low? So in the image we can perceibe this automatically reduce of quality?

Any explanation (or in Italian, spanish) or english for dummies?

thanks

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New writing strategy - variable buffering
« Reply #55 on: June 21, 2013, 04:48:33 AM »
Cluster size you pick when you format the card.

kaoshotbeatz

  • New to the forum
  • *
  • Posts: 5
Re: New writing strategy - variable buffering
« Reply #56 on: June 21, 2013, 05:05:52 AM »
@alex i intslled your 3 files and im using a promasters 1000x card 64 gigs on 5d3 still cant get continuous 1920 x 1080 only 320 frames  and 410 frames with hacked preview. but 1920x960 got 827 frames about 30 % more then before . With hacked preview I got 1725 frames, @  1920 x 872 is continuous  And I  was able to get about 120 frames @ 1920 x 1280 this is really good i still havent been able to convert files bigger then 4 gigs with raw magic. maybe my compiled version isnt right maybe you can email me one @kaoshotbeatz@gmail.com

xNiNELiVES

  • Senior
  • ****
  • Posts: 461
  • 5D II, 17-35mm F2.8L, 50mm F1.4, 70-200 F4L
Re: New writing strategy - variable buffering
« Reply #57 on: June 21, 2013, 05:44:21 AM »
Cluster size you pick when you format the card.

Yeah I know but what should it be? I think I read something on this thread about having to set your cluster size to a certain optimal value.

Holy Shit - I got 600 frames on 1792 x 672. Originally I had 450. Thats a 25 percent increase!

ted ramasola

  • Moderators
  • Hero Member
  • *****
  • Posts: 1250
Re: New writing strategy - variable buffering
« Reply #58 on: June 21, 2013, 07:58:20 AM »
I just ran a comparison test with this and jun 19 build by a.d.

the higher resolutions have changed to slightly smaller so I see slight improvements.

but strangely at 1728 x 972

A1ex variable buffering--- stops at  approx. 1,738 or 1,625 frames.

Jun 19 build----------------- records until card almost full. 10,000 = frames.

Is there a setting i should use when running this variable buffer build to get optimum performance?
5DmkII  / 7D
www.ramasolaproductions.com
Texas

xNiNELiVES

  • Senior
  • ****
  • Posts: 461
  • 5D II, 17-35mm F2.8L, 50mm F1.4, 70-200 F4L
Re: New writing strategy - variable buffering
« Reply #59 on: June 21, 2013, 08:04:33 AM »
I just ran a comparison test with this and jun 19 build by a.d.

the higher resolutions have changed to slightly smaller so I see slight improvements.

but strangely at 1728 x 972

A1ex variable buffering--- stops at  approx. 1,738 or 1,625 frames.

Jun 19 build----------------- records until card almost full. 10,000 = frames.

Is there a setting i should use when running this variable buffer build to get optimum performance?

Maybe you didn't enable the memory hack... Global draw should be off. Make sure fps over ride is set to on (24fps), sometimes my camera changed it to 30. All I can think of right now, maybe someone else can find out what happened.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12318
  • Emergencies only
Re: New writing strategy - variable buffering
« Reply #60 on: June 21, 2013, 08:09:11 AM »
@ted: the builds from a.d. contain a number of extra changes, e.g. allows non-mod16 resolutions (which I prefer to stay away from, for DMA alignment reasons), or auto-disabling global draw (I think this should be up to the user, not automatic).

This one is vanilla, there are no changes that I didn't review.

ted ramasola

  • Moderators
  • Hero Member
  • *****
  • Posts: 1250
Re: New writing strategy - variable buffering
« Reply #61 on: June 21, 2013, 08:21:59 AM »
@ted: the builds from a.d. contain a number of extra changes, e.g. allows non-mod16 resolutions (which I prefer to stay away from, for DMA alignment reasons), or auto-disabling global draw (I think this should be up to the user, not automatic).

This one is vanilla, there are no changes that I didn't review.

Ok. Thank you.

In that case when I run the rawrecmod that you posted, There are no special configurations or settings i should do aside from the usual? FPS over ride to=23.976
preview=hacked
global draw =OFF
??

I see no one is reporting results from the 5D2, is there a test I can do to help you in finding bug?
5DmkII  / 7D
www.ramasolaproductions.com
Texas

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12318
  • Emergencies only
Re: New writing strategy - variable buffering
« Reply #62 on: June 21, 2013, 08:43:08 AM »
Well... hacked preview has little or no effect on 5D2 afaik. I'd say try normal settings, global draw off, and compare with what you already have. Just make sure you have the same FPS and resolution for both.

mustardseed

  • Just arrived
  • *
  • Posts: 1
Re: New writing strategy - variable buffering
« Reply #63 on: June 21, 2013, 09:30:56 AM »
I tried this out on my 5D2 this afternoon. Previously the highest continuous 16:9 video I could record was 1600x900, but with this new module I can get continuous 1724x970 on my SanDisk 90MBs. Upscales to 1080p beautifully.

kgv5

  • Hero Member
  • *****
  • Posts: 643
Re: New writing strategy - variable buffering
« Reply #64 on: June 21, 2013, 09:32:09 AM »
With don't click me?

No, without.
www.pilotmovies.pl   5D Mark III, 6D, 550D

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: New writing strategy - variable buffering
« Reply #65 on: June 21, 2013, 10:07:33 AM »
2560 x 1072 23.976 fps over 700 frames before frame skipping.  104 MB/s.

Near to 2K! A real Cinema camcorder!

hammermina

  • New to the forum
  • *
  • Posts: 46
Re: New writing strategy - variable buffering
« Reply #66 on: June 21, 2013, 12:32:37 PM »
what the hell alex ;D

i got countinius shooting at 1280*388 in 720p mode at 23,97fps
and at 1280*544 in 1080p 191 frames on my 550d

massive!!!

Jackie

  • New to the forum
  • *
  • Posts: 8
Re: New writing strategy - variable buffering
« Reply #67 on: June 21, 2013, 12:36:45 PM »
On my 5D3 1920X818 with SanDisk Extreme 500 frames before and now is it 700+ frames so it work realy great :)
5D3 - Tamron SP 24-70mm F/2.8 Di VC USD - Canon EF 50mm f/1.4 USM - Canon EF 70-200mm f/4L USM - Yongnuo YN-560 II

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12318
  • Emergencies only
Re: New writing strategy - variable buffering
« Reply #68 on: June 21, 2013, 01:15:47 PM »
Just added a small startup optimization (speculative start): https://bitbucket.org/hudson/magic-lantern/commits/50209fff8bbb

Before, it was writing to the card as soon as there was one captured frame in the buffer. Even if this meant it was writing a little slower.

Now, it starts writing as soon as there is one frame captured, but the write command is issued in advance, for the entire contiguous memory area. Of course, subsequent frames will be captured while writing. As long as the file writing task is slower than frame capturing task, this trick will work.

(Don't confuse this with pre-buffering.)

Obviously, this technique is inherently unsafe. If, for some reason, the card controller starts writing 120% faster than we have measured, the risk is that we may save a frame that does not contain complete image data.

In theory, I can detect this situation, and I've added quite a few consistency checks. I did not encounter any lost frames yet.

Don't expect miracles from it. On my cards, I did not get any noticeable improvement, other than prettier buffer graphs (with large steps from the very beginning) and the mathematical proof that both startup and cruise "segments" are saved at the maximum speed possible with standard file I/O.

I expect it to work a little better on Ted's card, where yesterday's method didn't work very well.

And of course, since the optimization is a little unsafe, it's turned off by default.

Binary (same link): raw_rec.mo

N/A

  • Hero Member
  • *****
  • Posts: 576
  • Dreaming in 14 bit
Re: New writing strategy - variable buffering
« Reply #69 on: June 21, 2013, 02:02:48 PM »
600D, 1280x512 (640 mode), GD off, 410 frames.

:D
7D. 600D. Rokinon 35 cine. Sigma 30 1.4
Audio and video recording/production, Random Photography
Want to help with the latest development but don't know how to compile?

Redrocks

  • Member
  • ***
  • Posts: 151
  • 5D2 Windows
Re: New writing strategy - variable buffering
« Reply #70 on: June 21, 2013, 02:10:51 PM »
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
comparing A1ex's new build to the normal build I use (thanks a.d.)


A1ex: 1872 x 1250 @ 1:2

200
208
201
190
205

5x 2144 x 1078 @ 1:2

201
201
206
201
208

a.d.: 1880 x 1250 @1:2

174
174
174
174
174

5x 2152 x 1078 @1:2

185
184
185
184
185
__________________

A1ex: 1872 x 936 @2:1 (first unreliable ratio)

2077
2016
2429
2678
2171

5 x 2144 x 1072 @2:1 (first unreliable ratio)

211
209
207
207
211

a.d.: 1880 x 940 @2:1 (first unreliable ratio)

2061
2432
2289
2660
Battery gave out during final clip

5x 2152 x 1076 @2:1 (first unreliable ratio)

59
126
177
185
184
___________________

A1ex: 1872 x 1012 @1.85:1

674
738
676
725
679

5x 2144 x 1078 @1.85:1

207
200
208
201
208

a.d.: 1880 x 1016 @1.85:1

541
661
551
532
617

5x 2152 x 1078 @1.85:1

185
184
185
184
185
___________________

A1ex: 1872 x 1054 @16:9

475
488
466
470
468

5x 2144 x 1078 @16:9

204
205
206
203
208

a.d.: 1880 x 1058 @16:9

408
417
408
417
416

5x 2152 x 1078 @16:9

185
184
185
184
185

dariSSight#1

  • New to the forum
  • *
  • Posts: 47
  • Canon 5D Mark II
Re: New writing strategy - variable buffering
« Reply #71 on: June 21, 2013, 02:35:49 PM »
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
with A1ex's module from first page

1870 x 1250

200
208
201
190
205

5x 2144 x 1078

201
201
206
201
208

with current a.d. module on another Komputerbay

1880 x 1250

174
174
174
174
174

5x 2152 x 1078

185
184
185
184
185


Redrock are you checking for full ability with the 5D Mark II and we're getting at 1x with new module average 200 at 1870x1250, 5x 2144x1078? I will check my Lexar limitation and post back


Redrocks

  • Member
  • ***
  • Posts: 151
  • 5D2 Windows
Re: New writing strategy - variable buffering
« Reply #72 on: June 21, 2013, 03:48:23 PM »
Yes dariSSight, I aimed my first test at the maximum resolution and aspect ratio. I've since then updated the post with other aspect ratio tests.

dariSSight#1

  • New to the forum
  • *
  • Posts: 47
  • Canon 5D Mark II
Re: New writing strategy - variable buffering
« Reply #73 on: June 21, 2013, 04:46:22 PM »
5d2
Komputerbay 64GB 1000x
exact 24fps
Cinesco2 overlay (to simulate my typical usage)
comparing A1ex's new build to the normal build I use (thanks a.d.)


A1ex: 1872 x 1250 @ 1:2

200
208
201
190
205

5x 2144 x 1078 @ 1:2

201
201
206
201
208

a.d.: 1880 x 1250 @1:2

174
174
174
174
174

5x 2152 x 1078 @1:2

185
184
185
184
185
__________________

A1ex: 1872 x 936 @2:1 (first unreliable ratio)

2077
2016
2429
2678
2171

5 x 2144 x 1072 @2:1 (first unreliable ratio)

211
209
207
207
211

a.d.: 1880 x 940 @2:1 (first unreliable ratio)

2061
2432
2289
2660
Battery gave out during final clip

5x 2152 x 1076 @2:1 (first unreliable ratio)

59
126
177
185
184
___________________

A1ex: 1872 x 1012 @1.85:1

674
738
676
725
679

5x 2144 x 1078 @1.85:1

207
200
208
201
208

a.d.: 1880 x 1016 @1.85:1

541
661
551
532
617

5x 2152 x 1078 @1.85:1

185
184
185
184
185
___________________

A1ex: 1872 x 1054 @16:9

475
488
466
470
468

5x 2144 x 1078 @16:9

204
205
206
203
208

a.d.: 1880 x 1058 @16:9

408
417
408
417
416

5x 2152 x 1078 @16:9

185
184
185
184
185


Why has the resolution change from 1880 to 1872?

Redrocks

  • Member
  • ***
  • Posts: 151
  • 5D2 Windows
Re: New writing strategy - variable buffering
« Reply #74 on: June 21, 2013, 05:02:32 PM »
It's just a quirk of A1ex's build, I'm sure he will have a reason for the slight differences between his and a.d's resolutions. I actually set the resolution to 3584 for all the tests, but the numbers quoted are what the camera actually puts out.