550D/T2i raw video recording port official thread

Started by dlrpgmsvc, May 22, 2013, 12:11:45 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

a1ex

The 9x8 looks close to optimal for this case. I've got only 1910 frames with the variable buffer method in simulation (compared to 1565 for 9x8MB).

A question for you: what write speed are you getting if you only write one frame at a time? (that is, split the buffers in very small chunks equal to frame_size - and as many as you can - should be 81 stars for the numbers above). This will be helpful in fine-tuning the speed curve.

Edit: current simulation results (1280x426, 23.976, speed tuned to 21.16MB/s to match 1565 frames):

8MB + 2x32MB - 317 frames:


9x8MB - 1566 frames:


Variable buffering, starting from 8MB + 2x32MB - 1910 frames:


X: time in seconds; Y: number of buffered frames.

All 3 cases with the same write speed and the same amount of total memory.

Warning: this is pure theory, I didn't try this on any camera. But you can see the mathematical background behind raw recording, study the algorithms and suggest improvements.

Note: if write speed wouldn't vary with buffer size, the optimal algorithm would be trivial: just write one frame at a time.

qsara

to my experience, sandisk extreme cards not only capable of writing sequentially but also in parallel. I think we can push more data into card with that strategy.

Kim.dh

This is pretty awesome. Would this mean a few seconds of 1080p is possible?

a1ex

No. At 1920x1080, the simulation says:
current method: 19 frames
optimized by splitting and reordering buffers: 27 frames
variable buffering: 25

At 1728x972: 24/34/34 frames.

MaKsOZ

Quote from: a1ex on June 19, 2013, 08:12:41 PM
No. At 1920x1080, the simulation says:
current method: 19 frames
optimized by splitting and reordering buffers: 27 frames
variable buffering: 25

At 1728x972: 24/34/34 frames.

And at 720p what is the max frames ?

Also if we shoot with the 2.35 ratio, how many frame @1920 ?

Thanks Alex :)

a1ex

1280x720: 50/96/101.
1728x736: 31/50/51.

Try Xaint's zip, see if the theory makes any sense.

qsara

Quote from: a1ex on June 19, 2013, 08:12:41 PM
No. At 1920x1080, the simulation says:
current method: 19 frames
optimized by splitting and reordering buffers: 27 frames
variable buffering: 25

At 1728x972: 24/34/34 frames.

just tried xaint's code, I can get

60 frames 1920x480
64 frames 1728x576
114 frames 1728x432
280 frames 1728x346
196 frames 1600x400
837 frames 1600x320
126 frames 1472x490
600 frames 1472x368
151 frames 1344x584
242 frames 1344x448
88 frames 1280x720
106 frames 1280x640
114 frames 1280x582
151 frames 1280x544
138 frames 1280x536
196 frames 1280x512
227 frames 1280x480
471 frames 1280x426
continuous recording 1280x320

all at 23.976fps with Sandisk Extreme 30MB/s 8GB

mk11174

500D/T1i  550D/T2i  600D/T3i  700D/T5i

Rewind

Yes, the method of 8Mb+15Mb+15Mb+16Mb+16Mb gives me 164 frames with 1280x536 at 23.976 fps for now. It's quite a step forward.

For example, recently i could get only 192 frames at 1152x482 at 24fps, now it is more than 730 (usually 739).

dougie

great job, now getting continuous shooting @ 1152x432 on a 64GB sandisk extreme pro sdxc 95MB/s and the last rar posted by Xaint

other numbers are:

190f @ 1152x576 2:1
295f @ 1152x524 2.20:1
544f @ 1152x490 2.35:1
310f @ 1344x448 3:1
164f @ 1344x504 2.67:1
126f @ 1344x538 2.5:1
114f @ 1344x562 2.39:1
114f @ 1344x562 2.39:1
114f @ 1344x572 2.35:1
106f @ 1344x610 2.20:1
80f @ 1344x672 2:1
62f @ 1344x756 16:9
80f @ 1280x720 16:9
151f @ 1280x544 2.35:1
1010 @ 1280x426 3:1
280 @ 1280x480 2.67:1
666 @ 960x576 5:3
280 @ 960x640 3:2

marekk

what should I change in raw_rec.c to allocate 16MB memory chunks instead of 32 ?

xaint

Quote from: a1ex on June 19, 2013, 07:36:11 PM
The 9x8 looks close to optimal for this case. I've got only 1910 frames with the variable buffer method in simulation (compared to 1565 for 9x8MB).

A question for you: what write speed are you getting if you only write one frame at a time? (that is, split the buffers in very small chunks equal to frame_size - and as many as you can - should be 81 stars for the numbers above). This will be helpful in fine-tuning the speed curve.

Edit: current simulation results (1280x426, 23.976, speed tuned to 21.16MB/s to match 1565 frames):

Warning: this is pure theory, I didn't try this on any camera. But you can see the mathematical background behind raw recording, study the algorithms and suggest improvements.

Note: if write speed wouldn't vary with buffer size, the optimal algorithm would be trivial: just write one frame at a time.


I think it's not that easy :D

With frame_size sized buffers, it says 19.8Mb/sec  (~1100 frames). So I think we can rule this theory out :)

As I can see it now, it not just depends on the buffer size, but also on the resolution.   ??? Here's how:

With 1152x460 the 1x8Mb + 4x16Mb setup gives better results:









SortingBuffers                    FramesSeconds
asc9x8Mb4371~182
asc1x8Mb + 4x16Mb5168~215

But in 1280x426, inversely, the 9x8Mb gives the better results:









SortingBuffers                    FramesSeconds
asc9x8Mb1524~63
asc1x8Mb + 4x16Mb1395~58

The write speeds are better with the 1x8Mb + 4x16Mb setup, in BOTH cases! It says constant 21.0Mb/sec...
With 9x8Mb however it says 20.8-20.9Mb/sec in both cases!

All tests were performed twice, gives almost the same results! (but there may be some randomness... :D )


Maybe I'm stupid, but isn't it possible, that the 16Mb buffers are faster than the 8Mb ones?
If I believe in this diagram:


I've lost track, help me out!!!  ;D  ;D ;D ;D ;D

EDIT: I think the write speed vary. A lot. :D

notdabod

Quote from: xaint on June 19, 2013, 04:34:39 PM

To be honest :), I also thought, that the more buffers are better, even if the overall size isn't bigger, I don't know why I haven't tried so far: D, but here it is:

Some really quick test results here, on 550D with SanDisk ExtremePro 16GB, 1280x426 @ 24fps:












SortingBuffers                                               FramesSeconds
asc8Mb+31Mb+31Mb317~13
desc16Mb+16Mb+15Mb+15Mb+8Mb1326~55
asc8Mb+15Mb+15Mb+16Mb+16Mb1395~58
none15Mb+15Mb+16Mb+16Mb+8Mb1326~55

Try and let me know:
http://www.mediafire.com/download/rkq9kh63po8xxg4/ML_550D_Xaint.rar

Great findings by the way a1ex, I'm still experimenting with this, and with your speedsym.py script.. looks like it worth the try :) thanks!



EDIT (9 x 8Mb):







SortingBuffers                                                                         FramesSeconds
-8Mb+8Mb+8Mb+8Mb+8Mb+8Mb+8Mb+8Mb+8Mb1565~65

http://www.mediafire.com/download/rkq9kh63po8xxg4/ML_550D_Xaint.rar]http://www.mediafire.com/download/rkq9kh63po8xxg4/ML_550D_Xaint.rar

With this new xaint I get twice the number of frames!

8Mb+15Mb+15Mb+16Mb+16Mb

was getting:

80 frames at 1152x460 with 23.976 fps with my Kingston sd card which is the same speed as the sandisk 45 mb/s.

now getting

162 frames at 1152x460 with 23.976 fps with my Kingston sd card which is the same speed as the sandisk 45 mb/s.

xaint

Quote from: marekk on June 19, 2013, 09:57:02 PM
what should I change in raw_rec.c to allocate 16MB memory chunks instead of 32 ?

I don't know whether I'm doing it well or not, a1ex will tell, but:

       
                buffers[buffer_count].ptr = (void*)(((intptr_t)ptr + 4095) & ~4095);
                buffers[buffer_count].size = size/2 - 8192;
                buffers[buffer_count].used = 0;

                buffers[buffer_count+1].ptr = (void*)(((intptr_t)ptr+size/2 + 4095) & ~4095);
                buffers[buffer_count+1].size = size/2 - 8192;
                buffers[buffer_count+1].used = 0;


                buffer_count+=2;
                total += size;

a1ex

Okay guys, got some homework for you.

Take a look here: http://www.magiclantern.fm/forum/index.php?topic=5471

and run the same test (Debug - Card buffer benchmark), but this time in movie mode. The camera will shutdown after half an hour, so touch the half-shutter every now and then. Use any of the autoexecs from this thread.

Then post the logs.

xaint

Quote from: a1ex on June 19, 2013, 10:24:22 PM
Okay guys, got some homework for you.

Take a look here: http://www.magiclantern.fm/forum/index.php?topic=5471

and run the same test (Debug - Card buffer benchmark), but this time in movie mode. The camera will shutdown after half an hour, so touch the half-shutter every now and then. Use any of the autoexecs from this thread.

Then post the logs.

As you wish sir!  :)

I think there is a lot of factors, that affects the write speed, including the card itself, the buffer distribution, the order of the buffers, the frame_size, (the varying cpu load maybe?) and so on.. So good luck a1ex to find the best solution, what works best in all cases! :)
I'll try to help if I can! :)

qsara

I've noticed sth about sandisk extreme 45mb/s cards. after 6 months of intense use write rates drops about 1mb/s, because I've recently bought 3 new 45MB/s cards and they are all faster than my older 2 45MB/s cards. I'm surprised to see the result, because I also have a sandisk extreme III 20MB/s edition 6 yr old, works at same performance all along.

anyone noticed sth like that?

and @a1ex, is it possible to partition eos m's memory like that?

marekk

60D hangs with this ;)

Quote from: xaint on June 19, 2013, 10:12:45 PM
I don't know whether I'm doing it well or not, a1ex will tell, but:

       
                buffers[buffer_count].ptr = (void*)(((intptr_t)ptr + 4095) & ~4095);
                buffers[buffer_count].size = size/2 - 8192;
                buffers[buffer_count].used = 0;

                buffers[buffer_count+1].ptr = (void*)(((intptr_t)ptr+size/2 + 4095) & ~4095);
                buffers[buffer_count+1].size = size/2 - 8192;
                buffers[buffer_count+1].used = 0;


                buffer_count+=2;
                total += size;


dougie

Quote from: notdabod on June 19, 2013, 10:09:32 PM

With this new xaint I get twice the number of frames!

8Mb+15Mb+15Mb+16Mb+16Mb

was getting:

80 frames at 1152x460 with 23.976 fps with my Kingston sd card which is the same speed as the sandisk 45 mb/s.

now getting

162 frames at 1152x460 with 23.976 fps with my Kingston sd card which is the same speed as the sandisk 45 mb/s.


That's strange, I get 2242 frames @ 1152x460 23.976fps on a sandisk 95MB/s but the real bottle neck is the ~20MB/s 550d SD hw

marregi

Did my homework :)
Buffer size experiment
ML v2.3.NEXT.2013Jun19.550D109, f3631da68349+
Mode: movie 1920x1080 24p, Global Draw: OFF
Sandisk 16GB 45MB/s

10312704 195
15885312 202
80896 69
8949760 201
28661760 204
16264192 205
24521728 200
16424960 203
13148160 201
15619072 202
1658880 188
21001216 201
8548352 203
19760128 202
5965824 196
24906752 202
13968384 201
16770048 200
4542464 197
13063168 205
33263616 202
30312448 202
16713728 203
3887104 198
21841920 202
12846080 204
16192512 205
3333120 197
6719488 197
17313792 206
23374848 201
2865152 192
32488448 200
18588672 205
4186112 199
821248 179
3661824 200
18359296 201
19875840 201
22754304 205
13665280 202
24957952 203
28555264 203
33442816 205
10708992 200
20242432 203
8774656 199
11216896 203
23156736 201
12750848 201
12369920 204
2157568 188
15279104 203
29943808 200
30321664 204
19004416 206
268288 129
23778304 204
467968 152
1905664 194
32368640 204
19829760 203
26281984 200
24153088 204
11364352 202
32160768 204
5944320 199
24118272 204
28511232 204
31526912 205
15956992 203
1732608 193
13881344 205
6106112 198
17367040 204
21693440 200
30835712 201
1139712 177
13698048 205
19388416 203
1532928 186
16333824 203
15963136 202
23540736 205
23770112 205
7317504 201
6019072 201
13303808 202
32039936 204
13137920 199
24311808 201
25442304 204
8986624 202
25421824 204
2720768 191
18526208 202
12448768 202
6514688 199
6632448 199
17897472 202
26569728 205
22568960 205
11912192 200
22007808 204
30651392 202
29615104 204
18367488 203
21049344 201
15778816 203
24951808 202
5359616 199
23476224 201
17125376 201
3318784 194
12782592 204
30965760 202
1974272 194
32021504 206
24463360 202
28246016 206
23352320 204
27161600 203
25191424 205
10841088 203
22846464 202
19111936 201
26234880 204
32610304 202
14385152 202
18029568 203
11080704 201
3933184 197
22414336 201
8314880 200
27988992 204
20073472 205
23487488 203
30233600 204
25299968 202
2406400 193
28059648 204
2145280 190
27503616 201
20705280 205
28731392 202
2458624 191
19662848 204
6198272 199
21655552 205
32225280 201
21788672 202
16669696 202
12007424 202
4321280 196
14878720 204
10118144 199
2836480 191
21092352 202
17086464 202
1707008 192
17509376 205
28161024 204
18457600 201
16290816 201
10758144 203
17766400 203
28011520 203
19835904 201
11879424 203
18191360 202
12581888 201
15906816 205
6525952 199
30545920 200
934912 178
6738944 201
18920448 205
27171840 203
19916800 206
28598272 203
23926784 202
10470400 204
20392960 201
7282688 202
4824064 197
11820032 201
30988288 203
26221568 206
13527040 204
30140416 203
6785024 200
25762816 201
23324672 203
26011648 203
4475904 198
33500160 206
8732672 202
29461504 202
12411904 202
12294144 203
11481088 200
12786688 199
15858688 205
16142336 204
29415424 205
32114688 202
18460672 203
15672320 203
12832768 201
23447552 203
18734080 201
25832448 202
20690944 202
27086848 206
24894464 203
32919552 203
30015488 204
2870272 191
27643904 203
25778176 202
4189184 201
13642752 201
7861248 200
21999616 206
4232192 198
12140544 200
3162112 197
27608064 203
18791424 205
5019648 200
15958016 202
20426752 206
685056 164
18709504 202
6188032 201
6668288 203
30684160 200
9438208 204
20264960 205
1707008 192
4702208 202
27417600 201
31681536 204
1326080 183
9743360 201
29618176 204
28645376 201
21629952 207
21515264 211

mk11174

WOW!!! 1725 frames at 1280x426x23.976  Very nice!!!
500D/T1i  550D/T2i  600D/T3i  700D/T5i

1%

I did this to 50D and write speed fell... but idle time did too.

Well thing is size/2 makes ~13mb buffers so maybe having them at 16MB would be ideal.

xaint

Quote from: 1% on June 19, 2013, 11:58:02 PM
I did this to 50D and write speed fell... but idle time did too.

Well thing is size/2 makes ~13mb buffers so maybe having them at 16MB would be ideal.

What do you mean that size/2 makes ~13Mb buffers? How big the size then? I don't know the 50D, in 550D we have 2x31Mb, and the remaining 7-8Mb from the 75Mb shoot_malloc.

xaint

Homework :D

(The cam overheated, sorry. Today was the hottest day of the summer)

Mode: movie 1920x1080 25p, Global Draw: OFF
SanDisk ExtremePro 16GB



10312704 207
15885312 208
80896 86
8949760 207
28661760 208
16264192 206
24521728 209
16424960 207
13148160 208
15619072 206
1658880 195
21001216 208
8548352 206
19760128 208
5965824 204
24906752 207
13968384 207
16770048 207
4542464 202
13063168 206
33263616 208
30312448 208
16713728 208
3887104 203
21841920 206
12846080 207
16192512 207
3333120 202
6719488 204
17313792 207
23374848 206
2865152 198
32488448 208
18588672 207
4186112 203
821248 185
3661824 204
18359296 208
19875840 208
22754304 208
13665280 206
24957952 208
28555264 206
33442816 208
10708992 204
20242432 207
8774656 206
11216896 204
23156736 208
12750848 205
12369920 209
2157568 195
15279104 205
29943808 209
30321664 206
19004416 203
268288 145
23778304 209
467968 165
1905664 198
32368640 209
19829760 206
26281984 207
24153088 207
11364352 205
32160768 207
5944320 206
24118272 207
28511232 209
31526912 207
15956992 208
1732608 196
13881344 208
6106112 204
17367040 209
21693440 207
30835712 209
1139712 188
13698048 203
19388416 208
1532928 195
16333824 205
15963136 208
23540736 207
23770112 208
7317504 206
6019072 204
13303808 206
32039936 207
13137920 206
24311808 207
25442304 208
8986624 206
25421824 208
2720768 200
18526208 209
12448768 206
6514688 203
6632448 205
17897472 206
26569728 206
22568960 208
11912192 206
22007808 206
30651392 209
29615104 208
18367488 207
21049344 206
15778816 206
24951808 208
5359616 200
23476224 202
17125376 208
3318784 201
12782592 205
30965760 209
1974272 198
32021504 209
24463360 205
28246016 208
23352320 203
27161600 208
25191424 206
10841088 206
22846464 208
19111936 207
26234880 207
32610304 209
14385152 207
18029568 208
11080704 205
3933184 201
22414336 208
8314880 204
27988992 209
20073472 207
23487488 208
30233600 205
25299968 208
2406400 196
28059648 208
2145280 196
27503616 207
20705280 208
28731392 207
2458624 198
19662848 208
6198272 204
21655552 208
32225280 206
21788672 207
16669696 206
12007424 205
4321280 201
14878720 207
10118144 205
2836480 200
21092352 207
17086464 205
1707008 194
17509376 205
28161024 208
18457600 206
16290816 207
10758144 204
17766400 204
28011520 208
19835904 202
11879424 205
18191360 208
12581888 203
15906816 208
6525952 203
30545920 206
934912 186
6738944 204
18920448 204
27171840 208
19916800 207
28598272 207
23926784 207
10470400 206
20392960 206
7282688 201
4824064 202
11820032 207
30988288 205
26221568 204
13527040 207
30140416 204
6785024 202
25762816 207
23324672 205
26011648 205
4475904 200
33500160 205
8732672 206
29461504 207
12411904 203
12294144 207
11481088 205
12786688 205
15858688 206
16142336 203
29415424 206
32114688 208
18460672 205
15672320 206
12832768 207
23447552 206
18734080 206
25832448 206
20690944 207
27086848 206
24894464 208
32919552 207
30015488 208
2870272 197
27643904 208
25778176 206
4189184 202
13642752 204
7861248 203
21999616 202
4232192 202
12140544 206
3162112 201
27608064 208
18791424 204
5019648 203
15958016 208
20426752 207
685056 174
18709504 206
6188032 204
6668288 205
30684160 207
9438208 204
20264960 206
1707008 192
4702208 204
27417600 208
31681536 206
1326080 192
9743360 207
29618176 206
28645376 208
21629952 206
21515264 206
12170240 205
29555712 206
2771968 198
30593024 209
23996416 206
31484928 208


intrinsic

Is this buffering aspect has anything to do with increasing card write speed at lower FPS using FPS override? Or is it CPU problem?
I've first noticed that on 5dMkII, and 550D shows the same, it goes up to 22Mb/s at the beginning of the recording (than drops around 21), @FPS 8-12