Investigating cluster size in relation to card performance

[Shield]

Quick background - in the last week I've purchased three 64GB 1000x KomputerBay cards.
First one didn't work in the camera, but was fine in the computer.
Second one is fine and works as expected.
Third one works but is far slower, and only does around 60MB/s.

In my external quest for fast enough cards I am going to run various tests with benchmark results, and show the effect (if any) on the cluster size of the formatted cards.  One would think a large cluster size would be fine since it's only going to store very large data files, even if not super efficient for small files.

Example -

Card B:

Formatted for 16384 cluster size:

1920x1080 = 73, 73, 91 frames before crash
1880x1058 = 73, 121, 127 frames before crash
1720x968 = ran indefinitely

I will post screen captures too of known "good" cards, like my 32GB Lexar and the sole remaining Komputerbay that works.

So, it's really a crapshoot with the Komputerbay cards if you want 1920x1080 @ 24p.

[Shield]

I will also make the test somewhat scientific and do the same steps each time.  For example, pull the battery between tests, same ML build (May 27) and same ML settings for each (Globaldraw ON).

If anything we might find out that a specific cluster size can get someone over the hump for their desired resolution.  Plus, I'm ticked that 2 out of 3 cards were not as advertised.
Shawn

[Audionut]

I've got a 16GB Lexar 1000x CF.

It was formatted for 512kb cluster size as I wanted to not waste space with photos.  I was getting around 70 frames @ 1920x1080 before skipping.

I formatted to 16384 cluster size and got 660 frames before frame skipping.

Note:  The 16GB Lexar's are not as fast as the 36GB versions.

Reducing the aspect ratio to 1.85:1 is just enough to have continuous recording

Looking forward to your results Shield.

[noisyboy]

Niiiiiice 

[noix222]

any idea how can i format my card like that on a mac ? thanks guys.

[Shield]

I will post all the numbers here shortly.  This is boring as hell, but I'm too far into it to stop.
Keep in mind I'm running these tests on the "slow" 1000x Komputerbay card.

Here's some of the anecdotal results so far:

(3 tests each per resolution; test is until card skipped a frame)
Formatted for 4096 bytes cluster size:
1920x1080 = 100, 136, 100 frames.
1880x1058 = 154, 172, 163 frames
1720x968 = ran indefinitely

Formatted for 8192 bytes cluster size:
1920x1080 = 100, 63, 99 frames.
1880x1058 = 163,100, 172 frames
1720x968 = ran indefinitely

Formatted for 16 kilobytes cluster size:
1920x1080 = 45, 126, 136 frames.
1880x1058 =  117, 154, 163 frames
1720x968 = ran indefinitely

Formatted for 32 kilobytes cluster size:
1920x1080 = 46, 136, 145 frames.
1880x1058 = 172, 163, 172 frames
1720x968 = ran indefinitely

Formatted for 64 kilobytes cluster size:
1920x1080 =  145, 145, 109 frames.
1880x1058 =  172, 172, 181 frames
1720x968 = ran indefinitely

Formatted for 128 kilobytes cluster size:
1920x1080 =  109 136 100 frames.
1880x1058 = 172,172,172 frames
1720x968 = ran indefinitely

Formatted for 256 kilobytes cluster size:
1920x1080 =  136, 136, 100 frames.
1880x1058 =  163, 163, 162 frames
1720x968 = ran indefinitely

[Shield]

When these tests are done, I'm going to record perhaps the longest raw 1920x1080 5d3 movie ever done - a 63 GB one.  If the footer doesn't save I'm going to have to figure it out for the 64 GB size, but I have all the other footer sizes covered in my other thread.

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

[Shield]

Well I've saved all the screenshots and was going to get creative and do something in Excel.

But, the results are pretty much the same.  At least in Windows, with a 64GB card, use the default allocation size (128 kilobytes).

I will say performance got worse the larger the cluster size beginning @ 2048 kilobytes.

What a waste of an evening.

[Shield]

For a point of reference, here's the Lexar 32GB vs the "good" Komputerbay 64GB (both 1000x) :

(I realized I had done the test on the Komputerbay @ 5x zoom mode, but I'm very sick of testing and will not re-do it);

Lexar 32 GB:




Komputerbay 64GB card:

[Shield]

When these tests are done, I'm going to record perhaps the longest raw 1920x1080 5d3 movie ever done - a 63 GB one.  If the footer doesn't save I'm going to have to figure it out for the 64 GB size, but I have all the other footer sizes covered in my other thread.

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

Well the file size of the most boring movie of my desk and phone have been recorded.  1920x1080, 5d3.  Filesize is 62,467,329 KB, aka 62.4 GB.
Even with USB3 this is taking about 8 minutes to transfer over to a very fast disk array.
My guess is about 12:30 of footage.  Also, it shut off by itself; I'm curious if there's now a routine that auto-saves the footer.  I noticed in today's build (May 28th P.M. Lorenco's) that there's no longer an option to "enable over 4GB".  It just worked.  Excited to see how long the dng extraction will take on a 3.4GHZ overclocked i7 920....

[Shield]

Footer was AUTO saved!  17467 dng files to process.

THANK YOU MAGIC LANTERN!

[N/A]

Imax cameras only hold 3 minutes of 70mm film and take ~20 minutes to unload, so it could be worse 

[Audionut]

For a point of reference, here's the Lexar 32GB vs the "good" Komputerbay 64GB (both 1000x) :

The speeds are slow for a 32GB Lexar.  Have you aligned the partition?

http://www.sevenforums.com/tutorials/113967-ssd-alignment.html

[Shield]

The speeds are slow for a 32GB Lexar.  Have you aligned the partition?

http://www.sevenforums.com/tutorials/113967-ssd-alignment.html

Nope.  It's never let me down on 1920x1080 though.  If it starts too I'll look into it.

Any idea why one of my KomputerBay cards works in a pc but the camera can't read it?

[Audionut]

Nope.  It's never let me down on 1920x1080 though.  If it starts too I'll look into it.

That is strange.  My card benches slightly faster then yours, but I cannot maintain 1920x1080.



With changes to alignment and cluster size, I can do 1920x1036 though.

Any idea why one of my KomputerBay cards works in a pc but the camera can't read it?

No sorry.  You could try totally deleting the partition table and re-formatting.

[squig]

Turn global draw off!

[squig]

any idea how can i format my card like that on a mac ? thanks guys.

Wish I knew. Probably have to run wine AGAIN! 

[Audionut]

Turn global draw off!

In my case for this card, it made a difference of +-2MB/s.  I always have it off for RAW recording though.

[squig]

My Toshiba 64gb 1066x benches a couple of MB/s lower than the Lexar but it doesn't drop any 1080p frames.

[Shield]

Turn global draw off!

Now why in the world would I do that, considering I shoot with it on (focus peaking, histogram, zebras?).  It has to pass the "real world tests" for me; I can't shoot FF at larger apertures without some focus peaking.  I had already started the tests anyway with GD on.

[Audionut]

I think I've nailed it down to the card settings.  I changed from Record func. Auto switch card - to Standard and I can now record 1920x1080.  The buffer was still peaking hard on the third buffer indicator on first record and stayed stuck on the second indicator on a second recording.

[Shield]

That is strange.  My card benches slightly faster then yours, but I cannot maintain 1920x1080.



With changes to alignment and cluster size, I can do 1920x1036 though.

No sorry.  You could try totally deleting the partition table and re-formatting.

Tried that still nuttin...

Also I tried your align=1024 trick and my benchmarks got a bit slower.  No idea why.  I've shots hours upon hours @ 1920x1080 with the Lexar and never missed a beat - shot a full 32GB in the hot sun yesterday with focus peaking, histogram, global draw galore.  Perhaps the benchmarks aren't 100% accurate?  Maybe the data is being laid down on the disk in a manner that's not exactly 2048/1953 or any of the other buffer sizes?  Based on your last screenshot, there's just no way you shouldn't be able to do 1920x1080.

Here's another question for you - are you recording audio as a seperate wav?  Stick a generic SD card in the other slot; maybe that's your problem.  It'll automatically record the .wav to the other card.  I cannot do 1920x1080 + audio JUST on the CF either.

[Shield]

I think I've nailed it down to the card settings.  I changed from Record func. Auto switch card - to Standard and I can now record 1920x1080.  The buffer was still peaking hard on the third buffer indicator on first record and stayed stuck on the second indicator on a second recording.

Still sounds odd - see my last post.  Here's what mine does- it will do 2 ** for about 2-3 seconds at the beginning of the recording, then drop to a single * and never move.  On both the 64 Kompu-Serve card and the Lexar.  Exact same way.

[Audionut]

I don't do audio, I get the wav files but every software I own opens them up empty.

Turns out all of my problems are probably from only gauging speed on the first record.  Following recordings go much better.

[Shield]

I don't do audio, I get the wav files but every software I own opens them up empty.

Turns out all of my problems are probably from only gauging speed on the first record.  Following recordings go much better.

I find it funny that "Audionut" doesn't do audio.
Go into the camera menu (not ML) and you'll see that audio has probably been disabled.  Happened to me when I loaded the last ML build; now the wav files actually have content.
Shawn