550D/T2i raw video recording port official thread

Started by dlrpgmsvc, May 22, 2013, 12:11:45 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

funkysensation

Here is another Test video i did last Weekend in Copenhagen Denmark.



http://550draw.blogspot.de/

Ivoprod

Quote from: funkysensation on June 18, 2013, 03:37:32 PM
Here is another Test video i did last Weekend in Copenhagen Denmark.



http://550draw.blogspot.de/

Which build you use?
And What SD card ?
Also Interesting , moire is in very usable range, which lens you use?
5D Mark iii, Sandisk Extreme PRO 32 GB 1067x, Transcend 64GB 1000x,

funkysensation

I´m impressed too that moire is not that much visible. Try to shoot stuff like the shot at 01:45 you will get crazy with the intern H264. 
Lenses:
Zenitar Fisheye  16mm f2.8
Sigma 28-300mm

latest built I think
ML_RAW_MEM_FIX 

Sandisk Extreme Pro 95MB/s 32GB

a1ex

I've added a small buffer optimization to the raw rec simulator:

https://bitbucket.org/hudson/magic-lantern/commits/0db037b3b960

With the numbers from the example above:

1152x460 23.976fps 20.0MB/s => with current ML method, the simulation says 198 frames.
After optimization (only reordering and splitting the buffers differently) => 880 frames;

At 21.0 MB/s => 729 frames with current method, 2880 optimized.

Notice there's a slight difference from the previous numbers, because the latest script also models the slight speed loss from using small buffers.

Can you implement this buffering strategy in raw_rec and check if my theory is valid?

MaKsOZ


Kim.dh

Quote from: funkysensation on June 18, 2013, 03:37:32 PM
Here is another Test video i did last Weekend in Copenhagen Denmark.
http://vimeo.com/68607691

http://550draw.blogspot.de/
Nice! How do you avoid temporal aliasing in small details like leaves and roof tiles?

I got some footage of skerries covered with dried lichen (tiny details) which shows bad noise from temporal aliasing (I assume) in bright areas, which is driving me crazy trying to correct.

mk11174

This is the patch I am using in both Magic and Tragic module, nothing has changed since the first time we found out about the mem fix.
     /* try to recycle the waste */

-    if (waste >= 16*1024*1024 + 8192)

+    if (waste >= 8*1024*1024 + 8192)

     {

         buffers[buffer_count].ptr = (void*)(((intptr_t)(fullsize_buffers[0] + buf_size) + 4095) & ~4095);

         buffers[buffer_count].size = waste - 8192;

@@ -625,6 +625,17 @@
         buffer_count++;

         total += waste;

     }

+

+/* Sort Buffers */

+    static struct buff temp[10];

+    for (int i = 0; i < buffer_count; i++)

+ for(int j = 0; j < buffer_count- i - 1; j++)

+ if(buffers[j].size > buffers[j + 1].size) // > kisbuffer elöl

+            {

+                temp[j] = buffers[j];

+                buffers[j] = buffers[j + 1];

+                buffers[j + 1] = temp[j];

+            }


Tragic version is not from the main repo, that is from his own repo. So maybe something differ in the code that makes it work differ. Not sure.

I just use Magic version and it has not let me down yet, very happy with how it works considering our controller limitation, sometimes I get good frames, sometimes bad, I think it depends on the card condition at the time, seems to random to say it is in the code that causes it.

Maybe if you try without loading other modules to see if that makes a differ in performance, in this version I added the picview module too.

Go to the module menu and go to each module except Raw_Rec and press set on each and choose not to autoload them, then set the main autoload to on and restart camera so it only loads raw_rec.
500D/T1i  550D/T2i  600D/T3i  700D/T5i

JuanIrache

A1ex, if that patch works you'll have to tell us where to build you a statue.

In another order of things, would it be possible to keep the 10x crop for focusing purposes and avoid the camera crash? Something like ML turns Raw off the moment you go 10x and then turns it on when you're back to normal or 5x.

mk11174

not exactly sure if or what I should change unless the mem patch I mentioned in last post is already the best method which is the one we have been using.

Or am I suppose to change something else to try Alex's find?

I would be glad to test it if I knew what to change.

Do you simply want me to change if (waste >= 8*1024*1024 + 8192)  to if (waste >= 2 + 8*1024*1024)
500D/T1i  550D/T2i  600D/T3i  700D/T5i

mk11174

This version seems to run good for me, things I do with this one that helps is.
Turn Global Draw Off
Use FPS to set fps to 23.976 but 23 seems like it will go on forever
I was testing 1152x460
I use Canon Preview mode
And I used Magic Raw

But first try it with your normal settings you are use to and see if this version does better or worse.

If first record does not do good, try again and see how good next record does.

http://www.mediafire.com/download/25py137awp0s7x9/ML_550D_June_19_2013_Update.zip

UPDATE: After more tests I found this one was no better and recorded corrupted frames.
500D/T1i  550D/T2i  600D/T3i  700D/T5i

Rewind

These two last records doesn't work in photo mode.
But previous one was able to get as much as twice more frames compared to movie mode. What is the reason for that?

marekk

Quote from: mk11174 on June 18, 2013, 11:44:03 PM
This is the patch I am using in both Magic and Tragic module, nothing has changed since the first time we found out about the mem fix.
 
+ if(buffers[j].size > buffers[j + 1].size) // > kisbuffer elöl



Tragic Lantern raw_rec.c has  "if(buffers[j].size < buffers[j + 1].size)" .. It's better to sort it the other way ?

a1ex

Use the simulation script to find out.

Does that sorting algorithm actually work?! I can't understand it...

mk11174

Quote from: Rewind on June 19, 2013, 12:10:25 PM
These two last records doesn't work in photo mode.
But previous one was able to get as much as twice more frames compared to movie mode. What is the reason for that?
Raw record is disabled in photo mode, has nothing to do with the new versions here, it is just like that now.
500D/T1i  550D/T2i  600D/T3i  700D/T5i

mk11174

Quote from: a1ex on June 19, 2013, 12:29:29 PM
Use the simulation script to find out.

Does that sorting algorithm actually work?! I can't understand it...
Sorting code makes a huge differ on this camera. I added the sorting code to Magic Raw, it was already on the Tragic Raw
500D/T1i  550D/T2i  600D/T3i  700D/T5i

mk11174

Quote from: marekk on June 19, 2013, 12:17:45 PM
Tragic Lantern raw_rec.c has  "if(buffers[j].size < buffers[j + 1].size)" .. It's better to sort it the other way ?
To find out compare your results between Tragic and Magic, Magic uses it the opposite of Tragic.
500D/T1i  550D/T2i  600D/T3i  700D/T5i

xaint

Quote from: marekk on June 19, 2013, 12:17:45 PM
Tragic Lantern raw_rec.c has  "if(buffers[j].size < buffers[j + 1].size)" .. It's better to sort it the other way ?


Quote from: xaint on June 06, 2013, 05:33:29 AM

And I think it's better if the buffers are in ascending order (start writing ASAP):
/* Sort Buffers */
    static struct buff temp[10];
    for (int i = 0; i < buffer_count; i++)
        for(int j = 0; j < buffer_count- i - 1; j++)
           if(buffers[j].size > buffers[j + 1].size)
           {
                temp[j] = buffers[j];
                buffers[j] = buffers[j + 1];
                buffers[j + 1] = temp[j];
           }


(It seems it's no longer in the unified?)


Some test results here, on 550D with SanDisk ExtremePro 16GB, at 1280x426 @ 24fps without HaCKeD mode:






Buffers                 FramesSeconds
2x31Mb68<3
2x31Mb + 9Mb26511
9Mb + 2x31Mb27411.4

My version is probably not up to date, so take this into account!
And I don't know how it may affect the other models btw, need reviews...

xaint

Quote from: a1ex on June 18, 2013, 06:08:24 PM
Can you implement this buffering strategy in raw_rec and check if my theory is valid?

I'll check.

JuanIrache

Quote from: mk11174 on June 19, 2013, 11:47:29 AM
This version seems to run good for me, things I do with this one that helps is.
Turn Global Draw Off
Use FPS to set fps to 23.976 but 23 seems like it will go on forever
I was testing 1152x460
I use Canon Preview mode
And I used Magic Raw

But first try it with your normal settings you are use to and see if this version does better or worse.

If first record does not do good, try again and see how good next record does.

http://www.mediafire.com/download/25py137awp0s7x9/ML_550D_June_19_2013_Update.zip

UPDATE: After more tests I found this one was no better and recorded corrupted frames.

I'm getting pink frames with this one (*too). Never experienced this before, I think.

mk11174

Quote from: JuanIrache on June 19, 2013, 01:58:48 PM
I'm getting pink frames with this one. Never experienced this before, I think.
Same here, that is why I put an update about it having corrupt frames.
500D/T1i  550D/T2i  600D/T3i  700D/T5i

dlrpgmsvc

Quote from: mk11174 on June 19, 2013, 12:45:03 PM
Sorting code makes a huge differ on this camera. I added the sorting code to Magic Raw, it was already on the Tragic Raw

So, do this can be inserted in the raw rec module with a conditional compile statement only for 550d? If the conditionals are not implemented for this module, do we can implement them? So, we can take advantage of different strategies on different cameras! Let me know!
If you think it's impossible, you have lost beforehand

a1ex

Updated the simulation formula, now it's matches the real numbers perfectly for all test cases I've tried.

https://bitbucket.org/hudson/magic-lantern/commits/7eee493f6fc9

There's no best strategy for sorting; in some cases it's best to use biggest buffers first, in others, smallest buffers first. I think you will get better overall results if you call sim_frames and decide the sorting direction on the fly, right after allocating the buffers.

xaint

Quote from: a1ex on June 19, 2013, 02:44:17 PM
Updated the simulation formula, now it's matches the real numbers perfectly for all test cases I've tried.

https://bitbucket.org/hudson/magic-lantern/commits/7eee493f6fc9

There's no best strategy for sorting; in some cases it's best to use biggest buffers first, in others, smallest buffers first. I think you will get better overall results if you call sim_frames and decide the sorting direction on the fly, right after allocating the buffers.


To be honest :), I also thought, that the more buffers are better, even if the overall size isn't bigger, I don't know why I haven't tried so far: D, but here it is:

Some really quick test results here, on 550D with SanDisk ExtremePro 16GB, 1280x426 @ 24fps:












SortingBuffers                                               FramesSeconds
asc8Mb+31Mb+31Mb317~13
desc16Mb+16Mb+15Mb+15Mb+8Mb1326~55
asc8Mb+15Mb+15Mb+16Mb+16Mb1395~58
none15Mb+15Mb+16Mb+16Mb+8Mb1326~55

Try and let me know:
http://www.mediafire.com/download/rkq9kh63po8xxg4/ML_550D_Xaint.rar

Great findings by the way a1ex, I'm still experimenting with this, and with your speedsym.py script.. looks like it worth the try :) thanks!



EDIT (9 x 8Mb):







SortingBuffers                                                                         FramesSeconds
-8Mb+8Mb+8Mb+8Mb+8Mb+8Mb+8Mb+8Mb+8Mb1565~65

a1ex

Not always. Smaller buffers will reduce the write speed; the speed profile looks like this: http://www.magiclantern.fm/forum/index.php?topic=5471

Of course, the speed drop from 32M to 16M is very small, and the smaller granularity is useful here. But if the card is fast enough for continuous recording, it may work better with only the two 32MB buffers; the 8MB one may slow it down.

At startup and when the buffer is about to overflow, you want small buffers. In the middle of recording, when things are rolling smooth, you want large buffers. I'm investigating a writing strategy with variable buffer sizes, stay tuned.

xaint

Quote from: a1ex on June 19, 2013, 05:06:38 PM
I'm investigating a writing strategy with variable buffer sizes, stay tuned.

Me too :D