Magic Lantern Forum

Developing Magic Lantern => General Development Discussion => Topic started by: a1ex on July 02, 2014, 08:55:00 AM

Title: SRM job memory buffers
Post by: a1ex on July 02, 2014, 08:55:00 AM
Interesting find: if you call FA_CreateTestImage more than once, you need to call FA_DeleteTestImage in the same order (otherwise it will fail):

So, this is OK:
Code: [Select]
    struct JobClass * job1 = (void*) call("FA_CreateTestImage");
    struct JobClass * job2 = (void*) call("FA_CreateTestImage");

    call("FA_DeleteTestImage", job1);
    call("FA_DeleteTestImage", job2);

but this will fail:
Code: [Select]
    struct JobClass * job1 = (void*) call("FA_CreateTestImage");
    struct JobClass * job2 = (void*) call("FA_CreateTestImage");

    call("FA_DeleteTestImage", job2);
    call("FA_DeleteTestImage", job1);

Another nice trick: each job object has a contiguous raw image buffer, usually more than 32MB. With shoot_malloc, the maximum contiguous size is less than 32MB (depending on your camera). To access this buffer:

Code: [Select]
    struct JobClass * job = (void*) call("FA_CreateTestImage");
    void* buf = (void*) call("FA_GetCrawBuf", job);
   
    /* use your buffer */

    call("FA_DeleteTestImage", job);

So, it looks like we have a way to allocate some more memory, which is very useful for raw video. How much, I don't know yet.
Title: Re: SRM job memory buffers
Post by: a1ex on July 02, 2014, 10:31:20 AM
The underlying call for memory allocation is SRM_AllocateMemoryResourceFor1stJob (called from CreateSkeltonJob, easy to find from strings). To double-check, its second argument has an offset that matches the one from GetImageBuffer.

To free the memory, call SRM_FreeMemoryResourceFor1stJob (it's called from DeleteSkeltonJob, but has no strings - identify it from its first argument).

Example for 5D2:

GetImageBuffer returns (arg0->off_0x4)->off_0xC68, aka MEM(MEM(arg0 + 0x4) + 0xC68).

Code: [Select]
SRM_AllocateMemoryResourceFor1stJob(cbr_function, 0xC68 + something)
if take_semaphore(*0x3CC0, 0x64) != 0:
    DebugMsg(0, 22, "[JOB ERROR] SRM_AllocateMemoryResourceFor1stJob failed [%#x]', ...)

Code: [Select]
if (arg0->off_0x4)->off_0xC68 != 0:
    SRM_FreeMemoryResourceFor1stJob((arg0->off_0x4)->off_0xC68, 0, 0)

Notes and caveats:
- you don't need to enter factory mode to use this
- this is independent from shoot_malloc (you can use both, and using one doesn't affect the size or free space from the other)
- you can't tell how much to allocate; the buffer size is fixed (it tells you how much you've got)
- as long as you have one such buffer allocated, you can't take regular pictures (ERR70), but LiveView works fine
- if there's no more free RAM, the allocation will be delayed until you free something failed calls will be dropped, just wait for 100ms
- on 5D3, you can allocate 3 such buffers at a time (39MB x 3 = 117MB extra). On 5D2 and 550D, only 2.

Syntax:
Code: [Select]
void SRM_AllocateMemoryResourceFor1stJob(void (*callback)(uint32_t opaque, void* raw_buffer, uint32_t raw_buffer_size), uint32_t opaque);
void SRM_FreeMemoryResourceFor1stJob(void* raw_buffer, uint32_t unk1_zero, uint32_t unk2_zero);
Title: Re: SRM job memory buffers
Post by: jimmyD30 on July 02, 2014, 12:17:48 PM
@A1ex Reply #22

Strange that the order of deleting 'jobs' make a difference, I guess it relates to how the firmware prefers to manages memory?

Anyway... Mo' memory for raw video = :D
Title: Re: SRM job memory buffers
Post by: a1ex on July 02, 2014, 12:23:12 PM
Yeah, that was unexpected. The resource manager is quite complex; maybe g3gg0 will have a better understanding of why this happens.
Title: Re: SRM job memory buffers
Post by: Levas on July 02, 2014, 07:20:49 PM

- on 5D3, you can allocate 3 such buffers at a time (39MB x 3 = 117MB extra). On 5D2 and 550D, only 2.

Alex, are these 39MB buffers, additional on the already discovered memory buffers, used for fill-up with raw recording ?
The 6D has about 250MB of memory buffer with raw-recording at the moment.
The 250MB did already sound like the max, according to the point when burst mode slows down with raw photo mode.
raw images are like 35MB and the SD can write 40MB/s to the card.
Although there are reports of 9 seconds of writing to the SD card time to clear the buffer (with fast UHS-I card)in the 6D, so 40MB x 9 seconds is about 360MB of available buffer.

So there are probably 3 x 39MB buffers on the 6D. Would love to have these extra 117 MB for raw recording  :D
Title: Re: SRM job memory buffers
Post by: g3gg0 on July 02, 2014, 09:50:43 PM
- on 5D3, you can allocate 3 such buffers at a time (39MB x 3 = 117MB extra). On 5D2 and 550D, only 2.

i made some changes to mlv_rec to support this new memory interface to make some tests.
it looks very promising. here some results:

5D3 v1.1.3:
Memory details: total 277 MiB [exmem 238 MiB, SRM 39 MiB] in 89 slots

7D v2.0.3:
Memory details: total 371 MiB [exmem 182 MiB, SRM 188 MiB] in 164 slots

600D v1.0.3:
Memory details: total 129 MiB [exmem 67 MiB, SRM 62 MiB] in 44 slots

this is another powerful push. thanks a lot, alex :)

Title: Re: SRM job memory buffers
Post by: Levas on July 02, 2014, 11:10:22 PM
@Geggo,

Is there a way to test the changes you made to the mlv_rec module on a 6D (without the need to compile on my side  :-\) ?
At the moment, 6D is not supported in nightly builds...  :'(
I believe it's not as simple as put the new mlv_rec.mo file in an older nightly build, did tried that once, but it gives errors at camera startup.
Would love to have some extra memory with MLV recording  :)
Title: Re: SRM job memory buffers
Post by: g3gg0 on July 02, 2014, 11:23:08 PM
hm i dont have a recent 6D firmware dump i could use to find the stubs.
Title: Re: SRM job memory buffers
Post by: a1ex on July 02, 2014, 11:25:38 PM
These changes are not yet included in any nightly build (we are reverse engineering it as we speak, and sharing the details as soon as we discover them). Please be patient, and hope this will motivate the 6D developers to bring the port up to date.

Some quick tests on 5D3 123, first outside LiveView, second in LV:
(http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3.png) (http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3-lv.png)
Title: Re: SRM job memory buffers
Post by: N/A on July 02, 2014, 11:49:46 PM

7D v2.0.3:
Memory details: total 371 MiB [exmem 182 MiB, SRM 188 MiB] in 164 slots

:o

Such amaze. Much wow.
Title: Re: SRM job memory buffers
Post by: nikfreak on July 03, 2014, 08:18:05 AM
hm i dont have a recent 6D firmware dump i could use to find the stubs.

Here you are. These are for 6D_113:

Code: [Select]
NSTUB(0xFF0F0098,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0F40C8,  SRM_FreeMemoryResourceFor1stJob)
Title: Re: SRM job memory buffers
Post by: a1ex on July 03, 2014, 08:43:18 AM
Thanks, you will probably have to find a few more tonight :)
Title: Re: SRM job memory buffers
Post by: nikfreak on July 03, 2014, 08:59:22 AM
Thanks, you will probably have to find a few more tonight :)

No problem I am continuously watching commits on butbucket...
Title: Re: SRM job memory buffers
Post by: Levas on July 03, 2014, 09:24:38 AM
Silent picture discovery is huge.
But at this moment I'm way more excited about the possibility of getting some extra memory for MLV_rec  :P

Use MLV_rec a lot for high resolution short clips, these extra MB's mean a lot of extra seconds  :D


Title: Re: SRM job memory buffers
Post by: a1ex on July 03, 2014, 12:17:41 PM
Okay, here's an attempt to create some friendly wrappers for the new buffers: https://bitbucket.org/hudson/magic-lantern/pull-request/529

User-level API:
- srm_malloc_suite/srm_free_suite: will allocate the entire SRM memory (size autodetected) and return a memory suite, just like shoot_malloc_suite (preferred for raw_rec/mlv_rec)
- malloc/free: will use these buffers when you request very large sizes that can't be handled by shoot_malloc (32MB+); these are one level above srm_malloc_suite
- srm_malloc/srm_free: will use these buffers for any request higher than 25MB (even if the same buffers can be allocated from shoot_malloc)

Caveats:
- once somebody allocated from the SRM buffer, nobody else will be allowed to do so, and you will not be able to take pictures (this is because the backend must free the buffers in exactly the same order as allocated)
- you can, however, use malloc/free from two diferent tasks (first malloc will allocate all the SRM buffers, last free will give all buffers back to the system)
- with malloc, you can free the buffers in any order (the backend will take care of it)
- be careful allocating SRM buffers outside LiveView: you may not be able to enter LiveView until you free them
- depending on camera, some buffer sizes like 27MB may fail with malloc (because it will try shoot_malloc first, and it doesn't report the free space correctly [fixme]); workaround: srm_malloc will work.

For short-term processing that requires huge amounts of RAM, or for RAW recording, these wrappers should be just fine.

Porting notes:
- Stubs required: SRM_AllocateMemoryResourceFor1stJob, SRM_FreeMemoryResourceFor1stJob, CreateMemorySuite, DeleteMemorySuite, CreateMemoryChunk, AddMemoryChunk.
- Check the Free Memory dialog and post a screenshot (I already did for 5D3 123 earlier in this thread).
- Run the test from "don't click me" and post the screenshot created there.
- While this test is running, try taking some pictures (it should not crash, but the allocator may fail - it's normal, it should fail gracefully).

Some more thoughts:
g3gg0 is researching a way to merge two memory suites, so you would get the entire memory allocated in a single suite. With this, there would be - in theory - no need to modify the raw recording code, it would work out of the box.

However, if you enable the memory hack, it will allocate the memory outside LiveView and then get back into LiveView. The problem with this is that, if you allocate the entire RAM outside LV, the camera may not have enough RAM for the LiveView tasks (it happens on 5D3). So, I'd keep two separate allocators for now, and the user code will be responsible for having the proper context before calling (e.g. don't switch modes after allocating).
Title: Re: SRM job memory buffers
Post by: Audionut on July 03, 2014, 12:45:11 PM
Nice work.  Looking forward to see what can be done with all this extra RAM.  :)
Title: Re: SRM job memory buffers
Post by: nikfreak on July 03, 2014, 01:52:13 PM

So now see following. These are my findings for 6D_113:

Code: [Select]
NSTUB(0xFF0F0098,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0F40C8,  SRM_FreeMemoryResourceFor1stJob)
NSTUB(0xFFCCE714 - RAM_OFFSET,  CreateMemoryChunk)
NSTUB(0xFFCCF210 - RAM_OFFSET,  AddMemoryChunk)

If you need others then just say so....
Title: Re: SRM job memory buffers
Post by: kyrobb on July 03, 2014, 07:10:47 PM
I see that this has already been added to nightlies for the 7D, 5D3 and 600D. Any news on when the 550D can be tested?
Title: Re: SRM job memory buffers
Post by: a1ex on July 03, 2014, 07:35:20 PM
Nope, it wasn't (g3gg0 added the stubs without creating a branch first, and that triggered the nightly builds without any change in functionality)

Another note for raw recording: calling SRM_ChangeMemoryManagementForFactory might make a difference.

For example, for 550D, in LiveView, you get 75 + 1x31 MB in normal mode, and 43 + 3x31 in factory mode. 30MB extra is not bad, no?
On 5D2, in LiveView, you get 183 + 4x37 in normal mode, and 259 + 2x37 in factory mode (same with jpeg-small). No difference.
On 5D3, you get 163 + 3x39 in normal mode, and 127 + 4x39 in factory mode. No difference.

This might remove the need for picture quality tricks, at least on some cameras. Need to double-check if this is a persistent setting though.

(I'll let you continue the research during the next few weeks - sorry, family needs a little more attention, and it will keep me offline for a while)
Title: Re: SRM job memory buffers
Post by: dmilligan on July 04, 2014, 03:24:27 AM
1100D, I'm fairly certain these are right:
Code: [Select]
// NSTUB(    ???,  CreateMemorySuite)
NSTUB(0xFF071BD0,  DeleteMemorySuite)
// NSTUB(    ???,  CreateMemoryChunk)
NSTUB(0xFF072040,  AddMemoryChunk)

/** SRM JobMem **/
NSTUB(0xFF029268,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02BCF0,  SRM_FreeMemoryResourceFor1stJob)

I'm having a little trouble with CreateMemorySuite and CreateMemoryChunk, I have some leads:
Code: [Select]
NSTUB(0xFF0715BC, CreateMemoryChunk)
NSTUB(0xFF071D9C, CreateMemorySuite)
but I'm not sure if these are right, or how to verify them, any hints?

sub_FF071D9C+20:   _malloc(24)
sub_FF071D9C+40:   sub_FF0142D8('pSuite', 'PackMemory\\PackMem.c', 554)
sub_FF071D9C+100:   sub_FF0715BC(arg0, arg1, arg2)


sub_FF0715BC+20:   _malloc(40)
sub_FF0715BC+40:   sub_FF0142D8('pChunk', 'PackMemory\\PackMem.c', 244)
Title: Re: SRM job memory buffers
Post by: dmilligan on July 04, 2014, 03:54:46 AM
(https://bitbucket.org/dmilligan/magic-lantern/downloads/FreeMemory1100D.jpg)

the test fails but I guess that's because it's trying to allocate 25MB, and these are only 20MB on the 1100D?

(https://bitbucket.org/dmilligan/magic-lantern/downloads/1100Dsrm_mem_test.jpg)

Any interesting thing happened during the test: I tried to take some pics, they never got recorded to the SD card. If I tried to turn the camera off, it just kept saying "recording images", open the battery door and it says "don't take the battery out!", try to go into LV and it just says "busy...please wait" (maybe I have the stubs wrong?)
Title: Re: SRM job memory buffers
Post by: dmilligan on July 04, 2014, 05:39:40 PM
changed the test to 19MB, the first buffer works, but the second fails
Title: Re: SRM job memory buffers
Post by: dmilligan on July 04, 2014, 10:17:41 PM
(https://bitbucket.org/dmilligan/magic-lantern/downloads/FreeMemory60D.jpg)

holy sh*t!!!! 250MB more memory on the 60D!?

EDIT: 6x31MB in LV and Movie mode!
Title: Re: SRM job memory buffers
Post by: nikfreak on July 04, 2014, 11:11:51 PM
i made some changes to mlv_rec to support this new memory interface to make some tests.
it looks very promising. here some results:

5D3 v1.1.3:
Memory details: total 277 MiB [exmem 238 MiB, SRM 39 MiB] in 89 slots

7D v2.0.3:
Memory details: total 371 MiB [exmem 182 MiB, SRM 188 MiB] in 164 slots

600D v1.0.3:
Memory details: total 129 MiB [exmem 67 MiB, SRM 62 MiB] in 44 slots

this is another powerful push. thanks a lot, alex :)

after posting the missing stubs for 6D (see above), can anyone verify "SRM job total" for 6D? Could I find this out myself in QEMU w/o owning the cam?
Congrats dmilligan 250MB sounds amazing
Title: Re: SRM job memory buffers
Post by: Pelican on July 05, 2014, 12:18:51 AM
Another exciting discovery...  :D Congrats!!!
Title: Re: SRM job memory buffers
Post by: dmk on July 06, 2014, 09:24:46 AM
Very cool, thanks for taking the time to discover this!

Just to fuel excitement, what sorts of features will probably be able to happen with the extra memory that weren't possible before? Just looking forward to the future, not pushing, totally appreciating what you've done so far and curious about what's in store :D
Title: Re: SRM job memory buffers
Post by: Levas on July 06, 2014, 08:09:27 PM
If I read Alex post from juli 03, it makes no difference for things that use live view, so no memory gain here...(except 30MB extra for the 550d)
If I get it right it seems that live view uses the same amount of memory that is discovered.

Quote from Alex post:
For example, for 550D, in LiveView, you get 75 + 1x31 MB in normal mode, and 43 + 3x31 in factory mode. 30MB extra is not bad, no?
On 5D2, in LiveView, you get 183 + 4x37 in normal mode, and 259 + 2x37 in factory mode (same with jpeg-small). No difference.
On 5D3, you get 163 + 3x39 in normal mode, and 127 + 4x39 in factory mode. No difference.
Title: Re: SRM job memory buffers
Post by: dmilligan on July 06, 2014, 09:03:00 PM
If I get it right it seems that live view uses the same amount of memory that is discovered.
nope, some of it used by LV but not all of it, in fact it's not even the majority of it is used by LV, on 60D only 1/4 of SRM is used by LV (we get 186MB extra even in LV)
a1ex is talking about factory mode vs. not factory mode when he says "no difference", he's not talking about LV vs. not LV

I made some quick and dirty changes to mlv_rec to use the new memory interface and I was able to get 30-70% longer recording times (link (http://www.magiclantern.fm/forum/index.php?topic=5653.msg120999#msg120999)).

One thing I'll note: when using SRM memory with mlv_rec, "BUSY PLEASE WAIT" is displayed at the top of the screen, however recording works without a hitch. I assume this is because Canon h246 recording wants to use this memory.
Title: Re: SRM job memory buffers
Post by: Levas on July 06, 2014, 09:42:09 PM
 :o So there's more memory gained with MLV-recording!

Woohoo  :D

Title: Re: SRM job memory buffers
Post by: g3gg0 on July 06, 2014, 11:54:24 PM
I made some quick and dirty changes to mlv_rec to use the new memory interface and I was able to get 30-70% longer recording times (link (http://www.magiclantern.fm/forum/index.php?topic=5653.msg120999#msg120999)).

there are my changes: link (http://upload.g3gg0.de/pub_files/4ff199e7f0ac8b5321e37adee738ebd0/2014-07-02_20-56-20_r11391_.diff) (made before alex' updates to mem.c were made)
Title: Re: SRM job memory buffers
Post by: dubarry on July 07, 2014, 05:56:46 AM
Hey dmilligan, if you ever get the opportunity I would be very interested to know how many frames you are getting at

1728x736 p24 2.35

This of course is not a priority just if you get the opportunity...let us know.
Title: Re: SRM job memory buffers
Post by: Rewind on July 07, 2014, 08:00:01 AM
I want to find these stubs for 650D.
Is there sort of “easy way” to try that?

I mean point me please to some reading about that?
Title: Re: SRM job memory buffers
Post by: Audionut on July 07, 2014, 08:19:39 AM
http://www.magiclantern.fm/forum/index.php?topic=12177.0
Title: Re: SRM job memory buffers
Post by: mk11174 on July 07, 2014, 10:20:31 AM
On the 700D now it shows 154MB allocated when I choose to reserve space and before the mem patch it only said 92MB.

And I gain 1 second on 1920x1080@24p, so instead of 1 second it record 2 seconds.

This was the only test I did just to see if it did anything.

I made sure I formated card before each test as well.
Title: Re: SRM job memory buffers
Post by: Levas on July 07, 2014, 10:25:18 AM
Hey dmilligan, if you ever get the opportunity I would be very interested to know how many frames you are getting at

1728x736 p24 2.35

This of course is not a priority just if you get the opportunity...let us know.

If you know how much memory is available, you can calculate it yourself, the 60d can write 21MB/s to the SD card, the rest is buffered into the available memory.
Now the resolution you ask for,  takes up, 1728 x 736 x 24(frames/second) x 14(14bit canon raw) / 8 (conversion bits to bytes) /1000000(conversion bytes to MegaBytes) = 53,4 MB/second
1728*736*p24 resolution takes up 53,4 MB/second

I'm not sure how much free memory the 60d has, my guess is about 350MB now ?

That would make 53,4MB/second - 21MB/second(SD card writing speed in canon 60d) = 32,4MB/second that needs to be buffered every second.
350MB / 32,4MB = 11 Seconds of raw video

And after you filled up this memory, it takes up to 350MB/21MB = 17 seconds time to clear/write the complete filled up buffer to the SD card.

Title: Re: SRM job memory buffers
Post by: mk11174 on July 07, 2014, 01:31:46 PM
I want to find these stubs for 650D.
Is there sort of “easy way” to try that?

I mean point me please to some reading about that?
These are for my 700D
/** SRM JobMem **/
NSTUB(0xFF0E6C78,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0EA848,  SRM_FreeMemoryResourceFor1stJob)

So these should be for your 650D
/** SRM JobMem **/
NSTUB(0xFF0E6AE4,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0EA5B8,  SRM_FreeMemoryResourceFor1stJob)

Don't know the others yet, but all you need to try mlv_rec patch
Title: Re: SRM job memory buffers
Post by: Rewind on July 07, 2014, 03:08:55 PM
Ok, i've added these stubs to stubs.S, applied this patch  (http://upload.g3gg0.de/pub_files/4ff199e7f0ac8b5321e37adee738ebd0/2014-07-02_20-56-20_r11391_.diff) to the mlv_rec.c and compiled.
Everything went fine, ML loaded ok, module working. But no difference at all with regular mlv_rec.mo.

Am i missing something, additional steps needed? Or may be stubs addresses wrong?
Title: Re: SRM job memory buffers
Post by: mk11174 on July 07, 2014, 03:38:56 PM
Ok, i've added these stubs to stubs.S, applied this patch  (http://upload.g3gg0.de/pub_files/4ff199e7f0ac8b5321e37adee738ebd0/2014-07-02_20-56-20_r11391_.diff) to the mlv_rec.c and compiled.
Everything went fine, ML loaded ok, module working. But no difference at all with regular mlv_rec.mo.

Am i missing something, additional steps needed? Or may be stubs addresses wrong?
You have to define it as far as I know, look in the patch for the define, its long, or to test just comment out. Then try.

Also reformat card from camera for both test and see if mlv_rec allocates the 154MB before first recording. If you dont reformat you wont see it because it is already created.
Title: Re: SRM job memory buffers
Post by: Rewind on July 07, 2014, 05:35:25 PM
You have to define it as far as I know, look in the patch for the define, its long, or to test just comment out.

Got it. 154 MiB allocating instead of 92 on 650D.

Oh, by the way, when i enable trace write, i can't find it anywhere on the card. Nothing shows up in console either (when mlv_rec is working). How can i see the debug trace?
Title: Re: SRM job memory buffers
Post by: dubarry on July 07, 2014, 05:52:12 PM
If you know how much memory is available, you can calculate it yourself, the 60d can write 21MB/s to the SD card, the rest is buffered into the available memory.
Now the resolution you ask for,  takes up, 1728 x 736 x 24(frames/second) x 14(14bit canon raw) / 8 (conversion bits to bytes) /1000000(conversion bytes to MegaBytes) = 53,4 MB/second
1728*736*p24 resolution takes up 53,4 MB/second

I'm not sure how much free memory the 60d has, my guess is about 350MB now ?

That would make 53,4MB/second - 21MB/second(SD card writing speed in canon 60d) = 32,4MB/second that needs to be buffered every second.
350MB / 32,4MB = 11 Seconds of raw video

And after you filled up this memory, it takes up to 350MB/21MB = 17 seconds time to clear/write the complete filled up buffer to the SD card.

Thanks for that Levas.  I have found that the numbers often don't translate as black and white as that (considering unknown factors at this stage) but at least it gives me something to dream about.
Title: Re: SRM job memory buffers
Post by: mk11174 on July 07, 2014, 06:38:27 PM
Got it. 154 MiB allocating instead of 92 on 650D.

Oh, by the way, when i enable trace write, i can't find it anywhere on the card. Nothing shows up in console either (when mlv_rec is working). How can i see the debug trace?
Not sure, I never messed with any of that but gg3g0 would be the best one to ask.
Title: Re: SRM job memory buffers
Post by: g3gg0 on July 07, 2014, 07:48:20 PM
Oh, by the way, when i enable trace write, i can't find it anywhere on the card. Nothing shows up in console either (when mlv_rec is working). How can i see the debug trace?

you need trace.mo (http://ml.g3gg0.de/modules/trace.mo/trace.mo)
Title: Re: SRM job memory buffers
Post by: Rewind on July 07, 2014, 07:57:52 PM
Thank you.
One more impatience question:

May I ask you to give me some help to adopt your mlv_rec patch to raw_rec.mo?
Is it possible to use SRM job memory buffers in raw_rec at all?
This module treats memory differently, and I can't figure that out by myself.

Title: Re: SRM job memory buffers
Post by: dmilligan on July 08, 2014, 12:08:18 AM
With the latest changes that a1ex made to get a mem_suite from SRM, you can actually use SRM the same way both raw_rec and mlv_rec allocate memory using mem_suite. Basically you just need call srm_malloc_suite(0) to get a mem_suite for SRM, and then you can use it the same way as shoot_malloc_suite(0); This (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60) is how I did it for mlv_rec it should be basically the same for raw_rec.
Title: Re: SRM job memory buffers
Post by: dmilligan on July 08, 2014, 12:20:19 AM
Hey dmilligan, if you ever get the opportunity I would be very interested to know how many frames you are getting at

1728x736 p24 2.35
269 (~11s)

also I managed to get 505 frames (~21s) of 720p (better than the 460 I was getting yesterday)
Title: Re: SRM job memory buffers
Post by: dubarry on July 08, 2014, 01:22:48 AM
wow. I've always thought 10 secs would do for most cases but that 21 secs at 720p is sooo appealing.

And this is exactly what I meant .. why on earth would you be getting more frames (505) fine tuning things I suppose? This is great news.

Thank you for keeping us posted
Title: Re: SRM job memory buffers
Post by: N/A on July 08, 2014, 01:30:35 AM
Who wants to compile this for 7d so I can do some beta testing?  :D
Title: Re: SRM job memory buffers
Post by: Rewind on July 08, 2014, 06:27:31 AM
Need some clues on how to find the CreateMemoryChunk and AddMemoryChunk stubs.
(for 650d rom)

We have this:
(http://savepic.ru/5262574.jpg)

It says NSTUB ( 0x8B7C, CreateMemorySuite) in stubs.S, but ROM disassemble starts with ff000000, so how does that make sense? I suspect RAM_OFFSET has to be take into account. A1ex says in tutorial, that it is declared in stubs.S, but it's not the case for 650d. So how do I know my ram offset for particular camera?

for the AddMemoryChunk, i can find many strings like 32(Src,bytes,Unit... , and before them i usually got this:
(http://savepic.ru/5254369.jpg)
Does that mean, 8E58 has some relation to AddMemoryChunk start address?
Title: Re: SRM job memory buffers
Post by: mk11174 on July 08, 2014, 02:39:14 PM
(http://s18.postimg.org/cjjke7xrt/VRAM4.jpg)

Does this mean it works and the stubs are correct???

Strange that mlv_rec only reports 123MB now and not 154MB after compiling the srm_memory source

When I use the mlv_rec with patch from gg3g0 which he posted a few threads back, I get 154MB in mlv_rec again.

But my SRM Memory still only shows 1x31 does this sound ok?
Title: Re: SRM job memory buffers
Post by: mk11174 on July 08, 2014, 03:16:04 PM
Need some clues on how to find the CreateMemoryChunk and AddMemoryChunk stubs.
(for 650d rom)

We have this:
(http://savepic.ru/5262574.jpg)

It says NSTUB ( 0x8B7C, CreateMemorySuite) in stubs.S, but ROM disassemble starts with ff000000, so how does that make sense? I suspect RAM_OFFSET has to be take into account. A1ex says in tutorial, that it is declared in stubs.S, but it's not the case for 650d. So how do I know my ram offset for particular camera?

for the AddMemoryChunk, i can find many strings like 32(Src,bytes,Unit... , and before them i usually got this:
(http://savepic.ru/5254369.jpg)
Does that mean, 8E58 has some relation to AddMemoryChunk start address?

These are what I have for 700D now and I get working results and 1x31MB
NSTUB(0xFFA651FC - RAM_OFFSET,  CreateMemoryChunk)   //835C      // called from CreateMemorySuite
NSTUB(0xFFA65B88 - RAM_OFFSET,  AddMemoryChunk)        //8CE8      // called before " 32(Src,bytes,Unit)=(%#lx,%#lx,%#lx)" in many places; see also hAddSuite

So this is what you would put for 650D since your RAM-OFFSET is not defined
NSTUB(    0x835C,  CreateMemoryChunk)                      // called from CreateMemorySuite
NSTUB(    0x8CE8,  AddMemoryChunk)                           // called before " 32(Src,bytes,Unit)=(%#lx,%#lx,%#lx)" in many places; see also hAddSuite

Title: Re: SRM job memory buffers
Post by: Rewind on July 08, 2014, 04:25:54 PM
Thank you, mk11174 for your findings!
I'd prefer to know about some search 'algorithms' to find the stubs by myself, but i may suppose it is not what's easy to say in two words )

Anyway, now I get the same strange results as yours on 650D:
The test from 'don't click me' shows only 1x31 Mb buffer and dimilligan's mlv_rec only reports 123MB
But mlv patch from gg3g0 still works and gives 154 Mb.
Title: Re: SRM job memory buffers
Post by: mk11174 on July 08, 2014, 04:48:57 PM
Thank you, mk11174 for your findings!
I'd prefer to know about some search 'algorithms' to find the stubs by myself, but i may suppose it is not what's easy to say in two words )

Anyway, now I get the same strange results as yours on 650D:
The test from 'don't click me' shows only 1x31 Mb buffer and dimilligan's mlv_rec only reports 123MB
But mlv patch from gg3g0 still works and gives 154 Mb.

This way works to and makes more sense from the comments in the 5d3 stubs file
For 650D
NSTUB(    0x835C,  CreateMemoryChunk)        // called from CreateMemorySuite
NSTUB(    0x8E58,  AddMemoryChunk)           // called before " 32(Src,bytes,Unit)=(%#lx,%#lx,%#lx)" in many places; see also hAddSuite


For 700D written this way acts a bit flakey, so I guess best to use the way I wrote it for 650D which works solid
NSTUB(0xFFA651FC,  CreateMemoryChunk)        // called from CreateMemorySuite
NSTUB(0xFFA65CF8,  AddMemoryChunk)           // called before " 32(Src,bytes,Unit)=(%#lx,%#lx,%#lx)" in many places; see also hAddSuite


Also this way also keep the debug test going after picture takes.

Ah, cool and now it shows 2x31 also and 154MB again in mlv_rec

First time I tried it I had no change, but decided to do a full make clean then recompiled and now shows correct

(http://s30.postimg.org/e9w2ftyhd/VRAM5.jpg)

I use IDA to search, this is where I found them.
(http://s29.postimg.org/s7l4ndvxz/Capture1.jpg)

(http://s23.postimg.org/c7imtvmcr/Capture2.jpg)
Title: Re: SRM job memory buffers
Post by: Rewind on July 08, 2014, 05:10:13 PM
Yes, the 8E58 seems to be correct.
Also, now i get 4x31 MiB outside LV, but 2x31 in LV.
Don't actually understand that. Is it possible to use all 4x31 buffers for mlv recording?
Title: Re: SRM job memory buffers
Post by: mk11174 on July 08, 2014, 05:17:27 PM
Yes, the 8E58 seems to be correct.
Also, now i get 4x31 MiB outside LV, but 2x31 in LV.
Don't actually understand that. Is it possible to use all 4x31 buffers for mlv recording?
I have no clue how it works, and cool, never tried outside LV, lol
Title: Re: SRM job memory buffers
Post by: dmilligan on July 08, 2014, 05:24:05 PM
Don't actually understand that. Is it possible to use all 4x31 buffers for mlv recording?
No, LV is using 2 of the buffers itself, that's why you only get 2 in LV. When not in LV you get more b/c you get buffers that LV isn't using. The other buffers are needed by the Canon tasks that take a picture, so if you use them you can't take pictures (a1ex already mentioned this), until you release them again. So basically we are stealing memory from Canon's LV and picture taking tasks. We can use them so long as we are not trying to do one of the things we stole them from.
Title: Re: SRM job memory buffers
Post by: Rewind on July 08, 2014, 05:46:37 PM
No, LV is using 2 of the buffers itself, that's why you only get 2 in LV.

Now it's clear. Thank you.
Dmilligan, can you please help us to adopt these changes (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60) to legacy raw_rec module?

I am asking because its speed still perceptibly higher than mlv_rec, and that matters for low end cameras.
For example, with my old raw_rec build i've got about 80 seconds of 1536x640 at 24 fps.
But with mlv_rec even with new SRM job buffers only about 50 secs.

This is because raw_rec speed hovers around 38.4 MB/s, but mlv_rec only about 37.
If we'll be able to make raw_rec to use new srm buffers, it will be a huge addition to recording times!
Title: Re: SRM job memory buffers
Post by: Greg on July 08, 2014, 07:03:58 PM
That's correct stubs?

500D :
Code: [Select]
NSTUB(0xFF07B90C,  CreateMemoryChunk)
NSTUB(0xFF07C3A8,  AddMemoryChunk)
Title: Re: SRM job memory buffers
Post by: Rainmanwalking on July 08, 2014, 11:12:39 PM
With the latest changes that a1ex made to get a mem_suite from SRM, you can actually use SRM the same way both raw_rec and mlv_rec allocate memory using mem_suite. Basically you just need call srm_malloc_suite(0) to get a mem_suite for SRM, and then you can use it the same way as shoot_malloc_suite(0); This (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60) is how I did it for mlv_rec it should be basically the same for raw_rec.
I don't understand much of what your saying but, What I think I'm hearing is these latest changes should also benefit video shot in raw_rec which I use primarily and record sound externally.  That would be great news since a few more seconds could make a world of difference. 
Title: Re: SRM job memory buffers
Post by: dmilligan on July 09, 2014, 03:30:48 AM
Dmilligan, can you please help us to adopt these changes (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60) to legacy raw_rec module?
https://bitbucket.org/dmilligan/magic-lantern/commits/2c86608a549a7a3b1e85fd17b9acd8ed9214d8c0

Warning: I did not test this at all
Title: Re: SRM job memory buffers
Post by: dubarry on July 09, 2014, 05:54:31 AM
Thanks for that dmilligan (didn't even think legacy raw_rec mo was even used or supported anymore.)

After the bots do their thing I'll def try this out.

Glad you asked Rewind, glad to see I'm not the only old school raw_rec mo around here!
Title: Re: SRM job memory buffers
Post by: mk11174 on July 09, 2014, 09:53:20 AM
https://bitbucket.org/dmilligan/magic-lantern/commits/2c86608a549a7a3b1e85fd17b9acd8ed9214d8c0

Warning: I did not test this at all
Confirmed working, on 700D everything is doubled in record frame amount! Nice job dmilligan!
Title: Re: SRM job memory buffers
Post by: Rewind on July 09, 2014, 11:13:10 AM
Confirmed working on 650D. Thank you, dmilligan!

Results at 23.976 fps:

Resolution                     Old rec. time no SRM (GD off)       New rec. time with SRM (GD on)
1568 x 656 (2.39:1)      32 s                                              53 s
1600 x 672 (2.38:1)      18 s                                              32 s
1728 x 720 (2.40:1)      7 s                                                12 s

So we have x1.7 recording time increase even with Global Draw on
Title: Re: SRM job memory buffers
Post by: NedB on July 09, 2014, 11:51:45 AM
@mk11174: These results on the 650D from Rewind are very encouraging. If the recording times on the 550D for near-HD (like 1600x672, etc.) start getting up in the 10-20 seconds range, that changes things a lot. I'm turning to you because you were the driving force last year behind getting raw up and running on the 550D.

Am I correct in viewing the current situation as regards the increased SRM?:
1. Various contributors are busy finding stubs and compiling ML and associated modules for particular cameras
2. These compiled ML versions and modules are being tested by these contributors and they are reporting their results
3. Right now, if you want to try out and/or test these cutting-edge versions, you have to compile them yourself

Have you been able to work on the 550D in this context? Do you have an ML version/modules you could share for testing purposes?  Thanks very much in advance and Cheers!
Title: Re: SRM job memory buffers
Post by: nikfreak on July 09, 2014, 12:05:33 PM
Anyone able to compile for 6D and provide it for 6D owners to test recording time increase on 6D? Indeed, increased results posted here are very encouraging. I posted necessary stubs some days ago but am still trying to decide if I buy a 6D or move with something else like 7D. This could help me in making a decision and I think other 6D owners would like to test this, too.

So now see following. These are my findings for 6D_113:

Code: [Select]
NSTUB(0xFF0F0098,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0F40C8,  SRM_FreeMemoryResourceFor1stJob)
NSTUB(0xFFCCE714 - RAM_OFFSET,  CreateMemoryChunk)
NSTUB(0xFFCCF210 - RAM_OFFSET,  AddMemoryChunk)

If you need others then just say so....
Title: Re: SRM job memory buffers
Post by: Audionut on July 09, 2014, 12:16:07 PM
I posted necessary stubs some days ago but am still trying to decide if I buy a 6D

It would be great if you finally decide on a 6D, since this port is all but unmaintained currently, and you seem capable of keeping that port in an active state.  :)
Title: Re: SRM job memory buffers
Post by: nikfreak on July 09, 2014, 12:52:51 PM
It would be great if you finally decide on a 6D, since this port is all but unmaintained currently, and you seem capable of keeping that port in an active state.  :)

Hrrrr dunno dunno. I know I can maintain 6D platform and keep nightlies up2date. I did this for android device trees, too  :P
Will make a decision soon and ofc whichever canon body that will be I am going to contribute as much as I can to help this great project. If anyone knows where to grab a 6D for good price (also used) in Germany or Europe then PM me.
Title: Re: SRM job memory buffers
Post by: mk11174 on July 09, 2014, 02:00:54 PM
Anyone able to compile for 6D and provide it for 6D owners to test recording time increase on 6D? Indeed, increased results posted here are very encouraging. I posted necessary stubs some days ago but am still trying to decide if I buy a 6D or move with something else like 7D. This could help me in making a decision and I think other 6D owners would like to test this, too.
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip)
Title: Re: SRM job memory buffers
Post by: mk11174 on July 09, 2014, 02:03:51 PM
Have you been able to work on the 550D in this context? Do you have an ML version/modules you could share for testing purposes?  Thanks very much in advance and Cheers!
Hi, sorry, I have not been testing the 550D with this mem module yet, I think alex was and said he had 31megs extra, not sure if the stubs are added for that one yet, I would have to go find them first then it should be good to test, unless someone already knows them?
Title: Re: SRM job memory buffers
Post by: Rewind on July 09, 2014, 02:27:40 PM
Stubs for 550D (tested. working)

/** ExMem **/
NSTUB(0xFF06C678,  CreateMemorySuite)
NSTUB(0xFF06c4A8,  DeleteMemorySuite)

/** SRM JobMem **/
NSTUB(0xFF028590,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02AD5C,  SRM_FreeMemoryResourceFor1stJob)
NSTUB(0xFF06BEC8,  CreateMemoryChunk)
NSTUB(0xFF06C918,  AddMemoryChunk)
Title: Re: SRM job memory buffers
Post by: Rewind on July 09, 2014, 02:29:33 PM
Some quick tests with raw_rec for 550D at 23.976 fps:

Res.               Before               Now
1280x720      4 s                    8 s
1280x592      7 s                    14 s

So basically near the 2x rec. time increase.

But mlv_rec allocates 134 MiB — don't know, if it's new maximum for the 550d?
Title: Re: SRM job memory buffers
Post by: mk11174 on July 09, 2014, 03:05:42 PM
Have you been able to work on the 550D in this context? Do you have an ML version/modules you could share for testing purposes?  Thanks very much in advance and Cheers!
Here is a working version with the srm_memory update, and it def works!
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.550D109.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.550D109.zip)
Title: Re: SRM job memory buffers
Post by: mk11174 on July 09, 2014, 03:07:59 PM

But mlv_rec allocates 134 MiB — don't know, if it's new maximum for the 550d?
I think it is because the shoot total is like 20MB less then the 650D and 700D
Title: Re: SRM job memory buffers
Post by: nikfreak on July 09, 2014, 03:16:19 PM
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip)

Yeah I knew that it can campile as a1ex already mentioned it after posting the first stubs for 6d. Now I need to post this in 6D dev thread to get feedback. thanx for your efforts.
Title: Re: SRM job memory buffers
Post by: Levas on July 09, 2014, 04:47:20 PM
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip)

Downloaded this nightly build for 6d, it worsk, but no extra seconds of MLV recording.
In photo mode it has:
Shoot total of 255MB
SRM job total of 4 x 34MB

In movie mode:
Shoot total of 151MB
SRM job total of 3 x 34MB

So we're missing some memory in movie mode, but potentially it has 255MB + 3 x 34MB ?
Title: Re: SRM job memory buffers
Post by: Rewind on July 09, 2014, 05:14:27 PM
Downloaded this nightly build for 6d, it worsk, but no extra seconds of MLV recording.

This might be helpful:
Quote from: mk11174
Also reformat card from camera for both test and see if mlv_rec allocates the 154MB before first recording. If you dont reformat you wont see it because it is already created.
Title: Re: SRM job memory buffers
Post by: nikfreak on July 09, 2014, 05:47:40 PM
Also reformat card from camera for both test and see if mlv_rec allocates the 154MB before first recording. If you dont reformat you wont see it because it is already created.

@Levas please confirm...
Title: Re: SRM job memory buffers
Post by: josepvm on July 09, 2014, 06:18:13 PM
That's correct stubs?

500D :
Code: [Select]
NSTUB(0xFF07B90C,  CreateMemoryChunk)
NSTUB(0xFF07C3A8,  AddMemoryChunk)

No, these are not correct.

I have compared the dissambled code with the stubs for 550D posted above by Rewind, and I would say:

for 500D:
Code: [Select]
/** ExMem **/
NSTUB(0xFF06C0D0,  CreateMemorySuite)
NSTUB(0xFF06BEEC,  DeleteMemorySuite)
NSTUB(0xFF06B90C,  CreateMemoryChunk)
NSTUB(0xFF06C3A8,  AddMemoryChunk)


-------------edit !!!------------------------
/** SRM JobMem **/
NSTUB(0xFF02C5A0,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02D6B0,  SRM_FreeMemoryResourceFor1stJob)
-----------edit !!! ----this values are wrong !!!-------------





Tested in camera with a build of srm-memory branch.

The stubs API test completes Ok, no errors.

But I have not been able to verify the available SRM memory.   The Free Memory test shows 0MB for SRM Jobs, and ends crashing with Err. 70, I have to take battery off.
The "Don't click me!" test also crashes with Err.70

------------------------------------------------------
Edit: See my post below to see the correct values:  http://magiclantern.fm/forum/index.php?topic=12528.msg121487#msg121487 (http://magiclantern.fm/forum/index.php?topic=12528.msg121487#msg121487)

Title: Re: SRM job memory buffers
Post by: NedB on July 09, 2014, 06:19:14 PM
@mk11174: Man, that's service! Thanks very much for the upload. Have downloaded and will check to confirm Rewind's results for the 550D. But I have to admit I had forgotten that the SD bottleneck was only half as bad on the 650D as on the 550D, so I now realize we won't get a really great frame count for higher resolutions without some other amazing developer insight! Cheers...
Title: Re: SRM job memory buffers
Post by: Levas on July 09, 2014, 08:32:11 PM
@Levas please confirm...

Reformatted the card in camera (canon menu, with ML being restored) and checked the free memory.

Same totals.

Video mode
Shoot total 151MB and SRM total 3x34MB.

Photo mode 255MB and SRM total 4x34MB.

Also memory hack and extra mem hack in the raw-video menu didn't make any difference 

And shoot multiple clips with it, all the same recording length, same length as in older versions...
1792 x 750, 25P = 13/14 seconds of video (which is what too expect with 151MB+3x34MB)
Expect to have about 18 seconds of video(1792x750,25P) if you have 255MB shoot total and 3x34MB SRM available.




Title: Re: SRM job memory buffers
Post by: nikfreak on July 09, 2014, 09:02:21 PM
Reformatted the card in camera (canon menu, with ML being restored) and checked the free memory.

Same totals.

Video mode
Shoot total 151MB and SRM total 3x34MB.

Photo mode 255MB and SRM total 4x34MB.

Also memory hack and extra mem hack in the raw-video menu didn't make any difference 

And shoot multiple clips with it, all the same recording length, same length as in older versions...
1792 x 750, 25P = 13/14 seconds of video (which is what too expect with 151MB+3x34MB)
Expect to have about 18 seconds of video(1792x750,25P) if you have 255MB shoot total and 3x34MB SRM available.

Thx for providing feedback.
Title: Re: SRM job memory buffers
Post by: Levas on July 09, 2014, 09:13:08 PM
Probably already noticed by some of you,
But the SRM size is the exact same size as one raw image.

5D3 = 22MP, 22 x 14 bit / 8 = 39MB
6D = 20MP, 20 x 14 bit / 8 = 34MB
500D-550D-600D-650D etc. = 18MP, 18 x 14 bit / 8 = 31MB
500D = 15MP, 15 x 14 bit / 8 = 26MB
1100D = 12MP, 12 x 14 bit / 8 = 20MB
Title: Re: SRM job memory buffers
Post by: Levas on July 09, 2014, 10:57:06 PM
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip)

It seems that it doesn't work for the 6d, does this mean the stubs aren't right ?

Just checked the previous nightly build I'm using (15-03-2014) and it says the same about the free memory available.
Shoot total of 255MB in Photo mode and 151MB in video mode (SRM is of course not mentioned in this build).
The 151MB reported in video mode does however change "magically" too 255, otherwise I could never get 13 seconds of video in 1792x750, 25P resolution (58,8MB/sec.)
SD card writes about 39MB second, (58,8 - 39) x 13 seconds = 257MB...
So the previous builds of MLV_rec gives about 255MB buffer memory for video, although it shows a shoot total of 151MB in the Free Memory debug menu.

The Build from 09-07-2014 you made, does the same thing, it shows 151MB of shoot total in the free memory debug menu, but it gives however recording times for 255MB of memory.
This could mean the new build:
Uses the 3x 34 MB SRM memory, but doesn't magically higher the 151MB shoot total too 255MB as available in photo mode.
Or
It doesn't use the 3 x 34MB of SRM memory and still free's the shoot total of 255MB of memory.
Title: Re: SRM job memory buffers
Post by: nikfreak on July 10, 2014, 01:39:59 AM
Some quick tests on 5D3 123, first outside LiveView, second in LV:
(http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3.png) (http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3-lv.png)

On 5D3, you get 163 + 3x39 in normal mode, and 127 + 4x39 in factory mode. No difference.

@Levas look at page1. a1ex posted values for 5dmk3 and also later some comment for 5dmk3 (bolded above). Looks like 6D is simpy the same. No difference!
Seems like 7D / 600D and maybe others will be winners regarding usefulness of using SRM for mlv recording...
Title: Re: SRM job memory buffers
Post by: Audionut on July 10, 2014, 02:23:01 AM
I haven't been following this closely.  AFAIK, the SRM is never used in normal Canon tasks, so any memory from there is extra.

It looks like that comment for the 5D3, is only when calling via SRM_ChangeMemoryManagementForFactory, which, "might make a difference".  On the 5D3, calling via that routine, does not make a difference (over and above the SRM already found).

That's the way I read it.
Title: Re: SRM job memory buffers
Post by: dmilligan on July 10, 2014, 02:52:43 AM
That's mostly right except for:
the SRM is never used in normal Canon tasks
It is used by Canon tasks. Some of it is used by LV, if you allocate all of SRM outside of LV and then try to enter LV, you get err70 or 'please wait' and LV never starts. The rest of it is used by the tasks when taking a photo. If you allocate SRM, you can't take pictures till you free it (also err 70). Fortunately there's no need to take pictures during raw video recording.
Title: Re: SRM job memory buffers
Post by: Audionut on July 10, 2014, 03:11:02 AM
Oh of course.  Thanks.

Fortunately there's no need to take pictures during raw video recording.

Be interesting to see if it can be used for silent pics, especially the full resolution ones.
Title: Re: SRM job memory buffers
Post by: Levas on July 10, 2014, 12:07:41 PM
@Levas look at page1. a1ex posted values for 5dmk3 and also later some comment for 5dmk3 (bolded above). Looks like 6D is simpy the same. No difference!
Seems like 7D / 600D and maybe others will be winners regarding usefulness of using SRM for mlv recording...

The free memory values and changes between photo and video mode do look very alike for the 5d3 and 6d.
But what I don't get is how video mode/liveview can suck up so much memory on the 5d3 and 6d, there's more then 100MB taken by liveview.
How does this look in the 500d,550d,600d which for as far as I know, don't even have 100MB spare memory to be taken by live view ?

And where does the 255MB of free memory for MLV_rec we have in the old builds on the 6d come from ?
It has to do with the memory hack function available in the raw video menu (the memory hack is not needed by the new build of 09-07-14 to have 255MB of free memory, turning it on or off makes no differences).
In the description it says it takes extra memory outside of live view on the 5d3. But it works just as good on the 6d. (without the hack I have 151MB of memory for mlv recording buffer)
So can anyone tell where the "memory hack" function in the raw video menu get's it's extra memory from ?

Title: Re: SRM job memory buffers
Post by: dmilligan on July 10, 2014, 02:59:09 PM
But what I don't get is how video mode/liveview can suck up so much memory on the 5d3 and 6d, there's more then 100MB taken by liveview.

You have to remember ML is operating in the margins. Why would Canon give the camera extra memory that it didn't need? It's a miracle there's any memory for ML to use at all. There are many reasons LV would need memory, and lots of it (want specific reasons, ask a Canon engineer, or do some reverse engineering). Also, just because memory is allocated, doesn't mean it's actually being used, it just means its reserved (and we can't use it). So LV is 'reserving' this memory not necessarily using it (but who knows, it might be using it).

So can anyone tell where the "memory hack" function in the raw video menu get's it's extra memory from ?
It just gets it from shoot_malloc, like the rest of the memory it uses, it just calls shoot_malloc under different circumstances. Here's basically how the hack works (to the best of my understanding from looking at the code). Before allocating memory with shoot_malloc (which is some function similar to malloc (http://en.wikipedia.org/wiki/C_dynamic_memory_allocation) that's in the Canon firmware), LV is paused. Then we call shoot_malloc, then LV is resumed. Because LV is paused there is more memory available to shoot_malloc (why? ask a Canon engineer), we take it, and then LV doesn't seem to mind that it's gone.

I specifically put the srm_malloc call outside of the memory hack pause in my changes to raw_rec and mlv_rec (i.e. it doesn't try to malloc while LV is paused) for safety reasons -> I didn't want srm_malloc to steal memory that LV actually needed.

What would happen if you did? It might not make any difference, you might get more srm, or you might just crash the camera or err70. If you're feeling brave, move the srm_malloc_suite(0); call next to the shoot_malloc_suite(0); and find out.
Title: Re: SRM job memory buffers
Post by: Levas on July 10, 2014, 03:36:47 PM

I specifically put the srm_malloc call outside of the memory hack pause in my changes to raw_rec and mlv_rec (i.e. it doesn't try to malloc while LV is paused) for safety reasons -> I didn't want srm_malloc to steal memory that LV actually needed.

What would happen if you did? It might not make any difference, you might get more srm, or you might just crash the camera or err70. If you're feeling brave, move the srm_malloc_suite(0); call next to the shoot_malloc_suite(0); and find out.

I'm no coder at all, but everything is done in a certain sequence, I mean memory can't be called all at once.
What exactly is done, now the srm_malloc call is placed outside the memory hack ?
-Is the memory hack done first (so it pauses LV, and gets a total of 255MB's of memory)
-then calls srm_malloc (and maybe steals the 3 x 34MB from the extra memory that is gained with the live view hack, ending up with nothing extra ?)

So from what I read in your post, it is possible to put the srm_malloc call within the extra memory hack, which looks like this:
-the extra memory hack pauses live view, shoot_malloc is called, additionally we could call srm_malloc before resuming live view ?

With brave enough, you mean it is possible that I end up with a bricked camera ?
Or is it more likely that the camera runs somehow out of memory and if I get the battery out quick enough and put it back in it still works ?
Title: Re: SRM job memory buffers
Post by: dmilligan on July 10, 2014, 04:04:32 PM
-Is the memory hack done first
No, srm is allocated first, but it really doesn't matter (AFAIK), and this is just my crude implementation we're talking about, there are issues that need to be addressed for a proper implementation that can be merged

-then calls srm_malloc (and maybe steals the 3 x 34MB from the extra memory that is gained with the live view hack, ending up with nothing extra ?)
AFAIK, srm_malloc memory and shoot_malloc memory are different, you should be able to get both. The extra memory from "memory hack" is just extra from shoot_malloc => we call shoot_malloc a different way and it gives us more memory.

With brave enough, you mean it is possible that I end up with a bricked camera ?
It's always a possibility, however unlikely.

Or is it more likely that the camera runs somehow out of memory and if I get the battery out quick enough and put it back in it still works ?
Probably
Title: Re: SRM job memory buffers
Post by: Levas on July 10, 2014, 05:06:02 PM
If I get it right, the srm memory calls are implemented, but not yet in a way that it can be combined with the memory hack function.
So toggle memory hack function on and off has no influence because it isn't used at all in the build from 9 July for the 6d.

This means the implemented srm memory function does work on the 6d!  :D :D :D
Because I get raw video shooting times which corresponds with a buffer of 253MB. (151MB+ 3 x 34MB srm).
If the srm memory wasn't available we would expect shooting times which corresponds too 151MB.

For extra memory on 5d3 and 6d we now need an combination of the srm_memory function and the "memory hack" function.


 
Title: Re: SRM job memory buffers
Post by: josepvm on July 10, 2014, 05:25:50 PM

for 500D:
Code: [Select]

/** SRM JobMem **/
NSTUB(0xFF02C5A0,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02D6B0,  SRM_FreeMemoryResourceFor1stJob)




Sorry, these values I posted previously are not correct.

These are the good ones, for 500D

Code: [Select]
/** ExMem **/
NSTUB(0xFF06C0D0,  CreateMemorySuite)
NSTUB(0xFF06BEEC,  DeleteMemorySuite)
NSTUB(0xFF06B90C,  CreateMemoryChunk)
NSTUB(0xFF06C3A8,  AddMemoryChunk)


/** SRM JobMem **/
NSTUB(0xFF02A788,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02C6F8,  SRM_FreeMemoryResourceFor1stJob)
Title: Re: SRM job memory buffers
Post by: dmilligan on July 10, 2014, 05:29:14 PM
If I get it right, the srm memory calls are implemented, but not yet in a way that it can be combined with the memory hack function.
No that's wrong. Memory hack and shoot_malloc have nothing to do with SRM
Title: Re: SRM job memory buffers
Post by: Levas on July 10, 2014, 05:54:15 PM
No that's wrong. Memory hack and shoot_malloc have nothing to do with SRM

Probably saying it wrong, I mean the piece of code that calls the srm memory is in the build of 9 Juli and does work for the 6d.
But for 5d3 and 6d users it needs to work together with the memory hack function too have use.

Right now the already available memory hack gives us the same amount of free memory as the new srm function does without the memory hack.
Title: Re: SRM job memory buffers
Post by: Greg on July 10, 2014, 05:54:49 PM
These are the good ones, for 500D

Thanks!  :D

500D, photo and movie mode :

(http://s9.postimg.org/7wxcz11v3/VRAM0.png) (http://s9.postimg.org/jnbageunj/VRAM1.png)


Resolution                     time no SRM          time with SRM
1920 x 818 (crop)             37 frames                62 frames

10MB/s card, raw_rec
Title: Re: SRM job memory buffers
Post by: pittivale on July 10, 2014, 06:12:01 PM
(@nikfreak, @Levas)

This is my first post and I'm not much of a video maker.  ::)

BUT

I DO register a significant increase in record time with mlv_rec on a 6D with the nightly 2014Jul09 posted in this thread :D

I just tested various resolution (16:9) with the 2014May22 and 2014Jul09, without formatting card when changing version, and Global Draw: Allow. Write Speed showd on Display during Record: 35-38.1MB/s. Card: Lexar 600x

Here it is:

Horiz. Res.2014May222014Jul09
17923 sec. (Frame Skipped)6 sec. (Frame Skipped)
17284 sec. (Frame Skipped)7 sec. (Frame Skipped)
16006 sec. (Frame Skipped)12 sec. (Frame Skipped)
15368 sec. (Frame Skipped)15 sec. (Frame Skipped)
150410 sec. (Frame Skipped)18 sec. (Frame Skipped)
147211 sec. (Frame Skipped)20 sec. (Frame Skipped)
134438 sec. (Frame Skipped)57 sec. (Camera shuts down, File Size: 2,13GB)
128058 sec. (Camera shuts down, File Size: 2,08GB)58 sec. (Camera shuts down, File Size: 2,08GB)

Title: Re: SRM job memory buffers
Post by: Levas on July 10, 2014, 06:22:23 PM
(@nikfreak, @Levas)

This is my first post and I'm not much of a video maker.  ::)

BUT

I DO register a significant increase in record time with mlv_rec on a 6D with the nightly 2014Jul09 posted in this thread :D

I just tested various resolution (16:9) with the 2014May22 and 2014Jul09, without formatting card when changing version, and Global Draw: Allow. Write Speed showd on Display during Record: 35-38.1MB/s. Card: Lexar 600x

Here it is:

Horiz. Res.2014May222014Jul09
17923 sec. (Frame Skipped)6 sec. (Frame Skipped)
17284 sec. (Frame Skipped)7 sec. (Frame Skipped)
16006 sec. (Frame Skipped)12 sec. (Frame Skipped)
15368 sec. (Frame Skipped)15 sec. (Frame Skipped)
150410 sec. (Frame Skipped)18 sec. (Frame Skipped)
147211 sec. (Frame Skipped)20 sec. (Frame Skipped)
134438 sec. (Frame Skipped)57 sec. (Camera shuts down, File Size: 2,13GB)
128058 sec. (Camera shuts down, File Size: 2,08GB)58 sec. (Camera shuts down, File Size: 2,08GB)

@Pittivale

What happens to the recording times on the 2014May22 build, if you go into the raw recording menu and put the "memory hack" function on ?
Title: Re: SRM job memory buffers
Post by: pittivale on July 10, 2014, 06:46:51 PM
Activating memory hack increase record time to match 2014Jul09's times. (or 1 sec. less)
On pushing the record button, LiveView shuts down for a moment.
Title: Re: SRM job memory buffers
Post by: nikfreak on July 10, 2014, 07:09:14 PM
@pittivale thx for posting results
@Levas at least we can now be rest assured that my posted stubs are all ok and future srm related findings will work, too. If in any case new stubs need to be added in future then you can drop me a msg and I will post 'em for you. Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...

@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...

As far as I can tell we should also be able to use SRM for photo mode, too? Let's say increase burst rate and therefore fps?
Title: Re: SRM job memory buffers
Post by: surami on July 10, 2014, 08:42:56 PM
Would it be possible to send the frames from the SRM job memory buffers to the SD or CF (camera specific) card and to the USB port (maybe with Wi-Fi read out (http://www.youtube.com/watch?v=Our-AxmrDJM)) paralelly?

Edit: ... or split the frame and send the top half to the SD or CF card and the bottom part to the USB port. Later in post process merge the 2 parts to 1.
Title: Re: SRM job memory buffers
Post by: Greg on July 10, 2014, 09:10:03 PM
On the 500D picture quality (jpg raw) has an effect on the SRM.

             Photo mode          Movie mode
RAW         4x26MB                 2x26MB
JPG           3x26MB                 1x26MB
Title: Re: SRM job memory buffers
Post by: Levas on July 10, 2014, 10:41:10 PM
Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...
Have absolutely no idea how to do that, can anybody else help us here...

@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...

Don't know if I'm right, but my understanding is that raw video/photo data can't be compressed.
How would you compress data that's coming from a bayer pattern ? You need every pixel data to demosaic...
Title: Re: SRM job memory buffers
Post by: dmilligan on July 10, 2014, 11:07:49 PM
@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...
No way, it would require real-time compression of huge amounts of data on a dinky little ARM CPU. If we could compress in real-time we wouldn't need as much extra memory anyway, cause we could just write it out to the card compressed, thus requiring a lower card speed.

As far as I can tell we should also be able to use SRM for photo mode, too? Let's say increase burst rate and therefore fps?
Nope, photo mode is already using SRM when it takes pictures, and this is very likely what it is used for. When we allocate SRM, photo mode doesn't work (this is probably the 11th time I've stated this).

Don't know if I'm right, but my understanding is that raw video/photo data can't be compressed.
How would you compress data that's coming from a bayer pattern ? You need every pixel data to demosaic...
There are lots of ways to losslessly compress raw data (or any data for that matter). CR2 is actually compressed. It's not a matter of how, but the fact that there is just way to much data coming in way to fast. You can actually lossy compress raw data too, one example would be to reduce the bit depth from 14 to say 10. This is technically "compression" and it's lossy (the last 4 bits are lost), but the image can still be reconstructed. Unfortunately the CPU isn't even powerful enough to chop bits off. In fact the CPU isn't even fast enough to move the raw data around, we have to use the edmac (external memory controller) to move raw data around.
Title: Re: SRM job memory buffers
Post by: josepvm on July 10, 2014, 11:30:14 PM
On the 500D picture quality (jpg raw) has an effect on the SRM.

             Photo mode          Movie mode
RAW         4x26MB                 2x26MB
JPG           3x26MB                 1x26MB

Good point! 

I see the same results in my 500D.  And RAW+JPEG gives the same values than JPEG alone (3x26MB Photo, 1x26MB video)

So it seems the camera reserves one SRM buffer to convert the image to JPEG.

That's good, an added motivation to take photos always in RAW   ;)
Title: Re: SRM job memory buffers
Post by: Audionut on July 11, 2014, 03:15:44 AM
If in any case new stubs need to be added in future then you can drop me a msg and I will post 'em for you. Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...

I think we need CreateMemorySuite and DeleteMemorySuite before creating a pull request.
Title: Re: SRM job memory buffers
Post by: pompeiisneaks on July 11, 2014, 04:46:39 AM
These changes are not yet included in any nightly build (we are reverse engineering it as we speak, and sharing the details as soon as we discover them). Please be patient, and hope this will motivate the 6D developers to bring the port up to date.

Some quick tests on 5D3 123, first outside LiveView, second in LV:
(http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3.png) (http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3-lv.png)

I just did this on mine, here's the output on 5D3.123, seems the same.

(http://www.daviszone.org/~pdavis/scn1.png)
Title: Re: SRM job memory buffers
Post by: dyfid on July 11, 2014, 09:39:28 AM
Anyone with a 700D build? The recent nightly's won't boot, but the original 14th Feb Alpha boots everytime. http://www.magiclantern.fm/forum/index.php?topic=12617.new#new (http://www.magiclantern.fm/forum/index.php?topic=12617.new#new) and everything's ok with nightly 550D's with SRM so I think I'm doing it right.
Title: Re: SRM job memory buffers
Post by: mk11174 on July 11, 2014, 10:12:29 AM
Anyone with a 700D build? The recent nightly's won't boot, but the original 14th Feb Alpha boots everytime. http://www.magiclantern.fm/forum/index.php?topic=12617.new#new (http://www.magiclantern.fm/forum/index.php?topic=12617.new#new) and everything's ok with nightly 550D's with SRM so I think I'm doing it right.
Of course, I already took care of the 700D, will upload in a bit for you.  :)

SRM version for 700D https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip)

By the way, I just checked the recent Nightly from here http://builds.magiclantern.fm/#/ and it boots fine. The nighties do not contain the patches for the SRM_Memory though, only these test builds in this thread have the srm_memory patch.
Title: Re: SRM job memory buffers
Post by: nikfreak on July 11, 2014, 10:41:47 AM
I think we need CreateMemorySuite and DeleteMemorySuite before creating a pull request.

reateMemorySuite and DeleteMemorySuite already exist:
https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified (https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified)


Just those missing that I discovered and posted here....
Title: Re: SRM job memory buffers
Post by: dyfid on July 11, 2014, 11:01:11 AM
Of course, I already took care of the 700D, will upload in a bit for you.  :)

SRM version for 700D https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip)

By the way, I just checked the recent Nightly from here http://builds.magiclantern.fm/#/ and it boots fine. The nighties do not contain the patches for the SRM_Memory though, only these test builds in this thread have the srm_memory patch.

Thanks for the SRM build, but hmm, still no good, with Feb14th build on the card the 700D boots, when I put your build on the card, the battery flashes on/off as soon as the card is detected and it won't boot and I have to pull the battery with the card out to reset for it to boot after at all, even with no card in. The Canon Firmware is 1.1.1, but still the Feb14th works fine. What am i doing wrong?
Title: Re: SRM job memory buffers
Post by: mario1000 on July 11, 2014, 11:03:26 AM
I just did this on mine, here's the output on 5D3.123, seems the same.

(http://www.daviszone.org/~pdavis/scn1.png)

Do you have a compiled version for 5D3.113?
Title: Re: SRM job memory buffers
Post by: Audionut on July 11, 2014, 11:05:08 AM
reateMemorySuite and DeleteMemorySuite already exist:
https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified (https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified)


Just those missing that I discovered and posted here....

Oh yes thanks.

Done:  https://bitbucket.org/hudson/magic-lantern/pull-request/537/6d-missing-stubs-from-nikfreak-http/diff
Title: Re: SRM job memory buffers
Post by: mk11174 on July 11, 2014, 11:25:41 AM
Thanks for the SRM build, but hmm, still no good, with Feb14th build on the card the 700D boots, when I put your build on the card, the battery flashes on/off as soon as the card is detected and it won't boot and I have to pull the battery with the card out to reset for it to boot after at all, even with no card in. The Canon Firmware is 1.1.1, but still the Feb14th works fine. What am i doing wrong?
Well there you go, that is your problem, you have to update canon firmware to 113, the new ports are 113 not 111, so go to canon and update your camera, then get the nightly and reflash with the FIR file in the nightly and your good to go with ML, and if you want the SRM just use the zip in this thread.
Title: Re: SRM job memory buffers
Post by: vertigopix on July 11, 2014, 12:06:35 PM
Hi !
Does anybody have a 60D version ? So i can give you my results.

Cheers,

Christian
Title: Re: SRM job memory buffers
Post by: dyfid on July 11, 2014, 12:49:46 PM
Well there you go, that is your problem, you have to update canon firmware to 113, the new ports are 113 not 111, so go to canon and update your camera, then get the nightly and reflash with the FIR file in the nightly and your good to go with ML, and if you want the SRM just use the zip in this thread.

That was it,  :-[, 154mb reserved but now shooting as 20MB/s instead of 40MB/s with 1.1.1 & Feb14th ML? So not getting many frames, sure it was continuous at 720p?. Sandisk Extreme 95MB/s, ExFAT via EOS Card.
Title: Re: SRM job memory buffers
Post by: pompeiisneaks on July 11, 2014, 05:37:54 PM
Do you have a compiled version for 5D3.113?
I don't.  Realize this is a test version, mine couldn't take a picture, it kept saying 'busy' but I think that's because the code used is designed just to see if the ML code can allocate the memory.  If you want me to, I can do a build for 5D3.113, but I'm not going to test it first because rolling back to .113 is a pita.  So I won't have a chance to test it out, and it may not work. 

~Phil
Title: Re: SRM job memory buffers
Post by: pompeiisneaks on July 11, 2014, 06:02:24 PM
Here's the .113 version, again all the usual warnings, I used my build for a different .113 to test external monitors and it worked fine, but ymmv, but I'm pretty sure the codebase is stable and as usable in the srm-memory fork as any other.

https://www.dropbox.com/s/s136ezkzzad66hy/magiclantern-Nightly.2014Jul11.5D3113.zip (https://www.dropbox.com/s/s136ezkzzad66hy/magiclantern-Nightly.2014Jul11.5D3113.zip)
Title: Re: SRM job memory buffers
Post by: escho on July 11, 2014, 09:47:56 PM
Here the results of the mittelfranconian jury for my 600D:

Normal shoot mode:

(https://bitbucket.org/escho/escho-magiclantern/downloads/shoot.jpg)

LiveView:

(https://bitbucket.org/escho/escho-magiclantern/downloads/LV.jpg)

Test with "don´t click me"

(https://bitbucket.org/escho/escho-magiclantern/downloads/test.jpg)

No problems so far

Edgar
Title: Re: SRM job memory buffers
Post by: dyfid on July 12, 2014, 01:47:38 AM
Can anyone confirm what MB/s they're getting out of a 700D raw with the SRM code and FW 1.1.3?

I now only get 20MB/s same as a 550D where as before with FW 1.1.1 and the 700D Feb14th Alpha with exact same card, Sandisk Extreme Pro 95MB/s I was getting 40MB/s and continuous 720P. Can't get anywhere near that now.
Title: Re: SRM job memory buffers
Post by: mk11174 on July 12, 2014, 05:00:52 AM
Can anyone confirm what MB/s they're getting out of a 700D raw with the SRM code and FW 1.1.3?

I now only get 20MB/s same as a 550D where as before with FW 1.1.1 and the 700D Feb14th Alpha with exact same card, Sandisk Extreme Pro 95MB/s I was getting 40MB/s and continuous 720P. Can't get anywhere near that now.
Try using the nightly builds and see what you get, dont compare the builds from 1.1.1, with the SRM build, because there was no build for 1.1.1 with SRM, so to compare, you need to use the regular nightly builds and compare what you use to get when you were at 1.1.1.

And also what module are you testing, the raw_rec or mlv_rec or just a regular movie???

I have same card and these are my results in 1920x1080@24p setting both modules res to 1280x720

(http://s28.postimg.org/99jvi5365/VRAM0.jpg)
MLV_REC

(http://s15.postimg.org/k7o0uagej/VRAM1.jpg)
RAW_REC

Both jump around during record and give a normal reading as it did back in 1.1.1.

When using the SRM build, it does not affect the mb/s as far as I can tell, just records more frames.
Title: Re: SRM job memory buffers
Post by: dmilligan on July 12, 2014, 04:28:51 PM
Cleaned up my changes and created a pull request. It is mostly functionally the same as before, the code is just cleaner, and now there's a menu option to enable/disable srm memory for comparison purposes. If you can compile, please test.

https://bitbucket.org/hudson/magic-lantern/pull-request/540/update-raw_rec-and-mlv_rec-to-use-new-srm/diff
Title: Re: SRM job memory buffers
Post by: mk11174 on July 12, 2014, 04:42:11 PM
Cleaned up my changes and created a pull request. It is mostly functionally the same as before, the code is just cleaner, and now there's a menu option to enable/disable srm memory for comparison purposes. If you can compile, please test.

https://bitbucket.org/hudson/magic-lantern/pull-request/540/update-raw_rec-and-mlv_rec-to-use-new-srm/diff
Great idea, especially for testing, so much easier to test it out!
Title: Re: SRM job memory buffers
Post by: Rewind on July 12, 2014, 05:49:56 PM
This version (https://bitbucket.org/dmilligan/magic-lantern/commits/968258312ee5447ebe8dd4eedd6ff6ce12181337?at=unified) of raw_rec is slower than previous (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60).

results for 650D, 23.976 fps, GD off:

Resolution               First Implementation              This version
1568 x 656                62 s                                           50 s
1600 x 672                32 s                                           27 s
1728 x 720                12 s                                           10 s
Title: Re: SRM job memory buffers
Post by: escho on July 12, 2014, 06:12:36 PM
Short test with my 600D, using mlv_rec:

Code: [Select]
Recording 1728 x 972 until 1st skipped frame

1st try: without SRM: 28 frames, with SRM: 61 frames
2nd try: without SRM: 31 frames, with SRM: 59 frames

Code: [Select]
Recording 1728 x 972, 10s, skipped frames allowed

1st try: without SRM: 79 frames, with SRM: 110 frames

Really nice

Edgar
Title: Re: SRM job memory buffers
Post by: dmilligan on July 12, 2014, 10:41:40 PM
did some more tests, in general mlv_rec is outperforming raw_rec by a few frames in whatever mode

also recording times for crop mode are not quite as good, though this doesn't seem to be related to buffer space, looking at the trace, the same amount of buffer space is available, 353MB in 126 slots for 1728x972
Title: Re: SRM job memory buffers
Post by: Audionut on July 13, 2014, 01:32:08 AM
Quick test on 5D3.123.

MLV sound/Global Draw ON
No other hacks or card spanning.
Lexar 16GB 1000x
1920x1080x23.976
Default 13seconds
SRM 31seconds


1920x1280x23.976
Default 4 seconds
SRM 10 seconds

Title: Re: SRM job memory buffers
Post by: SoulChildPaul on July 13, 2014, 10:41:44 AM
Can anybody compile for 5D3.113 and .123 with mk11174 changes for me to test please
Title: Re: SRM job memory buffers
Post by: Andy600 on July 13, 2014, 01:02:11 PM
I think  :-\ these are the SRM stubs for the 50D

Code: [Select]
/** SRM JobMem **/
NSTUB(0xFF0A212C,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0A247C,  SRM_FreeMemoryResourceFor1stJob)

but I'm totally lost trying to find CreateMemorySuite and DeleteMemorySuite

Also, I'm not seeing 'SRM Job total' in Free Memory. I cloned the main repo and compiled a new build for the first time in months but module building fails with [module_strings.h] Error 1. Didn't compile from the SRM branch or have Python docutils installed  ::)

Found what I thought were the MemorySuite stubs but SRM memory shows 0x0 so they can't be correct.
Title: Re: SRM job memory buffers
Post by: josepvm on July 13, 2014, 03:27:16 PM
I think  :-\ these are the SRM stubs for the 50D
...
but I'm totally lost trying to find CreateMemorySuite and DeleteMemorySuite

I have found them for 500D, that should be quite similar, so I can give you some tips.

Search for the string "PackMemory\PackMem.c"

CreateMemorySuite should be a function like this one (500D):

 
Code: [Select]
loc_ff06c0d0: ; 18 refs
ff06c0d0: e92d41f0 push {r4, r5, r6, r7, r8, lr}
ff06c0d4: e1a07000 mov r7, r0
ff06c0d8: e3a00018 mov r0, #24
ff06c0dc: e1a08002 mov r8, r2
ff06c0e0: e1a06001 mov r6, r1
ff06c0e4: ebfeb3a1 bl loc_ff018f70
ff06c0e8: e1b04000 movs r4, r0
ff06c0ec: 059f212c ldreq r2, [pc, #300] ; ff06c220: (00000225)
ff06c0f0: 051f1158 ldreq r1, [pc, #-344] ; ff06bfa0: (ff06b860)  **"PackMemory\PackMem.c"
ff06c0f4: 028f0f47 addeq r0, pc, #284 ; ff06c218: (69755370)  *"pSuite"
ff06c0f8: 0bfea30f bleq loc_ff014d3c

And DeleteMemorySuite should be like this one:
Code: [Select]
loc_ff06beec: ; 53 refs
ff06beec: e92d47f0 push {r4, r5, r6, r7, r8, r9, sl, lr}
ff06bef0: e51f164c ldr r1, [pc, #-1612] ; ff06b8ac: (00002f60)
ff06bef4: e1a08000 mov r8, r0
ff06bef8: e5900000 ldr r0, [r0]
ff06befc: e5911000 ldr r1, [r1]
ff06bf00: e1500001 cmp r0, r1
ff06bf04: 159f20b0 ldrne r2, [pc, #176] ; ff06bfbc: (00000239)
ff06bf08: 124f1e6b subne r1, pc, #1712 ; ff06b860: (6b636150)  *"PackMemory\PackMem.c"
ff06bf0c: 128f00ac addne r0, pc, #172 ; ff06bfc0: (75537349)  *"IsSuiteSignature( hSuite )"
ff06bf10: 1bfea389 blne loc_ff014d3c

Quote
Also, I'm not seeing 'SRM Job total' in Free Memory.

Are you building the srm-memory branch?. The unified branch does not show the SRM values in FreeMemory.
Title: Re: SRM job memory buffers
Post by: josepvm on July 13, 2014, 03:43:19 PM
Found what I thought were the MemorySuite stubs but SRM memory shows 0x0 so they can't be correct.

Double check this ones:

Quote
Code: [Select]
/** SRM JobMem **/
NSTUB(0xFF0A212C,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0A247C,  SRM_FreeMemoryResourceFor1stJob)

They do not contain the strings "SRM_FreeMemoryResourceForDDD" or "SRM_AllocateMemoryResourceForDDD" but are called from places where these strings are present.

for SRM_AllocateMemoryResourceFor1stJob (500D) :
Code: [Select]
loc_ff0a8c18:
ff0a8c18: e2841b03 add r1, r4, #3072 ; 0xc00
ff0a8c1c: e59f0194 ldr r0, [pc, #404] ; ff0a8db8: (ff0a820c)
ff0a8c20: e281108c add r1, r1, #140 ; 0x8c
ff0a8c24: ebfe06d7 bl loc_ff02a788           // ---->>> This is the function
ff0a8c28: e59b0034 ldr r0, [fp, #52] ; 0x34
ff0a8c2c: e3a01064 mov r1, #100 ; 0x64
ff0a8c30: ebff0326 bl loc_ff0698d0
ff0a8c34: e3500000 cmp r0, #0
ff0a8c38: 11a03000 movne r3, r0
ff0a8c3c: 128f2f5e addne r2, pc, #376 ; ff0a8dbc: (5f4d5253)  *"SRM_AllocateMemoryResourceFor1stJob failed [%#x]"

And for SRM_FreeMemoryResourceFor1stJob (500D):

Code: [Select]
loc_ff0a9424:
ff0a9424: e5900c8c ldr r0, [r0, #3212] ; 0xc8c
ff0a9428: e3500000 cmp r0, #0
ff0a942c: 13a02000 movne r2, #0
ff0a9430: 13a01000 movne r1, #0
ff0a9434: 1bfe0caf blne loc_ff02c6f8      // ------->>> This is the function
ff0a9438: ebff0185 bl loc_ff069a54
ff0a943c: e51f1760 ldr r1, [pc, #-1888] ; ff0a8ce4: (00003f00)
ff0a9440: e591001c ldr r0, [r1, #28]
ff0a9444: e2500001 subs r0, r0, #1
ff0a9448: e581001c str r0, [r1, #28]
ff0a944c: 1a000004 bne loc_ff0a9464
ff0a9450: ebff1018 bl loc_ff06d4b8
ff0a9454: e28f2f72 add r2, pc, #456 ; ff0a9624: (6c656b53)  *"SkeltonJob Done"
ff0a9458: e3a01016 mov r1, #22
ff0a945c: e3a0008f mov r0, #143 ; 0x8f
ff0a9460: ebfef58c bl loc_ff066a98

In this one, search for "SkeltonJob"
Title: Re: SRM job memory buffers
Post by: Andy600 on July 13, 2014, 03:45:18 PM
@josepvm - thanks for your help :)

I'm compiling from the SRM branch now - wasn't before  ::)

This is the only one I can see as possible for CreateMemorySuite (it has 'ldreq' and I'm looking for similarities to what you posted)

Code: [Select]
loc_ff069068: ; 3 refs
ff069068: e92d41f0 push {r4, r5, r6, r7, r8, lr}
ff06906c:    e1b07002 movs r7, r2
ff069070: e1a08001 mov r8, r1
ff069074: e1a05000 mov r5, r0
ff069078: 059f114c  ldreq r1, [pc, #332] ; ff0691cc: (ff86895c)  **"PackMemory\PackMem.c"
ff06907c:         059f217c  ldreq r2, [pc, #380] ; ff069200: (000001f7)
ff069080: 028f0f5f         addeq r0, pc, #380 ; ff069204: (50282821)  *"!((PVOID)hClientPackHeap == NULL)"
ff069084: 0bfeae8d bleq loc_ff014ac0

for DeleteMemorySuite could it be:

Code: [Select]
ff068fb8: e92d47f0 push {r4, r5, r6, r7, r8, r9, sl, lr}
ff068fbc: e51f1624 ldr r1, [pc, #-1572] ; ff0689a0: (00002e7c)
ff068fc0: e1a08000 mov r8, r0
ff068fc4: e5900000 ldr r0, [r0]
ff068fc8: e5911000 ldr r1, [r1]
ff068fcc: e1500001 cmp r0, r1
ff068fd0: 159f2208 ldrne r2, [pc, #520] ; ff0691e0: (00000236)
ff068fd4: 124f1d1a subne r1, pc, #1664 ; ff06895c: (6b636150)  *"PackMemory\PackMem.c"
ff068fd8: 128f0f81         addne r0, pc, #516 ; ff0691e4: (75537349)  *"IsSuiteSignature( hSuite )"
ff068fdc: 1bfeaeb7 blne loc_ff014ac0

not sure which addresses to try though....I'm a total newb at this bit
Title: Re: SRM job memory buffers
Post by: Andy600 on July 13, 2014, 03:58:14 PM
ah, just saw your next post

Code: [Select]
ff8a231c:    e2811008 add r1, r1, #8
ff8a2320: ebfe23cb bl loc_ff82b254
ff8a2324: e5970028 ldr r0, [r7, #40] ; 0x28
ff8a2328: e3a01064 mov r1, #100 ; 0x64
ff8a232c:         ebff1180 bl loc_ff866934 <------------------------ this is referenced all the time when I search for SRM_AllocateMemoryResource so that must SRM_AllocateMemoryResourceFor1stJob?
ff8a2330: e3500000 cmp r0, #0
ff8a2334: 11a03000 movne r3, r0
ff8a2338: 124f2e3d subne r2, pc, #976 ; ff8a1f70: (424f4a5b)  *"[JOB ERROR] SRM_AllocateMemoryResourceFor1stJob failed [%#x]"
Title: Re: SRM job memory buffers
Post by: josepvm on July 13, 2014, 03:59:41 PM
DeleteMemorySuite looks Ok, but I'm not equally convinced about CreateMemorySuite.

Look for a similar structure in an address a little bit further:  0xFF069118 or near this value.
I have observed in 500D and in 550D a similar offset between DeleteMemorySuite and CreateMemorySuite. 
Title: Re: SRM job memory buffers
Post by: josepvm on July 13, 2014, 04:04:05 PM

Code: [Select]
ff8a231c:    e2811008 add r1, r1, #8
ff8a2320: ebfe23cb bl loc_ff82b254
ff8a2324: e5970028 ldr r0, [r7, #40] ; 0x28
ff8a2328: e3a01064 mov r1, #100 ; 0x64
ff8a232c:         ebff1180 bl loc_ff866934 <------------------------ this is referenced all the time when I search for SRM_AllocateMemoryResource so that must SRM_AllocateMemoryResourceFor1stJob?

Not this one. The previous call, three lines before, is the stub you are searching for :  loc_ff82b254
Title: Re: SRM job memory buffers
Post by: mk11174 on July 13, 2014, 04:32:13 PM
Can anybody compile for 5D3.113 and .123 with mk11174 changes for me to test please
I can only do the 5D3.113 because it is in the source I have.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul13.5D3113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul13.5D3113.zip)
Title: Re: SRM job memory buffers
Post by: keikun007 on July 13, 2014, 05:56:07 PM
600d and 650d are giving better results than 60d? how come?
Title: Re: SRM job memory buffers
Post by: a1ex on July 13, 2014, 08:55:09 PM
I'm back, glad to see the progress. Just give me some time to digest the new stuff, and I'll handle the pull requests in the next few days.
Title: Re: SRM job memory buffers
Post by: Andy600 on July 13, 2014, 09:13:38 PM
@josepvm or @a1ex - can you please specify the strings I need to look for for each stub needed (I think there are 5 missing stubs in total for SRM to work on the 50D!?) I'm getting a bit lost when it comes to MemoryChunk etc and still don't really understand what I'm looking at despite many reads of this and the other thread :-[

So for instance, for the
Code: [Select]
SRM_AllocateMemoryResourceFor1stJob stub , I searched for SRM_AllocateMemoryResourceFor1stJob failed [%#x] in Notepad++ which got me close (although  I was still wrong) and I presume
Code: [Select]
ff82b254 is correct as you indicated. I'm trying to see the pattern here because it doesn't make sense in my head yet.

Thanks again for your help :). I'm not sure if there are any devs still working on the 50D so I would like to at least try and keep it up to date with my extremely limited code knowledge.
Title: Re: SRM job memory buffers
Post by: ayshih on July 13, 2014, 11:24:05 PM
I'm not sure if there are any devs still working on the 50D so I would like to at least try and keep it up to date with my extremely limited code knowledge.
I'm still alive!  I just got back from a four-week work trip, so I'll need to get back up to speed on ML and then help you track down stubs.
Title: Re: SRM job memory buffers
Post by: ayshih on July 14, 2014, 01:30:44 AM
I've submitted a PR (https://bitbucket.org/hudson/magic-lantern/pull-request/543/srm-stubs-for-50d/diff) with the 50D stubs.  Thanks to @Andy600 and @josepvm for getting that ball rolling!  Here are the screenshots (outside LV, LV, and don't click me):

(https://www.dropbox.com/s/lje8j5ciodem5hm/srm-memory_outsideLV.png?dl=1)
(https://www.dropbox.com/s/5ux65ux5dkm63fh/srm-memory_insideLV.png?dl=1)
(https://www.dropbox.com/s/trnx58joostncd3/srm-memory_test.png?dl=1)

However, I starting taking a few pictures during the "don't click me" test, and on the second, I hit an "Err 80".  Has anyone else encountered any issues?
Title: Re: SRM job memory buffers
Post by: ayshih on July 14, 2014, 03:06:57 AM
However, I starting taking a few pictures during the "don't click me" test, and on the second, I hit an "Err 80".  Has anyone else encountered any issues?
Hmm, I haven't been able to reproduce this with further testing...
Title: Re: SRM job memory buffers
Post by: Andy600 on July 14, 2014, 04:27:35 AM
216mb extra! Nice :)

Just tried MLV and got this message:

Code: [Select]
Hack error at 0
expected e3a00032, got e1a00000

It recorded about the same amount of frames as without SRM. Also, you can get into crop mode only once and the zoom buttons become non functional requiring a battery pull.

And, has raw REC button been remapped? I get h264 recording with the SET button and raw with the LV (blue) button.


UPDATE: backing out of https://bitbucket.org/hudson/magic-lantern/commits/964f3df (https://bitbucket.org/hudson/magic-lantern/commits/964f3df) seems to have fixed all the problems :) Thanks a1ex!
Title: Re: SRM job memory buffers
Post by: Audionut on July 14, 2014, 04:58:57 AM
I noticed some undesirable behaviour with crop mode on a 5D3.

But haven't had the time to test things thoroughly.
Title: Re: SRM job memory buffers
Post by: dsManning on July 14, 2014, 05:15:22 PM
Hmm, I haven't been able to reproduce this with further testing...

Shots during "Don't Click Me" test worked fine for me on the 50D.  After trying to shoot .mlv with extra memory worked until card was full (forgot to clear it first).  Haven't seen error 80 yet, but trying on a clean 1000x card now to see how many frames I can get in crop mode.

Thanks for getting this going on the 50D, I like Andy was having a hard time looking for the stubs until it was cleared up a few posts above.
Title: Re: SRM job memory buffers
Post by: a1ex on July 15, 2014, 09:24:29 AM
Had a quick play with the raw recorders.

Good news: the memory hack is no longer needed on 5D3. The total memory with hack + SRM, usable for raw recording, is actually lower than just with SRM. This cleans up the code quite well, and removes the annoyances of the old memory hack.

Give it a quick try, and if all goes well, it will be in the nightly builds tonight or tomorrow. Including 6D and 500D.
Title: Re: SRM job memory buffers
Post by: Levas on July 15, 2014, 04:25:15 PM
Had a quick play with the raw recorders.

Good news: the memory hack is no longer needed on 5D3. The total memory with hack + SRM, usable for raw recording, is actually lower than just with SRM. This cleans up the code quite well, and removes the annoyances of the old memory hack.

Give it a quick try, and if all goes well, it will be in the nightly builds tonight or tomorrow. Including 6D and 500D.

According to some tests on the 6d, memory hack VS SRM made no difference

Activating memory hack increase record time to match 2014Jul09's times. (or 1 sec. less)
On pushing the record button, LiveView shuts down for a moment.

The memory hack gives an additional of 100MB, the SRM gives 3x 34MB on 6d.
How can that SRM only gives more memory on 5d3 then memory hack + SRM.
Am I missing something here ?
Title: Re: SRM job memory buffers
Post by: a1ex on July 15, 2014, 04:42:19 PM
The memory is managed in different ways by Canon in different modes (LV, outside LV, RAW, JPEG and so on). Some of it is reserved for shoot_malloc (for compressing photo data after capture), some of it is for SRM (for uncompressed just-captured raw photo data), some is for the operating system, some is for LiveView and so on. The largest ones are shoot, SRM and LiveView buffers, and their sum is more or less constant.

This means, the benefit for 6D is just cleaner code and a smaller delay before starting to record.
Title: Re: SRM job memory buffers
Post by: Levas on July 15, 2014, 07:03:49 PM
The memory is managed in different ways by Canon in different modes (LV, outside LV, RAW, JPEG and so on). Some of it is reserved for shoot_malloc (for compressing photo data after capture), some of it is for SRM (for uncompressed just-captured raw photo data), some is for the operating system, some is for LiveView and so on. The largest ones are shoot, SRM and LiveView buffers, and their sum is more or less constant.

This means, the benefit for 6D is just cleaner code and a smaller delay before starting to record.

Just to be sure if I understand this correct:
Are you saying we can't have best of both worlds, 100MB from the hack and 3 x 34 MB SRM memory ?
In video mode on 6d we have:
Shoot total 151MB and SRM total 3x34MB. (Total of 253MB)
In photo mode we have:
Shoot total 255MB and SRM total 4x34MB. (total of 391MB)

Don't know if I'm right, but my guess is there's enough total memory in the 6d to have (some) best of both worlds.
Right know, with SRM only, without the extra memory hack, there's 140MB hold back, live view of course need some of it, but I don't expect live view needs all 140MB.
Title: Re: SRM job memory buffers
Post by: a1ex on July 15, 2014, 07:47:29 PM
There are more allocators, not just shoot_malloc and SRM. If you trace the references to the worker function from AllocateMemory, you'll find around 10 or 20, but most of them are small. In addition, there are many small allocators in the resource manager. Feel free to reverse engineer them and squeeze even more RAM.

The total RAM size in these cameras is 512 MB, and 256 MB on most Rebels. That colorful line from the free memory dialog shows what we know about it, and from unknown areas, it tries to guess what's used and what might be free. A better guess can be obtained if you compile with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (but this delays the boot process a bit).

BTW, what's the reason for quoting the entire post right above yours? It would just double the thread length, if everybody would do the same.
Title: Re: SRM job memory buffers
Post by: Levas on July 15, 2014, 08:07:25 PM
Sorry about the quoting habit  :-[   
Looks indeed very stupid, if it's exactly one post under it   :D

The thing I'm worried about is that the extra memory hack is declared obsolete and will be deleted in the current builds.
I'm only worried about this because it looks like it can be used on the 6d aside with the SRM code.

Would love to test if the extra memory hack can be used in combination with the SRM code, but I don't have any coding skills.
And from previous post it seems, the build we tested on the 6d with the SRM code in it, somehow disables the extra memory hack.

If it turns out that extra memory hack and SRM code don't work together on the 6d, I see no reasons to not clean up the code by deleting the extra memory hack.
The way I see it, as a person without coding skills  ???, this could be a relatively easy way to get some extra memory for raw video buffer.
But then again, I have no coding skills, so maybe things are a lot harder to implement...

Title: Re: SRM job memory buffers
Post by: MA Visuals on July 15, 2014, 08:13:42 PM
Does this mean that the newly found SRM memory will be automatically be made available transparently to raw recording with no liveview resetting?  I was never able take advantage of the memory hack on the 5D3 because the long delay of resetting liveview caused my hdmi monitor to miss the initial 4 seconds of recording as the monitor reset itself.  This alone would have a huge impact for hdmi shooters.  Also, will small hacks still be available/useful?
Title: Re: SRM job memory buffers
Post by: a1ex on July 15, 2014, 08:35:40 PM
Would love to test if the extra memory hack can be used in combination with the SRM code, but I don't have any coding skills.

Try compiling 3d6a945, or ask Audionut here (http://www.magiclantern.fm/forum/index.php?topic=12608.0) to do it for you.

Quote
Does this mean that the newly found SRM memory will be automatically be made available transparently to raw recording with no liveview resetting?

Yes.
Title: Re: SRM job memory buffers
Post by: josepvm on July 15, 2014, 10:27:45 PM
I have been testing the recent additions to the srm-memory branch on 500D, today's build.

Shooting with mlv-rec module, using a low resolution, 864x486, to allow continuous recording on SD card.


I have noticed this issue: when using SRM memory, AF (with back focus button, "*") does not work at all during recording.

With exactly the same settings but disabling SRM memory use, AF works again.


So, maybe SRM memory is used for autofocusing by the Canon firmware?




Title: Re: SRM job memory buffers
Post by: a1ex on July 15, 2014, 10:30:42 PM
Does it help if you comment out the gui_uilock calls from exmem.c?

(ML blocks the shutter button so you can't take pictures by mistake and get err70)
Title: Re: SRM job memory buffers
Post by: josepvm on July 15, 2014, 11:41:33 PM
Does it help if you comment out the gui_uilock calls from exmem.c?

(ML blocks the shutter button so you can't take pictures by mistake and get err70)

Yes, it helps.  AF works now with SRM memory enabled.

Thanks, A1ex.

(some additional info: I have CFn functions adjusted to focus always with back focus button, not the shutter button)
 

Title: Re: SRM job memory buffers
Post by: josepvm on July 16, 2014, 10:38:47 AM
I have tested the new Nightly Build for 500D (Nightly.2014July16).

It still has no option to use SRM memory for raw video, but the shutter button lock is active, and AF does not work for video.

With the srm-memory branch build you have the option to disable SRM and then have AF working. In this nightly there is no such option, and no AF  :(


I wonder if a better solution would be possible here, to avoid taking pictures during video shooting but allowing AF with backfocus button.

Title: Re: SRM job memory buffers
Post by: a1ex on July 16, 2014, 10:55:49 AM
Autofocus works just fine here with the nightly. I didn't include the SRM yet, will do tonight.

The safeguard is not really to prevent taking pictures while shooting video; these routines are general-purpose, other code could make use of them, and I want the backend to prevent the race conditions with picture taking code (that's why the "don't click me" test tells you to take pictures while it's allocating and freeing memory).

Food for thought: when allocating the entire SRM memory, Canon code already locks the UI (it prints BUSY), but the autofocus still works. Figure out how it does that...

edit: I think it's solved, tested on 5D3: https://bitbucket.org/hudson/magic-lantern/commits/92536576a20d
Title: Re: SRM job memory buffers
Post by: josepvm on July 16, 2014, 12:55:41 PM
edit: I think it's solved, tested on 5D3: https://bitbucket.org/hudson/magic-lantern/commits/92536576a20d

Tested this solution on 500D, using srm-memory branch to build, works Ok.

I need to disable "Extra hacks" to get the AF working. When disabling the hacks, the AF works fine with SRM enabled.

I have done the "Don't click me" test with this changes, as you suggested, and I am able to take shots, record video and AF during the test.
But I don't know exactly how this test works and how it should be used, sorry   ??? In first place, I do not know if the test ends after a successfull run, or if it runs continously until I shut down the camera, as I did  ::) 


Disabling the "Extra hacks" is the key, also, for the Nightly build I tested earlier.  Now I've tested it again, disabling the Extra hacks, and AF works (no SRM, of course).

When I tested the srm-memory branch yesterday, I disabled the hacks also, because some days ago, testing mlv-rec, I got errors when starting video recording, and Greg suggested me to disable them.

But today, whith the new Nightly build, I saw mlv-rec worked without disabling the hacks, so I left them enabled ... but that's the reason for AF not working.






Title: Re: SRM job memory buffers
Post by: a1ex on July 16, 2014, 01:02:58 PM
The test simply checks if these routines play nice with Canon picture taking process. If you would comment out the gui lock tricks in exmem.c, you would get either lockups or err70 during the test.
Title: Re: SRM job memory buffers
Post by: Levas on July 16, 2014, 06:49:06 PM
Try compiling 3d6a945, or ask Audionut here (http://www.magiclantern.fm/forum/index.php?topic=12608.0) to do it for you.

@ Alex,
Audionut compiled 3d6a945 for me to test on the 6d.

Both options(mem hack and SRM) together didn't result in any more  memory... ::) or actually I mean  :'(

Did some testing and looked at the MLV file size
Both options = the same file size as Memory hack only 
SRM option only = slightly higher file size vs memory hack option and both options.
So it  looks like both options results in gaining the memory from the memory hack, but doesn't give us any SRM memory. ???

So you were right  :D
This means, the benefit for 6D is just cleaner code and a smaller delay before starting to record.
Title: Re: SRM job memory buffers
Post by: a1ex on July 16, 2014, 10:49:04 PM
Alright folks, the SRM memory changes are now in the nightly builds. The good old memory hack is gone, as well as a bunch of older code hacks.

On 5D3 and 6D you have the same total memory as before (+/- a few MB), but without having to blank the LiveView. On other cameras, you no longer need to switch to small JPEG or similar configurations. And if you are lucky, you should get a nice improvement in recording times at certain resolutions.

I don't know the exact figures - you tell me.

This is not the end of memory research, but I'll stop here for now. Feel free to investigate the factory mode, or try to find some more allocators from the resource manager.

Thanks to all the contributors from this thread!
Title: Re: SRM job memory buffers
Post by: dubarry on July 17, 2014, 12:21:37 AM
EVERYTHING IS AWESOME!!!  (Yep I'm a parent)

60D    1728x736 2.35:1  24fps 254 frames @ 10 seconds
          1728x972 16x9    24fps 170 frames @ 6 seconds

Honestly 10 seconds has been my dream for about what 2.5/3 years now.

That's bloody spectacular
thank you, thank you, thank you all.
Title: Re: SRM job memory buffers
Post by: MA Visuals on July 17, 2014, 12:44:46 AM
I can confirm a big increase in performance after the SRM update.  Was getting 15-20 seconds 30fps... now getting 30-40 seconds.  Double the recording time.  At least for me, this makes 30fps a more stable and reliable frame rate.  Also, 48 fps (squashed mode) went from 8 to 15 seconds. Again, much more dependable where before, it was borderline for me. These new times are with raw_rec.  I get similar relative increases with mlv_rec, just shorter actual recording times due to the extra overhead of the mlv format.   

Changes like this really do make a big difference.  For many HDMI monitor users, this eliminates the Liveview blackout that happens with the old memory hack prior to recording.  I could never deal with that during a production so never realized the benefits of that hack.  Thank you Alex and for any others who contributed to this.

The above was using a 5D3 and a SmallHD DP4 monitor.
Title: Re: SRM job memory buffers
Post by: N/A on July 17, 2014, 12:54:37 AM
Today's nighly build for 7d failed.  :-[

Changeset ab901b1
Title: Re: SRM job memory buffers
Post by: dmilligan on July 17, 2014, 01:10:02 AM
Today's nighly build for 7d failed.  :-[
looks like some missing stubs:
Code: [Select]
exmem.c:(.text+0x748): undefined reference to `CreateMemorySuite'
exmem.c:(.text+0x780): undefined reference to `CreateMemoryChunk'
exmem.c:(.text+0x7a8): undefined reference to `AddMemoryChunk'
exmem.c:(.text+0x88c): undefined reference to `DeleteMemorySuite'
Title: Re: SRM job memory buffers
Post by: dannyqt18 on July 17, 2014, 01:14:29 AM
simple MLV test on a 5D MARK ii with a 32gb 1000x Lexar:

NO SRM: shot at 1856x1044, 23.976fps for 16 sec

WITH SRM: shot at 1856x1044, 23.976fps for 39 sec (first time) 43 sec (second time)

very big leap for me, will test other resolutions and FPS times in the coming days.

Awesome job!!
Title: Re: SRM job memory buffers
Post by: N/A on July 17, 2014, 01:18:28 AM
looks like some missing stubs:
Code: [Select]
exmem.c:(.text+0x748): undefined reference to `CreateMemorySuite'
exmem.c:(.text+0x780): undefined reference to `CreateMemoryChunk'
exmem.c:(.text+0x7a8): undefined reference to `AddMemoryChunk'
exmem.c:(.text+0x88c): undefined reference to `DeleteMemorySuite'
Gotcha, how does one go about finding these stubs? I'll try do it tonight while I'm not busy.
Nevermind, found the tutorial. Yeah, it's going to take a lot longer than one night to learn that process. Poo.
Title: Re: SRM job memory buffers
Post by: Pelican on July 17, 2014, 11:02:18 AM
Alex already committed these stubs...
Title: Re: SRM job memory buffers
Post by: N/A on July 17, 2014, 03:41:26 PM
Yup, just noticed that. But I did learn how to compile, so that's a plus.

372 MB buffer, 13 seconds of 2496x1044 MLV footage. GD off, no sound. Not too shabby!  8)

Card write speeds on Lexar 1000x are still a bit inconsistent, on some recordings I get a steady 79-81 MB/s, and sometimes it drops to 65-75 MB/s. Maybe a freshly formatted card would help.
Title: Re: SRM job memory buffers
Post by: N/A on July 17, 2014, 08:59:24 PM
Is SRM also beneficial for h.264 recording? I would think that all the extra buffer memory would assist in using higher bitrates, but anything over 100 mbps instantly buffers out. Noticing a lot of instances of the buttons becoming unresponsive after switching back and forth between the ML menu and LV, although the display is still active. Requires a battery pull to fix.
Title: Re: SRM job memory buffers
Post by: dmilligan on July 17, 2014, 10:59:28 PM
Is SRM also beneficial for h.264 recording?
Canon is going to already be using all the memory they need for h.264.

Just to clarify something: we did not find new or extra memory in the cameras, we found a new memory allocator. SRM is just the name Canon gave one of their memory allocator functions (a memory allocator basically manages and keeps track of what tasks are using what memory).

In other words we found a new way to reserve memory for ourselves (ML) and prevent Canon tasks from using it (we can't use memory unless we can be sure that another task isn't going to try to use it, for obvious reasons). Before now, we "knew" about this memory (i.e. we know what valid range of RAM address are), we just weren't sure if/when Canon was using it, and we didn't know how to reserve it for ourselves (so we couldn't use it). Canon tasks certainly already know about this allocator and make use of it as they please.
Title: Re: SRM job memory buffers
Post by: N/A on July 18, 2014, 03:35:46 AM
That's what I had assumed, even though I was hoping otherwise. Cheers on the explanation.
Title: Re: SRM job memory buffers
Post by: x4kep on July 19, 2014, 11:30:03 AM
Canon 50D, card Transcend 16GB 600x TS16GCF600
Magic Lantern build jul 18 2014
Raw_rec (mlv_rec always gives the worst result)
Crop mode 1920x1080 24fps

SRM on
no raw, s-jpg2
GD on - 19s
GD off - 25s

s-raw2, no jpg
GD on - 19s
GD off - 27s

SRM off
no raw, s-jpg2
GD on - 11s
GD off - 15s

s-raw2, no jpg
GD on - 10s
GD off - 13s
always two pink frame each shot

Tragic Lantern build mar 3 2014 (no SRM)
Max Resolution
full mode - 1584x1056 (16:9 1584x892)
crop mode - 2000x1080

Raw_rec (mlv_rec always gives the worst result)
Crop mode 1920x1080 24fps
no raw, s-jpg2
GD on - 13s
GD off - 17s
no pink frames or other artifacts

ML raw module with SRM (jul 18 2014) give better recording time, but have pink frames and lower max resolution, than TL raw module (mar 3 2014)