Interesting find: if you call FA_CreateTestImage more than once, you need to call FA_DeleteTestImage in the same order (otherwise it will fail):
So, this is OK:
struct JobClass * job1 = (void*) call("FA_CreateTestImage");
struct JobClass * job2 = (void*) call("FA_CreateTestImage");
call("FA_DeleteTestImage", job1);
call("FA_DeleteTestImage", job2);
but this will fail:
struct JobClass * job1 = (void*) call("FA_CreateTestImage");
struct JobClass * job2 = (void*) call("FA_CreateTestImage");
call("FA_DeleteTestImage", job2);
call("FA_DeleteTestImage", job1);
Another nice trick: each job object has a contiguous raw image buffer, usually more than 32MB. With shoot_malloc, the maximum contiguous size is less than 32MB (depending on your camera). To access this buffer:
struct JobClass * job = (void*) call("FA_CreateTestImage");
void* buf = (void*) call("FA_GetCrawBuf", job);
/* use your buffer */
call("FA_DeleteTestImage", job);
So, it looks like we have a way to allocate some more memory, which is very useful for raw video. How much, I don't know yet.
The underlying call for memory allocation is SRM_AllocateMemoryResourceFor1stJob (called from CreateSkeltonJob, easy to find from strings). To double-check, its second argument has an offset that matches the one from GetImageBuffer.
To free the memory, call SRM_FreeMemoryResourceFor1stJob (it's called from DeleteSkeltonJob, but has no strings - identify it from its first argument).
Example for 5D2:
GetImageBuffer returns (arg0->off_0x4)->off_0xC68, aka MEM(MEM(arg0 + 0x4) + 0xC68).
SRM_AllocateMemoryResourceFor1stJob(cbr_function, 0xC68 + something)
if take_semaphore(*0x3CC0, 0x64) != 0:
DebugMsg(0, 22, "[JOB ERROR] SRM_AllocateMemoryResourceFor1stJob failed [%#x]', ...)
if (arg0->off_0x4)->off_0xC68 != 0:
SRM_FreeMemoryResourceFor1stJob((arg0->off_0x4)->off_0xC68, 0, 0)
Notes and caveats:
- you don't need to enter factory mode to use this
- this is independent from shoot_malloc (you can use both, and using one doesn't affect the size or free space from the other)
- you can't tell how much to allocate; the buffer size is fixed (it tells you how much you've got)
- as long as you have one such buffer allocated, you can't take regular pictures (ERR70), but LiveView works fine
- if there's no more free RAM, the allocation will be delayed until you free something failed calls will be dropped, just wait for 100ms
- on 5D3, you can allocate 3 such buffers at a time (39MB x 3 = 117MB extra). On 5D2 and 550D, only 2.
Syntax:
void SRM_AllocateMemoryResourceFor1stJob(void (*callback)(uint32_t opaque, void* raw_buffer, uint32_t raw_buffer_size), uint32_t opaque);
void SRM_FreeMemoryResourceFor1stJob(void* raw_buffer, uint32_t unk1_zero, uint32_t unk2_zero);
@A1ex Reply #22
Strange that the order of deleting 'jobs' make a difference, I guess it relates to how the firmware prefers to manages memory?
Anyway... Mo' memory for raw video = :D
Yeah, that was unexpected. The resource manager is quite complex; maybe g3gg0 will have a better understanding of why this happens.
Quote from: a1ex on July 02, 2014, 10:31:20 AM
- on 5D3, you can allocate 3 such buffers at a time (39MB x 3 = 117MB extra). On 5D2 and 550D, only 2.
Alex, are these 39MB buffers, additional on the already discovered memory buffers, used for fill-up with raw recording ?
The 6D has about 250MB of memory buffer with raw-recording at the moment.
The 250MB did already sound like the max, according to the point when burst mode slows down with raw photo mode.
raw images are like 35MB and the SD can write 40MB/s to the card.
Although there are reports of 9 seconds of writing to the SD card time to clear the buffer (with fast UHS-I card)in the 6D, so 40MB x 9 seconds is about 360MB of available buffer.
So there are probably 3 x 39MB buffers on the 6D. Would love to have these extra 117 MB for raw recording :D
Quote from: a1ex on July 02, 2014, 10:31:20 AM
- on 5D3, you can allocate 3 such buffers at a time (39MB x 3 = 117MB extra). On 5D2 and 550D, only 2.
i made some changes to mlv_rec to support this new memory interface to make some tests.
it looks very promising. here some results:
5D3 v1.1.3:Memory details: total 277 MiB [exmem 238 MiB,
SRM 39 MiB] in 89 slots
7D v2.0.3:Memory details: total 371 MiB [exmem 182 MiB,
SRM 188 MiB] in 164 slots
600D v1.0.3:Memory details: total 129 MiB [exmem 67 MiB,
SRM 62 MiB] in 44 slots
this is another powerful push. thanks a lot, alex :)
@Geggo,
Is there a way to test the changes you made to the mlv_rec module on a 6D (without the need to compile on my side :-\) ?
At the moment, 6D is not supported in nightly builds... :'(
I believe it's not as simple as put the new mlv_rec.mo file in an older nightly build, did tried that once, but it gives errors at camera startup.
Would love to have some extra memory with MLV recording :)
hm i dont have a recent 6D firmware dump i could use to find the stubs.
These changes are not yet included in any nightly build (we are reverse engineering it as we speak, and sharing the details as soon as we discover them). Please be patient, and hope this will motivate the 6D developers to bring the port up to date.
Some quick tests on 5D3 123, first outside LiveView, second in LV:
(http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3.png) (http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3-lv.png)
Quote from: g3gg0 on July 02, 2014, 09:50:43 PM
7D v2.0.3:
Memory details: total 371 MiB [exmem 182 MiB, SRM 188 MiB] in 164 slots
:o
Such amaze. Much wow.
Quote from: g3gg0 on July 02, 2014, 11:23:08 PM
hm i dont have a recent 6D firmware dump i could use to find the stubs.
Here you are. These are for 6D_113:
NSTUB(0xFF0F0098, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0F40C8, SRM_FreeMemoryResourceFor1stJob)
Thanks, you will probably have to find a few more tonight :)
Quote from: a1ex on July 03, 2014, 08:43:18 AM
Thanks, you will probably have to find a few more tonight :)
No problem I am continuously watching commits on butbucket...
Silent picture discovery is huge.
But at this moment I'm way more excited about the possibility of getting some extra memory for MLV_rec :P
Use MLV_rec a lot for high resolution short clips, these extra MB's mean a lot of extra seconds :D
Okay, here's an attempt to create some friendly wrappers for the new buffers: https://bitbucket.org/hudson/magic-lantern/pull-request/529
User-level API:
- srm_malloc_suite/srm_free_suite: will allocate the entire SRM memory (size autodetected) and return a memory suite, just like shoot_malloc_suite (preferred for raw_rec/mlv_rec)
- malloc/free: will use these buffers when you request very large sizes that can't be handled by shoot_malloc (32MB+); these are one level above srm_malloc_suite
- srm_malloc/srm_free: will use these buffers for any request higher than 25MB (even if the same buffers can be allocated from shoot_malloc)
Caveats:
- once somebody allocated from the SRM buffer, nobody else will be allowed to do so, and you will not be able to take pictures (this is because the backend must free the buffers in exactly the same order as allocated)
- you can, however, use malloc/free from two diferent tasks (first malloc will allocate all the SRM buffers, last free will give all buffers back to the system)
- with malloc, you can free the buffers in any order (the backend will take care of it)
- be careful allocating SRM buffers outside LiveView: you may not be able to enter LiveView until you free them
- depending on camera, some buffer sizes like 27MB may fail with malloc (because it will try shoot_malloc first, and it doesn't report the free space correctly [fixme]); workaround: srm_malloc will work.
For short-term processing that requires huge amounts of RAM, or for RAW recording, these wrappers should be just fine.
Porting notes:
- Stubs required: SRM_AllocateMemoryResourceFor1stJob, SRM_FreeMemoryResourceFor1stJob, CreateMemorySuite, DeleteMemorySuite, CreateMemoryChunk, AddMemoryChunk.
- Check the Free Memory dialog and post a screenshot (I already did for 5D3 123 earlier in this thread).
- Run the test from "don't click me" and post the screenshot created there.
- While this test is running, try taking some pictures (it should not crash, but the allocator may fail - it's normal, it should fail gracefully).
Some more thoughts:
g3gg0 is researching a way to merge two memory suites, so you would get the entire memory allocated in a single suite. With this, there would be - in theory - no need to modify the raw recording code, it would work out of the box.
However, if you enable the memory hack, it will allocate the memory outside LiveView and then get back into LiveView. The problem with this is that, if you allocate the entire RAM outside LV, the camera may not have enough RAM for the LiveView tasks (it happens on 5D3). So, I'd keep two separate allocators for now, and the user code will be responsible for having the proper context before calling (e.g. don't switch modes after allocating).
Nice work. Looking forward to see what can be done with all this extra RAM. :)
So now see following. These are my findings for 6D_113:
NSTUB(0xFF0F0098, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0F40C8, SRM_FreeMemoryResourceFor1stJob)
NSTUB(0xFFCCE714 - RAM_OFFSET, CreateMemoryChunk)
NSTUB(0xFFCCF210 - RAM_OFFSET, AddMemoryChunk)
If you need others then just say so....
I see that this has already been added to nightlies for the 7D, 5D3 and 600D. Any news on when the 550D can be tested?
Nope, it wasn't (g3gg0 added the stubs without creating a branch first, and that triggered the nightly builds without any change in functionality)
Another note for raw recording: calling SRM_ChangeMemoryManagementForFactory might make a difference.
For example, for 550D, in LiveView, you get 75 + 1x31 MB in normal mode, and 43 + 3x31 in factory mode. 30MB extra is not bad, no?
On 5D2, in LiveView, you get 183 + 4x37 in normal mode, and 259 + 2x37 in factory mode (same with jpeg-small). No difference.
On 5D3, you get 163 + 3x39 in normal mode, and 127 + 4x39 in factory mode. No difference.
This might remove the need for picture quality tricks, at least on some cameras. Need to double-check if this is a persistent setting though.
(I'll let you continue the research during the next few weeks - sorry, family needs a little more attention, and it will keep me offline for a while)
1100D, I'm fairly certain these are right:
// NSTUB( ???, CreateMemorySuite)
NSTUB(0xFF071BD0, DeleteMemorySuite)
// NSTUB( ???, CreateMemoryChunk)
NSTUB(0xFF072040, AddMemoryChunk)
/** SRM JobMem **/
NSTUB(0xFF029268, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02BCF0, SRM_FreeMemoryResourceFor1stJob)
I'm having a little trouble with CreateMemorySuite and CreateMemoryChunk, I have some leads:
NSTUB(0xFF0715BC, CreateMemoryChunk)
NSTUB(0xFF071D9C, CreateMemorySuite)
but I'm not sure if these are right, or how to verify them, any hints?
sub_FF071D9C+20: _malloc(24)
sub_FF071D9C+40: sub_FF0142D8('pSuite', 'PackMemory\\PackMem.c', 554)
sub_FF071D9C+100: sub_FF0715BC(arg0, arg1, arg2)
sub_FF0715BC+20: _malloc(40)
sub_FF0715BC+40: sub_FF0142D8('pChunk', 'PackMemory\\PackMem.c', 244)
(https://bitbucket.org/dmilligan/magic-lantern/downloads/FreeMemory1100D.jpg)
the test fails but I guess that's because it's trying to allocate 25MB, and these are only 20MB on the 1100D?
(https://bitbucket.org/dmilligan/magic-lantern/downloads/1100Dsrm_mem_test.jpg)
Any interesting thing happened during the test: I tried to take some pics, they never got recorded to the SD card. If I tried to turn the camera off, it just kept saying "recording images", open the battery door and it says "don't take the battery out!", try to go into LV and it just says "busy...please wait" (maybe I have the stubs wrong?)
changed the test to 19MB, the first buffer works, but the second fails
(https://bitbucket.org/dmilligan/magic-lantern/downloads/FreeMemory60D.jpg)
holy sh*t!!!! 250MB more memory on the 60D!?
EDIT: 6x31MB in LV and Movie mode!
Quote from: g3gg0 on July 02, 2014, 09:50:43 PM
i made some changes to mlv_rec to support this new memory interface to make some tests.
it looks very promising. here some results:
5D3 v1.1.3:
Memory details: total 277 MiB [exmem 238 MiB, SRM 39 MiB] in 89 slots
7D v2.0.3:
Memory details: total 371 MiB [exmem 182 MiB, SRM 188 MiB] in 164 slots
600D v1.0.3:
Memory details: total 129 MiB [exmem 67 MiB, SRM 62 MiB] in 44 slots
this is another powerful push. thanks a lot, alex :)
after posting the missing stubs for 6D (see above), can anyone verify "SRM job total" for 6D? Could I find this out myself in QEMU w/o owning the cam?
Congrats dmilligan 250MB sounds amazing
Another exciting discovery... :D Congrats!!!
Very cool, thanks for taking the time to discover this!
Just to fuel excitement, what sorts of features will probably be able to happen with the extra memory that weren't possible before? Just looking forward to the future, not pushing, totally appreciating what you've done so far and curious about what's in store :D
If I read Alex post from juli 03, it makes no difference for things that use live view, so no memory gain here...(except 30MB extra for the 550d)
If I get it right it seems that live view uses the same amount of memory that is discovered.
Quote from Alex post:
For example, for 550D, in LiveView, you get 75 + 1x31 MB in normal mode, and 43 + 3x31 in factory mode. 30MB extra is not bad, no?
On 5D2, in LiveView, you get 183 + 4x37 in normal mode, and 259 + 2x37 in factory mode (same with jpeg-small). No difference.
On 5D3, you get 163 + 3x39 in normal mode, and 127 + 4x39 in factory mode. No difference.
Quote from: Levas on July 06, 2014, 08:09:27 PM
If I get it right it seems that live view uses the same amount of memory that is discovered.
nope, some of it used by LV but not all of it, in fact it's not even the majority of it is used by LV, on 60D only 1/4 of SRM is used by LV (we get 186MB extra even in LV)
a1ex is talking about factory mode vs. not factory mode when he says "no difference", he's not talking about LV vs. not LV
I made some quick and dirty changes to mlv_rec to use the new memory interface and I was able to get 30-70% longer recording times (link (http://www.magiclantern.fm/forum/index.php?topic=5653.msg120999#msg120999)).
One thing I'll note: when using SRM memory with mlv_rec, "BUSY PLEASE WAIT" is displayed at the top of the screen, however recording works without a hitch. I assume this is because Canon h246 recording wants to use this memory.
:o So there's more memory gained with MLV-recording!
Woohoo :D
Quote from: dmilligan on July 06, 2014, 09:03:00 PM
I made some quick and dirty changes to mlv_rec to use the new memory interface and I was able to get 30-70% longer recording times (link (http://www.magiclantern.fm/forum/index.php?topic=5653.msg120999#msg120999)).
there are my changes: link (http://upload.g3gg0.de/pub_files/4ff199e7f0ac8b5321e37adee738ebd0/2014-07-02_20-56-20_r11391_.diff) (made before alex' updates to mem.c were made)
Hey dmilligan, if you ever get the opportunity I would be very interested to know how many frames you are getting at
1728x736 p24 2.35
This of course is not a priority just if you get the opportunity...let us know.
I want to find these stubs for 650D.
Is there sort of "easy way" to try that?
I mean point me please to some reading about that?
http://www.magiclantern.fm/forum/index.php?topic=12177.0
On the 700D now it shows 154MB allocated when I choose to reserve space and before the mem patch it only said 92MB.
And I gain 1 second on 1920x1080@24p, so instead of 1 second it record 2 seconds.
This was the only test I did just to see if it did anything.
I made sure I formated card before each test as well.
Quote from: dubarry on July 07, 2014, 05:56:46 AM
Hey dmilligan, if you ever get the opportunity I would be very interested to know how many frames you are getting at
1728x736 p24 2.35
This of course is not a priority just if you get the opportunity...let us know.
If you know how much memory is available, you can calculate it yourself, the 60d can write 21MB/s to the SD card, the rest is buffered into the available memory.
Now the resolution you ask for, takes up, 1728 x 736 x 24(frames/second) x 14(14bit canon raw) / 8 (conversion bits to bytes) /1000000(conversion bytes to MegaBytes) = 53,4 MB/second
1728*736*p24 resolution takes up 53,4 MB/second
I'm not sure how much free memory the 60d has, my guess is about 350MB now ?
That would make 53,4MB/second - 21MB/second(SD card writing speed in canon 60d) = 32,4MB/second that needs to be buffered every second.
350MB / 32,4MB = 11 Seconds of raw video
And after you filled up this memory, it takes up to 350MB/21MB = 17 seconds time to clear/write the complete filled up buffer to the SD card.
Quote from: Rewind on July 07, 2014, 08:00:01 AM
I want to find these stubs for 650D.
Is there sort of "easy way" to try that?
I mean point me please to some reading about that?
These are for my 700D
/** SRM JobMem **/
NSTUB(0xFF0E6C78, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0EA848, SRM_FreeMemoryResourceFor1stJob)
So these should be for your 650D
/** SRM JobMem **/
NSTUB(0xFF0E6AE4, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0EA5B8, SRM_FreeMemoryResourceFor1stJob)
Don't know the others yet, but all you need to try mlv_rec patch
Ok, i've added these stubs to stubs.S, applied this patch (http://upload.g3gg0.de/pub_files/4ff199e7f0ac8b5321e37adee738ebd0/2014-07-02_20-56-20_r11391_.diff) to the mlv_rec.c and compiled.
Everything went fine, ML loaded ok, module working. But no difference at all with regular mlv_rec.mo.
Am i missing something, additional steps needed? Or may be stubs addresses wrong?
Quote from: Rewind on July 07, 2014, 03:08:55 PM
Ok, i've added these stubs to stubs.S, applied this patch (http://upload.g3gg0.de/pub_files/4ff199e7f0ac8b5321e37adee738ebd0/2014-07-02_20-56-20_r11391_.diff) to the mlv_rec.c and compiled.
Everything went fine, ML loaded ok, module working. But no difference at all with regular mlv_rec.mo.
Am i missing something, additional steps needed? Or may be stubs addresses wrong?
You have to define it as far as I know, look in the patch for the define, its long, or to test just comment out. Then try.
Also reformat card from camera for both test and see if mlv_rec allocates the 154MB before first recording. If you dont reformat you wont see it because it is already created.
Quote from: mk11174 on July 07, 2014, 03:38:56 PM
You have to define it as far as I know, look in the patch for the define, its long, or to test just comment out.
Got it. 154 MiB allocating instead of 92 on 650D.
Oh, by the way, when i enable trace write, i can't find it anywhere on the card. Nothing shows up in console either (when mlv_rec is working). How can i see the debug trace?
Quote from: Levas on July 07, 2014, 10:25:18 AM
If you know how much memory is available, you can calculate it yourself, the 60d can write 21MB/s to the SD card, the rest is buffered into the available memory.
Now the resolution you ask for, takes up, 1728 x 736 x 24(frames/second) x 14(14bit canon raw) / 8 (conversion bits to bytes) /1000000(conversion bytes to MegaBytes) = 53,4 MB/second
1728*736*p24 resolution takes up 53,4 MB/second
I'm not sure how much free memory the 60d has, my guess is about 350MB now ?
That would make 53,4MB/second - 21MB/second(SD card writing speed in canon 60d) = 32,4MB/second that needs to be buffered every second.
350MB / 32,4MB = 11 Seconds of raw video
And after you filled up this memory, it takes up to 350MB/21MB = 17 seconds time to clear/write the complete filled up buffer to the SD card.
Thanks for that Levas. I have found that the numbers often don't translate as black and white as that (considering unknown factors at this stage) but at least it gives me something to dream about.
Quote from: Rewind on July 07, 2014, 05:35:25 PM
Got it. 154 MiB allocating instead of 92 on 650D.
Oh, by the way, when i enable trace write, i can't find it anywhere on the card. Nothing shows up in console either (when mlv_rec is working). How can i see the debug trace?
Not sure, I never messed with any of that but gg3g0 would be the best one to ask.
Quote from: Rewind on July 07, 2014, 05:35:25 PM
Oh, by the way, when i enable trace write, i can't find it anywhere on the card. Nothing shows up in console either (when mlv_rec is working). How can i see the debug trace?
you need trace.mo (http://ml.g3gg0.de/modules/trace.mo/trace.mo)
Thank you.
One more impatience question:
May I ask you to give me some help to adopt your mlv_rec patch to raw_rec.mo?
Is it possible to use SRM job memory buffers in raw_rec at all?
This module treats memory differently, and I can't figure that out by myself.
With the latest changes that a1ex made to get a mem_suite from SRM, you can actually use SRM the same way both raw_rec and mlv_rec allocate memory using mem_suite. Basically you just need call srm_malloc_suite(0) to get a mem_suite for SRM, and then you can use it the same way as shoot_malloc_suite(0); This (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60) is how I did it for mlv_rec it should be basically the same for raw_rec.
Quote from: dubarry on July 07, 2014, 05:56:46 AM
Hey dmilligan, if you ever get the opportunity I would be very interested to know how many frames you are getting at
1728x736 p24 2.35
269 (~11s)
also I managed to get 505 frames (~21s) of 720p (better than the 460 I was getting yesterday)
wow. I've always thought 10 secs would do for most cases but that 21 secs at 720p is sooo appealing.
And this is exactly what I meant .. why on earth would you be getting more frames (505) fine tuning things I suppose? This is great news.
Thank you for keeping us posted
Who wants to compile this for 7d so I can do some beta testing? :D
Need some clues on how to find the CreateMemoryChunk and AddMemoryChunk stubs.
(for 650d rom)
We have this:
(http://savepic.ru/5262574.jpg)
It says NSTUB ( 0x8B7C, CreateMemorySuite) in stubs.S, but ROM disassemble starts with ff000000, so how does that make sense? I suspect RAM_OFFSET has to be take into account. A1ex says in tutorial, that it is declared in stubs.S, but it's not the case for 650d. So how do I know my ram offset for particular camera?
for the AddMemoryChunk, i can find many strings like 32(Src,bytes,Unit... , and before them i usually got this:
(http://savepic.ru/5254369.jpg)
Does that mean, 8E58 has some relation to AddMemoryChunk start address?
(http://s18.postimg.org/cjjke7xrt/VRAM4.jpg)
Does this mean it works and the stubs are correct???
Strange that mlv_rec only reports 123MB now and not 154MB after compiling the srm_memory source
When I use the mlv_rec with patch from gg3g0 which he posted a few threads back, I get 154MB in mlv_rec again.
But my SRM Memory still only shows 1x31 does this sound ok?
Quote from: Rewind on July 08, 2014, 06:27:31 AM
Need some clues on how to find the CreateMemoryChunk and AddMemoryChunk stubs.
(for 650d rom)
We have this:
(http://savepic.ru/5262574.jpg)
It says NSTUB ( 0x8B7C, CreateMemorySuite) in stubs.S, but ROM disassemble starts with ff000000, so how does that make sense? I suspect RAM_OFFSET has to be take into account. A1ex says in tutorial, that it is declared in stubs.S, but it's not the case for 650d. So how do I know my ram offset for particular camera?
for the AddMemoryChunk, i can find many strings like 32(Src,bytes,Unit... , and before them i usually got this:
(http://savepic.ru/5254369.jpg)
Does that mean, 8E58 has some relation to AddMemoryChunk start address?
These are what I have for 700D now and I get working results and 1x31MB
NSTUB(0xFFA651FC - RAM_OFFSET, CreateMemoryChunk) //835C // called from CreateMemorySuite
NSTUB(0xFFA65B88 - RAM_OFFSET, AddMemoryChunk) //8CE8 // called before " 32(Src,bytes,Unit)=(%#lx,%#lx,%#lx)" in many places; see also hAddSuite
So this is what you would put for 650D since your RAM-OFFSET is not defined
NSTUB( 0x835C, CreateMemoryChunk) // called from CreateMemorySuite
NSTUB( 0x8CE8, AddMemoryChunk) // called before " 32(Src,bytes,Unit)=(%#lx,%#lx,%#lx)" in many places; see also hAddSuite
Thank you, mk11174 for your findings!
I'd prefer to know about some search 'algorithms' to find the stubs by myself, but i may suppose it is not what's easy to say in two words )
Anyway, now I get the same strange results as yours on 650D:
The test from 'don't click me' shows only 1x31 Mb buffer and dimilligan's mlv_rec only reports 123MB
But mlv patch from gg3g0 still works and gives 154 Mb.
Quote from: Rewind on July 08, 2014, 04:25:54 PM
Thank you, mk11174 for your findings!
I'd prefer to know about some search 'algorithms' to find the stubs by myself, but i may suppose it is not what's easy to say in two words )
Anyway, now I get the same strange results as yours on 650D:
The test from 'don't click me' shows only 1x31 Mb buffer and dimilligan's mlv_rec only reports 123MB
But mlv patch from gg3g0 still works and gives 154 Mb.
This way works to and makes more sense from the comments in the 5d3 stubs file
For 650D
NSTUB( 0x835C, CreateMemoryChunk) // called from CreateMemorySuite
NSTUB( 0x8E58, AddMemoryChunk) // called before " 32(Src,bytes,Unit)=(%#lx,%#lx,%#lx)" in many places; see also hAddSuite
For 700D written this way acts a bit flakey, so I guess best to use the way I wrote it for 650D which works solid
NSTUB(0xFFA651FC, CreateMemoryChunk) // called from CreateMemorySuite
NSTUB(0xFFA65CF8, AddMemoryChunk) // called before " 32(Src,bytes,Unit)=(%#lx,%#lx,%#lx)" in many places; see also hAddSuiteAlso this way also keep the debug test going after picture takes.
Ah, cool and now it shows 2x31 also and 154MB again in mlv_rec
First time I tried it I had no change, but decided to do a full make clean then recompiled and now shows correct
(http://s30.postimg.org/e9w2ftyhd/VRAM5.jpg)
I use IDA to search, this is where I found them.
(http://s29.postimg.org/s7l4ndvxz/Capture1.jpg)
(http://s23.postimg.org/c7imtvmcr/Capture2.jpg)
Yes, the 8E58 seems to be correct.
Also, now i get 4x31 MiB outside LV, but 2x31 in LV.
Don't actually understand that. Is it possible to use all 4x31 buffers for mlv recording?
Quote from: Rewind on July 08, 2014, 05:10:13 PM
Yes, the 8E58 seems to be correct.
Also, now i get 4x31 MiB outside LV, but 2x31 in LV.
Don't actually understand that. Is it possible to use all 4x31 buffers for mlv recording?
I have no clue how it works, and cool, never tried outside LV, lol
Quote from: Rewind on July 08, 2014, 05:10:13 PM
Don't actually understand that. Is it possible to use all 4x31 buffers for mlv recording?
No, LV is using 2 of the buffers itself, that's why you only get 2 in LV. When not in LV you get more b/c you get buffers that LV isn't using. The other buffers are needed by the Canon tasks that take a picture, so if you use them you can't take pictures (a1ex already mentioned this), until you release them again. So basically we are stealing memory from Canon's LV and picture taking tasks. We can use them so long as we are not trying to do one of the things we stole them from.
Quote from: dmilligan on July 08, 2014, 05:24:05 PM
No, LV is using 2 of the buffers itself, that's why you only get 2 in LV.
Now it's clear. Thank you.
Dmilligan, can you please help us to adopt these changes (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60) to legacy raw_rec module?
I am asking because its speed still perceptibly higher than mlv_rec, and that matters for low end cameras.
For example, with my old raw_rec build i've got about 80 seconds of 1536x640 at 24 fps.
But with mlv_rec even with new SRM job buffers only about 50 secs.
This is because raw_rec speed hovers around 38.4 MB/s, but mlv_rec only about 37.
If we'll be able to make raw_rec to use new srm buffers, it will be a huge addition to recording times!
That's correct stubs?
500D :
NSTUB(0xFF07B90C, CreateMemoryChunk)
NSTUB(0xFF07C3A8, AddMemoryChunk)
Quote from: dmilligan on July 08, 2014, 12:08:18 AM
With the latest changes that a1ex made to get a mem_suite from SRM, you can actually use SRM the same way both raw_rec and mlv_rec allocate memory using mem_suite. Basically you just need call srm_malloc_suite(0) to get a mem_suite for SRM, and then you can use it the same way as shoot_malloc_suite(0); This (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60) is how I did it for mlv_rec it should be basically the same for raw_rec.
I don't understand much of what your saying but, What I think I'm hearing is these latest changes should also benefit video shot in raw_rec which I use primarily and record sound externally. That would be great news since a few more seconds could make a world of difference.
Quote from: Rewind on July 08, 2014, 05:46:37 PM
Dmilligan, can you please help us to adopt these changes (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60) to legacy raw_rec module?
https://bitbucket.org/dmilligan/magic-lantern/commits/2c86608a549a7a3b1e85fd17b9acd8ed9214d8c0
Warning: I did not test this at all
Thanks for that dmilligan (didn't even think legacy raw_rec mo was even used or supported anymore.)
After the bots do their thing I'll def try this out.
Glad you asked Rewind, glad to see I'm not the only old school raw_rec mo around here!
Quote from: dmilligan on July 09, 2014, 03:30:48 AM
https://bitbucket.org/dmilligan/magic-lantern/commits/2c86608a549a7a3b1e85fd17b9acd8ed9214d8c0
Warning: I did not test this at all
Confirmed working, on 700D everything is doubled in record frame amount! Nice job dmilligan!
Confirmed working on 650D. Thank you, dmilligan!
Results at 23.976 fps:
Resolution Old rec. time no SRM (GD off) New rec. time with SRM (GD on)
1568 x 656 (2.39:1) 32 s 53 s
1600 x 672 (2.38:1) 18 s 32 s
1728 x 720 (2.40:1) 7 s 12 s
So we have x1.7 recording time increase even with Global Draw on
@mk11174: These results on the 650D from Rewind are very encouraging. If the recording times on the 550D for near-HD (like 1600x672, etc.) start getting up in the 10-20 seconds range, that changes things a lot. I'm turning to you because you were the driving force last year behind getting raw up and running on the 550D.
Am I correct in viewing the current situation as regards the increased SRM?:
1. Various contributors are busy finding stubs and compiling ML and associated modules for particular cameras
2. These compiled ML versions and modules are being tested by these contributors and they are reporting their results
3. Right now, if you want to try out and/or test these cutting-edge versions, you have to compile them yourself
Have you been able to work on the 550D in this context? Do you have an ML version/modules you could share for testing purposes? Thanks very much in advance and Cheers!
Anyone able to compile for 6D and provide it for 6D owners to test recording time increase on 6D? Indeed, increased results posted here are very encouraging. I posted necessary stubs some days ago but am still trying to decide if I buy a 6D or move with something else like 7D. This could help me in making a decision and I think other 6D owners would like to test this, too.
Quote from: nikfreak on July 03, 2014, 01:52:13 PM
So now see following. These are my findings for 6D_113:
NSTUB(0xFF0F0098, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0F40C8, SRM_FreeMemoryResourceFor1stJob)
NSTUB(0xFFCCE714 - RAM_OFFSET, CreateMemoryChunk)
NSTUB(0xFFCCF210 - RAM_OFFSET, AddMemoryChunk)
If you need others then just say so....
Quote from: nikfreak on July 09, 2014, 12:05:33 PM
I posted necessary stubs some days ago but am still trying to decide if I buy a 6D
It would be great if you finally decide on a 6D, since this port is all but unmaintained currently, and you seem capable of keeping that port in an active state. :)
Quote from: Audionut on July 09, 2014, 12:16:07 PM
It would be great if you finally decide on a 6D, since this port is all but unmaintained currently, and you seem capable of keeping that port in an active state. :)
Hrrrr dunno dunno. I know I can maintain 6D platform and keep nightlies up2date. I did this for android device trees, too :P
Will make a decision soon and ofc whichever canon body that will be I am going to contribute as much as I can to help this great project. If anyone knows where to grab a 6D for good price (also used) in Germany or Europe then PM me.
Quote from: nikfreak on July 09, 2014, 12:05:33 PM
Anyone able to compile for 6D and provide it for 6D owners to test recording time increase on 6D? Indeed, increased results posted here are very encouraging. I posted necessary stubs some days ago but am still trying to decide if I buy a 6D or move with something else like 7D. This could help me in making a decision and I think other 6D owners would like to test this, too.
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip)
Quote from: NedB on July 09, 2014, 11:51:45 AM
Have you been able to work on the 550D in this context? Do you have an ML version/modules you could share for testing purposes? Thanks very much in advance and Cheers!
Hi, sorry, I have not been testing the 550D with this mem module yet, I think alex was and said he had 31megs extra, not sure if the stubs are added for that one yet, I would have to go find them first then it should be good to test, unless someone already knows them?
Stubs for 550D (tested. working)
/** ExMem **/
NSTUB(0xFF06C678, CreateMemorySuite)
NSTUB(0xFF06c4A8, DeleteMemorySuite)
/** SRM JobMem **/
NSTUB(0xFF028590, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02AD5C, SRM_FreeMemoryResourceFor1stJob)
NSTUB(0xFF06BEC8, CreateMemoryChunk)
NSTUB(0xFF06C918, AddMemoryChunk)
Some quick tests with raw_rec for 550D at 23.976 fps:
Res. Before Now
1280x720 4 s 8 s
1280x592 7 s 14 s
So basically near the 2x rec. time increase.
But mlv_rec allocates 134 MiB — don't know, if it's new maximum for the 550d?
Quote from: NedB on July 09, 2014, 11:51:45 AM
Have you been able to work on the 550D in this context? Do you have an ML version/modules you could share for testing purposes? Thanks very much in advance and Cheers!
Here is a working version with the srm_memory update, and it def works!
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.550D109.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.550D109.zip)
Quote from: Rewind on July 09, 2014, 02:29:33 PM
But mlv_rec allocates 134 MiB — don't know, if it's new maximum for the 550d?
I think it is because the shoot total is like 20MB less then the 650D and 700D
Quote from: mk11174 on July 09, 2014, 02:00:54 PM
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip)
Yeah I knew that it can campile as a1ex already mentioned it after posting the first stubs for 6d. Now I need to post this in 6D dev thread to get feedback. thanx for your efforts.
Quote from: mk11174 on July 09, 2014, 02:00:54 PM
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip)
Downloaded this nightly build for 6d, it worsk, but no extra seconds of MLV recording.
In photo mode it has:
Shoot total of 255MB
SRM job total of 4 x 34MB
In movie mode:
Shoot total of 151MB
SRM job total of 3 x 34MB
So we're missing some memory in movie mode, but potentially it has 255MB + 3 x 34MB ?
Quote from: Levas on July 09, 2014, 04:47:20 PM
Downloaded this nightly build for 6d, it worsk, but no extra seconds of MLV recording.
This might be helpful:
Quote from: mk11174Also reformat card from camera for both test and see if mlv_rec allocates the 154MB before first recording. If you dont reformat you wont see it because it is already created.
Quote from: Rewind on July 09, 2014, 05:14:27 PM
Also reformat card from camera for both test and see if mlv_rec allocates the 154MB before first recording. If you dont reformat you wont see it because it is already created.
@Levas please confirm...
Quote from: Greg on July 08, 2014, 07:03:58 PM
That's correct stubs?
500D :
NSTUB(0xFF07B90C, CreateMemoryChunk)
NSTUB(0xFF07C3A8, AddMemoryChunk)
No, these are not correct.
I have compared the dissambled code with the stubs for 550D posted above by Rewind, and I would say:
for 500D:
/** ExMem **/
NSTUB(0xFF06C0D0, CreateMemorySuite)
NSTUB(0xFF06BEEC, DeleteMemorySuite)
NSTUB(0xFF06B90C, CreateMemoryChunk)
NSTUB(0xFF06C3A8, AddMemoryChunk)
-------------edit !!!------------------------
/** SRM JobMem **/
NSTUB(0xFF02C5A0, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02D6B0, SRM_FreeMemoryResourceFor1stJob)
-----------edit !!! ----this values are wrong !!!-------------
Tested in camera with a build of srm-memory branch.
The stubs API test completes Ok, no errors.
But I have not been able to verify the available SRM memory. The Free Memory test shows 0MB for SRM Jobs, and ends crashing with Err. 70, I have to take battery off.
The "Don't click me!" test also crashes with Err.70
------------------------------------------------------
Edit: See my post below to see the correct values: http://magiclantern.fm/forum/index.php?topic=12528.msg121487#msg121487 (http://magiclantern.fm/forum/index.php?topic=12528.msg121487#msg121487)
@mk11174: Man, that's service! Thanks very much for the upload. Have downloaded and will check to confirm Rewind's results for the 550D. But I have to admit I had forgotten that the SD bottleneck was only half as bad on the 650D as on the 550D, so I now realize we won't get a really great frame count for higher resolutions without some other amazing developer insight! Cheers...
Quote from: nikfreak on July 09, 2014, 05:47:40 PM
@Levas please confirm...
Reformatted the card in camera (canon menu, with ML being restored) and checked the free memory.
Same totals.
Video mode
Shoot total 151MB and SRM total 3x34MB.
Photo mode 255MB and SRM total 4x34MB.
Also memory hack and extra mem hack in the raw-video menu didn't make any difference
And shoot multiple clips with it, all the same recording length, same length as in older versions...
1792 x 750, 25P = 13/14 seconds of video (which is what too expect with 151MB+3x34MB)
Expect to have about 18 seconds of video(1792x750,25P) if you have 255MB shoot total and 3x34MB SRM available.
Quote from: Levas on July 09, 2014, 08:32:11 PM
Reformatted the card in camera (canon menu, with ML being restored) and checked the free memory.
Same totals.
Video mode
Shoot total 151MB and SRM total 3x34MB.
Photo mode 255MB and SRM total 4x34MB.
Also memory hack and extra mem hack in the raw-video menu didn't make any difference
And shoot multiple clips with it, all the same recording length, same length as in older versions...
1792 x 750, 25P = 13/14 seconds of video (which is what too expect with 151MB+3x34MB)
Expect to have about 18 seconds of video(1792x750,25P) if you have 255MB shoot total and 3x34MB SRM available.
Thx for providing feedback.
Probably already noticed by some of you,
But the SRM size is the exact same size as one raw image.
5D3 = 22MP, 22 x 14 bit / 8 = 39MB
6D = 20MP, 20 x 14 bit / 8 = 34MB
500D-550D-600D-650D etc. = 18MP, 18 x 14 bit / 8 = 31MB
500D = 15MP, 15 x 14 bit / 8 = 26MB
1100D = 12MP, 12 x 14 bit / 8 = 20MB
Quote from: mk11174 on July 09, 2014, 02:00:54 PM
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip)
It seems that it doesn't work for the 6d, does this mean the stubs aren't right ?
Just checked the previous nightly build I'm using (15-03-2014) and it says the same about the free memory available.
Shoot total of 255MB in Photo mode and 151MB in video mode (SRM is of course not mentioned in this build).
The 151MB reported in video mode does however change "magically" too 255, otherwise I could never get 13 seconds of video in 1792x750, 25P resolution (58,8MB/sec.)
SD card writes about 39MB second, (58,8 - 39) x 13 seconds = 257MB...
So the previous builds of MLV_rec gives about 255MB buffer memory for video, although it shows a shoot total of 151MB in the Free Memory debug menu.
The Build from 09-07-2014 you made, does the same thing, it shows 151MB of shoot total in the free memory debug menu, but it gives however recording times for 255MB of memory.
This could mean the new build:
Uses the 3x 34 MB SRM memory, but doesn't magically higher the 151MB shoot total too 255MB as available in photo mode.
Or
It doesn't use the 3 x 34MB of SRM memory and still free's the shoot total of 255MB of memory.
Quote from: a1ex on July 02, 2014, 11:25:38 PM
Some quick tests on 5D3 123, first outside LiveView, second in LV:
(http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3.png) (http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3-lv.png)
On 5D3, you get 163 + 3x39 in normal mode, and 127 + 4x39 in factory mode. No difference.
@Levas look at page1. a1ex posted values for 5dmk3 and also later some comment for 5dmk3 (bolded above). Looks like 6D is simpy the same.
No difference!
Seems like 7D / 600D and maybe others will be winners regarding usefulness of using SRM for mlv recording...
I haven't been following this closely. AFAIK, the SRM is never used in normal Canon tasks, so any memory from there is extra.
It looks like that comment for the 5D3, is only when calling via SRM_ChangeMemoryManagementForFactory, which, "might make a difference". On the 5D3, calling via that routine, does not make a difference (over and above the SRM already found).
That's the way I read it.
That's mostly right except for:
Quote from: Audionut on July 10, 2014, 02:23:01 AM
the SRM is never used in normal Canon tasks
It is used by Canon tasks. Some of it is used by LV, if you allocate all of SRM outside of LV and then try to enter LV, you get err70 or 'please wait' and LV never starts. The rest of it is used by the tasks when taking a photo. If you allocate SRM, you can't take pictures till you free it (also err 70). Fortunately there's no need to take pictures during raw video recording.
Oh of course. Thanks.
Quote from: dmilligan on July 10, 2014, 02:52:43 AM
Fortunately there's no need to take pictures during raw video recording.
Be interesting to see if it can be used for silent pics, especially the full resolution ones.
Quote from: nikfreak on July 10, 2014, 01:39:59 AM
@Levas look at page1. a1ex posted values for 5dmk3 and also later some comment for 5dmk3 (bolded above). Looks like 6D is simpy the same. No difference!
Seems like 7D / 600D and maybe others will be winners regarding usefulness of using SRM for mlv recording...
The free memory values and changes between photo and video mode do look very alike for the 5d3 and 6d.
But what I don't get is how video mode/liveview can suck up so much memory on the 5d3 and 6d, there's more then 100MB taken by liveview.
How does this look in the 500d,550d,600d which for as far as I know, don't even have 100MB spare memory to be taken by live view ?
And where does the 255MB of free memory for MLV_rec we have in the old builds on the 6d come from ?
It has to do with the memory hack function available in the raw video menu (the memory hack is not needed by the new build of 09-07-14 to have 255MB of free memory, turning it on or off makes no differences).
In the description it says it takes extra memory outside of live view on the 5d3. But it works just as good on the 6d. (without the hack I have 151MB of memory for mlv recording buffer)
So can anyone tell where the "memory hack" function in the raw video menu get's it's extra memory from ?
Quote from: Levas on July 10, 2014, 12:07:41 PM
But what I don't get is how video mode/liveview can suck up so much memory on the 5d3 and 6d, there's more then 100MB taken by liveview.
You have to remember ML is operating in the margins. Why would Canon give the camera extra memory that it didn't need? It's a miracle there's any memory for ML to use at all. There are many reasons LV would need memory, and lots of it (want specific reasons, ask a Canon engineer, or do some reverse engineering). Also, just because memory is allocated, doesn't mean it's actually being used, it just means its reserved (and we can't use it). So LV is 'reserving' this memory not necessarily using it (but who knows, it might be using it).
Quote from: Levas on July 10, 2014, 12:07:41 PM
So can anyone tell where the "memory hack" function in the raw video menu get's it's extra memory from ?
It just gets it from shoot_malloc, like the rest of the memory it uses, it just calls shoot_malloc under different circumstances. Here's basically how the hack works (to the best of my understanding from looking at the code). Before allocating memory with shoot_malloc (which is some function similar to malloc (http://en.wikipedia.org/wiki/C_dynamic_memory_allocation) that's in the Canon firmware), LV is paused. Then we call shoot_malloc, then LV is resumed. Because LV is paused there is more memory available to shoot_malloc (why? ask a Canon engineer), we take it, and then LV doesn't seem to mind that it's gone.
I specifically put the srm_malloc call outside of the memory hack pause in my changes to raw_rec and mlv_rec (i.e. it doesn't try to malloc while LV is paused) for safety reasons -> I didn't want srm_malloc to steal memory that LV actually needed.
What would happen if you did? It might not make any difference, you might get more srm, or you might just crash the camera or err70. If you're feeling brave, move the srm_malloc_suite(0); call next to the shoot_malloc_suite(0); and find out.
Quote from: dmilligan on July 10, 2014, 02:59:09 PM
I specifically put the srm_malloc call outside of the memory hack pause in my changes to raw_rec and mlv_rec (i.e. it doesn't try to malloc while LV is paused) for safety reasons -> I didn't want srm_malloc to steal memory that LV actually needed.
What would happen if you did? It might not make any difference, you might get more srm, or you might just crash the camera or err70. If you're feeling brave, move the srm_malloc_suite(0); call next to the shoot_malloc_suite(0); and find out.
I'm no coder at all, but everything is done in a certain sequence, I mean memory can't be called all at once.
What exactly is done, now the srm_malloc call is placed outside the memory hack ?
-Is the memory hack done first (so it pauses LV, and gets a total of 255MB's of memory)
-then calls srm_malloc (and maybe steals the 3 x 34MB from the extra memory that is gained with the live view hack, ending up with nothing extra ?)
So from what I read in your post, it is possible to put the srm_malloc call within the extra memory hack, which looks like this:
-the extra memory hack pauses live view, shoot_malloc is called, additionally we could call srm_malloc before resuming live view ?
With brave enough, you mean it is possible that I end up with a bricked camera ?
Or is it more likely that the camera runs somehow out of memory and if I get the battery out quick enough and put it back in it still works ?
Quote from: Levas on July 10, 2014, 03:36:47 PM
-Is the memory hack done first
No, srm is allocated first, but it really doesn't matter (AFAIK), and this is just my crude implementation we're talking about, there are issues that need to be addressed for a proper implementation that can be merged
Quote from: Levas on July 10, 2014, 03:36:47 PM
-then calls srm_malloc (and maybe steals the 3 x 34MB from the extra memory that is gained with the live view hack, ending up with nothing extra ?)
AFAIK, srm_malloc memory and shoot_malloc memory are different, you should be able to get both. The extra memory from "memory hack" is just extra from shoot_malloc => we call shoot_malloc a different way and it gives us more memory.
Quote from: Levas on July 10, 2014, 03:36:47 PM
With brave enough, you mean it is possible that I end up with a bricked camera ?
It's always a possibility, however unlikely.
Quote from: Levas on July 10, 2014, 03:36:47 PM
Or is it more likely that the camera runs somehow out of memory and if I get the battery out quick enough and put it back in it still works ?
Probably
If I get it right, the srm memory calls are implemented, but not yet in a way that it can be combined with the memory hack function.
So toggle memory hack function on and off has no influence because it isn't used at all in the build from 9 July for the 6d.
This means the implemented srm memory function does work on the 6d! :D :D :D
Because I get raw video shooting times which corresponds with a buffer of 253MB. (151MB+ 3 x 34MB srm).
If the srm memory wasn't available we would expect shooting times which corresponds too 151MB.
For extra memory on 5d3 and 6d we now need an combination of the srm_memory function and the "memory hack" function.
Quote from: josepvm on July 09, 2014, 06:18:13 PM
for 500D:
/** SRM JobMem **/
NSTUB(0xFF02C5A0, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02D6B0, SRM_FreeMemoryResourceFor1stJob)
Sorry, these values I posted previously are not correct.
These are the good ones, for 500D
/** ExMem **/
NSTUB(0xFF06C0D0, CreateMemorySuite)
NSTUB(0xFF06BEEC, DeleteMemorySuite)
NSTUB(0xFF06B90C, CreateMemoryChunk)
NSTUB(0xFF06C3A8, AddMemoryChunk)
/** SRM JobMem **/
NSTUB(0xFF02A788, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02C6F8, SRM_FreeMemoryResourceFor1stJob)
Quote from: Levas on July 10, 2014, 05:06:02 PM
If I get it right, the srm memory calls are implemented, but not yet in a way that it can be combined with the memory hack function.
No that's wrong. Memory hack and shoot_malloc have nothing to do with SRM
Quote from: dmilligan on July 10, 2014, 05:29:14 PM
No that's wrong. Memory hack and shoot_malloc have nothing to do with SRM
Probably saying it wrong, I mean the piece of code that calls the srm memory is in the build of 9 Juli and does work for the 6d.
But for 5d3 and 6d users it needs to work together with the memory hack function too have use.
Right now the already available memory hack gives us the same amount of free memory as the new srm function does without the memory hack.
Quote from: josepvm on July 10, 2014, 05:25:50 PM
These are the good ones, for 500D
Thanks! :D
500D, photo and movie mode :
(http://s9.postimg.org/7wxcz11v3/VRAM0.png) (http://s9.postimg.org/jnbageunj/VRAM1.png)
Resolution time no SRM time with SRM
1920 x 818 (crop) 37 frames 62 frames
10MB/s card, raw_rec
(@nikfreak, @Levas)
This is my first post and I'm not much of a video maker. ::)
BUT
I DO register a significant increase in record time with mlv_rec on a
6D with the nightly 2014Jul09 posted in this thread :D
I just tested various resolution (16:9) with the 2014May22 and 2014Jul09, without formatting card when changing version, and Global Draw: Allow. Write Speed showd on Display during Record: 35-38.1MB/s. Card: Lexar 600x
Here it is:
Horiz. Res. | 2014May22 | 2014Jul09 |
1792 | 3 sec. (Frame Skipped) | 6 sec. (Frame Skipped) |
1728 | 4 sec. (Frame Skipped) | 7 sec. (Frame Skipped) |
1600 | 6 sec. (Frame Skipped) | 12 sec. (Frame Skipped) |
1536 | 8 sec. (Frame Skipped) | 15 sec. (Frame Skipped) |
1504 | 10 sec. (Frame Skipped) | 18 sec. (Frame Skipped) |
1472 | 11 sec. (Frame Skipped) | 20 sec. (Frame Skipped) |
1344 | 38 sec. (Frame Skipped) | 57 sec. (Camera shuts down, File Size: 2,13GB) |
1280 | 58 sec. (Camera shuts down, File Size: 2,08GB) | 58 sec. (Camera shuts down, File Size: 2,08GB) |
Quote from: pittivale on July 10, 2014, 06:12:01 PM
(@nikfreak, @Levas)
This is my first post and I'm not much of a video maker. ::)
BUT
I DO register a significant increase in record time with mlv_rec on a 6D with the nightly 2014Jul09 posted in this thread :D
I just tested various resolution (16:9) with the 2014May22 and 2014Jul09, without formatting card when changing version, and Global Draw: Allow. Write Speed showd on Display during Record: 35-38.1MB/s. Card: Lexar 600x
Here it is:
Horiz. Res. | 2014May22 | 2014Jul09 |
1792 | 3 sec. (Frame Skipped) | 6 sec. (Frame Skipped) |
1728 | 4 sec. (Frame Skipped) | 7 sec. (Frame Skipped) |
1600 | 6 sec. (Frame Skipped) | 12 sec. (Frame Skipped) |
1536 | 8 sec. (Frame Skipped) | 15 sec. (Frame Skipped) |
1504 | 10 sec. (Frame Skipped) | 18 sec. (Frame Skipped) |
1472 | 11 sec. (Frame Skipped) | 20 sec. (Frame Skipped) |
1344 | 38 sec. (Frame Skipped) | 57 sec. (Camera shuts down, File Size: 2,13GB) |
1280 | 58 sec. (Camera shuts down, File Size: 2,08GB) | 58 sec. (Camera shuts down, File Size: 2,08GB) |
@Pittivale
What happens to the recording times on the 2014May22 build, if you go into the raw recording menu and put the "memory hack" function on ?
Activating memory hack increase record time to match 2014Jul09's times. (or 1 sec. less)
On pushing the record button, LiveView shuts down for a moment.
@pittivale thx for posting results
@Levas at least we can now be rest assured that my posted stubs are all ok and future srm related findings will work, too. If in any case new stubs need to be added in future then you can drop me a msg and I will post 'em for you. Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...
@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...
As far as I can tell we should also be able to use SRM for photo mode, too? Let's say increase burst rate and therefore fps?
Would it be possible to send the frames from the SRM job memory buffers to the SD or CF (camera specific) card and to the USB port (maybe with Wi-Fi read out (http://www.youtube.com/watch?v=Our-AxmrDJM)) paralelly?
Edit: ... or split the frame and send the top half to the SD or CF card and the bottom part to the USB port. Later in post process merge the 2 parts to 1.
On the 500D picture quality (jpg raw) has an effect on the SRM.
Photo mode Movie mode
RAW 4x26MB 2x26MB
JPG 3x26MB 1x26MB
Quote from: nikfreak on July 10, 2014, 07:09:14 PM
Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...
Have absolutely no idea how to do that, can anybody else help us here...
Quote from: nikfreak on July 10, 2014, 07:09:14 PM
@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...
Don't know if I'm right, but my understanding is that raw video/photo data can't be compressed.
How would you compress data that's coming from a bayer pattern ? You need every pixel data to demosaic...
Quote from: nikfreak on July 10, 2014, 07:09:14 PM
@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...
No way, it would require real-time compression of huge amounts of data on a dinky little ARM CPU. If we could compress in real-time we wouldn't need as much extra memory anyway, cause we could just write it out to the card compressed, thus requiring a lower card speed.
Quote from: nikfreak on July 10, 2014, 07:09:14 PM
As far as I can tell we should also be able to use SRM for photo mode, too? Let's say increase burst rate and therefore fps?
Nope, photo mode is already using SRM when it takes pictures, and this is very likely what it is used for. When we allocate SRM, photo mode doesn't work (this is probably the 11th time I've stated this).
Quote from: Levas on July 10, 2014, 10:41:10 PM
Don't know if I'm right, but my understanding is that raw video/photo data can't be compressed.
How would you compress data that's coming from a bayer pattern ? You need every pixel data to demosaic...
There are lots of ways to losslessly compress raw data (or any data for that matter). CR2 is actually compressed. It's not a matter of how, but the fact that there is just way to much data coming in way to fast. You can actually lossy compress raw data too, one example would be to reduce the bit depth from 14 to say 10. This is technically "compression" and it's lossy (the last 4 bits are lost), but the image can still be reconstructed. Unfortunately the CPU isn't even powerful enough to chop bits off. In fact the CPU isn't even fast enough to move the raw data around, we have to use the edmac (external memory controller) to move raw data around.
Quote from: Greg on July 10, 2014, 09:10:03 PM
On the 500D picture quality (jpg raw) has an effect on the SRM.
Photo mode Movie mode
RAW 4x26MB 2x26MB
JPG 3x26MB 1x26MB
Good point!
I see the same results in my 500D. And RAW+JPEG gives the same values than JPEG alone (3x26MB Photo, 1x26MB video)
So it seems the camera reserves one SRM buffer to convert the image to JPEG.
That's good, an added motivation to take photos always in RAW ;)
Quote from: nikfreak on July 10, 2014, 07:09:14 PM
If in any case new stubs need to be added in future then you can drop me a msg and I will post 'em for you. Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...
I think we need CreateMemorySuite and DeleteMemorySuite before creating a pull request.
Quote from: a1ex on July 02, 2014, 11:25:38 PM
These changes are not yet included in any nightly build (we are reverse engineering it as we speak, and sharing the details as soon as we discover them). Please be patient, and hope this will motivate the 6D developers to bring the port up to date.
Some quick tests on 5D3 123, first outside LiveView, second in LV:
(http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3.png) (http://a1ex.magiclantern.fm/bleeding-edge/srm-5d3-lv.png)
I just did this on mine, here's the output on 5D3.123, seems the same.
(http://www.daviszone.org/~pdavis/scn1.png)
Anyone with a 700D build? The recent nightly's won't boot, but the original 14th Feb Alpha boots everytime. http://www.magiclantern.fm/forum/index.php?topic=12617.new#new (http://www.magiclantern.fm/forum/index.php?topic=12617.new#new) and everything's ok with nightly 550D's with SRM so I think I'm doing it right.
Quote from: dyfid on July 11, 2014, 09:39:28 AM
Anyone with a 700D build? The recent nightly's won't boot, but the original 14th Feb Alpha boots everytime. http://www.magiclantern.fm/forum/index.php?topic=12617.new#new (http://www.magiclantern.fm/forum/index.php?topic=12617.new#new) and everything's ok with nightly 550D's with SRM so I think I'm doing it right.
Of course, I already took care of the 700D, will upload in a bit for you. :)
SRM version for 700D https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip)
By the way, I just checked the recent Nightly from here http://builds.magiclantern.fm/#/ and it boots fine. The nighties do not contain the patches for the SRM_Memory though, only these test builds in this thread have the srm_memory patch.
Quote from: Audionut on July 11, 2014, 03:15:44 AM
I think we need CreateMemorySuite and DeleteMemorySuite before creating a pull request.
reateMemorySuite and DeleteMemorySuite already exist:
https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified (https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified)
Just those missing that I discovered and posted here....
Quote from: mk11174 on July 11, 2014, 10:12:29 AM
Of course, I already took care of the 700D, will upload in a bit for you. :)
SRM version for 700D https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip)
By the way, I just checked the recent Nightly from here http://builds.magiclantern.fm/#/ and it boots fine. The nighties do not contain the patches for the SRM_Memory though, only these test builds in this thread have the srm_memory patch.
Thanks for the SRM build, but hmm, still no good, with Feb14th build on the card the 700D boots, when I put your build on the card, the battery flashes on/off as soon as the card is detected and it won't boot and I have to pull the battery with the card out to reset for it to boot after at all, even with no card in. The Canon Firmware is 1.1.1, but still the Feb14th works fine. What am i doing wrong?
Quote from: pompeiisneaks on July 11, 2014, 04:46:39 AM
I just did this on mine, here's the output on 5D3.123, seems the same.
(http://www.daviszone.org/~pdavis/scn1.png)
Do you have a compiled version for 5D3.113?
Quote from: nikfreak on July 11, 2014, 10:41:47 AM
reateMemorySuite and DeleteMemorySuite already exist:
https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified (https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified)
Just those missing that I discovered and posted here....
Oh yes thanks.
Done: https://bitbucket.org/hudson/magic-lantern/pull-request/537/6d-missing-stubs-from-nikfreak-http/diff
Quote from: dyfid on July 11, 2014, 11:01:11 AM
Thanks for the SRM build, but hmm, still no good, with Feb14th build on the card the 700D boots, when I put your build on the card, the battery flashes on/off as soon as the card is detected and it won't boot and I have to pull the battery with the card out to reset for it to boot after at all, even with no card in. The Canon Firmware is 1.1.1, but still the Feb14th works fine. What am i doing wrong?
Well there you go, that is your problem, you have to update canon firmware to 113, the new ports are 113 not 111, so go to canon and update your camera, then get the nightly and reflash with the FIR file in the nightly and your good to go with ML, and if you want the SRM just use the zip in this thread.
Hi !
Does anybody have a 60D version ? So i can give you my results.
Cheers,
Christian
Quote from: mk11174 on July 11, 2014, 11:25:41 AM
Well there you go, that is your problem, you have to update canon firmware to 113, the new ports are 113 not 111, so go to canon and update your camera, then get the nightly and reflash with the FIR file in the nightly and your good to go with ML, and if you want the SRM just use the zip in this thread.
That was it, :-[, 154mb reserved but now shooting as 20MB/s instead of 40MB/s with 1.1.1 & Feb14th ML? So not getting many frames, sure it was continuous at 720p?. Sandisk Extreme 95MB/s, ExFAT via EOS Card.
Quote from: mario1000 on July 11, 2014, 11:03:26 AM
Do you have a compiled version for 5D3.113?
I don't. Realize this is a test version, mine couldn't take a picture, it kept saying 'busy' but I think that's because the code used is designed just to see if the ML code can allocate the memory. If you want me to, I can do a build for 5D3.113, but I'm not going to test it first because rolling back to .113 is a pita. So I won't have a chance to test it out, and it may not work.
~Phil
Here's the .113 version, again all the usual warnings, I used my build for a different .113 to test external monitors and it worked fine, but ymmv, but I'm pretty sure the codebase is stable and as usable in the srm-memory fork as any other.
https://www.dropbox.com/s/s136ezkzzad66hy/magiclantern-Nightly.2014Jul11.5D3113.zip (https://www.dropbox.com/s/s136ezkzzad66hy/magiclantern-Nightly.2014Jul11.5D3113.zip)
Here the results of the mittelfranconian jury for my 600D:
Normal shoot mode:
(https://bitbucket.org/escho/escho-magiclantern/downloads/shoot.jpg)
LiveView:
(https://bitbucket.org/escho/escho-magiclantern/downloads/LV.jpg)
Test with "don´t click me"
(https://bitbucket.org/escho/escho-magiclantern/downloads/test.jpg)
No problems so far
Edgar
Can anyone confirm what MB/s they're getting out of a 700D raw with the SRM code and FW 1.1.3?
I now only get 20MB/s same as a 550D where as before with FW 1.1.1 and the 700D Feb14th Alpha with exact same card, Sandisk Extreme Pro 95MB/s I was getting 40MB/s and continuous 720P. Can't get anywhere near that now.
Quote from: dyfid on July 12, 2014, 01:47:38 AM
Can anyone confirm what MB/s they're getting out of a 700D raw with the SRM code and FW 1.1.3?
I now only get 20MB/s same as a 550D where as before with FW 1.1.1 and the 700D Feb14th Alpha with exact same card, Sandisk Extreme Pro 95MB/s I was getting 40MB/s and continuous 720P. Can't get anywhere near that now.
Try using the nightly builds and see what you get, dont compare the builds from 1.1.1, with the SRM build, because there was no build for 1.1.1 with SRM, so to compare, you need to use the regular nightly builds and compare what you use to get when you were at 1.1.1.
And also what module are you testing, the raw_rec or mlv_rec or just a regular movie???
I have same card and these are my results in 1920x1080@24p setting both modules res to 1280x720
(http://s28.postimg.org/99jvi5365/VRAM0.jpg)
MLV_REC
(http://s15.postimg.org/k7o0uagej/VRAM1.jpg)
RAW_REC
Both jump around during record and give a normal reading as it did back in 1.1.1.
When using the SRM build, it does not affect the mb/s as far as I can tell, just records more frames.
Cleaned up my changes and created a pull request. It is mostly functionally the same as before, the code is just cleaner, and now there's a menu option to enable/disable srm memory for comparison purposes. If you can compile, please test.
https://bitbucket.org/hudson/magic-lantern/pull-request/540/update-raw_rec-and-mlv_rec-to-use-new-srm/diff
Quote from: dmilligan on July 12, 2014, 04:28:51 PM
Cleaned up my changes and created a pull request. It is mostly functionally the same as before, the code is just cleaner, and now there's a menu option to enable/disable srm memory for comparison purposes. If you can compile, please test.
https://bitbucket.org/hudson/magic-lantern/pull-request/540/update-raw_rec-and-mlv_rec-to-use-new-srm/diff
Great idea, especially for testing, so much easier to test it out!
This version (https://bitbucket.org/dmilligan/magic-lantern/commits/968258312ee5447ebe8dd4eedd6ff6ce12181337?at=unified) of raw_rec is slower than previous (https://bitbucket.org/dmilligan/magic-lantern/commits/fb7d927e4b3d63f62fb2bbe3eed8a7e1a6e85e60).
results for 650D, 23.976 fps, GD off:
Resolution First Implementation This version
1568 x 656 62 s 50 s
1600 x 672 32 s 27 s
1728 x 720 12 s 10 s
Short test with my 600D, using mlv_rec:
Recording 1728 x 972 until 1st skipped frame
1st try: without SRM: 28 frames, with SRM: 61 frames
2nd try: without SRM: 31 frames, with SRM: 59 frames
Recording 1728 x 972, 10s, skipped frames allowed
1st try: without SRM: 79 frames, with SRM: 110 frames
Really nice
Edgar
did some more tests, in general mlv_rec is outperforming raw_rec by a few frames in whatever mode
also recording times for crop mode are not quite as good, though this doesn't seem to be related to buffer space, looking at the trace, the same amount of buffer space is available, 353MB in 126 slots for 1728x972
Quick test on 5D3.123.
MLV sound/Global Draw ON
No other hacks or card spanning.
Lexar 16GB 1000x
1920x1080x23.976
Default 13seconds
SRM 31seconds
1920x1280x23.976
Default 4 seconds
SRM 10 seconds
Can anybody compile for 5D3.113 and .123 with mk11174 changes for me to test please
I think :-\ these are the SRM stubs for the 50D
/** SRM JobMem **/
NSTUB(0xFF0A212C, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0A247C, SRM_FreeMemoryResourceFor1stJob)
but I'm totally lost trying to find CreateMemorySuite and DeleteMemorySuite
Also, I'm not seeing 'SRM Job total' in Free Memory. I cloned the main repo and compiled a new build for the first time in months but module building fails with [module_strings.h] Error 1. Didn't compile from the SRM branch or have Python docutils installed ::)
Found what I thought were the MemorySuite stubs but SRM memory shows 0x0 so they can't be correct.
Quote from: Andy600 on July 13, 2014, 01:02:11 PM
I think :-\ these are the SRM stubs for the 50D
...
but I'm totally lost trying to find CreateMemorySuite and DeleteMemorySuite
I have found them for 500D, that should be quite similar, so I can give you some tips.
Search for the string "PackMemory\PackMem.c"
CreateMemorySuite should be a function like this one (500D):
loc_ff06c0d0: ; 18 refs
ff06c0d0: e92d41f0 push {r4, r5, r6, r7, r8, lr}
ff06c0d4: e1a07000 mov r7, r0
ff06c0d8: e3a00018 mov r0, #24
ff06c0dc: e1a08002 mov r8, r2
ff06c0e0: e1a06001 mov r6, r1
ff06c0e4: ebfeb3a1 bl loc_ff018f70
ff06c0e8: e1b04000 movs r4, r0
ff06c0ec: 059f212c ldreq r2, [pc, #300] ; ff06c220: (00000225)
ff06c0f0: 051f1158 ldreq r1, [pc, #-344] ; ff06bfa0: (ff06b860) **"PackMemory\PackMem.c"
ff06c0f4: 028f0f47 addeq r0, pc, #284 ; ff06c218: (69755370) *"pSuite"
ff06c0f8: 0bfea30f bleq loc_ff014d3c
And DeleteMemorySuite should be like this one:
loc_ff06beec: ; 53 refs
ff06beec: e92d47f0 push {r4, r5, r6, r7, r8, r9, sl, lr}
ff06bef0: e51f164c ldr r1, [pc, #-1612] ; ff06b8ac: (00002f60)
ff06bef4: e1a08000 mov r8, r0
ff06bef8: e5900000 ldr r0, [r0]
ff06befc: e5911000 ldr r1, [r1]
ff06bf00: e1500001 cmp r0, r1
ff06bf04: 159f20b0 ldrne r2, [pc, #176] ; ff06bfbc: (00000239)
ff06bf08: 124f1e6b subne r1, pc, #1712 ; ff06b860: (6b636150) *"PackMemory\PackMem.c"
ff06bf0c: 128f00ac addne r0, pc, #172 ; ff06bfc0: (75537349) *"IsSuiteSignature( hSuite )"
ff06bf10: 1bfea389 blne loc_ff014d3c
Quote
Also, I'm not seeing 'SRM Job total' in Free Memory.
Are you building the srm-memory branch?. The unified branch does not show the SRM values in FreeMemory.
Quote from: Andy600 on July 13, 2014, 01:02:11 PM
Found what I thought were the MemorySuite stubs but SRM memory shows 0x0 so they can't be correct.
Double check this ones:
Quote
/** SRM JobMem **/
NSTUB(0xFF0A212C, SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0A247C, SRM_FreeMemoryResourceFor1stJob)
They do not contain the strings "SRM_FreeMemoryResourceForDDD" or "SRM_AllocateMemoryResourceForDDD" but are called from places where these strings are present.
for SRM_AllocateMemoryResourceFor1stJob (500D) :
loc_ff0a8c18:
ff0a8c18: e2841b03 add r1, r4, #3072 ; 0xc00
ff0a8c1c: e59f0194 ldr r0, [pc, #404] ; ff0a8db8: (ff0a820c)
ff0a8c20: e281108c add r1, r1, #140 ; 0x8c
ff0a8c24: ebfe06d7 bl loc_ff02a788 // ---->>> This is the function
ff0a8c28: e59b0034 ldr r0, [fp, #52] ; 0x34
ff0a8c2c: e3a01064 mov r1, #100 ; 0x64
ff0a8c30: ebff0326 bl loc_ff0698d0
ff0a8c34: e3500000 cmp r0, #0
ff0a8c38: 11a03000 movne r3, r0
ff0a8c3c: 128f2f5e addne r2, pc, #376 ; ff0a8dbc: (5f4d5253) *"SRM_AllocateMemoryResourceFor1stJob failed [%#x]"
And for SRM_FreeMemoryResourceFor1stJob (500D):
loc_ff0a9424:
ff0a9424: e5900c8c ldr r0, [r0, #3212] ; 0xc8c
ff0a9428: e3500000 cmp r0, #0
ff0a942c: 13a02000 movne r2, #0
ff0a9430: 13a01000 movne r1, #0
ff0a9434: 1bfe0caf blne loc_ff02c6f8 // ------->>> This is the function
ff0a9438: ebff0185 bl loc_ff069a54
ff0a943c: e51f1760 ldr r1, [pc, #-1888] ; ff0a8ce4: (00003f00)
ff0a9440: e591001c ldr r0, [r1, #28]
ff0a9444: e2500001 subs r0, r0, #1
ff0a9448: e581001c str r0, [r1, #28]
ff0a944c: 1a000004 bne loc_ff0a9464
ff0a9450: ebff1018 bl loc_ff06d4b8
ff0a9454: e28f2f72 add r2, pc, #456 ; ff0a9624: (6c656b53) *"SkeltonJob Done"
ff0a9458: e3a01016 mov r1, #22
ff0a945c: e3a0008f mov r0, #143 ; 0x8f
ff0a9460: ebfef58c bl loc_ff066a98
In this one, search for "SkeltonJob"
@josepvm - thanks for your help :)
I'm compiling from the SRM branch now - wasn't before ::)
This is the only one I can see as possible for CreateMemorySuite (it has 'ldreq' and I'm looking for similarities to what you posted)
loc_ff069068: ; 3 refs
ff069068: e92d41f0 push {r4, r5, r6, r7, r8, lr}
ff06906c: e1b07002 movs r7, r2
ff069070: e1a08001 mov r8, r1
ff069074: e1a05000 mov r5, r0
ff069078: 059f114c ldreq r1, [pc, #332] ; ff0691cc: (ff86895c) **"PackMemory\PackMem.c"
ff06907c: 059f217c ldreq r2, [pc, #380] ; ff069200: (000001f7)
ff069080: 028f0f5f addeq r0, pc, #380 ; ff069204: (50282821) *"!((PVOID)hClientPackHeap == NULL)"
ff069084: 0bfeae8d bleq loc_ff014ac0
for DeleteMemorySuite could it be:
ff068fb8: e92d47f0 push {r4, r5, r6, r7, r8, r9, sl, lr}
ff068fbc: e51f1624 ldr r1, [pc, #-1572] ; ff0689a0: (00002e7c)
ff068fc0: e1a08000 mov r8, r0
ff068fc4: e5900000 ldr r0, [r0]
ff068fc8: e5911000 ldr r1, [r1]
ff068fcc: e1500001 cmp r0, r1
ff068fd0: 159f2208 ldrne r2, [pc, #520] ; ff0691e0: (00000236)
ff068fd4: 124f1d1a subne r1, pc, #1664 ; ff06895c: (6b636150) *"PackMemory\PackMem.c"
ff068fd8: 128f0f81 addne r0, pc, #516 ; ff0691e4: (75537349) *"IsSuiteSignature( hSuite )"
ff068fdc: 1bfeaeb7 blne loc_ff014ac0
not sure which addresses to try though....I'm a total newb at this bit
ah, just saw your next post
ff8a231c: e2811008 add r1, r1, #8
ff8a2320: ebfe23cb bl loc_ff82b254
ff8a2324: e5970028 ldr r0, [r7, #40] ; 0x28
ff8a2328: e3a01064 mov r1, #100 ; 0x64
ff8a232c: ebff1180 bl loc_ff866934 <------------------------ this is referenced all the time when I search for SRM_AllocateMemoryResource so that must SRM_AllocateMemoryResourceFor1stJob?
ff8a2330: e3500000 cmp r0, #0
ff8a2334: 11a03000 movne r3, r0
ff8a2338: 124f2e3d subne r2, pc, #976 ; ff8a1f70: (424f4a5b) *"[JOB ERROR] SRM_AllocateMemoryResourceFor1stJob failed [%#x]"
DeleteMemorySuite looks Ok, but I'm not equally convinced about CreateMemorySuite.
Look for a similar structure in an address a little bit further: 0xFF069118 or near this value.
I have observed in 500D and in 550D a similar offset between DeleteMemorySuite and CreateMemorySuite.
Quote from: Andy600 on July 13, 2014, 03:58:14 PM
ff8a231c: e2811008 add r1, r1, #8
ff8a2320: ebfe23cb bl loc_ff82b254
ff8a2324: e5970028 ldr r0, [r7, #40] ; 0x28
ff8a2328: e3a01064 mov r1, #100 ; 0x64
ff8a232c: ebff1180 bl loc_ff866934 <------------------------ this is referenced all the time when I search for SRM_AllocateMemoryResource so that must SRM_AllocateMemoryResourceFor1stJob?
Not this one. The previous call, three lines before, is the stub you are searching for : loc_ff82b254
Quote from: SoulChildPaul on July 13, 2014, 10:41:44 AM
Can anybody compile for 5D3.113 and .123 with mk11174 changes for me to test please
I can only do the 5D3.113 because it is in the source I have.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul13.5D3113.zip (https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul13.5D3113.zip)
600d and 650d are giving better results than 60d? how come?
I'm back, glad to see the progress. Just give me some time to digest the new stuff, and I'll handle the pull requests in the next few days.
@josepvm or @a1ex - can you please specify the strings I need to look for for each stub needed (I think there are 5 missing stubs in total for SRM to work on the 50D!?) I'm getting a bit lost when it comes to MemoryChunk etc and still don't really understand what I'm looking at despite many reads of this and the other thread :-[
So for instance, for the SRM_AllocateMemoryResourceFor1stJob stub , I searched for SRM_AllocateMemoryResourceFor1stJob failed [%#x] in Notepad++ which got me close (although I was still wrong) and I presume ff82b254 is correct as you indicated. I'm trying to see the pattern here because it doesn't make sense in my head yet.
Thanks again for your help :). I'm not sure if there are any devs still working on the 50D so I would like to at least try and keep it up to date with my extremely limited code knowledge.
Quote from: Andy600 on July 13, 2014, 09:13:38 PM
I'm not sure if there are any devs still working on the 50D so I would like to at least try and keep it up to date with my extremely limited code knowledge.
I'm still alive! I just got back from a four-week work trip, so I'll need to get back up to speed on ML and then help you track down stubs.
I've submitted a PR (https://bitbucket.org/hudson/magic-lantern/pull-request/543/srm-stubs-for-50d/diff) with the 50D stubs. Thanks to @Andy600 and @josepvm for getting that ball rolling! Here are the screenshots (outside LV, LV, and don't click me):
(https://www.dropbox.com/s/lje8j5ciodem5hm/srm-memory_outsideLV.png?dl=1)
(https://www.dropbox.com/s/5ux65ux5dkm63fh/srm-memory_insideLV.png?dl=1)
(https://www.dropbox.com/s/trnx58joostncd3/srm-memory_test.png?dl=1)
However, I starting taking a few pictures during the "don't click me" test, and on the second, I hit an "Err 80". Has anyone else encountered any issues?
Quote from: ayshih on July 14, 2014, 01:30:44 AM
However, I starting taking a few pictures during the "don't click me" test, and on the second, I hit an "Err 80". Has anyone else encountered any issues?
Hmm, I haven't been able to reproduce this with further testing...
216mb extra! Nice :)
Just tried MLV and got this message:
Hack error at 0
expected e3a00032, got e1a00000
It recorded about the same amount of frames as without SRM. Also, you can get into crop mode only once and the zoom buttons become non functional requiring a battery pull.
And, has raw REC button been remapped? I get h264 recording with the SET button and raw with the LV (blue) button.
UPDATE: backing out of https://bitbucket.org/hudson/magic-lantern/commits/964f3df (https://bitbucket.org/hudson/magic-lantern/commits/964f3df) seems to have fixed all the problems :) Thanks a1ex!
I noticed some undesirable behaviour with crop mode on a 5D3.
But haven't had the time to test things thoroughly.
Quote from: ayshih on July 14, 2014, 03:06:57 AM
Hmm, I haven't been able to reproduce this with further testing...
Shots during "Don't Click Me" test worked fine for me on the 50D. After trying to shoot .mlv with extra memory worked until card was full (forgot to clear it first). Haven't seen error 80 yet, but trying on a clean 1000x card now to see how many frames I can get in crop mode.
Thanks for getting this going on the 50D, I like Andy was having a hard time looking for the stubs until it was cleared up a few posts above.
Had a quick play with the raw recorders.
Good news: the memory hack is no longer needed on 5D3. The total memory with hack + SRM, usable for raw recording, is actually lower than just with SRM. This cleans up the code quite well, and removes the annoyances of the old memory hack.
Give it a quick try, and if all goes well, it will be in the nightly builds tonight or tomorrow. Including 6D and 500D.
Quote from: a1ex on July 15, 2014, 09:24:29 AM
Had a quick play with the raw recorders.
Good news: the memory hack is no longer needed on 5D3. The total memory with hack + SRM, usable for raw recording, is actually lower than just with SRM. This cleans up the code quite well, and removes the annoyances of the old memory hack.
Give it a quick try, and if all goes well, it will be in the nightly builds tonight or tomorrow. Including 6D and 500D.
According to some tests on the 6d, memory hack VS SRM made no difference
Quote from: pittivale on July 10, 2014, 06:46:51 PM
Activating memory hack increase record time to match 2014Jul09's times. (or 1 sec. less)
On pushing the record button, LiveView shuts down for a moment.
The memory hack gives an additional of 100MB, the SRM gives 3x 34MB on 6d.
How can that SRM only gives more memory on 5d3 then memory hack + SRM.
Am I missing something here ?
The memory is managed in different ways by Canon in different modes (LV, outside LV, RAW, JPEG and so on). Some of it is reserved for shoot_malloc (for compressing photo data after capture), some of it is for SRM (for uncompressed just-captured raw photo data), some is for the operating system, some is for LiveView and so on. The largest ones are shoot, SRM and LiveView buffers, and their sum is more or less constant.
This means, the benefit for 6D is just cleaner code and a smaller delay before starting to record.
Quote from: a1ex on July 15, 2014, 04:42:19 PM
The memory is managed in different ways by Canon in different modes (LV, outside LV, RAW, JPEG and so on). Some of it is reserved for shoot_malloc (for compressing photo data after capture), some of it is for SRM (for uncompressed just-captured raw photo data), some is for the operating system, some is for LiveView and so on. The largest ones are shoot, SRM and LiveView buffers, and their sum is more or less constant.
This means, the benefit for 6D is just cleaner code and a smaller delay before starting to record.
Just to be sure if I understand this correct:
Are you saying we can't have best of both worlds, 100MB from the hack and 3 x 34 MB SRM memory ?
In video mode on 6d we have:
Shoot total 151MB and SRM total 3x34MB. (Total of 253MB)
In photo mode we have:
Shoot total 255MB and SRM total 4x34MB. (total of 391MB)
Don't know if I'm right, but my guess is there's enough total memory in the 6d to have (some) best of both worlds.
Right know, with SRM only, without the extra memory hack, there's 140MB hold back, live view of course need some of it, but I don't expect live view needs all 140MB.
There are more allocators, not just shoot_malloc and SRM. If you trace the references to the worker function from AllocateMemory, you'll find around 10 or 20, but most of them are small. In addition, there are many small allocators in the resource manager. Feel free to reverse engineer them and squeeze even more RAM.
The total RAM size in these cameras is 512 MB, and 256 MB on most Rebels. That colorful line from the free memory dialog shows what we know about it, and from unknown areas, it tries to guess what's used and what might be free. A better guess can be obtained if you compile with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (but this delays the boot process a bit).
BTW, what's the reason for quoting the entire post right above yours? It would just double the thread length, if everybody would do the same.
Sorry about the quoting habit :-[
Looks indeed very stupid, if it's exactly one post under it :D
The thing I'm worried about is that the extra memory hack is declared obsolete and will be deleted in the current builds.
I'm only worried about this because it looks like it can be used on the 6d aside with the SRM code.
Would love to test if the extra memory hack can be used in combination with the SRM code, but I don't have any coding skills.
And from previous post it seems, the build we tested on the 6d with the SRM code in it, somehow disables the extra memory hack.
If it turns out that extra memory hack and SRM code don't work together on the 6d, I see no reasons to not clean up the code by deleting the extra memory hack.
The way I see it, as a person without coding skills ???, this could be a relatively easy way to get some extra memory for raw video buffer.
But then again, I have no coding skills, so maybe things are a lot harder to implement...
Does this mean that the newly found SRM memory will be automatically be made available transparently to raw recording with no liveview resetting? I was never able take advantage of the memory hack on the 5D3 because the long delay of resetting liveview caused my hdmi monitor to miss the initial 4 seconds of recording as the monitor reset itself. This alone would have a huge impact for hdmi shooters. Also, will small hacks still be available/useful?
Quote from: Levas on July 15, 2014, 08:07:25 PM
Would love to test if the extra memory hack can be used in combination with the SRM code, but I don't have any coding skills.
Try compiling 3d6a945, or ask Audionut here (http://www.magiclantern.fm/forum/index.php?topic=12608.0) to do it for you.
QuoteDoes this mean that the newly found SRM memory will be automatically be made available transparently to raw recording with no liveview resetting?
Yes.
I have been testing the recent additions to the srm-memory branch on 500D, today's build.
Shooting with mlv-rec module, using a low resolution, 864x486, to allow continuous recording on SD card.
I have noticed this issue: when using SRM memory, AF (with back focus button, "*") does not work at all during recording.
With exactly the same settings but disabling SRM memory use, AF works again.
So, maybe SRM memory is used for autofocusing by the Canon firmware?
Does it help if you comment out the gui_uilock calls from exmem.c?
(ML blocks the shutter button so you can't take pictures by mistake and get err70)
Quote from: a1ex on July 15, 2014, 10:30:42 PM
Does it help if you comment out the gui_uilock calls from exmem.c?
(ML blocks the shutter button so you can't take pictures by mistake and get err70)
Yes, it helps. AF works now with SRM memory enabled.
Thanks, A1ex.
(some additional info: I have CFn functions adjusted to focus always with back focus button, not the shutter button)
I have tested the new Nightly Build for 500D (Nightly.2014July16).
It still has no option to use SRM memory for raw video, but the shutter button lock is active, and AF does not work for video.
With the srm-memory branch build you have the option to disable SRM and then have AF working. In this nightly there is no such option, and no AF :(
I wonder if a better solution would be possible here, to avoid taking pictures during video shooting but allowing AF with backfocus button.
Autofocus works just fine here with the nightly. I didn't include the SRM yet, will do tonight.
The safeguard is not really to prevent taking pictures while shooting video; these routines are general-purpose, other code could make use of them, and I want the backend to prevent the race conditions with picture taking code (that's why the "don't click me" test tells you to take pictures while it's allocating and freeing memory).
Food for thought: when allocating the entire SRM memory, Canon code already locks the UI (it prints BUSY), but the autofocus still works. Figure out how it does that...
edit: I think it's solved, tested on 5D3: https://bitbucket.org/hudson/magic-lantern/commits/92536576a20d
Quote from: a1ex on July 16, 2014, 10:55:49 AM
edit: I think it's solved, tested on 5D3: https://bitbucket.org/hudson/magic-lantern/commits/92536576a20d
Tested this solution on 500D, using srm-memory branch to build, works Ok.
I need to disable "Extra hacks" to get the AF working. When disabling the hacks, the AF works fine with SRM enabled.
I have done the "Don't click me" test with this changes, as you suggested, and I am able to take shots, record video and AF during the test.
But I don't know exactly how this test works and how it should be used, sorry ??? In first place, I do not know if the test ends after a successfull run, or if it runs continously until I shut down the camera, as I did ::)
Disabling the "Extra hacks" is the key, also, for the Nightly build I tested earlier. Now I've tested it again, disabling the Extra hacks, and AF works (no SRM, of course).
When I tested the srm-memory branch yesterday, I disabled the hacks also, because some days ago, testing mlv-rec, I got errors when starting video recording, and Greg suggested me to disable them.
But today, whith the new Nightly build, I saw mlv-rec worked without disabling the hacks, so I left them enabled ... but that's the reason for AF not working.
The test simply checks if these routines play nice with Canon picture taking process. If you would comment out the gui lock tricks in exmem.c, you would get either lockups or err70 during the test.
Quote from: a1ex on July 15, 2014, 08:35:40 PM
Try compiling 3d6a945, or ask Audionut here (http://www.magiclantern.fm/forum/index.php?topic=12608.0) to do it for you.
@ Alex,
Audionut compiled 3d6a945 for me to test on the 6d.
Both options(mem hack and SRM) together didn't result in any more memory... ::) or actually I mean :'(
Did some testing and looked at the MLV file size
Both options = the same file size as Memory hack only
SRM option only = slightly higher file size vs memory hack option and both options.
So it looks like both options results in gaining the memory from the memory hack, but doesn't give us any SRM memory. ???
So you were right :D
Quote from: a1ex on July 15, 2014, 04:42:19 PM
This means, the benefit for 6D is just cleaner code and a smaller delay before starting to record.
Alright folks, the SRM memory changes are now in the nightly builds. The good old memory hack is gone, as well as a bunch of older code hacks.
On 5D3 and 6D you have the same total memory as before (+/- a few MB), but without having to blank the LiveView. On other cameras, you no longer need to switch to small JPEG or similar configurations. And if you are lucky, you should get a nice improvement in recording times at certain resolutions.
I don't know the exact figures - you tell me.
This is not the end of memory research, but I'll stop here for now. Feel free to investigate the factory mode, or try to find some more allocators from the resource manager.
Thanks to all the contributors from this thread!
EVERYTHING IS AWESOME!!! (Yep I'm a parent)
60D 1728x736 2.35:1 24fps 254 frames @ 10 seconds
1728x972 16x9 24fps 170 frames @ 6 seconds
Honestly 10 seconds has been my dream for about what 2.5/3 years now.
That's bloody spectacular
thank you, thank you, thank you all.
I can confirm a big increase in performance after the SRM update. Was getting 15-20 seconds 30fps... now getting 30-40 seconds. Double the recording time. At least for me, this makes 30fps a more stable and reliable frame rate. Also, 48 fps (squashed mode) went from 8 to 15 seconds. Again, much more dependable where before, it was borderline for me. These new times are with raw_rec. I get similar relative increases with mlv_rec, just shorter actual recording times due to the extra overhead of the mlv format.
Changes like this really do make a big difference. For many HDMI monitor users, this eliminates the Liveview blackout that happens with the old memory hack prior to recording. I could never deal with that during a production so never realized the benefits of that hack. Thank you Alex and for any others who contributed to this.
The above was using a 5D3 and a SmallHD DP4 monitor.
Today's nighly build for 7d failed. :-[
Changeset ab901b1
Quote from: N/A on July 17, 2014, 12:54:37 AM
Today's nighly build for 7d failed. :-[
looks like some missing stubs:
exmem.c:(.text+0x748): undefined reference to `CreateMemorySuite'
exmem.c:(.text+0x780): undefined reference to `CreateMemoryChunk'
exmem.c:(.text+0x7a8): undefined reference to `AddMemoryChunk'
exmem.c:(.text+0x88c): undefined reference to `DeleteMemorySuite'
simple MLV test on a 5D MARK ii with a 32gb 1000x Lexar:
NO SRM: shot at 1856x1044, 23.976fps for 16 sec
WITH SRM: shot at 1856x1044, 23.976fps for 39 sec (first time) 43 sec (second time)
very big leap for me, will test other resolutions and FPS times in the coming days.
Awesome job!!
Quote from: dmilligan on July 17, 2014, 01:10:02 AM
looks like some missing stubs:
exmem.c:(.text+0x748): undefined reference to `CreateMemorySuite'
exmem.c:(.text+0x780): undefined reference to `CreateMemoryChunk'
exmem.c:(.text+0x7a8): undefined reference to `AddMemoryChunk'
exmem.c:(.text+0x88c): undefined reference to `DeleteMemorySuite'
Gotcha, how does one go about finding these stubs? I'll try do it tonight while I'm not busy.Nevermind, found the tutorial. Yeah, it's going to take a lot longer than one night to learn that process. Poo.
Alex already committed these stubs...
Yup, just noticed that. But I did learn how to compile, so that's a plus.
372 MB buffer, 13 seconds of 2496x1044 MLV footage. GD off, no sound. Not too shabby! 8)
Card write speeds on Lexar 1000x are still a bit inconsistent, on some recordings I get a steady 79-81 MB/s, and sometimes it drops to 65-75 MB/s. Maybe a freshly formatted card would help.
Is SRM also beneficial for h.264 recording? I would think that all the extra buffer memory would assist in using higher bitrates, but anything over 100 mbps instantly buffers out. Noticing a lot of instances of the buttons becoming unresponsive after switching back and forth between the ML menu and LV, although the display is still active. Requires a battery pull to fix.
Quote from: N/A on July 17, 2014, 08:59:24 PM
Is SRM also beneficial for h.264 recording?
Canon is going to already be using all the memory they need for h.264.
Just to clarify something: we did not find new or extra memory in the cameras, we found a new memory
allocator. SRM is just the name Canon gave one of their memory allocator functions (a memory allocator basically manages and keeps track of what tasks are using what memory).
In other words we found a new way to reserve memory for ourselves (ML) and prevent Canon tasks from using it (we can't use memory unless we can be sure that another task isn't going to try to use it, for obvious reasons). Before now, we "knew" about this memory (i.e. we know what valid range of RAM address are), we just weren't sure if/when Canon was using it, and we didn't know how to reserve it for ourselves (so we couldn't use it). Canon tasks certainly already know about this allocator and make use of it as they please.
That's what I had assumed, even though I was hoping otherwise. Cheers on the explanation.
Canon 50D, card Transcend 16GB 600x TS16GCF600
Magic Lantern build jul 18 2014
Raw_rec (mlv_rec always gives the worst result)
Crop mode 1920x1080 24fps
SRM on
no raw, s-jpg2
GD on - 19s
GD off - 25s
s-raw2, no jpg
GD on - 19s
GD off - 27s
SRM off
no raw, s-jpg2
GD on - 11s
GD off - 15s
s-raw2, no jpg
GD on - 10s
GD off - 13s
always two pink frame each shot
Tragic Lantern build mar 3 2014 (no SRM)
Max Resolution
full mode - 1584x1056 (16:9 1584x892)
crop mode - 2000x1080
Raw_rec (mlv_rec always gives the worst result)
Crop mode 1920x1080 24fps
no raw, s-jpg2
GD on - 13s
GD off - 17s
no pink frames or other artifacts
ML raw module with SRM (jul 18 2014) give better recording time, but have pink frames and lower max resolution, than TL raw module (mar 3 2014)