SRM job memory buffers

[Audionut]

Quick test on 5D3.123.

MLV sound/Global Draw ON
No other hacks or card spanning.
Lexar 16GB 1000x
1920x1080x23.976
Default 13seconds
SRM 31seconds


1920x1280x23.976
Default 4 seconds
SRM 10 seconds

[SoulChildPaul]

Can anybody compile for 5D3.113 and .123 with mk11174 changes for me to test please

[Andy600]

I think  these are the SRM stubs for the 50D

Code: [Select]

/** SRM JobMem **/
NSTUB(0xFF0A212C,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0A247C,  SRM_FreeMemoryResourceFor1stJob)

but I'm totally lost trying to find CreateMemorySuite and DeleteMemorySuite

Also, I'm not seeing 'SRM Job total' in Free Memory. I cloned the main repo and compiled a new build for the first time in months but module building fails with [module_strings.h] Error 1. Didn't compile from the SRM branch or have Python docutils installed 

Found what I thought were the MemorySuite stubs but SRM memory shows 0x0 so they can't be correct.

[josepvm]

I think  these are the SRM stubs for the 50D
...
but I'm totally lost trying to find CreateMemorySuite and DeleteMemorySuite

I have found them for 500D, that should be quite similar, so I can give you some tips.

Search for the string "PackMemory\PackMem.c"

CreateMemorySuite should be a function like this one (500D):

 

Code: [Select]

loc_ff06c0d0: ; 18 refs
ff06c0d0: e92d41f0 push {r4, r5, r6, r7, r8, lr}
ff06c0d4: e1a07000 mov r7, r0
ff06c0d8: e3a00018 mov r0, #24
ff06c0dc: e1a08002 mov r8, r2
ff06c0e0: e1a06001 mov r6, r1
ff06c0e4: ebfeb3a1 bl loc_ff018f70
ff06c0e8: e1b04000 movs r4, r0
ff06c0ec: 059f212c ldreq r2, [pc, #300] ; ff06c220: (00000225)
ff06c0f0: 051f1158 ldreq r1, [pc, #-344] ; ff06bfa0: (ff06b860)  **"PackMemory\PackMem.c"
ff06c0f4: 028f0f47 addeq r0, pc, #284 ; ff06c218: (69755370)  *"pSuite"
ff06c0f8: 0bfea30f bleq loc_ff014d3c

And DeleteMemorySuite should be like this one:

Code: [Select]

loc_ff06beec: ; 53 refs
ff06beec: e92d47f0 push {r4, r5, r6, r7, r8, r9, sl, lr}
ff06bef0: e51f164c ldr r1, [pc, #-1612] ; ff06b8ac: (00002f60)
ff06bef4: e1a08000 mov r8, r0
ff06bef8: e5900000 ldr r0, [r0]
ff06befc: e5911000 ldr r1, [r1]
ff06bf00: e1500001 cmp r0, r1
ff06bf04: 159f20b0 ldrne r2, [pc, #176] ; ff06bfbc: (00000239)
ff06bf08: 124f1e6b subne r1, pc, #1712 ; ff06b860: (6b636150)  *"PackMemory\PackMem.c"
ff06bf0c: 128f00ac addne r0, pc, #172 ; ff06bfc0: (75537349)  *"IsSuiteSignature( hSuite )"
ff06bf10: 1bfea389 blne loc_ff014d3c


Also, I'm not seeing 'SRM Job total' in Free Memory.

Are you building the srm-memory branch?. The unified branch does not show the SRM values in FreeMemory.

[josepvm]

Found what I thought were the MemorySuite stubs but SRM memory shows 0x0 so they can't be correct.

Double check this ones:

Code: [Select]

/** SRM JobMem **/
NSTUB(0xFF0A212C,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF0A247C,  SRM_FreeMemoryResourceFor1stJob)


They do not contain the strings "SRM_FreeMemoryResourceForDDD" or "SRM_AllocateMemoryResourceForDDD" but are called from places where these strings are present.

for SRM_AllocateMemoryResourceFor1stJob (500D) :

Code: [Select]

loc_ff0a8c18:
ff0a8c18: e2841b03 add r1, r4, #3072 ; 0xc00
ff0a8c1c: e59f0194 ldr r0, [pc, #404] ; ff0a8db8: (ff0a820c)
ff0a8c20: e281108c add r1, r1, #140 ; 0x8c
ff0a8c24: ebfe06d7 bl loc_ff02a788           // ---->>> This is the function
ff0a8c28: e59b0034 ldr r0, [fp, #52] ; 0x34
ff0a8c2c: e3a01064 mov r1, #100 ; 0x64
ff0a8c30: ebff0326 bl loc_ff0698d0
ff0a8c34: e3500000 cmp r0, #0
ff0a8c38: 11a03000 movne r3, r0
ff0a8c3c: 128f2f5e addne r2, pc, #376 ; ff0a8dbc: (5f4d5253)  *"SRM_AllocateMemoryResourceFor1stJob failed [%#x]"

And for SRM_FreeMemoryResourceFor1stJob (500D):

Code: [Select]

loc_ff0a9424:
ff0a9424: e5900c8c ldr r0, [r0, #3212] ; 0xc8c
ff0a9428: e3500000 cmp r0, #0
ff0a942c: 13a02000 movne r2, #0
ff0a9430: 13a01000 movne r1, #0
ff0a9434: 1bfe0caf blne loc_ff02c6f8      // ------->>> This is the function
ff0a9438: ebff0185 bl loc_ff069a54
ff0a943c: e51f1760 ldr r1, [pc, #-1888] ; ff0a8ce4: (00003f00)
ff0a9440: e591001c ldr r0, [r1, #28]
ff0a9444: e2500001 subs r0, r0, #1
ff0a9448: e581001c str r0, [r1, #28]
ff0a944c: 1a000004 bne loc_ff0a9464
ff0a9450: ebff1018 bl loc_ff06d4b8
ff0a9454: e28f2f72 add r2, pc, #456 ; ff0a9624: (6c656b53)  *"SkeltonJob Done"
ff0a9458: e3a01016 mov r1, #22
ff0a945c: e3a0008f mov r0, #143 ; 0x8f
ff0a9460: ebfef58c bl loc_ff066a98

In this one, search for "SkeltonJob"

[Andy600]

@josepvm - thanks for your help

I'm compiling from the SRM branch now - wasn't before 

This is the only one I can see as possible for CreateMemorySuite (it has 'ldreq' and I'm looking for similarities to what you posted)

Code: [Select]

loc_ff069068: ; 3 refs
ff069068: e92d41f0 push {r4, r5, r6, r7, r8, lr}
ff06906c:    e1b07002 movs r7, r2
ff069070: e1a08001 mov r8, r1
ff069074: e1a05000 mov r5, r0
ff069078: 059f114c  ldreq r1, [pc, #332] ; ff0691cc: (ff86895c)  **"PackMemory\PackMem.c"
ff06907c:         059f217c  ldreq r2, [pc, #380] ; ff069200: (000001f7)
ff069080: 028f0f5f         addeq r0, pc, #380 ; ff069204: (50282821)  *"!((PVOID)hClientPackHeap == NULL)"
ff069084: 0bfeae8d bleq loc_ff014ac0

for DeleteMemorySuite could it be:

Code: [Select]

ff068fb8: e92d47f0 push {r4, r5, r6, r7, r8, r9, sl, lr}
ff068fbc: e51f1624 ldr r1, [pc, #-1572] ; ff0689a0: (00002e7c)
ff068fc0: e1a08000 mov r8, r0
ff068fc4: e5900000 ldr r0, [r0]
ff068fc8: e5911000 ldr r1, [r1]
ff068fcc: e1500001 cmp r0, r1
ff068fd0: 159f2208 ldrne r2, [pc, #520] ; ff0691e0: (00000236)
ff068fd4: 124f1d1a subne r1, pc, #1664 ; ff06895c: (6b636150)  *"PackMemory\PackMem.c"
ff068fd8: 128f0f81         addne r0, pc, #516 ; ff0691e4: (75537349)  *"IsSuiteSignature( hSuite )"
ff068fdc: 1bfeaeb7 blne loc_ff014ac0

not sure which addresses to try though....I'm a total newb at this bit

[Andy600]

ah, just saw your next post

Code: [Select]

ff8a231c:    e2811008 add r1, r1, #8
ff8a2320: ebfe23cb bl loc_ff82b254
ff8a2324: e5970028 ldr r0, [r7, #40] ; 0x28
ff8a2328: e3a01064 mov r1, #100 ; 0x64
ff8a232c:         ebff1180 bl loc_ff866934 <------------------------ this is referenced all the time when I search for SRM_AllocateMemoryResource so that must SRM_AllocateMemoryResourceFor1stJob?
ff8a2330: e3500000 cmp r0, #0
ff8a2334: 11a03000 movne r3, r0
ff8a2338: 124f2e3d subne r2, pc, #976 ; ff8a1f70: (424f4a5b)  *"[JOB ERROR] SRM_AllocateMemoryResourceFor1stJob failed [%#x]"


[josepvm]

DeleteMemorySuite looks Ok, but I'm not equally convinced about CreateMemorySuite.

Look for a similar structure in an address a little bit further:  0xFF069118 or near this value.
I have observed in 500D and in 550D a similar offset between DeleteMemorySuite and CreateMemorySuite. 

[josepvm]

Code: [Select]

ff8a231c:    e2811008 add r1, r1, #8
ff8a2320: ebfe23cb bl loc_ff82b254
ff8a2324: e5970028 ldr r0, [r7, #40] ; 0x28
ff8a2328: e3a01064 mov r1, #100 ; 0x64
ff8a232c:         ebff1180 bl loc_ff866934 <------------------------ this is referenced all the time when I search for SRM_AllocateMemoryResource so that must SRM_AllocateMemoryResourceFor1stJob?


Not this one. The previous call, three lines before, is the stub you are searching for :  loc_ff82b254

[mk11174]

Can anybody compile for 5D3.113 and .123 with mk11174 changes for me to test please

I can only do the 5D3.113 because it is in the source I have.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul13.5D3113.zip

[keikun007]

600d and 650d are giving better results than 60d? how come?

[a1ex]

I'm back, glad to see the progress. Just give me some time to digest the new stuff, and I'll handle the pull requests in the next few days.

[Andy600]

@josepvm or @a1ex - can you please specify the strings I need to look for for each stub needed (I think there are 5 missing stubs in total for SRM to work on the 50D!?) I'm getting a bit lost when it comes to MemoryChunk etc and still don't really understand what I'm looking at despite many reads of this and the other thread

So for instance, for the

Code: [Select]

SRM_AllocateMemoryResourceFor1stJob stub , I searched for SRM_AllocateMemoryResourceFor1stJob failed [%#x] in Notepad++ which got me close (although  I was still wrong) and I presume

Code: [Select]

ff82b254 is correct as you indicated. I'm trying to see the pattern here because it doesn't make sense in my head yet.

Thanks again for your help . I'm not sure if there are any devs still working on the 50D so I would like to at least try and keep it up to date with my extremely limited code knowledge.

[ayshih]

I'm not sure if there are any devs still working on the 50D so I would like to at least try and keep it up to date with my extremely limited code knowledge.

I'm still alive!  I just got back from a four-week work trip, so I'll need to get back up to speed on ML and then help you track down stubs.

[ayshih]

I've submitted a PR with the 50D stubs.  Thanks to @Andy600 and @josepvm for getting that ball rolling!  Here are the screenshots (outside LV, LV, and don't click me):





However, I starting taking a few pictures during the "don't click me" test, and on the second, I hit an "Err 80".  Has anyone else encountered any issues?

[ayshih]

However, I starting taking a few pictures during the "don't click me" test, and on the second, I hit an "Err 80".  Has anyone else encountered any issues?

Hmm, I haven't been able to reproduce this with further testing...

[Andy600]

216mb extra! Nice

Just tried MLV and got this message:

Code: [Select]

Hack error at 0
expected e3a00032, got e1a00000

It recorded about the same amount of frames as without SRM. Also, you can get into crop mode only once and the zoom buttons become non functional requiring a battery pull.

And, has raw REC button been remapped? I get h264 recording with the SET button and raw with the LV (blue) button.


UPDATE: backing out of https://bitbucket.org/hudson/magic-lantern/commits/964f3df seems to have fixed all the problems Thanks a1ex!

[Audionut]

I noticed some undesirable behaviour with crop mode on a 5D3.

But haven't had the time to test things thoroughly.

[dsManning]

Hmm, I haven't been able to reproduce this with further testing...

Shots during "Don't Click Me" test worked fine for me on the 50D.  After trying to shoot .mlv with extra memory worked until card was full (forgot to clear it first).  Haven't seen error 80 yet, but trying on a clean 1000x card now to see how many frames I can get in crop mode.

Thanks for getting this going on the 50D, I like Andy was having a hard time looking for the stubs until it was cleared up a few posts above.

[a1ex]

Had a quick play with the raw recorders.

Good news: the memory hack is no longer needed on 5D3. The total memory with hack + SRM, usable for raw recording, is actually lower than just with SRM. This cleans up the code quite well, and removes the annoyances of the old memory hack.

Give it a quick try, and if all goes well, it will be in the nightly builds tonight or tomorrow. Including 6D and 500D.

[Levas]

Had a quick play with the raw recorders.

Good news: the memory hack is no longer needed on 5D3. The total memory with hack + SRM, usable for raw recording, is actually lower than just with SRM. This cleans up the code quite well, and removes the annoyances of the old memory hack.

Give it a quick try, and if all goes well, it will be in the nightly builds tonight or tomorrow. Including 6D and 500D.

According to some tests on the 6d, memory hack VS SRM made no difference

Activating memory hack increase record time to match 2014Jul09's times. (or 1 sec. less)
On pushing the record button, LiveView shuts down for a moment.

The memory hack gives an additional of 100MB, the SRM gives 3x 34MB on 6d.
How can that SRM only gives more memory on 5d3 then memory hack + SRM.
Am I missing something here ?

[a1ex]

The memory is managed in different ways by Canon in different modes (LV, outside LV, RAW, JPEG and so on). Some of it is reserved for shoot_malloc (for compressing photo data after capture), some of it is for SRM (for uncompressed just-captured raw photo data), some is for the operating system, some is for LiveView and so on. The largest ones are shoot, SRM and LiveView buffers, and their sum is more or less constant.

This means, the benefit for 6D is just cleaner code and a smaller delay before starting to record.

[Levas]

The memory is managed in different ways by Canon in different modes (LV, outside LV, RAW, JPEG and so on). Some of it is reserved for shoot_malloc (for compressing photo data after capture), some of it is for SRM (for uncompressed just-captured raw photo data), some is for the operating system, some is for LiveView and so on. The largest ones are shoot, SRM and LiveView buffers, and their sum is more or less constant.

This means, the benefit for 6D is just cleaner code and a smaller delay before starting to record.

Just to be sure if I understand this correct:
Are you saying we can't have best of both worlds, 100MB from the hack and 3 x 34 MB SRM memory ?
In video mode on 6d we have:
Shoot total 151MB and SRM total 3x34MB. (Total of 253MB)
In photo mode we have:
Shoot total 255MB and SRM total 4x34MB. (total of 391MB)

Don't know if I'm right, but my guess is there's enough total memory in the 6d to have (some) best of both worlds.
Right know, with SRM only, without the extra memory hack, there's 140MB hold back, live view of course need some of it, but I don't expect live view needs all 140MB.

[a1ex]

There are more allocators, not just shoot_malloc and SRM. If you trace the references to the worker function from AllocateMemory, you'll find around 10 or 20, but most of them are small. In addition, there are many small allocators in the resource manager. Feel free to reverse engineer them and squeeze even more RAM.

The total RAM size in these cameras is 512 MB, and 256 MB on most Rebels. That colorful line from the free memory dialog shows what we know about it, and from unknown areas, it tries to guess what's used and what might be free. A better guess can be obtained if you compile with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (but this delays the boot process a bit).

BTW, what's the reason for quoting the entire post right above yours? It would just double the thread length, if everybody would do the same.

[Levas]

Sorry about the quoting habit     
Looks indeed very stupid, if it's exactly one post under it   

The thing I'm worried about is that the extra memory hack is declared obsolete and will be deleted in the current builds.
I'm only worried about this because it looks like it can be used on the 6d aside with the SRM code.

Would love to test if the extra memory hack can be used in combination with the SRM code, but I don't have any coding skills.
And from previous post it seems, the build we tested on the 6d with the SRM code in it, somehow disables the extra memory hack.

If it turns out that extra memory hack and SRM code don't work together on the 6d, I see no reasons to not clean up the code by deleting the extra memory hack.
The way I see it, as a person without coding skills  , this could be a relatively easy way to get some extra memory for raw video buffer.
But then again, I have no coding skills, so maybe things are a lot harder to implement...