SRM job memory buffers

Started by a1ex, July 02, 2014, 08:55:00 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

nikfreak

Quote from: Rewind on July 09, 2014, 05:14:27 PM
Also reformat card from camera for both test and see if mlv_rec allocates the 154MB before first recording. If you dont reformat you wont see it because it is already created.

@Levas please confirm...
[size=8pt]70D.112 & 100D.101[/size]

josepvm

Quote from: Greg on July 08, 2014, 07:03:58 PM
That's correct stubs?

500D :
NSTUB(0xFF07B90C,  CreateMemoryChunk)
NSTUB(0xFF07C3A8,  AddMemoryChunk)


No, these are not correct.

I have compared the dissambled code with the stubs for 550D posted above by Rewind, and I would say:

for 500D:

/** ExMem **/
NSTUB(0xFF06C0D0,  CreateMemorySuite)
NSTUB(0xFF06BEEC,  DeleteMemorySuite)
NSTUB(0xFF06B90C,  CreateMemoryChunk)
NSTUB(0xFF06C3A8,  AddMemoryChunk)


-------------edit !!!------------------------
/** SRM JobMem **/
NSTUB(0xFF02C5A0,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02D6B0,  SRM_FreeMemoryResourceFor1stJob)
-----------edit !!! ----this values are wrong !!!-------------





Tested in camera with a build of srm-memory branch.

The stubs API test completes Ok, no errors.

But I have not been able to verify the available SRM memory.   The Free Memory test shows 0MB for SRM Jobs, and ends crashing with Err. 70, I have to take battery off.
The "Don't click me!" test also crashes with Err.70

------------------------------------------------------
Edit: See my post below to see the correct values:  http://magiclantern.fm/forum/index.php?topic=12528.msg121487#msg121487


NedB

@mk11174: Man, that's service! Thanks very much for the upload. Have downloaded and will check to confirm Rewind's results for the 550D. But I have to admit I had forgotten that the SD bottleneck was only half as bad on the 650D as on the 550D, so I now realize we won't get a really great frame count for higher resolutions without some other amazing developer insight! Cheers...
550D - Kit Lens | EF 50mm f/1.8 | Zacuto Z-Finder Pro 2.5x | SanDisk ExtremePro 95mb/s | Tascam DR-100MkII

Levas

Quote from: nikfreak on July 09, 2014, 05:47:40 PM
@Levas please confirm...

Reformatted the card in camera (canon menu, with ML being restored) and checked the free memory.

Same totals.

Video mode
Shoot total 151MB and SRM total 3x34MB.

Photo mode 255MB and SRM total 4x34MB.

Also memory hack and extra mem hack in the raw-video menu didn't make any difference 

And shoot multiple clips with it, all the same recording length, same length as in older versions...
1792 x 750, 25P = 13/14 seconds of video (which is what too expect with 151MB+3x34MB)
Expect to have about 18 seconds of video(1792x750,25P) if you have 255MB shoot total and 3x34MB SRM available.





nikfreak

Quote from: Levas on July 09, 2014, 08:32:11 PM
Reformatted the card in camera (canon menu, with ML being restored) and checked the free memory.

Same totals.

Video mode
Shoot total 151MB and SRM total 3x34MB.

Photo mode 255MB and SRM total 4x34MB.

Also memory hack and extra mem hack in the raw-video menu didn't make any difference 

And shoot multiple clips with it, all the same recording length, same length as in older versions...
1792 x 750, 25P = 13/14 seconds of video (which is what too expect with 151MB+3x34MB)
Expect to have about 18 seconds of video(1792x750,25P) if you have 255MB shoot total and 3x34MB SRM available.

Thx for providing feedback.
[size=8pt]70D.112 & 100D.101[/size]

Levas

Probably already noticed by some of you,
But the SRM size is the exact same size as one raw image.

5D3 = 22MP, 22 x 14 bit / 8 = 39MB
6D = 20MP, 20 x 14 bit / 8 = 34MB
500D-550D-600D-650D etc. = 18MP, 18 x 14 bit / 8 = 31MB
500D = 15MP, 15 x 14 bit / 8 = 26MB
1100D = 12MP, 12 x 14 bit / 8 = 20MB

Levas

Quote from: mk11174 on July 09, 2014, 02:00:54 PM
I have no idea if it even works since I don't have a 6D, but it compiled, so you can test it if you want. I put in the raw_rec and mlv_rec modules with the SRM changes. And added the stubs from this thread.
https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul09.6D113.zip

It seems that it doesn't work for the 6d, does this mean the stubs aren't right ?

Just checked the previous nightly build I'm using (15-03-2014) and it says the same about the free memory available.
Shoot total of 255MB in Photo mode and 151MB in video mode (SRM is of course not mentioned in this build).
The 151MB reported in video mode does however change "magically" too 255, otherwise I could never get 13 seconds of video in 1792x750, 25P resolution (58,8MB/sec.)
SD card writes about 39MB second, (58,8 - 39) x 13 seconds = 257MB...
So the previous builds of MLV_rec gives about 255MB buffer memory for video, although it shows a shoot total of 151MB in the Free Memory debug menu.

The Build from 09-07-2014 you made, does the same thing, it shows 151MB of shoot total in the free memory debug menu, but it gives however recording times for 255MB of memory.
This could mean the new build:
Uses the 3x 34 MB SRM memory, but doesn't magically higher the 151MB shoot total too 255MB as available in photo mode.
Or
It doesn't use the 3 x 34MB of SRM memory and still free's the shoot total of 255MB of memory.

nikfreak

Quote from: a1ex on July 02, 2014, 11:25:38 PM
Some quick tests on 5D3 123, first outside LiveView, second in LV:


On 5D3, you get 163 + 3x39 in normal mode, and 127 + 4x39 in factory mode. No difference.

@Levas look at page1. a1ex posted values for 5dmk3 and also later some comment for 5dmk3 (bolded above). Looks like 6D is simpy the same. No difference!
Seems like 7D / 600D and maybe others will be winners regarding usefulness of using SRM for mlv recording...
[size=8pt]70D.112 & 100D.101[/size]

Audionut

I haven't been following this closely.  AFAIK, the SRM is never used in normal Canon tasks, so any memory from there is extra.

It looks like that comment for the 5D3, is only when calling via SRM_ChangeMemoryManagementForFactory, which, "might make a difference".  On the 5D3, calling via that routine, does not make a difference (over and above the SRM already found).

That's the way I read it.

dmilligan

That's mostly right except for:
Quote from: Audionut on July 10, 2014, 02:23:01 AM
the SRM is never used in normal Canon tasks
It is used by Canon tasks. Some of it is used by LV, if you allocate all of SRM outside of LV and then try to enter LV, you get err70 or 'please wait' and LV never starts. The rest of it is used by the tasks when taking a photo. If you allocate SRM, you can't take pictures till you free it (also err 70). Fortunately there's no need to take pictures during raw video recording.

Audionut

Oh of course.  Thanks.

Quote from: dmilligan on July 10, 2014, 02:52:43 AM
Fortunately there's no need to take pictures during raw video recording.

Be interesting to see if it can be used for silent pics, especially the full resolution ones.

Levas

Quote from: nikfreak on July 10, 2014, 01:39:59 AM
@Levas look at page1. a1ex posted values for 5dmk3 and also later some comment for 5dmk3 (bolded above). Looks like 6D is simpy the same. No difference!
Seems like 7D / 600D and maybe others will be winners regarding usefulness of using SRM for mlv recording...

The free memory values and changes between photo and video mode do look very alike for the 5d3 and 6d.
But what I don't get is how video mode/liveview can suck up so much memory on the 5d3 and 6d, there's more then 100MB taken by liveview.
How does this look in the 500d,550d,600d which for as far as I know, don't even have 100MB spare memory to be taken by live view ?

And where does the 255MB of free memory for MLV_rec we have in the old builds on the 6d come from ?
It has to do with the memory hack function available in the raw video menu (the memory hack is not needed by the new build of 09-07-14 to have 255MB of free memory, turning it on or off makes no differences).
In the description it says it takes extra memory outside of live view on the 5d3. But it works just as good on the 6d. (without the hack I have 151MB of memory for mlv recording buffer)
So can anyone tell where the "memory hack" function in the raw video menu get's it's extra memory from ?


dmilligan

Quote from: Levas on July 10, 2014, 12:07:41 PM
But what I don't get is how video mode/liveview can suck up so much memory on the 5d3 and 6d, there's more then 100MB taken by liveview.

You have to remember ML is operating in the margins. Why would Canon give the camera extra memory that it didn't need? It's a miracle there's any memory for ML to use at all. There are many reasons LV would need memory, and lots of it (want specific reasons, ask a Canon engineer, or do some reverse engineering). Also, just because memory is allocated, doesn't mean it's actually being used, it just means its reserved (and we can't use it). So LV is 'reserving' this memory not necessarily using it (but who knows, it might be using it).

Quote from: Levas on July 10, 2014, 12:07:41 PM
So can anyone tell where the "memory hack" function in the raw video menu get's it's extra memory from ?
It just gets it from shoot_malloc, like the rest of the memory it uses, it just calls shoot_malloc under different circumstances. Here's basically how the hack works (to the best of my understanding from looking at the code). Before allocating memory with shoot_malloc (which is some function similar to malloc that's in the Canon firmware), LV is paused. Then we call shoot_malloc, then LV is resumed. Because LV is paused there is more memory available to shoot_malloc (why? ask a Canon engineer), we take it, and then LV doesn't seem to mind that it's gone.

I specifically put the srm_malloc call outside of the memory hack pause in my changes to raw_rec and mlv_rec (i.e. it doesn't try to malloc while LV is paused) for safety reasons -> I didn't want srm_malloc to steal memory that LV actually needed.

What would happen if you did? It might not make any difference, you might get more srm, or you might just crash the camera or err70. If you're feeling brave, move the srm_malloc_suite(0); call next to the shoot_malloc_suite(0); and find out.

Levas

Quote from: dmilligan on July 10, 2014, 02:59:09 PM

I specifically put the srm_malloc call outside of the memory hack pause in my changes to raw_rec and mlv_rec (i.e. it doesn't try to malloc while LV is paused) for safety reasons -> I didn't want srm_malloc to steal memory that LV actually needed.

What would happen if you did? It might not make any difference, you might get more srm, or you might just crash the camera or err70. If you're feeling brave, move the srm_malloc_suite(0); call next to the shoot_malloc_suite(0); and find out.

I'm no coder at all, but everything is done in a certain sequence, I mean memory can't be called all at once.
What exactly is done, now the srm_malloc call is placed outside the memory hack ?
-Is the memory hack done first (so it pauses LV, and gets a total of 255MB's of memory)
-then calls srm_malloc (and maybe steals the 3 x 34MB from the extra memory that is gained with the live view hack, ending up with nothing extra ?)

So from what I read in your post, it is possible to put the srm_malloc call within the extra memory hack, which looks like this:
-the extra memory hack pauses live view, shoot_malloc is called, additionally we could call srm_malloc before resuming live view ?

With brave enough, you mean it is possible that I end up with a bricked camera ?
Or is it more likely that the camera runs somehow out of memory and if I get the battery out quick enough and put it back in it still works ?

dmilligan

Quote from: Levas on July 10, 2014, 03:36:47 PM
-Is the memory hack done first
No, srm is allocated first, but it really doesn't matter (AFAIK), and this is just my crude implementation we're talking about, there are issues that need to be addressed for a proper implementation that can be merged

Quote from: Levas on July 10, 2014, 03:36:47 PM
-then calls srm_malloc (and maybe steals the 3 x 34MB from the extra memory that is gained with the live view hack, ending up with nothing extra ?)
AFAIK, srm_malloc memory and shoot_malloc memory are different, you should be able to get both. The extra memory from "memory hack" is just extra from shoot_malloc => we call shoot_malloc a different way and it gives us more memory.

Quote from: Levas on July 10, 2014, 03:36:47 PM
With brave enough, you mean it is possible that I end up with a bricked camera ?
It's always a possibility, however unlikely.

Quote from: Levas on July 10, 2014, 03:36:47 PM
Or is it more likely that the camera runs somehow out of memory and if I get the battery out quick enough and put it back in it still works ?
Probably

Levas

If I get it right, the srm memory calls are implemented, but not yet in a way that it can be combined with the memory hack function.
So toggle memory hack function on and off has no influence because it isn't used at all in the build from 9 July for the 6d.

This means the implemented srm memory function does work on the 6d!  :D :D :D
Because I get raw video shooting times which corresponds with a buffer of 253MB. (151MB+ 3 x 34MB srm).
If the srm memory wasn't available we would expect shooting times which corresponds too 151MB.

For extra memory on 5d3 and 6d we now need an combination of the srm_memory function and the "memory hack" function.



josepvm

Quote from: josepvm on July 09, 2014, 06:18:13 PM

for 500D:


/** SRM JobMem **/
NSTUB(0xFF02C5A0,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02D6B0,  SRM_FreeMemoryResourceFor1stJob)




Sorry, these values I posted previously are not correct.

These are the good ones, for 500D


/** ExMem **/
NSTUB(0xFF06C0D0,  CreateMemorySuite)
NSTUB(0xFF06BEEC,  DeleteMemorySuite)
NSTUB(0xFF06B90C,  CreateMemoryChunk)
NSTUB(0xFF06C3A8,  AddMemoryChunk)


/** SRM JobMem **/
NSTUB(0xFF02A788,  SRM_AllocateMemoryResourceFor1stJob)
NSTUB(0xFF02C6F8,  SRM_FreeMemoryResourceFor1stJob)

dmilligan

Quote from: Levas on July 10, 2014, 05:06:02 PM
If I get it right, the srm memory calls are implemented, but not yet in a way that it can be combined with the memory hack function.
No that's wrong. Memory hack and shoot_malloc have nothing to do with SRM

Levas

Quote from: dmilligan on July 10, 2014, 05:29:14 PM
No that's wrong. Memory hack and shoot_malloc have nothing to do with SRM

Probably saying it wrong, I mean the piece of code that calls the srm memory is in the build of 9 Juli and does work for the 6d.
But for 5d3 and 6d users it needs to work together with the memory hack function too have use.

Right now the already available memory hack gives us the same amount of free memory as the new srm function does without the memory hack.

Greg

Quote from: josepvm on July 10, 2014, 05:25:50 PM
These are the good ones, for 500D

Thanks!  :D

500D, photo and movie mode :




Resolution                     time no SRM          time with SRM
1920 x 818 (crop)             37 frames                62 frames

10MB/s card, raw_rec

pittivale

(@nikfreak, @Levas)

This is my first post and I'm not much of a video maker.  ::)

BUT

I DO register a significant increase in record time with mlv_rec on a 6D with the nightly 2014Jul09 posted in this thread :D

I just tested various resolution (16:9) with the 2014May22 and 2014Jul09, without formatting card when changing version, and Global Draw: Allow. Write Speed showd on Display during Record: 35-38.1MB/s. Card: Lexar 600x

Here it is:











Horiz. Res.2014May222014Jul09
17923 sec. (Frame Skipped)6 sec. (Frame Skipped)
17284 sec. (Frame Skipped)7 sec. (Frame Skipped)
16006 sec. (Frame Skipped)12 sec. (Frame Skipped)
15368 sec. (Frame Skipped)15 sec. (Frame Skipped)
150410 sec. (Frame Skipped)18 sec. (Frame Skipped)
147211 sec. (Frame Skipped)20 sec. (Frame Skipped)
134438 sec. (Frame Skipped)57 sec. (Camera shuts down, File Size: 2,13GB)
128058 sec. (Camera shuts down, File Size: 2,08GB)58 sec. (Camera shuts down, File Size: 2,08GB)


Levas

Quote from: pittivale on July 10, 2014, 06:12:01 PM
(@nikfreak, @Levas)

This is my first post and I'm not much of a video maker.  ::)

BUT

I DO register a significant increase in record time with mlv_rec on a 6D with the nightly 2014Jul09 posted in this thread :D

I just tested various resolution (16:9) with the 2014May22 and 2014Jul09, without formatting card when changing version, and Global Draw: Allow. Write Speed showd on Display during Record: 35-38.1MB/s. Card: Lexar 600x

Here it is:











Horiz. Res.2014May222014Jul09
17923 sec. (Frame Skipped)6 sec. (Frame Skipped)
17284 sec. (Frame Skipped)7 sec. (Frame Skipped)
16006 sec. (Frame Skipped)12 sec. (Frame Skipped)
15368 sec. (Frame Skipped)15 sec. (Frame Skipped)
150410 sec. (Frame Skipped)18 sec. (Frame Skipped)
147211 sec. (Frame Skipped)20 sec. (Frame Skipped)
134438 sec. (Frame Skipped)57 sec. (Camera shuts down, File Size: 2,13GB)
128058 sec. (Camera shuts down, File Size: 2,08GB)58 sec. (Camera shuts down, File Size: 2,08GB)

@Pittivale

What happens to the recording times on the 2014May22 build, if you go into the raw recording menu and put the "memory hack" function on ?

pittivale

Activating memory hack increase record time to match 2014Jul09's times. (or 1 sec. less)
On pushing the record button, LiveView shuts down for a moment.

nikfreak

@pittivale thx for posting results
@Levas at least we can now be rest assured that my posted stubs are all ok and future srm related findings will work, too. If in any case new stubs need to be added in future then you can drop me a msg and I will post 'em for you. Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...

@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...

As far as I can tell we should also be able to use SRM for photo mode, too? Let's say increase burst rate and therefore fps?
[size=8pt]70D.112 & 100D.101[/size]

surami

Would it be possible to send the frames from the SRM job memory buffers to the SD or CF (camera specific) card and to the USB port (maybe with Wi-Fi read out) paralelly?

Edit: ... or split the frame and send the top half to the SD or CF card and the bottom part to the USB port. Later in post process merge the 2 parts to 1.
550D + nightly ML