SRM job memory buffers

[Greg]

On the 500D picture quality (jpg raw) has an effect on the SRM.

             Photo mode          Movie mode
RAW         4x26MB                 2x26MB
JPG           3x26MB                 1x26MB

[Levas]

Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...

Have absolutely no idea how to do that, can anybody else help us here...

@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...

Don't know if I'm right, but my understanding is that raw video/photo data can't be compressed.
How would you compress data that's coming from a bayer pattern ? You need every pixel data to demosaic...

[dmilligan]

@all: as I am new to all this ML stuff and coming from android: I was able to use zram as kind of swapping to boost android roms and compress stuff into RAM. Could something like that be done for ML or am I pingpong'ing my brain too much about that? Such techniques would increase cpu load but offer around twice the amount of data to be compressed into RAM and as RAM (in our case here SRM, right?) is speedy enough this could definitiley increase continuous recording times. Could be worth a try but I don't know if zram modules could be adopted to ML OS...

No way, it would require real-time compression of huge amounts of data on a dinky little ARM CPU. If we could compress in real-time we wouldn't need as much extra memory anyway, cause we could just write it out to the card compressed, thus requiring a lower card speed.

As far as I can tell we should also be able to use SRM for photo mode, too? Let's say increase burst rate and therefore fps?

Nope, photo mode is already using SRM when it takes pictures, and this is very likely what it is used for. When we allocate SRM, photo mode doesn't work (this is probably the 11th time I've stated this).

Don't know if I'm right, but my understanding is that raw video/photo data can't be compressed.
How would you compress data that's coming from a bayer pattern ? You need every pixel data to demosaic...

There are lots of ways to losslessly compress raw data (or any data for that matter). CR2 is actually compressed. It's not a matter of how, but the fact that there is just way to much data coming in way to fast. You can actually lossy compress raw data too, one example would be to reduce the bit depth from 14 to say 10. This is technically "compression" and it's lossy (the last 4 bits are lost), but the image can still be reconstructed. Unfortunately the CPU isn't even powerful enough to chop bits off. In fact the CPU isn't even fast enough to move the raw data around, we have to use the edmac (external memory controller) to move raw data around.

[josepvm]

On the 500D picture quality (jpg raw) has an effect on the SRM.

             Photo mode          Movie mode
RAW         4x26MB                 2x26MB
JPG           3x26MB                 1x26MB

Good point! 

I see the same results in my 500D.  And RAW+JPEG gives the same values than JPEG alone (3x26MB Photo, 1x26MB video)

So it seems the camera reserves one SRM buffer to convert the image to JPEG.

That's good, an added motivation to take photos always in RAW   

[Audionut]

If in any case new stubs need to be added in future then you can drop me a msg and I will post 'em for you. Now i think it's time to commit the stubs to the repository so that they are merged into nightlies. If anyone can do this I would be grateful...

I think we need CreateMemorySuite and DeleteMemorySuite before creating a pull request.

[pompeiisneaks]

These changes are not yet included in any nightly build (we are reverse engineering it as we speak, and sharing the details as soon as we discover them). Please be patient, and hope this will motivate the 6D developers to bring the port up to date.

Some quick tests on 5D3 123, first outside LiveView, second in LV:

I just did this on mine, here's the output on 5D3.123, seems the same.

[dyfid]

Anyone with a 700D build? The recent nightly's won't boot, but the original 14th Feb Alpha boots everytime. http://www.magiclantern.fm/forum/index.php?topic=12617.new#new and everything's ok with nightly 550D's with SRM so I think I'm doing it right.

[mk11174]

Anyone with a 700D build? The recent nightly's won't boot, but the original 14th Feb Alpha boots everytime. http://www.magiclantern.fm/forum/index.php?topic=12617.new#new and everything's ok with nightly 550D's with SRM so I think I'm doing it right.

Of course, I already took care of the 700D, will upload in a bit for you. 

SRM version for 700D https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip

By the way, I just checked the recent Nightly from here http://builds.magiclantern.fm/#/ and it boots fine. The nighties do not contain the patches for the SRM_Memory though, only these test builds in this thread have the srm_memory patch.

[nikfreak]

I think we need CreateMemorySuite and DeleteMemorySuite before creating a pull request.

reateMemorySuite and DeleteMemorySuite already exist:
https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified


Just those missing that I discovered and posted here....

[dyfid]

Of course, I already took care of the 700D, will upload in a bit for you. 

SRM version for 700D https://bitbucket.org/mk11174/magic-lantern/downloads/magiclantern-Nightly.2014Jul11.700D113.zip

By the way, I just checked the recent Nightly from here http://builds.magiclantern.fm/#/ and it boots fine. The nighties do not contain the patches for the SRM_Memory though, only these test builds in this thread have the srm_memory patch.

Thanks for the SRM build, but hmm, still no good, with Feb14th build on the card the 700D boots, when I put your build on the card, the battery flashes on/off as soon as the card is detected and it won't boot and I have to pull the battery with the card out to reset for it to boot after at all, even with no card in. The Canon Firmware is 1.1.1, but still the Feb14th works fine. What am i doing wrong?

[mario1000]

I just did this on mine, here's the output on 5D3.123, seems the same.

Do you have a compiled version for 5D3.113?

[Audionut]

reateMemorySuite and DeleteMemorySuite already exist:
https://bitbucket.org/hudson/magic-lantern/src/c62e719cc9d54c6e801e5e1d97f880cfa9ad8d8f/platform/6D.113/stubs.S?at=unified


Just those missing that I discovered and posted here....

Oh yes thanks.

Done:  https://bitbucket.org/hudson/magic-lantern/pull-request/537/6d-missing-stubs-from-nikfreak-http/diff

[mk11174]

Thanks for the SRM build, but hmm, still no good, with Feb14th build on the card the 700D boots, when I put your build on the card, the battery flashes on/off as soon as the card is detected and it won't boot and I have to pull the battery with the card out to reset for it to boot after at all, even with no card in. The Canon Firmware is 1.1.1, but still the Feb14th works fine. What am i doing wrong?

Well there you go, that is your problem, you have to update canon firmware to 113, the new ports are 113 not 111, so go to canon and update your camera, then get the nightly and reflash with the FIR file in the nightly and your good to go with ML, and if you want the SRM just use the zip in this thread.

[vertigopix]

Hi !
Does anybody have a 60D version ? So i can give you my results.

Cheers,

Christian

[dyfid]

Well there you go, that is your problem, you have to update canon firmware to 113, the new ports are 113 not 111, so go to canon and update your camera, then get the nightly and reflash with the FIR file in the nightly and your good to go with ML, and if you want the SRM just use the zip in this thread.

That was it,  , 154mb reserved but now shooting as 20MB/s instead of 40MB/s with 1.1.1 & Feb14th ML? So not getting many frames, sure it was continuous at 720p?. Sandisk Extreme 95MB/s, ExFAT via EOS Card.

[pompeiisneaks]

Do you have a compiled version for 5D3.113?

I don't.  Realize this is a test version, mine couldn't take a picture, it kept saying 'busy' but I think that's because the code used is designed just to see if the ML code can allocate the memory.  If you want me to, I can do a build for 5D3.113, but I'm not going to test it first because rolling back to .113 is a pita.  So I won't have a chance to test it out, and it may not work. 

~Phil

[pompeiisneaks]

Here's the .113 version, again all the usual warnings, I used my build for a different .113 to test external monitors and it worked fine, but ymmv, but I'm pretty sure the codebase is stable and as usable in the srm-memory fork as any other.

https://www.dropbox.com/s/s136ezkzzad66hy/magiclantern-Nightly.2014Jul11.5D3113.zip

[escho]

Here the results of the mittelfranconian jury for my 600D:

Normal shoot mode:



LiveView:



Test with "don´t click me"



No problems so far

Edgar

[dyfid]

Can anyone confirm what MB/s they're getting out of a 700D raw with the SRM code and FW 1.1.3?

I now only get 20MB/s same as a 550D where as before with FW 1.1.1 and the 700D Feb14th Alpha with exact same card, Sandisk Extreme Pro 95MB/s I was getting 40MB/s and continuous 720P. Can't get anywhere near that now.

[mk11174]

Can anyone confirm what MB/s they're getting out of a 700D raw with the SRM code and FW 1.1.3?

I now only get 20MB/s same as a 550D where as before with FW 1.1.1 and the 700D Feb14th Alpha with exact same card, Sandisk Extreme Pro 95MB/s I was getting 40MB/s and continuous 720P. Can't get anywhere near that now.

Try using the nightly builds and see what you get, dont compare the builds from 1.1.1, with the SRM build, because there was no build for 1.1.1 with SRM, so to compare, you need to use the regular nightly builds and compare what you use to get when you were at 1.1.1.

And also what module are you testing, the raw_rec or mlv_rec or just a regular movie???

I have same card and these are my results in 1920x1080@24p setting both modules res to 1280x720


MLV_REC


RAW_REC

Both jump around during record and give a normal reading as it did back in 1.1.1.

When using the SRM build, it does not affect the mb/s as far as I can tell, just records more frames.

[dmilligan]

Cleaned up my changes and created a pull request. It is mostly functionally the same as before, the code is just cleaner, and now there's a menu option to enable/disable srm memory for comparison purposes. If you can compile, please test.

https://bitbucket.org/hudson/magic-lantern/pull-request/540/update-raw_rec-and-mlv_rec-to-use-new-srm/diff

[mk11174]

Cleaned up my changes and created a pull request. It is mostly functionally the same as before, the code is just cleaner, and now there's a menu option to enable/disable srm memory for comparison purposes. If you can compile, please test.

https://bitbucket.org/hudson/magic-lantern/pull-request/540/update-raw_rec-and-mlv_rec-to-use-new-srm/diff

Great idea, especially for testing, so much easier to test it out!

[Rewind]

This version of raw_rec is slower than previous.

results for 650D, 23.976 fps, GD off:

Resolution               First Implementation              This version
1568 x 656                62 s                                           50 s
1600 x 672                32 s                                           27 s
1728 x 720                12 s                                           10 s

[escho]

Short test with my 600D, using mlv_rec:

Code: [Select]

Recording 1728 x 972 until 1st skipped frame

1st try: without SRM: 28 frames, with SRM: 61 frames
2nd try: without SRM: 31 frames, with SRM: 59 frames

Code: [Select]

Recording 1728 x 972, 10s, skipped frames allowed

1st try: without SRM: 79 frames, with SRM: 110 frames

Really nice

Edgar

[dmilligan]

did some more tests, in general mlv_rec is outperforming raw_rec by a few frames in whatever mode

also recording times for crop mode are not quite as good, though this doesn't seem to be related to buffer space, looking at the trace, the same amount of buffer space is available, 353MB in 126 slots for 1728x972