Author Topic: New memory backend  (Read 26765 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #25 on: October 05, 2013, 04:26:31 PM »
Do you have some screenshots that show the source of the error?

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New memory backend
« Reply #26 on: October 05, 2013, 04:44:38 PM »
I would have to make you a gif/video of it eating the memory...

I had 2 problems:

1. 7D using alloc_dma_memory for FPS from slave (right inside eng_drv_out_lv)
Double free errors, underflow
*changed this to only allocate once at init, its just sizeof(uint32_t)
2. 6D murdered recording wav + H264
http://www.magiclantern.fm/forum/index.php?topic=8657.msg80747#msg80747


I changed *most* calls to use fio_malloc... small aloc to tmp_malloc 6D records for 27min +, 7d does 3 minutes and freezes

This morning I worked on 7d some more; boggled by the hard lock... go to bypass the backend completely, find I've missed the alloc_dma_memory calls in wav init

change to:
Code: [Select]
wav_buf[0] = (int *) _alloc_dma_memory(WAV_BUF_SIZE);
    wav_buf[1] = (int *) _alloc_dma_memory(WAV_BUF_SIZE);
   

so far so good knock on wood, recorded over 5 mins no lock...

step 3 is going back to fio/tmp malloc calls and fixing above then making sure 7D still lockup free.

what else could this be affecting?

*had another thought... could those calls actually be freed while recording after ~3 mins?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #27 on: October 05, 2013, 05:02:58 PM »
The backend is most likely catching a nasty bug, so don't bypass the backend just because it appears to work. This is not an error that can be safely ignored. The side effects for this kind of error vary from not noticeable in practice, to camera acting like drunk without finding any logical explanation.

I've recorded some short WAV clips on 5D2, all seems OK.

I'll try to check the 7D next week.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New memory backend
« Reply #28 on: October 05, 2013, 05:34:23 PM »
I kno... there is a memory leak... either I forgot a free or bypassing it shows beep.c has the leak in general record over 2 mins and look at memory in debug menu... I think part of the problem is small alloc, I don't think its possible to replace it with tmp_malloc.

I did this:

Code: [Select]
    wav_buf[0] = (int16_t*) _alloc_dma_memory(WAV_BUF_SIZE);
    wav_buf[1] = (int16_t*) _alloc_dma_memory(WAV_BUF_SIZE);
   
    rootq = SmallAlloc(sizeof(WRITE_Q));

Used fio_malloc/small for the rest and now still freezing, I'm going to try more combos like all fio_malloc/etc.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #29 on: October 05, 2013, 05:38:00 PM »
The alloc type is only a hint for choosing the allocator (whether it will it prefer to go into malloc/AllocateMemory/shoot_malloc/whatever).

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New memory backend
« Reply #30 on: October 05, 2013, 07:16:56 PM »
It seems to be choosing malloc tho..

No memory wrapper... malloc doesn't leak, allocatemem leaks very slowly, no freezing

FIO_malloc/_malloc + _alloc_dma (for initializer) - malloc leaks ~1kb/min - allocatemem doesn't leak (i think), no freezing for ~5mins

FIO_malloc/_malloc - malloc leaks ~1kb/min - allocatemem doesn't leak (i think), no freezing 6mins

fio_malloc/tmp_malloc - freeze ~2 min.

On 6D just fixing dma_memory to fio_malloc appears to fix everything.

I think the issue *might* be that it allocates dynamically based on the data its holding, if its too big it allocates it larger and then when it goes to free it may be freeing the wrong thing.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #31 on: October 05, 2013, 07:37:45 PM »
Understood. I think we need to rewrite it from scratch, with the method used for burst silent pics (zero CPU overhead) and maybe a buffering algorithm similar to raw_rec (or maybe just double-buffering with a larger buffer should do the trick). This will also remove the need for dma_memcpy.

Also I'd like an option to pass the buffers filled with audio data via a CBR (instead of saving to a file), which should make it easy to integrate audio in MLV.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New memory backend
« Reply #32 on: October 05, 2013, 07:50:58 PM »
Would be nice to have a rewrite... I tried edmac memcpy there and it would kill the screen/cam as soon as you left the wav menu...

Also if you rewrite, everything seems to be working with stereo int 16 sampled as 2 uint8, both channels are recorded.

16->16, had hiccups because of size but may not if using the silent pic method/edmac

the mono 8 was just :( in terms of quality and inability to do internal L, ext R, etc.


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #33 on: October 05, 2013, 08:27:32 PM »
Cool. It may be a good idea to write down what's needed for an audio API (e.g. a function for playing custom wav data, like in g3gg0's SMPTE module, or maybe sending out RS232 signals, or a function to record short samples of audio, maybe somebody wants to write a module that does voice recognition or plot a spectrogram? stuff like that)

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New memory backend
« Reply #34 on: October 05, 2013, 09:07:39 PM »
Selecting quality (maybe you only need 22.1 ref audio), maybe fixed duration, rs232/timecode out the headphone (or pseudo headphone). Or genlock can be done this way... cam 1 hp -> cam2 mic, vc versa etc

One rather nice thing, write the header at the end so you have real lenght.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #35 on: October 05, 2013, 09:38:07 PM »
Real length is possible now, since we have a file seek function. Back then we didn't.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #36 on: October 24, 2013, 10:15:03 AM »
For double free, I think I found the bug: if you look at add_write_q and write_q_dump, the latter is allowed to save a queue that was not yet filled completely. When this happens, that buffer is saved (and freed) twice.

It's not a bug in the memory backend, but in the wav recording code.

At a closer look, the bug is severe (the recording task will write on deallocated memory, so you can expect any kind of side effects), so I'll disable this feature until a solution is found.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3105
Re: New memory backend
« Reply #37 on: October 24, 2013, 03:11:30 PM »
ah, finally. nice :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #38 on: October 24, 2013, 04:01:13 PM »
Yeah, my bad for accepting code without fully understanding how it works.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New memory backend
« Reply #39 on: October 24, 2013, 05:37:03 PM »
What about static allocation of the buffer and only free when recording is finished?

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3105
Re: New memory backend
« Reply #40 on: October 24, 2013, 05:39:55 PM »
understanding the code? thats  rarely possible in reality.

static allocation? yep. allocation on recording, free on finishing.
permanent alloc/free/alloc makes no sense for such large blocks. (it really does that?)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #41 on: October 24, 2013, 05:46:38 PM »
My old implementation had just two buffers and no memcpy (but I think it did file I/O from the CBR). I'm thinking to try it again.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: New memory backend
« Reply #42 on: October 24, 2013, 05:49:18 PM »
Seems to use only a few 100KB, but I meant static allocation while recording instead of allocate -> write -> free
It looks like it uses 3*~200KB or so.

Otherwise maybe do it like silent pics but using allocate mem instead of shoot malloc.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #43 on: February 12, 2014, 03:40:59 PM »
Some tweaks to the memory backend (including a fix for err70 when memory gets fragmented):

https://bitbucket.org/hudson/magic-lantern/pull-request/384/memory-backend-improvements

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #44 on: February 12, 2014, 07:37:41 PM »
g3gg0 wrote a really cool module that exercises the memory backend: https://bitbucket.org/hudson/magic-lantern/commits/c7d8de335e664504755d4d2d7a0b6af1db995042

After tweaking it to be a little more aggressive (allocate a lot of small blocks and keep them allocated for a while), I ran a little test on 60D (of course, with the latest tweaks applied). There are 18 tasks flooding the backend with random malloc/free requests, and the total load varies around 10-50 MB:


mk11174

  • Hero Member
  • *****
  • Posts: 809
Re: New memory backend
« Reply #45 on: February 12, 2014, 07:47:09 PM »
On 700D the stub for GetMemoryInformation = 0x7498 so according to my IDA am I right that the GetSizeofMaxRegion =  0x7444???
500D/T1i  550D/T2i  600D/T3i  700D/T5i

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #46 on: February 12, 2014, 07:51:12 PM »
Yep. Easy to find, just time-consuming.

nanomad

  • Administrator
  • Hero Member
  • *****
  • Posts: 2918
  • All your websites are belong to us
Re: New memory backend
« Reply #47 on: February 12, 2014, 07:53:44 PM »
Suggestion: you can load an additional binary in IDA at the ram functions address space. It will make your life much easier.
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #48 on: February 13, 2014, 02:37:45 PM »
More updates:

- much faster allocation (the underlying memcheck library was filling every single buffer with 0xA5)
- updated heuristics (it should no longer try to allocate tiny blocks from shoot_malloc)
- re-implemented cache bit mangling => g3gg0's flood tests passed on 5D3 too, not just 60D (right now 2600 iterations x 8 tasks and counting)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11470
  • 5D Mark Free
Re: New memory backend
« Reply #49 on: February 13, 2014, 05:34:18 PM »
A bunch of flood tests in parallel on 5D3 (8 threads):


5D2 is even stronger (25 threads). The screenshot function can no longer keep up with screen redraws :D