Author Topic: uncompressed 14-bit RAW video recording  (Read 762795 times)

minimimi

  • Freshman
  • **
  • Posts: 68
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #125 on: May 04, 2013, 07:06:08 AM »
I see thanks. I will try it when I going back to home.
BTW, Alex is already suggested "memcpy is slow" , So I don't know we can get good result .
It's my hobby.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #126 on: May 04, 2013, 07:30:47 AM »
Dma memcpy is faster but I don't know what it will do while using the dma for other things.

Also can mess with exmem.c make the step bigger than 5.. the backup has to be > than the step or it will freeze the second time you shoot.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #127 on: May 04, 2013, 10:20:13 AM »
(not very active this weekend due to family events)

there are 3 possibilits for memcpy:
 - CPU: Load and Store words using more or less fast opcodes (LDMIA, STMIA with many registers)
 - DMA: simple DMA controller
 - EDMAC: special SDRAM DMA controller

the last one (EDMAC) seems to be the fastest, but is a bit complex to set up as it supports line skipping and stuff.
it is used also for transferring image (raw, yuv, jpeg) data between hardware like compressors, converters etc and RAM

DMA is a simple one and seems to be slower than EDMAC but faster than CPU.

CPU is the slowest of course ;)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #128 on: May 04, 2013, 06:06:44 PM »
Weird that dma copy == corrupt headers with dng burst. All 3 are separate, thats good at least.

shuggyrasklat

  • New to the forum
  • *
  • Posts: 38
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #129 on: May 04, 2013, 06:21:25 PM »
P337, I shot it with the burst feature of silent picture on the 5dmkii. With the "YUV/DNG recorder," you mean with the new mkiii firmware, or is it a newer feature, within ML, that I am unaware of?

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #130 on: May 04, 2013, 06:44:11 PM »
5dII modules aren't enabled.

shuggyrasklat

  • New to the forum
  • *
  • Posts: 38
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #131 on: May 04, 2013, 08:45:47 PM »
1%, is that being worked on?

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #132 on: May 04, 2013, 08:58:12 PM »
Hopefully... I don't have 5d2... I did 600D/EOSM (console broken) and 6D

minimimi

  • Freshman
  • **
  • Posts: 68
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #133 on: May 05, 2013, 03:02:41 AM »
(not very active this weekend due to family events)
Enjoy with your family.

there are 3 possibilits for memcpy:
 - CPU: Load and Store words using more or less fast opcodes (LDMIA, STMIA with many registers)
 - DMA: simple DMA controller
 - EDMAC: special SDRAM DMA controller
Let me confirm.
CPU
 -malloc & memcpy()
DMA
 -shoot_malloc & dma_memcpy()
EDMAC
 -CreateMemorySuite & (not found yet)

Correct?

And now I finding address of dma_memcpy, but I can't find it yet.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #134 on: May 05, 2013, 03:43:03 AM »
It should be
NSTUB(0xFF9EB650 - RAM_OFFSET, AJ_HP_Copy_n_DMA_channel_n_BLTDMA)

minimimi

  • Freshman
  • **
  • Posts: 68
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #135 on: May 05, 2013, 04:13:49 PM »
It should be
NSTUB(0xFF9EB650 - RAM_OFFSET, AJ_HP_Copy_n_DMA_channel_n_BLTDMA)

Thanks . But not enough speed..
Result is more slow with g3ggo's original method.

Test code is here.
Code: [Select]
diff -r 99bd629f7c57 modules/lv_rec/lv_rec.c
--- a/modules/lv_rec/lv_rec.c   Tue Apr 30 22:18:21 2013 +0200
+++ b/modules/lv_rec/lv_rec.c   Sun May 05 23:09:11 2013 +0900
@@ -416,8 +417,12 @@

         if(!skip_saving)
         {
-            FIO_WriteFile(save_file, UNCACHEABLE(&save_data->chunkData.chunkAddress[save_data->chunkData.chunkOffset]), avail);
+            dma_memcpy(shmem+written,UNCACHEABLE(&save_data->chunkData.chunkAddress[save_data->chunkData.chunkOffset]),avail);
             written += avail;
+            if(written == length)
+           {
+                FIO_WriteFile(save_file, shmem , length);
+            }
+
         }
diff -r 99bd629f7c57 platform/5D3.113/stubs.S
--- a/platform/5D3.113/stubs.S  Tue Apr 30 22:18:21 2013 +0200
+++ b/platform/5D3.113/stubs.S  Sun May 05 23:09:11 2013 +0900
@@ -69,7 +69,7 @@
 NSTUB(0xFF9E9AF0 - RAM_OFFSET, GetMemoryAddressOfMemoryChunk)
 NSTUB(0xFF9E9ED8 - RAM_OFFSET, GetNumberOfChunks)
 NSTUB(0xFF9E9DF8 - RAM_OFFSET, GetSizeOfMemorySuite)
-
+NSTUB(0xFF9EB650 - RAM_OFFSET, dma_memcpy)
 NSTUB(0xff9f3900 - RAM_OFFSET, AcquireRecursiveLock) // AJ_KernelDry_KerRLock.c
 NSTUB(0xff9e88e8 - RAM_OFFSET, AllocateMemory)
 NSTUB(0xff44c8c4, CreateDialogBox)


1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #136 on: May 05, 2013, 04:19:26 PM »
So the only other thing to try will be an equivalent to memcpy for edmac? So something like copy memory suite?

Also there are a couple of different memcpy around that one, on 6D i found a faster one but the files were screwed up for some reason (some kind of data was copied tho). Maybe poke around there and try the others for fun. Maybe the parameters are different.

There is also HPcopy.

Code: [Select]
int (*HPCopyAsync) (unsigned char *dst, unsigned char *src, int length, void (*cbr)(unsigned int), int ctx) = (int (*) (unsigned char *dst, unsigned char *src, int length, void (*cbr)(unsigned int), int ctx))0xCB80;

What is allocateHPmemory?

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #137 on: May 05, 2013, 05:04:02 PM »
Enjoy with your family.
Let me confirm.
CPU
 -malloc & memcpy()
DMA
 -shoot_malloc & dma_memcpy()
EDMAC
 -CreateMemorySuite & (not found yet)

Correct?

And now I finding address of dma_memcpy, but I can't find it yet.

nearly :)

for CPU copy it is better to work on cached memory (address & ~0x40000000)
whereas for (E)DMAC its important to work on uncached memory (address | 0x40000000).
so it is not important if you use malloc, alloc_dma_memory or  shoot_malloc.

but whenever you work with cached/uncached you have to ensure that the caches for this areas are flushed.
so when using DMA on memories that you accessed with CPU in a cached way, you have to flush before and after CPU access.


some example code for EDMAC memcpy (didnt test it, just wrote down right now):
see wiki page Register Map for some details on EDMAC. copy size granularity for EDMAC is 4096, so sizes must be a mutiple of 4096.
in this example i chose to copy from a continuous memory block to a newly allocated memSuite buffer.
you can change it to set up two memSuites from memory blocks or whatever. its just an example.

Code: [Select]
/* pick some free (check using debug menu) EDMAC channels write: 0x00-0x06, 0x10-0x16, 0x20-0x21. read: 0x08-0x0D, 0x18-0x1D,0x28-0x2B */
dmaChannelRead = 0x19
dmaChannelWrite = 0x11

/* both channels get connected to this... lets call it service. it will just output the data it gets as input */
dmaConnection = 6

/* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
dmaFlags = 0x20001000


/* create a memory suite from a already existing (continuous) memory block with given size. */
struct memSuite *memSuiteSource = CreateMemorySuite(<addr>, <size>, 0);
/* allocate a memory suite that is of given size (potentially fragmented) */
struct memSuite *memSuiteDest = shoot_malloc_suite(<length>);

/* only read channel will emit a callback when reading from memory is done. write channels would just silently wrap */
PackMem_RegisterEDmacCompleteCBRForMemorySuite(dmaChannelRead, &complete_cbr, 0);

/* connect the selected channels to 6 so any data read from RAM is passed to write channel */
ConnectWriteEDmac(dmaChannelWrite, dmaConnection);
ConnectReadEDmac(dmaChannelRead, dmaConnection);

/* setup EDMAC driver to handle memory suite copy. check return codes for being zero (OK)! if !=0 then the suite size was not a multiple of 4096 */
err = PackMem_SetEDmacForMemorySuite(dmaChannelWrite, memSuiteDest , dmaFlags);
err = PackMem_SetEDmacForMemorySuite(dmaChannelRead, memSuiteSource , dmaFlags);

/* start transfer. no flags for write, 2 for read channels */
PackMem_StartEDmac(dmaChannelWrite, 0);
PackMem_StartEDmac(dmaChannelRead, 2);
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

minimimi

  • Freshman
  • **
  • Posts: 68
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #138 on: May 05, 2013, 06:19:10 PM »
see wiki page Register Map for some details on EDMAC. copy size granularity for EDMAC is 4096, so sizes must be a mutiple of 4096.

Thankyou for your kind explanation.
I think If we use DMA memory , we can't write single operation to write files.(can't aggregate FIO_WriteFile)
If we use another memorySuite instance, WriteFile operation is same as current codes.
It means not speed up .....


But I'm learned memory things, It's really helpful. Thankyou again.

Modified
Just understood your code.
Will try it.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #139 on: May 05, 2013, 08:25:48 PM »
you are welcome.
it would be cool to see performance results of those three methods.

e.g. let them copy 2 MiB of data and check how long it took.
for EDMAC it would be better to copy 20 MiB as i guess it is very fast.

how to get timer ticks:

static uint32_t tskmon_get_timer_reg()
{
    return *(uint32_t*)0xC0242014;
}
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

kgv5

  • Hero Member
  • *****
  • Posts: 643
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #140 on: May 05, 2013, 11:19:14 PM »
Hi

is it possible to change crop amount top/bottom in LV rec? There is such option but when I am changing crop size it returns to "default" number (it doesn't save entered values). It would be nice to set such a crop to get 544 lines (1280x544 or 1840x544) and 800 lines (1840x800). It will corespond with 2.35:1 aspect ratio so one doesn't have to make black bars in post and it will save some bitrate.

Thanks
www.pilotmovies.pl   5D Mark III, 6D, 550D

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #141 on: May 06, 2013, 12:29:13 AM »
The non presets don't seem to work.

g3ggo just added footer data, I have to merge it and rebuild, so that solves some problems.

sRaw trick didn't work for EOS-M :(... actually Sraw didn't work at all on EOSM, just dies when you take a pic. Maybe 650D will have better luck.

minimimi

  • Freshman
  • **
  • Posts: 68
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #142 on: May 06, 2013, 05:41:31 AM »
you are welcome.
it would be cool to see performance results of those three methods.

e.g. let them copy 2 MiB of data and check how long it took.
for EDMAC it would be better to copy 20 MiB as i guess it is very fast.

hmm, I'm now porting your pseude-code to lv_rec.c, but shoot_malloc_suite is allocating chunked memory using memSuite.
So I think , the code makes a faster memory copy , but it needs a lot of FIO_WriteFile . It means a slowing down to writing a file I guess.

So we need to find how to write a file with DMA, I think..


Or , my old school experience,.. When we write a file , make a 1Gig or 2gig file with padding 0 before writing a file.
The write () don't need to allocate  disk spaces (sector) realtime, so it's a little bit faster.
In my experience, the file created by dd on UNIX at that time. So we need to implement the function or , find fseek IMHO.

kgv5

  • Hero Member
  • *****
  • Posts: 643
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #143 on: May 06, 2013, 01:35:29 PM »
LV rec set to YUV recording creates single *.yuv file. 422toIMg accepts *.422 files.
How are you guys processing *.yuv files to use them in premiere etc ?
www.pilotmovies.pl   5D Mark III, 6D, 550D

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #144 on: May 06, 2013, 01:47:53 PM »
The non presets don't seem to work.

g3ggo just added footer data, I have to merge it and rebuild, so that solves some problems.

sRaw trick didn't work for EOS-M :(... actually Sraw didn't work at all on EOSM, just dies when you take a pic. Maybe 650D will have better luck.

Is it possible to add g3ggo work to 5D3 so I can compile it and tested?
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #145 on: May 06, 2013, 04:59:39 PM »
Main repo has it, just enable TCC and modules.

sicetime

  • Freshman
  • **
  • Posts: 53
  • 5D III
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #146 on: May 06, 2013, 05:16:19 PM »
I can't find it now but I read that card speed is not the issue so would this work with sd cards as well? I am having trouble with my slow non code oriented mind, following along with the YUV 422 along side the DNG threads. Are both available for the 5d3? How come I can not find the cam on the nightly builds? tragic lantern 2.0 doesn't have a list of compatible cameras, is TL available for the 5d3? 

Also, unrelated question, has anyone used komputerbay 1000x cards? I want to pick up a few because of the amazing advancements here, but now am thinking that it would be fruitless... thoughts?


AlexVakulov

  • New to the forum
  • *
  • Posts: 3
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #147 on: May 06, 2013, 05:35:04 PM »
Hello to everyone!
From 600D, quality of succesful shots is amasing, but about a half of all series is broken. Why?
https://docs.google.com/file/d/0B4Dxk-qc9EApcjZpem5mbE5FRVE/edit?usp=sharing

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #148 on: May 06, 2013, 06:01:09 PM »
Sync on 600D is not so good. It looks like its catching it in the middle of the frame. I'll do a test again now that cutting from one file is convenient.

Quote
tragic lantern 2.0 doesn't have a list of compatible cameras

2.0 is 600D
6D is 6D + EOSM (needs more fixed)... you could also put back the raw size for lvrec and compile 5d3 off of it. Haven't recently though.

Yea, just checked, have this problem too. It sucks because I got over 120 frames on 720P with fps override. Ugh... 150 frames now and sync goes away after like 24. Tried display state and all parts of evf state.

Rush

  • Member
  • ***
  • Posts: 159
  • 5D3, EOS M
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #149 on: May 06, 2013, 07:34:58 PM »
I can't find it now but I read that card speed is not the issue so would this work with sd cards as well?
Card speed is not issue for RAW 1-2 sec bursts or RAW timelapse. But if you aim to record RAW continuously (which I hope will be available) - you need super duper fast CF at least 1000x.
Greetings from Russia!