Messages

I read codes.
The "File Browser" title string is not a just only for title string. It's also using identify the menu groups in the codes.
So I think , It can't do it . but we need a lot of codes to make it, I guess.

[Wrote reply but alex is making a code from scratch,,,,, ignore this.........]

Wrote reply but alex is making a code from scratch,,,,, ignore this.........

i wanted the split file to be really "split" just as if they were splitted by common split and merge utilities.
this means the payload is split over several files and at the end we have one footer that contains information for all frames.

simply concatenate the files to have a big movie file.

I already read and understand your codes.
But , concatenate time is really long.
So I think if the all of files has footer info, we don't need time to concatenate files and tempolary disk spaces.
Also , I'm tried to concat by cat command, I don't know why, but fopen failed (3.6Gig file on 32bit linux).
(?? cat command broken files???)

the reason for a footer is simple: in YUV mode you can process the files with the existing tools, you dont have to cut the footer away.
if i had added a header, it wont be that simple - you have to remove the header before you can use old tools.

when the tools are finished, it is as simple as specifying the first file M000000.RAW and it will autodetect that there is a .R00, R01 etc
and will virtually address them as one file.

If we can cut footer , we have no problem to use old tools when each files has footer info.

It's my thinking.

Just added a RAW to DNG converter: https://bitbucket.org/hudson/magic-lantern/commits/5bc3488c7a78

It's just a small command-line tool. Feel free to build a nice GUI on top of it.

For some reason, I couldn't compile it in 64-bit mode, so you may need the 32-bit compatibility libraries. Fixes are welcome

With a little luck, this program may work under Windows too.

Alex, Are you forget to add this ?(when the file size reached 2Gig , then rotation the file)
I'm tried to concatinate by cat command, but something wrong. (4gig of file not supported??)
And cat need a lot of time and disk spaces. So I think each file must have footer info.

Code Select Expand

diff -r 5bc3488c7a78 modules/lv_rec/lv_rec.c
--- a/modules/lv_rec/lv_rec.c   Sat May 11 14:35:52 2013 +0300
+++ b/modules/lv_rec/lv_rec.c   Sun May 12 21:51:27 2013 +0900
@@ -649,6 +649,8 @@
                     {
                         yPos++;
                         bmp_printf( FONT(FONT_MED, COLOR_WHITE, COLOR_BLACK), 30, 20 * yPos++, "Creating next file");
+                       save_data.frameCount = 1;
+                       lv_rec_save_footer(save_data.handle, &save_data);
                         FIO_CloseFile(save_data.handle);
                         save_data.handle = NULL;
                         lv_rec_update_suffix(&save_data);


I keep getting seg faults from the size, no matter what size I set:

Single file from 600D, W/H set to what dng burst reports

http://www.qfpost.com/file/d?g=4ED0ZAaJN

Single file from 6D http://www.qfpost.com/file/d?g=v8T0Ryl5m

I tried your 2files. Both are ...

Code Select Expand

$ ./raw2dng I0000000.RAW
Error: This ain't a lv_rec RAW file

So I guess you may upgrade lv_rec.mo first.
Alex added file footer to RAW in current code.
YOur file has no footer I guess.

Solved compilation error. Because ,CreateMemorySuite is imported to exmem.h

Code Select Expand

diff -r 22fd8107540b modules/lv_rec/lv_rec.c
--- a/modules/lv_rec/lv_rec.c   Thu May 09 00:10:56 2013 +0200
+++ b/modules/lv_rec/lv_rec.c   Thu May 09 11:32:32 2013 +0900
@@ -9,7 +9,6 @@

unsigned int exmem_clear(struct memSuite * hSuite, char fill);
unsigned int exmem_save_buffer(struct memSuite * hSuite, char *file);
-struct memSuite *CreateMemorySuite(unsigned int address, unsigned int size, unsigned int flags);

/* file footer data */
typedef struct
@@ -593,7 +592,7 @@
         data.dmaFlags = 0;

         /* create a memory suite that consists of lv_save_raw raw buffer */
-        data.memCopySuite = CreateMemorySuite(shamem_read(RAW_LV_EDMAC), save_data.frameSize, 0);
+        data.memCopySuite = CreateMemorySuite((void *)shamem_read(RAW_LV_EDMAC), save_data.frameSize, 0);
         PackMem_RegisterEDmacCompleteCBRForMemorySuite(data.dmaCopyChannel, &complete_cbr, 0);
         PackMem_RegisterEDmacPopCBRForMemorySuite(data.dmaCopyChannel, &pop_cbr, 0);
     }


Thanks for the time you take to help with this issue.  I really appreciated and I assume that this will have to be take care at some point so that other people using windows compiler will have to resolve in order to get some of these TCC module options.  So I hope that the time spent fixing this will help someone else too.

hmm, off topic sorry.
I'm worried about that people didn't understand the risk of this cutting-edge codes  who is this skill level such as you.
Then, maybe peoples will send canon customer center when their camera is bricked.
Canon need to pay more recovery cost , and canon will start to attack our community IMHO.

So maybe, I think the best thing is we do not support install support for cutting-edge codes.

So I think you can not install yourself , I recommend you to wait a release version.

In this case there is no "arm-none-eabi-ar" program under the gcc directory, I only found  the "arm-none-eabi-ar.exe"

1% means a
also add this line after CC=
AR=../../arm-toolchain473/bin/arm-none-eabi-ar.exe

Do you understand what meanning of "../" ?
So can you make a relative path from magic-lantern/tcc directory?

Then you may put this line after "-include $(TOP)/../Makefile.user" in tcc/Makefile

CC=../../path/of/your/arm-noneeabi-gcc.exe

I think you have no problem to use yagarto maybe,....

BTW, I'm also using windows, but we had a sometime problems on windows environment.
So I recommend you to use vmware player and build on Linux.
Ofcource I'm using linux with vmware player.

make[2]: ../../../arm-toolchain472/bin/arm-none-eabi-gcc-4.7.2.exe: No se encontró el programa

Is this means "program not found" . right?
Then you may try to check

Code Select Expand


# cd magic-lantern
# ../../../arm-toolchain472/bin/arm-none-eabi-gcc-4.7.2.exe


Show us a output of this .

[memory copy test results(without memSuite)]

memory copy test results(without memSuite)
I'm checked it but chunk copy is fast . It's misteirous.
and, I don't know uint32_t timer counter is how many cycled.So I don't know the result is correct or not.


The code is here, and these codes located at  run_test()

Code Select Expand

    void *m1 = malloc(20*1000*1000);
    void *m2 = malloc(20*1000*1000);
    void *sm1 = shoot_malloc(20*1000*1000);
    void *sm2 = shoot_malloc(20*1000*1000);
    void *ss1 = shoot_malloc_suite(20*1000*1000);
    void *ss2 = shoot_malloc_suite(20*1000*1000);

    uint32_t t1 = tskmon_get_timer_reg();
    uint32_t t2,tmp;
    int i,off;

    console_printf("malloc-ed 20M memcpy\n");
    memcpy(m1,m2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc-ed 20M memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            memcpy(m1+off,m2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc-ed 20M dma_memcpy\n");
    dma_memcpy(m1,m2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc-ed 20M dma_memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            dma_memcpy(m1+off,m2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc-ed 20M memcpy\n");
    memcpy(sm1,sm2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc-ed 20M memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            memcpy(sm1+off,sm2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc-ed 20M dma_memcpy\n");
    dma_memcpy(sm1,sm2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc-ed 20M dma_memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            dma_memcpy(sm1+off,sm2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc to shoot_malloc-ed 20M memcpy\n");
    memcpy(sm1,m2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc shoot_malloc-ed 20M memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            memcpy(sm1+off,m2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc to malloc 20M memcpy\n");
    memcpy(m1,sm2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc to malloc 20M memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            memcpy(m1+off,sm2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);





    t1 = tskmon_get_timer_reg();
    console_printf("malloc to shoot_malloc-ed 20M dma_memcpy\n");
    dma_memcpy(sm1,m2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc shoot_malloc-ed 20M dma_memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            dma_memcpy(sm1+off,m2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc to malloc 20M dma_memcpy\n");
    dma_memcpy(m1,sm2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc to malloc 20M dma_memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            dma_memcpy(m1+off,sm2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);




And result is:  (printf has no %u so I used %d(signed))

Code Select Expand

malloc-ed 20M memcpy
-485806
malloc-ed 20M memcpy 2m each
-71147
malloc-ed 20M dma_memcpy
-272364
malloc-ed 20M dma_memcpy 2m each
-524067
shoot_malloc-ed 20M memcpy
-484381
shoot_malloc-ed 20M memcpy 2m each
-81456
shoot_malloc-ed 20M dma_memcpy
-768859
shoot_malloc-ed 20M dma_memcpy 2m each
-527604
malloc to shoot_malloc-ed 20M memcpy
-561515
malloc shoot_malloc-ed 20M memcpy 2m each
-76210
shoot_malloc to malloc 20M memcpy
-494179
shoot_malloc to malloc 20M memcpy 2m each
-77837
malloc to shoot_malloc-ed 20M dma_memcpy
-266705
malloc shoot_malloc-ed 20M dma_memcpy 2m each
-513931
shoot_malloc to malloc 20M dma_memcpy
-268545
shoot_malloc to malloc 20M dma_memcpy 2m each
-537939


@RenatoPhoto

The key point is here.

Code Select Expand

make[2]: ../../../arm-toolchain462/bin/arm-none-eabi-gcc-4.6.2.exe: No se encontró el programa

I can't read it.
Also , we are already moved to gcc-4.6.2 to gcc-4.7
Upgrade first , and then check your gcc path.

@RenatoPhoto
I'm not English native speaker. So if you get a bad feeling from my message, please ignore.

Anyway, The answer of installing questions are including source codes. (I'm not asking anyone, but the source codes are said to me everything)
1% is answerd 1st message, but you don't read Makefile.inc and Makefile(or do not find tcc and module things in souce codes), so you can't understand it.
So I wrote the message.

@RenatoPhoto

I think you may not use developers time for your "installing" support.
If you want to use this function, you may learn development things.
Because, I think that it's better to spent developers time for actual development.
It's faster to get release version of this function when we don't use developers' time.

you are welcome.
it would be cool to see performance results of those three methods.

e.g. let them copy 2 MiB of data and check how long it took.
for EDMAC it would be better to copy 20 MiB as i guess it is very fast.

hmm, I'm now porting your pseude-code to lv_rec.c, but shoot_malloc_suite is allocating chunked memory using memSuite.
So I think , the code makes a faster memory copy , but it needs a lot of FIO_WriteFile . It means a slowing down to writing a file I guess.

So we need to find how to write a file with DMA, I think..


Or , my old school experience,.. When we write a file , make a 1Gig or 2gig file with padding 0 before writing a file.
The write () don't need to allocate  disk spaces (sector) realtime, so it's a little bit faster.
In my experience, the file created by dd on UNIX at that time. So we need to implement the function or , find fseek IMHO.

[Modified]

see wiki page Register Map for some details on EDMAC. copy size granularity for EDMAC is 4096, so sizes must be a mutiple of 4096.

Thankyou for your kind explanation.
I think If we use DMA memory , we can't write single operation to write files.(can't aggregate FIO_WriteFile)
If we use another memorySuite instance, WriteFile operation is same as current codes.
It means not speed up .....

But I'm learned memory things, It's really helpful. Thankyou again.

Modified
Just understood your code.
Will try it.

It should be
NSTUB(0xFF9EB650 - RAM_OFFSET, AJ_HP_Copy_n_DMA_channel_n_BLTDMA)

Thanks . But not enough speed..
Result is more slow with g3ggo's original method.

Test code is here.

Code Select Expand


diff -r 99bd629f7c57 modules/lv_rec/lv_rec.c
--- a/modules/lv_rec/lv_rec.c   Tue Apr 30 22:18:21 2013 +0200
+++ b/modules/lv_rec/lv_rec.c   Sun May 05 23:09:11 2013 +0900
@@ -416,8 +417,12 @@

         if(!skip_saving)
         {
-            FIO_WriteFile(save_file, UNCACHEABLE(&save_data->chunkData.chunkAddress[save_data->chunkData.chunkOffset]), avail);
+            dma_memcpy(shmem+written,UNCACHEABLE(&save_data->chunkData.chunkAddress[save_data->chunkData.chunkOffset]),avail);
             written += avail;
+            if(written == length)
+           {
+                FIO_WriteFile(save_file, shmem , length);
+            }
+
         }
diff -r 99bd629f7c57 platform/5D3.113/stubs.S
--- a/platform/5D3.113/stubs.S  Tue Apr 30 22:18:21 2013 +0200
+++ b/platform/5D3.113/stubs.S  Sun May 05 23:09:11 2013 +0900
@@ -69,7 +69,7 @@
NSTUB(0xFF9E9AF0 - RAM_OFFSET, GetMemoryAddressOfMemoryChunk)
NSTUB(0xFF9E9ED8 - RAM_OFFSET, GetNumberOfChunks)
NSTUB(0xFF9E9DF8 - RAM_OFFSET, GetSizeOfMemorySuite)
-
+NSTUB(0xFF9EB650 - RAM_OFFSET, dma_memcpy)
NSTUB(0xff9f3900 - RAM_OFFSET, AcquireRecursiveLock) // AJ_KernelDry_KerRLock.c
NSTUB(0xff9e88e8 - RAM_OFFSET, AllocateMemory)
NSTUB(0xff44c8c4, CreateDialogBox)



(not very active this weekend due to family events)

Enjoy with your family.

there are 3 possibilits for memcpy:
- CPU: Load and Store words using more or less fast opcodes (LDMIA, STMIA with many registers)
- DMA: simple DMA controller
- EDMAC: special SDRAM DMA controller

Let me confirm.
CPU
-malloc & memcpy()
DMA
-shoot_malloc & dma_memcpy()
EDMAC
-CreateMemorySuite & (not found yet)

Correct?

And now I finding address of dma_memcpy, but I can't find it yet.

First, I would like to say solved one importand bug in new_name() in my script.
Previous script cant named by Alex's method1.

Running on all bins now... for the ram segment it would be good if script jumped back to rom and got the name from there.. ramsub+ramoffset = sub in main rom. Naming strings might not be in the copied ram portion.

Let me confirm , that is we need to copy names from ROM to RAM, right?
Because some funcs are copyed from ROM to RAM. Is this a correct understanding?

I see thanks. I will try it when I going back to home.
BTW, Alex is already suggested "memcpy is slow" , So I don't know we can get good result .
It's my hobby.

dma memcpy didnt' work? shoot_malloc(0) should be the big memory

Anybody didn't find it. and I did a actual test like this

shoot_malloc(4M)
shoot_malloc(8M)
shoot_malloc(16M)

and check these returns. Then , I can find only 8M to use from ML.

[Modified]

I'm tried to aggregate FIO_WriteFile , like this ..
Because, I think contenious sequential access is most faster when we write file to filesystem.

Code Select Expand

@@ -416,8 +417,12 @@

         if(!skip_saving)
         {
-            FIO_WriteFile(save_file, UNCACHEABLE(&save_data->chunkData.chunkAddress[save_data->chunkData.chunkOffset]), avail);
+            memcpy(shmem+written,UNCACHEABLE(&save_data->chunkData.chunkAddress[save_data->chunkData.chunkOffset]),avail);
             written += avail;
+            if(written == length)
+            {
+                FIO_WriteFile(save_file, shmem , length);
+            }
+
         }

         length -= avail;
@@ -451,12 +456,20 @@
     strcpy(&data->fileSuffix[1], tmp);
}

void lv_rec_start()
{
     int yPos = 3;
     lv_rec_data_t data;
     lv_rec_save_data_t save_data;

+    shmem = shoot_malloc(8000000);
+    if(shmem == NULL)
+    {
+        NotifyBox(3000,"8M shoot memory allocate failed");
+        msleep(3000);
+        return;
+    }
+
     /* set all values to zero */
     memset(&data, 0x00, sizeof(lv_rec_data_t));
     memset(&save_data, 0x00, sizeof(lv_rec_save_data_t));



But memcpy from DMA mem to shoot mem is really slow. Let me know more faster memcpy method, If you know.
Also , shoot mem is only allocated about 8MB. where located more fast and bigger memory?

Modified
5D3 has no dma_memcpy

[Modified]

How to calculate saved .yuv fps?

Modified
I'm sorry , I had a mistaken to write another board....

On 600D? My zoom is 1/2 height on 6D. I only made it work there so far.. havn't put my UHS card in to see how well the burst bursts.

On my 5D3 yey!
yes 3592x1320 is 1/2 height. But the picture quality is not so bad , So I think it's a simuler function with 600D's x3 zoom. I mean I guess this is recording pixcel ratio is 1:1 maybe.
I have no idea yet , but I think this is one of hint for clalyfy this function IMHO.

Yeah but when it does work, it's just noise.
https://dl.dropboxusercontent.com/u/34113196/Camera%20stuff/more/78460003.DNG

hmm, My DNG has no noise....

I'm tried this function, then I found interesting thing.
You may push Zoom button and take a silent pic.
We can get 3592x1320 DNG.