RAW Video Module raw_rec [Compiling/Installation]

[Vegandelight]

I would rename it to 550D.sym, 6D.sym and so on. It will help when we'll use the same zip for more cameras.

Can it be named whatever we want in the modules dir, does it matter?

[1%]

Thats not bad... just really annoying to keep renaming MAGIC.SYM, if it used 6D.SYM then sure, otherwise same issue.

[a1ex]

Didn't notice the annoyance - "make install" creates the entire directory structure on the card, no need to rename manually.

[Vegandelight]

Finally got it running. Why doesnt make install copy raw_rev.o? maybe thats not even needed?

[fabiogalvanifurtado]

Hi, guys!

I got a 5D MKIII, but with the new 1.2.1 firmware. Magic Lantern only functions on the 1.1.3, that's it? Anyone could please tell me where to download the 1.1.3 version??

Best regards, this job you're doing is really amazing

[RenatoPhoto]

Hi, guys!

I got a 5D MKIII, but with the new 1.2.1 firmware. Magic Lantern only functions on the 1.1.3, that's it? Anyone could please tell me where to download the 1.1.3 version??

Best regards, this job you're doing is really amazing

Where to get Canon Firmware 1.1.3 for sure:
HERE: http://pel.hu/eoscard/

Also look at this:

How to install Magic Lantern on 5D Mark III
HERE: http://www.magiclantern.fm/forum/index.php?topic=2602.0

[stevefal]

Any idea why building is failing for me?

Makefile.user: http://popspring.com/mldrop/Makefile.txt

Output: http://popspring.com/mldrop/mlfail.txt

[escho]

Any idea why building is failing for me?

Makefile.user: http://popspring.com/mldrop/Makefile.txt

Output: http://popspring.com/mldrop/mlfail.txt

try to disable picoc

Edgar

[wolf]

Try to disable picoc.

Edit: didn't refresh my site

[stevefal]

try to disable picoc.

Thanks, but it still fails. Other ideas?

[created another thread at http://www.magiclantern.fm/forum/index.php?topic=5537.0]

[danap]

Obviously, current CF cards writing speeds stretches the possibilities of the raw recording on Canon  cameras. Therefore my question and possible suggestion for a new user managed bit depth:
Could the raw_rec module offer options to strip the 14bits depth down to 12, 10 and even 8 bits. The raw file would be much thinner and writable to a large number of CF cards and cameras?

[Colemar]

Could the raw_rec module offer options to strip the 14bits depth down to 12, 10 and even 8 bits.

I think all the raw-rec module is doing is saving a datastream the sensor is outputting anyway.  There is no processing happening which is why the camera will not overheat and the sensor will not be damaged.  Also why post workflow has not been solidified.  So no, it would not be possible to convert the 14bit datastream into a lower bitrate without processing in-camera, which defeats the purpose of this modification.

[vicnaum]

which defeats the purpose of this modification.

Well, I'd rather say that if possible - we should do something with uncompressed raw - cause 25 minutes at 128 GB card is just HUGE and unreal. Should be like REDCODE (20-40MB/s) or smth, cause 90MB/s is really overwhelm. Lossy or lossless - but not that huge.

The other question is - if it's feasible to process the full uncompressed RAW stream with our DIGIC, cause it's still a question to me if it is powerful enough to handle such huge amounts of data in realtime.

[Colemar]

Well, I'd rather say that if possible - we should do something with uncompressed raw - cause 25 minutes at 128 GB card is just HUGE and unreal.

The benefit of what Magic Lantern is doing here is pristine data straight from the sensor.  Compression ruins that and makes it just Lantern.  Voluminous data is a consequence of shooting raw, no getting around that.

The other question is - if it's feasible to process the full uncompressed RAW stream with our DIGIC,

the DIGIC processor is not processing the raw datastream.  The beauty of all of this is ML devs were able to bypass all of Canon's image processing and deliver a stream of image data that is 14bits deep and glorious.

[kgv5]

Well, I'd rather say that if possible - we should do something with uncompressed raw - cause 25 minutes at 128 GB card is just HUGE and unreal. Should be like REDCODE (20-40MB/s) or smth, cause 90MB/s is really overwhelm. Lossy or lossless - but not that huge.

The other question is - if it's feasible to process the full uncompressed RAW stream with our DIGIC, cause it's still a question to me if it is powerful enough to handle such huge amounts of data in realtime.

To save some space you can always decrease a resolution a little bit.
I have a 6D and unfortunatelly with SD only usable res (say ~400frames without any skip) for now is 1504x640. It is 2,35:1 aspect ratio.
Such a resolution has some advantages - one frame is about 1,64MB, bitrate ~37MB/s, it's still 14bit. After resizing in AE by ~130% resolution obviously looks worse than 1080p raw but the same as 1080p h264. There is much more details and dynamic range. I wonder how it compares with say prores 422HQ or even prores 444 (similar bitrate) in terms of image quality. With 64GB card you can have about 26 minutes of footage.

[vicnaum]

The benefit of what Magic Lantern is doing here is pristine data straight from the sensor.  Compression ruins that and makes it just Lantern.  Voluminous data is a consequence of shooting raw, no getting around that.
the DIGIC processor is not processing the raw datastream.  The beauty of all of this is ML devs were able to bypass all of Canon's image processing and deliver a stream of image data that is 14bits deep and glorious.

Yes, I know that it just flushes the buffers to card.

But even professional cameras (like RED) have lossy compressed RAW and no one cries about it. And I would prefer 1920x1080 lossy comressed, than 960x480 uncompressed.

The possibility of processing is still questionable.

[Colemar]

apologies I misunderstood.  I see now from the nametag you are shooting on a 600D, I assumed 5Dmk2 or 3.  Still, the fact that a Canon DSLR can compare to a camera from RED because of the efforts of volunteers is astonishing.  The fact they got it to shoot a picture with more color resolution than an Alexa is ridiculous.  I see your point on in-camera processing, just don't agree it's necessary.

[vicnaum]

apologies I misunderstood.  I see now from the nametag you are shooting on a 600D, I assumed 5Dmk2 or 3.  Still, the fact that a Canon DSLR can compare to a camera from RED because of the efforts of volunteers is astonishing.  The fact they got it to shoot a picture with more color resolution than an Alexa is ridiculous.  I see your point on in-camera processing, just don't agree it's necessary.

Right, on 5Dmk2-3 it's surely not so necessary (as long as you have tons of fast 128GB cards or guys invent CF->SSD converter). And surely great as a test. And an awesome breakthrough.

But now it looks more like a "proof-of-concept" to me. Cause for production use - when you need 2-4-6 hours of footage per day - you'll need to buy 10-15-20 CF cards for these bitrates, and assuming each one costs 300-400$ - you can get a used RED ONE for these money :-)

[dude]

Would it be possible to delete the Raw Files on Cam? Would be lot easier because you would have more space(deleting unusable takes) which is needed with raw.

Thanks

[Jackie]

Hello guys I have a litle problem when i want to compile normal nightly build everything is ok but when I enable TCC modules and memcheck in the makefile it end up like this and I think I have all the software updated to its newest versions any idea where is the problem ? I´m using gcc-arm-none-eabi-4_7-2013q1 and cygwin thanks in advance for any advice

Code: [Select]

Jackie@JACKIE-NTB ~
$ cd c:/cygwin/home/jackie/magic-lantern/platform/5D3.113

Jackie@JACKIE-NTB /home/jackie/magic-lantern/platform/5D3.113
$ make clean && make
[ CLEAN    ]   clean
[ CLEAN    ]   clean
[ CPP      ]   magiclantern.lds
~/gcc-arm-none-eabi-4_7-2013q1/bin/arm-none-eabi-gcc-4.7.3.exe    -c -o ../../src/entry.o ../../src/entry.S
[ AS       ]   ../../platform/5D3.113/stubs.o
echo 'const char build_version[] = "v2.3.NEXT.2013May22.5D3113";' > ../../platform/5D3.113/version.c
echo 'const char build_id[] = "'`hg id`'";' >> ../../platform/5D3.113/version.c
echo 'const char build_date[] ="'`date -u "+%Y-%m-%d %H:%M:%S"`'";' >> ../../platform/5D3.113/version.c
echo 'const char build_user[] = "'`whoami`@`hostname`'";'  >> ../../platform/5D3.113/version.c
[ CC       ]   version.o
[ CC       ]   boot-hack.o
[ CC       ]   ico.o
[ CC       ]   edmac.o
[ CC       ]   menu.o
[ CC       ]   debug.o
../../src/debug.c:4599:12: warning: 'zoom_in_pressed' defined but not used [-Wunused-variable]
[ CC       ]   cfn.o
[ CC       ]   gui.o
[ CC       ]   picstyle.o
[ CC       ]   exmem.o
[ CC       ]   bmp.o
[ CC       ]   font-dyn.o
[ CC       ]   config.o
[ CC       ]   stdio.o
[ CC       ]   bitrate-5d3.o
[ CC       ]   tweaks.o
[ CC       ]   lens.o
[ CC       ]   property.o
[ CC       ]   propvalues.o
[ CC       ]   gui-common.o
[ CC       ]   chdk-gui_draw.o
[ CC       ]   movtweaks.o
[ CC       ]   menuhelp.o
[ MENU IDX ]   menuindexentries.h
No menuindex.txt not running "python2 menuindex.py"
[ CC       ]   menuindex.o
[ CC       ]   focus.o
[ CC       ]   notify_box.o
[ CC       ]   dialog_test.o
[ CC       ]   vram.o
[ CC       ]   aj_port.o
[ CC       ]   fps-engio.o
[ CC       ]   shoot.o
[ CC       ]   hdr.o
[ CC       ]   lv-img-engio.o
[ CC       ]   state-object.o
[ CC       ]   tasks.o
[ CC       ]   beep.o
[ CC       ]   vsync-lite.o
[ CC       ]   tskmon.o
[ CC       ]   battery.o
[ CC       ]   imgconv.o
../../src/imgconv.c: In function 'yuv411_to_rgb':
../../src/imgconv.c:189:16: warning: unused variable 'c' [-Wunused-variable]
[ CC       ]   histogram.o
[ CC       ]   falsecolor.o
[ CC       ]   audio-ak.o
[ CC       ]   zebra.o
[ CC       ]   video_hacks.o
[ CC       ]   fio-5d3.o
[ CC       ]   misc.o
[ CC       ]   ph_info_disp.o
[ CC       ]   flexinfo.o
[ CC       ]   afma.o
[ CC       ]   raw.o
[ CC       ]   chdk-dng.o
[ CC       ]   edmac-memcpy.o
../../src/edmac-memcpy.c: In function 'edmac_memset':
../../src/edmac-memcpy.c:145:5: warning: passing argument 1 of 'memset' makes pointer from integer without a cast [enabled by default]
In file included from ../../src/edmac-memcpy.c:2:0:
../../src/dryos.h:56:15: note: expected 'void *' but argument is of type 'long unsigned int'
../../src/edmac-memcpy.c:148:5: warning: passing argument 1 of 'edmac_copy_rectangle_adv_start' makes pointer from integer without a cast [enabled by default]
../../src/edmac-memcpy.c:46:7: note: expected 'void *' but argument is of type 'long unsigned int'
../../src/edmac-memcpy.c:148:5: warning: passing argument 2 of 'edmac_copy_rectangle_adv_start' makes pointer from integer without a cast [enabled by default]
../../src/edmac-memcpy.c:46:7: note: expected 'void *' but argument is of type 'long unsigned int'
../../src/edmac-memcpy.c:157:9: warning: passing argument 1 of 'memset' makes pointer from integer without a cast [enabled by default]
In file included from ../../src/edmac-memcpy.c:2:0:
../../src/dryos.h:56:15: note: expected 'void *' but argument is of type 'long unsigned int'
[ CC       ]   asm.o
../../src/asm.c: In function 'seems_to_be_string':
../../src/asm.c:22:15: warning: incompatible implicit declaration of built-in function 'strlen' [enabled by default]
../../src/asm.c: In function 'asm_guess_func_name_from_string':
../../src/asm.c:36:26: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
[ CC       ]   memcheck.o
../../src/memcheck.c: In function 'memcheck_add':
../../src/memcheck.c:98:46: warning: assignment discards 'const' qualifier from pointer target type [enabled by default]
../../src/memcheck.c: In function 'memcheck_malloc':
../../src/memcheck.c:141:29: warning: incompatible implicit declaration of built-in function 'malloc' [enabled by default]
../../src/memcheck.c:149:26: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
../../src/memcheck.c:159:5: warning: return makes pointer from integer without a cast [enabled by default]
../../src/memcheck.c: In function 'memcheck_free':
../../src/memcheck.c:193:22: warning: pointer targets in initialization differ in signedness [-Wpointer-sign]
../../src/memcheck.c:207:9: warning: incompatible implicit declaration of built-in function 'free' [enabled by default]
../../src/memcheck.c:207:9: warning: passing argument 1 of 'free' makes pointer from integer without a cast [enabled by default]
../../src/memcheck.c:207:9: note: expected 'void *' but argument is of type 'unsigned int'
../../src/memcheck.c: In function 'memcheck_main':
../../src/memcheck.c:231:25: warning: initialization makes integer from pointer without a cast [enabled by default]
[ CC       ]   console.o
[ CC       ]   pico.o
In file included from ../../picoc/pico.c:1:0:
../../picoc/platform.h:68:0: warning: "malloc" redefined [enabled by default]
In file included from ../../picoc/../src/dryos.h:633:0,
                 from ../../picoc/platform.h:33,
                 from ../../picoc/pico.c:1:
../../picoc/../src/memcheck.h:10:0: note: this is the location of the previous definition
In file included from ../../picoc/pico.c:1:0:
../../picoc/platform.h:71:0: warning: "free" redefined [enabled by default]
In file included from ../../picoc/../src/dryos.h:633:0,
                 from ../../picoc/platform.h:33,
                 from ../../picoc/pico.c:1:
../../picoc/../src/memcheck.h:11:0: note: this is the location of the previous definition
In file included from ../../picoc/pico.c:13:0:
../../picoc/pico.c: In function 'PlatformExit':
../../picoc/platform_ml.c:62:1: warning: 'noreturn' function does return [enabled by default]
[ CC       ]   picoc.o
[ CC       ]   chdk-gui_script.o
[ CC       ]   tcc-glue.o
../../src/tcc-glue.c: In function 'tcc_compile_and_run':
../../src/tcc-glue.c:34:9: warning: incompatible implicit declaration of built-in function 'printf' [enabled by default]
../../src/tcc-glue.c:39:11: warning: 'main' is usually a function [-Wmain]
../../src/tcc-glue.c:66:9: warning: incompatible implicit declaration of built-in function 'printf' [enabled by default]
../../src/tcc-glue.c: In function 'open':
../../src/tcc-glue.c:146:5: warning: passing argument 2 of 'FIO_GetFileSize' from incompatible pointer type [enabled by default]
In file included from ../../src/tcc-glue.c:9:0:
../../src/dryos.h:532:1: note: expected 'uint32_t *' but argument is of type 'unsigned int *'
../../src/tcc-glue.c: In function 'read':
../../src/tcc-glue.c:179:5: warning: passing argument 2 of 'memcpy' makes pointer from integer without a cast [enabled by default]
In file included from ../../src/tcc-glue.c:9:0:
../../src/dryos.h:549:1: note: expected 'const void *' but argument is of type 'long unsigned int'
../../src/tcc-glue.c: In function 'close':
../../src/tcc-glue.c:187:5: warning: passing argument 1 of 'free_dma_memory' makes pointer from integer without a cast [enabled by default]
In file included from ../../src/tcc-glue.c:9:0:
../../src/dryos.h:558:1: note: expected 'const void *' but argument is of type 'int'
../../src/tcc-glue.c: In function 'fputc':
../../src/tcc-glue.c:240:1: warning: number of arguments doesn't match built-in prototype [enabled by default]
../../src/tcc-glue.c: In function 'tcc_compile_and_run':
../../src/tcc-glue.c:78:8: warning: 's' may be used uninitialized in this function [-Wuninitialized]
[ CC       ]   tcc-gui.o
[ CC       ]   script_lib.o
make -C ../../tcc
make[1]: Entering directory `/home/jackie/magic-lantern/tcc'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/jackie/magic-lantern/tcc'
[ CC       ]   module.o
[ AR       ]   strrchr.o
[ AR       ]   dietlibc.a
[ AR       ]   lib_a-setjmp.o
[ AR       ]   newlib-libc.a
[ LD       ]   magiclantern
tcc-gui.o: In function `handle_picoc_keys':
tcc-gui.c:(.text+0x518): multiple definition of `handle_picoc_keys'
picoc.o:picoc.c:(.text+0x660): first defined here
tcc-gui.o: In function `script_setup_param':
tcc-gui.c:(.text+0x5c8): multiple definition of `script_setup_param'
picoc.o:picoc.c:(.text+0x72c): first defined here
tcc-gui.o: In function `script_reset_params':
tcc-gui.c:(.text+0x620): multiple definition of `script_reset_params'
picoc.o:picoc.c:(.text+0x784): first defined here
tcc-gui.o: In function `script_open_submenu':
tcc-gui.c:(.text+0x9a8): multiple definition of `script_open_submenu'
picoc.o:picoc.c:(.text+0xb0c): first defined here
tcc-gui.o: In function `script_setup_title':
tcc-gui.c:(.text+0xa28): multiple definition of `script_setup_title'
picoc.o:picoc.c:(.text+0xb8c): first defined here
tcc-gui.o: In function `script_key_dequeue':
tcc-gui.c:(.text+0xa5c): multiple definition of `script_key_dequeue'
picoc.o:picoc.c:(.text+0xbc0): first defined here
tcc-gui.o: In function `script_key_enqueue':
tcc-gui.c:(.text+0xa9c): multiple definition of `script_key_enqueue'
picoc.o:picoc.c:(.text+0xc00): first defined here
tcc-gui.o:(.bss+0x4): multiple definition of `script_stop_requested'
picoc.o:(.bss+0x4): first defined here
dietlibc.a(vsscanf.o): In function `vsscanf':
vsscanf.c:(.text+0x5c): warning: warning: the scanf functions add several kilobytes of bloat.
dietlibc.a(sprintf.o): In function `sprintf':
sprintf.c:(.text+0x14): warning: warning: Avoid *sprintf; use *snprintf. It is more secure.
collect2.exe: error: ld returned 1 exit status
../../Makefile.inc:562: recipe for target `magiclantern' failed
make: *** [magiclantern] Error 1


[RenatoPhoto]

PicocC has to be disabled in Makefile.user

CONFIG_PICOC = n

Also make clean before you make

[Jackie]

PicocC has to be disabled in Makefile.user

CONFIG_PICOC = n

Also make clean before you make

Thank you it works after disabling the picoc lovely

[Mayo]

Noob question: I can compile magic lantern just fine, but what's the correct procedure to make the modules?

When I try to make raw_rec.o from /modules/raw_rec/ I get:

Code: [Select]

[ CC       ]   raw_rec.o
raw_rec.c:20:20: fatal error: module.h: No such file or directory
compilation terminated.
make: *** [raw_rec.o] Error 1
Thanks.

[RenatoPhoto]

Noob question: I can compile magic lantern just fine, but what's the correct procedure to make the modules?

When I try to make raw_rec.o from /modules/raw_rec/ I get:

Code: [Select]

[ CC       ]   raw_rec.o
raw_rec.c:20:20: fatal error: module.h: No such file or directory
compilation terminated.
make: *** [raw_rec.o] Error 1

Are you compiling in windows cygwin?

[Mayo]

Are you compiling in windows cygwin?

Yes.

Edit: Ah, nevermind, it was right before my eyes 

Code: [Select]

* - enable modules in Makefile.user (CONFIG_MODULES = y, CONFIG_TCC = y, CONFIG_PICOC = n, CONFIG_CONSOLE = y)
Edit 2: Ah well, same problem. Any help is much appreciated.

Edit 3: replaced TOP_DIR=$(PWD)/../.. with TOP_DIR=../.. in the makefile : I got raw_rec.mo to compile \o/
Now, on to raw2dng...