12-bit (and 10-bit) RAW video development discussion

[dmilligan]

I only tried with all 3, if I get some more time I'll try excluding them.

For anyone interested in doing further testing here's what I did (on top of current raw_video_10bit_12bit). To check what a1ex asked about above simply change the two #ifdef CONFIG_EOSM lines in the edmac_raw_slurp() function to #if 0 (try each one separately and both)

Code: [Select]

diff --git a/platform/EOSM.202/internals.h b/platform/EOSM.202/internals.h
--- a/platform/EOSM.202/internals.h
+++ b/platform/EOSM.202/internals.h
@@ -142,3 +142,6 @@
 
 /** Workaround for menu timeout in LiveView */
 #define CONFIG_MENU_TIMEOUT_FIX
+
+/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
+#define CONFIG_EDMAC_RAW_SLURP
diff --git a/src/edmac-memcpy.c b/src/edmac-memcpy.c
--- a/src/edmac-memcpy.c
+++ b/src/edmac-memcpy.c
@@ -388,6 +388,10 @@
 uint32_t raw_write_chan = 4;
 #endif
 
+#ifdef CONFIG_EOSM
+uint32_t raw_write_chan = 0x12;
+#endif
+
 
 static void edmac_slurp_complete_cbr (void* ctx)
 {
@@ -401,7 +405,11 @@
 void edmac_raw_slurp(void* dst, int w, int h)
 {
     /* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
+#ifdef CONFIG_EOSM
+    uint32_t dmaFlags = 0x20000000;
+#else
     uint32_t dmaFlags = 0x20001000;
+#endif
     
     /* @g3gg0: this callback does get called */
     RegisterEDmacCompleteCBR(raw_write_chan, &edmac_slurp_complete_cbr, 0);
@@ -420,6 +428,10 @@
     SetEDmac(raw_write_chan, (void*)dst, &dst_edmac_info, dmaFlags);
     
     /* start transfer. no flags for write, 2 for read channels */
+#ifdef CONFIG_EOSM
     StartEDmac(raw_write_chan, 0);
+#else
+    StartEDmac(raw_write_chan, 2);
+#endif
 }
 #endif /* CONFIG_EDMAC_RAW_SLURP */
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -114,6 +114,10 @@
 //~ #define DEFAULT_RAW_BUFFER MEM(0x25f1c + 0x34)  /* 123 */
 #endif
 
+#ifdef CONFIG_EOSM
+#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
+#endif
+
 #else
 
 /* with Canon lv_save_raw, just read it from EDMAC */
@@ -430,6 +434,12 @@
     *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
     return 1;
     #endif
+   
+    #ifdef CONFIG_EOSM
+    *width  = video_mode_crop ? 1872 : 1808;
+    *height = video_mode_crop ? 1060 : 727;
+    return 1;
+    #endif
 
     /* unknown camera? */
     return 0;


[dmilligan]

I guess another thing someone can try now that slurp is implemented is to change the raw type register to see if you can get a focus-pixel-free raw stream (looking at you @dfort).

Code: [Select]

#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
just change #undef to #define

[Teamsleepkid]

Test build pretty please?

[dfort]

(looking at you @dfort).

Yikes!

Ok--first I pulled in the latest commits and dmilligan's patches have already been applied, check.

Then I went into src/raw.c and change one line to:

Code: [Select]

#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
Unfortunately it doesn't compile for the EOSM:

Code: [Select]

[ CC       ]   raw.o
../../src/raw.c: In function 'raw_update_params_work':
../../src/raw.c:590:10: warning: #warning FIXME: are these values correct for 720p and crop modes? [-Wcpp]
         #warning FIXME: are these values correct for 720p and crop modes?
          ^
../../src/raw.c: At top level:
../../src/raw.c:2070:18: error: 'lv_raw_type' undeclared here (not in a function)
         .priv = &lv_raw_type,
                  ^
make: *** [raw.o] Error 1
I thought I tracked down the problem down to this:

Code: [Select]

#ifdef PREFERRED_RAW_TYPE
static int lv_raw_type = PREFERRED_RAW_TYPE;
#endif

So it looks like we need to define PREFERRED_RAW_TYPE which is commented out for the 650D and 700D:

Code: [Select]

/**
 * RAW_TYPE 78 (and others) have a frame-wide green pattern on them
 * ACR can clear them but also kills some details and does not do
 * a good job in general. TL;DR Use pink dot remover.
 * http://www.magiclantern.fm/forum/index.php?topic=6658.0
 */

/*
#ifdef CONFIG_700D
#define PREFERRED_RAW_TYPE 78
#define RAW_TYPE_ADDRESS 0x351B8
#endif

#ifdef CONFIG_650D
#define PREFERRED_RAW_TYPE 78
#define RAW_TYPE_ADDRESS 0x350B4
#endif
*/

We can do better than the old pink dot remover with MLVFS. 

In any case, I've got a 700D so I uncommented that section but it came up with the a similar error:

Code: [Select]

[ CC       ]   raw.o
../../src/raw.c:2065:6: error: #error Only implemented for CONFIG_EDMAC_RAW_SLURP.
     #error Only implemented for CONFIG_EDMAC_RAW_SLURP.
      ^
../../src/raw.c:2070:18: error: 'lv_raw_type' undeclared here (not in a function)
         .priv = &lv_raw_type,
                  ^
make: *** [raw.o] Error 1

@Teamsleepkid and other testers who haven't set up a development enviornment yet -- here's an updated package with the latest changes for all the platforms (except 1100D) and the modded mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Nov19.zip

[dmilligan]

You just need to define PREFFERED_RAW_TYPE for the EOSM. a1ex mentioned a few posts back that the default appears to be 0x22. Start from there and the use the menu to find the best one. You might want to search the forum for this topic, IIRC this was done a long time ago for other cameras, and there was some recent discussion about changing it for 5D3 to get rid of vertical banding. You might also want to split this discussion off into a new topic.

[glubber]

Did some tests on the 550D with latest build from dfort:

12bit is now stable in 1080p25 and 720p50 (fps override to 24 of course)!


12bit-1080p25

12bit-720p50


10bit still results in every second frame corrupted

10bit-720p50


5x zoom creates a camera crash (Err70)

Code: [Select]

ASSERT: dwDeadLine != 0xFFFFFFFF
at LVState.c:5842, task LiveViewMgr
lv:1 mode:20


Magic Lantern version : Nightly.2016Nov19.550D109
Mercurial changeset   : b56f3ee02864 (raw_video_10bit_12bit) tip
Built on 2016-11-20 06:00:40 UTC by rosiefort@RosieFoComputer.
Free Memory  : 196K + 1678K

Hope this helps in taking the 10/12 bit fork further.
To all involved: keep up the awesome work 

[ilia3101]

I tried the new builds on the 5D2 and pink frames are gone. Just split image on every other frame, about half of the frame for 10 bit is covered and most of it for 12 bit, the split part of the frame is always the same, seems to be from the first frame or something. Tested in 1856x1044 16:9 and 1856x1250 3:2 in 23.976fps.

[ddelreal]

So close. 550D 12 bit 8 seconds at 1600x600 (2.67:1) but frames out of order. They're numerically in order but not correct. Like they are numbered wrong in camera.

[dmilligan]

The only real changes from the previous version pertain only to EOSM.

[dfort]

You just need to define PREFFERED_RAW_TYPE for the EOSM. a1ex mentioned a few posts back that the default appears to be 0x22.

It seems that we also need to find RAW_TYPE_ADDRESS for the EOSM in order to get this working.

Code: [Select]

#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#define RAW_TYPE_ADDRESS 0xDEADBEEF
#endif

Otherwise it won't compile:

Code: [Select]

[ CC       ]   raw.o
../../src/raw.c: In function 'raw_update_params_work':
../../src/raw.c:595:10: warning: #warning FIXME: are these values correct for 720p and crop modes? [-Wcpp]
         #warning FIXME: are these values correct for 720p and crop modes?
          ^
In file included from ../../src/dryos.h:47:0,
                 from ../../src/raw.c:21:
../../src/raw.c: In function 'raw_lv_vsync':
../../src/raw.c:1538:42: error: 'RAW_TYPE_ADDRESS' undeclared (first use in this function)
         uint32_t raw_type_register = MEM(RAW_TYPE_ADDRESS-4);
                                          ^
../../src/mem.h:78:38: note: in definition of macro 'MEM'
 #define MEM(x) *(volatile uint32_t*)(x)
                                      ^
../../src/raw.c:1538:42: note: each undeclared identifier is reported only once for each function it appears in
         uint32_t raw_type_register = MEM(RAW_TYPE_ADDRESS-4);
                                          ^
../../src/mem.h:78:38: note: in definition of macro 'MEM'
 #define MEM(x) *(volatile uint32_t*)(x)
                                      ^
make: *** [raw.o] Error 1

Entering the wrong address will simply lock up the camera (requiring battery pull) when activating raw_rec.

[dmilligan]

I'm not sure what the address is, but I don't think you really need it since with this method there's no need to patch Canon code. The raw_type_register should be 0xC0F37014 according to a1ex.

[dfort]

Well without an RAW_TYPE_ADDRESS defined it won't compile. That address you gave me produced this ASSERT LOG:

Code: [Select]

ML ASSERT:
raw_type_register == 0xC0F08114 || raw_type_register == 0xC0F37014
at ../../src/raw.c:1539 (raw_lv_vsync), task Evf
lv:1 mode:3


Magic Lantern version : Nightly.2016Nov20.EOSM202
Mercurial changeset   : b56f3ee02864+ (raw_video_10bit_12bit)
Built on 2016-11-21 00:51:03 UTC by rosiefort@RosieFoComputer.
Free Memory  : 195K + 3406K

I doubt that's the right address for the EOSM. It looks like the address was found for the 5D3 and that's probably the only platform that will work with the "#define RAW_DEBUG_TYPE" trick:

Code: [Select]

#define RAW_TYPE_ADDRESS 0x2D168
I tried that address on the EOSM and it locks up the camera when activating raw_rec requiring a battery pull.

I've also got a 700D I could use to test this but that camera doesn't have CONFIG_EDMAC_RAW_SLURP enabled and getting that working is beyond me at the moment.

[dmilligan]

I didn't say that was the RAW_TYPE_ADDRESS, I said it was the raw_type_register, and that you don't really need RAW_TYPE_ADDRESS. Look carefully at the code (specifically the code where it's used and where the failed assert you posted references)

[dfort]

...you don't really need RAW_TYPE_ADDRESS...

If I understand your instruction, this should work:

Code: [Select]

#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#endif
However, it won't compile.

Changing this line from #undef to #define will also not compile which is how we got into defining PREFERRED_RAW_TYPE.

Code: [Select]

#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
I'm probably missing something obvious but the only way I've gotten #define RAW_DEBUG_TYPE to compile is to also come up with a value for RAW_TYPE_ADDRESS.

[Teamsleepkid]

thank you thank you thank you everyone

[Levas]

Did some testing with the 19 november build and new MLV_dump on the Canon 6d

First frame 00000.dng = always corrupted.

10 bit normal mode and crop mode-> always good
12 bit normal mode and crop mode-> sometimes wrong (diagonal purple stripes along the frames) sometimes good
I'm trying to figure out why 12 bit sometimes works and sometimes won't work ?

About the newest MLV_dump (I'm on osx), it only works with 10 bit.
In 12 bit mode, the newest MLV_dump finds about 40000 cold pixels, which results in frames with 40000 colored and black specles in them.
The same MLV files processed with MLV_dump version of may 2015 are fine.
In 10 bit mode, the newest MLV_dump finds zero cold pixels and gives normal frames.

[Levas]

The live view freezes during recording in 10 and 12 bit.
Do you think this will be fixable, is canon live view capable of showing 10/12bit stream ?

[dmilligan]

If I understand your instruction, this should work:

Code: [Select]

#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#endif
However, it won't compile.

You have to do more than just #define stuff, you will have to change the existing code, but it should be easy (I was not saying that you won't have to do anything at all, just that you don't have to do any reverse engineering to find that memory address, a simple code change should avoid that work). Look where RAW_TYPE_ADDRESS is actually being used and remove it. If you know the register value you don't need the address, because the address is only used to look up the register. This is why I told you to look carefully at the code.

[dmilligan]

The live view freezes during recording in 10 and 12 bit.
Do you think this will be fixable, is canon live view capable of showing 10/12bit stream ?

Implementing CONFIG_EDMAC_RAW_SLURP fixed the issue on EOSM.

[dfort]

...This is why I told you to look carefully at the code.

I can also look carefully at an airplane but I can't fly it.

Ok--figured it wasn't going to be this easy:

just change #undef to #define

I'll keep looking.

[domasa]

10 bit normal mode and crop mode-> always good

Do you see some real danger of damage camera 650D after use current build (19.11.) for 1 hour continuously  10-bit RAW recording?

[Levas]

I have no experience with such long recording times.
But I can't imagine that it can cause permanent damage.

[dfort]

I have no experience with such long recording times.
But I can't imagine that it can cause permanent damage.

Maybe not to the camera but to the viewer!

Just wanted to report that I'm getting great performance out of the latest EOSM changes. About 20 sec. with 10-bit at 1728x692. But hey, this is the EOSM which doesn't have mv1080 and it is running at 29.97 fps. With FPS override set to 23.98 fps it is continuous. Live View is good and no corrupt frames.

I've gotten it to compile with "#define RAW_DEBUG_TYPE" but still haven't been successful in bringing up the debug menus. Got to keep looking at the code.

[Danne]

About 20 sec. with 10-bit at 1728x692

Continuous when shooting 24fps.
*oops, already stated above.

[D_Odell]

Not sure if the first post is still active regarding testing for MLV Lite?

Just tried on latest ML build with 5D3 and got dropped frames after 3 sec. Not sure where it kicked in. Res was maxed 2:35.1.