12-bit (and 10-bit) RAW video development discussion

[Danne]

Use mlvfs or mlv_dump.

[VanGogh7]

That should be working. Does a regular nightly build work fine for you? Maybe delete the ML settings and try again. Turn on only the raw_rec module.

Yes nightly builds works well!

[dfort]

raw2cdng.1.7.4
mlrawviewer
MLVProducer

I doubt any of those will work with 10bit or 12bit. Like Danne said, try MLVFS if you can. Otherwise, the command line tool, mlv_dump will work if you use it like this:

Code: [Select]

mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
Obviously substituting the name of the MLV file you are converting. There was a tweak to mlv_dump to work with lower bit rates so I compiled Mac and Windows versions of mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/mlv_dump_2016Dec09.zip

[capncannabis]

@dfort is there a build to test this on the 600d?

[reddeercity]

On PC , only mlvfs can read 10 & 12bit correctly . Yes you can use mlv_dump But and that a big but  only 10bit can read by applications like A.E. ACR , BM Resolve
12bit from mlv_dump are un-readable as windows can't read them , I posted  here  with examples of 12bit mlv_dump . MLVFS thread and  mlvfs info

[dfort]

12bit from mlv_dump are un-readable as windows can't read them

I downloaded your 12bit file and it works with both MLVFS and mlv_dump. You need to use these options on mlv_dump:

Code: [Select]

mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
This is what it looks like without those options:


And this is what it looks like with the options:


[EDIT] Forgot to mention that I'm on a Mac and I cross compiled mlv_dump for Windows--it should work on your system.

[reddeercity]

Sweet , thanks when I used mlv_dump I didn't think I needed to uses those option

[dfort]

...I didn't think I needed to uses those option

Only for 10bit/12bit files. 14bit files should be fine without using those options.

[Asiyuk]

I try dec 4 build from @dfort with raw_rec (MLV Lite) on 5d3_1.1.3:
- 12 bit 1920x1080 full work except raw-based histogram while recording
- 12 bit in 5x zoom mode - freeze LV while recording (all lv full of bw dots), but video is ok.
- play doesn't work (pink dots video)

I use MLVFS + DaVinci in post, all good with files. I even use this 4dec build in yesterday commercial shooting.

PS: sorry for by bad english 

[eNnvi]

A question to people like a1ex and dmilligan:
how can we be sure about addresses we found?

we're porting raw_slurp to 700D and I think we found the right raw buffer address (lv is working good) but when we look at the file we've got only strange things.
here an example: https://drive.google.com/file/d/0BxMUvr5E8ANaaDdsS0JUVFFwVjA/view?usp=sharing

it happens at all bit depth.

Can it be related to the edmac channel instead of wrong buffer address? (ROM call to ROM on 700D anyway )

[a1ex]

how can we be sure about addresses we found?

Printing DEFAULT_RAW_BUFFER should give the same value(s) with both methods.

To see how Canon does it, look for something like StartMem1Edmac (only on models with EvfState). The older models (550D and earlier) are a bit more difficult.

The 700D should be very similar to EOS M.

[VanGogh7]

I doubt any of those will work with 10bit or 12bit. Like Danne said, try MLVFS if you can. Otherwise, the command line tool, mlv_dump will work if you use it like this:

Code: [Select]

mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
Obviously substituting the name of the MLV file you are converting. There was a tweak to mlv_dump to work with lower bit rates so I compiled Mac and Windows versions of mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/mlv_dump_2016Dec09.zip

MLVFS on Mac works well,

MLVFS on Win. I get in the cmd:

C:\>cd \mlvfs_X86

C:\ mlvfs.exe Z:\ --mlv_dir=E:\MLVFS
Unmont

So it doesn't create a virtual drive...where is the mistake? I installed Dokany and Mlvfs_x86

I tried to install the latest dokan version, now I have a other issue...dokanfuse.dll is missing..it's a pain!!!

Now, I have Win 7 64 bit Ultimate

the solution is to install the 32 bit Microsoft Visual C++ 2015 Redistributable (x86) - 14.0.23026 version

[teatotalTED]

What is unmont?

For me on Win10 64bit I use a batch file (MLVFS.bat) on my desktop, could add it to start up I guess but don't always need MLVFS. Batch file has one line in it as below.

cmd /k G:\MLVFS_x86\mlvfs.exe M:\ --mlv_dir=G:\MLV_FS\Projects

Double click fires up a cmd window and does the necessary.

[dfort]

Got CONFIG_EDMAC_RAW_SLURP working on the 700D

https://bitbucket.org/hudson/magic-lantern/pull-requests/781/700d-enable-config_edmac_raw_slurp/diff

@eNnvi -- the missing piece was:

edmac-memcpy.c

Code: [Select]

void edmac_raw_slurp(void* dst, int w, int h)
 {
     /* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
-#ifdef CONFIG_EOSM
+#if defined(CONFIG_700D) || defined(CONFIG_EOSM)
     uint32_t dmaFlags = 0x20000000;
 #else
     uint32_t dmaFlags = 0x20001000;

Note: this might also work for the 650D and maybe the 100D if we can get the DEFAULT_RAW_BUFFER address for those platforms.

[jc]

Has there been any further looking into how the losses cr2 raw compression is applied ? Could the discovery of running the firmware in QEMU or finding a the "Eeko" core help this discovery ?

[Danne]

Got CONFIG_EDMAC_RAW_SLURP working on the 700D

Breakthrough? 10bit/12bit kickin it with raw_rec?

[dfort]

10bit/12bit kickin it with raw_rec?

Yep--on the 700D. Who's next?

[teatotalTED]

550D ? But as A1ex says above, will be more difficult.

[Rewind]

this might also work for the 650D and maybe the 100D if we can get the DEFAULT_RAW_BUFFER address for those platforms.

Awesome!
Got 650D here, VM all set up and ready for testing. Have some experience with finding STUBS addresses. Can you give us some guidelines on how to find this god damn DEFAULT_RAW_BUFFER address?

[dfort]

Got 650D here, VM all set up and ready for testing. Have some experience with finding STUBS addresses. Can you give us some guidelines on how to find this god damn DEFAULT_RAW_BUFFER address?

Great, check dmilligan's posts. He provided all the hints we used to find the right address. You can check on the pull request for the 700D to make sure you find all the pieces that are needed for the 650D. They should be the same with the exception of the DEFAULT_RAW_BUFFER address.

[Rewind]

Thanks to dfort, i've managed to find the right address for 650D:

Code: [Select]

#ifdef CONFIG_650D
#define DEFAULT_RAW_BUFFER MEM(0x25B00 + 0x3C)
#endif
LiveView works during recording, no pink frames or glitches in extracted dngs (raw_rec, both 12- and 10-bit). Some shifting occurs though, focus pixel masks no longer works (MLVFS windows), need more testing.

Upd. Actually, there are no focusing pixels anymore on 650D (raw_rec, 14 bit as well as 12 and 10 bit), but there are these along the contrast edges (all over the frame). What am I missing? Were there some changes in raw framework?


What PREFERRED_RAW_TYPE should we use for 650D to get the old behavior (usual horizontal pattern of focusing pixel, removable by PDR etc.)?

[D_Odell]

Do you all get this from live view while doing crop recording?



Using 4th December build.

[VanGogh7]

I had this with Global Draw on...try with GD OFF

[Danne]

D_Odell. Are you using raw_twk? When in crop mode this will surely happen.

[dfort]

What PREFERRED_RAW_TYPE should we use for 650D to get the old behavior (usual horizontal pattern of focusing pixel, removable by PDR etc.)?

I'm seeing the same happening on the EOSM and 700D. I tested out some different RAW_TYPE's and found some that show the old focus pixel patterns, others that show only a few pixels, mostly around high contract boundaries and yet others that show a pixel pattern all over the image including areas that we've never seen focus pixels before. I reported on that in the Dealing with Focus Pixels in raw video topic.

We recently increased the RAW_DEBUG_TYPE limit and switched over to defining the PREFERRED_RAW_TYPE in hex instead of decimal values. The pull request is quite an interesting read.

To activate RAW_DEBUG_TYPE, define it here:

Code: [Select]

#undef RAW_DEBUG        /* define it to help with porting */
#undef RAW_DEBUG_DUMP   /* if you want to save the raw image buffer and the DNG from here */
#undef RAW_DEBUG_BLACK  /* for checking black level calibration */
#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
/* see also RAW_ZEBRA_TEST and RAW_SPOTMETER_TEST in zebra.c */

You also need to give it a starting value. Anything will do because you can change it from a menu:

Code: [Select]

#ifdef CONFIG_650D
#define PREFERRED_RAW_TYPE 0x12
#endif

This will give you a new Debug menu item that allows you to enter values from 0 to ffff (65,535).



Try them all out--that should keep you busy for a while. Actually, they repeat and many do not produce valid images or cause raw detect errors.

You said that the frame might have shifted? In the raw_lv_get_resolution function of raw.c you might check if the 650D uses either of these resolutions--I'm pretty sure it uses the 700D values:

Code: [Select]

    #ifdef CONFIG_600D
    *width  = zoom ? 2520 : mv1080crop ? 1952 : mv720  ? 1888 : 1888;
    *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
    return 1;
    #endif


Code: [Select]

    #ifdef CONFIG_700D
    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
    return 1;
    #endif

Are you planning on putting up a pull request for this? It would be great to get some feedback from other 650D users.