Author Topic: 12-bit (and 10-bit) RAW video development discussion  (Read 509287 times)

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6265
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #700 on: December 10, 2016, 01:38:12 AM »
Use mlvfs or mlv_dump.

VanGogh7

  • New to the forum
  • *
  • Posts: 14
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #701 on: December 10, 2016, 01:38:57 AM »
That should be working. Does a regular nightly build work fine for you? Maybe delete the ML settings and try again. Turn on only the raw_rec module.

Yes nightly builds works well!

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #702 on: December 10, 2016, 01:42:36 AM »
raw2cdng.1.7.4
mlrawviewer
MLVProducer

I doubt any of those will work with 10bit or 12bit. Like Danne said, try MLVFS if you can. Otherwise, the command line tool, mlv_dump will work if you use it like this:

Code: [Select]
mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
Obviously substituting the name of the MLV file you are converting. There was a tweak to mlv_dump to work with lower bit rates so I compiled Mac and Windows versions of mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/mlv_dump_2016Dec09.zip
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

capncannabis

  • New to the forum
  • *
  • Posts: 2
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #703 on: December 10, 2016, 03:51:53 AM »
@dfort is there a build to test this on the 600d?

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2141
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #704 on: December 10, 2016, 05:21:50 AM »
On PC , only mlvfs can read 10 & 12bit correctly . Yes you can use mlv_dump But and that a big but  only 10bit can read by applications like A.E. ACR , BM Resolve
12bit from mlv_dump are un-readable as windows can't read them , I posted  here  with examples of 12bit mlv_dump . MLVFS thread and  mlvfs info

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #705 on: December 10, 2016, 06:27:54 AM »
12bit from mlv_dump are un-readable as windows can't read them

I downloaded your 12bit file and it works with both MLVFS and mlv_dump. You need to use these options on mlv_dump:

Code: [Select]
mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
This is what it looks like without those options:


And this is what it looks like with the options:


[EDIT] Forgot to mention that I'm on a Mac and I cross compiled mlv_dump for Windows--it should work on your system.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2141
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #706 on: December 10, 2016, 06:50:15 AM »
Sweet , thanks when I used mlv_dump I didn't think I needed to uses those option

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #707 on: December 10, 2016, 08:29:44 AM »
...I didn't think I needed to uses those option

Only for 10bit/12bit files. 14bit files should be fine without using those options.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Asiyuk

  • New to the forum
  • *
  • Posts: 2
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #708 on: December 10, 2016, 11:03:48 AM »
I try dec 4 build from @dfort with raw_rec (MLV Lite) on 5d3_1.1.3:
- 12 bit 1920x1080 full work except raw-based histogram while recording
- 12 bit in 5x zoom mode - freeze LV while recording (all lv full of bw dots), but video is ok.
- play doesn't work (pink dots video)

I use MLVFS + DaVinci in post, all good with files. I even use this 4dec build in yesterday commercial shooting.

PS: sorry for by bad english  :-X
5dIII_1.1.3

eNnvi

  • New to the forum
  • *
  • Posts: 47
  • 700D
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #709 on: December 10, 2016, 03:09:39 PM »
A question to people like a1ex and dmilligan:
how can we be sure about addresses we found?

we're porting raw_slurp to 700D and I think we found the right raw buffer address (lv is working good) but when we look at the file we've got only strange things.
here an example: https://drive.google.com/file/d/0BxMUvr5E8ANaaDdsS0JUVFFwVjA/view?usp=sharing

it happens at all bit depth.

Can it be related to the edmac channel instead of wrong buffer address? (ROM call to ROM on 700D anyway :D )

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12359
  • Quarantined
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #710 on: December 10, 2016, 05:12:57 PM »
how can we be sure about addresses we found?

Printing DEFAULT_RAW_BUFFER should give the same value(s) with both methods.

To see how Canon does it, look for something like StartMem1Edmac (only on models with EvfState). The older models (550D and earlier) are a bit more difficult.

The 700D should be very similar to EOS M.

VanGogh7

  • New to the forum
  • *
  • Posts: 14
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #711 on: December 11, 2016, 12:09:50 PM »
I doubt any of those will work with 10bit or 12bit. Like Danne said, try MLVFS if you can. Otherwise, the command line tool, mlv_dump will work if you use it like this:

Code: [Select]
mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
Obviously substituting the name of the MLV file you are converting. There was a tweak to mlv_dump to work with lower bit rates so I compiled Mac and Windows versions of mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/mlv_dump_2016Dec09.zip


MLVFS on Mac works well,

MLVFS on Win. I get in the cmd:

C:\>cd \mlvfs_X86

C:\ mlvfs.exe Z:\ --mlv_dir=E:\MLVFS
Unmont


So it doesn't create a virtual drive...where is the mistake? I installed Dokany and Mlvfs_x86

I tried to install the latest dokan version, now I have a other issue...dokanfuse.dll is missing..it's a pain!!!

Now, I have Win 7 64 bit Ultimate

the solution is to install the 32 bit Microsoft Visual C++ 2015 Redistributable (x86) - 14.0.23026 version
:)

teatotalTED

  • New to the forum
  • *
  • Posts: 35
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #712 on: December 11, 2016, 07:37:36 PM »
What is unmont?

For me on Win10 64bit I use a batch file (MLVFS.bat) on my desktop, could add it to start up I guess but don't always need MLVFS. Batch file has one line in it as below.

cmd /k G:\MLVFS_x86\mlvfs.exe M:\ --mlv_dir=G:\MLV_FS\Projects

Double click fires up a cmd window and does the necessary.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #713 on: December 11, 2016, 09:34:29 PM »
Got CONFIG_EDMAC_RAW_SLURP working on the 700D

https://bitbucket.org/hudson/magic-lantern/pull-requests/781/700d-enable-config_edmac_raw_slurp/diff

@eNnvi -- the missing piece was:

edmac-memcpy.c
Code: [Select]
void edmac_raw_slurp(void* dst, int w, int h)
 {
     /* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
-#ifdef CONFIG_EOSM
+#if defined(CONFIG_700D) || defined(CONFIG_EOSM)
     uint32_t dmaFlags = 0x20000000;
 #else
     uint32_t dmaFlags = 0x20001000;

Note: this might also work for the 650D and maybe the 100D if we can get the DEFAULT_RAW_BUFFER address for those platforms.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

jc

  • New to the forum
  • *
  • Posts: 35
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #714 on: December 11, 2016, 09:49:11 PM »
Has there been any further looking into how the losses cr2 raw compression is applied ? Could the discovery of running the firmware in QEMU or finding a the "Eeko" core help this discovery ?

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6265
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #715 on: December 11, 2016, 10:15:20 PM »
Quote
Got CONFIG_EDMAC_RAW_SLURP working on the 700D
Breakthrough? 10bit/12bit kickin it with raw_rec?

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #716 on: December 11, 2016, 10:43:32 PM »
10bit/12bit kickin it with raw_rec?

Yep--on the 700D. Who's next?
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

teatotalTED

  • New to the forum
  • *
  • Posts: 35
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #717 on: December 12, 2016, 01:25:58 AM »
550D ? But as A1ex says above, will be more difficult.

Rewind

  • Senior
  • ****
  • Posts: 356
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #718 on: December 12, 2016, 05:34:46 AM »
this might also work for the 650D and maybe the 100D if we can get the DEFAULT_RAW_BUFFER address for those platforms.
Awesome!
Got 650D here, VM all set up and ready for testing. Have some experience with finding STUBS addresses. Can you give us some guidelines on how to find this god damn DEFAULT_RAW_BUFFER address?

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #719 on: December 12, 2016, 07:43:01 AM »
Got 650D here, VM all set up and ready for testing. Have some experience with finding STUBS addresses. Can you give us some guidelines on how to find this god damn DEFAULT_RAW_BUFFER address?

Great, check dmilligan's posts. He provided all the hints we used to find the right address. You can check on the pull request for the 700D to make sure you find all the pieces that are needed for the 650D. They should be the same with the exception of the DEFAULT_RAW_BUFFER address.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Rewind

  • Senior
  • ****
  • Posts: 356
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #720 on: December 12, 2016, 09:03:54 AM »
Thanks to dfort, i've managed to find the right address for 650D:

Code: [Select]
#ifdef CONFIG_650D
#define DEFAULT_RAW_BUFFER MEM(0x25B00 + 0x3C)
#endif

LiveView works during recording, no pink frames or glitches in extracted dngs (raw_rec, both 12- and 10-bit). Some shifting occurs though, focus pixel masks no longer works (MLVFS windows), need more testing.

Upd. Actually, there are no focusing pixels anymore on 650D (raw_rec, 14 bit as well as 12 and 10 bit), but there are these along the contrast edges (all over the frame). What am I missing? Were there some changes in raw framework?


What PREFERRED_RAW_TYPE should we use for 650D to get the old behavior (usual horizontal pattern of focusing pixel, removable by PDR etc.)?

D_Odell

  • Member
  • ***
  • Posts: 187
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #721 on: December 12, 2016, 04:33:41 PM »
Do you all get this from live view while doing crop recording?



Using 4th December build.
5D3 (OLPF removed) :: 1.1.3 :: Canon FD L Serie

VanGogh7

  • New to the forum
  • *
  • Posts: 14
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #722 on: December 12, 2016, 04:43:48 PM »
I had this with Global Draw on...try with GD OFF

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6265
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #723 on: December 12, 2016, 05:09:49 PM »
D_Odell. Are you using raw_twk? When in crop mode this will surely happen.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3743
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #724 on: December 12, 2016, 06:44:21 PM »
What PREFERRED_RAW_TYPE should we use for 650D to get the old behavior (usual horizontal pattern of focusing pixel, removable by PDR etc.)?

I'm seeing the same happening on the EOSM and 700D. I tested out some different RAW_TYPE's and found some that show the old focus pixel patterns, others that show only a few pixels, mostly around high contract boundaries and yet others that show a pixel pattern all over the image including areas that we've never seen focus pixels before. I reported on that in the Dealing with Focus Pixels in raw video topic.

We recently increased the RAW_DEBUG_TYPE limit and switched over to defining the PREFERRED_RAW_TYPE in hex instead of decimal values. The pull request is quite an interesting read.

To activate RAW_DEBUG_TYPE, define it here:

Code: [Select]
#undef RAW_DEBUG        /* define it to help with porting */
#undef RAW_DEBUG_DUMP   /* if you want to save the raw image buffer and the DNG from here */
#undef RAW_DEBUG_BLACK  /* for checking black level calibration */
#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
/* see also RAW_ZEBRA_TEST and RAW_SPOTMETER_TEST in zebra.c */

You also need to give it a starting value. Anything will do because you can change it from a menu:

Code: [Select]
#ifdef CONFIG_650D
#define PREFERRED_RAW_TYPE 0x12
#endif

This will give you a new Debug menu item that allows you to enter values from 0 to ffff (65,535).



Try them all out--that should keep you busy for a while. Actually, they repeat and many do not produce valid images or cause raw detect errors.

You said that the frame might have shifted? In the raw_lv_get_resolution function of raw.c you might check if the 650D uses either of these resolutions--I'm pretty sure it uses the 700D values:

Code: [Select]
    #ifdef CONFIG_600D
    *width  = zoom ? 2520 : mv1080crop ? 1952 : mv720  ? 1888 : 1888;
    *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
    return 1;
    #endif

Code: [Select]
    #ifdef CONFIG_700D
    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
    return 1;
    #endif

Are you planning on putting up a pull request for this? It would be great to get some feedback from other 650D users.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102