12-bit (and 10-bit) RAW video development discussion

[canneloni]

Is there still no build for the 100D ? I would like to test it and help as much as i can. It's really great what you guys are doing!

[DeafEyeJedi]

Actually @dfort and I are still waiting to hear back from @nikfreak on this very possibility (for both 100D & 70D) and will compile them as soon as we can -- we are just as anxious as you all are!   

[reddeercity]

@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test 
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here  .

 

[ddelreal]

Using MLVFS still getting every other half-pink frame. 550D.

[reddeercity]

Sorry guy should be Canon preview with overlays killed when recording for corrupt free @1044p 5D2
Edit:
30p is unstable , I can't now get it to record corruption free on every recording at the moment , some times it does so 50/50
but 23.976p is , sorry for the confusion I'll do some research  .

[andy kh]

No pro@reddeercity
Me too waiting to hear back from nikfreak for the 70D

[Deadcode]

@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test 
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here  .

 

Still not working with these settings. I will make a video around 8pm (GMT+1 ) about it from installing ML till opening the dng's in Davinci, maybe you can spot the wrong settings.

[Steve_FR]

(2016Nov09.EOSM202)

I don't experience any crashing on the Eos M when recording with either zoom mode or 3x crop mode enabled.
Video playback works too, in both color and b&w. Video is scrambled.  Raw @ 10bit. Tried with SRM Job ON and OFF.
I can record multiple clips back to back with no crash, aside from the expected preview frozen only during REC.

If I shoot in (mv720 I believe?) (no crop, no zoom) then I experience the crash mentioned, and have to pop battery. I tried with SRM Job on and off, and the crash still occurred, followed by the invalid header size error when playing back the clip that was attempting to be recording during the crash. I can still play the other clips fine that were recorded using 3x or zoom, but scrambled as expected.

[Deadcode]

@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test 
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here  .
 

I made a video about the process. What am i missing?

https://youtu.be/VI1u8Ouk_r8

[ilia3101]

@Deadcode @reddeercity I have also tried all the settings you list and many other options, I can't get it to work without half corrupt frames, or splitting/slicing on the lower half, and I have the same build situation on my camera. But it is definitely amazing for crop mode, as it can be full res 2144x1076 and continuous How might this issue be fixed, and what help can the users give?

[reddeercity]

@Deadcode , @Ilia3101 Ok I'll post a short video/images to how to get it working.


 

[Levas]

Tested the 4 november version of Daniel on the 6d.
It works, sort of, although image preview is halted, the 'allow global draw' option is missing in the menu ?
And i'm missing sound, but that's probably normal for Raw_rec module 

First frame is always corrupted.
10 bit works
12 bit works
But it doesn't work all the time, there was also a MLV file with all corrupted frames and a MLV file with frames with vertical lines corrupted, pink/magenta vertical lines and a posterized image behind it.
I've used MLV_dump.osx for getting the DNG's and viewed them in RawTherapee.

The vertical line correction in MLV_Dump always seems to use exactly 1.000 or 1 numbers with the 10bit files, the files look ok, but is it explainable that it is always exactly 1.000 ?

 

[dmilligan]

Enabling the new method on all models is desirable IMO, because we will have a single way of doing things. However, each model is likely to have its own quirks, so it's a bit of work.

Trying to get slurp working on EOSM. I assume all I need to do is find a free edmac write channel and then hardcode the DEFAULT_RAW_BUFFER?

The hint in the code is

Code: [Select]

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
 *
 * raw_buffer = get_raw_buffer()
 * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
 * ...
 * dump_file(filename, raw_buffer, 7*something...)
 */

I looked around in the disassembly of lv_raw_dump and called functions, but couldn't find any hardcoded values that looked like the other raw buffer addresses, at the beginning of lv_raw_dump EOSM appears to call two functions in RAM (not sure where to find these in ROM).

So I figured I would just call("lv_raw_dump"). Got a file on the card:
46798080.mm1

Also doesn't look much like the other addresses. (60D gave: 48332200.MM1, in raw.c the value is hardcoded as 0x5028)

Any tips?

[a1ex]

On 60D, the raw buffer address is returned by FF0F8B8C.

On EOS M, the function you are looking for is 10BF4. Add RAM_OFFSET from stubs.S =>  ffa7a184. This one returns MEM(0x404E4 + 0x44).

The raw buffer address is probably allocated dynamically (or maybe just hardcoded) from RscMgr.

[reddeercity]

@Deadcode look like your setup is ok , well after some time testing different configurations on 5D2 I can't seem now to get 10 or 12 bit working consistently
ever time . When the camera & card are cold (no operated for at least 3 Hours) I can record 10 or 12 bit 1856x1044 @ 24p & or 30p but can't repeat the corrupt free
recording and when it does work I notice a message on the very top LCD screen "busy" that flashes on and off for about the first 20 of so seconds then is gone .

So 10bit @ 1856x1044 23.976p & 29.97p are continuous if you can get a corrupt free recording .
of which I did a few times . Plus I was able to get 12bit @ 1856x1044 23.976p , 29.97p at this bit depth that can not be maintain for very long.

Below I have posted three links to my Google Drive where I have 3 Corrupt Free recordings of 10bit 1856x1044 @ 23.976 & 29.97p & one of 12bit 1856x1044 @ 23.976p
In the drive folder there is for each bit depth & frame rate the Original .MLV file 1.0Gb , 1.1Gb , 550Mb the True 10 & 12 bit DNGs exacted with mlv_dump . There about
75 dngs for each file .

Google drive links
12bit 1856x1044_23.976p     10bit 1856x1044_29.97p    10bit 1856x1044_23.976p

Now that bring me to the bit depth we have , there seem to be a problem with 12bit as Resolve or ACR/A.E. can't read or understand 12bit from mlv_dump.
10bit seen to read ok in A.E. & Resolve just 12bit. Why does this matter if you use MLVFS , yes  it does extract the cdngs ok from 10 & 12bit mlv's But
these are 16bit files (3.75Mb per frame) where as the 10 bit are (2.34Mb) & 12bit (2.8Mb) . So in some cases you are saving up to 1Mb per frame which can add up .
Plus the other factor is from 10bit (Billions of Colors) to 16bit(Trillions of Color) is the color accurate or will the color shift etc... .

Screen shot of A.E. CS6 with 12bit , can't read 12bit from mlv_dump 


ML_Raw_12bit_dngs_AE-CS6 by RedDeerCityTV, on Flickr

Screen shot of A.E. CS6 ACR with 12bit , can't read 12bit from mlv_dump 


ML_Raw_12bit_dngs_ACR-AE-CS6 by RedDeerCityTV, on Flickr


Screen shot of Blackmagic Resolve with 12bit , can't read 12bit from mlv_dump 


12.5_Blackmagic_Resolve_12bit_dng by RedDeerCityTV, on Flickr
 
Edit:

Screen Shots Blackmagic Resolve with True 10 bit from mlv_dump

 
12.5_Blackmagic_Resolve_10bit_dng by RedDeerCityTV, on Flickr


Screen Shots A.E. CS6 Timeline with True 10 bit from mlv_dump


ML_Raw_10bit_dngs_AE-CS6 by RedDeerCityTV, on Flickr


Screen Shots A.E. CS6 ACR with True 10 bit from mlv_dump


ML_Raw_10bit_dngs_ACR-AE-CS6 by RedDeerCityTV, on Flickr

[Teamsleepkid]

I'm not a scientist but 10 bit looks great in resolve 12.5 using mlvfs after white balance in the raw tab on eos m.

[reddeercity]

I'm not a scientist but 10 bit looks great in resolve 12.5 using mlvfs after white balance in the raw tab on eos m.

Sure it does because it's not 10bit anymore it's 16bit Cdng , that's point I was getting at MLVFS takes the 10bit image and up-samples
to 16bit even thou there's not 16 bits (well really 14bit) of information plus the file get blown up by at least 1Mb per frame.

Exact with mlv_dump for true 10bit and see if there's any difference .

[dmilligan]

There will be no difference. It's not an "up-sample". The values are exactly the same. The number 123 represented with 10 bits or 16 bits or 64 bits or a million bits is still the same value, 123. CPUs can't do math on 10 bit numbers so it's going to get converted to 16 (or maybe even 32 or 64) internally at some point regardless.

Also, MLVFS files are virtual, so they are not actually taking up any additional real space.

[dmilligan]

I tried a couple different edmac channels, but I'm just getting garbage, and with crop hack LV just freezes. Frame sizes are the ones reported by raw_info correct? That was the only other thing I needed to do?

[a1ex]

Frame sizes must be either hardcoded in raw_lv_get_resolution, or autodetected (as done on 5D3 on the crop_rec branch; you may have to tweak the width multiplier, as it's very likely to be lower than 8 ).

Once that is done, make sure raw_lv_vsync runs once for every frame (you can place bmp_printf's with small fonts there) and that raw_type_register is 0xC0F37014 (value for digic 5). Default raw type appears to be 0x22. Then, you could check whether edmac_slurp_complete_cbr gets called.

On 550D iirc, I couldn't get a raw stream until setting the raw type register.

You could also try the following changes:

Code: [Select]

raw_write_chan = 0x12
dmaFlags = 0x20000000
StartEDmac(raw_write_chan, 2)

Also, can you check the configuration of EDMAC channel #18 (Debug -> Show EDMAC) after enabling the LiveView raw stream with the old method?

The vsync hook looks ok to me.

A dm-spy log from LiveView might also help.

[Teamsleepkid]

Alex you talking about trying to get live view working?

[ch_d]

Any updates for the 5DM2?
Where can I find the latest builds?

Thank you all - you are the best 

[hjfilmspeed]

This is AWESOME!

[dmilligan]

You could also try the following changes:

Code: [Select]

raw_write_chan = 0x12
dmaFlags = 0x20000000
StartEDmac(raw_write_chan, 2)


That worked!

Very first try I got an Early Stop(5) and:

Code: [Select]

[89] raw_rec_task: NULL PTR (80c3e100,e1a00000)
pc=ff2c6020 lr=  ac5ad0 stack=1ddde0+0x1000
entry=abef78(0)
e1a00000 e59ff014 e59ff014 e59ff014
e59ff014 e1a00000 e59ff010 e59ff010

Everything worked fine after that, I did about 5 recordings in normal mode and about 5 in crop hack (I don't think there are any other video modes I can actually try with EOSM?). Also briefly tested 10 bit, appears to be working fine and Canon preview also working.

[a1ex]

All of those 3 changes are needed, right?

(I'm still unsure what EDMAC flags to use when, and how they affect the result...)

To diagnose the null ptr error, the mem_prot module can be helpful, as it can show the exact location that performed the invalid write. Or, a way to reproduce. There are some reports showing it on 60D as well.