12-bit (and 10-bit) RAW video development discussion

[zachnfine]

This is likely a question that was answered earlier in the thread - but I searched around and didn't find it. With this new feature can one shoot 10-bit on the 5D3 at higher than 1080p resolution in full-frame mode (i.e. not crop-mode)?

I've been doing fine recording 1080p23.98 to my Komputerbay 128Gb cards (1 1050x, 1 1066x) at 14-bit, which requires 82.97Mb/s. But if 10-bit is possible, then 2272x1278 @ 23.98 would be nearly the same bitrate at 82.99Mb/s - and that might be a perceptible increase in detail, even if downscaled to 1080p in post, or at least it'd leave a little extra headroom for stabilization. And if 10-bit drops too much usable color information, 12-bit at 2064x1161 23.98 is 82.19Mb/s.

Or maybe the card's capable of more and I'm aiming low with that bitrate, I haven't benchmarked. And maybe card spanning would be a help. It'd be fun to see how high a resolution can be recorded at 23.98.

[reddeercity]

Just look it up in raw.c and earlier in this thread

Thanks @a1ex
Ok I think I got my head around this a little better ,
raw.c- line160

#ifdef CONFIG_DIGIC_V
#define RAW_TYPE_REGISTER 0xC0F37014
#else
#define RAW_TYPE_REGISTER 0xC0F08114    /* PACK32_ISEL */
#endif

I see "0xC0F08114" etc.... stuff , question;  with the "#ifdef CONFIG_DIGIC_V" then it has #else , does (#else) mean Digic4
or would it make a difference to call "#ifdef CONFIG_DIGIC_IV" or is that a way of writing less code ?

As I see how other newer cams are configured does there need to be a "#ifdef CONFIG_xxx" for the 5D2 ?
maybe like this ?

Code: [Select]

#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
#endifFrom the Register Map "PACK32_ISEL" has  1-15 selectable variables
Would be right to put that variable in brackets ?
Or does "PACK32_ISEL" need to "#define" at all ?
Not sure how to write the Code/Syntax
Close you think , or way off base here .
Edit:
Because there 15 different variables does that mean the only way to figure it out is by trial & error ?

[kzv]

Hello!

First time posting, so thank you for all your great work on Magic Lantern and especially this experimental version!

Been testing it and works great except one thing -in last build (and some previous in 2016) there is the problem with the ISO table. When you change ISO during recording, it wont hide.

Here is the picture: https://drive.google.com/file/d/0Bw9gcCutoLPQenBkMGJ4SUdxUjQ/view?usp=sharing

Can you please have a look at this bug? I would be very grateful.

Thank you very much in advance!

Chris

[hyalinejim]

Yes, this happened to me last night when I was shooting a performance  (5D3 1.1.3 10bit)

While shooting, pressing the physical ISO button brings up the ISO overlay. I use the shutter wheel to change ISO. But then the overlay stays on screen until the camera stops recording.

[Surin Dmitriy]

You can push rate button and use joystick to change iso.

Thanks @a1ex
Ok I think I got my head around this a little better ,
raw.c- line160I see "0xC0F08114" etc.... stuff , question;  with the "#ifdef CONFIG_DIGIC_V" then it has #else , does (#else) mean Digic4
or would it make a difference to call "#ifdef CONFIG_DIGIC_IV" or is that a way of writing less code ?

As I see how other newer cams are configured does there need to be a "#ifdef CONFIG_xxx" for the 5D2 ?
maybe like this ?

Code: [Select]

#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
#endifFrom the Register Map "PACK32_ISEL" has  1-15 selectable variables
Would be right to put that variable in brackets ?
Or does "PACK32_ISEL" need to "#define" at all ?
Not sure how to write the Code/Syntax
Close you think , or way off base here .
Edit:
Because there 15 different variables does that mean the only way to figure it out is by trial & error ?

[budafilms]

@reddeercity

Are you looking for the way to twick Canon zoom for less magnification than 3x?

[ilia3101]

@budafilms I think reddeercity is trying to stop the live view from freezing on 5D2 when its recording(I always have that problem so I think its that)
@reddeercity have you had wrong(0 probably) black level in crop mode on your 5D2? it seems to be broken in 10bit-12bit branch since December, even shooting 14 bit, I keep getting a pink cast in crop mode no matter what settings or build (does MLVFS have a black level fix?)

[reddeercity]

@reddeercity Are you looking for the way to twick Canon zoom for less magnification than 3x?

No but that that would be a great dream 

@budafilms I think reddeercity is trying to stop the live view from freezing on 5D2 when its recording(I always have that problem so I think its that)

Correct ! in 1:1 plus stop frame corruption but if liveview works properly hopefully that will fix that.

@reddeercity have you had wrong(0 probably) black level in crop mode on your 5D2?
it seems to be broken in 10bit-12bit branch since December, even shooting 14 bit,
I keep getting a pink cast in crop mode no matter what settings or build (does MLVFS have a black level fix?)

No , are you using full mlv or mlv lite (.raw) ?
The video I posted was crop 10bit 2.1K mlv+audio (Full)
I need Audio when I record raw , I have been using just full mlv
No , MLVFS does not are you on windows or Mac?
I Thing if I'm not mistaken MLVProducer has W/B raw levels that can be adjusted on the fly in realtime "ACR" like 
Thou it's a windows base program it can be run under Wine on OSX ( I tried it on my MacPro4.1 on Yosemite)  works nice

The black level in crop has been like this for some time , it's has something to do with the mlv lite I think , not sure thou.
1:1 thou is ok in 10,12,14bit I'm sure , just not crop. If you read the mlv lite source code , there's some notes about it  I think.
 

[reddeercity]

After consulting with my IT friend asking for some basic help (I'm sure he's was laughing at me )
showing him my code sample , he return with a long email explaining why I should not try to code 
Will in the end he did give me a sample to work on

Code: [Select]

#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
if PACK_ISEL (3))
          {
           do something
           Return a valve
           }
#endif
this look better ? it's a work in process

I'm still learning , 

[Licaon_Kter]

Code: [Select]

if PACK_ISEL (3))
You're missing a ( after if I guess

[KelvinK]

There's a very annoying bug in experimental build (was there since very first versions), at least for 6D: in LV when the play videos, it can play 2-3 then screen turns black. To get back GUI you've to press trash-bin button.

Would be great if it can be fixed.

[RenatoPhoto]

Yes this bug is confirmed.  I have reported it before for 5D3.
Video playback is limited to about 3 videos, probably a problem in raw_twk module.

[KelvinK]

I don't think so. If you don't load raw_twk and record 14bit only, playback has same issue with 14b files.

[reddeercity]

Code: [Select]

if PACK_ISEL (3))You're missing a ( after if I guess

Ok Thanks
So like this

Code: [Select]

#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
if ( PACK_ISEL (3))
          {
           do something
           Return a valve
           }
#endif
Little closer , still need to find more on PACK32_ISEL stuff.
I haven tried to compile anything yet ...... soon , stay tune

[dmilligan]

What exactly are you trying to do? The code you have posted is just nonsense (I'm not trying to be mean).

PACK32 is the name Canon has given to an image processing hardware module in the camera. PACK32_ISEL is the name Canon has given to a register for this module that controls its behavior. a1ex has already given an explanation of what is known about what this module does and what this particular register probably controls here. Until you are at least to the point of thoroughly and completely understanding everything that is said in that message, there's no point in trying to write any code.

You can't just pick up a violin for the first time and expect to be able to play the Brahms violin concerto. If you really want to contribute and learn, you will have to start with something simpler and learn more about programming in general and ML specifically (by, for example, playing with Lua or doing an easy coding task).

[reddeercity]

What exactly are you trying to do? The code you have posted is just nonsense (I'm not trying to be mean).
 

No I understand,
it's big learning curve there's a lot to take in and there is a lot that's over my head .
I'm trying to get liveview in 1:1 with bit reduction in work with out liveview freezeing up and not have it produce alternating corrupt frames .

[dfort]

@reddeercity - I don't know much about coding or reverse engineering or playing the violin but one valuable lesson I learned from dmilligan is to look at the code. To get this 10bit/12bit thing working you need to figure out how to get CONFIG_EDMAC_RAW_SLURP working. Look at the internals.h file for any of the cameras that are supported and you'll see this:

internals.h

Code: [Select]

/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#define CONFIG_EDMAC_RAW_SLURP

Ah if it were only as easy as adding that one line, but there's more to it.

Currently the 5D2 uses the LiveView raw buffer:

raw.c

Code: [Select]

/**
 * LiveView raw buffer address
 * To find it, call("lv_save_raw") and look for an EDMAC channel that becomes active (Debug menu)
 **/

#if defined(CONFIG_5D2) || defined(CONFIG_50D)
#define RAW_LV_EDMAC 0xC0F04508
#endif

Cameras that have CONFIG_EDMAC_RAW_SLURP use the default raw buffer instead of the LiveView raw buffer.

raw.c

Code: [Select]

#ifdef CONFIG_EDMAC_RAW_SLURP
/* undefine so we don't use it by mistake */
#undef RAW_LV_EDMAC

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
 *
 * raw_buffer = get_raw_buffer()
 * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
 * ...
 * dump_file(filename, raw_buffer, 7*something...)
 */

#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif

So all you're looking for is that one hexadecimal number to unlock this feature. The question is how do you find it? Read the code, it tells you. Now the hard part is following those instructions and come up with the right address.

Back to the violin analogy, how do you get to Carnegie Hall? Practice.

[reddeercity]

@dfort Thanks !
I read so much code my head is spinning and of course I when down the wrong rabbit hole .
It all make sense now , I kind of thought that's what I being missing that CONFIG_EDMAC_RAW_SLURP  RAW_BUFFER address stuff
I just couldn't understand if that was needed to be done on digic4 (5D2 etc..) cams like digic5 cams (5d3 etc..)
now I know thank again .
Can you hear my violin warming up thou it's a bit scratchy must be something wrong with my bow     
When I can play the scales correctly , then I'll post 
 

[nikfreak]

Really your best bet is a1ex as he already confirmed that he tried to solve it for DIGIC IV.

Maybe next time?

You can try to solve it yourself but I can recommend to have a look into the state diagrams. Without having a look at them and understanding the vsync functions in ML you can also try to win a lottery before solving it for DIGIC IV. Finding the buffer sizes or address is just a look at the edmac screenshot and following the solution commited for 1100D... Just my 0.02$:

https://a1ex.bitbucket.io/ML/states/index.html

[reddeercity]

Thanks @nikfreak you have been very helpful .

[dfort]

...following the solution commited for 1100D...

Which branch are you looking at for the 1100D solution? Is it in the allocate-raw-lv-buffer?

The way the 1100D is handled is completely different than the 60D which AFAIK is the only other Digic IV camera that's working with 10bit/12bit.

Code: [Select]

#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif
...

#ifdef CONFIG_1100D
#define RAW_LV_BUFFER_ALLOC_SIZE (3906*968)
#endif

The 1100D isn't in the crop_rec or raw_video_10bit_12bit branches so is the build on the Experiments download page from the allocate-raw-lv-buffer branch? Note that the 1100D nightly build from the unified branch doesn't have the raw_rec or mlv_rec modules though it does have mlv_play, which is rather odd.

In any case, where can the Digic IV camera owners who are trying to solve the 10bit/12bit riddle find more information on how the 1100D was solved?

[nikfreak]

You probably expecting a final solution but my previous post maybe wasn't that clear. It was just about the edmac address (channel) and buffer size (width*height). What's known was committed to the allocate-raw-lv-buffer.

I already tried to explain it but won't be of any help in resolving it myself due to the complexity and lack of knowledge. Still crossing my fingers ofc for 7D/5D2 and hoping for a solution due to the CF write speed and my opinion is that 1100D is not even worth the headaches.

[SpcCb]

(...)
So all you're looking for is that one hexadecimal number to unlock this feature. The question is how do you find it? Read the code, it tells you. Now the hard part is following those instructions and come up with the right address.
(...)

I'm a bad violin player too, but if the goal is to find an hexa number, is it possible to use a sort of brute force method by scanning all numbers with a match (size of the expected informations?) to find the right number? I say that just in case, I imagine all high level devs had already thought about this or maybe there's a issue in this method.

@reddeercity > The problem here, like in many other parts of ML as you know, is that kind of hack is so complex that only a couple of persons can figure how to do it. For other violin players it took years to understand. Although I had 20 years of coding in 5 languages, my eyes blinking face of some parts of ML ^.^

[ShootMeAlready]

What's happening with raw_twk.mo?
It seems that its been left out in the cold.
Its able to display playback 10 bit raw MLV on the LCD.
The only issue I had with the 70D was it only works for the first clip, then if you try to playback
a second clip it hangs. 
Since it plays the first clip, it tells me that's its not initialized correctly for the second clip.
That should be a doable fix.  At least in terms of house of pain threshold.

[budafilms]

Raw Twk.mo on 5d3 1.1.3 , experimental build, show black frames only. In the computer the shoots were ok.