Magic Lantern Forum

Experimental builds (WIP) => Other experimental builds => Topic started by: d on May 22, 2013, 10:58:34 PM

Title: 12-bit (and 10-bit) RAW video development discussion
Post by: d on May 22, 2013, 10:58:34 PM
I've spent the last couple of days working on optimized 14-to-12-bit and 14-to-10-bit RAW conversion routines in native ARM assembly in hopes that 1) in spite of conventional wisdom the camera was in fact fast enough to do rudimentary "compression", and 2) the reduction in bitrate (15% and 29% respectively) would be a step towards making the 5D2 more viable for RAW recording at resolutions higher than 1280x720. There was only a limited amount of testing I could do with my very slow (40MB/s) cards but at 1280x720 on my 5D2 the conversion is fast enough for continuous "compressed" recording. I'd like to make the routines available to some people with faster cards and different cameras to see the extent to which this is viable at higher resolutions (10-bit 1880x1080 should theoretically be possible within the 5D2's apparent 60MB/s hardware limits).

Right now this should be treated as something highly experimental. Rather than posting a pre-built raw_rec.mo file I'm posting an assembled rawc.o file that can be linked with a compiled raw_rec.o into raw_rec.mo. The edits that need to be made to raw_rec.c (keep in mind that I'm working off the May 14th version so things have probably moved around) are:

New external functions:

extern void raw14_to_raw12(void *buffer, int size);
extern void raw14_to_raw10(void *buffer, int size);


New variable:

static int buffer_size_compressed = 0;

Inside the main recording loop after if (saving_buffer_index != capturing_buffer_index), add:

buffer_size_compressed = (buffer_size_used * 12) / 14;
raw14_to_raw12(buffers[saving_buffer_index], buffer_size_used);


for 12-bit, or:

buffer_size_compressed = (buffer_size_used * 10) / 14;
raw14_to_raw10(buffers[saving_buffer_index], buffer_size_used);


for 10-bit. Change every instance of buffer_size_used later in the function to buffer_size_compressed.

The native 14-bit RAW data is packed into halfwords, but it gets repacked into words in the conversion routines, so the C struct for getting at the 12-bit data in C looks like this:

struct raw12_pixblock
{
        unsigned int c_hi: 8;
        unsigned int b: 12;
        unsigned int a: 12;
        unsigned int f_hi: 4;
        unsigned int e: 12;
        unsigned int d: 12;
        unsigned int c_lo: 4;
        unsigned int h: 12;
        unsigned int g: 12;
        unsigned int f_lo: 8;
} __attribute__((packed));


Same deal with 10-bit data, but it's 32 bits shorter (and has 10-bit pixels obviously).

You can download the .o file at http://shoutingroomonly.com/rawc.zip
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 22, 2013, 11:07:53 PM
I wonder if implementing something like this on the 5D3 would make it easier to shoot at higher resolutions in crop mode?

Does anyone have the numbers on what the theoretical MB/frame would be at 3.6k 10bit?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 22, 2013, 11:20:40 PM
This could be the magic bullet to go with the magic lantern  8)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Hazer on May 22, 2013, 11:22:55 PM
5D2 with both Lexar and Komputerbay 1000x cards and happy to test.  However my gcc-fu is a little rusty and I'd be better off trusting you to upload a functioning module.  If you can do that, count me (and probably a bunch of others) in.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 22, 2013, 11:33:53 PM
did you made some speed tests?
as you are working on uncached memories, the performance wouldnt be good enough imo.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: d on May 22, 2013, 11:56:12 PM
Quote from: g3gg0 on May 22, 2013, 11:33:53 PM
did you made some speed tests?
as you are working on uncached memories, the performance wouldnt be good enough imo.

It's fast enough for continuous 12-bit 1280x720 on a 5D2 (6x 24MB buffers), and this is real world performance with actual recordings on an actual camera. I don't have any other cameras (or faster cards) to test with.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 12:11:02 AM
... wow.. cannot believe that.
can you post the asm code?
memcpy using 4 registers could not get more than 2-3 fps when testing around with YUV.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 12:12:47 AM
@g3gg0 - was that bang I just heard you falling off your chair?  ;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 12:16:09 AM
yeah..
well, i implemented an memcpy using LDMIA/STMIA for LV buffer copying and this was a dead end.
so i tried to get EDMAC working.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: ch_d on May 23, 2013, 12:33:44 AM
so d. if this is true and working - u r a hero to me. count me in for testing.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 23, 2013, 12:37:07 AM
Are there any vector-ish OP's on this implementation of ARM that can do shifts on multiple 16-bit integers packed into 64 or 128-bit registers?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: nanomad on May 23, 2013, 12:37:13 AM
I've removed the download. Call it an excess of safety but I don't want random code running on cameras especially of it doesn't have a source code attached.
I've sent the link to g3gg0 so he can review it and re-enable it if it's good
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Greg on May 23, 2013, 12:40:32 AM
Quote from: nanomad on May 23, 2013, 12:37:13 AM
I've removed the download. Call it an excess of safety but I don't want random code running on cameras especially of it doesn't have a source code attached.
I've sent the link to g3gg0 so he can review it and re-enable it if it's good
Good idea, the safety of our cameras is important.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Kunibert on May 23, 2013, 01:05:01 AM
Quote from: Grunf on May 23, 2013, 12:37:07 AM
Are there any vector-ish OP's on this implementation of ARM that can do shifts on multiple 16-bit integers packed into 64 or 128-bit registers?
As far as i know the Digic 5+ is a modification of this core:
http://www.arm.com/products/processors/classic/arm9/arm946.php?tab=Specifications
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Alia5 on May 23, 2013, 01:13:08 AM
Thanks! Now i don't trust the Devs anymore when they say that somethign would be impossible!
No serious, awesome job!
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 01:14:29 AM
here the removed link:
http://shoutingroomonly.com/rawc.zip

the CPU in digic is an ARM946, thats right.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 01:29:03 AM
This would be a HUGE win for SD cameras that can't record even close to CF.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Audionut on May 23, 2013, 01:33:22 AM
And those of us looking for highest resolution at 48-60fps.

The bottom 2 bits are pretty much just noise anyway.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 01:33:52 AM
Quote from: 1% on May 23, 2013, 01:29:03 AM
This would be a HUGE win for SD cameras that can't record even close to CF.

Are you gonna try with TL2 or is that a stupid question?  ;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 01:38:12 AM
I'll try to download it and just run it I guess... I'm not worried about this dude putting his blood sweat and tears in just to break my camera.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: silvertonesx24 on May 23, 2013, 01:42:14 AM
Am I correct in assuming that 10-bit and 12-bit will look no different than 14-bit (in real-world application of this camera)?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 01:46:48 AM
I've linked as follows:

MODULE_OBJS=raw_rec.o rawc.o


When I boot up I get this laughing skull and then camera starts smoking.....


Actually not sure if its working, will have to open the raws it produced. Not sure if I'm getting speed increase... looks like its doing something. Maybe have to add the modifications?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Alia5 on May 23, 2013, 01:48:30 AM
Quote from: silvertonesx24 on May 23, 2013, 01:42:14 AM
Am I correct in assuming that 10-bit and 12-bit will look no different than 14-bit (in real-world application of this camera)?

Reading this: http://francoismalan.com/2011/10/raw-12bit-or-14bit-lossy-or-lossless/
... i guess that it would only make a very very slight difference!

If this is really working... I just dont have words for how awesome this is...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 01:48:49 AM
Quote from: 1% on May 23, 2013, 01:38:12 AM
I'll try to download it and just run it I guess... I'm not worried about this dude putting his blood sweat and tears in just to break my camera.

I'll be a guinea pig if needed. Got my 50d yesterday and I'm willing to risk my 600 for something important like this. (Disclaimer. OK, the wife thinks the 600 is now hers but...)  ;)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 01:50:15 AM
Quote from: 1% on May 23, 2013, 01:46:48 AM
I've linked as follows:

MODULE_OBJS=raw_rec.o rawc.o


When I boot up I get this laughing skull and then camera starts smoking.....


Actually not sure if its working, will have to open the raws it produced. Not sure if I'm getting speed increase... looks like its doing something. Maybe have to add the modifications?

I was half-expecting you to have been rickrolled by it :D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 01:52:40 AM
Maybe he is from Canon giving us hints?  ???
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 01:55:53 AM
Its up in the 6D repo... see if it does anything.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: menoc on May 23, 2013, 01:58:48 AM
Quote from: DTSET123 on May 23, 2013, 01:52:40 AM
Maybe he is from Canon giving us hints?  ???

Please don't spook him . . .
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 01:59:35 AM
QuoteIts up in the 6D repo... see if it does anything.
im scared lol
Title: Re: 12-bit (and 10-bit) RAW video
Post by: RenatoPhoto on May 23, 2013, 02:01:37 AM
Probably Canons way of bricking all our cams!! 8)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:02:42 AM
Its will smith giving us all a virus. But must be slow acting cuz my cam still works :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 02:05:53 AM
is it faster tho?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: N/A on May 23, 2013, 02:06:30 AM
What the hell, t3i's are dirt cheap now, I'll give it a run if someone hooks a t3i link up.

Title: Re: 12-bit (and 10-bit) RAW video
Post by: RenatoPhoto on May 23, 2013, 02:06:40 AM
Quote from: 1% on May 23, 2013, 02:02:42 AM
Its will smith giving us all a virus. But must be slow acting cuz my cam still works :)

I am dying to know ...  I will write a How to (Brick)  Manual...   :o
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:07:19 AM
Try it... I can't tell as latest modules do like 30MB/s for me. I guess you'd have to see if you're getting any additional frames.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 02:14:10 AM
did not do anything for me... same speed
Title: Re: 12-bit (and 10-bit) RAW video
Post by: silvertonesx24 on May 23, 2013, 02:16:51 AM
Quote from: Alia5 on May 23, 2013, 01:48:30 AM
Reading this: http://francoismalan.com/2011/10/raw-12bit-or-14bit-lossy-or-lossless/
... i guess that it would only make a very very slight difference!

If this is really working... I just dont have words for how awesome this is...

Absolutely! I just used 5D raw for a paying client today (stupid, I know, but it ran without a hitch) and the only pain really was swapping cards around. Need to load up on a few 64gbs. Any space or efficiency savings are welcomed.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Audionut on May 23, 2013, 02:17:19 AM
Quote from: DTSET123 on May 23, 2013, 02:14:10 AM
did not do anything for me... same speed

Are the DNG frames in the RAW video smaller then without this mod?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 02:20:51 AM
just linking wont work. the raw rec module must call the routines to reduce the bit depth.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:21:38 AM
Ok, will edit then ... as per first post I guess?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 02:22:58 AM
QuoteAre the DNG frames in the RAW video smaller then without this mod?

nope its the same
g3gg0 did it work for you?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 02:23:32 AM
@1%: yep. (of course linking into the raw rec module makes more sense than in ML Core)

@DTSET123: nope, as i am busy with some other stuff atm
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 02:25:01 AM
 :-[
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 02:29:52 AM
I think he was answering 1%  ;)

We'll need to update raw_get_pixel() in raw2dng with the 12/10bit format, maybe store bit-depth in raw header and process raw accordingly?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:30:16 AM
Well its a bit fucked because new code uses a struct and not buffer_size_used, etc.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Greg on May 23, 2013, 02:40:24 AM
Here is the code of buffer_size_used - https://bitbucket.org/hudson/magic-lantern/src/84f657f209eb5b599ec32bd85718c9951c25b720/modules/raw_rec/raw_rec.c?at=unified
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:43:14 AM
I did this:



        if (saving_buffer_index != capturing_buffer_index)
        {
    if (!t0) t0 = get_ms_clock_value();
            void* ptr = buffers[saving_buffer_index].ptr;
            int size_used = buffers[saving_buffer_index].used;
buffer_size_compressed = (size_used * 12) / 14;
raw14_to_raw12(ptr, size_used);     
int r = FIO_WriteFile(f, ptr, buffer_size_compressed);
            if (r != buffer_size_compressed) goto abort;
            written += size_used;
//~ writtenL = (uint32_t) written;
            saving_buffer_index = mod(saving_buffer_index + 1, buffer_count);
        }



I think it needs more changes...says its writing at 33MB/s, lol

It compiled... will have to make new bin
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Audionut on May 23, 2013, 02:51:25 AM
Is the code just truncating bits, or is it compressing them into the remaining?

edit:  The OP states that the bits are being compressed.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:52:07 AM
I guess I'll find out?

http://pastebin.com/PviXPWNZ

It is saving files... I don't see any increase in speed but maybe a few extra frames... I think the change is still incomplete.

Pink garbage when I open them :(

Second shot had image cut up into squares like lv_rec used to do when it was out of sync.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 02:56:29 AM
Did you modify raw2dng ?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:59:37 AM
Nope.. feel free to try, I'm not the best coder... guy should have really provided source... may 14th is hella old and slow.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Greg on May 23, 2013, 03:02:45 AM
Quote from: 1% on May 23, 2013, 02:52:07 AM
I guess I'll find out?

http://pastebin.com/PviXPWNZ

It is saving files... I don't see any increase in speed but maybe a few extra frames... I think the change is still incomplete.

Pink garbage when I open them :(

Second shot had image cut up into squares like lv_rec used to do when it was out of sync.

On the 500D:
tcc : error: undefined symbol 'raw14_to_raw12'   :o
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 03:04:18 AM
Did oyu add the struct to raw.h? also link the .o in the makefile.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 03:08:55 AM
Quote from: 1% on May 23, 2013, 02:59:37 AM
Nope.. feel free to try, I'm not the best coder... guy should have really provided source... may 14th is hella old and slow.

I don't have all the required libs installed for this (And i really should be in bed right now), but in raw2dng.c



struct raw12_pixblock
{
        unsigned int c_hi: 8;
        unsigned int b: 12;
        unsigned int a: 12;
        unsigned int f_hi: 4;
        unsigned int e: 12;
        unsigned int d: 12;
        unsigned int c_lo: 4;
        unsigned int h: 12;
        unsigned int g: 12;
        unsigned int f_lo: 8;
} __attribute__((packed));

int raw_get_pixel(int x, int y) {
    struct raw12_pixblock * p = (void*)raw_info.buffer + y * raw_info.pitch + (x/8)*12;
    switch (x%8) {
        case 0: return p->a;
        case 1: return p->b;
        case 2: return p->c_lo | (p->c_hi << 8);
        case 3: return p->d;
        case 4: return p->e;
        case 5: return p->f_lo | (p->f_hi << 8);
        case 6: return p->g;
        case 7: return p->h;
    }
    return p->a;
}


If anyone wants to try  ;)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 03:14:04 AM
Gonna try it on the file I made already
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Greg on May 23, 2013, 03:17:25 AM
Quote from: 1% on May 23, 2013, 03:04:18 AM
Did oyu add the struct to raw.h?
Yes  ;)
Quote from: 1% on May 23, 2013, 03:04:18 AM
also link the .o in the makefile.
No  :o
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 03:18:47 AM
First one came out as pink shit with modified raw2dng.

Some of the frame pieces do come out... so it is kinda working.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: d on May 23, 2013, 04:56:14 AM
It looks like I accidentally uploaded an old, broken version of rawc.o. The correct, working version is now at the same URL. If you downloaded it before this reply went up you will need to re-download and and re-link it. Sorry about that.

Also, I noticed a mistake in the example raw_get_pixel posted earlier. The correct shifts for pixels C and F are:

raw->c_lo | (raw->c_hi << 4);
raw->f_lo | (raw->f_hi << 8);
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tin2tin on May 23, 2013, 10:19:23 AM
This is just... wow! The perspective of this is just incredible. Thank you for investing your time in this.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 23, 2013, 10:26:21 AM
(http://a1ex.magiclantern.fm/bleeding-edge/raw/bench14to12-playback.png) (http://a1ex.magiclantern.fm/bleeding-edge/raw/bench14to12-movie24p.png)

Impressive.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tin2tin on May 23, 2013, 11:26:52 AM
What is the 12/10 bit speed of 720p? Would this be usable for SD cards?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 11:30:04 AM
It looks about 25% reduced :) cool

@a1ex - have you shot anything with compression? How's it looking?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 23, 2013, 11:30:48 AM
Quote from: tin2tin on May 23, 2013, 11:26:52 AM
What is the 12/10 bit speed of 720p? Would this be usable for SD cards?

In theory, SD cards can give 22MB/s or 961194 bytes per frame. 1280x600 in 10 bits gives 960000 bytes.

In practice - we need a working build to test it (was the latest one 1% posted - the working one?)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: squig on May 23, 2013, 11:32:29 AM
Settle, settle, this development is only like 13 hours old, the little swimmers are still trying to find their way to the egg.  ::)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 23, 2013, 11:37:02 AM
I didn't shoot anything with it, only ran the benchmark. Right now I have no idea whether the algorithm is correct. You need an additional task to prepare the buffers, so the raw processing should become a pipeline, like this:
- source raw to cropped raw: via EDMAC, triggered from LiveView (EVF) task
- cropped raw (14-bit) to 12-bit raw
- file write task
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tihon on May 23, 2013, 12:07:13 PM
Quote from: vicnaum on May 23, 2013, 11:30:48 AM
In theory, SD cards can give 22MB/s or 961194 bytes per frame. 1280x600 in 10 bits gives 960000 bytes.

In practice - we need a working build to test it (was the latest one 1% posted - the working one?)
That so you can write 2.5k with 44 MB/s card? (10bit)) It can give a new live to mark 2

I think that it can be possible to wite a  2350x1100px on 5dm2 in 1:1 mode  with 12bit
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 23, 2013, 12:30:47 PM
Quote from: tihon on May 23, 2013, 12:07:13 PM
That so you can write 2.5k with 44 MB/s card? (10bit)) It can give a new live to mark 2
I think that it can be possible to wite a  2350x1100px on 5dm2 in 1:1 mode  with 12bit

44 MB/s is 1.922.389 bytes per frame @24fps, or 1.537.911 pixels in 10bit - that makes something around 1660x930px frame.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: t2scorp on May 23, 2013, 12:42:29 PM
Quote from: vicnaum on May 23, 2013, 12:30:47 PM
44 MB/s is 1.922.389 bytes per frame @24fps, or 1.537.911 pixels in 10bit - that makes something around 1660x930px frame.

Meaning that resolutions up to 2240x1260 @ 24fps at 83MB/s and 1920x1080 @24fps at 62MB/s should be possible?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 23, 2013, 12:53:34 PM
No, according to the benchmarks.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 12:54:15 PM
This is great news for the 50d. It should be able to record all it's available frame sizes unrestricted  :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: savale on May 23, 2013, 12:59:36 PM
Cool stuff! Don't know how to understand the benchmarks. Is that the "needed" bandwidth because of the less needed bits, or is it the actual bandwidth because of the extra calculations needed? (in that case the bandwidth is too low?)

I really want to look at the raw14_to_raw12 and raw14_to_raw10 methods, where can I find them?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: nanomad on May 23, 2013, 01:00:34 PM
They are inside the .o file d provided. In ARM ASM code once you de-compile it
Title: Re: 12-bit (and 10-bit) RAW video
Post by: eyeland on May 23, 2013, 01:24:12 PM
This is one of the threads that will wear down my F5 key ...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 01:29:46 PM
Do we know if this could be applied to higher resolutions than 1080p, or would the camera not be able to handle it?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AdamTheBoy on May 23, 2013, 03:21:06 PM
I'm pretty much a lamen, but it looks like you guys are figuring out how to retain the quality of RAW but save on space? This is such wonderful news.  So this means we may have longer record times and if the frames are a smaller size, fewer dropped frames?

Thanks to everyone involved, it's always exciting to see what people are achieving here. 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 03:46:15 PM
Have a better patch to raw_rec.mo? I think mine was a little lacking. 

IMO, should have an option in raw_rec 10/12/etc bit. So cameras like 600D can do 10 bit and push the res up to somewhere acceptable while 6D/5d3/etc can still do the higher ones, etc.



Title: Re: 12-bit (and 10-bit) RAW video
Post by: xxxbugxxxx on May 23, 2013, 03:51:49 PM
Just curious, does Canon hard code 14bit raw processing? Is it possible to change bitdepth in some native way to change the RAW AD converter to do 12bit or even 10 bit? I know on Nikon D600 you can choose between 14bit or 12bit. Wonder if that can be easier?

Thank you for the great work.

Title: Re: 12-bit (and 10-bit) RAW video
Post by: KenFTW on May 23, 2013, 03:57:04 PM
Quote from: eyeland on May 23, 2013, 01:24:12 PM
This is one of the threads that will wear down my F5 key ...

this and about three others... monumental stuff people, the rules have been changed.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 04:00:33 PM
The big question is... 

Is it going to be possible to shoot 3.6k @ 24fps, 10bit on the 5D3 without dropping frames?

If the percentages listed by d are correct, that would put the 3592 x 1320 at about 5MB/frame with a throughput of ~120MB/sec...  a bit much for most cards, but possible?

Is there anyway to drop it even lower to 8bit?  *I'd rather have color information... but if the resolution can be achieved it would certainly be a bonus.  :-)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: coutts on May 23, 2013, 04:02:33 PM
Quote from: xxxbugxxxx on May 23, 2013, 03:51:49 PM
Just curious, does Canon hard code 14bit raw processing? Is it possible to change bitdepth in some native way to change the RAW AD converter to do 12bit or even 10 bit? I know on Nikon D600 you can choose between 14bit or 12bit. Wonder if that can be easier?

Thank you for the great work.

There's a factory function named "FA_SetCRawBitNum", located at 0xFF47C0D8 in the 6D.113, seems to just store a number at 0x82734 . It can be called via the Call function, maybe this will work?

call( "FA_SetCRawBitNum", 10 );
Title: Re: 12-bit (and 10-bit) RAW video
Post by: N/A on May 23, 2013, 04:04:28 PM
Quote from: 1% on May 23, 2013, 03:46:15 PM
So cameras like 600D can do 10 bit and push the res up to somewhere acceptable while 6D/5d3/etc can still do the higher ones, etc.
600D would make a perfect b-roll if we could match resolution
Title: Re: 12-bit (and 10-bit) RAW video
Post by: bumkicho on May 23, 2013, 04:10:42 PM
This is such an important and exciting development!!
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 04:13:44 PM
I messed with that function... but was trying for H264.. I wonder if it cuts bits out of the "debug" raw we're using.

Called it and shot 2 silent pics... they are the same size :(
Title: Re: 12-bit (and 10-bit) RAW video
Post by: coutts on May 23, 2013, 04:15:33 PM
Quote from: 1% on May 23, 2013, 04:13:44 PM
I messed with that function... but was trying for H264.. I wonder if it cuts bits out of the "debug" raw we're using.
at first test with 5d2, the location is empty (0), maybe i need to call lv_save_raw first.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 04:21:13 PM
Location changes to a (10) when I call it.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 06:14:06 PM
Finally got everything running, but the output seems shifted. Can't tell yet if it's my modified raw_rec or modified raw2dng.
The corrupted dngs are the correct size though.

Also, I didn't write more frames at high resolutions compared to 14-bit, so this probably wouldn't help getting higher resolutions.
The only interest would be in saving card space  :-\
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 06:16:32 PM
Shifted.. like all tesselated? Post an example?

If the files are smaller at all then you'll get a few more frames... right now maybe CPU usage > size savings?  I think it had to go into a separate task like a1ex said to not interfere with writing.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 06:16:58 PM
Mayo: what camera are you on? Settings?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 06:44:38 PM
I'm on 6D, I shoot in photo mode, global draw off.

But wait a minute... I noticed raw2dng just saves the raw frames, but nowhere saves a bitdepth or linewidth...
So how could DNG software (like ACR) read this properly if they don't know it's 12-bit?
Does the DNG format support anything other than 14-bit?
If it does not, we need to convert 12-bit back to 14-bit to export as dng. (I'm really not sure I have the skills to do this properly).
Reading chdk-dng.c now...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 06:52:57 PM
I think DNG supports all bit depths but the DNG meta data will say it X bit.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 06:55:18 PM
Has anyone tested this on a 5D3 yet with the fastest card?

*I'm excited! 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 06:57:34 PM
We're barely getting it running :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: gebo on May 23, 2013, 07:05:51 PM
I'm checking these forums daily to see how things are developing. Congratulations, you're doing the "impossible"!
I'm so excited I even have trouble sleeping... :o
Title: Re: 12-bit (and 10-bit) RAW video
Post by: seanturco on May 23, 2013, 07:13:59 PM
I'm amazed by the work. I wish to be a developers one days :p
But I'm just a photographer  ;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: JohnBarlow on May 23, 2013, 07:14:06 PM
Maybe silly questions--->

Is this sRGB or ADobe RGB?

If the former will it cause a processor drain?

Is Adobe RGB possible seeing that we are dealing with a stream of stills?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 07:18:56 PM
Both? Neither? Whichever one you pick?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 07:23:20 PM
Quote from: 1% on May 23, 2013, 06:52:57 PM
I think DNG supports all bit depths but the DNG meta data will say it X bit.

Right, I found it, had to overwrite raw_info.bits_per_pixel=12; in raw2dng.

Now the image is good, only the colors are messed up.

Looks like this in irfan view (doesn't open in ACR):

(http://i.imgur.com/lsMeEFO.jpg)

(It's the bottom of a (very dirty) window)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: JohnBarlow on May 23, 2013, 07:24:26 PM
Do you mean you can have either?


Quote from: 1% on May 23, 2013, 07:18:56 PM
Both? Neither? Whichever one you pick?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 07:28:02 PM
Quote from: Mayo on May 23, 2013, 07:23:20 PM
Right, I found it, had to overwrite raw_info.bits_per_pixel=12; in raw2dng.

Now the image is good, only the colors are messed up.

Looks like this in irfan view (doesn't open in ACR):

(http://i.imgur.com/lsMeEFO.jpg)

(It's the bottom of a (very dirty) window)

Too bad the 'Predator' franchise is finished... :-/ 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 07:30:26 PM
Black level is wrong.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 07:30:46 PM
If only d posted his post-process workflow, this would be easier :P
Title: Re: 12-bit (and 10-bit) RAW video
Post by: IliasG on May 23, 2013, 07:33:19 PM
Quote from: Mayo on May 23, 2013, 07:23:20 PM
Right, I found it, had to overwrite raw_info.bits_per_pixel=12; in raw2dng.

Now the image is good, only the colors are messed up.

Looks like this in irfan view (doesn't open in ACR):

http://i.imgur.com/lsMeEFO.jpg

(It's the bottom of a (very dirty) window)

You also have to adapt the Black Level & White Level to 12 bit ..
Title: Re: 12-bit (and 10-bit) RAW video
Post by: chmee on May 23, 2013, 07:36:30 PM
Open the DNG-File in a Hex-Editor (recommending HxD).
Search for the BitsPerSample-Tag 0102 - think it will be on Position ~1BC
(there will be two entries, because the thumbnail needs this tag as well)
0201 0300 0100 0000 0E00 0000
0102 (or in little-endian 0201) and later on there is 000E (14Bit) -> 0E00. change that.

You find the Whitelevel and blacklevel with the tag-id's C61A and C61D, some bytes after the BitPerSampleTag. Example:

1AC6 0400 0100 0000 F807 0000 <-->07F8 = 2040
1DC6 0400 0100 0000 2E2D 0000 <--> 2D2E = 11566

Lower both. For Example Blacklevel to 1000 and Whitelevel to 4000.
(http://dl.phreekz.de/DNG_Example.jpg)

mfg chmee
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 23, 2013, 07:38:29 PM
Quote from: JohnBarlow on May 23, 2013, 07:14:06 PM
Is this sRGB or ADobe RGB?

RAW is beyond these. It becomes sRGB or Adobe RGB when exporting (baking) from bayer raw to RGB in any Raw Converter (ACR, Lightroom, etc).
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 07:54:36 PM
Modified reverse_bytes_order() for words instead of half-words in chdk-dng.c ,it's a bit better now.

static void FAST reverse_bytes_order(char* buf, int count)
{
long* buf32 = (long*) buf;
    int i;
    for (i = 0; i < count/4; i++)
    {
        long x = buf32[i];
buf[4*i+3] = x;
        buf[4*i+2] = x >> 8;
        buf[4*i+1] = x >> 16;
        buf[4*i] = x >>24;
    }
/*
    short* buf16 = (short*) buf;
    int i;
    for (i = 0; i < count/2; i++)
    {
        short x = buf16[i];
        buf[2*i+1] = x;
        buf[2*i] = x >> 8;
    }
*/
}


Quote from: chmee on May 23, 2013, 07:36:30 PM
Open the DNG-File in a Hex-Editor (recommending HxD).
Search for the BitsPerSample-Tag 0102 - think it will be on Position ~1BC
(there will be two entries, because the thumbnail needs this tag as well)
0201 0300 0100 0000 0E00 0000
0102 (or in little-endian 0201) and later on there is 000E (14Bit) -> 0E00. change that.

You find the Whitelevel and blacklevel with the tag-id's C61A and C61D, some bytes after the BitPerSampleTag. Example:

1AC6 0400 0100 0000 F807 0000 <-->07F8 = 2040
1DC6 0400 0100 0000 2E2D 0000 <--> 2D2E = 11566

Lower both. For Example Blacklevel to 1000 and Whitelevel to 4000.
(http://dl.phreekz.de/DNG_Example.jpg)

mfg chmee

Will try this now.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: chmee on May 23, 2013, 07:58:38 PM
ah. Forgot to say. AFAIK the Preview-Jpg is hardcoded on 14Bit-Transcode. Rename the .dng to .tif and look what the viewer is throwing out. 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 07:59:21 PM
Yep and then post a workable source for the mods so we can try it out in one fell swoop.

If its good will need depth selection for raw2dng/rawrec... I think you could even do silent pics this way too. 8-14 bit.

Real compression I think manipulates the whole image.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 08:06:50 PM
Almost there.

(http://i.imgur.com/f4D8A8v.jpg)

Now to find

1). where the black/white levels are wrongly set in chdk-dng.c
2). why the DNG file won't open in ACR
Title: Re: 12-bit (and 10-bit) RAW video
Post by: JohnBarlow on May 23, 2013, 08:07:55 PM
Thanks!

Quote from: vicnaum on May 23, 2013, 07:38:29 PM
RAW is beyond these. It becomes sRGB or Adobe RGB when exporting (baking) from bayer raw to RGB in any Raw Converter (ACR, Lightroom, etc).
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 08:25:29 PM
Wow, simply lowered black and white (*0.25f) in raw2dng before it exports

raw_info.bits_per_pixel=12;
raw_info.white_level *= 0.25f;
    raw_info.black_level *= 0.25f;


And now the file opens ACR  ;D

(http://i.imgur.com/3DhYy4u.jpg)

I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 08:45:24 PM
So what does mean for getting it up and running on the 5D3?  :-)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: bumkicho on May 23, 2013, 08:46:40 PM
Quote from: Mayo on May 23, 2013, 08:25:29 PM
I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Does it mean 12bit or 10bit raw is coming sometime very soon?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 09:03:15 PM
Well, I didn't touch 10-bit, but it should be fairly easy to add.

Here are the modified src files: https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing (https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing)

12-bit is currently hardcoded with the variable:

static int raw_bitdepth = 12;

...which should be set via a RAW menu option and exported in the RAW footer to be retrieved by raw2dng and processed accordingly.

Title: Re: 12-bit (and 10-bit) RAW video
Post by: scrax on May 23, 2013, 09:15:51 PM
Quote from: Mayo on May 23, 2013, 09:03:15 PM
Well, I didn't touch 10-bit, but it should be fairly easy to add.

Here are the modified src files: https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing (https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing)

12-bit is currently hardcoded with the variable:

static int raw_bitdepth = 12;

...which should be set via a RAW menu option and exported in the RAW footer to be retrieved by raw2dng and processed accordingly.

why don't you open a fork on bitbucket instead of google drive? It will be easier to follow (and learn from) your changes
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 09:22:55 PM
Quote from: scrax on May 23, 2013, 09:15:51 PM
why don't you open a fork on bitbucket instead of google drive? It will be easier to follow (and learn from) your changes

I've been a lone coder for 4-5 years now, and became a bit oblivious to source control :P
I had a look at the tragic lantern bitbucket (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all)) but i'm terrified to fuck up things for you guys... If you can quickly teach me how to commit my changes, i will gladly do it.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 23, 2013, 09:38:40 PM
Quote from: Kunibert on May 23, 2013, 01:05:01 AM
As far as i know the Digic 5+ is a modification of this core:
http://www.arm.com/products/processors/classic/arm9/arm946.php?tab=Specifications

Well as long as Arm 946 supports NEON, there is a neat quad-word SIMD shift instruction that is potentially able to do shift to 10 bits for one whole RGB pixel in one instruction:

http://infocenter.arm.com/help/topic/com.arm.doc.dui0489c/CIHEDCEG.html

In that case, there is even more speed to be gained. If not, we need to check into "Enhanced DSP instructions" which seems to be onboard on 946 for sure:
http://infocenter.arm.com/help/topic/com.arm.doc.dvi0022a/DVI0022A.pdf
Title: Re: 12-bit (and 10-bit) RAW video
Post by: bumkicho on May 23, 2013, 09:43:32 PM
Quote from: Mayo on May 23, 2013, 09:03:15 PM
Well, I didn't touch 10-bit, but it should be fairly easy to add.

Here are the modified src files: https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing (https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing)

12-bit is currently hardcoded with the variable:

static int raw_bitdepth = 12;

...which should be set via a RAW menu option and exported in the RAW footer to be retrieved by raw2dng and processed accordingly.

I would try 12bit to see if it works properly first. Then we can move onto adding 10bit. Well, I say "we" as if I am actually contributing anything to this development. Ha ha. Sorry. I meant I hoped you could.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: jordancolburn on May 23, 2013, 09:57:26 PM
Quote from: Mayo on May 23, 2013, 09:22:55 PM
I've been a lone coder for 4-5 years now, and became a bit oblivious to source control :P
I had a look at the tragic lantern bitbucket (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all)) but i'm terrified to fuck up things for you guys... If you can quickly teach me how to commit my changes, i will gladly do it.
Here's a really good Mercurial tutorial http://hginit.com/
I think most of the devs use it and I use it at work and for my own projects like my resume in LaTex and others.  Merging is a little strange at first, but it will make your life SO much easier in the long run.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tin2tin on May 23, 2013, 10:11:32 PM
Quote from: vicnaum on May 23, 2013, 11:30:48 AM
In theory, SD cards can give 22MB/s or 961194 bytes per frame. 1280x600 in 10 bits gives 960000 bytes.

In practice - we need a working build to test it (was the latest one 1% posted - the working one?)
Can't wait to hear if 1280x600 in 10 bits really is possible, so Cameras with SD cards only can benefit from the latest RAW magic.  :P 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 10:18:23 PM
Quote from: jordancolburn on May 23, 2013, 09:57:26 PM
Here's a really good Mercurial tutorial http://hginit.com/
I think most of the devs use it and I use it at work and for my own projects like my resume in LaTex and others.  Merging is a little strange at first, but it will make your life SO much easier in the long run.

Well, that looks nice and easy :P
Seriously though, the changes are really minor: just do a find 'raw_bitdepth' and all my changes are there (except for reverse_bytes_order() which is terribly hardcoded). If someone who's already setup can look through them it would be much safer than me fucking things up.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: scrax on May 23, 2013, 10:25:37 PM
Quote from: Mayo on May 23, 2013, 10:18:23 PM
Well, that looks nice and easy :P
Seriously though, the changes are really minor: just do a find 'raw_bitdepth' and all my changes are there (except for reverse_bytes_order() which is terribly hardcoded). If someone who's already setup can look through them it would be much safer than me fucking things up.

I think that quick way to share the code without any risk is, once installed hg, register yourself on bitbucket, go to magic lantern source and fork the repo, you will have your fork on bitbucket pubblic so any change you made will not screw anything in the ML source.
Once done the fork you have to download it on your pc (I use a GUI for managing those forks, SourceTree), make your changes and with "hg pull" update it on bitbucket.
When you are done with all your work you can submit from the fork page on bitbucket a pull request to the ML source, that way it will be reviewed before merging it.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 11:16:14 PM
Quote from: Grunf on May 23, 2013, 09:38:40 PMIf not, we need to check into "Enhanced DSP instructions" which seems to be onboard on 946 for sure:

this is for ARM946E-S vs. ARM946E which we have :)

but i didnt verify
Title: Re: 12-bit (and 10-bit) RAW video
Post by: trsaunders on May 23, 2013, 11:59:58 PM
I'm wondering: how is the RAW data (on the camera, before it is copied) aligned? is it 14bits with two zeros packed in 16 bits, or is each pixel packed tightly together?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 12:07:04 AM
Quote from: trsaunders on May 23, 2013, 11:59:58 PM
I'm wondering: how is the RAW data (on the camera, before it is copied) aligned? is it 14bits with two zeros packed in 16 bits, or is each pixel packed tightly together?

It's tight:

/**
* 14-bit encoding:

hi          lo
aaaaaaaaaaaaaabb
bbbbbbbbbbbbcccc
ccccccccccdddddd
ddddddddeeeeeeee
eeeeeeffffffffff
ffffgggggggggggg
gghhhhhhhhhhhhhh
*/



Quote from: scrax on May 23, 2013, 10:25:37 PM
I think that quick way to share the code without any risk is, once installed hg, register yourself on bitbucket, go to magic lantern source and fork the repo, you will have your fork on bitbucket pubblic so any change you made will not screw anything in the ML source.
Once done the fork you have to download it on your pc (I use a GUI for managing those forks, SourceTree), make your changes and with "hg pull" update it on bitbucket.
When you are done with all your work you can submit from the fork page on bitbucket a pull request to the ML source, that way it will be reviewed before merging it.

Thank you. I did all this, let's see how it goes.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: SonicVibe on May 24, 2013, 12:50:52 AM
Mayo, thanks for your work on this.

I think they meant for you to fork the magic lantern repo and not tragic lantern, which is already a fork of magic lantern.

https://bitbucket.org/hudson/magic-lantern
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 02:51:12 AM
Fork whichever... I can give you write permisison too if you want.

For 6D I wouldn't fork the 'main repo as its not updated.

Gonna start merging tomorrow morning or later on tonight...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: hirethestache on May 24, 2013, 04:22:10 AM
Im really rooting for the development of this on my 5D3, as I just purchased two more 5D3's and cant really afford a 1000x Lexar :-p
Title: Re: 12-bit (and 10-bit) RAW video
Post by: IliasG on May 24, 2013, 04:52:22 AM
Quote from: Mayo on May 23, 2013, 08:25:29 PM
Wow, simply lowered black and white (*0.25f) in raw2dng before it exports

raw_info.bits_per_pixel=12;
raw_info.white_level *= 0.25f;
    raw_info.black_level *= 0.25f;


And now the file opens ACR  ;D

(http://i.imgur.com/3DhYy4u.jpg)

I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Nice, thanks.

Is this posterisation down there ??.

Can you upload your dng file ??.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 06:57:12 AM
For some reason 12 bit stops faster than 14 bit right now.. 1200x676 I have to stop recording in 14bit but in 12 bit only 144 frames. Write speed drops to 26MB/s if LV is on and pretty much the same (28-30) if you hack it off. Global draw off seemed to help a tiny bit too. Still must be something stopping it early.

Or maybe 10 bit would be better?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 24, 2013, 07:57:16 AM
Quote from: 1% on May 24, 2013, 06:57:12 AM
For some reason 12 bit stops faster than 14 bit right now.. 1200x676 I have to stop recording in 14bit but in 12 bit only 144 frames. Write speed drops to 26MB/s if LV is on and pretty much the same (28-30) if you hack it off. Global draw off seemed to help a tiny bit too. Still must be something stopping it early.

Or maybe 10 bit would be better?

Maybe good thought to try extreme values, like 2-4 bits, to find out what stops it - bit processing, or card write. And if it's okay - then slowly raise it until the problem appears. Or don't write to card at all and see how much the processing only gives (something similar to a1ex benchmark, but with real data).
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 08:12:57 AM
It looks like a bug.. like its not clearing the buffers? 114 frames is about the size of memory or something?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a.d. on May 24, 2013, 09:16:47 AM
The same here on 5D2! This is the end at 114  ;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 24, 2013, 10:04:03 AM
Quote from: Mayo on May 23, 2013, 07:54:36 PM
Modified reverse_bytes_order() for words instead of half-words in chdk-dng.c ,it's a bit better now.

static void FAST reverse_bytes_order(char* buf, int count)
{
long* buf32 = (long*) buf;
    int i;
    for (i = 0; i < count/4; i++)
    {
        long x = buf32[i];
buf[4*i+3] = x;
        buf[4*i+2] = x >> 8;
        buf[4*i+1] = x >> 16;
        buf[4*i] = x >>24;
    }
/*
    short* buf16 = (short*) buf;
    int i;
    for (i = 0; i < count/2; i++)
    {
        short x = buf16[i];
        buf[2*i+1] = x;
        buf[2*i] = x >> 8;
    }
*/
}


Will try this now.

Hi!


    for (i = 0; i < count/2; i++)

What about pre-calculating "count/2" into it's own variable and use that variable instead in your "for"-loop, otherwise it will waste precious CPU cycles evaluating "count/2" for each loop (giving same results over and over again)?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 10:08:04 AM
You mean... raw2dng feels kinda slow?

;)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 24, 2013, 10:11:11 AM
Quote from: a1ex on May 24, 2013, 10:08:04 AM
You mean... raw2dng feels kinda slow?

;)

Ah, I didn't know which application this snippet of code belonged to. I just reacted when i saw the code :D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 10:15:06 AM
I don't know programming much. But it sounds to me that the program itself stalled itself and exit somewhere(or suspended by the OS), my wild guess.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 10:21:58 AM
FYI, gcc already moves this kind of computations outside loops.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 10:58:34 AM
buffer_size_compressed = (buffer_size_used * 12) / 14;

I don't know what is this, but if it changes to:

buffer_size_compressed = buffer_size_used  / 14;
buffer_size_compressed = buffer_size_compressed * 12;

It might have higher accuracy, I heard that some CPU cannot multiply too many times, or very soon it will have accuracy problem.

HA, I have been messing around. I quit.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 24, 2013, 11:46:57 AM
Quote from: g3gg0 on May 23, 2013, 11:16:14 PM
this is for ARM946E-S vs. ARM946E which we have :)

but i didnt verify

Ok! That's a pity :(

My understanding is that Canon licensed  basic ARM946E core and then added some of their ow stuff to it. do you know if there is any way to check for procedure/op -calls done from, for example H264 encoder? I bet it has $hitload of DSP/SIMD op's hardcoded on die in order to do real-time H264 encoding, some of those could be "re-used" for 4 x small int bitshift? (Or is raw data offloaded to buffer, to be handled by some sort of co-processor involved)

Pity "d" didn't make rawc.o ASM available. It would be very interesting to take a peek...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: nanomad on May 24, 2013, 12:59:31 PM
It's available, just de-compile it  ::)

H264 encoding is done by an external chip, so there's no need for fancy SIMD instructions sets on the ARM core :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 02:50:04 PM
Quote from: mucher on May 24, 2013, 10:58:34 AM
buffer_size_compressed = (buffer_size_used * 12) / 14;

I don't know what is this, but if it changes to:

buffer_size_compressed = buffer_size_used  / 14;
buffer_size_compressed = buffer_size_compressed * 12;

It might have higher accuracy, I heard that some CPU cannot multiply too many times, or very soon it will have accuracy problem.

HA, I have been messing around. I quit.

Or simply changes to:
buffer_size_compressed = (buffer_size_used / 14) *12;

Because if buffer_size_used is something big, *12 might make it very big, making it difficult to be divided by 14, so I think that it might be better to divide first, and then multiply. But this trick might not work anyway.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: nanomad on May 24, 2013, 04:26:56 PM
Don't do the compiler's job :P
He's usually good at it
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 04:31:36 PM
Its merged in and can be switched between... why does it stop tho at every res?

https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/src/7b5b918c68a2d7ba60d8c614e3576a3bc87785f4/modules/raw_rec/raw_rec.c
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 05:12:44 PM
I have noticed another thing:

static int buffer_size_compressed = 0;

If  buffer_size_compressed is divided by 14, the result is in float point, you might need to change it back to int. That is a pain in the axx thing too I guess.

I might be wrong.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:15:44 PM
sized_used isn't static so I can try that... the frames come out smaller when I open them so 12 bit should be helping.

buffer_size_compressed = (size_used * raw_bitdepth) / 14;

This doesn't make sense to me actually... why are we multiplying this out? The *12/14 calculation is to find width normally.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:17:38 PM
1%, I just realised

{
                .name = "Bit Depth",
                .priv = &raw_bitdepth,
                .min = 12,
                .max = 13,
                .choices = CHOICES("12 Bit", "14 Bit", ),                               
                .help2 = "Bit Depth",
            },


this will probably not work (can't test atm)... I wrote somewhere that raw_bitdepth would work as usual if it was anything else than 12.
It's wrong... It has to be either 12 or 14. Probably needs a menu update function and a LUT.
As for why it stops, no idea. :(
Probably

if (r != buffer_size_compressed) goto abort;

gets called, but no idea why.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:21:12 PM
It does work... when its 13 I get normal 14 bit I believe. You just have it set to if bit depth = 12 do this otherwise do that.

Also files opening in ACR just fine for me.

* Yea, both  files work fine... I just show one of each.. the 14 bit files play back in camera of course.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:24:59 PM
Quote from: 1% on May 24, 2013, 05:21:12 PM
It does work... when its 13 I get normal 14 bit I believe. You just have it set to if bit depth = 12 do this otherwise do that.

It's used as actual bitdepth in write_speed_update and lv_rec_save_footer()

int speed = (res_x * res_y * raw_bitdepth/8 / 1024) * fps / 100 / 1024;

footer.frameSize = footer.xRes * footer.yRes * raw_bitdepth/8;
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:30:13 PM
But for 14 bit we don't do this?

I took the line out and it still behaves similarly... its like it just dies out after a while even at 960x540.

I see how it would be a problem then with it being 14 or not.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:32:16 PM
Quote from: 1% on May 24, 2013, 05:15:44 PM
sized_used isn't static so I can try that... the frames come out smaller when I open them so 12 bit should be helping.

buffer_size_compressed = (size_used * raw_bitdepth) / 14;

This doesn't make sense to me actually... why are we multiplying this out? The *12/14 calculation is to find width normally.

It's just the compression ratio, it's the same for width and the whole image buffer. (to tell fwrite how much to write).

@mucher: not sure if you're serious, but all your assumptions are wrong...  :P
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:32:47 PM
Quote from: 1% on May 24, 2013, 05:30:13 PM
But for 14 bit we don't do this?

Yes, we do.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:33:54 PM
I guess using 13 works but some calculations will be wrong?

I dunno maybe we should try 10 bit it might be less cpu intensive?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:36:35 PM
I think what happens (regarding 13 instead of 14) is that raw2dng overwrites the wrong value anyway, so it does work in the end.

I really doubt 10-bit is less cpu intensive, but it would be nice to have as an option, for sure.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:48:46 PM
I don't get it tho...  it looks like its keeping up for a while and then something happens.

Also what sucks is I can't see CPU usuage... on 600D I would be able to.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 05:54:58 PM
Instead of trying to catch this unicorn (see benchmarks), I'd say it's better to investigate how to enable 12-bit output directly from DIGIC.

You can select different raw types from c0f37014: denoised, with different byte order, compressed and so on. Some test code: http://www.magiclantern.fm/forum/index.php?topic=5614.msg39696#msg39696
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 06:05:48 PM
Hey, don't act as if we knew this all along!  ;D

DIGIC compressed RAW... I'll try not to get overexcited this time.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: scrax on May 24, 2013, 06:07:39 PM
Quote from: Mayo on May 24, 2013, 06:05:48 PM
DIGIC compressed RAW... I'll try not to get overexcited this time.
;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 06:09:05 PM
Well, some of the raw types look kinda like this: http://www.nycresistor.com/2012/08/21/ghosts-in-the-rom/

The value at c0f37014 seems to be 6-bit, not sure what each bit means.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 06:09:59 PM
We had to test it, lol. But yea, I don't see this working right now. Forcing it lower via digic sounds like a smarter plan. Aren't those for defect detection tho?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 06:13:03 PM
Some of the raws are very noisy, others are cleaned up, but there are also scrambled ones. I don't know the image format, but I know for sure the digic can output 12-bit data.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 06:13:44 PM
One thing is for sure: Canon's CR2 are compressed... so it's out there.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 06:16:07 PM
Yeah, huffman compression. Also a JPEG encoder, see http://www.magiclantern.fm/forum/index.php?topic=2803.0
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 06:16:49 PM
There are several bit depths... that craw bits has several settings but for some reason they don't apply to pics or LV. I guess have to look at lv_af raw and find the address for 6D then we can start. I can probably undo the merge since this seems like a dead end unless D does something better with the module.

If we ever figure out how to use the TTJ path :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 06:32:47 PM
Quote from: Mayo on May 24, 2013, 05:32:16 PM
It's just the compression ratio, it's the same for width and the whole image buffer. (to tell fwrite how much to write).

@mucher: not sure if you're serious, but all your assumptions are wrong...  :P

I know nothing about camera, nothing about programming, but only waiting too anxiously to see when this good ML stuff will come to 7D(for me a jpg sequence is perfectly fine, because it is picture style compatible--so one does not have to correct the clips' color one by one, and jpg sequence can be true color full resolution too)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 06:37:31 PM
7D has dual procs.. no access to raw data easily. I'd do jpeg because of speed/size... a 2K jpeg would be OK... a smaller jpeg would not.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 24, 2013, 07:07:02 PM
Quote from: 1% on May 24, 2013, 05:48:46 PM
Also what sucks is I can't see CPU usuage... on 600D I would be able to.

There's no working build for 600d? d's asm lib only for 6d?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 08:58:01 PM
No point.. .if it doesn't run on the faster proc its not going to run on the slower one.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 24, 2013, 09:18:00 PM
Quote from: 1% on May 24, 2013, 08:58:01 PM
No point.. .if it doesn't run on the faster proc its not going to run on the slower one.

So 100% sure it's the CPU?

Bad :(
Title: Re: 12-bit (and 10-bit) RAW video
Post by: N/A on May 24, 2013, 09:18:50 PM
Can we overclock the processors? I know it would probably kill the camera or something, but say I wanted an excuse to buy a mkiii...

Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 09:42:11 PM
Nope, wish we could.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: savale on May 24, 2013, 09:51:51 PM
How about truncating the 2 lower bits? All the deep blacks have a lot of noise anyway so why not just throw them away... It will be less cpu intensive
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 24, 2013, 09:55:52 PM
How much CPU does it need more? I mean - what's the order - is it lacking 10% more CPU speed, or 500%?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 10:05:17 PM
Truncating might be worth a shot but I dunno how to do that.

Its lacking.. ie not helping over 14 bits even though working as expected. I can't even do 960x500 continuous with this. Unless D comes back and tells us what we're doing wrong its a dead end.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: bjacklee on May 24, 2013, 10:38:15 PM
tenacious "D" we need your help.. :D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 08:00:03 AM
Another maybe immature thought. 14bit color in raw video actually means 2^14 grade brightness, according to wiki.  So the 14bit to 12bit conversion seemingly should be:

int x; //brightness grade

x = x / (2^14);
x = x * (2^12);

Now things start to make sense to me.

Just my 2 coins.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 08:01:50 AM
Or the other way:

int x = x * (2 ^ 12);
x = x / (2^14);

Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 08:08:31 AM
But if x is int, and it is divided by a very large number, it will change to value 0 perhaps,


so I would rather change it to:

int x; //YUV brightness grade

register double y = x; //temporarily change it to double precision

y = y / (2^14);
y = y * (2^12);

x = y; //change the value back into int

What do you say developers?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: KMikhail on May 25, 2013, 11:18:34 AM
Why don't you just divide by 4? Which would be similar to bit shift by 2 bits >>

Bit shift was handy since the 80286 era, when x*320 was = x <<  8 + x << 6. But since then mul and div became very cheap.

Granted, if your integer isn't right alinged (least mening bits) you have to bit shift more to the right, and then back to the left. Masking would do the job too, but would require either a register or mem op.

BTW, according to spec ARM9 has 16 32bit MAD units, should be more than capable of basic integer math, no?

EDIT:

I see, you probably are trying to get a better rounding. For that purpose you can ADD/SUB something, prior to dividing. But dividing and then multiplying a double number won't give you the result, it is caled a floating number, for a reason. There is a set of specialty rounding operations for doubles. However, I doubt double operations are fast on such hardware.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 11:51:47 AM
Quote from: KMikhail on May 25, 2013, 11:18:34 AM
Why don't you just divide by 4? Which would be similar to bit shift by 2 bits >>

Bit shift was handy since the 80286 era, when x*320 was = x <<  8 + x << 6. But since then mul and div became very cheap.

Granted, if your integer isn't right alinged (least mening bits) you have to bit shift more to the right, and then back to the left. Masking would do the job too, but would require either a register or mem op.

BTW, according to spec ARM9 has 16 32bit MAD units, should be more than capable of basic integer math, no?

EDIT:

I see, you probably are trying to get a better rounding. For that purpose you can ADD/SUB something, prior to dividing. But dividing and then multiplying a double number won't give you the result, it is caled a floating number, for a reason. There is a set of specialty rounding operations for doubles. However, I doubt double operations are fast on such hardware.

What I worried was that accuracy problem, using int might not be accurate enough, and if the data were too small, divide by (2^14) might make it unusable at all, so I change it to floating point, and, I worried that float might not be accurate enough too, so I wanted it to use double instead. Modern CPU might be fast enough to handle double, my wild guess. If not, you can still change to use float, that might be accurate enough already. But if I have got the full source code, I will definitely compile it into my way, before loading it to my camera, including meddling a bit with that raw.c too.

BTW, ARM9's 16 x 32bit MAD unit is in floating point to my understanding.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: KMikhail on May 25, 2013, 12:08:01 PM
Quote from: mucher on May 25, 2013, 11:51:47 AM
What I worried was that accuracy problem, using int might not be accurate enough, and if the data were too small, divide by (2^14) might make it unusable at all, so I change it to floating point, and, I worried that float might not be accurate enough too, so I wanted it to use double instead. Modern CPU might be fast enough to handle double, my wild guess. If not, you can still change to use float, that might be accurate enough already. But if I have got the full source code, I will definitely compile it into my way, before loading it to my camera, including meddling a bit with that raw.c too.

BTW, ARM9's 16 x 32bit MAD unit is in floating point to my understanding.

int x; // 14 bit number, in a 16/32 bit format
x = (x +2) / 4; // rounding closer to normal rounding, not CPU binary world.

// x is 12 bits now

2 => 1; 3 => 1; 4 => 1; 5 => 1; 6 => 2; 7 => 2; 8 => 2;

Double is 64 bit, and by now is totally fine on modern CPU. But for raw handling in our Canons we need more integer performance for packing all this raw data into something more space efficient. Floats, though, are nice to have at times too.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 12:51:26 PM
Quote from: KMikhail on May 25, 2013, 12:08:01 PM
int x; // 14 bit number, in a 16/32 bit format
x = (x +2) / 4; // rounding closer to normal rounding, not CPU binary world.

// x is 12 bits now

2 => 1; 3 => 1; 4 => 1; 5 => 1; 6 => 2; 7 => 2; 8 => 2;

Double is 64 bit, and by now is totally fine on modern CPU. But for raw handling in our Canons we need more integer performance for packing all this raw data into something more space efficient. Floats, though, are nice to have at times too.

I don't understand your code very much.

ARM9 has 16 32bit floating units, let us assume that a double calculation costs 4 times more time to calculate than a float calculation, then the 16 32bit floating units' efficiency is like 16/4 = 4, so it can mean that there are 4 double's running at the same time, that can be translated to 4 frames at the same time. If all the calculation costs around 16-32 CPU cycles per frame(including time waiting for CPU's task scheduling), the CPU should by far have enough power to process 24fps in real-time. I reckon. So my modification should work, theoretically.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: eyeland on May 25, 2013, 05:50:14 PM
We seriously need a canon engineer to defect (again?) and give us a few spoilers...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 25, 2013, 06:13:22 PM
Why not hack at those registers in the noise thread... one of those probably enables 10/12/14 bit. Then nothing will have to run on the CPU.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 26, 2013, 09:48:43 AM
the ARM used in DIGIC has no VFP afaik.
this is an extra option when licensing ARM's IP.

so pricessing any floating point can only be done in software.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: budafilms on May 26, 2013, 10:04:11 AM
For those who have videos in 10 bits, the question is if that video it isn´t a quality of a good britrate in h264 with a prolost preset...?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 26, 2013, 10:28:05 AM
Then we have to stick to the original int thing :o

But the color level is still 2^14 >> 2^12 or >> 2^10, right? Mighty developers
Title: Re: 12-bit (and 10-bit) RAW video
Post by: squig on May 26, 2013, 10:31:58 AM
Quote from: budafilms on May 26, 2013, 10:04:11 AM
For those who have videos in 10 bits, the question is if that video it isn´t a quality of a good britrate in h264 with a prolost preset...?

This is a joke right?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 26, 2013, 11:46:28 AM
when cutting the least 4 bits you will usually cut off the noise that is said to be ~3-4 bits in bright areas
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 26, 2013, 01:08:50 PM
i just ran some code to read out VFP registers and can confirm now, that this HW has no VFP unit.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 26, 2013, 05:17:49 PM
Quote from: g3gg0 on May 26, 2013, 01:08:50 PM
i just ran some code to read out VFP registers and can confirm now, that this HW has no VFP unit.

So there's no need in it then :-) There are many other (fast) ways of bit conversion without using floating point calculations.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: IliasG on May 26, 2013, 10:14:01 PM
Of course the best is to get a ready 12bit raw or even 10 bit if it is gamma/log encoded. In fact I believe a 10bit log is better than 12bit linear.

But if this hack is difficult-impossible then the compression must be made in CPU.

I think that the way to go is 10bit log or rec.709 gamma encoding by a lookup table. No floats, no calculations .. all precalculated in at table.
Then the reverse (linearization table) can be in raw2dng or in the dedicated exif tag.

Something similar do BlackMagick and Sony and Nikon's etc.
BlackMagic encodes a 16bit linear raw in a 12bit gamma rec.709 (or log) file. The 12 log to 16 linear bitdepth sounds hyperbolic but we have to remember that BM uses only digital amplification (in fact it is just a exif tag .."Baseline exposure") for ISOs higher that the base so they need a bit more bitdepth.

ML needs 14bit linear to 12-10bit gamma or log and together with the analog ISO (up to 3200-6400) the expected file quality is at the same level as BM for mid ISO and higher at high ISO.

The question is how fast can this lookup be ??.

The lookup table with 16384 indexes and one column data with 1024 distinct values (10bit) is OK I think.

To make things better we can safely clip the 14bit data at about 1984 ( Black point is around  2048 - 64 = 1984) so any pixel with 14bit value <= 1984 will be mapped to 0 10bit. This way we can have a denser sampling of the useful range of values.

Looking at Blackmagick's and Nikon's linearisation tables ("Dump raw curve" with RawDigger) one can see that Nikon use an extended linear area at the darks and the exp area follows for midtones and highlights. It is usefull to keep data linear at the darks and without any manipulation .. usefull to avoid posterization and color shifts at the darks.

The 64 levels "below black" I choose is because so does BM (256 levels 16bit) .. and to leave footroom if something goes wrong (Black Level). It can be half (32 levels) with no problem.

Ideally we need a gamma with 16X multiplier at the linear part (so that no change needed there going from 14bit to 10) which extends around 20-40 levels (10bit) over "Black Level" (Nikon uses 400 linear levels at 14bits) .. and a relatively mild exp (^2.0) so that the highlight data stay relatively dense. Sadly my background in Maths is weak and i cannot match the linear with the exp part .. help needed ..

16X linear multiplier has the Prophoto gamma but it is linear area is very small, all falls in the "below black" area ..
Title: Re: 12-bit (and 10-bit) RAW video
Post by: xNiNELiVES on May 27, 2013, 06:20:05 AM
If the devs here get 12 bit working I think it will be possible for the 6D to shoot in 1880x817 resolution. From other photos taken in 14 and 12 bit raw I found the difference to be about 79% of the file size of a 14 bit photo. So with 1880x817 resolution requiring 55mb/s x .79 = 43.45mb/s. The highest 6D write speed I've seen is 41.7mb/s so within 2mb do you guys think it would be sustainable? Am I doing any of the math or analysis wrong?

Or is there a way to have real time 24p compression? Something to allow for higher resolutions?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Audionut on May 27, 2013, 06:27:15 AM
Quote from: xNiNELiVES on May 27, 2013, 06:20:05 AM
Or is there a way to have real time 24p compression? Something to allow for higher resolutions?

We should leave this thread to development discussion of making 12/10 bit encoding possible, before considering what advantages it will have with RAW recording.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 27, 2013, 06:36:20 AM
Quote from: IliasG on May 26, 2013, 10:14:01 PM
I think that the way to go is 10bit log or rec.709 gamma encoding by a lookup table. No floats, no calculations .. all precalculated in at table.
Then the reverse (linearization table) can be in raw2dng or in the dedicated exif tag.

Yeah, I also thought about the lookup table method. Should be faster than math.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: xNiNELiVES on May 27, 2013, 06:39:02 AM
Quote from: Audionut on May 27, 2013, 06:27:15 AM
We should leave this thread to development discussion of making 12/10 bit encoding possible, before considering what advantages it will have with RAW recording.

Ok :) it may be an incentive for other developers not yet dedicated implementing this feature yet, then again they probably already knew...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 27, 2013, 07:38:24 AM
Hello, Masters,

I just did a simple math, to see which part will change its value while converting from 14bit to 12bit or 10bit, and I use this formula:

We will try to find out which brightness value in 14bit world, after converting to 12bit by using int formula like x * 2^12 / 2^14, will change to the smallest value, which either 1 or 0 in int world:

int x; //the value to be found out

formula:

x * (2^12) / (2^14) = 1

the result:

x = 4

So any value in the original 14bit data smaller than 4 will be converted into 0 in the final 12bit data format, which totally consists 4 values: 0, 1, 2, 3

That means that in the original 14bit data, only brightness level from 0, 1, 2, 3 will be converted into value 0 in the final 12bit format. And these 4 values only consists only minimum fraction comparing with 14bit world, which is about 0.024%(4 / 2^14) of all the values in 14bit world, and its final form, which is total black level value = 0. What the good news is this part of data will not be transfer by HDMI interface(I heard that HDMI usually will not transfer brightness level from 0 - 5 in all the 0 - 255 values to the final equipments of display (like TV, projectors...).

To see the 10 bit scenerio:

formula:

x * (2^10) / (2^14) = 1

x = 16

There  will be totally 16 values converted into 0 while using int formula x * (2^10) / (2^14), that consists 0.098% of all the values in the 14bit data format, and these values might be neglected by HDMI interface anyway.

So, I think that using int formula x * (2^12) / (2^14),  or x * (2^10) / (2^14) into 10bit respecfully,  might be very viable methods.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 27, 2013, 07:48:17 AM
The bad news is that every 4 value in 14bit data will change to the same value in 12bit form, but then I realize that 12bit world has 4 times less value than 14bit world anyway.

What do you say masters? ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on May 27, 2013, 08:04:34 AM
HDMI doesn't belong here in any way, cause it's a display transfer format, but raw is not made for display, but for processing first. For transfer via HDMI sRGB values are compressed to 16-235 (in Rec.709 standard), and then recompressed back to full 0-255 range for display in TV (as far as I remember).

It's obvious that we'll have only every 16 number if we supress 14->10bit (each bit is a power of 2, so 2^4bits = 16), so we'll get 1024 values instead of 16535. But the good news is that's still more than sRGB 255 (although, need to try it, how it will Raw-convert, will there be banding, etc, etc).

Btw, to eliminate banding, we can use a bit of dithering while using the lookup table (but that's not so good as dithering in picture-space).

Anyways, all these are idle talks, until we get confirmation that this lookup-table thing will be realizable and fast on our little slow ARMs.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 27, 2013, 08:30:59 AM
Another thought.

In x * (2^12) / (2^14), one can only multiply first, or every value will become 0 if one divide by 2^14 first. So the worse scenerio can be this:

The largest number in 14bit data times 2^12, and then divide by 2^14, can be pains-takingly slow, but worth a try.

the largest value: 2^14 * 2^12 / 2 ^ 14, that means 16384 * 4096(that is 67108864
)/ 16384

And we can use precalculated values in the programs too, like instead of using x * (2^12) / (2^14) for easier reading, we can always use x * 4096 / 16384 instead.

As a last resort, we can use unsigned int instead of int, can we?

ARM might possibly be fast enough, because int is pretty fast to run, usually 1-a few CPU cycles in x86 world, and if we use unsigned int, it can be even faster.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 27, 2013, 08:37:21 AM
As the rounding issue, I don't know what these DIGICs will do, probably it will delete everything behind the decimal point to speed up running. But the difference can be like 2394 or 2395, can still be far more accurate than the human eyes can tell.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 27, 2013, 09:02:32 AM
@mucher:
not sure what you want to do with your calculation.
but the data has to be unshifted from the 32 bit words and doing a simple LSR#2 while building the 14 bit value
will get you the value as 12 bit and the same result.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: savale on May 27, 2013, 09:21:04 AM
First step:

mu suggestion: make it as fast as possible and see if it's feasible. So just truncate the 4 LSB bits using a bitmask / bitshifting:
something like this:


/* let's say the variable fourteenBitsValue contains the 14 bit value */
tenBitsValue = (fourteenBitsValue & 0b11111111110000) >>> 4; /* zero the 4 LSB bits and shift 4 bits */
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 27, 2013, 10:42:53 AM
Okay, back to rounding.

Lets simplify it for understanding - will work it out in a decimal system first.

We have 1001 and 1005 and want to shift it to the right by 1 digit, divide by 10. The double result of division is 100.1 and 100.5, nearest integers after NORMAL math rounding are 100 and 101. However, if you want to operate in the integer space you will have 100 and... 100, since CPU rounds DOWN to next integer. To avoid this we want to add 5 (half of the decimal base in power of the shift) to get normal rounding: 1001+5=1006 >> 1 = 100, 1005+5=1010 >> 1 = 101. Now, if you wish, you can shift them back. I am not sure how you want to align the final result (left or right).

To shift by 2 bits, divide by 4, you need to add 2^2 / 2= 2, before you shift, to preserve correct mathematical rounding. Then, if you need to re-align most meaningful digit to the left, you can shift it back.

I hope it is clear now. But, look-up table might be indeed a better alternative, unless referencing memory via ptr is expensive, I am not familiar with efficiency of DIGIC5+.

BTW why it was so important to round it so well? Perhaps for 10 bits?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mogs on May 27, 2013, 11:51:10 AM
The packed data looks pretty messy.  Dealing with this the most efficient way seems more important than rounding errors.

Once the packed data has been pulled apart, before repacking at a lower bit depth, it may be worth considering implementing something like this for a cleaner roll-off in the highlights.

(https://www.apertus.org/sites/default/files/dr-illustration-v03.png)
(from the apertus site, http://apertus.org)

If the knees were aligned with bit boundaries it should be possible to implement with only shifts and adds?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 27, 2013, 12:20:24 PM
The image is misleading.

1) Nowadays DR of digital sensors is enormous, compared to what it used to be, and outperforms many, if not most of the film (per area unit). Some BW might be still better, when other properties (grain) are poor.
2) Both linear digital and enhanced digital should stretch from (0,0) to the same topmost rightmost point, it is an allocation of levels per bits what makes it different (linear vs. curved), not the clipping (defined at the moment of reading the sensor).

At this moment I would prioritize ability to compress RAW 14 bit in some lossless/lossy way, and then chopping some bits off and possibly applying LUT. IMHO, of course.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 27, 2013, 01:11:43 PM
Quote from: savale on May 27, 2013, 09:21:04 AM

/* zero the 4 LSB bits and shift 4 bits */


not necessary as the lowest bits are already cut off when shifting right...

seriously, i dont get what you all are trying to calculate.
if its about proper rounding, well then add 8 before shifting 4 bits right.
any additional operation like adding, masking, shifting, whatever makes the code half as fast.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on May 27, 2013, 01:57:31 PM
Can someone make a test bitshifting build? I guess it shouldn't even needed to be based on d's object file.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 27, 2013, 02:05:43 PM
Quote from: a1ex on May 24, 2013, 05:54:58 PM
Instead of trying to catch this unicorn (see benchmarks), I'd say it's better to investigate how to enable 12-bit output directly from DIGIC.

You can select different raw types from c0f37014: denoised, with different byte order, compressed and so on. Some test code: http://www.magiclantern.fm/forum/index.php?topic=5614.msg39696#msg39696

This.

If we could make CMOS output shifted RAW data, then it would be much easier. It would be much easier even if it could output padded & aligned pixel data (so we don't have to "dig out" the RGB values from packed Bayer-data)., thus making it easier to shift diown to 10 bit in CPU
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 27, 2013, 02:29:09 PM
Quote from: vicnaum on May 27, 2013, 01:57:31 PM
Can someone make a test bitshifting build? I guess it shouldn't even needed to be based on d's object file.

what do you mean with that? thats exactly what d has done.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 27, 2013, 02:31:37 PM
Quote from: Grunf on May 27, 2013, 02:05:43 PM
If we could make CMOS output shifted RAW data, then it would be much easier. It would be much easier even if it could output padded & aligned pixel data (so we don't have to "dig out" the RGB values from packed Bayer-data)., thus making it easier to shift diown to 10 bit in CPU

if you want padded and aligned data, this means you will have a higher data rate.

(assuming you want it to be 16 bit aligned)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on May 27, 2013, 06:04:27 PM
Quote from: KMikhail on May 27, 2013, 10:42:53 AM
...

I hope it is clear now. But, look-up table might be indeed a better alternative, unless referencing memory via ptr is expensive, I am not familiar with efficiency of DIGIC5+.

BTW why it was so important to round it so well? Perhaps for 10 bits?

In case it is a question for me, yes with 10bits raw bayer we need very careful rounding and even with careful rounding 10bits depth can be inadequate. While with RGB 10bits can be (and are) OK with raw Bayer "master" there is the need to apply some heavy manipulations not needed for the ready to use RGB file. Demosaic, WB (which is a 1+ stop digital amplification for the weak channel), Color transformation from Bayer R-G1-B-G2 TO RGB (once again about 1 stop amp for the weak channel)

But with a lookup table we can have untouched the low values (keep the sampling at the same density as the 14bit starting data) where the problem is significant and gamma encode the rest.

We can even develop a curve imitating the S-log ..

Say we go for 14 to 10bit,
- we can clip at 32 (or even less) 14bit levels below "Black Level" instead of 2048 as it is now. Just map any 14bit level lower than 2016 (14bit) to zero. The remaining levels are 16384-2016 = 14368 which will be mapped (with any
- the first (say 128) 14bit values stay untouched by using a linear part with a slope of 16 y= 16*x
- then continue with a gamma .. say y= x^1/2
- if we want the S-curve then at the last 1/2 or 1/3 stop we invert the gamma y=x^2

Try to dump Nikon's D5200 raw curve with Rawdigger .. it's as described before .. linear-gamma-inverted gamma. Same proposal as AXIOM's Smart Dynamic Range

Let's hope the lookup table can be managed fastly by Digic .. can any expert give an opinion on this ??.
I think for a PC (X86 architecture) a lookup table is almost always preferable/faster.

One more compression scheme crosses my mind... Can we, during writing the raw data, use variable bit depth adapted to the real needs (values up to 256 really need 8bits). Then the same lookup table can be expanded with one more field giving this info ... :):).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 27, 2013, 07:40:26 PM
Quote from: g3gg0 on May 27, 2013, 01:11:43 PM
any additional operation like adding, masking, shifting, whatever makes the code half as fast.
and another memory access outside TCM memory will slow it down again a cycle or more.

so: 1/3th the performance.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 27, 2013, 10:30:41 PM
Quote from: g3gg0 on May 27, 2013, 02:31:37 PM
if you want padded and aligned data, this means you will have a higher data rate.

(assuming you want it to be 16 bit aligned)

It would lead to more data being transferred internally, but it would also mean that CPU doesn't need to do unpacking of the primaries, right?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jgerstel on May 27, 2013, 11:24:21 PM
Currently CF card IO is the bottleneck....
Imagine you can solve the 14 to 10 bit issue: this bottleneck is solved and resolutions can explode: what about 5d3 4K camera..... just an idea... :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 27, 2013, 11:33:58 PM
I'd settle for acceptable resolutions on SD cards.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 28, 2013, 12:08:34 AM
You know mates, what realy stumbles me.

With 5D3 we have 6fps at 22mp @ raw with lossless compression. 1920*1280 is 9 times smaller, so the same codec/routines should be able to pull up to (& who knows, may be beyond) 54fps, most likely 60fps, if you'd cut some fancy features. With 70% compression.

Canon definitely can pull, if it would care.

Is there any way to access RAW routines of original hardware/firmware?

Also, is the code multi-threaded? Apparently all these slowdowns with GUI/etc. is due to serial perferming?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 28, 2013, 12:44:51 AM
2 different processes. LV raw/photo raw

Maybe possible to shrink the actual read area of the sensor and then you get something like 640x480 120fps really line skipped. Some demo canon fw had this and it wasn't stable.

We don't know how to do that yet.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 28, 2013, 01:03:31 AM
Quote from: 1% on May 28, 2013, 12:44:51 AM
2 different processes. LV raw/photo raw
We don't know how to do that yet.

You probably saw my tests of video raw vs. photo raw. Binning definitely averages several pixels, having noise closer to resized 22 mp raw. I, personally, don't consider 1:1 mode valuable for me: I want to use full sensor and full coverage of my glass, that's what makes 5D3 standing out of everything else,  including DR, according to my test (1-2 stops better noise in blacks than 1:1).

So, if by any chance photo raw engine would be accessed to compress LV raw it won't slow it down significantly and give an outstanding ability to open these raws in DPP, where color is so nice and demosaicing works so well. Limitless abilities.

This would be more than worthy of a good donation :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 28, 2013, 05:18:43 AM
I am not expert in computing, but as far as I have known, the LUT things looks very good, but that sounds that we will have a lot of codes like: if x<y0 and if x>y1, x=y2, these kinds of codes are known to stall the CPU to a halt. d's method should be more suitable for CPU to do, and it already working real time reportedly by several one here who have actually loaded it into camera and tried, and we only don't know why the program halts. Maybe d can mercifully display his source code, maybe there will be someone who can look closely to see if there is something more we can do. d's method sound more straight-forward to me, and easier. To achieve better rounding and accuracy probably, soundingly to me, we only need to change from (x * 12) / 14 to (x * 2^12) / 2 ^ 14, and the cpu might be able to collaborate with that.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: savale on May 28, 2013, 08:11:42 AM
Quote from: g3gg0 on May 27, 2013, 02:29:09 PM
what do you mean with that? thats exactly what d has done.

Sorry I missed that... as I understand now the bits are truncated before putting them into the buffer. Is that correct?

I have two new ideas that might be worth checking:

1) what about manipulating the pointer instead of shifting. Would it be faster?

something like this:

- write 14 bits
- move the write pointer backwards 4 positions
- write 14 bits (which overwrite 4 bits of the previous value)
- move the write pointer backwards 4 positions
etc...

Details must be sorted out to make this work correctly (LSB / MSB)

2) a second thought: what about doing the conversion 14 to 10 bit at time when written to flash instead at time it's written into the buffer? I can imagine the results will be different. (I am not sure better, but might be worth a try)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on May 28, 2013, 08:16:00 AM
Quote from: mucher on May 28, 2013, 05:18:43 AM
I am not expert in computing, but as far as I have known, the LUT things looks very good, but that sounds that we will have a lot of codes like: if x<y0 and if x>y1, x=y2, these kinds of codes are known to stall the CPU to a halt. d's method should be more suitable for CPU to do, and it already working real time reportedly by several one here who have actually loaded it into camera and tried, and we only don't know why the program halts. Maybe d can mercifully display his source code, maybe there will be someone who can look closely to see if there is something more we can do. d's method sound more straight-forward to me, and easier. To achieve better rounding and accuracy probably, soundingly to me, we only need to change from (x * 12) / 14 to (x * 2^12) / 2 ^ 14, and the cpu might be able to collaborate with that.

Nope, LUT things will be just y = lut [ x ], and that's all.

And about the calculations you mention - they are easily done (like g3gg0 said) with bit shifting:
Raw14b: 13598 = 0011 0101 0001 1110
Shift 2 bits right =>12bit = 0000 1101 0100 0111 = 3399
13598 * 2^12 / 2^14 = 13598 * 4096 / 16384 = 3399.5 = 3400

So no need for calculations at all. Rsh solves it on cpu-elementary level.
If you need better rounding (like said), add half of the bits (2bits = 4, so add 2) to number before shifting:
(13598+2) = 13600
13600 Rsh 2 = 3400
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 28, 2013, 09:27:11 AM
Quote from: savale on May 28, 2013, 08:11:42 AM
2) a second thought: what about doing the conversion 14 to 10 bit at time when written to flash instead at time it's written into the buffer? I can imagine the results will be different. (I am not sure better, but might be worth a try)

sorry, but read into the raw_rec code.
as soon you understood how all that works, you will understand that these ideas are obsolete ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: savale on May 28, 2013, 10:32:23 AM
hehe ok :P I will
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Northernlight on May 28, 2013, 01:01:30 PM
Hi, I came across this article reg. the Canon C500 and it's 10-bit RAW capabilites.
Not sure if this was relevant under this thread or under discussion of ETTR, but I found it interesting (and I am sure most you already know this).

http://nofilmschool.com/2012/11/canon-c500-shipping-raw-4k/

"Adding gain adjustments at the sensor level produces a consistent stop range above and below the middle grey level, even at high ISOs, and reduces the overall noise in the image."

As I understand this; Canon's RAW format is special in the sense that the C500 add gain at sensor level before outputting the RAW stream, is this in a way, approaching a little more to ETTR? (to improve DR and SNR)

Q1: Could this "approach" be possible to do also with the RAW module from ML ? (if you hopefully succeed with 10bit RAW!)

BTW! I have no chance to follow the technical discussion in this thread, but:
Q2: Do you at this point have any estimates of what the DNG file size could be eg. @1920x1080, should you succeed with 10bit RAW?
(I am just curious to where 10bit might take us in case of fps/resolution)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tin2tin on May 28, 2013, 01:18:17 PM
Quote from: Northernlight on May 28, 2013, 01:01:30 PM
BTW! I have no chance to follow the technical discussion in this thread, but:
Q2: Do you at this point have any estimates of what the DNG file size could be eg. @1920x1080, should you succeed with 10bit RAW?

Quote from: vicnaum on May 23, 2013, 11:30:48 AM
In theory, SD cards can give 22MB/s or 961194 bytes per frame. 1280x600 in 10 bits gives 960000 bytes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 28, 2013, 02:50:00 PM
You guys could be looking at the registers to see if 10bit can be made in camera.. there are functions like set_craw bits, etc. Just have to find the correct one.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on May 28, 2013, 03:02:37 PM
Quote from: 1% on May 28, 2013, 02:50:00 PM
You guys could be looking at the registers to see if 10bit can be made in camera.. there are functions like set_craw bits, etc. Just have to find the correct one.

Could you tell please where can we look at these?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on May 28, 2013, 04:41:33 PM
Quote from: Northernlight on May 28, 2013, 01:01:30 PM

Q2: Do you at this point have any estimates of what the DNG file size could be eg. @1920x1080, should you succeed with 10bit RAW?
(I am just curious to where 10bit might take us in case of fps/resolution)

I have found this:

video bitrate calculator, shows frame size and bitrate with any resolution, framerate and bit depth.

http://web.forret.com/tools/video_fps.asp?width=1720&height=720&fps=24&space=raw&depth=14
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: xNiNELiVES on May 28, 2013, 05:24:28 PM
There is a specific canon raw video calculator.  I forgot where I found it. Google magic lantern raw and go to news.  There's dome Germany website and on the website there is a link to use it.

Never mind here it is: http://www.slashcam.de/tools/ml-raw-calc.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on May 28, 2013, 05:39:03 PM
Yep, but it doesnt have bit depth (or I cannot find this) so cannot compare 14 to 12 and 10 bits. This one I have found has this feature so we can calculate theoretical performance for 12 and 10 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Audionut on May 28, 2013, 05:42:53 PM
Guys, lets please stop with the, "oh wow with only 3 bit recording we can record 8k resolution" type posts, and please leave this thread solely for actual development discussion pertaining to the implementation of this feature.

Thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: inozcenterod on May 28, 2013, 06:40:01 PM
hello everybody im from VEN , basic knowedge of c++, how can I start with the ML 10 bits source, and if is possible to use some kind of lossles compression algorithm to minimize the size of data, or level up the cpu voltaje. (SORRY for my english)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 28, 2013, 07:07:08 PM
QuoteCould you tell please where can we look at these?

Look at regs touched by lv_save_raw and look for setcraw in strings.. there are 2 functions that set bits but it doesn't seem to be doing anything.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on May 28, 2013, 07:58:49 PM
Quote from: Northernlight on May 28, 2013, 01:01:30 PM
Hi, I came across this article reg. the Canon C500 and it's 10-bit RAW capabilites.
Not sure if this was relevant under this thread or under discussion of ETTR, but I found it interesting (and I am sure most you already know this).

http://nofilmschool.com/2012/11/canon-c500-shipping-raw-4k/

"Adding gain adjustments at the sensor level produces a consistent stop range above and below the middle grey level, even at high ISOs, and reduces the overall noise in the image."

As I understand this; Canon's RAW format is special in the sense that the C500 add gain at sensor level before outputting the RAW stream, is this in a way, approaching a little more to ETTR? (to improve DR and SNR)

Q1: Could this "approach" be possible to do also with the RAW module from ML ? (if you hopefully succeed with 10bit RAW!)

BTW! I have no chance to follow the technical discussion in this thread, but:
Q2: Do you at this point have any estimates of what the DNG file size could be eg. @1920x1080, should you succeed with 10bit RAW?
(I am just curious to where 10bit might take us in case of fps/resolution)

I wouldn't say Canon does this at the sensor Level, but at the Digic after digitization. There, they use the full 14bit depth to make calculations that a raw converter would have to do, which is problematic on 10bit data. They fight the problems described below

Quote from: IliasG on May 27, 2013, 06:04:27 PM
...., yes with 10bits raw bayer we need very careful rounding and even with careful rounding 10bits depth can be inadequate. While with RGB 10bits can be (and are) OK with raw Bayer "master" there is the need to apply some heavy manipulations not needed for the ready to use RGB file. Demosaic, WB (which is a 1+ stop digital amplification for the weak channel), Color transformation from Bayer R-G1-B-G2 TO RGB (once again about 1 stop amp for the weak channel)

But with a lookup table we can have untouched the low values (keep the sampling at the same density as the 14bit starting data) where the problem is significant and gamma encode the rest.

Canon say that:

- They have "baked" in raw the ISO.
Well, in photo mode, Canon 5DIII has the ISO fully "baked" in raw for analog ISO settings (100, 200 ... 12800, no gaps in raw histograms) and partially for intermediate and expanded ISOs where its mixed analog-digiral (we can see gaps in the raw histograms).
So do Sony (and others with Sony sensors) but it happens that they stop analog ISOs lower (1600 is usual).
BlackMagic uses only the base analog gain and every ISO higher than the basic is just a metadata flag (look for "Baseline exposure" in Exif)
Canon 1Dx uses analog ISO up to 51200 and for intermediate ISOs also. All raw histograms are without gaps so either it is analog ISO up to 51200 or the DAC digitize at 16bits and after 16to14bit conversion the histogram looks full (If I had to I'd put a bet on the second).  Maybe C500 goes up to same analog ISO also.

- They apply WB on the full bitdepth (14 or 16bit) raw data and then downconvert to 10bit. They gain accuracy this way and better tonality over all the range.
WB is in fact multipliers at each channel. For Canon's "Daylight" the multipliers are about R=2.0 G=1.0 B=1.5. The first thing a raw converter will do is apply these multipliers so the R & B channels will loose sampling density accordingly.

We could also "bake" a midway WB (say temp.4500K) in the proposed LUT but then we would need one LUT for each color and (if separate LUTs was possible) in the end any known raw converter would get crazy with WB/color transformations ... until someone decides to support the "crazy" ML-DNGs ..

- They use a log LUT on the 10bit output. This means they keep dense sampling where it counts (at the darks) and the sampling density decreases logarithmically as we go upright to the highlights. This way they can fit a 13-14 stop DR in 10bit log file.
It is the same we will do with the proposed 14 to 10bit LUT .. wish it is fast enough with the available resources ...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 28, 2013, 08:22:42 PM
Quote from: IliasG on May 28, 2013, 07:58:49 PM
...
The best way to use 10 bit space is to calculate the topmost value for each channel of the RAW, record it elsewhere and align all channels to the 11111111111111b. Likely after applying 14 bit LUT for highlights compression, then cleverly getting rid of last 4 bits. In the post, upon conversion to a normal DNG (which in fact might have all information in the lower of 14 bits after shiftin channels back and appying inverse LUT!) opposite operations should be performed.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on May 28, 2013, 10:17:01 PM
Quote from: KMikhail on May 28, 2013, 08:22:42 PM.

I suppose you talk generally and assuming the available computation resources are enough, not like what ML has currently available ..

Just aligning to the topmost channel is in many cases ineffective as even a single specular highlight drives all channels to align at the White Level cutoff point. Take a look at raw histograms of many video-raw samples, it happens very frequently for either all chanells or the 2 of 3. And  this is usually the case of a high DR shot when we mostly need the best behavior.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 28, 2013, 11:21:30 PM
another way could be:
- converting the 14 bit value into a CString by using sprintf and new CString(buffer)
- then using strlen to get the string length
- subtract 4 from that length
- copy that number of characters in an UTF-16 string
- code some utf16_atoi
- and save that value to CF card.

this can be done in VB too.

SCNR
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on May 28, 2013, 11:47:21 PM
@ g3gg0,

and .. will this be fast enough ?.

What we all wonder is ... if ML cannot find the lower bit-depth version ready from Canon (and this version can be of an acceptable quality or not at all ..), then which is the way to go ??
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 28, 2013, 11:49:48 PM
I cannot find that things mentioned in EOS SDK
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 29, 2013, 12:57:08 AM
Eos SDK? You need dumps from your camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 29, 2013, 06:32:36 PM
Quote from: IliasG on May 28, 2013, 10:17:01 PM
I suppose you talk generally and assuming the available computation resources are enough, not like what ML has currently available ..

Just aligning to the topmost channel is in many cases ineffective as even a single specular highlight drives all channels to align at the White Level cutoff point.

1) Generally, yes, aligning to the right might be not possible on the embedded cpu, though, could be done cleverly.
2) Nonetheless, that's the best way in general.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: fotojohni on June 02, 2013, 10:00:28 AM
Isn't the camera converting to 8bit during h.264 encoding?  Maybe you could just grab it there.  Anyway, I believe this is missing link.  Made a 2 minute youtube vid and I ended up with hundreds of GB's of footage in various formats along my workflow production line.  10 bit is better for a lot of reasons.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Clemens on June 04, 2013, 10:03:38 AM
Is there some progress on this topic? Would really love to test a 10 or 12 Bit build and help you with some feedback.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: xNiNELiVES on June 04, 2013, 06:32:06 PM
Same. I would like to test.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on June 04, 2013, 06:58:55 PM
The asm module wasn't fast enough... tried it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: xNiNELiVES on June 04, 2013, 07:06:09 PM
Quote from: 1% on June 04, 2013, 06:58:55 PM
The asm module wasn't fast enough... tried it.

So what does this mean.  It's not possible for live conversion to 12 bit?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on June 04, 2013, 07:21:08 PM
Yea, it just stopped working immediately.. it never flushed a single buffer. Maybe on 255M you'd get a decent string of frames.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: fotojohni on June 04, 2013, 11:03:37 PM
I think someone may have mentioned simply truncating the lower (or upper) two bits of data.  It's a loss of 2-3 stops of DR, but maybe worth it for low DR scenes where you don't need that extra data anyway.  I actually run into this situation quite frequently.

Another option could be to truncate the U+V (YUV, I'm assuming) then do a better algorithm for Y channel, just a thought.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on June 04, 2013, 11:10:47 PM
in raw data there is no YUV. its RGB. thats what is all about.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on June 05, 2013, 07:43:08 AM
Quote from: 1% on June 04, 2013, 07:21:08 PM
Yea, it just stopped working immediately.. it never flushed a single buffer. Maybe on 255M you'd get a decent string of frames.

So does it work slowly, or doesn't work at all? Maybe a buggy asm? If try at 320x240 or 160x120 res - is the concept working?

Maybe d's from canon, and he gave us an incomplete puzzle that we need to solve ourselves?
Or maybe d's from canon and he gave us a non working code sample so we lose our heart and don't try 10-12 bit raw again?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Audionut on June 05, 2013, 07:50:40 AM
Lets stop with the useless speculation and keep it on topic thanks.

The activity in this thread should make it pretty clear that the developers are interested in the idea, and being able to make it happen.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on June 05, 2013, 10:12:05 AM
Quote from: vicnaum on May 28, 2013, 08:16:00 AM
Nope, LUT things will be just y = lut [ x ], and that's all.

And about the calculations you mention - they are easily done (like g3gg0 said) with bit shifting:
Raw14b: 13598 = 0011 0101 0001 1110
Shift 2 bits right =>12bit = 0000 1101 0100 0111 = 3399
13598 * 2^12 / 2^14 = 13598 * 4096 / 16384 = 3399.5 = 3400

So no need for calculations at all. Rsh solves it on cpu-elementary level.
If you need better rounding (like said), add half of the bits (2bits = 4, so add 2) to number before shifting:
(13598+2) = 13600
13600 Rsh 2 = 3400

As per vicnaum. If this works, there will be no loss in DR, just lose lum levels because of that 10bit's less levels, but I understand very little of that hex level calculation and I have no time to check its validity.  And if vicnaum is right, there is no calculations to do except moving bits in HEX level, which sounds very promising to me.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: glnf on June 07, 2013, 01:50:57 AM
I don't want to distract anyone working on this. I just like to add a little clarifications for those that are not clear about the goal of this operation. It is actually a very simple idea. The RAW data from the sensor is 14 bits long. Each 14 bit number represents the grey value of a red, green or blue image pixel.

So a typical value looks like this: 1010 0101 0100 10 (the spaces are just for reading convenience)

We could translate the value into the decimal system (it is 10578) but there is not really a need for this. All we have to do is to get rid of the last two digits. So we end up with 1010 0101 0100. That is quite a different value. We write that onto the card. Then, when we retrieve the values from the card we simply add 00 at the end of the number, making it: 1010 0101 0100 00. This is almost the same value (decimal 10576) as initially. So we loose a tiny bit of precision but we save some space and reduce data rate.

There is by the way no need to do any rounding. Chopping off the lowest 2 (or alternatively 4) bits before writing the value to the card and replacing it with 2 or 4 "0" for further processing on the computer does the trick. Instead of adding "0" it would be slightly more elegant to add random values, hence mimicking (a very low level of) noise.

So far goes the theory. Since I haven't done any experiments with the camera I don't know if there are any additional obstacles, eg. a reverse or unusual bit order. (I remember reading something about black level in this thread and that sounded a bit odd to me. And no reason to come up with logs or whatever. That only harms the beauty of this trick.) Cheers, g
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Critical Point on June 07, 2013, 03:46:19 AM
So, lets say on a 600D, what would it mean a 10 or 12 bit raw ? What kind of resolutions are we talking about for 24 fps ? Please don't say 1080p...because I'm going to have a heart attack. :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hammermina on June 07, 2013, 06:31:59 AM
@ critical point

there will be no 1080/24p i think the max will be 720p if this mode will work
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: xNiNELiVES on June 07, 2013, 07:27:31 AM
Not to be a stickler guys but audio nut just told everyone to keep useless speculations aside. It'd best if we just keep out of this development discussion. If you were to open up a thread dedicated to general discussion of this topic that would be permissible.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on June 10, 2013, 12:32:45 AM
Maybe we can use some precalculated values. It is seemingly that DIGICs has accuracy problems when multiply too many times. But I am not sure how it will do in dividing, probably it will be fine. The equation changing to 10bit, x * 2^10 / 2^14, can actually turn into two parts x * ( 2^10 / 2^14), and 2^10 / 2 ^14 equals to 1/16, so the equation can change to x* 1 / 16, and that equals to x / 16, and likewise, changing to 12bit means x/4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on June 10, 2013, 12:51:42 AM
the last time:
devs are aware how to convert data from 14 bit to 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, and even 0 bits.
it is no riddle. it is no challenge.

can do this while sleeping, we can do this while steering a car and we can even do this in electrical circuits.
seriously.

the only issue is: THE CPU IS SLOW.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: driftwood on June 10, 2013, 12:55:47 AM
Exactly, the cpu is perpetually at  96% or greater ceiling at the moment.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: necronomfive on June 10, 2013, 12:59:49 AM
Quote from: mucher on June 10, 2013, 12:32:45 AM
...and that equals to x / 16, and likewise, changing to 12bit means x/4

Or in other words: shift right by 4 or 2 bits....  ::)

Concerning precalculated values: if CPU bus speed to memory is an issue (just look at the plain memcpy values), then I don't see how ADDITIONAL external memory fetches from a LUT will help this cause in any way.

I also see no way how 12-bit/10-bit compression will ever be achieved with the CPU. It was only designed to do GUI/filesystem management related stuff. The real processing is done by highly dedicated signal processing blocks inside the DIGIC. As said before, the only chance to get this working is finding an undocumented register bit in DIGIC which reduces the bit width of written sensor data.

With no documentation on DIGIC, this is like finding a small coin inside a barn full of hay, without any kind of guaranty IF this coin exists at all.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: necronomfive on June 10, 2013, 01:10:26 AM
.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on June 11, 2013, 10:08:53 PM
Quote from: necronomfive on June 10, 2013, 12:59:49 AM
...
Concerning precalculated values: if CPU bus speed to memory is an issue (just look at the plain memcpy values), then I don't see how ADDITIONAL external memory fetches from a LUT will help this cause in any way.

I also see no way how 12-bit/10-bit compression will ever be achieved with the CPU. It was only designed to do GUI/filesystem management related stuff. The real processing is done by highly dedicated signal processing blocks inside the DIGIC. As said before, the only chance to get this working is finding an undocumented register bit in DIGIC which reduces the bit width of written sensor data.

With no documentation on DIGIC, this is like finding a small coin inside a barn full of hay, without any kind of guaranty IF this coin exists at all.

Since I am not familiar with 5DIII internals and I have already asked 2-3 times how much time consuming would the LUT proposal be (no answer yet), can you give an estimation ??.

A clear answer from the start would keep the thread compact and on target ..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Chucho on June 12, 2013, 08:21:40 AM
I'm a little behide here, so changing the 14bit can be done by code but is the big challenge to find a way to do it in camera? If you search for CrawBit Invalid it will give you the acceptable arguments for FA_SetCRawBitNum() 0x10=16bit, 0xE=14bit, 0xC=12bit, and 0xA=10bit. Doing something this
void foo();
{
        call("FA_CaptureTestImage");                  
   call("FA_GetCrawBuf");
   call("FA_SetCRawBitNum",0xE);
        call("FA_DefectsTestImage");
   call("FA_CreateTestImage");
}

And then looking for the registers that change. Would that give us some sort of hint?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tin2tin on June 14, 2013, 08:12:12 AM
@ Chucho
Looks like a very interesting finding. I wonder what the devs can make of it?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on June 14, 2013, 10:16:44 AM
i am not 100% sure but i remember smth like that.
iirc it was setting up a bunch of registers that are related to the twoadd etc stuff.
so it was not a simple register change and the CMOS sends us 10 bit, but the parameter to one of those not-yet understoor modules.
if we understand what they do, we can redirect EDMAC output from CMOS to these modules etc.

that involves a lot of register setup.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: rudi on June 14, 2013, 12:48:24 PM
Quote from: g3gg0 on May 23, 2013, 12:16:09 AM
yeah..
well, i implemented an memcpy using LDMIA/STMIA for LV buffer copying and this was a dead end.
so i tried to get EDMAC working.

I don´t want to keep the idea of a 12 bit shifter alive, but i´m just interested in how much MB/s you recieved with LDMIA/STMIA.
the mysterious "d" seemed to reach about 30-40MB/s on an smaller modell.
and as far as i can see from the debug scrennshots a memcopy can reach over 70 MB/s. Ist that correct. Was that LDMIA/STMIA?
Some years ago a made heavy X86/MMX/SSE2 optimisations and looked some days ago at ARM optimizing.
As far as i can see (and what "d" did) you can sqeeze out a lot of bubbles and stalls out of hand optimizing.
Shifts are also free in ARM. Nearly every instruction can be combined with the barrel shifter at no cost. (I´m sure you are ware of this).
And you have a lot of Cache/Flush/Prefill Options in ARM.
But again, i don´t want to reanimate the idea, i´m just curious how fast optimized LDMIA/STMIA / memcpy on a 5DMK3 can be.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bjacklee on September 27, 2013, 02:47:02 PM
If only this is possible.. :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1berto on September 30, 2013, 12:36:42 PM
I don´t understant nothing of this, but IMO compression is the way to go, thats was the secret of RED cameras and the problem of BMCC workflow...

If you find a way to compress the raw data so that the image don't get compromissed and at the same time can be captured for the card.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Midphase on October 01, 2013, 10:28:43 AM
Sigh, for the nth time...it's not possible since the Canon CPU's aren't fast enough to perform the necessary calculations. They can barely keep up with writing the data to a CF card as it is. :o
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: pascal on October 01, 2013, 11:24:54 AM
Quote from: Midphase on October 01, 2013, 10:28:43 AM
Sigh, for the nth time...it's not possible since the Canon CPU's aren't fast enough to perform the necessary calculations. They can barely keep up with writing the data to a CF card as it is. :o
On the contrary if you add cpu time to conversion you remove cpu time from data write.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Audionut on October 01, 2013, 11:57:24 AM
Read the thread fully before making assumptions!

Feel free to PM me to have this thread reopened, if you have code that makes bitdepth reduction in camera possible.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 11, 2016, 05:37:11 PM
Quote from: Audionut on October 01, 2013, 11:57:24 AM
[...] to have this thread reopened, if you have code that makes bitdepth reduction in camera possible.

I think I've got such code, based on g3gg0's raw_twk (http://magiclantern.fm/forum/index.php?topic=13163.0) experiments.

https://bitbucket.org/hudson/magic-lantern/commits/c6fbba9

Timing: 120ms for a full-sized image (5936x3950).

Help is welcome - not just by testing (or asking for) the finished thing, but by trying to understand how Canon's image processor works, how to call it for other purposes (it can do lots of operations, such as add, sum, min, max, debayer, curves, filters, (m)jpeg compression, lossless raw compression... the main difficulty is understanding how to configure these image processing modules), and - of course - by turning this proof of concept into something useful.

Other relevant pages:

http://magiclantern.wikia.com/wiki/Register_Map
http://www.magiclantern.fm/forum/index.php?topic=13408
http://www.magiclantern.fm/forum/index.php?topic=6740.0
http://www.magiclantern.fm/forum/index.php?topic=16428.msg159735#msg159735

Happy hacking.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: markodarko on May 11, 2016, 07:05:22 PM
I've just read the entire thread (I wasn't around here in 2013) and I'm a little confused. Without standing on anyone's toes by not helping from a development POV, can someone explain why @d apparently got this working but others say that the CPU is not fast enough? (Or did I misread?)

Thanks,

Mark.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 11, 2016, 07:33:26 PM
Read the benchmarks and do the math.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: markodarko on May 11, 2016, 08:12:59 PM
Quote from: a1ex on May 11, 2016, 07:33:26 PM
Read the benchmarks and do the math.

:-D you take me for a learn-ed man, Mr. @a1ex.

I did take a look at these:

Quote from: a1ex on May 23, 2013, 10:26:21 AM
(http://a1ex.magiclantern.fm/bleeding-edge/raw/bench14to12-playback.png) (http://a1ex.magiclantern.fm/bleeding-edge/raw/bench14to12-movie24p.png)

Impressive.

...but I don't know what I'm looking for to do any math on. Or where to find a standard 14-bit benchmark to compare against. I figured that you saying "impressive" was a good thing though. :-)

Cheers,

Mark.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 12, 2016, 09:01:49 AM
I'm doing some research , to understand the possibility of a 10 or 12 bit raw video found this on the  http://magiclantern.wikia.com/wiki/Register_Map
under "Image Preprocessing"
0xC0F08120 - PACK16 Control register
0xC0F08124 - PACK16_MODE (mask 0x131)
             From ProcessPathForFurikake (raw_twk):
             -------x --xx----
                  111 (0x130) = 16-bit output
                  011 (0x030) = 14-bit output
                  010 (0x020) = 14-bit output
                  001 (0x010) = 12-bit output
                  000 (0x000) = 10-bit output

So the raw_twk is just for  DIGIC 5 processor  (5D3)  or is these addresses also validate for other camera.
I wonder thou if there enough cpu power to do this or is it a matter of available ram left from ML .
Thinking out loud .  :)
Yes I think this is worth getting involved  deeper , need to do more reading 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 12, 2016, 09:16:57 AM
@markodarko:

Since d's implementation is CPU-based, the numbers are impressive when compared to memcpy, or to my C version (which is used for the previews such as in mlv_play).

Now, imagine you want to record 1920x1080 at 24 fps. What numbers would you need here?

@reddeercity: that's the register I've tweaked to reduce the bit depth. This method doesn't use the main CPU to do the processing; it calls the dedicated hardware, very similar to the method we already use for copying raw video frames for recording (edmac).

For other cameras: I think the 60D will work just by finding the right stubs; the 5D2 has the functionality, but the exact implementation looks a bit different.

No need to worry about RAM: 12-bit frames are smaller than 14-bit ones, and the conversion doesn't need any extra memory (because it's done on dedicated processing hardware - very similar to a GPU).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: goldenchild9to5 on May 13, 2016, 07:23:20 PM
Man Magic Lantern keeps coming with awesome updates 10 - 12 Bits would be welcome..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 14, 2016, 04:06:20 AM
60D:

NSTUB(0xFF3C1C60, ProcessPathForFurikake)
NSTUB(0xFF3FC6FC, PACK16_Setup)


I'm assuming I need to merge raw-bit-depth and RscMgr_memory so that the patch hook will work (otherwise patch manager complains cache is locked). Unfortunately RscMgr_memory gives err 80 in LV.


on a side note: hmm... (https://en.wikipedia.org/wiki/Furikake)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 14, 2016, 07:54:06 AM
That's right - I just pushed a few changes to the patch manager, to allow patches like this to run on 60D. I had to use a long jump, because module code is loaded at the end of RscMgr, which is far from both ML core and ROM (so regular jumps don't work).

So, for 60D at least, you need to merge patchmgr as well.

I've also got a log in QEMU showing what registers are configured (without the ResLock part):

furikake-60d.log (http://a1ex.magiclantern.fm/bleeding-edge/prepro/furikake-60d.log)

5D3 is almost identical - except it uses different EDMAC channels.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 14, 2016, 06:28:49 PM
Is this only for testing on the big cams like 5d3 and 60d, I went to get the stubs, tried it, I get smaller file sizes for the 12 and 10 bit, but there scrambled and cam goes to black and freezes after done taking picture when I do 12 and 10 bit, 14 is fine?

//** Image processing paths */
NSTUB(0xFF420AE4, ProcessPathForFurikake)
NSTUB(0xFF5EF23C, PACK16_Setup)             // PACK16_MODE:%#lx and others

not 100% there right, especially the PACK16.

(http://s32.postimg.org/6lept4sj9/Capture1.jpg)

(http://s32.postimg.org/3ox79c9yt/Capture2.jpg)

Or is the problem this part that is only for 5D3?
+    /* hook to override the bit depth in PACK16 setup function */
    patch_hook_function(
        (uintptr_t)&PACK16_Setup,
        0xE92D4010,
        PACK16_Setup_hook,
        "PACK16: raw bit depth override"
    );
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 14, 2016, 06:46:16 PM
PACK16_Setup is correct. I'm not sure what's wrong - if the output is scrambled, probably the patch override didn't work. Are there any errors in Debug menu, under Memory Patches?

I'm also working on a lower-level implementation that doesn't need any extra stubs.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 14, 2016, 07:09:29 PM
Nope, it just says None.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 14, 2016, 08:36:24 PM
Okay, it's working on 60D (if I turn off GD, then I don't get the err80), avg. time is about 117 ms (268 MB/s !).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 14, 2016, 09:08:13 PM
EOSM also working, ~114 ms

NSTUB(0xff425150, ProcessPathForFurikake)
NSTUB(0xff5fa9e8, PACK16_Setup)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 14, 2016, 09:49:29 PM
Got same result on 550D, no freeze, but scrambled image for 10 and 12, unless its not even suppose to work on this cam anyway, but tried anyway.
NSTUB(0xFF353274, ProcessPathForFurikake)
NSTUB(0xFF37FBE8, PACK16_Setup)


Update!!, Works on 550D, had to do a Make Clean first, Ill try on 700D now.

Works on 700D as well! Just needed a make clean.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 14, 2016, 10:39:14 PM
On 700D MLV Format
14bit 820 ms 38 MiB/s
12bit 825 ms 32 MiB/s
10bit 739 ms 30 MiB/s

speed seems to jump around alot on all bits, gets faster after 2nd mlv is taken, usually by 100ms on each.

DNG Format
14bit 2970 ms 10 MiB/s
12bit 2169 ms 12 MiB/s
10bit 1960 ms 11 MiB/s



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 15, 2016, 01:25:07 AM
Just to clarify, I'm talking about just the speed of bit depth conversion, not the entire time it takes to save the file.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on May 15, 2016, 11:45:10 AM
Just to check if I understand things right.

There is a dedicated chip(the digic chip?) inside the canons which can do bit depth conversion.
The test results show it can convert about 30/35 MegaByte per second.
Full HD at 24/25 fps is about 90MegaByte/s.
So the chip isn't capable of doing it fast enough for full HD video

Could be use full though for full res silent picture time lapses. You're saving some memory card space.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on May 15, 2016, 01:26:39 PM
Read what a1ex wrote on previous page:

QuoteTiming: 120ms for a full-sized image (5936x3950).

As well as
Quote from: dmilligan on May 14, 2016, 08:36:24 PM
Okay, it's working on 60D (if I turn off GD, then I don't get the err80), avg. time is about 117 ms (268 MB/s !).

Quote from: Levas on May 15, 2016, 11:45:10 AM
Full HD at 24/25 fps is about 90MegaByte/s.
So the chip isn't capable of doing it fast enough for full HD video

Wrong, the "chip" does that already for you by providing FHD H264 files.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on May 15, 2016, 02:12:04 PM
Ah, so this is a big discovery :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Greg on May 15, 2016, 11:41:48 PM
500D :
NSTUB(0xFF327A58, PACK16_Setup)

5D2 :
NSTUB(0xFFB28D4C, PACK16_Setup)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chris_overseas on May 16, 2016, 11:02:43 AM
Stubs for 5D3 1.2.3 (untested):

NSTUB(0xFF3CCC14, ProcessPathForFurikake)
NSTUB(0xFF57CA1C, PACK16_Setup)


Edit: I just Googled Furikake. Interesting, though not sure what it has to do with a camera! https://en.wikipedia.org/wiki/Furikake :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on May 16, 2016, 12:48:26 PM
I'm really focus about this tread.

I think, It should be so usefull to be abel to have 10 bits and 12 bits.

- For exemple : 1920x1080 60p (12 or 10bits)
Or just by recording in good quality for 30 minutes or more.
But also 3K 10bit continuously ?

So, I ask for peoples in my cathegory (mean who are noobie).

Will it be possible for everybody having a 5D mIII or 60D (...), to easily use 10 and 12 bits ? A new module will come ?

Inifnity respect to every ML scientists !
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Greg on May 16, 2016, 10:41:33 PM
It works on old cameras. We need use the ProcessPath (without Furikake).

500D :

NSTUB(0xFF24E7F8, ProcessPath) //[DEFC]
NSTUB(0xFF327A58, PACK16_Setup)
//ProcessPath time : 96ms


We have only 3 values:
a1 = buf_in,
a2 = buf_out,
a3 = value 0 or 1 ?


So silent.c requires a small change
struct s_DefcData
{
    void *pSrcAddress;
    void *pDestAddr;
    uint32_t dstModeRaw;
} defc;


We can use this method on all cameras.


It might be interesting :

NSTUB(0xFF24FCCC, ProcessPath) //[SUB] DarkSubtraction
NSTUB(0xFF2506A0, ProcessPath) //[DEFD] related to TwoAdd
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Nutsy on May 17, 2016, 05:28:02 PM
What are the chances of this being developed for the 7d as well?
its very exciting stuff. but i fear maybe the 7D just wont be fast enough?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on May 17, 2016, 10:53:34 PM
What would this do for the 550D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Greg on May 19, 2016, 01:55:08 PM
Someone tried this with the 700D?
500D 10bit and 12bit (frames after DIGIC processing) does not have a hot and cold pixels.
There is a possibility that the DIGIC also corrects focus pixels.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 19, 2016, 03:54:50 PM
Yeah, I tried it on my 700D and posted results last week, but hmmmm, sounds like something I would like to test on the pixels, I didnt even think of checking the dots and pixels, just looked to see if it worked, lol.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on May 23, 2016, 11:27:57 PM
Sorry for being impatient, is there any progress? Anything even in early stage what could be tested on 5d3?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 24, 2016, 12:43:26 AM
You can compile the raw-bit-depth (https://bitbucket.org/hudson/magic-lantern/branch/raw-bit-depth) branch and try it with 10/12 full res silent pics.

From my quick attempt, normal silent pics don't work (and by extension, raw video) it seems b/c LV is running and these hardware modules are "busy" being utilized by Canon LV routines. Furikake just blocks until you exit LV. So to get this working for raw video it looks like we are going to need to understand more about how to manually setup these modules by overriding the low level registers that control them. That's a lot of reverse engineering work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 24, 2016, 07:25:17 AM
I have a semi-functional low-level implementation (it's basically the log I've posted earlier, converted to C code), but it only converts a picture at every other attempt. If anyone is willing to debug it, I can commit it in this state.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on May 24, 2016, 04:29:47 PM
I have to make a comment to follow this, I already though I was but I wasn't :P

From what I've followed so far; despite going from 16bit(14) to 10 or 12 it's still not faster to write to the card?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 24, 2016, 05:57:18 PM
if the kind of debugging u need is possible by me, i would be glad to help out any way i can, i owe you a few favors anyway alex.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on May 24, 2016, 06:19:31 PM
Count me in on the debugging whenever necessary @a1ex!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 24, 2016, 09:03:06 PM
Alright, pushed the code.

Finding the mistake might be as simple as solving one of those kid puzzles about finding the difference between two drawings (probably a typo in my code). Or - given that I couldn't figure out what was wrong in a couple of hours - it might be a little more difficult, as dmilligan already explained.

First, compile 15852ed and run "Fullres LV test" (from LiveView) to see how it should work. It will show a "full-res liveview", garbled in 12-bit and 10-bit modes (which is fine, because the drawing code expects 14-bit data).

Then, compile 21f337a and run the same test. It will do the first frame right, then it will freeze. Take the battery out.

If you add a timeout to the take_semaphore call, you will see it returning error every other frame (first correct, second bad, third correct, fourth bad and so on). Tested on 5D3 and 60D.

Note that regular full-res pictures work fine, because going to LiveView and back appears to perform the necessary cleanup (the one I've probably overlooked).

Also note that calling silent_pic_take_fullres in a loop, without going to LiveView, works fine, which puzzles me. This might help solving the mystery - turn "while(1) silent_pic_take_fullres(0)" into "silent_pic_fullres_lv_test()", step by step, and see where it fails.

To understand the code, see the log from reply #279, the Register Map wiki page, and the other links I've posted earlier for background info.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 24, 2016, 11:40:50 PM
On 700D, my first image is correct, second and on are garbled.

Also during 10bit, it freezes on 3rd image when I do a timeout, but continues on 12bit, but all garbled.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 25, 2016, 04:30:51 AM
Yeah, garbled image is what you are looking for, but without freezing.

First image is not converted to lower bit depth, that's why it looks good.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 25, 2016, 05:44:56 AM
Yeah, the only way I can get it to not freeze, is like you said add a timeout like this take_semaphore(edmac_write_done_sem, 100); which is what I think you meant to do, but only is stable at 12bit, 10 freezes after the 1st garbled frame passes. At 12bit, it goes fine with the correct garbled frames.

Not sure what to do with this  turn "while(1) silent_pic_take_fullres(0)" into "silent_pic_fullres_lv_test()", step by step, and see where it fails.

Don't see anything interesting I can find, but if you can think of anything for me to look at more closely and how, I will be glad to.

I was expecting to see text printed on screen, but I dont see any text at all, not sure if that is expected? I know that has nothing to do with the conversion, but I see  printf("Furikake time: %d us\n", (int)(t1 - t0));  So I assumed something would of printed?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 25, 2016, 06:04:14 AM
The two codes are doing pretty similar things: capture, display, convert bit depth. One is complex and works (it also saves the file, triggers overlays), the other is simple, but fails for some reason. So, to narrow down, it makes sense to simplify the complex code, step by step, to reach the simpler version, and write down at which step it fails, during the conversion process. It's very simple to do, but time-consuming.

To see the printf output, enable the debug console.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 25, 2016, 06:29:26 AM
Ok, printf stuff is visible now, thanks.

Wish I understood the code better, so I could know how to do the code step by step to narrow it down for you, I would take the time to do it, but honestly dont know which part is the complex one and which is the simple one, lol, most of the fuction is new, so wouldn't know where to begin and how to proceed,  sorry.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lucky Luke on June 07, 2016, 11:15:09 AM
successing 10bit recording, it would be worth to me buying this for 100€ or even more...and I guess to manymany others on the globe.
Any movements in this area?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on June 07, 2016, 11:35:26 AM
And another attempt to kill this project ...

Devs will drop out if ML goes commercial.
Donate or don't but don't expect anything to be done by throwing peanuts over the fence.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Nutsy on June 07, 2016, 11:50:55 AM
If I could afford to donate money I would... ML is a great project.

I'm on benefits (scum i know) and at the moment struggling to get by.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lucky Luke on June 09, 2016, 08:10:14 PM
"And another attempt to kill this project ..."

Really stupid insinuation.
Just my opinion on value and possibilities of 10Bit recording.
What is wrong to became commercial if it is worth?
We are taking so much for granted...
Yes, I would pay it even more than 200€ for having no drop outs because of the card reading.
This would be a real earthquake even now late in a 4K world.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on June 09, 2016, 08:52:24 PM
Can we please stay on topic and keep all discussion at a technical level. Posts like these just make it harder for those who might actually be able to contribute something to follow the conversation and find important information they need.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on July 03, 2016, 01:05:12 AM
Quote from: dmilligan on June 09, 2016, 08:52:24 PM
Can we please stay on topic and keep all discussion at a technical level. Posts like these just make it harder for those who might actually be able to contribute something to follow the conversation and find important information they need.

I agree.


I'm noob in code but doesn't matter. I do my best to understand and I would like to help you and try this new improvement.

Could somebody do a dropbox with the module ready to be try for 5D mark III 1.1.3 ?
Also, is it possible to chose in camera 10 and 12bit ? And can we chose resolution like when we use raw module ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Pierro777 on August 21, 2016, 06:43:45 AM
I'm new to the forum and just lurking a lot these past few days. I don't have much to contribute in terms of help (can't code etc.). But I just wanted to say you guys are all awesome. Keep up the amazing work! You help people create a lot of cool videos.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 02, 2016, 07:42:17 PM
On 60D, this appears to give 12-bit and 10-bit raw data for still pictures:


    MEM(0x2B308) = 0x10;   /* 12-bit */
    MEM(0x2B308) = 0;      /* 10-bit */


How did I came up with this magic value?

Look at the emulation log of the image capture process from QEMU (this post (http://www.magiclantern.fm/forum/index.php?topic=2864.msg174076#msg174076)).


[PREPRO] at ShootCa:FF1C5AB8 [0xC0F08094] <- 0x30      : PACK32_MODE


This code is called from FF1C5AB8; if you look it up in the disassembled code, it's inside engio_write. There are lots of calls to this function in the firmware; to see which one is used in our case, put this into 60D/debugmsg.gdb:


b *0xFF1C5A68
commands
  silent
  print_current_location
  printf "engio_write(0x%x)\n", $r0
  c
end


Compile autoexec.bin from minimal/qemu-frsp (qemu branch), copy it onto your SD image, and run the emulation:

./run_canon_fw.sh 60D,firmware="boot=1" -s -S -d io,int -singlestep & arm-none-eabi-gdb -x 60D/debugmsg.gdb


Result:

[ShootCapture:ff084a64 ] engio_write(0x2b2fc)
[PREPRO] at ShootCa:FF1C5AB8 [0xC0F08090] <- 0x80000000: PACK32_ENB
[PREPRO] at ShootCa:FF1C5AAC [0xC0F08094] <- 0x10      : PACK32_MODE
...


Now look at 0x2b2fc in a RAM dump using a hex editor, look at these lines (https://bitbucket.org/hudson/magic-lantern/commits/21f337ae919ed82e9aa1ebaf994e15ce1cb86f38?at=unified#Lmodules/silent/silent.cT1230) in the source code, and the mystery will clear up quickly.

I did not try to decode the 12-bit CR2 data, but the preview looks scrambled (as I would expect from 12-bit data interpreted as 14-bit).

It might be possible to change the bit depth of the LiveView stream with something similar.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 02, 2016, 08:55:33 PM
On 60D, in LiveView, this is enough to change the bit depth of the raw stream (valid values here (https://bitbucket.org/hudson/magic-lantern/commits/21f337ae919ed82e9aa1ebaf994e15ce1cb86f38?at=unified#Lmodules/silent/silent.cT1230)):


EngDrvOut(0xC0F08094, bit_depth_mode);


Now, all you have to do is to remove all hardcoded references to 14 bpp and bam! you have 12-bit and 10-bit raw recording. Example code to save a 10-bit DNG: dont-click-me.c (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/dont-click-me.c)

Looking forward to your pull request :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on November 02, 2016, 09:03:27 PM
excellent. thumbsup!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 02, 2016, 09:57:45 PM
 :D :D :D :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on November 02, 2016, 10:03:16 PM
Never regretted to have no popcorn in the house. Until now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on November 02, 2016, 10:10:47 PM
congrats for finding this @a1ex. Hopefully this won't raise Mark Free's street price in near future :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DavidSh on November 02, 2016, 10:49:27 PM
wow!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Pierro777 on November 02, 2016, 11:02:51 PM
Does 10bit raw mean it might be possible to have 60fps raw on the 80d someday ?!!!

Keep it up ! I wish I could help with anything lol.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hindra on November 02, 2016, 11:22:51 PM
That's awesome! Thank you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on November 02, 2016, 11:24:44 PM
Nice find Alex ! An additional small revolution in the magical world of magic lantern.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 03, 2016, 01:00:18 AM
Damn @a1ex, I smell the Christmas trees already...  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 03, 2016, 02:47:08 AM
Works on 60D. Can record 10bit full resolution for about 12 seconds. 720p is about 1 minute.

https://bitbucket.org/hudson/magic-lantern/commits/9c951e497666bb4f9fc8f933fa29bec12ce16030

Not using ML greyscale preview results in about 50% of frames being corrupted

Update:
Confirmed working on EOSM, however grayscale preview is broken (all you see is the last LV frame before you hit record), only tried with crop hack
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 03, 2016, 08:31:02 AM
The raw backend may be overwriting the raw_info structure and set it (or parts of it) back to 14 bits.

The easiest way is probably to block all overlays and other raw tools for other bit depths, and override the bit depth only while recording.

Supporting all bit depths in the raw backend requires a bit more work. Additionally, some features use direct pixel access (rather than using raw_get_pixel) for speed reasons; this access is probably fastest in 12-bit mode, as the packing should be a little easier (just a guess, didn't do any benchmarks).




Nice to see it working on the M; I've expecting it to work only on DIGIC 4. Here's why:

If you remember the "raw type" register (see PR #732 (https://bitbucket.org/hudson/magic-lantern/pull-requests/732/5d3-attempt-to-fix-vertical-stripes-so-it)) and look it up on the Register_Map (http://magiclantern.wikia.com/wiki/Register_Map) page, the register used on DIGIC 4 models (0xC0F08114) is labeled PACK32_ISEL (probably "input selection"). The bit depth register is 0xC0F08094 PACK32_MODE, so they must be related somehow. On 5D3, this register is 0xC0F37014, so I've expected the ISEL register to be moved as well. Didn't try yet.

Side note: PACK16 and PACK32 are probably image processing modules that write data from the image processor to the main RAM (the RAM visible from the ARM processor) and can convert it to various bit depths (hence the name "pack"). In a similar way, DSUNPACK, ADUNPACK and UNPACK24 are used to read image data from the ARM processor (which can be packed at various bit depths) and bring it somehow to other image processing modules. In all cases, the MODE register for each of these modules is used to configure the bit depth.

It isn't clear how to connect those processing modules together, but it's probably done with the ISEL switches (whose exact meaning is largely unknown, other than the above assumption about their naming).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 03, 2016, 08:42:09 AM
Quote from: dmilligan on November 03, 2016, 02:47:08 AM
Confirmed working on EOSM, however grayscale preview is broken (all you see is the last LV frame before you hit record), only tried with crop hack

If I remember correctly the grayscale preview was always broken on the M - or did that work at one point?
Regardless, this sounds like I really need to figure out a way to increase the frame size for raw recording on the M now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: myown on November 03, 2016, 09:26:07 AM
WOW!! You do such a great work!! Congratulations and many, many many thanks!

I have a 60D and wanne test this  :)
Do i have to consider something special?


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 03, 2016, 09:57:08 AM
Quote from: dmilligan on November 03, 2016, 02:47:08 AM
Not using ML greyscale preview results in about 50% of frames being corrupted

Figured it out - when getting the raw stream into main memory, we have to adjust frame size. On models that use CONFIG_EDMAC_RAW_SLURP (see http://builds.magiclantern.fm/features.html ), this parameter is under our control. On other models, this configuration is done in Canon code, which probably must be patched somehow.

Also, confirmed it working on 5D3.

To get clean image, you will have to change raw_info.bits_per_pixel from the raw recording module (this is not yet committed).

For more details, see the commit messages from this branch:
https://bitbucket.org/hudson/magic-lantern/commits/branch/raw_video_10bit_12bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on November 03, 2016, 11:01:34 AM
So what's more difficult? Switching all cams to use raw slurp in combination with running RAW_DEBUG_TYPE for all of 'em or better patching Canon code somehow. Already successfully used hardcoded values for slurp a while ago on 70D but the preferred raw type didn't match and I didn't feel like having time to try to find the best value for it at that time. Would other cameras still need hardcoded values for all modes when using raw slurp or do you have an idea to find / read them "magically" somehow...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 03, 2016, 05:09:48 PM
Enabling the new method on all models is desirable IMO, because we will have a single way of doing things. However, each model is likely to have its own quirks, so it's a bit of work.

Patching the ROM is probably easier, but pretty ugly if you ask me. You had problems with it on 70D (PR #734 (https://bitbucket.org/hudson/magic-lantern/pull-requests/734/cleaner-method-for-hiding-canons-bottom/diff#comment-18511247)), but I don't know where to begin debugging it. Maybe just merging the latest patchmgr could work, who knows.

BTW, for 70D, trying different raw types might reveal other streams, such as the dual pixel sub-images (just a guess, no idea how it's done on 5D4, but I wouldn't be surprised if Canon implemented it that way).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 04, 2016, 02:56:01 AM
Confirmed working on 60D now with Canon LV
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: goldenchild9to5 on November 04, 2016, 05:12:18 AM
@a1lex Great job.. that's just awesome smaller file sizes and longer recording times amazing.  This feature will be soo.. useful for the new gen canon's using 1080p 60fps 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on November 04, 2016, 08:45:50 AM
What's the state of the 550D in getting 10bit and 12bit.
(We all know the 550D is the most important camera of them all :P)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 04, 2016, 01:50:18 PM
Quote from: ItsMeLenny on November 04, 2016, 08:45:50 AM
What's the state of the 550D in getting 10bit and 12bit.
Maybe already working, somebody needs to try it: https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: araucaria on November 04, 2016, 02:19:42 PM
QuoteAudionut:
Is the code just truncating bits, or is it compressing them into the remaining?

edit:  The OP states that the bits are being compressed.

Curious to know what kind of compression it is. Great work, I'm very excited.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on November 04, 2016, 03:41:05 PM
Let me know if you need a 5DM2 (+VAF Filter) tester ...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on November 04, 2016, 04:43:50 PM
👍
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 04, 2016, 07:28:25 PM
Tested it on a 5D2, almost works, 10 bit works on every other frame, the rest are pink, and 12 bit also works on every other frame with the rest being completely pink. I have uploaded some DNG files made with MLP. I tried in full sensor resolution and 16:9, full sensor made 12 bit completely pink, but I only tested that once. Here are the 10 and 12 bit DNG files including 14 bit for comparison: https://drive.google.com/file/d/0BwvDlbhZgsGGY1BUM2tyN0YwdVk/view?usp=sharing (https://drive.google.com/file/d/0BwvDlbhZgsGGY1BUM2tyN0YwdVk/view?usp=sharing) The boring shot of a map is not enough to judge quality, but I think 10 and 12 bit look the good as 14 bit in this case.

"Disclaimer": I don't know if I missed anything, or did anything wrong, is there anything else to enable other than raw_rec module?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on November 04, 2016, 07:51:51 PM
What do you mean by "works on every other frame"?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 04, 2016, 07:56:38 PM
Quote from: Ilia3101 on November 04, 2016, 07:28:25 PM
Tested it on a 5D2, almost works, 10 bit works on every other frame, the rest are pink, and 12 bit also works on every other frame with the rest being completely pink.
This will be the case on all cameras except 5d3, 60D, and 600D, unless you set preview mode to ML grayscale. And you may notice the preview just looks like garbage once you start recording (ML grayscale preview can't handle anything but 14bit data yet)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 04, 2016, 07:58:54 PM
I mean this: first frame is fine, no corruption or pink artifacts, the next is all pink and corrupted, no image, then another fine one, another corrupted... and this cycle continues, so only half of the frames are not ruined, and when played back(which MlRawViewer refused to do), it just flashes pink every other frame.

Edit: @dmilligan just saw your post, thanks for clarifying, I will test again.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 04, 2016, 10:08:52 PM
Goods news here , I'm glad to see this thread back active.
I would like to test this on my 5D2 please ,  @Ilia3101 could I try your module ?
I don't have a environment to comply ml right now , just pm me  or post it here
:) 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 04, 2016, 10:36:08 PM
@reddeercity Uploaded the module: https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing (https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing), maybe you'll have some more success than I did.
After some more testing with ML grayscale, I have concluded that it does't yet fully work with the 5D mark ii, whatever I try, there's always something going wrong every other frame in 10 bit, either a pink frame or the bottom half is of another frame, and 12 bit seems to have weird colourful static all the time :'(.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 04, 2016, 10:43:22 PM
@lia3101 , thanks I'll do some testing and post the results  ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 05, 2016, 12:12:52 AM
Wow, lots of activity here. I've been asked to help compile some test builds so for a limited time only I'm posting everything that I could build on my bitbucket download area:

https://bitbucket.org/daniel_fort/magic-lantern/downloads

All platforms are there in one large zip file along with the command line tools, mlv_dump, raw2dng and cr2hdr, for both Mac and Windows.

Note that the 1100D.105 didn't compile and raw2dng needed this change in order to cross compile for Windows on a Mac:

raw2dng.exe: FORCE
$(call build,MINGW,$(MINGW_GCC) -c $(SRC_DIR)/chdk-dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR))
- $(call build,MINGW,$(MINGW_GCC) -c raw2dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64)
+ $(call build,MINGW,$(MINGW_GCC) -c raw2dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64 -std=c99)
$(call build,MINGW,$(MINGW_GCC) raw2dng.o chdk-dng.o -o raw2dng.exe -lm -m32)



I haven't tried any of this myself so -- Happy testing!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on November 05, 2016, 12:30:01 AM
Tried the 5D mark II with 12bits
Extract MLV with the new mlv_dump version and the line --dng --no-fixcp --no-stripes

I got pink frame issue too

Now i will test the 5DIII :)

Even if we got issues, i'm sure it will fixe of everyone do tests.

Thanks @dfort for all these news builds !

EDIT:

Ttried with the 5DIII :

10 bits = Stable  | FPS override enabled
12bits = Got pink frame only on the first frame | FPS override enabled
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 05, 2016, 12:54:05 AM
I tried on  my 5d3 113, just a speed test with sandisk extreme pro CF card 160MB/sek, i assume that despite image issues frames are recorded with their proper size (but i didnt check).

So i tried 3K 2,35:1 23,975fps- 14 bits - 134 frames
                                               -12 bit - 294 frames
                                               -10 bit - over 3 minutes recording, didnt check longer
1920x648 (16:9 after stretch) 60 fps     -14 bit 308 frames
                                                            - 10bit 1425 frames
1920x648 50fps seems to be continous in 10 bit.

1920x1080 37fps - 14bit - 239 frames
                             - 10bit - over minute, didnt check longer.

highest possible resolution 3584x1320 23,976 10bit - 160 frames (vs 61 with 14bits).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 05, 2016, 12:55:08 AM
Thanks a1ex, dmilligan,  Ilia3101 and dfort. Ran two test files with a 5D mark III and it is very promising. First fram garbled both for 10 and 12 bit but the rest looks fine. Disabled vertical stripes and cold pixels in mlv_dump. Needless to pint out it,s a blessing to see so much headroom left with the lower write speed even with 30 fps 1920x1080.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 05, 2016, 01:15:08 AM
Started testing with 10 & 12 bit  , as mention before on the 5D2 the Liveview is all scrambled even if you press the half shutter
for color preview in ML grey preview it's just colored scrambled lines. But I Found a trick !  :D  "Turn off Overlays when recording" in the overlay tab.
but of course that also kills the buffer overlay graph so you don't know any info of recording time etc... at least the Liveview is not all scrambled
and you can see what is being recorded .
I still need to test with HDMI & 3x Crop Mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: v8rrc on November 05, 2016, 01:28:21 AM
Tested on 6D 10bit seems to work, preview and playback garbled but dngs open in photoshop/lightroom fine. and Continuous OK at 1824 x 776 (2.35:1) 23.97FPS which is Great.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 05, 2016, 01:31:03 AM
Like reddeercity said, checked 5d3 113 with global draw OFF  - liveview is good then with 10 and 12 bits.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on November 05, 2016, 02:16:40 AM
As far as I can tell 10 bit was recording on the 550D, unsure how to play it back.
I was under the impression that it would result in a significantly lower bitrate, to the extent where I would be able to record full resolution @25fps, which it would only record half a second. Was that somebodies incorrect calculations in the past? (or am I thinking of 8bit).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 05, 2016, 03:28:54 AM
10/14 = 0.71 => you should see about 29% lower bitrate.


As a1ex has alluded to, the entire ML raw backend is basically hardcoded to only handle 14 bit raw data, so preview, playback and all that are not going to be working correctly. Fixing them is a lot of work (lots of tedious bit math), but should be pretty straight forward and requires no reverse engineering. I had a quick go at the preview code, but I'm not sure I understand all the transformations correctly. The result was starting to resemble the correct image but still very garbled.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 05, 2016, 04:07:48 AM
Some more test for Liveview , To get the overlays in liveview without being scrambled in 1:1 mode switch preview in the video menu in raw tab to
Auto or Canon preview then all overlays are enabled but not to sure if the meters for exposure etc... are correct this was for 12bit raw on 5D2.
For 3x Crop Canon preview , ML grey preview lockup preview but came back after recording is stop.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 05, 2016, 04:51:41 AM
Quote from: ItsMeLennyAs far as I can tell 10 bit was recording on the 550D...

Already have ML, how do I get 10 bit on my 550D?

Edit:
I found the files, do I just add to the folder on my SD card or do I actually have to reinstall ML?


Edit 2: Nevermind, got it working. Testing now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 05, 2016, 05:35:46 AM
Bunch of colored snow, but I used MLV Mystic. Recorded long though at 1728 using 10 bit, just can't see frames. This is on a 550D. Crop mode crashes camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 05, 2016, 07:00:57 AM
Completed some resolutions tests with 5D2 at 10 & 12bit with 3x Crop , the only resolutions atm that does not give corrupt on  ever other frame is
1856x1044 @ 10bit , ML Grey preview with overlays killed . Any other adjustment then this will give different type of frame corruption
from just total noise to part or the image is out of order and or with color noise bars across the image. 12 bit has corruption & or part of the image out of order
on all resolutions except in 3x Crop mode there 12 & 10 bit are free of corruption but there have a heavy pink cast just like the black level off .
Preview in Crop mode I used was ML Grey with overlays enable (default mode) . That's it for now .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DavidSh on November 05, 2016, 07:41:09 AM
Is there a build for 5d3 123 ?
Looking forward to do some tests...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 05, 2016, 03:40:05 PM
Quote from: ddelreal on November 05, 2016, 05:35:46 AM
Bunch of colored snow, but I used MLV Mystic.

You'll have to use the mlv_dump compiled from the raw_video_10bit_12bit branch with all processing options turned off (see previous replies) or MLVFS. These are the only converters I'm aware of that can handle non 14 bit data correctly (there might be others).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 05, 2016, 04:15:09 PM
Quote from: DavidSh on November 05, 2016, 07:41:09 AM
Is there a build for 5d3 123 ?
Looking forward to do some tests...

Short answer, no. Longer answer:

5D3.123, 70D.112 and 100D.101 have not been merged into the main trunk yet. Someone ( @nikfreak ?) could attempt to merge in the raw_video_10bit_12bit into these platforms.

1100D.105 build is currently failing but it has nothing to do with the 12-bit, 10-bit changes. The automatic builds on the download page are also failing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 05, 2016, 05:17:17 PM
Okay, so I used MLVFS and now I can see the frames. Every other frame is top half pinkish noise. I was trying 12FPS on 550D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 05, 2016, 08:03:37 PM
700D is working in 10-bit and 12-bit. Tested mv1080, mv1080crop, mv720 and zoom modes.

LiveView freezes when recording. In 5x zoom mode LiveView is in greyscale and becomes garbled when recording but turning off GlobalDraw fixes that.

About half of the time I'm getting a corrupted frame but only on the first frame:

(https://c7.staticflickr.com/6/5634/30496556750_d919626b81_n.jpg)

After that everything is beautiful! (Processed through MLVFS.)

(https://c5.staticflickr.com/6/5634/30760899156_b7e83081ae_z.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: krisdeak on November 05, 2016, 08:23:13 PM
Quote from: dfort on November 05, 2016, 08:03:37 PM
700D is working in 10-bit and 12-bit. Tested mv1080, mv1080crop, mv720 and zoom modes.

This is really really promising! Could you confirm the data rates achieved in 10-bit +  a rough estimate of how long we might be able to record like this, with the SD card hard limit taken into account?

I'll keep following this thread!

Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 05, 2016, 09:08:46 PM
Quote from: krisdeak on November 05, 2016, 08:23:13 PM
Could you confirm the data rates achieved in 10-bit

According to the ML display:

14-bit 36.8 MB/s @23.973p
12-bit 31.6 MB/s @23.973p
10-bit 26.3 MB/s @23.973p

I'm able to record continuously up to 1344x756 in 10-bit on the 700D. After that it stops automagically after a few seconds depending on the resolution. Basically if the bottom line on the ML menu is green it won't stop. For the wide screen enthusiasts that means 1536x654 on the 700D is doable.

I also wanted to report my findings on the EOSM. It works fine in the crop modes, mv1080crop and zoom with the same LiveView freeze and first frame corrupt issue as the 700D. There's something strange going on with the non-cropped modes. The camera records but the camera freezes when recording is stopped requiring a battery pull. The file is saved and appears to be mv720 (3x5) no matter what the Canon menu is set at which is normal behavior for the EOSM though there are some other idiosyncrasies with this build like it always defaults to 29.97 in non-cropped modes but in crop modes it will default to 23.98. Of course further testing is in order on this platform.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 05, 2016, 11:58:28 PM
Wonderful progress guys. So much activity as I've been trying to stay on top with you guys. Downloaded the builds for EOSM and 7D. Thanks @dfort!

Re: EOSM -- I can confirm that this works nice with MLP in spitting out DNG's/ProRes with the exception of first DNG being corrupted somehow. Tested on 24p, 30p & 60p.

Re: 7D -- This was shot in 48p @ 1728x692p it seems that I am getting that pink half corrupted frame on every other frame (for both 10-bit and 12-bit) as others have reported in here as well. Here's what they look like.

(https://c8.staticflickr.com/6/5468/30711689591_5a4ec3bce1.jpg) (https://flic.kr/p/NMTrkv)
(https://c4.staticflickr.com/6/5339/30711689811_bc128b4dc1.jpg) (https://flic.kr/p/NMTrpi)

https://vimeo.com/190398637

Before I can go ahead and test out a 5D3 -- I have to downgrade from 123 to 113 first so will report back by findings once I have them down.

Also any way I can get MLVFS to work with these 10-bit/12-bit files? Do I need to replace the binaries that @dfort shared with us and replace the ones inside MLVFS -- could you enlighten me on this @dmilligan please?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 06, 2016, 02:15:25 AM
Quote from: DeafEyeJedi on November 05, 2016, 11:58:28 PM
Also any way I can get MLVFS to work with these 10-bit/12-bit files? Do I need to replace the binaries that @dfort shared with us and replace the ones inside MLVFS -- could you enlighten me on this @dmilligan please?
MLVFS should already handle the files just fine. MLVFS does not contain or use binaries of ML tools.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 06, 2016, 02:19:03 AM
Yeah, MLVFS works. I've tried different frame rates and resolutions, still getting pink/corrupted every other frame in 10 and 12 bit. (550D)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 06, 2016, 06:42:23 AM
@DeafEyeJedi -- The binaries in the package I made have a commit (https://bitbucket.org/hudson/magic-lantern/commits/9b2547bb092e4c3c08265fe0d5948659d3e027db?at=raw_video_10bit_12bit) in the mlv_code to work with lower bit depth files. You should try using the binaries in the package from the command line and compare them to the ones from the unified branch. There's also a note:

Quotemlv_dump: fix DNG output for lower bit depths
You need to disable all processing, e.g. mlv_dump M00-1234.MLV --dng --no-fixcp --no-stripes

MLVFS can already handle these bit depths and like dmilligan explained doesn't use any of these binaries.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 06, 2016, 07:11:24 AM
Quote from: dmilligan on November 06, 2016, 02:15:25 AM
MLVFS should already handle the files just fine. MLVFS does not contain or use binaries of ML tools.

Apparently I needed to update the FUSE to make this work again. I could swear it was working right before the recent 10.12.1 update from Apple.

So indeed MLVFS works just fine with 10/12-bit files atm. Thanks!

(https://c7.staticflickr.com/6/5752/30506383910_dc52ee1fd3.jpg) (https://flic.kr/p/NtKc7o)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 06, 2016, 12:00:15 PM
On the Mighty 50D I get : (reported write speed in parenthesis)

in crop mode 1920*1080
~1100 frames in 14 bits (~77 Mbps)
Continuous in 12 bits (75.7 Mbps)
Continuous in 10 bits (74.7 Mbps)

in Full sensor (1568*1046)
Continuous in 14 bits (72.7 Mbps)
Continuous in 12 bits (74.6 Mbps ???)
Continuous in 10 bits (75.8 Mbps ??????)

So obviously, the reported bitrate is wrong, I'll have to check the recorded files
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilguercio on November 06, 2016, 12:30:41 PM
Any way to try the build for 6D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 06, 2016, 03:47:05 PM
Indeed, the numbers displayed are estimations of your write speed, not the bitrate of the recorded movie. The latter is easy to compute: width x height x bpp/8 * fps [MB/s].

The stdev of those estimations is a little high for my taste, but not a major issue.

However, a second look over the code uncovered a bug (present in the old raw_rec and mlv_lite): the speed estimation is only used for next recording if global draw is off. Easy to fix.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: anDyIII on November 06, 2016, 03:52:15 PM
I'm testing 10 and 12bit raw on my 5D Mark III, but why do I have MLV files if I use raw 1.0 to record?
Is it possible to add Status when recording to raw 1.0 to control in a better way how the recording is going?
Thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 06, 2016, 04:41:21 PM
Quote from: anDyIII on November 06, 2016, 03:52:15 PM
...why do I have MLV files if I use raw 1.0 to record?

Go to the module menu and check raw_rec. On the raw_video_10bit_12bit branch it should report "Raw video v1.1 (MLV Lite)"

Quote from: ilguercio on November 06, 2016, 12:30:41 PM
Any way to try the build for 6D?

That's included in the testing package I put in my bitbucket download area:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/raw_video_10bit_12bit_2016Nov04.zip

Note that I'm going to leave this up for a limited time. I may or may not update as changes come in depending on the demand of the testers. I'd really rather everyone search for my compiling tutorials and roll their own.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: extremelypoorfilmaker on November 06, 2016, 05:19:14 PM
just following the conversation :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 06, 2016, 05:36:05 PM
Quote from: dfort on November 06, 2016, 04:41:21 PM
Note that I'm going to leave this up for a limited time. I may or may not update as changes come in depending on the demand of the testers. I'd really rather everyone search for my compiling tutorials and roll their own.

Agreed -- this is a great way to keep all of our projects moving forward in a nice pace otherwise!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: anto on November 06, 2016, 06:33:56 PM
first, great!!! very impressive....
now I need a guidehow to use mlv_dump, please  ::)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tin2tin on November 06, 2016, 06:41:29 PM
Quotemlv_dump:
You need to disable all processing, e.g.
mlv_dump M00-1234.MLV --dng --no-fixcp --no-stripes
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 06, 2016, 09:25:17 PM
Or you can use MLVFS (http://www.magiclantern.fm/forum/index.php?topic=13152.msg127218#msg127218)  on PC or Mac no update needed.
I'm using  "Quick Mount" on PC with the old Pismo/mlvfs.dll (just right click on file & quickmount  :D )
I think it's about 1 year old extraction workflow and has no issue
with the 10 & 12bit raw files .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 06, 2016, 11:47:39 PM
Wonderful  job Guys

Tested on 5d mark III 113

Crop mode:

3584 x 1320 10bit 24fps 8/9 sec. (106.2 mbps)
3584 x 1320 12bit 24fps 4/5 sec. (106.2 mbps)
2880 x 1310 10bit 24fps Continous  (104.8 mbps)
2880 x 1080 12bit 24fps Continous  (104 mbps)

no frames corrupted

I cant belive myself i'm very very happy

Thanks very mutch for your job.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 07, 2016, 10:17:51 AM
this is awesome! thanks dfort for pointing me to this. live view still freezing on the eos m? just tried 1472x884 resolution in crop mode on eos m in 10 bit. looks very nice and detailed. just live view is freezing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 07, 2016, 10:29:50 AM
I am testing this on 5d3 113. I am really shocked, with the old PC mlvfs (with pismo) 10 and 12 bits works like a breeze :) I am using resolve 12.5, sometimes only the first frames are corrupted but the rest is perfect. 10 and 12 bits recording is really a huge boost in recording times.

EDIT:

The most important and useful modes for me now:

1920x648 (1920x1080 16:9 after desqueeze) 50 fps 12bit - continous - (previously only 2,35:1 was continous)
1920x648 (1920x1080 16:9 after desqueeze) 60 fps 10 bit - 25 sek

1920x1080 37 fps 12 bit - 13 sek
                              10 bit - continous

2560x1320 25 fps (max vertical resolution available) 1.94:1 aspect - 10 bit - continous

2880x1226 2,35:1 25fps 10bit  - over 2 minutes
                              24 fps 10 bit - continous

I checked those recordings in resolve and sometimes there are first and last frames corrupted but no problems during clips.

FEATURE REQUEST:

in mlv rec module there is possibility to choose more resolutions, could somebody please add those resolutions to the 10/12bit raw rec module?
3200x1320 (2.42:1)
3168x1320 (2.40:1)
3104x1320 (2,35:1)

There might be some good recording times in 10 bit so we coul utilize all available pixels in vertical axis.

                                                                         
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 07, 2016, 01:41:41 PM
Guys,

could anyone of you share compiled 5Dm3.113 version?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 07, 2016, 01:48:50 PM
Couple posts above...

https://bitbucket.org/daniel_fort/magic-lantern/downloads

there is raw video 10 12 bit zip, you have 5d3 113 inside the zip...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 07, 2016, 02:42:00 PM
For fine resolution increments, you just have to port this change (https://bitbucket.org/hudson/magic-lantern/pull-requests/627/added-fine-control-on-top-of-x-resolution/diff) to MLV Lite.

The change should be simple enough for anyone with minimal C skills (perfect for new contributors looking for something easy to get started).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 07, 2016, 02:49:37 PM
Quote from: kgv5 on November 07, 2016, 01:48:50 PM
Couple posts above...

https://bitbucket.org/daniel_fort/magic-lantern/downloads

there is raw video 10 12 bit zip, you have 5d3 113 inside the zip...

I cannot find module for 5D Mark 2. That means there is not a proper settings available for 5D2 yet where every 2nd frame is not pink?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on November 07, 2016, 03:35:19 PM
Quote from: Deadcode on November 07, 2016, 02:49:37 PM
I cannot find module for 5D Mark 2. That means there is not a proper settings available for 5D2 yet where every 2nd frame is not pink?


Check this post : http://www.magiclantern.fm/forum/index.php?topic=5601.msg174302#msg174302

Just replaces the module in your ML installation : ML / modules / raw_rec.mo


But yes, with the 5DII version there are pink frames issues. 10 and 12bits
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 07, 2016, 06:33:21 PM
Works great with 5Dm3 and fps override set to 37fps. Much better then 48fps in squeezed resolution. Any plans adding this to mlv_rec module, so that we could use audio as well?
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 08, 2016, 07:15:27 AM
@kaco -- How is 37p much better than 48p?

In terms of Motion (aliasing/moire) or Aspect Ratio/Resolution?

It may seem more ideal to go w 37p to avoid the desqueeze in post (resolution loss) but it's very minimal especially w 5D3 that it shouldn't even be a concern IMO.

Do you have any samples that show otherwise?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 08, 2016, 08:27:47 AM
Quote from: Deadcode on November 07, 2016, 02:49:37 PM
I cannot find module for 5D Mark 2. That means there is not a proper settings available for 5D2 yet where every 2nd frame is not pink?

Quote from: GutterPump on November 07, 2016, 03:35:19 PM
Just replaces the module in your ML installation : ML / modules / raw_rec.mo

There were some changes to the back ends so it isn't just a matter of dropping in the module. Though I don't think that will solve the every other (pink) frame issue.

Sorry, I must have missed it while compiling all the platforms. I just put a new build on my bitbucket download area.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 08, 2016, 08:56:01 AM
@DeafEyeJedi -- in 48/50p the vertical resolution is 1.67x smaller, which kills details and causes aliasing and moire. Most of wide angle shots are quite bad.

Switching to canon 25/24p mode and then overriding to 37fps means much more detail for me. BIG chance for me, crippled resolution in 48p mode was a reason for me to start looking for a new camera.

look at the aliased edges:
https://vimeo.com/148944224
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on November 08, 2016, 09:30:00 AM
Quote from: kaco on November 08, 2016, 08:56:01 AM
@DeafEyeJedi -- in 48/50p the vertical resolution is 1.67x smaller, which kills details and causes aliasing and moire. Most of wide angle shots are quite bad.

Switching to canon 25/24p mode and then overriding to 37fps means much more detail for me. BIG chance for me, crippled resolution in 48p mode was a reason for me to start looking for a new camera.

look at the aliased edges:
https://vimeo.com/148944224


Your work is just amazing ! Never saw your stuff before.
How did you grade it ? Which softs ? which luts (if used) ?
I dont saw informations.

For answer you about aliasing and moire on 50 and 60fps footage due to stretching, yes it's a small issue, but believe me, most of 95% people don't will see anything.
And for be honest, on your footage i'm not focused on it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 08, 2016, 11:29:03 AM
@GutterPump -- thanks, it's graded in Davinci. And really, using 490px vertical resolution in a 4K world is not so acceptable for professional work. :) This is game changing.

On topic -- is this going to be a part of mlv_rec? Is this in raw_rec module just during development?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 08, 2016, 12:24:06 PM
Results here with the 50D :

First frame
(http://www.justinbacle.com/temp/seq0.png)
Even frames
(http://www.justinbacle.com/temp/seq1.png)
Odd frames (except first one)
(http://www.justinbacle.com/temp/seq2.png)

Is it the same problem as the 5DmkII ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 08, 2016, 12:56:50 PM
Quote from: kaco on November 08, 2016, 08:56:01 AM
@DeafEyeJedi -- in 48/50p the vertical resolution is 1.67x smaller, which kills details and causes aliasing and moire. Most of wide angle shots are quite bad.

Switching to canon 25/24p mode and then overriding to 37fps means much more detail for me. BIG chance for me, crippled resolution in 48p mode was a reason for me to start looking for a new camera.

You should check out the crop_rec module that is in development, it allows you to record in 48/50p without the vertical squeeze.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 08, 2016, 03:41:32 PM
Quote from: rbrune on November 08, 2016, 12:56:50 PM
You should check out the crop_rec module that is in development, it allows you to record in 48/50p without the vertical squeeze.

@rbrune Thanks for the tip, I've already tested it. But AFAIK the preview in live view would be squeezed (1.64x) in vertical resolution? And because of the bitrate with full 1920x1080, without 10/12 bit I would not be able to get 48p continuos, not?
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: aschille84 on November 08, 2016, 03:46:16 PM
@kaco I tried it yesterday, you can have 672 px max vertical resolution. So you can get something like 1600x672 in 50/60p unsqueezed footage. You have to change aspect ratio close to 1:1 to get max horizontal resolution
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 08, 2016, 04:11:12 PM
Quote from: aschille84 on November 08, 2016, 03:46:16 PM
@kaco I tried it yesterday, you can have 672 px max vertical resolution. So you can get something like 1600x672 in 50/60p unsqueezed footage. You have to change aspect ratio close to 1:1 to get max vertical resolution

So 672px is limit for continuos shooting or is it a hard limit? If that's maximum (regardless bitrate), then this is not so great and a true 1920x1080 (or 1920x818) in 37fps is much more interesting.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aschille84 on November 08, 2016, 04:32:57 PM
Quote from: kaco on November 08, 2016, 04:11:12 PM
So 672px is limit for continuos shooting or is it a hard limit? If that's maximum (regardless bitrate), then this is not so great and a true 1920x1080 (or 1920x818) in 37fps is much more interesting.
Yes, its a hard limit regardless of bitrate. And I agree 37fps full HD is more useful and interesting.


Sent from my iPhone using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 08, 2016, 04:35:05 PM
Quote from: justinbacleIs it the same problem as the 5DmkII ?

Yes, and I also get it on the 550D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 08, 2016, 07:19:43 PM
Quote from: rbrune on November 08, 2016, 12:56:50 PM
You should check out the crop_rec module that is in development, it allows you to record in 48/50p without the vertical squeeze.

Quote from: kaco on November 08, 2016, 03:41:32 PM
@rbrune Thanks for the tip, I've already tested it. But AFAIK the preview in live view would be squeezed (1.64x) in vertical resolution? And because of the bitrate with full 1920x1080, without 10/12 bit I would not be able to get 48p continuos, not?

@rbrune -- Have you tried merging your crop_rec with raw_video_10bit_12bit? If you have a branch for that in your repository I can compile a new set of test builds. I'd do it myself but I'm not nearly the coder you are and am busy with some other projects at the moment.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 08, 2016, 08:08:00 PM
Quote from: dfort on November 08, 2016, 07:19:43 PM
@rbrune -- Have you tried merging your crop_rec with raw_video_10bit_12bit? If you have a branch for that in your repository I can compile a new set of test builds. I'd do it myself but I'm not nearly the coder you are and am busy with some other projects at the moment.
I had not but gave it a quick try. Recording in 10bit with 3x3 mode worked for me on the EOSM (of course without live preview, that still needs fixing on the EOSM).
Here you go:
https://bitbucket.org/rbrune/magic-lantern/branch/crop_rec_raw_video_10bit_12bit (https://bitbucket.org/rbrune/magic-lantern/branch/crop_rec_raw_video_10bit_12bit)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 08, 2016, 08:30:34 PM
Quote from: dfort on November 08, 2016, 07:19:43 PM
... I can compile a new set of test builds. I'd do it myself but I'm not nearly the coder you are and am busy with some other projects at the moment.

I replied in your thread (http://www.magiclantern.fm/forum/index.php?topic=16012.msg174580#msg174580your%20thread) about a possibility to give you a helping hand with this task. Thanks @rbrune for sharing your branch!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 08, 2016, 09:13:42 PM
Nice!

Ok--what's next, taking a1ex's hint (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174497#msg174497) and adding more of rbrune's wonderful work like added fine control on top of x-resolution presets and display of effective crop factor (https://bitbucket.org/hudson/magic-lantern/pull-requests/627/added-fine-control-on-top-of-x-resolution/diff) for MLV Lite? I was about to look into that this morning but got stuck in focus pixel land (http://www.magiclantern.fm/forum/index.php?topic=16054.msg174582#msg174582) with the EOSM crop_rec video mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 08, 2016, 11:45:42 PM
Quote from: rbrune on November 08, 2016, 08:08:00 PM
I had not but gave it a quick try. Recording in 10bit with 3x3 mode worked for me on the EOSM.

After playing with it some more I noticed that my EOSM crashes (screen freezes and I can hear the lens/AF working but no reaction to button presses) when I try to record a second 10bit mlv after a first one (with/without crop_mode).
Can someone confirm this? (just record a 10bit mlv stop, wait a couple second and immediately record a second one).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 09, 2016, 12:39:36 AM
Quote from: rbrune on November 08, 2016, 11:45:42 PM
After playing with it some more I noticed that my EOSM crashes (screen freezes and I can hear the lens/AF working but no reaction to button presses) when I try to record a second 10bit mlv after a first one (with/without crop_mode).
Can someone confirm this? (just record a 10bit mlv stop, wait a couple second and immediately record a second one).

My EOSM is crashing on the first 10bit mlv. I reported it a few days ago (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174349#msg174349).

QuoteIt works fine in the crop modes, mv1080crop and zoom with the same LiveView freeze and first frame corrupt issue as the 700D. There's something strange going on with the non-cropped modes. The camera records but the camera freezes when recording is stopped requiring a battery pull.

I just tried it with your crop_rec in 10bit and same thing. Works fine in Movie crop mode (600D movie crop-mode) and zoom mode. Well, if you accept LiveView freezing as "works fine."
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 09, 2016, 03:29:05 AM
@a1ex
QuoteFor fine resolution increments, you just have to port this change (https://bitbucket.org/hudson/magic-lantern/pull-requests/627/added-fine-control-on-top-of-x-resolution/diff) to MLV Lite.

The change should be simple enough for anyone with minimal C skills (perfect for new contributors looking for something easy to get started).

Got it working. Since the request was made on this topic I made the pull request for the raw_video_10bit_12bit branch:

https://bitbucket.org/hudson/magic-lantern/pull-requests/767/raw_rec-mlv-lite-now-handles-resolution/diff

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 09, 2016, 02:36:33 PM
@dfort, are you able to compile a module for 5d3 113 with fine resolution changes?
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 09, 2016, 08:23:29 PM
@kgv5 -- here's what I was able to compile today for a few of my DSLR's so far (be aware that Dual-ISO isn't included as it is still temporiarly broken) on my environment until then. all cameras and they are currently available for download (check below).

5D3.113, 7D.203 & EOSM.202 for now -- https://mega.nz/#F!PgFFQKxY!KR7Odshgs-HmL5o1EM5Z-g

New link here -- https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads

Thanks to @rbune for sharing your branch and to @dfort for the PR in getting crop_rec.mo & 10/12-bit merged together.

@Everyone else -- I'll eventually compile for the rest once I've finally resolved with the Dual-ISO module issue on this Mac.

*edit*

This has finally been fixed on my Mac. Got it compiled for all cameras running ML except for 1100D.105 & 5D3.123 (will eventually figure out how to get this merged for 5D3.123) and special Thanks goes out to @dfort for getting me back on track with the compiling environment.  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lars Steenhoff on November 09, 2016, 08:33:45 PM
Thanks for the build, will test it now on the 5dmk3

Main thing I'm looking forward to is to see next is in camera playback, its nice to verify the shots are on the card.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 09, 2016, 08:51:02 PM
Based on my recent tests with the mlv_playback.mo from these builds on 5D3.113 & 7D.203 which seems to playback files like normal.

Even if they (10/12-bit files) are corrupted. Nice!  :P

(https://c8.staticflickr.com/6/5477/30892960415_5e7f7a48b4_n.jpg) (https://flic.kr/p/P4UuQx)

However with the EOSM.202 -- if I try to playback an MLV file recorded with either 10 or 12 bit it would load (the blue 'loading' graphic line) up to the very moment when we would expect to see the footage to no avail.

(https://c1.staticflickr.com/6/5549/30582188200_5ba731d0a4_n.jpg) (https://flic.kr/p/NArH5U)

It still stays on the end of the 'Blue loading graphic' within the GUI -- is this what you are (or were) seeing as well @Lars Steenhoff?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 09, 2016, 09:07:38 PM
Quote from: kgv5 on November 09, 2016, 02:36:33 PM
@dfort, are you able to compile a module for 5d3 113 with fine resolution changes?

Sure, here you go.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit_fine_x_resolution.2016Nov09.5D3113.zip

I haven't tested it on that camera yet but now raw_rec should fine adjust just like it does in mlv_rec.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 09, 2016, 10:04:59 PM
Tested on 5d mark III 113
every things works ok
no frame corrupted

Crop Mode

3200x1200 23fps 10 bit Continous  (105.7 mbps)

3.5k Video Continous it's Amazing

Thank you very much  dfort
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 09, 2016, 10:25:40 PM
Thank you very much guys  :D This is incredible.

My measurments:

Everything tested with 24fps 10bit on 5d3 113, CF sandisk extreme pro 128gb 160mb/s

3168x1320 2.40:1 (1.81x crop) - 14 sec

3104x1320 2.35:1 (1.85x crop) - 18 sec
3104x1298 2.39:1 (1.85x crop) - 21 sec

3008x1280 2.35:1 (1.91x crop) - 38 sec
3008x1258 2.39:1 (1.91x crop) - 56 sec

2976x1266 2.35:1 (1.93x crop) - 60 sec
2976x1246 2.39:1 (1.93x crop) - 123 sec

2880x1226 2.35:1 (2x crop) - continous

And wider aspect ratios - just for fun:

3520x1320 2.67:1 (1,63x crop) - 6 sec
3296x1320 2.50:1 (1.74x crop) - 12 sec

Now I would like to see the comparison: 5d3 vs 5d4 - raw 10bit 3008x1258 (quite usable recording time - 56 sec) vs MJPEG 8bit 4k. Not so much different crops - 1.91x 5d3 vs 1.7x 5d4. Similar bitrates. That would be very interesting.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 09, 2016, 10:38:45 PM
Quote from: DeafEyeJedi on November 09, 2016, 08:51:02 PM
However with the EOSM.202 -- if I try to playback an MLV file recorded with either 10 or 12 bit it would load (the blue 'loading' graphic line) up to the very moment when we would expect to see the footage to no avail.

Can you upload a small MLV that shows the issue?

I assume the same MLV (recorded on EOS M) will fail to play in a different camera as well; can you check whether this is true?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 09, 2016, 11:00:16 PM
Quote from: a1ex on November 09, 2016, 10:38:45 PM
Can you upload a small MLV that shows the issue?

I assume the same MLV (recorded on EOS M) will fail to play in a different camera as well; can you check whether this is true?

Sure here you go (https://mega.nz/#!vw0z2SZY!L7EpC9MtIjrCTH9UlL6qdmaQcN4kQESAJbn4Gl5cVUohere%20you%20go) -- it's the same exact file (269.1 MB) from the screenshot above (M09-1246.MLV) and I can confirm that playing this particular file on a 7D which DOES playback fine until at the end which then I am prompted with this message...

(https://c8.staticflickr.com/6/5606/30885240615_020b72b310_n.jpg) (https://flic.kr/p/P4dW1x)

My gut feeling is telling me this may has to do with EOSM freezing while being recorded (even tho the file is still be written into card) and the only way to get the LiveView back on is to take the battery out after each take. Not sure if this would be helpful?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 09, 2016, 11:14:39 PM
The last few frames appear to have some trouble. Recording was stopped normally, or it crashed?

edit: looks like it crashed.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 10, 2016, 12:01:48 AM
I think we need to get the ML black and white preview working on the EOSM - the incorrectly setup LV for 10/12bit is probably messing with the camera and making it crash.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 10, 2016, 12:14:37 AM
With preview set to Canon, do you still get crashes?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 10, 2016, 08:23:18 AM
More testing on 5D Mark ii , by playing around with the SRM Job , SmallHacks & Preview I can now get 10bit 16x9 A.R. 1856x1044 29.97(68MB/s) , 23.976p(55MB/s)
continuous without any Frame corruption/pink noise  . I never have Raw Video 30p continuous before at near HD 1044p  :D
My settings:
29.97p(Set in Canon Menu)
Frame over ride - exact
Small Hacks - enabled
SRM Job      - disabled
Preview ML B/W with overlays killed when recording
Any other setting then this produce corrupted frames.

If you enable SRM Job you get the first frame corrupted with pink noise but all the rest are free of corruption.
with SRM Job disabled all frame including the first are free of corruption.

Now with 12bit I almost got it to product corruption free frame , there just a little color noise bar at the top of every other frame about 40 lines
Setting:
23.976p(continuous) (29.97p records for only 20-30 seconds)
Small Hacks - enabled
SRM Job      - enabled
Preview Canon with overlays killed when recording.

Curious that in 10bit Raw Video ,  Liveview needs to be in magic lantern black & white preview with overlays disabled
where 12bit Raw Video needs Liveview to be in Canon preview to be almost corruption free , any other preview cause total pink noise

I'll post some examples later .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 10, 2016, 08:36:15 AM
Quote from: a1ex on November 10, 2016, 12:14:37 AM
With preview set to Canon, do you still get crashes?

When set to Canon the EOSM crashes when stopping the first 10bit recording.
With ML B&W or Hacked it crashes when starting the second 10bit recording but the first records fine.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 10:11:52 AM
Quote from: rbrune on November 10, 2016, 08:36:15 AM
When set to Canon the EOSM crashes when stopping the first 10bit recording.
With ML B&W or Hacked it crashes when starting the second 10bit recording but the first records fine.

I couldn't reproduce any of the crashing (unless you guys are referring to the LiveView freezing) but that's not exactly crashing, is it? especially if it's being recorded blindly.

However upon reading @reddeercity post above re: SRM job memory -- decided to try and fiddle with it and now for some reason if I were to record in either 10 or 12 bit on the EOSM with the SRM job memory enabled OFF then mlv_play.mo doesn't load. (notice in this screenshot below that the invalid header error is at the beginning as oppose to the previous issue (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174661#msg174661previous%20issue))

(https://c4.staticflickr.com/6/5695/30804715771_1eb67a32c0_n.jpg) (https://flic.kr/p/NW7dMa)

Although it had issues earlier and now if I were to record in either 10 or 12 bit on EOSM with the SRM job memory enabled ON then mlv_play.mo does load as well as playback files like normal.

(https://c8.staticflickr.com/6/5477/30892960415_5e7f7a48b4_n.jpg) (https://flic.kr/p/P4UuQx)

Really?!? What's even more odd is the fact that now all of sudden I can playback files on EOSM when earlier I couldn't with that particular file (probably due to a crash while recording) but I honestly don't recall any crashes happening on my watch during this time other than the usual LiveView freeze (don't remember if I kept it rolling till dropped frames or if I had stopped recording myself) but then after trying to reproduce this glitch to no avail and now fiddling with this SRM job memory somehow did the trick or no?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 11:14:02 AM
got some (probably) stupid questions... Is 8 bit raw a thing? you know to achieve higher resolutions? how about monochrome black and white mode for the same reason? has anyone tried using an external monitor to see if that is also freezing while live view is freezing on 10 bit? I'd try it myself but i don't have any monitors or cables to do so. I'm talking about eos m. thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SiSS on November 10, 2016, 01:44:33 PM
I think the consensus is that 10-bit acquisition helps you for noticeably better end results than just upping resolution.
There's more to maneuver with 10-bit. Every smartphone and cheap action camera has 4K = and that doesn't mean those deliver anything worthwhile.
Also it's been said that difference between 8-bit and 10-bit is "last" to be noticed with your eyes (10-bit 12-bit not so much, tough there's the material to work with).
Real 10-bit and you have think C300 mk II and C500. Arri Alexa didn't have 4K until SXT and even that doesn't come from 4K sensor. SXT been out for month? So 99% of the films in the movie theaters still don't have native 4K. Tough 10-bit doesn't mean everything either.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SiSS on November 10, 2016, 01:49:52 PM
Quote from: kgv5 on November 09, 2016, 10:25:40 PM
Everything tested with 24fps 10bit on 5d3 113
2880x1226 2.35:1 (2x crop) - continous
Thanks kgv5, did you notice any difference in moire and/or rolling shutter?
Some test videos please!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 10, 2016, 02:32:28 PM
Little late at the party  :P
I never thought this was gonna happen, that the hardware in the cams simply were fixed for 14 bit.

But :o WOW :o Respect for these findings!!!  8)

So to put things in perspective...
Canon released a full-frame DSLR in 2008!!!  with some sort of full HD video feature in h.264 compression.
And all these time this cam could record 10 bit raw video in full HD continuously direct to the memory card  ;D
Canon could have saved some serious money by leaving out the chip that does the h.264 conversion(unless that is done by the digic  :P)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tullen on November 10, 2016, 05:11:26 PM
Hi. I admit I havent read through the whole thread, so forgive me if I missed it, but could this work on the 50D? Since I understand it has similarities with the 5D2.

Cheers
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 10, 2016, 05:34:42 PM
Quote from: Tullen on November 10, 2016, 05:11:26 PM
Hi. I admit I havent read through the whole thread, so forgive me if I missed it, but could this work on the 50D? Since I understand it has similarities with the 5D2.

Right now 1 out of 2 frames are garbage on the 50D. Same as the 5Dmk2 :)
Preliminary tests gives FullHD crop mode continuous w/ the 50D though
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 10, 2016, 05:56:36 PM
Quote from: justinbacle on November 10, 2016, 05:34:42 PM
Right now 1 out of 2 frames are garbage on the 50D. Same as the 5Dmk2 :)
Really !
Did you even Read any Of the post here or just decided to write Wrong & miss leading
Information ? This Is a big problem here on the forum with new users , not willing to use
Searches or read threads & post and expect someone to give them information .
And Yea FYI 5d2 works with 10bit corruption Free
Give your Facts right !
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 10, 2016, 05:56:56 PM
5D2, Crop mode, 1920x1076 25p 10/12 bit continous. First frame broken, others have strong magenta cast (MLVFS, Davinci)

Any ideas?

https://drive.google.com/file/d/0B9d46mzo8guUQkN1YTRCRnhsdTA/view?usp=sharing (https://drive.google.com/file/d/0B9d46mzo8guUQkN1YTRCRnhsdTA/view?usp=sharing)

reddeercity: would you please share the proper 5D2 settings for non crop mode? i tried what you posted days ago (10bit 1856x1044, ML Grey, Clear overlay) but it didnt worked, every 2nd frame corrupted
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 10, 2016, 06:06:17 PM
Quote from: reddeercity on November 10, 2016, 05:56:36 PM
Really !
Did you even Read any Of the post here or just decided to write Wrong & miss leading
Information ? This Is a big problem here on the forum with new users , not willing to use
Searches or read threads & post and expect someone to give them information .
And Yea FYI 5d2 works with 10bit corruption Free
Give your Facts right !

Sorry but there is no fix yet for the 50D. And post http://www.magiclantern.fm/forum/index.php?topic=5601.msg174548#msg174548 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174548#msg174548) said that the pink frames shouldn't be solved.
However there is a file https://bitbucket.org/daniel_fort/magic-lantern/downloads (https://bitbucket.org/daniel_fort/magic-lantern/downloads) specific to the 5dmk2, based on your reply, I guess it fixes pink frames on the 5dmk2.
Sorry if I gave wrong information, just trying to help here.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 10, 2016, 06:07:49 PM
Check my post at top of this page I gave my setting
That only works with 10bit right now . Try that if it dosn't work I can post more details.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 10, 2016, 06:42:03 PM
@Deadcode to fix magenta cast/black level on crop mode with 5D2 10bit use exiftool:
exiftool -BlackLevel=108 *.dng
This command fixes every frame in the directory, and for 12 bit the black level would be 4* that probably
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 10, 2016, 07:01:24 PM
Quote from: reddeercity on November 10, 2016, 06:07:49 PM
Check my post at top of this page I gave my setting
That only works with 10bit right now . Try that if it dosn't work I can post more details.
Tried your fix, at first it looked like it worked but it actually doesn't on my 50D. The part that used to be "noise" is now a static frame.
On this example, the top part is flickering but is not what should have been recorded. One every two frame is "splitted" like that.
(http://www.justinbacle.com/temp/M10-1949.png)
Both your fixes produced similar results (10 & 12 bits).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 10, 2016, 09:04:14 PM
Quote from: reddeercity on November 10, 2016, 06:07:49 PM
Check my post at top of this page I gave my setting
That only works with 10bit right now . Try that if it dosn't work I can post more details.



I tried exactly what you wrote on that post and still corrupted frames
29,970 exact, 1856x1044 10bit, Ml Grey, Clear when recording, SRM disabled, Small Hacks enabled.
I tried 12 bit setup too for 23,976p, i get the 40pixel corruption + the bottom half of the image is "wobbing".

Any suggestion?

*i used magiclantern-raw_video_10bit_12bit.2016Nov07.5D2212.zip
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 09:09:38 PM
Quote from: justinbacle on November 10, 2016, 07:01:24 PM
Tried your fix, at first it looked like it worked but it actually doesn't on my 50D. The part that used to be "noise" is now a static frame.
On this example, the top part is flickering but is not what should have been recorded. One every two frame is "splitted" like that.
(http://www.justinbacle.com/temp/M10-1949.png)
Both your fixes produced similar results (10 & 12 bits).

I can also confirm this so called 'splitted' or 'slicing' across the frame horizontally occurring on some of my 7D files while shooting 10/12-bit. I'll need to investigate furthermore to determine what is the culprit in here (fps, mv1080/mv720, etc) and will report my findings when I can.

Once I can reproduce short test files then I'll upload them to share.

Quote from: Quote from: kgv5 on Yesterday
@dfort, are you able to compile a module for 5d3 113 with fine resolution changes?

As promised -- I've managed to get it compiled with the latest 10/12-bit raw_rec.mo (includes finer resolution adjustment) similar to what you get in mlv_rec.mo merged together with crop_rec.mo -- special thanks goes to @dfort for making this possible on my system! 

All cameras running ML except for 1100D.105 & 5D3.123 (I plan on figuring out a way to get it compiled for 5D3.123 at some point) and remember to keep all bug reports on this thread if you plan on testing these builds out. Enjoy!

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 09:11:34 PM
this 10 bit raw is the best thing that ever happened. i don't think I've ever seen such a good image coming out of a camera. so excited for live view to be working on eosm. I've got an a7sii sitting in my room. i haven't even used it in a month. much prefer the image on this 100 dollar camera. absolutely hilarious.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 09:20:06 PM
@deafeyejedi can't seem to get your link to work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 10, 2016, 09:33:07 PM
I get "Access Denied."
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 09:36:23 PM
same here access denied.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 09:47:21 PM
i actually never got your mega link to work either lol. we gotta get this guy online. valuable guy :P there's bitbucket settings on the top right do you have "private profile" checked?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 10:07:33 PM
Ah, ha. Thanks for the hints and @dfort was also able to help me out by pointing me to fixing this by going to Admin > Access > Disable Private (I don't recall turning this ON so this must be a default thing, right?).  :o

In any case -- could you guys please let me know if the link works now and Thanks again @Walter Schulz for dealing with yet another one of these moments.  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 10:09:58 PM
Looks good now
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 10:11:56 PM
Quote from: Teamsleepkid on November 10, 2016, 10:09:58 PM
Looks good now

Thanks, Mate!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: canneloni on November 10, 2016, 10:26:31 PM
Is there still no build for the 100D ? I would like to test it and help as much as i can. It's really great what you guys are doing!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 10:28:18 PM
Actually @dfort and I are still waiting to hear back from @nikfreak (http://www.magiclantern.fm/forum/index.php?topic=16054.msg174586#msg174586hear%20back%20from%20@nikfreak) on this very possibility (for both 100D & 70D) and will compile them as soon as we can -- we are just as anxious as you all are!   ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 11, 2016, 01:43:51 AM
@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test  ::)
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry :-[

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here (https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing)  .




Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 11, 2016, 02:33:04 AM
Using MLVFS still getting every other half-pink frame. 550D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 11, 2016, 03:37:36 AM
Sorry guy should be Canon preview with overlays killed when recording for corrupt free @1044p 5D2
Edit:
30p is unstable , I can't now get it to record corruption free on every recording at the moment , some times it does so 50/50
but 23.976p is , sorry for the confusion I'll do some research  .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on November 11, 2016, 04:03:16 AM
No pro@reddeercity
Me too waiting to hear back from nikfreak for the 70D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 11, 2016, 09:07:33 AM
Quote from: reddeercity on November 11, 2016, 01:43:51 AM
@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test  ::)
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry :-[

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here (https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing)  .



Still not working with these settings. I will make a video around 8pm (GMT+1 :) ) about it from installing ML till opening the dng's in Davinci, maybe you can spot the wrong settings.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Steve_FR on November 11, 2016, 07:45:03 PM
(2016Nov09.EOSM202)

I don't experience any crashing on the Eos M when recording with either zoom mode or 3x crop mode enabled.
Video playback works too, in both color and b&w. Video is scrambled.  Raw @ 10bit. Tried with SRM Job ON and OFF.
I can record multiple clips back to back with no crash, aside from the expected preview frozen only during REC.

If I shoot in (mv720 I believe?) (no crop, no zoom) then I experience the crash mentioned, and have to pop battery. I tried with SRM Job on and off, and the crash still occurred, followed by the invalid header size error when playing back the clip that was attempting to be recording during the crash. I can still play the other clips fine that were recorded using 3x or zoom, but scrambled as expected.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 11, 2016, 09:57:26 PM
Quote from: reddeercity on November 11, 2016, 01:43:51 AM
@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test  ::)
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry :-[

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here (https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing)  .


I made a video about the process. What am i missing?

https://youtu.be/VI1u8Ouk_r8 (https://youtu.be/VI1u8Ouk_r8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 11, 2016, 11:32:36 PM
@Deadcode @reddeercity I have also tried all the settings you list and many other options, I can't get it to work without half corrupt frames, or splitting/slicing on the lower half, and I have the same build situation on my camera. But it is definitely amazing for crop mode, as it can be full res 2144x1076 and continuous :D :D :D :D :D :D :D :D :D :D :D How might this issue be fixed, and what help can the users give?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 12, 2016, 12:15:53 AM
@Deadcode , @Ilia3101 Ok I'll post a short video/images to how to get it working.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 12, 2016, 01:06:44 PM
Tested the 4 november version of Daniel on the 6d.
It works, sort of, although image preview is halted, the 'allow global draw' option is missing in the menu ?
And i'm missing sound, but that's probably normal for Raw_rec module  ;)

First frame is always corrupted.
10 bit works
12 bit works
But it doesn't work all the time, there was also a MLV file with all corrupted frames and a MLV file with frames with vertical lines corrupted, pink/magenta vertical lines and a posterized image behind it.
I've used MLV_dump.osx for getting the DNG's and viewed them in RawTherapee.

The vertical line correction in MLV_Dump always seems to use exactly 1.000 or 1 numbers with the 10bit files, the files look ok, but is it explainable that it is always exactly 1.000 ?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 12, 2016, 09:23:32 PM
Quote
Enabling the new method on all models is desirable IMO, because we will have a single way of doing things. However, each model is likely to have its own quirks, so it's a bit of work.
Trying to get slurp working on EOSM. I assume all I need to do is find a free edmac write channel and then hardcode the DEFAULT_RAW_BUFFER?

The hint in the code is

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
*
* raw_buffer = get_raw_buffer()
* sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
* ...
* dump_file(filename, raw_buffer, 7*something...)
*/


I looked around in the disassembly of lv_raw_dump and called functions, but couldn't find any hardcoded values that looked like the other raw buffer addresses, at the beginning of lv_raw_dump EOSM appears to call two functions in RAM (not sure where to find these in ROM).

So I figured I would just call("lv_raw_dump"). Got a file on the card:
46798080.mm1

Also doesn't look much like the other addresses. (60D gave: 48332200.MM1, in raw.c the value is hardcoded as 0x5028)

Any tips?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 12, 2016, 10:33:25 PM
On 60D, the raw buffer address is returned by FF0F8B8C.

On EOS M, the function you are looking for is 10BF4. Add RAM_OFFSET from stubs.S =>  ffa7a184. This one returns MEM(0x404E4 + 0x44).

The raw buffer address is probably allocated dynamically (or maybe just hardcoded) from RscMgr.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 13, 2016, 05:23:49 AM
@Deadcode look like your setup is ok , well after some time testing different configurations on 5D2 I can't seem now to get 10 or 12 bit working consistently
ever time . When the camera & card are cold (no operated for at least 3 Hours) I can record 10 or 12 bit 1856x1044 @ 24p & or 30p but can't repeat the corrupt free
recording and when it does work I notice a message on the very top LCD screen "busy" that flashes on and off for about the first 20 of so seconds then is gone .

So 10bit @ 1856x1044 23.976p & 29.97p are continuous if you can get a corrupt free recording .
of which I did a few times . Plus I was able to get 12bit @ 1856x1044 23.976p , 29.97p at this bit depth that can not be maintain for very long.

Below I have posted three links to my Google Drive where I have 3 Corrupt Free recordings of 10bit 1856x1044 @ 23.976 & 29.97p & one of 12bit 1856x1044 @ 23.976p
In the drive folder there is for each bit depth & frame rate the Original .MLV file 1.0Gb , 1.1Gb , 550Mb the True 10 & 12 bit DNGs exacted with mlv_dump . There about
75 dngs for each file .

Google drive links
12bit 1856x1044_23.976p (https://drive.google.com/drive/folders/0B3rFQHNAb91MX2V0am1CYXNNRGs?usp=sharing)     10bit 1856x1044_29.97p (https://drive.google.com/drive/folders/0B3rFQHNAb91MbWE1T2gwdy11eHM?usp=sharing)    10bit 1856x1044_23.976p (https://drive.google.com/drive/folders/0B3rFQHNAb91MTU5hOE5hS1JKWm8?usp=sharing)

Now that bring me to the bit depth we have , there seem to be a problem with 12bit as Resolve or ACR/A.E. can't read or understand 12bit from mlv_dump.
10bit seen to read ok in A.E. & Resolve just 12bit. Why does this matter if you use MLVFS , yes  it does extract the cdngs ok from 10 & 12bit mlv's But
these are 16bit files (3.75Mb per frame) where as the 10 bit are (2.34Mb) & 12bit (2.8Mb) . So in some cases you are saving up to 1Mb per frame which can add up .
Plus the other factor is from 10bit (Billions of Colors) to 16bit(Trillions of Color) is the color accurate or will the color shift etc... .

Screen shot of A.E. CS6 with 12bit , can't read 12bit from mlv_dump 

(https://c4.staticflickr.com/6/5797/25311866219_fc85c3438c.jpg) (https://flic.kr/p/EyHVYx)
ML_Raw_12bit_dngs_AE-CS6 (https://flic.kr/p/EyHVYx) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Screen shot of A.E. CS6 ACR with 12bit , can't read 12bit from mlv_dump 

(https://c1.staticflickr.com/6/5683/30831184632_ec787db8ed.jpg) (https://flic.kr/p/NYrT3Q)
ML_Raw_12bit_dngs_ACR-AE-CS6 (https://flic.kr/p/NYrT3Q) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr


Screen shot of Blackmagic Resolve with 12bit , can't read 12bit from mlv_dump 

(https://c3.staticflickr.com/6/5825/30831184642_b4660b7183.jpg) (https://flic.kr/p/NYrT41)
12.5_Blackmagic_Resolve_12bit_dng (https://flic.kr/p/NYrT41) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Edit:

Screen Shots Blackmagic Resolve with True 10 bit from mlv_dump

(https://c6.staticflickr.com/6/5609/30949677445_6da74f3424.jpg) (https://flic.kr/p/P9VbQn)
12.5_Blackmagic_Resolve_10bit_dng (https://flic.kr/p/P9VbQn) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr


Screen Shots A.E. CS6 Timeline with True 10 bit from mlv_dump

(https://c4.staticflickr.com/6/5654/30949677475_7025edfccc.jpg) (https://flic.kr/p/P9VbQT)
ML_Raw_10bit_dngs_AE-CS6 (https://flic.kr/p/P9VbQT) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr


Screen Shots A.E. CS6 ACR with True 10 bit from mlv_dump

(https://c8.staticflickr.com/6/5654/30949677495_7c621cb37a.jpg) (https://flic.kr/p/P9VbRe)
ML_Raw_10bit_dngs_ACR-AE-CS6 (https://flic.kr/p/P9VbRe) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 13, 2016, 09:34:31 PM
I'm not a scientist but 10 bit looks great in resolve 12.5 using mlvfs after white balance in the raw tab on eos m.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 14, 2016, 12:10:27 AM
Quote from: Teamsleepkid on November 13, 2016, 09:34:31 PM
I'm not a scientist but 10 bit looks great in resolve 12.5 using mlvfs after white balance in the raw tab on eos m.
Sure it does because it's not 10bit anymore it's 16bit Cdng , that's point I was getting at MLVFS takes the 10bit image and up-samples
to 16bit even thou there's not 16 bits (well really 14bit) of information plus the file get blown up by at least 1Mb per frame.

Exact with mlv_dump for true 10bit and see if there's any difference .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 14, 2016, 12:47:43 AM
There will be no difference. It's not an "up-sample". The values are exactly the same. The number 123 represented with 10 bits or 16 bits or 64 bits or a million bits is still the same value, 123. CPUs can't do math on 10 bit numbers so it's going to get converted to 16 (or maybe even 32 or 64) internally at some point regardless.

Also, MLVFS files are virtual, so they are not actually taking up any additional real space.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 14, 2016, 03:35:08 AM
I tried a couple different edmac channels, but I'm just getting garbage, and with crop hack LV just freezes. Frame sizes are the ones reported by raw_info correct? That was the only other thing I needed to do?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 14, 2016, 09:54:32 AM
Frame sizes must be either hardcoded in raw_lv_get_resolution, or autodetected (as done on 5D3 on the crop_rec branch; you may have to tweak the width multiplier, as it's very likely to be lower than 8 ).

Once that is done, make sure raw_lv_vsync runs once for every frame (you can place bmp_printf's with small fonts there) and that raw_type_register is 0xC0F37014 (value for digic 5). Default raw type appears to be 0x22. Then, you could check whether edmac_slurp_complete_cbr gets called.

On 550D iirc, I couldn't get a raw stream until setting the raw type register.

You could also try the following changes:

raw_write_chan = 0x12
dmaFlags = 0x20000000
StartEDmac(raw_write_chan, 2)


Also, can you check the configuration of EDMAC channel #18 (Debug -> Show EDMAC) after enabling the LiveView raw stream with the old method?

The vsync hook looks ok to me.

A dm-spy log from LiveView might also help.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 14, 2016, 09:58:36 AM
Alex you talking about trying to get live view working?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on November 16, 2016, 04:37:55 PM
Any updates for the 5DM2?
Where can I find the latest builds?

Thank you all - you are the best  8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hjfilmspeed on November 18, 2016, 02:29:42 PM
This is AWESOME!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 19, 2016, 10:01:36 PM
Quote from: a1ex on November 14, 2016, 09:54:32 AM
You could also try the following changes:

raw_write_chan = 0x12
dmaFlags = 0x20000000
StartEDmac(raw_write_chan, 2)

That worked!

Very first try I got an Early Stop(5) and:

[89] raw_rec_task: NULL PTR (80c3e100,e1a00000)
pc=ff2c6020 lr=  ac5ad0 stack=1ddde0+0x1000
entry=abef78(0)
e1a00000 e59ff014 e59ff014 e59ff014
e59ff014 e1a00000 e59ff010 e59ff010


Everything worked fine after that, I did about 5 recordings in normal mode and about 5 in crop hack (I don't think there are any other video modes I can actually try with EOSM?). Also briefly tested 10 bit, appears to be working fine and Canon preview also working.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 19, 2016, 10:10:39 PM
All of those 3 changes are needed, right?

(I'm still unsure what EDMAC flags to use when, and how they affect the result...)

To diagnose the null ptr error, the mem_prot module can be helpful, as it can show the exact location that performed the invalid write. Or, a way to reproduce. There are some reports showing it on 60D as well.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 19, 2016, 10:32:40 PM
I only tried with all 3, if I get some more time I'll try excluding them.

For anyone interested in doing further testing here's what I did (on top of current raw_video_10bit_12bit). To check what a1ex asked about above simply change the two #ifdef CONFIG_EOSM lines in the edmac_raw_slurp() function to #if 0 (try each one separately and both)


diff --git a/platform/EOSM.202/internals.h b/platform/EOSM.202/internals.h
--- a/platform/EOSM.202/internals.h
+++ b/platform/EOSM.202/internals.h
@@ -142,3 +142,6 @@

/** Workaround for menu timeout in LiveView */
#define CONFIG_MENU_TIMEOUT_FIX
+
+/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
+#define CONFIG_EDMAC_RAW_SLURP
diff --git a/src/edmac-memcpy.c b/src/edmac-memcpy.c
--- a/src/edmac-memcpy.c
+++ b/src/edmac-memcpy.c
@@ -388,6 +388,10 @@
uint32_t raw_write_chan = 4;
#endif

+#ifdef CONFIG_EOSM
+uint32_t raw_write_chan = 0x12;
+#endif
+

static void edmac_slurp_complete_cbr (void* ctx)
{
@@ -401,7 +405,11 @@
void edmac_raw_slurp(void* dst, int w, int h)
{
     /* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
+#ifdef CONFIG_EOSM
+    uint32_t dmaFlags = 0x20000000;
+#else
     uint32_t dmaFlags = 0x20001000;
+#endif
     
     /* @g3gg0: this callback does get called */
     RegisterEDmacCompleteCBR(raw_write_chan, &edmac_slurp_complete_cbr, 0);
@@ -420,6 +428,10 @@
     SetEDmac(raw_write_chan, (void*)dst, &dst_edmac_info, dmaFlags);
     
     /* start transfer. no flags for write, 2 for read channels */
+#ifdef CONFIG_EOSM
     StartEDmac(raw_write_chan, 0);
+#else
+    StartEDmac(raw_write_chan, 2);
+#endif
}
#endif /* CONFIG_EDMAC_RAW_SLURP */
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -114,6 +114,10 @@
//~ #define DEFAULT_RAW_BUFFER MEM(0x25f1c + 0x34)  /* 123 */
#endif

+#ifdef CONFIG_EOSM
+#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
+#endif
+
#else

/* with Canon lv_save_raw, just read it from EDMAC */
@@ -430,6 +434,12 @@
     *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
     return 1;
     #endif
+   
+    #ifdef CONFIG_EOSM
+    *width  = video_mode_crop ? 1872 : 1808;
+    *height = video_mode_crop ? 1060 : 727;
+    return 1;
+    #endif

     /* unknown camera? */
     return 0;
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 19, 2016, 10:47:45 PM
I guess another thing someone can try now that slurp is implemented is to change the raw type register to see if you can get a focus-pixel-free raw stream (looking at you @dfort).


#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */

just change #undef to #define
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 20, 2016, 12:00:44 AM
Test build pretty please?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 20, 2016, 07:30:20 AM
Quote from: dmilligan on November 19, 2016, 10:47:45 PM
(looking at you @dfort).

Yikes!

Ok--first I pulled in the latest commits and dmilligan's patches have already been applied, check.

Then I went into src/raw.c and change one line to:

#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */

Unfortunately it doesn't compile for the EOSM:

[ CC       ]   raw.o
../../src/raw.c: In function 'raw_update_params_work':
../../src/raw.c:590:10: warning: #warning FIXME: are these values correct for 720p and crop modes? [-Wcpp]
         #warning FIXME: are these values correct for 720p and crop modes?
          ^
../../src/raw.c: At top level:
../../src/raw.c:2070:18: error: 'lv_raw_type' undeclared here (not in a function)
         .priv = &lv_raw_type,
                  ^
make: *** [raw.o] Error 1


I thought I tracked down the problem down to this:

#ifdef PREFERRED_RAW_TYPE
static int lv_raw_type = PREFERRED_RAW_TYPE;
#endif


So it looks like we need to define PREFERRED_RAW_TYPE which is commented out for the 650D and 700D:

/**
* RAW_TYPE 78 (and others) have a frame-wide green pattern on them
* ACR can clear them but also kills some details and does not do
* a good job in general. TL;DR Use pink dot remover.
* http://www.magiclantern.fm/forum/index.php?topic=6658.0
*/

/*
#ifdef CONFIG_700D
#define PREFERRED_RAW_TYPE 78
#define RAW_TYPE_ADDRESS 0x351B8
#endif

#ifdef CONFIG_650D
#define PREFERRED_RAW_TYPE 78
#define RAW_TYPE_ADDRESS 0x350B4
#endif
*/


We can do better than the old pink dot remover with MLVFS.  8)

In any case, I've got a 700D so I uncommented that section but it came up with the a similar error:

[ CC       ]   raw.o
../../src/raw.c:2065:6: error: #error Only implemented for CONFIG_EDMAC_RAW_SLURP.
     #error Only implemented for CONFIG_EDMAC_RAW_SLURP.
      ^
../../src/raw.c:2070:18: error: 'lv_raw_type' undeclared here (not in a function)
         .priv = &lv_raw_type,
                  ^
make: *** [raw.o] Error 1


@Teamsleepkid and other testers who haven't set up a development enviornment yet -- here's an updated package with the latest changes for all the platforms (except 1100D) and the modded mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Nov19.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 20, 2016, 02:10:23 PM
You just need to define PREFFERED_RAW_TYPE for the EOSM. a1ex mentioned a few posts back that the default appears to be 0x22. Start from there and the use the menu to find the best one. You might want to search the forum for this topic, IIRC this was done a long time ago for other cameras, and there was some recent discussion about changing it for 5D3 to get rid of vertical banding. You might also want to split this discussion off into a new topic.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: glubber on November 20, 2016, 06:55:06 PM
Did some tests on the 550D with latest build from dfort:

12bit is now stable in 1080p25 and 720p50 (fps override to 24 of course)!


(https://c7.staticflickr.com/6/5512/30316190854_8afcbb259e_n.jpg) (https://flic.kr/p/NbWpkL)12bit-1080p25 (https://flic.kr/p/NbWpkL)

(https://c4.staticflickr.com/6/5701/31137213675_eb89b1e692_n.jpg) (https://flic.kr/p/PrumPD)12bit-720p50 (https://flic.kr/p/PrumPD)


10bit still results in every second frame corrupted

(https://c2.staticflickr.com/6/5732/31137213585_83b0323d36_n.jpg) (https://flic.kr/p/PrumN6)10bit-720p50 (https://flic.kr/p/PrumN6)


5x zoom creates a camera crash (Err70)

ASSERT: dwDeadLine != 0xFFFFFFFF
at LVState.c:5842, task LiveViewMgr
lv:1 mode:20


Magic Lantern version : Nightly.2016Nov19.550D109
Mercurial changeset   : b56f3ee02864 (raw_video_10bit_12bit) tip
Built on 2016-11-20 06:00:40 UTC by rosiefort@RosieFoComputer.
Free Memory  : 196K + 1678K


Hope this helps in taking the 10/12 bit fork further.
To all involved: keep up the awesome work  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 20, 2016, 07:26:02 PM
I tried the new builds on the 5D2 and pink frames are gone. Just split image on every other frame, about half of the frame for 10 bit is covered and most of it for 12 bit, the split part of the frame is always the same, seems to be from the first frame or something. Tested in 1856x1044 16:9 and 1856x1250 3:2 in 23.976fps.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 20, 2016, 08:24:37 PM
So close. 550D 12 bit 8 seconds at 1600x600 (2.67:1) but frames out of order. They're numerically in order but not correct. Like they are numbered wrong in camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 12:57:31 AM
The only real changes from the previous version pertain only to EOSM.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 21, 2016, 01:17:51 AM
Quote from: dmilligan on November 20, 2016, 02:10:23 PM
You just need to define PREFFERED_RAW_TYPE for the EOSM. a1ex mentioned a few posts back that the default appears to be 0x22.

It seems that we also need to find RAW_TYPE_ADDRESS for the EOSM in order to get this working.

#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#define RAW_TYPE_ADDRESS 0xDEADBEEF
#endif


Otherwise it won't compile:

[ CC       ]   raw.o
../../src/raw.c: In function 'raw_update_params_work':
../../src/raw.c:595:10: warning: #warning FIXME: are these values correct for 720p and crop modes? [-Wcpp]
         #warning FIXME: are these values correct for 720p and crop modes?
          ^
In file included from ../../src/dryos.h:47:0,
                 from ../../src/raw.c:21:
../../src/raw.c: In function 'raw_lv_vsync':
../../src/raw.c:1538:42: error: 'RAW_TYPE_ADDRESS' undeclared (first use in this function)
         uint32_t raw_type_register = MEM(RAW_TYPE_ADDRESS-4);
                                          ^
../../src/mem.h:78:38: note: in definition of macro 'MEM'
#define MEM(x) *(volatile uint32_t*)(x)
                                      ^
../../src/raw.c:1538:42: note: each undeclared identifier is reported only once for each function it appears in
         uint32_t raw_type_register = MEM(RAW_TYPE_ADDRESS-4);
                                          ^
../../src/mem.h:78:38: note: in definition of macro 'MEM'
#define MEM(x) *(volatile uint32_t*)(x)
                                      ^
make: *** [raw.o] Error 1


Entering the wrong address will simply lock up the camera (requiring battery pull) when activating raw_rec.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 01:46:09 AM
I'm not sure what the address is, but I don't think you really need it since with this method there's no need to patch Canon code. The raw_type_register should be 0xC0F37014 according to a1ex.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 21, 2016, 02:15:49 AM
Well without an RAW_TYPE_ADDRESS defined it won't compile. That address you gave me produced this ASSERT LOG:

ML ASSERT:
raw_type_register == 0xC0F08114 || raw_type_register == 0xC0F37014
at ../../src/raw.c:1539 (raw_lv_vsync), task Evf
lv:1 mode:3


Magic Lantern version : Nightly.2016Nov20.EOSM202
Mercurial changeset   : b56f3ee02864+ (raw_video_10bit_12bit)
Built on 2016-11-21 00:51:03 UTC by rosiefort@RosieFoComputer.
Free Memory  : 195K + 3406K


I doubt that's the right address for the EOSM. It looks like the address was found for the 5D3 and that's probably the only platform that will work with the "#define RAW_DEBUG_TYPE" trick:

#define RAW_TYPE_ADDRESS 0x2D168

I tried that address on the EOSM and it locks up the camera when activating raw_rec requiring a battery pull.

I've also got a 700D I could use to test this but that camera doesn't have CONFIG_EDMAC_RAW_SLURP enabled and getting that working is beyond me at the moment.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 03:03:20 AM
I didn't say that was the RAW_TYPE_ADDRESS, I said it was the raw_type_register, and that you don't really need RAW_TYPE_ADDRESS. Look carefully at the code (specifically the code where it's used and where the failed assert you posted references)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 21, 2016, 05:00:15 AM
Quote from: dmilligan on November 21, 2016, 03:03:20 AM
...you don't really need RAW_TYPE_ADDRESS...

If I understand your instruction, this should work:

#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#endif


However, it won't compile.

Changing this line from #undef to #define will also not compile which is how we got into defining PREFERRED_RAW_TYPE.

#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */

I'm probably missing something obvious but the only way I've gotten #define RAW_DEBUG_TYPE to compile is to also come up with a value for RAW_TYPE_ADDRESS.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 21, 2016, 06:54:54 AM
thank you thank you thank you everyone :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 21, 2016, 10:36:36 AM
Did some testing with the 19 november build and new MLV_dump on the Canon 6d

First frame 00000.dng = always corrupted.

10 bit normal mode and crop mode-> always good
12 bit normal mode and crop mode-> sometimes wrong (diagonal purple stripes along the frames) sometimes good
I'm trying to figure out why 12 bit sometimes works and sometimes won't work ?


About the newest MLV_dump (I'm on osx), it only works with 10 bit.
In 12 bit mode, the newest MLV_dump finds about 40000 cold pixels, which results in frames with 40000 colored and black specles in them.
The same MLV files processed with MLV_dump version of may 2015 are fine.
In 10 bit mode, the newest MLV_dump finds zero cold pixels and gives normal frames.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 21, 2016, 10:40:56 AM
The live view freezes during recording in 10 and 12 bit.
Do you think this will be fixable, is canon live view capable of showing 10/12bit stream ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 01:04:13 PM
Quote from: dfort on November 21, 2016, 05:00:15 AM
If I understand your instruction, this should work:

#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#endif


However, it won't compile.

You have to do more than just #define stuff, you will have to change the existing code, but it should be easy (I was not saying that you won't have to do anything at all, just that you don't have to do any reverse engineering to find that memory address, a simple code change should avoid that work). Look where RAW_TYPE_ADDRESS is actually being used and remove it. If you know the register value you don't need the address, because the address is only used to look up the register. This is why I told you to look carefully at the code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 01:18:03 PM
Quote from: Levas on November 21, 2016, 10:40:56 AM
The live view freezes during recording in 10 and 12 bit.
Do you think this will be fixable, is canon live view capable of showing 10/12bit stream ?
Implementing CONFIG_EDMAC_RAW_SLURP fixed the issue on EOSM.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 21, 2016, 05:24:00 PM
Quote from: dmilligan on November 21, 2016, 01:04:13 PM
...This is why I told you to look carefully at the code.

I can also look carefully at an airplane but I can't fly it.

Ok--figured it wasn't going to be this easy:

Quotejust change #undef to #define

I'll keep looking.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domasa on November 22, 2016, 03:59:19 PM
Quote10 bit normal mode and crop mode-> always good
Do you see some real danger of damage camera 650D after use current build (19.11.) for 1 hour continuously  10-bit RAW recording?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 23, 2016, 12:32:29 AM
I have no experience with such long recording times.
But I can't imagine that it can cause permanent damage.



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 23, 2016, 09:53:05 AM
Quote from: Levas on November 23, 2016, 12:32:29 AM
I have no experience with such long recording times.
But I can't imagine that it can cause permanent damage.

Maybe not to the camera but to the viewer!

Just wanted to report that I'm getting great performance out of the latest EOSM changes. About 20 sec. with 10-bit at 1728x692. But hey, this is the EOSM which doesn't have mv1080 and it is running at 29.97 fps. With FPS override set to 23.98 fps it is continuous. Live View is good and no corrupt frames.

I've gotten it to compile with "#define RAW_DEBUG_TYPE" but still haven't been successful in bringing up the debug menus. Got to keep looking at the code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 23, 2016, 11:24:08 PM
QuoteAbout 20 sec. with 10-bit at 1728x692
Continuous when shooting 24fps.
*oops, already stated above.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on November 24, 2016, 12:43:20 AM
Not sure if the first post is still active regarding testing for MLV Lite?

Just tried on latest ML build with 5D3 and got dropped frames after 3 sec. Not sure where it kicked in. Res was maxed 2:35.1.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 24, 2016, 04:21:03 PM
Messing around with compiling magic lantern for 6d
and was wondering about the bit modes:

Probably already tried by other users, but did somebody test what happens when you use values of 0x020 or 0x040 ?

+    /* for PACK16_MODE, DSUNPACK_MODE, ADUNPACK_MODE (mask 0x131) */
Add a comment to this line
+    const uint32_t MODE_16BIT       = 0x130;
+    const uint32_t MODE_14BIT       = 0x030;
+    const uint32_t MODE_12BIT       = 0x010;
+    const uint32_t MODE_10BIT       = 0x000;
+    const uint32_t bit_modes[] = { MODE_14BIT, MODE_12BIT, MODE_10BIT };
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 24, 2016, 05:25:44 PM
#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
Quotejust change #undef to #define

Ok--so it wasn't that simple but eventually it compiled.

Quote from: dmilligan on November 21, 2016, 03:03:20 AM
Look carefully at the code

static struct menu_entry debug_menus[] = {
    {
        .name = "LV raw type",
        .priv = &lv_raw_type,
        .max = 64,
        .help = "Choose what type of raw stream we should use in LiveView.",
        .help2 = "See lv_af_raw, lv_rshd_raw, lv_set_raw, KindOfCraw...",
    },
};


There it is!

(https://c4.staticflickr.com/6/5538/31215434515_25d49d5d65.jpg)

So here's what I'm doing to get there--obviously still a work in progress:

https://bitbucket.org/daniel_fort/magic-lantern/branch/10bit_12bit_experiments#commits
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 24, 2016, 08:22:59 PM
Thanks for this work being done and especially now on the eos M. I did some testing with the raw stream from debug menu. Ran a recording on stream 64-45 but all of them exhibits the same focus pixel pattern. Are we certain streams are actually changed? How many streams are there anyway?
Also want to highlight the crop rec 3x3 720p mode which will yield superior H.264 HDR footage on the eos M and of course even better with the 5D mark III. Here is my workaround to output correct aspect ratio through ffmpeg.
http://www.magiclantern.fm/forum/index.php?topic=15108.msg175407#msg175407
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 24, 2016, 09:14:45 PM
I might have found a way to use 16px (or even 8px) increments horizontally without crashing Digic 5 cameras. The trick was to make sure W * H (yes, the product) is multiple of 16 bytes. The previous assumption was that W must be multiple of 8 bytes, and since H was always an even value, this appeared to work. Then, I've noticed 1904x1072 works fine, while 1872x1054 crashes. In both cases, (W*14/8) mod 8 is 4, so according to my previous hypothesis, they both should crash. Also, 1872x1072 works, 1872x1052 works, 1870x1052 fails, 1870x1056 works. See the pattern?

Still wondering why it works that way.

Committed on the 12-bit branch. This should make pixel (http://www.magiclantern.fm/forum/index.php?topic=5533.msg106435#msg106435) peepers (http://www.magiclantern.fm/forum/index.php?topic=11205.350) happy ;)

Quote from: Danne on November 24, 2016, 08:22:59 PM
I did some testing with the raw stream from debug menu. [...] Are we certain streams are actually changed?

If you add a variable (with a menu) that is not used anywhere else, guess what will happen ;)

Check commit c3488df.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on November 25, 2016, 12:47:18 AM
Is there any build with crop + 10 12 bit + fine resolution? I find crop + 10 12 bit, also find 10 12 bit + fine resolution...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 25, 2016, 01:08:44 AM
Wow! happy to see the higher resolution back on the 5D2, it works at 1872 pixels, but it has a bug that allows it to go up to 1879, and once it has gone up there, it goes down to odd numbers like 1863, and the weird numbers cause recording to stop after 3 seconds and then all kinds of 'corruption' messages appear. But 1872 works!!!!! only tried 14 bit so far :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 25, 2016, 01:21:26 AM
Good catch, pushed a fix. Try it at all bit depths ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 25, 2016, 02:28:45 AM
Quote from: a1ex on November 24, 2016, 09:14:45 PM
I might have found a way to use 16px (or even 8px) increments horizontally without crashing Digic 5 cameras.  Also, 1872x1072 works, 1872x1052 works, 1870x1052 fails, 1870x1056 works. See the pattern?
This should make pixel (http://www.magiclantern.fm/forum/index.php?topic=5533.msg106435#msg106435) peepers (http://www.magiclantern.fm/forum/index.php?topic=11205.350) happy ;)
Thanks A1ex  , Cool The Mighty 5D2 Gets it's wings back  :D
Funny  :P  that's one of the mean reasons I started to get a compiling machine (Ubuntu) up and running , made my day !
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 25, 2016, 09:09:52 AM
compiled new update from a1ex on 5d2 , yes I'm back to 1872x1054 , 10bit thou was all corrupted (colored lines )12bit was almost perfect
every other frame had the top 100 lines or so shifted over a bit . This was a quick test used Auto preview for liveview that was good no lockup or freeze there .
Sample Cdng's 12bit 1872x1054 (https://www.dropbox.com/sh/zv2ee400cjt68id/AAC4Kp5GSRu5BAZ4bCBW-8aka?dl=0)  Sample all corrupted Cdng's 10bit 1872x1054 (https://www.dropbox.com/sh/39p7m7tzzf26wvw/AABgiyAhzI0GbaDcxtGCOnxra?dl=0) from my dropbox
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 25, 2016, 09:44:23 AM
QuoteIf you add a variable (with a menu) that is not used anywhere else, guess what will happen  :P
Not much I guess  :P
And I need to learn C...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 25, 2016, 01:24:39 PM
5D2 update: whatever changes happened overnight, it has turned 1872 in to 1880, but now there are constant messages all over the screen saying things I don't understand, but it does work, 1880x1056 works, so does 1880x1248, but not 1880x1250, which seems to fit the pattern.
(http://gdurl.com/ftU2)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on November 25, 2016, 01:39:18 PM
1250 is not divisible by 16? See A1ex post up a bit.^ I dunno thou.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: NedB on November 25, 2016, 09:16:49 PM
Guys - sorry for (stupid?) question (not a beginner, been using ML since 2012). Downloaded and installed dfort's 10-bit/12-bit build from Nov. 19 for my 550D. But, try as I might, I cannot find where to set the bit depth to 10-bit or 12-bit for testing purposes. Anybody wanna cut me some slack and point it out explicitly for me? Does it require certain modules to be loaded or other settings to be set correctly in order to appear? Thanks, gentlemen!

NedB


Edit: Thank you very much for insanely quick reply, Levas!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 25, 2016, 09:20:53 PM
10 and 12 bit options are only available in the raw_rec module and not available in the probably more used MLV_rec module.
So enable raw_rec module and unleash the power  ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 25, 2016, 11:05:15 PM
First of all -- thanks to dmilligan for having the patience to work with me on this.

NO MORE FOCUS PIXELS!

(https://c4.staticflickr.com/6/5607/30431903123_21e0ee762d_z.jpg)

Here's what does the trick--only working on the EOSM at this time:

#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 18
#endif


This is a before shot (using PREFERRED_RAW_TYPE 34)

(https://c1.staticflickr.com/6/5522/31203782576_d42de1350a_z.jpg)

and here is what it looks like with PREFERRED_RAW_TYPE 18

(https://c6.staticflickr.com/6/5503/31125394261_7f266185aa_z.jpg)

I'm still running some tests and haven't gotten it working with the crop_rec module yet but I wanted to share the news.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 25, 2016, 11:33:31 PM
Holy shit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lars Steenhoff on November 26, 2016, 12:15:55 AM
Great work  guys!  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 26, 2016, 12:23:24 AM
You guys are the real Focus Pocus!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on November 26, 2016, 02:43:36 AM
updated mlv_rec and mlv_play for bit depth selection.
mlv_play requires raw_twk (http://magiclantern.fm/forum/index.php?topic=13163.0) if you want 10/12 bit playback.

on my 5D3 setup recording still is a bit buggy, every other frame is corrupted.
anyone can confirm?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on November 26, 2016, 03:42:55 AM
5D2: I can confirm every other frame is corrupted in 10 and 12 bit mlv_rec, top half of the image in 10 bit and a small bar of noise at the top for 12 bit. mlv_play + raw_twk does work with 10 and 12 bit, but everything is very green and crushed, probably black level.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 03:49:36 AM
Quote from: g3gg0 on November 26, 2016, 02:43:36 AM
...every other frame is corrupted...

I don't have a 5D3 to run the test I just did with the EOSM but I saw lots of PREFERRED_RAW_TYPE settings that had the alternating frame corruption issue you are describing. I went through all of the 64 possible settings until I found the one that doesn't show focus pixels. Maybe it is time to recheck the PREFERRED_RAW_TYPE setting on other cameras too?

Checked out your latest changes to mlv_dump but I'm getting a "Segmentation fault: 11" when turning on "--cs2x2". It looks like this isn't an issue specific to the 10bit_12bit branch.

http://www.magiclantern.fm/forum/index.php?topic=7122.msg167507#msg167507
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 26, 2016, 05:54:40 AM
@dfort great stuff the only thing missing is MV1080 , I think I'll keep my eye on a good used EOSM  :)
@g3gg0 same as @Ilia3101 on my 5D2 even tried different Liveview previews setting even the ones that freeze up Liveview
Also tried  29.97p fps in canon menu 1856x1044 raw mlv. Still the same as 24p

Corrupted Frame 10bit 1856x1044 23.976p MLV
(https://c5.staticflickr.com/6/5580/30423382844_c5ccf57e01_n.jpg) (https://flic.kr/p/NmpMMw)
M25-2137_000545 (https://flic.kr/p/NmpMMw) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Good Frame 10bit 1856x1044 23.976p MLV
(https://c7.staticflickr.com/6/5662/30877145430_d1ed1a0ae0_n.jpg) (https://flic.kr/p/P3vrAo)
M25-2137_000546 (https://flic.kr/p/P3vrAo) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Corrupted Frame 12bit 1856x1044 23.976p MLV
(https://c5.staticflickr.com/6/5559/31100842892_ac433d85bf_n.jpg) (https://flic.kr/p/PogX4b)
M25-2139_000000 (https://flic.kr/p/PogX4b) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Good Frame 12bit 1856x1044 23.976p MLV
(https://c7.staticflickr.com/6/5483/31100841422_9d13150edf_n.jpg) (https://flic.kr/p/PogWBQ)
M25-2139_000001 (https://flic.kr/p/PogWBQ) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 07:43:54 AM
Quote from: reddeercity on November 26, 2016, 05:54:40 AM
@dfort great stuff the only thing missing is MV1080

The really great stuff for the EOSM is happening with @rbrune here:

http://www.magiclantern.fm/forum/index.php?topic=16608.msg175360#msg175360

A camera you can pick up for $150-$200 USD that can shoot 10-bit raw video--and you can adapt just about any lens from a C-mount to a PL-mount?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 08:51:53 AM
Did a quick test with the updated mlv_rec from g3gg0 on the 6d and got ZERO corrupted frames with 10bit.
Recorded 3 MLV files in 10bit and all frames are fine, even the first ones.

I used mlv_dump.osx (build from may 2015) and all frames look fine in Rawtherapee.



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 26, 2016, 08:53:34 AM
no focus pixels!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 08:53:41 AM
Forget to mention.
The canon liveview freezes during recording.
The ML grayscale doesn't freeze during recording, but doesn't look like it can handle 10bit right now.

Is this normal, does canon live view freezes with the use of 10bit on all camera models ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 08:56:01 AM
Gonna shoot some files on my 5D mark III soon.
QuoteChecked out your latest changes to mlv_dump but I'm getting a "Segmentation fault: 11" when turning on "--cs2x2". It looks like this isn't an issue specific to the 10bit_12bit branch.
This issue is solved in unified so it,s in the 10-bit/12-bit code?

http://www.magiclantern.fm/forum/index.php?topic=7122.msg174592#msg174592
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 26, 2016, 09:30:10 AM
Quote from: Levas on November 26, 2016, 08:51:53 AM
...I used mlv_dump.osx (build from may 2015)...

Really?

Quote from: Levas on November 26, 2016, 08:51:53 AM
...and all frames look fine in Rawtherapee.

Care to share screenshots?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 09:34:58 AM
Quote from: Danne on November 26, 2016, 08:56:01 AM
This issue is solved in unified so it,s in the 10-bit/12-bit code?

Yes--so it looks like somehow this commit (https://bitbucket.org/hudson/magic-lantern/commits/33843737460f1b0323a7b46fc0e81198ea44ab9b) didn't make it into the 10bit_12bit branch. Just applied it and --cs2x2 works fine.

This is needed to get rid of the hot pixels that sometimes appear in high contrast boundaries. Not sure if those are only in cameras that have (had?) the focus pixel issue.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 10:17:32 AM
g3gg0, you are king. And major thanks to Greg for work on this via DARK code? http://www.magiclantern.fm/forum/index.php?topic=13408.msg172108#msg172108. Who did I forget?
Just added 10/12-bit branch into unified and added the raw_twk.mo from here http://www.magiclantern.fm/forum/index.php?topic=13163.msg174991#msg174991
and playback is more or less realtime with all bits 10,12,14. it even works on the eos M!, just tested briefly.

Quoteon my 5D3 setup recording still is a bit buggy, every other frame is corrupted.
anyone can confirm?
I get corrupted files(white level off I think) on my 5D mark III every now and then in both 10 and 12 bit. Also from time to time a text with "raw detect error"(I think it says) on the left. Canon eos M is the more stable participant right now it seems(thanks dmilligan, a1ex, dfort and others)
Shortened examples of corrupted files
test_1
https://drive.google.com/file/d/0B4tCJMlOYfircGtCRlNSWFFGU2M/view?usp=sharing
test_2
https://drive.google.com/file/d/0B4tCJMlOYfirckxRMmJqMm1EaFE/view?usp=sharing
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on November 26, 2016, 12:30:22 PM
@Danne:
happy to hear that :)

yeah the black/white level thing is maybe still an issue. tried to fix it by scaling white level properly when upconverting to 16 bit in mlv_play.
but maybe there is still smth twisted.

does mlv_lite perform better with recording, i.e. no corrupted frames?
didnt test t.b.h.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 12:44:04 PM
Only tested with mlv lite. Is 10 bit working with mlv_rec?? Not home to test atm.
One quirk with eos m is that play, pause, delete buttons are suppressed when running playback through raw_twk.mo. Buttons works great with 5D mark III.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on November 26, 2016, 01:25:24 PM
i updated it also, so yes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 03:26:32 PM
Will check later. Most greatful.
If anybody wants to try here are builds for 5D mark III and eos m with latest stuff. Raw_twk.mo included.(removed for now, causing corrupted black level)

Briefly tested. If anything breaks you get to keep all pieces.

*Black level issues fixed by g3gg0. New builds up
https://bitbucket.org/Dannephoto/cr2hdr/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 26, 2016, 04:48:00 PM
Quote
Will check later. Most greatful.
If anybody wants to try here are builds for 5D mark III and eos m with latest stuff. Raw_twk.mo included.
Briefly tested. If anything breaks you get to keep all pieces.
https://bitbucket.org/Dannephoto/cr2hdr/downloads

Hi i tested the last build on 5d Mark III sometimes it work correctly
but often the video is completely black
i send you a dng : https://www.dropbox.com/s/odn6xxrubhup6hz/M28-1717_000000.dng?dl=0



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 05:12:15 PM
@baccinoluigi
Any difference in stability between mlv_rec and mlv_lite?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 05:15:16 PM
Had some time to do some more mlv_rec testing on 6d.

Tested 10bit, 12bit and 14bit.
10bit works, for all frames and both old mlv_dump and newest mlv_dump works.
12bit works, only old mlv_dump gives purple vertical lines, newest mlv_dump detects false cold pixels and gives frames with too much specles because of the false cold pixel detection.
14bit, still works ;D

I'm using rawtherapee to view the files.

Uploading some dng files right now (few minutes uploading remaining)
http://drive.google.com/open?id=0B1BxGc3dfMDabVFDeG11QjBmWlk (http://drive.google.com/open?id=0B1BxGc3dfMDabVFDeG11QjBmWlk)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 26, 2016, 05:23:15 PM
@Danne

Tested again but mlv_rec and mlv_lite have the same issue

14 bit is more stable
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 07:30:17 PM
Still wondering, how is liveview during recording on other cams ?
On the 6d, canon liveview freezes during 10bit recording, Grayscale preview is messed up during 10bit recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 26, 2016, 09:16:50 PM
@baccinoluigi:

Workaround:

exiftool *.dng -BlackLevel=128


Is the issue specific to a particular build, or all of them are buggy?

@Levas:

Your question was already answered by dmilligan earlier in this thread.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: robertgl on November 26, 2016, 09:23:06 PM
Hello 50D user here
1. Magenta frames. It shows up on 10,12 and 14 when using the available modified fw. I haven't tried to change the black levels with mlv_dump, assuming that's the way to do it

2. The first frame keeps showing up in the even frames. So it's like 0,1,0,3,0,5,0,7,0,9,

3. at 1856x*** I get full frames interspersed with that continuing first frame showing up in even frames, while if I go to 1920x**** I get torn frames  with most of the frame being the first frame repeating, while a small sliver is the properly sequenced frame

4. Grayscale preview seems to run fine when recording

5. fingers crossed that the bugs can be ironed out because recordings look amazing and have a smaller files size  8)

Am exporting with : mlv_dump --dng --no-fixcp --no-stripes
debayering in rawtherapee
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on November 26, 2016, 09:29:56 PM
testing on 700D, builded from official hudson repository, raw_video_10bit_12bit branch about 10 minutes ago.

Found a "BUG", setting full HD res, no crop mode, going over the limit it uses max res possible on camera (1736x976) not divisible by 16 so crash happens after a few secs recording, this in both mlv_rec and raw_rec modules.

here logs after that crash
https://drive.google.com/open?id=0BxMUvr5E8ANaMDZUbkJnTHFya28  LOG000.log
https://drive.google.com/open?id=0BxMUvr5E8ANaci12Y2xUSU1MREE CRASH00.log

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 26, 2016, 09:44:04 PM
@a1ex
QuoteIs the issue specific to a particular build, or all of them are buggy?

The issue it's just in the build downloaded here : https://bitbucket.org/Dannephoto/cr2hdr/downloads
whith raw_twk.mo for 5dMkIII
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 10:04:22 PM
Difference between raw_rec (MLV Lite) and mlv_rec on EOSM.

raw_rec is fine at all bit depths.

mlv_rec is fine at 14 bit, shows some corruption on every other frame at 12 bit and the corruption is worse at 10 bit.

12bit:
(https://c5.staticflickr.com/6/5769/31115024412_468e17ea64_z.jpg)

10bit:
(https://c5.staticflickr.com/6/5781/30437512284_4e5101ab1c_z.jpg)

No focus pixels though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 10:30:18 PM
Tested some more around raw_twk on my eos m. Now I,m not sure if raw_twk is to be implemented on all cams. Anyway. Since it works really fast I think it,s a badass feature.
When recording mlv_rec and sound on an error message is shown Threads failed to start. When raw_twk.mo is off recording works. Is sound even working on eos m to begin with? I am not too familiar with this cam.
Mlv_play module has no start, stop, pause buttons when run through raw_twk. Can,t scroll around to the next video etc.
Like dfort points out mlv lite is producing good files while mlv_rec inherits corruption.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 10:54:59 PM
It seems raw_twk is causing mlv_lite(raw_rec) to produce the files with bad black levels on the 5D mark III. It happens once you start mlv_play and after this the next file usually turns out bad. When raw_twk is off everything seems fine when running mlv_lite. I will change my shared builds to builds without the raw_twk.mo for now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 10:58:48 PM
@alex ah I missed that one from Dmilligan
@Dmilligan

Can you help me here, I can see the following code in edmac_memcpy.c (see italic below)
I wanna fix live view during recording for 6d, how do I find a free edmac channel with free edmac channels ?


/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#ifdef CONFIG_EDMAC_RAW_SLURP

/* for other cameras, find a free channel with find_free_edmac_channels  */
#ifdef CONFIG_5D3
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_60D
uint32_t raw_write_chan = 1;
#endif

#ifdef CONFIG_600D
// write-index 1, 4, 6, 8, 10, 11, 13
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_EOSM
uint32_t raw_write_chan = 0x12;
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 11:03:51 PM
I can see with show edmac in the Debug menu and scroll trough the channels, that channel 7, 8 and 13 are empty...
Is it that easy ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 11:16:41 PM
Well...I can confirm that adding this line


#ifdef CONFIG_6D
uint32_t raw_write_chan = 7;
#endif



And compiling mlv_rec doesn't work... :-\
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on November 26, 2016, 11:17:58 PM
@Danne:
black level issue should be fixed now
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 11:37:27 PM
Great man!
Confirmed fixed. Tried with 10-bit and with raw_rec on 5D mark III. raw_twk works and no green messed up files. Will put up new builds here again.
https://bitbucket.org/Dannephoto/cr2hdr/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 11:38:10 PM
False alarm on the focus pixel elimination. a1ex pointed out that this has been tried in the past:

https://www.magiclantern.fm/forum/index.php?topic=6658.msg69400#msg69400

and things haven't changed with 10bit 12bit:

(https://c3.staticflickr.com/6/5459/30892543730_be804328eb_z.jpg)

Bummer.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 11:42:02 PM
And cold pixel fix on this?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 27, 2016, 12:01:53 AM
Quote from: Danne on November 26, 2016, 11:42:02 PM
And cold pixel fix on this?

Doesn't fix it.

I'm running at test now to see if maybe there is some sort of a pattern to this. In any case this doesn't seem to be a 10bit/12bit issue so we're off topic.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 27, 2016, 12:03:09 AM
@Danne
Tested on my 5dMkIII 10 and 12 bit
it works great
Thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on November 27, 2016, 12:07:48 AM
Quote from: Danne on November 26, 2016, 11:37:27 PM
Great man!
Confirmed fixed. Tried with 10-bit and with raw_rec on 5D mark III. raw_twk works and no green messed up files. Will put up new builds here again.
https://bitbucket.org/Dannephoto/cr2hdr/downloads

Can you add crop rec module to your build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 27, 2016, 12:38:03 AM
@eNnvi: 1736x976 is divisible by 16 bytes at all bit depths. For 700D, the horizontal resolution is 1808. Therefore, EDMAC configuration must be, with the notation from here (http://www.magiclantern.fm/forum/index.php?topic=18315):

10bpp: (2170, skip 90) x 975, 2170
12bpp: (2604, skip 108) x 975, 2604
14bpp: (3038, skip 126) x 975, 3038

All these configurations work fine on 5D3, so I'm not sure what the issue is.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on November 27, 2016, 01:00:27 AM
Ok still have to rightly understand the code (expecially Emacs, my arm knowledge is stuck on Cortex m0 and m4 from stm)

I'll open another thread to understand the whole firmware (i know it's hard) so i Can begin writing a wiki for developers and later for users.

So if i understood it right your calc is (W*H*bpp)%16==0 ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 27, 2016, 07:35:59 AM
Doing some more testing with 5D2 on 10-12bit raw with updated resolutions , A very strange thing is happen when I use 1872x#### in 10 or 14bit I get corruption but 12bit is almost good just some tearing on top 50 or so line no color noise the image is good . When I select 1856x#### 14bit is fine. It seems that 1872 only work with 12bit without crashing (early stop with all corrupted Cdng's, color bars and about 100 no more then that ) with Canon preview . Anything with 1880 in all 3 bit ranges are totally corrupted and crashes the recording session , ( early stop etc..) but not the camera . You would think in 14bit the higher resolutions above 1856 should not be corrupted , even if it's too much data bandwidth  it just should drop a frame and stop recording .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on November 27, 2016, 11:27:53 AM
The corruption seen in the 10bit and 12bit dforts 543 post looks like its to do with bit planes https://en.m.wikipedia.org/wiki/Bit_plane, guess this is obvious to those understand math and code, not me.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 27, 2016, 11:36:28 AM
Trying to fix live view during recording for 6d and found some more pieces of the CONFIG_EDMAC_RAW_SLURP puzzle.

I found some references in edmac-memcpy.c and raw.c
Already added some extra code for the 6d to the emac-memcpy.c file.(see end of this post)
Now I found this piece of code in raw.c, I need to find DEFAULT_RAW_BUFFER MEM address for the 6d.
It says on top in the descriptions how to get that address(see bold text), but I'm too stupid too understand it :P
Can anyone help me here and tell me how to find this memory address?

#ifdef CONFIG_EDMAC_RAW_SLURP
/* undefine so we don't use it by mistake */
#undef RAW_LV_EDMAC

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
*
* raw_buffer = get_raw_buffer()
* sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
* ...
* dump_file(filename, raw_buffer, 7*something...)
*/


#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif

#ifdef CONFIG_600D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x51FC))
#endif

#ifdef CONFIG_5D3
#define DEFAULT_RAW_BUFFER MEM(0x2600C + 0x2c)  /* 113 */
//~ #define DEFAULT_RAW_BUFFER MEM(0x25f1c + 0x34)  /* 123 */
#endif

#ifdef CONFIG_EOSM
#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
#endif

#else

/* with Canon lv_save_raw, just read it from EDMAC */
#define DEFAULT_RAW_BUFFER shamem_read(RAW_LV_EDMAC)

#endif


In edmac-memcpy.c I already added the 6d to this piece of code
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#ifdef CONFIG_EDMAC_RAW_SLURP

/* for other cameras, find a free channel with find_free_edmac_channels  */
#ifdef CONFIG_5D3
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_60D
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_600D
// write-index 1, 4, 6, 8, 10, 11, 13
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_EOSM
uint32_t raw_write_chan = 0x12;
#endif

#ifdef CONFIG_6D
uint32_t raw_write_chan = 8;
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 27, 2016, 01:49:48 PM
a1ex gave me some tips a couple pages back. Start by looking up the function lv_raw_dump in Canon firmware (pretty easy to find, it's an eventproc). It should be calling a function that returns a hardcoded pointer (or in 60D's case pointer to a pointer) to the raw buffer address.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 27, 2016, 09:49:34 PM
I read the tips Alex gave you and I'm even more confused now, and that stuff was in raw_rec module, although the code should be much the same.
How do I look up the LV_raw_dump function in Canon firmware ?
I've made a rom dump, load it in a texteditor and found two times the text 'lv_raw_dump', but around it was a bunch of unreadable rom code.

Maybe you or Alex can give some more fool proof help ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 28, 2016, 04:17:36 AM
https://www.magiclantern.fm/forum/index.php?topic=12177.0

I like to use the HTML pages generated by ARM Console (http://magiclantern.wikia.com/wiki/GPL_Tools/ARM_console). Getting it setup is no easy task and it make take several days or more to run.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 28, 2016, 11:15:50 AM
QuoteGetting it setup is no easy task and it make take several days or more to run.

Real motivating words :P

I was hoping for something more simple, but hey, if this is the way how to get live view back, I'll shall take a look at it.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on November 28, 2016, 09:47:53 PM
BUILD magiclantern-Nightly.2016Nov26.5D3113_raw_twk

When I press Zoom for focus, absolutly brick the camera.
Battery out resolve the problem.

10 Bits has the pink problem on the top  and 12 bits image on the higlights.
Not problem at all in 14 bits

Changing the aspect to 4x3 (1920 x 1280) has pink color on the higlights as the 12 bits.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 11:04:05 AM
I'm first trying to go this route to find the 6d address for lv_raw_dump
https://www.magiclantern.fm/forum/index.php?topic=12177.0 (https://www.magiclantern.fm/forum/index.php?topic=12177.0)

But I'm a little stuck, I do have 'disassemble.pl' file half working.
I do get the ROM1.BIN.strings file, but the other two, .dis and .labels are empty.
Terminal output:
------------------------------------------------------------------
string dump

create elf file
sh: arm-elf-objcopy: command not found
sh: arm-elf-objcopy: command not found
label scan
sh: arm-elf-objdump: command not found

disassemble and string lookup
sh: arm-elf-objdump: command not found

job complete!

--------------------------------------------------------------------
I think I don't have the path locations right.
The 'gcc-arm-none-eabi-4_8-2013q4' folder is in my user root directory.
And the original disassemble.pl file also asks for '/gcc-4.1-arm/bin/' directory, which I don't have (but that is probably old version of the 'gcc-arm-none-eabi-4_8-2013q4'folder' ?

This is now in my disassemble.pl file
----------------------------------------------------
# adjust these for your needs (note final slash):
#$path = "$ENV{'HOME'}/gcc-arm-none-eabi-4_8-2013q4/bin/";
$path = "";
----------------------------------------------------

Anybody now how to setup the path to the /gcc-arm-none-eabi-4_8-2013q4/bin/ directory which is in the root of my user directory on mac osx
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on November 29, 2016, 01:15:10 PM
(http://i.imgur.com/9nHd6FZ.png)

I tried the @nikfreak :s 7d build and it didn't work at all. 10 and also 12 bit neither of these worked (i tried mlv_rec). I tried crop mode and regular mode.  Same thing also showed on the camera screen when playing back.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 29, 2016, 01:31:35 PM
Well it works with a little corruption here and there for the 7D(Used Nikfreak,s build). Raw_rec both 10bit and 12bit. SOme frames are perfect so there is potential. Tried with raw_twk added as well but symbols_engio error.

(https://s16.postimg.org/pcgzgxdk5/Screen_Shot_2016_11_29_at_13_28_51.png)

(https://s16.postimg.org/ribabfh0l/Screen_Shot_2016_11_29_at_13_28_42.png)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on November 29, 2016, 01:48:23 PM
@Oswald: you better use MLVFS and check the DNG's instead of using MLRAWVIEWER. Otherwise you won't have luck.

Quote from: Oswald on November 29, 2016, 01:15:10 PM
...10 and also 12 bit neither of these worked (i tried mlv_rec).

Use raw_rec!!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 01:49:14 PM
@Oswald: Is that screenshot from MLrawviewer ?
MLrawviewer doesn't work with 10bit files, at least not the version I have with Canon 6d files.
I use MLV_dump to get dng out of the MLV files and can view them in RawTherapee.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 29, 2016, 02:55:38 PM
Quote from: Levas on November 29, 2016, 11:04:05 AM
Anybody now how to setup the path to the /gcc-arm-none-eabi-4_8-2013q4/bin/ directory which is in the root of my user directory on mac osx
It should be as easy as changing the line: $path=""; to simply point at the path of your arm gcc bin folder (any easy way to get that is to drag and drop that folder on a terminal window.


$path="~/gcc-arm-none-eabi-4_8-2013q4/bin/";
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on November 29, 2016, 03:05:14 PM
I got it working with MLVFS, sorry about it. I tested again and 12bit and 10bit crop mode litlle bit working, i see few totally corrupted frames and other good ones. I see huge potential with filming with continues 10bit raw with resolution of 2520 x 1052.

http://imgur.com/a/icoiU

Second is 10bit and first one is 12bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 04:07:06 PM
@dmilligan
Thanks!

I was confused by this piece of code in the sample path.
"$ENV{'HOME'}

After the path fix I needed to duplicate 'arm-none-eabi-objcopy' and 'arm-none-eabi-objdump' and renamed them to 'arm-elf-objcopy' and 'arm-elf-objdump'
And now it works!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 04:54:15 PM
First of all, I have no idea what I'm doing  :P ;D

Found this piece of code in the ROM1.BIN.dis file.

ff35a8d8:    ebf7afc0    bl   loc_ff1467e0
ff35a8dc:    e59f10a8    ldr   r1, [pc, #168]   ; ff35a98c: (ff359efc)
ff35a8e0:    e28f00a8    add   r0, pc, #168   ; ff35a990: (725f766c)  *"lv_raw_dump"


So am I right saying function lv_raw_buffer is called with address ff1467e0 ?
But now what to do, because this isn't the memory address I'm looking for ?
Skim the ROM1.BIN.dis file for ff1467e0 ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 29, 2016, 06:49:56 PM
This sounds very exiting! Is it possible to try it on 5dmk3? Special module or ml version required?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 29, 2016, 07:07:16 PM
Quote from: Markus on November 29, 2016, 06:49:56 PM
Special module or ml version required?

You need to compile this branch:

https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit

If you aren't able to compile but want to, check out some of the tutorials located here: http://www.magiclantern.fm/forum/index.php?board=25.0

If you still can't compile but want to test, ask if someone will compile a build for your camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 29, 2016, 08:33:02 PM
Quote from: dfort on November 29, 2016, 07:07:16 PM
You need to compile this branch:

https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit

If you aren't able to compile but want to, check out some of the tutorials located here: http://www.magiclantern.fm/forum/index.php?board=25.0

Definitely worth checking that tut out!  8)

Quote from: dfort on November 29, 2016, 07:07:16 PM
If you still can't compile but want to test, ask if someone will compile a build for your camera.

I went nuts on compiling again this morning (Thanks @dfort for the follow-up re: SourceTree :D) and here the builds for all cameras running ML (except for 1100D.105):

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads

Meanwhile I am still working on to get @nikfreak's branch working on my system in order to spit out with the latest and greatest into the 70D & 100D and as soon as they are up and running definitely will share them.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 08:37:24 PM
Just wondering, is it possible to disassemble a RAM dump from the camera with 'disassemble.pl' ?
Ans if so, at which address does it start, the ROM loads at 0xFF000000, but what value to use for the RAM dump ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 29, 2016, 09:21:02 PM
Thanks! gonna try DeafEyeJedi's Compiled file! :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on November 29, 2016, 09:45:52 PM
Tried the latest firmware build for the 50D... Unable to locate the Bit-Depth option in the menu (either w/ the RAW or MLV module). Also, MLV Sound is listed as a module w/ the 50D and causes an error to occur. (Just thought to try it out and see what happens).

The MLV and .RAW Modules in the menu say they were last updated in August of 2016.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 09:49:13 PM
@DeafEyeJedi
Just tried the build you made for 6d, and noticed it doesn't have 10/12 bit options in mlv_rec and raw_rec.
Just saying(Already have a build for 6d with 10/12bit options)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 29, 2016, 10:15:09 PM
Same problem here for 5d3 123. Can't find settings for bit depth.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 30, 2016, 12:28:13 AM
Quote from: LEVISDAVIS on November 29, 2016, 09:45:52 PM
Tried the latest firmware build for the 50D... Unable to locate the Bit-Depth option in the menu (either w/ the RAW or MLV module). Also, MLV Sound is listed as a module w/ the 50D and causes an error to occur. (Just thought to try it out and see what happens).

The MLV and .RAW Modules in the menu say they were last updated in August of 2016.
Quote from: Levas on November 29, 2016, 09:49:13 PM
@DeafEyeJedi
Just tried the build you made for 6d, and noticed it doesn't have 10/12 bit options in mlv_rec and raw_rec.
Just saying(Already have a build for 6d with 10/12bit options)
Quote from: Markus on November 29, 2016, 10:15:09 PM
Same problem here for 5d3 123. Can't find settings for bit depth.

Thanks guys for noticing this. Dumb mistake on my part (for not double checking builds in beforehand) as I was in a hurry to compile them all before work. ::)

Uploaded new files and this time I checked for 5D3.113, EOSM and 7D which all includes the bit options. Now off I go to figure out how to merge the crop_rec.mo together with this. I'll be sending you an PM soon @dfort!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on November 30, 2016, 03:08:29 AM
Nevermind: (Turned camera On/Off and it's fine... and also... I assumed that I would be able to see the footage in camera and that I would not need to use MLFVS.)

(https://s16.postimg.org/a5zqoyjtt/VRAM1_50_D_San_Disk_UDMA_7_120_MBs.jpg) (https://postimg.org/image/a5zqoyjtt/)

Interesting results with a SanDisk UDMA 7 120MB/s... I'm getting "Matrix" looking images with 10-bit and 12-bit... It's about time I'm seeing the Matrix. Unfortunately, that not something I was hoping to show people. I'd rather just have the power.  8) Secondly, I started getting text on the screen. I turned the camera off and back on and will eventually get text on the screen. The readout does go away when recording, though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 30, 2016, 06:27:02 AM
@LEVISDAVIS -- you sure you didn't accidentally enable on lua.mo under Modules within ML menu?

@100D.101 owners -- as promised that I'd get this one compiled and ready for test (Thanks @nikfreak for sharing your branch) and again PLEASE keep all bug reports on this thread. Enjoy!

100D.101: https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads/ML_10bit_12bit_2016Nov29.100D101.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 30, 2016, 08:00:24 AM
Is it only possible to test this on 5D3 fw113? No fw123 version? If there Isn't I'll guess I'll just downgrade but 123 would be sweet.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 30, 2016, 08:02:00 AM
Afaik 5D3.123 is included @Markus -- my wrong and thanks for catching that @Markus! Just got home from work. Will double check SourceTree and see if I can get it compiled for 5D3.123. Until then.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 30, 2016, 09:07:47 AM
5D3.123 is on a separate branch. I don't have a 5D3 to test so there's no guarantee that this will work so use at your own risk:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Nov30.5D3123.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 30, 2016, 10:01:25 AM
Didn't get far with fw123 version. Crash when enableing raw rec and/or mlv rec module.  8)

I can share the crash log when I get home from work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: otherman on November 30, 2016, 11:13:54 AM
Is it possible shoot in 10bit with dual-iso, and recreate 14bit in post?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on November 30, 2016, 05:50:26 PM
I tested the 100D (i have b model) build and it is working without any corruption, but live view freezes when starting recording. And also, recording with 12bit, it occanly says raw detect error.

Edit: it says every time raw detect error, when trying to film 12 bit raw. But when you click it again, it starts to record.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 01, 2016, 12:06:19 AM
Trying out 10bit on a 550D using DeafEyeJedi's recent 'I went nuts again this morning' build, 1280 wide 2.35:1 23.976 fps overide and MLVFS installed for the first time today, recording works great, no freezing or odd behavior except top quarter of playback in Resolve via MLVFS every other frame is offset to the right by a consistent 16 pixels, so top quarter of the image jumps from side to side on playback.

50D 10bit every other frame top quarter 'corrupt'. Same portion of the frame as the 550D but 50D shows multicolor, which looks like an incomplete bit plane stack and it appears to be offset to the right by a consistent 16 pixels.

Tested playback with MLVFS and viewed DNG's via MLVFS in Rawtherapee.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 01, 2016, 05:20:14 AM
im also getting "the matrix" levisdavis was talking about on my eos m.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 01, 2016, 05:26:52 AM
@teatotalTED - "recording works great, no freezing or odd behavior" - is that really true? You mean, no LV freezing during recording on 550D = realtime monitoring works?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on December 01, 2016, 07:27:18 AM
Realtime, full colour Liveview monitoring indeed works for me with 10 bit raw on the 550D! Very exciting! Haven't checked the files yet however.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 01, 2016, 07:35:01 AM
So this is the way to go, if we want to fix LiveView for other cameras, am I right? If it works for 550D, it may work for others.
I have some time and some skills. Need some push to the right direction. Please give me some clues, where to dig in, if it make sense.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 01, 2016, 07:40:46 AM
Very exciting indeed, no freezing on 550D or 50D. Getting contnuous 1280 at 2.35:1 10bit on 550D but then big jump to 1600 wide, smaller increments of 16 pixels as discussed earlier on the thread would be good, can't find it in the menus.

So nearly there just need the 16 pixel offset every other frame resolved.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on December 01, 2016, 07:54:35 AM
Quote from: teatotalTEDVery exciting indeed, no freezing on 550D or 50D. Getting contnuous 1280 at 2.35:1 10bit on 550D but then big jump to 1600 wide, smaller increments of 16 pixels as discussed earlier on the thread would be good, can't find it in the menus.

So nearly there just need the 16 pixel offset every other frame resolved.

Which build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on December 01, 2016, 08:09:08 AM
Scroll the wheel for increments in between 1280 and 1600. Works nicely on DeafEyeJedi's latest build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 01, 2016, 09:00:31 AM
Alright guys here we go with Round Two of me going nuts!  :P

This time it includes the crop_rec.mo together with 10/12bit options along with a raw_twk.mo that has been inserted as well for each build.

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads

Special credit goes out to @Danne and @dfort for pushing this forward (Thanks for putting up with my shit this AM) as they were extremely helpful on making this merge possible.

Also I'm still in the process of trying (still a noob myself) to merge this together with 100D/70D from @nikfreak's branch. So will need more time on this meanwhile the previous build from yesterday is still online for now until then. Enjoy!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on December 01, 2016, 09:41:18 AM
Also I'm still in the process of trying (still a noob myself) to merge this together with 100D/70D from @nikfreak's branch.

Sounds great@deafeyejedi
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hammermina on December 01, 2016, 11:59:24 AM
didnt work on 550d :(
got only color pixel mashup,but the recording times are amazing...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on December 01, 2016, 12:06:08 PM
Post-processing tool used?

1. Try ExFAT file system. Format card using cardreader, copy ML files to card and redo installation.
2. Try lower resolutions.
Report back with results.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 01, 2016, 12:17:40 PM
Crashlog from yesterday when I tried to get 10-12bit working on 5d3 123:

ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3


Magic Lantern version : Nightly.2016Nov30.5D3123
Mercurial changeset   : 083982718d31+fd61d21d1429+ (5D3-113_123_10bit_12bit)
Built on 2016-11-30 08:02:13 UTC by rosiefort@RosieFoComputer.
Free Memory  : 165K + 3736K

Not sure if this is helpful?
I'll try out 113.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 01, 2016, 12:18:20 PM
7D users: Please try this build and let me know if it's any better (or worse) than yesterday's build?

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads/crop_rec10bit_12bit_twk.2016Dec01.7D203.zip

Happy December and Thanks to @nikfreak for the tip (http://www.magiclantern.fm/forum/index.php?topic=9848.msg175779#msg175779tip).

*edit*

Thanks guys for tests and my apologies for this bad build. Will get to the bottom of this after some cleaning up.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hammermina on December 01, 2016, 12:20:49 PM
ok now it worked but picture glitches....
i can only preview in 550d on desktop there ist still pixelmashup
i use raw2cdng 1.75
what do you use?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on December 01, 2016, 12:59:59 PM
Quote from: DeafEyeJedi on December 01, 2016, 09:00:31 AM
Alright guys here we go with Round Two of me going nuts!  :P

This time it includes the crop_rec.mo together with 10/12bit options along with a raw_twk.mo that has been inserted as well for each build.

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads

Special credit goes out to @Danne and @dfort for pushing this forward (Thanks for putting up with my shit this AM) as they were extremely helpful on making this merge possible.

Also I'm still in the process of trying (still a noob myself) to merge this together with 100D/70D from @nikfreak's branch. So will need more time on this meanwhile the previous build from yesterday is still online for now until then. Enjoy!


All fine, but x5 zoom crash the camera first time whith in the raw twk activated. If deactivate reboot and activate raw twk again works ok/.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 01, 2016, 02:00:37 PM
http://www.magiclantern.fm/forum/index.php?topic=5601.msg175786#msg175786
The build for 7D above is broken. Don,t use it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 01, 2016, 02:05:31 PM
5D3 113 Results:

Works
12bit & 10bit RAW Full HD
3xcrop with good preview works. Even with 10bit.
5XZoom without rawtwk loaded works
recorded 2.5k successfully but preview bugged out when the recording starts.

Things that dont:
12bit MLV screen tearing in the top part of the frame as MitchLally noticed.
10bit MLV gives colorful corruption on every other frame top third part
5xZoom with raw twk results in crash as Surin Dmitriy noted
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 01, 2016, 02:07:05 PM
@Markus
Any difference in corruption for between raw_rec and mlv_rec?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on December 01, 2016, 02:12:28 PM
Confirmed 5X Zoom causes freeze up, pop battery to rectify. Turned 5x off in Liveview Zoom Tweaks to continue testing.

Tested in mlv_rec only as most of my work requires sound. 5D3 113, tested all 3 bit depths. Global Draw on, mlv_play, raw_twk loaded. 1920x1080 24p. Used deafeyejedi's latest build. Used MLVFS (vert stripe fix on) and Davinci Resolve 12.5 to decode. Exported at h264.

Here is what I have:

https://vimeo.com/193871899/694e0a161e
(definitely worth hitting that download button to see the original file)

As you can see, 14bit is normal, 12bit looks normal, 10bit every second(ish) frame is corrupted in top third. Plays back in mlv_play this way too.

Upon further inspection – testing 12bit whilst moving the camera I encountered this (keep an eye on the top portion of the frame):

https://vimeo.com/193872350/9a8cd67e0b

Also it seems that it isn't decoding all the frames...  Same number of frames in the folder as 10/14 bit... Could this be a resolve issue? Or MLVFS?

Awesome progress so far, thank you to everyone working on this. I am suuuper keen to see at least 12bit working in mlv_rec! That would be a dream come true.

Here's some size comparisons:

https://s15.postimg.org/71b8gur6j/Screen_Shot_2016_12_01_at_10_23_34_pm.jpg
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 01, 2016, 02:24:20 PM
Raw_rec works without glitches 10bit/12bit?
Is raw_twk causing the 5x zoom to freeze?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 01, 2016, 02:39:57 PM
On 550D rawrec 10bit produces multicolor corruption to top 1/3rd of every other frame. mlvrec 10bit produces 16pixel offset to right every other frame but image intact. Playback in camera totally corrupt.

(https://s14.postimg.org/v97tf4zcx/Good_1_1_1.gif)


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on December 01, 2016, 02:53:42 PM
Yes it seems to be caused by raw_twk. Can zoom 5X fine without this module loaded.

Raw_rec works without glitches 10bit/12bit.

Here's raw_rec at 10 and 12 respectively: https://vimeo.com/193878531/e7db2974ed

Exactly same workflow as mentioned in last post.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 01, 2016, 03:11:41 PM
QuoteRaw_rec works without glitches 10bit/12bit?
Is raw_twk causing the 5x zoom to freeze?

@Danne
Yes rawrec 10-12bit works as far as I can see.
Yes raw_twk causes freeze.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 01, 2016, 06:28:27 PM
Quote from: reddeercity on November 27, 2016, 07:35:59 AM
Doing some more testing with 5D2 on 10-12bit raw with updated resolutions. [...] Anything with 1880 in all 3 bit ranges are totally corrupted and crashes the recording session , ( early stop etc..) but not the camera .

1880 works just fine here on 5D2 (just some minor UI quirks when fine-tuning the resolution). Playback tested on classic mlv_play (14-bit only) and checked the first DNG with mlv_dump (all bit depths).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on December 01, 2016, 06:47:25 PM
Canon 50D: 11/29 & 11/30 Firmware: Getting the same 10-bit and 12-bit results as the 550D (@ teatotalTED), it's the bottom portion of the screen that repeats. FYI: The image splits in the middle of the frame. However, one time it split at the top 2/3rds. The split image repeats every other frame. :o

Meanwhile: 14-bit is as solid as ever. (** First time using MLVFS and the image is really something worth broadcasting! **)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on December 01, 2016, 08:23:21 PM
I tried the Deafeyejedi:s 7d build and it worked worse than previous one:
1) when trying to film with raw_lite, it just says data corruption at slot 21, 22, 23 or something like that.
2) crop mode freezes whole live view, but not the camera
3) i think that dng:s are more corrupt than before.

Also i tested the your 100d build and it has same problem than before, so, when recording start, it freezes the live view.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 01, 2016, 08:54:03 PM
Quote from: a1ex on December 01, 2016, 06:28:27 PM
1880 works just fine here on 5D2 (just some minor UI quirks when fine-tuning the resolution). Playback tested on classic mlv_play (14-bit only) and checked the first DNG with mlv_dump (all bit depths).
Ok , I'll recompile the build and try again maybe I messed up some where or missed something .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Steve_FR on December 01, 2016, 08:59:13 PM
Tested Eos M running Crop_rec nov30th build from DeafEyeJedi.

Recording Raw 10 bit @ 1728x692 (2.50:1) using movie crop mode works smoothly, with live view display working great during raw REC.
Playback of the last 10bit raw clip using mlv_play and raw_twk enabled works great, but there are no controls to start, stop, track back or track forward. I can interrupt the raw playback with a half-shutter press, and return to normal liveview. I can play the clip as many times as I want if I keep interrupting it before it finishes, but if I allow the clip to play until the last frame, the camera appears to hang on that frame. A half shutter returns to liveview. Afterward, raw playback soft hangs on a black screen (half shutter returns to liveView). The only way for me to see the raw playback again is to power cycle the camera. The camera is not crashed or frozen, though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 02, 2016, 12:08:58 AM
Ok share my report.

Tested on 700D, after selecting 10bit it doesn't allow to select 12 bit, it shows in the dropdown list but non way to select it.

Anyway i think we MUST find a unique way for testing as i can see that many corrupted frames take place only when converting from .mlv to the dng sequence.

So please when talking about corrupted frames also say what decodifier you're using and (eventually) the options with it.



Here is my very short test:
Camera: 700D
Decodified with: MLVFS (running as service)
OS: Windows
BPP: 10 bit
Resolution: 1736x784
Memory card: Scheda SanDisk Extreme 32 GB SDHC UHS-I - 45 MB/S
raw_twk: no
Corrupted frames: none
Build: 30th november from DeafEyeJedi
Notes:
got liveview working, quite slow to be honest but fast enough for now ;)


I'll do further testing on saturday morning (yeah sunlight shots :D )

I'd like to know what's the repo we should consider if we want to help (so check last versions and so on), is the hudson's official bitbucket repo or should i browse many other repos?


If you need help (also programming, got some skill in C) just ask
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 12:33:50 AM
Quote from: eNnvi on December 02, 2016, 12:08:58 AM
Tested on 700D, after selecting 10bit it doesn't allow to select 12 bit, it shows in the dropdown list but non way to select it.

I made new test builds and selecting 12 bit is working fine on the 700D. Haven't done extensive tests yet but it seems that mlv_rec has the corrupt frames issue while raw_rec (MLV Lite) is ok.

Lots of exciting development going on. A few of us have been merging 10bit_12bit with crop_rec and raw_twk and posting test builds. Here's my latest--all platforms including 5D3.123 but excluding 1100D, 70D and 100D. I'll keep it up for a few days or when there are new commits to the development branches.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.zip

Note that crop_rec only works with 5D3 and EOSM and some combinations of modules aren't working. Lots of testing opportunities but please make sure you report where you got the build, date, if you are using mlv_rec or raw_rec and if you find an issue report it in the appropriate forum topic.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 02, 2016, 04:32:40 AM
Quote from: dfort on December 02, 2016, 12:33:50 AMselecting 12 bit is working fine on the 700D
So LV display during recording works on 700D too? No freezing?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 05:29:01 AM
Quote from: Rewind on December 02, 2016, 04:32:40 AM
So LV display during recording works on 700D too? No freezing?

No--during recording LiveView still freezes. This was also the case with the EOSM but it was resolved so there's still hope for the other platforms. I believe that it was the 
EOSM: enable CONFIG_EDMAC_RAW_SLURP (https://bitbucket.org/hudson/magic-lantern/commits/193dc5989d6f8807804ffb1ede2597967ef33f1d?at=raw_video_10bit_12bit) commit by dmilligan that did the trick.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 02, 2016, 07:17:17 AM
I can confirm on 5D2 1880 does work in all bit ranges , of course it's only stable/uncorrupted in 14bit
10 , 12 have issue of course but liveview never froze up . I didn't have time today to recompile
the 10_12bit expermential branch so I tested out dfort build as it seem to be update .
Set liveview to Auto , FYI . I made 3 short 20 second test videos (h264) from 10 ,12 ,14 bit
in my Dropbox  https://www.dropbox.com/sh/lh5lxr74moyrnj5/AAAFDzHGYxoafVLWkX71R5R0a?dl=0
you can there download (50MB each) or watch them in dropbox
Beware ! 10 & 12 bit may made you sea sick  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 08:58:20 AM
@reddeercity

Are you using mlv_rec or raw_rec (MLV Lite)? I've gotten better results on the EOSM from raw_rec.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 02, 2016, 11:02:55 AM
Did some changes to raw_rec (mlv lite), not really related to bit depth, but since this became the "de facto" bleeding edge branch for raw recording, I've merged them. The functional ones are:

- fix for this issue (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174390#msg174390) (may affect recording times if - and only if - global draw is on).

- a change borrowed from mlv_rec (async edmac calls) that might help with general frame corruption issues (that already happen in unified in some edge cases); it will not help with frame corruptions related by bit depth changes.

Currently benchmarking these changes; will post results here (http://www.magiclantern.fm/forum/index.php?topic=17091.msg175088#msg175088).

For new builds, I recommend including only those cameras that can record without corrupted frames, at least at some settings. For the cameras that show corruption every other frame, the behavior will be unchanged. I know the cause and already suggested possible fixes  a month ago (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181;topicseen#msg174181). Until somebody implements one of these fixes (hint: see how it was done on EOS M), reports of image corruption (repeated for every new build), for those models that are already known not to work, are simply not useful.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Peter Linov on December 02, 2016, 11:35:27 AM
5DMIII 1.2.3 
CRASH 00-
ASSERT: hLvJob->hJpegMemSuiteat ./Epp/Vram/VramStage.c:891, task Epplv:1 mode:3
Magic Lantern version : Nightly.2016Dec01.5D3123 Mercurial changeset   
: 385bc44ed158+b3dfbe7194f3+ (5D3-113_123_10bit_12bit) tip
Built on 2016-12-01 23:09:48 UTC by rosiefort@RosieFoComputer.
Free Memory  : 146K + 3712K
CRASH01-
ASSERT: IsSuiteSignature( hSuite ) at ./PackMemory/PackMem.c:599, task Epp
lv:0 mode:3Magic Lantern version : Nightly.2016Dec01.5D3123
Mercurial changeset   : 385bc44ed158+b3dfbe7194f3+ (5D3-113_123_10bit_12bit) tip
Built on 2016-12-01 23:09:48 UTC by rosiefort@RosieFoComputer.
Free Memory  : 146K + 3728K
CRASH02-
ASSERT: hLvJob->hJpegMemSuiteat ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3 Magic Lantern version : Nightly.2016Dec01.5D3123
Mercurial changeset   : 385bc44ed158+b3dfbe7194f3+ (5D3-113_123_10bit_12bit) tip
Built on 2016-12-01 23:09:48 UTC by rosiefort@RosieFoComputer.
Free Memory  : 153K + 3719K


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 04:10:40 PM
@Peter_Linov

Didn't think it was going to work but it was worth a shot. What were the conditions that caused the crash?

Looks like for now we should limit the test builds to the 5D3.113 and EOSM.202 and make sure we're in raw_rec instead of mlv_rec. These test builds don't have the mlv_rec and mlv_snd modules to avoid confusion.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D3113.zip
https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.EOSM202.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 02, 2016, 06:35:17 PM
The 550D is also very close, 12bit works fine, 10bit image intact just 16 pixel offset to right every other frame, (mlvrec not rawrec which looks to be a trainrec on the 550D)

A1ex suggests above that there's a fix for this from a month ago or should this be brought up on the 550D camera specific thread instead of here where priorities seem weighted elsewhere, I guess because of ownership of EOS-M & 5D3? There's many more 550D ML users than EOS-M and for good reason?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 02, 2016, 06:42:04 PM
Quote from: teatotalTED on December 02, 2016, 06:35:17 PM
... There's many more 550D ML users than EOS-M and for good reason?

I'd take an EOSM over 550D on any given day and no pun intended.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 07:26:00 PM
Quote from: teatotalTED on December 02, 2016, 06:35:17 PM
The 550D is also very close...

I didn't know that. Here's a test build for the 550D without mlv_rec and mlv_snd like the other test builds I recently posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.550D109.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hammermina on December 02, 2016, 08:58:35 PM
@dfort

550d build works :)12 bit all resolutions :) :)
10 bit frames are messed up...

go for it :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 02, 2016, 09:08:26 PM
12bit worked great via mlvrec in dforts build from before, so 12bit now works in rawrec to.

10bit in rawrec is vertically jumping up and down 16 pixels every other frame rather than moving to the right 16 pixels every other frame as with the mlvrec module from dforts earlier build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 10:29:12 PM
@teatotalTED

Actually it is the same build. What you are reporting is the difference between mlv_rec and raw_rec (MLV Lite).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 03, 2016, 04:08:42 AM
Quote from: dfort on December 02, 2016, 08:58:20 AM
@reddeercity
Are you using mlv_rec or raw_rec (MLV Lite)? I've gotten better results on the EOSM from raw_rec.
raw_rec (mlv lite) actually it's getting better on 5D2  , 10bit used to be totally pink noise & 12bit ever other frame was totally pink noise.
But ever once in a while I have perfect recording + clean liveview mostly in Crop mode . I would say crop mode on 5d2 is more stable right now .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 03, 2016, 07:20:48 AM
The same build? As yours from yesterday posted at 12.08:58? How can it be the latest doesn't have mlvrec and mlvsnd in it? Might be same code base but a new build? 12bit on 550D worked fine on yesterdays build of 12.08:58 via mlvrec, all I was saying was that 12bit working on 550D was nothing new to the latest rawrec only build. And finally yes I was comparing 12.08:58 build 10bit 550D via mlvrec 16 pixel shift horizontal with latest rawrec only build 10bit 16 pixel shift vertical, both very close to solution for 550D, so to first say only concentrate on EOS-M and 5D3, then say only concentrate on rawrec, with regard to 550D either which way makes little difference its so close. Does the latest rawrec build include possible fix highlighted by A1ex in post #627, is that what we're supposed to be testing here in the latest build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 03, 2016, 10:10:02 AM
That's a race condition working in your favor :)

The process is very sensitive to timing, and will be so until the real issue is addressed. The DMA channel is configured for 14-bit image size, we only copy 12-bit (or 10-bit) data from it, and what happens with the excess data... Canon knows. That's the cause of the artifacts you describe. It is a random process, as the LiveView timings are not very exact (I think the image capture is done at fixed intervals, but the processing timing is affected by the task scheduler, to some extent).

So, just because it works for you in some (lucky) conditions, that doesn't mean it's getting any closer.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 03, 2016, 10:55:11 AM
Yeah anticipated the response to presumption of 'so close', in reality I'm well aware it ain't that simple, just would like to see 550D in the running, 10bit on cameras with restricted write speeds is a good priority to many users, for me my 50D is go to and barely use the 550D. Lucky settings, there's no blackmagic, pun intended, in the combination of settings, just chose mlvrec 10bit and it either works or doesn't, don't consider myself a tester just have bit of time at the moment to mess.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on December 04, 2016, 05:44:28 AM
Tried out dfort's latest build on the 550D. In 10 bit every second frame is perfect, but in the others only the top 3rd is correct. The bottom 2/3rds are just a repeat of the 1st frame I believe. Liveview still works great and the record times are awesome!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on December 04, 2016, 11:10:15 AM
From Post #22
>>
Reading this: http://francoismalan.com/2011/10/raw-12bit-or-14bit-lossy-or-lossless/
... i guess that it would only make a very very slight difference!

If this is really working... I just dont have words for how awesome this is...
>>

Here is the other argument..
>>
There is a difference between underexposed sunny scene and shooting in pitch dark interiors or at night.

I did the tests myself and 14 bit has huge advantage under extreme dark conditions.

Compression concerns the highlights only and I assume it works the same way for the high-key scenes – white on white.

It's true, in every-day outdoors shooting, 14bit/uncompressed doesn't really matter.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 04, 2016, 07:12:36 PM
New test builds:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip
https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec04.EOSM202.zip

Seems like we may be adding too much into the recent test builds like crop_rec and raw_twk and reporting the results in this topic. How about we test just a few things at a time like raw_rec and mlv_rec and make sure to indicate which one you are using in your reports.

Also, cameras other than the 5D3.113 and EOSM.202 should be updated before making more test builds for those platforms:

Quote from: a1ex on December 02, 2016, 11:02:55 AM
For new builds, I recommend including only those cameras that can record without corrupted frames, at least at some settings. For the cameras that show corruption every other frame, the behavior will be unchanged. I know the cause and already suggested possible fixes  a month ago (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181;topicseen#msg174181). Until somebody implements one of these fixes (hint: see how it was done on EOS M), reports of image corruption (repeated for every new build), for those models that are already known not to work, are simply not useful.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 04, 2016, 07:39:02 PM
Did some more research and it seems that the cameras that have the fewest issues are the ones that have CONFIG_EDMAC_RAW_SLURP. Looks like dmilligan got it working on the EOSM (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175184#msg175184).

According to the features list (https://builds.magiclantern.fm/features.html) the 60D and 600D have this feature. Anyone out there want raw_video_10bit_12bit builds of those platforms to test?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 04, 2016, 07:53:11 PM
I've recorded in 10bit, many times, on the 6d and there are no corrupted frames, both in raw_rec and mlv_rec modules.
Only problem is that liveview freezes (no config_edmac_raw_slurp).
I'm stuck with figuring out how to find the address for the 6d to configure config_edmac_raw_slurp(see few post back in this topic).
http://www.magiclantern.fm/forum/index.php?topic=5601.msg175682#msg175682 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175682#msg175682)

But I can assure, that there are no corrupted frames on the 6d with 10bit recording.
Although 12bit has sometimes the purple banding issues on every frame.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 04, 2016, 08:59:14 PM
Quote from: Levas on December 04, 2016, 07:53:11 PM
Only problem is that liveview freezes (no config_edmac_raw_slurp).

disabled raw_twk module?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 04, 2016, 09:23:21 PM
With and without raw_twk enabled, live view freezes during 10bit recording on 6d.
But as far as I know, you need to configure config_raw_edmac_slurp to fix this.
At the moment this only done for 4 cams, 5d3, eosm, 60d and 600d

With 14 bit, live view works like it should be.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 04, 2016, 10:36:05 PM
@Levas - I'm trying to do the same with the 700D and seem to be stuck in the same spot you were in a week ago (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175569#msg175569).

I ran find_free_edmac_channels and found that channels 0 and 8 "seems to work" but I have no idea where to find the DEFAULT_RAW_BUFFER MEM address. I've got an EOSM so I'm looking a disassembly of the ROM1.BIN. Found "lv_raw_dump" but don't understand how dmilligan got MEM(0x404E4 + 0x44) (https://bitbucket.org/hudson/magic-lantern/commits/193dc5989d6f8807804ffb1ede2597967ef33f1d?at=unified) out of that.


diff --git a/platform/700D.114/internals.h b/platform/700D.114/internals.h
--- a/platform/700D.114/internals.h
+++ b/platform/700D.114/internals.h
@@ -139,3 +139,7 @@

/** Workaround for menu timeout in LiveView */
#define CONFIG_MENU_TIMEOUT_FIX
+
+/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
+#define CONFIG_EDMAC_RAW_SLURP
+
diff --git a/src/edmac-memcpy.c b/src/edmac-memcpy.c
--- a/src/edmac-memcpy.c
+++ b/src/edmac-memcpy.c
@@ -396,6 +396,11 @@
uint32_t raw_write_chan = 4;
#endif

+#ifdef CONFIG_700D
+// channels 0 and 8 "seems to work"
+uint32_t raw_write_chan = 8;
+#endif
+
#ifdef CONFIG_EOSM
uint32_t raw_write_chan = 0x12;
#endif
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -118,6 +118,11 @@
#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
#endif

+#ifdef CONFIG_700D
+// copied from EOSM -- need to find 700D address
+#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
+#endif
+
#else

/* with Canon lv_save_raw, just read it from EDMAC */


[EDIT] Seems like this was tried before on the 6D:

http://www.magiclantern.fm/forum/index.php?topic=13155.msg127251#msg127251
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on December 04, 2016, 11:09:53 PM
At the beginning of lv_raw_dump on the M there is a call made to a function in ram (branch instruction). On 60D this function call is to ROM not RAM. We can use RAM_OFFSET to find the location of that function in rom (Canon firmware copies some functions to RAM and runs them there rather than running them directly from ROM).

#define RAM_OFFSET 0xFFA69590


This leads to this function in ROM:

FFA7A184: ldr r0, [pc, #808] ; 0xffa7a4b4: pointer to 0x404e4
FFA7A188: ldr r0, [r0, #68] ; 0x44
FFA7A18C: bx lr


This code loads 0x404e4 into register 0, then adds 0x44 to register 0, then it returns (register 0 is the return value). In C:

return 0x404e4 + 0x44;


On 60D:

FF0F8B8C: ldr r0, [pc, #732] ; 0xff0f8e70: pointer to 0x4ff8 FF25B020 FF25B360 FF25B3A4 FF25AF2C FF25B31C
FF0F8B90: ldr r0, [r0, #48] ; 0x30
FF0F8B94: ldr r0, [r0]
FF0F8B98: bx lr

Which is equivalent to:

return *(0x4ff8 + 0x30);


Then lv_raw_dump uses this return value as a pointer to the location of the raw buffer. The ML macro MEM() is simply a pointer dereference. So Canon firmware always keeps a pointer to the location of the raw buffer at 0x404e4 + 0x44 on the EOS M. The actual raw buffer may be dynamically allocated, but it's current address is always kept at memory address 0x404e4 + 0x44 (yes, this is a pointer to a pointer, on 60D, it's actually a pointer to a pointer to a pointer). I guess these buffer addresses are probably kept in static arrays or structs and the + 0x44 part is the array index or struct offset. If it is indeed a data structure, it might be interesting to look at and around 0x404e4 and see what else might be kept there.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: capncannabis on December 05, 2016, 05:17:58 AM
Is there a build to test this on the 600d?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 05, 2016, 05:30:46 AM
Quote from: dmilligan on December 04, 2016, 11:09:53 PM
At the beginning of lv_raw_dump on the M there is a call made to a function in ram (branch instruction)...We can use RAM_OFFSET to find the location of that function in rom...

#define RAM_OFFSET 0xFFA69590


This leads to this function in ROM:

FFA7A184: ldr r0, [pc, #808] ; 0xffa7a4b4: pointer to 0x404e4
FFA7A188: ldr r0, [r0, #68] ; 0x44
FFA7A18C: bx lr


Let's see if I understand. This means that the function call should be specifying an address of 0x10BF4 (FFA7A184 - RAM_OFFSET) but there is no such address in the the EOSM disassembly that I've got. There was enough of a hint in that function in the EOSM disassembly for me to look up that same function in the 700D and I came up with 0x25B0C + 0x3C. I'm pretty confident about that. However, the RAM_OFFSET on the 700D is written in a different format from the EOSM:

stubs.S
#define RAM_OFFSET (0xFFA5E8B0-0x1900) // Nanomad: some functions are copied to RAM at around ff0c0098; they have to be called from RAM...


Oh boy. I would assume that the RAM_OFFSET is 0xFFA5CFB0 but there must be a reason it is written the way it is.

Running find_free_edmac_channels seemed pretty straight forward but --

edmac-memcpy.c
#ifdef CONFIG_700D
// channels 0 and 8 "seems to work" with find_free_edmac_channels
// but these values don't work. Still searching.
uint32_t raw_write_chan = 20;
#endif


I tried every channel from 0-20 and found several that seemed to work but the MLV file was just a bunch of noise. The EOSM is written in hex (0x12) while the other platforms are in decimal values so maybe theres a reason for that too?

Sorry to be a PITA but it seems that config_edmac_raw_slurp on the 700D is possible if we can find the right numbers to plug in. Here's my latest work in progress:

diff --git a/platform/700D.114/internals.h b/platform/700D.114/internals.h
--- a/platform/700D.114/internals.h
+++ b/platform/700D.114/internals.h
@@ -139,3 +139,7 @@

/** Workaround for menu timeout in LiveView */
#define CONFIG_MENU_TIMEOUT_FIX
+
+/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
+#define CONFIG_EDMAC_RAW_SLURP
+
diff --git a/src/edmac-memcpy.c b/src/edmac-memcpy.c
--- a/src/edmac-memcpy.c
+++ b/src/edmac-memcpy.c
@@ -396,6 +396,12 @@
uint32_t raw_write_chan = 4;
#endif

+#ifdef CONFIG_700D
+// channels 0 and 8 "seems to work" with find_free_edmac_channels
+// but these values don't work. Still searching.
+uint32_t raw_write_chan = 20;
+#endif
+
#ifdef CONFIG_EOSM
uint32_t raw_write_chan = 0x12;
#endif
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -118,6 +118,10 @@
#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
#endif

+#ifdef CONFIG_700D
+#define DEFAULT_RAW_BUFFER MEM(0x25B0C + 0x3C)
+#endif
+
#else

/* with Canon lv_save_raw, just read it from EDMAC */
@@ -447,6 +451,12 @@
     return 1;
     #endif

+    #ifdef CONFIG_700D
+    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
+    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
+    return 1;
+    #endif
+
     /* unknown camera? */
     return 0;


One last question about this:

raw.c
    #ifdef CONFIG_EOSM
    *width  = video_mode_crop ? 1872 : 1808;
    *height = video_mode_crop ? 1060 : 727;
    return 1;
    #endif


Is video_mode_crop the same as mv1080crop or is there a reason why it is coded this way for the EOSM and would this also apply to the 700D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on December 05, 2016, 05:22:36 PM
Well, as you know, the EOSM only has two video modes we can actually use for raw video (crop hack and mv720 - LV "standby" or whatever you want to call it), so I didn't bother looking up the buffer sizes for the other ones that only start when recording h.264. For 700D you will probably need all of them.

I believe there is method in crop_rec that avoids the hardcoded buffer sizes (have not looked at that code)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 05, 2016, 08:07:42 PM
working to find addresses for 700D too.

@dfort can u pls update your repo so we'd be able to work on the same files? ;)

had to work in the past few days so didn't had the time to write even a line :(

now disassembling rom and ram
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 05, 2016, 08:18:48 PM
@eNnvi -- commited what I've got so far:

https://bitbucket.org/daniel_fort/magic-lantern/commits/e0463878deec23ab856d05879def00a78e6d0d6d
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 05, 2016, 08:43:16 PM
Quote from: dfort on December 05, 2016, 05:30:46 AM
Oh boy. I would assume that the RAM_OFFSET is 0xFFA5CFB0 but there must be a reason it is written the way it is.

this way you know two important informations.
where the RAM address is and from where it gets copied.

writing 0xFFA5E8B0 would throw away that information.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 05, 2016, 11:38:59 PM
Quote from: dmilligan on December 05, 2016, 05:22:36 PM
Well, as you know, the EOSM only has two video modes we can actually use for raw video...

There is a third one -- zoom. The magnifying lens icon is on one of the touch screens so it isn't very convenient to get into that mode but it has a 2592x1108 full raw buffer and allows shooting up to 2496x1074 images. Of course 24fps is out of the question but it should be usable at lower frame rates especially with the lower bit depths.

The only video mode missing on the EOSM is mv1080 or at least that buffer seems to be out of reach for now. All of the video modes except for the "Movie crop mode" (a.k.a. mv1080crop or video_mode_crop) default to 29.97fps (even when set to PAL) so this is a quirky little camera to say the least.

At some point perhaps someone will find the key to unlocking mv1080 on the EOSM. Also, lower bit depth will eventually be working on other platforms so shouldn't this work?

#if defined(CONFIG_100D) || defined(CONFIG_650D) || defined(CONFIG_700D) || defined(CONFIG_EOSM)
    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
    return 1;
#endif


These are the cameras that exhibit the focus pixel issue. They all have identical full raw buffer sizes. I also made a new discovery while searching for video streams that don't show focus pixels but I'll save that post for the focus pixel topic (http://www.magiclantern.fm/forum/index.php?topic=16054.new#new).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 06, 2016, 05:07:19 AM
@dfort anything new on the dec4th build for eosm? I'm using mlv lite 1.1 raw record module. its working well in 10 bit raw. there's been a bunch of builds and i feel like I'm missing something new. haha
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2016, 06:13:25 AM
You can check up on new commits to this branch with this link:

https://bitbucket.org/hudson/magic-lantern/commits/branch/raw_video_10bit_12bit

I don't want to merge too many branches together until this is well tested. How about trying mlv_rec with mlv_snd in 10bit?

I'm also testing the latest cr2hdr.app that uses MLVFS and can't get the mlv_snd working with mlv_rec. It should work because the WAV file sounds fine. Also, applying FPS override to 23.98 causes corruption at the top 1/3rd of every other frame with mlv_rec but it is fine with raw_rec (MLV Lite).

On the EOSM using the Dec. 4 build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 07, 2016, 04:05:36 AM
Hi guys,  its possible to compile one for the 7D,  want to make some testing

Thanks  :) 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 07, 2016, 07:55:42 AM
Quote from: arrinkiiii on December 07, 2016, 04:05:36 AM
Hi guys,  its possible to compile one for the 7D...

It is possible but is there anything new to report on the 7D? Correct me if I'm wrong but only the 5D3.113 and EOSM.202 are doing the EDMAC_RAW_SLURPING thing necessary for this to work properly.

Quote from: eNnvi on December 05, 2016, 08:07:42 PM
working to find addresses for 700D too.

Any luck yet?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on December 07, 2016, 08:20:55 AM
For future reference... Just wanted to jump back in and say that the 10-bit + 12-bit RAW recordings from the Nov 29 & Nov 30th build created magenta frames during 5X Crop mode for the Canon 50D. This includes the 14-bit RAW as well.

Jumping out...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 07, 2016, 08:21:12 AM
So I'm wondering if this is a bug or something but doing some testing with 3xCrop mode on 5D2 with build from dfort (https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.zip) Dec./1/2016 . Just the basic modules for raw video.
With raw_rec (mlv lite) in fine resolutions 10 & 14bit  2152x1072 is the max. selectable resolution, while in 12bit I can expand the size to 2152x1076 but not 10 or 14bit.
Should it not but the same thought out the bit range ? Too bad the image are dark pink not very easy to correct like bad black levels , Good thing thou is in 12bit with Full MLV+audio in
3X crop mode I have perfect corruption Free recording  :D with correct black levels in All Liveview modes , but only the canon preview was glitch free no Lv freeze or B/W lines mess .
Thou framing is off it doesn't freeze up very useable and with audio too . I didn't try 10 bit in 3x Crop to see if it's a problem will try later , so it seems 3x crop recording to MLV 2.0(full)
works in 12bit  corruption free (useable). 8)

Edit: Sorry the size in 3xcrop MLV 2.0(full) was 2144x1076 @23.976p
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 07, 2016, 12:15:36 PM

Thanks dfort. I thought that exist, even with bad frames, a version for the 7D  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 07, 2016, 01:16:11 PM
There's one in the 7D thread posted by me.

Edit: http://www.magiclantern.fm/forum/index.php?topic=9848.msg175503#msg175503

That one is confirmed to work
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 07, 2016, 02:08:34 PM

Thanks nikfreak  8)    Will test it and give some feedback.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: robertgl on December 07, 2016, 04:33:29 PM
@660 re:50D

What resolutions & frame speeds did you shoot at? You did not get repeating frames or torn frames?  thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 07, 2016, 08:26:09 PM
@reddeercity
Quote from: reddeercity on December 07, 2016, 08:21:12 AM
10 & 14bit  2152x1072 is the max. selectable resolution, while in 12bit I can expand the size to 2152x1076 but not 10 or 14bit.
Should it not but the same thought out the bit range ?
(If I understand it correctly): The whole image has to be a multiple of 16 bytes, and 1076x2152 in 14 bit is not, so it reduces vertical resolution. In new 1880, the max vertical resolution is 1248 instead of 1250, highest resolution you can get is 1872x1250 in 12 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 08, 2016, 01:15:17 AM
ok i'm stuck on a (probably) stupid thing: find the right address for raw_slurp

i disassembled my 700D .bin file, i found the lv_raw_dump function in rom but i don't get how to get the right address from here.

I mean..

ff36c804: 725f766c subsvc r7, pc, #108, 12 ; ff36c7a0: (72615020)
ff36c808: 645f7761 ldrbvs r7, [pc], #-1889 ; ff36c810 <_binary_ROM1_BIN_start+0x36c810>
ff36c80c: 00706d75 rsbseq r6, r0, r5, ror sp
ff36c810: ff36beb0 ; <UNDEFINED> instruction: 0xff36beb0


following http://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969 dmilligan says there should be a branch instruction but i don't get where it actually is. I mean the subsvc should mean something like
r7 = pc - (108<<12) right?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 08, 2016, 01:19:56 AM
this is no valid code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 08, 2016, 04:39:39 AM
Quote from: Ilia3101 on December 07, 2016, 08:26:09 PM
@reddeercity(If I understand it correctly): The whole image has to be a multiple of 16 bytes, and 1076x2152 in 14 bit is not, so it reduces vertical resolution. In new 1880, the max vertical resolution is 1248 instead of 1250, highest resolution you can get is 1872x1250 in 12 bit.
No you misunderstood , I'm talking about 3x crop mode not full frame with the new 10-12 raw_rec (mlv lite) with a1ex update fine resolution's , in 10 bit & 14 bit in 3xCrop mode 1072 is maximum vertical and 12bit can be expanded to 1076 on the same build . It shouldn't matter if it's 10 , 12 or 14 bit the  vertical resolution in 3x crop should not change & yes I understand what happen with the code . If you check out the full MLV 2.0+audio in 3x crop mode the maximum vertical resolution is 1076 and of course in 12bit (mlv2.0) @ 2144x1076 all frames are clean with Canon Liveview .

Edit: On MLV 2.0 with bit range reduction (10-12bit) update in 3x Crop the maximum vertical resolution stays the same thou out all 3 bit ranges of 1076

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 08, 2016, 05:03:54 AM
5D2 , To just confirm MLV 2.0 10bit in 3X crop @ 2144x1076 23.976p + audio are free of any corruptions/shearing etc... .  with Canon Lv.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 08, 2016, 09:29:19 AM
So I been trying to correct the pink/magenta 3x Crop on 5d2 with raw_rec(mlv lite) with reduced bit range (working with 12bit)
well exiftool tell me the white & black level are totally out to lunch compared to mlv 2.0 in crop mode
I end up with black level of 63 & white level of 937 on the magenta frame and on the non magenta frame (full mlv2.0) black level 448 & white level 3750.
Not sure why mlv lite uses these level compared to mlv2.0 which are correct , anyway to fix the magenta frames  I used Exiftool to set the white & black level
to what my eyes see as normal WB image. So after some time of trial & error I set the black level to 110 and the white level to 730 ,
it doesn't seem right to have just a little differences between the
black & white levels as compared to the other images levels below is the magenta frame and then the corrected frame with exiftool.

  (https://c5.staticflickr.com/1/255/30660487884_d2cec3c4d2_n.jpg) (https://flic.kr/p/NHn1Rs)
12bit_3Xcrop magenta frame  (https://flic.kr/p/NHn1Rs) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://c1.staticflickr.com/1/679/30660487824_b50b83cb01_n.jpg) (https://flic.kr/p/NHn1Qq)
12bit_3Xcrop_corrected_levels (https://flic.kr/p/NHn1Qq) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

syntax for windows in command prompt
exiftool -IFD0:BlackLevel=110 *.dng
exiftool -IFD0:whiteLevel=730 *.dng
If anyone what's to check the original magenta Cdng frame 2144x1076  link below
https://www.dropbox.com/s/91a1blyb2qix31f/original_M08-0008_000004.dng?dl=0


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on December 08, 2016, 11:06:00 AM
Quote from: reddeercity on December 08, 2016, 04:39:39 AM
No you misunderstood , I'm talking about 3x crop mode not full frame with the new 10-12 raw_rec (mlv lite) with a1ex update fine resolution's , in 10 bit & 14 bit in 3xCrop mode 1072 is maximum vertical and 12bit can be expanded to 1076 on the same build . It shouldn't matter if it's 10 , 12 or 14 bit the  vertical resolution in 3x crop should not change & yes I understand what happen with the code . If you check out the full MLV 2.0+audio in 3x crop mode the maximum vertical resolution is 1076 and of course in 12bit (mlv2.0) @ 2144x1076 all frames are clean with Canon Liveview .

Edit: On MLV 2.0 with bit range reduction (10-12bit) update in 3x Crop the maximum vertical resolution stays the same thou out all 3 bit ranges of 1076

No, he is right, the bit depth matters. You can check with some simple math:


((2152*1072*10bit)/8bit)/16bytes = 180230
((2152*1072*12bit)/8bit)/16bytes = 216276
((2152*1072*14bit)/8bit)/16bytes = 252322

((2152*1076*10bit)/8bit)/16bytes = 180902.5
((2152*1076*12bit)/8bit)/16bytes = 217083
((2152*1076*14bit)/8bit)/16bytes = 253263.5

Only modes that give an integer value are valid and can be used to record with. So yes you can record with 12bit @ 2152x1076 but not with 10/14bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 08, 2016, 03:16:56 PM
Tried to read me to understanding this build but I can't manage to record without getting pink pixelated images. Live view is only stripes of black and white. Does it work on exfat? I have only tried in crop mode. Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 08, 2016, 10:47:17 PM
Quote from: rbrune on December 08, 2016, 11:06:00 AM
No, he is right, the bit depth matters. You can check with some simple math:


((2152*1072*10bit)/8bit)/16bytes = 180230
((2152*1072*12bit)/8bit)/16bytes = 216276
((2152*1072*14bit)/8bit)/16bytes = 252322

((2152*1076*10bit)/8bit)/16bytes = 180902.5
((2152*1076*12bit)/8bit)/16bytes = 217083
((2152*1076*14bit)/8bit)/16bytes = 253263.5

Only modes that give an integer value are valid and can be used to record with. So yes you can record with 12bit @ 2152x1076 but not with 10/14bit.

Thanks for the math , so then as per example when I'm using mlv 2.0 @ 12bit 2144x1076
((2144*1076*12bit)/8bit)/16bytes = 216276
If I did my math right "216276" is a good  integer value , so valve that fall in to the ".5" etc... don't work right ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 09, 2016, 01:02:09 AM
Where Can I fine the file to test out, and is this the best build so far?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 09, 2016, 07:37:24 AM
Quote from: dariSSight on December 09, 2016, 01:02:09 AM
Where Can I fine the file to test out, and is this the best build so far?
Click me (https://www.dropbox.com/s/oodspk5jits334u/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip?dl=0)  :)
this was compiled by dfort , if you what to be able to compile yourself read my post here (http://www.magiclantern.fm/forum/index.php?topic=18259.msg175475#msg175475) very easy to setup in less then 30min. and start compiling , there a short video tutorial to help explain .
I assume you need it for 5d2 right ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh on December 09, 2016, 11:06:02 AM
Hi where I can find the latest experimental 10bit 12bit build for the 5DMark III 1.1.3
Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 09, 2016, 12:32:08 PM
Is there a new version for 5D3 for mlv 10-12bit testing.

I filmed 50p 10bit continious with half vertical res on the 5D3 with the previous build :) (raw rec/ mlv light)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh on December 09, 2016, 01:07:59 PM
Hi, you mean the nightly bulids or the beta testing? I have a experimental bulid, but I have pinky noise over the whole frame ( 10bit and 12bit)...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jai554 on December 09, 2016, 05:28:15 PM
Anyone tested the latest 60D build?

thanks :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2016, 07:58:45 PM
Quote from: Markus on December 09, 2016, 12:32:08 PM
Is there a new version for 5D3 for mlv 10-12bit testing.

There hasn't been any new commits since I posted this build:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip

I only put up 5D3.113 and EOSM.202 and took down all the other builds because my understanding was that only these two platforms are stable enough for testing and the developers are working on the other platforms so they aren't quite ready for further testing. However, it seems that @reddeercity is seeing some promising results with a 5D2.212 build I put up a while ago so maybe we should check and see if the latest changes show any improvement. Like the most recent builds I posted this is the straight raw_video_10bit_12bit branch without any crop_rec or raw_twk merging going on.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec09.5D2212.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kamranjon on December 09, 2016, 08:22:30 PM
Was able to record 2240x1200 at 10-bit continuous on 7D in crop mode with the build Nikfreak provided.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jpegmasterjesse on December 09, 2016, 08:27:57 PM
Appreciate you posting that, dfort, I've been following along excitedly but not really knowing where to try the more stable builds for 5d2.

In theory, is 10/12 bit Dual-ISO possible on any camera?  As far as I remember Dual-ISO video on 5d2 never made it beyond pretty glitchy.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 09, 2016, 08:33:44 PM
dfort, thanks for the 5D3 10bit_12bit link ( 1.1.3) Now why I have allways pink frames with this build on my 5D3?
(http://img5.fotos-hochladen.net/uploads/aufnahme4vdm37lpsj9.jpg) (http://www.fotos-hochladen.net)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2016, 08:51:48 PM
Quote from: VanGogh7 on December 09, 2016, 08:33:44 PM
Now why I have allways pink frames with this build on my 5D3?

Are you using mlv_rec or raw_rec?

raw_rec (MLV Lite) is more stable at this point.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 09, 2016, 09:41:37 PM
I try

5DIII 1.1.3
crop_rec10bit_12bit_twk.2016Nov30.5D3113
raw_twk
raw_rec
mlv_play

When I push zoom to 5xZOOM in 10bit mode the camera just freeze and it's same for all 10 12 and 14bit...

Did I do something wrong ?


magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip
I try the new build and it's same than VanGogh7.

Pink frame with raw_rec in 5xZoom or normal 1080p
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2016, 09:44:47 PM
Quote from: MAG on December 09, 2016, 09:41:37 PM
I try

5DIII 1.1.3
crop_rec10bit_12bit_twk.2016Nov30.5D3113

Try the build I just posted:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec09.5D2212.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 09, 2016, 10:27:46 PM
?
Quote from: dfort on December 09, 2016, 09:44:47 PM
Try the build I just posted:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec09.5D2212.zip
The link is for 5DMII


With this build --> crop_rec10bit_12bit_twk.2016Nov30.5D3113

I can use crop mode with nice result like full preview in 1x1 but it's impossible to use 5xZoom with custom resolution more than 1080p because Freeze.

Also I try to 1080p 16:9 in All 1920 25p with fps override 40fps but the final video just extremely bug. Is there any option to slowmo without stretch in post ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 09, 2016, 11:06:13 PM
@dfort
Tryed the dec 4 build with mlv + sound full HD 10,12 & 14 bit
12 bit suffered from screentearing. Even a little more severe than the previous one I tryed
10 bit upper one third part on every other fame corrupted
14 bit OK
All bitrates in mlv-lite worked as before.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 09, 2016, 11:29:47 PM
wait testing build for my 50D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2016, 11:53:27 PM
Quote from: MAG on December 09, 2016, 10:27:46 PM
The link is for 5DMII

Oops, gave you the wrong link. Try this:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip

Quote from: MAG on December 09, 2016, 10:27:46 PM
Is there any option to slowmo without stretch in post ?

That can be done with the crop_rec module. The build I pointed you to doesn't have crop_rec or raw_twk because we probably should be getting the basic functions working before adding those modules.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 10, 2016, 12:10:07 AM
Quote from: dfort on December 09, 2016, 08:51:48 PM
Are you using mlv_rec or raw_rec?

raw_rec (MLV Lite) is more stable at this point.

Yes, and pink frames at 10bit and 12bit...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2016, 12:23:53 AM
QuoteYes, and pink frames at 10bit and 12bit...
? raw_rec or mlv_rec? There is a difference.
raw_rec(mlv_lite) is working very good with 5D mark III(10bit,12bit).

On a sidenote I,ve been using the crop_rec with 5D mark 3 and it works well. I also get good results with raw_twk module and that module is the only one working if you want to view 10 and 12bit footage in cam. I merged and compiled with updated unified code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 10, 2016, 12:28:37 AM
I use raw_rec ( MLV Lite)...and my build  magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip have not the raw_twk module
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2016, 12:35:18 AM
When you say pink frames at 10bit, 12bit are you referring to when viewing files in camera or viewing dng files on your computer?
It won,t work previewing on your computer camera without the raw_twk module.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 12:40:02 AM
Quote from: Markus on December 09, 2016, 11:06:13 PM
All bitrates in mlv-lite worked as before.

So no change in this build? At least it hasn't gotten worse!

Quote from: CITY-U1001 on December 09, 2016, 11:29:47 PM
wait testing build for my 50D

I don't want to put out a bunch of builds for platforms that are known not to work yet. My understanding is that only the 5D3.113 and EOSM.202 are working properly and ready for testing. Apparently the 5D2 and 7D are somewhat stable so if you really want to be a crash test dummy, here you go:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec09.50D109.zip

Quote from: Danne on December 10, 2016, 12:23:53 AM
On a sidenote I,ve been using the crop_rec with 5D mark 3 and it works well. I also get good results with raw_twk module and that module is the only one working if you want to view 10 and 12bit footage in cam. I merged and compiled with updated unified code.

Great. You should be able to drop in a module into the ML/modules folder on the card. These should be platform and branch independent--well, in theory. Sometime the backends need to be tweak for a module to work.

raw_twk topic (http://magiclantern.fm/forum/index.php?topic=13163.0) and download link (http://www.magiclantern.fm/modules/modules/raw_twk.mo/raw_twk.mo).

crop_rec topic (http://www.magiclantern.fm/forum/index.php?topic=17021.0) -- does anyone have a link of just the module or does this module require a certain branch? I think it only works on 5D3 and EOSM but I'm not sure.

Quote from: Danne on December 10, 2016, 12:35:18 AM
It won,t work previewing on your computer without the raw_twk module.

Really? I thought raw_twk only affects in camera playback. I use MLVFS to look at the files on my computer and it works fine without raw_twk.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2016, 12:44:08 AM
Correction. raw_twk only affects camera previewing of course  :P

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 12:48:09 AM
Quote from: VanGogh7 on December 10, 2016, 12:28:37 AM
I use raw_rec ( MLV Lite)...and my build  magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip have not the raw_twk module

That should be working. Does a regular nightly build work fine for you? Maybe delete the ML settings and try again. Turn on only the raw_rec module.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 10, 2016, 12:52:13 AM
Quote from: Danne on December 10, 2016, 12:35:18 AM
When you say pink frames at 10bit, 12bit are you referring to when viewing files in camera or viewing dng files on your computer?
It won,t work previewing on your computer without the raw_twk module.


on both camera and pc..

I just downloaded the raw_twk.mo for this build magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip,

It seems to works now on my 5D3 1.1.3! ... it works on cam previewing...
but not on pc. Which converter you use??
I tried different converter with no success:

raw2cdng.1.7.4
mlrawviewer
MLVProducer
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2016, 01:38:12 AM
Use mlvfs or mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 10, 2016, 01:38:57 AM
Quote from: dfort on December 10, 2016, 12:48:09 AM
That should be working. Does a regular nightly build work fine for you? Maybe delete the ML settings and try again. Turn on only the raw_rec module.

Yes nightly builds works well!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 01:42:36 AM
Quote from: VanGogh7 on December 10, 2016, 12:52:13 AM
raw2cdng.1.7.4
mlrawviewer
MLVProducer

I doubt any of those will work with 10bit or 12bit. Like Danne said, try MLVFS if you can. Otherwise, the command line tool, mlv_dump will work if you use it like this:

mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]

Obviously substituting the name of the MLV file you are converting. There was a tweak to mlv_dump to work with lower bit rates so I compiled Mac and Windows versions of mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/mlv_dump_2016Dec09.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: capncannabis on December 10, 2016, 03:51:53 AM
@dfort is there a build to test this on the 600d?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 10, 2016, 05:21:50 AM
On PC , only mlvfs can read 10 & 12bit correctly . Yes you can use mlv_dump But and that a big but  only 10bit can read by applications like A.E. ACR , BM Resolve
12bit from mlv_dump are un-readable as windows can't read them , I posted  here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174826#msg174826)  with examples of 12bit mlv_dump . MLVFS (http://www.magiclantern.fm/forum/index.php?topic=13152.msg127218#msg127218) thread and  mlvfs info (https://bitbucket.org/dmilligan/mlvfs)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 06:27:54 AM
Quote from: reddeercity on December 10, 2016, 05:21:50 AM
12bit from mlv_dump are un-readable as windows can't read them

I downloaded your 12bit file and it works with both MLVFS and mlv_dump. You need to use these options on mlv_dump:

mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]

This is what it looks like without those options:
(https://c3.staticflickr.com/1/322/31171410490_a941c28df1_z.jpg)

And this is what it looks like with the options:
(https://c1.staticflickr.com/1/542/31171411080_017415b6bf_z.jpg)

[EDIT] Forgot to mention that I'm on a Mac and I cross compiled mlv_dump for Windows--it should work on your system.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 10, 2016, 06:50:15 AM
Sweet , thanks when I used mlv_dump I didn't think I needed to uses those option
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 08:29:44 AM
Quote from: reddeercity on December 10, 2016, 06:50:15 AM
...I didn't think I needed to uses those option

Only for 10bit/12bit files. 14bit files should be fine without using those options.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Asiyuk on December 10, 2016, 11:03:48 AM
I try dec 4 build from @dfort with raw_rec (MLV Lite) on 5d3_1.1.3:
- 12 bit 1920x1080 full work except raw-based histogram while recording
- 12 bit in 5x zoom mode - freeze LV while recording (all lv full of bw dots), but video is ok.
- play doesn't work (pink dots video)

I use MLVFS + DaVinci in post, all good with files. I even use this 4dec build in yesterday commercial shooting.

PS: sorry for by bad english  :-X
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 10, 2016, 03:09:39 PM
A question to people like a1ex and dmilligan:
how can we be sure about addresses we found?

we're porting raw_slurp to 700D and I think we found the right raw buffer address (lv is working good) but when we look at the file we've got only strange things.
here an example: https://drive.google.com/file/d/0BxMUvr5E8ANaaDdsS0JUVFFwVjA/view?usp=sharing

it happens at all bit depth.

Can it be related to the edmac channel instead of wrong buffer address? (ROM call to ROM on 700D anyway :D )
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 10, 2016, 05:12:57 PM
Quote from: eNnvi on December 10, 2016, 03:09:39 PM
how can we be sure about addresses we found?

Printing DEFAULT_RAW_BUFFER should give the same value(s) with both methods.

To see how Canon does it, look for something like StartMem1Edmac (only on models with EvfState). The older models (550D and earlier) are a bit more difficult.

The 700D should be very similar to EOS M.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 11, 2016, 12:09:50 PM
Quote from: dfort on December 10, 2016, 01:42:36 AM
I doubt any of those will work with 10bit or 12bit. Like Danne said, try MLVFS if you can. Otherwise, the command line tool, mlv_dump will work if you use it like this:

mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]

Obviously substituting the name of the MLV file you are converting. There was a tweak to mlv_dump to work with lower bit rates so I compiled Mac and Windows versions of mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/mlv_dump_2016Dec09.zip


MLVFS on Mac works well,

MLVFS on Win. I get in the cmd:

C:\>cd \mlvfs_X86

C:\ mlvfs.exe Z:\ --mlv_dir=E:\MLVFS
Unmont


So it doesn't create a virtual drive...where is the mistake? I installed Dokany and Mlvfs_x86

I tried to install the latest dokan version, now I have a other issue...dokanfuse.dll is missing..it's a pain!!!

Now, I have Win 7 64 bit Ultimate

the solution is to install the 32 bit Microsoft Visual C++ 2015 Redistributable (x86) - 14.0.23026 version
:)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 11, 2016, 07:37:36 PM
What is unmont?

For me on Win10 64bit I use a batch file (MLVFS.bat) on my desktop, could add it to start up I guess but don't always need MLVFS. Batch file has one line in it as below.

cmd /k G:\MLVFS_x86\mlvfs.exe M:\ --mlv_dir=G:\MLV_FS\Projects

Double click fires up a cmd window and does the necessary.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 11, 2016, 09:34:29 PM
Got CONFIG_EDMAC_RAW_SLURP working on the 700D

https://bitbucket.org/hudson/magic-lantern/pull-requests/781/700d-enable-config_edmac_raw_slurp/diff

@eNnvi -- the missing piece was:

edmac-memcpy.c
void edmac_raw_slurp(void* dst, int w, int h)
{
     /* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
-#ifdef CONFIG_EOSM
+#if defined(CONFIG_700D) || defined(CONFIG_EOSM)
     uint32_t dmaFlags = 0x20000000;
#else
     uint32_t dmaFlags = 0x20001000;


Note: this might also work for the 650D and maybe the 100D if we can get the DEFAULT_RAW_BUFFER address for those platforms.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jc on December 11, 2016, 09:49:11 PM
Has there been any further looking into how the losses cr2 raw compression is applied ? Could the discovery of running the firmware in QEMU or finding a the "Eeko" core help this discovery ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 11, 2016, 10:15:20 PM
QuoteGot CONFIG_EDMAC_RAW_SLURP working on the 700D
Breakthrough? 10bit/12bit kickin it with raw_rec?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 11, 2016, 10:43:32 PM
Quote from: Danne on December 11, 2016, 10:15:20 PM
10bit/12bit kickin it with raw_rec?

Yep--on the 700D. Who's next?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 12, 2016, 01:25:58 AM
550D ? But as A1ex says above, will be more difficult.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 12, 2016, 05:34:46 AM
Quote from: dfort on December 11, 2016, 09:34:29 PM
this might also work for the 650D and maybe the 100D if we can get the DEFAULT_RAW_BUFFER address for those platforms.
Awesome!
Got 650D here, VM all set up and ready for testing. Have some experience with finding STUBS addresses. Can you give us some guidelines on how to find this god damn DEFAULT_RAW_BUFFER address?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2016, 07:43:01 AM
Quote from: Rewind on December 12, 2016, 05:34:46 AM
Got 650D here, VM all set up and ready for testing. Have some experience with finding STUBS addresses. Can you give us some guidelines on how to find this god damn DEFAULT_RAW_BUFFER address?

Great, check dmilligan's posts. He provided all the hints we used to find the right address. You can check on the pull request for the 700D (https://bitbucket.org/hudson/magic-lantern/pull-requests/781/700d-enable-config_edmac_raw_slurp/diff) to make sure you find all the pieces that are needed for the 650D. They should be the same with the exception of the DEFAULT_RAW_BUFFER address.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 12, 2016, 09:03:54 AM
Thanks to dfort, i've managed to find the right address for 650D:

#ifdef CONFIG_650D
#define DEFAULT_RAW_BUFFER MEM(0x25B00 + 0x3C)
#endif


LiveView works during recording, no pink frames or glitches in extracted dngs (raw_rec, both 12- and 10-bit). Some shifting occurs though, focus pixel masks no longer works (MLVFS windows), need more testing.

Upd. Actually, there are no focusing pixels anymore on 650D (raw_rec, 14 bit as well as 12 and 10 bit), but there are these along the contrast edges (all over the frame). What am I missing? Were there some changes in raw framework?
(http://s2.1pic.org/files/2016/12/12/71a532d2cd490e8b5db6.png)

What PREFERRED_RAW_TYPE should we use for 650D to get the old behavior (usual horizontal pattern of focusing pixel, removable by PDR etc.)?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 12, 2016, 04:33:41 PM
Do you all get this from live view while doing crop recording?

(http://forumbilder.se/FBVC0/img-2009.JPG)

Using 4th December build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 12, 2016, 04:43:48 PM
I had this with Global Draw on...try with GD OFF
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 12, 2016, 05:09:49 PM
D_Odell. Are you using raw_twk? When in crop mode this will surely happen.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2016, 06:44:21 PM
Quote from: Rewind on December 12, 2016, 09:03:54 AM
What PREFERRED_RAW_TYPE should we use for 650D to get the old behavior (usual horizontal pattern of focusing pixel, removable by PDR etc.)?

I'm seeing the same happening on the EOSM and 700D. I tested out some different RAW_TYPE's and found some that show the old focus pixel patterns, others that show only a few pixels, mostly around high contract boundaries and yet others that show a pixel pattern all over the image including areas that we've never seen focus pixels before. I reported on that in the Dealing with Focus Pixels in raw video topic (http://www.magiclantern.fm/forum/index.php?topic=16054.msg176032#msg176032).

We recently increased the RAW_DEBUG_TYPE limit and switched over to defining the PREFERRED_RAW_TYPE in hex instead of decimal values. The pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/780/increase-raw_debug_type-limit/diff) is quite an interesting read.

To activate RAW_DEBUG_TYPE, define it here:

#undef RAW_DEBUG        /* define it to help with porting */
#undef RAW_DEBUG_DUMP   /* if you want to save the raw image buffer and the DNG from here */
#undef RAW_DEBUG_BLACK  /* for checking black level calibration */
#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
/* see also RAW_ZEBRA_TEST and RAW_SPOTMETER_TEST in zebra.c */


You also need to give it a starting value. Anything will do because you can change it from a menu:

#ifdef CONFIG_650D
#define PREFERRED_RAW_TYPE 0x12
#endif


This will give you a new Debug menu item that allows you to enter values from 0 to ffff (65,535).

(https://c3.staticflickr.com/1/766/31229380130_7893122693.jpg)

Try them all out--that should keep you busy for a while. Actually, they repeat and many do not produce valid images or cause raw detect errors.

You said that the frame might have shifted? In the raw_lv_get_resolution function of raw.c you might check if the 650D uses either of these resolutions--I'm pretty sure it uses the 700D values:

    #ifdef CONFIG_600D
    *width  = zoom ? 2520 : mv1080crop ? 1952 : mv720  ? 1888 : 1888;
    *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
    return 1;
    #endif


    #ifdef CONFIG_700D
    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
    return 1;
    #endif


Are you planning on putting up a pull request for this? It would be great to get some feedback from other 650D users.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 12, 2016, 06:48:33 PM
I've disassembled the 50D's FW and found 4 instances where 'lv_raw_dump' is mentioned but after that I'm lost  ??? - yes I did read @dmilligan's post (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969)) but I still don't know what I'm looking at.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2016, 07:34:44 PM
Quote from: Andy600 on December 12, 2016, 06:48:33 PM
...found 4 instances where 'lv_raw_dump' is mentioned but after that I'm lost...

Yeah, I found the same in my EOSM and 700D disassembly. I had to look for the address that dmilligan used on the EOSM and find the equivalent in the 700D. In art that would be like tracing a master's drawing but hey, it worked.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 12, 2016, 07:42:44 PM
there are 4 because 2 are real lv_raw_dump, the other 2 shoulde be lv_raw_dump2

anyway the same instances of lv_raw_dump should take you to the same spot (address jump)

from here it's harder :D

there should be a sub that actually loads the address of the raw buffer, you need to find that! how? well, read the lv_raw_dump function (the bl from the string you found) and check when the return pointer is used for reading or writing something
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 12, 2016, 08:10:03 PM
Quote from: Danne on December 12, 2016, 05:09:49 PM
D_Odell. Are you using raw_twk? When in crop mode this will surely happen.
Hi Danne! No not at that time, when used it was completely green..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 12, 2016, 09:05:49 PM
Quote from: dfort on December 12, 2016, 06:44:21 PM
To activate RAW_DEBUG_TYPE, define it here:
Got it. Tested, and at least 0x10 PREFERRED_RAW_TYPE works like we used to (650D, raw_rec, both 10 and 12 bit). No tearing, no artifacting, no shifting (even PDR works). Absolutely usable for shooting already. Thank you dfort!

Quote from: dfort on December 12, 2016, 06:44:21 PM
The pull request is quite an interesting read
Indeed. I'll try other raw types as well. By the way, when using mlv_rec instead of raw_rec, i got some tearing in 10 bit mode: every other frame's top third is slightly shifted (5 or 6 pix) to the top.

Quote from: dfort on December 12, 2016, 06:44:21 PM
Are you planning on putting up a pull request for this
Shame on me, but i spent some years in different kind of activity, and i just forget everything about Git, Bitbucket, branching etc. Actually coding is way more understandable :) I'll try to refresh and join later. As for 650D's CONFIG_EDMAC_RAW_SLURP, it may be easier for you to just add the proper DEFAULT_RAW_BUFFER MEM(0x25B00 + 0x3C) in your commit. Other changes are basically the same as yours for 700D obviously.
Or I can share my test build for other 650D users if this make sense.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 12, 2016, 09:15:13 PM
From this analysis (http://www.magiclantern.fm/forum/index.php?topic=18393.0), PREFERRED_RAW_TYPE 0x10 appears to be the best choice for all DIGIC 5 and 6 cameras, and likely 0x5 for all DIGIC 4.

The EDMAC channel should probably be 0x12 for all DIGIC 5 models (unless used by something else).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 12, 2016, 09:31:01 PM
Quote from: eNnvi on December 12, 2016, 07:42:44 PM
...from here it's harder :D

there should be a sub that actually loads the address of the raw buffer, you need to find that! how? well, read the lv_raw_dump function (the bl from the string you found) and check when the return pointer is used for reading or writing something

Ok, but I don't understand the last bit  :-\

the bl is loc_ff064830 but what do I do with it? How do I check?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2016, 11:36:12 PM
Quote from: Rewind on December 12, 2016, 09:05:49 PM
...when using mlv_rec instead of raw_rec, i got some tearing in 10 bit mode: every other frame's top third is slightly shifted (5 or 6 pix) to the top...

Yeah, noticed that too, raw_rec (MLV Lite) is working better at this time.

Quote from: Rewind on December 12, 2016, 09:05:49 PM
...it may be easier for you to just add the proper DEFAULT_RAW_BUFFER MEM(0x25B00 + 0x3C) in your commit. Other changes are basically the same as yours for 700D obviously.
Or I can share my test build for other 650D users if this make sense.

I updated my pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/781/650d-and-700d-enable/diff) so it includes the 650D. @Rewind -- are you on bitbucket? There are a few Rewinds there so I'm not sure if one of those is you. In any case, I'd appreciate if you check it out and make sure I didn't leave out anything important.

I made a new set of test builds mainly for 650D and 700D users but found out that the 60D and 600D has CONFIG_EDMAC_RAW_SLURP defined and heard reports that the 7D was also working so I went ahead and built all the platforms. This time I didn't zip them all up one bundle so we can check up on the number of downloads per platform. Also took down the older builds to avoid confusion.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kamranjon on December 13, 2016, 01:33:12 AM
Just tested this new build on 7D - it works! - the live view in some cases works as well which is awesome - what i had to do was: i turn global draw/liveview on, and then i turn it off, and when i hit record i am able to see what i am shooting! If i don't do that first it doesn't seem to work, it just freezes like it had previously.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on December 13, 2016, 07:38:18 AM
Canon 50D to MLVFS Workflow w/ 12-12-16 Build:

Module Report: RAW Lite

1. 10-bit and 12-bit display a repeating first frame; every other frame.

2. 5X features "pink frames" in 10-bit, 12-bit, and 14-bit; every frame.

3. 14-bit works great; aside from 5X zoom.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 13, 2016, 09:03:40 AM
Just tested 500D, 550D, 600D and the 7D with dfort latest compilations. (raw_rec, mlv_lite)

500D , raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
550D, raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
600D, raw10bit, 12bit works flawlessly! raw slurp enabled already?
7D, raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 12:10:27 PM
Will the 650D/700D port work on the 750D? Or is that a different kettle entirely... ?   :-\
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on December 13, 2016, 12:22:59 PM
How to saddle a horse not born yet?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 12:26:52 PM
Thanks Walter. What's delaying the birth? Testers? Major technical differences? I'm genuinely interested.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on December 13, 2016, 12:46:09 PM
As always: Lack of people with skills (Assembler and C for embedded devices, preferable ARM architecture) and spare time at hand. It will take (estimated) some hundred hours - partly mind numbing - to port ML to a new platform. And this is not just a new platform but a different processor generation, too.
And such a person will be asked to maintain the cam and its ML port. If there is a new Canon firmware and it is decided to port ML for this new firmware: Yupp, reverse engineer it a second time (should be easier, but PITA, 2nd edition).
See threads for 750D, 760D and 80D. 7D2 will have to wait.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 12:49:19 PM
Thanks for the comprehensive reply Walter.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 13, 2016, 12:51:18 PM
First i didn't understand why you post it here as the thread is about something else.
Second you Can get your cam, study assembly, C coding, arm architetture by your own and help porting the code speeding up a lot the porting process
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 12:55:06 PM
@ eNnvi Walter just explained the exact same thing to me – but with tact.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 13, 2016, 03:21:00 PM
Sorry i didn't mean to be rude but that's what was told me sometime ago (for another project) and i tried and studied and now i Can at least understand the concept and the c code behind ml but i'm still learning a lot about assembly (even if i studied it at University). There's no limit about you but about how much you want to help and learn :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 04:23:57 PM
@eNnvi Thanks. No offence taken. :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 13, 2016, 05:15:14 PM
Quote from: Danne on December 13, 2016, 09:03:40 AM
500D , raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
550D, raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
600D, raw10bit, 12bit works flawlessly! raw slurp enabled already?
7D, raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)

This makes sense because the 600D is the only camera on the list that has CONFIG_EDMAC_RAW_SLURP defined. I just went through all the platforms and checked the ones that are "slurping":















1100D.105No. Last Successful Build (2015-05-03 00:25) What's up with that?
500D.111No
50D.109No but @Andy600 is working on it.
550D.109No
5D2.212No but @reddeercity is reporting some success.
5D3.113Yes
600D.102Yes
60D.111Yes
650D.104Yes with pending pull request (test build posted)
6D.116No
700D.114Yes with pending pull request (test build posted)
7D.203No but maybe @nikfreak is working on it?
EOSM.202Yes
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 13, 2016, 05:26:07 PM
Working on 6D and 7D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 14, 2016, 01:34:47 AM
Thanks @dfort for the latest updates.

@EOSM users:

Are you guys able to get mlv_play.mo to work w 10/12-bit files after each recording takes? I notice sometimes I have to either turn off camera and back on to get mlv_play.mo to work properly otherwise I'll get a black screen stuck on LCD liveview.

Btw it's nice to be able to see LiveView while recording. Also confirmed to work in crop_mode.mo but not in 5x zoom mode.

https://vimeo.com/195551541

More test to come later with 5D3 & 7D.

Perhaps I'll wait until @eNnvi provides a new build for the 7D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: saulbass on December 14, 2016, 01:49:47 AM
@dfort - thanks for the recent 650D update - (magiclantern-raw_video_10bit_12bit.2016Dec12.650D104.zip).

10bit RAW 1280 x 720 works continuously - only issue are the focus pixels and some image tearing but I'm not using my fastest SD card.

With the MLV sound added to this the video is very choppy.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 14, 2016, 02:54:11 AM
Hi and thanks for ALL  :)

I test the new version for the 7D and i can record continuos and with the camera screen (LiveView) good (canon preview), but in camera with mlv_play.mo i just see pink/flashy image. In computer, after use MPV player i see this pink and good image blinking very fast.

Cheers
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 14, 2016, 03:25:03 AM
Quote from: eNnvi on December 13, 2016, 05:26:07 PM
Working on 6D and 7D

Great. Make sure and check out the comment @nikfreak posted on the 650D/700D pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/781/650d-and-700d-enable/diff#comment-28272540). You're probably way ahead of me!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on December 16, 2016, 06:26:49 AM
Can someone upgrade the first post, it's from 2013...!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 16, 2016, 06:47:52 PM
i think i got 6D almost working, need my tester to tell me if the last build works.

Can someone please share me a 60D ROM1.bin file? need it to find the way to find (strange to say :P ) the address of 7D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 16, 2016, 08:13:20 PM
Great work eNnvi  :)
Rawslurp for 6d.
Happily willing to test, is the source somewhere available on bitbucket?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 16, 2016, 09:41:16 PM
eNnvi: unfortunately, the 60D won't help you solving the 7D. It's old generation, and the dual processor makes it probably the most difficult of all models.

From what I've tried so far on those old models (without much success yet), we have at least two problems:
- the raw buffer is not allocated all the time (as it's done on the new models) -> easy to fix, see the 1100D raw branch
- the vsync hook currently used doesn't appear to be the best for getting a raw image
- the exact behavior of EDMAC in case of input/output dimension mismatch is not very well known

Solving those models may require either some serious low-level research, or a great dose of luck. I'm just scratching the surface here (http://www.magiclantern.fm/forum/index.php?topic=18315).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 16, 2016, 10:04:43 PM
Im getting some really great images in lowlight with 10 and 12-bit mod. Really superb work from you guys. Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 17, 2016, 06:47:14 PM
Did a little testing on my 600D.
At 12 bit, 1280 @ 2:35 / 23.9fps, got 31s.  For Indy film, this is about 90% of your shots.
At 10bit, 1280 @ 2:35 /23.9fps, continuous.  For Indy film, the few shots longer than 30s switch to 10 bit. 

Selected MLV_raw, and notice that playback at 12 bit crashes camera, but at 10 bit, it doesn't but its preview is only white with verticle lines. raw_twk.mo provides 12bit & 10bit previews

I am mega impressed with the improvements. 
1280 @ 2:35/24 fps 10 bit raw, finally makes the humblest of cams a useful raw video cam (new learner or student).
Sorry I have not been using raw video in over 1.5 years.
I am using Windows 10, what's the best automated work-flow to convert my MLV_raw into PPro or Resolve friendly format?
(PM please as I dont want to go off topic on a dev forum). 

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 17, 2016, 07:18:44 PM
For 10/12bit playback add the excellent raw_twk.mo module which g3gg0 recently polished.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on December 17, 2016, 08:46:16 PM
How's 550D? Last time I tried it had pink every other frame.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 17, 2016, 09:32:33 PM
How hard is to type 550D in your browser's search box?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on December 18, 2016, 12:06:50 AM
I did, I just thought that maybe somebody had info that wasn't yet shared here. Sorry I ruffled your feathers.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 01:05:32 AM
Quote from: Danne on December 17, 2016, 07:18:44 PM
For 10/12bit playback add the excellent raw_twk.mo module which g3gg0 recently polished.

i just polished mlv_play and raw_twk and pushed them into unified.

mlv_play directly uses raw_twk now if available.
-> no raw preview lockups during recording

(also added a zoom feature when pressing zoom button during playback)

needs for sure some tests.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 18, 2016, 05:47:58 AM
Processed the 600D 1280x544 (2:35), 24fps (canon default) for 29s.

Used Mystic to convert to DNG, but got few lost frames.  In LR saved to TIFFs 16bit sRGB, and built clip in AE with skipped frames.   Got wierd wabble when panned.

Tried again by DNG dropped from card into Resolve Lite, had bigtime shadow noise but no wierd wabble on same pan. 


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 18, 2016, 07:22:03 AM
@ShootMeAlready the only Correct extractors are MLVFS and new updated mlv_dump
all others are not designed to extract bit reduced raw (10-12bit) you may get a some what usable
image from 10bit raw with other convertors but it will not look right or have noise etc.... .
the others are designed to read 14bit raw . If others do it will be only by chance .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 18, 2016, 07:31:15 AM
Quote from: g3gg0 on December 18, 2016, 01:05:32 AM
i just polished mlv_play and raw_twk and pushed them into unified.

Could you please merge unified into raw_video_10bit_12bit or at least raw_twk into raw_video_10bit_12bit? Seems like the reduced bit raw files really need it.

Quote from: g3gg0 on December 18, 2016, 01:05:32 AM
mlv_rec directly uses raw_twk now if available.

You mean mlv_play, right?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 18, 2016, 09:07:31 AM
Quote from: dfort on December 18, 2016, 07:31:15 AM
Could you please merge unified into raw_video_10bit_12bit or at least raw_twk into raw_video_10bit_12bit? Seems like the reduced bit raw files really need it.

+1

Quote from: dfort on December 18, 2016, 07:31:15 AM
You mean mlv_play, right?

Pretty sure that's what he meant.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 12:27:31 PM
yeah i meant mlv_play of course. corrected that last post.
grafted that patches into the branch. happy testing ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 18, 2016, 01:07:53 PM
Quote from: g3gg0 on December 18, 2016, 01:05:32 AM
i just polished mlv_play and raw_twk and pushed them into unified.

The unified branch doesn't have variable bit depth support in raw.c, so I'm not sure what was the reason for pushing directly to unified. Ran a quick test on 5D3 and recordings at lower bit depths are corrupted.

Sounds like a rushed move to me. Looks like the mlv_rec changes were committed by mistake, sorry, all fine now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 01:33:32 PM
wait, you used unified (which doesnt support 10/12 bit video) to play back files that were recorded with raw_video_10bit_12bit branch?
how did you get the lower bit depth recordings with it? :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 01:51:38 PM
ah now i understood. initially i wanted to graft the mlv_play patch that adds variable bit depth only.
things went wrong and the whole patch was grafted into unified, giving 10/12/14 bit mlv_rec in unified which obviously doesn't work :)

fixed now, thanks a1ex
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 18, 2016, 04:48:44 PM
@g3gg0

I see that raw_twk is now in the raw_video_10bit_12bit branch--Thanks!

A couple of questions.

Shouldn't the raw_twk module be added to Makefile.modules.default on the raw_video_10bit_12bit branch since it is required for playing back 10/12 bit video?

Does raw_twk load automatically with mlv_play or does the user have to turn on both modules? Obviously having to turn on just mlv_play would be better but merging raw_twk into mlv_play might be even better, if that's possible.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 05:55:15 PM
i'd say it makes the most sense in ML core.
but it is not as good as i want it to be, guess a1ex sees it likewise.

so either keep it as optional module and research how to make stuff clean, faster and more stable
or integrate that half-cooked stuff into core, requiring some workarounds.

well, i'd prefer to keep it as an (surprisingly stable) option for those who need the high performance.
and meanwhile we look for the golden solution with full engine support.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on December 18, 2016, 07:14:53 PM
@all
could someone post some 10/12bit samples? (samples-thread (https://www.magiclantern.fm/forum/index.php?topic=11899.0))

regards chmee
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 18, 2016, 08:42:07 PM
Some initial testing around the blazing fast raw_twk module. I also tested it with the magnificent crop_rec module on a 5D mark III.

Pros
5D mark III mlv_lite
1 - The 3xzoom mode grayscale preview crash bug is gone!
2 - Playback works great also with the magnificent crop_rec module(tested centred 5xzoom)
3 - 10bit, 12bit, 14bit works

550D mlv_lite
1 -  Briefly tested on a 550D and all seems to work the same as for the 5D mark III but with the lack of slurp stream causing 10bit, 12bit files corruption(another issue not related to raw_twk).

eos m mlv_lite
1 - All is working but no overlay buttons. Can,t delete etc.

Cons (initial testing summary)
1 - Scrolling back and forth between mlv files will soon cause the previewing to turn black. This behaviour exists for both mlv_lite and mlv_rec
2 - Eos m still lacks play, pause, next, delete buttons


@Chmee, posted some samples of my cat and linked the clips in your samples thread.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on December 18, 2016, 08:56:36 PM
Quote from: Danne on December 18, 2016, 08:42:07 PM
Some initial testing around the blazing fast raw_twk module. I also tested it with the magnificent crop_rec module on a 5D mark III.

Pros
5D mark III mlv_lite
1 - The 3xzoom mode grayscale preview crash bug is gone!
2 - Playback works great also with the magnificent crop_rec module(tested centred 5xzoom)
3 - 10bit, 12bit, 14bit works



Amazing new ! Can't wait to test the new 10/12bits build coupled with the super crop_rec module version :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 18, 2016, 09:09:45 PM
Quote from: jc on December 11, 2016, 09:49:11 PM
Has there been any further looking into how the losses cr2 raw compression is applied ?

Yes (http://www.magiclantern.fm/forum/index.php?topic=18443).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: AWPStar on December 18, 2016, 10:59:05 PM
I just tried to decode 10 and 12 bit's raw files.
http://www.magiclantern.fm/forum/index.php?topic=15271.msg176632#msg176632

I think i can implement it in several days.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 19, 2016, 12:03:22 AM
I don't know if what I have achieved counts as successful, but I think I've got closest to working 12 bit on the 5D2. As we all know raw_rec produces split frames all the time, and I haven't had any success with it, but mlv_rec produces no split frames and only a small bar of noise at the top in 12 bit, or a large one in 10 bit, so I tried to move it down using digital dolly, and it actually got rid of the bar of noise! Clean, 12 bit, 1856x1044, continuous... AND MLV... on the 5D2... with potential to have audio!!! but of course tilt shifted down.

Proof:


If mlv_rec has something that is more functional it could be transferred to raw_rec right? Also there must be a work around to get it in the middle, maybe by adding blank space at the top of the video and making it so that digital dollying all the way down is now the middle? I'm sure this is progress, isn't it?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 19, 2016, 12:32:16 AM
Posted a new set of test builds of the raw_video_10bit_12bit branch now with the new and improved raw_twk module.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 19, 2016, 01:35:42 AM
Quote from: Danne on December 18, 2016, 08:42:07 PM
eos m mlv_lite
1 - All is working but no overlay buttons. Can,t delete etc.

Found the same EOSM issue here. Turn off raw_twk and the overlay buttons are back but of course it doesn't playback 10bit files.

Works fine on the 700D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 19, 2016, 02:28:32 AM
mlv_rec got a fix for the alternating frame errors with lower bit depths, along with some other stuff that was outdated or simply odd.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 05:41:33 AM
QuoteFound the same EOSM issue here. Turn off raw_twk and the overlay buttons are back but of course it doesn't playback 10bit files
?
My tests was done around raw_twk module with eos m and other cameras here.
http://www.magiclantern.fm/forum/index.php?topic=5601.msg176617#msg176617
There were never any issues with mlv_play alone on eos m.

Quotemlv_rec got a fix for the alternating frame errors
Great! More than welcome.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 19, 2016, 07:26:43 AM
Quote from: g3gg0 on December 19, 2016, 02:28:32 AM
mlv_rec got a fix for the alternating frame errors with lower bit depths, along with some other stuff that was outdated or simply odd.

Alternating frame errors with mlv_rec at 10bit are still present on the EOSM and 700D. It seems to be better but I haven't done a regression test.

[EDIT] Oops missed that last commit -- yes, the alternating frame errors with lower bit depths is fixed. Thanks! Now we can record audio with 10/12bit raw video.

Making a new set of test builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 08:05:47 AM
In already? That is brilliant.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 19, 2016, 08:41:13 AM
Yay for audio
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 19, 2016, 10:42:36 AM
Super awsome!  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 01:42:09 PM
5D mark III
Just tested mlv_rec with audio on 5D mark III and both 10bit and 12bit works really good. Very nice. Crop_rec module is also working with audio and all frames looked great. I didn't,t test this very much so more testing.

Eos m
I keep bugging about this cam but its a great little cam that needs all the love it can get.
Records mlv_rec 10/12bit fine. If raw_twk.mo is added and you try to record sound will make the camera crash kind of. There is a faulty message saying 'threads failed to start' I think and then the camera needs to be restarted. If disabling raw_twk everything works, except for previewing 10/12bit files of course.(Yeah, and button overlays still missing with raw_twk enabled)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 19, 2016, 01:53:22 PM
the EOS M has too few thread ids. so this is kind of a known issue we cannot do much about :(
less modules that use threads would help.
(also reworking modules that they only start/end threads when needed would help)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 01:59:55 PM
I see, all in all great working cam delivering continuous 10bit recordings on highest resolution.
Any thoughts on mlv_play buttons? They seem vanished into oblivion once raw_twk starts running?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 19, 2016, 02:51:13 PM
mlv_play issues -> http://www.magiclantern.fm/forum/index.php?topic=9062.0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on December 19, 2016, 02:57:46 PM
Eager to test 10bit mlv on 70D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 03:00:40 PM
@g3gg0
Thanks a lot for taking the time on this.
Just to be sure here. Mlv_play buttons(delete, play, pause, scroll etc) If not going through raw_twk. Link you are posting tells me I have to go through file manager instead? Conclusion is that raw_twk can,t maintain the otherwise working buttons?

*Updated more info here
http://www.magiclantern.fm/forum/index.php?topic=9062.msg176677#msg176677
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 19, 2016, 03:52:58 PM
50D 10bit
last build 10bit_12bit.2016Dec18.50D109
view with MLVProducer
(http://image.prntscr.com/image/483c01587cfa4e43abbeeab24277e763.jpeg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: AWPStar on December 19, 2016, 04:01:28 PM
@CITY-U1001
QuoteI just tried to decode 10 and 12 bit's raw files.
http://www.magiclantern.fm/forum/index.php?topic=15271.msg176632#msg176632

I think i can implement it in several days.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 19, 2016, 09:02:56 PM
Quote from: Danne on December 19, 2016, 01:42:09 PM
5D mark III
Just tested mlv_rec with audio on 5D mark III and both 10bit and 12bit works really good. Very nice. Crop_rec module is also working with audio and all frames looked great. I didn't,t test this very much so more testing.

Eos m
I keep bugging about this cam but its a great little cam that needs all the love it can get.
Records mlv_rec 10/12bit fine. If raw_twk.mo is added and you try to record sound will make the camera crash kind of. There is a faulty message saying 'threads failed to start' I think and then the camera needs to be restarted. If disabling raw_twk everything works, except for previewing 10/12bit files of course.(Yeah, and button overlays still missing with raw_twk enabled)

Hi Danne, how is the workflow to convert 10bit including the soundfile?  for the MLV I use MLVFS..and for the soundfile named ?? MLV_REC.TMP ??

thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 19, 2016, 09:04:07 PM
New build "magiclantern-raw_video_10bit_12bit.2016Dec18.5D3113"

Tested one 5D mark III 1.1.3 with mlv_rec and raw_twk

Work on 10, 12 bit also in crop mode in high res really nicely ! Also work in slowmo 50p with stretch mode.
Could it be possible to do somethings and record slowmo without stretch in full frame mode ?

dfort, I have no idea how you did it but you are dat king ! Huge deep thanks and respect
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 19, 2016, 09:06:29 PM
Having the same problems that @Ilia3101 have but with the 7D

Going trie the new version of @dfort
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 09:10:20 PM
@VanGogh7
Audio is included and revealed automagically when opening the file in mlvfs. Mlv_dump will build a solid wav file as well upon converting to dng files.

@MAG
With the crop_rec module I think it,s posible to record unstretched mv720 mode(not mv1920)

Also if previewing the MLV files with raw_twk enabled please test to go back a few files then forward a few files. Do this a couple of times and see if screen turns black?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 19, 2016, 09:40:33 PM

@dfort

With your last version for the 7D it got the same problems... maybe is needed to merge/compile the new update from the mlv_rec that got a fix for the alternating frame errors in lower bit depths.

Thanks=))
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 19, 2016, 09:47:20 PM
@Danne

Thanks, strange, first on my half-full Card ( Komputerbay 64GB ) MLFVS was not able to create a .wav file from the MLV_REC.TMP. After formatting the card now all works fine! :)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 09:52:45 PM
You have to enable mlv_snd module. MLV_REC.tmp has nothing to do with this. Anyway, seems you have it working so let,s keep up the 10/12bit testing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 19, 2016, 09:59:52 PM
Quote from: Danne on December 19, 2016, 09:10:20 PM
@MAG
With the crop_rec module I think it,s posible to record unstretched mv720 mode(not mv1920)

Also if previewing the MLV files with raw_twk enabled please test to go back a few files then forward a few files. Do this a couple of times and see if screen turns black?

Yes in playback, the screen turns black after couple of time. (raw_twk and mlv_play)

Also when I say record crop mode, I mean in ZOOM mode. The build I have doesn't have "crop_mode" module (2016Dec18.5D3113)

So, it's impossible to record fullframe 1080p without stretch ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 10:03:38 PM
I think maximum is 38 fps mv1080. Also let,s try to stay on topic here. You can dig out a lot of info with 'search' regarding stretch/unstretch footage.
Thanks for verifying raw_twk issue
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aschille84 on December 19, 2016, 10:08:25 PM
6D: 10/12-bit with mlv_snd working in raw_rec and mlv_rec. Nice!

In cam preview with raw_twk working, but goes black when scrolling thru 3 clips or more.

Live view freezes on canon/ml greyscale without global draw preview, scrambled grey on ml greyscale w/global draw on.

latest @dfort build "magiclantern-raw_video_10bit_12bit.2016Dec18.6D116"
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 10:18:30 PM
QuoteLive view freezes on canon/ml greyscale without global draw preview, scrambled grey on ml greyscale w/global draw on.
How do you mean? Does the camera freeze when previewing with grayscale?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aschille84 on December 19, 2016, 10:30:27 PM
Quote from: Danne on December 19, 2016, 10:18:30 PM
How do you mean? Does the camera freeze when previewing with grayscale?


On the 6D normal canon preview freezes in a still frame. I doesnt freeze on greyscale, its scrambled grey as earlier when global draw is on. So, limited preview options.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 19, 2016, 10:41:01 PM
At this moment a bunch of camera have working live view while recording 10/12 bit (5D3, 60D, 600D, 650D, 700D and the EOS-M)
other ones have freezing live view while recording 10/12 bit video.



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on December 20, 2016, 07:52:28 AM
A brief summary of the test MLV-REC 10bit RAW on 600D:
RAW_twk - works
MLV_ snd - works
Most of the raw-videos converted with MLV_dump are badly decoded.

:-)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 20, 2016, 08:01:04 AM
When using mlv_dump.
http://www.magiclantern.fm/forum/index.php?topic=5601.msg176263#msg176263
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DavidSh on December 20, 2016, 09:26:14 AM
Any news on 5d3.123?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on December 20, 2016, 03:54:20 PM
@D

Can you update the first post with the latest results?

- what works at the moment
- tools for process on pc or/and mac at the moment

It's just a couple of lines.

Thank you.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 20, 2016, 07:23:53 PM
Using 18dec build and the crop marks or actual pixels aren't shown in live view on 113 5D3. If I remembered correct the 4 Dec showed correct crop/actual pixels while recording. Anyone experiencing the same? Problem is that it's impossible to frame while rolling. Best, David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 20, 2016, 08:21:16 PM
Quote from: D_Odell on December 20, 2016, 07:23:53 PM
Using 18dec build and the crop marks or actual pixels aren't shown in live view on 113 5D3.

5D3.113 has the most amount of downloads by far so someone should be able to comment on that.

Just for fun, here are the download counts of the December 18 test builds. Checkmarks are for cameras that are working with CONFIG_EDMAC_RAW_SLURP:














5D3.113-28
7D.203-8
5D2.212-6
EOSM.202-6
550D.109-5
6D.116-5
600D.102-4
50D.109-3
60D.111-3
650D.104-3
700D.114-3
500D.111-1

Quote from: DavidSh on December 20, 2016, 09:26:14 AM
Any news on 5d3.123?

That is a bit more challenging to build and I don't have a camera to check it out but here's a build for any 5D3.123 user who would like to check it out--please report your findings.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123.zip

Quote from: budafilms on December 20, 2016, 03:54:20 PM
@D

Can you update the first post with the latest results?

@d started this topic on May 22, 2013, has only made 6 posts and hasn't been seen on the forum for over two years.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 20, 2016, 08:51:13 PM
Quote from: D_Odell on December 20, 2016, 07:23:53 PM
Using 18dec build and the crop marks or actual pixels aren't shown in live view on 113 5D3. If I remembered correct the 4 Dec showed correct crop/actual pixels while recording. Anyone experiencing the same? Problem is that it's impossible to frame while rolling. Best, David

I try the 18dec 5DIII there is no crop_rec and when I zoom record I don't have good liveview...

May be dfort do some test before adding crop_rec ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 20, 2016, 09:57:21 PM
Quote from: MAG on December 20, 2016, 08:51:13 PM
May be dfort do some test before adding crop_rec ?

I'm doing test builds of just the raw_video_10bit_12bit branch for now. The crop_rec module is only working for the 5D3 and EOSM, is on a different branch and I don't want to introduce yet another variable until all possible cameras are working. If you want to experiment with crop_rec try merging branches--it really isn't that hard to compile and there are several tutorials in the General Development Discussion (http://www.magiclantern.fm/forum/index.php?board=25.0) area of the forum.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on December 20, 2016, 11:37:55 PM
10bit raw - video test /build 18dec 2016/

www.youtube.com/watch?v=XH3yVrSHVQQ

"magiclantern-raw_video_10bit_12bit.2016Dec18.600D102.zip"

:: EOS 600D ::
MLV_REC + MLV_SND, ISO 400, 1728x648, 23,98fps

Most of recorded videos are fine.
Some of them are bad decoded. Some of videos are without sound.

Original video file:
https://ulozto.cz/!JoQQdlZxLl9X/10bit-raw-video-test-build-18dec-2016-mp4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 21, 2016, 12:52:21 AM
5d3 113 mlv + sound 10, 12 & 14 bit seams to be working for me as well.

:)

Gonna try 123 later.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vstrglv on December 21, 2016, 10:07:59 AM
Quote from: MAG on December 20, 2016, 08:51:13 PM
If you want to experiment with crop_rec try merging branches--it really isn't that hard to compile and there are several tutorials in the General Development Discussion area of the forum.
It is not easy. There are many conflicts when merging Unified into crop_rec branch.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on December 21, 2016, 01:22:54 PM
ok, put 10 and 12bit into raw2cdng (https://www.magiclantern.fm/forum/index.php?topic=5618.1325). if someone s still using this tool :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on December 21, 2016, 02:35:08 PM
Quote from: chmee on December 21, 2016, 01:22:54 PM
ok, put 10 and 12bit into raw2cdng (https://www.magiclantern.fm/forum/index.php?topic=5618.1325). if someone s still using this tool :)


Of course, im still using your soft as mlv viewer and extractor. Thanks for your update.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on December 21, 2016, 08:19:19 PM
Where is the module for for 5d3 1.1.3?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ali Oliya on December 22, 2016, 08:37:44 AM
Why this 10bit/12bit build is not available for download in  nightly builds? (5d3 - 1.1.3)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on December 22, 2016, 08:55:53 AM
Because it is still in development. There is a "main repository" containing code considered to be tested/evaluated/peer reviewed. New features will tested by whoever dares to do so (= not afraid taking additional risks) in so called "branches". If (and only if) a dev is convinced code has been verified it will make it into "main" and then you will find it in the nightlies.

If you want to test this one (and taking all the responsibilities) you may use downloads compiled by members or build your own development environment. There are some quite good tutorials by dfort working perfectly for beginners like me.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on December 22, 2016, 09:13:57 AM
Test of 10bit RAW_REC with EOS 600D.
www.youtube.com/watch?v=WZ9unJonmOY
Build: "magiclantern-raw_video_10bit_12bit.2016Dec18.600D102.zip"
RAW_REC, ISO 400-800, 1728x972, shutter: 1/15s, 12fps (postprocess: RawTherapee, Virtualdub, Avisynth, Kdenlive)
Original video file: https://ulozto.cz/!SbnYaSK0n6gx/karvina-frystat-2016-avi

Danne: thank you for the advice ;-)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 09:47:19 AM
As seen some post before, eNnvi tried to make CONFIG_EDMAC_RAW_SLURP work on the 6d.
One thing is working, live view doesn't freeze anymore when recording 10/12 bit on the 6d  :D
However new problem now is, the recording gives scrambled/pink frames  :(

So we're halfway there...
Since live view works, the address for CONFIG_EDMAC_RAW_SLURP is probably right and looks like this in the 'raw.c' file:
#ifdef CONFIG_6D
// found at 0xFFCE513C (ram func)
#define DEFAULT_RAW_BUFFER MEM(0x76d6c + 0x2C)
#endif


Question is now, why do we end up with scrambled/pink frames. Are we not using a right free edmac channel in 'edmac-memcpy.c' ?
which is configured as follows:
#if defined(CONFIG_650D) || defined(CONFIG_700D) || defined(CONFIG_EOSM) || defined(CONFIG_6D)
uint32_t raw_write_chan = 0x12;
#endif


I'm not an expert in this, but I'm wondering why some camera addresses are written as 0xXX and for other camera's like the 5d3 for example as a plain normal number:

/* for other cameras, find a free channel with find_free_edmac_channels  */
#ifdef CONFIG_5D3
uint32_t raw_write_chan = 4;
#endif


I'm wondering if I could brick my camera when trying out a normal written number address for the 6d like:
#ifdef CONFIG_6D
uint32_t raw_write_chan = 8;
#endif


eNnvi mentions that the scrambled/pink frames problem could also be due to wrong edmac flag settings.
We tried both options for 6d, the first one gives pink frames, the second gives scrambled frames.
/* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
#if defined(CONFIG_650D) || defined(CONFIG_700D) || defined(CONFIG_EOSM) || defined(CONFIG_6D)
    uint32_t dmaFlags = 0x20000000;
#else
    uint32_t dmaFlags = 0x20001000;
#endif


So anybody got suggestions how to get this working, and is it safe to try out some different free edmac channels, or will it brick my camera when I pick the wrong one?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 22, 2016, 10:08:03 AM
It is safe to try it out. Both settings won't brick your camera. I would try out one after another not both at the same time. For finding free channels you can compile this (https://bitbucket.org/hudson/magic-lantern/branch/edmac). Check the latest commits. a1ex has rewritten find free edmac channels. So should be no problem for you to find other unused ones.

For dmaflags I recommend doing it like seen on unified branch:
https://bitbucket.org/hudson/magic-lantern/src/4f72a0eabb16c87ace571f494e588598b467c157/src/edmac-memcpy.c?at=unified&fileviewer=file-view-default#edmac-memcpy.c-401
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 22, 2016, 10:22:21 AM
Near LVx1_StartMem1Edmac: channel = 0x12, flags = 0x60000000.

The meaning of this flag is not really known, but this benchmark gives some insights:
5D3 edmac_memcpy (see also transfer size from Register Map (http://magiclantern.wikia.com/wiki/Register_Map#EDMAC)):

0x40001000 => 653 MB/s (16 bytes per transfer)
0x20001000 => 625 MB/s (8 bytes)
0x40000000 => 585 MB/s (4 bytes)
0x20000000 => 380 MB/s (2 bytes)
0x60000000 => 585 MB/s (same as 0x40000000?)
0x60001000 => 653 MB/s (same as 0x40001000?)


So, I'd expect it to work with both 0x60000000 and 0x40000000. Can you confirm?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 10:35:18 AM
Alex, are you asking me to try those values on the 6d ?

So instead of:
uint32_t dmaFlags = 0x20000000;
Try out:
uint32_t dmaFlags = 0x60000000; (or  uint32_t dmaFlags = 0x40000000;

@NikFreak
QuoteIt is safe to try it out. Both settings won't brick your camera.
That's all I need to know  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 22, 2016, 10:36:02 AM
Nice code insights. Whats the synonym for LVx1_StartMem1Edmac on 7D.203? Something around FF27E924 with dmaflag 0x40000000?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 22, 2016, 11:18:03 AM
StartPass_x1 CrawAddr : %lx / KindOfCraw : %d.

Old models appear to require some more changes though...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on December 22, 2016, 11:48:57 AM
18 dec 113 build first tests:

mlv 10 bit 39 fps - wrong black level
mlv 10 bit 24 fps - ok
raw rec 10 bit 39 fps - ok

crop mode (5x zoom) lv doesnt work

the last build whithin the crop_rec module doesnt have issues whith lv, but had another problems such as bad first frame and mlv_rec corrupt frames
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on December 22, 2016, 12:06:17 PM
Quote from: Surin Dmitriy on December 22, 2016, 11:48:57 AM
18 dec 113 build first tests:

mlv 10 bit 39 fps - wrong black level
mlv 10 bit 24 fps - ok
raw rec 10 bit 39 fps - ok

crop mode (5x zoom) lv doesnt work

the last build whithin the crop_rec module doesnt have issues whith lv, but had another problems such as bad first frame and mlv_rec corrupt frames

After restarting, removing the card, 5x crop mode lv start to work..?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2016, 07:48:20 PM
Quote from: LevasI'm not an expert in this, but I'm wondering why some camera addresses are written as 0xXX and for other camera's like the 5d3 for example as a plain normal number:

The address can be in either hex or decimal. Near the top of raw.c look for this line:

#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */


and change it to #define and you'll be able to try different addresses (in hex) from the Debug menu. Read the comments on this merged pull request for more information:

https://bitbucket.org/hudson/magic-lantern/pull-requests/780/increase-raw_debug_type-limit/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 07:59:16 PM
I wanted to do some testing, but can't get it to work.

I've downloaded the latest magic lantern 'raw_video_10bit_12bit' branch from https://bitbucket.org/hudson/magic-lantern
And I've got an altered 'raw.c' file and an altered 'edmac-memcpy.c' file from eNnvi, which I put in the latest version of raw_video_10_12bit branch, I overwrite the original raw.c and edmac-memcpy.c files.

Then I go to platforms and make a new build for the 6d.
But when I test this new build, live view freezes... (Although the recorded 10bit MLV files give good frames)
While eNnvi got me some builds where live view doesn't freeze ?
Am I doing something wrong here, as far as I understand the only two files altered for using RAW SLURP are the raw.c and the edmac-memcpy.c files ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2016, 08:53:46 PM
You also need internals.h for the 6D to define the slurp thing--away from computer at the moment.


Sent from my iPhone using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 09:13:57 PM
Thanks, that was the missing piece. Compiled a build with working live view myself.
Now it's time to do some testing  8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 09:22:22 PM
@Alex
I can conform that using
uint32_t dmaFlags = 0x40000000

gives the same results as
uint32_t dmaFlags = 0x20000000

pink garbage frames  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 09:46:49 PM
@dfort

I did the #define thing on RAW_DEBUG_TYPE
But I get this error with compiling
../../src/raw.c:2113:18: error: 'lv_raw_type' undeclared here (not in a function)
         .priv = &lv_raw_type,
                  ^
make: *** [raw.o] Error 1


Tried some different edmac channels, changing the edmac-memcpy.c, but so far, no luck.
Got one black live view and one really slow live view, like 5 fps or something.
Would be a lot quicker if you can define it in the debug menu.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on December 22, 2016, 10:32:52 PM
@Walter Schulz

I want to take de risk. I can test in the real world. I have a film production company and we work everyday with Canon (5D, 7D, 6D), Sony and Black Magic.

I don't have skills and time to compile. But time to test when I film (not trees and cats, of course ;)).

If you or other dev can share the compile, I will test.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2016, 10:35:43 PM
You need to give it a starting value. Any value will do because you can change it. Look at the 60D as an example.


Sent from my iPhone using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 22, 2016, 11:16:50 PM
Quote from: Levas on December 22, 2016, 09:22:22 PM
...
pink garbage frames  :P

Then try 0x60000000.

Quote from: a1ex on December 22, 2016, 11:18:03 AM
StartPass_x1 CrawAddr : %lx / KindOfCraw : %d.

Old models appear to require some more changes though...
Finding DEFAULT_RAW_BUFFER for 7D causes some headaches. Cannot find anything EVF related in the 7D ROM though other DIGIC IV like 60D/600D are no problem regarding that. can you help in finding DEFAULT_RAW_BUFFER for 7D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 22, 2016, 11:21:38 PM
Sure, reply #754. I already tried, without success.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 03:46:37 AM
@Levas did you get it to compile with "#define RAW_DEBUG_TYPE" ?

What I was trying to tell you on my cell phone is to look for this:

raw.c
#ifdef CONFIG_60D
#define PREFERRED_RAW_TYPE 5
#endif


And do the same for the 6D. Then go to the Debug menu to try different values:

(https://c3.staticflickr.com/1/766/31229380130_7893122693.jpg)

Start with 0x1 and go up from there. After a while you'll get repeating results. You might want to look at the playback in camera with mlv_play and also load raw_twk if you're using 10/12 bit options.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 08:27:17 AM
Thanks @Dfort
I already got the debug option working, but ran out off battery...

It's a new day and got a fully loaded battery 8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 11:34:47 AM
Ok, I tested a lot today and allmost  all I saw were frames that could be described as a pink mess  :P
There is one setting which gives me scrambled pink frames, so still a mess, but you can recognize some stuff in it.
Free edmac channel
uint32_t raw_write_chan = 0x12;
and dma flag
  uint32_t dmaFlags = 0x20000000;

But I also find some resolution settings furtheron in the raw.c file.
Could it be that the scrambled picture is caused by faulty settings in this part of raw.c

#ifdef CONFIG_EDMAC_RAW_SLURP

    /* params useful for hardcoding buffer sizes, according to video mode */
    int mv = is_movie_mode();
    int mv720 = mv && video_mode_resolution == 1;
    int mv1080 = mv && video_mode_resolution == 0;
    int mv640 = mv && video_mode_resolution == 2;
    int mv1080crop = mv && video_mode_resolution == 0 && video_mode_crop;
    int mv640crop = mv && video_mode_resolution == 2 && video_mode_crop;
   
    /* note: 6D reports 129 = 0x81 for zoom x1, and it behaves just like plain (unzoomed) LiveView) */
    int zoom = (lv_dispsize & 0xF) > 1;

    /* silence warnings; not all cameras have all these modes */
    (void)mv640; (void)mv720; (void)mv1080; (void)mv640; (void)mv1080crop; (void)mv640crop; (void)zoom;

    #ifdef CONFIG_5D3
    /* don't know how to get the resolution without relying on Canon's lv_save_raw */
    *width  = zoom ? 3744 : mv720 ? 2080 : 2080;
    *height = zoom ? 1380 : mv720 ?  692 : 1318;    /* height must be exact! copy it from Debug->EDMAC */
    return 1;
    #endif

    #ifdef CONFIG_60D
    *width  = zoom ? 2520 : mv640crop ? 920 : mv720 || mv640 ? 1888 : 1888;
    *height = zoom ? 1106 : mv640crop ? 624 : mv720 || mv640 ?  720 : 1182;
    return 1;
    #endif

    #ifdef CONFIG_600D
    *width  = zoom ? 2520 : mv1080crop ? 1952 : mv720  ? 1888 : 1888;
    *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
    return 1;
    #endif
   
    #if defined(CONFIG_650D) || defined(CONFIG_700D)
    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
    return 1;
    #endif

    #ifdef CONFIG_EOSM
    *width  = video_mode_crop ? 1872 : 1808;
    *height = video_mode_crop ? 1060 : 727;
    return 1;
    #endif

    #ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ?  634 : mv720 ?  720 : 1208;    /* find correct mv720 height -- must be exact! */   
    return 1;
    #endif
/ silent picture in 1080 mode = 1830 x 1224 /
/ silent picture in 720 mode = 1830 x 634 /
/ silent picture in zoom mode = 2688 x 960 /
/ image buffers 1080 mode (in debug menu) shows 720 x 480, 1808 x 1206 /
/ Some EDMAC channels show 3360 x 1207 or 3616 x 1205 /
/ image buffers 720 mode (in debug menu) shows 720 x 480, 1280 x 720 /
/ Some EDMAC channels show 2560 x 720 or 2560 x 719 or 3360 x 719 /
/ image buffers in 1080 zoom are 720x480, 1104x736 /
/ Some EDMAC channels show 2208 x 736 or 2208 x 735 or 4844 x 633 /
   
   #ifdef CONFIG_7D
    *width  = zoom ? 2520 : mv1080crop ? 1952 : mv720  ? 1888 : 1888;
    *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
    return 1;
    #endif

    /* unknown camera? */
    return 0;


For 6d I tried some different settings, but I'm not sure which resolutions to put where, so I added some text in the file which shows resolutions of silent pics and image buffers and edmac channels.
Does somebody have an idea which numbers to use ?

    #ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ?  634 : mv720 ?  720 : 1208;    /* find correct mv720 height -- must be exact! */   
    return 1;
    #endif
/ silent picture in 1080 mode = 1830 x 1224 /
/ silent picture in 720 mode = 1830 x 634 /
/ silent picture in zoom mode = 2688 x 960 /
/ image buffers 1080 mode (in debug menu) shows 720 x 480, 1808 x 1206 /
/ Some EDMAC channels show 3360 x 1207 or 3616 x 1205 /
/ image buffers 720 mode (in debug menu) shows 720 x 480, 1280 x 720 /
/ Some EDMAC channels show 2560 x 720 or 2560 x 719 or 3360 x 719 /
/ image buffers in 1080 zoom are 720x480, 1104x736 /
/ Some EDMAC channels show 2208 x 736 or 2208 x 735 or 4844 x 633 /
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 12:12:47 PM
So envvi guided you and gave you some code snippets? Where did you guys get


#ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ?  634 : mv720 ?  720 : 1208;    /* find correct mv720 height -- must be exact! */   
    return 1;
    #endif


from? You need to call lv_save_raw to read the values from "Show EDMAC". The value for width shown there must be undertaken a math. the formula is:

shown width * 8 / 14


You can replace this in debug.c


static void run_test()
{
call("lv_save_raw", 1);
}


and just call it afterwards from "Don't click me" for all video / zoom modes. the shown height in edmac needs to be added +1. For e.g. if it shows 1205 then it is 1206. Height values isn't alowed to be odd. It needs to be an even value.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 12:36:36 PM
Thanks Nikfreak, I'll try the debug.c and dont click me method.

eNnvi tried to make soms builds for the 6d.
Only they didn't work perfect.
I asked eNnvi for the raw.c and edmac-memcpy.c file so I could test some stuff.
eNnvi doesn't have a 6d, but seems to know somebody else with a 6d who also tested the builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 23, 2016, 12:54:56 PM
Sorry giusto been really busy at work these data, this evening i'll try to get the help of levas and find the right width and height buffer pointers.

I got those data looking at the 6d edmac while recordman and the width from the silent pic.

Now i realize that was not the best way to do that. Levas i'll text you via PM when i'll be back home
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 12:57:42 PM
probably the values are ok for mv720 and 1x zoom but definitely wrong height for the zoom mode. I would recheck all of them in all modes as explained above.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 01:36:33 PM
Original values in the file from eNnvi were
*width  = zoom ? 2768 : mv720 ? 1920 : 1920;
*height = zoom ?  736 : mv720 ?  720 : 1208;

Don't know if I did it right, cause changing the debug.c and pushing the don't click me didn't change the values in the show edmac screens  :-\
But these values I did found in one edmac channel:

zoom
4844 x 987 which translates too (2768 x 988)

1080mode
3360 x 1251  which translates too  (1920 x 1252)

720mode
3360 x 662  which translates too  (1920 x 662)

So my raw.c file is now like this (which still gives scrambled frames, but could be due other issues like free edmac channel etc.)
#ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ? 988 : mv720 ?  662 : 1252;    /* find correct mv720 height -- must be exact! */   
    return 1;
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 01:53:46 PM
 :o
I tried the latest resolution values with this dma flag in edmac-memcpy.c

uint32_t dmaFlags = 0x60000000

And looks like we have working live view and NORMAL FRAMES on 6d 
:D :D :D :D :D :D :D :D :D :D :D

I'll test some more, to be really sure it works  ;D

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 02:16:20 PM
Yep, 10 bit recording works flawlessly on the 6d, with live view :D

Although with 12 bit recording something weird is still going on (as mentioned before in this topic).
with 12 bit recording MLV_dump finds a lot of cold pixels and the image has specles all over it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 02:26:31 PM
congrats Levas. Was worth all your efforts and guess it feels great making yourself a xmas gift. Submitting a PR @bitbucket will make it perfect  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 23, 2016, 02:26:53 PM
Wow. Beautiful work on the 6D everyone and Levas :).

Maybe needless to say but are you disabling cold pixel and vertical stripes settings in mlv_dump?
http://www.magiclantern.fm/forum/index.php?topic=5601.msg176263#msg176263
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 02:40:54 PM
@Alex
uint32_t dmaFlags = 0x40000000
Works too, so before it didn't probably work because of the wrong resolution settings.

@Nikfreak
Submitting @bitbucket, what do you think I am, a developer  :P
And my coding skills aren't that good,in order to use the right dmaFlags my version, for sure, won't work for some other cams that worked before ;D

@Danne
You're right, disabling stripe fix and cold pixel fix gives good frames.
But I always thought the stripe fix was nessecarry  ?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 23, 2016, 02:46:17 PM
Could you also try with edmac flags of 700D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 03:33:31 PM
@eNnvi
I've tried the following dma flag as used for 700d
uint32_t dmaFlags = 0x20000000

And it causes the 'RAW DETECT ERROR' and when it will record it gives the scrambled frames.

So the only settings that were wrong in your build are the height value settings for the different video modes (width values were all good)
#ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ? 988 : mv720 ?  662 : 1252;    /* find correct mv720 height -- must be exact! */   
    return 1;
#endif

And the dma flag setting, which must be either
uint32_t dmaFlags = 0x40000000 or uint32_t dmaFlags = 0x60000000
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 03:46:51 PM
So, confirmed (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176849#msg176849)!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 03:51:02 PM
Yes confirmed, both values work
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 03:59:31 PM
Is somebody willing to do a pull request for this for the 6d?

I can send the pieces of code that needs to be altered/added in a pm.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 04:40:05 PM
Quote from: Levas on December 23, 2016, 03:59:31 PM
Is somebody willing to do a pull request for this for the 6d?

Sure, no problem. Let's get some developers reviewing the changes you and eNnvi made and post a test build for other 6D users.

@nikfreak -- do you have a branch in your repo that I can use to make test builds for the 70D and 100D?

Off topic -- Just a suggestion. It would be great to get those platforms merged and perhaps prune the branch for the 1100D if that platform is no longer supported.

[EDIT]
Back on topic -- so if CONFIG_EDMAC_RAW_SLURP was working on the 1100D at one time why was it removed? Seems like that camera could really use the 10bit/12bit treatment though at this point it doesn't work at all with raw video.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 06:36:28 PM
just updated the 100D PR.
70D PR should already contain all relevant info needed as far as I remember.

Wishing ya fun.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on December 23, 2016, 07:49:24 PM
You guys are amazing with the improvements upon the great foundations of Magic Lantern.
Thanks to all and congratulations on this new breakthrough!

I use the 1.23 firmware so finally I had a little time to test it with magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123.zip (great work dfort)
WOW! WOW! WOW!
I tested the mlv_rec and raw_rec with the raw_twk.  Great accomplishments here dfort, g3gg0, and all contributors!

Video quality:  I cannot see any difference at iso 6400 or at iso 100.  All of them look superb on 10/12/14 bits.
Video size: Amazing 40% lighter dng frames going from 14 bits to 10 bits.  Pixel peeping the dngs at 300% show no difference between 10/12/14 bit!
Color and Exposure are identical.  Bumping exposure by 3ev on Photoshop does not bring up any differences on shadows.

Standard Max Recording Resolutions 5D3 with 64 GB Lexar 1066:
Origninal mlv_rec + spanning + Auto preview + sound at 1920x1080 30fps same with crop module: 17 sec max recording
Original mlv_rec + Auto preview + sound at 1920x1080 30fps with crop module: 17 sec max recording
Original raw_rec + Auto preview at 1920x1080 30fps same with crop module: 23 sec max recording

10/12/14 BIT TESTING
With magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123
Note: I cannot find the card spaning on new mlv_rec

14 bit
mlv_rec + canon preview + sound at 1920x1080 30fps same with crop module: 16 seconds recording time
raw_rec + canon preview 23 sec at 1920x1080 30fps: 23 seconds recording time
raw_rec + canon preview 32 sec at 1920x1080 30fps at 5x croped: 35 seconds recording time!! dont know why the croped records faster.

12 bit
mlv_rec + Auto preview + sound at 1920x1080 30fps: continuous recording
mlv_rec + Auto preview + sound at 1920x1080 30fps at 5X cropped: continuous recording
Max continuous recording mlv_rec + Auto preview + sound at 1952x1098 or smaller at 30fps

raw_rec + canon preview 23 sec at 1920x1080 30fps: continuous recording
raw_rec + canon preview 32 sec at 1920x1080 30fps at 5x croped: continuous recording
Max continuous recording raw_rec Continuous recording with 2000x1126 30fps or smaller resolutions.

note: mlv_play crashes (play quits and camera has some error since the red led keeps blinking) on crop video larger than 1968x1108

10bit
Max continuous recording mlv_rec + Auto preview + sound at 2144x1206 or smaller resolutions at 30fps.
Max continuous recording raw_rec Continuous recording with 2208x1242 or smaller resolutions at 30fps.

Although I do not contribute to the forum due to time constrains I do come here all the time and look for all the updates on 5D3 1.23.  I use my 5D3 often and take lots of bird photos and RAW video.
I truly enjoy all of the amazing things you guys accomplish.  It is stunning!
THANK YOU!  THANK YOU!  THANK YOU!
Merry Christmas and happy new year to all!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 23, 2016, 07:55:26 PM
pull request for 6D done.

now i'll try to work in 7D (i like the hard things ;) ). As i don't own one i have to ask if anybody want to help me (possibly having that camera :D )
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 08:07:17 PM
Looks like @a1ex already merged the 6D pull request, that was fast.

There's also another change I didn't see coming. The 700D now has "#define PREFERRED_RAW_TYPE 0x10" while the 650D still has "#define PREFERRED_RAW_TYPE 78" commented out. They both used to be commented out. What's the reason for that?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 23, 2016, 08:11:56 PM
Quote from: dfort on December 23, 2016, 04:40:05 PM
Back on topic -- so if CONFIG_EDMAC_RAW_SLURP was working on the 1100D at one time why was it removed?

It wasn't removed. There are some missing bits from the raw branch (such as skip offsets), so the branch wasn't merged. Start reading from here (http://www.magiclantern.fm/forum/index.php?topic=1009.msg119468#msg119468); some of those bits are already there and just need to be put together in a commit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 08:45:11 PM
Thanks for the tips. Danne has access to an 1100D so we might team up and work on it, though it isn't a high priority for either of us.

Quote from: a1ex on June 22, 2014, 06:09:14 PMJust noticed the 1100D build was broken for about one week, and nobody reported the error.
Is there anybody still using this camera, or should I discontinue this port?

So it wasn't intentional?@dmilligan also has one but it looks like he isn't using it much these days - and there's a math problem:

Quote from: dmilligan on December 19, 2016, 02:41:02 AM
The 1100D nightly build was intentionally disabled because a1ex noticed some incorrect values for FPS override timers. I've never cared enough to try and fix it (the Maths involved hurt my head :P ), I just compile it with the old incorrect values.

Getting the 1100D working isn't my goal, I'm just trying to get a better understanding of ML and looking at CONFIG_EDMAC_RAW_SLURP on that camera opened this can of worms. In any case, years ago I did some beta testing and the company sent me their lowest end system. Why not their most powerful? Because they wanted to make sure their minimal configuration was still valid or if they should discontinue support for it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 08:59:54 PM
6D added to the test builds:

https://bitbucket.org/daniel_fort/magic-lantern/downloads

Make sure to load the raw_twk module if you want in camera playback of 10/12 bit files.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 24, 2016, 12:40:39 AM
Quote from: nikfreak on December 23, 2016, 06:36:28 PM
just updated the 100D PR.
70D PR should already contain all relevant info needed as far as I remember.

Thanks. 100D was pretty much ready. Merged the 70D_merge_fw112 branch with raw_video_10bit_12bit.

Test builds for 100D and 70D posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on December 24, 2016, 01:20:47 AM
Quote from: dfort on December 23, 2016, 08:45:11 PM
So it wasn't intentional?
1100D has been broken multiple times in the past, sometimes on purpose, sometimes on accident. It is currently broken on purpose. The 1100D was probably the easiest platform to break on accident because of its unusual combination of CONFIG options enabled (so not being careful about #ifdefs would be more likely to result in code that won't compile without being noticed, for example 1100D could do QR raw but not LV raw, but there's a lot of shared code between the two).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on December 24, 2016, 02:19:27 AM
[quote Merged the 70D_merge_fw112 branch with raw_video_10bit_12bit.

Test builds for 100D and 70D posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
[/quote]

i wil be able to test 70D tonight after reaching home. Thank q dfort. I have been waiting for so long
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 24, 2016, 09:04:52 AM
70D tried a quick test.

1832@ 2.35 apsect, 24fps 10bit, could record for over 1 minute (probably continuous), the movie review but it never builds the movie because it hangs/bricks requiring battery pull and two resets to get right.
1832@ 2.35 aspect, 30fps 10bit, could record for 18s, but same hang/brick.
1832@ 2.35 aspect, 24fps 12bit, could record for 28s, but same hang/brick. 
1846@ 2.35 aspect, 60fps 12bit, 3X mode on, could record for 9s, but same hang/brick.
1920@ 2.35 aspect, 24 fps 10bit, 5X zoom, could record for over 1 minute (probably continuous), two second in got a "hack error" but it kept on recording until I hit stop rec.,  but same hang/brick occurs. Not sure with hack error to trust the record time.
2220 @2.35 aspect, 24 fps 10bit, 5Xzoom,  hack error at 0 sec kept recording over 1 minute, but when I switched to photo mode it showed 51 sec record, so I cant trust either number.

It did produce MLV files, but Mystic could not convert any MLV to DNG.  This same program worked with my 600D 10/12 bit MLV files.
Will try mlv_dump on the files once I have more time. 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on December 24, 2016, 09:54:03 AM
Live view still freezes with 100d. I just tested it. I downloaded the @dfort build.  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 24, 2016, 11:40:12 AM
dfort probably didn't enable edmac_raw_slurp before compiling for 100d. Just a guess as I didn't try myself.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 24, 2016, 01:19:51 PM
@dfort and @reddeercity for your help to make 5D Mark II even better, which build is the best so far Dec1, Dec12, or Dec18  I never got Dec9? I had some corruption with Dec18 but I'm going to do some more test, also will there be a update build from @dfort for 5DMarkIi soon?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 24, 2016, 08:09:55 PM
Quote from: nikfreak on December 24, 2016, 11:40:12 AM
dfort probably didn't enable edmac_raw_slurp before compiling for 100d. Just a guess as I didn't try myself.

@nikfreak - Could you make a 10bit_12bit branch in your repository that includes the 70D and 100D? I don't have these cameras available for testing so I can't be sure that I put all the pieces in the right place. Just a thought--maybe do your 70D and 100D pull requests on the raw_video_10bit_12bit branch? Seems that this is where all the action is happening and since this is a development branch maybe the developers with the merge key are more receptive to adding new platforms here as opposed to the "stable" unified branch?

@dariSSight - The 5D2.212 won't be much better with 10bit/12bit until CONFIG_EDMAC_RAW_SLURP is working on that camera. Seems to be a matter of finding a free edmac channel and sniffing out where the DEFAULT_RAW_BUFFER is located -- it is different on every camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 24, 2016, 08:49:20 PM
Looks like there's lots of requests for crop_rec with the 5D3.113. Since that platform seems to be well tested with 10bit/12bit I put up a test build that includes crop_rec:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit_crop_rec.2016Dec24.5D3113.zip

The 5D3.123 test build also includes the crop_rec module.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ewinemiller on December 24, 2016, 09:56:13 PM
Quote from: dfort on December 24, 2016, 08:49:20 PM
The 5D3.123 test build also includes the crop_rec module.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123.zip

Hey dfort.

I appreciate the builds, but if I try your 5D3 1.23 build from the 20th, it won't load the modules. I get failed to link modules error.

Regards.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Camerite on December 24, 2016, 10:30:47 PM
The 5D3 123 build seems to load all modules but crop rec. When activated the camera gives errors on all modules.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kamranjon on December 24, 2016, 10:59:41 PM
More than happy to help provide testing on the 7D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 24, 2016, 11:27:20 PM
Quote from: ewinemiller on December 24, 2016, 09:56:13 PM
I appreciate the builds, but if I try your 5D3 1.23 build from the 20th, it won't load the modules. I get failed to link modules error.

Strange, check reply #862 from @RenatoPhoto

Quote from: kamranjon on December 24, 2016, 10:59:41 PM
More than happy to help provide testing on the 7D.

There is a test build posted but like the 5D2 it needs the CONFIG_EDMAC_RAW_SLURP treatment for it to work properly.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vstrglv on December 25, 2016, 08:24:33 AM
Quote from: dfort on December 24, 2016, 08:49:20 PM
Looks like there's lots of requests for crop_rec with the 5D3.113. Since that platform seems to be well tested with 10bit/12bit I put up a test build that includes crop_rec:
https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit_crop_rec.2016Dec24.5D3113.zip
Thank you very much! Quick test - frame issue:
10bit,12bit, crop1:1 and not crop mode - preview framing during recording  is not correct. All screen is filled with image (there are no horizontal black stripes).
Sometimes this issue takes place in 14bit mode too. I could not find a regularity.   
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on December 25, 2016, 10:57:01 AM
tried dfort's test build for 70D but no luck
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ewinemiller on December 25, 2016, 03:54:16 PM
Quote from: dfort on December 24, 2016, 11:27:20 PM
Strange, check reply #862 from @RenatoPhoto

It's crop_rec.mo. If I enable it, no modules load. I don't think he tested that. Looks like he was using the classic crop stuff.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on December 25, 2016, 04:43:57 PM
Guys!  I did not use crop.rec!!!
@dfort

I used the standard crop mode using the magnify button..  So it works on crop mode.
crop.rec is and advanced version of crop mode that centers the images and has other nice features.
It is very good but is not incorporated into unified.. it is a separate branch.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vstrglv on December 25, 2016, 04:54:24 PM
Quote from: vstrglv on December 25, 2016, 08:24:33 AM
Thank you very much! Quick test - frame issue:
10bit,12bit, crop1:1 and not crop mode - preview framing during recording  is not correct. All screen is filled with image (there are no horizontal black stripes).
Sometimes this issue takes place in 14bit mode too.
This issue takes place only for mvl_rec mode. If preview options is set to "ML Grayscale"  this issue disappears and  mode is colour during  recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on December 25, 2016, 05:55:15 PM
@dfort
Also forgot to mention that at the start/stop of recording 5D3-123 with mlv_rec moldule I get this message:
Hack error at 0:
excpectede3a00032, got e1a00000
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 26, 2016, 09:10:21 PM
@g3gg0 updated mlv_dump and put up binaries for Windows, Mac and Linux so I removed the mlv_dump builds I was hosting. These new builds work with 10bit/12bit files.
http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip

Some of you are posting links to the test builds which is fine but just a warning that I've been removing old builds to avoid confusion. That means that you'll need to save previous versions in case something breaks or you want to make a regression test.

There have been some reports that things aren't working properly on a few cameras that should be working. It might be the way I'm compiling. In most cases I'm just compiling the raw_video_10bit_12bit branch but in others I merged crop_rec and other branches. Maybe I should be merging unified too since eventually the goal is to get this into the nightly builds?

Here are the latest download counts of the test builds. Checkmarks are for cameras that are working with CONFIG_EDMAC_RAW_SLURP:



















5D3.123-76* with crop_rec (Dec 20)
5D3.113-55* without crop_rec (Dec 18)
5D3.113-51* with crop_rec (Dec 24)
6D.116-24* new build works with 10bit/12bit (Dec 23)
5D2.212-21
7D.203-15
EOSM.202-14
100D.101-14* might have build issues? (Dec 23)
600D.102-13
70D.112-13* might have build issues? (Dec 23)
700D.114-12
60D.111-10
650D.104-10
1100D.105-9* allocate_raw_buffer branch has CONFIG_EDMAC_RAW_SLURP working but not 10bit/12bit (Dec 19)
550D.109-8
50D.109-6
500D.111-3

Although this might not be an accurate list of cameras sorted by popularity, it does show quite a bit of interest in 10bit/12bit raw video from users of the higher end and full frame cameras. I was surprised by the amount of interest in the 5D3.123--looks like users want raw video and clean HDMI output.

There a few older models left to solve. Check out reply #754 for some hints:

Quote from: a1ex on December 16, 2016, 09:41:16 PMFrom what I've tried so far on those old models (without much success yet), we have at least two problems:
- the raw buffer is not allocated all the time (as it's done on the new models) -> easy to fix, see the 1100D raw branch
- the vsync hook currently used doesn't appear to be the best for getting a raw image
- the exact behavior of EDMAC in case of input/output dimension mismatch is not very well known

Solving those models may require either some serious low-level research, or a great dose of luck. I'm just scratching the surface here (http://www.magiclantern.fm/forum/index.php?topic=18315).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lars Steenhoff on December 27, 2016, 12:53:14 AM
for monitoring with an external monitor clean hdmi is quite nice.

and as a bonus the zoom mode wich shows in black and white with the 5x crop mode is showing a preview on the external monitor in color. ( last time i checked wich is a while ago)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Quentin on December 27, 2016, 08:23:16 AM
How fast in terms of FPS/Frame size is 12/10 bit RAW recording ?
I tried to follow the thread but its since 2013 and was inmature.
Any clues ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 27, 2016, 08:39:01 AM
You can use the usual formula to check if it fits in your camera controller's speed limit:
W * H * BD * FPS / 8 /1024 / 1024
(W &  H — frame width and height in pixels, BD — bitdepth, which now can be 10 or 12 instead of 14, FPS is frames per second, and converting to megabytes per second). Result is required speed in MB/s

For example, for 650D we have a speed limit in 40MB/s.
We wish to film 10 bit raw video 1728x736 at 25fps.
1728*736*10*25/8/1024/1024 = 37.9 MB/s which fits fine in controller's limit. So in practice we can shoot continuously with these parameters.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Quentin on December 27, 2016, 08:42:59 AM
Thank you very much  :)
Its reasonable. Cant wait to try it out.
ML is so cool
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 28, 2016, 04:12:34 PM
On my 70D MLV files tried the latest mlv_dump, all were failed.  This agrees with mystic not being able to convert any of the same MLV.

G:\DCIM\100CANON>mlv_dump --dng M23-2332.MLV

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M23-2332.MLV'
   - Convert to DNG frames
   - Output into 'M23-2332_'
File M23-2332.MLV opened
File M23-2332.M00 not existing.
Processing...
[ERROR] Invalid block size at position 0x00000400
Processed 0 video frames
Done

my 600D MLV works fine with same program.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 28, 2016, 04:36:28 PM
@nikfreak -- Need your help with the 70D. Could you create a 10bit/12bit branch in your repository with the appropriate merges? I got it to compile but I must have missed something. Don't have a 70D here to check things out. It would be great if you could do the same for the 100D.

Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: maxotics on December 28, 2016, 04:54:39 PM
I'm happy to test the 12/10 bit on my 7D.  But would need instructions.  I can't make complete sense of it reading threads.  Again, only if you need someone to test.  I can wait ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 28, 2016, 05:53:00 PM
600D, 1280 @2.35, 24 fps, 10 bit. 
I am getting pink frames (at top 25%) every other frame.
What's this about?
By bit depth & rate calculation it should be continuous.
My card is Sandisk Ex. Pro, 95 Mb/s so its not the card speed.


Reformat card, loaded latest 600D, all is good. Was able to get [email protected]/24fps  for about 2 seconds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: whyfy on December 28, 2016, 06:49:16 PM
Canon 60D 10Bit RAW video, Overexposure testing https://youtu.be/BlmajDezBB4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 29, 2016, 08:53:51 AM
Quote from: maxotics on December 28, 2016, 04:54:39 PM
I'm happy to test the 12/10 bit on my 7D.

This isn't working on the 7D yet. A few people are looking into it. The hard part seems to be finding the memory location for DEFAULT_RAW_BUFFER.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 29, 2016, 09:44:23 AM
maxotics check out 10 bit raw on your eosm. I saw your vids before on youtube and vimeo. and its one of the main reasons i even got an eosm. theres not much info out there..you had some of the only raw vids available to watch. 10 bit raw on eosm is really great. give it a try.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 29, 2016, 09:51:02 AM
@quentin check this out http://rbrune.github.io/mlraw/ (http://rbrune.github.io/mlraw/) noticed it has settings for 10 bit and 12 bit now:)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 29, 2016, 01:27:49 PM
Ones again. Tremendous work devs! Really appreciate the development.

If I could wish for the coming build, since I can't program, or have any code skills (I only make hardware and such mods). When shooting MLV instead of RAW I get these problems:


I know I can't ask any of this to be done but it would be swell!

Best,
David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: maxotics on December 29, 2016, 03:26:38 PM
Thanks Teamsleepkid, was wondering about that.  Will try it on the EOSM.  It always warms my heart when someone says, "Hey Max, you turned me onto Crack, thanks!" :)  Is there an instruction page anywhere, hard for me to piece together from threads.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on December 29, 2016, 04:46:09 PM
I just tried dfort's 12/18 7D build and 10 and 12 bit are working fine (but with grayscale preview frozen if used). What is not supposed to be working?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: snipeua on December 29, 2016, 07:28:33 PM
Dfort please make a new build at 1.13, the same as you did at 1.23! Thank you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 29, 2016, 08:13:27 PM
Max there's a download a few pages back like on page 35 it's dforts build. Hope I'm not stepping on his toes:) it's a ml folder you've done it before.. install magic lantern. Then in the settings under raw it says 10 bit 12 bit. I use 10 bit because it's higher resolution. For post I think you have to use mlvfs or mlv dump? I have no idea about other software I use mlvfs works great. Might be more addicting than crack:)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 29, 2016, 10:36:28 PM
Quote from: snipeua on December 29, 2016, 07:28:33 PM
Dfort please make a new build at 1.13, the same as you did at 1.23! Thank you!

What's the problem with this build? It should have the same features as the 5D3.123 build I recently posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit_crop_rec.2016Dec24.5D3113.zip

Quote from: Teamsleepkid on December 29, 2016, 08:13:27 PM
...it's dforts build. Hope I'm not stepping on his toes:)...

No worries. We're all sharing our experiences.

Quote from: Frank7D on December 29, 2016, 04:46:09 PM
...What is not supposed to be working? ...grayscale preview frozen if used...

Just rearranged your comment to answer your own question. Ok, there's more to it than that. Apparently 5D2 and 7D users are reporting some success with 10bit/12bit even though all the pieces aren't quite finished yet.

Quote from: maxotics on December 29, 2016, 03:26:38 PM
Thanks Teamsleepkid, was wondering about that.  Will try it on the EOSM.  It always warms my heart when someone says, "Hey Max, you turned me onto Crack, thanks!" :)  Is there an instruction page anywhere, hard for me to piece together from threads.

@maxotics -- I'm another one who saw your raw video tests on the EOSM and got me interested in Magic Lantern. To try out 10bit/12bit from the test builds I posted turn on the mlv_rec and/or raw_rec modules. raw_rec is actually MLV Lite in this branch so you'll end up with MLV files instead of the RAW files you were used to when you were doing your tests. For processing the 10bit/12bit MLV files use either MLVFS (http://www.magiclantern.fm/forum/index.php?topic=13152.0) or g3gg0's new mlv_dump builds (http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip). If you want to record sound with mlv_rec you can turn on mlv_snd but note that you still can't record audio with raw_rec. You can also playback in camera by loading mlv_play but for 10bit/12bit playback you'll also need to load raw_twk. Hope this helps.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 29, 2016, 11:03:53 PM
Audio won,t work with raw_twk module on the eos m. You cannot delete video other than from file manager with raw_twk enabled. Other than that Eos M is one hell of a little beast with 10bit enabled. I use it a lot nowadays.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domasa on December 30, 2016, 12:06:31 AM
QuoteW * H * BD * FPS / 8 /1024 / 1024
(W &  H — frame width and height in pixels, BD — bitdepth, which now can be 10 or 12 instead of 14, FPS is frames per second, and converting to megabytes per second)

Is it right? Where is BD for second and third color? (R-G-B)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jpegmasterjesse on December 30, 2016, 12:18:17 AM
I'm having some success with my 5d2 and the December 19th builds.

MLV 10 BIT in Crop Mode is working perfectly minus frozen preview.
GIF: https://gfycat.com/ReflectingMadCod

However standard recording (not in crop mode) every other frame contains part of the 1st frame.
GIF: https://gfycat.com/ThoseObeseAllosaurus
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on December 30, 2016, 12:41:07 AM
So on 7D I can get the Canon preview to work if I have preview set to "auto" and before I press record I cycle through the various displays using the "info" key and end on Canon preview. This is while using the 5x crop mode (not sure about non-crop yet).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 01:14:37 AM
Hello everyone,

     I tested 10 bit Raw Video recording on a Canon T3i. I was able to record 1280 x 544 for 25 minutes on a 32 GB Sandisk Extreme Pro 95 MB/SEC SD Card.

     One pink frame was recorded on the 6th DNG, and there are pink, green, blue tints, on the highlights of a fan I recorded.

     However, they aren't on every dng frame, it seems every other.

     I exported an H.264 version of the video and included 6 dngs along with MLVFS settings.

     Here is a Google Drive link to these files: https://drive.google.com/drive/folders/0Bz5n0qYxIkVON0g1WUVVMDM1NWc?usp=sharing (https://drive.google.com/drive/folders/0Bz5n0qYxIkVON0g1WUVVMDM1NWc?usp=sharing)

     MLRawviewer 1.4.3 will not play these files, they look like white static.

     Also, using MLVFS, I'm getting Embedded when trying to use Cinelog-C profile in Adobe Camera Raw. I will make sure to post to the MLVFS Forum, but I just wanted to mention this for 10 bit raw recording. This might be due to the camera model though.

      Sound worked fine for this video, I just have it muted for privacy reasons (was having a conversation with someone while this was recording).

      This was from DFort's December 18th Build, found here: https://bitbucket.org/daniel_fort/magic-lantern/downloads (https://bitbucket.org/daniel_fort/magic-lantern/downloads)

      The settings I had for recording this video: 1280 x 544 Resolution, 2.35:1 Aspect Ratio, 10bpp, Global Draw Allow, Preview Options Auto, and the rest of the settings default. Override Audio settings on with 30 db Mic Boost. MLV Sound On. MLV_PLAY, MLV_REC, MLV_SND, AND RAW_TWK were the modules enabled.

     I was using a Lanparte LPE8 Battery Brick with LPE8 Dummy Battery to help avoid overheating.

      I hope this helps in some way!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 30, 2016, 02:29:36 AM
@PaulHarwood856

MLRawviewer won't work with 10bit/12bit MLV's.

Why do you have Bad Pixel Fix set to Aggressive?

One corrupted frame on a 25 minute recording is pretty good. Of course it would be better if there were none.

The color highlights on the fan are a problem. Would you be willing to try a debug build to see if we can fix that?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 30, 2016, 02:38:13 AM
@Paulharwood856.
Also try this version of mlv_dump (ml-dng branch) since it will work 10/12/14 bit files as well as provide correct white balance, camera model id tags(Not using localized camera model tags which can,t be used with acr dcp files)
mlv_dump
https://bitbucket.org/Dannephoto/ml-dng-dannephoto/downloads/mlv_dump

Code here ml-dng (dmilligan) reworked upon by bouncyball, maintained and updated with g3gg0 latest mlv_dump changes which I recently added.
https://bitbucket.org/Dannephoto/ml-dng-dannephoto
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 02:51:52 AM
Hey dfort,

     I found Bad Pixel Fix helps with the 7D, maybe I'm wrong? Maybe not for the T3i? Yeah I can try a debug build, that would be great. Can you guide me into what needs to be done? I haven't done this before.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 02:54:44 AM
Hey Danne,

     Ok great, I will try this version of mlv_dump, thanks! I'll make sure to check this page you linked to see updates and changes. Awesome!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 30, 2016, 02:56:36 AM
Also try without bad pixel fix and see if that get rid of the strange highligts
QuoteThe color highlights on the fan are a problem.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 30, 2016, 03:27:13 AM
Quote from: PaulHarwood856 on December 30, 2016, 02:51:52 AM
Yeah I can try a debug build, that would be great. Can you guide me into what needs to be done? I haven't done this before.

Ok, here's the debug build:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_RAW_DEBUG_TYPE.2016Dec29.600D102.zip

It might work right away. The issue is that several cameras need to have the PREFERRED_RAW_TYPE defined, the 600D might be one of them. I started with a value of 0x10 which might do the trick but you can try other values. Go to the Debug menu, select the first item and you'll be able to change it.

(https://c3.staticflickr.com/1/766/31229380130_7893122693.jpg)

Here's a more detailed explanation of what is going on:

https://bitbucket.org/hudson/magic-lantern/pull-requests/780/increase-raw_debug_type-limit/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: maxotics on December 30, 2016, 04:20:57 AM
I was able to do various  10-bit videos on the EOS-M.  1600x900 is amazing!  Files play fine in latest MLVProducer.   Amazing!  dfort you are DA MAN :)   I see builds for the 7D and it sounds like others have got it to work.  Should I stick with the EOS-M or should I, can I, try on 7D? 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 04:21:54 AM
Hey Danne,

     I just tried without Bad Pixel Fix, and the strange highlights were still there, but much less pronounced.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 04:32:08 AM
Hey dfort,

     Ok great, I will try this debug build out! Thanks.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 04:34:41 AM
Hey maxotics,

     If you get any success with the 7D, please post here. I haven't been able to get the settings right. Thanks.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 05:51:42 AM
QuoteHello everyone,

     I tested 10 bit Raw Video recording on a Canon T3i. I was able to record 1280 x 544 for 25 minutes on a 32 GB Sandisk Extreme Pro 95 MB/SEC SD Card.

     One pink frame was recorded on the 6th DNG, and there are pink, green, blue tints, on the highlights of a fan I recorded.

     However, they aren't on every dng frame, it seems every other.

     I exported an H.264 version of the video and included 6 dngs along with MLVFS settings.

     Here is a Google Drive link to these files: https://drive.google.com/drive/folders/0Bz5n0qYxIkVON0g1WUVVMDM1NWc?usp=sharing

     MLRawviewer 1.4.3 will not play these files, they look like white static.

     Also, using MLVFS, I'm getting Embedded when trying to use Cinelog-C profile in Adobe Camera Raw. I will make sure to post to the MLVFS Forum, but I just wanted to mention this for 10 bit raw recording. This might be due to the camera model though.

      Sound worked fine for this video, I just have it muted for privacy reasons (was having a conversation with someone while this was recording).

      This was from DFort's December 18th Build, found here: https://bitbucket.org/daniel_fort/magic-lantern/downloads

      The settings I had for recording this video: 1280 x 544 Resolution, 2.35:1 Aspect Ratio, 10bpp, Global Draw Allow, Preview Options Auto, and the rest of the settings default. Override Audio settings on with 30 db Mic Boost. MLV Sound On. MLV_PLAY, MLV_REC, MLV_SND, AND RAW_TWK were the modules enabled.

     I was using a Lanparte LPE8 Battery Brick with LPE8 Dummy Battery to help avoid overheating.

      I hope this helps in some way!

- Paul Harwood
Modify message

Although MLRawviewer will not play 10 bit or 12 bit .MLV files, I was able to install MLVProducer via Wineskin Winery for Mac. This plays 10 bit, 12 bit, and 14 bit files, nice!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 30, 2016, 06:49:36 AM
Quote from: domasa on December 30, 2016, 12:06:31 AM
Is it right? Where is BD for second and third color? (R-G-B)
This is right. You forgot the Bayer pattern. There are only WxH pixels on the matrix. All three colors are among them.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 30, 2016, 07:35:39 AM
Quote from: maxotics on December 30, 2016, 04:20:57 AM
Should I stick with the EOS-M or should I, can I, try on 7D? 

The EOSM should be working fine. From the reports testers have been posting it looks like it works on the 7D in crop mode which is sort of weird because CONFIG_EDMAC_RAW_SLURP is not defined on that camera.

Of course you're welcome to try it yourself and post your results.

BTW--I noticed some differences in the resolutions displayed between mlv_rec and raw_rec (MLV Lite) on the EOSM:

mlv_rec
(https://c7.staticflickr.com/1/390/31134400734_edc3d9586b.jpg)

raw_rec
(https://c3.staticflickr.com/1/743/31826501562_1566348afb.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 30, 2016, 09:33:59 AM
Quote from: dfort on December 30, 2016, 07:35:39 AM
BTW--I noticed some differences in the resolutions displayed between mlv_rec and raw_rec (MLV Lite) on the EOSM:
That's because only bit reduction (10-12) was implemented and not a1ex's fine resolutions code , so basically it's the same old MLV+audio
with bit reductions . Some thing on my 5D2

Quote from: dfort on December 30, 2016, 07:35:39 AM
looks like it works on the 7D in crop mode which is sort of weird because CONFIG_EDMAC_RAW_SLURP is not defined on that camera.
Don't forget the 5D2 work fine in mlv2.0 crop mode too  ;D  flawlessly
in fact I just finished a recording 8:00 green screen shot @ 2144x1076 10bit @23.976 + audio 16 bit 48khz
cool thing was 8 min at over 2K = only 30GB when extracted I had over 49GB each frame was 4.5MB @16bit Cdng (11,190 frame) and even had my HDMI Zacuto Evf  connected
All frames clear of any corruption  8)
I thing I notice was normally I have a few hot pixel in crop mode but noticed there was only one in 10bit mode , so does bit reductions reduce the load on the sensor ?
or just lucky .   

Edit: I'll post the results in a day or two
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on December 30, 2016, 11:51:22 AM
Hi,

I'm new to this forum, I 've read all this thread about 10/12 bit raw and many other threads on raw, so I apologise if my question is too simple.
First above all, I want to thanks the magic lantern team  :)  ! and reddeercity for his 5D2 support with tutorials all along so many years ...
Now my question : with my 5D2 and Dec18.5D2212 software, I can't get "crop" or "not crop" without Liveview freezing.
So reddeercity how can you film 8 min ? with hdmi output ? with "magic" settings ?

Let me know, please.

David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on December 30, 2016, 11:54:01 AM
Testing of 10bit-raw video /EOS 600D/ at very low light conditions (ISO 1600-6400)
www.youtube.com/watch?v=FVdyuj1OHJg
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 30, 2016, 09:05:13 PM
@reddeercity which build did you use for flawless 10bit 8min recording?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on December 31, 2016, 01:09:54 AM
@reddeercity - yes, link please.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on January 01, 2017, 12:41:20 AM
@paulharwood

From my experience with ML raw, hot pixels increase with higher ISO and record time.
Try ISO 100 or 400 max, and limit the record time per clip.  How long did you record with T3i before hot pixel appeared?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 01, 2017, 05:56:55 PM
Hey ShootMeAlready,

     This was a test in case of low light conditions. So I had the ISO at 3200. I'll make sure to do the same test at ISO 100 or 400 to see if there are no hot pixels. Thanks a bunch!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 01, 2017, 05:59:32 PM
QuoteOk, here's the debug build:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_RAW_DEBUG_TYPE.2016Dec29.600D102.zip

It might work right away. The issue is that several cameras need to have the PREFERRED_RAW_TYPE defined, the 600D might be one of them. I started with a value of 0x10 which might do the trick but you can try other values. Go to the Debug menu, select the first item and you'll be able to change it.



Here's a more detailed explanation of what is going on:

https://bitbucket.org/hudson/magic-lantern/pull-requests/780/increase-raw_debug_type-limit/diff

Hey dfort,

     So far I've tested 0x10 in the debug build, and got hot pixels, but no highlight issues with the fan in the shot. I'll keep testing, but just wanted to give you an update.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 01, 2017, 06:34:02 PM
Quote from: a1ex on December 12, 2016, 09:15:13 PM
From this analysis (http://www.magiclantern.fm/forum/index.php?topic=18393.0), PREFERRED_RAW_TYPE 0x10 appears to be the best choice for all DIGIC 5 and 6 cameras, and likely 0x5 for all DIGIC 4.

Can you confirm the above value? (600D, and also other cameras where PREFERRED_RAW_TYPE is not yet defined)

You should get:
- no bad pixels
- no changes from digital ISO (e.g. ISO 800 should be identical to 640 on the raw histogram)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 01, 2017, 06:52:02 PM
Hey Alex,

     Here is a link a dng from the 0x10 Debug. I saw the hot pixels in MLV Producer, but in this dng generated from cr2hdr.app I am only seeing a little artifacting in the candle and pink highlights in the fan. Also, is getting Err 70 normal when trying to playback raw files in a debug build?

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 01, 2017, 07:14:49 PM
0x10 is not relevant for 600D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 01, 2017, 10:44:51 PM
Oops -- @PaulHarwood856 could you go into the Debug menu, change it to 0x5 and run another test please?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on January 01, 2017, 11:47:26 PM
As it is now well known, the 5D2 and most other cameras don't work in 10/12 bit, and definitely won't until CONFIG_EDMAC_RAW_SLURP is enabled, so I have decided I would like to help fix it on the 5D2. All I can do right now is compile Magic Lantern(with my own idiotic changes), and I've successfully ran the ROM disassembly script, so I don't really have much of a clue how to do anything, but I do have too much free time(sometimes), so is this something I could do with limited ability but lots of free time? Most complex thing I've ever done with C is getting two Arduinos transmitting numbers at each other at slightly over 1 Mbit/hour(yeah that slow :D) with analog 433mhz antennas, so I don't know if my coding skills are enough.

Basically all I ask is, might I have enough ability to get it done or help out? and if so, where to start?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 02, 2017, 12:13:25 AM
@ch_d  & dariSSight sorry for the delay , so the build I'm using is from dec.01/2016.
But I don't think that really matters , as mlv 2.0+audio code has not changed other then
implementing bit reduction code , so every thing else should be the same , where as with raw_rec (mlv Lite)
code has gone under I lot of work to get the limited amount of cam to work at this point.
So I'm betting that any of the MLV2.0 builds with bit reduction should work find ,
just in case there don't work for you guy's here the one I used  10bit_12bit_raw_2016Dec01.5D2212 (https://www.dropbox.com/s/oodspk5jits334u/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip?dl=0).
This build came from dfort so I didn't even compile the build myself.

Now on to something different , I said I would post the results of my 5D2 10bit 2.1K green screen 8:00 min shot .
This was just a test , but work so well that I put more work in to it and ended up as a short talk/explanation of 10,12bit raw video.
I was very impressed was the results , even thou I had a few issue with setting the shot up, focusing(as I was by myself , No auto focus in 3x crop mode)
The setting I used was mlv+audio 10bit 2144x1076 23.976p from 30p (not sure if 24 from 30 make any differences)
Auto preview for Liveview , audio set to 16bit 48khz (I ran the audio from my Zoom N4h in to the ext. mic input on cam)
ISO 400 , EF 24-70 f2.8L set to 35mm (Crop mode -- 2.60 crop x 35mm= 91mm) and had my zactuo evf connected (I needed something to reference to even thou it only shows
the center crop of the 2.1K image & B/W , it let me know when the recording stops but never did  :D
Process the 10bit mlv+audio on my PC with acr a.e. (on the my Mac the 10bit frames don't show up just all black from mlvfs, do show image in Resolve thou)
as I wanted to sharping up the image ( little soft on the focus) started with the adobe neutral profile and just pulled down the contrast & desaturated (flatted out)
export as 16bit tiff , imported in to apple compressor made my prores4444xq , graded in FCPX . I could have spent more time grading but I was looking for a quick turn around.
Any ways I hope everyone Enjoys it.
Happy New Years to everyone here on the forum  :)


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 02, 2017, 04:41:50 AM
@reddeercity

Very good video and explanation about the last news!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 02, 2017, 07:44:28 AM
A few of us have been poking around the lv_raw_dump code for various cameras and it looks like the 1100D has very similar code to the 600D. I was able to get a grasp how to find the DEFAULT_RAW_BUFFER for two other similar cameras, the EOSM and 700D but the code for these cameras are vastly different--maybe because the 1100D/600D (T3/T3i) are Digic 4 while the EOSM/700D (T5i) are Digic 5 cameras?

It looks like the only cameras that are not working with 10/12 bit are all Digic 4 or dual Digic 4 cameras. The 60D and 600D are also Digic 4 and they are working so perhaps it will be possible to get all of the ML supported cameras working if we can understand how to follow these instructions?

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
*
* raw_buffer = get_raw_buffer()
* sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
* ...
* dump_file(filename, raw_buffer, 7*something...)
*/


[EDIT] Thanks to @dmilligan and @eNnvi for taking the time to explain the EOSM code to me.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 02, 2017, 08:59:48 AM
5D mark III 1.1.3 works like a charm.

- Only in zoom 5x can't visualize. But if you press Info, can visualize at 10x zoom.
- I tried 10bits with zoom mode, 3,5K 4:1/5/1 2,5K, etc... aspect ratio and everything works perfect, continuous recording, except over 105 Mb/s or more.

52 Degree Celsius after 30 minutes playing and changing. I don't know if is too much. (A tea it's around 70????  :D )
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 02, 2017, 07:22:34 PM
I'm going to be cleaning up the mess in my bitbucket download area  and posting new test builds but first a download count and preview of coming attractions:

Checkmarks are for cameras that are working with CONFIG_EDMAC_RAW_SLURP:


















5D3.123-118* made a few attempts at merging crop_rec but was unsuccessful
5D3.113-56* attempted merging crop_rec with limited success and had 129 downloads
6D.116-58
5D2.212-31
7D.203-22
EOSM.202-19
600D.102-17
70D.112-17* might have build issues? (Dec 23)
100D.101-16* might have build issues? (Dec 23)
650D.104-15
1100D.105-15* allocate_raw_buffer branch has CONFIG_EDMAC_RAW_SLURP working but not 10bit/12bit (Dec 19)
60D.111-14
700D.114-13
550D.109-11
500D.111-8
50D.109-6

The next round of test builds will not include the 70D and 100D because there might have been build issues with them and since they are not merged into the main repository I'll need the help of nikfreak with them. Also, the 1100D won't go up until we can track down the DEFAULT_RAW_BUFFER for that camera. The 600D, 650D and 700D test builds will define the PREFERRED_RAW_TYPE to see if that helps those cameras.

Give it a few hours for the new test builds to show up here:

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on January 02, 2017, 07:40:54 PM
Oh very unfortunate for being a  70D user
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 02, 2017, 09:20:31 PM
Quote from: andy kh on January 02, 2017, 07:40:54 PM
Oh very unfortunate for being a  70D user

70D and 100D users should be giving their feedback to the developers so these platforms will be accepted into the unified branch. Once they are in the main branch adding them to the raw_video_10bit_12bit branch is much easier.

https://bitbucket.org/hudson/magic-lantern/pull-requests/760/adding-support-for-the-70d-112/diff
https://bitbucket.org/hudson/magic-lantern/pull-requests/757/adding-support-for-the-eos-100d-sl1/diff

This also applies to 5D3.123 users:

https://bitbucket.org/hudson/magic-lantern/pull-requests/611/support-for-5d3-firmwares-123-and-113-in/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 02, 2017, 11:44:25 PM
Hey dfort,

     I'll make sure to test 0x5 on the debug build. Thanks for the heads up.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 02, 2017, 11:49:11 PM
Hey reddeercity,

     Thanks so much for making this video! This explains everything really well, much appreciated!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Asiyuk on January 02, 2017, 11:56:38 PM
Hi guys,

I have a problem with 3 of 100 video that I shoot just few days ago. Other 97 video files are  really good. I shoot 10bit mlv on my 5d3_1.1.3 with 24dec Dfort's build. The problem is that all dng sequence is extremly dark, contrast and all green. I attach some screenshots from davinci resolve and 2 original dng. I tried  also adobe after effects and lightroom, but have the same problem. I used mlvfs and mlv_dump to get dng, but i see this problem.

(https://pp.vk.me/c837638/v837638034/186c6/cQpCE7qTUXg.jpg)
(https://pp.vk.me/c837638/v837638034/186ce/4fcyWCp1_a8.jpg)


here is dng: https://we.tl/1Dp6BTOiB8 (https://we.tl/1Dp6BTOiB8)

Does enyone have some advice for me?

English is my third language, but I hope, you understand :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 03, 2017, 01:17:04 AM
@PaulHarwood856 & budafilms thanks for the positive feed back  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 03, 2017, 03:49:48 AM
Test builds are up.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

@reddeercity - Thanks for the video. So it was shot in 10bit on a 5D2 in crop mode? If you think it works good now think how good it will be once that camera is working with CONFIG_EDMAC_RAW_SLURP.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SpcCb on January 03, 2017, 04:27:44 AM
Thanks reddeercity, nice presentation ;)

So it's working only in crop mode on the 5D2? Or you are using this mode because of aliasing in scaled mode?
Ok, I saw in the other thread you said there'r some problems in full mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 03, 2017, 04:58:05 AM
Quote from: dfort on January 03, 2017, 03:49:48 AM
@reddeercity - Thanks for the video. So it was shot in 10bit on a 5D2 in crop mode?
Thanks
Yes with full mlv ,  was a test originally to see if it could record corruption free for extended time.

Quote from: dfort on January 03, 2017, 03:49:48 AM
If you think it works good now think how good it will be once that camera is working with CONFIG_EDMAC_RAW_SLURP.
Yes I know I've been looking in to that , there's a lot to learn to understand about CONFIG_EDMAC_RAW_SLURP on older cams

Quote from: SpcCb on January 03, 2017, 04:27:44 AM
Thanks reddeercity, nice presentation ;)
So it's working only in crop mode on the 5D2? Or you are using this mode because of aliasing in scaled mode?
Ok, I saw in the other thread you said there'r some problems in full mode.
Thanks
Yes 5d2 Full frame still has frames corruption problems
I wanted to see if things are still stable and reliable with bit reduction or just hit and miss .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 03, 2017, 09:30:22 AM
Quote from: dfort on January 03, 2017, 03:49:48 AM
Test builds are up.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

@reddeercity - Thanks for the video. So it was shot in 10bit on a 5D2 in crop mode? If you think it works good now think how good it will be once that camera is working with CONFIG_EDMAC_RAW_SLURP.
Nice job dfort! Is there a change log I can read? Will try it out tonight. Best, David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 03, 2017, 06:09:16 PM
Asiyuk: Sometimes I have the same problem with dark-green colors on video /10 bit raw build 18dec/ EOS 600D

dfort: Thanks for new build ! I will try
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 03, 2017, 07:26:12 PM
Can you post a sample? Could be black levels are off. How often is it happening?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 03, 2017, 08:19:06 PM
@Danne - Asiyuk uploaded a sample on Reply #942

Quotehere is dng: https://we.tl/1Dp6BTOiB8

Though maybe you would prefer an MLV file?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 03, 2017, 08:37:47 PM
Black and white levels are off. After correction.
(https://s28.postimg.org/sse0iqdvh/Screen_Shot_2017_01_03_at_20_36_49.png)

For 600D enter terminal:
cd folder/with/dng/files
exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dng
hit enter
For 5D markIII
exiftool "-BlackLevel=2046" "-whitelevel=16383" *.dng

Seems exiftool won,t work with the 5D mark III file. YOu can use exiv2 maybe or try mlv_dump or another converter and then add correct white/black levels.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on January 03, 2017, 09:02:39 PM
@dfort Thanks for all your doing for The Goal of lower bit rate. I do have a question to ask and maybe its been asked before but why does the 5D Mark II stay at MLV 1856 and MLV Lite 1880?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 03, 2017, 09:28:18 PM
Alright guys I know I'm a tad late for this party once again and I apologize. Really hate slowing you guys down. But you guys are all doing great pushing this project forward!  :)

Ran @dfort's latest (2017Jan02) for 5D3.113 and notice after recordings of 10/12-bit files while in mlv_play.mo I get this some kind of '...stack overflow' errors which produced several crash logs occasionally (even spitted out COREDUMP.DAT whatever that is) within period of 2-5 minutes or so. Here they are stored into my dropbox (https://www.dropbox.com/sh/yvw0mbobxjvi7gm/AAAOzCzLebRDCRarHVp1MS0aa?dl=0dropbox) just in case they give us hints or not.

I ran a bunch of test files in both mv1080 and mv720 with MLV + Audio and all seems well. Even in 5x crop mode at 2.5k/3.2k for shorter takes if you will.

https://vimeo.com/197948723

Now upon trying to upgrade firmware (in order for me to uninstall ML and do a clean upgrade to 5D3.123) which seems to not work with your build. I get stuck with a black screen along with a steady orange light next to the LED blinker (not blinking tho) and battery pull out fixes this. But then thought maybe I had to download a nightly from unified would help for me to do this to no avail.  :-\

Not sure if it was worth sharing this but I felt it was. Anyone else experiencing this?

Off I go to try and do 5D3.123 update (will probably have to use EOS Utility tho I shouldn't have to and suspect this has to do with the autoexec.bin?) and will report my findings then.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on January 03, 2017, 09:40:55 PM
You. Don't. Need. EOS. Utility. With. 1.2.3. And. 1.1.3.
You don't need to "clean up cam" before going to 1.2.3. or 1.1.3 either.
Just format a card to remove card's boot option (either use ML's "remove ML" format option or card reader). Copy Canon firmware file and upgrade/downgrade.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 03, 2017, 09:42:46 PM
To Asiyuk "dark-green" picture.
The worst videos are already deleted. But something like that:
"dng picture"
http://www.uschovna.cz/zasilka/LILKZ228LKN9X8ZA-SX9/

Original raw video file:
http://www.uschovna.cz/zasilka/LIIWK6FJT47XUGD8-IJG/

BUILD:
magiclantern-raw_video_10bit_12bit.2016Dec18.600D102
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 03, 2017, 10:02:45 PM
Check reply 951 Honza.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 03, 2017, 10:46:51 PM
Danne: After command:
exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dng

Picture looks much better, but not completely pure  ;-)

http://www.uschovna.cz/zasilka/LIRRW9FAWK638482-F4X/
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 03, 2017, 11:01:45 PM
Yes, some corruption going on as well. Well, to be expected with bleeding edge firmware builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 03, 2017, 11:02:14 PM
Quote from: Walter Schulz on January 03, 2017, 09:40:55 PM
You. Don't. Need. EOS. Utility. With. 1.2.3. And. 1.1.3.
You don't need to "clean up cam" before going to 1.2.3. or 1.1.3 either.
Just format a card to remove card's boot option (either use ML's "remove ML" format option or card reader). Copy Canon firmware file and upgrade/downgrade.

Got it, Walter. I was hoping not to do this because I had a whole bunch of random files that turned out to be awesome (from moments when I forget to set the 5D3 to record onto CF instead of SD) from the past few years even had files from 2015. So yeah you were right I had to format (prefer Disk Utility to keep the ExFat just because...) and was able to update straight up to 123 from in cam even with the bootflag enabled, ha!

Off I go to work and maybe I'll be able to shoot some dark scenes with ISO 6400 in 10-bit slo-mo and do some DF avg automation scripting from @Danne's latest and greatest cr2hdr.app (http://www.magiclantern.fm/forum/index.php?topic=15108.msg177259#msg177259cr2hdr.app) (even previews 10/12-bit flawlessly in MLRawViewer 1.4.3) while running in MLVFS mode. Holy shit dude!  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 03, 2017, 11:38:11 PM
Before:
(https://pp.vk.me/c837638/v837638034/186c6/cQpCE7qTUXg.jpg)

After:
(https://s28.postimg.org/sse0iqdvh/Screen_Shot_2017_01_03_at_20_36_49.png)

Good to know it is salvageable but how did that happen on 3 of 100 videos?

Quote from: DeafEyeJedi on January 03, 2017, 09:28:18 PM
Ran @dfort's latest (2017Jan02) for 5D3.113 and notice after recordings of 10/12-bit files while in mlv_play.mo I get this some kind of '...stack overflow' errors which produced several crash logs occasionally (even spitted out COREDUMP.DAT whatever that is) within period of 2-5 minutes or so.

Looks like you're loading way too many modules. Try loading just the one's you're testing first. I don't know how to read those logs but my guess is that it looks like you're low on memory.

Free Memory  : 144K + 3569K

The COREDUMP.DAT shows even less memory available.

Free Memory  : 135K + 3562K
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 04, 2017, 01:35:51 AM
FYI, the workarounds for the green cast (aka wrong black level) bug are very well known to those used to read sticky topics ;)

http://www.magiclantern.fm/forum/index.php?topic=11664.0
http://www.magiclantern.fm/forum/index.php?topic=11942.0

Now, why does it happen at low bit depths?

My best guess is that our raw backend sometimes decides to compute the black level at the same time with the bit depth switch. There are some sanity checks to avoid computing the black level from an invalid image, but they are not 100% effective. Disabling them should increase the probability of encountering this bug.

The only other tasks (besides raw recording) that could trigger a refresh on the raw parameters under normal usage are the raw overlays, so you now know what to disable until this issue is fixed.

The fix would be in the raw backend: bit depth changes should be done in raw.c and guarded by raw_sem, and lower bit depths should be handled differently. Some possibilities (not yet sure what to choose):

* disable all raw processing (e.g. overlays) at lower bit depths
   + minimal amount of coding
   - no overlays while recording at lower bit depths (including grayscale previews and so on)
* perform all raw processing on 16-bit buffers, as in raw_twk:
   + fast (all raw processing will be faster)
   - may requires custom memory management for raw buffers, as Canon code reserves memory for a 14-bit buffer
   - will overflow outside LiveView, as we cannot allocate a continuous block larger than a 14-bit full res picture
   - on the fly conversion to 16-bit could work, but gives additional complexity
* convert lower bit depths to 14-bit, with raw_twk code:
   + would reuse current 14-bit code
   - no speed gains from raw_twk
* create the low bit depth image from the main 14-bit raw buffer, instead of edmac_copy_rectangle
   + would reuse current 14-bit code
   + a similar code structure would also work for lossless compression
   - no speed gains from raw_twk
* variations of the above

Quote from: dfort on January 03, 2017, 11:38:11 PM
I don't know how to read those logs but my guess is that it looks like you're low on memory.
Free Memory  : 144K + 3569K
Free Memory  : 135K + 3562K

From mem.c, look for "malloc" and "AllocateMemory" at preferred_free_space. The above is not a low memory condition; quite the opposite. Some background info here (http://www.magiclantern.fm/forum/index.php?topic=8358.0).

Quote from: dfort on January 03, 2017, 11:38:11 PM
Looks like you're loading way too many modules. Try loading just the one's you're testing first.

This advice is perfectly valid; not just for memory reasons, but also for other kinds of code interactions. For example, a module might start its own tasks, and those are a scarce resource on certain cameras. Also, raw_rec and mlv_rec are known to conflict with each other on functions exported to other modules.

I'm thinking to place a warning on the Modules tab, to be displayed if you load too many modules. They are not meant to be loaded all at once.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on January 04, 2017, 09:55:06 AM
Hi !

@reddeercity

Thanks for your video, I saw it but I still have a question : is your liveview was freezing during recording ?

I've installed your version (with the link you've gave us) and copied all the settings I could see in your video,

I didn't manage with my 5D2  to have a liveview unfreezing in crop mode + MLV2.0 ( with a hdmi connected or without ).

Do you have a "magic" settings or perhaps I've missed something ?

David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: NoCp_Albert on January 04, 2017, 08:18:38 PM
Hallo Danne

Hopefully I can help a bit with my test for your last magiclantern-raw_video_10bit_12bit.2017Jan02.5D3123 upload.

First I made the same test as DeafEyeJedi in his Video on 5D3113 firmware. I used the same settings and the results are similar what you see on this Video. But, when I replay with MLV Player, I got clear Video clips.

My Standard on 5d3 -123 works with Canon Setting PAL and 1920 x 25 fps. Now, with your new Builds raw_video_10bit_12bit.2017Jan02.5D3123 and raw_video_10bit_12bit.2017Jan02.5D3113 the Camera switch during 3xCrop Mode in to NFTS (Canon Menu still 1920x25fps) and shows me the Frame rates goes to 29,776 fps, but this is NTFS not PAL.
By 29,766 fps and 2400x1320 - 16-9 recording Time is 8 - 12 Seconds. If I use FPS override, I can select 25 from 30 fps and it recording works continually without problem.

If I select the maximum 3520x1320 - 25fps the record stops with Frame skipping after 5 seconds and I must take the Battery out to start the Camera again. There is only the message about "Frame skipping"

Hope this Info is helpful for you. Generally I use MLV with Sound bei 1920x25 and use some times 3x Crop mode for Nature (Bird) documentaries with Lenses from 12 to 600mm. Your last magiclantern-crop_rec.2016Dec28.5D3123.zip works beautiful, only missing the 10bit mode for more frames.

Big thank you to all ML Developers
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ricbi1 on January 04, 2017, 09:14:27 PM
Did some tests on 5d3 and like others have reported 10bit works great in non-crop mode. I also pushed FPS to 60 at full HD and stopped recording at 1min + at 2:35 aspect ratio.
Really impressive!
Looking forward to doing more tests.
Recording to KomputerBay x1000 64gb
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 04, 2017, 09:28:06 PM
Just for information, I've got a few video files shot in 12 bit with wrong black level too, on the 6d.
I've shot about 25 mlv files during the day and got three files with green dark frames, never switched raw video settings, only turned camera on and off and adjusted diafragma and iso.
And did some switching between video and photo mode.

So it happens on multiple platforms.
Good to know where this weird green frames are coming from and how it is fixable ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on January 04, 2017, 09:50:50 PM
Quote from: ricbi1 on January 04, 2017, 09:14:27 PM
Did some tests on 5d3 and like others have reported 10bit works great in non-crop mode. I also pushed FPS to 60 at full HD and stopped recording at 1min + at 2:35 aspect ratio.
Really impressive!
Looking forward to doing more tests.
Recording to KomputerBay x1000 64gb

FullHD 60 fps? Its impossible)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ricbi1 on January 04, 2017, 10:19:08 PM
Meaning 1920x squeeze factor from Magic Lantern... but 1920 HD
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 04, 2017, 10:56:38 PM
Did some small updates, mostly refactors to reduce the need to hardcode model-specific values. Details here (https://bitbucket.org/hudson/magic-lantern/pull-requests/778/raw-fixes-part-3).

Would be nice to get feedback (whether things are still working fine) on all models that use EDMAC_RAW_SLURP. Behavior on older models is unlikely to be changed.

I had another attempt at fixing 5D2, unfortunately without success. Maybe next time. The changes that worked were committed to allocate-raw-lv-buffer (currently useful for 1100D).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 05, 2017, 02:57:42 AM
Quote from: benoit on January 04, 2017, 09:55:06 AM
Hi !@reddeercity 
Thanks for your video, I saw it but I still have a question : is your liveview was freezing during recording ?
Your welcome , No freezing liveview work as normal . Lv set to auto preview ,

Quote from: benoit on January 04, 2017, 09:55:06 AMDo you have a "magic" settings or perhaps I've missed something ?
My 5d2 ml cam setting (https://www.dropbox.com/sh/le3wt4fqf61kmic/AAA6Cqzs4kOb33GGiPLBg2Jxa?dl=0)

I did have 30p ntsc in canon menu and frame over ride to 23.976p


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on January 05, 2017, 12:40:39 PM
Whaaa  :D
Thanks a lot Reddeercity !
Yours settings files rules !
Now I have greyscale liveview on 5D2 , great !
On my settings I changed PAL to NTSC without results on unfreezing.
But with yours settings, I changed NTSC to PAL and liveview freezes ... so NTSC with your settings is the graal.
So I think, NTSC is not the only one parameter to tweak.

I post my old settings to find the differences with yours because I don't understand the meaning of each parameter on these config files.

magic.cfg :

# Magic Lantern Nightly.2016Dec01.5D2212 (b3dfbe7194f3 (x-perimental) tip)
# Built on 2016-12-01 22:52:58 UTC by rosiefort@RosieFoComputer
# Configuration saved on 2017/01/05 09:04:53
beta.warn = 5
menu.first = -7
time.indicator = 1
movie.log = 1
bv.av = 35
bv.tv = 99
bv.iso = 117
bv.auto = 1
rec.notify = 4
screen_layout.ext = 0
shutter.lock.value = 99
shutter.lock = 1
enable-liveview = 2
movie.restart = 1
fps.preset = 1
fps.override.idx = 32
fps.override = 1
zoom.disable.x10 = 1
hdrv.iso.b = 112
hdrv.iso.a = 88
battery.drain.rate.rev = 149
hist.meter = 0
audio.mgain = 0
audio.dgain.r = 18
audio.dgain.l = 18
audio.monitoring = 0
audio.input-choice = 0
crop.playback = 1
spotmeter.draw = 0
clear.preview = 4
waveform.draw = 0
zebra.draw = 0
global.draw = 0
disp.mode.x = 4


# Config file for module mlv_rec (MLV_REC.MO)

mlv.video.enabled = 1
mlv.bpp = 0
mlv.write_speed = 6153
mlv.display_rec_info = 2
mlv.warm_up = 2
mlv.buffer_fill_method = 0
mlv.show_graph = 1

Thanks again.
David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: danilaswift on January 05, 2017, 08:14:09 PM
Hello. Guys please lend the old assembly for canon 650d. In the new build of 2017 I was not happy with the video quality (ie, 12bit). Video is too sharp and a lot of red dots. In the old version, this was not.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 05, 2017, 10:14:47 PM
Short video test of 10bit raw with EOS 600D:
https://www.youtube.com/watch?v=OJwMxwI3-YE
Build dfort: magiclantern-raw_video_10bit_12bit.2017Jan02.600D102
I recorded 32 pcs of mlv files (raw_rec.mo). All was withouth damage or green cast.
The shutter time 1/30s was not good choice -> flickering
Original video file: https://ulozto.cz/!FwuHK8wc9CET/10bitraw-video-test-karvina-frystat-mp4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 05, 2017, 11:56:15 PM
Quote from: benoit on January 05, 2017, 12:40:39 PM
Thanks a lot Reddeercity !Yours settings files rules ! Now I have greyscale liveview on 5D2 , great ! On my settings I changed PAL to NTSC without results on unfreezing.
But with yours settings, I changed NTSC to PAL and liveview freezes ... so NTSC with your settings is the graal.
So I think, NTSC is not the only one parameter to tweak.
I post my old settings to find the differences with yours because I don't understand the meaning of each parameter on these config files.
magic.cfg :
# Magic Lantern Nightly.2016Dec01.5D2212 (b3dfbe7194f3 (x-perimental) tip)
# Built on 2016-12-01 22:52:58 UTC by rosiefort@RosieFoComputer
# Configuration saved on 2017/01/05 09:04:53
beta.warn = 5
menu.first = -7
time.indicator = 1
movie.log = 1
bv.av = 35
bv.tv = 99
bv.iso = 117
bv.auto = 1
rec.notify = 4
screen_layout.ext = 0
shutter.lock.value = 99
shutter.lock = 1
enable-liveview = 2
movie.restart = 1
fps.preset = 1
fps.override.idx = 32
fps.override = 1
zoom.disable.x10 = 1
hdrv.iso.b = 112
hdrv.iso.a = 88
battery.drain.rate.rev = 149
hist.meter = 0
audio.mgain = 0
audio.dgain.r = 18
audio.dgain.l = 18
audio.monitoring = 0
audio.input-choice = 0
crop.playback = 1
spotmeter.draw = 0
clear.preview = 4
waveform.draw = 0
zebra.draw = 0
global.draw = 0
disp.mode.x = 4


Looks like you have Auto Settings enabled , all my settings are manual .
So set the programs dial on the top left hand side to "M" and I use manual WB in Kevlin also ISO .
I use no Auto setting (only exception is ML Liveview Preview).
Only load the min needed for raw video to avoid problems .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 06, 2017, 12:34:29 AM
New test builds posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

Downloads for the Jan 2 test builds. Checkmarks are for cameras that are working with CONFIG_EDMAC_RAW_SLURP:















5D3.113-35
5D2.212-27
5D3.123-23
6D.116-13
60D.111-9
7D.203-8
50D.109-8
600D.102-8
EOSM.202-7
700D.114-7
550D.109-7
650D.104-5
500D.111-1
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on January 06, 2017, 11:01:00 AM
I shot a wedding last week – 10bit MLV lite on 5D3. All shot at 60p. It is quite stable. Had 2 clips with black level issues much like aforementioned in this thread. 1 clip completely corrupt, not sure why. Here's the link: https://youtu.be/KIjUZafUEBA
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 06, 2017, 02:55:30 PM
Added 10/12-bit builds to the nightly server:

http://builds.magiclantern.fm/experiments.html

Current builds are probably identical to dfort's (didn't really check); future builds will just track the raw_video_10bit_12bit branch. 1100D is included as well, but raw recording support is quite incomplete. Please report and fill in the missing bits (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176935#msg176935).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 06, 2017, 04:36:06 PM
Would love to test 5D3 123..  but it is missing..
Thanks for your support!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: maxotics on January 06, 2017, 06:02:43 PM
I'm looking forward to the 7D.  FYI, it doesn't seem you can use normal MLV with the EOS-M 10/12-bit build.  It will say "thread not starting" or something like that.   So seems like that is a 10-bit build only.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 06, 2017, 06:16:12 PM
Turn off raw_twk. That and snd.mo don,t go well together.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 06, 2017, 06:38:02 PM
Beginners question: what is and what does CONFIG_EDMAC_RAW_SLURP do? Curious..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 06, 2017, 07:45:05 PM
A method that retrieves the raw data from Canon's image processing pipeline to main memory by configuring the DMA (EDMAC) directly, with our own parameters, rather than calling Canon code (lv_save_raw).

See here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181#msg174181) and look it up in the source code, or here on the forum.

Initially implemented for 5D3 1.2.3 to fix corrupted frames (http://www.magiclantern.fm/forum/index.php?topic=10443.msg100979#msg100979).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 06, 2017, 07:58:19 PM
It seems it also can be used as a cheat sheet for dev's to pick up certain EDM addresses already provided for each specific settings in order to save time and get things implemented rather quicker basically.

Here's what the first page upon running EDMAC (after 'Don't click me!') looks like:

(https://c8.staticflickr.com/1/545/32146424775_7521cb4aee.jpg) (https://flic.kr/p/QYEQ82)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RTI on January 06, 2017, 09:12:01 PM
Currently testing 10bit on my canon 6D
-I'm getting random slowdowns (using a lexar UHS-I 128 633, max write speed with 6D is around 38-39Mb/s) - quite often I get 9-10 MB when trying to record (can't reproduce it).
-Also, battery percentage always shows 100% (when in global draw ofc, in canon menu the right values are displayed).
-ISO display seems to show only full EV values, no 1/3EV.
-mlv_play doesn't work, shows some corrupt images.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 07, 2017, 12:29:24 AM
If you want the latest builds, bookmark this address:

https://builds.magiclantern.fm/experiments.html

I'm still putting up builds for the 5D3.123 and the platforms without CONFIG_EDMAC_RAW_SLURP. Seems that there is still some limited 10bit/12bit functionality on the 5D2 and maybe 7D. It is also a way to keep track on which platforms still need to catch up.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

Hum, 121 downloads for the 550D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on January 07, 2017, 05:21:58 AM
Yes, the 7D works fine (for my purposes) in 5x crop mode with the exception of not having true live framing (but I wasn't using that even with 14 bit). I have seen no corruption issues in crop mode with 10 and 12 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: NoCp_Albert on January 07, 2017, 07:02:14 AM
Hallo dfort

the last build from https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2017Jan05.5D3123.zip shows me the problem during mlv_play, without loading the raw_rec module I get corrupted files to see, if I see the recorded frames at MLVProducer everything is ok. If I load the raw-rec module and record with mlv_rec also in mlv-play I see the clip correct. This was not happen on your build from 2 January 2017.

If I like to go back from 5xCrop used with 50fps by 10bit 1920x648 to 5xCrop 233x1314 25fps 10bit must switch of and on to work with the new setting. With 5x crop the lifeview shows the the images are jumping from normal size to 5x crop sometimes, if you like i prepare a video to show what I mean.

When I try to use normaly 1920x1080 25fps by 10, 12, 14 bit the record stops after 12 seconds bei "Frame skipping"
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 07, 2017, 10:56:08 AM
I tried the -a(average all frames) option in mlv_dump.osx on some 12 bit files for creating darkframes.
But the average file doesn't look right...it looks like a weird purple frame (when extracting the MLV files to dng's, I get normal black frames...)
Could it be that the average all frames option in mlv_dump doesn't work (yet :D) on 10 and 12 bit files ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 11:24:02 AM
Do you get the same with 10-bit? I use it without issues in cr2hdr.app.
g3gg0 rewrote code so it works with 14bit code in mlv_dump. Very great.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 07, 2017, 11:39:05 AM
Quote from: Levas on January 07, 2017, 10:56:08 AM
I tried the -a(average all frames) option in mlv_dump.osx on some 12 bit files for creating darkframes.
But the average file doesn't look right...it looks like a weird purple frame (when extracting the MLV files to dng's, I get normal black frames...)
Could it be that the average all frames option in mlv_dump doesn't work (yet :D) on 10 and 12 bit files ?

will investigate that
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 11:53:51 AM
Just ran darkframe averaging on my eos m in 12bit and all was working fine on that particular cam. Make sure to use the very latest version of mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 07, 2017, 11:55:21 AM
ok thanks. you ran it on osx ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 12:05:10 PM
Yup macOS Sierra.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 07, 2017, 12:35:49 PM
Sounds like I'm doing something wrong here...
I'll try the latest mlv_dump, is there a standard download location for the newest mlv_dump?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 12:50:47 PM
It,s in the 10/12bit branch.
http://www.magiclantern.fm/forum/index.php?topic=7122.msg177020#msg177020
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on January 07, 2017, 04:58:09 PM
Quote from: dfort on January 07, 2017, 12:29:24 AM
Hum, 121 downloads for the 550D?

That'll be because there's a lot of 550D users, more than EOS M and for good reason. Joking aside, 12bit works so some improvement for limited card write speed, obvious 10bit is eagerly awaited no doubt.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 07, 2017, 07:08:09 PM
Quote from: NoCp_Albert on January 07, 2017, 07:02:14 AM
...if you like i prepare a video to show what I mean.

That might help. I'm having a hard time following what changed from January 2 to January 5 that would cause those issues. It might be a bad merge on my part.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 07, 2017, 10:02:41 PM
Some color and resolution testing of latest :  magiclantern-raw_video_10bit_12bit.2017Jan05.5D3123

Original image from CR2 (cropped)  taken in manual mode ISO 100   White Balance in manual at 3900K
When open in Adobe Camera Raw WB reads:  Temperature +3900 and Tint +7
(https://s20.postimg.org/ncva4g0bx/From_CR2_image.jpg)

Recorded video in Manual Mode ISO100 WB=3900K and extracted DNG for comparison with good exposure and then with low light but bumped exposure by +4EV

TEST1: Properly exposed video Image from DNG.. Note that when I open with Adobe Camera RAW it reads: Temperature +5500 and  Tint +19
On the left are the un-edited images and on the right I have used "Auto WB" in Adobe Camera Raw
(https://s20.postimg.org/pipkyy3sd/comparison.jpg)

TEST2: Low light exposed video image from DNG..  Note that when I open with Adobe Camera RAW it reads: Temperature +5500 and  Tint +19
On the left are the un-edited images and on the right I have used "Auto WB" in Adobe Camera Raw
Exposure has been edited to +4 EV
(https://s20.postimg.org/vwjpzbezx/comparison_bad_expose_4ev_3_X.jpg)

Question 1: Why is MLV not properly passing White Balance to ACR in the DNG format.
Question 2: Why is there some significant White Balance tweaking required in 10 bit.

NOTE: I cannot find any significant difference in resolution between 10-12-14 bit.  After pixel peeping I see a tiny amount if difference between 10 and 14 bit.. only on low exposure areas.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 10:21:26 PM
Thanks for tests. Wonder what darkframe averaging would do for those 10-bit files.
Hard coded wb values in mlv_dump. Until ml-dng code finds the way into unified.
If important you can dig out mlv_dump in cr2hdr.app and use that one. Bouncyball put together missing white balance parts in the ml-dng code dmilligan uses in mlvfs and in his ml-dng code into that version of mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 07, 2017, 11:04:54 PM
@Danne I dont think there is one available for 10bit and windows??
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 07, 2017, 11:08:27 PM
Quote from: RenatoPhoto on January 07, 2017, 10:02:41 PM
Question 2: Why is there some significant White Balance tweaking required in 10 bit.

Black level may need some fine-tuning to reduce roundoff errors.

When you simply discard some bits, the resulting signal, when scaled back, will have lower values compared to the original. For 14->10 bit conversion, the difference will be between 0 and 15 DN.

Therefore, when interpreting 10-bit data, it's probably best to adjust the black level by 0.5 LSB. This would require scaling the 10-bit data to a higher bit depth (at least 11).

For latest mlv_dump, which scales the 10-bit data back to 14-bit levels (multiplies by 16), the workaround would be:

exiftool *.dng -BlackLevel=2040


(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2014%20bit.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202032.jpg)
(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202040.jpg)


[14-bit 2047    10to14 2032;
10to14 2048    10to14 2040]


Test image from here (http://www.eoshd.com/comments/topic/21004-12-or-10-bit-raw-magic-lantern/?do=findComment&comment=168058).

For a fix, the raw recording module should probably save the 14-bit black/white levels and let mlv_dump do the scaling. Or, remove the autodetection and just hardcode the reference black level (e.g. 2048 on 5D3); in this case, the roundoff error will be predictable.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 12:37:42 AM
Thanks A1lex:  Great explanation and solution..

After applying the exif fix here are the results:

(https://s20.postimg.org/3yzi1vf71/Compare_after_exif.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on January 08, 2017, 01:26:33 AM
Sorry guys if i'm taking so long but i'm really in a short of spare time.

I'm still trying to understand what's the big difference between digic 4 and digic 5 for raw_slurp

Io to now i think i just discovered hot water as i'm trying ti figure out how Memory is dinamically allocated in Canon firmware. As a1ex said a few posts ago probably the raw buffer address is a variabile inside a structure. My Guess is that in digic 5 this struct is statically allocated while in digic 4 this is dinamically allocated via a Memory lib (for garbage collection? With debugging symbols and assert management i think)
This should also explain why in digic 5 se get a pointer while in digic 4 we get a pointer to a pointer.

What i don't understand is why the offsets from the struct base address are so different from camera to camera.

Is it possibile to dump a certain part of RAM while running? From address X get N bytes?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 08, 2017, 02:12:39 AM
@eNnvi -- You can make fresh dumps of ROM0.BIN, ROM1.BIN and RAM4.BIN from the Debug menu.

Looks like the cameras that need to get the CONFIG_EDMAC_RAW_SLURP treatment are all Digic 4:







5D2.212-Digic 4
7D.203-Dual Digic 4
50D.109-Digic 4
500D.111-Digic 4
550D.109-Digic 4

The 60D, 600D and 1100D are also Digic 4 but I have not found out how these were resolved. I keep getting lost tracing lv_raw_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on January 08, 2017, 04:54:49 AM
Quote from: a1ex on January 07, 2017, 11:08:27 PM
Therefore, when interpreting 10-bit data, it's probably best to adjust the black level by 0.5 LSB.
Or perhaps even just adding back 0-15 units of noise?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on January 08, 2017, 06:09:15 AM
So why is the 600D experimental?  It seemed to be one of the most full functioned?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 08, 2017, 07:14:55 AM
Quote from: ShootMeAlready on January 08, 2017, 06:09:15 AM
So why is the 600D experimental?  It seemed to be one of the most full functioned?

It isn't the 600D that's experimental, it is "10/12-bit RAW video" that is experimental.

This might be the first time that a development branch is being compiled on the Jenkins server so that ML users can have a preview of coming attractions. Note that there is also a section for "Non-CPU lens info" and a "Crop mode recording (crop_rec)" section too. Yeah, I know, users want 10/12-bit and crop_rec but that requires resolving merge conflicts and I already botched it up a few times because I don't have a 5D3 for testing so you can ask someone else to do it for you or learn how to compile ML and try it yourself. It isn't that hard and there are several tutorials in the General Development Discussion (http://www.magiclantern.fm/forum/index.php?board=25.0) section of the forum. (I'm starting to sound like an old man repeating the same thing over and over.)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on January 08, 2017, 10:14:39 AM
Quote from: dfort on January 08, 2017, 02:12:39 AM

The 60D, 600D and 1100D are also Digic 4 but I have not found out how these were resolved.

Those use EVF_STATE being DIGIC4. All other DIGIC4's using LVSTATE due to the missing EVF_STATE are the ones that are not working. a1ex has the state diagrams published on his bitbucket repo and allocating the buffer is not the problem (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176545#msg176545)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on January 08, 2017, 11:04:43 AM
I tried mlv_rec (and mlv_play + raw_twk) with the nightly experimental version for the 50D using non-crop, 5x crop and 10x crop at 10, 12 and 14bit.

10 and 12bit recording exhibit frame tearing with the lower portion of the image always being frame 1 but there were no colored/noisy corrupt frames. 14bit has no corrupt frames.

I don't remember the 50D ever recording in 10x but it does now  ??? (though it might just be that I have never used it before  :P)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 08, 2017, 11:52:47 AM
So I've downloaded the 25th december 2016 version of mlv_dump and now I get normal dark dark frames from averaging a mlv file.
And I noticed the differince in file size, this version of MLV dump creates 14 bit files, where the older mlv_dump created 12 or 10 bit files.
Is this for compatibility reasons or are there other advantages that the output files are converted to 14 bit ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 11:58:33 AM
Quote from: Levas on January 08, 2017, 11:52:47 AM
So I've downloaded the 25th december 2016 version of mlv_dump and now I get normal dark dark frames from averaging a mlv file.
And I noticed the differince in file size, this version of MLV dump creates 14 bit files, where the older mlv_dump created 12 or 10 bit files.
Is this for compatibility reasons or are there other advantages that the output files are converted to 14 bit ?

can you tell me the exact commandline, the bit depths of the input files and the bit depth the output file had?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 08, 2017, 12:06:52 PM
I'm using an old script which was once available on magic lantern, made some alterations for averaging.
Input file is 12 bit mlv, output file is a single frame 14 bit mlv

workingDir=`dirname "$0"`
cd "${workingDir}"

sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
sudo chmod +x /usr/bin/mlv_dump

for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    mv ./"$BASE".M* ./$BASE
    cd ./$BASE
    /usr/bin/mlv_dump --no-fixcp --no-stripes -o ${BASE}_frame_ $FILE -a
    cd ..
done
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 12:56:18 PM
"--no-fixcp --no-stripes" should not have any effect. they are for dng export only.


root@linux-dev:/media/sf_M_DRIVE/raw/10_12_14bpp# mlv_dump M24-2240.MLV -v | grep bits_per_pixel
      bits_per_pixel   12
root@linux-dev:/media/sf_M_DRIVE/raw/10_12_14bpp# mlv_dump --no-fixcp --no-stripes M24-2240.MLV -o M24-2240_avg.mlv -a > /dev/null
root@linux-dev:/media/sf_M_DRIVE/raw/10_12_14bpp# mlv_dump M24-2240_avg.mlv -v | grep bits_per_pixel
      bits_per_pixel   12


which mlv_dump did you use?
when in doubt, always this one: http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 08, 2017, 01:44:14 PM
www.youtube.com/watch?v=UWFVnE4-xp0

Test of 10 bit raw video recorded on Canon EOS 600D during daylight.
Resolution: 1728x576, 23,97fps, ISO 100, mlv_rec.mo + mlv_snd.mo

BUILD/dfort/: magiclantern-raw_video_10bit_12bit.2017Jan05.600D102

I recorded 25 mlv video files. Four files were corupted (green cast, frame-skipping).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 01:58:39 PM
"vibration" could be a stuck frame. then you will see parts of (or the whole) older frame.
it could be the first frame, or even the second-to-last frame you see there.

green cast is no corruption, just a black level error.
fix it (as described here a few times) by correcting the black level in DNG files, or by supplying "--black-fix=<value>" to mlv_dump with value depending on the bit depth.
when calling without any value "--black-fix" it will set black level to 2048
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 08, 2017, 02:52:59 PM
Quote from: dmilligan on January 08, 2017, 04:54:49 AM
Or perhaps even just adding back 0-15 units of noise?

(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20off8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20off8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20r2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20r2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20r4.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20r4.jpg)
(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20r8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20r8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20r16.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20r16.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20n1.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20n1.jpg)
(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20n2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20n2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20n4.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20n4.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20n8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20n8.jpg)

Which one looks best? Clicking on each thumbnail will open the full test image.

Render settings:

ufraw-batch --out-type=jpg --exposure=5 --temperature=2000 --green=1 --interpolation=ahd --color-smoothing --clip=film
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 03:02:23 PM
@g3goo  When I use --black-fix the dng cannot be read on Adobe Camera Raw and the command prompt shows "Cold pixels : 200000" see below:

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M07-1506.mlv'
   - Setting black level to 2048
   - Convert to DNG frames
   - Output into 'M07-1506.raw'
File M07-1506.mlv opened
File M07-1506.m00 not existing.
Processing...


Vertical stripes correction:
  1.00000    1      1      1      1      1      1      1
Cold pixels : 200000
Reached end of chunk 1/1 after 306 blocks
Processed 146 video frames
Done

edit: using the windows version
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 08, 2017, 03:13:34 PM
to g3gg0:
1) "vibration" it could be
2) to fix "green cast" I used command: exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dng
In this case 2/3 of picture is OK, but 1/3 (upper part of frame) is not usable.
Sample is here. I can't upload picture to this forum :-/
http://uschovna.cz/zasilka/LIBLRNMY6XS63UNG-9LL/49WFXLN2CF
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 03:19:44 PM
@a1ex:  Seems like n1 or r2 have less color noise than others.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 03:36:52 PM
@A1ex. r2 and r4 looks good. Bottom three not so.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 03:42:57 PM
@Renatophoto.
I never used --black-fix option in mlv_dump before but just tried it and it worked over here. Did you also specify the black level number like this --black-fix=2048
Black level setting has nothing to do with cold pixels so not sure why you would be getting that.
Could you specify the full command line output you are using?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 08, 2017, 04:04:10 PM
@Geggo
I did use the same mlv_dump as in your link.
The build from 25th december for osx

When I use that build on 12 bit mlv's, I'll get 14 bit dng's  ???
Which is nice, as in you can view them normal in finder, but the downside is that file size is also increased  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 05:23:40 PM
@Danne
mlv_dump --dng --black-fix=204 -o %Input%.raw %Input%.mlv
I suspect is something to do with ACR.  If I use exiftool then it works..  I was hoping to eliminate the exiftool step.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 05:31:01 PM
Tried putting --black fix command before the --dng?
Are you on mac? Could you upload a dng?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 05:40:59 PM
Yea, no change.. here is a dng
https://drive.google.com/file/d/0B9z8Y0rg-pu8SjFaMUtyeTRzT0E/view?usp=sharing
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 05:55:49 PM
I don,t think you are doing it right. Try something like this. It absolutely works over here.
mlv_dump --black-fix=2048 --dng My.MLV

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 08, 2017, 06:04:21 PM
Quote from: Andy600 on January 08, 2017, 11:04:43 AM
I don't remember the 50D ever recording in 10x but it does now  ???

So it joins the 5D2 and 7D as cameras that shouldn't be working but are, sometimes, at certain settings? Does the 50D have zoom mode? Zoom in the LiveView to 5x or 10x and record video? I believe that's the setting that's working on the 5D2 but I'm not sure.

Quote from: nikfreak on January 08, 2017, 02:12:39 PMThose use EVF_STATE being DIGIC4. All other DIGIC4's using LVSTATE due to the missing EVF_STATE are the ones that are not working. a1ex has the state diagrams published on his bitbucket repo and allocating the buffer is not the problem (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176545#msg176545)

So the 60D, 600D, 1100D use EVF_STATE while the 5D2, 7D, 50D, 500D, 550D use LVSTATE and are missing the EVF_STATE? Learning how the 600D was resolved #define DEFAULT_RAW_BUFFER MEM(MEM(0x51FC)) isn't going to help with the 550D even though lv_raw_dump starts out very similar on both cameras? I keep getting lost and not knowing how the 600D address was discovered I'm clueless when it comes to trying to figure out these platforms. After @dmilligan explained how he traced lv_raw_dump on the EOSM figuring it out on the 700D and 650D was relatively easy. (Thanks to @eNnvi for explaining dmilligan's explaination  :D )

https://bitbucket.org/hudson/magic-lantern/pull-requests/781/650d-and-700d-enable/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 06:05:50 PM
@Danne  No Go!
As I said it seems to be something wrong with ACR..  If I use the exiftool then it works.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 06:11:43 PM
Quote from: Levas on January 08, 2017, 04:04:10 PM
When I use that build on 12 bit mlv's, I'll get 14 bit dng's  ???
vs.
Quote from: Levas on January 08, 2017, 04:04:10 PM
Input file is 12 bit mlv, output file is a single frame 14 bit mlv

do we talk about the MLV -> MLV(averaged) script that produces 14 bit files
or about DNG conversion which is 14 bit only yet?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 06:13:44 PM
Quote from: RenatoPhoto on January 08, 2017, 06:05:50 PM
@Danne  No Go!
As I said it seems to be something wrong with ACR..  If I use the exiftool then it works.

is it better with '--no-fixcp --no-stripes' ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 06:25:09 PM
@Renatophoto.
This command can,t work which you put up.
mlv_dump --dng --black-fix=204 -o %Input%.raw %Input%.mlv
You can,t even create a new MLV with another black level. In case it would have worked it would look like this.
mlv_dump --black-fix=2048 -o Output.MLV Input.MLV

It only works when spitting out dng files. Try exactly what I put out here.
mlv_dump --dng --black-fix=2048 INPUT.MLV

If still not working upload part of you mlv file

@Levas
QuoteInput file is 12 bit mlv, output file is a single frame 14 bit mlv
I thought this was to be expected, especially since g3gg0 updated code so all bits(10/12/14) would work with the 14bit code in mlv_dump?


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 06:45:03 PM
Danne: I tried your suggestion and it does not work.
g3gg0: I tried but no luck either.

I think it something related to my system and ACR.  I will use exiftool to do the black level correction until it is fixed directly on the mlv_rec.mo
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 08, 2017, 06:46:57 PM
@Geggo

MLV conversion to dng's gives 14bit dng's even with 12 bit mlv's.
So 12 bit MLV gives 14 bit dng frames
Doesn't matter if average is used or not, always end up with 14 bit dng's

The MLV_dump build I used before, 20 November 2016 build, gave 12 bit dng's or 10 bit or 14 bit, depending whatever bit depth the mlv file was recorded.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 08:17:10 PM
so MLV averaging produces the correct format, right?
i am asking because you reported the generated MLV has 14 bits although it should have 12 bits.

DNGs are *always* 14 bit in the branches i know. maybe you used mlv_dump builds from other sources.
do not know about the pending PR if it would produce 12bpp DNGs


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 08, 2017, 10:52:07 PM
Quote from: Danne on January 08, 2017, 03:36:52 PM
@A1ex. r2 and r4 looks good. Bottom three not so.

Agreed!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 09, 2017, 04:46:42 PM
3X resolution testing  of latest dfort:  magiclantern-raw_video_10bit_12bit.2017Jan05.5D3123

In liveview (zoom 3x) the ML Grey Scale preview flickers when selecting aspect ratios beyond 16:9 i.e 1.85:1, 2:1, etc

Using cinnema ratios like 2.35:1 In liveview (zoom 3x) in ML Grey Scale preview the areas above and below the video, which should be black, have images (lines here) on the right half of the screen.  see below:
(https://s20.postimg.org/8ldzt6u99/VRAM1.jpg)


With big resolutions I get some bad frames:

3008x1320:
(https://s20.postimg.org/bgr301y99/3008x1320.jpg)

3520x1320:
(https://s20.postimg.org/xh7fgogx9/3520x1320.jpg)

Best Regards,
Renato
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 09, 2017, 05:56:46 PM
3X resolution testing  of latest dfort:  magiclantern-raw_video_10bit_12bit.2017Jan05.5D3123

Also ML Grey Scale turns to Canon preview while recording..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 09, 2017, 06:23:30 PM
@Geggo
Quoteso MLV averaging produces the correct format, right?

Yes, the resulting average MLV has the right format, the average MLV is 12 bit, when input MLV is 12 bit.

Only thing new to me was that when extracting dng's, it gives 14 bit dng's no matter what the input is, 10/12 or 14 bit.
I was used to a MLV_dump which gave dng's in the same bit depth as the recording was.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 10, 2017, 07:15:47 AM
I made a test with a card Tone 14/12/10 bits
5d Mark III 1.1.3
Build January 5

I hope this can be usefull for the comunity.

Full info into the video.

https://youtu.be/owo2WWcyC28
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on January 10, 2017, 11:50:23 AM
Quote from: budafilms on January 10, 2017, 07:15:47 AM
I made a test with a card Tone 14/12/10 bits
5d Mark III 1.1.3
Build January 5

I hope this can be usefull for the comunity.

Full info into the video.

https://youtu.be/owo2WWcyC28

Is this before or after the black level fix posted by A1ex above for 10 bit files. It is supposed to remove the purple colour cast in 10 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 10, 2017, 04:38:09 PM
I didn't apply any correction, no black level 2048 in 5d Mark III.
But if @a1ex need a new test I can make it again!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 10, 2017, 04:48:18 PM
Here is the latest version of mlv_dump with g3gg0 latest commits. It,s vital to specify black level.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump

Source: https://bitbucket.org/hudson/magic-lantern/commits/75f61f30b6e2a6fb1a136d538b8fe190919024a9?at=unified

Specify like this if working with a 5D mark III
mlv_dump --black-fix=2048 --dng your.MLV
Correction:
mlv_dump --black-fix=2040 --dng your.MLV
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 10, 2017, 04:50:58 PM
... 2040 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177802#msg177802) ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 10, 2017, 04:55:12 PM
Thanks, corrected.
WHat is the black level number for 12bit? The same as for 10bit?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tracerman on January 10, 2017, 05:51:10 PM
Hi,

I've tested 10bit and 12bit mlv_rec on the 550D : magiclantern-raw_video_10bit_12bit.2017Jan05.550D109

10 bit recording is indeed eagerly awaited on this platform, because 1280x544(2.35) at 23.98 fps makes about 20MBps
which is the maximum rate I can get from it (SandDisk 64Gb ExtremePro) so its pretty cool :)
big thanks to the ml devs, it gives a new life to this (ok rather old) camera but with (it seems) a lot of users still ...

I tested it in this configuration, recording goes OK (green light!) but when I try to read back it's not quite OK yet
(I've tried with mlv_play on the camera (with raw_twk loaded), with mlvfs on Linux and mlvproducer on Win7 with the same result) :

https://youtu.be/iJWjrpm-HHI

In 10 bit mode : it seems that 1 frame out of 2 is in fact the 1st recorded frame, and the other has problems on the top third of the frame.

In 12 bit mode : every frame is split vertically in 2 and offset in time and/or vertically (I can post another video if you need).

14 bit mode is all OK but recording stops after a few seconds (buffer full).

If I can be of some help in testing just let me know, having at least 10bit mlv_rec working would be awesome !

Thanks !

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 10, 2017, 06:24:15 PM
@a1ex @danne

Yes it works! 5d3.
No color pink/magenta and blue. Tested 10/12/14 bits

./mlv_dump --black-fix=2040 --dng M10-0135.MLV

(https://s29.postimg.org/z0rrmw3tv/M10_0136_000206.jpg) (https://postimg.org/image/z0rrmw3tv/)

(https://s29.postimg.org/phi2tfgbn/M10_0137_000188.jpg) (https://postimg.org/image/phi2tfgbn/)

(https://s29.postimg.org/60xd6wl7n/M10_0135_000415.jpg) (https://postimg.org/image/60xd6wl7n/)


image upload no limit (https://postimage.org/)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 10, 2017, 06:45:43 PM
Could you upload the dng files? And name them 10/12/14-bit so one can differ them.
Shouldn,t 14-bit have 2048 black level? 10 and 12 can,t be both 2040 or what am I missing?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 10, 2017, 06:56:02 PM
@Danne

I applied the same 2040 to 10/12/14 bits.

https://www.dropbox.com/sh/26id73senhicchy/AADDKTM9PefdK6rY0n7AhIa3a?dl=0


off topic: resolve studio (pay version) has a command called Noise Reduction. I don't know if this is usefull: https://www.youtube.com/watch?v=x5BhDlwJukY
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 10, 2017, 08:07:47 PM
I need a 14-bit DNG sample from every single camera model, from a recent build (not older than 2017Jan05), either from experimental nightly or from dfort (doesn't matter). It must have some overexposed highlights and should be extracted with:

mlv_dump --dng --no-fixcp --no-stripes input.MLV


Please post them in this thread: http://www.magiclantern.fm/forum/index.php?topic=11899

In particular, I expect surprises on 6D and 1100D, but I'd like to check all models.

(sorry for the large font, but I'd like to merge this (https://bitbucket.org/hudson/magic-lantern/pull-requests/778/raw-fixes-part-3) in the main builds before 5D3.123, 100D, 70D, and I need these samples to make sure everything is fine)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 10, 2017, 09:51:38 PM
Some more crash logs (Error 70) recently with @dfort's latest 5D3.123 (2017Jan05) and this usually happens after previewing files through mlv_play.mo so not sure if they are related or even have anything to do with raw_twk.mo?

https://mega.nz/#F!vhdRET6S!rIZA_Ko_BMjxQGbwIm-d4w
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 10, 2017, 10:29:49 PM
@dfort:
Testing 7D.203 with :  magiclantern-raw_video_10bit_12bit.2017Jan05.7D203

14 bit works well..

12bit: I get damaged frame on every other frame, see below:
(https://s20.postimg.org/fwe3j2019/12bit_7_D.jpg)

10bit: I get damaged frame on every other frame, see below:
(https://s20.postimg.org/oq5005mzx/10bit_7_D.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 11, 2017, 12:04:53 AM
Quote from: budafilms on January 10, 2017, 06:24:15 PM
@a1ex @danne

Yes it works! 5d3.
No color pink/magenta and blue. Tested 10/12/14 bits

./mlv_dump --black-fix=2040 --dng M10-0135.MLV

(https://s29.postimg.org/z0rrmw3tv/M10_0136_000206.jpg) (https://postimg.org/image/z0rrmw3tv/)

(https://s29.postimg.org/phi2tfgbn/M10_0137_000188.jpg) (https://postimg.org/image/phi2tfgbn/)

(https://s29.postimg.org/60xd6wl7n/M10_0135_000415.jpg) (https://postimg.org/image/60xd6wl7n/)


image upload no limit (https://postimage.org/)


Looking at your tests I get increased blue in 12 and 10-bit. It gos higher and higher per bit. Look below:

(http://forumbilder.se/G74CS/10bits-vs-12-and-14.png)

It's really easy to fix in post per se, but for your info the blue/magenta is still there.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 11, 2017, 01:58:49 AM
Quote from: RenatoPhoto on January 10, 2017, 10:29:49 PM
Testing 7D.203

Right, that's expected. I believe that is also happening on the 5D2 but crop (zoom) mode is working on that camera. Does the 7D have that feature? Press the magnifying glass button to 5x or 10x and record video. Which feature is that on the features list (https://builds.magiclantern.fm/features.html)?

That isn't the solution to  but it is interesting that it works on the 5D2. Maybe it also works on the 7D?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on January 11, 2017, 05:29:14 AM
dfort, if you mean regular classic 5x zoom crop mode, then yes the 7D has that feature and 10 and 12 bit have no corruption issues when using that mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 11, 2017, 07:18:21 AM
@D_Odell

Thanks for the comparation and design. But I see a big mistake (it's my house and I know how the light goes over the table, white and stanley, and less behind that line and on the floor).

If you see carefully, horizontally (the same design like your version but not vertical) you can see is not a increment of blue or magenta. There is a difference in the noise between 10 and 14. But no incresed the blue and pink.

Other way to comprobe this is zoom in at 1000%. Something not usefull in the reality. Remember video has "movement" between pixels.

If you want to try the horizontally design, comparation only over the Card Tone, could be interesting and more accurate.

;)

Edit: the light behind the tone card (kitcheen) is Fluorescent (blue and pink), and over the card, other room, its tungsten 3200K.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 11, 2017, 07:37:35 AM
Could you post an example using black level 2048 on all three picures Budafilm?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 11, 2017, 07:53:29 AM
@Danne

Reply #1050 to you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 11, 2017, 09:16:22 AM
I,d say run all footage with black level 2048(5D mark III). A1ex seems to agree, at least counting from yesterday https://bitbucket.org/hudson/magic-lantern/commits/e3ef8c75cf5de2e4018098d21ae6d1d3bcd183d5
The 14-bit file is slightly different. White balance, color depth?

Order: 10/12/14
(https://s29.postimg.org/9j7pjt6uv/Screen_Shot_2017_01_11_at_08_58_50.png)
(https://s29.postimg.org/wjecw54on/Screen_Shot_2017_01_11_at_08_58_57.png)
(https://s29.postimg.org/h9ehoy96f/Screen_Shot_2017_01_11_at_08_59_06.png)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on January 11, 2017, 10:07:25 AM
I see at the  top of the pict. in the blue band a good visible difference between the 10,12,14 bit. 14bit has more details, see at the reflections in it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ottoga on January 11, 2017, 12:18:30 PM
@DFORT

10/12bit Testing

Camera 7D
ML Build: magiclantern-raw_video_10bit_12bit.2017Jan02.7D203 (DFort's)

Modules Loaded:   Fileman, mlv_rec, mlv_play, mlv_snd, pic_view & raw_twk                           

DNG's Extracted using mlv_dump (07/01/2017 build), parameters used --dng --cs2x2
Post processing/editing of extracted dngs - none.


Results of 10bit test using mlv_rec

Frame 000000 looks good with no colour casts, hot/cold pixels etc.
Frame 000001 looks good with no colour casts, hot/cold pixels etc.
Frame 000002 Has the top 1/3 of the image distorted and barely discernable as an image.
                       The bottom  2/3rds of the image look good.
Frame 000003 Same as frames 000000 and 000001.
frame 000004 Same as frame 000002

All subsequent  pairs of frames i.e: 000005&000006, 000007&000008 etc. behave the same as frames 000003&000004.

What is actualy happening here?  from what I can see:

1) All of the frames that have the top 1/3 distorted are a combination of the actual image that should be within that frame and a static image that is constantly repeating and overwriting the actual image. Having shuffled through the frames I believe that it is the distorted part of the frame that is the actual image for the respective frame. The undistorted part is likely from frame 000000.

2) Apart from frame 000000, all of the frames that are distortion free is actually a static image repeating itself. Likely from frame 000000.



Results of 12bit test using mlv_rec

Frame 000000 has the top 1 cm of the image displaced a few pixels to the right but is otherwise OK.
Frame 000001 has the top 1cm corrupted with the balance of the image looking normal
Frame 000002 No Corruption and the image looks normal.
Frame 000003 On first glance looks normal but upon closer inspection it is in fact corrupted. It is in fact a blended image with the top 1 cm of the image being
                       from one image and the balance from another (statc image).
Frame 000004 No Corruption and the image looks normal.
Frame 000005 Like frame 000003 it is a blended image with the top 1 cm of the image being from one image and the balance
                       from another (static image).

All subsequent  pairs of frames i.e: 000006&000007, 000008&000009 etc. behave the same as frames 000004&000005.

What is actually happening with the 12bt mlv? It is similar but different to what is happening with the the 10bit mlv.  For the 12bit mlv, the first two frames 000000&000001 are corrupt. From there on in every odd numbered frame is corrupt (a static repeating image) whilst every even numbered frame is ok. I proved this by:

   - Deleting all of the odd numbered dng frames.
   - Imported the even numbered dng frames in PowerDirector (corrupt frame 000000 is included)
   - Exported to an 1920 x 1080p mpeg4 video.

File Links

The below links are extracts from the original 10 and 12bit mlv files and contain 24 frames each. Also I have add ed link to the 12 bit video generated from PowerDirector.

10bit mlv extract - https://www.dropbox.com/s/hx61s68c4mcjdxq/7d_10bit_extract.mlv?dl=0

12bit mlv extract - https://www.dropbox.com/s/kp31jkj4vsun6t9/7d_12bit_extract.mlv?dl=0

MPEG4 from 12bit (even numbered frames) - https://www.dropbox.com/s/zg4hpbiqxlponuk/7D_14bit_test.mp4?dl=0 


Hopefully this helps the devs get a step closer to resolving the 10/12bit raw for the 7D. 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on January 11, 2017, 12:35:14 PM
@ottoga:
7D is only confirmed (http://www.magiclantern.fm/forum/index.php?topic=9848.msg177514#msg177514) to produce "good" results with 10/12 bit when using crop/zoom mode. Otherwise you will get corrupted frames
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on January 11, 2017, 12:41:00 PM
Probably a beginner thought, im not even at novice lvl in coding, but it's like a synchronisation slip/clock error happens when you chop 2/4 bits from the incoming stream while recording 10/12 bit with 50D/7D/5D2
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on January 11, 2017, 01:36:46 PM
a1ex has already explained it: http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181#msg174181

If your camera has the "raw slurp" method implemented, everything will work. If it does not, then the solution is to implement it. That requires some reverse engineering. If slurp is not implemented on your camera, the fact that you get any good frames at all is just luck (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175902#msg175902).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 11, 2017, 01:38:49 PM
Quote from: Danne on January 11, 2017, 09:16:22 AM
I,d say run all footage with black level 2048(5D mark III). A1ex seems to agree, at least counting from yesterday https://bitbucket.org/hudson/magic-lantern/commits/e3ef8c75cf5de2e4018098d21ae6d1d3bcd183d5

Only applies to mlv_dump builds containing that change (such as the one from http://builds.magiclantern.fm/utilities.html ).

Older mlv_dump builds would require 2048 for 14 bpp, 2046 for 12 bpp and 2040 for 10 bpp.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on January 11, 2017, 03:23:52 PM
@A1ex

Did you saw my post  (http://www.magiclantern.fm/forum/index.php?topic=17795.msg177218#msg177218) about this issue in blown highlight when using mlv_dump for fix verticals stripes ?

It is a known issue ? Or maybe it is me who has not understood something.

When using MLVFS all is ok about stripes fix, even for blown highlight.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 11, 2017, 04:02:33 PM
Here is the latest mlv_dump for mac.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 11, 2017, 07:28:39 PM
@Danne @A1ex and Raw Devs.

Samples of 5d3 build Jan6 in 10/12/14 bits with MLV_DUMP with black fix at 2040, 2046, 2048 in all bit dephs.

Inside the folder I'd included MLV_DUMP and MLV.MO for check versions.

Let's go foward!

https://www.dropbox.com/sh/ruu0sclnxb7xk6m/AAAYRuRuLjlSAjz9lmOsL8dCa?dl=0

:P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 11, 2017, 08:45:13 PM
@dfort
Further 7D testing (croped) magiclantern-raw_video_10bit_12bit.2017Jan05.7D203
Croped mode generates proper dngs at 10/12/14 bits.
LiveView during recording (at 5x) goes to Canon View but sometimes locks up (no motion) in GrayScale.
10x zoom does not work.  It can be made to work if in LiveView at 5X yo enter ML menue and then return to LiveView.  Pressing zoom again will bring 10x, and then pressing again will go back to 1X.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on January 12, 2017, 06:37:15 PM
Is there a working 5d3 123 version with mlv 10,12,14bit and crop rec version available to try? I downloaded the one from experiments but it only had Mlv-Lite module for raw.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 12, 2017, 07:08:05 PM
mlv_rec not available yet, only with raw_rec available here: https://builds.magiclantern.fm/experiments.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 13, 2017, 12:48:39 AM
Black level issues should be fixed now; no more trickery should be needed with latest builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 13, 2017, 04:49:02 AM
@a1ex,
" no more trickery" means no more mlv dump black fixes?

What's next? Mpeg?

:)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 13, 2017, 08:21:48 AM
New builds on the Experimentals download page (https://builds.magiclantern.fm/experiments.html) are ready. I took down the builds I've been hosting. These cameras do not yet support CONFIG_EDMAC_RAW_SLURP but when someone puts up a pull request I'll try to compile a build for testing.






5D2.212
7D.203
50D.109
550D.109
500D.111
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Licaon_Kter on January 13, 2017, 08:55:25 AM
So close to merger, thanks for the continous development.  :o
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 13, 2017, 05:44:34 PM
Had a shoot today. First day with latest nightly and today with Jan 11 10-12bit build. External monitor did work at first (worked without any problem on Nightly), some shots were made then later the monitor only froze when connecting to the 5D3 with that build. Anyone experienced the same? Monitor was an Small HD DP6 5,6".

Otherways the build worked great!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on January 13, 2017, 06:41:13 PM
@dfort: i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Sganzerla on January 13, 2017, 10:21:58 PM
I'm going to post something that may be of interest, even if I'm not sure it is related to this area of the forum, or the 'post processing' one.

Using the experimental builds today - 5D MKIII 1.2.3 13th January - I tested the crop mode 3x3 720p and FPS override at 70fps (67fps) in 10/12/14bit modes.
Using raw2cdng software with "16bit CDNG maximized" option ticked, I opened the 3 files in Photoshop (ACR) and noticed that the blue cast in the shadows is not there anymore, but there are differences related to the magenta/green cast now, being 10bit files greener than 14bit files. I don't know if it is possible to see the exact difference from the image bellow, but there is a link to download these files if anyone is interested.

10bit TOP
12bit MIDDLE
14bit BELLOW

(http://i1063.photobucket.com/albums/t506/Sganzerla/cdng-differences_zps7mny8ail.jpg)

The download link of the 3 files is here:
https://goo.gl/YE4hcN (https://goo.gl/YE4hcN)

The difference is not solved with tint adjustment, unfortunately.

By the way, I don't know if people have tried this and seen before or not, but on a side note, related to software, MLV Mystic is producing noticeable more magenta images than raw2cdng from the same 14bit files too. So I don't know if it is something software related or code related.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 13, 2017, 10:28:04 PM
Any difference if you convert with latest mlv_dump from https://builds.magiclantern.fm/utilities.html ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 13, 2017, 11:40:53 PM
Quote from: Markus on January 12, 2017, 06:37:15 PM
Is there a working 5d3 123 version with mlv 10,12,14bit and crop rec version available to try? I downloaded the one from experiments but it only had Mlv-Lite module for raw.

Top of the Experimentals (https://builds.magiclantern.fm/experiments.html) download page. That build includes mlv_rec.

Quote from: ch_d on January 13, 2017, 06:41:13 PM
@dfort: i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)

You don't need to make a pull request to test it. I took down the builds I made for cameras that don't have CONFIG_EDMAC_RAW_SLURP which the 5D2 is one of them. @reddeercity made a very complete test of that camera so we have a good idea of what's working and what still needs to be done. If you want to do some additional testing I'd suggest getting in touch with him.

@g3gg0

There is an issue with raw_twk on the EOSM. When mlv_rec, mlv_snd, mlv_play and raw_twk are loaded and MLV Sound is turned on a "Threads failed to start" error appears on the screen and the camera must be shut down or it may even require a battery pull. This happens with any bit depth setting for picture and sound. If the raw_twk module is not loaded this will not happen.

(https://c1.staticflickr.com/1/333/32252532236_662daf5049.jpg)

I was unable to reproduce this on the 700D so this issue may be specific to the EOSM.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 14, 2017, 01:11:41 AM
yeah, already mentioned a few times.
the EOS M has very few threads available and some modules need threads for their work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Sganzerla on January 14, 2017, 01:23:55 AM
Quote from: a1ex on January 13, 2017, 10:28:04 PM
Any difference if you convert with latest mlv_dump from https://builds.magiclantern.fm/utilities.html ?

Thanks a1ex for the insight. I tought mlv_dump was a software for developers!
If I make it work, I'll come back with info. I think my English is not helping very much on this issue, but I'll try to find what is happening.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 14, 2017, 01:33:48 AM
mlv_dump is a reference implementation on how to read MLV files and how to extract the frames.
it is by far not the most powerful toy to play with MLV files, but it offers a few advanced features.

so when in doubt, use mlv_dump. if its output is unexpected then we can help :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 14, 2017, 01:46:15 AM
Mlv_dump, the more powerful tool imo. Strong features. Now working with 10/12/14 bits without the need to specify black/white level. And also:
- darkframe handling
- flatframe handling
- Possible to parallelize exporting processing
- and more...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 14, 2017, 03:50:55 AM
Quote from: g3gg0 on January 14, 2017, 01:11:41 AM
yeah, already mentioned a few times.
the EOS M has very few threads available and some modules need threads for their work.

Sorry, I searched the forum before posting but came up with the other definition of threads.

Quote from: g3gg0 on January 14, 2017, 01:33:48 AM
it is by far not the most powerful toy to play with MLV...

The definition of modesty.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on January 14, 2017, 04:35:29 AM
threads = tasks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 14, 2017, 06:15:45 AM
Quote from: ch_d on January 13, 2017, 06:41:13 PM
i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)
@ch_d no problem , I'll host the 5D2 10-12bit experimental build in my bitbucket account for the next 4-6 weeks
Or until 5D2 and older cams have liveview functional in 1:1 mode.
This build is the one I use in full mlv+audio in crop mode (2.61 crop) 2144x1076 23.976p @ 10bit + audio. (65MB/s write speed)
Please note 1:1 liveveiw is not working while recording raw bit reduction (corrupt frames) 14bit Ok .
With the build there the raw_twk.mo to play back 10-12bit in camera so it must be loaded with mlv_play & .mlv or .raw(mlv lite).
10bit_12bit_raw_twk.2016Dec01.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip) by chance if I can figure out the problem
with the liveview in bit reduction  ::) then I'll share my builds here.

If older cams e.g. 7D , 50D , 550D 500D user wish to try bit reductions with limited functionality ( I know 7D works in crop mode like the 5D2 but not sure about the rest)
I can post those cams builds also if requested , there will be from dec.1/2016 same as the one I posted for 5D2




Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on January 14, 2017, 01:28:08 PM
1:1 not working: That´s a pity. I don´t like the crop mode (because of FOV / DOF).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 14, 2017, 02:28:32 PM
@dfort:
yeah sorry, threads and tasks can be used as a synonym in our case, as there is no memory segregation/separation or anything between them.
threads are known as kind of light weight tasks in context of the parent task.
i should use task as the word canons os uses.


good to hear that you all like mlv_dump how it is today :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on January 14, 2017, 03:28:31 PM
For those who use mlv_dump and who like to do things simply, I recommend this tutorial (http://www.magiclantern.fm/forum/index.php?topic=10526.0)

It is easy also to make different styles of dng extraction commands in the folder "send to" by duplicating .cmd files.
It's interesting when we need to do some tests with news features.
(http://image.noelshack.com/fichiers/2017/02/1484403957-mlvdump.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 15, 2017, 03:12:07 PM
www.youtube.com/watch?v=__k0o0ycYk0

Test of 10 bit raw video recorded on Canon EOS 600D during daylight ISO 100.
Resolution: 1728x672, 23,97fps, ISO 100, raw_rec.mo
BUILD/dfort/: magiclantern-raw_video_10bit_12bit.2017Jan13.600D102
I recorded 23 raw video files. Three files were with a "green cast".
Command: exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dng didn't fix it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 15, 2017, 05:47:23 PM
Try the latest builds from the Experiments (https://builds.magiclantern.fm/experiments.html) download page. I believe there were some recent changes to the way black level is handled.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on January 15, 2017, 06:17:29 PM
@Sganzerla
as @g3gg0 said, mlv_dump is the reference, all other tools are written by "independent" coders. so, if other tools are in need of a fix, write them (as i am the coder of raw2cdng)

regards chmee
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 15, 2017, 08:30:35 PM
Can someone post the link, or a list with the commands lines of Mlv_dump?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 15, 2017, 08:39:42 PM
My examples of "green cast" - black level. Original and after fix
Shot with EOS 600D:
Resolution: 1728x672, 23,97fps, ISO 100, raw_rec.mo, magiclantern-raw_video_10bit_12bit.2017Jan13.600D102,
http://www.uschovna.cz/zasilka/LIVYCU4ETGKEGXFD-ZAK/
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on January 15, 2017, 08:40:33 PM
Quote from: budafilms on January 15, 2017, 08:30:35 PM
Can someone post the link, or a list with the commands lines of Mlv_dump?
If you run mlv_dump without any other options, it will print that for you.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 15, 2017, 08:53:46 PM
Usage: mlv_dump [-o output_file] [-rscd] [-l compression_level(0-9)] <inputfile>

Parameters:
-o output_file      set the filename to write into
-v                  verbose output
--batch             output message suitable for batch processing

-- DNG output --
--dng               output frames into separate .dng files. set prefix with -o
--no-cs             no chroma smoothing (default)
--cs2x2             2x2 chroma smoothing
--cs3x3             3x3 chroma smoothing
--cs5x5             5x5 chroma smoothing
--no-fixcp          do not fix cold pixels
--fixcp2            fix non-static (moving) cold pixels (slow)
--no-stripes        do not fix vertical stripes in highlights

-- RAW output --
-r                  output into a legacy raw file for e.g. raw2dng

-- MLV output --
-b bits             convert image data to given bit depth per channel (1-16)
-z bits             zero the lowest bits, so we have only specified number of b
its containing data (1-16) (improves compression rate)
-f frames           frames to save. e.g. '12' saves the first 12 frames, '12-40
' saves frames 12 to 40.
-A fpsx1000         Alter the video file's FPS metadata
-x                  build xref file (indexing)
-m                  write only metadata, no audio or video frames
-n                  write no metadata, only audio and video frames

-- Image manipulation --
-a                  average all frames in <inputfile> and output a single-frame MLV from it
--avg-vertical      [DARKFRAME ONLY] average the resulting frame in vertical direction, so we will extract vertical banding
--avg-horizontal    [DARKFRAME ONLY] average the resulting frame in horizontaldirection, so we will extract horizontal banding
-s mlv_file         subtract the reference frame in given file from every single frame during processing
-t mlv_file         use the reference frame in given file as flat field (gain correction)

-- Processing --
-e                  delta-encode frames to improve compression, but lose random
access capabilities
-X type             extract only block type
-I mlv_file         inject data from given MLV file right after MLVI header
-c                  (re-)compress video and audio frames using LZMA (set bpp to
16 to improve compression rate)
-d                  decompress compressed video and audio frames using LZMA
-l level            set compression level from 0=fastest to 9=best compression

-- bugfixes --
--black-fix=value   set black level to <value> (fix green/magenta cast). if no
value given, it will be set to 2048.
--white-fix=value   set white level to <value>. if no value given, it will be s
et to 15000.
--fix-bug=id        fix some special bugs. *only* to be used if given instructi
on by developers.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 16, 2017, 01:56:31 AM
Wonderfull guys!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ansiivan on January 17, 2017, 01:20:33 AM
Hi. Why regular chamber mov, much brighter raw vide? The settings are the same.
Canon 1100d,  magiclantern-raw_video_10bit_12bit.2017Jan13.1100D105.zip
Fps 25, iso1600.
mov
https://www.youtube.com/watch?v=2o4Umf92yLM

Fps 25, iso1600.
raw video
https://www.youtube.com/watch?v=hOEtEuKAdLs
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on January 17, 2017, 12:32:36 PM
Hi guys,

on Saturday, I've sucesfully shot 2TB of MLV's in 10bit. From 1400 filex, approx 150 had wrong black level. Since calling mlv_dump 150 times and then sorting all the files would be too complicated, I've wrote little windows batch script. I am not a programmer so finding the first file in a folder and even the iteration is very very messy. But works. :)


@echo off
setlocal EnableDelayedExpansion
set mypath=%~dp0
set workingDir=E:\footage\mlv\
set fixDir=E:\footage\cdng-fixed\
set moveDir=E:\footage\mlv-processed\


E:
cd %workingDir%
for /r %%i in (*) do (
set file=%%i

!mypath!\mlv_dump.exe --black-fix=2040 --dng "!file!"

for %%A in ("!file!") do (
set foldername=%%~dpA
set filename=%%~nxA
)
echo !filename!

cd "!fixDir!"
md "!filename!"
cd "!filename!"

for /r "%workingDir%" %%f in (*.dng) do move "%%f" >nul
for /r "%workingDir%" %%f in (*.wav) do move "%%f" >nul

move  "!file!" "!moveDir!"
)
D:
cd %mypath%
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on January 17, 2017, 03:04:54 PM
BTW, may I process *all* files with --black-fix=2040, even those which have correct value? If I want to process only those wrong ones, I need to extract all and visually check.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 17, 2017, 05:44:17 PM
On 5D3 and probably all other recent cameras, latest mlv_dump requires --black-fix=2048, regardless of ML build (since this post) (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178162#msg178162).

With latest ML bulid from the Experiments page AND latest mlv_dump, there should be no need to change the black level (since this post) (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178319#msg178319).

Checklist:
- with 10-bit files, the black level from MLV (visible with mlv_dump -v) should be 128 (127 is wrong)
- with 14-bit files, the black level should be close to 2048 (small variations are OK)
- the black level from the DNG (always upconverted to 14 bits) should be close to 2048 (e.g. 2050 is fine, 2032 is wrong)
- there should be no color difference between 10 and 14 bit files in shadows

If any of the above is not true for you with the latest experimental build and latest mlv_dump, please report.

For authors of other postprocessing tools: to avoid color casts at lower bit depths, conversion to a higher bit depth must be done by adding 0.5 LSB to the input signal (explained here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177802#msg177802)). Black level scaling requires no trickery (e.g. 128 -> 2048).

Relevant commits: e3ef8c7 (https://bitbucket.org/hudson/magic-lantern/commits/e3ef8c75cf5de2e4018098d21ae6d1d3bcd183d5) and a31228c (https://bitbucket.org/hudson/magic-lantern/commits/a31228c7b7bed6805e3dc5e9e0b894ed75b033b4).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vstrglv on January 17, 2017, 05:53:10 PM
Build  magiclantern-crop3x.2017Jan13.5D3113 (2017-01-12 23:35) – crop+10bit,12bit. MLV raw.
DNG conversion raw2cdng 1.7.9 and  mvl_dump (2017-01-13 00:55).
DNG for crop(1x1)_Xbit_ raw2cdng and DNG for crop(1x1)_Xbit_ mlv_dump look very similar (except  green colour cast for 10bit raw2cdng)  in ACR if Noise Reduction (Luminance) is set to zero. But if Noise Reduction is increased its effect is more noticeable  for mlv_dump convertion.
X is 10bit, 12bit or 14bit. This effect takes place for normal mode (not crop) too.
https://www.dropbox.com/s/q6pj4kspt8l794t/M14-1338_10b_crop_mvl_dump.dng?dl=0
https://www.dropbox.com/s/i6vpllko75d43wl/M14-1338_10b_crop_raw2cdng.dng?dl=0

(https://c1.staticflickr.com/1/737/32218454142_cc0b3d211c_o.jpg) (https://flic.kr/p/R62ZWE)M14-1338_10b_crop_0NR_raw2cdng (https://flic.kr/p/R62ZWE) by Vladimir Strigalev (https://www.flickr.com/photos/141116628@N06/), on Flickr
(https://c1.staticflickr.com/1/330/32329871916_1d88800d55_o.jpg) (https://flic.kr/p/RfT3yL)M14-1338_10b_crop_0NR_mvl_dump (https://flic.kr/p/RfT3yL) by Vladimir Strigalev (https://www.flickr.com/photos/141116628@N06/), on Flickr
(https://c1.staticflickr.com/1/450/32218698302_b7129a1bcc_o.jpg) (https://flic.kr/p/R64fwj)M14-1338_10b_crop_30NR_raw2cdng (https://flic.kr/p/R64fwj) by Vladimir Strigalev (https://www.flickr.com/photos/141116628@N06/), on Flickr
(https://c1.staticflickr.com/1/277/32368983145_3e9e3e18df_o.jpg) (https://flic.kr/p/RjkuYi)M14-1338_10b_crop_30NR_mvl_dump (https://flic.kr/p/RjkuYi) by Vladimir Strigalev (https://www.flickr.com/photos/141116628@N06/), on Flickr
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on January 17, 2017, 07:09:42 PM
Quote from: a1ex on January 17, 2017, 05:44:17 PM
On 5D3 and probably all other recent cameras, latest mlv_dump requires --black-fix=2048, regardless of ML build (since this post) (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178162#msg178162).

With latest ML bulid from the Experiments page AND latest mlv_dump, there should be no need to change the black level (since this post) (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178319#msg178319).

Great! However I still find batch calling mlv_dump much better then mlv-fs. Is there any way how to make real cdng from mlv_dump? So that Resolve could read audio and correct fps? Or this wouldn't work because mlv_dump is not indexing the files?

Thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 18, 2017, 02:10:40 AM
Resolve read Cdng with audio from mlvfs as one file with correct level in real time on PC's and Mac's .
batch calling on command is very slow plus you end up with lots of dng's that fill up hard drive space .
MLVFS doesn't do that it produces Virtual Cdng so no hard drive killer here , you even read off the CF or SD cards
depending on your computer hardware .
FYI: mlv_dump produces compliance dng that work in resolve or at least used to that could be broken thou with bit reduction
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 18, 2017, 02:48:08 AM
QuoteGreat! However I still find batch calling mlv_dump much better then mlv-fs. Is there any way how to make real cdng from mlv_dump?
You can process cdng files with this
http://www.magiclantern.fm/forum/index.php?topic=15108.msg146822#msg146822

The program works with a mlv_dump version which has code mixed in from a specific ml-dng branch version. Code updated to latest 10/12/14 bit the same as latest mlv_dump(g3gg0/a1ex). Works with resolve(audio embedding).
When installed head over to the mlv_dump menu (m) from the main menu and you can do setting changes from there such as black/white level etc.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 19, 2017, 12:24:50 PM
Using 13th of Jan build and the frame freezes in black and white live view while shooting raw_rec with 5x. Any others experiencing the same?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tracerman on January 19, 2017, 01:11:37 PM
@dfort

Hi, I'm up for testing 10/12 bit mlv_rec on 550D if you like (I don't know how to make a pull request either :/).

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 19, 2017, 01:19:19 PM
Is this happening with or without raw_twk enabled? Try with raw_twk disabled.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 19, 2017, 03:53:58 PM
There 's only a windows version of MLV_dump available on the magic lantern downloads page:
https://builds.magiclantern.fm/utilities.html (https://builds.magiclantern.fm/utilities.html)

Can somebody add newest version of MLV_dump for osX ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 19, 2017, 04:16:46 PM
mlv_dump for mac
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 19, 2017, 10:36:42 PM
Quote from: Danne on January 19, 2017, 01:19:19 PM
Is this happening with or without raw_twk enabled? Try with raw_twk disabled.
Still happening with raw_twk disabled. Guess its something with the build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 20, 2017, 08:33:57 AM
Quote from: Tracerman on January 19, 2017, 01:11:37 PM
@dfort

Hi, I'm up for testing 10/12 bit mlv_rec on 550D if you like (I don't know how to make a pull request either :/).

Nothing new for the 550D yet and we know what needs to be done before we're ready for any further testing.


Sent from my iPad using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on January 20, 2017, 02:01:04 PM
i dont see 10 bit for 50D   ???
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 20, 2017, 06:49:04 PM
mlv_dump for mac updated
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump

fixes here(dark/flatframe, mlv_lite)
https://bitbucket.org/hudson/magic-lantern/commits/89daf093c81fbe86be3ae27e4acce8fa79bf91f3
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 20, 2017, 07:31:31 PM
Thanks SO much for sharing this @Danne and for the quick fixes @a1ex!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tracerman on January 21, 2017, 12:30:33 PM
@dfort

Thanks for your reply, feel free to let me know when/if you need testers for 550D.

Cheers
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on January 21, 2017, 04:44:04 PM
Quote from: CITY-U1001 on January 20, 2017, 02:01:04 PM
i dont see 10 bit for 50D   ???
50D is not supported (yet).
(Same as 5dmk2 as they don't have CONFIG_EDMAC_RAW_SLURP ?)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bobbyshots2 on January 21, 2017, 07:14:30 PM
I want to try and test the t2i (550d) with 10bit - 12bit raw. If anybody has a link to  the module they could send me or the full ml download. that would be great.!
Thanks in advance!
One Love
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on January 21, 2017, 07:35:51 PM
Ok, mlv_dump works to get me DNG.
I can colour correct usingbut how to get them into Ppro?  If I save as JPEG from LR, it complains that bit depth is not supported.

Is there a way to get 10bit scaled to 14bit depth?

Found that DNG works very nice in Resolve.
However LR becomes useless in this flow, since its corrections are not recognized when DNG is input to Resolve.
One can export from LR into jpeg but that limits Ppro or Resolve's abilities.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 21, 2017, 08:05:18 PM
Looks like it is worth repeating that the following cameras need to get the CONFIG_EDMAC_RAW_SLURP treatment before it is worth doing any further tests with them.







5D2.212-Digic 4
7D.203-Dual Digic 4
50D.109-Digic 4
500D.111-Digic 4
550D.109-Digic 4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 22, 2017, 02:29:02 AM
Quote from: justinbacle on January 21, 2017, 04:44:04 PM
50D is not supported (yet).
(Same as 5dmk2 as they don't have CONFIG_EDMAC_RAW_SLURP ?)
Not really true , not sure about 50D but the 5D2 work in Crop mode 100% (10-12bit) with Audio , overlays  in MLV full  see the Proof (http://www.magiclantern.fm/forum/index.php?topic=11205.msg177454#msg177454)
Even thought 1:1 liveview products some corruption in 10bit but 12bit can be used if you crop out
the top 50-100 line do to a color noise bar see here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175483#msg175483) look at the 3rd image I posted .

I can't say that the 50D or 550 (T2i) will have the same level of functionality in crop mode as the 5D2 but as @dfort said
Quote from: dfort on January 21, 2017, 08:05:18 PM
..... need to get the CONFIG_EDMAC_RAW_SLURP treatment before it is worth doing any further tests with them.

We do need more user's who can help figure the Digic 4 liveview issue (5d2,50d,7d,500,550) so I will provide the link to those builds in hope more development can be made .
My reasoning is if users have at least a limited functioning version of  raw video bit reduction , this hopefully will give more incentive to push development on the digic 4.
link to my bitbucket downloads for 10-12bit reductions 5d2,50d,7d,500,550 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/)



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Nikolas on January 22, 2017, 10:53:56 AM
Thanks to developers for latest build for 1100d with 10/12bit raw video (2017-01-20)! Compared with the previous version (december 2016), record of raw video works very stable. 10/12 bit really is supported. in the previous build had a lot of colored dots, now they have disappeared.

But in the latest build, strong processing gives vertical banding (pattern noise?). Like i said, in the previous version were only colored dots, vertical banding been absent. Image quality with hard processing is very important for me (I use raw video for astrophotography).

I tries to use different combinations of recording settings and to use subtraction dark frames, but it not work.

Dear developers, Is it possible to fix in the next build? Or how i can make a correction of these bands?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 22, 2017, 12:11:59 PM
Interesting.
How are you subtracting darkframes? Can you upload a darkframe and a clip with banding?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Greg on January 22, 2017, 05:24:29 PM
#ifdef CONFIG_80D_101
#define DEFAULT_RAW_BUFFER MEM(0x66B0 + 0x54)
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Sganzerla on January 23, 2017, 12:57:22 AM
@a1ex
@g3gg0
@GutterPump
@chmee
@RenatoPhoto


Regarding my post [b#1081]]#1081 (//http://)[/b] I finally (after some struggle) learnt how to process .MLV with mlv_dump and could see that there is no color cast between 14bit and 10bit files using this software.

Thanks for all of your input!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 24, 2017, 05:34:38 AM
The Bit depth in raw_rec and mlv_rec are slightly different:

(https://c1.staticflickr.com/1/778/32372240161_023bc27e4b.jpg)

(https://c1.staticflickr.com/1/579/32372240041_24667767bd.jpg)

Note that mlv_rec says "bpp" for bits per pixel.

In addition, the up/down buttons go the opposite direction in raw_rec so I made a small change and put in a pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/808/conform-raw_rec-bit-depth-menu-to-match/diff).

(https://c1.staticflickr.com/1/454/32115323320_0a7413a16e.jpg)

(https://c1.staticflickr.com/1/521/32115323440_abc5798e4f.jpg)

Hope you like the small improvements and hope this doesn't break anything--tested and working fine over here.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on January 24, 2017, 09:04:48 AM
Canon 50D_01-14-2017 109 Firmware Test

All Processed w/ CMD and MLVFS:

NOTES:
     
             RAW REC 1.1 works in 14-Bit Full Sensor Readout and Canon Preview.

             MLV 2.0 works in 14-bit Full Sensor and 5X Zoom w/Canon Preview. However, it is super close to working in 12-bit.



- - - - - - - - - - - - - - - RAW RECORD 1.1 LITE - - - - - - - - - - - - - - -

10-Bit - Full Sensor Readout - - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Repeating Frame; every other frame. Top 1/3 of the image is an inverted data sector / bad write.

10-Bit 5X Zoom - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Purple Frames. Repeating Frame; every other frame. Top 1/3 of the image is separated from bottom 2/3. Bottom 2/3 features a "wiggly zoom-in" distortion appearing in a timed pattern.


- - - - - - - - - - - - - - - RAW RECORD 1.1 LITE

** Best Results w/ 5X Zoom:

12-Bit Full Sensor Readout - - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Repeating Frame; every other frame. Top 1/6 of the image is an inverted data sector / bad write (still repeats image).

12-Bit 5X Zoom - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W do not work - -
   RESULTS: Purple Frames. Repeating Frame; every other frame. Top 1/3 of the image is separated from bottom 2/3.


- - - - - - - - - - - - - - - RAW RECORD 1.1 LITE

14-Bit Full Sensor Readout - - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Full Sensor Readout is great.

14-Bit 5X Zoom - - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS:  Purple Frames. Repeating Frame; every other frame. Top 1/3 of the image is separated from bottom 2/3. Bottom 2/3 features a "wiggly zoom-in" distortion appearing in a timed pattern.



- - - - - - - - - - - - - - - MLV 2.0 - - - - - - - - - - - - - - -


14-Bit Full Sensor Readout - - MLV 2.0 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Full Sensor Readout is great.

14-Bit 5X Zoom - - MLV 2.0 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: 5X Zoom Results Record Great. However, FRAMING IS OFF.


- - - - - - - - - - - - - - - MLV 2.0

12-Bit Full Sensor Readout - - MLV 2.0 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Top 1/8 of the image is a repeating inverted data sector / bad write every other frame.

12-Bit 5X Zoom - MLV 2.0 - - Canon Preview Only - Hacked / B&W do not work - -
   RESULTS: Top 1/3 of frame appears to be separated from both 2/3 and repeating every other frame. NO PURPLE FRAMES.


- - - - - - - - - - - - - - - MLV 2.0

10-Bit Full Sensor Readout - - MLV 2.0 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Repeating Frame; every other frame. Top 1/3 of the image is an inverted data sector / bad write.

10-Bit 5X Zoom - MLV 2.0 - - Canon Preview Only - Hacked / B&W do not work - -
   RESULTS: Top 1/3 of frame appears to be separated from both 2/3. NO PURPLE FRAMES.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 24, 2017, 09:13:03 AM
Quote from: dfort on January 21, 2017, 08:05:18 PM
Looks like it is worth repeating that the following cameras need to get the CONFIG_EDMAC_RAW_SLURP treatment before it is worth doing any further tests with them.







5D2.212-Digic 4
7D.203-Dual Digic 4
50D.109-Digic 4
500D.111-Digic 4
550D.109-Digic 4

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lars Steenhoff on January 24, 2017, 11:05:39 AM
For any tests done I think its worth mentioning wich memory card is used to isolate any card speed issues.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 24, 2017, 12:23:25 PM
... for any *speed* tests (hint, hint) (http://www.magiclantern.fm/forum/index.php?topic=17091)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on January 24, 2017, 06:24:38 PM
The internal temp reading on the 50D is reading, "Calm and collected."

Komputerbay 128 1066x, by the way.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bobbyshots2 on January 28, 2017, 07:37:29 PM
Iv been testing the  10-12 bit on the 550d (t2i)
I get good results on the 12 bit using mlv 2.0
on the 10 bit the top part of the frame is jittery or "repeating" as i see someother user posting
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hjfilmspeed on January 29, 2017, 05:29:34 AM
I am super impressed by the Jan 13 experimental build the Incorporates the 10bit and crop mode. 10bit is a dream to work with. If we could just get a better preview for high resolution. But 10 bit has been working flawlessly for me. So many possibilities here for higher fps and resolution bravo.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tracerman on January 29, 2017, 11:00:20 AM
@bobbyshots2 what build did you use ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on January 29, 2017, 04:32:16 PM
Quote from: a1ex on January 24, 2017, 09:13:03 AM
Quote from: dfort
Looks like it is worth repeating that the following cameras need to get the CONFIG_EDMAC_RAW_SLURP treatment before it is worth doing any further tests with them.

5D2.212   -   Digic 4
7D.203   -   Dual Digic 4
50D.109   -   Digic 4
500D.111   -   Digic 4
550D.109   -   Digic 4

Sorry for fullquote. You highlighted 50D. Work is done for other cams?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 29, 2017, 04:40:04 PM
Nope.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bobbyshots2 on January 29, 2017, 10:51:54 PM
I am using the download from the link that reddeercity posted previously.
the build say 01-12-2017
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads

* i notice now that there is a new date 01-14-2017 that i will have to try.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on January 29, 2017, 11:00:25 PM
@a1ex and dfort: 7D here but no clue. Not that fluent in code. Directions, please!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 29, 2017, 11:08:36 PM
Unsuccessful on 5D2 after trying for a couple of evenings...

For new models, it's pretty straightforward if one looks at the commits. The 1200D is probably the only new model not yet ported. Old models have a very different codebase for LiveView (though the hardware is likely to work in the same way).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 30, 2017, 07:42:45 PM
Quote from: Walter Schulz on January 29, 2017, 11:00:25 PM
@a1ex and dfort: 7D here but no clue. Not that fluent in code. Directions, please!

Me neither! Fluent in code that is. I helped a bit with the 700D but only because it is very similar to the EOSM which dmilligan previously figured out. Those older models are proving very difficult to crack. There's an 1100D build on the Experiments (https://builds.magiclantern.fm/experiments.html) download page but I haven't seen any reports on how well it is working. Though looking that camera probably won't get you very far with the 7D or 5D2 because it is a slightly newer model.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 31, 2017, 02:11:04 AM
Quote from: a1ex on January 29, 2017, 11:08:36 PM
Unsuccessful on 5D2 after trying for a couple of evenings...
Old models have a very different codebase for LiveView (though the hardware is likely to work in the same way).
Sounds encouraging  :)

At the risk of not knowing anything about liveview code (over my head) is there something my 5d2 or myself can help with ?
I can compile , I understand some of the code but I get easily confuse at times but willing to try . 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 04, 2017, 06:23:12 AM
Ok doing some research on the liveview problem on the 5D2 in 1:1
Come across wiki Register_Map (http://magiclantern.wikia.com/wiki/Register_Map) found Liveview Registers stuff .

Quote0xC0F140C0 does luma scaling
0xC0F140C4 does chroma scaling
0xC0F140C8 sets the bmp overlay mode
0xC0F140E8 [s32] pixels to skip - causes distorted image
0xC0F140F0 [s32] pixels to skip - causes distorted image
0xC0F37014 LV raw type (see lv_af_raw, lv_set_raw) on DIGIC5 (DIGIC4 uses PACK32_ISEL)
0xC0F371FC unpacking mode (see PACK16_MODE) for the input module that feeds image data to lossless encoder

0xC0F37014 LV raw type seems to point to  "PACK32_ISEL" for digic4 (5d2) (from ML code I seen this is where digic5 cam gets configured with "LV raw type" if I'm not mistaken

Quote0xC0F08090 - PACK32 Control register
0xC0F08094 - PACK32_MODE

0xC0F08114 - PACK32_ISEL (0-15)
Edit: Sorry posted wrong address  ::)
I see the variable (0-15) now
nice to know what those are .

How's can this be configured ?

Like this for the "PACK16_MODE" ?

Quote0xC0F08120 - PACK16 Control register
0xC0F08124 - PACK16_MODE (mask 0x131)
             From ProcessPathForFurikake (raw_twk):
             -------x --xx----
                  111 (0x130) = 16-bit output
                  011 (0x030) = 14-bit output
                  010 (0x020) = 14-bit output
                  001 (0x010) = 12-bit output
                  000 (0x000) = 10-bit output

I hope this is on the right track , What do the developers think ?
If this is correct how would I processed from here.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 04, 2017, 10:37:38 AM
Quote from: reddeercity on February 04, 2017, 06:23:12 AM
0xC0F08114 - PACK32_ISEL (0-15)

Just look it up in raw.c and earlier in this thread (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174177#msg174177) ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on February 04, 2017, 02:18:18 PM
When recording on these new experimental builds the raw histogram switches over to the standard histogram on record. Have I missed some simple setting or is it i bug?

5d3 113 crop + 10-14bit experimental version.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 04, 2017, 02:26:17 PM
All raw overlays are currently hardcoded for 14-bit data. See this post (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177531#msg177531) for more details.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on February 05, 2017, 06:29:08 AM
raw_twk.mo issue with 70D

For 10 bit raw, if you want to have liveview playback working
you need to copy raw_twk.mo to your ML->modules folder on your card.
*** It seems to work for one pass, but 2nd clip it hangs your camera ***
*** Looks like that its not initializing properly for 2nd clip ***

here is a link for the raw_twk.mo I used.
https://www.dropbox.com/sh/arb43xa6svvjljr/AACyi51IrJfMGs3ZQ8EC_6hOa?dl=0

The modules that need to be enabled for it to work include
file_man
mlv_play
mlv_rec
mlv_send
raw_rec
raw_twk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zachnfine on February 05, 2017, 07:01:48 AM
This is likely a question that was answered earlier in the thread - but I searched around and didn't find it. With this new feature can one shoot 10-bit on the 5D3 at higher than 1080p resolution in full-frame mode (i.e. not crop-mode)?

I've been doing fine recording 1080p23.98 to my Komputerbay 128Gb cards (1 1050x, 1 1066x) at 14-bit, which requires 82.97Mb/s. But if 10-bit is possible, then 2272x1278 @ 23.98 would be nearly the same bitrate at 82.99Mb/s - and that might be a perceptible increase in detail, even if downscaled to 1080p in post, or at least it'd leave a little extra headroom for stabilization. And if 10-bit drops too much usable color information, 12-bit at 2064x1161 23.98 is 82.19Mb/s.

Or maybe the card's capable of more and I'm aiming low with that bitrate, I haven't benchmarked. And maybe card spanning would be a help. It'd be fun to see how high a resolution can be recorded at 23.98.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 05, 2017, 07:52:20 AM
Quote from: a1ex on February 04, 2017, 10:37:38 AM
Just look it up in raw.c and earlier in this thread (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174177#msg174177) ;)
Thanks @a1ex
Ok I think I got my head around this a little better ,
raw.c- line160 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/src/48bb172d28daa6e82247191f9cef94bda0a07217/src/raw.c?fileviewer=file-view-default#raw.c-160)
Quote#ifdef CONFIG_DIGIC_V
#define RAW_TYPE_REGISTER 0xC0F37014
#else
#define RAW_TYPE_REGISTER 0xC0F08114    /* PACK32_ISEL */
#endif
I see "0xC0F08114" etc.... stuff , question;  with the "#ifdef CONFIG_DIGIC_V" then it has #else , does (#else) mean Digic4
or would it make a difference to call "#ifdef CONFIG_DIGIC_IV" or is that a way of writing less code ?

As I see how other newer cams are configured does there need to be a "#ifdef CONFIG_xxx" for the 5D2 ?
maybe like this ?
#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
#endif

From the Register Map "PACK32_ISEL" has  1-15 selectable variables
Would be right to put that variable in brackets ?
Or does "PACK32_ISEL" need to "#define" at all ?
Not sure how to write the Code/Syntax
Close you think , or way off base here .
Edit:
Because there 15 different variables does that mean the only way to figure it out is by trial & error ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kzv on February 05, 2017, 06:21:41 PM
Hello!

First time posting, so thank you for all your great work on Magic Lantern and especially this experimental version!

Been testing it and works great except one thing -in last build (and some previous in 2016) there is the problem with the ISO table. When you change ISO during recording, it wont hide.

Here is the picture: https://drive.google.com/file/d/0Bw9gcCutoLPQenBkMGJ4SUdxUjQ/view?usp=sharing

Can you please have a look at this bug? I would be very grateful.

Thank you very much in advance!

Chris
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hyalinejim on February 06, 2017, 03:01:34 AM
Yes, this happened to me last night when I was shooting a performance  (5D3 1.1.3 10bit)

While shooting, pressing the physical ISO button brings up the ISO overlay. I use the shutter wheel to change ISO. But then the overlay stays on screen until the camera stops recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on February 06, 2017, 09:19:07 AM
You can push rate button and use joystick to change iso.

Quote from: reddeercity on February 05, 2017, 07:52:20 AM
Thanks @a1ex
Ok I think I got my head around this a little better ,
raw.c- line160 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/src/48bb172d28daa6e82247191f9cef94bda0a07217/src/raw.c?fileviewer=file-view-default#raw.c-160)I see "0xC0F08114" etc.... stuff , question;  with the "#ifdef CONFIG_DIGIC_V" then it has #else , does (#else) mean Digic4
or would it make a difference to call "#ifdef CONFIG_DIGIC_IV" or is that a way of writing less code ?

As I see how other newer cams are configured does there need to be a "#ifdef CONFIG_xxx" for the 5D2 ?
maybe like this ?
#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
#endif

From the Register Map "PACK32_ISEL" has  1-15 selectable variables
Would be right to put that variable in brackets ?
Or does "PACK32_ISEL" need to "#define" at all ?
Not sure how to write the Code/Syntax
Close you think , or way off base here .
Edit:
Because there 15 different variables does that mean the only way to figure it out is by trial & error ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on February 06, 2017, 04:34:21 PM
@reddeercity

Are you looking for the way to twick Canon zoom for less magnification than 3x?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on February 06, 2017, 05:13:49 PM
@budafilms I think reddeercity is trying to stop the live view from freezing on 5D2 when its recording(I always have that problem so I think its that)
@reddeercity have you had wrong(0 probably) black level in crop mode on your 5D2? it seems to be broken in 10bit-12bit branch since December, even shooting 14 bit, I keep getting a pink cast in crop mode no matter what settings or build (does MLVFS have a black level fix?)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 07, 2017, 06:16:04 AM
Quote from: budafilms on February 06, 2017, 04:34:21 PM
@reddeercity Are you looking for the way to twick Canon zoom for less magnification than 3x?
No but that that would be a great dream  :)
Quote from: Ilia3101 on February 06, 2017, 05:13:49 PM
@budafilms I think reddeercity is trying to stop the live view from freezing on 5D2 when its recording(I always have that problem so I think its that)
Correct ! in 1:1 plus stop frame corruption but if liveview works properly hopefully that will fix that.

Quote from: Ilia3101 on February 06, 2017, 05:13:49 PM
@reddeercity have you had wrong(0 probably) black level in crop mode on your 5D2?
it seems to be broken in 10bit-12bit branch since December, even shooting 14 bit,
I keep getting a pink cast in crop mode no matter what settings or build (does MLVFS have a black level fix?)
No , are you using full mlv or mlv lite (.raw) ?
The video I posted was crop 10bit 2.1K mlv+audio (Full)
I need Audio when I record raw , I have been using just full mlv
No , MLVFS does not are you on windows or Mac?
I Thing if I'm not mistaken MLVProducer (http://www.magiclantern.fm/forum/index.php?topic=15271.msg148538#msg148538) has W/B raw levels that can be adjusted on the fly in realtime "ACR" like  :D
Thou it's a windows base program it can be run under Wine on OSX ( I tried it on my MacPro4.1 on Yosemite)  works nice

The black level in crop has been like this for some time , it's has something to do with the mlv lite I think , not sure thou.
1:1 thou is ok in 10,12,14bit I'm sure , just not crop. If you read the mlv lite source code , there's some notes about it  I think.
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 07, 2017, 06:49:43 AM
After consulting with my IT friend asking for some basic help (I'm sure he's was laughing at me )
showing him my code sample , he return with a long email explaining why I should not try to code  :P :P
Will in the end he did give me a sample to work on
#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
if PACK_ISEL (3))
          {
           do something
           Return a valve
           }
#endif


this look better ? it's a work in process

I'm still learning ,  :)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Licaon_Kter on February 07, 2017, 08:00:37 AM
if PACK_ISEL (3))

You're missing a ( after if I guess
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KelvinK on February 07, 2017, 01:35:31 PM
There's a very annoying bug in experimental build (was there since very first versions), at least for 6D: in LV when the play videos, it can play 2-3 then screen turns black. To get back GUI you've to press trash-bin button.

Would be great if it can be fixed.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on February 07, 2017, 02:48:53 PM
Yes this bug is confirmed.  I have reported it before for 5D3.
Video playback is limited to about 3 videos, probably a problem in raw_twk module.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KelvinK on February 07, 2017, 03:23:29 PM
I don't think so. If you don't load raw_twk and record 14bit only, playback has same issue with 14b files.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 08, 2017, 01:55:04 AM
Quote from: Licaon_Kter on February 07, 2017, 08:00:37 AM
if PACK_ISEL (3))
You're missing a ( after if I guess
Ok Thanks
So like this
#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
if ( PACK_ISEL (3))
          {
           do something
           Return a valve
           }
#endif


Little closer , still need to find more on PACK32_ISEL stuff.
I haven tried to compile anything yet ...... soon , stay tune
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on February 08, 2017, 04:06:55 AM
What exactly are you trying to do? The code you have posted is just nonsense (I'm not trying to be mean).

PACK32 is the name Canon has given to an image processing hardware module in the camera. PACK32_ISEL is the name Canon has given to a register (https://en.wikipedia.org/wiki/Hardware_register) for this module that controls its behavior. a1ex has already given an explanation of what is known about what this module does and what this particular register probably controls here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174177#msg174177). Until you are at least to the point of thoroughly and completely understanding everything that is said in that message, there's no point in trying to write any code.

You can't just pick up a violin for the first time and expect to be able to play the Brahms violin concerto. If you really want to contribute and learn, you will have to start with something simpler and learn more about programming in general and ML specifically (by, for example, playing with Lua or doing an easy coding task).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 08, 2017, 04:49:05 AM
Quote from: dmilligan on February 08, 2017, 04:06:55 AM
What exactly are you trying to do? The code you have posted is just nonsense (I'm not trying to be mean).
No I understand,
it's big learning curve there's a lot to take in and there is a lot that's over my head .
I'm trying to get liveview in 1:1 with bit reduction in work with out liveview freezeing up and not have it produce alternating corrupt frames .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 08, 2017, 05:28:39 AM
@reddeercity - I don't know much about coding or reverse engineering or playing the violin but one valuable lesson I learned from dmilligan is to look at the code. To get this 10bit/12bit thing working you need to figure out how to get CONFIG_EDMAC_RAW_SLURP working. Look at the internals.h file for any of the cameras that are supported and you'll see this:

internals.h
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#define CONFIG_EDMAC_RAW_SLURP


Ah if it were only as easy as adding that one line, but there's more to it.

Currently the 5D2 uses the LiveView raw buffer:

raw.c
/**
* LiveView raw buffer address
* To find it, call("lv_save_raw") and look for an EDMAC channel that becomes active (Debug menu)
**/

#if defined(CONFIG_5D2) || defined(CONFIG_50D)
#define RAW_LV_EDMAC 0xC0F04508
#endif


Cameras that have CONFIG_EDMAC_RAW_SLURP use the default raw buffer instead of the LiveView raw buffer.

raw.c
#ifdef CONFIG_EDMAC_RAW_SLURP
/* undefine so we don't use it by mistake */
#undef RAW_LV_EDMAC

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
*
* raw_buffer = get_raw_buffer()
* sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
* ...
* dump_file(filename, raw_buffer, 7*something...)
*/

#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif


So all you're looking for is that one hexadecimal number to unlock this feature. The question is how do you find it? Read the code, it tells you. Now the hard part is following those instructions and come up with the right address.

Back to the violin analogy, how do you get to Carnegie Hall? Practice.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 08, 2017, 06:01:27 AM
@dfort Thanks ! :D
I read so much code my head is spinning and of course I when down the wrong rabbit hole .
It all make sense now , I kind of thought that's what I being missing that CONFIG_EDMAC_RAW_SLURP  RAW_BUFFER address stuff
I just couldn't understand if that was needed to be done on digic4 (5D2 etc..) cams like digic5 cams (5d3 etc..)
now I know thank again .
Can you hear my violin warming up thou it's a bit scratchy must be something wrong with my bow  :P   
When I can play the scales correctly , then I'll post 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on February 08, 2017, 08:49:30 AM
Really your best bet is a1ex as he already confirmed that he tried to solve it for DIGIC IV (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176545#msg176545).

Maybe next time? (https://www.magiclantern.fm/forum/index.php?topic=5601.msg177584#msg177584)

You can try to solve it yourself but I can recommend to have a look into the state diagrams. Without having a look at them and understanding the vsync functions in ML you can also try to win a lottery before solving it for DIGIC IV. Finding the buffer sizes or address is just a look at the edmac screenshot and following the solution commited for 1100D... Just my 0.02$:

https://a1ex.bitbucket.io/ML/states/index.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 09, 2017, 08:37:06 AM
Thanks @nikfreak you have been very helpful .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 09, 2017, 03:29:34 PM
Quote from: nikfreak on February 08, 2017, 08:49:30 AM
...following the solution commited for 1100D...

Which branch are you looking at for the 1100D solution? Is it in the allocate-raw-lv-buffer?

The way the 1100D is handled is completely different than the 60D which AFAIK is the only other Digic IV camera that's working with 10bit/12bit.

#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif
...

#ifdef CONFIG_1100D
#define RAW_LV_BUFFER_ALLOC_SIZE (3906*968)
#endif


The 1100D isn't in the crop_rec or raw_video_10bit_12bit branches so is the build on the Experiments download page from the allocate-raw-lv-buffer branch? Note that the 1100D nightly build from the unified branch doesn't have the raw_rec or mlv_rec modules though it does have mlv_play, which is rather odd.

In any case, where can the Digic IV camera owners who are trying to solve the 10bit/12bit riddle find more information on how the 1100D was solved?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on February 09, 2017, 05:37:22 PM
You probably expecting a final solution but my previous post maybe wasn't that clear. It was just about the edmac address (channel) and buffer size (width*height). What's known was committed to the allocate-raw-lv-buffer.

I already tried to explain it (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177826#msg177826) but won't be of any help in resolving it myself due to the complexity (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176890#msg176890) and lack of knowledge. Still crossing my fingers ofc for 7D/5D2 and hoping for a solution due to the CF write speed and my opinion is that 1100D is not even worth the headaches.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SpcCb on February 10, 2017, 06:27:10 AM
Quote from: dfort on February 08, 2017, 05:28:39 AM(...)
So all you're looking for is that one hexadecimal number to unlock this feature. The question is how do you find it? Read the code, it tells you. Now the hard part is following those instructions and come up with the right address.
(...)
I'm a bad violin player too, but if the goal is to find an hexa number, is it possible to use a sort of brute force method by scanning all numbers with a match (size of the expected informations?) to find the right number? I say that just in case, I imagine all high level devs had already thought about this or maybe there's a issue in this method.

@reddeercity > The problem here, like in many other parts of ML as you know, is that kind of hack is so complex that only a couple of persons can figure how to do it. For other violin players it took years to understand. Although I had 20 years of coding in 5 languages, my eyes blinking face of some parts of ML ^.^
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on February 10, 2017, 06:31:34 PM
What's happening with raw_twk.mo?
It seems that its been left out in the cold.
Its able to display playback 10 bit raw MLV on the LCD.
The only issue I had with the 70D was it only works for the first clip, then if you try to playback
a second clip it hangs. 
Since it plays the first clip, it tells me that's its not initialized correctly for the second clip.
That should be a doable fix.  At least in terms of house of pain threshold.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on February 12, 2017, 07:56:38 AM
Raw Twk.mo on 5d3 1.1.3 , experimental build, show black frames only. In the computer the shoots were ok.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on February 12, 2017, 04:10:53 PM
@ShootMeAlready Think I've had that problem on my 5D2 for a while in the 10-12bit builds and raw_twk, lets me play back a few clips then goes black until mlv_play is restarted(was already happening in December)
@reddeercity Thanks, I didn't realise mlv_rec had black level fixed in crop mode. Just an idea: Couldn't black levels just be hardcoded for each camera so there doesn't have to be problems all the time?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: balliamo1 on February 13, 2017, 07:27:58 PM
Hello
about the 10-bit version, I would like to know if the 550D will be supported and who you reckon a download will be available.
Thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on February 13, 2017, 11:50:43 PM
Quote from: balliamo1 on February 13, 2017, 07:27:58 PM
Hello
about the 10-bit version, I would like to know if the 550D will be supported and who you reckon a download will be available.
Thanks

It's already there somewhere, tested it a while ago, scroll back through the comments and you should find it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: balliamo1 on February 14, 2017, 07:29:53 PM
I mean is a zip file available? I am not familiar with software code testing
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 16, 2017, 08:42:37 AM
Doing some research & testing on 5d2 full mlv+audio 10-12bit (I'm not doing any work with mlv lite because of no Audio for now so it's low on my priority list)
I have Qemu eos emluator setup with my rom file (5d2) , I'm almost ready to step fully in to this to work on 10/12bit issue on 5d2/digic4 although I had been discourage by a few developer
I have assistance from my best friend who happen to be a programmer/coder , with are closing in on the problem I believe. When I have something to show I'll post the results
either as Code or a test private build.

On a different note I was trying selected resolution sizes to see if this has any effect to frame corruption in 12bit and it seems that any vertical resolution be on 865 has
pink noise bar corruption on the top few lines , as you increase the vertical resolution to 1114 the pink noise bar just increase in height . It seem the top part of the sensor
always as this  corruption bar but as you crop the resolution vertically you don't record it , I tried 1856x844 2.20:1 A.R. 12bit 23.976p Canon liveview and all is well no frame shearing/tearing
all frames are good plus audio. So Any one that wants to try 12bit in 1:1 FHD I have been using the test build in my bitbucket downloads 10bit_12bit_2016Dec01.5D2212 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip)

In case anyone interested in how many digic4 cam users that have download 10-12bit test builds look below

(https://c1.staticflickr.com/4/3794/32086259354_85a048686d.jpg) (https://flic.kr/p/QTmt3b)
bitbucket downloads digic4 cam (https://flic.kr/p/QTmt3b) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

5D2 with 76 downloads next 7d & 550d with 17 , that's what gives me incentive to continue on  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on February 16, 2017, 12:17:41 PM
@reddeercity
I am sure many users will be happy to know that you are working on porting this digic 4 cameras.  Make sure you post a link to the download page when you need testing.  I did not know that you had these builds!  :o
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 18, 2017, 03:43:58 AM
Quote from: RenatoPhoto on February 16, 2017, 12:17:41 PM
@reddeercity
Make sure you post a link to the download page when you need testing.
Yes I will  ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on February 20, 2017, 02:45:19 PM

Also for the 7D?? Many thanks :) 8)  Looking forward to the links =))
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: steeth on February 20, 2017, 02:54:45 PM
I´m having a weird issue with the 10-12 bit version. I´m using a 5D Mark III, with the latest feb 11 1.2.3 experimental 10-12 bit version. Sorry if this has been covered before, I´ve searched all the possible combinations of words I can think of, and read thru as many threads I can but haven´t found anything regarding this issue.

I convert my MLV files using the latest version of CR2HDR, and I use Resolve as my choice of development software. After I started using the experimental 10-12 bit build to try out 12 bit recording, I noticed that some clips in the media pool inside Resolve came out as folders, and not clips. Inside those folders are still frames, and smaller corrupt parts of that particular clip making the clip useless, which I thought was weird..

I checked the folder with the dng files for that particular clip and noticed something really strange. At first I thought that there were missing frames in the sequence, but I have not checked the "frame skipping" option in the ML menu. So I checked the dng folder of one of the corrupt files, and all the dng files are there, but for some reason some of the dng files are named with the file extension as capitol letters (.DNG) and not lower case letters as usual (.dng)!? And that particular dng sequence doesn´t load in either AE or Resolve as it says it´s corrupt. In Resolve, each .DNG file comes out as a single still frame.. If I manually change the dng files that have upper case file endings (.DNG to .dng), the clip works for some reason!?

I have tried everything I can think of to re-create the issue, but it´s completely random and doesn´t happen when I use 14 bit. It only happens when I change to 12 bit, and there is nothing indicating that it has happened to a clip, nor can I figure out anything that might cause it as it seems extremely random.

Has anyone else experienced this issue? I get this issue on about an average of 2-3 clips per 64 gb card I use. And some have a lot of .DNG file endings in them, making it a real hazzle to manually re-name all of them :)

(http://i67.tinypic.com/14ad2qx.png)

Kind regards
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on February 20, 2017, 03:07:23 PM
Not sure if 10/12 bit are optimized working with dualiso mlv files. Better stick with 14-bit if that works better. If you want resolve embedded folders you can run your mlv files through cr2hdr.app(mac).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 20, 2017, 03:59:52 PM
As long as the input to cr2hdr is up-converted to 14 bits, I don't see why it wouldn't work. But I've never tested this scenario.

With cr2hdr, uppercase DNG means converted files; mlv_dump outputs lowercase dng, so those files probably couldn't be converted. If their bit depth is 14, upload one of them; otherwise, try a newer mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: steeth on February 20, 2017, 05:02:43 PM
Thanks a lot for your replies Danne and a1ex.

Sorry if I didn´t make it clear in my post, but I´m not shooting nor trying to convert dualiso files, just standard regular 25fps mlv files. And I´m using the latest cr2hdr.app to convert my .mlv files.

The strange thing is that 99% of the time, both 10 bit and 12 bit .mlv files convert properly as lowercase .dng frames that works properly in both AE and Resolve. I don´t do anything different, nor do I change any settings in either my camera or in the cr2hdr.app when converting. Sometimes I can record and fill up 2-3 64 GB cards with no problem whatsoever converting 12 bit .mlv files with cr2hdr.app. All clips are recorded and converted to regular .dng frames as they should, and then out of nowwhere, 2-3 clips has these random .DNG uppercase file extension on just some random frames in a converted clip as shown in my previous image, making both AE and Resolve think that the sequence is corrupt. And if I manually change these random .DNG frames to .dng instead, they open just fine in both AE and Resolve?

So I don´t change any settings between clips or cards, I don´t do anything different when converting my .mlv files from the card in the cr2hdr.app, and I still get these random upper case .DNG frames randomly inside a .dng sequence which I think is so strange?

Has anyone else experienced this? And if so, is there a solution or am I doing something wrong? I´m sorry to say that I´m just a filmmaker and I have no knowledge of nor any experience in how to use code, read code or search for errors etc, so I can´t contribute with anything but my experience in using Magic Lantern unfortunately.

Thanks again for taking the time to read my post and replying.

Kind regards.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on February 20, 2017, 07:13:03 PM
Can you post a dng file that is turned into upper case dng in the cr2hdr.app thread?
http://www.magiclantern.fm/forum/index.php?topic=15108.msg174767#msg174767
I don,t really get why it would happen in the middle of a sequence. What the script does it tries the first file with cr2hdr to see if it will create a dual iso dng and if so it will continue to run cr2hdr on all the files. Otherwise not. This would however mean that the first dng file would be mistaken for a dualiso file and maybe that is the case here.
If the problem is what I described, cr2hdr is fooled somehow, you can easily disable the cr2hdr function for MLV files in cr2hdr.app and won,t even test for dualiso movie files. Especially if you never use dualiso. Of course this has nothing to do with this thread so just continue the c2hdr thread by posting a dng and maybe even a sample of a troubling MLV file in that thread instead.

*Update
Just added a switch in mlv_dump settings menu in cr2hdr.app which disables dualiso automation.
http://www.magiclantern.fm/forum/index.php?topic=15108.new#new
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: steeth on February 20, 2017, 09:56:10 PM
Thanks a lot Danne for taking the time to answer me regarding this. I´ll definitely post one of the mlv files that has this problem in the thread you mentioned. And I´ll download and test your updated cr2hdr.app build to see if the problem goes away. And now that you mention it, from what I remember, in every mlv file I´ve converted that has those upper case .DNG frames randomly in the sequence, I believe that the absolute first frame always have had a .DNG frame, and then several .dng frames, and then random .DNG frames throughout the sequence.. But I´m not sure as I haven´t looked for it on the first frame.. I´ll definitely upload a problematic .mlv file that has these issues so you can take a look. And sorry for posting in the wrong section, I didn´t know if it was caused by the 10-12 bit build or during the conversion process.

Thanks again.

And as a side note, you don´t happen to be Swedish do you? If you are, I´m from Sundsvall in Sweden :) Not that it has anything to do with anything, just curious :)

Kind regards.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on February 20, 2017, 10:10:52 PM
Yup, swedish and polish. Creme de la creme :).
Ok, if the first one popped then the switch will solve the issue.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 21, 2017, 06:45:09 AM
I'm looking in to raw record process , 
I seems to be going in circles can someone please point me to parts of the source code that capture the raw buffer lv (which I assume hopefully , correct me if I'm wrong)
is the buffer we use for raw video . I what to step thought the lines of code for 5d2 .
Thanks in advance .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kizza1234 on February 23, 2017, 05:03:30 AM
I am so impressed with the development of magic lantern throughout my short span of using it its truly amazing, and the devs are doing such an amazing job!!! Magic lantern is so so so so good, and now with 10bit 50p 1080P continuous and stable? Its literally a dream come true!

I have installed the module and so far so good on 5D III, makes 50p 1080p (vertical upscaled 1.67x) continuous from what i can test.
Although as some other people have pointed out crop mode in 1080p 50p doesnt work as half the
frame is cut off top and bottom (image here: https://drive.google.com/open?id=0B1q4BC6n6pcNVTBwcUNXamRTRmc

Main Q:
But i was just wondering if anybody know the difference in dynamic range from 10bit to 12bit to 14bit?
From what i can see there is slightly less dynamic range every step down from 14bit, biggest difference i think is from 12bit to 10bit....
It is my understanding 14bit 1080p raw is around 11.8-11.5 stops?  :'(
Is 10bit something like 10 stops?

I might want to add that i use RawFlow to extract my files and 10bit extracts fine without any issues i can see.
Link: https://www.magiclantern.fm/forum/index.php?topic=13338.0
Its by far the easiest way i have found to extract files without any complications, and the images that get extracted are amazing!

Thanks Guys
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: alejandro rumolino on February 23, 2017, 12:10:24 PM
For 6D here the experimental loop https://builds.magiclantern.fm/experiments.html
I uploaded it and it works smoothly in 16/9 1600x900 10-bit, it does not have the possibility of the audio and I had a problem after filming, my canon does not start more for 1 hour ... so uninstall it ... use MLVtoMovie in .mov after
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on March 02, 2017, 04:36:19 PM
Kizza the DR stops or range is a matter of sensor size and performance.
You can have 20bit raw out of a small sensor, and it will not be better than a 10bit raw from a big sensor that has wide pixel separation, low noise. 

Humans are not very good at distinguishing colour depth, so 4 bit colour verses 2bit we are not as quick. Folks cant discern any difference in a 14bit raw vs a 12bit raw photo.  Where you get the big benefit is in pushing the raw data, like adjusting exposure in post, in that you can push 14bit raw much more than 10bit raw.

I think humans can distinguish 12bit raw vs 10bit raw photos, but that's only in certain shots and not always. 10 bit raw provides greater depth than compressed video, so one should be able to distinguish these, especially in low light shooting detail.  10bit will generally do a better job with highlight details (assuming not overexposed).  So rather than quote stops of light, think more about depth and detail in low/high light.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on March 03, 2017, 11:08:37 PM
Quote from: ShootMeAlready on March 02, 2017, 04:36:19 PMHumans are not very good at distinguishing colour depth, so 4 bit colour verses 2bit we are not as quick. Folks cant discern any difference in a 14bit raw vs a 12bit raw photo.  Where you get the big benefit is in pushing the raw data, like adjusting exposure in post, in that you can push 14bit raw much more than 10bit raw.

I think humans can distinguish 12bit raw vs 10bit raw photos, but that's only in certain shots and not always. 10 bit raw provides greater depth than compressed video, so one should be able to distinguish these, especially in low light shooting detail.  10bit will generally do a better job with highlight details (assuming not overexposed).  So rather than quote stops of light, think more about depth and detail in low/high light.
^ Sounds like poetry.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on March 04, 2017, 09:44:03 PM
Using 10bit_12bit.2017Feb11.5D3 build, and can't get ML to load with SND module activated (it says something like can't load modules, with a grey box). Any idea or do any of you have the same problem? Shooting a lot of sync footage tomorrow so would help a lot for to the post process. I wish not to use nightly because of the extra resolution on 10bit while doing zoom view..

Thank you!

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on March 05, 2017, 02:53:47 AM
Try only loading the 2 modules you need. Nothing else.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 05, 2017, 04:18:42 AM
You mean "failed to link modules" ?

Are you trying to load mlv_snd with mlv_lite? That won't work. You need to use mlv_snd with mlv_rec.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on March 05, 2017, 07:05:55 AM
Quote from: dfort on March 05, 2017, 04:18:42 AM
You mean "failed to link modules" ?

Are you trying to load mlv_snd with mlv_lite? That won't work. You need to use mlv_snd with mlv_rec.
Thats correct. Thank you for the heads up!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 06, 2017, 04:43:21 AM
Hey guys I've been getting some weird behaviour from Mark III 10bit MLV files converted to DNG with MLVFS. Probably 1 in 50 MLV files seems corrupted.

It looks like the Black Level issue that we're all familiar with – but I can't correct it using exiftool. I got the black level changed to 2048 but did not solve the issue.

What might be happening here and is there a way to fix it? Would someone care to try exiftool to confirm? Here is a DNG file (untouched) from one of the corrupted 10 bit MLVs.

https://www.dropbox.com/s/m77zmz6m6h9huik/M03-1839_000008.dng?dl=0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 06, 2017, 05:49:35 AM
No joy with exiftool , tried with mlv_dump , results below 
Will there is problems , Can you upload the corrupted MLV file if not too big,
you skinny can it down with mlvdiag.1.0.1.zip (http://dl.phreekz.de/raw2cdng/mlvdiag.1.0.1.zip) if on pc
C:\New\test>mlv_dump -o updated.dng --black-fix=valve M03-1839_000008.dng
MLV Dumper v1.0
-----------------
Mode of operation:
   - Input MLV file: 'M03-1839_000008.dng'
   - Setting black level to 1
   - Rewrite MLV
   - Output into 'updated.dng'
File M03-1839_000008.dng opened
Processing...
[ERROR] Invalid block size at position 0x00000000
Processed 0 video frames
[ERROR] Failed to rewrite header in .MLV file
Done


You have some odd white & black levels
Unique Camera Model             : Canon EOS 5D Mark III
Black Level                     : 418
White Level                     : 1018

Light valve seems off too
Light Value                     : 3.1

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 06, 2017, 08:10:04 AM
It is 8.5GB file. I'm on Mac.

I found a similarly corrupted file that is much smaller.

https://www.dropbox.com/s/rmkyyg6q3nn47yj/M03-1356.MLV?dl=0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 06, 2017, 08:11:30 AM
Setting black level to 128 fixes it. There's a bug in MLVFS that doesn't let you change black level with exiftool, but didn't manage to look into it yet.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 06, 2017, 09:17:48 AM
I used mlv_dump instead to convert to DNG then used exiftool black level to 2040 – seems to have fixed it.

Although now there is an issue with the top section of the frame. It looks smeared.

https://www.dropbox.com/s/ucathd53p1megkt/M03-1839_000001.jpg?dl=0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 06, 2017, 10:10:45 AM
Just a note. mlv_dump supports specifying both black and white level manually.
Also had issues with mlvfs dng files and exiftool. DNG files turning black or not letting exiftool changing tags.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 06, 2017, 01:08:18 PM
Quote from: MitchLally on March 06, 2017, 09:17:48 AM
Although now there is an issue with the top section of the frame. It looks smeared.

With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 06, 2017, 11:14:08 PM
Just a little update on the digic4 cams downloads

(https://c1.staticflickr.com/4/3756/32450022274_6a54bbc22d.jpg) (https://flic.kr/p/RruR5E)
10-12bit downloads_DigicIV (https://flic.kr/p/RruR5E) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Seams 5D2 users are very hungry for Bit reduction  148 downloads and counting  8)
I don't think even 5d3 has that many bit reduction downloads
Long Live the mighty Digic IV Cams 7 years old+ and still in the game  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 07, 2017, 07:24:09 AM
Quote from: a1ex on March 06, 2017, 01:08:18 PM
With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.

I'm on mac – is there an updated mlv_dump for mac?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 07, 2017, 07:32:28 AM
mlv_dump binary
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump

cr2hdr.app(uses mlv_dump)
http://www.magiclantern.fm/forum/index.php?topic=15108.msg146822#msg146822
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on March 11, 2017, 01:52:00 PM
Whats going on with in the raw_twk, mlv_play and 10 12 bit??? A lot of people confirm the issue whith playback - you can only review 2-3 clips before you get black screen?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 11, 2017, 02:40:29 PM
http://wiki.magiclantern.fm/faq#any_progress_on_xyz
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Anstech on March 13, 2017, 02:07:54 AM
I'm a programmer in a very different field, and I'm just starting to try to get up to speed with understanding the ML / Digic context, so please excuse any naivety in this question:

One way that ArriRaw, amongst others, reduces its data requirements is to write the 16bit linear data from the sensor into an uncompressed raw 12bit log data stream, which - given the familiar inefficiency of linear encoding of light values - drastically reduces frame sizes.

The current 14-to- 12- and 10-bit strategy is an amazingly impressive piece of work, and I can see took years to achieve, and a lot of specialist knowledge. If I understand correctly, it's a bit-chop lossy reduction in representation precision. Obviously the raw-log-encoded method would be less of a quality drop from 14- to 10-bit, if it existed.

So before I 'go off on one' (as my old AI programming tutor used to put it when he saw me disappearing down a research rabbit hole / dead end again):

Is e.g. a 14-bit to 10-bit log-encoding of the raw uncompressed (linear) sensor data - rather than a lossy bit-chop - computationally plausible in this context? Or is it waaay too ambitious for the resources available on-camera during recording?

It's caught my imagination, but I realise it might be a silly idea if you know enough - so if that's the case, I'll be sensible and take on a far more manageable task from the 'to-do' list as a 'teach myself' project instead...

Thank you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 13, 2017, 08:45:50 AM
On CPU, even a simple memcpy is too slow (http://www.magiclantern.fm/forum/index.php?topic=5601.msg166928#msg166928). So, the only remaining possibility is finding out how to configure Canon's image processor to do that.

Relevant links here (https://www.magiclantern.fm/forum/index.php?topic=16428.msg159735#msg159735), here (https://www.magiclantern.fm/forum/index.php?topic=16428.msg159735#msg159735), here (https://www.magiclantern.fm/forum/index.php?topic=18838.msg179076#msg179076) and here (http://www.magiclantern.fm/forum/index.php?topic=18222.msg174806#msg174806).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TKez on March 22, 2017, 04:24:14 AM
Chiming in to add/confirm reports on 550D every 2nd frame is a copy of the last frame, except the top 3rd which looks like current frame, only shifted on X axis.

If I can contribute to testing in anyway, gimme a job! 550D is 'B' cam, but love to get more than a couple of seconds of Raw at full res.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ricbi1 on March 22, 2017, 04:25:27 AM
Anstech... if you achieved that, it would be great! Just trying to encourage coz I can't do code for my life. Absolute ZERO. I tried over the years but no, can't do sir! Thanks everyone who is coding for sharing their efforts and developments!  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Eugenia on March 25, 2017, 11:44:15 PM
Hello, first post here. :)
I have two questions, that I couldn't find answers for:

1. What is the max res of the EOS M using 10 bit, no crop, at 24p continuous?

2. What is the max res of the 5D MkII using 10 bit, no crop, at 24p continuous? Why is there no 10-12bit ZIP file to download for the 5D-MkII? I couldn't find any on the exp. download page.

thx!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: far.in.out on March 26, 2017, 11:05:25 PM
Hey guys. Sorry for asking such a question but I really have no time to read all the thread, been away for too long...
What is the status on 12/10 bit raw? Is it working? Are there any downsides compared to 14bit?
I'm mainly interested in 50D as it's the camera that I have.
If there's a compiled module for my cam please give me the link. Thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on March 27, 2017, 01:06:43 AM
Red deer city has some builds uploaded a few pages back, including 50D, I know for a fact that the 50d can't film 10/12 bit without corruption in full sensor, but maybe it works well in crop mode like the 5D2, give it a try.
edit: it's on page 46
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: beauchampy on March 27, 2017, 12:07:25 PM
Just came back from Malta filming a client piece entirely using 12bit and 3x3 50p. Flying by the seat of my pants, but the footage is lovely and not a single corrupt frame (5d3). I'll post some results next week. As far as I am concerned, this is more than ready for commercial use IMO.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 27, 2017, 12:27:24 PM
atleast 4 footage gets corrupted out of 10 with wrong black level.  im using 70D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 27, 2017, 12:50:48 PM
@andy kh

This is for you: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: beauchampy on March 27, 2017, 08:52:43 PM
Quote from: andy kh on March 27, 2017, 12:27:24 PM
atleast 4 footage gets corrupted out of 10 with wrong black level.  im using 70D

I'm using cr2hdr which enables you to specify a black level. Not sure what it is on the 70D but on my 5D3 I'm using 2048.

I do have the odd clip here and there with strange pink highlight clipping. I had also specified a white level of 1163.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 27, 2017, 10:15:51 PM
Could you post an example MLV which shows the corruption? You could shorten it before uploading. Use mlv_dump and open up terminal and do following:
mlv_dump -f 5 -o Output.MLV Input.MLV

mlv_dump
http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip

Input.MLV being the corrupted file and Output.MLV the name of the shortened file.
Beware to do things right since a wrong command could erase the file in question.

By the way. I think black and white levels should be around this for the 70D
White level 12277
Black level   2049
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 27, 2017, 11:13:25 PM
@Danne  please check this out. i use mlvp and set the black level to 125 for all 10bit mlv raw.
https://expirebox.com/download/fcfa483143425529e11d9cd2f4c43776.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 27, 2017, 11:27:59 PM
Yes, classic black level issue. From your file after extracting with mlv_dump
Black Level                     : 7312
White Level                     : 14992

When reprocessing and specifying numbers below picture is perfect.
Black Level                     : 2049
White Level                     : 12277

In mlvp you specify 10bit levels so they differ. Above numbers are for upressed to 14bit. Visually the picture should look the same for both 10bit and 14bit after fixing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 27, 2017, 11:39:27 PM
@Danne thanks for checking it out. corrupted files are so many. as i mention earlier atleast3 to 4 out of 10 files get corrupted everytime i record in 10bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 27, 2017, 11:55:41 PM
I don,t know the status of 70D and 10/12bit recording but you could use mlv_dump for windows and specify both white and black level for all files that you export. Or stick to mlvp if that,s more convenient. Here is mlv_dump for windows
https://builds.magiclantern.fm/jenkins/job/mlv_dump/20/artifact/src/modules/mlv_rec/mlv_dump.exe

Specify like this
mlv_dump --dng --white-fix=12277 --black-fix=2049 Input.MLV

ANd why not do the same command with bouncyball´s version mlv_dump on steroids :)
http://www.magiclantern.fm/forum/index.php?topic=18975.msg180083#msg180083
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 28, 2017, 12:11:43 AM
@Danne
i wil try bounchyball's version as i prefer to export in cdng
thanks for your help
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 28, 2017, 04:22:19 AM
If on Windows use MLVProducer (http://www.magiclantern.fm/forum/index.php?topic=15271.msg148538#msg148538) , support 10 & 12 bit full mlv , mlv lite & old raw plus exports as Cdng if needed .
Also has slider for black level with real time preview like A.E. ACR plus many more grading options , I MUST have tool in windows along mlvfs .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 28, 2017, 10:05:06 AM
@reddeercity i have been using mlvp for sometime and wil continue to use it. it has some problems though. cdng exported from mlvp has a decrete amount of magenta cast while exporting to resolve or premiere pro. it can be fixed by adjusting the white balance though. awpstar is aware of the magenta cast as someone in the forum pointed out with image. and after converting to h264 422 or 444, im not able to import them in resolve. while clicking the folder containing converted h264 422 in resolve, all i can see is red icons written offline. i wil share with more details in mlvp forum
thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: beauchampy on March 28, 2017, 12:50:08 PM
@Danne I've been trying to find out more information for black level / white level on the 5d3. I recently forced white level to 1163 using cr2hdr (is that your app btw!?). Can it be pushed around even more for increased dynamic range?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 28, 2017, 01:00:23 PM
You can push whitelevel to 16383, it,s maximum white point.
Black level 2048. I do think default for 5D mark III is set to 16383 already.
Yes, my scripts bundled in automator. Utilizes a mlv_dump version with code from all over the place.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 28, 2017, 01:10:46 PM
The black level is the signal level you get in the absence of light. It's a positive value because:
- the value is not the same for every sensel (hence the need for dark frame subtraction)
- values below black level do appear (because of the additive noise); a noise reduction or a frame stacking step can make use of those sub-black values.

"Pushing" the black level to a lower value would be the same as changing the input curve (rendering deep shadows in a lighter tone, rather than black). You probably liked the look, but there's no additional dynamic range to gain this way.

If the black level is incorrectly set above the real value, you will see additional detail from lowering it. Otherwise, there's little or no point in changing it.

Changing the black level also changes the colors in shadows, because the white balance multipliers are usually not equal. If you set all of them to 1, the color cast introduced by changing black level disappears.

On the other hand, there might be additional dynamic range to gain from pushing white level to a higher value. With recent versions, this additional gain will be less than 0.02 EV.

/**
* White level
*
* With PREFERRED_RAW_TYPE set to CCD, most cameras appear to clip above 16300
* (most of them actually use the full range, until 16382)
*
* one size fits all: 16200 may sacrifice up to 0.02 EV of highlights
* that is, log2((16382-2048) / (16000-2048))
*/
#define WHITE_LEVEL 16200


However, if you find pink highlights with this value, I'd be interested to see a sample MLV.

@Danne: pushing white to 16383 may be a little optimistic; there are exposure settings that clip to this value and there are exposure settings that clip to a slightly lower level (including 5D3). Usually, if white level is above clipping point, you get pink highlights; but in some cases, the result is pretty good anyway, so it's a bit of an edge case.

You can check the white level in the samples in this thread: http://www.magiclantern.fm/forum/index.php?topic=11899.0

e.g. with octave (requires read_raw.m (https://github.com/apertus-open-source-cinema/misc-tools-utilities/blob/master/darkframes/read_raw.m)):

a = read_raw("frame.dng");
prctile(a(:), [90 95 99 99.9 99.99 99.999])


Note the max value from the image can exceed the clipping point from one sensel (happens with hot pixels), that's why I prefer checking it with percentiles.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: beauchampy on March 28, 2017, 04:52:16 PM
@a1ex @danne

Just to clarify. What should the black and white levels be for a) true colour representation and b) maximum dynamic range on the 5d3?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 28, 2017, 05:43:34 PM
I would go with this for white balance. Thanks a1ex.
Quoteone size fits all: 16200 may sacrifice up to 0.02 EV of highlights
Black level 2048

However Davinci resolve probably expects 5D mark III around 15000 if you start messing with their highlight recovery so you simply have to check what looks alright, how you choose to work with raw material and what NLE you are working in.

Do have a nice evening read in Baldavengers resolve frenzy over here
http://www.magiclantern.fm/forum/index.php?topic=15801.msg164822#msg164822
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on March 28, 2017, 06:35:00 PM
Been getting decent results w 5D3 footage by using @Danne's cr2hdr.app (black levels @ 2048 and white levels @ 16383) but now according to @a1ex it seems the new 'sweet spot' would be 16200 for the whites, correct?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 28, 2017, 06:48:07 PM
Safety margin, not sweet spot. Very well explained.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on March 28, 2017, 07:15:24 PM
Well perhaps I should have mentioned 'safety spot' rather than. This is all great to know and thanks guys!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Haeri on March 28, 2017, 10:49:15 PM
Any compiled 550D builds around? Would really like to try it out :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 30, 2017, 06:38:33 PM
Theres a few others here having the black level issues popping up with 10bit. I have a suspicion that it could be a result of having raw_twk loaded or "extra_hacks" selected in the mlv menu. Its one of the two so far I've had no false black levels with both switched off. I will test extensively over the weekend. Anyone else who has had black level issues tried this? And is it even possible that another module would interfere? I'm clueless on the coding so all I can do is test for you guys.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 31, 2017, 06:44:07 AM
a Little update for the 5D2 , Dual ISO Video Does work with 10-12bit Full MLV. the file corruption for 1:1 is still there thou 3x Crop work OK
did produce a few Hot pixels , by fine tuning the ISO range and a few more setting there can be eliminated . I was mostly looking to see
if it would work and it does . I only had 1 liveview freeze the needed a battery pull ,but I did a Super Fast pan before recording and it was in
3x crop mode with mlv_lite  before recording , after recording started it didn't seem to be a problem.
So at this point Dual ISO Video is functional in 3x Crop mode and all frame have proper black level with the 20bit option toggled in MLVFS PC Version Web GUI.
If you what to try it out here is the link 5D2212_Video_dual_iso.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5D2212_Video_dual_iso.zip) !!! Beware !!! This could brick your camera (will probably not :D)  I had to say that !
If anything happen like a liveview freeze just pull the battery out asap.
One more thing this is for just the 5D2 not sure if it will work with other Digic IV cams
Until next time  8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on March 31, 2017, 08:47:42 AM
I tried the dual ISO recently too, and it was really difficult to get the freeze to happen(maybe it's improved?). I still think the freeze needs to be fixed for it to be included in main build though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on April 04, 2017, 03:29:48 PM
So I was using the Dec build for 10-12 bit raw video on the 7D; here's a little update to show some love for the camera.

While I was recording 10bit (various resolutions) my screen is killed instantly until it stops recording and writing, but it still records at normal FPS, with and without override.
When I was recording 12bit, the camera functioned normally and I had no issues manning my camera as I normally do; but the video output only gets a rough 15-20 frames a second, making the video look like a Chaplin film with color.

14 bit works just as fine. I did both 5x and normal modes.

Also, while recording 12, and I believe 10, The recording frame was not aligned properly- here's what I mean: http://imgur.com/a/sIb9W

To anybody working on the 7d, or to who can point me in the right direction, I'd like to contribute and accelerate the development of this firmware for the 7D as much as I can. So if you'd like some info or logs from me, please let me know what I can do.


Check my signature if you want to see an example of the footage and the quality (fixed from tech errors)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rduquematos on April 07, 2017, 08:31:26 AM
is 10 bit raw available on the 50d? thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on April 11, 2017, 11:59:30 PM
Quote from: rduquematos on April 07, 2017, 08:31:26 AM
is 10 bit raw available on the 50d? thanks

yes, work
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on April 12, 2017, 12:25:05 AM
Nested quote --

Quote from: CITY-U1001 on April 11, 2017, 11:59:30 PM
Quote from: rduquematos on April 07, 2017, 08:31:26 AM
is 10 bit raw available on the 50d? thanks
yes, work

Really? Since when? It isn't on the Experiments download page (https://builds.magiclantern.fm/experiments.html) and I haven't seen any commits on bitbucket for it.

It would be nice to get it working properly on the 50D -- and the 5D2, 7D, 500D and 550D but those who have tried can tell you, it isn't that easy.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 12, 2017, 03:45:50 AM
Quote from: dfort on April 12, 2017, 12:25:05 AM
yes, work
Really? Since when? ...... It would be nice to get it working properly on the 50D -- and the 5D2, 7D, 500D and 550D but those who have tried can tell you, it isn't that easy.
+1  what dfort said ,
@CITY-U1001 if you mean 3x crop will that's not working fully is it , when 1:1 works that's when you can say it works.
I've been working on the 5d2 for mouths now and I still can't get it to work right in 1:1 , 3x crop work @ 10bit but that's not working fully.
If you have it working  then Please share the code/info so I can advance the 5d2 .   
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mirko91 on April 15, 2017, 10:59:37 PM
Hi guys, where can I download a compiled version with 10/12bit module for a 5d3 1.2.3? There were some links around but they are all missing. Thanks to anyone!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: pc_bel on April 24, 2017, 11:46:59 AM
http://builds.magiclantern.fm/experiments.html (http://builds.magiclantern.fm/experiments.html)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SkalickyJ on April 24, 2017, 03:19:55 PM
Hi, I tried to shoot 1728x736 10 bit ml-raw on canon 700d. But i get files full of green,pink and blue dots, its almost impossible to see the original image i shot, because its covered with this dots.  Any idea, what could be wrong? Thanks! https://ibb.co/dF9GLk (https://ibb.co/dF9GLk)


btw. card speed should be ok, 14bit raw video at 1280x720 is working great...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on April 25, 2017, 03:33:07 PM
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/  :-\ but not fresh build
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: theBilalFakhouri on April 26, 2017, 12:19:06 PM
Quote from: SkalickyJ on April 24, 2017, 03:19:55 PM
Hi, I tried to shoot 1728x736 10 bit ml-raw on canon 700d. But i get files full of green,pink and blue dots, its almost impossible to see the original image i shot, because its covered with this dots.  Any idea, what could be wrong? Thanks! https://ibb.co/dF9GLk (https://ibb.co/dF9GLk)


btw. card speed should be ok, 14bit raw video at 1280x720 is working great...

Your workflow process did not support 10-12bit
Use MLVProducer (the best,fastest software i used in MLV files Process) (also support 10-12-14bit and removing focus point automatically)
Download From Here:
https://www.magiclantern.fm/forum/index.php?topic=15271.0
after installing the program you must change some settings to encod MLV To DNG
(https://preview.ibb.co/hxPCAk/IMG.jpg)
Now Use in file menu Open MLV/RAW >> Chose your mlv file >> and again click File >> Render Task
You can open multiple MLVs And Click Render Queue
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rduquematos on May 05, 2017, 02:53:46 AM
Quote from: CITY-U1001 on April 25, 2017, 03:33:07 PM
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/  :-\ but not fresh build
tried it on my 50d, 12bit works fine, 10bit I get some pink frames on the top
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 08, 2017, 01:44:36 AM
Working on 10-12bit 1:1 on 5D2 , as we know 3x crop works with 10 & 12 bit reduction but not 1:1 really well .
Thou in 12bit 1:1 you can record with corruption on every other frame on the top 100 or so lines .
I'm currently seeing the feasible of using bit reduction in a up coming documentary I been hired to shoot in 2 weeks for the sierra club (usa)
My needs are long shoot times , so low bite rate fits the bill .
Here a list of resolution's size I was possibly thinking of using with 12bit in 1:1 (as there corruption on the top of frame)
with the adjusted crop to remove corruption. I also listed the file size for 60 second or so to get max time for minimum card space
The base line is 14 bit 1856x928 23.976 fps (My minimum spec. for HD broadcast as it need to be up upscaled)
1856x928  23.976 14bit 59.9 seconds 1437 frames 4.05 GB 1:1 Liveview  ( 2:1 A.R.)
1856x1114 23.976 12bit 64.8 seconds 1554 frames 4.51 GB (crop adjust 1856x988) 126 line crop 1.878542510121457 A.R.
1856x1044 23.976 12bit 64.8 seconds 1554 frames 4.23 GB (Crop Adjust 1856x953) 91 line crop  1.947534102833158 A.R.
1856x1004 23.976 12bit 60.3 seconds 1447 frames 3.78 GB (crop Adjust 1856x932) 72 lines crop 1.991416309012876 A.R.
1856x928  23.976 12bit 60.1 seconds 1442 frames 3.49 GB (crop adjust 1856x895) 33 lines crop 2.073743016759777 A.R.
   

So There is really no saving after cropping , but 3x crop mode is a different matter all together .
Here 10bit really save me , as I can have to 20min per 64GB Card @ 1920x1076 @ 23.976fps
Thou the field of view is reduced , looks like the 24mm will gets a work out  :D
1920x1076 23.976 10bit 59.9 seconds 1437 frames 3.47 GB  (3x Crop Mode)  1.784386617100372 A.R.
So I save about 500 MB per minute which will add up fast. 4x 64gb card should be around 80 min's of 10bit 2K or 1920x1076.
Now I can use my slower 60MB/s CF cards to get
1856x928 23.976  10bit 60.1 seconds 1443 frames 2.91 GB  (3x crop)  (2:1  A.R.)

It looks like I'll being shooting in 10bit 3x crop for this doc.

Now for the real reason I started this post , as I said I'm work on 1:1 liveview 5d2 10-12bit.
I've being working on older code with .raw , just to keep it as simple as possible before I move on to .mlv full & mlv_lite.
At this Point I can record 10bit & 12bit , with out any lockups or screen freezes all overlays are working while recording bit reduction.
When not in bit reduction it return to 14bit and of course it work without problems.

Now the Problem  ::) there is always a problem , my issue is decoding the .raw to  dngs
My question is can I convert the .raw to .mlv ? as mlv_dump seem not to understand .raw
I'm currently uploading some of the .raw file to my Google drive to share , when done I'll post the links.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 08, 2017, 04:04:09 AM
Quote from: reddeercity on May 08, 2017, 01:44:36 AM
I'm currently uploading some of the .raw file to my Google drive to share , when done I'll post the links.
10 & 12 bit 1:1
https://drive.google.com/file/d/0BxNY4d4Aa7KuM1JlR3JnSTZpNlE/view?usp=sharing     1GB
https://drive.google.com/file/d/0BxNY4d4Aa7KualhHNnJ0Ujc3Ym8/view?usp=sharing    899MB
https://drive.google.com/file/d/0BxNY4d4Aa7KuSFpzcjJudHVUblk/view?usp=sharing       1GB
https://drive.google.com/file/d/0BxNY4d4Aa7KuSE1yZ2xqWERkR1k/view?usp=sharing    1GB
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 08, 2017, 05:09:58 AM
Should have taken my own advice "search box"  :P
Any ways yes you can convert .raw to .mlv with raw2dng .
now I have a more workable file .

Edit:
After All that , it's reading as a 14bit file , so the image is totally corrupted
should be either 10 or 12 bit .
Top file is my 10-12bit .raw file that I converted to .mlv
Second files is 14bit , was recorded in the session as the bit reduction files
Third is a older 2k 10bit .mlv  , just to be sure it's decoding correctly .
(https://c1.staticflickr.com/5/4156/33681092884_09fc15aa4c_c.jpg) (https://flic.kr/p/Tjhpu1)
raw2cdng_bit_reduction (https://flic.kr/p/Tjhpu1) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

With MLVFS quick mount on PC I get this from the cdng's from the .raw converted to .mlv , black level is way out to lunch , but it's a start .
ExifTool Version Number         : 10.33
File Name                       : m07-1552_000000.dng
Directory                       : C:/New
File Size                       : 3.3 MB
File Modification Date/Time     : 2017:01:07 14:51:59-07:00
File Access Date/Time           : 2017:05:07 21:01:52-06:00
File Creation Date/Time         : 2017:05:07 21:01:52-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution Image
Image Width                     : 1872
Image Height                    : 1054
Bits Per Sample                 : 16
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Fill Order                      : Normal
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Strip Offsets                   : 65536
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Rows Per Strip                  : 1054
Strip Byte Counts               : 3384466
Planar Configuration            : Chunky
Software                        : MLVFS
Modify Date                     : 2017:00:07 15:51:59
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Exposure Time                   : 1/50
F Number                        : 1.0
ISO                             : 81
Sensitivity Type                : ISO Speed
Exif Version                    : 0230
Subject Distance                : 0 m
Focal Length                    : 50.0 mm
Lens Model                      : Some Great 50mm f/1
DNG Version                     : 1.4.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Linearization Table             : (Binary data 87187 bytes, use -b option to ext
ract)
Black Level                     : 7794
White Level                     : 15000
Default Crop Origin             : 0 0
Default Crop Size               : 1872 1054
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
As Shot Neutral                 : 0.3885467303 1 0.7199849091
Baseline Exposure               : 0
Calibration Illuminant 1        : D65
Active Area                     : 0 0 1054 1872
Frame Rate                      : 23.976
Baseline Exposure Offset        : 0
Aperture                        : 1.0
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 1872x1054
Megapixels                      : 2.0
Shutter Speed                   : 1/50
Focal Length                    : 50.0 mm
Light Value                     : 5.9
-- press RETURN --
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on May 08, 2017, 02:25:25 PM
@reddeercity
i didnt read the last bunch of thread-entries. is it my tool or something else? if its my tool, give me some advice to fix.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 09, 2017, 01:24:48 AM
@chmee  , No your app is fine I was showing the issue I was having with my custom raw_rec module exporting to 10 or 12bit for 5d2 1:1
As 14bit & 10bit 3x crop .raw works fine from 5D2 .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 09, 2017, 05:58:06 AM
After checking my 10-12bit 1:1 files with  mlvdiag.1.0.1. from chmee (converted .raw to .mlv with raw2dng) I notice I'm missing the Null Block (Alignment Fill)
Or is it just a block for .mlv format ?
Is the bit depth written in the MLVI block or RAWI block ? or is it in a different block .
my file read as 14bit in MLVProducer & raw2cdng when I recorded in 10 & 12 bit , thou the 14bit .raw are fine .
I guess I will have too go check my code .   
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on May 09, 2017, 09:01:24 AM
Bitdepth is in the raw_info structure which is part of the RAWI block; raw_info can be found in raw.h in src folder
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 11, 2017, 07:41:39 AM
I'm a little closer , I being working on 2 different branch's Compressed Raw & Memspy with the Old ML core ( why the old core , because raw_rec is very simple and easier to understand)
actually the only problem I'm having with the Old core is decoding the different bit depth's from the .raw files (excluding 14bit)  , I have even implemented 16bit encoding as a test .
You my ask how do I know it's recording at 10 , 12 , 16 ? By the data rata , reference to the old dec.1/2016 10-12bit build in my bitbucket downloads and 16bit say's it's data rate is
91MB/s @ 1872x1054 23.976 , Thou I can only save about 200 frames , yes 16bit is not realistic to record , it was to see if my coding was correct . I'll discuss the decoding problem later on .

I wanted to talk about the compressed raw branch with 10-12bit , thou I'm not trying to implement compressed raw just yet it was a way to get a cleaner 10bit 1:1 image
Source Code is here  magic-lantern_compressed_raw.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magic-lantern_compressed_raw.zip) and there is a build I made for testing magiclantern-Nightly.Compressed.Raw.Bit.Reduction.2017.May11.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-Nightly.Compressed.Raw.Bit.Reduction.2017.May11.5D2212.zip) .
Caveat , 10bit is only cleaner with Auto preview , there is a from of sheering but both halfs of the image are clean
and there is movement in all frames no frozen images , it's far from perfect to say the least but no pink corrupted frame , 12bit still have the same top 100 or so line of corrupted .
As nominal , this build could brick your camera  ::) and you get to keep both halves  :P

Edit: Just a note about the old core .raw decoding problem , basically for some reason be on my understanding the .raw files keep writting14bit to the RAWI block instead of 10 or 12 etc...
I tried this from a1ex , but seem not to work , thou I'm very sure I have not implemented it correctly 
http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181#msg174181
QuoteTo get clean image, you will have to change raw_info.bits_per_pixel from the raw recording module

   strcpy((char*)footer.magic, "RAWM");
    footer.xRes = res_x;
    footer.yRes = res_y;
    footer.frameSize = frame_size;
    footer.frameCount = frame_count - 1; /* last frame is usually gibberish */
    footer.frameSkip = 1;
   
    footer.sourceFpsx1000 = fps_get_current_x1000();
    footer.raw_info.bits_per_pixel = raw_info.bits_per_pixel;

    int written = FIO_WriteFile(save_file, &footer, sizeof(lv_rec_file_footer_t));

Where it say's "footer.raw_info" I add ".bits_per_pixel"
Any help would be greatly appreciated .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 11, 2017, 08:48:17 AM
here a test 10bit 1:1 1856x1044 file 5d2 from the compressed raw branch
https://www.dropbox.com/s/3znqpbd9ur0o1xe/M10-2043.mov?dl=0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on May 11, 2017, 09:11:47 AM
@reddeercity Can't tell if you've got things working :-\ Have you got reduced bitdepths working?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on May 11, 2017, 04:50:41 PM
@chmee adding an option to change black level would be great. some 10bit raws are unusable because of wrong black level
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 12, 2017, 06:07:56 AM
Quote from: Ilia3101 on May 11, 2017, 09:11:47 AM
@reddeercity Can't tell if you've got things working :-\ Have you got reduced bitdepths working?
In compressed raw branch , yes with full mlv+audio  haven't tried with mlv lite yet ( I need audio so mlv lite is low on my list)
As you know 3x crop is good , this is for 1:1.
The video in my dropbox link is a 10bit  1:1 23.976fps , normally the image is half pink corruption , so improvement there
Also yes bit reduction in the old core but still can't decode the .raw file reads as a 14bit .
If I could get this fixed I can move on to mlv.

Edit: is there a way to extract frames at low level with correct bit depth and bypass the header ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 16, 2017, 07:03:47 AM
I asked a question "is there a way to extract frames at low level with correct bit depth and bypass the header ?"
Will I answered my own question , In a way yes . What do I mean by that , will you need to modify blocks with something like
HxD a hex editor , that on wins platform there should be a Mac version hex editor out there.
I loaded a true 10bit MLV_Lite file (1872x1052) and then my .RAW Old ML Core custom build raw_rec.mo 10bit recorded (1872x1054) with wrong bit depth information
Reads as 14bit as per post #1256 . Thought trial & error I found which blocks are for depth bit & black/white levels
Offset 00000060 :   0A 00 00 00 6F 00
That's from the  donor file (10bit mlv lite)
Offset 00000060:   0E 00 00 00 FF 06
Blocks from the original .raw which is wrong bit depth (14bit) should have been 10bit.
So "0A" is 10bit & "0E" is 14bit  and "6F 00" gives correct white & black levels @ 10bit pixel depth

Original .raw converted to .mlv with raw2dng with wrong bit depth (14) recorded as 10bit in 1:1 5d2
https://drive.google.com/file/d/0BxNY4d4Aa7KuSnpMdDZtS1Y5RGs/view?usp=sharing
Modified block bit depth blocks & black/white levels
https://drive.google.com/file/d/0BxNY4d4Aa7KuNGFYZ09tRlRZbzQ/view?usp=sharing
Corrected 10bit Cdngs from MLVProducer (frames , M08-1755-3_00001.dng & M08-1755-3_00002.dng)
https://drive.google.com/file/d/0BxNY4d4Aa7KuNGFYZ09tRlRZbzQ/view?usp=sharing
https://drive.google.com/file/d/0BxNY4d4Aa7KucnByQ1lyLWVMZ28/view?usp=sharing

(https://c1.staticflickr.com/5/4167/33878844593_581677aa1c_z.jpg) (https://flic.kr/p/TBKWbe)
Old-ML-Core10bit (https://flic.kr/p/TBKWbe) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://c1.staticflickr.com/5/4189/34688598565_29d7831842_z.jpg) (https://flic.kr/p/URj8PB)
Orginial_MLV-RAW (https://flic.kr/p/URj8PB) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://c1.staticflickr.com/5/4164/33878844913_969731e03d_z.jpg) (https://flic.kr/p/TBKWgK)
Modified_MLV-RAW (https://flic.kr/p/TBKWgK) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr




Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: durb on May 17, 2017, 12:33:26 AM
Hello! I can fix this 12bit dng from 5d mark 2 ?
I deleted my mlv file and I only have a dng sequence.

(http://thumb.ibb.co/ifORVk/Ybw_Isf1j_Tk.jpg) (http://ibb.co/ifORVk)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: AWPStar on May 17, 2017, 08:43:42 PM
Quote from: reddeercity on May 16, 2017, 07:03:47 AMI asked a question "is there a way to extract frames at low level with correct bit depth and bypass the header ?"
At the start of MLVP's, I wanted to implement the header editor. But nobody was interested.
I think it could be useful.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 18, 2017, 03:48:19 AM
Quote from: AWPStar on May 17, 2017, 08:43:42 PM
At the start of MLVP's, I wanted to implement the header editor. But nobody was interested.
I think it could be useful.
Yes , You got my vote
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on May 18, 2017, 06:33:43 AM
Quote from: AWPStar on May 17, 2017, 08:43:42 PM
At the start of MLVP's, I wanted to implement the header editor. But nobody was interested.
I think it could be useful.


Quote from: reddeercity on May 18, 2017, 03:48:19 AM
Yes , You got my vote

+1
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 19, 2017, 12:50:13 AM
Quote from: AWPStar on May 17, 2017, 08:43:42 PM
At the start of MLVP's, I wanted to implement the header editor. But nobody was interested.
I think it could be useful.

see this (https://bitbucket.org/hudson/magic-lantern/commits/7a1a31459136ec2785bd049cf8f4bc077f7f11f3), might be useful.


-- MLV autopsy --
--skip-block <block#>        skip given block number
--skip-type <type>           skip given block type (e.g. VIDF, AUDF etc)
--extract <block#>           extract only the block at given into output file
--replace <block#>           replace block with data from given source file
--autopsy-file <file>        extract/insert from this file
--payload-only               features above extract/replace affect not the whole block, but only payload
--header-only                features above extract/replace affect not the whole block, but only header
--relaxed                    do not exit on every error, skip blocks that are erroneous
-m                           write only metadata, no audio or video frames
-n                           write no metadata, only audio and video frames
-I <mlv_file>                inject data from given MLV file right after MLVI header
-X type                      extract only block type


example use case:

./mlv_dump corrupt.mlv --extract 15 --header-only --autopsy-file header.bin
<edit header.bin>
./mlv_dump corrupt.mlv --replace 15 --header-only --autopsy-file header.bin -o fixed.mlv

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KeithWu on May 24, 2017, 03:05:14 AM
Hey guys im new here, but ive got some info that might help some people working on the 550D builds.

From my experiences with mlv_rec based on the 01-14 build
10 bit : works well except every second frame the top third of the image is shifted right. Making a constant shift in video (back and forth)
12 bit : no problems with frames however every second frame is a repeat of frame 1. therefore in a 24p video only 12 unique frames.
14 bit: works as expected
no dropped frames with mlv_rec which is v good.

Finally i can't really figure out how to compile and i noticed nothing new has been put out for the 550D.
So i was just wondering if someone could compile a new one (unless you guys stopped working on it)
because i would love to keep testing builds and giving feedback!

Thanks!


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: goldenchild9to5 on May 26, 2017, 10:28:01 PM
Can someone please share a stable version of the 10 - 12bit Build with me..  I've tried the last Build and I'm getting Pink frames all over the place, and I can't find the older Builds.   
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on May 26, 2017, 11:24:42 PM
https://builds.magiclantern.fm/jenkins/view/Experiments/job/raw_video_10bit_12bit/
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: goldenchild9to5 on May 27, 2017, 12:51:28 AM
@Walter Schulz Thank you.. appreciate it  8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mp52 on May 29, 2017, 06:04:31 AM
Hello,

I recently picked up a 550D and having some fun with ML. Currently am able to get continuous RAW 1280x326 2.35:1 12 bit @24fps cropped without sound using mlv_rec. As I understand it, sound does not work with any overridden frame rate settings - correct? This is using the last compressed bit build I can find for the 550D from Jan 14th.

No luck yet with 10 bit recording. Just a quick thanks to all who've developed (are developing) ML - I'm hooked! Also to echo Keith Wu's post from a few days ago - there are still a few of us 550D users out there keen to test some more recent builds if any of the gurus are up to it.

Quote from: KeithWu on May 24, 2017, 03:05:14 AM
Finally i can't really figure out how to compile and i noticed nothing new has been put out for the 550D.
So i was just wondering if someone could compile a new one (unless you guys stopped working on it)
because i would love to keep testing builds and giving feedback!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 30, 2017, 06:12:15 AM
Quote from: KeithWu on May 24, 2017, 03:05:14 AM
Hey guys im new here, but ive got some info that might help some people working on the 550D builds.
Thanks  but there isn't anything new since Jan. 2017
If you what to help I need more people to get involved with reverse engineer of Liveview
reading this (http://www.magiclantern.fm/forum/index.php?topic=19336.msg184661#msg184661) will help , some of my 10-12bit  development work  here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg184474#msg184474)


Quote from: KeithWu on May 24, 2017, 03:05:14 AM
Finally i can't really figure out how to compile .......
Believe it or not up until about 7 mouths ago I had no idea how to compile magic lantern
let alone compile different branches e.g. 10-12bit raw video etc. ....
So the best way was to jump in with both foot and make lots of mistakes & read lots of posts .
So ended up with the following , (should work on MAC also)
http://www.magiclantern.fm/forum/index.php?topic=18259.msg174934#msg174934
http://www.magiclantern.fm/forum/index.php?topic=18259.msg175475#msg175475

For the Digic4 cam users: There still a lot of people very interested in this , the download speaks for them selves .
(https://c1.staticflickr.com/5/4224/34171448933_5ed630ce20_z.jpg) (https://flic.kr/p/U4BBfD)
Updated 10-12bit downloads (https://flic.kr/p/U4BBfD) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

So yea I'm working on this , but very slowly.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mp52 on May 31, 2017, 01:08:42 PM
Thanks for the update reddeercity - encouraging stats and great to hear there's still activity for the Digic4 cam. I'd like to help out down the line once I'm familiar with the camera and ML (only had it a couple of weeks!). I am an OSX user and programmer - not C but have compiled C apps.

Right now I'm trying to determine what theexpected behaviour of crop_rec on the 550D - so I can establish whether it should be working or not in any ML builds. I'm currently running your 1 Dec 10/12bit build for which I assume crop_rec is not meant to be functional? The module is enabled but the setting is dimmed under all the combinations of video and bit-depth settings I tried. Am I right in thinking that, to invoke, crop mode, I just need to have the module enabled, set any native 720/1080p mode and push Zoom in LV once before starting to record? When I do this with this build the camera chokes almost immediately on record start and needs a reboot.

From crash log:
ASSERT: dwDeadLine != 0xFFFFFFFF
at LVState.c:5842, task LiveViewMgr
lv:1 mode:20

Magic Lantern version : Nightly.2016Dec01.550D109


Is there a recent build you can recommend which does have working crop_rec for the 550D (14bit). I'd like to see it working so in time I can find a way to help with getting 10/12bit crop_rec support available for this camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on June 01, 2017, 03:59:28 AM
crop_rec   (http://www.magiclantern.fm/forum/index.php?topic=17021.msg165196#msg165196)is mainly for 5d3 ,
The build has all possible modules and not necessary apply to All Cameras. The only modules you need are mlv_rec.mo , mlv_snd.mo & or raw_rec.mo(mlv_lite) but all my work is around full mlv+audio .
Here the source code for 10-12bit video , this is what I being working from , and my goal is 1:1 functional thou I think the best chance is 12bit as it is has less corruption (read over the last 15 pages on the this thread and I think you will understand)
https://bitbucket.org/hudson/magic-lantern/src/43176afffdca?at=raw_video_10bit_12bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: enricominisini on June 22, 2017, 04:06:48 PM
hi, i read all the discussion but i don't understand how to fix this problem with frames with the 600d

(https://thumb.ibb.co/dSkCxk/Cattura.png) (https://imgbb.com/)


thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on June 22, 2017, 04:34:53 PM
http://www.catb.org/esr/faqs/smart-questions.html#intro
And an unprocessed file to play with would be fine, too.

Ceterum censeo experimental nightly build page esse delendam.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on June 24, 2017, 10:44:13 PM
hey i just heard about the new 10 bit RAW recording option and i just wanted to know what the reason anyone would want to record 10 bit when there is already 14. is there like better quality or longer recording times at higher resolutions? thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on June 24, 2017, 10:48:56 PM
No, quality is not better, but 10 bit gives 29% smaller file sizes, which allows for more footage in the same space and longer record time for slow cameras.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on June 25, 2017, 09:34:44 PM
ooo nice . thats real neat. thank you. I am testing it out now on my 700D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on June 25, 2017, 09:53:56 PM
Quote from: enricominisini on June 22, 2017, 04:06:48 PM
hi, i read all the discussion but i don't understand how to fix this problem with frames with the 600d

(https://thumb.ibb.co/dSkCxk/Cattura.png) (https://imgbb.com/)


thanks

hey did you figure out how? i have the same problem with 700D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on June 25, 2017, 10:39:28 PM
oh wait someone back through the previous pages said to use mlv producer. download that .it worked for me.
https://sourceforge.net/projects/mlvproducer/files/latest/download?source=files
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DomeCGI on June 27, 2017, 05:50:22 PM
Can someone please advice me on the 10- 12 bit mode for the 7D ? it's works? wich resolutions e frame rates?
Is possible for ppl that have tried the new 10-12 bit mode add the data with new data sheets on the ML raw charts?

Thnak you all very much in advance.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on June 27, 2017, 07:28:47 PM
Lookup ML's Twitter account (https://www.twitter.com/autoexec_bin) and follow linked EOSHD page about "Magic Lantern Raw Video – Current camera capabilities, updated for 2017". If you follow the link you will get some info about DiGiC IV not supported. User Reddeercity is working on first DiGiC IV port for this feature but there is no release date. Don't hold your breath. 7D unique architecture (Dual DiGiC IV) -> Some times is sucks to be unique.
My advice: If there is a feature missing for your cam act like this feature will be missing forever.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Boboche on July 01, 2017, 07:50:33 PM
Any reasons why there's no builds for the 5d2 anymore? I've scrubbed the forums and while there was pink issues at the beginning, it looked like if everything was working fine?, I see the post above states its only 14bits but at some point there were still some features partially working, what's wrong exactly? Impossible to do due to architecture or more work needed?

cheers,
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: walter_schulz on July 01, 2017, 08:05:10 PM
Read post above yours.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on July 01, 2017, 09:40:16 PM
Just tested 10 12 bit 28 jun. Something completely wrong whithin the play module - all I can see - no image warning...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on July 02, 2017, 12:09:45 AM
Except for some asserts that pop up the console after a few seconds of playback, it works fine here. Solved the asserts - they were related to the fix for this issue (https://www.magiclantern.fm/forum/index.php?topic=19761).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on July 02, 2017, 01:32:57 AM
hey you guys something is wrong with the june update on this 10-12 bit raw experiment.
im using the 700D.
in the May update i could record 10bit
in a multitude of different resolutions easily and continuously.
but in this new June update i can not.
its like my write speed maxs out at around 33mbs per second. (hovers from around 27 to 33)
so i cant really record anything without it stopping after around 15 seconds.
i was recording 1600 at 2:1 aspect ratio and 1280 at aspect 4:3 ratio.
those are the only ones i really used although im pretty sure it applies to every scenario.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on July 02, 2017, 01:35:20 AM
also i never could play back 10 bit raw video with MLV_PLAY module.
even with the May update.
it would just show black
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on July 02, 2017, 07:45:50 AM
Screenshot showing your settings and speed difference?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on July 02, 2017, 11:27:28 AM
Quote from: a1ex on July 02, 2017, 12:09:45 AM
Except for some asserts that pop up the console after a few seconds of playback, it works fine here. Solved the asserts - they were related to the fix for this issue (https://www.magiclantern.fm/forum/index.php?topic=19761).

I could sucesfully replay this files using older builds...I read the topic you pointed but dont understand hot to fix the bug...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on July 02, 2017, 01:20:07 PM
Can you upload a small file that can be replayed in the older version, but not in the newer?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on July 02, 2017, 04:22:08 PM

(http://thumb.ibb.co/nsXVQF/20170702_171306.jpg) (http://ibb.co/nsXVQF)

(http://thumb.ibb.co/mALZkF/20170702_171353.jpg) (http://ibb.co/mALZkF)
Quote from: a1ex on July 02, 2017, 01:20:07 PM
Can you upload a small file that can be replayed in the older version, but not in the newer?

The first foto - playback whithin the newer build, the second 1012bit2017Apr025d3123
https://drive.google.com/open?id=0B9HplG9-d6ywZGVEZ3VZaW5oUkE
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on July 02, 2017, 05:04:31 PM
header start looks a bit uncommon.

[MLVI]
[RAWI][INFO][RTCI][EXPO][LENS][IDNT][WBAL][STYL][VERS]+++++[EXPO][LENS][RTCI][EXPO][LENS][WBAL]
[VIDF][NULL]
[VIDF][NULL]
[VIDF]...


is this recorded with mlv_rec instead of mlv_lite?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on July 02, 2017, 06:27:04 PM
can you check if you had a lot of .idx files on your card that wasn't able to play back?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on July 03, 2017, 01:34:17 AM
Quote from: g3gg0 on July 02, 2017, 06:27:04 PM
can you check if you had a lot of .idx files on your card that wasn't able to play back?

No I had only idx related to the footage. Also I deleted all IDX and try to replay again - nothing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on July 03, 2017, 01:39:43 AM
Quote from: g3gg0 on July 02, 2017, 05:04:31 PM
header start looks a bit uncommon.

[MLVI]
[RAWI][INFO][RTCI][EXPO][LENS][IDNT][WBAL][STYL][VERS]+++++[EXPO][LENS][RTCI][EXPO][LENS][WBAL]
[VIDF][NULL]
[VIDF][NULL]
[VIDF]...


is this recorded with mlv_rec instead of mlv_lite?
Yes
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on July 03, 2017, 05:33:19 AM
hey alex i will say that i used the exact settings on both versions and even made sure to  only activate  the raw record/play/sound module. installed the different versions both twice just to  make sure that the install didnt mess up the first time. but i will provide screen shots. what is it again that you want me to screen shot? the raw video settings and for the speedtest do you want me to use the card speed test in debugging mode? thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on July 04, 2017, 10:13:42 AM
The writing speed indicator from the MLV recording module (either during recording, or afterwards) should be fine.

I don't have a 700D to test, but I can try to reproduce on 5D3.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on July 07, 2017, 10:15:49 PM
i just test the 10bit expirimental build on 2 july 17 for 70D. just after  started recording the camera hangs. stop buttons or any other buttons wouldn't work so i have to pull out the battery. i can share a video very soon if needed to see what i mean


edit: while the camera is recordng if i press any buttons like stop button, my camera hangs so i have to pull out the battery. however the file is save in the memory card. i can't preview in  the camera but i can preview in my laptop with mlrarviewer
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on July 07, 2017, 10:48:28 PM
That's unlikely to help. A lot more helpful would be a detailed (!) list of what works and what not.

If raw video is not working at all, it could be worth trying other ways of capturing raw frames from LiveView (silent pictures, raw_diag) or other builds (e.g. lua_fix).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on July 07, 2017, 11:38:33 PM
oh hey alex sorry for the long response time but i went to go download the june version again to show you but it was already replaced by the july version sorry.
good thing is the july version doesnt seem to have the problem the june version had and compared to the may version seems to be even better! for instance the write speed seems to be more consistent. resolutions i use to use that technically did shoot continous (even though magic lantern had a yellow camera icon by it) now are recognized by magic lantern as a continous shoot.  so that feels good as far as usability goes.
one thing did happen when i tryed to play back the file though. it started gathering all the frames and then it slowly wrote this into existence (see images) and then it was just plastering onto my camera screen and the only way to get it off it seems was to turn off the camera(i could till see through it and it wasnt frozen)

(http://thumb.ibb.co/gLXj7v/20170707_155829_HDR_2.jpg) (http://ibb.co/gLXj7v)

(http://thumb.ibb.co/kgwoZa/20170707_163119_HDR.jpg) (http://ibb.co/kgwoZa)

(http://thumb.ibb.co/iUAFEa/20170707_162818_HDR.jpg) (http://ibb.co/iUAFEa)


btw i mean play back of 10bit raw recordings
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RainZ on July 12, 2017, 02:01:06 AM
How can I use this 12bit/10bit raw feature on my camera?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on July 12, 2017, 03:08:03 PM
With care or not at all.
Depends on the cam you have. Top of page -> Downloads -> Downloads (drop-down menu) -> Experiments.

@Admins: Please edit first post of this thread and add some links/text/instructions according to other threads.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on July 14, 2017, 12:54:10 PM
I see that 4k crop build has the fix of the black frame issue at the mlv_play module...Could anybody add this fix to 10_12 bit build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: histor on July 22, 2017, 03:12:36 PM
I see a problem with ghost images in this build. It shows the overlay, but immediately prints error log.
Looks like it was resolved in Unified. https://bitbucket.org/hudson/magic-lantern/commits/e0c1b48a2558975e23545eaeea9eced6de72ab97 (https://bitbucket.org/hudson/magic-lantern/commits/e0c1b48a2558975e23545eaeea9eced6de72ab97)
ML ASSERT:
count <= 8192
at ../../src/fio-ml.c:528 (FIO_ReadFile), task cls_task
lv:1 mode:3


Magic Lantern version : raw_video_10bit_12bit.2017Jul03.600D102
Mercurial changeset   : c38f583da163+83173fc0953b+ (raw_video_10bit_12bit) tip
Built on 2017-07-02 22:50:32 UTC by jenkins@nightly.
Free Memory  : 208K + 795K
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: papasapien on July 26, 2017, 07:32:49 PM
okay so i have a theory on why playback doesnt work for 10-12 bit recordings (ive only tested 10 bit)(tested 14 bit to and looks to be the same)
i have a 700D
so playback only seems to work when a file is under 1GB, anything over 1GB will bring up the console message saying something about fio_malloc, and it wont go away until you restart your camera or go to the debug settings and turn on and off show console.
it could be something like, anything over 978mb or something random but 1GB seems to be the safe bet here.
so maybe a fix could be since it only shows up on files over 1GB then maybe there is something that says files over 1GB display message or something like that. so all that would need to be done is to find out whats telling it to  say that and tell it not to? i dont know, im not a coder or anything. its probably more complicated then that but its just a theory.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on July 27, 2017, 01:39:08 AM
Yes it's easy to get rid of the error message, but the message is not the problem, it was intentionally put there to highlight the underlying bug and encourage someone to fix it.

http://www.magiclantern.fm/forum/index.php?topic=19761.msg186060#msg186060
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tonij on August 01, 2017, 02:50:32 AM
7D Users,

I played around with the 10/12bit Magic Lantern build for 7D found here: https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.7D203.zip  (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.7D203.zip)
I was shocked when I managed over 2min of continuous 2520x1200 video @ 24fps with working Liveview and in camera playback! I used a Lexar 64GB 1066x card.

Here's a guide for noobs like myself who wish to try it out.

-In Modules menu enable mlv_play.mo, raw_rec.mo and raw_twk.mo.
-In Movie menu set FPS override to 24 FPS Exact.
-Enable RAW video and set Bit depth to 10, return to Liveview and do 5x zoom, return to Movie menu set resolution to 2520x1200,
-In Overlay menu set Global Draw to OFF, return to Liveview in 5x zoom
-Hit record!

When the camera is turned off and in some other situations when you attempt to record in 5x zoom mode Liveview will freeze. Go back to Overlay menu, enable Global Draw again, return to Liveview 5x zoom, go back again and disable Global Draw, return to Liveview one last time, hit record and you will again be able to view what you are recording.

Most of the time I get around 1min40sec of continuous 2.5k video :)

Edit: I've noticed there's a few pixles in the bottom right corner of the video that do strange things..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: flofifull on August 02, 2017, 02:27:53 PM
Quote from: tonij on August 01, 2017, 02:50:32 AM
7D Users,

I played around with the 10/12bit Magic Lantern build for 7D found here: https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.7D203.zip  (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.7D203.zip)
I was shocked when I managed over 2min of continuous 2520x1200 video @ 24fps with working Liveview and in camera playback! I used a Lexar 64GB 1066x card.

Here's a guide for noobs like myself who wish to try it out.

-In Modules menu enable mlv_play.mo, raw_rec.mo and raw_twk.mo.
-In Movie menu set FPS override to 24 FPS Exact.
-Enable RAW video and set Bit depth to 10, return to Liveview and do 5x zoom, return to Movie menu set resolution to 2520x1200,
-In Overlay menu set Global Draw to OFF, return to Liveview in 5x zoom
-Hit record!

When the camera is turned off and in some other situations when you attempt to record in 5x zoom mode Liveview will freeze. Go back to Overlay menu, enable Global Draw again, return to Liveview 5x zoom, go back again and disable Global Draw, return to Liveview one last time, hit record and you will again be able to view what you are recording.

Most of the time I get around 1min40sec of continuous 2.5k video :)

Edit: I've noticed there's a few of pixles in the bottom right corner of video that do strange things..

Thanks Tonij !
It's working !
I've made some test this morning and i get around 73 seconds of 2.5K 10-bit with 1000x card :

https://vimeo.com/228049559
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on August 04, 2017, 06:01:11 PM
50d and 7d are both digic 4. I don't have my 50d atm. Could a 50d owner kindly check this on their 50d?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on August 04, 2017, 11:38:38 PM
Not sure that anyone has a 50D build out there, but then again, I didn't know that there was a 7D build either. Neither are listed on the Experiment Downloads page.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Malik on August 06, 2017, 01:35:33 AM
In Canon 70D (firmware 1.1.2) 10 and 12 Bit do not work, the camera stops. Further use without removing the battery is not possible.
There is no menu item to select the number of frames rate
This is what is written on the memory card   https://yadi.sk/i/FJIQ-wbe3LkRDs (https://yadi.sk/i/FJIQ-wbe3LkRDs)
This is how the menu looks after hanging up https://yadi.sk/i/ZH4eFzWl3LkR9d (https://yadi.sk/i/ZH4eFzWl3LkR9d)
After a couple of minutes error 80 or 70 appears
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tonij on August 06, 2017, 04:09:42 AM
50D
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.50D109.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: halbmoki on August 06, 2017, 01:24:46 PM
Thank you, tonij. I'm happy to test the bleeding edge builds, but have a hard time getting into making them myself.

I recorded a few seconds of 14, 12 and 10bit each at 1280x544px and converted to DNGs with MLVmystic. Seems to work except for a few corrupted frames (top part hyper-saturated and magenta) with 10bit recording.

So it seems to almost-work on 50D. I'll do a few more tests next week... Anything special I should do?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on August 06, 2017, 10:53:13 PM
Yes thank you tonji. Excited to test on my 50D. Did not have good luck with the 550D build, but I thought it was a common issue for Digic 4 cameras.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on August 07, 2017, 05:49:26 PM
Quote from: halbmoki on August 06, 2017, 01:24:46 PM
...converted to DNGs with MLVmystic. Seems to work except for a few corrupted frames (top part hyper-saturated and magenta) with 10bit recording.

Any reason why to use MLVmystic instead of the latest and greatest coming from those who have constantly been hard at work (for obvious reasons) by keeping things up to date with all the bells and whistles in their apps/converters especially when it comes to using experimental stuff?

Also do you run on Mac or Win?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Francis Frenkel on August 07, 2017, 06:45:42 PM
Quote from: DeafEyeJedi on August 07, 2017, 05:49:26 PM
Any reason why to use MLVmystic instead of the latest and greatest coming from those who have constantly been hard at work (for obvious reasons) by keeping things up to date with all the bells and whistles in their apps/converters especially when it comes to using experimental stuff?

I use MLVMystic (on windows 7) because it's perfectly stable.
What are for you the best "uptodate" applications for dng ?

Francis
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on August 07, 2017, 06:48:47 PM
https://bitbucket.org/Dannephoto/batch_mlv/overview
Batch_mlv?
Runs Bouncyball beefed up mlv_dump version and produces cdng files to any bitrate you like.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: halbmoki on August 13, 2017, 12:09:02 PM
Quote from: DeafEyeJedi on August 07, 2017, 05:49:26 PM
Any reason why to use MLVmystic instead of the latest and greatest coming from those who have constantly been hard at work (for obvious reasons) by keeping things up to date with all the bells and whistles in their apps/converters especially when it comes to using experimental stuff?

Also do you run on Mac or Win?

I run Win7 64bit.
Some time ago, MLVmystic was the most stable and easy to use, so I just stuck with that. Always keeping up with the latest and greatest development is fun, but sometimes "working well enough" is all I need. If you could point me in the right direction for a current converter with a decent interface, I'd be happy to try something else, though. Maybe the corrupted frames at lower bitrates came from using an outdated converter, not from the recording itself.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on August 13, 2017, 09:11:56 PM
Hello, I was playing around with 10 bit and 12 bit (1st time testing them so I think I missed a lot)
Using version of july03. on a 6D.
There is something that I think I missunderstood, wanted to shoot in 10 bit so I could go higher with the resolution, but I couldn't  get higher than 1823*1029 on the 6D, instead I got 7 seconds of filming.
Usually on the 14bit (of the nightly build) I could film between 5-6 seconds. So I thought it wasn't a big deal.
tried the 14 bit of the experimental build i couldn't get to 4 seconds. any Idea why?
I tried to install the nightly build again , I was able to shoot 5-ish. why is there a difference between the two 14 bits?
note that I'm filming the same subject in all the tests.

And my other question is if I choose 10 bit is there a way to film with higher res and instead of more seconds?
Thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on August 15, 2017, 06:02:48 AM
Perhaps a sticky for this thread
- latest build links, some like 50D, 70D are not regularily updated
- latest mlv raw comverters for 14bit, 12 bit & 10bit

I think these two stickies type thread might speed things along, as these get asked often ...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on August 17, 2017, 08:07:50 AM
@kyrobb Please update with your 50D experience.

Quote from: kyrobb on August 06, 2017, 10:53:13 PM
Yes thank you tonji. Excited to test on my 50D. Did not have good luck with the 550D build, but I thought it was a common issue for Digic 4 cameras.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on August 19, 2017, 04:05:29 AM
50D for me has the same issues in 10 and 12 bit as my 550D. Top third of frame turns pink and green every other frame while bottom two thirds constantly jumps back to frame 1. I've seen the same reported of the 5D2. Has anyone had confirmed success with the 7D? I understand these are all Digic 4 cameras, although the 7D is dual correct?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on August 19, 2017, 07:31:08 AM
"Has anyone had confirmed success with the 7D?"
Yes, but only in zoom crop mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Zeusjuice on August 20, 2017, 04:25:17 AM
Hello,

Im new to magic lantern and have the 70D.  I downloaded and successfully installed the latest beta nightly build but that was before I saw this download page for experiments. I want to be able to do 10 bit raw recording. Am i suppose to uninstall the Beta nightly build and then install the experimental?  Or do I keep the original beta magic lantern files installed and add this experiment download to it?  Any help is greatly appreciated as I am a complete noob.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on August 20, 2017, 10:45:16 PM
Just posted new builds for 70D and 100D - do they finally work with 10/12-bit recording?

Also, the fio_malloc issue in mlv_play should be solved, although I didn't get the error here. Let me know if it still doesn't work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: markkro88 on August 21, 2017, 05:37:54 AM
hey alex,

im getting that fio_malloc issue in mlv_play error on the 6d for 10bit and 12bit with the august 20th update for files over 1gb. can you please fix it so i can preview the recordings.  Also the shutter speed is showing 0" when its actually at 50. not sure why that is happening since it was working fine on the July update. thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on August 21, 2017, 08:15:22 AM
Sorry, unable to reproduce either error on 5D3...

For the shutter speed issue, I don't even know where to look (screenshot, please).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: markkro88 on August 21, 2017, 09:22:41 AM

(https://thumb.ibb.co/nKaK6k/IMG_0517.jpg) (https://ibb.co/nKaK6k)

(https://thumb.ibb.co/jpPOXQ/IMG_0515.jpg) (https://ibb.co/jpPOXQ)

(https://thumb.ibb.co/eDFxsQ/IMG_0516.jpg) (https://ibb.co/eDFxsQ)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on August 21, 2017, 10:50:56 AM
Found - compiled the older sources by mistake (so the build was identical to Jul03). New builds posted.

That means the shutter speed issue depends on some of your settings that were changed since July. I'm unable to trigger this behavior on 5D3, even on auto modes. Did a few checks in QEMU, but could not find anything obviously wrong. What does the FPS indicator show? (top info bar, on the right side)

If you can compile ML, please print the values from fps-engio.c, get_current_shutter_reciprocal_x1000.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on August 21, 2017, 03:38:23 PM
Quote from: a1ex on August 20, 2017, 10:45:16 PM
Just posted new builds for 70D and 100D - do they finally work with 10/12-bit recording?

Also, the fio_malloc issue in mlv_play should be solved, although I didn't get the error here. Let me know if it still doesn't work.

everything works like a charm so far. no more playback issue. no more wrong black level issue. thank you so much for this build@Alex
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on August 22, 2017, 02:27:26 PM
I have a small request that I tried to do myself a ~a month ago but couldn't find where all the settings variables were to create a new one, and not certain where bitdepth is even set.

Basically here's the idea: For Digic IV cameras (ones where 10/12 bit doesnt work) - have a seperate bit depth selector in the menu for crop mode and full frame, as 10/12 bit seems to work in crop mode only for these cams. Maybe do it with something like #ifdef (5d2|50d|7d etc...)

Reason: if 10/12 bit setting is accidentaly left on after using crop mode, it can ruin some shots afterwards. (speaking from experience  ;) )
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on August 23, 2017, 01:30:50 PM
What max res is available on 7D, 5d2 or 50d ? Irrespective of  frame corruption.
I think we can get 2.5K max in crop mode on these cameras. Is anyone able to go higher than 1080P ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on August 23, 2017, 02:22:53 PM
You are post #1339
reading #1314 should answer your question regarding 7D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on August 23, 2017, 08:12:03 PM
@nikfreak Thanks.

The mapping from 14-bit to 10-bit is currently linear.

Here is the log mapping for SI-2K camera - output = Log base 90 (input*89+1).

> http://cineform.blogspot.in/2007/09/10-bit-log-vs-12-bit-linear.html

This is a pretty standard log profile that has LUTs for LOG to LIN conversion. Also does a better job with 10-bit RAW compared to linear.

@A1ex Any chance we can try get a log profile in here??
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: OlRivrRat on August 23, 2017, 08:36:04 PM
   Does this take some Specific Set of Settings to make It work ? On SL1 I get 2-3Sec's of Rec

& then it Quits ~

                     ORR ~ DeanB
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on August 23, 2017, 08:46:32 PM
Quote from: zcream on August 23, 2017, 08:12:03 PM
@A1ex Any chance we can try get a log profile in here??

If you can figure out how to change the linear mapping using Canon's image processing modules (since the CPU is way too slow for this), then yes. It's a matter of figuring out how to configure the image processing paths. You'll have to find out what to use for a LUT, gamma correction or any other nonlinear operation.

There is ongoing research on this (see e.g. EDMAC internals, raw_twk, lossless compression/decompression, EekoAddRawPath, ResLock, dm-spy-experiments). Feel free to investigate and share your notes (what you have tried, what was the result and so on).

Some low-hanging fruits to get started: FA_MaxSelectTestImage, FA_SubtractTestImage.

Canon's RAW to JPG processing paths and their creative effects from recent models are probably good choices to look at. Start by getting detailed logs on the dm-spy-experiments branch (https://www.magiclantern.fm/forum/index.php?topic=2388.0).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Zeusjuice on August 25, 2017, 08:31:38 PM
I'm currently using the new experiment 10 / 12 bit build on my 70D.  Here are my problems so far unsure if it is related to the build or just my workflow any help diagnosing is extremely appreciated and would receive hero status in my book for any help given.

There is a delay sometimes until actual recording registered in the footage.
I was able to get some footage later and everything was working really  really well.  I did 30 seconds of test footage but after converting mlv files to cdng using mlvmystic and dropping them into davinci resolve 12.5  - all the footage had unstable frame rates and would play back differently each time.  it would go from 2 frames per second to 24 with missing frames in between.  ( the raw footage i did see played back even though out of frame rate looked amazing though so I'm close)

I made sure to set the settings using a lower resolution and in an aspect ratio that showed green for continuous recording with room to spare and using a card with a high write speed that far exceeds my cameras ability.

Is this a local problem when i converted the MLV files using mystic? ( mystic i see is old and I had no luck with the converters i found on ML so far)
Is this a local problem for my computer ? my computer only has 4gb of ram which I know is like bare minimum but I'm using davinci resolve 12.5 which i thought would work with my set up in 10 bit
Is there a setting I'm unaware of in ML that would keep the frame rate stable?

Or is this build just not stable yet?

I'm very new to ML and resolve and the problems I am experiencing could be just a local issue I'm not catching with my inexperience.

Either way I'm really happy there are pro's working on this 10 bit raw recording I think that's so freaking cool and I'm very excited to see where this goes and thank you for working on these builds so we can get these great features out of our canon cameras.  ML is a big part of why I got a canon 70D.

Again Thanks in Advance for any help or suggestions offered by our members here.

Thanks,
Zeus
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on August 25, 2017, 09:04:51 PM
@Zeusjuice:

Probably mystic. Try one of the alternatives mentioned in the linked post:
http://www.magiclantern.fm/forum/index.php?topic=10466.msg187771#msg187771
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Zeusjuice on August 25, 2017, 09:55:01 PM
@nikfreak:

Thank you so much, not just for your help today but for all that you do!! 

Thank You,
Zeus
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on August 26, 2017, 12:03:25 PM
I've downloaded the latest experiment version for the 6D (21august)

1-When u activate the FPSoverride the whole image becomes brighter and overexposed , and looks like shutter speed doesn't affect it the only thing that worked was going to f8 or f16, didn't have that prob with previous version (Jul03)
**FPS settings: 24fps, fix, and the rest didn't touch it

2-I'm also getting this error every time I start recording on an empty SD card (error was in both versions jul03 and august21)
   "FAILED: "WBAL3: queued:7 failed: 1 (requeued)" and I only get it once it doesn't seem to affect the video though or WB, and have a feeling I only get it when there's nothing on the SD card and I hit Record
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Malik on August 26, 2017, 04:00:07 PM
Canon 70D

10 Bit  (http://thumb.ibb.co/m1NeSQ/12.png) (http://ibb.co/m1NeSQ)

12Bit  (http://thumb.ibb.co/dBbpSQ/10.png) (http://ibb.co/dBbpSQ)

When recording there are no problems, the problem is that color noise is recorded.
The section chose FPS - no (raw-twk).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on August 26, 2017, 06:46:16 PM
Probably been asked before, but is 8 bit raw a possibility? Would it offer any improvement in video quality over h.264? Just thinking about cams with slower write speeds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on August 27, 2017, 12:29:14 AM
Take a 14bit MLV file and output it as 8bit and have a look at the output. Use mlv_dump on steroids for this:
mlv_dump -b 8 --dng Input.MLV
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on August 28, 2017, 12:54:22 AM
In my very unprofessional opinion, 8 bit raw files still look better than the standard H.264 videos. I think they're still sharper. Recovery in shadows and highlights is no comparison to 14 bit, but I still feel it could be useful for people with slower cams. But is it possible?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on August 28, 2017, 01:09:31 AM
Can you upload 8bit example files where it shows no posterization for example?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on August 28, 2017, 01:19:33 AM
I tested this some time ago, and not very thoroughly, but I don't remember seeing posterization problems. Perhaps I didn't output as 8 bit correctly. Will have to retest. Thank you for the info.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on August 30, 2017, 08:53:55 AM
Quote from: keepersdungeon on August 26, 2017, 12:03:25 PM
I've downloaded the latest experiment version for the 6D (21august)

1-When u activate the FPSoverride the whole image becomes brighter and overexposed , and looks like shutter speed doesn't affect it the only thing that worked was going to f8 or f16, didn't have that prob with previous version (Jul03)
**FPS settings: 24fps, fix, and the rest didn't touch it

2-I'm also getting this error every time I start recording on an empty SD card (error was in both versions jul03 and august21)
   "FAILED: "WBAL3: queued:7 failed: 1 (requeued)" and I only get it once it doesn't seem to affect the video though or WB, and have a feeling I only get it when there's nothing on the SD card and I hit Record
So nobody is having the errors mentioned above? If I'm missing something would really like some help from you guys.
Thanks

Sent from my Pixel using Tapatalk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: OlRivrRat on August 31, 2017, 02:34:44 AM
      @KeepersDungeon

   Quote from: keepersdungeon on August 26, 2017, 12:03:25 PM

I've downloaded the latest experiment version for the 6D (21august)

   1-When u activate the FPSoverride the whole image becomes brighter and overexposed , and looks like shutter speed
doesn't affect it the only thing that worked was going to f8 or f16, didn't have that prob with previous version (Jul03)
   **FPS settings: 24fps, fix, and the rest didn't touch it

   In FPSoverride do You have "OptimizeFor" set to "LowLight" ~ If so switch to some other pram' & see it that helps ~

                                 ORR ~ DeanB
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on September 02, 2017, 03:45:03 PM
Quote from: OlRivrRat on August 31, 2017, 02:34:44 AM
      @KeepersDungeon

   Quote from: keepersdungeon on August 26, 2017, 12:03:25 PM

I've downloaded the latest experiment version for the 6D (21august)

   1-When u activate the FPSoverride the whole image becomes brighter and overexposed , and looks like shutter speed
doesn't affect it the only thing that worked was going to f8 or f16, didn't have that prob with previous version (Jul03)
   **FPS settings: 24fps, fix, and the rest didn't touch it

   In FPSoverride do You have "OptimizeFor" set to "LowLight" ~ If so switch to some other pram' & see it that helps ~

                                 ORR ~ DeanB
Hello, thanks for you reply.
I tried them all they all did the same thing on x5 or x10 the image becomes brighter

Sent from my Pixel using Tapatalk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: overflow on September 15, 2017, 07:58:30 PM
Quote from: tonij on August 01, 2017, 02:50:32 AM
7D Users,

I played around with the 10/12bit Magic Lantern build for 7D found here: https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.7D203.zip  (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.7D203.zip)
I was shocked when I managed over 2min of continuous 2520x1200 video @ 24fps with working Liveview and in camera playback! I used a Lexar 64GB 1066x card.

Here's a guide for noobs like myself who wish to try it out.

-In Modules menu enable mlv_play.mo, raw_rec.mo and raw_twk.mo.
-In Movie menu set FPS override to 24 FPS Exact.
-Enable RAW video and set Bit depth to 10, return to Liveview and do 5x zoom, return to Movie menu set resolution to 2520x1200,
-In Overlay menu set Global Draw to OFF, return to Liveview in 5x zoom
-Hit record!

When the camera is turned off and in some other situations when you attempt to record in 5x zoom mode Liveview will freeze. Go back to Overlay menu, enable Global Draw again, return to Liveview 5x zoom, go back again and disable Global Draw, return to Liveview one last time, hit record and you will again be able to view what you are recording.

Most of the time I get around 1min40sec of continuous 2.5k video :)

Edit: I've noticed there's a few of pixles in the bottom right corner of video that do strange things..

Thanks tonij, I followed your step by step instructions and I manage to record in 10 bit successfully on my 7d. However, no matter what I do the liveview always freezes. How did you manage to see what you're recording?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tonij on September 20, 2017, 11:10:20 AM
Quote from: overflow on September 15, 2017, 07:58:30 PM
Thanks tonij, I followed your step by step instructions and I manage to record in 10 bit successfully on my 7d. However, no matter what I do the liveview always freezes. How did you manage to see what you're recording?

Did you follow my guide exactly? When it does freeze do this:

Go back to Overlay menu, enable Global Draw again, return to Liveview 5x zoom, go back again and disable Global Draw, return to Liveview one last time, hit record and you will again be able to view what you are recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: overflow on September 24, 2017, 04:48:00 PM
Quote from: tonij on September 20, 2017, 11:10:20 AM
Did you follow my guide exactly? When it does freeze do this:

Go back to Overlay menu, enable Global Draw again, return to Liveview 5x zoom, go back again and disable Global Draw, return to Liveview one last time, hit record and you will again be able to view what you are recording.

Thanks again. There's something odd, I tried your step by step instructions and it still freezes. Do you have any other special settings on ML?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: overflow on September 28, 2017, 01:54:31 PM
Quote from: reddeercity on May 08, 2017, 01:44:36 AM
Working on 10-12bit 1:1 on 5D2 , as we know 3x crop works with 10 & 12 bit reduction but not 1:1 really well .
Thou in 12bit 1:1 you can record with corruption on every other frame on the top 100 or so lines .
I'm currently seeing the feasible of using bit reduction in a up coming documentary I been hired to shoot in 2 weeks for the sierra club (usa)
My needs are long shoot times , so low bite rate fits the bill .
Here a list of resolution's size I was possibly thinking of using with 12bit in 1:1 (as there corruption on the top of frame)
with the adjusted crop to remove corruption. I also listed the file size for 60 second or so to get max time for minimum card space
The base line is 14 bit 1856x928 23.976 fps (My minimum spec. for HD broadcast as it need to be up upscaled)
1856x928  23.976 14bit 59.9 seconds 1437 frames 4.05 GB 1:1 Liveview  ( 2:1 A.R.)
1856x1114 23.976 12bit 64.8 seconds 1554 frames 4.51 GB (crop adjust 1856x988) 126 line crop 1.878542510121457 A.R.
1856x1044 23.976 12bit 64.8 seconds 1554 frames 4.23 GB (Crop Adjust 1856x953) 91 line crop  1.947534102833158 A.R.
1856x1004 23.976 12bit 60.3 seconds 1447 frames 3.78 GB (crop Adjust 1856x932) 72 lines crop 1.991416309012876 A.R.
1856x928  23.976 12bit 60.1 seconds 1442 frames 3.49 GB (crop adjust 1856x895) 33 lines crop 2.073743016759777 A.R.
   

So There is really no saving after cropping , but 3x crop mode is a different matter all together .
Here 10bit really save me , as I can have to 20min per 64GB Card @ 1920x1076 @ 23.976fps
Thou the field of view is reduced , looks like the 24mm will gets a work out  :D
1920x1076 23.976 10bit 59.9 seconds 1437 frames 3.47 GB  (3x Crop Mode)  1.784386617100372 A.R.
So I save about 500 MB per minute which will add up fast. 4x 64gb card should be around 80 min's of 10bit 2K or 1920x1076.
Now I can use my slower 60MB/s CF cards to get
1856x928 23.976  10bit 60.1 seconds 1443 frames 2.91 GB  (3x crop)  (2:1  A.R.)

It looks like I'll being shooting in 10bit 3x crop for this doc.

Now for the real reason I started this post , as I said I'm work on 1:1 liveview 5d2 10-12bit.
I've being working on older code with .raw , just to keep it as simple as possible before I move on to .mlv full & mlv_lite.
At this Point I can record 10bit & 12bit , with out any lockups or screen freezes all overlays are working while recording bit reduction.
When not in bit reduction it return to 14bit and of course it work without problems.

Now the Problem  ::) there is always a problem , my issue is decoding the .raw to  dngs
My question is can I convert the .raw to .mlv ? as mlv_dump seem not to understand .raw
I'm currently uploading some of the .raw file to my Google drive to share , when done I'll post the links.

I couldn't get the 7d to work without freezing the liveview. So I bought a used 5d2 and spent last evening modifying and compiling the code on a trial and error. I can now get no corruption consistently by disabling the double buffer, but there's a bit of image shifting appearing at the top part from the previous frame.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on September 29, 2017, 08:56:24 AM
It is interesting that some cameras that haven't been coded to work with 12/10bit seem to be working fine in crop mode. What I recently discovered is that some of the cameras in the 12/10bit branch are not working in crop mode (the 600D "Move crop mode). I'm testing with the EOSM and 700D. Anyone care to try it out and report back?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on September 29, 2017, 11:27:44 AM
Quote from: dfort on September 29, 2017, 08:56:24 AM
What I recently discovered is that some of the cameras in the 12/10bit branch are not working in crop mode (the 600D "Move crop mode). I'm testing with the EOSM and 700D. Anyone care to try it out and report back?

70D works fine
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on September 29, 2017, 11:52:58 AM
100D works fine too, even with sound!  Just playback of already recorded files yields a black screen but this used to work on previous builds, so I guess, it could be easily fixed.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on September 30, 2017, 07:20:02 AM
Thanks for the feedback. What I should have said was if you're seeing flickering on the bottom of the LV when recording in the 600D Movie crop mode.

By the way, it has been a while since unified was merged into this branch so I gave it a shot and it merged cleanly. Tested on the EOSM and 700D (firmware version 115) and it is working fine--though it still has the flickering I just mentioned.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: JimiHendrix on October 19, 2017, 06:23:01 PM
Hi, I am using this build with my 6d for 10bit/12bit video and am encountering a strange shutter speed bug that I don't get on the main Nightly Build. It seems that whatever I set the manual shutter speed too (M mode) in Canon it always displays 1/27.01th 333deg TV5.6 in the expo menu and it won't let me change it. It displays a different figure if I change the frame rate but seems impossible to get the standard 1/50th sec. I've tried disabling things and all kinds of combinations to no avail.

Any help would be much appreciated! Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on October 21, 2017, 01:20:34 AM
Is it a a display issue, or does the image brightness stay fixed as well?

The 6D is blindly maintained - can you record a video of the camera screen to show the issue, and also the difference from the regular nightly build?

In manual movie mode at 25 fps, ML should show 1/33.333 when Canon shutter speed is set to 1/30, and 1/50.000 when it's 1/50. Do you get these values on the nightly build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: JimiHendrix on November 02, 2017, 07:57:44 PM
Hi, thanks a lot for your reply and apologies for my slow response!

I've made video showing how my shutter speed is set to 1/50th and ISO 160 in manual mode. I then go to the ML Menu and it shows 1/27.01th 333deg TV5.6 ...the video is here https://drive.google.com/file/d/0B4n8mHr_jKcLUVY2Vy1GTmFXWEU/view?usp=sharing

I also notice (and you can see in the video at 0.31 secs) the ISO seems to change to 200.

With the regular nightly build I get the values you have said in your message.

Thanks again!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 03, 2017, 07:24:51 AM
Unable to evaluate the issue from Canon info screen - there are some important details not printed there. Press INFO to show ML screen.

Unable to locate the comparison with regular nightly (must be with the same settings in Canon and ML menus).

What happens at other shutter speeds?

What are your modified ML settings? (Delta menu; in particular, FPS override and Shutter fine-tuning)

From what I could tell, the ISO changes were triggered by autofocus (Canon firmware does that). Then, ML's gradual exposure makes the change a bit more obvious, but that only happens one second after AF, so it shouldn't be a show stopper.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PlantDaniel on November 06, 2017, 02:00:51 PM
I have issues with my 6D recording 10 bit 60fps RAW Video (using the highest resolution possible)

1. Extreme Aliasing: I have only shot 24fps RAW on a 70D before, which had pretty crippling aliasing already. The 6D tops it though - I don't know whether that is a known issue of the 6D or if it is due to the stretching I have to apply in post when working with Slow Motion footage. Anyways - any suggestions on how to decrease aliasing (in camera or in post)?

2. Weird Colour Artifacts: In Landscape shots, where a lot of the scene is in focus, I get weird colour lines of purple/blue and orange colour in my image. They seem to be only present in the area where the focus sits. Is that a known issue, if so: What can I do about it?

Here is a video showcasing the problems, pay attention to the structure of the rock, where the focus sits: https://www.youtube.com/watch?v=_PvSyJ_RX5g

I am using the 10 bit/12 bit nightly build fro Aug21 2017.

Post processing was as follows: Convert MLV to DNG via raw2cdng 1.7.9 - import into Davinci Resolve 14. No colour correction was added, the clip was slowed down to 24fps in post. Resolve is not the issue - I have rendered it with After Effects too, same problems.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: OlegK on November 23, 2017, 06:33:00 PM
Hello Guys!
I installed Magic Lantern with 10-12 bit support on a 6D, and noticed one bug.
Shutter number seems to be stucked, when I change it its changes, I can see it but number not changing, its always show 26.
How can I fix it?
.
(http://thumb.ibb.co/cUN3Nm/1.jpg) (http://ibb.co/cUN3Nm)

cheap image hosting (http://imgbb.com/)



On other builds it works just fine.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on November 24, 2017, 05:16:15 PM
i wait good mode for 50d  ::) now badly frames in 10bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2017, 08:06:06 PM
I might have something that is starting to work for Digic IV cameras that use LVState instead of EVF_STATE. This includes the 5D2, 50D, 500D, 550D and 7D. It is still an early work in progress but using the the excellent work @nanomad did with the 1100D on the allocate-raw-lv-buffer branch and some hints from @a1ex I was able to get CONFIG_EDMAC_RAW_SLURP working on LVState cameras. (Note: Canon uses the LVState string but in the ML code it is called LV_STATE.)

Now for the weird part. It used to be that these cameras sort of worked with 10bit/12bit when in zoom mode but not in regular mv1080 and mv720 modes. With this build mv1080 works but the other modes don't. In fact if you switch over to another mode then back to mv1080, it stops working. Deleting the SETTINGS folder from the card clears it up. Any help with this would be greatly appreciated.

Here is my work in progress: https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/16/10bit-12bit-for-lvstate-cameras/diff

I also put up test builds on my Bitbucket downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) for the 7D, 5D2, 50D, 550D and 1100D. I still need to get the SRM_BUFFER_SIZE for the 500D in order to get that camera working. I'll make it easy, use the find-SRM_BUFFER_SIZE build also on the downloads page. Simply take a simple silent still and it should show up on the screen.

The 1100D is a special case. I suspect that the 10bit/12bit build on the experiments page probably isn't working. Even though it is an EVF_STATE camera it might need the allocate-raw-lv-buffer method so if you have access to an 1100D please test and report. I could also use the SRM_BUFFER_SIZE for the 1100D but I can't help you with a special build because the silent module isn't compiling on that camera.

Speaking of the silent module, it isn't compiling in the raw_video_10bit_12bit branch either. Need to look into that.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: masc on December 06, 2017, 08:34:37 PM
Great progress! I tried out your version for 5D2.
Tested standard setting + mlv_rec enabled + 1856x1044 + 10bit/12bit/14bit.
Unfortunately, the picture on the screen starts flickering when starting recording. It is okay again, when recording is stopped. I watched the clips on the computer: it looks the same as on the cameras screen. That happens in all 3 bitdepth modes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2017, 08:45:55 PM
@masc - thanks for the report. I just uploaded a allocate-raw-lv-buffer version for the 5D2. Could you try it and report back? It will look and behave like an older build before mlv_lite but the raw video backend works differently.

Found the problem with the silent module not compiling in the raw_video_10bit_12bit branch:

-    raw_set_preview_rect(raw_info.active_area.x1, raw_info.active_area.y1, raw_info.active_area.x2 - raw_info.active_area.x1, raw_info.active_area.y2 - raw_info.active_area.y1);
+    raw_set_preview_rect(raw_info.active_area.x1, raw_info.active_area.y1, raw_info.active_area.x2 - raw_info.active_area.x1, raw_info.active_area.y2 - raw_info.active_area.y1, 1);
...
-            raw_set_preview_rect(raw_info.active_area.x1, raw_info.active_area.y1, raw_info.active_area.x2 - raw_info.active_area.x1, raw_info.active_area.y2 - raw_info.active_area.y1);
+            raw_set_preview_rect(raw_info.active_area.x1, raw_info.active_area.y1, raw_info.active_area.x2 - raw_info.active_area.x1, raw_info.active_area.y2 - raw_info.active_area.y1, 0);


However, a better fix would be to merge unified because the raw_video_10bit_12bit branch is falling way behind. It still has 700D.114 instead of 115.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: masc on December 06, 2017, 09:00:34 PM
Tried your allocate-raw-lv-buffer.2017Dec06.5D2212.zip:
- I activated mlv_rec
- after restart and LV enable, in the left corner below is a "BUSY"
- I activated 1856x1044 again
- start recording - picture on screen is no more actualized
- stop recording - live view is back
Then I watched the clip on the computer: it looks corrupted in the same way as the other 10/12/14 bit clips.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2017, 09:13:23 PM
Darn it.

Oh well, this is a work in progress. Any hints would be greatly appreciated.

[EDIT] This is what it is looking like on the 7D:

mv1080 mode 1736x976 10-bit
(https://farm5.staticflickr.com/4581/37992331985_050de17024_z.jpg) (https://flic.kr/p/ZTfCED)

zoom mode 1920x1080 10-bit
(https://farm5.staticflickr.com/4524/24013982787_9ed2bc5b33_z.jpg) (https://flic.kr/p/CA2WrZ)

Changing the video mode or resolution seems to garble it up. Maybe this has to do with the buffer addresses changing on Digic IV cameras that use LVState?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2017, 09:59:41 PM
Posted allocate-raw-lv-buffer builds on my Bitbucket downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). It will probably be easier if we can make sure it is working on these before trying it with 10bit/12bit.

@masc - I reduced the RAW_LV_BUFFER_ALLOC_SIZE on the 5D2 so give it another try.

Here's what I'm doing if anyone wants to follow along.

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff

Click on the Commits tab to see the latest changes. If I remember to do it, I'll make the builds before committing the changes so you can open the autoexec.bin file in a text editor and read the latest news:

Magic Lantern Nightly.2017Dec06.5D2212
Camera   : 5D2
Firmware : 212
Changeset: c039449005d2+ (allocate-raw-lv-buffer-wip) tip
Built on : 2017-12-06 20:57:03 by rosiefort@RosieFoComputer

diff -r c039449005d2 src/raw.c
--- a/src/raw.c Wed Dec 06 12:45:04 2017 -0800
+++ b/src/raw.c Wed Dec 06 12:57:03 2017 -0800
@@ -120,7 +120,7 @@
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif

-#ifdef CONFIG_5D2
+#ifdef CONFIG_5D2 // keep working on this until it works!
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x2000)
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 06, 2017, 10:03:32 PM
Quote from: dfort on December 06, 2017, 08:06:06 PM
In fact if you switch over to another mode then back to mv1080, it stops working.

Does it help if you start the camera in that other mode, rather than switching to it?

If yes (very likely IMO), the SRM buffer may have to be freed and reallocated again when the video mode changes (because Canon's memory layout changes). Disabling raw video before switching to the new video mode, and re-enabling it afterwards, should do the trick. On crop_rec_4k, mlv_lite does something about this, but the raw backend doesn't.

For 5D2 - I also had trouble with it and could not figure out why (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177584#msg177584).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: masc on December 06, 2017, 10:15:39 PM
Tried out again with latest allocate version. I did not change the mode (3x / 10x zoom). Switched on, LV enabled, record, stop. While recording, picture on screen is not refreshed.
(https://image.ibb.co/n5HnfG/M06_2209_frame_12.png)
5D2, 1856x1044, 25fps.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 06, 2017, 10:52:50 PM
This is soo exciting!!!! Hoping 5D2 isn't impossible to do, though it might be as a1ex had difficulties.

@dfort Why change the buffer alloc size to be smaller, how could that random of a change help?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2017, 10:58:54 PM
I think we're close on the 5D2. Let's see what happens when we shrink the RAW_LV_BUFFER_ALLOC_SIZE even more. The 5D2 has a huge SRM_BUFFER_SIZE and maybe the overhead is a percentage of the size instead of a fixed amount that can be used on all cameras. Just a guess. I kept changing it until it worked on the 7D.

New build posted (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). This time you should see this in autoexec.bin:

Magic Lantern Nightly.2017Dec06.5D2212
Camera   : 5D2
Firmware : 212
Changeset: c039449005d2+ (allocate-raw-lv-buffer-wip) tip
Built on : 2017-12-06 21:37:06 by rosiefort@RosieFoComputer

diff -r c039449005d2 src/raw.c
--- a/src/raw.c Wed Dec 06 12:45:04 2017 -0800
+++ b/src/raw.c Wed Dec 06 13:37:06 2017 -0800
@@ -121,7 +121,7 @@
#endif

#ifdef CONFIG_5D2
-#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x2000)
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x4000)
#endif

#ifdef CONFIG_50D


Once we get it working on allocate-raw-lv-buffer we'll try it on 10bit/12bit.

Quote from: a1ex on December 06, 2017, 10:03:32 PM
Does it help if you start the camera in that other mode, rather than switching to it?

Didn't work. The only thing that works consistently is deleting the SETTINGS directory. Sometimes switching to another mode works but most of the time it doesn't.

Quote from: a1ex on December 06, 2017, 10:03:32 PM
On crop_rec_4k, mlv_lite does something about this, but the raw backend doesn't.

Guess I could try using the mlv_lite module from the crop_rec_4k branch. After all, that's where we want to end up with this!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 06, 2017, 11:07:13 PM
Tried out your new build, artifacts very similar to what masc had, only first frame intact.

Wow... Lots of memories came back with that old raw_rec module ::)

QuoteJust a guess. I kept changing it until it worked on the 7D.
Really interesting. I might try a few different sizes myself then. Did you have to shrink it a lot on 7D for it to work? What's the minimum size it can go to? The amount needed to fit a whole (full-size)frame?

QuoteDoes it help if you start the camera in that other mode, rather than switching to it?
I did it all in the same mode, but could the 5D2 possibly be switching modes by itself at the start? I remember it had some weird double startup procedure or something...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2017, 11:23:51 PM
Quote from: Ilia3101 on December 06, 2017, 11:07:13 PM
Really interesting. I might try a few different sizes myself then. Did you have to shrink it a lot on 7D for it to work? What's the minimum size it can go to? The amount needed to fit a whole (full-size)frame?

Start reading from here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg194161#msg194161) and you'll see how I got to the current code. Once you get allocate-raw-lv-buffer (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff) working apply it to this version of raw_video_10bit_12bit (http://raw_video_10bit_12bit).

Great to have someone who can compile checking this stuff out. Please let us know if you find something that works.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2017, 11:31:19 PM
@masc - Let's try one more time before calling it quits for the day. New allocate-raw-lv-buffer for the 5D2 uploaded.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 07, 2017, 12:07:53 AM
Quote from: dfort on December 06, 2017, 11:23:51 PM
Start reading from here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg194161#msg194161) and you'll see how I got to the current code. Once you get allocate-raw-lv-buffer (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff) working apply it to this version of raw_video_10bit_12bit (http://raw_video_10bit_12bit).

Thanks a lot. I'll read through that, and see what I can do, starting with reducing buffer size.

Also tried the new build, didn't do it :(
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 07, 2017, 12:56:13 AM
Ok--I played around some more with the 10bit/12bit stuff and merged in the latest changes. It seems a bit more stable, at least in mv1080 mode. Updated all of those builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 07, 2017, 08:58:15 AM
Just a thought. If you can compile ML and have one of the Digic IV cameras that we're trying to get working, try different values for this:

src/edmac-memcpy.c
#elif defined(LV_STATE)
uint32_t raw_write_chan = 0x10; /* DIGIC IV LVState cameras */


I tried everything from 0x0 through 0x20 (this is hex so the count goes from 0-F) so that's the first 32 possibilites. You might notice that there is some code to find free edmac channels but experimenting around I found that sometimes what isn't listed might work better. You'll know when it doesn't work. I only tried it in mv1080 mode so maybe you need to switch to a different channel depending on the video mode?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: OlRivrRat on December 07, 2017, 05:51:40 PM
                     @DFort

           When You Say "Updated all of those builds" just Which "all of those builds" are You referring to & Where can the Updated Builds be found?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 07, 2017, 05:56:25 PM
Check the downloads page:
http://www.magiclantern.fm/forum/index.php?topic=19336.msg194178#msg194178
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 07, 2017, 05:59:39 PM
Dfort, you seem to have reached a very important milestone - normal 1728x972 MLV video is now working in both: the MLV and RAW-video modes at 10, 12 and 14 bits.  CONGRATULATIONS !!!

After playing with your latest built on my 7D for several hours last night, I would like to provide some feedback.

1) In the normal uncropped mode I get continuous recording with synchronous sound at the maximum 1728x972 resolution at 10, 12 and 14 bits.  Color preview works just fine, so does focusing and framing.  Framing is maintained also during recording which is extremely useful in practical video work.  In-camera playback works only for the 14-bit MLV files.  With 10 and 12 bits a black screen is observed during playback but the counter works and shows the actual number of recorded frames.  Maybe,a working RAW_TWK module could solve this playback issue at 10 and 12 bits.

2) A problem occurs at 5x magnification mode.  When the magnification button is pressed  the screen freezes and no further adjustmens or focusing are possible.  However, when in this condition, the recording button is pressed, actual recording takes place and in the MLV-video mode recording with synchronous sound is possible even at the highest resolution available (2496x1198).  The MLV file looks gorgeous, there are no corrupt frames and sound works perfectly!  There are no "hick-ups" either, those frame jumps that are so annoying in the 100D.  Unfortunately, the frozen screen makes framing and recomposing during recording impossible.  So, this wonderful 5x magnification mode is usable only for static camera shots on a tripod.  Here are some record times on a 16GB Transcend x1000 CF card:

     * 14-bit (2496x1198) resolution with sound at 24 fps - about 70-75 frames
     * 12-bit (2496x1198) resolution with sound at 24 fps - about 140-160 frames
     * 10-bit (2496x1198) resolution with sound at 24 fps - about 420-510 frames (fantastic !!!)

Interestingly, the number of frames differs at each shot, although a constant bitrate is used.  Typically, the first shot after camera is turned on is quite short (say 240 frames at 10 bits in the above resolution), then the second shot is quite long (say 500 frames)and the next shots vary between 420 and 500 shots, if one succeeds in getting out of the frozen-screen state without turning camera off, (see #4 below).

3) A second, more serious problem occurs when you want to get out of the 5x magnification mode to shoot the next clip.  The only way to do this is by turning the camera off, taking the battery out, putting it back in and restarting the camera twice.  Only then you are back in the normal uncropped mode where you can compose, focus, meter and prepare for the next shot.  One workaround is to first turn MLV-video off, use LifeView to compose your shot, meter and focus in the 5x magnification mode, then turn MLV-video on again and take your footage.  This you can do only once.  Then you again have to pull the battery out and restart twice to get out of the frozen-screen state. 

4) Sometimes, when you playback the last recorded clip, (even just with black screen), the camera will return to its normal state, so you don't have to pull the battery out but sometimes it works and sometimes it doesn't.  I was unable to reproduce the issue.  I also played with the different preview modes but this does not resolve the frozen screen problem either as this was the case with older builds when you could get a normal screen after switching between those preview modes.     

I hope this is helpful to those who haven't been able to test yet.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 07, 2017, 08:21:03 PM
Quote from: OlRivrRat on December 07, 2017, 05:51:40 PM
           When You Say "Updated all of those builds" just Which "all of those builds" are You referring to & Where can the Updated Builds be found?

I post test builds on my Bitbucket downloads page:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/

Though I tend to update or clear them out without much fanfare.

@IDA_ML

Wow, thanks for the great feedback. Yeah, I've been doing a lot of battery pulling with this experiment.

a1ex suggested looking at mlv_lite from the crop_rec_4k branch because it handles special cases for 10bit/12bit. Apparently there are certain frame sizes that don't work with reduced bit rates. In any case, it might be easier to move the changes from the allocate-raw-lv-buffer branch to the crop_rec_4k branch rather than cherry pick code from mlv_lite.

I believe that I've got the addresses for lossless (https://www.magiclantern.fm/forum/index.php?topic=18443.msg193534#msg193534) on the 7D but don't have it working just yet. Of course that's another topic.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: masc on December 07, 2017, 08:54:30 PM
Hej, I played a bit around again with your latest version for 5D2. I have some news:
- 640x360, 25fps, 10bit -> OK!!!
- 960x540, 25fps, 10bit -> OK!!!
- 1280x720, 25fps, 10bit -> not OK
- 1600x900, 25fps, 10bit -> not OK
- 640x360, 25fps, 12bit -> OK!!!
- 960x540, 25fps, 12bit -> not OK
fps override:
- 1856x1044, 2.5fps, 10bit -> OK!!!
- 1856x1044, 20fps, 10bit -> OK!!!
- 1856x1044, 22fps, 10bit -> not OK
- the very first frame is always OK
- if the screen freezes while recording, the file was not OK. If the file was OK, I had a preview while recording.
Maybe that helps...  ;)
Unfortunately I don't have installed the toolchain needed for compiling...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 07, 2017, 11:02:33 PM
@dfort WOW!!!! 10 bit works on the 5D mark II !!!... pretty much. I just shot a 10 bit clip at 1880x1056 in MLV Lite. MLV App confirms: Bits / Pixel: 10

I found the main factor for making it work is having fps override enabled, even without changing the framerate. I'm sure knowing that could be useful.
And I agree with masc's findings, display frozen=fail, not frozen=successful clip.

Also have you uploaded the source of the 10-12bit builds that you posted? I'm not very good at navigating bitbucket so don't know where to find it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 08, 2017, 12:25:36 AM
Quote from: Ilia3101 on December 07, 2017, 11:02:33 PM
Also have you uploaded the source of the 10-12bit builds that you posted? I'm not very good at navigating bitbucket so don't know where to find it.

Sure, the dummy pull request is here so you can see my changes:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/16/10bit-12bit-for-lvstate-cameras/diff

If you just want to get my repository set to that branch right away simply:

hg clone -u raw_video_10bit_12bit_LVState-wip https://bitbucket.org/daniel_fort/magic-lantern

Assuming that you have Mercurial installed.

[EDIT] If you already have a magic-lantern directory and don't want to clobber it try this:

hg clone -u raw_video_10bit_12bit_LVState-wip https://bitbucket.org/daniel_fort/magic-lantern magic-lantern-dfort
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 08, 2017, 05:14:08 PM
@dfort - Had a very quick look at your 50D 10/12bit build. It's promising!

10bit 1584x856 ok / 2000*1076 (crop) ok - continuous recording on a relatively slow 800x CF card  :)
12bit 1584x856 ok / 2000*1076 (crop) fails with some frame tearing and intermittent pink frames
14bit 1584x856 ok / 2000*1076 (crop) fails with some frame tearing and intermittent pink frames

all 25p


The allocate-raw-lv-buffer build produces corrupt images in crop and non-crop modes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 08, 2017, 09:17:05 PM
Quote from: dfort on December 08, 2017, 12:25:36 AM
If you just want to get my repository set to that branch right away simply:

hg clone -u raw_video_10bit_12bit_LVState-wip https://bitbucket.org/daniel_fort/magic-lantern

Thanks a lot for that, it worked wondefully. 14 bit recording has issues and I was wrong about just having fps override enabled fixing things, the framerate also has to be changed a bit.

Is it still worth toying about with making RAW_LV_BUFFER_ALLOC_SIZE smaller? Could it help get rid of some of the issues like some settings not working or pink frames?

10 bit 5D2 1880x1248 frame:

(http://thumb.ibb.co/dPSM9b/5d10bit.png) (http://ibb.co/dPSM9b)


Quote from: dfort on December 07, 2017, 08:58:15 AM
Just a thought. If you can compile ML and have one of the Digic IV cameras that we're trying to get working, try different values for this:

src/edmac-memcpy.c
#elif defined(LV_STATE)
uint32_t raw_write_chan = 0x10; /* DIGIC IV LVState cameras */


I tried everything from 0x0 through 0x20 (this is hex so the count goes from 0-F) so that's the first 32 possibilites. You might notice that there is some code to find free edmac channels but experimenting around I found that sometimes what isn't listed might work better. You'll know when it doesn't work. I only tried it in mv1080 mode so maybe you need to switch to a different channel depending on the video mode?
Will try this also, hopefully with some good results. No risk of bricking or anything?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 08, 2017, 09:19:12 PM
Quote from: dfort on December 07, 2017, 08:58:15 AM
Just a thought. If you can compile ML and have one of the Digic IV cameras that we're trying to get working, try different values for this:

src/edmac-memcpy.c
#elif defined(LV_STATE)
uint32_t raw_write_chan = 0x10; /* DIGIC IV LVState cameras */


I tried everything from 0x0 through 0x20 (this is hex so the count goes from 0-F) so that's the first 32 possibilites. You might notice that there is some code to find free edmac channels but experimenting around I found that sometimes what isn't listed might work better. You'll know when it doesn't work. I only tried it in mv1080 mode so maybe you need to switch to a different channel depending on the video mode?

Haven't used the 50D for ages and had to get back up to speed with compiling ML but there now :)

Are you saying to try all channels? I can't remember how to check for free edmac channels (hint anyone?) but surely it's better to discount edmac channels that are in use rather than compiling ml 32x or more?

Trying to get my head around the architecture - are corrupt frames like I'm getting with your 10/12bit build symptomatic of using a shared edmac channel?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 08, 2017, 10:27:26 PM
Good to see Andy600 and Ilia3101 jumping in.

@a1ex found the SRM_BUFFER_SIZE for the 500D (http://www.magiclantern.fm/forum/index.php?topic=11864.msg194327#msg194327) so it looks like all of the Digic IV that use LVState might be able to work with 10bit/12bit. Well maybe just hobble along at a few resolution settings for now.

500D build is up on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).

Now that there is some momentum on this how about some of you that have these old cameras and can compile ML start experimenting with it and see if you can get it working good enough to get it accepted into the Experiments builds (https://builds.magiclantern.fm/experiments.html)?

Here's a dummy pull request so you can see what's been done so far:
https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/16/10bit-12bit-for-lvstate-cameras/diff

If you want to grab my branch and play around with it, here's how you can do it without overwriting your local repository:

hg clone -u raw_video_10bit_12bit_LVState-wip https://bitbucket.org/daniel_fort/magic-lantern magic-lantern-dfort

@Andy600 - It makes sense that you should try only the free edmac channels. I remember running something a while back that would show which channels were free but then I found out that there were other channels that weren't reported that worked better so I just kept trying. I got to that 0x10 value fairly early in the process so and there are probably other channels to try out so that's a good place to experiment.

Here is the edmac screen on a 7D. It looks like 19-22 (0x13 - 0x16) might be open write channels but I'm not really sure about that. Note that what I'm using 0x10 is 16 decimal which is the channel that has that big write buffer. (Take this with a grain of salt.)

(https://farm5.staticflickr.com/4556/38106830844_5c0fb2db6b.jpg) (https://flic.kr/p/214ntbU)

@Ilia3101 - I'm not sure if reducing the RAW_LV_BUFFER_ALLOC_SIZE any further helps. I found out on the 7D that using the full SRM_BUFFER_SIZE wouldn't work at all. a1ex suggested reducing it by 0x100 but I had to go all the way to 0x1000 for it to work. Again, that could probably use some more experimenting.

By the way, I keep trying to get this Digic IV 10bit/12bit stuff working in the crop_rec_4k branch but keep failing miserably. It would be great to get some help with that.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 01:49:29 AM
The 50D seems stable using EDMAC 01 and 02 :D - and for some reason I decided to work backwards from 1F ::)

Tested at 25p and maximum mlv-lite 50D frame resolutions for non-crop (1584*1056) and crop (2000*1076). I've had no pink or torn frame issues and filled a 32gb card with 10, 12 and 14bit raw video.

I've uploaded the EDMAC 01 build here if any 50D users want to try it (entirely at your own risk!! Usual rules apply - if it breaks your camera I can't help!).
Removed to avoid any confusion. Please download from @dfort's repo (https://bitbucket.org/daniel_fort/magic-lantern/downloads/)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2017, 02:32:31 AM
Ha ha -- like the saying goes, everything you find is in the last place you look for it.

Following your lead I updated edmac-memcpy.c. Getting ready for the possibility of each camera using a different channel.

Magic Lantern Nightly.2017Dec08.50D109
Camera   : 50D
Firmware : 109
Changeset: 269205a6d2d5+ (raw_video_10bit_12bit_LVState-wip)
Built on : 2017-12-09 01:19:40 by rosiefort@RosieFoComputer

diff -r 269205a6d2d5 src/edmac-memcpy.c
--- a/src/edmac-memcpy.c Fri Dec 08 12:18:33 2017 -0800
+++ b/src/edmac-memcpy.c Fri Dec 08 17:19:40 2017 -0800
@@ -341,8 +341,16 @@
uint32_t raw_write_chan = 0x4;  /* 0x12 gives corrupted frames on 1.2.3, http://www.magiclantern.fm/forum/index.php?topic=10443 */
#elif defined(EVF_STATE)
uint32_t raw_write_chan = 0x12; /* 60D and newer, including all DIGIC V */
-#elif defined(LV_STATE)
-uint32_t raw_write_chan = 0x10; /* DIGIC IV LVState cameras */
+#elif defined(CONFIG_7D)
+uint32_t raw_write_chan = 0x10;
+#elif defined(CONFIG_5D2)
+uint32_t raw_write_chan = 0x10;
+#elif defined(CONFIG_50D)
+uint32_t raw_write_chan = 0x01;
+#elif defined(CONFIG_500D)
+uint32_t raw_write_chan = 0x10;
+#elif defined(CONFIG_550D)
+uint32_t raw_write_chan = 0x10;
#endif

static void edmac_slurp_complete_cbr (void* ctx)


Also posted a new 50D build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 04:27:37 AM
I wouldn't be surprised if 01 and 02 also work for the 5D2  ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2017, 07:10:16 AM
0x01 and 0x02 are also working on the 7D but what I'm noticing is that when turning on the camera the LV skips until you start recording then it smooths out. 0x10 doesn't do that.

For those of you who are wondering what we're talking about. Andy600 and I are trying to find the best raw_write_chan to use.

/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#ifdef CONFIG_EDMAC_RAW_SLURP

#if defined(CONFIG_5D3)
uint32_t raw_write_chan = 0x4;  /* 0x12 gives corrupted frames on 1.2.3, http://www.magiclantern.fm/forum/index.php?topic=10443 */
#elif defined(EVF_STATE)
uint32_t raw_write_chan = 0x12; /* 60D and newer, including all DIGIC V */
#elif defined(CONFIG_7D)
uint32_t raw_write_chan = 0x10; /* tested 0x01 through 0x60 (0x01 and 0x02 also work) */
#elif defined(CONFIG_5D2)
uint32_t raw_write_chan = 0x10;
#elif defined(CONFIG_50D)
uint32_t raw_write_chan = 0x01; /* 0x01 and 0x02 reported to work by Andy600 */
#elif defined(CONFIG_500D)
uint32_t raw_write_chan = 0x10;
#elif defined(CONFIG_550D)
uint32_t raw_write_chan = 0x10;
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 04:08:54 PM
No skipping or frame issues on the 50D. Tested a lot more today. Recorded 300Gb+ at different resolutions and not had any issues 8).

The LV skipping might be something to do with movie mode!? The 50D doesn't have a dedicated movie mode (it was added by a1ex) so maybe those EDMAC channels are not shared/used on the 50D but are on the 7D, 5D2 etc. Just speculating.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2017, 04:16:33 PM
@Andy600 -- Seems logical. Does 0x10 also work on the 50D? Do you find 0x1 more stable? Here's my experience on the 7D, maybe you can check these settings on the 50D.

Some things are consistant on the 7D, like recording up to 1736x976 (16x9) mv1080 at 10bit but as soon as I increase the vertical resolution to the next step 1736x1040 (5:3) it breaks up.

This morning I tried zoom mode and although the LV refresh rate was rather low I got good video at every resolution all the way up to 2520x1192 at 10bit. Don't get too excited, next time I tried it all I got was a garbled LV and corrupted frames. About the only thing consistant in zoom mode is that once you press the magnifying lens button you can't get out of that mode without restarting the camera. Still, there is a glimmer of hope there. I saved the test--I'll keep it on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) for a few days. Don't judge the artistic content, I shot it without even getting out of bed.

(https://farm5.staticflickr.com/4633/38900307992_282e24a46a.jpg) (https://flic.kr/p/22gufio)

7D 2520x1192 zoom mode at 10bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 05:17:19 PM
Quote from: dfort on December 09, 2017, 04:16:33 PM
@Andy600 -- Seems logical. Does 0x10 also work on the 50D? Do you find 0x1 more stable?

0x10 works for some resolutions but not others http://www.magiclantern.fm/forum/index.php?topic=5601.msg194350#msg194350 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194350#msg194350)

I get, dare I say, no problems using channels 0x01 and 0x02 at all resolutions in 10, 12 and 14bit. Tried most channels up to 1F. Most don't work and the ones that did (0x0A, 0x0B, 0x03) had issues (corrupt LV, EDMAC timeout etc).

For the 50D 0x01 and 0x02 are the ones to use. I'm interested to see if those channels work equally well on the 5D2. I don't have one to test but I suggest uploading a couple of 5D2 test builds for someone that does (or I can build/upload).

I would like to try pushing the resolution but don't know where to start. The 50D is maxed at 2000*1076 currently (crop).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: masc on December 09, 2017, 05:47:40 PM
Quote from: Andy600 on December 09, 2017, 05:17:19 PM
I'm interested to see if those channels work equally well on the 5D2. I don't have one to test but I suggest uploading a couple of 5D2 test builds for someone that does (or I can build/upload).
If you could do that, I could test... not this weekend, but next week I'll find some time for testing... ;) That is so exciting!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2017, 06:42:42 PM
Put up the 5D2 test builds recommended by @Andy600 - raw_video_10bit_12bit_raw_write_chan=0x01 and 0x02. On my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).

Of course if more users would run through the simple compiling tutorials for Mac (http://www.magiclantern.fm/forum/index.php?topic=16012.0) and Windows (http://www.magiclantern.fm/forum/index.php?topic=15894.0) and run tests like @Andy600, development would move at a faster pace.

Quote from: Andy600 on December 09, 2017, 05:17:19 PM
...no problems using channels 0x00 and 0x01 at all resolutions in 10, 12 and 14bit...

You mean 0x1 and 0x02, right?

Quote from: Andy600 on December 09, 2017, 05:17:19 PM
I would like to try pushing the resolution but don't know where to start. The 50D is maxed at 2000*1076 currently (crop).

That sounds about right. The 50D has a 15.1 megapixels sensor while the 7D has a 17.9 megapixels sensor. Increasing resolution beyond zoom mode is handled by the crop_rec module.

Speaking of crop and zoom, it can get rather confusing. When I was working on the focus pixel issue with EOSM/100D/650D/700D I added this to one of my scripts. The buffer sizes are different on these Digic IV models but the description of the various video modes still apply. We need to concentrate on mv1080, mv720 and zoom modes for the LVState cameras (7D/50D/5D2/500D/550D):

# Mode          Buffer Size   Notes
#
# mv1080      - 1808x1190  -  Uses full sensor area with a 3x3
#                             line skipping pattern.
#                             Not available on the EOSM.
#                             Canon menu setting 1920x1080 24/25/30.
#
# mv720       - 1808x727   -  Uses full sensor area with a 3x5
#                             line skipping pattern.
#                             Needs to be stretched by 1.67x
#                             vertically - up to 50/60 fps.
#                             Limited to 24/30 on EOSM.
#                             Canon menu setting 1280x720 50/60.
#
# mv640       - 1808x1190  -  Basically the same as mv1080 mode when
#                             shooting MLV/RAW.
#                             Canon menu setting 640x480 25/30.
#
# mv1080crop  - 1872x1060  -  High resolution and largest
#                             sustainable frame sizes on all cameras.
#
# mv640crop   -            -  Not available on these cameras.
#
# zoom        - 2592x1108  -  Same 5x magnification as mv1080crop but
#                             can pan around sensor and achieve
#                             higher resolution than mv1080crop.
#
# crop_rec    - 1808x727   -  Requires crop_rec module.
#                             Uses mv720 raw buffer but records in
#                             3x3 line skipping like mv1080.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 06:48:52 PM
Quote from: dfort

You mean 0x1 and 0x02, right?


Doh  ::)

Yes, I mean 0x01 and 0x02
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 07:02:25 PM
Quote from: dfort on December 09, 2017, 06:42:42 PM
Speaking of crop and zoom, it can get rather confusing. When I was working on the focus pixel issue with EOSM/100D/650D/700D I added this to one of my scripts. The buffer sizes are different on these Digic IV models but the description of the various video modes still apply. We need to concentrate on mv1080, mv720 and zoom modes for the LVState cameras (7D/50D/5D2/500D/550D):

# Mode          Buffer Size   Notes
#
# mv1080      - 1808x1190  -  Uses full sensor area with a 3x3
#                             line skipping pattern.
#                             Not available on the EOSM.
#                             Canon menu setting 1920x1080 24/25/30.
#
# mv720       - 1808x727   -  Uses full sensor area with a 3x5
#                             line skipping pattern.
#                             Needs to be stretched by 1.67x
#                             vertically - up to 50/60 fps.
#                             Limited to 24/30 on EOSM.
#                             Canon menu setting 1280x720 50/60.
#
# mv640       - 1808x1190  -  Basically the same as mv1080 mode when
#                             shooting MLV/RAW.
#                             Canon menu setting 640x480 25/30.
#
# mv1080crop  - 1872x1060  -  High resolution and largest
#                             sustainable frame sizes on all cameras.
#
# mv640crop   -            -  Not available on these cameras.
#
# zoom        - 2592x1108  -  Same 5x magnification as mv1080crop but
#                             can pan around sensor and achieve
#                             higher resolution than mv1080crop.
#
# crop_rec    - 1808x727   -  Requires crop_rec module.
#                             Uses mv720 raw buffer but records in
#                             3x3 line skipping like mv1080.


currently no 720 mode on the 50D.

I tested 1080 and 640 modes with those EDMAC channels and all work although I have just been through all the dngs and did have a couple of random corrupt 10 and 12bit frames when set to 640 mode so it's not perfect. 1080 mode DNGs have no issues :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2017, 07:23:15 PM
Seems like the 50D is the most stable with this 10bit/12bit experiment. Maybe since it wasn't designed for video the channels and buffers and what not are fixed in one place instead of moving around like what seems to be happening on the other cameras.

Is that mv640 mode of any use or is it pretty much the same as mv1080?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 07:39:57 PM
It affects H.264 recording sizes but for raw recording, aside from the couple of bad frames I mentioned, it seems to be the same as mv1080 mode.

mv1080 at all bit depths and frame sizes seems to be rock solid 8). If it can be made to work with the crop_rec module and compression it should be quite good :) (actually, it's already good)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2017, 08:13:26 PM
I'd really like to get this merged into the raw_video_10bit_12bit branch so it could be included on the Experiments downloads page (https://builds.magiclantern.fm/experiments.html) but so far it seems that the only camera that is ready for this is the 50D. Maybe I can make a pull request for just the 50D?

@a1ex, any thoughts? Is this experiment heading in the right direction?

Quote from: Andy600 on December 09, 2017, 07:39:57 PM
If it can be made to work with the crop_rec module and compression it should be quite good :)

I agree. It should be possible. Just requires some time and perseverance. Of course if more ML users would get involved it would help our chances. Sort of like the infinite monkey theorem (https://en.wikipedia.org/wiki/Infinite_monkey_theorem), the chances may be extremely low (but technically not zero).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 09, 2017, 08:35:13 PM
From what I remember, the problem with 5D2 was similar to what it was reported (lack of good sync in LiveView), so I believe edmac_raw_slurp must be called from another place (somehow in sync with the LiveView capture process - probably the ReadOutDone event). I've started to look into that here (http://www.magiclantern.fm/forum/index.php?topic=18315.0), but didn't get too far. Still, the edmac module is compatible with DIGIC 4, so you are encouraged to try it and log the LiveView activity in various modes.

The last graph from that page is in the "edmac" branch, but not very interesting for the current issue; everything else is in mainline (edmac module). Didn't upload the drawing script yet, but you can upload some logs and I'll draw the graphs for you. If it won't require too much fiddling, I'll upload the script as well.

The 50D doesn't seem to be 100% clean either (the two video modes are likely identical in standby, so I assume the "rock solid" report was just pure luck (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175902#msg175902)). Just FYI - its firmware and video mode implementation is very close to 5D2 1.0.x (the older firmware without manual video controls).

My older attempt for 5D2 was pretty much the same as in the current PR, just less polished. Back then, I've used dmaFlags = 0x20000000, raw_write_chan = 5, PREFERRED_RAW_TYPE 5 and redirected_raw_buffer = malloc(2050*14/8*2000).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 08:42:47 PM
mv640 had a couple of corrupt frames but I've recorded a lot with video mode set to mv1080 without issues at 10,12 and 14bit both crop and non-crop. No bad frames or LV glitches. Yes, might be luck but seems solid for me. What can I do to push it to break?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 09, 2017, 09:04:39 PM
1584х on 50D in no crop ? i see only 1568 on my camera  :-\
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 09, 2017, 09:07:39 PM
Quote from: CITY-U1001 on December 09, 2017, 09:04:39 PM
1584х on 50D in no crop ? i see only 1568 on my camera  :-\

1584 is because I'm testing 10/12bit with the MLV-Lite module.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 09, 2017, 09:18:36 PM
on 50D work good in 23.976fps  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 09, 2017, 09:27:14 PM
Quote from: Andy600 on December 09, 2017, 08:42:47 PM
What can I do to push it to break?

Different frame rates and different overlay configurations in ML (zebras, magic zoom etc). These are likely to affect the LiveView sync.

Some models and/or video modes might be luckier than others, or they might have a higher tolerance for timing errors. The 50D might be lucky because of the lower resolution.

I might be able to give a better answer after some hours of analyzing LiveView logs (to cross-check what happens on 60D and newer vs 550D and older models), but not right now. Want to check the timing relationships between the EDMAC transfers from connection #0, and the ReadOutDone event or HEAD interrupts, or whatever else might signal a completed image capture. The tools for doing that are edmac.mo and dm-spy-experiments. Some real-time plotting code that can be reused is in mlv_lite on crop_rec_4k (EDMAC plots).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 10, 2017, 12:00:10 AM
Edmac channel 0x02 works perfectly for recording 10/12 bit on 5D2 without pink frames and without needing fps override, however crop mode is broken.

Interestingly... channel 0x01 produced same split frame problems as 5D2 had right at the start of 10/12 bit development. So 0x01 doesn't work at all on 5D2.

I noticed a strange problem though... the liveview seemed to refresh at around 12fps and stutter a lot, but not when recording... what is happening?
However on channel 0x01 it always stuttered and refreshed slowly even when recording.

I was using code fetched with this:
hg clone -u raw_video_10bit_12bit_LVState-wip https://bitbucket.org/daniel_fort/magic-lantern magic-lantern-dfort

Just recorded a "rock solid" 3-chunk, 3-minute, no corrupt frame 10 bit MLV :D

Quote from: a1ex on December 09, 2017, 09:27:14 PM
Different frame rates and different overlay configurations in ML (zebras, magic zoom etc). These are likely to affect the LiveView sync.
I tried doing this to make it break... framerate at some weird 24.88 fps, overlays, zebras, histograms, waveform everything that can be: enabled, still got a flawless 10 bit recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2017, 02:29:14 AM
@Ilia3101 - So like the opthamologist says, which looks better A or B? Seems like 0x2 works on the 5D2 but the LiveView stutters until you start recording, just like on the 7D. I'm finding 0x10 doesn't exhibit this yet it records mv1080 at 1736x976 at all bit depths just fine. However, increasing the vertical resolution doesn't work. Sometimes it works in zoom mode but the LiveView either stutters or freezes--usually it doesn't work at all.

@Andy600 - Do you feel lucky punk? Just kidding. Same question as above. Is 0x1 or 0x2 really better than 0x10 on the 50D or do you just want to be different from everybody else?

Speaking of different, I'm not getting any responses from the 1100D camp, just one download. I tried getting the SRM_BUFFER_SIZE in QEMU but can't get to the ML menu. I'll upload a find-SRM_BUFFER_SIZE build but that camera is way behind, most modules don't even compile. Ironic that it was the work that was done on the 1100D that got this 10bit/12bit on Digic IV thing started.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2017, 03:41:02 AM
Great news for the 1100D. Found a user who is willing to do some testing.

@nuvon -- I uploaded a find-SRM_BUFFER_SIZE build for the 1100D on my Bitbucket downloads page. It is using the latest code base which means that many of the modules won't compile. Don't worry about it, we just want one piece of information so follow these steps:

In the ML Debug menu go to Free Memory option:

(https://farm5.staticflickr.com/4595/38911743852_f286e0da5c.jpg) (https://flic.kr/p/22huRMo)

This should display the SRM_BUFFER_SIZE. Write it down and report back.

(https://farm5.staticflickr.com/4539/25076572648_2e0fc3f318.jpg) (https://flic.kr/p/EcVZoQ)

(These screenshots were done on an EOSM so it might look a little different on the 1100D.)

There is also a build for the 1100D (raw_video_10bit_12bit.2017Dec06.1100D105.zip) on the same downloads page with all the goodies for the 1100D including mlv_lite and it should work with 10bit/12bit raw video. It would be great if you could try that build and report back.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 10, 2017, 08:40:50 AM
I can confirm with @Ilia3101 on 5D2 in 1:1 non crop mode with mlv_lite & mlv full+audio ,
I can record 10 , 12 , 14 bit with out any frame corruption or out of sync Lv preview.
I downloaded the source code , changed write channel from 0x01 to 0x02 in edmac-memcpy.c  , compiled it locally in my VM.
Tested 1856x928 , 1872x936 , 1880x1056 , 1880x1016 , 1880x1128 , 1880x1248 ,  23.976 with frame over ride to exact in Auto Preview with mlv_lite
Pressed the half shutter while recording to get actuate framing with out issue , in 10 , 12bit (14bit where short 200 frame captures)

In mlv full with audio I tested 1856x928,  1856x1044 , 1856x1114 , 1856x1238 , 1856x1248 ,  23.976 with frame over ride to exact in Auto Preview ,
also pressed the half shutter while recording to get actuate framing with out issue , in 10 , 12bit (14bit where short 200 frame captures)


3x Crop mode is broken , it seems to keep the first frame it see as a alternating frame while the other frame has movement so
like frame #1 is frozen and frame #2 has normal movement and keeps repeating that pattern but frame #1 is not broken -- as in
artifacts or shearing with pink frame just frozen . I tested mlv_lite 2152x1072 & Full mlv + audio 2144x1074

*Note* Full MLV +Audio : audio was sync and was perfect , not problems.

My Camera setting and build information
magic.cfg
# Magic Lantern Nightly.2017Dec10.5D2212 (ee95e8825183+ (raw_video_10bit_12bit_LVState-wip) tip)
# Built on 2017-12-10 03:33:12 UTC by ml@ml-pc
# Configuration saved on 2017/12/10 00:17:36
beta.warn = 10
menu.first = -4
movie.log = 1
rec.notify = 0
enable-liveview = 2
fps.preset = 1
fps.override.idx = 32
fps.override = 1
battery.drain.rate.rev = 69
hist.log = 0
spotmeter.draw = 0
zebra.draw = 0
disp.mode.x = 149



mlv_lite.cfg
# Config file for module mlv_lite (MLV_LITE.MO)

raw.video.enabled = 1
raw.res_x_fine = -40
raw.write.speed = 6399
raw.warm.up = 2
raw.bppi = 0


mlv_rec.cfg
# Config file for module mlv_rec (MLV_REC.MO)

mlv.video.enabled = 1
mlv.res.x.fine = 64
mlv.bpp = 0
mlv.write_speed = 6699
mlv.display_rec_info = 2
mlv.buffer_fill_method = 0


mlv_snd.cfg
# Config file for module mlv_snd (MLV_SND.MO) mlv.snd.enabled = 1

This code looks promising for the 3.5/UHD development with custom buffers , I'll see if I can implement this in to my source code.

Edit: if anyone that can't compile and what to try out  1:1 10 & 12bit raw video with mlv_lite & full mlv+audio I uploaded to my bitbucket downloads magiclantern-10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip)



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 10, 2017, 09:02:54 AM
Quote from: dfort on December 10, 2017, 02:29:14 AM
@Andy600 - Do you feel lucky punk? Just kidding. Same question as above. Is 0x1 or 0x2 really better than 0x10 on the 50D or do you just want to be different from everybody else?

To put it simply, 0x01 and 0x02 work with every setting and resolution when set to mv1080 but I had a couple of bad frames with a random LV glitch when set to mv640 mode[/url]. I have tested more as @a1ex suggested (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194423#msg194423) and, so far, I've had no additional issues. Call it a favorable race condition, luck or whatever but LV and recording on the 50D appear to be very stable when it's set to mv1080 with channel 0x01 or 0x02. I'm still testing.

There appears to be no difference between these channels but I can only test empirically and have nowhere near @a1ex's understanding of how these cameras work. He may well know of good reasons why those channels shouldn't be used.

0x10 has some issues (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194350#msg194350) with crop mode recording and glitchy, unstable LV.

I get that it would be simpler for all Digic IV cameras to use the same channels but I think the 50D is going to be a bit different by virtue of it's 'added' movie modes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 10, 2017, 09:31:28 AM
Quote from: a1ex on December 09, 2017, 08:35:13 PM
... (somehow in sync with the LiveView capture process - probably the ReadOutDone event)....

what happened to http://a1ex.bitbucket.org/ML/states/ ??
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 10, 2017, 10:11:27 AM
Quote from: nikfreak on December 10, 2017, 09:31:28 AM
what happened to http://a1ex.bitbucket.org/ML/states/ ??

Good catch - they renamed it to bitbucket.io. Found this guide (https://confluence.atlassian.com/bitbucket/publishing-a-website-on-bitbucket-cloud-221449776.html) and renaming the old repository was enough to do the trick.

=> http://a1ex.bitbucket.io/ML/states/index.html

Also updated old forum links.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 10, 2017, 10:46:03 AM
@a1ex - I just compiled the edmac module. Is there anything useful I can provide you from the 50D?

movie mode was off for this:

(https://images2.imgbox.com/23/d6/XaVmJrMb_o.jpg) (http://imgbox.com/XaVmJrMb)

and on for this:

(https://images2.imgbox.com/e9/57/QbdpuRf1_o.jpg) (http://imgbox.com/QbdpuRf1)

0x02 looks good and compared to the 7D (http://www.magiclantern.fm/forum/index.php?topic=19336.msg194105#msg194105) where the largest channel appears to be 0x10 it makes sense with my findings!? (unless I'm not properly understanding this :-\)

also, the camera freezes when I press 'log EDMAC usage'
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 10, 2017, 11:00:34 AM
This says channel 1 seems free (does it also come up as free if you scan for free channels?), but channel 2 is used (and a transfer was active in the moment you took the screenshot). Channel 0 is used to write the HD buffer (the input data to H.264, but it's also updated in standby) and channel 18 = 0x12 writes the LV buffer (the preview image - 720x480 on 3:2 screens, about 720 x 480*(4/3)/(3/2) = 720x426 on 4:3 screens, 2 bytes per pixel - UYVY aka YUV422 in Canon strings). This matches the definitions from consts.h - just giving some examples on how to interpret these numbers.

Channel 13 is reading from the HD buffer (maybe for resizing it or whatever it does), but its geometry is likely more complex, so the overview screen only prints its size. You may scroll to detailed view to get the full configuration. We don't need it for this experiment - I'm just showing how you can use it to explore things.

The camera freeze has to be debugged (not sure where to start). Does it work outside LiveView, if you use it to log a still image capture? Does it help if you increase LOG_INTERVAL?

Then, there are a bunch of logging tools in the dm-spy-experiments branch (such as logging EDMAC transfers, or state object transitions). Sample snippet from 500D, which I'm studying for QEMU emulation (found its debug messages a bit more verbose than usual).

dm-0007.log-15329-782D1> **INT-F9h*:000928bc:00:00: *** TryPostEvent(LiveViewMgr, 0x3, 0x0, 0x0), from ff0da628
dm-0007.log-15330-78327> **INT-F9h*:00000558:99:02: WriteEDmacFlickerCompleteCBR
dm-0007.log-15331-7834E> **INT-F9h*:0000057c:00:00: <<< INT-F9h done
dm-0007.log:15332:7838F> LiveViewMg:000920c0:00:00: *** LVState: (3) --3--> (4)          ff0e1288 (x=b18710 z=0 t=0)
dm-0007.log-15333-783E7> LiveViewMg:ff0e12b0:98:02: SetPsave 0xFFFFF0 RecRes:2 LvRes:2
dm-0007.log-15334-7844A> **INT-5Eh*:00092118:00:00: >>> INT-5Eh EDMAC#9 ff18f7d8(9)
dm-0007.log-15335-78488> **INT-5Eh*:00092198:00:00:     addr 734fd4, ptr 736474, size , flags 0
dm-0007.log-15336-784CB> **INT-5Eh*:0000057c:00:00: <<< INT-5Eh done
dm-0007.log-15337-784EF> **INT-5Dh*:00092118:00:00: >>> INT-5Dh EDMAC#8 ff18f7d8(8)


In these logs, messages tagged with *** are from our logging handlers (such as the LVState transition - state_transition_log), the >>> and <<< are showing interrupts, while plain messages (not tagged with anything) come from Canon code. The dm-spy-experiments code is pretty much a swiss army knife (http://www.magiclantern.fm/forum/index.php?topic=16534.msg192377#msg192377) and requires fiddling - you may place logging hooks basically anywhere in Canon code (whenever you think they may provide useful info). Start by using predefined logging functions, then feel free to invent new ones. Caveat: some of them may conflict, depending on their ROM address, so you can't just enable them all at the same time. Refer to the dm-spy-experiments topic (http://www.magiclantern.fm/forum/index.php?topic=2388) for details.

Example of messages I could only find on 500D (full log, without raw (http://a1ex.magiclantern.fm/bleeding-edge/qemu/500D-lv/dm-0010.log)):

dm-0010.log:4326:E6762>    Startup:ff170334:9b:01: TG-InitializeLvTgDriver
dm-0010.log:4327:E6816>    Startup:ff17039c:9b:01: dwInitComm[0]=0x12800001
dm-0010.log:4328:E6863>    Startup:ff1703c8:9b:01: wInit[0]=0x00000000
dm-0010.log:4329:E6896>    Startup:ff1703f4:9b:01: dwInit[0]=0x00074046
dm-0010.log:4330:E68E7>    Startup:ff170424:9b:01: pAdjustAnalgVoltage[0][0]=0x00140000
dm-0010.log:18305:222F2> LiveViewMg:ff170580:9b:01: TG-PowerOnLiveViewDevice
dm-0010.log:18306:2231A> LiveViewMg:ff17059c:9b:01: PowerOnE3LAT
dm-0010.log:18360:2377F> LiveViewMg:ff1705d4:9b:01: PowerOnE31LAT
dm-0010.log:18395:25F5B> LiveViewMg:ff170624:9b:01: CancelADTG_RESET
dm-0010.log:18396:25F85> LiveViewMg:ff170644:9b:01: EnableSerialPort
dm-0010.log:20113:4419C> LiveViewMg:ff170680:9b:01: TG-ReadyLiveViewDevice
dm-0010.log:20114:441BF> LiveViewMg:ff1706a4:9b:01: 2-1.AFE_InitSetting 74046 2804c
dm-0010.log:20169:44ABA> LiveViewMg:ff1706d4:9b:01: 1-1.TG_InitSetting 12800001 11a42c01
dm-0010.log:20459:48A39> LiveViewMg:ff170764:9b:01: PowerOnE32LAT
dm-0010.log:20460:48A63> LiveViewMg:ff17077c:9b:01: 2-2.BaseVoltage_Setting
dm-0010.log:20471:48C33> LiveViewMg:ff1707a0:9b:01: 3-1.CMOS_InitSetting
dm-0010.log:20494:48F9D> LiveViewMg:ff1707bc:9b:01: 1-2.2Acc1SettingForLV
dm-0010.log:21327:5B493> LiveViewMg:ff170d8c:9b:01: TG-StartupLiveViewDevice 4
dm-0010.log:21328:5B4B8> LiveViewMg:ff170d30:9b:01: 1-4/1-12.ReadOut_Setting
dm-0010.log:21336:5B615> LiveViewMg:ff170dbc:9b:01: 1-6.Condition_Setting
dm-0010.log:21341:5B6DD> LiveViewMg:ff1709f4:9b:01: 2-7.AFE Timing3[0] 92b0b bab0b
dm-0010.log:21349:5B82C> LiveViewMg:ff170ab4:9b:01: 2-3.AFE_GainSetting
dm-0010.log:21360:5B9E8> LiveViewMg:ff170dec:9b:01: 2-4.Cal_Start
dm-0010.log:21365:5BAA6> LiveViewMg:ff170e10:9b:01: 1-8.Release_PSAVE
dm-0010.log:21370:5BB6D> LiveViewMg:ff170e38:9b:01: 1-10.AnimComm_Setting [4]
dm-0010.log:21751:5FB72> LiveViewMg:ff170be0:9b:01: 3-4.CMOS SR_Setting [0=0x0]
dm-0010.log:22124:B1B3F> LiveViewMg:ff1713b0:9b:01: 1-13.SR_2ndFrame_Setting
dm-0010.log:22270:B9C14> LiveViewMg:ff171090:9b:01: TG-SetPsave[2-6]Off 4 553648112
dm-0010.log:22390:BCA3D> LiveViewMg:ff171108:9b:01: TG-ResetPsave[2-6]On 4 553648112
dm-0010.log:22391:BCA6B> LiveViewMg:ff171728:9b:01: 2-6.AFE Standby
dm-0010.log:22396:BCB2D> LiveViewMg:ff170be0:9b:01: 3-4.CMOS SR_Setting [0=0x0]
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 10, 2017, 11:26:50 AM
Edited my post because the first image was with movie record disabled ::). See the second image.

The scan says 'seems to work' for channels 2-6, 8, 10-13, 19-22 and 24-29.

Logging still freezes with LV off. Where do I set the interval?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 10, 2017, 01:30:30 PM
The "seems to work" only looks at whether the tested channel is locked (http://www.magiclantern.fm/forum/index.php?topic=6740.0) by some other code (usually Canon code) or not. One may assume Canon code would always lock the channels they use, and unlock them when they no longer need them, but take it with a grain of salt.

The green color on channel 2 (which you have reused for raw capture) means that channel is transferring data. I don't think it's a good idea to use it, even if it appears to work at first sight. To see the activity in nearly real-time, press and hold the up or down key in the EDMAC screen (which won't change the current selection, but will cause the menu to redraw faster). Whenever there's green on any of the channels, even for a very short time, that means "stay away from it".

For LOG_INTERVAL:

Quote from: dfort on February 08, 2017, 05:28:39 AM
I don't know much about coding or reverse engineering or playing the violin but one valuable lesson I learned from dmilligan is to look at the code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: masc on December 10, 2017, 02:09:19 PM
I can confirm the results of @reddeercity from here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194448#msg194448) on my 5D2.

This time it even works without frame rate override for me @25fps. mlv_lite 1880x1056@10/12bit, mlv_rec+mlv_snd 1856x1044@10/12bit+audio works like a charme (both non-crop). Yeah. :)

When not recording the liveview was relativly slow, maybe around 2fps and a "BUSY" was written in the corner left below of the screen and in LCD display. Using the standard nightly this is more fluently... But when recording all is fine again (okay... there are nearly no ML elements on the screen...)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 10, 2017, 02:38:57 PM
@dfort Channel 0x10 does not work without fps override on 5D2, retested it with the same code 0x02 was working on. Maybe I'll have to go searching for one that doesn't have the stutter 0x02 and 0x01 has.
0x02 is best on 5D2 as of this moment.

BTW what will the process of getting crop mode not broken again be like?


... I'll try the edmac module

Update:
(http://image.ibb.co/gHy6vG/edmac5d2.png)

@andy600 channel 1 and 2 flash green sometimes, 0 is always green. 4,5,6 are always grey.  (replying to post below) Nope, only 0x05 is always grey. @a1ex but 0x02 works so well despite green :( what to do...

Are grey channels the ones to try?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 10, 2017, 02:46:55 PM
Quote from: a1ex on December 10, 2017, 01:30:30 PM
... Whenever there's green on any of the channels, even for a very short time, that means "stay away from it".

Got it. 0x02 flashes green but 0x01 doesn't. I'm using 0x01 on the 50D so I assume it's safe!?

@dfort this needs checking on the 5D2 with the EDMAC module.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 10, 2017, 02:55:54 PM
Good advice about the code.

Log interval must be line 334!? https://bitbucket.org/hudson/magic-lantern/src/97b73e7a781e78f3364555c5c30afbd3c0e5cae8/modules/edmac/edmac.c?at=unified&fileviewer=file-view-default#edmac.c-334 (https://bitbucket.org/hudson/magic-lantern/src/97b73e7a781e78f3364555c5c30afbd3c0e5cae8/modules/edmac/edmac.c?at=unified&fileviewer=file-view-default#edmac.c-334)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 10, 2017, 03:41:12 PM
1 is likely safe on 50D, but not on 5D2.

5 is likely used only when entering LiveView or when changing video mode, so it might be OK to reuse. The yellow ones may also be worth trying. No idea why some channels, with apparently identical configurations, give different results.

For color coding, the advice from dfort dmilligan also applies.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2017, 05:33:16 PM
Just waking up here in LA LA Land and whoa--lots of reading to do.

So it looks like the 50D can use channel 0x1 and the 5D2 will go back 0x10 and will require the fps override workaround until we can find something that works better.

Where I left off last night was failing on yet another attempt at merging the work in progress for the Digic IV LVState cameras into the crop_rec_4k branch. The reason why I'm trying so hard is to get on the newest mlv_lite.

Something that I noticed this time was that the code we're testing uses RAW_LV_BUFFER_ALLOC_SIZE in place of DEFAULT_RAW_BUFFER. Check out this function in the allocate-raw-lv-buffer branch:

static void* raw_get_default_lv_buffer()
{
#if !defined(CONFIG_EDMAC_RAW_SLURP)
    return (void*) shamem_read(RAW_LV_EDMAC);
#elif defined(DEFAULT_RAW_BUFFER)
    return (void*) DEFAULT_RAW_BUFFER;
#elif defined(CONFIG_ALLOCATE_RAW_LV_BUFFER)
    return raw_allocated_lv_buffer;
#else
    #error DEFAULT_RAW_BUFFER not configured.
#endif
}


The only camera that looks to be coded properly in the crop_rec_4k branch is the 5D3 and it has both DEFAULT_RAW_BUFFER and CONFIG_ALLOCATE_RAW_LV_BUFFER so the above code won't work.

What about the other cameras that are functioning in the crop_rec_4k branch? Seems there are some hacks in there like using this from the 5D3:

#define DEFAULT_RAW_BUFFER_SIZE (0x4ee00000 - 0x4d600100)

It is there in the 5D3.113 disassembly. It changed somewhat in 5D3.123 because the DEFAULT_RAW_BUFFER moved to 0x48600000 [EDIT + 0X100 so that should be 0x48600100] but I can't find it or anything like it in the newest 1.3.5 firmware or in the 100D/650D/700D/EOSM disassemblies.

[EDIT2] Think I'm misunderstanding this because there are plenty of references to 0x48600000 in all of the 5D3 firmware versions.

The comments on this commit (https://bitbucket.org/hudson/magic-lantern/commits/fce0d99f05bfc0e6598f0cf5f752e838d28bf86a) bring to light that this will work for now but we will eventually "need the correct value for resolutions higher than Canon firmware." I'm going a little off topic with this but @reddeercity -- take note.

So I thought I'd go through some links that might help understand this, like the LVState diagram for the 7D and Holy S***

(https://farm5.staticflickr.com/4556/24094350407_4427095897_c.jpg) (https://a1ex.bitbucket.io/ML/states/7D-alt/sm20.htm)

The reason why I've been so active lately was because my wife has been on vacation. She returns this afternoon and if she sees these charts hanging on the walls she will for sure think I lost it.

Funny that a1ex is saying to follow my advice to look at the code, which was really dmilligan's advice to me. [EDIT] Attribution corrected.

What I feel like right now is more like this:

Quote from: High Anxiety by Mel Brooks
I got it. I got it. I got it.
[thump]
I ain't got it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bouncyball on December 10, 2017, 05:46:46 PM
 :o 8) :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2017, 06:58:36 PM
@dfort
Haha, too funny :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 10, 2017, 08:09:04 PM
i use raw_video_10bit_12bit.2017Dec08.50D109, but now see new raw_video_10bit_12bit.2017Dec10.50D109.zip what deference  ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 10, 2017, 08:33:32 PM
Quote from: dfort on December 09, 2017, 04:16:33 PM
This morning I tried zoom mode and although the LV refresh rate was rather low I got good video at every resolution all the way up to 2520x1192 at 10bit. Don't get too excited, next time I tried it all I got was a garbled LV and corrupted frames. About the only thing consistant in zoom mode is that once you press the magnifying lens button you can't get out of that mode without restarting the camera. Still, there is a glimmer of hope there.

Dfort,

There is a way of getting out of the frozen 5x-magnification mode on the 7D without restarting the camera (see my post #1390 on page 56).  After you record a clip in that mode, just press the play button to play it back in the camera.  You get a black screen with a frame counter working.  After getting out of this black screen, (press half shutter), the camera returns to its normal state and you do not need to restart it.  It does not work always but most of the time. 

By the way, 10/12-bit recording in the 5x-magnification mode was working fine, up to 2520x1200 resolution, in the Dec. 1/2016 built that you can download from the download page of Reddeercity.  In that built, the frozen-screen issue could easily be fixed just by switching back and fort from Auto to Canon mode in the preview submenu.  After doing that, the screen would return to its normal LifeView state and would allow composing, metering and focusing for all clips to follow. As far as I remember, there was also a working raw_twk module in that built, so it was possible to playback your clips in the camera to check focus and composition for example - extremely useful!

From the practical point of view, it is very important to fix the 10/12-bit recording in the 5x-magnification mode because of its increased vertical resolution (1200p) which is unique to the 7D.  Maybe, if you compare your code with the one from the above Dec. 1/2016 built, the problem could be more easily found and fixed.  Just a suggestion.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2017, 09:20:58 PM
Quote from: CITY-U1001 on December 10, 2017, 08:09:04 PM
what deference  ?

Check this:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/16/10bit-12bit-for-lvstate-cameras/commits

You will see the latest commits. Look for the one that matches the changset on the build you are using. (7a58a90...)

(https://farm5.staticflickr.com/4528/38929622592_d679d59e48.jpg) (https://flic.kr/p/22j5uvw)

Note that on Bitbucket is is truncated but that's fine. You can also open autoexec.bin in a text editor to see the information including any changes that were not committed.

Short answer, I wanted to make sure that the posted builds are using channel 0x1 for the 50D and 0x10 for the 5D2.

@IDA_ML - Thanks for reminding me about your tip. I was able to get out of 5x mode using the play button. Of course we want to get out of using workarounds like that. As far as the builds @reddeercity put up, could you do me a favor and open autoexc.bin in a text editor and report who built it? I do most of my development on rosiefort@RosieFoComputer. If it is mine you can find the source code by searching my repository for that changeset. None of this requires coding skills, it can all be done on the Bitbucket website.

Something I'm trying now is why not throw out this whole CONFIG_ALLOCATE_RAW_LV_BUFFER thing for now and instead of this:

#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)

do this:

#define DEFAULT_RAW_BUFFER (SRM_BUFFER_SIZE - 0x1000)

It worked the same on the 7D and this would be much easier to get working in the crop_rec_4k branch. But wait, that isn't the "real" DEFAULT_RAW_BUFFER. Looking a little closer:

#ifdef CONFIG_EDMAC_RAW_SLURP
/* undefine so we don't use it by mistake */
#undef RAW_LV_EDMAC


So what if we used it on purpose? The only change needed in raw.c would be to remove that line undefining RAW_LV_EDMAC. It worked! Well, let me qualify that, it has the same issues on the 7D with the vertical resolution being limited to 976 and zoom mode doesn't work. In other words, it is working the same as before.

Hum, does that mean we have a rather large fudge factor on these buffer sizes or am I having a lucky day?

[EDIT] Or has this all been a trip down the wrong path because certainly there is a good reason to undefine RAW_LV_EDMAC. It looks like it was done a while back (https://bitbucket.org/hudson/magic-lantern/commits/52184ce01e74ce2f71bf3ae7c4b5c3b543cb77c5) to fix an issue with pink frames. Of course the same thing can be done without having to define RAW_LV_EDMAC though it would take more than a one line change to raw.c.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 10, 2017, 09:43:19 PM
Quote from: dfort on December 10, 2017, 09:20:58 PM
#define DEFAULT_RAW_BUFFER (SRM_BUFFER_SIZE - 0x1000)

Do not do this. Camera bricking risk is real.

Allocate it, use a pre-allocated one, or find some unused*) memory block. Don't try random pointers hoping they will work.

*) If you decide to use this one, double-check it's really unused (test it in any camera mode you may think of - photo, movie, zoom, various FPS, while recording, external monitors, and make sure it's not going to be reused by other Canon code while you are updating it). Tools you may use: CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP, CONFIG_RSCMGR_UNUSED_SPACE_TEST, or write your own RAM analysis tools.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 10, 2017, 10:04:44 PM
Dfort,

I just mentioned a built that was working for me before and where I found it.  I was just trying to help.  I am sorry if I did it the wrong way.  I didn't mean to.  You know how much I appreciate your work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2017, 11:16:24 PM
Quote from: a1ex on December 10, 2017, 09:43:19 PM
Do not do this. Camera bricking risk is real.

Oops--but isn't that essentially what is going on in the allocate-raw-lv-buffer branch?

How about this for the 7D?

#define DEFAULT_RAW_BUFFER shamem_read(0xC0F26008)

This isn't supposed to work with CONFIG_EDMAC_RAW_SLURP? I can't quite follow the instructions in the comments, much less write my own RAM analysis tools.

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
*
* raw_buffer = get_raw_buffer()
* sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
* ...
* dump_file(filename, raw_buffer, 7*something...)
*/


Quote from: IDA_ML on December 10, 2017, 10:04:44 PM
I was just trying to help.  I am sorry if I did it the wrong way.  I didn't mean to.  You know how much I appreciate your work.

Ditto from me to a1ex and everyone else participating in this.

My comments to you were similar to a1ex's to me. Instead of giving you the answer I'm trying to show you how to find the answer. Like the old proverb, "Give a man a fish and he eats for a day, teach a man to fish and he'll sit in a boat all day drinking beer."

Ok--here's the fish:

Magic Lantern Nightly.2016Dec01.5D2212
Camera   : 5D2
Firmware : 212
Changeset: b3dfbe7194f3 (x-perimental) tip
Built on : 2016-12-01 22:53:11 by rosiefort@RosieFoComputer
-e


Here (https://bitbucket.org/daniel_fort/magic-lantern/commits/b3dfbe7194f37256b15a3103f4e0f9320c952880) is that commit in Bitbucket. if you cloned my repository this is how do it on your local copy using Mercurial:

hg log -v -r b3dfbe7194f3
changeset:   14629:b3dfbe7194f3
branch:      x-perimental
user:        Daniel Fort <[email protected]>
date:        Thu Dec 01 14:19:54 2016 -0800
files:       src/raw.c
description:
added note about PREFERRED_RAW_TYPE 18 for EOSM


You can get to that changeset by doing this:

hg update -r b3dfbe7194f3

I just tried that version on the 7D and LV doesn't get garbled up like in the latest experiments but the 5x issue is still there and the MLV files aren't working properly. My understanding is that it needs CONFIG_EDMAC_RAW_SLURP and this old version doesn't have it defined on the Digic IV LVState cameras. Back when 10bit/12bit was a new thing I was posting builds for people who couldn't compile but eventually stopped doing that when the Experiments page (https://builds.magiclantern.fm/experiments.html) went up. Only the cameras ready for testing are on that page. Consider the work we're doing here pre-Alpha so you should be ready to dive into this much deeper than the casual user.

[EDIT] Just a reminder, we're all here to learn and share. Well, at least some of us are.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 10, 2017, 11:43:47 PM
Quote from: dfort on December 10, 2017, 11:16:24 PM
Oops--but isn't that essentially what is going on in the allocate-raw-lv-buffer branch?

Nope - https://en.wikipedia.org/wiki/C_dynamic_memory_allocation

RAW_LV_BUFFER_ALLOC_SIZE gets passed to fio_malloc; this returns a pointer to the allocated buffer, which is then stored in raw_allocated_lv_buffer (and used to save the LiveView frame there). The same happens in crop_rec_4k, with some additional bounds checking (which is only valid on 5D3 - the other models use dummy values just to compile).

DEFAULT_RAW_BUFFER is used as a pointer to some valid raw buffer (allocated by Canon code). On recent models (60D and newer), it is allocated when entering LiveView (evfActive), but that pointer is no longer valid when leaving LiveView (it's still there, but stale); on older models I believe it's allocated on request (by whatever code gets triggered by lv_save_raw) edit: it's also allocated when entering LiveView! edit2: false alarm, I forgot photo raw overlays enabled during the test!

5D3 1.2.3 (https://bitbucket.org/hudson/magic-lantern/commits/1ee967) (it appears to allocate 0x1cae000 bytes; MEM1 is the raw buffer we are looking for):

79152>        Evf:ff16b0dc:a9:03: [2] Flicker Address[0x4d318000][0x52518000]
791E9>        Evf:000aef04:00:00: *** lv_raw_buf(0x0, 0x0, 0x94897c, 0xc, 0x4d318000, 0x52518000, 0x1cae000), from ff1650d4
79200>        Evf:ff16b334:a9:03: Mem1 Address[0x4d31a000][0]


500D (full log (http://a1ex.magiclantern.fm/bleeding-edge/raw/500D/dm-0004.log)):

1B512> LiveViewAn:ff1f6224:9a:07: [00000000:043A8818] lvaStart
...
1B944> LiveViewMg:00092b38:00:00: [00000000:043A8818] *** LVState: (1) --20--> (1)          ff0e5424 (x=869970 z=4099b360 t=1)
1B9B4> LiveViewMg:ff0e5470:98:03: [00000000:043A8818] HPCopyAsync pvParam:0x4099b360 -copy-> Lucky:0x43fd4420
1BA24> LiveViewMg:ff0de974:9b:03: [00000000:043A8818] GetResource 0x1000 (0xF07/0x1F07)
1BB16> LiveViewMg:ff1f8e28:99:02: [00869A28:00000000] SetFlickerPassParameter
...
(after a while; LiveView working normally without raw for a few seconds)
...
42F97>   run_test:00059408:00:00: [00869A28:00000000] *** call lv_save_raw ***
43011>   run_test:00093268:00:00: [00869A28:00000000] *** TryPostEvent(LiveViewMgr, 0x9, 0x1, 0x200), from ff032728
430BD> LiveViewMg:00092b38:00:00: [00869A28:00000000] *** LVState: (2) --9--> (2)          ff0e56ec (x=869970 z=1 t=200)
43126> LiveViewMg:ff0e5738:98:16: [00869A28:00000000] Set Debug Flag 0x000006A1(0x08200)
...
7FB00> LiveViewMg:ff1fb03c:99:02: [00869A28:00000000] StopQuarkPass
7FB5F> LiveViewMg:ff1f5a94:98:02: [00869A28:463390A4] GetEngineResource SUCCESS Res:4, Free:fffffffc


The numbers in brackets are (added this to my_DebugMsg to print the possible raw buffer address on every debug message):

    len += snprintf(buf+len, buf_size-len, "[%08X:%08X] ", MEM(0x15CB0), MEM(MEM(0x15CB0) + 0x328));


Addresses coming from:

DebugMsg(153, 4, "StartPass_x1 CrawAddr : %lx / KindOfCraw : %d", *(v15CB0 + 0x328), *(v15CB0 + 0x32C));
BC24B> LiveViewMg:ff1f7190:99:04: [00869A7C:463390A4] StartPass_x1 CrawAddr : 463390a4 / KindOfCraw : 0


so DEFAULT_RAW_BUFFER on 500D would be MEM(MEM(0x15CB0) + 0x328) = MEM(MEM(0x15CA4 + 0xC) + 0x328).
7D: MEM(MEM(0x17BBC+0xC) + 0x338), not tested.
(addresses are good, but they are only valid with lv_save_raw)

BTW - that "Debug Flag" from the 500D log is actually the LiveView RAW mode in Canon firmware (a feature they have used for debugging). That should explain why the x5/x10 modes in very old models are broken (pink previews, unusable x10 etc), or why the LV raw buffer on 1100D is reused for something else (as this camera has very little RAM).





Back to your question.

Pointers are memory locations; they are not interchangeable with buffer sizes (even if the C compiler may accept this). The reason it might happened to work (sort of) is the large value of SRM_BUFFER_SIZE - it points somewhere way above the DryOS core (http://www.magiclantern.fm/forum/index.php?topic=5071.msg166843#msg166843), so if it screws up things, it might do so in some non-obvious way (remember those obvious bugs from 5D3.134).

With shamem_read(0xC0F26008), this reads the address previously configured on EDMAC channel 0x10, by some existing code. Without CONFIG_EDMAC_RAW_SLURP, that's Canon code enabled by lv_save_raw; they have configured that channel with whatever buffer they have allocated for the LV raw image. The point of the new method is to bypass Canon code so we can have full control over it (such as, specify a different "raw type" or - in our case - a lower EDMAC transfer size for 10/12-bit), but since we are not going to call lv_save_raw, there will be no Canon code to allocate a raw buffer for us. So that won't work - you'll get some old value and the results are unpredictable.

Why do you want to change it? The only valid reason I can see would be to avoid the BUSY screen - in this case, just allocate as much as you need. Find the largest frame size you want to use (usually that's the raw resolution in 5x zoom) and allocate a buffer large enough for that (but not much larger). In this case, the memory backend will see no reason to use SRM buffers (which may print the BUSY message while they are allocated).




BTW, what's that -e doing there? That may indicate yet another bug in the Makefiles (a similar one was on Mac when generating module READMEs). This didn't help locating it:

grep -nr -- " -e" Mak* */Mak* */*/Mak*





P.S. 60D EvfState (compare to 7D's LVState):
(http://a1ex.magiclantern.fm/bleeding-edge/raw/60D/60D-Evf.png)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 11, 2017, 10:45:44 AM
Quote from: a1ex on December 10, 2017, 11:00:34 AM
Does it help if you increase LOG_INTERVAL?

Nope. Tried 500ms, 1sec, 2sec, 5sec intervals with LV on/off, movie mode enabled/disabled etc and it still freezes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 11, 2017, 11:23:54 AM
Quote from: reddeercity on December 10, 2017, 08:40:50 AM
I can confirm with @Ilia3101 on 5D2 in 1:1 non crop mode with mlv_lite & mlv full+audio ,
I can record 10 , 12 , 14 bit with out any frame corruption or out of sync Lv preview.
I downloaded the source code , changed write channel from 0x01 to 0x02 in edmac-memcpy.c  , compiled it locally in my VM.
Tested 1856x928 , 1872x936 , 1880x1056 , 1880x1016 , 1880x1128 , 1880x1248 ,  23.976 with frame over ride to exact in Auto Preview with mlv_lite
Pressed the half shutter while recording to get actuate framing with out issue , in 10 , 12bit (14bit where short 200 frame captures)

In mlv full with audio I tested 1856x928,  1856x1044 , 1856x1114 , 1856x1238 , 1856x1248 ,  23.976 with frame over ride to exact in Auto Preview ,
also pressed the half shutter while recording to get actuate framing with out issue , in 10 , 12bit (14bit where short 200 frame captures)


3x Crop mode is broken , it seems to keep the first frame it see as a alternating frame while the other frame has movement so
like frame #1 is frozen and frame #2 has normal movement and keeps repeating that pattern but frame #1 is not broken -- as in
artifacts or shearing with pink frame just frozen . I tested mlv_lite 2152x1072 & Full mlv + audio 2144x1074

*Note* Full MLV +Audio : audio was sync and was perfect , not problems.

My Camera setting and build information
magic.cfg
# Magic Lantern Nightly.2017Dec10.5D2212 (ee95e8825183+ (raw_video_10bit_12bit_LVState-wip) tip)
# Built on 2017-12-10 03:33:12 UTC by ml@ml-pc
# Configuration saved on 2017/12/10 00:17:36
beta.warn = 10
menu.first = -4
movie.log = 1
rec.notify = 0
enable-liveview = 2
fps.preset = 1
fps.override.idx = 32
fps.override = 1
battery.drain.rate.rev = 69
hist.log = 0
spotmeter.draw = 0
zebra.draw = 0
disp.mode.x = 149



mlv_lite.cfg
# Config file for module mlv_lite (MLV_LITE.MO)

raw.video.enabled = 1
raw.res_x_fine = -40
raw.write.speed = 6399
raw.warm.up = 2
raw.bppi = 0


mlv_rec.cfg
# Config file for module mlv_rec (MLV_REC.MO)

mlv.video.enabled = 1
mlv.res.x.fine = 64
mlv.bpp = 0
mlv.write_speed = 6699
mlv.display_rec_info = 2
mlv.buffer_fill_method = 0


mlv_snd.cfg
# Config file for module mlv_snd (MLV_SND.MO) mlv.snd.enabled = 1

This code looks promising for the 3.5/UHD development with custom buffers , I'll see if I can implement this in to my source code.

Edit: if anyone that can't compile and what to try out  1:1 10 & 12bit raw video with mlv_lite & full mlv+audio I uploaded to my bitbucket downloads magiclantern-10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip)

Redeercity Thanks for the testing Build, Its a big step toward at least a 12bit build. I did a quick test with 12bit setting and I will be dong some more test during this week. How do I use your setting above, is it a past and copy trick?

Test 1-with Stabilization filter
"https://player.vimeo.com/video/246759604"
Test 1-without Stabilization filter
"https://player.vimeo.com/video/246758398"
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 11, 2017, 11:40:59 AM
@dariSSight read the last few posts and do not use any 5D2 build that uses EDMAC channels 0x01 or 0x02 for raw recording!!

More research needs to be done.

50D testers - the 50D builds using EDMAC 0x01 should be ok.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 11, 2017, 01:36:51 PM
Solved lock-ups in edmac.mo and polished it a bit - changes on the "edmac" branch. Merges cleanly into raw_video_10bit_12bit, if you prefer to try it there.

PR open (http://bitbucket.org/hudson/magic-lantern/pull-requests/886/edmac-module-fixes-for-digic-4-connection/diff).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 11, 2017, 04:03:39 PM
Quote from: a1ex on December 11, 2017, 01:36:51 PM
Solved lock-ups in edmac.mo and polished it a bit - changes on the "edmac" branch. Merges cleanly into raw_video_10bit_12bit, if you prefer to try it there.

PR open (http://bitbucket.org/hudson/magic-lantern/pull-requests/886/edmac-module-fixes-for-digic-4-connection/diff).

Working now :)

Not sure what you need from this but I ran the EDMAC test until it finished and also enabled logging. Uploaded the log files here (https://bitbucket.org/andy600/magic-lantern/downloads/).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 11, 2017, 04:22:38 PM
Quote from: Andy600 on December 11, 2017, 04:03:39 PM
Not sure what you need from this but I ran the EDMAC test until it finished and also enabled logging. Uploaded the log files here (https://bitbucket.org/andy600/magic-lantern/downloads/).

Looks fine.

Try logging at 50us (or 100us or more if it freezes the image or has other problems) a few scenarios in LiveView:
- without raw video and without photo raw overlays enabled (e.g. global draw off, no other modules loaded)
- with raw video enabled, old method (run it on top of regular nightly or raw_video_10bit_12bit)
- with raw video enabled, new method (on top of dfort's modified version)

Also, for my own curiosity: log a still photo, short exposure, outside LiveView, with 500us (or 1 ms if that doesn't cover the entire process). Look at the blue LED to see when it logs and when it stops.

Take a look at the EDMAC connections screen as well (Show EDMAC channels, scroll to the right).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 11, 2017, 04:57:34 PM
Quote from: a1ex
- without raw video and without photo raw overlays enabled (e.g. global draw off, no other modules loaded)

all logged at 50u

Without LV (https://bitbucket.org/andy600/magic-lantern/downloads/01_NO_LV_edmac000.log)

With LV (https://bitbucket.org/andy600/magic-lantern/downloads/01_WITH_LV_edmac001.log)

Quote from: a1ex
- with raw video enabled, old method (run it on top of regular nightly or raw_video_10bit_12bit)

still to do

Quote from: a1ex
- with raw video enabled, new method (on top of dfort's modified version)

using dfort mod (https://bitbucket.org/andy600/magic-lantern/downloads/03_edmac001.log)

Quote from: a1ex
Also, for my own curiosity: log a still photo, short exposure, outside LiveView, with 500us

500u (https://bitbucket.org/andy600/magic-lantern/downloads/photo_500u_edmac002.log)

Quote from: a1ex
(or 1 ms if that doesn't cover the entire process).

1ms (https://bitbucket.org/andy600/magic-lantern/downloads/photo_1s_edmac003.log)

Quote from: a1ex
Look at the blue LED to see when it logs and when it stops.

Stays on then flickers briefly before showing preview and 'logged' message.

Quote from: a1ex
Take a look at the EDMAC connections screen as well (Show EDMAC channels, scroll to the right).

Ok. what am I looking for?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 11, 2017, 06:56:35 PM
I'm at work today so I can't test but I can compile.

Merged  the EDMAC module: fixes for DIGIC 4, connection map (https://bitbucket.org/hudson/magic-lantern/pull-requests/886/edmac-module-fixes-for-digic-4-connection/diff) pull request and updated the builds on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).

Lots of studying and testing still to do on this.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 11, 2017, 10:58:23 PM
Pictures: plain LiveView (1) vs LiveView with raw enabled using new method (2, 3):

(http://a1ex.magiclantern.fm//bleeding-edge/capture/50D/edmac-stby.png)

(http://a1ex.magiclantern.fm//bleeding-edge/capture/50D/edmac-raw-dfort.png)

(http://a1ex.magiclantern.fm//bleeding-edge/capture/50D/edmac-raw-dfort-long.png)

Notice how channel #2 writes some data into RAM, but not for every single frame. It's transferring every now and then, either 32x15 or 32x225 or 32x240 bytes.

What exactly it's doing? To find out:
- capture a log using dm-spy-experiments branch, make sure to include EDMAC setup functions (SetEDmac, StartEDmac etc) and see what debug messages are nearby
- dump the data from that memory address and attempt to make some sense out of it

Already having such a log done for 500D (linked earlier), channel #2 is configured with same transfer size, so I'm looking there:

1E2F2> LiveViewMg:ff0da8e4:98:02: SetWbIntegRegisterForAe 0
1E31F> LiveViewMg:ff1fabf0:99:02: StartWbPass
1F3D5> LiveViewMg:000927a0:00:00: *** ConnectWriteEDmac(0x2, 0x2), from ff1fac44
1F46A> LiveViewMg:000927a0:00:00: *** RegisterEDmacCompleteCBR(0x2, 0xff1fa56c, 0x43ffa1e4), from ff1fac54
1F4D0> LiveViewMg:000927a0:00:00: *** SetEDmac(0x2, 0x43ff83e0, 0x15a200, 0x20000000), from ff1fac68
1F76E> LiveViewMg:000927a0:00:00: *** StartEDmac(0x2, 0x0), from ff1fac74


White balance.

That's why channel #2 is not safe to reuse on 50D (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194461#msg194461), 5D2 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194490#msg194490), 550D (http://www.magiclantern.fm/forum/index.php?topic=19336.msg187341#msg187341), 500D and 7D (https://www.magiclantern.fm/forum/index.php?topic=19336.msg194105#msg194105) (links pointing to EDMAC screenshots).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 12, 2017, 06:32:44 AM
A brief feedback on the Dec. 11-th build for the 7D that I have tested:
==============================================

Good news:

1) If you add the RAW-tweak module from the old Dec. 1-st/2016 build, discussed above, to the Modules directory, you can now playback your 10/12-bit recorded clips in the camera!  Playback works fine on normal and 5x-crop recorded clips.  You can find this build here:

http://www.magiclantern.fm/forum/index.php?topic=5601.msg187903#msg187903

Bad news:

2) The overall camera behaviour remains pretty much the same as described in my post #1390 on page 56.  Camera screen still freezes when you press the magnification button once but 5x-crop recording with sound works just fine in this condition up to 2496x1198 resolution.  Getting out of the frozen screen state does no longer require camera restart with taking battery out, all you need to do is to press the play button and the camera returns to its normal LifeView screen where framing, focusing and metering is possible.

Conclusion:
------------
In the normal non-crop mode, the camera provides very stable and consistent 10/12-bit continuous recording with sound up to 1728x972 resolution and up to 30 fps.  If the frozen-screen issue in the 5x-magnification mode could be fixed this would make the EOS 7D a fully capable camera for 10/12-bit RAW video recording with synchronous sound and resolutions up to 2496x1198 at 24 fps.

Keep up your excellent work, guys!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tonij on December 12, 2017, 07:45:10 AM
Regarding the 7D frozen screen issue in the 5x magnification mode, is there anything that could be learned from this build: http://www.magiclantern.fm/forum/index.php?topic=5601.msg187903#msg187903 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg187903#msg187903)
It has a working liveview (canon mode) with global draw off for 10/12 bit in 5x mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2017, 08:35:41 AM
Quote from: tonij on December 12, 2017, 07:45:10 AM
is there anything that could be learned from this build

Sure, another one of my old builds coming back to haunt me.

Magic Lantern Nightly.2016Dec01.7D203
Camera   : 7D
Firmware : 203
Changeset: b3dfbe7194f3 (x-perimental) tip
Built on : 2016-12-01 23:00:07 by rosiefort@RosieFoComputer
-e


@a1ex - that "-e" is showing up only on these old builds.

Quote from: tonij on December 12, 2017, 07:45:10 AM
It has a working liveview (canon mode) with global draw off for 10/12 bit in 5x mode.

My understanding is that it works only in zoom (5x) mode but now that I've got access to a 7D I'm not having much luck getting that working.

Quote from: tonij on August 01, 2017, 02:50:32 AM
-In Movie menu set FPS override to 24 FPS Exact.
...
-In Overlay menu set Global Draw to OFF, return to Liveview in 5x zoom
...
Edit: I've noticed there's a few pixles in the bottom right corner of the video that do strange things..

Is it necessary to set FPS override? That is something being reported on the 5D2 on these latest experiments too. Also, is turning off Global Draw necessary?

Pixels on the bottom right doing "strange things?"

That old build doesn't have CONFIG_EDMAC_RAW_SLURP or CONFIG_ALLOCATE_RAW_LV_BUFFER which is what the latest test test builds for Digic IV LVState are using. Maybe it was just luck that it worked. With the new builds you should have mv1080 and mv720 pretty much working and sometimes zoom mode, depending on the luck factor.

By the way I have been re-reading this topic, making corrections on some of my posts and studying up on things.

Late breaking news -- @nuvon did find the SRM_BUFFER_SIZE for the 1100D. It is: 0x14e8000
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 12, 2017, 05:23:16 PM
Quote from: dfort on December 12, 2017, 08:35:41 AM
Sure, another one of my old builds coming back to haunt me.

Ha ha, Dfort, this old build of yours keeps coming back to haunt you because this is the best ML build made for the 7D in the history of mankind!  Once it came out, I installed it and used it throughout the entire 2017 for my amateur videos.  If worked so well at 10 bits with sound in the 5x-crop mode that I never thought to replace it.  And if I needed the normal mode, I used it at 14 bits - still very stable.  The only way to stop this old buddy haunting you is to write a better one - possibly with 4K crop recording and lossless compression.  Only then, your old trusted Dec. 1/2016 build will find peace and will rest in peace too  ;) ;) ;) ! 
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2017, 09:04:00 PM
Quote from: IDA_ML on December 12, 2017, 05:23:16 PM
...this old build of yours keeps coming back to haunt you because this is the best ML build made for the 7D in the history of mankind!

Yes, we can learn from the past. Fortunately all this is being done on Bitbucket and reported on the forum so it is rather easy to go back and see what was being done back then.

I got a PM from @DeafEyeJedi. Hopefully he'll share some of the video he shot on the 7D to illustrate this issue:

Quote from: DeafEyeJedi on December 12, 2017, 10:14:19 AM
First MLV LITE 10-bit works like normal but then afterward it's all choppy with first frame repeating every other frame or so. We have seen this behavior before on other cameras. Don't remember which.

It is quite disappointing to shoot a bunch of footage in the field only to find out when you get home that it didn't turn out so I updated the test builds (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) with raw_twk. This way you can preview 10bit/12bit MLV's in camera. Today's builds also include the edmac module.

Noticed that there have been downloads of all cameras in one day so there's interest in this. It would be good to get some more reports from testers.

Quote from: a1ex on December 10, 2017, 09:43:19 PM
Don't try random pointers hoping they will work.

Ok--so now that you pointed out that DEFAULT_RAW_BUFFER is returned as a pointer and CONFIG_ALLOCATE_RAW_LV_BUFFER is not, I can see there is a significant difference between the two.

#elif defined(DEFAULT_RAW_BUFFER)
    return (void*) DEFAULT_RAW_BUFFER;
#elif defined(CONFIG_ALLOCATE_RAW_LV_BUFFER)
    return raw_allocated_lv_buffer;


I'm still very much a beginner in this stuff and haven't quite wrapped my head around what a pointer is. Most of what I learned in coding comes from Googling stackoverflow (https://stackoverflow.com/questions/11626786/what-does-void-mean-and-how-to-use-it). This is a rather simplistic answer but it helps clear things up a bit:

Quote from: stackoverflow (paraphrased)
(void*) - "One can say void is nothingness void* is everything"

There's a lot more reading and testing to do. One thing to try is CONFIG_RSCMGR_UNUSED_SPACE_TEST. Looking at the code it says:

        /* take an unused block from RscMgr and manage it with AllocateMemory's low-level allocators */
        /* the block must be tested to make sure it's really free before using it */
        /* to do so, define CONFIG_RSCMGR_UNUSED_SPACE_TEST,
         * then use and abuse the camera for a while in this configuration */
...
        .preferred_max_alloc_size = 5 * 1024 * 1024,


5 * 1024 * 1024 = 5242880 or 0x500000 Right? We're shrinking SRM_BUFFER_SIZE somewhat ( 0x1F80000 - 0x1000 on the 7D) but that is a much larger value than the preferred_max_alloc_size noted there.

I take it that playing around with CONFIG_ALLOCATE_RAW_LV_BUFFER is safer than DEFAULT_RAW_BUFFER but don't we need both of these in order to make the jump to the crop_rec_4k branch? As I recall the whole issue why the Digic IV LVState cameras weren't working properly was because nobody was able to find the DEFAULT_RAW_BUFFER on these cameras. As @IDA_ML, @reddeercity and others have noted they can record 10bit in zoom mode but this might just be a "lucky" coincidence.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 12, 2017, 09:45:35 PM
Quote from: dfort on December 12, 2017, 09:04:00 PM
Ok--so now that you pointed out that DEFAULT_RAW_BUFFER is returned as a pointer and CONFIG_ALLOCATE_RAW_LV_BUFFER is not, I can see there is a significant difference between the two.

They are both pointers; the only difference is who allocates them (Canon code or our code).

Quote from: dfort on December 12, 2017, 09:04:00 PM
As I recall the whole issue why the Digic IV LVState cameras weren't working properly was because nobody was able to find the DEFAULT_RAW_BUFFER on these cameras.

No (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176545#msg176545), the whole issue was (and still is) proper sync to avoid corrupted frames (in all video modes).

The sync issue has two components. An important one was frame size in 10/12-bit modes (which affected EDMAC behavior - it went out of sync), which was solved by RAW_SLURP. From current reports, this was not sufficient to solve all camera models x video modes. The remaining issue is probably that RAW_SLURP does not always start transferring the raw data at the right moment (like Canon's lv_save_raw hopefully does). We need to find when to call raw_lv_vsync: once for every LiveView frame - we are already doing that - but when exactly? This time we have to sync with Canon's image capture process. The newer models (60D and newer) already do that (evfReadOutDoneInterrupt).

The buffer does not affect sync at all (it's just a place in memory where the data is going to be stored) and the EDMAC channel must not conflict with anything else, but that's common sense. These were required by RAW_SLURP. BTW - changing that 0x1000 is not going to make things any better or worse, as long as the allocation succeeds.

It is also possible to solve the frame size issue without RAW_SLURP (and therefore without a custom buffer and without changing the EDMAC channel), with cache patches. That will keep the other shortcomings of lv_save_raw (side effects in x5/x10 on most old models) and the backend for cache patching is still quite fragile in my opinion, so I'd prefer to avoid this route.

The keys for solving the sync issue are in that big LVState diagram, the debug logs from LiveView (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194464#msg194464) (dm-spy-experiments branch; for models other than 500D you will have to find some stubs (https://bitbucket.org/hudson/magic-lantern/src/bc46b081b25bd901cd5b604faf0c86774ac6794a/src/dm-spy-extra.c?fileviewer=file-view-default#dm-spy-extra.c-264)) or EDMAC activity logs/graphs (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194545#msg194545) (currently captured for 50D, but incomplete). I hope to be able (but cannot promise) to solve the sync issues after analyzing all these logs.

All the logs have to be captured in 3 LiveView modes (see #1450 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194545#msg194545)) in order to have something to compare. Repeat for all video modes (1080p, 720p, 5x, whatever else your camera has). Repeat for all camera models (not just 50D). The two logging tasks are also for camera models already present on the Experiments page (for double-checking and for having something to compare against).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 12, 2017, 09:50:51 PM
50D
MLV_rec 1568х882 10bit all good, on Transcend 400x 32Gb. raw_twk work fine for mlv_play.
build: raw_video_10bit_12bit.2017Dec12.50D109
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 13, 2017, 09:28:38 AM
Man, you are moving fast, Dfort!  Amazing work!  Thank you so much!

I have noticed that you have a 7D now.  Congratulations!  I am sure, once you get a feeling for this fantastic camera, you will never reach for your other ones.  I have been testing your latest builds for hours now and I still have some juice in the battery left.

I have tested your Dec. 12 build on my 7D but I have to run for work now.  I will report on what I found later.

Keep up!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tonij on December 13, 2017, 10:01:23 AM
Shot some 10bit today with the Dec 12 build on 7d.
All 24fps exact in normal mode and 5x mode with sound.
Found myself having to restart the camera after each recorded clip or the next clip would get that "earthquake shaking"
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 13, 2017, 03:55:50 PM
Not sure where to put this but here we go:
Tried compiling and enable a version from dfort branch crop_rec_4k_7D_wip and when mlv_lite enabled camera always starts off with error 70. Maybe stubs aren´t correct? Didn´t check into it too much.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: coco770108 on December 13, 2017, 04:21:30 PM
Thx @dfort
I have tried the last version (2017dec12) released from your bitbucket, on 500D it can be work on normal mode, in 5X mode the screen will be frozen.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 13, 2017, 05:28:49 PM
Here is a brief feedback on the raw_video_10bit_12bit.2017Dec12.7D203 version:

Good news:
=======

1) It is amazing how long the battery life of the 7D is with these new builds.  I played last night for several hours and also this morning for a while and I sill have some energy left in my battery although it is at least 5 years old.  Surprisingly, the maximum temperature I reached did not exceed 51 deg. C which is very good.

2) Raw-tweak works very well and is extremely useful.  The same module used in the Dec.1/2016 build works flawlessly in your current builds as well.  Amazing!

Bad news:
=======

3) In the normal uncropped modes I no longer get this clean, stable and non corrupt video that I got on the December 11 build.  As Tonij said, I get "earthquake shaking" all the time, regardless of the resolution and bitrate set and this happens in both: the MLV and RAW video modes.  With my card (Transcend 16GB/x1000), "earthquake shaking" happens all the time, even if I turn the camera off and then on again.

4) Also in the 5x-crop modes things have gotten worse.  Sometimes the screen freezes, sometimes it shows jumping corrupt frames, and all this makes composing and metering impossible.  Focus does not engage when I press the back button for focusing.  I have to get out of this mode by pressing the play button and only then composing, focusing and metering is possible. 

5) Recording in the 5x-crop mode is still possible at the maximum 2496x1198 resolution but I rarely get more than 300 frames recorded at 24 fps.  With the Dec. 11 build I was sometimes able to record 500-520 frames without a single corrupt frame. Now, I get lots of corrupt (jumping) frames with the Dec. 12 build. 

6) I have noticed that features from the crop_rec_4K branch have been added (e.g. prerecording). I did not have time to play with these features but given the fact that recording is not stable and consistent enough, this does not make much sense.

7) I would like to update this post with another observation of mine concerning RAW-video recording with the RAW_REC module.  Sometimes recording in the 5x-magnification mode is OK, sometimes the recorded clip is full of corrupt frames, regardless of the resolution.  When corrupt frames start to appear in the clips, the camera needs to be turned off and on again and then one has the chance to record a few more clips without corruption.  At some random and not reproducible moment the corrupt frames start appearing again.  Sometimes this happens also in the normal uncropped mode but less frequently.  The MLV_REC module seems to work in a more stable way and with synchronous sound. 

My suggestion to Dfort:
==============
Including too many new features at this point, especially crop_rec_4K, in my opinion, does not make much sense.  Fixing things one at a time and getting more stable and consistent operation of the existing ones would be better.  I would suggest that you go back to your December 11 build and start from there.  That build was quite stable in the uncropped modes and also recorded fine in the 5x-ones.  You could try to fix the freezing-screen issue first which, in my opinion, was the only serious problem with that Dec.11 build.  It may help if you again compare your code from the old "haunting" Dec. 1/2016 build in its 5x-crop recording part with your current code and see if that can help.  There might be some slight difference that causes the current problems.  I installed that old build again yesterday and confirm: in the 5x-crop mode you can easily get out of the frozen-screen state just by switching the preview from Canon to Auto and back to Canon or by turning Global Draw off and on again as Tonij described.  Once you unlock the frozen screen, LifeView and recording in that mode work normally for all 5x-crop clips to come.  You have to repeat this screen unlocking procedure only if you turn the camera off and on again or if you switch to the uncropped mode and then again to the 5x-crop mode.

I hope, this helps and keep my thumbs pressed for more achievements from you !   
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 13, 2017, 08:55:30 PM
Quote from: Danne on December 13, 2017, 03:55:50 PM
Tried compiling and enable a version from dfort branch crop_rec_4k_7D_wip...

Quote from: dfort on November 30, 2017, 07:12:14 AM
Attacking this all at once is overwhelming, believe me I tried.

@Danne -- You discovered one of my experimental branches. I was able to get the 7D to compile but there are a lot of dummy stubs in there. It was too much trying to get all of the features working at once so I decided to work on this one feature at a time. With a1ex's guidance a few of us were able to get some features working on various Digic V cameras but these older Digic IV models are more challenging.

@coco770108 -- Welcome! Good to know that someone is testing the 500D. a1ex is running that one in QEMU and much of what we know about these Digic IV LVState cameras seem to be coming from the research done on that camera.

Quote from: tonij on December 13, 2017, 10:01:23 AM
Found myself having to restart the camera after each recorded clip or the next clip would get that "earthquake shaking"

I noticed that shaking too. Seems like the sync issue is somewhat random but restarting clears it up? We should consider that when capturing the logs.

@IDA_ML -- We've had some private conversations so I know you are a scientist and your reports are the most detailed. I'm not adding any crop_rec_4k features yet and the only things that have changed recently were the addition of a couple of modules that should not change anything unless you enable those modules, in theory at least.

Testers - what would really help is if you could provide the logs that a1ex is asking for:

Quote from: a1ex on December 12, 2017, 09:45:35 PM
The keys for solving the sync issue are in that big LVState diagram, the debug logs from LiveView (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194464#msg194464) (dm-spy-experiments branch; for models other than 500D you will have to find some stubs (https://bitbucket.org/hudson/magic-lantern/src/bc46b081b25bd901cd5b604faf0c86774ac6794a/src/dm-spy-extra.c?fileviewer=file-view-default#dm-spy-extra.c-264)) or EDMAC activity logs/graphs (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194545#msg194545) (currently captured for 50D, but incomplete). I hope to be able (but cannot promise) to solve the sync issues after analyzing all these logs.

All the logs have to be captured in 3 LiveView modes (see #1450 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194545#msg194545)) in order to have something to compare. Repeat for all video modes (1080p, 720p, 5x, whatever else your camera has). Repeat for all camera models (not just 50D). The two logging tasks are also for camera models already present on the Experiments page (for double-checking and for having something to compare against).

This will be more complicated. Andy600 has already created some of these logs on the 50D. I know that Danne, DeafEyeJedi and others have submitted logs - IDA_ML you can do this! I'm currently working on getting those missing stubs and will post a special version of the dm-spy-experiments branch.

I haven't tried it yet but if anyone wants to get a jump on this here's my dm-spy-experiments work in progress branch (https://bitbucket.org/daniel_fort/magic-lantern/branch/dm-spy-experiments_7D?dest=dm-spy-experiments#commits).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 13, 2017, 11:45:51 PM
I see. I was aiming for to inject ErwinH mlv_lite.c and mlv_snd.c into that specific version. Great work by the way.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 14, 2017, 01:23:38 AM
Quote from: Danne on December 13, 2017, 11:45:51 PM
I was aiming for to inject ErwinH mlv_lite.c and mlv_snd.c into that specific version.

I'd like to get the new mlv_lite merged into this but haven't succeeded with that yet. Maybe someone else could have a go at it?

Quote from: Andy600 on December 10, 2017, 02:46:55 PM
Got it. 0x02 flashes green but 0x01 doesn't. I'm using 0x01 on the 50D so I assume it's safe!?

@dfort this needs checking on the 5D2 with the EDMAC module.

I don't have access to a 5D2. On the 7D both 0x01 and 0x02 flashes green. The "Find free EDMAC channels" of the EDMAC module confirmed that channel #16 (0x10) "seems to work!" Others that are listed as seems to work are: 3-6, 10-13, 16-17, 19-22, and 25-29.

Compare this to the 50D:

Quote from: Andy600 on December 10, 2017, 11:26:50 AM
The scan says 'seems to work' for channels 2-6, 8, 10-13, 19-22 and 24-29.

So the code to find free channels seemed to work on the 7D but we're using channel 1 on the 50D so it validated my earlier post:

Quote from: dfort on December 07, 2017, 08:58:15 AM
You might notice that there is some code to find free edmac channels but experimenting around I found that sometimes what isn't listed might work better.

Testers -- the EDMAC module is in the latest builds that I posted (https://bitbucket.org/daniel_fort/magic-lantern/downloads/):

(https://farm5.staticflickr.com/4686/24175086187_dd53d39a18.jpg) (https://flic.kr/p/CQgCSP)

(https://farm5.staticflickr.com/4588/38156314745_41d244a52a.jpg) (https://flic.kr/p/218K62B)

Here are some 7D EDMAC logs ([EDIT] mv1080 video mode):

EDMAC model test (https://pastebin.com/D9bfkAWq)
50µs without raw video and without photo raw overlays enabled (https://pastebin.com/ANphfruh)
50µs with raw video enabled, old method (run it on top of regular nightly) (https://pastebin.com/46nve3Qj)
50µs with raw video enabled, new method (on top of dfort's modified version) (https://pastebin.com/WEvUy2hG)
500µs still photo, short exposure, outside LiveView (https://pastebin.com/c5G6vFqr)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 14, 2017, 04:19:43 AM
ON 5D2 Set to movie mode -- shot a short 10bit mlv + take a Full Res Silent photo while edmac.mo was logging
edmac000_5d2.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/edmac000_5d2.log) & EDMAC.LOG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/EDMAC.LOG) from dfort builds # Magic Lantern Nightly.2017Dec12.5D2212 (94fa79203875+ (raw_video_10bit_12bit_LVState-wip) tip)
# Built on 2017-12-12 18:21:30 UTC by rosiefort@RosieFoComputer
# Configuration saved on 2017/12/13 20:06:55


Channel#16 has the buffer I use for UHD , but I see I bigger buffer at EDMAC#12 that I can try out , lot of info -- need to study more

(http://preview.ibb.co/cZxtC6/Edmac_Log_1.png) (http://ibb.co/k1UjKm)
(http://image.ibb.co/jvq8em/Edmac_Log_2.png) (http://imgbb.com/)

Misc. Screen Shots while EDMAC Logging -- Lasts image after I took a full res silent photo

(http://image.ibb.co/kOcJC6/VRAM9_1_small.png) (http://imgbb.com/) (http://image.ibb.co/c3xks6/VRAM10_2_small.png) (http://imgbb.com/) (http://image.ibb.co/fGTKKm/VRAM11_3_small.png) (http://imgbb.com/)

Edit: Also took a plain silent photo and notice it was 1880x1248 instead of 1880x1250 which I though was strange .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 14, 2017, 02:32:05 PM
Quote from: reddeercity on December 14, 2017, 04:19:43 AM
Channel#16 has the buffer I use for UHD , but I see I bigger buffer at EDMAC#12 that I can try out , lot of info -- need to study more

Show EDMAC channels (EDMAC module settings) and push the joystick up. Does the channel flicker green? If it does you can't use it.


(https://images2.imgbox.com/51/54/6aSgvlKB_o.jpg)


e.g. Ch2 here does flicker green but Ch1 does not.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 14, 2017, 06:24:31 PM
Quote from: dfort on December 14, 2017, 01:23:38 AM
The "Find free EDMAC channels" of the EDMAC module confirmed that channel #16 (0x10) "seems to work!" Others that are listed as seems to work are: 3-6, 10-13, 16-17, 19-22, and 25-29.
About which camera? Are you on about the yellow <w!> channels? Is it worth trying a few of those out?

Incase anyone missed it: 5D2 EDMAC screenshot (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194490#msg194490) (with some info to read).
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 14, 2017, 08:53:50 PM
I'm testing on a 7D.

You can try other channels but the main issue seems to be sync, not the channel we're using.

It would be great if you could run those EDMAC module tests and post logs for the 5D2.


Sent from my iPhone using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 15, 2017, 12:04:08 PM
Dfort,

I tried to generate some logs on the 7D but I am not sure if I am doing this porperly.  Here is what I did:

1) I activate "Start logging"
2) I go back to the RAW or MLV video mode and see the "Press shutter halfway to start" message
3) I press the record button to start video recording and then I press half shutter
4) I get the "Out of memory message".  No log file is recorded.

A log file gets recorded only if I skip step 3) above but is it really supposed to work that way?  Isn't it supposed to log during real-time video recording to understand what is going on?

I also looked at some of the log files that you provided but this is like reading Chinese to me.  Also, my old eyes cannot read these sequences of small numbers and text any more.  I am sorry to say that but I cannot help here either.

Please provide more detailed instructions on how this logging procedure should be run and also EXACTLY at what camera settings, (more details please, on how the camera should be set).  We testers cannot provide useful information if we do not understand how things work.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 15, 2017, 05:49:38 PM
Quote from: IDA_ML on December 15, 2017, 12:04:08 PM
...
3) I press the record button to start video recording and then I press half shutter
...

A log file gets recorded only if I skip step 3) above but is it really supposed to work that way?

I believe so.

Quote from: IDA_ML on December 15, 2017, 12:04:08 PM
Isn't it supposed to log during real-time video recording to understand what is going on?

Not for this test, though I'm not 100% sure.

Quote from: IDA_ML on December 15, 2017, 12:04:08 PM
I also looked at some of the log files that you provided but this is like reading Chinese to me.

If you read through the previous posts you'll see that a1ex runs these logs through a script that creates graphics that clear up what is going on. This is what we found out with Andy600's 50D log files:

Quote from: a1ex on December 11, 2017, 10:58:23 PM
Pictures: plain LiveView (1) vs LiveView with raw enabled using new method (2, 3):

(http://a1ex.magiclantern.fm//bleeding-edge/capture/50D/edmac-stby.png)

(http://a1ex.magiclantern.fm//bleeding-edge/capture/50D/edmac-raw-dfort.png)

(http://a1ex.magiclantern.fm//bleeding-edge/capture/50D/edmac-raw-dfort-long.png)

Notice how channel #2 writes some data into RAM, but not for every single frame. It's transferring every now and then, either 32x15 or 32x225 or 32x240 bytes.

As you can see this is much easier to read.

Next steps to take:

Quote from: a1ex on December 11, 2017, 10:58:23 PM
What exactly it's doing? To find out:
- capture a log using dm-spy-experiments branch, make sure to include EDMAC setup functions (SetEDmac, StartEDmac etc) and see what debug messages are nearby
- dump the data from that memory address and attempt to make some sense out of it

We're missing some stubs for the 7D, 50D and 1100D in the dm-spy-experiments branch. That's what I'm working on. Should have it in a couple of days or so.

By the way, I've been thinking about that old build and will do some more testing on it. It only works in zoom mode but maybe by changing the EDMAC write channel it might work in mv1080 and mv720 modes—and probably stop working in zoom mode? Also, the FPS override might be a key to solving the sync issue.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 15, 2017, 07:18:46 PM
Thanks for this clarification, Dfort.  Complex stuff!  It will be very difficult to handle without A1ex's contribution, I think.  Good that he is willing to help!

By the way, yesterday I talked to a friend of mine who uses his 50D extensively for RAW video work. I can't tell you how happy he is with your latest developments on his favorite camera.  So, what is going on in this thread makes a lot of sense to many people all over the globe, believe me!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 16, 2017, 06:34:40 AM
Quote from: dfort on December 15, 2017, 05:49:38 PM
Also, the FPS override might be a key to solving the sync issue.

This is what I have been wondering as well...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilia3101 on December 16, 2017, 02:40:16 PM
Quote from: dfort on December 14, 2017, 08:53:50 PM
It would be great if you could run those EDMAC module tests and post logs for the 5D2.

Quote from: a1ex on December 11, 2017, 04:22:38 PM
Try logging at 50us (or 100us or more if it freezes the image or has other problems) a few scenarios in LiveView:
- without raw video and without photo raw overlays enabled (e.g. global draw off, no other modules loaded)
- with raw video enabled, old method (run it on top of regular nightly or raw_video_10bit_12bit)
- with raw video enabled, new method (on top of dfort's modified version)

I ran two log tests:
1. no modules loaded (unified)
2. with mlv_lite enabled (unified)

Not sure what "new method" means though. I can do more if needed.

Download: https://drive.google.com/file/d/1xwEypSKD43PJ-8-agUMYBGVk7LxGAkVo/view?usp=sharing
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 16, 2017, 04:59:11 PM
Quote from: Ilia3101 on December 16, 2017, 02:40:16 PM
Not sure what "new method" means though. I can do more if needed.

The "new method" means using the build that I posted on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). Make sure to use one of the "raw_video_10bit_12bit" builds. I'll be posting some dm-spy-experiments builds soon.

Speaking of -- I have found the stubs needed for the dm-spy-experiments tests on the 7D and 50D. The stubs were already there on the 500D and 5D2 and I found the few missing on the 550D. Some of the stubs conflict and must be turned off but I'm not sure how to go about testing which stubs to comment out. For starters I'm using the 500D as an example though the 5D2 has a different set of stubs commented out. Once I run a test on the 7D I'll start posting dm-spy-experiments builds.

Here are the tests we should be concentrating on. Every time I reread one of a1ex's post I seem to find something new, like making sure we get logs using all video modes.

Quote from: a1ex on December 12, 2017, 09:45:35 PM
The keys for solving the sync issue are in that big LVState diagram, the debug logs from LiveView (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194464#msg194464) (dm-spy-experiments branch; for models other than 500D you will have to find some stubs (https://bitbucket.org/hudson/magic-lantern/src/bc46b081b25bd901cd5b604faf0c86774ac6794a/src/dm-spy-extra.c?fileviewer=file-view-default#dm-spy-extra.c-264)) or EDMAC activity logs/graphs (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194545#msg194545) (currently captured for 50D, but incomplete). I hope to be able (but cannot promise) to solve the sync issues after analyzing all these logs.

All the logs have to be captured in 3 LiveView modes (see #1450 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194545#msg194545)) in order to have something to compare. Repeat for all video modes (1080p, 720p, 5x, whatever else your camera has). Repeat for all camera models (not just 50D). The two logging tasks are also for camera models already present on the Experiments page (for double-checking and for having something to compare against).

About the 1100D, it is an EVF_STATE camera while the others use LVState, there seems to have been very little interest in that camera, no reports about how the build posted on the official Experiments page  (https://builds.magiclantern.fm/experiments.html)is working and many of the modules on the Nightly build (https://builds.magiclantern.fm/) aren't compiling yet no 1100D user has brought this up. Finding these stubs is a PITA and if there isn't feedback from 1100D users then it isn't worth the effort. A few more points on the 1100D, the video specs are very unimpressive and the memory is rather small so the chances of getting this camera working in the crop_rec_4k branch seem rather dismal.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 16, 2017, 07:07:56 PM
my camera 50D, i can make test.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 16, 2017, 07:31:51 PM
modules on
mlv_play, mlv_rec, mlv_twk, edmac, ettr.
difference:
edmac000 - movie records 640x480
edmac001 - movie records 1920x1080
start log in lv
canon menu sRAW2
raw video 1568x848 23.976 fps
http://dropmefiles.com/mIFeM files
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 16, 2017, 09:44:09 PM
Quote from: dfort on December 16, 2017, 04:59:11 PM
I'll be posting some dm-spy-experiments builds soon.........  The stubs were already there on the 500D and 5D2  Once I run a test on the 7D I'll start posting dm-spy-experiments builds.
I complied theses  memspy build for my 3.5K/UHD thread , I think they will work , I was using them to find LV buffer info etc....
Nightly.memspy.lv.raw.buffer.extended.resoultion.2017Apr24.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-Nightly.memspy.lv.raw.buffer.extended.resoultion.2017Apr24.5D2212.zip) or magiclantern-v2.3.NEXT.Memspy.2017Apr23.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-v2.3.NEXT.Memspy.2017Apr23.5D2212.zip) (Old Core)
This Stub Test  my be useful I make some time ago stubtest_5D2.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/stubtest_5D2.log) also some misc. dm_Logs I made a while ago for research & development for 3K/UHD 5D2 Raw development and Other Digic IV Cams  (http://www.magiclantern.fm/forum/index.php?topic=19336.msg187766#msg18776)
dm-0000.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0000.log) dm-0001.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0001.log) dm-0002.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0002.log) dm-0003.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0003.log) dm-0004.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0004.log) dm-0005.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0005.log) dm-0006.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0006.log) plus a few misc. --  digic00.LOG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/digic00.LOG) & log001.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/log001.log)
I don't remember the conditions or camera setting for the log test . Of course these are from my 5D2 , it may help .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 16, 2017, 10:29:36 PM
Now that I got dm-spy-experiments pimped out I thought I'd start going over some previous posts.

Quote from: a1ex on December 10, 2017, 09:43:19 PM
Allocate it, use a pre-allocated one, or find some unused*) memory block. Don't try random pointers hoping they will work.

*) If you decide to use this one, double-check it's really unused (test it in any camera mode you may think of - photo, movie, zoom, various FPS, while recording, external monitors, and make sure it's not going to be reused by other Canon code while you are updating it). Tools you may use: CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP, CONFIG_RSCMGR_UNUSED_SPACE_TEST, or write your own RAM analysis tools.

Not sure what to look for (yeah I know, search the forum) but thought this was interesting:

(https://farm5.staticflickr.com/4594/38212182825_06b08a3e86.jpg) (https://flic.kr/p/21dFqEn)

(https://farm5.staticflickr.com/4636/27316660769_bed8b52c4c.jpg) (https://flic.kr/p/HBT2xP)

(https://farm5.staticflickr.com/4645/38212182005_94417acae7.jpg) (https://flic.kr/p/21dFqqe)

I also started looking at some logs and trying to make sense of them. Looks like the new EDmac stubs are working.

grep EDmac DM-0000.LOG
D3CBA> **INT-5Ah*:ff28226c:99:02: WriteEDmacWbIntegCompleteCBR 0
D3D16> **INT-58h*:00000564:99:02: WriteEDmacYuvCompleteCBR
D3DEC> **INT-83h*:00000564:99:02: WriteEDmacQuarkCompleteCBR
D3E26> **INT-8Ah*:00000564:99:02: WriteEDmacVramCompleteCBR_x1
D748A> LiveViewMg:00ca6ebc:00:00: *** SetEDmac(0x9, 0x4056ff9c, 0x1364d4, 0x40000000), from ff27ed4c
D74F7> LiveViewMg:00ca6ebc:00:00: *** ConnectReadEDmac(0x0, 0xf), from ff27ed58
D7552> LiveViewMg:00ca6ebc:00:00: *** RegisterEDmacCompleteCBR(0x9, 0xff27dbc0, 0x0), from ff27ed68
D75A9> LiveViewMg:00ca6ebc:00:00: *** StartEDmac(0x9, 0x1), from ff27ed84
D78FE> LiveViewMg:00ca6ebc:00:00: *** SetEDmac(0x12, 0x5f11d800, 0x136498, 0x20000000), from ff27e4f8
D7961> LiveViewMg:00ca6ebc:00:00: *** StartEDmac(0x12, 0x2), from ff27e504
D7A12> LiveViewMg:00ca6ebc:00:00: *** RegisterEDmacCompleteCBR(0x10, 0xff27dbbc, 0x0), from ff27e598
D7A7D> LiveViewMg:00ca6ebc:00:00: *** SetEDmac(0x10, 0x443c0d24, 0xff56f474, 0x20000000), from ff27e678
D7AD2> LiveViewMg:00ca6ebc:00:00: *** StartEDmac(0x10, 0x1), from ff27e684


Not sure what sort of test to run on the dm-spy-experiments branch but if anyone wants to compile it and try various options the branch I'm working on is here (https://bitbucket.org/daniel_fort/magic-lantern/branch/dm-spy-experiments_7D?dest=dm-spy-experiments#diff).

Back to creating a full set of EDMAC logs:

1080p no raw 50µs (https://pastebin.com/Qhc2Cn6q)
720p no raw 50µs (https://pastebin.com/K6Y4Sb2U)
480p no raw 50µs (https://pastebin.com/2hVwTCWF)
5x zoom no raw 50µs (https://pastebin.com/cittQY8j)

mv1080 mlv_lite 50µs (https://pastebin.com/kQVyx3Sh)
mv720 mlv_lite 50µs (https://pastebin.com/XhSiPmUh)
mv640 mlv_lite 50µs (https://pastebin.com/eBLeSa5y)
zoom mlv_lite 50µs (https://pastebin.com/fA619M3t)

mv1080 mlv_lite allocate-raw-lv-buffer 50µs (https://pastebin.com/gxUjeE2j)
mv720 mlv_lite allocate-raw-lv-buffer 50µs (https://pastebin.com/GJzYbLqQ)
mv640 mlv_lite allocate-raw-lv-buffer 50µs (https://pastebin.com/NmpzdctS)
zoom mlv_lite allocate-raw-lv-buffer 50µs (https://pastebin.com/kJkDdtLn)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 17, 2017, 12:54:06 AM
Quote from: Andy600 on December 11, 2017, 11:40:59 AM
@dariSSight read the last few posts and do not use any 5D2 build that uses EDMAC channels 0x01 or 0x02 for raw recording!!

More research needs to be done.

50D testers - the 50D builds using EDMAC 0x01 should be ok.
So can I use Dfort December 12 build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 17, 2017, 01:09:05 AM
Quote from: dariSSight on December 17, 2017, 12:54:06 AM
So can I use Dfort December 12 build?

According to the code:

edmac-memcpy.c
#elif defined(CONFIG_5D2)
uint32_t raw_write_chan = 0x10; /* 0x01 and 0x02 also work but not safe. Needs fps override as a workaround */


The current 5D2 build should be good to use for testing and logging.

Quote from: IDA_ML on December 12, 2017, 05:23:16 PM
Ha ha, Dfort, this old build of yours keeps coming back to haunt you because this is the best ML build made for the 7D in the history of mankind!

As promised I took another look at this build. I could finally get a good zoom mode recording using mlv_rec but not with raw_rec (before it became mlv_lite) and only if FPS override was turned on. However, it does work with 10/12bit and the video looks good.

Taking a look at the old code for the 7D:

edmac-memcpy.c
#elif defined(CONFIG_7D)
uint32_t edmac_read_chan = 0x0A;  /*Read 0x19 0x0D 0x0B 0x0A(82MB/S)*/
uint32_t edmac_write_chan = 0x06; /* Write 0x5 0x6 0x4 (LV) */
//5 zoom, 6 not - improved performance (no HDMI related tearing)


I tried changing the write channel to 0x10 which is what we're using in these experimental builds in hopes that it will work with mv1080 mode but no luck. How about going the other way and on the raw_video_10bit_12bit_LVState-wip branch change the write channel from 0x10 to 0x6 to see if that gets zoom mode working? Nope, didn't work. So it looks like that old build just happens to be a lucky coincidence that it works as well as it does.
Title: Re: 12/10-bit RAW Video Shutter Speed Issue
Post by: 6D_ML on December 17, 2017, 05:42:31 AM
I'm excited about 10-bit RAW recording option. The footage I was able to record so far looks as good as a 14-bit one.
I noticed that shutter speed can't be adjusted in ML menu, EXPO tab (stuck at 1/25.49), while top panel shows shutter speed changes accordingly. Video exif indicates 1/25.49 speed, when top panel may indicate anything from 1/30 to 1/4000s. magiclantern-raw_video_10bit_12bit.2017Aug21.6D116.zip - Latest Build (2017-08-21 08:24) is used.
When switching to Nightly Build shutter speed adjustment works fine. What debugging procedures are required to help troubleshoot the shutter speed issue in 10/12-bit RAW video mode?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 17, 2017, 09:57:46 AM
Read from #1365 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg191914#msg191914) - the questions are still valid.

If it's a regression (issue not present in regular nightly), try older builds (http://builds.magiclantern.fm/jenkins/view/Experiments/job/raw_video_10bit_12bit/) to narrow down. If that doesn't help, compile from source, running "hg bisect".
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 17, 2017, 10:22:01 AM
Quote from: dfort on December 17, 2017, 01:09:05 AM
As promised I took another look at this build. I could finally get a good zoom mode recording using mlv_rec but not with raw_rec (before it became mlv_lite) and only if FPS override was turned on. However, it does work with 10/12bit and the video looks good.

Do you mean to say that you have both: the normal uncropped as well as 5x-zoom mode in 10/12 bits working now?  If that is the case, then you have acomplished another achievement, Dfort !!!  You have broken your own record for the best ML build for the 7D in the history of mankind!  Congratulations !!!

Lucky coincidence or not, if these two modes work properly now, this makes the 7D an extremely capable RAW-video camera for resolutions up to 2496x1198 WITH synchronous sound and up to 20 and more seconds recording times even at the maximum resolution, 10 bits and 24 fps !!! You can hardly immagine how happy will all 7D owners be hearing this news!  I cannot wait to test.  Could you please post a new build with these features working?

As far as the RAW-rec module is concerned, in my opinion, further work on it makes sense only after it becomes MLV-lite.  So please, don't worry about it for now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 17, 2017, 06:24:07 PM
Quote from: IDA_ML on December 17, 2017, 10:22:01 AM
Do you mean to say that you have both: the normal uncropped as well as 5x-zoom mode in 10/12 bits working now?

No, it means that the old build works well in 5x-zoom mode and the new builds sort of work in the uncropped mv1080 and mv720 modes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 17, 2017, 09:04:11 PM
Quote from: a1ex on December 17, 2017, 09:57:46 AM
Read from #1365 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg191914#msg191914) - the questions are still valid.

If it's a regression (issue not present in regular nightly), try older builds (http://builds.magiclantern.fm/jenkins/view/Experiments/job/raw_video_10bit_12bit/) to narrow down. If that doesn't help, compile from source, running "hg bisect".

Alex, thank you for your reply and providing the link to older builds. Let me answer some questions you asked previously.
Quote from: a1exIs it a a display issue, or does the image brightness stay fixed as well?
Replies based on Build #26 (Aug 21, 2017 10:24:50 AM) for 6D1.1.6 with no settings changed or modules loaded.
Image brightness does not change when adjusting shutter speed. It's actually frozen in ML menu, but  top screen and Canon version of the overlay settings show shutter speed value changes properly.
Shutter speed is set to the following values based on FPS set in Canon video recording settings:
1920/30 FPS → 1/32.43(45) - Fixed
1920/24 FPS → 1/25.52(54) - Fixed
1280/60 FPS → 1/70.03(11) - Fixed

I tried Build #24 (Aug 20, 2017 10:36:50 PM), and it has the same shutter speed adjustment  problem as Build #26 (Aug 21, 2017 10:24:50 AM). There is a difference in shutter speed displayed though:

(https://farm5.staticflickr.com/4725/38404568704_56507e9bb7.jpg)
Shutter_Speed_Disabled_in_Video_Mode_magiclantern-raw_video_10bit_12bit.2017Aug20.6D116

(https://farm5.staticflickr.com/4601/25247639908_d6cc9c33a0.jpg)
Shutter_Speed_Disabled_in_Video_Mode_magiclantern-raw_video_10bit_12bit.2017Aug21.6D116

Build #22 (Jul 3, 2017 12:47:09 AM) does not suffer from this shutter speed issue. So something when wrong with Build #24 for 6D model.

(https://farm5.staticflickr.com/4635/39114677891_46a732060a.jpg)
Shutter_Speed_Selection_OK_in_Video_Mode_magiclantern-raw_video_10bit_12bit.2017Jul03.6D116
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 17, 2017, 10:10:27 PM
Hm... now that's a mystery. The main difference between the two is this backend (https://bitbucket.org/hudson/magic-lantern/pull-requests/672) (for 100D/70D, but also affects 6D) and a few small fixes on the raw side.

Can you run the same tests on the latest build (#31) ? (not really expecting to work, but... who knows; it has nothing fancy, other than syncing with latest stuff from mainline and the above PR)

BTW, I see ISO is set to Auto. Are you in M mode?

edit: I might remember something; please PM me a RAM dump (Debug -> Dump ROM and RAM, from both #22 and #31).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 12georgiadis on December 17, 2017, 10:56:40 PM
Dfort, Andy600, Ilia3101, a1ex, IDA_ML, reddeercity and others, fantastic evolution for the 7D and all DigicIV ! Good resurrection ! That adds a range of new possibilities for casual raw shooting. I hope that all this work on 10/12 would reach the stable graal. We can even dream of a crop_rec_4k on 7D now, with cropmode and H264 proxy features !
I can try to run some test after christmas. Is there any tutorial for EDMAC tools ? And I can test the builds too. Keep it up !
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 17, 2017, 11:19:48 PM
Quote from: a1ex on December 17, 2017, 10:10:27 PM
Can you run the same tests on the latest build (#31) ?
Build# 31 still has the same shutter speed issue. The stuck value is different though:

(https://farm5.staticflickr.com/4589/24256592587_8cfb9c2ced.jpg)

Quote from: a1ex on December 17, 2017, 10:10:27 PM
BTW, I see ISO is set to Auto. Are you in M mode?
Indeed, it was M mode with autoISO, but switching to Av, Tv, P modes does not remedy stuck SS values.

Quote from: a1ex on December 17, 2017, 10:10:27 PM
edit: I might remember something; please PM me a RAM dump (Debug -> Dump ROM and RAM, from both #22 and #31).
PM sent with RAM dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 17, 2017, 11:38:14 PM
Fix pushed (issue confirmed in QEMU, but could not test the actual behavior). Please try build #32.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 18, 2017, 04:40:28 AM
Quote from: a1ex on December 17, 2017, 11:38:14 PM
Fix pushed (issue confirmed in QEMU, but could not test the actual behavior). Please try build #32.
Tried Build #32. The issue persists, but now the shutter speed value is equal to 360° of the selected FPS. Here's a screenshot for 1920/30FPS:

(https://farm5.staticflickr.com/4636/38413016164_b259484458.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 18, 2017, 06:06:25 AM
Both RAM dumps were taken under the same conditions, right? (same shutter speed, same video mode etc)

Found another set of addresses, not an exact match (likely because the RAM dumps were not taken at the same shutter speeds), but let's try that one as well (build #33). If that doesn't work, please send me a ROM dump from that build as well.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 18, 2017, 09:37:30 AM
Quote from: dfort on December 17, 2017, 01:09:05 AM
According to the code:

edmac-memcpy.c
#elif defined(CONFIG_5D2)
uint32_t raw_write_chan = 0x10; /* 0x01 and 0x02 also work but not safe. Needs fps override as a workaround */


The current 5D2 build should be good to use for testing and logging.

As promised I took another look at this build. I could finally get a good zoom mode recording using mlv_rec but not with raw_rec (before it became mlv_lite) and only if FPS override was turned on. However, it does work with 10/12bit and the video looks good.


I don't know if I'm doing anything wrong but I tested Dec 12,2017 build and its nonfunctional, choppy footage files, freeze while in record. Is there any build you have for non crop and 5xzoom I can test that's ready? So far best build for testing non-crop 10bit 12bit is Reddeercity Dec10,2017noncropBuild but you said don't use that build, and Dec1,2016/5xCropBuild(I think that's a Dfort build).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on December 18, 2017, 10:26:41 AM
Quote from: dariSSight on December 18, 2017, 09:37:30 AM
Is there any build you have for non crop and 5xzoom I can test that's ready?

Obviously not.  Please read post 1488 above.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 18, 2017, 02:35:14 PM
Quote from: a1ex on December 18, 2017, 06:06:25 AM
Both RAM dumps were taken under the same conditions, right? (same shutter speed, same video mode etc)

Found another set of addresses, not an exact match (likely because the RAM dumps were not taken at the same shutter speeds), but let's try that one as well (build #33). If that doesn't work, please send me a ROM dump from that build as well.
Build #33 allows for shutter speed changes now! PM with ROM files sent.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 18, 2017, 06:40:24 PM
Looks like the 6D is up and running properly. Thanks @6D_ML for the excellent bug reporting and following a1ex's instructions.

Quote from: dariSSight on December 18, 2017, 09:37:30 AM
I tested Dec 12,2017 build and its nonfunctional, choppy footage files, freeze while in record.

Are you setting FPS override to 24 or 23.976? That seems to be the key on the 5D2.

Quote from: dariSSight on December 18, 2017, 09:37:30 AM
So far best build for testing non-crop 10bit 12bit is Reddeercity Dec10,2017noncropBuild but you said don't use that build...

Well, if it works for you, go for it. I'm trying to figure out how to get all the video modes working like they are on the Digic V cameras and the Digic IV models that use EVF_STATE (60D and 600D). Technically the 1100D falls in that last category but there doesn't seem to be much feedback on that camera.

Top priority is 7D, 5D2 and 50D. Sounds like the 50D is already working with this new code. Where are the 500D and 550D testers?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 19, 2017, 06:26:15 AM
I can confirm -- raw_video_10bit_12bit.2017Dec12.5D2212.zip  build totally fails in all bit depths & crop even 14bit .
As @dariSSight says "choppy footage"
Frame over ride enable -- canon fps 24 , ML fps 23.976 (I always have frame over enable to exact fyi )
this what I have in my original source in download , which worked
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
#endif

the 12/12/2017 build changed to this and doesn't
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x4000)
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 19, 2017, 07:47:23 PM
Actually my December 12 build is using (SRM_BUFFER_SIZE - 0xF000) for the 5D2. You can check the history on Bitbucket using the Blame tool:

(https://farm5.staticflickr.com/4633/38455126424_f99ae5ecbb_z.jpg) (https://flic.kr/p/21A9zk7)

The way I got to that is by feedback from 5D2 users though honestly it would make more sense if we would subtracting the same amount for all cameras. I began by using the method used on the 1100D in the allocate-raw-lv-buffer branch and changed to using SRM_BUFFER_SIZE and subtracting a bit for overhead as per a1ex's suggestion:

Quote from: a1ex on December 05, 2017, 09:11:02 PM
SRM_BUFFER_SIZE would work if you allocate with srm_malloc_suite. With the malloc wrappers, there will be a small overhead from the memory backend, so... just reduce the size by 0x100 bytes or so. Didn't try, but this should be generic for any model that's going to implement full-res LiveView.

Testing on the 7D I determined that we needed to shave off 0x1000 from SRM_BUFFER_SIZE. Please run some tests on the 5D2 and let me know what works best.

Note that I started down this path assuming that 10bit/12bit was working on the 1100D because there was an experimental build posted on the downloads page but it turns out that nobody reported whether or not it was working. This whole allocate-raw-lv-buffer might turn out to be a dead end except that it seems to be working great on the 50D. Go figure.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 19, 2017, 08:19:20 PM
Quote from: dfort on December 18, 2017, 06:40:24 PM
Looks like the 6D is up and running properly. Thanks @6D_ML for the excellent bug reporting and following a1ex's instructions.

I did not expect  A1ex to release those last builds so quickly, and I lagged behind with the testing. Well done A1ex!
So far, great results for 6D video shooters:
Continuous 10-bit RAW recording 1728x724 (1.05x) @ 23.976 fps with sound.
Continuous 10-bit  RAW recording in crop mode 1920x720 (2.85x) @ 23.976 fps with sound.

Live liew during crop mode video recording does not work yet in 10/12-bit mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 20, 2017, 06:07:14 AM
Quote from: dfort on December 19, 2017, 07:47:23 PM
Note that I started down this path assuming that 10bit/12bit was working on the 1100D because there was an experimental build posted on the downloads page but it turns out that nobody reported whether or not it was working.
This whole allocate-raw-lv-buffer might turn out to be a dead end except that it seems to be working great on the 50D. Go figure.
That what I been using in my 3.5k/UHD thread with limited success (3584x1068),
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
It was a1ex that suggested that I fellow the 1100D 10-12bit implementation
but I know I made mistakes  (sometimes the custom buffer will not load more then likely my coding skills or lack there of) I know I don't add the internals.h stuff

1100d_internals.h from10-12bit branch
/** Perfect sync using EVF_STATE **/
#define CONFIG_EVF_STATE_SYNC


Not sure if you are using this code , haven't check yet .
found this post claims to have run the Canon 1100d,  magiclantern-raw_video_10bit_12bit.2017Jan13.1100D105 build but does not say if they really used 10/12bit or just 14bit
http://www.magiclantern.fm/forum/index.php?topic=5601.msg178589#msg178589
so I would have to assume 10-12bit with RAW_LV_BUFFER_ALLOC_SIZE works on 1100d
another post from a 1100d 10-12bit build user -- http://www.magiclantern.fm/forum/index.php?topic=5601.msg178914#msg178914

Notice this allocate-raw-lv-buffer#Lsrc/raw.cT1798 (https://bitbucket.org/hudson/magic-lantern/commits/5647189c555541e63e31f5b67302b60c396e75b1?at=allocate-raw-lv-buffer#Lsrc/raw.cT1798) for the 1100d , I know I haven't include this in my source code yet -- may you have @dfort  found them in this commit here (https://bitbucket.org/hudson/magic-lantern/commits/5647189c555541e63e31f5b67302b60c396e75b1?at=allocate-raw-lv-buffer)
I guess I never fully implemented all the 1100d allocate-raw-lv state code , will update my source and try out

Edit: something interesting in vsync-lite.c (https://bitbucket.org/hudson/magic-lantern/commits/16946357438b2af805172909df2f86666201264d?at=new-lv-buffer-detection#chg-src/vsync-lite.c) at cF4T4 (https://bitbucket.org/hudson/magic-lantern/commits/16946357438b2af805172909df2f86666201264d?at=new-lv-buffer-detection#Lsrc/vsync-lite.cF4T4) from this commit 1694635  (https://bitbucket.org/hudson/magic-lantern/commits/16946357438b2af805172909df2f86666201264d?at=new-lv-buffer-detection)
I find this interesting also raw.c : Allow the backend to allocate a buffer for RAW lv if CONFIG_EDMAC_RAW_SLURP and CONFIG_BROKEN_RAW_LV_BUFFER are defined (https://bitbucket.org/hudson/magic-lantern/commits/6348f69e0f57a90f1d4861c9b22ae570aeb6b241?at=allocate-raw-lv-buffer)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 20, 2017, 11:41:58 AM
Short report:

Tested 550D, following build:
https://bitbucket.org/daniel_fort/magic-lantern/downloads/raw_video_10bit_12bit.2017Dec12.550D109.zip
10bit choppy
12bit choppy
14bit working
Tried set override fps to 24, still choppy

Also tested 10/12/14bit on a 1100D, same choppy sync issues. Also a live view message about Hack error 0, expecting some other stub.
Following build:
https://bitbucket.org/daniel_fort/magic-lantern/downloads/raw_video_10bit_12bit.2017Dec12.1100D105.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 20, 2017, 12:24:45 PM
@Danne: while you have the 1100D, do you mind capturing the full set of VRAM dumps? http://www.magiclantern.fm/forum/index.php?topic=12375.0

(don't really need the external monitor ones; just the ones on internal LCD)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 20, 2017, 01:07:04 PM
I'm on it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 20, 2017, 02:10:00 PM
Some progress. Seems only video mode for 1100D is mv720(1280x720):

https://bitbucket.org/Dannephoto/magic-lantern/downloads/Vram_dumps_1100D.zip

Hopefully useful. I could create as below:

3.dump live view with  5x zoom enabled (standby)
4.dump live view with 10x zoom enabled (standby)
7.dump live view in 720p video mode (standby)
8.dump live view in 720p video mode (recording)
14.enable raw video recording (via ML menu) press dump and quickly press play (to review a mlv video)
15.disable raw video, switch to photo mode (from mode dial or dedicated switch)
16.dump live view in photo mode
17.press dump then quickly press play (to review a photo)
18.For this last one, Image review must be enabled and set to at least 5 sec. (canon menu) after you press "dump VRAM" you must quickly take a picture, and the dump must be captured during image review (automatically review a photo after taken)

I got the camera and could continue tonight or tomorrow if I get specifics.

I am working with this ML build:
https://builds.magiclantern.fm/jenkins/job/raw_video_10bit_12bit/33/artifact/platform/1100D.105/magiclantern-raw_video_10bit_12bit.2017Dec18.1100D105.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 20, 2017, 02:26:23 PM
Thanks - definitely useful (as this camera has some unusual aspect ratios in its YUV buffers). There are a few 0-byte DNGs (guess: out of memory?), but I think I can fill in the gaps with what I already have.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 20, 2017, 02:28:04 PM
Yes, noticed the 0 bytes. Occured when movie recording on. Maybe other occasions too. Glad to help.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 20, 2017, 03:46:16 PM
Quote from: reddeercity on December 20, 2017, 06:07:14 AM
Edit: something interesting in vsync-lite.c (https://bitbucket.org/hudson/magic-lantern/commits/16946357438b2af805172909df2f86666201264d?at=new-lv-buffer-detection#chg-src/vsync-lite.c) at cF4T4 (https://bitbucket.org/hudson/magic-lantern/commits/16946357438b2af805172909df2f86666201264d?at=new-lv-buffer-detection#Lsrc/vsync-lite.cF4T4) from this commit 1694635  (https://bitbucket.org/hudson/magic-lantern/commits/16946357438b2af805172909df2f86666201264d?at=new-lv-buffer-detection)
I find this interesting also raw.c : Allow the backend to allocate a buffer for RAW lv if CONFIG_EDMAC_RAW_SLURP and CONFIG_BROKEN_RAW_LV_BUFFER are defined (https://bitbucket.org/hudson/magic-lantern/commits/6348f69e0f57a90f1d4861c9b22ae570aeb6b241?at=allocate-raw-lv-buffer)

new-lv-buffer-detection (https://bitbucket.org/hudson/magic-lantern/branch/new-lv-buffer-detection#diff) - Yet another old experimental branch for the 1100D. There might be something there that we could try out, like we did with allocate-raw-lv-buffer (https://bitbucket.org/hudson/magic-lantern/branch/allocate-raw-lv-buffer#diff). Thanks for pointing it out.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rob_6 on December 20, 2017, 07:57:26 PM
@dfort,

I see that you have already done some testing with your 7D and the build that you are working on. Is there any additional testing you might need? I am happy to help if needed. :)

Thanks!

Rob
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: djkraq on December 20, 2017, 08:21:23 PM
I can't seem to find the build for the 70D.  Is it still there on the experimental page?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 21, 2017, 03:45:03 AM
Quote from: rob_6 on December 20, 2017, 07:57:26 PM
Is there any additional testing you might need?

Sure -- if you find a problem, tell us the issue, the steps to reproduce it and share any workaround you may discover.

Quote from: djkraq on December 20, 2017, 08:21:23 PM
I can't seem to find the build for the 70D.  Is it still there on the experimental page?

There is a Nightly build (https://builds.magiclantern.fm/) but if you want a 10bit/12bit experimental build you'll have to roll your own or bug @nikfreak for one.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Stathman on December 22, 2017, 05:24:45 PM
Quote from: dfort on December 21, 2017, 03:45:03 AM
There is a Nightly build (https://builds.magiclantern.fm/) but if you want a 10bit/12bit experimental build you'll have to roll your own or bug @nikfreak for one.

There was a 10bit/12bit experimental build for 70D.
Do you know why it has been removed?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2017, 05:58:21 PM
Don't know anything about a 70D build for 10bit/12bit - that camera isn't in the raw_video_10bit_12bit branch though it is in the crop_rec_4k branch. There isn't an official experimental build though @esas has reported on the Canon 70D (http://www.magiclantern.fm/forum/index.php?topic=14309.msg195013#msg195013) topic that he has a build that seems to work with 10bit/12bit.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Stathman on December 22, 2017, 06:57:19 PM
Quote from: dfort on December 22, 2017, 05:58:21 PM
Don't know anything about a 70D build for 10bit/12bit - that camera isn't in the raw_video_10bit_12bit branch though it is in the crop_rec_4k branch. There isn't an official experimental build though @esas has reported on the Canon 70D (http://www.magiclantern.fm/forum/index.php?topic=14309.msg195013#msg195013) topic that he has a build that seems to work with 10bit/12bit.

Some days ago there was a build for 70D 10bit/12bit in the experiments section.
I have the file on my computer and this is the one i use on my 70D.
In fact, this is the reason i bought it in the first place!
Although, i will try the new build asap!
Thanks  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2017, 07:00:07 PM
@Stathman - could you open the autoexec.bin file in a text editor and post the message at the beginning of the file? It should show where that build came from.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Stathman on December 22, 2017, 07:02:29 PM
Sure!

Magic Lantern raw_video_10bit_12bit.2017Aug21.70D112
Camera   : 70D
Firmware : 112
Changeset: 870cedc16e2d+87bd456b1128+ (raw_video_10bit_12bit) tip
Built on : 2017-08-21 08:31:05 by jenkins@nightly
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2017, 07:21:59 PM
Interesting. That looks like a merge of two branches. The Wayback Machine captured the Experiments download page (https://web.archive.org/web/20170820210345/https://builds.magiclantern.fm/experiments.html) the day before the build you've got and sure enough there was a 70D build there. I can't answer your question over why it went away.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 22, 2017, 07:25:02 PM
Quote from: dfort on December 22, 2017, 05:58:21 PM
Don't know anything about a 70D build for 10bit/12bit - that camera isn't in the raw_video_10bit_12bit branch though it is in the crop_rec_4k branch. There isn't an official experimental build though @esas has reported on the Canon 70D (http://www.magiclantern.fm/forum/index.php?topic=14309.msg195013#msg195013) topic that he has a build that seems to work with 10bit/12bit.
Is there a update test build for 5D2, that we can test over the holiday. I apologize for my lack of coding ability, so if you can direct a non-coder in the direction of learning how to build test firmware for 5D2? I'm very willing to learn and i have the next 11 day off work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 22, 2017, 09:57:59 PM
Here is my take on comparing 10bit vs 14bit on 6D*1.1.6 Build 33.

mlv_rec module for recording RAW video (1.00x crop). →  MLV files converted into cinemaDNG (lossless) with MLV App v0.12 alpha. → DNGs to JPGs in LR 6.5. JPG conversion settings should be embedded in DNGs.

Expo settings: 1/30s, iso100, 30fps.
Image Fine-tuning:
ML Digital ISO: 0.0 EV
Black Level: 0
Shutter fine-tuning: +39 units

WB adjusted in ML EXPO tab (Auto adjust Kelvin + G/M) for 14-bit recording and remained fixed for 10bit mode. 
Click on thumb for full size JPG.

(https://farm5.staticflickr.com/4642/27441483719_5cae2522a4.jpg) (https://c1.staticflickr.com/5/4642/27441483719_a2fe5eec5e_o.jpg)
14bit / 30fps
Download DNG (https://drive.google.com/file/d/1JBd0B8n2QC-RHZ6hJaJiKvnxci4z12Ax/view?usp=sharing)

(https://farm5.staticflickr.com/4687/27441483439_b9d8ac641f.jpg) (https://c1.staticflickr.com/5/4687/27441483439_2ee27c4712_o.jpg)
10bit / 30fps
Download DNG (https://drive.google.com/open?id=1R_mGzLH2k-4JgNHhEx-AqQl2dCopJTD-)

Is there a flaw in the above workflow or a 10-bit raw footage is not very forgiving for raising blacks and shadows compared to a 14-bit file for such a scene?
I've also shot less contrasty scenes that didn't require much shadows/blacks pushing in post, and 10/14bit footage looks indistinguishable, unless you start pixel peeping at 200%+ zoom level.

Also, I noticed that in Build 33, when I set in Pref tab, "ISO/Kelvin" - ON and "Use Set button" - ON, they do not appear in LV before or during recording. Pressing UP/DOWN/SET buttons does not change anything.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 22, 2017, 10:03:14 PM
@6D_ML which build are you using and is there a 5D2 build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 23, 2017, 03:40:52 AM
Quote from: dariSSight on December 22, 2017, 10:03:14 PM
@6D_ML which build are you using and is there a 5D2 build?
Build 33: http://builds.magiclantern.fm/jenkins/view/Experiments/job/raw_video_10bit_12bit/33/ (http://builds.magiclantern.fm/jenkins/view/Experiments/job/raw_video_10bit_12bit/33/). It looks like 5D2 is not supported yet.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2017, 04:05:17 AM
Quote from: dariSSight on December 22, 2017, 07:25:02 PM
...if you can direct a non-coder in the direction of learning how to build test firmware for 5D2?

You don't need to know how to code in order to compile ML from source. I wrote a couple of tutorials for the methods that I use most.

Mac tutorial (http://www.magiclantern.fm/forum/index.php?topic=16012.0)
Windows, Cygwin tutorial (http://www.magiclantern.fm/forum/index.php?topic=15894.0)

If you are running Windows 10 there is a way to run Ubuntu on it. Here's a discussion on that topic. (http://www.magiclantern.fm/forum/index.php?topic=20214.0) Cool thing about this method is that you can also run QEMU--with Cygwin you can't.

Since Danne got involved testing the 1100D I went ahead and looked up those stubs needed in the dm-spy-experiments branch (https://bitbucket.org/daniel_fort/magic-lantern/branch/dm-spy-experiments_7D). Still haven't been able to figure out a good test to run with that branch but it probably involves merging it into the experimental 10bit/12bit branch I've been using and creating some in camera logs.

@dariSSight - I'll post a 5D2 build that does this:

#ifdef CONFIG_5D2
-#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif


That will be the Dec.22 build. What this does is to shrink the custom buffer we're using to the same amount that is working on the other cameras. Check it against the previous build, which I'll leave up for a while, and see if there is any difference between the two. Look for these unofficial test builds on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 23, 2017, 08:35:49 AM
Quote from: dfort on December 23, 2017, 04:05:17 AM
You don't need to know how to code in order to compile ML from source. I wrote a couple of tutorials for the methods that I use most.

Mac tutorial (http://www.magiclantern.fm/forum/index.php?topic=16012.0)
Windows, Cygwin tutorial (http://www.magiclantern.fm/forum/index.php?topic=15894.0)

If you are running Windows 10 there is a way to run Ubuntu on it. Here's a discussion on that topic. (http://www.magiclantern.fm/forum/index.php?topic=20214.0) Cool thing about this method is that you can also run QEMU--with Cygwin you can't.
@dariSSight - I'll post a 5D2 build that does this:

#ifdef CONFIG_5D2
-#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif


That will be the Dec.22 build. What this does is to shrink the custom buffer we're using to the same amount that is working on the other cameras. Check it against the previous build, which I'll leave up for a while, and see if there is any difference between the two. Look for these unofficial test builds on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).

@dfort Thanks for the links to Tutorials. I'm a Mac user, I'm up at 3am to jump right in, and I'll do a clean install of Dec 12 build because it gave me Choppy jumpy video when I use it last, but maybe I did something wrong but I'll test again along with Dec 22 build.Thanks Again
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2017, 03:19:50 PM
I'm amazed by the quality difference between 14 and 10 bit

But explainable, downloaded the original dng's and checked them.
In the 14bit file you can see the wall(or shadows) are exposed at the left border/lower limit of the histogram, BUT within limits off the histogram, no cutoff in histogram.
When you do an exposure correction of 3 stops on the 14 bit file, the left side of the histogram starts low and raises, no cut off in detail.
Bottom line, you nailed the exposure for lifting the shadows.

The 10 bit file has the same exposure, but the lower 4 bits of data are cut off.
If you raise it with an exposure of 3 stops, you can see the histogram starts high, so the data in the shadows is cut off.
You nailed the exposure time for 14 bit, but for 10 bit this means detail loss in the shadows.

Makes all sense, but didn't think about it too much before, the quality difference is a lot. 
I'm using the raw histogram data to expose correctly, but does the raw histogram take in account if you're recording 14/12 or 10 bit ?
Cause it seems if you're planning on lifting the shadows, and if your shadows are at the all left end of the histogram, you're better of with 14 bit.







Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2017, 04:49:34 PM
Of course you should also compare 10bit raw against 8bit H.264.

The sweet spot seems to be 12bit. However, the big improvements come with lossless compression. I think that's what made the Black Magic Pocket Cinema Camera such a big hit, it was the 12bit lossless raw recording feature. Well, that and the log 10bit ProRes but we probably can't do that.

Quote from: Levas on December 23, 2017, 03:19:50 PM
...does the raw histogram take in account if you're recording 14/12 or 10 bit ?

Good question.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 23, 2017, 04:57:06 PM
The histogram is always computed from 14-bit data, but only the top 12 stops are displayed (count them). Maybe something to consider in the next histogram iteration (http://www.magiclantern.fm/forum/index.php?topic=20882.0).

Long answer (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177531;topicseen#msg177531) (computing the histogram from 12-bit or 10-bit data, i.e. while recording at a lower bit depth, is not trivial, although not very difficult either).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 23, 2017, 05:03:03 PM
wait new test build for 50D  :) i see for 5D2 something change ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2017, 05:36:55 PM
@CITY-U1001 - The 5D2 test doesn't apply to the 50D. See Reply #1524 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg195042#msg195042)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 25, 2017, 01:59:42 AM
Thanks and Happy Holidays to Magic Lantern and a special THANKS and HAPPY HOLIDAYS to dfort for helping me out on setting up for builds, I have a build on my desktop now and I guess I'm read to pitch in.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 27, 2017, 04:17:58 AM
Quote from: dfort on December 23, 2017, 04:49:34 PM
Of course you should also compare 10bit raw against 8bit H.264.

I did. In case of 6D, while detail and dynamic range are noticeably higher in 10 bit RAW, 8bit H.264 file has an edge with a relatively "consistent and accurate" color, whereas 12/10bit file gets polluted with a green & yellow cast in blacks/shadows. This temperature/tint difference is also noticeable between 14 bit (best IQ results) and 12/10 bit files. Has anyone noticed similar temp/tint shifts with other models?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 27, 2017, 07:27:29 AM
Is the color shift visible with mlv_dump?

If using other converter, is it following this recommendation (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178627#msg178627)? (if not, you know where you have to report the issue)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on December 28, 2017, 04:40:48 AM
dforts 550D build works flawlessly for me, in both 10 and 12 bit. The 50D build however goes a bit wonky for me. About every 30ish frames I'll get one frame where the bottom 3rd of the frame offsets to the left slightly. This happens in both 10 bit and 12 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 28, 2017, 05:51:40 AM
Quote from: a1ex on December 27, 2017, 07:27:29 AM
Is the color shift visible with mlv_dump?
With mlv_dump (option 15), 10bit DNGs are identical to those produced by MLV App (color shift visible)

Quote from: a1ex on December 27, 2017, 07:27:29 AM
If using other converter, is it following this recommendation (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178627#msg178627)? (if not, you know where you have to report the issue)
I couldn't check black level with mlv_dump since "-v" is not listed as an option, and it doesn't do any operation. What did the trick to remedy the color shift in 10 bit files is the option 4 (converting image data to 14 bit depth per channel).

(https://farm5.staticflickr.com/4733/38466850125_2408979228.jpg) (https://c1.staticflickr.com/5/4733/38466850125_e6632967f9_o.jpg)
Download 10bit DNG interpreted as 14bit (https://drive.google.com/open?id=1pOQJJwg83vHSBN28E1Jnx-xtpDT8L57s)


(https://farm5.staticflickr.com/4687/27441483439_b9d8ac641f.jpg) (https://c1.staticflickr.com/5/4687/27441483439_2ee27c4712_o.jpg)
Download 10bit DNG (https://drive.google.com/open?id=1R_mGzLH2k-4JgNHhEx-AqQl2dCopJTD-)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Sganzerla on December 28, 2017, 05:58:01 AM
Quote from: 6D_ML on December 27, 2017, 04:17:58 AM
I did. In case of 6D, while detail and dynamic range are noticeably higher in 10 bit RAW, 8bit H.264 file has an edge with a relatively "consistent and accurate" color, whereas 12/10bit file gets polluted with a green & yellow cast in blacks/shadows. This temperature/tint difference is also noticeable between 14 bit (best IQ results) and 12/10 bit files. Has anyone noticed similar temp/tint shifts with other models?

With 5D MKIII there is a magenta color shift in the blacks (with mlv_dump), the only reason I quit using 12/10bit lossless.
Edit: Didn't know about this 14bit conversion solution for the color shift. Will try soon my own tests...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 28, 2017, 07:24:37 AM
Not sure what "option 15" is, and the EXIF info in the DNG says Adobe Photoshop Lightroom, so I couldn't tell whether you were using the right version - try the command-line mlv_dump from the Experiments (http://builds.magiclantern.fm/experiments.html) page.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 28, 2017, 08:52:32 AM
Quote from: kyrobb on December 28, 2017, 04:40:48 AM
dforts 550D build works flawlessly for me, in both 10 and 12 bit. The 50D build however goes a bit wonky for me.

Glad to hear the 550D is working but those issues on the 50D is news to me.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 28, 2017, 10:33:44 AM
What build is working with the 550D? Last I tested I had timing?(wacky recordings) issues.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 28, 2017, 01:24:40 PM
Quote from: kyrobb on December 28, 2017, 04:40:48 AM
The 50D build however goes a bit wonky for me. About every 30ish frames I'll get one frame where the bottom 3rd of the frame offsets to the left slightly. This happens in both 10 bit and 12 bit.

What build are you using and which app for MLV>DNG?

Can't reproduce it here.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 28, 2017, 02:28:04 PM
Quote from: a1ex on December 28, 2017, 07:24:37 AM
Not sure what "option 15" is, and the EXIF info in the DNG says Adobe Photoshop Lightroom, so I couldn't tell whether you were using the right version - try the command-line mlv_dump from the Experiments (http://builds.magiclantern.fm/experiments.html) page.

I used batch_mlv_in-out (http://www.magiclantern.fm/forum/index.php?topic=10526.msg193871#msg193871) to output 10bit DNG above. I need to read a manual for command-line mlv_dump before I make it output DNGs ::)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 28, 2017, 03:10:59 PM
Okay, so the linked mlv_dump is following my advice when used with -b 14, but not by default (it has the defaults slightly changed - our version upconverts to 14-bit by default).

To avoid color shift in shadows, the DNG bit depth needs to be at least 1 bit above the recorded bit depth (otherwise, the 0.5 LSB adjustment won't be applied).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 28, 2017, 04:29:37 PM
Quote from: 6D_ML on December 28, 2017, 02:28:04 PM
I need to read a manual for command-line mlv_dump before I make it output DNGs ::)

Just run mlv_dump without any input/output or options. The crop_rec_4k version it will print out this:


MLV Dumper
-----------------

[ERROR] Error: Missing input filename
Usage: mlv_dump [options] <inputfile>
Parameters:
  -o output_file      write video data into a MLV file
  -v                  verbose output
  --version           print version information
  --batch             format output message suitable for batch processing
  --relaxed           do not exit on every error, skip blocks that are erroneous

-- DNG output --
  --dng               output frames into separate .dng files. set prefix with -o
  --no-cs             no chroma smoothing (default)
  --cs2x2             2x2 chroma smoothing
  --cs3x3             3x3 chroma smoothing
  --cs5x5             5x5 chroma smoothing
  --no-fixcp          do not fix cold pixels
  --fixcp2            fix non-static (moving) cold pixels (slow)
  --no-stripes        do not fix vertical stripes in highlights

-- RAW output --
  -r                  output into a legacy raw file for e.g. raw2dng

-- MLV output --
  -b bits             convert image data to given bit depth per channel (1-16)
  -z bits             zero the lowest bits, so we have only specified number of bits containing data (1-16) (improves compression rate)
  -f frames           frames to save. e.g. '12' saves frames 0 to 12, '12-40' saves frames 12 to 40.
  -A fpsx1000         Alter the video file's FPS metadata
  -x                  build xref file (indexing)

-- MLV autopsy --
  --skip-block <block#>        skip given block number, as if it wasn't present
  --skip-type <type>           skip given block type (e.g. VIDF, AUDF, etc), as if they weren't present
  --extract <block#>           extract the block at given position into autopsy file
  --extract-type <type>        extract the block type (e.g. VERS, LENS, etc) into autopsy file
  --replace <block#>           replace block with data from given autopsy file; requires --autopsy-file
  --payload-only               extract/replace affect not the whole block, but only payload
  --header-only                extract/replace affect not the whole block, but only header
  --autopsy-file <file>        extract/replace from this file
  --hex                        extract prints the selected data as hexdump on screen
  --ascii                      extract prints the selected data as ASCII on screen (only suitable for VERS and DEBG)
  --visualize                  visualize block types, most likely you want to use --skip-xref along with it

-- MLV manipulation --
  --skip-xref                  skip loading .IDX (XREF) file, read block in the MLV file's order instead of presorted
  -m                           write only metadata, no audio or video frames
  -n                           write no metadata, only audio and video frames
  -I <mlv_file>                inject data from given MLV file right after MLVI header
  -X type                      extract only block type int output file

-- Image manipulation --
  -a                  average all frames in <inputfile> and output a single-frame MLV from it
  --avg-vertical      [DARKFRAME ONLY] average the resulting frame in vertical direction, so we will extract vertical banding
   --avg-horizontal   [DARKFRAME ONLY] average the resulting frame in horizontal direction, so we will extract horizontal banding
  -s mlv_file         subtract the reference frame in given file from every single frame during processing
  -t mlv_file         use the reference frame in given file as flat field (gain correction)

-- Processing --
  -e                  delta-encode frames to improve compression, but lose random access capabilities
  -c                  compress video and audio frames using LJ92. if already compressed, then decompress and recompress again.
                      specify twice to pass through unmodified compressed (lossless) data to DNG which speeds up writing, but skips preprocessing
  -d                  decompress compressed video and audio frames using LZMA or LJ92

-- bugfixes --
  --black-fix=value   set black level to <value> (fix green/magenta cast). if no value given, it will be set to 2048.
  --white-fix=value   set white level to <value>. if no value given, it will be set to 15000.
  --fix-bug=id        fix some special bugs. *only* to be used if given instruction by developers.


So if I'm understanding what is going on--the version of mlv_dump you are using needs this:

mlv_dump --dng -b 14 <inputfile>

If you use the "official" crop_rec_4k mlv_dump you can skip the "-b 14" option because it will save to 14 bit by default:

mlv_dump --dng <inputfile>

To understand why the 0.5 LSB adjustment won't be applied, read Reply #1004 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177802#msg177802).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on December 28, 2017, 06:03:32 PM
I see. I'm using MLV app and converting to ProRes. The issue is present both in the app and after export. Maybe an MLV app issue? I'll have to check with mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 29, 2017, 04:54:46 AM
Quote from: a1ex on December 27, 2017, 07:27:29 AM
Is the color shift visible with mlv_dump?

Managed to extract a DNG (download link (https://drive.google.com/open?id=1HWzB_U-9uPLBf9Yq4vDodCogPxxt__sn)) with mlv_dump (https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/63/artifact/modules/mlv_rec/mlv_dump.exe) default conversion (mlv_dump --dng myfile.mlv). The color shift is there, and I'd rate its severity between results produced by "batch_mlv_in-out" (best) and "MVL App" (worst). There is also the highest level of blue hue present in clipped blacks with mlv_dump. Hopefully, devs can tackle the color shift issue with 10bit files. I can help so far with feedback only.

Quote from: dfort on December 28, 2017, 04:29:37 PM
If you use the "official" crop_rec_4k mlv_dump you can skip the "-b 14" option because it will save to 14 bit by default:
mlv_dump --dng <inputfile>

dfort, thanks for the tip. In my case I had to use the following cmd syntax to extract DNGs:
root_path_to\mlv_dump --dng root_path_to\MLVfile.mlv
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 29, 2017, 09:54:47 AM
This DNG renders identically to the first one from #1536 (the only difference between the two being some hot pixels and vertical stripes, only noticeable if you subtract the raw data in the two images). Overall, the median difference between the two is 0.


dcraw -a -W -b 16 M21-2259_*.dng


Results: new (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/M21-2259_MLV_Dump_000203.png) vs old (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/M21-2259_10bit-to-14bit_000203.png).

No idea what further improvements you are expecting - I am unable to see any difference between the two files after rendering.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 6D_ML on December 29, 2017, 02:38:46 PM
Thanks for your reply A1ex. Lightroom renders files the way I described above. RawTherapee produces results very similar to yours with no difference.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 03, 2018, 04:56:20 PM
Dfort,

Many people who want to experiment with their 7Ds keep asking me where they can download the latest stable 7D build with 10/12-bit RAW recording.  I direct them to your download area but all they find there is the Dec. 12-th build which does not work well with the 7D, (earthquake motion with normal uncropped RAW video).  The Dec. 11-th build was much better and more stable.  I was wondering if you might be able to upload your Dec. 11-th build with a RAW-tweak module for the 7D again.

Thank you.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 03, 2018, 09:33:48 PM
The code for the 7D wasn't touched between the Dec-11 and Dec-12 builds.

Check it out:

https://bitbucket.org/daniel_fort/magic-lantern/branch/raw_video_10bit_12bit_LVState-wip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 03, 2018, 10:12:31 PM
Well, there must be some difference, otherwise, why would you post two absolutely identical builds on two subsequent days?  I tested both builds extensively and provided detailed feedback (see posts 1390 and 1454 for the Dec. 5 and  Dec. 11 builds, respectively and compare them with post 1465 describing the behavior of the Dec. 12-th build).   If you do not trust the feedback information that I provided, you could test these builds by yourself with your own 7D.  There must be some explanation why the two builds behave so differently.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 04, 2018, 05:04:13 AM
Quote from: IDA_ML on January 03, 2018, 10:12:31 PM
Well, there must be some difference, otherwise, why would you post two absolutely identical builds on two subsequent days?  I tested both builds extensively and provided detailed feedback (see posts 1390 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194278#msg194278) and 1454 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194563#msg194563) for the Dec. 5 and  Dec. 11 builds, respectively and compare them with post 1465 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg194615#msg194615) describing the behavior of the Dec. 12-th build).   If you do not trust the feedback information that I provided, you could test these builds by yourself with your own 7D.  There must be some explanation why the two builds behave so differently.

I added links on your references to previous posts because it took a while to look them up.

If you look at the autoexec.bin files in a text editor you will see that the builds aren't exactly identical but the changes that were made between Dec. 11 and Dec. 12 shouldn't have changed anything on the 7D. I did do some tests on the two builds and didn't see any difference but as long as I was updating some of the other builds I updated all of them. I haven't found a way to sort the downloads page in Bitbucket (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) and wanted to make sure all the builds were next to each other.

One thing that might be causing some issues is that I merged in the edmac module on Dec. 11. I posted a regression test before the merge (Dec. 10) and after which should be that Dec. 11 build that is working for you. Note that sometimes I'll post more than one build in a day so if you could, please open the autoexec.bin file on that build that is working so well for you and post it here so we can track down the changeset.

src/edmac-memcpy.c
    resLock = CreateResLockEntry(resIds, 4);
   
    ASSERT(resLock);

    /* just to make sure we have this stub */
    static int AbortEDmac_check __attribute__((used)) = &AbortEDmac;
}

INIT_FUNC("edmac_memcpy", edmac_memcpy_init);

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 04, 2018, 06:17:08 AM
I used a build from december 6th that seemed to work(it´s named dec 5th, raw_video_10bit_12bit.2017Dec05.7D203.zip):
http://www.magiclantern.fm/forum/index.php?topic=19336.msg194188#msg194188

I think this is the version still on my computer, opened up the autoexe.bin:
Magic Lantern Nightly.2017Dec05.7D203
Camera   : 7D
Firmware : 203
Changeset: 870cedc16e2d+8f8e391db888+ (raw_video_10bit_12bit_allocate-raw-lv-buffer)
Built on : 2017-12-06 04:13:07 by rosiefort@RosieFoComputer

diff -r 870cedc16e2d src/edmac-memcpy.c
--- a/src/edmac-memcpy.c Thu Aug 17 12:07:00 2017 +0300
+++ b/src/edmac-memcpy.c Tue Dec 05 20:13:07 2017 -0800
@@ -341,6 +341,8 @@
uint32_t raw_write_chan = 0x4;  /* 0x12 gives corrupted frames on 1.2.3, http://www.magiclantern.fm/forum/index.php?topic=10443 */
#elif defined(EVF_STATE)
uint32_t raw_write_chan = 0x12; /* 60D and newer, including all DIGIC V */
+#elif defined(LV_STATE)
+uint32_t raw_write_chan = 0x10; /* older DIGIC IV LVState cameras */
#endif

static void edmac_slurp_complete_cbr (void* ctx)
diff -r 870cedc16e2d src/raw.c
--- a/src/raw.c Thu Aug 17 12:07:00 2017 +0300
+++ b/src/raw.c Tue Dec 05 20:13:07 2017 -0800
@@ -133,10 +133,45 @@
#define DEFAULT_RAW_BUFFER MEM(0x76d6c + 0x2C)
#endif

-#else
+#ifdef CONFIG_1100D
+#define RAW_LV_BUFFER_ALLOC_SIZE (3906*968)
+#endif
+
+#ifdef CONFIG_7D
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
+#endif
+
+#ifdef CONFIG_5D2
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
+#endif
+
+#ifdef CONFIG_50D
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
+#endif
+
+#ifdef CONFIG_550D
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
+#endif

-/* with Canon lv_save_raw, just read it from EDMAC */
-#define DEFAULT_RAW_BUFFER shamem_read(RAW_LV_EDMAC)
+#else // "Traditional" RAW LV buffer detection
+
+/**
+ * LiveView raw buffer address
+ * To find it, call("lv_save_raw") and look for an EDMAC channel that becomes active (Debug menu)
+ **/
+
+#if defined(CONFIG_5D2) || defined(CONFIG_50D)
+#define RAW_LV_EDMAC 0xC0F04508
+#endif
+
+#if defined(CONFIG_500D) || defined(CONFIG_550D) || defined(CONFIG_7D)
+#define RAW_LV_EDMAC 0xC0F26008
+#endif
+
+#if defined(CONFIG_DIGIC_V) || defined(CONFIG_600D) || defined(CONFIG_60D)
+/* probably all new cameras use this address */
+#define RAW_LV_EDMAC 0xC0F26208
+#endif

#endif

@@ -399,6 +434,23 @@
extern void reverse_bytes_order(char* buf, int count);

#ifdef CONFIG_RAW_LIVEVIEW
+
+#if defined(CONFIG_ALLOCATE_RAW_LV_BUFFER)
+static void* raw_allocated_lv_buffer = 0;
+#endif
+
+static void* raw_get_default_lv_buffer()
+{
+#if !defined(CONFIG_EDMAC_RAW_SLURP)
+    return (void*) shamem_read(RAW_LV_EDMAC);
+#elif defined(DEFAULT_RAW_BUFFER)
+    return (void*) DEFAULT_RAW_BUFFER;
+#elif defined(CONFIG_ALLOCATE_RAW_LV_BUFFER)
+    return raw_allocated_lv_buffer;
+#else
+    #error DEFAULT_RAW_BUFFER not configured.
+#endif
+}
/* returns 1 on success */
static int raw_lv_get_resolution(int* width, int* height)
{
@@ -521,7 +573,7 @@
         }
         #endif

-        raw_info.buffer = (void*) DEFAULT_RAW_BUFFER;
+        raw_info.buffer = raw_get_default_lv_buffer();
         
         if (!raw_info.buffer)
         {
@@ -1581,7 +1633,7 @@
void FAST raw_lv_vsync()
{
     /* where should we save the raw data? */
-    void* buf = redirected_raw_buffer ? redirected_raw_buffer : (void*) DEFAULT_RAW_BUFFER;
+    void* buf = redirected_raw_buffer ? redirected_raw_buffer : raw_get_default_lv_buffer();
     
     if (buf && lv_raw_enabled)
     {
@@ -1851,6 +1903,14 @@
#ifndef CONFIG_EDMAC_RAW_SLURP
     call("lv_save_raw", 1);
#endif
+
+#ifdef CONFIG_ALLOCATE_RAW_LV_BUFFER
+    uint32_t old = cli();
+    if(!raw_allocated_lv_buffer) {
+        raw_allocated_lv_buffer = fio_malloc(RAW_LV_BUFFER_ALLOC_SIZE);
+    }
+    sei(old);
+#endif
}

static void raw_lv_disable()
@@ -1860,6 +1920,15 @@
#ifndef CONFIG_EDMAC_RAW_SLURP
     call("lv_save_raw", 0);
#endif
+
+#ifdef CONFIG_ALLOCATE_RAW_LV_BUFFER
+    uint32_t old = cli();
+    if(raw_allocated_lv_buffer) {
+        free(raw_allocated_lv_buffer);
+        raw_allocated_lv_buffer = 0;
+    }
+    sei(old);
+#endif
}

int raw_lv_is_enabled()
diff -r 870cedc16e2d platform/7D.203/consts.h
--- a/platform/7D.203/consts.h Thu Aug 17 12:07:00 2017 +0300
+++ b/platform/7D.203/consts.h Tue Dec 05 20:13:07 2017 -0800
@@ -262,6 +262,7 @@
// see "Malloc Information"
#define MALLOC_STRUCT 0x249e4
#define MALLOC_FREE_MEMORY (MEM(MALLOC_STRUCT + 8) - MEM(MALLOC_STRUCT + 0x1C)) // "Total Size" - "Allocated Size"
+#define SRM_BUFFER_SIZE 0x1F80000   /* print it from srm_malloc_cbr */

//~ max volume supported for beeps
#define ASIF_MAX_VOL 5
diff -r 870cedc16e2d platform/7D.203/internals.h
--- a/platform/7D.203/internals.h Thu Aug 17 12:07:00 2017 +0300
+++ b/platform/7D.203/internals.h Tue Dec 05 20:13:07 2017 -0800
@@ -129,3 +129,7 @@

/** Use joystick for one-finger menu navigation */
#define CONFIG_JOY_CENTER_ACTIONS
+
+/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
+#define CONFIG_EDMAC_RAW_SLURP
+#define CONFIG_ALLOCATE_RAW_LV_BUFFER


How do one search for a changeset in source tree?
Changeset: 870cedc16e2d+8f8e391db888+

All I can find is your link to an early build in this post but it´s been removed:
http://www.magiclantern.fm/forum/index.php?topic=19336.msg194178#msg194178
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 04, 2018, 06:36:46 AM
Interesting, that was from a different branch. The first thing I tried was to see if the allocate raw lv buffer experiment that @nanomad did on the 1100D could be ported over to other Digic 4 cameras. I jumped to another branch because the point of all this is to get these cameras integrated into the experimental branches that are built on the Jenkins (https://jenkins-ci.org/) server.

Quote from: Danne on January 04, 2018, 06:17:08 AM
How do one search for a changeset in source tree?
Changeset: 870cedc16e2d+8f8e391db888+

All I can find is your link to an early build in this post but it´s been removed:
http://www.magiclantern.fm/forum/index.php?topic=19336.msg194178#msg194178

You can find it either on the Bitbucket website:

(https://farm5.staticflickr.com/4734/27712237579_7e3c7b21e7_z.jpg) (https://flic.kr/p/JdQsKD)

Or in a clone of my repository using this command:

hg update 870cedc16e2d+8f8e391db888
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 04, 2018, 07:01:54 AM
ah, thanks a lot.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 04, 2018, 10:16:57 AM
Quote from: dfort on January 04, 2018, 05:04:13 AM
please open the autoexec.bin file on that build that is working so well for you and post it here so we can track down the changeset.

I have a better idea, Dfort.  I have packed your Dec. 11-th build directly from my camera into a .RAR archive that you can download in the next 7 days from here:

https://we.tl/BkF9sH3Ymk

In this way, you do not only have the autoexec.bin file but also the entire build, including the config file, so you can see what camera settings I have been using.  I believe, I have just added a raw_tweak module to the Modules directory to enable in-camera palyback of the recorded clips.  If you please upload the build as it is to your 7D you will be able to precisely reproduce all the tests that I made and provided feedback on.

After several more tests, I have to admit that this Dec. 11-th build is not very stable either but it WORKS!  When I use the normal uncropped modes (RAW-video (MLV) and RAW-video, 1728x972 resolution @24 fps, 10/12 bit), the first recording after camera has been switched on is perfectly OK.  Subsequent recordings sometimes work out fine, sometimes get shaky.  If this happens, I have to turn camera off and on again to make it record properly, without earthquake shaking.  In the 5x-magnification mode there is no earthquake shaking and I can record clips even at the maximum 2496x1198 resolution with sound but the frozen screen during recording makes the usage of this fantastic mode very limited.

I believe, the first 10/12 bit recording after camera is turned on is a very good starting point to identify the condition at which earthquake shaking does not occur.

I hope this helps and keep my thumbs pressed that you succeed!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 04, 2018, 05:13:20 PM
Thanks @IDA_ML

This is interesting, the "Miracle Build" was compiled on the Windows 7 system that I used at work--using the Cygwin environment. It shouldn't make any difference if ML is compiled on Windows, Linux or Mac but maybe? It would be great if you could test it against the Dec. 11 Mac build I posted yesterday.

@Danne -- check out the branch, it isn't the allocate-raw-lv-buffer-wip branch. Backing out the changes since Dec. 11 will only remove the raw_twk module. The other changes were only on the 1100D and 5D2.

Magic Lantern Nightly.2017Dec11.7D203
Changeset: 7a58a90ecbb4+fee1ee2bf3b3+ (raw_video_10bit_12bit_LVState-wip) tip
Camera   : 7D
Firmware : 203
Built on : 2017-12-11 17:22:07 by dfort@QANTAS

diff -r 7a58a90ecbb4 src/edmac-memcpy.c
--- a/src/edmac-memcpy.c Sun Dec 10 08:40:46 2017 -0800
+++ b/src/edmac-memcpy.c Mon Dec 11 09:22:07 2017 -0800
@@ -81,6 +81,9 @@
     resLock = CreateResLockEntry(resIds, 4);
     
     ASSERT(resLock);
+
+    /* just to make sure we have this stub */
+    static int AbortEDmac_check __attribute__((used)) = &AbortEDmac;
}

INIT_FUNC("edmac_memcpy", edmac_memcpy_init);
Changeset: 7a58a90ecbb4+fee1ee2bf3b3+ (raw_video_10bit_12bit_LVState-wip) tip
Built on : 2017-12-11 17:22:07 by dfort@QANTAS





diff -r 7a58a90ecbb4 src/edmac-memcpy.c
--- a/src/edmac-memcpy.c Sun Dec 10 08:40:46 2017 -0800
+++ b/src/edmac-memcpy.c Mon Dec 11 09:22:08 2017 -0800
@@ -81,6 +81,9 @@
     resLock = CreateResLockEntry(resIds, 4);
     
     ASSERT(resLock);
+
+    /* just to make sure we have this stub */
+    static int AbortEDmac_check __attribute__((used)) = &AbortEDmac;
}

INIT_FUNC("edmac_memcpy", edmac_memcpy_init);


Interesting that the changes that were not committed are showing up twice. I've never seen that before.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 04, 2018, 06:38:41 PM
Quote from: dfort on January 04, 2018, 05:13:20 PM
It would be great if you could test it against the Dec. 11 Mac build I posted yesterday.

LOL, Dfort, things seem to get quite messy here.  I don't know which build you mean.  I do not see a "Dec. 11 Mac build" in your download area.  Could you please send me a download link and I will test it for you.

By the way, I might be able to find your very first build with the RAW_Slurp issue resolved, that you posted, (I believe, this was the Dec. 5-th or 6-th build).  I remember, it was quite buggy and there was no in-camera playback but in the uncropped RAW-video modes, it seemed to provide 10/12 bit clips without earthquake shaking.  I you want that build as a starting point, please let me know and I will look for it on my home PC.

One more suggestion.  As I reported, with the Dec. 11-th Miracle build, the first 10/12-bit recording after camera is turned on, works fine - no shaking.  With the second and following ones there is no guarantee that you will not get a shaky clip and you never know which one that clip will be until you play it.  Maybe you could implement some kind of an automatic routine that simulates the camera turn-on condition right after the first clip is recorded.  In this way, the camera will always think that it has just been turned on and will record the next clip without shaking.  Then the routine is run again after the second clip and then again and again until all clips are properly recorded.  I know, this is not a very elegant solution but if it works, why not?  How about that?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 04, 2018, 10:32:15 PM
Dfort,

I have some very good news for you!  I think, I found out what is causing the earthquake shaking in the 7D videos.  It is not your builds, it is the RAW TWEAK MODULE that we added to enable in-camera playback!

As I said, I found your December 6-th build on my computer and started testing it again.  I was able to record many clips one after the other with both: the RAW-video (MLV) and the RAW-video modules in the normal uncropped mode at 24fps and 1728x972 and 1736x976 resolutions, respectively.  I downloaded them to the computer and palyed them back after conversion to H.264 files.  And guess what.  There was not a single clip with the earthquake shaking!  All clips were perfectly OK without a single corrupt frame in them.  In the RAW-video (MLV) mode sound also works synchronously.

Then I decided to add a Raw-tweak module to allow in-camera playback and view the clips right after they have been recorded.  And voila, just the first clip after camera is turned on is OK.  All the ones that follow are shaky!  This is clearly seen during in-camera playback.  And I also confirmed the shakyness in the computer.  Then I disabled the RAW-tweak module again and took several more clips in the above modes.  The shakyness was gone!

So, my conclusion is that not your modules but the RAW-tweak module needs a fix. 

I have now uploaded for your reference that Dec. 6-th build as I downloaded it originally a month ago (without the RAW-tweak module):

https://we.tl/swGMxT851M

If you could please repeat my tests with this build on your 7D, I am sure, you will be very pleasantly surprised and will know how to proceed. So, don't dispare but keep up your excellent work!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 12georgiadis on January 04, 2018, 11:15:17 PM
I can confirm what IDA_ML is saying about crashing issue due to raw playback, both on the 7D and 5dmk3. And it's not related to one build. I stopped to use it because I had to reset the camera so many time. That's why h264 proxy playback is a god function to playback clips after recording on any camera. It's stable on 5Dmk3. I hope it can be generalized on others cameras


Envoyé de mon iPhone en utilisant Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on January 05, 2018, 01:14:04 AM
Happy New Year To My Fellow Magic Lantern Enthusiasts,

I just read the possible corrections to the builds I've been testing but they give me shake Choppy video, but I will try it without RAW TWK module.

@dfort @reddeercity
When I make a build on Mac is it correct that I should open up Sourcetree, double click on raw_video_10bit_12bit_LVState-wip in Clone, then open terminal paste

cd Magic-LanternCloneRepository/platform/5D2.212
make zip
cp *.zip ~/Desktop
make clean

Take zip from desktop, install it on cf card, Also when we make edits do we edit the raw.c in src folder? And how do we know what to edit?

I know these question are all over the place but they will help me figure out how to assist in the builds. Thanks again
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on January 05, 2018, 02:10:49 AM
These links (may take about 30min to upload) are to video with choppy video, also it is still choppy outcome even with raw_twk.mo turned off

https://www.dropbox.com/s/e5717umzd2hhb0d/CropM04-1943.mov?dl=0
https://www.dropbox.com/s/6hh783vws66befm/Non-CropM04-1942.mov?dl=0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 05, 2018, 04:00:39 PM
To the developers familiar with the RAW-tweak module

This module urgently needs troubleshooting and fixing, otherwise it undermines the developers' efforts to port crop_4k RAW video to various cameras !  

As far as I understand, the RAW-tweak module is used to playback 10 and 12-bit uncompressed MLV clips in the camera.  I does this job very well and is quite useful to quickly review the clips right after they have been recorded.  If it is not loaded or disabled, a black screen occurs when trying to playback a 10/12-bit clip.  Unfortunately, when the module is available in the Modules directory and activated, it seems to interfere with other RAW video recording features, in particular with the MLV-lite module, and causes a lot of trouble.  That is why, in most cases, it has to be disabled to get the work done.

Below, please find a brief description on what happens when the RAW-tweak module is active on some of the cameras:

The EOS 7D with the Dec. 6-th 2017 build by Dfort:
================================

In the normal uncropped modes (RAW-video (MLV) and RAW-video, 1728x972 resolution @24 fps, 10/12 bit), the first recording after camera has been switched on is perfectly OK.  Subsequent recordings get shaky and this can be clearly observed during in-camera playback.  If this happens, the camera has to be turned off and on again to make it record properly, without earthquake shaking for one more clip.  Once the module is disabled, all subsequent clips in the above modes get perfectly recorded and there are no glitches, corrupt frames or shaking.  No camera turning-off is required.  These error free clips can also be viewed in the camera after the module is activated again to verify that there is no shaking. 

The EOS 100D/SL1, Dec. 19-th Experimental build
===============================

The RAW-tweak module does not work in any of the RAW-video modes.  When the record button is pressed, the "Threads failed to start" message appears and the screen turns black.  A red dot and a "Starting" message appears next to the RAW-video mode selected but recording never starts.  When the module is disabled, recording works normally but when it is enabled again to playback the recorded clips, a black screen occurs.  When now halfshutter is pressed the red LED starts blinking but the screen remains black.  The red LED does not stop blinking even after the camera is turned off.  The only way out of this state is a battery pull.  The RAW-tweak module has to be disabled for proper camera operation.

The EOS 5DMkIII (several builds with the RAW-tweak module included)
============================================

See the brief explanation by 12georgiadis in his post #1559 above.

-------------------------------------------------------------------------------

I hope, the above behavior can be confirmed also by other users and the RAW-tweak module will receive a fix to allow further work on the latest developments.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 09, 2018, 10:25:39 AM
Are 12georgiadis and I the only people having trouble with the RAW-tweak module?  Please report on your experience with it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Kharak on January 09, 2018, 02:17:56 PM
5D3 123

I dont have any issues with RAW_twk.mo and recording. Not with MLV_Lite or MLV


I had some random issues at times not being able to play MLV's in camera except for the last shot. But I believe I fixed it by removing some IDX files.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 09, 2018, 02:33:33 PM
What about owners of of other camera models (EOS M and 700D)?  Does the RAW-tweak module cause problems with the latest experimental builds with crop-4K recording on your cameras?  How about the cameras working with 10/12-bit recording?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 12, 2018, 12:19:38 AM
raw_twk is causing problems on the EOSM possibly because it has maxed out on the number of threads it can handle (https://www.magiclantern.fm/forum/index.php?topic=9062.msg176692#msg176692).

This might also explain why things were working better on my older 7D builds before I added the raw_twk module.

As far as the future of the raw_twk module, I think that the plan might be to eventually incorporate something more efficient in mlv_play to playback lossless compressed files. Then again if we can get H.264 proxy working you will be able to playback the H.264 file with audio in camera.

Off topic: @IDA_ML -- There has been lots of work recently to get the focus pixel map files working for 8...12-bit lossless compressed files in MLVFS. Most of the testing was done on EOSM and 700D but it could really use a thorough test from an experienced 100D user.  ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 25, 2018, 07:26:06 AM
I took another look at a video recorded on the 7D in 10bit:

Block: RAWI
...
    Res:  2520x1192
    raw_info:
...
      height           1224
      width            2520
...
      bits_per_pixel   10


The image size and raw buffer check out as valid:

In [1]: 2520*14//8*1224 % 16
Out[1]: 0

In [2]: 2520*14//8*1192 % 16
Out[2]: 0


Raw buffer height (1224) - skip_top (26) = 1198 so why is the Image height only 1192?

In [3]: 2520*14//8*1198 % 16
Out[3]: 12


Not valid. Ok--this makes sense.

But it looks like there is some doubt these are the right skip settings:

raw.c from the raw_video_10bit_12bit and unified branches:
        #ifdef CONFIG_7D
        #warning FIXME: are these values correct for 720p and crop modes?
        skip_top    = 26;
        skip_left   = zoom ? 0 : 256;
        #endif


So it looks like we should run raw_diag and shoot some simple silent DNG's on the 7D and figure out where the out of bounds area is located on the various video modes, right?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 26, 2018, 12:25:24 PM
Dfort,

I have a strategic question on the 7D development to you.  You said that the 10/12 bit incompressed RAW video development is a tough nut to crack and I fully agree with you keeping in mind how much time and efforts you invested in this development.  My question to you is: 

Do we really need 10/12 bit uncompressed? 

Since October last year, I have been shooting 8...11 bits and 12 bits losslessly compressed video with sound (continuous recording at 24fps) on the 100D and could not be happier.  Video quality is fantastic and files are finally reduced in size to reasonable levels.  Since then, I never felt the necessity to film at 10, 12 or 14 bits uncompressed.  So, to continue my question on the 7D let's consider the hypothesis that the 10/12 bit incompressed is left alone for a while and porting 4K_crop_recording is proceeded with.  Would that be possible and if yes wouldn't that be easier and with better chances for success than 10/12 bit incompressed?  If the answer to that question is yes, then, keeping in mind the fast write speed of the 7D, we would have continuous 8...11 bit or even 12-bit LL recording WITH SOUND in the 2520x1200 resolution!  Imagine for how many people such video would be a dream come through !!!

Let us know what you think.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 26, 2018, 03:28:28 PM
My understanding is that we need to get 10/12 bit uncompressed before attempting to compress it. Maybe we could skip 10/12 bit and go to 14 bit compressed but then everyone will still want 10/12 bit compressed.

I think the way we dealt with the 100D "hiccup" issue might bring us a step closer to figuring out how to resolve the "earthquake" issue in 10/12 bit on the 7D. If nothing else, those skip values should be verified. This is something you can do with the raw_dia module. No coding necessary. It is available on the bottom of the modules downloads page (https://builds.magiclantern.fm/modules.html).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on January 26, 2018, 04:07:27 PM
Dfort,

Your Dec. 06-th, 2017 build with the raw_tweak module disabled works perfectly OK with 10/12 bits uncompressed RAW-video in the normal uncropped mode on the 7D.  No "earthquake shaking" is observed!  For stable 10/12-bit operation at the 5x-magnification mode, I still use the Dec. 1, 2016 build - the "haunting" one.  Once you get the Preview working properly, (by switching preview modes back and forth), this 5x-mode also provides excellent and very stable results with sound even at the highest 2496x1198 resolution.  I love filming with these two builds!  May be, starting from them is a good idea ???

I have now downloaded the raw_diag.mo but I don't know how and what to test and expect.  I don't know how to "save the RAW buffer" either.  If you give me detailed instructions on how to test, at what camera settings and what to expect, I can give it a try.  I cannot perform tests if I don't understand what they are doing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 26, 2018, 06:41:38 PM
A good place to start is reading the CMOS/ADTG/Digic register investigation on ISO (https://www.magiclantern.fm/forum/index.php?topic=10111.0) topic.

Things to do:


raw_diag does several tests but what we're looking for is being able to save an image that looks something like this:

(https://farm5.staticflickr.com/4705/28132432609_2612f82374.jpg) (https://flic.kr/p/JRY56x)

Notice how that blue line is showing the active area defined by ML.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 27, 2018, 05:33:23 PM
Looks like @dariSSight found something that works better on the 5D2 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg196439#msg196439). I had this before but it didn't make sense that only the 5D2 would be different but hey--so much of this stuff doesn't make sense to me!

src/raw.c
#ifdef CONFIG_5D2
- #define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
+ #define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
#endif


Don't have a 5D2 to test so I'll just follow his lead. Made the change and posted a test build on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: masc on January 27, 2018, 06:59:55 PM
@dfort: I tried it with my 5D2:
mlv_rec + mlv_snd: 1856x1044 10bit/12bit flickers pink. When enabling fps override @20fps all is fine. @23.976 it flickers again.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on January 28, 2018, 12:09:59 AM
These are the 2 Builds it gives me when I use Terminal
https://www.dropbox.com/sh/acqkc6zebq7ryq6/AAB7AqgJGoKwJQECPi6yrifya?dl=0

so should I keep using -
cd Magic-LanternCloneRepository/platform/5D2.212
make zip
cp *.zip ~/Desktop
make clean

I do have two Directory I think 1.magic-lantern & 2.Magic-LanternCloneRepository, so should I delete 1.magic-lantern Directory
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 28, 2018, 07:55:45 AM
Both of those builds are almost identical. The only difference being the date you saved and compiled them. It is easy to figure out if you open autoexec.bin in a text editor:

Magic Lantern Nightly.2018Jan16.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 857ed765edf8+ (raw_video_10bit_12bit_LVState-wip)
Built on : 2018-01-16 10:17:43 by dariSSightEntertainment@dariSSiinneyMBP

diff -r 857ed765edf8 src/raw.c
--- a/src/raw.c Fri Dec 22 19:08:35 2017 -0800
+++ b/src/raw.c Tue Jan 16 05:17:43 2018 -0500
@@ -142,7 +142,7 @@
#endif

#ifdef CONFIG_5D2
-#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
#endif

#ifdef CONFIG_50D


Magic Lantern Nightly.2018Jan27.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 857ed765edf8+ (raw_video_10bit_12bit_LVState-wip)
Built on : 2018-01-27 07:02:58 by dariSSightEntertainment@dariSSiinneyMBP

diff -r 857ed765edf8 src/raw.c
--- a/src/raw.c Fri Dec 22 19:08:35 2017 -0800
+++ b/src/raw.c Sat Jan 27 02:02:58 2018 -0500
@@ -142,7 +142,7 @@
#endif

#ifdef CONFIG_5D2
-#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
#endif

#ifdef CONFIG_50D


So you cloned my repository twice? No big deal but of course you don't really need to do that. You can delete one or the other, it doesn't matter. In fact you can rename one of them magic-lantern-dfort (my forum name) or magic-lantern-daniel_fort (my Bitbucket name) it doesn't really matter. What is important is that you know which repository you're working from. You may want to clone the main (hudson) repository so you also have access to the latest "official" source code. I tend to update my repository quite often and merge in the latest changes from the hudson repository.

At some point if you want to submit a pull request so your changes get into the main repository you should fork the hudson repository. There's lots of tips on the Internet for using hg (Mercurial) or the gui front end, SourceTree (https://www.sourcetreeapp.com/).

In any case, what you are doing looks fine. At least I don't see any problems from here.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 31, 2018, 12:00:28 AM
Just tried to backport raw backend's memory management from crop_rec_4k - updated the allocate-raw-lv-buffer branch (which was initially created to add raw video support for 1100D) and merged it into raw_video_10bit_12bit. I ran a quick test on 60D, with a DEFAULT_RAW_BUFFER_SIZE modified to 4MB to make sure it reallocates on larger resolutions (this size was larger than what's required in 1080p, but smaller than what's required in x5 zoom), but that was pretty much all I've tried.

Unfortunately it doesn't build on older models (those without CONFIG_EDMAC_RAW_SLURP), so we are one step back from integration into mainline. But at least 1100D joins the party (thanks Danne (http://www.magiclantern.fm/forum/index.php?topic=1009.msg196544#msg196544)), hopefully as a first class citizen (but staying that way depends on user feedback).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 31, 2018, 09:48:19 PM
Quote from: a1ex on December 10, 2017, 11:43:47 PM
DEFAULT_RAW_BUFFER is used as a pointer to some valid raw buffer (allocated by Canon code). On recent models (60D and newer), it is allocated when entering LiveView (evfActive), but that pointer is no longer valid when leaving LiveView (it's still there, but stale)

Some more notes about this.

On models with EvfState, the raw buffer address (or a pointer to it) can be found in debug logs as "Mem1 Address".

On 60D, 600D, 1100D, 1200D and 1300D (other models seem to be different), a bunch of buffers are preallocated when entering LiveView and kept in some structure; they are retrieved by ADST_GetMemAddrArray (no idea what it stands for). With the memory browser (enabled by default in the dm-spy-experiments branch, or startup-log builds (http://builds.magiclantern.fm/jenkins/view/Experiments/job/startup-log/)), you can look at 0x26400 (60D) or 0x8A90 (1100D), dereference the pointer and see all these buffers aligned.

I'm trying to find out the size of this buffer.

First puzzle: where it's allocated? QEMU doesn't help - the emulation doesn't go that far.

Since I have no idea where that happens, except that it's somewhere in LiveView initialization (which is a huge jungle), I'm going to print some buffer from this structure along with each debug message (my_DebugMsg in dm-spy.c, dm-spy-experiments branch), and see where this is set:

    len += snprintf( buf+len, buf_size-len, "%05X> %s:%08x:%08x: ", us_timer, task_name_padded, lr-4, MEM(MEM(0x26400)) );



EC8C8>        Gmt:ff0fcf64:e1a00000: [GMT] gmtInit
EC93E>        Gmt:ff0fcfcc:00000000: [GMT] gmtStart:0x2
...
D3CE7>        Gmt:ff101fec:00000000: [GMT][WAKU] Event 15 Result State 6->6
D3D6D>        Gmt:ff0f8bc8:41b07800: [EVF] evfComAct(0)


Still, between these two messages, there's a bunch of stuff happening, so I still have no idea where that buffer is allocated.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 01, 2018, 12:12:56 AM
I hope this helps with some info -- Run startup-log.2018Jan29.5D2212 (https://builds.magiclantern.fm/jenkins/view/Experiments/job/startup-log/) to get so Log files I uploaded to my bitbucket account
dm-0000-nonraw-startup-log.2018Jan29-5D2.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0000%5Bnonraw-startup-log.2018Jan29%5D.log) , dm-0000-raw-startup-log.2018Jan29-5D2.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0000%5Braw-startup-log.2018Jan29%5D.log) , dm-0001raw-recording-startup-log.2018Jan29-5D2.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0001%5Braw-recording-startup-log.2018Jan29%5D.log) .
First logs is after loading up dm-spymem built without any thing loaded , next is with raw enabled the final one is with raw recording in 1:1 (1856x928 23.976p)



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 01, 2018, 12:51:29 AM
Thanks, would require a bit of context switch to analyze them, as LVState models are a completely different beast from EVFState.

I've got a hypothesis about the raw buffer on EvfState models - full chunks (size = srm_buffer_size) are allocated for LiveView use (SRM_AllocateMemoryResourceForLiveViewYuv) and then divided later; the addresses end up in the ADST_GetMemAddrArray (ADdress STore?) at some point, but where that exactly happens is a mystery. In any case, we've got an experiment named CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP that fills the entire memory with a known pattern, so we can see what parts of memory are used and what parts are likely free (in the Free Memory dialog - the color bar). With this tool, I've noticed the raw buffer address appears to be repeatable (always allocated in the same spot), and even if it's not, it's likely just in a different chunk - but that chunk is going to be divided in the same way.

Here's a test I'd like to run on all EvfState models (those with 10/12-bit raw video builds on the Experiments page). Instructions in the commit message (https://bitbucket.org/hudson/magic-lantern/commits/e799e89f67877825c6a4dc2868f87bed1196432b). In addition, I'd like a screenshot of the Free Memory dialog in movie mode. I've ran the test on 60D and 5D3 1.1.3; these are the only EvfState models I own.

The test can be done on either on the allocate-raw-lv-buffer branch (possibly better for older models), or on the current crop_rec_4k branch. I don't expect this to affect the results in any way, so just pick one. If it doesn't work, pick the other.

Will edit the following summary as the test results will arrive:

5D3 1.1.3: 0x4B152000 - 0x4CDFFFFC (repeatable; tested photo mode, 1080p25, 720p50, x5, with and without HDMI monitor)
5D3 1.2.3: ? (old notes: 0x4d31a000 - 0x4ee00000)
6D: 0x4B328000 - 0x4CFFFFFC
650D: same values as 700D/EOSM
700D: same values as EOSM (repeatable; tested mv720, mv1080, mv1080crop, x5)
100D: 46798100-46CC40C4
70D: 4B328000-4CFFFFFC on camera, 4B328000-4D7FFFF0 in QEMU
EOSM: 46798080 - 47F24060 (repeatable; tested mv720, mv1080, mv1080crop, x5)
EOSM2: 48798100 - 48CC40C4

60D: 48332200 - 49F6807C (repeatable; tested photo mode, 1080p25, 720p50, x5, with and without HDMI monitor)
600D: ? (internals like 60D)
1100D: ? (rough guess: 0x46b00000 - 0x473b0000)
1200D: ? (requires merging)
1300D: placeholder (not yet ready to run the test, but it's in the same group)

Hopefully these are all models with CONFIG_EDMAC_RAW_SLURP.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 01, 2018, 09:18:12 AM
EOSM is a little quirky. Couldn't get screenshot to work while the Show Memory display was open so I did a quick cell phone shot--in movie mode:

(https://farm5.staticflickr.com/4623/39307761334_fd65aee5e4.jpg) (https://flic.kr/p/22TuxZ1)

The camera always starts up in mv720 mode:

(https://farm5.staticflickr.com/4654/39307650384_66a925c162.jpg) (https://flic.kr/p/22TtZ15)

but it does display changes--I switched to Movie Crop Mode and zoom mode:

(https://farm5.staticflickr.com/4658/40017055891_efa57ccf49.jpg) (https://flic.kr/p/23YaSxv)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 01, 2018, 09:50:14 AM
If the camera cannot start in the tested mode, you can start the camera with raw video turned off, then switch to that mode (x5, crop mode, whatever), then enable raw video. The raw buffer size detection is performed once, right before this buffer gets filled with data (it's harder to do it afterwards).

Could not reproduce the screenshot issue in QEMU (it captures the screenshot of the Free Memory dialog and saves it correctly)...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 01, 2018, 05:50:27 PM
Ok--here's the EOSM started up in 1080p mode (Canon menu). Starting in 720p mode gives the same results. Tried again with the screenshot of the Show memory screen, still won't work. After a few tries the led doesn't even do the countdown blink.

(https://farm5.staticflickr.com/4707/39992484962_ba416280dc.jpg) (https://flic.kr/p/23VZWsN)

Move Crop Mode
(https://farm5.staticflickr.com/4703/39314333804_f9d90a91c2.jpg) (https://flic.kr/p/22U5eKs)

zoom mode
(https://farm5.staticflickr.com/4678/40023980631_e90b45b619.jpg) (https://flic.kr/p/23YMn2z)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 01, 2018, 10:14:39 PM
700D

Can't do a screenshot of the Show memory display on the 700D either.

(https://farm5.staticflickr.com/4717/25155276537_1409c9178b.jpg) (https://flic.kr/p/EjTnjr)

mv1080
(https://farm5.staticflickr.com/4712/25157255567_db3640629d.jpg) (https://flic.kr/p/Ek4vBD)

mv720
(https://farm5.staticflickr.com/4752/26155285598_c1d96bc508.jpg) (https://flic.kr/p/FRfFaY)

zoom
(https://farm5.staticflickr.com/4675/28249096659_c6db57079c.jpg) (https://flic.kr/p/K3h1fF)

mv1080crop - Movie Crop Mode
(https://farm5.staticflickr.com/4742/39996493642_5231fc3493.jpg) (https://flic.kr/p/23Wmu6Y)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Arthur21Dayne on February 02, 2018, 11:37:00 PM
Hey guys, Im pretty new to ML, especially to 10 and 12 bit hacks, which im using on my eos m rn, I also have a 5d2 and a 50d so I was wondering if there will ever be a 10 bit or crop mode (3x3) hack for any of those cameras in near future. Ive been around the forum a while now and couldnt find a satisfying answer to that question. Is anybody working on it? Would it even be possible, if not, why not?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: canneloni on February 04, 2018, 01:16:55 PM
Did the test on 100D (hope I did it right) with latest crop_rec build (magiclantern-crop_rec_4k.2018Feb04.100D101). Folder with pictures can be found here:

https://drive.google.com/open?id=1nHumQo7ZMGHq7lz-Ml6FPgdyGBRSGv7_

While doing the free_memory test, the camera crashed and created a log file, wich also can be found in the linked folder.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 04, 2018, 02:44:23 PM
Quote from: Arthur21Dayne on February 02, 2018, 11:37:00 PM
...have a 5d2 and a 50d...Is anybody working on it?

I gave it a shot. It is partially working on the 5D2 and much better on the 50D. Haven't given up but ran into a wall. Start reading from here:

https://www.magiclantern.fm/forum/index.php?topic=5601.msg194222#msg194222

There are still test builds posted on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).

[EDIT] There have been lots of changes to the allocate-raw-lv-buffer and raw_video_10bit_12bit branches. It probably won't make any difference to the Digic 4 LVState cameras, 5D2/50D/500D/550D/7D, but I merged in the latest changes and posted new test builds to my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). Please report only good news!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 05, 2018, 02:23:44 AM
Just for kicks I tested the latest (5D2 10-12bit) knowing nothing was really changed.
As to be expected 1:1 & 3x crop_mode are broken in all bit depths with mlv_lite (I thought 14bit work be ok but not)
full mlv + audio , 1:1 broken in all bit depths thou 3x crop mode seem to be recording ok from the very short clips I shot (10-15 sec.)
So this makes me believe that there has to be something in mlv_lite that causes a total failure in all bit depth + in 3x crop ,
but full mlv+audio in 3x crop mode seems to be some what functional (liveview framing not correct , pink cast on lcd screen)
but this does not cause corrupt frames so far.  I did get a "Raw error detect" and crashed ml , only after I tried moving the 3x crop window while recording (this feature used to work in nighty's)
ML ASSERT:
raw_info.bits_per_pixel == bpp
at ../../src/raw.c:2020 (raw_lv_request_bpp), task raw_rec_task
lv:1 mode:3
raw_rec_task stack: 176f68 [177118-176118]
0xUNKNOWN  @ ff8773e0:177110
0x000904C8 @ 929328:176fc0
0x0004EA74 @ 90548:176f98
0x0004E378 @ 4ead0:176f68
Magic Lantern version : Nightly.2018Feb04.5D2212
Mercurial changeset   : 24d40e1a8c3c (raw_video_10bit_12bit_LVState-wip)
Built on 2018-02-04 17:03:59 UTC by rosiefort@RosieFoComputer.
Free Memory  : 133K + 3968K

This happen when a had the crash from 3x crop by trying to move the crop window while recording raw
(https://preview.ibb.co/hNGcQc/VRAM16.png) (https://ibb.co/mitfdx)

Notice there a conflict in scr/raw pull-request 16 - 10bit-12bit-for-lvstate-cameras/diff  (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/16/10bit-12bit-for-lvstate-cameras/diff) at raw.cT1635 (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/16/10bit-12bit-for-lvstate-cameras/diff#Lsrc/raw.cT1635) , did you manage to resolve it ?

the resolution change was from 3x crop & 1:1 , so that prove it ,2312x1127 is crop mode up to 2152 & 1:1 2040x1267 full HD/full res
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 05, 2018, 04:06:40 AM
Quote from: reddeercity on February 05, 2018, 02:23:44 AM
Notice there a conflict in scr/raw pull-request 16 - 10bit-12bit-for-lvstate-cameras/diff  (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/16/10bit-12bit-for-lvstate-cameras/diff) at raw.cT1635 (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/16/10bit-12bit-for-lvstate-cameras/diff#Lsrc/raw.cT1635) , did you manage to resolve it ?

Thanks for pointing that out. It is a problem. Then again I started this with the assumption that what was needed on the Digic 4 LVState cameras was to find the DEFAULT_RAW_BUFFER. Turned out I was wrong (https://www.magiclantern.fm/forum/index.php?topic=5601.msg194585#msg194585).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 05, 2018, 06:48:39 AM
I see , yes I remember reading this forgot all about this .
did a google search for RAW_SLURP to understand the syntax and usage etc. ... to see if there something else would work , came up with this  "slurp_raw" not sure if that's the same thing
see link below .
http://search.cpan.org/~dagolden/Path-Tiny-0.104/lib/Path/Tiny.pm#slurp,_slurp_raw,_slurp_utf8
https://metacpan.org/pod/Path::Tiny#spew,-spew_raw,-spew_utf8

There is something that's been overlooked I think and that's the LCD Display it's self , I mean that digic iv from what I understand is 4:1:1 color space were the digic v & up are 4:2:2
as explained by a1ex here (https://www.magiclantern.fm/forum/index.php?topic=14732.msg142463#msg142463) in Display access from bootloader / Portable binary test thread
I have this feeling that there need to be something to take in account for the differences , I guess that could be the Vsync issue but from my experience with broadcast equipment this seems
it need to be conformed to 4:2:2 color (up sample) , this my be why 3x crop mode works (normally) because we are using only part of the screen . But I could  be total wrong too  :D


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 05, 2018, 08:46:09 AM
Ok that confirm it from my dm-startup log , LCD screen is YUV 4:1:1 , but is that just for face detection ?
C7654> LiveViewMg:ffa0c904:18:01: [DETFEN] DetectFace_SetDataYUV411VType
Color space address , possible to modify to 4:2:2 , or is it hardware only 4:1:1
D28A1> LiveViewMg:ff8e057c:9b:03: YuvAddress = 0x50000064 -> 0x50000080
I see a error in the log too , not sure what this means , everything seem to work ok
EC92E> LiveViewMg:00096198:00:00: *** register_interrupt("HEADERROR", 0x6c, 0xff987004 "[CAP]:ISR >>>>>>>>> HEAD ERROR >>> AF SHIFTER ERROR", 0x0), from ff9871d0
It's late can see any more , continue tomorrow .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 05, 2018, 09:08:44 AM
I was talking about bootloader (think about the Loading message from the firmware update). There, the image overlay (unused in Canon code, just in our demo) is YUV411 in D4 and YUV422 in D5. On DIGIC <= 5, the display hardware accepts 2 image overlays: BMP (at the top), palette-based, with transparency, and image (YUV-based, BT 601 for old models / 709 for newer ones, not 100% sure where is the age threshold). On the bootloader, D4/5 use a 4-bit palette for BMP; in main firmware, they both use a 8-bit palette.

In LiveView (main firmware), it's 422 on both (unless you connect a low-res monitor, when it drops to 411, thus breaking some of our overlays). On DIGIC 2 (unsure about DIGIC 3), the image buffers (such as what gets displayed on the LCD) are 411. The firmware may use additional buffers for other stuff (didn't really look into these).

"Slurp" is our own naming, you won't find it anywhere else. It refers to the transfer of the raw image data into main memory, after it was already captured by Canon code; the transfer itself is done with our own code (which allows us to fine-tune the parameters and the timing - when it happens).

That message is not an error. It means "register an interrupt handler for interrupt 0x6c, named HEADERROR (http://www.magiclantern.fm/forum/index.php?topic=21526.msg196300), so that whenever this event happens - probably some error in the image capture process - the function 0xff987004 gets called (and that function uses that string, which is likely to provide a hint on what that function does, so it gets displayed in the log)".

If this low-level stuff sounds strange, be sure to check the Basic concepts (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/HACKING.rst#rst-header-basic-concepts) section from the QEMU guide. Just be careful not to confuse "register an event handler" with "MMIO register" ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 06, 2018, 04:16:41 AM
Thank you , So much to understand  :D That prove it You can teach a OLD Dog new tricks  :P

(https://petcube.com/blog/content/images/2017/12/smart-dog-math.gif)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 08, 2018, 05:05:59 AM
Just Thinking out load , been wondering about raw_type for the digic 4 I think it's type 5 , would a different type work ? or is there not other setting .
Can't find any info yet , still looking . 

Edit:is there any info about lv_raw_size ? 5d2 uses "16" is there any other sizes ? or how can I find if there is ?  still looking
/* 5D2 uses lv_raw_size >> 16
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 08, 2018, 05:44:03 AM
I think I found raw_type or something close in  lv_set_raw / lv_select_raw  (https://www.magiclantern.fm/forum/index.php?topic=18393.msg176315#msg176315) thread but it's listed for 60d
can I assume for all digic 4 ?

RSHD     => 0x0B
SHADE    => 0x01
HIVSHD   => 0x07
ORBIT    => 0x09
DEFCORRE => 0x04
CCD      => 0x05 (currently used)
DEFMARK  => 0x06

So this means CCD , make sense  is for imaging , but I also have read something about RSHD has something to do with it or it may be the Liveview LCD can't find it right now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on February 09, 2018, 01:04:45 PM
Without going into details:
https://builds.magiclantern.fm/features.html -> Internals -> CONFIG_EDMAC_RAW_SLURP
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 10, 2018, 12:08:08 AM
Not sure what you mean here ?
I know that " CONFIG_EDMAC_RAW_SLURP" is not working on 5D2 , in fact on all digic 4 cam that what I'm tying to get functional
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 10, 2018, 12:18:57 AM
Don't worry too much about CONFIG_EDMAC_RAW_SLURP (https://www.magiclantern.fm/forum/index.php?topic=5601.msg194585#msg194585). There is a difference between the 60D and 600D which are EVF_STATE cameras and 5D2/7D/50D/500D/550D which are LVState cameras. All EVF_STATE cameras supported by ML have been ported to use the reduced bit depth (8..12bit) code in the raw_video_10bit_12bit branch. There is a sync issue to resolve with LVState cameras.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on February 10, 2018, 01:33:37 AM
Quote from: reddeercity on February 10, 2018, 12:08:08 AM
Not sure what you mean here ?

I answered a person who prefered to delete his/her question about this one.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 10, 2018, 05:53:36 AM
Ha ha--so we've been having a discussion over nothing?

@reddeercity and I have been on this quest to get these remaining Digic 4 LVState cameras (5D2/7D/50D/500D/550D) working on this 10bit/12bit branch and ultimately on the crop_rec_4k branch. The strange part is that if we don't do anything at all with the code and just compile this branch, some of the settings seem to work quite well. Modifying the code we've been able to get some other modes partially working at the expense of the modes that worked without tweaking the code. Sort of like robbing Peter to pay Paul.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 10, 2018, 07:18:16 AM
Yea I know , I still convinced that it will work , I did some test the other day with the 10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip)
That I complied with the change I did (mainly the write channel ) ( the one that a1ex said not to use but works in 1:1)
I what to see what problem or crashes would happen in hopes of maybe seeing something that can help resolve this problem .
So with mlv_lite  without hdmi connected there was no issue crashes etc. ...  but with hdmi in standby ,
I would get the flicker with pink  or out of sequence image(see below) when I start recording raw video the flicker stops and liveview is good to record from hdmi .
I can't say the same as  full mlv + audio I got a memory overflow  ??? didn't matter if hdmi was connected or not .
printed a message on screen "mlv_rec.c line 161" has to do with the SRM memory
maybe audio pushed it over the edge not sure .
(https://thumb.ibb.co/fNDyT7/mlv_lite_10_12bit.png) (https://ibb.co/fNDyT7)  (https://thumb.ibb.co/d5VCo7/VRAM6.png) (https://ibb.co/d5VCo7)

Next I'm going to be looking at big LVState diagram , I did look them over a year ago but really didn't understand that much now I hope I can make head or tales out of it

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 10, 2018, 07:53:25 PM
Quote from: reddeercity on February 10, 2018, 07:18:16 AM
That I complied with the change I did (mainly the write channel ) ( the one that a1ex said not to use but works in 1:1)

Interesting, you changed the write channel and 1:1 works again?

@alex, I've been thinking about the discovery that my experimental builds for the LVState Digic 4 cameras work better when you first use it then it gets into the "earthquake" mode. Maybe the buffer needs to be cleaned out before recording? I'm not really sure how all this works but it seems that there is something left over in the buffer that throws things out of whack. Deleting the ML settings, restarting the camera, set mlv_lite or mlv_raw to 10/12bit and chances are it will work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 11, 2018, 07:52:04 AM
Quote from: dfort on February 10, 2018, 07:53:25 PM
Interesting, you changed the write channel and 1:1 works again?
Yes , changed write channel from 0x01 to 0x02 in edmac-memcpy.c

Also here  (https://www.magiclantern.fm/forum/index.php?topic=5601.msg184371#msg184371) I did test your original 10-12bit build from my downloads in bitbucket  10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip) there I only tested full mlv+audio and as you know 3x crop work perfectly with bit reductions , mainly I record 10bit @ 2144x1076 23.976p and 1:1@ 12bit does work with the top 100 or so lines being corrupted but still useable (just crop off the bad part) . Be on that there not other problems , it just has the 10-12bit code implemented without RAW_SLURP .
See image below , 1st is 12bit 1856x1044 @ 23.976p 2nd is the next frame in the sequence .
(https://thumb.ibb.co/iMCAt7/12bit_pink_bar_M04_2120_00001.jpg) (https://ibb.co/iMCAt7)  (https://thumb.ibb.co/hOUC6S/12bit_pink_bar_M04_2120_00002.jpg) (https://ibb.co/hOUC6S)
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/12bit_M18-0124.MLV --- here is the only 12bit 1:1 I got to work without any corruption (190MB) from  ‎March ‎04, ‎2017,
But for the life of me I can't remember what I changed and I can't find the source code right now  :-[




Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 11, 2018, 08:52:50 AM
Found something that may help a little How do State Machines change states? (https://www.magiclantern.fm/forum/index.php?topic=4344.msg24325#msg24325) ok maybe not
@dfort
Found something that may be helpful from raw.c-104 (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/src/bcfa214064995813a2f4172dddc3fcc60b87af66/src/raw.c?at=unified&fileviewer=file-view-default#raw.c-104)

#ifdef CONFIG_7D
//~ #define DEFAULT_RAW_BUFFER MEM(0x2b14)
//~ #define DEFAULT_RAW_BUFFER shamem_read(0xC0F26008)
#define DEFAULT_RAW_BUFFER 0x43C0D24
//~ #define PREFERRED_RAW_TYPE 16
//~ #define RAW_TYPE_ADDRESS 0x1D38
#endif

It a old experiment from @onepercent , this may be in the backend of the code but I could not find it .I know it's not raw_slurp but
Hope it's helpful
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andrew_dotdot on February 23, 2018, 06:52:40 PM
Sound on the 70D?

I'm a longtime Magic Lantern user, but this part of ML really has a steep learning curve.

I have a 70D and the 12-bit modules are installed and they load and I get video in 10 12 and 14 bit (yay!) using mlv_lite/raw_rec, mlv_play and raw_twk -- but SOUND? Not so much. I look in the MLV with MLVFS, and there's no .wav.

In the past, I've used old-regular MLV with good results and sound, but then you have the mlv_snd module -- and no such module exists in the modules directory of the 70D 12-bit download; and (not unexpectedly) bringing in a mlv_snd module from another build doesn't work at all.

Is the sound working in the 70D at all, and if so, how does one get that working?

Thx so much for this branch of the ML experience!

:A)>
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 24, 2018, 10:47:26 PM
Quote from: a1ex on February 01, 2018, 12:51:29 AM
EOSM2: ? (requires merging)

Filling in the blanks:

EOSM2: 48798100 - 48CC40C4

Here's the Free Memory screen:

(https://farm5.staticflickr.com/4713/39567838895_f657e0b98f.jpg) (https://flic.kr/p/23htvYT)

Like the EOSM it starts in mv720 mode:

(https://farm5.staticflickr.com/4721/38653389150_1c222c4355.jpg) (https://flic.kr/p/21TEHVY)

Looks like it is crying out for help. Not able to make screenshots and the console keeps shrinking down so it is impossible to read the information needed for DEFAULT_RAW_BUFFER_SIZE.

zoom mode:
(https://farm5.staticflickr.com/4655/26593136498_6a213f1d64.jpg) (https://flic.kr/p/GvWLZq)

mv1080crop mode:
(https://farm5.staticflickr.com/4676/26593136178_f687028cd7.jpg) (https://flic.kr/p/GvWLTU)

Raw buffer sizes pretty much as expected.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: saulbass on February 26, 2018, 12:33:35 AM
Some console screens from 650D using dforts - allocate-raw-lv-buffer.2018Feb25.650D104.zip

I tried changing photo/video/lv/crop/720p/1080p modes but the numbers seem to stay the same.


(https://thumb.ibb.co/czMpcc/IMG_4603.jpg) (https://ibb.co/czMpcc)

(https://thumb.ibb.co/eYumOH/IMG_4597.jpg) (https://ibb.co/eYumOH)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 26, 2018, 01:14:51 AM
Quote from: saulbass on February 26, 2018, 12:33:35 AM
I tried changing photo/video/lv/crop/720p/1080p modes but the numbers seem to stay the same.

Did you try starting the camera with mlv_lite active? You can also start with mlv_lite off then turn it on and it should update:

Quote- open the console (Debug menu) and enable something that uses LiveView RAW features (raw video, raw histogram etc)
- test by starting the camera in all video modes (photo, 1080p, 720p, crop, x5 etc)

Quote from: a1ex on February 01, 2018, 09:50:14 AM
If the camera cannot start in the tested mode, you can start the camera with raw video turned off, then switch to that mode (x5, crop mode, whatever), then enable raw video. The raw buffer size detection is performed once, right before this buffer gets filled with data (it's harder to do it afterwards).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: saulbass on February 26, 2018, 10:51:56 PM
Quote from: dfort on February 26, 2018, 01:14:51 AM
Did you try starting the camera with mlv_lite active? You can also start with mlv_lite off then turn it on and it should update:

unfortunately trying repeatedly with the allocate-raw-lv-buffer.2018Feb25.650D104 build, its not possible to activate mlv_lite or crop_rec
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 27, 2018, 02:33:40 AM
Quote from: saulbass on February 26, 2018, 10:51:56 PM
unfortunately trying repeatedly with the allocate-raw-lv-buffer.2018Feb25.650D104 build, its not possible to activate mlv_lite or crop_rec

That's very strange. Ok, so according to the instructions:

Quote from: a1ex on February 01, 2018, 12:51:29 AM
The test can be done on either on the allocate-raw-lv-buffer branch (possibly better for older models), or on the current crop_rec_4k branch. I don't expect this to affect the results in any way, so just pick one. If it doesn't work, pick the other.

I tried compiling crop_rec_4k with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (config-defines.h) but ran into this:

../../src/boot-hack.c: In function 'copy_and_restart':
../../src/boot-hack.c:109:5: error: implicit declaration of function 'memset64' [-Werror=implicit-function-declaration]
     memset64(0x00D00000, 0x124B1DE0 /* RA(W)VIDEO*/, 0x1FE00000 - 0x00D00000);
     ^
cc1: some warnings being treated as errors
make: *** [boot-hack.o] Error 1


Got past that by copying the memset64 code from the allocate-raw-lv-buffer in stdio.c and mem.h but it wasn't showing the "Raw buffer guess" in the console so I didn't post the build. There's also memset64 in zebra.c that replaces a large block of memset calls and in mem_bench.c that is identical in the two branches.

Bottom line, I gave it a shot but the compile errors should be fixed and tested by someone with more experience than me before the crop_rec_4k branch will be able to run this test (https://www.magiclantern.fm/forum/index.php?topic=5601.msg196632#msg196632).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: saulbass on February 28, 2018, 03:44:55 AM
dfort, I had another go with your "allocate-raw-lv-buffer.2018Feb25.650D104.zip" and it would appear that the problem may lie with the crop.rec module. If I switch on the console and then load mlv-lite and reboot - everything initially looks okay. But when I then load the crop-rec module - both modules appear with a 'Er' sign after rebooting. Similarly if I load the crop-rec first - I get a straight croprec 'Er' on rebooting.

Not sure if this is of any help.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 28, 2018, 01:47:49 PM
Quote from: saulbass on February 28, 2018, 03:44:55 AM
I had another go with your "allocate-raw-lv-buffer.2018Feb25.650D104.zip" and it would appear that the problem may lie with the crop.rec module.

Hum, the crop_rec module isn't in the allocate-raw-lv-buffer branch (https://bitbucket.org/hudson/magic-lantern/src/81b9e49a460df76be65cc7837dd3217827ab9292/modules/?at=allocate-raw-lv-buffer). That probably means I didn't run "make clean" in the source top directory before compiling. There is no reason to load the crop_rec for this test. The only module that should be loaded is mlv_lite. In fact the instructions a1ex gave in the commits page (https://bitbucket.org/hudson/magic-lantern/commits/e799e89f67877825c6a4dc2868f87bed1196432b) indicates that you should be able to do the test without loading any module:

Quote- open the console (Debug menu) and enable something that uses LiveView RAW features (raw video, raw histogram etc)

Let's try it again. I posted a new allocate-raw-lv-buffer build for the 650D with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: saulbass on March 01, 2018, 02:21:15 AM
dfort - I think that this build seems to work now - I've posted some images - hopefully they're what is needed. If not let me know and I will redo.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: saulbass on March 01, 2018, 02:31:30 AM

(https://thumb.ibb.co/deWKFx/IMG_4656.jpg) (https://ibb.co/deWKFx)

(https://thumb.ibb.co/edAqNc/IMG_4654.jpg) (https://ibb.co/edAqNc)

(https://thumb.ibb.co/bZ1mvx/IMG_4651.jpg) (https://ibb.co/bZ1mvx)

(https://thumb.ibb.co/dcH4hc/IMG_4647.jpg) (https://ibb.co/dcH4hc)


4 images from a 650D switching LV on off, photo, 720p, 1080p, 1x5, 5x5.
If you need me to identify which is which let me know.
Using dforts allocate-raw-lv-buffer.2018Feb28.650D104.zip

Hope this helps.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 01, 2018, 09:14:02 PM
Quote from: saulbass on March 01, 2018, 02:31:30 AM
Hope this helps.

It sure does. Thanks for running this test. Turned out pretty much as expected:

Quote from: a1ex on February 01, 2018, 12:51:29 AM
650D: same values as 700D/EOSM
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 05, 2018, 09:09:27 PM
Poking around raw.c so I thought I'd revisit this:

https://www.magiclantern.fm/forum/index.php?topic=5601.msg196632#msg196632

Looks like this test might have already been run on the 70D? It was run recently on the 650D so I'll add it to the Lossless Compression updates for all Digic 5 cameras (https://bitbucket.org/hudson/magic-lantern/pull-requests/909/lossless-compression-updates-for-all-digic/diff) pull request since this is all related. Still missing on the 100D so I posted a wake up call (https://www.magiclantern.fm/forum/index.php?topic=16040.msg198126;topicseen#msg198126) for 100D users.

The 600D and 1100D haven't been tested yet though there are raw_video_10bit_12bit builds for those cameras. These cameras have very limited memory so maybe the smaller DEFAULT_RAW_BUFFER_SIZE might be all these cameras can handle?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 06, 2018, 09:02:41 PM
Quote from: canneloni on February 04, 2018, 01:16:55 PM
Did the test on 100D (hope I did it right) with latest crop_rec build (magiclantern-crop_rec_4k.2018Feb04.100D101).

Your test doesn't look right. Did you compile it yourself with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP like in the instructions (https://bitbucket.org/hudson/magic-lantern/commits/e799e89f67877825c6a4dc2868f87bed1196432b)? Let me know if you need help with that.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 07, 2018, 01:36:59 AM
Quote from: a1ex on February 01, 2018, 12:51:29 AM
The test can be done on either on the allocate-raw-lv-buffer branch (possibly better for older models), or on the current crop_rec_4k branch. I don't expect this to affect the results in any way, so just pick one. If it doesn't work, pick the other.

The 100D isn't in the allocate-raw-lv-buffer branch but when I tried defining CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP on the crop_rec_4k branch I got this:

../../src/boot-hack.c: In function 'copy_and_restart':
../../src/boot-hack.c:105:5: error: implicit declaration of function 'memset64' [-Werror=implicit-function-declaration]
     memset64(0x00D00000, 0x124B1DE0 /* RA(W)VIDEO*/, 0x1FE00000 - 0x00D00000);
     ^
cc1: some warnings being treated as errors
make: *** [boot-hack.o] Error 1


This seems to affect all platforms.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 07, 2018, 06:24:07 AM
Hm, I should undo b831cb1 (since memset64 has slightly different behavior from standard memset, and this code relies on that). BTW, nobody noticed Magic Zoom focus confirmation is broken because of that?! It happened on all experimental builds 2 weeks ago, on all camera models...

(I'm sorry, I no longer use the camera often enough to notice such quirks, but I was hoping the others do...)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Majkovic on March 07, 2018, 03:41:12 PM
I did that tests with dforts build for 100D.
https://drive.google.com/open?id=1wFWKtmashdtGA8ult5H3GYUXbeK8zL7U
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: OlRivrRat on March 07, 2018, 10:51:22 PM
       Link to images of tests > I Hope !

https://photos.app.goo.gl/59Y19hUcS8TtJ1lu1
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 07, 2018, 11:13:33 PM
@Majkovic and @OlRivrRat - Thanks for the tests!

Quote from: a1ex on February 01, 2018, 12:51:29 AM
100D: 46798100-46CC40C4

Wondering if that "FIXME" message anything to worry about? Got the same on the EOSM2 (https://www.magiclantern.fm/forum/index.php?topic=5601.msg197778#msg197778).

Raw buffer guess:  46798100-46CC40C4 (5.2MB, using 9.0MB FIXME)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 07, 2018, 11:36:03 PM
Unless somebody starts porting the 4K and full-res LiveView on these cameras, it won't make any difference. These size checks will never be triggered with default LiveView resolutions; these code changes are more about future-proofing the codebase. Well, some years ago we suspected the 1100D might have a smaller buffer (for some reason, it didn't work, and - not knowing what we were doing - we believed it was that, as it worked fine with a custom buffer). Turns out it was a mistake - the 1100D raw buffer is large enough for all LiveView resolutions (estimated by counting the pixels from the Free Memory screenshot; still need to run the same test to find out the exact value).

100D/EOSM2 x5: 2592 * 1108 * 14/8 = 5025888 (smaller than 5.2MB).

On 700D, these changes were (https://www.magiclantern.fm/forum/index.php?topic=19300.msg197870#msg197870) already (https://www.magiclantern.fm/forum/index.php?topic=19300.msg197702#msg197702) helpful (https://www.magiclantern.fm/forum/index.php?topic=19300.msg197697#msg197697).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 12, 2018, 07:28:40 AM
Battle testing the 10-12bit broken 3xcrop5D2212 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip)  LVState-wip branch from dfort ,  to see what bricks it . As we know the write channel (https://www.magiclantern.fm/forum/index.php?topic=5601.msg194559#msg194559)  I'm using (0x2)has a very small amount of data being written to as a1ex said .
We do know that this build breaks 3x Crop_mode , hence the name . HDMI does not brick it , connected at bootup & after ml running plugged it in & no problems , thou the liveview was slow almost seems to running a half speed (18-12fps) once recording started  liveview returned to normal speed (no delay)(without HDMI connected) (D4/5D2 kills the ml overlays with full mlv+audio , clean output recordable) , did notice to 10x zoom the black levels was bad (pink-ish normally is canon liveview) 5X zoom or 3X Crop mode was normal .

Frame override was enable to 23.976 , so no problems there -- audio was enable to 44.1 kHz , no problems again . The only issue I had that cause a lockup/frozen liveview (needed a battery pull) when I changed the ISO I always use analog pure ISO 100 , 200 , 400 & sometimes 800 not digital pushed or pulled and when I move the dial to change ISO I landed on a digital one (320) & it locked up  :-X quick battery pull and I was back in business . I could reproduce the lockup at every digital ISO , the pure ISO's all work without lockup up to 1600 ISO (yes it pure also , found that out in my disassembly) .  After I got the right ISO (200) I proceeded to set to 12bit @ 1856x1044+audio , started to record a 7+ min talking head , 30GB had a skipped frame at 11299 frames so it stop , I was done just before that by a few seconds . It was a perfect recoding no corruption , the 1st frame had out of sync image bit the rest are all good (perfect) . I didn't use 10bit , thou I did test it and not problems other then the digital ISO thing , I find there's more noise
then I like (I filmed in doors with less then desirable light conditions ) so shadow get noisy when pushed in post.

What does all this mean , not sure but it is usable , but then again it could blowup and you get to kept all the parts  :P :P :P
If anyone want's to use it for a project you must test test test & hope doesn't it lockup (more then likely it would be ok) but Great Caution must be Observed
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 12, 2018, 09:43:26 PM
Worked with andy kh on the 70D and came up with this:

70D: 4B328000 - 4CFFFFFC (repeatable; tested mv1080, x5)

(https://farm5.staticflickr.com/4794/39879568445_6f1e40c73f.jpg) (https://flic.kr/p/23L2dji)

mv1080
(https://farm5.staticflickr.com/4772/25903482097_25bb5f2e4a.jpg) (https://flic.kr/p/Ft17Li)

5x zoom
(https://farm5.staticflickr.com/4796/25903481987_2d9828e21a.jpg) (https://flic.kr/p/Ft17Jp)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 16, 2018, 06:28:00 AM
started to grade the experimental 12bit @1856x1044 1:1 from my 5D2 very nice , I don't what to post a lot of images here (I make post somewhere else)
so I'll post links to the orginal dng's & the export from CS6 A.E. also the graded frame exports for FCPX with
film convert pro , I used the RED MX camera source profile (closest to raw footage) .
By the way it's just my ugly old face -- so you being warned  :P
M11-0006_002688.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M11-0006_002688.dng)  no-grade.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/12bit%205D%20Markii_no-grade.png)  full grade_black backgound (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/12bit%205D%20Markii_1.png)   full grade_white backgound (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/12bit%205D%20Markii.png)

Edit:shot at 70mm (24-70f2.8L canon) F4 200 ISO @ 1/53th second (closest I can get to 1/48) and no preprocessing with mlv_dump option
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 21, 2018, 03:41:34 AM
the images I posted are a little soft so I found a really sharp in focus frame 12bit for 5D2 1:1
Link below to the dng from mlvfs (quick mount win7) so it's 16bit & tif from A.E+ACR ,
when I got more time I'll post a true 12bit for mlv_dump .
M11-0006_006342.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M11-0006_006342.dng)
M11-0006_06342.tif (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M11-0006_06342.tif)
I basically just normalize the .dng in to legal color space from a.e.

Edit: What I find amazing is the absence of any aliasing and moiré at least I don't see any
the other thing I wonder about is the native bit depth from the sensor , is it really 14bit ? or a up-sample 12bit?
If I read it right the digic 3 is native 12bit sooo very interesting , just a thought  ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 26, 2018, 08:16:35 AM
Posted my 12bit experimental & digic4 development update video on 5D MarkII thread if anyone interested to see
12bit 1856x1044+audio 1:1  ;D
https://www.magiclantern.fm/forum/index.php?topic=11205.msg198906#msg198906
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on March 26, 2018, 02:29:00 PM
Nice to see that you are doing some pretty good progress on the Digic 4 cameras, Reddeercity.  And you seem to be pretty close to turning our fantastic Digic 4 cameras into real ML RAW video shooting beasts.  Keep up the good work!

I have a question.  How do you get rid of the ugly moire and aliasing issue in the 12bit 1856x1044+audio 1:1 mode? 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TrEK on March 26, 2018, 09:54:31 PM
Quote from: reddeercity on February 10, 2018, 07:18:16 AM
Yea I know , I still convinced that it will work , I did some test the other day with the 10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip)
That I complied with the change I did (mainly the write channel ) ( the one that a1ex said not to use but works in 1:1)
I what to see what problem or crashes would happen in hopes of maybe seeing something that can help resolve this problem .
So with mlv_lite  without hdmi connected there was no issue crashes etc. ...  but with hdmi in standby ,
I would get the flicker with pink  or out of sequence image(see below) when I start recording raw video the flicker stops and liveview is good to record from hdmi .
I can't say the same as  full mlv + audio I got a memory overflow  ??? didn't matter if hdmi was connected or not .
printed a message on screen "mlv_rec.c line 161" has to do with the SRM memory
maybe audio pushed it over the edge not sure .
(https://thumb.ibb.co/fNDyT7/mlv_lite_10_12bit.png) (https://ibb.co/fNDyT7)  (https://thumb.ibb.co/d5VCo7/VRAM6.png) (https://ibb.co/d5VCo7)

Next I'm going to be looking at big LVState diagram , I did look them over a year ago but really didn't understand that much now I hope I can make head or tales out of it


hello!
i have such trouble too
what you did with it ?>
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TrEK on March 26, 2018, 09:56:09 PM
Quote from: IDA_ML on March 26, 2018, 02:29:00 PM
Nice to see that you are doing some pretty good progress on the Digic 4 cameras, Reddeercity.  And you seem to be pretty close to turning our fantastic Digic 4 cameras into real ML RAW video shooting beasts.  Keep up the good work!

I have a question.  How do you get rid of the ugly moire and aliasing issue in the 12bit 1856x1044+audio 1:1 mode?
where i can download CORRECT 10/12 ML for Canon 5dm2 ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 28, 2018, 05:34:00 AM
Quote from: TrEK on March 26, 2018, 09:56:09 PM
where i can download CORRECT 10/12 ML for Canon 5dm2 ?
search it , it's not hard to find  ;)
hint ,maybe on this page , clue : has something to do with "broken 3xcrop_mode" and you may have or not quoted it in a post  ::)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 28, 2018, 07:20:15 AM
Quote from: IDA_ML on March 26, 2018, 02:29:00 PM
Nice to see that you are doing some pretty good progress on the Digic 4 cameras, Reddeercity.  And you seem to be pretty close to turning our fantastic Digic 4 cameras into real ML RAW video shooting beasts.  Keep up the good work!
It was not just me @dfort had a lot to do with it not to mention a1ex , g3gg0 and other's .
Quote from: IDA_ML on March 26, 2018, 02:29:00 PM
I have a question.  How do you get rid of the ugly moiré and aliasing issue in the 12bit 1856x1044+audio 1:1 mode?
read here , tried to explain it in this post
https://www.magiclantern.fm/forum/index.php?topic=11205.msg198964#msg198964
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TrEK on March 28, 2018, 08:58:08 PM
Quote from: reddeercity on March 28, 2018, 05:34:00 AM
search it , it's not hard to find  ;)
hint ,maybe on this page , clue : has something to do with "broken 3xcrop_mode" and you may have or not quoted it in a post  ::)

sorry but i dont understand

i have raw_video_10bit_12bit_LVState-wip.2018Feb04.5D2212 but its bot work correctly
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 29, 2018, 06:04:56 AM
No that's not it , you already posted it in quote from me .
The reason I'm being a little difficult is I want you search for it , Because it could be dangerous , I have accepted the risk that it could fry the camera or corrupt  the firmware .

Look thought your recent posts (this page)

check post #1623
QuoteAs we know the write channel (https://www.magiclantern.fm/forum/index.php?topic=5601.msg194559#msg194559)  I'm using (0x2)has a very small amount of data being written to as a1ex said
QuoteWhat does all this mean , not sure but it is usable , but then again it could blowup and you get to kept all the parts  :P :P :P
If anyone want's to use it for a project you must test test test & hope doesn't it lockup (more then likely it would be ok) but Great Caution must be Observed
I'm serious about this , no joke !

If all else fails look in my bitbucket downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 01, 2018, 06:51:28 AM
Happy Easter everyone !! :))

Found some time now to get back working on digic4 issue ,
as promised I extracted a few true 12bit frames from my 5D2 on the experimentally 12bit 1:1 mode with the latest mlv_dump on the download page .
These are from the same mlv 30GB file I made . I uploaded the same frames as before but in true 12bit so everyone can compare to the mlvfs 16bit extraction I made before.
Exiftool info , yea it' only 2.8MB pre frame , where as the 14bit is around 3.8MB
ExifTool Version Number         : 10.45
File Name                       : M11-0006_002688.dng
Directory                       : C:/Newfolder
File Size                       : 2.8 MB
File Modification Date/Time     : 2018:03:31 22:05:13-06:00
File Access Date/Time           : 2018:03:31 22:41:30-06:00
File Creation Date/Time         : 2018:03:31 22:41:30-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution Image
Image Width                     : 1856
Image Height                    : 1044
Bits Per Sample                 : 12
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Fill Order                      : Normal
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Strip Offsets                   : 1198
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Rows Per Strip                  : 1044
Strip Byte Counts               : 2906496
Planar Configuration            : Chunky
Software                        : MLV_DUMP CDNG
Modify Date                     : 2018:03:11 00:08:36
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Exposure Time                   : 1/53
F Number                        : 4.0
ISO                             : 200
Sensitivity Type                : ISO Speed
Exif Version                    : 0230
Subject Distance                : 172 m
Focal Length                    : 70.0 mm
Focal Plane X Resolution        : 1283.070596
Focal Plane Y Resolution        : 1302.713987
Focal Plane Resolution Unit     : inches
Lens Model                      : EF24-70mm f/2.8L USM
DNG Version                     : 1.4.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Black Level                     : 448
White Level                     : 4050
Default Scale                   : 1 1
Default Crop Origin             : 0 0
Default Crop Size               : 1856 1044
Color Matrix 1                  : 0.5309 -0.0229 -0.0336 -0.6241 1.3265 0.3337 -
0.0817 0.1215 0.6664
Color Matrix 2                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
As Shot Neutral                 : 0.5017448176 1 0.5119116727
Baseline Exposure               : 0
Camera Serial Number            : ********
Calibration Illuminant 1        : Standard Light A
Calibration Illuminant 2        : D65
Active Area                     : 0 0 1044 1856
Forward Matrix 1                : 0.8924 -0.1041 0.176 0.4351 0.6621 -0.0972 0.0
505 -0.1562 0.9308
Forward Matrix 2                : 0.8924 -0.1041 0.176 0.4351 0.6621 -0.0972 0.0
505 -0.1562 0.9308
Time Codes                      : 00:01:52.00
Frame Rate                      : 23.976
Reel Name                       : M11-0006.mlv
Baseline Exposure Offset        : 0
Aperture                        : 4.0
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 1856x1044
Megapixels                      : 1.9
Scale Factor To 35 mm Equivalent: 1.0
Shutter Speed                   : 1/53
Circle Of Confusion             : 0.029 mm
Depth Of Field                  : inf (33.77 m - inf)
Field Of View                   : 28.0 deg
Focal Length                    : 70.0 mm (35 mm equivalent: 72.1 mm)
Hyperfocal Distance             : 42.00 m
Light Value                     : 8.7


12bit_M11-0006_006342.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/12bit_M11-0006_006342.dng)  12bit_M11-0006_002688.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/12bit_M11-0006_002688.dng) &  corrupted   12bit_M11-0006_000001.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/12bit_M11-0006_000001.dng) , the only one I could find .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 02, 2018, 03:16:24 AM
Further test of the 10-12bit with broken 3xCrop_mode , run some 10bit tests and no problems recording up to 1856x1248 @23.976 1:1 liveview FHD (not 3xcrop_mode)
Looks like it's continuous or very close to it , I stop it after 1200 frames . There is one consisted factor across 10 ,12 & 14bit on this version of bit reduction is every clip has the second frame corrupted no matter what bit depth , yes even 14bit after that all frames are good . I should clarify the broken 3xCrop_mode  ,will it's not really broken you can still use it to get accurate focus just when you record raw video in all bit depths every other frame is frozen with movement in the other alternating frames .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 06, 2018, 03:50:08 AM
trying some different write channels on the D4/5d2 (10-12bit)
0x03 , 0x04 , 0x05 , 0x10 , 0x12 , 0x13 , 0x16
All either error out , (raw detect error) or have corruption in liveview
Still the only one that works so far is 0x02
hoping to stumble across a channel that 3xcrop works with FHD .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 07, 2018, 06:08:17 AM
I see there's being downloads of the 10-12bit broken 3xcrop_mode with 0x02 write channel (which is not totally safe)

(https://preview.ibb.co/kdQLzx/10_12bit_downloads_1.png) (https://ibb.co/m3JvXH)
(https://preview.ibb.co/no1FXH/10_12bit_downloads.png) (https://ibb.co/iwXNCH)

Can these people give some feed back please , I like to know about crashes/lockups and anything else of interest .
how is the 10bit in all video modes ? same for 12bit ? plus how it react to HDMI , FPS change , Over Ride Lock on or off , 25p , 30p ,
W/B changes , Picture Style liveview changes/adjustments .
What ever brick it , don't need to report about (digital ISO freezes the cam.) when adjusting ,
I know about it (work around ) turn mlv_rec module off reboot then adjust to pure ISO (100,200 etc. ..) .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jpegmasterjesse on April 08, 2018, 04:06:21 AM
I'm using it for 12-bit & dual-iso and haven't had any serious problems beyond the ones you mentioned.  I also haven't tested it in that many conditions yet - I'll try HDMI soon and report back.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 09, 2018, 05:37:14 AM
Thanks , that sound good.
Quote from: jpegmasterjesse on April 08, 2018, 04:06:21 AM
I'm using it for 12-bit & dual-iso
Never thought of trying with Dual ISO video -- you do mean video right ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jpegmasterjesse on April 09, 2018, 06:03:40 AM
Yes - and the MLVapp developers have made it work pretty flawlessly. I can do dual-iso 12bit with darkframe subtraction to get some pretty nice frames.

I know someone questioned why I'd pair 12bit with dual iso as 12bit loses some dynamic range, but I find that to be an ideal combination, especially with continuous shooting being an option at 12.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 09, 2018, 07:42:44 AM
Great to hear ,
Yes , I agree -- the reduce bit depth to 12bit gives cleaner shadow while retaining the hi-lights be on my vision that's recoverable in MLVProducer(the best mlv post app yet) & A.E. ACR
and of course artifact are almost all eliminated if not all ! I find with reduce bits the line skipping seem to be a non issue , this may sound strange ,
I have gut feeling that 14bit=4 trillion shades of color  is just to many shades for canon pixel binding & line skipping on 5D2
where I think 12bit=68 billion shades of colors , sort of dither the  color or soften it . That's one reason why in grading apps
I always reduce contrast & saturation heavily  (can always be adding later in the NLE) .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jai554 on April 14, 2018, 07:35:49 PM
Hi,
I tried to copy the MLV rec module into the latest 60d ML firmware 1.1.2 test build
The module fail to load
Is there anything else need to be copied?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on April 14, 2018, 09:17:05 PM
60D.112 just got ported last night to the unified branch. The changes need to be merged into raw_video_10bit_12bit and compiled on that branch. This is something that we'll get around to eventually but first let's see if passes some basic tests.

You're more than welcome to clone my repository and merge branches. Better yet, get the 60D working on the crop_rec_4k branch along with the sd_uhs module. Wheeeeeee!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TrEK on April 15, 2018, 04:42:25 PM
Quote from: jpegmasterjesse on April 09, 2018, 06:03:40 AM
Yes - and the MLVapp developers have made it work pretty flawlessly. I can do dual-iso 12bit with darkframe subtraction to get some pretty nice frames.

I know someone questioned why I'd pair 12bit with dual iso as 12bit loses some dynamic range, but I find that to be an ideal combination, especially with continuous shooting being an option at 12.

how you use DUAL ISO in video recording ? and how you edit it on postproduction ?
MLVFS will be good for it ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jpegmasterjesse on April 15, 2018, 07:04:12 PM
This: https://drive.google.com/open?id=19ZgqOpGWgs0af6HF0DFAqVh7ZYdacsbg

It's an old module that was being tested on the 5D2.  I don't have any documentation on who created it.

You can replace the old dual-iso module in your modules folder on the card with this one.  No need to reinstall ML. Just drag and replace.

You will probably have varying degrees of luck with different programs in trying to convert and view these easily.  I highly recommend MLVApp.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on April 15, 2018, 09:21:39 PM
Why are we talking about Dual ISO on the 12-bit (and 10-bit) RAW video development discussion?

Let's continue discussing it here: Video - dual iso for 5D2 (https://www.magiclantern.fm/forum/index.php?topic=16854.msg199991#msg199991)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: lyolyalya on April 16, 2018, 06:52:02 PM
sorry for the noob question but what zip i need to use with 500D to test?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on April 29, 2018, 06:10:44 PM
I'm Using the latest build from jan31 for the 6D, and was trying the x5 crop mode while zooming. The display looks fine zoomed in (in grayscale) but when I hit record the display shifts to the left in the LV. I remember seeing this somewhere but I can't seem to find it.

Is this a normal behavior?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on April 29, 2018, 07:45:18 PM
That doesn't sound "normal" and Jan 31 isn't the latest build for the 6D. Try the latest crop_rec_4k build from the experiments download page (https://builds.magiclantern.fm/experiments.html).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on May 01, 2018, 11:40:55 AM
Sorry for the late reply, I meant latest for the 10-12bit Raw video.
I tried to remove all and re-extracting a clean version. It seemed to "fix" it now it doesn't shift to the left, it zooms a bit more when I hit record than what u see on the LV before u do so but at least it's centered.
I did get couple of error msgs but they don't seem to affect anything:
1- I use to get this before, it kinda happen the 1st time when u turn on the can and u record something it says "Failed 'WBAL'. queued: 7 failed:1 (requeued)"

2- this one is new "Raw error" It happened twice but I couldn't replicate it and I'm not sure why it happened. 1st time I saw it was when I was tweaking the FPS override to 24 after 5x zoom crop. But like  I said I could replicate it.

It's kinda hard to debug without any logs. If there's something else I can do to help let me know

Sent from my Pixel using Tapatalk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 02, 2018, 06:26:12 AM
Ok--got it.

I would recommend using the crop_rec_4k builds on the 6D. It has everything that is in the 10/12bit build and a whole lot more. The 6D has relatively recently started working with lossless compression so you might want to try that out too.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on May 02, 2018, 02:41:22 PM
Quote from: dfort on May 02, 2018, 06:26:12 AM
Ok--got it.

I would recommend using the crop_rec_4k builds on the 6D. It has everything that is in the 10/12bit build and a whole lot more. The 6D has relatively recently started working with lossless compression so you might want to try that out too.
Alright I'll give it a try. Thanks again @dfort
Any idea what is this WBAL error?

Sent from my Pixel using Tapatalk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 02, 2018, 04:01:18 PM
Quote from: keepersdungeon on May 02, 2018, 02:41:22 PM
Any idea what is this WBAL error?

Need more information. What are your settings? Are you using mlv_lite? Auto WB? Can you get a screenshot of the error?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on May 03, 2018, 09:41:32 PM
Using mlv_rec, not in auto WB, recording in 12bit, 1824*1094, 24fps, no FPS override, in M mode, that's pretty much it. Sorry the image is a bit blurry but u can read the msg at the bottom.
It happens mostly on the 1st time u start recording after turning on the Cam.
(https://thumb.ibb.co/jZ4ijS/Screenshot_20180503_213521_2.png) (https://ibb.co/jZ4ijS)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 03, 2018, 11:47:04 PM
So this is on the 6D running the latest crop_rec_4k build? We're getting off topic. I'll point to this from the 6D topic because it hasn't been reported on any other camera that I know of.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: keepersdungeon on May 04, 2018, 10:16:23 AM
Quote from: dfort on May 03, 2018, 11:47:04 PM
So this is on the 6D running the latest crop_rec_4k build? We're getting off topic. I'll point to this from the 6D topic because it hasn't been reported on any other camera that I know of.
No this is from the 10-12 bit build of Jan 13. I did not try the crop_rec_4k build yet. I hesitate on posting on the 6D topic or here seen that this is the build I'm using . And my apologies if I went off topic or made it confusing

Sent from my Pixel using Tapatalk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 04, 2018, 05:35:26 PM
You're not off topic if you are reporting an issue with the 10-12 bit branch but could you please try to reproduce the issue using the crop_rec_4k branch?

Another thing you might try is to reset settings to factory defaults and see if the issue is still there.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: blubbblubb on May 06, 2018, 07:27:55 PM
I tried the 10/12 Bit one and i cant reproduce the WBAL error (altough i had to use a lower resolution to record more than just 1-2 seconds), so maybe not a error affecting all 6D's?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zootrope on May 10, 2018, 06:30:55 PM
70D
Maybe some screenshots are useless, i don't know...
(i've used this ML version from dfort : crop_rec_4k.57614b3.2018Mar14.70D112)

Raw Rec @10bit @1080p (1824x1026)

(https://thumb.ibb.co/htczkd/1824x1026_10bit_MEM.jpg) (https://ibb.co/htczkd)


(https://thumb.ibb.co/h62XOy/1824x1026_10bit.jpg) (https://ibb.co/h62XOy)


Raw Rec @10bit @720p

(https://thumb.ibb.co/eLPe3y/1280x720_10bit_MEM.jpg) (https://ibb.co/eLPe3y)


(https://thumb.ibb.co/n1CowJ/1280x720_10bit.jpg) (https://ibb.co/n1CowJ)


Raw Rec @12bit @1080p (1824x1026)

(https://thumb.ibb.co/jwstVd/1824x1026_12bit_MEM.jpg) (https://ibb.co/jwstVd)


(https://thumb.ibb.co/c0a2Oy/1824x1026_12bit.jpg) (https://ibb.co/c0a2Oy)


Raw Rec @12bit @720p

(https://thumb.ibb.co/cg8dwJ/1280x720_12bit_MEM.jpg) (https://ibb.co/cg8dwJ)


(https://thumb.ibb.co/hp97qd/1280x720_12bit.jpg) (https://ibb.co/hp97qd)


Raw Rec @10bit 3x3 @1080p (1824x1026)

(https://thumb.ibb.co/kotiVd/1824x1026_10bit_3x3_MEM.jpg) (https://ibb.co/kotiVd)


(https://thumb.ibb.co/hkKU3y/1824x1026_10bit_3x3.jpg) (https://ibb.co/hkKU3y)


Raw Rec @10bit 3x3 @720p

(https://thumb.ibb.co/f8xMGJ/1280x720_10bit_3x3_MEM.jpg) (https://ibb.co/f8xMGJ)


(https://thumb.ibb.co/dUQK3y/1280x720_10bit_3x3.jpg) (https://ibb.co/dUQK3y)


Raw Rec @12bit 3x3 @1080p (1824x1026)

(https://thumb.ibb.co/giNMGJ/1824x1026_12bit_3x3_MEM.jpg) (https://ibb.co/giNMGJ)


(https://thumb.ibb.co/h41nqd/1824x1026_12bit_3x3.jpg) (https://ibb.co/h41nqd)


Raw Rec @12bit 3x3 @720p

(https://thumb.ibb.co/kVjqAd/1280x720_12bit_3x3_MEM.jpg) (https://ibb.co/kVjqAd)


(https://thumb.ibb.co/hCVVAd/1280x720_12bit_3x3.jpg) (https://ibb.co/hCVVAd)


Raw Rec @10bit x5 @1080p

(https://thumb.ibb.co/cpfQbJ/1920x1080_10bit_x5_MEM.jpg) (https://ibb.co/cpfQbJ)


(https://thumb.ibb.co/fG1j3y/1920x1080_10bit_x5.jpg) (https://ibb.co/fG1j3y)


Raw Rec @10bit x5 @720p

(https://thumb.ibb.co/nFvSOy/1280x720_10bit_x5_MEM.jpg) (https://ibb.co/nFvSOy)


(https://thumb.ibb.co/dh5YwJ/1280x720_10bit_x5.jpg) (https://ibb.co/dh5YwJ)


Raw Rec @12bit x5 @1080p

(https://thumb.ibb.co/irRj3y/1920x1080_12bit_x5_MEM.jpg) (https://ibb.co/irRj3y)


(https://thumb.ibb.co/d6F5Ad/1920x1080_12bit_x5.jpg) (https://ibb.co/d6F5Ad)


Raw Rec @12bit x5 @720p

(https://thumb.ibb.co/dU5SOy/1280x720_12bit_x5_MEM.jpg) (https://ibb.co/dU5SOy)


(https://thumb.ibb.co/fmURGJ/1280x720_12bit_x5.jpg) (https://ibb.co/fmURGJ)

delete my (https://deleteacc.com/g)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 10, 2018, 06:56:16 PM
@zootrope - Not sure what problem you are pointing out. Are you able to record video files with that build and are the MLV files valid? Something you might try is the magiclantern-raw_video_10bit_12bit.2018Jan31.70D112.zip build that is posted on the experiments downloads page (https://builds.magiclantern.fm/experiments.html).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zootrope on May 10, 2018, 08:35:38 PM
Quote from: dfort on May 10, 2018, 06:56:16 PM
@zootrope - Not sure what problem you are pointing out. Are you able to record video files with that build and are the MLV files valid? Something you might try is the magiclantern-raw_video_10bit_12bit.2018Jan31.70D112.zip build that is posted on the experiments downloads page (https://builds.magiclantern.fm/experiments.html).
Maybe i'm doing wrong...
my previous post was a reply to this request : https://www.magiclantern.fm/forum/index.php?topic=5601.msg196632#msg196632

Just wanted to help  :-X
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 10, 2018, 09:02:48 PM
Got it -- Your tests do help. Looks like you did it right.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 17, 2018, 07:30:32 AM
Try to find the default_raw_buffer for 5d2 , compiled the allocate-raw-lv-buffer branch  (https://bitbucket.org/hudson/magic-lantern/commits/e799e89f67877825c6a4dc2868f87bed1196432b)
Quotecompile with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (config-defines.h)
I did this , but I'm not getting the same info as posted here dfort https://www.magiclantern.fm/forum/index.php?topic=5601.msg197778#msg197778

(https://image.ibb.co/j92Qoy/VRAM1_small.png) (https://imgbb.com/)

and this 
(https://image.ibb.co/gPDWTy/VRAM6_small.png) (https://imgbb.com/)
Does this work on digic 4 of is this a digic 5 thing only ?

I record small raw video took some silent picture , the only thing show up is (c17ec) see second screen shot .
I'll look over the code and see if I make a mistake , oh yea I open the console to see what I'm getting
any help would great .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 17, 2018, 08:01:18 AM
from a1ex
https://www.magiclantern.fm/forum/index.php?topic=5601.msg196632#msg196632
QuoteI've got a hypothesis about the raw buffer on EvfState models - full chunks (size = srm_buffer_size) are allocated for LiveView use (SRM_AllocateMemoryResourceForLiveViewYuv) and then divided later; the addresses end up in the ADST_GetMemAddrArray (ADdress STore?) at some point,

would this be the same on d4 -- full chunks (size = srm_buffer_size) ?
in my case
chunk #7 size=1a2bfc0 (c17ec)
chunk #8 size=179ffc0 (c17ec)

so is "c17ec" the srm buffer size or is this nothing to do with it ?

edit: I know now why it didn't work there no raw_slurp enabled in allocate-raw-lv-buffer branch for d4/5d2 , I'll have to manually input the code .
Tomorrow's task :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 22, 2018, 06:44:12 AM
Using my modified 10-12bit (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10_12bit_broken_3xcrop_mode_2017Dec10.5D2212.zip) for 5D2
Magic Lantern Nightly.2017Dec10.5D2212
Camera   : 5D2
Firmware : 212
Changeset: ee95e8825183+ (raw_video_10bit_12bit_LVState-wip)

Shot a 40GB talking head interview 9:50 min.  (13663 frames) 12bit 1856x1044 23.976p+audio 44.1 kHz ,
no problem or issue , didn't have hdmi connected .
set to 800 ISO , lens set to f7 (ef24-70f2.8L) @24mm --  I had a hard time trying to find aliasing patterns
check it out , in the interview I had two ladies , one with striped clothing and looking at it I'm having a hard time
seeing  aliasing patterns , linked file below
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M20-1417_000448.dng
I extracted with the new mlv_dump in the crop_rec_4k branch here (https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/72/artifact/modules/mlv_rec/mlv_dump.exe) -- I exported the frames at the same bit depth as I recorded (12bit) so each frame is 2.77MB
To exact to the correct bit depth with mlv_dump you need to add " --no-fixcp" to option if not they will be exported as 14 or 16bit no matter what bit depth you record at
mlv_dump --dng --no-fixcp M20-1417.MLV
so if you record at 10bit or 12bit this will export to the same bit depth as recorded.

The other good thing is now all my slow CF Card (SanDisk-60MB/s) are now usable :) at a good resolution 10bit 1792x968 @23.976p+audio (write speed needed 49MB/s) continuous
would be nice if we could get other's to help get 10-12bit fully ported to 5d2 and all D4 cams .
wishful thinking :))
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on May 22, 2018, 12:05:17 PM
Excellent progress, Reddeercity!  I can't wait to see 10-12 bit working on the 7D too. 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on May 22, 2018, 12:25:49 PM
Great stuff redderrcity!

If the most recent experiments are yielding [almost] continuous recording in 12bit (but obviously no 3x crop mode) what exactly are the additional steps required to '...get 10-12bit fully ported to 5d2 and all D4 cams'?

Haven't you already achieved the goal?



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 23, 2018, 06:43:54 AM
@IDA_ML , thanks -- I thought 7D was partially working in 1:1 with 10-12bit on the same "LVState-wip" branch ? Maybe look under dfort bitbucket downloads .
If not I can provide a 7d test build with my changes , I don't have a 7d to test and I haven't got qmeu up and running just yet (very close) to test in the emulator.

QuoteIf the most recent experiments are yielding [almost] continuous recording in 12bit (but obviously no 3x crop mode) what exactly are the additional steps required to '...get 10-12bit fully ported to 5d2 and all D4 cams'?
@timbytheriver , thanks
I can't really tell you or any one what steps to exactly take , most of this discover was really just a lucky break and some experiments push it to be some what functional .
If anything there needs to be more reverse engineering  research in to LiveView Sync .
The latest code for 10-12bit depends on "raw_slurp" which is a way to collect all the raw data from  channel"0" and use srm memory pack to write the data
(has 4 different address it writes from to speed up data transfer to the memory card). and at this time it's only fully supported in digic 5  cams .
There a big problem with LV sync ( not sure how to fix) that's what is messing up 3x crop_mode.
QuoteHaven't you already achieved the goal?
No , when there's a nightly build download in https://builds.magiclantern.fm/experiments.html for 10-12bit then I would have reach one of my goals .
There is a few broken thinks that I live with so I can use 10-12bit for production work .
1st: digital ISO (160,300,320,500, etc. ...) will lock up/freeze liveview , needs a battery pull to recover .
The reason (I believe) is the write channel I use 0x02 for "raw_slurp" (further investigation by a1ex noted that there a small amount data written to the same channel for time to time)
( the normal channel is 0x06 for basic ml nightly build in 14bit) this is the only channel that produces clean frames in bit reduction bit in 14bit there are still messed up so
only 10-12bit works no 14bit.
2nd: 3x crop_mode is broken , even in 14bit so all it's good for right now is for checking focus .
and again this is related to Liveview sync being messed up , It's much deeper then I can explain , if you read back a few pages I think you will get the idea .

So once raw_slurp is figured out on the digic 4 cams this will open them up to all the feature that digic5 cams have .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on May 23, 2018, 08:19:53 AM
Quote from: reddeercity on May 23, 2018, 06:43:54 AM
@IDA_ML , thanks -- I thought 7D was partially working in 1:1 with 10-12bit on the same "LVState-wip" branch ? Maybe look under dfort bitbucket downloads .
If not I can provide a 7d test build with my changes , I don't have a 7d to test and I haven't got qmeu up and running just yet (very close) to test in the emulator.

Yes, Dfort almost got it working without earthquake shaking on the 7D but it was not stable enough and 5x-magnification mode was not working on that build at all.

I am traveling right now but if you post a new 7D build with your changes, I will be happy to test when I come back in a week or so.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on May 23, 2018, 11:25:35 AM
Thanks. So sorting raw_slurp on the digic 4 cams is the deal-breaker to getting a stable nightly?

Maybe that could be pinned as a sticky to the head of the 5DMK2 thread so that anyone can see that as a important milestone?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on May 23, 2018, 01:40:45 PM
Reddeercity,

On page 56 in my post 1390 I have provided detailed feedback on how the 7D works with Dfort's build that I mentioned in my previous post - the one without earthquake shaking in the Normal uncropped mode at 10/12 - bits.  I cannot find that build in his repository but I may have it somewhere on my home computer.  Unfortunately, as I said, I am traveling right now and do not have access to it.  If Dfort cannot find it either, I can look for it in a week from now, when I get back home.  Please read on from page 56.  You may find some more useful information for your further work.

Good luck!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on May 23, 2018, 02:02:28 PM
Is this the branch?
https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 23, 2018, 06:43:53 PM
I believe the build is still on my downloads page (http://my%20downloads%20page).

raw_video_10bit_12bit_LVState-wip.2018Feb04.7D203.zip

There are also some older experiments on the reddeercity repository downloads page (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/).

My suggestion, as always, is to set up a development environment and try making some changes. For example, how much should we shave off of the SRM_BUFFER_SIZE in order to get this to work or is that not even necessary? You don't need to know how to code to try different values:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff#chg-src/raw.c
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on June 10, 2018, 02:54:39 PM
Long time no see / hear guys. What´s going on - Any news on the 10/12 bit progress on 5DM2?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on June 11, 2018, 01:19:48 AM
read below  :)) it may make you smile 
https://www.magiclantern.fm/forum/index.php?topic=5601.msg198349#msg198349
https://www.magiclantern.fm/forum/index.php?topic=5601.msg198528#msg198528
https://www.magiclantern.fm/forum/index.php?topic=5601.msg198749#msg198749
https://www.magiclantern.fm/forum/index.php?topic=5601.msg198907#msg198907
https://www.magiclantern.fm/forum/index.php?topic=11205.msg198906#msg198906
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: togg on June 25, 2018, 12:44:48 AM
Where can I find a recap post on 12 bit? I tried to look for it but didn't find it. I'm curious to know how much space is saved and if the image quality is really the same and only noise is taken out.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on July 24, 2018, 04:12:44 PM
Quote from: togg on June 25, 2018, 12:44:48 AM
Where can I find a recap post on 12 bit? I tried to look for it but didn't find it. I'm curious to know how much space is saved and if the image quality is really the same and only noise is taken out.
I hope someone digs up the 50d and tries 12-bit raw on it also. I cannot atm

Sent from my Lenovo TB-8703F using Tapatalk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: olegp2005 on August 07, 2018, 02:34:28 PM
Full-frame live view is not possible technically on 5DM2 (3X)?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vincent pierre on August 30, 2018, 03:58:21 PM
Hi,

Oh my god ! Thank you.

Eos 700d - firmware 1.1.5 - Experimental ML firmware 1.1.5 10-12 bit raw.
I can shoot [email protected] [2.35:1 ratio] in raw 10bits ( and kill pink dots with MLVPRODUCER )

I will test this week end in real life shotting conditions. But... Enormous Thanks !

Amazing Work ! Long life to Magic lantern. And thank you for all this working hours, ML is a genius job.
Thank to the ML community.

Best regards from France
Vince
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on September 11, 2018, 04:08:48 AM
Quote from: olegp2005 on August 07, 2018, 02:34:28 PM
Full-frame live view is not possible technically on 5DM2 (3X)?
This should change your mind (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205614#msg205614) I bet
For full width LiveView:
CMOS[2] = 0x00E
C0F06008: 0x27705DB (only the lowest half appears to matter on this camera)
C0F06084: 0x30037 (copied lowest half from photo mode)
C0F06088: 0x46A0B87 (copied lowest half from photo mode)
=> 5632x1074 in mlv_rec, 12.5 FPS.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 17, 2018, 07:23:52 PM
A tiny present (https://bitbucket.org/hudson/magic-lantern/commits/3e3fac1e9c8e3b2ae956a0613c1e49c705831257) for 5D2 users: 10/12-bit recording working without synchronization issues, from what I could tell. Experimental build (https://builds.magiclantern.fm/experiments.html) available.

I've only tested a short clip at 1080p30, 1880x1056 10-bit, mlv_lite + mlv_play + raw_twk, with and without the linked commit.

Happy anniversary (https://en.wikipedia.org/wiki/Canon_EOS_5D_Mark_II)!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on September 17, 2018, 10:18:42 PM
That's great @a1ex -- thanks , I'll give it a complete test drive today with hdmi devices
Petty cool for a 10 year old camera that stated a whole revolution in Video/film production .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: waza57 on September 18, 2018, 01:00:46 AM
wonderful!

I just tested and, for me, I see 2 big improvements:

1. The preview is much better.
2. 10 and 12 bits works in zoom x1 mode. (until now only worked for me in x5)

I will try to understand how it work and see what it gives in my experimental  crop_rec_4k_5D2 branch.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on September 18, 2018, 03:22:36 AM
I'm pretty sure these are the stubs for the 7D:

/** LiveView RAW patches **/
NSTUB(0xFF280084, StartImagePass_x1_SetEDmac)               /* right before the SetEDmac call from StartPass_x10 CrawAddr / Kind */
NSTUB(0xFF27F43C, StartImagePass_x5_SetEDmac)               /* FIXME: why it fails when the hook is placed on the BL instruction?! */


Now to figure out why it doesn't build on this branch:

/Library/Developer/CommandLineTools/usr/bin/make -C ../../platform/7D_MASTER.203
make[1]: *** No rule to make target `../../src/patch.h', needed by `gdb.o'.  Stop.
make[1]: *** Waiting for unfinished jobs....
[ VERSION  ]   ../../platform/7D_MASTER.203/version.bin
[ CC       ]   master_boot.o
make: *** [all] Error 2
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on September 18, 2018, 04:16:14 AM
Had a quick look at the code and looks like we are using the liveview hack from 3x crop_mode , nice find
Works ok , some little issue -- h264 proxy is enabled , did causes a "raw error"
For kick I tried it and got a this
ML ASSERT:
raw_info.bits_per_pixel == bpp
at ../../src/raw.c:2178 (raw_lv_request_bpp), task raw_rec_task
lv:1 mode:3
raw_rec_task stack: 193098 [193178-192178]
0xUNKNOWN  @ ff8773e0:193170
0x00090404 @ 950e7c:1930f0
0x0004EA38 @ 90484:1930c8
0x0004E378 @ 4ea94:193098
Magic Lantern version : raw_video_10bit_12bit.2018Sep17.5D2212
Mercurial changeset   : 3e3fac1e9c8e (raw_video_10bit_12bit_LVState) tip
Built on 2018-09-17 17:07:46 UTC by jenkins@nightly.
Free Memory  : 124K + 3949K

more then likely this should be disable until fixed , if this is even possible to work on 5d2 .

in 10bit didn't like 30p with "frame over ride" enable , had a very hard time being continues .
without "frame over ride" no real problems at 30p .
HDMI devices are problematic , had my Zacuto Evf with pass thru to the Ninja hard drive recorder 
with pink frame glitchs (looks like too much overhead from the overlayes) in standby not so bad when recording raw video .
If I kill overlays when recording raw a.k.a. clean output the pink glitchs are almost all gone from HDMI preview .
If I force HDMI to "VGA" the pink frame glitchs are gone but can't use the HDMI recorder (can't record SD/VGA)
the Evf work great at "VGA" mode , just like before magic lantern on just canon code .
Nice think about Zacuto Evf , it was design for the 5D Mark ii low res hdmi preview so it as a preset to scale it to full screen .
So we may loose full 1080i HDMI preview ,
@a1ex could we add the "kill overlays" when recording mlv's like the full mlv_rec ? this make the HDMI preview more stable
from my tests .
24p mode is more stable all around with "frame over ride" enabled .
Tried different preview modes , Auto work fine  , Framing with low res preview freezes when record raw but can be over ridden
with the half shutter for live preview -- I thought the low res preview was live , I think in full mlv it is but not a deal breaker .

Hopefully we can enable sound like the 5d3 with mlv_lite .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on September 18, 2018, 09:51:43 AM
Quote from: a1ex on September 17, 2018, 07:23:52 PM
A tiny present (https://bitbucket.org/hudson/magic-lantern/commits/3e3fac1e9c8e3b2ae956a0613c1e49c705831257) for 5D2 users: 10/12-bit recording working without synchronization issues!

@a1ex What a present ! Thanks a lot, I see that your work on dual pixel on 5DIV don't make you forget the old and goody 5DII ! 5D2 users allways need you. Hope your next contribution on 5D2 branch will not be in ten years ! ;)

Thanks reddeercity, waza57, dfort to believe in this old D4 cam ! Keep up the good work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 18, 2018, 09:58:55 AM
Ha, I actually hoped to get arbitrary resolutions (up to full-res LiveView), 720p50/60, CF overclocking, and if all works well, start looking into MJPEG. Unfortunately I've ended up spending an entire week of intensive work just for making the Lua tests (https://bitbucket.org/hudson/magic-lantern/branch/lua_fix) pass on 5D2 and 500D, without breaking newer models...

Had to take a break from that, so I've looked into 5D4 & co.

Then kept bumping into issues with crop_rec (unexpected, as the proof of concept (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205622#msg205622) worked smoothly), ran out of time and committed only what worked reasonably well, i.e. 10/12-bit recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on September 18, 2018, 04:11:41 PM
Thanks for your very detailed answer.
For ML developpers, I understand that roadmap is great and for me too ! but as a 5D2 user, the bigger problem it's exact framing.
I know the ML forum rules and I'm asking nothing to developpers. I want just to dialogue with technical people about hdmi output and liveview freezing.
Could we find more easily the liveview bug if we were independant of hdmi ouput engine by switching it off ? (I don't know how or even if it's possible, i'm not ml developper)
I don't have any background on Edmac, interupts, sync frame etc... but I allways split stuff to make it easier. Since users don't use hdmi output and liveview at the same time, it will be a nice feature to switch off the one we don't want. Just a thought.
I'm just thinking out loud, for developpers is certainly non sense or bullshit but who knows with this ML team ...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 18, 2018, 04:37:40 PM
Sorry, I don't get the question. HDMI functionality is disabled as soon as you unplug the cable.

What LiveView bug are you talking about? Slow preview when you need accurate framing? That one is not affected by HDMI*), and it's a lot more complex than getting the raw image data and saving it to card (long answer (https://www.magiclantern.fm/forum/index.php?topic=19300.msg202052#msg202052)).

*) Well, one could compare the configuration of Canon's image processing modules with or without HDMI to figure out how to change the output size of the live preview. But the current slowdown certainly does not come from ML supporting - to some limited extent - external monitors.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kwstas on September 18, 2018, 04:59:25 PM
Thnx a1ex for your present! Just wanted to mention while LV is faster in previewing, in recording freezes (5x).

Also I was wondering instead of zooming to 5x,if it is possible to have a plain white crop frame enabled for 5x recording but continue to see whole frame as 1x.... something like a Leica rangefinder crop frame..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on September 18, 2018, 05:14:24 PM
@ae1x thanks for the clarification and the long answer ... (a bit hard to understand for me)

Hope one day, it will be a short answer like "liveview never freezes or slowdown !"
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TBar15 on September 20, 2018, 07:56:49 AM
I downloaded the experimental build for the 5D2 linked by @a1ex, but I am having a bit of trouble implementing the 10 or 12 bit modes for the RAW recording. When activating the mlv_rec.mo or mlv_lite.mo, I only see that 14 bit footage is referenced, so I am curious about how to switch to 10 bit or 12 bit. I also am missing the raw_twk.mo module referenced in @a1ex's post in addition to crop_rec.mo; have others reported the same thing? I have been trying RAW recording on my 5D2's for a couple years now, and this looked like a promising result for me to try to adopt. Any help would be appreciated, and I am glad to post any screen information or log files from my specific case if requested.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on September 20, 2018, 02:08:15 PM
 You will find 10 and 12 bit selection under ML menu : MOVIE/RAW Video/Bit Depth.
Hope it helps.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on September 21, 2018, 11:04:18 AM
Hello guys.
Thank you very much for this excellent work on the 10bit 5DM2 version... Work really well.
Is it possible to implement sound into this build?

Thank you very much.
8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on September 21, 2018, 01:04:53 PM
Tried to pattern match stubs for 50D:

/** LiveView RAW patches **/
NSTUB(0xFF1DB254, StartImagePass_x1_SetEDmac)               /* right before the SetEDmac call from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
NSTUB(0xFF1DBCE9, StartImagePass_x5_SetEDmac)               /* FIXME: why it fails when the hook is placed on the BL instruction?! */


Movie records in 10bit but can't be reviewed on camera (black frames) nor with mlRawViewer (artefacts).
I'm not sure about the stubs, looks a bit different in code
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on September 21, 2018, 01:13:36 PM
Developing with dcraw?
dcraw -T input.dng
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 21, 2018, 01:26:10 PM
Stubs are OK, except there's no Thumb bit this time.

You (still) need raw_twk to review 10/12-bit files in camera.

On 5D2, when compiling without CONFIG_EDMAC_RAW_PATCH, I get some sync issues (half of the frame is sometimes from previous frame). Compiling with CONFIG_EDMAC_RAW_PATCH (i.e. default configuration on the raw_video_10bit_12bit_LVState branch) solves this issue, but does not address any other quirks, such as in-camera playback or preview/overlays while recording at lower bit depths.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on September 21, 2018, 03:11:34 PM
Quote from: a1ex on September 21, 2018, 01:26:10 PM
On 5D2, when compiling without CONFIG_EDMAC_RAW_PATCH, I get some sync issues (half of the frame is sometimes from previous frame). Compiling with CONFIG_EDMAC_RAW_PATCH (i.e. default configuration on the raw_video_10bit_12bit_LVState branch) solves this issue, but does not address any other quirks, such as in-camera playback or preview/overlays while recording at lower bit depths.

I get half of the frame issue with CONFIG_EDMAC_RAW_PATCH !
Tried to undefined it and will  run the same way on camera, isn't this strange?

Also Preview of Raw video is set to auto, which will looks like Framing when not recording and Frozen when rec start, but I can set to Realtime without apparently any issues... Why?

Quote from: Danne on September 21, 2018, 01:13:36 PM
Developing with dcraw?
dcraw -T input.dng

Compiled raw_tweak and can playback in camera
Not sure how to use dcraw with mlv, do I need to convert to dng with mlv_dump first?

Tried with MLVFS but CF refuses to cooperate...
So I downloaded MLV App 1.0 and it playback .mlv files like mlv_play
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 21, 2018, 03:21:12 PM
My bad, the correct stubs should start with 0xFF9 instead of 0xFF1. That means, you should patch some address that Canon code is going to execute, not a mirrored copy.

Make sure the hook is actually executing (add some printf, for example).

Preview: the heuristic is probably not aware of 10/12-bit limitations, will check. Real-time preview has incorrect framing in x5.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on September 21, 2018, 04:15:28 PM
Quote from: a1ex on September 21, 2018, 03:21:12 PM
the correct stubs should start with 0xFF9 instead of 0xFF1. That means, you should patch some address that Canon code is going to execute, not a mirrored copy.

Make sure the hook is actually executing (add some printf, for example).

/** LiveView RAW patches **/
NSTUB(0xff9db254, StartImagePass_x1_SetEDmac)               /* right before the SetEDmac call from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
NSTUB(0xff9dbce8, StartImagePass_x5_SetEDmac)               /* FIXME: why it fails when the hook is placed on the BL instruction?! */


Looks good! I didn't tried to set second stub to BL instruction yet.

Will show up 2 patches in PatchMgr Dialog (lv_x1 and lv_x5) and value returned by raw_lv_get_resolution() in raw_lv_setedmac_patch()i s "1"; also no branch due to errors in patch_hook_function()
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on September 24, 2018, 12:23:46 PM
Hi

I definitely have a problem!

I've installed @a1ex 's 'tiny present' 10bit build for 5D2 from https://builds.magiclantern.fm/experiments.html (https://builds.magiclantern.fm/experiments.html)

A) I'm not seeing mlv_rec.mo or mlv_snd.mo in that 5D2 build.

B) I'm not seeing 10/12bit Bit-Depth as an option under ML menu : MOVIE/RAW Video/Bit Depth.

C) Should I be loading raw_twk.mo to enable 10/12bit?

Confused! Am I missing some extra files?


Thanks.
Tim
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on September 24, 2018, 10:18:43 PM
I just did : delete my old ml directory, copy the new one.
Load the mlv_lite + mlv_play + raw_twk modules and don't forget to restart your 5D2 to see the options  MOVIE/RAW Video/Bit Depth.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TBar15 on September 25, 2018, 02:24:37 AM
In my case, I am missing the raw_twk.mo module. Are we needing to actually compile that module for ourselves, or is it supposed to be included in the experimental build? I have tried compiling, but that is spitting back errors, unfortunately.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on September 25, 2018, 04:46:56 AM
 @TBar15 , What are talking about  ? Everything you need is in the build .
https://builds.magiclantern.fm/experiments.html
(https://image.ibb.co/iVW6NU/10_12bit_download_link.png) (https://imgbb.com/)
Just follow the really sample instructions that's being posted
Quote from: benoit on September 24, 2018, 10:18:43 PM
Load the mlv_lite + mlv_play + raw_twk modules and don't forget to restart your 5D2 to see the options  MOVIE/RAW Video/Bit Depth.
How is this not straight forward ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on September 25, 2018, 10:53:41 AM
Thanks @benoit That fixed it!  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on September 25, 2018, 01:10:01 PM
Sorry guys - I have to jump in:
When I load mlv_lite + mlv_play + raw_twk I have no sound.
On the audio tab there is this orange writings: "Sound recording is disabled. Enable it from Canon menu."
But it is enabled in Canon menu

Thank you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 25, 2018, 01:30:57 PM
That message looks outdated; when recording raw video, it should recommend mlv_snd instead.

On this branch, mlv_snd is compatible only with mlv_rec, not mlv_lite.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on September 25, 2018, 04:35:50 PM
OK, I ask differently. Is it possible to record sound with the latest "anniversary build"?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on September 25, 2018, 08:25:53 PM
Yes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Audionut on September 26, 2018, 05:12:41 AM
Quote from: a1ex on September 25, 2018, 01:30:57 PM
On this branch, mlv_snd is compatible only with mlv_rec, not mlv_lite.

When I download the 5D2 build of 10/12-bit RAW video from the experiments page (https://builds.magiclantern.fm/experiments.html), there is no mlv_snd module in that build.
Do we have to drag mlv_snd module from a different build?
Which build do you recommend getting the mlv_snd module from?
Is it possible to have mlv_snd module built in this branch so that the auto generated builds from this branch have the mlv_snd module automagically?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 26, 2018, 09:08:15 AM
Sorry, didn't notice it was missing; previous feedback appears to reference mlv_rec, so I've assumed it was OK.

Fixed, and also included the 50D (mostly untested, except for the low-level notes from aprofiti).

The error message about sound recording is still outdated; feel free to suggest something better.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on September 26, 2018, 10:08:06 AM
@a1ex
Thanks – but I'm now getting the following errors on boot with that build.
(https://s3.eu-west-2.amazonaws.com/images.rivertim.uk/grab.jpg) ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on September 26, 2018, 10:13:02 AM
Please read reply #1708
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on September 26, 2018, 12:33:49 PM
fyi: When I connect my smallHD DP6 I got bad frames every few seconds with mlv_rec + mlv_snd + raw_twk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on September 26, 2018, 02:37:30 PM
Quote from: Walter Schulz on September 26, 2018, 10:13:02 AM
Please read reply #1708

@Walter Schultz I did that. It doesn't appear to refer to my problem at all. Have I missed something?   ::)


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on September 26, 2018, 02:46:27 PM
Quote from: timbytheriver on September 26, 2018, 02:37:30 PM
@Walter Schultz I did that. It doesn't appear to refer to my problem at all. Have I missed something?   ::)

Try mlv_rec and no mlv_lite.
Just load: mlv_rec, mlv_snd and raw_twk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on September 26, 2018, 02:54:02 PM
@ch_d Thanks! But I thought mlv_rec wasn't the 10/12bit ? That seems to have fixed it!  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Audionut on September 27, 2018, 06:24:55 AM
Quote from: a1ex on September 26, 2018, 09:08:15 AM
The error message about sound recording is still outdated; feel free to suggest something better.

May I have a link to the code snippet please?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on September 27, 2018, 06:31:56 AM
Checking the updated 10-12bit 5d2 , mlv_rec + mlv_snd + mlv_play + raw_twk
@24p+ Frame Over ride , everything works, 10 ,12 & 14bit looks good .
HDMI work great @ 1080i while recording raw @ 10bit 1856x1044 , Did notice any pink flickering frames on liveview & HDMI .
Had my Zacuto Evf pass thought to the HDMI Ninja hard drive recorder , Looked perfect ! even if I switch GD to allow for overlays
doesn't miss a beat .
30p is a little more touchy , It has a hard time with Frame Over Ride enabled --
it takes a long time to get to continuous (sometimes is just stays at 9800 frames) 10bit 1856x1044 29.97fps .
Without Frame Over Ride enable it goes continuous pretty quickly .
HDMI with 10bit 30p is a no starter ! Too much overhead to record raw continuously , get around 2500 frames .
12bit 30p is just too much at the current write speed ( maybe in the future it won't be  ;D )

12bit @ 24p is still good to use with Frame Over Ride + HDMI , even had GD allowed for overlays and didn't seem to slow it down  :D
but I only let go for a few minute's .

I did get 2 files with wrong black levels , 1 at 122 (should be 112) & one at 115 , both where at 29.97 fps with HDMI .
All else was fine , Audio is there & good 

One last thing the edmac.mo "wrong api version" or something like that was printed on screen .
Would not load , not big deal though .
Oh and the mlv_play.mo & raw_twk.mo work great by the way  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 27, 2018, 06:25:38 PM
Quote from: Audionut on September 27, 2018, 06:24:55 AM
May I have a link to the code snippet please?

Quote from: ch_d on September 25, 2018, 01:10:01 PM
On the audio tab there is this orange writings: "Sound recording is disabled. Enable it from Canon menu."


cd magic-lantern/
grep -nr "Sound recording is disabled. Enable it from Canon menu."
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on September 27, 2018, 07:57:31 PM
What I have noticed on the latest build is that the batteries are draining very fast.
Regularly I worked with a 2013 version on my 5DM2 and the batteries lasted longer.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on September 27, 2018, 08:03:10 PM
Comparison under identical test conditions, please. For example, camera should be in LiveView with both builds (important!), at the same frame rate, with the same overlays enabled, same LCD brightness, same battery, same charging method and so on.

For each build you are going to test:
- Write down the time it takes for the battery to go from say 90% to say 50%.
- Take a screenshot of the battery indicator in the Debug menu at the end of each test (i.e. when the battery indicator reaches 50%)

Run the test a few times to evaluate the repeatability of the results.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on September 28, 2018, 07:07:19 AM
Quote from: ch_d on September 27, 2018, 07:57:31 PM
What I have noticed on the latest build is that the batteries are draining very fast.
Regularly I worked with a 2013 version on my 5DM2 and the batteries lasted longer.
I would have to disagree with this , It's the total opposite . I very surprised on how long the battery life is with the new core (2015-2018).
I have 3 batteries , when I go to a shoot & I'm always in standby (HDMI feed to Evf->Montior->Hard drive recorder-backup to Prores) scene set up , focus , exposure , etc. ....
Sometimes up to 30min before I start recording raw video and doesn't move on the battery indicator , when filming I can shoot for more then 30min with around 75% battery life left .
So 3 battery will last me around 5 hours with recording raw & standby mode (depending on the situation & temperature or cam)
Even when I'm developing , using tools like adtg_gui.mo , I can sit there and go thought registers for hours (2-3) before I can totally drain a battery .
If anything since 2013 the battery life has increased a great deal , can't give number right now , but if needed I can do some endurance tests to proof it .

Maybe there a problem with the batteries ? how old are they ? are they Canon batteries ?
I seem problems with aftermarket batteries not lasting , also your charger station , how is it -- working correctly , there again is a Canon charger ?
Maybe there's a problem with the chip in the battery and it's not reporting correctly . 
Do you let the battery drain to near "0" ? e.g. 10% before changing or do you re-charge at something like 50% ? that can cause a bad charge memory .
All deep cycle batteries no matter what the application (Camera's , Cell phone's or automotive ) needs to be conditioned to remember the voltage & amperage
Sorry I when off topic here . Just trying help  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on September 28, 2018, 09:53:39 AM
Quote from: reddeercity on September 28, 2018, 08:16:21 AM
5D2212_Video_dual_iso.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5D2212_Video_dual_iso.zip)
I haven't tested it on the latest builds like 10-12bit or mlv_lite  but it work fine with plain vanilla builds 14bit , you may have to use a older build .
On 5D2, I just tried your posted module and a1ex 26Sept release : 10bits - Dual iso - MLV no sound - 3x mode (not crop rec) 1984*1074 @24fps. It works  8) But on another clip, I had to pull off the battery. Thanks every one for dual iso 10bits on 5D2!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on October 05, 2018, 11:30:23 AM
With @a1ex's 10/12bit build working so well on 5D2, is there a reason why we can't get a standard 1080 x 1920 24p preset available on the 5D2? I may well have missed something obvious...  :o
Thanks.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on October 05, 2018, 12:46:20 PM
Thanks for the 50d check-in. I don't have access to my 50d ATM, could someone kindly test on the 50d..

Sent from my Lenovo TB-8703F using Tapatalk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on October 05, 2018, 03:39:12 PM
Quote from: zcream on October 05, 2018, 12:46:20 PM
Thanks for the 50d check-in. I don't have access to my 50d ATM, could someone kindly test on the 50d..

Sent from my Lenovo TB-8703F using Tapatalk


https://bitbucket.org/hudson/magic-lantern/commits/db4ee396f4a261d33688b528c39447682f871a07#general-comments (https://bitbucket.org/hudson/magic-lantern/commits/db4ee396f4a261d33688b528c39447682f871a07#general-comments)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andrew_dotdot on October 10, 2018, 02:49:05 PM
Quote from: a1ex on September 26, 2018, 09:08:15 AM
Sorry, didn't notice it was missing; previous feedback appears to reference mlv_rec, so I've assumed it was OK.

Fixed, and also included the 50D (mostly untested, except for the low-level notes from aprofiti).

The error message about sound recording is still outdated; feel free to suggest something better.

The above post made me hopeful that the new latest 70D package might also now include the sound module, which has always been missing in this branch. Is there a recommended version of the sound module to use for the 70D, or is the sound definitely not working on this camera with this branch?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on October 10, 2018, 04:20:28 PM
Um... I really need a way to fail the build if certain modules are not included. Currently, if some module throws a compile error, the build system assumes it's simply not compatible with the current camera and removes it, with a warning message buried somewhere in the logs (i.e. not visible on the front page).

I don't want to manually "whitelist" modules either, so... here's an idea: fail the build whenever a module that used to work, now fails to compile.

Rough draft of what I think it might work (not tested):

- there should be a platform-specific (!) list of required modules (different wording welcome)
- when running "make zip" from scratch, all modules that happen to build successfully should be marked as "required" (i.e. added to that list)
- "make clean" should not erase that list (that's the whole point - to detect modules that worked in previous builds)
- there should be some way to erase that list (possibly with a short help when the build fails because of some module in that list)
- if some required module would fail to build, fail the entire build
- if some new module is successfully built when running "make zip", auto-add it to the list of required modules
- partial builds (make install ML_MODULES="foo bar baz") should keep working; I use these often, as it compiles much faster that way
- for example: platform/70D.112/Makefile.modules.required (or maybe modules/Makefile.modules.required.70D.112) might contain:

ML_MODULES_REQUIRED ?= \
mlv_lite \
mlv_play \
mlv_rec \
mlv_snd \
        ... (autogenerated list)


Who's going to help? May require a bit of Makefile hacking (something I'm not good at); otherwise, a Python script might do the job, too. You may come up with a different workflow, too, as long as it does the job.

Build updated. For cameras other than 70D, it's exactly the same as before.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on October 22, 2018, 11:04:42 PM
Quote from: reddeercity on September 25, 2018, 04:46:56 AM
@TBar15 , What are talking about  ? Everything you need is in the build .
https://builds.magiclantern.fm/experiments.html
(https://image.ibb.co/iVW6NU/10_12bit_download_link.png) (https://imgbb.com/)
Just follow the really sample instructions that's being postedHow is this not straight forward ?
I'm testing out firmware a lot you guys are doing, I'm wondering are anyone doing professional work with magiclantern-raw_video_10bit_12bit.2018Oct10.5D2212.zip? I didn't get the September 17 version, but  magiclantern-raw_video_10bit_12bit.2018Oct10.5D2212.zip looks stable. Please let me know what you guys think. THANKS FOR ANY RESPONSE
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: banertop on November 15, 2018, 10:54:09 PM
Hi guys,

please, how can i record 2880x1080p with 5d2? Is this possible with 10bit, 12bit build from experimental page (witch i try but no success)?,

or, it is another build? Witch one?

Tnx, and sorry if i asked something that was already asked before....sometimes, it is really hard to find information on forum, there are on different topics.....

I am shooting with 5d2 after a while, so getting in shape again>))

tnx for the help
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on November 15, 2018, 11:01:09 PM
https://www.magiclantern.fm/forum/index.php?topic=19336.msg208394;topicseen#msg208394
and look out for reddeercity's posts.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: banertop on November 16, 2018, 12:10:28 AM
i will.

thank you very much
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 16, 2018, 06:28:27 AM
Did you look here 3K/UHD 5D2 Raw development and Other Digic IV Cams (https://www.magiclantern.fm/forum/index.php?topic=19336.msg182476#msg182476)
It really not that hard to find thing here if you use the search engine , by the way always check the first post of any thread for possible downloads . 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: banertop on November 16, 2018, 08:34:41 PM
yes, i see it...

it is more clear now.....

i bought 5d ii (again) after seeing your YT video about 2880p progress....it is really amazing,,,thanks a lot
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: manudom on December 04, 2018, 10:04:05 PM
It does not work on my 5D3. I tested with 10 bit in crop x5 mode and I only get extremely noisy frames.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on February 18, 2019, 04:32:00 PM
How's development on 7d 10/12 bit?

I'm running 2.0.6 but I am going back to 2.0.3 for more testing purposes and all.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 18, 2019, 09:18:20 PM
What needs to happen on the 7D and other Digic 4 cameras is to find a few stubs like aprofiti did on the 50D (https://bitbucket.org/hudson/magic-lantern/commits/db4ee396f4a261d33688b528c39447682f871a07?at=raw_video_10bit_12bit_LVState). That's a first step. I tried but couldn't find one of them, can't remember which one at the moment.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on February 20, 2019, 05:29:19 PM
& just how can a normy like me do so?

Or is that beyond my capability without knowing the code process and all?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on February 20, 2019, 06:24:36 PM
Quote from: dfort on February 18, 2019, 09:18:20 PM
What needs to happen on the 7D and other Digic 4 cameras is to find a few stubs like aprofiti did on the 50D (https://bitbucket.org/hudson/magic-lantern/commits/db4ee396f4a261d33688b528c39447682f871a07?at=raw_video_10bit_12bit_LVState). That's a first step. I tried but couldn't find one of them, can't remember which one at the moment.
Quote from: a1ex on September 21, 2018, 03:21:12 PM
the correct stubs should start with 0xFF9 instead of 0xFF1. That means, you should patch some address that Canon code is going to execute, not a mirrored copy.

Make sure the hook is actually executing (add some printf, for example).

@dfort Can you try these ones?

NSTUB(0xFF27E674, StartImagePass_x1_SetEDmac)
NSTUB(0xFF27F444, StartImagePass_x5_SetEDmac) 


First one It's a bit different but function called looks likely the same.
Looking into the disassembly I can't find a mirror for both stubs in the 0xFF9 range or similar like on 50D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 20, 2019, 07:54:34 PM
@aprofiti - Thanks for the tips. Couldn't get very far. There's something in the mysterious 7D Master code that is causing a compile error on the raw_video_10bit_12bit_LVState branch.

[ VERSION  ]   ../../platform/7D_MASTER.203/version.bin
[ CC       ]   master_boot.o
master_boot.c: In function 'isrlog':
master_boot.c:173:5: error: implicit declaration of function 'clean_d_cache' [-Werror=implicit-function-declaration]
     clean_d_cache();
     ^
cc1: some warnings being treated as errors
make[1]: *** [master_boot.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on February 20, 2019, 08:02:35 PM
That's real bummer @dfort... Thanks for sharing @aprofiti!

Regardless I bet you this only brought us closer to perfection. Indeed it was well worth the effort. :-X

Perhaps this whole mysterious thing is benefiting all the 7D bodies itself in the long run after all. Ha.

In terms of wear and tear. It's sitting like a lifeless puppy ready to burst out 'n grow w ML yet once again!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on February 20, 2019, 09:48:24 PM
I replaced with _clean_d_cache() and found the reference; error is probably due to some refactor like this commit (https://bitbucket.org/hudson/magic-lantern/commits/4c33f6439858a029301283de9c3f957d7d6fcf21?at=patchmgr).
Tried also with sync_caches() and I had to change to _sync_caches()

But it doesn't compile for 7D (no problems for 50D without any changes needed):

Using ~/gcc-arm-none-eabi-5_4-2016q3/bin/arm-none-eabi-gcc (preferred).
[ VERSION  ]   ../../platform/7D_MASTER.203/version.bin
[ CC       ]   master_boot.o
[ LD       ]   magiclantern
gdb.o: In function `gdb_unarm_bkpt':
gdb.c:(.text+0x2ec): undefined reference to `unpatch_memory'
gdb.o: In function `gdb_arm_bkpt':
gdb.c:(.text+0x380): undefined reference to `patch_instruction'
make: *** [magiclantern] Error 1

It strange because they are defined in patch.c and his header is already included in gdb.c...

@a1ex maybe is there some makefile/external reference/header to update for 7D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on February 21, 2019, 07:43:40 AM
Quote from: DeafEyeJedi on February 20, 2019, 08:02:35 PM
It's sitting like a lifeless puppy ready to burst out 'n grow w ML yet once again!

This same hope makes me keep my 7D too.  I still use it with the legendary Dec. 1-st 2016 Build by Dfort and every time I see this unique beautiful cinematic look of the video coming out of it, I keep saying to myself - no, I will keep this good old boy.  Maybe, some day it will receive the ML functionality it deserves.  In my country, we have a saying: "The hope dies last" ...

A1ex,

You state so often on the forum that you are here to help.  The 7D is one example where your help would be invaluable.  You can hardly imagine how many people you would make happy if you could help Dfort, Aprofiti and other skilled people implement 4k crop recording to the 7D.  I know, this is low-priority for you but sometimes, just a few helpful hints in the right direction can create miracles!  The miracle happened already when you helped Reddeercity revive the 5D2.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on February 21, 2019, 09:26:55 AM
Dec 1st 2016 Build? *EDIT* - it's Dec 3

Looks like I'll be using that one from now on until further notice!

Yes, if the 7D gets some love on the development side, I will gladly be your guinea pig all day.

I help but NOT throw away my 7d or switch out.

I wish the low light was great, but I hardly ever shoot low light, so it's pointless for me.

I don't care for 4k as much as I do 10-12 bit raw recording. I get more footage time out of my CF Card :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 21, 2019, 05:19:23 PM
Quote from: aprofiti on February 20, 2019, 09:48:24 PM
I replaced with _clean_d_cache() and found the reference; error is probably due to some refactor like this commit (https://bitbucket.org/hudson/magic-lantern/commits/4c33f6439858a029301283de9c3f957d7d6fcf21?at=patchmgr).

Same here on platform/7D_MASTER.203/master_boot.c.

Quote from: aprofiti on February 20, 2019, 09:48:24 PM
But it doesn't compile for 7D (no problems for 50D without any changes needed):

Right, it compiles fine on all the other cameras. The problem is in patchmgr so it should be fixed on that branch and re-merged into raw_video_10bit_12bit_LVState. Ran hg bisect on it:

The first bad revision is:
changeset:   16796:ac33688aae07
branch:      patchmgr
parent:      16445:e7a7eadbe2a0
parent:      16512:97b73e7a781e
user:        alex@thinkpad
date:        Wed Jan 24 00:04:28 2018 +0100
summary:     Merged unified into patchmgr


Looks like I've been here before:

https://bitbucket.org/hudson/magic-lantern/commits/ac33688aae076a3c54c3de9f064be54e28e9e324
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 21, 2019, 06:59:05 PM
A little rough around the edges but something to start playing around with on the 7D:

raw_video_10bit_12bit_LVState_7D_experiments (https://bitbucket.org/daniel_fort/magic-lantern/commits/b41b699834cbdbbc2b03b97f9b8cd170236706d6)

Test build on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on February 21, 2019, 07:41:08 PM
Thanks for this @dfort! Finally downgraded my 7D from 2.0.6 back to thee good ole' 2.0.3 and will definitely give these rough edges a little test run.

*EDIT*

Just a friendly reminder to those who may find themselves in a similar situation whereas the MLV files are corrupted while shooting in 2.5K (x5 zoom) while the trick in here is to set Preview to 'REALTIME' while 'AUTO/FRAMING/FREEZE' won't do you any good. Just reminded myself of this upon playing w this build.  ;D

Same goes if you press half-shutter to show 'FRAMING' which will often show a frozen preview from last recorded clip. Must press Canon menu in/out to reset this. Sometimes forcing to turn off/on cam if that doesn't help. Indeed a bit hacky but definitely well worth the effort!

Obviously this is all temporarily solutions for now. Have fun and remember to utilize ETTR as much as possible for best color-noise reduction especially for a 7D (can pull down -2.00 more or less in MLV App) and let's try to kill as many bugs as possible while we're at it!

*EDIT2*

Also noticed a similar pattern with how if one were to press half-shutter to check preview while recording it can cause a corruption (pinkish effect) within MLV take. Just for a slight second (during the moment of pressing) as I also suspect this phenomenon happens while utilizing EOSM 4.5K stuff. Need to investigate this more!  :o
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on February 21, 2019, 08:29:24 PM
Good to see a new update on the 10/12 bit build.

I guess I'll try it out.

I can't do 5x zoom for normal use, so I don't consider it. It's not worth it for me, personally, to do so.
I'd rather 900p 14 bit raw footage with my 7D to make magic.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on February 21, 2019, 08:57:56 PM
Quote from: domo94 on February 21, 2019, 08:29:24 PM
Good to see a new update on the 10/12 bit build.

Perhaps it's been put there for a reason, right?

Quote from: domo94 on February 21, 2019, 08:29:24 PM
I guess I'll try it out.

Hopefully you can help us hunt down some bugs while you're at it.

Quote from: domo94 on February 21, 2019, 08:29:24 PM
I can't do 5x zoom for normal use, so I don't consider it. It's not worth it for me, personally, to do so.
I'd rather 900p 14 bit raw footage with my 7D to make magic.

Well if you call that magic then how do we expect @dfort to create even more magic when we refused to force ourselves to dig through puddle of mud?

Nevertheless... I think it is worth to try and fine-tune this mighty 7D into the 4K_crop_rec experimental branch while we have this rare opportunity.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on February 21, 2019, 09:02:50 PM
I guess so.

I meant that more in the sense of me just not using 10/12 bit.

I will most def help in the development because I love my 7D to death.

I'll run a couple tests today.

Let's see if I can even use the glitchiness of it in order to make some cool in camera visual effects for music videos, lol.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 21, 2019, 10:01:18 PM
Ha ha -- let's call this the Max Headroom build.

Actually, I've seen much worse. Try 10bit 12bit for some lovely pink highlights. These issues have been resolved on other cameras. It is just a matter of searching the forum and commits.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on February 21, 2019, 10:05:41 PM
Running tests right now. The same set up that Jedi just recommended and then some of my own.

Nothing is turning out good.

Best I got was MLV Lite module with 12bit standard zoom video. It was most stable with little jitters and skips and frame corruption.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on February 21, 2019, 10:37:47 PM
Quote from: domo94 on February 21, 2019, 10:05:41 PM
Running tests right now. The same set up that Jedi just recommended and then some of my own.

Nothing is turning out good.

Best I got was MLV Lite module with 12bit standard zoom video. It was most stable with little jitters and skips and frame corruption.

Have you tried resetting Canon menu and custom settings?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on February 21, 2019, 11:59:09 PM
I will reset my settings tonight and start from scratch.

I also notice every once in a while, my camera gets REALLY REALLY slow and laggy.

I'll want to maneuver through menus and selections and the camera would respond like 2-5 seconds later. It varies.

This happens with 2.0.3 for me a lot, almost rarely with 2.0.6 if I remember correctly, but I hardly ever, if not EVER, experienced it on 2.0.6.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 22, 2019, 12:59:54 AM
My 7D.206 port doesn't use the MASTER processor and the experimental 10/12bit also isn't using it because it won't compile. Maybe that makes a difference?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on February 22, 2019, 12:22:32 PM
Today is my lucky day!  Opening this thread, I found that Dfort posted a new Build for the EOS 7D.  I am so happy, Dfort, that you didn't give up on the good old 7D!  Thank you so much!

I did not have time to test for more than a few minutes - had to run for work, but I noticed that you have fixed the most nasty 7D bug for the last two years - the freezing preview screen at 10 and 12 bits and 2,5K which I had to unfreeze by changing the preview twice before starting each new recording.  Now, I have to do this only once, after camera turn on.  Fantastic!  Not quite WYSIWYG due to 5x magnification but I can frame now!

What is not working yet is the earthquake shaking at 10 and 12 bits in the normal 1728x972 mode.  But you really are pretty close to solving that problem too.  Every second frame is broken, the rest is OK.

Now I get continuous recording with sound at 2496x1198/10bit/24fps and also at 1728x972/14-bit in the old MLV mode.  As expected, image quality and cinematic look are gorgeous and MLVApp does a hell of a job processing the files.  This is also evident from DeafEyeJedi's video and his supermodel - the cat, beautifully illustrating what the 7D is up to!  I had almost forgotten how it feels like working with this high dynamic range, gorgeous colors and large processing headroom.

I am sure, if the 7D receives the attention and developers' effort it deserves, it will pay off to all of us.  So, just keep going, Dfort !!!

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 22, 2019, 07:30:02 PM
Not much closer than we were before. It looks like we still haven't found the right stubs. At least now the 7D will compile on the raw_video_10bit_12bit_LVState branch, albeit without compiling 7D_MASTER.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on February 22, 2019, 07:59:02 PM
That is very good. Now IDA_ML or some other energetic 7D user could start firing up adtg_gui and get busy with crop rec presets.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on February 23, 2019, 08:52:26 AM
I have now also tested the "RAW video" mode of the Feb. 21 build.  Unfortunately, things do not work so well here:

Working:
----------
1)  The 1736x976/14bits/24fps choice.  Camera provides about 35 sec. of recording time in that mode.  If you need continuous recording, just use the 1728x972/14bits/24fps choice in the "RAW video (MLV)" mode.  It works very well, also with sound.  No corrupt frames either.

Not working:
--------------
2)  All other selection choices: 1736x976 at 10/12 bits and the 5x zoom mode at 2520x1192 at 10/12/14 bits provide corrupt frames and earthquake shaking.

3)  Trying to record 2496x1196 at 14 bits in both: the RAW video and RAW video (MLV) modes does not stop recording properly, although the MLV file gets recorded.  You need to turn camera off and pull the battery to continue working.

Note:
------
If you add the raw_twk.mo from the Dec. 1, 2016 build to the modules directory and activate it in the Modules menu, you will be able to playback your MLV videos in the camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 25, 2019, 08:18:11 AM
I think I found something that may help with 7d in 10-12bit .
I being investigating canon LV with adtg_gi.mo on my 5d2 ,
I think I have recreated the every other fame corruption plus as @IDA_ML puts it "earthquake shaking"
c0f08188 (raw.width)aka HIV_H_SIZE , can breaks raw buffer if not set to the same as the raw resolution
and can give this  "earthquake" thing .
So maybe check c0f08188 and see what it doing & maybe do a image dump & check the raw buffer size to see if these match
e.g I know on 5d2 in 3xcrop the raw buffer is 2152(H)  & C0f08188=0x907(2311) so if you add the OB area to the raw buffer
2152+160 = 2312 -- so it matches . The question is , does the 7D raw image buffer match the C0f08188 in 10/12bit ?
6 second H264 clip 22MB
EarthQuake-Shaking_M25-0003.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/EarthQuake-Shaking_M25-0003.mov)
MLV - 500MB 1856x1248_29.97
M25-0003.MLV (https://drive.google.com/file/d/1DeBqmp0UFzGdggsua3cjkow2KaWK_gNI/view?usp=sharing)
What I did to recreate the "earthquake shaking" is set c0f08188 from(default) 4d707f7 -> 4d70907 in 1:1 FHD so 2039 to 2311
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on February 25, 2019, 10:13:40 AM
Thank you very much, Reddeercity!  Any hint from knowledgeable and experienced people like you could be very helpful. 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: scrax on February 26, 2019, 12:02:18 PM
Quote from: a1ex on February 01, 2018, 12:51:29 AM
600D: ? (internals like 60D)

I've tried on my 600D, here screen captures for 640x480 and crop modes
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 28, 2019, 03:29:22 AM
@scrax - There's more to the test than that:

Quote from: a1ex on February 01, 2018, 12:51:29 AM
Here's a test I'd like to run on all EvfState models (those with 10/12-bit raw video builds on the Experiments page). Instructions in the commit message (https://bitbucket.org/hudson/magic-lantern/commits/e799e89f67877825c6a4dc2868f87bed1196432b). In addition, I'd like a screenshot of the Free Memory dialog in movie mode. I've ran the test on 60D and 5D3 1.1.3; these are the only EvfState models I own.

@reddeercity - Appreciate the comments about tweaking registers with adtg_gui but that module doesn't build on the raw_video_10bit_12bit_LVState so it will require some more work before we can try that.

I've been pulling out what little hair I have left because I would have sworn that the stubs @aprofiti posted (https://www.magiclantern.fm/forum/index.php?topic=5601.msg212347#msg212347) were working on my 7D. Now it is back to only working in 5x zoom mode which was always sort of almost working anyway even without the patch manager doing its magic.

There's a note about the stubs needing to start at 0xFF9 (https://www.magiclantern.fm/forum/index.php?topic=5601.msg206380#msg206380) but the 7D disassembly doesn't match the mirroring of the 5D2 and 50D which are so far the only two cameras working on that branch--right?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 28, 2019, 07:16:46 AM
@dfort , what about digital poke ?
Before a1ex fixed adtg_gui.mo for d4's (broken from some d5 code etc... ) waza57 &  myself used digital poke to extended raw resolutions  .
This committed db30a11 (https://bitbucket.org/waza57/magic-lantern/commits/db30a11dca7b9e531d69a74e0f5ff9947fe05068) is for enabling digic poke on 5d2 , should work on 7D i would think. I can see if i can compile adtg_gui mo. for 7d on my local source.
I Idea ! , did you try the adtg_gui.mo (https://builds.magiclantern.fm/jenkins/job/iso-research/14/artifact/modules/adtg_gui/adtg_gui.mo) from the https://builds.magiclantern.fm/modules.html download page ?
That's the one I've been using on my 5D2 for the last mouth or so . It should work on top of any build .

Also if you can't get the redirect buffer to work , there's still waza 57 Rom Hack (https://www.magiclantern.fm/forum/index.php?topic=19336.msg210955#msg210955)  to extend resolutions
Food for thought .

Quote from: dfort on February 28, 2019, 03:29:22 AM
...... 5D2 and 50D which are so far the only two cameras working on that branch--right?
Yes
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on February 28, 2019, 01:12:34 PM
Quote from: dfort on February 28, 2019, 03:29:22 AM
There's a note about the subs needing to start at 0xFF9 (https://www.magiclantern.fm/forum/index.php?topic=5601.msg206380#msg206380) but the 7D disassembly doesn't match the mirroring of the 5D2 and 50D which are so far the only two cameras working on that branch--right?
Are we sure that there is a mirroring also on the 7D?

If I look in both 50D and 5D2 disassembly I can find 4 references for the string "StartPass_x1 CrawAddr : %lx / KindOfCraw : %d" / "StartImagePass_x1 CrawAddr : %lx / KindOfCraw : %d", but only 2 for 7D; same for the second stub.

Try inserting some debug messages to check patch execution:

diff --git a/src/patch.c b/src/patch.c
--- a/src/patch.c
+++ b/src/patch.c
@@ -926,6 +926,7 @@

int patch_hook_function(uintptr_t addr, uint32_t orig_instr, patch_hook_function_cbr logging_function, const char * description)
{
+    printf("In patch_hook_function()\n");
     int err = 0;

     /* ensure thread safety */
@@ -944,6 +945,7 @@
     
     if (logging_slot < 0)
     {
+        printf("patch_hook_function() - Error no logging slot available!\n");
         snprintf(last_error, sizeof(last_error), "Patch error at %x (no logging slot)", addr);
         puts(last_error);
         err = E_PATCH_TOO_MANY_PATCHES;
@@ -957,6 +959,7 @@
     if (!check_jump_range((uint32_t) &hook->reloc_insn, (uint32_t) addr + 4) ||
         !check_jump_range((uint32_t) addr,              (uint32_t) hook))
     {
+        printf("patch_hook_function() - Error jump out of range!\n");
         snprintf(last_error, sizeof(last_error), "Patch error at %x (jump out of range)", addr);
         puts(last_error);
         err = E_PATCH_UNKNOWN_ERROR;
@@ -989,12 +992,14 @@

     /* since we have modified some code in RAM, sync the caches */
     sync_caches();
-   
+
+    printf("patch_hook_function() - Patching the original instruction!\n");
     /* patch the original instruction to jump to the logging code */
     err = patch_instruction(addr, orig_instr, B_INSTR(addr, hook), description);
     
     if (err)
     {
+        printf("patch_hook_function() - Error something went wrong?\n");
         /* something went wrong? */
         memset(hook, 0, sizeof(union logging_hook_code));
         goto end;
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -1755,6 +1755,7 @@
     int ok = raw_lv_get_resolution(&width, &height);
     if (ok)
     {
+      //  printf("raw_lv_setedmac_patch() - Updating EDMAC image size\n");
         /* update EDMAC image size */
         int pitch = width * raw_info.bits_per_pixel / 8;
         static struct edmac_info dst_edmac_info;
@@ -2021,8 +2022,11 @@
#ifndef CONFIG_EDMAC_RAW_SLURP
     call("lv_save_raw", 1);
#ifdef CONFIG_EDMAC_RAW_PATCH
+    printf("patch_hook_function() - Trying to patch RAW LV x1\n");
     patch_hook_function((uint32_t) &StartImagePass_x1_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x1");
+    printf("patch_hook_function() - Trying to patch RAW LV x5\n");
     patch_hook_function((uint32_t) &StartImagePass_x5_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x5");
+    printf("patch_hook_function() - Done\n");
#endif
#endif

@@ -2066,6 +2070,7 @@
#ifndef CONFIG_EDMAC_RAW_SLURP
     call("lv_save_raw", 0);
#ifdef CONFIG_EDMAC_RAW_PATCH
+    printf("Unpatching RAW LV x1/x5\n");
     unpatch_memory((uint32_t) &StartImagePass_x1_SetEDmac);
     unpatch_memory((uint32_t) &StartImagePass_x5_SetEDmac);
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 28, 2019, 08:51:32 PM
@aprofiti - Thanks for the patch.

Quote from: aprofiti on February 20, 2019, 06:24:36 PM
@dfort Can you try these ones?

NSTUB(0xFF27E674, StartImagePass_x1_SetEDmac)
NSTUB(0xFF27F444, StartImagePass_x5_SetEDmac) 


(https://farm8.staticflickr.com/7839/33364048028_91540d2571.jpg) (https://flic.kr/p/SQgt6o)

Looks like the "Jump range error" message was already patch.c but I didn't think of opening the console -- Doh!

Ok--so these aren't the stubs we're looking for.

Quote from: aprofiti on February 20, 2019, 06:24:36 PM
First one It's a bit different but function called looks likely the same.
Looking into the disassembly I can't find a mirror for both stubs in the 0xFF9 range or similar like on 50D

Right, on the 7D these stubs show up only once. I did take a close look at them and it seems that the second one (StartImagePass_x5_SetEDmac) can be matched perfectly to what works on the 5D2/50D but the stub you came up with is slightly different. Is there a reason for that? Here's the "perfect" match:

platform/7D.203/stubs.S
NSTUB(0xFF27F43C, StartImagePass_x5_SetEDmac)


Looking at the pieces of the puzzle that fit, don't fit or are missing:

src/raw.c
    printf("patch_hook_function() - Trying to patch RAW LV x5");
    patch_hook_function((uint32_t) &StartImagePass_x5_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x5");


I get that "E3A03202" is the instruction, "mov r3, #536870912; 0x20000000" which is what the StartImagePass_x5_SetEDmac stub is pointing at. I also looked at the patch_hook_function definition to try and figure out what is being passed to that function:

src/patch.h
int patch_hook_function(uintptr_t addr, uint32_t orig_instr, patch_hook_function_cbr logging_function, const char * description);


So--I guess that one looks good but the first stub, StartImagePass_x1_SetEDmac is definitely different. The closest match I could find is the same as the one you found:

platform/7D.203/stubs.S
NSTUB(0xFF27E674, StartImagePass_x1_SetEDmac)


However, the instruction at that address is, "E3A00010" which is "mov r0, #16" so maybe we need to modify the patch_hook_function call to this?

src/raw.c
    printf("patch_hook_function() - Trying to patch RAW LV x1");
    patch_hook_function((uint32_t) &StartImagePass_x1_SetEDmac, 0xE3A00010, raw_lv_setedmac_patch, "RAW LV x1");


So how did that work out? Same errors as before. Does this mean we're patching a mirrored copy and not executable code?

Quote from: a1ex on September 21, 2018, 03:21:12 PM
That means, you should patch some address that Canon code is going to execute, not a mirrored copy.

Where's the code? Show me the code.

Quote from: reddeercity on February 28, 2019, 07:16:46 AM
@dfort , what about digital poke ?
Before a1ex fixed adtg_gui.mo for d4's (broken from some d5 code etc... ) waza57 &  myself used digital poke to extended raw resolutions  .

I'm not extending raw resolutions at this point, simply trying to get 10/12-bit working on the 7D.

Quote from: reddeercity on February 28, 2019, 07:16:46 AM
I Idea ! , did you try the adtg_gui.mo (https://builds.magiclantern.fm/jenkins/job/iso-research/14/artifact/modules/adtg_gui/adtg_gui.mo) from the https://builds.magiclantern.fm/modules.html download page ?

That module comes from the iso-research branch and the 7D isn't supported. I found the stubs:

modules/adtg_gui/adtg_gui.c
    else if (is_camera("7D", "2.0.3"))
    {
        ADTG_WRITE_FUNC = 0xFF2C0944; //"[REG] @@@@@@@@@@@@ Start ADTG[CS:%lx]"
        CMOS_WRITE_FUNC = 0xFF2C0B3C; //"[REG] ############ Start CMOS"
        ENGIO_WRITE_FUNC = 0xFF1F6B20;  // from stubs
        ENG_DRV_OUT_FUNC = 0xFF1F675C;
    }


The 7D_MASTER.203 doesn't build on the iso-research branch so it needs the same Makefile hack (https://bitbucket.org/hudson/magic-lantern/commits/ac33688aae076a3c54c3de9f064be54e28e9e324) I used on the raw_video_10bit_12bit_LVState branch. So now it builds, adtg_gui launches and --- it doesn't work!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 28, 2019, 09:17:28 PM
Jump out of range: see the 1300D thread for some background info. On ARM, you cannot jump more than 32 MB with one instruction; you would have to patch two instructions in the original code (i.e. a use long jump).

On 7D, it's easy to fix - in the dm-spy-experiments branch (where I need to patch stuff for logging purposes) I'm starting ML on this camera with the "classic" method (PR #731). I had some trouble with that PR on 60D, though; need to double-check. The new boot method should fix the patch issues.

[ offtopic: on 1300D, that trick won't work, because of the ROM layout; there we really need to use long jumps; I have a local fix that I need to clean up ]

Unfortunately, adtg_gui is not going to work on 7D after some pattern matching. First, one has to find a way to run code on the master processor starting from autoexec.bin. Next, one has to implement ADTG/CMOS logging hooks on the master processor, and forward all the info about these register writes, back into the slave processor, so they can be displayed on the GUI (and hope it's fast enough to avoid interference with the photo capture process). It's really not trivial; possibly more difficult than porting ML on DIGIC 6/7/8.

Mirroring: all Canon cameras use it, but ROM sizes differ. 5D2 uses 8MB for the main ROM (what we call ROM1); the "slave" side of 7D uses 1 x 16MB for the same ROM. Newer cameras use 32MB for the main ROM. Memory map (including ROM mirrors) is displayed in QEMU at startup.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 28, 2019, 10:37:22 PM
Quote from: a1ex on February 28, 2019, 09:17:28 PM
It's really not trivial; possibly more difficult than porting ML on DIGIC 6/7/8.

Not good news. Especially considering that I still haven't finished porting the EOSM2 which should be easy.

Quote from: a1ex on February 28, 2019, 09:17:28 PM
Memory map (including ROM mirrors) is displayed in QEMU at startup.

You mean like this?

./run_canon_fw.sh 7D,firmware=boot=0 -d debugmsg

DebugMsg=0xFF0776AC (from GDB script)
Lockdown read 1
Lockdown read 1
Lockdown read 0
Lockdown read 0
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
Lockdown read 5
Lockdown read 5
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror
C0000000 - CFFFFFFF: eos.mmio
[EOS] enabling code execution logging.
[EOS] loading './7D/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './7D/ROM1.BIN' to 0xF8000000-0xF8FFFFFF
[MPU] FIXME: using generic MPU spells for 7D.
...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 28, 2019, 10:44:16 PM
Indeed. On 7D you get:


F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror


On 5D2/50D, just noticed I need to fix it (.rom1_size = 0x00800000; at least qemu prints a warning):


F8000000 - F87FFFFF: eos.rom1
F8800000 - F8FFFFFF: eos.rom1_mirror
F9000000 - F97FFFFF: eos.rom1_mirror
F9800000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FA7FFFFF: eos.rom1_mirror
FA800000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FB7FFFFF: eos.rom1_mirror
FB800000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FC7FFFFF: eos.rom1_mirror
FC800000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FD7FFFFF: eos.rom1_mirror
FD800000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FE7FFFFF: eos.rom1_mirror
FE800000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FF7FFFFF: eos.rom1_mirror
FF800000 - FFFFFFFF: eos.rom1_mirror


The last copy of this ROM is actually used by Canon for code execution, so... that's what we need to patch.

According to g3gg0, the main copy of the ROM (the one used for reflashing and other low-level access) is at F8000000, so I've tried to model the memory map in the same way.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on February 28, 2019, 10:59:47 PM
Quote from: dfort on February 28, 2019, 08:51:32 PM
Right, on the 7D these stubs show up only once. I did take a close look at them and it seems that the second one (StartImagePass_x5_SetEDmac) can be matched perfectly to what works on the 5D2/50D but the stub you came up with is slightly different. Is there a reason for that? Here's the "perfect" match:
platform/7D.203/stubs.S
NSTUB(0xFF27F43C, StartImagePass_x5_SetEDmac)

Maybe I mixed the the two when looking again at the address before posting. Need to recheck...

To use as reference... this is what is showing on 50D:

(https://i.ibb.co/xCDqCBJ/VRAM28-2.jpg) (https://ibb.co/xCDqCBJ) (https://i.ibb.co/y48M6wz/VRAM29.jpg) (https://ibb.co/y48M6wz)

Last message in console show up when exiting from LV

Quote from: a1ex on February 28, 2019, 09:17:28 PM
Jump out of range: see the 1300D thread for some background info. On ARM, you cannot jump more than 32 MB with one instruction; you would have to patch two instructions in the original code (i.e. a use long jump).

On 7D, it's easy to fix - in the dm-spy-experiments branch (where I need to patch stuff for logging purposes) I'm starting ML on this camera with the "classic" method (PR #731). I had some trouble with that PR on 60D, though; need to double-check. The new boot method should fix the patch issues.
just to have a better pictures:
Do it need two instructions in ARM mode into one, so 64bit length in total for the long jump instruction?
Is it done automatically by the patchmanger backend or need to manually add a new "instruction patch request" to make it works?
Or Do we need to fork the changes from PR #731 and apply there the patches instead?

Quote from: a1ex on February 28, 2019, 09:17:28 PM
Memory map (including ROM mirrors) is displayed in QEMU at startup.
Yeah, remembered about this, so tried to look at 50D to understand what address range to use, but found that I can't run it anymore...
Something screwed at yesterday after installing exiv2 using packet manager...

./run_canon_fw.sh 50D, firmware=boot=0

DebugMsg=0xFF863B10 (from GDB script)
qemu-system-arm: -M 50D,: drive with bus=0, unit=0 (index=0) exists

Reinstalled QEMU but still there... need to figure out what do do....
Then finally forget to post the message for dfort...

On the PR #731 I read: "enable this on 7D (done, please test), maybe also 50D and 700D"
Is there something I can do?
Testing or looking for something? (in that case need some explanation to understand what to do)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 28, 2019, 11:29:25 PM
If you boot the 7D with the current method (i.e. loading ML in RscMgr memory), ML will be loaded far away from the ROM. In this case, you will need long jumps (two ARM instructions for one jump, i.e. 64 bits or two int32's to patch). No, patch manager won't do that for you yet, but I'm working on it (I've got it halfway working for 1300D).

However, you can boot the 7D with the "classic" method (see the PR). In this case, you no longer need long jumps, as ML will be loaded close to the main ROM (close enough to use short jumps). We've already tested that some time ago, on the dm-spy-experiments branch, and IIRC it worked.

50D: the only benefit from that PR would be some extra general-purpose memory (10MB, see here (https://www.magiclantern.fm/forum/index.php?topic=5071.msg166815#msg166815)). On 700D, there is an extra 4MB block. This extra memory is not essential; one has to make sure that block is really unused by Canon firmware; it's not uncommon for some memory to appear unused in the RscMgr map, but to be actually used under certain conditions. This was the trouble with 60D (and the reason that PR was stalled), so... let's focus on what's actually broken for now.

50D memory map in QEMU: it's the same as 5D2, but I need to fix the code (as the current repo version will display the one for 7D and other newer models).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 01, 2019, 09:12:03 PM
Quote from: a1ex on February 28, 2019, 11:29:25 PM
However, you can boot the 7D with the "classic" method (see the PR).

Just checking -- are you referring to this PR (https://bitbucket.org/hudson/magic-lantern/pull-requests/731/rscmgr-memory-60d-todo-7d-maybe-also-50d/diff)?

Then once I get that working in my raw_video_10bit_12bit_LVState_7D_experiments branch then the stubs I found should work? Looks like they are already pointing to a mirrored copy (0xFF) that is used by Canon for code execution. (or did I misunderstand this?)

/** LiveView RAW patches **/
NSTUB(0xFF27E674, StartImagePass_x1_SetEDmac)
NSTUB(0xFF27F444, StartImagePass_x5_SetEDmac)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 02, 2019, 04:53:22 AM
Yeah, I know--stop asking questions and just do as the man says.

It works.

(https://farm8.staticflickr.com/7815/47252476941_c1f1174742.jpg) (https://flic.kr/p/2eZxi9i)

Had to take into account the different instructions on the 7D stubs:

src/raw.c
#ifndef CONFIG_EDMAC_RAW_SLURP
    call("lv_save_raw", 1);
#ifdef CONFIG_EDMAC_RAW_PATCH
#ifdef CONFIG_7D
    patch_hook_function((uint32_t) &StartImagePass_x1_SetEDmac, 0xE3A00010, raw_lv_setedmac_patch, "RAW LV x1");
#else
    patch_hook_function((uint32_t) &StartImagePass_x1_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x1");
#endif
    patch_hook_function((uint32_t) &StartImagePass_x5_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x5");
#endif
#endif


Note that something like this should also work for the 500D and 550D. I haven't found the stubs on the 60D, 600D or 1100D but I didn't look very hard.

The good news is now it will record 10-bit 12-bit without the earthquake effect or corrupted frames -- minimally tested. However, now zoom mode (a.k.a. 2.5K or whatever) doesn't work. In fact it crashes. Don't know if this is a problem on the 5D2/50D as well or just a 7D issue. Ironically, this was the only setting that worked when patching wasn't working.

Next step is to enable CONFIG_DIGIC_POKE since getting adtg_gui to work on the 7D isn't likely at this point. Looks like @waza57 had to do some extra work on it for the 5D2. Then we can continue this discussion on the 3K/UHD 5D2 Raw development and Other Digic IV Cams (https://www.magiclantern.fm/forum/index.php?topic=19336.0).

In the meantime -- a new 7D test build is on my Bitbucket downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on March 02, 2019, 08:48:04 AM
That's a bummer that it didn't work out in favor for the 2.5K stuff while in 5x zoom. Regardless I'll give this new 7D build a test run and Thanks for this @dfort!

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on March 02, 2019, 09:07:58 PM
Hello all,

I am happy to provide some ffedback on the first significant progress with the EOS 7D for the last two years.  Thanks to Dfort's systematic efforts and helpful hints provided by A1ex, Aprofiti, Reddeercity and others, with today's experimental build, we finally have 10 and 12 bit uncompressed MLV video working.  I would like to thank all these people for making this remarkable progress possible.  Here is what works and what doesn't:

RAW video (MLV):
---------------------
All three bitdepths: 10, 12 and 14 bits uncompressed provide continuous recording with sound at 24fps.  If the preview mode is set to "Canon" in the sub menu, we also have a nice real time WYSIWYG color preview.  Recording is very stable and no corrupt frames are observed.  The maximum frame rate that can be set is 25fps.  If you add the raw_twk.mo module to the Modules folder, in-camera playback is possible.  This module is available in the Dec. 1-st, 2016 build.

For those of you who wish to see how 10-bit uncompressed RAW video from the 7D looks like, here are two short videos shot at 1728x972/24 fps:

https://we.tl/t-vJz7bBt9l4

As I mentioned in my post #1757, at 5x-magnification, the camera can also record in the 2496x1198 at all 10/12/14 bit depths.  At 10 bits/24fps recording is continuous, with sound and there are no corrupt frames in the footage.  Also very stable but prevew is 5x and not WYSIWYG.

RAW video:
-------------
As Dfort mentioned and DeafEyeJedi showed in his video, this mode does not work.  Footage is corrupt and camera crashes ocasionally, hard to reproduce though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 02, 2019, 09:56:33 PM
I want to look a little deeper into why zoom mode worked before patching but not after patching. When I was posting builds a few years ago for testing 10/12bit I wasn't aware back then that several cameras weren't ported to this branch at the time--yet they worked in zoom mode. Now that we have a few LV State, Digic 4 cameras ported maybe they all have this same issue?

Could some 50D and 5D2 users please download the latest 10/12-bit RAW video build from the Experiments downloads page (https://builds.magiclantern.fm/experiments.html) and report back whether or not zoom mode is working on those cameras with the raw_video_10bit_12bit_LVState (https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit_LVState) branch.

Back to the 7D, my development branch has some ugly hacks in it and now other cameras won't compile on it so it is going to take some more work before this can be merged. However, for those who want to keep moving the 7D to the next level I enabled CONFIG_DIGIC_POKE (https://magiclantern.fandom.com/wiki/Register_Map/Brute_Force) on the latest test build and posted it on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). Careful when using it. You might want to read through the 3K/UHD 5D2 Raw development and Other Digic IV Cams (https://www.magiclantern.fm/forum/index.php?topic=19336.0) topic to see which registers to tweak. Yeah it isn't as cool as adtg_gui but it seems to be working on the 7D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 03, 2019, 12:54:36 AM
3x crop_mode 10-12bit has always works since dec/2016 .
yes , 5D2 works with the ML 10-12bit release on the
experimental download page works in 3x crop_mode (5x zoom) as well as 1:1 FHD

You will need 3x crop_mode working for 3k/UHD , pretty much everything is done in 3x crop_mode on d4 cams .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on March 03, 2019, 01:27:37 AM
@dfort Have you already tried to use the "not so good" stub for x5 mode?
Is the different version of patch_hook_function() needed because of the sligthly different disassembly?

Can't test x5 mode on 50D, but I think it should works.
Will try to record a quick video as soon as possible.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 03, 2019, 03:12:57 AM
Quote from: aprofiti on March 03, 2019, 01:27:37 AM
Is the different version of patch_hook_function() needed because of the sligthly different disassembly?

That's right. The 50D and 5D2 have the same instruction for the StartImagePass_x1 stub but the 7D, 500D and 550D have a different instruction so it needs to be adjusted when invoking the patch_hook_function or it will show an error on the console and the patch isn't applied.

Quote from: reddeercity on March 03, 2019, 12:54:36 AM
5D2 works with the ML 10-12bit release on the
experimental download page works in 3x crop_mode (5x zoom) as well as 1:1 FHD

Hum, let's try this again. Took my latest test build, ran Dump DIGIC registers (https://pastebin.com/3yjGQZjw) just to see what registers show up on the log, fired up mlv_rec and what??? It works!!! Tried both mlv_rec and mlv_lite. At one point I did get some corrupt frames with mlv_lite but couldn't replicate it. So it looks like this latest build might be working working. Something that may have made a difference is that with these various branches merged together I now need to run "make clean" in the installer directory or subsequent attempts at compiling fail. Maybe that last test build didn't have good "make" cleaning before compiling.

It isn't perfect, not by a long shot. Turning the camera on and off isn't a problem. Note that in prior attempts it sort of worked when first starting up a virgin build but then showed corrupted frames after subsequent startups. However, now when pulling out the card and reinserting it, the camera won't start up--it needs a battery pull. Well, battery pulls seems to fix a lot of problems but what is the problem here? Maybe it should unpatch on shutdown? IDK.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 03, 2019, 04:55:23 AM
Had a look at the reg's , there seem to line up with 5D2 -- shouldn't  be too hard to port to crop_rec .
the Interesting ones are
c0f06008:  22d022d    *Timer A*
c0f06014:      59b   *Timer B*
c0f0713c:      4cc   *Head Timer 3*
c0f07150:      538  *Head Timer 4*
c0f06084:    10143             
c0f06088:  4ca062f *Raw Resolution Size*
c0f08184:      4c9   *Preview Raw Height*
c0f08188:      9d7   *Preview Raw Total Width+ob area*
c0f08518:  4c909d7 *Preview Raw Height & Width*


raw res
4ca=>1226 (horizontal)
62f=>1583 x2->3166 (vertical) Maybe 3X crop_mode ? not sure on the off set
raw height head timers 4cc->1225 & 538->1336
Wow , there a lot of over head there , on the 5d2 I set
both head timer to the same valve e.g.4cc , it gives higher frame rate on extended res's   
c0f08184 , 8188 & 8518 work together for real time preview
8184 - raw height preview (set to the same as c0f0713c & 7150)
8188 - total preview raw width + OB area  (e,g, 5d2 in 3xcrop_mode offset=160 or OB area)
8518 - preview raw height & width from 8184 & 8188
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: AF-OFF on March 03, 2019, 10:43:04 PM
Quote from: dfort on March 02, 2019, 09:56:33 PM
Could some 50D and 5D2 users please download the latest 10/12-bit RAW video build from the Experiments downloads page (https://builds.magiclantern.fm/experiments.html) and report back whether or not zoom mode is working on those cameras with the raw_video_10bit_12bit_LVState (https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit_LVState) branch.

Yes, works for me on 50D . I can zoom in, and record 2000x1078 rav video , zoom out and record 1584x1056 , both 12bit uncompressed and 24fps.
See screenshot of damaged frame - each zoomed clip has two damged frames at the beginnig, frame 002 and 003.
Magnification (5x 10x) is without pink hue, I have had using another build.



(https://i.ibb.co/r3sQWQ2/50d.jpg) (https://ibb.co/r3sQWQ2)


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aprofiti on March 03, 2019, 11:32:08 PM
Quote from: AF-OFF on March 03, 2019, 10:43:04 PM
Yes, works for me on 50D . I can zoom in, and record 2000x1078 rav video , zoom out and record 1584x1056 , both 12bit uncompressed and 24fps.
See screenshot of damaged frame - each zoomed clip has two damged frames at the beginnig, frame 002 and 003.
Magnification (5x 10x) is without pink hue, I have had using another build.

(https://i.ibb.co/r3sQWQ2/50d.jpg) (https://ibb.co/r3sQWQ2)

Confirmed. Same here with a quick test in 5x mode using mlv_lite.

Tried to record a clip with mlv_rec and it doesn't show broken second and third frames
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 04, 2019, 03:08:23 AM
Thanks--so it looks like the 7D is performing about as well as can be expected at this point. Moving on to the next stage, getting crop_rec working on this camera.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on March 04, 2019, 07:20:30 AM
Quote from: dfort on March 04, 2019, 03:08:23 AM
...so it looks like the 7D is performing about as well as can be expected at this point.

Pretty much...



Quote from: dfort on March 04, 2019, 03:08:23 AM
Moving on to the next stage, getting crop_rec working on this camera.

It's definitely getting warmer...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 04, 2019, 07:44:49 AM
@DeafEyeJedi - can you reproduce the conditions that caused the corrupted frames? Something similar happened here but I couldn't reproduce it. I'm pretty sure it was a reduced bit setting with mlv_lite. Note that both mlv_lite and mlv_rec are working on the test build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: AF-OFF on March 04, 2019, 02:33:29 PM
Quote from: dfort on March 04, 2019, 07:44:49 AM
I'm pretty sure it was a reduced bit setting with mlv_lite. Note that both mlv_lite and mlv_rec are working on the test build.

@dfort same on the 50d, the Raw video module instead of mlv lite, does not produce bad frames in any mode when zoomed , 1984x1079
preview set to canon, when zoomed 5x, when recording very nice bw preview.
mlv lite has color preview. (but 2 bad frames at beginning of each zoomed recording)
really like this bw preview.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 05, 2019, 05:30:21 AM
mlv_Lite has been a problem on d4 cams , I never use it  (besides it not having audio support) .
The pink frames/corruption are a result of system overload (can't keep up)
In mlv_lite you need to kill overlays to stop the pink frame/corruption , the display code is different then mlv_rec
where there's no pink frames/corruption . In mlv_rec , there's code to  kill overlays when recording raw video at least on the 5D2
and you can choose between debug info (record GB/time/buffer load) or a movie icon with time recorded . The overlays take up a lot of resources .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 05, 2019, 06:42:55 AM
Put this in mlv_lite. Should work and bring you black and white preview while recording:
Change:
        (need_for_speed && !get_halfshutter_pressed())
            ? RAW_PREVIEW_GRAY_ULTRA_FAST
            : RAW_PREVIEW_COLOR_HALFRES

to:
        (need_for_speed && !get_halfshutter_pressed())
            ? RAW_PREVIEW_GRAY_ULTRA_FAST
    : cam_5d2 && RAW_IS_RECORDING ? RAW_PREVIEW_GRAY_ULTRA_FAST
            : RAW_PREVIEW_COLOR_HALFRES

Turning off global draw should be easy enough manually but it can be done in mlv_lite too. Also maybe worth to look into reg 7150 and 713c and see if they can be tweaked for corruption.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 05, 2019, 07:16:57 AM
yes but this take CPU cycle , and still a impact to performance .
with "kill Overlays" code it picks up 10-15MB/s (65MB/s to 79MB/s) write speed and uses
Canon Liveview (full Color) so no lag . Also mlv_lite causes hdmi to be unusable on 5D2
nothing but pink & corrupted frames . mlv_lite is really not design for d4 cams but for the much
faster d5 cams .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 05, 2019, 07:01:32 PM
Quote from: reddeercity on March 05, 2019, 07:16:57 AM
...mlv_lite causes hdmi to be unusable on 5D2
nothing but pink & corrupted frames . mlv_lite is really not design for d4 cams but for the much
faster d5 cams .

Maybe -- or maybe it is because what you are using has some experimental work that I did a while back that didn't work properly so I abandoned it. Check out the commit messages on the waza57 repository (https://bitbucket.org/waza57/magic-lantern/commits/all).

(https://farm8.staticflickr.com/7905/46376376485_688325b661.jpg) (https://flic.kr/p/2dE8412)

It might also be because you aren't using the latest version of mlv_lite (https://bitbucket.org/hudson/magic-lantern/commits/all?search=mlv_lite). Look at some of the work that was put into it since the branch you are using was last updated to the main repository:

(https://farm8.staticflickr.com/7825/46567887524_0f3cdd28ac.jpg) (https://flic.kr/p/2dX3Ayu)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 06, 2019, 07:29:44 PM
Quote from: Danne on March 05, 2019, 06:42:55 AM
Put this in mlv_lite...

I tried this:

        (need_for_speed && !get_halfshutter_pressed())
            ? RAW_PREVIEW_GRAY_ULTRA_FAST
            : cam_7d && RAW_IS_RECORDING ? RAW_PREVIEW_GRAY_ULTRA_FAST
            : RAW_PREVIEW_COLOR_HALFRES


Didn't work.

I also did some more testing between mlv_lite and mlv_rec. I'm getting clean video on all settings using mlv_rec but mlv_lite won't work with 10/12bit on zoom mode. There is also a chance of getting corrupt frames on other settings using mlv_lite.

On Digic 5 cameras mlv_lite has evolved and surpassed mlv_rec in pretty much every way but the raw_video_10bit_12bit_LVState branch is using a rather early version of mlv_lite so this isn't surprising. One of the features that would be great to have is lossless compression and only mlv_lite works with that. However, this is yet another challenge for Digic 4 development.

ProcessTwoInTwoOutLosslessPath topic:
Quote from: a1ex on March 06, 2019, 12:10:40 AM
I had many attempts to get it working on DIGIC 4, but didn't document them, because... it just didn't work in LiveView.

Anyway--drifting off topic.

Right now the raw_video_10bit_12bit_LVState branch is working with 5D2 and 50D. My 7D branch is also working but it has some ugly hacks and the other platforms won't compile on it. It should be possible to get 10/12-bit working on all of the other ML supported Digic 4 cameras. Finding the stubs was pretty easy on the 500D and 550D so let's document that:

platform/500D.111/stubs.S
/** LiveView RAW patches **/
NSTUB(0xff1f7154, StartImagePass_x1_SetEDmac)
NSTUB(0xff1f7dc0, StartImagePass_x5_SetEDmac)


platform/550D.109/stubs.S
/** LiveView RAW patches **/
NSTUB(0xff242f04, StartImagePass_x1_SetEDmac)
NSTUB(0xff243cc4, StartImagePass_x5_SetEDmac)


Anyone care to verify that? Test builds on my Bitbucket downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).

How about finding these stubs for the 60D and 600D?

[EDIT] Just remembered that the 60D and 600D are EVF_STATE and not LV_STATE so this doesn't apply. Is 10/12bit already working on these cameras?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 07, 2019, 04:32:19 PM
Camera 500D:
Tested 10bit and 12bit on mlv_rec. Completely scrambled
tested 10bit mlv_lite Completely scrambled

Cam went back into work cupboard. No more tests until monday on my part...

Edit.
600D already working.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 07, 2019, 05:48:42 PM
Hum--I'll double check everything. On Monday could you check if the patches are being applied? I can't get it to show up in QEMU.

I guess there's no point including mlv_lite on these Digic 4, LV State test builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on March 07, 2019, 07:08:23 PM
I have a question to those that may be able to help.  The 7D now provides a nice preview in the magnified RAW video (MLV) mode which makes framing possible.  My question concerns the preview during recording which now is 5x-magnified.  Would it be possible to make it WySiWyG, even if it is only black and white and with a reduced frame rate?  WySiWyG preview works quite well with the 100D.  Maybe, it could be implemented into the 7D too.  If that happens, this would further enhance framing, especially when you move the camera and need precise framing to avoid unwanted objects entering the scene.  I am sure, 7D users will greatly appreciate such a preview enhancement.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 07, 2019, 07:24:11 PM
Mirrored copy, long jumps -- re-reading Reply #1772 (https://www.magiclantern.fm/forum/index.php?topic=5601.msg212695#msg212695) maybe the 500D and 550D need to be booted using the "classic" method like what worked for the 7D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on March 08, 2019, 01:12:41 AM
Quote from: dfort on March 07, 2019, 07:24:11 PM
...maybe the 500D and 550D need to be booted using the "classic" method like what worked for the 7D?

Sounds about right...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on March 08, 2019, 06:20:42 AM
Holy crap,

I got busy for 1-2 weeks and you guys flipped the switch so hard on the 7D.

Y'all are such a blessing! Wow.

Thank you so much for your interest and hard work on the 7d though I know it's a hardcore camera.
I can't wait to start shooting with this camera, it seems the low light on it is better than 14 bit, based on Jedi's videos he uploaded.

Time to run some tests. Having 10-12 bit raw capabilities is RIDICULOUS for this camera. Oh man, the stability and security of recording this camera is going to provide is amazing.
I may not need to sell after all...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on March 08, 2019, 06:58:52 AM
Just ran some slow-mo tests.

Go to Canon menu and drop to 720 fps, shoot at maximum resolution possible, bring aspect ratio down to 2.20:1 / 2.35:1 and you get continuous slow-mo, 10-bit raw footage, no corruptions, no dropped frames, excellent footage.

1600 is max ISO for 7D still. You can bump to 2000 as well, 2500 + is total crap.

Funny though when you get to 6400 ISO, you actually get less grain/artifact than anything below. Would only use in very necessary times where I just NEED some footage of something. or maybe even just risk some footage and do heavy post work to make it look clean.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on March 08, 2019, 09:31:34 AM
Quote from: domo94 on March 08, 2019, 06:20:42 AM
Oh man, the stability and security of recording this camera is going to provide is amazing.  I may not need to sell after all...

Ha, I'd rather sell my entire photo gear but the 7D - never !!!  As far as high ISO performance is concerned, you should try AETTR to see what this beast is really up to.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on March 08, 2019, 09:47:45 AM
Some days I feel like a king with the 7d, others not so much.

I'm still not sure how AETTR works just yet.
Care to explain?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on March 08, 2019, 10:24:47 AM
Just activate it in the Expo menu, adjust it and set triggering to your liking in the sub menu and use it before you shoot a still or video.  It will automatically analyze your scene and expose it to the right without blowing up the highlights.  This will maximise the signal-to-noise ratio (SNR) in the shadows and provide a very clean image.  Works very well.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 08, 2019, 05:21:45 PM
Darn it -- thought I was onto something that would work with the 500D/550D by using RscMgr memory but no such luck (https://bitbucket.org/hudson/magic-lantern/pull-requests/731/rscmgr-memory-60d-todo-7d-maybe-also-50d/diff#comment-94344444). Sorry for the false hope.

Not sure why 10/12-bit isn't working properly on these cameras yet. Seems to me that it should be possible.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on March 08, 2019, 05:30:07 PM
Quote from: domo94 on March 08, 2019, 09:47:45 AM
I'm still not sure how AETTR works just yet.
Care to explain?

Tip: Follow the links in feature matrix and you will be directed to https://wiki.magiclantern.fm/ettr
Happy reading!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jpegmasterjesse on March 08, 2019, 11:16:37 PM
Quote from: dfort on March 08, 2019, 05:21:45 PM
Darn it -- thought I was onto something that would work with the 500D/550D by using RscMgr memory but no such luck (https://bitbucket.org/hudson/magic-lantern/pull-requests/731/rscmgr-memory-60d-todo-7d-maybe-also-50d/diff#comment-94344444). Sorry for the false hope.

Not sure why 10/12-bit isn't working properly on these cameras yet. Seems to me that it should be possible.

It seems to be at the same point that the 5D2 was at a few years ago - 10/12bit was working in 5x mode but not 1:1. I'm still not sure what the most stable version of 10/12 bit is for the 5d2 but maybe the progress there will provide a clue.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 09, 2019, 04:37:39 AM
Quote from: jpegmasterjesse on March 08, 2019, 11:16:37 PM
.......
I'm still not sure what the most stable version of 10/12 bit is for the 5d2 but maybe the progress there will provide a clue.
It been stable since Oct/2018 and it's the same code that I have in the 3k crop_rec for 5d2 , so yea it's stable 
https://builds.magiclantern.fm/jenkins/job/raw_video_10bit_12bit/51/artifact/platform/5D2.212/magiclantern-raw_video_10bit_12bit.2018Oct10.5D2212.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 09, 2019, 06:40:13 AM
Quote from: reddeercity on March 09, 2019, 04:37:39 AM
It been stable since Oct/2018...

Actually just a little before that date. This is the commit (https://bitbucket.org/hudson/magic-lantern/commits/3e3fac1e9c8e3b2ae956a0613c1e49c705831257) that got 10/12 bit working with 1080p on the 5D2. Before that it was the same on all Digic 4 which was that it worked in zoom mode but not in mv1080 or mv720 mode. aprofiti figured it out for the 50D (https://bitbucket.org/hudson/magic-lantern/commits/db4ee396f4a261d33688b528c39447682f871a07?at=raw_video_10bit_12bit_LVState) and I was able to do it for the 7D with a lot of guidance from a1ex and some ugly hacks (https://bitbucket.org/daniel_fort/magic-lantern/commits/b41b699834cbdbbc2b03b97f9b8cd170236706d6?at=raw_video_10bit_12bit_LVState_7D_experiments) by merging in the RscMgr_memory (https://bitbucket.org/hudson/magic-lantern/pull-requests/731/rscmgr-memory-60d-todo-7d-maybe-also-50d/diff) branch so it isn't in the main repository just yet.

It should be possible to get 10/12 bit working on the 500D and 550D which are the remaining Digic 4 LV State cameras but so far nobody has cracked that code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GregMal on March 09, 2019, 01:47:46 PM
Big difference between "should work" and "working" sometimes, right? What seems to be the biggest hurdle do you think?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on March 17, 2019, 01:22:42 AM
Here's a quick test with the 10bit 60fps raw on the Canon 7D at the latest 2.0.3 build.

Shot in 2.35:1 ratio at 59.94 with a 24mm pancake prime lens.

https://vimeo.com/324755693

Colors still look amazing,

some aliasing as usually, chromatic aberration, but nothing that my VAF filter can't fix and tidy up. Mind you, I did NOT shoot with my VAF filter for this. I just ran outside for 5 minutes and did the recording thing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on March 17, 2019, 11:43:12 AM
Quote from: dfort on March 09, 2019, 06:40:13 AM
This is the commit (https://bitbucket.org/hudson/magic-lantern/commits/3e3fac1e9c8e3b2ae956a0613c1e49c705831257) that got 10/12 bit working with 1080p on the 5D2.

!! 1080p on the 5D2?  Did I miss something?  ??? Tried the build above but I'm only able to load 1856 x 1004 max at 10bit. This has always been the case for me. What modules / settings yields a straight 1080p @ 16:9 on the 5D2?

Cheers.

KB 1000x CF card
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: theBilalFakhouri on March 17, 2019, 01:55:29 PM
@timbytheriver

There is no Camera that supported by ML can do Real 1080p except 5D3 , and another camera like 5D2 and 700D the res is a little bit lower, but if you are shooting in x5 mode you can get real 1080p but in crop mode only not the full sensor size.

1080p means a little bit lower resolution in the other cameras except in 5D3 . This is technical AFAIK we can't do real 1080p in full sensor size and this is related to sensors read out and how many mega pixels we have in the sensor.

More accurate way to say what we mean in 1080p in other cameras is to use word "mv1080" as Canon call it maybe; this is the mode when selecting 1080p in Canon Menu and without x5 crop mode but not necessarily the res is exactly 1080p. (H.264 does up scaling to 1080p).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on March 17, 2019, 02:44:35 PM
@theBilalFakhouri

Wow. I get you – but that is confusing!  :P

IMHO a non-standard way of describing a well-known/agreed industry format can lead to lots of confusion. I would have thought most pros believe 1080p to mean 1920x1080. These days 16:9 is the 'mother' ratio from which all the others – 1:85, 2:35 etc, are derived. It's certainly taken years for me to become familiar with the slightly esoteric way in which formats are discussed within the ML family.  ;)

I'd say it could lead to disappointment (especially in new ML adopters) if formats ratios/dimensions are described as, say 1080p, when that is not actually what they deliver – compared to those same descriptions used within the wider cineog community.

On my 5D3 I'm happy to describe my 1920x1080p, 24fps, 16:9 as 1080p – and know that another non-ML user understands what that output means.

On my 5D2 – by my own logic – I'd feel more honest to describe its max 1856x1004, 24fps, 16:9 as 1004p – but it would certainly require some explaining to others; and it doesn't sound quite so impressive.

I'm not complaining – I'm full of praise and gratitude to the ML gods who gave all this to us. Just sharing my opinion – leading possibly to a discussion around a unified and meaningful output naming – understood by all.

Cheers!

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 17, 2019, 02:51:39 PM
Upscale to 1920x1080p. You can't see the difference.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on March 17, 2019, 02:53:53 PM
@Danne. I do – and it still looks superb!  :D But that was not my point (see above).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 18, 2019, 05:36:10 AM
Well, the commit I pointed out (https://bitbucket.org/hudson/magic-lantern/commits/3e3fac1e9c8e3b2ae956a0613c1e49c705831257) does say:

QuoteRaw backend: experimental method for overriding raw stream parameters in Canon firmware (CONFIG_EDMAC_RAW_PATCH)
Stubs for 5D2. Clean footage in 1080p30 10-bit!

Like Bilal said--technically, only the 5D3 can do 1920x1080 raw video using the entire sensor area (3x3 sampling). Other cameras, including the 5D2, can do 1920x1080 by cropping in and using 1x1 sampling (a.k.a. zoom mode). Also note that the 5D3 also does horizontal and vertical pixel binning so it is much less susceptible to aliasing and moiré when using 3x3 sampling.

What made the 5D2 such a revolutionary camera was being able to shoot video (albeit lossy compressed H.264 -- at 30fps when it first came out) on a large sensor so that the depth of field was very shallow compared to the relatively small sensors of the video cameras from that era.

Back on topic, I've got an experimental build that allows 10/12bit reduced bit depth on the 7D but can't figure out a way to get it working on the 500D and 550D. Checking out the experiments download page (https://builds.magiclantern.fm/experiments.html) it looks like those are the only cameras not working with 10/12bit. Well, there isn't a build for the 100D but I know that camera works--it just hasn't been merged into the raw_video_10bit_12bit_LVState yet.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on March 22, 2019, 04:08:40 AM
I'm getting a ton of inconsistency with new 7D 10/12/14 bit RAW build.

I get 2:30 of recording time at 60fps, 1600x408 - 2.35:1 aspect ratio, auto live view, 12 bit raw, (with and without) external monitor connected.

Same thing at 10 bit I get about 1:25.

Ironically 14 bit is the most stable, but no 60fps, but yes 24 fps. 2.35:1 ratio, even up to 16:9 at times.

I'm randomly skipping frames and sometimes RAW module will just start recording and skip a frame right away.

As I'm typing this, I'm getting full continuous 60fps raw at 10bit, same settings, but I chose 'HACKED' preview instead of auto preview.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 22, 2019, 06:28:59 AM
Are you talking about magiclantern-crop_rec_4k_mlv_snd_experiments_waza57.2019Mar21.7D203? That one is the latest mega-merged experimental build. How is it compared to the prior couple of experimental builds that I posted? I'm just doing some quick tests on the 7D before posting.

In case you are on the magiclantern-raw_video_10bit_12bit_LVState build - make sure you are using mlv_rec and not mlv_lite. There have been issues with mlv_lite with Digic 4 LV State cameras on that branch.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on March 22, 2019, 07:18:45 AM
Currently on LV State Feb 21. 2019

There was an updated build? I was not aware of that.

Let me upgrade and see.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IDA_ML on March 22, 2019, 01:26:21 PM
Domo94,

Please try the March 02 version and see how it works for you.  Don't forget to report your findings!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 23, 2019, 02:29:08 AM
Right, let's stay on topic and continue testing the 10/12bit development here. Those changes I made to get the 7D working resulted in a huge changeset (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/28/raw-video-10bit-12bit-lvstate-7d/diff) so it probably won't make it into a pull request in this current state.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 24, 2019, 07:28:49 AM
@a1ex - Found a problem with the 700D.115 10/12-bit RAW video build posted on the experiments downloads page (https://builds.magiclantern.fm/experiments.html). This is recording 10-bit video with mlv_lite but a crash with a similar log happens with mlv_rec. It also saved a COREDUMP.DAT file.

[2] raw_rec_task: NULL PTR (80c3e100,e1a00000)
pc=    c288 lr=    c288 stack=1abdd0+0x1000
entry=abad7c(0)
e1a00000 e59ff014 e59ff014 e59ff014
e59ff014 e1a00000 e59ff010 e59ff010

Magic Lantern version : raw_video_10bit_12bit.2018Oct10.700D115
Mercurial changeset   : db4ee396f4a2 (raw_video_10bit_12bit_LVState) tip
Built on 2018-10-10 14:16:48 UTC by jenkins@nightly.
Free Memory  : 128K + 3254K


I know that I have used this branch before and didn't have that issue so I ran hg bisect and it came up with this:

The first bad revision is:
changeset:   17692:fc5e8aa04abe
branch:      raw_video_10bit_12bit_LVState
user:        alex@thinkpad
date:        Wed Sep 26 08:14:44 2018 +0200
summary:     mlv_rec: fix compilation


Don't see anything there specific to the 700D so maybe some other cameras are affected? Note that this came up while trying to get the 7D working on this branch in a way that won't break any of the other cameras. I'm almost there. Got it working and the EOSM seems to be fine with it. So far it is just the 700D though I haven't tried the 5D3 yet--crash looks kind of scary. Easy to test (if you dare) and see if your camera is affected because builds are posted for every camera except the 7D.

[EDIT] There's also a build for the 700D.114 on the downloads page. Is there a reason for that?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on March 24, 2019, 09:48:48 AM
Ok, so far, I've been shooting at 1600 x 408 at the following settings with mlv_rec:

10-bit
59.94 fps
2.35:1 ratio

Global Draw: Allow
Preview Options: HaCKeD
Status When: Icon

Card Warm Up: 128 MB (I have 128GB CF Card)
Use SRM Job Memory: On

- I believe I changed the SRM and Buffer Options after recording 60fps, so some may not be accurate -

My camera is slow upon startup after I removed the card and disconnected battery for a while. It's super delayed when I move through menus.

Sometimes, when I power off and then on, and I record, I get 'RAW DETECT ERROR' and I have to turn off module and turn it on again for it to work as such:
I first record raw and I skip frames at 2 seconds~
The second time it works fairly well, I believe I skip frames after a while, can't remember.

I also have 'load modules after crash'  enabled. - > *If I have this feature enabled, I think it should have some sort of check in the beginning to see if it crashed or not, if it did, have it purge a memory or something?*

It seems to be some sort of memory issue? It remembers the settings but it cannot function sometimes or properly.
IS IT possible to do a purge of cache or something upon start up? (I'm just jotting possible ideas down for my understanding of how this all works. I have no code experience this far.)
When I also go to make sure my settings are correct when I restart the raw record functionality on the video menu, the word at the bottom of the menu describing each option give a last recorded measure of the raw recording frame rate limit. When I skip frame and go to restart, I notice that the frames change every time. The way the values are measured seems to update every time.

I have a link showing the last recorded clips.

Some have corrupted frames. The subsequent clip has less every time.

The third clip has 1 corrupted frame.

https://drive.google.com/open?id=1SoURFm6HwRrdkQqpGuv3YiU_jpV7L-Z3

~~

Is there anyway to get Magic Zoom to work on an external monitor?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 24, 2019, 11:15:06 AM
Quote from: dfort on March 24, 2019, 07:28:49 AM
raw_rec_task: NULL PTR

Posted a new set of builds, with what I think it might be the fix. I've temporarily merged the "old" (but "clean") raw_video_10bit_12bit branch with new-dryos-task-hooks (a change there might have fixed it; need to refresh my memory). Please report back, as I don't remember being able to reproduce this issue.

If that works, I know what I have to do - merge the qemu branch into mainline, then merge new-dryos-task-hooks. Then lua_fix, but that's not so much about video recording.

Then, I need to do some serious cleanup on the patch manager, as I still don't trust that code for mainline (besides breaking the 7D build, which is rather minor, as the master code is not used at all in regular builds).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 24, 2019, 05:25:59 PM
Quote from: a1ex on March 24, 2019, 11:15:06 AM
If that works, I know what I have to do...

It works. Tested on the 700D and EOSM.

As long as you are merging, would it be possible to include RscMgr_memory in raw_video_10bit_12bit_LVState? That is working great on the 7D as I posted on Reply #1774 (https://www.magiclantern.fm/forum/index.php?topic=5601.msg212733#msg212733) and doesn't seem to break the other cameras. On the 7D I did have to block the master from compiling:

platform/7D.203/Makefile
# all::
# $(MAKE) -C $(PLATFORM_PATH)/7D_MASTER.203


Is this an acceptable temp workaround?

Quote from: domo94 on March 24, 2019, 09:48:48 AM
59.94 fps

I take it you have the Canon menu set for 1280/60fps? That is going to give you some heavy aliasing. If we can ever get the crop_rec module working on the 7D we should be able to do 60fps at 3x3 sampling which will improve the resolution. Well, getting lossless compression working would help too. Getting 10/12-bit working is just one of the steps needed on the 7D.

Title: 8-bit RAW ?
Post by: yourboylloyd on March 26, 2019, 04:18:55 AM
I know that this sounds crazy, but is 8bit RAW possible? Sorry if I sound like a noob.

The best feature of shooting RAW on my 5D2 is the aspect ratio control. I can use a lot more of the sensor than shooting in H.264 allows me to. Shooting in 3:2 and 4:3 allows me more creative control in shooting for instagram/facebook because of the larger vertical resolution these sites are intended for. It also helps when shooting anamorphic because of the 1.33x horizontal squeeze!

If 8-bit raw sounds too weird, then is there a possible way to control aspect ratio when recording in H.264?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on March 26, 2019, 02:11:13 PM
Quote from: yourboylloyd on March 26, 2019, 04:18:55 AM
I know that this sounds crazy, but is 8bit RAW possible? Sorry if I sound like a noob.

The best feature of shooting RAW on my 5D2 is the aspect ratio control. I can use a lot more of the sensor than shooting in H.264 allows me to. Shooting in 3:2 and 4:3 allows me more creative control in shooting for instagram/facebook because of the larger vertical resolution these sites are intended for. It also helps when shooting anamorphic because of the 1.33x horizontal squeeze!

If 8-bit raw sounds too weird, then is there a possible way to control aspect ratio when recording in H.264?

You won't get any benefit from it, the size is pretty much the same as 10 and 12 bit.
Plus I would assume it would result in banding.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TrEK on April 02, 2019, 11:40:49 PM
Hello friends
I see that its available new verision on 10/12 bit ML RAW for Canon 5dmk2 and Canon 5dmk3
Latest Build (2019-03-24 09:58) :D

But where i can watch differencess with previous version ? :-[
thank
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: scrax on April 03, 2019, 12:46:17 AM
Quote from: TrEK on April 02, 2019, 11:40:49 PM
Hello friends
I see that its available new verision on 10/12 bit ML RAW for Canon 5dmk2 and Canon 5dmk3
Latest Build (2019-03-24 09:58) :D

But where i can watch differencess with previous version ? :-[
thank

after the links to download for each camera there is a link to the bitbucket commits (https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit_LVState), just check the date to see what's new from the previous version you are using
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 03, 2019, 06:23:42 AM
No not for 5D2 , there no updated commits for "magiclantern-raw_video_10bit_12bit.2019Mar24.5D2212"
the latest is 2018-09-26 (https://bitbucket.org/hudson/magic-lantern/commits/db4ee396f4a261d33688b528c39447682f871a07?at=raw_video_10bit_12bit_LVState) , so I'm not sure what has changed .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TrEK on April 03, 2019, 10:23:50 PM
Quote from: reddeercity on April 03, 2019, 06:23:42 AM
No not for 5D2 , there no updated commits for "magiclantern-raw_video_10bit_12bit.2019Mar24.5D2212"
the latest is 2018-09-26 (https://bitbucket.org/hudson/magic-lantern/commits/db4ee396f4a261d33688b528c39447682f871a07?at=raw_video_10bit_12bit_LVState) , so I'm not sure what has changed .

[
(https://i.ibb.co/ctXXCWw/1.png) (https://ibb.co/ctXXCWw)

(https://i.ibb.co/ypj3y0j/image.png) (https://ibb.co/ypj3y0j)


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on April 03, 2019, 10:38:20 PM
The only change is reply #1823 (that is, compiling from other branch on newer models; no changes on the repository). Apparently it fixes some null pointer errors on Rebel cameras (that for some reason I could never reproduce).

I doubt they changed anything on 5D2/3, but I do not use these builds, so... YMMV.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TrEK on April 03, 2019, 10:43:36 PM
Quote from: scrax on April 03, 2019, 12:46:17 AM
after the links to download for each camera there is a link to the bitbucket commits (https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit_LVState), just check the date to see what's new from the previous version you are using
friend ... i can not understand.
my current version on Canon 5dmk3 -2018.Oct.10

(https://i.ibb.co/ph93cjd/IMG-0229.jpg) (https://ibb.co/ph93cjd)


but now is available - 2019.Mar.24


where on this link (https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit_LVState) i can see difference between current and new versions ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: scrax on April 03, 2019, 11:35:08 PM
Quote from: TrEK on April 03, 2019, 10:43:36 PM
friend ... i can not understand.
my current version on Canon 5dmk3 -2018.Oct.10

(https://i.ibb.co/ph93cjd/IMG-0229.jpg) (https://ibb.co/ph93cjd)


but now is available - 2019.Mar.24


where on this link (https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit_LVState) i can see difference between current and new versions ?

Alex already answered, there are small changes only for other models, but if you look on the linked pages you should see that last commit (or last changes) is 26 september 2018  and since you version is older it already has all the commits, none new so nothing changed in the code
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 04, 2019, 05:09:46 AM
Quote from: a1ex on April 03, 2019, 10:38:20 PM
The only change is reply #1823 (that is, compiling from other branch on newer models; no changes on the repository). Apparently it fixes some null pointer errors on Rebel cameras (that for some reason I could never reproduce).
I doubt they changed anything on 5D2/3, but I do not use these builds, so... YMMV.
Everything is ok on 5D2 , I can't see anything that different .
tested mlv_lite  & mlv_rec , but mlv_rec is much faster on the 5D2 then mlv_lite .
Mainly because of the different liveview preview & no "Kill Liveview Overlays" like "mlv_rec"
which give about 10MB/s more write speed , best I can do is around 65 MB/s on MLV_lite
On mlv_rec I can get around 75 MB/s write speed .
would be nice to have "Kill Liveview Over lays" in mlv_lite the same as mlv_rec.

edit: tried h264 proxy , started to record a h264 .mov ( had the h264 overlay stuff)
but after a few second the liveview froze , had to do a battery pull .
check the CF card , had a h264.mov there about 18 second clip & the start of a mlv file with "0" data
so it kind of worked just didn't save the raw steam data , I may look in to this at a much later date .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on April 04, 2019, 02:30:46 PM
7D 10-12 bit Mar.2.2019 Build

Had an entire video be corrupted.

I'm still managing the best types of shooting settings.


(https://i.ibb.co/JR54gW6/image.png) (https://ibb.co/JR54gW6)


Shooting 60fps, 1600x408, 10 bit raw.

Hacked Preview output into a Feelworld Monitor.

Anybody need some tests?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TrEK on April 06, 2019, 09:35:19 PM
Quote from: scrax on April 03, 2019, 11:35:08 PM
Alex already answered, there are small changes only for other models, but if you look on the linked pages you should see that last commit (or last changes) is 26 september 2018  and since you version is older it already has all the commits, none new so nothing changed in the code

thank you for answer
but do u know if will improve any new changes for canon 5dmk2/3 in new versions of ML 10/12 bit ?
and tell me pls how i can filter branch with change only for canon 5dmk2/3 or watch it in tree-list


(https://i.ibb.co/DLJ9RPh/image.png) (https://ibb.co/DLJ9RPh)

(https://i.ibb.co/Symj4M0/2.png) (https://ibb.co/Symj4M0)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on April 11, 2019, 08:53:35 PM
Quote from: AF-OFF on March 03, 2019, 10:43:04 PM
Yes, works for me on 50D . I can zoom in, and record 2000x1078 rav video , zoom out and record 1584x1056 , both 12bit uncompressed and 24fps.
See screenshot of damaged frame - each zoomed clip has two damged frames at the beginnig, frame 002 and 003.
Magnification (5x 10x) is without pink hue, I have had using another build.



(https://i.ibb.co/r3sQWQ2/50d.jpg) (https://ibb.co/r3sQWQ2)

Any idea why 50d is not capable of 2.5k in zoom mode? Isn't the chipset similar to 7d and 5d2?

Sent from my Redmi 4A using Tapatalk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Dionisgr on April 30, 2019, 10:45:04 AM
On my Canon 650d, this firmware does not start With standard optics just a black screen, and with manual optics writes an error idle: stack overflow: free=0 used=160 and then does not start!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on April 30, 2019, 02:02:45 PM
@Dionisgr - Which build? Open the autoexec.bin file with a text editor and you should see the changeset and who compiled it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Dionisgr on April 30, 2019, 04:33:02 PM
That's what it says at the beginning of the file!
QuoteMagic Lantern raw_video_10bit_12bit.2019Mar24.650D104
Camera   : 650D
Firmware : 104
Changeset: d318f774b24e+94a19d94813f+ (raw_video_10bit_12bit)
Built on : 2019-03-24 10:05:06 by jenkins@nightly
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 05, 2019, 09:36:31 AM
@Dionisgr - Finally got back to this. I reported a problem back in Reply #1821 (https://www.magiclantern.fm/forum/index.php?topic=5601.msg213871#msg213871) and a1ex made a "temporary fix (https://www.magiclantern.fm/forum/index.php?topic=5601.msg213876#msg213876)" meaning the changes were not committed and pushed. If you look at the changeset: d318f774b24e+94a19d94813f+ the plus sign between two changeset hashes are an indication that there was a merge and the trailing plus sign shows that there were some other changes. If you look at the autoexec.bin file in a text editor you'll also see a rather long "diff" which can be used to patch the source code.

There's more work to be done on the raw_video_10bit_12bit branches so I would recommend using one of the crop_rec_4k branches instead. You can record 10/12bit on the 650D with those branches.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 05, 2019, 09:55:44 AM
That means, whatever caused the error... will remain a mystery. For example, should I consider merging that fix (new-dryos-task-hooks (https://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d)) into mainline, or not yet?

If the issue can be reproduced, the contents of the ML/SETTINGS directory might be enough to trigger it (possibly on other camera models as well, not just on 650D).

Otherwise, the above-mentioned build boots just fine in QEMU, but that's pretty much all I could test.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 05, 2019, 07:59:46 PM
@Dionisgr - If you're still following this please post the contents of your ML/SETTINGS and I'll try to reproduce the error on my 700D.

In addition I posted a new-dryos-task-hooks build for the 650D on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). Appreciate it if you could try it out and post your findings on the pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d/diff#chg-modules/sf_dump/sf_dump.c).

@a1ex - Isn't the problem with new-dryos-task-hooks is that it breaks the 500D and possibly some of the other older cameras?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 05, 2019, 08:11:32 PM
To my knowledge, new-dryos-task-hooks doesn't break anything (besides the above report, which is uncertain), but I didn't run extensive tests either. It should actually fix all these "null pointer" errors on various models, and these changes are compatible with all future models (including M50).

So, after QEMU, it would be the next candidate for merging into mainline, as long as it doesn't break things.

Not sure what you are talking about, regarding 500D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 05, 2019, 09:15:00 PM
Quote from: a1ex on May 05, 2019, 08:11:32 PM
Not sure what you are talking about, regarding 500D.

This:
https://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d/diff#comment-48243615
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 06, 2019, 08:13:31 AM
That should have been fixed in 0391160 (https://bitbucket.org/hudson/magic-lantern/commits/0391160eb560).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 06, 2019, 06:11:01 PM
Quote from: a1ex on May 05, 2019, 09:55:44 AM
...should I consider merging that fix (new-dryos-task-hooks (https://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d)) into mainline, or not yet?

Yes, please. In addition to fixing the "null pointer" issues it is needed for the EOSM2, 1300D and probably some other ports in progress. It would be good to propagate new-dryos-task-hooks throughout the various development branches.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Dionisgr on May 07, 2019, 11:35:15 AM
@dfort @a1ex The link shows all the files in the ML folder https://drive.google.com/drive/folders/1VC4LqKf-IOwIzUBu5L3bSqMNo_TVeYgc?usp=sharing (https://drive.google.com/drive/folders/1VC4LqKf-IOwIzUBu5L3bSqMNo_TVeYgc?usp=sharing)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 07, 2019, 01:34:01 PM
Could not reproduce; it boots just fine in QEMU.

However, you have loaded quite a few modules (some for still photos, others for video); maybe one of them is causing the issue. Delete the SETTINGS directory to start with a fresh configuration; then you'll be able to narrow down and figure out what module is causing the issue.

Was this configuration working with some older build? If yes, which one?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 07, 2019, 03:30:57 PM
Could not reproduce on 700D either. Also noticed that the edmac module should be either updated or removed from the builds posted on the experiments downloads page.

(https://live.staticflickr.com/65535/40829873483_b9da3b8064.jpg) (https://flic.kr/p/25cZLTV)

@Dionisgr - Same suggestion, start with a fresh ML configuration. If the issue persists, clearing the Canon settings sometimes helps.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Dionisgr on May 07, 2019, 08:30:36 PM
This configuration worked with assemblies: magiclantern-Nightly.2017Mar30.650D104 and  magiclantern-Nightly.2018Jul03.650D104
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 07, 2019, 09:05:34 PM
The configuration changed because raw_twk isn't included in those builds.

Remove the ML/SETTINGS folder, enable modules one at a time and maybe we can figure out what is causing the problem.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on May 16, 2019, 06:40:38 PM
Due to the official 7D mk I thread being locked...

I just want to thank @dfort SO MUCH for more updates and work done to the 7D.

I can't wait to download and install the new build for 10-12-14 bit raw recording.

Even with the build from March, this camera is holding up really really strong.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: togg on May 26, 2019, 07:21:50 AM
I'm a little bit lost :) If I want to use mlv lite with sound and compression should I use this one or crop rec from last year? Which one is less bug prone?

edit: ok I tested it and the answer seems to be the crop rec branch? I was just surprised to see that the last update is from last year and this one is more recent.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on May 29, 2019, 01:56:00 AM
Quote from: domo94 on May 16, 2019, 06:40:38 PM
Due to the official 7D mk I thread being locked...

Unlocked--feel free to post on that topic.

Quote from: togg on May 26, 2019, 07:21:50 AM
I was just surprised to see that the last update is from last year and this one is more recent.

The Digic 4 cameras (7D, 5D2, 50D, 500D, 550D, 60D, 600D, 1100D) are not supported in the crop_rec branches. The raw_video_10bit_12bit_LVState adds support for these older cameras. Note that 10/12bit works but not the lossless compression that is incorporated in the crop_rec_4k branches.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Spups on July 08, 2019, 08:52:37 AM
Hi!
First post here, been lurking a few months trying to learn everything I can about ML and I think it's time to show off some 10 bit footage!
A big shoutout to Dfort, thank you for everything you've put into these builds!



Video shot on Canon 7D
Lens was Helios 44-2 + ND
No Grade
Processing was as follows:
7D 10 bit > MLVAPP (LOG-C) > Prores 422 > Adobe Premiere (LOG-C to REC709) > 4k blow up > HEVC

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on July 15, 2019, 06:25:56 PM
@Spups

Nice!

I love it.
I was looking into that lens! How do you like it?
What build did you use and what RAW settings did you record with?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Spups on July 16, 2019, 11:55:48 AM
@domo94

Cheers!

The 44-2 is a very capable lens for video.
Having the stepless aperture ring is nice, and the focus ring is as smooth as butter.
IQ is very good, some of the sharpest old glass in it's price range.
Build quality is outstanding, no rattles or loose parts - you could practically use it as a hammer, lol.

Some things to note:
Stepless aperture is easy to nudge so be aware.
Focus ring has a long travel (270 degrees) so you get a work out pulling focus.
there is an unpainted ring inside the 44-2 that causes flaring wide open, stop down a step and it fixes it.

Overall a very fun lens, with a very fun price tag. I paid £30 for mine and the only time it leaves my 7D is when I need to get close and use my Sigma Superwide II 24mm
I shoot 10 bit @ 23.976p (low jello mode) on DForts may build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: histor on July 16, 2019, 02:56:21 PM
I used 44-2 for a long time on 600d without a problem. Cheap, easy to repair/disassemble if you need.  Great ergonomics. On a full frame it looks differently – corners are never sharp (not good for stitching panoramas, for example), the whole image seems to be much more "artistic". But you need sufficient subject.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on July 19, 2019, 06:22:22 AM
Quote from: Spups on July 16, 2019, 11:55:48 AM
I shoot 10 bit @ 23.976p (low jello mode) on DForts may build.

10 bit and what resolution? I need to know.
What about all the extra settings such as buffer speed method?
That video looked great!

Encounter any problems?!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on July 19, 2019, 07:58:49 AM
Quote from: Spups on July 16, 2019, 11:55:48 AM
The 44-2 is a very capable lens for video.
Having the stepless aperture ring is nice, and the focus ring is as smooth as butter.
IQ is very good, some of the sharpest old glass in it's price range.
Build quality is outstanding, no rattles or loose parts - you could practically use it as a hammer, lol.

Do you need any extra mounts or adapters for this lens to work on the 7d?
Does it work full frame?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Spups on July 20, 2019, 11:25:00 PM
@domo94

Resolution was the maximum for 16:9 capture: 1728x972. This can be increased to 1728 x 1152 if you shoot in 3:2 (max resolution the 7D can achieve without x3 crop AFAIK). I may well shoot in 3:2 for a future video, the cost being some black bars on the sides.
I don't touch any buffer settings as I've not encountered any problems with continous recording. I have however changed the liveview settings to use canons view (this stops magic lantern from putting the liveview to black and white when zooming in to confirm focus)

The 44-2 works perfectly on a full frame body, though don't expect razor sharp corners wide open (even my APS-C gets soft around the edge at F2) so you may find F5.6+ works better if global sharpness is a big concern.
you will need an M42 to canon EF adapter. these can be bought very inexpensively, mine was made by Neewer.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KirbyLikes525 on August 05, 2019, 03:29:57 PM
Quote from: Spups on July 08, 2019, 08:52:37 AM
Processing was as follows:
7D 10 bit > MLVAPP (LOG-C) > Prores 422 > Adobe Premiere (LOG-C to REC709) > 4k blow up > HEVC

Just curious why you didn't MLVAPP (Rec.709) > Prores 422 instead?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: masc on August 05, 2019, 08:12:47 PM
Quote from: KirbyLikes525 on August 05, 2019, 03:29:57 PM
Just curious why you didn't MLVAPP (Rec.709) > Prores 422 instead?
Most people who export with a LOG profile from MLVApp like to grade in another program. LOG offers much more room than Rec.709, which mostly cuts shadows and highlights, when not correcting before the export in MLVApp.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KirbyLikes525 on August 05, 2019, 08:30:24 PM
Quote from: masc on August 05, 2019, 08:12:47 PM
Most people who export with a LOG profile from MLVApp like to grade in another program. LOG offers much more room than Rec.709, which mostly cuts shadows and highlights, when not correcting before the export in MLVApp.

Gotcha, thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Mike Mout on August 10, 2019, 01:11:20 PM
Hi everybody , I'm new here, I just installed ML on my 5DMK3 and I have issues when I'm shooting in 1920x818 at 25fps 10bits ratio 2.35.1
the image is perfect but I have no sound or just a few seconds... generally the sound is being recorded around 24 seconds after the beginning of the recording and stops few seconds later (but the video recording still runs...)
Does any of you have the same problem ?
Sorry for my bad English , I'm French...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Dobfek on March 10, 2020, 10:08:30 PM
Hello! I am using this branch for a quite while now on my 600D. Unfortunately it has the SD card speed bottleneck. I would like to record raw in a better resolution but only at 8 bits. This way an "almost" 720p resolution would be possible without the compression artifacts.(I know it can be pointless as h264 has the same color depth, but the artifacts and color banding can be minimized.) Would it be possible, to implement an 8 bit raw mode? Thank you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: micromaster on January 14, 2022, 12:19:19 AM
Hello,

I use on my 7d with the magiclantern-raw_video_10bit_12bit_LVState.2019May15.7D203 (Dfort)
It works quite well even at high resolutions (10bit) but the big drawback is that the framing during recording (preview option) in ML Grayscale mode remains frozen.
In "Canon" mode the framing is smooth but much too zoomed in to be really usable.
Do you know of a version for the 7d that does not have this defect?
If so can you give me the link?

Thank you anticipating your answer.

(sorry for my English coming from a translator)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Spups on January 20, 2022, 07:01:30 PM
Quote from: micromaster on January 14, 2022, 12:19:19 AM
Hello,

I use on my 7d with the magiclantern-raw_video_10bit_12bit_LVState.2019May15.7D203 (Dfort)
It works quite well even at high resolutions (10bit) but the big drawback is that the framing during recording (preview option) in ML Grayscale mode remains frozen.
In "Canon" mode the framing is smooth but much too zoomed in to be really usable.
Do you know of a version for the 7d that does not have this defect?
If so can you give me the link?

Thank you anticipating your answer.

(sorry for my English coming from a translator)



I've tested this just now on my 7d, we share the same build and the problem persists on mine. I think it's due to this build but I'll take the working 10bit over dodgy 2k!
I reckon best way would be take a few test shots to get your reference before the take. or use greyscale for reference then back to canon for recording- Don't know about you but I only use 2k on static landscapes or cityscapes due to exaggerated CMOS slush lol, so it wouldn't slow the workflow much?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: theBilalFakhouri on December 26, 2023, 07:02:28 PM
Let's blow the dust a little bit from this thread :P

Yesterday I was looking into 10/12-bit branch, mainly for DIGIC 4 models (the older ones), what I found:

-Both 5D2 and 50D have working 12/10-bit in uncompressed MLV (Walter did some tests and verified that using mlv_rec.mo, thanks!).
Official builds are available here (https://builds.magiclantern.fm/experiments.html) under 10/12-bit RAW video.

-12/10-bit does also work on 7D, a1ex and dfort worked on it before, but the commit wasn't merged back to main repo, check this (https://www.magiclantern.fm/forum/index.php?topic=5601.msg213192#msg213192), also some (https://www.magiclantern.fm/forum/index.php?topic=5601.msg213578#msg213578) footage (https://www.magiclantern.fm/forum/index.php?topic=5601.msg218412#msg218412).

-Remaining models are 550D and 500D, what's up with them?
dfort actually did make a try in the past (https://www.magiclantern.fm/forum/index.php?topic=5601.msg212997#msg212997) without success back then, what was happening?

After taking a look today, I found that StartImagePass_x1_SetEDmac is slightly different on 500D and 550D compared to 5D2 and 50D, it turned out to be a bit similar to 7D actually, which mean we need to patch the address for StartImagePass_x1_SetEDmac like we did on 7D (https://www.magiclantern.fm/forum/index.php?topic=5601.msg212733#msg212733), also the addresses which were picked by dfort for StartImagePass_x1_SetEDmac were actually not correct, and probably no one checked them in the past other than dfort.

-Adding support to 500D and 550D:

First define CONFIG_EDMAC_RAW_PATCH in internals.h for each camera:
/** experimental - patch raw buffer parameters in Canon code */
#define CONFIG_EDMAC_RAW_PATCH


Add these addresses in stubs.S:

500D:
/** LiveView RAW patches **/
NSTUB(0xff1f7160, StartImagePass_x1_SetEDmac)            /* right before the SetEDmac call from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
NSTUB(0xff1f7dc0, StartImagePass_x5_SetEDmac)


550D:
/** LiveView RAW patches **/
NSTUB(0xff242f10, StartImagePass_x1_SetEDmac)            /* right before the SetEDmac call from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
NSTUB(0xff243cc4, StartImagePass_x5_SetEDmac)


Do this tweak in raw.c:

#ifdef CONFIG_EDMAC_RAW_PATCH
#if defined(CONFIG_550D) || defined(CONFIG_500D)
    patch_hook_function((uint32_t) &StartImagePass_x1_SetEDmac, 0xE3A00010, raw_lv_setedmac_patch, "RAW LV x1");
#else
    patch_hook_function((uint32_t) &StartImagePass_x1_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x1");
#endif
    patch_hook_function((uint32_t) &StartImagePass_x5_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x5");
#endif


-Compiled builds for running tests:

magiclantern-Nightly.2023Dec26.500D111_V2.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/magiclantern-Nightly.2023Dec26.500D111_V2.zip)
magiclantern-Nightly.2023Dec26.550D109_V2.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/magiclantern-Nightly.2023Dec26.550D109_V2.zip)

Walter did some tests on 550D using mlv_rec.mo in 10-bit in 1080p mode, first clip was corrupted other were fine! (Thanks again).
For users who have 500D or 550D, please make recording tests in 12-bit and 10-bit in different video modes and frame rates, and don't forget to share you results!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 26, 2023, 09:58:56 PM
Nice. Thought 10bit existed already on 5d2 but maybe not 8).
Is there any hope for lossless action  :P?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: theBilalFakhouri on December 27, 2023, 09:12:04 AM
Quote from: Danne on December 26, 2023, 09:58:56 PM
Nice. Thought 10bit existed already on 5d2..

Thanks!

Yes, your thought is correct, Walter made some tests to verify if it stable on 5D2, he didn't have corrupted frames and LiveView was working during recording in 10-bit,
I read some reports that said LiveView will be frozen while recording in 10-bit on 5D2, also others reported clips with corrupted frames.

Corrupted frames issue might still be a thing today (not sure what is cuasing it, but it seems to happen sometime), we probably need more tests using both mlv_rec.mo (with mlv_snd.mo too) and mlv_lite.mo for now let's make the tests on 500D and 550D (https://www.magiclantern.fm/forum/index.php?topic=5601.msg245649#msg245649).




Can't give promises on lossless :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Skinny on December 27, 2023, 10:15:32 AM
Awesome! I'll try this new 550D build :)

10/12 bits for 5D2 was there for a very long time, I almost never experienced any problems with the build on the "experiments" page. I have recorded thousands of clips with this build, it is very stable.

Quote from: Danne on December 26, 2023, 09:58:56 PM
Is there any hope for lossless action  :P?

Yeah lossless is something that could make a lot of difference for these cameras..


========

I have recorded a few clips, then battery died :D
It seems like they are perfectly fine! No pink frames or anything suspicious.. Tried both mlv_lite and raw_rec.
Up to 6 seconds with 1728x724 (max width 1,39 ar) 23,976 fps. This is actually not bad at all! With some slo-mo in post this camera could be used as a b-roll cam :)

And mlv_play doesn't work, it is counting but the frames are black.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: theBilalFakhouri on December 29, 2023, 02:23:14 PM
Thanks for the tests Skinny, that's awesome!

Quote from: Skinny on December 27, 2023, 10:15:32 AM
And mlv_play doesn't work, it is counting but the frames are black.

Yes, same on all ML builds and models, you need to load raw_twk.mo to get 12-bit and 10-bit uncompressed MLV playback support (not sure how much stable the module is).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 29, 2023, 04:08:40 PM
If anyone can solve lossless on digic IV it will be bilal 👌.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: koljanych on January 04, 2024, 09:38:16 AM
Quote from: theBilalFakhouri on December 26, 2023, 07:02:28 PM
For users who have 500D or 550D, please make recording tests in 12-bit and 10-bit in different video modes and frame rates, and don't forget to share you results!

550D

canon mode 1080p25
mlv_rec 1280x720
14 bit: 134, 132, 123 frames
12 bit: 159, 184, 186
10 bit: 252, 282, 277

canon mode 720
mlv_rec 1280x432
14 bit: 218, 200, 196 frames
12 bit: 250, 266, 280
10 bit: 372, 393, 351

a mlv  is played normally by raw_twk.mo

in general the results are very similar to raw_video_10bit_12bit_LVState-wip.2018Feb04.550D109

images: https://disk.yandex.ru/d/66wB-lVlDTRmeA