Magic Lantern Forum

Developing Magic Lantern => General Development Discussion => Topic started by: d on May 22, 2013, 10:58:34 PM

Title: 12-bit (and 10-bit) RAW video development discussion
Post by: d on May 22, 2013, 10:58:34 PM
I've spent the last couple of days working on optimized 14-to-12-bit and 14-to-10-bit RAW conversion routines in native ARM assembly in hopes that 1) in spite of conventional wisdom the camera was in fact fast enough to do rudimentary "compression", and 2) the reduction in bitrate (15% and 29% respectively) would be a step towards making the 5D2 more viable for RAW recording at resolutions higher than 1280x720. There was only a limited amount of testing I could do with my very slow (40MB/s) cards but at 1280x720 on my 5D2 the conversion is fast enough for continuous "compressed" recording. I'd like to make the routines available to some people with faster cards and different cameras to see the extent to which this is viable at higher resolutions (10-bit 1880x1080 should theoretically be possible within the 5D2's apparent 60MB/s hardware limits).

Right now this should be treated as something highly experimental. Rather than posting a pre-built raw_rec.mo file I'm posting an assembled rawc.o file that can be linked with a compiled raw_rec.o into raw_rec.mo. The edits that need to be made to raw_rec.c (keep in mind that I'm working off the May 14th version so things have probably moved around) are:

New external functions:

Code: [Select]
extern void raw14_to_raw12(void *buffer, int size);
extern void raw14_to_raw10(void *buffer, int size);

New variable:

Code: [Select]
static int buffer_size_compressed = 0;
Inside the main recording loop after if (saving_buffer_index != capturing_buffer_index), add:

Code: [Select]
buffer_size_compressed = (buffer_size_used * 12) / 14;
raw14_to_raw12(buffers[saving_buffer_index], buffer_size_used);

for 12-bit, or:

Code: [Select]
buffer_size_compressed = (buffer_size_used * 10) / 14;
raw14_to_raw10(buffers[saving_buffer_index], buffer_size_used);

for 10-bit. Change every instance of buffer_size_used later in the function to buffer_size_compressed.

The native 14-bit RAW data is packed into halfwords, but it gets repacked into words in the conversion routines, so the C struct for getting at the 12-bit data in C looks like this:

Code: [Select]
struct raw12_pixblock
{
        unsigned int c_hi: 8;
        unsigned int b: 12;
        unsigned int a: 12;
        unsigned int f_hi: 4;
        unsigned int e: 12;
        unsigned int d: 12;
        unsigned int c_lo: 4;
        unsigned int h: 12;
        unsigned int g: 12;
        unsigned int f_lo: 8;
} __attribute__((packed));

Same deal with 10-bit data, but it's 32 bits shorter (and has 10-bit pixels obviously).

You can download the .o file at http://shoutingroomonly.com/rawc.zip
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 22, 2013, 11:07:53 PM
I wonder if implementing something like this on the 5D3 would make it easier to shoot at higher resolutions in crop mode?

Does anyone have the numbers on what the theoretical MB/frame would be at 3.6k 10bit?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 22, 2013, 11:20:40 PM
This could be the magic bullet to go with the magic lantern  8)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Hazer on May 22, 2013, 11:22:55 PM
5D2 with both Lexar and Komputerbay 1000x cards and happy to test.  However my gcc-fu is a little rusty and I'd be better off trusting you to upload a functioning module.  If you can do that, count me (and probably a bunch of others) in.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 22, 2013, 11:33:53 PM
did you made some speed tests?
as you are working on uncached memories, the performance wouldnt be good enough imo.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: d on May 22, 2013, 11:56:12 PM
did you made some speed tests?
as you are working on uncached memories, the performance wouldnt be good enough imo.

It's fast enough for continuous 12-bit 1280x720 on a 5D2 (6x 24MB buffers), and this is real world performance with actual recordings on an actual camera. I don't have any other cameras (or faster cards) to test with.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 12:11:02 AM
... wow.. cannot believe that.
can you post the asm code?
memcpy using 4 registers could not get more than 2-3 fps when testing around with YUV.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 12:12:47 AM
@g3gg0 - was that bang I just heard you falling off your chair?  ;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 12:16:09 AM
yeah..
well, i implemented an memcpy using LDMIA/STMIA for LV buffer copying and this was a dead end.
so i tried to get EDMAC working.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: ch_d on May 23, 2013, 12:33:44 AM
so d. if this is true and working - u r a hero to me. count me in for testing.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 23, 2013, 12:37:07 AM
Are there any vector-ish OP's on this implementation of ARM that can do shifts on multiple 16-bit integers packed into 64 or 128-bit registers?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: nanomad on May 23, 2013, 12:37:13 AM
I've removed the download. Call it an excess of safety but I don't want random code running on cameras especially of it doesn't have a source code attached.
I've sent the link to g3gg0 so he can review it and re-enable it if it's good
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Greg on May 23, 2013, 12:40:32 AM
I've removed the download. Call it an excess of safety but I don't want random code running on cameras especially of it doesn't have a source code attached.
I've sent the link to g3gg0 so he can review it and re-enable it if it's good
Good idea, the safety of our cameras is important.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Kunibert on May 23, 2013, 01:05:01 AM
Are there any vector-ish OP's on this implementation of ARM that can do shifts on multiple 16-bit integers packed into 64 or 128-bit registers?
As far as i know the Digic 5+ is a modification of this core:
http://www.arm.com/products/processors/classic/arm9/arm946.php?tab=Specifications
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Alia5 on May 23, 2013, 01:13:08 AM
Thanks! Now i don't trust the Devs anymore when they say that somethign would be impossible!
No serious, awesome job!
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 01:14:29 AM
here the removed link:
http://shoutingroomonly.com/rawc.zip

the CPU in digic is an ARM946, thats right.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 01:29:03 AM
This would be a HUGE win for SD cameras that can't record even close to CF.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Audionut on May 23, 2013, 01:33:22 AM
And those of us looking for highest resolution at 48-60fps.

The bottom 2 bits are pretty much just noise anyway.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 01:33:52 AM
This would be a HUGE win for SD cameras that can't record even close to CF.

Are you gonna try with TL2 or is that a stupid question?  ;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 01:38:12 AM
I'll try to download it and just run it I guess... I'm not worried about this dude putting his blood sweat and tears in just to break my camera.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: silvertonesx24 on May 23, 2013, 01:42:14 AM
Am I correct in assuming that 10-bit and 12-bit will look no different than 14-bit (in real-world application of this camera)?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 01:46:48 AM
I've linked as follows:

MODULE_OBJS=raw_rec.o rawc.o


When I boot up I get this laughing skull and then camera starts smoking.....


Actually not sure if its working, will have to open the raws it produced. Not sure if I'm getting speed increase... looks like its doing something. Maybe have to add the modifications?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Alia5 on May 23, 2013, 01:48:30 AM
Am I correct in assuming that 10-bit and 12-bit will look no different than 14-bit (in real-world application of this camera)?

Reading this: http://francoismalan.com/2011/10/raw-12bit-or-14bit-lossy-or-lossless/
... i guess that it would only make a very very slight difference!

If this is really working... I just dont have words for how awesome this is...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 01:48:49 AM
I'll try to download it and just run it I guess... I'm not worried about this dude putting his blood sweat and tears in just to break my camera.

I'll be a guinea pig if needed. Got my 50d yesterday and I'm willing to risk my 600 for something important like this. (Disclaimer. OK, the wife thinks the 600 is now hers but...)  ;)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 01:50:15 AM
I've linked as follows:

MODULE_OBJS=raw_rec.o rawc.o


When I boot up I get this laughing skull and then camera starts smoking.....


Actually not sure if its working, will have to open the raws it produced. Not sure if I'm getting speed increase... looks like its doing something. Maybe have to add the modifications?

I was half-expecting you to have been rickrolled by it :D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 01:52:40 AM
Maybe he is from Canon giving us hints?  ???
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 01:55:53 AM
Its up in the 6D repo... see if it does anything.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: menoc on May 23, 2013, 01:58:48 AM
Maybe he is from Canon giving us hints?  ???

Please don't spook him . . .
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 01:59:35 AM
Quote
Its up in the 6D repo... see if it does anything.
im scared lol
Title: Re: 12-bit (and 10-bit) RAW video
Post by: RenatoPhoto on May 23, 2013, 02:01:37 AM
Probably Canons way of bricking all our cams!! 8)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:02:42 AM
Its will smith giving us all a virus. But must be slow acting cuz my cam still works :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 02:05:53 AM
is it faster tho?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: N/A on May 23, 2013, 02:06:30 AM
What the hell, t3i's are dirt cheap now, I'll give it a run if someone hooks a t3i link up.

Title: Re: 12-bit (and 10-bit) RAW video
Post by: RenatoPhoto on May 23, 2013, 02:06:40 AM
Its will smith giving us all a virus. But must be slow acting cuz my cam still works :)

I am dying to know ...  I will write a How to (Brick)  Manual...   :o
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:07:19 AM
Try it... I can't tell as latest modules do like 30MB/s for me. I guess you'd have to see if you're getting any additional frames.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 02:14:10 AM
did not do anything for me... same speed
Title: Re: 12-bit (and 10-bit) RAW video
Post by: silvertonesx24 on May 23, 2013, 02:16:51 AM
Reading this: http://francoismalan.com/2011/10/raw-12bit-or-14bit-lossy-or-lossless/
... i guess that it would only make a very very slight difference!

If this is really working... I just dont have words for how awesome this is...

Absolutely! I just used 5D raw for a paying client today (stupid, I know, but it ran without a hitch) and the only pain really was swapping cards around. Need to load up on a few 64gbs. Any space or efficiency savings are welcomed.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Audionut on May 23, 2013, 02:17:19 AM
did not do anything for me... same speed

Are the DNG frames in the RAW video smaller then without this mod?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 02:20:51 AM
just linking wont work. the raw rec module must call the routines to reduce the bit depth.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:21:38 AM
Ok, will edit then ... as per first post I guess?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 02:22:58 AM
Quote
Are the DNG frames in the RAW video smaller then without this mod?

nope its the same
g3gg0 did it work for you?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 02:23:32 AM
@1%: yep. (of course linking into the raw rec module makes more sense than in ML Core)

@DTSET123: nope, as i am busy with some other stuff atm
Title: Re: 12-bit (and 10-bit) RAW video
Post by: DTSET123 on May 23, 2013, 02:25:01 AM
 :-[
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 02:29:52 AM
I think he was answering 1%  ;)

We'll need to update raw_get_pixel() in raw2dng with the 12/10bit format, maybe store bit-depth in raw header and process raw accordingly?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:30:16 AM
Well its a bit fucked because new code uses a struct and not buffer_size_used, etc.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Greg on May 23, 2013, 02:40:24 AM
Here is the code of buffer_size_used - https://bitbucket.org/hudson/magic-lantern/src/84f657f209eb5b599ec32bd85718c9951c25b720/modules/raw_rec/raw_rec.c?at=unified
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:43:14 AM
I did this:

Code: [Select]

        if (saving_buffer_index != capturing_buffer_index)
        {
    if (!t0) t0 = get_ms_clock_value();
            void* ptr = buffers[saving_buffer_index].ptr;
            int size_used = buffers[saving_buffer_index].used;
buffer_size_compressed = (size_used * 12) / 14;
raw14_to_raw12(ptr, size_used);     
int r = FIO_WriteFile(f, ptr, buffer_size_compressed);
            if (r != buffer_size_compressed) goto abort;
            written += size_used;
//~ writtenL = (uint32_t) written;
            saving_buffer_index = mod(saving_buffer_index + 1, buffer_count);
        }


I think it needs more changes...says its writing at 33MB/s, lol

It compiled... will have to make new bin
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Audionut on May 23, 2013, 02:51:25 AM
Is the code just truncating bits, or is it compressing them into the remaining?

edit:  The OP states that the bits are being compressed.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:52:07 AM
I guess I'll find out?

http://pastebin.com/PviXPWNZ

It is saving files... I don't see any increase in speed but maybe a few extra frames... I think the change is still incomplete.

Pink garbage when I open them :(

Second shot had image cut up into squares like lv_rec used to do when it was out of sync.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 02:56:29 AM
Did you modify raw2dng ?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 02:59:37 AM
Nope.. feel free to try, I'm not the best coder... guy should have really provided source... may 14th is hella old and slow.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Greg on May 23, 2013, 03:02:45 AM
I guess I'll find out?

http://pastebin.com/PviXPWNZ

It is saving files... I don't see any increase in speed but maybe a few extra frames... I think the change is still incomplete.

Pink garbage when I open them :(

Second shot had image cut up into squares like lv_rec used to do when it was out of sync.

On the 500D:
tcc : error: undefined symbol 'raw14_to_raw12'   :o
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 03:04:18 AM
Did oyu add the struct to raw.h? also link the .o in the makefile.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 03:08:55 AM
Nope.. feel free to try, I'm not the best coder... guy should have really provided source... may 14th is hella old and slow.

I don't have all the required libs installed for this (And i really should be in bed right now), but in raw2dng.c

Code: [Select]

struct raw12_pixblock
{
        unsigned int c_hi: 8;
        unsigned int b: 12;
        unsigned int a: 12;
        unsigned int f_hi: 4;
        unsigned int e: 12;
        unsigned int d: 12;
        unsigned int c_lo: 4;
        unsigned int h: 12;
        unsigned int g: 12;
        unsigned int f_lo: 8;
} __attribute__((packed));

int raw_get_pixel(int x, int y) {
    struct raw12_pixblock * p = (void*)raw_info.buffer + y * raw_info.pitch + (x/8)*12;
    switch (x%8) {
        case 0: return p->a;
        case 1: return p->b;
        case 2: return p->c_lo | (p->c_hi << 8);
        case 3: return p->d;
        case 4: return p->e;
        case 5: return p->f_lo | (p->f_hi << 8);
        case 6: return p->g;
        case 7: return p->h;
    }
    return p->a;
}

If anyone wants to try  ;)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 03:14:04 AM
Gonna try it on the file I made already
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Greg on May 23, 2013, 03:17:25 AM
Did oyu add the struct to raw.h?
Yes  ;)
also link the .o in the makefile.
No  :o
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 03:18:47 AM
First one came out as pink shit with modified raw2dng.

Some of the frame pieces do come out... so it is kinda working.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: d on May 23, 2013, 04:56:14 AM
It looks like I accidentally uploaded an old, broken version of rawc.o. The correct, working version is now at the same URL. If you downloaded it before this reply went up you will need to re-download and and re-link it. Sorry about that.

Also, I noticed a mistake in the example raw_get_pixel posted earlier. The correct shifts for pixels C and F are:

Code: [Select]
raw->c_lo | (raw->c_hi << 4);
raw->f_lo | (raw->f_hi << 8);
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tin2tin on May 23, 2013, 10:19:23 AM
This is just... wow! The perspective of this is just incredible. Thank you for investing your time in this.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 23, 2013, 10:26:21 AM
(http://a1ex.magiclantern.fm/bleeding-edge/raw/bench14to12-playback.png) (http://a1ex.magiclantern.fm/bleeding-edge/raw/bench14to12-movie24p.png)

Impressive.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tin2tin on May 23, 2013, 11:26:52 AM
What is the 12/10 bit speed of 720p? Would this be usable for SD cards?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 11:30:04 AM
It looks about 25% reduced :) cool

@a1ex - have you shot anything with compression? How's it looking?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 23, 2013, 11:30:48 AM
What is the 12/10 bit speed of 720p? Would this be usable for SD cards?

In theory, SD cards can give 22MB/s or 961194 bytes per frame. 1280x600 in 10 bits gives 960000 bytes.

In practice - we need a working build to test it (was the latest one 1% posted - the working one?)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: squig on May 23, 2013, 11:32:29 AM
Settle, settle, this development is only like 13 hours old, the little swimmers are still trying to find their way to the egg.  ::)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 23, 2013, 11:37:02 AM
I didn't shoot anything with it, only ran the benchmark. Right now I have no idea whether the algorithm is correct. You need an additional task to prepare the buffers, so the raw processing should become a pipeline, like this:
- source raw to cropped raw: via EDMAC, triggered from LiveView (EVF) task
- cropped raw (14-bit) to 12-bit raw
- file write task
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tihon on May 23, 2013, 12:07:13 PM
In theory, SD cards can give 22MB/s or 961194 bytes per frame. 1280x600 in 10 bits gives 960000 bytes.

In practice - we need a working build to test it (was the latest one 1% posted - the working one?)
That so you can write 2.5k with 44 MB/s card? (10bit)) It can give a new live to mark 2

I think that it can be possible to wite a  2350x1100px on 5dm2 in 1:1 mode  with 12bit
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 23, 2013, 12:30:47 PM
That so you can write 2.5k with 44 MB/s card? (10bit)) It can give a new live to mark 2
I think that it can be possible to wite a  2350x1100px on 5dm2 in 1:1 mode  with 12bit

44 MB/s is 1.922.389 bytes per frame @24fps, or 1.537.911 pixels in 10bit - that makes something around 1660x930px frame.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: t2scorp on May 23, 2013, 12:42:29 PM
44 MB/s is 1.922.389 bytes per frame @24fps, or 1.537.911 pixels in 10bit - that makes something around 1660x930px frame.

Meaning that resolutions up to 2240x1260 @ 24fps at 83MB/s and 1920x1080 @24fps at 62MB/s should be possible?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 23, 2013, 12:53:34 PM
No, according to the benchmarks.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Andy600 on May 23, 2013, 12:54:15 PM
This is great news for the 50d. It should be able to record all it's available frame sizes unrestricted  :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: savale on May 23, 2013, 12:59:36 PM
Cool stuff! Don't know how to understand the benchmarks. Is that the "needed" bandwidth because of the less needed bits, or is it the actual bandwidth because of the extra calculations needed? (in that case the bandwidth is too low?)

I really want to look at the raw14_to_raw12 and raw14_to_raw10 methods, where can I find them?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: nanomad on May 23, 2013, 01:00:34 PM
They are inside the .o file d provided. In ARM ASM code once you de-compile it
Title: Re: 12-bit (and 10-bit) RAW video
Post by: eyeland on May 23, 2013, 01:24:12 PM
This is one of the threads that will wear down my F5 key ...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 01:29:46 PM
Do we know if this could be applied to higher resolutions than 1080p, or would the camera not be able to handle it?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AdamTheBoy on May 23, 2013, 03:21:06 PM
I'm pretty much a lamen, but it looks like you guys are figuring out how to retain the quality of RAW but save on space? This is such wonderful news.  So this means we may have longer record times and if the frames are a smaller size, fewer dropped frames?

Thanks to everyone involved, it's always exciting to see what people are achieving here. 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 03:46:15 PM
Have a better patch to raw_rec.mo? I think mine was a little lacking. 

IMO, should have an option in raw_rec 10/12/etc bit. So cameras like 600D can do 10 bit and push the res up to somewhere acceptable while 6D/5d3/etc can still do the higher ones, etc.



Title: Re: 12-bit (and 10-bit) RAW video
Post by: xxxbugxxxx on May 23, 2013, 03:51:49 PM
Just curious, does Canon hard code 14bit raw processing? Is it possible to change bitdepth in some native way to change the RAW AD converter to do 12bit or even 10 bit? I know on Nikon D600 you can choose between 14bit or 12bit. Wonder if that can be easier?

Thank you for the great work.

Title: Re: 12-bit (and 10-bit) RAW video
Post by: KenFTW on May 23, 2013, 03:57:04 PM
This is one of the threads that will wear down my F5 key ...

this and about three others... monumental stuff people, the rules have been changed.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 04:00:33 PM
The big question is... 

Is it going to be possible to shoot 3.6k @ 24fps, 10bit on the 5D3 without dropping frames?

If the percentages listed by d are correct, that would put the 3592 x 1320 at about 5MB/frame with a throughput of ~120MB/sec...  a bit much for most cards, but possible?

Is there anyway to drop it even lower to 8bit?  *I'd rather have color information... but if the resolution can be achieved it would certainly be a bonus.  :-)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: coutts on May 23, 2013, 04:02:33 PM
Just curious, does Canon hard code 14bit raw processing? Is it possible to change bitdepth in some native way to change the RAW AD converter to do 12bit or even 10 bit? I know on Nikon D600 you can choose between 14bit or 12bit. Wonder if that can be easier?

Thank you for the great work.

There's a factory function named "FA_SetCRawBitNum", located at 0xFF47C0D8 in the 6D.113, seems to just store a number at 0x82734 . It can be called via the Call function, maybe this will work?
Code: [Select]
call( "FA_SetCRawBitNum", 10 );
Title: Re: 12-bit (and 10-bit) RAW video
Post by: N/A on May 23, 2013, 04:04:28 PM
So cameras like 600D can do 10 bit and push the res up to somewhere acceptable while 6D/5d3/etc can still do the higher ones, etc.
600D would make a perfect b-roll if we could match resolution
Title: Re: 12-bit (and 10-bit) RAW video
Post by: bumkicho on May 23, 2013, 04:10:42 PM
This is such an important and exciting development!!
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 04:13:44 PM
I messed with that function... but was trying for H264.. I wonder if it cuts bits out of the "debug" raw we're using.

Called it and shot 2 silent pics... they are the same size :(
Title: Re: 12-bit (and 10-bit) RAW video
Post by: coutts on May 23, 2013, 04:15:33 PM
I messed with that function... but was trying for H264.. I wonder if it cuts bits out of the "debug" raw we're using.
at first test with 5d2, the location is empty (0), maybe i need to call lv_save_raw first.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 04:21:13 PM
Location changes to a (10) when I call it.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 06:14:06 PM
Finally got everything running, but the output seems shifted. Can't tell yet if it's my modified raw_rec or modified raw2dng.
The corrupted dngs are the correct size though.

Also, I didn't write more frames at high resolutions compared to 14-bit, so this probably wouldn't help getting higher resolutions.
The only interest would be in saving card space  :-\
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 06:16:32 PM
Shifted.. like all tesselated? Post an example?

If the files are smaller at all then you'll get a few more frames... right now maybe CPU usage > size savings?  I think it had to go into a separate task like a1ex said to not interfere with writing.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 06:16:58 PM
Mayo: what camera are you on? Settings?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 06:44:38 PM
I'm on 6D, I shoot in photo mode, global draw off.

But wait a minute... I noticed raw2dng just saves the raw frames, but nowhere saves a bitdepth or linewidth...
So how could DNG software (like ACR) read this properly if they don't know it's 12-bit?
Does the DNG format support anything other than 14-bit?
If it does not, we need to convert 12-bit back to 14-bit to export as dng. (I'm really not sure I have the skills to do this properly).
Reading chdk-dng.c now...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 06:52:57 PM
I think DNG supports all bit depths but the DNG meta data will say it X bit.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 06:55:18 PM
Has anyone tested this on a 5D3 yet with the fastest card?

*I'm excited! 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 06:57:34 PM
We're barely getting it running :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: gebo on May 23, 2013, 07:05:51 PM
I'm checking these forums daily to see how things are developing. Congratulations, you're doing the "impossible"!
I'm so excited I even have trouble sleeping... :o
Title: Re: 12-bit (and 10-bit) RAW video
Post by: seanturco on May 23, 2013, 07:13:59 PM
I'm amazed by the work. I wish to be a developers one days :p
But I'm just a photographer  ;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: JohnBarlow on May 23, 2013, 07:14:06 PM
Maybe silly questions--->

Is this sRGB or ADobe RGB?

If the former will it cause a processor drain?

Is Adobe RGB possible seeing that we are dealing with a stream of stills?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 07:18:56 PM
Both? Neither? Whichever one you pick?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 07:23:20 PM
I think DNG supports all bit depths but the DNG meta data will say it X bit.

Right, I found it, had to overwrite raw_info.bits_per_pixel=12; in raw2dng.

Now the image is good, only the colors are messed up.

Looks like this in irfan view (doesn't open in ACR):

(http://i.imgur.com/lsMeEFO.jpg)

(It's the bottom of a (very dirty) window)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: JohnBarlow on May 23, 2013, 07:24:26 PM
Do you mean you can have either?


Both? Neither? Whichever one you pick?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 07:28:02 PM
Right, I found it, had to overwrite raw_info.bits_per_pixel=12; in raw2dng.

Now the image is good, only the colors are messed up.

Looks like this in irfan view (doesn't open in ACR):

(http://i.imgur.com/lsMeEFO.jpg)

(It's the bottom of a (very dirty) window)

Too bad the 'Predator' franchise is finished... :-/ 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 07:30:26 PM
Black level is wrong.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 07:30:46 PM
If only d posted his post-process workflow, this would be easier :P
Title: Re: 12-bit (and 10-bit) RAW video
Post by: IliasG on May 23, 2013, 07:33:19 PM
Right, I found it, had to overwrite raw_info.bits_per_pixel=12; in raw2dng.

Now the image is good, only the colors are messed up.

Looks like this in irfan view (doesn't open in ACR):

http://i.imgur.com/lsMeEFO.jpg

(It's the bottom of a (very dirty) window)

You also have to adapt the Black Level & White Level to 12 bit ..
Title: Re: 12-bit (and 10-bit) RAW video
Post by: chmee on May 23, 2013, 07:36:30 PM
Open the DNG-File in a Hex-Editor (recommending HxD).
Search for the BitsPerSample-Tag 0102 - think it will be on Position ~1BC
(there will be two entries, because the thumbnail needs this tag as well)
Code: [Select]
0201 0300 0100 0000 0E00 00000102 (or in little-endian 0201) and later on there is 000E (14Bit) -> 0E00. change that.

You find the Whitelevel and blacklevel with the tag-id's C61A and C61D, some bytes after the BitPerSampleTag. Example:
Code: [Select]
1AC6 0400 0100 0000 F807 0000 <-->07F8 = 2040
1DC6 0400 0100 0000 2E2D 0000 <--> 2D2E = 11566
Lower both. For Example Blacklevel to 1000 and Whitelevel to 4000.
(http://dl.phreekz.de/DNG_Example.jpg)

mfg chmee
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 23, 2013, 07:38:29 PM
Is this sRGB or ADobe RGB?

RAW is beyond these. It becomes sRGB or Adobe RGB when exporting (baking) from bayer raw to RGB in any Raw Converter (ACR, Lightroom, etc).
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 07:54:36 PM
Modified reverse_bytes_order() for words instead of half-words in chdk-dng.c ,it's a bit better now.

Code: [Select]
static void FAST reverse_bytes_order(char* buf, int count)
{
long* buf32 = (long*) buf;
    int i;
    for (i = 0; i < count/4; i++)
    {
        long x = buf32[i];
buf[4*i+3] = x;
        buf[4*i+2] = x >> 8;
        buf[4*i+1] = x >> 16;
        buf[4*i] = x >>24;
    }
/*
    short* buf16 = (short*) buf;
    int i;
    for (i = 0; i < count/2; i++)
    {
        short x = buf16[i];
        buf[2*i+1] = x;
        buf[2*i] = x >> 8;
    }
*/
}

Open the DNG-File in a Hex-Editor (recommending HxD).
Search for the BitsPerSample-Tag 0102 - think it will be on Position ~1BC
(there will be two entries, because the thumbnail needs this tag as well)
Code: [Select]
0201 0300 0100 0000 0E00 00000102 (or in little-endian 0201) and later on there is 000E (14Bit) -> 0E00. change that.

You find the Whitelevel and blacklevel with the tag-id's C61A and C61D, some bytes after the BitPerSampleTag. Example:
Code: [Select]
1AC6 0400 0100 0000 F807 0000 <-->07F8 = 2040
1DC6 0400 0100 0000 2E2D 0000 <--> 2D2E = 11566
Lower both. For Example Blacklevel to 1000 and Whitelevel to 4000.
(http://dl.phreekz.de/DNG_Example.jpg)

mfg chmee

Will try this now.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: chmee on May 23, 2013, 07:58:38 PM
ah. Forgot to say. AFAIK the Preview-Jpg is hardcoded on 14Bit-Transcode. Rename the .dng to .tif and look what the viewer is throwing out. 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 23, 2013, 07:59:21 PM
Yep and then post a workable source for the mods so we can try it out in one fell swoop.

If its good will need depth selection for raw2dng/rawrec... I think you could even do silent pics this way too. 8-14 bit.

Real compression I think manipulates the whole image.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 08:06:50 PM
Almost there.

(http://i.imgur.com/f4D8A8v.jpg)

Now to find

1). where the black/white levels are wrongly set in chdk-dng.c
2). why the DNG file won't open in ACR
Title: Re: 12-bit (and 10-bit) RAW video
Post by: JohnBarlow on May 23, 2013, 08:07:55 PM
Thanks!

RAW is beyond these. It becomes sRGB or Adobe RGB when exporting (baking) from bayer raw to RGB in any Raw Converter (ACR, Lightroom, etc).
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 08:25:29 PM
Wow, simply lowered black and white (*0.25f) in raw2dng before it exports

Code: [Select]
raw_info.bits_per_pixel=12;
raw_info.white_level *= 0.25f;
    raw_info.black_level *= 0.25f;

And now the file opens ACR  ;D

(http://i.imgur.com/3DhYy4u.jpg)

I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Title: Re: 12-bit (and 10-bit) RAW video
Post by: AnotherDave on May 23, 2013, 08:45:24 PM
So what does mean for getting it up and running on the 5D3?  :-)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: bumkicho on May 23, 2013, 08:46:40 PM
I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Does it mean 12bit or 10bit raw is coming sometime very soon?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 09:03:15 PM
Well, I didn't touch 10-bit, but it should be fairly easy to add.

Here are the modified src files: https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing (https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing)

12-bit is currently hardcoded with the variable:

Code: [Select]
static int raw_bitdepth = 12;
...which should be set via a RAW menu option and exported in the RAW footer to be retrieved by raw2dng and processed accordingly.

Title: Re: 12-bit (and 10-bit) RAW video
Post by: scrax on May 23, 2013, 09:15:51 PM
Well, I didn't touch 10-bit, but it should be fairly easy to add.

Here are the modified src files: https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing (https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing)

12-bit is currently hardcoded with the variable:

Code: [Select]
static int raw_bitdepth = 12;
...which should be set via a RAW menu option and exported in the RAW footer to be retrieved by raw2dng and processed accordingly.

why don't you open a fork on bitbucket instead of google drive? It will be easier to follow (and learn from) your changes
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 09:22:55 PM
why don't you open a fork on bitbucket instead of google drive? It will be easier to follow (and learn from) your changes

I've been a lone coder for 4-5 years now, and became a bit oblivious to source control :P
I had a look at the tragic lantern bitbucket (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all)) but i'm terrified to fuck up things for you guys... If you can quickly teach me how to commit my changes, i will gladly do it.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 23, 2013, 09:38:40 PM
As far as i know the Digic 5+ is a modification of this core:
http://www.arm.com/products/processors/classic/arm9/arm946.php?tab=Specifications

Well as long as Arm 946 supports NEON, there is a neat quad-word SIMD shift instruction that is potentially able to do shift to 10 bits for one whole RGB pixel in one instruction:

http://infocenter.arm.com/help/topic/com.arm.doc.dui0489c/CIHEDCEG.html

In that case, there is even more speed to be gained. If not, we need to check into "Enhanced DSP instructions" which seems to be onboard on 946 for sure:
http://infocenter.arm.com/help/topic/com.arm.doc.dvi0022a/DVI0022A.pdf
Title: Re: 12-bit (and 10-bit) RAW video
Post by: bumkicho on May 23, 2013, 09:43:32 PM
Well, I didn't touch 10-bit, but it should be fairly easy to add.

Here are the modified src files: https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing (https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing)

12-bit is currently hardcoded with the variable:

Code: [Select]
static int raw_bitdepth = 12;
...which should be set via a RAW menu option and exported in the RAW footer to be retrieved by raw2dng and processed accordingly.

I would try 12bit to see if it works properly first. Then we can move onto adding 10bit. Well, I say "we" as if I am actually contributing anything to this development. Ha ha. Sorry. I meant I hoped you could.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: jordancolburn on May 23, 2013, 09:57:26 PM
I've been a lone coder for 4-5 years now, and became a bit oblivious to source control :P
I had a look at the tragic lantern bitbucket (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all)) but i'm terrified to fuck up things for you guys... If you can quickly teach me how to commit my changes, i will gladly do it.
Here's a really good Mercurial tutorial http://hginit.com/
I think most of the devs use it and I use it at work and for my own projects like my resume in LaTex and others.  Merging is a little strange at first, but it will make your life SO much easier in the long run.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tin2tin on May 23, 2013, 10:11:32 PM
In theory, SD cards can give 22MB/s or 961194 bytes per frame. 1280x600 in 10 bits gives 960000 bytes.

In practice - we need a working build to test it (was the latest one 1% posted - the working one?)
Can't wait to hear if 1280x600 in 10 bits really is possible, so Cameras with SD cards only can benefit from the latest RAW magic.  :P 
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 23, 2013, 10:18:23 PM
Here's a really good Mercurial tutorial http://hginit.com/
I think most of the devs use it and I use it at work and for my own projects like my resume in LaTex and others.  Merging is a little strange at first, but it will make your life SO much easier in the long run.

Well, that looks nice and easy :P
Seriously though, the changes are really minor: just do a find 'raw_bitdepth' and all my changes are there (except for reverse_bytes_order() which is terribly hardcoded). If someone who's already setup can look through them it would be much safer than me fucking things up.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: scrax on May 23, 2013, 10:25:37 PM
Well, that looks nice and easy :P
Seriously though, the changes are really minor: just do a find 'raw_bitdepth' and all my changes are there (except for reverse_bytes_order() which is terribly hardcoded). If someone who's already setup can look through them it would be much safer than me fucking things up.

I think that quick way to share the code without any risk is, once installed hg, register yourself on bitbucket, go to magic lantern source and fork the repo, you will have your fork on bitbucket pubblic so any change you made will not screw anything in the ML source.
Once done the fork you have to download it on your pc (I use a GUI for managing those forks, SourceTree), make your changes and with "hg pull" update it on bitbucket.
When you are done with all your work you can submit from the fork page on bitbucket a pull request to the ML source, that way it will be reviewed before merging it.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 23, 2013, 11:16:14 PM
If not, we need to check into "Enhanced DSP instructions" which seems to be onboard on 946 for sure:

this is for ARM946E-S vs. ARM946E which we have :)

but i didnt verify
Title: Re: 12-bit (and 10-bit) RAW video
Post by: trsaunders on May 23, 2013, 11:59:58 PM
I'm wondering: how is the RAW data (on the camera, before it is copied) aligned? is it 14bits with two zeros packed in 16 bits, or is each pixel packed tightly together?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 12:07:04 AM
I'm wondering: how is the RAW data (on the camera, before it is copied) aligned? is it 14bits with two zeros packed in 16 bits, or is each pixel packed tightly together?

It's tight:

Code: [Select]
/**
* 14-bit encoding:

hi          lo
aaaaaaaaaaaaaabb
bbbbbbbbbbbbcccc
ccccccccccdddddd
ddddddddeeeeeeee
eeeeeeffffffffff
ffffgggggggggggg
gghhhhhhhhhhhhhh
*/


I think that quick way to share the code without any risk is, once installed hg, register yourself on bitbucket, go to magic lantern source and fork the repo, you will have your fork on bitbucket pubblic so any change you made will not screw anything in the ML source.
Once done the fork you have to download it on your pc (I use a GUI for managing those forks, SourceTree), make your changes and with "hg pull" update it on bitbucket.
When you are done with all your work you can submit from the fork page on bitbucket a pull request to the ML source, that way it will be reviewed before merging it.

Thank you. I did all this, let's see how it goes.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: SonicVibe on May 24, 2013, 12:50:52 AM
Mayo, thanks for your work on this.

I think they meant for you to fork the magic lantern repo and not tragic lantern, which is already a fork of magic lantern.

https://bitbucket.org/hudson/magic-lantern
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 02:51:12 AM
Fork whichever... I can give you write permisison too if you want.

For 6D I wouldn't fork the 'main repo as its not updated.

Gonna start merging tomorrow morning or later on tonight...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: hirethestache on May 24, 2013, 04:22:10 AM
Im really rooting for the development of this on my 5D3, as I just purchased two more 5D3's and cant really afford a 1000x Lexar :-p
Title: Re: 12-bit (and 10-bit) RAW video
Post by: IliasG on May 24, 2013, 04:52:22 AM
Wow, simply lowered black and white (*0.25f) in raw2dng before it exports

Code: [Select]
raw_info.bits_per_pixel=12;
raw_info.white_level *= 0.25f;
    raw_info.black_level *= 0.25f;

And now the file opens ACR  ;D

(http://i.imgur.com/3DhYy4u.jpg)

I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Nice, thanks.

Is this posterisation down there ??.

Can you upload your dng file ??.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 06:57:12 AM
For some reason 12 bit stops faster than 14 bit right now.. 1200x676 I have to stop recording in 14bit but in 12 bit only 144 frames. Write speed drops to 26MB/s if LV is on and pretty much the same (28-30) if you hack it off. Global draw off seemed to help a tiny bit too. Still must be something stopping it early.

Or maybe 10 bit would be better?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 24, 2013, 07:57:16 AM
For some reason 12 bit stops faster than 14 bit right now.. 1200x676 I have to stop recording in 14bit but in 12 bit only 144 frames. Write speed drops to 26MB/s if LV is on and pretty much the same (28-30) if you hack it off. Global draw off seemed to help a tiny bit too. Still must be something stopping it early.

Or maybe 10 bit would be better?

Maybe good thought to try extreme values, like 2-4 bits, to find out what stops it - bit processing, or card write. And if it's okay - then slowly raise it until the problem appears. Or don't write to card at all and see how much the processing only gives (something similar to a1ex benchmark, but with real data).
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 08:12:57 AM
It looks like a bug.. like its not clearing the buffers? 114 frames is about the size of memory or something?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a.d. on May 24, 2013, 09:16:47 AM
The same here on 5D2! This is the end at 114  ;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 24, 2013, 10:04:03 AM
Modified reverse_bytes_order() for words instead of half-words in chdk-dng.c ,it's a bit better now.

Code: [Select]
static void FAST reverse_bytes_order(char* buf, int count)
{
long* buf32 = (long*) buf;
    int i;
    for (i = 0; i < count/4; i++)
    {
        long x = buf32[i];
buf[4*i+3] = x;
        buf[4*i+2] = x >> 8;
        buf[4*i+1] = x >> 16;
        buf[4*i] = x >>24;
    }
/*
    short* buf16 = (short*) buf;
    int i;
    for (i = 0; i < count/2; i++)
    {
        short x = buf16[i];
        buf[2*i+1] = x;
        buf[2*i] = x >> 8;
    }
*/
}

Will try this now.

Hi!


    for (i = 0; i < count/2; i++)

What about pre-calculating "count/2" into it's own variable and use that variable instead in your "for"-loop, otherwise it will waste precious CPU cycles evaluating "count/2" for each loop (giving same results over and over again)?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 10:08:04 AM
You mean... raw2dng feels kinda slow?

;)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 24, 2013, 10:11:11 AM
You mean... raw2dng feels kinda slow?

;)

Ah, I didn't know which application this snippet of code belonged to. I just reacted when i saw the code :D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 10:15:06 AM
I don't know programming much. But it sounds to me that the program itself stalled itself and exit somewhere(or suspended by the OS), my wild guess.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 10:21:58 AM
FYI, gcc already moves this kind of computations outside loops.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 10:58:34 AM
buffer_size_compressed = (buffer_size_used * 12) / 14;

I don't know what is this, but if it changes to:

buffer_size_compressed = buffer_size_used  / 14;
buffer_size_compressed = buffer_size_compressed * 12;

It might have higher accuracy, I heard that some CPU cannot multiply too many times, or very soon it will have accuracy problem.

HA, I have been messing around. I quit.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 24, 2013, 11:46:57 AM
this is for ARM946E-S vs. ARM946E which we have :)

but i didnt verify

Ok! That's a pity :(

 My understanding is that Canon licensed  basic ARM946E core and then added some of their ow stuff to it. do you know if there is any way to check for procedure/op -calls done from, for example H264 encoder? I bet it has $hitload of DSP/SIMD op's hardcoded on die in order to do real-time H264 encoding, some of those could be "re-used" for 4 x small int bitshift? (Or is raw data offloaded to buffer, to be handled by some sort of co-processor involved)

Pity "d" didn't make rawc.o ASM available. It would be very interesting to take a peek...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: nanomad on May 24, 2013, 12:59:31 PM
It's available, just de-compile it  ::)

H264 encoding is done by an external chip, so there's no need for fancy SIMD instructions sets on the ARM core :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 02:50:04 PM
buffer_size_compressed = (buffer_size_used * 12) / 14;

I don't know what is this, but if it changes to:

buffer_size_compressed = buffer_size_used  / 14;
buffer_size_compressed = buffer_size_compressed * 12;

It might have higher accuracy, I heard that some CPU cannot multiply too many times, or very soon it will have accuracy problem.

HA, I have been messing around. I quit.

Or simply changes to:
buffer_size_compressed = (buffer_size_used / 14) *12;

Because if buffer_size_used is something big, *12 might make it very big, making it difficult to be divided by 14, so I think that it might be better to divide first, and then multiply. But this trick might not work anyway.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: nanomad on May 24, 2013, 04:26:56 PM
Don't do the compiler's job :P
He's usually good at it
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 04:31:36 PM
Its merged in and can be switched between... why does it stop tho at every res?

https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/src/7b5b918c68a2d7ba60d8c614e3576a3bc87785f4/modules/raw_rec/raw_rec.c
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 05:12:44 PM
I have noticed another thing:

static int buffer_size_compressed = 0;

If  buffer_size_compressed is divided by 14, the result is in float point, you might need to change it back to int. That is a pain in the axx thing too I guess.

I might be wrong.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:15:44 PM
sized_used isn't static so I can try that... the frames come out smaller when I open them so 12 bit should be helping.

Code: [Select]
buffer_size_compressed = (size_used * raw_bitdepth) / 14;
This doesn't make sense to me actually... why are we multiplying this out? The *12/14 calculation is to find width normally.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:17:38 PM
1%, I just realised

Code: [Select]
{
                .name = "Bit Depth",
                .priv = &raw_bitdepth,
                .min = 12,
                .max = 13,
                .choices = CHOICES("12 Bit", "14 Bit", ),                               
                .help2 = "Bit Depth",
            },

this will probably not work (can't test atm)... I wrote somewhere that raw_bitdepth would work as usual if it was anything else than 12.
It's wrong... It has to be either 12 or 14. Probably needs a menu update function and a LUT.
As for why it stops, no idea. :(
Probably

Code: [Select]
if (r != buffer_size_compressed) goto abort;
gets called, but no idea why.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:21:12 PM
It does work... when its 13 I get normal 14 bit I believe. You just have it set to if bit depth = 12 do this otherwise do that.

Also files opening in ACR just fine for me.

* Yea, both  files work fine... I just show one of each.. the 14 bit files play back in camera of course.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:24:59 PM
It does work... when its 13 I get normal 14 bit I believe. You just have it set to if bit depth = 12 do this otherwise do that.

It's used as actual bitdepth in write_speed_update and lv_rec_save_footer()

int speed = (res_x * res_y * raw_bitdepth/8 / 1024) * fps / 100 / 1024;

footer.frameSize = footer.xRes * footer.yRes * raw_bitdepth/8;
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:30:13 PM
But for 14 bit we don't do this?

I took the line out and it still behaves similarly... its like it just dies out after a while even at 960x540.

I see how it would be a problem then with it being 14 or not.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:32:16 PM
sized_used isn't static so I can try that... the frames come out smaller when I open them so 12 bit should be helping.

Code: [Select]
buffer_size_compressed = (size_used * raw_bitdepth) / 14;
This doesn't make sense to me actually... why are we multiplying this out? The *12/14 calculation is to find width normally.

It's just the compression ratio, it's the same for width and the whole image buffer. (to tell fwrite how much to write).

@mucher: not sure if you're serious, but all your assumptions are wrong...  :P
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:32:47 PM
But for 14 bit we don't do this?

Yes, we do.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:33:54 PM
I guess using 13 works but some calculations will be wrong?

I dunno maybe we should try 10 bit it might be less cpu intensive?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 05:36:35 PM
I think what happens (regarding 13 instead of 14) is that raw2dng overwrites the wrong value anyway, so it does work in the end.

I really doubt 10-bit is less cpu intensive, but it would be nice to have as an option, for sure.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 05:48:46 PM
I don't get it tho...  it looks like its keeping up for a while and then something happens.

Also what sucks is I can't see CPU usuage... on 600D I would be able to.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 05:54:58 PM
Instead of trying to catch this unicorn (see benchmarks), I'd say it's better to investigate how to enable 12-bit output directly from DIGIC.

You can select different raw types from c0f37014: denoised, with different byte order, compressed and so on. Some test code: http://www.magiclantern.fm/forum/index.php?topic=5614.msg39696#msg39696
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 06:05:48 PM
Hey, don't act as if we knew this all along!  ;D

DIGIC compressed RAW... I'll try not to get overexcited this time.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: scrax on May 24, 2013, 06:07:39 PM
DIGIC compressed RAW... I'll try not to get overexcited this time.
;D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 06:09:05 PM
Well, some of the raw types look kinda like this: http://www.nycresistor.com/2012/08/21/ghosts-in-the-rom/

The value at c0f37014 seems to be 6-bit, not sure what each bit means.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 06:09:59 PM
We had to test it, lol. But yea, I don't see this working right now. Forcing it lower via digic sounds like a smarter plan. Aren't those for defect detection tho?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 06:13:03 PM
Some of the raws are very noisy, others are cleaned up, but there are also scrambled ones. I don't know the image format, but I know for sure the digic can output 12-bit data.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Mayo on May 24, 2013, 06:13:44 PM
One thing is for sure: Canon's CR2 are compressed... so it's out there.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: a1ex on May 24, 2013, 06:16:07 PM
Yeah, huffman compression. Also a JPEG encoder, see http://www.magiclantern.fm/forum/index.php?topic=2803.0
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 06:16:49 PM
There are several bit depths... that craw bits has several settings but for some reason they don't apply to pics or LV. I guess have to look at lv_af raw and find the address for 6D then we can start. I can probably undo the merge since this seems like a dead end unless D does something better with the module.

If we ever figure out how to use the TTJ path :)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 24, 2013, 06:32:47 PM
It's just the compression ratio, it's the same for width and the whole image buffer. (to tell fwrite how much to write).

@mucher: not sure if you're serious, but all your assumptions are wrong...  :P

I know nothing about camera, nothing about programming, but only waiting too anxiously to see when this good ML stuff will come to 7D(for me a jpg sequence is perfectly fine, because it is picture style compatible--so one does not have to correct the clips' color one by one, and jpg sequence can be true color full resolution too)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 06:37:31 PM
7D has dual procs.. no access to raw data easily. I'd do jpeg because of speed/size... a 2K jpeg would be OK... a smaller jpeg would not.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 24, 2013, 07:07:02 PM
Also what sucks is I can't see CPU usuage... on 600D I would be able to.

There's no working build for 600d? d's asm lib only for 6d?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 08:58:01 PM
No point.. .if it doesn't run on the faster proc its not going to run on the slower one.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 24, 2013, 09:18:00 PM
No point.. .if it doesn't run on the faster proc its not going to run on the slower one.

So 100% sure it's the CPU?

Bad :(
Title: Re: 12-bit (and 10-bit) RAW video
Post by: N/A on May 24, 2013, 09:18:50 PM
Can we overclock the processors? I know it would probably kill the camera or something, but say I wanted an excuse to buy a mkiii...

Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 09:42:11 PM
Nope, wish we could.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: savale on May 24, 2013, 09:51:51 PM
How about truncating the 2 lower bits? All the deep blacks have a lot of noise anyway so why not just throw them away... It will be less cpu intensive
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 24, 2013, 09:55:52 PM
How much CPU does it need more? I mean - what's the order - is it lacking 10% more CPU speed, or 500%?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 24, 2013, 10:05:17 PM
Truncating might be worth a shot but I dunno how to do that.

Its lacking.. ie not helping over 14 bits even though working as expected. I can't even do 960x500 continuous with this. Unless D comes back and tells us what we're doing wrong its a dead end.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: bjacklee on May 24, 2013, 10:38:15 PM
tenacious "D" we need your help.. :D
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 08:00:03 AM
Another maybe immature thought. 14bit color in raw video actually means 2^14 grade brightness, according to wiki.  So the 14bit to 12bit conversion seemingly should be:

int x; //brightness grade

x = x / (2^14);
x = x * (2^12);

Now things start to make sense to me.

Just my 2 coins.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 08:01:50 AM
Or the other way:

int x = x * (2 ^ 12);
x = x / (2^14);

Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 08:08:31 AM
But if x is int, and it is divided by a very large number, it will change to value 0 perhaps,


so I would rather change it to:

int x; //YUV brightness grade

register double y = x; //temporarily change it to double precision

y = y / (2^14);
y = y * (2^12);

x = y; //change the value back into int

What do you say developers?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: KMikhail on May 25, 2013, 11:18:34 AM
Why don't you just divide by 4? Which would be similar to bit shift by 2 bits >>

Bit shift was handy since the 80286 era, when x*320 was = x <<  8 + x << 6. But since then mul and div became very cheap.

Granted, if your integer isn't right alinged (least mening bits) you have to bit shift more to the right, and then back to the left. Masking would do the job too, but would require either a register or mem op.

BTW, according to spec ARM9 has 16 32bit MAD units, should be more than capable of basic integer math, no?

EDIT:

I see, you probably are trying to get a better rounding. For that purpose you can ADD/SUB something, prior to dividing. But dividing and then multiplying a double number won't give you the result, it is caled a floating number, for a reason. There is a set of specialty rounding operations for doubles. However, I doubt double operations are fast on such hardware.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 11:51:47 AM
Why don't you just divide by 4? Which would be similar to bit shift by 2 bits >>

Bit shift was handy since the 80286 era, when x*320 was = x <<  8 + x << 6. But since then mul and div became very cheap.

Granted, if your integer isn't right alinged (least mening bits) you have to bit shift more to the right, and then back to the left. Masking would do the job too, but would require either a register or mem op.

BTW, according to spec ARM9 has 16 32bit MAD units, should be more than capable of basic integer math, no?

EDIT:

I see, you probably are trying to get a better rounding. For that purpose you can ADD/SUB something, prior to dividing. But dividing and then multiplying a double number won't give you the result, it is caled a floating number, for a reason. There is a set of specialty rounding operations for doubles. However, I doubt double operations are fast on such hardware.

What I worried was that accuracy problem, using int might not be accurate enough, and if the data were too small, divide by (2^14) might make it unusable at all, so I change it to floating point, and, I worried that float might not be accurate enough too, so I wanted it to use double instead. Modern CPU might be fast enough to handle double, my wild guess. If not, you can still change to use float, that might be accurate enough already. But if I have got the full source code, I will definitely compile it into my way, before loading it to my camera, including meddling a bit with that raw.c too.

BTW, ARM9's 16 x 32bit MAD unit is in floating point to my understanding.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: KMikhail on May 25, 2013, 12:08:01 PM
What I worried was that accuracy problem, using int might not be accurate enough, and if the data were too small, divide by (2^14) might make it unusable at all, so I change it to floating point, and, I worried that float might not be accurate enough too, so I wanted it to use double instead. Modern CPU might be fast enough to handle double, my wild guess. If not, you can still change to use float, that might be accurate enough already. But if I have got the full source code, I will definitely compile it into my way, before loading it to my camera, including meddling a bit with that raw.c too.

BTW, ARM9's 16 x 32bit MAD unit is in floating point to my understanding.

int x; // 14 bit number, in a 16/32 bit format
x = (x +2) / 4; // rounding closer to normal rounding, not CPU binary world.

// x is 12 bits now

2 => 1; 3 => 1; 4 => 1; 5 => 1; 6 => 2; 7 => 2; 8 => 2;

Double is 64 bit, and by now is totally fine on modern CPU. But for raw handling in our Canons we need more integer performance for packing all this raw data into something more space efficient. Floats, though, are nice to have at times too.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 25, 2013, 12:51:26 PM
int x; // 14 bit number, in a 16/32 bit format
x = (x +2) / 4; // rounding closer to normal rounding, not CPU binary world.

// x is 12 bits now

2 => 1; 3 => 1; 4 => 1; 5 => 1; 6 => 2; 7 => 2; 8 => 2;

Double is 64 bit, and by now is totally fine on modern CPU. But for raw handling in our Canons we need more integer performance for packing all this raw data into something more space efficient. Floats, though, are nice to have at times too.

I don't understand your code very much.

ARM9 has 16 32bit floating units, let us assume that a double calculation costs 4 times more time to calculate than a float calculation, then the 16 32bit floating units' efficiency is like 16/4 = 4, so it can mean that there are 4 double's running at the same time, that can be translated to 4 frames at the same time. If all the calculation costs around 16-32 CPU cycles per frame(including time waiting for CPU's task scheduling), the CPU should by far have enough power to process 24fps in real-time. I reckon. So my modification should work, theoretically.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: eyeland on May 25, 2013, 05:50:14 PM
We seriously need a canon engineer to defect (again?) and give us a few spoilers...
Title: Re: 12-bit (and 10-bit) RAW video
Post by: 1% on May 25, 2013, 06:13:22 PM
Why not hack at those registers in the noise thread... one of those probably enables 10/12/14 bit. Then nothing will have to run on the CPU.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 26, 2013, 09:48:43 AM
the ARM used in DIGIC has no VFP afaik.
this is an extra option when licensing ARM's IP.

so pricessing any floating point can only be done in software.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: budafilms on May 26, 2013, 10:04:11 AM
For those who have videos in 10 bits, the question is if that video it isn´t a quality of a good britrate in h264 with a prolost preset...?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: mucher on May 26, 2013, 10:28:05 AM
Then we have to stick to the original int thing :o

But the color level is still 2^14 >> 2^12 or >> 2^10, right? Mighty developers
Title: Re: 12-bit (and 10-bit) RAW video
Post by: squig on May 26, 2013, 10:31:58 AM
For those who have videos in 10 bits, the question is if that video it isn´t a quality of a good britrate in h264 with a prolost preset...?

This is a joke right?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 26, 2013, 11:46:28 AM
when cutting the least 4 bits you will usually cut off the noise that is said to be ~3-4 bits in bright areas
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 26, 2013, 01:08:50 PM
i just ran some code to read out VFP registers and can confirm now, that this HW has no VFP unit.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 26, 2013, 05:17:49 PM
i just ran some code to read out VFP registers and can confirm now, that this HW has no VFP unit.

So there's no need in it then :-) There are many other (fast) ways of bit conversion without using floating point calculations.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: IliasG on May 26, 2013, 10:14:01 PM
Of course the best is to get a ready 12bit raw or even 10 bit if it is gamma/log encoded. In fact I believe a 10bit log is better than 12bit linear.

But if this hack is difficult-impossible then the compression must be made in CPU.

I think that the way to go is 10bit log or rec.709 gamma encoding by a lookup table. No floats, no calculations .. all precalculated in at table.
Then the reverse (linearization table) can be in raw2dng or in the dedicated exif tag.

Something similar do BlackMagick and Sony and Nikon's etc.
BlackMagic encodes a 16bit linear raw in a 12bit gamma rec.709 (or log) file. The 12 log to 16 linear bitdepth sounds hyperbolic but we have to remember that BM uses only digital amplification (in fact it is just a exif tag .."Baseline exposure") for ISOs higher that the base so they need a bit more bitdepth.

ML needs 14bit linear to 12-10bit gamma or log and together with the analog ISO (up to 3200-6400) the expected file quality is at the same level as BM for mid ISO and higher at high ISO.

The question is how fast can this lookup be ??.

The lookup table with 16384 indexes and one column data with 1024 distinct values (10bit) is OK I think.

To make things better we can safely clip the 14bit data at about 1984 ( Black point is around  2048 - 64 = 1984) so any pixel with 14bit value <= 1984 will be mapped to 0 10bit. This way we can have a denser sampling of the useful range of values.

Looking at Blackmagick's and Nikon's linearisation tables ("Dump raw curve" with RawDigger) one can see that Nikon use an extended linear area at the darks and the exp area follows for midtones and highlights. It is usefull to keep data linear at the darks and without any manipulation .. usefull to avoid posterization and color shifts at the darks.

The 64 levels "below black" I choose is because so does BM (256 levels 16bit) .. and to leave footroom if something goes wrong (Black Level). It can be half (32 levels) with no problem.

Ideally we need a gamma with 16X multiplier at the linear part (so that no change needed there going from 14bit to 10) which extends around 20-40 levels (10bit) over "Black Level" (Nikon uses 400 linear levels at 14bits) .. and a relatively mild exp (^2.0) so that the highlight data stay relatively dense. Sadly my background in Maths is weak and i cannot match the linear with the exp part .. help needed ..

16X linear multiplier has the Prophoto gamma but it is linear area is very small, all falls in the "below black" area ..
Title: Re: 12-bit (and 10-bit) RAW video
Post by: xNiNELiVES on May 27, 2013, 06:20:05 AM
If the devs here get 12 bit working I think it will be possible for the 6D to shoot in 1880x817 resolution. From other photos taken in 14 and 12 bit raw I found the difference to be about 79% of the file size of a 14 bit photo. So with 1880x817 resolution requiring 55mb/s x .79 = 43.45mb/s. The highest 6D write speed I've seen is 41.7mb/s so within 2mb do you guys think it would be sustainable? Am I doing any of the math or analysis wrong?

Or is there a way to have real time 24p compression? Something to allow for higher resolutions?
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Audionut on May 27, 2013, 06:27:15 AM
Or is there a way to have real time 24p compression? Something to allow for higher resolutions?

We should leave this thread to development discussion of making 12/10 bit encoding possible, before considering what advantages it will have with RAW recording.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: vicnaum on May 27, 2013, 06:36:20 AM
I think that the way to go is 10bit log or rec.709 gamma encoding by a lookup table. No floats, no calculations .. all precalculated in at table.
Then the reverse (linearization table) can be in raw2dng or in the dedicated exif tag.

Yeah, I also thought about the lookup table method. Should be faster than math.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: xNiNELiVES on May 27, 2013, 06:39:02 AM
We should leave this thread to development discussion of making 12/10 bit encoding possible, before considering what advantages it will have with RAW recording.

Ok :) it may be an incentive for other developers not yet dedicated implementing this feature yet, then again they probably already knew...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 27, 2013, 07:38:24 AM
Hello, Masters,

I just did a simple math, to see which part will change its value while converting from 14bit to 12bit or 10bit, and I use this formula:

We will try to find out which brightness value in 14bit world, after converting to 12bit by using int formula like x * 2^12 / 2^14, will change to the smallest value, which either 1 or 0 in int world:

int x; //the value to be found out

formula:

x * (2^12) / (2^14) = 1

the result:

x = 4

So any value in the original 14bit data smaller than 4 will be converted into 0 in the final 12bit data format, which totally consists 4 values: 0, 1, 2, 3

That means that in the original 14bit data, only brightness level from 0, 1, 2, 3 will be converted into value 0 in the final 12bit format. And these 4 values only consists only minimum fraction comparing with 14bit world, which is about 0.024%(4 / 2^14) of all the values in 14bit world, and its final form, which is total black level value = 0. What the good news is this part of data will not be transfer by HDMI interface(I heard that HDMI usually will not transfer brightness level from 0 - 5 in all the 0 - 255 values to the final equipments of display (like TV, projectors...).

To see the 10 bit scenerio:

formula:

x * (2^10) / (2^14) = 1

x = 16

There  will be totally 16 values converted into 0 while using int formula x * (2^10) / (2^14), that consists 0.098% of all the values in the 14bit data format, and these values might be neglected by HDMI interface anyway.

So, I think that using int formula x * (2^12) / (2^14),  or x * (2^10) / (2^14) into 10bit respecfully,  might be very viable methods.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 27, 2013, 07:48:17 AM
The bad news is that every 4 value in 14bit data will change to the same value in 12bit form, but then I realize that 12bit world has 4 times less value than 14bit world anyway.

What do you say masters? ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on May 27, 2013, 08:04:34 AM
HDMI doesn't belong here in any way, cause it's a display transfer format, but raw is not made for display, but for processing first. For transfer via HDMI sRGB values are compressed to 16-235 (in Rec.709 standard), and then recompressed back to full 0-255 range for display in TV (as far as I remember).

It's obvious that we'll have only every 16 number if we supress 14->10bit (each bit is a power of 2, so 2^4bits = 16), so we'll get 1024 values instead of 16535. But the good news is that's still more than sRGB 255 (although, need to try it, how it will Raw-convert, will there be banding, etc, etc).

Btw, to eliminate banding, we can use a bit of dithering while using the lookup table (but that's not so good as dithering in picture-space).

Anyways, all these are idle talks, until we get confirmation that this lookup-table thing will be realizable and fast on our little slow ARMs.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 27, 2013, 08:30:59 AM
Another thought.

In x * (2^12) / (2^14), one can only multiply first, or every value will become 0 if one divide by 2^14 first. So the worse scenerio can be this:

The largest number in 14bit data times 2^12, and then divide by 2^14, can be pains-takingly slow, but worth a try.

the largest value: 2^14 * 2^12 / 2 ^ 14, that means 16384 * 4096(that is 67108864
)/ 16384

And we can use precalculated values in the programs too, like instead of using x * (2^12) / (2^14) for easier reading, we can always use x * 4096 / 16384 instead.

As a last resort, we can use unsigned int instead of int, can we?

ARM might possibly be fast enough, because int is pretty fast to run, usually 1-a few CPU cycles in x86 world, and if we use unsigned int, it can be even faster.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 27, 2013, 08:37:21 AM
As the rounding issue, I don't know what these DIGICs will do, probably it will delete everything behind the decimal point to speed up running. But the difference can be like 2394 or 2395, can still be far more accurate than the human eyes can tell.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 27, 2013, 09:02:32 AM
@mucher:
not sure what you want to do with your calculation.
but the data has to be unshifted from the 32 bit words and doing a simple LSR#2 while building the 14 bit value
will get you the value as 12 bit and the same result.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: savale on May 27, 2013, 09:21:04 AM
First step:

mu suggestion: make it as fast as possible and see if it's feasible. So just truncate the 4 LSB bits using a bitmask / bitshifting:
something like this:

Code: [Select]
/* let's say the variable fourteenBitsValue contains the 14 bit value */
tenBitsValue = (fourteenBitsValue & 0b11111111110000) >>> 4; /* zero the 4 LSB bits and shift 4 bits */
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 27, 2013, 10:42:53 AM
Okay, back to rounding.

Lets simplify it for understanding - will work it out in a decimal system first.

We have 1001 and 1005 and want to shift it to the right by 1 digit, divide by 10. The double result of division is 100.1 and 100.5, nearest integers after NORMAL math rounding are 100 and 101. However, if you want to operate in the integer space you will have 100 and... 100, since CPU rounds DOWN to next integer. To avoid this we want to add 5 (half of the decimal base in power of the shift) to get normal rounding: 1001+5=1006 >> 1 = 100, 1005+5=1010 >> 1 = 101. Now, if you wish, you can shift them back. I am not sure how you want to align the final result (left or right).

To shift by 2 bits, divide by 4, you need to add 2^2 / 2= 2, before you shift, to preserve correct mathematical rounding. Then, if you need to re-align most meaningful digit to the left, you can shift it back.

I hope it is clear now. But, look-up table might be indeed a better alternative, unless referencing memory via ptr is expensive, I am not familiar with efficiency of DIGIC5+.

BTW why it was so important to round it so well? Perhaps for 10 bits?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mogs on May 27, 2013, 11:51:10 AM
The packed data looks pretty messy.  Dealing with this the most efficient way seems more important than rounding errors.

Once the packed data has been pulled apart, before repacking at a lower bit depth, it may be worth considering implementing something like this for a cleaner roll-off in the highlights.

(https://www.apertus.org/sites/default/files/dr-illustration-v03.png)
(from the apertus site, http://apertus.org)

If the knees were aligned with bit boundaries it should be possible to implement with only shifts and adds?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 27, 2013, 12:20:24 PM
The image is misleading.

1) Nowadays DR of digital sensors is enormous, compared to what it used to be, and outperforms many, if not most of the film (per area unit). Some BW might be still better, when other properties (grain) are poor.
2) Both linear digital and enhanced digital should stretch from (0,0) to the same topmost rightmost point, it is an allocation of levels per bits what makes it different (linear vs. curved), not the clipping (defined at the moment of reading the sensor).

At this moment I would prioritize ability to compress RAW 14 bit in some lossless/lossy way, and then chopping some bits off and possibly applying LUT. IMHO, of course.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 27, 2013, 01:11:43 PM
Code: [Select]
/* zero the 4 LSB bits and shift 4 bits */

not necessary as the lowest bits are already cut off when shifting right...

seriously, i dont get what you all are trying to calculate.
if its about proper rounding, well then add 8 before shifting 4 bits right.
any additional operation like adding, masking, shifting, whatever makes the code half as fast.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on May 27, 2013, 01:57:31 PM
Can someone make a test bitshifting build? I guess it shouldn't even needed to be based on d's object file.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 27, 2013, 02:05:43 PM
Instead of trying to catch this unicorn (see benchmarks), I'd say it's better to investigate how to enable 12-bit output directly from DIGIC.

You can select different raw types from c0f37014: denoised, with different byte order, compressed and so on. Some test code: http://www.magiclantern.fm/forum/index.php?topic=5614.msg39696#msg39696

This.

If we could make CMOS output shifted RAW data, then it would be much easier. It would be much easier even if it could output padded & aligned pixel data (so we don't have to "dig out" the RGB values from packed Bayer-data)., thus making it easier to shift diown to 10 bit in CPU
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 27, 2013, 02:29:09 PM
Can someone make a test bitshifting build? I guess it shouldn't even needed to be based on d's object file.

what do you mean with that? thats exactly what d has done.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: g3gg0 on May 27, 2013, 02:31:37 PM
If we could make CMOS output shifted RAW data, then it would be much easier. It would be much easier even if it could output padded & aligned pixel data (so we don't have to "dig out" the RGB values from packed Bayer-data)., thus making it easier to shift diown to 10 bit in CPU

if you want padded and aligned data, this means you will have a higher data rate.

(assuming you want it to be 16 bit aligned)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on May 27, 2013, 06:04:27 PM
...

I hope it is clear now. But, look-up table might be indeed a better alternative, unless referencing memory via ptr is expensive, I am not familiar with efficiency of DIGIC5+.

BTW why it was so important to round it so well? Perhaps for 10 bits?

In case it is a question for me, yes with 10bits raw bayer we need very careful rounding and even with careful rounding 10bits depth can be inadequate. While with RGB 10bits can be (and are) OK with raw Bayer "master" there is the need to apply some heavy manipulations not needed for the ready to use RGB file. Demosaic, WB (which is a 1+ stop digital amplification for the weak channel), Color transformation from Bayer R-G1-B-G2 TO RGB (once again about 1 stop amp for the weak channel)

But with a lookup table we can have untouched the low values (keep the sampling at the same density as the 14bit starting data) where the problem is significant and gamma encode the rest.

We can even develop a curve imitating the S-log ..

Say we go for 14 to 10bit,
- we can clip at 32 (or even less) 14bit levels below "Black Level" instead of 2048 as it is now. Just map any 14bit level lower than 2016 (14bit) to zero. The remaining levels are 16384-2016 = 14368 which will be mapped (with any
- the first (say 128) 14bit values stay untouched by using a linear part with a slope of 16 y= 16*x
- then continue with a gamma .. say y= x^1/2
- if we want the S-curve then at the last 1/2 or 1/3 stop we invert the gamma y=x^2

Try to dump Nikon's D5200 raw curve with Rawdigger .. it's as described before .. linear-gamma-inverted gamma. Same proposal as AXIOM's Smart Dynamic Range

Let's hope the lookup table can be managed fastly by Digic .. can any expert give an opinion on this ??.
I think for a PC (X86 architecture) a lookup table is almost always preferable/faster.

One more compression scheme crosses my mind... Can we, during writing the raw data, use variable bit depth adapted to the real needs (values up to 256 really need 8bits). Then the same lookup table can be expanded with one more field giving this info ... :):).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 27, 2013, 07:40:26 PM
any additional operation like adding, masking, shifting, whatever makes the code half as fast.
and another memory access outside TCM memory will slow it down again a cycle or more.

so: 1/3th the performance.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: Grunf on May 27, 2013, 10:30:41 PM
if you want padded and aligned data, this means you will have a higher data rate.

(assuming you want it to be 16 bit aligned)

It would lead to more data being transferred internally, but it would also mean that CPU doesn't need to do unpacking of the primaries, right?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jgerstel on May 27, 2013, 11:24:21 PM
Currently CF card IO is the bottleneck….
Imagine you can solve the 14 to 10 bit issue: this bottleneck is solved and resolutions can explode: what about 5d3 4K camera….. just an idea... :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 27, 2013, 11:33:58 PM
I'd settle for acceptable resolutions on SD cards.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 28, 2013, 12:08:34 AM
You know mates, what realy stumbles me.

With 5D3 we have 6fps at 22mp @ raw with lossless compression. 1920*1280 is 9 times smaller, so the same codec/routines should be able to pull up to (& who knows, may be beyond) 54fps, most likely 60fps, if you'd cut some fancy features. With 70% compression.

Canon definitely can pull, if it would care.

Is there any way to access RAW routines of original hardware/firmware?

Also, is the code multi-threaded? Apparently all these slowdowns with GUI/etc. is due to serial perferming?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 28, 2013, 12:44:51 AM
2 different processes. LV raw/photo raw

Maybe possible to shrink the actual read area of the sensor and then you get something like 640x480 120fps really line skipped. Some demo canon fw had this and it wasn't stable.

We don't know how to do that yet.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 28, 2013, 01:03:31 AM
2 different processes. LV raw/photo raw
We don't know how to do that yet.

You probably saw my tests of video raw vs. photo raw. Binning definitely averages several pixels, having noise closer to resized 22 mp raw. I, personally, don't consider 1:1 mode valuable for me: I want to use full sensor and full coverage of my glass, that's what makes 5D3 standing out of everything else,  including DR, according to my test (1-2 stops better noise in blacks than 1:1).

So, if by any chance photo raw engine would be accessed to compress LV raw it won't slow it down significantly and give an outstanding ability to open these raws in DPP, where color is so nice and demosaicing works so well. Limitless abilities.

This would be more than worthy of a good donation :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 28, 2013, 05:18:43 AM
I am not expert in computing, but as far as I have known, the LUT things looks very good, but that sounds that we will have a lot of codes like: if x<y0 and if x>y1, x=y2, these kinds of codes are known to stall the CPU to a halt. d's method should be more suitable for CPU to do, and it already working real time reportedly by several one here who have actually loaded it into camera and tried, and we only don't know why the program halts. Maybe d can mercifully display his source code, maybe there will be someone who can look closely to see if there is something more we can do. d's method sound more straight-forward to me, and easier. To achieve better rounding and accuracy probably, soundingly to me, we only need to change from (x * 12) / 14 to (x * 2^12) / 2 ^ 14, and the cpu might be able to collaborate with that.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: savale on May 28, 2013, 08:11:42 AM
what do you mean with that? thats exactly what d has done.

Sorry I missed that... as I understand now the bits are truncated before putting them into the buffer. Is that correct?

I have two new ideas that might be worth checking:

1) what about manipulating the pointer instead of shifting. Would it be faster?

something like this:

- write 14 bits
- move the write pointer backwards 4 positions
- write 14 bits (which overwrite 4 bits of the previous value)
- move the write pointer backwards 4 positions
etc...

Details must be sorted out to make this work correctly (LSB / MSB)

2) a second thought: what about doing the conversion 14 to 10 bit at time when written to flash instead at time it's written into the buffer? I can imagine the results will be different. (I am not sure better, but might be worth a try)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on May 28, 2013, 08:16:00 AM
I am not expert in computing, but as far as I have known, the LUT things looks very good, but that sounds that we will have a lot of codes like: if x<y0 and if x>y1, x=y2, these kinds of codes are known to stall the CPU to a halt. d's method should be more suitable for CPU to do, and it already working real time reportedly by several one here who have actually loaded it into camera and tried, and we only don't know why the program halts. Maybe d can mercifully display his source code, maybe there will be someone who can look closely to see if there is something more we can do. d's method sound more straight-forward to me, and easier. To achieve better rounding and accuracy probably, soundingly to me, we only need to change from (x * 12) / 14 to (x * 2^12) / 2 ^ 14, and the cpu might be able to collaborate with that.

Nope, LUT things will be just y = lut [ x ], and that's all.

And about the calculations you mention - they are easily done (like g3gg0 said) with bit shifting:
Raw14b: 13598 = 0011 0101 0001 1110
Shift 2 bits right =>12bit = 0000 1101 0100 0111 = 3399
13598 * 2^12 / 2^14 = 13598 * 4096 / 16384 = 3399.5 = 3400

So no need for calculations at all. Rsh solves it on cpu-elementary level.
If you need better rounding (like said), add half of the bits (2bits = 4, so add 2) to number before shifting:
(13598+2) = 13600
13600 Rsh 2 = 3400
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 28, 2013, 09:27:11 AM
2) a second thought: what about doing the conversion 14 to 10 bit at time when written to flash instead at time it's written into the buffer? I can imagine the results will be different. (I am not sure better, but might be worth a try)

sorry, but read into the raw_rec code.
as soon you understood how all that works, you will understand that these ideas are obsolete ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: savale on May 28, 2013, 10:32:23 AM
hehe ok :P I will
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Northernlight on May 28, 2013, 01:01:30 PM
Hi, I came across this article reg. the Canon C500 and it's 10-bit RAW capabilites.
Not sure if this was relevant under this thread or under discussion of ETTR, but I found it interesting (and I am sure most you already know this).

http://nofilmschool.com/2012/11/canon-c500-shipping-raw-4k/

"Adding gain adjustments at the sensor level produces a consistent stop range above and below the middle grey level, even at high ISOs, and reduces the overall noise in the image."

As I understand this; Canon's RAW format is special in the sense that the C500 add gain at sensor level before outputting the RAW stream, is this in a way, approaching a little more to ETTR? (to improve DR and SNR)

Q1: Could this "approach" be possible to do also with the RAW module from ML ? (if you hopefully succeed with 10bit RAW!)

BTW! I have no chance to follow the technical discussion in this thread, but:
Q2: Do you at this point have any estimates of what the DNG file size could be eg. @1920x1080, should you succeed with 10bit RAW?
(I am just curious to where 10bit might take us in case of fps/resolution)
Title: Re: 12-bit (and 10-bit) RAW video
Post by: tin2tin on May 28, 2013, 01:18:17 PM
BTW! I have no chance to follow the technical discussion in this thread, but:
Q2: Do you at this point have any estimates of what the DNG file size could be eg. @1920x1080, should you succeed with 10bit RAW?

In theory, SD cards can give 22MB/s or 961194 bytes per frame. 1280x600 in 10 bits gives 960000 bytes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 28, 2013, 02:50:00 PM
You guys could be looking at the registers to see if 10bit can be made in camera.. there are functions like set_craw bits, etc. Just have to find the correct one.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on May 28, 2013, 03:02:37 PM
You guys could be looking at the registers to see if 10bit can be made in camera.. there are functions like set_craw bits, etc. Just have to find the correct one.

Could you tell please where can we look at these?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on May 28, 2013, 04:41:33 PM

Q2: Do you at this point have any estimates of what the DNG file size could be eg. @1920x1080, should you succeed with 10bit RAW?
(I am just curious to where 10bit might take us in case of fps/resolution)

I have found this:

video bitrate calculator, shows frame size and bitrate with any resolution, framerate and bit depth.

http://web.forret.com/tools/video_fps.asp?width=1720&height=720&fps=24&space=raw&depth=14
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: xNiNELiVES on May 28, 2013, 05:24:28 PM
There is a specific canon raw video calculator.  I forgot where I found it. Google magic lantern raw and go to news.  There's dome Germany website and on the website there is a link to use it.

Never mind here it is: http://www.slashcam.de/tools/ml-raw-calc.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on May 28, 2013, 05:39:03 PM
Yep, but it doesnt have bit depth (or I cannot find this) so cannot compare 14 to 12 and 10 bits. This one I have found has this feature so we can calculate theoretical performance for 12 and 10 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Audionut on May 28, 2013, 05:42:53 PM
Guys, lets please stop with the, "oh wow with only 3 bit recording we can record 8k resolution" type posts, and please leave this thread solely for actual development discussion pertaining to the implementation of this feature.

Thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: inozcenterod on May 28, 2013, 06:40:01 PM
hello everybody im from VEN , basic knowedge of c++, how can I start with the ML 10 bits source, and if is possible to use some kind of lossles compression algorithm to minimize the size of data, or level up the cpu voltaje. (SORRY for my english)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 28, 2013, 07:07:08 PM
Quote
Could you tell please where can we look at these?

Look at regs touched by lv_save_raw and look for setcraw in strings.. there are 2 functions that set bits but it doesn't seem to be doing anything.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on May 28, 2013, 07:58:49 PM
Hi, I came across this article reg. the Canon C500 and it's 10-bit RAW capabilites.
Not sure if this was relevant under this thread or under discussion of ETTR, but I found it interesting (and I am sure most you already know this).

http://nofilmschool.com/2012/11/canon-c500-shipping-raw-4k/

"Adding gain adjustments at the sensor level produces a consistent stop range above and below the middle grey level, even at high ISOs, and reduces the overall noise in the image."

As I understand this; Canon's RAW format is special in the sense that the C500 add gain at sensor level before outputting the RAW stream, is this in a way, approaching a little more to ETTR? (to improve DR and SNR)

Q1: Could this "approach" be possible to do also with the RAW module from ML ? (if you hopefully succeed with 10bit RAW!)

BTW! I have no chance to follow the technical discussion in this thread, but:
Q2: Do you at this point have any estimates of what the DNG file size could be eg. @1920x1080, should you succeed with 10bit RAW?
(I am just curious to where 10bit might take us in case of fps/resolution)

I wouldn't say Canon does this at the sensor Level, but at the Digic after digitization. There, they use the full 14bit depth to make calculations that a raw converter would have to do, which is problematic on 10bit data. They fight the problems described below

...., yes with 10bits raw bayer we need very careful rounding and even with careful rounding 10bits depth can be inadequate. While with RGB 10bits can be (and are) OK with raw Bayer "master" there is the need to apply some heavy manipulations not needed for the ready to use RGB file. Demosaic, WB (which is a 1+ stop digital amplification for the weak channel), Color transformation from Bayer R-G1-B-G2 TO RGB (once again about 1 stop amp for the weak channel)

But with a lookup table we can have untouched the low values (keep the sampling at the same density as the 14bit starting data) where the problem is significant and gamma encode the rest.

Canon say that:

- They have "baked" in raw the ISO.
Well, in photo mode, Canon 5DIII has the ISO fully "baked" in raw for analog ISO settings (100, 200 ... 12800, no gaps in raw histograms) and partially for intermediate and expanded ISOs where its mixed analog-digiral (we can see gaps in the raw histograms).
So do Sony (and others with Sony sensors) but it happens that they stop analog ISOs lower (1600 is usual).
BlackMagic uses only the base analog gain and every ISO higher than the basic is just a metadata flag (look for "Baseline exposure" in Exif)
Canon 1Dx uses analog ISO up to 51200 and for intermediate ISOs also. All raw histograms are without gaps so either it is analog ISO up to 51200 or the DAC digitize at 16bits and after 16to14bit conversion the histogram looks full (If I had to I'd put a bet on the second).  Maybe C500 goes up to same analog ISO also.

- They apply WB on the full bitdepth (14 or 16bit) raw data and then downconvert to 10bit. They gain accuracy this way and better tonality over all the range.
WB is in fact multipliers at each channel. For Canon's "Daylight" the multipliers are about R=2.0 G=1.0 B=1.5. The first thing a raw converter will do is apply these multipliers so the R & B channels will loose sampling density accordingly.

We could also "bake" a midway WB (say temp.4500K) in the proposed LUT but then we would need one LUT for each color and (if separate LUTs was possible) in the end any known raw converter would get crazy with WB/color transformations ... until someone decides to support the "crazy" ML-DNGs ..

- They use a log LUT on the 10bit output. This means they keep dense sampling where it counts (at the darks) and the sampling density decreases logarithmically as we go upright to the highlights. This way they can fit a 13-14 stop DR in 10bit log file.
It is the same we will do with the proposed 14 to 10bit LUT .. wish it is fast enough with the available resources ...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 28, 2013, 08:22:42 PM
...
The best way to use 10 bit space is to calculate the topmost value for each channel of the RAW, record it elsewhere and align all channels to the 11111111111111b. Likely after applying 14 bit LUT for highlights compression, then cleverly getting rid of last 4 bits. In the post, upon conversion to a normal DNG (which in fact might have all information in the lower of 14 bits after shiftin channels back and appying inverse LUT!) opposite operations should be performed.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on May 28, 2013, 10:17:01 PM
.

I suppose you talk generally and assuming the available computation resources are enough, not like what ML has currently available ..

Just aligning to the topmost channel is in many cases ineffective as even a single specular highlight drives all channels to align at the White Level cutoff point. Take a look at raw histograms of many video-raw samples, it happens very frequently for either all chanells or the 2 of 3. And  this is usually the case of a high DR shot when we mostly need the best behavior.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on May 28, 2013, 11:21:30 PM
another way could be:
 - converting the 14 bit value into a CString by using sprintf and new CString(buffer)
 - then using strlen to get the string length
 - subtract 4 from that length
 - copy that number of characters in an UTF-16 string
 - code some utf16_atoi
 - and save that value to CF card.

this can be done in VB too.

SCNR
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on May 28, 2013, 11:47:21 PM
@ g3gg0,

and .. will this be fast enough ?.

What we all wonder is ... if ML cannot find the lower bit-depth version ready from Canon (and this version can be of an acceptable quality or not at all ..), then which is the way to go ??
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on May 28, 2013, 11:49:48 PM
I cannot find that things mentioned in EOS SDK
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on May 29, 2013, 12:57:08 AM
Eos SDK? You need dumps from your camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KMikhail on May 29, 2013, 06:32:36 PM
I suppose you talk generally and assuming the available computation resources are enough, not like what ML has currently available ..

Just aligning to the topmost channel is in many cases ineffective as even a single specular highlight drives all channels to align at the White Level cutoff point.

1) Generally, yes, aligning to the right might be not possible on the embedded cpu, though, could be done cleverly.
2) Nonetheless, that's the best way in general.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: fotojohni on June 02, 2013, 10:00:28 AM
Isn't the camera converting to 8bit during h.264 encoding?  Maybe you could just grab it there.  Anyway, I believe this is missing link.  Made a 2 minute youtube vid and I ended up with hundreds of GB's of footage in various formats along my workflow production line.  10 bit is better for a lot of reasons.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Clemens on June 04, 2013, 10:03:38 AM
Is there some progress on this topic? Would really love to test a 10 or 12 Bit build and help you with some feedback.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: xNiNELiVES on June 04, 2013, 06:32:06 PM
Same. I would like to test.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on June 04, 2013, 06:58:55 PM
The asm module wasn't fast enough... tried it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: xNiNELiVES on June 04, 2013, 07:06:09 PM
The asm module wasn't fast enough... tried it.

So what does this mean.  It's not possible for live conversion to 12 bit?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1% on June 04, 2013, 07:21:08 PM
Yea, it just stopped working immediately.. it never flushed a single buffer. Maybe on 255M you'd get a decent string of frames.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: fotojohni on June 04, 2013, 11:03:37 PM
I think someone may have mentioned simply truncating the lower (or upper) two bits of data.  It's a loss of 2-3 stops of DR, but maybe worth it for low DR scenes where you don't need that extra data anyway.  I actually run into this situation quite frequently.

Another option could be to truncate the U+V (YUV, I'm assuming) then do a better algorithm for Y channel, just a thought.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on June 04, 2013, 11:10:47 PM
in raw data there is no YUV. its RGB. thats what is all about.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vicnaum on June 05, 2013, 07:43:08 AM
Yea, it just stopped working immediately.. it never flushed a single buffer. Maybe on 255M you'd get a decent string of frames.

So does it work slowly, or doesn't work at all? Maybe a buggy asm? If try at 320x240 or 160x120 res - is the concept working?

Maybe d's from canon, and he gave us an incomplete puzzle that we need to solve ourselves?
Or maybe d's from canon and he gave us a non working code sample so we lose our heart and don't try 10-12 bit raw again?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Audionut on June 05, 2013, 07:50:40 AM
Lets stop with the useless speculation and keep it on topic thanks.

The activity in this thread should make it pretty clear that the developers are interested in the idea, and being able to make it happen.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on June 05, 2013, 10:12:05 AM
Nope, LUT things will be just y = lut [ x ], and that's all.

And about the calculations you mention - they are easily done (like g3gg0 said) with bit shifting:
Raw14b: 13598 = 0011 0101 0001 1110
Shift 2 bits right =>12bit = 0000 1101 0100 0111 = 3399
13598 * 2^12 / 2^14 = 13598 * 4096 / 16384 = 3399.5 = 3400

So no need for calculations at all. Rsh solves it on cpu-elementary level.
If you need better rounding (like said), add half of the bits (2bits = 4, so add 2) to number before shifting:
(13598+2) = 13600
13600 Rsh 2 = 3400

As per vicnaum. If this works, there will be no loss in DR, just lose lum levels because of that 10bit's less levels, but I understand very little of that hex level calculation and I have no time to check its validity.  And if vicnaum is right, there is no calculations to do except moving bits in HEX level, which sounds very promising to me.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: glnf on June 07, 2013, 01:50:57 AM
I don't want to distract anyone working on this. I just like to add a little clarifications for those that are not clear about the goal of this operation. It is actually a very simple idea. The RAW data from the sensor is 14 bits long. Each 14 bit number represents the grey value of a red, green or blue image pixel.

So a typical value looks like this: 1010 0101 0100 10 (the spaces are just for reading convenience)

We could translate the value into the decimal system (it is 10578) but there is not really a need for this. All we have to do is to get rid of the last two digits. So we end up with 1010 0101 0100. That is quite a different value. We write that onto the card. Then, when we retrieve the values from the card we simply add 00 at the end of the number, making it: 1010 0101 0100 00. This is almost the same value (decimal 10576) as initially. So we loose a tiny bit of precision but we save some space and reduce data rate.

There is by the way no need to do any rounding. Chopping off the lowest 2 (or alternatively 4) bits before writing the value to the card and replacing it with 2 or 4 "0" for further processing on the computer does the trick. Instead of adding "0" it would be slightly more elegant to add random values, hence mimicking (a very low level of) noise.

So far goes the theory. Since I haven't done any experiments with the camera I don't know if there are any additional obstacles, eg. a reverse or unusual bit order. (I remember reading something about black level in this thread and that sounded a bit odd to me. And no reason to come up with logs or whatever. That only harms the beauty of this trick.) Cheers, g
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Critical Point on June 07, 2013, 03:46:19 AM
So, lets say on a 600D, what would it mean a 10 or 12 bit raw ? What kind of resolutions are we talking about for 24 fps ? Please don't say 1080p...because I'm going to have a heart attack. :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hammermina on June 07, 2013, 06:31:59 AM
@ critical point

there will be no 1080/24p i think the max will be 720p if this mode will work
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: xNiNELiVES on June 07, 2013, 07:27:31 AM
Not to be a stickler guys but audio nut just told everyone to keep useless speculations aside. It'd best if we just keep out of this development discussion. If you were to open up a thread dedicated to general discussion of this topic that would be permissible.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mucher on June 10, 2013, 12:32:45 AM
Maybe we can use some precalculated values. It is seemingly that DIGICs has accuracy problems when multiply too many times. But I am not sure how it will do in dividing, probably it will be fine. The equation changing to 10bit, x * 2^10 / 2^14, can actually turn into two parts x * ( 2^10 / 2^14), and 2^10 / 2 ^14 equals to 1/16, so the equation can change to x* 1 / 16, and that equals to x / 16, and likewise, changing to 12bit means x/4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on June 10, 2013, 12:51:42 AM
the last time:
devs are aware how to convert data from 14 bit to 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, and even 0 bits.
it is no riddle. it is no challenge.

can do this while sleeping, we can do this while steering a car and we can even do this in electrical circuits.
seriously.

the only issue is: THE CPU IS SLOW.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: driftwood on June 10, 2013, 12:55:47 AM
Exactly, the cpu is perpetually at  96% or greater ceiling at the moment.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: necronomfive on June 10, 2013, 12:59:49 AM
...and that equals to x / 16, and likewise, changing to 12bit means x/4

Or in other words: shift right by 4 or 2 bits....  ::)

Concerning precalculated values: if CPU bus speed to memory is an issue (just look at the plain memcpy values), then I don't see how ADDITIONAL external memory fetches from a LUT will help this cause in any way.

I also see no way how 12-bit/10-bit compression will ever be achieved with the CPU. It was only designed to do GUI/filesystem management related stuff. The real processing is done by highly dedicated signal processing blocks inside the DIGIC. As said before, the only chance to get this working is finding an undocumented register bit in DIGIC which reduces the bit width of written sensor data.

With no documentation on DIGIC, this is like finding a small coin inside a barn full of hay, without any kind of guaranty IF this coin exists at all.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: necronomfive on June 10, 2013, 01:10:26 AM
.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: IliasG on June 11, 2013, 10:08:53 PM
...
Concerning precalculated values: if CPU bus speed to memory is an issue (just look at the plain memcpy values), then I don't see how ADDITIONAL external memory fetches from a LUT will help this cause in any way.

I also see no way how 12-bit/10-bit compression will ever be achieved with the CPU. It was only designed to do GUI/filesystem management related stuff. The real processing is done by highly dedicated signal processing blocks inside the DIGIC. As said before, the only chance to get this working is finding an undocumented register bit in DIGIC which reduces the bit width of written sensor data.

With no documentation on DIGIC, this is like finding a small coin inside a barn full of hay, without any kind of guaranty IF this coin exists at all.

Since I am not familiar with 5DIII internals and I have already asked 2-3 times how much time consuming would the LUT proposal be (no answer yet), can you give an estimation ??.

A clear answer from the start would keep the thread compact and on target ..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Chucho on June 12, 2013, 08:21:40 AM
I'm a little behide here, so changing the 14bit can be done by code but is the big challenge to find a way to do it in camera? If you search for CrawBit Invalid it will give you the acceptable arguments for FA_SetCRawBitNum() 0x10=16bit, 0xE=14bit, 0xC=12bit, and 0xA=10bit. Doing something this
void foo();
{
        call("FA_CaptureTestImage");                  
   call("FA_GetCrawBuf");
   call("FA_SetCRawBitNum",0xE);
        call("FA_DefectsTestImage");
   call("FA_CreateTestImage");
}

And then looking for the registers that change. Would that give us some sort of hint?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tin2tin on June 14, 2013, 08:12:12 AM
@ Chucho
Looks like a very interesting finding. I wonder what the devs can make of it?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on June 14, 2013, 10:16:44 AM
i am not 100% sure but i remember smth like that.
iirc it was setting up a bunch of registers that are related to the twoadd etc stuff.
so it was not a simple register change and the CMOS sends us 10 bit, but the parameter to one of those not-yet understoor modules.
if we understand what they do, we can redirect EDMAC output from CMOS to these modules etc.

that involves a lot of register setup.
Title: Re: 12-bit (and 10-bit) RAW video
Post by: rudi on June 14, 2013, 12:48:24 PM
yeah..
well, i implemented an memcpy using LDMIA/STMIA for LV buffer copying and this was a dead end.
so i tried to get EDMAC working.

I don´t want to keep the idea of a 12 bit shifter alive, but i´m just interested in how much MB/s you recieved with LDMIA/STMIA.
the mysterious "d" seemed to reach about 30-40MB/s on an smaller modell.
and as far as i can see from the debug scrennshots a memcopy can reach over 70 MB/s. Ist that correct. Was that LDMIA/STMIA?
Some years ago a made heavy X86/MMX/SSE2 optimisations and looked some days ago at ARM optimizing.
As far as i can see (and what "d" did) you can sqeeze out a lot of bubbles and stalls out of hand optimizing.
Shifts are also free in ARM. Nearly every instruction can be combined with the barrel shifter at no cost. (I´m sure you are ware of this).
And you have a lot of Cache/Flush/Prefill Options in ARM.
But again, i don´t want to reanimate the idea, i´m just curious how fast optimized LDMIA/STMIA / memcpy on a 5DMK3 can be.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bjacklee on September 27, 2013, 02:47:02 PM
If only this is possible.. :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: 1berto on September 30, 2013, 12:36:42 PM
I don´t understant nothing of this, but IMO compression is the way to go, thats was the secret of RED cameras and the problem of BMCC workflow...

If you find a way to compress the raw data so that the image don't get compromissed and at the same time can be captured for the card.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Midphase on October 01, 2013, 10:28:43 AM
Sigh, for the nth time...it's not possible since the Canon CPU's aren't fast enough to perform the necessary calculations. They can barely keep up with writing the data to a CF card as it is. :o
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: pascal on October 01, 2013, 11:24:54 AM
Sigh, for the nth time...it's not possible since the Canon CPU's aren't fast enough to perform the necessary calculations. They can barely keep up with writing the data to a CF card as it is. :o
On the contrary if you add cpu time to conversion you remove cpu time from data write.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Audionut on October 01, 2013, 11:57:24 AM
Read the thread fully before making assumptions!

Feel free to PM me to have this thread reopened, if you have code that makes bitdepth reduction in camera possible.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 11, 2016, 05:37:11 PM
[...] to have this thread reopened, if you have code that makes bitdepth reduction in camera possible.

I think I've got such code, based on g3gg0's raw_twk (http://magiclantern.fm/forum/index.php?topic=13163.0) experiments.

https://bitbucket.org/hudson/magic-lantern/commits/c6fbba9

Timing: 120ms for a full-sized image (5936x3950).

Help is welcome - not just by testing (or asking for) the finished thing, but by trying to understand how Canon's image processor works, how to call it for other purposes (it can do lots of operations, such as add, sum, min, max, debayer, curves, filters, (m)jpeg compression, lossless raw compression... the main difficulty is understanding how to configure these image processing modules), and - of course - by turning this proof of concept into something useful.

Other relevant pages:

http://magiclantern.wikia.com/wiki/Register_Map
http://www.magiclantern.fm/forum/index.php?topic=13408
http://www.magiclantern.fm/forum/index.php?topic=6740.0
http://www.magiclantern.fm/forum/index.php?topic=16428.msg159735#msg159735

Happy hacking.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: markodarko on May 11, 2016, 07:05:22 PM
I've just read the entire thread (I wasn't around here in 2013) and I'm a little confused. Without standing on anyone's toes by not helping from a development POV, can someone explain why @d apparently got this working but others say that the CPU is not fast enough? (Or did I misread?)

Thanks,

Mark.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 11, 2016, 07:33:26 PM
Read the benchmarks and do the math.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: markodarko on May 11, 2016, 08:12:59 PM
Read the benchmarks and do the math.

:-D you take me for a learn-ed man, Mr. @a1ex.

I did take a look at these:

(http://a1ex.magiclantern.fm/bleeding-edge/raw/bench14to12-playback.png) (http://a1ex.magiclantern.fm/bleeding-edge/raw/bench14to12-movie24p.png)

Impressive.

...but I don't know what I'm looking for to do any math on. Or where to find a standard 14-bit benchmark to compare against. I figured that you saying "impressive" was a good thing though. :-)

Cheers,

Mark.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 12, 2016, 09:01:49 AM
I'm doing some research , to understand the possibility of a 10 or 12 bit raw video found this on the  http://magiclantern.wikia.com/wiki/Register_Map
 under "Image Preprocessing"
Code: [Select]
0xC0F08120 - PACK16 Control register
0xC0F08124 - PACK16_MODE (mask 0x131)
             From ProcessPathForFurikake (raw_twk):
             -------x --xx----
                  111 (0x130) = 16-bit output
                  011 (0x030) = 14-bit output
                  010 (0x020) = 14-bit output
                  001 (0x010) = 12-bit output
                  000 (0x000) = 10-bit output
So the raw_twk is just for  DIGIC 5 processor  (5D3)  or is these addresses also validate for other camera.
I wonder thou if there enough cpu power to do this or is it a matter of available ram left from ML .
Thinking out loud .  :)
Yes I think this is worth getting involved  deeper , need to do more reading 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 12, 2016, 09:16:57 AM
@markodarko:

Since d's implementation is CPU-based, the numbers are impressive when compared to memcpy, or to my C version (which is used for the previews such as in mlv_play).

Now, imagine you want to record 1920x1080 at 24 fps. What numbers would you need here?

@reddeercity: that's the register I've tweaked to reduce the bit depth. This method doesn't use the main CPU to do the processing; it calls the dedicated hardware, very similar to the method we already use for copying raw video frames for recording (edmac).

For other cameras: I think the 60D will work just by finding the right stubs; the 5D2 has the functionality, but the exact implementation looks a bit different.

No need to worry about RAM: 12-bit frames are smaller than 14-bit ones, and the conversion doesn't need any extra memory (because it's done on dedicated processing hardware - very similar to a GPU).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: goldenchild9to5 on May 13, 2016, 07:23:20 PM
Man Magic Lantern keeps coming with awesome updates 10 - 12 Bits would be welcome..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 14, 2016, 04:06:20 AM
60D:
Code: [Select]
NSTUB(0xFF3C1C60, ProcessPathForFurikake)
NSTUB(0xFF3FC6FC, PACK16_Setup)

I'm assuming I need to merge raw-bit-depth and RscMgr_memory so that the patch hook will work (otherwise patch manager complains cache is locked). Unfortunately RscMgr_memory gives err 80 in LV.


on a side note: hmm... (https://en.wikipedia.org/wiki/Furikake)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 14, 2016, 07:54:06 AM
That's right - I just pushed a few changes to the patch manager, to allow patches like this to run on 60D. I had to use a long jump, because module code is loaded at the end of RscMgr, which is far from both ML core and ROM (so regular jumps don't work).

So, for 60D at least, you need to merge patchmgr as well.

I've also got a log in QEMU showing what registers are configured (without the ResLock part):

furikake-60d.log (http://a1ex.magiclantern.fm/bleeding-edge/prepro/furikake-60d.log)

5D3 is almost identical - except it uses different EDMAC channels.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 14, 2016, 06:28:49 PM
Is this only for testing on the big cams like 5d3 and 60d, I went to get the stubs, tried it, I get smaller file sizes for the 12 and 10 bit, but there scrambled and cam goes to black and freezes after done taking picture when I do 12 and 10 bit, 14 is fine?

Code: [Select]
//** Image processing paths */
NSTUB(0xFF420AE4, ProcessPathForFurikake)
NSTUB(0xFF5EF23C, PACK16_Setup)             // PACK16_MODE:%#lx and others
not 100% there right, especially the PACK16.

(http://s32.postimg.org/6lept4sj9/Capture1.jpg)

(http://s32.postimg.org/3ox79c9yt/Capture2.jpg)

Or is the problem this part that is only for 5D3?
Code: [Select]
+    /* hook to override the bit depth in PACK16 setup function */
    patch_hook_function(
        (uintptr_t)&PACK16_Setup,
        0xE92D4010,
        PACK16_Setup_hook,
        "PACK16: raw bit depth override"
    );
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 14, 2016, 06:46:16 PM
PACK16_Setup is correct. I'm not sure what's wrong - if the output is scrambled, probably the patch override didn't work. Are there any errors in Debug menu, under Memory Patches?

I'm also working on a lower-level implementation that doesn't need any extra stubs.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 14, 2016, 07:09:29 PM
Nope, it just says None.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 14, 2016, 08:36:24 PM
Okay, it's working on 60D (if I turn off GD, then I don't get the err80), avg. time is about 117 ms (268 MB/s !).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 14, 2016, 09:08:13 PM
EOSM also working, ~114 ms
Code: [Select]
NSTUB(0xff425150, ProcessPathForFurikake)
NSTUB(0xff5fa9e8, PACK16_Setup)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 14, 2016, 09:49:29 PM
Got same result on 550D, no freeze, but scrambled image for 10 and 12, unless its not even suppose to work on this cam anyway, but tried anyway.
Code: [Select]
NSTUB(0xFF353274, ProcessPathForFurikake)
NSTUB(0xFF37FBE8, PACK16_Setup)

Update!!, Works on 550D, had to do a Make Clean first, Ill try on 700D now.

Works on 700D as well! Just needed a make clean.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 14, 2016, 10:39:14 PM
On 700D MLV Format
14bit 820 ms 38 MiB/s
12bit 825 ms 32 MiB/s
10bit 739 ms 30 MiB/s

speed seems to jump around alot on all bits, gets faster after 2nd mlv is taken, usually by 100ms on each.

DNG Format
14bit 2970 ms 10 MiB/s
12bit 2169 ms 12 MiB/s
10bit 1960 ms 11 MiB/s



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 15, 2016, 01:25:07 AM
Just to clarify, I'm talking about just the speed of bit depth conversion, not the entire time it takes to save the file.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on May 15, 2016, 11:45:10 AM
Just to check if I understand things right.

There is a dedicated chip(the digic chip?) inside the canons which can do bit depth conversion.
The test results show it can convert about 30/35 MegaByte per second.
Full HD at 24/25 fps is about 90MegaByte/s.
So the chip isn't capable of doing it fast enough for full HD video

Could be use full though for full res silent picture time lapses. You're saving some memory card space.
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on May 15, 2016, 01:26:39 PM
Read what a1ex wrote on previous page:

Quote
Timing: 120ms for a full-sized image (5936x3950).

As well as
Okay, it's working on 60D (if I turn off GD, then I don't get the err80), avg. time is about 117 ms (268 MB/s !).

Full HD at 24/25 fps is about 90MegaByte/s.
So the chip isn't capable of doing it fast enough for full HD video

Wrong, the "chip" does that already for you by providing FHD H264 files.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on May 15, 2016, 02:12:04 PM
Ah, so this is a big discovery :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Greg on May 15, 2016, 11:41:48 PM
500D :
Code: [Select]
NSTUB(0xFF327A58, PACK16_Setup)
5D2 :
Code: [Select]
NSTUB(0xFFB28D4C, PACK16_Setup)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chris_overseas on May 16, 2016, 11:02:43 AM
Stubs for 5D3 1.2.3 (untested):

Code: [Select]
NSTUB(0xFF3CCC14, ProcessPathForFurikake)
NSTUB(0xFF57CA1C, PACK16_Setup)

Edit: I just Googled Furikake. Interesting, though not sure what it has to do with a camera! https://en.wikipedia.org/wiki/Furikake :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on May 16, 2016, 12:48:26 PM
I'm really focus about this tread.

I think, It should be so usefull to be abel to have 10 bits and 12 bits.

- For exemple : 1920x1080 60p (12 or 10bits)
Or just by recording in good quality for 30 minutes or more.
But also 3K 10bit continuously ?

So, I ask for peoples in my cathegory (mean who are noobie).

Will it be possible for everybody having a 5D mIII or 60D (...), to easily use 10 and 12 bits ? A new module will come ?

Inifnity respect to every ML scientists !
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Greg on May 16, 2016, 10:41:33 PM
It works on old cameras. We need use the ProcessPath (without Furikake).

500D :
Code: [Select]
NSTUB(0xFF24E7F8, ProcessPath) //[DEFC]
NSTUB(0xFF327A58, PACK16_Setup)
//ProcessPath time : 96ms

We have only 3 values:
Code: [Select]
a1 = buf_in,
a2 = buf_out,
a3 = value 0 or 1 ?

So silent.c requires a small change
Code: [Select]
struct s_DefcData
{
    void *pSrcAddress;
    void *pDestAddr;
    uint32_t dstModeRaw;
} defc;

We can use this method on all cameras.


It might be interesting :
Code: [Select]
NSTUB(0xFF24FCCC, ProcessPath) //[SUB] DarkSubtraction
NSTUB(0xFF2506A0, ProcessPath) //[DEFD] related to TwoAdd
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Nutsy on May 17, 2016, 05:28:02 PM
What are the chances of this being developed for the 7d as well?
its very exciting stuff. but i fear maybe the 7D just wont be fast enough?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on May 17, 2016, 10:53:34 PM
What would this do for the 550D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Greg on May 19, 2016, 01:55:08 PM
Someone tried this with the 700D?
500D 10bit and 12bit (frames after DIGIC processing) does not have a hot and cold pixels.
There is a possibility that the DIGIC also corrects focus pixels.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 19, 2016, 03:54:50 PM
Yeah, I tried it on my 700D and posted results last week, but hmmmm, sounds like something I would like to test on the pixels, I didnt even think of checking the dots and pixels, just looked to see if it worked, lol.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on May 23, 2016, 11:27:57 PM
Sorry for being impatient, is there any progress? Anything even in early stage what could be tested on 5d3?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on May 24, 2016, 12:43:26 AM
You can compile the raw-bit-depth (https://bitbucket.org/hudson/magic-lantern/branch/raw-bit-depth) branch and try it with 10/12 full res silent pics.

From my quick attempt, normal silent pics don't work (and by extension, raw video) it seems b/c LV is running and these hardware modules are "busy" being utilized by Canon LV routines. Furikake just blocks until you exit LV. So to get this working for raw video it looks like we are going to need to understand more about how to manually setup these modules by overriding the low level registers that control them. That's a lot of reverse engineering work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 24, 2016, 07:25:17 AM
I have a semi-functional low-level implementation (it's basically the log I've posted earlier, converted to C code), but it only converts a picture at every other attempt. If anyone is willing to debug it, I can commit it in this state.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on May 24, 2016, 04:29:47 PM
I have to make a comment to follow this, I already though I was but I wasn't :P

From what I've followed so far; despite going from 16bit(14) to 10 or 12 it's still not faster to write to the card?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 24, 2016, 05:57:18 PM
if the kind of debugging u need is possible by me, i would be glad to help out any way i can, i owe you a few favors anyway alex.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on May 24, 2016, 06:19:31 PM
Count me in on the debugging whenever necessary @a1ex!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 24, 2016, 09:03:06 PM
Alright, pushed the code.

Finding the mistake might be as simple as solving one of those kid puzzles about finding the difference between two drawings (probably a typo in my code). Or - given that I couldn't figure out what was wrong in a couple of hours - it might be a little more difficult, as dmilligan already explained.

First, compile 15852ed and run "Fullres LV test" (from LiveView) to see how it should work. It will show a "full-res liveview", garbled in 12-bit and 10-bit modes (which is fine, because the drawing code expects 14-bit data).

Then, compile 21f337a and run the same test. It will do the first frame right, then it will freeze. Take the battery out.

If you add a timeout to the take_semaphore call, you will see it returning error every other frame (first correct, second bad, third correct, fourth bad and so on). Tested on 5D3 and 60D.

Note that regular full-res pictures work fine, because going to LiveView and back appears to perform the necessary cleanup (the one I've probably overlooked).

Also note that calling silent_pic_take_fullres in a loop, without going to LiveView, works fine, which puzzles me. This might help solving the mystery - turn "while(1) silent_pic_take_fullres(0)" into "silent_pic_fullres_lv_test()", step by step, and see where it fails.

To understand the code, see the log from reply #279, the Register Map wiki page, and the other links I've posted earlier for background info.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 24, 2016, 11:40:50 PM
On 700D, my first image is correct, second and on are garbled.

Also during 10bit, it freezes on 3rd image when I do a timeout, but continues on 12bit, but all garbled.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 25, 2016, 04:30:51 AM
Yeah, garbled image is what you are looking for, but without freezing.

First image is not converted to lower bit depth, that's why it looks good.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 25, 2016, 05:44:56 AM
Yeah, the only way I can get it to not freeze, is like you said add a timeout like this take_semaphore(edmac_write_done_sem, 100); which is what I think you meant to do, but only is stable at 12bit, 10 freezes after the 1st garbled frame passes. At 12bit, it goes fine with the correct garbled frames.

Not sure what to do with this  turn "while(1) silent_pic_take_fullres(0)" into "silent_pic_fullres_lv_test()", step by step, and see where it fails.

Don't see anything interesting I can find, but if you can think of anything for me to look at more closely and how, I will be glad to.

I was expecting to see text printed on screen, but I dont see any text at all, not sure if that is expected? I know that has nothing to do with the conversion, but I see  printf("Furikake time: %d us\n", (int)(t1 - t0));  So I assumed something would of printed?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on May 25, 2016, 06:04:14 AM
The two codes are doing pretty similar things: capture, display, convert bit depth. One is complex and works (it also saves the file, triggers overlays), the other is simple, but fails for some reason. So, to narrow down, it makes sense to simplify the complex code, step by step, to reach the simpler version, and write down at which step it fails, during the conversion process. It's very simple to do, but time-consuming.

To see the printf output, enable the debug console.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mk11174 on May 25, 2016, 06:29:26 AM
Ok, printf stuff is visible now, thanks.

Wish I understood the code better, so I could know how to do the code step by step to narrow it down for you, I would take the time to do it, but honestly dont know which part is the complex one and which is the simple one, lol, most of the fuction is new, so wouldn't know where to begin and how to proceed,  sorry.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lucky Luke on June 07, 2016, 11:15:09 AM
successing 10bit recording, it would be worth to me buying this for 100€ or even more...and I guess to manymany others on the globe.
Any movements in this area?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on June 07, 2016, 11:35:26 AM
And another attempt to kill this project ...

Devs will drop out if ML goes commercial.
Donate or don't but don't expect anything to be done by throwing peanuts over the fence.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Nutsy on June 07, 2016, 11:50:55 AM
If I could afford to donate money I would... ML is a great project.

I'm on benefits (scum i know) and at the moment struggling to get by.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lucky Luke on June 09, 2016, 08:10:14 PM
"And another attempt to kill this project ..."

Really stupid insinuation.
Just my opinion on value and possibilities of 10Bit recording.
What is wrong to became commercial if it is worth?
We are taking so much for granted...
Yes, I would pay it even more than 200€ for having no drop outs because of the card reading.
This would be a real earthquake even now late in a 4K world.
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on June 09, 2016, 08:52:24 PM
Can we please stay on topic and keep all discussion at a technical level. Posts like these just make it harder for those who might actually be able to contribute something to follow the conversation and find important information they need.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on July 03, 2016, 01:05:12 AM
Can we please stay on topic and keep all discussion at a technical level. Posts like these just make it harder for those who might actually be able to contribute something to follow the conversation and find important information they need.

I agree.


I'm noob in code but doesn't matter. I do my best to understand and I would like to help you and try this new improvement.

Could somebody do a dropbox with the module ready to be try for 5D mark III 1.1.3 ?
Also, is it possible to chose in camera 10 and 12bit ? And can we chose resolution like when we use raw module ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Pierro777 on August 21, 2016, 06:43:45 AM
I'm new to the forum and just lurking a lot these past few days. I don't have much to contribute in terms of help (can't code etc.). But I just wanted to say you guys are all awesome. Keep up the amazing work! You help people create a lot of cool videos.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 02, 2016, 07:42:17 PM
On 60D, this appears to give 12-bit and 10-bit raw data for still pictures:

Code: [Select]
    MEM(0x2B308) = 0x10;   /* 12-bit */
    MEM(0x2B308) = 0;      /* 10-bit */

How did I came up with this magic value?

Look at the emulation log of the image capture process from QEMU (this post (http://www.magiclantern.fm/forum/index.php?topic=2864.msg174076#msg174076)).

Code: [Select]
[PREPRO] at ShootCa:FF1C5AB8 [0xC0F08094] <- 0x30      : PACK32_MODE

This code is called from FF1C5AB8; if you look it up in the disassembled code, it's inside engio_write. There are lots of calls to this function in the firmware; to see which one is used in our case, put this into 60D/debugmsg.gdb:

Code: [Select]
b *0xFF1C5A68
commands
  silent
  print_current_location
  printf "engio_write(0x%x)\n", $r0
  c
end

Compile autoexec.bin from minimal/qemu-frsp (qemu branch), copy it onto your SD image, and run the emulation:
Code: [Select]
./run_canon_fw.sh 60D,firmware="boot=1" -s -S -d io,int -singlestep & arm-none-eabi-gdb -x 60D/debugmsg.gdb

Result:
Code: [Select]
[ShootCapture:ff084a64 ] engio_write(0x2b2fc)
[PREPRO] at ShootCa:FF1C5AB8 [0xC0F08090] <- 0x80000000: PACK32_ENB
[PREPRO] at ShootCa:FF1C5AAC [0xC0F08094] <- 0x10      : PACK32_MODE
...

Now look at 0x2b2fc in a RAM dump using a hex editor, look at these lines (https://bitbucket.org/hudson/magic-lantern/commits/21f337ae919ed82e9aa1ebaf994e15ce1cb86f38?at=unified#Lmodules/silent/silent.cT1230) in the source code, and the mystery will clear up quickly.

I did not try to decode the 12-bit CR2 data, but the preview looks scrambled (as I would expect from 12-bit data interpreted as 14-bit).

It might be possible to change the bit depth of the LiveView stream with something similar.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 02, 2016, 08:55:33 PM
On 60D, in LiveView, this is enough to change the bit depth of the raw stream (valid values here (https://bitbucket.org/hudson/magic-lantern/commits/21f337ae919ed82e9aa1ebaf994e15ce1cb86f38?at=unified#Lmodules/silent/silent.cT1230)):

Code: [Select]
EngDrvOut(0xC0F08094, bit_depth_mode);

Now, all you have to do is to remove all hardcoded references to 14 bpp and bam! you have 12-bit and 10-bit raw recording. Example code to save a 10-bit DNG: dont-click-me.c (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/dont-click-me.c)

Looking forward to your pull request :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on November 02, 2016, 09:03:27 PM
excellent. thumbsup!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 02, 2016, 09:57:45 PM
 :D :D :D :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on November 02, 2016, 10:03:16 PM
Never regretted to have no popcorn in the house. Until now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on November 02, 2016, 10:10:47 PM
congrats for finding this @a1ex. Hopefully this won't raise Mark Free's street price in near future :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DavidSh on November 02, 2016, 10:49:27 PM
wow!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Pierro777 on November 02, 2016, 11:02:51 PM
Does 10bit raw mean it might be possible to have 60fps raw on the 80d someday ?!!!

Keep it up ! I wish I could help with anything lol.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hindra on November 02, 2016, 11:22:51 PM
That's awesome! Thank you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on November 02, 2016, 11:24:44 PM
Nice find Alex ! An additional small revolution in the magical world of magic lantern.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 03, 2016, 01:00:18 AM
Damn @a1ex, I smell the Christmas trees already...  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 03, 2016, 02:47:08 AM
Works on 60D. Can record 10bit full resolution for about 12 seconds. 720p is about 1 minute.

https://bitbucket.org/hudson/magic-lantern/commits/9c951e497666bb4f9fc8f933fa29bec12ce16030

Not using ML greyscale preview results in about 50% of frames being corrupted

Update:
Confirmed working on EOSM, however grayscale preview is broken (all you see is the last LV frame before you hit record), only tried with crop hack
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 03, 2016, 08:31:02 AM
The raw backend may be overwriting the raw_info structure and set it (or parts of it) back to 14 bits.

The easiest way is probably to block all overlays and other raw tools for other bit depths, and override the bit depth only while recording.

Supporting all bit depths in the raw backend requires a bit more work. Additionally, some features use direct pixel access (rather than using raw_get_pixel) for speed reasons; this access is probably fastest in 12-bit mode, as the packing should be a little easier (just a guess, didn't do any benchmarks).



Nice to see it working on the M; I've expecting it to work only on DIGIC 4. Here's why:

If you remember the "raw type" register (see PR #732 (https://bitbucket.org/hudson/magic-lantern/pull-requests/732/5d3-attempt-to-fix-vertical-stripes-so-it)) and look it up on the Register_Map (http://magiclantern.wikia.com/wiki/Register_Map) page, the register used on DIGIC 4 models (0xC0F08114) is labeled PACK32_ISEL (probably "input selection"). The bit depth register is 0xC0F08094 PACK32_MODE, so they must be related somehow. On 5D3, this register is 0xC0F37014, so I've expected the ISEL register to be moved as well. Didn't try yet.

Side note: PACK16 and PACK32 are probably image processing modules that write data from the image processor to the main RAM (the RAM visible from the ARM processor) and can convert it to various bit depths (hence the name "pack"). In a similar way, DSUNPACK, ADUNPACK and UNPACK24 are used to read image data from the ARM processor (which can be packed at various bit depths) and bring it somehow to other image processing modules. In all cases, the MODE register for each of these modules is used to configure the bit depth.

It isn't clear how to connect those processing modules together, but it's probably done with the ISEL switches (whose exact meaning is largely unknown, other than the above assumption about their naming).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 03, 2016, 08:42:09 AM
Confirmed working on EOSM, however grayscale preview is broken (all you see is the last LV frame before you hit record), only tried with crop hack

If I remember correctly the grayscale preview was always broken on the M - or did that work at one point?
Regardless, this sounds like I really need to figure out a way to increase the frame size for raw recording on the M now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: myown on November 03, 2016, 09:26:07 AM
WOW!! You do such a great work!! Congratulations and many, many many thanks!

I have a 60D and wanne test this  :)
Do i have to consider something special?


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 03, 2016, 09:57:08 AM
Not using ML greyscale preview results in about 50% of frames being corrupted

Figured it out - when getting the raw stream into main memory, we have to adjust frame size. On models that use CONFIG_EDMAC_RAW_SLURP (see http://builds.magiclantern.fm/features.html ), this parameter is under our control. On other models, this configuration is done in Canon code, which probably must be patched somehow.

Also, confirmed it working on 5D3.

To get clean image, you will have to change raw_info.bits_per_pixel from the raw recording module (this is not yet committed).

For more details, see the commit messages from this branch:
https://bitbucket.org/hudson/magic-lantern/commits/branch/raw_video_10bit_12bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on November 03, 2016, 11:01:34 AM
So what's more difficult? Switching all cams to use raw slurp in combination with running RAW_DEBUG_TYPE for all of 'em or better patching Canon code somehow. Already successfully used hardcoded values for slurp a while ago on 70D but the preferred raw type didn't match and I didn't feel like having time to try to find the best value for it at that time. Would other cameras still need hardcoded values for all modes when using raw slurp or do you have an idea to find / read them "magically" somehow...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 03, 2016, 05:09:48 PM
Enabling the new method on all models is desirable IMO, because we will have a single way of doing things. However, each model is likely to have its own quirks, so it's a bit of work.

Patching the ROM is probably easier, but pretty ugly if you ask me. You had problems with it on 70D (PR #734 (https://bitbucket.org/hudson/magic-lantern/pull-requests/734/cleaner-method-for-hiding-canons-bottom/diff#comment-18511247)), but I don't know where to begin debugging it. Maybe just merging the latest patchmgr could work, who knows.

BTW, for 70D, trying different raw types might reveal other streams, such as the dual pixel sub-images (just a guess, no idea how it's done on 5D4, but I wouldn't be surprised if Canon implemented it that way).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 04, 2016, 02:56:01 AM
Confirmed working on 60D now with Canon LV
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: goldenchild9to5 on November 04, 2016, 05:12:18 AM
@a1lex Great job.. that's just awesome smaller file sizes and longer recording times amazing.  This feature will be soo.. useful for the new gen canon's using 1080p 60fps 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on November 04, 2016, 08:45:50 AM
What's the state of the 550D in getting 10bit and 12bit.
(We all know the 550D is the most important camera of them all :P)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 04, 2016, 01:50:18 PM
What's the state of the 550D in getting 10bit and 12bit.
Maybe already working, somebody needs to try it: https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: araucaria on November 04, 2016, 02:19:42 PM
Quote
Audionut:
Is the code just truncating bits, or is it compressing them into the remaining?

edit:  The OP states that the bits are being compressed.

Curious to know what kind of compression it is. Great work, I'm very excited.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on November 04, 2016, 03:41:05 PM
Let me know if you need a 5DM2 (+VAF Filter) tester ...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on November 04, 2016, 04:43:50 PM
👍
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 04, 2016, 07:28:25 PM
Tested it on a 5D2, almost works, 10 bit works on every other frame, the rest are pink, and 12 bit also works on every other frame with the rest being completely pink. I have uploaded some DNG files made with MLP. I tried in full sensor resolution and 16:9, full sensor made 12 bit completely pink, but I only tested that once. Here are the 10 and 12 bit DNG files including 14 bit for comparison: https://drive.google.com/file/d/0BwvDlbhZgsGGY1BUM2tyN0YwdVk/view?usp=sharing (https://drive.google.com/file/d/0BwvDlbhZgsGGY1BUM2tyN0YwdVk/view?usp=sharing) The boring shot of a map is not enough to judge quality, but I think 10 and 12 bit look the good as 14 bit in this case.

"Disclaimer": I don't know if I missed anything, or did anything wrong, is there anything else to enable other than raw_rec module?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on November 04, 2016, 07:51:51 PM
What do you mean by "works on every other frame"?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 04, 2016, 07:56:38 PM
Tested it on a 5D2, almost works, 10 bit works on every other frame, the rest are pink, and 12 bit also works on every other frame with the rest being completely pink.
This will be the case on all cameras except 5d3, 60D, and 600D, unless you set preview mode to ML grayscale. And you may notice the preview just looks like garbage once you start recording (ML grayscale preview can't handle anything but 14bit data yet)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 04, 2016, 07:58:54 PM
I mean this: first frame is fine, no corruption or pink artifacts, the next is all pink and corrupted, no image, then another fine one, another corrupted... and this cycle continues, so only half of the frames are not ruined, and when played back(which MlRawViewer refused to do), it just flashes pink every other frame.

Edit: @dmilligan just saw your post, thanks for clarifying, I will test again.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 04, 2016, 10:08:52 PM
Goods news here , I'm glad to see this thread back active.
 I would like to test this on my 5D2 please ,  @Ilia3101 could I try your module ?
I don't have a environment to comply ml right now , just pm me  or post it here
 :) 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 04, 2016, 10:36:08 PM
@reddeercity Uploaded the module: https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing (https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing), maybe you'll have some more success than I did.
After some more testing with ML grayscale, I have concluded that it does't yet fully work with the 5D mark ii, whatever I try, there's always something going wrong every other frame in 10 bit, either a pink frame or the bottom half is of another frame, and 12 bit seems to have weird colourful static all the time :'(.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 04, 2016, 10:43:22 PM
@lia3101 , thanks I'll do some testing and post the results  ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 05, 2016, 12:12:52 AM
Wow, lots of activity here. I've been asked to help compile some test builds so for a limited time only I'm posting everything that I could build on my bitbucket download area:

https://bitbucket.org/daniel_fort/magic-lantern/downloads

All platforms are there in one large zip file along with the command line tools, mlv_dump, raw2dng and cr2hdr, for both Mac and Windows.

Note that the 1100D.105 didn't compile and raw2dng needed this change in order to cross compile for Windows on a Mac:

Code: [Select]
raw2dng.exe: FORCE
  $(call build,MINGW,$(MINGW_GCC) -c $(SRC_DIR)/chdk-dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR))
- $(call build,MINGW,$(MINGW_GCC) -c raw2dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64)
+ $(call build,MINGW,$(MINGW_GCC) -c raw2dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64 -std=c99)
  $(call build,MINGW,$(MINGW_GCC) raw2dng.o chdk-dng.o -o raw2dng.exe -lm -m32)
 

I haven't tried any of this myself so -- Happy testing!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on November 05, 2016, 12:30:01 AM
Tried the 5D mark II with 12bits
Extract MLV with the new mlv_dump version and the line
Code: [Select]
--dng --no-fixcp --no-stripes
I got pink frame issue too

Now i will test the 5DIII :)

Even if we got issues, i'm sure it will fixe of everyone do tests.

Thanks @dfort for all these news builds !

EDIT:

Ttried with the 5DIII :

10 bits = Stable  | FPS override enabled
12bits = Got pink frame only on the first frame | FPS override enabled
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 05, 2016, 12:54:05 AM
I tried on  my 5d3 113, just a speed test with sandisk extreme pro CF card 160MB/sek, i assume that despite image issues frames are recorded with their proper size (but i didnt check).

So i tried 3K 2,35:1 23,975fps- 14 bits - 134 frames
                                               -12 bit - 294 frames
                                               -10 bit - over 3 minutes recording, didnt check longer
1920x648 (16:9 after stretch) 60 fps     -14 bit 308 frames
                                                            - 10bit 1425 frames
1920x648 50fps seems to be continous in 10 bit.

1920x1080 37fps - 14bit - 239 frames
                             - 10bit - over minute, didnt check longer.

highest possible resolution 3584x1320 23,976 10bit - 160 frames (vs 61 with 14bits).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 05, 2016, 12:55:08 AM
Thanks a1ex, dmilligan,  Ilia3101 and dfort. Ran two test files with a 5D mark III and it is very promising. First fram garbled both for 10 and 12 bit but the rest looks fine. Disabled vertical stripes and cold pixels in mlv_dump. Needless to pint out it,s a blessing to see so much headroom left with the lower write speed even with 30 fps 1920x1080.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 05, 2016, 01:15:08 AM
Started testing with 10 & 12 bit  , as mention before on the 5D2 the Liveview is all scrambled even if you press the half shutter
for color preview in ML grey preview it's just colored scrambled lines. But I Found a trick !  :D  "Turn off Overlays when recording" in the overlay tab.
but of course that also kills the buffer overlay graph so you don't know any info of recording time etc... at least the Liveview is not all scrambled
and you can see what is being recorded .
I still need to test with HDMI & 3x Crop Mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: v8rrc on November 05, 2016, 01:28:21 AM
Tested on 6D 10bit seems to work, preview and playback garbled but dngs open in photoshop/lightroom fine. and Continuous OK at 1824 x 776 (2.35:1) 23.97FPS which is Great.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 05, 2016, 01:31:03 AM
Like reddeercity said, checked 5d3 113 with global draw OFF  - liveview is good then with 10 and 12 bits.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on November 05, 2016, 02:16:40 AM
As far as I can tell 10 bit was recording on the 550D, unsure how to play it back.
I was under the impression that it would result in a significantly lower bitrate, to the extent where I would be able to record full resolution @25fps, which it would only record half a second. Was that somebodies incorrect calculations in the past? (or am I thinking of 8bit).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 05, 2016, 03:28:54 AM
10/14 = 0.71 => you should see about 29% lower bitrate.


As a1ex has alluded to, the entire ML raw backend is basically hardcoded to only handle 14 bit raw data, so preview, playback and all that are not going to be working correctly. Fixing them is a lot of work (lots of tedious bit math), but should be pretty straight forward and requires no reverse engineering. I had a quick go at the preview code, but I'm not sure I understand all the transformations correctly. The result was starting to resemble the correct image but still very garbled.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 05, 2016, 04:07:48 AM
Some more test for Liveview , To get the overlays in liveview without being scrambled in 1:1 mode switch preview in the video menu in raw tab to
Auto or Canon preview then all overlays are enabled but not to sure if the meters for exposure etc... are correct this was for 12bit raw on 5D2.
For 3x Crop Canon preview , ML grey preview lockup preview but came back after recording is stop.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 05, 2016, 04:51:41 AM
Quote from: ItsMeLenny
As far as I can tell 10 bit was recording on the 550D...

Already have ML, how do I get 10 bit on my 550D?

Edit:
I found the files, do I just add to the folder on my SD card or do I actually have to reinstall ML?


Edit 2: Nevermind, got it working. Testing now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 05, 2016, 05:35:46 AM
Bunch of colored snow, but I used MLV Mystic. Recorded long though at 1728 using 10 bit, just can't see frames. This is on a 550D. Crop mode crashes camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 05, 2016, 07:00:57 AM
Completed some resolutions tests with 5D2 at 10 & 12bit with 3x Crop , the only resolutions atm that does not give corrupt on  ever other frame is
1856x1044 @ 10bit , ML Grey preview with overlays killed . Any other adjustment then this will give different type of frame corruption
from just total noise to part or the image is out of order and or with color noise bars across the image. 12 bit has corruption & or part of the image out of order
on all resolutions except in 3x Crop mode there 12 & 10 bit are free of corruption but there have a heavy pink cast just like the black level off .
Preview in Crop mode I used was ML Grey with overlays enable (default mode) . That's it for now .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DavidSh on November 05, 2016, 07:41:09 AM
Is there a build for 5d3 123 ?
Looking forward to do some tests...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 05, 2016, 03:40:05 PM
Bunch of colored snow, but I used MLV Mystic.

You'll have to use the mlv_dump compiled from the raw_video_10bit_12bit branch with all processing options turned off (see previous replies) or MLVFS. These are the only converters I'm aware of that can handle non 14 bit data correctly (there might be others).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 05, 2016, 04:15:09 PM
Is there a build for 5d3 123 ?
Looking forward to do some tests...

Short answer, no. Longer answer:

5D3.123, 70D.112 and 100D.101 have not been merged into the main trunk yet. Someone ( @nikfreak ?) could attempt to merge in the raw_video_10bit_12bit into these platforms.

1100D.105 build is currently failing but it has nothing to do with the 12-bit, 10-bit changes. The automatic builds on the download page are also failing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 05, 2016, 05:17:17 PM
Okay, so I used MLVFS and now I can see the frames. Every other frame is top half pinkish noise. I was trying 12FPS on 550D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 05, 2016, 08:03:37 PM
700D is working in 10-bit and 12-bit. Tested mv1080, mv1080crop, mv720 and zoom modes.

LiveView freezes when recording. In 5x zoom mode LiveView is in greyscale and becomes garbled when recording but turning off GlobalDraw fixes that.

About half of the time I'm getting a corrupted frame but only on the first frame:

(https://c7.staticflickr.com/6/5634/30496556750_d919626b81_n.jpg)

After that everything is beautiful! (Processed through MLVFS.)

(https://c5.staticflickr.com/6/5634/30760899156_b7e83081ae_z.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: krisdeak on November 05, 2016, 08:23:13 PM
700D is working in 10-bit and 12-bit. Tested mv1080, mv1080crop, mv720 and zoom modes.

This is really really promising! Could you confirm the data rates achieved in 10-bit +  a rough estimate of how long we might be able to record like this, with the SD card hard limit taken into account?

I'll keep following this thread!

Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 05, 2016, 09:08:46 PM
Could you confirm the data rates achieved in 10-bit

According to the ML display:

14-bit 36.8 MB/s @23.973p
12-bit 31.6 MB/s @23.973p
10-bit 26.3 MB/s @23.973p

I'm able to record continuously up to 1344x756 in 10-bit on the 700D. After that it stops automagically after a few seconds depending on the resolution. Basically if the bottom line on the ML menu is green it won't stop. For the wide screen enthusiasts that means 1536x654 on the 700D is doable.

I also wanted to report my findings on the EOSM. It works fine in the crop modes, mv1080crop and zoom with the same LiveView freeze and first frame corrupt issue as the 700D. There's something strange going on with the non-cropped modes. The camera records but the camera freezes when recording is stopped requiring a battery pull. The file is saved and appears to be mv720 (3x5) no matter what the Canon menu is set at which is normal behavior for the EOSM though there are some other idiosyncrasies with this build like it always defaults to 29.97 in non-cropped modes but in crop modes it will default to 23.98. Of course further testing is in order on this platform.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 05, 2016, 11:58:28 PM
Wonderful progress guys. So much activity as I've been trying to stay on top with you guys. Downloaded the builds for EOSM and 7D. Thanks @dfort!

Re: EOSM -- I can confirm that this works nice with MLP in spitting out DNG's/ProRes with the exception of first DNG being corrupted somehow. Tested on 24p, 30p & 60p.

Re: 7D -- This was shot in 48p @ 1728x692p it seems that I am getting that pink half corrupted frame on every other frame (for both 10-bit and 12-bit) as others have reported in here as well. Here's what they look like.

(https://c8.staticflickr.com/6/5468/30711689591_5a4ec3bce1.jpg) (https://flic.kr/p/NMTrkv)
(https://c4.staticflickr.com/6/5339/30711689811_bc128b4dc1.jpg) (https://flic.kr/p/NMTrpi)


Before I can go ahead and test out a 5D3 -- I have to downgrade from 123 to 113 first so will report back by findings once I have them down.

Also any way I can get MLVFS to work with these 10-bit/12-bit files? Do I need to replace the binaries that @dfort shared with us and replace the ones inside MLVFS -- could you enlighten me on this @dmilligan please?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 06, 2016, 02:15:25 AM
Also any way I can get MLVFS to work with these 10-bit/12-bit files? Do I need to replace the binaries that @dfort shared with us and replace the ones inside MLVFS -- could you enlighten me on this @dmilligan please?
MLVFS should already handle the files just fine. MLVFS does not contain or use binaries of ML tools.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 06, 2016, 02:19:03 AM
Yeah, MLVFS works. I've tried different frame rates and resolutions, still getting pink/corrupted every other frame in 10 and 12 bit. (550D)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 06, 2016, 06:42:23 AM
@DeafEyeJedi -- The binaries in the package I made have a commit (https://bitbucket.org/hudson/magic-lantern/commits/9b2547bb092e4c3c08265fe0d5948659d3e027db?at=raw_video_10bit_12bit) in the mlv_code to work with lower bit depth files. You should try using the binaries in the package from the command line and compare them to the ones from the unified branch. There's also a note:

Quote
mlv_dump: fix DNG output for lower bit depths
You need to disable all processing, e.g. mlv_dump M00-1234.MLV --dng --no-fixcp --no-stripes

MLVFS can already handle these bit depths and like dmilligan explained doesn't use any of these binaries.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 06, 2016, 07:11:24 AM
MLVFS should already handle the files just fine. MLVFS does not contain or use binaries of ML tools.

Apparently I needed to update the FUSE to make this work again. I could swear it was working right before the recent 10.12.1 update from Apple.

So indeed MLVFS works just fine with 10/12-bit files atm. Thanks!

(https://c7.staticflickr.com/6/5752/30506383910_dc52ee1fd3.jpg) (https://flic.kr/p/NtKc7o)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 06, 2016, 12:00:15 PM
On the Mighty 50D I get : (reported write speed in parenthesis)

in crop mode 1920*1080
~1100 frames in 14 bits (~77 Mbps)
Continuous in 12 bits (75.7 Mbps)
Continuous in 10 bits (74.7 Mbps)

in Full sensor (1568*1046)
Continuous in 14 bits (72.7 Mbps)
Continuous in 12 bits (74.6 Mbps ???)
Continuous in 10 bits (75.8 Mbps ??????)

So obviously, the reported bitrate is wrong, I'll have to check the recorded files
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ilguercio on November 06, 2016, 12:30:41 PM
Any way to try the build for 6D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 06, 2016, 03:47:05 PM
Indeed, the numbers displayed are estimations of your write speed, not the bitrate of the recorded movie. The latter is easy to compute: width x height x bpp/8 * fps [MB/s].

The stdev of those estimations is a little high for my taste, but not a major issue.

However, a second look over the code uncovered a bug (present in the old raw_rec and mlv_lite): the speed estimation is only used for next recording if global draw is off. Easy to fix.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: anDyIII on November 06, 2016, 03:52:15 PM
I'm testing 10 and 12bit raw on my 5D Mark III, but why do I have MLV files if I use raw 1.0 to record?
Is it possible to add Status when recording to raw 1.0 to control in a better way how the recording is going?
Thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 06, 2016, 04:41:21 PM
...why do I have MLV files if I use raw 1.0 to record?

Go to the module menu and check raw_rec. On the raw_video_10bit_12bit branch it should report "Raw video v1.1 (MLV Lite)"

Any way to try the build for 6D?

That's included in the testing package I put in my bitbucket download area:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/raw_video_10bit_12bit_2016Nov04.zip

Note that I'm going to leave this up for a limited time. I may or may not update as changes come in depending on the demand of the testers. I'd really rather everyone search for my compiling tutorials and roll their own.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: extremelypoorfilmaker on November 06, 2016, 05:19:14 PM
just following the conversation :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 06, 2016, 05:36:05 PM
Note that I'm going to leave this up for a limited time. I may or may not update as changes come in depending on the demand of the testers. I'd really rather everyone search for my compiling tutorials and roll their own.

Agreed -- this is a great way to keep all of our projects moving forward in a nice pace otherwise!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: anto on November 06, 2016, 06:33:56 PM
first, great!!! very impressive....
now I need a guidehow to use mlv_dump, please  ::)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: tin2tin on November 06, 2016, 06:41:29 PM
Quote
mlv_dump:
You need to disable all processing, e.g.
mlv_dump M00-1234.MLV --dng --no-fixcp --no-stripes
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 06, 2016, 09:25:17 PM
Or you can use MLVFS (http://www.magiclantern.fm/forum/index.php?topic=13152.msg127218#msg127218)  on PC or Mac no update needed.
I'm using  "Quick Mount" on PC with the old Pismo/mlvfs.dll (just right click on file & quickmount  :D )
I think it's about 1 year old extraction workflow and has no issue
with the 10 & 12bit raw files .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 06, 2016, 11:47:39 PM
Wonderful  job Guys

Tested on 5d mark III 113

Crop mode:

3584 x 1320 10bit 24fps 8/9 sec. (106.2 mbps)
3584 x 1320 12bit 24fps 4/5 sec. (106.2 mbps)
2880 x 1310 10bit 24fps Continous  (104.8 mbps)
2880 x 1080 12bit 24fps Continous  (104 mbps)

no frames corrupted

I cant belive myself i'm very very happy

Thanks very mutch for your job.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 07, 2016, 10:17:51 AM
this is awesome! thanks dfort for pointing me to this. live view still freezing on the eos m? just tried 1472x884 resolution in crop mode on eos m in 10 bit. looks very nice and detailed. just live view is freezing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 07, 2016, 10:29:50 AM
I am testing this on 5d3 113. I am really shocked, with the old PC mlvfs (with pismo) 10 and 12 bits works like a breeze :) I am using resolve 12.5, sometimes only the first frames are corrupted but the rest is perfect. 10 and 12 bits recording is really a huge boost in recording times.

EDIT:

The most important and useful modes for me now:

1920x648 (1920x1080 16:9 after desqueeze) 50 fps 12bit - continous - (previously only 2,35:1 was continous)
1920x648 (1920x1080 16:9 after desqueeze) 60 fps 10 bit - 25 sek

1920x1080 37 fps 12 bit - 13 sek
                              10 bit - continous

2560x1320 25 fps (max vertical resolution available) 1.94:1 aspect - 10 bit - continous

2880x1226 2,35:1 25fps 10bit  - over 2 minutes
                              24 fps 10 bit - continous

I checked those recordings in resolve and sometimes there are first and last frames corrupted but no problems during clips.

FEATURE REQUEST:

in mlv rec module there is possibility to choose more resolutions, could somebody please add those resolutions to the 10/12bit raw rec module?
3200x1320 (2.42:1)
3168x1320 (2.40:1)
3104x1320 (2,35:1)

There might be some good recording times in 10 bit so we coul utilize all available pixels in vertical axis.

                                                                         
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 07, 2016, 01:41:41 PM
Guys,

could anyone of you share compiled 5Dm3.113 version?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 07, 2016, 01:48:50 PM
Couple posts above...

https://bitbucket.org/daniel_fort/magic-lantern/downloads

there is raw video 10 12 bit zip, you have 5d3 113 inside the zip...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 07, 2016, 02:42:00 PM
For fine resolution increments, you just have to port this change (https://bitbucket.org/hudson/magic-lantern/pull-requests/627/added-fine-control-on-top-of-x-resolution/diff) to MLV Lite.

The change should be simple enough for anyone with minimal C skills (perfect for new contributors looking for something easy to get started).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 07, 2016, 02:49:37 PM
Couple posts above...

https://bitbucket.org/daniel_fort/magic-lantern/downloads

there is raw video 10 12 bit zip, you have 5d3 113 inside the zip...

I cannot find module for 5D Mark 2. That means there is not a proper settings available for 5D2 yet where every 2nd frame is not pink?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on November 07, 2016, 03:35:19 PM
I cannot find module for 5D Mark 2. That means there is not a proper settings available for 5D2 yet where every 2nd frame is not pink?


Check this post : http://www.magiclantern.fm/forum/index.php?topic=5601.msg174302#msg174302

Just replaces the module in your ML installation : ML / modules / raw_rec.mo


But yes, with the 5DII version there are pink frames issues. 10 and 12bits
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 07, 2016, 06:33:21 PM
Works great with 5Dm3 and fps override set to 37fps. Much better then 48fps in squeezed resolution. Any plans adding this to mlv_rec module, so that we could use audio as well?
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 08, 2016, 07:15:27 AM
@kaco -- How is 37p much better than 48p?

In terms of Motion (aliasing/moire) or Aspect Ratio/Resolution?

It may seem more ideal to go w 37p to avoid the desqueeze in post (resolution loss) but it's very minimal especially w 5D3 that it shouldn't even be a concern IMO.

Do you have any samples that show otherwise?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 08, 2016, 08:27:47 AM
I cannot find module for 5D Mark 2. That means there is not a proper settings available for 5D2 yet where every 2nd frame is not pink?

Just replaces the module in your ML installation : ML / modules / raw_rec.mo

There were some changes to the back ends so it isn't just a matter of dropping in the module. Though I don't think that will solve the every other (pink) frame issue.

Sorry, I must have missed it while compiling all the platforms. I just put a new build on my bitbucket download area.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 08, 2016, 08:56:01 AM
@DeafEyeJedi -- in 48/50p the vertical resolution is 1.67x smaller, which kills details and causes aliasing and moire. Most of wide angle shots are quite bad.

Switching to canon 25/24p mode and then overriding to 37fps means much more detail for me. BIG chance for me, crippled resolution in 48p mode was a reason for me to start looking for a new camera.

look at the aliased edges:
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on November 08, 2016, 09:30:00 AM
@DeafEyeJedi -- in 48/50p the vertical resolution is 1.67x smaller, which kills details and causes aliasing and moire. Most of wide angle shots are quite bad.

Switching to canon 25/24p mode and then overriding to 37fps means much more detail for me. BIG chance for me, crippled resolution in 48p mode was a reason for me to start looking for a new camera.

look at the aliased edges:


Your work is just amazing ! Never saw your stuff before.
How did you grade it ? Which softs ? which luts (if used) ?
I dont saw informations.

For answer you about aliasing and moire on 50 and 60fps footage due to stretching, yes it's a small issue, but believe me, most of 95% people don't will see anything.
And for be honest, on your footage i'm not focused on it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 08, 2016, 11:29:03 AM
@GutterPump -- thanks, it's graded in Davinci. And really, using 490px vertical resolution in a 4K world is not so acceptable for professional work. :) This is game changing.

On topic -- is this going to be a part of mlv_rec? Is this in raw_rec module just during development?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 08, 2016, 12:24:06 PM
Results here with the 50D :

First frame
(http://www.justinbacle.com/temp/seq0.png)
Even frames
(http://www.justinbacle.com/temp/seq1.png)
Odd frames (except first one)
(http://www.justinbacle.com/temp/seq2.png)

Is it the same problem as the 5DmkII ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 08, 2016, 12:56:50 PM
@DeafEyeJedi -- in 48/50p the vertical resolution is 1.67x smaller, which kills details and causes aliasing and moire. Most of wide angle shots are quite bad.

Switching to canon 25/24p mode and then overriding to 37fps means much more detail for me. BIG chance for me, crippled resolution in 48p mode was a reason for me to start looking for a new camera.

You should check out the crop_rec module that is in development, it allows you to record in 48/50p without the vertical squeeze.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 08, 2016, 03:41:32 PM
You should check out the crop_rec module that is in development, it allows you to record in 48/50p without the vertical squeeze.

@rbrune Thanks for the tip, I've already tested it. But AFAIK the preview in live view would be squeezed (1.64x) in vertical resolution? And because of the bitrate with full 1920x1080, without 10/12 bit I would not be able to get 48p continuos, not?
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: aschille84 on November 08, 2016, 03:46:16 PM
@kaco I tried it yesterday, you can have 672 px max vertical resolution. So you can get something like 1600x672 in 50/60p unsqueezed footage. You have to change aspect ratio close to 1:1 to get max horizontal resolution
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on November 08, 2016, 04:11:12 PM
@kaco I tried it yesterday, you can have 672 px max vertical resolution. So you can get something like 1600x672 in 50/60p unsqueezed footage. You have to change aspect ratio close to 1:1 to get max vertical resolution

So 672px is limit for continuos shooting or is it a hard limit? If that's maximum (regardless bitrate), then this is not so great and a true 1920x1080 (or 1920x818) in 37fps is much more interesting.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aschille84 on November 08, 2016, 04:32:57 PM
So 672px is limit for continuos shooting or is it a hard limit? If that's maximum (regardless bitrate), then this is not so great and a true 1920x1080 (or 1920x818) in 37fps is much more interesting.
Yes, its a hard limit regardless of bitrate. And I agree 37fps full HD is more useful and interesting.


Sent from my iPhone using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 08, 2016, 04:35:05 PM
Quote from: justinbacle
Is it the same problem as the 5DmkII ?

Yes, and I also get it on the 550D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 08, 2016, 07:19:43 PM
You should check out the crop_rec module that is in development, it allows you to record in 48/50p without the vertical squeeze.

@rbrune Thanks for the tip, I've already tested it. But AFAIK the preview in live view would be squeezed (1.64x) in vertical resolution? And because of the bitrate with full 1920x1080, without 10/12 bit I would not be able to get 48p continuos, not?

@rbrune -- Have you tried merging your crop_rec with raw_video_10bit_12bit? If you have a branch for that in your repository I can compile a new set of test builds. I'd do it myself but I'm not nearly the coder you are and am busy with some other projects at the moment.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 08, 2016, 08:08:00 PM
@rbrune -- Have you tried merging your crop_rec with raw_video_10bit_12bit? If you have a branch for that in your repository I can compile a new set of test builds. I'd do it myself but I'm not nearly the coder you are and am busy with some other projects at the moment.
I had not but gave it a quick try. Recording in 10bit with 3x3 mode worked for me on the EOSM (of course without live preview, that still needs fixing on the EOSM).
Here you go:
https://bitbucket.org/rbrune/magic-lantern/branch/crop_rec_raw_video_10bit_12bit (https://bitbucket.org/rbrune/magic-lantern/branch/crop_rec_raw_video_10bit_12bit)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 08, 2016, 08:30:34 PM
... I can compile a new set of test builds. I'd do it myself but I'm not nearly the coder you are and am busy with some other projects at the moment.

I replied in your thread (http://www.magiclantern.fm/forum/index.php?topic=16012.msg174580#msg174580your thread) about a possibility to give you a helping hand with this task. Thanks @rbrune for sharing your branch!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 08, 2016, 09:13:42 PM
Nice!

Ok--what's next, taking a1ex's hint (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174497#msg174497) and adding more of rbrune's wonderful work like added fine control on top of x-resolution presets and display of effective crop factor (https://bitbucket.org/hudson/magic-lantern/pull-requests/627/added-fine-control-on-top-of-x-resolution/diff) for MLV Lite? I was about to look into that this morning but got stuck in focus pixel land (http://www.magiclantern.fm/forum/index.php?topic=16054.msg174582#msg174582) with the EOSM crop_rec video mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 08, 2016, 11:45:42 PM
I had not but gave it a quick try. Recording in 10bit with 3x3 mode worked for me on the EOSM.

After playing with it some more I noticed that my EOSM crashes (screen freezes and I can hear the lens/AF working but no reaction to button presses) when I try to record a second 10bit mlv after a first one (with/without crop_mode).
Can someone confirm this? (just record a 10bit mlv stop, wait a couple second and immediately record a second one).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 09, 2016, 12:39:36 AM
After playing with it some more I noticed that my EOSM crashes (screen freezes and I can hear the lens/AF working but no reaction to button presses) when I try to record a second 10bit mlv after a first one (with/without crop_mode).
Can someone confirm this? (just record a 10bit mlv stop, wait a couple second and immediately record a second one).

My EOSM is crashing on the first 10bit mlv. I reported it a few days ago (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174349#msg174349).

Quote
It works fine in the crop modes, mv1080crop and zoom with the same LiveView freeze and first frame corrupt issue as the 700D. There's something strange going on with the non-cropped modes. The camera records but the camera freezes when recording is stopped requiring a battery pull.

I just tried it with your crop_rec in 10bit and same thing. Works fine in Movie crop mode (600D movie crop-mode) and zoom mode. Well, if you accept LiveView freezing as "works fine."
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 09, 2016, 03:29:05 AM
@a1ex
Quote
For fine resolution increments, you just have to port this change (https://bitbucket.org/hudson/magic-lantern/pull-requests/627/added-fine-control-on-top-of-x-resolution/diff) to MLV Lite.

The change should be simple enough for anyone with minimal C skills (perfect for new contributors looking for something easy to get started).

Got it working. Since the request was made on this topic I made the pull request for the raw_video_10bit_12bit branch:

https://bitbucket.org/hudson/magic-lantern/pull-requests/767/raw_rec-mlv-lite-now-handles-resolution/diff

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 09, 2016, 02:36:33 PM
@dfort, are you able to compile a module for 5d3 113 with fine resolution changes?
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 09, 2016, 08:23:29 PM
@kgv5 -- here's what I was able to compile today for a few of my DSLR's so far (be aware that Dual-ISO isn't included as it is still temporiarly broken) on my environment until then. all cameras and they are currently available for download (check below).

5D3.113, 7D.203 & EOSM.202 for now -- https://mega.nz/#F!PgFFQKxY!KR7Odshgs-HmL5o1EM5Z-g

New link here -- https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads

Thanks to @rbune for sharing your branch and to @dfort for the PR in getting crop_rec.mo & 10/12-bit merged together.

@Everyone else -- I'll eventually compile for the rest once I've finally resolved with the Dual-ISO module issue on this Mac.

*edit*

This has finally been fixed on my Mac. Got it compiled for all cameras running ML except for 1100D.105 & 5D3.123 (will eventually figure out how to get this merged for 5D3.123) and special Thanks goes out to @dfort for getting me back on track with the compiling environment.  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lars Steenhoff on November 09, 2016, 08:33:45 PM
Thanks for the build, will test it now on the 5dmk3

Main thing I'm looking forward to is to see next is in camera playback, its nice to verify the shots are on the card.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 09, 2016, 08:51:02 PM
Based on my recent tests with the mlv_playback.mo from these builds on 5D3.113 & 7D.203 which seems to playback files like normal.

Even if they (10/12-bit files) are corrupted. Nice!  :P

(https://c8.staticflickr.com/6/5477/30892960415_5e7f7a48b4_n.jpg) (https://flic.kr/p/P4UuQx)

However with the EOSM.202 -- if I try to playback an MLV file recorded with either 10 or 12 bit it would load (the blue 'loading' graphic line) up to the very moment when we would expect to see the footage to no avail.

(https://c1.staticflickr.com/6/5549/30582188200_5ba731d0a4_n.jpg) (https://flic.kr/p/NArH5U)

It still stays on the end of the 'Blue loading graphic' within the GUI -- is this what you are (or were) seeing as well @Lars Steenhoff?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 09, 2016, 09:07:38 PM
@dfort, are you able to compile a module for 5d3 113 with fine resolution changes?

Sure, here you go.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit_fine_x_resolution.2016Nov09.5D3113.zip

I haven't tested it on that camera yet but now raw_rec should fine adjust just like it does in mlv_rec.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 09, 2016, 10:04:59 PM
Tested on 5d mark III 113
every things works ok
no frame corrupted

Crop Mode

3200x1200 23fps 10 bit Continous  (105.7 mbps)

3.5k Video Continous it's Amazing

Thank you very much  dfort
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kgv5 on November 09, 2016, 10:25:40 PM
Thank you very much guys  :D This is incredible.

My measurments:

Everything tested with 24fps 10bit on 5d3 113, CF sandisk extreme pro 128gb 160mb/s

3168x1320 2.40:1 (1.81x crop) - 14 sec

3104x1320 2.35:1 (1.85x crop) - 18 sec
3104x1298 2.39:1 (1.85x crop) - 21 sec

3008x1280 2.35:1 (1.91x crop) - 38 sec
3008x1258 2.39:1 (1.91x crop) - 56 sec

2976x1266 2.35:1 (1.93x crop) - 60 sec
2976x1246 2.39:1 (1.93x crop) - 123 sec

2880x1226 2.35:1 (2x crop) - continous

And wider aspect ratios - just for fun:

3520x1320 2.67:1 (1,63x crop) - 6 sec
3296x1320 2.50:1 (1.74x crop) - 12 sec

Now I would like to see the comparison: 5d3 vs 5d4 - raw 10bit 3008x1258 (quite usable recording time - 56 sec) vs MJPEG 8bit 4k. Not so much different crops - 1.91x 5d3 vs 1.7x 5d4. Similar bitrates. That would be very interesting.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 09, 2016, 10:38:45 PM
However with the EOSM.202 -- if I try to playback an MLV file recorded with either 10 or 12 bit it would load (the blue 'loading' graphic line) up to the very moment when we would expect to see the footage to no avail.

Can you upload a small MLV that shows the issue?

I assume the same MLV (recorded on EOS M) will fail to play in a different camera as well; can you check whether this is true?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 09, 2016, 11:00:16 PM
Can you upload a small MLV that shows the issue?

I assume the same MLV (recorded on EOS M) will fail to play in a different camera as well; can you check whether this is true?

Sure here you go (https://mega.nz/#!vw0z2SZY!L7EpC9MtIjrCTH9UlL6qdmaQcN4kQESAJbn4Gl5cVUohere you go) -- it's the same exact file (269.1 MB) from the screenshot above (M09-1246.MLV) and I can confirm that playing this particular file on a 7D which DOES playback fine until at the end which then I am prompted with this message...

(https://c8.staticflickr.com/6/5606/30885240615_020b72b310_n.jpg) (https://flic.kr/p/P4dW1x)

My gut feeling is telling me this may has to do with EOSM freezing while being recorded (even tho the file is still be written into card) and the only way to get the LiveView back on is to take the battery out after each take. Not sure if this would be helpful?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 09, 2016, 11:14:39 PM
The last few frames appear to have some trouble. Recording was stopped normally, or it crashed?

edit: looks like it crashed.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 10, 2016, 12:01:48 AM
I think we need to get the ML black and white preview working on the EOSM - the incorrectly setup LV for 10/12bit is probably messing with the camera and making it crash.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 10, 2016, 12:14:37 AM
With preview set to Canon, do you still get crashes?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 10, 2016, 08:23:18 AM
More testing on 5D Mark ii , by playing around with the SRM Job , SmallHacks & Preview I can now get 10bit 16x9 A.R. 1856x1044 29.97(68MB/s) , 23.976p(55MB/s)
continuous without any Frame corruption/pink noise  . I never have Raw Video 30p continuous before at near HD 1044p  :D
My settings:
29.97p(Set in Canon Menu)
Frame over ride - exact
Small Hacks - enabled
SRM Job      - disabled
Preview ML B/W with overlays killed when recording
Any other setting then this produce corrupted frames.

If you enable SRM Job you get the first frame corrupted with pink noise but all the rest are free of corruption.
with SRM Job disabled all frame including the first are free of corruption.

Now with 12bit I almost got it to product corruption free frame , there just a little color noise bar at the top of every other frame about 40 lines
Setting:
23.976p(continuous) (29.97p records for only 20-30 seconds)
Small Hacks - enabled
SRM Job      - enabled
Preview Canon with overlays killed when recording.

Curious that in 10bit Raw Video ,  Liveview needs to be in magic lantern black & white preview with overlays disabled
where 12bit Raw Video needs Liveview to be in Canon preview to be almost corruption free , any other preview cause total pink noise

I'll post some examples later .
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on November 10, 2016, 08:36:15 AM
With preview set to Canon, do you still get crashes?

When set to Canon the EOSM crashes when stopping the first 10bit recording.
With ML B&W or Hacked it crashes when starting the second 10bit recording but the first records fine.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 10:11:52 AM
When set to Canon the EOSM crashes when stopping the first 10bit recording.
With ML B&W or Hacked it crashes when starting the second 10bit recording but the first records fine.

I couldn't reproduce any of the crashing (unless you guys are referring to the LiveView freezing) but that's not exactly crashing, is it? especially if it's being recorded blindly.

However upon reading @reddeercity post above re: SRM job memory -- decided to try and fiddle with it and now for some reason if I were to record in either 10 or 12 bit on the EOSM with the SRM job memory enabled OFF then mlv_play.mo doesn't load. (notice in this screenshot below that the invalid header error is at the beginning as oppose to the previous issue (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174661#msg174661previous issue))

(https://c4.staticflickr.com/6/5695/30804715771_1eb67a32c0_n.jpg) (https://flic.kr/p/NW7dMa)

Although it had issues earlier and now if I were to record in either 10 or 12 bit on EOSM with the SRM job memory enabled ON then mlv_play.mo does load as well as playback files like normal.

(https://c8.staticflickr.com/6/5477/30892960415_5e7f7a48b4_n.jpg) (https://flic.kr/p/P4UuQx)

Really?!? What's even more odd is the fact that now all of sudden I can playback files on EOSM when earlier I couldn't with that particular file (probably due to a crash while recording) but I honestly don't recall any crashes happening on my watch during this time other than the usual LiveView freeze (don't remember if I kept it rolling till dropped frames or if I had stopped recording myself) but then after trying to reproduce this glitch to no avail and now fiddling with this SRM job memory somehow did the trick or no?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 11:14:02 AM
got some (probably) stupid questions... Is 8 bit raw a thing? you know to achieve higher resolutions? how about monochrome black and white mode for the same reason? has anyone tried using an external monitor to see if that is also freezing while live view is freezing on 10 bit? I'd try it myself but i don't have any monitors or cables to do so. I'm talking about eos m. thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SiSS on November 10, 2016, 01:44:33 PM
I think the consensus is that 10-bit acquisition helps you for noticeably better end results than just upping resolution.
There's more to maneuver with 10-bit. Every smartphone and cheap action camera has 4K = and that doesn't mean those deliver anything worthwhile.
Also it's been said that difference between 8-bit and 10-bit is "last" to be noticed with your eyes (10-bit 12-bit not so much, tough there's the material to work with).
Real 10-bit and you have think C300 mk II and C500. Arri Alexa didn't have 4K until SXT and even that doesn't come from 4K sensor. SXT been out for month? So 99% of the films in the movie theaters still don't have native 4K. Tough 10-bit doesn't mean everything either.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SiSS on November 10, 2016, 01:49:52 PM
Everything tested with 24fps 10bit on 5d3 113
2880x1226 2.35:1 (2x crop) - continous
Thanks kgv5, did you notice any difference in moire and/or rolling shutter?
Some test videos please!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 10, 2016, 02:32:28 PM
Little late at the party  :P
I never thought this was gonna happen, that the hardware in the cams simply were fixed for 14 bit.

But :o WOW :o Respect for these findings!!!  8)

So to put things in perspective...
Canon released a full-frame DSLR in 2008!!!  with some sort of full HD video feature in h.264 compression.
And all these time this cam could record 10 bit raw video in full HD continuously direct to the memory card  ;D
Canon could have saved some serious money by leaving out the chip that does the h.264 conversion(unless that is done by the digic  :P)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tullen on November 10, 2016, 05:11:26 PM
Hi. I admit I havent read through the whole thread, so forgive me if I missed it, but could this work on the 50D? Since I understand it has similarities with the 5D2.

Cheers
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 10, 2016, 05:34:42 PM
Hi. I admit I havent read through the whole thread, so forgive me if I missed it, but could this work on the 50D? Since I understand it has similarities with the 5D2.

Right now 1 out of 2 frames are garbage on the 50D. Same as the 5Dmk2 :)
Preliminary tests gives FullHD crop mode continuous w/ the 50D though
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 10, 2016, 05:56:36 PM
Right now 1 out of 2 frames are garbage on the 50D. Same as the 5Dmk2 :)
Really !
Did you even Read any Of the post here or just decided to write Wrong & miss leading
Information ? This Is a big problem here on the forum with new users , not willing to use
Searches or read threads & post and expect someone to give them information .
And Yea FYI 5d2 works with 10bit corruption Free
Give your Facts right !
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 10, 2016, 05:56:56 PM
5D2, Crop mode, 1920x1076 25p 10/12 bit continous. First frame broken, others have strong magenta cast (MLVFS, Davinci)

Any ideas?
 
https://drive.google.com/file/d/0B9d46mzo8guUQkN1YTRCRnhsdTA/view?usp=sharing (https://drive.google.com/file/d/0B9d46mzo8guUQkN1YTRCRnhsdTA/view?usp=sharing)

reddeercity: would you please share the proper 5D2 settings for non crop mode? i tried what you posted days ago (10bit 1856x1044, ML Grey, Clear overlay) but it didnt worked, every 2nd frame corrupted
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 10, 2016, 06:06:17 PM
Really !
Did you even Read any Of the post here or just decided to write Wrong & miss leading
Information ? This Is a big problem here on the forum with new users , not willing to use
Searches or read threads & post and expect someone to give them information .
And Yea FYI 5d2 works with 10bit corruption Free
Give your Facts right !

Sorry but there is no fix yet for the 50D. And post http://www.magiclantern.fm/forum/index.php?topic=5601.msg174548#msg174548 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174548#msg174548) said that the pink frames shouldn't be solved.
However there is a file https://bitbucket.org/daniel_fort/magic-lantern/downloads (https://bitbucket.org/daniel_fort/magic-lantern/downloads) specific to the 5dmk2, based on your reply, I guess it fixes pink frames on the 5dmk2.
Sorry if I gave wrong information, just trying to help here.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 10, 2016, 06:07:49 PM
Check my post at top of this page I gave my setting
That only works with 10bit right now . Try that if it dosn't work I can post more details.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 10, 2016, 06:42:03 PM
@Deadcode to fix magenta cast/black level on crop mode with 5D2 10bit use exiftool:
Code: [Select]
exiftool -BlackLevel=108 *.dngThis command fixes every frame in the directory, and for 12 bit the black level would be 4* that probably
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on November 10, 2016, 07:01:24 PM
Check my post at top of this page I gave my setting
That only works with 10bit right now . Try that if it dosn't work I can post more details.
Tried your fix, at first it looked like it worked but it actually doesn't on my 50D. The part that used to be "noise" is now a static frame.
On this example, the top part is flickering but is not what should have been recorded. One every two frame is "splitted" like that.
(http://www.justinbacle.com/temp/M10-1949.png)
Both your fixes produced similar results (10 & 12 bits).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 10, 2016, 09:04:14 PM
Check my post at top of this page I gave my setting
That only works with 10bit right now . Try that if it dosn't work I can post more details.



I tried exactly what you wrote on that post and still corrupted frames
29,970 exact, 1856x1044 10bit, Ml Grey, Clear when recording, SRM disabled, Small Hacks enabled.
I tried 12 bit setup too for 23,976p, i get the 40pixel corruption + the bottom half of the image is "wobbing".

Any suggestion?

*i used magiclantern-raw_video_10bit_12bit.2016Nov07.5D2212.zip
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 09:09:38 PM
Tried your fix, at first it looked like it worked but it actually doesn't on my 50D. The part that used to be "noise" is now a static frame.
On this example, the top part is flickering but is not what should have been recorded. One every two frame is "splitted" like that.
(http://www.justinbacle.com/temp/M10-1949.png)
Both your fixes produced similar results (10 & 12 bits).

I can also confirm this so called 'splitted' or 'slicing' across the frame horizontally occurring on some of my 7D files while shooting 10/12-bit. I'll need to investigate furthermore to determine what is the culprit in here (fps, mv1080/mv720, etc) and will report my findings when I can.

Once I can reproduce short test files then I'll upload them to share.

Quote from: Quote from: kgv5 on Yesterday
@dfort, are you able to compile a module for 5d3 113 with fine resolution changes?

As promised -- I've managed to get it compiled with the latest 10/12-bit raw_rec.mo (includes finer resolution adjustment) similar to what you get in mlv_rec.mo merged together with crop_rec.mo -- special thanks goes to @dfort for making this possible on my system! 

All cameras running ML except for 1100D.105 & 5D3.123 (I plan on figuring out a way to get it compiled for 5D3.123 at some point) and remember to keep all bug reports on this thread if you plan on testing these builds out. Enjoy!

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 09:11:34 PM
this 10 bit raw is the best thing that ever happened. i don't think I've ever seen such a good image coming out of a camera. so excited for live view to be working on eosm. I've got an a7sii sitting in my room. i haven't even used it in a month. much prefer the image on this 100 dollar camera. absolutely hilarious.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 09:20:06 PM
@deafeyejedi can't seem to get your link to work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 10, 2016, 09:33:07 PM
I get "Access Denied."
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 09:36:23 PM
same here access denied.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 09:47:21 PM
i actually never got your mega link to work either lol. we gotta get this guy online. valuable guy :P there's bitbucket settings on the top right do you have "private profile" checked?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 10:07:33 PM
Ah, ha. Thanks for the hints and @dfort was also able to help me out by pointing me to fixing this by going to Admin > Access > Disable Private (I don't recall turning this ON so this must be a default thing, right?).  :o

In any case -- could you guys please let me know if the link works now and Thanks again @Walter Schulz for dealing with yet another one of these moments.  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 10, 2016, 10:09:58 PM
Looks good now
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 10:11:56 PM
Looks good now

Thanks, Mate!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: canneloni on November 10, 2016, 10:26:31 PM
Is there still no build for the 100D ? I would like to test it and help as much as i can. It's really great what you guys are doing!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 10, 2016, 10:28:18 PM
Actually @dfort and I are still waiting to hear back from @nikfreak (http://www.magiclantern.fm/forum/index.php?topic=16054.msg174586#msg174586hear back from @nikfreak) on this very possibility (for both 100D & 70D) and will compile them as soon as we can -- we are just as anxious as you all are!   ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 11, 2016, 01:43:51 AM
@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test  ::)
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry :-[

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here (https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing)  .

 


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 11, 2016, 02:33:04 AM
Using MLVFS still getting every other half-pink frame. 550D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 11, 2016, 03:37:36 AM
Sorry guy should be Canon preview with overlays killed when recording for corrupt free @1044p 5D2
Edit:
30p is unstable , I can't now get it to record corruption free on every recording at the moment , some times it does so 50/50
but 23.976p is , sorry for the confusion I'll do some research  .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on November 11, 2016, 04:03:16 AM
No pro@reddeercity
Me too waiting to hear back from nikfreak for the 70D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 11, 2016, 09:07:33 AM
@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test  ::)
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry :-[

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here (https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing)  .

 

Still not working with these settings. I will make a video around 8pm (GMT+1 :) ) about it from installing ML till opening the dng's in Davinci, maybe you can spot the wrong settings.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Steve_FR on November 11, 2016, 07:45:03 PM
(2016Nov09.EOSM202)

I don't experience any crashing on the Eos M when recording with either zoom mode or 3x crop mode enabled.
Video playback works too, in both color and b&w. Video is scrambled.  Raw @ 10bit. Tried with SRM Job ON and OFF.
I can record multiple clips back to back with no crash, aside from the expected preview frozen only during REC.

If I shoot in (mv720 I believe?) (no crop, no zoom) then I experience the crash mentioned, and have to pop battery. I tried with SRM Job on and off, and the crash still occurred, followed by the invalid header size error when playing back the clip that was attempting to be recording during the crash. I can still play the other clips fine that were recorded using 3x or zoom, but scrambled as expected.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on November 11, 2016, 09:57:26 PM
@Deadcode , Ok I'll go thought my 5D2 Cam Setup
1) CF Card - 1066X Lexar , but any CF card 1000x or better. (FYI 10bit 1044p 30fps needs 68MB/s sustained write speed where as 24p only needs 55MB/s)
2) Canon Menu , Set FPS to 30p and video system to NTSC (because it works for me)
3) Set camera to Manual Mode  "M"
4) Load only modules you need ,  "5D2_212.sym , file_man.mo , raw_rec.mo ,  mlv_play.mo , mlv_snd.mo.
5) Either Load Raw or MLV not both at the same time
6) Frame Over Ride , set to 29.97 Exact.
7) Raw Video Tab , 1856x1044 , 10bit , 16x9 A.R. , Small Hacks - enabled , SRM Job - disabled
*Edit*
Sorry made a mistake , too many test  ::)
8 ) Preview ML B/W with overlays killed when recording
      Should be Canon Preview , Very Sorry :-[

That it , and I used  "magiclanertan-Nightly.2016Oct09.5D2212" 10 & 12 bit experimental build from here (https://drive.google.com/file/d/0BwvDlbhZgsGGbHBxeVpBTXJ4Zlk/view?usp=sharing)  .
 

I made a video about the process. What am i missing?

https://youtu.be/VI1u8Ouk_r8 (https://youtu.be/VI1u8Ouk_r8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 11, 2016, 11:32:36 PM
@Deadcode @reddeercity I have also tried all the settings you list and many other options, I can't get it to work without half corrupt frames, or splitting/slicing on the lower half, and I have the same build situation on my camera. But it is definitely amazing for crop mode, as it can be full res 2144x1076 and continuous :D :D :D :D :D :D :D :D :D :D :D How might this issue be fixed, and what help can the users give?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 12, 2016, 12:15:53 AM
@Deadcode , @Ilia3101 Ok I'll post a short video/images to how to get it working.


 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 12, 2016, 01:06:44 PM
Tested the 4 november version of Daniel on the 6d.
It works, sort of, although image preview is halted, the 'allow global draw' option is missing in the menu ?
And i'm missing sound, but that's probably normal for Raw_rec module  ;)

First frame is always corrupted.
10 bit works
12 bit works
But it doesn't work all the time, there was also a MLV file with all corrupted frames and a MLV file with frames with vertical lines corrupted, pink/magenta vertical lines and a posterized image behind it.
I've used MLV_dump.osx for getting the DNG's and viewed them in RawTherapee.

The vertical line correction in MLV_Dump always seems to use exactly 1.000 or 1 numbers with the 10bit files, the files look ok, but is it explainable that it is always exactly 1.000 ?

 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 12, 2016, 09:23:32 PM
Quote
Enabling the new method on all models is desirable IMO, because we will have a single way of doing things. However, each model is likely to have its own quirks, so it's a bit of work.
Trying to get slurp working on EOSM. I assume all I need to do is find a free edmac write channel and then hardcode the DEFAULT_RAW_BUFFER?

The hint in the code is
Code: [Select]
/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
 *
 * raw_buffer = get_raw_buffer()
 * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
 * ...
 * dump_file(filename, raw_buffer, 7*something...)
 */

I looked around in the disassembly of lv_raw_dump and called functions, but couldn't find any hardcoded values that looked like the other raw buffer addresses, at the beginning of lv_raw_dump EOSM appears to call two functions in RAM (not sure where to find these in ROM).

So I figured I would just call("lv_raw_dump"). Got a file on the card:
46798080.mm1

Also doesn't look much like the other addresses. (60D gave: 48332200.MM1, in raw.c the value is hardcoded as 0x5028)

Any tips?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 12, 2016, 10:33:25 PM
On 60D, the raw buffer address is returned by FF0F8B8C.

On EOS M, the function you are looking for is 10BF4. Add RAM_OFFSET from stubs.S =>  ffa7a184. This one returns MEM(0x404E4 + 0x44).

The raw buffer address is probably allocated dynamically (or maybe just hardcoded) from RscMgr.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 13, 2016, 05:23:49 AM
@Deadcode look like your setup is ok , well after some time testing different configurations on 5D2 I can't seem now to get 10 or 12 bit working consistently
ever time . When the camera & card are cold (no operated for at least 3 Hours) I can record 10 or 12 bit 1856x1044 @ 24p & or 30p but can't repeat the corrupt free
recording and when it does work I notice a message on the very top LCD screen "busy" that flashes on and off for about the first 20 of so seconds then is gone .

So 10bit @ 1856x1044 23.976p & 29.97p are continuous if you can get a corrupt free recording .
of which I did a few times . Plus I was able to get 12bit @ 1856x1044 23.976p , 29.97p at this bit depth that can not be maintain for very long.

Below I have posted three links to my Google Drive where I have 3 Corrupt Free recordings of 10bit 1856x1044 @ 23.976 & 29.97p & one of 12bit 1856x1044 @ 23.976p
In the drive folder there is for each bit depth & frame rate the Original .MLV file 1.0Gb , 1.1Gb , 550Mb the True 10 & 12 bit DNGs exacted with mlv_dump . There about
75 dngs for each file .

Google drive links
12bit 1856x1044_23.976p (https://drive.google.com/drive/folders/0B3rFQHNAb91MX2V0am1CYXNNRGs?usp=sharing)     10bit 1856x1044_29.97p (https://drive.google.com/drive/folders/0B3rFQHNAb91MbWE1T2gwdy11eHM?usp=sharing)    10bit 1856x1044_23.976p (https://drive.google.com/drive/folders/0B3rFQHNAb91MTU5hOE5hS1JKWm8?usp=sharing)

Now that bring me to the bit depth we have , there seem to be a problem with 12bit as Resolve or ACR/A.E. can't read or understand 12bit from mlv_dump.
10bit seen to read ok in A.E. & Resolve just 12bit. Why does this matter if you use MLVFS , yes  it does extract the cdngs ok from 10 & 12bit mlv's But
these are 16bit files (3.75Mb per frame) where as the 10 bit are (2.34Mb) & 12bit (2.8Mb) . So in some cases you are saving up to 1Mb per frame which can add up .
Plus the other factor is from 10bit (Billions of Colors) to 16bit(Trillions of Color) is the color accurate or will the color shift etc... .

Screen shot of A.E. CS6 with 12bit , can't read 12bit from mlv_dump 

(https://c4.staticflickr.com/6/5797/25311866219_fc85c3438c.jpg) (https://flic.kr/p/EyHVYx)
ML_Raw_12bit_dngs_AE-CS6 (https://flic.kr/p/EyHVYx) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Screen shot of A.E. CS6 ACR with 12bit , can't read 12bit from mlv_dump 

(https://c1.staticflickr.com/6/5683/30831184632_ec787db8ed.jpg) (https://flic.kr/p/NYrT3Q)
ML_Raw_12bit_dngs_ACR-AE-CS6 (https://flic.kr/p/NYrT3Q) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr


Screen shot of Blackmagic Resolve with 12bit , can't read 12bit from mlv_dump 

(https://c3.staticflickr.com/6/5825/30831184642_b4660b7183.jpg) (https://flic.kr/p/NYrT41)
12.5_Blackmagic_Resolve_12bit_dng (https://flic.kr/p/NYrT41) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
 
Edit:

Screen Shots Blackmagic Resolve with True 10 bit from mlv_dump

 (https://c6.staticflickr.com/6/5609/30949677445_6da74f3424.jpg) (https://flic.kr/p/P9VbQn)
12.5_Blackmagic_Resolve_10bit_dng (https://flic.kr/p/P9VbQn) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr


Screen Shots A.E. CS6 Timeline with True 10 bit from mlv_dump

(https://c4.staticflickr.com/6/5654/30949677475_7025edfccc.jpg) (https://flic.kr/p/P9VbQT)
ML_Raw_10bit_dngs_AE-CS6 (https://flic.kr/p/P9VbQT) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr


Screen Shots A.E. CS6 ACR with True 10 bit from mlv_dump

(https://c8.staticflickr.com/6/5654/30949677495_7c621cb37a.jpg) (https://flic.kr/p/P9VbRe)
ML_Raw_10bit_dngs_ACR-AE-CS6 (https://flic.kr/p/P9VbRe) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 13, 2016, 09:34:31 PM
I'm not a scientist but 10 bit looks great in resolve 12.5 using mlvfs after white balance in the raw tab on eos m.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 14, 2016, 12:10:27 AM
I'm not a scientist but 10 bit looks great in resolve 12.5 using mlvfs after white balance in the raw tab on eos m.
Sure it does because it's not 10bit anymore it's 16bit Cdng , that's point I was getting at MLVFS takes the 10bit image and up-samples
to 16bit even thou there's not 16 bits (well really 14bit) of information plus the file get blown up by at least 1Mb per frame.

Exact with mlv_dump for true 10bit and see if there's any difference .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 14, 2016, 12:47:43 AM
There will be no difference. It's not an "up-sample". The values are exactly the same. The number 123 represented with 10 bits or 16 bits or 64 bits or a million bits is still the same value, 123. CPUs can't do math on 10 bit numbers so it's going to get converted to 16 (or maybe even 32 or 64) internally at some point regardless.

Also, MLVFS files are virtual, so they are not actually taking up any additional real space.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 14, 2016, 03:35:08 AM
I tried a couple different edmac channels, but I'm just getting garbage, and with crop hack LV just freezes. Frame sizes are the ones reported by raw_info correct? That was the only other thing I needed to do?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 14, 2016, 09:54:32 AM
Frame sizes must be either hardcoded in raw_lv_get_resolution, or autodetected (as done on 5D3 on the crop_rec branch; you may have to tweak the width multiplier, as it's very likely to be lower than 8 ).

Once that is done, make sure raw_lv_vsync runs once for every frame (you can place bmp_printf's with small fonts there) and that raw_type_register is 0xC0F37014 (value for digic 5). Default raw type appears to be 0x22. Then, you could check whether edmac_slurp_complete_cbr gets called.

On 550D iirc, I couldn't get a raw stream until setting the raw type register.

You could also try the following changes:
Code: [Select]
raw_write_chan = 0x12
dmaFlags = 0x20000000
StartEDmac(raw_write_chan, 2)

Also, can you check the configuration of EDMAC channel #18 (Debug -> Show EDMAC) after enabling the LiveView raw stream with the old method?

The vsync hook looks ok to me.

A dm-spy log from LiveView might also help.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 14, 2016, 09:58:36 AM
Alex you talking about trying to get live view working?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on November 16, 2016, 04:37:55 PM
Any updates for the 5DM2?
Where can I find the latest builds?

Thank you all - you are the best  8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hjfilmspeed on November 18, 2016, 02:29:42 PM
This is AWESOME!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 19, 2016, 10:01:36 PM
You could also try the following changes:
Code: [Select]
raw_write_chan = 0x12
dmaFlags = 0x20000000
StartEDmac(raw_write_chan, 2)
That worked!

Very first try I got an Early Stop(5) and:
Code: [Select]
[89] raw_rec_task: NULL PTR (80c3e100,e1a00000)
pc=ff2c6020 lr=  ac5ad0 stack=1ddde0+0x1000
entry=abef78(0)
e1a00000 e59ff014 e59ff014 e59ff014
e59ff014 e1a00000 e59ff010 e59ff010

Everything worked fine after that, I did about 5 recordings in normal mode and about 5 in crop hack (I don't think there are any other video modes I can actually try with EOSM?). Also briefly tested 10 bit, appears to be working fine and Canon preview also working.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 19, 2016, 10:10:39 PM
All of those 3 changes are needed, right?

(I'm still unsure what EDMAC flags to use when, and how they affect the result...)

To diagnose the null ptr error, the mem_prot module can be helpful, as it can show the exact location that performed the invalid write. Or, a way to reproduce. There are some reports showing it on 60D as well.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 19, 2016, 10:32:40 PM
I only tried with all 3, if I get some more time I'll try excluding them.

For anyone interested in doing further testing here's what I did (on top of current raw_video_10bit_12bit). To check what a1ex asked about above simply change the two #ifdef CONFIG_EOSM lines in the edmac_raw_slurp() function to #if 0 (try each one separately and both)

Code: [Select]
diff --git a/platform/EOSM.202/internals.h b/platform/EOSM.202/internals.h
--- a/platform/EOSM.202/internals.h
+++ b/platform/EOSM.202/internals.h
@@ -142,3 +142,6 @@
 
 /** Workaround for menu timeout in LiveView */
 #define CONFIG_MENU_TIMEOUT_FIX
+
+/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
+#define CONFIG_EDMAC_RAW_SLURP
diff --git a/src/edmac-memcpy.c b/src/edmac-memcpy.c
--- a/src/edmac-memcpy.c
+++ b/src/edmac-memcpy.c
@@ -388,6 +388,10 @@
 uint32_t raw_write_chan = 4;
 #endif
 
+#ifdef CONFIG_EOSM
+uint32_t raw_write_chan = 0x12;
+#endif
+
 
 static void edmac_slurp_complete_cbr (void* ctx)
 {
@@ -401,7 +405,11 @@
 void edmac_raw_slurp(void* dst, int w, int h)
 {
     /* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
+#ifdef CONFIG_EOSM
+    uint32_t dmaFlags = 0x20000000;
+#else
     uint32_t dmaFlags = 0x20001000;
+#endif
     
     /* @g3gg0: this callback does get called */
     RegisterEDmacCompleteCBR(raw_write_chan, &edmac_slurp_complete_cbr, 0);
@@ -420,6 +428,10 @@
     SetEDmac(raw_write_chan, (void*)dst, &dst_edmac_info, dmaFlags);
     
     /* start transfer. no flags for write, 2 for read channels */
+#ifdef CONFIG_EOSM
     StartEDmac(raw_write_chan, 0);
+#else
+    StartEDmac(raw_write_chan, 2);
+#endif
 }
 #endif /* CONFIG_EDMAC_RAW_SLURP */
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -114,6 +114,10 @@
 //~ #define DEFAULT_RAW_BUFFER MEM(0x25f1c + 0x34)  /* 123 */
 #endif
 
+#ifdef CONFIG_EOSM
+#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
+#endif
+
 #else
 
 /* with Canon lv_save_raw, just read it from EDMAC */
@@ -430,6 +434,12 @@
     *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
     return 1;
     #endif
+   
+    #ifdef CONFIG_EOSM
+    *width  = video_mode_crop ? 1872 : 1808;
+    *height = video_mode_crop ? 1060 : 727;
+    return 1;
+    #endif
 
     /* unknown camera? */
     return 0;
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 19, 2016, 10:47:45 PM
I guess another thing someone can try now that slurp is implemented is to change the raw type register to see if you can get a focus-pixel-free raw stream (looking at you @dfort).

Code: [Select]
#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
just change #undef to #define
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 20, 2016, 12:00:44 AM
Test build pretty please?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 20, 2016, 07:30:20 AM
(looking at you @dfort).

Yikes!

Ok--first I pulled in the latest commits and dmilligan's patches have already been applied, check.

Then I went into src/raw.c and change one line to:

Code: [Select]
#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
Unfortunately it doesn't compile for the EOSM:

Code: [Select]
[ CC       ]   raw.o
../../src/raw.c: In function 'raw_update_params_work':
../../src/raw.c:590:10: warning: #warning FIXME: are these values correct for 720p and crop modes? [-Wcpp]
         #warning FIXME: are these values correct for 720p and crop modes?
          ^
../../src/raw.c: At top level:
../../src/raw.c:2070:18: error: 'lv_raw_type' undeclared here (not in a function)
         .priv = &lv_raw_type,
                  ^
make: *** [raw.o] Error 1

I thought I tracked down the problem down to this:

Code: [Select]
#ifdef PREFERRED_RAW_TYPE
static int lv_raw_type = PREFERRED_RAW_TYPE;
#endif

So it looks like we need to define PREFERRED_RAW_TYPE which is commented out for the 650D and 700D:

Code: [Select]
/**
 * RAW_TYPE 78 (and others) have a frame-wide green pattern on them
 * ACR can clear them but also kills some details and does not do
 * a good job in general. TL;DR Use pink dot remover.
 * http://www.magiclantern.fm/forum/index.php?topic=6658.0
 */

/*
#ifdef CONFIG_700D
#define PREFERRED_RAW_TYPE 78
#define RAW_TYPE_ADDRESS 0x351B8
#endif

#ifdef CONFIG_650D
#define PREFERRED_RAW_TYPE 78
#define RAW_TYPE_ADDRESS 0x350B4
#endif
*/

We can do better than the old pink dot remover with MLVFS.  8)

In any case, I've got a 700D so I uncommented that section but it came up with the a similar error:

Code: [Select]
[ CC       ]   raw.o
../../src/raw.c:2065:6: error: #error Only implemented for CONFIG_EDMAC_RAW_SLURP.
     #error Only implemented for CONFIG_EDMAC_RAW_SLURP.
      ^
../../src/raw.c:2070:18: error: 'lv_raw_type' undeclared here (not in a function)
         .priv = &lv_raw_type,
                  ^
make: *** [raw.o] Error 1

@Teamsleepkid and other testers who haven't set up a development enviornment yet -- here's an updated package with the latest changes for all the platforms (except 1100D) and the modded mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Nov19.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 20, 2016, 02:10:23 PM
You just need to define PREFFERED_RAW_TYPE for the EOSM. a1ex mentioned a few posts back that the default appears to be 0x22. Start from there and the use the menu to find the best one. You might want to search the forum for this topic, IIRC this was done a long time ago for other cameras, and there was some recent discussion about changing it for 5D3 to get rid of vertical banding. You might also want to split this discussion off into a new topic.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: glubber on November 20, 2016, 06:55:06 PM
Did some tests on the 550D with latest build from dfort:

12bit is now stable in 1080p25 and 720p50 (fps override to 24 of course)!


(https://c7.staticflickr.com/6/5512/30316190854_8afcbb259e_n.jpg) (https://flic.kr/p/NbWpkL)12bit-1080p25 (https://flic.kr/p/NbWpkL)

(https://c4.staticflickr.com/6/5701/31137213675_eb89b1e692_n.jpg) (https://flic.kr/p/PrumPD)12bit-720p50 (https://flic.kr/p/PrumPD)


10bit still results in every second frame corrupted

(https://c2.staticflickr.com/6/5732/31137213585_83b0323d36_n.jpg) (https://flic.kr/p/PrumN6)10bit-720p50 (https://flic.kr/p/PrumN6)


5x zoom creates a camera crash (Err70)

Code: [Select]
ASSERT: dwDeadLine != 0xFFFFFFFF
at LVState.c:5842, task LiveViewMgr
lv:1 mode:20


Magic Lantern version : Nightly.2016Nov19.550D109
Mercurial changeset   : b56f3ee02864 (raw_video_10bit_12bit) tip
Built on 2016-11-20 06:00:40 UTC by rosiefort@RosieFoComputer.
Free Memory  : 196K + 1678K

Hope this helps in taking the 10/12 bit fork further.
To all involved: keep up the awesome work  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 20, 2016, 07:26:02 PM
I tried the new builds on the 5D2 and pink frames are gone. Just split image on every other frame, about half of the frame for 10 bit is covered and most of it for 12 bit, the split part of the frame is always the same, seems to be from the first frame or something. Tested in 1856x1044 16:9 and 1856x1250 3:2 in 23.976fps.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on November 20, 2016, 08:24:37 PM
So close. 550D 12 bit 8 seconds at 1600x600 (2.67:1) but frames out of order. They're numerically in order but not correct. Like they are numbered wrong in camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 12:57:31 AM
The only real changes from the previous version pertain only to EOSM.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 21, 2016, 01:17:51 AM
You just need to define PREFFERED_RAW_TYPE for the EOSM. a1ex mentioned a few posts back that the default appears to be 0x22.

It seems that we also need to find RAW_TYPE_ADDRESS for the EOSM in order to get this working.

Code: [Select]
#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#define RAW_TYPE_ADDRESS 0xDEADBEEF
#endif

Otherwise it won't compile:

Code: [Select]
[ CC       ]   raw.o
../../src/raw.c: In function 'raw_update_params_work':
../../src/raw.c:595:10: warning: #warning FIXME: are these values correct for 720p and crop modes? [-Wcpp]
         #warning FIXME: are these values correct for 720p and crop modes?
          ^
In file included from ../../src/dryos.h:47:0,
                 from ../../src/raw.c:21:
../../src/raw.c: In function 'raw_lv_vsync':
../../src/raw.c:1538:42: error: 'RAW_TYPE_ADDRESS' undeclared (first use in this function)
         uint32_t raw_type_register = MEM(RAW_TYPE_ADDRESS-4);
                                          ^
../../src/mem.h:78:38: note: in definition of macro 'MEM'
 #define MEM(x) *(volatile uint32_t*)(x)
                                      ^
../../src/raw.c:1538:42: note: each undeclared identifier is reported only once for each function it appears in
         uint32_t raw_type_register = MEM(RAW_TYPE_ADDRESS-4);
                                          ^
../../src/mem.h:78:38: note: in definition of macro 'MEM'
 #define MEM(x) *(volatile uint32_t*)(x)
                                      ^
make: *** [raw.o] Error 1

Entering the wrong address will simply lock up the camera (requiring battery pull) when activating raw_rec.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 01:46:09 AM
I'm not sure what the address is, but I don't think you really need it since with this method there's no need to patch Canon code. The raw_type_register should be 0xC0F37014 according to a1ex.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 21, 2016, 02:15:49 AM
Well without an RAW_TYPE_ADDRESS defined it won't compile. That address you gave me produced this ASSERT LOG:

Code: [Select]
ML ASSERT:
raw_type_register == 0xC0F08114 || raw_type_register == 0xC0F37014
at ../../src/raw.c:1539 (raw_lv_vsync), task Evf
lv:1 mode:3


Magic Lantern version : Nightly.2016Nov20.EOSM202
Mercurial changeset   : b56f3ee02864+ (raw_video_10bit_12bit)
Built on 2016-11-21 00:51:03 UTC by rosiefort@RosieFoComputer.
Free Memory  : 195K + 3406K

I doubt that's the right address for the EOSM. It looks like the address was found for the 5D3 and that's probably the only platform that will work with the "#define RAW_DEBUG_TYPE" trick:

Code: [Select]
#define RAW_TYPE_ADDRESS 0x2D168
I tried that address on the EOSM and it locks up the camera when activating raw_rec requiring a battery pull.

I've also got a 700D I could use to test this but that camera doesn't have CONFIG_EDMAC_RAW_SLURP enabled and getting that working is beyond me at the moment.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 03:03:20 AM
I didn't say that was the RAW_TYPE_ADDRESS, I said it was the raw_type_register, and that you don't really need RAW_TYPE_ADDRESS. Look carefully at the code (specifically the code where it's used and where the failed assert you posted references)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 21, 2016, 05:00:15 AM
...you don't really need RAW_TYPE_ADDRESS...

If I understand your instruction, this should work:

Code: [Select]
#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#endif

However, it won't compile.

Changing this line from #undef to #define will also not compile which is how we got into defining PREFERRED_RAW_TYPE.

Code: [Select]
#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
I'm probably missing something obvious but the only way I've gotten #define RAW_DEBUG_TYPE to compile is to also come up with a value for RAW_TYPE_ADDRESS.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 21, 2016, 06:54:54 AM
thank you thank you thank you everyone :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 21, 2016, 10:36:36 AM
Did some testing with the 19 november build and new MLV_dump on the Canon 6d

First frame 00000.dng = always corrupted.

10 bit normal mode and crop mode-> always good
12 bit normal mode and crop mode-> sometimes wrong (diagonal purple stripes along the frames) sometimes good
I'm trying to figure out why 12 bit sometimes works and sometimes won't work ?


About the newest MLV_dump (I'm on osx), it only works with 10 bit.
In 12 bit mode, the newest MLV_dump finds about 40000 cold pixels, which results in frames with 40000 colored and black specles in them.
The same MLV files processed with MLV_dump version of may 2015 are fine.
In 10 bit mode, the newest MLV_dump finds zero cold pixels and gives normal frames.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 21, 2016, 10:40:56 AM
The live view freezes during recording in 10 and 12 bit.
Do you think this will be fixable, is canon live view capable of showing 10/12bit stream ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 01:04:13 PM
If I understand your instruction, this should work:

Code: [Select]
#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 0x22
#endif

However, it won't compile.

You have to do more than just #define stuff, you will have to change the existing code, but it should be easy (I was not saying that you won't have to do anything at all, just that you don't have to do any reverse engineering to find that memory address, a simple code change should avoid that work). Look where RAW_TYPE_ADDRESS is actually being used and remove it. If you know the register value you don't need the address, because the address is only used to look up the register. This is why I told you to look carefully at the code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 21, 2016, 01:18:03 PM
The live view freezes during recording in 10 and 12 bit.
Do you think this will be fixable, is canon live view capable of showing 10/12bit stream ?
Implementing CONFIG_EDMAC_RAW_SLURP fixed the issue on EOSM.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 21, 2016, 05:24:00 PM
...This is why I told you to look carefully at the code.

I can also look carefully at an airplane but I can't fly it.

Ok--figured it wasn't going to be this easy:

Quote
just change #undef to #define

I'll keep looking.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domasa on November 22, 2016, 03:59:19 PM
Quote
10 bit normal mode and crop mode-> always good
Do you see some real danger of damage camera 650D after use current build (19.11.) for 1 hour continuously  10-bit RAW recording?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 23, 2016, 12:32:29 AM
I have no experience with such long recording times.
But I can't imagine that it can cause permanent damage.



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 23, 2016, 09:53:05 AM
I have no experience with such long recording times.
But I can't imagine that it can cause permanent damage.

Maybe not to the camera but to the viewer!

Just wanted to report that I'm getting great performance out of the latest EOSM changes. About 20 sec. with 10-bit at 1728x692. But hey, this is the EOSM which doesn't have mv1080 and it is running at 29.97 fps. With FPS override set to 23.98 fps it is continuous. Live View is good and no corrupt frames.

I've gotten it to compile with "#define RAW_DEBUG_TYPE" but still haven't been successful in bringing up the debug menus. Got to keep looking at the code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 23, 2016, 11:24:08 PM
Quote
About 20 sec. with 10-bit at 1728x692
Continuous when shooting 24fps.
*oops, already stated above.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on November 24, 2016, 12:43:20 AM
Not sure if the first post is still active regarding testing for MLV Lite?

Just tried on latest ML build with 5D3 and got dropped frames after 3 sec. Not sure where it kicked in. Res was maxed 2:35.1.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 24, 2016, 04:21:03 PM
Messing around with compiling magic lantern for 6d
and was wondering about the bit modes:

Probably already tried by other users, but did somebody test what happens when you use values of 0x020 or 0x040 ?

+    /* for PACK16_MODE, DSUNPACK_MODE, ADUNPACK_MODE (mask 0x131) */
Add a comment to this line
+    const uint32_t MODE_16BIT       = 0x130;
+    const uint32_t MODE_14BIT       = 0x030;
+    const uint32_t MODE_12BIT       = 0x010;
+    const uint32_t MODE_10BIT       = 0x000;
+    const uint32_t bit_modes[] = { MODE_14BIT, MODE_12BIT, MODE_10BIT };
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 24, 2016, 05:25:44 PM
Code: [Select]
#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
Quote
just change #undef to #define

Ok--so it wasn't that simple but eventually it compiled.

Look carefully at the code

Code: [Select]
static struct menu_entry debug_menus[] = {
    {
        .name = "LV raw type",
        .priv = &lv_raw_type,
        .max = 64,
        .help = "Choose what type of raw stream we should use in LiveView.",
        .help2 = "See lv_af_raw, lv_rshd_raw, lv_set_raw, KindOfCraw...",
    },
};

There it is!

(https://c4.staticflickr.com/6/5538/31215434515_25d49d5d65.jpg)

So here's what I'm doing to get there--obviously still a work in progress:

https://bitbucket.org/daniel_fort/magic-lantern/branch/10bit_12bit_experiments#commits
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 24, 2016, 08:22:59 PM
Thanks for this work being done and especially now on the eos M. I did some testing with the raw stream from debug menu. Ran a recording on stream 64-45 but all of them exhibits the same focus pixel pattern. Are we certain streams are actually changed? How many streams are there anyway?
Also want to highlight the crop rec 3x3 720p mode which will yield superior H.264 HDR footage on the eos M and of course even better with the 5D mark III. Here is my workaround to output correct aspect ratio through ffmpeg.
http://www.magiclantern.fm/forum/index.php?topic=15108.msg175407#msg175407
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 24, 2016, 09:14:45 PM
I might have found a way to use 16px (or even 8px) increments horizontally without crashing Digic 5 cameras. The trick was to make sure W * H (yes, the product) is multiple of 16 bytes. The previous assumption was that W must be multiple of 8 bytes, and since H was always an even value, this appeared to work. Then, I've noticed 1904x1072 works fine, while 1872x1054 crashes. In both cases, (W*14/8) mod 8 is 4, so according to my previous hypothesis, they both should crash. Also, 1872x1072 works, 1872x1052 works, 1870x1052 fails, 1870x1056 works. See the pattern?

Still wondering why it works that way.

Committed on the 12-bit branch. This should make pixel (http://www.magiclantern.fm/forum/index.php?topic=5533.msg106435#msg106435) peepers (http://www.magiclantern.fm/forum/index.php?topic=11205.350) happy ;)

I did some testing with the raw stream from debug menu. [...] Are we certain streams are actually changed?

If you add a variable (with a menu) that is not used anywhere else, guess what will happen ;)

Check commit c3488df.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on November 25, 2016, 12:47:18 AM
Is there any build with crop + 10 12 bit + fine resolution? I find crop + 10 12 bit, also find 10 12 bit + fine resolution...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 25, 2016, 01:08:44 AM
Wow! happy to see the higher resolution back on the 5D2, it works at 1872 pixels, but it has a bug that allows it to go up to 1879, and once it has gone up there, it goes down to odd numbers like 1863, and the weird numbers cause recording to stop after 3 seconds and then all kinds of 'corruption' messages appear. But 1872 works!!!!! only tried 14 bit so far :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 25, 2016, 01:21:26 AM
Good catch, pushed a fix. Try it at all bit depths ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 25, 2016, 02:28:45 AM
I might have found a way to use 16px (or even 8px) increments horizontally without crashing Digic 5 cameras.  Also, 1872x1072 works, 1872x1052 works, 1870x1052 fails, 1870x1056 works. See the pattern?
This should make pixel (http://www.magiclantern.fm/forum/index.php?topic=5533.msg106435#msg106435) peepers (http://www.magiclantern.fm/forum/index.php?topic=11205.350) happy ;)
Thanks A1ex  , Cool The Mighty 5D2 Gets it's wings back  :D
Funny  :P  that's one of the mean reasons I started to get a compiling machine (Ubuntu) up and running , made my day !
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 25, 2016, 09:09:52 AM
compiled new update from a1ex on 5d2 , yes I'm back to 1872x1054 , 10bit thou was all corrupted (colored lines )12bit was almost perfect
every other frame had the top 100 lines or so shifted over a bit . This was a quick test used Auto preview for liveview that was good no lockup or freeze there .
Sample Cdng's 12bit 1872x1054 (https://www.dropbox.com/sh/zv2ee400cjt68id/AAC4Kp5GSRu5BAZ4bCBW-8aka?dl=0)  Sample all corrupted Cdng's 10bit 1872x1054 (https://www.dropbox.com/sh/39p7m7tzzf26wvw/AABgiyAhzI0GbaDcxtGCOnxra?dl=0) from my dropbox
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 25, 2016, 09:44:23 AM
Quote
If you add a variable (with a menu) that is not used anywhere else, guess what will happen  :P
Not much I guess  :P
And I need to learn C...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 25, 2016, 01:24:39 PM
5D2 update: whatever changes happened overnight, it has turned 1872 in to 1880, but now there are constant messages all over the screen saying things I don't understand, but it does work, 1880x1056 works, so does 1880x1248, but not 1880x1250, which seems to fit the pattern.
(http://gdurl.com/ftU2)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on November 25, 2016, 01:39:18 PM
1250 is not divisible by 16? See A1ex post up a bit.^ I dunno thou.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: NedB on November 25, 2016, 09:16:49 PM
Guys - sorry for (stupid?) question (not a beginner, been using ML since 2012). Downloaded and installed dfort's 10-bit/12-bit build from Nov. 19 for my 550D. But, try as I might, I cannot find where to set the bit depth to 10-bit or 12-bit for testing purposes. Anybody wanna cut me some slack and point it out explicitly for me? Does it require certain modules to be loaded or other settings to be set correctly in order to appear? Thanks, gentlemen!

NedB


Edit: Thank you very much for insanely quick reply, Levas!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 25, 2016, 09:20:53 PM
10 and 12 bit options are only available in the raw_rec module and not available in the probably more used MLV_rec module.
So enable raw_rec module and unleash the power  ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 25, 2016, 11:05:15 PM
First of all -- thanks to dmilligan for having the patience to work with me on this.

NO MORE FOCUS PIXELS!

(https://c4.staticflickr.com/6/5607/30431903123_21e0ee762d_z.jpg)

Here's what does the trick--only working on the EOSM at this time:

Code: [Select]
#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 18
#endif

This is a before shot (using PREFERRED_RAW_TYPE 34)

(https://c1.staticflickr.com/6/5522/31203782576_d42de1350a_z.jpg)

and here is what it looks like with PREFERRED_RAW_TYPE 18

(https://c6.staticflickr.com/6/5503/31125394261_7f266185aa_z.jpg)

I'm still running some tests and haven't gotten it working with the crop_rec module yet but I wanted to share the news.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 25, 2016, 11:33:31 PM
Holy shit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lars Steenhoff on November 26, 2016, 12:15:55 AM
Great work  guys!  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 26, 2016, 12:23:24 AM
You guys are the real Focus Pocus!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on November 26, 2016, 02:43:36 AM
updated mlv_rec and mlv_play for bit depth selection.
mlv_play requires raw_twk (http://magiclantern.fm/forum/index.php?topic=13163.0) if you want 10/12 bit playback.

on my 5D3 setup recording still is a bit buggy, every other frame is corrupted.
anyone can confirm?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on November 26, 2016, 03:42:55 AM
5D2: I can confirm every other frame is corrupted in 10 and 12 bit mlv_rec, top half of the image in 10 bit and a small bar of noise at the top for 12 bit. mlv_play + raw_twk does work with 10 and 12 bit, but everything is very green and crushed, probably black level.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 03:49:36 AM
...every other frame is corrupted...

I don't have a 5D3 to run the test I just did with the EOSM but I saw lots of PREFERRED_RAW_TYPE settings that had the alternating frame corruption issue you are describing. I went through all of the 64 possible settings until I found the one that doesn't show focus pixels. Maybe it is time to recheck the PREFERRED_RAW_TYPE setting on other cameras too?

Checked out your latest changes to mlv_dump but I'm getting a "Segmentation fault: 11" when turning on "--cs2x2". It looks like this isn't an issue specific to the 10bit_12bit branch.

http://www.magiclantern.fm/forum/index.php?topic=7122.msg167507#msg167507
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 26, 2016, 05:54:40 AM
@dfort great stuff the only thing missing is MV1080 , I think I'll keep my eye on a good used EOSM  :)
@g3gg0 same as @Ilia3101 on my 5D2 even tried different Liveview previews setting even the ones that freeze up Liveview
Also tried  29.97p fps in canon menu 1856x1044 raw mlv. Still the same as 24p

Corrupted Frame 10bit 1856x1044 23.976p MLV
 (https://c5.staticflickr.com/6/5580/30423382844_c5ccf57e01_n.jpg) (https://flic.kr/p/NmpMMw)
M25-2137_000545 (https://flic.kr/p/NmpMMw) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Good Frame 10bit 1856x1044 23.976p MLV
(https://c7.staticflickr.com/6/5662/30877145430_d1ed1a0ae0_n.jpg) (https://flic.kr/p/P3vrAo)
M25-2137_000546 (https://flic.kr/p/P3vrAo) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Corrupted Frame 12bit 1856x1044 23.976p MLV
(https://c5.staticflickr.com/6/5559/31100842892_ac433d85bf_n.jpg) (https://flic.kr/p/PogX4b)
M25-2139_000000 (https://flic.kr/p/PogX4b) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Good Frame 12bit 1856x1044 23.976p MLV
(https://c7.staticflickr.com/6/5483/31100841422_9d13150edf_n.jpg) (https://flic.kr/p/PogWBQ)
M25-2139_000001 (https://flic.kr/p/PogWBQ) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 07:43:54 AM
@dfort great stuff the only thing missing is MV1080

The really great stuff for the EOSM is happening with @rbrune here:

http://www.magiclantern.fm/forum/index.php?topic=16608.msg175360#msg175360

A camera you can pick up for $150-$200 USD that can shoot 10-bit raw video--and you can adapt just about any lens from a C-mount to a PL-mount?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 08:51:53 AM
Did a quick test with the updated mlv_rec from g3gg0 on the 6d and got ZERO corrupted frames with 10bit.
Recorded 3 MLV files in 10bit and all frames are fine, even the first ones.

I used mlv_dump.osx (build from may 2015) and all frames look fine in Rawtherapee.



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on November 26, 2016, 08:53:34 AM
no focus pixels!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 08:53:41 AM
Forget to mention.
The canon liveview freezes during recording.
The ML grayscale doesn't freeze during recording, but doesn't look like it can handle 10bit right now.

Is this normal, does canon live view freezes with the use of 10bit on all camera models ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 08:56:01 AM
Gonna shoot some files on my 5D mark III soon.
Quote
Checked out your latest changes to mlv_dump but I'm getting a "Segmentation fault: 11" when turning on "--cs2x2". It looks like this isn't an issue specific to the 10bit_12bit branch.
This issue is solved in unified so it,s in the 10-bit/12-bit code?

http://www.magiclantern.fm/forum/index.php?topic=7122.msg174592#msg174592
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 26, 2016, 09:30:10 AM
...I used mlv_dump.osx (build from may 2015)...

Really?

...and all frames look fine in Rawtherapee.

Care to share screenshots?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 09:34:58 AM
This issue is solved in unified so it,s in the 10-bit/12-bit code?

Yes--so it looks like somehow this commit (https://bitbucket.org/hudson/magic-lantern/commits/33843737460f1b0323a7b46fc0e81198ea44ab9b) didn't make it into the 10bit_12bit branch. Just applied it and --cs2x2 works fine.

This is needed to get rid of the hot pixels that sometimes appear in high contrast boundaries. Not sure if those are only in cameras that have (had?) the focus pixel issue.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 10:17:32 AM
g3gg0, you are king. And major thanks to Greg for work on this via DARK code? http://www.magiclantern.fm/forum/index.php?topic=13408.msg172108#msg172108. Who did I forget?
Just added 10/12-bit branch into unified and added the raw_twk.mo from here http://www.magiclantern.fm/forum/index.php?topic=13163.msg174991#msg174991
and playback is more or less realtime with all bits 10,12,14. it even works on the eos M!, just tested briefly.

Quote
on my 5D3 setup recording still is a bit buggy, every other frame is corrupted.
anyone can confirm?
I get corrupted files(white level off I think) on my 5D mark III every now and then in both 10 and 12 bit. Also from time to time a text with "raw detect error"(I think it says) on the left. Canon eos M is the more stable participant right now it seems(thanks dmilligan, a1ex, dfort and others)
Shortened examples of corrupted files
test_1
https://drive.google.com/file/d/0B4tCJMlOYfircGtCRlNSWFFGU2M/view?usp=sharing
test_2
https://drive.google.com/file/d/0B4tCJMlOYfirckxRMmJqMm1EaFE/view?usp=sharing
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on November 26, 2016, 12:30:22 PM
@Danne:
happy to hear that :)

yeah the black/white level thing is maybe still an issue. tried to fix it by scaling white level properly when upconverting to 16 bit in mlv_play.
but maybe there is still smth twisted.

does mlv_lite perform better with recording, i.e. no corrupted frames?
didnt test t.b.h.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 12:44:04 PM
Only tested with mlv lite. Is 10 bit working with mlv_rec?? Not home to test atm.
One quirk with eos m is that play, pause, delete buttons are suppressed when running playback through raw_twk.mo. Buttons works great with 5D mark III.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on November 26, 2016, 01:25:24 PM
i updated it also, so yes.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 03:26:32 PM
Will check later. Most greatful.
If anybody wants to try here are builds for 5D mark III and eos m with latest stuff. Raw_twk.mo included.(removed for now, causing corrupted black level)

Briefly tested. If anything breaks you get to keep all pieces.

*Black level issues fixed by g3gg0. New builds up
https://bitbucket.org/Dannephoto/cr2hdr/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 26, 2016, 04:48:00 PM
Quote
Will check later. Most greatful.
If anybody wants to try here are builds for 5D mark III and eos m with latest stuff. Raw_twk.mo included.
Briefly tested. If anything breaks you get to keep all pieces.
https://bitbucket.org/Dannephoto/cr2hdr/downloads

Hi i tested the last build on 5d Mark III sometimes it work correctly
but often the video is completely black
i send you a dng : https://www.dropbox.com/s/odn6xxrubhup6hz/M28-1717_000000.dng?dl=0

 
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 05:12:15 PM
@baccinoluigi
Any difference in stability between mlv_rec and mlv_lite?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 05:15:16 PM
Had some time to do some more mlv_rec testing on 6d.

Tested 10bit, 12bit and 14bit.
10bit works, for all frames and both old mlv_dump and newest mlv_dump works.
12bit works, only old mlv_dump gives purple vertical lines, newest mlv_dump detects false cold pixels and gives frames with too much specles because of the false cold pixel detection.
14bit, still works ;D

I'm using rawtherapee to view the files.

Uploading some dng files right now (few minutes uploading remaining)
http://drive.google.com/open?id=0B1BxGc3dfMDabVFDeG11QjBmWlk (http://drive.google.com/open?id=0B1BxGc3dfMDabVFDeG11QjBmWlk)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 26, 2016, 05:23:15 PM
@Danne
 
Tested again but mlv_rec and mlv_lite have the same issue

14 bit is more stable
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 07:30:17 PM
Still wondering, how is liveview during recording on other cams ?
On the 6d, canon liveview freezes during 10bit recording, Grayscale preview is messed up during 10bit recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 26, 2016, 09:16:50 PM
@baccinoluigi:

Workaround:
Code: [Select]
exiftool *.dng -BlackLevel=128

Is the issue specific to a particular build, or all of them are buggy?

@Levas:

Your question was already answered by dmilligan earlier in this thread.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: robertgl on November 26, 2016, 09:23:06 PM
Hello 50D user here
1. Magenta frames. It shows up on 10,12 and 14 when using the available modified fw. I haven't tried to change the black levels with mlv_dump, assuming that's the way to do it

2. The first frame keeps showing up in the even frames. So it's like 0,1,0,3,0,5,0,7,0,9,

3. at 1856x*** I get full frames interspersed with that continuing first frame showing up in even frames, while if I go to 1920x**** I get torn frames  with most of the frame being the first frame repeating, while a small sliver is the properly sequenced frame

4. Grayscale preview seems to run fine when recording

5. fingers crossed that the bugs can be ironed out because recordings look amazing and have a smaller files size  8)

Am exporting with : mlv_dump --dng --no-fixcp --no-stripes
debayering in rawtherapee
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on November 26, 2016, 09:29:56 PM
testing on 700D, builded from official hudson repository, raw_video_10bit_12bit branch about 10 minutes ago.

Found a "BUG", setting full HD res, no crop mode, going over the limit it uses max res possible on camera (1736x976) not divisible by 16 so crash happens after a few secs recording, this in both mlv_rec and raw_rec modules.

here logs after that crash
https://drive.google.com/open?id=0BxMUvr5E8ANaMDZUbkJnTHFya28  LOG000.log
https://drive.google.com/open?id=0BxMUvr5E8ANaci12Y2xUSU1MREE CRASH00.log

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 26, 2016, 09:44:04 PM
@a1ex
Quote
Is the issue specific to a particular build, or all of them are buggy?

The issue it's just in the build downloaded here : https://bitbucket.org/Dannephoto/cr2hdr/downloads
whith raw_twk.mo for 5dMkIII
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 10:04:22 PM
Difference between raw_rec (MLV Lite) and mlv_rec on EOSM.

raw_rec is fine at all bit depths.

mlv_rec is fine at 14 bit, shows some corruption on every other frame at 12 bit and the corruption is worse at 10 bit.

12bit:
(https://c5.staticflickr.com/6/5769/31115024412_468e17ea64_z.jpg)

10bit:
(https://c5.staticflickr.com/6/5781/30437512284_4e5101ab1c_z.jpg)

No focus pixels though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 10:30:18 PM
Tested some more around raw_twk on my eos m. Now I,m not sure if raw_twk is to be implemented on all cams. Anyway. Since it works really fast I think it,s a badass feature.
When recording mlv_rec and sound on an error message is shown Threads failed to start. When raw_twk.mo is off recording works. Is sound even working on eos m to begin with? I am not too familiar with this cam.
Mlv_play module has no start, stop, pause buttons when run through raw_twk. Can,t scroll around to the next video etc.
Like dfort points out mlv lite is producing good files while mlv_rec inherits corruption.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 10:54:59 PM
It seems raw_twk is causing mlv_lite(raw_rec) to produce the files with bad black levels on the 5D mark III. It happens once you start mlv_play and after this the next file usually turns out bad. When raw_twk is off everything seems fine when running mlv_lite. I will change my shared builds to builds without the raw_twk.mo for now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 10:58:48 PM
@alex ah I missed that one from Dmilligan
@Dmilligan

Can you help me here, I can see the following code in edmac_memcpy.c (see italic below)
I wanna fix live view during recording for 6d, how do I find a free edmac channel with free edmac channels ?


/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#ifdef CONFIG_EDMAC_RAW_SLURP

/* for other cameras, find a free channel with find_free_edmac_channels  */
#ifdef CONFIG_5D3
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_60D
uint32_t raw_write_chan = 1;
#endif

#ifdef CONFIG_600D
// write-index 1, 4, 6, 8, 10, 11, 13
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_EOSM
uint32_t raw_write_chan = 0x12;
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 11:03:51 PM
I can see with show edmac in the Debug menu and scroll trough the channels, that channel 7, 8 and 13 are empty...
Is it that easy ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 26, 2016, 11:16:41 PM
Well...I can confirm that adding this line


#ifdef CONFIG_6D
uint32_t raw_write_chan = 7;
#endif



And compiling mlv_rec doesn't work... :-\
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on November 26, 2016, 11:17:58 PM
@Danne:
black level issue should be fixed now
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 11:37:27 PM
Great man!
Confirmed fixed. Tried with 10-bit and with raw_rec on 5D mark III. raw_twk works and no green messed up files. Will put up new builds here again.
https://bitbucket.org/Dannephoto/cr2hdr/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 26, 2016, 11:38:10 PM
False alarm on the focus pixel elimination. a1ex pointed out that this has been tried in the past:

https://www.magiclantern.fm/forum/index.php?topic=6658.msg69400#msg69400

and things haven't changed with 10bit 12bit:

(https://c3.staticflickr.com/6/5459/30892543730_be804328eb_z.jpg)

Bummer.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 26, 2016, 11:42:02 PM
And cold pixel fix on this?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 27, 2016, 12:01:53 AM
And cold pixel fix on this?

Doesn't fix it.

I'm running at test now to see if maybe there is some sort of a pattern to this. In any case this doesn't seem to be a 10bit/12bit issue so we're off topic.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: baccinoluigi on November 27, 2016, 12:03:09 AM
@Danne
Tested on my 5dMkIII 10 and 12 bit
it works great
Thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on November 27, 2016, 12:07:48 AM
Great man!
Confirmed fixed. Tried with 10-bit and with raw_rec on 5D mark III. raw_twk works and no green messed up files. Will put up new builds here again.
https://bitbucket.org/Dannephoto/cr2hdr/downloads

Can you add crop rec module to your build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on November 27, 2016, 12:38:03 AM
@eNnvi: 1736x976 is divisible by 16 bytes at all bit depths. For 700D, the horizontal resolution is 1808. Therefore, EDMAC configuration must be, with the notation from here (http://www.magiclantern.fm/forum/index.php?topic=18315):

10bpp: (2170, skip 90) x 975, 2170
12bpp: (2604, skip 108) x 975, 2604
14bpp: (3038, skip 126) x 975, 3038

All these configurations work fine on 5D3, so I'm not sure what the issue is.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on November 27, 2016, 01:00:27 AM
Ok still have to rightly understand the code (expecially Emacs, my arm knowledge is stuck on Cortex m0 and m4 from stm)

I'll open another thread to understand the whole firmware (i know it's hard) so i Can begin writing a wiki for developers and later for users.

So if i understood it right your calc is (W*H*bpp)%16==0 ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on November 27, 2016, 07:35:59 AM
Doing some more testing with 5D2 on 10-12bit raw with updated resolutions , A very strange thing is happen when I use 1872x#### in 10 or 14bit I get corruption but 12bit is almost good just some tearing on top 50 or so line no color noise the image is good . When I select 1856x#### 14bit is fine. It seems that 1872 only work with 12bit without crashing (early stop with all corrupted Cdng's, color bars and about 100 no more then that ) with Canon preview . Anything with 1880 in all 3 bit ranges are totally corrupted and crashes the recording session , ( early stop etc..) but not the camera . You would think in 14bit the higher resolutions above 1856 should not be corrupted , even if it's too much data bandwidth  it just should drop a frame and stop recording .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on November 27, 2016, 11:27:53 AM
The corruption seen in the 10bit and 12bit dforts 543 post looks like its to do with bit planes https://en.m.wikipedia.org/wiki/Bit_plane, guess this is obvious to those understand math and code, not me.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 27, 2016, 11:36:28 AM
Trying to fix live view during recording for 6d and found some more pieces of the CONFIG_EDMAC_RAW_SLURP puzzle.

I found some references in edmac-memcpy.c and raw.c
Already added some extra code for the 6d to the emac-memcpy.c file.(see end of this post)
Now I found this piece of code in raw.c, I need to find DEFAULT_RAW_BUFFER MEM address for the 6d.
It says on top in the descriptions how to get that address(see bold text), but I'm too stupid too understand it :P
Can anyone help me here and tell me how to find this memory address?

#ifdef CONFIG_EDMAC_RAW_SLURP
/* undefine so we don't use it by mistake */
#undef RAW_LV_EDMAC

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
 *
 * raw_buffer = get_raw_buffer()
 * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
 * ...
 * dump_file(filename, raw_buffer, 7*something...)
 */


#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif

#ifdef CONFIG_600D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x51FC))
#endif

#ifdef CONFIG_5D3
#define DEFAULT_RAW_BUFFER MEM(0x2600C + 0x2c)  /* 113 */
//~ #define DEFAULT_RAW_BUFFER MEM(0x25f1c + 0x34)  /* 123 */
#endif

#ifdef CONFIG_EOSM
#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
#endif

#else

/* with Canon lv_save_raw, just read it from EDMAC */
#define DEFAULT_RAW_BUFFER shamem_read(RAW_LV_EDMAC)

#endif


In edmac-memcpy.c I already added the 6d to this piece of code
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#ifdef CONFIG_EDMAC_RAW_SLURP

/* for other cameras, find a free channel with find_free_edmac_channels  */
#ifdef CONFIG_5D3
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_60D
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_600D
// write-index 1, 4, 6, 8, 10, 11, 13
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_EOSM
uint32_t raw_write_chan = 0x12;
#endif

#ifdef CONFIG_6D
uint32_t raw_write_chan = 8;
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 27, 2016, 01:49:48 PM
a1ex gave me some tips a couple pages back. Start by looking up the function lv_raw_dump in Canon firmware (pretty easy to find, it's an eventproc). It should be calling a function that returns a hardcoded pointer (or in 60D's case pointer to a pointer) to the raw buffer address.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 27, 2016, 09:49:34 PM
I read the tips Alex gave you and I'm even more confused now, and that stuff was in raw_rec module, although the code should be much the same.
How do I look up the LV_raw_dump function in Canon firmware ?
I've made a rom dump, load it in a texteditor and found two times the text 'lv_raw_dump', but around it was a bunch of unreadable rom code.

Maybe you or Alex can give some more fool proof help ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 28, 2016, 04:17:36 AM
https://www.magiclantern.fm/forum/index.php?topic=12177.0

I like to use the HTML pages generated by ARM Console (http://magiclantern.wikia.com/wiki/GPL_Tools/ARM_console). Getting it setup is no easy task and it make take several days or more to run.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 28, 2016, 11:15:50 AM
Quote
Getting it setup is no easy task and it make take several days or more to run.

Real motivating words :P

I was hoping for something more simple, but hey, if this is the way how to get live view back, I'll shall take a look at it.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on November 28, 2016, 09:47:53 PM
BUILD magiclantern-Nightly.2016Nov26.5D3113_raw_twk

When I press Zoom for focus, absolutly brick the camera.
Battery out resolve the problem.

10 Bits has the pink problem on the top  and 12 bits image on the higlights.
Not problem at all in 14 bits

Changing the aspect to 4x3 (1920 x 1280) has pink color on the higlights as the 12 bits.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 11:04:05 AM
I'm first trying to go this route to find the 6d address for lv_raw_dump
https://www.magiclantern.fm/forum/index.php?topic=12177.0 (https://www.magiclantern.fm/forum/index.php?topic=12177.0)

But I'm a little stuck, I do have 'disassemble.pl' file half working.
I do get the ROM1.BIN.strings file, but the other two, .dis and .labels are empty.
Terminal output:
------------------------------------------------------------------
string dump

create elf file
sh: arm-elf-objcopy: command not found
sh: arm-elf-objcopy: command not found
label scan
sh: arm-elf-objdump: command not found

disassemble and string lookup
sh: arm-elf-objdump: command not found

job complete!

--------------------------------------------------------------------
I think I don't have the path locations right.
The 'gcc-arm-none-eabi-4_8-2013q4' folder is in my user root directory.
And the original disassemble.pl file also asks for '/gcc-4.1-arm/bin/' directory, which I don't have (but that is probably old version of the 'gcc-arm-none-eabi-4_8-2013q4'folder' ?

This is now in my disassemble.pl file
----------------------------------------------------
# adjust these for your needs (note final slash):
#$path = "$ENV{'HOME'}/gcc-arm-none-eabi-4_8-2013q4/bin/";
$path = "";
----------------------------------------------------

Anybody now how to setup the path to the /gcc-arm-none-eabi-4_8-2013q4/bin/ directory which is in the root of my user directory on mac osx
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on November 29, 2016, 01:15:10 PM
(http://i.imgur.com/9nHd6FZ.png)

I tried the @nikfreak :s 7d build and it didn't work at all. 10 and also 12 bit neither of these worked (i tried mlv_rec). I tried crop mode and regular mode.  Same thing also showed on the camera screen when playing back.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on November 29, 2016, 01:31:35 PM
Well it works with a little corruption here and there for the 7D(Used Nikfreak,s build). Raw_rec both 10bit and 12bit. SOme frames are perfect so there is potential. Tried with raw_twk added as well but symbols_engio error.

(https://s16.postimg.org/pcgzgxdk5/Screen_Shot_2016_11_29_at_13_28_51.png)

(https://s16.postimg.org/ribabfh0l/Screen_Shot_2016_11_29_at_13_28_42.png)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on November 29, 2016, 01:48:23 PM
@Oswald: you better use MLVFS and check the DNG's instead of using MLRAWVIEWER. Otherwise you won't have luck.

...10 and also 12 bit neither of these worked (i tried mlv_rec).

Use raw_rec!!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 01:49:14 PM
@Oswald: Is that screenshot from MLrawviewer ?
MLrawviewer doesn't work with 10bit files, at least not the version I have with Canon 6d files.
I use MLV_dump to get dng out of the MLV files and can view them in RawTherapee.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on November 29, 2016, 02:55:38 PM
Anybody now how to setup the path to the /gcc-arm-none-eabi-4_8-2013q4/bin/ directory which is in the root of my user directory on mac osx
It should be as easy as changing the line: $path=""; to simply point at the path of your arm gcc bin folder (any easy way to get that is to drag and drop that folder on a terminal window.

Code: [Select]
$path="~/gcc-arm-none-eabi-4_8-2013q4/bin/";
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on November 29, 2016, 03:05:14 PM
I got it working with MLVFS, sorry about it. I tested again and 12bit and 10bit crop mode litlle bit working, i see few totally corrupted frames and other good ones. I see huge potential with filming with continues 10bit raw with resolution of 2520 x 1052.

http://imgur.com/a/icoiU

Second is 10bit and first one is 12bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 04:07:06 PM
@dmilligan
Thanks!

I was confused by this piece of code in the sample path.
"$ENV{'HOME'}

After the path fix I needed to duplicate 'arm-none-eabi-objcopy' and 'arm-none-eabi-objdump' and renamed them to 'arm-elf-objcopy' and 'arm-elf-objdump'
And now it works!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 04:54:15 PM
First of all, I have no idea what I'm doing  :P ;D

Found this piece of code in the ROM1.BIN.dis file.

ff35a8d8:    ebf7afc0    bl   loc_ff1467e0
ff35a8dc:    e59f10a8    ldr   r1, [pc, #168]   ; ff35a98c: (ff359efc)
ff35a8e0:    e28f00a8    add   r0, pc, #168   ; ff35a990: (725f766c)  *"lv_raw_dump"


So am I right saying function lv_raw_buffer is called with address ff1467e0 ?
But now what to do, because this isn't the memory address I'm looking for ?
Skim the ROM1.BIN.dis file for ff1467e0 ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 29, 2016, 06:49:56 PM
This sounds very exiting! Is it possible to try it on 5dmk3? Special module or ml version required?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 29, 2016, 07:07:16 PM
Special module or ml version required?

You need to compile this branch:

https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit

If you aren't able to compile but want to, check out some of the tutorials located here: http://www.magiclantern.fm/forum/index.php?board=25.0

If you still can't compile but want to test, ask if someone will compile a build for your camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 29, 2016, 08:33:02 PM
You need to compile this branch:

https://bitbucket.org/hudson/magic-lantern/branch/raw_video_10bit_12bit

If you aren't able to compile but want to, check out some of the tutorials located here: http://www.magiclantern.fm/forum/index.php?board=25.0

Definitely worth checking that tut out!  8)

If you still can't compile but want to test, ask if someone will compile a build for your camera.

I went nuts on compiling again this morning (Thanks @dfort for the follow-up re: SourceTree :D) and here the builds for all cameras running ML (except for 1100D.105):

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads

Meanwhile I am still working on to get @nikfreak's branch working on my system in order to spit out with the latest and greatest into the 70D & 100D and as soon as they are up and running definitely will share them.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 08:37:24 PM
Just wondering, is it possible to disassemble a RAM dump from the camera with 'disassemble.pl' ?
Ans if so, at which address does it start, the ROM loads at 0xFF000000, but what value to use for the RAM dump ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 29, 2016, 09:21:02 PM
Thanks! gonna try DeafEyeJedi's Compiled file! :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on November 29, 2016, 09:45:52 PM
Tried the latest firmware build for the 50D... Unable to locate the Bit-Depth option in the menu (either w/ the RAW or MLV module). Also, MLV Sound is listed as a module w/ the 50D and causes an error to occur. (Just thought to try it out and see what happens).

The MLV and .RAW Modules in the menu say they were last updated in August of 2016.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on November 29, 2016, 09:49:13 PM
@DeafEyeJedi
Just tried the build you made for 6d, and noticed it doesn't have 10/12 bit options in mlv_rec and raw_rec.
Just saying(Already have a build for 6d with 10/12bit options)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 29, 2016, 10:15:09 PM
Same problem here for 5d3 123. Can't find settings for bit depth.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 30, 2016, 12:28:13 AM
Tried the latest firmware build for the 50D... Unable to locate the Bit-Depth option in the menu (either w/ the RAW or MLV module). Also, MLV Sound is listed as a module w/ the 50D and causes an error to occur. (Just thought to try it out and see what happens).

The MLV and .RAW Modules in the menu say they were last updated in August of 2016.
@DeafEyeJedi
Just tried the build you made for 6d, and noticed it doesn't have 10/12 bit options in mlv_rec and raw_rec.
Just saying(Already have a build for 6d with 10/12bit options)
Same problem here for 5d3 123. Can't find settings for bit depth.

Thanks guys for noticing this. Dumb mistake on my part (for not double checking builds in beforehand) as I was in a hurry to compile them all before work. ::)

Uploaded new files and this time I checked for 5D3.113, EOSM and 7D which all includes the bit options. Now off I go to figure out how to merge the crop_rec.mo together with this. I'll be sending you an PM soon @dfort!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on November 30, 2016, 03:08:29 AM
Nevermind: (Turned camera On/Off and it's fine... and also... I assumed that I would be able to see the footage in camera and that I would not need to use MLFVS.)

(https://s16.postimg.org/a5zqoyjtt/VRAM1_50_D_San_Disk_UDMA_7_120_MBs.jpg) (https://postimg.org/image/a5zqoyjtt/)

Interesting results with a SanDisk UDMA 7 120MB/s... I'm getting "Matrix" looking images with 10-bit and 12-bit... It's about time I'm seeing the Matrix. Unfortunately, that not something I was hoping to show people. I'd rather just have the power.  8) Secondly, I started getting text on the screen. I turned the camera off and back on and will eventually get text on the screen. The readout does go away when recording, though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 30, 2016, 06:27:02 AM
@LEVISDAVIS -- you sure you didn't accidentally enable on lua.mo under Modules within ML menu?

@100D.101 owners -- as promised that I'd get this one compiled and ready for test (Thanks @nikfreak for sharing your branch) and again PLEASE keep all bug reports on this thread. Enjoy!

100D.101: https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads/ML_10bit_12bit_2016Nov29.100D101.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 30, 2016, 08:00:24 AM
Is it only possible to test this on 5D3 fw113? No fw123 version? If there Isn't I'll guess I'll just downgrade but 123 would be sweet.
Title: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on November 30, 2016, 08:02:00 AM
Afaik 5D3.123 is included @Markus -- my wrong and thanks for catching that @Markus! Just got home from work. Will double check SourceTree and see if I can get it compiled for 5D3.123. Until then.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on November 30, 2016, 09:07:47 AM
5D3.123 is on a separate branch. I don't have a 5D3 to test so there's no guarantee that this will work so use at your own risk:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Nov30.5D3123.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on November 30, 2016, 10:01:25 AM
Didn't get far with fw123 version. Crash when enableing raw rec and/or mlv rec module.  8)

I can share the crash log when I get home from work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: otherman on November 30, 2016, 11:13:54 AM
Is it possible shoot in 10bit with dual-iso, and recreate 14bit in post?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on November 30, 2016, 05:50:26 PM
I tested the 100D (i have b model) build and it is working without any corruption, but live view freezes when starting recording. And also, recording with 12bit, it occanly says raw detect error.

Edit: it says every time raw detect error, when trying to film 12 bit raw. But when you click it again, it starts to record.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 01, 2016, 12:06:19 AM
Trying out 10bit on a 550D using DeafEyeJedi's recent 'I went nuts again this morning' build, 1280 wide 2.35:1 23.976 fps overide and MLVFS installed for the first time today, recording works great, no freezing or odd behavior except top quarter of playback in Resolve via MLVFS every other frame is offset to the right by a consistent 16 pixels, so top quarter of the image jumps from side to side on playback.

50D 10bit every other frame top quarter 'corrupt'. Same portion of the frame as the 550D but 50D shows multicolor, which looks like an incomplete bit plane stack and it appears to be offset to the right by a consistent 16 pixels.

Tested playback with MLVFS and viewed DNG's via MLVFS in Rawtherapee.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 01, 2016, 05:20:14 AM
im also getting "the matrix" levisdavis was talking about on my eos m.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 01, 2016, 05:26:52 AM
@teatotalTED - "recording works great, no freezing or odd behavior" - is that really true? You mean, no LV freezing during recording on 550D = realtime monitoring works?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on December 01, 2016, 07:27:18 AM
Realtime, full colour Liveview monitoring indeed works for me with 10 bit raw on the 550D! Very exciting! Haven't checked the files yet however.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 01, 2016, 07:35:01 AM
So this is the way to go, if we want to fix LiveView for other cameras, am I right? If it works for 550D, it may work for others.
I have some time and some skills. Need some push to the right direction. Please give me some clues, where to dig in, if it make sense.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 01, 2016, 07:40:46 AM
Very exciting indeed, no freezing on 550D or 50D. Getting contnuous 1280 at 2.35:1 10bit on 550D but then big jump to 1600 wide, smaller increments of 16 pixels as discussed earlier on the thread would be good, can't find it in the menus.

So nearly there just need the 16 pixel offset every other frame resolved.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on December 01, 2016, 07:54:35 AM
Quote from: teatotalTED
Very exciting indeed, no freezing on 550D or 50D. Getting contnuous 1280 at 2.35:1 10bit on 550D but then big jump to 1600 wide, smaller increments of 16 pixels as discussed earlier on the thread would be good, can't find it in the menus.

So nearly there just need the 16 pixel offset every other frame resolved.

Which build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on December 01, 2016, 08:09:08 AM
Scroll the wheel for increments in between 1280 and 1600. Works nicely on DeafEyeJedi's latest build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 01, 2016, 09:00:31 AM
Alright guys here we go with Round Two of me going nuts!  :P

This time it includes the crop_rec.mo together with 10/12bit options along with a raw_twk.mo that has been inserted as well for each build.

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads

Special credit goes out to @Danne and @dfort for pushing this forward (Thanks for putting up with my shit this AM) as they were extremely helpful on making this merge possible.

Also I'm still in the process of trying (still a noob myself) to merge this together with 100D/70D from @nikfreak's branch. So will need more time on this meanwhile the previous build from yesterday is still online for now until then. Enjoy!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on December 01, 2016, 09:41:18 AM
Also I'm still in the process of trying (still a noob myself) to merge this together with 100D/70D from @nikfreak's branch.

Sounds great@deafeyejedi
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hammermina on December 01, 2016, 11:59:24 AM
didnt work on 550d :(
got only color pixel mashup,but the recording times are amazing...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on December 01, 2016, 12:06:08 PM
Post-processing tool used?

1. Try ExFAT file system. Format card using cardreader, copy ML files to card and redo installation.
2. Try lower resolutions.
Report back with results.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 01, 2016, 12:17:40 PM
Crashlog from yesterday when I tried to get 10-12bit working on 5d3 123:

ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3


Magic Lantern version : Nightly.2016Nov30.5D3123
Mercurial changeset   : 083982718d31+fd61d21d1429+ (5D3-113_123_10bit_12bit)
Built on 2016-11-30 08:02:13 UTC by rosiefort@RosieFoComputer.
Free Memory  : 165K + 3736K

Not sure if this is helpful?
I'll try out 113.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 01, 2016, 12:18:20 PM
7D users: Please try this build and let me know if it's any better (or worse) than yesterday's build?

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads/crop_rec10bit_12bit_twk.2016Dec01.7D203.zip

Happy December and Thanks to @nikfreak for the tip (http://www.magiclantern.fm/forum/index.php?topic=9848.msg175779#msg175779tip).

*edit*

Thanks guys for tests and my apologies for this bad build. Will get to the bottom of this after some cleaning up.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hammermina on December 01, 2016, 12:20:49 PM
ok now it worked but picture glitches....
i can only preview in 550d on desktop there ist still pixelmashup
i use raw2cdng 1.75
what do you use?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on December 01, 2016, 12:59:59 PM
Alright guys here we go with Round Two of me going nuts!  :P

This time it includes the crop_rec.mo together with 10/12bit options along with a raw_twk.mo that has been inserted as well for each build.

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads

Special credit goes out to @Danne and @dfort for pushing this forward (Thanks for putting up with my shit this AM) as they were extremely helpful on making this merge possible.

Also I'm still in the process of trying (still a noob myself) to merge this together with 100D/70D from @nikfreak's branch. So will need more time on this meanwhile the previous build from yesterday is still online for now until then. Enjoy!


All fine, but x5 zoom crash the camera first time whith in the raw twk activated. If deactivate reboot and activate raw twk again works ok/.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 01, 2016, 02:00:37 PM
http://www.magiclantern.fm/forum/index.php?topic=5601.msg175786#msg175786
The build for 7D above is broken. Don,t use it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 01, 2016, 02:05:31 PM
5D3 113 Results:

Works
12bit & 10bit RAW Full HD
3xcrop with good preview works. Even with 10bit.
5XZoom without rawtwk loaded works
recorded 2.5k successfully but preview bugged out when the recording starts.

Things that dont:
12bit MLV screen tearing in the top part of the frame as MitchLally noticed.
10bit MLV gives colorful corruption on every other frame top third part
5xZoom with raw twk results in crash as Surin Dmitriy noted
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 01, 2016, 02:07:05 PM
@Markus
Any difference in corruption for between raw_rec and mlv_rec?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on December 01, 2016, 02:12:28 PM
Confirmed 5X Zoom causes freeze up, pop battery to rectify. Turned 5x off in Liveview Zoom Tweaks to continue testing.

Tested in mlv_rec only as most of my work requires sound. 5D3 113, tested all 3 bit depths. Global Draw on, mlv_play, raw_twk loaded. 1920x1080 24p. Used deafeyejedi's latest build. Used MLVFS (vert stripe fix on) and Davinci Resolve 12.5 to decode. Exported at h264.

Here is what I have:

694e0a161e
(definitely worth hitting that download button to see the original file)

As you can see, 14bit is normal, 12bit looks normal, 10bit every second(ish) frame is corrupted in top third. Plays back in mlv_play this way too.

Upon further inspection – testing 12bit whilst moving the camera I encountered this (keep an eye on the top portion of the frame):

9a8cd67e0b

Also it seems that it isn't decoding all the frames...  Same number of frames in the folder as 10/14 bit... Could this be a resolve issue? Or MLVFS?

Awesome progress so far, thank you to everyone working on this. I am suuuper keen to see at least 12bit working in mlv_rec! That would be a dream come true.

Here's some size comparisons:

https://s15.postimg.org/71b8gur6j/Screen_Shot_2016_12_01_at_10_23_34_pm.jpg
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 01, 2016, 02:24:20 PM
Raw_rec works without glitches 10bit/12bit?
Is raw_twk causing the 5x zoom to freeze?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 01, 2016, 02:39:57 PM
On 550D rawrec 10bit produces multicolor corruption to top 1/3rd of every other frame. mlvrec 10bit produces 16pixel offset to right every other frame but image intact. Playback in camera totally corrupt.

(https://s14.postimg.org/v97tf4zcx/Good_1_1_1.gif)


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on December 01, 2016, 02:53:42 PM
Yes it seems to be caused by raw_twk. Can zoom 5X fine without this module loaded.

Raw_rec works without glitches 10bit/12bit.

Here's raw_rec at 10 and 12 respectively:
e7db2974ed

Exactly same workflow as mentioned in last post.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 01, 2016, 03:11:41 PM
Quote
Raw_rec works without glitches 10bit/12bit?
Is raw_twk causing the 5x zoom to freeze?

@Danne
Yes rawrec 10-12bit works as far as I can see.
Yes raw_twk causes freeze.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 01, 2016, 06:28:27 PM
Doing some more testing with 5D2 on 10-12bit raw with updated resolutions. [...] Anything with 1880 in all 3 bit ranges are totally corrupted and crashes the recording session , ( early stop etc..) but not the camera .

1880 works just fine here on 5D2 (just some minor UI quirks when fine-tuning the resolution). Playback tested on classic mlv_play (14-bit only) and checked the first DNG with mlv_dump (all bit depths).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on December 01, 2016, 06:47:25 PM
Canon 50D: 11/29 & 11/30 Firmware: Getting the same 10-bit and 12-bit results as the 550D (@ teatotalTED), it's the bottom portion of the screen that repeats. FYI: The image splits in the middle of the frame. However, one time it split at the top 2/3rds. The split image repeats every other frame. :o

Meanwhile: 14-bit is as solid as ever. (** First time using MLVFS and the image is really something worth broadcasting! **)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on December 01, 2016, 08:23:21 PM
I tried the Deafeyejedi:s 7d build and it worked worse than previous one:
1) when trying to film with raw_lite, it just says data corruption at slot 21, 22, 23 or something like that.
2) crop mode freezes whole live view, but not the camera
3) i think that dng:s are more corrupt than before.

Also i tested the your 100d build and it has same problem than before, so, when recording start, it freezes the live view.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 01, 2016, 08:54:03 PM
1880 works just fine here on 5D2 (just some minor UI quirks when fine-tuning the resolution). Playback tested on classic mlv_play (14-bit only) and checked the first DNG with mlv_dump (all bit depths).
Ok , I'll recompile the build and try again maybe I messed up some where or missed something .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Steve_FR on December 01, 2016, 08:59:13 PM
Tested Eos M running Crop_rec nov30th build from DeafEyeJedi.

Recording Raw 10 bit @ 1728x692 (2.50:1) using movie crop mode works smoothly, with live view display working great during raw REC.
Playback of the last 10bit raw clip using mlv_play and raw_twk enabled works great, but there are no controls to start, stop, track back or track forward. I can interrupt the raw playback with a half-shutter press, and return to normal liveview. I can play the clip as many times as I want if I keep interrupting it before it finishes, but if I allow the clip to play until the last frame, the camera appears to hang on that frame. A half shutter returns to liveview. Afterward, raw playback soft hangs on a black screen (half shutter returns to liveView). The only way for me to see the raw playback again is to power cycle the camera. The camera is not crashed or frozen, though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 02, 2016, 12:08:58 AM
Ok share my report.

Tested on 700D, after selecting 10bit it doesn't allow to select 12 bit, it shows in the dropdown list but non way to select it.

Anyway i think we MUST find a unique way for testing as i can see that many corrupted frames take place only when converting from .mlv to the dng sequence.

So please when talking about corrupted frames also say what decodifier you're using and (eventually) the options with it.



Here is my very short test:
Camera: 700D
Decodified with: MLVFS (running as service)
OS: Windows
BPP: 10 bit
Resolution: 1736x784
Memory card: Scheda SanDisk Extreme 32 GB SDHC UHS-I - 45 MB/S
raw_twk: no
Corrupted frames: none
Build: 30th november from DeafEyeJedi
Notes:
got liveview working, quite slow to be honest but fast enough for now ;)


I'll do further testing on saturday morning (yeah sunlight shots :D )

I'd like to know what's the repo we should consider if we want to help (so check last versions and so on), is the hudson's official bitbucket repo or should i browse many other repos?


If you need help (also programming, got some skill in C) just ask
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 12:33:50 AM
Tested on 700D, after selecting 10bit it doesn't allow to select 12 bit, it shows in the dropdown list but non way to select it.

I made new test builds and selecting 12 bit is working fine on the 700D. Haven't done extensive tests yet but it seems that mlv_rec has the corrupt frames issue while raw_rec (MLV Lite) is ok.

Lots of exciting development going on. A few of us have been merging 10bit_12bit with crop_rec and raw_twk and posting test builds. Here's my latest--all platforms including 5D3.123 but excluding 1100D, 70D and 100D. I'll keep it up for a few days or when there are new commits to the development branches.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.zip

Note that crop_rec only works with 5D3 and EOSM and some combinations of modules aren't working. Lots of testing opportunities but please make sure you report where you got the build, date, if you are using mlv_rec or raw_rec and if you find an issue report it in the appropriate forum topic.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 02, 2016, 04:32:40 AM
selecting 12 bit is working fine on the 700D
So LV display during recording works on 700D too? No freezing?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 05:29:01 AM
So LV display during recording works on 700D too? No freezing?

No--during recording LiveView still freezes. This was also the case with the EOSM but it was resolved so there's still hope for the other platforms. I believe that it was the 
EOSM: enable CONFIG_EDMAC_RAW_SLURP (https://bitbucket.org/hudson/magic-lantern/commits/193dc5989d6f8807804ffb1ede2597967ef33f1d?at=raw_video_10bit_12bit) commit by dmilligan that did the trick.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 02, 2016, 07:17:17 AM
I can confirm on 5D2 1880 does work in all bit ranges , of course it's only stable/uncorrupted in 14bit
10 , 12 have issue of course but liveview never froze up . I didn't have time today to recompile
the 10_12bit expermential branch so I tested out dfort build as it seem to be update .
Set liveview to Auto , FYI . I made 3 short 20 second test videos (h264) from 10 ,12 ,14 bit
in my Dropbox  https://www.dropbox.com/sh/lh5lxr74moyrnj5/AAAFDzHGYxoafVLWkX71R5R0a?dl=0
you can there download (50MB each) or watch them in dropbox
Beware ! 10 & 12 bit may made you sea sick  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 08:58:20 AM
@reddeercity

Are you using mlv_rec or raw_rec (MLV Lite)? I've gotten better results on the EOSM from raw_rec.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 02, 2016, 11:02:55 AM
Did some changes to raw_rec (mlv lite), not really related to bit depth, but since this became the "de facto" bleeding edge branch for raw recording, I've merged them. The functional ones are:

- fix for this issue (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174390#msg174390) (may affect recording times if - and only if - global draw is on).

- a change borrowed from mlv_rec (async edmac calls) that might help with general frame corruption issues (that already happen in unified in some edge cases); it will not help with frame corruptions related by bit depth changes.

Currently benchmarking these changes; will post results here (http://www.magiclantern.fm/forum/index.php?topic=17091.msg175088#msg175088).

For new builds, I recommend including only those cameras that can record without corrupted frames, at least at some settings. For the cameras that show corruption every other frame, the behavior will be unchanged. I know the cause and already suggested possible fixes  a month ago (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181;topicseen#msg174181). Until somebody implements one of these fixes (hint: see how it was done on EOS M), reports of image corruption (repeated for every new build), for those models that are already known not to work, are simply not useful.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Peter Linov on December 02, 2016, 11:35:27 AM
5DMIII 1.2.3 
CRASH 00-
ASSERT: hLvJob->hJpegMemSuiteat ./Epp/Vram/VramStage.c:891, task Epplv:1 mode:3
Magic Lantern version : Nightly.2016Dec01.5D3123 Mercurial changeset   
: 385bc44ed158+b3dfbe7194f3+ (5D3-113_123_10bit_12bit) tip
Built on 2016-12-01 23:09:48 UTC by rosiefort@RosieFoComputer.
Free Memory  : 146K + 3712K
CRASH01-
ASSERT: IsSuiteSignature( hSuite ) at ./PackMemory/PackMem.c:599, task Epp
lv:0 mode:3Magic Lantern version : Nightly.2016Dec01.5D3123
Mercurial changeset   : 385bc44ed158+b3dfbe7194f3+ (5D3-113_123_10bit_12bit) tip
Built on 2016-12-01 23:09:48 UTC by rosiefort@RosieFoComputer.
Free Memory  : 146K + 3728K
CRASH02-
ASSERT: hLvJob->hJpegMemSuiteat ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3 Magic Lantern version : Nightly.2016Dec01.5D3123
Mercurial changeset   : 385bc44ed158+b3dfbe7194f3+ (5D3-113_123_10bit_12bit) tip
Built on 2016-12-01 23:09:48 UTC by rosiefort@RosieFoComputer.
Free Memory  : 153K + 3719K


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 04:10:40 PM
@Peter_Linov

Didn't think it was going to work but it was worth a shot. What were the conditions that caused the crash?

Looks like for now we should limit the test builds to the 5D3.113 and EOSM.202 and make sure we're in raw_rec instead of mlv_rec. These test builds don't have the mlv_rec and mlv_snd modules to avoid confusion.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D3113.zip
https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.EOSM202.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 02, 2016, 06:35:17 PM
The 550D is also very close, 12bit works fine, 10bit image intact just 16 pixel offset to right every other frame, (mlvrec not rawrec which looks to be a trainrec on the 550D)

A1ex suggests above that there's a fix for this from a month ago or should this be brought up on the 550D camera specific thread instead of here where priorities seem weighted elsewhere, I guess because of ownership of EOS-M & 5D3? There's many more 550D ML users than EOS-M and for good reason?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 02, 2016, 06:42:04 PM
... There's many more 550D ML users than EOS-M and for good reason?

I'd take an EOSM over 550D on any given day and no pun intended.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 07:26:00 PM
The 550D is also very close...

I didn't know that. Here's a test build for the 550D without mlv_rec and mlv_snd like the other test builds I recently posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.550D109.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hammermina on December 02, 2016, 08:58:35 PM
@dfort

550d build works :)12 bit all resolutions :) :)
10 bit frames are messed up...

go for it :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 02, 2016, 09:08:26 PM
12bit worked great via mlvrec in dforts build from before, so 12bit now works in rawrec to.

10bit in rawrec is vertically jumping up and down 16 pixels every other frame rather than moving to the right 16 pixels every other frame as with the mlvrec module from dforts earlier build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 02, 2016, 10:29:12 PM
@teatotalTED

Actually it is the same build. What you are reporting is the difference between mlv_rec and raw_rec (MLV Lite).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 03, 2016, 04:08:42 AM
@reddeercity
Are you using mlv_rec or raw_rec (MLV Lite)? I've gotten better results on the EOSM from raw_rec.
raw_rec (mlv lite) actually it's getting better on 5D2  , 10bit used to be totally pink noise & 12bit ever other frame was totally pink noise.
But ever once in a while I have perfect recording + clean liveview mostly in Crop mode . I would say crop mode on 5d2 is more stable right now .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 03, 2016, 07:20:48 AM
The same build? As yours from yesterday posted at 12.08:58? How can it be the latest doesn't have mlvrec and mlvsnd in it? Might be same code base but a new build? 12bit on 550D worked fine on yesterdays build of 12.08:58 via mlvrec, all I was saying was that 12bit working on 550D was nothing new to the latest rawrec only build. And finally yes I was comparing 12.08:58 build 10bit 550D via mlvrec 16 pixel shift horizontal with latest rawrec only build 10bit 16 pixel shift vertical, both very close to solution for 550D, so to first say only concentrate on EOS-M and 5D3, then say only concentrate on rawrec, with regard to 550D either which way makes little difference its so close. Does the latest rawrec build include possible fix highlighted by A1ex in post #627, is that what we're supposed to be testing here in the latest build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 03, 2016, 10:10:02 AM
That's a race condition working in your favor :)

The process is very sensitive to timing, and will be so until the real issue is addressed. The DMA channel is configured for 14-bit image size, we only copy 12-bit (or 10-bit) data from it, and what happens with the excess data... Canon knows. That's the cause of the artifacts you describe. It is a random process, as the LiveView timings are not very exact (I think the image capture is done at fixed intervals, but the processing timing is affected by the task scheduler, to some extent).

So, just because it works for you in some (lucky) conditions, that doesn't mean it's getting any closer.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 03, 2016, 10:55:11 AM
Yeah anticipated the response to presumption of 'so close', in reality I'm well aware it ain't that simple, just would like to see 550D in the running, 10bit on cameras with restricted write speeds is a good priority to many users, for me my 50D is go to and barely use the 550D. Lucky settings, there's no blackmagic, pun intended, in the combination of settings, just chose mlvrec 10bit and it either works or doesn't, don't consider myself a tester just have bit of time at the moment to mess.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kyrobb on December 04, 2016, 05:44:28 AM
Tried out dfort's latest build on the 550D. In 10 bit every second frame is perfect, but in the others only the top 3rd is correct. The bottom 2/3rds are just a repeat of the 1st frame I believe. Liveview still works great and the record times are awesome!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zcream on December 04, 2016, 11:10:15 AM
From Post #22
>>
Reading this: http://francoismalan.com/2011/10/raw-12bit-or-14bit-lossy-or-lossless/
... i guess that it would only make a very very slight difference!

If this is really working... I just dont have words for how awesome this is...
>>

Here is the other argument..
>>
There is a difference between underexposed sunny scene and shooting in pitch dark interiors or at night.

I did the tests myself and 14 bit has huge advantage under extreme dark conditions.

Compression concerns the highlights only and I assume it works the same way for the high-key scenes – white on white.

It’s true, in every-day outdoors shooting, 14bit/uncompressed doesn’t really matter.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 04, 2016, 07:12:36 PM
New test builds:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip
https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec04.EOSM202.zip

Seems like we may be adding too much into the recent test builds like crop_rec and raw_twk and reporting the results in this topic. How about we test just a few things at a time like raw_rec and mlv_rec and make sure to indicate which one you are using in your reports.

Also, cameras other than the 5D3.113 and EOSM.202 should be updated before making more test builds for those platforms:

For new builds, I recommend including only those cameras that can record without corrupted frames, at least at some settings. For the cameras that show corruption every other frame, the behavior will be unchanged. I know the cause and already suggested possible fixes  a month ago (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181;topicseen#msg174181). Until somebody implements one of these fixes (hint: see how it was done on EOS M), reports of image corruption (repeated for every new build), for those models that are already known not to work, are simply not useful.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 04, 2016, 07:39:02 PM
Did some more research and it seems that the cameras that have the fewest issues are the ones that have CONFIG_EDMAC_RAW_SLURP. Looks like dmilligan got it working on the EOSM (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175184#msg175184).

According to the features list (https://builds.magiclantern.fm/features.html) the 60D and 600D have this feature. Anyone out there want raw_video_10bit_12bit builds of those platforms to test?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 04, 2016, 07:53:11 PM
I've recorded in 10bit, many times, on the 6d and there are no corrupted frames, both in raw_rec and mlv_rec modules.
Only problem is that liveview freezes (no config_edmac_raw_slurp).
I'm stuck with figuring out how to find the address for the 6d to configure config_edmac_raw_slurp(see few post back in this topic).
http://www.magiclantern.fm/forum/index.php?topic=5601.msg175682#msg175682 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175682#msg175682)

But I can assure, that there are no corrupted frames on the 6d with 10bit recording.
Although 12bit has sometimes the purple banding issues on every frame.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 04, 2016, 08:59:14 PM
Only problem is that liveview freezes (no config_edmac_raw_slurp).

disabled raw_twk module?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 04, 2016, 09:23:21 PM
With and without raw_twk enabled, live view freezes during 10bit recording on 6d.
But as far as I know, you need to configure config_raw_edmac_slurp to fix this.
At the moment this only done for 4 cams, 5d3, eosm, 60d and 600d

With 14 bit, live view works like it should be.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 04, 2016, 10:36:05 PM
@Levas - I'm trying to do the same with the 700D and seem to be stuck in the same spot you were in a week ago (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175569#msg175569).

I ran find_free_edmac_channels and found that channels 0 and 8 "seems to work" but I have no idea where to find the DEFAULT_RAW_BUFFER MEM address. I've got an EOSM so I'm looking a disassembly of the ROM1.BIN. Found "lv_raw_dump" but don't understand how dmilligan got MEM(0x404E4 + 0x44) (https://bitbucket.org/hudson/magic-lantern/commits/193dc5989d6f8807804ffb1ede2597967ef33f1d?at=unified) out of that.


Code: [Select]
diff --git a/platform/700D.114/internals.h b/platform/700D.114/internals.h
--- a/platform/700D.114/internals.h
+++ b/platform/700D.114/internals.h
@@ -139,3 +139,7 @@
 
 /** Workaround for menu timeout in LiveView */
 #define CONFIG_MENU_TIMEOUT_FIX
+
+/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
+#define CONFIG_EDMAC_RAW_SLURP
+
diff --git a/src/edmac-memcpy.c b/src/edmac-memcpy.c
--- a/src/edmac-memcpy.c
+++ b/src/edmac-memcpy.c
@@ -396,6 +396,11 @@
 uint32_t raw_write_chan = 4;
 #endif
 
+#ifdef CONFIG_700D
+// channels 0 and 8 "seems to work"
+uint32_t raw_write_chan = 8;
+#endif
+
 #ifdef CONFIG_EOSM
 uint32_t raw_write_chan = 0x12;
 #endif
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -118,6 +118,11 @@
 #define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
 #endif
 
+#ifdef CONFIG_700D
+// copied from EOSM -- need to find 700D address
+#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
+#endif
+
 #else
 
 /* with Canon lv_save_raw, just read it from EDMAC */

[EDIT] Seems like this was tried before on the 6D:

http://www.magiclantern.fm/forum/index.php?topic=13155.msg127251#msg127251
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on December 04, 2016, 11:09:53 PM
At the beginning of lv_raw_dump on the M there is a call made to a function in ram (branch instruction). On 60D this function call is to ROM not RAM. We can use RAM_OFFSET to find the location of that function in rom (Canon firmware copies some functions to RAM and runs them there rather than running them directly from ROM).
Code: [Select]
#define RAM_OFFSET 0xFFA69590

This leads to this function in ROM:
Code: [Select]
FFA7A184: ldr r0, [pc, #808] ; 0xffa7a4b4: pointer to 0x404e4
FFA7A188: ldr r0, [r0, #68] ; 0x44
FFA7A18C: bx lr

This code loads 0x404e4 into register 0, then adds 0x44 to register 0, then it returns (register 0 is the return value). In C:
Code: [Select]
return 0x404e4 + 0x44;

On 60D:
Code: [Select]
FF0F8B8C: ldr r0, [pc, #732] ; 0xff0f8e70: pointer to 0x4ff8 FF25B020 FF25B360 FF25B3A4 FF25AF2C FF25B31C
FF0F8B90: ldr r0, [r0, #48] ; 0x30
FF0F8B94: ldr r0, [r0]
FF0F8B98: bx lr
Which is equivalent to:
Code: [Select]
return *(0x4ff8 + 0x30);

Then lv_raw_dump uses this return value as a pointer to the location of the raw buffer. The ML macro MEM() is simply a pointer dereference. So Canon firmware always keeps a pointer to the location of the raw buffer at 0x404e4 + 0x44 on the EOS M. The actual raw buffer may be dynamically allocated, but it's current address is always kept at memory address 0x404e4 + 0x44 (yes, this is a pointer to a pointer, on 60D, it's actually a pointer to a pointer to a pointer). I guess these buffer addresses are probably kept in static arrays or structs and the + 0x44 part is the array index or struct offset. If it is indeed a data structure, it might be interesting to look at and around 0x404e4 and see what else might be kept there.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: capncannabis on December 05, 2016, 05:17:58 AM
Is there a build to test this on the 600d?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 05, 2016, 05:30:46 AM
At the beginning of lv_raw_dump on the M there is a call made to a function in ram (branch instruction)...We can use RAM_OFFSET to find the location of that function in rom...
Code: [Select]
#define RAM_OFFSET 0xFFA69590

This leads to this function in ROM:
Code: [Select]
FFA7A184: ldr r0, [pc, #808] ; 0xffa7a4b4: pointer to 0x404e4
FFA7A188: ldr r0, [r0, #68] ; 0x44
FFA7A18C: bx lr

Let's see if I understand. This means that the function call should be specifying an address of 0x10BF4 (FFA7A184 - RAM_OFFSET) but there is no such address in the the EOSM disassembly that I've got. There was enough of a hint in that function in the EOSM disassembly for me to look up that same function in the 700D and I came up with 0x25B0C + 0x3C. I'm pretty confident about that. However, the RAM_OFFSET on the 700D is written in a different format from the EOSM:

stubs.S
Code: [Select]
#define RAM_OFFSET (0xFFA5E8B0-0x1900) // Nanomad: some functions are copied to RAM at around ff0c0098; they have to be called from RAM...

Oh boy. I would assume that the RAM_OFFSET is 0xFFA5CFB0 but there must be a reason it is written the way it is.

Running find_free_edmac_channels seemed pretty straight forward but --

edmac-memcpy.c
Code: [Select]
#ifdef CONFIG_700D
// channels 0 and 8 "seems to work" with find_free_edmac_channels
// but these values don't work. Still searching.
uint32_t raw_write_chan = 20;
#endif

I tried every channel from 0-20 and found several that seemed to work but the MLV file was just a bunch of noise. The EOSM is written in hex (0x12) while the other platforms are in decimal values so maybe theres a reason for that too?

Sorry to be a PITA but it seems that config_edmac_raw_slurp on the 700D is possible if we can find the right numbers to plug in. Here's my latest work in progress:

Code: [Select]
diff --git a/platform/700D.114/internals.h b/platform/700D.114/internals.h
--- a/platform/700D.114/internals.h
+++ b/platform/700D.114/internals.h
@@ -139,3 +139,7 @@
 
 /** Workaround for menu timeout in LiveView */
 #define CONFIG_MENU_TIMEOUT_FIX
+
+/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
+#define CONFIG_EDMAC_RAW_SLURP
+
diff --git a/src/edmac-memcpy.c b/src/edmac-memcpy.c
--- a/src/edmac-memcpy.c
+++ b/src/edmac-memcpy.c
@@ -396,6 +396,12 @@
 uint32_t raw_write_chan = 4;
 #endif
 
+#ifdef CONFIG_700D
+// channels 0 and 8 "seems to work" with find_free_edmac_channels
+// but these values don't work. Still searching.
+uint32_t raw_write_chan = 20;
+#endif
+
 #ifdef CONFIG_EOSM
 uint32_t raw_write_chan = 0x12;
 #endif
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -118,6 +118,10 @@
 #define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
 #endif
 
+#ifdef CONFIG_700D
+#define DEFAULT_RAW_BUFFER MEM(0x25B0C + 0x3C)
+#endif
+
 #else
 
 /* with Canon lv_save_raw, just read it from EDMAC */
@@ -447,6 +451,12 @@
     return 1;
     #endif
 
+    #ifdef CONFIG_700D
+    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
+    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
+    return 1;
+    #endif
+
     /* unknown camera? */
     return 0;
 

One last question about this:

raw.c
Code: [Select]
    #ifdef CONFIG_EOSM
    *width  = video_mode_crop ? 1872 : 1808;
    *height = video_mode_crop ? 1060 : 727;
    return 1;
    #endif

Is video_mode_crop the same as mv1080crop or is there a reason why it is coded this way for the EOSM and would this also apply to the 700D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on December 05, 2016, 05:22:36 PM
Well, as you know, the EOSM only has two video modes we can actually use for raw video (crop hack and mv720 - LV "standby" or whatever you want to call it), so I didn't bother looking up the buffer sizes for the other ones that only start when recording h.264. For 700D you will probably need all of them.

I believe there is method in crop_rec that avoids the hardcoded buffer sizes (have not looked at that code)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 05, 2016, 08:07:42 PM
working to find addresses for 700D too.

@dfort can u pls update your repo so we'd be able to work on the same files? ;)

had to work in the past few days so didn't had the time to write even a line :(

now disassembling rom and ram
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 05, 2016, 08:18:48 PM
@eNnvi -- commited what I've got so far:

https://bitbucket.org/daniel_fort/magic-lantern/commits/e0463878deec23ab856d05879def00a78e6d0d6d
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 05, 2016, 08:43:16 PM
Oh boy. I would assume that the RAM_OFFSET is 0xFFA5CFB0 but there must be a reason it is written the way it is.

this way you know two important informations.
where the RAM address is and from where it gets copied.

writing 0xFFA5E8B0 would throw away that information.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 05, 2016, 11:38:59 PM
Well, as you know, the EOSM only has two video modes we can actually use for raw video...

There is a third one -- zoom. The magnifying lens icon is on one of the touch screens so it isn't very convenient to get into that mode but it has a 2592x1108 full raw buffer and allows shooting up to 2496x1074 images. Of course 24fps is out of the question but it should be usable at lower frame rates especially with the lower bit depths.

The only video mode missing on the EOSM is mv1080 or at least that buffer seems to be out of reach for now. All of the video modes except for the "Movie crop mode" (a.k.a. mv1080crop or video_mode_crop) default to 29.97fps (even when set to PAL) so this is a quirky little camera to say the least.

At some point perhaps someone will find the key to unlocking mv1080 on the EOSM. Also, lower bit depth will eventually be working on other platforms so shouldn't this work?

Code: [Select]
#if defined(CONFIG_100D) || defined(CONFIG_650D) || defined(CONFIG_700D) || defined(CONFIG_EOSM)
    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
    return 1;
#endif

These are the cameras that exhibit the focus pixel issue. They all have identical full raw buffer sizes. I also made a new discovery while searching for video streams that don't show focus pixels but I'll save that post for the focus pixel topic (http://www.magiclantern.fm/forum/index.php?topic=16054.new#new).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 06, 2016, 05:07:19 AM
@dfort anything new on the dec4th build for eosm? I'm using mlv lite 1.1 raw record module. its working well in 10 bit raw. there's been a bunch of builds and i feel like I'm missing something new. haha
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 06, 2016, 06:13:25 AM
You can check up on new commits to this branch with this link:

https://bitbucket.org/hudson/magic-lantern/commits/branch/raw_video_10bit_12bit

I don't want to merge too many branches together until this is well tested. How about trying mlv_rec with mlv_snd in 10bit?

I'm also testing the latest cr2hdr.app that uses MLVFS and can't get the mlv_snd working with mlv_rec. It should work because the WAV file sounds fine. Also, applying FPS override to 23.98 causes corruption at the top 1/3rd of every other frame with mlv_rec but it is fine with raw_rec (MLV Lite).

On the EOSM using the Dec. 4 build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 07, 2016, 04:05:36 AM
Hi guys,  its possible to compile one for the 7D,  want to make some testing

Thanks  :) 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 07, 2016, 07:55:42 AM
Hi guys,  its possible to compile one for the 7D...

It is possible but is there anything new to report on the 7D? Correct me if I'm wrong but only the 5D3.113 and EOSM.202 are doing the EDMAC_RAW_SLURPING thing necessary for this to work properly.

working to find addresses for 700D too.

Any luck yet?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on December 07, 2016, 08:20:55 AM
For future reference... Just wanted to jump back in and say that the 10-bit + 12-bit RAW recordings from the Nov 29 & Nov 30th build created magenta frames during 5X Crop mode for the Canon 50D. This includes the 14-bit RAW as well.

Jumping out...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 07, 2016, 08:21:12 AM
So I'm wondering if this is a bug or something but doing some testing with 3xCrop mode on 5D2 with build from dfort (https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.zip) Dec./1/2016 . Just the basic modules for raw video.
With raw_rec (mlv lite) in fine resolutions 10 & 14bit  2152x1072 is the max. selectable resolution, while in 12bit I can expand the size to 2152x1076 but not 10 or 14bit.
Should it not but the same thought out the bit range ? Too bad the image are dark pink not very easy to correct like bad black levels , Good thing thou is in 12bit with Full MLV+audio in
3X crop mode I have perfect corruption Free recording  :D with correct black levels in All Liveview modes , but only the canon preview was glitch free no Lv freeze or B/W lines mess .
Thou framing is off it doesn't freeze up very useable and with audio too . I didn't try 10 bit in 3x Crop to see if it's a problem will try later , so it seems 3x crop recording to MLV 2.0(full)
 works in 12bit  corruption free (useable). 8)

Edit: Sorry the size in 3xcrop MLV 2.0(full) was 2144x1076 @23.976p
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 07, 2016, 12:15:36 PM

Thanks dfort. I thought that exist, even with bad frames, a version for the 7D  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 07, 2016, 01:16:11 PM
There's one in the 7D thread posted by me.

Edit: http://www.magiclantern.fm/forum/index.php?topic=9848.msg175503#msg175503

That one is confirmed to work
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 07, 2016, 02:08:34 PM

Thanks nikfreak  8)    Will test it and give some feedback.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: robertgl on December 07, 2016, 04:33:29 PM
@660 re:50D

What resolutions & frame speeds did you shoot at? You did not get repeating frames or torn frames?  thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on December 07, 2016, 08:26:09 PM
@reddeercity
10 & 14bit  2152x1072 is the max. selectable resolution, while in 12bit I can expand the size to 2152x1076 but not 10 or 14bit.
Should it not but the same thought out the bit range ?
(If I understand it correctly): The whole image has to be a multiple of 16 bytes, and 1076x2152 in 14 bit is not, so it reduces vertical resolution. In new 1880, the max vertical resolution is 1248 instead of 1250, highest resolution you can get is 1872x1250 in 12 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 08, 2016, 01:15:17 AM
ok i'm stuck on a (probably) stupid thing: find the right address for raw_slurp

i disassembled my 700D .bin file, i found the lv_raw_dump function in rom but i don't get how to get the right address from here.

I mean..
Code: [Select]
ff36c804: 725f766c subsvc r7, pc, #108, 12 ; ff36c7a0: (72615020)
ff36c808: 645f7761 ldrbvs r7, [pc], #-1889 ; ff36c810 <_binary_ROM1_BIN_start+0x36c810>
ff36c80c: 00706d75 rsbseq r6, r0, r5, ror sp
ff36c810: ff36beb0 ; <UNDEFINED> instruction: 0xff36beb0

following http://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969 dmilligan says there should be a branch instruction but i don't get where it actually is. I mean the subsvc should mean something like
Code: [Select]
r7 = pc - (108<<12) right?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 08, 2016, 01:19:56 AM
this is no valid code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 08, 2016, 04:39:39 AM
@reddeercity(If I understand it correctly): The whole image has to be a multiple of 16 bytes, and 1076x2152 in 14 bit is not, so it reduces vertical resolution. In new 1880, the max vertical resolution is 1248 instead of 1250, highest resolution you can get is 1872x1250 in 12 bit.
No you misunderstood , I'm talking about 3x crop mode not full frame with the new 10-12 raw_rec (mlv lite) with a1ex update fine resolution's , in 10 bit & 14 bit in 3xCrop mode 1072 is maximum vertical and 12bit can be expanded to 1076 on the same build . It shouldn't matter if it's 10 , 12 or 14 bit the  vertical resolution in 3x crop should not change & yes I understand what happen with the code . If you check out the full MLV 2.0+audio in 3x crop mode the maximum vertical resolution is 1076 and of course in 12bit (mlv2.0) @ 2144x1076 all frames are clean with Canon Liveview .

Edit: On MLV 2.0 with bit range reduction (10-12bit) update in 3x Crop the maximum vertical resolution stays the same thou out all 3 bit ranges of 1076
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 08, 2016, 05:03:54 AM
5D2 , To just confirm MLV 2.0 10bit in 3X crop @ 2144x1076 23.976p + audio are free of any corruptions/shearing etc... .  with Canon Lv.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 08, 2016, 09:29:19 AM
So I been trying to correct the pink/magenta 3x Crop on 5d2 with raw_rec(mlv lite) with reduced bit range (working with 12bit)
well exiftool tell me the white & black level are totally out to lunch compared to mlv 2.0 in crop mode
I end up with black level of 63 & white level of 937 on the magenta frame and on the non magenta frame (full mlv2.0) black level 448 & white level 3750.
Not sure why mlv lite uses these level compared to mlv2.0 which are correct , anyway to fix the magenta frames  I used Exiftool to set the white & black level
to what my eyes see as normal WB image. So after some time of trial & error I set the black level to 110 and the white level to 730 ,
it doesn't seem right to have just a little differences between the
black & white levels as compared to the other images levels below is the magenta frame and then the corrected frame with exiftool.

  (https://c5.staticflickr.com/1/255/30660487884_d2cec3c4d2_n.jpg) (https://flic.kr/p/NHn1Rs)
12bit_3Xcrop magenta frame  (https://flic.kr/p/NHn1Rs) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://c1.staticflickr.com/1/679/30660487824_b50b83cb01_n.jpg) (https://flic.kr/p/NHn1Qq)
12bit_3Xcrop_corrected_levels (https://flic.kr/p/NHn1Qq) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

syntax for windows in command prompt
exiftool -IFD0:BlackLevel=110 *.dng
exiftool -IFD0:whiteLevel=730 *.dng
If anyone what's to check the original magenta Cdng frame 2144x1076  link below
https://www.dropbox.com/s/91a1blyb2qix31f/original_M08-0008_000004.dng?dl=0


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rbrune on December 08, 2016, 11:06:00 AM
No you misunderstood , I'm talking about 3x crop mode not full frame with the new 10-12 raw_rec (mlv lite) with a1ex update fine resolution's , in 10 bit & 14 bit in 3xCrop mode 1072 is maximum vertical and 12bit can be expanded to 1076 on the same build . It shouldn't matter if it's 10 , 12 or 14 bit the  vertical resolution in 3x crop should not change & yes I understand what happen with the code . If you check out the full MLV 2.0+audio in 3x crop mode the maximum vertical resolution is 1076 and of course in 12bit (mlv2.0) @ 2144x1076 all frames are clean with Canon Liveview .

Edit: On MLV 2.0 with bit range reduction (10-12bit) update in 3x Crop the maximum vertical resolution stays the same thou out all 3 bit ranges of 1076

No, he is right, the bit depth matters. You can check with some simple math:

Code: [Select]
((2152*1072*10bit)/8bit)/16bytes = 180230
((2152*1072*12bit)/8bit)/16bytes = 216276
((2152*1072*14bit)/8bit)/16bytes = 252322

((2152*1076*10bit)/8bit)/16bytes = 180902.5
((2152*1076*12bit)/8bit)/16bytes = 217083
((2152*1076*14bit)/8bit)/16bytes = 253263.5
Only modes that give an integer value are valid and can be used to record with. So yes you can record with 12bit @ 2152x1076 but not with 10/14bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 08, 2016, 03:16:56 PM
Tried to read me to understanding this build but I can't manage to record without getting pink pixelated images. Live view is only stripes of black and white. Does it work on exfat? I have only tried in crop mode. Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 08, 2016, 10:47:17 PM
No, he is right, the bit depth matters. You can check with some simple math:

Code: [Select]
((2152*1072*10bit)/8bit)/16bytes = 180230
((2152*1072*12bit)/8bit)/16bytes = 216276
((2152*1072*14bit)/8bit)/16bytes = 252322

((2152*1076*10bit)/8bit)/16bytes = 180902.5
((2152*1076*12bit)/8bit)/16bytes = 217083
((2152*1076*14bit)/8bit)/16bytes = 253263.5
Only modes that give an integer value are valid and can be used to record with. So yes you can record with 12bit @ 2152x1076 but not with 10/14bit.

Thanks for the math , so then as per example when I'm using mlv 2.0 @ 12bit 2144x1076
Code: [Select]
((2144*1076*12bit)/8bit)/16bytes = 216276If I did my math right "216276" is a good  integer value , so valve that fall in to the ".5" etc... don't work right ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 09, 2016, 01:02:09 AM
Where Can I fine the file to test out, and is this the best build so far?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 09, 2016, 07:37:24 AM
Where Can I fine the file to test out, and is this the best build so far?
Click me (https://www.dropbox.com/s/oodspk5jits334u/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip?dl=0)  :)
this was compiled by dfort , if you what to be able to compile yourself read my post here (http://www.magiclantern.fm/forum/index.php?topic=18259.msg175475#msg175475) very easy to setup in less then 30min. and start compiling , there a short video tutorial to help explain .
I assume you need it for 5d2 right ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh on December 09, 2016, 11:06:02 AM
Hi where I can find the latest experimental 10bit 12bit build for the 5DMark III 1.1.3
Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 09, 2016, 12:32:08 PM
Is there a new version for 5D3 for mlv 10-12bit testing.

I filmed 50p 10bit continious with half vertical res on the 5D3 with the previous build :) (raw rec/ mlv light)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh on December 09, 2016, 01:07:59 PM
Hi, you mean the nightly bulids or the beta testing? I have a experimental bulid, but I have pinky noise over the whole frame ( 10bit and 12bit)...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jai554 on December 09, 2016, 05:28:15 PM
Anyone tested the latest 60D build?

thanks :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2016, 07:58:45 PM
Is there a new version for 5D3 for mlv 10-12bit testing.

There hasn't been any new commits since I posted this build:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip

I only put up 5D3.113 and EOSM.202 and took down all the other builds because my understanding was that only these two platforms are stable enough for testing and the developers are working on the other platforms so they aren't quite ready for further testing. However, it seems that @reddeercity is seeing some promising results with a 5D2.212 build I put up a while ago so maybe we should check and see if the latest changes show any improvement. Like the most recent builds I posted this is the straight raw_video_10bit_12bit branch without any crop_rec or raw_twk merging going on.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec09.5D2212.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kamranjon on December 09, 2016, 08:22:30 PM
Was able to record 2240x1200 at 10-bit continuous on 7D in crop mode with the build Nikfreak provided.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jpegmasterjesse on December 09, 2016, 08:27:57 PM
Appreciate you posting that, dfort, I've been following along excitedly but not really knowing where to try the more stable builds for 5d2.

In theory, is 10/12 bit Dual-ISO possible on any camera?  As far as I remember Dual-ISO video on 5d2 never made it beyond pretty glitchy.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 09, 2016, 08:33:44 PM
dfort, thanks for the 5D3 10bit_12bit link ( 1.1.3) Now why I have allways pink frames with this build on my 5D3?
(http://img5.fotos-hochladen.net/uploads/aufnahme4vdm37lpsj9.jpg) (http://www.fotos-hochladen.net)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2016, 08:51:48 PM
Now why I have allways pink frames with this build on my 5D3?

Are you using mlv_rec or raw_rec?

raw_rec (MLV Lite) is more stable at this point.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 09, 2016, 09:41:37 PM
I try

5DIII 1.1.3
crop_rec10bit_12bit_twk.2016Nov30.5D3113
raw_twk
raw_rec
mlv_play

When I push zoom to 5xZOOM in 10bit mode the camera just freeze and it's same for all 10 12 and 14bit...

Did I do something wrong ?


magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip
I try the new build and it's same than VanGogh7.

Pink frame with raw_rec in 5xZoom or normal 1080p
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2016, 09:44:47 PM
I try

5DIII 1.1.3
crop_rec10bit_12bit_twk.2016Nov30.5D3113

Try the build I just posted:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec09.5D2212.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 09, 2016, 10:27:46 PM
?
Try the build I just posted:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec09.5D2212.zip
The link is for 5DMII


With this build --> crop_rec10bit_12bit_twk.2016Nov30.5D3113

I can use crop mode with nice result like full preview in 1x1 but it's impossible to use 5xZoom with custom resolution more than 1080p because Freeze.

Also I try to 1080p 16:9 in All 1920 25p with fps override 40fps but the final video just extremely bug. Is there any option to slowmo without stretch in post ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 09, 2016, 11:06:13 PM
@dfort
Tryed the dec 4 build with mlv + sound full HD 10,12 & 14 bit
12 bit suffered from screentearing. Even a little more severe than the previous one I tryed
10 bit upper one third part on every other fame corrupted
14 bit OK
All bitrates in mlv-lite worked as before.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 09, 2016, 11:29:47 PM
wait testing build for my 50D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 09, 2016, 11:53:27 PM
The link is for 5DMII

Oops, gave you the wrong link. Try this:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip

Is there any option to slowmo without stretch in post ?

That can be done with the crop_rec module. The build I pointed you to doesn't have crop_rec or raw_twk because we probably should be getting the basic functions working before adding those modules.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 10, 2016, 12:10:07 AM
Are you using mlv_rec or raw_rec?

raw_rec (MLV Lite) is more stable at this point.

Yes, and pink frames at 10bit and 12bit...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2016, 12:23:53 AM
Quote
Yes, and pink frames at 10bit and 12bit...
? raw_rec or mlv_rec? There is a difference.
raw_rec(mlv_lite) is working very good with 5D mark III(10bit,12bit).

On a sidenote I,ve been using the crop_rec with 5D mark 3 and it works well. I also get good results with raw_twk module and that module is the only one working if you want to view 10 and 12bit footage in cam. I merged and compiled with updated unified code.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 10, 2016, 12:28:37 AM
I use raw_rec ( MLV Lite)...and my build  magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip have not the raw_twk module
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2016, 12:35:18 AM
When you say pink frames at 10bit, 12bit are you referring to when viewing files in camera or viewing dng files on your computer?
It won,t work previewing on your computer camera without the raw_twk module.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 12:40:02 AM
All bitrates in mlv-lite worked as before.

So no change in this build? At least it hasn't gotten worse!

wait testing build for my 50D

I don't want to put out a bunch of builds for platforms that are known not to work yet. My understanding is that only the 5D3.113 and EOSM.202 are working properly and ready for testing. Apparently the 5D2 and 7D are somewhat stable so if you really want to be a crash test dummy, here you go:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec09.50D109.zip

On a sidenote I,ve been using the crop_rec with 5D mark 3 and it works well. I also get good results with raw_twk module and that module is the only one working if you want to view 10 and 12bit footage in cam. I merged and compiled with updated unified code.

Great. You should be able to drop in a module into the ML/modules folder on the card. These should be platform and branch independent--well, in theory. Sometime the backends need to be tweak for a module to work.

raw_twk topic (http://magiclantern.fm/forum/index.php?topic=13163.0) and download link (http://www.magiclantern.fm/modules/modules/raw_twk.mo/raw_twk.mo).

crop_rec topic (http://www.magiclantern.fm/forum/index.php?topic=17021.0) -- does anyone have a link of just the module or does this module require a certain branch? I think it only works on 5D3 and EOSM but I'm not sure.

It won,t work previewing on your computer without the raw_twk module.

Really? I thought raw_twk only affects in camera playback. I use MLVFS to look at the files on my computer and it works fine without raw_twk.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2016, 12:44:08 AM
Correction. raw_twk only affects camera previewing of course  :P

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 12:48:09 AM
I use raw_rec ( MLV Lite)...and my build  magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip have not the raw_twk module

That should be working. Does a regular nightly build work fine for you? Maybe delete the ML settings and try again. Turn on only the raw_rec module.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 10, 2016, 12:52:13 AM
When you say pink frames at 10bit, 12bit are you referring to when viewing files in camera or viewing dng files on your computer?
It won,t work previewing on your computer without the raw_twk module.


on both camera and pc..

I just downloaded the raw_twk.mo for this build magiclantern-raw_video_10bit_12bit.2016Dec04.5D3113.zip,

It seems to works now on my 5D3 1.1.3! ... it works on cam previewing...
but not on pc. Which converter you use??
I tried different converter with no success:

raw2cdng.1.7.4
mlrawviewer
MLVProducer
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 10, 2016, 01:38:12 AM
Use mlvfs or mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 10, 2016, 01:38:57 AM
That should be working. Does a regular nightly build work fine for you? Maybe delete the ML settings and try again. Turn on only the raw_rec module.

Yes nightly builds works well!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 01:42:36 AM
raw2cdng.1.7.4
mlrawviewer
MLVProducer

I doubt any of those will work with 10bit or 12bit. Like Danne said, try MLVFS if you can. Otherwise, the command line tool, mlv_dump will work if you use it like this:

Code: [Select]
mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
Obviously substituting the name of the MLV file you are converting. There was a tweak to mlv_dump to work with lower bit rates so I compiled Mac and Windows versions of mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/mlv_dump_2016Dec09.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: capncannabis on December 10, 2016, 03:51:53 AM
@dfort is there a build to test this on the 600d?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 10, 2016, 05:21:50 AM
On PC , only mlvfs can read 10 & 12bit correctly . Yes you can use mlv_dump But and that a big but  only 10bit can read by applications like A.E. ACR , BM Resolve
12bit from mlv_dump are un-readable as windows can't read them , I posted  here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174826#msg174826)  with examples of 12bit mlv_dump . MLVFS (http://www.magiclantern.fm/forum/index.php?topic=13152.msg127218#msg127218) thread and  mlvfs info (https://bitbucket.org/dmilligan/mlvfs)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 06:27:54 AM
12bit from mlv_dump are un-readable as windows can't read them

I downloaded your 12bit file and it works with both MLVFS and mlv_dump. You need to use these options on mlv_dump:

Code: [Select]
mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
This is what it looks like without those options:
(https://c3.staticflickr.com/1/322/31171410490_a941c28df1_z.jpg)

And this is what it looks like with the options:
(https://c1.staticflickr.com/1/542/31171411080_017415b6bf_z.jpg)

[EDIT] Forgot to mention that I'm on a Mac and I cross compiled mlv_dump for Windows--it should work on your system.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 10, 2016, 06:50:15 AM
Sweet , thanks when I used mlv_dump I didn't think I needed to uses those option
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 10, 2016, 08:29:44 AM
...I didn't think I needed to uses those option

Only for 10bit/12bit files. 14bit files should be fine without using those options.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Asiyuk on December 10, 2016, 11:03:48 AM
I try dec 4 build from @dfort with raw_rec (MLV Lite) on 5d3_1.1.3:
- 12 bit 1920x1080 full work except raw-based histogram while recording
- 12 bit in 5x zoom mode - freeze LV while recording (all lv full of bw dots), but video is ok.
- play doesn't work (pink dots video)

I use MLVFS + DaVinci in post, all good with files. I even use this 4dec build in yesterday commercial shooting.

PS: sorry for by bad english  :-X
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 10, 2016, 03:09:39 PM
A question to people like a1ex and dmilligan:
how can we be sure about addresses we found?

we're porting raw_slurp to 700D and I think we found the right raw buffer address (lv is working good) but when we look at the file we've got only strange things.
here an example: https://drive.google.com/file/d/0BxMUvr5E8ANaaDdsS0JUVFFwVjA/view?usp=sharing

it happens at all bit depth.

Can it be related to the edmac channel instead of wrong buffer address? (ROM call to ROM on 700D anyway :D )
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 10, 2016, 05:12:57 PM
how can we be sure about addresses we found?

Printing DEFAULT_RAW_BUFFER should give the same value(s) with both methods.

To see how Canon does it, look for something like StartMem1Edmac (only on models with EvfState). The older models (550D and earlier) are a bit more difficult.

The 700D should be very similar to EOS M.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 11, 2016, 12:09:50 PM
I doubt any of those will work with 10bit or 12bit. Like Danne said, try MLVFS if you can. Otherwise, the command line tool, mlv_dump will work if you use it like this:

Code: [Select]
mlv_dump --no-fixcp --no-stripes --dng [YOUR.MLV]
Obviously substituting the name of the MLV file you are converting. There was a tweak to mlv_dump to work with lower bit rates so I compiled Mac and Windows versions of mlv_dump:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/mlv_dump_2016Dec09.zip


MLVFS on Mac works well,

MLVFS on Win. I get in the cmd:

C:\>cd \mlvfs_X86

C:\ mlvfs.exe Z:\ --mlv_dir=E:\MLVFS
Unmont


So it doesn't create a virtual drive...where is the mistake? I installed Dokany and Mlvfs_x86

I tried to install the latest dokan version, now I have a other issue...dokanfuse.dll is missing..it's a pain!!!

Now, I have Win 7 64 bit Ultimate

the solution is to install the 32 bit Microsoft Visual C++ 2015 Redistributable (x86) - 14.0.23026 version
:)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 11, 2016, 07:37:36 PM
What is unmont?

For me on Win10 64bit I use a batch file (MLVFS.bat) on my desktop, could add it to start up I guess but don't always need MLVFS. Batch file has one line in it as below.

cmd /k G:\MLVFS_x86\mlvfs.exe M:\ --mlv_dir=G:\MLV_FS\Projects

Double click fires up a cmd window and does the necessary.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 11, 2016, 09:34:29 PM
Got CONFIG_EDMAC_RAW_SLURP working on the 700D

https://bitbucket.org/hudson/magic-lantern/pull-requests/781/700d-enable-config_edmac_raw_slurp/diff

@eNnvi -- the missing piece was:

edmac-memcpy.c
Code: [Select]
void edmac_raw_slurp(void* dst, int w, int h)
 {
     /* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
-#ifdef CONFIG_EOSM
+#if defined(CONFIG_700D) || defined(CONFIG_EOSM)
     uint32_t dmaFlags = 0x20000000;
 #else
     uint32_t dmaFlags = 0x20001000;

Note: this might also work for the 650D and maybe the 100D if we can get the DEFAULT_RAW_BUFFER address for those platforms.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jc on December 11, 2016, 09:49:11 PM
Has there been any further looking into how the losses cr2 raw compression is applied ? Could the discovery of running the firmware in QEMU or finding a the "Eeko" core help this discovery ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 11, 2016, 10:15:20 PM
Quote
Got CONFIG_EDMAC_RAW_SLURP working on the 700D
Breakthrough? 10bit/12bit kickin it with raw_rec?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 11, 2016, 10:43:32 PM
10bit/12bit kickin it with raw_rec?

Yep--on the 700D. Who's next?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on December 12, 2016, 01:25:58 AM
550D ? But as A1ex says above, will be more difficult.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 12, 2016, 05:34:46 AM
this might also work for the 650D and maybe the 100D if we can get the DEFAULT_RAW_BUFFER address for those platforms.
Awesome!
Got 650D here, VM all set up and ready for testing. Have some experience with finding STUBS addresses. Can you give us some guidelines on how to find this god damn DEFAULT_RAW_BUFFER address?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2016, 07:43:01 AM
Got 650D here, VM all set up and ready for testing. Have some experience with finding STUBS addresses. Can you give us some guidelines on how to find this god damn DEFAULT_RAW_BUFFER address?

Great, check dmilligan's posts. He provided all the hints we used to find the right address. You can check on the pull request for the 700D (https://bitbucket.org/hudson/magic-lantern/pull-requests/781/700d-enable-config_edmac_raw_slurp/diff) to make sure you find all the pieces that are needed for the 650D. They should be the same with the exception of the DEFAULT_RAW_BUFFER address.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 12, 2016, 09:03:54 AM
Thanks to dfort, i've managed to find the right address for 650D:

Code: [Select]
#ifdef CONFIG_650D
#define DEFAULT_RAW_BUFFER MEM(0x25B00 + 0x3C)
#endif

LiveView works during recording, no pink frames or glitches in extracted dngs (raw_rec, both 12- and 10-bit). Some shifting occurs though, focus pixel masks no longer works (MLVFS windows), need more testing.

Upd. Actually, there are no focusing pixels anymore on 650D (raw_rec, 14 bit as well as 12 and 10 bit), but there are these along the contrast edges (all over the frame). What am I missing? Were there some changes in raw framework?
(http://s2.1pic.org/files/2016/12/12/71a532d2cd490e8b5db6.png)

What PREFERRED_RAW_TYPE should we use for 650D to get the old behavior (usual horizontal pattern of focusing pixel, removable by PDR etc.)?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 12, 2016, 04:33:41 PM
Do you all get this from live view while doing crop recording?

(http://forumbilder.se/FBVC0/img-2009.JPG)

Using 4th December build.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 12, 2016, 04:43:48 PM
I had this with Global Draw on...try with GD OFF
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 12, 2016, 05:09:49 PM
D_Odell. Are you using raw_twk? When in crop mode this will surely happen.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2016, 06:44:21 PM
What PREFERRED_RAW_TYPE should we use for 650D to get the old behavior (usual horizontal pattern of focusing pixel, removable by PDR etc.)?

I'm seeing the same happening on the EOSM and 700D. I tested out some different RAW_TYPE's and found some that show the old focus pixel patterns, others that show only a few pixels, mostly around high contract boundaries and yet others that show a pixel pattern all over the image including areas that we've never seen focus pixels before. I reported on that in the Dealing with Focus Pixels in raw video topic (http://www.magiclantern.fm/forum/index.php?topic=16054.msg176032#msg176032).

We recently increased the RAW_DEBUG_TYPE limit and switched over to defining the PREFERRED_RAW_TYPE in hex instead of decimal values. The pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/780/increase-raw_debug_type-limit/diff) is quite an interesting read.

To activate RAW_DEBUG_TYPE, define it here:

Code: [Select]
#undef RAW_DEBUG        /* define it to help with porting */
#undef RAW_DEBUG_DUMP   /* if you want to save the raw image buffer and the DNG from here */
#undef RAW_DEBUG_BLACK  /* for checking black level calibration */
#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
/* see also RAW_ZEBRA_TEST and RAW_SPOTMETER_TEST in zebra.c */

You also need to give it a starting value. Anything will do because you can change it from a menu:

Code: [Select]
#ifdef CONFIG_650D
#define PREFERRED_RAW_TYPE 0x12
#endif

This will give you a new Debug menu item that allows you to enter values from 0 to ffff (65,535).

(https://c3.staticflickr.com/1/766/31229380130_7893122693.jpg)

Try them all out--that should keep you busy for a while. Actually, they repeat and many do not produce valid images or cause raw detect errors.

You said that the frame might have shifted? In the raw_lv_get_resolution function of raw.c you might check if the 650D uses either of these resolutions--I'm pretty sure it uses the 700D values:

Code: [Select]
    #ifdef CONFIG_600D
    *width  = zoom ? 2520 : mv1080crop ? 1952 : mv720  ? 1888 : 1888;
    *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
    return 1;
    #endif

Code: [Select]
    #ifdef CONFIG_700D
    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
    return 1;
    #endif

Are you planning on putting up a pull request for this? It would be great to get some feedback from other 650D users.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 12, 2016, 06:48:33 PM
I've disassembled the 50D's FW and found 4 instances where 'lv_raw_dump' is mentioned but after that I'm lost  ??? - yes I did read @dmilligan's post (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969)) but I still don't know what I'm looking at.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2016, 07:34:44 PM
...found 4 instances where 'lv_raw_dump' is mentioned but after that I'm lost...

Yeah, I found the same in my EOSM and 700D disassembly. I had to look for the address that dmilligan used on the EOSM and find the equivalent in the 700D. In art that would be like tracing a master's drawing but hey, it worked.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 12, 2016, 07:42:44 PM
there are 4 because 2 are real lv_raw_dump, the other 2 shoulde be lv_raw_dump2

anyway the same instances of lv_raw_dump should take you to the same spot (address jump)

from here it's harder :D

there should be a sub that actually loads the address of the raw buffer, you need to find that! how? well, read the lv_raw_dump function (the bl from the string you found) and check when the return pointer is used for reading or writing something
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 12, 2016, 08:10:03 PM
D_Odell. Are you using raw_twk? When in crop mode this will surely happen.
Hi Danne! No not at that time, when used it was completely green..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 12, 2016, 09:05:49 PM
To activate RAW_DEBUG_TYPE, define it here:
Got it. Tested, and at least 0x10 PREFERRED_RAW_TYPE works like we used to (650D, raw_rec, both 10 and 12 bit). No tearing, no artifacting, no shifting (even PDR works). Absolutely usable for shooting already. Thank you dfort!

The pull request is quite an interesting read
Indeed. I'll try other raw types as well. By the way, when using mlv_rec instead of raw_rec, i got some tearing in 10 bit mode: every other frame's top third is slightly shifted (5 or 6 pix) to the top.

Are you planning on putting up a pull request for this
Shame on me, but i spent some years in different kind of activity, and i just forget everything about Git, Bitbucket, branching etc. Actually coding is way more understandable :) I'll try to refresh and join later. As for 650D's CONFIG_EDMAC_RAW_SLURP, it may be easier for you to just add the proper DEFAULT_RAW_BUFFER MEM(0x25B00 + 0x3C) in your commit. Other changes are basically the same as yours for 700D obviously.
Or I can share my test build for other 650D users if this make sense.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 12, 2016, 09:15:13 PM
From this analysis (http://www.magiclantern.fm/forum/index.php?topic=18393.0), PREFERRED_RAW_TYPE 0x10 appears to be the best choice for all DIGIC 5 and 6 cameras, and likely 0x5 for all DIGIC 4.

The EDMAC channel should probably be 0x12 for all DIGIC 5 models (unless used by something else).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on December 12, 2016, 09:31:01 PM
...from here it's harder :D

there should be a sub that actually loads the address of the raw buffer, you need to find that! how? well, read the lv_raw_dump function (the bl from the string you found) and check when the return pointer is used for reading or writing something

Ok, but I don't understand the last bit  :-\

the bl is loc_ff064830 but what do I do with it? How do I check?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 12, 2016, 11:36:12 PM
...when using mlv_rec instead of raw_rec, i got some tearing in 10 bit mode: every other frame's top third is slightly shifted (5 or 6 pix) to the top...

Yeah, noticed that too, raw_rec (MLV Lite) is working better at this time.

...it may be easier for you to just add the proper DEFAULT_RAW_BUFFER MEM(0x25B00 + 0x3C) in your commit. Other changes are basically the same as yours for 700D obviously.
Or I can share my test build for other 650D users if this make sense.

I updated my pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/781/650d-and-700d-enable/diff) so it includes the 650D. @Rewind -- are you on bitbucket? There are a few Rewinds there so I'm not sure if one of those is you. In any case, I'd appreciate if you check it out and make sure I didn't leave out anything important.

I made a new set of test builds mainly for 650D and 700D users but found out that the 60D and 600D has CONFIG_EDMAC_RAW_SLURP defined and heard reports that the 7D was also working so I went ahead and built all the platforms. This time I didn't zip them all up one bundle so we can check up on the number of downloads per platform. Also took down the older builds to avoid confusion.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kamranjon on December 13, 2016, 01:33:12 AM
Just tested this new build on 7D - it works! - the live view in some cases works as well which is awesome - what i had to do was: i turn global draw/liveview on, and then i turn it off, and when i hit record i am able to see what i am shooting! If i don't do that first it doesn't seem to work, it just freezes like it had previously.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on December 13, 2016, 07:38:18 AM
Canon 50D to MLVFS Workflow w/ 12-12-16 Build:

Module Report: RAW Lite

1. 10-bit and 12-bit display a repeating first frame; every other frame.

2. 5X features "pink frames" in 10-bit, 12-bit, and 14-bit; every frame.

3. 14-bit works great; aside from 5X zoom.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 13, 2016, 09:03:40 AM
Just tested 500D, 550D, 600D and the 7D with dfort latest compilations. (raw_rec, mlv_lite)

500D , raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
550D, raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
600D, raw10bit, 12bit works flawlessly! raw slurp enabled already?
7D, raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 12:10:27 PM
Will the 650D/700D port work on the 750D? Or is that a different kettle entirely… ?   :-\
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on December 13, 2016, 12:22:59 PM
How to saddle a horse not born yet?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 12:26:52 PM
Thanks Walter. What's delaying the birth? Testers? Major technical differences? I'm genuinely interested.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on December 13, 2016, 12:46:09 PM
As always: Lack of people with skills (Assembler and C for embedded devices, preferable ARM architecture) and spare time at hand. It will take (estimated) some hundred hours - partly mind numbing - to port ML to a new platform. And this is not just a new platform but a different processor generation, too.
And such a person will be asked to maintain the cam and its ML port. If there is a new Canon firmware and it is decided to port ML for this new firmware: Yupp, reverse engineer it a second time (should be easier, but PITA, 2nd edition).
See threads for 750D, 760D and 80D. 7D2 will have to wait.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 12:49:19 PM
Thanks for the comprehensive reply Walter.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 13, 2016, 12:51:18 PM
First i didn't understand why you post it here as the thread is about something else.
Second you Can get your cam, study assembly, C coding, arm architetture by your own and help porting the code speeding up a lot the porting process
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 12:55:06 PM
@ eNnvi Walter just explained the exact same thing to me – but with tact.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 13, 2016, 03:21:00 PM
Sorry i didn't mean to be rude but that's what was told me sometime ago (for another project) and i tried and studied and now i Can at least understand the concept and the c code behind ml but i'm still learning a lot about assembly (even if i studied it at University). There's no limit about you but about how much you want to help and learn :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: timbytheriver on December 13, 2016, 04:23:57 PM
@eNnvi Thanks. No offence taken. :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 13, 2016, 05:15:14 PM
500D , raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
550D, raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)
600D, raw10bit, 12bit works flawlessly! raw slurp enabled already?
7D, raw10bit, 12bit working, earthquake picture, scrambled lines on top(Check LEVISDAVIS description)

This makes sense because the 600D is the only camera on the list that has CONFIG_EDMAC_RAW_SLURP defined. I just went through all the platforms and checked the ones that are "slurping":

1100D.105No. Last Successful Build (2015-05-03 00:25) What's up with that?
500D.111No
50D.109No but @Andy600 is working on it.
550D.109No
5D2.212No but @reddeercity is reporting some success.
5D3.113Yes
600D.102Yes
60D.111Yes
650D.104Yes with pending pull request (test build posted)
6D.116No
700D.114Yes with pending pull request (test build posted)
7D.203No but maybe @nikfreak is working on it?
EOSM.202Yes
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 13, 2016, 05:26:07 PM
Working on 6D and 7D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 14, 2016, 01:34:47 AM
Thanks @dfort for the latest updates.

@EOSM users:

Are you guys able to get mlv_play.mo to work w 10/12-bit files after each recording takes? I notice sometimes I have to either turn off camera and back on to get mlv_play.mo to work properly otherwise I'll get a black screen stuck on LCD liveview.

Btw it's nice to be able to see LiveView while recording. Also confirmed to work in crop_mode.mo but not in 5x zoom mode.


More test to come later with 5D3 & 7D.

Perhaps I'll wait until @eNnvi provides a new build for the 7D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: saulbass on December 14, 2016, 01:49:47 AM
@dfort - thanks for the recent 650D update - (magiclantern-raw_video_10bit_12bit.2016Dec12.650D104.zip).

10bit RAW 1280 x 720 works continuously - only issue are the focus pixels and some image tearing but I’m not using my fastest SD card.

With the MLV sound added to this the video is very choppy.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 14, 2016, 02:54:11 AM
Hi and thanks for ALL  :)

I test the new version for the 7D and i can record continuos and with the camera screen (LiveView) good (canon preview), but in camera with mlv_play.mo i just see pink/flashy image. In computer, after use MPV player i see this pink and good image blinking very fast.

Cheers
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 14, 2016, 03:25:03 AM
Working on 6D and 7D

Great. Make sure and check out the comment @nikfreak posted on the 650D/700D pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/781/650d-and-700d-enable/diff#comment-28272540). You're probably way ahead of me!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on December 16, 2016, 06:26:49 AM
Can someone upgrade the first post, it's from 2013...!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 16, 2016, 06:47:52 PM
i think i got 6D almost working, need my tester to tell me if the last build works.

Can someone please share me a 60D ROM1.bin file? need it to find the way to find (strange to say :P ) the address of 7D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 16, 2016, 08:13:20 PM
Great work eNnvi  :)
Rawslurp for 6d.
Happily willing to test, is the source somewhere available on bitbucket?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 16, 2016, 09:41:16 PM
eNnvi: unfortunately, the 60D won't help you solving the 7D. It's old generation, and the dual processor makes it probably the most difficult of all models.

From what I've tried so far on those old models (without much success yet), we have at least two problems:
- the raw buffer is not allocated all the time (as it's done on the new models) -> easy to fix, see the 1100D raw branch
- the vsync hook currently used doesn't appear to be the best for getting a raw image
- the exact behavior of EDMAC in case of input/output dimension mismatch is not very well known

Solving those models may require either some serious low-level research, or a great dose of luck. I'm just scratching the surface here (http://www.magiclantern.fm/forum/index.php?topic=18315).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 16, 2016, 10:04:43 PM
Im getting some really great images in lowlight with 10 and 12-bit mod. Really superb work from you guys. Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 17, 2016, 06:47:14 PM
Did a little testing on my 600D.
At 12 bit, 1280 @ 2:35 / 23.9fps, got 31s.  For Indy film, this is about 90% of your shots.
At 10bit, 1280 @ 2:35 /23.9fps, continuous.  For Indy film, the few shots longer than 30s switch to 10 bit. 

Selected MLV_raw, and notice that playback at 12 bit crashes camera, but at 10 bit, it doesn't but its preview is only white with verticle lines. raw_twk.mo provides 12bit & 10bit previews

I am mega impressed with the improvements. 
1280 @ 2:35/24 fps 10 bit raw, finally makes the humblest of cams a useful raw video cam (new learner or student).
Sorry I have not been using raw video in over 1.5 years.
I am using Windows 10, what's the best automated work-flow to convert my MLV_raw into PPro or Resolve friendly format?
(PM please as I dont want to go off topic on a dev forum). 

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 17, 2016, 07:18:44 PM
For 10/12bit playback add the excellent raw_twk.mo module which g3gg0 recently polished.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on December 17, 2016, 08:46:16 PM
How's 550D? Last time I tried it had pink every other frame.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 17, 2016, 09:32:33 PM
How hard is to type 550D in your browser's search box?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ddelreal on December 18, 2016, 12:06:50 AM
I did, I just thought that maybe somebody had info that wasn't yet shared here. Sorry I ruffled your feathers.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 01:05:32 AM
For 10/12bit playback add the excellent raw_twk.mo module which g3gg0 recently polished.

i just polished mlv_play and raw_twk and pushed them into unified.

mlv_play directly uses raw_twk now if available.
-> no raw preview lockups during recording

(also added a zoom feature when pressing zoom button during playback)

needs for sure some tests.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 18, 2016, 05:47:58 AM
Processed the 600D 1280x544 (2:35), 24fps (canon default) for 29s.

Used Mystic to convert to DNG, but got few lost frames.  In LR saved to TIFFs 16bit sRGB, and built clip in AE with skipped frames.   Got wierd wabble when panned.

Tried again by DNG dropped from card into Resolve Lite, had bigtime shadow noise but no wierd wabble on same pan. 


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 18, 2016, 07:22:03 AM
@ShootMeAlready the only Correct extractors are MLVFS and new updated mlv_dump
all others are not designed to extract bit reduced raw (10-12bit) you may get a some what usable
image from 10bit raw with other convertors but it will not look right or have noise etc.... .
the others are designed to read 14bit raw . If others do it will be only by chance .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 18, 2016, 07:31:15 AM
i just polished mlv_play and raw_twk and pushed them into unified.

Could you please merge unified into raw_video_10bit_12bit or at least raw_twk into raw_video_10bit_12bit? Seems like the reduced bit raw files really need it.

mlv_rec directly uses raw_twk now if available.

You mean mlv_play, right?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on December 18, 2016, 09:07:31 AM
Could you please merge unified into raw_video_10bit_12bit or at least raw_twk into raw_video_10bit_12bit? Seems like the reduced bit raw files really need it.

+1

You mean mlv_play, right?

Pretty sure that's what he meant.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 12:27:31 PM
yeah i meant mlv_play of course. corrected that last post.
grafted that patches into the branch. happy testing ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 18, 2016, 01:07:53 PM
i just polished mlv_play and raw_twk and pushed them into unified.

The unified branch doesn't have variable bit depth support in raw.c, so I'm not sure what was the reason for pushing directly to unified. Ran a quick test on 5D3 and recordings at lower bit depths are corrupted.

Sounds like a rushed move to me. Looks like the mlv_rec changes were committed by mistake, sorry, all fine now.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 01:33:32 PM
wait, you used unified (which doesnt support 10/12 bit video) to play back files that were recorded with raw_video_10bit_12bit branch?
how did you get the lower bit depth recordings with it? :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 01:51:38 PM
ah now i understood. initially i wanted to graft the mlv_play patch that adds variable bit depth only.
things went wrong and the whole patch was grafted into unified, giving 10/12/14 bit mlv_rec in unified which obviously doesn't work :)

fixed now, thanks a1ex
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 18, 2016, 04:48:44 PM
@g3gg0

I see that raw_twk is now in the raw_video_10bit_12bit branch--Thanks!

A couple of questions.

Shouldn't the raw_twk module be added to Makefile.modules.default on the raw_video_10bit_12bit branch since it is required for playing back 10/12 bit video?

Does raw_twk load automatically with mlv_play or does the user have to turn on both modules? Obviously having to turn on just mlv_play would be better but merging raw_twk into mlv_play might be even better, if that's possible.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 18, 2016, 05:55:15 PM
i'd say it makes the most sense in ML core.
but it is not as good as i want it to be, guess a1ex sees it likewise.

so either keep it as optional module and research how to make stuff clean, faster and more stable
or integrate that half-cooked stuff into core, requiring some workarounds.

well, i'd prefer to keep it as an (surprisingly stable) option for those who need the high performance.
and meanwhile we look for the golden solution with full engine support.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on December 18, 2016, 07:14:53 PM
@all
could someone post some 10/12bit samples? (samples-thread (https://www.magiclantern.fm/forum/index.php?topic=11899.0))

regards chmee
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 18, 2016, 08:42:07 PM
Some initial testing around the blazing fast raw_twk module. I also tested it with the magnificent crop_rec module on a 5D mark III.

Pros
5D mark III mlv_lite
1 - The 3xzoom mode grayscale preview crash bug is gone!
2 - Playback works great also with the magnificent crop_rec module(tested centred 5xzoom)
3 - 10bit, 12bit, 14bit works

550D mlv_lite
1 -  Briefly tested on a 550D and all seems to work the same as for the 5D mark III but with the lack of slurp stream causing 10bit, 12bit files corruption(another issue not related to raw_twk).

eos m mlv_lite
1 - All is working but no overlay buttons. Can,t delete etc.

Cons (initial testing summary)
1 - Scrolling back and forth between mlv files will soon cause the previewing to turn black. This behaviour exists for both mlv_lite and mlv_rec
2 - Eos m still lacks play, pause, next, delete buttons


@Chmee, posted some samples of my cat and linked the clips in your samples thread.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on December 18, 2016, 08:56:36 PM
Some initial testing around the blazing fast raw_twk module. I also tested it with the magnificent crop_rec module on a 5D mark III.

Pros
5D mark III mlv_lite
1 - The 3xzoom mode grayscale preview crash bug is gone!
2 - Playback works great also with the magnificent crop_rec module(tested centred 5xzoom)
3 - 10bit, 12bit, 14bit works



Amazing new ! Can't wait to test the new 10/12bits build coupled with the super crop_rec module version :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 18, 2016, 09:09:45 PM
Has there been any further looking into how the losses cr2 raw compression is applied ?

Yes (http://www.magiclantern.fm/forum/index.php?topic=18443).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: AWPStar on December 18, 2016, 10:59:05 PM
I just tried to decode 10 and 12 bit's raw files.
http://www.magiclantern.fm/forum/index.php?topic=15271.msg176632#msg176632

I think i can implement it in several days.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on December 19, 2016, 12:03:22 AM
I don't know if what I have achieved counts as successful, but I think I've got closest to working 12 bit on the 5D2. As we all know raw_rec produces split frames all the time, and I haven't had any success with it, but mlv_rec produces no split frames and only a small bar of noise at the top in 12 bit, or a large one in 10 bit, so I tried to move it down using digital dolly, and it actually got rid of the bar of noise! Clean, 12 bit, 1856x1044, continuous... AND MLV... on the 5D2... with potential to have audio!!! but of course tilt shifted down.

Proof:

If mlv_rec has something that is more functional it could be transferred to raw_rec right? Also there must be a work around to get it in the middle, maybe by adding blank space at the top of the video and making it so that digital dollying all the way down is now the middle? I'm sure this is progress, isn't it?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 19, 2016, 12:32:16 AM
Posted a new set of test builds of the raw_video_10bit_12bit branch now with the new and improved raw_twk module.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 19, 2016, 01:35:42 AM
eos m mlv_lite
1 - All is working but no overlay buttons. Can,t delete etc.

Found the same EOSM issue here. Turn off raw_twk and the overlay buttons are back but of course it doesn't playback 10bit files.

Works fine on the 700D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 19, 2016, 02:28:32 AM
mlv_rec got a fix for the alternating frame errors with lower bit depths, along with some other stuff that was outdated or simply odd.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 05:41:33 AM
Quote
Found the same EOSM issue here. Turn off raw_twk and the overlay buttons are back but of course it doesn't playback 10bit files
?
My tests was done around raw_twk module with eos m and other cameras here.
http://www.magiclantern.fm/forum/index.php?topic=5601.msg176617#msg176617
There were never any issues with mlv_play alone on eos m.

Quote
mlv_rec got a fix for the alternating frame errors
Great! More than welcome.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 19, 2016, 07:26:43 AM
mlv_rec got a fix for the alternating frame errors with lower bit depths, along with some other stuff that was outdated or simply odd.

Alternating frame errors with mlv_rec at 10bit are still present on the EOSM and 700D. It seems to be better but I haven't done a regression test.

[EDIT] Oops missed that last commit -- yes, the alternating frame errors with lower bit depths is fixed. Thanks! Now we can record audio with 10/12bit raw video.

Making a new set of test builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 08:05:47 AM
In already? That is brilliant.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 19, 2016, 08:41:13 AM
Yay for audio
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 19, 2016, 10:42:36 AM
Super awsome!  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 01:42:09 PM
5D mark III
Just tested mlv_rec with audio on 5D mark III and both 10bit and 12bit works really good. Very nice. Crop_rec module is also working with audio and all frames looked great. I didn't,t test this very much so more testing.

Eos m
I keep bugging about this cam but its a great little cam that needs all the love it can get.
Records mlv_rec 10/12bit fine. If raw_twk.mo is added and you try to record sound will make the camera crash kind of. There is a faulty message saying 'threads failed to start' I think and then the camera needs to be restarted. If disabling raw_twk everything works, except for previewing 10/12bit files of course.(Yeah, and button overlays still missing with raw_twk enabled)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 19, 2016, 01:53:22 PM
the EOS M has too few thread ids. so this is kind of a known issue we cannot do much about :(
less modules that use threads would help.
(also reworking modules that they only start/end threads when needed would help)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 01:59:55 PM
I see, all in all great working cam delivering continuous 10bit recordings on highest resolution.
Any thoughts on mlv_play buttons? They seem vanished into oblivion once raw_twk starts running?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on December 19, 2016, 02:51:13 PM
mlv_play issues -> http://www.magiclantern.fm/forum/index.php?topic=9062.0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on December 19, 2016, 02:57:46 PM
Eager to test 10bit mlv on 70D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 03:00:40 PM
@g3gg0
Thanks a lot for taking the time on this.
Just to be sure here. Mlv_play buttons(delete, play, pause, scroll etc) If not going through raw_twk. Link you are posting tells me I have to go through file manager instead? Conclusion is that raw_twk can,t maintain the otherwise working buttons?

*Updated more info here
http://www.magiclantern.fm/forum/index.php?topic=9062.msg176677#msg176677
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on December 19, 2016, 03:52:58 PM
50D 10bit
last build 10bit_12bit.2016Dec18.50D109
view with MLVProducer
(http://image.prntscr.com/image/483c01587cfa4e43abbeeab24277e763.jpeg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: AWPStar on December 19, 2016, 04:01:28 PM
@CITY-U1001
Quote
I just tried to decode 10 and 12 bit's raw files.
http://www.magiclantern.fm/forum/index.php?topic=15271.msg176632#msg176632

I think i can implement it in several days.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 19, 2016, 09:02:56 PM
5D mark III
Just tested mlv_rec with audio on 5D mark III and both 10bit and 12bit works really good. Very nice. Crop_rec module is also working with audio and all frames looked great. I didn't,t test this very much so more testing.

Eos m
I keep bugging about this cam but its a great little cam that needs all the love it can get.
Records mlv_rec 10/12bit fine. If raw_twk.mo is added and you try to record sound will make the camera crash kind of. There is a faulty message saying 'threads failed to start' I think and then the camera needs to be restarted. If disabling raw_twk everything works, except for previewing 10/12bit files of course.(Yeah, and button overlays still missing with raw_twk enabled)

Hi Danne, how is the workflow to convert 10bit including the soundfile?  for the MLV I use MLVFS..and for the soundfile named ?? MLV_REC.TMP ??

thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 19, 2016, 09:04:07 PM
New build "magiclantern-raw_video_10bit_12bit.2016Dec18.5D3113"

Tested one 5D mark III 1.1.3 with mlv_rec and raw_twk

Work on 10, 12 bit also in crop mode in high res really nicely ! Also work in slowmo 50p with stretch mode.
Could it be possible to do somethings and record slowmo without stretch in full frame mode ?

dfort, I have no idea how you did it but you are dat king ! Huge deep thanks and respect
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 19, 2016, 09:06:29 PM
Having the same problems that @Ilia3101 have but with the 7D

Going trie the new version of @dfort
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 09:10:20 PM
@VanGogh7
Audio is included and revealed automagically when opening the file in mlvfs. Mlv_dump will build a solid wav file as well upon converting to dng files.

@MAG
With the crop_rec module I think it,s posible to record unstretched mv720 mode(not mv1920)

Also if previewing the MLV files with raw_twk enabled please test to go back a few files then forward a few files. Do this a couple of times and see if screen turns black?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on December 19, 2016, 09:40:33 PM

@dfort

With your last version for the 7D it got the same problems... maybe is needed to merge/compile the new update from the mlv_rec that got a fix for the alternating frame errors in lower bit depths.

Thanks=))
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on December 19, 2016, 09:47:20 PM
@Danne

Thanks, strange, first on my half-full Card ( Komputerbay 64GB ) MLFVS was not able to create a .wav file from the MLV_REC.TMP. After formatting the card now all works fine! :)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 09:52:45 PM
You have to enable mlv_snd module. MLV_REC.tmp has nothing to do with this. Anyway, seems you have it working so let,s keep up the 10/12bit testing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 19, 2016, 09:59:52 PM
@MAG
With the crop_rec module I think it,s posible to record unstretched mv720 mode(not mv1920)

Also if previewing the MLV files with raw_twk enabled please test to go back a few files then forward a few files. Do this a couple of times and see if screen turns black?

Yes in playback, the screen turns black after couple of time. (raw_twk and mlv_play)

Also when I say record crop mode, I mean in ZOOM mode. The build I have doesn't have "crop_mode" module (2016Dec18.5D3113)

So, it's impossible to record fullframe 1080p without stretch ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 10:03:38 PM
I think maximum is 38 fps mv1080. Also let,s try to stay on topic here. You can dig out a lot of info with 'search' regarding stretch/unstretch footage.
Thanks for verifying raw_twk issue
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aschille84 on December 19, 2016, 10:08:25 PM
6D: 10/12-bit with mlv_snd working in raw_rec and mlv_rec. Nice!

In cam preview with raw_twk working, but goes black when scrolling thru 3 clips or more.

Live view freezes on canon/ml greyscale without global draw preview, scrambled grey on ml greyscale w/global draw on.

latest @dfort build "magiclantern-raw_video_10bit_12bit.2016Dec18.6D116"
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 19, 2016, 10:18:30 PM
Quote
Live view freezes on canon/ml greyscale without global draw preview, scrambled grey on ml greyscale w/global draw on.
How do you mean? Does the camera freeze when previewing with grayscale?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: aschille84 on December 19, 2016, 10:30:27 PM
How do you mean? Does the camera freeze when previewing with grayscale?


On the 6D normal canon preview freezes in a still frame. I doesnt freeze on greyscale, its scrambled grey as earlier when global draw is on. So, limited preview options.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 19, 2016, 10:41:01 PM
At this moment a bunch of camera have working live view while recording 10/12 bit (5D3, 60D, 600D, 650D, 700D and the EOS-M)
other ones have freezing live view while recording 10/12 bit video.



Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on December 20, 2016, 07:52:28 AM
A brief summary of the test MLV-REC 10bit RAW on 600D:
RAW_twk - works
MLV_ snd - works
Most of the raw-videos converted with MLV_dump are badly decoded.

:-)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 20, 2016, 08:01:04 AM
When using mlv_dump.
http://www.magiclantern.fm/forum/index.php?topic=5601.msg176263#msg176263
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DavidSh on December 20, 2016, 09:26:14 AM
Any news on 5d3.123?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on December 20, 2016, 03:54:20 PM
@D

Can you update the first post with the latest results?

- what works at the moment
- tools for process on pc or/and mac at the moment

It's just a couple of lines.

Thank you.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 20, 2016, 07:23:53 PM
Using 18dec build and the crop marks or actual pixels aren't shown in live view on 113 5D3. If I remembered correct the 4 Dec showed correct crop/actual pixels while recording. Anyone experiencing the same? Problem is that it's impossible to frame while rolling. Best, David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 20, 2016, 08:21:16 PM
Using 18dec build and the crop marks or actual pixels aren't shown in live view on 113 5D3.

5D3.113 has the most amount of downloads by far so someone should be able to comment on that.

Just for fun, here are the download counts of the December 18 test builds. Checkmarks are for cameras that are working with CONFIG_EDMAC_RAW_SLURP:

5D3.113-28
7D.203-8
5D2.212-6
EOSM.202-6
550D.109-5
6D.116-5
600D.102-4
50D.109-3
60D.111-3
650D.104-3
700D.114-3
500D.111-1

Any news on 5d3.123?

That is a bit more challenging to build and I don't have a camera to check it out but here's a build for any 5D3.123 user who would like to check it out--please report your findings.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123.zip

@D

Can you update the first post with the latest results?

@d started this topic on May 22, 2013, has only made 6 posts and hasn't been seen on the forum for over two years.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MAG on December 20, 2016, 08:51:13 PM
Using 18dec build and the crop marks or actual pixels aren't shown in live view on 113 5D3. If I remembered correct the 4 Dec showed correct crop/actual pixels while recording. Anyone experiencing the same? Problem is that it's impossible to frame while rolling. Best, David

I try the 18dec 5DIII there is no crop_rec and when I zoom record I don't have good liveview...

May be dfort do some test before adding crop_rec ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 20, 2016, 09:57:21 PM
May be dfort do some test before adding crop_rec ?

I'm doing test builds of just the raw_video_10bit_12bit branch for now. The crop_rec module is only working for the 5D3 and EOSM, is on a different branch and I don't want to introduce yet another variable until all possible cameras are working. If you want to experiment with crop_rec try merging branches--it really isn't that hard to compile and there are several tutorials in the General Development Discussion (http://www.magiclantern.fm/forum/index.php?board=25.0) area of the forum.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on December 20, 2016, 11:37:55 PM
10bit raw - video test /build 18dec 2016/

www.youtube.com/watch?v=XH3yVrSHVQQ

"magiclantern-raw_video_10bit_12bit.2016Dec18.600D102.zip"

:: EOS 600D ::
MLV_REC + MLV_SND, ISO 400, 1728x648, 23,98fps

Most of recorded videos are fine.
Some of them are bad decoded. Some of videos are without sound.

Original video file:
https://ulozto.cz/!JoQQdlZxLl9X/10bit-raw-video-test-build-18dec-2016-mp4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on December 21, 2016, 12:52:21 AM
5d3 113 mlv + sound 10, 12 & 14 bit seams to be working for me as well.

 :)

Gonna try 123 later.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vstrglv on December 21, 2016, 10:07:59 AM
If you want to experiment with crop_rec try merging branches--it really isn't that hard to compile and there are several tutorials in the General Development Discussion area of the forum.
It is not easy. There are many conflicts when merging Unified into crop_rec branch.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on December 21, 2016, 01:22:54 PM
ok, put 10 and 12bit into raw2cdng (https://www.magiclantern.fm/forum/index.php?topic=5618.1325). if someone s still using this tool :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on December 21, 2016, 02:35:08 PM
ok, put 10 and 12bit into raw2cdng (https://www.magiclantern.fm/forum/index.php?topic=5618.1325). if someone s still using this tool :)


Of course, im still using your soft as mlv viewer and extractor. Thanks for your update.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on December 21, 2016, 08:19:19 PM
Where is the module for for 5d3 1.1.3?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ali Oliya on December 22, 2016, 08:37:44 AM
Why this 10bit/12bit build is not available for download in  nightly builds? (5d3 - 1.1.3)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on December 22, 2016, 08:55:53 AM
Because it is still in development. There is a "main repository" containing code considered to be tested/evaluated/peer reviewed. New features will tested by whoever dares to do so (= not afraid taking additional risks) in so called "branches". If (and only if) a dev is convinced code has been verified it will make it into "main" and then you will find it in the nightlies.

If you want to test this one (and taking all the responsibilities) you may use downloads compiled by members or build your own development environment. There are some quite good tutorials by dfort working perfectly for beginners like me.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on December 22, 2016, 09:13:57 AM
Test of 10bit RAW_REC with EOS 600D.
www.youtube.com/watch?v=WZ9unJonmOY
Build: "magiclantern-raw_video_10bit_12bit.2016Dec18.600D102.zip"
RAW_REC, ISO 400-800, 1728x972, shutter: 1/15s, 12fps (postprocess: RawTherapee, Virtualdub, Avisynth, Kdenlive)
Original video file: https://ulozto.cz/!SbnYaSK0n6gx/karvina-frystat-2016-avi

Danne: thank you for the advice ;-)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 09:47:19 AM
As seen some post before, eNnvi tried to make CONFIG_EDMAC_RAW_SLURP work on the 6d.
One thing is working, live view doesn't freeze anymore when recording 10/12 bit on the 6d  :D
However new problem now is, the recording gives scrambled/pink frames  :(

So we're halfway there...
Since live view works, the address for CONFIG_EDMAC_RAW_SLURP is probably right and looks like this in the 'raw.c' file:
#ifdef CONFIG_6D
// found at 0xFFCE513C (ram func)
#define DEFAULT_RAW_BUFFER MEM(0x76d6c + 0x2C)
#endif


Question is now, why do we end up with scrambled/pink frames. Are we not using a right free edmac channel in 'edmac-memcpy.c' ?
which is configured as follows:
#if defined(CONFIG_650D) || defined(CONFIG_700D) || defined(CONFIG_EOSM) || defined(CONFIG_6D)
uint32_t raw_write_chan = 0x12;
#endif


I'm not an expert in this, but I'm wondering why some camera addresses are written as 0xXX and for other camera's like the 5d3 for example as a plain normal number:

/* for other cameras, find a free channel with find_free_edmac_channels  */
#ifdef CONFIG_5D3
uint32_t raw_write_chan = 4;
#endif


I'm wondering if I could brick my camera when trying out a normal written number address for the 6d like:
#ifdef CONFIG_6D
uint32_t raw_write_chan = 8;
#endif


eNnvi mentions that the scrambled/pink frames problem could also be due to wrong edmac flag settings.
We tried both options for 6d, the first one gives pink frames, the second gives scrambled frames.
/* see wiki, register map, EDMAC what the flags mean. they are for setting up copy block size */
#if defined(CONFIG_650D) || defined(CONFIG_700D) || defined(CONFIG_EOSM) || defined(CONFIG_6D)
    uint32_t dmaFlags = 0x20000000;
#else
    uint32_t dmaFlags = 0x20001000;
#endif


So anybody got suggestions how to get this working, and is it safe to try out some different free edmac channels, or will it brick my camera when I pick the wrong one?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 22, 2016, 10:08:03 AM
It is safe to try it out. Both settings won't brick your camera. I would try out one after another not both at the same time. For finding free channels you can compile this (https://bitbucket.org/hudson/magic-lantern/branch/edmac). Check the latest commits. a1ex has rewritten find free edmac channels. So should be no problem for you to find other unused ones.

For dmaflags I recommend doing it like seen on unified branch:
https://bitbucket.org/hudson/magic-lantern/src/4f72a0eabb16c87ace571f494e588598b467c157/src/edmac-memcpy.c?at=unified&fileviewer=file-view-default#edmac-memcpy.c-401
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 22, 2016, 10:22:21 AM
Near LVx1_StartMem1Edmac: channel = 0x12, flags = 0x60000000.

The meaning of this flag is not really known, but this benchmark gives some insights:
5D3 edmac_memcpy (see also transfer size from Register Map (http://magiclantern.wikia.com/wiki/Register_Map#EDMAC)):
Code: [Select]
0x40001000 => 653 MB/s (16 bytes per transfer)
0x20001000 => 625 MB/s (8 bytes)
0x40000000 => 585 MB/s (4 bytes)
0x20000000 => 380 MB/s (2 bytes)
0x60000000 => 585 MB/s (same as 0x40000000?)
0x60001000 => 653 MB/s (same as 0x40001000?)

So, I'd expect it to work with both 0x60000000 and 0x40000000. Can you confirm?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 10:35:18 AM
Alex, are you asking me to try those values on the 6d ?

So instead of:
uint32_t dmaFlags = 0x20000000;
Try out:
uint32_t dmaFlags = 0x60000000; (or  uint32_t dmaFlags = 0x40000000;

@NikFreak
Quote
It is safe to try it out. Both settings won't brick your camera.
That's all I need to know  :D
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 22, 2016, 10:36:02 AM
Nice code insights. Whats the synonym for LVx1_StartMem1Edmac on 7D.203? Something around FF27E924 with dmaflag 0x40000000?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 22, 2016, 11:18:03 AM
StartPass_x1 CrawAddr : %lx / KindOfCraw : %d.

Old models appear to require some more changes though...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on December 22, 2016, 11:48:57 AM
18 dec 113 build first tests:

mlv 10 bit 39 fps - wrong black level
mlv 10 bit 24 fps - ok
raw rec 10 bit 39 fps - ok

crop mode (5x zoom) lv doesnt work

the last build whithin the crop_rec module doesnt have issues whith lv, but had another problems such as bad first frame and mlv_rec corrupt frames
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on December 22, 2016, 12:06:17 PM
18 dec 113 build first tests:

mlv 10 bit 39 fps - wrong black level
mlv 10 bit 24 fps - ok
raw rec 10 bit 39 fps - ok

crop mode (5x zoom) lv doesnt work

the last build whithin the crop_rec module doesnt have issues whith lv, but had another problems such as bad first frame and mlv_rec corrupt frames

After restarting, removing the card, 5x crop mode lv start to work..?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2016, 07:48:20 PM
Quote from: Levas
I'm not an expert in this, but I'm wondering why some camera addresses are written as 0xXX and for other camera's like the 5d3 for example as a plain normal number:

The address can be in either hex or decimal. Near the top of raw.c look for this line:

Code: [Select]
#undef RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */

and change it to #define and you'll be able to try different addresses (in hex) from the Debug menu. Read the comments on this merged pull request for more information:

https://bitbucket.org/hudson/magic-lantern/pull-requests/780/increase-raw_debug_type-limit/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 07:59:16 PM
I wanted to do some testing, but can't get it to work.

I've downloaded the latest magic lantern 'raw_video_10bit_12bit' branch from https://bitbucket.org/hudson/magic-lantern
And I've got an altered 'raw.c' file and an altered 'edmac-memcpy.c' file from eNnvi, which I put in the latest version of raw_video_10_12bit branch, I overwrite the original raw.c and edmac-memcpy.c files.

Then I go to platforms and make a new build for the 6d.
But when I test this new build, live view freezes... (Although the recorded 10bit MLV files give good frames)
While eNnvi got me some builds where live view doesn't freeze ?
Am I doing something wrong here, as far as I understand the only two files altered for using RAW SLURP are the raw.c and the edmac-memcpy.c files ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2016, 08:53:46 PM
You also need internals.h for the 6D to define the slurp thing--away from computer at the moment.


Sent from my iPhone using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 09:13:57 PM
Thanks, that was the missing piece. Compiled a build with working live view myself.
Now it's time to do some testing  8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 09:22:22 PM
@Alex
I can conform that using
uint32_t dmaFlags = 0x40000000

gives the same results as
uint32_t dmaFlags = 0x20000000

pink garbage frames  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 22, 2016, 09:46:49 PM
@dfort

I did the #define thing on RAW_DEBUG_TYPE
But I get this error with compiling
../../src/raw.c:2113:18: error: 'lv_raw_type' undeclared here (not in a function)
         .priv = &lv_raw_type,
                  ^
make: *** [raw.o] Error 1


Tried some different edmac channels, changing the edmac-memcpy.c, but so far, no luck.
Got one black live view and one really slow live view, like 5 fps or something.
Would be a lot quicker if you can define it in the debug menu.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on December 22, 2016, 10:32:52 PM
@Walter Schulz

I want to take de risk. I can test in the real world. I have a film production company and we work everyday with Canon (5D, 7D, 6D), Sony and Black Magic.

I don't have skills and time to compile. But time to test when I film (not trees and cats, of course ;)).

If you or other dev can share the compile, I will test.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 22, 2016, 10:35:43 PM
You need to give it a starting value. Any value will do because you can change it. Look at the 60D as an example.


Sent from my iPhone using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 22, 2016, 11:16:50 PM
...
pink garbage frames  :P

Then try 0x60000000.

StartPass_x1 CrawAddr : %lx / KindOfCraw : %d.

Old models appear to require some more changes though...
Finding DEFAULT_RAW_BUFFER for 7D causes some headaches. Cannot find anything EVF related in the 7D ROM though other DIGIC IV like 60D/600D are no problem regarding that. can you help in finding DEFAULT_RAW_BUFFER for 7D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 22, 2016, 11:21:38 PM
Sure, reply #754. I already tried, without success.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 03:46:37 AM
@Levas did you get it to compile with "#define RAW_DEBUG_TYPE" ?

What I was trying to tell you on my cell phone is to look for this:

raw.c
Code: [Select]
#ifdef CONFIG_60D
#define PREFERRED_RAW_TYPE 5
#endif

And do the same for the 6D. Then go to the Debug menu to try different values:

(https://c3.staticflickr.com/1/766/31229380130_7893122693.jpg)

Start with 0x1 and go up from there. After a while you'll get repeating results. You might want to look at the playback in camera with mlv_play and also load raw_twk if you're using 10/12 bit options.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 08:27:17 AM
Thanks @Dfort
I already got the debug option working, but ran out off battery...

It's a new day and got a fully loaded battery 8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 11:34:47 AM
Ok, I tested a lot today and allmost  all I saw were frames that could be described as a pink mess  :P
There is one setting which gives me scrambled pink frames, so still a mess, but you can recognize some stuff in it.
Free edmac channel
 uint32_t raw_write_chan = 0x12;
and dma flag
  uint32_t dmaFlags = 0x20000000;

But I also find some resolution settings furtheron in the raw.c file.
Could it be that the scrambled picture is caused by faulty settings in this part of raw.c

#ifdef CONFIG_EDMAC_RAW_SLURP

    /* params useful for hardcoding buffer sizes, according to video mode */
    int mv = is_movie_mode();
    int mv720 = mv && video_mode_resolution == 1;
    int mv1080 = mv && video_mode_resolution == 0;
    int mv640 = mv && video_mode_resolution == 2;
    int mv1080crop = mv && video_mode_resolution == 0 && video_mode_crop;
    int mv640crop = mv && video_mode_resolution == 2 && video_mode_crop;
   
    /* note: 6D reports 129 = 0x81 for zoom x1, and it behaves just like plain (unzoomed) LiveView) */
    int zoom = (lv_dispsize & 0xF) > 1;

    /* silence warnings; not all cameras have all these modes */
    (void)mv640; (void)mv720; (void)mv1080; (void)mv640; (void)mv1080crop; (void)mv640crop; (void)zoom;

    #ifdef CONFIG_5D3
    /* don't know how to get the resolution without relying on Canon's lv_save_raw */
    *width  = zoom ? 3744 : mv720 ? 2080 : 2080;
    *height = zoom ? 1380 : mv720 ?  692 : 1318;    /* height must be exact! copy it from Debug->EDMAC */
    return 1;
    #endif

    #ifdef CONFIG_60D
    *width  = zoom ? 2520 : mv640crop ? 920 : mv720 || mv640 ? 1888 : 1888;
    *height = zoom ? 1106 : mv640crop ? 624 : mv720 || mv640 ?  720 : 1182;
    return 1;
    #endif

    #ifdef CONFIG_600D
    *width  = zoom ? 2520 : mv1080crop ? 1952 : mv720  ? 1888 : 1888;
    *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
    return 1;
    #endif
   
    #if defined(CONFIG_650D) || defined(CONFIG_700D)
    *width  = zoom ? 2592 : mv1080crop ? 1872 : mv720  ? 1808 : 1808;
    *height = zoom ? 1108 : mv1080crop ? 1060 : mv720  ?  720 : 1190;
    return 1;
    #endif

    #ifdef CONFIG_EOSM
    *width  = video_mode_crop ? 1872 : 1808;
    *height = video_mode_crop ? 1060 : 727;
    return 1;
    #endif

    #ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ?  634 : mv720 ?  720 : 1208;    /* find correct mv720 height -- must be exact! */   
    return 1;
    #endif
/ silent picture in 1080 mode = 1830 x 1224 /
/ silent picture in 720 mode = 1830 x 634 /
/ silent picture in zoom mode = 2688 x 960 /
/ image buffers 1080 mode (in debug menu) shows 720 x 480, 1808 x 1206 /
/ Some EDMAC channels show 3360 x 1207 or 3616 x 1205 /
/ image buffers 720 mode (in debug menu) shows 720 x 480, 1280 x 720 /
/ Some EDMAC channels show 2560 x 720 or 2560 x 719 or 3360 x 719 /
/ image buffers in 1080 zoom are 720x480, 1104x736 /
/ Some EDMAC channels show 2208 x 736 or 2208 x 735 or 4844 x 633 /
   
   #ifdef CONFIG_7D
    *width  = zoom ? 2520 : mv1080crop ? 1952 : mv720  ? 1888 : 1888;
    *height = zoom ? 1106 : mv1080crop ? 1048 : mv720  ?  720 : 1182;
    return 1;
    #endif

    /* unknown camera? */
    return 0;


For 6d I tried some different settings, but I'm not sure which resolutions to put where, so I added some text in the file which shows resolutions of silent pics and image buffers and edmac channels.
Does somebody have an idea which numbers to use ?

    #ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ?  634 : mv720 ?  720 : 1208;    /* find correct mv720 height -- must be exact! */   
    return 1;
    #endif
/ silent picture in 1080 mode = 1830 x 1224 /
/ silent picture in 720 mode = 1830 x 634 /
/ silent picture in zoom mode = 2688 x 960 /
/ image buffers 1080 mode (in debug menu) shows 720 x 480, 1808 x 1206 /
/ Some EDMAC channels show 3360 x 1207 or 3616 x 1205 /
/ image buffers 720 mode (in debug menu) shows 720 x 480, 1280 x 720 /
/ Some EDMAC channels show 2560 x 720 or 2560 x 719 or 3360 x 719 /
/ image buffers in 1080 zoom are 720x480, 1104x736 /
/ Some EDMAC channels show 2208 x 736 or 2208 x 735 or 4844 x 633 /
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 12:12:47 PM
So envvi guided you and gave you some code snippets? Where did you guys get

Code: [Select]
#ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ?  634 : mv720 ?  720 : 1208;    /* find correct mv720 height -- must be exact! */   
    return 1;
    #endif

from? You need to call lv_save_raw to read the values from "Show EDMAC". The value for width shown there must be undertaken a math. the formula is:

shown width * 8 / 14


You can replace this in debug.c

Code: [Select]
static void run_test()
{
call("lv_save_raw", 1);
}

and just call it afterwards from "Don't click me" for all video / zoom modes. the shown height in edmac needs to be added +1. For e.g. if it shows 1205 then it is 1206. Height values isn't alowed to be odd. It needs to be an even value.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 12:36:36 PM
Thanks Nikfreak, I'll try the debug.c and dont click me method.

eNnvi tried to make soms builds for the 6d.
Only they didn't work perfect.
I asked eNnvi for the raw.c and edmac-memcpy.c file so I could test some stuff.
eNnvi doesn't have a 6d, but seems to know somebody else with a 6d who also tested the builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 23, 2016, 12:54:56 PM
Sorry giusto been really busy at work these data, this evening i'll try to get the help of levas and find the right width and height buffer pointers.

I got those data looking at the 6d edmac while recordman and the width from the silent pic.

Now i realize that was not the best way to do that. Levas i'll text you via PM when i'll be back home
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 12:57:42 PM
probably the values are ok for mv720 and 1x zoom but definitely wrong height for the zoom mode. I would recheck all of them in all modes as explained above.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 01:36:33 PM
Original values in the file from eNnvi were
*width  = zoom ? 2768 : mv720 ? 1920 : 1920;
*height = zoom ?  736 : mv720 ?  720 : 1208;

Don't know if I did it right, cause changing the debug.c and pushing the don't click me didn't change the values in the show edmac screens  :-\
But these values I did found in one edmac channel:

zoom
4844 x 987 which translates too (2768 x 988)

1080mode
3360 x 1251  which translates too  (1920 x 1252)

720mode
3360 x 662  which translates too  (1920 x 662)

So my raw.c file is now like this (which still gives scrambled frames, but could be due other issues like free edmac channel etc.)
#ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ? 988 : mv720 ?  662 : 1252;    /* find correct mv720 height -- must be exact! */   
    return 1;
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 01:53:46 PM
 :o
I tried the latest resolution values with this dma flag in edmac-memcpy.c

uint32_t dmaFlags = 0x60000000

And looks like we have working live view and NORMAL FRAMES on 6d 
 :D :D :D :D :D :D :D :D :D :D :D

I'll test some more, to be really sure it works  ;D

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 02:16:20 PM
Yep, 10 bit recording works flawlessly on the 6d, with live view :D

Although with 12 bit recording something weird is still going on (as mentioned before in this topic).
with 12 bit recording MLV_dump finds a lot of cold pixels and the image has specles all over it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 02:26:31 PM
congrats Levas. Was worth all your efforts and guess it feels great making yourself a xmas gift. Submitting a PR @bitbucket will make it perfect  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 23, 2016, 02:26:53 PM
Wow. Beautiful work on the 6D everyone and Levas :).

Maybe needless to say but are you disabling cold pixel and vertical stripes settings in mlv_dump?
http://www.magiclantern.fm/forum/index.php?topic=5601.msg176263#msg176263
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 02:40:54 PM
@Alex
uint32_t dmaFlags = 0x40000000
Works too, so before it didn't probably work because of the wrong resolution settings.

@Nikfreak
Submitting @bitbucket, what do you think I am, a developer  :P
And my coding skills aren't that good,in order to use the right dmaFlags my version, for sure, won't work for some other cams that worked before ;D

@Danne
You're right, disabling stripe fix and cold pixel fix gives good frames.
But I always thought the stripe fix was nessecarry  ?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 23, 2016, 02:46:17 PM
Could you also try with edmac flags of 700D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 03:33:31 PM
@eNnvi
I've tried the following dma flag as used for 700d
uint32_t dmaFlags = 0x20000000

And it causes the 'RAW DETECT ERROR' and when it will record it gives the scrambled frames.

So the only settings that were wrong in your build are the height value settings for the different video modes (width values were all good)
#ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ? 988 : mv720 ?  662 : 1252;    /* find correct mv720 height -- must be exact! */   
    return 1;
#endif

And the dma flag setting, which must be either
uint32_t dmaFlags = 0x40000000 or uint32_t dmaFlags = 0x60000000
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 03:46:51 PM
So, confirmed (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176849#msg176849)!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 03:51:02 PM
Yes confirmed, both values work
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on December 23, 2016, 03:59:31 PM
Is somebody willing to do a pull request for this for the 6d?

I can send the pieces of code that needs to be altered/added in a pm.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 04:40:05 PM
Is somebody willing to do a pull request for this for the 6d?

Sure, no problem. Let's get some developers reviewing the changes you and eNnvi made and post a test build for other 6D users.

@nikfreak -- do you have a branch in your repo that I can use to make test builds for the 70D and 100D?

Off topic -- Just a suggestion. It would be great to get those platforms merged and perhaps prune the branch for the 1100D if that platform is no longer supported.

[EDIT]
Back on topic -- so if CONFIG_EDMAC_RAW_SLURP was working on the 1100D at one time why was it removed? Seems like that camera could really use the 10bit/12bit treatment though at this point it doesn't work at all with raw video.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 23, 2016, 06:36:28 PM
just updated the 100D PR.
70D PR should already contain all relevant info needed as far as I remember.

Wishing ya fun.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on December 23, 2016, 07:49:24 PM
You guys are amazing with the improvements upon the great foundations of Magic Lantern.
Thanks to all and congratulations on this new breakthrough!

I use the 1.23 firmware so finally I had a little time to test it with magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123.zip (great work dfort)
WOW! WOW! WOW!
I tested the mlv_rec and raw_rec with the raw_twk.  Great accomplishments here dfort, g3gg0, and all contributors!

Video quality:  I cannot see any difference at iso 6400 or at iso 100.  All of them look superb on 10/12/14 bits.
Video size: Amazing 40% lighter dng frames going from 14 bits to 10 bits.  Pixel peeping the dngs at 300% show no difference between 10/12/14 bit!
Color and Exposure are identical.  Bumping exposure by 3ev on Photoshop does not bring up any differences on shadows.

Standard Max Recording Resolutions 5D3 with 64 GB Lexar 1066:
Origninal mlv_rec + spanning + Auto preview + sound at 1920x1080 30fps same with crop module: 17 sec max recording
Original mlv_rec + Auto preview + sound at 1920x1080 30fps with crop module: 17 sec max recording
Original raw_rec + Auto preview at 1920x1080 30fps same with crop module: 23 sec max recording

10/12/14 BIT TESTING
With magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123
Note: I cannot find the card spaning on new mlv_rec

14 bit
mlv_rec + canon preview + sound at 1920x1080 30fps same with crop module: 16 seconds recording time
raw_rec + canon preview 23 sec at 1920x1080 30fps: 23 seconds recording time
raw_rec + canon preview 32 sec at 1920x1080 30fps at 5x croped: 35 seconds recording time!! dont know why the croped records faster.

12 bit
mlv_rec + Auto preview + sound at 1920x1080 30fps: continuous recording
mlv_rec + Auto preview + sound at 1920x1080 30fps at 5X cropped: continuous recording
Max continuous recording mlv_rec + Auto preview + sound at 1952x1098 or smaller at 30fps

raw_rec + canon preview 23 sec at 1920x1080 30fps: continuous recording
raw_rec + canon preview 32 sec at 1920x1080 30fps at 5x croped: continuous recording
Max continuous recording raw_rec Continuous recording with 2000x1126 30fps or smaller resolutions.

note: mlv_play crashes (play quits and camera has some error since the red led keeps blinking) on crop video larger than 1968x1108

10bit
Max continuous recording mlv_rec + Auto preview + sound at 2144x1206 or smaller resolutions at 30fps.
Max continuous recording raw_rec Continuous recording with 2208x1242 or smaller resolutions at 30fps.

Although I do not contribute to the forum due to time constrains I do come here all the time and look for all the updates on 5D3 1.23.  I use my 5D3 often and take lots of bird photos and RAW video.
I truly enjoy all of the amazing things you guys accomplish.  It is stunning!
THANK YOU!  THANK YOU!  THANK YOU!
Merry Christmas and happy new year to all!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on December 23, 2016, 07:55:26 PM
pull request for 6D done.

now i'll try to work in 7D (i like the hard things ;) ). As i don't own one i have to ask if anybody want to help me (possibly having that camera :D )
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 08:07:17 PM
Looks like @a1ex already merged the 6D pull request, that was fast.

There's also another change I didn't see coming. The 700D now has "#define PREFERRED_RAW_TYPE 0x10" while the 650D still has "#define PREFERRED_RAW_TYPE 78" commented out. They both used to be commented out. What's the reason for that?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on December 23, 2016, 08:11:56 PM
Back on topic -- so if CONFIG_EDMAC_RAW_SLURP was working on the 1100D at one time why was it removed?

It wasn't removed. There are some missing bits from the raw branch (such as skip offsets), so the branch wasn't merged. Start reading from here (http://www.magiclantern.fm/forum/index.php?topic=1009.msg119468#msg119468); some of those bits are already there and just need to be put together in a commit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 08:45:11 PM
Thanks for the tips. Danne has access to an 1100D so we might team up and work on it, though it isn't a high priority for either of us.

Just noticed the 1100D build was broken for about one week, and nobody reported the error.
Is there anybody still using this camera, or should I discontinue this port?

So it wasn't intentional?@dmilligan also has one but it looks like he isn't using it much these days - and there's a math problem:

The 1100D nightly build was intentionally disabled because a1ex noticed some incorrect values for FPS override timers. I've never cared enough to try and fix it (the Maths involved hurt my head :P ), I just compile it with the old incorrect values.

Getting the 1100D working isn't my goal, I'm just trying to get a better understanding of ML and looking at CONFIG_EDMAC_RAW_SLURP on that camera opened this can of worms. In any case, years ago I did some beta testing and the company sent me their lowest end system. Why not their most powerful? Because they wanted to make sure their minimal configuration was still valid or if they should discontinue support for it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 23, 2016, 08:59:54 PM
6D added to the test builds:

https://bitbucket.org/daniel_fort/magic-lantern/downloads

Make sure to load the raw_twk module if you want in camera playback of 10/12 bit files.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 24, 2016, 12:40:39 AM
just updated the 100D PR.
70D PR should already contain all relevant info needed as far as I remember.

Thanks. 100D was pretty much ready. Merged the 70D_merge_fw112 branch with raw_video_10bit_12bit.

Test builds for 100D and 70D posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on December 24, 2016, 01:20:47 AM
So it wasn't intentional?
1100D has been broken multiple times in the past, sometimes on purpose, sometimes on accident. It is currently broken on purpose. The 1100D was probably the easiest platform to break on accident because of its unusual combination of CONFIG options enabled (so not being careful about #ifdefs would be more likely to result in code that won't compile without being noticed, for example 1100D could do QR raw but not LV raw, but there's a lot of shared code between the two).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on December 24, 2016, 02:19:27 AM
[quote Merged the 70D_merge_fw112 branch with raw_video_10bit_12bit.

Test builds for 100D and 70D posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
[/quote]

i wil be able to test 70D tonight after reaching home. Thank q dfort. I have been waiting for so long
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 24, 2016, 09:04:52 AM
70D tried a quick test.

1832@ 2.35 apsect, 24fps 10bit, could record for over 1 minute (probably continuous), the movie review but it never builds the movie because it hangs/bricks requiring battery pull and two resets to get right.
1832@ 2.35 aspect, 30fps 10bit, could record for 18s, but same hang/brick.
1832@ 2.35 aspect, 24fps 12bit, could record for 28s, but same hang/brick. 
1846@ 2.35 aspect, 60fps 12bit, 3X mode on, could record for 9s, but same hang/brick.
1920@ 2.35 aspect, 24 fps 10bit, 5X zoom, could record for over 1 minute (probably continuous), two second in got a "hack error" but it kept on recording until I hit stop rec.,  but same hang/brick occurs. Not sure with hack error to trust the record time.
2220 @2.35 aspect, 24 fps 10bit, 5Xzoom,  hack error at 0 sec kept recording over 1 minute, but when I switched to photo mode it showed 51 sec record, so I cant trust either number.

It did produce MLV files, but Mystic could not convert any MLV to DNG.  This same program worked with my 600D 10/12 bit MLV files.
Will try mlv_dump on the files once I have more time. 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Oswald on December 24, 2016, 09:54:03 AM
Live view still freezes with 100d. I just tested it. I downloaded the @dfort build.  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on December 24, 2016, 11:40:12 AM
dfort probably didn't enable edmac_raw_slurp before compiling for 100d. Just a guess as I didn't try myself.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 24, 2016, 01:19:51 PM
@dfort and @reddeercity for your help to make 5D Mark II even better, which build is the best so far Dec1, Dec12, or Dec18  I never got Dec9? I had some corruption with Dec18 but I'm going to do some more test, also will there be a update build from @dfort for 5DMarkIi soon?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 24, 2016, 08:09:55 PM
dfort probably didn't enable edmac_raw_slurp before compiling for 100d. Just a guess as I didn't try myself.

@nikfreak - Could you make a 10bit_12bit branch in your repository that includes the 70D and 100D? I don't have these cameras available for testing so I can't be sure that I put all the pieces in the right place. Just a thought--maybe do your 70D and 100D pull requests on the raw_video_10bit_12bit branch? Seems that this is where all the action is happening and since this is a development branch maybe the developers with the merge key are more receptive to adding new platforms here as opposed to the "stable" unified branch?

@dariSSight - The 5D2.212 won't be much better with 10bit/12bit until CONFIG_EDMAC_RAW_SLURP is working on that camera. Seems to be a matter of finding a free edmac channel and sniffing out where the DEFAULT_RAW_BUFFER is located -- it is different on every camera.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 24, 2016, 08:49:20 PM
Looks like there's lots of requests for crop_rec with the 5D3.113. Since that platform seems to be well tested with 10bit/12bit I put up a test build that includes crop_rec:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit_crop_rec.2016Dec24.5D3113.zip

The 5D3.123 test build also includes the crop_rec module.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ewinemiller on December 24, 2016, 09:56:13 PM
The 5D3.123 test build also includes the crop_rec module.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2016Dec20.5D3123.zip

Hey dfort.

I appreciate the builds, but if I try your 5D3 1.23 build from the 20th, it won't load the modules. I get failed to link modules error.

Regards.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Camerite on December 24, 2016, 10:30:47 PM
The 5D3 123 build seems to load all modules but crop rec. When activated the camera gives errors on all modules.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kamranjon on December 24, 2016, 10:59:41 PM
More than happy to help provide testing on the 7D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 24, 2016, 11:27:20 PM
I appreciate the builds, but if I try your 5D3 1.23 build from the 20th, it won't load the modules. I get failed to link modules error.

Strange, check reply #862 from @RenatoPhoto

More than happy to help provide testing on the 7D.

There is a test build posted but like the 5D2 it needs the CONFIG_EDMAC_RAW_SLURP treatment for it to work properly.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vstrglv on December 25, 2016, 08:24:33 AM
Looks like there's lots of requests for crop_rec with the 5D3.113. Since that platform seems to be well tested with 10bit/12bit I put up a test build that includes crop_rec:
https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit_crop_rec.2016Dec24.5D3113.zip
Thank you very much! Quick test - frame issue:
10bit,12bit, crop1:1 and not crop mode - preview framing during recording  is not correct. All screen is filled with image (there are no horizontal black stripes).
Sometimes this issue takes place in 14bit mode too. I could not find a regularity.   
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on December 25, 2016, 10:57:01 AM
tried dfort's test build for 70D but no luck
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ewinemiller on December 25, 2016, 03:54:16 PM
Strange, check reply #862 from @RenatoPhoto

It's crop_rec.mo. If I enable it, no modules load. I don't think he tested that. Looks like he was using the classic crop stuff.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on December 25, 2016, 04:43:57 PM
Guys!  I did not use crop.rec!!!
@dfort

I used the standard crop mode using the magnify button..  So it works on crop mode.
crop.rec is and advanced version of crop mode that centers the images and has other nice features.
It is very good but is not incorporated into unified.. it is a separate branch.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vstrglv on December 25, 2016, 04:54:24 PM
Thank you very much! Quick test - frame issue:
10bit,12bit, crop1:1 and not crop mode - preview framing during recording  is not correct. All screen is filled with image (there are no horizontal black stripes).
Sometimes this issue takes place in 14bit mode too.
This issue takes place only for mvl_rec mode. If preview options is set to "ML Grayscale"  this issue disappears and  mode is colour during  recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on December 25, 2016, 05:55:15 PM
@dfort
Also forgot to mention that at the start/stop of recording 5D3-123 with mlv_rec moldule I get this message:
Hack error at 0:
excpectede3a00032, got e1a00000
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 26, 2016, 09:10:21 PM
@g3gg0 updated mlv_dump and put up binaries for Windows, Mac and Linux so I removed the mlv_dump builds I was hosting. These new builds work with 10bit/12bit files.
http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip

Some of you are posting links to the test builds which is fine but just a warning that I've been removing old builds to avoid confusion. That means that you'll need to save previous versions in case something breaks or you want to make a regression test.

There have been some reports that things aren't working properly on a few cameras that should be working. It might be the way I'm compiling. In most cases I'm just compiling the raw_video_10bit_12bit branch but in others I merged crop_rec and other branches. Maybe I should be merging unified too since eventually the goal is to get this into the nightly builds?

Here are the latest download counts of the test builds. Checkmarks are for cameras that are working with CONFIG_EDMAC_RAW_SLURP:

5D3.123-76* with crop_rec (Dec 20)
5D3.113-55* without crop_rec (Dec 18)
5D3.113-51* with crop_rec (Dec 24)
6D.116-24* new build works with 10bit/12bit (Dec 23)
5D2.212-21
7D.203-15
EOSM.202-14
100D.101-14* might have build issues? (Dec 23)
600D.102-13
70D.112-13* might have build issues? (Dec 23)
700D.114-12
60D.111-10
650D.104-10
1100D.105-9* allocate_raw_buffer branch has CONFIG_EDMAC_RAW_SLURP working but not 10bit/12bit (Dec 19)
550D.109-8
50D.109-6
500D.111-3

Although this might not be an accurate list of cameras sorted by popularity, it does show quite a bit of interest in 10bit/12bit raw video from users of the higher end and full frame cameras. I was surprised by the amount of interest in the 5D3.123--looks like users want raw video and clean HDMI output.

There a few older models left to solve. Check out reply #754 for some hints:

From what I've tried so far on those old models (without much success yet), we have at least two problems:
- the raw buffer is not allocated all the time (as it's done on the new models) -> easy to fix, see the 1100D raw branch
- the vsync hook currently used doesn't appear to be the best for getting a raw image
- the exact behavior of EDMAC in case of input/output dimension mismatch is not very well known

Solving those models may require either some serious low-level research, or a great dose of luck. I'm just scratching the surface here (http://www.magiclantern.fm/forum/index.php?topic=18315).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lars Steenhoff on December 27, 2016, 12:53:14 AM
for monitoring with an external monitor clean hdmi is quite nice.

and as a bonus the zoom mode wich shows in black and white with the 5x crop mode is showing a preview on the external monitor in color. ( last time i checked wich is a while ago)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Quentin on December 27, 2016, 08:23:16 AM
How fast in terms of FPS/Frame size is 12/10 bit RAW recording ?
I tried to follow the thread but its since 2013 and was inmature.
Any clues ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 27, 2016, 08:39:01 AM
You can use the usual formula to check if it fits in your camera controller's speed limit:
W * H * BD * FPS / 8 /1024 / 1024
(W &  H — frame width and height in pixels, BD — bitdepth, which now can be 10 or 12 instead of 14, FPS is frames per second, and converting to megabytes per second). Result is required speed in MB/s

For example, for 650D we have a speed limit in 40MB/s.
We wish to film 10 bit raw video 1728x736 at 25fps.
1728*736*10*25/8/1024/1024 = 37.9 MB/s which fits fine in controller's limit. So in practice we can shoot continuously with these parameters.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Quentin on December 27, 2016, 08:42:59 AM
Thank you very much  :)
Its reasonable. Cant wait to try it out.
ML is so cool
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 28, 2016, 04:12:34 PM
On my 70D MLV files tried the latest mlv_dump, all were failed.  This agrees with mystic not being able to convert any of the same MLV.

G:\DCIM\100CANON>mlv_dump --dng M23-2332.MLV

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M23-2332.MLV'
   - Convert to DNG frames
   - Output into 'M23-2332_'
File M23-2332.MLV opened
File M23-2332.M00 not existing.
Processing...
[ERROR] Invalid block size at position 0x00000400
Processed 0 video frames
Done

my 600D MLV works fine with same program.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 28, 2016, 04:36:28 PM
@nikfreak -- Need your help with the 70D. Could you create a 10bit/12bit branch in your repository with the appropriate merges? I got it to compile but I must have missed something. Don't have a 70D here to check things out. It would be great if you could do the same for the 100D.

Thanks!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: maxotics on December 28, 2016, 04:54:39 PM
I'm happy to test the 12/10 bit on my 7D.  But would need instructions.  I can't make complete sense of it reading threads.  Again, only if you need someone to test.  I can wait ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on December 28, 2016, 05:53:00 PM
600D, 1280 @2.35, 24 fps, 10 bit. 
I am getting pink frames (at top 25%) every other frame.
What's this about?
By bit depth & rate calculation it should be continuous.
My card is Sandisk Ex. Pro, 95 Mb/s so its not the card speed.


Reformat card, loaded latest 600D, all is good. Was able to get 1920@2.35/24fps  for about 2 seconds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: whyfy on December 28, 2016, 06:49:16 PM
Canon 60D 10Bit RAW video, Overexposure testing
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 29, 2016, 08:53:51 AM
I'm happy to test the 12/10 bit on my 7D.

This isn't working on the 7D yet. A few people are looking into it. The hard part seems to be finding the memory location for DEFAULT_RAW_BUFFER.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 29, 2016, 09:44:23 AM
maxotics check out 10 bit raw on your eosm. I saw your vids before on youtube and vimeo. and its one of the main reasons i even got an eosm. theres not much info out there..you had some of the only raw vids available to watch. 10 bit raw on eosm is really great. give it a try.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 29, 2016, 09:51:02 AM
@quentin check this out http://rbrune.github.io/mlraw/ (http://rbrune.github.io/mlraw/) noticed it has settings for 10 bit and 12 bit now:)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on December 29, 2016, 01:27:49 PM
Ones again. Tremendous work devs! Really appreciate the development.

If I could wish for the coming build, since I can't program, or have any code skills (I only make hardware and such mods). When shooting MLV instead of RAW I get these problems:


I know I can't ask any of this to be done but it would be swell!

Best,
David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: maxotics on December 29, 2016, 03:26:38 PM
Thanks Teamsleepkid, was wondering about that.  Will try it on the EOSM.  It always warms my heart when someone says, "Hey Max, you turned me onto Crack, thanks!" :)  Is there an instruction page anywhere, hard for me to piece together from threads.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on December 29, 2016, 04:46:09 PM
I just tried dfort's 12/18 7D build and 10 and 12 bit are working fine (but with grayscale preview frozen if used). What is not supposed to be working?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: snipeua on December 29, 2016, 07:28:33 PM
Dfort please make a new build at 1.13, the same as you did at 1.23! Thank you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on December 29, 2016, 08:13:27 PM
Max there's a download a few pages back like on page 35 it's dforts build. Hope I'm not stepping on his toes:) it's a ml folder you've done it before.. install magic lantern. Then in the settings under raw it says 10 bit 12 bit. I use 10 bit because it's higher resolution. For post I think you have to use mlvfs or mlv dump? I have no idea about other software I use mlvfs works great. Might be more addicting than crack:)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 29, 2016, 10:36:28 PM
Dfort please make a new build at 1.13, the same as you did at 1.23! Thank you!

What's the problem with this build? It should have the same features as the 5D3.123 build I recently posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit_crop_rec.2016Dec24.5D3113.zip

...it's dforts build. Hope I'm not stepping on his toes:)...

No worries. We're all sharing our experiences.

...What is not supposed to be working? ...grayscale preview frozen if used...

Just rearranged your comment to answer your own question. Ok, there's more to it than that. Apparently 5D2 and 7D users are reporting some success with 10bit/12bit even though all the pieces aren't quite finished yet.

Thanks Teamsleepkid, was wondering about that.  Will try it on the EOSM.  It always warms my heart when someone says, "Hey Max, you turned me onto Crack, thanks!" :)  Is there an instruction page anywhere, hard for me to piece together from threads.

@maxotics -- I'm another one who saw your raw video tests on the EOSM and got me interested in Magic Lantern. To try out 10bit/12bit from the test builds I posted turn on the mlv_rec and/or raw_rec modules. raw_rec is actually MLV Lite in this branch so you'll end up with MLV files instead of the RAW files you were used to when you were doing your tests. For processing the 10bit/12bit MLV files use either MLVFS (http://www.magiclantern.fm/forum/index.php?topic=13152.0) or g3gg0's new mlv_dump builds (http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip). If you want to record sound with mlv_rec you can turn on mlv_snd but note that you still can't record audio with raw_rec. You can also playback in camera by loading mlv_play but for 10bit/12bit playback you'll also need to load raw_twk. Hope this helps.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 29, 2016, 11:03:53 PM
Audio won,t work with raw_twk module on the eos m. You cannot delete video other than from file manager with raw_twk enabled. Other than that Eos M is one hell of a little beast with 10bit enabled. I use it a lot nowadays.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domasa on December 30, 2016, 12:06:31 AM
Quote
W * H * BD * FPS / 8 /1024 / 1024
(W &  H — frame width and height in pixels, BD — bitdepth, which now can be 10 or 12 instead of 14, FPS is frames per second, and converting to megabytes per second)

Is it right? Where is BD for second and third color? (R-G-B)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: jpegmasterjesse on December 30, 2016, 12:18:17 AM
I'm having some success with my 5d2 and the December 19th builds.

MLV 10 BIT in Crop Mode is working perfectly minus frozen preview.
GIF: https://gfycat.com/ReflectingMadCod

However standard recording (not in crop mode) every other frame contains part of the 1st frame.
GIF: https://gfycat.com/ThoseObeseAllosaurus
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on December 30, 2016, 12:41:07 AM
So on 7D I can get the Canon preview to work if I have preview set to "auto" and before I press record I cycle through the various displays using the "info" key and end on Canon preview. This is while using the 5x crop mode (not sure about non-crop yet).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 01:14:37 AM
Hello everyone,

     I tested 10 bit Raw Video recording on a Canon T3i. I was able to record 1280 x 544 for 25 minutes on a 32 GB Sandisk Extreme Pro 95 MB/SEC SD Card.

     One pink frame was recorded on the 6th DNG, and there are pink, green, blue tints, on the highlights of a fan I recorded.

     However, they aren't on every dng frame, it seems every other.

     I exported an H.264 version of the video and included 6 dngs along with MLVFS settings.

     Here is a Google Drive link to these files: https://drive.google.com/drive/folders/0Bz5n0qYxIkVON0g1WUVVMDM1NWc?usp=sharing (https://drive.google.com/drive/folders/0Bz5n0qYxIkVON0g1WUVVMDM1NWc?usp=sharing)

     MLRawviewer 1.4.3 will not play these files, they look like white static.

     Also, using MLVFS, I'm getting Embedded when trying to use Cinelog-C profile in Adobe Camera Raw. I will make sure to post to the MLVFS Forum, but I just wanted to mention this for 10 bit raw recording. This might be due to the camera model though.

      Sound worked fine for this video, I just have it muted for privacy reasons (was having a conversation with someone while this was recording).

      This was from DFort's December 18th Build, found here: https://bitbucket.org/daniel_fort/magic-lantern/downloads (https://bitbucket.org/daniel_fort/magic-lantern/downloads)

      The settings I had for recording this video: 1280 x 544 Resolution, 2.35:1 Aspect Ratio, 10bpp, Global Draw Allow, Preview Options Auto, and the rest of the settings default. Override Audio settings on with 30 db Mic Boost. MLV Sound On. MLV_PLAY, MLV_REC, MLV_SND, AND RAW_TWK were the modules enabled.

     I was using a Lanparte LPE8 Battery Brick with LPE8 Dummy Battery to help avoid overheating.

      I hope this helps in some way!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 30, 2016, 02:29:36 AM
@PaulHarwood856

MLRawviewer won't work with 10bit/12bit MLV's.

Why do you have Bad Pixel Fix set to Aggressive?

One corrupted frame on a 25 minute recording is pretty good. Of course it would be better if there were none.

The color highlights on the fan are a problem. Would you be willing to try a debug build to see if we can fix that?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 30, 2016, 02:38:13 AM
@Paulharwood856.
Also try this version of mlv_dump (ml-dng branch) since it will work 10/12/14 bit files as well as provide correct white balance, camera model id tags(Not using localized camera model tags which can,t be used with acr dcp files)
mlv_dump
https://bitbucket.org/Dannephoto/ml-dng-dannephoto/downloads/mlv_dump

Code here ml-dng (dmilligan) reworked upon by bouncyball, maintained and updated with g3gg0 latest mlv_dump changes which I recently added.
https://bitbucket.org/Dannephoto/ml-dng-dannephoto
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 02:51:52 AM
Hey dfort,

     I found Bad Pixel Fix helps with the 7D, maybe I'm wrong? Maybe not for the T3i? Yeah I can try a debug build, that would be great. Can you guide me into what needs to be done? I haven't done this before.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 02:54:44 AM
Hey Danne,

     Ok great, I will try this version of mlv_dump, thanks! I'll make sure to check this page you linked to see updates and changes. Awesome!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on December 30, 2016, 02:56:36 AM
Also try without bad pixel fix and see if that get rid of the strange highligts
Quote
The color highlights on the fan are a problem.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 30, 2016, 03:27:13 AM
Yeah I can try a debug build, that would be great. Can you guide me into what needs to be done? I haven't done this before.

Ok, here's the debug build:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_RAW_DEBUG_TYPE.2016Dec29.600D102.zip

It might work right away. The issue is that several cameras need to have the PREFERRED_RAW_TYPE defined, the 600D might be one of them. I started with a value of 0x10 which might do the trick but you can try other values. Go to the Debug menu, select the first item and you'll be able to change it.

(https://c3.staticflickr.com/1/766/31229380130_7893122693.jpg)

Here's a more detailed explanation of what is going on:

https://bitbucket.org/hudson/magic-lantern/pull-requests/780/increase-raw_debug_type-limit/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: maxotics on December 30, 2016, 04:20:57 AM
I was able to do various  10-bit videos on the EOS-M.  1600x900 is amazing!  Files play fine in latest MLVProducer.   Amazing!  dfort you are DA MAN :)   I see builds for the 7D and it sounds like others have got it to work.  Should I stick with the EOS-M or should I, can I, try on 7D? 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 04:21:54 AM
Hey Danne,

     I just tried without Bad Pixel Fix, and the strange highlights were still there, but much less pronounced.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 04:32:08 AM
Hey dfort,

     Ok great, I will try this debug build out! Thanks.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 04:34:41 AM
Hey maxotics,

     If you get any success with the 7D, please post here. I haven't been able to get the settings right. Thanks.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on December 30, 2016, 05:51:42 AM
Quote
Hello everyone,

     I tested 10 bit Raw Video recording on a Canon T3i. I was able to record 1280 x 544 for 25 minutes on a 32 GB Sandisk Extreme Pro 95 MB/SEC SD Card.

     One pink frame was recorded on the 6th DNG, and there are pink, green, blue tints, on the highlights of a fan I recorded.

     However, they aren't on every dng frame, it seems every other.

     I exported an H.264 version of the video and included 6 dngs along with MLVFS settings.

     Here is a Google Drive link to these files: https://drive.google.com/drive/folders/0Bz5n0qYxIkVON0g1WUVVMDM1NWc?usp=sharing

     MLRawviewer 1.4.3 will not play these files, they look like white static.

     Also, using MLVFS, I'm getting Embedded when trying to use Cinelog-C profile in Adobe Camera Raw. I will make sure to post to the MLVFS Forum, but I just wanted to mention this for 10 bit raw recording. This might be due to the camera model though.

      Sound worked fine for this video, I just have it muted for privacy reasons (was having a conversation with someone while this was recording).

      This was from DFort's December 18th Build, found here: https://bitbucket.org/daniel_fort/magic-lantern/downloads

      The settings I had for recording this video: 1280 x 544 Resolution, 2.35:1 Aspect Ratio, 10bpp, Global Draw Allow, Preview Options Auto, and the rest of the settings default. Override Audio settings on with 30 db Mic Boost. MLV Sound On. MLV_PLAY, MLV_REC, MLV_SND, AND RAW_TWK were the modules enabled.

     I was using a Lanparte LPE8 Battery Brick with LPE8 Dummy Battery to help avoid overheating.

      I hope this helps in some way!

- Paul Harwood
Modify message

Although MLRawviewer will not play 10 bit or 12 bit .MLV files, I was able to install MLVProducer via Wineskin Winery for Mac. This plays 10 bit, 12 bit, and 14 bit files, nice!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Rewind on December 30, 2016, 06:49:36 AM
Is it right? Where is BD for second and third color? (R-G-B)
This is right. You forgot the Bayer pattern. There are only WxH pixels on the matrix. All three colors are among them.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on December 30, 2016, 07:35:39 AM
Should I stick with the EOS-M or should I, can I, try on 7D? 

The EOSM should be working fine. From the reports testers have been posting it looks like it works on the 7D in crop mode which is sort of weird because CONFIG_EDMAC_RAW_SLURP is not defined on that camera.

Of course you're welcome to try it yourself and post your results.

BTW--I noticed some differences in the resolutions displayed between mlv_rec and raw_rec (MLV Lite) on the EOSM:

mlv_rec
(https://c7.staticflickr.com/1/390/31134400734_edc3d9586b.jpg)

raw_rec
(https://c3.staticflickr.com/1/743/31826501562_1566348afb.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on December 30, 2016, 09:33:59 AM
BTW--I noticed some differences in the resolutions displayed between mlv_rec and raw_rec (MLV Lite) on the EOSM:
That's because only bit reduction (10-12) was implemented and not a1ex's fine resolutions code , so basically it's the same old MLV+audio
with bit reductions . Some thing on my 5D2

looks like it works on the 7D in crop mode which is sort of weird because CONFIG_EDMAC_RAW_SLURP is not defined on that camera.
Don't forget the 5D2 work fine in mlv2.0 crop mode too  ;D  flawlessly
in fact I just finished a recording 8:00 green screen shot @ 2144x1076 10bit @23.976 + audio 16 bit 48khz
cool thing was 8 min at over 2K = only 30GB when extracted I had over 49GB each frame was 4.5MB @16bit Cdng (11,190 frame) and even had my HDMI Zacuto Evf  connected
All frames clear of any corruption  8)
I thing I notice was normally I have a few hot pixel in crop mode but noticed there was only one in 10bit mode , so does bit reductions reduce the load on the sensor ?
or just lucky .   

Edit: I'll post the results in a day or two
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on December 30, 2016, 11:51:22 AM
Hi,

I'm new to this forum, I 've read all this thread about 10/12 bit raw and many other threads on raw, so I apologise if my question is too simple.
First above all, I want to thanks the magic lantern team  :)  ! and reddeercity for his 5D2 support with tutorials all along so many years ...
Now my question : with my 5D2 and Dec18.5D2212 software, I can't get "crop" or "not crop" without Liveview freezing.
So reddeercity how can you film 8 min ? with hdmi output ? with "magic" settings ?

Let me know, please.

David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on December 30, 2016, 11:54:01 AM
Testing of 10bit-raw video /EOS 600D/ at very low light conditions (ISO 1600-6400)
www.youtube.com/watch?v=FVdyuj1OHJg
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on December 30, 2016, 09:05:13 PM
@reddeercity which build did you use for flawless 10bit 8min recording?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on December 31, 2016, 01:09:54 AM
@reddeercity - yes, link please.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on January 01, 2017, 12:41:20 AM
@paulharwood

From my experience with ML raw, hot pixels increase with higher ISO and record time.
Try ISO 100 or 400 max, and limit the record time per clip.  How long did you record with T3i before hot pixel appeared?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 01, 2017, 05:56:55 PM
Hey ShootMeAlready,

     This was a test in case of low light conditions. So I had the ISO at 3200. I'll make sure to do the same test at ISO 100 or 400 to see if there are no hot pixels. Thanks a bunch!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 01, 2017, 05:59:32 PM
Quote
Ok, here's the debug build:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-10bit_12bit_RAW_DEBUG_TYPE.2016Dec29.600D102.zip

It might work right away. The issue is that several cameras need to have the PREFERRED_RAW_TYPE defined, the 600D might be one of them. I started with a value of 0x10 which might do the trick but you can try other values. Go to the Debug menu, select the first item and you'll be able to change it.



Here's a more detailed explanation of what is going on:

https://bitbucket.org/hudson/magic-lantern/pull-requests/780/increase-raw_debug_type-limit/diff

Hey dfort,

     So far I've tested 0x10 in the debug build, and got hot pixels, but no highlight issues with the fan in the shot. I'll keep testing, but just wanted to give you an update.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 01, 2017, 06:34:02 PM
From this analysis (http://www.magiclantern.fm/forum/index.php?topic=18393.0), PREFERRED_RAW_TYPE 0x10 appears to be the best choice for all DIGIC 5 and 6 cameras, and likely 0x5 for all DIGIC 4.

Can you confirm the above value? (600D, and also other cameras where PREFERRED_RAW_TYPE is not yet defined)

You should get:
- no bad pixels
- no changes from digital ISO (e.g. ISO 800 should be identical to 640 on the raw histogram)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 01, 2017, 06:52:02 PM
Hey Alex,

     Here is a link a dng from the 0x10 Debug. I saw the hot pixels in MLV Producer, but in this dng generated from cr2hdr.app I am only seeing a little artifacting in the candle and pink highlights in the fan. Also, is getting Err 70 normal when trying to playback raw files in a debug build?

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 01, 2017, 07:14:49 PM
0x10 is not relevant for 600D.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 01, 2017, 10:44:51 PM
Oops -- @PaulHarwood856 could you go into the Debug menu, change it to 0x5 and run another test please?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on January 01, 2017, 11:47:26 PM
As it is now well known, the 5D2 and most other cameras don't work in 10/12 bit, and definitely won't until CONFIG_EDMAC_RAW_SLURP is enabled, so I have decided I would like to help fix it on the 5D2. All I can do right now is compile Magic Lantern(with my own idiotic changes), and I've successfully ran the ROM disassembly script, so I don't really have much of a clue how to do anything, but I do have too much free time(sometimes), so is this something I could do with limited ability but lots of free time? Most complex thing I've ever done with C is getting two Arduinos transmitting numbers at each other at slightly over 1 Mbit/hour(yeah that slow :D) with analog 433mhz antennas, so I don't know if my coding skills are enough.

Basically all I ask is, might I have enough ability to get it done or help out? and if so, where to start?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 02, 2017, 12:13:25 AM
@ch_d  & dariSSight sorry for the delay , so the build I'm using is from dec.01/2016.
But I don't think that really matters , as mlv 2.0+audio code has not changed other then
implementing bit reduction code , so every thing else should be the same , where as with raw_rec (mlv Lite)
code has gone under I lot of work to get the limited amount of cam to work at this point.
So I'm betting that any of the MLV2.0 builds with bit reduction should work find ,
just in case there don't work for you guy's here the one I used  10bit_12bit_raw_2016Dec01.5D2212 (https://www.dropbox.com/s/oodspk5jits334u/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip?dl=0).
This build came from dfort so I didn't even compile the build myself.

Now on to something different , I said I would post the results of my 5D2 10bit 2.1K green screen 8:00 min shot .
This was just a test , but work so well that I put more work in to it and ended up as a short talk/explanation of 10,12bit raw video.
I was very impressed was the results , even thou I had a few issue with setting the shot up, focusing(as I was by myself , No auto focus in 3x crop mode)
The setting I used was mlv+audio 10bit 2144x1076 23.976p from 30p (not sure if 24 from 30 make any differences)
Auto preview for Liveview , audio set to 16bit 48khz (I ran the audio from my Zoom N4h in to the ext. mic input on cam)
ISO 400 , EF 24-70 f2.8L set to 35mm (Crop mode -- 2.60 crop x 35mm= 91mm) and had my zactuo evf connected (I needed something to reference to even thou it only shows
the center crop of the 2.1K image & B/W , it let me know when the recording stops but never did  :D
Process the 10bit mlv+audio on my PC with acr a.e. (on the my Mac the 10bit frames don't show up just all black from mlvfs, do show image in Resolve thou)
as I wanted to sharping up the image ( little soft on the focus) started with the adobe neutral profile and just pulled down the contrast & desaturated (flatted out)
export as 16bit tiff , imported in to apple compressor made my prores4444xq , graded in FCPX . I could have spent more time grading but I was looking for a quick turn around.
Any ways I hope everyone Enjoys it.
Happy New Years to everyone here on the forum  :)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 02, 2017, 04:41:50 AM
@reddeercity

Very good video and explanation about the last news!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 02, 2017, 07:44:28 AM
A few of us have been poking around the lv_raw_dump code for various cameras and it looks like the 1100D has very similar code to the 600D. I was able to get a grasp how to find the DEFAULT_RAW_BUFFER for two other similar cameras, the EOSM and 700D but the code for these cameras are vastly different--maybe because the 1100D/600D (T3/T3i) are Digic 4 while the EOSM/700D (T5i) are Digic 5 cameras?

It looks like the only cameras that are not working with 10/12 bit are all Digic 4 or dual Digic 4 cameras. The 60D and 600D are also Digic 4 and they are working so perhaps it will be possible to get all of the ML supported cameras working if we can understand how to follow these instructions?

Code: [Select]
/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
 *
 * raw_buffer = get_raw_buffer()
 * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
 * ...
 * dump_file(filename, raw_buffer, 7*something...)
 */

[EDIT] Thanks to @dmilligan and @eNnvi for taking the time to explain the EOSM code to me.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 02, 2017, 08:59:48 AM
5D mark III 1.1.3 works like a charm.

- Only in zoom 5x can't visualize. But if you press Info, can visualize at 10x zoom.
- I tried 10bits with zoom mode, 3,5K 4:1/5/1 2,5K, etc... aspect ratio and everything works perfect, continuous recording, except over 105 Mb/s or more.

52 Degree Celsius after 30 minutes playing and changing. I don't know if is too much. (A tea it's around 70????  :D )
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 02, 2017, 07:22:34 PM
I'm going to be cleaning up the mess in my bitbucket download area  and posting new test builds but first a download count and preview of coming attractions:

Checkmarks are for cameras that are working with CONFIG_EDMAC_RAW_SLURP:

5D3.123-118* made a few attempts at merging crop_rec but was unsuccessful
5D3.113-56* attempted merging crop_rec with limited success and had 129 downloads
6D.116-58
5D2.212-31
7D.203-22
EOSM.202-19
600D.102-17
70D.112-17* might have build issues? (Dec 23)
100D.101-16* might have build issues? (Dec 23)
650D.104-15
1100D.105-15* allocate_raw_buffer branch has CONFIG_EDMAC_RAW_SLURP working but not 10bit/12bit (Dec 19)
60D.111-14
700D.114-13
550D.109-11
500D.111-8
50D.109-6

The next round of test builds will not include the 70D and 100D because there might have been build issues with them and since they are not merged into the main repository I'll need the help of nikfreak with them. Also, the 1100D won't go up until we can track down the DEFAULT_RAW_BUFFER for that camera. The 600D, 650D and 700D test builds will define the PREFERRED_RAW_TYPE to see if that helps those cameras.

Give it a few hours for the new test builds to show up here:

https://bitbucket.org/daniel_fort/magic-lantern/downloads
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on January 02, 2017, 07:40:54 PM
Oh very unfortunate for being a  70D user
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 02, 2017, 09:20:31 PM
Oh very unfortunate for being a  70D user

70D and 100D users should be giving their feedback to the developers so these platforms will be accepted into the unified branch. Once they are in the main branch adding them to the raw_video_10bit_12bit branch is much easier.

https://bitbucket.org/hudson/magic-lantern/pull-requests/760/adding-support-for-the-70d-112/diff
https://bitbucket.org/hudson/magic-lantern/pull-requests/757/adding-support-for-the-eos-100d-sl1/diff

This also applies to 5D3.123 users:

https://bitbucket.org/hudson/magic-lantern/pull-requests/611/support-for-5d3-firmwares-123-and-113-in/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 02, 2017, 11:44:25 PM
Hey dfort,

     I'll make sure to test 0x5 on the debug build. Thanks for the heads up.

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: PaulHarwood856 on January 02, 2017, 11:49:11 PM
Hey reddeercity,

     Thanks so much for making this video! This explains everything really well, much appreciated!

- Paul Harwood
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Asiyuk on January 02, 2017, 11:56:38 PM
Hi guys,

I have a problem with 3 of 100 video that I shoot just few days ago. Other 97 video files are  really good. I shoot 10bit mlv on my 5d3_1.1.3 with 24dec Dfort's build. The problem is that all dng sequence is extremly dark, contrast and all green. I attach some screenshots from davinci resolve and 2 original dng. I tried  also adobe after effects and lightroom, but have the same problem. I used mlvfs and mlv_dump to get dng, but i see this problem.

(https://pp.vk.me/c837638/v837638034/186c6/cQpCE7qTUXg.jpg)
(https://pp.vk.me/c837638/v837638034/186ce/4fcyWCp1_a8.jpg)


here is dng: https://we.tl/1Dp6BTOiB8 (https://we.tl/1Dp6BTOiB8)

Does enyone have some advice for me?

English is my third language, but I hope, you understand :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 03, 2017, 01:17:04 AM
@PaulHarwood856 & budafilms thanks for the positive feed back  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 03, 2017, 03:49:48 AM
Test builds are up.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

@reddeercity - Thanks for the video. So it was shot in 10bit on a 5D2 in crop mode? If you think it works good now think how good it will be once that camera is working with CONFIG_EDMAC_RAW_SLURP.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SpcCb on January 03, 2017, 04:27:44 AM
Thanks reddeercity, nice presentation ;)

So it's working only in crop mode on the 5D2? Or you are using this mode because of aliasing in scaled mode?
Ok, I saw in the other thread you said there'r some problems in full mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 03, 2017, 04:58:05 AM
@reddeercity - Thanks for the video. So it was shot in 10bit on a 5D2 in crop mode?
 
Thanks
Yes with full mlv ,  was a test originally to see if it could record corruption free for extended time.

If you think it works good now think how good it will be once that camera is working with CONFIG_EDMAC_RAW_SLURP.
Yes I know I've been looking in to that , there's a lot to learn to understand about CONFIG_EDMAC_RAW_SLURP on older cams

Thanks reddeercity, nice presentation ;)
So it's working only in crop mode on the 5D2? Or you are using this mode because of aliasing in scaled mode?
Ok, I saw in the other thread you said there'r some problems in full mode.
Thanks
Yes 5d2 Full frame still has frames corruption problems
I wanted to see if things are still stable and reliable with bit reduction or just hit and miss .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 03, 2017, 09:30:22 AM
Test builds are up.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

@reddeercity - Thanks for the video. So it was shot in 10bit on a 5D2 in crop mode? If you think it works good now think how good it will be once that camera is working with CONFIG_EDMAC_RAW_SLURP.
Nice job dfort! Is there a change log I can read? Will try it out tonight. Best, David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 03, 2017, 06:09:16 PM
Asiyuk: Sometimes I have the same problem with dark-green colors on video /10 bit raw build 18dec/ EOS 600D

dfort: Thanks for new build ! I will try
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 03, 2017, 07:26:12 PM
Can you post a sample? Could be black levels are off. How often is it happening?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 03, 2017, 08:19:06 PM
@Danne - Asiyuk uploaded a sample on Reply #942

Quote
here is dng: https://we.tl/1Dp6BTOiB8

Though maybe you would prefer an MLV file?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 03, 2017, 08:37:47 PM
Black and white levels are off. After correction.
(https://s28.postimg.org/sse0iqdvh/Screen_Shot_2017_01_03_at_20_36_49.png)

For 600D enter terminal:
cd folder/with/dng/files
Code: [Select]
exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dnghit enter
For 5D markIII
Code: [Select]
exiftool "-BlackLevel=2046" "-whitelevel=16383" *.dng
Seems exiftool won,t work with the 5D mark III file. YOu can use exiv2 maybe or try mlv_dump or another converter and then add correct white/black levels.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dariSSight on January 03, 2017, 09:02:39 PM
@dfort Thanks for all your doing for The Goal of lower bit rate. I do have a question to ask and maybe its been asked before but why does the 5D Mark II stay at MLV 1856 and MLV Lite 1880?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 03, 2017, 09:28:18 PM
Alright guys I know I'm a tad late for this party once again and I apologize. Really hate slowing you guys down. But you guys are all doing great pushing this project forward!  :)

Ran @dfort's latest (2017Jan02) for 5D3.113 and notice after recordings of 10/12-bit files while in mlv_play.mo I get this some kind of '...stack overflow' errors which produced several crash logs occasionally (even spitted out COREDUMP.DAT whatever that is) within period of 2-5 minutes or so. Here they are stored into my dropbox (https://www.dropbox.com/sh/yvw0mbobxjvi7gm/AAAOzCzLebRDCRarHVp1MS0aa?dl=0dropbox) just in case they give us hints or not.

I ran a bunch of test files in both mv1080 and mv720 with MLV + Audio and all seems well. Even in 5x crop mode at 2.5k/3.2k for shorter takes if you will.


Now upon trying to upgrade firmware (in order for me to uninstall ML and do a clean upgrade to 5D3.123) which seems to not work with your build. I get stuck with a black screen along with a steady orange light next to the LED blinker (not blinking tho) and battery pull out fixes this. But then thought maybe I had to download a nightly from unified would help for me to do this to no avail.  :-\

Not sure if it was worth sharing this but I felt it was. Anyone else experiencing this?

Off I go to try and do 5D3.123 update (will probably have to use EOS Utility tho I shouldn't have to and suspect this has to do with the autoexec.bin?) and will report my findings then.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on January 03, 2017, 09:40:55 PM
You. Don't. Need. EOS. Utility. With. 1.2.3. And. 1.1.3.
You don't need to "clean up cam" before going to 1.2.3. or 1.1.3 either.
Just format a card to remove card's boot option (either use ML's "remove ML" format option or card reader). Copy Canon firmware file and upgrade/downgrade.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 03, 2017, 09:42:46 PM
To Asiyuk "dark-green" picture.
The worst videos are already deleted. But something like that:
"dng picture"
http://www.uschovna.cz/zasilka/LILKZ228LKN9X8ZA-SX9/

Original raw video file:
http://www.uschovna.cz/zasilka/LIIWK6FJT47XUGD8-IJG/

BUILD:
magiclantern-raw_video_10bit_12bit.2016Dec18.600D102
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 03, 2017, 10:02:45 PM
Check reply 951 Honza.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 03, 2017, 10:46:51 PM
Danne: After command:
exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dng

Picture looks much better, but not completely pure  ;-)

http://www.uschovna.cz/zasilka/LIRRW9FAWK638482-F4X/
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 03, 2017, 11:01:45 PM
Yes, some corruption going on as well. Well, to be expected with bleeding edge firmware builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 03, 2017, 11:02:14 PM
You. Don't. Need. EOS. Utility. With. 1.2.3. And. 1.1.3.
You don't need to "clean up cam" before going to 1.2.3. or 1.1.3 either.
Just format a card to remove card's boot option (either use ML's "remove ML" format option or card reader). Copy Canon firmware file and upgrade/downgrade.

Got it, Walter. I was hoping not to do this because I had a whole bunch of random files that turned out to be awesome (from moments when I forget to set the 5D3 to record onto CF instead of SD) from the past few years even had files from 2015. So yeah you were right I had to format (prefer Disk Utility to keep the ExFat just because...) and was able to update straight up to 123 from in cam even with the bootflag enabled, ha!

Off I go to work and maybe I'll be able to shoot some dark scenes with ISO 6400 in 10-bit slo-mo and do some DF avg automation scripting from @Danne's latest and greatest cr2hdr.app (http://www.magiclantern.fm/forum/index.php?topic=15108.msg177259#msg177259cr2hdr.app) (even previews 10/12-bit flawlessly in MLRawViewer 1.4.3) while running in MLVFS mode. Holy shit dude!  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 03, 2017, 11:38:11 PM
Before:
(https://pp.vk.me/c837638/v837638034/186c6/cQpCE7qTUXg.jpg)

After:
(https://s28.postimg.org/sse0iqdvh/Screen_Shot_2017_01_03_at_20_36_49.png)

Good to know it is salvageable but how did that happen on 3 of 100 videos?

Ran @dfort's latest (2017Jan02) for 5D3.113 and notice after recordings of 10/12-bit files while in mlv_play.mo I get this some kind of '...stack overflow' errors which produced several crash logs occasionally (even spitted out COREDUMP.DAT whatever that is) within period of 2-5 minutes or so.

Looks like you're loading way too many modules. Try loading just the one's you're testing first. I don't know how to read those logs but my guess is that it looks like you're low on memory.

Code: [Select]
Free Memory  : 144K + 3569K
The COREDUMP.DAT shows even less memory available.

Code: [Select]
Free Memory  : 135K + 3562K
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 04, 2017, 01:35:51 AM
FYI, the workarounds for the green cast (aka wrong black level) bug are very well known to those used to read sticky topics ;)

http://www.magiclantern.fm/forum/index.php?topic=11664.0
http://www.magiclantern.fm/forum/index.php?topic=11942.0

Now, why does it happen at low bit depths?

My best guess is that our raw backend sometimes decides to compute the black level at the same time with the bit depth switch. There are some sanity checks to avoid computing the black level from an invalid image, but they are not 100% effective. Disabling them should increase the probability of encountering this bug.

The only other tasks (besides raw recording) that could trigger a refresh on the raw parameters under normal usage are the raw overlays, so you now know what to disable until this issue is fixed.

The fix would be in the raw backend: bit depth changes should be done in raw.c and guarded by raw_sem, and lower bit depths should be handled differently. Some possibilities (not yet sure what to choose):

* disable all raw processing (e.g. overlays) at lower bit depths
   + minimal amount of coding
   - no overlays while recording at lower bit depths (including grayscale previews and so on)
* perform all raw processing on 16-bit buffers, as in raw_twk:
   + fast (all raw processing will be faster)
   - may requires custom memory management for raw buffers, as Canon code reserves memory for a 14-bit buffer
   - will overflow outside LiveView, as we cannot allocate a continuous block larger than a 14-bit full res picture
   - on the fly conversion to 16-bit could work, but gives additional complexity
* convert lower bit depths to 14-bit, with raw_twk code:
   + would reuse current 14-bit code
   - no speed gains from raw_twk
* create the low bit depth image from the main 14-bit raw buffer, instead of edmac_copy_rectangle
   + would reuse current 14-bit code
   + a similar code structure would also work for lossless compression
   - no speed gains from raw_twk
* variations of the above

I don't know how to read those logs but my guess is that it looks like you're low on memory.
Code: [Select]
Free Memory  : 144K + 3569K
Code: [Select]
Free Memory  : 135K + 3562K

From mem.c, look for "malloc" and "AllocateMemory" at preferred_free_space. The above is not a low memory condition; quite the opposite. Some background info here (http://www.magiclantern.fm/forum/index.php?topic=8358.0).

Looks like you're loading way too many modules. Try loading just the one's you're testing first.

This advice is perfectly valid; not just for memory reasons, but also for other kinds of code interactions. For example, a module might start its own tasks, and those are a scarce resource on certain cameras. Also, raw_rec and mlv_rec are known to conflict with each other on functions exported to other modules.

I'm thinking to place a warning on the Modules tab, to be displayed if you load too many modules. They are not meant to be loaded all at once.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on January 04, 2017, 09:55:06 AM
Hi !

@reddeercity

Thanks for your video, I saw it but I still have a question : is your liveview was freezing during recording ?

I've installed your version (with the link you've gave us) and copied all the settings I could see in your video,

I didn't manage with my 5D2  to have a liveview unfreezing in crop mode + MLV2.0 ( with a hdmi connected or without ).

Do you have a "magic" settings or perhaps I've missed something ?

David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: NoCp_Albert on January 04, 2017, 08:18:38 PM
Hallo Danne

Hopefully I can help a bit with my test for your last magiclantern-raw_video_10bit_12bit.2017Jan02.5D3123 upload.

First I made the same test as DeafEyeJedi in his Video on 5D3113 firmware. I used the same settings and the results are similar what you see on this Video. But, when I replay with MLV Player, I got clear Video clips.

My Standard on 5d3 -123 works with Canon Setting PAL and 1920 x 25 fps. Now, with your new Builds raw_video_10bit_12bit.2017Jan02.5D3123 and raw_video_10bit_12bit.2017Jan02.5D3113 the Camera switch during 3xCrop Mode in to NFTS (Canon Menu still 1920x25fps) and shows me the Frame rates goes to 29,776 fps, but this is NTFS not PAL.
By 29,766 fps and 2400x1320 - 16-9 recording Time is 8 - 12 Seconds. If I use FPS override, I can select 25 from 30 fps and it recording works continually without problem.

If I select the maximum 3520x1320 - 25fps the record stops with Frame skipping after 5 seconds and I must take the Battery out to start the Camera again. There is only the message about "Frame skipping"

Hope this Info is helpful for you. Generally I use MLV with Sound bei 1920x25 and use some times 3x Crop mode for Nature (Bird) documentaries with Lenses from 12 to 600mm. Your last magiclantern-crop_rec.2016Dec28.5D3123.zip works beautiful, only missing the 10bit mode for more frames.

Big thank you to all ML Developers
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ricbi1 on January 04, 2017, 09:14:27 PM
Did some tests on 5d3 and like others have reported 10bit works great in non-crop mode. I also pushed FPS to 60 at full HD and stopped recording at 1min + at 2:35 aspect ratio.
Really impressive!
Looking forward to doing more tests.
Recording to KomputerBay x1000 64gb
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 04, 2017, 09:28:06 PM
Just for information, I've got a few video files shot in 12 bit with wrong black level too, on the 6d.
I've shot about 25 mlv files during the day and got three files with green dark frames, never switched raw video settings, only turned camera on and off and adjusted diafragma and iso.
And did some switching between video and photo mode.

So it happens on multiple platforms.
Good to know where this weird green frames are coming from and how it is fixable ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on January 04, 2017, 09:50:50 PM
Did some tests on 5d3 and like others have reported 10bit works great in non-crop mode. I also pushed FPS to 60 at full HD and stopped recording at 1min + at 2:35 aspect ratio.
Really impressive!
Looking forward to doing more tests.
Recording to KomputerBay x1000 64gb

FullHD 60 fps? Its impossible)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ricbi1 on January 04, 2017, 10:19:08 PM
Meaning 1920x squeeze factor from Magic Lantern... but 1920 HD
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 04, 2017, 10:56:38 PM
Did some small updates, mostly refactors to reduce the need to hardcode model-specific values. Details here (https://bitbucket.org/hudson/magic-lantern/pull-requests/778/raw-fixes-part-3).

Would be nice to get feedback (whether things are still working fine) on all models that use EDMAC_RAW_SLURP. Behavior on older models is unlikely to be changed.

I had another attempt at fixing 5D2, unfortunately without success. Maybe next time. The changes that worked were committed to allocate-raw-lv-buffer (currently useful for 1100D).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 05, 2017, 02:57:42 AM
Hi !@reddeercity 
Thanks for your video, I saw it but I still have a question : is your liveview was freezing during recording ?
Your welcome , No freezing liveview work as normal . Lv set to auto preview ,

 
Do you have a "magic" settings or perhaps I've missed something ?
My 5d2 ml cam setting (https://www.dropbox.com/sh/le3wt4fqf61kmic/AAA6Cqzs4kOb33GGiPLBg2Jxa?dl=0)

I did have 30p ntsc in canon menu and frame over ride to 23.976p


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: benoit on January 05, 2017, 12:40:39 PM
Whaaa  :D
Thanks a lot Reddeercity !
Yours settings files rules !
Now I have greyscale liveview on 5D2 , great !
On my settings I changed PAL to NTSC without results on unfreezing.
But with yours settings, I changed NTSC to PAL and liveview freezes ... so NTSC with your settings is the graal.
So I think, NTSC is not the only one parameter to tweak.

I post my old settings to find the differences with yours because I don't understand the meaning of each parameter on these config files.

magic.cfg :

# Magic Lantern Nightly.2016Dec01.5D2212 (b3dfbe7194f3 (x-perimental) tip)
# Built on 2016-12-01 22:52:58 UTC by rosiefort@RosieFoComputer
# Configuration saved on 2017/01/05 09:04:53
beta.warn = 5
menu.first = -7
time.indicator = 1
movie.log = 1
bv.av = 35
bv.tv = 99
bv.iso = 117
bv.auto = 1
rec.notify = 4
screen_layout.ext = 0
shutter.lock.value = 99
shutter.lock = 1
enable-liveview = 2
movie.restart = 1
fps.preset = 1
fps.override.idx = 32
fps.override = 1
zoom.disable.x10 = 1
hdrv.iso.b = 112
hdrv.iso.a = 88
battery.drain.rate.rev = 149
hist.meter = 0
audio.mgain = 0
audio.dgain.r = 18
audio.dgain.l = 18
audio.monitoring = 0
audio.input-choice = 0
crop.playback = 1
spotmeter.draw = 0
clear.preview = 4
waveform.draw = 0
zebra.draw = 0
global.draw = 0
disp.mode.x = 4


# Config file for module mlv_rec (MLV_REC.MO)

mlv.video.enabled = 1
mlv.bpp = 0
mlv.write_speed = 6153
mlv.display_rec_info = 2
mlv.warm_up = 2
mlv.buffer_fill_method = 0
mlv.show_graph = 1

Thanks again.
David
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: danilaswift on January 05, 2017, 08:14:09 PM
Hello. Guys please lend the old assembly for canon 650d. In the new build of 2017 I was not happy with the video quality (ie, 12bit). Video is too sharp and a lot of red dots. In the old version, this was not.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 05, 2017, 10:14:47 PM
Short video test of 10bit raw with EOS 600D:
Build dfort: magiclantern-raw_video_10bit_12bit.2017Jan02.600D102
I recorded 32 pcs of mlv files (raw_rec.mo). All was withouth damage or green cast.
The shutter time 1/30s was not good choice -> flickering
Original video file: https://ulozto.cz/!FwuHK8wc9CET/10bitraw-video-test-karvina-frystat-mp4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 05, 2017, 11:56:15 PM
Thanks a lot Reddeercity !Yours settings files rules ! Now I have greyscale liveview on 5D2 , great ! On my settings I changed PAL to NTSC without results on unfreezing.
But with yours settings, I changed NTSC to PAL and liveview freezes ... so NTSC with your settings is the graal.
So I think, NTSC is not the only one parameter to tweak.
I post my old settings to find the differences with yours because I don't understand the meaning of each parameter on these config files.
magic.cfg :
Code: [Select]
# Magic Lantern Nightly.2016Dec01.5D2212 (b3dfbe7194f3 (x-perimental) tip)
# Built on 2016-12-01 22:52:58 UTC by rosiefort@RosieFoComputer
# Configuration saved on 2017/01/05 09:04:53
beta.warn = 5
menu.first = -7
time.indicator = 1
movie.log = 1
bv.av = 35
bv.tv = 99
bv.iso = 117
bv.auto = 1
rec.notify = 4
screen_layout.ext = 0
shutter.lock.value = 99
shutter.lock = 1
enable-liveview = 2
movie.restart = 1
fps.preset = 1
fps.override.idx = 32
fps.override = 1
zoom.disable.x10 = 1
hdrv.iso.b = 112
hdrv.iso.a = 88
battery.drain.rate.rev = 149
hist.meter = 0
audio.mgain = 0
audio.dgain.r = 18
audio.dgain.l = 18
audio.monitoring = 0
audio.input-choice = 0
crop.playback = 1
spotmeter.draw = 0
clear.preview = 4
waveform.draw = 0
zebra.draw = 0
global.draw = 0
disp.mode.x = 4

Looks like you have Auto Settings enabled , all my settings are manual .
So set the programs dial on the top left hand side to "M" and I use manual WB in Kevlin also ISO .
I use no Auto setting (only exception is ML Liveview Preview).
Only load the min needed for raw video to avoid problems .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 06, 2017, 12:34:29 AM
New test builds posted.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

Downloads for the Jan 2 test builds. Checkmarks are for cameras that are working with CONFIG_EDMAC_RAW_SLURP:

5D3.113-35
5D2.212-27
5D3.123-23
6D.116-13
60D.111-9
7D.203-8
50D.109-8
600D.102-8
EOSM.202-7
700D.114-7
550D.109-7
650D.104-5
500D.111-1
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on January 06, 2017, 11:01:00 AM
I shot a wedding last week – 10bit MLV lite on 5D3. All shot at 60p. It is quite stable. Had 2 clips with black level issues much like aforementioned in this thread. 1 clip completely corrupt, not sure why. Here's the link:
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 06, 2017, 02:55:30 PM
Added 10/12-bit builds to the nightly server:

http://builds.magiclantern.fm/experiments.html

Current builds are probably identical to dfort's (didn't really check); future builds will just track the raw_video_10bit_12bit branch. 1100D is included as well, but raw recording support is quite incomplete. Please report and fill in the missing bits (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176935#msg176935).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 06, 2017, 04:36:06 PM
Would love to test 5D3 123..  but it is missing..
Thanks for your support!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: maxotics on January 06, 2017, 06:02:43 PM
I'm looking forward to the 7D.  FYI, it doesn't seem you can use normal MLV with the EOS-M 10/12-bit build.  It will say "thread not starting" or something like that.   So seems like that is a 10-bit build only.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 06, 2017, 06:16:12 PM
Turn off raw_twk. That and snd.mo don,t go well together.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 06, 2017, 06:38:02 PM
Beginners question: what is and what does CONFIG_EDMAC_RAW_SLURP do? Curious..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 06, 2017, 07:45:05 PM
A method that retrieves the raw data from Canon's image processing pipeline to main memory by configuring the DMA (EDMAC) directly, with our own parameters, rather than calling Canon code (lv_save_raw).

See here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181#msg174181) and look it up in the source code, or here on the forum.

Initially implemented for 5D3 1.2.3 to fix corrupted frames (http://www.magiclantern.fm/forum/index.php?topic=10443.msg100979#msg100979).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 06, 2017, 07:58:19 PM
It seems it also can be used as a cheat sheet for dev's to pick up certain EDM addresses already provided for each specific settings in order to save time and get things implemented rather quicker basically.

Here's what the first page upon running EDMAC (after 'Don't click me!') looks like:

(https://c8.staticflickr.com/1/545/32146424775_7521cb4aee.jpg) (https://flic.kr/p/QYEQ82)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RTI on January 06, 2017, 09:12:01 PM
Currently testing 10bit on my canon 6D
-I'm getting random slowdowns (using a lexar UHS-I 128 633, max write speed with 6D is around 38-39Mb/s) - quite often I get 9-10 MB when trying to record (can't reproduce it).
-Also, battery percentage always shows 100% (when in global draw ofc, in canon menu the right values are displayed).
-ISO display seems to show only full EV values, no 1/3EV.
-mlv_play doesn't work, shows some corrupt images.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 07, 2017, 12:29:24 AM
If you want the latest builds, bookmark this address:

https://builds.magiclantern.fm/experiments.html

I'm still putting up builds for the 5D3.123 and the platforms without CONFIG_EDMAC_RAW_SLURP. Seems that there is still some limited 10bit/12bit functionality on the 5D2 and maybe 7D. It is also a way to keep track on which platforms still need to catch up.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

Hum, 121 downloads for the 550D?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on January 07, 2017, 05:21:58 AM
Yes, the 7D works fine (for my purposes) in 5x crop mode with the exception of not having true live framing (but I wasn't using that even with 14 bit). I have seen no corruption issues in crop mode with 10 and 12 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: NoCp_Albert on January 07, 2017, 07:02:14 AM
Hallo dfort

the last build from https://bitbucket.org/daniel_fort/magic-lantern/downloads/magiclantern-raw_video_10bit_12bit.2017Jan05.5D3123.zip shows me the problem during mlv_play, without loading the raw_rec module I get corrupted files to see, if I see the recorded frames at MLVProducer everything is ok. If I load the raw-rec module and record with mlv_rec also in mlv-play I see the clip correct. This was not happen on your build from 2 January 2017.

If I like to go back from 5xCrop used with 50fps by 10bit 1920x648 to 5xCrop 233x1314 25fps 10bit must switch of and on to work with the new setting. With 5x crop the lifeview shows the the images are jumping from normal size to 5x crop sometimes, if you like i prepare a video to show what I mean.

When I try to use normaly 1920x1080 25fps by 10, 12, 14 bit the record stops after 12 seconds bei "Frame skipping"
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 07, 2017, 10:56:08 AM
I tried the -a(average all frames) option in mlv_dump.osx on some 12 bit files for creating darkframes.
But the average file doesn't look right...it looks like a weird purple frame (when extracting the MLV files to dng's, I get normal black frames...)
Could it be that the average all frames option in mlv_dump doesn't work (yet :D) on 10 and 12 bit files ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 11:24:02 AM
Do you get the same with 10-bit? I use it without issues in cr2hdr.app.
g3gg0 rewrote code so it works with 14bit code in mlv_dump. Very great.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 07, 2017, 11:39:05 AM
I tried the -a(average all frames) option in mlv_dump.osx on some 12 bit files for creating darkframes.
But the average file doesn't look right...it looks like a weird purple frame (when extracting the MLV files to dng's, I get normal black frames...)
Could it be that the average all frames option in mlv_dump doesn't work (yet :D) on 10 and 12 bit files ?

will investigate that
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 11:53:51 AM
Just ran darkframe averaging on my eos m in 12bit and all was working fine on that particular cam. Make sure to use the very latest version of mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 07, 2017, 11:55:21 AM
ok thanks. you ran it on osx ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 12:05:10 PM
Yup macOS Sierra.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 07, 2017, 12:35:49 PM
Sounds like I'm doing something wrong here...
I'll try the latest mlv_dump, is there a standard download location for the newest mlv_dump?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 12:50:47 PM
It,s in the 10/12bit branch.
http://www.magiclantern.fm/forum/index.php?topic=7122.msg177020#msg177020
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: teatotalTED on January 07, 2017, 04:58:09 PM
Hum, 121 downloads for the 550D?

That'll be because there's a lot of 550D users, more than EOS M and for good reason. Joking aside, 12bit works so some improvement for limited card write speed, obvious 10bit is eagerly awaited no doubt.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 07, 2017, 07:08:09 PM
...if you like i prepare a video to show what I mean.

That might help. I'm having a hard time following what changed from January 2 to January 5 that would cause those issues. It might be a bad merge on my part.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 07, 2017, 10:02:41 PM
Some color and resolution testing of latest :  magiclantern-raw_video_10bit_12bit.2017Jan05.5D3123

Original image from CR2 (cropped)  taken in manual mode ISO 100   White Balance in manual at 3900K
When open in Adobe Camera Raw WB reads:  Temperature +3900 and Tint +7
(https://s20.postimg.org/ncva4g0bx/From_CR2_image.jpg)

Recorded video in Manual Mode ISO100 WB=3900K and extracted DNG for comparison with good exposure and then with low light but bumped exposure by +4EV

TEST1: Properly exposed video Image from DNG.. Note that when I open with Adobe Camera RAW it reads: Temperature +5500 and  Tint +19
On the left are the un-edited images and on the right I have used "Auto WB" in Adobe Camera Raw
(https://s20.postimg.org/pipkyy3sd/comparison.jpg)

TEST2: Low light exposed video image from DNG..  Note that when I open with Adobe Camera RAW it reads: Temperature +5500 and  Tint +19
On the left are the un-edited images and on the right I have used "Auto WB" in Adobe Camera Raw
Exposure has been edited to +4 EV
(https://s20.postimg.org/vwjpzbezx/comparison_bad_expose_4ev_3_X.jpg)

Question 1: Why is MLV not properly passing White Balance to ACR in the DNG format.
Question 2: Why is there some significant White Balance tweaking required in 10 bit.

NOTE: I cannot find any significant difference in resolution between 10-12-14 bit.  After pixel peeping I see a tiny amount if difference between 10 and 14 bit.. only on low exposure areas.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 07, 2017, 10:21:26 PM
Thanks for tests. Wonder what darkframe averaging would do for those 10-bit files.
Hard coded wb values in mlv_dump. Until ml-dng code finds the way into unified.
If important you can dig out mlv_dump in cr2hdr.app and use that one. Bouncyball put together missing white balance parts in the ml-dng code dmilligan uses in mlvfs and in his ml-dng code into that version of mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 07, 2017, 11:04:54 PM
@Danne I dont think there is one available for 10bit and windows??
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 07, 2017, 11:08:27 PM
Question 2: Why is there some significant White Balance tweaking required in 10 bit.

Black level may need some fine-tuning to reduce roundoff errors.

When you simply discard some bits, the resulting signal, when scaled back, will have lower values compared to the original. For 14->10 bit conversion, the difference will be between 0 and 15 DN.

Therefore, when interpreting 10-bit data, it's probably best to adjust the black level by 0.5 LSB. This would require scaling the 10-bit data to a higher bit depth (at least 11).

For latest mlv_dump, which scales the 10-bit data back to 14-bit levels (multiplies by 16), the workaround would be:
Code: [Select]
exiftool *.dng -BlackLevel=2040

(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2014%20bit.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202032.jpg)
(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202040.jpg)

Code: [Select]
[14-bit 2047    10to14 2032;
 10to14 2048    10to14 2040]

Test image from here (http://www.eoshd.com/comments/topic/21004-12-or-10-bit-raw-magic-lantern/?do=findComment&comment=168058).

For a fix, the raw recording module should probably save the 14-bit black/white levels and let mlv_dump do the scaling. Or, remove the autodetection and just hardcode the reference black level (e.g. 2048 on 5D3); in this case, the roundoff error will be predictable.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 12:37:42 AM
Thanks A1lex:  Great explanation and solution..

After applying the exif fix here are the results:

(https://s20.postimg.org/3yzi1vf71/Compare_after_exif.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: eNnvi on January 08, 2017, 01:26:33 AM
Sorry guys if i'm taking so long but i'm really in a short of spare time.

I'm still trying to understand what's the big difference between digic 4 and digic 5 for raw_slurp

Io to now i think i just discovered hot water as i'm trying ti figure out how Memory is dinamically allocated in Canon firmware. As a1ex said a few posts ago probably the raw buffer address is a variabile inside a structure. My Guess is that in digic 5 this struct is statically allocated while in digic 4 this is dinamically allocated via a Memory lib (for garbage collection? With debugging symbols and assert management i think)
This should also explain why in digic 5 se get a pointer while in digic 4 we get a pointer to a pointer.

What i don't understand is why the offsets from the struct base address are so different from camera to camera.

Is it possibile to dump a certain part of RAM while running? From address X get N bytes?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 08, 2017, 02:12:39 AM
@eNnvi -- You can make fresh dumps of ROM0.BIN, ROM1.BIN and RAM4.BIN from the Debug menu.

Looks like the cameras that need to get the CONFIG_EDMAC_RAW_SLURP treatment are all Digic 4:

5D2.212-Digic 4
7D.203-Dual Digic 4
50D.109-Digic 4
500D.111-Digic 4
550D.109-Digic 4

The 60D, 600D and 1100D are also Digic 4 but I have not found out how these were resolved. I keep getting lost tracing lv_raw_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on January 08, 2017, 04:54:49 AM
Therefore, when interpreting 10-bit data, it's probably best to adjust the black level by 0.5 LSB.
Or perhaps even just adding back 0-15 units of noise?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on January 08, 2017, 06:09:15 AM
So why is the 600D experimental?  It seemed to be one of the most full functioned?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 08, 2017, 07:14:55 AM
So why is the 600D experimental?  It seemed to be one of the most full functioned?

It isn't the 600D that's experimental, it is "10/12-bit RAW video" that is experimental.

This might be the first time that a development branch is being compiled on the Jenkins server so that ML users can have a preview of coming attractions. Note that there is also a section for "Non-CPU lens info" and a "Crop mode recording (crop_rec)" section too. Yeah, I know, users want 10/12-bit and crop_rec but that requires resolving merge conflicts and I already botched it up a few times because I don't have a 5D3 for testing so you can ask someone else to do it for you or learn how to compile ML and try it yourself. It isn't that hard and there are several tutorials in the General Development Discussion (http://www.magiclantern.fm/forum/index.php?board=25.0) section of the forum. (I'm starting to sound like an old man repeating the same thing over and over.)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on January 08, 2017, 10:14:39 AM

The 60D, 600D and 1100D are also Digic 4 but I have not found out how these were resolved.

Those use EVF_STATE being DIGIC4. All other DIGIC4's using LVSTATE due to the missing EVF_STATE are the ones that are not working. a1ex has the state diagrams published on his bitbucket repo and allocating the buffer is not the problem (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176545#msg176545)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Andy600 on January 08, 2017, 11:04:43 AM
I tried mlv_rec (and mlv_play + raw_twk) with the nightly experimental version for the 50D using non-crop, 5x crop and 10x crop at 10, 12 and 14bit.

10 and 12bit recording exhibit frame tearing with the lower portion of the image always being frame 1 but there were no colored/noisy corrupt frames. 14bit has no corrupt frames.

I don't remember the 50D ever recording in 10x but it does now  ??? (though it might just be that I have never used it before  :P)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 08, 2017, 11:52:47 AM
So I've downloaded the 25th december 2016 version of mlv_dump and now I get normal dark dark frames from averaging a mlv file.
And I noticed the differince in file size, this version of MLV dump creates 14 bit files, where the older mlv_dump created 12 or 10 bit files.
Is this for compatibility reasons or are there other advantages that the output files are converted to 14 bit ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 11:58:33 AM
So I've downloaded the 25th december 2016 version of mlv_dump and now I get normal dark dark frames from averaging a mlv file.
And I noticed the differince in file size, this version of MLV dump creates 14 bit files, where the older mlv_dump created 12 or 10 bit files.
Is this for compatibility reasons or are there other advantages that the output files are converted to 14 bit ?

can you tell me the exact commandline, the bit depths of the input files and the bit depth the output file had?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 08, 2017, 12:06:52 PM
I'm using an old script which was once available on magic lantern, made some alterations for averaging.
Input file is 12 bit mlv, output file is a single frame 14 bit mlv

workingDir=`dirname "$0"`
cd "${workingDir}"

sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
sudo chmod +x /usr/bin/mlv_dump

for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    mv ./"$BASE".M* ./$BASE
    cd ./$BASE
    /usr/bin/mlv_dump --no-fixcp --no-stripes -o ${BASE}_frame_ $FILE -a
    cd ..
done
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 12:56:18 PM
"--no-fixcp --no-stripes" should not have any effect. they are for dng export only.

Code: [Select]
root@linux-dev:/media/sf_M_DRIVE/raw/10_12_14bpp# mlv_dump M24-2240.MLV -v | grep bits_per_pixel
      bits_per_pixel   12
root@linux-dev:/media/sf_M_DRIVE/raw/10_12_14bpp# mlv_dump --no-fixcp --no-stripes M24-2240.MLV -o M24-2240_avg.mlv -a > /dev/null
root@linux-dev:/media/sf_M_DRIVE/raw/10_12_14bpp# mlv_dump M24-2240_avg.mlv -v | grep bits_per_pixel
      bits_per_pixel   12

which mlv_dump did you use?
when in doubt, always this one: http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 08, 2017, 01:44:14 PM
www.youtube.com/watch?v=UWFVnE4-xp0

Test of 10 bit raw video recorded on Canon EOS 600D during daylight.
Resolution: 1728x576, 23,97fps, ISO 100, mlv_rec.mo + mlv_snd.mo

BUILD/dfort/: magiclantern-raw_video_10bit_12bit.2017Jan05.600D102

I recorded 25 mlv video files. Four files were corupted (green cast, frame-skipping).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 01:58:39 PM
"vibration" could be a stuck frame. then you will see parts of (or the whole) older frame.
it could be the first frame, or even the second-to-last frame you see there.

green cast is no corruption, just a black level error.
fix it (as described here a few times) by correcting the black level in DNG files, or by supplying "--black-fix=<value>" to mlv_dump with value depending on the bit depth.
when calling without any value "--black-fix" it will set black level to 2048
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 08, 2017, 02:52:59 PM
Or perhaps even just adding back 0-15 units of noise?

(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20off8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20off8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20r2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20r2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20r4.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20r4.jpg)
(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20r8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20r8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20r16.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20r16.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20n1.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20n1.jpg)
(http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20n2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20n2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20n4.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20n4.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/C%2010%20bit%202048%20n8.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/raw/10bit/full/C%2010%20bit%202048%20n8.jpg)

Which one looks best? Clicking on each thumbnail will open the full test image.

Render settings:
Code: [Select]
ufraw-batch --out-type=jpg --exposure=5 --temperature=2000 --green=1 --interpolation=ahd --color-smoothing --clip=film
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 03:02:23 PM
@g3goo  When I use --black-fix the dng cannot be read on Adobe Camera Raw and the command prompt shows "Cold pixels : 200000" see below:

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M07-1506.mlv'
   - Setting black level to 2048
   - Convert to DNG frames
   - Output into 'M07-1506.raw'
File M07-1506.mlv opened
File M07-1506.m00 not existing.
Processing...


Vertical stripes correction:
  1.00000    1      1      1      1      1      1      1
Cold pixels : 200000
Reached end of chunk 1/1 after 306 blocks
Processed 146 video frames
Done

edit: using the windows version
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 08, 2017, 03:13:34 PM
to g3gg0:
1) "vibration" it could be
2) to fix "green cast" I used command: exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dng
In this case 2/3 of picture is OK, but 1/3 (upper part of frame) is not usable.
Sample is here. I can't upload picture to this forum :-/
http://uschovna.cz/zasilka/LIBLRNMY6XS63UNG-9LL/49WFXLN2CF
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 03:19:44 PM
@a1ex:  Seems like n1 or r2 have less color noise than others.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 03:36:52 PM
@A1ex. r2 and r4 looks good. Bottom three not so.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 03:42:57 PM
@Renatophoto.
I never used --black-fix option in mlv_dump before but just tried it and it worked over here. Did you also specify the black level number like this --black-fix=2048
Black level setting has nothing to do with cold pixels so not sure why you would be getting that.
Could you specify the full command line output you are using?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 08, 2017, 04:04:10 PM
@Geggo
I did use the same mlv_dump as in your link.
The build from 25th december for osx

When I use that build on 12 bit mlv's, I'll get 14 bit dng's  ???
Which is nice, as in you can view them normal in finder, but the downside is that file size is also increased  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 05:23:40 PM
@Danne
mlv_dump --dng --black-fix=204 -o %Input%.raw %Input%.mlv
I suspect is something to do with ACR.  If I use exiftool then it works..  I was hoping to eliminate the exiftool step.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 05:31:01 PM
Tried putting --black fix command before the --dng?
Are you on mac? Could you upload a dng?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 05:40:59 PM
Yea, no change.. here is a dng
https://drive.google.com/file/d/0B9z8Y0rg-pu8SjFaMUtyeTRzT0E/view?usp=sharing
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 05:55:49 PM
I don,t think you are doing it right. Try something like this. It absolutely works over here.
Code: [Select]
mlv_dump --black-fix=2048 --dng My.MLV
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 08, 2017, 06:04:21 PM
I don't remember the 50D ever recording in 10x but it does now  ???

So it joins the 5D2 and 7D as cameras that shouldn't be working but are, sometimes, at certain settings? Does the 50D have zoom mode? Zoom in the LiveView to 5x or 10x and record video? I believe that's the setting that's working on the 5D2 but I'm not sure.

Those use EVF_STATE being DIGIC4. All other DIGIC4's using LVSTATE due to the missing EVF_STATE are the ones that are not working. a1ex has the state diagrams published on his bitbucket repo and allocating the buffer is not the problem (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176545#msg176545)

So the 60D, 600D, 1100D use EVF_STATE while the 5D2, 7D, 50D, 500D, 550D use LVSTATE and are missing the EVF_STATE? Learning how the 600D was resolved #define DEFAULT_RAW_BUFFER MEM(MEM(0x51FC)) isn't going to help with the 550D even though lv_raw_dump starts out very similar on both cameras? I keep getting lost and not knowing how the 600D address was discovered I'm clueless when it comes to trying to figure out these platforms. After @dmilligan explained how he traced lv_raw_dump on the EOSM figuring it out on the 700D and 650D was relatively easy. (Thanks to @eNnvi for explaining dmilligan's explaination  :D )

https://bitbucket.org/hudson/magic-lantern/pull-requests/781/650d-and-700d-enable/diff
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 06:05:50 PM
@Danne  No Go!
As I said it seems to be something wrong with ACR..  If I use the exiftool then it works.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 06:11:43 PM
When I use that build on 12 bit mlv's, I'll get 14 bit dng's  ???
vs.
Input file is 12 bit mlv, output file is a single frame 14 bit mlv

do we talk about the MLV -> MLV(averaged) script that produces 14 bit files
or about DNG conversion which is 14 bit only yet?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 06:13:44 PM
@Danne  No Go!
As I said it seems to be something wrong with ACR..  If I use the exiftool then it works.

is it better with '--no-fixcp --no-stripes' ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 08, 2017, 06:25:09 PM
@Renatophoto.
This command can,t work which you put up.
Code: [Select]
mlv_dump --dng --black-fix=204 -o %Input%.raw %Input%.mlvYou can,t even create a new MLV with another black level. In case it would have worked it would look like this.
Code: [Select]
mlv_dump --black-fix=2048 -o Output.MLV Input.MLV
It only works when spitting out dng files. Try exactly what I put out here.
Code: [Select]
mlv_dump --dng --black-fix=2048 INPUT.MLV
If still not working upload part of you mlv file

@Levas
Quote
Input file is 12 bit mlv, output file is a single frame 14 bit mlv
I thought this was to be expected, especially since g3gg0 updated code so all bits(10/12/14) would work with the 14bit code in mlv_dump?


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 08, 2017, 06:45:03 PM
Danne: I tried your suggestion and it does not work.
g3gg0: I tried but no luck either.

I think it something related to my system and ACR.  I will use exiftool to do the black level correction until it is fixed directly on the mlv_rec.mo
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 08, 2017, 06:46:57 PM
@Geggo

MLV conversion to dng's gives 14bit dng's even with 12 bit mlv's.
So 12 bit MLV gives 14 bit dng frames
Doesn't matter if average is used or not, always end up with 14 bit dng's

The MLV_dump build I used before, 20 November 2016 build, gave 12 bit dng's or 10 bit or 14 bit, depending whatever bit depth the mlv file was recorded.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 08, 2017, 08:17:10 PM
so MLV averaging produces the correct format, right?
i am asking because you reported the generated MLV has 14 bits although it should have 12 bits.

DNGs are *always* 14 bit in the branches i know. maybe you used mlv_dump builds from other sources.
do not know about the pending PR if it would produce 12bpp DNGs


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 08, 2017, 10:52:07 PM
@A1ex. r2 and r4 looks good. Bottom three not so.

Agreed!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 09, 2017, 04:46:42 PM
3X resolution testing  of latest dfort:  magiclantern-raw_video_10bit_12bit.2017Jan05.5D3123

In liveview (zoom 3x) the ML Grey Scale preview flickers when selecting aspect ratios beyond 16:9 i.e 1.85:1, 2:1, etc

Using cinnema ratios like 2.35:1 In liveview (zoom 3x) in ML Grey Scale preview the areas above and below the video, which should be black, have images (lines here) on the right half of the screen.  see below:
(https://s20.postimg.org/8ldzt6u99/VRAM1.jpg)


With big resolutions I get some bad frames:

3008x1320:
(https://s20.postimg.org/bgr301y99/3008x1320.jpg)

3520x1320:
(https://s20.postimg.org/xh7fgogx9/3520x1320.jpg)

Best Regards,
Renato
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 09, 2017, 05:56:46 PM
3X resolution testing  of latest dfort:  magiclantern-raw_video_10bit_12bit.2017Jan05.5D3123

Also ML Grey Scale turns to Canon preview while recording..
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 09, 2017, 06:23:30 PM
@Geggo
Quote
so MLV averaging produces the correct format, right?

Yes, the resulting average MLV has the right format, the average MLV is 12 bit, when input MLV is 12 bit.

Only thing new to me was that when extracting dng's, it gives 14 bit dng's no matter what the input is, 10/12 or 14 bit.
I was used to a MLV_dump which gave dng's in the same bit depth as the recording was.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 10, 2017, 07:15:47 AM
I made a test with a card Tone 14/12/10 bits
5d Mark III 1.1.3
Build January 5

I hope this can be usefull for the comunity.

Full info into the video.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on January 10, 2017, 11:50:23 AM
I made a test with a card Tone 14/12/10 bits
5d Mark III 1.1.3
Build January 5

I hope this can be usefull for the comunity.

Full info into the video.


Is this before or after the black level fix posted by A1ex above for 10 bit files. It is supposed to remove the purple colour cast in 10 bit.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 10, 2017, 04:38:09 PM
I didn't apply any correction, no black level 2048 in 5d Mark III.
But if @a1ex need a new test I can make it again!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 10, 2017, 04:48:18 PM
Here is the latest version of mlv_dump with g3gg0 latest commits. It,s vital to specify black level.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump

Source: https://bitbucket.org/hudson/magic-lantern/commits/75f61f30b6e2a6fb1a136d538b8fe190919024a9?at=unified

Specify like this if working with a 5D mark III
mlv_dump --black-fix=2048 --dng your.MLV
Correction:
mlv_dump --black-fix=2040 --dng your.MLV
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 10, 2017, 04:50:58 PM
... 2040 (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177802#msg177802) ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 10, 2017, 04:55:12 PM
Thanks, corrected.
WHat is the black level number for 12bit? The same as for 10bit?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tracerman on January 10, 2017, 05:51:10 PM
Hi,

I've tested 10bit and 12bit mlv_rec on the 550D : magiclantern-raw_video_10bit_12bit.2017Jan05.550D109

10 bit recording is indeed eagerly awaited on this platform, because 1280x544(2.35) at 23.98 fps makes about 20MBps
which is the maximum rate I can get from it (SandDisk 64Gb ExtremePro) so its pretty cool :)
big thanks to the ml devs, it gives a new life to this (ok rather old) camera but with (it seems) a lot of users still ...

I tested it in this configuration, recording goes OK (green light!) but when I try to read back it's not quite OK yet
(I've tried with mlv_play on the camera (with raw_twk loaded), with mlvfs on Linux and mlvproducer on Win7 with the same result) :


In 10 bit mode : it seems that 1 frame out of 2 is in fact the 1st recorded frame, and the other has problems on the top third of the frame.

In 12 bit mode : every frame is split vertically in 2 and offset in time and/or vertically (I can post another video if you need).

14 bit mode is all OK but recording stops after a few seconds (buffer full).

If I can be of some help in testing just let me know, having at least 10bit mlv_rec working would be awesome !

Thanks !

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 10, 2017, 06:24:15 PM
@a1ex @danne

Yes it works! 5d3.
No color pink/magenta and blue. Tested 10/12/14 bits

./mlv_dump --black-fix=2040 --dng M10-0135.MLV

(https://s29.postimg.org/z0rrmw3tv/M10_0136_000206.jpg) (https://postimg.org/image/z0rrmw3tv/)

(https://s29.postimg.org/phi2tfgbn/M10_0137_000188.jpg) (https://postimg.org/image/phi2tfgbn/)

(https://s29.postimg.org/60xd6wl7n/M10_0135_000415.jpg) (https://postimg.org/image/60xd6wl7n/)


image upload no limit (https://postimage.org/)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 10, 2017, 06:45:43 PM
Could you upload the dng files? And name them 10/12/14-bit so one can differ them.
Shouldn,t 14-bit have 2048 black level? 10 and 12 can,t be both 2040 or what am I missing?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 10, 2017, 06:56:02 PM
@Danne

I applied the same 2040 to 10/12/14 bits.

https://www.dropbox.com/sh/26id73senhicchy/AADDKTM9PefdK6rY0n7AhIa3a?dl=0


off topic: resolve studio (pay version) has a command called Noise Reduction. I don't know if this is usefull:
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 10, 2017, 08:07:47 PM
I need a 14-bit DNG sample from every single camera model, from a recent build (not older than 2017Jan05), either from experimental nightly or from dfort (doesn't matter). It must have some overexposed highlights and should be extracted with:
Code: [Select]
mlv_dump --dng --no-fixcp --no-stripes input.MLV

Please post them in this thread: http://www.magiclantern.fm/forum/index.php?topic=11899

In particular, I expect surprises on 6D and 1100D, but I'd like to check all models.

(sorry for the large font, but I'd like to merge this (https://bitbucket.org/hudson/magic-lantern/pull-requests/778/raw-fixes-part-3) in the main builds before 5D3.123, 100D, 70D, and I need these samples to make sure everything is fine)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 10, 2017, 09:51:38 PM
Some more crash logs (Error 70) recently with @dfort's latest 5D3.123 (2017Jan05) and this usually happens after previewing files through mlv_play.mo so not sure if they are related or even have anything to do with raw_twk.mo?

https://mega.nz/#F!vhdRET6S!rIZA_Ko_BMjxQGbwIm-d4w
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 10, 2017, 10:29:49 PM
@dfort:
Testing 7D.203 with :  magiclantern-raw_video_10bit_12bit.2017Jan05.7D203

14 bit works well..

12bit: I get damaged frame on every other frame, see below:
(https://s20.postimg.org/fwe3j2019/12bit_7_D.jpg)

10bit: I get damaged frame on every other frame, see below:
(https://s20.postimg.org/oq5005mzx/10bit_7_D.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 11, 2017, 12:04:53 AM
@a1ex @danne

Yes it works! 5d3.
No color pink/magenta and blue. Tested 10/12/14 bits

./mlv_dump --black-fix=2040 --dng M10-0135.MLV

(https://s29.postimg.org/z0rrmw3tv/M10_0136_000206.jpg) (https://postimg.org/image/z0rrmw3tv/)

(https://s29.postimg.org/phi2tfgbn/M10_0137_000188.jpg) (https://postimg.org/image/phi2tfgbn/)

(https://s29.postimg.org/60xd6wl7n/M10_0135_000415.jpg) (https://postimg.org/image/60xd6wl7n/)


image upload no limit (https://postimage.org/)


Looking at your tests I get increased blue in 12 and 10-bit. It gos higher and higher per bit. Look below:

(http://forumbilder.se/G74CS/10bits-vs-12-and-14.png)

It's really easy to fix in post per se, but for your info the blue/magenta is still there.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 11, 2017, 01:58:49 AM
Testing 7D.203

Right, that's expected. I believe that is also happening on the 5D2 but crop (zoom) mode is working on that camera. Does the 7D have that feature? Press the magnifying glass button to 5x or 10x and record video. Which feature is that on the features list (https://builds.magiclantern.fm/features.html)?

That isn't the solution to  but it is interesting that it works on the 5D2. Maybe it also works on the 7D?

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Frank7D on January 11, 2017, 05:29:14 AM
dfort, if you mean regular classic 5x zoom crop mode, then yes the 7D has that feature and 10 and 12 bit have no corruption issues when using that mode.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 11, 2017, 07:18:21 AM
@D_Odell

Thanks for the comparation and design. But I see a big mistake (it's my house and I know how the light goes over the table, white and stanley, and less behind that line and on the floor).

If you see carefully, horizontally (the same design like your version but not vertical) you can see is not a increment of blue or magenta. There is a difference in the noise between 10 and 14. But no incresed the blue and pink.

Other way to comprobe this is zoom in at 1000%. Something not usefull in the reality. Remember video has "movement" between pixels.

If you want to try the horizontally design, comparation only over the Card Tone, could be interesting and more accurate.

 ;)

Edit: the light behind the tone card (kitcheen) is Fluorescent (blue and pink), and over the card, other room, its tungsten 3200K.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 11, 2017, 07:37:35 AM
Could you post an example using black level 2048 on all three picures Budafilm?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 11, 2017, 07:53:29 AM
@Danne

Reply #1050 to you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 11, 2017, 09:16:22 AM
I,d say run all footage with black level 2048(5D mark III). A1ex seems to agree, at least counting from yesterday https://bitbucket.org/hudson/magic-lantern/commits/e3ef8c75cf5de2e4018098d21ae6d1d3bcd183d5
The 14-bit file is slightly different. White balance, color depth?

Order: 10/12/14
(https://s29.postimg.org/9j7pjt6uv/Screen_Shot_2017_01_11_at_08_58_50.png)
(https://s29.postimg.org/wjecw54on/Screen_Shot_2017_01_11_at_08_58_57.png)
(https://s29.postimg.org/h9ehoy96f/Screen_Shot_2017_01_11_at_08_59_06.png)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: VanGogh7 on January 11, 2017, 10:07:25 AM
I see at the  top of the pict. in the blue band a good visible difference between the 10,12,14 bit. 14bit has more details, see at the reflections in it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ottoga on January 11, 2017, 12:18:30 PM
@DFORT

10/12bit Testing

Camera 7D
ML Build: magiclantern-raw_video_10bit_12bit.2017Jan02.7D203 (DFort's)

Modules Loaded:   Fileman, mlv_rec, mlv_play, mlv_snd, pic_view & raw_twk                           

DNG's Extracted using mlv_dump (07/01/2017 build), parameters used --dng --cs2x2
Post processing/editing of extracted dngs - none.


Results of 10bit test using mlv_rec

Frame 000000 looks good with no colour casts, hot/cold pixels etc.
Frame 000001 looks good with no colour casts, hot/cold pixels etc.
Frame 000002 Has the top 1/3 of the image distorted and barely discernable as an image.
                       The bottom  2/3rds of the image look good.
Frame 000003 Same as frames 000000 and 000001.
frame 000004 Same as frame 000002

All subsequent  pairs of frames i.e: 000005&000006, 000007&000008 etc. behave the same as frames 000003&000004.

What is actualy happening here?  from what I can see:

1) All of the frames that have the top 1/3 distorted are a combination of the actual image that should be within that frame and a static image that is constantly repeating and overwriting the actual image. Having shuffled through the frames I believe that it is the distorted part of the frame that is the actual image for the respective frame. The undistorted part is likely from frame 000000.

2) Apart from frame 000000, all of the frames that are distortion free is actually a static image repeating itself. Likely from frame 000000.



Results of 12bit test using mlv_rec

Frame 000000 has the top 1 cm of the image displaced a few pixels to the right but is otherwise OK.
Frame 000001 has the top 1cm corrupted with the balance of the image looking normal
Frame 000002 No Corruption and the image looks normal.
Frame 000003 On first glance looks normal but upon closer inspection it is in fact corrupted. It is in fact a blended image with the top 1 cm of the image being
                       from one image and the balance from another (statc image).
Frame 000004 No Corruption and the image looks normal.
Frame 000005 Like frame 000003 it is a blended image with the top 1 cm of the image being from one image and the balance
                       from another (static image).

All subsequent  pairs of frames i.e: 000006&000007, 000008&000009 etc. behave the same as frames 000004&000005.

What is actually happening with the 12bt mlv? It is similar but different to what is happening with the the 10bit mlv.  For the 12bit mlv, the first two frames 000000&000001 are corrupt. From there on in every odd numbered frame is corrupt (a static repeating image) whilst every even numbered frame is ok. I proved this by:

   - Deleting all of the odd numbered dng frames.
   - Imported the even numbered dng frames in PowerDirector (corrupt frame 000000 is included)
   - Exported to an 1920 x 1080p mpeg4 video.

File Links

The below links are extracts from the original 10 and 12bit mlv files and contain 24 frames each. Also I have add ed link to the 12 bit video generated from PowerDirector.

10bit mlv extract - https://www.dropbox.com/s/hx61s68c4mcjdxq/7d_10bit_extract.mlv?dl=0

12bit mlv extract - https://www.dropbox.com/s/kp31jkj4vsun6t9/7d_12bit_extract.mlv?dl=0

MPEG4 from 12bit (even numbered frames) - https://www.dropbox.com/s/zg4hpbiqxlponuk/7D_14bit_test.mp4?dl=0 


Hopefully this helps the devs get a step closer to resolving the 10/12bit raw for the 7D. 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on January 11, 2017, 12:35:14 PM
@ottoga:
7D is only confirmed (http://www.magiclantern.fm/forum/index.php?topic=9848.msg177514#msg177514) to produce "good" results with 10/12 bit when using crop/zoom mode. Otherwise you will get corrupted frames
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Deadcode on January 11, 2017, 12:41:00 PM
Probably a beginner thought, im not even at novice lvl in coding, but it's like a synchronisation slip/clock error happens when you chop 2/4 bits from the incoming stream while recording 10/12 bit with 50D/7D/5D2
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on January 11, 2017, 01:36:46 PM
a1ex has already explained it: http://www.magiclantern.fm/forum/index.php?topic=5601.msg174181#msg174181

If your camera has the "raw slurp" method implemented, everything will work. If it does not, then the solution is to implement it. That requires some reverse engineering. If slurp is not implemented on your camera, the fact that you get any good frames at all is just luck (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175902#msg175902).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 11, 2017, 01:38:49 PM
I,d say run all footage with black level 2048(5D mark III). A1ex seems to agree, at least counting from yesterday https://bitbucket.org/hudson/magic-lantern/commits/e3ef8c75cf5de2e4018098d21ae6d1d3bcd183d5

Only applies to mlv_dump builds containing that change (such as the one from http://builds.magiclantern.fm/utilities.html ).

Older mlv_dump builds would require 2048 for 14 bpp, 2046 for 12 bpp and 2040 for 10 bpp.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on January 11, 2017, 03:23:52 PM
@A1ex

Did you saw my post  (http://www.magiclantern.fm/forum/index.php?topic=17795.msg177218#msg177218) about this issue in blown highlight when using mlv_dump for fix verticals stripes ?

It is a known issue ? Or maybe it is me who has not understood something.

When using MLVFS all is ok about stripes fix, even for blown highlight.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 11, 2017, 04:02:33 PM
Here is the latest mlv_dump for mac.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 11, 2017, 07:28:39 PM
@Danne @A1ex and Raw Devs.

Samples of 5d3 build Jan6 in 10/12/14 bits with MLV_DUMP with black fix at 2040, 2046, 2048 in all bit dephs.

Inside the folder I'd included MLV_DUMP and MLV.MO for check versions.

Let's go foward!

https://www.dropbox.com/sh/ruu0sclnxb7xk6m/AAAYRuRuLjlSAjz9lmOsL8dCa?dl=0

 :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 11, 2017, 08:45:13 PM
@dfort
Further 7D testing (croped) magiclantern-raw_video_10bit_12bit.2017Jan05.7D203
Croped mode generates proper dngs at 10/12/14 bits.
LiveView during recording (at 5x) goes to Canon View but sometimes locks up (no motion) in GrayScale.
10x zoom does not work.  It can be made to work if in LiveView at 5X yo enter ML menue and then return to LiveView.  Pressing zoom again will bring 10x, and then pressing again will go back to 1X.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on January 12, 2017, 06:37:15 PM
Is there a working 5d3 123 version with mlv 10,12,14bit and crop rec version available to try? I downloaded the one from experiments but it only had Mlv-Lite module for raw.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 12, 2017, 07:08:05 PM
mlv_rec not available yet, only with raw_rec available here: https://builds.magiclantern.fm/experiments.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 13, 2017, 12:48:39 AM
Black level issues should be fixed now; no more trickery should be needed with latest builds.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 13, 2017, 04:49:02 AM
@a1ex,
" no more trickery" means no more mlv dump black fixes?

What's next? Mpeg?

:)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 13, 2017, 08:21:48 AM
New builds on the Experimentals download page (https://builds.magiclantern.fm/experiments.html) are ready. I took down the builds I've been hosting. These cameras do not yet support CONFIG_EDMAC_RAW_SLURP but when someone puts up a pull request I'll try to compile a build for testing.

5D2.212
7D.203
50D.109
550D.109
500D.111
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Licaon_Kter on January 13, 2017, 08:55:25 AM
So close to merger, thanks for the continous development.  :o
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 13, 2017, 05:44:34 PM
Had a shoot today. First day with latest nightly and today with Jan 11 10-12bit build. External monitor did work at first (worked without any problem on Nightly), some shots were made then later the monitor only froze when connecting to the 5D3 with that build. Anyone experienced the same? Monitor was an Small HD DP6 5,6".

Otherways the build worked great!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on January 13, 2017, 06:41:13 PM
@dfort: i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Sganzerla on January 13, 2017, 10:21:58 PM
I'm going to post something that may be of interest, even if I'm not sure it is related to this area of the forum, or the 'post processing' one.

Using the experimental builds today - 5D MKIII 1.2.3 13th January - I tested the crop mode 3x3 720p and FPS override at 70fps (67fps) in 10/12/14bit modes.
Using raw2cdng software with "16bit CDNG maximized" option ticked, I opened the 3 files in Photoshop (ACR) and noticed that the blue cast in the shadows is not there anymore, but there are differences related to the magenta/green cast now, being 10bit files greener than 14bit files. I don't know if it is possible to see the exact difference from the image bellow, but there is a link to download these files if anyone is interested.

10bit TOP
12bit MIDDLE
14bit BELLOW

(http://i1063.photobucket.com/albums/t506/Sganzerla/cdng-differences_zps7mny8ail.jpg)

The download link of the 3 files is here:
https://goo.gl/YE4hcN (https://goo.gl/YE4hcN)

The difference is not solved with tint adjustment, unfortunately.

By the way, I don't know if people have tried this and seen before or not, but on a side note, related to software, MLV Mystic is producing noticeable more magenta images than raw2cdng from the same 14bit files too. So I don't know if it is something software related or code related.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 13, 2017, 10:28:04 PM
Any difference if you convert with latest mlv_dump from https://builds.magiclantern.fm/utilities.html ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 13, 2017, 11:40:53 PM
Is there a working 5d3 123 version with mlv 10,12,14bit and crop rec version available to try? I downloaded the one from experiments but it only had Mlv-Lite module for raw.

Top of the Experimentals (https://builds.magiclantern.fm/experiments.html) download page. That build includes mlv_rec.

@dfort: i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)

You don't need to make a pull request to test it. I took down the builds I made for cameras that don't have CONFIG_EDMAC_RAW_SLURP which the 5D2 is one of them. @reddeercity made a very complete test of that camera so we have a good idea of what's working and what still needs to be done. If you want to do some additional testing I'd suggest getting in touch with him.

@g3gg0

There is an issue with raw_twk on the EOSM. When mlv_rec, mlv_snd, mlv_play and raw_twk are loaded and MLV Sound is turned on a "Threads failed to start" error appears on the screen and the camera must be shut down or it may even require a battery pull. This happens with any bit depth setting for picture and sound. If the raw_twk module is not loaded this will not happen.

(https://c1.staticflickr.com/1/333/32252532236_662daf5049.jpg)

I was unable to reproduce this on the 700D so this issue may be specific to the EOSM.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 14, 2017, 01:11:41 AM
yeah, already mentioned a few times.
the EOS M has very few threads available and some modules need threads for their work.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Sganzerla on January 14, 2017, 01:23:55 AM
Any difference if you convert with latest mlv_dump from https://builds.magiclantern.fm/utilities.html ?

Thanks a1ex for the insight. I tought mlv_dump was a software for developers!
If I make it work, I'll come back with info. I think my English is not helping very much on this issue, but I'll try to find what is happening.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 14, 2017, 01:33:48 AM
mlv_dump is a reference implementation on how to read MLV files and how to extract the frames.
it is by far not the most powerful toy to play with MLV files, but it offers a few advanced features.

so when in doubt, use mlv_dump. if its output is unexpected then we can help :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 14, 2017, 01:46:15 AM
Mlv_dump, the more powerful tool imo. Strong features. Now working with 10/12/14 bits without the need to specify black/white level. And also:
- darkframe handling
- flatframe handling
- Possible to parallelize exporting processing
- and more...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 14, 2017, 03:50:55 AM
yeah, already mentioned a few times.
the EOS M has very few threads available and some modules need threads for their work.

Sorry, I searched the forum before posting but came up with the other definition of threads.

it is by far not the most powerful toy to play with MLV...

The definition of modesty.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on January 14, 2017, 04:35:29 AM
threads = tasks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 14, 2017, 06:15:45 AM
i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)
@ch_d no problem , I'll host the 5D2 10-12bit experimental build in my bitbucket account for the next 4-6 weeks
Or until 5D2 and older cams have liveview functional in 1:1 mode.
This build is the one I use in full mlv+audio in crop mode (2.61 crop) 2144x1076 23.976p @ 10bit + audio. (65MB/s write speed)
Please note 1:1 liveveiw is not working while recording raw bit reduction (corrupt frames) 14bit Ok .
With the build there the raw_twk.mo to play back 10-12bit in camera so it must be loaded with mlv_play & .mlv or .raw(mlv lite).
10bit_12bit_raw_twk.2016Dec01.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip) by chance if I can figure out the problem
with the liveview in bit reduction  ::) then I'll share my builds here.

If older cams e.g. 7D , 50D , 550D 500D user wish to try bit reductions with limited functionality ( I know 7D works in crop mode like the 5D2 but not sure about the rest)
I can post those cams builds also if requested , there will be from dec.1/2016 same as the one I posted for 5D2




Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ch_d on January 14, 2017, 01:28:08 PM
1:1 not working: That´s a pity. I don´t like the crop mode (because of FOV / DOF).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: g3gg0 on January 14, 2017, 02:28:32 PM
@dfort:
yeah sorry, threads and tasks can be used as a synonym in our case, as there is no memory segregation/separation or anything between them.
threads are known as kind of light weight tasks in context of the parent task.
i should use task as the word canons os uses.


good to hear that you all like mlv_dump how it is today :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: GutterPump on January 14, 2017, 03:28:31 PM
For those who use mlv_dump and who like to do things simply, I recommend this tutorial (http://www.magiclantern.fm/forum/index.php?topic=10526.0)

It is easy also to make different styles of dng extraction commands in the folder "send to" by duplicating .cmd files.
It's interesting when we need to do some tests with news features.
(http://image.noelshack.com/fichiers/2017/02/1484403957-mlvdump.jpg)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 15, 2017, 03:12:07 PM
www.youtube.com/watch?v=__k0o0ycYk0

Test of 10 bit raw video recorded on Canon EOS 600D during daylight ISO 100.
Resolution: 1728x672, 23,97fps, ISO 100, raw_rec.mo
BUILD/dfort/: magiclantern-raw_video_10bit_12bit.2017Jan13.600D102
I recorded 23 raw video files. Three files were with a "green cast".
Command: exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dng didn't fix it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 15, 2017, 05:47:23 PM
Try the latest builds from the Experiments (https://builds.magiclantern.fm/experiments.html) download page. I believe there were some recent changes to the way black level is handled.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: chmee on January 15, 2017, 06:17:29 PM
@Sganzerla
as @g3gg0 said, mlv_dump is the reference, all other tools are written by "independent" coders. so, if other tools are in need of a fix, write them (as i am the coder of raw2cdng)

regards chmee
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 15, 2017, 08:30:35 PM
Can someone post the link, or a list with the commands lines of Mlv_dump?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: honza123 on January 15, 2017, 08:39:42 PM
My examples of "green cast" - black level. Original and after fix
Shot with EOS 600D:
Resolution: 1728x672, 23,97fps, ISO 100, raw_rec.mo, magiclantern-raw_video_10bit_12bit.2017Jan13.600D102,
http://www.uschovna.cz/zasilka/LIVYCU4ETGKEGXFD-ZAK/
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on January 15, 2017, 08:40:33 PM
Can someone post the link, or a list with the commands lines of Mlv_dump?
If you run mlv_dump without any other options, it will print that for you.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on January 15, 2017, 08:53:46 PM
Usage: mlv_dump [-o output_file] [-rscd] [-l compression_level(0-9)] <inputfile>

Code: [Select]
Parameters:
 -o output_file      set the filename to write into
 -v                  verbose output
 --batch             output message suitable for batch processing

-- DNG output --
 --dng               output frames into separate .dng files. set prefix with -o
 --no-cs             no chroma smoothing (default)
 --cs2x2             2x2 chroma smoothing
 --cs3x3             3x3 chroma smoothing
 --cs5x5             5x5 chroma smoothing
 --no-fixcp          do not fix cold pixels
 --fixcp2            fix non-static (moving) cold pixels (slow)
 --no-stripes        do not fix vertical stripes in highlights

-- RAW output --
 -r                  output into a legacy raw file for e.g. raw2dng

-- MLV output --
 -b bits             convert image data to given bit depth per channel (1-16)
 -z bits             zero the lowest bits, so we have only specified number of b
its containing data (1-16) (improves compression rate)
 -f frames           frames to save. e.g. '12' saves the first 12 frames, '12-40
' saves frames 12 to 40.
 -A fpsx1000         Alter the video file's FPS metadata
 -x                  build xref file (indexing)
 -m                  write only metadata, no audio or video frames
 -n                  write no metadata, only audio and video frames

-- Image manipulation --
 -a                  average all frames in <inputfile> and output a single-frame MLV from it
 --avg-vertical      [DARKFRAME ONLY] average the resulting frame in vertical direction, so we will extract vertical banding
 --avg-horizontal    [DARKFRAME ONLY] average the resulting frame in horizontaldirection, so we will extract horizontal banding
 -s mlv_file         subtract the reference frame in given file from every single frame during processing
 -t mlv_file         use the reference frame in given file as flat field (gain correction)

-- Processing --
 -e                  delta-encode frames to improve compression, but lose random
 access capabilities
 -X type             extract only block type
 -I mlv_file         inject data from given MLV file right after MLVI header
 -c                  (re-)compress video and audio frames using LZMA (set bpp to
 16 to improve compression rate)
 -d                  decompress compressed video and audio frames using LZMA
 -l level            set compression level from 0=fastest to 9=best compression

-- bugfixes --
 --black-fix=value   set black level to <value> (fix green/magenta cast). if no
value given, it will be set to 2048.
 --white-fix=value   set white level to <value>. if no value given, it will be s
et to 15000.
 --fix-bug=id        fix some special bugs. *only* to be used if given instructi
on by developers.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on January 16, 2017, 01:56:31 AM
Wonderfull guys!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ansiivan on January 17, 2017, 01:20:33 AM
Hi. Why regular chamber mov, much brighter raw vide? The settings are the same.
Canon 1100d,  magiclantern-raw_video_10bit_12bit.2017Jan13.1100D105.zip
Fps 25, iso1600.
mov

Fps 25, iso1600.
raw video
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on January 17, 2017, 12:32:36 PM
Hi guys,

on Saturday, I've sucesfully shot 2TB of MLV's in 10bit. From 1400 filex, approx 150 had wrong black level. Since calling mlv_dump 150 times and then sorting all the files would be too complicated, I've wrote little windows batch script. I am not a programmer so finding the first file in a folder and even the iteration is very very messy. But works. :)

Code: [Select]
@echo off
setlocal EnableDelayedExpansion
set mypath=%~dp0
set workingDir=E:\footage\mlv\
set fixDir=E:\footage\cdng-fixed\
set moveDir=E:\footage\mlv-processed\


E:
cd %workingDir%
for /r %%i in (*) do (
set file=%%i

!mypath!\mlv_dump.exe --black-fix=2040 --dng "!file!"

for %%A in ("!file!") do (
set foldername=%%~dpA
set filename=%%~nxA
)
echo !filename!

cd "!fixDir!"
md "!filename!"
cd "!filename!"

for /r "%workingDir%" %%f in (*.dng) do move "%%f" >nul
for /r "%workingDir%" %%f in (*.wav) do move "%%f" >nul

move  "!file!" "!moveDir!"
)
D:
cd %mypath%
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on January 17, 2017, 03:04:54 PM
BTW, may I process *all* files with --black-fix=2040, even those which have correct value? If I want to process only those wrong ones, I need to extract all and visually check.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 17, 2017, 05:44:17 PM
On 5D3 and probably all other recent cameras, latest mlv_dump requires --black-fix=2048, regardless of ML build (since this post) (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178162#msg178162).

With latest ML bulid from the Experiments page AND latest mlv_dump, there should be no need to change the black level (since this post) (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178319#msg178319).

Checklist:
- with 10-bit files, the black level from MLV (visible with mlv_dump -v) should be 128 (127 is wrong)
- with 14-bit files, the black level should be close to 2048 (small variations are OK)
- the black level from the DNG (always upconverted to 14 bits) should be close to 2048 (e.g. 2050 is fine, 2032 is wrong)
- there should be no color difference between 10 and 14 bit files in shadows

If any of the above is not true for you with the latest experimental build and latest mlv_dump, please report.

For authors of other postprocessing tools: to avoid color casts at lower bit depths, conversion to a higher bit depth must be done by adding 0.5 LSB to the input signal (explained here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177802#msg177802)). Black level scaling requires no trickery (e.g. 128 -> 2048).

Relevant commits: e3ef8c7 (https://bitbucket.org/hudson/magic-lantern/commits/e3ef8c75cf5de2e4018098d21ae6d1d3bcd183d5) and a31228c (https://bitbucket.org/hudson/magic-lantern/commits/a31228c7b7bed6805e3dc5e9e0b894ed75b033b4).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: vstrglv on January 17, 2017, 05:53:10 PM
Build  magiclantern-crop3x.2017Jan13.5D3113 (2017-01-12 23:35) – crop+10bit,12bit. MLV raw.
DNG conversion raw2cdng 1.7.9 and  mvl_dump (2017-01-13 00:55).
DNG for crop(1x1)_Xbit_ raw2cdng and DNG for crop(1x1)_Xbit_ mlv_dump look very similar (except  green colour cast for 10bit raw2cdng)  in ACR if Noise Reduction (Luminance) is set to zero. But if Noise Reduction is increased its effect is more noticeable  for mlv_dump convertion.
X is 10bit, 12bit or 14bit. This effect takes place for normal mode (not crop) too.
https://www.dropbox.com/s/q6pj4kspt8l794t/M14-1338_10b_crop_mvl_dump.dng?dl=0
https://www.dropbox.com/s/i6vpllko75d43wl/M14-1338_10b_crop_raw2cdng.dng?dl=0

(https://c1.staticflickr.com/1/737/32218454142_cc0b3d211c_o.jpg) (https://flic.kr/p/R62ZWE)M14-1338_10b_crop_0NR_raw2cdng (https://flic.kr/p/R62ZWE) by Vladimir Strigalev (https://www.flickr.com/photos/141116628@N06/), on Flickr
(https://c1.staticflickr.com/1/330/32329871916_1d88800d55_o.jpg) (https://flic.kr/p/RfT3yL)M14-1338_10b_crop_0NR_mvl_dump (https://flic.kr/p/RfT3yL) by Vladimir Strigalev (https://www.flickr.com/photos/141116628@N06/), on Flickr
(https://c1.staticflickr.com/1/450/32218698302_b7129a1bcc_o.jpg) (https://flic.kr/p/R64fwj)M14-1338_10b_crop_30NR_raw2cdng (https://flic.kr/p/R64fwj) by Vladimir Strigalev (https://www.flickr.com/photos/141116628@N06/), on Flickr
(https://c1.staticflickr.com/1/277/32368983145_3e9e3e18df_o.jpg) (https://flic.kr/p/RjkuYi)M14-1338_10b_crop_30NR_mvl_dump (https://flic.kr/p/RjkuYi) by Vladimir Strigalev (https://www.flickr.com/photos/141116628@N06/), on Flickr
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kaco on January 17, 2017, 07:09:42 PM
On 5D3 and probably all other recent cameras, latest mlv_dump requires --black-fix=2048, regardless of ML build (since this post) (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178162#msg178162).

With latest ML bulid from the Experiments page AND latest mlv_dump, there should be no need to change the black level (since this post) (http://www.magiclantern.fm/forum/index.php?topic=5601.msg178319#msg178319).

Great! However I still find batch calling mlv_dump much better then mlv-fs. Is there any way how to make real cdng from mlv_dump? So that Resolve could read audio and correct fps? Or this wouldn't work because mlv_dump is not indexing the files?

Thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 18, 2017, 02:10:40 AM
Resolve read Cdng with audio from mlvfs as one file with correct level in real time on PC's and Mac's .
batch calling on command is very slow plus you end up with lots of dng's that fill up hard drive space .
MLVFS doesn't do that it produces Virtual Cdng so no hard drive killer here , you even read off the CF or SD cards
depending on your computer hardware .
FYI: mlv_dump produces compliance dng that work in resolve or at least used to that could be broken thou with bit reduction
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 18, 2017, 02:48:08 AM
Quote
Great! However I still find batch calling mlv_dump much better then mlv-fs. Is there any way how to make real cdng from mlv_dump?
You can process cdng files with this
http://www.magiclantern.fm/forum/index.php?topic=15108.msg146822#msg146822
 
The program works with a mlv_dump version which has code mixed in from a specific ml-dng branch version. Code updated to latest 10/12/14 bit the same as latest mlv_dump(g3gg0/a1ex). Works with resolve(audio embedding).
When installed head over to the mlv_dump menu (m) from the main menu and you can do setting changes from there such as black/white level etc.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 19, 2017, 12:24:50 PM
Using 13th of Jan build and the frame freezes in black and white live view while shooting raw_rec with 5x. Any others experiencing the same?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tracerman on January 19, 2017, 01:11:37 PM
@dfort

Hi, I'm up for testing 10/12 bit mlv_rec on 550D if you like (I don't know how to make a pull request either :/).

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 19, 2017, 01:19:19 PM
Is this happening with or without raw_twk enabled? Try with raw_twk disabled.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Levas on January 19, 2017, 03:53:58 PM
There 's only a windows version of MLV_dump available on the magic lantern downloads page:
https://builds.magiclantern.fm/utilities.html (https://builds.magiclantern.fm/utilities.html)

Can somebody add newest version of MLV_dump for osX ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 19, 2017, 04:16:46 PM
mlv_dump for mac
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on January 19, 2017, 10:36:42 PM
Is this happening with or without raw_twk enabled? Try with raw_twk disabled.
Still happening with raw_twk disabled. Guess its something with the build?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 20, 2017, 08:33:57 AM
@dfort

Hi, I'm up for testing 10/12 bit mlv_rec on 550D if you like (I don't know how to make a pull request either :/).

Nothing new for the 550D yet and we know what needs to be done before we're ready for any further testing.


Sent from my iPad using Tapatalk
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on January 20, 2017, 02:01:04 PM
i dont see 10 bit for 50D   ???
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 20, 2017, 06:49:04 PM
mlv_dump for mac updated
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump

fixes here(dark/flatframe, mlv_lite)
https://bitbucket.org/hudson/magic-lantern/commits/89daf093c81fbe86be3ae27e4acce8fa79bf91f3
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on January 20, 2017, 07:31:31 PM
Thanks SO much for sharing this @Danne and for the quick fixes @a1ex!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tracerman on January 21, 2017, 12:30:33 PM
@dfort

Thanks for your reply, feel free to let me know when/if you need testers for 550D.

Cheers
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: justinbacle on January 21, 2017, 04:44:04 PM
i dont see 10 bit for 50D   ???
50D is not supported (yet).
(Same as 5dmk2 as they don't have CONFIG_EDMAC_RAW_SLURP ?)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bobbyshots2 on January 21, 2017, 07:14:30 PM
I want to try and test the t2i (550d) with 10bit - 12bit raw. If anybody has a link to  the module they could send me or the full ml download. that would be great.!
Thanks in advance!
One Love
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on January 21, 2017, 07:35:51 PM
Ok, mlv_dump works to get me DNG.
I can colour correct usingbut how to get them into Ppro?  If I save as JPEG from LR, it complains that bit depth is not supported.

Is there a way to get 10bit scaled to 14bit depth?

Found that DNG works very nice in Resolve.
However LR becomes useless in this flow, since its corrections are not recognized when DNG is input to Resolve.
One can export from LR into jpeg but that limits Ppro or Resolve's abilities.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 21, 2017, 08:05:18 PM
Looks like it is worth repeating that the following cameras need to get the CONFIG_EDMAC_RAW_SLURP treatment before it is worth doing any further tests with them.

5D2.212-Digic 4
7D.203-Dual Digic 4
50D.109-Digic 4
500D.111-Digic 4
550D.109-Digic 4
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 22, 2017, 02:29:02 AM
50D is not supported (yet).
(Same as 5dmk2 as they don't have CONFIG_EDMAC_RAW_SLURP ?)
Not really true , not sure about 50D but the 5D2 work in Crop mode 100% (10-12bit) with Audio , overlays  in MLV full  see the Proof (http://www.magiclantern.fm/forum/index.php?topic=11205.msg177454#msg177454)
Even thought 1:1 liveview products some corruption in 10bit but 12bit can be used if you crop out
the top 50-100 line do to a color noise bar see here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg175483#msg175483) look at the 3rd image I posted .

I can't say that the 50D or 550 (T2i) will have the same level of functionality in crop mode as the 5D2 but as @dfort said
..... need to get the CONFIG_EDMAC_RAW_SLURP treatment before it is worth doing any further tests with them.

We do need more user's who can help figure the Digic 4 liveview issue (5d2,50d,7d,500,550) so I will provide the link to those builds in hope more development can be made .
My reasoning is if users have at least a limited functioning version of  raw video bit reduction , this hopefully will give more incentive to push development on the digic 4.
link to my bitbucket downloads for 10-12bit reductions 5d2,50d,7d,500,550 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/)

 

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Nikolas on January 22, 2017, 10:53:56 AM
Thanks to developers for latest build for 1100d with 10/12bit raw video (2017-01-20)! Compared with the previous version (december 2016), record of raw video works very stable. 10/12 bit really is supported. in the previous build had a lot of colored dots, now they have disappeared.

But in the latest build, strong processing gives vertical banding (pattern noise?). Like i said, in the previous version were only colored dots, vertical banding been absent. Image quality with hard processing is very important for me (I use raw video for astrophotography).

I tries to use different combinations of recording settings and to use subtraction dark frames, but it not work.

Dear developers, Is it possible to fix in the next build? Or how i can make a correction of these bands?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on January 22, 2017, 12:11:59 PM
Interesting.
How are you subtracting darkframes? Can you upload a darkframe and a clip with banding?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Greg on January 22, 2017, 05:24:29 PM
Code: [Select]
#ifdef CONFIG_80D_101
#define DEFAULT_RAW_BUFFER MEM(0x66B0 + 0x54)
#endif
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Sganzerla on January 23, 2017, 12:57:22 AM
@a1ex
@g3gg0
@GutterPump
@chmee
@RenatoPhoto


 Regarding my post [b#1081]]#1081 (http://)[/b] I finally (after some struggle) learnt how to process .MLV with mlv_dump and could see that there is no color cast between 14bit and 10bit files using this software.

Thanks for all of your input!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 24, 2017, 05:34:38 AM
The Bit depth in raw_rec and mlv_rec are slightly different:

(https://c1.staticflickr.com/1/778/32372240161_023bc27e4b.jpg)

(https://c1.staticflickr.com/1/579/32372240041_24667767bd.jpg)

Note that mlv_rec says "bpp" for bits per pixel.

In addition, the up/down buttons go the opposite direction in raw_rec so I made a small change and put in a pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/808/conform-raw_rec-bit-depth-menu-to-match/diff).

(https://c1.staticflickr.com/1/454/32115323320_0a7413a16e.jpg)

(https://c1.staticflickr.com/1/521/32115323440_abc5798e4f.jpg)

Hope you like the small improvements and hope this doesn't break anything--tested and working fine over here.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on January 24, 2017, 09:04:48 AM
Canon 50D_01-14-2017 109 Firmware Test

All Processed w/ CMD and MLVFS:

NOTES:
     
             RAW REC 1.1 works in 14-Bit Full Sensor Readout and Canon Preview.

             MLV 2.0 works in 14-bit Full Sensor and 5X Zoom w/Canon Preview. However, it is super close to working in 12-bit.



- - - - - - - - - - - - - - - RAW RECORD 1.1 LITE - - - - - - - - - - - - - - -

10-Bit - Full Sensor Readout - - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Repeating Frame; every other frame. Top 1/3 of the image is an inverted data sector / bad write.

10-Bit 5X Zoom - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Purple Frames. Repeating Frame; every other frame. Top 1/3 of the image is separated from bottom 2/3. Bottom 2/3 features a "wiggly zoom-in" distortion appearing in a timed pattern.


- - - - - - - - - - - - - - - RAW RECORD 1.1 LITE

** Best Results w/ 5X Zoom:

12-Bit Full Sensor Readout - - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Repeating Frame; every other frame. Top 1/6 of the image is an inverted data sector / bad write (still repeats image).

12-Bit 5X Zoom - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W do not work - -
   RESULTS: Purple Frames. Repeating Frame; every other frame. Top 1/3 of the image is separated from bottom 2/3.


- - - - - - - - - - - - - - - RAW RECORD 1.1 LITE

14-Bit Full Sensor Readout - - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Full Sensor Readout is great.

14-Bit 5X Zoom - - RAW RECORD 1.1 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS:  Purple Frames. Repeating Frame; every other frame. Top 1/3 of the image is separated from bottom 2/3. Bottom 2/3 features a "wiggly zoom-in" distortion appearing in a timed pattern.



- - - - - - - - - - - - - - - MLV 2.0 - - - - - - - - - - - - - - -


14-Bit Full Sensor Readout - - MLV 2.0 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Full Sensor Readout is great.

14-Bit 5X Zoom - - MLV 2.0 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: 5X Zoom Results Record Great. However, FRAMING IS OFF.


- - - - - - - - - - - - - - - MLV 2.0

12-Bit Full Sensor Readout - - MLV 2.0 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Top 1/8 of the image is a repeating inverted data sector / bad write every other frame.

12-Bit 5X Zoom - MLV 2.0 - - Canon Preview Only - Hacked / B&W do not work - -
   RESULTS: Top 1/3 of frame appears to be separated from both 2/3 and repeating every other frame. NO PURPLE FRAMES.


- - - - - - - - - - - - - - - MLV 2.0

10-Bit Full Sensor Readout - - MLV 2.0 - - Canon Preview Only - Hacked / B&W Freeze Image on LCD - -
   RESULTS: Repeating Frame; every other frame. Top 1/3 of the image is an inverted data sector / bad write.

10-Bit 5X Zoom - MLV 2.0 - - Canon Preview Only - Hacked / B&W do not work - -
   RESULTS: Top 1/3 of frame appears to be separated from both 2/3. NO PURPLE FRAMES.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 24, 2017, 09:13:03 AM
Looks like it is worth repeating that the following cameras need to get the CONFIG_EDMAC_RAW_SLURP treatment before it is worth doing any further tests with them.

5D2.212-Digic 4
7D.203-Dual Digic 4
50D.109-Digic 4
500D.111-Digic 4
550D.109-Digic 4

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Lars Steenhoff on January 24, 2017, 11:05:39 AM
For any tests done I think its worth mentioning wich memory card is used to isolate any card speed issues.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 24, 2017, 12:23:25 PM
... for any *speed* tests (hint, hint) (http://www.magiclantern.fm/forum/index.php?topic=17091)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: LEVISDAVIS on January 24, 2017, 06:24:38 PM
The internal temp reading on the 50D is reading, "Calm and collected."

Komputerbay 128 1066x, by the way.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bobbyshots2 on January 28, 2017, 07:37:29 PM
Iv been testing the  10-12 bit on the 550d (t2i)
I get good results on the 12 bit using mlv 2.0
on the 10 bit the top part of the frame is jittery or "repeating" as i see someother user posting
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hjfilmspeed on January 29, 2017, 05:29:34 AM
I am super impressed by the Jan 13 experimental build the Incorporates the 10bit and crop mode. 10bit is a dream to work with. If we could just get a better preview for high resolution. But 10 bit has been working flawlessly for me. So many possibilities here for higher fps and resolution bravo.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Tracerman on January 29, 2017, 11:00:20 AM
@bobbyshots2 what build did you use ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on January 29, 2017, 04:32:16 PM
Quote from: dfort
Looks like it is worth repeating that the following cameras need to get the CONFIG_EDMAC_RAW_SLURP treatment before it is worth doing any further tests with them.

5D2.212   -   Digic 4
7D.203   -   Dual Digic 4
50D.109   -   Digic 4
500D.111   -   Digic 4
550D.109   -   Digic 4

Sorry for fullquote. You highlighted 50D. Work is done for other cams?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 29, 2017, 04:40:04 PM
Nope.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: bobbyshots2 on January 29, 2017, 10:51:54 PM
I am using the download from the link that reddeercity posted previously.
the build say 01-12-2017
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads

* i notice now that there is a new date 01-14-2017 that i will have to try.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Walter Schulz on January 29, 2017, 11:00:25 PM
@a1ex and dfort: 7D here but no clue. Not that fluent in code. Directions, please!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on January 29, 2017, 11:08:36 PM
Unsuccessful on 5D2 after trying for a couple of evenings...

For new models, it's pretty straightforward if one looks at the commits. The 1200D is probably the only new model not yet ported. Old models have a very different codebase for LiveView (though the hardware is likely to work in the same way).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on January 30, 2017, 07:42:45 PM
@a1ex and dfort: 7D here but no clue. Not that fluent in code. Directions, please!

Me neither! Fluent in code that is. I helped a bit with the 700D but only because it is very similar to the EOSM which dmilligan previously figured out. Those older models are proving very difficult to crack. There's an 1100D build on the Experiments (https://builds.magiclantern.fm/experiments.html) download page but I haven't seen any reports on how well it is working. Though looking that camera probably won't get you very far with the 7D or 5D2 because it is a slightly newer model.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on January 31, 2017, 02:11:04 AM
Unsuccessful on 5D2 after trying for a couple of evenings...
Old models have a very different codebase for LiveView (though the hardware is likely to work in the same way).
Sounds encouraging  :)

At the risk of not knowing anything about liveview code (over my head) is there something my 5d2 or myself can help with ?
I can compile , I understand some of the code but I get easily confuse at times but willing to try . 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 04, 2017, 06:23:12 AM
Ok doing some research on the liveview problem on the 5D2 in 1:1
Come across wiki Register_Map (http://magiclantern.wikia.com/wiki/Register_Map) found Liveview Registers stuff .

Quote
0xC0F140C0 does luma scaling
0xC0F140C4 does chroma scaling
0xC0F140C8 sets the bmp overlay mode
0xC0F140E8 [s32] pixels to skip - causes distorted image
0xC0F140F0 [s32] pixels to skip - causes distorted image
0xC0F37014 LV raw type (see lv_af_raw, lv_set_raw) on DIGIC5 (DIGIC4 uses PACK32_ISEL)
0xC0F371FC unpacking mode (see PACK16_MODE) for the input module that feeds image data to lossless encoder

0xC0F37014 LV raw type seems to point to  "PACK32_ISEL" for digic4 (5d2) (from ML code I seen this is where digic5 cam gets configured with "LV raw type" if I'm not mistaken

Quote
0xC0F08090 - PACK32 Control register
0xC0F08094 - PACK32_MODE

0xC0F08114 - PACK32_ISEL (0-15)
Edit: Sorry posted wrong address  ::)
I see the variable (0-15) now
nice to know what those are .

How's can this be configured ?

Like this for the "PACK16_MODE" ?

Quote
0xC0F08120 - PACK16 Control register
0xC0F08124 - PACK16_MODE (mask 0x131)
             From ProcessPathForFurikake (raw_twk):
             -------x --xx----
                  111 (0x130) = 16-bit output
                  011 (0x030) = 14-bit output
                  010 (0x020) = 14-bit output
                  001 (0x010) = 12-bit output
                  000 (0x000) = 10-bit output

I hope this is on the right track , What do the developers think ?
If this is correct how would I processed from here.


Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 04, 2017, 10:37:38 AM
0xC0F08114 - PACK32_ISEL (0-15)

Just look it up in raw.c and earlier in this thread (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174177#msg174177) ;)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Markus on February 04, 2017, 02:18:18 PM
When recording on these new experimental builds the raw histogram switches over to the standard histogram on record. Have I missed some simple setting or is it i bug?

5d3 113 crop + 10-14bit experimental version.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 04, 2017, 02:26:17 PM
All raw overlays are currently hardcoded for 14-bit data. See this post (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177531#msg177531) for more details.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on February 05, 2017, 06:29:08 AM
raw_twk.mo issue with 70D

For 10 bit raw, if you want to have liveview playback working
you need to copy raw_twk.mo to your ML->modules folder on your card.
*** It seems to work for one pass, but 2nd clip it hangs your camera ***
*** Looks like that its not initializing properly for 2nd clip ***

here is a link for the raw_twk.mo I used.
https://www.dropbox.com/sh/arb43xa6svvjljr/AACyi51IrJfMGs3ZQ8EC_6hOa?dl=0

The modules that need to be enabled for it to work include
file_man
mlv_play
mlv_rec
mlv_send
raw_rec
raw_twk

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: zachnfine on February 05, 2017, 07:01:48 AM
This is likely a question that was answered earlier in the thread - but I searched around and didn't find it. With this new feature can one shoot 10-bit on the 5D3 at higher than 1080p resolution in full-frame mode (i.e. not crop-mode)?

I've been doing fine recording 1080p23.98 to my Komputerbay 128Gb cards (1 1050x, 1 1066x) at 14-bit, which requires 82.97Mb/s. But if 10-bit is possible, then 2272x1278 @ 23.98 would be nearly the same bitrate at 82.99Mb/s - and that might be a perceptible increase in detail, even if downscaled to 1080p in post, or at least it'd leave a little extra headroom for stabilization. And if 10-bit drops too much usable color information, 12-bit at 2064x1161 23.98 is 82.19Mb/s.

Or maybe the card's capable of more and I'm aiming low with that bitrate, I haven't benchmarked. And maybe card spanning would be a help. It'd be fun to see how high a resolution can be recorded at 23.98.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 05, 2017, 07:52:20 AM
Just look it up in raw.c and earlier in this thread (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174177#msg174177) ;)
Thanks @a1ex
Ok I think I got my head around this a little better ,
raw.c- line160 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/src/48bb172d28daa6e82247191f9cef94bda0a07217/src/raw.c?fileviewer=file-view-default#raw.c-160)
Quote
#ifdef CONFIG_DIGIC_V
#define RAW_TYPE_REGISTER 0xC0F37014
#else
#define RAW_TYPE_REGISTER 0xC0F08114    /* PACK32_ISEL */
#endif
I see "0xC0F08114" etc.... stuff , question;  with the "#ifdef CONFIG_DIGIC_V" then it has #else , does (#else) mean Digic4
or would it make a difference to call "#ifdef CONFIG_DIGIC_IV" or is that a way of writing less code ?

As I see how other newer cams are configured does there need to be a "#ifdef CONFIG_xxx" for the 5D2 ?
maybe like this ?
Code: [Select]
#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
#endif
From the Register Map "PACK32_ISEL" has  1-15 selectable variables
Would be right to put that variable in brackets ?
Or does "PACK32_ISEL" need to "#define" at all ?
Not sure how to write the Code/Syntax
Close you think , or way off base here .
Edit:
Because there 15 different variables does that mean the only way to figure it out is by trial & error ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kzv on February 05, 2017, 06:21:41 PM
Hello!

First time posting, so thank you for all your great work on Magic Lantern and especially this experimental version!

Been testing it and works great except one thing -in last build (and some previous in 2016) there is the problem with the ISO table. When you change ISO during recording, it wont hide.

Here is the picture: https://drive.google.com/file/d/0Bw9gcCutoLPQenBkMGJ4SUdxUjQ/view?usp=sharing

Can you please have a look at this bug? I would be very grateful.

Thank you very much in advance!

Chris
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: hyalinejim on February 06, 2017, 03:01:34 AM
Yes, this happened to me last night when I was shooting a performance  (5D3 1.1.3 10bit)

While shooting, pressing the physical ISO button brings up the ISO overlay. I use the shutter wheel to change ISO. But then the overlay stays on screen until the camera stops recording.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on February 06, 2017, 09:19:07 AM
You can push rate button and use joystick to change iso.

Thanks @a1ex
Ok I think I got my head around this a little better ,
raw.c- line160 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/src/48bb172d28daa6e82247191f9cef94bda0a07217/src/raw.c?fileviewer=file-view-default#raw.c-160)I see "0xC0F08114" etc.... stuff , question;  with the "#ifdef CONFIG_DIGIC_V" then it has #else , does (#else) mean Digic4
or would it make a difference to call "#ifdef CONFIG_DIGIC_IV" or is that a way of writing less code ?

As I see how other newer cams are configured does there need to be a "#ifdef CONFIG_xxx" for the 5D2 ?
maybe like this ?
Code: [Select]
#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
#endif
From the Register Map "PACK32_ISEL" has  1-15 selectable variables
Would be right to put that variable in brackets ?
Or does "PACK32_ISEL" need to "#define" at all ?
Not sure how to write the Code/Syntax
Close you think , or way off base here .
Edit:
Because there 15 different variables does that mean the only way to figure it out is by trial & error ?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on February 06, 2017, 04:34:21 PM
@reddeercity

Are you looking for the way to twick Canon zoom for less magnification than 3x?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on February 06, 2017, 05:13:49 PM
@budafilms I think reddeercity is trying to stop the live view from freezing on 5D2 when its recording(I always have that problem so I think its that)
@reddeercity have you had wrong(0 probably) black level in crop mode on your 5D2? it seems to be broken in 10bit-12bit branch since December, even shooting 14 bit, I keep getting a pink cast in crop mode no matter what settings or build (does MLVFS have a black level fix?)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 07, 2017, 06:16:04 AM
@reddeercity Are you looking for the way to twick Canon zoom for less magnification than 3x?
No but that that would be a great dream  :)
@budafilms I think reddeercity is trying to stop the live view from freezing on 5D2 when its recording(I always have that problem so I think its that)
Correct ! in 1:1 plus stop frame corruption but if liveview works properly hopefully that will fix that.

@reddeercity have you had wrong(0 probably) black level in crop mode on your 5D2?
it seems to be broken in 10bit-12bit branch since December, even shooting 14 bit,
I keep getting a pink cast in crop mode no matter what settings or build (does MLVFS have a black level fix?)

No , are you using full mlv or mlv lite (.raw) ?
The video I posted was crop 10bit 2.1K mlv+audio (Full)
I need Audio when I record raw , I have been using just full mlv
No , MLVFS does not are you on windows or Mac?
I Thing if I'm not mistaken MLVProducer (http://www.magiclantern.fm/forum/index.php?topic=15271.msg148538#msg148538) has W/B raw levels that can be adjusted on the fly in realtime "ACR" like  :D
Thou it's a windows base program it can be run under Wine on OSX ( I tried it on my MacPro4.1 on Yosemite)  works nice

The black level in crop has been like this for some time , it's has something to do with the mlv lite I think , not sure thou.
1:1 thou is ok in 10,12,14bit I'm sure , just not crop. If you read the mlv lite source code , there's some notes about it  I think.
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 07, 2017, 06:49:43 AM
After consulting with my IT friend asking for some basic help (I'm sure he's was laughing at me )
showing him my code sample , he return with a long email explaining why I should not try to code  :P :P
Will in the end he did give me a sample to work on
Code: [Select]
#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
if PACK_ISEL (3))
          {
           do something
           Return a valve
           }
#endif

this look better ? it's a work in process

I'm still learning ,  :)

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Licaon_Kter on February 07, 2017, 08:00:37 AM
Code: [Select]
if PACK_ISEL (3))
You're missing a ( after if I guess
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KelvinK on February 07, 2017, 01:35:31 PM
There's a very annoying bug in experimental build (was there since very first versions), at least for 6D: in LV when the play videos, it can play 2-3 then screen turns black. To get back GUI you've to press trash-bin button.

Would be great if it can be fixed.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on February 07, 2017, 02:48:53 PM
Yes this bug is confirmed.  I have reported it before for 5D3.
Video playback is limited to about 3 videos, probably a problem in raw_twk module.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: KelvinK on February 07, 2017, 03:23:29 PM
I don't think so. If you don't load raw_twk and record 14bit only, playback has same issue with 14b files.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 08, 2017, 01:55:04 AM
Code: [Select]
if PACK_ISEL (3))You're missing a ( after if I guess
Ok Thanks
So like this
Code: [Select]
#ifdef CONFIG_5D2
#define PACK32_ISEL (3)
if ( PACK_ISEL (3))
          {
           do something
           Return a valve
           }
#endif

Little closer , still need to find more on PACK32_ISEL stuff.
I haven tried to compile anything yet ...... soon , stay tune
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dmilligan on February 08, 2017, 04:06:55 AM
What exactly are you trying to do? The code you have posted is just nonsense (I'm not trying to be mean).

PACK32 is the name Canon has given to an image processing hardware module in the camera. PACK32_ISEL is the name Canon has given to a register (https://en.wikipedia.org/wiki/Hardware_register) for this module that controls its behavior. a1ex has already given an explanation of what is known about what this module does and what this particular register probably controls here (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174177#msg174177). Until you are at least to the point of thoroughly and completely understanding everything that is said in that message, there's no point in trying to write any code.

You can't just pick up a violin for the first time and expect to be able to play the Brahms violin concerto. If you really want to contribute and learn, you will have to start with something simpler and learn more about programming in general and ML specifically (by, for example, playing with Lua or doing an easy coding task).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 08, 2017, 04:49:05 AM
What exactly are you trying to do? The code you have posted is just nonsense (I'm not trying to be mean).
 
No I understand,
it's big learning curve there's a lot to take in and there is a lot that's over my head .
I'm trying to get liveview in 1:1 with bit reduction in work with out liveview freezeing up and not have it produce alternating corrupt frames .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 08, 2017, 05:28:39 AM
@reddeercity - I don't know much about coding or reverse engineering or playing the violin but one valuable lesson I learned from dmilligan is to look at the code. To get this 10bit/12bit thing working you need to figure out how to get CONFIG_EDMAC_RAW_SLURP working. Look at the internals.h file for any of the cameras that are supported and you'll see this:

internals.h
Code: [Select]
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#define CONFIG_EDMAC_RAW_SLURP

Ah if it were only as easy as adding that one line, but there's more to it.

Currently the 5D2 uses the LiveView raw buffer:

raw.c
Code: [Select]
/**
 * LiveView raw buffer address
 * To find it, call("lv_save_raw") and look for an EDMAC channel that becomes active (Debug menu)
 **/

#if defined(CONFIG_5D2) || defined(CONFIG_50D)
#define RAW_LV_EDMAC 0xC0F04508
#endif

Cameras that have CONFIG_EDMAC_RAW_SLURP use the default raw buffer instead of the LiveView raw buffer.

raw.c
Code: [Select]
#ifdef CONFIG_EDMAC_RAW_SLURP
/* undefine so we don't use it by mistake */
#undef RAW_LV_EDMAC

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
 *
 * raw_buffer = get_raw_buffer()
 * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
 * ...
 * dump_file(filename, raw_buffer, 7*something...)
 */

#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif

So all you're looking for is that one hexadecimal number to unlock this feature. The question is how do you find it? Read the code, it tells you. Now the hard part is following those instructions and come up with the right address.

Back to the violin analogy, how do you get to Carnegie Hall? Practice.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 08, 2017, 06:01:27 AM
@dfort Thanks ! :D
I read so much code my head is spinning and of course I when down the wrong rabbit hole .
It all make sense now , I kind of thought that's what I being missing that CONFIG_EDMAC_RAW_SLURP  RAW_BUFFER address stuff
I just couldn't understand if that was needed to be done on digic4 (5D2 etc..) cams like digic5 cams (5d3 etc..)
now I know thank again .
Can you hear my violin warming up thou it's a bit scratchy must be something wrong with my bow  :P   
When I can play the scales correctly , then I'll post 
 
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on February 08, 2017, 08:49:30 AM
Really your best bet is a1ex as he already confirmed that he tried to solve it for DIGIC IV (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176545#msg176545).

Maybe next time? (https://www.magiclantern.fm/forum/index.php?topic=5601.msg177584#msg177584)

You can try to solve it yourself but I can recommend to have a look into the state diagrams. Without having a look at them and understanding the vsync functions in ML you can also try to win a lottery before solving it for DIGIC IV. Finding the buffer sizes or address is just a look at the edmac screenshot and following the solution commited for 1100D... Just my 0.02$:

https://a1ex.bitbucket.io/ML/states/index.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 09, 2017, 08:37:06 AM
Thanks @nikfreak you have been very helpful .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on February 09, 2017, 03:29:34 PM
...following the solution commited for 1100D...

Which branch are you looking at for the 1100D solution? Is it in the allocate-raw-lv-buffer?

The way the 1100D is handled is completely different than the 60D which AFAIK is the only other Digic IV camera that's working with 10bit/12bit.

Code: [Select]
#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif
...

#ifdef CONFIG_1100D
#define RAW_LV_BUFFER_ALLOC_SIZE (3906*968)
#endif

The 1100D isn't in the crop_rec or raw_video_10bit_12bit branches so is the build on the Experiments download page from the allocate-raw-lv-buffer branch? Note that the 1100D nightly build from the unified branch doesn't have the raw_rec or mlv_rec modules though it does have mlv_play, which is rather odd.

In any case, where can the Digic IV camera owners who are trying to solve the 10bit/12bit riddle find more information on how the 1100D was solved?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: nikfreak on February 09, 2017, 05:37:22 PM
You probably expecting a final solution but my previous post maybe wasn't that clear. It was just about the edmac address (channel) and buffer size (width*height). What's known was committed to the allocate-raw-lv-buffer.

I already tried to explain it (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177826#msg177826) but won't be of any help in resolving it myself due to the complexity (http://www.magiclantern.fm/forum/index.php?topic=5601.msg176890#msg176890) and lack of knowledge. Still crossing my fingers ofc for 7D/5D2 and hoping for a solution due to the CF write speed and my opinion is that 1100D is not even worth the headaches.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SpcCb on February 10, 2017, 06:27:10 AM
(...)
So all you're looking for is that one hexadecimal number to unlock this feature. The question is how do you find it? Read the code, it tells you. Now the hard part is following those instructions and come up with the right address.
(...)
I'm a bad violin player too, but if the goal is to find an hexa number, is it possible to use a sort of brute force method by scanning all numbers with a match (size of the expected informations?) to find the right number? I say that just in case, I imagine all high level devs had already thought about this or maybe there's a issue in this method.

@reddeercity > The problem here, like in many other parts of ML as you know, is that kind of hack is so complex that only a couple of persons can figure how to do it. For other violin players it took years to understand. Although I had 20 years of coding in 5 languages, my eyes blinking face of some parts of ML ^.^
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on February 10, 2017, 06:31:34 PM
What's happening with raw_twk.mo?
It seems that its been left out in the cold.
Its able to display playback 10 bit raw MLV on the LCD.
The only issue I had with the 70D was it only works for the first clip, then if you try to playback
a second clip it hangs. 
Since it plays the first clip, it tells me that's its not initialized correctly for the second clip.
That should be a doable fix.  At least in terms of house of pain threshold.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: budafilms on February 12, 2017, 07:56:38 AM
Raw Twk.mo on 5d3 1.1.3 , experimental build, show black frames only. In the computer the shoots were ok.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on February 12, 2017, 04:10:53 PM
@ShootMeAlready Think I've had that problem on my 5D2 for a while in the 10-12bit builds and raw_twk, lets me play back a few clips then goes black until mlv_play is restarted(was already happening in December)
@reddeercity Thanks, I didn't realise mlv_rec had black level fixed in crop mode. Just an idea: Couldn't black levels just be hardcoded for each camera so there doesn't have to be problems all the time?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: balliamo1 on February 13, 2017, 07:27:58 PM
Hello
about the 10-bit version, I would like to know if the 550D will be supported and who you reckon a download will be available.
Thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ItsMeLenny on February 13, 2017, 11:50:43 PM
Hello
about the 10-bit version, I would like to know if the 550D will be supported and who you reckon a download will be available.
Thanks

It's already there somewhere, tested it a while ago, scroll back through the comments and you should find it.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: balliamo1 on February 14, 2017, 07:29:53 PM
I mean is a zip file available? I am not familiar with software code testing
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 16, 2017, 08:42:37 AM
Doing some research & testing on 5d2 full mlv+audio 10-12bit (I'm not doing any work with mlv lite because of no Audio for now so it's low on my priority list)
I have Qemu eos emluator setup with my rom file (5d2) , I'm almost ready to step fully in to this to work on 10/12bit issue on 5d2/digic4 although I had been discourage by a few developer
I have assistance from my best friend who happen to be a programmer/coder , with are closing in on the problem I believe. When I have something to show I'll post the results
either as Code or a test private build.

On a different note I was trying selected resolution sizes to see if this has any effect to frame corruption in 12bit and it seems that any vertical resolution be on 865 has
pink noise bar corruption on the top few lines , as you increase the vertical resolution to 1114 the pink noise bar just increase in height . It seem the top part of the sensor
always as this  corruption bar but as you crop the resolution vertically you don't record it , I tried 1856x844 2.20:1 A.R. 12bit 23.976p Canon liveview and all is well no frame shearing/tearing
all frames are good plus audio. So Any one that wants to try 12bit in 1:1 FHD I have been using the test build in my bitbucket downloads 10bit_12bit_2016Dec01.5D2212 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.5D2212.zip)

In case anyone interested in how many digic4 cam users that have download 10-12bit test builds look below

(https://c1.staticflickr.com/4/3794/32086259354_85a048686d.jpg) (https://flic.kr/p/QTmt3b)
bitbucket downloads digic4 cam (https://flic.kr/p/QTmt3b) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

5D2 with 76 downloads next 7d & 550d with 17 , that's what gives me incentive to continue on  :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: RenatoPhoto on February 16, 2017, 12:17:41 PM
@reddeercity
I am sure many users will be happy to know that you are working on porting this digic 4 cameras.  Make sure you post a link to the download page when you need testing.  I did not know that you had these builds!  :o
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 18, 2017, 03:43:58 AM
@reddeercity
Make sure you post a link to the download page when you need testing.
Yes I will  ;D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: arrinkiiii on February 20, 2017, 02:45:19 PM

Also for the 7D?? Many thanks :) 8)  Looking forward to the links =))
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: steeth on February 20, 2017, 02:54:45 PM
I´m having a weird issue with the 10-12 bit version. I´m using a 5D Mark III, with the latest feb 11 1.2.3 experimental 10-12 bit version. Sorry if this has been covered before, I´ve searched all the possible combinations of words I can think of, and read thru as many threads I can but haven´t found anything regarding this issue.

I convert my MLV files using the latest version of CR2HDR, and I use Resolve as my choice of development software. After I started using the experimental 10-12 bit build to try out 12 bit recording, I noticed that some clips in the media pool inside Resolve came out as folders, and not clips. Inside those folders are still frames, and smaller corrupt parts of that particular clip making the clip useless, which I thought was weird..

I checked the folder with the dng files for that particular clip and noticed something really strange. At first I thought that there were missing frames in the sequence, but I have not checked the "frame skipping" option in the ML menu. So I checked the dng folder of one of the corrupt files, and all the dng files are there, but for some reason some of the dng files are named with the file extension as capitol letters (.DNG) and not lower case letters as usual (.dng)!? And that particular dng sequence doesn´t load in either AE or Resolve as it says it´s corrupt. In Resolve, each .DNG file comes out as a single still frame.. If I manually change the dng files that have upper case file endings (.DNG to .dng), the clip works for some reason!?

I have tried everything I can think of to re-create the issue, but it´s completely random and doesn´t happen when I use 14 bit. It only happens when I change to 12 bit, and there is nothing indicating that it has happened to a clip, nor can I figure out anything that might cause it as it seems extremely random.

Has anyone else experienced this issue? I get this issue on about an average of 2-3 clips per 64 gb card I use. And some have a lot of .DNG file endings in them, making it a real hazzle to manually re-name all of them :)

(http://i67.tinypic.com/14ad2qx.png)

Kind regards
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on February 20, 2017, 03:07:23 PM
Not sure if 10/12 bit are optimized working with dualiso mlv files. Better stick with 14-bit if that works better. If you want resolve embedded folders you can run your mlv files through cr2hdr.app(mac).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on February 20, 2017, 03:59:52 PM
As long as the input to cr2hdr is up-converted to 14 bits, I don't see why it wouldn't work. But I've never tested this scenario.

With cr2hdr, uppercase DNG means converted files; mlv_dump outputs lowercase dng, so those files probably couldn't be converted. If their bit depth is 14, upload one of them; otherwise, try a newer mlv_dump.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: steeth on February 20, 2017, 05:02:43 PM
Thanks a lot for your replies Danne and a1ex.

Sorry if I didn´t make it clear in my post, but I´m not shooting nor trying to convert dualiso files, just standard regular 25fps mlv files. And I´m using the latest cr2hdr.app to convert my .mlv files.

The strange thing is that 99% of the time, both 10 bit and 12 bit .mlv files convert properly as lowercase .dng frames that works properly in both AE and Resolve. I don´t do anything different, nor do I change any settings in either my camera or in the cr2hdr.app when converting. Sometimes I can record and fill up 2-3 64 GB cards with no problem whatsoever converting 12 bit .mlv files with cr2hdr.app. All clips are recorded and converted to regular .dng frames as they should, and then out of nowwhere, 2-3 clips has these random .DNG uppercase file extension on just some random frames in a converted clip as shown in my previous image, making both AE and Resolve think that the sequence is corrupt. And if I manually change these random .DNG frames to .dng instead, they open just fine in both AE and Resolve?

So I don´t change any settings between clips or cards, I don´t do anything different when converting my .mlv files from the card in the cr2hdr.app, and I still get these random upper case .DNG frames randomly inside a .dng sequence which I think is so strange?

Has anyone else experienced this? And if so, is there a solution or am I doing something wrong? I´m sorry to say that I´m just a filmmaker and I have no knowledge of nor any experience in how to use code, read code or search for errors etc, so I can´t contribute with anything but my experience in using Magic Lantern unfortunately.

Thanks again for taking the time to read my post and replying.

Kind regards.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on February 20, 2017, 07:13:03 PM
Can you post a dng file that is turned into upper case dng in the cr2hdr.app thread?
http://www.magiclantern.fm/forum/index.php?topic=15108.msg174767#msg174767
I don,t really get why it would happen in the middle of a sequence. What the script does it tries the first file with cr2hdr to see if it will create a dual iso dng and if so it will continue to run cr2hdr on all the files. Otherwise not. This would however mean that the first dng file would be mistaken for a dualiso file and maybe that is the case here.
If the problem is what I described, cr2hdr is fooled somehow, you can easily disable the cr2hdr function for MLV files in cr2hdr.app and won,t even test for dualiso movie files. Especially if you never use dualiso. Of course this has nothing to do with this thread so just continue the c2hdr thread by posting a dng and maybe even a sample of a troubling MLV file in that thread instead.

*Update
Just added a switch in mlv_dump settings menu in cr2hdr.app which disables dualiso automation.
http://www.magiclantern.fm/forum/index.php?topic=15108.new#new
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: steeth on February 20, 2017, 09:56:10 PM
Thanks a lot Danne for taking the time to answer me regarding this. I´ll definitely post one of the mlv files that has this problem in the thread you mentioned. And I´ll download and test your updated cr2hdr.app build to see if the problem goes away. And now that you mention it, from what I remember, in every mlv file I´ve converted that has those upper case .DNG frames randomly in the sequence, I believe that the absolute first frame always have had a .DNG frame, and then several .dng frames, and then random .DNG frames throughout the sequence.. But I´m not sure as I haven´t looked for it on the first frame.. I´ll definitely upload a problematic .mlv file that has these issues so you can take a look. And sorry for posting in the wrong section, I didn´t know if it was caused by the 10-12 bit build or during the conversion process.

Thanks again.

And as a side note, you don´t happen to be Swedish do you? If you are, I´m from Sundsvall in Sweden :) Not that it has anything to do with anything, just curious :)

Kind regards.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on February 20, 2017, 10:10:52 PM
Yup, swedish and polish. Creme de la creme :).
Ok, if the first one popped then the switch will solve the issue.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on February 21, 2017, 06:45:09 AM
I'm looking in to raw record process , 
I seems to be going in circles can someone please point me to parts of the source code that capture the raw buffer lv (which I assume hopefully , correct me if I'm wrong)
is the buffer we use for raw video . I what to step thought the lines of code for 5d2 .
Thanks in advance .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: kizza1234 on February 23, 2017, 05:03:30 AM
I am so impressed with the development of magic lantern throughout my short span of using it its truly amazing, and the devs are doing such an amazing job!!! Magic lantern is so so so so good, and now with 10bit 50p 1080P continuous and stable? Its literally a dream come true!

I have installed the module and so far so good on 5D III, makes 50p 1080p (vertical upscaled 1.67x) continuous from what i can test.
Although as some other people have pointed out crop mode in 1080p 50p doesnt work as half the
frame is cut off top and bottom (image here: https://drive.google.com/open?id=0B1q4BC6n6pcNVTBwcUNXamRTRmc

Main Q:
But i was just wondering if anybody know the difference in dynamic range from 10bit to 12bit to 14bit?
From what i can see there is slightly less dynamic range every step down from 14bit, biggest difference i think is from 12bit to 10bit....
It is my understanding 14bit 1080p raw is around 11.8-11.5 stops?  :'(
Is 10bit something like 10 stops?

I might want to add that i use RawFlow to extract my files and 10bit extracts fine without any issues i can see.
Link: https://www.magiclantern.fm/forum/index.php?topic=13338.0
Its by far the easiest way i have found to extract files without any complications, and the images that get extracted are amazing!

Thanks Guys
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: alejandro rumolino on February 23, 2017, 12:10:24 PM
For 6D here the experimental loop https://builds.magiclantern.fm/experiments.html
I uploaded it and it works smoothly in 16/9 1600x900 10-bit, it does not have the possibility of the audio and I had a problem after filming, my canon does not start more for 1 hour ... so uninstall it ... use MLVtoMovie in .mov after
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ShootMeAlready on March 02, 2017, 04:36:19 PM
Kizza the DR stops or range is a matter of sensor size and performance.
You can have 20bit raw out of a small sensor, and it will not be better than a 10bit raw from a big sensor that has wide pixel separation, low noise. 

Humans are not very good at distinguishing colour depth, so 4 bit colour verses 2bit we are not as quick. Folks cant discern any difference in a 14bit raw vs a 12bit raw photo.  Where you get the big benefit is in pushing the raw data, like adjusting exposure in post, in that you can push 14bit raw much more than 10bit raw.

I think humans can distinguish 12bit raw vs 10bit raw photos, but that's only in certain shots and not always. 10 bit raw provides greater depth than compressed video, so one should be able to distinguish these, especially in low light shooting detail.  10bit will generally do a better job with highlight details (assuming not overexposed).  So rather than quote stops of light, think more about depth and detail in low/high light.

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on March 03, 2017, 11:08:37 PM
Humans are not very good at distinguishing colour depth, so 4 bit colour verses 2bit we are not as quick. Folks cant discern any difference in a 14bit raw vs a 12bit raw photo.  Where you get the big benefit is in pushing the raw data, like adjusting exposure in post, in that you can push 14bit raw much more than 10bit raw.

I think humans can distinguish 12bit raw vs 10bit raw photos, but that's only in certain shots and not always. 10 bit raw provides greater depth than compressed video, so one should be able to distinguish these, especially in low light shooting detail.  10bit will generally do a better job with highlight details (assuming not overexposed).  So rather than quote stops of light, think more about depth and detail in low/high light.
^ Sounds like poetry.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on March 04, 2017, 09:44:03 PM
Using 10bit_12bit.2017Feb11.5D3 build, and can't get ML to load with SND module activated (it says something like can't load modules, with a grey box). Any idea or do any of you have the same problem? Shooting a lot of sync footage tomorrow so would help a lot for to the post process. I wish not to use nightly because of the extra resolution on 10bit while doing zoom view..

Thank you!

Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Teamsleepkid on March 05, 2017, 02:53:47 AM
Try only loading the 2 modules you need. Nothing else.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on March 05, 2017, 04:18:42 AM
You mean "failed to link modules" ?

Are you trying to load mlv_snd with mlv_lite? That won't work. You need to use mlv_snd with mlv_rec.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: D_Odell on March 05, 2017, 07:05:55 AM
You mean "failed to link modules" ?

Are you trying to load mlv_snd with mlv_lite? That won't work. You need to use mlv_snd with mlv_rec.
Thats correct. Thank you for the heads up!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 06, 2017, 04:43:21 AM
Hey guys I've been getting some weird behaviour from Mark III 10bit MLV files converted to DNG with MLVFS. Probably 1 in 50 MLV files seems corrupted.

It looks like the Black Level issue that we're all familiar with – but I can't correct it using exiftool. I got the black level changed to 2048 but did not solve the issue.

What might be happening here and is there a way to fix it? Would someone care to try exiftool to confirm? Here is a DNG file (untouched) from one of the corrupted 10 bit MLVs.

https://www.dropbox.com/s/m77zmz6m6h9huik/M03-1839_000008.dng?dl=0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 06, 2017, 05:49:35 AM
No joy with exiftool , tried with mlv_dump , results below 
Will there is problems , Can you upload the corrupted MLV file if not too big,
you skinny can it down with mlvdiag.1.0.1.zip (http://dl.phreekz.de/raw2cdng/mlvdiag.1.0.1.zip) if on pc
Code: [Select]
C:\New\test>mlv_dump -o updated.dng --black-fix=valve M03-1839_000008.dng
 MLV Dumper v1.0
-----------------
Mode of operation:
   - Input MLV file: 'M03-1839_000008.dng'
   - Setting black level to 1
   - Rewrite MLV
   - Output into 'updated.dng'
File M03-1839_000008.dng opened
Processing...
[ERROR] Invalid block size at position 0x00000000
Processed 0 video frames
[ERROR] Failed to rewrite header in .MLV file
Done

You have some odd white & black levels
Code: [Select]
Unique Camera Model             : Canon EOS 5D Mark III
Black Level                     : 418
White Level                     : 1018
Light valve seems off too
Code: [Select]
Light Value                     : 3.1
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 06, 2017, 08:10:04 AM
It is 8.5GB file. I'm on Mac.

I found a similarly corrupted file that is much smaller.

https://www.dropbox.com/s/rmkyyg6q3nn47yj/M03-1356.MLV?dl=0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 06, 2017, 08:11:30 AM
Setting black level to 128 fixes it. There's a bug in MLVFS that doesn't let you change black level with exiftool, but didn't manage to look into it yet.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 06, 2017, 09:17:48 AM
I used mlv_dump instead to convert to DNG then used exiftool black level to 2040 – seems to have fixed it.

Although now there is an issue with the top section of the frame. It looks smeared.

https://www.dropbox.com/s/ucathd53p1megkt/M03-1839_000001.jpg?dl=0
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 06, 2017, 10:10:45 AM
Just a note. mlv_dump supports specifying both black and white level manually.
Also had issues with mlvfs dng files and exiftool. DNG files turning black or not letting exiftool changing tags.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 06, 2017, 01:08:18 PM
Although now there is an issue with the top section of the frame. It looks smeared.

With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 06, 2017, 11:14:08 PM
Just a little update on the digic4 cams downloads

(https://c1.staticflickr.com/4/3756/32450022274_6a54bbc22d.jpg) (https://flic.kr/p/RruR5E)
10-12bit downloads_DigicIV (https://flic.kr/p/RruR5E) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Seams 5D2 users are very hungry for Bit reduction  148 downloads and counting  8)
I don't think even 5d3 has that many bit reduction downloads
Long Live the mighty Digic IV Cams 7 years old+ and still in the game  :P
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 07, 2017, 07:24:09 AM
With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.

I'm on mac – is there an updated mlv_dump for mac?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 07, 2017, 07:32:28 AM
mlv_dump binary
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump

cr2hdr.app(uses mlv_dump)
http://www.magiclantern.fm/forum/index.php?topic=15108.msg146822#msg146822
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Surin Dmitriy on March 11, 2017, 01:52:00 PM
Whats going on with in the raw_twk, mlv_play and 10 12 bit??? A lot of people confirm the issue whith playback - you can only review 2-3 clips before you get black screen?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 11, 2017, 02:40:29 PM
http://wiki.magiclantern.fm/faq#any_progress_on_xyz
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Anstech on March 13, 2017, 02:07:54 AM
I'm a programmer in a very different field, and I'm just starting to try to get up to speed with understanding the ML / Digic context, so please excuse any naivety in this question:

One way that ArriRaw, amongst others, reduces its data requirements is to write the 16bit linear data from the sensor into an uncompressed raw 12bit log data stream, which - given the familiar inefficiency of linear encoding of light values - drastically reduces frame sizes.

The current 14-to- 12- and 10-bit strategy is an amazingly impressive piece of work, and I can see took years to achieve, and a lot of specialist knowledge. If I understand correctly, it's a bit-chop lossy reduction in representation precision. Obviously the raw-log-encoded method would be less of a quality drop from 14- to 10-bit, if it existed.

So before I 'go off on one' (as my old AI programming tutor used to put it when he saw me disappearing down a research rabbit hole / dead end again):

Is e.g. a 14-bit to 10-bit log-encoding of the raw uncompressed (linear) sensor data - rather than a lossy bit-chop - computationally plausible in this context? Or is it waaay too ambitious for the resources available on-camera during recording?

It's caught my imagination, but I realise it might be a silly idea if you know enough - so if that's the case, I'll be sensible and take on a far more manageable task from the 'to-do' list as a 'teach myself' project instead...

Thank you!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 13, 2017, 08:45:50 AM
On CPU, even a simple memcpy is too slow (http://www.magiclantern.fm/forum/index.php?topic=5601.msg166928#msg166928). So, the only remaining possibility is finding out how to configure Canon's image processor to do that.

Relevant links here (https://www.magiclantern.fm/forum/index.php?topic=16428.msg159735#msg159735), here (https://www.magiclantern.fm/forum/index.php?topic=16428.msg159735#msg159735), here (https://www.magiclantern.fm/forum/index.php?topic=18838.msg179076#msg179076) and here (http://www.magiclantern.fm/forum/index.php?topic=18222.msg174806#msg174806).
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: TequilaKez on March 22, 2017, 04:24:14 AM
Chiming in to add/confirm reports on 550D every 2nd frame is a copy of the last frame, except the top 3rd which looks like current frame, only shifted on X axis.

If I can contribute to testing in anyway, gimme a job! 550D is 'B' cam, but love to get more than a couple of seconds of Raw at full res.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: ricbi1 on March 22, 2017, 04:25:27 AM
Anstech... if you achieved that, it would be great! Just trying to encourage coz I can't do code for my life. Absolute ZERO. I tried over the years but no, can't do sir! Thanks everyone who is coding for sharing their efforts and developments!  :)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Eugenia on March 25, 2017, 11:44:15 PM
Hello, first post here. :)
I have two questions, that I couldn't find answers for:

1. What is the max res of the EOS M using 10 bit, no crop, at 24p continuous?

2. What is the max res of the 5D MkII using 10 bit, no crop, at 24p continuous? Why is there no 10-12bit ZIP file to download for the 5D-MkII? I couldn't find any on the exp. download page.

thx!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: far.in.out on March 26, 2017, 11:05:25 PM
Hey guys. Sorry for asking such a question but I really have no time to read all the thread, been away for too long...
What is the status on 12/10 bit raw? Is it working? Are there any downsides compared to 14bit?
I'm mainly interested in 50D as it's the camera that I have.
If there's a compiled module for my cam please give me the link. Thanks.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on March 27, 2017, 01:06:43 AM
Red deer city has some builds uploaded a few pages back, including 50D, I know for a fact that the 50d can't film 10/12 bit without corruption in full sensor, but maybe it works well in crop mode like the 5D2, give it a try.
edit: it's on page 46
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: beauchampy on March 27, 2017, 12:07:25 PM
Just came back from Malta filming a client piece entirely using 12bit and 3x3 50p. Flying by the seat of my pants, but the footage is lovely and not a single corrupt frame (5d3). I'll post some results next week. As far as I am concerned, this is more than ready for commercial use IMO.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 27, 2017, 12:27:24 PM
atleast 4 footage gets corrupted out of 10 with wrong black level.  im using 70D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 27, 2017, 12:50:48 PM
@andy kh

This is for you: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: beauchampy on March 27, 2017, 08:52:43 PM
atleast 4 footage gets corrupted out of 10 with wrong black level.  im using 70D

I'm using cr2hdr which enables you to specify a black level. Not sure what it is on the 70D but on my 5D3 I'm using 2048.

I do have the odd clip here and there with strange pink highlight clipping. I had also specified a white level of 1163.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 27, 2017, 10:15:51 PM
Could you post an example MLV which shows the corruption? You could shorten it before uploading. Use mlv_dump and open up terminal and do following:
Code: [Select]
mlv_dump -f 5 -o Output.MLV Input.MLV
mlv_dump
http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip

Input.MLV being the corrupted file and Output.MLV the name of the shortened file.
Beware to do things right since a wrong command could erase the file in question.

By the way. I think black and white levels should be around this for the 70D
White level 12277
Black level   2049
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 27, 2017, 11:13:25 PM
@Danne  please check this out. i use mlvp and set the black level to 125 for all 10bit mlv raw.
https://expirebox.com/download/fcfa483143425529e11d9cd2f4c43776.html
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 27, 2017, 11:27:59 PM
Yes, classic black level issue. From your file after extracting with mlv_dump
Black Level                     : 7312
White Level                     : 14992

When reprocessing and specifying numbers below picture is perfect.
Black Level                     : 2049
White Level                     : 12277

In mlvp you specify 10bit levels so they differ. Above numbers are for upressed to 14bit. Visually the picture should look the same for both 10bit and 14bit after fixing.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 27, 2017, 11:39:27 PM
@Danne thanks for checking it out. corrupted files are so many. as i mention earlier atleast3 to 4 out of 10 files get corrupted everytime i record in 10bit
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 27, 2017, 11:55:41 PM
I don,t know the status of 70D and 10/12bit recording but you could use mlv_dump for windows and specify both white and black level for all files that you export. Or stick to mlvp if that,s more convenient. Here is mlv_dump for windows
https://builds.magiclantern.fm/jenkins/job/mlv_dump/20/artifact/src/modules/mlv_rec/mlv_dump.exe

Specify like this
Code: [Select]
mlv_dump --dng --white-fix=12277 --black-fix=2049 Input.MLV
ANd why not do the same command with bouncyball´s version mlv_dump on steroids :)
http://www.magiclantern.fm/forum/index.php?topic=18975.msg180083#msg180083
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 28, 2017, 12:11:43 AM
@Danne
i wil try bounchyball's version as i prefer to export in cdng
thanks for your help
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 28, 2017, 04:22:19 AM
If on Windows use MLVProducer (http://www.magiclantern.fm/forum/index.php?topic=15271.msg148538#msg148538) , support 10 & 12 bit full mlv , mlv lite & old raw plus exports as Cdng if needed .
Also has slider for black level with real time preview like A.E. ACR plus many more grading options , I MUST have tool in windows along mlvfs .
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: andy kh on March 28, 2017, 10:05:06 AM
@reddeercity i have been using mlvp for sometime and wil continue to use it. it has some problems though. cdng exported from mlvp has a decrete amount of magenta cast while exporting to resolve or premiere pro. it can be fixed by adjusting the white balance though. awpstar is aware of the magenta cast as someone in the forum pointed out with image. and after converting to h264 422 or 444, im not able to import them in resolve. while clicking the folder containing converted h264 422 in resolve, all i can see is red icons written offline. i wil share with more details in mlvp forum
thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: beauchampy on March 28, 2017, 12:50:08 PM
@Danne I've been trying to find out more information for black level / white level on the 5d3. I recently forced white level to 1163 using cr2hdr (is that your app btw!?). Can it be pushed around even more for increased dynamic range?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 28, 2017, 01:00:23 PM
You can push whitelevel to 16383, it,s maximum white point.
Black level 2048. I do think default for 5D mark III is set to 16383 already.
Yes, my scripts bundled in automator. Utilizes a mlv_dump version with code from all over the place.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: a1ex on March 28, 2017, 01:10:46 PM
The black level is the signal level you get in the absence of light. It's a positive value because:
- the value is not the same for every sensel (hence the need for dark frame subtraction)
- values below black level do appear (because of the additive noise); a noise reduction or a frame stacking step can make use of those sub-black values.

"Pushing" the black level to a lower value would be the same as changing the input curve (rendering deep shadows in a lighter tone, rather than black). You probably liked the look, but there's no additional dynamic range to gain this way.

If the black level is incorrectly set above the real value, you will see additional detail from lowering it. Otherwise, there's little or no point in changing it.

Changing the black level also changes the colors in shadows, because the white balance multipliers are usually not equal. If you set all of them to 1, the color cast introduced by changing black level disappears.

On the other hand, there might be additional dynamic range to gain from pushing white level to a higher value. With recent versions, this additional gain will be less than 0.02 EV.
Code: [Select]
/**
 * White level
 *
 * With PREFERRED_RAW_TYPE set to CCD, most cameras appear to clip above 16300
 * (most of them actually use the full range, until 16382)
 *
 * one size fits all: 16200 may sacrifice up to 0.02 EV of highlights
 * that is, log2((16382-2048) / (16000-2048))
 */
#define WHITE_LEVEL 16200

However, if you find pink highlights with this value, I'd be interested to see a sample MLV.

@Danne: pushing white to 16383 may be a little optimistic; there are exposure settings that clip to this value and there are exposure settings that clip to a slightly lower level (including 5D3). Usually, if white level is above clipping point, you get pink highlights; but in some cases, the result is pretty good anyway, so it's a bit of an edge case.

You can check the white level in the samples in this thread: http://www.magiclantern.fm/forum/index.php?topic=11899.0

e.g. with octave (requires read_raw.m (https://github.com/apertus-open-source-cinema/misc-tools-utilities/blob/master/darkframes/read_raw.m)):
Code: [Select]
a = read_raw("frame.dng");
prctile(a(:), [90 95 99 99.9 99.99 99.999])

Note the max value from the image can exceed the clipping point from one sensel (happens with hot pixels), that's why I prefer checking it with percentiles.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: beauchampy on March 28, 2017, 04:52:16 PM
@a1ex @danne

Just to clarify. What should the black and white levels be for a) true colour representation and b) maximum dynamic range on the 5d3?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 28, 2017, 05:43:34 PM
I would go with this for white balance. Thanks a1ex.
Quote
one size fits all: 16200 may sacrifice up to 0.02 EV of highlights
Black level 2048

However Davinci resolve probably expects 5D mark III around 15000 if you start messing with their highlight recovery so you simply have to check what looks alright, how you choose to work with raw material and what NLE you are working in.

Do have a nice evening read in Baldavengers resolve frenzy over here
http://www.magiclantern.fm/forum/index.php?topic=15801.msg164822#msg164822
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on March 28, 2017, 06:35:00 PM
Been getting decent results w 5D3 footage by using @Danne's cr2hdr.app (black levels @ 2048 and white levels @ 16383) but now according to @a1ex it seems the new 'sweet spot' would be 16200 for the whites, correct?
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Danne on March 28, 2017, 06:48:07 PM
Safety margin, not sweet spot. Very well explained.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: DeafEyeJedi on March 28, 2017, 07:15:24 PM
Well perhaps I should have mentioned 'safety spot' rather than. This is all great to know and thanks guys!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Haeri on March 28, 2017, 10:49:15 PM
Any compiled 550D builds around? Would really like to try it out :D
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: MitchLally on March 30, 2017, 06:38:33 PM
Theres a few others here having the black level issues popping up with 10bit. I have a suspicion that it could be a result of having raw_twk loaded or "extra_hacks" selected in the mlv menu. Its one of the two so far I've had no false black levels with both switched off. I will test extensively over the weekend. Anyone else who has had black level issues tried this? And is it even possible that another module would interfere? I'm clueless on the coding so all I can do is test for you guys.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on March 31, 2017, 06:44:07 AM
a Little update for the 5D2 , Dual ISO Video Does work with 10-12bit Full MLV. the file corruption for 1:1 is still there thou 3x Crop work OK
did produce a few Hot pixels , by fine tuning the ISO range and a few more setting there can be eliminated . I was mostly looking to see
if it would work and it does . I only had 1 liveview freeze the needed a battery pull ,but I did a Super Fast pan before recording and it was in
3x crop mode with mlv_lite  before recording , after recording started it didn't seem to be a problem.
So at this point Dual ISO Video is functional in 3x Crop mode and all frame have proper black level with the 20bit option toggled in MLVFS PC Version Web GUI.
If you what to try it out here is the link 5D2212_Video_dual_iso.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5D2212_Video_dual_iso.zip) !!! Beware !!! This could brick your camera (will probably not :D)  I had to say that !
If anything happen like a liveview freeze just pull the battery out asap.
One more thing this is for just the 5D2 not sure if it will work with other Digic IV cams
Until next time  8)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: Ilia3101 on March 31, 2017, 08:47:42 AM
I tried the dual ISO recently too, and it was really difficult to get the freeze to happen(maybe it's improved?). I still think the freeze needs to be fixed for it to be included in main build though.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: domo94 on April 04, 2017, 03:29:48 PM
So I was using the Dec build for 10-12 bit raw video on the 7D; here's a little update to show some love for the camera.

While I was recording 10bit (various resolutions) my screen is killed instantly until it stops recording and writing, but it still records at normal FPS, with and without override.
When I was recording 12bit, the camera functioned normally and I had no issues manning my camera as I normally do; but the video output only gets a rough 15-20 frames a second, making the video look like a Chaplin film with color.

14 bit works just as fine. I did both 5x and normal modes.

Also, while recording 12, and I believe 10, The recording frame was not aligned properly- here's what I mean: http://imgur.com/a/sIb9W

To anybody working on the 7d, or to who can point me in the right direction, I'd like to contribute and accelerate the development of this firmware for the 7D as much as I can. So if you'd like some info or logs from me, please let me know what I can do.


Check my signature if you want to see an example of the footage and the quality (fixed from tech errors)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rduquematos on April 07, 2017, 08:31:26 AM
is 10 bit raw available on the 50d? thanks
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on April 11, 2017, 11:59:30 PM
is 10 bit raw available on the 50d? thanks

yes, work
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: dfort on April 12, 2017, 12:25:05 AM
Nested quote --

is 10 bit raw available on the 50d? thanks
yes, work

Really? Since when? It isn't on the Experiments download page (https://builds.magiclantern.fm/experiments.html) and I haven't seen any commits on bitbucket for it.

It would be nice to get it working properly on the 50D -- and the 5D2, 7D, 500D and 550D but those who have tried can tell you, it isn't that easy.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on April 12, 2017, 03:45:50 AM
yes, work
Really? Since when? ...... It would be nice to get it working properly on the 50D -- and the 5D2, 7D, 500D and 550D but those who have tried can tell you, it isn't that easy.
+1  what dfort said ,
@CITY-U1001 if you mean 3x crop will that's not working fully is it , when 1:1 works that's when you can say it works.
I've been working on the 5d2 for mouths now and I still can't get it to work right in 1:1 , 3x crop work @ 10bit but that's not working fully.
If you have it working  then Please share the code/info so I can advance the 5d2 .   
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: mirko91 on April 15, 2017, 10:59:37 PM
Hi guys, where can I download a compiled version with 10/12bit module for a 5d3 1.2.3? There were some links around but they are all missing. Thanks to anyone!
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: pc_bel on April 24, 2017, 11:46:59 AM
http://builds.magiclantern.fm/experiments.html (http://builds.magiclantern.fm/experiments.html)
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: SkalickyJ on April 24, 2017, 03:19:55 PM
Hi, I tried to shoot 1728x736 10 bit ml-raw on canon 700d. But i get files full of green,pink and blue dots, its almost impossible to see the original image i shot, because its covered with this dots.  Any idea, what could be wrong? Thanks! https://ibb.co/dF9GLk (https://ibb.co/dF9GLk)


btw. card speed should be ok, 14bit raw video at 1280x720 is working great...
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: CITY-U1001 on April 25, 2017, 03:33:07 PM
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/  :-\ but not fresh build
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: theBilalFakhouri on April 26, 2017, 12:19:06 PM
Hi, I tried to shoot 1728x736 10 bit ml-raw on canon 700d. But i get files full of green,pink and blue dots, its almost impossible to see the original image i shot, because its covered with this dots.  Any idea, what could be wrong? Thanks! https://ibb.co/dF9GLk (https://ibb.co/dF9GLk)


btw. card speed should be ok, 14bit raw video at 1280x720 is working great...

Your workflow process did not support 10-12bit
Use MLVProducer (the best,fastest software i used in MLV files Process) (also support 10-12-14bit and removing focus point automatically)
Download From Here:
https://www.magiclantern.fm/forum/index.php?topic=15271.0
after installing the program you must change some settings to encod MLV To DNG
(https://preview.ibb.co/hxPCAk/IMG.jpg)
Now Use in file menu Open MLV/RAW >> Chose your mlv file >> and again click File >> Render Task
You can open multiple MLVs And Click Render Queue
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: rduquematos on May 05, 2017, 02:53:46 AM
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/  :-\ but not fresh build
tried it on my 50d, 12bit works fine, 10bit I get some pink frames on the top
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 08, 2017, 01:44:36 AM
Working on 10-12bit 1:1 on 5D2 , as we know 3x crop works with 10 & 12 bit reduction but not 1:1 really well .
Thou in 12bit 1:1 you can record with corruption on every other frame on the top 100 or so lines .
I'm currently seeing the feasible of using bit reduction in a up coming documentary I been hired to shoot in 2 weeks for the sierra club (usa)
My needs are long shoot times , so low bite rate fits the bill .
Here a list of resolution's size I was possibly thinking of using with 12bit in 1:1 (as there corruption on the top of frame)
with the adjusted crop to remove corruption. I also listed the file size for 60 second or so to get max time for minimum card space
The base line is 14 bit 1856x928 23.976 fps (My minimum spec. for HD broadcast as it need to be up upscaled)
Code: [Select]
1856x928  23.976 14bit 59.9 seconds 1437 frames 4.05 GB 1:1 Liveview  ( 2:1 A.R.)
Code: [Select]
1856x1114 23.976 12bit 64.8 seconds 1554 frames 4.51 GB (crop adjust 1856x988) 126 line crop 1.878542510121457 A.R.
1856x1044 23.976 12bit 64.8 seconds 1554 frames 4.23 GB (Crop Adjust 1856x953) 91 line crop  1.947534102833158 A.R.
1856x1004 23.976 12bit 60.3 seconds 1447 frames 3.78 GB (crop Adjust 1856x932) 72 lines crop 1.991416309012876 A.R.
1856x928  23.976 12bit 60.1 seconds 1442 frames 3.49 GB (crop adjust 1856x895) 33 lines crop 2.073743016759777 A.R.
   

So There is really no saving after cropping , but 3x crop mode is a different matter all together .
Here 10bit really save me , as I can have to 20min per 64GB Card @ 1920x1076 @ 23.976fps
Thou the field of view is reduced , looks like the 24mm will gets a work out  :D
Code: [Select]
1920x1076 23.976 10bit 59.9 seconds 1437 frames 3.47 GB  (3x Crop Mode)  1.784386617100372 A.R.
So I save about 500 MB per minute which will add up fast. 4x 64gb card should be around 80 min's of 10bit 2K or 1920x1076.
Now I can use my slower 60MB/s CF cards to get
Code: [Select]
1856x928 23.976  10bit 60.1 seconds 1443 frames 2.91 GB  (3x crop)  (2:1  A.R.)
It looks like I'll being shooting in 10bit 3x crop for this doc.

Now for the real reason I started this post , as I said I'm work on 1:1 liveview 5d2 10-12bit.
I've being working on older code with .raw , just to keep it as simple as possible before I move on to .mlv full & mlv_lite.
At this Point I can record 10bit & 12bit , with out any lockups or screen freezes all overlays are working while recording bit reduction.
When not in bit reduction it return to 14bit and of course it work without problems.

Now the Problem  ::) there is always a problem , my issue is decoding the .raw to  dngs
My question is can I convert the .raw to .mlv ? as mlv_dump seem not to understand .raw
I'm currently uploading some of the .raw file to my Google drive to share , when done I'll post the links.
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 08, 2017, 04:04:09 AM
I'm currently uploading some of the .raw file to my Google drive to share , when done I'll post the links.
10 & 12 bit 1:1
https://drive.google.com/file/d/0BxNY4d4Aa7KuM1JlR3JnSTZpNlE/view?usp=sharing     1GB
https://drive.google.com/file/d/0BxNY4d4Aa7KualhHNnJ0Ujc3Ym8/view?usp=sharing    899MB
https://drive.google.com/file/d/0BxNY4d4Aa7KuSFpzcjJudHVUblk/view?usp=sharing       1GB
https://drive.google.com/file/d/0BxNY4d4Aa7KuSE1yZ2xqWERkR1k/view?usp=sharing    1GB
Title: Re: 12-bit (and 10-bit) RAW video development discussion
Post by: reddeercity on May 08, 2017, 05:09:58 AM
Should have taken my own advice "search box"  :P
Any ways yes you can convert .raw to .mlv with raw2dng .
now I have a more workable file .

Edit:
After All that , it's reading as a 14bit file , so the image is totally corrupted
should be either 10 or 12 bit .
Top file is my 10-12bit .raw file that I converted to .mlv
Second files is 14bit , was recorded in the session as the bit reduction files
Third is a older 2k 10bit .mlv  , just to be sure it's decoding correctly .
(https://c1.staticflickr.com/5/4156/33681092884_09fc15aa4c_c.jpg) (https://flic.kr/p/Tjhpu1)
raw2cdng_bit_reduction (https://flic.kr/p/Tjhpu1) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

With MLVFS quick mount on PC I get this from the cdng's from the .raw converted to .mlv , black level is way out to lunch , but it's a start .
Code: [Select]
ExifTool Version Number         : 10.33
File Name                       : m07-1552_000000.dng
Directory                       : C:/New
File Size                       : 3.3 MB
File Modification Date/Time     : 2017:01:07 14:51:59-07:00
File Access Date/Time           : 2017:05:07 21:01:52-06:00
File Creation Date/Time         : 2017:05:07 21:01:52-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution Image
Image Width                     : 1872
Image Height                    : 1054
Bits Per Sample                 : 16
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Fill Order                      : Normal
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Strip Offsets                   : 65536
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Rows Per Strip                  : 1054
Strip Byte Counts               : 3384466
Planar Configuration            : Chunky
Software                        : MLVFS
Modify Date                     : 2017:00:07 15:51:59
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Exposure Time                   : 1/50
F Number                        : 1.0
ISO                             : 81
Sensitivity Type