12-bit (and 10-bit) RAW video development discussion

Started by d, May 22, 2013, 10:58:34 PM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

1%

Fork whichever... I can give you write permisison too if you want.

For 6D I wouldn't fork the 'main repo as its not updated.

Gonna start merging tomorrow morning or later on tonight...

hirethestache

Im really rooting for the development of this on my 5D3, as I just purchased two more 5D3's and cant really afford a 1000x Lexar :-p
@HireTheStache
www.HireTheStache.com
C100, 5D3, 5D2, 6D

IliasG

Quote from: Mayo on May 23, 2013, 08:25:29 PM
Wow, simply lowered black and white (*0.25f) in raw2dng before it exports

raw_info.bits_per_pixel=12;
raw_info.white_level *= 0.25f;
    raw_info.black_level *= 0.25f;


And now the file opens ACR  ;D



I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Nice, thanks.

Is this posterisation down there ??.

Can you upload your dng file ??.

1%

For some reason 12 bit stops faster than 14 bit right now.. 1200x676 I have to stop recording in 14bit but in 12 bit only 144 frames. Write speed drops to 26MB/s if LV is on and pretty much the same (28-30) if you hack it off. Global draw off seemed to help a tiny bit too. Still must be something stopping it early.

Or maybe 10 bit would be better?

vicnaum

Quote from: 1% on May 24, 2013, 06:57:12 AM
For some reason 12 bit stops faster than 14 bit right now.. 1200x676 I have to stop recording in 14bit but in 12 bit only 144 frames. Write speed drops to 26MB/s if LV is on and pretty much the same (28-30) if you hack it off. Global draw off seemed to help a tiny bit too. Still must be something stopping it early.

Or maybe 10 bit would be better?

Maybe good thought to try extreme values, like 2-4 bits, to find out what stops it - bit processing, or card write. And if it's okay - then slowly raise it until the problem appears. Or don't write to card at all and see how much the processing only gives (something similar to a1ex benchmark, but with real data).

1%

It looks like a bug.. like its not clearing the buffers? 114 frames is about the size of memory or something?

a.d.

The same here on 5D2! This is the end at 114  ;D

Grunf

Quote from: Mayo on May 23, 2013, 07:54:36 PM
Modified reverse_bytes_order() for words instead of half-words in chdk-dng.c ,it's a bit better now.

static void FAST reverse_bytes_order(char* buf, int count)
{
long* buf32 = (long*) buf;
    int i;
    for (i = 0; i < count/4; i++)
    {
        long x = buf32[i];
buf[4*i+3] = x;
        buf[4*i+2] = x >> 8;
        buf[4*i+1] = x >> 16;
        buf[4*i] = x >>24;
    }
/*
    short* buf16 = (short*) buf;
    int i;
    for (i = 0; i < count/2; i++)
    {
        short x = buf16[i];
        buf[2*i+1] = x;
        buf[2*i] = x >> 8;
    }
*/
}


Will try this now.

Hi!


    for (i = 0; i < count/2; i++)

What about pre-calculating "count/2" into it's own variable and use that variable instead in your "for"-loop, otherwise it will waste precious CPU cycles evaluating "count/2" for each loop (giving same results over and over again)?

a1ex

You mean... raw2dng feels kinda slow?

;)

Grunf

Quote from: a1ex on May 24, 2013, 10:08:04 AM
You mean... raw2dng feels kinda slow?

;)

Ah, I didn't know which application this snippet of code belonged to. I just reacted when i saw the code :D

mucher

I don't know programming much. But it sounds to me that the program itself stalled itself and exit somewhere(or suspended by the OS), my wild guess.

a1ex

FYI, gcc already moves this kind of computations outside loops.

mucher

buffer_size_compressed = (buffer_size_used * 12) / 14;

I don't know what is this, but if it changes to:

buffer_size_compressed = buffer_size_used  / 14;
buffer_size_compressed = buffer_size_compressed * 12;

It might have higher accuracy, I heard that some CPU cannot multiply too many times, or very soon it will have accuracy problem.

HA, I have been messing around. I quit.

Grunf

Quote from: g3gg0 on May 23, 2013, 11:16:14 PM
this is for ARM946E-S vs. ARM946E which we have :)

but i didnt verify

Ok! That's a pity :(

My understanding is that Canon licensed  basic ARM946E core and then added some of their ow stuff to it. do you know if there is any way to check for procedure/op -calls done from, for example H264 encoder? I bet it has $hitload of DSP/SIMD op's hardcoded on die in order to do real-time H264 encoding, some of those could be "re-used" for 4 x small int bitshift? (Or is raw data offloaded to buffer, to be handled by some sort of co-processor involved)

Pity "d" didn't make rawc.o ASM available. It would be very interesting to take a peek...

nanomad

It's available, just de-compile it  ::)

H264 encoding is done by an external chip, so there's no need for fancy SIMD instructions sets on the ARM core :)
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

mucher

Quote from: mucher on May 24, 2013, 10:58:34 AM
buffer_size_compressed = (buffer_size_used * 12) / 14;

I don't know what is this, but if it changes to:

buffer_size_compressed = buffer_size_used  / 14;
buffer_size_compressed = buffer_size_compressed * 12;

It might have higher accuracy, I heard that some CPU cannot multiply too many times, or very soon it will have accuracy problem.

HA, I have been messing around. I quit.

Or simply changes to:
buffer_size_compressed = (buffer_size_used / 14) *12;

Because if buffer_size_used is something big, *12 might make it very big, making it difficult to be divided by 14, so I think that it might be better to divide first, and then multiply. But this trick might not work anyway.

nanomad

Don't do the compiler's job :P
He's usually good at it
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

1%


mucher

I have noticed another thing:

static int buffer_size_compressed = 0;

If  buffer_size_compressed is divided by 14, the result is in float point, you might need to change it back to int. That is a pain in the axx thing too I guess.

I might be wrong.

1%

sized_used isn't static so I can try that... the frames come out smaller when I open them so 12 bit should be helping.

buffer_size_compressed = (size_used * raw_bitdepth) / 14;

This doesn't make sense to me actually... why are we multiplying this out? The *12/14 calculation is to find width normally.

Mayo

1%, I just realised

{
                .name = "Bit Depth",
                .priv = &raw_bitdepth,
                .min = 12,
                .max = 13,
                .choices = CHOICES("12 Bit", "14 Bit", ),                               
                .help2 = "Bit Depth",
            },


this will probably not work (can't test atm)... I wrote somewhere that raw_bitdepth would work as usual if it was anything else than 12.
It's wrong... It has to be either 12 or 14. Probably needs a menu update function and a LUT.
As for why it stops, no idea. :(
Probably

if (r != buffer_size_compressed) goto abort;

gets called, but no idea why.

1%

It does work... when its 13 I get normal 14 bit I believe. You just have it set to if bit depth = 12 do this otherwise do that.

Also files opening in ACR just fine for me.

* Yea, both  files work fine... I just show one of each.. the 14 bit files play back in camera of course.

Mayo

Quote from: 1% on May 24, 2013, 05:21:12 PM
It does work... when its 13 I get normal 14 bit I believe. You just have it set to if bit depth = 12 do this otherwise do that.

It's used as actual bitdepth in write_speed_update and lv_rec_save_footer()

int speed = (res_x * res_y * raw_bitdepth/8 / 1024) * fps / 100 / 1024;

footer.frameSize = footer.xRes * footer.yRes * raw_bitdepth/8;

1%

But for 14 bit we don't do this?

I took the line out and it still behaves similarly... its like it just dies out after a while even at 960x540.

I see how it would be a problem then with it being 14 or not.

Mayo

Quote from: 1% on May 24, 2013, 05:15:44 PM
sized_used isn't static so I can try that... the frames come out smaller when I open them so 12 bit should be helping.

buffer_size_compressed = (size_used * raw_bitdepth) / 14;

This doesn't make sense to me actually... why are we multiplying this out? The *12/14 calculation is to find width normally.

It's just the compression ratio, it's the same for width and the whole image buffer. (to tell fwrite how much to write).

@mucher: not sure if you're serious, but all your assumptions are wrong...  :P