Messages

One thing is for sure: Canon's CR2 are compressed... so it's out there.

Hey, don't act as if we knew this all along! 

DIGIC compressed RAW... I'll try not to get overexcited this time.

I think what happens (regarding 13 instead of 14) is that raw2dng overwrites the wrong value anyway, so it does work in the end.

I really doubt 10-bit is less cpu intensive, but it would be nice to have as an option, for sure.

But for 14 bit we don't do this?

Yes, we do.

sized_used isn't static so I can try that... the frames come out smaller when I open them so 12 bit should be helping.

Code Select Expand

buffer_size_compressed = (size_used * raw_bitdepth) / 14;

This doesn't make sense to me actually... why are we multiplying this out? The *12/14 calculation is to find width normally.

It's just the compression ratio, it's the same for width and the whole image buffer. (to tell fwrite how much to write).

@mucher: not sure if you're serious, but all your assumptions are wrong... 

It does work... when its 13 I get normal 14 bit I believe. You just have it set to if bit depth = 12 do this otherwise do that.

It's used as actual bitdepth in write_speed_update and lv_rec_save_footer()

int speed = (res_x * res_y * raw_bitdepth/8 / 1024) * fps / 100 / 1024;

footer.frameSize = footer.xRes * footer.yRes * raw_bitdepth/8;

1%, I just realised

Code Select Expand

{
                .name = "Bit Depth",
                .priv = &raw_bitdepth,
                .min = 12,
                .max = 13,
                .choices = CHOICES("12 Bit", "14 Bit", ),                               
                .help2 = "Bit Depth",
            },

this will probably not work (can't test atm)... I wrote somewhere that raw_bitdepth would work as usual if it was anything else than 12.
It's wrong... It has to be either 12 or 14. Probably needs a menu update function and a LUT.
As for why it stops, no idea.
Probably

Code Select Expand

if (r != buffer_size_compressed) goto abort;

gets called, but no idea why.

I'm wondering: how is the RAW data (on the camera, before it is copied) aligned? is it 14bits with two zeros packed in 16 bits, or is each pixel packed tightly together?

It's tight:

Code Select Expand

/**
* 14-bit encoding:

hi          lo
aaaaaaaaaaaaaabb
bbbbbbbbbbbbcccc
ccccccccccdddddd
ddddddddeeeeeeee
eeeeeeffffffffff
ffffgggggggggggg
gghhhhhhhhhhhhhh
*/

I think that quick way to share the code without any risk is, once installed hg, register yourself on bitbucket, go to magic lantern source and fork the repo, you will have your fork on bitbucket pubblic so any change you made will not screw anything in the ML source.
Once done the fork you have to download it on your pc (I use a GUI for managing those forks, SourceTree), make your changes and with "hg pull" update it on bitbucket.
When you are done with all your work you can submit from the fork page on bitbucket a pull request to the ML source, that way it will be reviewed before merging it.

Thank you. I did all this, let's see how it goes.

Here's a really good Mercurial tutorial http://hginit.com/
I think most of the devs use it and I use it at work and for my own projects like my resume in LaTex and others.  Merging is a little strange at first, but it will make your life SO much easier in the long run.

Well, that looks nice and easy
Seriously though, the changes are really minor: just do a find 'raw_bitdepth' and all my changes are there (except for reverse_bytes_order() which is terribly hardcoded). If someone who's already setup can look through them it would be much safer than me fucking things up.

why don't you open a fork on bitbucket instead of google drive? It will be easier to follow (and learn from) your changes

I've been a lone coder for 4-5 years now, and became a bit oblivious to source control
I had a look at the tragic lantern bitbucket (https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/all) but i'm terrified to fuck up things for you guys... If you can quickly teach me how to commit my changes, i will gladly do it.

If you guys want to try 12-bit recording, here are my raw_rec.mo and 12bitraw2dng.exe (for win32)

https://docs.google.com/file/d/0Bw98R-sUdECPang2ZkVDUElfZU0/edit?usp=sharing

I only tried in 1840x1228, and it was worse than 14-bit for me (35 frames instead of 40 frames on a Sandisk Extreme Pro 95MBs).

Maybe it's better in lower res, and the raw14to12 routines should have their own task as 1% said.... But I need to take a break for now

Well, I didn't touch 10-bit, but it should be fairly easy to add.

Here are the modified src files: https://docs.google.com/file/d/0Bw98R-sUdECPZkpjQ2RPWi1yVjQ/edit?usp=sharing

12-bit is currently hardcoded with the variable:

Code Select Expand

static int raw_bitdepth = 12;

...which should be set via a RAW menu option and exported in the RAW footer to be retrieved by raw2dng and processed accordingly.

Wow, simply lowered black and white (*0.25f) in raw2dng before it exports

Code Select Expand

raw_info.bits_per_pixel=12;
raw_info.white_level *= 0.25f;
    raw_info.black_level *= 0.25f;

And now the file opens ACR 



I think I'm done here...
1%, I'll send you my modified src files, I trust you can tidy it up and merge with yours?

Almost there.



Now to find

1). where the black/white levels are wrongly set in chdk-dng.c
2). why the DNG file won't open in ACR

Modified reverse_bytes_order() for words instead of half-words in chdk-dng.c ,it's a bit better now.

Code Select Expand

static void FAST reverse_bytes_order(char* buf, int count)
{
long* buf32 = (long*) buf;
    int i;
    for (i = 0; i < count/4; i++)
    {
        long x = buf32[i];
buf[4*i+3] = x;
        buf[4*i+2] = x >> 8;
        buf[4*i+1] = x >> 16;
        buf[4*i] = x >>24;
    }
/*
    short* buf16 = (short*) buf;
    int i;
    for (i = 0; i < count/2; i++)
    {
        short x = buf16[i];
        buf[2*i+1] = x;
        buf[2*i] = x >> 8;
    }
*/
}

Open the DNG-File in a Hex-Editor (recommending HxD).
Search for the BitsPerSample-Tag 0102 - think it will be on Position ~1BC
(there will be two entries, because the thumbnail needs this tag as well)

Code Select Expand

0201 0300 0100 0000 0E00 0000
0102 (or in little-endian 0201) and later on there is 000E (14Bit) -> 0E00. change that.

You find the Whitelevel and blacklevel with the tag-id's C61A and C61D, some bytes after the BitPerSampleTag. Example:

Code Select Expand


1AC6 0400 0100 0000 F807 0000 <-->07F8 = 2040
1DC6 0400 0100 0000 2E2D 0000 <--> 2D2E = 11566

Lower both. For Example Blacklevel to 1000 and Whitelevel to 4000.


mfg chmee

Will try this now.

If only d posted his post-process workflow, this would be easier

I think DNG supports all bit depths but the DNG meta data will say it X bit.

Right, I found it, had to overwrite raw_info.bits_per_pixel=12; in raw2dng.

Now the image is good, only the colors are messed up.

Looks like this in irfan view (doesn't open in ACR):



(It's the bottom of a (very dirty) window)

I'm on 6D, I shoot in photo mode, global draw off.

But wait a minute... I noticed raw2dng just saves the raw frames, but nowhere saves a bitdepth or linewidth...
So how could DNG software (like ACR) read this properly if they don't know it's 12-bit?
Does the DNG format support anything other than 14-bit?
If it does not, we need to convert 12-bit back to 14-bit to export as dng. (I'm really not sure I have the skills to do this properly).
Reading chdk-dng.c now...

Finally got everything running, but the output seems shifted. Can't tell yet if it's my modified raw_rec or modified raw2dng.
The corrupted dngs are the correct size though.

Also, I didn't write more frames at high resolutions compared to 14-bit, so this probably wouldn't help getting higher resolutions.
The only interest would be in saving card space 

Yeah, but I want to modify it, to add support for 10bit and 12bit raw. Almost done

Thanks

I compile for 6D.

I figured out the CONFIG_XXXX stuff. raw_rec.mo compiles now.

I'm now trying to figure out how to compile raw2dng (the makefile apparently uses mingw, so i'm doing this now).

Are you compiling in windows cygwin?

Yes.

Edit: Ah, nevermind, it was right before my eyes 

Code Select Expand

* - enable modules in Makefile.user (CONFIG_MODULES = y, CONFIG_TCC = y, CONFIG_PICOC = n, CONFIG_CONSOLE = y)

Edit 2: Ah well, same problem. Any help is much appreciated.

Edit 3: replaced TOP_DIR=$(PWD)/../.. with TOP_DIR=../.. in the makefile : I got raw_rec.mo to compile \o/
Now, on to raw2dng...

Noob question: I can compile magic lantern just fine, but what's the correct procedure to make the modules?

When I try to make raw_rec.o from /modules/raw_rec/ I get:

Code Select Expand

[ CC       ]   raw_rec.o
raw_rec.c:20:20: fatal error: module.h: No such file or directory
compilation terminated.
make: *** [raw_rec.o] Error 1

Thanks.

Nope.. feel free to try, I'm not the best coder... guy should have really provided source... may 14th is hella old and slow.

I don't have all the required libs installed for this (And i really should be in bed right now), but in raw2dng.c

Code Select Expand



struct raw12_pixblock
{
        unsigned int c_hi: 8;
        unsigned int b: 12;
        unsigned int a: 12;
        unsigned int f_hi: 4;
        unsigned int e: 12;
        unsigned int d: 12;
        unsigned int c_lo: 4;
        unsigned int h: 12;
        unsigned int g: 12;
        unsigned int f_lo: 8;
} __attribute__((packed));

int raw_get_pixel(int x, int y) {
    struct raw12_pixblock * p = (void*)raw_info.buffer + y * raw_info.pitch + (x/8)*12;
    switch (x%8) {
        case 0: return p->a;
        case 1: return p->b;
        case 2: return p->c_lo | (p->c_hi << 8);
        case 3: return p->d;
        case 4: return p->e;
        case 5: return p->f_lo | (p->f_hi << 8);
        case 6: return p->g;
        case 7: return p->h;
    }
    return p->a;
}

If anyone wants to try 

Did you modify raw2dng ?