Reading 14 bit RAW

[ilia3101]

I am an absolute code noob AND right now I'm trying to decode 14 bit RAW video in C. I have looked at stack overflow, and seems like I'd need to read 7 bytes/56 bits to get 4 x 14 bit numbers, and thats what I'm doing, but its not working and I can't see why not, isn't a Magic Lantern RAW frame a tightly packed sequence of 14 bits? or is there some kind of gaps?
I'm taking 7 bytes, and shifting them like this:

Code Select Expand

for (uint32_t RAWbyte = 0; RAWbyte < RAWFrameSize; RAWbyte += 7)
{
    image16bit[(RAWbyte/7) * 4]     = ( (RAWFrame[RAWbyte] << 6) | (RAWFrame[RAWbyte + 1] >> 2) );
    image16bit[(RAWbyte/7) * 4 + 1] = ( (RAWFrame[RAWbyte + 1] << 12) | (RAWFrame[RAWbyte + 2] << 4) | (RAWFrame[RAWbyte + 3] >> 4) );
    image16bit[(RAWbyte/7) * 4 + 2] = ( (RAWFrame[RAWbyte + 3] << 10) | (RAWFrame[RAWbyte + 4] << 2) | (RAWFrame[RAWbyte + 5] >> 6) );
    image16bit[(RAWbyte/7) * 4 + 3] = ( (RAWFrame[RAWbyte + 5] << 8) | RAWFrame[RAWbyte + 6] );
}
easier to read:

Code Select Expand


pixel0 = ( (byte0 << 6) | (byte1 >> 2) );
pixel1 = ( (byte1 << 12) | (byte2 << 4) | (byte3 >> 4) );
pixel2 = ( (byte3 << 10) | (byte4 << 2) | (byte5 >> 6) );
pixel3 = ( (byte5 << 8) | byte6 );
Is there an error in the bit math or is the RAW format more complex?
Here is the output btw:

Close up:

I tried looking at mlv_dump source but its way too confusing for me.
Thanks to anyone willing to explain

[a1ex]

mlv_dump handles all bit depths with the same routine; you may find raw2dng or raw_get_pixel from raw.c (which are hardcoded for 14-bit) easier to understand. Make sure you look at MLVFS as well.

struct raw_pixblock from raw.h may be useful, too. However, bit packing order is not well-defined in the C standard, so... it just happens to work with gcc (and might break with other compilers) [1] [2].

So, to find what shifts to use, the easiest way is probably by looking at disassembly code created by the compiler.

Tip: in ML, there's a shortcut to look at disassembly, either from ML core or from a module: tag any non-static function with DUMP_ASM, then run "make dump_asm".

[ilia3101]

Thank you! Spent a while being confused looking at the things you suggested, raw_pixblock was most confusing at first, but told me everything I needed. Didn't realise the pixels are ordered in such a pattern, thanks a lot. Do 10 and 12 bit have their own pix block struct? And/or do they just follow the same mirrored kind of pattern?

[a1ex]

Most likely yes, but didn't try. Here's what I've used for Apertus:

Code Select Expand


/* two pixels packed as 12-bit (Apertus raw12) */
struct raw12_twopix
{
    unsigned a_hi: 8;
    unsigned b_hi: 4;
    unsigned a_lo: 4;
    unsigned b_lo: 8;
} __attribute__((packed));


Note the bytes in uncompressed Canon raw must be swapped to get valid DNG, at 10, 12 and 14-bit (see reverse_bytes_order in chdk-dng.c), but not at 16.

DNG spec:

If BitsPerSample is not equal to 8 or 16 or 32, then the bits must be packed into bytes using the TIFF default FillOrder of 1 (big-endian), even if the TIFF file itself uses little-endian byte order.

The 16-bit data provided by Canon (which is really 14-bit padded with zeros) is already little endian.

[ilia3101]

I see, finally got the swapped bytes all working:
(same shot)

Although my code has ended up pretty much the same as raw_get_pixel at least I understand how the order works now.
Sorry to be bothering you with tonnes of questions, but where could I find information about how to read the MLV headers? I've looked at mlv.h but have no idea how to identify which block its reading, I'm guessing

Code Select Expand

uint8_t     blockType[4]; would contain this info, but what values does blockType[4] have for each block?

[bouncyball]

Here is all information you need along with mlv.h

[ilia3101]

Thats exactly what I needed, thanks a lot! surprised I never found it

[ilia3101]

Hello, I have another noob question about RAW video/processing.

Still doing the same RAW processing code, now reading the headers properly and got the image looking pretty good already with amaze debayer:

(little pink highlight bug to fix)

Next step is colour, how do I use the matrices:

Code Select Expand

    #define CAM_COLORMATRIX1                       \
     4716, 10000,      603, 10000,    -830, 10000, \
    -7798, 10000,    15474, 10000,    2480, 10000, \
    -1496, 10000,     1937, 10000,    6651, 10000?

I have almost no idea about colour and colour spaces. Could someone give me a hint on what to do?
Matrixes are necessary for all cameras to match is that right?
Should white balance(channel multiplying) need to be done after matrix is applied?

p.s
I'll release the code soon(weeks maybe couple of months) with GPL, once it is actually useable for people other than me.
Also working on a GUI for it (mac, but maybe other OSes after because processing code is simple C)

[Danne]

Nice looking. What are you aiming for? You can see cam matrices (color matrix 1 an color matrix 2) in mlvfs sources. Check dng.c.
Multipliers turned into AsShotNeutral tag in your dng file.

[ilia3101]

Currently aiming to use matrices correctly. Had a look at dng.c in mlvfs source; What is the difference between ColorMatrix1 and ColorMatrix2? are they just different 'looks', or do both need to be applied to RAW data?

That image was without any matrices, just raw colour.

[a1ex]

This should help: https://rcsumner.net/raw_guide/RAWguide.pdf

The DNG spec also explains what's up with these matrices; Andy600 also has a couple of helpful posts on this.

[g3gg0]

i had some comments about this in
https://bitbucket.org/hudson/magic-lantern/src/d5915d61349a3bf61011bd0d21fbbaacc86691c4/contrib/g3gg0-tools/MLVViewSharp/DebayerBase.cs?at=unified&fileviewer=file-view-default#DebayerBase.cs-122

basically the camera's coefficients give you a matrix to compute (XYZ) -> (Bayer channels).
you have to inverse that matrix then you have  (Bayer channels) ->  (XYZ).
then you multiply it with a (XYZ) -> (RGB) matrix and you have RGB colors.

all the weird stuff you can see in my source is adding kelvin WB (which happens in cone domain)

maybe it helps a bit

[ilia3101]

@A1ex thanks, that is useful.
@g3gg0 couldn't really tell much from the code, but thank you, useful to know what the matrices actually do

According to that document, ColourMatrix2 contains XYZ to camera conversion, so(as g3gg0 says) that would need to be reversed and used to get RAW -> XYZ, then do XYZ -> RGB... etc
I'm still confused about one thing though, at what point does white balance multipliers need to be applied?/in what space?
The PDF says: raw data -> linearisation -> white balance -> demosaicing -> colour space correction -> brightness and contrast control -> final image
So the camera->XYZ->RGB matrix process happens after white balance?
But how could that be the case? if it is, correcting white balance after colour space correction would be wrong, which makes no sense to me.
Or would it be done before white balance, which "(Bayer channels) ->  (XYZ)" seems to imply - "Bayer" suggests white balance has not been done yet.

TL;DR In what order to do: (Bayer channels)->(XYZ)->(RGB) and White Balancing?

Apologies for not understanding anything.

Edit: @Danne I realised "What are you aiming for?" must have meant in general, well basically an all in one app for MLV thats free and open source and good(maybe)

[a1ex]

The PDF says: raw data -> linearisation -> white balance -> demosaicing -> colour space correction -> brightness and contrast control -> final image
So the camera->XYZ->RGB matrix process happens after white balance?

That's my understanding. At least, the AMaZE demosaicing algorithm depends on the raw data being white-balanced before running it. It uses this trick to recover high-frequency details and desaturate (hide) the color artifacts.

The simplest way to do white balance is just multiplying the linear raw data with some scalars. After that, demosaic and apply the matrix to convert from linear raw to linear RGB. Note that Adobe's matrices already contain a white balancing component and must be decomposed (at least that's how dcraw does it).

However, proper white balance is a bit more complex, but that's another can of worms:

http://web.stanford.edu/~sujason/ColorBalancing/adaptation.html
http://www.brucelindbloom.com/Eqn_ChromAdapt.html
http://www.brucelindbloom.com/Eqn_RGB_XYZ_Matrix.html
http://ninedegreesbelow.com/photography/xyz-rgb.html

[ilia3101]

Ok, thanks for clearing up my own understanding/confusion, its weird how it works but good to know.

Yeah, I noticed AMaZE worked better when white balance was done before debayering.

I think I'll stick to simple white balance with multiplication for the time being

Digital images need to be white balanced, yet in the real world eyes do it for us.

[g3gg0]

from what i know, white balance has to be done in XYZ domain, like the comments in source say:
/* RAW --> RAW-WB --> XYZ --> Kelvin-WB --> XYZ --> (s)RGB --> RGB-WB */
CamToRgbMatrix = WhiteBalanceMatrix * RGBToXYZMatrix.Inverse() * xyzKelvinWb * XYZToCamMatrix.Inverse() * WhiteBalanceMatrixRaw;

with:
Matrix xyzKelvinWb = coneDomain.Inverse() * xyzScale * coneDomain;

which means, convert the camera image into XYZ color representation domain, then convert it into a cone domain (e.b. using bradford)
within that cone domain kelvin conversion is quite simple.
then all the way back from cone -> XYZ -> RGB

[ilia3101]

My understanding of white balance: process image so that white/grey in real life, no matter what colour the lighting is, appears perfectly white or grey on the screen.
Of course subject to sensible/creative decisions, sodium vapour just doesn't look right in white.

So technically multiplying channels is one way to do it, although probably not the most correct.

I'll look in to the cone method with bradford. The links from a1ex look promising and I'll look at the c# code once again.

Thanks for enlightening me g3gg0. The can of worms is very opened

Also what do real RAW converters do? do they use cone?

[g3gg0]

once you get used to "this bayer RGB tristimulus" is a vector in some color domain,
there are color domains like the camera's bayer filters, XYZ, cone domain and RGB and
that there are matrices that, when you multiply a color triplet with, convert them from one domain into another,
then you quite understand the magic of all the stuff.

and the source code in C# represents these vector/matrix-multiplications as if you would do them in math.
the final matrix 'CamToRgbMatrix' is merely just a combination of all that weird space/kelvin conversion matrices.
so the result of all setup is one matrix that converts "bayer pixels" to "final RGB".

the only pain in the ass is then understanding "kelvin" white balance.
but in cone domain converting kelvin values is just simple "scaling" the single values.

back then it was fun to learn although i don't really like math.

[bouncyball]

Nice conversation guys!

@Ilia3101: really appreciate what you are up to, thought about doing this many times but I'm not a close friend of math either

br,
bb

[ilia3101]

I've been getting my head around this stuff today, finally starting to see clearer after your last post @g3gg0. Today just happens to be a very slow day for me(did not sleep). Also I couldn't find much about any colour domain called 'cone', but one comes up on Wikipedia that features both Bradford and eye cone cells - LMS colour space, is that the right one?
@bouncyball 

[g3gg0]

also interesting: https://books.google.de/books?id=suLqBgAAQBAJ&printsec=frontcover&hl=de&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false

i understood the "cone response domain" same as you, some color space which is probably related to cone sensitivity.
and it seems to make kelvin-wb easier to process.

although i didn't dig deeper as it was good enough for my goals.

[ilia3101]

An update: I am integrating the RAW processing code (that I've had help with on here) with a user interface finally! Only started the blending yesterday (actually 90% today). Right now, I've only connected the exposure and temperature slider, so no pretty saturated images yet 

Here's a teaser:

Yes, its hurried and would have been a whole lot better even if I'd posted later today, but I'm just overexcited that it's working.

The sliders are weird like that cos the contrast is done like an S-curve. I already regret doing the interface with Apple Cocoa, should have done it in GTK; but oh well. I'll release the code (and app) hopefully by the end of this month/start of July, whenever it works to a basic extent. Also don't know how to go about exporting compressed video, all I know how to do right now is a .bmp sequence which is ridiculous... 24bpp = 1.7x bigger

[Danne]

Awesome man. Let me know if you need some testing done.

[ilia3101]

Thanks. Soon

http://www.youtube.com/watch?v=2EFOhRNYMBE&feature=youtu.be

Won't do anymore pointless videos till its done
As you can see the current version has a bad case of sony highlights (which was fixed before I liked it to the ui, need to remember how)

[ilia3101]

Hi Gz, I have another question about reading MLV, this time about resolution metadata.

I know that when MLV is recorded in 50/60fps, the video resolution is 3/5 the real height becuase it uses 3x5 binning/scaling/skipping, or what is called mv720 around here... so it needs vertical streching by 1.66x

I asked for a sample in the MLV App thread, but haven't got any at this time yet, so I found an EOSM sample (with Danne cats ) in the ML Samplefiles thread, and tried it out (the EOSM is always mv720), and the vertical resolution in RAWI.yRes was 692, and it was also 692/693 when I tried to extract it from RAWI.raw_info.... so how can an MLV software possibly be sure if the image is supposed to be unsqueezed or not?

How can I tell if an image is compressed from the headers? or is the EOSM weird in some way? Do other cameras say unsqueezed resolution in RAWI.xRes and recorded resolution in RAW.raw_info.height, after all the comment on xRes is: /* Configured video resolution, may differ from payload resolution */