Converting legacy RAW to a valid MLV is now possible

Started by bouncyball, May 02, 2016, 04:00:51 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

bouncyball

So what is that all about:

I've modified raw2dng code and added the fully compatible (hopefully ;)) MLV output with all required info blocks and metadata.


usage:

raw2dng file.raw [prefix|--mlv [sidecar]]

  prefix    will create prefix000000.dng, prefix0000001.dng and so on.
   --mlv    will output MLV with unprocessed raw data and the same name as input.
sidecar    if needed specify (prerecorded or any) MLV file to override meaningless
            metadata values in IDNT, EXPO, LENS and WBAL blocks


It fully supports multiple file chunks in DNG and MLV mode. Produced MLV files tested and working with mlv_dump, mlvfs, MlRawViewer. Compiled for Linux and Win32. OSX binaries kindly provided by Danne.

I needed this tool and I've done it for myself, but if our _master_ devs or anyone feel like this is useful addition it's a honor to contribute at least something to the great ML project community.

Download latest source and binaries or compile it from ML source (it's merged to Unified branch).



Update:

Here is a small tool which automatically sets proper frameCount value to MLV file header (development raw_rec, raw v1.1 mlv lite case where it's zero not any more, patched).


usage:

mlv_setframes file.mlv [--set]

   --set    if specified actually writes frameCount to file
            otherwise just outputs the information

   Extra testing option:
   --set0x00000000    sets zero frameCount to any mlv file



Download: Source, Binaries

The binary looks for proper MLV/MXX file not by extension but a content of a file, makes sure the file has to be changed and only after that alters the value if additionally --set option specified.

You can safely run it on any folder with any mixed files in it like this: "for file in *.*; do ./mlv_setframes "$file"; done".
However *.M* wildcard is better ;) performance wise.

If --set is not specified it changes nothing - just outputs a few info about processed files. With --set0x00000000 you can go back to original state.
Note: It does not alter file modification time. Which I guess essential at least for me.



Update 2:

Here is a mlv_dump version from Dmilligan's magic-lantern repo. He did substitute original dng handling (chdk-dng.c) with his own library (dng.c) which is a base of mlvfs dng handling code. dng.c from latest mlvfs is quite different from the version of this branch, so I merged some sweet code from latest file and it seems worked out ok.

What we got after that:
1. Very similar to MLVFS dng output except it's 14bit and there are no forward matrixes in the header.
2. Fully works (hopefully): white balance, timecode, default scale.
3. Some fixes to mlv_dump to work properly including crash when using chroma smooth option.

I want to thank David for his great work on dng.c and mlvfs in general and Danne for his ideas, pointing out to this great reworked version of mlv_dump and helping out to test the program during development process. As well as all ML leading devs here and all the community.

Latest source is here.



Update 3:

There is more advanced and up to date version called "mlv_dump on steroids"



Update 4:

Now "mlv_dump on steroids" merged to official ML's crop_rec_4k branch Link


Regards
bb

Danne

Great work! Will test this tonight. Very useful if working as advertised.

bouncyball

Thanx Danne, just be aware that it does not support multi chunk RAWs as the original raw2dng does not support them either. And I was too tired ( lazy? :P ) to implement it myself.

Danne

Not really sure if I do this right.

...deleted...

*update
Actually got it to work with a sidecar MLV This is great stuff.

Daniels-MBP:~ Daniel$ /Users/Daniel/Desktop/raw2dng /Users/Daniel/Desktop/Myfiles/M12-1427.RAW --mlv /Users/Daniel/Desktop/Myfiles/A_mlv_raw_ORIGINALS/M12-1424.MLV

Resolution  : 1920 x 1080
Frames      : 43
Frame size  : 3629056 bytes
FPS         : 23.976
Black level : 2047
White level : 15000

Using sidecar file '/Users/Daniel/Desktop/Myfiles/A_mlv_raw_ORIGINALS/M12-1424.MLV'
Found EXPO LENS IDNT WBAL block(s)

Processing frame 43 of 43 writing MLV... Done.



*update 2
Never mind my question. It was too easy.
Without a sidecar file
Daniels-MBP:~ Daniel$ /Users/Daniel/Desktop/raw2dng /Users/Daniel/Desktop/Myfiles/M12-1427.RAW --mlv

Great stuff. I can cat any spanned file in MLP and the use this tool to create MLV files.
By the way. Did you notice the problem playing MLV files in MlRawViewer coming from mlv_lite? Maybe you know what,s it about?

*update
You created some new "unique camera model name" like Canon EOS 5DX Mark Free. Not sure how this will sing with converters and color matrices? Could it keep the regular unique camera model names you think? Ok, I see. This is some universal canon tag for all cams.

DeafEyeJedi

Damn boys... BB & Danne going at it like no other! Love this stuff so far and Thanks Danne for getting this ball rolling for OS X as well as to BB for setting the foundation on this legacy stuff!

Looking forward to playing with this ... This is so awesome actually and I can already feel it despite not touching it yet.

Thanks again for sharing a brilliant modified coding @bouncyball!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

bouncyball

Yeah... as RAW footer have no idea about information which is stored in EXPO, LENS, IDNT and WBAL blocks for MLV files I had to invent some default values to fill them up. Or... you can record a little mlv/mlvlite file before you record the RAW and then use it like sidecar with all flavor of real metadata to fill in.

Compatibility is a real issue for example mlrawviewer does not like mlvs without expo block and crashes. I tried to emulate minimal (like MLV_Lite is) and at the same time the most compatible case. If some software does not like the new EOS model ;) we can change that to anything it accepts.

Thank you guys:
@Danne: for compiling and testing it on MAC
@DeafEyeJedi: for the kind words and testing desire (as per usual) :)

edit:
QuoteBy the way. Did you notice the problem playing MLV files in MlRawViewer coming from mlv_lite? Maybe you know what,s it about?
Yes, I've been also thinking about it but have no idea why mlrawvewer does not like it. MLV lite has all requred blocks and more... :). so maybe it related to VIDF raw data size or something...

regards
bb

Danne

Here is the modified raw2dng for testing(use at own risk of course)
raw2dng
https://drive.google.com/file/d/0B4tCJMlOYfirTTk0dGNrbXAxcW8/view?usp=sharing

I wonder if it,s only the lack of reporting amount of frames in mlv_lite causing mlrawviewer showing black screen?

Anyway. it,s probably possible to insert the correct unique camera model name by looking at the dcraw color matrice 1. That way at least the name would be correct without the use of a side car file. Not that I don,t mind using the sidecar workflow, very clever. One could even rearrange the color matrices from this info. Here is the info regarding most cams. Now some cams share the same matrices. That will cause trouble of course. Original understanding about this I got from Chmee, author of raw2cdng.
RAW_matrices
https://drive.google.com/file/d/0B4tCJMlOYfirOUVvM3QxSnJnYjg/view?usp=sharing

DeafEyeJedi

Re: mlv_lite files showing black screen on MLRV ... I've learned that if you go into directory mode (backspace) within MLRV which then shows all files in place and each has its own previews shown from the first DNG's (assuming so) which leads me to Danne's theory w the lack of reforming amount of frames?
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

tecgen

Canon 5D Mark II, 50D, 550D/Rebel T2i, EF 40mm f/2.8 STM, Sigma 18-35mm f/1.8, EF 85 f1.8, EF 135 f2.8 SF, Zoom H2n

bouncyball

@Danne

QuoteI wonder if it,s only the lack of reporting amount of frames in mlv_lite causing mlrawviewer showing black screen?
Yep right! tested like this:

mlv_dump -r mlvlite.mlv -> mlvlite.raw -> raw2dng mlvlite.raw --mlv mlvlite_copy.mlv -> mlvlite.mlv.
This produces mlv with correct frame number filled in by mlv_dump -> raw process. mlrawviewer plays the file. no black screen.

Analyzed orig and converted mlv with mlv_dump -v.
Two differences so far:
1. Frame number 0 vs real number
2. VIDF size differs by 32 byte. It's because of the 'frameSpace' field in VIDF structure (necessary for EDMAC alignment). Vanilla MLV lite's value is 32 and in raw2dng I use 0 (zero byte padding before actual frame data). There is no need to respect EDMAC on PC :)

Then I HexEdited orig MLV lite and just put at offset 0x00000024 the correct frame value and... mlrawviewer plays file without a hickup :)

QuoteAnyway. it,s probably possible to insert the correct unique camera model name by looking at the dcraw color matrice 1
Thank you for pointing it out, I will look at it.

regards

Danne

Great findings.
I posted about this in the mlv_lite thread.
A question. How do one find and alter the offset number in hex? I,m trying to search for the number 0x00000024 in a mlv lite via hex fiend(for mac) file but nothing?
Could I ask for a little description around this?
Thanks
/D

bouncyball

@Danne

MLV file header 52 bytes long so:

$ xxd -l 52 test.mlv
0000000: 4d4c 5649 3400 0000 7632 2e30 0000 0000  MLVI4...v2.0....
0000010: 589d dfec 9618 0fcb 0000 0100 0100 0000  X...............
0000020: 0100 0000 0000 0000 0000 0000 a85d 0000  .............]..
0000030: e803 0000

4 Red bytes represent frame number value. In MLV Lite they're always 0x00

Let's assume we want frame number 282 dec = 0x0000011a hex. They should be interpreted backwards e.g 1a 01 00 00

use printf and dd in shell:
$ printf '\x1A\x01\x00\x00' | dd conv=notrunc of=test.mlv bs=4 seek=9
1+0 records in
1+0 records out
4 bytes (4 B) copied, 0.000141183 s, 28.3 kB/s

skip 9x4 byte (36byte=0x00000024 offset) and change 4 bytes from this offset to 0x0000011a

again
$ xxd -l 52 test.mlv
0000000: 4d4c 5649 3400 0000 7632 2e30 0000 0000  MLVI4...v2.0....
0000010: 589d dfec 9618 0fcb 0000 0100 0100 0000  X...............
0000020: 0100 0000 1a01 0000 0000 0000 a85d 0000  .............]..
0000030: e803 0000

regards
bb

Danne

Thank you very much. It works and I understand a lot better how to work with hex numbers now. Great.

bouncyball


Danne

Could you see what is wrong here. I try to insert 89 frames I do
printf '%x\n' 89
59

which will give me 0x0000059 I check this and it seems right.
echo $((0x0000059))
89

What is suppose to happen here?
printf '\x1A\x01\x00\x00' I tried following which gives me 2309 frames  :P
printf '\x05\x09\x00\x00'  | dd conv=notrunc of=my.mlv bs=4 seek=9

*update
I,m too sloppy. Found the solution  :P
printf '\x59\x00\x00\x00'  | dd conv=notrunc of=my.mlv bs=4 seek=9
89 frames

bouncyball

The correct command is:

printf '\x59\x00\x00\x00'   | dd conv=notrunc of=my.mlv bs=4 seek=9

You have to represent it as 4 byte (32bit) long number (unsigned long int) and  then reverse order of BYTEs.

In my previous example 282 = 0x11a = 0x 01 1a (bytes) = 0x 00 00 01 1a (adding 2 zero bytes for alignment to long int).
reverse them 1a 01 00 00 and push to file.

In your case 89 = 0x59 = 0x 00 00 00 59. Reversed 59 00 00 00

that's all :)
bb

Danne

hehe, thanks again. I updated my question just before you posted the solution. I was a bit slow. Hex is a bit hard to grasp at first sight  :P

bouncyball

Noo, it's just the byte order in a file a little bit misleading :)

bouncyball

Implemented default camera names according to matrix values. Thanx to Danne.

Block: IDNT
  Offset: 0x000000e8
    Size: 84
    Time: 1.300000 ms
     Camera Name:   'Canon EOS 5D Mark III'
     Camera Serial: 'E055DF4EE'
     Camera Model:  0x80000285


If we can collect "Camera Model" values for the following cameras:

Canon EOS 5D Mark III = 0x80000285
Canon EOS 5D Mark II = ?
Canon EOS 6D = ?
Canon EOS 7D = ?
Canon EOS 50D = ?
Canon EOS 60D = ?
Canon EOS 700D = ?
Canon EOS 600D = ?
Canon EOS 500D = ?
Canon EOS 1100D = ?

will fill it into the appropriate field.
Just redownload .C file from the first post.

bb

Danne

Exactly what we,re after. Tested working with 5D mark III. Is it possible to get these camera model tags from CR2 files or CR2 files converted to dng files with adobe dng converter? In that case I got all the raw files for this.

a1ex

Hey, didn't you say this a short while ago?

Quote from: bouncyball on April 18, 2016, 07:15:51 PM
I wish I had the C programming skills like you guys :)

Nice work - contributions like this are very encouraging. My C skills were a lot more rusty when I started tweaking ML, a few years ago, so it's nice to see you didn't give up.

To answer your last question: http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html

Danne

Great A1ex! Thanks
    switch(raw_info.color_matrix1[0])
    {
        case 6722:
            memcpy(idnt_hdr.cameraName, "Canon EOS 5D Mark III", 32);
            idnt_hdr.cameraModel = 0x80000285;
            break;
        case 4716:
            memcpy(idnt_hdr.cameraName, "Canon EOS 5D Mark II", 32);
            idnt_hdr.cameraModel = 0x80000218;
            break;
        case 7034:
            memcpy(idnt_hdr.cameraName, "Canon EOS 6D", 32);
            idnt_hdr.cameraModel = 0x80000302;
            break;
        case 6844:
            memcpy(idnt_hdr.cameraName, "Canon EOS 7D", 32);
            idnt_hdr.cameraModel = 0x80000250;
            break;
        case 4920:
            memcpy(idnt_hdr.cameraName, "Canon EOS 50D", 32);
            idnt_hdr.cameraModel = 0x80000261;
            break;
        case 6719:
            memcpy(idnt_hdr.cameraName, "Canon EOS 60D", 32);
            idnt_hdr.cameraModel = 0x80000287;
            break;
        case 6602:
            memcpy(idnt_hdr.cameraName, "Canon EOS 700D", 32);
            idnt_hdr.cameraModel = 0x80000326;
            break;
        case 6461:
            memcpy(idnt_hdr.cameraName, "Canon EOS 600D", 32);
            idnt_hdr.cameraModel = 0x80000286;
            break;
        case 4763:
            memcpy(idnt_hdr.cameraName, "Canon EOS 500D", 32);
            idnt_hdr.cameraModel = 0x80000252;
            break;
        case 6444:
            memcpy(idnt_hdr.cameraName, "Canon EOS 1100D", 32);
            idnt_hdr.cameraModel = 0x80000288 ;
            break;
        default:
            memcpy(idnt_hdr.cameraName, "Canon EOS 5DX Mark Free", 32);
            idnt_hdr.cameraModel = 0x8000F4EE;
    }


Maybe put in for these cameras as well or are they not recognized?
0x80000325   Canon EOS 70D
0x80000346   Canon EOS 100D
0x80000270   Canon EOS 550D
0x80000331   Canon EOS M
*update. Right, their color matrix 1 has doubles. Memory falls short.

Feel free to double check according to the Canon CanonModelID Values from A1ex exiftool link.

bouncyball

@a1ex

QuoteHey, didn't you say this a short while ago?
Yup :)

Thanks A1ex, it's the first time after almost 10 year period of inactivity in C programming. It's been a real adventure digging into the ML code (mlv_rec.c, mlv_lite.c and mlv_dump.c).

regards
bb

bouncyball

@Danne: Good job :)

Quote*update. Right, their color matrix 1 has doubles. Memory falls short.
Yes, I've just decided to use the highest model name among them. If some software will use the model name for color matrix assignment job will be done anyway.

.C file link updated.

bb

Danne

Thanks bouncyball for more greatness. Tested on a 700D and 5D mark III RAW file and it puts in the correct camera model for these cameras.
Here is the latest version for mac.
raw2dng
https://drive.google.com/file/d/0B4tCJMlOYfirTTk0dGNrbXAxcW8/view?usp=sharing