So what is that all about:
I've modified raw2dng code and added the fully compatible (hopefully ;)) MLV output with all required info blocks and metadata.
usage:
raw2dng file.raw [prefix|--mlv [sidecar]]
prefix will create prefix000000.dng, prefix0000001.dng and so on.
--mlv will output MLV with unprocessed raw data and the same name as input.
sidecar if needed specify (prerecorded or any) MLV file to override meaningless
metadata values in IDNT, EXPO, LENS and WBAL blocks
It fully supports multiple file chunks in DNG and MLV mode. Produced MLV files tested and working with mlv_dump, mlvfs, MlRawViewer. Compiled for Linux and Win32. OSX binaries kindly provided by Danne.
I needed this tool and I've done it for myself, but if our _master_ devs or anyone feel like this is useful addition it's a honor to contribute at least something to the great ML project community.
Download (https://bitbucket.org/bouncyball/mlv-tools/downloads) latest source and binaries or compile it from ML source (it's merged to Unified branch).
Update:
Here is a small tool which automatically sets proper frameCount value to MLV file header (development raw_rec, raw v1.1 mlv lite case where it's zero not any more, patched (http://www.magiclantern.fm/forum/index.php?topic=17185.msg166987#msg166987)).
usage:
mlv_setframes file.mlv [--set]
--set if specified actually writes frameCount to file
otherwise just outputs the information
Extra testing option:
--set0x00000000 sets zero frameCount to any mlv file
Download: Source (https://bitbucket.org/bouncyball/mlv-tools/src), Binaries (https://bitbucket.org/bouncyball/mlv-tools/downloads)
The binary looks for proper MLV/MXX file not by extension but a content of a file, makes sure the file has to be changed and only after that alters the value if additionally --set option specified.
You can safely run it on any folder with any mixed files in it like this: "for file in *.*; do ./mlv_setframes "$file"; done".
However *.M* wildcard is better ;) performance wise.
If --set is not specified it changes nothing - just outputs a few info about processed files. With --set0x00000000 you can go back to original state.
Note: It does not alter file modification time. Which I guess essential at least for me.
Update 2:
Here is a mlv_dump version from Dmilligan's magic-lantern repo. He did substitute original dng handling (chdk-dng.c) with his own library (dng.c) which is a base of mlvfs dng handling code. dng.c from latest mlvfs is quite different from the version of this branch, so I merged some sweet code from latest file and it seems worked out ok.
What we got after that:
1. Very similar to MLVFS dng output except it's 14bit and there are no forward matrixes in the header.
2. Fully works (hopefully): white balance, timecode, default scale.
3. Some fixes to mlv_dump to work properly including crash when using chroma smooth option.
I want to thank David for his great work on dng.c and mlvfs in general and Danne for his ideas, pointing out to this great reworked version of mlv_dump and helping out to test the program during development process. As well as all ML leading devs here and all the community.
Latest source is here (https://bitbucket.org/Dannephoto/ml-dng-dannephoto/src).
Update 3:
There is more advanced and up to date version called "mlv_dump on steroids" (http://www.magiclantern.fm/forum/index.php?topic=18975.0)
Update 4:
Now "mlv_dump on steroids" merged to official ML's crop_rec_4k branch Link (https://bitbucket.org/hudson/magic-lantern/src/0075d0fa70f229ec972d7831aa756c4a9280e625/modules/mlv_rec/?at=crop_rec_4k)
Regards
bb
Great work! Will test this tonight. Very useful if working as advertised.
Thanx Danne, just be aware that it does not support multi chunk RAWs as the original raw2dng does not support them either. And I was too tired ( lazy? :P ) to implement it myself.
Not really sure if I do this right.
...deleted...
*update
Actually got it to work with a sidecar MLV This is great stuff.
Daniels-MBP:~ Daniel$ /Users/Daniel/Desktop/raw2dng /Users/Daniel/Desktop/Myfiles/M12-1427.RAW --mlv /Users/Daniel/Desktop/Myfiles/A_mlv_raw_ORIGINALS/M12-1424.MLV
Resolution : 1920 x 1080
Frames : 43
Frame size : 3629056 bytes
FPS : 23.976
Black level : 2047
White level : 15000
Using sidecar file '/Users/Daniel/Desktop/Myfiles/A_mlv_raw_ORIGINALS/M12-1424.MLV'
Found EXPO LENS IDNT WBAL block(s)
Processing frame 43 of 43 writing MLV... Done.
*update 2
Never mind my question. It was too easy.
Without a sidecar file
Daniels-MBP:~ Daniel$ /Users/Daniel/Desktop/raw2dng /Users/Daniel/Desktop/Myfiles/M12-1427.RAW --mlv
Great stuff. I can cat any spanned file in MLP and the use this tool to create MLV files.
By the way. Did you notice the problem playing MLV files in MlRawViewer coming from mlv_lite? Maybe you know what,s it about?
*update
You created some new "unique camera model name" like Canon EOS 5DX Mark Free. Not sure how this will sing with converters and color matrices? Could it keep the regular unique camera model names you think? Ok, I see. This is some universal canon tag for all cams.
Damn boys... BB & Danne going at it like no other! Love this stuff so far and Thanks Danne for getting this ball rolling for OS X as well as to BB for setting the foundation on this legacy stuff!
Looking forward to playing with this ... This is so awesome actually and I can already feel it despite not touching it yet.
Thanks again for sharing a brilliant modified coding @bouncyball!
Yeah... as RAW footer have no idea about information which is stored in EXPO, LENS, IDNT and WBAL blocks for MLV files I had to invent some default values to fill them up. Or... you can record a little mlv/mlvlite file before you record the RAW and then use it like sidecar with all flavor of real metadata to fill in.
Compatibility is a real issue for example mlrawviewer does not like mlvs without expo block and crashes. I tried to emulate minimal (like MLV_Lite is) and at the same time the most compatible case. If some software does not like the new EOS model ;) we can change that to anything it accepts.
Thank you guys:
@Danne: for compiling and testing it on MAC
@DeafEyeJedi: for the kind words and testing desire (as per usual) :)
edit:
QuoteBy the way. Did you notice the problem playing MLV files in MlRawViewer coming from mlv_lite? Maybe you know what,s it about?
Yes, I've been also thinking about it but have no idea why mlrawvewer does not like it. MLV lite has all requred blocks and more... :). so maybe it related to VIDF raw data size or something...
regards
bb
Here is the modified raw2dng for testing(use at own risk of course)
raw2dng
https://drive.google.com/file/d/0B4tCJMlOYfirTTk0dGNrbXAxcW8/view?usp=sharing
I wonder if it,s only the lack of reporting amount of frames in mlv_lite causing mlrawviewer showing black screen?
Anyway. it,s probably possible to insert the correct unique camera model name by looking at the dcraw color matrice 1. That way at least the name would be correct without the use of a side car file. Not that I don,t mind using the sidecar workflow, very clever. One could even rearrange the color matrices from this info. Here is the info regarding most cams. Now some cams share the same matrices. That will cause trouble of course. Original understanding about this I got from Chmee, author of raw2cdng.
RAW_matrices
https://drive.google.com/file/d/0B4tCJMlOYfirOUVvM3QxSnJnYjg/view?usp=sharing
Re: mlv_lite files showing black screen on MLRV ... I've learned that if you go into directory mode (backspace) within MLRV which then shows all files in place and each has its own previews shown from the first DNG's (assuming so) which leads me to Danne's theory w the lack of reforming amount of frames?
I compiled a version of the new raw2dng from bouncyball for OSX.
https://bitbucket.org/tecgen/magic-lantern/downloads/raw2dng
@Danne
QuoteI wonder if it,s only the lack of reporting amount of frames in mlv_lite causing mlrawviewer showing black screen?
Yep right! tested like this:
mlv_dump -r mlvlite.mlv -> mlvlite.raw -> raw2dng mlvlite.raw --mlv mlvlite_copy.mlv -> mlvlite.mlv.
This produces mlv with correct frame number filled in by mlv_dump -> raw process. mlrawviewer plays the file. no black screen.
Analyzed orig and converted mlv with mlv_dump -v.
Two differences so far:
1. Frame number 0 vs real number
2. VIDF size differs by 32 byte. It's because of the 'frameSpace' field in VIDF structure (necessary for EDMAC alignment). Vanilla MLV lite's value is 32 and in raw2dng I use 0 (zero byte padding before actual frame data). There is no need to respect EDMAC on PC :)
Then I HexEdited orig MLV lite and just put at offset 0x00000024 the correct frame value and... mlrawviewer plays file without a hickup :)
QuoteAnyway. it,s probably possible to insert the correct unique camera model name by looking at the dcraw color matrice 1
Thank you for pointing it out, I will look at it.
regards
Great findings.
I posted about this in the mlv_lite thread.
A question. How do one find and alter the offset number in hex? I,m trying to search for the number 0x00000024 in a mlv lite via hex fiend(for mac) file but nothing?
Could I ask for a little description around this?
Thanks
/D
@Danne
MLV file header 52 bytes long so:
$ xxd -l 52 test.mlv
0000000: 4d4c 5649 3400 0000 7632 2e30 0000 0000 MLVI4...v2.0....
0000010: 589d dfec 9618 0fcb 0000 0100 0100 0000 X...............
0000020: 0100 0000 0000 0000 0000 0000 a85d 0000 .............]..
0000030: e803 0000
4 Red bytes represent frame number value. In MLV Lite they're always 0x00
Let's assume we want frame number 282 dec = 0x0000011a hex. They should be interpreted backwards e.g 1a 01 00 00
use printf and dd in shell:
$ printf '\x1A\x01\x00\x00' | dd conv=notrunc of=test.mlv bs=4 seek=9
1+0 records in
1+0 records out
4 bytes (4 B) copied, 0.000141183 s, 28.3 kB/s
skip 9x4 byte (36byte=0x00000024 offset) and change 4 bytes from this offset to 0x0000011a
again
$ xxd -l 52 test.mlv
0000000: 4d4c 5649 3400 0000 7632 2e30 0000 0000 MLVI4...v2.0....
0000010: 589d dfec 9618 0fcb 0000 0100 0100 0000 X...............
0000020: 0100 0000 1a01 0000 0000 0000 a85d 0000 .............]..
0000030: e803 0000
regards
bb
Thank you very much. It works and I understand a lot better how to work with hex numbers now. Great.
U R Welcome :)
Could you see what is wrong here. I try to insert 89 frames I do
printf '%x\n' 89
59
which will give me 0x0000059 I check this and it seems right.
echo $((0x0000059))
89
What is suppose to happen here?
printf '\x1A\x01\x00\x00' I tried following which gives me 2309 frames :P
printf '\x05\x09\x00\x00' | dd conv=notrunc of=my.mlv bs=4 seek=9
*update
I,m too sloppy. Found the solution :P
printf '\x59\x00\x00\x00' | dd conv=notrunc of=my.mlv bs=4 seek=9
89 frames
The correct command is:
printf '\x59\x00\x00\x00' | dd conv=notrunc of=my.mlv bs=4 seek=9
You have to represent it as 4 byte (32bit) long number (unsigned long int) and then reverse order of BYTEs.
In my previous example 282 = 0x11a = 0x 01 1a (bytes) = 0x 00 00 01 1a (adding 2 zero bytes for alignment to long int).
reverse them 1a 01 00 00 and push to file.
In your case 89 = 0x59 = 0x 00 00 00 59. Reversed 59 00 00 00
that's all :)
bb
hehe, thanks again. I updated my question just before you posted the solution. I was a bit slow. Hex is a bit hard to grasp at first sight :P
Noo, it's just the byte order in a file a little bit misleading :)
Implemented default camera names according to matrix values. Thanx to Danne.
Block: IDNT
Offset: 0x000000e8
Size: 84
Time: 1.300000 ms
Camera Name: 'Canon EOS 5D Mark III'
Camera Serial: 'E055DF4EE'
Camera Model: 0x80000285
If we can collect "Camera Model" values for the following cameras:
Canon EOS 5D Mark III = 0x80000285
Canon EOS 5D Mark II = ?
Canon EOS 6D = ?
Canon EOS 7D = ?
Canon EOS 50D = ?
Canon EOS 60D = ?
Canon EOS 700D = ?
Canon EOS 600D = ?
Canon EOS 500D = ?
Canon EOS 1100D = ?
will fill it into the appropriate field.
Just redownload .C file from the first post.
bb
Exactly what we,re after. Tested working with 5D mark III. Is it possible to get these camera model tags from CR2 files or CR2 files converted to dng files with adobe dng converter? In that case I got all the raw files for this.
Hey, didn't you say this a short while ago?
Quote from: bouncyball on April 18, 2016, 07:15:51 PM
I wish I had the C programming skills like you guys :)
Nice work - contributions like this are very encouraging. My C skills were a lot more rusty when I started tweaking ML, a few years ago, so it's nice to see you didn't give up.
To answer your last question: http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html
Great A1ex! Thanks
switch(raw_info.color_matrix1[0])
{
case 6722:
memcpy(idnt_hdr.cameraName, "Canon EOS 5D Mark III", 32);
idnt_hdr.cameraModel = 0x80000285;
break;
case 4716:
memcpy(idnt_hdr.cameraName, "Canon EOS 5D Mark II", 32);
idnt_hdr.cameraModel = 0x80000218;
break;
case 7034:
memcpy(idnt_hdr.cameraName, "Canon EOS 6D", 32);
idnt_hdr.cameraModel = 0x80000302;
break;
case 6844:
memcpy(idnt_hdr.cameraName, "Canon EOS 7D", 32);
idnt_hdr.cameraModel = 0x80000250;
break;
case 4920:
memcpy(idnt_hdr.cameraName, "Canon EOS 50D", 32);
idnt_hdr.cameraModel = 0x80000261;
break;
case 6719:
memcpy(idnt_hdr.cameraName, "Canon EOS 60D", 32);
idnt_hdr.cameraModel = 0x80000287;
break;
case 6602:
memcpy(idnt_hdr.cameraName, "Canon EOS 700D", 32);
idnt_hdr.cameraModel = 0x80000326;
break;
case 6461:
memcpy(idnt_hdr.cameraName, "Canon EOS 600D", 32);
idnt_hdr.cameraModel = 0x80000286;
break;
case 4763:
memcpy(idnt_hdr.cameraName, "Canon EOS 500D", 32);
idnt_hdr.cameraModel = 0x80000252;
break;
case 6444:
memcpy(idnt_hdr.cameraName, "Canon EOS 1100D", 32);
idnt_hdr.cameraModel = 0x80000288 ;
break;
default:
memcpy(idnt_hdr.cameraName, "Canon EOS 5DX Mark Free", 32);
idnt_hdr.cameraModel = 0x8000F4EE;
}
Maybe put in for these cameras as well or are they not recognized?
0x80000325 Canon EOS 70D
0x80000346 Canon EOS 100D
0x80000270 Canon EOS 550D
0x80000331 Canon EOS M
*update. Right, their color matrix 1 has doubles. Memory falls short.
Feel free to double check according to the Canon CanonModelID Values from A1ex exiftool link.
@a1ex
QuoteHey, didn't you say this a short while ago?
Yup :)
Thanks A1ex, it's the first time after almost 10 year period of inactivity in C programming. It's been a real adventure digging into the ML code (mlv_rec.c, mlv_lite.c and mlv_dump.c).
regards
bb
@Danne: Good job :)
Quote*update. Right, their color matrix 1 has doubles. Memory falls short.
Yes, I've just decided to use the highest model name among them. If some software will use the model name for color matrix assignment job will be done anyway.
.C file link updated.
bb
Thanks bouncyball for more greatness. Tested on a 700D and 5D mark III RAW file and it puts in the correct camera model for these cameras.
Here is the latest version for mac.
raw2dng
https://drive.google.com/file/d/0B4tCJMlOYfirTTk0dGNrbXAxcW8/view?usp=sharing
Update
Now it supports multi chunk RAW files In both DNG and MLV mode. Link on the first post updated.
@Danne, DeafEyeJedi:
Can you try it with more than 2 chunks. I have no camera at the moment. Splitted manually the file I have to several pieces and it's ok but would be nice to test it on raws right out of the camera.
regards
bb
Fully working. Tested with 23gb 5 chunk RAW file. You,ve created a RAW demolisher ;).
Latest compilation
raw2dng
https://drive.google.com/file/d/0B4tCJMlOYfirTUZHeUR2TEdlSWM/view?usp=sharing
On a sidenote you actually finally solved the spanned file issue with raw2dng so now instead of using cat the dng files will export just like mlv_dump. How good is that. Great!
Thank you for helping Danne :)
As Audionut said a short while ago:
QuoteTo create a RAW to MLV convertor would require a bulk of time from a developer to develop the software, which would be used by few, once, to create the files, then be useless.
Well... that's right.
So let's demolish legacy stuff... Long live to MLV Lite :)
regards
bb
Agreed. To speed up things I could integrate your raw to mlv workflow in MLP. Could put in a little pop menu asking for creation of mlv files. Could even be an added option to utilize mlv sidecar files?
It's a honor to be a small part of your project :)
However as linux/win user and owner of very old mac mini and also old hackintosh machine never used macosx for serious video stuff just for some testing.
bb
Cool. Thanks :).
Just created a simplistic menu around your fine changes in raw2dng and put in MLP
http://www.magiclantern.fm/forum/index.php?topic=13512.msg166801#msg166801
I really have to try MLP :)
My quadruple boot :P (Linux/Win10/MacOSX ML/Androidx86 RemixOS) Asus Zenbook's performing not that bad with its old low power i7. Will it work on mountain lion?
bb
hm, I think so :). I had problems with a user trying to get it to work on a really old mac with 32-bit architecture but it should work otherwise.
@bouncyball: what about creating a pull request, to merge your work in the main repository and get proper credits?
@a1ex
Will put it into repository with pleasure. Had no time and just want to polish some fseek->fseeko64 stuff for more win32 compatibility.
regards
bb
QuoteWill it work on mountain lion?
I installed Yosemite some months ago on a Thinkpad x201 with an old i5 processor. It works quite well. I couldn't get Wifi working, even after I installed a patched BIOS and changed the internal card with a compatible (but maybe broken) one. Currently I use a cheap tiny USB Wifi adapter as workaround. Good luck!
@tecgen
I installed MLP on osx mountain lion and it works smoothly. Just cpu (2xCore i7 ultralow voltage series) is too slow for video encoding.
OT: Yes I'm also using usb realtec chip based wifi. Could not find any Intel Centrino wifi driver for osx. I even managed the touch screen ;) and full mac power management with proper intel stepping work under osx, but not wifi. And it was ~3.5 years ago... never updated osx since then.
regards
bb
Hi
I was a little bit annoyed with the fact that mlv lite files are not playable in the First-And-Only-Crossplatform-Realtime-MLV-Player-With-Extras and great OpenGL overlay interface. Yes you guessed it - it's MLRawViewer! I'm sure lots of you guys are using it like just a player.
It was sugested by Danne that this behavior was caused by 0 value of the FrameCount in MLV header produced by new raw_rec module (mlvlite). Proof (http://www.magiclantern.fm/forum/index.php?topic=17185.msg166559#msg166559) is here.
Two facts:
1. Unfortunately it seems that baldand not gonna continue to develop the project further, his last post dated "March 22, 2015, 10:21:17AM", and this problem not gonna be fixed. Very sad.
2. This framecount value is no big deal for any software which indexes complex mlv files (spanned/single) before using them. And I guess raw_rec does not pushes it into file after recording for some performance wise reason. And we will not get this value pushed to the file any soon :).
That's why I cooked small easily usable utility to address this mlrawiewer black screen issue.
Additional info and links are on updated first page.
Regards
bb
A better way would be to update the frame count in mlv_lite source code. And I bet it's easier as well - the only difference from plain C is the naming of the file I/O functions (instead of fseek you have FIO_SeekSkipFile, but they work in the same way).
This can be done at the end of the recording, so there's no worry about performance. Maybe just a little care for spanned files.
hehe I new you would reply about no performance issue :) and yes - care about spanned files (you mentioned it while back). My coding is so rusty after years, needed some training...
Regarding of creating raw2dng pull request how can I manage it. I'm good at downloading from the repo not uploading/updating something etc...
regards
bb
Nice one bouncyball. You created what I was thinking of doing. MlRawViewer is the better viewer so hopefully mlv-lite gets the the needed update in raw_rec or I would like to make your fix working in mlp.
Nice that you got MLP up and running by the way @tecgen @bouncyball.
@bouncyball: good, job. thank you :)
Quote from: a1ex on May 12, 2016, 07:40:13 PM
A better way would be to update the frame count in mlv_lite source code.
Done. (https://bitbucket.org/hudson/magic-lantern/commits/47e2c4e)
@g3gg0: Thanks, u r welcome :)
@dmilligan: I'm glad my effort of speeding up elimination of this issue is successfull ;)
regards
bb
QuoteI'm glad my effort of speeding up elimination of this issue is successfull
Amen :P
Excellent work @bouncyball!
Hi all
Updated first post (Update 2).
regards
bb
Most welcome contribution. Great work.
Thanks, keep generating useful requests :)
See what you have done @bouncyball ... This is big time stuff and all the Thanks to you for stream rolling this into a giant bouncy ball that's going to be rolling downhill for a long time. Possibly forever. Legend stuff actually. Okay enough said and I gotta go to @Danne's MLP (http://www.magiclantern.fm/forum/index.php?topic=13512.msg167549#msg167549MLP) thread for the rest...
@DeafEyeJedi: LOL :)