Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

[g3gg0]

yeah except the re-use of "internal" structs for the file format, its fine.
RAWC due to "raw crop" ?

[a1ex]

Yeah, tried some more names, but since it's very similar to RAWI (adding some additional information about the raw buffer), that's what I've got.

[g3gg0]

simply CROP wont get the point i guess due to the binning info, same for BINN(ing)
SMPL(ling) looks a bit hmm mismatching, sounds like audio stuff.
SENS(or) could be a candidate, but then i'd embed bayer pattern info and e.g. a pixel scan order (why not?), maybe same for TOPO
maybe ARCH(itecture) for sensor/scanning architecture info or SCAN?
CAPA (capture area), REGN (region) or SATT (sensor/scanning attributes) _{no SCAT please}
CAPI for capture info also looks okay to me

if you fall in love with one of those, good. if not, fine as well
better thinking a bit too long about names than later being mad because it was misleading

[togg]

I'm not sure it's the right place but I'll link it here: seems like I found a bunch of important and strange bugs in the module as of today.

http://www.magiclantern.fm/forum/index.php?topic=19277.0

[Danne]

Noticed ACR won,t open files with maximum white level set to 16384 which is default when trying to push white level above this in mlv_dump(10/12bit branch and compressed_raw branch). Maximum is 16383 for acr to chew on. Not sure if all places with 16384 in mlv_dump.c should be set to 16383 so I won,t do a pull request. I did following in mlv_dump.c in compressed_raw branch. The white fix code isn,t in unified yet I think.

Code: [Select]

            case 'W':
                if(!optarg)
                {
                    white_fix = 15000;
                }
                else
                {
                    white_fix = MIN(16383, MAX(1, atoi(optarg)));
                }
                break;
               
             
            case 'B':
                if(!optarg)
                {
                    black_fix = 2048;
                }
                else
                {
                    black_fix = MIN(16383, MAX(1, atoi(optarg)));
                }
                break;

When set to 16384

               

[loknar]

Hi,
is there a reason, why mlv_dump for linux isn't auto built anymore? Last available binary version is from 26th December 2016. I'm asking because i need "--white-fix" switch that has been implemented later.

I tried to compile it myself, but I got some errors that I am unable to correct  .

[Danne]

Hi. I encountered an annoying event around the -f function in mlv_dump. For example. When extracting a selected frame from a bigger MLV file mlv_dump not only extracts the file itself but it keeps processing VIDF blocks until it reaches the end of the file. Now this causes a serious lag overhead and if you have a MLV file that´s a few gb in size it´s a way too hefty price to pay for extracting only one or a few frames.
In my scriptbased tool cr2hdr.app I run this very fine function to parallelize exports of dng frames in chunks of 4 so the extra overhead will add up hogging resources. Question naturally is if this function could work so that as soon as the last selected dng frame is processed into a dng mlv_dump itself will end VIDF processing?

example usage:

Code: [Select]

mlv_dump --dng -f 1-2 Input.MLV
I could always kill mlv_dump processing from within terminal but the much nicer solution would be to do it from mlv_dump.


*update. Did some extensive testing and realize this overhead is also caused when going from let´s say

Code: [Select]

mlv_dump --dng -f 400-600 For now I will simply exclude the parallellizing in cr2hdr.app.

[g3gg0]

hi danne

yeah i am aware of that. i also fell over this issue a few times.
but when implementing the functionality, we must be aware that
blocks are sorted randomly in the file / files (yeah frame 1 could be in the last .Mxx)

so one solution could be, allow selecting frames only in cases where an index file is being used. (first do a run with -x)
this at least guarantees that the frames that appear are in order (sorted by time, frame number *should* be consecutive too).
in case of a missing frame (due to frame drop in mlv_rec) this would also work well.

for files processed without a previously created index, the old method would have to be used.


this seemed a bit odd to me in the first place.
still want this feature?

[g3gg0]

mlv_dump for linux

can you try the one i added there?
https://builds.magiclantern.fm/jenkins/view/Experiments/job/crop_rec_4k/24/

[Danne]

It´s a little odd, I give you that. I got the idea from so-rose, a linux user here at the forum. I think the feature could be useful if the indexing won´t hog the processing which it does now. I just realised this after parallelizing mlv_dump dng extraction for about one year  . (Sorry all cr2hdr.app users, new version already up). For bigger files this is a real showstopper I noticed so just reverted to normal --dng processing.
For other things based around the -f function I could easily kill mlv_dump indexing via kill pid processing.
Below, the simplistic script idea. Hard to tell if there will be a real speed increase and it might be too much hassle here to make it roll effectively. This also must work with all other stuff like vertical stripes and so on.

Code: [Select]

#frame extraction in mlv_dump makes multithreading possible. Thanks to so-rose for highlighting this possibility
    frct=$(mlv_dump "$FILE" | awk '/Processed/ { print $2; }')
    mlv_dump -f 0-$(($frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1))-$(($frct/4+1+$frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1+$frct/4+1))-$(($frct/4+1+$frct/4+1+$frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1+$frct/4+1+$frct/4+1))-$frct --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE"

[g3gg0]

also thought about multi core usage, but things get weird when using some ancient things like delta mode..

[g3gg0]

maybe someone could try the OSX binary too?
https://builds.magiclantern.fm/jenkins/view/Experiments/job/crop_rec_4k/24/

[Danne]

Here´s a tested working mlv_dump_crop_rec4k binary for mac OSX.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_crop_rec4k

Tested some parallell mlv_dump processing on a ssd drive to check for improvements. It sure is possible to speed up things here. Ran 4 mlv files in parallell vs lined up one by one.

4 in parallell
02 Min, 4 Sec

Lined up one by one
03 Min, 37 Sec

My test command(parallell)
mlv_dump --dng M28-2003.MLV &
mlv_dump --dng M28-2004.MLV &
mlv_dump --dng M28-2006.MLV &
time mlv_dump --dng M28-2005.MLV

[Danne]

A little feature request: Ability to add white balance metadata to the processed dng files via mlv_dump. AsShotNeutral information. User could add multipliers for preset wb like sunny, shady, flourouscent etc but there is also the possibility to add manually computed multipliers working with dcraw.
Maybe a --wb= switch?

[DeafEyeJedi]

A little feature request: Ability to add white balance metadata to the processed dng files via mlv_dump. AsShotNeutral information. User could add multipliers for preset wb like sunny, shady, flourouscent etc but there is also the possibility to add manually computed multipliers working with dcraw.
Maybe a --wb= switch?

+1 and this is a really good idea!

[g3gg0]

does the WBAL block help anything? you can select it in camera.
or do you want to "correct" this when exporting to DNG?

[Danne]

Solely for the purpose to correct wb to the dng files in post.
In the dng:

Code: [Select]

As Shot Neutral                 : 0.5793613707 1 0.5545483871Something like following would be quite useful imo:
example:

Code: [Select]

mlv_dump --dng --wb=0.6793613707 1 0.7545483871

[Danne]

@g3gg0
I wonder if there could be an abort safeguard for a certain error that can occur when creating shortened MLV files?
http://www.magiclantern.fm/forum/index.php?topic=20418.msg189471#msg189471

typically one would run something like this:

Code: [Select]

mlv_dump -f 5 -o Output.MLV Input.MLVWhen both input and output are named the same the whole MLV file is emptied from content, poff.

Code: [Select]

mlv_dump -f 5 -o Output.MLV Output.MLV

[g3gg0]

doh... yeah easy to prevent

[pulsar124]

I  just upgraded to 6D (from 50D), and noticed that I cannot open MLV files made with 6D, using the tools which worked fine with 50D (a year ago). I am using a recent nightly build (magiclantern-Nightly.2017Aug18.6D116), and load module mlv_rec and silent. I take full resolution silent pictures. There seem to be no issues taking the photos, the file size seem to be right, but neither mlv_dump nor MLV Mystic work for me. The Mystic would run way too fast, and there would be no DNG files produced. mlv_dump -v seems to report no errors, runs a bit longer (prints the exif info), but then again no files are generated.

Is it a known issue? How could I troubleshoot this?

Here is one of these files (5 shots; 177 MB):

http://syam.no-ip.org/pub/11370000.MLV

Perhaps I don't understand how to use mlv_dump? My command is

mlv_dump --dng 11370000.MLV

Thanks!

[g3gg0]

Code: [Select]

Block: RAWI
  Offset: 0x00000034
  Number: 1
    Size: 180
    Time: 4.339000 ms
    Res:  5568x3723
    raw_info:
      api_version      0x00000001
      height           3723
      width            5568
      pitch            9744
      frame_size       0x02298AB0
      bits_per_pixel   14
      black_level      2047
      white_level      14783
      active_area.y1   52
      active_area.x1   72
      active_area.y2   3723
      active_area.x2   5568
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1

a vertical resolution that is not a multiple of two? uhm... quite unusual..?

nevertheless, mlv_dump should not crash. preparing a fix.

[Danne]

Try:
Mac
http://www.magiclantern.fm/forum/index.php?topic=15108.0
Windows
http://www.magiclantern.fm/forum/index.php?topic=10526.msg188558#msg188558

Tried your file on mac and working.

[g3gg0]

crashed on my machine. depends on how memory layout appears to be.
reason: detect_vertical_stripes_coeffs() goes through all line pairs and if there is an uneven number of lines, it accesses beyond allocated memory

wait till the build job finished, then pick your executable (.exe for windows)

[pulsar124]

Thanks, I'll try this now. I wonder why I got the odd number of pixels - perhaps it's normal for 6D?

EDIT: g3gg0: I am still getting the same behaviour, with your latest build. BTW the program doesn't crash on me - it just runs too fast, produces no error messages, but also no DNG files. I am using Windows 7 64 bit.

[pulsar124]

Danne: your mlv_dump.exe worked, thanks a bunch! I just copied your mlv_dump.exe inside my images folder, run it from there, and it worked without issues, for both small (5 shots) and large (144 shots) MLV files I made with 6D.