Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

Started by g3gg0, July 15, 2013, 10:58:23 PM

Previous topic - Next topic

0 Members and 4 Guests are viewing this topic.

g3gg0

yeah except the re-use of "internal" structs for the file format, its fine.
RAWC due to "raw crop" ?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

a1ex

Yeah, tried some more names, but since it's very similar to RAWI (adding some additional information about the raw buffer), that's what I've got.

g3gg0

simply CROP wont get the point i guess due to the binning info, same for BINN(ing)
SMPL(ling) looks a bit hmm mismatching, sounds like audio stuff.
SENS(or) could be a candidate, but then i'd embed bayer pattern info and e.g. a pixel scan order (why not?), maybe same for TOPO
maybe ARCH(itecture) for sensor/scanning architecture info or SCAN?
CAPA (capture area), REGN (region) or SATT (sensor/scanning attributes) no SCAT please
CAPI for capture info also looks okay to me

if you fall in love with one of those, good. if not, fine as well :)
better thinking a bit too long about names than later being mad because it was misleading :D
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

togg

I'm not sure it's the right place but I'll link it here: seems like I found a bunch of important and strange bugs in the module as of today.

http://www.magiclantern.fm/forum/index.php?topic=19277.0

Danne

Noticed ACR won,t open files with maximum white level set to 16384 which is default when trying to push white level above this in mlv_dump(10/12bit branch and compressed_raw branch). Maximum is 16383 for acr to chew on. Not sure if all places with 16384 in mlv_dump.c should be set to 16383 so I won,t do a pull request. I did following in mlv_dump.c in compressed_raw branch. The white fix code isn,t in unified yet I think.

            case 'W':
                if(!optarg)
                {
                    white_fix = 15000;
                }
                else
                {
                    white_fix = MIN(16383, MAX(1, atoi(optarg)));
                }
                break;
               
             
            case 'B':
                if(!optarg)
                {
                    black_fix = 2048;
                }
                else
                {
                    black_fix = MIN(16383, MAX(1, atoi(optarg)));
                }
                break;



When set to 16384

               

loknar

Hi,
is there a reason, why mlv_dump for linux isn't auto built anymore? Last available binary version is from 26th December 2016. I'm asking because i need "--white-fix" switch that has been implemented later.

I tried to compile it myself, but I got some errors that I am unable to correct  :-[.

Danne

Hi. I encountered an annoying event around the -f function in mlv_dump. For example. When extracting a selected frame from a bigger MLV file mlv_dump not only extracts the file itself but it keeps processing VIDF blocks until it reaches the end of the file. Now this causes a serious lag overhead and if you have a MLV file that´s a few gb in size it´s a way too hefty price to pay for extracting only one or a few frames.
In my scriptbased tool cr2hdr.app I run this very fine function to parallelize exports of dng frames in chunks of 4 so the extra overhead will add up hogging resources. Question naturally is if this function could work so that as soon as the last selected dng frame is processed into a dng mlv_dump itself will end VIDF processing?

example usage:
mlv_dump --dng -f 1-2 Input.MLV

I could always kill mlv_dump processing from within terminal but the much nicer solution would be to do it from mlv_dump.


*update. Did some extensive testing and realize this overhead is also caused when going from let´s say mlv_dump --dng -f 400-600 For now I will simply exclude the parallellizing in cr2hdr.app.

g3gg0

hi danne

yeah i am aware of that. i also fell over this issue a few times.
but when implementing the functionality, we must be aware that
blocks are sorted randomly in the file / files (yeah frame 1 could be in the last .Mxx)

so one solution could be, allow selecting frames only in cases where an index file is being used. (first do a run with -x)
this at least guarantees that the frames that appear are in order (sorted by time, frame number *should* be consecutive too).
in case of a missing frame (due to frame drop in mlv_rec) this would also work well.

for files processed without a previously created index, the old method would have to be used.


this seemed a bit odd to me in the first place.
still want this feature?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

It´s a little odd, I give you that. I got the idea from so-rose, a linux user here at the forum. I think the feature could be useful if the indexing won´t hog the processing which it does now. I just realised this after parallelizing mlv_dump dng extraction for about one year  :P. (Sorry all cr2hdr.app users, new version already up). For bigger files this is a real showstopper I noticed so just reverted to normal --dng processing.
For other things based around the -f function I could easily kill mlv_dump indexing via kill pid processing.
Below, the simplistic script idea. Hard to tell if there will be a real speed increase and it might be too much hassle here to make it roll effectively. This also must work with all other stuff like vertical stripes and so on.

#frame extraction in mlv_dump makes multithreading possible. Thanks to so-rose for highlighting this possibility
    frct=$(mlv_dump "$FILE" | awk '/Processed/ { print $2; }')
    mlv_dump -f 0-$(($frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1))-$(($frct/4+1+$frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1+$frct/4+1))-$(($frct/4+1+$frct/4+1+$frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1+$frct/4+1+$frct/4+1))-$frct --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE"


g3gg0

also thought about multi core usage, but things get weird when using some ancient things like delta mode..
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

Here´s a tested working mlv_dump_crop_rec4k binary for mac OSX.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_crop_rec4k

Tested some parallell mlv_dump processing on a ssd drive to check for improvements. It sure is possible to speed up things here. Ran 4 mlv files in parallell vs lined up one by one.

4 in parallell
02 Min, 4 Sec

Lined up one by one
03 Min, 37 Sec

My test command(parallell)
mlv_dump --dng M28-2003.MLV &
mlv_dump --dng M28-2004.MLV &
mlv_dump --dng M28-2006.MLV &
time mlv_dump --dng M28-2005.MLV

Danne

A little feature request: Ability to add white balance metadata to the processed dng files via mlv_dump. AsShotNeutral information. User could add multipliers for preset wb like sunny, shady, flourouscent etc but there is also the possibility to add manually computed multipliers working with dcraw.
Maybe a --wb= switch?

DeafEyeJedi

Quote from: Danne on May 31, 2017, 07:55:04 PM
A little feature request: Ability to add white balance metadata to the processed dng files via mlv_dump. AsShotNeutral information. User could add multipliers for preset wb like sunny, shady, flourouscent etc but there is also the possibility to add manually computed multipliers working with dcraw.
Maybe a --wb= switch?

+1 and this is a really good idea!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

g3gg0

does the WBAL block help anything? you can select it in camera.
or do you want to "correct" this when exporting to DNG?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

Solely for the purpose to correct wb to the dng files in post.
In the dng:
As Shot Neutral                 : 0.5793613707 1 0.5545483871
Something like following would be quite useful imo:
example:
mlv_dump --dng --wb=0.6793613707 1 0.7545483871

Danne

@g3gg0
I wonder if there could be an abort safeguard for a certain error that can occur when creating shortened MLV files?
http://www.magiclantern.fm/forum/index.php?topic=20418.msg189471#msg189471

typically one would run something like this:
mlv_dump -f 5 -o Output.MLV Input.MLV
When both input and output are named the same the whole MLV file is emptied from content, poff.
mlv_dump -f 5 -o Output.MLV Output.MLV

g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

pulsar124

I  just upgraded to 6D (from 50D), and noticed that I cannot open MLV files made with 6D, using the tools which worked fine with 50D (a year ago). I am using a recent nightly build (magiclantern-Nightly.2017Aug18.6D116), and load module mlv_rec and silent. I take full resolution silent pictures. There seem to be no issues taking the photos, the file size seem to be right, but neither mlv_dump nor MLV Mystic work for me. The Mystic would run way too fast, and there would be no DNG files produced. mlv_dump -v seems to report no errors, runs a bit longer (prints the exif info), but then again no files are generated.

Is it a known issue? How could I troubleshoot this?

Here is one of these files (5 shots; 177 MB):

http://syam.no-ip.org/pub/11370000.MLV

Perhaps I don't understand how to use mlv_dump? My command is

mlv_dump --dng 11370000.MLV

Thanks!

g3gg0


Block: RAWI
  Offset: 0x00000034
  Number: 1
    Size: 180
    Time: 4.339000 ms
    Res:  5568x3723
    raw_info:
      api_version      0x00000001
      height           3723
      width            5568
      pitch            9744
      frame_size       0x02298AB0
      bits_per_pixel   14
      black_level      2047
      white_level      14783
      active_area.y1   52
      active_area.x1   72
      active_area.y2   3723
      active_area.x2   5568
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1


a vertical resolution that is not a multiple of two? uhm... quite unusual..?

nevertheless, mlv_dump should not crash. preparing a fix.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!


g3gg0

crashed on my machine. depends on how memory layout appears to be.
reason: detect_vertical_stripes_coeffs() goes through all line pairs and if there is an uneven number of lines, it accesses beyond allocated memory

wait till the build job finished, then pick your executable (.exe for windows)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

pulsar124

Thanks, I'll try this now. I wonder why I got the odd number of pixels - perhaps it's normal for 6D?

EDIT: g3gg0: I am still getting the same behaviour, with your latest build. BTW the program doesn't crash on me - it just runs too fast, produces no error messages, but also no DNG files. I am using Windows 7 64 bit.

pulsar124

Danne: your mlv_dump.exe worked, thanks a bunch! I just copied your mlv_dump.exe inside my images folder, run it from there, and it worked without issues, for both small (5 shots) and large (144 shots) MLV files I made with 6D.