Author Topic: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)  (Read 807765 times)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3025
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2100 on: March 22, 2017, 11:53:16 PM »
yeah except the re-use of "internal" structs for the file format, its fine.
RAWC due to "raw crop" ?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2101 on: March 23, 2017, 12:23:58 AM »
Yeah, tried some more names, but since it's very similar to RAWI (adding some additional information about the raw buffer), that's what I've got.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3025
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2102 on: March 23, 2017, 12:59:02 AM »
simply CROP wont get the point i guess due to the binning info, same for BINN(ing)
SMPL(ling) looks a bit hmm mismatching, sounds like audio stuff.
SENS(or) could be a candidate, but then i'd embed bayer pattern info and e.g. a pixel scan order (why not?), maybe same for TOPO
maybe ARCH(itecture) for sensor/scanning architecture info or SCAN?
CAPA (capture area), REGN (region) or SATT (sensor/scanning attributes) no SCAT please
CAPI for capture info also looks okay to me

if you fall in love with one of those, good. if not, fine as well :)
better thinking a bit too long about names than later being mad because it was misleading :D
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

togg

  • Senior
  • ****
  • Posts: 278
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2103 on: March 27, 2017, 11:26:46 PM »
I'm not sure it's the right place but I'll link it here: seems like I found a bunch of important and strange bugs in the module as of today.

http://www.magiclantern.fm/forum/index.php?topic=19277.0

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2104 on: April 07, 2017, 08:01:56 AM »
Noticed ACR won,t open files with maximum white level set to 16384 which is default when trying to push white level above this in mlv_dump(10/12bit branch and compressed_raw branch). Maximum is 16383 for acr to chew on. Not sure if all places with 16384 in mlv_dump.c should be set to 16383 so I won,t do a pull request. I did following in mlv_dump.c in compressed_raw branch. The white fix code isn,t in unified yet I think.

Code: [Select]
            case 'W':
                if(!optarg)
                {
                    white_fix = 15000;
                }
                else
                {
                    white_fix = MIN(16383, MAX(1, atoi(optarg)));
                }
                break;
               
             
            case 'B':
                if(!optarg)
                {
                    black_fix = 2048;
                }
                else
                {
                    black_fix = MIN(16383, MAX(1, atoi(optarg)));
                }
                break;


When set to 16384

               

loknar

  • New to the forum
  • *
  • Posts: 11
  • 550D, EOS M
mlv_dump for linux
« Reply #2105 on: April 20, 2017, 09:36:13 PM »
Hi,
is there a reason, why mlv_dump for linux isn't auto built anymore? Last available binary version is from 26th December 2016. I'm asking because i need "--white-fix" switch that has been implemented later.

I tried to compile it myself, but I got some errors that I am unable to correct  :-[.

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2106 on: May 29, 2017, 02:50:45 PM »
Hi. I encountered an annoying event around the -f function in mlv_dump. For example. When extracting a selected frame from a bigger MLV file mlv_dump not only extracts the file itself but it keeps processing VIDF blocks until it reaches the end of the file. Now this causes a serious lag overhead and if you have a MLV file that´s a few gb in size it´s a way too hefty price to pay for extracting only one or a few frames.
In my scriptbased tool cr2hdr.app I run this very fine function to parallelize exports of dng frames in chunks of 4 so the extra overhead will add up hogging resources. Question naturally is if this function could work so that as soon as the last selected dng frame is processed into a dng mlv_dump itself will end VIDF processing?

example usage:
Code: [Select]
mlv_dump --dng -f 1-2 Input.MLV
I could always kill mlv_dump processing from within terminal but the much nicer solution would be to do it from mlv_dump.


*update. Did some extensive testing and realize this overhead is also caused when going from let´s say
Code: [Select]
mlv_dump --dng -f 400-600 For now I will simply exclude the parallellizing in cr2hdr.app.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3025
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2107 on: May 29, 2017, 08:45:08 PM »
hi danne

yeah i am aware of that. i also fell over this issue a few times.
but when implementing the functionality, we must be aware that
blocks are sorted randomly in the file / files (yeah frame 1 could be in the last .Mxx)

so one solution could be, allow selecting frames only in cases where an index file is being used. (first do a run with -x)
this at least guarantees that the frames that appear are in order (sorted by time, frame number *should* be consecutive too).
in case of a missing frame (due to frame drop in mlv_rec) this would also work well.

for files processed without a previously created index, the old method would have to be used.


this seemed a bit odd to me in the first place.
still want this feature?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3025
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2109 on: May 29, 2017, 09:20:12 PM »
It´s a little odd, I give you that. I got the idea from so-rose, a linux user here at the forum. I think the feature could be useful if the indexing won´t hog the processing which it does now. I just realised this after parallelizing mlv_dump dng extraction for about one year  :P. (Sorry all cr2hdr.app users, new version already up). For bigger files this is a real showstopper I noticed so just reverted to normal --dng processing.
For other things based around the -f function I could easily kill mlv_dump indexing via kill pid processing.
Below, the simplistic script idea. Hard to tell if there will be a real speed increase and it might be too much hassle here to make it roll effectively. This also must work with all other stuff like vertical stripes and so on.

Code: [Select]
#frame extraction in mlv_dump makes multithreading possible. Thanks to so-rose for highlighting this possibility
    frct=$(mlv_dump "$FILE" | awk '/Processed/ { print $2; }')
    mlv_dump -f 0-$(($frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1))-$(($frct/4+1+$frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1+$frct/4+1))-$(($frct/4+1+$frct/4+1+$frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
    mlv_dump -f $(($frct/4+1+$frct/4+1+$frct/4+1))-$frct --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE"

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3025
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2110 on: May 29, 2017, 09:45:01 PM »
also thought about multi core usage, but things get weird when using some ancient things like delta mode..
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3025
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2111 on: May 29, 2017, 10:10:31 PM »
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2112 on: May 30, 2017, 10:10:22 AM »
Here´s a tested working mlv_dump_crop_rec4k binary for mac OSX.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_crop_rec4k

Tested some parallell mlv_dump processing on a ssd drive to check for improvements. It sure is possible to speed up things here. Ran 4 mlv files in parallell vs lined up one by one.

4 in parallell
02 Min, 4 Sec

Lined up one by one
03 Min, 37 Sec

My test command(parallell)
mlv_dump --dng M28-2003.MLV &
mlv_dump --dng M28-2004.MLV &
mlv_dump --dng M28-2006.MLV &
time mlv_dump --dng M28-2005.MLV

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2113 on: May 31, 2017, 07:55:04 PM »
A little feature request: Ability to add white balance metadata to the processed dng files via mlv_dump. AsShotNeutral information. User could add multipliers for preset wb like sunny, shady, flourouscent etc but there is also the possibility to add manually computed multipliers working with dcraw.
Maybe a --wb= switch?

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2955
  • 5D3 / M1 / 7D / 70D / SL1
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2114 on: August 05, 2017, 05:32:29 AM »
A little feature request: Ability to add white balance metadata to the processed dng files via mlv_dump. AsShotNeutral information. User could add multipliers for preset wb like sunny, shady, flourouscent etc but there is also the possibility to add manually computed multipliers working with dcraw.
Maybe a --wb= switch?

+1 and this is a really good idea!
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3025
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2115 on: August 05, 2017, 11:57:47 AM »
does the WBAL block help anything? you can select it in camera.
or do you want to "correct" this when exporting to DNG?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2116 on: August 05, 2017, 12:25:04 PM »
Solely for the purpose to correct wb to the dng files in post.
In the dng:
Code: [Select]
As Shot Neutral                 : 0.5793613707 1 0.5545483871Something like following would be quite useful imo:
example:
Code: [Select]
mlv_dump --dng --wb=0.6793613707 1 0.7545483871

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2117 on: September 01, 2017, 02:42:13 PM »
@g3gg0
I wonder if there could be an abort safeguard for a certain error that can occur when creating shortened MLV files?
http://www.magiclantern.fm/forum/index.php?topic=20418.msg189471#msg189471

typically one would run something like this:
Code: [Select]
mlv_dump -f 5 -o Output.MLV Input.MLVWhen both input and output are named the same the whole MLV file is emptied from content, poff.
Code: [Select]
mlv_dump -f 5 -o Output.MLV Output.MLV

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3025
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #2118 on: September 01, 2017, 11:34:02 PM »
doh... yeah easy to prevent
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!