mlv_dump on steroids

[bouncyball]

Hi there,

First of all I want to admit: this is not the first time when someone modified mlv_dump. I also did it once not so long while ago but this time it's little bit different. This one based on latest vanilla MLVFS source files edited and changed by me.

Also, I can say that digging into MLVFS sources is a real joy . I love it starting from the idea and ending with the very cool multiplatform implementation. I want to thank David Milligan for this gem, special thanks to Danne (he always supports my development attempts), also a1ex and g3gg0 for all this thing called ML project moving forward and all active maintainers and devs of other ports and all the community around the project. Thank you guys!

Now about the subject:

UPDATE:
UPDATE2:
UPDATE3 (2018):
UPDATE4 (last!): It is merged to official ML's crop_rec_4k branch: Link

( hopefully ) it:
1. Saves 16bit any bit depth Cinema DNG files. e.g. 10/12/14 bits compressed or uncompressed, "--no-bitpack" exports 16 bit uncompressed DNGs.
2. Supports any bit depth raw data as input compressed or uncompressed
3. Has additional or reworked switch options:

Code Select Expand


---
  --no-audio          for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination MLV
---
  --no-fixfp          do not fix focus pixels
  --fixcp2            use aggressive method for revealing more bad pixels
  --no-stripes        do not fix vertical stripes in highlights
  --force-stripes     compute stripe correction for every frame
  --is-dualiso        use dual iso compatible horizontal interpolation of focus and bad pixels
  --is-croprec        generate focus map for crop_rec mode
  --save-bpm          save bad pixels to .BPM file
  --fixpn             fix pattern noise
  --deflicker=value   per-frame exposure compensation. value is target median in raw units ex: 3072 (default)
  --no-bitpack        write DNG files with unpacked to 16 bit raw data
  --show-progress     show DNG file creation progress. ignored when -v or --batch is specified
                      also works when compressing MLV to MLV and shows compression ratio for each frame
  --fpi <method>      focus pixel interpolation method: 0 (mlvfs), 1 (raw2dng), default is 1
  --bpi <method>      bad pixel interpolation method: 0 (mlvfs), 1 (raw2dng), default is 0

---
  -c                  compress video frames using LJ92. if input is lossless, then decompress and recompress again.
  -d                  decompress compressed video and audio frames using LZMA or LJ92
  -p                  pass through original raw data without processing, it works for lossless or uncompressed raw

"--force-stripes" forces stripe detection for each frame separately. Slow but more effective on some footage
"-p" option replaces "-c -c" and passes through unmodified data for both uncompressed or lossless compressed raw.
"--no-audio" for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination MLV, it's forced when "-f" used
"--no-fixfp" option deactivates fixing of focus pixels"
"--is-croprec" option  generates focus map for crop_rec mode"

4. Supports focus pixels the same way MLVFS does, also has ability to override and load '.fpm' manually if file has the same name as input MLV. Automatically fixes focus pixels for losless or uncompressed MLVs without special maps. If could not fix some dots for some special case map file can be specified as usual to override auto fixing. Auto detects crop_rec mode
5. Saves/Loads '.bpm' (bad pixel map) files. Name of the file same as input MLV
6. Can fix pattern noise
7. Has deflicker option of MLVFS
8. --show-progress prints every operation's done while saving DNGs, also works when compressing MLV to MLV and shows compression ratio for each frame
9. Implemented advanced WAV header with iXML for demanding NLEs
10. Includes 'crop_rec_4k' branch goodies/changes
11. Correctly fills up 'Unique Camera Model' tag in dng header and has additional camera matrixes
12. With "-b" switch you can convert bit depth and related black/white levels and export to that bit depth DNG (uncompressed only)
13. Chroma smooth code updated to the more advanced one
14. Added pixel interpolation method from raw2dng
15. Introduced two command line switches: "--fpi" and "--bpi" (see usage), by default, for focus pixels RAW2DNG method (correct edge pixel processing added) and for bad pixels MLVFS method is used
16. Supports new focus pixel maps w/header, generated by fpmutil (in addition to original simple .fpm), 'fpmutil' binaries can be downloaded here
17. Well... maybe I forgot something to mention

'fpmutil' also updated and can be downloaded from the links above as usual

Any feedback is welcome!

Here are links to the fork and downloads.
Here is a branch with simplified changeset to understand, with less effort, how 'mlv_dump.c' was modified and what has changed.

Now it's merged to Magic Lantern repository: Link

The best regards
bb

[Levas]

Only option I'm missing, since 10/12 bit raw recording  , is the option for different bit depths when creating dng's.

I know it's crazy, but I keep the dng's as backup and throw away the MLV files
So an option for 10, 12 or 14 bit dng output, according to my recording bit depth in the original MLV, would be killer future .

[Danne]

On steroids I say. It,s beautiful rework and addons. Don,t forget to mention white balance support as well as darkframe and flatframe inclusion. I,ll check deeper into this once I have some time. Here is a binary for mac meanwhile.
Great work.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_macOS

[bouncyball]

@Levas:
Yeah I understand, but the whole purpose of using mlvfs universal implementation is to convert any bit depth sources to 16 bit and then do raw processing on 16 bit buffers (downconverting again is a bad idea). Maybe some day experimental hardware raw compression from silent pictures goes to the mlv_rec/mlv_lite who knows.

@Danne:
Hey! Thanx for the binary. Uploaded.

[mothaibaphoto]

Hello, bouncyball. It's a incredibly great.
Why you don't a "Hero member" yet? Yes, that message counters just nonsense...
Ok, on topic: Ironically, I just finished to transcode a couple of TB with old gold mlv_dump.
Actually, due to this issue
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173295#msg173295
i used 2 versions of mlv_dump: one for the dual iso shots, and another(which corrupts dual iso) for the rest -
because that one handles vertical stripes better, to my eye.
As far as i know, that modification finally was removed:
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173394#msg173394
So, the first question - what version of vertical stripes fix code do you use?
Is it possible to have both via swich or something like that?
And, second: there was a discussion in MLVFS thread, that MLVFS does not analyse every frame for vertical stripes for performance reason,
and i had some persisted stripes on panned shots with MLVFS.
What is your approach here?
Best regards, and really great thanks for your contribution.

[mothaibaphoto]

@Levas: I keep DNG too. I compress it with Adobe DNG converter.
Virtually there is no difference in size between 14 and 16 bit versions after compression.
And, i guess, 10 and 12 bit will have no benefit in size after compression.

[bouncyball]

So, the first question - what version of vertical stripes fix code do you use?
Is it possible to have both via swich or something like that?
And, second: there was a discussion in MLVFS thread, that MLVFS does not analyse every frame for vertical stripes for performance reason,

Well, It's exact MLVFS stripe removal code not from raw2dng however I guess technically they are quite similar.

What do you mean by "possible to have both via swich"?

The stripes analyzed on zero frame or 1st frame of a range, they could be analyzed on every frame though. Can be implemented via switch.

[mothaibaphoto]

What do you mean by "possible to have both via swich"?

I mean with this:
https://bitbucket.org/hudson/magic-lantern/commits/ab24965b3f4d45e4cba10214bc0061f36b440730
and without

The stripes analyzed on zero frame or 1st frame of a range, they could be analyzed on every frame though. Can be implemented via switch.

It really matters for some shots.
As long as performance not so important when pre-converting, it would be very useful to have this switch.

[bouncyball]

Ok understood. MLVFS doe not include that fix from a1ex. Also, I did not port the dualiso stuff from mlvfs anyway.

Will think about --stripes2 switch.

[Danne]

Tested some clips with fpm files and it works great just like in mlvfs. Tried the load .bpm file but it doesn,t seem to come through. The bpm files loads but the pixels aren,t patched. In your older .map code which I,m using now the dots are gone but the same coordinates don,t apply here. Maybe the zeroes at the end are disturbing?

My test files here: https://drive.google.com/file/d/0B4tCJMlOYfira2hKTEhIdmwzZ3c/view?usp=sharing
Note that this file won,t work with mlvfs fpm files since it,s a 700D crop_rec test file(experimental build from dfort). It,s the same as for eosm so it works with map files and it also loads the .bpm file in your steroid version but focus pixels are still present.

Code Select Expand

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/Daniel/Desktop/test/700D_crop_rec_test.MLV'
   - Convert to DNG frames
   - Output into '/Users/Daniel/Desktop/test/700D_crop_rec_test_'
File /Users/Daniel/Desktop/test/700D_crop_rec_test.MLV opened
File /Users/Daniel/Desktop/test/700D_crop_rec_test.M00 not existing.
Processing...

Using bad pixel map: '/Users/Daniel/Desktop/test/700D_crop_rec_test.bpm'
39060 pixels loaded

Vertical stripes correction: 'UNNEEDED'
  1.00000  1.00000  1.00014  1.00079  0.99968  0.99924  0.99899  1.00012

Extracting frames...
Current frame : '/Users/Daniel/Desktop/test/700D_crop_rec_test_000001.dng'
Reached end of chunk 1/1 after 9 blocks
Processed 2 video frames
Done

[Levas]

@mothaibaphoto
Thanks for the tip for using Adobe DNG converter, will try it out.

[bouncyball]

@Danne: I'll take a look

[mothaibaphoto]

Sorry for offtopic, but:
@Levas:
Keep in mind that after DNG compressor your DNG are not CDNG anymore. This means you loose FPS tag at the very least and may have some problems with your post software.

[Levas]

No problem, using RawTherapee for editing DNG's, export as TIFF sequence and use these in DaVinci Resolve

[bouncyball]

It really matters for some shots.
As long as performance not so important when pre-converting, it would be very useful to have this switch.

Had no time until today. Implemented --force-stripes switch. Please report the result.

[Danne]

For mac
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_steroids_1.0

[mothaibaphoto]

Cool, but I have no footage to test on now

[bouncyball]

Hi guys,

I updated downloads on the first post. Latest binaries (linux/mac/win) with lot of internal changes and bug fixes.
I hope someone's using it, really need some beta tester's feedback

bb

[DeafEyeJedi]

Thanks for this @bouncyball and just downloaded. Will test and report back my findings after trying it out with a DF Avg process test. 

*edit*

Actually probably due to my idiotic moment but I notice the mlv_dump (for Mac OS) file I downloaded isn't a binary -- did I miss a step in this?

[g3gg0]

hmh, maybe i missed the point.
but why develop a separate branch that grows too far from the main tree?

[bouncyball]

@DeafEyeJedi: Thanks for your help

Actually probably due to my idiotic moment but I notice the mlv_dump (for Mac OS) file I downloaded isn't a binary -- did I miss a step in this?

Hmm... strange. The binary compiled/linked and running well under my virtual Sierra. I just stripped off the debug info by strip command to make it smaller. Did you rename it to mlv_dump and make it executable?

[bouncyball]

@g3gg0: Do you mean I gotta make pull request?

but why develop a separate branch that grows too far from the main tree?

The point is that there were a lot of things to do (remove very mlvs specific code, add my functions and make some optimizations/bugfixes, still want to add some more stuff) and at that moment I hesitated to do PR. Right now I think it's possible.

I've started it as a fork of unified, then included goodies from 10/12 branch, then just merged it to that branch and it's there now. It even merges to the unified cleanly at least the way as 10/12bit branch does.

regards
bb

[Danne]

@deafeyejedi
chmod u=rwx mlv_dump
Enter

[bouncyball]

@Danne: Yup. Right

[DeafEyeJedi]

@deafeyejedi
chmod u=rwx mlv_dump
Enter

That did it. Thanks!