Author Topic: mlv_dump on steroids  (Read 1737 times)

bouncyball

  • Member
  • ***
  • Posts: 147
mlv_dump on steroids
« on: February 16, 2017, 07:10:10 PM »
Hi there,

First of all I want to admit: this is not the fist time when someone modified mlv_dump. I also did it once not so long while ago but this time it's little bit different. This one based on latest vanilla MLVFS source files edited and changed by me.

Also, I can say that digging into MLVFS sources is a real joy :D. I love it starting from the idea and ending with the very cool multiplatform implementation. I want to thank David Milligan for this gem, special thanks to Danne (he always supports my development attempts), also a1ex and g3gg0 for all this thing called ML project moving forward and all active maintainers and devs of other ports and all the community around the project. Thank you guys!

Now about the subject:

( hopefully ;) ) it:
1. Saves 16bit Cinema DNG files
2. Supports any bit depth raw data as input
3. Has additional or reworked switch options:
Code: [Select]
-- DNG output --
 --dng               output frames into separate .dng files. set prefix with -o
 --no-cs             no chroma smoothing (default)
 --cs2x2             2x2 chroma smoothing
 --cs3x3             3x3 chroma smoothing
 --cs5x5             5x5 chroma smoothing
 --no-fixcp          do not fix bad pixels
 --fixcp2            use aggressive method for revealing more bad pixels
 --save-bpm          save bad pixels to .BPM file
 --no-stripes        do not fix vertical stripes in highlights
 --force-stripes     compute stripe correction for every frame
 --fixpn             fix pattern noise
 --deflicker=value   per-frame exposure compensation. value is target median in raw units ex: 3072 (default)
 --show-progress     show DNG file creation progress. ignored when -v or --batch is specified

4. Supports focus pixels the same way MLVFS does, also has ability to override and load '.fpm' manually if file has the same name as input MLV
5. Saves/Loads '.bpm' (bad pixel map) files. Name of the file same as input MLV
6. Can fix pattern noise
7. Has deflicker option of MLVFS
8. --show-progress prints every operation's done while saving DNGs
9. Implemented advanced WAV header with iXML for demanding NLEs
10. Includes 'raw_video_10bit_12bit' branch goodies/changes
11. Correctly fills up 'Unique Camera Model' tag in dng header and has additional camera matrixes
12. well... maybe I forgot something to mention here

Any feedback is welcome!

Here are links to the fork and downloads.

The best regards
bb

Levas

  • Hero Member
  • *****
  • Posts: 834
  • 6d - Nightly build user
Re: mlv_dump on steroids
« Reply #1 on: February 16, 2017, 07:41:44 PM »
Only option I'm missing, since 10/12 bit raw recording  ;D, is the option for different bit depths when creating dng's.

I know it's crazy, but I keep the dng's as backup and throw away the MLV files :P
So an option for 10, 12 or 14 bit dng output, according to my recording bit depth in the original MLV, would be killer future 8).

Danne

  • Hero Member
  • *****
  • Posts: 2719
Re: mlv_dump on steroids
« Reply #2 on: February 16, 2017, 08:14:34 PM »
On steroids I say. It,s beautiful rework and addons. Don,t forget to mention white balance support as well as darkframe and flatframe inclusion. I,ll check deeper into this once I have some time. Here is a binary for mac meanwhile.
Great work.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_macOS

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #3 on: February 17, 2017, 06:13:07 AM »
@Levas:
Yeah I understand, but the whole purpose of using mlvfs universal implementation is to convert any bit depth sources to 16 bit and then do raw processing on 16 bit buffers (downconverting again is a bad idea). Maybe some day experimental hardware raw compression from silent pictures goes to the mlv_rec/mlv_lite who knows.

@Danne:
Hey! Thanx for the binary. Uploaded.

mothaibaphoto

  • Senior
  • ****
  • Posts: 311
  • pesky kid
Re: mlv_dump on steroids
« Reply #4 on: February 17, 2017, 06:38:56 AM »
Hello, bouncyball. It's a incredibly great.
Why you don't a "Hero member" yet? Yes, that message counters just nonsense...
Ok, on topic: Ironically, I just finished to transcode a couple of TB with old gold mlv_dump.
Actually, due to this issue
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173295#msg173295
i used 2 versions of mlv_dump: one for the dual iso shots, and another(which corrupts dual iso) for the rest -
because that one handles vertical stripes better, to my eye.
As far as i know, that modification finally was removed:
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173394#msg173394
So, the first question - what version of vertical stripes fix code do you use?
Is it possible to have both via swich or something like that?
And, second: there was a discussion in MLVFS thread, that MLVFS does not analyse every frame for vertical stripes for performance reason,
and i had some persisted stripes on panned shots with MLVFS.
What is your approach here?
Best regards, and really great thanks for your contribution.

mothaibaphoto

  • Senior
  • ****
  • Posts: 311
  • pesky kid
Re: mlv_dump on steroids
« Reply #5 on: February 17, 2017, 06:44:24 AM »
@Levas: I keep DNG too. I compress it with Adobe DNG converter.
Virtually there is no difference in size between 14 and 16 bit versions after compression.
And, i guess, 10 and 12 bit will have no benefit in size after compression.

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #6 on: February 17, 2017, 07:57:52 AM »
So, the first question - what version of vertical stripes fix code do you use?
Is it possible to have both via swich or something like that?
And, second: there was a discussion in MLVFS thread, that MLVFS does not analyse every frame for vertical stripes for performance reason,
Well, It's exact MLVFS stripe removal code not from raw2dng however I guess technically they are quite similar.

What do you mean by "possible to have both via swich"?

The stripes analyzed on zero frame or 1st frame of a range, they could be analyzed on every frame though. Can be implemented via switch.

mothaibaphoto

  • Senior
  • ****
  • Posts: 311
  • pesky kid
Re: mlv_dump on steroids
« Reply #7 on: February 17, 2017, 08:36:03 AM »
What do you mean by "possible to have both via swich"?
I mean with this:
https://bitbucket.org/hudson/magic-lantern/commits/ab24965b3f4d45e4cba10214bc0061f36b440730
and without
The stripes analyzed on zero frame or 1st frame of a range, they could be analyzed on every frame though. Can be implemented via switch.
It really matters for some shots.
As long as performance not so important when pre-converting, it would be very useful to have this switch.

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #8 on: February 17, 2017, 08:58:10 AM »
Ok understood. MLVFS doe not include that fix from a1ex. Also, I did not port the dualiso stuff from mlvfs anyway.

Will think about --stripes2 switch.

Danne

  • Hero Member
  • *****
  • Posts: 2719
Re: mlv_dump on steroids
« Reply #9 on: February 17, 2017, 09:06:32 AM »
Tested some clips with fpm files and it works great just like in mlvfs. Tried the load .bpm file but it doesn,t seem to come through. The bpm files loads but the pixels aren,t patched. In your older .map code which I,m using now the dots are gone but the same coordinates don,t apply here. Maybe the zeroes at the end are disturbing?

My test files here: https://drive.google.com/file/d/0B4tCJMlOYfira2hKTEhIdmwzZ3c/view?usp=sharing
Note that this file won,t work with mlvfs fpm files since it,s a 700D crop_rec test file(experimental build from dfort). It,s the same as for eosm so it works with map files and it also loads the .bpm file in your steroid version but focus pixels are still present.

Code: [Select]
MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/Daniel/Desktop/test/700D_crop_rec_test.MLV'
   - Convert to DNG frames
   - Output into '/Users/Daniel/Desktop/test/700D_crop_rec_test_'
File /Users/Daniel/Desktop/test/700D_crop_rec_test.MLV opened
File /Users/Daniel/Desktop/test/700D_crop_rec_test.M00 not existing.
Processing...

Using bad pixel map: '/Users/Daniel/Desktop/test/700D_crop_rec_test.bpm'
39060 pixels loaded

Vertical stripes correction: 'UNNEEDED'
  1.00000  1.00000  1.00014  1.00079  0.99968  0.99924  0.99899  1.00012

Extracting frames...
Current frame : '/Users/Daniel/Desktop/test/700D_crop_rec_test_000001.dng'
Reached end of chunk 1/1 after 9 blocks
Processed 2 video frames
Done

Levas

  • Hero Member
  • *****
  • Posts: 834
  • 6d - Nightly build user
Re: mlv_dump on steroids
« Reply #10 on: February 17, 2017, 09:31:05 AM »
@mothaibaphoto
Thanks for the tip for using Adobe DNG converter, will try it out.


bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #11 on: February 17, 2017, 09:41:27 AM »
@Danne: I'll take a look

mothaibaphoto

  • Senior
  • ****
  • Posts: 311
  • pesky kid
Re: mlv_dump on steroids
« Reply #12 on: February 17, 2017, 09:58:55 AM »
Sorry for offtopic, but:
@Levas:
Keep in mind that after DNG compressor your DNG are not CDNG anymore. This means you loose FPS tag at the very least and may have some problems with your post software.

Levas

  • Hero Member
  • *****
  • Posts: 834
  • 6d - Nightly build user
Re: mlv_dump on steroids
« Reply #13 on: February 17, 2017, 10:11:44 AM »
No problem, using RawTherapee for editing DNG's, export as TIFF sequence and use these in DaVinci Resolve 8)

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #14 on: February 19, 2017, 05:12:57 PM »
It really matters for some shots.
As long as performance not so important when pre-converting, it would be very useful to have this switch.

Had no time until today. Implemented --force-stripes switch. Please report the result.

Danne

  • Hero Member
  • *****
  • Posts: 2719

mothaibaphoto

  • Senior
  • ****
  • Posts: 311
  • pesky kid
Re: mlv_dump on steroids
« Reply #16 on: February 19, 2017, 06:08:46 PM »
Cool, but I have no footage to test on now :(

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #17 on: March 06, 2017, 07:58:46 PM »
Hi guys,

I updated downloads on the first post. Latest binaries (linux/mac/win) with lot of internal changes and bug fixes.
I hope someone's using it, really need some beta tester's feedback :P

bb

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2769
  • 5D3 / M1 / 7D / 70D / SL1
Re: mlv_dump on steroids
« Reply #18 on: March 06, 2017, 08:52:02 PM »
Thanks for this @bouncyball and just downloaded. Will test and report back my findings after trying it out with a DF Avg process test.  8)

*edit*

Actually probably due to my idiotic moment but I notice the mlv_dump (for Mac OS) file I downloaded isn't a binary -- did I miss a step in this?
5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101


bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #20 on: March 07, 2017, 07:19:32 AM »
@DeafEyeJedi: Thanks for your help :)

Actually probably due to my idiotic moment but I notice the mlv_dump (for Mac OS) file I downloaded isn't a binary -- did I miss a step in this?
Hmm... strange. The binary compiled/linked and running well under my virtual Sierra. I just stripped off the debug info by strip command to make it smaller. Did you rename it to mlv_dump and make it executable?

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #21 on: March 07, 2017, 07:46:25 AM »
@g3gg0: Do you mean I gotta make pull request? :D

but why develop a separate branch that grows too far from the main tree?
The point is that there were a lot of things to do (remove very mlvs specific code, add my functions and make some optimizations/bugfixes, still want to add some more stuff) and at that moment I hesitated to do PR. Right now I think it's possible.

I've started it as a fork of unified, then included goodies from 10/12 branch, then just merged it to that branch and it's there now. It even merges to the unified cleanly at least the way as 10/12bit branch does.

regards
bb

Danne

  • Hero Member
  • *****
  • Posts: 2719
Re: mlv_dump on steroids
« Reply #22 on: March 07, 2017, 07:54:07 AM »
@deafeyejedi
chmod u=rwx mlv_dump
Enter

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #23 on: March 07, 2017, 08:01:33 AM »
@Danne: Yup. Right :)

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2769
  • 5D3 / M1 / 7D / 70D / SL1
Re: mlv_dump on steroids
« Reply #24 on: March 07, 2017, 08:10:44 AM »
@deafeyejedi
chmod u=rwx mlv_dump
Enter

That did it. Thanks!
5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

k2121

  • New to the forum
  • *
  • Posts: 9
Re: mlv_dump on steroids
« Reply #25 on: March 07, 2017, 10:25:40 AM »
Hi,

In Windows 7 Service Pack 1 64-bit, program mlv_dump not working.

mlv_dump crash

Code: [Select]
Problem signature:
   Problem Event Name: APPCRASH
   Application Name: mlv_dump.exe
   Application Version: 0.0.0.0
   Timestamp applications: 589fb28f
   The name of the module with an error: ntdll.dll
   The module version with the error: 6.1.7601.23455
   The timestamp of the module with an error: 573a5463
   Exception code: c0000005
   Moving the exception: 00012c22
   OS Version: 6.1.7601.2.1.0.256.1
   Locale ID 1045
   Additional information 1: 0a9e
   Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
   Additional Information 3: 0a9e
   Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

Read our online privacy statement:
   http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0415

If the principle of confidentiality of information online is not available, please read our privacy statement offline:
   C: \ Windows \ system32 \ en-US \ erofflps.txt

Raw Video

Is there mlv_dump_steroids_1.0 version for Windows?
650d.104 ML • SX1.200h CHDK

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #26 on: March 07, 2017, 01:15:51 PM »
Yay! Feedback at last! :)

Well in reality this is the problem of your .IDX xref file. It's somehow wrong or corrupted (file size is wrong and block_size is wrong after XREF header). In all versions of mlv_dump it's gonna dump the core (will crash).

Just delete this .IDX file and if you really need it, build one with the command 'mlv_dump -x your.mlv' and you'll see the difference in sizes: Your's 5980 bytes vs Generated 4504 bytes.

Regards
bb

Danne

  • Hero Member
  • *****
  • Posts: 2719
Re: mlv_dump on steroids
« Reply #27 on: March 07, 2017, 03:33:43 PM »
I just finished a cr2hdr_steroids.app(MAC) build for testing which utilize the steroid version of mlv_dump. Download here
https://bitbucket.org/Dannephoto/cr2hdr/downloads/cr2hdr_steroids.dmg
Source code for this version is inside the dmg.

It works the same as cr2hdr.app you start out with a main menu and from there you head over to the mlv_dump settings menu by selecting (m).
https://bitbucket.org/Dannephoto/cr2hdr/overview

Note that you reach darkframe and flatframe automation from the main menu, not from mlv_dump menu.

Also updated to the latest focus pixel(dfort) script with the -n switch
https://bitbucket.org/daniel_fort/ml-focus-pixels

Main menu


mlv_dump menu


bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #28 on: March 07, 2017, 03:44:24 PM »
Hey man! Great news, thank you :)

Danne

  • Hero Member
  • *****
  • Posts: 2719
Re: mlv_dump on steroids
« Reply #29 on: March 07, 2017, 07:08:00 PM »
Uploaded a new cr2hdr_steroids version. Some small fixes around darkframe automation.
https://bitbucket.org/Dannephoto/cr2hdr/downloads/cr2hdr_steroids.dmg

All tests below with eosm in crop_rec

Tested darkframe and flatframe handling and confirmed working. Like with g3gg0/a1ex 10/12 bit mlv_dump darkframe subtraction works good with 14 and 12 bit files but with 10bit not so very nice. Instead of subtracting color noise it seems to add to it a little with 10bit. Good to know if recording 10bit footage.
A not so very nice problem is later chroma smooth(dualiso code) vs raw2dng chroma smooth code. As cs 2x2 is great with raw2dng code and for cameras in need of it like eosm the cs code in dualiso code can,t really cope with focus pixels. Have a look.
Also about focus pixels it seems not all of them are taken care of. Check to the right on the picture. Maybe the -n switch needs some checking into @dfort?

Chroma smooth artifacts


Focus pixels



Darkframe testing. Last two picture flatframe averaging

14bit original


14bit Darkframed


12bit original


12bit Darkframed


10bit original


10bit Darkframed


Original (not flatframed)


Flatframed


DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2769
  • 5D3 / M1 / 7D / 70D / SL1
mlv_dump on steroids
« Reply #30 on: March 08, 2017, 07:48:59 AM »
Loving the update so far @Danne and I really dig the new mlv_dump menu within your app. It's so funny that 10-bit looks worse after DF avg process. Seems 12-bit may be the 'sweet spot' after all. Thanks for sharing and will test thoroughly before I report anything.
5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #31 on: March 08, 2017, 08:56:28 AM »
Yes. CS issue exacly as in mlvfs. I guess the solution is: http://www.magiclantern.fm/forum/index.php?topic=7122.msg179319#msg179319.

Danne

  • Hero Member
  • *****
  • Posts: 2719
Re: mlv_dump on steroids
« Reply #32 on: March 08, 2017, 01:42:03 PM »
Think I got something working with the -n swith in the focus pixel list.

This:
Code: [Select]
##
# output_row() function
# Write a row of focus pixel coordinates to the output file.
#
output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
      echo -e "$x \t $y" >> "$output"
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
      if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
        if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
          if [[ "$format" == fpm ]]; then
            echo -e "$x \t $y" >> "$output"
          else echo -e "$x \t $y \t 0" >> "$output"
          fi
        fi
      fi
    fi
  fi
  done
}


instead of:
Code: [Select]
##
# output_row() function
# Write a row of focus pixel coordinates to the output file.
#
output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
      fi
      if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
        if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
          if [[ "$format" == fpm ]]; then
            echo -e "$x \t $y" >> "$output"
          else echo -e "$x \t $y \t 0" >> "$output"
          fi
        fi
      fi
    fi
  done
}

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #33 on: March 08, 2017, 05:38:36 PM »
Nicely done! :)

output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
+       if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
+         if [[ "$format" == fpm ]]; then
+            echo -e "$x \t $y" >> "$output"
+         else.
+            echo -e "$x \t $y \t 0" >> "$output"
+          fi
+        fi
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
        if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
          if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
            if [[ "$format" == fpm ]]; then
              echo -e "$x \t $y" >> "$output"
            else.
              echo -e "$x \t $y \t 0" >> "$output"
            fi
          fi
        fi
      fi
    fi
  done
}

Just a small correction. Now this does not break dcraw style output (zero at the end of line) and makes sure there're no negative coordinates (x,y >= 0).

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2769
  • 5D3 / M1 / 7D / 70D / SL1
Re: mlv_dump on steroids
« Reply #34 on: March 08, 2017, 06:16:32 PM »
Loving the collaborations from you two!
5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 2719
Re: mlv_dump on steroids
« Reply #35 on: March 08, 2017, 06:30:04 PM »
That extra dcraw else function isn,t really working. I think the
Code: [Select]
if [[ "$format" == fpm ]] Needs an actual fpm file to be fed to the script else it won,t recognize outputting fpm and it will take it for a dcraw output anyway. At least that,s what my tests shown.
I think like this. The -n switch will probably only be used for the mlvfs focus pixel code in your steroid version and otherwise we run the script without the -n switch and then we,re back to the extra 0.

*Unless I specify the -f for fpm file, huh, just noticed that little detail. Will test again.
Ok, your little extras are working since I specified the -f switch as well. Then I tested the old focus pixel code with the -f switch but that din,t work, go figure. I will add your extra lines to dforts script. Great.

Updated cr2hdr_steroids.app with the latest fpm.sh script.

bouncyball

  • Member
  • ***
  • Posts: 147
Re: mlv_dump on steroids
« Reply #36 on: March 09, 2017, 09:36:37 AM »
Yup that -f is there from the beginning :)

-f -n does what I really was after when adding -n.

bb