Magic Lantern Forum

Developing Magic Lantern => General Development => Topic started by: bouncyball on February 16, 2017, 07:10:10 PM

Title: mlv_dump on steroids
Post by: bouncyball on February 16, 2017, 07:10:10 PM
Hi there,

First of all I want to admit: this is not the first time when someone modified mlv_dump. I also did it once not so long while ago but this time it's little bit different. This one based on latest vanilla MLVFS source files edited and changed by me.

Also, I can say that digging into MLVFS sources is a real joy :D. I love it starting from the idea and ending with the very cool multiplatform implementation. I want to thank David Milligan for this gem, special thanks to Danne (he always supports my development attempts), also a1ex and g3gg0 for all this thing called ML project moving forward and all active maintainers and devs of other ports and all the community around the project. Thank you guys!

Now about the subject:

UPDATE:
UPDATE2:
UPDATE3 (2018):
UPDATE4 (last!): It is merged to official ML's crop_rec_4k branch: Link (https://bitbucket.org/hudson/magic-lantern/src/0075d0fa70f229ec972d7831aa756c4a9280e625/modules/mlv_rec/?at=crop_rec_4k)

( hopefully ;) ) it:
1. Saves 16bit any bit depth Cinema DNG files. e.g. 10/12/14 bits compressed or uncompressed, "--no-bitpack" exports 16 bit uncompressed DNGs.
2. Supports any bit depth raw data as input compressed or uncompressed
3. Has additional or reworked switch options:
Code: [Select]
---
  --no-audio          for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination MLV
---
  --no-fixfp          do not fix focus pixels
  --fixcp2            use aggressive method for revealing more bad pixels
  --no-stripes        do not fix vertical stripes in highlights
  --force-stripes     compute stripe correction for every frame
  --is-dualiso        use dual iso compatible horizontal interpolation of focus and bad pixels
  --is-croprec        generate focus map for crop_rec mode
  --save-bpm          save bad pixels to .BPM file
  --fixpn             fix pattern noise
  --deflicker=value   per-frame exposure compensation. value is target median in raw units ex: 3072 (default)
  --no-bitpack        write DNG files with unpacked to 16 bit raw data
  --show-progress     show DNG file creation progress. ignored when -v or --batch is specified
                      also works when compressing MLV to MLV and shows compression ratio for each frame
  --fpi <method>      focus pixel interpolation method: 0 (mlvfs), 1 (raw2dng), default is 1
  --bpi <method>      bad pixel interpolation method: 0 (mlvfs), 1 (raw2dng), default is 0

---
  -c                  compress video frames using LJ92. if input is lossless, then decompress and recompress again.
  -d                  decompress compressed video and audio frames using LZMA or LJ92
  -p                  pass through original raw data without processing, it works for lossless or uncompressed raw
"--force-stripes" forces stripe detection for each frame separately. Slow but more effective on some footage
"-p" option replaces "-c -c" and passes through unmodified data for both uncompressed or lossless compressed raw.
"--no-audio" for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination MLV, it's forced when "-f" used
"--no-fixfp" option deactivates fixing of focus pixels"
"--is-croprec" option  generates focus map for crop_rec mode"

4. Supports focus pixels the same way MLVFS does, also has ability to override and load '.fpm' manually if file has the same name as input MLV. Automatically fixes focus pixels for losless or uncompressed MLVs without special maps. If could not fix some dots for some special case map file can be specified as usual to override auto fixing. Auto detects crop_rec mode
5. Saves/Loads '.bpm' (bad pixel map) files. Name of the file same as input MLV
6. Can fix pattern noise
7. Has deflicker option of MLVFS
8. --show-progress prints every operation's done while saving DNGs, also works when compressing MLV to MLV and shows compression ratio for each frame
9. Implemented advanced WAV header with iXML for demanding NLEs
10. Includes 'crop_rec_4k' branch goodies/changes
11. Correctly fills up 'Unique Camera Model' tag in dng header and has additional camera matrixes
12. With "-b" switch you can convert bit depth and related black/white levels and export to that bit depth DNG (uncompressed only)
13. Chroma smooth code updated to the more advanced one
14. Added pixel interpolation method from raw2dng
15. Introduced two command line switches: "--fpi" and "--bpi" (see usage), by default, for focus pixels RAW2DNG method (correct edge pixel processing added) and for bad pixels MLVFS method is used
16. Supports new focus pixel maps w/header, generated by fpmutil (https://bitbucket.org/bouncyball/mlv-tools/overview) (in addition to original simple .fpm), 'fpmutil' binaries can be downloaded here (https://bitbucket.org/bouncyball/mlv-tools/downloads/)
17. Well... maybe I forgot something to mention

'fpmutil' also updated and can be downloaded from the links above as usual

Any feedback is welcome!

Here are links to the fork and downloads (https://bitbucket.org/bouncyball/magic-lantern/downloads/).
Here is a branch with simplified changeset to understand, with less effort, how 'mlv_dump.c' was modified and what has changed.

Now it's merged to Magic Lantern repository: Link (https://bitbucket.org/hudson/magic-lantern/src/0075d0fa70f229ec972d7831aa756c4a9280e625/modules/mlv_rec/?at=crop_rec_4k)

The best regards
bb
Title: Re: mlv_dump on steroids
Post by: Levas on February 16, 2017, 07:41:44 PM
Only option I'm missing, since 10/12 bit raw recording  ;D, is the option for different bit depths when creating dng's.

I know it's crazy, but I keep the dng's as backup and throw away the MLV files :P
So an option for 10, 12 or 14 bit dng output, according to my recording bit depth in the original MLV, would be killer future 8).
Title: Re: mlv_dump on steroids
Post by: Danne on February 16, 2017, 08:14:34 PM
On steroids I say. It,s beautiful rework and addons. Don,t forget to mention white balance support as well as darkframe and flatframe inclusion. I,ll check deeper into this once I have some time. Here is a binary for mac meanwhile.
Great work.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_macOS
Title: Re: mlv_dump on steroids
Post by: bouncyball on February 17, 2017, 06:13:07 AM
@Levas:
Yeah I understand, but the whole purpose of using mlvfs universal implementation is to convert any bit depth sources to 16 bit and then do raw processing on 16 bit buffers (downconverting again is a bad idea). Maybe some day experimental hardware raw compression from silent pictures goes to the mlv_rec/mlv_lite who knows.

@Danne:
Hey! Thanx for the binary. Uploaded.
Title: Re: mlv_dump on steroids
Post by: mothaibaphoto on February 17, 2017, 06:38:56 AM
Hello, bouncyball. It's a incredibly great.
Why you don't a "Hero member" yet? Yes, that message counters just nonsense...
Ok, on topic: Ironically, I just finished to transcode a couple of TB with old gold mlv_dump.
Actually, due to this issue
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173295#msg173295
i used 2 versions of mlv_dump: one for the dual iso shots, and another(which corrupts dual iso) for the rest -
because that one handles vertical stripes better, to my eye.
As far as i know, that modification finally was removed:
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173394#msg173394
So, the first question - what version of vertical stripes fix code do you use?
Is it possible to have both via swich or something like that?
And, second: there was a discussion in MLVFS thread, that MLVFS does not analyse every frame for vertical stripes for performance reason,
and i had some persisted stripes on panned shots with MLVFS.
What is your approach here?
Best regards, and really great thanks for your contribution.
Title: Re: mlv_dump on steroids
Post by: mothaibaphoto on February 17, 2017, 06:44:24 AM
@Levas: I keep DNG too. I compress it with Adobe DNG converter.
Virtually there is no difference in size between 14 and 16 bit versions after compression.
And, i guess, 10 and 12 bit will have no benefit in size after compression.
Title: Re: mlv_dump on steroids
Post by: bouncyball on February 17, 2017, 07:57:52 AM
So, the first question - what version of vertical stripes fix code do you use?
Is it possible to have both via swich or something like that?
And, second: there was a discussion in MLVFS thread, that MLVFS does not analyse every frame for vertical stripes for performance reason,
Well, It's exact MLVFS stripe removal code not from raw2dng however I guess technically they are quite similar.

What do you mean by "possible to have both via swich"?

The stripes analyzed on zero frame or 1st frame of a range, they could be analyzed on every frame though. Can be implemented via switch.
Title: Re: mlv_dump on steroids
Post by: mothaibaphoto on February 17, 2017, 08:36:03 AM
What do you mean by "possible to have both via swich"?
I mean with this:
https://bitbucket.org/hudson/magic-lantern/commits/ab24965b3f4d45e4cba10214bc0061f36b440730
and without
The stripes analyzed on zero frame or 1st frame of a range, they could be analyzed on every frame though. Can be implemented via switch.
It really matters for some shots.
As long as performance not so important when pre-converting, it would be very useful to have this switch.
Title: Re: mlv_dump on steroids
Post by: bouncyball on February 17, 2017, 08:58:10 AM
Ok understood. MLVFS doe not include that fix from a1ex. Also, I did not port the dualiso stuff from mlvfs anyway.

Will think about --stripes2 switch.
Title: Re: mlv_dump on steroids
Post by: Danne on February 17, 2017, 09:06:32 AM
Tested some clips with fpm files and it works great just like in mlvfs. Tried the load .bpm file but it doesn,t seem to come through. The bpm files loads but the pixels aren,t patched. In your older .map code which I,m using now the dots are gone but the same coordinates don,t apply here. Maybe the zeroes at the end are disturbing?

My test files here: https://drive.google.com/file/d/0B4tCJMlOYfira2hKTEhIdmwzZ3c/view?usp=sharing
Note that this file won,t work with mlvfs fpm files since it,s a 700D crop_rec test file(experimental build from dfort). It,s the same as for eosm so it works with map files and it also loads the .bpm file in your steroid version but focus pixels are still present.

Code: [Select]
MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/Daniel/Desktop/test/700D_crop_rec_test.MLV'
   - Convert to DNG frames
   - Output into '/Users/Daniel/Desktop/test/700D_crop_rec_test_'
File /Users/Daniel/Desktop/test/700D_crop_rec_test.MLV opened
File /Users/Daniel/Desktop/test/700D_crop_rec_test.M00 not existing.
Processing...

Using bad pixel map: '/Users/Daniel/Desktop/test/700D_crop_rec_test.bpm'
39060 pixels loaded

Vertical stripes correction: 'UNNEEDED'
  1.00000  1.00000  1.00014  1.00079  0.99968  0.99924  0.99899  1.00012

Extracting frames...
Current frame : '/Users/Daniel/Desktop/test/700D_crop_rec_test_000001.dng'
Reached end of chunk 1/1 after 9 blocks
Processed 2 video frames
Done
Title: Re: mlv_dump on steroids
Post by: Levas on February 17, 2017, 09:31:05 AM
@mothaibaphoto
Thanks for the tip for using Adobe DNG converter, will try it out.

Title: Re: mlv_dump on steroids
Post by: bouncyball on February 17, 2017, 09:41:27 AM
@Danne: I'll take a look
Title: Re: mlv_dump on steroids
Post by: mothaibaphoto on February 17, 2017, 09:58:55 AM
Sorry for offtopic, but:
@Levas:
Keep in mind that after DNG compressor your DNG are not CDNG anymore. This means you loose FPS tag at the very least and may have some problems with your post software.
Title: Re: mlv_dump on steroids
Post by: Levas on February 17, 2017, 10:11:44 AM
No problem, using RawTherapee for editing DNG's, export as TIFF sequence and use these in DaVinci Resolve 8)
Title: Re: mlv_dump on steroids
Post by: bouncyball on February 19, 2017, 05:12:57 PM
It really matters for some shots.
As long as performance not so important when pre-converting, it would be very useful to have this switch.

Had no time until today. Implemented --force-stripes switch. Please report the result.
Title: Re: mlv_dump on steroids
Post by: Danne on February 19, 2017, 05:50:17 PM
For mac
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_steroids_1.0
Title: Re: mlv_dump on steroids
Post by: mothaibaphoto on February 19, 2017, 06:08:46 PM
Cool, but I have no footage to test on now :(
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 06, 2017, 07:58:46 PM
Hi guys,

I updated downloads on the first post. Latest binaries (linux/mac/win) with lot of internal changes and bug fixes.
I hope someone's using it, really need some beta tester's feedback :P

bb
Title: Re: mlv_dump on steroids
Post by: DeafEyeJedi on March 06, 2017, 08:52:02 PM
Thanks for this @bouncyball and just downloaded. Will test and report back my findings after trying it out with a DF Avg process test.  8)

*edit*

Actually probably due to my idiotic moment but I notice the mlv_dump (for Mac OS) file I downloaded isn't a binary -- did I miss a step in this?
Title: Re: mlv_dump on steroids
Post by: g3gg0 on March 06, 2017, 11:13:02 PM
hmh, maybe i missed the point.
but why develop a separate branch that grows too far from the main tree?
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 07, 2017, 07:19:32 AM
@DeafEyeJedi: Thanks for your help :)

Actually probably due to my idiotic moment but I notice the mlv_dump (for Mac OS) file I downloaded isn't a binary -- did I miss a step in this?
Hmm... strange. The binary compiled/linked and running well under my virtual Sierra. I just stripped off the debug info by strip command to make it smaller. Did you rename it to mlv_dump and make it executable?
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 07, 2017, 07:46:25 AM
@g3gg0: Do you mean I gotta make pull request? :D

but why develop a separate branch that grows too far from the main tree?
The point is that there were a lot of things to do (remove very mlvs specific code, add my functions and make some optimizations/bugfixes, still want to add some more stuff) and at that moment I hesitated to do PR. Right now I think it's possible.

I've started it as a fork of unified, then included goodies from 10/12 branch, then just merged it to that branch and it's there now. It even merges to the unified cleanly at least the way as 10/12bit branch does.

regards
bb
Title: Re: mlv_dump on steroids
Post by: Danne on March 07, 2017, 07:54:07 AM
@deafeyejedi
chmod u=rwx mlv_dump
Enter
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 07, 2017, 08:01:33 AM
@Danne: Yup. Right :)
Title: Re: mlv_dump on steroids
Post by: DeafEyeJedi on March 07, 2017, 08:10:44 AM
@deafeyejedi
chmod u=rwx mlv_dump
Enter

That did it. Thanks!
Title: Re: mlv_dump on steroids
Post by: k2121 on March 07, 2017, 10:25:40 AM
Hi,

In Windows 7 Service Pack 1 64-bit, program mlv_dump not working.

mlv_dump crash (https://drive.google.com/open?id=0B5jIPDo4rZ60OTB3VFFhTHBpTDA)

Code: [Select]
Problem signature:
   Problem Event Name: APPCRASH
   Application Name: mlv_dump.exe
   Application Version: 0.0.0.0
   Timestamp applications: 589fb28f
   The name of the module with an error: ntdll.dll
   The module version with the error: 6.1.7601.23455
   The timestamp of the module with an error: 573a5463
   Exception code: c0000005
   Moving the exception: 00012c22
   OS Version: 6.1.7601.2.1.0.256.1
   Locale ID 1045
   Additional information 1: 0a9e
   Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
   Additional Information 3: 0a9e
   Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

Read our online privacy statement:
   http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0415

If the principle of confidentiality of information online is not available, please read our privacy statement offline:
   C: \ Windows \ system32 \ en-US \ erofflps.txt

Raw Video  (https://drive.google.com/drive/folders/0B5jIPDo4rZ60ZHpGNEd6bVN0TWM?usp=sharing)

Is there mlv_dump_steroids_1.0 version for Windows?
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 07, 2017, 01:15:51 PM
Yay! Feedback at last! :)

Well in reality this is the problem of your .IDX xref file. It's somehow wrong or corrupted (file size is wrong and block_size is wrong after XREF header). In all versions of mlv_dump it's gonna dump the core (will crash).

Just delete this .IDX file and if you really need it, build one with the command 'mlv_dump -x your.mlv' and you'll see the difference in sizes: Your's 5980 bytes vs Generated 4504 bytes.

Regards
bb
Title: Re: mlv_dump on steroids
Post by: Danne on March 07, 2017, 03:33:43 PM
I just finished a cr2hdr_steroids.app(MAC) build for testing which utilize the steroid version of mlv_dump. Download here
https://bitbucket.org/Dannephoto/cr2hdr/downloads/cr2hdr_steroids.dmg
Source code for this version is inside the dmg.

It works the same as cr2hdr.app you start out with a main menu and from there you head over to the mlv_dump settings menu by selecting (m).
https://bitbucket.org/Dannephoto/cr2hdr/overview

Note that you reach darkframe and flatframe automation from the main menu, not from mlv_dump menu.

Also updated to the latest focus pixel(dfort) script with the -n switch
https://bitbucket.org/daniel_fort/ml-focus-pixels

Main menu
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs29.postimg.org%2Fegr7nupmv%2FScreen_Shot_2017_03_07_at_15_20_52.png&hash=5f65dfd662dfb3414bb2d574a300ee89)

mlv_dump menu
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs29.postimg.org%2Fpf2h61e87%2FScreen_Shot_2017_03_07_at_15_20_29.png&hash=e7f981ee48a3cb38be5afc29217da140)
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 07, 2017, 03:44:24 PM
Hey man! Great news, thank you :)
Title: Re: mlv_dump on steroids
Post by: Danne on March 07, 2017, 07:08:00 PM
Uploaded a new cr2hdr_steroids version. Some small fixes around darkframe automation.
https://bitbucket.org/Dannephoto/cr2hdr/downloads/cr2hdr_steroids.dmg

All tests below with eosm in crop_rec

Tested darkframe and flatframe handling and confirmed working. Like with g3gg0/a1ex 10/12 bit mlv_dump darkframe subtraction works good with 14 and 12 bit files but with 10bit not so very nice. Instead of subtracting color noise it seems to add to it a little with 10bit. Good to know if recording 10bit footage.
A not so very nice problem is later chroma smooth(dualiso code) vs raw2dng chroma smooth code. As cs 2x2 is great with raw2dng code and for cameras in need of it like eosm the cs code in dualiso code can,t really cope with focus pixels. Have a look.
Also about focus pixels it seems not all of them are taken care of. Check to the right on the picture. Maybe the -n switch needs some checking into @dfort?

Chroma smooth artifacts
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs10.postimg.org%2Ft20rp2yw9%2FScreen_Shot_2017_03_07_at_19_10_48.png&hash=5cdee9a6f79d86d28e5fd483c0321b5a)

Focus pixels
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2F8wjxkshb7%2FScreen_Shot_2017_03_07_at_19_14_50.png&hash=bde663e358192438a0b6a493282a17fa)


Darkframe testing. Last two picture flatframe averaging

14bit original
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2F6sncwc403%2F01_14bit.jpg&hash=579ec114b3845064179212a1de81a586)

14bit Darkframed
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2Fpdbl3epfn%2F02_14bit.jpg&hash=949f6afa0ece44936f0b99a823e26974)

12bit original
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2Fqyquv7znn%2F01_12bit.jpg&hash=ef27a6e450861e2f3c1fcc5f8654085d)

12bit Darkframed
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2Fz9co31v7n%2F02_12bit.jpg&hash=35f53344ca9edb57bc06b893614b17d4)

10bit original
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2Fp5jct5p37%2F01_10bit.jpg&hash=355a65fdaadefe91005f2dd73b6ac758)

10bit Darkframed
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2F9padwm9tv%2F02_10bit.jpg&hash=15626a3bdf94d6fee6523676b45f24bf)

Original (not flatframed)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2Fij1adpwsj%2F01_flat.jpg&hash=4c97b89eccfd5b29d24cadcc097558ec)

Flatframed
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs3.postimg.org%2Fdphj8v0ar%2F02_flat.jpg&hash=fdc53065a45fe4bdc809d264703cc3bf)
Title: mlv_dump on steroids
Post by: DeafEyeJedi on March 08, 2017, 07:48:59 AM
Loving the update so far @Danne and I really dig the new mlv_dump menu within your app. It's so funny that 10-bit looks worse after DF avg process. Seems 12-bit may be the 'sweet spot' after all. Thanks for sharing and will test thoroughly before I report anything.
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 08, 2017, 08:56:28 AM
Yes. CS issue exacly as in mlvfs. I guess the solution is: http://www.magiclantern.fm/forum/index.php?topic=7122.msg179319#msg179319 (http://www.magiclantern.fm/forum/index.php?topic=7122.msg179319#msg179319).
Title: Re: mlv_dump on steroids
Post by: Danne on March 08, 2017, 01:42:03 PM
Think I got something working with the -n swith in the focus pixel list.

This:
Code: [Select]
##
# output_row() function
# Write a row of focus pixel coordinates to the output file.
#
output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
      echo -e "$x \t $y" >> "$output"
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
      if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
        if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
          if [[ "$format" == fpm ]]; then
            echo -e "$x \t $y" >> "$output"
          else echo -e "$x \t $y \t 0" >> "$output"
          fi
        fi
      fi
    fi
  fi
  done
}


instead of:
Code: [Select]
##
# output_row() function
# Write a row of focus pixel coordinates to the output file.
#
output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
      fi
      if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
        if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
          if [[ "$format" == fpm ]]; then
            echo -e "$x \t $y" >> "$output"
          else echo -e "$x \t $y \t 0" >> "$output"
          fi
        fi
      fi
    fi
  done
}
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 08, 2017, 05:38:36 PM
Nicely done! :)

output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
+       if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
+         if [[ "$format" == fpm ]]; then
+            echo -e "$x \t $y" >> "$output"
+         else.
+            echo -e "$x \t $y \t 0" >> "$output"
+          fi
+        fi
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
        if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
          if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
            if [[ "$format" == fpm ]]; then
              echo -e "$x \t $y" >> "$output"
            else.
              echo -e "$x \t $y \t 0" >> "$output"
            fi
          fi
        fi
      fi
    fi
  done
}

Just a small correction. Now this does not break dcraw style output (zero at the end of line) and makes sure there're no negative coordinates (x,y >= 0).
Title: Re: mlv_dump on steroids
Post by: DeafEyeJedi on March 08, 2017, 06:16:32 PM
Loving the collaborations from you two! [emoji56]
Title: Re: mlv_dump on steroids
Post by: Danne on March 08, 2017, 06:30:04 PM
That extra dcraw else function isn,t really working. I think the
Code: [Select]
if [[ "$format" == fpm ]] Needs an actual fpm file to be fed to the script else it won,t recognize outputting fpm and it will take it for a dcraw output anyway. At least that,s what my tests shown.
I think like this. The -n switch will probably only be used for the mlvfs focus pixel code in your steroid version and otherwise we run the script without the -n switch and then we,re back to the extra 0.

*Unless I specify the -f for fpm file, huh, just noticed that little detail. Will test again.
Ok, your little extras are working since I specified the -f switch as well. Then I tested the old focus pixel code with the -f switch but that din,t work, go figure. I will add your extra lines to dforts script. Great.

Updated cr2hdr_steroids.app with the latest fpm.sh script.
Title: Re: mlv_dump on steroids
Post by: bouncyball on March 09, 2017, 09:36:37 AM
Yup that -f is there from the beginning :)

-f -n does what I really was after when adding -n.

bb
Title: Re: mlv_dump on steroids
Post by: g3gg0 on June 14, 2017, 03:21:03 PM
can we partially merge the goodies from that branch into the main branch?

i am afraid that this will end in both branches to develop in their own context and finally get un-merge-able.
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 14, 2017, 09:06:23 PM
Hi g3gg0 thanx for asking.

can we partially merge the goodies from that branch into the main branch?
Sure we can talk about it more deeply. I always wanted to do this but my version is quite different (different from dmilligans version as well) and I kinda thought to ask you and a1ex to just add it as a separate branch (I agree: not the best idea). Actually, to the "mlv_dump.c" itself, I always tried to make changes as subtle as possible. I think I succeeded more or less and all big changes are only regarding DNG handling and WAV advanced header. All processing of the RAW data before saving DNG is taking place outside of the mlv_dump.c.

i am afraid that this will end in both branches to develop in their own context and finally get un-merge-able.
I did some important commits last two days (now it supports any bit depth compressed/uncompressed DNG output) and did not test merge-ability yet. However, usually I'm very carefull about it and always merge any new changes you make.

BR,
BB
Title: Re: mlv_dump on steroids
Post by: Danne on June 17, 2017, 05:31:58 PM
Thanks for latest changes. Being able to output native bit 10/12/16 with mlv_dump stripes, chroma smooth etc added is magic. Also working with compression with all bits after final strip byte fix to match dng checksum.
Also looking forward to some sort of merge story involving mlv_dump unified as g3gg0 points out. How that would be done idk but where's there competence there's possibilities. Always been missing white balance and some of the matrices.
Insights around the strip byte tag would perhaps make it possible to calculate the correct strip byte tag for when exporting compressed 10bit/12bit dng files with crop_rec_4k build now?
Great work Bouncyball.
Title: Re: mlv_dump on steroids
Post by: g3gg0 on June 17, 2017, 11:40:31 PM
well, its already such a huuuuuuge changeset that it is almost impossible to review and understand.
if it was asked to merge, i would have no clue what (side) effects merging would bring with.

also not sure if i want to have e.g. deflicker in mlv_dump.
(don't get me wrong, it's for sure a good piece of software, but mlv_dump wasn't meant as post processing tool.
it fixes all the very camera specific issues and is made for a solid mlv-to-other-tools converter)

just having a PR for this branch is very unlikely to get merged.
you would have to split the changes up.
Title: Re: mlv_dump on steroids
Post by: Danne on June 17, 2017, 11:54:39 PM
Indeed. more mlvfs like approach. And hey, I only scratch my head in amazement when I look into code in mlv_dump. Good inspiration.
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 20, 2017, 07:27:42 AM
Hi g3gg0,

I really understand your point. Huge changeset is not gonna get merged in any case and that is why I hesitated to make PR. Even 1st changes were big enough (but review-able) because of "chdk_dng.c" been replaced by "dng.c" to support CDNG and more...

I think you are familiar with dmilligan's dng lib more or less (actually I modified it but the idea is the same) and if you accept the fact that in order to merge goodies from this version chdk_dng.c should be replaced, then I will try my best to split changes and start merging them to crop_rec_4k branch.

also not sure if i want to have e.g. deflicker in mlv_dump.
Well :) it's the smallest and least important part of changes and takes place outside of mlv_dump.c. Can be removed anytime.

regards
bb
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 20, 2017, 01:25:31 PM
Hey guys!

Finally I updated the binaries to the latest version. Lots of changes are made from last time. Look at the 1st post of this thread. Download links are the same.

And... any feedback is welcome as usual!

br,
bb
Title: Re: mlv_dump on steroids
Post by: Kharak on June 20, 2017, 07:59:17 PM
Hey Bouncyball

I am interested in trying out --force-stripes. I am having issues with Vertical Stripes in a lot of my footage.

Could you post an example how I would write the command ? I am not sure where to put in --force-stripes.
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 21, 2017, 08:56:40 AM
@Kharak

Command like this:
Code: [Select]
mlv_dump --dng --force-stripes --show-progress INPUT.MLV
should do the trick.

Also have a look at flat field method as g3gg0 mentioned here (http://www.magiclantern.fm/forum/index.php?topic=19300.msg186057#msg186057)
Title: Re: mlv_dump on steroids
Post by: Kharak on June 25, 2017, 03:07:55 AM
@ BB

Thank you, sorry for the late reply. I have been so busy.

The --force-stripes does a very good job. But it is very slow as you said.

I just tried converting a compressed mlv without --force-stripes.

Code: [Select]
mlv_dump --dng --show-progress INPUT.MLV

And it said "Vertical stripes correction: 'NEEDED'" and then proceeded to show the numbers under it. Does this mean it did perform a "normal" Vertical Stripe Fix or that I need to do it. I tried without --force-stripes hoping I could achieve same results as Danne with his CR2HDR app; he got straight out of the box with an older Vertical Stripe fix code, identical results to the --force-stripes. I don't know if CR2HDR app with that (old) Vertical Stripe Code is a slow process. But I am wondering why this old stripe fix is working so well and I have to use (a seemingly new code?) --force-stripes to remove mine? Is it a Mac and Win thing? :)

I also tried the same MLV:

Code: [Select]
mlv_dump_s --dng --fixpn --show-progress input.MLV
It said "Vertical Stripe Correction: Unneded". Yet extremely slow, maybe slower than --force-stripes (did not take time). But half way through the conversion I could see I was just adding pattern noise, I was hoping maybe this --fixpn could attack the vertical stripes, i mean in reality Vertical Stripes are Pattern Noise, no?


And last but not least, is there any way to batch process with you mlv_dump or for some form of automation?

Thank you.
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 25, 2017, 08:35:32 AM
@Kharak

And it said "Vertical stripes correction: 'NEEDED'" and then proceeded to show the numbers under it. Does this mean it did perform a "normal" Vertical Stripe Fix or that I need to do it.
It's just an information, that after evaluating of the 1st frame there is a real need to correct stripes. No intervention required from user. Normal vertical stripes correction is done according to 1st frame analysis.

he got straight out of the box with an older Vertical Stripe fix code, identical results to the --force-stripes. But I am wondering why this old stripe fix is working so well and I have to use (a seemingly new code?) --force-stripes to remove mine? Is it a Mac and Win thing? :)
Good question :) (no mac or windows thing). This mlv_dump uses all the same raw processing algorithms what used in MLVFS but it does not mean these algos are better than some other ones used in other code. Not only stripe fix but also chroma smooth and bad/focus pixel interpolation is different and yields different results. Investigation of this subject is on my schedule, just had low priority until recently. Thanks for sharing experience with both stripes code. Are you certain when you say that older normal stripes removal gives same results as --force-stripes? Can you show some images illustrating this?

I was hoping maybe this --fixpn could attack the vertical stripes, i mean in reality Vertical Stripes are Pattern Noise, no?
Pattern noise removal method is not the same as stripes removal even if it helps to make stripes less visible.

And last but not least, is there any way to batch process with you mlv_dump or for some form of automation?
It's a command line utility. Of course it's intended to use for batch processing in shell out of the box. Under windows, unfortunatelly, there is no any user friendly, ready to use, automator like Danne's cr2hdr.app.

bb
Title: Re: mlv_dump on steroids
Post by: Danne on June 25, 2017, 10:54:49 AM
I fired up windows under virtual box so been fiddling some windows commands today. Who knows where that ends  :P
http://www.magiclantern.fm/forum/index.php?topic=10526.msg186219#msg186219
By the way. How does --force-stripes work? Is it calculating every dng file individually or only the first?
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 25, 2017, 01:05:53 PM
Hey Danne!

Hahaha welcome to winworld :). Well... there is a rudimentary support of bash in Win10 :P, very, VERY limited to use with full blown scripts. If you someday, somehow decide to port your great work you're gonna need to use powershell (can be even made GUI app with it).

Regarding "--force-stripes": it has force prefix because it means analyze every single frame and correct it. That's why it takes ages :)

bb
Title: Re: mlv_dump on steroids
Post by: Danne on June 25, 2017, 03:34:32 PM
Gleaned on bash in windows 10. Will check out power shell definetaly :).
Seems there is need for a second vert stripe option, "always on" to be able to overide automation code and at the same time avoid forcing it on every dng.
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 25, 2017, 05:46:23 PM
Well... it "always on" when no stripe related switch is specified:

Code: [Select]
no switch        - analyze 1st frame, process all frames according to gathered data (code decides and prints out whether correction 'NEEDED' or 'UNNEEDED')
--no-stripes     - do not correct stripes at all
--force-stripes  - analyze every frame, process current frame according to gathered data (code decides and prints out whether correction 'NEEDED' or 'UNNEEDED')

Did you mean something else? :)
Title: Re: mlv_dump on steroids
Post by: Danne on June 25, 2017, 05:57:31 PM
Always on, sure about that? Kharak is reporting bad results when force stripes isn´t selected? I think he needs to put up some demo pics here...
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 25, 2017, 05:59:29 PM
Yes normal correction is on by default. Look at output with --show-progress. There was a serious bug in there, stripe correction was done only to 1st frame. Sorry about that... will upload new fixed binaries soon. Thanks to Danne for the hint :)

Sure :) I already asked him about it.
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 25, 2017, 07:09:15 PM
Fixed binaries are uploaded. Link is the same @ 1st post.
Title: Re: mlv_dump on steroids
Post by: Kharak on June 26, 2017, 06:14:38 AM
You still need examples?

I jumped to the new script and mlv_dump Danne posted here:
http://www.magiclantern.fm/forum/index.php?topic=10526.new#new
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 26, 2017, 09:49:27 AM
No thank you :)
Title: Re: mlv_dump on steroids
Post by: Danne on June 26, 2017, 09:57:51 AM
Well. What would be good is to compare latest vertical stripes code in unified and crop_rec_4k versions of mlv_dump to a1ex older vertical stripes code. If there are a lot of footage that doesn´t get the stripe treatment we need to know. Could you check and compare Kharak and see what you come up with? You can simply do this test by exchanging mlv_dump in batch_mlv.
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 26, 2017, 10:17:32 AM
Well good idea. I don's have havily striped footage at the moment to test with.

@Kharak: If you do the test anyway test this mlv_dump as well. Also cut original mlv file(s) to have 1st 5-10 frames and put up them somewhere please.
Title: Re: mlv_dump on steroids
Post by: Kharak on June 26, 2017, 04:47:47 PM
Sure can do, but how do I cut the MLV's?

 I used to use MLV_Diag from Chmee to cut the files, but it does not work with the lossless MLV's.
Title: Re: mlv_dump on steroids
Post by: Danne on June 26, 2017, 04:50:19 PM
Code: [Select]
mlv_dump -f 5 -o OUTPUT.MLV INPUT.MLV
Beware output name must be different from input name or you end up with a 0 byte file.
Title: Re: mlv_dump on steroids
Post by: Kharak on June 26, 2017, 05:13:14 PM
okey, thank you.
Title: Re: mlv_dump on steroids
Post by: Kharak on June 27, 2017, 03:09:02 AM
I am uploading 3 Folders with DNG's from 3 separate mlv_dumps all containing 6 DNG's each, hope it enough. Bouncyball's mlv_dump, Danne's dump and latest Crop_rec dump (june 19th). All on the same mlv. The one I posted jpeg examples of in the crop_rec 4k thread. Good to have a standard.

@Danne,

 I can not use your latest mlv_dump, it requires cygwin1.dll or tells me to "please reinstall the program". You mentioned something about win 10, I am on win 8.1 maybe that is why. So I used the previous version that I was using before.

This crazy connection. Download 4-5 kb/s and upload 130-150 kb/s. Thought I had to postpone these uploads 10 days from now.

I send you PM's when it is done.
Title: Re: mlv_dump on steroids
Post by: Danne on June 27, 2017, 09:02:03 AM
Fixed the mlv_dump issue Kharak.
Can you post examples of the difference between vertical stripe code among the mlv_dump versions? A few close ups here would do the trick. Also link to your dng files. Would be great to also have the shortened MLV files. If we need to do some retests we could easliy recheck against the same files.
Title: Re: mlv_dump on steroids
Post by: Kharak on June 28, 2017, 06:45:15 PM
Yes I can, but I can not post the originals I sent you, publicly. I could send the DNG/mlv's to the relevant Devs, but that is not very open source.

So I have some other examples like this one I posted already in crop 4k thread once, much more visible stripes too. https://mega.nz/#!8YIHGIJB!hzewH2JBX55vELsy2MPdp3gPSo2mTzlNKe_CQIVqu_o
But maybe this particular issue has more to do with highlight stripes. The other examples I gave you were stripes in the shadows. anyways, yours and Bouncyballs mlv_dumps fixed these stripes aswell, so if you like, I can send this. I got a very shady connection at the moment, but I will try and upload dng's and mlv's.

I have to cut the mlv in the middle somewhere to get this example in a 6 frame MLV, some code? suggestions?

This example has no flaring the first few seconds and that made, according to a1ex, the crop_rec4k mlv_dump to not remove it. (yours do though).
Title: Re: mlv_dump on steroids
Post by: Danne on June 28, 2017, 06:49:11 PM
You can cut anywhere in the MLV file thanks to g3gg0 indexing code. For example:
mlv_dump -f 25-30 -o output.MLV input.MLV
Title: Re: mlv_dump on steroids
Post by: Kharak on June 29, 2017, 11:14:01 PM
Only took 50 attempts! but here it is.

Experimental Crop_rec4k MLV_dump - DNG's (June 19th Build):
https://mega.nz/#F!pURzVaJQ!jGGhzZAWtHXRm-gX7LEfhQ

Danne's Batch MLV_dump - DNG's:
https://mega.nz/#F!hZREiJZK!2j1SfZGur31XZJ0p2gvsbA

Bouncyball's MLV_dump on Steroids - DNG's:
https://mega.nz/#F!9dZm3QSA!Q2XhjUCzWSWR5nE8NkEc4A

The cut MLV:
https://mega.nz/#!cF43DDqQ!z1J-HlaQscli88b0BeCzxfBRrV6HC6X-i1L1o88Kob0

I hope these can help. Let me know if the Scene should be different somehow. This example was my initial foré in to Vertical Stripe fix with the experimentals, but later I found out that I was having severe issues in the shadows aswell.

Title: Re: mlv_dump on steroids
Post by: bouncyball on June 30, 2017, 03:46:29 PM
@Kharak: Thank you. Downloaded MLV.
Title: Re: mlv_dump on steroids
Post by: Danne on June 30, 2017, 04:01:46 PM
Thanks K. Will check asap.
Title: Re: mlv_dump on steroids
Post by: Kharak on June 30, 2017, 10:00:24 PM
I am posting another example showing Vertical Stripes in the shadows, no matter what MLV_dump I use they are there. FYI: of the bat in this case I see no difference in the shadow stripes between MLV_dump from Bouncyball/Danne or A1ex's Experimental MLV_Dump (june 19th). I did not pull all of them together in to and editor to check differences. No matter, stripes are there.

On a static scene the Vertical Stripes are well hidden, but as soon as movement is going on, either camera movement or subject movement across shadows, they become very visible.

Look Camera Right, the shadow on his cheek and throat, MLV Example:
https://mega.nz/#!sAZCABKK!XHWF4aGHue6Uu1356gufLS7_7NALDx5D5ucQG3iZPJs

Set MLRawViewer Curve to Log-c add a punchy LUT of your choosing. Preferably one that just barely crushes blacks and raises highlights to almost blow out. Raise the EV to +1.51 (or there abouts) to emulate the Cinelog-C Workflow, where you increase the Signal Strength, this should highlight the stripes. And ofcourse play at 24 FPS.

These stripes are not just visible to the Pixel Peepers but for the "normal" audience too. And hiding the stripes in blacks is not a workable solution, way too much DR is lost to crush them out.

The problems persists after Noise Reduction as the algorithms "think" it's important information, atleast noise dithers the stripes somewhat, but I don't like 5D noise in those medium dark shadows, and stripes are still visible with or without NR. I took some screenshots with NR and full grade, but on the screenshots they disappear because no movement, I hope you can replicate, if not let me know.

When playing the converted DNG's in MLRawViewer and having Vertical Stripe fix ON, the Vertical Stripes that MLV_dump could not get rid of, disappear. At first it looks perfect! But at a closer look I see it introduces Horizontal stripes in its place instead, not as severe as the Vertical Stripes, but Horizontal Stripes are there. MLRawViewer will most of the time remove the Vertical Stripes that MLV_dump could not, but at total random will leave chunky Vertical Lines through the frame where any Black or high contrast source is in an image or these Horizontal small lines in medium to dark shadows.

Unfortunately I can't export Prores from MLRawViwer, that just stopped working for me many months ago.

Let me know if I can do something else.
Title: Re: mlv_dump on steroids
Post by: a1ex on June 30, 2017, 10:26:47 PM
http://www.magiclantern.fm/forum/index.php?topic=13152.msg170924#msg170924

This is called pattern noise (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/#patternnoise), and it's unrelated to what we call "vertical stripes". If a dark frame doesn't fix it, there is an experimental algorithm in MLVFS, although, for best results, you may have to adjust things directly in the source code.

Try exposing properly next time. Since the highlights are not critical in this case, you should have increased the exposure by 2 stops (which would make a big difference in shadows and midtones, but will also require some highlight recovery capabilities from your raw processor). To clip the skin tones (near the window), you would need +3.5 stops on top of your current exposure level, so there's plenty of headroom.

To recap:
- pattern noise in shadows -> use a dark frame
- vertical stripes in highlights (sharp, not noisy) -> use a flat frame
- both kind of defects -> use both calibration frames.

For exposure:
- clouds, sky, buildings, out of focus backgrounds: clipping 1 or 2 channels should be OK
- skin tones: do not clip anything

(unfortunately Auto ETTR doesn't have face detection)

For existing footage, you should be able to record the calibration frames afterwards.
Title: Re: mlv_dump on steroids
Post by: Danne on June 30, 2017, 10:30:48 PM
fyi
Pattern noise setting is ported to mlv_dump for steroids.
Title: Re: mlv_dump on steroids
Post by: a1ex on June 30, 2017, 10:40:48 PM
Thanks for pointing out - will definitely experiment with this option as soon as my PC will be able to play videos again.
Title: Re: mlv_dump on steroids
Post by: Danne on June 30, 2017, 11:41:12 PM
Cool.
And @Kharak. Also try out chroma smooth settings. 2x2 with darkframes works wonders on eosm for instance.
Title: Re: mlv_dump on steroids
Post by: Kharak on July 01, 2017, 03:16:29 AM
@Danne.

What are the dis/advantages of using Chroma Smoothing? I must have read something about it years ago, that it crushes details or something, because I've been avoiding it like the plague. Besides, if Chroma Smoothing is a fix for these issues and has no obvious caveats, shouldn't it be On as standard for every converter? For me, anything that destroys detail I avoid, especially on 5D3 because the perceived resolution is so small as is.

And how do I do FPN fix in mlv_dump? Do I need to make FPN Frames like Darkframes ?

@a1ex

Okey thank you, so its FPN. So now the silly questions. MLRawViewer manages to get rid of them with its vertical Stripe fix, but introduces artifacts like horizontal stripes and Chunky vertical stripes in black/dark spots and/or high contrast sources instead. Is it in anyway possible to combine the best of both worlds? It is my understanding that MLRawViewer uses OpenGL shaders with combination of different coding languages, ty @bouncyball, but it does a fine job at removing these vertical FPN except for generating other artifacts. Thoughts?

I find the exposure to be fine for my style, I like creamy soft highlight roll-offs and ETTR'ing too much usually ruins that, especially blowing out two channels, that is just not recoverable. This shot could have been +0.5 EV higher, maybe +1, that's a big maybe. But for my shooting style it's good enough. The shadows hold a lot of information underneath the noise, something Temporal NR is good at extracting. I just can't get rid of the FPN.
Quick grade: https://mega.nz/#!FVwXSSpI!4lyl105BGrLCU7fcPEJVxqoX1jVxVaGZpl5MzhU2gbY .jpeg

Or maybe I need to learn to Pull exposure better in post.

This FPN in the shadows was not a problem for me with this shooting style in Vanilla 24p 3x3 1080p. I think the sensor is pushed too much in 60p mode, introducing more FPN than normal. Just a hunch I have.

btw bit off topic, a line above in the link you posted:
Quote
The read noise can be isolated by taking a "black frame" image, an exposure with the lens cap on and the highest available shutter speed; there are thus no photons captured, and only the electronic noise from reading the sensor remains.
Is this the actual proper way of doing a Darkframe ? Setting shutter speed to the highest available?

And yet again, everything points to Darkframe+Flatframe for the ultimate image. But for any kind of production that is not possible as is, on Windows atleast. Its just way too much manual work, decompressing, sorting ISO's, sorting lenses, reprossesing for Darkframe+Flatframe, to me it just doesn't seem feasible. Not the Flatframe thing atleast. I have some Vintage lenses that have quite big variations in the Vignetting at the different F-stops, from 1.4 up to 5.6 and they are all Manual. Here's crossing fingers for Danne's venture in to Windows Darkframe automation. ;)

As always, let me know what else I can help with.
Title: Re: mlv_dump on steroids
Post by: Danne on July 01, 2017, 08:13:58 AM
Try a darkframe. Probably gonna work best. Chroma smooth 2x2 is fine in my world but it might not be needed.
When doing a darkframe film with exact same setting as the footage, lens cap on, shoot 2-3 seconds.
Creating a storage folder with darkframes for batch_mlv shouldn't be too hard. Just need some time.

You need to decompress your footage too before doing the mlv_dump averaging. If it's too many steps just upload a vanilla darkframe and I test it for you.
Title: Re: mlv_dump on steroids
Post by: a1ex on July 01, 2017, 09:07:47 AM
MLRawViewer manages to get rid of them with its vertical Stripe fix, but introduces artifacts like horizontal stripes and Chunky vertical stripes in black/dark spots and/or high contrast sources instead. Is it in anyway possible to combine the best of both worlds? It is my understanding that MLRawViewer uses OpenGL shaders with combination of different coding languages

Unfortunately, MLRawViewer just locks up my current machine (bringing down the entire operating system), so it's not straightforward for me to play with this code. Otherwise I'd probably maintain this program, to some extent.

Quote
This shot could have been +0.5 EV higher, maybe +1, that's a big maybe.

To see how it would have looked at +1 EV, do this:
Code: [Select]
exiftool M08-1044-cut_000000.dng -WhiteLevel -BlackLevel
White Level                     : 16200
Black Level                     : 2047

exiftool M08-1044-cut_000000.dng -WhiteLevel=9123

How did I come up with this number? (2047 + 16200) / 2.

Proof that taking the average of black and white is equivalent with increasing the exposure 1 stop (regarding overall look and highlight clipping point, but not noise levels) if left as an exercise for the reader.

Result...
Code: [Select]
octave:1> a = read_raw('M08-1044-cut_000000.dng');
octave:2> prctile(a(:), [1 5 10 50 90 99 99.9 99.99 100])'
ans =
   2054.0   2066.0   2076.0   2176.0   3285.0   6054.0   8156.0   8828.6   9113.0

That means, not a single pixel would have been clipped in this frame by increasing the exposure by 1 stop (which means, no information would have been lost). You should be able to grade this file identically to the original (please try).

To simulate 2 stops, try white level 5585, and for 1.5 stops, 7050. Math is left as exercise.

Code: [Select]
octave:3> prctile(a(1:2:end,1:2:end)(:), [1 5 10 50 90 99 99.9 99.99 100])'
   2050.0   2059.0   2067.0   2126.0   2767.0   4212.5   5019.0   5236.2   5364.0

That means, with 2 stops, no pixel on the red channel would have been clipped (therefore, no highlights should be clipped to pure white).

My JPEGs (0, +1, +1.5, +2 from white level, +3, +2, +1.5, +1 from exposure slider; for other settings, change extension to .ufraw):

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fufraw-mod%2FM08-1044-cut_000000.jpg&hash=13e70528341e2fada017154900ce3376) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fufraw-mod%2FM08-1044-cut_000000%2B1ev.jpg&hash=7e57c1dbb31e94685ab7c4beb186223f)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fufraw-mod%2FM08-1044-cut_000000%2B1.5ev.jpg&hash=3708cc62579fa193b07fcc3be95d007c) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fufraw-mod%2FM08-1044-cut_000000%2B2ev.jpg&hash=e06526210c596ef9728c168a2c496af3)

The highlights in the last two (where some channels are clipped) are desaturated (they do not retain the original colors, but they are not clipped to white either). That's how highlight recovery works in ufraw. To my knowledge, many commercial raw processing programs perform a lot better in this area (curious to see how Danne would render this).

Quote
btw bit off topic, a line above in the link you posted:
> The read noise can be isolated by taking a "black frame" image, an exposure with the lens cap on and the highest
> available shutter speed; there are thus no photons captured, and only the electronic noise from reading the sensor remains.

Is this the actual proper way of doing a Darkframe ? Setting shutter speed to the highest available?

Not sure how exactly you reached this conclusion; this page (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/) (next time, make sure the article you are talking about can be identified, e.g. with a link) mentions the highest shutter speed as a way to analyze the read noise component (without e.g. thermal noise, which depends on exposure, or PRNU, which depends on the number of photons captured), not as the best way to capture dark frames.

I'd take the dark frames under the same conditions as the original clip. Probably some settings (like shutter speed) won't matter much, but I didn't experiment with it.

Quote
I have some Vintage lenses that have quite big variations in the Vignetting at the different F-stops, from 1.4 up to 5.6

Flat frame would correct the vignetting in those lenses as well - but you will need one averaged flat MLV at each aperture. Repeat for each ISO and some usual shutter speeds, and you'll probably end up with a 3D matrix of correction frames. That would definitely require some sort of automation.
Title: Re: mlv_dump on steroids
Post by: Danne on July 01, 2017, 09:25:49 AM
Very informative, educative. Will try to find more time do some more testing around it. Just started vacay...
Title: Re: mlv_dump on steroids
Post by: bouncyball on July 01, 2017, 09:53:22 AM
Hey guys! Thanks for lots of very interesting talking here :)

Otherwise I'd probably maintain this program, to some extent.
I'm sure every community member would be extremely thankful for that :)
Title: Re: mlv_dump on steroids
Post by: g3gg0 on July 01, 2017, 10:49:57 AM
Flat frame would correct the vignetting in those lenses as well - but you will need one averaged flat MLV at each aperture. Repeat for each ISO and some usual shutter speeds, and you'll probably end up with a 3D matrix of correction frames. That would definitely require some sort of automation.

as proposed in early MLV times, we could use FLAT and DARK frame types instead of VIDF.
a "correction.mlv" then would contain (sorted by block time):

MLVI, RAWI, RAWC, LENS, (misc info), EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, ...

where EXPO and FLAT+DARK are a series of correction frames for the EXPO settings.
(yeah DARK isn't possible as of now so we would omit it, but maybe somewhen we can do that)

possible correction frames: FLAT, DARK, BIAS
(see http://deepskystacker.free.fr/english/faq.htm )
Title: Re: mlv_dump on steroids
Post by: Danne on July 01, 2017, 12:58:14 PM
An idea right out of my ass. What about creating one for all isos darkframe with rec trigger? Record a sequence with iso 100 then pause, set iso 200 record a sequence pause, iso 300 etc, then average this all iso MLV file to one hell of a darkframe?
Title: Re: mlv_dump on steroids
Post by: g3gg0 on July 01, 2017, 01:14:49 PM
yes, thats what i wrote with the sequence above ;D
EXPO <= ISO, exposure time
LENS <= lens, aperture, focal length

Code: [Select]
typedef struct {
    uint8_t     blockType[4];
    uint32_t    blockSize;    /* total frame size */
    uint64_t    timestamp;    /* hardware counter timestamp for this frame (relative to recording start) */
    uint32_t    isoMode;    /* 0=manual, 1=auto */
    uint32_t    isoValue;    /* camera delivered ISO value */
    uint32_t    isoAnalog;    /* ISO obtained by hardware amplification (most full-stop ISOs, except extreme values) */
    uint32_t    digitalGain;    /* digital ISO gain (1024 = 1 EV) - it's not baked in the raw data, so you may want to scale it or adjust the white level */
    uint64_t    shutterValue;    /* exposure time in microseconds */
}  mlv_expo_hdr_t;

typedef struct {
    uint8_t     blockType[4];
    uint32_t    blockSize;    /* total frame size */
    uint64_t    timestamp;    /* hardware counter timestamp for this frame (relative to recording start) */
    uint16_t    focalLength;    /* in mm */
    uint16_t    focalDist;    /* in mm (65535 = infinite) */
    uint16_t    aperture;    /* f-number * 100 */
    uint8_t     stabilizerMode;    /* 0=off, 1=on, (is the new L mode relevant) */
    uint8_t     autofocusMode;    /* 0=off, 1=on */
    uint32_t    flags;    /* 1=CA avail, 2=Vign avail, ... */
    uint32_t    lensID;    /* hexadecimal lens ID (delivered by properties?) */
    uint8_t     lensName[32];    /* full lens string */
    uint8_t     lensSerial[32]; /* full lens serial number */
}  mlv_lens_hdr_t;

Title: Re: mlv_dump on steroids
Post by: Danne on July 01, 2017, 01:31:57 PM
Quote
(yeah DARK isn't possible as of now so we would omit it, but maybe somewhen we can do that)
So we can uncomment some abort stops in mlv_dump to try this?
With all due respect flat frames needs some more testing and going throughs. As opposed to darkframes it's more tinkering time getting it to work to the better. Removing vignetting often causes grain and other side effects such as unexpected pinks in highlights. Not being able to upload examples isn't ideal here but I see if I find time here. Darkframes works very reliably imo.
Title: Re: mlv_dump on steroids
Post by: a1ex on July 01, 2017, 02:08:25 PM
Removing vignetting often causes grain

Because of exposure pushing in the dark areas? Yes, that's expected.

Quote
unexpected pinks in highlights

White level may need to be adjusted; unfortunately, this will result in clipping useful highlights (that were not clipped in the vignetted area of the frame).

Possible workarounds:
- include some highlight recovery code in mlv_dump (hard, there are an infinite number of methods to do this, as it must guess what's in the clipped channels; that's the job of the raw processing software)
- variable white level support in raw processing programs (does any app support something like a white frame, or other sort of variable white level?)
- apply the vignetting correction after developing the raw image (after the highlight recovery is applied, but while still in some linear space) - depends on the processing app.
- decompose the flat frame into vignette correction (radial averaging, easy) and defects correction (dust spots, vertical stripes, whatever), and apply just the second part.

From all those, the last item appears to be the most practical to me; mlv_dump already does vertical averaging, so it's easy to add a similar option. A very large blurring filter might work as well, if the correction frame is not uniform for some reason.
Title: Re: mlv_dump on steroids
Post by: g3gg0 on July 01, 2017, 02:27:12 PM
i would keep mlv_dump as focused as possible.

if some correction is necessary in "mlv-space", i.e. where we act camera specific, where we are tightly coupled to mlv_lite/mlv_rec, i absolutely agree implementing that in mlv_dump.
also some "primitive" corrections like chroma smoothing, bad pixel maps etc is well placed there.
also the DARK, BIAS, FLAT correction is quite camera - or mlv-workflow - specific and its good to place it in mlv_dump.
lens vignetting correction, well, thats something that we get "for free" using FLAT frames.

more complex and generic corrections might be best somewere else. (hightlight recovery for example)
Title: Re: mlv_dump on steroids
Post by: bouncyball on July 01, 2017, 06:13:34 PM
i would keep mlv_dump as focused as possible.
I've got a vague feeling that I'm gonna remove a lot of stuff from my PR :P

also the DARK, BIAS, FLAT correction is quite camera - or mlv-workflow - specific and its good to place it in mlv_dump.
DARK, FLAT frame types!
Fantastic, simple, effective proposal for "correction.mlv".

more complex and generic corrections might be best somewere else. (hightlight recovery for example)
Would be so nice to not rely on any raw processor and have several advanced hightlight recovery methods in the arsenal. Could be implemented outside of "mlv_dump.c" just during exporting DNGs. Or maybe introduce plugin framework? ;)
Title: Re: mlv_dump on steroids
Post by: g3gg0 on July 01, 2017, 11:36:23 PM
these frames were proposed from the beginning on ;)
well, mlv_dump already has this framework also from the very beginning. lua callouts.

if we integrate highlight recovery - then why not noise reduction, rotation, cropping, lens correction, deflickering, red eye reduction etc.
for me this is definitely raw manipulation and nothing that is a) required to make mlv usable or b) very camera specific.

 
Title: Re: mlv_dump on steroids
Post by: Danne on July 02, 2017, 12:20:45 AM
This test mainly verifies this
Quote
That means, not a single pixel would have been clipped in this frame by increasing the exposure by 1 stop (which means, no information would have been lost). You should be able to grade this file identically to the original (please try).
Mainly this suggests pushing f-stop more "to the right" to achieve cleaner shadows prone to fpn and stripes.

Opened up the files in ACR, altered white level acording to a1ex test above http://www.magiclantern.fm/forum/index.php?topic=18975.msg186641#msg186641.
Matched exposure accordingly(+2, +1.5, +1, 0)
Note white level 5585 and white clip warning. Check 4th pic.
From 5th pic and onwards some heavy highlight recovery and acr behaviour.
Final pics good old dcraw -H 2 recovery.

16200
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fpmu8wyuvd%2Fimage.jpg&hash=716334d71ffc2bb1bc19f1132cd04acf)

9123
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fkcpa5oamh%2Fimage.jpg&hash=38d7b06dfc620b8240616af2d1d7d574)

7050
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2F5brl25hah%2Fimage.jpg&hash=2f07f3269e64847331bbea8683e6eaea)

5585 Note that highlight clip warning is shown in ACR, upper right corner
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fx0e89o4ax%2Fimage.jpg&hash=0a3dda93bda2cbbc513aa40b01205266)




Now some heavy highlight recovery. Slider all the way to the left
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2F8h7lemaix%2Fimage.jpg&hash=4d037fa1255936c3540689a8deccf9d8)


The result by eye with highlight recovery applied. Pic one and two without any clipping. One stop push seems ok as stated by a1ex before.
16200
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fuf428epjd%2Fimage.jpg&hash=2adc7d21d756d7c123db5299006d2d95)

9123(1 stop push)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fe1k2p99e1%2Fimage.jpg&hash=03708c5ce9dd64828834022e4c39463c)



Next two are showing some funky rebuilding artefacts. Do have in mind we are pulling slider to the maximum.
7050
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fjogfmqbwp%2Fimage.jpg&hash=517a5b5b3afdb5a734f420b973a6aaef)

5585
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fvg4axj6ix%2Fimage.jpg&hash=9478a5f15ecc4bbc8bdc0a6f628fc8a3)




dcraw output verifies this as well. I used following command
dcraw -T -w -H 2 INPUT.dng
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Frert5m3m1%2F091_16200.jpg&hash=b813e033b778502feb548718dfc49065)

16200
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fy3yclmoy1%2F091.jpg&hash=8ac6d396405da09f23c9ad91e8d905ae)

9123
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fh5zbzsfk9%2F092.jpg&hash=d8270a515852c526e1b4f3641c995819)

7050
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2F5jf7yns95%2F093.jpg&hash=ee5ddb1be408d9ac54ef85fbdeb4283c)

5585
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs2.postimg.org%2Fwfz76zb2h%2F094.jpg&hash=f404abad5d477814b084f128d15e69e4)
Title: Re: mlv_dump on steroids
Post by: bouncyball on July 02, 2017, 09:24:19 AM
well, mlv_dump already has this framework also from the very beginning. lua callouts.
btw, never used it myself. Should be tested if I broke something after my changes. And no one can do this better than you ;)

if we integrate highlight recovery - then why not noise reduction, rotation, cropping, lens correction, deflickering, red eye reduction etc.
That's right agree, but would be cool, all in one Swiss Army knife, well perhaps red eye is too much :P
Title: Re: mlv_dump on steroids
Post by: g3gg0 on July 02, 2017, 09:30:10 AM
That's right agree, but would be cool, all in one Swiss Army knife, well perhaps red eye is too much :P


ehm. this was ironic... :)
Title: Re: mlv_dump on steroids
Post by: bouncyball on July 02, 2017, 09:41:10 AM
ehm. this was ironic... :)
I know ;)
Title: Re: mlv_dump on steroids
Post by: bouncyball on July 02, 2017, 09:45:07 AM
@Danne

Thanks for thorough investigation!
Title: Re: mlv_dump on steroids
Post by: Kharak on July 03, 2017, 04:16:57 AM
So I did a DFA on the example you all are looking at, it did not help with the FPN. I think it actually made it worse. Mentioned this before on 'mlv_dump on steroids' before Bouncyball fixed a bug that only made VS corrections to the first frame; it's as if Darkframe Averaging makes more contrast for the stripes, but this might be bollocks from me. Again, just a hunch and you guys know a lot better than me what the correction/averaging and so on are actually doing.

I took the decompressed file in to MLVProducer where I made a FPN profile to try and remove it. In parts on his cheek and throat, the results were better, but it also introduced vertical FPN stripes in a few other places instead, like in brighter splotches in the background and so on. Comparing MLVProducer's output, the MLV_dump DFA output and just normal Vertical Stripe fix direct from MLV_dump (danne). It looks to me as normal Output from MLV_dump with VS Fix had the best result of the three.
 Maybe someone else could have a go with the DFA, in case I made a mistake. I don't remember what ISO I shot this at, after decompression, the MLV says ISO 100 and shutter 1/120, the shutter is accurate for sure. So I assume the ISO is true aswell?

All three outputs vary in Colour and Luminance. Which might have skewed my perception of the Striped results (less contrast, more contrast, green darks, magenta darks etc..) Bottom line is, the FPN is still there. Unfortunately I don't have a lot of time to fiddle around matching these frames and outputs, I got a music video coming up which requires a lot of my attention on planning these days. But I still wanna help out as much as I possibly can, I really wanna get rid of this FPN, I mean I think the FPN in those shadows is surrounded with totally usable shadow details (after NR techniques), its just the Stripes making it unusable (to me). 

MLVProducer made the MLV 2 Frames longer than the other converters, so I don't know what's going on there, I outputted 59.976 from MLVProducer, as it read the file, maybe it doesn't know how to count properly or interpolates somewhere along the line. The software is a bit funky sometimes, but I also never really used it, I was more of a RAW2CDNG guy.

Another question. When a MLV has been decompressed, it shows Metadata like ISO and Shutter speed, but after DNG's have been made, that metadata is gone? Wouldn't it be an idea to include that aswell in the DNG output?
Title: Re: mlv_dump on steroids
Post by: a1ex on July 03, 2017, 07:25:11 AM
Well, I'm actually looking for some sample footage with dark frame correction, and another one also with flat frame, as I've just started a test suite for mlv_dump (http://builds.magiclantern.fm/jenkins/view/Utilities/job/mlv_dump_4k_test/). Currently I've used the MLV from this thread (hope you don't mind), and a buggy MLV from another thread.

So, if you have a small dark MLV to match this clip, I can look into it.
Title: Re: mlv_dump on steroids
Post by: Danne on July 03, 2017, 11:11:09 AM
Not sure if these MLV frames and included flatframes could help any. Shot with eosm 14bit. Compressed with lj92(for upload size reasons, bad connection) in post so obviously will need decompressing back.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/eosm_flatframes.zip
332MB
Title: Re: mlv_dump on steroids
Post by: a1ex on July 03, 2017, 11:38:54 AM
Thanks - ideally it should be a real-world example with visible defects that would be corrected by these frames (maybe extract a short clip from one of your dark/flat experiments, where this correction makes a difference).

If possible, it should have some clipped highlights as well.

Currently I'm trying to set up a visual diff, to highlight image changes between different versions. These things are already used in web development - some examples (https://sqa.stackexchange.com/questions/2459/compare-screenshots-of-rendered-web-pages). I've also posted a proof of concept for ML menus (http://magiclantern.fm/forum/index.php?topic=12396.0).
Title: Re: mlv_dump on steroids
Post by: Danne on July 03, 2017, 11:53:52 AM
I see what I can do.
Title: Re: mlv_dump on steroids
Post by: Kharak on July 03, 2017, 02:50:00 PM
@a1ex

I am uploading a unprocessed dark mlv, ISO 100 1/120.

I cut the MLV down to 30 Frames, I hope it is enough, the connection makes this the very max size for me 41 some MB. If you need more, you gotta wait 1-2 days and I'll have 3G or possibly real internet ;)


EDIT:
ISO 100 mlv
https://mega.nz/#!8NgVAAaK!9-870FPyEXgTXA9FGaV2za5NxkHRM8WFmJke661sqDo
Title: Re: mlv_dump on steroids
Post by: a1ex on July 03, 2017, 09:44:29 PM
Thanks - here's my attempt, but unfortunately the dark frame introduced additional artifacts...

Maybe these dark frames really have to be recorded for each clip (ideally at the beginning and at the end - or just at the end, to avoid missing the action).

https://builds.magiclantern.fm/jenkins/view/Utilities/job/mlv_dump_darktest/ (0 = plain, 1 = with dark frame)
Title: Re: mlv_dump on steroids
Post by: ilia3101 on July 05, 2017, 09:56:36 AM
Maybe this isn't relevant to this thread, but could there be a darkframe feature added to mlv/lite where it closes the shutter for a short time at the end of recording and records a few frames in "DARK" blocks? Mlv specific apps could take good advantage of this ;)
Title: Re: mlv_dump on steroids
Post by: Danne on July 05, 2017, 10:09:05 AM
That's a nice idea but it would prolong every clip with around two seconds. Also note that recording a darkframe for every clip isn't needed so then maybe it would be useful as a menu option.

Another idea I have is to build a lua script which would record a short darkframe sample of each iso on any specific cam and then the user could use these files in a dark frame storage. Let's say one loads the script and then it ends when all files are recorded instead of manually having to do this.
Title: Re: mlv_dump on steroids
Post by: ilia3101 on July 05, 2017, 10:18:03 AM
Yeah, it would be nice as a menu option. A script is a good idea, but I think noise pattern might be different at different shutter speeds, probably a good approximation though. Does anyone else think it's worth adding this as an option?
Title: Re: mlv_dump on steroids
Post by: bouncyball on July 05, 2017, 10:44:10 AM
I don't think that electronic rolling shutter can be turned off and sensor still be scanned and data read out still be functional. Otherwise we'd be able to use whole sensor as optical black area. Am I wrong?

If mechanical shutter could be closed in live view that would be another story.
Title: Re: mlv_dump on steroids
Post by: g3gg0 on July 05, 2017, 01:47:31 PM
alex and i already discussed that stuff while ago.
main problem is, we maybe could make a 0-exposue readout. this is a BIAS frame iirc.
a DARK frame involves the correct exposure time and ISO (or at least correct exposure and a fitting BIAS frame for the ISO etc)

we could somewhen also move the mirror/shutter down and then expose for a DARK frame,
but currently we do not know how to achieve this.
this is most likely controlled in the MPU which we didnt reverse engineer yet.
Title: Re: mlv_dump on steroids
Post by: bouncyball on July 25, 2017, 07:27:18 PM
Hi guys!

Some more updates on the 1st post, download link is the same.

regards
bb
Title: Re: mlv_dump on steroids
Post by: Danne on July 25, 2017, 10:46:25 PM
The power of "c". Great work B.

Running time command comparing creation time building the first croprec focus pixel file for eosm:
fmputil
real   0m0.021s
user   0m0.012s
sys   0m0.005s

fpm.sh
real   0m4.717s
user   0m10.848s
sys   0m2.304s
Title: Re: mlv_dump on steroids
Post by: bouncyball on July 26, 2017, 12:21:04 PM
Nice, it's about 225 times faster :)
Title: Re: mlv_dump on steroids
Post by: Kharak on August 16, 2017, 09:39:56 PM
@bouncyball @Danne @a1ex @ g3gg0 @anyone else interested.

Bug Report:

I talked to A1ex about this one, as I feared the Lossless compression in 5D3 had messed up badly on two shots I made, it looks like data corruption or failed compression. To my great relief and anyone else using Lossless MLV, it is not the camera. A1ex thinks it is some kind of filtering or some bug of some sort in the 'mlv_dump on Steroids' causing this.

It's not underflow (that was my first guess, but then I've compared the DNGs in octave and it's more likely to be some sort of filtering operation). If the mlv_dump on steroids has some option to output unprocessed raw data, you could try that.

JPEG screenshot:
https://mega.nz/#!hAo32YwD!iSr0T0XTJdKZ3FJupV6tApUhk7rcixlr5xde8PW7PJI


DNG:
https://mega.nz/#!IIBSDbBT!r4cwb58zCnbuZwcxcPoXY9rixJ2jr3uok5iFQktgtzQ

MLV:
https://mega.nz/#!lZ4A0SyD!Fl1SxrgbM0XtP7RL8vPwK43q9ePhBkSR2iK1bowCsiw some 500 MB.

The "corruption" is stuck in specific spots, its actually in many different places if you look closely its just much more visible in the "high probable aliasing" of the building. Perhaps Vertical Stripe fix messing up?

Both shots where this messed up are of this same building, but it might just aswell be in other shots I have, I just really noticed it on this one.

Converted with Danne's Batch Script combined with "his" mlv_dump on steroids.

5D3 - 123
June 19th build


Let me know if you want any more info.
Title: Re: mlv_dump on steroids
Post by: Danne on August 16, 2017, 09:56:42 PM
I tried processing the file over here on windows and I couldn´t reproduce the artefacts in dng 245. Are you using any additional settings in the menu when developing? I just ran it without any adding.
Title: Re: mlv_dump on steroids
Post by: Kharak on August 16, 2017, 10:01:04 PM
No, no additional processing.

And it is reproducable, again and again.

With your Batch Script Menu, I tried outputting Non-Processed Raw (3). But the program crashes.
Title: Re: mlv_dump on steroids
Post by: Danne on August 16, 2017, 10:16:46 PM
Just tried -p option with and without spaces, in the same folder to another drive etc, without darkframes folder and with. It all worked outputting raw dng files with the setting -p. Strange.

You could try outside the script by:
Code: [Select]
mlv_dump -p --dng INPUT.MLV

You don´t have any darkframes in your folder? Could you try disbling vertical stripes fix. Also disabling cold pixel fix.
Title: Re: mlv_dump on steroids
Post by: Kharak on August 17, 2017, 12:02:23 AM

´´It is not the Darkframes causing it, it is the combination of Cold Pixel Fix with Darkframes causing this corruption. First tried disabling Vertical Stripe fix with DF's and then with --no-fixcp and now it works with Darkframes.

Thank you Danne and A1ex!
Title: Re: mlv_dump on steroids
Post by: Danne on August 17, 2017, 01:04:29 AM
Nice catch Kharak.
Title: Re: mlv_dump on steroids
Post by: a1ex on August 17, 2017, 01:09:31 AM
Can you upload a dark frame for this one, so I can add it to the test suite?
Title: Re: mlv_dump on steroids
Post by: Kharak on August 17, 2017, 09:08:38 AM
Yes, will update this post when done.

Edit:

DF MLV straight out of camera:
https://mega.nz/#!dAg0jQjZ!bs2kX3TwL5UD-kXL7_fCgRQI_nC7-eFwLYD0l6gYKFY ISO 100 216 MB


Averaged DF via Mlv dump on Steroids:
https://mega.nz/#!1AgkwAwR!J0-1xq7emM9-2UFIL4Kk0NMtKMxcrK0Gemow6vmR7F8 2.6 MB
Title: Re: mlv_dump on steroids
Post by: bouncyball on August 17, 2017, 10:17:20 AM
@kharak

Switch uses both mlv_dump versions, for simplicity me and Danne are calling them: ml-dng and steroid version :).
Can you be more specific about version you used? Steroid supports two interpolation methods for bad pixel removal - from mlvfs (more like cr2hdr) and from raw2dng which is used by standard mlv_dump.

Try both in conjunction with dark frame:
Code: [Select]
mlv_dump -bpi 0
mlv_dump -bpi 1
I'm really curious if there will be difference.

thanx
bb
Title: Re: mlv_dump on steroids
Post by: Danne on August 17, 2017, 10:43:10 AM
Actually, he is using batch_mlv which uses your steroid version only named mlv_dump  :P
I should rename it. Anyway. Interpolation method tests are important. I havn't inserted this to batch_mlv but as soon as Kharak uploads his darkframe shot we could check this.
Title: Re: mlv_dump on steroids
Post by: bouncyball on August 17, 2017, 10:46:25 AM
Ah well, never mind. I tried it myself with your posted darkframe sample. Either bad pixel removal method causes artifacts, different ones thought. Can't say which one is better :P both are ugly in it's own way.

I guess if dark/flat frame is used there is no need to fix badpixels or stripes after that.

bb
Title: Re: mlv_dump on steroids
Post by: bouncyball on August 17, 2017, 10:54:00 AM
@Danne: ah I forgot that Kharak uses windows :)
Title: Re: mlv_dump on steroids
Post by: Danne on August 17, 2017, 01:34:49 PM
I just ran the provided darkframe and matching MLV file from Kharak through my version of mlv_dump in Switch and it was not affected by any corruption in file nr 245. My ml-dng version should be closer to unified mlv_dump. I did not disable cold pixel fix.
https://bitbucket.org/Dannephoto/magic-lantern/branch/ml-dng-unified_11b
I could reproduce the corruption with mlv_dump_on_steroids in Switch.

Not affected version of mlv_dump:
terminal output:(beginning)
Code: [Select]
MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: 'M31-1527.MLV'
   - Decompressing before writing DNG
   - Enforcing 14bpp for DNG output
   - Convert to DNG frames
   - Subtract reference frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'
   - Output into 'M31-1527_1_2017-07-31_0001_C0000_'
File M31-1527.MLV opened
File M31-1527.M00 not existing.
Loading subtract (dark) frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'


terminal output(ending):
Code: [Select]
Processing...

Vertical stripes correction:
  1.000  1.000  1.004  1.003  0.998  1.001  1.000  1.003

Frame0 : cold pixels found: 0                             

Reached end of chunk 1/1 after 367 blocks
Processed 348 video frames at 59.98 FPS (5.80 s)
Done
Title: Re: mlv_dump on steroids
Post by: bouncyball on August 17, 2017, 05:26:17 PM
Hmm... It says: "Frame0 : cold pixels found: 0"  this equals to not touching data and not fixing anything hence no artifacts.

Now I understand better what's happening. After dark frame subtraction, MLVFS/cr2hdr bad pixels scaning code tries to find all flavors of bad pixels eg: cold, hot, dead...

raw2dng code is for cold pixel revealing only. In steroid for revealing bad pixels I've used only advanced scaning code from MLVFS but for interpolation both methods are available trough the switches.

bb
Title: Re: mlv_dump on steroids
Post by: Danne on August 17, 2017, 07:30:39 PM
For reliability sake, all pixel fix methods should be default set to off. When pixel peepers see dead, cold, hot things in their dng files they should reprocess with correct pixel method set.
Otherwise we are gonna keep on narrowing down these little never ending mysteries for a long time :)
Title: Re: mlv_dump on steroids
Post by: Kharak on August 17, 2017, 09:56:31 PM
I think I have only ever seen the Cold Pixel Counter say 1, once. Are Cold Pixels usually an issue?

How does a Cold Pixel look like? Is it a dead pixel e.g. Black?
Title: Re: mlv_dump on steroids
Post by: bouncyball on August 18, 2017, 11:11:30 AM
I guess it's more like dead pixel and is nearly black. Someone correct me please :)
Title: Re: mlv_dump on steroids
Post by: Danne on August 18, 2017, 12:36:21 PM
Think they can come in different colors.
http://www.magiclantern.fm/forum/index.php?topic=17795.msg171191#msg171191
Title: Re: mlv_dump on steroids
Post by: Kharak on August 18, 2017, 01:42:36 PM
Okey, luckily not an isuue I come across on the 5d3. Perhaps the 3x3 bayer helps get rid of it, as it has 8 other pixels to draw on in the case of a cold pixel?
Title: Re: mlv_dump on steroids
Post by: Danne on August 18, 2017, 01:47:58 PM
On the 5D mark III there is seldom any use of cold pixel fix. If you use darkframes they take care of them anyway. As well as the hot pixels if any.
Title: Re: mlv_dump on steroids
Post by: a1ex on August 18, 2017, 07:01:52 PM
Cold pixels used to be a problem before this PR (https://bitbucket.org/hudson/magic-lantern/pull-requests/778); see also this analysis (http://www.magiclantern.fm/forum/index.php?topic=18393.0). I think most of them were actually pixels marked as bad by Canon firmware (they mark them as 0, so they are very easy to detect by our "cold pixel" heuristic). Some of them were not 0; they were only below the black level, although I don't remember the specific conditions one would get such artifact.

Need to double-check whether there are still cold pixels on other models (60D and 600D were troublesome, possibly also 550D and 500D, not sure about others). One has to compare a MLV (or a DNG developed with all processing turned off) from any recent build, vs any 2016 build.

Some of the "cold pixels" used to appear near very strong highlights (there were some bug reports about this for 5D3, with green artifacts: this one (http://www.magiclantern.fm/forum/index.php?topic=7122.msg109315#msg109315) or this one (https://bitbucket.org/hudson/magic-lantern/issues/2593/green-noise-in-fire) or this one (http://www.magiclantern.fm/forum/index.php?topic=16401)). These are dynamic and won't be fixed with default processing settings (they require scanning every frame for bad pixels, which is very slow: option --fixcp2).
Title: Re: mlv_dump on steroids
Post by: Kharak on August 18, 2017, 10:06:41 PM
Those examples all seem to derive from bad post-processing software of the mlv/raw. No?

The only thing i see as a consistent issue with the 5D3 is with red channel in high contast to other colours, be it black or blue or green etc. there is an example on the forum from maybe 2 years ago or more where a guy showed an example from some red flowers in the foreground with a blue sky behind (red and blue channel interacting) and he had grey pixels howering over the red colours (in the blue sky) some few pixels away. I cant find it, but remember it as later I myself have seen this a lot. It comes mostly from "really" red sources.

I spend so much time on fishing vessels shooting for clients and for personal use. Here the issue comes up a lot, the bright red/orange Oil Clothes that is the typical colour of any fishermans outfit, in contrast to basically any other darker colour will exerborate this behaviour. Gray pixels hovering over the edge of the Red/orange.

I also recall back in the day shooting on boats, pre dslr, shooting with Panasonic and sony, crappy bit depths, they could not reproduce the true colours of the Outfits, they would turn in to a red blob/mess or orange mess, very difficult to work with.

Dont have my computer, so no examples to show from.

The colour is very intense, its made to be as visible as possible in case of MOB or Blackouts.

Title: Re: mlv_dump on steroids
Post by: DeafEyeJedi on August 19, 2017, 01:56:54 AM
On the 5D mark III there is seldom any use of cold pixel fix. If you use darkframes they take care of them anyway. As well as the hot pixels if any.

+1  8)
Title: Re: mlv_dump on steroids
Post by: bouncyball on August 19, 2017, 12:08:25 PM
I also recall back in the day shooting on boats, pre dslr, shooting with Panasonic and sony, crappy bit depths, they could not reproduce the true colours of the Outfits, they would turn in to a red blob/mess or orange mess, very difficult to work with.
Yeah I remember that mess of deep saturated colors with my SONY and JVC analog and early DV cameras. :) That JVC had 25p progressive scan mode and it hooked me, funny the DR was about 5-6 stops I guess :P

If you'll find the time, post gray pixel artifact examples please.

bb
Title: Re: mlv_dump on steroids
Post by: Kharak on August 19, 2017, 12:24:50 PM
I will find you some examples.
Title: Re: mlv_dump on steroids
Post by: mothaibaphoto on August 20, 2017, 04:56:52 AM
I guess if dark/flat frame is used there is no need to fix badpixels or stripes after that.
bb
If only that was true, but not :(
I've processed file with --no-stripes --no-fixcp twice - one time with black frame, one - without and than pixel peep the difference.
Hot pixels(bright ones) are really gone with black frame, cold/bad - no sure, i dont see any on both shots,
BUT, vertical stripes are the same on both!!! keep investigating, though...
Title: Re: mlv_dump on steroids
Post by: Kharak on September 03, 2017, 06:46:10 PM
Here is an example with the Red Channel having difficulties with the high contrast edges.

Like in this example, look at the Boots of the guy closest to camera, the Red Oil Pants are outlining the boots around the edges. The edges get  This is pretty much consistent from 5D3.

It makes the Boot look "cut out" of the picture or placed there. The Gray pixels appear all over the edges.

When Sharpening the image which I always do, this effect becomes very obvious.

DNG:
https://mega.nz/#!1MoRjCKZ!yU7zbQFKMEpEi4VxCqYIqOdkfVLzDvfnKgOf3JKt-aQ
Title: Re: mlv_dump on steroids
Post by: bouncyball on September 22, 2017, 04:57:44 PM
@Kharak

Sorry about very late answer :)

It makes the Boot look "cut out" of the picture or placed there. The Gray pixels appear all over the edges.
When Sharpening the image which I always do, this effect becomes very obvious.
Yeah you are right, now I understand what did you mean.

bb
Title: Re: mlv_dump on steroids
Post by: ErwinH on October 24, 2017, 10:33:30 AM
I was working with the latest version of the crop_rec_4k branch on my 700D and the current linux (and windows) version of mlv_dump on steroids doesn't handle focus pixels any more. If you can update it, would be much appreciated. (I've compiled my own version for now, but I'm sure there are others with the same issue.)
Title: Re: mlv_dump on steroids
Post by: dfort on October 24, 2017, 03:32:51 PM
Focus pixel fixing has been breaking on various experimental builds. Seen it happen on the EOSM too. Not sure what is going on but in one instance everything was shifted 8 pixels to the right.

Could you document which build, what settings, etc. you're seeing the focus pixels so we can try to reproduce the issue? Also posting a short MLV file would help. There's a way of trimming down an MLV file to only a few frames.

Code: [Select]
mlv_dump -o output.mlv -f 5 input.mlv
Title: Re: mlv_dump on steroids
Post by: bouncyball on October 24, 2017, 07:04:48 PM
I've compiled my own version for now, but I'm sure there are others with the same issue
What do you mean by this? Did you patched code of mlv_dump (+shift/-shift pixels) or managed to forge focus map for your particular case?

[/quote]
Could you document which build, what settings, etc. you're seeing the focus pixels so we can try to reproduce the issue? Also posting a short MLV file would help.
+1
Title: Re: mlv_dump on steroids
Post by: bouncyball on October 24, 2017, 07:12:59 PM
If you meant that only MacOS binary is updated and not Win/Linux ones - then yes, that's right ;)

Both now uploaded (https://bitbucket.org/bouncyball/magic-lantern/downloads/).
Title: Re: mlv_dump on steroids
Post by: ErwinH on October 24, 2017, 07:41:18 PM
This.

If you meant that only MacOS binary is updated and not Win/Linux ones - then yes, that's right ;)

Both now uploaded (https://bitbucket.org/bouncyball/magic-lantern/downloads/).

When I have a little more time I'll look at the different versions and where it goes wrong. I'll come back on this.
Title: Re: mlv_dump on steroids
Post by: ErwinH on October 25, 2017, 11:38:02 AM
I have a recording where the old mlv_dump-on-steroids fails and the new one works:

Old version:
Code: [Select]
MLV Dumper
-----------------

Last update:  33ac267 on 2017-07-25 14:35:07 UTC by bouncyball:
pixel_proc: now supported both: newer .fpm with header or origin...
Build date:   2017-07-25 17:09:30 UTC

New version:
Code: [Select]
MLV Dumper
-----------------

Last update:  f44ffe0 on 2017-10-15 13:02:25 UTC by bouncyball:
pixel_proc: changes to focus pixel map generator
Build date:   2017-10-24 08:10:55 UTC

The command I use is:
Code: [Select]
~/mlv_dump.linux --fpi 0 -f 5 --dng
I'll send the mlv file per PM.

Code: [Select]
MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: 'M17-1504.MLV'
   - Verbose messages
   - Verify file structure
   - Dump all block information
File M17-1504.MLV opened
File M17-1504.M00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 5477059852982235116
    FPS         : 25.000000
    File        : 0 / 0
    Frames Video: 74094
    Frames Audio: 0
    Class Video : 0x00000001
    Class Audio : 0x00000000
Block: RAWI
  Offset: 0x00000034
  Number: 1
    Size: 180
    Time: 0.782000 ms
    Res:  1280x720
    raw_info:
      api_version      0x00000001
      height           1189
      width            1808
      pitch            2260
      frame_size       0x002900A4
      bits_per_pixel   10
      black_level      128
      white_level      1013
      active_area.y1   28
      active_area.x1   72
      active_area.y2   1185
      active_area.x2   1808
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: RAWC
  Offset: 0x000000e8
  Number: 2
    Size: 32
    Time: 0.797000 ms
    raw_capture_info:
      sensor res      5184x3456
      sensor crop     1.62 (APS-C)
      sampling        3x3 (read 1 line, skip 2, bin 3 columns)
Block: IDNT
  Offset: 0x00000108
  Number: 3
    Size: 84
    Time: 0.811000 ms
     Camera Name:   'Canon EOS REBEL T5i'
     Camera Serial: '9C93E2F79'
     Camera Model:  0x80000326
Block: EXPO
  Offset: 0x0000015c
  Number: 4
    Size: 40
    Time: 0.823000 ms
     ISO Mode:   0
     ISO:        400
     ISO Analog: 88
     ISO DGain:  0/1024 EV
     Shutter:    20000 microseconds (1/50.00)
Block: LENS
  Offset: 0x00000184
  Number: 5
    Size: 96
    Time: 0.847000 ms
     Name:        'EF-S18-135mm f/3.5-5.6 IS STM'
     Serial:      ''
     Focal Len:   24 mm
     Focus Dist:  132 mm
     Aperture:    f/4.00
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x0000102E
     Flags:       0x00000000
Block: RTCI
  Offset: 0x000001e4
  Number: 6
    Size: 44
    Time: 0.863000 ms
     Date:        17.10.2017
     Time:        15:04:43 (GMT+0)
     Zone:        ''
     Day of week: 2
     Day of year: 289
     Daylight s.: 0
Block: WBAL
  Offset: 0x00000210
  Number: 7
    Size: 44
    Time: 1.495000 ms
     Mode:   9
     Kelvin:   5500
     Gain R:   483
     Gain G:   1024
     Gain B:   660
     Shift GM:   0
     Shift BA:   0
Block: VERS
  Offset: 0x0000023c
  Number: 8
    Size: 172
    Time: 58.597000 ms
  String: 'mlv_lite built 2017-10-16 20:22:53 UTC; commit 8ee7858 on 2017-10-16 20:21:52 UTC by alex: mlv_lite: fix 8..12-bit lossless in 720p on small cameras '
Title: Re: mlv_dump on steroids
Post by: bouncyball on October 25, 2017, 12:40:11 PM
As you named it - "Old" mlv_dump supports focus pixel correction only by .fpm map files, while "New" one has built in map generator, hence does not require map file in the binary directory. For overriding the built in generator you should put the map file beside mlv_dump and if the name of the file matches the cameraID and raw buffer resolution it will be used instead.

Your sample MLV is recorded for mv1080 video mode and should be processed w/o issue, but... as you can see raw buffer resolution is 1808x1189, not 1190 and there is no .fpm file with that name by default. That is why "Old" version did not work for you out of the box (you had to rename the file to 80000326_1808x1189.fpm, this is the established standard used after MLVFS). On contrary "New" mlv_dump does not care about vertical resolution mismatches and will work for any Y resolution.

There is also one thing. If you're gonna process MLV recorded in crop_rec mode, then to activate correct built in map generator mlv_dump needs the switch '--is-croprec' on the command line. It is temporary solution until I implement parsing of RAWC header information to make sure this mode is indeed crop_rec.

regards
bb
Title: Re: mlv_dump on steroids
Post by: ErwinH on October 25, 2017, 02:43:15 PM
Thank you for the explanation. Good that it wasn't anything serious.

I'll be shooting a new session tomorrow and will test it again.
Title: Re: mlv_dump on steroids
Post by: ErwinH on October 26, 2017, 03:59:17 PM
Small bug report. If you set the output to a folder -o DNGFOLDER/ the wav file isn't exported.

If you set it to -o DNGFOLDER/prefix the wav file is exported.
Title: Re: mlv_dump on steroids
Post by: bouncyball on October 26, 2017, 06:19:49 PM
Thanks, I'll check it out.

Edit: is that the same in official/vanilla mlv_dump?
Title: Re: mlv_dump on steroids
Post by: ErwinH on October 27, 2017, 03:28:59 PM
Official mlv_dump has the same issue.
Title: Re: mlv_dump on steroids
Post by: bouncyball on October 27, 2017, 06:45:23 PM
OK thank you.
Title: Re: mlv_dump on steroids
Post by: SharpieUser on December 07, 2017, 08:15:20 PM
Hello, I've been following these forums for over a year now, finally decided to make an account. I'm really stumped right now, and have a few questions. :D

Will (mlv dump on steroids) process my raw, dual iso files? and is there a guide on how to use this mlv dump?

the (batch mlv in out) program works great but it does not have any option for dual iso, does it? maybe i missed somthing.


I can process dual iso raw video on my macbook already through the mlvfs, but it takes WAY to long to transfer back over to my pc. around 20mins when copying a single 1g file off of the fuse mount ???.

Thank you for reading. any help will be greatly appreciated.

Title: Re: mlv_dump on steroids
Post by: bouncyball on December 10, 2017, 12:22:24 PM
@SharpieUser

mlv_dump does not support dual iso processing, just fixing bad/focus pixels for dual iso with '--is-dualiso' switch. I wanted to implement this feature but then gave up, not worth it.

Instead I implemented it to the MLV App (http://www.magiclantern.fm/forum/index.php?topic=20025.0). Try it and and tell what do you think :)

regards
bb
Title: Re: mlv_dump on steroids
Post by: a1ex on December 20, 2017, 12:31:20 PM
Possible bug: http://www.magiclantern.fm/forum/index.php?topic=15088.msg194917#msg194917
Title: Re: mlv_dump on steroids
Post by: bouncyball on December 21, 2017, 04:48:41 PM
@a1ex

Hmm... LINK (https://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d/diff). Could this be corrected/done in camera in the future? To not mess with channel data offsetting by 128,256,512,1024 in post? This (http://www.magiclantern.fm/forum/index.php?topic=15088.msg194950#msg194950) will be temporary solution for post tools or am I missing the basic idea?

bb
Title: Re: mlv_dump on steroids
Post by: a1ex on December 21, 2017, 06:10:03 PM
Well....
- the link is for new-style DryOS task switching hooks on recent D5 models; it should not impact any functionality (makes the way we are overriding tasks more "unified", adds CPU usage, some error checks, stuff like that)
- these 128,256,512,1024 black level offsets are a LJ92 encoding bug on 6D, that we have no idea how to track down, so I thought it may be a better idea to work around this in post
- the sample DNG files (coming from mlv_dump on steroids) could not be adjusted with exiftool -BlackLevel (but others found out they can be adjusted with -IFD0:BlackLevel)
- the same sample DNG files (coming from mlv_dump on steroids) could not pass dng_validate.exe tests (and that's the bug report for you).
- edit: OP found out he was using an older version (https://www.magiclantern.fm/forum/index.php?topic=15088.msg194960#msg194960), so please disregard.
Title: Re: mlv_dump on steroids
Post by: JADURCA on December 22, 2017, 02:22:06 PM
"Unfortunately, MLRawViewer just locks up my current machine (bringing down the entire operating system), so it's not straightforward for me to play with this code. Otherwise I'd probably maintain this program, to some extent."

@a1ex Then we should avoid it. Is there another player for just viewing MLV files?
Title: Re: mlv_dump on steroids
Post by: bouncyball on December 22, 2017, 04:16:49 PM
Thx a1ex, got it. I'll try to fix 6D case in software, but... I really do not like this ;)


btw, I compiled dng_validate for linux. If someone needs it, it's here (http://nic.caucasus.net/ml/?dir=&download=dng_validate). Note: It might require some libs to install.
Title: Re: mlv_dump on steroids
Post by: bouncyball on December 22, 2017, 04:55:56 PM
@a1ex Then we should avoid it. Is there another player for just viewing MLV files?
a1ex said this before @hwdbk made his changes to support losless MLVs. Link is Here (https://bitbucket.org/hwdbk/mlrawviewer/downloads/). It is not finished thought and needs further development.

Meanwhile you can play MLVs with MLV App (https://ilia3101.github.io/MLV-App/) for all 3 OSes (Mac, Linux, Win), or use mlvproducer (https://ilia3101.github.io/MLV-App/) (windows only).

bb
Title: Re: mlv_dump on steroids
Post by: bouncyball on January 12, 2018, 05:10:49 PM
Hey guys some updates to the 'mlv_dump os'. Added feature to auto generate focus pixel maps according to latest break through development.

Downloads are on the first post as usual.

bb
Title: Re: mlv_dump on steroids
Post by: DeafEyeJedi on January 12, 2018, 11:03:54 PM
Thanks, @bouncyball!
Title: Re: mlv_dump on steroids
Post by: bouncyball on January 13, 2018, 05:03:17 PM
Thanks, @bouncyball!
:D
Title: Re: mlv_dump on steroids
Post by: JADURCA on January 15, 2018, 06:42:21 PM
a1ex said this before @hwdbk made his changes to support losless MLVs. Link is Here (https://bitbucket.org/hwdbk/mlrawviewer/downloads/). It is not finished thought and needs further development.

Meanwhile you can play MLVs with MLV App (https://ilia3101.github.io/MLV-App/) for all 3 OSes (Mac, Linux, Win), or use mlvproducer (https://ilia3101.github.io/MLV-App/) (windows only).

bb

Thanks! After I wrote this, found MLV App and mlv_dump on steroids. I'm using both actively, mlv_dump os for darkframes high ISO footages. MLV App dont have this darkframe feature right?
Title: Re: mlv_dump on steroids
Post by: bouncyball on January 15, 2018, 07:32:13 PM
MLV App dont have this darkframe feature right?
Not at this moment, but... ;)
Title: Re: mlv_dump on steroids
Post by: JADURCA on January 16, 2018, 05:33:59 PM
Hey guys some updates to the 'mlv_dump os'. Added feature to auto generate focus pixel maps according to latest break through development.

Downloads are on the first post as usual.

bb

Great! I just download it and overwrite mlv_dump file. Right?

Thanks!
Title: Re: mlv_dump on steroids
Post by: JADURCA on January 17, 2018, 12:04:42 AM
Having problems with both apps now. MLV app and mlv_dump os   :-\
Title: Re: mlv_dump on steroids
Post by: masc on January 17, 2018, 10:39:36 AM
Having problems with both apps now. MLV app and mlv_dump os   :-\
Could you be more precise please? Maybe you can provide a small MLV for debugging as well?
Title: Re: mlv_dump on steroids
Post by: IDA_ML on January 17, 2018, 12:27:21 PM
I also continue having problems with MLVApp on both: my laptop Lenovo 3000 N200 with 4 GB of RAM and my desktop PC - QuadCore Q6600, MoBo DP35DP, 8 GB DDR2 RAM, GTX750 Ti GPU.  On both of them I have Win7x64 Ultimate running.  MLV_dump on steroids works on both PCs fine.

I tried the 64-bit version 013 of MLVApp on both of them.  When I click on the software icon it starts normally and opens the main window.  When I try to import an MLV file, regardless of its size, resolution, bitrate and/or compression, the software crashes and the message "MLVApp.exe has stopped working" appears.  I also tried the 32-bit version and observed the same behavior.

Interestingly, the software works flawlessly on a different laptop of a friend of mine who has exactly the same version of Win7x64 installed.   His laptop is a Dell Precision M6600 with i7 2720 and 16 GB of RAM.  Could it be that MLVApp requires at least 16 GB of RAM to work properly?
Title: Re: mlv_dump on steroids
Post by: bouncyball on January 17, 2018, 01:18:17 PM
I'm running it w/o issues in Win10x64 on my 5 year old Asus Zenbook laptop with 4gb mem.

Edit: And in virtual machine with Windows7 x64 (4GB) under Vmware ESXi @ my work.
Title: Re: mlv_dump on steroids
Post by: bouncyball on January 17, 2018, 01:29:49 PM
Having problems with both apps now. MLV app and mlv_dump os   :-\
Please tell us what kind of problem is it.
Title: Re: mlv_dump on steroids
Post by: JADURCA on January 17, 2018, 01:41:30 PM
Upgraded from Windows 7 64-bit  to Windows 10 64-bit. Maybe is that the problem? Where I can download mlv__dump on steroids again? To try again right away and send you photos in case it don't work.
Title: Re: mlv_dump on steroids
Post by: masc on January 17, 2018, 02:10:54 PM
Hm... strange. I have MLVApp running on Win7 64bit Core i3 4GB RAM, Win8.1Pro 64bit Core i7 8GB RAM, OSX 10.9 16GB RAM, 2x OSX 10.11 8 & 24GB RAM and Ubuntu 12.04LTS VM without that problem. It is bad, that Windows never tells the reason for a crash...
Do you activate "AMaZE Cached"? This crashes Windows 32bit versions on my systems.
Title: Re: mlv_dump on steroids
Post by: JADURCA on January 17, 2018, 02:51:47 PM
@masc True is that MLV app is not the one crashing, is mlv_dump on steroids. I'm having other problems that can't be reproduced with MLV app, so I can't use it anymore for extracting, just for playing mlv files. Can you help where to locate mlv_dump on steroids? I can't find it to download again...
Title: Re: mlv_dump on steroids
Post by: masc on January 17, 2018, 03:04:35 PM
Ah okay. Then I missunderstood. Is it here?
https://bitbucket.org/bouncyball/magic-lantern/downloads/
Title: Re: mlv_dump on steroids
Post by: Danne on January 17, 2018, 03:12:13 PM
Here are the latest versions of mlv_dump on steroids both mac and windows
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_os.zip

If on mac make sure you give it priviligies:
Code: [Select]
chmod u=rwx mlv_dump_osenter
Title: Re: mlv_dump on steroids
Post by: bouncyball on January 17, 2018, 03:23:52 PM
Thanx Danne. When I'll update it with latest stuff compile the mac binary for me. My macos VM does not boot anymore ... IDK why.
Title: Re: mlv_dump on steroids
Post by: JADURCA on January 17, 2018, 03:33:50 PM
I think I'm confused here, I'm looking for an archive called "batch_mlv_in-out" to download it so I can retest it using mlv batch process and darkframe folders, all of that. Im not making myself clear.
Title: Re: mlv_dump on steroids
Post by: Danne on January 17, 2018, 03:51:34 PM
Ok, you can replace it with the one provided in the link. Just rename mlv_dump_os.exe to mlv_dump.exe and replace the one inside. If you still have issues please post about them in the other forum post.
Title: Re: mlv_dump on steroids
Post by: JADURCA on January 17, 2018, 03:54:04 PM
Ok, you can replace it with the one provided in the link. Just rename mlv_dump_os.exe to mlv_dump.exe and replace the one inside. If you still have issues please post about them in the other forum post.

Thanks! Then I'm renaming and replacing to test. Can you please tell me the title name of the other forum post?
Title: Re: mlv_dump on steroids
Post by: Danne on January 17, 2018, 04:00:58 PM
The one where you got batch_mlv_in-out from. Try pasting in search.
Title: Re: mlv_dump on steroids
Post by: DeafEyeJedi on January 17, 2018, 05:41:08 PM
MLV App dont have this darkframe feature right?

Not at this moment, but... ;)

Dude... don't be such a tease!!!  :P
Title: Re: mlv_dump on steroids
Post by: bouncyball on January 17, 2018, 06:48:25 PM
Dude... don't be such a tease!!!  :P
All right, all right! I won't be... until... :P
Title: Re: mlv_dump on steroids
Post by: JADURCA on January 17, 2018, 08:03:55 PM
C'mon don't play with our feelings  ;D
Title: Re: mlv_dump on steroids
Post by: tajino on April 26, 2018, 11:01:23 PM
Is there any way I could save only odd or even frames out from an input MLV file and output that into a new MLV file?

I need to do this to save some space of redundant duplicated frames of some MLV written by mlv_rec.mo that wrote duplicated frames in 1fps frame override mode in 1x crop on 5D2.

I tried -f switch but that only allows input of frame or frame range, not comma delimited frame numbers, -f 0,2,4,6,8 for example.

Any ideas would be appreciated. Thanks!
Title: Re: mlv_dump on steroids
Post by: bouncyball on April 27, 2018, 01:30:21 PM
Yeah... unfortunately this is not supported by mlv_dump directly (MLV -> MLV).

What's wrong with exporting DNGs and just picking the needed ones?
Title: Re: mlv_dump on steroids
Post by: tajino on April 27, 2018, 05:42:24 PM
Yeah... unfortunately this is not supported by mlv_dump directly (MLV -> MLV).

What's wrong with exporting DNGs and just picking the needed ones?

Nothing wrong with it, that's what I did. It is just that I would usually keep the source MLV, just in case in I need to re-dump the MLV with a newer/better mlv_dump with improved vertical banding removal, etc. maybe who knows.

Title: Re: mlv_dump on steroids
Post by: bouncyball on April 27, 2018, 07:17:14 PM
I understand your point.

But IMHO the better way to fix this with mlv_rec.mo to not duplicate frames when fps = 1. Implementing something like even or odd only frame saving is not that hard but barely widely usable, despite duplicates can turn out be like: 01010111010100 (given that 0 and 1 are frames) this requires picking not only odd/even frames and will be even more work for developer :).

regards
bb
Title: Re: mlv_dump on steroids
Post by: tajino on April 27, 2018, 09:18:19 PM
How about -f "frameslist.csv" ?
make the -f switch when referencing a user generated .csv to only include those frames listed in it.

Title: Re: mlv_dump on steroids
Post by: bouncyball on April 29, 2018, 08:19:06 PM
This should be done in the mlv_dump code not in the code I added, so maybe you will try to ask g3gg0 b/c this proggie is his exclusive child ;)
Title: Re: mlv_dump on steroids
Post by: g3gg0 on April 29, 2018, 09:00:54 PM
this proggie is his exclusive child ;)

not correct ;)
accepting pull requests if they fit in cleanly.
and such addition sounds if it can be done in a clean way.
Title: Re: mlv_dump on steroids
Post by: bouncyball on April 30, 2018, 11:30:08 AM
not correct ;)
Well... right ;). Sorry about being too lazy.

@g3gg0: BTW could this be done with mlv_dump's LUA scripting feature?
Title: Re: mlv_dump on steroids
Post by: g3gg0 on April 30, 2018, 01:02:56 PM
yeah good point. could be done, have to check this.
maybe we have to add a callback but no big issue.
Title: Re: mlv_dump on steroids
Post by: Danne on April 30, 2018, 01:15:00 PM
I like this  :)
Title: Re: mlv_dump on steroids
Post by: tajino on May 01, 2018, 12:15:26 AM
Thanks bouncyball & g3gg0  :D looking forward to the feature addition.
Title: Re: mlv_dump on steroids
Post by: reddeercity on June 02, 2018, 06:00:59 AM
Not sure if this is the right place , getting a
Code: [Select]
Win32 Expceptionsdebugging it in Microsoft Visual Basic 2015
Code: [Select]
Exception thrown at 0x7548A04C (msvcrt.dll) in mlv_dump.exe: 0xC0000005: Access violation reading location 0x02D31000.
Unhandled exception at 0x7548A04C (msvcrt.dll) in mlv_dump.exe: 0xC0000005: Access violation reading location 0x02D31000.
mlv_dump.exe (downloaded today) from  Branch: crop_rec_4k (https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/72/artifact/modules/mlv_rec/mlv_dump.exe) on
the https://builds.magiclantern.fm/experiments.html page -- running win7 pro ,

in call stacks this
Code: [Select]
> msvcrt.dll!___ascii_strnicmp()
  ntdll.dll!77acfaea()
  [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
seems this line in the disassembly
Code: [Select]
7548A04C 66 0F 6F 4E 10       movdqa      xmm1,xmmword ptr [esi+10h]
most of this is over my head but i thought it my be useful for others.
No sure if it's just my test .mlv that's the problem , wondering if any one else has had a   "Win32 Expceptions"
Title: Re: mlv_dump on steroids
Post by: bouncyball on June 02, 2018, 12:26:18 PM
Is that exception happens when just running the binary or if you specify some switched to do some job?

Please post exact CLI command and exact MLV you used.