mlv_dump on steroids

Started by bouncyball, February 16, 2017, 07:10:10 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

k2121

Hi,

In Windows 7 Service Pack 1 64-bit, program mlv_dump not working.

mlv_dump crash

Problem signature:
   Problem Event Name: APPCRASH
   Application Name: mlv_dump.exe
   Application Version: 0.0.0.0
   Timestamp applications: 589fb28f
   The name of the module with an error: ntdll.dll
   The module version with the error: 6.1.7601.23455
   The timestamp of the module with an error: 573a5463
   Exception code: c0000005
   Moving the exception: 00012c22
   OS Version: 6.1.7601.2.1.0.256.1
   Locale ID 1045
   Additional information 1: 0a9e
   Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
   Additional Information 3: 0a9e
   Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

Read our online privacy statement:
   http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0415

If the principle of confidentiality of information online is not available, please read our privacy statement offline:
   C: \ Windows \ system32 \ en-US \ erofflps.txt


Raw Video

Is there mlv_dump_steroids_1.0 version for Windows?
650d.104 ML • SX1.200h CHDK

bouncyball

Yay! Feedback at last! :)

Well in reality this is the problem of your .IDX xref file. It's somehow wrong or corrupted (file size is wrong and block_size is wrong after XREF header). In all versions of mlv_dump it's gonna dump the core (will crash).

Just delete this .IDX file and if you really need it, build one with the command 'mlv_dump -x your.mlv' and you'll see the difference in sizes: Your's 5980 bytes vs Generated 4504 bytes.

Regards
bb

Danne

I just finished a cr2hdr_steroids.app(MAC) build for testing which utilize the steroid version of mlv_dump. Download here
https://bitbucket.org/Dannephoto/cr2hdr/downloads/cr2hdr_steroids.dmg
Source code for this version is inside the dmg.

It works the same as cr2hdr.app you start out with a main menu and from there you head over to the mlv_dump settings menu by selecting (m).
https://bitbucket.org/Dannephoto/cr2hdr/overview

Note that you reach darkframe and flatframe automation from the main menu, not from mlv_dump menu.

Also updated to the latest focus pixel(dfort) script with the -n switch
https://bitbucket.org/daniel_fort/ml-focus-pixels

Main menu


mlv_dump menu


bouncyball

Hey man! Great news, thank you :)

Danne

Uploaded a new cr2hdr_steroids version. Some small fixes around darkframe automation.
https://bitbucket.org/Dannephoto/cr2hdr/downloads/cr2hdr_steroids.dmg

All tests below with eosm in crop_rec

Tested darkframe and flatframe handling and confirmed working. Like with g3gg0/a1ex 10/12 bit mlv_dump darkframe subtraction works good with 14 and 12 bit files but with 10bit not so very nice. Instead of subtracting color noise it seems to add to it a little with 10bit. Good to know if recording 10bit footage.
A not so very nice problem is later chroma smooth(dualiso code) vs raw2dng chroma smooth code. As cs 2x2 is great with raw2dng code and for cameras in need of it like eosm the cs code in dualiso code can,t really cope with focus pixels. Have a look.
Also about focus pixels it seems not all of them are taken care of. Check to the right on the picture. Maybe the -n switch needs some checking into @dfort?

Chroma smooth artifacts


Focus pixels



Darkframe testing. Last two picture flatframe averaging

14bit original


14bit Darkframed


12bit original


12bit Darkframed


10bit original


10bit Darkframed


Original (not flatframed)


Flatframed


DeafEyeJedi

Loving the update so far @Danne and I really dig the new mlv_dump menu within your app. It's so funny that 10-bit looks worse after DF avg process. Seems 12-bit may be the 'sweet spot' after all. Thanks for sharing and will test thoroughly before I report anything.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

bouncyball


Danne

Think I got something working with the -n swith in the focus pixel list.

This:
##
# output_row() function
# Write a row of focus pixel coordinates to the output file.
#
output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
      echo -e "$x \t $y" >> "$output"
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
      if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
        if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
          if [[ "$format" == fpm ]]; then
            echo -e "$x \t $y" >> "$output"
          else echo -e "$x \t $y \t 0" >> "$output"
          fi
        fi
      fi
    fi
  fi
  done
}



instead of:
##
# output_row() function
# Write a row of focus pixel coordinates to the output file.
#
output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
      fi
      if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
        if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
          if [[ "$format" == fpm ]]; then
            echo -e "$x \t $y" >> "$output"
          else echo -e "$x \t $y \t 0" >> "$output"
          fi
        fi
      fi
    fi
  done
}


bouncyball

Nicely done! :)

output_row() {
  for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
    if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
      if [[ "$cropXY" == no ]]; then
        ((x = $i))
        ((y = $j))
+       if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
+         if [[ "$format" == fpm ]]; then
+            echo -e "$x \t $y" >> "$output"
+         else.
+            echo -e "$x \t $y \t 0" >> "$output"
+          fi
+        fi
      else
        ((x = $i - $cropX))
        ((y = $j - $cropY))
        if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
          if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
            if [[ "$format" == fpm ]]; then
              echo -e "$x \t $y" >> "$output"
            else.
              echo -e "$x \t $y \t 0" >> "$output"
            fi
          fi
        fi
      fi
    fi
  done
}

Just a small correction. Now this does not break dcraw style output (zero at the end of line) and makes sure there're no negative coordinates (x,y >= 0).

DeafEyeJedi

Loving the collaborations from you two! [emoji56]
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

That extra dcraw else function isn,t really working. I think the if [[ "$format" == fpm ]] Needs an actual fpm file to be fed to the script else it won,t recognize outputting fpm and it will take it for a dcraw output anyway. At least that,s what my tests shown.
I think like this. The -n switch will probably only be used for the mlvfs focus pixel code in your steroid version and otherwise we run the script without the -n switch and then we,re back to the extra 0.

*Unless I specify the -f for fpm file, huh, just noticed that little detail. Will test again.
Ok, your little extras are working since I specified the -f switch as well. Then I tested the old focus pixel code with the -f switch but that din,t work, go figure. I will add your extra lines to dforts script. Great.

Updated cr2hdr_steroids.app with the latest fpm.sh script.

bouncyball

Yup that -f is there from the beginning :)

-f -n does what I really was after when adding -n.

bb

g3gg0

can we partially merge the goodies from that branch into the main branch?

i am afraid that this will end in both branches to develop in their own context and finally get un-merge-able.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

bouncyball

Hi g3gg0 thanx for asking.

Quote from: g3gg0 on June 14, 2017, 03:21:03 PM
can we partially merge the goodies from that branch into the main branch?
Sure we can talk about it more deeply. I always wanted to do this but my version is quite different (different from dmilligans version as well) and I kinda thought to ask you and a1ex to just add it as a separate branch (I agree: not the best idea). Actually, to the "mlv_dump.c" itself, I always tried to make changes as subtle as possible. I think I succeeded more or less and all big changes are only regarding DNG handling and WAV advanced header. All processing of the RAW data before saving DNG is taking place outside of the mlv_dump.c.

Quote from: g3gg0 on June 14, 2017, 03:21:03 PM
i am afraid that this will end in both branches to develop in their own context and finally get un-merge-able.
I did some important commits last two days (now it supports any bit depth compressed/uncompressed DNG output) and did not test merge-ability yet. However, usually I'm very carefull about it and always merge any new changes you make.

BR,
BB

Danne

Thanks for latest changes. Being able to output native bit 10/12/16 with mlv_dump stripes, chroma smooth etc added is magic. Also working with compression with all bits after final strip byte fix to match dng checksum.
Also looking forward to some sort of merge story involving mlv_dump unified as g3gg0 points out. How that would be done idk but where's there competence there's possibilities. Always been missing white balance and some of the matrices.
Insights around the strip byte tag would perhaps make it possible to calculate the correct strip byte tag for when exporting compressed 10bit/12bit dng files with crop_rec_4k build now?
Great work Bouncyball.

g3gg0

well, its already such a huuuuuuge changeset that it is almost impossible to review and understand.
if it was asked to merge, i would have no clue what (side) effects merging would bring with.

also not sure if i want to have e.g. deflicker in mlv_dump.
(don't get me wrong, it's for sure a good piece of software, but mlv_dump wasn't meant as post processing tool.
it fixes all the very camera specific issues and is made for a solid mlv-to-other-tools converter)

just having a PR for this branch is very unlikely to get merged.
you would have to split the changes up.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

Indeed. more mlvfs like approach. And hey, I only scratch my head in amazement when I look into code in mlv_dump. Good inspiration.

bouncyball

Hi g3gg0,

I really understand your point. Huge changeset is not gonna get merged in any case and that is why I hesitated to make PR. Even 1st changes were big enough (but review-able) because of "chdk_dng.c" been replaced by "dng.c" to support CDNG and more...

I think you are familiar with dmilligan's dng lib more or less (actually I modified it but the idea is the same) and if you accept the fact that in order to merge goodies from this version chdk_dng.c should be replaced, then I will try my best to split changes and start merging them to crop_rec_4k branch.

Quote from: g3gg0 on June 17, 2017, 11:40:31 PM
also not sure if i want to have e.g. deflicker in mlv_dump.
Well :) it's the smallest and least important part of changes and takes place outside of mlv_dump.c. Can be removed anytime.

regards
bb

bouncyball

Hey guys!

Finally I updated the binaries to the latest version. Lots of changes are made from last time. Look at the 1st post of this thread. Download links are the same.

And... any feedback is welcome as usual!

br,
bb

Kharak

Hey Bouncyball

I am interested in trying out --force-stripes. I am having issues with Vertical Stripes in a lot of my footage.

Could you post an example how I would write the command ? I am not sure where to put in --force-stripes.
once you go raw you never go back

bouncyball

@Kharak

Command like this:

mlv_dump --dng --force-stripes --show-progress INPUT.MLV

should do the trick.

Also have a look at flat field method as g3gg0 mentioned here

Kharak

@ BB

Thank you, sorry for the late reply. I have been so busy.

The --force-stripes does a very good job. But it is very slow as you said.

I just tried converting a compressed mlv without --force-stripes.

mlv_dump --dng --show-progress INPUT.MLV

And it said "Vertical stripes correction: 'NEEDED'" and then proceeded to show the numbers under it. Does this mean it did perform a "normal" Vertical Stripe Fix or that I need to do it. I tried without --force-stripes hoping I could achieve same results as Danne with his CR2HDR app; he got straight out of the box with an older Vertical Stripe fix code, identical results to the --force-stripes. I don't know if CR2HDR app with that (old) Vertical Stripe Code is a slow process. But I am wondering why this old stripe fix is working so well and I have to use (a seemingly new code?) --force-stripes to remove mine? Is it a Mac and Win thing? :)

I also tried the same MLV:

mlv_dump_s --dng --fixpn --show-progress input.MLV

It said "Vertical Stripe Correction: Unneded". Yet extremely slow, maybe slower than --force-stripes (did not take time). But half way through the conversion I could see I was just adding pattern noise, I was hoping maybe this --fixpn could attack the vertical stripes, i mean in reality Vertical Stripes are Pattern Noise, no?


And last but not least, is there any way to batch process with you mlv_dump or for some form of automation?

Thank you.
once you go raw you never go back

bouncyball

@Kharak

Quote from: Kharak on June 25, 2017, 03:07:55 AM
And it said "Vertical stripes correction: 'NEEDED'" and then proceeded to show the numbers under it. Does this mean it did perform a "normal" Vertical Stripe Fix or that I need to do it.
It's just an information, that after evaluating of the 1st frame there is a real need to correct stripes. No intervention required from user. Normal vertical stripes correction is done according to 1st frame analysis.

Quote from: Kharak on June 25, 2017, 03:07:55 AM
he got straight out of the box with an older Vertical Stripe fix code, identical results to the --force-stripes. But I am wondering why this old stripe fix is working so well and I have to use (a seemingly new code?) --force-stripes to remove mine? Is it a Mac and Win thing? :)
Good question :) (no mac or windows thing). This mlv_dump uses all the same raw processing algorithms what used in MLVFS but it does not mean these algos are better than some other ones used in other code. Not only stripe fix but also chroma smooth and bad/focus pixel interpolation is different and yields different results. Investigation of this subject is on my schedule, just had low priority until recently. Thanks for sharing experience with both stripes code. Are you certain when you say that older normal stripes removal gives same results as --force-stripes? Can you show some images illustrating this?

Quote from: Kharak on June 25, 2017, 03:07:55 AM
I was hoping maybe this --fixpn could attack the vertical stripes, i mean in reality Vertical Stripes are Pattern Noise, no?
Pattern noise removal method is not the same as stripes removal even if it helps to make stripes less visible.

Quote from: Kharak on June 25, 2017, 03:07:55 AM
And last but not least, is there any way to batch process with you mlv_dump or for some form of automation?
It's a command line utility. Of course it's intended to use for batch processing in shell out of the box. Under windows, unfortunatelly, there is no any user friendly, ready to use, automator like Danne's cr2hdr.app.

bb

Danne

I fired up windows under virtual box so been fiddling some windows commands today. Who knows where that ends  :P
http://www.magiclantern.fm/forum/index.php?topic=10526.msg186219#msg186219
By the way. How does --force-stripes work? Is it calculating every dng file individually or only the first?

bouncyball

Hey Danne!

Hahaha welcome to winworld :). Well... there is a rudimentary support of bash in Win10 :P, very, VERY limited to use with full blown scripts. If you someday, somehow decide to port your great work you're gonna need to use powershell (can be even made GUI app with it).

Regarding "--force-stripes": it has force prefix because it means analyze every single frame and correct it. That's why it takes ages :)

bb