Author Topic: mlv_dump function checker  (Read 1139 times)

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5919
mlv_dump function checker
« on: February 07, 2018, 04:00:18 PM »
Been doing tests of mlv_dump(g3gg0,A1ex) version in crop_rec_4k branch which contains all the stuff from dmilligan mlvfs version and a lot of refined functions such as bits option, fixed cold pixel fix handling for 10/12bit files along with a croprec function and much more all put together very nicely by bouncyball in his mlv_dump on steroids and now also merged into crop_rec_4k branch.

To speed up testing I put up a multiprocessing script which do following:
Spits out single dng files testing most functions in mlv_dump and then creates tif previews for manual pixelpeeping. The script also tests compress and decompress functions on MLV and dng files, darkframe, flatframe functions etc.

How to use the script:
Place a sample MLV in a folder along with a version of mlv_dump from crop_rec_4k branch and then run the script in that folder.
Out comes dng files and other test related info and also a folder containing tiff previews.

Script here:
Code: [Select]
cat <<'EOF' > MLV_test_Script

#multiprocessing mlv_dump overall function checker

#first check for dependencies

#check for homebrew
if ! [ -f "/usr/local/bin/brew" ]
then
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
fi

#check for dcraw
if ! [ -f "/usr/local/bin/dcraw" ]
then
brew install dcraw
fi

#check for exiftool
if ! [ -f "/usr/local/bin/exiftool" ]
then
brew install exiftool
fi


#!/bin/bash
workingDir=`dirname "$0"`
cd "${workingDir}"

export PATH="${workingDir}":$PATH

########################Output simple dng file#############################
#start with a pass through file
  mlv_dump -p --dng -f 0 -o 000_-p "$(ls *.MLV | head -1)" && mv 000_-p000000.dng 000_-p.dng && exiftool 000_-p.dng > 000_-p.txt &
  mlv_dump --dng -f 0 -o 001_--dng "$(ls *.MLV | head -1)" && mv 001_--dng000000.dng 001_--dng.dng && exiftool 001_--dng.dng > 001_--dng.txt &
  mlv_dump --dng --no-cs -f 0 -o "002_--dng --no-cs" "$(ls *.MLV | head -1)" && mv "002_--dng --no-cs000000.dng" "002_--dng --no-cs.dng" && exiftool "002_--dng --no-cs.dng" > "002_--dng --no-cs".txt &
  mlv_dump --dng --cs2x2 -f 0 -o "003_--dng --cs2x2" "$(ls *.MLV | head -1)" && mv "003_--dng --cs2x2000000.dng" "003_--dng --cs2x2.dng" && exiftool "003_--dng --cs2x2.dng" > "003_--dng --cs2x2".txt &
  mlv_dump --dng --cs3x3 -f 0 -o "004_--dng --cs3x3" "$(ls *.MLV | head -1)" && mv "004_--dng --cs3x3000000.dng" "004_--dng --cs3x3.dng" && exiftool "004_--dng --cs3x3.dng" > "004_--dng --cs3x3".txt &
  mlv_dump --dng --cs5x5 -f 0 -o "005_--dng --cs5x5" "$(ls *.MLV | head -1)" && mv "005_--dng --cs5x5000000.dng" "005_--dng --cs5x5.dng" && exiftool "005_--dng --cs5x5.dng" > "005_--dng --cs5x5".txt &
  mlv_dump --dng --no-fixfp -f 0 -o "006_--dng --no-fixfp" "$(ls *.MLV | head -1)" && mv "006_--dng --no-fixfp000000.dng" "006_--dng --no-fixfp.dng" && exiftool "006_--dng --no-fixfp.dng" > "006_--dng --no-fixfp".txt &
  mlv_dump --dng --no-fixcp -f 0 -o "007_--dng --no-fixcp" "$(ls *.MLV | head -1)" && mv "007_--dng --no-fixcp000000.dng" "007_--dng --no-fixcp.dng" && exiftool "007_--dng --no-fixcp.dng" > "007_--dng --no-fixcp".txt &
  mlv_dump --dng --no-fixcp --no-fixfp -f 0 -o "008_--dng --no-fixcp --no-fixfp" "$(ls *.MLV | head -1)" && mv "008_--dng --no-fixcp --no-fixfp000000.dng" "008_--dng --no-fixcp --no-fixfp.dng" && exiftool "008_--dng --no-fixcp --no-fixfp.dng" > "008_--dng --no-fixcp --no-fixfp".txt &
  mlv_dump --dng --fixcp2 -f 0 -o "009_--dng --fixcp2" "$(ls *.MLV | head -1)" && mv "009_--dng --fixcp2000000.dng" "009_--dng --fixcp2.dng" && exiftool "009_--dng --fixcp2.dng" > "009_--dng --fixcp2".txt &
  mlv_dump --dng --no-stripes -f 0 -o "010_--dng --no-stripes" "$(ls *.MLV | head -1)" && mv "010_--dng --no-stripes000000.dng" "010_--dng --no-stripes.dng" && exiftool "010_--dng --no-stripes.dng" > "010_--dng --no-stripes".txt &
  mlv_dump --dng --force-stripes  -f 0 -o "011_--dng --force-stripes " "$(ls *.MLV | head -1)" && mv "011_--dng --force-stripes 000000.dng" "011_--dng --force-stripes.dng" && exiftool "011_--dng --force-stripes.dng" > "011_--dng --force-stripes".txt &
  mlv_dump --dng --is-dualiso -f 0 -o "012_--dng --is-dualiso" "$(ls *.MLV | head -1)" && mv "012_--dng --is-dualiso000000.dng" "012_--dng --is-dualiso.dng" && exiftool "012_--dng --is-dualiso.dng" > "012_--dng --is-dualiso".txt &
  mlv_dump --dng --is-croprec -f 0 -o "013_--dng --is-croprec" "$(ls *.MLV | head -1)" && mv "013_--dng --is-croprec000000.dng" "013_--dng --is-croprec.dng" && exiftool "013_--dng --is-croprec.dng" > "013_--dng --is-croprec".txt &
  mlv_dump --dng --fixpn -f 0 -o "014_--dng --fixpn" "$(ls *.MLV | head -1)" && mv "014_--dng --fixpn000000.dng" "014_--dng --fixpn.dng" && exiftool "014_--dng --fixpn.dng" > "014_--dng --fixpn".txt &
  mlv_dump --dng --deflicker=3000 -f 0 -o "015_--dng --deflicker=3000" "$(ls *.MLV | head -1)" && mv "015_--dng --deflicker=3000000000.dng" "015_--dng --deflicker=3000(will look bad).dng" && exiftool "015_--dng --deflicker=3000(will look bad).dng" > "015_--dng --deflicker=3000(will look bad)".txt &
  mlv_dump --dng --no-bitpack -f 0 -o "016_--dng --no-bitpack" "$(ls *.MLV | head -1)" && mv "016_--dng --no-bitpack000000.dng" "016_--dng --no-bitpack.dng" && exiftool "016_--dng --no-bitpack.dng" > "016_--dng --no-bitpack".txt &
  mlv_dump --dng --fpi 0 -f 0 -o "017_--dng --fpi 0 MLVFS" "$(ls *.MLV | head -1)" && mv "017_--dng --fpi 0 MLVFS000000.dng" "017_--dng --fpi 0 MLVFS.dng" && exiftool "017_--dng --fpi 0 MLVFS.dng" > "017_--dng --fpi 0 MLVFS".txt &
  mlv_dump --dng --fpi 1 -f 0 -o "018_--dng --fpi 1 raw2dng" "$(ls *.MLV | head -1)" && mv "018_--dng --fpi 1 raw2dng000000.dng" "018_--dng --fpi 1 raw2dng.dng" && exiftool "018_--dng --fpi 1 raw2dng.dng" > "018_--dng --fpi 1 raw2dng".txt &
  mlv_dump --dng --bpi 0 -f 0 -o "019_--dng --bpi 0 MLVFS" "$(ls *.MLV | head -1)" && mv "019_--dng --bpi 0 MLVFS000000.dng" "019_--dng --bpi 0 MLVFS.dng" && exiftool "019_--dng --bpi 0 MLVFS.dng" > "019_--dng --bpi 0 MLVFS".txt &
  mlv_dump --dng --bpi 1 -f 0 -o "020_--dng --bpi 1 raw2dng" "$(ls *.MLV | head -1)" && mv "020_--dng --bpi 1 raw2dng000000.dng" "020_--dng --bpi 1 raw2dng.dng" && exiftool "020_--dng --bpi 1 raw2dng.dng" > "020_--dng --bpi 1 raw2dng".txt &
  mlv_dump --dng -b 8 -f 0 -o "021_--dng -b 8" "$(ls *.MLV | head -1)" && mv "021_--dng -b 8000000.dng" "021_--dng -b 8.dng" && exiftool "021_--dng -b 8.dng" > "021_--dng -b 8".txt &
  mlv_dump --dng -b 10 -f 0 -o "022_--dng -b 10" "$(ls *.MLV | head -1)" && mv "022_--dng -b 10000000.dng" "022_--dng -b 10.dng" && exiftool "022_--dng -b 10.dng" > "022_--dng -b 10".txt &
  mlv_dump --dng -b 12 -f 0 -o "023_--dng -b 12" "$(ls *.MLV | head -1)" && mv "023_--dng -b 12000000.dng" "023_--dng -b 12.dng" && exiftool "023_--dng -b 12.dng" > "023_--dng -b 12".txt &
  mlv_dump --dng -b 14 -f 0 -o "024_--dng -b 14" "$(ls *.MLV | head -1)" && mv "024_--dng -b 14000000.dng" "024_--dng -b 14.dng" && exiftool "024_--dng -b 14.dng" > "024_--dng -b 14".txt &
  mlv_dump -c --dng -f 0 -o 025_-c "$(ls *.MLV | head -1)" && mv 025_-c000000.dng 025_-c.dng && exiftool 025_-c.dng > 025_-c.txt &
  mlv_dump --dng --relaxed -f 0 -o "026_--dng --relaxed" "$(ls *.MLV | head -1)" && mv "026_--dng --relaxed000000.dng" "026_--dng --relaxed.dng" && exiftool "026_--dng --relaxed.dng" > "026_--dng --relaxed".txt &
  bl=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/black_level/ { print $2; exit }')
  wl=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/white_level/ { print $2; exit }')
  mlv_dump --dng --black-fix=$bl -f 0 -o "030_--dng --black-fix=$bl" "$(ls *.MLV | head -1)" && mv "030_--dng --black-fix="$bl"000000.dng" "030_--dng --black-fix=$bl.dng" && exiftool "030_--dng --black-fix=$bl.dng" > "030_--dng --black-fix=$bl".txt &
  mlv_dump --dng --white-fix=$wl -f 0 -o "031_--dng --white-fix=$wl" "$(ls *.MLV | head -1)" && mv "031_--dng --white-fix="$wl"000000.dng" "031_--dng --white-fix=$wl.dng" && exiftool "031_--dng --white-fix=$wl.dng" > "031_--dng --white-fix=$wl".txt &
  mlv_dump -a -o "$(ls *.MLV | head -1)dark.MLV" $(ls *.MLV | head -1) && mlv_dump --dng "$(ls *.MLV | head -1)dark.MLV" && mv "$(ls *.MLV | head -1)dark_000000.dng" "027_--dng darkframe(will average all frames into one).dng" && mlv_dump --dng -f 0 -o "028_--dng -s subtract averaged frame" "$(ls *.MLV | head -1)" -s "$(ls *.MLV | head -1)dark.MLV" && mv "028_--dng -s subtract averaged frame000000.dng" "028_--dng -s subtract averaged frame(should look bad).dng" && exiftool "028_--dng -s subtract averaged frame(should look bad).dng" > "028_--dng -s subtract averaged frame(should look bad)".txt && mlv_dump --dng -f 0 -o "029_--dng -t subtract flatfield frame" "$(ls *.MLV | head -1)" -t "$(ls *.MLV | head -1)dark.MLV" && mv "029_--dng -t subtract flatfield frame000000.dng" "029_--dng -t subtract flatfield frame(should look bad).dng" && exiftool "029_--dng -t subtract flatfield frame(should look bad).dng" > "029_--dng -t subtract flatfield frame(should look bad)".txt && rm "$(ls *.MLV | head -1)dark.MLV" && mlv_dump -c -o "032_-c(compress MLV).MLV" "$(ls *.MLV | head -1)" && mlv_dump -d -o "033_-d(decompress MLV).MLV" "032_-c(compress MLV).MLV" &
  mlv_dump -r -o "034_test.RAW" "$(ls *.MLV | head -1)" &

#wait for jobs to end
    wait < <(jobs -p)

  if ls *.wav >/dev/null 2>&1
  then
  rm *.wav
  mlv_dump --dng --no-audio -f 0 -o "035_--dng --no-audio" "$(ls *.MLV | head -1)" && mv "035_--dng --no-audio000000.dng" "035_--dng --no-audio.dng" && exiftool "035_--dng --no-audio.dng" > "035_--dng --no-audio".txt
  if ls *.wav >/dev/null 2>&1
  then
  rm *.wav
  mv "035_--dng --no-audio.dng" "035_--dng --no-audio(not working!).dng" && exiftool "035_--dng --no-audio(not working!).dng" > "035_--dng --no-audio(not working!)"txt
  fi
  else
  audio=$(printf "%s\n" "  --no-audio          for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination ML")
  fi

echo "
  --version           print version information
  --batch             format output message suitable for batch processing
  --save-bpm          save bad pixels to .BPM file
  --show-progress     show DNG file creation progress. ignored when -v or --batch is specified
$audio

-- MLV output --
  -b bits             convert image data to given bit depth per channel (1-16)
  -z bits             zero the lowest bits, so we have only specified number of bits containing data (1-16) (improves compression rate)
  -f frames           frames to save. e.g. '12' saves frames 0 to 12, '12-40' saves frames 12 to 40. forces --no-audio switch
  -A fpsx1000         Alter the video file's FPS metadata
  -x                  build xref file (indexing)

-- MLV autopsy --
  --skip-block <block#>        skip given block number, as if it wasn't present
  --skip-type <type>           skip given block type (e.g. VIDF, AUDF, etc), as if they weren't present
  --extract <block#>           extract the block at given position into autopsy file
  --extract-type <type>        extract the block type (e.g. VERS, LENS, etc) into autopsy file
  --replace <block#>           replace block with data from given autopsy file; requires --autopsy-file
  --payload-only               extract/replace affect not the whole block, but only payload
  --header-only                extract/replace affect not the whole block, but only header
  --autopsy-file <file>        extract/replace from this file
  --hex                        extract prints the selected data as hexdump on screen
  --ascii                      extract prints the selected data as ASCII on screen (only suitable for VERS and DEBG)
  --visualize                  visualize block types, most likely you want to use --skip-xref along with it

-- MLV manipulation --
  --skip-xref                  skip loading .IDX (XREF) file, read block in the MLV file's order instead of presorted
  -I <mlv_file>                inject data from given MLV file right after MLVI header
  -X type                      extract only block type int output file" > 036_not_tested

########################Check essential metadata#############################
    date=$(mlv_dump -v "$(ls *.MLV | head -1)" | grep 'Date' | head -1 | awk 'FNR == 1 {print $2; exit}')
    bit=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/bits_per_pixel/ { print $2; exit }')
    res=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/Res/ { print $2; exit }')
    iso=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/ISO:/ { print $2; exit }')
    fra=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/FPS/ { print $3; exit }')
    cn=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/Camera Name/ { print $3,$4,$5,$6,$7,$8; exit }')
    frames=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/Frames/ { print $3; exit}')
    rawc=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/sampling/ { print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10; exit}')
    bl=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/black_level/ { print $2; exit }')
    wl=$(mlv_dump -v "$(ls *.MLV | head -1)" | awk '/white_level/ { print $2; exit }')

echo "Checking for basic metadata from your MLV file
" > 037_mlv_metadata
echo "###date###" >> 037_mlv_metadata
echo "$date
" >> 037_mlv_metadata
echo "###bit###" >> 037_mlv_metadata
echo "$bit
" >> 037_mlv_metadata
echo "###resolution###" >> 037_mlv_metadata
echo "$res
" >> 037_mlv_metadata
echo "###iso###" >> 037_mlv_metadata
echo "$iso
" >> 037_mlv_metadata
echo "###FPS###" >> 037_mlv_metadata
echo "$fra
" >> 037_mlv_metadata
echo "###Camera model###" >> 037_mlv_metadata
echo "$cn
" >> 037_mlv_metadata
echo "###frames###" >> 037_mlv_metadata
echo "$frames
" >> 037_mlv_metadata
echo "###rawc###" >> 037_mlv_metadata
echo "$rawc
" >> 037_mlv_metadata
echo "###white level###" >> 037_mlv_metadata
echo "$wl
" >> 037_mlv_metadata
echo "###black level###" >> 037_mlv_metadata
echo "$bl
" >> 037_mlv_metadata
echo "Does all look ok?" >> 037_mlv_metadata

#build dcraw tif previews and exiftool files
  find . -maxdepth 1 -mindepth 1 -name '*.dng' -print0 | xargs -0 -P 8 -n 1 dcraw -T
  mkdir -p 038_tifpreviews_exiftool
  mv *.tiff 038_tifpreviews_exiftool
  mv *.txt 038_tifpreviews_exiftool
  cd 038_tifpreviews_exiftool
  open .
  cd -

EOF
. MLV_test_Script && rm MLV_test_Script
clear
echo "
MLV_test_script has finished and you can now check your files"


Typically the folder content should look like this when finished:

000_-p.dng
001_--dng.dng
002_--dng --no-cs.dng
003_--dng --cs2x2.dng
004_--dng --cs3x3.dng
005_--dng --cs5x5.dng
006_--dng --no-fixfp.dng
007_--dng --no-fixcp.dng
008_--dng --no-fixcp --no-fixfp.dng
009_--dng --fixcp2.dng
010_--dng --no-stripes.dng
011_--dng --force-stripes .dng
012_--dng --is-dualiso.dng
013_--dng --is-croprec.dng
014_--dng --fixpn.dng
015_--dng --deflicker=3000(will look bad).dng
016_--dng --no-bitpack.dng
017_--dng --fpi 0 MLVFS.dng
018_--dng --fpi 1 raw2dng.dng
019_--dng --bpi 0 MLVFS.dng
020_--dng --bpi 1 raw2dng.dng
021_--dng -b 8.dng
022_--dng -b 10.dng
023_--dng -b 12.dng
024_--dng -b 14.dng
025_-c.dng
026_--dng --relaxed.dng
027_--dng darkframe(will average all frames into one).dng
028_--dng -s subtract averaged frame(should look bad).dng
029_--dng -t subtract flatfield frame(should look bad).dng
030_--dng --black-fix=512.dng
031_--dng --white-fix=4050.dng
032_-c(compress MLV).MLV
033_-d(decompress MLV).MLV
034_test.RAW
035_--dng --no-audio.dng
036_not_tested
037_mlv_metadata
038_tifpreviews_exiftool
EOSM_12bit_crop_rec.MLV
mlv_dump

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12290
  • Emergencies only
Re: mlv_dump function checker
« Reply #1 on: February 07, 2018, 04:35:09 PM »
Great stuff, this should run on Jenkins. My previous attempt to do something similar is far from complete:

https://builds.magiclantern.fm/jenkins/view/Utilities/job/mlv_dump_4k_test/

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5919
Re: mlv_dump function checker
« Reply #2 on: February 07, 2018, 05:08:29 PM »
Cool, will try to continue and add ways to check every function in there...

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: mlv_dump function checker
« Reply #3 on: February 07, 2018, 08:37:03 PM »
perfect!
excellent work
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5919
Re: mlv_dump function checker
« Reply #4 on: February 07, 2018, 09:00:26 PM »
Thanks for noticing. Appreciated.

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 782
Re: mlv_dump function checker
« Reply #5 on: February 08, 2018, 08:25:50 AM »
Works great! Could be added exiftool output for DNGs.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5919
Re: mlv_dump function checker
« Reply #6 on: February 08, 2018, 09:55:50 AM »
Added and updated the script in first post! Good idea.
Exiftool txt files will be placed inside this folder along with tiff previews:
038_tifpreviews_exiftool

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 782
Re: mlv_dump function checker
« Reply #7 on: February 08, 2018, 04:45:33 PM »
Thanks for this addition. Works great and fast! :)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3737
Re: mlv_dump function checker
« Reply #8 on: February 08, 2018, 06:39:04 PM »
Speaking of exiftool -- here's a pull request to always grab the latest version when using a feature in the ML build scripts that downloads exiftool along with compiling dcraw and cr2hdr and puts them in a neat package:

https://bitbucket.org/hudson/magic-lantern/pull-requests/899/always-use-the-latest-version-of-exiftool/diff

5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102