crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[Danne]

Tried the bc trickery but seems I´ve been building a different code nest so decided to go for some other roundtrippin´:

Code: [Select]

for i in *.MOV; do
duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i")
if (( $(echo "$duration > 3" |bc -l) )); then
snippet=$(echo 2)
else
snippet=$(echo 3)
fi
first_black=$(ffmpeg -i "$i" -to 3 -vf "blackdetect=d=0.5:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
near_end=$(echo $duration - $snippet | bc -l)
last_black=$(ffmpeg -ss $near_end -i "$i" -vf "blackdetect=d=0.1:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
if ! [ x"$last_black" = x ]; then
last_black=$(echo $duration - $last_black | bc -l)
fi
trimmed=$(echo $last_black - $first_black - 0.05 | bc -l)
if ! [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
elif ! [ x"$first_black" = x ] && [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $duration -vcodec copy -acodec copy new_"$i"
elif [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
fi
done
Too tired to explain stuff right now. Cutting the last frames with - 0.05 seems closer to perfect then - 0.01 from where I´m sitting.
Great script snippets today. Learnt a lot.

[Danne]

Did some minor changes which now seems to yield perfect cuts at start and ending in the script. Since we now how to delimit reencoding by using -to I tried the reverse filter again to achieve blackframe detection of the ending part.

this part:

Code: [Select]

near_end=$(echo $duration - $snippet | bc -l)
last_black=$(ffmpeg -ss $near_end -i "$i" -vf "blackdetect=d=0.1:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)becomes this:

Code: [Select]

last_black=$(ffmpeg -i "$i" $snippet -vf "reverse,""blackdetect=d=0.1:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)trimming section back to 0.01 from 0.05:

Code: [Select]

trimmed=$(echo $last_black - $first_black - 0.01 | bc -l)
Full script:(updated)

Code: [Select]

for i in *.MOV; do
duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i")
if (( $(echo "$duration < 5" |bc -l) )); then
snippet=$(echo 2)
else
snippet=$(echo 5)
fi
first_black=$(ffmpeg -i "$i" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
last_black=$(ffmpeg -i "$i" -to $snippet -vf "reverse,""blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
if ! [ x"$last_black" = x ]; then
last_black=$(echo $duration - $last_black | bc -l)
fi
trimmed=$(echo $last_black - $first_black - 0.01 | bc -l)
if ! [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
elif ! [ x"$first_black" = x ] && [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $duration -vcodec copy -acodec copy new_"$i"
elif [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
fi
done

[Lars Steenhoff]

https://www.dropbox.com/s/pjebqj2yf2bdf3t/YORAN800_S001_S001_T009.MOV?dl=0

I'm trying to run the script to remove the black frames and installed ffmpeg and it seems to process the file,
but the quicktime does not change.

I have attached the input quicktime file

Is there something that I need to change?

Code: [Select]

thunderbooksmbp:black thunderbook$ ls
YORAN800_S001_S001_T009.MOV
thunderbooksmbp:black thunderbook$ for i in *.MOV; do
> duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i")
> if (( $(echo "$duration > 3" |bc -l) )); then
> snippet=$(echo 2)
> else
> snippet=$(echo 3)
> fi
> first_black=$(ffmpeg -i "$i" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
> last_black=$(ffmpeg -i "$i" -to $snippet -vf "reverse,""blackdetect=d=0.1:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
> if ! [ x"$last_black" = x ]; then
> last_black=$(echo $duration - $last_black | bc -l)
> fi
> trimmed=$(echo $last_black - $first_black - 0.01 | bc -l)
> if ! [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
> ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
> elif ! [ x"$first_black" = x ] && [ x"$last_black" = x ]; then
> ffmpeg -ss $first_black -i "$i" -t $duration -vcodec copy -acodec copy new_"$i"
> elif [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
> ffmpeg -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
> fi
> done
thunderbooksmbp:black thunderbook$

[Danne]

You are probably missing ffprobe. Easiest install for mac is to do this in terminal:

Code: [Select]

brew install ffmpegWill install ffmpeg, ffplay, ffprobe.
If already installed try:

Code: [Select]

brew uninstall ffmpegthen

Code: [Select]

brew install ffmpegagain.

Here is homebrew by the way.
https://brew.sh/index_se.html


*Update. Just tried your file and indeed, it doesn´t process, go figure!
Will check further. Stay tuned.

This isn´t an original canon H.264 MOV file but prores file right? Think you want to run original H264 files for this workflow. I also noted that your black sequence is pretty long at the end. Probably more than three seconds, is that right?

[Danne]

@Lars Steenhoff
Tested some more and I managed to clean your prores file as well. I had to change threshold settings and also black length estimation prolonged to 5 seconds from earlier 3. Now it needs to be tested on darker footage so it doesn´t break elsewhere. Seemed to work on my test files over here. Corrected an error as well.

New script.

Code: [Select]

for i in *.MOV; do
duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i")
if (( $(echo "$duration < 5" |bc -l) )); then
snippet=$(echo 2)
else
snippet=$(echo 5)
fi
first_black=$(ffmpeg -i "$i" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
last_black=$(ffmpeg -i "$i" -to $snippet -vf "reverse,""blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
if ! [ x"$last_black" = x ]; then
last_black=$(echo $duration - $last_black | bc -l)
fi
trimmed=$(echo $last_black - $first_black - 0.01 | bc -l)
if ! [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
elif ! [ x"$first_black" = x ] && [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $duration -vcodec copy -acodec copy new_"$i"
elif [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
fi
done

[Lars Steenhoff]

Thanks, I installed the homebrew and ffprobe.

the reason why this is a prores file, is that I recorded it with and external recorder triggered by the hdmi record trigger from h264.  I did not remember when I upload it was the pores version.

Actually I aslo have the internal camera file, it has the same length of black file.
https://www.dropbox.com/s/s8xzqghjjaddnod/953A9793.MOV?dl=0

The new script cleaned up both of them well 

EDIT:  it seems the h264 still has some black part at the end.

All-I vs IPB
I had the camera on IPB I don't know if thats affects anything for the black frames?
I mean how the file can be cut of, or this does not matter?

Code: [Select]

thunderbooksmbp:black thunderbook$ for i in *.MOV; do duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i"); if (( $(echo "$duration > 3" |bc -l) )); then snippet=$(echo 2); else snippet=$(echo 3); fi; first_black=$(ffmpeg -i "$i" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2); last_black=$(ffmpeg -i "$i" -to $snippet -vf "reverse,""blackdetect=d=0.1:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2); if ! [ x"$last_black" = x ]; then last_black=$(echo $duration - $last_black | bc -l); fi; trimmed=$(echo $last_black - $first_black - 0.01 | bc -l); if ! [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"; elif ! [ x"$first_black" = x ] && [ x"$last_black" = x ]; then ffmpeg -ss $first_black -i "$i" -t $duration -vcodec copy -acodec copy new_"$i"; elif [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then ffmpeg -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"; fi; done
thunderbooksmbp:black thunderbook$ for i in *.MOV; do
> duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i")
> if (( $(echo "$duration < 5" |bc -l) )); then
> snippet=$(echo 2)
> else
> snippet=$(echo 5)
> fi
> first_black=$(ffmpeg -i "$i" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
> last_black=$(ffmpeg -i "$i" -to $snippet -vf "reverse,""blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
> if ! [ x"$last_black" = x ]; then
> last_black=$(echo $duration - $last_black | bc -l)
> fi
> trimmed=$(echo $last_black - $first_black - 0.01 | bc -l)
> if ! [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
> ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
> elif ! [ x"$first_black" = x ] && [ x"$last_black" = x ]; then
> ffmpeg -ss $first_black -i "$i" -t $duration -vcodec copy -acodec copy new_"$i"
> elif [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
> ffmpeg -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
> fi
> done
ffmpeg version 3.3.3 Copyright (c) 2000-2017 the FFmpeg developers
  built with Apple LLVM version 8.1.0 (clang-802.0.41)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.3.3 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-frei0r --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-opencl --enable-videotoolbox --disable-lzma --enable-libopenjpeg --disable-decoder=jpeg2000 --extra-cflags=-I/usr/local/Cellar/openjpeg/2.2.0/include/openjpeg-2.2 --enable-nonfree --enable-vda
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '953A9793.MOV':
  Metadata:
    major_brand     : qt 
    minor_version   : 537331968
    compatible_brands: qt  CAEP
    com.apple.quicktime.make: Canon
    com.apple.quicktime.model: Canon EOS 5D Mark III
    com.apple.quicktime.rating.user: 0.000000
    creation_time   : 2017-08-27T16:44:02.000000Z
  Duration: 00:00:17.48, start: 0.000000, bitrate: 27129 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 25531 kb/s, 25 fps, 25 tbr, 25k tbn, 50k tbc (default)
    Metadata:
      creation_time   : 2017-08-27T16:44:02.000000Z
      timecode        : 17:54:07:22
    Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2017-08-27T16:44:02.000000Z
      timecode        : 17:54:07:22
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2017-08-27T16:44:02.000000Z
      timecode        : 17:54:07:22
Output #0, mov, to 'new_953A9793.MOV':
  Metadata:
    major_brand     : qt 
    minor_version   : 537331968
    compatible_brands: qt  CAEP
    com.apple.quicktime.make: Canon
    com.apple.quicktime.model: Canon EOS 5D Mark III
    com.apple.quicktime.rating.user: 0.000000
    encoder         : Lavf57.71.100
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 25531 kb/s, 25 fps, 25 tbr, 25k tbn, 25k tbc (default)
    Metadata:
      creation_time   : 2017-08-27T16:44:02.000000Z
      timecode        : 17:54:07:22
    Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2017-08-27T16:44:02.000000Z
      timecode        : 17:54:07:22
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  330 fps=0.0 q=-1.0 Lsize=   48488kB time=00:00:12.77 bitrate=31090.9kbits/s speed= 163x   
video:46004kB audio:2476kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.017650%
ffmpeg version 3.3.3 Copyright (c) 2000-2017 the FFmpeg developers
  built with Apple LLVM version 8.1.0 (clang-802.0.41)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.3.3 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-frei0r --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-opencl --enable-videotoolbox --disable-lzma --enable-libopenjpeg --disable-decoder=jpeg2000 --extra-cflags=-I/usr/local/Cellar/openjpeg/2.2.0/include/openjpeg-2.2 --enable-nonfree --enable-vda
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f8936003400] overread end of atom '?mak' by 2 bytes
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f8936003400] overread end of atom '?swr' by 3 bytes
Guessed Channel Layout for Input Stream #0.0 : 4.0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'YORAN800_S001_S001_T009.MOV':
  Metadata:
    major_brand     : qt 
    minor_version   : 537199360
    compatible_brands: qt 
    creation_time   : 2017-08-27T17:30:00.000000Z
    make            : Atomos
    make-eng        : Atomos
    encoder         : NinjaBld - 5.2   
    encoder-eng     : NinjaBld - 5.2   
    timecode        : 17:54:07:24
  Duration: 00:00:17.20, start: 0.000000, bitrate: 156029 kb/s
    Stream #0:0(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, 4.0, s16, 3072 kb/s (default)
    Metadata:
      creation_time   : 2017-08-27T17:30:00.000000Z
    Stream #0:1(eng): Video: prores (apch / 0x68637061), yuv422p10le(bt709, top coded first (swapped)), 1920x1080, 152908 kb/s, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 2500 tbn, 2500 tbc (default)
    Metadata:
      creation_time   : 2017-08-27T17:30:00.000000Z
      encoder         : Apple ProRes 422 (HQ)
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2017-08-27T17:30:00.000000Z
      timecode        : 17:54:07:24
Output #0, mov, to 'new_YORAN800_S001_S001_T009.MOV':
  Metadata:
    major_brand     : qt 
    minor_version   : 537199360
    compatible_brands: qt 
    timecode        : 17:54:07:24
    make            : Atomos
    make-eng        : Atomos
    encoder         : Lavf57.71.100
    Stream #0:0(eng): Video: prores (apch / 0x68637061), yuv422p10le(bt709, top coded first (swapped)), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 152908 kb/s, 25 fps, 25 tbr, 10k tbn, 2500 tbc (default)
    Metadata:
      creation_time   : 2017-08-27T17:30:00.000000Z
      encoder         : Apple ProRes 422 (HQ)
    Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, 4.0, s16, 3072 kb/s (default)
    Metadata:
      creation_time   : 2017-08-27T17:30:00.000000Z
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  318 fps=0.0 q=-1.0 Lsize=  236864kB time=00:00:12.69 bitrate=152834.9kbits/s speed=52.2x   
video:232082kB audio:4776kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.002831%

[Lars Steenhoff]

Code: [Select]

ffmpeg -i INPUT.MOV -c copy -map 0:a OUTPUT.WAV
Next I try to extract the wav file, and want to change the script to process all the .mov files in the folder and make the wav file with the corresponding names.

to allow Davinci Resolve to import the audio and raw file sync up.

Here is the corresponding MLV for testing
https://www.dropbox.com/s/hhnn68a24mer6q6/953A9793.MLV?dl=0

And all the files together
https://www.dropbox.com/sh/a3a5wtyppgddnt4/AABh0Snrmb3fJC4T4CqrvTVra?dl=0

[Danne]

I will put davinci resolve compliance and automation workflows in Switch eventually. For dng folder embedding you´ll need some additional wav metadata. You can inject the data with bwfmetaedit.
You can check row 190 and onwards in here:
https://bitbucket.org/Dannephoto/switch/src/fa38c2375f43b051bf54268e3912dd6c946321a9/source_code/mlv_dump_01.txt?at=default&fileviewer=file-view-default

If you want to get the wav files from the script you can run this right away next to your MOV files:

Code: [Select]

for i in *.MOV; do
duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i")
if (( $(echo "$duration < 5" |bc -l) )); then
snippet=$(echo 2)
else
snippet=$(echo 5)
fi
first_black=$(ffmpeg -i "$i" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
last_black=$(ffmpeg -i "$i" -to $snippet -vf "reverse,""blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
if ! [ x"$last_black" = x ]; then
last_black=$(echo $duration - $last_black | bc -l)
fi
trimmed=$(echo $last_black - $first_black - 0.01 | bc -l)
if ! [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
ffmpeg -i new_"$i" -c copy -map 0:a new_$(echo "$i" | cut -d "." -f1).WAV
elif ! [ x"$first_black" = x ] && [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $duration -vcodec copy -acodec copy new_"$i"
ffmpeg -i new_"$i" -c copy -map 0:a new_$(echo "$i" | cut -d "." -f1).WAV
elif [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
ffmpeg -i new_"$i" -c copy -map 0:a new_$(echo "$i" | cut -d "." -f1).WAV
fi
done

EDIT:  it seems the h264 still has some black part at the end.

will check.
Seems there is a pure white frame at the end of the newly created file. Not sure what´s going on.

[Lars Steenhoff]

I'm not sure if its something todo with All-I vs IPB
I had the camera on IPB I don't know if thats affects anything for the black frames.

Because the prores does it well with the same settings. ( prores is All-I )
and because we don't recompress the .MOV file maybe there are only certain frames that are allowed to be cut. 
Purely speculating.

[Danne]

Actually, I think it´s the floating point numbers. FFmpeg can´t cut the files with that precision on the frames so it does some crude cutting instead(Now this is speculation) However. Here´s a version which simply shaves off the the floating point leaving us with integer values. Didn´t test very much but it cuts you example H.264 file. Probably would be better if we could round numbers.
By the way. I actually managed to trick a dark shot for blackframe with the latest changes that worked on your prores file.
-vf "blackdetect=d=0.1:pix_th=0.09"
If you´re working with in cam MOV files I suggest you set the script to:
-vf "blackdetect=d=0.1:pix_th=0.01" in both places in the script.

*erased* not working.

[Lars Steenhoff]

This new scripts cuts off a bit too much, it also cuts the footage in the end.

I put them all in top of each other in resolve and manually cut the original black frames to see the difference

 

[Danne]

Thanks for pm with files. I guess it will be pretty hard to match exactly all the time, especially the ending part. I see to it that wav will match and work in Switch. Easier to cut wav when you extracted the amount of dng files. Thanks for testing and feedback.

[Danne]

If anybody wants to run MOV proxy/wav automation in Switch I just released a version which supports it.
http://www.magiclantern.fm/forum/index.php?topic=15108.msg189314#msg189314

[Lars Steenhoff]

cool, I will check it soon !

[a1ex]

Found a way to allocate the entire SRM memory without locking the exposure controls... sort of:

- When allocating the entire SRM memory (normally used for still photos, for holding uncompressed 14-bit raw data), Canon code locks the shutter button and shows the BUSY screen (to prevent you from taking any more pictures until some buffer becomes free again).
- Unfortunately, they also lock the exposure controls after you press the shutter halfway.
- I've tried a bunch of tricks, such as calling their unlock routines used in this allocator (ClearBusy, RealClearBusy); they did unlock the GUI, but pressing the shutter fully (by mistake or for whatever reason) gave ERR80. Unlocking the GUI for a short time and re-locking it again did not re-activate the exposure controls.
- Low-level note: RealSetBusy/RealClearBusy only changes a GPIO; the MPU does react to this change.
- There is another way to lock the GUI - using PROP_ICU_UILOCK (our wrapper is gui_uilock). That one lets you lock only the shutter button (UILOCK_SHUTTER), but does not lock the exposure controls.
- Guess what: having both locks enabled (UILOCK_SHUTTER and Canon's SRM lock) works and DOES NOT LOCK THE EXPOSURE CONTROLS!!!

Therefore, one possibility would be to avoid the UILOCK_NONE state as long as the entire SRM memory is allocated.

However, this has another issue: half-shutter events are no longer present. Not sure how to solve it...

The MPU does send a message that tells the main CPU to show the BUSY screen. Maybe I should just interpret that as half-shutter press. I believe the message is some part of:

Code: [Select]

45B7B> **INT-36h*:000afe84:00:00: *** mpu_recv(26 24 0a 08 6f 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 01 01 00 00 00 00 00 00), from ff2e87f8
CDA4B> **INT-36h*:000afe84:00:00: *** mpu_recv(26 24 0a 08 6f 00 01 00 01 01 a0 10 00 68 01 01 50 20 30 01 01 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00), from ff2e87f8

Good luck decoding that, though...

[Danne]

Fixing the busy screen. How cool is that. Seems a deep mystery...

If anyone is still interested there is a way to get rid of that last black frame by doing multiple runs on stubborn files. By putting in a while loop we can keep recopy the file until it chops the very last blackframe out.
Some talk about drop frames and black frames over here.
https://video.stackexchange.com/questions/10383/remove-black-frames-from-vob-files-with-ffmpeg

Code: [Select]

for i in *.MOV; do
duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i")
if (( $(echo "$duration < 5" |bc -l) )); then
snippet=$(echo 2)
else
snippet=$(echo 5)
fi
first_black=$(ffmpeg -i "$i" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
last_black=$(ffmpeg -i "$i" -to $snippet -vf "reverse,""blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
if ! [ x"$last_black" = x ]; then
last_black=$(echo $duration - $last_black | bc -l)
fi
trimmed=$(echo $last_black - $first_black - 0.01 | bc -l)
if ! [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
ffmpeg -i new_"$i" -c copy -map 0:a new_$(echo "$i" | cut -d "." -f1).WAV
elif ! [ x"$first_black" = x ] && [ x"$last_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $duration -vcodec copy -acodec copy new_"$i"
ffmpeg -i new_"$i" -c copy -map 0:a new_$(echo "$i" | cut -d "." -f1).WAV
elif [ x"$first_black" = x ] && ! [ x"$last_black" = x ]; then
ffmpeg -ss 0 -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
ffmpeg -i new_"$i" -c copy -map 0:a new_$(echo "$i" | cut -d "." -f1).WAV
fi
#rerun ending part until we´re satisfied
while ! [ x"$last_black" = x ]; do
duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i new_"$i")
last_black=$(ffmpeg -i new_"$i" -to $snippet -vf "reverse,""blackdetect=d=0.01:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
if ! [ x"$last_black" = x ]; then
trimmed=$(echo $duration - $last_black | bc -l)
ffmpeg -ss 0 -i new_"$i" -t $trimmed -vcodec copy -acodec copy new_new_"$i"
rm new_"$i"
mv new_new_"$i" new_"$i"
fi
done
done

[a1ex]

Better: count the frames in the "sidecar" MLV and trim the MOV to the same number of frames.

[12georgiadis]

Hello,
When I use the proxy mode, H264 files get hot pixels. Is there a tool to remove it on H264 files ? By the way, when using 10/12 compressed mode, I get pink zone in the highlight. Is it removable ? the goal is not to correct it for master file but to have a clean proxy for offline editing.

[denishogan]

Hi All

I have been using Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter) - continuous*) at 5 FPS! with success to capture still macro.

Given the low frame rate (5fps) has anyone used this , if so what is the recommend shutter speed for such slow rate under ML for a stationary subject/object. I see that "they" state to double the shutter speed of the FPS. which means if its 5fps then my speed should be 1/10. Given the subject is stationery and there is no movement is there any advantage of increasing the shutter speed. I have done some tests and cannot tell the difference other then loss of light.

cheers
Denis

[Danne]

Better: count the frames in the "sidecar" MLV and trim the MOV to the same number of frames.

This is what I do with the wav file. Seems the way to go for proxy files as well. Just tested and verified a perfect cut from the problematic file from Lars Steenhoff. And it´s fast. Since first cut always seems correctly detected, ending part can be calculated via mlv_dump and the MLV file.

Needs later version of mlv_dump to work and the corresponding MLV next to the proxy file.

Code: [Select]

for i in *.MOV; do
MLV=$(echo "$i" | cut -d "." -f1 | tail -c 5).MLV
duration=$(ffprobe -loglevel quiet -of compact=p=0:nk=1 -show_entries format=duration -i "$i")
if (( $(echo "$duration < 5" |bc -l) )); then
snippet=$(echo 2)
else
snippet=$(echo 5)
fi
first_black=$(ffmpeg -i "$i" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.09" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
#grab amount of dng frames from MLV metadata
frct=$(mlv_dump *"$MLV" | awk '/Processed/ { print $2; }')
FPS=$(mlv_dump *"$MLV" | awk '/Processed/ { print $6; }')     
trimmed=$(echo $frct/$FPS | bc -l | awk 'FNR == 1 {print}')
if ! [ x"$first_black" = x ]; then
ffmpeg -ss $first_black -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
elif [ x"$first_black" = x ]; then
ffmpeg -ss 0 -i "$i" -t $trimmed -vcodec copy -acodec copy new_"$i"
fi
done
The trustworthy number is to grab:

Code: [Select]

frct=$(mlv_dump *"$MLV" | awk '/Processed/ { print $2; }')Grabbing the Processed number is a bit slow on bigger MLV files since it will process the whole file before giving you the information. There is a Frames Video: metadata tag which could be used but it´s never exact so not useful here.
I will rewrite the processing in Switch maybe tonight...

[a1ex]

You can specify -vframes to avoid doing the (possibly inexact) FPS math.

[Danne]

-vframes, never heard of. Is there any difference running -vframes from only -v? Speed increase?
My understanding is that this:

Code: [Select]

frct=$(mlv_dump *"$MLV" | awk '/Processed/ { print $2; }')
FPS=$(mlv_dump *"$MLV" | awk '/Processed/ { print $6; }') this:

Code: [Select]

frct=$(mlv_dump -vframes *"$MLV" | awk '/Processed/ { print $2; }')
FPS=$(mlv_dump -vframes *"$MLV" | awk '/Processed/ { print $6; }') and this:

Code: [Select]

frct=$(mlv_dump -v *"$MLV" | awk '/Processed/ { print $2; }')
FPS=$(mlv_dump -v *"$MLV" | awk '/Processed/ { print $6; }') Will do exactly the same thing?

[a1ex]

My bad, it's -frames:v. Things have changed since I've last read the manual

[Danne]

hehe.
Let´s see.
I get this:

Code: [Select]

dans-MacBook-Pro:~ dan$ mlv_dump -frames:v /Users/dan/Desktop/MLV_files/h264proxy/953A9793.MLV

 MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: '/Users/dan/Desktop/MLV_files/h264proxy/953A9793.MLV'
   - Verify file structure
File /Users/dan/Desktop/MLV_files/h264proxy/953A9793.MLV opened
File /Users/dan/Desktop/MLV_files/h264proxy/953A9793.M00 not existing.
Processing...

Reached end of chunk 1/1 after 328 blocks
Processed 315 video frames at 25.00 FPS (12.60 s)
Done
And exactly the same without:

Code: [Select]

dans-MacBook-Pro:~ dan$ mlv_dump /Users/dan/Desktop/MLV_files/h264proxy/953A9793.MLV

 MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: '/Users/dan/Desktop/MLV_files/h264proxy/953A9793.MLV'
   - Verify file structure
File /Users/dan/Desktop/MLV_files/h264proxy/953A9793.MLV opened
File /Users/dan/Desktop/MLV_files/h264proxy/953A9793.M00 not existing.
Processing...

Reached end of chunk 1/1 after 328 blocks
Processed 315 video frames at 25.00 FPS (12.60 s)
Done
Default -frames:v setting when nothing selected? How to verify?

[a1ex]

It's not an undocumented option (hint).