Switch for macOS Catalina/Linux (former cr2hdr.app)

[Danne]

I understand. It could be done like this.
MLV folder is generated and both MLV and MOV are moved inside this folder. When processing is done the cleaned proxy is moved to a proxy folder, MOV original goes back to origin place and MLV is moved to A_ORIGINALS.

You could also use menu setting (C) and assign a new output folder. This way both MOV and MLV stay untouched in its origin folder.

[Lars Steenhoff]

Ok custom output folder is good enough for how I will use it (C)

It does make a temp folder originals but it deletes it after, seems like a redundant step?
Thanks

[Danne]

Yes, it's more like a safety step since there could be files wandering in and out of that folder. Hard to keep track of all processrs nowadays . If it's empty when all processing is done it is removed.

[12georgiadis]

Danne, it's working now, thanks. I tried a test in Davinci but the conformation isn't working. The CDNG's timecode is reset to 00:00:00:00 and the .mov and cdng don't have the exact same name (aditionnal count for DNG in the name). The conformation can work if the two sources share same reelname + same timecode. Do you think we can add .mov timecode, add reelname and copy it to CDNG ?

[Danne]

Oh my, it never ends  .
I'll take a look. Stay tuned.

Do you by any chance have any links to workflows which includes linking proxies like you describe? I never do this myself.

Regarding reelname. Please print out an example how your files looks now after processed in Switch and then how they should be named to work with resolve.

[12georgiadis]

Do you by any chance have any links to workflows which includes linking proxies like you describe? I never do this myself.

Regarding reelname. Please print out an example how your files looks now after processed in Switch and then how they should be named to work with resolve.

Ok Danne, it seems more easier than I expected. I have a successfull test connecting a ProRes and CDNG (from Black magic camera). The two clips only need to match the same timecode. No reelname to add, and no problem with DNG source clip name.
My source file from canon proxy mov had a TC : 19:44:38:03
The DNG processed in Switch had a TC : 00:00:00:00

There are two solutions to get a successfull conformation : we can copy the mov proxy TC to the DNG during processing. Or reset TC from canon proxy mov to 00:00:00:00 during the black frame removing process.

To make a conformation test, here is the workflow :
1) one folder containing MLV + mov proxy
2) run switch to get new files w/o black frame
3) open free demo version of FCPX. File => Import proxy file, drag drop in the timeline. Make some cuts, trim and change order of the clip
to make a sort of film edit
4) Go to file => export XML (whatever version)
5) Open resolve 14 (free)
6) import DNG
7)File => import XML
8 when there is the pop up menu = untick "automatically import source clips into media pool"
9) if resolve relinks to the DNG and the timeline is similar to what you did in FCPX, it's matching.
Note : we can then replicate the workflow on any NLE (open source, Premiere, Avid...) using XML. Maybe AAF can work too.
If you need a screencast, I can provide one

[Danne]

Are you actually getting synched MOV files with cdng files in resolve from following those steps described above? If you edit stuff in fcpx and export your xml, how can resolve now where to link?
Maybe a screenrecord will help .
I will be busy today but as soon as I have time I will experiment with this. If anybody else wants to get their hands dirty feel free to fire up exiftool, ffmpeg, exiv2 or whatever needed to modify and get a correct relinking...


You can try to modify first and last time code tag in the cdng sequence to have the same start and ending like in the MOV file. Install exiftool and do for instance this:

Code: [Select]

exiftool "-TimeCodes=17:54:07.22" 953A9793_1_2017-08-27_0001_C0000_000000.dngLast dng:

Code: [Select]

exiftool "-TimeCodes=17:54:20.15" 953A9793_1_2017-08-27_0001_C0000_000306.dngThis is just my example based on wht the MOV is telling my in resolve. Have no idea how it works.

[Danne]

New version.

- Added timecode start of 00:00:00:00 in MOV file to match the dng time code in resolve(needs testing)
- Some refactoring

commit:
https://bitbucket.org/Dannephoto/switch/commits/e926c347e5b7fd72d6cd2d7702e2fc7fedab3d68

All recent MOV file seems to match in and out timecode. All files but the testfiles with the leaves from Lars Steenhoff. Good file to have.

[12georgiadis]

Hi Danne, I just arrived in bangalore so I'll test it on my new footage tonight. Thanks !


Envoyé de mon iPhone en utilisant Tapatalk

[12georgiadis]

Danne, I made successfull test in Resolve! I tried both with the CDNG generated by switch and the CDNG from MLVFS (this one without sound of course). TC + name are matching. So now, we can edit offline with the H264 ML proxy in any NLE, export XML to send in/out cut info and relink in Resolve. In Resolve, you just have to import the DNG and after that to import the XML and untick import media (which import the proxy...).

To save more space and time, if you add an option to only reset the TC of the MOV proxy (without generating CDNG in the same time), we could edit and just transcode the necessary clips in DNG at the end (it's called a "consolidate").

I think it is a solid step forward for indie production !

[Danne]

Cool!
Generate only proxy with corrected tc and cleaned from black frames without developing MLV could be an option in the menu. We still need the corresponding MLV files though to achieve the correct end frame.

[12georgiadis]

Cool!
Generate only proxy with corrected tc and cleaned from black frames without developing MLV could be an option in the menu. We still need the corresponding MLV files though to achieve the correct end frame.

Yes, of course we need MLV. This option Would be Nice.
By the way, is there a tool in switch that can remove hot pixels from h264 proxys ? I also noticed pink hightlight with 10/12 bit


Envoyé de mon iPhone en utilisant Tapatalk

[Danne]

Here´s an example using the delogo filter in ffmpeg.
http://www.madpanic.tv/blog/files/fixing-dead-pixels-with-ffmpeg.html

Using this is probably meaning you need to reencode the file itself and then the much better proxy solution would be to create proxys inside Switch. That will also cure any highlight issues.

[12georgiadis]

Excellent article ! However, as you said, it means reencoding, so it's better to edit straight with H264 proxy with dead pixels or making proxies from MLV in Switch. Thanks anyway !

[Danne]

New version

- added support for creating "cleaned" proxies only without processing of MLV files(multithreaded of course  )
You find the setting in both (m) mlv_dump and (ms) mlv_dump_on_steroids menus at the bottom. If not selected proxy+MLV processing works as before.


- refined black detection with two more thresholds
Very simple:

Code: [Select]

    first_black=$(ffmpeg -i *"$MOV" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.01" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
    if [ x"$first_black" = x ]; then
    first_black=$(ffmpeg -i *"$MOV" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.04" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
    if [ x"$first_black" = x ]; then
    first_black=$(ffmpeg -i *"$MOV" -to $snippet -vf "blackdetect=d=0.1:pix_th=0.08" -an -f null - 2>&1 | grep -o "black_duration:.*" | cut -d ":" -f2)
    fi
    fi
commit:
https://bitbucket.org/Dannephoto/switch/commits/e0a8ad7a67468ebb3f32a6bfd19876be4a4692a5

[12georgiadis]

beautiful ! I'll test it !
I made some average calculations for 10 hours of footage :
rec. 1080p23.976 MLV raw (not comp) + CDNG transcode = 5.6TB av.
rec. 1080p23.976 MLV raw comp + H264 proxys = 1.940TB av.

[Danne]

Nice.
Could you check for this issue? MLV_lite is broken in the middle of the file in later versions of crop_rec_4k. Some metadata missing.
http://www.magiclantern.fm/forum/index.php?topic=19300.msg189583#msg189583

Also if things are actually working, what build are you using? I think it affects longer takes around more than 2 or 3 minutes long.

I found out my version of mlv_dump has been buggy. Fixed it and uploaded a new version.

[12georgiadis]

Tested new version with proxy only mode and it works perfectly.
for storage space : 10hrs footage MLV comp + proxy H264 + proxy H264 black removed = 2.1TB av.
I think it's good to keep original proxy.

[Danne]

Great. Thanks for testing with bulk footage.

I think it's good to keep original proxy.

All originals are kept into A_ORIGINALS right?

[12georgiadis]

Great. Thanks for testing with bulk footage.All originals are kept into A_ORIGINALS right?

Yes, I advice people not to delete content in A_ORIGINALS

[togg]

Doubt: why does the counter go beyong 100% when converting only one file? it seems like 100 represents only one chunck of the whole video, a single file, if this is the case is there a way to have it fixed somehow? not that important just a thing

[Danne]

Doubt: why does the counter go beyong 100% when converting only one file?

Well, the reason for the inaccuracy is actually partly because of your reports a few months ago. We tested two scenarios of which one was slowing down MLV batches several minutes because of the frame accuracy issue.

It´s either:

Code: [Select]

    for FILE in `ls -A1 *.MLV *.mlv | grep -v 'avg_\|ft_' 2>/dev/null`; do
    $mlv_dump -v "$FILE" | awk '/Processed/ { print $2; }' >> /tmp/DUALISO/MLVprogress_bar3
    doneor faster but less accurate(but still pretty accurate)

Code: [Select]

    for FILE in `ls -A1 *.MLV *.mlv | grep -v 'avg_\|ft_' 2>/dev/null`; do
    $mlv_dump -v "$FILE" | awk '/Frames/ { print $3; exit}' >> /tmp/DUALISO/MLVprogress_bar3
    done
Grabbing the "Processed" number requires all files processed before giving you the more accurate frame number. This will severely slow down overall processing. Fetching "Frames" however does not since it´s already in the header metadata. Why the inaccuracy? My guess is that the frame number tag is an estimate depending on size of the MLV file and aspect ratio recorded. Probably also depending on if recorded lossless/10/12/14bit as well. So speed is more important than accuracy in this case imo. Still nice to see that there even was "Frames" tag in there...

[togg]

oooh! right, yea absolutelly. speed is more important.

[Danne]

New version uploaded.

Got word from dfort who´s tested a 30 minute MLV with included proxy MOV file. MOV files are of course cut in 4gb chunks  so I added support for this in Switch. It will cat the files and then run as before.

I noticed that all MLV files are treated individually. Even .M00, .M01 etc so this made the percent counter count way above 100% before since I missed to include the frames for .M00, .M01 etc. This is hopefully fixed. Maybe @Togg could have a go here.

Also added some more codec settings rather than only ProRes output in the ProRes menu.

Commits:
https://bitbucket.org/Dannephoto/switch/commits/59905ecc1368a8ea2a29417397a2c36aaa355b22
https://bitbucket.org/Dannephoto/switch/commits/dc648a19d878b27a59bcabc9911bd8c0ffdea0c6
https://bitbucket.org/Dannephoto/switch/commits/ee3dfa54f36600beb257802706c098c4fdcba9f9

[Danne]

New version.

Been testing refining under the radar. Mostly been feedbacked from dfort. Firstly I fixed proxy cleaning with all kinds of scenarious, short, long, different prefixes and spanned files. Another issue was that the cleaning scripts would consistently cut a few frames too many at the end on longer takes. The solution here seems to be to use -vframes rather than the other math I was doing before. A1ex pointed this out a while ago but I didn´t put this to some real tests until today when feedbacked from dfort.

Commits here:
https://bitbucket.org/Dannephoto/switch/commits/all