Switch for macOS Catalina/Linux (former cr2hdr.app)

[ErwinH]

Thank you!

I was running the process on a Macbook Pro 13" 2015 i5. I know it's not the fastest, but it's workable.

The MLV is 45.76GB large and contains 31345 frames and it takes 50 minutes to extract the dng files from the MLV.  The process of creating a prores movie takes ages Maybe you can add the running times to the log.txt file?

The SSH part, is simple running the command from a console, terminal, like pressing ctrl+alt+f1 on a linux box. If it's possible to start the run command for a task from the command prompt would be very helpful. I am aware of the fact that multithreaded might be an issue.

It would also be helpful to change the number of parallel tasks, because working with MLV's that are 20 minutes long takes up a lot of disk space. 4x100 GB of temp data is a lot!

[Danne]

The MLV is 45.76GB large and contains 31345 frames and it takes 50 minutes to extract the dng files from the MLV.  The process of creating a prores movie takes ages Maybe you can add the running times to the log.txt file?

I have total running time in the notifier window which should pop up when done. Are you thinking some estimating time in LOG.txt file or a revealing the frame by frame process in ffmpeg? I intentionally neglected this to not let the LOG grow too big.
About speed, I think the bottleneck is the pipe through dcraw. Higher quality is demanding. Might be some alternate prores codec which could help speed as well but probably will create bigger files. Could check into this.

The SSH part, is simple running the command from a console, terminal, like pressing ctrl+alt+f1 on a linux box. If it's possible to start the run command for a task from the command prompt would be very helpful. I am aware of the fact that multithreaded might be an issue.

It could be possible to start Switch outside automator by double clicking a command script. The Switch_MAIN_DBG.command is more or less doing this but it needs paths which now are established through automator.

It would also be helpful to change the number of parallel tasks, because working with MLV's that are 20 minutes long takes up a lot of disk space. 4x100 GB of temp data is a lot!

I could add a switch which would let the user choose number of processes manually. On a side note you can assign a different output location if you want to export to let´s say an external drive. Do this with (O) setting in main menu.
By the way. Are we talkning about MLV to dng processing or the dng to prores processing?

[ErwinH]

I have total running time in the notifier window which should pop up when done.

This summary would be nice to have in the LOG.txt file. You could add the runtime of a task after that task is done.

I am using external drives, because the internal drive is way to small on my MBPr. 256GB isn't that much once you start doing video, but still I think it's a good thing to be able to alter the amount of processes based on preferences, other tasks, etc.

MLV is 43GB.
DNG's are 88GB
Prores is 35GB

The Prores image is ok, but waiting 11 hours for it to build is unpractical and it's more convenient to use the DNG files instead of the Prores file.

[Danne]

Just finished some tests and there are alternatives which will speed up things considerably. I ran 101 frames in a dng folder and it goes like this:

Based on running 101 frames

With the settings right now highest FFmpeg quality and dcraw -q 3:
41 sec dcraw -q 3 FFmpeg -vcodec prores_ks -pix_fmt yuv444p10

Altering -q setting in dcraw
27 sec dcraw -q 0 FFmpeg -vcodec prores_ks -pix_fmt yuv444p10

New prores codec(422)
28 sec dcraw -q 3 FFmpeg -vcodec prores -pix_fmt yuv422p10le
14 sec dcraw -q 0 FFmpeg -vcodec prores -pix_fmt yuv422p10le

Pick your poison .

In real life I think it´s actually pretty hard to see any difference here. Hard to say how color is affected with dcraw -q alterations. Anyhow. If there is any interest and some use for prores I could smack in some speed up alternatives in the prores menu.

[ErwinH]

Great! Thanks for looking for solutions! I even like the 422 codec a lot! 444 is way overkill for a lot of use cases. Mine is one of them.

If you want I'll give it a go on bigger files.

[Danne]

Just pushed fixes for
 
- Specifying thread amount
- Added total time to LOG.txt

I´ll put in the alternative codec and then you might wanna check quality and speed comparison for bigger files. Stay tuned...

[Danne]

Ok, here we go. New upload in first post. Following changes suggested from ErwinH added:

- Specify thread amount
- Add total time to LOG.txt
- Switch to much faster prores422 codec(less than half processing time)

Main menu (CP) for changing thread processing


In ProRes menu (07) to switch codec


Note that I also switch from -q 3 to -q 0 in dcraw when selecting new codec. Do check the final output and compare.

[ErwinH]

5758 frames of 1360x764:

Using codec yuv422p10le

Code: [Select]

Total running time:   
0 Days, 00 Hrs, 55 Min, 40 Sec
X to FFmpeg ProRes:   
XxXx
mlv_dump:   
0 Days, 00 Hrs, 08 Min, 24 Sec
raw2dng:   
XxXx
dcraw_FFmpeg ProRes:   
0 Days, 00 Hrs, 47 Min, 11 Sec
cr2hdr(CR2 files):   
XxXx

Still quite a lot, 20-30 seconds of work for every second of footage.

This is with the old version (with a small change)

[Danne]

Try the latest version. I changed to prores instead of prores_ks and altered -q setting in dcraw which makes all the difference.

[ErwinH]

Can you also change the output directory to ProRes422 instead of ProRes4444? So you can keep different versions apart?

1522 frames 1:01

Code: [Select]

Total running time:   
0 Days, 00 Hrs, 04 Min, 11 Sec
X to FFmpeg ProRes:   
XxXx
mlv_dump:   
0 Days, 00 Hrs, 01 Min, 22 Sec
raw2dng:   
XxXx
dcraw_FFmpeg ProRes:   
0 Days, 00 Hrs, 02 Min, 44 Sec
cr2hdr(CR2 files):   
XxXx
That's a nice improvement! This way it's a lot easier to use the footage.

Can you also remove the DNG files after the run is complete?

[Danne]

Sure. Just check the quality and speed increase of the 422 files and if all is ok I do the folder change as well.

[ErwinH]

Quality check isn't my cup of tea. But the speed improvement is huge! 4:1 instead of 20:1.

[Danne]

Cool. Well, I did a sloppy check before but cannot spot any difference. I'll add the 422 folder in an hour or so. Thanks for feedback and ideas.

[ErwinH]

Can you also add something to clean up the temp dng files, so larger queue's don't need as much space as needed now.

Another improvement would be to finish one file at the time. First extract the dng's from the first file and convert it to ProRes, remove the dng's and then move on to the next file.

[Danne]

I´ll see if I could do deletion of dng folders possible.
Waiting for files before moving to the next is not a priority. More important to limit parallell processing and to keep the flow going for speed reasons. In the end files will all be sorted.

Isn´t the "set thread manually" to 1 doing exactly what you want? Processing one file at the time?

[Danne]

New version uploaded.
Added a switch for deletion of dng folders after ProRes exports. It´s at the bottom of th ProRes menu.

Another improvement would be to finish one file at the time. First extract the dng's from the first file and convert it to ProRes, remove the dng's and then move on to the next file.

Setting the threads to 1 in main menu (CP) and setting the (X) switch in ProRes menu would probably do exactly what you ask for.

Regarding deletion of dng folders. The switch can be set and run from within the ProRes menu. As soon as you leave the ProRes menu this switch will be reset. Reason for this is that there is a good chance you´ll forget it next time you start switch and then the damage is done.

For complexity reasons I´ll probably won´t add a prores422 folder. All files will end up in the ProRes4444 folder instead.

[DeafEyeJedi]

Nice updates, @Danne!

...As soon as you leave the ProRes menu this switch will be reset. Reason for this is that there is a good chance you´ll forget it next time you start switch and then the damage is done.

+1

[ErwinH]

Isn´t the "set thread manually" to 1 doing exactly what you want? Processing one file at the time?

I think I messed up a setting skipping the conversion to ProRes, which gave the impression it was doing the MLV_dump first for every file, and after that doing the conversion to ProRes.

As soon as you leave the ProRes menu this switch will be reset. Reason for this is that there is a good chance you´ll forget it next time you start switch and then the damage is done.

-1

What is the use for the DNG files after you have converted them to ProRes. You can quite quickly extract them from the MLV and they just take up space.

[Danne]

There could be cases when one saves and works strictly from folders with dng files. Not good practice to set permanent deletion flags hidden in submenus. If you really want to keep the flag set you can simply erase following lines inside the Menu.command:

Code: [Select]

#remove destructive switch
if [ -f /tmp/DUALISO/DEL_DNG ]
then
rm /tmp/DUALISO/DEL_DNG
fi
Another tip. You could build and create prores without the need to convert the MLV files by running MLVFS from inside the main menu (ml). Once in the menu select (A) and MLV files will be turned into dng folders. All MLV files will be placed in A_ORIGINALS folder inside two subfolders. Now you can move into (p) prores folder and start processing prores files. Once don open up Switch and go back to (ml) and:
you can then get back your MLV files via menu by doing (EJ) once you´re done or manually dig them out from the subfolder in A_ORIGINALS.
It will not be as fast as with already processed physical dng files but maybe fast enough.

[ErwinH]

MLVFS works, and saves a lot of space, but the development of a 2 minute clip takes 45 minutes.

dcraw / ffmpeg runs at 4-4.5 fps at the moment, on a slow virtual machine running Ubuntu Xenial, using extracted dng's. Storage is a samba mounted network storage, so that's not in it's advantage either.

My MBPr, using external USB3.0 SSD storage runs at approximately 3-3.5 fps using MLVFS.

More testing results:
Profile 2?
Debayer 0

Code: [Select]

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 0 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores -pix_fmt yuv422p10le -n -r 25 M23-1452.mov
frame=  100 fps=4.4 q=-0.0 Lsize=   51746kB time=00:00:04.00 bitrate=105975.6kbits/s   

Debayer 3

Code: [Select]

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores -pix_fmt yuv422p10le -n -r 25 -profile:v 2 M23-1452_profile2_q3.mov
frame=  100 fps=2.4 q=-0.0 Lsize=   48464kB time=00:00:04.00 bitrate=99255.2kbits/s   

Quality 3 (Hq)
Debayer 0

Code: [Select]

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 0 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores -pix_fmt yuv422p10le -n -r 25 -profile:v 3 M23-1452_profile3.mov
frame=  100 fps=4.4 q=-0.0 Lsize=   77486kB time=00:00:04.00 bitrate=158690.4kbits/s   
Debayer 3

Code: [Select]

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores -pix_fmt yuv422p10le -n -r 25 -profile:v 3 M23-1452_profile3_q3.mov
frame=  100 fps=2.4 q=-0.0 Lsize=   70703kB time=00:00:04.00 bitrate=144800.6kbits/s   
Prores_ks
Quality 2
Debayer 3

Code: [Select]

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores_ks -pix_fmt yuv422p10le -n -r 25 -profile:v 2 M23-1452_proresks_profile2_q3.mov
frame=  100 fps=2.1 q=-0.0 Lsize=   35701kB time=00:00:04.00 bitrate=73114.9kbits/s

Code: [Select]

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores_ks -pix_fmt yuv422p10le -n -r 25 -profile:v 3 M23-1452_proresks_profile3_q3.mov
frame=  100 fps=2.1 q=-0.0 Lsize=   53619kB time=00:00:04.00 bitrate=109811.1kbits/s
Prores_ks 4444

Code: [Select]

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores_ks -pix_fmt yuv444p10 -n -r 25 M23-1452_proresks_4444.mov
frame=  100 fps=1.8 q=-0.0 Lsize=   79918kB time=00:00:04.00 bitrate=163672.1kbits/s   

On the MBPr
100 frames read from MLVFS vs 100 frames read from storage:
Storage:

Code: [Select]

frame=  100 fps= 11 q=-0.0 Lsize=   50995kB time=00:00:03.96 bitrate=105491.5kbits/s speed=0.446x
MLVFS:

Code: [Select]

frame=  100 fps=6.6 q=-0.0 Lsize=   50995kB time=00:00:03.96 bitrate=105491.5kbits/s speed=0.261x   

[Danne]

New version in first post

Added:
- MOV proxy/wav file integration when paired with corresponding MLV file.
Follow_this_post

Work like this:
1 - Film with proxy setting on while recording MLV files to camera
2 - Add MLV files and MOV proxy files to a folder and run Switch

What will happen:
Switch will match the corresponding MLV/MOV proxy and then cut out the black frames from the MOV file. In the process it will also generate a wav file which will be matched with the dng files to the corresponding MLV file. Additional metadata information will be injected to the wav file, needed for davinci resolve.
I tried to match files also if they are named slightly different. This seems to be the case when the camera prefix preset is running. Typically you would get a match like this:
2U0A7621.MLV
IMG_7621.MOV
as opposed to:
2U0A7621.MLV
2U0A7621.MOV
Anyway. Switch should be able to handle both scenarios.

What else:
When all processing is done there will be a "cleaned" MOV proxy file next to your  DNG folder. The original MOV file will be moved to A_ORIGINALS.

Commits:
https://bitbucket.org/Dannephoto/switch/commits/183ac6b2096ead0bb7d637f7c73ff95183dc2d10
https://bitbucket.org/Dannephoto/switch/commits/f84a76a6d2016705adb6e5a386cf5591b2731afd

Replaced ffprobe dependency with good old exiftool(Phil Harvey)

Thanks to a1ex for script feedback and very cool MOV proxy recording setting.
Thanks Lars Steenhoff for testing and feedback.

[a1ex]

Typically you would get a match like this:
2U0A7621.MLV
IMG_7621.MOV
as opposed to:
2U0A7621.MLV
2U0A7621.MOV

How to reproduce? I get 3U3B for both MLV and MOV...

[Danne]

Switching to sd card (2) in canon setting
File name setting to default IMG_ in canon menu

MLV comes out with Preset code prefix 2U0A
MOV comes out with prefix IMG_
Of course changing the File name setting to Preset code fixes it...

Is canon select card (2) setting reset to CF card upon restart? I know preferred card in ML menu is set to CF card but what if I want to keep the MOV proxy workflow going without going back to canon menu again?

[a1ex]

Okay, my default is 3U3B (after clearing Canon settings). Indeed, if you change this prefix from Canon menu, the new one is not recognized by ML (but it's not IMG_ either). Should add some more tests about this (currently this is tested in api_test.lua for photos only, without changing Canon setting).

The "preferred card" setting in ML changes Canon card at startup - it does nothing else. MLV from mlv_lite always goes to CF if you have one inserted, otherwise SD. H.264 always goes to the card selected in Canon menu.

So, just leave "preferred card" off (it's off by default).

[Danne]

Ok, so reproduced. Good. I also had to reset some configurations to get the behaviour reproduced over here.

Code: [Select]

So, just leave "preferred card" off (it's off by default)preferred card to off fixes it however Preferred card CF is the default upon installation over here...