Author Topic: MLP - Mac OSX batch processing workflow (former cr2hdr-r)  (Read 342706 times)

swinxx

  • Hero Member
  • *****
  • Posts: 597
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #50 on: November 14, 2014, 02:19:01 PM »
@danne:
- as mlvfs is mlv only, this would indeed be a great idea.
- i have deleted the images from your app but will convert again for posting here. one difference here is that the conversion with your app is flickering here, and that in my scenario, the picture quality seems better. you will see when i post the pics.

greets sw

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #51 on: November 14, 2014, 06:48:34 PM »
Hey Swinxx and Deafeyejedi. I,m trying to convert longer files containing more than 1000 frames. Encountered an issue when converting a longer mlv file which seemed to stop the conversion after around 700 dng:s. I look at the file that stopped the conversion but it seems fine. On rerun it stops around the same place but not on the exact same dng. Some hick up with the binary maybe.
I then tried a longer raw which seemed to work. Could anybody try a longer raw vs a longer mlv to see if the issue is reproduced?


*update. noticed this behaviour also for raw sequences. It seems the cr2hdr has som issues maybe with blurry files, hmm

Segmentation fault: 11


DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3081
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #52 on: November 14, 2014, 07:32:28 PM »
Copy that @Danne -- will get on it now and then let you all know if it does in fact reproduce the issue or not.

Stand by...
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #53 on: November 14, 2014, 07:43:09 PM »
I think you can skip the long tests. Not exactly sure what is causing the hickup but probably the cr2hdr binary has some automation going on which could lead to make it stop. Found this in terminal. Seen it sometimes before.

Segmentation fault: 11

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3081
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #54 on: November 14, 2014, 08:23:22 PM »
about to start an MLV folder containing 1181 files and you mentioned that I didn't have to do the long test -- how exactly do I shorten the routine towards to final results from the binary itself?

let me know if I should wait for your suggestion otherwise I'll just go ahead with the normal routine (long test)...
thx
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #55 on: November 14, 2014, 09:05:36 PM »
If you want to stop automator right click the wheel at the top screen and abort or stop, can,t remember exactly. It will probbly run a while and the put the originals in a folder.
You could try run it to see how your file behave.
Thanks
/D

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #56 on: November 14, 2014, 11:08:57 PM »
Ok, so issue regarding segmentation fault 11 seem software and picture related. Not file format spanned or exfat related.
Might occur when a file can,t be handled with cr2hdr and converted directly. Pic might be blurry to bright, or something else. It seems to work when retrying so I put in some reruns of the converter if it should stop randomly while running. Of course dual iso should be used in contrasty situations for best results and not judged from badly uncontrasty test files as with the case where I got the problem.

swinxx

  • Hero Member
  • *****
  • Posts: 597
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #57 on: November 15, 2014, 08:38:15 AM »
@danne.
i can not see a wheel when your app is working?
i have osx 10.10

greets sw

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #58 on: November 15, 2014, 08:46:08 AM »
Ok, I see. I,m still on mavericks. Could it be Yosemite related? I,m curious if the app runs the exiftool white/blacklevel or not on your file since you had flicker.

the wheel (to the left)


I,m trying to use mlvfs dual iso implementation but I get a lagg effect on folders and in workflow. Only really tested on AE. This is when I work in fullres mode just before exporting. Preview mode seems to be working fine. What are you experiencing Swinxx?

swinxx

  • Hero Member
  • *****
  • Posts: 597
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #59 on: November 15, 2014, 10:23:38 AM »
Hi, my workflow is different:
Mount -> Copy to new folder -> Import the printed files into Resolve -> voila
The thing is that the copy procedure is as dmilligan described -> slow (1-3fps)

Will search for the wheel ;)

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #60 on: November 15, 2014, 10:34:28 AM »
So you drag the folders to desktop first and gets them converted? If so, then conversion with cr2hdr-r should be significantly faster? Did you try dragging chunks of files to start parallell processing with mlvfs?

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #61 on: November 15, 2014, 11:18:24 AM »
Shared a new version
Handles whitespace and also makes the app try a few extra reruns just in case cr2hdr20bit don,t succeed converting problematic files the first time.

cr2hdr-r 0.2b "app"
https://drive.google.com/folderview?id=0B4tCJMlOYfirckJfdkRWU09icWc&usp=sharing

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3081
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #62 on: November 15, 2014, 11:39:28 AM »
@Danne -- once again thanks for your superb app... indeed it is faster than MLVFS (<-- I still love your app @dmilligan esp w the option to use DR & CC PP 2014) but when it comes to converting dual-iso w cr2hdr20-bit... c2rhdr-r is the one to go with and yet the latest update just makes it even more durable and reliable!  :)

SJ
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #63 on: November 15, 2014, 11:58:58 AM »
Nice to hear someone using this :). Thanks.
Gotta try the preview and realtime possibilities with mlvfs some more. If there is a bottleneck anywhere I, m certain dmilligan will solve this. For now I stick with my automator baby :). Which of course, from my part, is a bunch of compiled script lines ftom the community and information out there in cyberspace:)

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3081
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #64 on: November 16, 2014, 09:29:51 AM »
Ran a test with several RAW/MLV files over 1000 frames (few were over 2k frames as well) and all seems to be running fine. No errors were able to be duplicated per your request.
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3081
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #65 on: November 16, 2014, 10:07:20 AM »
FYI -- I always use exfat on my cards, in case you were curious.
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #66 on: November 16, 2014, 10:25:28 AM »
Sounds good, thanks for testing! :)

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #67 on: November 19, 2014, 10:04:02 PM »
I,ve started playing with exiftool and there is a lot to learn. I have created an experimental build. It converts much as before but after conversion is done it gives you back the metadata(MLV) into the converted DNG file. MLV of course has a lot more to tell than RAW. I also implemented a whitelevel command which should give a correct(same level) whitelevel reading read from the first dng and applied after conversion.

If anybody wants to try it out feel free. Do copies of originals!

https://drive.google.com/file/d/0B4tCJMlOYfirN1pkLTAzNTFwZUE/view?usp=sharing

This is used for copying mlv metadata, adding it back and applying whitelevel

exiftool *.dng -X > /tmp/a.xml -overwrite_original
find "${workingDir}" -name '*.dng' -print0 | xargs -0 -P 4 -n 1 cr2hdr20bit
exiftool -tagsfromfile /tmp/a.xml -all:all *.DNG -overwrite_original
exiftool -tagsfromfile *.DNG -exif:whitelevel -overwrite_original


DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3081
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #68 on: November 20, 2014, 01:10:18 AM »
Excellent ideas on the update for your app... Will definitely test them out tonight as soon as I get home!
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

swinxx

  • Hero Member
  • *****
  • Posts: 597
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #69 on: November 20, 2014, 05:59:20 AM »
Thank you, me too

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #70 on: November 20, 2014, 07:56:04 AM »
As for know the metadata is being applied to mlv files when the process starts from an original mlv raw file. When you run the app on a folder with already converted dng files  it works as before and with no mlv data changed after conversion except for whitelevel.
If you test please check for flicker  since I didn,t include blacklevel correction. Seems to work up to 70000 on whitelevel and I,m not sure if the files go any further.

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #71 on: November 25, 2014, 06:26:02 PM »
-deleted-

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3081
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #72 on: November 25, 2014, 07:05:32 PM »
Meaning it wasn't worth to update? So should I downgrade back to previous version regarding your app?

Thx!
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #73 on: November 25, 2014, 07:27:43 PM »
Uploading a new one in a while.

Danne

  • Hero Member
  • *****
  • Posts: 4270
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #74 on: November 25, 2014, 07:32:49 PM »
From first post.

Experimental cr2hdr-r 0.2d "app"(DaVinci Resolve)
https://drive.google.com/file/d/0B4tCJMlOYfirOHdVSkpwSmlQVU0/view?usp=sharing

Thanks to dmilligan for pointers and information about wav information and TimeCode(exifdata).

Works with RAW(no audio of course) on different framerates.
Works on MLV files with audio (Only when filmed in 25fps for now)

Above version converts as usual but with naming scheme working with DaVinci resolve. It runs audio+16-bit DNG from converted MLV files in Da Vinci resolve(only in 25fps for now) RAW is also working(Without audio of course).
Since I simply copy the missing ixml information into the audio(wav) the information for now will be hardcoded to master speed 25000. Audio and DNG will merge only when filmed in 25fps for now.
There are some new nice binaries added, SoX(shortening audio) and BWF MetaEdit and also a created bmcc.xml file. Run the command installer and then you,re good to go.