Author Topic: MLP - Mac OSX batch processing workflow (former cr2hdr-r)  (Read 323592 times)

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3066
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #25 on: October 18, 2014, 10:33:17 AM »
*update*

Although I did it twice just to make sure. For some reason after your app is completed and I get an error when trying to open the newly converted DNG's. Tried again with LR5 -- same.



Will try restarting the macbook pro and try one last time before I hit the sack...

5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #26 on: October 18, 2014, 10:55:34 AM »
Could you upload a non working DNG?

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3066
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #27 on: October 18, 2014, 06:22:29 PM »
@Danne -- my fault I fell asleep last night and here is the DNG you requested...

https://www.dropbox.com/s/t3bswzveeg0z6ti/M20-1628_000000.DNG?dl=0
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #28 on: October 18, 2014, 07:04:43 PM »
Thanks.
It,s the white and black level that didn,t finish. Sure you let the script work til the end?
You can run the app on a converted folder and it will apply black and white level and skip conversion. Should be ok after that.
Get back after trying.

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3066
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #29 on: October 18, 2014, 07:12:44 PM »
I actually did let it run through overnight so I'm assuming it did apply the white/black levels toward the end?

I am currently converting again with the newest app that you mentioned to me in PM from last night and will upload a sample DNG as soon as its completed..

It's still in the process so I'll wait for the white/black levels to be applied...
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #30 on: October 18, 2014, 07:13:57 PM »
Cool. Maybe something interrupted the process, not sure.

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3066
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #31 on: October 18, 2014, 08:02:42 PM »
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #32 on: October 18, 2014, 08:21:49 PM »
Yes, it works? It is opening overe here. Not at your computer?

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3066
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #33 on: October 19, 2014, 06:11:53 AM »
@Danne -- all good. apparently it opens up just fine via ACR/LR5 as soon as I got home tonight. Both takes. Perhaps the first take was due to not being fully completed (as you stated regarding the black/white levels not being applied) as strange as that could be... not sure how this happened but I wouldn't worry about it.

Could be due to my old poor 2006 macbook pro running Mavricks, ha?

will upload the test footage as soon as Vimeo is back into service from their so called maintance.

Thanks again for everything!
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #34 on: November 01, 2014, 08:26:15 PM »
Hi all! I made some changes in this app and added a description about what in the first post. Mainly added the possibility to cat spanned files. This is oldschool I know but could be nice though to at least have the app handle the files one way or the other than not at all. Or use Mlvfs or Mlrawviewer for export before cr2hdr conversion.

cr2hdr-r 0.2 "app"
https://drive.google.com/folderview?id=0B4tCJMlOYfirckJfdkRWU09icWc&usp=sharing

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3066
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #35 on: November 02, 2014, 08:03:08 PM »
Thanks for the update @Danne -- today my baseball game got cancelled so I'll take this opportunity to go out and shoot some dual-iso 2.5k RAW footage...

Be in touch!
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #36 on: November 03, 2014, 12:11:00 AM »
Cool. Looking forward for some 2, 5k :)

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3066
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #37 on: November 13, 2014, 06:12:09 AM »
@Danne...question -- any idea if its possible to enable an option in order to export dual-iso text files after its renders?

So I can read the reports to verify the DR and such...

I thought it was producing these files in the previous version but not the new one unless I'm missing something?

Thx
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #38 on: November 13, 2014, 06:58:51 AM »
Don,t really know how to get that textfile, sorry. You can use the binary directly in terminal and the processing will be shown while the dng:s are baked. Around 2 minutes in https://www.youtube.com/watch?v=tCszO1nGrwc&list=UUomeOeghS6wanMOCQ8BtH_A

swinxx

  • Hero Member
  • *****
  • Posts: 597
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #39 on: November 13, 2014, 09:00:27 AM »
@danne:
Hello Danne, thank you for your tool. But isn´t "Full DualIso" provided with MLVFS? So why should i use your tool instead?

best wishes, sw

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #40 on: November 13, 2014, 11:31:06 AM »
One word.  RAW

Is mlvfs exporting multithreaded? Gotta check that out. This workflow runs 4 dngs in parallell.

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3066
  • 5D3 / M1 / 7D / 70D / SL1
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #41 on: November 13, 2014, 07:36:47 PM »
@swinxx -- I also think it's because I noticed when converting 20-bit dual ISO w MLVFS it comes out under exposed whereas if using @Danne's app it gives you well exposed DNG's as well as fixing black/white levels for dual-ISO videos which is a PLUS!

If you rather use MLVFS -- you just gotta apply the black/white levels and/or adjust the exposures manually along with highlight recovery as oppose to @Danne's app when the highlights seems to be already taken care of because if I do try to reduce the highlights it basically makes them go a bit too low so I kind of like how @Danne's app does all the 'dirty' work for you if that makes sense...
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #42 on: November 13, 2014, 08:03:56 PM »
Do you know if it is multithreaded deafeyejedi?

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3212
  • 60Da / 1100D / EOSM
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #43 on: November 13, 2014, 10:16:33 PM »
Do you know if it is multithreaded deafeyejedi?
It is...sort of. It can process multiple frames at the same time (in separate threads), but it will only do so if you actually ask it to, i.e. if you actually try to read both DNGs at the same time. If you're reading the DNGs from a post processing app that supports "pre-fetching" or loading multiple frames concurrently, then MLVFS will be multi-threaded. I know SpeedGrade supports this and there's even a setting where you can specify the number of frames it will "pre-fetch" (it's a good idea to set it to the number CPU cores you have for this). If you only try and read one DNG at a time, then MLVFS is only going to use one thread to process that DNG. An example would be the Finder, if you use the Finder to copy files out of MLVFS mount, it only copies one file at a time. If you wanted it to be "multithreaded" you could start multiple copy operations (for example start to copy out half the DNGs, and then start to copy out the second half).

Now, I suppose that I could support "pre-fetching" internally to MLVFS => when one frame is asked for, go a head and process the next one (or the next several) in a separate thread, so it will be ready when it's asked for, but that will be some work to implement.

If you rather use MLVFS -- you just gotta apply the black/white levels and/or adjust the exposures manually
That is not true. You should not need to adjust the black/white level with MLVFS. If you have an example file where MLVFS gives the wrong black or white level, then please share it with me so I can fix MLVFS.

As far as manual exposure/highlights adjustment goes, I've never met a dual ISO image I didn't need to make some sort of manual adjustment anyway. In fact, that's sort of the point (to have lots of ability to push the image really hard).

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #44 on: November 13, 2014, 10:46:29 PM »
Been busy working lately but will dive into some more testing soon. A lot of good stuff with mlvfs and also mlrawviewer lately.

Quote
You should not need to adjust the black/white level with MLVFS.
Yea, I was under the impression mlvfs would set the same consistent white and black level for dual iso dng:s. Gotta investigate further.

Since I mostly use dual iso for 3x zoom I,ve been working with raw but I,ve been trying some mlv stuff too and it it,s really nice too although I miss maybe one extra step in performance boost but that is ok. Nice thing with mlvfs is of course the no need for first conversion step, big timesaver.



swinxx

  • Hero Member
  • *****
  • Posts: 597
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #45 on: November 14, 2014, 12:27:12 AM »
@danne

how do i know when the app is finished?

update: i have converted a dual iso mlv, but sorry mate, i have to tell you that after converting, its flickering..
perhaps i am doing something wrong.

i installed all your required files with the command app, then i opened up the cr2hdr-r pointed to the directory of my dual iso mlv folder, and waited. when i now import those files and play it back, its flickering.

Update:
i have to point out that my shot is problematic, however... when i convert my shot with MLVFS, it is Davinci Resolve Conform and Not Flickering.


thx sw

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #46 on: November 14, 2014, 07:13:43 AM »
Did you install exiftool? Not sure why it won, t work. You could post some sample variations for me to look at or simply stick to mlvfs since it, s mlv. Would be nice to find what is wrong though.
You see a small wheel at the top when the conversion is finished. It is an automator thing.
I think mlvfs and cr2hdr-r is using the same binary from Alex so flicker problem should be with white/blacklevel coming from exiftool.
Thanks

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3212
  • 60Da / 1100D / EOSM
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #47 on: November 14, 2014, 01:07:04 PM »
I think mlvfs and cr2hdr-r is using the same binary from Alex
MLVFS does not use the cr2hdr binary. It has it's own implementation of cr2hdr. Much of the code came originally from cr2hdr, but I tweaked the code a lot and there are some important differences. The biggest difference is that MLVFS outputs CDNGs instead of plain old DNGs. Also, the black and white levels come from the metadata in the original MLV file, while cr2hdr computes them from the actual image data. This means that MLVFS's DNGs are all going to already have the same black and white levels and its a little faster (since we don't have to do some computations on the image data).

swinxx

  • Hero Member
  • *****
  • Posts: 597
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #48 on: November 14, 2014, 01:57:39 PM »
hello dmilligan and danne!
i must say that the image, which comes out of mlvfs from dual iso mlvs is brilliant and it makes a big difference!
all in all it is a slow process (as far as i can tell) but superb in terms of quality.
Thanks dmilligan for that great addon!

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: OSX dual iso dng/CR2 batch conversion ”app”
« Reply #49 on: November 14, 2014, 02:10:44 PM »
What,s the difference Swinxx? You could post two samples comparing? THe quality of the converted dng shouldn,t change unless dmilligan also tweaked the fullres or amazed code?

Nice to hear you efforts dmilligan, cdng, tweaked code. Have you created a mac binary which perhaps could work the same as a1ex cr2hdr20bit? Would be nice if cdng, original metadata and same levels would find it,s way into this app as well and then we also would have a raw converter :)

//D