Author Topic: raw2dng.app 0.13 Osx GUI development, bugs and updates  (Read 365182 times)

bashi

  • Freshman
  • **
  • Posts: 53
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #175 on: June 06, 2013, 02:00:14 PM »
@scrax,
i think prores -qscale 1 is overkill. I have done some "research" on this the last couple of weeks and came up with this:

ffmpeg -f image2 -i %6d.tiff -r 24 -vcodec prores_kostya -profile:v 2 -qscale:v 6 -vendor ap10 -pix_fmt yuv422p10le -threads 4 -y -r 24 ProResHigh.mov

Let me explain some things:

the first -r 24 is necessary since ffmpeg standard framerate for image sequence would be 25
-prores_kostya is supposed to be a bit better than prores
-profile 0-3 is setting Proxy, LQ, SQ, HQ quality - SQ and HQ give almost same quality. I use 2 or 3 there
-qscale:v 6 Settings between 5 - 10 should be fine. Lower the better but bigger file size.
-vendor ap10 Makes the resulting video file to appear as to be made by quicktime - i think it is more compatible
-pix_fmt yuv422p10le set it to 422. You can use yuv444p10le to make 444. Not sure if it really makes a difference - i rembember i got same file size for 422 and 444 - maybe ffmpeg does something wrong or i do.
-threads 4 just to use all of my 4 threads
the last -r 24 sets output to 24 fps - not entirely sure it's necessary.

as for raw to dng - i found dcraw to work fine and fast - ufraw is based on dcraw.
my settings there:

dcraw -T -H 0 -6 -W -q 3 -S 13000 -a ******.dng

again some info:
-T makes tiff
-H 0 is highlight treating - -H 0 is save to use
-6 makes 16bit output
-W don 't apply auto-brightening of image
-q 3 quality scale
-S 13000 Saturation i found works best for 5DIII - this is not necessary to set
-a is auto WhiteBalance - works ok in most cases - but not in all

I do somehow think the user should have the choice if he wants the raw2dng makes jpg (or tif) and/or video too. What i mean the raw2dng app should be able to do raw to dng conversion only - in case somebody has a different workflow. Not that i think it should not be able to do other things too.

On this whole Auto-Conversion, the thing that bugs me most at the moment is the White Balance setting... Since the dng don't have WB from Camera embedded/included (why actually not? it should! ;-) we need to do it automatically - which, as said, is not optimal in some cases - or do it manually. It just have not found a way yet to make this nicely in batch converting...

Thanks all for your work. bashi

bashi

  • Freshman
  • **
  • Posts: 53
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #176 on: June 06, 2013, 02:04:35 PM »
As for having the ffmpeg and ufraw or dcraw as external installation would make sense at the moment, until we found out how to bundle the necessary dylib and whatever is needed.

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #177 on: June 06, 2013, 02:30:45 PM »
I've already updated the code for using already installed tools, will add dcraw.
I've an idea for add a way to open an options menu. Maybe here we can let user chose default workflow:

So far they could be:

1. Only raw2dng (simple/default)
2. dng, tiff prores (prores)
3. dng, tiff jpg prores mjpg (all)
4. toggle use dcraw or ufraw
5. Set work folder (to import and convert from card)
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #178 on: June 06, 2013, 04:49:49 PM »
I've already updated the code for using already installed tools, will add dcraw.
I've an idea for add a way to open an options menu. Maybe here we can let user chose default workflow:

So far they could be:

1. Only raw2dng (simple/default)
2. dng, tiff prores (prores)
3. dng, tiff jpg prores mjpg (all)
4. toggle use dcraw or ufraw
5. Set work folder (to import and convert from card)

This is so cool :)

As stated, the white balance with the magenta really needs the files to be processed through raw DNG at the moment so yes, number 1 option should be only raw to dng/cdng. That,s the fullproof workhorse :). After that adding the tiff, jpg options and maybe sum it up with the ProResHQ422 which is a moviefile format. Heck, or maybe just divide in two steps. 1 being original raw2dng and the 2 all the rest?
Important feature would definitely be the batch conversion tool. To have them converted place in folders works fine. Maybe to have the ProRes files placed in the same folder would be to hard to achieve? I don,t think placing tiffs och jpegs in one filder would work due to the huge amounts of files but prores would be moviefile-segments.
The workfolder thingy. I think most users simply drag the files to a folder to the computer and go from there. I know I do anyway :)

I,m sure it will work fine with some beta-testing :)

Gooood work Scrax!

Hazer

  • Freshman
  • **
  • Posts: 56
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #179 on: June 06, 2013, 06:22:53 PM »
Hi.  Thanks so much for your continued work on this.

This version breaks the 5D2 metadata -- .dng files open up as Camera model "Canikon" in Photoshop on Mac, and picture styles are no longer available.

For reference, here are the metadata tags from a v0.10 that correctly identify the 5D2:

Make                            : Canon
Model                           : Canon EOS 5D Mark II
UniqueCameraModel               : Canon EOS 5D Mark II

If you can include these again in the next compile, that would be fabulous.

Thank you!

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #180 on: June 07, 2013, 05:47:58 AM »
Hi.  Thanks so much for your continued work on this.

This version breaks the 5D2 metadata -- .dng files open up as Camera model "Canikon" in Photoshop on Mac, and picture styles are no longer available.

For reference, here are the metadata tags from a v0.10 that correctly identify the 5D2:

Make                            : Canon
Model                           : Canon EOS 5D Mark II
UniqueCameraModel               : Canon EOS 5D Mark II

If you can include these again in the next compile, that would be fabulous.

Thank you!
Can you check if with 0.10 they are correct using the same .RAW
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #181 on: June 07, 2013, 09:12:48 AM »
RAW2DNG  0.11 create a foilder called jpg empty. This is for what? prepare a folder to leave the files worked in ACR? Thanks.

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #182 on: June 07, 2013, 01:00:20 PM »
the jpg folder is empty because of ffmpeg error

I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

Hazer

  • Freshman
  • **
  • Posts: 56
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #183 on: June 07, 2013, 08:52:59 PM »
Can you check if with 0.10 they are correct using the same .RAW

I just checked on the same raw file and yes, the older version that I have adds the 5d2 metadata, whereas 0.11 does not.  Thomas Worth on the RawMagic thread indicated that there is no camera metadata in the actual .RAW files.  So it would seem that the older version that I'm using must have "Canon 5D Mark II" hard-coded.  This may be a custom version that was uploaded at one point just for the 5D2.

If that is indeed the case, it would be great if Raw2DNG had some kind of drop-down to specify which camera you're using, so it could set the correct metadata.

Eventually, the best solution would be for the .RAW files themselves to be set with the correct metadata at capture time -- camera model, white balance, and so forth.  Of course everybody's plate is pretty full right no, so we'll get there when we get there.

Anyway thanks again for your work on this!

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #184 on: June 08, 2013, 10:24:52 AM »
Hi, maybe is a report.

I was tried again with 0.11 and i transform from RAW to DNG around 4 gb.
After that, I open, color processing and exporter from ACR to tiff folder.

So, I have 9 gigabites.

Compressor cannot open the file sequence, but with other RAW2DNG, I didnĀ“t have that problem.
I divided the 9 GB file in two and Compressor works like ever before.

Any idea?

Bioskop.Inc

  • Member
  • ***
  • Posts: 109
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #185 on: June 08, 2013, 02:49:49 PM »
Hi, thanks for this GUI its been great so far...
However, with the latest 60D compile v.11 just produces empty folders - both DNG & JPEG (but Jpeg was always like this).
In the DNG folder it has produced a log file & it has the normal stuff on it, apart from it just says processing file 1 of 129 & then nothing else...

I have reverted back to v.10 & it works just fine...strange?!

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #186 on: June 08, 2013, 07:40:33 PM »
Hi Scrax! I see Alex updated the rawtodng.exe today, some kind of purple fringing issue? Are those issues exactly the same on your app? Are you using the same source coed for both mac and windows?

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7293
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #187 on: June 08, 2013, 09:06:50 PM »
Hi Scrax! Tried converting a rawclip filmed with 50fps. In version 0.11 it creates two empty folders , dng, and jpg, nothing gets converted. In version before, 0.10 conversion from 50fps works really good bu tnot in the latest?
//D

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #188 on: June 09, 2013, 12:01:37 AM »
I just checked on the same raw file and yes, the older version that I have adds the 5d2 metadata, whereas 0.11 does not.  Thomas Worth on the RawMagic thread indicated that there is no camera metadata in the actual .RAW files.  So it would seem that the older version that I'm using must have "Canon 5D Mark II" hard-coded.  This may be a custom version that was uploaded at one point just for the 5D2.

If that is indeed the case, it would be great if Raw2DNG had some kind of drop-down to specify which camera you're using, so it could set the correct metadata.

Eventually, the best solution would be for the .RAW files themselves to be set with the correct metadata at capture time -- camera model, white balance, and so forth.  Of course everybody's plate is pretty full right no, so we'll get there when we get there.

Anyway thanks again for your work on this!

In source I saw once the string Canikon so i suppose it's from there, and I think it's not camera specific so the module can be "universal", maybe alex could explain us better if it's possible to link it to the camera model.

Hi, thanks for this GUI its been great so far...
However, with the latest 60D compile v.11 just produces empty folders - both DNG & JPEG (but Jpeg was always like this).
In the DNG folder it has produced a log file & it has the normal stuff on it, apart from it just says processing file 1 of 129 & then nothing else...

I have reverted back to v.10 & it works just fine...strange?!
can you post somewhere the log and where you have raw2dng.app and the raw file (full addres),is strange that someone can use 0.11 and someothers have problem generating the .dng, but could be that in the clean-up I've made some typos. will try to find the problem tomorrow and post a 0.12 withthe lates rawa2dng sources

Hi Scrax! I see Alex updated the rawtodng.exe today, some kind of purple fringing issue? Are those issues exactly the same on your app? Are you using the same source coed for both mac and windows?

Yes the source code for raw2dng is the same for win, osx and linux. Will update it in 0.12
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #189 on: June 09, 2013, 12:03:48 AM »
We can embed the camera model in the header, but it needs updating all the converters. So... at the next major file format upgrade.

tin2tin

  • Member
  • ***
  • Posts: 158
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #190 on: June 09, 2013, 12:33:20 AM »
If you consider an update/change of the raw/dng format,  please consider to include creation time and date, because this data then can be used for embedded time code and reel name in the later process.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #191 on: June 09, 2013, 12:37:08 AM »
Sure. The problem with current metadata is that we can't extend it without breaking current converters, so the first step is to have something where we can add things easily.

I can also get microsecond-level timestamps for each frame, if it helps.

Bioskop.Inc

  • Member
  • ***
  • Posts: 109
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #192 on: June 09, 2013, 02:37:36 AM »
can you post somewhere the log and where you have raw2dng.app and the raw file (full addres),is strange that someone can use 0.11 and someothers have problem generating the .dng, but could be that in the clean-up I've made some typos. will try to find the problem tomorrow and post a 0.12 withthe lates rawa2dng sources

Here's the log file that was generated, but as you'll see it just seemed to get stuck:
https://dl.dropboxusercontent.com/u/90827273/_log.txt

& i put the RAW2DNG into the applications folder (as i did with the other versions). Normally i just open it & drag the Raw files into it - it does the rest. I just put the Raw files in a folder on my desktop. Its all worked fine until the recent update/compile this morning - v.10 works fine & v.11 was working fine until Sat newest 60D build.

Also, since the RAWMagic v.4 will include 60D (well all our cameras) it might not be a problem for much longer.
But always useful for Silent Pics etc...

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #193 on: June 09, 2013, 05:37:02 AM »
Is the last version the link in the first topic? I was reading in Hudson and is talking about new fixes...   

bashi

  • Freshman
  • **
  • Posts: 53
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #194 on: June 09, 2013, 01:55:25 PM »
@scrax
there is away to convert from dng directly to video, without saving images. Using PIPE function. I'm new to this stuff, so i don't really know much about, but here a shell script that works:

Code: [Select]
for f in "$@"; do shell script
cd $f
/opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | /opt/local/bin/ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 3 -qscale:v 5 -vendor ap10 -pix_fmt yuv422p10le -threads 4 -y -r 24 $f/ProResHigh.mov
echo "$f converted"
done

input $@ in this case folders with the dng
the important thing is in draw the "-c" = Write image data to standard output
and in ffmpeg the "-f image2pipe -vcodec ppm" and the " -i pipe:0"

This way is faster plus it does not need that much disk space.

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #195 on: June 09, 2013, 02:33:18 PM »
@scrax
there is away to convert from dng directly to video, without saving images. Using PIPE function. I'm new to this stuff, so i don't really know much about, but here a shell script that works:

Code: [Select]
for f in "$@"; do shell script
cd $f
/opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | /opt/local/bin/ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 3 -qscale:v 5 -vendor ap10 -pix_fmt yuv422p10le -threads 4 -y -r 24 $f/ProResHigh.mov
echo "$f converted"
done

input $@ in this case folders with the dng
the important thing is in draw the "-c" = Write image data to standard output
and in ffmpeg the "-f image2pipe -vcodec ppm" and the " -i pipe:0"

This way is faster plus it does not need that much disk space.

Nice find, so we just need to have installed installed ffmpeg, dcraw seems to work from the package.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

bashi

  • Freshman
  • **
  • Posts: 53
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #196 on: June 09, 2013, 04:13:28 PM »
as addition: if we add a & behind it it gets even faster since all the conversion is done simultaneously with background jobs. (tried to explain as good as possible)

so in my example it looks like:
Code: [Select]
dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 2 -qscale:v 6 -vendor ap10 -pix_fmt yuv422p10le -y -r 24 ProResHigh.mov &
(i removed the -threads 4 from ffmpeg too - doesn't make sense anymore)

a concern for me is still how to set proper White balance and avoid the automatic there...

aaphotog

  • Senior
  • ****
  • Posts: 356
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #197 on: June 09, 2013, 07:29:36 PM »
Latest version of raw2dng not working with the latest compile of ML
I drag the folder(or individual file) onto the app, it starts to process but only takes a fraction of a second for each raw. Creates both the dng and jpg folders, and they're both empty

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #198 on: June 09, 2013, 08:45:05 PM »
Latest version of raw2dng not working with the latest compile of ML
I drag the folder(or individual file) onto the app, it starts to process but only takes a fraction of a second for each raw. Creates both the dng and jpg folders, and they're both empty
I've compiled a new raw2dng from source I suppose recent changes broke it., time to goto 0.12

as addition: if we add a & behind it it gets even faster since all the conversion is done simultaneously with background jobs. (tried to explain as good as possible)

so in my example it looks like:
Code: [Select]
dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 2 -qscale:v 6 -vendor ap10 -pix_fmt yuv422p10le -y -r 24 ProResHigh.mov &
(i removed the -threads 4 from ffmpeg too - doesn't make sense anymore)

a concern for me is still how to set proper White balance and avoid the automatic there...
so far the other code can generate the mov but it's empty, i need to make some more test.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

selfreflexion

  • New to the forum
  • *
  • Posts: 2
Re: raw2dng.app 0.11 Osx GUI development, bugs and updates
« Reply #199 on: June 09, 2013, 10:00:02 PM »
I get this:
raw2dng converter GUI for OsX
Beta ver.0.11

M08-1643 File Supported
Generating jpg preview
dyld: Library not loaded: /usr/local/lib/libexiv2.10.dylib
  Referenced from: /Applications/raw2dng.app/Contents/Resources/ufraw-batch
  Reason: image not found
/Applications/raw2dng.app/Contents/Resources/script: line 31: 46298 Trace/BPT trap: 5       "$UFRAW" --out-type=jpg --out-path=../"$FILE_NAME"_jpg *.dng >> "$LOG"
Generating avi preview
dyld: Library not loaded: /usr/local/Cellar/ffmpeg/1.1.3/lib/libavdevice.54.dylib
  Referenced from: /Applications/raw2dng.app/Contents/Resources/ffmpeg
  Reason: image not found
/Applications/raw2dng.app/Contents/Resources/script: line 31: 46300 Trace/BPT trap: 5       "$FFMPEG" -i ../"$FILE_NAME"_jpg/%6d.jpg -vcodec mjpeg -qscale 1 ../"$FILE_NAME".avi >> "$LOG"






the files come out fine but it makes two folders, one of them blank

I also did a clean install of macports after, still the same output