raw2dng.app 0.13 Osx GUI development, bugs and updates

[bashi]

@scrax,
i think prores -qscale 1 is overkill. I have done some "research" on this the last couple of weeks and came up with this:

ffmpeg -f image2 -i %6d.tiff -r 24 -vcodec prores_kostya -profile:v 2 -qscale:v 6 -vendor ap10 -pix_fmt yuv422p10le -threads 4 -y -r 24 ProResHigh.mov

Let me explain some things:

the first -r 24 is necessary since ffmpeg standard framerate for image sequence would be 25
-prores_kostya is supposed to be a bit better than prores
-profile 0-3 is setting Proxy, LQ, SQ, HQ quality - SQ and HQ give almost same quality. I use 2 or 3 there
-qscale:v 6 Settings between 5 - 10 should be fine. Lower the better but bigger file size.
-vendor ap10 Makes the resulting video file to appear as to be made by quicktime - i think it is more compatible
-pix_fmt yuv422p10le set it to 422. You can use yuv444p10le to make 444. Not sure if it really makes a difference - i rembember i got same file size for 422 and 444 - maybe ffmpeg does something wrong or i do.
-threads 4 just to use all of my 4 threads
the last -r 24 sets output to 24 fps - not entirely sure it's necessary.

as for raw to dng - i found dcraw to work fine and fast - ufraw is based on dcraw.
my settings there:

dcraw -T -H 0 -6 -W -q 3 -S 13000 -a ******.dng

again some info:
-T makes tiff
-H 0 is highlight treating - -H 0 is save to use
-6 makes 16bit output
-W don 't apply auto-brightening of image
-q 3 quality scale
-S 13000 Saturation i found works best for 5DIII - this is not necessary to set
-a is auto WhiteBalance - works ok in most cases - but not in all

I do somehow think the user should have the choice if he wants the raw2dng makes jpg (or tif) and/or video too. What i mean the raw2dng app should be able to do raw to dng conversion only - in case somebody has a different workflow. Not that i think it should not be able to do other things too.

On this whole Auto-Conversion, the thing that bugs me most at the moment is the White Balance setting... Since the dng don't have WB from Camera embedded/included (why actually not? it should! ;-) we need to do it automatically - which, as said, is not optimal in some cases - or do it manually. It just have not found a way yet to make this nicely in batch converting...

Thanks all for your work. bashi

[bashi]

As for having the ffmpeg and ufraw or dcraw as external installation would make sense at the moment, until we found out how to bundle the necessary dylib and whatever is needed.

[scrax]

I've already updated the code for using already installed tools, will add dcraw.
I've an idea for add a way to open an options menu. Maybe here we can let user chose default workflow:

So far they could be:

1. Only raw2dng (simple/default)
2. dng, tiff prores (prores)
3. dng, tiff jpg prores mjpg (all)
4. toggle use dcraw or ufraw
5. Set work folder (to import and convert from card)

[Danne]

I've already updated the code for using already installed tools, will add dcraw.
I've an idea for add a way to open an options menu. Maybe here we can let user chose default workflow:

So far they could be:

1. Only raw2dng (simple/default)
2. dng, tiff prores (prores)
3. dng, tiff jpg prores mjpg (all)
4. toggle use dcraw or ufraw
5. Set work folder (to import and convert from card)

This is so cool

As stated, the white balance with the magenta really needs the files to be processed through raw DNG at the moment so yes, number 1 option should be only raw to dng/cdng. That,s the fullproof workhorse . After that adding the tiff, jpg options and maybe sum it up with the ProResHQ422 which is a moviefile format. Heck, or maybe just divide in two steps. 1 being original raw2dng and the 2 all the rest?
Important feature would definitely be the batch conversion tool. To have them converted place in folders works fine. Maybe to have the ProRes files placed in the same folder would be to hard to achieve? I don,t think placing tiffs och jpegs in one filder would work due to the huge amounts of files but prores would be moviefile-segments.
The workfolder thingy. I think most users simply drag the files to a folder to the computer and go from there. I know I do anyway

I,m sure it will work fine with some beta-testing

Gooood work Scrax!

[Hazer]

Hi.  Thanks so much for your continued work on this.

This version breaks the 5D2 metadata -- .dng files open up as Camera model "Canikon" in Photoshop on Mac, and picture styles are no longer available.

For reference, here are the metadata tags from a v0.10 that correctly identify the 5D2:

Make                            : Canon
Model                           : Canon EOS 5D Mark II
UniqueCameraModel               : Canon EOS 5D Mark II

If you can include these again in the next compile, that would be fabulous.

Thank you!

[scrax]

Hi.  Thanks so much for your continued work on this.

This version breaks the 5D2 metadata -- .dng files open up as Camera model "Canikon" in Photoshop on Mac, and picture styles are no longer available.

For reference, here are the metadata tags from a v0.10 that correctly identify the 5D2:

Make                            : Canon
Model                           : Canon EOS 5D Mark II
UniqueCameraModel               : Canon EOS 5D Mark II

If you can include these again in the next compile, that would be fabulous.

Thank you!

Can you check if with 0.10 they are correct using the same .RAW

[budafilms]

RAW2DNG  0.11 create a foilder called jpg empty. This is for what? prepare a folder to leave the files worked in ACR? Thanks.

[scrax]

the jpg folder is empty because of ffmpeg error

[Hazer]

Can you check if with 0.10 they are correct using the same .RAW

I just checked on the same raw file and yes, the older version that I have adds the 5d2 metadata, whereas 0.11 does not.  Thomas Worth on the RawMagic thread indicated that there is no camera metadata in the actual .RAW files.  So it would seem that the older version that I'm using must have "Canon 5D Mark II" hard-coded.  This may be a custom version that was uploaded at one point just for the 5D2.

If that is indeed the case, it would be great if Raw2DNG had some kind of drop-down to specify which camera you're using, so it could set the correct metadata.

Eventually, the best solution would be for the .RAW files themselves to be set with the correct metadata at capture time -- camera model, white balance, and so forth.  Of course everybody's plate is pretty full right no, so we'll get there when we get there.

Anyway thanks again for your work on this!

[budafilms]

Hi, maybe is a report.

I was tried again with 0.11 and i transform from RAW to DNG around 4 gb.
After that, I open, color processing and exporter from ACR to tiff folder.

So, I have 9 gigabites.

Compressor cannot open the file sequence, but with other RAW2DNG, I didn´t have that problem.
I divided the 9 GB file in two and Compressor works like ever before.

Any idea?

[Bioskop.Inc]

Hi, thanks for this GUI its been great so far...
However, with the latest 60D compile v.11 just produces empty folders - both DNG & JPEG (but Jpeg was always like this).
In the DNG folder it has produced a log file & it has the normal stuff on it, apart from it just says processing file 1 of 129 & then nothing else...

I have reverted back to v.10 & it works just fine...strange?!

[Danne]

Hi Scrax! I see Alex updated the rawtodng.exe today, some kind of purple fringing issue? Are those issues exactly the same on your app? Are you using the same source coed for both mac and windows?

[Danne]

Hi Scrax! Tried converting a rawclip filmed with 50fps. In version 0.11 it creates two empty folders , dng, and jpg, nothing gets converted. In version before, 0.10 conversion from 50fps works really good bu tnot in the latest?
//D

[scrax]

I just checked on the same raw file and yes, the older version that I have adds the 5d2 metadata, whereas 0.11 does not.  Thomas Worth on the RawMagic thread indicated that there is no camera metadata in the actual .RAW files.  So it would seem that the older version that I'm using must have "Canon 5D Mark II" hard-coded.  This may be a custom version that was uploaded at one point just for the 5D2.

If that is indeed the case, it would be great if Raw2DNG had some kind of drop-down to specify which camera you're using, so it could set the correct metadata.

Eventually, the best solution would be for the .RAW files themselves to be set with the correct metadata at capture time -- camera model, white balance, and so forth.  Of course everybody's plate is pretty full right no, so we'll get there when we get there.

Anyway thanks again for your work on this!

In source I saw once the string Canikon so i suppose it's from there, and I think it's not camera specific so the module can be "universal", maybe alex could explain us better if it's possible to link it to the camera model.

Hi, thanks for this GUI its been great so far...
However, with the latest 60D compile v.11 just produces empty folders - both DNG & JPEG (but Jpeg was always like this).
In the DNG folder it has produced a log file & it has the normal stuff on it, apart from it just says processing file 1 of 129 & then nothing else...

I have reverted back to v.10 & it works just fine...strange?!

can you post somewhere the log and where you have raw2dng.app and the raw file (full addres),is strange that someone can use 0.11 and someothers have problem generating the .dng, but could be that in the clean-up I've made some typos. will try to find the problem tomorrow and post a 0.12 withthe lates rawa2dng sources

Hi Scrax! I see Alex updated the rawtodng.exe today, some kind of purple fringing issue? Are those issues exactly the same on your app? Are you using the same source coed for both mac and windows?

Yes the source code for raw2dng is the same for win, osx and linux. Will update it in 0.12

[a1ex]

We can embed the camera model in the header, but it needs updating all the converters. So... at the next major file format upgrade.

[tin2tin]

If you consider an update/change of the raw/dng format,  please consider to include creation time and date, because this data then can be used for embedded time code and reel name in the later process.

[a1ex]

Sure. The problem with current metadata is that we can't extend it without breaking current converters, so the first step is to have something where we can add things easily.

I can also get microsecond-level timestamps for each frame, if it helps.

[Bioskop.Inc]

can you post somewhere the log and where you have raw2dng.app and the raw file (full addres),is strange that someone can use 0.11 and someothers have problem generating the .dng, but could be that in the clean-up I've made some typos. will try to find the problem tomorrow and post a 0.12 withthe lates rawa2dng sources

Here's the log file that was generated, but as you'll see it just seemed to get stuck:
https://dl.dropboxusercontent.com/u/90827273/_log.txt

& i put the RAW2DNG into the applications folder (as i did with the other versions). Normally i just open it & drag the Raw files into it - it does the rest. I just put the Raw files in a folder on my desktop. Its all worked fine until the recent update/compile this morning - v.10 works fine & v.11 was working fine until Sat newest 60D build.

Also, since the RAWMagic v.4 will include 60D (well all our cameras) it might not be a problem for much longer.
But always useful for Silent Pics etc...

[budafilms]

Is the last version the link in the first topic? I was reading in Hudson and is talking about new fixes...   

[bashi]

@scrax
there is away to convert from dng directly to video, without saving images. Using PIPE function. I'm new to this stuff, so i don't really know much about, but here a shell script that works:

Code: [Select]

for f in "$@"; do shell script
cd $f
/opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | /opt/local/bin/ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 3 -qscale:v 5 -vendor ap10 -pix_fmt yuv422p10le -threads 4 -y -r 24 $f/ProResHigh.mov
echo "$f converted"
done
input $@ in this case folders with the dng
the important thing is in draw the "-c" = Write image data to standard output
and in ffmpeg the "-f image2pipe -vcodec ppm" and the " -i pipe:0"

This way is faster plus it does not need that much disk space.

[scrax]

@scrax
there is away to convert from dng directly to video, without saving images. Using PIPE function. I'm new to this stuff, so i don't really know much about, but here a shell script that works:

Code: [Select]

for f in "$@"; do shell script
cd $f
/opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | /opt/local/bin/ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 3 -qscale:v 5 -vendor ap10 -pix_fmt yuv422p10le -threads 4 -y -r 24 $f/ProResHigh.mov
echo "$f converted"
done
input $@ in this case folders with the dng
the important thing is in draw the "-c" = Write image data to standard output
and in ffmpeg the "-f image2pipe -vcodec ppm" and the " -i pipe:0"

This way is faster plus it does not need that much disk space.

Nice find, so we just need to have installed installed ffmpeg, dcraw seems to work from the package.

[bashi]

as addition: if we add a & behind it it gets even faster since all the conversion is done simultaneously with background jobs. (tried to explain as good as possible)

so in my example it looks like:

Code: [Select]

dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 2 -qscale:v 6 -vendor ap10 -pix_fmt yuv422p10le -y -r 24 ProResHigh.mov &
(i removed the -threads 4 from ffmpeg too - doesn't make sense anymore)

a concern for me is still how to set proper White balance and avoid the automatic there...

[aaphotog]

Latest version of raw2dng not working with the latest compile of ML
I drag the folder(or individual file) onto the app, it starts to process but only takes a fraction of a second for each raw. Creates both the dng and jpg folders, and they're both empty

[scrax]

Latest version of raw2dng not working with the latest compile of ML
I drag the folder(or individual file) onto the app, it starts to process but only takes a fraction of a second for each raw. Creates both the dng and jpg folders, and they're both empty

I've compiled a new raw2dng from source I suppose recent changes broke it., time to goto 0.12

as addition: if we add a & behind it it gets even faster since all the conversion is done simultaneously with background jobs. (tried to explain as good as possible)

so in my example it looks like:

Code: [Select]

dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 2 -qscale:v 6 -vendor ap10 -pix_fmt yuv422p10le -y -r 24 ProResHigh.mov &
(i removed the -threads 4 from ffmpeg too - doesn't make sense anymore)

a concern for me is still how to set proper White balance and avoid the automatic there...

so far the other code can generate the mov but it's empty, i need to make some more test.

[selfreflexion]

I get this:
raw2dng converter GUI for OsX
Beta ver.0.11

M08-1643 File Supported
Generating jpg preview
dyld: Library not loaded: /usr/local/lib/libexiv2.10.dylib
  Referenced from: /Applications/raw2dng.app/Contents/Resources/ufraw-batch
  Reason: image not found
/Applications/raw2dng.app/Contents/Resources/script: line 31: 46298 Trace/BPT trap: 5       "$UFRAW" --out-type=jpg --out-path=../"$FILE_NAME"_jpg *.dng >> "$LOG"
Generating avi preview
dyld: Library not loaded: /usr/local/Cellar/ffmpeg/1.1.3/lib/libavdevice.54.dylib
  Referenced from: /Applications/raw2dng.app/Contents/Resources/ffmpeg
  Reason: image not found
/Applications/raw2dng.app/Contents/Resources/script: line 31: 46300 Trace/BPT trap: 5       "$FFMPEG" -i ../"$FILE_NAME"_jpg/%6d.jpg -vcodec mjpeg -qscale 1 ../"$FILE_NAME".avi >> "$LOG"






the files come out fine but it makes two folders, one of them blank

I also did a clean install of macports after, still the same output