raw2dng.app 0.13 Osx GUI development, bugs and updates

[bashi]

i've hacked the raw2dng script convert file() to:

Code Select Expand

convert_file() {
      cd "$FLDR$FILE_NAME"_dng
      "$RAW2DNG" "$FLDR$FILE_NAME".RAW
      /opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 *.dng | /usr/local/bin/ffmbc -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores -profile hq -b 120M -pix_fmt yuv444p10 -y -r 24 "$FILE_NAME".mov & >> "$LOG"
}

i switched ffmpeg to ffmbc. It seems to be easier to configure. (the bitrate setting works in ffmbc - i haven't got this to work in ffmpeg. this could be a very easy way to set quality of video - set bitrate and set quality hq, std, lt or proxy)
http://code.google.com/p/ffmbc/

the quality settings - i'm still trying to find the sweet spot. for ffmpeg -profile:v 2 -qscale 6 seems to be too low. i would recommend -profile:v 2 without -qscale. this gives roughly 120MBit/s. -profile:v 3 is about 185MBit/s

the code above for ffmpeg would look like:

Code Select Expand

convert_file() {
      cd "$FLDR$FILE_NAME"_dng
      "$RAW2DNG" "$FLDR$FILE_NAME".RAW
      /opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -a -o 1 *.dng | /usr/local/bin/ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 2 -pix_fmt yuv444p10 -y -r 24 "$FILE_NAME".mov & >> "$LOG"
}

(i removed the dcraw -s 13000 here - this is saturation level for my 5diii - don't think this value is good for other cameras...)

now, if we could make raw2dng pipe as well we could convert .RAW to ProRes (or any other video) directly without .dng files ;-) Somehow i like the idea of archiving the .RAW instead of a bunch of .dng seq..

EDIT: The framerate fixed to 24 is not very nice. the raw2dng reads the framerate when it makes .dng - is there a way to get to this information and put a variable there? 

[epyonxero]

Any idea when 0.12 will be available?

[mkrjf]

When using raw2dng v.11 on latest SnowLeopard - it always gives segmentation fault.
Do I need to install some other software for it to work?
Thx
Mike

[aaphotog]

Any updates?
When using raw2dng, after dragging my .raw files onto the app, Im getting 2 folders(jpg and dng) and they are BOTH EMPTY!
Im using the newest build of Magic Lantern

[bashi]

can the ones of you that have issues please do the following:

-right-click on raw2dng.app and select "Show Package Contents"
go to raw2dng.app/Contents/MacOS/
and double-click raw2dng there

this will start raw2dng together with a terminal console

now drag-drop your .raw file on it
look at the Terminal Console and copy/paste the stuff that is output there into here as a comment

this gives  at least the chance of seeing where the problem is.

[consmi]

Hey guys, dont know if this was mentioned before, but:

Had no problem converting to dng, but I use the program Aperture, not Photoshop. When I try to open the files in Aperture, It says that the file format is unrecognized.

Solutions?

[upthinklab]

can the ones of you that have issues please do the following:

-right-click on raw2dng.app and select "Show Package Contents"
go to raw2dng.app/Contents/MacOS/
and double-click raw2dng there

this will start raw2dng together with a terminal console

now drag-drop your .raw file on it
look at the Terminal Console and copy/paste the stuff that is output there into here as a comment

this gives  at least the chance of seeing where the problem is.

I'm having the same issue with 0.11. I opened through terminal as you have instructed, but I only get feedback in the raw2dng GUI. Here's what it says:


raw2dng converter GUI for OsX
Beta ver.0.11

M11-2220 File Supported
Generating jpg preview
dyld: Library not loaded: /usr/local/lib/libexiv2.10.dylib
  Referenced from: /Applications/raw2dng.app/Contents/Resources/ufraw-batch
  Reason: image not found
/Applications/raw2dng.app/Contents/Resources/script: line 31:  2925 Trace/BPT trap: 5       "$UFRAW" --out-type=jpg --out-path=../"$FILE_NAME"_jpg *.dng >> "$LOG"
Generating avi preview
dyld: Library not loaded: /usr/local/Cellar/ffmpeg/1.1.3/lib/libavdevice.54.dylib
  Referenced from: /Applications/raw2dng.app/Contents/Resources/ffmpeg
  Reason: image not found
/Applications/raw2dng.app/Contents/Resources/script: line 31:  2927 Trace/BPT trap: 5       "$FFMPEG" -i ../"$FILE_NAME"_jpg/%6d.jpg -vcodec mjpeg -qscale 1 ../"$FILE_NAME".avi >> "$LOG"

[upthinklab]

I would like to add a feature request:

Can the DNG sequence include the .RAW clip name in the naming structure? Right now, I just get a folder (properly named) and a DNG sequence going from 0000.dng through the final frame. As a result, I cannot easily batch process the sequences in After Effects without first manually making comps with unique naming. If I import 100 different image sequences in AE, they all look the same and it's tedious to rename to keep everything organized.

it would be best if it renamed like so:

M11-2220_0000.dng
M11-2220_0001.dng
...and so on

Based on some earlier posts, it looked as though this might have been working at one point in time. ?

Thanks for the great work, guys!

[mkrjf]

I had similar output as other user opening as you suggested:
log.txt:
Resolution  : 1920 x 1080
Frames      : 18
Frame size  : 3629056 bytes
Frame skip  : 1
FPS         : 0.000

Processing frame 1 of 18...

App Details:

raw2dng converter GUI for OsX
Beta ver.0.11

M12-0229 File Supported
/Users/user/Downloads/raw2dng.app/Contents/Resources/script: line 31:   281 Segmentation fault      "$RAW2DNG" "$FLDR$FILE_NAME".RAW >> "$LOG"
Generating jpg preview
dyld: Library not loaded: /usr/local/lib/libexiv2.10.dylib
  Referenced from: /Users/user/Downloads/raw2dng.app/Contents/Resources/ufraw-batch
  Reason: image not found
/Users/user/Downloads/raw2dng.app/Contents/Resources/script: line 31:   284 Trace/BPT trap          "$UFRAW" --out-type=jpg --out-path=../"$FILE_NAME"_jpg *.dng >> "$LOG"
Generating avi preview
dyld: Library not loaded: /usr/local/Cellar/ffmpeg/1.1.3/lib/libavdevice.54.dylib
  Referenced from: /Users/user/Downloads/raw2dng.app/Contents/Resources/ffmpeg
  Reason: image not found
/Users/user/Downloads/raw2dng.app/Contents/Resources/script: line 31:   285 Trace/BPT trap          "$FFMPEG" -i ../"$FILE_NAME"_jpg/%6d.jpg -vcodec mjpeg -qscale 1 ../"$FILE_NAME".avi >> "$LOG"


MY GUESS: ffmpeg dynamic library is not installed as expected or is incompatible with this version of OS (10.6..

I have ffmpeg working just fine on this computer - if what you are doing has cmd line equivalent - please share and I can just use ffmpeg
Otherwise please provide version with static linked library not dynamic linked.

[aaphotog]

My hard drives are getting full. Im recording, but unable to actually convert anything

[Cusain]

Hey guys, dont know if this was mentioned before, but:

Had no problem converting to dng, but I use the program Aperture, not Photoshop. When I try to open the files in Aperture, It says that the file format is unrecognized.

Solutions?

Hi, download (free) Adobe DNG Converter from the Adobe website (Windows and Mac versions) and use it to process your files into camera raw DNG files which Aperture can import and process.
Photoshop Elements (which is often thrown in free with scanners and the like) can also convert the raw2dng files into TIFFS, and also eliminates hot pixels if you have that problem (as I have on my old 50D, though only in video).

[epyonxero]

I think v 0.11 may be incompatible with raw files created by the latest ML Raw version. RawMagic works.

[S3Dcentre]

Hey, I have noticed that our the raw2dng.app inserting vertical lines into digital negatives. If i export in Apple Pro Res 4444, the vertical lines are still present, but if I convert the dng .RAW video sequence into an h.264 mp4 video file, it compresses them away.

I am not sure if this is our Mark III sensor, or an issue with compression. See Images Below.


This 100% clip from a .RAW sequence converted in raw2dng, and then exported as apple pro res 4444 in After Effects CS6.
[/b]

100% clip from .RAW sqeuence convereted in raw2dng, then exported as h.264 mpeg (.mp4) in After Effects CS6.
[/b]



Any information would be great, thanks.

[scrax]

New version uploaded, updated the app with last bin (so it should work with also latest ML generated raw files). ffmpeg and dcraw should work now. link

For vertical lines latest ML code and 0.12 should fix it

[squig]

Hey, I have noticed that our the raw2dng.app inserting vertical lines into digital negatives. If i export in Apple Pro Res 4444, the vertical lines are still present, but if I convert the dng .RAW video sequence into an h.264 mp4 video file, it compresses them away.

I am not sure if this is our Mark III sensor, or an issue with compression.

It's a sensor issue and we're not sure what causes it. The strange thing is it doesn't affect every 5D, I've never had a problem with it. http://www.magiclantern.fm/forum/index.php?topic=5614.0

[KMA_WWC]

It's a sensor issue and we're not sure what causes it. The strange thing is it doesn't affect every 5D, I've never had a problem with it. http://www.magiclantern.fm/forum/index.php?topic=5614.0

Same things here. Haven't seen that issue on my camera on any builds. I have even OLPF removed.

[squig]

I suspect it only affects a particular batch of cameras.

[mkorho]

New version uploaded, updated the app with last bin (so it should work with also latest ML generated raw files). ffmpeg and dcraw should work now. link

For vertical lines latest ML code and 0.12 should fix it

TO DO:
Extract dead pixel list from dng and use it in dcraw to remove dead pixels.

Dcraw wont read the dead pixel information from the source file but needs an external list file.

[Danne]

New version uploaded, updated the app with last bin (so it should work with also latest ML generated raw files). ffmpeg and dcraw should work now. link

For vertical lines latest ML code and 0.12 should fix it

Thanks Scrax! It,s working fine . Good work. The ProRes looks fine. Also the white balance issue seems fine. Is the white balance information recorded from what you see in the camera to the ProRes files?
I got a few questions. When I remove the dcraw and ffmpeg from your app, It simply converts the dng;s again. Is that the way I should do if I want to skip the ProRes part? Sometimes only the dng,s would be enough.
Also, is there an alternative If I simply only wants the ProRes files and skip the dng,s? Also, what bitrate and quality is the ProRes in? 422HQ, 444?

A lot of questions, I know. Still, I believe this is one of the best converters out there.
Thanks a lot Scrax!
//D

off topic
Could someone explain or link me to how to convert spanned files on the rawtodng converter? Seem that I,ve missed that part of my raw-education lessons .

[bashi]

Thanks Scrax! It,s working fine . Good work. The ProRes looks fine. Also the white balance issue seems fine. Is the white balance information recorded from what you see in the camera to the ProRes files?
I got a few questions. When I remove the dcraw and ffmpeg from your app, It simply converts the dng;s again. Is that the way I should do if I want to skip the ProRes part? Sometimes only the dng,s would be enough.
Also, is there an alternative If I simply only wants the ProRes files and skip the dng,s? Also, what bitrate and quality is the ProRes in? 422HQ, 444?

A lot of questions, I know. Still, I believe this is one of the best converters out there.
Thanks a lot Scrax!
//D

off topic
Could someone explain or link me to how to convert spanned files on the rawtodng converter? Seem that I,ve missed that part of my raw-education lessons .

White Balance currently uses dcraw automatic - quite ok in lot of cases - not in all...

unfortunately ml does not seem to be record WB in Raw...

the ProRes is about 422HQ with qscale 5 - about 76MBit/s - but it is somewhat dependend on input.

@scrax: to get FPS out of raw2dng:

Code Select Expand

convert_file() {
   cd "$FLDR$FILE_NAME"_dng
   "$RAW2DNG" "$FLDR$FILE_NAME".RAW > r2dlog.txt
   
   FPS=`awk '/FPS/ { print $3; }' "$FLDR$FILE_NAME"_dng/r2dlog.txt`
   
   rm r2dlog.txt
   
   printf "Generating ProResHQ 422\n"

and then in ffmpeg replace the two -r 24 by -r $FPS

there might be a way without saving r2dlog...


i have been playing around with gui creation based on your script - there is pasha to make ui from script. I have it working here with Image preview and manual WB and some more options.
Included ffmbc and dcrawU (i guess U means universal?), tried to build with static libs (no idea if it worked ;-)
https://www.dropbox.com/s/gxm2vvdtbjxl9tl/r2v.zip
i have seen you started with white balance option and cocoadialog... (I will get the rgbg multipliers for the 5d III, can send them to you)

[scrax]

I've used what Bashi suggested so basically:
WB is on auto for now.
It should be yuv422p10le if it means something to you.
I suppose it could be 444 too maybe

So far we can't make the ProRes without the dng part.

Do you mean the .RAW .R00 .R01 and so on? I'm thinking that they could be merged and converted from raw2dng.app but how are they called?

For now you can from terminal use that:

Code Select Expand

cat drop_first.RAW drop_other.R00 again_with.R01 and_so_on.R02 >> merged.RAW
cat is the command used to merge the files, then you have to drop them in the correct order into the terminal window so they will have the correct address
>> those thing will save the cat output to a new file called
merged.RAW the name of the file merged

[scrax]

White Balance currently uses dcraw automatic - quite ok in lot of cases - not in all...

unfortunately ml does not seem to be record WB in Raw...

the ProRes is about 422HQ with qscale 5 - about 76MBit/s - but it is somewhat dependend on input.

@scrax: to get FPS out of raw2dng:

Code Select Expand

convert_file() {
   cd "$FLDR$FILE_NAME"_dng
   "$RAW2DNG" "$FLDR$FILE_NAME".RAW > r2dlog.txt
   
   FPS=`awk '/FPS/ { print $3; }' "$FLDR$FILE_NAME"_dng/r2dlog.txt`
   
   rm r2dlog.txt
   
   printf "Generating ProResHQ 422\n"

and then in ffmpeg replace the two -r 24 by -r $FPS

there might be a way without saving r2dlog...


i have been playing around with gui creation based on your script - there is pasha to make ui from script. I have it working here with Image preview and manual WB and some more options.
Included ffmbc and dcrawU (i guess U means universal?), tried to build with static libs (no idea if it worked ;-)
https://www.dropbox.com/s/gxm2vvdtbjxl9tl/r2v.zip
i have seen you started with white balance option and cocoadialog... (I will get the rgbg multipliers for the 5d III, can send them to you)

downloading now, Good to hear that you are working on this too
do you think it will be better to have a repo to share our code?

[Danne]

Thanks all!
yes, they,re calle r00 etc, will try the terminal later. Real nice work. Thanks for clarifying on white balance.
Will check in later, gotta go now for now.
Thanks !
//D

[scrax]

My hard drives are getting full. Im recording, but unable to actually convert anything

0.12 works?

[squig]

I did a quick card full test, no probs. Prores is cool but can you make it an option?