Messages

Since we,re on the ProRes topic. Will the files contain similar dynamic range possibilities as with the dng,s? I compared the raw imported with the ginger plugin(trial) and the generated ProRes file. It seems contrasts and highlight information differs. Maybe one could get a flatter profile ProRes file somehow? The first three are the ginger plugin one and the other three are the ProRes. Might it be a white balance thing?

Magic Lantern RAW is 14bit, ProRes is 10bit. There is no way (i know of) to increase ProRes bit depth.
There might be the possibility to output raw video (not sure if it would be 14bit), but FileSize is similar to CinemaDNG, so i don't see it make sense, better work of CinemaDNG sequence then.

the raw converter in use (dcraw) can do gamma correction, so we should be able to get a flatter profile. It can also do Highlight blending and reconstruction. But those things are not in raw2dng.app for now.

White balance has influence on Highlights too.

We should consider use pashua instead of cocoadialog and keep Platypus as wrapper. Pashua will be good for the preference page maybe.
Your app is working in making the dng, then it show the options for video rendering but video is not generated.

The big issue I see so far is that in batch conversion setting wb for each file will be not ideal.

My idea is that settings should be something that is related to app and not to raw file. For example if we enable ProRes conversion in settings all the file imported will be converted, if we set a working folder all file go in that folder and so on.
Things like wb, fps etc. should be automatic and right. Maybe ML could make a video.log of raw captures too like for normal video, and we can use that for finding out needed settings...

EDIT: I can open a settings dialog by dropping raw2dng.app into his window

Code Select Expand


set_pref(){
    printf "Preference panel to be added!\n"
}

##### Main

if [[ ! "$1" ]]; then
   usage
else
   if [[ "$1" == *raw2dng.app ]]; then
      set_pref
   fi
   #set_workdir
...


use platypus and pasha seems ideal. will try this.

what i think of:
on first dialog have checkbox advanced and combo box White Balance with default Auto. if advanced unchecked it will work as usual and convert all raw with this White Balance setting.
if advanced checked it will open for each raw a preview thing to make settings and then render.

additionally on first dialog checkbox image and video, to make img seq and/or video.

this would allow to batch process but as well fine tuning of each. 

EDIT: I can see that it can make sense to edit all videos with the same settings. At the same time i see cases where it would be great to edit each. So an option to edit settings for all or for each could make sense.

I've added fps as you suggested early, now I'm looking into a way for having some options to set, but they should be optional (for advanced use only).

something like an advanced checkbox on first window perhaps... not really sure how to change platypus default window to do that. so far i'm not getting "warm" with platypus and cocoadialog. as far as i see you can only trigger dialog popup (i might be wrong) which is not great to use.

i think the app (the ideal app) should be something like:
-be as easy as possible
-give as much control over the image (raw->img process) as possible (if desired, advanced option). Since those settings gets "backed" into resulting img seq or video.
-be suitable for batch process

btw, did my app work on your computer?

i don't like on mine:
-no drag drop on app (in file selection does work)
-not nice ui (but hey, a ui ;-)
-no feedback about what it is doing ( while raw2dng and dcraw-video)

Yes it is.

All the 3x and 5x zoom mode raw recording have more than 5 dead pixels, at least on my camera.
It is funny that nobody else is seeing them. (or dont care)
To me red, green and blue pixels in video is a NO NO.
And manual removal is too painful process.

Adobe camera raw and Ufraw can read the dead (hot) pixel list from the source file but dcraw cant.

will look into it... (we just switched ufraw to dcraw - might switch back then ;-)
it's not too high of a priority for me though.

@scrax

here your script edited for fps and white balance dialog with cocoadialog working:

Code Select Expand

set_wb(){

  "$CD" standard-dropdown \
  --text "Pick a wb" --title "wb Options" --items "Auto" "Daylight" "Shade" "Cloudy" "Tungsten" "White Fluorescent"\
  --string-output | tail -1 > wb.txt
   
  wb=`awk ' { print $1; }' "$FLDR$FILE_NAME"_dng/wb.txt`
  rm wb.txt
  #Set rgb multiplier
  if [[ $wb = Auto ]]
  then
      Wtemp="-a"
 
  elif [[ $wb = *Daylight* ]]
  then
      Wtemp="-r 1.950195 1.000000 1.605469 1.000000"
 
  elif [[ $wb = *Shade* ]]
  then
  Wtemp="-r 2.226562 1.000000 1.369141 1.000000"
 
  elif [[ $wb = *Cloudy* ]]
  then
  Wtemp="-r 2.089844 1.000000 1.477539 1.000000"
 
  elif [[ $wb = *Tungsten* ]]
  then
  Wtemp="-r 1.412109 1.000000 2.392578 1.000000"

  elif [[ $wb = *White* ]]
  then
  Wtemp="-r 1.720703 1.000000 2.291016 1.000000"
 
   
  fi
}

convert_file() {
   cd "$FLDR$FILE_NAME"_dng
   "$RAW2DNG" "$FLDR$FILE_NAME".RAW > r2dlog.txt
   
   FPS=`awk '/FPS/ { print $3; }' "$FLDR$FILE_NAME"_dng/r2dlog.txt`
   
   rm r2dlog.txt
   
   set_wb
   
   printf "Generating ProResHQ 422\n"
   #find "$FLDR$FILE_NAME"_dng -type f -name "*.dng" -print | \
   #while read dng
   #do
      "$DCRAW" -c -H 0 -6 $Wtemp -W -q 3 *.dng | "$FFMPEG" -f image2pipe -vcodec ppm -r $FPS -i pipe:0 -vcodec prores_kostya -profile:v 3 -vendor ap10 -pix_fmt yuv422p10le -y -r $FPS "$FLDR$FILE_NAME".mov
   #done
   printf "Done\n"
}

here the canon 5d III dcraw white balance rgbg multipliers:

Daylight
1.950195 1.000000 1.605469 1.000000
Shade
2.226562 1.000000 1.369141 1.000000
Cloudy
2.089844 1.000000 1.477539 1.000000
Tungsten
1.412109 1.000000 2.392578 1.000000
White Fluorescent
1.720703 1.000000 2.291016 1.000000

TO DO:
Extract dead pixel list from dng and use it in dcraw to remove dead pixels.

Dcraw wont read the dead pixel information from the source file but needs an external list file.

this is a feature request?

downloading now, Good to hear that you are working on this too
do you think it will be better to have a repo to share our code?

this might make sense at some point. but first have a look at it and then we can see. my coding isn't really great ;-)
note: the wb multipliers in my app are wrong.

Thanks Scrax! It,s working fine . Good work. The ProRes looks fine. Also the white balance issue seems fine. Is the white balance information recorded from what you see in the camera to the ProRes files?
I got a few questions. When I remove the dcraw and ffmpeg from your app, It simply converts the dng;s again. Is that the way I should do if I want to skip the ProRes part? Sometimes only the dng,s would be enough.
Also, is there an alternative If I simply only wants the ProRes files and skip the dng,s? Also, what bitrate and quality is the ProRes in? 422HQ, 444?

A lot of questions, I know. Still, I believe this is one of the best converters out there.
Thanks a lot Scrax!
//D

off topic
Could someone explain or link me to how to convert spanned files on the rawtodng converter? Seem that I,ve missed that part of my raw-education lessons .

White Balance currently uses dcraw automatic - quite ok in lot of cases - not in all...

unfortunately ml does not seem to be record WB in Raw...

the ProRes is about 422HQ with qscale 5 - about 76MBit/s - but it is somewhat dependend on input.

@scrax: to get FPS out of raw2dng:

Code Select Expand

convert_file() {
   cd "$FLDR$FILE_NAME"_dng
   "$RAW2DNG" "$FLDR$FILE_NAME".RAW > r2dlog.txt
   
   FPS=`awk '/FPS/ { print $3; }' "$FLDR$FILE_NAME"_dng/r2dlog.txt`
   
   rm r2dlog.txt
   
   printf "Generating ProResHQ 422\n"

and then in ffmpeg replace the two -r 24 by -r $FPS

there might be a way without saving r2dlog...


i have been playing around with gui creation based on your script - there is pasha to make ui from script. I have it working here with Image preview and manual WB and some more options.
Included ffmbc and dcrawU (i guess U means universal?), tried to build with static libs (no idea if it worked ;-)
https://www.dropbox.com/s/gxm2vvdtbjxl9tl/r2v.zip
i have seen you started with white balance option and cocoadialog... (I will get the rgbg multipliers for the 5d III, can send them to you)

can the ones of you that have issues please do the following:

-right-click on raw2dng.app and select "Show Package Contents"
go to raw2dng.app/Contents/MacOS/
and double-click raw2dng there

this will start raw2dng together with a terminal console

now drag-drop your .raw file on it
look at the Terminal Console and copy/paste the stuff that is output there into here as a comment

this gives  at least the chance of seeing where the problem is.

i've hacked the raw2dng script convert file() to:

Code Select Expand

convert_file() {
      cd "$FLDR$FILE_NAME"_dng
      "$RAW2DNG" "$FLDR$FILE_NAME".RAW
      /opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 *.dng | /usr/local/bin/ffmbc -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores -profile hq -b 120M -pix_fmt yuv444p10 -y -r 24 "$FILE_NAME".mov & >> "$LOG"
}

i switched ffmpeg to ffmbc. It seems to be easier to configure. (the bitrate setting works in ffmbc - i haven't got this to work in ffmpeg. this could be a very easy way to set quality of video - set bitrate and set quality hq, std, lt or proxy)
http://code.google.com/p/ffmbc/

the quality settings - i'm still trying to find the sweet spot. for ffmpeg -profile:v 2 -qscale 6 seems to be too low. i would recommend -profile:v 2 without -qscale. this gives roughly 120MBit/s. -profile:v 3 is about 185MBit/s

the code above for ffmpeg would look like:

Code Select Expand

convert_file() {
      cd "$FLDR$FILE_NAME"_dng
      "$RAW2DNG" "$FLDR$FILE_NAME".RAW
      /opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -a -o 1 *.dng | /usr/local/bin/ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 2 -pix_fmt yuv444p10 -y -r 24 "$FILE_NAME".mov & >> "$LOG"
}

(i removed the dcraw -s 13000 here - this is saturation level for my 5diii - don't think this value is good for other cameras...)

now, if we could make raw2dng pipe as well we could convert .RAW to ProRes (or any other video) directly without .dng files ;-) Somehow i like the idea of archiving the .RAW instead of a bunch of .dng seq..

EDIT: The framerate fixed to 24 is not very nice. the raw2dng reads the framerate when it makes .dng - is there a way to get to this information and put a variable there? 

as addition: if we add a & behind it it gets even faster since all the conversion is done simultaneously with background jobs. (tried to explain as good as possible)

so in my example it looks like:

Code Select Expand

dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 2 -qscale:v 6 -vendor ap10 -pix_fmt yuv422p10le -y -r 24 ProResHigh.mov &

(i removed the -threads 4 from ffmpeg too - doesn't make sense anymore)

a concern for me is still how to set proper White balance and avoid the automatic there...

@scrax
there is away to convert from dng directly to video, without saving images. Using PIPE function. I'm new to this stuff, so i don't really know much about, but here a shell script that works:

Code Select Expand

for f in "$@"; do shell script
cd $f
/opt/local/bin/dcraw -c -H 0 -6 -W -q 3 -S 13000 -a -o 1 ******.dng | /opt/local/bin/ffmpeg -f image2pipe -vcodec ppm -r 24 -i pipe:0 -vcodec prores_kostya -profile:v 3 -qscale:v 5 -vendor ap10 -pix_fmt yuv422p10le -threads 4 -y -r 24 $f/ProResHigh.mov
echo "$f converted"
done

input $@ in this case folders with the dng
the important thing is in draw the "-c" = Write image data to standard output
and in ffmpeg the "-f image2pipe -vcodec ppm" and the " -i pipe:0"

This way is faster plus it does not need that much disk space.

As for having the ffmpeg and ufraw or dcraw as external installation would make sense at the moment, until we found out how to bundle the necessary dylib and whatever is needed.

@scrax,
i think prores -qscale 1 is overkill. I have done some "research" on this the last couple of weeks and came up with this:

ffmpeg -f image2 -i %6d.tiff -r 24 -vcodec prores_kostya -profile:v 2 -qscale:v 6 -vendor ap10 -pix_fmt yuv422p10le -threads 4 -y -r 24 ProResHigh.mov

Let me explain some things:

the first -r 24 is necessary since ffmpeg standard framerate for image sequence would be 25
-prores_kostya is supposed to be a bit better than prores
-profile 0-3 is setting Proxy, LQ, SQ, HQ quality - SQ and HQ give almost same quality. I use 2 or 3 there
-qscale:v 6 Settings between 5 - 10 should be fine. Lower the better but bigger file size.
-vendor ap10 Makes the resulting video file to appear as to be made by quicktime - i think it is more compatible
-pix_fmt yuv422p10le set it to 422. You can use yuv444p10le to make 444. Not sure if it really makes a difference - i rembember i got same file size for 422 and 444 - maybe ffmpeg does something wrong or i do.
-threads 4 just to use all of my 4 threads
the last -r 24 sets output to 24 fps - not entirely sure it's necessary.

as for raw to dng - i found dcraw to work fine and fast - ufraw is based on dcraw.
my settings there:

dcraw -T -H 0 -6 -W -q 3 -S 13000 -a ******.dng

again some info:
-T makes tiff
-H 0 is highlight treating - -H 0 is save to use
-6 makes 16bit output
-W don 't apply auto-brightening of image
-q 3 quality scale
-S 13000 Saturation i found works best for 5DIII - this is not necessary to set
-a is auto WhiteBalance - works ok in most cases - but not in all

I do somehow think the user should have the choice if he wants the raw2dng makes jpg (or tif) and/or video too. What i mean the raw2dng app should be able to do raw to dng conversion only - in case somebody has a different workflow. Not that i think it should not be able to do other things too.

On this whole Auto-Conversion, the thing that bugs me most at the moment is the White Balance setting... Since the dng don't have WB from Camera embedded/included (why actually not? it should! ;-) we need to do it automatically - which, as said, is not optimal in some cases - or do it manually. It just have not found a way yet to make this nicely in batch converting...

Thanks all for your work. bashi

Since i now see the app is basically a shell script, i'm thinking about implementing dcraw and ffmpeg. Dcraw for make tiff out of DNG and ffmpeg to make Video from tiff. I have tinkered the last Days with them and Automator, to batch-convert them.
But there are some issues for me... I'm not familiar with Menu builder (or how its called, the thing to make UI), and it would almost need some UI for it, out of following reasons: for dcraw i would like to set at least the WhiteBalance. Automatic does not work well enough. But this would almost require a preview of at least one DNG file. Other variant would be to have only setting for Tungsten, Daylight and others. (Kelvin would be a bit difficult sine draw does not have Kelvin, but RGB multipliers).
for ffmpeg it would make sense to be able to choose Codec or probably some Preset Codec/Container combination. (ffmpeg shell execution is quite complex, and i'm not a programmer myself, except some basic Shell/Phyton)

About xcode menu builder. Since you have used it somewhat, what you think about it? Is it difficult or relative "easy" to learn? I'm willing to spend some time. Just let me know what you thing about it there.

This for now, and now some questions:
1. Is somebody already planing an app to do this?
2. How do you feel about having this in raw2dng app? Should it be a separate app? (I think it kind of makes sense to have all together, maybe with options to do DNG and/or Tiff (or jpeg) and/or Video.)
3. Do you think there will be World Peace? ;-)

Well, let's see...

Ok, here is the INIT Code that works in all Cases:

INIT
for i in $*
    do
    if [ -d "$i" ]; then
        FILES_FLDR=$i/
        echo "SEARCHING: $FILES_FLDR"
        checkFLDR       
    elif [ -f "$i" ]; then
        file_is=$i
        importFILE       
    else
    echo "Drop a ML .RAW video file and/or folder here"
fi
done
exit 0

@scrax, would you mind testing it, and if it works maybe put it in app if you like?

Hey, that works like a charm. I've had a look at the it and changed the script so it would import all files put on it, if you drop multiple Raw on it, not just the first one.

EDITII:forget the following and look next post ;-)

In INIT after checkFLDR:

elif [ -f "$1" ]; then
for i in $*
do file_is=$i
importFILE
done
else

not so sure about check for $1 and then operate on $*.... but it works in all Chases (Single File drop, Multiple File drop, Folder drop). Thank you.
EDIT: It does not work if drop Folder and Files... So still room for improvement ;-) I might look at this as well.

Ok, great, thanks.

cool job.

A bit less ridiculous?  thanks for the tip.

And this is not changeable at all? If not, we have to live with it, if yes it makes sense to make it work.

Isn't this something that could and should be done in camera? imo clearly yes!

That's a ridiculous workflow suggestion ;-) Why should i copy the App seven times to make seven Raw conversions (at the same time)? lol
Why does it need to copy the Dng at all? the raw2dng in terminal allows direct conversion, not? (raw2dng input output)

Audio Sync Wav (internal) with Raw Video is not working. I think there's something wrong with the Wav. VLC gives me length of over 1 hour for a some Minutes Recording. And it's not just offset but stretched too. Is it really 48000Hz?

Thanks for this Tool.

I have some complaints: I don't think it's very elegant to put the Converted Files into the Raw2Dng.app itself and copy them later... Converting Raw on external Drive - and everything gets copied twice, once to my System HD (Which is not thaat Big - SSD), and then back to drive. Sound like a hack to me ;-)
This as well messes things up, if one wants to convert multiple Raw at the same time, since they overwrite each others results...

Can't it be made that the conversion happens where the RAW file is? (without copying stuff)

This copy thing as well causes my Mac to freeze occasionally.. ;-)