raw2dng.app 0.13 Osx GUI development, bugs and updates

[bashi]

downloading now, Good to hear that you are working on this too
do you think it will be better to have a repo to share our code?

this might make sense at some point. but first have a look at it and then we can see. my coding isn't really great ;-)
note: the wb multipliers in my app are wrong.

[bashi]

TO DO:
Extract dead pixel list from dng and use it in dcraw to remove dead pixels.

Dcraw wont read the dead pixel information from the source file but needs an external list file.

this is a feature request?

[bashi]

here the canon 5d III dcraw white balance rgbg multipliers:

Daylight
1.950195 1.000000 1.605469 1.000000
Shade
2.226562 1.000000 1.369141 1.000000
Cloudy
2.089844 1.000000 1.477539 1.000000
Tungsten
1.412109 1.000000 2.392578 1.000000
White Fluorescent
1.720703 1.000000 2.291016 1.000000

[mkorho]

this is a feature request?

Yes it is.

All the 3x and 5x zoom mode raw recording have more than 5 dead pixels, at least on my camera.
It is funny that nobody else is seeing them. (or dont care)
To me red, green and blue pixels in video is a NO NO.
And manual removal is too painful process.

Adobe camera raw and Ufraw can read the dead (hot) pixel list from the source file but dcraw cant.

[bashi]

@scrax

here your script edited for fps and white balance dialog with cocoadialog working:

Code: [Select]

set_wb(){

  "$CD" standard-dropdown \
  --text "Pick a wb" --title "wb Options" --items "Auto" "Daylight" "Shade" "Cloudy" "Tungsten" "White Fluorescent"\
  --string-output | tail -1 > wb.txt
   
  wb=`awk ' { print $1; }' "$FLDR$FILE_NAME"_dng/wb.txt`
  rm wb.txt
  #Set rgb multiplier
  if [[ $wb = Auto ]]
  then
      Wtemp="-a"
 
  elif [[ $wb = *Daylight* ]]
  then
      Wtemp="-r 1.950195 1.000000 1.605469 1.000000"
 
  elif [[ $wb = *Shade* ]]
  then
  Wtemp="-r 2.226562 1.000000 1.369141 1.000000"
 
  elif [[ $wb = *Cloudy* ]]
  then
  Wtemp="-r 2.089844 1.000000 1.477539 1.000000"
 
  elif [[ $wb = *Tungsten* ]]
  then
  Wtemp="-r 1.412109 1.000000 2.392578 1.000000"

  elif [[ $wb = *White* ]]
  then
  Wtemp="-r 1.720703 1.000000 2.291016 1.000000"
 
   
  fi
}

convert_file() {
   cd "$FLDR$FILE_NAME"_dng
   "$RAW2DNG" "$FLDR$FILE_NAME".RAW > r2dlog.txt
   
   FPS=`awk '/FPS/ { print $3; }' "$FLDR$FILE_NAME"_dng/r2dlog.txt`
   
   rm r2dlog.txt
   
   set_wb
   
   printf "Generating ProResHQ 422\n"
   #find "$FLDR$FILE_NAME"_dng -type f -name "*.dng" -print | \
   #while read dng
   #do
      "$DCRAW" -c -H 0 -6 $Wtemp -W -q 3 *.dng | "$FFMPEG" -f image2pipe -vcodec ppm -r $FPS -i pipe:0 -vcodec prores_kostya -profile:v 3 -vendor ap10 -pix_fmt yuv422p10le -y -r $FPS "$FLDR$FILE_NAME".mov
   #done
   printf "Done\n"
}

[bashi]

Yes it is.

All the 3x and 5x zoom mode raw recording have more than 5 dead pixels, at least on my camera.
It is funny that nobody else is seeing them. (or dont care)
To me red, green and blue pixels in video is a NO NO.
And manual removal is too painful process.

Adobe camera raw and Ufraw can read the dead (hot) pixel list from the source file but dcraw cant.

will look into it... (we just switched ufraw to dcraw - might switch back then ;-)
it's not too high of a priority for me though.

[scrax]

@scrax

here your script edited for fps and white balance dialog with cocoadialog working:

I've added fps as you suggested early, now I'm looking into a way for having some options to set, but they should be optional (for advanced use only).

[bashi]

I've added fps as you suggested early, now I'm looking into a way for having some options to set, but they should be optional (for advanced use only).

something like an advanced checkbox on first window perhaps... not really sure how to change platypus default window to do that. so far i'm not getting "warm" with platypus and cocoadialog. as far as i see you can only trigger dialog popup (i might be wrong) which is not great to use.

i think the app (the ideal app) should be something like:
-be as easy as possible
-give as much control over the image (raw->img process) as possible (if desired, advanced option). Since those settings gets "backed" into resulting img seq or video.
-be suitable for batch process

btw, did my app work on your computer?

i don't like on mine:
-no drag drop on app (in file selection does work)
-not nice ui (but hey, a ui ;-)
-no feedback about what it is doing ( while raw2dng and dcraw-video)

[scrax]

We should consider use pashua instead of cocoadialog and keep Platypus as wrapper. Pashua will be good for the preference page maybe.
 Your app is working in making the dng, then it show the options for video rendering but video is not generated.

The big issue I see so far is that in batch conversion setting wb for each file will be not ideal.

My idea is that settings should be something that is related to app and not to raw file. For example if we enable ProRes conversion in settings all the file imported will be converted, if we set a working folder all file go in that folder and so on.
Things like wb, fps etc. should be automatic and right. Maybe ML could make a video.log of raw captures too like for normal video, and we can use that for finding out needed settings...

EDIT: I can open a settings dialog by dropping raw2dng.app into his window

Code: [Select]

set_pref(){
    printf "Preference panel to be added!\n"
}

##### Main

if [[ ! "$1" ]]; then
   usage
else
   if [[ "$1" == *raw2dng.app ]]; then
      set_pref
   fi
   #set_workdir
...


[bashi]

We should consider use pashua instead of cocoadialog and keep Platypus as wrapper. Pashua will be good for the preference page maybe.
 Your app is working in making the dng, then it show the options for video rendering but video is not generated.

The big issue I see so far is that in batch conversion setting wb for each file will be not ideal.

My idea is that settings should be something that is related to app and not to raw file. For example if we enable ProRes conversion in settings all the file imported will be converted, if we set a working folder all file go in that folder and so on.
Things like wb, fps etc. should be automatic and right. Maybe ML could make a video.log of raw captures too like for normal video, and we can use that for finding out needed settings...

EDIT: I can open a settings dialog by dropping raw2dng.app into his window

Code: [Select]

set_pref(){
    printf "Preference panel to be added!\n"
}

##### Main

if [[ ! "$1" ]]; then
   usage
else
   if [[ "$1" == *raw2dng.app ]]; then
      set_pref
   fi
   #set_workdir
...


use platypus and pasha seems ideal. will try this.

what i think of:
on first dialog have checkbox advanced and combo box White Balance with default Auto. if advanced unchecked it will work as usual and convert all raw with this White Balance setting.
if advanced checked it will open for each raw a preview thing to make settings and then render.

additionally on first dialog checkbox image and video, to make img seq and/or video.

this would allow to batch process but as well fine tuning of each. 

EDIT: I can see that it can make sense to edit all videos with the same settings. At the same time i see cases where it would be great to edit each. So an option to edit settings for all or for each could make sense.

[scrax]

So far I've a 0.13 candidate that will use cocoadialog to set working folder and if Prores should be generated or not.

[Danne]

Since we,re on the ProRes topic. Will the files contain similar dynamic range possibilities as with the dng,s? I compared the raw imported with the ginger plugin(trial) and the generated ProRes file. It seems contrasts and highlight information differs. Maybe one could get a flatter profile ProRes file somehow? The first three are the ginger plugin one and the other three are the ProRes. Might it be a white balance thing?

[bashi]

Since we,re on the ProRes topic. Will the files contain similar dynamic range possibilities as with the dng,s? I compared the raw imported with the ginger plugin(trial) and the generated ProRes file. It seems contrasts and highlight information differs. Maybe one could get a flatter profile ProRes file somehow? The first three are the ginger plugin one and the other three are the ProRes. Might it be a white balance thing?

Magic Lantern RAW is 14bit, ProRes is 10bit. There is no way (i know of) to increase ProRes bit depth.
There might be the possibility to output raw video (not sure if it would be 14bit), but FileSize is similar to CinemaDNG, so i don't see it make sense, better work of CinemaDNG sequence then.

the raw converter in use (dcraw) can do gamma correction, so we should be able to get a flatter profile. It can also do Highlight blending and reconstruction. But those things are not in raw2dng.app for now.

White balance has influence on Highlights too.

[Danne]

Thanks a lot for answering

[Jason Montalvo]

Hello and thank you for the new version of raw2dng. Just wanted to report that when the app is generating the prores file it doesn't seem to be able to make a file greater than 645 frames. Any else experienced this ?

Thank You Again !!

[bashi]

Hello and thank you for the new version of raw2dng. Just wanted to report that when the app is generating the prores file it doesn't seem to be able to make a file greater than 645 frames. Any else experienced this ?

Thank You Again !!

thanks for report. not good ;-) will look into it.
If you click in raw2dng.app on the Details button left, after it failed, can you copy the last part of the text output bellow to here (or pasteall.org)?

[Jason Montalvo]

@Bashi

I just tried it again and now it seems to have converted the entire clip to a prores file. I'll try more and let you know if I discover anything.

[Doyle4]

Sweet this version is working well for me and the ProRes output is a nice feature too, drag n drop into afters, change color bit from 8bit to 16 and using adjustment layers all seems sweet!

Iv noticed that it dosnt convert the very last frame so i got 187 out of 188, this isnt a major problem at all but i thought id let you know.

Thanks!

[bashi]

@Bashi

I just tried it again and now it seems to have converted the entire clip to a prores file. I'll try more and let you know if I discover anything.

it's bashi ;-) Good to hear, and please let us know of any issues.

@Doyle4: Thanks for the info.

[noix222]

just converted some files with 0.12 to me it worked pretty good... i still prefer to proRes4444 with AE. It would be amazing if we could batch convert with raw2dng and have the option to create just DNG's or DNG's and proRes... anyway its killing it. thanks guys!

[bashi]

just converted some files with 0.12 to me its worked pretty good... i still prefer to proRes4444 with AE. It would be amazing if we could batch convert with raw2dng and have the option to create just DNG's or DNG's and proRes... anyway its killing it. thanks guys!

“-pix_fmt yuv444p10″ specifies that you want to use the ProRes 4444 codec specifically

we can do that. ;-)

so this would be prores 4444 12bit.
(until now i had some confusion with those pixel formats)

the option to make dng and/or video will come soon.

[noix222]

we can do that. ;-)

so this would be prores 4444 12bit.
(until now i had some confusion with those pixel formats)

the option to make dng and/or video will come soon.

that's awesome thanks!!! its possible to batch convert multiple .RAW files with raw2dng? I went through a lot of time going one by one

[8beeeaaat]

I using RAW module for 60D made by marekk.
http://www.magiclantern.fm/forum/index.php?topic=5653.0

^{UPDATE #9:
Hot/dead pixels removed automatically by ACR (you need fresh raw2dng)
ML with raw_rec compiled hourly by rocket's scripts:
https://www.dropbox.com/sh/1kitzbmanifb6sf/qgkHzW3gn2}


The ProRes files were made by raw2dng.app 0.12 have some Hot/dead pixels.

Thanks.

[budafilms]

Hi Guys, what kind of Pro Res is the file created by RAW2DNG: 422, 4444, LT? Because in Quicktime information only said the creator´s name. Thanks!

[bashi]

that's awesome thanks!!! its possible to batch convert multiple .RAW files with raw2dng? I went through a lot of time going one by one

just drag 'n drop all files on it, or folder containing them.

Hi Guys, what kind of Pro Res is the file created by RAW2DNG: 422, 4444, LT? Because in Quicktime information only said the creator´s name. Thanks!

currently it's 422 HQ, but 4444 is possible and will be enabled.