Author Topic: raw2exr in Linux  (Read 16269 times)

johnnybgood

  • Just arrived
  • *
  • Posts: 1
raw2exr in Linux
« on: June 13, 2013, 06:27:12 PM »
I've been looking for a way of converting RAW (actually DNG) to .exr... I work with VFX in blender and want to try out the raw goodness under Linux (ubuntu). I've kicked out the RAW into DNG, but getting some sort of floating point EXR would be good practice for my workflow.

I've tried a couple of things:

-UFRaw/DCraw doesn't seem to accept .exr output
-Running the .dng through a RAW developing program (darktable). This worked for 500-600 frames, but Darktable seems to only work with very few images- when I want to bring out more than a thousand frames it hits the roof (on a recent, reasonably spec'd PC) and does some very nasty crashing, darktable won't open anymore on my machine..

What would other people here do to convert between these formats?

deleted.account

  • Member
  • ***
  • Posts: 154
Re: raw2exr in Linux
« Reply #1 on: June 13, 2013, 07:41:40 PM »
I use DT both via the GUI or via the command line darktable-cli. Hundreds of frames rather than thousands though.

I've found DT a bit unstable for sure but am running PPA unstable and git builds. You could try using OpenCL in DT  if you're vid card is up to it, that may help.

Regarding floating point EXR's I think they're half float at best, via DT 16bit rather than 32bit full float not that it really matters much about having 32bit, massive files and little to benefit storing images that way for the most of us.

Imagemagick will also give you EXR output, you'll need to compile a hdri version possibly unless things have changed with IM.

Another option might be to use dcraw to go dng to 16bit tif, you'll have control over debayer algo, WB, sensor saturation level, color space but going to tif will bake WB but Blenders tools are totally lacking for raw anyway.

**EDIT**

Here's a simple script I just hacked for the CLI to create 16bit EXR's from the .dng's:

for file in *.dng ; do
dcraw -c -w -H 1 -o 5 -q 3 -4 $file | convert - -depth 16 "./EXR/$(basename "$file" .dng).exr"
done

If you put Raw2dng.exe in a folder along with your dng's, create a sub folder called 'EXR', save the above shell script above in there too, give it execute permissions, make sure you have dcraw and a hdri build of imagemagick installed, then fire up a terminal, cd to folder and start the shell script. You should get 16bit exr's in the EXR folder. :-) Of coarse you can use tif or png for 16bit too but blender will read the exr's as half float or float whatever.

If you import the exr's via the Movie Clip editor change the color space input to XYZ.

If you don't want XYZ color space the dcraw settings and explanation is here:

http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm

So you could change the dcraw settings to suit. I just did:

-c    send to standard out
-w   auto white balance
-h    '1' linear mode no clipped highlights
-o    '5' XYZ color space (0=none (no colour management), 1=sRGB, 2=AdobeRGB, 3=WideGamut, 4=ProPhoto, 5=XYZ)
-q    '3' Best quality Bayer demosaicing (0=bilinear, 1=VNG, 2=PPG, 3=AHD)
-4    Creates linear 16bit rather than 8bit gamma

If raw2dng would pipe to standard out we could avoid creating the dng's at all. Obviously a lot more could be added to the script like mkdir for each named RAW and sending the .dngs or EXR's to those named folders etc but meh...




chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: raw2exr in Linux
« Reply #2 on: June 14, 2013, 11:09:13 AM »
as i've read, the single (small) exr has a dynamicrange of about 30ELV, so its clearly enough for this kind of data :)

deleted.account

  • Member
  • ***
  • Posts: 154
Re: raw2exr in Linux
« Reply #3 on: June 15, 2013, 03:49:51 PM »
Updated the shell script to handle all the raws in a folder.

Creates a folder for each raw using the raw file name -> extracts the dngs to that folder -> creates a sub folder called EXR -> dcraw + IM handles dng to 16bit exr as above. Option to set tmp folder, I use a separate drive for tmp to hold temporary pipe data.

export TMPDIR=/media/......
for file in *.RAW ; do
mkdir './Out/'$file'/'
wine ./raw2dng.exe $file './Out/'$file'/'
mkdir './Out/'$file'/EXR'
   for out in "./Out/$file/*.dng" ; do
      dcraw -c -w -H 1 -o 5 -q 3 -4 $out | convert - -depth 16 "./Out/$file/EXR/%06d.exr"
   done
done

brapodam

  • Member
  • ***
  • Posts: 121
Re: raw2exr in Linux
« Reply #4 on: June 15, 2013, 03:59:45 PM »
Do you absolutely have to import every frame? I notice darktable has a tendency to crash if you import too many photos. If you only need to import a couple frames to preview adjustments, I suggest creating a separate folder and storing a couple of frames inside, and importing them into darktable to edit. After which you can use darktable-cli to export everything based on one xmp file. I don't know about normal video, but this is what I do when I try RAW video time lapses. I just bring in the first and last frame (stored in a subfolder) into darktable, do my adjustments, copy the xmp file into the main folder, and run the bash loop in the main folder:


i=0
for file in *.dng; do
  output=$(printf "exported/%06d.jpg" ${i})
  darktable-cli "${file}" "000000.dng.xmp" "${output}"
  let i++
done

Basically exports everything based on the xmp file of the first frame into a subfolder called exported. You can just change the jpg extension to exr, and the output filename to whatever you want.

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: raw2exr in Linux
« Reply #5 on: June 15, 2013, 04:46:47 PM »

wine ./raw2dng.exe $file './Out/'$file'/'


Why do you use raw2dng.exe in Linux?  I work with raw2dng without exe and without wine.  ;)

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

deleted.account

  • Member
  • ***
  • Posts: 154
Re: raw2exr in Linux
« Reply #6 on: June 15, 2013, 04:49:17 PM »
You're absolutely right, just like the UFRaw workflow listed in the output of the raw2dng converter.

After trying out DT on video, the devs suggested I looked at the darktable-cli for a number of reasons, some relating to the GUI handling and workflow.

http://blendervse.wordpress.com/2013/03/21/darktable-for-video/

The only restriction on darktable-cli I think is it's limited to 8bpp output, so for exporting 16bit formats such as EXR, I don't think it will do it. I may have just been using it wrong though but I got an error message about DT's API.


deleted.account

  • Member
  • ***
  • Posts: 154
Re: raw2exr in Linux
« Reply #7 on: June 15, 2013, 04:52:20 PM »
Why do you use raw2dng.exe in Linux?  I work with raw2dng without exe and without wine.  ;)

Edgar

Simple, I haven't found a Linux binary, I haven't been interested to try compiling it myself, raw2dng like many windows binaries run just as fast under wine, for the purpose of this it does the job. :-) I'm sure I can think of a few more reasons. :-) But if you have a linux binary or link to that you'd like to share ;-) and it works on Ubuntu 13.04 then great.

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: raw2exr in Linux
« Reply #8 on: June 15, 2013, 05:43:02 PM »
I compile it myself. But here you find one:

https://bitbucket.org/OtherOnePercent/tragic-lantern-2.0/downloads

It´s a little bit old. Maybe 1% will upload a new one some day.

Edgar

https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

houz

  • New to the forum
  • *
  • Posts: 2
Re: raw2exr in Linux
« Reply #9 on: June 17, 2013, 01:48:37 PM »
The only restriction on darktable-cli I think is it's limited to 8bpp output, so for exporting 16bit formats such as EXR, I don't think it will do it. I may have just been using it wrong though but I got an error message about DT's API.

While it's still not possible to set the bit depth using the --bpp option there is now a workaround that will allow you to set the bit depth for darktable-cli:

Code: [Select]
darktable-cli "${file}" "000000.dng.xmp" "${output}.png" --hq 1 --core --conf plugins/imageio/format/png/bpp=16
This will export to PNG and force 16 bit output (and enable high quality export). When exporting to EXR you don't need to specify the bit depth, that is always 16 bit.

The --core and --conf options are not in any released version yet, so you have to compile darktable yourself.

That being said, we are currently thinking about supporting video workflows better. Could someone provide a sample raw video? Just a few seconds would be enough. I couldn't find anything to download.

Tobias

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: raw2exr in Linux
« Reply #10 on: June 17, 2013, 07:14:39 PM »


That being said, we are currently thinking about supporting video workflows better. Could someone provide a sample raw video? Just a few seconds would be enough. I couldn't find anything to download.

Tobias

Hallo Tobias

Nice to see a darktable developer in the magiclantern forum.

Before you try to support ml-raw-video in darktable, please talk to Alex about his plans with the raw-video format.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

deleted.account

  • Member
  • ***
  • Posts: 154
Re: raw2exr in Linux
« Reply #11 on: June 17, 2013, 10:46:39 PM »
houz, thanks for the extra info and 16bit route from darktable-cli.

Regarding a raw sample, I guess you're looking for a better one than I emailed to you a few weeks ago? I know it was poor and only off a 550D at 960x408, so totally understanding you looking for a better sample. ;-)

Quote
Before you try to support ml-raw-video in darktable, please talk to Alex about his plans with the raw-video format.

Sure but there's a MASSIVE difference between being able to open a ML Raw file and
Quote
supporting video workflows better.
as I'm sure houz is fully aware, from loading large image sequences, been able to play through an image sequence, even trim in/out's, interpolated keyframed effects and mattes / blending, ffmpeg export even, not suggesting these are the scope of what the DT devs are considering but still massive difference and it's not necessarily only raw format support, color managed, 16bit OpenCL processing in Lab with more photo image orientated processing tools can be very useful to non raw video formats too.

houz

  • New to the forum
  • *
  • Posts: 2
Re: raw2exr in Linux
« Reply #12 on: June 19, 2013, 05:59:41 PM »
Yes, I have your video, and that file is what I use to test my code. But "unfortunately" it doesn't suffer from the stripes issue that gets addressed in raw2dng, so I can't test that stuff with your video.

My current state is that I can load the .RAW file in darktable, I can get the 1st frame, and white balance is correct, too. The rest should come soon, too.

darethehair

  • New to the forum
  • *
  • Posts: 13
Re: raw2exr in Linux
« Reply #13 on: July 11, 2013, 04:51:50 PM »
Why do you use raw2dng.exe in Linux?  I work with raw2dng without exe and without wine.  ;)

Edgar, I am continuing to scan the forum for a solution, but can you tell us how you successfully compiled 'raw2dng' on Linux?  Thanks!

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: raw2exr in Linux
« Reply #14 on: July 11, 2013, 07:46:09 PM »
Edgar, I am continuing to scan the forum for a solution, but can you tell us how you successfully compiled 'raw2dng' on Linux?  Thanks!

Clone MagicLantern unified from the repo to your harddisk.
Set your system so, that you could compile MagicLantern with modules enabled
Goto
Code: [Select]
~/magic-lantern/modules/raw_rec and type
Code: [Select]
makeThat´s all

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

darethehair

  • New to the forum
  • *
  • Posts: 13
Re: raw2exr in Linux
« Reply #15 on: July 12, 2013, 02:50:29 AM »
Clone MagicLantern unified from the repo to your harddisk.
Set your system so, that you could compile MagicLantern with modules enabled
Goto
Code: [Select]
~/magic-lantern/modules/raw_rec and type
Code: [Select]
makeThat´s all

Edgar

Thanks Edgar!  Actually, from what you said, I now realized how non-trivial the task of simply compiling 'raw2dng' is i.e. the entire MagicLantern repository is needed, and (even though this confuses me greatly) I apparently also need an ARM cross-compile environment -- even though I am compiling for a binary on my X86 regular desktop (!).

In any case, from bits and pieces, for the sake of others curious, this is what worked for me:

Code: [Select]
$ cd Downloads
$ wget https://launchpad.net/gcc-arm-embedded/4.7/4.7-2013-q2-update/+download/gcc-arm-none-eabi-4_7-2013q2-20130614-linux.tar.bz2
$ tar -xf gcc-arm-none-eabi-4_7-2013q2-20130614-linux.tar.bz2
$ mv gcc-arm-none-eabi-4_7-2013q2 ~/gcc-arm-non-eabi
$ echo 'export PATH=$PATH:~/gcc-arm-none-eabi/bin' >> ~/.bashrc
$ source ~/.bashrc
...
[cd to where the magic lantern stuff will exist]
...

$ hg clone https://bitbucket.org/hudson/magic-lantern/
$ cd magic-lantern
$ hg update unified
$ cd modules/raw_rec
$ make CROSS_COMPILE=arm-none-eabi- GCC_VERSION=-4.7.4
$ sudo cp raw2dng /usr/local/bin

The new binary 'raw2dng' then looked like it would work if I had a 'raw' file to give it:

Code: [Select]
$ ./raw2dng

usage:

./raw2dng file.raw [prefix]

 => will create prefix000000.dng, prefix0000001.dng and so on.

If there is a simpler way of doing this, I'd sure be interested :)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: raw2exr in Linux
« Reply #16 on: July 12, 2013, 07:47:53 AM »
"make raw2dng" should work without asking for a cross-compiler, though I didn't try to delete mine :D

darethehair

  • New to the forum
  • *
  • Posts: 13
Re: raw2exr in Linux
« Reply #17 on: July 12, 2013, 03:07:59 PM »
"make raw2dng" should work without asking for a cross-compiler, though I didn't try to delete mine :D

Yep, looks like you are correct (which makes sense):

Code: [Select]
$ make raw2dng
[ GCC      ]   raw2dng
[ GCC      ]   raw2dng
[ GCC      ]   raw2dng

I guess if I ever do decide to compile *all* of ML, I will have the infrastructure to do that :)