Automatic color grading script for photos (uniform_exposure.py)

[42kwisatzhaderach]

Howdy a1ex, good show with DualISO it's impressive. I'm following in the footsteps of asthanaiitk and I am running into the same errors. The script said that it failed to create the output dir and working dir (though the folders did get created) and then I got the error below. I added print X as you suggested and it returned [].

C:\Python27\ArcGIS10.1>python Uniform_exposure.py
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\_MG_3260.CR2:
--temperature=5500 --green=1
Invalid Parameter - -type
X
[]
Traceback (most recent call last):
  File "Uniform_exposure.py", line 277, in <module>
    mm, mh, ms = get_medians(r)
ValueError: need more than 0 values to unpack


After that I tried to print the lines under get_raw_data_for_median. It returned:

C:\Python27\ArcGIS10.1>python Uniform_exposure.py
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\_MG_3260.CR2:
--temperature=5500 --green=1
dcraw -c -h -4 'raw\_MG_3260.CR2'
convert - -type Grayscale -gravity Center  -scale 500x500 -format %c histogram:i
nfo:-
Invalid Parameter - -type
Traceback (most recent call last):
  File "Uniform_exposure.py", line 277, in <module>
    mm, mh, ms = get_medians(r)
ValueError: need more than 0 values to unpack
 
I guess that means there is something off with '- -type'? I'm out of my league here.

Anywho... if I run C:\Python27\ArcGIS10.1>dcraw -c -h -4 raw\_MG_3260.CR2
I get: "Will not write an image to the terminal!" Hmmm. Any thoughts?

[a1ex]

The complete commnd looks like this:

Code: [Select]

dcraw -c -h -4 IMG_1234.CR2 | convert - -type Grayscale -gravity Center -scale 500x500 -format %c histogram:info:-

If you want to run dcraw as standalone, drop the -c option. Here it's needed in order to pass the image data from dcraw to another program (convert) using a pipe. The dash from convert means the same thing: accept image data from the pipe (here, from dcraw), instead of reading it from a file.

The output of this command should be histogram data.

[42kwisatzhaderach]

Hey; thanks for the speedy reply. I didn't know that convert was a separate program (or about pipes).

If I run: C:\Python27\ArcGIS10.1>dcraw -h -4 raw\_MG_3261.CR2 it does output a *.ppm file (which I suppose is good).

But if I add the rest back I get:
C:\Python27\ArcGIS10.1>dcraw -c -h -4 raw\_MG3261.CR2 | convert - -type Grayscale -gravity Center -scale 500x500 -format %c histogram:info:-
raw\_MG3261.CR2: No such file or directory
Invalid Parameter - -type

It does the same if I run it on an image not in the raw folder:
C:\Python27\ArcGIS10.1>dcraw -c -h -4 _MG3260.CR2 | convert - -type Grayscale -g
ravity Center -scale 500x500 -format %c histogram:info:-
_MG3260.CR2: No such file or directory
Invalid Parameter - -type

[a1ex]

What happens if you try this?

Code: [Select]

convert _MG_3261.ppm -type Grayscale -gravity Center -scale 500x500 -format %c histogram:info:-

Also, what versions of ImageMagick (convert) and dcraw are you using?

[42kwisatzhaderach]

I'm was using ImageMagick-7.0.0-Q16 which is weird because the site says the newest version is 6.8.7-8. I switched to 6.8.7-8 and that sure made something happen (when I ran the code you suggested). It's also possible I just had imagemagick installed in the wrong place though I didn't see convert as an exe in it's folder. Anywho...

New results:
C:\Python27\ArcGIS10.1>python Uniform_exposure.py

raw\_MG_3261.CR2:
--temperature=5500 --green=1
dcraw -c -h -4 'raw\_MG_3261.CR2'
convert - -type Grayscale -gravity Center  -scale 500x500 -format %c histogram:i
nfo:-
    midtones: brightness level  3665 => exposure +1.94 EV
  highlights: brightness level 61357 => exposure +0.72,-0.50 EV
     shadows: brightness level   629 => exposure +0.67 EV (skipping)
 (midtones) (<type 'exceptions.WindowsError'>, WindowsError(2, 'The system canno
t find the file specified'), <traceback object at 0x021DAC88>)
(highlights (<type 'exceptions.WindowsError'>, WindowsError(2, 'The system canno
t find the file specified'), <traceback object at 0x021DAE68>)
. (<type 'exceptions.WindowsError'>, WindowsError(2, 'The system cannot find the
 file specified'), <traceback object at 0x021DA788>)
) (enfuse) (<type 'exceptions.WindowsError'>, WindowsError(2, 'The system cannot
 find the file specified'), <traceback object at 0x021DAE68>)

 [100% done, ETA 0:00:00]...


I'm using enblend-enfuse-4.1.2-win64, and 64bit dcraw that I found here: http://www.fastpictureviewer.com/bin/dcraw.zip (v9.19/1.456, 2013/06/16, x86, x64).

[a1ex]

Alright, so now the image histogram is read properly, now it seems to require ufraw_batch.exe in the executable path.

I now have a fresh copy of Windows 8 installed, so I'll give it a try.

[42kwisatzhaderach]

OK so it seems like I just keep installing things in the wrong place. Thanks for putting up with me. I copied the contents of the ufraw bin folder into the working (python) directory and now it successfully creates the three files in the temp folder.

I am presumably still missing something there though. New output:
C:\Python27\ArcGIS10.1>python Uniform_exposure.py
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\_MG_3261.CR2:
--temperature=5500 --green=1
dcraw -c -h -4 'raw\_MG_3261.CR2'
convert - -type Grayscale -gravity Center  -scale 500x500 -format %c histogram:i
nfo:-
    midtones: brightness level  3665 => exposure +1.94 EV
  highlights: brightness level 61357 => exposure +0.72,-0.50 EV
     shadows: brightness level   629 => exposure +0.67 EV (skipping)
 (midtones) (highlights.) (enfuse) (<type 'exceptions.WindowsError'>, WindowsErr
or(2, 'The system cannot find the file specified'), <traceback object at 0x021DB
300>)

 [100% done, ETA 0:00:00]...

I hope it works for you.

[a1ex]

Check dev.log to see what commands it's trying to run. Or, in function run, print the command before executing it.

[42kwisatzhaderach]

Contents of dev.log:

ufraw-batch --out-type=jpg --overwrite --temperature=5500 --green=1  --exposure=1.93514362066 'raw\_MG_3261.CR2' --output='tmp\_MG_3261-m.jpg' --shrink=1
ufraw-batch: Loaded raw\_MG_3261.CR2
ufraw-batch: Saved C:\Python27\ArcGIS10.1\tmp\_MG_3261-m.jpg

ufraw-batch --out-type=jpg --overwrite --temperature=5500 --green=1  --exposure=0.71757181033 'raw\_MG_3261.CR2' --output='tmp\_MG_3261-h0.jpg' --shrink=1
ufraw-batch: Loaded raw\_MG_3261.CR2
ufraw-batch: Saved C:\Python27\ArcGIS10.1\tmp\_MG_3261-h0.jpg

ufraw-batch --out-type=jpg --overwrite --temperature=5500 --green=1  --exposure=-0.5 'raw\_MG_3261.CR2' --output='tmp\_MG_3261-h1.jpg' --shrink=1
ufraw-batch: Loaded raw\_MG_3261.CR2
ufraw-batch: Saved C:\Python27\ArcGIS10.1\tmp\_MG_3261-h1.jpg

enfuse 'tmp\_MG_3261-m.jpg' 'tmp\_MG_3261-h0.jpg' 'tmp\_MG_3261-h1.jpg' --gray-projector=value --saturation-weight=0 --exposure-sigma=0.3 -o 'jpg\_MG_3261.jpg'

enfuse: info: input image "tmp\_MG_3261-m.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: info: input image "tmp\_MG_3261-h0.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: info: input image "tmp\_MG_3261-h1.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: warning: no usable resolution found in first image "tmp\_MG_3261-m.jpg";
enfuse: warning:   will use 300 dpi
enfuse: info: loading next image: tmp\_MG_3261-m.jpg 1/1
enfuse: info: loading next image: tmp\_MG_3261-h0.jpg 1/1
enfuse: info: loading next image: tmp\_MG_3261-h1.jpg 1/1

exiftool -TagsFromFile 'raw\_MG_3261.CR2' -comment="overall_bias=0; highlight_level=20000; midtone_level=10000; shadow_level=1000; ufraw_options='--temperature=5500 --green=1 '; midtones: brightness level  3665 => exposure +1.94 EV; highlights: brightness level 61357 => exposure +0.72,-0.50 EV ; shadows: brightness level   629 => exposure +0.67 EV (skipping)" -ThumbnailImage= -PreviewImage= -Orientation= -z -overwrite_original 'jpg\_MG_3261.jpg'
    0 image files updated

    1 files weren't updated due to errors


Error: File not found - jpg/_MG_3261.jpg

[a1ex]

Got it working under Windows 8. This was pretty hard, lots of quirks.

Step by step:
- download the script (uniform_exposure.py) and save it somewhere (let's say in c:\expotest)
- download Python 2.7.6 and install it
- download ImageMagick and install it
- download UFRaw, install it, and add C:\Program Files (x86)\UFRaw\bin to your PATH (it's in My Computer, Properties, Advanced, Environment Variables)
- download dcraw.exe and exiftool.exe (I used the ones from cr2hdr.zip) and place them in the same directory as uniform_exposure.py
- download enfuse and place enfuse.exe and vcomp110.dll in the same directory as uniform_exposure.py

Everything installed, now the actual usage:
- create a subdirectory named raw (c:\expotest\raw) and place some CR2 or DNG files there
- double-click on uniform_exposure.py

That's it. The output files will be in the jpg folder.

Advanced: tweak the parameters by editing the script. Optional: edit override_settings to customize each file or group of files. Enable the fullsize flag for maximum quality. When you want to re-develop some files with different settings, delete the JPEGs and run the script again.

[42kwisatzhaderach]

Awesome. I just ran your new python script and that did it (I didn't have to reinstall anything; lucky), thanks for taking the time to write a step by step install, and to trouble shoot. Now I will spend all day batch converting dualISO DNG's.

[shahed26]

awesome. Thanks alex

[engardeknave]

I couldn't get the path to work for UFRaw or ImageMagick. I always have problems using the path statement so I usually include full paths in my scripts. If anyone else has problems with this you can specify paths in the script like this: \"C:\\Program Files (x86)\\UFRaw\\bin\\ufraw-batch.exe\"

Then it turned out I didn't like what it did to my photo. But thanks for introducing me to all these utilities, A1ex. The odyssey of automating my post-process continues.

[a1ex]

My PATH string looks like this:

Code: [Select]

C:\Program Files\ImageMagick-6.8.7-Q16;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\UFRaw\bin

(ImageMagick installer did it automatically, but for ufraw I had to edit it myself).

Can you show an example of what it did to your photo and what you expected to get?

I'm still researching how to make it preserve the percentile levels in the final output (basically exposure and contrast); when that will work, the script will also be suitable for flicker-free video and timelapse.

[engardeknave]

I'm not sure what it's supposed to do exactly besides grade photos to your personal taste. I was thinking that it might make the shadows and highlights more even, and it does seem to be doing this, but not nearly as much as I need. So it wouldn't save me any steps yet. I don't know what you mean by preserve percentile levels. I'm always changing them dramatically.

There were also problems with white balance (very green in some photos) and what looked like some lens correction (didn't auto-trim), and I work with .tifs anyway. Do you still want to see files?

The biggest bottleneck in my processing workflow is manually correcting levels and color in my photos. I'm not sure how much it can be automated.

[a1ex]

I have no idea what you are talking about without seeing some files

[engardeknave]

Ok, so here are three .cr2s and three .jpg outputs. http://we.tl/PiccXQ0yoD

There's also a folder in which I have some finished files. I am not expecting a script to produce files like this. It isn't even a fair comparison, because the finished files come from enfused brackets while the others are single frames. But I didn't think it would make sense not to show you in what general direction I'm hoping to go with this.

[a1ex]

Well, the goal of this script is exactly what you said: automating the levels and color correction. Basically I want all the pictures in a set to look pretty much similar (with customizable look).

The green cast is present in the original CR2. The script doesn't do any adjustments to white balance (it defaults to 5500K and green=1, and you can change it either by editing the script or by tweaking settings in ufraw GUI and saving an ID file in the raw directory).

I'm also thinking at a workflow where you edit a few reference images, you show them to the script as training data, and then all the other pics will get developed with a similar look (similar brightness, contrast, saturation, highlight/shadow levels, stuff like that).

[engardeknave]

I think the really difficult task is grading color in mixed light. I think when I do this I look at the color of the light sources, often reducing the orange/yellow saturation and the overall temperature for indoor shots. Then sometimes I kill the blue saturation coming in from the windows. So the light sources should generally become more white.

So get the brightest areas, look at slightly less bright areas next to them, maybe compare the saturation level of those areas to the other colors in the photo somehow?, then reduce the saturation of the colors in the bright spots (usually yellow/orange/blue/cyan) until their level of saturation is comensurate with other prominent colors in the photo.

[a1ex]

My quick attempt (manual WB and midtone_level = 100000):



(images resized to load faster)

Indeed, the reference shots have a lot more highlight detail. At first I thought "wtf, is ufraw that bad at recovering highlights?!"

[a1ex]

After fine-tuning the script a little, I've set up a folder with reference test images, so you can see what to expect from it. Some of them are my pics, others were taken from this thread or the dual ISO one. Nothing fancy regarding composition; I've only picked some shots I've considered tricky and relevant for testing purposes.

https://www.dropbox.com/sh/013ewzjdjtj7p55/rBM16yO5Rh

I've also ran the same test shots through ufraw with exposure=0 (to see how they would look without any adjustments applied) and through RawTherapee in "auto" mode (to see how some other "auto" algorithm would handle them).

You can also see the "constant median" experiment. It's not enabled by default, since I find it too aggressive on some shots.

You can also notices the current limitations:
- the script is not able to adapt to low-key or high-key scenes (it just makes everything equally brigtht)
- some shots are more contrasty, while others have very little contrast (not sure how to describe it mathematically)
- some underexposed or low-key shots become noisy after processing with this script

Suggestions welcome.

[cineNoisis]

I don't know what i'm doing wrong but it doesn't work for me. It creates the jpg and the tmp folders but nothing else. I have followed the steps, i did tha PATH work etc, but nothing! (Win7 - 64bit)

[a1ex]

Try following these steps: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html

[glubber]

I don't know what i'm doing wrong but it doesn't work for me. It creates the jpg and the tmp folders but nothing else. I have followed the steps, i did tha PATH work etc, but nothing! (Win7 - 64bit)

Plz post the output message of the cmd-window or the dev.log, so other user might have a clue what's going wrong.

If you run uniform_exposure.py from inside the command prompt, the window will stay open.
Little user hint: http://www.tech-recipes.com/rx/232/right-click-to-open-a-command-prompt/

At least that helped me to get the app running 

[cineNoisis]

I was hoping that there are other people having already solved the kind of problem i have with the script, and that the info i gave could help recognizing a specific situation!  I was wrong!
I don't know if it helps, but i tried to drag and drop the .py file to Python cmd and it turned to this https://drive.google.com/file/d/0B5GyTJrcBLXzQXNBMHE4a1lQY0U/edit?usp=sharing

Thanks glubber, i hope what I did is the same!

I opened it with windows cmd now, and we have this https://drive.google.com/file/d/0B5GyTJrcBLXzbkU5c0wtVGhfc2s/edit?usp=sharing