Automatic color grading script for photos (uniform_exposure.py)

Started by a1ex, July 10, 2013, 08:35:55 PM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

a1ex

I don't think the script is compatible with Python 3. Patch welcome :D

cineNoisis

Just uninstalled 3 and installed the proper now (while hoping this was the problem only) but unfortunately nothing! the cmd window is doing exactly the same as before, and the python window is doing "nothing" https://drive.google.com/file/d/0B5GyTJrcBLXzSkZIRFVBdHBVRE0/edit?usp=sharing

if it helps i can open teamviewer (i you believe is something that worths checking, and not just a stupid problem that i can't solve) :)


**Just tested to a 2nd PC (clean install Win7 64bit), and the same problem occurs

glubber

Well this looks worser than i thought.  :(

According to my experience (infact it's just guessing) i would use a short path without "blanks". Like 'd:/expotest'.

sorry thats all i can do.
EOS 550D // Sigma 18-200 // Sigma 18-70 // Canon 10-18 STM

dbast

Hi A1ex,

nice to see my code got applied.

While playing around a bit more if found out some things:

- exiftool: dcraw and ufraw-batch have sRGB as default output colorspace, no matter what icc-profile and colorspace the input file has. So theoretical it can be dangerous to copy over the original icc-profile and colorspace with exiftool, because those maybe don't fit to the image anymore and could lead to wrong colors. Therefore the following patch removes the icc-profile and sets the colorspace to sRGB while copying with exiftool (I hope this is enough and things like the "White Point" tag, "Primary Chromaticities" tag or the "Gamma" tag are not applied by any viewer when opening a file as they do it with color profiles before viewing. I did not see that in the wild, this is more like a theoretical fix):
https://gist.github.com/dbast/7760376

- jpeg2tiff: The final output of the whole script is a jpeg file enfused out of many jpeg files. Why not let all itermediate steps to be lossless=tiff and only the last step (=enfuse) to be jpg=lossy? So no information can get lost in any step but the last step. And in the last step it can be controlled/changed by setting a jpeg quality. A little side note: the ufraw-batch default jpeg-quality is 85 and the enfuse default jpeg-quality is 98 (showed by identify -verbose picture.jpg | grep quality). The following patch converts the whole script to do this:
https://gist.github.com/dbast/7760407

With this patch applied the tmp directory gets very big, but well it's the tmp directory and can be deleted from time to time. It is interessting that even the resulting enfused-files are getting bigger while using the same enfuse jpeg quality=98 (the default) as befor, which indicates that less information is lost in the steps befor. Eventhough much biger intermediate images have to be handled, the scipt doesn't get any slower, atleast in my benchmarks.

Both patches applied:
https://gist.github.com/dbast/7760426

Happy image developing.

Daniel

engardeknave

With midtone_level = 100000 it's starting to get a lot more interesting. However, I can't test it much without being able to input my .tifs that came from Enfuse. And I'd need .tif output to use it anyway.

Also is it doing lens correction somewhere? I'm getting a little indentation on the bottom and top of my images.

42kwisatzhaderach

Yeah, I'm getting indentation too but on all sides. To be fair I was @11mm. Here is an example: http://www.douglasaden.com/_MG_3899.jpg

cineNoisis

Quote from: glubber on December 02, 2013, 10:30:12 PM
Well this looks worser than i thought.  :(

According to my experience (infact it's just guessing) i would use a short path without "blanks". Like 'd:/expotest'.

sorry thats all i can do.

I have already tried this (with C: and D:) but without any improvement!

Trying different ways, and i opened it with IDLE and clicked RUN MODULE and it shows this:
>>>
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\IMG_0478.CR2:
--temperature=5500 --green=1

>>>

engardeknave

This is UFRaw automatically lens correcting. Open a file in UFRaw, go to the lens section, click the gear icon to turn off lens correction, then go to options at the bottom and save the configuration.

a1ex

@cineNoisis: to run the script from command-line, don't run it from Python prompt, but from CMD prompt. Follow the instructions from glubber to open a command prompt, then just type uniform_exposure.py in the console.

Then paste the output and the contents of log files (especially dev.log)

Quotewith midtone_level = 100000 it's starting to get a lot more interesting.

Actually I think it's better to raise shadow_level here (the effect should be similar though). I think the EV step size also matters (now it's roughly 1 EV, but in the beginning I tried with only 3 pictures - one for midtones, one for highlights and one for shadows - but got heavy HDR-like halos in some cases).

QuoteAnd I'd need .tif output to use it anyway.

I also thought to add an option for 16-bit tif output. dbast is getting pretty close to this with the patches.

dbast

Oh, I already had the 16bit tiff option in and tested it, but then forgot it. Here are the updated files:
https://gist.github.com/dbast/7760407
https://gist.github.com/dbast/7760426

cineNoisis

Quote from: a1ex on December 03, 2013, 04:40:25 AM
@cineNoisis: to run the script from command-line, don't run it from Python prompt, but from CMD prompt. Follow the instructions from glubber to open a command prompt, then just type uniform_exposure.py in the console.

Then paste the output and the contents of log files (especially dev.log)

As I said before, I have already done this, and the output was this: https://drive.google.com/file/d/0B5GyTJrcBLXzbkU5c0wtVGhfc2s/edit?usp=sharing
The dev.log (which is the only file I get after running the script) is totally empty!

a1ex

Wait a minute, the contents of line 237 does not match the error message. Make sure you have the latest version.

glubber

Did you try to run the command prompt out of the path "D:\projects\cam stuff\automatic color grading script\" instead out of "c:\user\vasilis"?
i just tried your way and i get the same error (upper window)
But "shift+rightclick" works with me (lower window)



     
sorry for short explenation but i'm in a hurry.

EOS 550D // Sigma 18-200 // Sigma 18-70 // Canon 10-18 STM

a1ex

Updates:
- option for 16-bit tif output
- option for customizing EV step size (as in ML auto bracketing)
- minor tweaks, patches from dbast...

dbast

Hi A1ex,

i created a multiprocessing version, by just parallelizing the loop over all files. It needed some refactoring to get a consistent console output. For debugging Pool(1) can be used.

It can generate a lot of IO, that gets even more going fullsize. It benefits from huge amounts of RAM, where the tmp files can be cached. 

On a Core i7 920 (4 Cores) it is about 395% faster than the normal version :)

Patch: https://gist.github.com/dbast/7918513
Patch applied: https://gist.github.com/dbast/7918532

Happy image developing.

Daniel

afre

Hello A1ex, uniform_exposure.py doesn't work after I install ufraw-mod with build_lin.sh. Terminal gets grumpy: "convert.im6: no images defined `histogram:info:-' @ error/convert.c/ConvertImageCommand/3044." I don't know much about this stuff; however, ufraw-batch seems to work fine by itself but imagemagick doesn't seem to agree with the pipe.

a1ex

Noticed it too. Here's a dirty fix until I'll figure out what went wrong:


# HG changeset patch
# User alex@thinkpad
# Date 1388859102 -7200
# Node ID aa4164103e2f46db24f187747a8ee2ef698527ba
# Parent  13e3617e21c1fa9559e888b8d8d59201e79d8a74
Fix a mysterious incompatibility between ufraw tif output and ImageMagick (maybe I've broken something in ufraw-mod?)

diff -r 13e3617e21c1 -r aa4164103e2f uniform_exposure.py
--- a/uniform_exposure.py Wed Dec 04 12:59:57 2013 +0200
+++ b/uniform_exposure.py Sat Jan 04 20:11:42 2014 +0200
@@ -156,7 +156,7 @@
     return X

def get_histogram_data_for_raw(file):
-    cmd1 = 'ufraw-batch --exposure=0 --gamma=1 --clip=digital --shrink=10 --grayscale=luminance --out-depth=16 --output=- --create-id=no --silent "%s"' % file
+    cmd1 = 'ufraw-batch --exposure=0 --gamma=1 --clip=digital --shrink=10 --grayscale=luminance --out-depth=16 --out-type=ppm --output=- --create-id=no --silent "%s"' % file
     cmd2 = "convert - -gravity Center %s -format %%c histogram:info:-" % ("-crop 67%x67%" if samyang8ff else "")
     cmd = cmd1 + " | " + cmd2
     return get_histogram_data_work(file, cmd)
@@ -193,7 +193,7 @@
     if 0:
         level_min = min(ans)
         level_max = max(ans)
-        cmd1 = 'ufraw-batch --exposure=0 --gamma=1 --clip=digital --shrink=10 --grayscale=luminance --out-depth=16 --output=- --silent "%s"' % file
+        cmd1 = 'ufraw-batch --exposure=0 --gamma=1 --clip=digital --shrink=10 --grayscale=luminance --out-depth=16 --out-type=ppm --output=- --silent "%s"' % file
         cmd_dbg = cmd1 + ' | convert - -gravity Center -level %s,%s -solarize 65534 -threshold 0 "%s" ' % (level_min-1, level_max+1, change_ext(file, "-test.jpg"))
         print cmd_dbg
         run(cmd_dbg)

riclim

After run the .py script, it show this, anyone know what happen?

QuotePython 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> ================================ RESTART ================================
>>>
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\IMG_6549.DNG:
--temperature=5500 --green=1

>>>

a1ex

Probably it could not run ufraw (but it seems to have created the output and working folders, so it did something). Did you find any log files?

If not, the only way to troubleshoot it is to run it step-by-step (add some print instructions to see where it stops).

engardeknave

Just stumbled upon this. Results look very compelling. Math looks hilariously impenetrable.

Automated colour grading using colour distribution transfer [pdf]

a1ex

Nice results.

Quote
The overall method takes around a second per image at PAL resolution on a 2 GHz machine using a commercial implementation of the method.

So, there is already a program that implements this, no need to reinvent the wheel :)

gunar

Hi,
First I would like to thank you for the great work!
I use this script as the standard procedure for the Dual ISO images of my 550D.

My questions regards the image resolution:
The raw (CR2) image has a resolution of 5184x3456 pixels.
The resulting DNG (generated with ML's cr2hdr) has half of the vertical resolution, which is understood: 2601x1732 pixels.
The final jpg generated by this uniform_exposure.py script has the same resolution as the DNG - logic: 2601x1732 pixels.

Today I've tried Luminance-HDR for more "dramatic" effects. I used the DNG as input (2601x1732) and choose the output resolution to be 100% of the input file.

Interestingly the outputfile now had the same dimensions as the CR2 input file: 5184x3456.
When I zoom to 100% it looks like all details seem still to be retained. (See 100% comparison center crop at the bottom)  :o

I'm a complete noob, when it comes to technical details.
Can someone explain, why the output of Luminance-HDR has higher resolution than the output of uniform-exposure.py, while the details seem to be retained and the same file was used as input?
Could also uniform_exposure.py output the original resolution of the CR2 rather than of the DNG.

The files I used:
CR2 by 550D using ML Dual-ISO
DNG by ML cr2hdr
JGP by uniform_exposure.py
JPG by Luminance-HDR

Thanks for your help!

Left: output of uniform_exposure.py (2601x1732)    ||   Right: output of Luminance-HDR (5184x3456)

a1ex

By default, this script outputs half-res images (it's faster). For full-res output, edit the script and set "fullsize = True".

gunar


afre

Hello A1ex, How would you determine the level settings? I set highlight_level and midtone_level to the max of my photo set, and shadow_level to the min. I haven't verified this but the max seems to correspond to -NormalWhiteLevel and/or -SpecularWhiteLevel. highlight_level and midtone_level are the same in your script; I did the same for my settings.

Edit: made concise.