Automatic color grading script for photos (uniform_exposure.py)

[scurrvy2020]

Is anyone mirroring the automatic color grading script?  I'm not able to download it because drop box has stopped the link.

[brapodam]

Is anyone mirroring the automatic color grading script?  I'm not able to download it because drop box has stopped the link.

Here: https://docs.google.com/file/d/0BzFGDdZw7pVwUnFKVjZoQXE4WVU/edit?usp=sharing

[optik]

Forgive my ignorance but where does this script go or what application uses the script the only thing that I have but I remember that uses Python is NUKE.

[Canon eos m]

Forgive my ignorance but where does this script go or what application uses the script the only thing that I have but I remember that uses Python is NUKE.

Optik, I would like to know that too.

I have downloaded UFRAW, Enfuse & GIMP. Another question that I have try to send a picture from UFRAW to GIMP, I get the following message "Error activating GIMP, failed to execute helper program (invalid argument)". Not sure what to do. I have to instead save the file and open separately in GIMP.

You need UFRAW + GIMP because the DNG pictures generated by "Silent Picture" do not open in ACR or PS6.

Please let me know if you figure out what to do with the *.py file.

[dbals]

The first few lines in the code:
"# Develop a bunch of raw pics so they look pretty much equally exposed.
# Copyright (2013) a1ex. License: GPL.

# Requires python, numpy, dcraw, ufraw, enfuse and ImageMagick."


hope this helps,
Dan B

[Silkway]

is there manual to use this script please?

[Silkway]

is there manual to use this script please?

also can anybody put links for all this necessary applications to download?

[scurrvy2020]

Hello,

I've been trying to use this color grading script.  I was not able to find Numpy by itself, so i downloaded the anaconda build of python that supposedly includes numpy and scimpy.  I get this error when I try to run the script:

D:\Pictures\dualiso>python uniform_exposure.py
Qt: Untested Windows version 6.2 detected!
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\D77B4190.DNG:
Invalid Parameter - -type
D:\Install Folder\Anaconda\lib\site-packages\numpy\core\_methods.py:57: RuntimeW
arning: invalid value encountered in double_scalars
  ret = ret / float(rcount)

raw\D77B4191.DNG:
Invalid Parameter - -type

Any help?

[a1ex]

Can you try to find at what line from the script you get the error?

(use some print instructions)

[Audionut]

Sorry for the long delay in response.  I forgot all about this thread.

It's simple when working in stops (log2). ISO 1600 should have 4 stops less DR than ISO100. DXO measures it at only 1stop lower so the gain is 3 stops.

Yes, because the ADC in Canon bodies is noisy.  By using the PGA to boost signal levels before sending the signal to the ADC, the total read noise is reduced.

http://www.sensorgen.info/CanonEOS_5D_MkIII.html

note:  On a quick test I preformed, the gain is 3.64EV on 1 single test shot according to the noise results determined by cr2hdr.  On my normal settings ISO 100/400, the gain is often greater then 2 stops, typically I see 2.3EV increase in DR as reported by cr2hdr.

At -10.5EV SNR is 1.0 .. totally crap .. but I would call it simply crap even at -9EV where SNR is around 4.0 ..

Indeed, @ ISO 200 in my testing, -8.5EV is the limit for my own eyes.

Can you clarify .. what's the cost as you see it ?.

Reduced resolution in the highlights and shadows.

[scurrvy2020]

Can you try to find at what line from the script you get the error?

(use some print instructions)

Hello a1ex,

I inserted some print commands.  I'm not sure if this is how you would have done it but here it is.  This is what I get when I execute:

PS D:\pictures\dualiso> python uniform_exposure.py
0.1
Qt: Untested Windows version 6.2 detected!
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'
1
2
3
4
5
6
9
10     (this is around line 188)
raw\D77B4190.DNG:
Invalid Parameter - -type
m1  (line 112)
D:\Install Folder\Anaconda\lib\site-packages\numpy\core\_methods.py:57: RuntimeWarning: invalid value encountered in dou
ble_scalars
  ret = ret / float(rcount)
m1.1
9
10
raw\D77B4191.DNG:
Invalid Parameter - -type
m1
m1.1
PS D:\pictures\dualiso>


print "4"
def get_medians(file):
    X = get_raw_data_for_median(file)
    print "m1"
    # for midtones
    m = median(X)
    mm = float(m)
    print "m1.1"
    # for highlights
    m = median(X) # 50%
    m = median(X[X > m]) # 75%
    m = median(X[X > m]) # 87.5%
    m = median(X[X > m]) # 93.75%
    m = median(X[X > m]) # 96.875%
    m = median(X[X > m]) # 98.4375%
    m = median(X[X > m])
    m = median(X[X > m])
    m = median(X[X > m])
    m = median(X[X > m])
    mh = float(m) if isfinite(m) else float(max(X))
    print "m2"

[a1ex]

This part does not require numpy:

Code: [Select]

def get_raw_data_for_median(file):
    cmd1 = "dcraw -c -d -4 -h -T '%s'" % file
    cmd2 = "convert - -type Grayscale -gravity Center %s -scale %s -format %%c histogram:info:-" % ("-crop 67%x67%" if samyang8ff else "", "100x100" if dummy_mode else "1000x1000")

    if debug_dcraw: # use this to troubleshoot the dcraw conversion
        cmd_dbg = cmd1 + " | convert - -type Grayscale -gravity Center -scale 500x500 " + change_ext(file, "-debug.jpg")
        print cmd_dbg
        os.system(cmd_dbg)
   
    p1 = subprocess.Popen(shlex.split(cmd1), stdout=subprocess.PIPE)
    p2 = subprocess.Popen(shlex.split(cmd2), stdin=p1.stdout, stdout=subprocess.PIPE)
    lines = p2.communicate()[0].split("\n")
    X = []
    for l in lines[1:]:
        p1 = l.find("(")
        if p1 > 0:
            p2 = l.find(",", p1)
            level = int(l[p1+1:p2])
            count = int(l[:p1-2])
            X.append((level, count))
    return X

def get_percentile(file, percentile):
    X = get_raw_data_for_median(file)
   
    total = sum([count for level, count in X])
    target = total * percentile / 100
   
    acc = 0
    for level, count in X:
        acc += count
        if acc >= target:

            if debug_percentile: # use this to see where the percentile level is
                cmd1 = "dcraw -c -d -4 -h -T '%s'" % file
                cmd_dbg = cmd1 + " | convert - -type Grayscale -gravity Center -threshold %g '%s' " % (level-1, change_ext(file, "-test.jpg"))
                os.system(cmd_dbg)

            return level
    return -1

It's from another script (work in progress for timelapse deflickering), so you need to tweak it a little (you need 3 percentiles here, not just one).

[midnite]

...

[l_d_allan]

Something along these lines: http://www.guillermoluijk.com/article/nonoise/index_en.htm

off-topic, but I was trying to find a link to download the Histogrammar utility described ... (for Win-7-64 in my case). Available?

Or can the equivalent viewing of 16-bit histograms be done with RawDigger?

[djronbxs]

im trying to run the script but im having an error at Line 60:

\uniform_exposure.py", line 60
    except: print "Warning: could not create output dir '%s'" % out_dir
                                                                                           ^
SyntaxError: invalid syntax


can you help me please ?

thanks
ronald

[a1ex]

Try using Python 2, not 3.

[asthanaiitk]

A big thank and hello from me..

I have been enjoying magic lantern and dual iso specifically. However my efforts to make automatic color grading just doesn't work for me even after installing all the mentioned programs. I have installed:

python 2.7
numpy-MKL-1.7.1.win-amd64-py2.7
dcraw
ufraw-0.19.2-2
enfuse 4.0
ImageMagick-6.8.7-0-Q16-x64-dll

I still get "Runtime warning: invalid value encountered in double_scalars ret=ret/float(rcount)"
and "invalid parameter - -type"

I don't have Python programming experience to try to find out the source of error. If anybody knows how to fix it, please help. If somebody who has it working could point to right sources/ combinations of the program as well.. that would help too!

Thanks

[dbast]

Hi A1ex,

thank you for your color grading script.

I'm using it on a Mac running OS X. Here are the steps i did to get the script running including some modifications/additions.

1.    Installing dependencies with homebrew http://brew.sh (should be also possible via MacPorts or Fink):

• brew install dcraw ufraw imagemagick
• and optional: brew install jpegoptim exiftool
  

2.   Manually installing enfuse (it is maybe available via Macports and Fink or homebrew-science someday):

• Download: http://sourceforge.net/projects/enblend/files/enblend-enfuse/enblend-enfuse-4.0/enblend-enfuse-4.0-mac.tar.gz/download e.g. to ~/Downloads
• Uncompress it: cd ~/Downloads; tar -xzf enblend-enfuse-4.0-mac.tar.gz
• Copy it to /usr/bin: sudo cp ~/Downloads/enblend-enfuse-4.0-mac/enfuse-openmp /usr/bin/enfuse

3. Modifications to uniform_exposure.py:

• pylab -> numpy: Changing "from pylab import *" to "from numpy import *", as numpy itself already provides every needed function. (While both python and numpy are available on OS X by default without any need of installation, the script imports pylab instead of numpy, which is neither available by default on OS X nor is it easily installable).
• exclude dot-files: OS X uses hidden dot-files like .DS_Store to store finder data. Exclude dot-files from processing, as it is unlikely a raw-image-filename starts with a dot.
• jpegoptim (optional): Somehow the Huffman tables of enfused jpeg images are not fully compressed. jpegoptim provides lossless (!) and fast optimization of those tables to the maximum, reducing the filesize of some images up to 20% without loosing quality.
• exiftool (optional): By going through developed jpeg-images i sometimes like to have the exif-data and the script-processing parameters at hand, so i copy over the exif data (without old preview/thumbnail-images and without orientation as ufraw already takes care of it) from the raw-image and add the processing parameters as comment (Maybe some more exif-data should be excluded to prevent confusion).

Here is the diff and the code:
https://gist.github.com/dbast/6989576
https://gist.github.com/dbast/6989591

Happy image processing.

Thanks,

Daniel

[a1ex]

Nice! Looking forward to try it out.

You may also want to try these enfuse options: --gray-projector=value --saturation-weight=0 --exposure-sigma=0.5

(still playing with it, but I think these values will handle highlights a bit better).

[asthanaiitk]

Hi
If anybody has working combination of script and needed programs for window 7 64bit, please point to it/ upload.
Thanks

[a1ex]

I've created a repository for this script and added some tweaks, most important being:
- the patch from dbast (EXIF info enabled by default)
- no longer needs numpy
- you can override settings for each image

https://bitbucket.org/a1ex/uniform-exposure/commits/all

I'm also experimenting with capturing electronic level data for each picture in a sidecar file (e.g. IMG_1234.LEV), and this script can use this info to rotate the pictures automatically (in 0.5 degree increments). If anyone is interested in this feature, I'll try to polish it and include it in ML.

[asthanaiitk]

I've created a repository for this script and added some tweaks, most important being:
- the patch from dbast (EXIF info enabled by default)
- no longer needs numpy
- you can override settings for each image

https://bitbucket.org/a1ex/uniform-exposure/commits/all

I'm also experimenting with capturing electronic level data for each picture in a sidecar file (e.g. IMG_1234.LEV), and this script can use this info to rotate the pictures automatically (in 0.5 degree increments). If anyone is interested in this feature, I'll try to polish it and include it in ML.

Hi A1ex

Any idea, why this could be happening:

Traceback (most recent call last):
  File "C:\Camera\Magic Lantern\cr2hdr\uniform_exposure1.py", line 275, in <module>
    mm, mh, ms = get_medians(r)
ValueError: need more than 0 values to unpack

[a1ex]

In get_percentiles, after X = get_raw_data_for_median(file), can you print X?

[asthanaiitk]

In get_percentiles, after X = get_raw_data_for_median(file), can you print X?

X is empty : []

[a1ex]

Then, in get_raw_data_for_median, try to print lines, or try running the dcraw commands (print cmd1, cmd2, then run cmd1 | cmd2 in the command prompt).