Magic Lantern Forum

Most of the time I'm too lazy to adjust colors in my photos, so I wrote a script that does this exactly the way I want. It gets roughly 80-90% of pics just right for my taste; for the others I can do manual adjustments.

The script is a few months old, so it was fine-tuned over the time. It's similar to the "Post Deflicker" feature from ML, but optimized for regular photos, not for timelapse.

What it does:
- applies exposure compensation to all photos (so the end result looks consistent)
- performs shadow and highlight recovery, with the "HDR from a single RAW" tonemapping trick and enfuse

Download: uniform_exposure.py (https://bitbucket.org/a1ex/uniform-exposure/raw/tip/uniform_exposure.py) or browse the Mercurial repository (https://bitbucket.org/a1ex/uniform-exposure/).

Usage: look in the comments. There are no command-line params; edit the script to tweak stuff.

Dependencies: python, numpy, dcraw, ufraw, enfuse and ImageMagick.

Sample photo developed with the script, from this DNG, (http://acoutts.com/a1ex/0R0A0640-fullres-soft.DNG) without any post-processing after running the script:

(http://acoutts.com/a1ex/0R0A0640-thumb.jpg) (http://acoutts.com/a1ex/0R0A0640-fullres-soft.jpg)

How I've got that DNG, I'll tell you later these days.

Caveats:
- By default it develops at half-res, for speed. To use full resolution, set --shrink=1 instead of 2.
- I use a modified ufraw, so you may not get exactly the same result for this sample DNG.
- I didn't try it for timelapse. It may flicker a bit more than ML post deflicker, but you can try it.
- Don't expect it to be a speed demon. It will spend several seconds on a raw file. So, first select your shots that you want to keep, then fire the script and let it crunch your pics.
- For Windows, see this step-by-step guide (http://www.magiclantern.fm/forum/index.php?topic=7022.msg90245#msg90245).

Tips & tricks:
- To re-develop a picture with different settings, just delete that jpeg and run the script again.
- You can use any kind of raw files recognized by dcraw (not just Canon). I've tried only CR2 and DNG.

You can post sample pics here, I'd like to see how well it works for you.

Enjoy!

great result, Alex.

If you enable the hot pixel reduction in ufraw with a sensitivity of about 10, than the 2 hot pixels on the lower right side will go, too, I think. :)

Edgar

Any idea what's happening here on a win x64 machine?

raw\D46A7239.CR2:
Invalid Parameter - -type
C:\Python27\lib\site-packages\numpy\core\_methods.py:57: RuntimeWarning: invalid
value encountered in double_scalars
  ret = ret / float(rcount)

Seems promising.

0_0 Testing mode_activated

Cool!

Another sample, this time a video frame from 5D3:

(http://a1ex.magiclantern.fm/bleeding-edge/isoless/luke/batman-compare.jpg)

(bottom left is ISO 100, top right is with a small ADTG trick, at the same shutter speed and aperture; both graded with the above script so I could compare them)

DNG (http://a1ex.magiclantern.fm/bleeding-edge/isoless/luke/000000.dng)

Still fine-tuning the details.

Quote from: a1ex on July 13, 2013, 10:36:47 PM
top right is with a small ADTG trick

Makes a massive difference.

that looks really great

is it possible that i can test this with my 5dmk3...?

greets. swinxx

 :o it looks great!

Quote from: a1ex on July 13, 2013, 10:36:47 PM
Another sample, this time a video frame from 5D3:

(http://a1ex.magiclantern.fm/bleeding-edge/isoless/luke/batman-compare.jpg)

(bottom left is ISO 100, top right is with a small ADTG trick, at the same shutter speed and aperture; both graded with the above script so I could compare them)

DNG (http://a1ex.magiclantern.fm/bleeding-edge/isoless/luke/000000.dng)

Still fine-tuning the details.

ISP ONE HUNDRED... Is the noisy one?
How'd it get so noisy?
Pitch black room, you got it that bright in post?

It looks brighter ISO 100 with 5EV  :o

hello,

the question should not be: how did you get so noisy? :) ..

the question should be: how did you remove so much noise with your next masterpiece of code.. :)

i am thrilled!!!!
++++ great job

Easy, with that theory that almost nobody believed: ISO 1600 is less noisy than ISO 100.

;)

Something along these lines: http://www.guillermoluijk.com/article/nonoise/index_en.htm

Quote from: a1ex on July 14, 2013, 10:50:45 AM
Easy, with that theory that almost nobody believed: ISO 1600 is less noisy than ISO 100.

Lol.  Pretty tricky being able to keep the lighter bits at ISO 100!

More exciting code from the Dev team  :)

It requires a few photos?

No, just one.

You strive for perfection. I like that. I don't understand the tools process though.  :P

Quote from: a1ex on July 14, 2013, 10:50:45 AM
Easy, with that theory that almost nobody believed: ISO 1600 is less noisy than ISO 100.

;)

Something along these lines: http://www.guillermoluijk.com/article/nonoise/index_en.htm

Well, that theory is pretty established, even Dr Emil Martinec made an experiment-presentation using the same idea you use
http://www.dpreview.com/forums/post/28749589
http://www.dpreview.com/forums/post/28750076

The miraculous is the way you gather all this info from a Canon boby in a single file !!!.

It's the best Canon users could hope .. a clear increase of at least 2-3 stops in DR. And this makes Canons Isoless cameras !!.

Can't keep waiting to learn what is this "small ADTG trick".

Quote from: IliasG on July 14, 2013, 11:07:30 PM
what is this "small ADTG trick".

A1ex Delivers The Goodshit.

Quote from: IliasG on July 14, 2013, 11:07:30 PM
It's the best Canon users could hope .. a clear increase of at least 2-3 stops in DR.

3 stops should be the expected increase in DR.

Quote from: a1exIn particular, with the 5D Mark III, at ISO 100 you get 10.97 stops of dynamic range, and at
ISO 1600 you get 9.94 stops according to DxO [5]. When raising the ISO from 100 to 1600, you
are shifting the dynamic range towards the shadows by 4 stops and shortening it by 1 stop. In
other words, you throw away 4 stops of highlight data (they get clipped), and you get in return
3 stops of shadow detail.

16 bit dng  :o

(http://imageshack.us/a/img834/2865/2jb.png)

Quote from: Audionut on July 15, 2013, 02:42:13 AM
3 stops should be the expected increase in DR.

I was way off the mark.

Quote from: Greg on July 15, 2013, 02:53:34 AM
16 bit dng  :o

Indeed.

This isn't the exact process used by the code from a1ex, but should be a good representation of what to expect.

Single ISO 100 shot.
(https://s15.postimg.cc/i81h3vfy3/ISO100.png)

Enfused ISO 100 and ISO 1600 shots.
(https://s15.postimg.cc/3ouc2h28r/enfused.png)

Of course, when a1ex lets us play with his code, it won't take 2 different shots to produce the enfused image above :)

edit:  And some comparisons of extreme shadow areas.

ISO 100 with +4 EV in post.
(https://s15.postimg.cc/53vwr68gr/100.jpg)

Enfused.
(https://s15.postimg.cc/gszwf620b/enused.jpg)


And some examples of not so extreme shadow areas.

ISO 100 with +4 EV in post.
(https://s15.postimg.cc/66639r1kr/100-1.jpg)

Enfused.
(https://s15.postimg.cc/s5chwxfu3/enfused-1.jpg)

I tried running the script and while I got the output files, opening them produces this error message: Could not load image 'imagename.jpg". Error interpreting JPEG image file (Suspension not allowed here)

Anyone knows what is going on? Screenshot:
(http://i155.photobucket.com/albums/s289/brapodam/Screenshot/screenshot_zpsb202f2a4.png)

The errors that appear in the command prompt seem to be related to EXIF info, which shouldn't affect the image

Can you upload dev.log and maybe try to open the intermediate files from tmp?

ImageMagick gave some warnings, but the exposure levels seem to be recognized. Probably something with enfuse (try running it manually on the images from tmp).

Quote from: Audionut on July 15, 2013, 02:42:13 AM
3 stops should be the expected increase in DR.

As Alex wrote, 3 stops is expected for 5DIII. It depends on the model .. for 50D using ISOs range 100-1600 DR increase is expected at a bit more than 2 stops.
It also depends on the ISOs used .. On 5DIII ISOs range 200 - 1600 will give 2 stops 100-3200 will give a bit more than 3 stops ..

Sure, here is the dev.log: https://docs.google.com/file/d/0BzFGDdZw7pVwTmFSTW9TUFdIb3M/edit?usp=sharing

Intermediate files open fine, I'll try enfuse now

EDIT: enfuse produces a file with the same error, though I don't see anything unusual in the command prompt when I run enfuse

Quoteenfuse: info: input image "00000001-h0.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: info: input image "00000001-h1.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: info: input image "00000001-m.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: info: input image "00000001-s0.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: info: input image "00000001-s1.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: warning: no usable resolution found in first image "00000001-h0.jpg";
enfuse: warning:   will use 300 dpi
enfuse: info: loading next image: 00000001-h0.jpg 1/1
enfuse: info: loading next image: 00000001-h1.jpg 1/1
enfuse: info: loading next image: 00000001-m.jpg 1/1
enfuse: info: loading next image: 00000001-s0.jpg 1/1
enfuse: info: loading next image: 00000001-s1.jpg 1/1

Here's a final output file: https://docs.google.com/file/d/0BzFGDdZw7pVwRlZLMzh1d3lMcFU/edit?usp=sharing

Can you upload a final jpeg file?

I have enfuse 4.0-753b534c819d, what version is yours? I know syntax changed a little between 3.x and 4.x, but I'm not using any advanced options here.

Strangely enough though, GIMP is able to open it. Some programs, like the default image viewer in Ubuntu, as well as Avidemux (which I use for time lapse sequences) are unable to open it

Here, imagemagick says:


convert: Sorry, there are legal restrictions on arithmetic coding `00000001.jpg' @ error/jpeg.c/EmitMessage/235.


So... try to output a png or tiff or something:


j  = os.path.join(out_dir, change_ext(f, ".tif"))


Quote from: IliasG on July 15, 2013, 09:37:52 AM
As Alex wrote, 3 stops is expected for 5DIII.

I haven't got my head around how the logarithmic SNR ratio plays into the equation.
Judging from the histogram (and others I've been playing with) I posted above, the tonal quality of a single ISO 100 shot falls to crap around -10.5EV.  The histogram of the enfused shots, show all but 100% tonal quality over the entire 16EV.  That's more then 3 stops improvement :)  I haven't done any testing by eye.

http://www.clarkvision.com/articles/evaluation-canon-5diii/

QuoteTable 2 shows the noise as a function of ISO in image form. The images illustrate several things: 1) lower banding noise at higher ISOs. 2) Better detection of smaller signals at higher ISOs (the random noise decreases). 3) At a certain high ISO, improvements decrease, meaning there is no benefit to higher ISO. Note, ISO is a post sensor gain and does not increase sensitivity. Increasing ISO digitizes a smaller range (see Table 1) but does improve the noise floor. While some banding noise is apparent at ISO 1600, would not be noticeable in many images. At ISO 3200 and higher, banding noise is almost negligible.

Quote from: IliasG on July 15, 2013, 09:37:52 AM
It depends on the model............It also depends on the ISOs used

Indeed.  Just like everything in life, there are compromises.  From what I've seen of a1ex's code so far, this little trick doesn't come for free ;)

Quote from: a1ex on July 15, 2013, 09:53:52 AM
Here, imagemagick says:

Code Select Expand


convert: Sorry, there are legal restrictions on arithmetic coding `00000001.jpg' @ error/jpeg.c/EmitMessage/235.


So... try to output a png or tiff or something:

Code Select Expand


j  = os.path.join(out_dir, change_ext(f, ".tif"))


Ok, setting the output file to png works. Thanks!

Cool.

For timelapse, this script may not be ideal; you probably want the same settings on all files, rather than analyzing every single image and deciding how many exposures it needs.

If you disable highlight/shadow recovery (e.g. by setting the thresholds to extreme values), you should get similar results as with ML post deflicker. Maybe slightly less accurate because of downsampling.

Hi, does this ISO trick also work for video captures? I see you merge 2 ISO images somehow. But that does not work für video production then, unless we capture something static right?

You'll see when I'll announce it. There are still some little glitches in the workflow, and I'll try to publish the code these days.

Quote from: a1ex on July 15, 2013, 10:07:37 AM
Cool.

For timelapse, this script may not be ideal; you probably want the same settings on all files, rather than analyzing every single image and deciding how many exposures it needs.

If you disable highlight/shadow recovery (e.g. by setting the thresholds to extreme values), you should get similar results as with ML post deflicker. Maybe slightly less accurate because of downsampling.

I doubt setting the thresholds to extreme values will help, especially in something like a sunrise/sunset/day to night time lapse. If you set highlight level to the max (65k or something), for the dark photos in the whole sequence, the script will try to push the highlights up, causing what is supposed to be a dark sky to appear bright.

Nevertheless, I'll still try playing around with the thresholds to see if it works. Maybe something like setting the shadow level to 1 but lowering the highlight level from the default value.

UPDATE: setting shadow_level to 1 and highlight_level to 500000 as suggested by a1ex produced results similar to post deflicker (no highlight or shadow recovery) for time lapse purposes. Exposure transitions were smooth for the most part but there were a few inconsistencies towards the end of the video

Quote# from 0 to 65535
highlight_level = 50000
midtone_level = 10000
shadow_level = 1000

The values there can't be 0. Using 0 as the shadow level results in a divide by 0 error  :)

Setting highlight_level to 500000 and shadow_level to 1 should disable any correction. It's out of range, but it works because highlight correction is always negative compared to midtones, shadow correction is always positive.

Can I expect this magical HDR readout mode to read the sensor quickly enough to be hand-holdable, or is it just a way to do HDR capture on a tripod, but in a single file (by effectively doing multiple silent exposures at different ISOs, for example)

I dream of getting clean shadows from handheld shots. I have written a tonemapping program of sorts that benefits enormously from clean shadows, but there are so many times when I don't have a tripod for bracketed shots.

I,m amazed

Quote from: a1ex on July 14, 2013, 02:04:50 PM
No, just one.

I'm not sure I've got it. You have found/invented something that increase the DR of a single photo?

Quote from: a1ex on July 14, 2013, 10:50:45 AM
Easy, with that theory that almost nobody believed: ISO 1600 is less noisy than ISO 100.

;)

Something along these lines: http://www.guillermoluijk.com/article/nonoise/index_en.htm

Dang, I wish I learned about this earlier  ;D I've been dabbling with enfuse by pushing an image in different directions and merging them together and thought, "Damn, it's great that I can see lots of shadow+highlight detail in one exposure now, but man it's noisy in the shadows." Now I gotta grab some new footage while it's still day outside at a high ISO and play with it later.  :o

Quote from: togg on July 16, 2013, 12:14:16 AM
I'm not sure I've got it. You have found/invented something that increase the DR of a single photo?

The DR is there, he just found a good way to squeeze all of it out of the image. I don't think you can "increase the DR of a single photo"; what's captured in a raw photo is stored in the raw photo, you just need to be able to bring it out.

Quote from: brapodam on July 16, 2013, 02:07:11 AM
The DR is there, he just found a good way to squeeze all of it out of the image. I don't think you can "increase the DR of a single photo"; what's captured in a raw photo is stored in the raw photo, you just need to be able to bring it out.

No, I don't think that's the case. The DR is not there; this supposedly actually reduces the noise in the shadows relative to an ordinary image readout.

Im so confused. So far it seams to be a a way to merge 2 exposures but not a standard HDR way. I read a little and from what i understand one exposure is a normal exposure possibly protecting highlights a little and the other is for shadows, not to bring up shadows but to clean them up. But ive always found canon iso to be strange. i thought sensors had a native iso so how can canon have all these iso options. is it just meta data or what? I would love to try this what ever it is.

I guess there is more behind it than simple automatic color grading.
:)

I guess this also mean that now you know how the heck the ISO speed works on a Canon DSLR.
Any hints?

Quote from: Audionut on July 15, 2013, 09:58:07 AM
I haven't got my head around how the logarithmic SNR ratio plays into the equation.

It's simple when working in stops (log2). ISO 1600 should have 4 stops less DR than ISO100. DXO measures it at only 1stop lower so the gain is 3 stops.

QuoteJudging from the histogram (and others I've been playing with) I posted above, the tonal quality of a single ISO 100 shot falls to crap around -10.5EV.  The histogram of the enfused shots, show all but 100% tonal quality over the entire 16EV.  That's more then 3 stops improvement :)  I haven't done any testing by eye.

At -10.5EV SNR is 1.0 .. totally crap .. but I would call it simply crap even at -9EV where SNR is around 4.0 ..

(http://imageshack.us/a/img834/2865/2jb.png)

Look at how many values have population ..  8500 from 50000 available at best. On 0R0A0640-fullres-soft.DNG I count 12800 from 52000 available.
Raw levels density starts at 1:1 at the darks and goes up to 1:9 at highlights. Looks like a 14bit log file linearized at 16bit

QuoteIndeed.  Just like everything in life, there are compromises.  From what I've seen of a1ex's code so far, this little trick doesn't come for free ;)

Can you clarify .. what's the cost as you see it ?.

I hoped, the dual iso trick would work with my 600D. But as I see in the code of alex, it only works on 5d3 and 7d. Nevertheless great work!

Edgar

The mystery should clear up now.

Announced here: http://www.magiclantern.fm/forum/index.php?topic=7139

Really magic !!!!!!

Quote from: a1ex on July 16, 2013, 06:36:37 PM
The mystery should clear up now.

Announced here: http://www.magiclantern.fm/forum/index.php?topic=7139

Now we are talking :D I want to read the pdf to understand how all of that was possible.

Is anyone mirroring the automatic color grading script?  I'm not able to download it because drop box has stopped the link.

Quote from: scurrvy2020 on July 18, 2013, 02:22:45 PM
Is anyone mirroring the automatic color grading script?  I'm not able to download it because drop box has stopped the link.

Here: https://docs.google.com/file/d/0BzFGDdZw7pVwUnFKVjZoQXE4WVU/edit?usp=sharing

Forgive my ignorance but where does this script go or what application uses the script the only thing that I have but I remember that uses Python is NUKE.

Quote from: optik on August 07, 2013, 10:37:40 PM
Forgive my ignorance but where does this script go or what application uses the script the only thing that I have but I remember that uses Python is NUKE.

Optik, I would like to know that too.

I have downloaded UFRAW, Enfuse & GIMP. Another question that I have try to send a picture from UFRAW to GIMP, I get the following message "Error activating GIMP, failed to execute helper program (invalid argument)". Not sure what to do. I have to instead save the file and open separately in GIMP.

You need UFRAW + GIMP because the DNG pictures generated by "Silent Picture" do not open in ACR or PS6.

Please let me know if you figure out what to do with the *.py file.

The first few lines in the code:
"# Develop a bunch of raw pics so they look pretty much equally exposed.
# Copyright (2013) a1ex. License: GPL.

# Requires python, numpy, dcraw, ufraw, enfuse and ImageMagick."


hope this helps,
Dan B

is there manual to use this script please?

Quote from: Silkway on August 12, 2013, 05:16:11 PM
is there manual to use this script please?

also can anybody put links for all this necessary applications to download?

Hello,

I've been trying to use this color grading script.  I was not able to find Numpy by itself, so i downloaded the anaconda build of python that supposedly includes numpy and scimpy.  I get this error when I try to run the script:

D:\Pictures\dualiso>python uniform_exposure.py
Qt: Untested Windows version 6.2 detected!
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\D77B4190.DNG:
Invalid Parameter - -type
D:\Install Folder\Anaconda\lib\site-packages\numpy\core\_methods.py:57: RuntimeW
arning: invalid value encountered in double_scalars
  ret = ret / float(rcount)

raw\D77B4191.DNG:
Invalid Parameter - -type

Any help?

Can you try to find at what line from the script you get the error?

(use some print instructions)

Sorry for the long delay in response.  I forgot all about this thread.

Quote from: IliasG on July 16, 2013, 05:37:06 PM
It's simple when working in stops (log2). ISO 1600 should have 4 stops less DR than ISO100. DXO measures it at only 1stop lower so the gain is 3 stops.

Yes, because the ADC in Canon bodies is noisy.  By using the PGA to boost signal levels before sending the signal to the ADC, the total read noise is reduced.

http://www.sensorgen.info/CanonEOS_5D_MkIII.html

note:  On a quick test I preformed, the gain is 3.64EV on 1 single test shot according to the noise results determined by cr2hdr.  On my normal settings ISO 100/400, the gain is often greater then 2 stops, typically I see 2.3EV increase in DR as reported by cr2hdr.

Quote from: IliasG on July 16, 2013, 05:37:06 PM
At -10.5EV SNR is 1.0 .. totally crap .. but I would call it simply crap even at -9EV where SNR is around 4.0 ..

Indeed, @ ISO 200 in my testing, -8.5EV is the limit for my own eyes.

Quote from: IliasG on July 16, 2013, 05:37:06 PM
Can you clarify .. what's the cost as you see it ?.

Reduced resolution in the highlights and shadows.

Quote from: a1ex on September 07, 2013, 05:06:38 AM
Can you try to find at what line from the script you get the error?

(use some print instructions)

Hello a1ex,

I inserted some print commands.  I'm not sure if this is how you would have done it but here it is.  This is what I get when I execute:

PS D:\pictures\dualiso> python uniform_exposure.py
0.1
Qt: Untested Windows version 6.2 detected!
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'
1
2
3
4
5
6
9
10     (this is around line 188)
raw\D77B4190.DNG:
Invalid Parameter - -type
m1  (line 112)
D:\Install Folder\Anaconda\lib\site-packages\numpy\core\_methods.py:57: RuntimeWarning: invalid value encountered in dou
ble_scalars
  ret = ret / float(rcount)
m1.1
9
10
raw\D77B4191.DNG:
Invalid Parameter - -type
m1
m1.1
PS D:\pictures\dualiso>


print "4"
def get_medians(file):
    X = get_raw_data_for_median(file)
    print "m1"
    # for midtones
    m = median(X)
    mm = float(m)
    print "m1.1"
    # for highlights
    m = median(X) # 50%
    m = median(X[X > m]) # 75%
    m = median(X[X > m]) # 87.5%
    m = median(X[X > m]) # 93.75%
    m = median(X[X > m]) # 96.875%
    m = median(X[X > m]) # 98.4375%
    m = median(X[X > m])
    m = median(X[X > m])
    m = median(X[X > m])
    m = median(X[X > m])
    mh = float(m) if isfinite(m) else float(max(X))
    print "m2"

This part does not require numpy:


def get_raw_data_for_median(file):
    cmd1 = "dcraw -c -d -4 -h -T '%s'" % file
    cmd2 = "convert - -type Grayscale -gravity Center %s -scale %s -format %%c histogram:info:-" % ("-crop 67%x67%" if samyang8ff else "", "100x100" if dummy_mode else "1000x1000")

    if debug_dcraw: # use this to troubleshoot the dcraw conversion
        cmd_dbg = cmd1 + " | convert - -type Grayscale -gravity Center -scale 500x500 " + change_ext(file, "-debug.jpg")
        print cmd_dbg
        os.system(cmd_dbg)
   
    p1 = subprocess.Popen(shlex.split(cmd1), stdout=subprocess.PIPE)
    p2 = subprocess.Popen(shlex.split(cmd2), stdin=p1.stdout, stdout=subprocess.PIPE)
    lines = p2.communicate()[0].split("\n")
    X = []
    for l in lines[1:]:
        p1 = l.find("(")
        if p1 > 0:
            p2 = l.find(",", p1)
            level = int(l[p1+1:p2])
            count = int(l[:p1-2])
            X.append((level, count))
    return X

def get_percentile(file, percentile):
    X = get_raw_data_for_median(file)
   
    total = sum([count for level, count in X])
    target = total * percentile / 100
   
    acc = 0
    for level, count in X:
        acc += count
        if acc >= target:

            if debug_percentile: # use this to see where the percentile level is
                cmd1 = "dcraw -c -d -4 -h -T '%s'" % file
                cmd_dbg = cmd1 + " | convert - -type Grayscale -gravity Center -threshold %g '%s' " % (level-1, change_ext(file, "-test.jpg"))
                os.system(cmd_dbg)

            return level
    return -1


It's from another script (work in progress for timelapse deflickering), so you need to tweak it a little (you need 3 percentiles here, not just one).

...

Quote from: a1ex on July 14, 2013, 10:50:45 AM
Something along these lines: http://www.guillermoluijk.com/article/nonoise/index_en.htm

off-topic, but I was trying to find a link to download the Histogrammar utility described ... (for Win-7-64 in my case). Available?

Or can the equivalent viewing of 16-bit histograms be done with RawDigger?

im trying to run the script but im having an error at Line 60:

\uniform_exposure.py", line 60
    except: print "Warning: could not create output dir '%s'" % out_dir
                                                                                           ^
SyntaxError: invalid syntax


can you help me please ?

thanks
ronald

Try using Python 2, not 3.

A big thank and hello from me..

I have been enjoying magic lantern and dual iso specifically. However my efforts to make automatic color grading just doesn't work for me even after installing all the mentioned programs. I have installed:

python 2.7
numpy-MKL-1.7.1.win-amd64-py2.7
dcraw
ufraw-0.19.2-2
enfuse 4.0
ImageMagick-6.8.7-0-Q16-x64-dll

I still get "Runtime warning: invalid value encountered in double_scalars ret=ret/float(rcount)"
and "invalid parameter - -type"

I don't have Python programming experience to try to find out the source of error. If anybody knows how to fix it, please help. If somebody who has it working could point to right sources/ combinations of the program as well.. that would help too!

Thanks

Hi A1ex,

thank you for your color grading script.

I'm using it on a Mac running OS X. Here are the steps i did to get the script running including some modifications/additions.

1.    Installing dependencies with homebrew http://brew.sh (should be also possible via MacPorts or Fink):

• brew install dcraw ufraw imagemagick
• and optional: brew install jpegoptim exiftool
  

2.   Manually installing enfuse (it is maybe available via Macports and Fink or homebrew-science someday):

• Download: http://sourceforge.net/projects/enblend/files/enblend-enfuse/enblend-enfuse-4.0/enblend-enfuse-4.0-mac.tar.gz/download e.g. to ~/Downloads
• Uncompress it: cd ~/Downloads; tar -xzf enblend-enfuse-4.0-mac.tar.gz
• Copy it to /usr/bin: sudo cp ~/Downloads/enblend-enfuse-4.0-mac/enfuse-openmp /usr/bin/enfuse

3. Modifications to uniform_exposure.py:

• pylab -> numpy: Changing "from pylab import *" to "from numpy import *", as numpy itself already provides every needed function. (While both python and numpy are available on OS X by default without any need of installation, the script imports pylab instead of numpy, which is neither available by default on OS X nor is it easily installable).
• exclude dot-files: OS X uses hidden dot-files like .DS_Store to store finder data. Exclude dot-files from processing, as it is unlikely a raw-image-filename starts with a dot.
• jpegoptim (optional): Somehow the Huffman tables of enfused jpeg images are not fully compressed. jpegoptim provides lossless (!) and fast optimization of those tables to the maximum, reducing the filesize of some images up to 20% without loosing quality.
• exiftool (optional): By going through developed jpeg-images i sometimes like to have the exif-data and the script-processing parameters at hand, so i copy over the exif data (without old preview/thumbnail-images and without orientation as ufraw already takes care of it) from the raw-image and add the processing parameters as comment (Maybe some more exif-data should be excluded to prevent confusion).

Here is the diff and the code:
https://gist.github.com/dbast/6989576
https://gist.github.com/dbast/6989591

Happy image processing.

Thanks,

Daniel

Nice! Looking forward to try it out.

You may also want to try these enfuse options: --gray-projector=value --saturation-weight=0 --exposure-sigma=0.5

(still playing with it, but I think these values will handle highlights a bit better).

Hi
If anybody has working combination of script and needed programs for window 7 64bit, please point to it/ upload.
Thanks

I've created a repository for this script and added some tweaks, most important being:
- the patch from dbast (EXIF info enabled by default)
- no longer needs numpy
- you can override settings for each image

https://bitbucket.org/a1ex/uniform-exposure/commits/all

I'm also experimenting with capturing electronic level data for each picture in a sidecar file (e.g. IMG_1234.LEV), and this script can use this info to rotate the pictures automatically (in 0.5 degree increments). If anyone is interested in this feature, I'll try to polish it and include it in ML.

Quote from: a1ex on October 26, 2013, 01:28:01 PM
I've created a repository for this script and added some tweaks, most important being:
- the patch from dbast (EXIF info enabled by default)
- no longer needs numpy
- you can override settings for each image

https://bitbucket.org/a1ex/uniform-exposure/commits/all

I'm also experimenting with capturing electronic level data for each picture in a sidecar file (e.g. IMG_1234.LEV), and this script can use this info to rotate the pictures automatically (in 0.5 degree increments). If anyone is interested in this feature, I'll try to polish it and include it in ML.

Hi A1ex

Any idea, why this could be happening:

Traceback (most recent call last):
  File "C:\Camera\Magic Lantern\cr2hdr\uniform_exposure1.py", line 275, in <module>
    mm, mh, ms = get_medians(r)
ValueError: need more than 0 values to unpack

In get_percentiles, after X = get_raw_data_for_median(file), can you print X?

Quote from: a1ex on October 27, 2013, 11:20:38 PM
In get_percentiles, after X = get_raw_data_for_median(file), can you print X?

X is empty : []

Then, in get_raw_data_for_median, try to print lines, or try running the dcraw commands (print cmd1, cmd2, then run cmd1 | cmd2 in the command prompt).

Howdy a1ex, good show with DualISO it's impressive. I'm following in the footsteps of asthanaiitk and I am running into the same errors. The script said that it failed to create the output dir and working dir (though the folders did get created) and then I got the error below. I added print X as you suggested and it returned [].

C:\Python27\ArcGIS10.1>python Uniform_exposure.py
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\_MG_3260.CR2:
--temperature=5500 --green=1
Invalid Parameter - -type
X
[]
Traceback (most recent call last):
  File "Uniform_exposure.py", line 277, in <module>
    mm, mh, ms = get_medians(r)
ValueError: need more than 0 values to unpack


After that I tried to print the lines under get_raw_data_for_median. It returned:

C:\Python27\ArcGIS10.1>python Uniform_exposure.py
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\_MG_3260.CR2:
--temperature=5500 --green=1
dcraw -c -h -4 'raw\_MG_3260.CR2'
convert - -type Grayscale -gravity Center  -scale 500x500 -format %c histogram:i
nfo:-
Invalid Parameter - -type
Traceback (most recent call last):
  File "Uniform_exposure.py", line 277, in <module>
    mm, mh, ms = get_medians(r)
ValueError: need more than 0 values to unpack

I guess that means there is something off with '- -type'? I'm out of my league here.

Anywho... if I run C:\Python27\ArcGIS10.1>dcraw -c -h -4 raw\_MG_3260.CR2
I get: "Will not write an image to the terminal!" Hmmm. Any thoughts?

The complete commnd looks like this:

dcraw -c -h -4 IMG_1234.CR2 | convert - -type Grayscale -gravity Center -scale 500x500 -format %c histogram:info:-


If you want to run dcraw as standalone, drop the -c option. Here it's needed in order to pass the image data from dcraw to another program (convert) using a pipe. The dash from convert means the same thing: accept image data from the pipe (here, from dcraw), instead of reading it from a file.

The output of this command should be histogram data.

Hey; thanks for the speedy reply. I didn't know that convert was a separate program (or about pipes).

If I run: C:\Python27\ArcGIS10.1>dcraw -h -4 raw\_MG_3261.CR2 it does output a *.ppm file (which I suppose is good).

But if I add the rest back I get:
C:\Python27\ArcGIS10.1>dcraw -c -h -4 raw\_MG3261.CR2 | convert - -type Grayscale -gravity Center -scale 500x500 -format %c histogram:info:-
raw\_MG3261.CR2: No such file or directory
Invalid Parameter - -type

It does the same if I run it on an image not in the raw folder:
C:\Python27\ArcGIS10.1>dcraw -c -h -4 _MG3260.CR2 | convert - -type Grayscale -g
ravity Center -scale 500x500 -format %c histogram:info:-
_MG3260.CR2: No such file or directory
Invalid Parameter - -type

What happens if you try this?


convert _MG_3261.ppm -type Grayscale -gravity Center -scale 500x500 -format %c histogram:info:-


Also, what versions of ImageMagick (convert) and dcraw are you using?

I'm was using ImageMagick-7.0.0-Q16 which is weird because the site says the newest version is 6.8.7-8. I switched to 6.8.7-8 and that sure made something happen (when I ran the code you suggested). It's also possible I just had imagemagick installed in the wrong place though I didn't see convert as an exe in it's folder. Anywho...

New results:
C:\Python27\ArcGIS10.1>python Uniform_exposure.py

raw\_MG_3261.CR2:
--temperature=5500 --green=1
dcraw -c -h -4 'raw\_MG_3261.CR2'
convert - -type Grayscale -gravity Center  -scale 500x500 -format %c histogram:i
nfo:-
    midtones: brightness level  3665 => exposure +1.94 EV
  highlights: brightness level 61357 => exposure +0.72,-0.50 EV
     shadows: brightness level   629 => exposure +0.67 EV (skipping)
(midtones) (<type 'exceptions.WindowsError'>, WindowsError(2, 'The system canno
t find the file specified'), <traceback object at 0x021DAC88>)
(highlights (<type 'exceptions.WindowsError'>, WindowsError(2, 'The system canno
t find the file specified'), <traceback object at 0x021DAE68>)
. (<type 'exceptions.WindowsError'>, WindowsError(2, 'The system cannot find the
file specified'), <traceback object at 0x021DA788>)
) (enfuse) (<type 'exceptions.WindowsError'>, WindowsError(2, 'The system cannot
find the file specified'), <traceback object at 0x021DAE68>)

[100% done, ETA 0:00:00]...


I'm using enblend-enfuse-4.1.2-win64, and 64bit dcraw that I found here: http://www.fastpictureviewer.com/bin/dcraw.zip (http://www.fastpictureviewer.com/bin/dcraw.zip) (v9.19/1.456, 2013/06/16, x86, x64).

Alright, so now the image histogram is read properly, now it seems to require ufraw_batch.exe in the executable path.

I now have a fresh copy of Windows 8 installed, so I'll give it a try.

OK so it seems like I just keep installing things in the wrong place. Thanks for putting up with me. I copied the contents of the ufraw bin folder into the working (python) directory and now it successfully creates the three files in the temp folder.

I am presumably still missing something there though. New output:
C:\Python27\ArcGIS10.1>python Uniform_exposure.py
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\_MG_3261.CR2:
--temperature=5500 --green=1
dcraw -c -h -4 'raw\_MG_3261.CR2'
convert - -type Grayscale -gravity Center  -scale 500x500 -format %c histogram:i
nfo:-
    midtones: brightness level  3665 => exposure +1.94 EV
  highlights: brightness level 61357 => exposure +0.72,-0.50 EV
     shadows: brightness level   629 => exposure +0.67 EV (skipping)
(midtones) (highlights.) (enfuse) (<type 'exceptions.WindowsError'>, WindowsErr
or(2, 'The system cannot find the file specified'), <traceback object at 0x021DB
300>)

[100% done, ETA 0:00:00]...

I hope it works for you.

Check dev.log to see what commands it's trying to run. Or, in function run, print the command before executing it.

Contents of dev.log:

ufraw-batch --out-type=jpg --overwrite --temperature=5500 --green=1  --exposure=1.93514362066 'raw\_MG_3261.CR2' --output='tmp\_MG_3261-m.jpg' --shrink=1
ufraw-batch: Loaded raw\_MG_3261.CR2
ufraw-batch: Saved C:\Python27\ArcGIS10.1\tmp\_MG_3261-m.jpg

ufraw-batch --out-type=jpg --overwrite --temperature=5500 --green=1  --exposure=0.71757181033 'raw\_MG_3261.CR2' --output='tmp\_MG_3261-h0.jpg' --shrink=1
ufraw-batch: Loaded raw\_MG_3261.CR2
ufraw-batch: Saved C:\Python27\ArcGIS10.1\tmp\_MG_3261-h0.jpg

ufraw-batch --out-type=jpg --overwrite --temperature=5500 --green=1  --exposure=-0.5 'raw\_MG_3261.CR2' --output='tmp\_MG_3261-h1.jpg' --shrink=1
ufraw-batch: Loaded raw\_MG_3261.CR2
ufraw-batch: Saved C:\Python27\ArcGIS10.1\tmp\_MG_3261-h1.jpg

enfuse 'tmp\_MG_3261-m.jpg' 'tmp\_MG_3261-h0.jpg' 'tmp\_MG_3261-h1.jpg' --gray-projector=value --saturation-weight=0 --exposure-sigma=0.3 -o 'jpg\_MG_3261.jpg'

enfuse: info: input image "tmp\_MG_3261-m.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: info: input image "tmp\_MG_3261-h0.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: info: input image "tmp\_MG_3261-h1.jpg" does not have an alpha channel;
enfuse: info: assuming all pixels should contribute to the final image
enfuse: warning: no usable resolution found in first image "tmp\_MG_3261-m.jpg";
enfuse: warning:   will use 300 dpi
enfuse: info: loading next image: tmp\_MG_3261-m.jpg 1/1
enfuse: info: loading next image: tmp\_MG_3261-h0.jpg 1/1
enfuse: info: loading next image: tmp\_MG_3261-h1.jpg 1/1

exiftool -TagsFromFile 'raw\_MG_3261.CR2' -comment="overall_bias=0; highlight_level=20000; midtone_level=10000; shadow_level=1000; ufraw_options='--temperature=5500 --green=1 '; midtones: brightness level  3665 => exposure +1.94 EV; highlights: brightness level 61357 => exposure +0.72,-0.50 EV ; shadows: brightness level   629 => exposure +0.67 EV (skipping)" -ThumbnailImage= -PreviewImage= -Orientation= -z -overwrite_original 'jpg\_MG_3261.jpg'
    0 image files updated

    1 files weren't updated due to errors


Error: File not found - jpg/_MG_3261.jpg

Got it working under Windows 8. This was pretty hard, lots of quirks. (https://bitbucket.org/a1ex/uniform-exposure/commits/all)

Step by step:
- download the script (uniform_exposure.py (https://bitbucket.org/a1ex/uniform-exposure/raw/tip/uniform_exposure.py)) and save it somewhere (let's say in c:\expotest)
- download Python 2.7.6 (http://www.python.org/download/) and install it
- download ImageMagick (http://www.imagemagick.org/script/binary-releases.php#windows) and install it
- download UFRaw (http://sourceforge.net/projects/ufraw/files/ufraw/), install it, and add C:\Program Files (x86)\UFRaw\bin to your PATH (it's in My Computer, Properties, Advanced, Environment Variables)
- download dcraw.exe and exiftool.exe (I used the ones from cr2hdr.zip (http://acoutts.com/a1ex/cr2hdr.zip)) and place them in the same directory as uniform_exposure.py
- download enfuse (http://sourceforge.net/projects/enblend/files/latest/download) and place enfuse.exe and vcomp110.dll in the same directory as uniform_exposure.py

Everything installed, now the actual usage:
- create a subdirectory named raw (c:\expotest\raw) and place some CR2 or DNG files there
- double-click on uniform_exposure.py

That's it. The output files will be in the jpg folder.

Advanced: tweak the parameters by editing the script. Optional: edit override_settings to customize each file or group of files. Enable the fullsize flag for maximum quality. When you want to re-develop some files with different settings, delete the JPEGs and run the script again.

Awesome. I just ran your new python script and that did it (I didn't have to reinstall anything; lucky), thanks for taking the time to write a step by step install, and to trouble shoot. Now I will spend all day batch converting dualISO DNG's.

awesome. Thanks alex

I couldn't get the path to work for UFRaw or ImageMagick. I always have problems using the path statement so I usually include full paths in my scripts. If anyone else has problems with this you can specify paths in the script like this: \"C:\\Program Files (x86)\\UFRaw\\bin\\ufraw-batch.exe\"

Then it turned out I didn't like what it did to my photo. But thanks for introducing me to all these utilities, A1ex. The odyssey of automating my post-process continues.

My PATH string looks like this:

C:\Program Files\ImageMagick-6.8.7-Q16;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\UFRaw\bin


(ImageMagick installer did it automatically, but for ufraw I had to edit it myself).

Can you show an example of what it did to your photo and what you expected to get?

I'm still researching how to make it preserve the percentile levels in the final output (basically exposure and contrast); when that will work, the script will also be suitable for flicker-free video and timelapse.

I'm not sure what it's supposed to do exactly besides grade photos to your personal taste. I was thinking that it might make the shadows and highlights more even, and it does seem to be doing this, but not nearly as much as I need. So it wouldn't save me any steps yet. I don't know what you mean by preserve percentile levels. I'm always changing them dramatically.

There were also problems with white balance (very green in some photos) and what looked like some lens correction (didn't auto-trim), and I work with .tifs anyway. Do you still want to see files?

The biggest bottleneck in my processing workflow is manually correcting levels and color in my photos. I'm not sure how much it can be automated.

I have no idea what you are talking about without seeing some files ;)

Ok, so here are three .cr2s and three .jpg outputs. http://we.tl/PiccXQ0yoD

There's also a folder in which I have some finished files. I am not expecting a script to produce files like this. It isn't even a fair comparison, because the finished files come from enfused brackets while the others are single frames. But I didn't think it would make sense not to show you in what general direction I'm hoping to go with this.

Well, the goal of this script is exactly what you said: automating the levels and color correction. Basically I want all the pictures in a set to look pretty much similar (with customizable look).

The green cast is present in the original CR2. The script doesn't do any adjustments to white balance (it defaults to 5500K and green=1, and you can change it either by editing the script or by tweaking settings in ufraw GUI and saving an ID file in the raw directory).

I'm also thinking at a workflow where you edit a few reference images, you show them to the script as training data, and then all the other pics will get developed with a similar look (similar brightness, contrast, saturation, highlight/shadow levels, stuff like that).

I think the really difficult task is grading color in mixed light. I think when I do this I look at the color of the light sources, often reducing the orange/yellow saturation and the overall temperature for indoor shots. Then sometimes I kill the blue saturation coming in from the windows. So the light sources should generally become more white.

So get the brightest areas, look at slightly less bright areas next to them, maybe compare the saturation level of those areas to the other colors in the photo somehow?, then reduce the saturation of the colors in the bright spots (usually yellow/orange/blue/cyan) until their level of saturation is comensurate with other prominent colors in the photo.

My quick attempt (manual WB and midtone_level = 100000):

(http://a1ex.magiclantern.fm/bleeding-edge/IMG_9330.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/IMG_9283.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/IMG_9931.jpg)

(images resized to load faster)

Indeed, the reference shots have a lot more highlight detail. At first I thought "wtf, is ufraw that bad at recovering highlights?!"

After fine-tuning the script a little, I've set up a folder with reference test images, so you can see what to expect from it. Some of them are my pics, others were taken from this thread or the dual ISO one. Nothing fancy regarding composition; I've only picked some shots I've considered tricky and relevant for testing purposes.

https://www.dropbox.com/sh/013ewzjdjtj7p55/rBM16yO5Rh

I've also ran the same test shots through ufraw with exposure=0 (to see how they would look without any adjustments applied) and through RawTherapee in "auto" mode (to see how some other "auto" algorithm would handle them).

You can also see the "constant median" experiment. It's not enabled by default, since I find it too aggressive on some shots.

You can also notices the current limitations:
- the script is not able to adapt to low-key or high-key scenes (it just makes everything equally brigtht)
- some shots are more contrasty, while others have very little contrast (not sure how to describe it mathematically)
- some underexposed or low-key shots become noisy after processing with this script

Suggestions welcome.

I don't know what i'm doing wrong but it doesn't work for me. It creates the jpg and the tmp folders but nothing else. I have followed the steps, i did tha PATH work etc, but nothing! (Win7 - 64bit)

Try following these steps: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html

Quote from: cineNoisis on December 02, 2013, 04:12:04 PM
I don't know what i'm doing wrong but it doesn't work for me. It creates the jpg and the tmp folders but nothing else. I have followed the steps, i did tha PATH work etc, but nothing! (Win7 - 64bit)

Plz post the output message of the cmd-window or the dev.log, so other user might have a clue what's going wrong.

If you run uniform_exposure.py from inside the command prompt, the window will stay open.
Little user hint: http://www.tech-recipes.com/rx/232/right-click-to-open-a-command-prompt/ (http://www.tech-recipes.com/rx/232/right-click-to-open-a-command-prompt/)

At least that helped me to get the app running  :)

I was hoping that there are other people having already solved the kind of problem i have with the script, and that the info i gave could help recognizing a specific situation!  I was wrong! :)
I don't know if it helps, but i tried to drag and drop the .py file to Python cmd and it turned to this https://drive.google.com/file/d/0B5GyTJrcBLXzQXNBMHE4a1lQY0U/edit?usp=sharing

Thanks glubber, i hope what I did is the same!

I opened it with windows cmd now, and we have this https://drive.google.com/file/d/0B5GyTJrcBLXzbkU5c0wtVGhfc2s/edit?usp=sharing

I don't think the script is compatible with Python 3. Patch welcome :D

Just uninstalled 3 and installed the proper now (while hoping this was the problem only) but unfortunately nothing! the cmd window is doing exactly the same as before, and the python window is doing "nothing" https://drive.google.com/file/d/0B5GyTJrcBLXzSkZIRFVBdHBVRE0/edit?usp=sharing

if it helps i can open teamviewer (i you believe is something that worths checking, and not just a stupid problem that i can't solve) :)


**Just tested to a 2nd PC (clean install Win7 64bit), and the same problem occurs

Well this looks worser than i thought.  :(

According to my experience (infact it's just guessing) i would use a short path without "blanks". Like 'd:/expotest'.

sorry thats all i can do.

Hi A1ex,

nice to see my code got applied.

While playing around a bit more if found out some things:

- exiftool: dcraw and ufraw-batch have sRGB as default output colorspace, no matter what icc-profile and colorspace the input file has. So theoretical it can be dangerous to copy over the original icc-profile and colorspace with exiftool, because those maybe don't fit to the image anymore and could lead to wrong colors. Therefore the following patch removes the icc-profile and sets the colorspace to sRGB while copying with exiftool (I hope this is enough and things like the "White Point" tag, "Primary Chromaticities" tag or the "Gamma" tag are not applied by any viewer when opening a file as they do it with color profiles before viewing. I did not see that in the wild, this is more like a theoretical fix):
https://gist.github.com/dbast/7760376

- jpeg2tiff: The final output of the whole script is a jpeg file enfused out of many jpeg files. Why not let all itermediate steps to be lossless=tiff and only the last step (=enfuse) to be jpg=lossy? So no information can get lost in any step but the last step. And in the last step it can be controlled/changed by setting a jpeg quality. A little side note: the ufraw-batch default jpeg-quality is 85 and the enfuse default jpeg-quality is 98 (showed by identify -verbose picture.jpg | grep quality). The following patch converts the whole script to do this:
https://gist.github.com/dbast/7760407

With this patch applied the tmp directory gets very big, but well it's the tmp directory and can be deleted from time to time. It is interessting that even the resulting enfused-files are getting bigger while using the same enfuse jpeg quality=98 (the default) as befor, which indicates that less information is lost in the steps befor. Eventhough much biger intermediate images have to be handled, the scipt doesn't get any slower, atleast in my benchmarks.

Both patches applied:
https://gist.github.com/dbast/7760426

Happy image developing.

Daniel

With midtone_level = 100000 it's starting to get a lot more interesting. However, I can't test it much without being able to input my .tifs that came from Enfuse. And I'd need .tif output to use it anyway.

Also is it doing lens correction somewhere? I'm getting a little indentation on the bottom and top of my images.

Yeah, I'm getting indentation too but on all sides. To be fair I was @11mm. Here is an example: http://www.douglasaden.com/_MG_3899.jpg (http://www.douglasaden.com/_MG_3899.jpg)

Quote from: glubber on December 02, 2013, 10:30:12 PM
Well this looks worser than i thought.  :(

According to my experience (infact it's just guessing) i would use a short path without "blanks". Like 'd:/expotest'.

sorry thats all i can do.

I have already tried this (with C: and D:) but without any improvement!

Trying different ways, and i opened it with IDLE and clicked RUN MODULE and it shows this:
>>>
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\IMG_0478.CR2:
--temperature=5500 --green=1

>>>

This is UFRaw automatically lens correcting. Open a file in UFRaw, go to the lens section, click the gear icon to turn off lens correction, then go to options at the bottom and save the configuration.

@cineNoisis: to run the script from command-line, don't run it from Python prompt, but from CMD prompt. Follow the instructions from glubber to open a command prompt, then just type uniform_exposure.py in the console.

Then paste the output and the contents of log files (especially dev.log)

Quotewith midtone_level = 100000 it's starting to get a lot more interesting.

Actually I think it's better to raise shadow_level here (the effect should be similar though). I think the EV step size also matters (now it's roughly 1 EV, but in the beginning I tried with only 3 pictures - one for midtones, one for highlights and one for shadows - but got heavy HDR-like halos in some cases).

QuoteAnd I'd need .tif output to use it anyway.

I also thought to add an option for 16-bit tif output. dbast is getting pretty close to this with the patches.

Oh, I already had the 16bit tiff option in and tested it, but then forgot it. Here are the updated files:
https://gist.github.com/dbast/7760407
https://gist.github.com/dbast/7760426

Quote from: a1ex on December 03, 2013, 04:40:25 AM
@cineNoisis: to run the script from command-line, don't run it from Python prompt, but from CMD prompt. Follow the instructions from glubber to open a command prompt, then just type uniform_exposure.py in the console.

Then paste the output and the contents of log files (especially dev.log)

As I said before, I have already done this, and the output was this: https://drive.google.com/file/d/0B5GyTJrcBLXzbkU5c0wtVGhfc2s/edit?usp=sharing
The dev.log (which is the only file I get after running the script) is totally empty!

Wait a minute, the contents of line 237 does not match the error message. Make sure you have the latest version.

Did you try to run the command prompt out of the path "D:\projects\cam stuff\automatic color grading script\" instead out of "c:\user\vasilis"?
i just tried your way and i get the same error (upper window)
But "shift+rightclick" works with me (lower window)


(http://img223.imagevenue.com/loc1059/th_086639566_python1_122_1059lo.jpg) (http://img223.imagevenue.com/img.php?image=086639566_python1_122_1059lo.jpg)
     
sorry for short explenation but i'm in a hurry.

Updates:
- option for 16-bit tif output
- option for customizing EV step size (as in ML auto bracketing)
- minor tweaks, patches from dbast...

Hi A1ex,

i created a multiprocessing version, by just parallelizing the loop over all files. It needed some refactoring to get a consistent console output. For debugging Pool(1) can be used.

It can generate a lot of IO, that gets even more going fullsize. It benefits from huge amounts of RAM, where the tmp files can be cached. 

On a Core i7 920 (4 Cores) it is about 395% faster than the normal version :)

Patch: https://gist.github.com/dbast/7918513
Patch applied: https://gist.github.com/dbast/7918532

Happy image developing.

Daniel

Hello A1ex, uniform_exposure.py doesn't work after I install ufraw-mod with build_lin.sh. Terminal gets grumpy: "convert.im6: no images defined `histogram:info:-' @ error/convert.c/ConvertImageCommand/3044." I don't know much about this stuff; however, ufraw-batch seems to work fine by itself but imagemagick doesn't seem to agree with the pipe.

Noticed it too. Here's a dirty fix until I'll figure out what went wrong:


# HG changeset patch
# User alex@thinkpad
# Date 1388859102 -7200
# Node ID aa4164103e2f46db24f187747a8ee2ef698527ba
# Parent  13e3617e21c1fa9559e888b8d8d59201e79d8a74
Fix a mysterious incompatibility between ufraw tif output and ImageMagick (maybe I've broken something in ufraw-mod?)

diff -r 13e3617e21c1 -r aa4164103e2f uniform_exposure.py
--- a/uniform_exposure.py Wed Dec 04 12:59:57 2013 +0200
+++ b/uniform_exposure.py Sat Jan 04 20:11:42 2014 +0200
@@ -156,7 +156,7 @@
     return X

def get_histogram_data_for_raw(file):
-    cmd1 = 'ufraw-batch --exposure=0 --gamma=1 --clip=digital --shrink=10 --grayscale=luminance --out-depth=16 --output=- --create-id=no --silent "%s"' % file
+    cmd1 = 'ufraw-batch --exposure=0 --gamma=1 --clip=digital --shrink=10 --grayscale=luminance --out-depth=16 --out-type=ppm --output=- --create-id=no --silent "%s"' % file
     cmd2 = "convert - -gravity Center %s -format %%c histogram:info:-" % ("-crop 67%x67%" if samyang8ff else "")
     cmd = cmd1 + " | " + cmd2
     return get_histogram_data_work(file, cmd)
@@ -193,7 +193,7 @@
     if 0:
         level_min = min(ans)
         level_max = max(ans)
-        cmd1 = 'ufraw-batch --exposure=0 --gamma=1 --clip=digital --shrink=10 --grayscale=luminance --out-depth=16 --output=- --silent "%s"' % file
+        cmd1 = 'ufraw-batch --exposure=0 --gamma=1 --clip=digital --shrink=10 --grayscale=luminance --out-depth=16 --out-type=ppm --output=- --silent "%s"' % file
         cmd_dbg = cmd1 + ' | convert - -gravity Center -level %s,%s -solarize 65534 -threshold 0 "%s" ' % (level_min-1, level_max+1, change_ext(file, "-test.jpg"))
         print cmd_dbg
         run(cmd_dbg)


After run the .py script, it show this, anyone know what happen?

QuotePython 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> ================================ RESTART ================================
>>>
Warning: could not create output dir 'jpg'
Warning: could not create working dir 'tmp'

raw\IMG_6549.DNG:
--temperature=5500 --green=1

>>>

Probably it could not run ufraw (but it seems to have created the output and working folders, so it did something). Did you find any log files?

If not, the only way to troubleshoot it is to run it step-by-step (add some print instructions to see where it stops).

Just stumbled upon this. Results look very compelling. Math looks hilariously impenetrable.

Automated colour grading using colour distribution transfer (http://web4.cs.ucl.ac.uk/teaching/3085/archive/2010/labs5/fpitie.pdf) [pdf]

Nice results.

Quote
The overall method takes around a second per image at PAL resolution on a 2 GHz machine using a commercial implementation of the method.

So, there is already a program that implements this, no need to reinvent the wheel :)

Hi,
First I would like to thank you for the great work!
I use this script as the standard procedure for the Dual ISO images of my 550D.

My questions regards the image resolution:
The raw (CR2) image has a resolution of 5184x3456 pixels.
The resulting DNG (generated with ML's cr2hdr) has half of the vertical resolution, which is understood: 2601x1732 pixels.
The final jpg generated by this uniform_exposure.py script has the same resolution as the DNG - logic: 2601x1732 pixels.

Today I've tried Luminance-HDR for more "dramatic" effects. I used the DNG as input (2601x1732) and choose the output resolution to be 100% of the input file.

Interestingly the outputfile now had the same dimensions as the CR2 input file: 5184x3456.
When I zoom to 100% it looks like all details seem still to be retained. (See 100% comparison center crop at the bottom)  :o

I'm a complete noob, when it comes to technical details.
Can someone explain, why the output of Luminance-HDR has higher resolution than the output of uniform-exposure.py, while the details seem to be retained and the same file was used as input?
Could also uniform_exposure.py output the original resolution of the CR2 rather than of the DNG.

The files I used:
CR2 by 550D using ML Dual-ISO (https://dl-web.dropbox.com/get/dual_iso_resolution/IMG_1770.CR2?_subject_uid=4080925&w=AACSSUoxRHBE1ZYn6aMKBd8dx3McmRDBIvuG2bZHzdtNqQ)
DNG by ML cr2hdr (https://dl-web.dropbox.com/get/dual_iso_resolution/IMG_1770.DNG?_subject_uid=4080925&w=AAAFBbY54ECUWoZ7tntenwC4t6sOx5Z53YfqkdUOaKlFSw)
JGP by uniform_exposure.py (https://dl-web.dropbox.com/get/dual_iso_resolution/IMG_1770.jpg?_subject_uid=4080925&w=AADHP2f7ArFQb962X0oA6YphekRD4HyMaalZ_ARY2NyMoA)
JPG by Luminance-HDR (https://dl-web.dropbox.com/get/dual_iso_resolution/IMG_1770_pregamma_1_mantiuk06_contrast_mapping_0.1_saturation_factor_2_detail_factor_5.JPEG?_subject_uid=4080925&w=AABMSUIk5ccoqP-SjWgJwHgY6AKMosUEooEW7xXabHfeOg)

Thanks for your help!

Left: output of uniform_exposure.py (2601x1732)    ||   Right: output of Luminance-HDR (5184x3456)
(https://dl-web.dropbox.com/get/dual_iso_resolution/compare@100%.png?_subject_uid=4080925&w=AACgv-gh1NEaU9rSHkf9n6ULMZr4Vwa3xttuwyfSqUdz7w)

By default, this script outputs half-res images (it's faster). For full-res output, edit the script and set "fullsize = True".

Great, thanks!

Just tried it out and it works for me...

Hello A1ex, How would you determine the level settings? I set highlight_level and midtone_level to the max of my photo set, and shadow_level to the min. I haven't verified this but the max seems to correspond to -NormalWhiteLevel and/or -SpecularWhiteLevel. highlight_level and midtone_level are the same in your script; I did the same for my settings.

Edit: made concise.

Highlight, midtone and shadow levels are used to decide what exposures to use for developing the image. Values are normally from 0 to 65535 (though you can use higher values to push the exposure even more).

From this:

def get_medians(file):
    return get_percentiles(file, [50, 99.99, 1])


you can see that:

- Midtones are the median brightness of the image. So, if you don't use any highlight/shadow recovery (that is, no HDR processing, just a plain curve), you will get JPEGs with the same median brightness. That's effectively the deflicker algorithm used in ML for timelapse.

- Highlights are metered at 99.99th percentile (so, the brightest 0.01% of all pixels will be considered "specular highlights" and ignored). A lower highlight_level will reduce the exposure used for highlight recovery (so it will recover more highlights).

Note: the darkest image will be developed at no less than -0.5 EV, because usually there's no more useful highlight detail below this level.

- Shadows are metered at 1st percentile (maybe I should choose the 5th percentile, as in ETTR). This would basically meter the darkest object in your scene. So, if you want brighter shadows, increase shadow_level.

So, midtone_level gives the basic exposure (for mitdones), and the other two are used to decide whether you need highlight/shadow recovery (and how much). If the exposure values are too far apart, it will insert some intermediate values, according to ev_step (because enfuse seems to give less ghosting with small EV increments in the bracketed sequence).

Once it decides the exposure values, the script invokes ufraw and enfuse, which do the hard work.

Tip: to disable highlight/shadow recovery, set highlight_level = 1e10 and shadow_level = 1 (extreme values, which will be interpreted as "no highlight/shadow correction needed"). If you do that, the script will do a simple deflickering, with no HDR processing.

Thanks A1ex, The reason for my query is that I feel like my settings are arbitrary. I would like to set things with more precision and confidence, getting a desired result without having to wrestle with the parameters too much. E.g. apply 10% highlight and 5% shadow recovery. It's meant to save time after all and I don't think it's doing that for me at the moment ::)

Hi everyone,

First of all, thank you for this amazing software and more generally for magic lantern.
I realize each step described in Reply #84 by a1ex (each version of each software), but I have the same problem that cineNoisis. When I launch the program  two folders are created (jpg and tmp) but they are empty.
I have tried several paths but none works.

I have the following message:

I also try to patch ufraw but I have the following error message:

I use windows 8.1 pro x64
I am sorry but I am not familiar with coding, so any help will be welcome

Regards

Hi again, maybe I have found something but I still have troubles.
My ms-dos starts automatically with C:\Users\Sebastian. So I put all files in this directory and afterward I get this new error message:
Temperature=5500 –green=1
'convert' is not recognized as internal or external command
Before I moved the files to this directory the script also created empty jpg and tmp folders in C:\Users\Sebastian.
I hope that could help understand my problem (or maybe not)

The last error means you should also install ImageMagick (which provides the "convert" command).

Hi a1ex,

Effectively it was a problem of ImageMagic, the first time I didn't pick all the options.
But (unfortunately) now at the end of the installation I have this error message:
unable to execute file: ppm
shellExecuteEx failed; code 2.
The file specified cannot be found
But at the end a dialog says the installation is done. I can run 'uniform_exposure.py' but my picture is very strange.
So I don't know if it has something to do with the error message or if I have to change the parameters of 'uniform_exposure.py' (I suppose it is still a problem with Image Magic).

http://www.casimages.com/i/150310092529869962.jpg.html (http://www.casimages.com/i/150310092529869962.jpg.html)

Thank you again.

Your image is a dual ISO one? If so, you need to convert it first to DNG with cr2hdr.

(you can leave both the CR2 and DNG in the raw directory - the script will pick the DNG if it finds both files)

OK nice! thank you very much

hello, how do I change the file output quality in uniform_exposure script
for 100% or change the output to .tif, I would use for my photos. the script is many great but would like to customize it to my needs. I know some programming. thank you and thanks for the great work team ML.

Quote from: 88oak88 on March 10, 2015, 11:52:20 PM
[...]change the output to .tif [...] I know some programming. [...]

In this case, finding the answer is as simple as opening the script and pressing CTRL-F, T, I, F (at least with Firefox).

resolved modified and running script. thank you

...is there a way of generating roughly the correct amount of midtone/highlight/shadow levels needed for a specific image?  ...maybe based on the histogram?  ...or is it just based on try and error to get a pleasant result?  :-\

Yes, take a look at "override_settings" (there is an example commented out).

...ok fine but how do you achieve the calculation of the "best guest" of the midtone/highlight/shadow ?  ...maybe by doing a histogram redistribution/equalisation and comparing with the original?  (don't know I'm just speculating or talking rubbish!!)

Otherwise did you see my PM?  ;)

The script looks at the histogram to decide the midtone exposure level, and how much highlight/shadow correction is needed for each image. Usually I run a test pass at low resolution (faster), then take a quick look at all the pictures, and I override the settings for those pictures that don't look the way I like (just trial and error). I also adjust other things like WB, saturation or crop/rotate in ufraw (and save .ufraw files where needed - the script will use them if they are present). Once those adjustments are done, I run the script again at higher resolution for the final output.

Hello.
I write from Italy (google translator)
I'm in love with ME and Dual Iso and I'm starting to use the fine program of self-correction.
I wondered if it was possible to add an option that would allow to have outbound another CR2 or DNG with its file .xmp so as to re-open it directly in Camera Raw (curves have already settled) and deal with only the color?
It could be an idea that still allows you to manage the final phase of the conversion.

Did anyone already have the idea to clone / transform / code this script for Lightroom or implement it as a Photoshop script / plugin. Heck, I even don't know if that would be possible but it came just up to my mind...

As a starting point, one can use dmilligan's deflicker script: www.magiclantern.fm/forum/index.php?topic=8850
It uses histogram analysis as well, so it should be easy to implement something similar.

hi
suddenly I have this problem ...
What could it be?

sorry as you upload a photo?

(http://i.imgur.com/1aETjfx.jpg)

This is the foto.All'improvviso started this difetto.Non know if I inadvertently changed something,

hi,
I'm a complete noob here  :'(
can i get help?
I've installed everything but its not run as expected; i think the one is about adding path to computer; i'm using windows 8.1; anybody?

thanks

@a1ex:

I didn't understood how this script works; I would like to understand how to apply a tonal curve so that I can decide wich raw value will correspond to the middle gray. I shoot dual iso 100/3200 and I use an exposure meter and I set aperture and shutter speed for a choosed EI from 100 up to 3200, then when I develop the image I must fight with the curve tool... it's a nightmare because it is only 255x255...
I would like to meter the raw values of middle gray for every EI from 100 to 3200, and then write a script that applies a tonal curve to retrieve the proper EI.

can you help me?)
i have cr2 dual iso photos, is it possible to add dual iso procession to python batch?

Correct me if i've missed something, but, any GUI or app available here for these scripts already?

Quote from: asthanaiitk on October 27, 2013, 10:51:05 PM

Hi A1ex

Any idea, why this could be happening:

Traceback (most recent call last):
  File "C:\Camera\Magic Lantern\cr2hdr\uniform_exposure1.py", line 275, in <module>
    mm, mh, ms = get_medians(r)
ValueError: need more than 0 values to unpack

Non-coder here, I'm having the same issue as this guy.

MacOS High Sierra, Xcode installed, then brew.sh and packages as instructed on dbast's post. (Enblend-enfuse on brew too).

Is this something specific with one of the dependencies?

Quote from: 350D on February 26, 2016, 05:32:40 PM
Correct me if i've missed something, but, any GUI or app available here for these scripts already?

This would be fantastic for us non programmers (or not technically capable). I attempt to follow the instructions but get lost almost immediately.

Всем Здравствуйте!Не могу скачать скрипт автоматической цветокоррекции фотографий (uniform_exposure.py) Подскажите каким способом еще его можно скачать на мой комп. или скиньте файл на почту/ [email protected]/ .Буду очень благодарен!Спасибо

https://web.archive.org/web/20161108063213/https://bitbucket.org/a1ex/uniform-exposure/raw/tip/uniform_exposure.py