Experiment - reducing aliasing in post using optical flow

[a1ex]

Just playing with this dataset and https://github.com/pathak22/pyflow

Do you mean you want to try some super resolution algorithms ?
I have about 45 frames of this castle, before I start panning to the right...
Uploading frame 0 to 45 right now, takes about half an hour.
Same link as before.
http://drive.google.com/drive/folders/0B1BxGc3dfMDaRmtKc2tOa3dHMTA?usp=sharing

- before: dcraw M27-1337-frame_000002.dng
- after: averaged with frames 1 and 3, warped with optical flow to match frame 2





To reproduce the above result, get the files below, install the dependencies (follow comments and error messages), then type:

Code Select Expand


make -j2 M27-1337_frame_000002-a.jpg


or "make -j8" to render the entire sequence on a quad-core processor.



Makefile (use Firefox for copying the text; Google Chrome and Safari will not work!)

Code Select Expand


# experiment: attempt to reduce aliasing on hand-held footage using optical flow
# requires https://github.com/pathak22/pyflow

# replace with path to pyflow repository
FLOW=python ~/src/pyflow/flow.py

# default target: render all frames as jpg
all: $(patsubst %.dng,%-a.jpg,$(wildcard M27-1337_frame_*.dng))

# render DNGs with dcraw
%.ppm: %.dng
dcraw $<

# helper to specify dependencies on previous or next image
# assumes the file name pattern is: prefix_000123 (underscore followed by 6 digits)
# fixme: easier way to... increment a number in Makefile?!
inc = $(shell stem="$1"; echo $${stem%_*}_$$(printf "%06d" $$((10\#$${stem//*_/}+$2))) )

# enable secondary expansion (needed below)
.SECONDEXPANSION:

# next or previous frames
%-n.png: %.ppm $$(call inc,%,1).ppm
$(FLOW) $^ $@

%-p.png: %.ppm $$(call inc,%,-1).ppm
$(FLOW) $^ $@

# average
%-a.png: %.ppm %-n.png %-p.png
convert -average $^ $@

# fallback rules: first / last file will only have "next" / "previous" neighbours
# FIXME: these rules may be chosen incorrectly instead of the above in some edge cases; if in doubt, delete them and see if it helps
%-a.png: %.ppm %-n.png
convert -average $^ $@

%-a.png: %.ppm %-p.png
convert -average $^ $@

# convert to jpg
%.jpg: %.ppm
convert $< $@
%.jpg: %.png
convert $< $@

# 100% crops
%-crop.jpg: %.jpg
convert $< -crop 400x300+900+650 $@

# careful if you have other files in this directory ;)
clean:
rm -f *.ppm *.jpg *.png *.npy

# do not delete intermediate files
.SECONDARY:

# example:
# make -j8
# make -j2 M27-1337_frame_000002-a.jpg


flow.py:

Code Select Expand


# Modified the demo from https://github.com/pathak22/pyflow
# -> just save the warped image and the computed flow; filenames from command line

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
# from __future__ import unicode_literals
import numpy as np
from PIL import Image
import time
import pyflow
import sys

try:
    print("%s %s -> %s" % (sys.argv[1], sys.argv[2], sys.argv[3]))
except:
    print("usage: %s input1.jpg input2.jpg output.npy" % sys.argv[0])
    raise SystemExit

im1 = np.array(Image.open(sys.argv[1]))
im2 = np.array(Image.open(sys.argv[2]))
im1 = im1.astype(float) / 255.
im2 = im2.astype(float) / 255.

# Flow Options:
alpha = 0.012
ratio = 0.75
minWidth = 20
nOuterFPIterations = 7
nInnerFPIterations = 1
nSORIterations = 30
colType = 0  # 0 or default:RGB, 1:GRAY (but pass gray image with shape (h,w,1))

s = time.time()
u, v, im2W = pyflow.coarse2fine_flow(
    im1, im2, alpha, ratio, minWidth, nOuterFPIterations, nInnerFPIterations,
    nSORIterations, colType)
e = time.time()
print('Time Taken: %.2f seconds for image of size (%d, %d, %d)' % (
    e - s, im1.shape[0], im1.shape[1], im1.shape[2]))

flow = np.concatenate((u[..., None], v[..., None]), axis=2)
np.save(sys.argv[3] + ".npy", flow)

import cv2
cv2.imwrite(sys.argv[3], im2W[:, :, ::-1] * 255)


Exercise for the reader: use more frames to compute the correction.

Have fun.

[Levas]

That footage looks familiar 

Interesting stuff to experiment with, seems to clear up a lot of the moire.

[Danne]

It seems very promising indeed. Amazing really. Not affecting sharpness. Never seen such results.

[Kharak]

How does this affect the parts of the image that are without Moire?

Edit: Just clicked on the pictures.

[DeafEyeJedi]

- before: dcraw M27-1337-frame_000002.dng
- after: averaged with frames 1 and 3, warped with optical flow to match frame 2

Definitely looks promising. Fun times are ahead!

[Levas]

Trying to duplicate what you did here.

Didn't succeed.

Did this in terminal

Code Select Expand

git clone https://github.com/pathak22/pyflow.git
cd pyflow/
python setup.py build_ext -i
python demo.py 


But with 'python setup.py build_ext -I'
I get the following error:

Traceback (most recent call last):
  File "setup.py", line 8, in <module>
    from Cython.Build import cythonize
ImportError: No module named Cython.Build

[Danne]

Did you try:

Code Select Expand

pip install Cython

[a1ex]

You may need to install pip first. There are two ways to do this on Mac (or maybe more):

- sudo easy_install pip (then, sudo pip install Cython)
- without sudo: pip2 after installing homebrew and hg. After running the commands suggested by homebrew: pip2 install Cython; python setup.py build_ext -i

You'll also need PIL (sudo pip install Pillow / pip2 install Pillow), OpenCV (brew install opencv3), dcraw and ImageMagick (brew install dcraw imagemagick). Also requires some fiddling (e.g. copying code doesn't work well from Safari, python modules don't work out of the box - read brew's messages - and so on).

Tested on a Mac VM after installing this. The hardest part is getting used to the Mac UI (e.g. how do you open the Home directory in the file browser? how do you select all the files from a zip in order to drag them?!)

Tip: only experiment with one frame at a time (it's not very fast).

[Levas]

Wow, lots of information.
Ok, installed all the packages, did the following commands in terminal:

sudo easy_install pip
sudo pip install Cython

cd pyflow/
python setup.py build_ext -I

sudo pip install Pillow
brew install opencv3
brew install dcraw imagemagick

Now when I use the command:
make -j2 M27-1337_frame_000002-a.jpg
I get an error message:
Makefile:12: *** missing separator.  Stop.

I'm not sure about my makefile, I assume I needed to make a file in Pyflow directory with the name 'makefile' (no extension) with the code you had in your first post.
To be sure the makefile is not in a wrong text format, rich tekst or something, I copied one from the Magiclantern directory and replaced the tekst with the tekst in your post.

I have Pyflow in my user home directory,
So this alteration should do ?

# replace with path to pyflow repository
FLOW=python ~/pyflow/Flow.py

[a1ex]

copying code doesn't work well from Safari

Maybe you can report a bug to Apple after you get it working

edit: Google Chrome has the same issue, Firefox works.

edit: wget also has the issue, what the...

[Levas]

Read that the first time 
Thought I got it right with using existing makefile from magic lantern and selecting the lines manually etc...
But now that I have copied the tekst of code with using the quote button on your post, to see plain tekst of your message, it works.

It does a lot, takes about 1 minute to calculate stuff for only averaging frame 2.
But at the end I get an error, no module CV2.

make -j2 M27-1337_frame_000002-a.jpg
python ~/pyflow/flow.py M27-1337_frame_000002.ppm M27-1337_frame_000003.ppm M27-1337_frame_000002-n.png
python ~/pyflow/flow.py M27-1337_frame_000002.ppm M27-1337_frame_000001.ppm M27-1337_frame_000002-p.png
M27-1337_frame_000002.ppm M27-1337_frame_000003.ppm -> M27-1337_frame_000002-n.png
M27-1337_frame_000002.ppm M27-1337_frame_000001.ppm -> M27-1337_frame_000002-p.png
Constructing pyramid...done!
Constructing pyramid...done!
Pyramid level 14
Pyramid level 14
Pyramid level 13
Pyramid level 13
Pyramid level 12
Pyramid level 12
Pyramid level 11
Pyramid level 11
Pyramid level 10
Pyramid level 10
Pyramid level 9
Pyramid level 9
Pyramid level 8
Pyramid level 8
Pyramid level 7
Pyramid level 7
Pyramid level 6
Pyramid level 6
Pyramid level 5
Pyramid level 5
Pyramid level 4
Pyramid level 4
Pyramid level 3
Pyramid level 3
Pyramid level 2
Pyramid level 2
Pyramid level 1
Pyramid level 1
Pyramid level 0
Pyramid level 0
Time Taken: 55.16 seconds for image of size (1026, 1824, 3)
Time Taken: 55.17 seconds for image of size (1026, 1824, 3)
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/magic_lantern/pyflow/flow.py", line 45, in <module>
  File "/Users/magic_lantern/pyflow/flow.py", line 45, in <module>
    import cv2
ImportError: No module named cv2
    import cv2
ImportError: No module named cv2
make: *** [M27-1337_frame_000002-n.png] Error 1
make: *** Waiting for unfinished jobs....
make: *** [M27-1337_frame_000002-p.png] Error 1

Sorry for all, probably noob, questions 

[Levas]

Interested in making this work, curious what it can do with high iso noise 

For as far I can tell, you're using pyflow to align images and once aligned, your averaging the frame with previous and next frame.
Is that what's happening ?

[a1ex]

When you install opencv with brew, it will tell you to run some commands if you want to develop with it (or something along these lines). Once you do that, it should be able to find cv2 (aka OpenCV). At least that's how it worked here.

I've used a modified version of this for vertical pattern noise removal (used more neighbouring frames, reduced the weight of frames without camera motion and only kept the vertical offsets from the correction). Regular noise reduction also works (advice: limit the correction to +/- some small quantity, e.g. 5 units on a 0-255 scale).

For more fun stuff, look at https://people.csail.mit.edu/celiu/ (watch the video on the front page).



Just got another sample file for testing; result:

dcraw -a -b 2:
Before (original, uncorrected): M18-1635_000003-dcraw.jpg
Corrected with 1 frame before/after: M18-1635_000003-dcraw-a11.jpg
Corrected with 3 frames before/after: M18-1635_000003-dcraw-a33.jpg

Same example with RawTherapee, LMMSE:
Before (original, uncorrected): M18-1635_000003-RT-LMMSE.jpg
Corrected with 1 frame before/after: M18-1635_000003-RT-LMMSE-a11.jpg
Corrected with 3 frames before/after: M18-1635_000003-RT-LMMSE-a33.jpg

[Levas]

When you install opencv with brew, it will tell you to run some commands if you want to develop with it (or something along these lines). Once you do that, it should be able to find cv2 (aka OpenCV). At least that's how it worked here.

Found the command you're talking about, it works now.
Thanks.

Time to do some testing with it 

[Danne]

How did you solve opencv issue Levas? I´m getting:

Code Select Expand

Time Taken: 55.89 seconds for image of size (1026, 1824, 3)
RuntimeError: module compiled against API version 0xb but this version of numpy is 0x9
Traceback (most recent call last):
  File "/Users/dan/pyflow/flow.py", line 45, in <module>
    import cv2
ImportError: numpy.core.multiarray failed to import
make: *** [M27-1337_frame_000002-p.png] Error 1
dans-MBP:pyflow dan$

Probably need to install the latest numpy version as well...

*Update. Working now. Did:

Code Select Expand

sudo -H pip install numpy --upgrade

[a1ex]

Also noticed LMMSE already handles color artifacts pretty well, so I've rendered the frames with RawTherapee and applied the same script. This is not fully automated - after batch processing with RT or any other external program, convert the images to ppm, delete the dcraw rule from the Makefile and run the remaining stuff. Result:

[Levas]

Knew about the LMMSE method controlling color moire really good, THE reason I've learned to use RawTherapee 

But I'm wondering, how to convert the images to ppm    (it's mentioned in your workflow as if every photo/videographer uses the ppm format  )
And is there a way to export the final; image as tif file instead of jpg ?

[a1ex]

convert foobar.tif foobar.ppm

I had some trouble saving 16-bit tif from OpenCV, but 16-bit PNG appears to work fine (convert it to tif afterwards). For 8-bit output, just change the extensions in the Makefile; for 16-bit output, it's something like: cv2.imwrite(out, im2.astype('uint16')); might also require scaling.

The default output format is 8-bit PNG, btw (and the default input is 8-bit PPM because that's what dcraw outputs by default). I've used JPEG just for uploading smaller files.

[Levas]

Anybody already find out how to do more then one previous and one next frame ?
Alex is using 3 previous/next frames here:
https://www.magiclantern.fm/forum/index.php?topic=20995.msg193400#msg193400

[a1ex]

Very easy:

Code Select Expand


# similar to %-n.png and %-p.png, but using frames i+2, i-2, i+3 and i-3
%-n2.png: %.ppm $$(call inc,%,2).ppm
$(FLOW) $^ $@

%-p2.png: %.ppm $$(call inc,%,-2).ppm
$(FLOW) $^ $@

%-n3.png: %.ppm $$(call inc,%,3).ppm
$(FLOW) $^ $@

%-p3.png: %.ppm $$(call inc,%,-3).ppm
$(FLOW) $^ $@

# average current frame with 3 warped frames before and 3 after
%-a33.png: %.ppm %-n.png %-p.png %-n2.png %-p2.png %-n3.png %-p3.png
convert -average $^ $@


=> make whatever_001234-a33.png (or jpg, since there is a rule that converts from png to jpg)

[bpv5P]

A integration of it on HDRMerge would be cool...

[Danne]

ffmpeg has a very fast averaging filter. Here´s an example using the tblend filter averaging consecutive frames. Obviously it will produce a slight blur but maybe one can use something like minterpolate filter to build some in between frame.

You need ffmpeg and dcraw and then you can run the following command directly on two dng files to output highres tif files:

Averaged output

Code Select Expand

find . -maxdepth 1 -iname '*.dng' -print0 | xargs -0 dcraw +M -c -6 -W -q 3 | ffmpeg -f image2pipe -vcodec ppm -i pipe:0 -y -pix_fmt rgb24 -t 2 -vf tblend=all_mode=average AVERAGED%03d.tif



Unaveraged output

Code Select Expand

find . -maxdepth 1 -iname '*.dng' -print0 | xargs -0 dcraw +M -c -6 -W -q 3 | ffmpeg -f image2pipe -vcodec ppm -i pipe:0 -y -pix_fmt rgb24 NOTAVERAGED%03d.tif



Averaged


Unaveraged



ffmpeg tblend filter:
https://ffmpeg.org/ffmpeg-filters.html#blend_002c-tblend

Minterpolate:
https://ffmpeg.org/ffmpeg-filters.html#minterpolate

[Danne]

Seems there are more ways to play with this. By using enblend-enfuse and hugin(align image tool) one can elmininate aliasing and moiré. Inspired by a1ex post I did following.

Install:
brew install dcraw
brew cask install hugin
brew install enblend-enfuse

In a folder place two files let´s say for instance(skip one file in between):
M27-1337_frame_000001.dng
M27-1337_frame_000003.dng

Then run following command inside the folder containing the two dng files:

Code Select Expand

dcraw +M -6 -W -q 3 -T *.dng && /Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned.tiff $(ls *.tiff | awk 'FNR == 1 {print}') $(ls *.tiff | awk 'FNR == 2 {print}') && enfuse -o enfused_aligned.tiff aligned.tiff0000.tif aligned.tiff0001.tif && rm aligned.tiff0000.tif aligned.tiff0001.tif

The final file will be called enfused_aligned.tiff and should be sharp and clean from aliasing and moiré.
The first file M27-1337_frame_000001.dng can be replaced with enfused_aligned.tiff so then it´s just a matter of building a script that will iterate a whole movie sequence working through the dng sequence/replacing in consecutive order.

enfused_aligned


original

[a1ex]

maybe one can use something like minterpolate filter to build some in between frame.

Code Select Expand


ffmpeg -framerate 30 -i M27-1337_frame_%06d.ppm -filter_complex "minterpolate=fps=60:mi_mode=mci:me_mode=bidir" -y interp_%06d.ppm


M27-1337_frame_000000.ppm -> interp_000002.ppm
M27-1337_frame_000001.ppm -> interp_000004.ppm
M27-1337_frame_000002.ppm -> interp_000006.ppm
...

It's very fast; try changing some parameters to get better quality.

To keep only the interpolated frames:

Code Select Expand


ffmpeg -framerate 30 -i M27-1337_frame_%06d.ppm -filter_complex "minterpolate=fps=60:mi_mode=mci:me_mode=bidir,select=not(mod(n-1\,2))" -r 30 -y interp_%06d.ppm


[Danne]

Nice syntax. However sharpness is affected(possibly masking aliasing better). Compare with the alignment roundtrip(huging/enfuse):

ffmpeg


hugin/enfuse