Experiment - reducing aliasing in post using optical flow

Started by a1ex, November 19, 2017, 07:49:22 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

IDA_ML

If this effort succeeds, does that mean that expensive VAF filters will no longer be necessary for the cropped Canon models?  If that is the case, then this is really a big thing and I keep my thumbs pressed for all developers involved.  Good work, guys!

Danne

I think it´s dependent to motion but handheld should work.

Levas

Quote from: IDA_ML on November 21, 2017, 02:42:51 PM
If this effort succeeds, does that mean that expensive VAF filters will no longer be necessary for the cropped Canon models?  If that is the case, then this is really a big thing and I keep my thumbs pressed for all developers involved.  Good work, guys!

For as far as I can tell, VAF filters where never necessary  :P

http://www.magiclantern.fm/forum/index.php?topic=19687.msg185429#msg185429

That's the beauty of raw, many options in post proces  ;D

Levas

Sorry for asking here and not googling  :P

But is there an easy command line option, with convert, for converting all tif's in one direcory to ppm files ?


dfort

This reminds me of the image stacking experiment I made in the Large image, low fps vs. Small image, high fps topic a while back. With optical flow you shouldn't get the motion artifacts.

bpv5P

Quote from: Levas on November 21, 2017, 10:50:59 PM
Sorry for asking here and not googling  :P

But is there an easy command line option, with convert, for converting all tif's in one direcory to ppm files ?

See command "convert" inside the imagemagick package. If you're on debian-based gnu/linux system, use "sudo apt-get install imagemagick" and then "convert foo.tif bar.ppm".

a1ex


Danne

@Levas
Note that dcraw can convert straight to ppm as well:
find . -maxdepth 1 -iname '*.dng' -print0 | xargs -0 -P 4 -n 1 dcraw +M -6 -W -w -q 3
This command will also work where a for loop or ls command will break with "argument list too long" issues hitting rooftop around 6000 files...


Levas

Thanks,

Never would have guessed this one:

for f in *.tif; do  echo "Converting $f"; convert "$f"  "$(basename "$f" .tif).ppm"; done

Levas

Now I'm trying to average out a whole batch of ppm files with flow.py

I thought, the command

make j8

would do batch process, but it does only the clean operation in the makefile  :P
So the number behind the j is just the number of CPUjobs it starts ?

I have a sequence of files with the following name ' Test_000000.ppm' to 'Test_000121.ppm'
So the prefix should be according to the rules _6digits
What command will make all average files for the whole sequence ?

a1ex

It's "make -j8", and unless you have edited the Makefile, it should render all frames.

Here's a nicer version of the "all" rule (it process frames sequentially and should be easier to understand, but all sub-frames required by one image are still done in parallel):


all:
for f in *.dng; do \
$(MAKE) $${f%.dng}-a.jpg; \
rm -f *.npy; \
done


If your inputs are .jpg (same as outputs), I suggest using case sensitivity to tell the difference between inputs and outputs (for example, make the inputs uppercase). For example:


# use *.JPG for input files and *.jpg for output files
%.ppm: %.JPG:
convert $< $@

all:
for f in *.JPG do \
$(MAKE) $${f%.JPG}-a.jpg; \
rm -f *.npy; \
done


That removes the need to run a bash for loop before "make".

When rendering all frames, you may also want to delete the intermediate files. To do so, just comment out .SECONDARY. The *.npy files are an exception, since they are not created with Makefile rules (so "make" won't know it has to clean them up), hence the "rm -f *.npy" in the above loop.

Levas

Thanks, works  :D

(Had deleted all unused functions in the makefile, the 'all:' function was already deleted   :-[, my fault  :P)

Danne

Built a batch transcoding script aorund these dependencies:

brew install dcraw
brew cask install hugin
brew install enblend-enfuse


When run on a dng sequence it will end up with tif files which will be aligned and enfused hopefully reducing moiré. All dng files will be placed in a folder named tmp:

mkdir -p tmp
while grep 'dng' <<< $(find . -maxdepth 1 -iname '*.dng')
do
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 1') & pid1=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 2') & pid2=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 3') & pid3=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 4') & pid4=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 5') & pid5=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 6') & pid6=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 7') & pid7=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 8') & pid8=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 9') & pid9=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 10') & pid10=$!
#wait for jobs to end
    wait < <(jobs -p)

mv $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 1') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 2') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 3') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 4') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 5') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 6') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 7') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 8') tmp

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned.tiff $(ls *.tiff | awk 'FNR == 1') $(ls *.tiff | awk 'FNR == 3') && enfuse -o $(ls *.tiff | awk 'FNR == 1' | cut -d "." -f1).tif aligned.tiff0000.tif aligned.tiff0001.tif && rm aligned.tiff0000.tif aligned.tiff0001.tif & pid1=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned02.tiff $(ls *.tiff | awk 'FNR == 2') $(ls *.tiff | awk 'FNR == 4') && enfuse -o $(ls *.tiff | awk 'FNR == 2' | cut -d "." -f1).tif aligned02.tiff0000.tif aligned02.tiff0001.tif && rm aligned02.tiff0000.tif aligned02.tiff0001.tif & pid2=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned03.tiff $(ls *.tiff | awk 'FNR == 3') $(ls *.tiff | awk 'FNR == 5') && enfuse -o $(ls *.tiff | awk 'FNR == 3' | cut -d "." -f1).tif aligned03.tiff0000.tif aligned03.tiff0001.tif && rm aligned03.tiff0000.tif aligned03.tiff0001.tif & pid3=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned04.tiff $(ls *.tiff | awk 'FNR == 4') $(ls *.tiff | awk 'FNR == 6') && enfuse -o $(ls *.tiff | awk 'FNR == 4' | cut -d "." -f1).tif aligned04.tiff0000.tif aligned04.tiff0001.tif && rm aligned04.tiff0000.tif aligned04.tiff0001.tif & pid4=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned05.tiff $(ls *.tiff | awk 'FNR == 5') $(ls *.tiff | awk 'FNR == 7') && enfuse -o $(ls *.tiff | awk 'FNR == 5' | cut -d "." -f1).tif aligned05.tiff0000.tif aligned05.tiff0001.tif && rm aligned05.tiff0000.tif aligned05.tiff0001.tif & pid5=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned06.tiff $(ls *.tiff | awk 'FNR == 6') $(ls *.tiff | awk 'FNR == 8') && enfuse -o $(ls *.tiff | awk 'FNR == 6' | cut -d "." -f1).tif aligned06.tiff0000.tif aligned06.tiff0001.tif && rm aligned06.tiff0000.tif aligned06.tiff0001.tif & pid6=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned07.tiff $(ls *.tiff | awk 'FNR == 7') $(ls *.tiff | awk 'FNR == 9') && enfuse -o $(ls *.tiff | awk 'FNR == 7' | cut -d "." -f1).tif aligned07.tiff0000.tif aligned07.tiff0001.tif && rm aligned07.tiff0000.tif aligned07.tiff0001.tif & pid7=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned08.tiff $(ls *.tiff | awk 'FNR == 8') $(ls *.tiff | awk 'FNR == 10') && enfuse -o $(ls *.tiff | awk 'FNR == 8' | cut -d "." -f1).tif aligned08.tiff0000.tif aligned08.tiff0001.tif && rm aligned08.tiff0000.tif aligned08.tiff0001.tif & pid8=$!
#wait for jobs to end
    wait < <(jobs -p)

rm $(ls *.tiff | awk 'FNR == 1') $(ls *.tiff | awk 'FNR == 2') $(ls *.tiff | awk 'FNR == 3') $(ls *.tiff | awk 'FNR == 4') $(ls *.tiff | awk 'FNR == 5') $(ls *.tiff | awk 'FNR == 6') $(ls *.tiff | awk 'FNR == 7') $(ls *.tiff | awk 'FNR == 8')
done

Danne

Tested against enfuse/hugin to produce tifs with reduced aliasing. Would be nice to have it checked against optical flow/ffmpeg too. Running out of time here..

A test file here(1200D mv720):
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M23-1009_aliasing.MLV
develop settings in Switch:
ms (mlv_dump_on_steroids)
02 (cs2x2)

dng(opened in acr, sharpness set to 0)
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M23-1009_1_2017-11-23_0001_C0000_000005.dng
(crop)



tif
(crop)
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M23-1009_1_2017-11-23_0001_C0000_000005.tif



Danne

On the theme super resolution I played with merging and aligning more and more images and the result gets better(crops):

My test file:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M23-1643_aliasing.MLV

Images 1_3


Images 1_2_3_4_5


Images 1_2_3_4_5_6_7_8_9_10


Images 1_2_3_4_5_6_7_8_9_10_11_12_13_14_15




Full image width:
Images 1_3


Images 1_2_3_4_5_6_7_8_9_10_11_12_13_14_15



DeafEyeJedi

Quote from: Danne on November 23, 2017, 04:50:14 PM
On the theme super resolution I played with merging and aligning more and more images and the result gets better(crops):

O0o0o0ohh hell yeah baby this is so bad ass!!!

Also is it just me but I noticed that even the shadows seems to boost up slightly in 'Images 1_2_3_4_5_6_7_8_9_10_11_12_13_14_15' or no?  8)
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

So this is super resolution using optical flow? Wonder how it would works with video. How long does it take to render each frame?

Danne

I'm using enfuse/hugin alignment. Rewrote script part to be able to work 15 files.
http://www.magiclantern.fm/forum/index.php?topic=20999.msg193531#msg193531

Well, processing between 5-10 images producing one image will take a while if running movie sequences.

Teamsleepkid

if you can get something thats fast and better than normal with video everyone will be happy. build it into the new switch danne:)
EOS M


Levas

Using the optical flow script, first post this topic, to enhance some old 320x240 motion jpg avi files.
upscaled the resolution 3 times each dimension, so 960x720 and then run the optical flow script, altered it to let it average out 17 frames (8 previous, 8 next).
The result is actually not bad. It still lacks some detail of course, but shadow noise and jpg blocking artifacts are all gone.

@Alex
In the makefile I see the average frame is made by the command convert.
But when I type convert in terminal, I don't see an option for average ?
I'm curious if there are more options instead of average, maybe median or something ?




a1ex

Sounds cool - mind sharing a few frames?

For median and other methods for combining images: https://www.imagemagick.org/discourse-server/viewtopic.php?t=17519 and https://www.imagemagick.org/script/command-line-options.php#evaluate-sequence

Would be nice to pick or weight the images to average, based on sharpness, similarity to original, whether there's some slight camera movement... probably best done in a Python or Octave script.

Levas

Had to search for an avi without relatives in it, found one  :D
I think this is a good example of what aligning and averaging can do.
Before (upscaled tif from 320x240 to 960x720 with Lightroom

Same frame, but averaged with 8 previous and 8 next frames:


Not very detailed, but much better then the original.
Think median will give more detailed results, will check out the links you send.

Deadcode

Great job Levas!

Could you please do some tests with my 5D2 footage? You can download it from the following link:
https://wetransfer.com/downloads/7747cb1b86614e01dc1acb9bf11bd2a220171124170652/f7fd5be4b588f945da4898c6c41feb9e20171124170652/f83ff8

I think on this example "motion picture" means a totally different thing... This was my worst experience with moire/aliasing. Yes you can clean up a lot with chroma blur. i filmed this about a year ago, now im filming around F16 if im shooting with infinity focus, so now moire/aliasing is not that much of a problem. Im curious what can you do with this if you are using your method.

Levas

EDIT:
Got a averaged frame too, frame 9 averaged with 8 previous and 8 next.
Can't hardly see the difference with the TIF I got with RawTherapee.
For the pixelpeepers, 3 files on google drive, original DNG, the Rawtherapee TIF and the averaged DNG files:
https://drive.google.com/drive/folders/1yOffbd7OzRnr6LEKoJqXIddZ5bFLu03c?usp=sharing




That one was rather easy  8)

The optical flow script wasn't even needed for this, only RawTherapee was used:
See this post for a youtube link to see the workflow:
http://www.magiclantern.fm/forum/index.php?topic=19687.msg185429#msg185429

Just to see what happens, I'm going to use the optical flow script on frame 9 too.
But that's gonna take a while, will upload it when its done.