Author Topic: Experiment - reducing aliasing in post using optical flow  (Read 3078 times)

IDA_ML

  • Member
  • ***
  • Posts: 186
Re: Experiment - reducing aliasing in post using optical flow
« Reply #25 on: November 21, 2017, 02:42:51 PM »
If this effort succeeds, does that mean that expensive VAF filters will no longer be necessary for the cropped Canon models?  If that is the case, then this is really a big thing and I keep my thumbs pressed for all developers involved.  Good work, guys!

Danne

  • Hero Member
  • *****
  • Posts: 3611
Re: Experiment - reducing aliasing in post using optical flow
« Reply #26 on: November 21, 2017, 02:47:55 PM »
I think it´s dependent to motion but handheld should work.

Levas

  • Hero Member
  • *****
  • Posts: 969
  • 6d - Nightly build user
Re: Experiment - reducing aliasing in post using optical flow
« Reply #27 on: November 21, 2017, 10:44:33 PM »
If this effort succeeds, does that mean that expensive VAF filters will no longer be necessary for the cropped Canon models?  If that is the case, then this is really a big thing and I keep my thumbs pressed for all developers involved.  Good work, guys!

For as far as I can tell, VAF filters where never necessary  :P

http://www.magiclantern.fm/forum/index.php?topic=19687.msg185429#msg185429

That's the beauty of raw, many options in post proces  ;D

Levas

  • Hero Member
  • *****
  • Posts: 969
  • 6d - Nightly build user
Re: Experiment - reducing aliasing in post using optical flow
« Reply #28 on: November 21, 2017, 10:50:59 PM »
Sorry for asking here and not googling  :P

But is there an easy command line option, with convert, for converting all tif's in one direcory to ppm files ?


dfort

  • Hero Member
  • *****
  • Posts: 2300
Re: Experiment - reducing aliasing in post using optical flow
« Reply #29 on: November 21, 2017, 11:44:35 PM »
This reminds me of the image stacking experiment I made in the Large image, low fps vs. Small image, high fps topic a while back. With optical flow you shouldn't get the motion artifacts.
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

bpv5P

  • Member
  • ***
  • Posts: 169
Re: Experiment - reducing aliasing in post using optical flow
« Reply #30 on: November 22, 2017, 06:50:11 AM »
Sorry for asking here and not googling  :P

But is there an easy command line option, with convert, for converting all tif's in one direcory to ppm files ?

See command "convert" inside the imagemagick package. If you're on debian-based gnu/linux system, use "sudo apt-get install imagemagick" and then "convert foo.tif bar.ppm".

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10704
  • 5D Mark Free
Re: Experiment - reducing aliasing in post using optical flow
« Reply #31 on: November 22, 2017, 09:26:44 AM »
@bpv5P: if you want to help, try reading the question (quoting it is not enough) and maybe also some of the previous posts.

On-topic:

https://www.cyberciti.biz/faq/bash-loop-over-file/
https://stackoverflow.com/questions/20483331/how-to-copy-and-rename-file-with-for-loop-in-bash-script
https://askubuntu.com/questions/60401/batch-processing-tif-images-converting-tif-to-jpeg

Spoiler:
for f in *.tif; do convert $f ${f%.tif}.ppm; done

Danne

  • Hero Member
  • *****
  • Posts: 3611
Re: Experiment - reducing aliasing in post using optical flow
« Reply #32 on: November 22, 2017, 10:06:18 AM »
@Levas
Note that dcraw can convert straight to ppm as well:
Code: [Select]
find . -maxdepth 1 -iname '*.dng' -print0 | xargs -0 -P 4 -n 1 dcraw +M -6 -W -w -q 3This command will also work where a for loop or ls command will break with "argument list too long" issues hitting rooftop around 6000 files...


Levas

  • Hero Member
  • *****
  • Posts: 969
  • 6d - Nightly build user
Re: Experiment - reducing aliasing in post using optical flow
« Reply #33 on: November 22, 2017, 10:20:50 AM »
Thanks,

Never would have guessed this one:
Code: [Select]
for f in *.tif; do  echo "Converting $f"; convert "$f"  "$(basename "$f" .tif).ppm"; done

Levas

  • Hero Member
  • *****
  • Posts: 969
  • 6d - Nightly build user
Re: Experiment - reducing aliasing in post using optical flow
« Reply #34 on: November 22, 2017, 10:42:22 AM »
Now I'm trying to average out a whole batch of ppm files with flow.py

I thought, the command
Code: [Select]
make j8
would do batch process, but it does only the clean operation in the makefile  :P
So the number behind the j is just the number of CPUjobs it starts ?

I have a sequence of files with the following name ' Test_000000.ppm' to 'Test_000121.ppm'
So the prefix should be according to the rules _6digits
What command will make all average files for the whole sequence ?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10704
  • 5D Mark Free
Re: Experiment - reducing aliasing in post using optical flow
« Reply #35 on: November 22, 2017, 11:49:38 AM »
It's "make -j8", and unless you have edited the Makefile, it should render all frames.

Here's a nicer version of the "all" rule (it process frames sequentially and should be easier to understand, but all sub-frames required by one image are still done in parallel):

Code: [Select]
all:
for f in *.dng; do \
$(MAKE) $${f%.dng}-a.jpg; \
rm -f *.npy; \
done

If your inputs are .jpg (same as outputs), I suggest using case sensitivity to tell the difference between inputs and outputs (for example, make the inputs uppercase). For example:

Code: [Select]
# use *.JPG for input files and *.jpg for output files
%.ppm: %.JPG:
convert $< [email protected]

all:
for f in *.JPG do \
$(MAKE) $${f%.JPG}-a.jpg; \
rm -f *.npy; \
done

That removes the need to run a bash for loop before "make".

When rendering all frames, you may also want to delete the intermediate files. To do so, just comment out .SECONDARY. The *.npy files are an exception, since they are not created with Makefile rules (so "make" won't know it has to clean them up), hence the "rm -f *.npy" in the above loop.

Levas

  • Hero Member
  • *****
  • Posts: 969
  • 6d - Nightly build user
Re: Experiment - reducing aliasing in post using optical flow
« Reply #36 on: November 22, 2017, 10:54:20 PM »
Thanks, works  :D

(Had deleted all unused functions in the makefile, the 'all:' function was already deleted   :-[, my fault  :P)

Danne

  • Hero Member
  • *****
  • Posts: 3611
Re: Experiment - reducing aliasing in post using optical flow
« Reply #37 on: November 22, 2017, 11:56:53 PM »
Built a batch transcoding script aorund these dependencies:

brew install dcraw
brew cask install hugin
brew install enblend-enfuse


When run on a dng sequence it will end up with tif files which will be aligned and enfused hopefully reducing moiré. All dng files will be placed in a folder named tmp:

Code: [Select]
mkdir -p tmp
while grep 'dng' <<< $(find . -maxdepth 1 -iname '*.dng')
do
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 1') & pid1=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 2') & pid2=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 3') & pid3=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 4') & pid4=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 5') & pid5=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 6') & pid6=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 7') & pid7=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 8') & pid8=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 9') & pid9=$!
dcraw +M -6 -W -w -q 3 -T $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 10') & pid10=$!
#wait for jobs to end
    wait < <(jobs -p)

mv $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 1') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 2') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 3') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 4') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 5') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 6') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 7') $(find . -maxdepth 1 -iname '*.dng' | awk 'FNR == 8') tmp

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned.tiff $(ls *.tiff | awk 'FNR == 1') $(ls *.tiff | awk 'FNR == 3') && enfuse -o $(ls *.tiff | awk 'FNR == 1' | cut -d "." -f1).tif aligned.tiff0000.tif aligned.tiff0001.tif && rm aligned.tiff0000.tif aligned.tiff0001.tif & pid1=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned02.tiff $(ls *.tiff | awk 'FNR == 2') $(ls *.tiff | awk 'FNR == 4') && enfuse -o $(ls *.tiff | awk 'FNR == 2' | cut -d "." -f1).tif aligned02.tiff0000.tif aligned02.tiff0001.tif && rm aligned02.tiff0000.tif aligned02.tiff0001.tif & pid2=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned03.tiff $(ls *.tiff | awk 'FNR == 3') $(ls *.tiff | awk 'FNR == 5') && enfuse -o $(ls *.tiff | awk 'FNR == 3' | cut -d "." -f1).tif aligned03.tiff0000.tif aligned03.tiff0001.tif && rm aligned03.tiff0000.tif aligned03.tiff0001.tif & pid3=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned04.tiff $(ls *.tiff | awk 'FNR == 4') $(ls *.tiff | awk 'FNR == 6') && enfuse -o $(ls *.tiff | awk 'FNR == 4' | cut -d "." -f1).tif aligned04.tiff0000.tif aligned04.tiff0001.tif && rm aligned04.tiff0000.tif aligned04.tiff0001.tif & pid4=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned05.tiff $(ls *.tiff | awk 'FNR == 5') $(ls *.tiff | awk 'FNR == 7') && enfuse -o $(ls *.tiff | awk 'FNR == 5' | cut -d "." -f1).tif aligned05.tiff0000.tif aligned05.tiff0001.tif && rm aligned05.tiff0000.tif aligned05.tiff0001.tif & pid5=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned06.tiff $(ls *.tiff | awk 'FNR == 6') $(ls *.tiff | awk 'FNR == 8') && enfuse -o $(ls *.tiff | awk 'FNR == 6' | cut -d "." -f1).tif aligned06.tiff0000.tif aligned06.tiff0001.tif && rm aligned06.tiff0000.tif aligned06.tiff0001.tif & pid6=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned07.tiff $(ls *.tiff | awk 'FNR == 7') $(ls *.tiff | awk 'FNR == 9') && enfuse -o $(ls *.tiff | awk 'FNR == 7' | cut -d "." -f1).tif aligned07.tiff0000.tif aligned07.tiff0001.tif && rm aligned07.tiff0000.tif aligned07.tiff0001.tif & pid7=$!

/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned08.tiff $(ls *.tiff | awk 'FNR == 8') $(ls *.tiff | awk 'FNR == 10') && enfuse -o $(ls *.tiff | awk 'FNR == 8' | cut -d "." -f1).tif aligned08.tiff0000.tif aligned08.tiff0001.tif && rm aligned08.tiff0000.tif aligned08.tiff0001.tif & pid8=$!
#wait for jobs to end
    wait < <(jobs -p)

rm $(ls *.tiff | awk 'FNR == 1') $(ls *.tiff | awk 'FNR == 2') $(ls *.tiff | awk 'FNR == 3') $(ls *.tiff | awk 'FNR == 4') $(ls *.tiff | awk 'FNR == 5') $(ls *.tiff | awk 'FNR == 6') $(ls *.tiff | awk 'FNR == 7') $(ls *.tiff | awk 'FNR == 8')
done

Danne

  • Hero Member
  • *****
  • Posts: 3611
Re: Experiment - reducing aliasing in post using optical flow
« Reply #38 on: November 23, 2017, 09:56:40 AM »
Tested against enfuse/hugin to produce tifs with reduced aliasing. Would be nice to have it checked against optical flow/ffmpeg too. Running out of time here..

A test file here(1200D mv720):
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M23-1009_aliasing.MLV
develop settings in Switch:
ms (mlv_dump_on_steroids)
02 (cs2x2)

dng(opened in acr, sharpness set to 0)
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M23-1009_1_2017-11-23_0001_C0000_000005.dng
(crop)



tif
(crop)
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M23-1009_1_2017-11-23_0001_C0000_000005.tif



Danne

  • Hero Member
  • *****
  • Posts: 3611
Re: Experiment - reducing aliasing in post using optical flow
« Reply #39 on: November 23, 2017, 04:50:14 PM »
On the theme super resolution I played with merging and aligning more and more images and the result gets better(crops):

My test file:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M23-1643_aliasing.MLV

Images 1_3


Images 1_2_3_4_5


Images 1_2_3_4_5_6_7_8_9_10


Images 1_2_3_4_5_6_7_8_9_10_11_12_13_14_15




Full image width:
Images 1_3


Images 1_2_3_4_5_6_7_8_9_10_11_12_13_14_15



DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3065
  • 5D3 / M1 / 7D / 70D / SL1
Re: Experiment - reducing aliasing in post using optical flow
« Reply #40 on: November 23, 2017, 05:01:24 PM »
On the theme super resolution I played with merging and aligning more and more images and the result gets better(crops):

O0o0o0ohh hell yeah baby this is so bad ass!!!

Also is it just me but I noticed that even the shadows seems to boost up slightly in 'Images 1_2_3_4_5_6_7_8_9_10_11_12_13_14_15' or no?  8)
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

dfort

  • Hero Member
  • *****
  • Posts: 2300
Re: Experiment - reducing aliasing in post using optical flow
« Reply #41 on: November 23, 2017, 05:48:15 PM »
So this is super resolution using optical flow? Wonder how it would works with video. How long does it take to render each frame?
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

Danne

  • Hero Member
  • *****
  • Posts: 3611
Re: Experiment - reducing aliasing in post using optical flow
« Reply #42 on: November 23, 2017, 05:58:22 PM »
I'm using enfuse/hugin alignment. Rewrote script part to be able to work 15 files.
http://www.magiclantern.fm/forum/index.php?topic=20999.msg193531#msg193531

Well, processing between 5-10 images producing one image will take a while if running movie sequences.

Teamsleepkid

  • Member
  • ***
  • Posts: 188
Re: Experiment - reducing aliasing in post using optical flow
« Reply #43 on: November 23, 2017, 09:28:24 PM »
if you can get something thats fast and better than normal with video everyone will be happy. build it into the new switch danne:)
EOS M

Danne

  • Hero Member
  • *****
  • Posts: 3611

Levas

  • Hero Member
  • *****
  • Posts: 969
  • 6d - Nightly build user
Re: Experiment - reducing aliasing in post using optical flow
« Reply #45 on: November 24, 2017, 02:50:32 PM »
Using the optical flow script, first post this topic, to enhance some old 320x240 motion jpg avi files.
upscaled the resolution 3 times each dimension, so 960x720 and then run the optical flow script, altered it to let it average out 17 frames (8 previous, 8 next).
The result is actually not bad. It still lacks some detail of course, but shadow noise and jpg blocking artifacts are all gone.

@Alex
In the makefile I see the average frame is made by the command convert.
But when I type convert in terminal, I don't see an option for average ?
I'm curious if there are more options instead of average, maybe median or something ?




a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10704
  • 5D Mark Free
Re: Experiment - reducing aliasing in post using optical flow
« Reply #46 on: November 24, 2017, 02:59:51 PM »
Sounds cool - mind sharing a few frames?

For median and other methods for combining images: https://www.imagemagick.org/discourse-server/viewtopic.php?t=17519 and https://www.imagemagick.org/script/command-line-options.php#evaluate-sequence

Would be nice to pick or weight the images to average, based on sharpness, similarity to original, whether there's some slight camera movement... probably best done in a Python or Octave script.

Levas

  • Hero Member
  • *****
  • Posts: 969
  • 6d - Nightly build user
Re: Experiment - reducing aliasing in post using optical flow
« Reply #47 on: November 24, 2017, 05:26:01 PM »
Had to search for an avi without relatives in it, found one  :D
I think this is a good example of what aligning and averaging can do.
Before (upscaled tif from 320x240 to 960x720 with Lightroom

Same frame, but averaged with 8 previous and 8 next frames:


Not very detailed, but much better then the original.
Think median will give more detailed results, will check out the links you send.

Deadcode

  • New to the forum
  • *
  • Posts: 41
Re: Experiment - reducing aliasing in post using optical flow
« Reply #48 on: November 24, 2017, 06:31:53 PM »
Great job Levas!

Could you please do some tests with my 5D2 footage? You can download it from the following link:
https://wetransfer.com/downloads/7747cb1b86614e01dc1acb9bf11bd2a220171124170652/f7fd5be4b588f945da4898c6c41feb9e20171124170652/f83ff8

I think on this example "motion picture" means a totally different thing... This was my worst experience with moire/aliasing. Yes you can clean up a lot with chroma blur. i filmed this about a year ago, now im filming around F16 if im shooting with infinity focus, so now moire/aliasing is not that much of a problem. Im curious what can you do with this if you are using your method.

Levas

  • Hero Member
  • *****
  • Posts: 969
  • 6d - Nightly build user
Re: Experiment - reducing aliasing in post using optical flow
« Reply #49 on: November 24, 2017, 07:26:49 PM »
EDIT:
Got a averaged frame too, frame 9 averaged with 8 previous and 8 next.
Can't hardly see the difference with the TIF I got with RawTherapee.
For the pixelpeepers, 3 files on google drive, original DNG, the Rawtherapee TIF and the averaged DNG files:
https://drive.google.com/drive/folders/1yOffbd7OzRnr6LEKoJqXIddZ5bFLu03c?usp=sharing




That one was rather easy  8)

The optical flow script wasn't even needed for this, only RawTherapee was used:
See this post for a youtube link to see the workflow:
http://www.magiclantern.fm/forum/index.php?topic=19687.msg185429#msg185429

Just to see what happens, I'm going to use the optical flow script on frame 9 too.
But that's gonna take a while, will upload it when its done.