Extreme macro with ML

Started by pulsar124, April 19, 2020, 01:06:30 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

pulsar124


yourboylloyd

Wow this is amazing! This feels almost microscope level of small. And they are all so sharp! Who knew that sanding paper looked like that.

I'm curious  more about how this was done. Which camera/lens/fstops/etc did you use? Did you have to blast your subjects with a crazy amount of lighting? I see the tutorial about the fast stacker so I'll give it a read and it'll probably answer a lot of these questions.
Join the ML discord! https://discord.gg/H7h6rfq

a1ex

Yeah, really nice.

I'd appreciate a sequence of raw image files (e.g. the one of the CPU pin, or the ant head, since they seem to be a little blurrier than others), and some notes about postprocessing.

pulsar124

Hi,

Some of these photos have lots of details on Flickr (most of them are old, so I won't even remember the details if they are not there!). Indeed, a few of them are microscopic (in the sense that I use a pro microscope objective with some extension tubes and/or telephoto lens). The top 3 shots are made with 20:1 magnification, using Olympus ULWD Neo Splan 20 0.40 8/- f=180 microscope objective (I was lucky to get one on ebay for ~150$; normally ULWD ones sell for $400+). I also have a 10:1 Nikon microscope objective, for 10:1 macro. For lower magnifications, I use a very decent "macro lens" - old film enlarger lens Schneider Componon 28mm f4, with and without extension tubes (gives me 1.5-5:1 macro).

A lot of postprocessing goes into shots like this, especially extreme macro like 20:1 which suffer significantly from diffraction. (At 20:1, my 6D pixel size is only 0.3 um - much smaller than the light wavelength). It typically involves multiple sharpening steps, at different scales. I wrote my own scripts and programs for the whole workflow, from raw images processing, bad pixels removal and white balance, to alignment, to focus stacking, and multi-scale sharpening:

https://github.com/pulsar123/Macro-scripts

I think I am getting better with PP, that's why my more recent shots (sanding paper) look sharper than earlier ones (like CPU pin).

I am still doing research into proper PP for focus stacks. My holy grail is to be able to apply 3D deconvolution to the stack before focus stacking - this should remove all those glowing blobs around bright parts of the stacked image. Right now, I am playing with ImageJ with DeconvolutionLab2 plugin (medical software, used for microscopy).

In terms of light - it is not that extreme. For many shots I use a flash with a softbox (I figured out how to use flash with FRSP - the trick is to find a delay, specific to your camera, between triggering FRSP and triggering the flash), typically at 1:8 power. The FRSP exposure is so long (0.4s for my 6D), that one can just use interior ceiling lights for proper exposure (but I think sharpness suffers, because of long exposure - from microvibrations). E.g., this wire spiral from a wire nut:

Macro mountain by SyamAstro (750,000 views - thank you!), on Flickr

More examples:

https://www.flickr.com/photos/syamastro/albums/72157627584126318

In particular (10:1 macro),

Macro 10:1 by SyamAstro (750,000 views - thank you!), on Flickr

and (6:1)

Peppercorn by SyamAstro (750,000 views - thank you!), on Flickr



a1ex

Quote from: pulsar124 on April 20, 2020, 06:13:56 PM
I wrote my own scripts and programs for the whole workflow, from raw images processing, bad pixels removal and white balance, to alignment, to focus stacking, and multi-scale sharpening:

https://github.com/pulsar123/Macro-scripts

Very nice. Scripting around align_image_stack and enfuse, with additional processing :)

The reason I've asked for a test image sequence: I was recently asked by the Apertus folks to help them with a focus stack of a PCB, where align_image_stack was failing (and, as it turned out later, enfuse had suboptimal output on that image sequence). The main problem was the image sequence itself, which apparently was captured wide-open with an old manual lens; I've recommended them some new settings (along the lines of stopping down the aperture and using ISO 100, even if that meant a slightly long exposure), but re-shooting that particular PCB wasn't an option at the time (don't ask me why). So, I ended up rewriting align_image_stack and the focus stacking part of enfuse, from scratch, as Octave scripts, for that particular image sequence. Yes, in that case it was easier - for me - to reinvent the wheel, rather than fine-tuning the existing software.

When seeing your images, I thought my modified align_image_stack / enfuse versions might give slightly better output, so I was tempted to give it a try. Now that I see you are using an workflow based on align_image_stack and enfuse, you might be able to try them yourself (there are minimal modifications on the command line). Please find my scripts here:

http://files.apertus.org/align_image_stack/

The first script is align_image_stack_finetune.m. It uses an initial brute force search, followed by coordinate descent (parameters: X and Y offsets, zoom aka focus breathing, optional rotation), first on low-resolution copies of the original images, then, increasing the resolution until 1/8 x 1/8 of the original (you can change that by editing the script). It aligns all the images in the sequence, to the middle image (as specified on the command line). It's significantly slower than the original align_image_stack, but it worked out of the box on the PCB test sequence (a few images available on that link, but not the entire sequence). It should handle images with tiny depth of field, where align_image_stack is unable to find its control points.

The second one is focus_stack.m, which outputs a weighted average of a pre-aligned image sequence. Weight is computed by looking at how much each image differs from a blurred copy of itself (computed as grayscale, for speed reasons). The output contains less halo artifacting, compared to enfuse, although I didn't use any deconvolution. I haven't tried it on other images.

Before: PB5-enfuse.jpg
After: PB5-focus_stack.jpg

If you decide to try these scripts, please share the results. If this focus stacking algorithm can be useful to a wider audience, I'm happy to contribute it upstream (to be included in the official enfuse). With align_image_stack, it's a bit more complicated, as my approach is fundamentally different; besides, it would take significant work to convert it to another programming language (haven't checked the upstream sources yet).

c_joerg

Quote from: a1ex on April 21, 2020, 07:34:38 AM
If you decide to try these scripts, please share the results.

I'd like to try it...
I tried MATLAB but the differences are too big. It is already failing
pkg load image
I am not allowed to install Octave on my company computer ..
EOS R


a1ex

Quote from: c_joerg on April 21, 2020, 09:03:46 AM
pkg load image

Haven't touched Matlab in years, but you might be able to run the scripts by commenting any "pkg load" lines, converting comments from # to % and fixing up stuff as you go. Hopefully the changes required are minimal, but no guarantees.

https://en.wikibooks.org/wiki/MATLAB_Programming/Differences_between_Octave_and_MATLAB

Also, looks like octave has a --traditional mode, which should warn about (known) Matlab compatibility issues edit: nope, one has to use warning('on', 'Octave:language-extension'). If you can't get the scripts working otherwise, I might give it a try.

Quote from: c_joerg on April 21, 2020, 09:03:46 AM
I am not allowed to install Octave on my company computer ..

Well, assuming you are using Windows, Octave can be simply unzipped and ran; no need to install it system-wide. Therefore, as long as you are allowed to use a web browser and a zip extractor, I expect it to work.

https://wiki.octave.org/Octave_for_Microsoft_Windows

c_joerg

Quote from: a1ex on April 21, 2020, 12:30:18 PM
Haven't touched Matlab in years, but you might be able to run the scripts by commenting any "pkg load" lines, converting comments from # to % and fixing up stuff as you go.

That was the first thing what I did...

But I'm not sure how to use it in MATLAB (Pass the image to the script)
Usage: octave-cli focus_stack.m foo.png bar.png baz.png

When I do similar thinks in MATLAB (like Median over Images), I only give a path to the script. Then the script processes every image in this folder.

Quote from: a1ex on April 21, 2020, 12:30:18 PM
Well, assuming you are using Windows, Octave can be simply unzipped and ran; no need to install it system-wide. Therefore, as long as you are allowed to use a web browser and a zip extractor, I expect it to work.

https://wiki.octave.org/Octave_for_Microsoft_Windows

I think I will try this....
EOS R

pulsar124

@a1ex: Thanks, I'll give your scripts a try. Actually, ever since I switched to always run align_image_stack from the middle image (this is what my Align.sh script does), I never had issues with this program. So I am more interested in your focus stacking script, as bright halos bother me a lot in my stacked images. As I mentioned, I am looking into 3D deconvolution of bright field microscopy (which is more or less what macro focus stacking deals with), which would be the best tool for the task. One can try to use theoretical PSF (e.g. this recent Science paper https://www.sciencedirect.com/science/article/pii/S0006349517309840), but they are complicated to compute, and I am sure are not exact match to real life macro photography. I'll try to measure PSF myself - one needs a tiny (smaller than the resolution limit) light object against dark background. I am thinking perhaps a reflection of a single bright LED (placed at a large distance) in a small metal ball from a bearing might do the trick. Or perhaps a few tiny particles of flour on a clean glass plate.

Have you seen this software: https://github.com/PetteriAimonen/focus-stack  ? It is super fast (both aligning and stacking are using GPU), but many times the alignment part failed on my, due to some "failed to converge" issue.

c_joerg

Quote from: a1ex on April 21, 2020, 12:30:18 PM
Well, assuming you are using Windows, Octave can be simply unzipped and ran; no need to install it system-wide.

Nothing is simply at beginning ;)


E:\New\Stack_Test\M3_100mm_Stack_Test\LR>C:\Tools\octave\mingw64\bin\octave-cli focus_stack.m IMG_5051.jpg IMG_5052.jpg IMG_5053.jpg
warning: addpath: __OH__/share/octave/packages/image-2.10.0: No such file or directory
warning: called from
    load_packages_and_dependencies at line 48 column 5
    load_packages at line 47 column 3
    pkg at line 461 column 7
    focus_stack at line 12 column 1
Processing IMG_5051.jpg .error: could not find any INDEX file in directory __OH__/share/octave/packages/image-2.10.0, try 'pkg rebuild all' to generate missing INDEX files
error: called from
    describe>parse_pkg_idx at line 94 column 5
    describe at line 59 column 40
    pkg at line 555 column 43
    __unimplemented__>check_package at line 540 column 15
    __unimplemented__ at line 321 column 11
    focus_stack at line 44 column 7


I comment that out
   
# pkg load image



E:\New\Stack_Test\M3_100mm_Stack_Test\LR>C:\Tools\octave\mingw64\bin\octave-cli focus_stack.m IMG_5051.jpg IMG_5052.jpg IMG_5053.jpg
warning: function name 'show' does not agree with function filename 'E:\New\Stack_Test\M3_100mm_Stack_Test\LR\focus_stack.m'
error: 'im' undefined near line 16 column 10
error: called from
    focus_stack at line 16 column 3

   

EOS R

Robby24


c_joerg

I have now managed to port the script to MATLAB. Quite a few changes are necessary. Especially in the operations of 2-dimensional and 3-dimensional matrices. The most important thing was that I learned a lot about the algorithm.

I am really impressed about the script. It works much better than CS6. 
Maybe I made a mistake with the porting. So far I have not carried out an algin. I'll port that next week.

Here is the MATLAB version of the script and the result from 70 images
https://drive.google.com/open?id=1qaQAEvg7g_Zllbq7BDPHxjgqIYSj3zJB

This is the result from HeliconFocus. HeliconFocus crops very strongly and the viewing angle is different

EOS R

nikfreak

Quote from: pulsar124 on April 20, 2020, 06:13:56 PM
...
(I figured out how to use flash with FRSP - the trick is to find a delay, specific to your camera, between triggering FRSP and triggering the flash), typically at 1:8 power. The FRSP exposure is so long (0.4s for my 6D)...

Please explain quoted part in detail!
Are you using radio triggers whch you can adjust / program a delay or do you do this with ML specific code. I am very interested in this. And I thought the shutter limit was already solved by a1ex for FRPS - at least on 5D3 !?!?!? So you could probably use more flash power. Pleas tell me I can use some kind of "tail sync hack" with frsp. I need this adjustable delay for flash trigger function :D
[size=8pt]70D.112 & 100D.101[/size]

pulsar124

Quote from: nikfreak on April 23, 2020, 09:38:08 PM
Please explain quoted part in detail!
Are you using radio triggers whch you can adjust / program a delay or do you do this with ML specific code. I am very interested in this. And I thought the shutter limit was already solved by a1ex for FRPS - at least on 5D3 !?!?!? So you could probably use more flash power. Pleas tell me I can use some kind of "tail sync hack" with frsp. I need this adjustable delay for flash trigger function :D

My trick does work (I used it with 50D and 6D), but it requires an external Arduino gadget which controls separately the camera shutter and the flash (in my case, it also controls the focus stacking rail of my design - https://pulsar124.fandom.com/wiki/Fast_Stacker). All the details are here:

https://pulsar124.fandom.com/wiki/User_guide#Electronic_shutter

pulsar124

@a1ex: I just tried your focus stacking (not  the aligning) Octave script on my recent stack. Specifically, it is ~3.5:1 macro with a very good quality macro lens (Componon Schneider 28mm f/4 enlarger lens), 14 um steps, 120 raw images in a stack. First image was processed with enfuse, second - with your script. I did not apply any sharpening. Overall, your script produces more clear results (less halo around bright objects). But it does leave some artifacts (instead of halo, now it's a ring), which will cause issues if you still plan to do some fairly significant PP (sharpening, 2D deconvolution etc). In particular, see the left bottom corner of the image, with the bright piece of copper.

I suspect your script will be the best when used with fairly low magnifications (like in your original example) - less than 1. For magnifications lager than 1, a significant PP sharpening is always involved, and that works better with the enfuse approach.

And of course, another obvious advantage of your script - it doesn't use much memory (~1.5GB). With the ~100 16-bit TIFFs stack, enfuse used almost 20GB of RAM. One can use my Macro-Scripts instead (two-phase enfuse application, which reduces RAM consumption by a square root of N), but it produces even more prominent halos.

Again, 3D deconvolution would be the ultimate solution for halos and hazyness in macro shots...

enfuse by First Last, on Flickr

focus_stack by First Last, on Flickr




Quote from: a1ex on April 21, 2020, 07:34:38 AM
Very nice. Scripting around align_image_stack and enfuse, with additional processing :)

The reason I've asked for a test image sequence: I was recently asked by the Apertus folks to help them with a focus stack of a PCB, where align_image_stack was failing (and, as it turned out later, enfuse had suboptimal output on that image sequence). The main problem was the image sequence itself, which apparently was captured wide-open with an old manual lens; I've recommended them some new settings (along the lines of stopping down the aperture and using ISO 100, even if that meant a slightly long exposure), but re-shooting that particular PCB wasn't an option at the time (don't ask me why). So, I ended up rewriting align_image_stack and the focus stacking part of enfuse, from scratch, as Octave scripts, for that particular image sequence. Yes, in that case it was easier - for me - to reinvent the wheel, rather than fine-tuning the existing software.

When seeing your images, I thought my modified align_image_stack / enfuse versions might give slightly better output, so I was tempted to give it a try. Now that I see you are using an workflow based on align_image_stack and enfuse, you might be able to try them yourself (there are minimal modifications on the command line). Please find my scripts here:

http://files.apertus.org/align_image_stack/

The first script is align_image_stack_finetune.m. It uses an initial brute force search, followed by coordinate descent (parameters: X and Y offsets, zoom aka focus breathing, optional rotation), first on low-resolution copies of the original images, then, increasing the resolution until 1/8 x 1/8 of the original (you can change that by editing the script). It aligns all the images in the sequence, to the middle image (as specified on the command line). It's significantly slower than the original align_image_stack, but it worked out of the box on the PCB test sequence (a few images available on that link, but not the entire sequence). It should handle images with tiny depth of field, where align_image_stack is unable to find its control points.

The second one is focus_stack.m, which outputs a weighted average of a pre-aligned image sequence. Weight is computed by looking at how much each image differs from a blurred copy of itself (computed as grayscale, for speed reasons). The output contains less halo artifacting, compared to enfuse, although I didn't use any deconvolution. I haven't tried it on other images.

Before: PB5-enfuse.jpg
After: PB5-focus_stack.jpg

If you decide to try these scripts, please share the results. If this focus stacking algorithm can be useful to a wider audience, I'm happy to contribute it upstream (to be included in the official enfuse). With align_image_stack, it's a bit more complicated, as my approach is fundamentally different; besides, it would take significant work to convert it to another programming language (haven't checked the upstream sources yet).

nikfreak

a1ex do you know an easy way (some lines of code) to trigger the flash hot shoe within frsp?
[size=8pt]70D.112 & 100D.101[/size]

v8lve

Would be great  8).  So far I have been using a stand alone controller from mjkzz for firing flash during FRSP and second curtain.


https://www.mjkzz.com/single-post/2018/05/08/Flashes-Continuous-Lights-And-Flash-Triggering-Modes