Depth map generation using focus stacking

[Karma Lies]

Hi! This is my first post, but I've been a longtime user of ML. I just graduated from Purdue with a degree in computer science, where I researched the concept of generating depth maps for a scene using a DSLR. My vision is to be able to take an input stack of images as DNGs which are then processed into a single, all-in-focus DNG with an embedded depth map. That could open up the creative post-processing potential for focus stacks in RAW editors like Photoshop/Lightroom since those programs already support the use of embedded depth masks.

At the project's current state, there's quite a bit of room for improvement in polishing up the generated depth maps. Although, just a few improvements (i.e. image segmentation, Homography/ECC alignment) to the core pipeline of the project could improve the quality of these depth maps to near perfection. The algorithm is currently fed by JPEG inputs for ease of fleshing out the core of the program, but I want to eventually integrate DNG processing for the above reasons.

The work I've done so far is standalone to ML, but I would be interested in involving my project in some ML development (if deemed justifiable or necessary beyond the current focus stacking module).

Down the road, I would even like to integrate depth by defocus algorithms to generate a depth map from 2 or 3 shallow depth-of-field shots. That would unlock some potentially game-changing portrait post-processing capabilities.

Feel free to check out the project's GitHub repository: https://github.com/tonyketcham/Depth-Mapper

I would love some more contributors in manifesting this vision. To my knowledge, I haven't seen any other open source projects like this for creatives -- only a couple proprietary depth-map generators and then some algorithms described in academic papers without any source code.

[yokashin]

Welcome to the forum 
You may be interested in a similar topic.

https://www.magiclantern.fm/forum/index.php?topic=24170.msg217588#msg217588

[histor]

Welcome and... you have some real giants to compete with. Zerene Stacker is definitely the first one to mention.

[names_are_hard]

Cool project, and congratulations on graduating

I skimmed your code, it looks approachable.  It's clearly producing usable depth maps!  I like macro work a lot so I could use a nice automated focus stacking workflow, I might have a play.  The demo image I would guess has some artifacts from Jpeg compression, especially obvious around the head of the panther figurine.  The white aura means it thinks there's pixels close to the viewer, right?  The shapes there look like Jpeg spatial quantisation to me.  The random noise in the lower-right corner also looks funky though, and that's probably unrelated.

What were you hoping to gain from ML integration?  I can see some angles, ML can control focus and customise info embedded in the image.  Focus peaking info might be useful to embed.

You may well enjoy the work of Levon Biss:
https://levonbiss.com/macros/macro-information
http://microsculpture.net/

[ilia3101]

Amazing!!!

I know this is not strictly ML related, but I am glad you came here. This forum is a great place for open source image processing.

[chris_overseas]

Hi Ilia3101, looks like an interesting and promising project. Are you familiar with the dual pixel RAW mode of the newer Canon cameras? It seems like (multiple?) dual pixel images could be a good candidate to use as input to your algorithm.

Here are a few relevant links:

https://photographylife.com/what-is-dual-pixel-raw-in-the-canon-5d-mark-iv
https://www.magiclantern.fm/forum/index.php?topic=17695.msg172097#msg172097
https://a1ex.magiclantern.fm/bleeding-edge/5D4/5d4-dual-pixel.html
https://a1ex.magiclantern.fm/bleeding-edge/5D4/dual-pixel/5D4-lv-daf-raw.html
https://www.osapublishing.org/oe/abstract.cfm?uri=oe-24-12-12868