Author Topic: Depth map generation using focus stacking  (Read 1200 times)

Karma Lies

  • Just arrived
  • *
  • Posts: 1
Depth map generation using focus stacking
« on: December 17, 2019, 08:36:14 AM »
Hi! This is my first post, but I've been a longtime user of ML. I just graduated from Purdue with a degree in computer science, where I researched the concept of generating depth maps for a scene using a DSLR. My vision is to be able to take an input stack of images as DNGs which are then processed into a single, all-in-focus DNG with an embedded depth map. That could open up the creative post-processing potential for focus stacks in RAW editors like Photoshop/Lightroom since those programs already support the use of embedded depth masks.

At the project's current state, there's quite a bit of room for improvement in polishing up the generated depth maps. Although, just a few improvements (i.e. image segmentation, Homography/ECC alignment) to the core pipeline of the project could improve the quality of these depth maps to near perfection. The algorithm is currently fed by JPEG inputs for ease of fleshing out the core of the program, but I want to eventually integrate DNG processing for the above reasons.

The work I've done so far is standalone to ML, but I would be interested in involving my project in some ML development (if deemed justifiable or necessary beyond the current focus stacking module).

Down the road, I would even like to integrate depth by defocus algorithms to generate a depth map from 2 or 3 shallow depth-of-field shots. That would unlock some potentially game-changing portrait post-processing capabilities.

Feel free to check out the project's GitHub repository: https://github.com/tonyketcham/Depth-Mapper

I would love some more contributors in manifesting this vision. To my knowledge, I haven't seen any other open source projects like this for creatives -- only a couple proprietary depth-map generators and then some algorithms described in academic papers without any source code.

yokashin

  • Member
  • ***
  • Posts: 102
Re: Depth map generation using focus stacking
« Reply #1 on: December 17, 2019, 08:42:04 PM »
Welcome to the forum  :)
You may be interested in a similar topic.

https://www.magiclantern.fm/forum/index.php?topic=24170.msg217588#msg217588

histor

  • Freshman
  • **
  • Posts: 60
Re: Depth map generation using focus stacking
« Reply #2 on: December 17, 2019, 08:52:59 PM »
Welcome and... you have some real giants to compete with. Zerene Stacker is definitely the first one to mention.

names_are_hard

  • Contributor
  • Senior
  • *****
  • Posts: 293
  • 200D idiot
Re: Depth map generation using focus stacking
« Reply #3 on: December 17, 2019, 08:59:43 PM »
Cool project, and congratulations on graduating :)

I skimmed your code, it looks approachable.  It's clearly producing usable depth maps!  I like macro work a lot so I could use a nice automated focus stacking workflow, I might have a play.  The demo image I would guess has some artifacts from Jpeg compression, especially obvious around the head of the panther figurine.  The white aura means it thinks there's pixels close to the viewer, right?  The shapes there look like Jpeg spatial quantisation to me.  The random noise in the lower-right corner also looks funky though, and that's probably unrelated.

What were you hoping to gain from ML integration?  I can see some angles, ML can control focus and customise info embedded in the image.  Focus peaking info might be useful to embed.

You may well enjoy the work of Levon Biss:
https://levonbiss.com/macros/macro-information
http://microsculpture.net/

ilia3101

  • Moderators
  • Hero Member
  • *****
  • Posts: 924
Re: Depth map generation using focus stacking
« Reply #4 on: December 17, 2019, 10:49:39 PM »
Amazing!!!

I know this is not strictly ML related, but I am glad you came here. This forum is a great place for open source image processing.

chris_overseas

  • Moderators
  • Member
  • *****
  • Posts: 233
Re: Depth map generation using focus stacking
« Reply #5 on: December 18, 2019, 10:28:00 AM »
Hi Ilia3101, looks like an interesting and promising project. Are you familiar with the dual pixel RAW mode of the newer Canon cameras? It seems like (multiple?) dual pixel images could be a good candidate to use as input to your algorithm.

Here are a few relevant links:

https://photographylife.com/what-is-dual-pixel-raw-in-the-canon-5d-mark-iv
https://www.magiclantern.fm/forum/index.php?topic=17695.msg172097#msg172097
https://a1ex.magiclantern.fm/bleeding-edge/5D4/5d4-dual-pixel.html
https://a1ex.magiclantern.fm/bleeding-edge/5D4/dual-pixel/5D4-lv-daf-raw.html
https://www.osapublishing.org/oe/abstract.cfm?uri=oe-24-12-12868
EOS R5 1.1.0 | Canon 16-35mm f4.0L | Tamron SP 24-70mm f/2.8 Di VC USD G2 | Canon 70-200mm f2.8L IS II | Canon 100-400mm f4.5-5.6L II | Canon 800mm f5.6L | Canon 100mm f2.8L macro | Sigma 14mm f/1.8 DG HSM Art | Yongnuo YN600EX-RT II