Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - Taynkbot

Here's an idea I would like to hear some thoughts on.

When I snap a normal picture, the field of view of my sensor is laid out like a sheet over the pixels, which we could consider to be like lots of red party cups fitted closely together.
Looking at the sheet on the whole, we'll see the outline of lots of rings (being the rims of the cups), but the majority of the sheet will be drooping in the cups. In this example, most of the light reaching the sensor will end up illuminating one pixel or another.

In the lite version of this proposal, I want to discuss the possibility of capturing the parts of the FOV that don't ultimately land on a pixel, i.e., they land on the rim of a cup or between rims. By utilizing small vibrations that I think can be generated using an image stabilizer or the environment, we can do the equivalent of wiggling the sheet around. As we wiggle it, the parts of the sheet that before landed in between the cups, will eventually end up in one cup or another. Thus, we should be able to take a handful of pictures of the same scene, combine them, and end up with slightly higher resolution.

That seems possible, but not all that useful. The main hope I have is to use this far more dramatically.

(The following is just a thought experiment, and I make no attempt to describe this method mathematically).

Let's imagine we have a single-pixel (this being composed of the typical three color sub-sensors) camera. Let's consider two different lenses focusing two different fields of view, one focused narrowly (A), and one wide angle (B).

Regardless of the narrowness of the field (assuming it's not so narrow as to require QM considerations), there will always be some blending of details. This is quite clear with B, where we might imagine a 90 degree field of view focused on our single pixel. Composing an image where a green lawn takes up the lower half of this FOV and a red building takes up the FOV's upper half, will result in an image that is blended yellow. We can still refer to the original three color channels to see the contribution each has, but no information will be recorded about where within the FOV they come from. But this is only true with a snapshot.

With a narrower focus, blending will still occur, but do to the nature of things, the narrower the view the less the disparity across the FOV. If we use lens A, with say a 1 degree FOV, from the same distance that we used lens B, we can much more accurately represent the image by stitching together something like a thousand pictures with small angle adjustments. This isn't miraculous, as we've essentially used our single pixel to replicate temporally the job that a higher res sensor could do spatially, but it segues into my next point.

Let's aim lens A at some transition point between the red house and the green grass. For the sake of this argument, let's consider an infinitely sharp transition line between the physical house and grass, meaning that any blending of red and green in our sensor is do to the blending described above and isn't inherent in the scene. How might we reconstruct an image that shows this contrast even beyond the resolution and FOV combination might allow? This is the purpose of this analysis.

We begin at the transition point described above and allow our camera to vibrate, to move randomly, with the maximum distance of displacement resulting in a half degree of FOV displacement. So, as our sensor moves around, it will record data from a total area of 2 degrees FOV. To simplify this down to what's relevant, let's only consider Y axis displacement. Let Y=0 be our starting position, where the FOV is cleanly divided horizontally with the red house on top and the green grass on bottom. Let Y=-1 be a half a degree moved down, so the grass fills 100% of the FOV; let Y=+1 do likewise but with regard to the house. Snapping a picture at Y=-1 will result in a purely green image; snapping a picture at Y=+1 will result in a purely red image; snapping a picture at Y=0 will result in a yellow image; snapping a picture anywhere in this range will result in some intermediate level of blending.

We can intuit from the setup described above that it should be possible to construct an image of a higher level of contrast than the sensor can provide. It seems we can sharpen the line of contrast as much as we want, regardless of the resolution of the sensor or FOV of the focus, by using a couple pieces of additional information.
In a world where infinitely precise measurements were possible, a single pixel could theoretically resolve a quasar, but we have some constraints on the information we can obtain.
We will be limited by the noise in the environment, noise in the sensor/camera, the A/D converter that changes the analog input of each color sub-pixel into some number of bits (which reduces the information for the sake of fidelity), and we will be limited by the accuracy with which we can measure the position of the camera at the moment each picture was taken (the key concept to this method).

To put this all together, we would begin with a certain number of pictures taken from a single vantage point while the camera was in some way vibrating or moving very subtly from one frame to the next. Within the metadata of each image, we would include the position of the camera (either by measuring absolute position in some way, or by using an accelerometer of some kind to measure the acceleration of the camera from frame to frame). Assuming we can measure the position with an accuracy within the FOV of a single pixel, then we should be able to use this method to improve the resolution of a single pixel by combining the position information with the color levels at that position, and comparing this to the color levels of nearby positions. As described, this is all scalable, and so it should be possible to implement even in cameras of very high res.

In theory, any camera coupled with an accelerometer of sufficient accuracy could be used to increase resolution beyond the resolution of a sensor. Though random vibrations from the environment could be used to achieve the displacement required, they would be practically random in direction and magnitude, and so more pictures would need to be taken of the same scene than strictly necessary. The optimal setup would be to have a mechanism for very controlled movements that could move the sensor in precisely calibrated ways so as to reduce redundancy, and allow for full coverage.

I don't know the details of how accurately we can measure positions in this way, and depending, this method might be lost to noise. Still, I found the thought experiment interesting, and would appreciate thoughts and elaborations.
I would like advice on using my dual-pixel (dp) 5D4 to generate a depth map. Here is a paper I found discussing what I believe to be a compatible idea (though I'm not entirely sure if Canon's dp is capable of the differential phase detection outlined here):

I would like to attempt to generate a depth map manually, in post, from a dp RAW image before getting more ambitious and trying to incorporate it into ML. If it is possible in post, I would like to attempt to automate the process in-camera (though I admit I'm fuzzy on how computationally heavy are the algorithms needed to generate a depth map) by creating an option similar to the option that records both RAW and JPEG versions of an image, but in this case I would like the JPEG to be a generated depth map.
The in-camera option will probably be too intensive for my camera to perform acceptably, but I would appreciate advice and more information.

Do you know of any programs that already exist that can take two images, half a pixel apart, and combine them into a stereoscopic image? Any open source code that might aid me would also be very welcome.

Finally, for the stickler sticklebacks who want to describe how inferior this kind of approach is compared to a properly generated depth map..... I'm not asking for perfection. Also, I don't want to have to buy a Light L16, but I am up for donations of this sort:

Here are some similar posts that didn't explicitly discuss the possibility of exploiting dp.
Maybe one of you would be kind enough to enlighten me on a problem I'm having. I can't use the set + scroll dial command, the set + erase in playback, actually none of the arrow hotkeys work for me. So what do you think my problem is? Are the features not yet added, or do I need to do something to fix it?

Also, I don't know if you guys want ideas, but I'll throw mine out there and you can take it or leave it. Would it be possible to add to the FPS Override, increasing the setting to allow a frame rate such as 128 but at a loss of resolution, possible 640X480? If you COULD add it, I would really benefit from it, if not, I'd love to understand what would prevent a mod like that.

Magic Lantern is absolutely delightful and it's been a deep pleasure exploring. I absolutely respect all of your creativity and ingenuity in writing this awesome program.  Have a fantastic week, guys!
So I was thinking that it'd be awesome if you could increase the video frame rate. I know you probably can't get a whole lot faster than 30fps on 1080 but maybe increase frame rate on lower quality videos so we could get some high speed shots!
ONLY a suggestion though and only if you think it'd be worth throwing in (you may already be doing it, I don't know). Thanks again guys! Your work is awesome and I appreciate it!

I know right? It seems more like a firmware update than a new camera. A new 360 turn-able dial? A new body finish? Come on! I really hope that the lack of work they put into the T5i went into the 7DMKII.
I really hope you guys continue to finish ML for the T4i. The T5i doesn't look like that much of an upgrade from the T4i and I don't want to have to waste the money on it.
Just letting you know I'd REALLY appreciate if you continued programming the T4i's ML. You guys are awesome!
dngrhm, hey, thanks! I forgot about that trick. For some reason I was thinking it wasn't working with my remote. I guess it flips up two seconds before and then takes the picture.
Anyway, thanks again :D
Just a suggestion, I thought that if you could add a setting that pulls the mirror up before taking the picture; that could reduce camera shake. Maybe add a setting that let you decide how much time to keep the mirror up before taking the picture.
JUST an idea from a guy that has NO experience in this but knows it's a challenging thing to do. So if you think this would be worth adding, that'd be awesome. But I definitely don't expect you to do it if you don't want to :D I'm very happy with all the features you've already added and I don't want to be a pain. Thanks for all your work!

@BrightGarden, I just turned my setting, touch to shoot, off, went to HDR and back, but the setting was still off. I guess I don't have that problem. However, I have noticed that it's turned on a couple of times without me setting it... I thought I was losing my mind, I guess not.
Excited for this to be released! This sounds like very stable programming from what I've read... nobody's had their camera blow up. Thanks for all your hard work, I've never heard of this before today but it looks amazing! Although I'm unable to financially support, I'll give all the moral support I can :D Peace!