adding new defishing tables? how to do it?

[dmg]

I maintain panotools (the remapping library at the core of hugin, responsible for "defishing") and I am one of the co-authors of the Panini projection. I would like to contribute to ML by creating LUT tables for various fisheye lenses (I have several of them). But, I need a bit of help. Where do I find the file xy.png and the script: vram/defish-lut.m? Could somebody provide me some hints on where to get started?

thank you!

--dmg

[a1ex]

Hi dmg,

The xy.png was renamed to xy.tiff and can be found here: https://bitbucket.org/hudson/magic-lantern/src/tip/vram

In theory, all you have to do is to create the PTO files, customize make.sh, and run it. Then, filenames are hardcoded in ML source for now, but can be changed to work just like the cropmarks.

The file names are limited to 8.3, so it may be a good idea to embed a string or some metadata in the LUT files.

[dmg]

Hi dmg,

The xy.png was renamed to xy.tiff and can be found here: https://bitbucket.org/hudson/magic-lantern/src/tip/vram

In theory, all you have to do is to create the PTO files, customize make.sh, and run it. Then, filenames are hardcoded in ML source for now, but can be changed to work just like the cropmarks.

The file names are limited to 8.3, so it may be a good idea to embed a string or some metadata in the LUT files.

Thanks A1ex, that is exactly what I was looking for.

Looking at the current PTO files it seems that only remapping is done, nothing specific to the distortion of the lens is done. This has the advantage that all yone needs is LUT tables per field of view. 

For fixed-fov this method works great. There are the 8mm (Samyang, Sigma, Peleng), 10.5 (Nikon, shaved on a Canon FF), and 15 mm fisheyes (Canon). Those are the 3 commonly used ones in Canon.

For zooms (such as the new 8-15) this is problematic. A future feature might be to use the Focal length of the lens to choose the LUT. This feature will work well for sigma and canon lenses, but it will not work for the Nikon, Samyang, or Peleng (perhaps specify a default if no metadata is available.

Now, this is a "bug": If one presses the preview button twice, it does the remapping of the remapped images. Perhaps rotating through the LUTs would be an idea--but I am not sure it is a good one.

--dmg

[dmg]

one more thing.

Even if two lenses are of the same focal length, they will project very different images. The best way to compare them is this page, by Michael Thoby (I'll email him to see if I can get him to join--he is the ultimate expert in fish eyes in my view)

http://michel.thoby.free.fr/Canon_8-15mm/Fisheyes_galore/Fisheye_Images_Galore.html

It is interesting to see that at 8mm each of the lenses has a very different projected image. This basically means that ML needs LUTs per lens. I will start creating them for the lenses I have. In the current version of ML the user will have to "preload" the one that he/she wants to use the most.

--dmg

[dmg]

Ok. I was able to understand how the remapping works. it is very clever.

What I like is that it does not have to be perfect. I just needs to be close enough to give an idea of what the photos will look like.

But some suggestions:

* the Samyang has a very different projection, and instead of f2, it should use f10 (stereographic)
* the nikkor 10.5 should use the f20 (Thoby)
* for other lenses, f2 is good.

But the critical parameter is the input FOV of the lens. Here is where each of them has very different parameters.

For the Canon 8-15mm at 8mm, the fov should be 264 (the lens rapidly compresses as you get to infinity, while the samyang does it at a smaller rate).

I have created LUT tables for the canon8-15 at 8mm. I guess that the question is, what would be the best way to share them?

for an example of the impact of a custom LUT for the canon8, see the following http://turingmachine.org/~dmg/temp/lut

My screenshots are slightly squeezed (vertically). Am I doing something wrong while converting them? I used python ../../../src/422-tif.py . to convert them.

--dmg

[a1ex]

Very nice info, thanks a lot.

Are your LUT examples screensots from the 5D/50D/500D playback screen? (looks so, because of the black bar). These cameras are stretching the image by a factor of 8/9 (the image is 3:2 but the screen is 4:3).

About remapping algorithm, which you probably already understood.
* I use a 8-bit LUT which should be small enough. A 16-bit LUT is more likely to cause issues on 60D, which is low on memory. But accuracy is not enough.
* The LUT only stores a quarter of the picture, since the defishing is symmetrical (I assume yaw=0, pitch=0, roll=0).
* In LiveView it only has to be fast (though it's still pretty slow). In playback mode it can be a bit slower, so I average neighbouring coordinates from the LUT to reduce the quantization error a bit. Also, the playback algorithm works at pixel level. The source image is 720x480 (sometimes more on external monitors).
* The quality achieved by "nona" from the same source image is excellent IMO (I've tried on a screenshot with small text and antialiasing was just perfect). I've experimented with a floating-point algorithm and bilinear interpolation, but it was very slow.

About making these LUTs available to general public - that's my job, I need to remove the hardcoded paths from ML and accept user LUTs.

[dmg]

Very nice info, thanks a lot.

Are your LUT examples screensots from the 5D/50D/500D playback screen? (looks so, because of the black bar). These cameras are stretching the image by a factor of 8/9 (the image is 3:2 but the screen is 4:3).

Yes, but I think something else is going on too. I think the bottom half is more compressed than the top half.

About remapping algorithm, which you probably already understood.
* I use a 8-bit LUT which should be small enough. A 16-bit LUT is more likely to cause issues on 60D, which is low on memory. But accuracy is not enough.
* The LUT only stores a quarter of the picture, since the defishing is symmetrical (I assume yaw=0, pitch=0, roll=0).

you also assume d,e parameters should be zero. If not, they might show a small artifact right in the center of the image.

* In LiveView it only has to be fast (though it's still pretty slow). In playback mode it can be a bit slower, so I average neighbouring coordinates from the LUT to reduce the quantization error a bit. Also, the playback algorithm works at pixel level. The source image is 720x480 (sometimes more on external monitors).
* The quality achieved by "nona" from the same source image is excellent IMO (I've tried on a screenshot with small text and antialiasing was just perfect). I've experimented with a floating-point algorithm and bilinear interpolation, but it was very slow.

I agree, that is the big challenge. But I think that, even with a very rough approximation, it is useful, even in liveview. I see myself using it to get me an idea of what a shot would be before it is remapped in the computer.

About making these LUTs available to general public - that's my job, I need to remove the hardcoded paths from ML and accept user LUTs.

I'll write a document describing how to create and load a user defined LUT with 2.3.


--dmg

[a1ex]

Good news regarding defishing: I was able to run it in almost real-time (probably 10-15 fps) on 5D Mark II, by caching the offset for every single pixel pair in a big look-up table.

It's pretty memory intensive (the new LUT takes 360x480x4 bytes (700K) and is camera-specific, but can be computed from the generic LUT fairly quickly).

I think it's reasonable to go with a 16-bit generic LUT (180x240 area with 16-bit coordinates), and defishing to be done from the high-resolution image buffer (1 or 2 megapixels). There are still some memory issues to be solved. With some interpolation, the generic LUT can be even smaller, but for this I need to brush up my math skills a bit.

Regarding support for more lenses, I suggest sharing only the PTO files, so I can compile the LUTs and find some easy way to integrate them in the menus. I thought at having the LUTs organized by folders, one folder containing LUT files for each lens.

For example:
SAMYANG8: rectilinear, Panini, maybe also vertical Panini.
NIKKOR10.5: rectilinear, Panini

The analytic description of those transformation is also an option, but for this I really need your help. There is floating point arithmetic in ML, so building the LUTs on the fly from analytic formulas shouldn't be a huge problem.

[dmg]

Sorry Alex, i haven't been reading the Forum.

One potential challenge with having one directory per lens, is what happens if one has a zoom? like the EF 8-15?