## Digital deformation model for fisheye image rectification |

Optics Express, Vol. 20, Issue 20, pp. 22252-22261 (2012)

http://dx.doi.org/10.1364/OE.20.022252

Acrobat PDF (1003 KB)

### Abstract

Fisheye lens can provide a wide view over 180°. It then has prominence advantages in three dimensional reconstruction and machine vision applications. However, the serious deformation in the image limits fisheye lens’s usage. To overcome this obstacle, a new rectification method named DDM (Digital Deformation Model) is developed based on two dimensional perspective transformation. DDM is a type of digital grid representation of the deformation of each pixel on CCD chip which is built by interpolating the difference between the actual image coordinate and pseudo-ideal coordinate of each mark on a control panel. This method obtains the pseudo-ideal coordinate according to two dimensional perspective transformation by setting four mark’s deformations on image. The main advantages are that this method does not rely on the optical principle of fisheye lens and has relatively less computation. In applications, equivalent pinhole images can be obtained after correcting fisheye lens images using DDM.

© 2012 OSA

## 1. Introduction

3. J. Willneff and O. Wenisch, “The calibration of wide-angle lens cameras using perspective and non-perspective projections in the context of realtime tracking applications,” Proc. SPIE **8085**, 80850S–80850S-9 (2011). [CrossRef]

5. S. Abraham and W. Forstner, “Fish-eye-stereo calibration and epipolar rectification,” ISPRS J Photogramm. **59**(5), 278–288 (2005). [CrossRef]

7. A. Heikkil, “Geometric camera calibration using circular control points,” IEEE T Pattern Anal. **22**(10), 1066–1077 (2000). [CrossRef]

8. M. Grossberg and S. Nayar, “The raxel imaging model and ray-based calibration,” Int J Comput Vision **61**(2), 119–137 (2005). [CrossRef]

10. Z. Zhang, “A flexible new technique for camera calibration,” IEEE T Pattern Anal. **22**(11), 1330–1334 (2000). [CrossRef]

11. D. Schneider, E. Schwalbe, and H. Maas, “Validation of geometric models for fisheye lenses,” ISPRS J Photogramm. **64**(3), 259–266 (2009). [CrossRef]

13. J. Kannala and S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” IEEE T Pattern Anal. **28**(8), 1335–1340 (2006). [CrossRef]

14. D. Gennery, “Generalized camera calibration including fish-eye lenses,” Int J Comput Vision **68**(3), 239–266 (2006). [CrossRef]

13. J. Kannala and S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” IEEE T Pattern Anal. **28**(8), 1335–1340 (2006). [CrossRef]

13. J. Kannala and S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” IEEE T Pattern Anal. **28**(8), 1335–1340 (2006). [CrossRef]

## 2. Fisheye lens model

*θ*is the angle between the optical axis and incoming ray,

*r*is the distance between the image point and the principal point,

*f*is the focal length. The most common model of fisheye lens may be equidistance projection. The schematic description of different projections for fisheye lens are illustrated in Fig. 1(a) and the difference between pinhole lens and fisheye lens is shown in Fig. 1(b) [13

**28**(8), 1335–1340 (2006). [CrossRef]

*x*̂,

*ŷ*) is point coordinate on image, (

*X*,

*Y*,

*Z*) denotes the 3D object coordinate,

*f*,

_{x}*f*are the focal lengths in

_{y}*x*,

*y*axis respectively, (

*x*

_{0},

*y*

_{0}) is the image coordinate of principal point,

*s*is the skewness factor between two axes on the image plane,

*λ*is the scale coefficient.

*R*,

*T*represent the rotation matrix and translation vector of the image. Then, the pinhole imaging process can be written as:

*x*,

*y*) (distorted) and the ideal point (

*x*̂,

*ŷ*) is expressed as:

*x*

_{0},

*y*

_{0}) represents the coordinate of principal point,

*k*

_{1},

*k*

_{2}are the radial distortion coefficients. It is obvious that iterations are needed to optimize parameters in the process of calibration. At the same time, iterative computation is still necessary in process of image correction. Distortion correction is to obtain the grey value of each pixel from original image. That is to say, we should compute the corresponding point (

*x*,

*y*) on the original image for each pixel (

*x*̂,

*ŷ*) on the corrected image to be generated. According to Eqs. (5)–(6), this process is iterative. This is the case of common pinhole camera. For fisheye lens, the deformation (Δ

*x*, Δ

*y*) is a generalization that includes the non-perspective projection and all kinds of distortions which violate the ideal imaging model. The imaging model is much more complex than above which makes the process of correction be rather complex.

## 3. Digital deformation model

*x*,

*y*axis or radial direction. The height includes not only the deformation caused by projection, radial and tangential distortions of lens, error of electric performance of CCD, but also any other errors caused by medium between object and lens. In all, the value of deformation should comprise all systemic and accidental errors which make the image and object violate the collinear restriction. The purpose of establishing DDM of a camera is to express the total deformation and correct the images taken in the similar condition. Images after correcting will meet the pinhole relationship to object.

*A*

_{3}

*Z*,

*A*

_{7}

*Z*,

*A*

_{11}

*Z*are all the same when all objects are on a same plane. In this case, after simplification, Eq. (4) can be rewritten as: The relationship between

*A*

_{1},

*A*

_{2}, ⋯

*A*

_{11}and

*B*

_{1},

*B*

_{2}, ⋯

*B*

_{8}is as follows:

*x*̂,

*ŷ*) is the mark’s ideal image coordinate and (

*X*,

*Y*) is the corresponding object coordinate,

*B*

_{1},

*B*

_{2}, ⋯

*B*

_{8}are the transform coefficients which can be accurately computed when the image coordinates and object coordinates of four marks are known. Due to the influence of all kinds of errors, the actual imaging process including fisheye lens can be described as Eq. (9) by combining Eqs. (6) and (7).

*x*

_{0},

*y*

_{0}are constant. So, we can easily obtain Eq. (10) by using other symbols. The relationship between two groups of coefficients will not detailed.

*x*, Δ

*y*) is unknown, the coefficients

*C*

_{1},

*C*

_{2}, ⋯

*C*

_{8}can’t be computed. It means this problem is really an ill-posed problem. To obtain its solution, our method artificially sets the deformations of four points to zero and computes the coefficients

*C*′

_{1},

*C*′

_{2}, ⋯

*C*′

_{8}on basis of Eq. (10). The pseudo-ideal coordinate (

*x*′,

*y*′) of each mark can then be obtained. It is clear that the pseudo-ideal image and control panel meet collinear relationship. The difference between pseudo-ideal image and actual image is:

*x*, Δ

*y*) is retrieved from DDM. According to the definition of DDM, the corrected value of each pixel is not the actual deformation, while the corrected images still meet the pinhole condition to object. That is to say, an equivalent pinhole image is produced after correction using DDM. Then, we can see DDM actually provides a flexible solution to solve the ill-posed problem in Eq. (10).

## 4. Algorithm details

- Establish two dimensional control panel composed by a certain number of marks and obtain marks’ spatial coordinates (
*X*,_{i}*Y*)_{i}*i*= 1, 2, ⋯ ,*N*. The two factors of marks’ shape and physical property are mainly considered. As those in many applications, circle marks are usually adopted due to its isotropic. Empirically, highly contrasted images may be obtained when reflective materials are used to make the marks. High accuracy measurement equipment such as theodolite helps to increase the accuracy of the marks’ spatial coordinates. - Capture the image of this control panel using the fisheye lens camera. The control panel should fill the whole field of view and the image contrast must be sufficient.
- Segment each mark and obtain image coordinates (
*x*,_{i}*y*)_{i}*i*= 1, 2, ⋯ ,*N*for all marks. The grey gravity center of each mark should be its location on image if the mark is symmetry. - Choose four marks on image near to the corners of image with approximately same distances to image center and set their deformations to zero. The purpose of setting the deformations of four marks is to compute the pseudo-ideal transformation coefficients and then to obtain a pseudo-ideal image. Theoretically, four points can be randomly chosen. However, when four marks on image with same distance to image center, their actual deformations are close. Then, the pseudo-ideal image to be established will be approximately parallel to the actual image. I.e, we hope the external parameters of the pseudo-ideal image be closely equal to those of actual image. Accordingly, when DDM is used to correct other images taken by this camera, the corrected image seems to be taken in the same position as actual image. We choose the four points on corners simply due to their deformations are largest on the whole image.
- Compute the perspective transformation coefficients
*C*′_{1},*C*′_{2}, ⋯*C*′_{8}on basis of Eq. (10) according to the least squares adjustment. - Compute the pseudo-ideal image coordinates (
*x*′,_{i}*y*′)_{i}*i*= 1, 2, ⋯,*N*for all marks using Eq. (7), in which the coefficients are*C*′_{1},*C*′_{2}, ⋯*C*′_{8}. The pseudo-ideal image strictly meets 2D perspective transformation to the control panel. - Compute difference between actual and pseudo-ideal coordinate for each mark.
- Interpolate the deformation for each integer pixel based on the bilinear rule. To perform this task, we construct a rectangle mesh model for all marks on the image, then, the nearest marks of each pixel can be found by judging which rectangle is the pixel located in. After interpolating the deformations of all pixels, DDM has been established.
- Correct images obtained in same condition using DDM.

## 5. Experiments and results

### 5.1. Fisheye images rectification using DDM

*mm*. In this experiment, we intend to establish the DDM of a camera with resolution of 3024 × 2016, lens model is F/2.8G ED with a 180-degree angle of view, focus length is 10.5

*mm*made by Nikon. Fig. 2(a) is the image of the control panel taken using this fisheye camera. Fig. 2(b) shows the DDM in radial direction, whose height is

*d*,

_{x}*d*are the deformation values in

_{y}*x*,

*y*axis respectively.

16. Z. Kang, L. Zhang, and S. Zlatanova, “An automatic mosaicking method for building facade texture mapping using a monocular close-range image sequence,” ISPRS J Photogramm. **65**(3), 282–293 (2010). [CrossRef]

### 5.2. Quantitative evaluation

*x*,

*y*) is the coordinate of point on white line,

*d*is the distance from each point on the white curve line to the blue straight line.

*M*= 0.44

_{X}*mm*,

*M*= 0.42

_{Y}*mm*,

*M*= 0.94

_{Z}*mm*. In establishing DDM, this camera’s optical axis and 2D control panel are approximately perpendicular to each other. Moreover, their intersection is close to the center of 2D control panel. In addition, distance between camera and 2D control panel is about 0.5m since this camera has a wide view and the control panel should try to fill the whole image. In measuring 3D control panel, distance from camera to object is about 2m. The size of three dimensional control panel is about 2.5

*m*× 2

*m*× 2

*m*. Then, the relative accuracies are

*X*: 1/5000,

*Y*: 1/5000,

*Z*: 1/2000.

## 6. Conclusion

## Acknowledgments

## References and links

1. | H. Bakstein and T. Pajdla, “Panoramic mosaicing with a field of view lens,” in |

2. | Y. Jia, H. Lu, and A. Xu, “Fish-eye lens camera calibration for stereo vision system,” Chinese J Comput. |

3. | J. Willneff and O. Wenisch, “The calibration of wide-angle lens cameras using perspective and non-perspective projections in the context of realtime tracking applications,” Proc. SPIE |

4. | A. Parian and A. Gruen, “Panoramic camera calibration using 3D straight lines,” presented at ISPRS Panoramic Photogrammetry Workshop, Berlin, Germany, 24–25 Feb. 2005. |

5. | S. Abraham and W. Forstner, “Fish-eye-stereo calibration and epipolar rectification,” ISPRS J Photogramm. |

6. | P. Sturm and S. Maybank, “On plane-based camera calibration: a general glgorithm, singularities, applications,” in |

7. | A. Heikkil, “Geometric camera calibration using circular control points,” IEEE T Pattern Anal. |

8. | M. Grossberg and S. Nayar, “The raxel imaging model and ray-based calibration,” Int J Comput Vision |

9. | I. Akio, Y. Kazukiyo, M. Nobuya, and K. Yuichiro, “Calibrating view angle and lens distortion of the nikon fisheye converter FC-E8,” J Forest Res. |

10. | Z. Zhang, “A flexible new technique for camera calibration,” IEEE T Pattern Anal. |

11. | D. Schneider, E. Schwalbe, and H. Maas, “Validation of geometric models for fisheye lenses,” ISPRS J Photogramm. |

12. | J. Kannala and S. Brandt, “A generic camera calibration method for fish-eye lenses,” in |

13. | J. Kannala and S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” IEEE T Pattern Anal. |

14. | D. Gennery, “Generalized camera calibration including fish-eye lenses,” Int J Comput Vision |

15. | V. Orekhov, B. Abidi, C. Broaddus, and M. Abidi, “Universal camera calibration with automatic distortion model selection,” in |

16. | Z. Kang, L. Zhang, and S. Zlatanova, “An automatic mosaicking method for building facade texture mapping using a monocular close-range image sequence,” ISPRS J Photogramm. |

**OCIS Codes**

(150.1488) Machine vision : Calibration

(080.1753) Geometric optics : Computation methods

**History**

Original Manuscript: June 1, 2012

Revised Manuscript: July 29, 2012

Manuscript Accepted: September 5, 2012

Published: September 13, 2012

**Virtual Issues**

Vol. 7, Iss. 11 *Virtual Journal for Biomedical Optics*

**Citation**

Wenguang Hou, Mingyue Ding, Nannan Qin, and Xudong Lai, "Digital deformation model for fisheye image rectification," Opt. Express **20**, 22252-22261 (2012)

http://www.opticsinfobase.org/vjbo/abstract.cfm?URI=oe-20-20-22252

Sort: Year | Journal | Reset

### References

- H. Bakstein and T. Pajdla, “Panoramic mosaicing with a field of view lens,” in Proceedings of IEEE Conference on Omnidirectional Vision (IEEE, 2002), pp. 60–67.
- Y. Jia, H. Lu, and A. Xu, “Fish-eye lens camera calibration for stereo vision system,” Chinese J Comput.23(11), 1215–1219 (2002).
- J. Willneff and O. Wenisch, “The calibration of wide-angle lens cameras using perspective and non-perspective projections in the context of realtime tracking applications,” Proc. SPIE8085, 80850S–80850S-9 (2011). [CrossRef]
- A. Parian and A. Gruen, “Panoramic camera calibration using 3D straight lines,” presented at ISPRS Panoramic Photogrammetry Workshop, Berlin, Germany, 24–25 Feb. 2005.
- S. Abraham and W. Forstner, “Fish-eye-stereo calibration and epipolar rectification,” ISPRS J Photogramm.59(5), 278–288 (2005). [CrossRef]
- P. Sturm and S. Maybank, “On plane-based camera calibration: a general glgorithm, singularities, applications,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 1999), pp. 432–437.
- A. Heikkil, “Geometric camera calibration using circular control points,” IEEE T Pattern Anal.22(10), 1066–1077 (2000). [CrossRef]
- M. Grossberg and S. Nayar, “The raxel imaging model and ray-based calibration,” Int J Comput Vision61(2), 119–137 (2005). [CrossRef]
- I. Akio, Y. Kazukiyo, M. Nobuya, and K. Yuichiro, “Calibrating view angle and lens distortion of the nikon fisheye converter FC-E8,” J Forest Res.9(3), 177–181 (2004). [CrossRef]
- Z. Zhang, “A flexible new technique for camera calibration,” IEEE T Pattern Anal.22(11), 1330–1334 (2000). [CrossRef]
- D. Schneider, E. Schwalbe, and H. Maas, “Validation of geometric models for fisheye lenses,” ISPRS J Photogramm.64(3), 259–266 (2009). [CrossRef]
- J. Kannala and S. Brandt, “A generic camera calibration method for fish-eye lenses,” in Proceedings of International Conference on Pattern Recognition (IEEE, 2004), pp. 10–13.
- J. Kannala and S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” IEEE T Pattern Anal.28(8), 1335–1340 (2006). [CrossRef]
- D. Gennery, “Generalized camera calibration including fish-eye lenses,” Int J Comput Vision68(3), 239–266 (2006). [CrossRef]
- V. Orekhov, B. Abidi, C. Broaddus, and M. Abidi, “Universal camera calibration with automatic distortion model selection,” in Proceedings of International Conference on Image Processing (IEEE, 2007), pp. 397–400.
- Z. Kang, L. Zhang, and S. Zlatanova, “An automatic mosaicking method for building facade texture mapping using a monocular close-range image sequence,” ISPRS J Photogramm.65(3), 282–293 (2010). [CrossRef]

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.