## Computational optical distortion correction using a radial basis function-based mapping method |

Optics Express, Vol. 20, Issue 14, pp. 14906-14920 (2012)

http://dx.doi.org/10.1364/OE.20.014906

Acrobat PDF (2941 KB)

### Abstract

A distortion mapping and computational image unwarping method based on a network interpolation that uses radial basis functions is presented. The method is applied to correct distortion in an off-axis head-worn display (HWD) presenting up to 23% highly asymmetric distortion over a 27°x21° field of view. A 10^{−5} mm absolute error of the mapping function over the field of view was achieved. The unwarping efficacy was assessed using the image-rendering feature of optical design software. Correlation coefficients between unwarped images seen through the HWD and the original images, as well as edge superimposition results, are presented. In an experiment, images are prewarped using radial basis functions for a recently built, off-axis HWD with a 20° diagonal field of view in a 4:3 ratio. Real-time video is generated by a custom application with 2 ms added latency and is demonstrated.

© 2012 OSA

## 1. Introduction

2. R. Y. Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE J. Robot. Autom. **3**(4), 323–344 (1987). [CrossRef]

6. J. M. Sasian, “Image plane tilt in optical systems,” Opt. Eng. **31**(3), 527 (1992). [CrossRef]

*correspondence*, of which a set is illustrated in Fig. 1 . The search for a distortion model of a given system generally aims to model a set of correspondences over a desired field of view (FOV).

## 2. Overview of distortion mapping methods

2. R. Y. Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE J. Robot. Autom. **3**(4), 323–344 (1987). [CrossRef]

8. J. Weng, P. Cohen, and M. Herniou, “Camera calibration with distortion models and accuracy evaluations,” IEEE Trans. Pattern Anal. Mach. Intell. **14**(10), 965–980 (1992). [CrossRef]

12. J. P. Rolland, “Wide-angle, off-axis, see-through head-mounted display,” Opt. Eng. **39**(7), 1760–1767 (2000). [CrossRef]

13. K. Fuerschbach, J. P. Rolland, and K. P. Thompson, “A new family of optical systems employing φ-polynomial surfaces,” Opt. Express **19**(22), 21919–21928 (2011). [CrossRef] [PubMed]

8. J. Weng, P. Cohen, and M. Herniou, “Camera calibration with distortion models and accuracy evaluations,” IEEE Trans. Pattern Anal. Mach. Intell. **14**(10), 965–980 (1992). [CrossRef]

16. J. Wang, F. Shi, J. Zhang, and Y. Liu, “A new calibration model of camera lens distortion,” Pattern Recognit. **41**(2), 607–615 (2008). [CrossRef]

22. C. C. Leung, P. C. K. Kwok, K. Y. Zee, and F. H. Y. Chan, “B-spline interpolation for bend intra-oral radiographs,” Comput. Biol. Med. **37**(11), 1565–1571 (2007). [CrossRef] [PubMed]

23. P. Cerveri, S. Ferrari, and N. A. Borghese, “Calibration of TV cameras through RBF networks,” Proc. SPIE **3165**, 312–318 (1997). [CrossRef]

23. P. Cerveri, S. Ferrari, and N. A. Borghese, “Calibration of TV cameras through RBF networks,” Proc. SPIE **3165**, 312–318 (1997). [CrossRef]

25. D. N. Fogel, “Image Rectification with Radial Basis Functions: Applications to RS/GIS Data Integration,” in *Proceedings of the Third International Conference on Integrating GIS and Environmental Modeling,*(Sante Fe, 1996).http://www.ncgia.ucsb.edu/conf/SANTA_FE_CD-ROM/sf_papers/fogel_david/santafe.html

22. C. C. Leung, P. C. K. Kwok, K. Y. Zee, and F. H. Y. Chan, “B-spline interpolation for bend intra-oral radiographs,” Comput. Biol. Med. **37**(11), 1565–1571 (2007). [CrossRef] [PubMed]

## 3. Principle of the RBF-based distortion mapping method and its application to a computational unwarping task

^{®}) model of an off-axis eyeglass display [12

12. J. P. Rolland, “Wide-angle, off-axis, see-through head-mounted display,” Opt. Eng. **39**(7), 1760–1767 (2000). [CrossRef]

### 3.1Optical system distortion mapping

*n*equal21

^{2 }points in visual space (evenly positioned on a rectangular grid satisfying the 25% oversized FOV requirement), distorted point coordinates were computed by the Distortion Grid option in CODE V to give the correspondences needed for the determination of the mapping parameters. Normalization to the horizontal FOV maximum value was performed for computation convenience. Using the local RBF-based mapping method with

*N*multiquadric basis functions [21], each correspondence between a point on the regular grid

*M’*and its distorted counterpart

_{j}(x_{j}^{’},y_{j}^{’})*M*is written as follows: where,

_{j}(x_{j},y_{j})*R*represents the

_{i }*i*basis function centered at (

^{th}*x*), with basis function weights

_{basis_center_i}, y_{basis_center_i}*α*;

_{xy,i}*p*is a polynomial of degree

_{m}(x_{j}, y_{j})*m*that assures polynomial precision of degree

*m; j*is an integer that runs from 1 to

*N*; and

*χ*is a scaling factor to be chosen heuristically based on the specific application. Figure 3 shows a graphical representation of the algorithm.

*n = N*).The basis centers were placed at the original grid points location. The characteristic radii (

*r*)

_{i}_{i = 1..n}of all basis functions were taken to be constant and equal to the minimum spacing between 2 original points on the grid. The multiquadric exponent,

*µ*, was set equal to −2, as in [21]. The 2N-equations linear system formed by Eqs. (1) and (2) repeated for the

*N*correspondences was solved by least squares minimization to find the basis functions weights (

*α*)

_{xi}_{i = 1..n}and (

*α*)

_{yi}_{i = 1..n}. Once learned, those weights completely define the RBF-based mapping of the HWD distortion. The RBF distortion mapping is compared to the distortion grid output in Fig. 4 yielding accuracies higher than 3 μm across a high density of test points within the effective FOV, which we found to be more than 100 times better than that obtained with the next best model discussed by Weng [8

8. J. Weng, P. Cohen, and M. Herniou, “Camera calibration with distortion models and accuracy evaluations,” IEEE Trans. Pattern Anal. Mach. Intell. **14**(10), 965–980 (1992). [CrossRef]

### 3.2 Image Prewarping Using RBF-based distortion mapping

*N*) (rows by columns) to be displayed with the HWD from [12

_{y}x N_{x}12. J. P. Rolland, “Wide-angle, off-axis, see-through head-mounted display,” Opt. Eng. **39**(7), 1760–1767 (2000). [CrossRef]

*N*), from the (

_{y_storage}x N_{x_storage}*N*) input image pixels and forming the final prewarped image from it.

_{y}x N_{x}*N*) for method 1, and equal to (

_{y}x N_{x}*N*) for method 2 (by definition of this method). The precise number of pixels in the storage matrix for method 1 depends on the specific distortion mapping. In this case, the (

_{y}x N_{x}*N*) input pixels are not moved to (

_{y}x N_{x}*N*) distinct locations due to the existence of pixel overlaps. This occurs when initially adjacent pixels are moved closer than 1 pixel apart by the distortion function, and thus, have identical distorted coordinates. Similar to these pixel overlaps, holes occur between pixels that are moved further than 1 pixel apart. These holes are corrected with a filling algorithm. After the hole-filling correction step, the storage matrix contains the desired hole free

_{y}x N_{x}*N*image, which has been prewarped following the desired distortion mapping. Figure 5 shows, as an example, an input grayscale image (a grid) with SVGA resolution (i.e.

_{y_storage}x N_{x_storage}*N*equal 600 x 800) and its corresponding prewarped image for both methods. Uniform gray pixels were added around the effective distorted image during the prewarping process for further assessment convenience of the unwarped images. Smoother prewarped images in the case of method 2 may be observed in Fig. 5.

_{y}x N_{x }### 3.3 Computational Distortion Unwrapping Simulation

*direct*HWD model (i.e. microdisplay in object space) that computes the appearance of a 2D input object imaged through an optical system. The input object consisted of an (

*N*) equal 600x800 prewarped image filling the display FOV (18.5 mm-diagonal) and translated to account for the reverse system (the visual space for the reverse system is the object space) chief ray offset in the prewarped image. The resulting rendered images from methods 1 and 2 are shown in Fig. 6 .

_{y}x N_{x}*A*) were quantitatively compared to the original images (

*B*) using the correlation coefficient (CC) given byWhere

*N*is the number of pixels,

*A*and

_{i}*B*are the pixel values of the i

_{i }^{th}pixel, and

*µ*are the mean values of

_{A,B}*A*and

*B*. A value of CC equal 1 denotes perfect correlation between two images. The unwarped images resulting from the 2D IMS option were cropped along the uniform gray border enclosing the original image pattern and corrected from the system vignetting effect using the rendered image of a uniform white rectangle. The original image was downsampled in MATLAB to the cropped unwarped image resolution using the

*imresize*function.

## 4. Prewarping an image for an off-axis HWD

27. J. P. McGuire Jr., “Next-generation head-mounted display,” Proc. SPIE **7618**, 761804, 761804-8 (2010). [CrossRef]

*Distortion Grid*function in the optical design software used in the previous section. Note that the

*Distortion Grid*function only generates a small sample of correspondences from discrete points in the FOV. From the entire set of correspondences established to get the gold standard, we select a small subset of them and aim to predict the entire set with the RBF mapping function. As previously mentioned, there are inherent edge effects that occur as a result of RBF interpolation, so one must cleverly select the subset of correspondences.

*χ*in Eq. (3) is chosen heuristically per application. Simulations showed that

*χ*equal 24 yields the most accurate results, as quantified in Fig. 11(b) .Results also show that the process is not sensitive to the exact value of

*χ*around 24.

28. B. Fornberg, E. Larsson, and N. Flyer, “Stable computations with Gaussian radial basis functions,” SIAM J. Sci. Comput. **33**(2), 869–892 (2011). [CrossRef]

## 5. Real-time distortion mapping for use with HWDs

## 6. Conclusion and discussion

## Acknowledgments

^{TM}

_{.}

## References and links

1. | W. Faig, “Calibration of close-range photogrammetric systems: Mathematical formulation,” Photogramm. Eng. Remote Sensing |

2. | R. Y. Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE J. Robot. Autom. |

3. | W. Robinett and J.P. Rolland, “A computational model for the stereoscopic optics of a head mounted-display,” Presence (Camb. Mass.) |

4. | P. Cerveri, C. Forlani, A. Pedotti, and G. Ferrigno, “Hierarchical radial basis function networks and local polynomial un-warping for X-ray image intensifier distortion correction: a comparison with global techniques,” Med. Biol. Eng. Comput. |

5. | W. T. Welford, |

6. | J. M. Sasian, “Image plane tilt in optical systems,” Opt. Eng. |

7. | B. A. Watson and L. F. Hodges, “Using texture maps to correct for optical distortion in head-mounted-displays,” in |

8. | J. Weng, P. Cohen, and M. Herniou, “Camera calibration with distortion models and accuracy evaluations,” IEEE Trans. Pattern Anal. Mach. Intell. |

9. | A. Basu, S. Licardie, A. Basu, and S. Licardie, “Alternative models for fish-eye lenses,” Pattern Recognit. Lett. |

10. | F. Devernay, O. Faugeras, F. Devernay, and O. Faugeras, “Straight lines have to be straight,” Mach. Vis. Appl. |

11. | D. Clause and A. W. Fitzgibbon, “A rational function lens distortion model for general cameras,” in |

12. | J. P. Rolland, “Wide-angle, off-axis, see-through head-mounted display,” Opt. Eng. |

13. | K. Fuerschbach, J. P. Rolland, and K. P. Thompson, “A new family of optical systems employing φ-polynomial surfaces,” Opt. Express |

14. | C. Slama, |

15. | J. Heikkilä and O. Silven, “A four-step camera calibration procedure with implicit image correction,” in |

16. | J. Wang, F. Shi, J. Zhang, and Y. Liu, “A new calibration model of camera lens distortion,” Pattern Recognit. |

17. | A. E. Conrady, “Decentered lens systems,” Monthly Notices of The Royal Astr Society |

18. | D. C. Brown, “Decentering distortion of lenses,” Photogramm. Eng. Remote Sensing |

19. | R. I. Hartley and T. Saxena, “The cubic rational polynomial camera model,” |

20. | G. Q. Wei and S. D. Ma, “A Complete two-plane camera calibration method and experimental comparisons,” in |

21. | D. Ruprecht, H. Muller, D. Ruprecht, and H. Muller, “Image warping with scattered data interpolation,” IEEE Comput. Graph. Appl. |

22. | C. C. Leung, P. C. K. Kwok, K. Y. Zee, and F. H. Y. Chan, “B-spline interpolation for bend intra-oral radiographs,” Comput. Biol. Med. |

23. | P. Cerveri, S. Ferrari, and N. A. Borghese, “Calibration of TV cameras through RBF networks,” Proc. SPIE |

24. | G. E. Martin, |

25. | D. N. Fogel, “Image Rectification with Radial Basis Functions: Applications to RS/GIS Data Integration,” in |

26. | X. Zhu, R. M. Rangayyan, and A. L. Ellis, |

27. | J. P. McGuire Jr., “Next-generation head-mounted display,” Proc. SPIE |

28. | B. Fornberg, E. Larsson, and N. Flyer, “Stable computations with Gaussian radial basis functions,” SIAM J. Sci. Comput. |

**OCIS Codes**

(100.0100) Image processing : Image processing

(120.2820) Instrumentation, measurement, and metrology : Heads-up displays

**ToC Category:**

Image Processing

**History**

Original Manuscript: April 17, 2012

Revised Manuscript: June 4, 2012

Manuscript Accepted: June 5, 2012

Published: June 19, 2012

**Citation**

Aaron Bauer, Sophie Vo, Keith Parkins, Francisco Rodriguez, Ozan Cakmakci, and Jannick P. Rolland, "Computational optical distortion correction using a radial basis function-based mapping method," Opt. Express **20**, 14906-14920 (2012)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-20-14-14906

Sort: Year | Journal | Reset

### References

- W. Faig, “Calibration of close-range photogrammetric systems: Mathematical formulation,” Photogramm. Eng. Remote Sensing41, 1479–1486 (1975).
- R. Y. Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE J. Robot. Autom.3(4), 323–344 (1987). [CrossRef]
- W. Robinett and J.P. Rolland, “A computational model for the stereoscopic optics of a head mounted-display,” Presence (Camb. Mass.)1, 45–62 (1992).
- P. Cerveri, C. Forlani, A. Pedotti, and G. Ferrigno, “Hierarchical radial basis function networks and local polynomial un-warping for X-ray image intensifier distortion correction: a comparison with global techniques,” Med. Biol. Eng. Comput.41(2), 151–163 (2003).
- W. T. Welford, Aberrations of Optical Systems (CRC Press, 1986).
- J. M. Sasian, “Image plane tilt in optical systems,” Opt. Eng.31(3), 527 (1992). [CrossRef]
- B. A. Watson and L. F. Hodges, “Using texture maps to correct for optical distortion in head-mounted-displays,” in Proceedings of the Virtual Reality Annual Int. Symposium, 172–178 (1995).
- J. Weng, P. Cohen, and M. Herniou, “Camera calibration with distortion models and accuracy evaluations,” IEEE Trans. Pattern Anal. Mach. Intell.14(10), 965–980 (1992). [CrossRef]
- A. Basu, S. Licardie, A. Basu, and S. Licardie, “Alternative models for fish-eye lenses,” Pattern Recognit. Lett.16(4), 433–441 (1995).
- F. Devernay, O. Faugeras, F. Devernay, and O. Faugeras, “Straight lines have to be straight,” Mach. Vis. Appl.13(1), 14–24 (2001).
- D. Clause and A. W. Fitzgibbon, “A rational function lens distortion model for general cameras,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 213–219 (2005).
- J. P. Rolland, “Wide-angle, off-axis, see-through head-mounted display,” Opt. Eng.39(7), 1760–1767 (2000). [CrossRef]
- K. Fuerschbach, J. P. Rolland, and K. P. Thompson, “A new family of optical systems employing φ-polynomial surfaces,” Opt. Express19(22), 21919–21928 (2011). [CrossRef] [PubMed]
- C. Slama, Manual of Photogrammetry, (Amer. Soc. of Photogrammetry, 1980).
- J. Heikkilä and O. Silven, “A four-step camera calibration procedure with implicit image correction,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1106–1112 (1997).
- J. Wang, F. Shi, J. Zhang, and Y. Liu, “A new calibration model of camera lens distortion,” Pattern Recognit.41(2), 607–615 (2008). [CrossRef]
- A. E. Conrady, “Decentered lens systems,” Monthly Notices of The Royal Astr Society39, 384–390 (1919).
- D. C. Brown, “Decentering distortion of lenses,” Photogramm. Eng. Remote Sensing32, 444–462 (1966).
- R. I. Hartley and T. Saxena, “The cubic rational polynomial camera model,”in Proceedings of Defense Advanced Research Projects Agency Image Understanding Workshop, 649–653 (1997).
- G. Q. Wei and S. D. Ma, “A Complete two-plane camera calibration method and experimental comparisons,” in Proceedings of IEEE 4th International Conference on Computer Vision, 439–446 (1993).
- D. Ruprecht, H. Muller, D. Ruprecht, and H. Muller, “Image warping with scattered data interpolation,” IEEE Comput. Graph. Appl.15(2), 37–43 (1995).
- C. C. Leung, P. C. K. Kwok, K. Y. Zee, and F. H. Y. Chan, “B-spline interpolation for bend intra-oral radiographs,” Comput. Biol. Med.37(11), 1565–1571 (2007). [CrossRef] [PubMed]
- P. Cerveri, S. Ferrari, and N. A. Borghese, “Calibration of TV cameras through RBF networks,” Proc. SPIE3165, 312–318 (1997). [CrossRef]
- G. E. Martin, Transformation Geometry: An Introduction to Symmetry (Springer, 1982).
- D. N. Fogel, “Image Rectification with Radial Basis Functions: Applications to RS/GIS Data Integration,” in Proceedings of the Third International Conference on Integrating GIS and Environmental Modeling,(Sante Fe, 1996). http://www.ncgia.ucsb.edu/conf/SANTA_FE_CD-ROM/sf_papers/fogel_david/santafe.html
- X. Zhu, R. M. Rangayyan, and A. L. Ellis, Digital Image Processing for Ophthalmology: Detection of the Optic Nerve Head (Morgan & Claypool, 2011), Chap 3.
- J. P. McGuire., “Next-generation head-mounted display,” Proc. SPIE7618, 761804, 761804-8 (2010). [CrossRef]
- B. Fornberg, E. Larsson, and N. Flyer, “Stable computations with Gaussian radial basis functions,” SIAM J. Sci. Comput.33(2), 869–892 (2011). [CrossRef]

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.