## Multi-sensor image registration based on algebraic projective invariants |

Optics Express, Vol. 21, Issue 8, pp. 9824-9838 (2013)

http://dx.doi.org/10.1364/OE.21.009824

Acrobat PDF (1607 KB)

### Abstract

A new automatic feature-based registration algorithm is presented for multi-sensor images with projective deformation. Contours are firstly extracted from both reference and sensed images as basic features in the proposed method. Since it is difficult to design a projective-invariant descriptor from the contour information directly, a new feature named Five Sequential Corners (FSC) is constructed based on the corners detected from the extracted contours. By introducing algebraic projective invariants, we design a descriptor for each FSC that is ensured to be robust against projective deformation. Further, no gray scale related information is required in calculating the descriptor, thus it is also robust against the gray scale discrepancy between the multi-sensor image pairs. Experimental results utilizing real image pairs are presented to show the merits of the proposed registration method.

© 2013 OSA

## 1. Introduction

## 2. The proposed image registration algorithm

### 2.1 Overview

### 2.2 Construction of the FSC

**: For a given contour, an FSC is defined as a group of five corners on the contour that are traversed in a sequential order. The FSC can be denoted aswhere**

*Definition 1*_{i}is the sequence number of the FSC,

_{p1i}.

**: Two FSCs**

*Definition 2*_{gref}and

**: From Definition 1, it can be seen that an FSC is determined not only by the coordinates, but also the sequence in which the five corners are traversed. For example,**

*Remark 1*_{N}corners after the step of corner detection (namely Step 2 of Fig. 1), if we randomly choose the subset of five corners and use all the possible sequences of the five corners to construct FSCs, then

_{N}will result in a larger number of FSCs and a heavy computational load of the proposed method. To avoid this, we shall propose a construction strategy to reduce the number of FSCs while guaranteeing that enough pairs of the corresponding FSCs can be constructed from the reference and sensed images. Since the contours extracted in Step 1 of the proposed method can be classified into two types, i.e. the closed contour and the open contour, a construction strategy will be designed for each type of contours respectively.

_{.}extracted contour is denoted as:where

_{j'th}extracted contour, of which the total number is

_{nj}. The corners

_{Nc}is the total number of contours extracted from the image.

**Algorithm 1: FSC construction**

**Input**: the whole set of corners

**Output**: a set of FSCs, named

**for**

**do**2.

**if**the

_{i'th}contour is

*open*and

**do**3.

**for**

**do**4. record

**end for**6.

**elseif**the

*closed*and

**do**7.

**for**

**do**8. record

**end for**10.

**end if**11.

**end for**

**: In Line 2 and Line 6 of Algorithm 1,**

*Remark 2*_{ni}denotes the number of corners on the

_{i'th}extracted contour, for

_{i=1,2,⋯,5}are the five pairs of corresponding corners. It can also be seen that

_{}

_{g¯1={p1',p2',p3',p4',p5'},g¯2={p5',p4',p3',p2',p1'}}are the two constructed FSCs for the reference image and the sensed image in Fig. 3, respectively. Moreover,

_{{g2,g¯2}}are two pairs of corresponding FSCs obtained from the image pair in Fig. 3 according to Definition 2.

_{n}corners (

_{n≥5}). If the contour is closed, such number will be

_{2n}. Suppose that the total number of the corners in C is

### 2.3 Design of descriptors for the FSC

**: In fact, increasing the dimensions of a descriptor will improve its ability to identify the pair of corresponding features in the step of feature matching. However, it will also increase the computational load. According to the experimental results on various image pairs, a ten-dimensional descriptor is selected as a good compromise.**

*Remark 3*### 2.4 Feature matching

_{des(g1)}and

_{des(g2)}on a certain direction.

**Algorithm 2: Feature Matching**

**Input**: two sets of FSCs

**Output**: a set of matched corners

_{F};1.

**for**

**do**2. find

_{g¯j}and

_{g¯k}respectively;3. calculate

**if**

**and**

_{gi}is also

_{g¯j}’s nearest neighbors from

**do**5. record

**end if**7.

**end for**

**: In Line 4 of Algorithm 2,**

*Remark 4*## 3. Analysis of accuracy rate and repetition rate of the proposed algorithm

_{σ}is a compromise between these two rates. As the repetition rate changes slowly when

## 4. Experiments

### 4.1 Robustness Validation

**Test Set 1**: Fig. 5(a) and 5(c). Only projective deformation exists in the image pair of this set;•

**Test Set 2**: Fig. 5(a) and 5(b). Only gray scale discrepancy exists, in the image pair of this set;•

**Test Set 3**: Fig. 5(a) and 5(d). Both projective deformation and gray scale discrepancy exists in the image pair of this test set.

- i) It is known that four pairs of corners can determine a projective model with eight parameters involved. It can be seen that nearly 10 corresponding pairs of corners are obtained after feature matching for the three test sets. Thus the projective transformation models corresponding to all the test sets can be estimated effectively;
- ii) The accuracy rates and repetition rates for all the three test tests are over 0.4 and 0.5, respectively. Although different degrees of projective deformation and gray scale discrepancy exist in the image pairs, similar registration results in terms of the accuracy rates and repetition rates can be observed. Based on this, satisfying robustness of the proposed registration method against projective deformation and gray scale discrepancy is demonstrated.

### 4.2 Performance Comparison with Existing Representative Algorithms

## 5. Conclusion

## Acknowledgment

## References and links

1. | B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. |

2. | J. J. Arthur, L. J. Kramer, and R. E. Bailey, “Flight test comparison between enhanced vision (FLIR) and synthetic vision systems,” (SPIE-INT SOC Optical Engineering, BELLINGHAM, 2005), pp. 25–36. |

3. | R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam, “Image change detection algorithms: a systematic survey,” IEEE Trans. Image Process. |

4. | J. P. Pluim, J. B. Maintz, and M. A. Viergever, “Mutual-information-based registration of medical images: a survey,” IEEE Trans. Med. Imaging |

5. | H. Luan, F. Qi, Z. Xue, L. Chen, and D. Shen, “Multimodality image registration by maximization of quantitative-qualitative measure of mutual information,” Pattern Recognit. |

6. | F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, “Multimodality image registration by maximization of mutual information,” IEEE Trans. Med. Imaging |

7. | A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, “Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient,” IEEE Trans. Image Process. |

8. | J. P. Heather and M. I. Smith, “Multimodal image registration with applications to image fusion,” 2005 7th International Conference on Information Fusion (FUSION), Vols 1 and 2, 372–379 (2005). |

9. | M. Irani and P. Anandan, “Robust multi-sensor image alignment,” Sixth International Conference on Computer Vision, 959–966 (1998). |

10. | Z. H. Zhang, C. H. Pan, and S. D. Ma, “An automatic method of coarse registration between multi-source satellite images,” Proceedings of the 2004 Intelligent Sensors, Sensor Networks & Information Processing Conference, 205–209 (2004). |

11. | M. A. Ali and D. A. Clausi, “Automatic registration of SAR and visible band remote sensing images,” 2002 IEEE International Geoscience and Remote Sensing Symposium. 24th Canadian Symposium on Remote Sensing. Proceedings (Cat. No.02CH37380), 1331–1333 (2002). |

12. | H. Li, B. S. Manjunath, and S. K. Mitra, “A contour-based approach to multisensor image registration,” IEEE Trans. Image Process. |

13. | S. Dawn, V. Saxena, and B. Sharma, “Remote Sensing Image Registration Techniques: A Survey,” Image and Signal Processing. Proceedings 4th International Conference, ICISP 2010, 103–112 (2010). |

14. | Z. L. Song and J. P. Zhang, “Remote Sensing Image Registration Based on Retrofitted SURF Algorithm and Trajectories Generated From Lissajous Figures,” IEEE Geosci. Remote Sens. Lett. |

15. | Z. L. Song, S. Li, and T. F. George, “Remote sensing image registration approach based on a retrofitted SIFT algorithm and Lissajous-curve trajectories,” Opt. Express |

16. | N. Alajlan, I. El Rube, M. S. Kamel, and G. Freeman, “Shape retrieval using triangle-area representation and dynamic space warping,” Pattern Recognit. |

17. | D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. |

18. | H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis. Image Underst. |

19. | J. L. Mundy and A. Zisserman, |

20. | J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. |

21. | F. Mokhtarian and R. Suomela, “Robust image corner detection through curvature scale space,” IEEE Trans. Pattern Anal |

22. | X. C. He and N. H. C. Yung, “Corner detector based on global and local curvature properties,” Opt. Eng. |

23. | M. A. Fischler and R. C. Bolles, “Random Sample Consensus - A Paradigm For Model-Fitting With Applications To Image-Analysis And Automated Cartography,” Commun. ACM |

24. | K. Mikolajczyk and C. Schmid, “Performance evaluation of local descriptors,” IEEE Trans. Pattern Anal. Mach. Intell. |

**OCIS Codes**

(100.2000) Image processing : Digital image processing

(100.5010) Image processing : Pattern recognition

(100.3008) Image processing : Image recognition, algorithms and filters

**ToC Category:**

Image Processing

**History**

Original Manuscript: January 9, 2013

Revised Manuscript: March 17, 2013

Manuscript Accepted: April 2, 2013

Published: April 12, 2013

**Citation**

Bin Li, Wei Wang, and Hao Ye, "Multi-sensor image registration based on algebraic projective invariants," Opt. Express **21**, 9824-9838 (2013)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-21-8-9824

Sort: Year | Journal | Reset

### References

- B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput.21(11), 977–1000 (2003).
- J. J. Arthur, L. J. Kramer, and R. E. Bailey, “Flight test comparison between enhanced vision (FLIR) and synthetic vision systems,” (SPIE-INT SOC Optical Engineering, BELLINGHAM, 2005), pp. 25–36.
- R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam, “Image change detection algorithms: a systematic survey,” IEEE Trans. Image Process.14(3), 294–307 (2005).
- J. P. Pluim, J. B. Maintz, and M. A. Viergever, “Mutual-information-based registration of medical images: a survey,” IEEE Trans. Med. Imaging22(8), 986–1004 (2003).
- H. Luan, F. Qi, Z. Xue, L. Chen, and D. Shen, “Multimodality image registration by maximization of quantitative-qualitative measure of mutual information,” Pattern Recognit.41(1), 285–298 (2008).
- F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, “Multimodality image registration by maximization of mutual information,” IEEE Trans. Med. Imaging16(2), 187–198 (1997).
- A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, “Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient,” IEEE Trans. Image Process.12(12), 1495–1511 (2003).
- J. P. Heather and M. I. Smith, “Multimodal image registration with applications to image fusion,” 2005 7th International Conference on Information Fusion (FUSION), Vols 1 and 2, 372–379 (2005).
- M. Irani and P. Anandan, “Robust multi-sensor image alignment,” Sixth International Conference on Computer Vision, 959–966 (1998).
- Z. H. Zhang, C. H. Pan, and S. D. Ma, “An automatic method of coarse registration between multi-source satellite images,” Proceedings of the 2004 Intelligent Sensors, Sensor Networks & Information Processing Conference, 205–209 (2004).
- M. A. Ali and D. A. Clausi, “Automatic registration of SAR and visible band remote sensing images,” 2002 IEEE International Geoscience and Remote Sensing Symposium. 24th Canadian Symposium on Remote Sensing. Proceedings (Cat. No.02CH37380), 1331–1333 (2002).
- H. Li, B. S. Manjunath, and S. K. Mitra, “A contour-based approach to multisensor image registration,” IEEE Trans. Image Process.4(3), 320–334 (1995).
- S. Dawn, V. Saxena, and B. Sharma, “Remote Sensing Image Registration Techniques: A Survey,” Image and Signal Processing. Proceedings 4th International Conference, ICISP 2010, 103–112 (2010).
- Z. L. Song and J. P. Zhang, “Remote Sensing Image Registration Based on Retrofitted SURF Algorithm and Trajectories Generated From Lissajous Figures,” IEEE Geosci. Remote Sens. Lett.7(3), 491–495 (2010).
- Z. L. Song, S. Li, and T. F. George, “Remote sensing image registration approach based on a retrofitted SIFT algorithm and Lissajous-curve trajectories,” Opt. Express18(2), 513–522 (2010).
- N. Alajlan, I. El Rube, M. S. Kamel, and G. Freeman, “Shape retrieval using triangle-area representation and dynamic space warping,” Pattern Recognit.40(7), 1911–1920 (2007).
- D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis.60(2), 91–110 (2004).
- H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis. Image Underst.110(3), 346–359 (2008).
- J. L. Mundy and A. Zisserman, Geometric Invariance in Computer Vision (the MIT Press, 1992).
- J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell.8(6), 679–698 (1986).
- F. Mokhtarian and R. Suomela, “Robust image corner detection through curvature scale space,” IEEE Trans. Pattern Anal20(12), 1376–1381 (1998).
- X. C. He and N. H. C. Yung, “Corner detector based on global and local curvature properties,” Opt. Eng.47, 057008 (2008).
- M. A. Fischler and R. C. Bolles, “Random Sample Consensus - A Paradigm For Model-Fitting With Applications To Image-Analysis And Automated Cartography,” Commun. ACM24, 381–395 (1981).
- K. Mikolajczyk and C. Schmid, “Performance evaluation of local descriptors,” IEEE Trans. Pattern Anal. Mach. Intell.27(10), 1615–1630 (2005).

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.