OSA's Digital Library

Optics Express

Optics Express

  • Editor: Andrew M. Weiner
  • Vol. 21, Iss. 8 — Apr. 22, 2013
  • pp: 9824–9838
« Show journal navigation

Multi-sensor image registration based on algebraic projective invariants

Bin Li, Wei Wang, and Hao Ye  »View Author Affiliations


Optics Express, Vol. 21, Issue 8, pp. 9824-9838 (2013)
http://dx.doi.org/10.1364/OE.21.009824


View Full Text Article

Acrobat PDF (1607 KB)





Browse Journals / Lookup Meetings

Browse by Journal and Year


   


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools

Share
Citations

Abstract

A new automatic feature-based registration algorithm is presented for multi-sensor images with projective deformation. Contours are firstly extracted from both reference and sensed images as basic features in the proposed method. Since it is difficult to design a projective-invariant descriptor from the contour information directly, a new feature named Five Sequential Corners (FSC) is constructed based on the corners detected from the extracted contours. By introducing algebraic projective invariants, we design a descriptor for each FSC that is ensured to be robust against projective deformation. Further, no gray scale related information is required in calculating the descriptor, thus it is also robust against the gray scale discrepancy between the multi-sensor image pairs. Experimental results utilizing real image pairs are presented to show the merits of the proposed registration method.

© 2013 OSA

1. Introduction

Image registration is a process to align two images which were acquired in the same scene, while from different sensors at different times [1

1. B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).

]. The two images are often referred to as the reference image and the sensed image, respectively [1

1. B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).

]. The purpose of registration is to estimate the coordinate transformation model, according to which the sensed image can be well registered to the reference image. Since the images captured through different sensors will involve distinct characteristics, fused images delivering more information than the original images may be expected after performing the image registration. This constitutes the main reason that automatic registration of multi-sensor images has become an indispensable step in many practical applications including change detection, medical diagnosis, and synthetic vision [2

2. J. J. Arthur, L. J. Kramer, and R. E. Bailey, “Flight test comparison between enhanced vision (FLIR) and synthetic vision systems,” (SPIE-INT SOC Optical Engineering, BELLINGHAM, 2005), pp. 25–36.

4

4. J. P. Pluim, J. B. Maintz, and M. A. Viergever, “Mutual-information-based registration of medical images: a survey,” IEEE Trans. Med. Imaging 22(8), 986–1004 (2003).

].

However, compared with fruitful results reported on the image registration for single-sensor images, the effective registration algorithms for multi-sensor images are still limited mainly due to the following two challenges. The first one is the gray scale discrepancy between corresponding pixels of the reference and sensed images. As two different sensors have different operation wavelengths, there is no strict one-to-one correspondence between the intensities of the two sensed images. The second challenge is geometric deformation between the image pairs. Note that the translation, scaling, rotation, affine and projective transformations are some representative deformation models. Clearly, it will be rather difficult to design multi-sensor registration algorithms if the deformation models are complex.

The existing multi-sensor image registration schemes can roughly be classified into two categories: the area-based and the feature-based methods [1

1. B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).

]. In area-based registration methods, similarity measures, such as Mutual information (MI) based [5

5. H. Luan, F. Qi, Z. Xue, L. Chen, and D. Shen, “Multimodality image registration by maximization of quantitative-qualitative measure of mutual information,” Pattern Recognit. 41(1), 285–298 (2008).

7

7. A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, “Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient,” IEEE Trans. Image Process. 12(12), 1495–1511 (2003).

] and correlation based [8

8. J. P. Heather and M. I. Smith, “Multimodal image registration with applications to image fusion,” 2005 7th International Conference on Information Fusion (FUSION), Vols 1 and 2, 372–379 (2005).

,9

9. M. Irani and P. Anandan, “Robust multi-sensor image alignment,” Sixth International Conference on Computer Vision, 959–966 (1998).

] similarity measures, are usually introduced. By searching the parameter space, the estimated transformation model parameters obtained by maximizing the similarity measures are often accepted as the final results. Since the parameter searching space will expand exponentially with the deformation complexity increases, this type of methods may cost extremely long time to find satisfactory results in the cases of affine and projective deformations. In contrast to these, salient features are extracted from reference and sensed images separately in feature-based registration methods. By matching the extracted features, a set of corresponding points can be obtained, which will then be utilized to estimate the parameters in the transformation models directly. Since parameter searching is not required in the feature-based methods, feature-based methods have advantages in computational efficiency over the area-based methods, especially when the images with complex deformations are dealt with.

It is worth mentioning that the contour features are often adopted in the feature-based multi-sensor image registration for its good robustness against gray scale discrepancy [10

10. Z. H. Zhang, C. H. Pan, and S. D. Ma, “An automatic method of coarse registration between multi-source satellite images,” Proceedings of the 2004 Intelligent Sensors, Sensor Networks & Information Processing Conference, 205–209 (2004).

12

12. H. Li, B. S. Manjunath, and S. K. Mitra, “A contour-based approach to multisensor image registration,” IEEE Trans. Image Process. 4(3), 320–334 (1995).

]. However, the contour-based methods proposed in [10

10. Z. H. Zhang, C. H. Pan, and S. D. Ma, “An automatic method of coarse registration between multi-source satellite images,” Proceedings of the 2004 Intelligent Sensors, Sensor Networks & Information Processing Conference, 205–209 (2004).

12

12. H. Li, B. S. Manjunath, and S. K. Mitra, “A contour-based approach to multisensor image registration,” IEEE Trans. Image Process. 4(3), 320–334 (1995).

] are not applicable if severe affine or projective deformations exist in the image pairs. This is because that these complex geometric deformations may significantly destroy the structure information of the contours, which will affect the accuracies of the registration results. In addition to the contour features, some registration methods based on other features have been proposed for the multi-sensor images recently. Wong et.al [13

13. S. Dawn, V. Saxena, and B. Sharma, “Remote Sensing Image Registration Techniques: A Survey,” Image and Signal Processing. Proceedings 4th International Conference, ICISP 2010, 103–112 (2010).

] adopted the local maximum moment of the phase congruency as an image feature, which is shown to be invariant to the gray scale discrepancy and affine deformation. Song et al. [14

14. Z. L. Song and J. P. Zhang, “Remote Sensing Image Registration Based on Retrofitted SURF Algorithm and Trajectories Generated From Lissajous Figures,” IEEE Geosci. Remote Sens. Lett. 7(3), 491–495 (2010).

,15

15. Z. L. Song, S. Li, and T. F. George, “Remote sensing image registration approach based on a retrofitted SIFT algorithm and Lissajous-curve trajectories,” Opt. Express 18(2), 513–522 (2010).

] introduced the affine-invariance property of Triangle Area Representation (TAR) [16

16. N. Alajlan, I. El Rube, M. S. Kamel, and G. Freeman, “Shape retrieval using triangle-area representation and dynamic space warping,” Pattern Recognit. 40(7), 1911–1920 (2007).

] into two representative feature-based single-sensor image registration methods, i.e. the Scale-Invariant Feature Transform -SIFT [17

17. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).

] and the Speeded Up Robust Features -SURF [18

18. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).

], to deal with multi-sensor images with affine deformation. Nevertheless, the aforementioned methods cannot handle the image pairs with projective deformations. To the best of the authors’ knowledge, very few results in feature-based image registration are available to deal with multi-sensor images in the presence of projective deformations.

In this paper, a feature-based registration method will be proposed for the multi-sensor images with projective geometric deformation. Considering the fact that the contour-based features are robust to gray scale discrepancy of the images and the corners on a contour will not vanish after arbitrary projective transformations, we firstly select the corners on the extracted contours as basic feature points. It should be emphasized that if the framework of traditional feature-based methods is adopted, i.e. to design descriptor for the feature points directly [1

1. B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).

], it is difficult to ensure that the designed descriptor can keep invariant when the projective deformation and gray scale discrepancy exist in the multi-sensor image pair simultaneously. To overcome this difficulty, instead of describing the detected corners directly, an intermediate step of constructing a new feature, called Five Sequential Corners (FSC), is introduced in this paper. Then a descriptor is designed for each FSC by employing the algebraic projective invariants [19

19. J. L. Mundy and A. Zisserman, Geometric Invariance in Computer Vision (the MIT Press, 1992).

]. Based on these, the set of corresponding FSC pairs can be obtained and the set of corresponding corner pairs will be determined in further. It will be shown that our proposed feature-based registration scheme has good robustness against both projective deformation and gray scale discrepancy between the image pairs.

2. The proposed image registration algorithm

2.1 Overview

The framework of the proposed registration algorithm is shown in Fig. 1
Fig. 1 The framework of the proposed method.
. The brief introductions to all the steps involved in Fig. 1 are given as follows.

Step 1: Detect edges by employing Canny [20

20. J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986).

] edge detector to obtain a binary edge map from each input image, and extract contours by the Curvature Scale Space (CSS) algorithm introduced in [21

21. F. Mokhtarian and R. Suomela, “Robust image corner detection through curvature scale space,” IEEE Trans. Pattern Anal 20(12), 1376–1381 (1998).

].

Step 2: Detect the corners from each extracted contour by adopting the contour-based corner detector proposed by He et al. [22

22. X. C. He and N. H. C. Yung, “Corner detector based on global and local curvature properties,” Opt. Eng. 47, 057008 (2008).

].

Step 3: Construct the newly introduced feature FSCs on the basis of the corners. Each FSC is determined not only by the coordinates of the five corners chosen, but also by the sequence that the five corners are traversed.

Step 4: Describe each FSC using a ten-dimensional vector, in which each element is an algebraic projective invariant.

Step 5: Match the FSCs by comparing their descriptors, based on which the correspondences of the corners can be obtained.

Step 6: Estimate the parameters of transformation model using the RANSAC algorithm [23

23. M. A. Fischler and R. C. Bolles, “Random Sample Consensus - A Paradigm For Model-Fitting With Applications To Image-Analysis And Automated Cartography,” Commun. ACM 24, 381–395 (1981).

] based on the set of corresponding corners.

In the frameworks of the most existing feature-based methods, which generally consist of feature detection, feature description, feature matching and transformation estimation [1

1. B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).

], the descriptors are designed directly for the feature points. Different from this traditional framework, the feature points (i.e. the corners detected in Step 2) are used to construct FSCs firstly in our proposed algorithm (see Step 3). A descriptor will then be designed for each FSC in Step 4 by introducing the algebraic projective invariants, which will be shown to be robust against both the projective deformation and gray scale discrepancy between the image pairs.

It is discussed in Step 3 that a FSC cannot be determined solely by the coordinates of the five chosen corners. In fact, if an FSC can be determined once the five corners are specified, the algebraic projective invariants can still be introduced to design the feature descriptor for the FSC. However, only the correspondences of the FSCs can be obtained by comparing their descriptors rather than the correspondences of the corners. Obviously, the collection of the former correspondences is not sufficient to estimate the final transformation model. To address this problem, the sequence information of the five corners selected is also included in defining an FSC as indicated in Step 3. By doing this, we can obtain the matched corners according to the matched FSCs finally. More detailed illustration about this issue will be given in Sec. 2.2.

2.2 Construction of the FSC

In this section, we will give a detailed introduction to the strategy of constructing the new feature FSC. As shown in Fig.1, a set of FSCs will be constructed from the reference and sensed images respectively in Step 3 of the proposed method. Note that the two constructed sets of the FSCs should contain sufficiently large number of pairs of corresponding elements, so that we can obtain enough matched points which will be used to estimate the final parameters of the transformation model in later steps of the proposed method.

In this paper, the definitions of an FSC and a pair corresponding FSCs are given as follows.

Definition 1: For a given contour, an FSC is defined as a group of five corners on the contour that are traversed in a sequential order. The FSC can be denoted as
gi={p1i,p2i,p3i,p4i,p5i}
(1)
where i is the sequence number of the FSC, p1i=(x1i,y1i), p2i=(x2i,y2i), …, p5i=(x5i,y5i) are the five corners with (xki,yki) representing the coordinate of the corner p1i.

Definition 2: Two FSCs gref and gsen (constructed from the reference and sensed image, respectively) are regarded as a pair of corresponding FSCs, if and only if the following conditions are satisfied:(1): gref and gsen are constructed from five pairs of the corresponding corners.(2): The corresponding corners are recorded in the same sequence in gref and gsen, respectively.

Remark 1: From Definition 1, it can be seen that an FSC is determined not only by the coordinates, but also the sequence in which the five corners are traversed. For example, g={pa,pb,pc,pd,pe} and g˜={pb,pa,pc,pd,pe} represent two different FSCs although they are defined from the same five corners. As will be discussed in Sec. 2.4, such definition of the FSC can guarantee that five pairs of corresponding corners can be determined from one pair of corresponding FSCs.

Based on the above definitions, the FSCs are constructed by the following procedure. Firstly, a subset of five corners is selected from the set of detected corners. Secondly, the FSCs are constructed from the five corners in the chosen subset by specifying different recorded sequences of them. Therefore, the number of the FSCs depends on the number of chosen subsets as well as the number of FSCs that a subset can construct. For example, suppose we get N corners after the step of corner detection (namely Step 2 of Fig. 1), if we randomly choose the subset of five corners and use all the possible sequences of the five corners to construct FSCs, then (5N)=N!/[5!(N5)!] subsets and 5! FSCs from each subset will be obtained. Thus N!/(N5)! FSCs will be constructed in total. Obviously, a larger N will result in a larger number of FSCs and a heavy computational load of the proposed method. To avoid this, we shall propose a construction strategy to reduce the number of FSCs while guaranteeing that enough pairs of the corresponding FSCs can be constructed from the reference and sensed images. Since the contours extracted in Step 1 of the proposed method can be classified into two types, i.e. the closed contour and the open contour, a construction strategy will be designed for each type of contours respectively.

In this paper, the set of corners detected from the .. extracted contour is denoted as:
cj={p1j,p2j,...,pnjj}
(2)
where {pkj}k=1,2,,njare the corners on the j'th extracted contour, of which the total number is nj. The corners {pkj}k=1,2,,nj are recorded in the following way. For the case of open contour, either terminal corner is recorded first, from which the rest corners are recorded in a sequential order along the contour. For the case of closed contour, the first corner to be recorded is selected arbitrarily, from which the rest corners are recorded in a clockwise direction along the contour. For example, for the open contour as shown in Fig. 2(a)
Fig. 2 (a) An open contour. (b) A closed contour.
, if the top terminal corner pa is recorded first, then the sequence of the corners recorded is {pa,pb,pc}; for the closed contour as shown in Fig. 2(b), if the corner p¯b is recorded first, then the sequence of the corners recorded is {p¯b,p¯c,p¯d,p¯e,p¯a}.From Eq. (2), we can divide the entire set of corners detected from an image (denoted as C) into several subsets based on different contour information, i.e.,
C={c1,c2,...,cNc}
(3)
where Nc is the total number of contours extracted from the image.

The detailed procedure to construct the FSCs is formally presented in Algorithm 1.Algorithm 1: FSC constructionInput: the whole set of corners C={c1,c2,...,cNc};Output: a set of FSCs, named G;1. for i=1:Nc do2. if the i'th contour is open and ni5 do3. for j=1:(ni4) do4. record {pji,pj+1i,pj+2i,pj+3i,pj+4i}and {pj+4i,pj+3i,pj+2i,pj+1i,pji} into G;5. end for6. elseif the i'th contour is closed and ni5 do7. for j=1:ni do8. record {pji,pj+1i,pj+2i,pj+3i,pj+4i}and {pj+4i,pj+3i,pj+2i,pj+1i,pji} into G;9. end for10. end if11. end for

Remark 2: In Line 2 and Line 6 of Algorithm 1, nidenotes the number of corners on the i'th extracted contour, for i{1,2,,Nc}. From Line 8, if the value of the subscript (j+k) for k{1,2,3,4} is larger than ni, pj+kiwill be replaced with pj+kniiwhen Algorithm 1 is under execution.

We shall use an example to better illustrate the idea of Algorithm 1. Given a reference image as shown in Fig. 3(a)
Fig. 3 An example of five pairs of corresponding corners between the reference and sensed images.
, p1, p2, p3, p4 and p5 are five corners detected from a contour. Suppose that there is a sensed image in Fig. 3(b), where p1', p2', p3', p4', p5' are the five corners detected from the contour. Clearly,{pi,pi'} for i=1,2,,5 are the five pairs of corresponding corners. It can also be seen that p1, p2, p3, p4, p5 and p1', p2', p3', p4', p5' are arranged in the same order on the contours in the two images. This is because the sequence of the five points on a contour can keep invariant after arbitrary projective transformations between the reference image and the sensed image.According to Line 2-4 of Algorithm 1, for a group of five successive corners on an open contour, only two FSCs indicating opposite traversal directions will be constructed by Algorithm 1. For example, g1={p1,p2,p3,p4,p5},g2={p5,p4,p3,p2,p1} and g¯1={p1',p2',p3',p4',p5'},g¯2={p5',p4',p3',p2',p1'} are the two constructed FSCs for the reference image and the sensed image in Fig. 3, respectively. Moreover, {g1,g¯1}and {g2,g¯2} are two pairs of corresponding FSCs obtained from the image pair in Fig. 3 according to Definition 2.

Note that if only one traversal direction had been considered in constructing the FSCs, it cannot be ensured that sufficiently large number of corresponding FSC pairs will be obtained since the first traversed corners for the group of corresponding five corners may not be matched. To be more concrete, if p1, p2, p3, p4 and p5are sequentially traversed once for Fig. 3(a), only the FSC g1 is constructed for the reference image. However, if p5',p4',p3',p2',p1' are sequentially traversed once for Fig. 3(b), only the FSC g¯2 is constructed for the sensed image. Therefore, no pair of corresponding FSCs is obtained from the image pair in Fig. 3.

Apart from these, the number of the FSCs to be constructed will be 2(n4), if there is an open contour with n corners (n5). If the contour is closed, such number will be2n. Suppose that the total number of the corners in C is N, i.e. n1+n2++nNc=N, then at most 2(n1+n2++nNc)=2N FSCs can be constructed by Algorithm 1. Obviously, compared with the aforementioned randomly choosing strategy with which N!/(N5)! FSCs will be constructed, Algorithm 1 can effectively reduce the number of the FSCs to be constructed.

2.3 Design of descriptors for the FSC

Generally, a good feature descriptor should satisfy the following two requirements:

(1) it is robust against the differences such as projective geometric deformation and gray scale discrepancy between the reference and sensed images.

(2) the descriptors for a pair of non-corresponding features should be distinctly different from each other.

In this paper, the algebraic projective invariants [19

19. J. L. Mundy and A. Zisserman, Geometric Invariance in Computer Vision (the MIT Press, 1992).

] will be utilized to ensure that the proposed descriptor satisfies the above two requirements.

For a group of five co-plane points, pi=(xi,yi), with no three of which being co-linear, two functionally independent projective invariants can be defined as [19

19. J. L. Mundy and A. Zisserman, Geometric Invariance in Computer Vision (the MIT Press, 1992).

]:
{INV1(p1,p2,p3,p4,p5)=det(m431)det(m521)det(m421)det(m531)INV2(p1,p2,p3,p4,p5)=det(m421)det(m531)det(m432)det(m521)
(4)
where det(m)denotes the determinant of mand mijk=(qi,qj,qk)with qi=(xi,yi,1)T.

If p¯i=(xi,yi) is the corresponding point of pi=(xi,yi) after an arbitrary projective transform, the following equations hold [19

19. J. L. Mundy and A. Zisserman, Geometric Invariance in Computer Vision (the MIT Press, 1992).

],
INVi(p1,p2,p3,p4,p5)=INVi(p¯1,p¯2,p¯3,p¯4,p¯5),i=1,2
(5)
Based on the introduced projective invariants, we adopt a ten-dimensional vector to describe each FSC. Take the FSC, g={pa,pb,pc,pd,pe}, for example, its descriptor is given as:

des(g)=[INV1(pa,pb,pc,pd,pe)INV2(pa,pb,pc,pd,pe)INV1(pb,pc,pd,pe,pa)INV2(pb,pc,pd,pe,pa)INV1(pc,pd,pe,pa,pb)INV2(pc,pd,pe,pa,pb)INV1(pd,pe,pa,pb,pc)INV2(pd,pe,pa,pb,pc)INV1(pe,pa,pb,pc,pd)INV2(pe,pa,pb,pc,pd)]
(6)

Remark 3: In fact, increasing the dimensions of a descriptor will improve its ability to identify the pair of corresponding features in the step of feature matching. However, it will also increase the computational load. According to the experimental results on various image pairs, a ten-dimensional descriptor is selected as a good compromise.

The properties of the descriptor proposed above are summarized as follows:

(1) It is composed of ten projective invariants, which can maintain unchanged after arbitrary projective deformations according to Eq. (5). Thus the proposed descriptor is robust against projective deformations.

(2) Since no gray scale information is used in calculating the proposed descriptor, it is robust against gray scale discrepancy between the corresponding pixels of the image pairs.

(3) From the definition of the invariants in Eq. (4), the values of the invariants will be changed once the coordinates or the recording sequence of the five corners change. Therefore, different FSCs will have different descriptors, which enable us to identify the corresponding FSCs by comparing their descriptors.

2.4 Feature matching

The purpose of the feature matching step (i.e. Step 5 of Fig. 1) is to obtain a set of matched corners which can be used to estimate the parameters of the transformation model (i.e. Step 6 of Fig. 1). In this section, we first get the matched FSCs by comparing their descriptors, then obtain five matched corners from each pair of the matched FSCs. To this end, we define the following distance measure to indicate the similarity of two descriptors.

Ddes(g1),des(g2)=i=110(aibi)2ai2+bi2
(7)

where des(g1)=[a1a2a10]T and des(g2)=[b1b2b10]T denote two arbitrary descriptors. In Eq. (7), each component represents the difference between des(g1) and des(g2) on a certain direction.

The detailed feature matching procedure is described in Algorithm 2:Algorithm 2: Feature MatchingInput: two sets of FSCs G1={g1,g2,,gN1},G2={g¯1,g¯2,,g¯N2} and their descriptors;Output: a set of matched corners F;1. for i=1:N1 do2. find gi’s nearest and second nearest neighbors from G2, and record them as g¯jand g¯k respectively;3. calculate dr=Ddes(gi),des(g¯j)/Ddes(gi),des(g¯k);4. if dr<σ and gi is also g¯j’s nearest neighbors from G1 do5. record (p1i,p¯1j), (p2i,p¯2j), (p3i,p¯3j), (p4i,p¯4j) and (p5i,p¯5j) into F;6. end if7. end for

Remark 4: In Line 4 of Algorithm 2, σ is a threshold belonging to the interval [0,1]. Sec. 3.2 will be devoted to discuss how to choose σproperly.

3. Analysis of accuracy rate and repetition rate of the proposed algorithm

For a feature matching algorithm, the purpose is to extract the corresponding features exactly from two sets of features. However, some wrong matches will inevitably exist in its output which is a set of matched features. Besides, not all the corresponding pairs can be covered by the input sets. In this regard, the accuracy rate and repetition rate are two standard indexes introduced to indicate whether a feature matching result is good or not [24

24. K. Mikolajczyk and C. Schmid, “Performance evaluation of local descriptors,” IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1615–1630 (2005).

]. The accuracy rate is defined as the ratio of the number of corresponding pairs in the output to the number of the matched pairs in the output, i.e.

accuracyrate=#correspondingpairsinthe output#matchesinthe output
(8)

The repetition rate is defined as the ratio of the number of corresponding pairs in the output to the number of corresponding ones in the input, i.e.
repetitionrate=#correspondingpairsinoutput#correspondingpairsininput
(9)
In this section, the following simulation model is considered to generate the testing data
{x¯=ax+by+cgx+hy+1+αRcos(2πβ)y¯==dx+ey+fgx+hy+1+αRsin(2πβ)
(10)
where
H=[abcdefgh1]
(11)
is a projective model, (x,y) and (x,¯y¯) denote a chosen point from reference image and its corresponding point in the sensed image. (αRcos(2πβ),αRsin(2πβ)) represents the influence of a positioning error which is the distance between the extracted corner and the real corner on the image in real applications. R is a constant, α and β are two random variables with uniform distributions [0,1]

4. Experiments

In this section, an experiment is firstly conducted to validate the robustness of the proposed registration method against projective deformation and gray scale discrepancy. Then comparative results of the proposed method and two existing representative feature-based image registration methods, i.e. SIFT and SURF algorithms [17

17. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).

,18

18. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).

], are provided.

4.1 Robustness Validation

The test images in Fig. 5
Fig. 5 Test images for robustness validation experiment.
are adopted as the experimental database to show the robustness of the proposed method. Fig. 5(a) and 5(b) are a pair of well registered thermal infrared and visual images. Fig. 5(c) and 5(d) are the transformed images of Fig. 5(a) and 5(b), respectively, based on different projective models, which are known. As indicated in Fig. 5, three test sets are constructed from Fig. 5(a)-(d):• Test Set 1: Fig. 5(a) and 5(c). Only projective deformation exists in the image pair of this set;• Test Set 2: Fig. 5(a) and 5(b). Only gray scale discrepancy exists, in the image pair of this set;• Test Set 3: Fig. 5(a) and 5(d). Both projective deformation and gray scale discrepancy exists in the image pair of this test set.

The experiment results are provided in Table 1

Table 1. Feature Matching Results of Test Sets 1-3 by the Proposed Algorithm

table-icon
View This Table
| View All Tables
. It is worth mentioning that since Fig. 5(a), 5(b) are well registered, and the projective models corresponding to Fig. 5(c), 5(d) are both known, the number of corresponding (corner) pairs in both the input and output of the algorithm for each test set can be counted. Thus the accuracy rates and the repetition rates as defined in Eqs. (8) and (9) of the registration results for all three test sets can be explicitly calculated. Besides, the corresponding pairs in the matched results for each test set are shown in Fig. 6
Fig. 6 The corresponding pairs in the matched results by the proposed method for each test set.
.According to Table 1, the following conclusions can be drawn.

  • i) It is known that four pairs of corners can determine a projective model with eight parameters involved. It can be seen that nearly 10 corresponding pairs of corners are obtained after feature matching for the three test sets. Thus the projective transformation models corresponding to all the test sets can be estimated effectively;
  • ii) The accuracy rates and repetition rates for all the three test tests are over 0.4 and 0.5, respectively. Although different degrees of projective deformation and gray scale discrepancy exist in the image pairs, similar registration results in terms of the accuracy rates and repetition rates can be observed. Based on this, satisfying robustness of the proposed registration method against projective deformation and gray scale discrepancy is demonstrated.

4.2 Performance Comparison with Existing Representative Algorithms

Three pairs of multi-sensor images as shown in Figs. 7
Fig. 7 Test Set 4. (a) Reference image. (b) Sensed image.
, 8
Fig. 8 Test Set 5. (a) Reference image. (b) Sensed image.
, and 9
Fig. 9 Test Set 6. (a) Reference image. (b) Sensed image.
are used as the test data to compare the performance of the proposed method with the well-known SIFT and SURF algorithms [17

17. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).

,18

18. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).

]. Note that the source codes of SIFT and SURF algorithms used in conducting the comparative experiment can be found on “www.cs.ubc.ca/~lowe/keypoints/” and “www.chrisevansdev.com/computer-visionopensurf.html”, respectively. In contrast to the robustness validation experiment presented above, the actual projective model corresponding to the image pairs in Fig. 7-9 are not known. Thus 10 pairs of the corresponding points are manually selected from each image pair and the Root-Mean-Squared Error (RMSE) is utilized to evaluate the accuracies of the registration results.Observing Fig. 7-9, both projective deformation and gray scale discrepancy are involved in the three test sets. In Test Set 4, the image pair is visual and near infrared images, respectively. The gray scale discrepancy for this set is much weaker than those for Test Sets 5 and 6. The registration performances of SIFT, SURF and our proposed algorithm for Test Sets 4-6 are given in Table 2

Table 2. Performance Comparison of SIFT, SURF and the Proposed Algorithms

table-icon
View This Table
| View All Tables
, including the number of obtained matches in the outputs and RMSE of the registration results.

It can be seen that SIFT and SURF can obtain enough matches to estimate the projective model in Test Set 4, however, the RMSE of the registration results by using these two algorithms are too large to be acceptable. This is because that the matches obtained by SIFT and SURF contain not enough corresponding corners (correct matches). For Test Sets 5 and 6, both SIFT and SURF deliver none matches in the outputs. Therefore, these two algorithms failed to provide valid registration results due to the effects of much serious gray scale discrepancy between the image pairs. In contrast to these, our proposed method can provide good registration results for all the three test sets with RMSE less than one. The aligned images by the proposed method are provided in Fig. 10
Fig. 10 Aligned images by the proposed method. (a) Test set 1. (b) Test set 2. (c) Test set 3.
.

5. Conclusion

In this paper, a new feature-based multi-sensor image registration algorithm is proposed based on contour feature extraction and algebraic projective invariants. Since the designed feature descriptor depends on the coordinate information of the corners only and has no relation with the gray scale information, the proposed method has good robustness against the gray scale discrepancy between the corresponding pixels. Moreover, the algebraic projective invariants guarantee that the feature descriptor is also robust against the projective deformation. The experiment results are given to show the effectiveness of the proposed scheme

Acknowledgment

This work was supported in part by the 973 Program of China under Grant 2010CB731800 and in part by the National Natural Science Foundation of China under Grants 61290324, 60974059, 60736026, 61021063 and 61203068.

References and links

1.

B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).

2.

J. J. Arthur, L. J. Kramer, and R. E. Bailey, “Flight test comparison between enhanced vision (FLIR) and synthetic vision systems,” (SPIE-INT SOC Optical Engineering, BELLINGHAM, 2005), pp. 25–36.

3.

R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam, “Image change detection algorithms: a systematic survey,” IEEE Trans. Image Process. 14(3), 294–307 (2005).

4.

J. P. Pluim, J. B. Maintz, and M. A. Viergever, “Mutual-information-based registration of medical images: a survey,” IEEE Trans. Med. Imaging 22(8), 986–1004 (2003).

5.

H. Luan, F. Qi, Z. Xue, L. Chen, and D. Shen, “Multimodality image registration by maximization of quantitative-qualitative measure of mutual information,” Pattern Recognit. 41(1), 285–298 (2008).

6.

F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, “Multimodality image registration by maximization of mutual information,” IEEE Trans. Med. Imaging 16(2), 187–198 (1997).

7.

A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, “Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient,” IEEE Trans. Image Process. 12(12), 1495–1511 (2003).

8.

J. P. Heather and M. I. Smith, “Multimodal image registration with applications to image fusion,” 2005 7th International Conference on Information Fusion (FUSION), Vols 1 and 2, 372–379 (2005).

9.

M. Irani and P. Anandan, “Robust multi-sensor image alignment,” Sixth International Conference on Computer Vision, 959–966 (1998).

10.

Z. H. Zhang, C. H. Pan, and S. D. Ma, “An automatic method of coarse registration between multi-source satellite images,” Proceedings of the 2004 Intelligent Sensors, Sensor Networks & Information Processing Conference, 205–209 (2004).

11.

M. A. Ali and D. A. Clausi, “Automatic registration of SAR and visible band remote sensing images,” 2002 IEEE International Geoscience and Remote Sensing Symposium. 24th Canadian Symposium on Remote Sensing. Proceedings (Cat. No.02CH37380), 1331–1333 (2002).

12.

H. Li, B. S. Manjunath, and S. K. Mitra, “A contour-based approach to multisensor image registration,” IEEE Trans. Image Process. 4(3), 320–334 (1995).

13.

S. Dawn, V. Saxena, and B. Sharma, “Remote Sensing Image Registration Techniques: A Survey,” Image and Signal Processing. Proceedings 4th International Conference, ICISP 2010, 103–112 (2010).

14.

Z. L. Song and J. P. Zhang, “Remote Sensing Image Registration Based on Retrofitted SURF Algorithm and Trajectories Generated From Lissajous Figures,” IEEE Geosci. Remote Sens. Lett. 7(3), 491–495 (2010).

15.

Z. L. Song, S. Li, and T. F. George, “Remote sensing image registration approach based on a retrofitted SIFT algorithm and Lissajous-curve trajectories,” Opt. Express 18(2), 513–522 (2010).

16.

N. Alajlan, I. El Rube, M. S. Kamel, and G. Freeman, “Shape retrieval using triangle-area representation and dynamic space warping,” Pattern Recognit. 40(7), 1911–1920 (2007).

17.

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).

18.

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).

19.

J. L. Mundy and A. Zisserman, Geometric Invariance in Computer Vision (the MIT Press, 1992).

20.

J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986).

21.

F. Mokhtarian and R. Suomela, “Robust image corner detection through curvature scale space,” IEEE Trans. Pattern Anal 20(12), 1376–1381 (1998).

22.

X. C. He and N. H. C. Yung, “Corner detector based on global and local curvature properties,” Opt. Eng. 47, 057008 (2008).

23.

M. A. Fischler and R. C. Bolles, “Random Sample Consensus - A Paradigm For Model-Fitting With Applications To Image-Analysis And Automated Cartography,” Commun. ACM 24, 381–395 (1981).

24.

K. Mikolajczyk and C. Schmid, “Performance evaluation of local descriptors,” IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1615–1630 (2005).

OCIS Codes
(100.2000) Image processing : Digital image processing
(100.5010) Image processing : Pattern recognition
(100.3008) Image processing : Image recognition, algorithms and filters

ToC Category:
Image Processing

History
Original Manuscript: January 9, 2013
Revised Manuscript: March 17, 2013
Manuscript Accepted: April 2, 2013
Published: April 12, 2013

Citation
Bin Li, Wei Wang, and Hao Ye, "Multi-sensor image registration based on algebraic projective invariants," Opt. Express 21, 9824-9838 (2013)
http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-21-8-9824


Sort:  Author  |  Year  |  Journal  |  Reset  

References

  1. B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput.21(11), 977–1000 (2003).
  2. J. J. Arthur, L. J. Kramer, and R. E. Bailey, “Flight test comparison between enhanced vision (FLIR) and synthetic vision systems,” (SPIE-INT SOC Optical Engineering, BELLINGHAM, 2005), pp. 25–36.
  3. R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam, “Image change detection algorithms: a systematic survey,” IEEE Trans. Image Process.14(3), 294–307 (2005).
  4. J. P. Pluim, J. B. Maintz, and M. A. Viergever, “Mutual-information-based registration of medical images: a survey,” IEEE Trans. Med. Imaging22(8), 986–1004 (2003).
  5. H. Luan, F. Qi, Z. Xue, L. Chen, and D. Shen, “Multimodality image registration by maximization of quantitative-qualitative measure of mutual information,” Pattern Recognit.41(1), 285–298 (2008).
  6. F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, “Multimodality image registration by maximization of mutual information,” IEEE Trans. Med. Imaging16(2), 187–198 (1997).
  7. A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, “Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient,” IEEE Trans. Image Process.12(12), 1495–1511 (2003).
  8. J. P. Heather and M. I. Smith, “Multimodal image registration with applications to image fusion,” 2005 7th International Conference on Information Fusion (FUSION), Vols 1 and 2, 372–379 (2005).
  9. M. Irani and P. Anandan, “Robust multi-sensor image alignment,” Sixth International Conference on Computer Vision, 959–966 (1998).
  10. Z. H. Zhang, C. H. Pan, and S. D. Ma, “An automatic method of coarse registration between multi-source satellite images,” Proceedings of the 2004 Intelligent Sensors, Sensor Networks & Information Processing Conference, 205–209 (2004).
  11. M. A. Ali and D. A. Clausi, “Automatic registration of SAR and visible band remote sensing images,” 2002 IEEE International Geoscience and Remote Sensing Symposium. 24th Canadian Symposium on Remote Sensing. Proceedings (Cat. No.02CH37380), 1331–1333 (2002).
  12. H. Li, B. S. Manjunath, and S. K. Mitra, “A contour-based approach to multisensor image registration,” IEEE Trans. Image Process.4(3), 320–334 (1995).
  13. S. Dawn, V. Saxena, and B. Sharma, “Remote Sensing Image Registration Techniques: A Survey,” Image and Signal Processing. Proceedings 4th International Conference, ICISP 2010, 103–112 (2010).
  14. Z. L. Song and J. P. Zhang, “Remote Sensing Image Registration Based on Retrofitted SURF Algorithm and Trajectories Generated From Lissajous Figures,” IEEE Geosci. Remote Sens. Lett.7(3), 491–495 (2010).
  15. Z. L. Song, S. Li, and T. F. George, “Remote sensing image registration approach based on a retrofitted SIFT algorithm and Lissajous-curve trajectories,” Opt. Express18(2), 513–522 (2010).
  16. N. Alajlan, I. El Rube, M. S. Kamel, and G. Freeman, “Shape retrieval using triangle-area representation and dynamic space warping,” Pattern Recognit.40(7), 1911–1920 (2007).
  17. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis.60(2), 91–110 (2004).
  18. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis. Image Underst.110(3), 346–359 (2008).
  19. J. L. Mundy and A. Zisserman, Geometric Invariance in Computer Vision (the MIT Press, 1992).
  20. J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell.8(6), 679–698 (1986).
  21. F. Mokhtarian and R. Suomela, “Robust image corner detection through curvature scale space,” IEEE Trans. Pattern Anal20(12), 1376–1381 (1998).
  22. X. C. He and N. H. C. Yung, “Corner detector based on global and local curvature properties,” Opt. Eng.47, 057008 (2008).
  23. M. A. Fischler and R. C. Bolles, “Random Sample Consensus - A Paradigm For Model-Fitting With Applications To Image-Analysis And Automated Cartography,” Commun. ACM24, 381–395 (1981).
  24. K. Mikolajczyk and C. Schmid, “Performance evaluation of local descriptors,” IEEE Trans. Pattern Anal. Mach. Intell.27(10), 1615–1630 (2005).

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.


« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited