## Photographic stitching with optimized object and color matching based on image derivatives

Optics Express, Vol. 15, Issue 12, pp. 7689-7696 (2007)

http://dx.doi.org/10.1364/OE.15.007689

Acrobat PDF (279 KB)

### Abstract

In this paper, a novel optimization-based stitching method is presented. It minimizes an energy function defined with derivatives up to the second order. We have identified some appropriate choices for its parameters, allowing it to reduce artifacts such as ghosting, color inconsistency, and misalignment. To accelerate the computation, a multi-resolution technique is introduced. The significant speedup and memory saving make it possible for use in hand-held capturing devices.

© 2007 Optical Society of America

## 1. Introduction

2. F. Meyer and S. Beucher, “Morphological segmentation,” J. Vis. Comm. and Imag. Rep. **1**, 21–46 (1990). [CrossRef]

4. P. Soille, “Morphological image compositing,” IEEE Trans. Pat. Anal. and Mach. Intell. **28**, 673–683 (2006). [CrossRef]

6. C. T. Hsu and J. L. Wu, “Multiresolution mosaic,” IEEE Trans. Cons. Elec. **42**, 981–990 (1996). [CrossRef]

8. M. S. Su, W. L. Hwang, and K. Y. Cheng, “Analysis on multiresolution mosaic images,” IEEE Trans. Imag. Proc. **13**, 952–959 (2004). [CrossRef]

## 2. Our optimization stitching method

*g*ô(

*x*,

*y*)} of the mosaic result

*g*ô(

*x*,

*y*), which includes derivatives up to the second order, where

*g*ô(

*x*,

*y*) for every pixel location. Here, Ω

_{1}and Ω

_{2}are the capturing regions of the input images

*g*

_{1}(

*x*,

*y*) and

*g*

_{2}(

*x*,

*y*) respectively.

*D*is the

^{i}f*i*

^{th}derivative of the real function

*f*such that

*D*

^{0}

*f*=

*f*,

*α*(

*x*,

*y*) is abinary mask indicating if

*g*

_{1}(

*x*,

*y*) or

*g*

_{2}(

*x*,

*y*) is selected.

*β*(

_{i}*x*,

*y*) is also abinary mask representing whether the

*i*

^{th}derivative is selected at pixel (

*x*,

*y*). In Section 2.1, we explain how to find a curve to avoid passing through inconsistent objects and use it to design

*α*(

*x*,

*y*). In Section 2.2, we will describe the properties of the derivatives and our design of

*β*(

_{i}*x*,

*y*).

### 2.1. Optimal cut based on intensities and gradients

*α*(

*x*,

*y*) equals 1 if we want to select a pixel from

*g*

_{1}(

*x*,

*y*) and 0 otherwise. As with other pixel selection method [1, 4

4. P. Soille, “Morphological image compositing,” IEEE Trans. Pat. Anal. and Mach. Intell. **28**, 673–683 (2006). [CrossRef]

*g*

_{1}(

*x*,

*y*). Usually, double-edge artifact can be prevented as no pixels are mixed. Here we introduce a method to give a cut without passing through moving objects. First, we capture the intensity dissimilarity from the overlap region. To compensate for photometric inconsistency, we subtract the mean intensity from each pixel for each input image beforehand. The dissimilarity

*a*(

*x*,

*y*) is represented by their absolute differences:

*a*(

*x*,

*y*). Next, we measure the gradient consistency

*b*(

*x*,

*y*):

*ρ*(∙) denotes a morphological gradient operator with structuring element

_{A}*A*[10]. The size of A increases with more misalignment such that the same edge in

*g*

_{1}(

*x*,

*y*) and

*g*

_{2}(

*x*,

*y*) can manifest as an edge in

*b*(

*x*,

*y*). We combine

*a*(

*x*,

*y*) and

*b*(

*x*,

*y*) to form a mask image

*c*(

*x*,

*y*):

2. F. Meyer and S. Beucher, “Morphological segmentation,” J. Vis. Comm. and Imag. Rep. **1**, 21–46 (1990). [CrossRef]

*c*(

*x*,

*y*) to form an image with labels for selecting the input images. In theory, the optimal cut is located along consistent edges that are not on moving objects. We then assign

*α*(

*x*,

*y*) = 1 to the side of

*g*

_{1}(

*x*,

*y*) and

*α*(

*x*,

*y*) = 0 otherwise. For color images, we can choose any one of the color planes to compute the cut and assign its corresponding mask

*α*(

*x*,

*y*) for all the planes. In Sect. 4, our experiment shows that this algorithm works well even on the plane with severe photometric inconsistency.

### 2.2. Transition smoothness and fidelity

*β*(

_{i}*x*,

*y*) determines the contribution of the derivatives. The zeroth derivatives (i.e. image intensities) contain the complete information of an image and do not reject inconsistency. The first derivatives remove the mean intensities of an image, removing uniform inconsistency because the input images can match each other regardless of their global differences. For the second derivatives, both mean intensities and gradients are removed from an image. Hence, spatially varying inconsistency can also be reduced when images are matched in this domain. Readers can refer to our earlier work [11, 13] for a detailed mathematical analysis.

*β*(

_{2}*x*,

*y*) = 1 in the region and 0 elsewhere. In addition, users can assign either the zeroth or the first derivatives to the remaining region according to their requirement on fidelity relative to the input images. If matching the mean intensities is preferred, the first derivatives should be selected. We assign

*β*

_{1}(

*x*,

*y*) = 1 in the region and 0 elsewhere, while setting

*β*

_{0}(

*x*,

*y*) = 0 for all pixels. Similarly, if they prefer to keep the intensities, we swap

*β*

_{0}(

*x*,

*y*) and

*β*

_{1}(

*x*,

*y*) in this assignment rule. These features favor the subjective requirement of the users.

## 3. Basic implementation and multi-resolution enhancement technique

*g*ˆ(

*x*,

*y*) denoted as

*g*ˆ

_{0}(

*x*,

*y*), such as when we simply merge

*g*

_{1}(

*x*,

*y*) and

*g*

_{2}(x,

*y*) according to the pixel locations with respect to the cut, i.e.

## 4. Experimental results and discussion

15. “The Panorama Factory V4.4 for Windows XP,” http://www.panoramafactory.com/.

### 4.1. Comparisons on identifying an optimal cut

4. P. Soille, “Morphological image compositing,” IEEE Trans. Pat. Anal. and Mach. Intell. **28**, 673–683 (2006). [CrossRef]

*α*(

*x*,

*y*), where black is 0 and white is 1, can be created according to this curve.

### 4.2. Comparisons on stitching methods

*α*(

*x*,

*y*) created above to eliminate ghosting. Also, we apply the multi-resolution enhancement technique for Wavelet Blending, Gradient Stitching and our method. It decomposes the images into 6 wavelet levels and thus the approximation image is 30 × 40 pixels each. For our method, the optimization process can finish in about 10 seconds for each channel.

*M*is the MSE between the result and the two input images within the overlap. Smaller

_{s}*M*implies that the result is more similar to the average of the images, and thus the transition is smoother. The fidelity metric

_{s}*M*is the MSE outside the overlap regions. A smaller

_{f}*M*means better fidelity.

_{f}*M*and

_{s}*M*for (a) Exposure Compensation, (b) Wavelet Blending, (c) Gradient Stitching, and (d) and (e) our method with different

_{f}*β*(

_{i}*x*,

*y*). In (d), we assign 1 to the middle 65% vertical portion of

*β*

_{2}(

*x*,

*y*),

*β*

_{0}(

*x*,

*y*) = 1 in the remaining area, and

*β*

_{1}(

*x*,

*y*) = 0 for all pixels. In (e), we assign 1 to the 85% portion of

*β*

_{2}(

*x*,

*y*) instead. Note that since the vehicles shown in Fig. 1(b) are removed in most of the results, we do not count the errors in

*M*for those regions. In the Table, Exposure Compensation gives the smallest

_{s}*M*, followed by our method with the two settings. Thus, our technique gives a smoother result than Wavelet Blending and Gradient Stitching. Moreover, the result with our second parameter setting is smoother than the first one. Our first design achieves the smallest

_{s}*M*, followed by our second one. Therefore, our method can maintain the content better than the other methods.

_{f}*M*) and fidelity metric (

_{s}*M*) after stitching. We can observe that the comparison can match with that presented in this paper.

_{f}### 4.3. Comparisons on decomposition level count

## 5. Conclusion

## References and links

1. | F. P. Araújo Jr. and N. J. Leite, “A morphological algorithm for photomosaicking,” in |

2. | F. Meyer and S. Beucher, “Morphological segmentation,” J. Vis. Comm. and Imag. Rep. |

3. | M. Uyttendaele, A. Eden, and R. Szeliski, “Eliminating ghosting and exposure artifacts in image mosaics,” in |

4. | P. Soille, “Morphological image compositing,” IEEE Trans. Pat. Anal. and Mach. Intell. |

5. | A. Levin, A. Zomet, S. Peleg, and Y. Weiss, “Seamless image stitching in the gradient domain,” in |

6. | C. T. Hsu and J. L. Wu, “Multiresolution mosaic,” IEEE Trans. Cons. Elec. |

7. | M. S. Su, W. L. Hwang, and K. Y. Cheng, “Variational calculus approach to multiresolution image mosaic,” in |

8. | M. S. Su, W. L. Hwang, and K. Y. Cheng, “Analysis on multiresolution mosaic images,” IEEE Trans. Imag. Proc. |

9. | A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in Proceedings of SIGGRAPH ’01: 28th Annual Conference on Computer Graphics and Interactive Techniques (ACM, 2001), pp. 341–346. |

10. | R. Gonzalez and R. Woods, |

11. | S. T. Suen, E. Y. Lam, and K. K. Wong, “Digital photograph stitching with optimized matching of gradient and curvature,” Proc. SPIE |

12. | S. Boyd and L. Vandenberghe, |

13. | S. T. Suen, E. Y. Lam, and K. K. Wong, “Photographic mosaic for camera phones based on minimization of curvature value variations,” Tech. rep., Department of Electrical and Electronic Engineering, The University of Hong Kong (2006), http://www.eee.hku.hk/research/research_reports.htm. |

14. | G. Strang and T. Nguyen, |

15. | “The Panorama Factory V4.4 for Windows XP,” http://www.panoramafactory.com/. |

**OCIS Codes**

(100.2000) Image processing : Digital image processing

(110.4190) Imaging systems : Multiple imaging

(330.1690) Vision, color, and visual optics : Color

**ToC Category:**

Image Processing

**History**

Original Manuscript: March 12, 2007

Revised Manuscript: May 15, 2007

Manuscript Accepted: May 29, 2007

Published: June 7, 2007

**Virtual Issues**

Vol. 2, Iss. 7 *Virtual Journal for Biomedical Optics*

**Citation**

Simon T. Suen, Edmund Y. Lam, and Kenneth K. Wong, "Photographic stitching with optimized object and color matching based on image derivatives," Opt. Express **15**, 7689-7696 (2007)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-15-12-7689

Sort: Year | Journal | Reset

### References

- F. P. Junior and N. J. Leite, "A Morphological Algorithm for Photomosaicking," 8th European Signal Processing Conference 3, 1881-1884 (1996).
- F. Meyer and S. Beucher, "Morphological Segmentation," J. Visual Comm. and Image Representation 1, 21-46 (1990). [CrossRef]
- M. Uyttendaele, A. Eden, and R. Szeliski, "Eliminating Ghosting and Exposure Artifacts in Image Mosaics," IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2, 509-516 (2001).
- P. Soille, "Morphological Image Compositing," IEEE Trans. Pattern Analysis and Machine Intelligence 28, 673- 683 (2006). [CrossRef]
- A. Levin, A. Zomet, S. Peleg, and Y. Weiss, "Seamless image stitching in the Gradient Domain," 8th European Conference on Computer Vision 4, 377-389 (2004).
- C. T. Hsu and J. L. Wu, "Multiresolution Mosaic," IEEE Trans. Consumer Electronics 42, 981-990 (1996). [CrossRef]
- M. S. Su, W. L. Hwang, and K. Y. Cheng, "Variational calculus approach to multiresolution image mosaic," International Conference on Image Processing 2, 245 - 245 (2001).
- M. S. Su, W. L. Hwang, and K. Y. Cheng, "Analysis on multiresolution mosaic images," IEEE Trans. Image Processing 13, 952-959 (2004). [CrossRef]
- A. A. Efros andW. T. Freeman, "Image quilting for texture synthesis and transfer," SIGGRAPH ’01: 28th Annual Conference on Computer Graphics and Interactive Techniques, 341-346 (2001).
- S. T. Suen, E. Y. Lam, and K. K. Wong, "Digital photograph stitching with optimized matching of gradient and curvature," Proc. SPIE 6069, 139-154 (2006).
- S. T. Suen, E. Y. Lam, and K. K. Wong, "Photographic mosaic for camera phones based on minimization of curvature value variations," Tech. rep., Department of Electrical and Electronic Engineering, the University of Hong Kong (2006), http://www.eee.hku.hk/research/research_reports.htm.
- G. Strang and T. Nguyen, Wavelets and Filter Banks (MA: Wellesley-Cambridge Press, 1996).
- "The Panorama Factory V4.4 for Windows XP," http://www.panoramafactory.com/.

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.