OSA's Digital Library

Optics Letters

Optics Letters

| RAPID, SHORT PUBLICATIONS ON THE LATEST IN OPTICAL DISCOVERIES

  • Editor: Alan E. Willner
  • Vol. 38, Iss. 17 — Sep. 1, 2013
  • pp: 3471–3474
« Show journal navigation

Super-resolution reconstruction based on incoherent optical aperture synthesis

Tao Sun, Jianguo Liu, Hongshi Yan, Gareth Morgan, and Wangli Chen  »View Author Affiliations


Optics Letters, Vol. 38, Issue 17, pp. 3471-3474 (2013)
http://dx.doi.org/10.1364/OL.38.003471


View Full Text Article

Acrobat PDF (1112 KB)





Browse Journals / Lookup Meetings

Browse by Journal and Year


   


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools

Share
Citations

Abstract

Based on incoherent optical synthetic aperture, a super-resolution reconstruction method is proposed with pixel-wise phase correlation superposition and sparse a priori constraints deconvolution. A “full spatial-frequency” image is synthesized from multiview low resolution images captured from successive exposures of a single focus plane array with shift-motion over a short range. Owing to a displacement matrix of 1/30th pixel accuracy and high-order differential L1 norm sparse regularization, the experimental results show that this approach achieves a state-of-the-art result with a resolution gain factor of about 3.0 times. It is demonstrated theoretically and experimentally to be a good and practical solution to improve spatial resolution beyond the resolving power limitation of conventional imaging systems and at low cost.

© 2013 Optical Society of America

Diffraction theory reveals that an optical imaging system acts as a low-bandpass filter, sparing the low spatial frequencies of an object’s spectrum but cutting off high frequency information. Some of the high spatial-frequency signal information will be lost almost completely, due to the finite size of the lens apertures. Moreover, for digital imaging, the finite number of detectors in a sensor array (e.g., a CCD) is another factor that limits an imaging system’s spatial bandwidth. How to obtain a full spatial-frequency image is an important and fundamental research topic. Many attempts have been made to improve the resolving power of optical imaging systems, typically requiring large size lenses and complicated configurations that are costly and not always practical. However, recent advances in imaging technology have shown that combining the design of optics and optical electronics with digital image processing can lead to more effective imaging systems [1

1. J. N. Mait, R. Athale, and J. van der Gracht, Opt. Express 11, 2093 (2003). [CrossRef]

]. For instance, the well-known synthetic aperture (SA) concept has been tested as a promising approach using computational imaging to increase resolving power in many applications [2

2. G. O. Reynolds and D. J. Cronin, J. Opt. Soc. Am 60, 634 (1970).

], which can extend the spatial frequency of an imaging system beyond its normal limitations.

The optical aperture synthesis technique is widely used in various imaging systems for holographic, microscopic, and astronomical imaging with different modes. In the imaging system of digital holography, a high resolution image can be achieved from two or more real or virtual subapertures via a combination of low resolution images through postprocessing, on condition that the object’s complex amplitude is available and the CCD is shifted to different off-axis positions [3

3. B. Katz and J. Rosen, Opt. Express 18, 962 (2010). [CrossRef]

5

5. V. Micó, L. Granero, Z. Zalevsky, and J. García, J. Opt. A 11, 125408 (2009). [CrossRef]

]. Then the SA will improve the system resolution. As another promising method, distributed aperture or multiaperture synthesis [6

6. D. Rabb, D. Jameson, A. Stokes, and J. Stafford, Opt. Express 18, 10334 (2010). [CrossRef]

] is able to generate high-resolution images from an array of small apertures but with a more complex setup and at higher cost. However, most of the above-mentioned methods are usually restricted to coherent imaging within in a laboratory setting, requiring complex configuration. Furthermore, the speckle noise of coherent imaging is often unavoidable and results in a serious negative impact on image quality.

For both spatially and temporally incoherent illumination, a high-resolution image can also be generated from a SA by superposing, on a single photographic plate, successive exposures [7

7. G. W. Stroke, Opt. Commun. 1, 283 (1970). [CrossRef]

]. In [7

7. G. W. Stroke, Opt. Commun. 1, 283 (1970). [CrossRef]

], it has been experimentally demonstrated that the synthesis of a high-resolution image could be achieved in a laboratory. In this experiment, however, a set of aperture “masks” were placed in front of a large size lens to avoid the problem of focus plane array position detection. We do not think that such an approach can fully recover the spatial resolution of the large aperture lens, but it has a hidden advantage: the smaller aperture enables greater depth-of-field while reconstructing higher resolution images.

Triggered by the above SA experiments, we present a computational imaging approach for super-resolution reconstruction (SRR) by synthesizing a virtual large aperture based on an incoherent imaging system. The key idea here is to consider a sequence of low resolution images of the same scene captured by one lens but with minor shifts, as images taken separately by an array of identical lenses or a phase shifted CCD camera with successive exposures. Then, a high spatial resolution image will be produced by exploiting temporal freedom of the imaging system according to the information capacity theory of optical systems [8

8. I. J. Cox and C. J. R. Sheppard, J. Opt. Soc. Am. 3, 1152 (1986). [CrossRef]

]. In this SA mode, only a series of intensity images is required but benefiting from an identical lens and a flexible setup. The reconstruction image will have the higher spatial resolution of the synthesized larger aperture lens but meanwhile have the greater depth of field offered by the small aperture lens.

Superposition of the multiviews is one of the key steps for SA-based SRR, requiring accurate image registration, which has been studied in [9

9. A. E. Tippie, A. Kumar, and J. R. Fienup, Opt. Express 19, 12027 (2011). [CrossRef]

] with a secondary camera to determine the CCD position. The accuracy of the displacement estimation was close to 1/7th pixel. In this Letter, however, the superposition is based on the displacement matrix with accuracy of 1/30th pixel, and then sparse a priori constraints deconvolution is applied for SRR. The approach can efficiently overcome degradation due to the limited size of the lens as well as the size of the CCD pixel. The final result of a SA generation extends the cutoff frequency of the imaging system much beyond that of an individual small lens aperture and thus yields a super-resolution image.

When the object illumination is incoherent, the imaging system can be expressed as
Ii(x,y)=+Io(ξ,η)h(xξ,yη)d(ξ)d(η).
(1)
Thus, the image intensity Ii is a convolution of the incoherent intensity impulse response h with the ideal image intensity Io. For digital imaging, the spatial integral effect is characterized by the system point spread function (PSF) that is composed of PSFopt and PSFccd. For simplicity, they are assumed to be modeled, respectively, by a 2D Gaussian function and a 2D rectangle function. Then the discrete digital images from different viewpoints, i, are represented by gi(x,y). The mathematical discrete description of an imaging system integral sum is rewritten as Eq. (2):
gi(x,y)=fPSFoptPSFccd+ni(x,y),
(2)
where f is the ideal image intensity.

Figure 1 illustrates a simplified scheme of the idea of this incoherent optical SA-based SRR method from a sequence of low resolution images, which are taken by a conventional imaging system under both spatially and temporally incoherent illumination. A camera with a small lens is assumed to shift over a relatively short range, with random subpixel increments, capturing consecutive images. The convolution operation in Eq. (2), a spatial integration effect, makes the high spatial frequency components largely unresolvable. The aim of this study is thus to find an optimal estimation of the SRR image f^ when a set of images gi(x,y) is available from the shift operation, each of which only contains partial information. Superposition based on pixel-wise phase-correlation and sparse a priori constraints deconvolution is performed for the incoherent SA-based SRR.

Fig. 1. Observation and SA reconstruction for conventional imaging system.

The CCD relative positions between the multiviews are calculated using a pixel-wise phase-correlation method [10

10. G. L. K. Morgan, J. G. Liu, and H. S. Yan, IEEE Trans. Geosci. Remote Sens. 35, 3456 (2010).

], which produces a subpixel accuracy displacement matrix. The phase correlation function is defined as
c^(k,l)=F1F*{gi}F{gi+1}|F*{gi}F{gi+1}|.
(3)
With Fourier inverse of the normalized cross-power spectrum of pairs of images, this method measures the motion vector from the phase shift directly. So it is relatively insensitive to frequency-dependent noise and illumination variations of the multiview images. As a result, a local displacement matrix with 1/30th pixel accuracy can be achieved with pixel-wise phase correlation combined with 2D Gaussian fitting. This is an experimentally validated good solution for superposition of multiple low resolution images captured with minor shift-motion approximately in the same imaging plane, which makes the proposed SA-based SRR method superior to any interpolation algorithm.

Then, the a priori constraints deconvolution is used to retrieve the high-frequency components by alleviating aliasing. Finding an optimization approximation f^ only from a set of low spatial observations gi is an ill-posed linear inverse problem, which generally leads to a convex minimizing problem. Mathematically,
f^=argmin(λR(f))s.tAfg2=0.
(4)

R(f) is a regularization term that incorporates a priori knowledge of the desired high resolution image. For large underdetermined systems of linear equations, there are various regularization operators. However, in this Letter, the L1 norm sparse prior regularization term including a first order differential operator as well as a second order differential operator are taken into account in this convex minimizing problem, formulated as
f^=argmin(λ(|Df|+|D2f|))s.tAfg2=0.
(5)

|Df| is a first-order-differential regularization term. Without |D2f|, Eq. (5) is equivalent to the total variation image restoration algorithm, which produces a sparse solution. It is especially effective on image deblurring with piecewise constant and piecewise linear image components, but otherwise staircase effects will be created, which is often the case for images of natural scenes. To alleviate the staircase effect, |D2f| for the second order differential operator is added as a smoothing regularity term in the proposed method, as shown in Eq. (5). This modification improves smoothness while keeping image structure/edges sharp and alleviating effectively the ringing artifacts generally resulted from deconvolution algorithm. Equation (5) is solved by the corresponding Euler–Lagrange equation with CG iteration algorithm. The viability of the proposed method has been demonstrated using standard simulation test images as well as real image datasets.

First, the 1951 United States Air Force (USAF) resolution target (AF target) was used for experimental validation and to quantify the improvement in image resolution. Figure 2(a) shows a sequence of 25 low resolution images each with vertical or horizontal 0.2 pixel displacement increments between consecutive images. They are generated from the ground-truth high-resolution image through blurring by a Gaussian function with delta 1.3, a down-sample factor 5, and Gaussian additive noise with noise level 0.005.

Fig. 2. AF target images for SA-based SRR: (a) low resolution image sequence; (b) the SRR reconstruction result with the proposed method; (c) and (d) present center parts ROIs of one low resolution image in (a); (e) and (f) present center parts of ROIs of the SRR result (b) corresponding to the ROIs (c) and (d).

The bar elements (groups/n:1, elements/m:2) shown in Fig. 2(d), a boxed region of interest (ROI) boxed area in Fig. 2(a), are hardly resolvable. By contrast, the high-resolution image synthesized using the proposed SA-based SRR, shown in Fig. 2(b), is sharply enhanced. The texture is fairly sharp. Figures 2(e) and 2(f) showing the corresponding ROIs to Figs. 2(c) and 2(d) demonstrate the improvement of image spatial resolution. Figure 2(e) shows that the bar elements (n:2, m:6) are easy resolvable.

Figure 3 presents 1D column profiles of horizontal bar elements of group 2 before and after the SA-based SRR. Figure 3(a) shows five profiles (n:2, m:26) out of 25 low resolution images, which are completely unresolved. By contrast, the profiles after processing of superposition and SRR, shown, respectively, as blue dashed line and a solid gray line in Fig. 3(b), are clearly resolved with sharp edges. For the AF target, an improvement by (n bars, m elements) is equivalent to a resolution improvement by a linear factor of 2(n+m/6). Note that the spatial resolution gain of the proposed method is about 2.8–3.2.

Fig. 3. 1D column profiles analysis of ROI (group 2, elements 2–6) in AF target.

Figure 4 shows the normalized power spectrum of the 25th low resolution image, the red bottom line, out of the AF target sequence, the SRR image in black, and the ground truth image in the blue. The blue dashed line and the black line are closely matched and both are far away from the red line at higher frequencies. This indicates that the high frequency components are substantially recovered by the SA-based SRR approach.

Fig. 4. Images normalization power spectrum.

A further test was carried out using aerial images captured with a vertical viewing mode in incoherent illumination. Figure 5 presents the consecutive aerial images and the SRR result. The camera moved along one direction, and the initial motion vector estimation is demonstrated in Fig. 6 with subpixel displacement. Figure 5(a) shows 50 low resolution subimages. The original grayscale images have a resolution of 480×512 pixels with a very high overlapping rate(85%), which were assumed to be captured approximately in the same image plane. A high spatial resolution image was synthesized as shown in Fig. 5(b). Compared with Fig. 5(a), Fig. 5(b) shows that the spatial resolution has been effectively improved. The ROI, boxed area in Fig. 5(e), shows much finer structures than the corresponding ROI boxed area in Figs. 5(c) and 5(d), which are results of different interpolation methods. It indicates that more higher spatial frequency components, beyond what the interpolation and deconvolution algorithms can achieve from one single image, have been recovered from the aliased low resolution images without introducing ringing artifacts.

Fig. 5. Aerial image sequence and the SRR result.
Fig. 6. Aerial camera initial motion vector estimation.

Figure 7(a) presents 30 low-resolution images captured with a camera in a slow panning motion under incoherent illumination. The original grayscale images have a resolution of 360×288 pixels with a very high overlapping rate. The initial estimation of camera motion vectors is shown in Fig. 8 with random and subpixel displacement for each image. Figure 7(b) shows the SA-based SRR result. Compared with Fig. 7(a), Fig. 7(b) is fairly sharp. For comparison of details, ROI 1 and ROI 2 in Fig. 7(a), two boxed areas in red, are presented in Figs. 7(c) and 7(d) and Figs. 7(f) and 7(g), which are results of different interpolation algorithms. The characters in ROI 1 and building structures detail in ROI 2 are clearly resolvable with significant resolution improvement, as shown in Figs. 7(e) and 7(h). It shows that the spatial resolution has been effectively improved with much finer structures and sharp edge without introducing ringing artifacts, what cannot be achieved with a interpolation or a deconvolution algorithm.

Fig. 7. Experiment with real video images: (a) presents 30 frames low images; (b) presents the SRR result; (c), (d), and (e) show, respectively, ROI 1 nearest interpolation image, bicubic interpolation, and SRR result; (f), (g), and (h) present, respectively, ROI two nearest interpolation image, bicubic interpolation, and SRR result.
Fig. 8. Camera initial motion vectors estimation.

In this Letter, an incoherent optical SA-based SRR approach is proposed experimentally and demonstrated for multiview low resolution images taken by incoherent imaging systems with simple and flexible configurations. Characterized by accurate superposition based on pixel-wise phase correlation and deconvolution constrained by a L1 norm regularization term involving a first order differential operator and a second order differential operator, the proposed method achieves a state-of-the-art result. It dramatically improves the image spatial resolution in experiments using controlled test images as well as two real image datasets. The spatial resolution gain factor is about 3.0 and displacement matrix estimation is as accurate as 1/30th pixel.

This work is supported by NSFC (Grant 41171450) and China Scholarship Council.

References

1.

J. N. Mait, R. Athale, and J. van der Gracht, Opt. Express 11, 2093 (2003). [CrossRef]

2.

G. O. Reynolds and D. J. Cronin, J. Opt. Soc. Am 60, 634 (1970).

3.

B. Katz and J. Rosen, Opt. Express 18, 962 (2010). [CrossRef]

4.

J. H. Massig, Opt. Lett. 27, 2179 (2002). [CrossRef]

5.

V. Micó, L. Granero, Z. Zalevsky, and J. García, J. Opt. A 11, 125408 (2009). [CrossRef]

6.

D. Rabb, D. Jameson, A. Stokes, and J. Stafford, Opt. Express 18, 10334 (2010). [CrossRef]

7.

G. W. Stroke, Opt. Commun. 1, 283 (1970). [CrossRef]

8.

I. J. Cox and C. J. R. Sheppard, J. Opt. Soc. Am. 3, 1152 (1986). [CrossRef]

9.

A. E. Tippie, A. Kumar, and J. R. Fienup, Opt. Express 19, 12027 (2011). [CrossRef]

10.

G. L. K. Morgan, J. G. Liu, and H. S. Yan, IEEE Trans. Geosci. Remote Sens. 35, 3456 (2010).

OCIS Codes
(100.0100) Image processing : Image processing
(100.6640) Image processing : Superresolution
(110.1758) Imaging systems : Computational imaging

ToC Category:
Image Processing

History
Original Manuscript: May 31, 2013
Revised Manuscript: July 31, 2013
Manuscript Accepted: August 2, 2013
Published: August 30, 2013

Virtual Issues
Vol. 8, Iss. 10 Virtual Journal for Biomedical Optics

Citation
Tao Sun, Jianguo Liu, Hongshi Yan, Gareth Morgan, and Wangli Chen, "Super-resolution reconstruction based on incoherent optical aperture synthesis," Opt. Lett. 38, 3471-3474 (2013)
http://www.opticsinfobase.org/ol/abstract.cfm?URI=ol-38-17-3471


Sort:  Author  |  Year  |  Journal  |  Reset  

References

  1. J. N. Mait, R. Athale, and J. van der Gracht, Opt. Express 11, 2093 (2003). [CrossRef]
  2. G. O. Reynolds and D. J. Cronin, J. Opt. Soc. Am 60, 634 (1970).
  3. B. Katz and J. Rosen, Opt. Express 18, 962 (2010). [CrossRef]
  4. J. H. Massig, Opt. Lett. 27, 2179 (2002). [CrossRef]
  5. V. Micó, L. Granero, Z. Zalevsky, and J. García, J. Opt. A 11, 125408 (2009). [CrossRef]
  6. D. Rabb, D. Jameson, A. Stokes, and J. Stafford, Opt. Express 18, 10334 (2010). [CrossRef]
  7. G. W. Stroke, Opt. Commun. 1, 283 (1970). [CrossRef]
  8. I. J. Cox and C. J. R. Sheppard, J. Opt. Soc. Am. 3, 1152 (1986). [CrossRef]
  9. A. E. Tippie, A. Kumar, and J. R. Fienup, Opt. Express 19, 12027 (2011). [CrossRef]
  10. G. L. K. Morgan, J. G. Liu, and H. S. Yan, IEEE Trans. Geosci. Remote Sens. 35, 3456 (2010).

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.


« Previous Article

OSA is a member of CrossRef.

CrossCheck Deposited