Diffraction theory reveals that an optical imaging system acts as a low-bandpass filter, sparing the low spatial frequencies of an object’s spectrum but cutting off high frequency information. Some of the high spatial-frequency signal information will be lost almost completely, due to the finite size of the lens apertures. Moreover, for digital imaging, the finite number of detectors in a sensor array (e.g., a CCD) is another factor that limits an imaging system’s spatial bandwidth. How to obtain a full spatial-frequency image is an important and fundamental research topic. Many attempts have been made to improve the resolving power of optical imaging systems, typically requiring large size lenses and complicated configurations that are costly and not always practical. However, recent advances in imaging technology have shown that combining the design of optics and optical electronics with digital image processing can lead to more effective imaging systems [1
1. J. N. Mait, R. Athale, and J. van der Gracht, Opt. Express 11, 2093 (2003). [CrossRef]
]. For instance, the well-known synthetic aperture (SA) concept has been tested as a promising approach using computational imaging to increase resolving power in many applications [2
2. G. O. Reynolds and D. J. Cronin, J. Opt. Soc. Am 60, 634 (1970).
], which can extend the spatial frequency of an imaging system beyond its normal limitations.
Triggered by the above SA experiments, we present a computational imaging approach for super-resolution reconstruction (SRR) by synthesizing a virtual large aperture based on an incoherent imaging system. The key idea here is to consider a sequence of low resolution images of the same scene captured by one lens but with minor shifts, as images taken separately by an array of identical lenses or a phase shifted CCD camera with successive exposures. Then, a high spatial resolution image will be produced by exploiting temporal freedom of the imaging system according to the information capacity theory of optical systems [8
8. I. J. Cox and C. J. R. Sheppard, J. Opt. Soc. Am. 3, 1152 (1986). [CrossRef]
]. In this SA mode, only a series of intensity images is required but benefiting from an identical lens and a flexible setup. The reconstruction image will have the higher spatial resolution of the synthesized larger aperture lens but meanwhile have the greater depth of field offered by the small aperture lens.
Superposition of the multiviews is one of the key steps for SA-based SRR, requiring accurate image registration, which has been studied in [9
9. A. E. Tippie, A. Kumar, and J. R. Fienup, Opt. Express 19, 12027 (2011). [CrossRef]
] with a secondary camera to determine the CCD position. The accuracy of the displacement estimation was close to
th pixel. In this Letter, however, the superposition is based on the displacement matrix with accuracy of
th pixel, and then sparse a priori
constraints deconvolution is applied for SRR. The approach can efficiently overcome degradation due to the limited size of the lens as well as the size of the CCD pixel. The final result of a SA generation extends the cutoff frequency of the imaging system much beyond that of an individual small lens aperture and thus yields a super-resolution image.
When the object illumination is incoherent, the imaging system can be expressed as
Thus, the image intensity
is a convolution of the incoherent intensity impulse response
with the ideal image intensity
. For digital imaging, the spatial integral effect is characterized by the system point spread function (PSF) that is composed of
. For simplicity, they are assumed to be modeled, respectively, by a 2D Gaussian function and a 2D rectangle function. Then the discrete digital images from different viewpoints,
, are represented by
. The mathematical discrete description of an imaging system integral sum is rewritten as Eq. (2
is the ideal image intensity.
illustrates a simplified scheme of the idea of this incoherent optical SA-based SRR method from a sequence of low resolution images, which are taken by a conventional imaging system under both spatially and temporally incoherent illumination. A camera with a small lens is assumed to shift over a relatively short range, with random subpixel increments, capturing consecutive images. The convolution operation in Eq. (2
), a spatial integration effect, makes the high spatial frequency components largely unresolvable. The aim of this study is thus to find an optimal estimation of the SRR image
when a set of images
is available from the shift operation, each of which only contains partial information. Superposition based on pixel-wise phase-correlation and sparse a priori
constraints deconvolution is performed for the incoherent SA-based SRR.
Fig. 1. Observation and SA reconstruction for conventional imaging system.
The CCD relative positions between the multiviews are calculated using a pixel-wise phase-correlation method [10
10. G. L. K. Morgan, J. G. Liu, and H. S. Yan, IEEE Trans. Geosci. Remote Sens. 35, 3456 (2010).
], which produces a subpixel accuracy displacement matrix. The phase correlation function is defined as
With Fourier inverse of the normalized cross-power spectrum of pairs of images, this method measures the motion vector from the phase shift directly. So it is relatively insensitive to frequency-dependent noise and illumination variations of the multiview images. As a result, a local displacement matrix with
th pixel accuracy can be achieved with pixel-wise phase correlation combined with 2D Gaussian fitting. This is an experimentally validated good solution for superposition of multiple low resolution images captured with minor shift-motion approximately in the same imaging plane, which makes the proposed SA-based SRR method superior to any interpolation algorithm.
Then, the a priori
constraints deconvolution is used to retrieve the high-frequency components by alleviating aliasing. Finding an optimization approximation
only from a set of low spatial observations
is an ill-posed linear inverse problem, which generally leads to a convex minimizing problem. Mathematically,
is a regularization term that incorporates a priori
knowledge of the desired high resolution image. For large underdetermined systems of linear equations, there are various regularization operators. However, in this Letter, the
norm sparse prior regularization term including a first order differential operator as well as a second order differential operator are taken into account in this convex minimizing problem, formulated as
is a first-order-differential regularization term. Without
, Eq. (5
) is equivalent to the total variation image restoration algorithm, which produces a sparse solution. It is especially effective on image deblurring with piecewise constant and piecewise linear image components, but otherwise staircase effects will be created, which is often the case for images of natural scenes. To alleviate the staircase effect,
for the second order differential operator is added as a smoothing regularity term in the proposed method, as shown in Eq. (5
). This modification improves smoothness while keeping image structure/edges sharp and alleviating effectively the ringing artifacts generally resulted from deconvolution algorithm. Equation (5
) is solved by the corresponding Euler–Lagrange equation with CG iteration algorithm. The viability of the proposed method has been demonstrated using standard simulation test images as well as real image datasets.
First, the 1951 United States Air Force (USAF) resolution target (AF target) was used for experimental validation and to quantify the improvement in image resolution. Figure 2(a)
shows a sequence of 25 low resolution images each with vertical or horizontal 0.2 pixel displacement increments between consecutive images. They are generated from the ground-truth high-resolution image through blurring by a Gaussian function with delta 1.3, a down-sample factor 5, and Gaussian additive noise with noise level 0.005.
Fig. 2. AF target images for SA-based SRR: (a) low resolution image sequence; (b) the SRR reconstruction result with the proposed method; (c) and (d) present center parts ROIs of one low resolution image in (a); (e) and (f) present center parts of ROIs of the SRR result (b) corresponding to the ROIs (c) and (d).
The bar elements (
) shown in Fig. 2(d)
, a boxed region of interest (ROI) boxed area in Fig. 2(a)
, are hardly resolvable. By contrast, the high-resolution image synthesized using the proposed SA-based SRR, shown in Fig. 2(b)
, is sharply enhanced. The texture is fairly sharp. Figures 2(e)
showing the corresponding ROIs to Figs. 2(c)
demonstrate the improvement of image spatial resolution. Figure 2(e)
shows that the bar elements (
) are easy resolvable.
presents 1D column profiles of horizontal bar elements of group 2 before and after the SA-based SRR. Figure 3(a)
shows five profiles (
) out of 25 low resolution images, which are completely unresolved. By contrast, the profiles after processing of superposition and SRR, shown, respectively, as blue dashed line and a solid gray line in Fig. 3(b)
, are clearly resolved with sharp edges. For the AF target, an improvement by (
elements) is equivalent to a resolution improvement by a linear factor of
. Note that the spatial resolution gain of the proposed method is about 2.8–3.2.
Fig. 3. 1D column profiles analysis of ROI (group 2, elements 2–6) in AF target.
shows the normalized power spectrum of the 25th low resolution image, the red bottom line, out of the AF target sequence, the SRR image in black, and the ground truth image in the blue. The blue dashed line and the black line are closely matched and both are far away from the red line at higher frequencies. This indicates that the high frequency components are substantially recovered by the SA-based SRR approach.
Fig. 4. Images normalization power spectrum.
A further test was carried out using aerial images captured with a vertical viewing mode in incoherent illumination. Figure 5
presents the consecutive aerial images and the SRR result. The camera moved along one direction, and the initial motion vector estimation is demonstrated in Fig. 6
with subpixel displacement. Figure 5(a)
shows 50 low resolution subimages. The original grayscale images have a resolution of
pixels with a very high overlapping rate(
), which were assumed to be captured approximately in the same image plane. A high spatial resolution image was synthesized as shown in Fig. 5(b)
. Compared with Fig. 5(a)
, Fig. 5(b)
shows that the spatial resolution has been effectively improved. The ROI, boxed area in Fig. 5(e)
, shows much finer structures than the corresponding ROI boxed area in Figs. 5(c)
, which are results of different interpolation methods. It indicates that more higher spatial frequency components, beyond what the interpolation and deconvolution algorithms can achieve from one single image, have been recovered from the aliased low resolution images without introducing ringing artifacts.
Fig. 5. Aerial image sequence and the SRR result.
Fig. 6. Aerial camera initial motion vector estimation.
presents 30 low-resolution images captured with a camera in a slow panning motion under incoherent illumination. The original grayscale images have a resolution of
pixels with a very high overlapping rate. The initial estimation of camera motion vectors is shown in Fig. 8
with random and subpixel displacement for each image. Figure 7(b)
shows the SA-based SRR result. Compared with Fig. 7(a)
, Fig. 7(b)
is fairly sharp. For comparison of details, ROI 1 and ROI 2 in Fig. 7(a)
, two boxed areas in red, are presented in Figs. 7(c)
and Figs. 7(f)
, which are results of different interpolation algorithms. The characters in ROI
and building structures detail in ROI
are clearly resolvable with significant resolution improvement, as shown in Figs. 7(e)
. It shows that the spatial resolution has been effectively improved with much finer structures and sharp edge without introducing ringing artifacts, what cannot be achieved with a interpolation or a deconvolution algorithm.
Fig. 7. Experiment with real video images: (a) presents 30 frames low images; (b) presents the SRR result; (c), (d), and (e) show, respectively, ROI 1 nearest interpolation image, bicubic interpolation, and SRR result; (f), (g), and (h) present, respectively, ROI two nearest interpolation image, bicubic interpolation, and SRR result.
Fig. 8. Camera initial motion vectors estimation.
In this Letter, an incoherent optical SA-based SRR approach is proposed experimentally and demonstrated for multiview low resolution images taken by incoherent imaging systems with simple and flexible configurations. Characterized by accurate superposition based on pixel-wise phase correlation and deconvolution constrained by a L1 norm regularization term involving a first order differential operator and a second order differential operator, the proposed method achieves a state-of-the-art result. It dramatically improves the image spatial resolution in experiments using controlled test images as well as two real image datasets. The spatial resolution gain factor is about 3.0 and displacement matrix estimation is as accurate as th pixel.