## Video rate spectral imaging using a coded aperture snapshot spectral imager

Optics Express, Vol. 17, Issue 8, pp. 6368-6388 (2009)

http://dx.doi.org/10.1364/OE.17.006368

Acrobat PDF (1911 KB)

### Abstract

We have previously reported on coded aperture snapshot spectral imagers (CASSI) that can capture a full frame spectral image in a snapshot. Here we describe the use of CASSI for spectral imaging of a dynamic scene at video rate. We describe significant advances in the design of the optical system, system calibration procedures and reconstruction method. The new optical system uses a double Amici prism to achieve an in-line, direct view configuration, resulting in a substantial improvement in image quality. We describe NeAREst, an algorithm for estimating the instantaneous three-dimensional spatio-spectral data cube from CASSI’s two-dimensional array of encoded and compressed measurements. We utilize CASSI’s snapshot ability to demonstrate a spectral image video of multi-colored candles with live flames captured at 30 frames per second.

© 2009 Optical Society of America

## 1. Introduction

*λ*at each spatial location (

*x*,

*y*) in an image. Thus, it acquires a three-dimensional (3D) data cube of spatio-spectral information, (

*x*,

*y*,

*λ*), about the scene being imaged. The knowledge of the spectrum at various points in the image can be useful in identifying the composition and structure of objects in the scene being observed by the imager.

1. J. Mooney, V. Vickers, and A. Brodzik, “High throughput hyperspectral infrared camera,” J. Opt. Soc. Am. A **14**, 2951–2961 (1997). [CrossRef]

2. C. Volin, B. Ford, M. Descour, J. Garcia, D. Wilson, P. Maker, and G. Bearman, “High-speed spectral imager for imaging transient fluorescent phenomena,” Appl. Opt. **37**, 8112–8119 (1998). [CrossRef]

3. K. Hege, D. O’Connell, W. Johnson, S. Basty, and E. Dereniak, “Hyperspectral imaging for astronomy and space surveillance,” Proc. SPIE **5159**,380–391 (2003). [CrossRef]

4. W. Johnson, D. Wilson, W. Fink, M. Humayun, and G. Bearman, “Snapshot hyperspectral imaging in ophthalmology,” J. Biomed. Opt. **12**, 0140,361–0140,367 (2007). [CrossRef]

5. M. Descour, C. Volin, E. Dereniak, K. Thorne, A. Schumacher, D. Wilson, and P. Maker, “Demonstration of a high-speed nonscanning imaging spectrometer,” Opt. Lett. **22**, 1271–1273 (1997). [CrossRef] [PubMed]

6. N. Gat, G. Scriven, J. Garman, M. D. Li, and J. Zhang, “Development of four-dimensional imaging spectrometers (4D-IS),” Proc. SPIE **6302**, 63020M (2006). [CrossRef]

7. A. Wagadarikar, R. John, R. Willett, and D. J. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. **47**, B44–B51 (2008). [CrossRef]

8. M. Gehm, R. John, D. J. Brady, R. Willett, and T. Schulz, “Single-shot compressive spectral imaging with a dual-disperser architecture,” Opt. Express **15**, 14,013–14,027 (2007). [CrossRef]

7. A. Wagadarikar, R. John, R. Willett, and D. J. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. **47**, B44–B51 (2008). [CrossRef]

## 2. Description of the direct view CASSI system

*μ*m on each side). This limits the spatial size of the reconstructed data cube to a maximum of 256 × 248 spatial elements. The bandpass filter (Omega Optical) limits the spectral range of the system to 450-650 nm. A relay lens (Edmund Optics stock part C45762) relays the image from the plane of the coded aperture to the CCD. Spectral dispersion is introduced using a custom designed (Shanghai Optics Inc.) double Amici prism. This prism consists of three separate prisms cemented together. The first and third prisms are made of SK2, a medium dispersion crown glass (low-medium

*n*and

_{d}*v*on the glass map), while the second prism is made of SF4, a higher dispersion flint glass (high

_{d}*n*and

_{d}*v*on the glass map). Based on the indices of refraction of each prism, Snell’s law can be used to choose angles of these prisms to ensure that rays corresponding to the center wavelength of 550 nm pass through the prism undeviated. Rays corresponding to adjacent wavelengths disperse to either side of the optical axis, resulting in a direct view prism. This configuration is useful because all the components of the system can be placed in a line, which makes system alignment much easier than that of the previous prototype. Furthermore, wavelength dependent anamorphic distortion is minimized because the path lengths of all wavelengths from the prism to the CCD are similar. Figure 1(b) shows an on-axis ray bundle passing through the custom designed double Amici prism. The CCD detector (AVT Marlin, Allied Vision Technologies) is an 8-bit camera with 9.9 × 9.9 (

_{d}*μ*m)

^{2}pixels and has its strongest response at 500 nm, with a relative response greater than 0.7 between 450-650 nm.

*μ*m at 450 nm, 3-8

*μ*m at 550 nm, and 3-8

*μ*m at 650 nm. The radius of the Airy disk ranges between 4.4-6.4

*μ*m over 450-650 nm. The system is not diffraction limited, as the spot sizes are larger than the size of the Airy disk, a result of the double Amici prism perturbing the imaging properties of the relay lens.

## 3. Light propagation model

*f*

_{0}(

*x*′′,

*y*′′,

*λ*). Then the power spectral density after the coded aperture is

*T*(

*x*,

*y*) is the transmission pattern printed on the coded aperture as shown in Fig. 2. The pattern is designed as an array of square features, with each feature being composed of 2×2 elements. Each element has the same size as that of a detector pixel, Δ. Let

*t*represent the binary value at the (

_{i,j}*i, j*)

*element, with a 1 representing a transmissive code element and a 0 representing an opaque code element. Then,*

^{th}*T*(

*x*,

*y*) can be described as

*h*(

*x*′ −

*x*,

*y*′ −

*y*,l) represents the shift-invariant optical impulse response of the relay lens and the double Amici prism.

*ϕ*(

*λ*) describes the dispersion induced by the double Amici prism. The detector array measures the intensity of incident light rather than the spectral density. As a result, the image on the detector array is the result of an integration process over the spectral range Λ. Furthermore, the detector array is spatially pixelated by the pixel function

*m*,

*n*)

*pixel is*

^{th}*= Ω*

_{i,j,m,n}

_{i,n,m,n}*δ*, so that

_{j,n}*x*′′ =

*x*′ −iΔ and

*y*′′ =

*y*′ −

*n*Δ,

*x*′′′ =

*x*−

*m*Δ and

*y*′′′ =

*y*−

*n*Δ,

*ϕ*(

*λ*) =

*αλ*, and letting

*f*= Ω

_{i,n,m+i}*. As a result, the array of detector measurements can be written as*

_{i,n,m}*f*′

*=*

_{m,n,k}*f*so that

_{m−k,n,k}**H**in (12) represents the CASSI system operator. It is a non-negative, binary matrix that maps voxels of the three-dimensional sampled and sheared data cube to pixels of the detector array. As the number of pixels on the detector used for the measurement is smaller than the number of aperture code elements multiplied by the number of spectral channels, the system of equations is under-determined.

*f*

_{0}(

*x*′ ,

*y*′ ,

*λ*), being filtered by the CASSI system transfer function, which ultimately determines the spatial and spectral resolution of the data cube estimate. The spatial resolution of the data cube depends on the point spread function,

*h*(

*x*′,

*x, y*′,

*y*,

*λ*), of the relay optics and double Amici prism optical transfer function, the pixel width, Δ, numerical reconstruction effects, and the size of a feature on the coded aperture. However, if we ignore reconstruction effects, the spatial resolution is approximately given by the width and height of the smallest feature on the coded aperture, or a 2 × 2 block of detector pixels (19.8

*μ*m × 19.8

*μ*m). Using larger features on the coded aperture will produce larger transmissive areas, but will also rely more heavily on numerical reconstruction to estimate the spatial content of objects in the data cube that are being imaged on to opaque areas of the coded aperture.

## 4. CASSI calibration

*h*(

*x*,

*x*′,

*y*,

*y*′,

*λ*) was shift invariant, the dispersion by the double Amici prism was linear, and that there was one-to-one mapping between elements of the aperture code to the detector pixels. However, in reality, the point spread function varies across the field, the dispersion is non-linear over CASSI’s spectral range, and there are subpixel misalignments between the aperture code features and detector pixels. As a result, the elements of

**H**are not necessarily binary as in (12). The quality of the data cube estimate generated by any numerical estimation algorithm strongly depends on how well this matrix is characterized.

### 4.1. Calibration process

**Illumination control**: Every effort was made to illuminate the aperture code with the light from the monochromator as uniformly as possible.**Shot-noise reduction**: At each wavelength, 10 CCD frames were captured and averaged to reduce the impact of shot and readout noise.**Background subtraction**: At each wavelength, 10 dark frames were captured at the same exposure time as the bright frames and averaged. The averaged dark frame at each wavelength was then subtracted from its corresponding bright calibration frame.**Exposure time adjustment**: To improve the signal-to-noise ratio (SNR) of the aperture code image at each wavelength, the exposure time at each wavelength was scaled so that the mean counts over the coded aperture in the CCD measurements at all wavelengths were similar.**Light source spectral intensity distribution**: Light from the source at each wavelength was measured with a photodiode having a known responsivity curve to obtain a calibration curve for the non-uniform spectral intensity of the light source.

### 4.2. Calibration results

*ϕ*(

*λ*) for the double Amici prism. The dispersion at the blue end of the spectrum is much greater than that at the red end. The black crosshairs in Fig. 7 identify the 33 wavelengths that define the centers of the 33 spectral channels.

## 5. Numerical estimation of spectral images

9. A. Wagadarikar, N. Pitsianis, X. Sun, and D. Brady, “Spectral Image Estimation for Coded Aperture Snapshot Spectral Imagers,” Proc. SPIE **7076**, 707602 (2008). [CrossRef]

### 5.1. A nested set of discretized systems

*m*,

*n*) detector pixel is the integration of the following components, each attributed to a particular wavelength

*λ*,

*ϕ*(

*λ*) is the dispersion at wavelength

*λ*and

*A*is the (

_{i,j}*i*,

*j*)

*virtual sub-aperture at a particular scale. For example, at the finest scale, the size of the sub-aperture is the same size as the detector pixels, while at a coarser scale, the size of this sub-aperture may be the same size as the smallest aperture code element, i.e. 2×2 detector pixels. In (13),*

^{th}*T*(

_{i,j}*x*′,

*y*′) ≥ 0 represents the effective code function at the (

*i*,

*j*)

*sub-aperture. We note, once again, that even when the sub-aperture is of the same size as the code elements, it is not necessarily equal to the binary-valued function in (2). The intensity measured at detector pixel (*

^{th}*n*,

*m*) can be described as

*P*denotes the spatial support of pixel (

_{n,m}*n*,

*m*) and Λ

*denotes the*

_{k}*k*spectral channel, which is determined by the calibration process and the scale at which the system of equations is being constructed.

^{th}*A*} as the sub-apertures and {Λ

_{ij}*} as the spectral channels, there is a system of equations relating a discrete 3D spatio-spectral data cube to the complete 2D array of CASSI measurements,*

_{k}*x*,

_{i}*y*,

_{j}*λ*) ∈

_{k}*A*× Λ

_{i,j}*, where*

_{k}*Q*(

*m*,

*n*;

*i*,

*j*,

*k*) by applying numerical quadrature for integration:

*w*and

_{m,n}*w*are the quadrature weights associated with the spatial discretization and

_{i,j}*w*is the quadrature weight associated with the spectral discretization.

_{k}*A*} and {Λ

_{ij}*}. In a CASSI system, pixelization at the detector is uniform on a 2D cartesian grid. We assume from now on that the discretization of the aperture at a given scale is also uniform on a cartesian grid. Under such conditions,*

_{k}*w*

_{m,n}*W*is constant over all (

_{i,j}*m*,

*n*;

*i*,

*j*).

### 5.2. Accommodation of CASSI system specifics

**Q**at each discretization scale. Through the calibration process, the spectral channels are determined so that the centers of each channel, at the finest scale, are separated by one column of detector pixels. Thus, the position of a channel corresponds to a fixed dispersion in terms of detector pixels relative to a fixed spectral channel. At a coarser scale, we can relate the dispersion of a spectral channel to the channel index

*k*in a linear fashion as

*α*represents the ratio of the dispersion range per spectral channel to the length of detector pixel. For example, if the centers of two neighboring spectral channels are two pixels apart in dispersion,

_{λ}*α*= 2. Although (17) is similar to the linear assumption in (9), it holds true for the case of non-linear dispersion under the condition that the channels are not equally partitioned in bandwidth. The non-linearity of dispersion is captured by the non-uniform spectral quadrature weights

_{λ}*w*across the spectral channels.

_{k}*c*=

_{k}*w*varies only with the channel index under the assumption of uniform spatial partitions.

_{m,n}w_{i,j}w_{k}*h*, of the imaging system from the coded aperture to the detector array, which consists of the relay lens and the double Amici prism. We approximate

*h*with two factors, one that is spatially shift invariant, and another that captures the spatial variation at each spectral channel. Specifically, the operator

*Q*is expressed in terms of the calibrated factors,

*h*

^{[k]}is assumed to be the spatially shift invariant part of the point spread function that depends on the imaging optics and the

*k*spectral channel.

^{th}*T*

_{i,j}^{[k]}is the effective grayscale code calibrated at the corresponding channel and compensates for the difference between

*h*(

*x*,

*y*,

*λ*) and

_{k}*h*

^{[k]}(

*x*,

*y*).

*A*} and {Λ

_{ij}*}. They differ from Equations (15) and (16) in that the calibration process is taken into account and the reconstruction process is considered as well. Our experiments show that reconstruction of a data cube using the calibrated operators is much superior to one that uses operators that do not account for the calibration process.*

_{k}### 5.3. The variational method for reconstruction

*f*(

*x*,

_{i}*y*,

_{j}*λ*) at a given scale to a measurement on a detector pixel

_{k}*g*for a calibrated CASSI. This system can be written in a matrix-vector expression,

_{n,m}**f**

*is a 2D slice of the 3D data cube associated with the*

_{k}*k*spectral channel, and

^{th}**Q**

*is the sub-matrix relating*

_{k}**f**

*to its contribution to the detector measurements. Across the different discretization scales, the system varies in the degree of freedom, with the degree being lower at a coarser scale. The matrix*

_{k}**Q**at a coarse scale has fewer columns than that at a finer scale, but the former is not necessarily a sub-matrix of the latter. NeAREst transforms the linear systems at each scale into a variational formulation. Specifically, Csiszar’s version of the Kullback-Leibler (KL) divergence is used as the objective function,

**g**. In (21), the second term is the algebraic difference between

**g**and

**Qf**, while the first term is the weighted difference in the bitwise representation, with

*g*as the weights. This seems similar to the regularization approaches we have used in the past, including the Gradient Projection for Sparse Reconstruction (GPSR) algorithm and the Two-step Iterative Shrinkage and Thresholding (TwIST) algorithm [9

_{n,m}9. A. Wagadarikar, N. Pitsianis, X. Sun, and D. Brady, “Spectral Image Estimation for Coded Aperture Snapshot Spectral Imagers,” Proc. SPIE **7076**, 707602 (2008). [CrossRef]

### 5.4. Efficient iterations by exploiting the nested structure of NeAREst

**Q**is invoked. Due to the non-uniform spectral weights,

**Q**

_{k}**f**

*are computed separately.*

_{k}*Q*

_{k}**f**

*involves a convolution with*

_{k}*h*

^{[k]}and a masking with

*T*

_{i,j}^{[k]}. Strictly speaking, there is another stage of propagation of light from the scene to the aperture as well, which can fortunately be merged with

*h*

^{[k]}.

**f**are explicitly formed in the image space, regardless of the intermediate representation of

**f**in an alternate basis with a multi-scale representation, such as a wavelet basis. Therefore, the computational cost per iteration is at least as much as that for the matrix-vector multiplication with

**Q**. In this aspect, NeAREst has the advantage that the matrix size is smaller at a coarser scale, lowering the computational cost per iteration at coarser levels. In comparison, a GPSR-like method accesses the largest matrix entirely at every iteration even when the number of representation scales is small. In addition, the computational cost per iteration increases substantially with an increase in the number of representation scales as well as with an increase in the variety of basis functions used.

### 5.5. Reconstruction of a spectral image video from CASSI measurements

### 5.6. Additional remarks

## 6. Experimental results & discussion

### 6.1. Quantitative validation with a non-imaging, reference spectrometer

## 7. Conclusion

## Acknowledgments

## References and links

1. | J. Mooney, V. Vickers, and A. Brodzik, “High throughput hyperspectral infrared camera,” J. Opt. Soc. Am. A |

2. | C. Volin, B. Ford, M. Descour, J. Garcia, D. Wilson, P. Maker, and G. Bearman, “High-speed spectral imager for imaging transient fluorescent phenomena,” Appl. Opt. |

3. | K. Hege, D. O’Connell, W. Johnson, S. Basty, and E. Dereniak, “Hyperspectral imaging for astronomy and space surveillance,” Proc. SPIE |

4. | W. Johnson, D. Wilson, W. Fink, M. Humayun, and G. Bearman, “Snapshot hyperspectral imaging in ophthalmology,” J. Biomed. Opt. |

5. | M. Descour, C. Volin, E. Dereniak, K. Thorne, A. Schumacher, D. Wilson, and P. Maker, “Demonstration of a high-speed nonscanning imaging spectrometer,” Opt. Lett. |

6. | N. Gat, G. Scriven, J. Garman, M. D. Li, and J. Zhang, “Development of four-dimensional imaging spectrometers (4D-IS),” Proc. SPIE |

7. | A. Wagadarikar, R. John, R. Willett, and D. J. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. |

8. | M. Gehm, R. John, D. J. Brady, R. Willett, and T. Schulz, “Single-shot compressive spectral imaging with a dual-disperser architecture,” Opt. Express |

9. | A. Wagadarikar, N. Pitsianis, X. Sun, and D. Brady, “Spectral Image Estimation for Coded Aperture Snapshot Spectral Imagers,” Proc. SPIE |

10. | X. Sun and N. Pitsianis, “Solving non-negative linear inverse problems with the NeAREst method,” Proc. SPIE |

**OCIS Codes**

(110.0110) Imaging systems : Imaging systems

(120.6200) Instrumentation, measurement, and metrology : Spectrometers and spectroscopic instrumentation

(300.6190) Spectroscopy : Spectrometers

**ToC Category:**

Imaging Systems

**History**

Original Manuscript: January 28, 2009

Revised Manuscript: March 19, 2009

Manuscript Accepted: March 20, 2009

Published: April 2, 2009

**Citation**

Ashwin A. Wagadarikar, Nikos P. Pitsianis, Xiaobai Sun, and David J. Brady, "Video rate spectral imaging using a coded aperture snapshot spectral imager," Opt. Express **17**, 6368-6388 (2009)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-17-8-6368

Sort: Year | Journal | Reset

### References

- J. Mooney, V. Vickers, and A. Brodzik, "High throughput hyperspectral infrared camera," J. Opt. Soc. Am. A 14, 2951-2961 (1997). [CrossRef]
- C. Volin, B. Ford, M. Descour, J. Garcia, D. Wilson, P. Maker, and G. Bearman, "High-speed spectral imager for imaging transient fluorescent phenomena," Appl. Opt. 37, 8112-8119 (1998). [CrossRef]
- K. Hege, D. O’Connell, W. Johnson, S. Basty, and E. Dereniak, "Hyperspectral imaging for astronomy and space surveillance," Proc. SPIE 5159, 380-391 (2003). [CrossRef]
- W. Johnson, D. Wilson, W. Fink, M. Humayun, and G. Bearman, "Snapshot hyperspectral imaging in ophthalmology," J. Biomed. Opt. 12, 0140,361-0140,367 (2007). [CrossRef]
- M. Descour, C. Volin, E. Dereniak, K. Thorne, A. Schumacher, D. Wilson, and P. Maker, "Demonstration of a high-speed nonscanning imaging spectrometer," Opt. Lett. 22, 1271-1273 (1997). [CrossRef] [PubMed]
- N. Gat, G. Scriven, J. Garman, M. D. Li, and J. Zhang, "Development of four-dimensional imaging spectrometers (4D-IS)," Proc. SPIE 6302, 63020M (2006). [CrossRef]
- A. Wagadarikar, R. John, R. Willett, and D. J. Brady, "Single disperser design for coded aperture snapshot spectral imaging," Appl. Opt. 47, B44-B51 (2008). [CrossRef]
- M. Gehm, R. John, D. J. Brady, R. Willett, and T. Schulz, "Single-shot compressive spectral imaging with a dual-disperser architecture," Opt. Express 15, 14,013-14,027 (2007). [CrossRef]
- A. Wagadarikar, N. Pitsianis, X. Sun, and D. Brady, "Spectral Image Estimation for Coded Aperture Snapshot Spectral Imagers," Proc. SPIE 7076, 707602 (2008). [CrossRef]
- X. Sun and N. Pitsianis, "Solving non-negative linear inverse problems with the NeAREst method," Proc. SPIE 7074, 7074E (2008).

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

### Figures

Fig. 1. |
Fig. 2. |
Fig. 3. |

Fig. 4. |
Fig. 5. |
Fig. 6. |

Fig. 7. |
Fig. 8. |
Fig. 9. |

Fig. 10. |
Fig. 11. |
Fig. 12. |

### Multimedia

Multimedia Files | Recommended Software |

» Media 1: MOV (3055 KB) | QuickTime |

» Media 2: MOV (3942 KB) | QuickTime |

« Previous Article | Next Article »

OSA is a member of CrossRef.