## 3D passive integral imaging using compressive sensing |

Optics Express, Vol. 20, Issue 24, pp. 26624-26635 (2012)

http://dx.doi.org/10.1364/OE.20.026624

Acrobat PDF (3918 KB)

### Abstract

Passive 3D sensing using integral imaging techniques has been well studied in the literature. It has been shown that a scene can be reconstructed at various depths using several 2D elemental images. This provides the ability to reconstruct objects in the presence of occlusions, and passively estimate their 3D profile. However, high resolution 2D elemental images are required for high quality 3D reconstruction. Compressive Sensing (CS) provides a way to dramatically reduce the amount of data that needs to be collected to form the elemental images, which in turn can reduce the storage and bandwidth requirements. In this paper, we explore the effects of CS in acquisition of the elemental images, and ultimately on passive 3D scene reconstruction and object recognition. Our experiments show that the performance of passive 3D sensing systems remains robust even when elemental images are recovered from very few compressive measurements.

© 2012 OSA

## 1. Introduction and background review

8. S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express **12**(3), 483–491 (2004). [CrossRef] [PubMed]

*x*and

*y*direction each with a different perspective of a 3D scene. To reconstruct the 3D scene optically from the captured 2D elemental images, the rays are reversely propagated from the elemental images through a display microlens array that is similar to the pickup microlens array. It does not require the special glasses to observe 3D images. In a synthetic aperture integral imaging (SAII) as shown in Fig. 1(a) , the 2D elemental images with different perspectives can be obtained through a camera array. For 3D computational reconstruction, back-projections of the elemental images through virtual pinhole array are used as depicted in Fig. 1(b), and averaged at the desired depth to reconstruct the image at that point. Thus 3D objects can be reconstructed at different depths in this manner.

9. E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory **52**(2), 489–509 (2006). [CrossRef]

20. Y. Rivenson and A. Stern, “Conditions for practicing compressive Fresnel holography,” Opt. Lett. **36**(17), 3365–3367 (2011). [CrossRef] [PubMed]

9. E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory **52**(2), 489–509 (2006). [CrossRef]

10. D. Donoho, “Compressed Sensing,” IEEE Trans. Inf. Theory **52**(4), 1289–1306 (2006). [CrossRef]

*m*=

*O*(

*N*log(

*N*/

*k*)) non-adaptive measurements which contain the information necessary to reconstruct any object with

*N*pixels at an accuracy comparable to that which would be possible if the

*k*most important coefficients of that object were directly observable. Thus one can have perfect (or near-perfect) image/signal reconstruction with far fewer samples than the Shannon sampling theorem implies provided the following assumptions are true:

- 1) There is structure in the image (i.e., the image is not random) ↔ there exist a basis for which the representation of the image of interest is sparse ↔ the image is compressible, and
- 2) The notion of “observing a sample” is generalized to include linear projections of the image in addition to the instantaneous image pixel intensity.

14. R. Muise, “Compressive imaging: An application,” SIAM J. Imaging Science **2**(4), 1255–1276 (2009). [CrossRef]

## 2. A review of compressive sensing

17. J. Tropp and A. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inf. Theory **53**(12), 4655–4666 (2007). [CrossRef]

**be a 64-element vector representing an 8 × 8 grayscale image**

*i**I*ordered lexigraphically. Furthermore, let

*A*be an

*n*× 64,

*n*= 64 encoding matrix used to subsample the input image. Then the goal of compressive sensing is to reconstruct

**given an n-element vector**

*i***by solvingfor**

*y***. As**

*i**A*has fewer rows than columns, this is an underdetermined system and therefore has infinitely many solutions.

**is sparse. This is generally not the case for typical imagery; however, we can transform the system into an equivalent sparse system by assuming we can represent the desired image**

*i***as =**

*i**ϕ*

**, where the matrix**

*c**ϕ*represents a compression basis (i.e. wavelet, DCT, or an over-complete dictionary).

*I*. We consider the image

*I*as split into 8 × 8 regions and reconstruct each one independently. Once each patch has been reconstructed, the regions are reassembled to form the final image.

**. Each 2 × 2 region represents a single element of the sensor. Therefore this mask will allow us to reconstruct an 8 × 8 region from only 16 pixels, resulting in 4x compression.**

*y**A*as a 16 × 64 matrix. For each small 2 × 2 block in the mask, we consider the 8 × 8 matrix

*A'*containing all 0s except where the corresponding pixel in the smaller block is on. This matrix

*A'*is then ordered lexigraphically, transposed, and set as a row of the resulting matrix

*A*. The

*A*matrix corresponding to the mask in Fig. 2(a) is shown in Fig. 2(b).

*A*, we must solve the underdetermined systemfor the coefficient vector,

**. Since this is an underdetermined system, there are infinitely many solutions (i.e. there are many 8 × 8 “images” which fit the data we collected). If we fit all of the underlying assumptions, the theory of compressive imaging states that the sparsest solution will be the correct image. Our 8 × 8 image is then approximated by**

*c***=**

*i**ϕ*

**. There are two details which remain, how do we choose our basis matrix**

*c**ϕ*and how do we find the sparsest solution?

*ϕ*, empirically on real world data to generate an overcomplete representation of 8 × 8 image patches that will guarantee the sparsity required by the compressive imaging theory.

16. M. Aharon, M. Elad, and A. M. Bruckstein, “The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. **54**(11), 4311–4322 (2006). [CrossRef]

*k*-means algorithm, the K-SVD algorithm, by Aharon, Elad, and Bruckstein, clusters the input chips and then creates basis vectors from them using singular-value decomposition (SVD). Each input chip is then reassigned to a cluster based upon its similarity to the set of basis vectors. The process iterates until the set of basis vectors can reconstruct the input chips within a specified amount of error.

17. J. Tropp and A. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inf. Theory **53**(12), 4655–4666 (2007). [CrossRef]

## 3. 3D integral imaging with compressive sensing

*ϕ*. The sensing matrix was generated as in Fig. 2 but with the higher compression sampling previously described. At 1/16 compression ratio, 4 measurements are collected for each 8 × 8 block and for the 1/64 compression ratio, only one measurement is collected for each 8 × 8 block. Each color channel in the elemental image was assumed to be sensed and reconstructed separately and independently. Figure 4 shows the results for a typical elemental image using OMP for the image reconstruction algorithm.

*I*, are recorded by SAII, sensed at different compression rates,

_{kl}8. S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express **12**(3), 483–491 (2004). [CrossRef] [PubMed]

*f*is the focal length of camera lens,

*p*,

_{x}*p*are the distance between cameras,

_{y}*N*,

_{x}*N*are the number of pixels for each elemental image in

_{y}*x*and

*y*directions,

*c*,

_{x}*c*are the image sensor size,

_{y}*z*is the reconstruction depth,

*S*,

_{x}*S*are the shifting pixel number for 3D reconstruction in

_{y}*x*and

*y*directions,

*k*,

*l*are the index of each elemental image, and

*O*(

*x*,

*y*) is the superposition matrix at the reconstruction plane, respectively.

## 4. Experimental results

*f*= 50mm, the distance between cameras is

*p*= 2mm, 10 × 10 elemental images are used to reconstruct 3D images, and each elemental image has 2184 × 1456 pixels. Figures 6(a) and 6(b) show the original elemental images with and without background. Yellow car and green car are located at

*z*= 340mm and

*z*= 440mm. Tree and background are positioned at

*z*= 610mm and

*z*= 800mm, respectively.

21. A. Van Nevel and A. Mahalanobis, “Comparative study of MACH filter variants using LADAR imagery,” Opt. Eng. **42**, 541–550 (2002). [CrossRef]

21. A. Van Nevel and A. Mahalanobis, “Comparative study of MACH filter variants using LADAR imagery,” Opt. Eng. **42**, 541–550 (2002). [CrossRef]

*M*(

*u*,

*v*),

*D*(

*u*,

*v*) and

*S*(

*u*,

*v*) are an object’s average image, power spectrum and spectral variance, respectively. We assume that the image is corrupted by additive noise with power spectrum

*C*(

*u*,

*v*). The parameters can be chosen to provide an optimal trade-off between the various performance metrics such as sharpness of the correlation peak (controlled by “

*α*” for

*D*), the stability of the peak value in the presence of distortions (controlled by “

*β*” for

*S*), and the response to additive noise (determined by “

*γ*” on

*C*). Since it is difficult to minimize all of these quantities simultaneously, the parameters

*α*,

*β*, and

*γ*allow us to arrive at the best overall tradeoff between the performance metrics. By carefully selecting

*α*and

*β*we can realize the smallest possible increase in correlation energy while minimizing the peak variance. In addition,

*γ*can be selected to yield minimize variance due to noise. To identify the 3D target, we correlate the MACH filter with the reconstructed image and measure the performance metrics.

## 5. Conclusion

## References and links

1. | G. Lippmann, “La Photographie Integrale,” C.-R. Acad. des Sci. |

2. | L. Yang, M. McCormick, and N. Davies, “Discussion of the optics of a new 3-D imaging system,” Appl. Opt. |

3. | F. Okano, J. Arai, K. Mitani, and M. Okui, “Real-time integral imaging based on extremely high resolution video system,” Proc. IEEE |

4. | A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE |

5. | R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, “Progress in 3-D multiperspective display by integral imaging,” Proc. IEEE |

6. | J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. |

7. | J.-S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. |

8. | S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express |

9. | E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory |

10. | D. Donoho, “Compressed Sensing,” IEEE Trans. Inf. Theory |

11. | M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “An Architecture for Compressive Imaging,” Proc. Int. Conf. on Image Processing (2006). |

12. | M. Duarte, M. Wakin, S. Sarvotham, D. Baron, and R. Baraniuk, “Distributed Compressed Sensing of Jointly Sparse Signals,” Asilomar Conf. on Signals, Systems, and Computers 1537–1541 (2005). |

13. | D. L. Donoho, Y. Tsaig, I. Drori, and J. L. Starck, “Sparse Solution of Underdetermined Linear Equations by Stagewise Orthogonal Matching Pursuit,” Dep. of Stat., Stanford Univ., Technical Report 2006–2, April (2006). |

14. | R. Muise, “Compressive imaging: An application,” SIAM J. Imaging Science |

15. | D. Bottisti and R. Muise, “Image exploitation from encoded measurements,” Proc. SPIE 8165 816518 (2011). |

16. | M. Aharon, M. Elad, and A. M. Bruckstein, “The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. |

17. | J. Tropp and A. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inf. Theory |

18. | Y. Rivenson and A. Stern, “Compressed imaging with a separable sensing operator,” IEEE Signal Process. Lett. |

19. | A. Stern, “Compressed imaging system with linear sensors,” Opt. Lett. |

20. | Y. Rivenson and A. Stern, “Conditions for practicing compressive Fresnel holography,” Opt. Lett. |

21. | A. Van Nevel and A. Mahalanobis, “Comparative study of MACH filter variants using LADAR imagery,” Opt. Eng. |

22. | E. Elhara, A. Stern, O. Hadar, and B. Javidi, “A hybrid compression method for integral images using discrete wavelet transform and discrete cosine transform,” IEEE JDT |

**OCIS Codes**

(110.0110) Imaging systems : Imaging systems

(110.6880) Imaging systems : Three-dimensional image acquisition

(150.6910) Machine vision : Three-dimensional sensing

**ToC Category:**

Imaging Systems

**History**

Original Manuscript: August 22, 2012

Revised Manuscript: October 29, 2012

Manuscript Accepted: November 1, 2012

Published: November 12, 2012

**Citation**

Myungjin Cho, Abhijit Mahalanobis, and Bahram Javidi, "3D passive integral imaging using compressive sensing," Opt. Express **20**, 26624-26635 (2012)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-20-24-26624

Sort: Year | Journal | Reset

### References

- G. Lippmann, “La Photographie Integrale,” C.-R. Acad. des Sci.146, 446–451 (1908).
- L. Yang, M. McCormick, and N. Davies, “Discussion of the optics of a new 3-D imaging system,” Appl. Opt.27(21), 4529–4534 (1988). [CrossRef] [PubMed]
- F. Okano, J. Arai, K. Mitani, and M. Okui, “Real-time integral imaging based on extremely high resolution video system,” Proc. IEEE94(3), 490–501 (2006). [CrossRef]
- A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE94(3), 591–607 (2006). [CrossRef]
- R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, “Progress in 3-D multiperspective display by integral imaging,” Proc. IEEE97(6), 1067–1077 (2009). [CrossRef]
- J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt.48(34), H77–H94 (2009). [CrossRef] [PubMed]
- J.-S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett.27(13), 1144–1146 (2002). [CrossRef] [PubMed]
- S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express12(3), 483–491 (2004). [CrossRef] [PubMed]
- E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory52(2), 489–509 (2006). [CrossRef]
- D. Donoho, “Compressed Sensing,” IEEE Trans. Inf. Theory52(4), 1289–1306 (2006). [CrossRef]
- M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “An Architecture for Compressive Imaging,” Proc. Int. Conf. on Image Processing (2006).
- M. Duarte, M. Wakin, S. Sarvotham, D. Baron, and R. Baraniuk, “Distributed Compressed Sensing of Jointly Sparse Signals,” Asilomar Conf. on Signals, Systems, and Computers 1537–1541 (2005).
- D. L. Donoho, Y. Tsaig, I. Drori, and J. L. Starck, “Sparse Solution of Underdetermined Linear Equations by Stagewise Orthogonal Matching Pursuit,” Dep. of Stat., Stanford Univ., Technical Report 2006–2, April (2006).
- R. Muise, “Compressive imaging: An application,” SIAM J. Imaging Science2(4), 1255–1276 (2009). [CrossRef]
- D. Bottisti and R. Muise, “Image exploitation from encoded measurements,” Proc. SPIE 8165816518 (2011).
- M. Aharon, M. Elad, and A. M. Bruckstein, “The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process.54(11), 4311–4322 (2006). [CrossRef]
- J. Tropp and A. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inf. Theory53(12), 4655–4666 (2007). [CrossRef]
- Y. Rivenson and A. Stern, “Compressed imaging with a separable sensing operator,” IEEE Signal Process. Lett.16(6), 449–452 (2009). [CrossRef]
- A. Stern, “Compressed imaging system with linear sensors,” Opt. Lett.32(21), 3077–3079 (2007). [CrossRef] [PubMed]
- Y. Rivenson and A. Stern, “Conditions for practicing compressive Fresnel holography,” Opt. Lett.36(17), 3365–3367 (2011). [CrossRef] [PubMed]
- A. Van Nevel and A. Mahalanobis, “Comparative study of MACH filter variants using LADAR imagery,” Opt. Eng.42, 541–550 (2002). [CrossRef]
- E. Elhara, A. Stern, O. Hadar, and B. Javidi, “A hybrid compression method for integral images using discrete wavelet transform and discrete cosine transform,” IEEE JDT3, 321–325 (2007).

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.