## Three dimensional visualization by photon counting computational Integral Imaging

Optics Express, Vol. 16, Issue 7, pp. 4426-4436 (2008)

http://dx.doi.org/10.1364/OE.16.004426

Acrobat PDF (867 KB)

### Abstract

In this paper, we present three dimensional (3D) object reconstruction using photon-counted elemental images acquired by a passive 3D Integral Imaging (II) system. The maximum likelihood (ML) estimator is derived to reconstruct the irradiance of the 3D scene pixels and the reliability of the estimator is described by confidence intervals. For applications in photon scarce environments, our proposed technique provides 3D reconstruction for better visualization as well as significant reduction in the computational burden and required bandwidth for transmission of integral images. The performance of the reconstruction is illustrated qualitatively and compared quantitatively with Peak to Signal to Noise Ratio (PSNR) criterion.

© 2008 Optical Society of America

## 1. Introduction

1. Y. Frauel, T. Naughton, O. Matoba, E. Tahajuerce, and B. Javidi, “Three Dimensional Imaging and Display Using Computational Holographic Imaging,” Proc. IEEE Journal **94**636–654, (2006). [CrossRef]

5. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE **94**, 591–607 (2006). [CrossRef]

8. Hoshino, F. Okano, H. Isono, and I. Yuyama, “Analysis of resolution limitation of integral photography,” J. Opt. Soc. Am. A **15**, 2059–2065 (1998). [CrossRef]

11. S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. **27**, 1144–1146 (2002). [CrossRef]

12. T. Okoshi, “Three-dimensional displays,” Proc. IEEE **68**, 548–564 (1980). [CrossRef]

13. Y. Igarishi, H. Murata, and M. Ueda, “3D display system using a computer-generated integral photograph,” Jpn. J. Appl. Phys. **17**, 1683–1684 (1978). [CrossRef]

6. H. Arimoto and B. Javidi, “Integrate three-dimensional imaging with computed reconstruction,” Opt. Lett. **26**, 157–159 (2001). [CrossRef]

11. S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. **27**, 1144–1146 (2002). [CrossRef]

14. Tavakoli, M. Danesh Panah, B. Javidi, and E. Watson, “Performance of 3D integral imaging with position uncertainty,” Opt. Express **15**, 11889–11902 (2007). [CrossRef] [PubMed]

20. Erdmann and K. J. Gabriel, “High resolution digital photography by use of a scanning microlens array,” Appl. Opt. **40**, 5592–5599 (2001). [CrossRef]

21. K. Nitta, R. Shogenji, S. Miyatake, and J. Tanida, “Image reconstruction for thin observation module by bound optics by using the iterative backprojection method,” Appl. Opt. **45**, 2893–2900 (2006). [CrossRef] [PubMed]

22. G. M. Morris, “Scene matching using photon-limited images,” J. Opt. Soc. Am. A. **1**, 482–488 (1984). [CrossRef]

27. P. A. Hiskett, G. S. Buller, A. Y. Loudon, J. M. Smith, I Gontijo, A. C. Walker, P. D. Townsend, and M. J. Robertson, “Performance and design of InGaAs/InP photodiodes for single-photon counting at 1.55 um,” Appl. Opt. **39**, 6818–6829 (2000). [CrossRef]

28. L. Duraffourg, J.-M. Merolla, J.-P. Goedgebuer, N. Butterlin, and W. Rhods, “Photon Counting in the 1540-nm Wavelength Region with a Germanium Avalanche photodiode,” IEEE J. Quantum Electron. **37**, 75–79 (2001). [CrossRef]

30. M. Guillaume, P. Melon, and P. Refregier, “Maximum-likelihood estimation of an astronomical image from a sequence at low photon levels,” J. Opt. Soc. Am. A. **15**, 2841–2848 (1998). [CrossRef]

32. E. Kolaczyk, “Bayesian multi-scale models for Poisson processes,” J. Amer. Stat. Assoc. **94**, 920–933 (1999). [CrossRef]

33. S. Yeom, B. Javidi, and E. Watson, “Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging,” Opt. Express **15**, 1513–1533 (2007). [CrossRef] [PubMed]

## 2. Three dimensional imaging and computational reconstruction

*S*and

_{x}*S*in

_{y}*x*and

*y*directions respectively in order to capture the elemental images. An array of these elemental images and the reconstruction at two planes where the objects are located is shown in Fig. 2.

*z*=

*z*from the lens of the imaging device, the magnification factor is

_{0}*M*=

_{0}*z*

_{0}/

*g*where g is the distance between pick up grid and image plane [see Fig.1]. Thus another approach for computational reconstruction is applied exclusive of magnification. It is illustrated in Fig. 2 that if the camera shifts by

*S*, the image of the object located at

_{x}*z*=

*z*

_{0}, shifts by

*S*/

_{x}*M*. Thus, the objects are reconstructed by shifting the elemental images opposite to the direction of the image shift due to camera motion. This is equivalent to shrinking the pickup grid with the factor of

_{0}*S*/

_{x}*M*and

_{0}*S*/

_{y}*M*in x and y direction respectively.

_{0}*k*and

*l*indicate the location of elemental image, I

_{kl}, in the pickup grid. Notice that the size of the objects reconstructed with this method is

*1*/

*M*

_{0}of their actual size and if the actual size is required the reconstructed plane should be magnified.

## 3. Photon-counting detection model

*hυ*, where

*h*is Plank’s constant (6.6262×10

^{-34}) and

*υ*is the mean frequency of the quasi monochromatic light source. Suppose that the energy incident on one pixel of the photosurface during the time Δ

*T*be

*E*, then the mean number of photons detected during this time interval can be expressed as(

_{x}*ηE*)/(

_{x}*hυ*), where

*η*≤1 is the sensor quantum efficiency and represents the average number of photoelectrons generated by each incident photon [31].

*T*, i.e.

*E*=

_{x}*I*Δ

_{x}*T*. Consequently the irradiance is proportional to the mean number of photoevents. On the other hand, the statistical properties of the photoevents show that the probability of the number of photons detected in a time interval, ΔT, smaller than the coherence time of the light, by the photosurface, smaller than the coherence area of the incident light, follows Poisson density function [31]. However, in practical cases of our interest, neither of the above conditions hold, i.e. the pixel area and exposure time are significantly larger than the coherence area and coherence time of the passive illumination. Nevertheless, for polarized thermal illumination with high degrees of coherence freedom, the degeneracy parameter approaches zero so the probability of detecting

*C*photons at pixel

*x*during the exposure time,

*C*, given the irradiance

_{x}*I*follows Poisson distribution as expressed in Eq. (2) [31].

_{x}*I*is the normalized irradiance at pixel

_{x}*x*such that

*N*is the total number of pixels of the image. In order to simulate a photon-counted image that has

_{T}*N*number of photons in average, a Poisson Random number

_{p}*C*with the mean parameter

_{x}*N*is generated,

_{p}I_{x}*C*|

_{x}*I*~

_{x}*Poisson*(

*N*). It is confirmed with Eq. (3) that the expected number of photons in the generated image is

_{p}I_{x}*N*. So the normalization is required to meet the constraint on the number of photons per image.

_{p}## 4. Three Dimensional reconstruction using photon-counted elemental images

*z*from the pick up grid, is captured at pixel

_{0}*p*≡(

*x,y*) of the first sensor. As discussed in section 2, the image of this object pixel appears on the elemental images periodically in the positions as follows:

*C*(

_{kl}*p*+Δ

*p*)} where

_{kl}*N*is constant and equal to the expected number of photons per elemental image. Our purpose is to estimate the irradiance based on these set of photon counts for each pixel of the 3D object. The likelihood estimation for the hypothesis

_{p}*α*, where

*α*∈(0,1) and is chosen according to the desired accuracy. It is shown in appendix A how one gets the following expression for the confidence intervals of the grayscale irradiance;

*KL*is the total number of elemental images, and

*z*

_{α/2}is the upper 100(

*α*/2)

_{%}point of the standard normal distribution and is known for each desired α as shown in Appendix A.

*N*, or number of elemental images, KL, confidence intervals shrink that is equivalent to the decrease of the estimation uncertainty. On the other hand, decrease of the number of photons can be compensated by increasing the number of elemental images.

_{p}*N*. Suppose two objects which are identical except in size, are being imaged separately while the number of photons per elemental image is the same for both of them. Intuitively, we expect more error for the irradiance estimation of the larger object. This fact is explained according to the confidential intervals as follows. The larger object has more number of pixels and consequently to meet the condition explained in section 3; i.e.

_{T}## 5. Experimental results

^{2}with and 16×16 elemental images are captured.

_{p}=10

^{3}and N

_{p}=10

^{5}in part (a) and (d) respectively while the corresponding irradiance image is captured from the center of the pick up grid. Clearly, recognizing the objects visually is not trivial in the 2D image with N

_{p}=10

^{3}. It is illustrated that the objects become recognizable after 3D computational reconstruction using all the gathered 2D photon-counted elemental images and the results can be compared qualitatively with the reconstruction using irradiance elemental images presented in Fig. 3. The movies of reconstruction from z=24cm to z=40cm is presented in Fig. 5 which shows that the objects become in focus at their corresponding distances even by using elemental images with very low number of photons.

*I*is the maximum irradiance of the gray scale elemental images.

_{max}*N*, the error decreases and as a result PSNR increases exponentially. The increase of the PSNR is confirmed with the confidence intervals, Eq. (8) which proves that the estimation error is proportional to

_{p}*N*).

_{p}^{-5}], so with

*N*=10

_{p}^{3}, the probability of counting more than one photon per pixel is less than 0.03%, i.e. Pr(

*C*>2|

*I*)<0.03 as calculated by Eq. (2). Therefore, we can assume to good approximation that the count for each pixel is either zero or one. Thus, with little error we treat photon-counted images with very low number of photons as binary images.

_{max}35. N. Otsu, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Trans. Syst. Man. Cybern **9**, 62–66 (1979). [CrossRef]

*N*=10

_{p}^{3}, shown in Fig. 5. It is interesting to note that using a 16×16 array of elemental images, 1000 photo-counts per elemental image (average), and a mean illumination wavelength of 500 nm, the total received energy would only be approximately 10

^{-16}J.

## 6. Conclusion

## Appendix A

*C*(

_{kl}*p*+Δ

*p*)} where

_{kl}*z*.

_{0}*α*∈(0,1), we claim that:

*z*

_{α/2}is the upper 100(

*α*/2)% point of the standard normal distribution that is obtained for each desired α as follows:

## References and links

1. | Y. Frauel, T. Naughton, O. Matoba, E. Tahajuerce, and B. Javidi, “Three Dimensional Imaging and Display Using Computational Holographic Imaging,” Proc. IEEE Journal |

2. | Javidi and F. Okano, eds., |

3. | M. Levoy and P. Hanrahan. “Light field rendering” Proc. ACM Siggarph, ACM Press , 31–42 (1996). |

4. | B. Javidi, S.-H. Hong, and O. Matoba, “Multi dimensional optical sensors and imaging systems,” Appl. Opt. |

5. | Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE |

6. | H. Arimoto and B. Javidi, “Integrate three-dimensional imaging with computed reconstruction,” Opt. Lett. |

7. | Stern and B. Javidi, “3-D computational synthetic aperture integral imaging (COMPSAII),” Opt. Express |

8. | Hoshino, F. Okano, H. Isono, and I. Yuyama, “Analysis of resolution limitation of integral photography,” J. Opt. Soc. Am. A |

9. | M. G. Lippmann, “Epreuves reversibles donnant la sensation du relief,” J. Phys. |

10. | H. E. Ives, “Optical properties of a Lippmann lenticuled sheet,” J. Opt. Soc. Am. |

11. | S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. |

12. | T. Okoshi, “Three-dimensional displays,” Proc. IEEE |

13. | Y. Igarishi, H. Murata, and M. Ueda, “3D display system using a computer-generated integral photograph,” Jpn. J. Appl. Phys. |

14. | Tavakoli, M. Danesh Panah, B. Javidi, and E. Watson, “Performance of 3D integral imaging with position uncertainty,” Opt. Express |

15. | Wilburn, N. Joshi, V. Vaish, A. Barth, A. Adams, M. Horowitz, and M. Levoy, “High performance imaging using large camera arrays,” Proc. of the ACM |

16. | M. Martínez-Corral, B. Javidi, R. Martínez-Cuenca, and G. Saavedra, “Integral imaging with improved depth of field by use of amplitude modulated microlens array,” Appl. Opt. |

17. | O. Matoba, E. Tajahuerce, and B. Javidi, “Real-time three-dimensional object recognition with multiple perspectives imaging,” Appl. Opt. |

18. | Y. Frauel and B. Javidi, “Digital three-dimensional image correlation by use of computer-reconstructed integral imaging,” Appl. Opt. |

19. | F. A. Sadjadi and A. Mahalanobis, “Target-adaptive polarimetric synthetic aperture radar target discrimination using maximum average correlation height filters,” Appl. Opt. |

20. | Erdmann and K. J. Gabriel, “High resolution digital photography by use of a scanning microlens array,” Appl. Opt. |

21. | K. Nitta, R. Shogenji, S. Miyatake, and J. Tanida, “Image reconstruction for thin observation module by bound optics by using the iterative backprojection method,” Appl. Opt. |

22. | G. M. Morris, “Scene matching using photon-limited images,” J. Opt. Soc. Am. A. |

23. | G. M. Morris, “Image correlation at low light levels: a computer simulation,” Appl. Opt. |

24. | Watson and G. M. Morris, “Comparison of infrared up conversion methods for photon-limited imaging,” J. Appl. Phys. |

25. | Watson and G. M. Morris, “Imaging thermal objects with photon-counting detector,” Appl. Opt. |

26. | D. Stucki, G. Ribordy, A. Stefanov, H. Zbinden, J. G. Rarity, and T. Wall, “Photon counting for quantum key distribution with Peltier cooled InGaAs/InP APDs,” J. Mod. Opt. |

27. | P. A. Hiskett, G. S. Buller, A. Y. Loudon, J. M. Smith, I Gontijo, A. C. Walker, P. D. Townsend, and M. J. Robertson, “Performance and design of InGaAs/InP photodiodes for single-photon counting at 1.55 um,” Appl. Opt. |

28. | L. Duraffourg, J.-M. Merolla, J.-P. Goedgebuer, N. Butterlin, and W. Rhods, “Photon Counting in the 1540-nm Wavelength Region with a Germanium Avalanche photodiode,” IEEE J. Quantum Electron. |

29. | K. Lange and R. Carson, “EM reconstruction algorithms for emission and transmission tomography,” Proc. IEEE J. Comput. Assist. Tomogr. |

30. | M. Guillaume, P. Melon, and P. Refregier, “Maximum-likelihood estimation of an astronomical image from a sequence at low photon levels,” J. Opt. Soc. Am. A. |

31. | J. W. Goodman, |

32. | E. Kolaczyk, “Bayesian multi-scale models for Poisson processes,” J. Amer. Stat. Assoc. |

33. | S. Yeom, B. Javidi, and E. Watson, “Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging,” Opt. Express |

34. | N. Mukhopadhyay, |

35. | N. Otsu, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Trans. Syst. Man. Cybern |

**OCIS Codes**

(030.5260) Coherence and statistical optics : Photon counting

(100.3010) Image processing : Image reconstruction techniques

(100.6890) Image processing : Three-dimensional image processing

(110.6880) Imaging systems : Three-dimensional image acquisition

**ToC Category:**

Image Processing

**History**

Original Manuscript: January 3, 2008

Revised Manuscript: March 8, 2008

Manuscript Accepted: March 9, 2008

Published: March 17, 2008

**Virtual Issues**

Vol. 3, Iss. 4 *Virtual Journal for Biomedical Optics*

**Citation**

Behnoosh Tavakoli, Bahram Javidi, and Edward Watson, "Three dimensional visualization by photon
counting computational Integral Imaging," Opt. Express **16**, 4426-4436 (2008)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-16-7-4426

Sort: Year | Journal | Reset

### References

- Y. Frauel, T. Naughton, O. Matoba, E. Tahajuerce, and B. Javidi, "Three Dimensional Imaging and Display Using Computational Holographic Imaging," Proc. IEEE Journal 94 636-654, (2006). [CrossRef]
- Javidi and F. Okano, eds., Three Dimensional Television, Video, and Display Technologies (Springer, Berlin, 2002).
- M. Levoy and P. Hanrahan. "Light field rendering" Proc. ACM Siggarph, ACM Press, 31-42 (1996).
- B. Javidi, S.-H. Hong, and O. Matoba, "Multi dimensional optical sensors and imaging systems," Appl. Opt. 45, 2986-2994 (2006). [CrossRef] [PubMed]
- Stern and B. Javidi, "Three-dimensional image sensing, visualization, and processing using integral imaging," Proc. IEEE 94, 591-607 (2006). [CrossRef]
- H. Arimoto and B. Javidi, "Integrate three-dimensional imaging with computed reconstruction," Opt. Lett. 26, 157-159 (2001). [CrossRef]
- Stern and B. Javidi, "3-D computational synthetic aperture integral imaging (COMPSAII)," Opt. Express 11, 2446-2451 (2003). [CrossRef] [PubMed]
- Hoshino, F. Okano, H. Isono, and I. Yuyama, "Analysis of resolution limitation of integral photography," J. Opt. Soc. Am. A 15, 2059-2065 (1998). [CrossRef]
- M. G. Lippmann, "Epreuves reversibles donnant la sensation du relief," J. Phys. 7, 821-825 (1908).
- H. E. Ives, "Optical properties of a Lippmann lenticuled sheet," J. Opt. Soc. Am. 21, 171-176 (1931). [CrossRef]
- S. Jang and B. Javidi, "Three-dimensional synthetic aperture integral imaging," Opt. Lett. 27, 1144-1146 (2002). [CrossRef]
- T. Okoshi, "Three-dimensional displays," Proc. IEEE 68, 548-564 (1980). [CrossRef]
- Y. Igarishi, H. Murata, and M. Ueda, "3D display system using a computer-generated integral photograph," Jpn. J. Appl. Phys. 17, 1683-1684 (1978). [CrossRef]
- Tavakoli, M. Danesh Panah, B. Javidi, and E. Watson, "Performance of 3D integral imaging with position uncertainty," Opt. Express 15, 11889-11902 (2007). [CrossRef] [PubMed]
- Wilburn, N. Joshi, V. Vaish, A. Barth, A. Adams, M. Horowitz, and M. Levoy, "High performance imaging using large camera arrays," Proc. of the ACM 24, 765-776 (2005).
- M. Martínez-Corral, B. Javidi, R. Martínez-Cuenca, and G. Saavedra, "Integral imaging with improved depth of field by use of amplitude modulated microlens array," Appl. Opt. 43, 5806-5813 (2004). [CrossRef] [PubMed]
- O. Matoba, E. Tajahuerce, and B. Javidi, "Real-time three-dimensional object recognition with multiple perspectives imaging," Appl. Opt. 40, 3318-3325 (2001). [CrossRef]
- Y. Frauel and B. Javidi, "Digital three-dimensional image correlation by use of computer-reconstructed integral imaging," Appl. Opt. 41, 5488-5496 (2002). [CrossRef] [PubMed]
- F. A. Sadjadi and A. Mahalanobis, "Target-adaptive polarimetric synthetic aperture radar target discrimination using maximum average correlation height filters," Appl. Opt. 45, 3063-3070 (2006). [CrossRef] [PubMed]
- Erdmann and K. J. Gabriel, "High resolution digital photography by use of a scanning microlens array," Appl. Opt. 40, 5592-5599 (2001). [CrossRef]
- K. Nitta, R. Shogenji, S. Miyatake, and J. Tanida, "Image reconstruction for thin observation module by bound optics by using the iterative backprojection method," Appl. Opt. 45, 2893-2900 (2006). [CrossRef] [PubMed]
- G. M. Morris, "Scene matching using photon-limited images," J. Opt. Soc. Am. A. 1, 482-488 (1984). [CrossRef]
- G. M. Morris, "Image correlation at low light levels: a computer simulation," Appl. Opt. 23, 3152-3159 (1984). [CrossRef] [PubMed]
- Watson and G. M. Morris, "Comparison of infrared up conversion methods for photon-limited imaging," J. Appl. Phys. 67, 6075-6084 (1990). [CrossRef]
- Watson and G. M. Morris, "Imaging thermal objects with photon-counting detector," Appl. Opt. 31, 4751-4757 (1992). [CrossRef] [PubMed]
- D. Stucki, G. Ribordy, A. Stefanov, H. Zbinden, J. G. Rarity, and T. Wall, "Photon counting for quantum key distribution with Peltier cooled InGaAs/InP APDs," J. Mod. Opt. 48, 1967-1981 (2001). [CrossRef]
- P. A. Hiskett, G. S. Buller, A. Y. Loudon, J. M. Smith, I Gontijo, A. C. Walker, P. D. Townsend, and M. J. Robertson, "Performance and design of InGaAs/InP photodiodes for single-photon counting at 1.55 um," Appl. Opt. 39, 6818-6829 (2000). [CrossRef]
- L. Duraffourg, J.-M. Merolla, J.-P. Goedgebuer, N. Butterlin, and W. Rhods, "Photon Counting in the 1540-nm Wavelength Region with a Germanium Avalanche photodiode," IEEE J. Quantum Electron. 37, 75-79 (2001). [CrossRef]
- K. Lange and R. Carson, "EM reconstruction algorithms for emission and transmission tomography," Proc. IEEE J. Comput. Assist. Tomogr. 8, 306-316 (1984).
- M. Guillaume, P. Melon, and P. Refregier, "Maximum-likelihood estimation of an astronomical image from a sequence at low photon levels," J. Opt. Soc. Am. A. 15, 2841-2848 (1998). [CrossRef]
- J. W. Goodman, Statistical optics (John Wiley & Sons, inc., 1985), Chap 9.
- E. Kolaczyk, "Bayesian multi-scale models for Poisson processes," J. Amer. Stat. Assoc. 94, 920-933 (1999). [CrossRef]
- S. Yeom, B. Javidi, and E. Watson, "Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging," Opt. Express 15, 1513-1533 (2007). [CrossRef] [PubMed]
- N. Mukhopadhyay, Probability and Statistical Inference (Marcel Dekker, Inc. New York, 2000).
- N. Otsu, "A Threshold Selection Method from Gray-Level Histograms," IEEE Trans. Syst. Man. Cybern 9, 62-66 (1979). [CrossRef]

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.