## Photon-counting passive 3D image sensing for reconstruction and recognition of partially occluded objects

Optics Express, Vol. 15, Issue 24, pp. 16189-16195 (2007)

http://dx.doi.org/10.1364/OE.15.016189

Acrobat PDF (693 KB)

### Abstract

In this paper, we discuss the reconstruction and the recognition of partially occluded objects using photon counting integral imaging (II). Irradiance scenes are numerically reconstructed for the reference target in three-dimensional (3D) space. Photon counting scenes are estimated for unknown input objects using maximum likelihood estimation (MLE) of Poisson parameter. We propose nonlinear matched filtering in 3D space to recognize partially occluded targets. The recognition performance is substantially improved from the nonlinear matched filtering of elemental images without 3D reconstruction. The discrimination capability is analyzed in terms of Fisher ratio (FR) and receiver operating characteristic (ROC) curves.

© 2007 Optical Society of America

## 1. Introduction

10. S. Yeom, B. Javidi, and E. Watson, “Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging,” Opt. Express15, 1513–1533 (2007), http://www.opticsinfobase.org/abstract.cfm?URI=oe-15-4-1513. [CrossRef] [PubMed]

13. H. Kwon and N. M. Nasrabadi, “Kernel RX-algorithm: a nonlinear anomaly detector for hyperspectral imagery,” IEEE Trans. on Geosci. Remote Sens. **43**, 388–397 (2005). [CrossRef]

2. G. M. Morris, “Scene matching using photon-limited images,” J. Opt. Soc. Am. A. **1**, 482–488 (1984). [CrossRef]

3. E. A. Watson and G. M. Morris, “Comparison of infrared upconversion methods for photon-limited imaging,” J. Appl. Phys. **67**, 6075–6084 (1990). [CrossRef]

4. D. Stucki, G. Ribordy, A. Stefanov, H. Zbinden, J. G. Rarity, and T. Wall, “Photon counting for quantum key distribution with Peltier cooled InGaAs/InP APDs,” J. Mod. Opt. **48**, 1967–1981 (2001). [CrossRef]

7. J. G. Rarity, T. E. Wall, K. D. Ridley, P. C. M. Owens, and P. R. Tapster, “Single-photon counting for the 1300–1600-nm range by use of Peltier-cooled and passively quenched InGaAs avalanche photodiodes,” Appl. Opt. **39**, 6746–6753 (2000). [CrossRef]

8. P. Refregier, F. Goudail, and G. Delyon, “Photon noise effect on detection in coherent active images,” Opt. Lett. **29**, 162–164 (2004). [CrossRef] [PubMed]

9. S. Yeom, B. Javidi, and E. Watson, “Photon counting passive 3D image sensing for automatic target recognition,” Opt. Express13, 9310–9330 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-23-9310. [CrossRef] [PubMed]

10. S. Yeom, B. Javidi, and E. Watson, “Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging,” Opt. Express15, 1513–1533 (2007), http://www.opticsinfobase.org/abstract.cfm?URI=oe-15-4-1513. [CrossRef] [PubMed]

22. S. H. Hong and B. Javidi, “Distortion-tolerant 3D recognition of occluded objects using computational integral imaging,” Opt. Express14, 12085–12095 (2006), http://www.opticsinfobase.org/abstract.cfm?URI=oe-14-25-12085. [CrossRef] [PubMed]

21. B. Javidi, R. Ponce-Diaz, and S-. H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. **31**, 1106–1108 (2006). [CrossRef] [PubMed]

22. S. H. Hong and B. Javidi, “Distortion-tolerant 3D recognition of occluded objects using computational integral imaging,” Opt. Express14, 12085–12095 (2006), http://www.opticsinfobase.org/abstract.cfm?URI=oe-14-25-12085. [CrossRef] [PubMed]

19. S. H. Hong, J. S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express12, 483–491 (2004), http://www.opticsinfobase.org/abstract.cfm?URI=oe-12-3-483. [CrossRef] [PubMed]

9. S. Yeom, B. Javidi, and E. Watson, “Photon counting passive 3D image sensing for automatic target recognition,” Opt. Express13, 9310–9330 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-23-9310. [CrossRef] [PubMed]

## 2. Computational reconstruction of integral imaging

### 2.1 Reconstruction of irradiance information

19. S. H. Hong, J. S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express12, 483–491 (2004), http://www.opticsinfobase.org/abstract.cfm?URI=oe-12-3-483. [CrossRef] [PubMed]

*A*in Fig. 1(b) is located at [

*i*,

_{A}*j*,

_{A}*z*] on the 3D object surface. The power density at the point

_{A}*A*is denoted as

*x*. Let

_{A}*x*be the captured irradiance corresponding to the point

_{n}*A*on the imaging plane of the

*n*-th micro-lenslet. The points are located at [

*i*,

_{n}*j*,

_{n}*g*],

_{A}*n*=1,…,

*N*and assumed to have a unit area; o

_{A}*is the center of the*

_{n}*n*-th micro-lenslet. Under the assumption that the distance (

*z*) between the point

_{A}*A*and the micro-lens array is large, the same power is transferred from

*x*and collected for

_{A}*x*and

_{A}*x*,

_{n}*n*=1,…,

*N*are approximately the same. Therefore,

_{A}*x*can be estimated as proportional to the average of

_{A}*x*.

_{n}### 2.2 Reconstruction of photon counting information

9. S. Yeom, B. Javidi, and E. Watson, “Photon counting passive 3D image sensing for automatic target recognition,” Opt. Express13, 9310–9330 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-23-9310. [CrossRef] [PubMed]

10. S. Yeom, B. Javidi, and E. Watson, “Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging,” Opt. Express15, 1513–1533 (2007), http://www.opticsinfobase.org/abstract.cfm?URI=oe-15-4-1513. [CrossRef] [PubMed]

*i*can be given by

*y*(

*i*) is the number of photons detected at pixel

*i*,

*N*is an expected number of photon-counts in the scene,

_{P}*x*(

*i*) is irradiance at pixel

*i*,

*N*is the total number of pixels in the scene. Let

_{T}*y*,

_{n}*n*=1,…,

*N*be the photon counts detected with the parameter

_{A}*λ*which is associated with

_{n}*x*. Let us consider

_{n}*λ*which is associated with

_{A}*x*. With the assumption that

_{A}*x*,

_{A}*λ*. Therefore, the joint probability density function of photon counts is calculated as

_{n}*λ*is

_{A}## 3. Recognition of occluded objects from the photon counting scene

*v*of the photon-counting image [9

*x*(

_{r}*i*) is the irradiance at pixel

*i*of the reference image of the target

*r*, and

*y*(

_{s}*i*) is the photon counts at pixel

*i*of the unknown input object

*s*. The first and second order characteristics of the nonlinear matched filtering is that the mean of

*C*(1) is constant and the variance is approximately proportional to 1/

_{rs}*N*[9

_{P}*x̂*(

_{r}*i*;

*z*) is the reconstructed irradiance information of the reference target at the depth

_{d}*z*;

_{d}*i*denotes the position of the virtual voxels on the reconstruction plane Ω

*, thus, voxels on the plane Ω*

_{d}*have the same longitudinal distance (*

_{d}*z*) from the micro-lens array;

_{d}*(*λ ^

_{s}*i*;

*z*) is the estimated Poisson parameter on the reconstruction plane Ω

_{d}*d*for the unknown input target. It is noted that the sailing factors between

*x*and

_{A}*x*,

_{n}*n*=1,…,

*N*are set at 1 without the loss of generality for the reconstruction and recognition purpose.

_{A}*d*,

*d*=1,…,

*N*, where

_{d}*N*is the total number of depth levels. The nonlinearity decided by

_{d}*v*in the denominator in Eq. (7) causes the same first and second order characteristics with Eq. (6) since

*x̂*(

_{r}*i*) and

*(*λ ^

_{s}*i*) are merely the linear combination of the irradiance and the photon counts in the elemental images.

*FR*) as the performance metric [3

3. E. A. Watson and G. M. Morris, “Comparison of infrared upconversion methods for photon-limited imaging,” J. Appl. Phys. **67**, 6075–6084 (1990). [CrossRef]

## 4. Experimental and simulation results

### 4.1 Reconstruction results

*mm*, and the focal length of each micro-lenslet is about 3

*mm*. The size of the cars is about 4.5

*cm*×2.5

*cm*×2.5

*cm*. To simulate the partial occlusion, a tree model is placed between the toy car and the micro-lens array. The distance between the imaging lens and the micro-lens array is 9.5

*cm*, and the distance between the micro-lens array and the occluding objects is 4~5

*cm*and the distance between the micro-lens array and the toy car is 9.5

*cm*. Figure 2 shows the elemental images of the white car for the reference image and the partially occluded white toy car for the true-class target and the yellow car for the false-class target. The size of the elemental image array is 1161×1419 pixels and the number of elemental images is 18×22. Figure 3 shows the central parts of elemental images in Fig. 2. Movies in Fig. 4 show that the reconstructed sectional images in the 3D space for the partially occluded true-class target. The reconstruction distance of Fig. 4(a) is 30~50

*mm*and the distance for Fig. 4(b) is 66~80

*mm*.

### 4.2 Recognition results

*v*=1. It is noted that only the gray level of irradiance in Fig. 3 is used for photo-counts simulation. Several values of

*N*, mean number of photo-counts in the entire image, are used to test the recognition performances. Figures 5(a) and 5(b) show the experimental results of nonlinear matched filtering for elemental images, that is without reconstruction [See Eq. (6)] and with reconstruction in 3D space [See Eq. (7)], respectively. The red solid line graph represents the sample mean for the true class target with occlusion and the blue dotted line graph is the sample mean for the false class target. Error bars stand for

_{P}*m*±

_{rs}*σ*where

_{rs}*m*and

_{rs}*σ*are the sample mean and the sample standard deviation of the matched filtering, respectively.

_{rs}*N*in Eq. (7) is set at 1 where the depth of reconstruction plane is 74

_{d}*mm*.

*N*=500 and

_{p}*N*=100, respectively. It is shown that the recognition performance of the proposed technique is substantially improved compared from the conventional method.

_{p}## 5. Conclusions

## Acknowledgements

## References and links

1. | J. W. Goodman, |

2. | G. M. Morris, “Scene matching using photon-limited images,” J. Opt. Soc. Am. A. |

3. | E. A. Watson and G. M. Morris, “Comparison of infrared upconversion methods for photon-limited imaging,” J. Appl. Phys. |

4. | D. Stucki, G. Ribordy, A. Stefanov, H. Zbinden, J. G. Rarity, and T. Wall, “Photon counting for quantum key distribution with Peltier cooled InGaAs/InP APDs,” J. Mod. Opt. |

5. | P. A. Hiskett, G. S. Buller, A. Y. Loudon, J. M. Smith, I Gontijo, A. C. Walker, P. D. Townsend, and M. J. Robertson, “Performance and design of InGaAs/InP photodiodes for single-photon counting at 1.55 um,” Appl. Opt. |

6. | L. Duraffourg, J.-M. Merolla, J.-P. Goedgebuer, N. Butterlin, and W. Rhods, “Photon counting in the 1540-nm wavelength region with a Germanium avalanche photodiode,” IEEE J. Quantum Electron. |

7. | J. G. Rarity, T. E. Wall, K. D. Ridley, P. C. M. Owens, and P. R. Tapster, “Single-photon counting for the 1300–1600-nm range by use of Peltier-cooled and passively quenched InGaAs avalanche photodiodes,” Appl. Opt. |

8. | P. Refregier, F. Goudail, and G. Delyon, “Photon noise effect on detection in coherent active images,” Opt. Lett. |

9. | S. Yeom, B. Javidi, and E. Watson, “Photon counting passive 3D image sensing for automatic target recognition,” Opt. Express13, 9310–9330 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-23-9310. [CrossRef] [PubMed] |

10. | S. Yeom, B. Javidi, and E. Watson, “Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging,” Opt. Express15, 1513–1533 (2007), http://www.opticsinfobase.org/abstract.cfm?URI=oe-15-4-1513. [CrossRef] [PubMed] |

11. | F. Sadjadi, ed., |

12. | A. Mahalanobis, R. R. Muise, S. R. Stanfill, and A. V. Nevel, “Design and application of quadratic correlation filters for target detection,” IEEE Trans. on Aerosp. Electron. Syst. |

13. | H. Kwon and N. M. Nasrabadi, “Kernel RX-algorithm: a nonlinear anomaly detector for hyperspectral imagery,” IEEE Trans. on Geosci. Remote Sens. |

14. | J.-S. Jang and B. Javidi, “Time-multiplexed integral imaging for 3D sensing and display,” Optics and Photonics News |

15. | J. Y. Son, V. V. Saveljev, J. S. Kim, S. S. Kim, and B. Javidi, “Viewing zones in three-dimensional imaging systems based on lenticular, parallax-barrier, and microlens-array plates,” Appl. Opt. |

16. | A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proceedings of the IEEE |

17. | Y. Frauel, E. Tajahuerce, O. Matoba, M.A. Castro, and B. Javidi, “Comparison of passive ranging integral imaging and active imaging digital holography for 3D object recognition,” Appl. Opt. |

18. | Raul Martinez-Cuenca, Amparo Pons, Genaro Saavedra, Manuel Martinez-Corral, and Bahram Javidi, “Optically-corrected elemental images for undistorted Integral image display,” Opt. Express14, 9657–9663 (2006), http://www.opticsinfobase.org/abstract.cfm?URI=oe-14-21-9657. [CrossRef] [PubMed] |

19. | S. H. Hong, J. S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express12, 483–491 (2004), http://www.opticsinfobase.org/abstract.cfm?URI=oe-12-3-483. [CrossRef] [PubMed] |

20. | J.-H. Park, J. Kim, and B. Lee, “Three-dimensional optical correlator using a sub-image array,” Opt. Express13, 5116–5126 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-13-5116. [CrossRef] [PubMed] |

21. | B. Javidi, R. Ponce-Diaz, and S-. H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. |

22. | S. H. Hong and B. Javidi, “Distortion-tolerant 3D recognition of occluded objects using computational integral imaging,” Opt. Express14, 12085–12095 (2006), http://www.opticsinfobase.org/abstract.cfm?URI=oe-14-25-12085. [CrossRef] [PubMed] |

23. | V. Vaish, R. Szeliski, C. L. Zitnick, S. B. Kang, and M. Levoy, “Reconstructing occluded surfaces using synthetic: stereo, focus and robust measures,” Proceedings of the IEEE CVPR’06 (2006). |

24. | S. M. Kay, |

**OCIS Codes**

(040.3780) Detectors : Low light level

(100.3010) Image processing : Image reconstruction techniques

(100.6890) Image processing : Three-dimensional image processing

(110.6880) Imaging systems : Three-dimensional image acquisition

(100.3008) Image processing : Image recognition, algorithms and filters

(100.4992) Image processing : Pattern, nonlinear correlators

**ToC Category:**

Image Processing

**History**

Original Manuscript: September 27, 2007

Revised Manuscript: November 12, 2007

Manuscript Accepted: November 13, 2007

Published: November 21, 2007

**Citation**

Seokwon Yeom, Bahram Javidi, Chae-wook Lee, and Edward Watson, "Photon-counting passive 3D image sensing for reconstruction and recognition of partially occluded objects," Opt. Express **15**, 16189-16195 (2007)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-15-24-16189

Sort: Year | Journal | Reset

### References

- J. W. Goodman, Statistical Optics, (Jonh wiley & Sons, inc., 1985), Chap 9.
- G. M. Morris, "Scene matching using photon-limited images," J. Opt. Soc. Am. A. 1, 482-488 (1984). [CrossRef]
- E. A. Watson and G. M. Morris, "Comparison of infrared upconversion methods for photon-limited imaging," J. Appl. Phys. 67, 6075-6084 (1990). [CrossRef]
- D. Stucki, G. Ribordy, A. Stefanov, H. Zbinden, J. G. Rarity, and T. Wall, "Photon counting for quantum key distribution with Peltier cooled InGaAs/InP APDs," J. Mod. Opt. 48, 1967-1981 (2001). [CrossRef]
- P. A. Hiskett, G. S. Buller, A. Y. Loudon, J. M. Smith, I Gontijo, A. C. Walker, P. D. Townsend, and M. J. Robertson, "Performance and design of InGaAs/InP photodiodes for single-photon counting at 1.55 um," Appl. Opt. 39, 6818-6829 (2000). [CrossRef]
- L. Duraffourg, J.-M. Merolla, J.-P. Goedgebuer, N. Butterlin, and W. Rhods, "Photon counting in the 1540-nm wavelength region with a Germanium avalanche photodiode," IEEE J. Quantum Electron. 37, 75-79 (2001). [CrossRef]
- J. G. Rarity, T. E. Wall, K. D. Ridley, P. C. M. Owens, and P. R. Tapster, "Single-photon counting for the 1300-1600-nm range by use of Peltier-cooled and passively quenched InGaAs avalanche photodiodes," Appl. Opt. 39, 6746-6753 (2000). [CrossRef]
- P. Refregier, F. Goudail, and G. Delyon, "Photon noise effect on detection in coherent active images," Opt. Lett. 29, 162-164 (2004). [CrossRef] [PubMed]
- S. Yeom, B. Javidi, and E. Watson, "Photon counting passive 3D image sensing for automatic target recognition," Opt. Express 13, 9310-9330 (2005). [CrossRef] [PubMed]
- S. Yeom, B. Javidi, and E. Watson, "Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging," Opt. Express 15, 1513-1533 (2007). [CrossRef] [PubMed]
- F. Sadjadi, ed., Selected Papers on Automatic Target Recognition, (SPIE-CDROM, 1999).
- A. Mahalanobis, R. R. Muise, S. R. Stanfill, and A. V. Nevel, "Design and application of quadratic correlation filters for target detection," IEEE Trans. on Aerosp. Electron. Syst. 40, 837-850 (2004). [CrossRef]
- H. Kwon and N. M. Nasrabadi, "Kernel RX-algorithm: a nonlinear anomaly detector for hyperspectral imagery," IEEE Trans. on Geosci.Remote Sens. 43, 388-397 (2005). [CrossRef]
- J.-S. Jang and B. Javidi, "Time-multiplexed integral imaging for 3D sensing and display," Optics and Photonics News 15, 36-43 (2004).
- J. Y. Son, V. V. Saveljev, J. S. Kim, S. S. Kim, and B. Javidi, "Viewing zones in three-dimensional imaging systems based on lenticular, parallax-barrier, and microlens-array plates," Appl. Opt. 43, 4985-4992 (2004). [CrossRef] [PubMed]
- A. Stern and B. Javidi, "Three-dimensional image sensing, visualization, and processing using integral imaging," Proceedings of the IEEE 94, 591- 607 (2006). [CrossRef]
- Y. Frauel, E. Tajahuerce, O. Matoba, M.A. Castro, and B. Javidi, "Comparison of passive ranging integral imaging and active imaging digital holography for 3D object recognition," Appl. Opt. 43, 452-462 (2004). [CrossRef] [PubMed]
- Raul Martinez-Cuenca, Amparo Pons, Genaro Saavedra, Manuel Martinez-Corral, and Bahram Javidi, "Optically-corrected elemental images for undistorted Integral image display," Opt. Express 14, 9657-9663 (2006). [CrossRef] [PubMed]
- S. H. Hong, J. S. Jang, and B. Javidi, "Three-dimensional volumetric object reconstruction using computational integral imaging," Opt. Express 12, 483-491 (2004). [CrossRef] [PubMed]
- J.-H. Park, J. Kim, and B. Lee, "Three-dimensional optical correlator using a sub-image array," Opt. Express 13, 5116-5126 (2005). [CrossRef] [PubMed]
- B. Javidi, R. Ponce-Diaz, and S-. H. Hong, "Three-dimensional recognition of occluded objects by using computational integral imaging," Opt. Lett. 31, 1106-1108 (2006). [CrossRef] [PubMed]
- S. H. Hong and B. Javidi, "Distortion-tolerant 3D recognition of occluded objects using computational integral imaging," Opt. Express 14, 12085-12095 (2006). [CrossRef] [PubMed]
- V. Vaish, R. Szeliski, C. L. Zitnick, S. B. Kang, and M. Levoy, "Reconstructing occluded surfaces using synthetic: stereo, focus and robust measures," Proceedings of the IEEE CVPR’06 (2006).
- S. M. Kay, Fundamentals of Statistical Signal Processing (Prentice Hall, New Jersey, 1993).

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.