## Three dimensional object recognition with photon counting imagery in the presence of noise |

Optics Express, Vol. 18, Issue 25, pp. 26450-26460 (2010)

http://dx.doi.org/10.1364/OE.18.026450

Acrobat PDF (1044 KB)

### Abstract

Three dimensional (3D) imaging systems have been recently suggested for passive sensing and recognition of objects in photon-starved environments where only a few photons are emitted or reflected from the object. In this paradigm, it is important to make optimal use of limited information carried by photons. We present a statistical framework for 3D passive object recognition in presence of noise. Since in quantum-limited regime, detector dark noise is present, our approach takes into account the effect of noise on information bearing photons. The model is tested when background noise and dark noise sources are present for identifying a target in a 3D scene. It is shown that reliable object recognition is possible in photon-counting domain. The results suggest that with proper translation of physical characteristics of the imaging system into the information processing algorithms, photon-counting imagery can be used for object classification.

© 2010 Optical Society of America

## 1. Introduction

2. J. R. Janesick, *Scientific Charge-Coupled Devices (SPIE Press Monograph Vol. PM83)* (SPIE Publications, 2001), 1st ed. [CrossRef]

3. F. Dubois, “Automatic spatial frequency selection algorithm for pattern recognition by correlation,” Appl. Opt. **32**, 4365–4371 (1993). [CrossRef] [PubMed]

7. H. Kwon and N. M. Nasrabadi, “Kernel matched subspace detectors for hyperspectral target detection.” IEEE Trans Pattern Anal Mach Intell **28**, 178–194 (2006). [CrossRef] [PubMed]

8. B. Javidi, R. Ponce-Díaz, and S.-H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. **31**, 1106–1108 (2006). [CrossRef] [PubMed]

9. O. Matoba, E. Tajahuerce, and B. Javidi, “Real-time three-dimensional object recognition with multiple perspectives imaging,” Appl. Opt. **40**, 3318–3325 (2001). [CrossRef]

10. G. M. Morris, “Scene matching using photon-limited images,” J. Opt. Soc. Am. A **1**, 482–488 (1984). [CrossRef]

11. E. A. Watson and G. M. Morris, “Imaging thermal objects with photon-counting detectors,” Appl. Opt. **31**, 4751–4757 (1992). [CrossRef] [PubMed]

12. S. Yeom, B. Javidi, and E. Watson, “Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging,” Opt. Express **15**, 1513–1533 (2007). [CrossRef] [PubMed]

13. S. Yeom, B. Javidi, and E. Watson, “Photon counting passive 3d image sensing for automatic target recognition,” Opt. Express **13**, 9310–9330 (2005). [CrossRef] [PubMed]

14. I. Moon and B. Javidi, “Three dimensional imaging and recognition using truncated photon counting model and parametric maximum likelihood estimator.” Opt Express **17**, 15709–15715 (2009). [CrossRef] [PubMed]

15. S. R. Narravula, M. M. Hayat, and B. Javidi, “Information theoretic approach for assessing image fidelity in photon-counting arrays,” Opt. Express **18**, 2449–2466 (2010). [CrossRef] [PubMed]

16. S. Yeom, B. Javidi, C. wook Lee, and E. Watson, “Photon-counting passive 3d image sensing for reconstruction and recognition of partially occluded objects,” Opt. Express **15**, 16189–16195 (2007). [CrossRef] [PubMed]

## 2. Multi-View Photon-Counting 3D Sensing

24. R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, “Progress in 3-d multiperspective display by integral imaging,” Proceedings of the IEEE **97**, 1067–1077 (2009). [CrossRef]

25. J.-S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. **27**, 1144–1146 (2002). [CrossRef]

26. S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express **12**, 483–491 (2004). [CrossRef] [PubMed]

*k*-th photon-counted elemental image be

*M*is the total number of pixels in each elemental image. We show the back-propagation of such elemental image at distance

*z*as

**R**

*↑*

_{k}*, which according to back-propagation based reconstruction can be written as: in which*

^{z}*p̃*=

*pf*/

*μz*is the pickup grid pitch (

*p*) normalized by sensor pixel pitch (

*μ*) and magnification (

*z*/

*f*) [see Fig. 1].

*z*can be reconstructed by integrating its associated image pixels on all sensors, that is for the

*i*-th object space point at distance

*z*one has

*z*=

*z*

_{0}represent the light field distribution in the object space at that particular plane, i.e.

**R**↑

*= {R*

^{z}*:*

^{i}*i*= 1 ...

*M*}. Similarly, the light field distribution in 3D object space can be reconstructed at

*Q*intermittent planes, such that:

### 2.1. Photon Counting Imagery Model

27. B. Javidi, P. Refregier, and P. Willett, “Optimum receiver design for pattern recognition with nonoverlapping target and scene noise.” Opt Lett **18**, 1660 (1993). [CrossRef] [PubMed]

28. E. A. Richards, “Limitations in optical imaging devices at low light levels,” Appl. Opt. **8**, 1999–2005 (1969). [CrossRef] [PubMed]

2. J. R. Janesick, *Scientific Charge-Coupled Devices (SPIE Press Monograph Vol. PM83)* (SPIE Publications, 2001), 1st ed. [CrossRef]

*n*, to each pixel such that statistics of dark counts is preserved. Therefore, the resulting irradiance (

_{d}*r*) incident on the

*i*-th pixel of

*k*-th sensor in a multi-view imaging system can be modeled as a combination of object (

*s*), background (

*n*) and dark-count equivalent (

_{B}*n*) irradiances as following: in which one dimensional scripted notation is used for brevity. Also,

_{d}*k*-th elemental image so that

*w*is different for each elemental image and is known a priori as part of reference object information. Likewise, background noise can be different in each elemental image due to varying sensor viewpoints. In addition,

_{k}*α*accounts for the potential difference between unknown object and reference irradiances. Throughout this paper, the pre-superscript, post-superscript and post-subscript for each symbol denote object class, pixel index and signal source, respectively. For example,

*i*-th pixel of a

*j*-th class object support function as seen from

*k*-th elemental image.

## 3. Pattern Recognition with 3D Photon-Counting Imagery

*H*. Given the photon-counting imagery, the maximum-likelihood (ML) decision criterion is to choose between one of the hypothesis such that an objective function (e.g. probability of error) is minimized. In a binary classification problem,

_{j}*H*

_{1}is selected if object 1 is more likely to have produced the observed photon-counting data. For mathematical brevity, we assume that all object classes are equally likely and that the cost of error is the same for all misclassifications. Given a photon counting dataset,

**R**, a convenient way to use ML decision theory is to calculate the likelihood ratio,

*ℓ*(.), between the two classes and make decisions based on the outcome [30]: where ℒ(.) denotes the likelihood function.

*H*can be expanded as: where

_{j}*i*of

*k*-th elemental image. The innermost product in Eq. (7) is on

*M*pixels of each elemental image, the second product is on set of

*K*elemental images and the outermost product is on

*Q*reconstruction planes.

*k*-th elemental image, the conditional density of the number of counts for

*i*-th pixel,

*j*denoting the class hypothesis, i.e. where 𝒫(.) is the Poisson transformation following the form of Eq. (4). Combining Eqs. (8) and (9) and substituting in Eq. (7), and taking the logarithm yields: where

*ĩ*=

*i*–

*p̃k*for conciseness. The log likelihood in Eq. (10) can be calculated based on the a priori reference object normalized irradiance (

*s*and

*w*), sensor’s characteristic dark count rate (

*n*) and total counts registered at each pixel of the sensor (r). Note that unlike correlation based techniques, the likelihood in Eq. (10) would penalize a high energy background if the object is not present or expected in the scene, i.e. where

_{d}*s*≪ 1 but

^{i}*r*> 0. This reduces the false positive rate and improves the recognition performance.

^{i}*α*to find its estimate

*α̂*: From Eq. (11) the solution for

*α̂*is that of a high-order polynomial which does not yield a closed form expression. However, for small enough dark noise,

*n*≪

_{d}*s*,

*α̂*can be simply found as: which can be calculated separately and substituted for

*α*in Eq. (10). If

*n*≥

_{d}*s*, one can calculate

*α̂*by applying numerical non-linear solvers, such as Newton’s method, on Eq. (11). Note that only pixels with nonzero counts need to be taken into account to find a solution for

*α̂*in Eq. (11). In photon counting domain, only a small number of pixels are expected to report counts, which in turn simplifies calculation of

*α̂*.

*j*= 1, 2,...,

*J*. In case of the binary classification between two distinct objects, the labeling strategy based on the likelihood ratio in Eq. (6) can be rewritten with log-likelihood values. The resulting decision rule is:

*O*(

*n*) where

*n*represents the total number of pixels that belong to the object in all images. Note that in contrast to conventional intensity images that have a large number of 8 or 12 bit pixels, photon counting images are the outcome of a Poisson process [Eq. (4)] with very low number of incident photons. Such images typically require only 1 or 2 bit pixel elements for detection. This results in substantially reduced number of non-zero pixels in the image and directly translates into reduced storage and computational requirements.

## 4. Experimental Results

### 4.1. Multi-View 3D Imaging

*p*= 16 mm and

_{x}*p*= 10 mm respectively and the focal plane size, i.e. sensor size, is 24×36 mm with pixel size of

_{y}*μ*= 10

*μ*m. The imaging optics has a fixed focal length of 24 mm with

*f*# = 5.4. For each elemental image the object support

**w**

*is extracted by thresholding. The reference objects are imaged under controlled illumination against a dark background. Note that the unknown input scenes need not to be imaged in the same illumination condition and can include background noise of arbitrary pattern and brightness [see Eq. (11)]. As unknown input objects, the same two objects are presented to the imaging system in a different pose (comparing to reference objects) with additional pine-tree foliage background. Figure 3 illustrates 16 (out of 121) views of one of the objects.*

_{k}**R**, as described in Section 2. The volumetric reconstruction for both photon-counting imagery and reference objects’ 3D images are generated using Eq. (2) based on which Eq. (10) is used to find the log likelihood with respect to both object hypotheses.

### 4.2. Recognition Performance

*n*= 0, and the reconstructed photon-counting 3D image of the object only contains the photo-counts. In the second scenario, sensors are assumed to have a fixed dark count rate as a result of which the total dark counts (

_{d}*N*) increase proportional to the total photon counts (

_{dc}*N*). In both cases, the illumination conditions are similar, thus

_{ph}*α̂*is set to 1.

*H*

_{1}, and false,

*H*

_{2}class reference objects.

**R**|

*H*

_{1}) – log ℒ(

**R**|

*H*

_{2}) is used for classification. This quantity, along with its standard deviation, is plotted in Fig. 5(a) for various values of total photon count,

*N*.

_{ph}*n*, is chosen such that the expected number of dark counts combined for all elemental images,

_{d}*N*, is always 27 times more than that of photon-counts, i.e

_{dc}*N*= 27

_{dc}*N*. This results in a constant ratio of object photons to dark counts equal to 0.037 that is preserved in all experiments. The resulting log likelihood difference and its standard deviation is plotted in Fig. 5(b).

_{ph}*N*in both scenarios.

_{ph}## 5. Conclusion

## Acknowledgments

## References and links

1. | J. W. Goodman, |

2. | J. R. Janesick, |

3. | F. Dubois, “Automatic spatial frequency selection algorithm for pattern recognition by correlation,” Appl. Opt. |

4. | F. Sadjadi, ed., |

5. | V. Page, F. Goudail, and P. Refregier, “Improved robustness of target location in nonhomogeneous backgrounds by use of the maximum-likelihood ratio test location algorithm,” Opt Lett |

6. | A. Mahalanobis, R. R. Muise, and S. R. Stanfill, “Quadratic correlation filter design methodology for target detection and surveillance applications,” Appl Opt |

7. | H. Kwon and N. M. Nasrabadi, “Kernel matched subspace detectors for hyperspectral target detection.” IEEE Trans Pattern Anal Mach Intell |

8. | B. Javidi, R. Ponce-Díaz, and S.-H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. |

9. | O. Matoba, E. Tajahuerce, and B. Javidi, “Real-time three-dimensional object recognition with multiple perspectives imaging,” Appl. Opt. |

10. | G. M. Morris, “Scene matching using photon-limited images,” J. Opt. Soc. Am. A |

11. | E. A. Watson and G. M. Morris, “Imaging thermal objects with photon-counting detectors,” Appl. Opt. |

12. | S. Yeom, B. Javidi, and E. Watson, “Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging,” Opt. Express |

13. | S. Yeom, B. Javidi, and E. Watson, “Photon counting passive 3d image sensing for automatic target recognition,” Opt. Express |

14. | I. Moon and B. Javidi, “Three dimensional imaging and recognition using truncated photon counting model and parametric maximum likelihood estimator.” Opt Express |

15. | S. R. Narravula, M. M. Hayat, and B. Javidi, “Information theoretic approach for assessing image fidelity in photon-counting arrays,” Opt. Express |

16. | S. Yeom, B. Javidi, C. wook Lee, and E. Watson, “Photon-counting passive 3d image sensing for reconstruction and recognition of partially occluded objects,” Opt. Express |

17. | M. G. Lippmann, “La photographie intégrale,,” Comptes-rendus de l’Académie des Sciences |

18. | C. B. Burckhardt, “Optimum parameters and resolution limitation of integral photography,” J. Opt. Soc. Amer |

19. | T. Okoshi, “Three-dimensional displays,” Proceedings of the IEEE |

20. | M. C. Forman, N. Davies, and M. McCormick, “Continuous parallax in discrete pixelated integral three-dimensional displays.” J Opt Soc Am A Opt Image Sci Vis |

21. | A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proceedings of the IEEE |

22. | F. Okano, J. Arai, K. Mitani, and M. Okui, “Real-time integral imaging based on extremely high resolution video system,” Proceedings of the IEEE |

23. | B. Javidi, F. Okano, and J.-Y. Son, eds., |

24. | R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, “Progress in 3-d multiperspective display by integral imaging,” Proceedings of the IEEE |

25. | J.-S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. |

26. | S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express |

27. | B. Javidi, P. Refregier, and P. Willett, “Optimum receiver design for pattern recognition with nonoverlapping target and scene noise.” Opt Lett |

28. | E. A. Richards, “Limitations in optical imaging devices at low light levels,” Appl. Opt. |

29. | P. Rfrgier, |

30. | A. Papoulis and S. Pillai, |

31. | R. J. Schalkoff, |

**OCIS Codes**

(030.5260) Coherence and statistical optics : Photon counting

(100.6890) Image processing : Three-dimensional image processing

(110.6880) Imaging systems : Three-dimensional image acquisition

**ToC Category:**

Image Processing

**History**

Original Manuscript: August 16, 2010

Revised Manuscript: November 5, 2010

Manuscript Accepted: November 17, 2010

Published: December 2, 2010

**Citation**

Mehdi DaneshPanah, Bahram Javidi, and Edward A. Watson, "Three dimensional object recognition with photon counting imagery in the presence of noise," Opt. Express **18**, 26450-26460 (2010)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-18-25-26450

Sort: Year | Journal | Reset

### References

- J. W. Goodman, Statistical Optics (Wiley-Interscience, 1985), Wiley classics ed.
- J. R. Janesick, Scientific Charge-Coupled Devices (SPIE Press Monograph Vol. PM83) (SPIE Publications, 2001), 1st ed. [CrossRef]
- F. Dubois, "Automatic spatial frequency selection algorithm for pattern recognition by correlation," Appl. Opt. 32, 4365-4371 (1993). [CrossRef] [PubMed]
- F. Sadjadi, ed., Selected Papers on Automatic Target Recognition (SPIE-CDROM, 1999).
- V. Page, F. Goudail, and P. Refregier, "Improved robustness of target location in nonhomogeneous backgrounds by use of the maximum-likelihood ratio test location algorithm," Opt. Lett. 24, 1383-1385 (1999). [CrossRef]
- A. Mahalanobis, R. R. Muise, and S. R. Stanfill, "Quadratic correlation filter design methodology for target detection and surveillance applications," Appl. Opt. 43, 5198-5205 (2004). [CrossRef] [PubMed]
- H. Kwon, and N. M. Nasrabadi, "Kernel matched subspace detectors for hyperspectral target detection," IEEE Trans. Pattern Anal. Mach. Intell. 28, 178-194 (2006). [CrossRef] [PubMed]
- B. Javidi, R. Ponce-Díaz, and S.-H. Hong, "Three-dimensional recognition of occluded objects by using computational integral imaging," Opt. Lett. 31, 1106-1108 (2006). [CrossRef] [PubMed]
- O. Matoba, E. Tajahuerce, and B. Javidi, "Real-time three-dimensional object recognition with multiple perspectives imaging," Appl. Opt. 40, 3318-3325 (2001). [CrossRef]
- G. M. Morris, "Scene matching using photon-limited images," J. Opt. Soc. Am. A 1, 482-488 (1984). [CrossRef]
- E. A. Watson, and G. M. Morris, "Imaging thermal objects with photon-counting detectors," Appl. Opt. 31, 4751-4757 (1992). [CrossRef] [PubMed]
- S. Yeom, B. Javidi, and E. Watson, "Three-dimensional distortion-tolerant object recognition using photon counting integral imaging," Opt. Express 15, 1513-1533 (2007). [CrossRef] [PubMed]
- S. Yeom, B. Javidi, and E. Watson, "Photon counting passive 3d image sensing for automatic target recognition," Opt. Express 13, 9310-9330 (2005). [CrossRef] [PubMed]
- I. Moon, and B. Javidi, "Three dimensional imaging and recognition using truncated photon counting model and parametric maximum likelihood estimator," Opt. Express 17, 15709-15715 (2009). [CrossRef] [PubMed]
- S. R. Narravula, M. M. Hayat, and B. Javidi, "Information theoretic approach for assessing image fidelity in photon-counting arrays," Opt. Express 18, 2449-2466 (2010). [CrossRef] [PubMed]
- S. Yeom, B. Javidi, C. W. Lee, and E. Watson, "Photon-counting passive 3d image sensing for reconstruction and recognition of partially occluded objects," Opt. Express 15, 16189-16195 (2007). [CrossRef] [PubMed]
- M. G. Lippmann, "La photographie intégrale," Comptes-rendus de l’Académie des Sciences 146, 446-451 (1908).
- C. B. Burckhardt, "Optimum parameters and resolution limitation of integral photography," J. Opt. Soc. Am. 58, 71-76 (1968). [CrossRef]
- T. Okoshi, "Three-dimensional displays," Proc. IEEE 68, 548-564 (1980). [CrossRef]
- M. C. Forman, N. Davies, and M. McCormick, "Continuous parallax in discrete pixilated integral three-dimensional displays," J. Opt. Soc. Am. A 20, 411-420 (2003). [CrossRef] [PubMed]
- A. Stern, and B. Javidi, "Three-dimensional image sensing, visualization, and processing using integral imaging," Proc. IEEE 94, 591-607 (2006). [CrossRef]
- F. Okano, J. Arai, K. Mitani, and M. Okui, "Real-time integral imaging based on extremely high resolution video system," Proc. IEEE 94, 490-501 (2006). [CrossRef]
- B. Javidi, F. Okano, and J.-Y. Son, eds., Three-Dimensional Imaging, Visualization, and Display (Signals and Communication Technology) (Springer, 2008), 1st ed.
- R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, "Progress in 3-d multiperspective display by integral imaging," Proc. IEEE 97, 1067-1077 (2009). [CrossRef]
- J.-S. Jang, and B. Javidi, "Three-dimensional synthetic aperture integral imaging," Opt. Lett. 27, 1144-1146 (2002). [CrossRef]
- S.-H. Hong, J.-S. Jang, and B. Javidi, "Three-dimensional volumetric object reconstruction using computational integral imaging," Opt. Express 12, 483-491 (2004). [CrossRef] [PubMed]
- B. Javidi, P. Refregier, and P. Willett, "Optimum receiver design for pattern recognition with nonoverlapping target and scene noise," Opt. Lett. 18, 1660 (1993). [CrossRef] [PubMed]
- E. A. Richards, "Limitations in optical imaging devices at low light levels," Appl. Opt. 8, 1999-2005 (1969). [CrossRef] [PubMed]
- P. Refregier, Noise Theory and Application to Physics (Springer, 2004), 1st ed.
- A. Papoulis, and S. Pillai, Probability, Random Variables and Stochastic Processes (McGraw Hill Higher Education, 2002), 4th ed.
- R. J. Schalkoff, Pattern Recognition: Statistical, Structural and Neural Approaches (Wiley, 1991), 1st ed.

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.