## Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging

Optics Express, Vol. 11, Issue 26, pp. 3528-3541 (2003)

http://dx.doi.org/10.1364/OE.11.003528

Acrobat PDF (402 KB)

### Abstract

In this paper we present a high-resolution technique to passively sense, detect and recognize a 3D object using computational integral imaging. We show that the use of a non-stationary microlens array improves the longitudinal distance estimation quantization error. The proposed method overcomes the Nyquist upper limit for the resolution. We use 3D non-linear correlation to recognize the 3D coordinates and shape of the desired object.

© 2003 Optical Society of America

## 1. Introduction

1. H. E. Ives, “Optical properties of a Lippmann lenticulated sheet,” J. Opt. Soc. Am. **21**, 171–176 (1931). [CrossRef]

6. B. Javidi and E. Tajahuerce, Three-dimensional object recognition by use of digital holography,” Opt. Lett. **25**, 610–612 (2000). [CrossRef]

7. J. Rosen, “three dimensional electro-optical correlation,” J. Opt. Soc. Am. A **15**, 430–436 (1998). [CrossRef]

9. Y. Frauel and B. Javidi, “Digital three-dimensional image correlation by use of computer reconstructed integral imaging,” Appl. Opt. **41**, 5488–5496 (2002). [CrossRef] [PubMed]

10. J. Jang and B. Javidi, “Improved viewing resolution of three dimensional integral imaging by use of non-stationary micro-optics,” Opt. Lett. **27**, 324–326 (2002). [CrossRef]

11. A. Stern and B. Javidi, “3D Image Sensing and Reconstruction with Time-Division Multiplexed Computational Integral Imaging (CII),” Applied Optics-IP **42**, 7036–7042 (2003). [CrossRef]

14. S. Kim, N. Bose, and H. Valenzuela, “Recursive reconstruction of high resolution image from noisy undersampled multiframes,” IEEE trans. On Acoustics, Speech and Signal Processing **38**, 1013–1027 (1990). [CrossRef]

## 2. High resolution 3D image reconstruction using time multiplexed integral imaging

10. J. Jang and B. Javidi, “Improved viewing resolution of three dimensional integral imaging by use of non-stationary micro-optics,” Opt. Lett. **27**, 324–326 (2002). [CrossRef]

### 2.1 Maximum possible depth estimation resolution

*d*from the microlens array, and

*ϕ*is the microlens diameter. Using one-dimensional notation, let

*n*be the microlens number with reference to the center microlens. Using Fig. 3, the projection of an object pixel at the CCD plane for two different longitudinal depths can be represented by:

*z*=(

*z*-

_{2}*z*) and

_{1}*Δx*=(

*x*-

_{1}*x*).

_{2}*Δz*in Eq. (2)] needed to produce a shift of one pixel in an elemental image. While calculating the best possible depth resolution, we have to set a minimum z distance such that all the elemental images are focused at the same plane and are not overlapped. We use a microlens array with each microlens having a diameter

*ϕ*=1, and a focal length of 5.2mm. Figure 4 shows the depth change as a function of the longitudinal distance of the 3D object, z

_{1}, that produces a one pixel shift at the CCD plane for n=6 and n=10. It is worth to mention that as the microlens number gets lower (that is, the lenslet becomes closer to the optical axis), a higher depth change is needed to produce one pixel shift in the projected image plane. It is clear that a stationary microlens array cannot produce high accuracy to estimate the depth of a 3D object.

*c*×

*c*, and the minumum acceptable distance by the system is z

_{min}. We can show that the depth change from z

_{min}that produces a one pixel shift on the CCD is given by:

### 2.2 High resolution integral image generation

*N*times has an effect equivelent to reducing the pixel size by

*N*times which improves, theoretically, the depth estimation accuracy by as much as

*N*times.

11. A. Stern and B. Javidi, “3D Image Sensing and Reconstruction with Time-Division Multiplexed Computational Integral Imaging (CII),” Applied Optics-IP **42**, 7036–7042 (2003). [CrossRef]

*f*(

*x*) be a low resolution image, and

*g*(

*x*) be the reconstructed high resolution image. The reconstructed high resolution image at iteration

*j*is given by:

*h*is the optical system point spread function, ↓

_{psf}^{D}is down sampling, ↑

^{U}is up-sampling, ⊗ stands for correlation,

*N*is the number of low resolution images and

*i*takes a value from 1 to

*N*.

*h*is the microlens point spread function which is obtained using a point source placed at a far distance.

_{psf}*h*is the impulse response of the back projection function and is obtaind by psudo invers of the point spread function. Figure 5 shows the reconstructed high-resolution elemental image from a set of four low-resolution images. In Fig. 5, the resolution enhancement in the horizontal direction is achieved through interpolation.

_{bp}## 3. High resolution 3D reconstruction and depth estimation:

*ϕ*/

*N*to store another array of elemental integral images. We repeat this step until we store

*N*arrays of elemental integral images. Figure 7 shows an example of a sequence of low-resolution time multiplexed elemental integral images.

*Δx*.

*Δz*. To locate a certain object pixel in the other elemental images we use a window that is centered on this pixel with a size of

_{min}*W*×

_{x}*W*. We chose the window size as small as possible to increase the longitudinal distance estimation.

_{y}*u*in the 3D object such that the longitudinal resolution is preserved by Δz is:

*z*, as a function of the object pixel coordinate

_{max}*u*for different microlenses. Both cases of a stationary and moving microlens array are illustrated. The total number microlens is

*n*.

*d*(

_{n,m}*x,y*) between a window centered at coordinates

*x*and

_{p}*y*in the center elemental image and a moving window, centered at

_{q}*x*,

_{i}*y*, over the integral image corresponding to lens (

_{j}*n,m*) is given by:

*I*(

_{n.m}*x,y*) is the elemental image projected by lens (

*n,m*), (

*x,y*) are the spatial coordinates, and

*N*,

_{x}*N*are the number of rows and columns in the elemental image.

_{y}*W*,

_{x}*W*are the correlation window width and height. Then,

_{y}*ẑ*(

_{n, m}*x, y*) using the elemental image corresponding to lens number (

*n,m*). The Actual depth of the 3D object coordinates (

*u,v*) can be presented as:

*n*(

_{z}*u,v;n,m,ẑ*) is the longitudinal depth estimation error and

*ẑ*(

*u, v; n, m*) is the depth estimation. The estimation error can be represented as a uniform random variable with zero mean and a variance equals to Δ

*z*

^{2}(

*u, v; n, m, ẑ*)/12, where

*Δz*is computed using Eq. (3). The final depth estimation for an object coordinates (

*u,ν*) can be given by:

*N*, and

_{u}*N*are the number of active rows and columns in the microlens array used in computing

_{v}*ẑ*(

*u,v;m, n*)). Assuming that the quantization errors in depth estimation in different elemental images are statistically independent and the number of elemental images is large enough, the final depth is:

*n*(

_{z}*u,v*) is a Gaussian random variable with zero mean and a variance equals:

## 4. 3D object detection

19. B. Javidi, “Nonlinear joint power spectrum based optical correlators,” Appl. Opt. **28**, 2358–2367 (1989). [CrossRef] [PubMed]

*r*(

*x,y,z*) and the reconstructed input scene be denoted as

*s*(

*x,y,z*). The correlation output is given by:

*S*| and |

*R*| are the absolute value of Fourier transform of

*s*(

*x,y,z*) and

*r*(

*x,y,z*), respectively.

*ϕ*and

_{S}*ϕ*are the phases of the Fourier transform of

_{R}*s*(

*x,y,z*) and

*r*(

*x,y,z*), respectively.

*IFT*stands for the inverse Fourier transform, and

*k*is the non-linearity that is between 1 and 0. The reference object coordinates (

*x,y,z*) is determined according to the maximum of C.

## 5. 3D recognition experimental results and simulations

19. B. Javidi, “Nonlinear joint power spectrum based optical correlators,” Appl. Opt. **28**, 2358–2367 (1989). [CrossRef] [PubMed]

*z*distance of the 3D object properly. Figure 15(c) shows the correlation output at plane that is 79 mm from the lenslet, which is the location of the other object. As we can see from the figure there was a sharp peak at the location of the desired 3D object at plane that is 75mm from the lenslet.

## 6. Conclusion

## References and links

1. | H. E. Ives, “Optical properties of a Lippmann lenticulated sheet,” J. Opt. Soc. Am. |

2. | J. Caulfield, |

3. | F. Okano, J. Arai, H. Hoshino, and I. Yuyama, “Three dimensional video system based on integral photography,” Opt. Eng. |

4. | H. Arimoto and B. Javidi, “Integral three-dimensional imaging with digital reconstruction,” Opt. Lett. |

5. | T. Okoshi, |

6. | B. Javidi and E. Tajahuerce, Three-dimensional object recognition by use of digital holography,” Opt. Lett. |

7. | J. Rosen, “three dimensional electro-optical correlation,” J. Opt. Soc. Am. A |

8. | J. Rosen, “Electrooptical correlators for three-dimensional pattern recognition” in Image Recognition and Classification: Algorithms, Systems, and Applications, B. Javidi Ed., Marcel Dekker, NY2002. |

9. | Y. Frauel and B. Javidi, “Digital three-dimensional image correlation by use of computer reconstructed integral imaging,” Appl. Opt. |

10. | J. Jang and B. Javidi, “Improved viewing resolution of three dimensional integral imaging by use of non-stationary micro-optics,” Opt. Lett. |

11. | A. Stern and B. Javidi, “3D Image Sensing and Reconstruction with Time-Division Multiplexed Computational Integral Imaging (CII),” Applied Optics-IP |

12. | R. Tsu and T. Huang, “Multi-frame image restoration and registration,” in advances in computer vision and image processing 1, 317–339, JAI press (1984). |

13. | A. Teklap, |

14. | S. Kim, N. Bose, and H. Valenzuela, “Recursive reconstruction of high resolution image from noisy undersampled multiframes,” IEEE trans. On Acoustics, Speech and Signal Processing |

15. | D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using subpixel displacement,” Proceeding of the IEEE computer society conference on computer vision and pattern recognition, 742–746, June (1988). |

16. | B. Frieden and H. Aumann, “Image reconstruction from multiple 1-D scans using filtered localization projection,” Appl. Optics |

17. | N. Shah and A. Zakhor, “Multiframe spatial resolution enhancement of color video,” in Proceeding of the IEEE international conference on image processing, Lausanne, Switzerland, 985–988, Sept. (1996). |

18. | R. Schultz and R. Stevenson, “Improved definition video frame enhancement,” in Proceeding of the IEEE international conference of Acoustics, Speech and Signal Processing, 2169–2172, Detroit, MI (1995). |

19. | B. Javidi, “Nonlinear joint power spectrum based optical correlators,” Appl. Opt. |

20. | A. Mahalanobis, “On the optimality of the MACH filter for detection of targets in noise” Opt. Eng. |

21. | C. Chesnaud, Ph. Réfrégier, and V. Boulet, “Statistical region snake based segmentation adapted to different physical noise models,” IEEE Transactions on Pattern Analysis and Machine Intelligence |

22. | B. Javidi and J.L. Homer, |

**OCIS Codes**

(100.4550) Image processing : Correlators

(100.5010) Image processing : Pattern recognition

(100.6890) Image processing : Three-dimensional image processing

(120.1880) Instrumentation, measurement, and metrology : Detection

**ToC Category:**

Research Papers

**History**

Original Manuscript: November 11, 2003

Revised Manuscript: December 5, 2003

Published: December 29, 2003

**Citation**

Sherif Kishk and Bahram Javidi, "Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging," Opt. Express **11**, 3528-3541 (2003)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-11-26-3528

Sort: Journal | Reset

### References

- H. E. Ives, "Optical properties of a Lippmann lenticulated sheet," J. Opt. Soc. Am. 21, 171-176 (1931). [CrossRef]
- J. Caulfield, Handbook of Optical Holography, (Academic, London, 1979).
- F. Okano, J. Arai, H. Hoshino and I. Yuyama, �??Three dimensional video system based on integral photography,�?? Opt. Eng. 38, 1072-1077 (1999). [CrossRef]
- H. Arimoto and B. Javidi, �??Integral three-dimensional imaging with digital reconstruction,�?? Opt. Lett. 26, 157-159 (2001). [CrossRef]
- T. Okoshi, Three-dimensional Imaging Techniques, (Academic, NY, 1971).
- B. Javidi and E. Tajahuerce, Three-dimensional object recognition by use of digital holography,�?? Opt. Lett. 25, 610-612 (2000). [CrossRef]
- J. Rosen, �??three dimensional electro-optical correlation,�?? J. Opt. Soc. Am. A 15, 430-436 (1998). [CrossRef]
- J. Rosen, "Electrooptical correlators for three-dimensional pattern recognition" in Image Recognition and Classification: Algorithms, Systems, and Applications, B. Javidi Ed., Marcel Dekker, NY 2002.
- Y. Frauel and B. Javidi, �??Digital three-dimensional image correlation by use of computer reconstructed integral imaging,�?? Appl. Opt. 41, 5488-5496 (2002). [CrossRef] [PubMed]
- J. Jang and B. Javidi, �??Improved viewing resolution of three dimensional integral imaging by use of nonstationary micro-optics,�?? Opt. Lett. 27, 324-326 (2002). [CrossRef]
- A. Stern and B. Javidi, " 3D Image Sensing and Reconstruction with Time-Division Multiplexed Computational Integral Imaging (CII)," Applied Optics-IP 42, 7036-7042 (2003). [CrossRef]
- R. Tsu and T. Huang, �??Multi-frame image restoration and registration,�?? in advances in computer vision and image processing 1, 317-339, JAI press (1984).
- A. Teklap, Digital Video Processing, (Prentice Hall, NJ, 1995).
- S. Kim, N. Bose, and H. Valenzuela, �??Recursive reconstruction of high resolution image from noisy undersampled multiframes,�?? IEEE trans. On Acoustics, Speech and Signal Processing 38, 1013-1027 (1990). [CrossRef]
- D. Keren, S. Peleg, and R. Brada, �??Image sequence enhancement using subpixel displacement,�?? Proceeding of the IEEE computer society conference on computer vision and pattern recognition, 742-746, June (1988).
- B. Frieden, and H. Aumann, �??Image reconstruction from multiple 1-D scans using filtered localization projection,�?? Appl. Optics 26, 3615-1621 (1987). [CrossRef]
- N. Shah, and A. Zakhor, �??Multiframe spatial resolution enhancement of color video,�?? in Proceeding of the IEEE international conference on image processing, Lausanne, Switzerland, 985-988, Sept. (1996).
- R. Schultz, and R. Stevenson, �??Improved definition video frame enhancement,�?? in Proceeding of the IEEE international conference of Acoustics, Speech and Signal Processing, 2169-2172, Detroit, MI (1995).
- B. Javidi, �??�??Nonlinear joint power spectrum based optical correlators,�??�?? Appl. Opt. 28, 2358�??2367 (1989). [CrossRef] [PubMed]
- A. Mahalanobis, �??On the optimality of the MACH filter for detection of targets in noise�?? Opt. Eng. 36, 2642-2648 (1997). [CrossRef]
- C. Chesnaud, Ph. Réfrégier and V. Boulet, "Statistical region snake based segmentation adapted to different physical noise models," IEEE Transactions on Pattern Analysis and Machine Intelligence 21, 1145-1157 (1999). [CrossRef]
- B. Javidi and J.L. Homer, Real-time Optical Information Processing, (Academic Press, NY 1994).

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

### Figures

Fig. 1. |
Fig. 2. |
Fig. 3. |

Fig. 4. |
Fig. 5. |
Fig. 6. |

Fig. 7. |
Fig. 8. |
Fig. 9. |

Fig. 10. |
Fig. 11. |
Fig. 12. (a) |

Fig. 13. |
Fig. 14. |
Fig. 15. (a) |

Fig. 16. |
||

### Supplementary Material

» Media 1: GIF (30 KB)

» Media 2: GIF (55 KB)

» Media 3: AVI (675 KB)

» Media 4: AVI (196 KB)

« Previous Article | Next Article »

OSA is a member of CrossRef.