OSA's Digital Library

Optics Express

Optics Express

  • Editor: C. Martijn de Sterke
  • Vol. 20, Iss. 21 — Oct. 8, 2012
  • pp: 23044–23052
« Show journal navigation

Depth extraction of 3D objects using axially distributed image sensing

Suk-Pyo Hong, Donghak Shin, Byung-Gook Lee, and Eun-Soo Kim  »View Author Affiliations


Optics Express, Vol. 20, Issue 21, pp. 23044-23052 (2012)
http://dx.doi.org/10.1364/OE.20.023044


View Full Text Article

Acrobat PDF (3975 KB)





Browse Journals / Lookup Meetings

Browse by Journal and Year


   


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools

Share
Citations

Abstract

Axially distributed image sensing (ADS) technique is capable of capturing 3D objects and reconstructing high-resolution slice plane images for 3D objects. In this paper, we propose a computational method for depth extraction of 3D objects using ADS. In the proposed method, the high-resolution elemental images are recorded by simply moving the camera along the optical axis and the recorded elemental images are used to generate a set of 3D slice images using the computational reconstruction algorithm based on ray back-projection. To extract depth of 3D object, we propose the simple block comparison algorithm between the first elemental image and a set of 3D slice images. This provides a simple computation process and robustness for depth extraction. To demonstrate our method, we carry out the preliminary experiments of three scenarios for 3D objects and the results are presented. To our best knowledge, this is the first report to extract the depth information using an ADS method.

© 2012 OSA

1. Introduction

Acquiring depth information from three-dimensional (3D) objects in real world is an important issue in many diverse fields of computer vision, 3D display and 3D recognition. Three-dimensional (3D) passive imaging makes it possible to extract depth information by recording different perspectives of 3D objects [1

1. T. Okoshi, Three-dimensional Imaging Techniques (Academic Press, 1976).

9

9. D. Shin and B. Javidi, “Three-dimensional imaging and visualization of partially occluded objects using axially distributed stereo image sensing,” Opt. Lett. 37(9), 1394–1396 (2012). [CrossRef] [PubMed]

]. Among 3D passive imaging technologies, integral imaging has been widely studied for depth extraction [10

10. J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, “Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification,” Appl. Opt. 43(25), 4882–4895 (2004). [CrossRef] [PubMed]

17

17. M. DaneshPanah and B. Javidi, “Profilometry and optical slicing by passive three-dimensional imaging,” Opt. Lett. 34(7), 1105–1107 (2009). [CrossRef] [PubMed]

]. In original structure of integral imaging, the lens array is used to record 3D objects. However, the recorded elemental images have low-resolution so that the extracted depth is poor. To overcome this problem, synthetic aperture integral imaging (SAII) was proposed [18

18. J.-S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. 27(13), 1144–1146 (2002). [CrossRef] [PubMed]

21

21. M. Cho and B. Javidi, “Three-dimensional visualization of objects in turbid water using integral imaging,” J. Disp. Technol. 6(10), 544–547 (2010). [CrossRef]

], in which the multiple cameras or moving cameras are used to acquire depth information. Even though it can provide high-resolution depth information, the system structure of integral imaging is complex because of 2D grid array structure of many cameras or moving a camera along both horizontal and vertical direction. Recently, a simple modification structure of SAII was proposed. This is called an axially distributed image sensing (ADS) method [8

8. D. Shin and B. Javidi, “3D visualization of partially occluded objects using axially distributed sensing,” J. Disp. Technol. 7(5), 223–225 (2011). [CrossRef]

,22

22. R. Schulein, M. DaneshPanah, and B. Javidi, “3D imaging with axially distributed sensing,” Opt. Lett. 34(13), 2012–2014 (2009). [CrossRef] [PubMed]

24

24. D. Shin and B. Javidi, “Visualization of 3D objects in scattering medium using axially distributed sensing,” J. Disp. Technol. 8(6), 317–320 (2012). [CrossRef]

], in which longitudinal perspective information is obtained by translating a camera along its optical axis. The high-resolution elemental images recorded from ADS can be used for acquiring high-resolution depth information of real-world 3D objects. However, it has a disadvantage of the limitation of the object pickup zone. Despite this limitation, the ADS is an attractive way to provide high-resolution elemental images and simple axial movement.

2. Depth extraction method using ADS

The proposed depth extraction method using ADS is shown in Fig. 1
Fig. 1 Proposed depth extraction processes using ADS
. It is divided into three sub-parts: ADS pickup part, digital reconstruction part and the depth extraction part.

2.1 ADS pickup

The ADS pickup part of 3D objects is shown in Fig. 2
Fig. 2 Pickup process in ADS
, in which a single camera records elemental images by moving the camera along the optical axis [7

7. M. DaneshPanah, B. Javidi, and E. A. Watson, “Three dimensional imaging with randomly distributed sensors,” Opt. Express 16(9), 6368–6377 (2008). [CrossRef] [PubMed]

].

We suppose that the focal length of imaging lens is defined as g. Then, the different elemental images are captured along optical axis (z axis) if 3D objects are located at a distance z1 away from the first camera. We can record k elemental images by shifting a camera with k-1 times. Then, the last camera is located at zk. If Δz is the separation distance between two adjacent cameras in ADS pickup process as shown in Fig. 2, the ith elemental image is captured from the camera located at a distance of di = Z-z1-(i-1)Δz from the object. Since we can capture each elemental image at a different camera position, it contains the object image with different scale level. In other words, when i = 1, the object image is the smallest in elemental image because the camera position is farthest from 3D object. While, the object image is largest when i = k.

2.2 Slice image reconstruction using the recorded elemental images

The second part of the proposed depth extraction method is digital reconstruction. The aim of the digital reconstruction part is to generate slice plane image using the recorded elemental images from the first part of ADS pickup. The digital reconstruction process is shown in Fig. 3
Fig. 3 Principle of digital reconstruction in ADS
. The digital reconstruction process of 3D objects is the inverse process of ADS pickup. It can be implemented on the basis of an inverse mapping procedure through a pinhole model [7

7. M. DaneshPanah, B. Javidi, and E. A. Watson, “Three dimensional imaging with randomly distributed sensors,” Opt. Express 16(9), 6368–6377 (2008). [CrossRef] [PubMed]

]. Each camera in the ADS pickup part is modeled as pinhole and elemental image at the original camera position as shown in Fig. 3. We assume that a reconstruction plane for the computational reconstruction of the 3D image is located at distance z = L. Each elemental image is inversely projected through each corresponding pinhole to the reconstruction plane at L. Then, the ith inversely projected elemental image is magnified with mi = (L-zi)/g. At the reconstruction plane, all inversely mapped elemental images are superimposed each other with the different magnifications. In Fig. 3, we assume that Ei is the ith elemental image with the size of p × q, and IL is the superimposed image of all the inversely mapped image of the elemental image at the reconstruction plane L. IL is given by
IL=1ki=1kUiEi
(1)
where Ui is upsampling operator for magnification of Ei at the reconstruction plane of z = L-zi and the size of IL is m1p × m1q.

Since the superimposition of all elemental images can require large computation load due to the large magnification factor, we can modify Eq. (1) by using the downsampling operator Dr of image by a factor of r. Then, it becomes
IL1ki=1kDrUiEi
(2)
In order to generate the 3D volume information, we repeat this process for different distances.

2.3 Depth extraction process

In the depth extraction part of the proposed method, we extract depth information using many slice plane images from digital reconstruction described in Eq. (2). The reconstructed slice plane images consist of different mixture images with focused images and blurred images according to the reconstruction distances. The focused images of 3D object are reconstructed only at the original position of 3D object. While, blurred images are shown out of original position. Based on this principle, we want to find the focused image part in the reconstructed slice plane images.

The depth extraction algorithm used in this paper is shown in Fig. 4
Fig. 4 Block comparison algorithm between the first elemental image and slice images
. This is based on separation between focused image and defocused image. To find the focused image part in each slice plane image, we use block comparison between the first elemental image and slice plane images. Note that the first elemental image is composed of only focused images of 3D objects. We select block images in both first elemental image and the slice plane images. The block images have the same position as shown in Fig. 4. Then we apply block comparison algorithm into two selective block images. Here, block comparison calculates intensity error between the block image of the first elemental image (DrU1E1) and a set of block images of slice images (IL) reconstructed at L. The intensity error between the blocks along the reconstruction distance (L) can be defined as the sum of absolute difference (SAD):
SADL=i=1bj=1b|DrU1E1(x+i,y+j)IL(x+i,y+j)|
(3)
where the block size is b × b.

Using SAD result, the depth of each point (x,y) can be extracted by finding L value where SADL is minimized over the range of L. This can be mathematically formulated as:
L^(x,y)=argminLSADL(x,y)
(4)
where the size of L^ is m1p/r × m1q/r.

The depth map can be obtained by calculating depth of all points in the images.

2.4 Depth accuracy in ADS

The accuracy of extracted depth information is dependent of the step size of L value. We can calculate the minimum step size of δ value using Fig. 5
Fig. 5 Ray diagram for calculation of minimum step size
. As shown in Fig. 5, when we suppose two L values (L and L + δ), the inversely magnified elemental image is superimposed at the reconstruction plane. If the object is located a distance d from the axis and c is the pixel size of the camera sensor, the elemental image should be superimposed at different pixel of the reconstructed plane to separate the reconstructed plane image. The minimum step size of δ value is obtained when the pixel position in the reconstruction plane has the difference of 1 pixel. Then, it becomes
δ=(Lz1)cd.
(5)
For practical parameters, we simulated Eq. (5) as shown in Fig. 6
Fig. 6 Calculation results of minimum step size for practical parameters.
. As d increases, δ can be reduced. When c = 5 μm, d = 100 mm, L = 500 mm, and z1 = 300 mm, the minimum step size δ becomes 10 μm.

3. Experiments and results

To demonstrate our depth extraction method using ADS, we performed the preliminary experiments for three scenarios composed of different 3D objects. The experimental structure is shown in Fig. 7
Fig. 7 Three scenarios for optical experiments object
.

First objects were composed of simple characters ‘k’ and ‘w’. Second scenario has two objects of ‘car’ and ‘sign’ toys. Last one is three ‘car’ objects with continuous depth. Front object and behind object are positioned between approximately 350 mm and 500 away from the first camera, respectively. They are located at approximately 70 mm from the optical axis. We use a Nikon camera (D7000) whose pixels are 3872 × 2592. The imaging lens with focal length f = 40 mm is used in this experiments. The camera is translated at Δz = 3 mm increments for a total of K = 41 elemental images and a total displacement distance of 120 mm. Among them, some elemental images for second scenario are shown in Fig. 8
Fig. 8 Four examples of the recorded elemental images for second object (a) first elemental image (b) 10th elemental image (c) 20th elemental image (d) 30th elemental image
.

After recording the elemental images using ADS, we reconstruct slice plane image for 3D objects. The recorded 41 elemental images are used to the digital reconstruction algorithm using Eq. (2). In digital reconstruction process, the pinhole gap was 40 mm and the interval of reconstruction plane was 10 mm, which is much larger than the minimum step size of Eq. (5). To simplify the computational calculations, the downsampling factor was 1/U1 and then Ui was normalized by U1. The reconstruction plane was moved from 300 mm to 600 mm. Then we can reconstruct 31 slice plane images. Some reconstructed slice plane images for second objects of Fig. 7(b) are shown in Fig. 9
Fig. 9 Reconstructed slice images for second objects (a) at 350 mm (b) at 420 mm (c) at 500 mm.
.

The ‘car’ object was focused at L = 350 as shown in Fig. 9(a).While, the ‘sign’ object was generated clearly at L = 500 mm as shown in Fig. 9(c). And, the blurred image was obtained at L = 420 mm as shown in Fig. 9(b).

Now, we estimated the depth of the object using the first reference elemental image and 31 slice plane images. We apply the block comparison algorithm to these images. The block size was 8 × 8. The estimated depths are shown in Fig. 10
Fig. 10 Extracted depth map for three scenarios
. The depth results were well extracted. This result reveals that the proposed method can extract the 3D information of object effectively.

4. Conclusion

In conclusion, we have presented a depth extraction method using ADS. In the proposed method, the high-resolution elemental images were recorded by simply moving the camera along the optical axis and the recorded elemental images were used to generate a set of 3D slice images using the computational reconstruction algorithm based on ray back-projection. To extract depth of 3D object, the block comparison algorithm between the first elemental image and a set of 3D slice images was used. Since ADS provides high-resolution slice plane images of 3D objects, we can extract high-resolution 3D depth information. We performed the preliminary experiment of 3D object and presented the results to show the usefulness of the proposed method.

Acknowledgment

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MEST) (No. 2012-0009223)

References and links

1.

T. Okoshi, Three-dimensional Imaging Techniques (Academic Press, 1976).

2.

J.-S. Ku, K.-M. Lee, and S.-U. Lee, “Multi-image matching for a general motion stereo camera model,” Pattern Recognit. 34(9), 1701–1712 (2001). [CrossRef]

3.

A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006). [CrossRef]

4.

K. Itoh, W. Watanabe, H. Arimoto, and K. Isobe, “Coherence-based 3-D and spectral imaging and laser-scanning microscopy,” Proc. IEEE 94(3), 608–628 (2006). [CrossRef]

5.

J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48(34), H77–H94 (2009). [CrossRef] [PubMed]

6.

Y.-M. Kim, K.-H. Hong, and B. Lee, “Recent researches based on integral imaging display method,” 3D Research 1(1), 17–27 (2010). [CrossRef]

7.

M. DaneshPanah, B. Javidi, and E. A. Watson, “Three dimensional imaging with randomly distributed sensors,” Opt. Express 16(9), 6368–6377 (2008). [CrossRef] [PubMed]

8.

D. Shin and B. Javidi, “3D visualization of partially occluded objects using axially distributed sensing,” J. Disp. Technol. 7(5), 223–225 (2011). [CrossRef]

9.

D. Shin and B. Javidi, “Three-dimensional imaging and visualization of partially occluded objects using axially distributed stereo image sensing,” Opt. Lett. 37(9), 1394–1396 (2012). [CrossRef] [PubMed]

10.

J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, “Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification,” Appl. Opt. 43(25), 4882–4895 (2004). [CrossRef] [PubMed]

11.

J.-H. Park, Y. Kim, J. Kim, S.-W. Min, and B. Lee, “Three-dimensional display scheme based on integral imaging with three-dimensional information processing,” Opt. Express 12(24), 6020–6032 (2004). [CrossRef] [PubMed]

12.

G. Passalis, N. Sgouros, S. Athineos, and T. Theoharis, “Enhanced reconstruction of three-dimensional shape and texture from integral photography images,” Appl. Opt. 46(22), 5311–5320 (2007). [CrossRef] [PubMed]

13.

D.-H. Shin, B.-G. Lee, and J.-J. Lee, “Occlusion removal method of partially occluded 3D object using sub-image block matching in computational integral imaging,” Opt. Express 16(21), 16294–16304 (2008). [CrossRef] [PubMed]

14.

J.-H. Jung, K. Hong, G. Park, I. Chung, J.-H. Park, and B. Lee, “Reconstruction of three-dimensional occluded object using optical flow and triangular mesh reconstruction in integral imaging,” Opt. Express 18(25), 26373–26387 (2010). [CrossRef] [PubMed]

15.

D.-C. Hwang, D.-H. Shin, S.-C. Kim, and E.-S. Kim, “Depth extraction of three-dimensional objects in space by the computational integral imaging reconstruction technique,” Appl. Opt. 47(19), D128–D135 (2008). [CrossRef] [PubMed]

16.

B.-G. Lee, H.-H. Kang, and E.-S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research 1(2), 2–10 (2010). [CrossRef]

17.

M. DaneshPanah and B. Javidi, “Profilometry and optical slicing by passive three-dimensional imaging,” Opt. Lett. 34(7), 1105–1107 (2009). [CrossRef] [PubMed]

18.

J.-S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. 27(13), 1144–1146 (2002). [CrossRef] [PubMed]

19.

A. Stern and B. Javidi, “3-D computational synthetic aperture integral imaging (COMPSAII),” Opt. Express 11(19), 2446–2451 (2003). [CrossRef] [PubMed]

20.

Y. S. Hwang, S.-H. Hong, and B. Javidi, “Free view 3-D visualization of occluded objects by using computational synthetic aperture integral imaging,” J. Disp. Technol. 3(1), 64–70 (2007). [CrossRef]

21.

M. Cho and B. Javidi, “Three-dimensional visualization of objects in turbid water using integral imaging,” J. Disp. Technol. 6(10), 544–547 (2010). [CrossRef]

22.

R. Schulein, M. DaneshPanah, and B. Javidi, “3D imaging with axially distributed sensing,” Opt. Lett. 34(13), 2012–2014 (2009). [CrossRef] [PubMed]

23.

D. Shin, M. Cho, and B. Javidi, “Three-dimensional optical microscopy using axially distributed image sensing,” Opt. Lett. 35(21), 3646–3648 (2010). [CrossRef] [PubMed]

24.

D. Shin and B. Javidi, “Visualization of 3D objects in scattering medium using axially distributed sensing,” J. Disp. Technol. 8(6), 317–320 (2012). [CrossRef]

OCIS Codes
(100.6890) Image processing : Three-dimensional image processing
(110.4190) Imaging systems : Multiple imaging
(110.6880) Imaging systems : Three-dimensional image acquisition

ToC Category:
Imaging Systems

History
Original Manuscript: July 20, 2012
Revised Manuscript: September 17, 2012
Manuscript Accepted: September 18, 2012
Published: September 24, 2012

Citation
Suk-Pyo Hong, Donghak Shin, Byung-Gook Lee, and Eun-Soo Kim, "Depth extraction of 3D objects using axially distributed image sensing," Opt. Express 20, 23044-23052 (2012)
http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-20-21-23044


Sort:  Author  |  Year  |  Journal  |  Reset  

References

  1. T. Okoshi, Three-dimensional Imaging Techniques (Academic Press, 1976).
  2. J.-S. Ku, K.-M. Lee, and S.-U. Lee, “Multi-image matching for a general motion stereo camera model,” Pattern Recognit.34(9), 1701–1712 (2001). [CrossRef]
  3. A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE94(3), 591–607 (2006). [CrossRef]
  4. K. Itoh, W. Watanabe, H. Arimoto, and K. Isobe, “Coherence-based 3-D and spectral imaging and laser-scanning microscopy,” Proc. IEEE94(3), 608–628 (2006). [CrossRef]
  5. J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt.48(34), H77–H94 (2009). [CrossRef] [PubMed]
  6. Y.-M. Kim, K.-H. Hong, and B. Lee, “Recent researches based on integral imaging display method,” 3D Research1(1), 17–27 (2010). [CrossRef]
  7. M. DaneshPanah, B. Javidi, and E. A. Watson, “Three dimensional imaging with randomly distributed sensors,” Opt. Express16(9), 6368–6377 (2008). [CrossRef] [PubMed]
  8. D. Shin and B. Javidi, “3D visualization of partially occluded objects using axially distributed sensing,” J. Disp. Technol.7(5), 223–225 (2011). [CrossRef]
  9. D. Shin and B. Javidi, “Three-dimensional imaging and visualization of partially occluded objects using axially distributed stereo image sensing,” Opt. Lett.37(9), 1394–1396 (2012). [CrossRef] [PubMed]
  10. J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, “Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification,” Appl. Opt.43(25), 4882–4895 (2004). [CrossRef] [PubMed]
  11. J.-H. Park, Y. Kim, J. Kim, S.-W. Min, and B. Lee, “Three-dimensional display scheme based on integral imaging with three-dimensional information processing,” Opt. Express12(24), 6020–6032 (2004). [CrossRef] [PubMed]
  12. G. Passalis, N. Sgouros, S. Athineos, and T. Theoharis, “Enhanced reconstruction of three-dimensional shape and texture from integral photography images,” Appl. Opt.46(22), 5311–5320 (2007). [CrossRef] [PubMed]
  13. D.-H. Shin, B.-G. Lee, and J.-J. Lee, “Occlusion removal method of partially occluded 3D object using sub-image block matching in computational integral imaging,” Opt. Express16(21), 16294–16304 (2008). [CrossRef] [PubMed]
  14. J.-H. Jung, K. Hong, G. Park, I. Chung, J.-H. Park, and B. Lee, “Reconstruction of three-dimensional occluded object using optical flow and triangular mesh reconstruction in integral imaging,” Opt. Express18(25), 26373–26387 (2010). [CrossRef] [PubMed]
  15. D.-C. Hwang, D.-H. Shin, S.-C. Kim, and E.-S. Kim, “Depth extraction of three-dimensional objects in space by the computational integral imaging reconstruction technique,” Appl. Opt.47(19), D128–D135 (2008). [CrossRef] [PubMed]
  16. B.-G. Lee, H.-H. Kang, and E.-S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research1(2), 2–10 (2010). [CrossRef]
  17. M. DaneshPanah and B. Javidi, “Profilometry and optical slicing by passive three-dimensional imaging,” Opt. Lett.34(7), 1105–1107 (2009). [CrossRef] [PubMed]
  18. J.-S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett.27(13), 1144–1146 (2002). [CrossRef] [PubMed]
  19. A. Stern and B. Javidi, “3-D computational synthetic aperture integral imaging (COMPSAII),” Opt. Express11(19), 2446–2451 (2003). [CrossRef] [PubMed]
  20. Y. S. Hwang, S.-H. Hong, and B. Javidi, “Free view 3-D visualization of occluded objects by using computational synthetic aperture integral imaging,” J. Disp. Technol.3(1), 64–70 (2007). [CrossRef]
  21. M. Cho and B. Javidi, “Three-dimensional visualization of objects in turbid water using integral imaging,” J. Disp. Technol.6(10), 544–547 (2010). [CrossRef]
  22. R. Schulein, M. DaneshPanah, and B. Javidi, “3D imaging with axially distributed sensing,” Opt. Lett.34(13), 2012–2014 (2009). [CrossRef] [PubMed]
  23. D. Shin, M. Cho, and B. Javidi, “Three-dimensional optical microscopy using axially distributed image sensing,” Opt. Lett.35(21), 3646–3648 (2010). [CrossRef] [PubMed]
  24. D. Shin and B. Javidi, “Visualization of 3D objects in scattering medium using axially distributed sensing,” J. Disp. Technol.8(6), 317–320 (2012). [CrossRef]

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.


« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited