## Extraction of location coordinates of 3-D objects from computationally reconstructed integral images basing on a blur metric

Optics Express, Vol. 16, Issue 6, pp. 3623-3635 (2008)

http://dx.doi.org/10.1364/OE.16.003623

Acrobat PDF (614 KB)

### Abstract

In this paper, a novel approach to effectively extract location coordinates of 3-D objects employing a blur metric has been proposed. With elemental images of 3-D objects, plane object images (POIs) were reconstructed along the output plane using the CIIR (computational integral imaging reconstruction) algorithm, in which only the POIs reconstructed on the output planes where 3-D objects were originally located are focused whereas the other ones are blurred. Therefore, by calculating these blur metrics of the reconstructed POIs depth data of 3-D objects could be extracted. That is, the blur metric is the lowest on the focused point, but it starts to increase as (fill in the blank) moves away from that point. Accordingly, by finding out the points of inflection in the map of blur metric variation, the output planes where the objects were located were finally detected. To show the feasibility of our proposed scheme, some experiments were carried out and its results are presented as well.

© 2008 Optical Society of America

## 1. Introduction

1. J.-I. Park and S. Inoue, “Acquisition of sharp depth map from multiple cameras,” Signal Processing: Image Commun. **14**, 7–19 (1998). [CrossRef]

3. G. J. Iddan and G. Yahav, “Three-dimensional imaging in the studio and elsewhere,” Proc. SPIE **4298**, 48–55 (2000). [CrossRef]

4. J.-H. Lee, J.-H. Ko, K.-J. Lee, J.-H. Jang, and E.-S. Kim, “Implementation of stereo camera-based automatic unmanned ground vehicle system for adaptive target detection,” Proc. SPIE **5608**, 188–197 (2004). [CrossRef]

5. J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, “Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification,” Appl. Opt. **43**, 4882–4895 (2004). [CrossRef] [PubMed]

6. J.-H. Park, Y. Kim, J. Kim, S.-W. Min, and B. Lee, “Three-dimensional display scheme based on integral imaging with three-dimensional information processing,” Opt. Express. **12**, 6020–6032 (2004). [CrossRef] [PubMed]

7. S.-W. Min, B. Javidi, and B. Lee, “Enhanced three-dimensional integral imaging system by use of double display devices,” Appl. Opt. **42**, 4186–4195 (2003). [CrossRef] [PubMed]

8. B. Javidi, R. Ponce-Díaz, and S. -H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. , **31**, 1106–1108 (2006). [CrossRef] [PubMed]

9. J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, “Resolution-enhanced three-dimensional image correlator using computationally reconstructed integral images,” Opt Commun. **276**, 72–79 (2007). [CrossRef]

## 2. Operational characteristics of the integral imaging system

## 3. Proposed blur metric-based depth extraction method

11. S. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express. **12**, 483–491 (2004). [CrossRef] [PubMed]

### 3.1 Pickup part

*z*= 30

_{1}*mm*,

*z*= 45

_{2}*mm*and

*z*= 60

_{3}*mm*in front of the pinhole array, respectively. In the pickup part, elemental images of the 3-D test object are computationally picked up. Here, the distance between the virtual pinhole array and the elemental image plane is assumed to be 3

*mm*. The resolution of the picked-up EIA is given by 1,292 by 760 because it is assumed that the lenslet array is consisted of 34 by 20 lenslets and the resolution of each lenslet is 38 by 38 pixels. Resolution of each of the three 2-D objects is given by 430 by 360 pixels, and its center location in the image plane of 1,292 by 760 is set to be (291, 548), (599, 201) and (970, 520), respectively.

### 3.2 Reconstruction part

*z*(

_{l}*z*=

*L*) [11

11. S. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express. **12**, 483–491 (2004). [CrossRef] [PubMed]

*z*, each picked-up elemental image is projected inversely through the corresponding virtual pinhole array. By digitally simulating the reconstruction process basing on the ray optics approach, the EIA can be inversely magnified according to the magnification factor of

_{l}*M*=

*L*/

*g*, in which M is the ratio of the distance between the virtual pinhole array and the reconstructed image plane (

*L*) to the distance between the virtual pinhole array and the EIA plane (-

*g*).

*M*>

*1*, the inversely mapped images through each virtual pinhole array are overlapped and summated with each other on the reconstructed image plane of

*z*. Assuming that the respective sizes of an elemental image in the vertical and horizontal directions are given by

_{l}*a*and

*b*, the vertical and the horizontal sizes of the mapped image on the reconstructed image plane are given by

*M*and

_{a}*M*, respectively according to the magnification factor of

_{b}*M*=

*L*/

*g*. The enlarged elemental image is overlapped and summed at the corresponding pixels of the reconstruction image plane. For the complete reconstruction of the POI at a given distance, the process mentioned above is repeatedly performed to all of the obtained elemental images through each corresponding pinhole.

*Δz*, a set of POIs can be finally reconstructed.

*Δz*= 3

*mm*.

*z*= 30

_{1}*mm*,

*z*= 45

_{2}*mm*and

*z*= 60

_{3}*mm*where the objects were located during the pickup process are clearly focused, but getting away from these planes, the POIs are getting out of focus and appearing to be blurred. Therefore, depth data of the objects could be obtained from the estimation of the focusing (or defocusing) parameters of the reconstructed POIs, so-called a blur metric.

### 3.3 Depth extraction part

#### 3.3.1 Estimation of the blur metric

*I(x,y)*is given, in which

*x*and

*y*are the row and column coordinate in an image, respectively. Additionally,

*I*is the

_{i}(x,y)*i*th channel of the image

*I(x,y)*, i.e.,

*i*= 1 for gray channel,

*i*= 3 for R, G, B channel. The gradient at any pixel point of

*P(x,y)*can be calculated by use of a 2-D directional derivative (sobel operator has been used in this paper) as Eq. (1).

*G*and

_{x}*G*as shown in Eq. (2) and Eq. (3).

_{y}*P*be a local maximum edge point in a POI, then its magnitude and orientation are given by |

_{e}(x_{e},y_{e})*| and*

^{∇}I_{i}(x_{e},y_{e})*θ*, respectively. In order to calculate the spatial variation, the

_{i}(x_{e},y_{e})*ψ*-axis is introduced here, in which the origin of the

*ψ*-axis is at the pixel point of

*P*and the direction is the normal to the

_{e}(x_{e},y_{e})*θ*, as shown in Fig. 7.

_{i}(x_{e},y_{e})*m*and

_{l}(x_{l},y_{l})*m*as the nearest local minima on left and right side of the local maximum point

_{r}(x_{r},y_{r})*P*, respectively i.e.,

_{e}(x_{e},y_{e})*m*is on the negative

_{l}*ψ*–axis and

*m*is on the positive

_{l}*ψ*–axis because the origin of the

*ψ*–axis is at

*P*. As a discrete probability distribution with the mean at the point of

_{e}(x_{e},y_{e})*P*, corresponding to

_{e}(x_{e},y_{e})*ψ*= 0, mathematically, the spatial variance is calculated by Eq. (4).

*β*(

_{i}*P*) for a local maximum edge point

_{e}*P*can be obtained by computing the weighted average of the standard deviation

_{e}(x_{e},y_{e})*σ*and the magnitude of an edge |

_{i}*|, which is given by Eq. (5) [15].*

^{∇}I_{i}(p_{e})#### 3.3.2 Calculation of the mean blur metric

*β*. And then,

_{imean}*β*is divided by the ratio of the number of local maximum points per channel

_{imean}*M*to the number of pixels in a POI

_{i}*N*. Therefore,

*α*represents the percentage of local maximum edge points per channel in a POI.

*β*is then given by Eq. (7) and it can provide discrimination in depth detection of the target objects per POI, not per point.

_{MBM}*N*is the number of pixels in a POI and

*M*is the number of local maximum points per channel. From Eq. (7), the MBM of each POI can be calculated and their results are depicted in Fig. 8. Figure 8 shows a variation of the MBM along the output plane and we can see three points of inflection in this figure, which indicates that there are three potential objects. As noted above, the MBM becomes to be the lowest at the focused point, where the object was originally located, whereas the MBM sharply increases moving away from that point.

_{i}*z*= 30

*mm*, on which the MBM value was found to be 30.47×10

^{-4}, whereas it sharply increased up to 54.57×10

^{-4}, 50.17×10

^{-4}on the neighboring points of 27 and 33

*mm*, respectively. At the same time the gradients of the MBM on the neighboring points of 27 and 33

*mm*were calculated to be - 8.03×10

^{-4}and +6.56×10

^{-4}, respectively. This change of the gradient of the MBM from negative to the positive value finally confirmed that there should be a point of inflection between them and that one object might exist on the plane of

*z*= 30

*mm*. Therefore, from Table 1, the right output planes where the ‘Target 1’, ‘Target 2’ and ‘Target 3’ were originally located could be easily found to be

*z*= 30, 45 and 60

*mm*, respectively.

*z*= 30, 45 and 60

*mm*, where the objects were originally located. Figure 9 shows lateral profiles of their correlation results. From the correlation outputs of Figs. 9(a), 9(b), 9(c), NCC (normalized cross correlation) values and lateral correlation positions for the ‘Target 1’, ‘Target 2’ and ‘Target 3’ were found to be 0.8659, 0.5723, 0.8087 and (291, 548), (599, 201), (970, 520), respectively. Finally 3D location coordinates of the test objects in space can be obtained by (291, 548, 30), (599, 201, 45) and (970, 520, 60), respectively at the Cartesian coordinates system basing on the experimental results of Fig. 8 and Fig. 9. By comparing these results with the original location coordinates of the test objects it was found that they are exactly the same, which might confirm that the proposed method can provide a good discrimination performance in detection of depth and location coordinate of the target objects in space.

### 3.4 Depth extraction for the case of totally overlapped objects

*z*= 30

_{1}*mm*,

*z*= 45

_{2}*mm*and

*z*= 60

_{3}*mm*in front of the lenslet array, respectively just like those of Fig. 3, except that they are totally overlapped along the

*z*-direction. The employed lenslet array is identical with that of Fig. 3. Resolution of each of three 2-D objects is also given by 430 by 360 pixels, but the center locations of the objects are set to be (622, 375), (623, 378) and (625, 372), respectively, so that they are totally overlapped along the

*z*-direction.

*z*= 45

*mm*, on which the MBM value was found to be 27.0×10

^{-4}, whereas it gradually increased up to 30.97×10-4, 33.87×10

^{-4}on the neighboring points of 42 and 48

*mm*, respectively. At the same time the gradients of the MBM on the neighboring points of 42 and 48

*mm*were calculated to be -1.32×10

^{-4}and +2.29×10

^{-4}, respectively. This change of the gradient of the MBM from the negative to the positive value finally confirmed that there should be a point of inflection between them and one object might exist on the plane of

*z*= 45

*mm*. Therefore, from Table 2, the right output planes where the ‘Target 1’, ‘Target 2’ and ‘Target 3’ were originally located could be found to be

*z*= 30, 45 and 60

*mm*, respectively.

*z*= 30, 45 and 60

*mm*, lateral positions of ‘Target 1’, ‘Target 2’ and ‘Target 3’ were found to be (622, 375), (623, 378) and (625, 372), respectively. Accordingly, 3D location coordinates of the test objects were finally obtained to be (622, 375, 30), (623, 378, 45) and (625, 372, 60), respectively. Comparing these results with the original location coordinates of the test objects they are also found to be the same with each other.

*z*= 30, 45 and 60

*mm*where the objects were originally located, so that depth data of the three objects could be accurately detected from the proposed method whether they might be overlapped or not along the output plane. However there are some differences in gradient and MBM values between two extreme cases. First, there were much bigger changes in gradient values around the inflection points in ‘Case 1’ than ‘Case 2’. These results are due to relatively weaker interaction between target objects in ‘Case 1’ comparing with that in ‘Case 2’. Second, the overall MBM values of the ‘Case 2’ are relatively lower than those of the ‘Case 1’. This is due to

*M*which is the number of local maximum points in a POI of ‘Case 2’ is much less than that of the ‘Case 1’.

_{i}*z*-direction, discrimination performance of the points of inflection in the MBM curve might be getting deteriorated a little bit. Anyhow, through successful experiments on depth extraction for two extreme cases of totally non-overlapped and overlapped objects, the feasibility of the proposed method in the practical applications can be validated.

### 3.5 Depth extraction for the case of real objects

*z*= 30 mm and

_{o1}*z*= 45 mm, respectively in front of the lenslet array. Here, a lenslet array with 34×20 lenslets was used, which was located at z=0 mm. Each lenslet size is 1.08 mm, the focal length of a lenslet is 3 mm and a single elemental image is composed of 38×38 pixels.

_{o2}*z*= 30 and 45

*mm*, respectively. Comparing these results with the original location coordinates of the test objects, they were found to be the same with each other, which finally confirmed the feasibility of the proposed method even in the practical implementations.

*z*= 30 and 45

*mm*, lateral positions of ‘Object 1’ and ‘Object 2’ were also additionally found to be (430, 285) and (735, 308), respectively. Accordingly, 3D location coordinates of the real objects were finally obtained to be (430, 285, 30) and (735, 308, 45), respectively.

## 4. Conclusion

## Acknowledgment

## References and links

1. | J.-I. Park and S. Inoue, “Acquisition of sharp depth map from multiple cameras,” Signal Processing: Image Commun. |

2. | J.-H. Ko and E.-S. Kim, “Stereoscopic video surveillance system for detection of Target’s 3D location coordinates and moving trajectories,” Opt. Commun. |

3. | G. J. Iddan and G. Yahav, “Three-dimensional imaging in the studio and elsewhere,” Proc. SPIE |

4. | J.-H. Lee, J.-H. Ko, K.-J. Lee, J.-H. Jang, and E.-S. Kim, “Implementation of stereo camera-based automatic unmanned ground vehicle system for adaptive target detection,” Proc. SPIE |

5. | J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, “Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification,” Appl. Opt. |

6. | J.-H. Park, Y. Kim, J. Kim, S.-W. Min, and B. Lee, “Three-dimensional display scheme based on integral imaging with three-dimensional information processing,” Opt. Express. |

7. | S.-W. Min, B. Javidi, and B. Lee, “Enhanced three-dimensional integral imaging system by use of double display devices,” Appl. Opt. |

8. | B. Javidi, R. Ponce-Díaz, and S. -H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. , |

9. | J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, “Resolution-enhanced three-dimensional image correlator using computationally reconstructed integral images,” Opt Commun. |

10. | H. Arimoto and B. Javidi, “Integral three-dimensional imaging with digital reconstruction,” Opt. Lett. |

11. | S. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express. |

12. | D.-H. Shin, E.-S. Kim, and B. Lee, “Computational reconstruction technique of three-dimensional object in integral imaging using a Lenslet Array,” Jpn. J. Appl. Phys. |

13. | D.-H. Shin, M. Cho, K.-C. Park, and E.-S. Kim, “Computational technique of volumetric object reconstruction in integral imaging by use of real and virtual image fields,” ETRI J. |

14. | P. Marziliano, F. Dufaux, S. Winkler, and T. Ebrahimi, “A no-reference perceptual blur metric,” in the Proceedings of the International Conference on Image Processing , |

15. | Y. C. Chung, J. M. Wang, R. R. Bailey, and S. W. Chen, “A non-parametric blur measure based on edge analysis for image processing applications,” in |

16. | R. Youmaran and A. Adler, “Using red-eye to improve face detection in low quality video image,” |

17. | Z. Rahman, D. J. Jobson, and G. A. Woodell, “A multiscale retinex for color rendition and dynamic range compression,” NASA Langley Technical Report (1996). |

**OCIS Codes**

(100.6890) Image processing : Three-dimensional image processing

(110.0110) Imaging systems : Imaging systems

(110.6880) Imaging systems : Three-dimensional image acquisition

**ToC Category:**

Image Processing

**History**

Original Manuscript: January 2, 2008

Revised Manuscript: February 14, 2008

Manuscript Accepted: February 28, 2008

Published: March 4, 2008

**Citation**

Dong-Choon Hwang, Kwang-Jin Lee, Seung-Cheol Kim, and Eun-Soo Kim, "Extraction of location coordinates of 3-D objects
from computationally reconstructed integral
images basing on a blur metric," Opt. Express **16**, 3623-3635 (2008)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-16-6-3623

Sort: Year | Journal | Reset

### References

- J.-I. Park and S. Inoue, "Acquisition of sharp depth map from multiple cameras," Signal Processing: Image Commun. 14, 7-19 (1998). [CrossRef]
- J.-H. Ko and E.-S. Kim, "Stereoscopic video surveillance system for detection of Target's 3D location coordinates and moving trajectories," Opt. Commun. 191, 100-107 (2006).
- G. J. Iddan and G. Yahav, "Three-dimensional imaging in the studio and elsewhere," Proc. SPIE 4298, 48-55 (2000). [CrossRef]
- J.-H. Lee, J.-H. Ko, K.-J. Lee, J.-H. Jang, and E.-S. Kim, "Implementation of stereo camera-based automatic unmanned ground vehicle system for adaptive target detection," Proc. SPIE 5608, 188-197 (2004). [CrossRef]
- J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, "Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification," Appl. Opt. 43, 4882-4895 (2004). [CrossRef] [PubMed]
- J.-H. Park, Y. Kim, J. Kim, S.-W. Min, and B. Lee, "Three-dimensional display scheme based on integral imaging with three-dimensional information processing," Opt. Express. 12, 6020-6032 (2004). [CrossRef] [PubMed]
- S.-W. Min, B. Javidi, and B. Lee, "Enhanced three-dimensional integral imaging system by use of double display devices," Appl. Opt. 42, 4186-4195 (2003). [CrossRef] [PubMed]
- B. Javidi, R. Ponce-Díaz, and S. -H. Hong, "Three-dimensional recognition of occluded objects by using computational integral imaging," Opt. Lett. 31, 1106-1108 (2006). [CrossRef] [PubMed]
- J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, "Resolution-enhanced three-dimensional image correlator using computationally reconstructed integral images," Opt Commun. 276, 72-79 (2007). [CrossRef]
- H. Arimoto and B. Javidi, "Integral three-dimensional imaging with digital reconstruction," Opt. Lett. 26, 157-159 (2001). [CrossRef]
- S. Hong, J.-S. Jang, and B. Javidi, "Three-dimensional volumetric object reconstruction using computational integral imaging," Opt. Express. 12, 483-491 (2004). [CrossRef] [PubMed]
- D.-H. Shin, E.-S. Kim, and B. Lee, "Computational reconstruction technique of three-dimensional object in integral imaging using a Lenslet Array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005). [CrossRef]
- D.-H. Shin, M. Cho, K.-C. Park, and E.-S. Kim, "Computational technique of volumetric object reconstruction in integral imaging by use of real and virtual image fields," ETRI J. 27, 208-712 (2005). [CrossRef]
- P. Marziliano, F. Dufaux, S. Winkler, and T. Ebrahimi, "A no-reference perceptual blur metric," in the Proceedings of the International Conference on Image Processing, 3, 57-60 (2002).
- Y. C. Chung, J. M. Wang, R. R. Bailey, and S. W. Chen, "A non-parametric blur measure based on edge analysis for image processing applications," in the Proceedings of IEEE Conference on Cybernetics and Intelligent Systems (IEEE, 2004), 1, pp. 356-360.
- R. Youmaran, A. Adler, "Using red-eye to improve face detection in low quality video image," IEEE Canadian Conference on Electrical and Computer Engineering (IEEE, 2006), pp. 1940-1943.
- Z. Rahman, D. J. Jobson, and G. A. Woodell, "A multiscale retinex for color rendition and dynamic range compression," NASA Langley Technical Report (1996).

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

### Figures

Fig. 1. |
Fig. 2. |
Fig. 3. |

Fig. 4. |
Fig. 5. |
Fig. 6. |

Fig. 7. |
Fig. 8. |
Fig. 9. |

Fig. 10. |
Fig. 11. |
Fig. 12. |

Fig. 13. |
||

« Previous Article | Next Article »

OSA is a member of CrossRef.