OSA's Digital Library

Optics Express

Optics Express

  • Editor: J. H. Eberly
  • Vol. 8, Iss. 6 — Mar. 12, 2001
  • pp: 322–327
« Show journal navigation

Subpixel microscopic deformation analysis using correlation and artificial neural networks

Mark C. Pitter, Chung W. See, and Michael G. Somekh  »View Author Affiliations


Optics Express, Vol. 8, Issue 6, pp. 322-327 (2001)
http://dx.doi.org/10.1364/OE.8.000322


View Full Text Article

Acrobat PDF (912 KB)





Browse Journals / Lookup Meetings

Browse by Journal and Year


   


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools

Share
Citations

Abstract

Microscopic deformation analysis has been performed using digital image correlation and artificial neural networks (ANNs). Cross-correlations of small image regions before and after deformation contain a peak, the position of which indicates the displacement to pixel accuracy. Subpixel resolution has been achieved here by nonintegral pixel shifting and by training ANNs to estimate the fractional part of the displacement. Results from displaced and thermally stressed microelectronic devices indicate these techniques can achieve comparable accuracies to other subpixel techniques and that the use of ANNs can facilitate very fast analysis without knowledge of the analytical form of the image correlation function.

© Optical Society of America

1. Introduction

Digital image correlation is an established technique for the measurement of in-plane displacement and strain fields. Full-field deformation between two images can be calculated directly by cross-correlating small image regions or subimages [1

1. D. Vogel, A. Schubert, W. Faust, R. Dudek, and B. Michel, “MicroDAC - a novel approach to measure in situ deformation fields of microscopic scale,” Microelectron. Reliab. 36, 1939–1942 (1996). [CrossRef]

7

7. T. C. Chu, W. F. Ranson, M. A. Sutton, and W. H. Peters, “Applications of digital-image-correlation techniques to experimental mechanics,” Exp. Mech. 25, 232–244 (1985). [CrossRef]

]. The position of the correlation peak indicates the displacement and strain fields can be calculated by differentiation. Image correlation techniques have been applied to many imaging modalities including optical microscopy [1

1. D. Vogel, A. Schubert, W. Faust, R. Dudek, and B. Michel, “MicroDAC - a novel approach to measure in situ deformation fields of microscopic scale,” Microelectron. Reliab. 36, 1939–1942 (1996). [CrossRef]

], SEM [1

1. D. Vogel, A. Schubert, W. Faust, R. Dudek, and B. Michel, “MicroDAC - a novel approach to measure in situ deformation fields of microscopic scale,” Microelectron. Reliab. 36, 1939–1942 (1996). [CrossRef]

] and STM [2

2. G. Vendroux, N. Schmidt, and W. G. Knauss, “Submicron deformation field measurements: Part 3. demonstration of deformation determinations,” Exp. Mech. 38, 154–160 (1998). [CrossRef]

] and used for applications as diverse as elastography of tissue [3

3. J. M. Schmitt, “OCT elastography: imaging microscopic deformation and strain of tissue,” Opt. Express 3, 199–211 (1998), http://www.opticsexpress.org/oearchive/source/5793.htm. [CrossRef] [PubMed]

] and real-time velocity measurement [4

4. Y. Kobayashi, T. Takemori, N. Mukohzaka, N. Yoshida, and S. Fukishima, “Real-time velocity measurement by the use of a speckle-pattern correlation system that incorporates a ferroelectric liquid-crystal spatial light modulator,” Appl. Opt. 33, 2785–2794 (1994). [CrossRef] [PubMed]

].

The motivation behind the work in this paper was the need for a non-contacting optical system to analyze thermal deformation in microelectronics devices. Microscopic deformation analysis using correlation (MDAC or microDAC [1

1. D. Vogel, A. Schubert, W. Faust, R. Dudek, and B. Michel, “MicroDAC - a novel approach to measure in situ deformation fields of microscopic scale,” Microelectron. Reliab. 36, 1939–1942 (1996). [CrossRef]

]) was chosen because, unlike alternative optical techniques such as microscopic moiré [8

8. B. Han, “Recent advances of moiré and microscopic moiré interferometry for thermal deformation analyses of microelectronic devices,” Exp. Mech. 38, 278–288 (1998).

], no special sample preparation, vibration isolation or complex optics are required. In fact, the algorithms described here could be applied to the output of any imaging system, provided sufficient correlation was present between subsequent images.

Although in principle an optical MDAC system can be used to achieve high strain measurement throughput without requiring special skills of the operator, it does have some inherent disadvantages when compared with microscopic moiré. The first of these results from the need to use a subimage of typically 16×16 to 64×64 pixels to calculate each displacement. Subimages can be overlapped to some extent, but the lateral resolution of MDAC is always much lower than the optical resolution of the imaging system. The second restriction when using MDAC is the computational overhead of interpolating the results to obtain subpixel accuracy. Typically, the true displacement between two digitized subimages will contain some non-integral element and locating the correlation peak directly from the digital correlation is only accurate to ±0.5 pixels. This is insufficient for many purposes, including the analysis of thermal deformation in microelectronics devices. Since the sensitivity of MDAC scales with magnification, lateral resolution can be improved by using an array of high resolution images to cover the required field of view. But naturally this greatly increases the computational demands on the system.

2. MDAC using successive approximation nonintegral pixel shifting

In Ref. 5, Sjödahl describes using a combination of curve-fitting to the correlation peak and nonintegral pixel shifting of subimages to achieve subpixel displacement sensitivity with images of laser speckle patterns. We developed SA-NPS in an attempt to replicate the high accuracy of Sjödahl’s technique without the need for an analytical expression to describe the shape of the correlation peak. It has been demonstrated that NPS can be used to increase the overlap of the initial and deformed images. Aligning the subimages to subpixel accuracy prior to calculating the correlation coefficient is essential if high precision is required. It removes a displacement dependent systematic error from the curve-fitting algorithm and allows displacement resolution of as little as 0.01 pixels [5

5. M. Sjödahl, “Electronic speckle photography: increased accuracy by nonintegral pixel shifting,” Appl. Opt. 33, 6667–6673 (1994). [CrossRef] [PubMed]

, 6

6. M. Sjödahl, “Accuracy in electronic speckle photography,” Appl. Opt. 36, 2875–2885 (1997). [CrossRef] [PubMed]

]. However, laser speckle pattern images are well understood in terms of feature size and shape and are typically of high contrast [9

9. J. W. Goodman, Statistical optics (Wiley, New York, 1985).

]. The images we wished to analyze, and hence the shape of their correlation functions, are less easily characterized and are typically broader. This is demonstrated in Figure 1 which shows two typical micrographs of cross-sections through a plastic quad flat pack (PQFP) microelectronic device. Sample 1, an area of packaging material, possesses some of the properties of laser speckle patterns, having a fine, homogenous structure. Sample 2 however contains regions of very different image properties. Inspection of correlation peaks generated from the two samples revealed much greater variation in sharpness and d.c. background for sample 2. This fact made it very difficult to select a suitable and reliable function or family of functions to curve-fit to the correlation peaks.

Fig. 1. Micrographs (320×320µm) of cross-sections through an PQFP microelectronics device. Sample 1 is packaging material and possesses a fine homogenous structure. Sample 2 contains regions of copper (lower area), silicon (upper right), packaging (upper left) and filler (center).

3. MDAC using artificial neural networks

Artificial neural networks are biologically inspired electronic circuit architectures which can be used to build a generalizable model of a data set [11

11. K. Swingler, Applying neural networks - a practical guide (Academic Press, London1996).

]. Model building consists of ‘training’ the network by exposing it to examples of the data. Training can be ‘supervised’ where the network is repeatedly shown examples of input data together with the desired output, or ‘unsupervised’ where the network is used to find subtle relationships within the input data. Many basic ANN architectures can be simulated on a standard PC with widely available code. The networks used in this paper were trained and tested with code inherited from Ref. 12.

We trained and tested a range of MLP architectures with data generated from images of sample 1 and sample 2 in an attempt to create a robust yet accurate model. Training sets were generated by randomly selecting a 24×24 pixel subimage from an image of the sample, subpixel shifting a copy by a known, random amount and cross-correlating with the original subimage. Two MLP input vectors were stored (one each for the x and y displacements) along with the actual subpixel displacements. Training sets of 10,000 examples were generated, along with independent test sets. The MLP input vectors were derived from the central row and column of the cross-correlation peak. Attempts to train MLPs using absolute correlation values failed, however using the gradients of the correlation peak was successful. Preprocessing the MLP input data by calculating the difference between adjacent correlation values reduced the network’s task from differentiating and estimating the intercept with the horizontal axis to simply estimating the intercept.

Fig. 2. Test performance of several MLP architectures on sample 1 (lower plots) and sample 2.

The test set performance of MLPs for a range of input and hidden node configurations is shown in Fig. 2. It should be noted that the back-propagation algorithm does not guarantee to locate the global error minima, so several networks were trained for each architecture in an attempt to optimize performance. The number of network inputs (I) determines the number of correlation gradient values used to calculate the displacement. The number of hidden nodes (H) in combination with I determines the number of weights, W=I(H+1) and hence the complexity of the network. It can be seen from Fig. 2 that performance increases with both the number of inputs and with the total number of weights, but that a significantly more complex network is needed to analyze sample 2 than sample 1. For sample 1 (lower five plots) a network of just 21 weights (6 input and 3 hidden nodes) performs very nearly as well as one containing 88 weights (10 input and 8 hidden nodes). An optimal ANN has as few weights as possible while achieving the required performance, so for sample 1 a network with 6 input and 3 hidden nodes was selected (denoted MLP-1) and for sample 2 a network with 8 inputs and 8 hidden units was used (MLP-2) for the results in Sect. 4. The average error of these ANNs on computer generated test data was 0.023 and 0.044 pixels respectively.

A useful feature of the MLP architecture is its speed of operation once training is completed. Calculating the output of an MLP requires one forward pass through the network. Mathematically, this amounts to multiplying the input vector (I elements) by a matrix containing the weights between input and hidden nodes (I×H elements). The result is passed through a non-linear operator (typically a hyperbolic tangent or sigmoidal function) and then multiplied by a H element vector containing the weights between the hidden layer and the output node. A final pass though a non-linear operator generates the output.

4. Results and discussion

To assess the baseline performance of SA-NPS and the MLPs on real images, the techniques were first used to measure pure displacement in samples 1 and 2. The results displayed below were generated by imaging the sample at 20 displacements between 1 and 16 pixels (0.6 and 10 µm). For each image the error was calculated as the standard deviation of the displacement results for each subimage. The errors quoted below are the mean errors over the 20 measurements. Imaging was performed using a 0.30 N.A. (x10) microscope objective illumination with an LED (λpeak=590 nm). The object pixel size was 628 nm to ensure correct image sampling.

Table 1. Performance of successive approximation NPS MDAC for pure translation.

table-icon
View This Table
| View All Tables

Table 1 illustrates both the high accuracy possible with NPS and the trade-off between lateral resolution (related to the subimage size) and precision. It can also be seen that SA-NPS performs better on the more homogenous sample. Table 2 summarizes the results obtained using MLP-1 and MLP-2. A single application of a network (1 iteration) produces inferior results to SA-NPS, but with a reduction in the computational overhead of greater than an order of magnitude. It was found that more accurate measurements were obtained by using the results of the first MLP iteration to subpixel shift the deformed subimage and improve overlap. A second iteration with the MLP reduced the mean error and also greatly attenuated displacement dependent systematic errors as reported in Ref. 5. The results in Table 2 show a simple but specialized filter (MLP-1) can achieve high precision on a homogenous sample, but performs very badly on a more complicated image. A more complex, general-purpose filter (MLP-2) can attain reasonable accuracy on a inhomogeneous sample without sacrificing significant performance on simple images.

Table 2. Performance of ANN MDAC for pure translation.

table-icon
View This Table
| View All Tables

The results of applying SA-NPS and MLP-2 (with 2 iterations) to images of a thermally stressed PQFP are shown in Figs. 3 and 4. It can be seen that SA-NPS appears to have performed poorly on the packaging area of the device. This is probably due to the very low light levels in this region. The MLP seems to have less problem here as its training data contained examples from this region. The MLP seems to work least well on the filler area and it is possible a specially trained network may help in this region.

5. Summary

Two techniques for subpixel MDAC have been described and tested. The first of these uses SA-NPS to realize high accuracy without the need for an analytical function to describe the form of the correlation peak. However, its use is limited by the amount of computer processing required to implement it. The second technique uses specially trained artificial neural networks to achieve subpixel MDAC of comparable accuracy to SA-NPS but with a much reduced computational overhead. The results in Sect. 4 establish the feasibility of the neural approach. They indicate that an MLP trained over a wide range of input data can perform well, but suggest that more specialized filters may perform better. Future work will investigate the potential of real-time MDAC using ANNs and adaptive MDAC using a categorization unit with a bank of specialized filters.

Fig. 3. Unfiltered in-plane displacement map calculated using NPS with a 24×24 subimage for a temperature rise from 20 °C to 50 °C. The length and direction of the arrows indicate the magnitude and direction of the displacement for each subimage.
Fig. 4. Unfiltered in-plane displacement map calculated using MLP-2 for a temperature rise from 20 °C to 50 °C. The displacement fields of Figs. 3 and 4 may be smoothed and differentiated to produce strain maps.

Acknowledgements

This work was partially supported by the European Union under the Fifth Framework Programme for RTD in the specific programme Competitive and Sustainable Growth, project number GRD1-1999-10xxx OPTIMISM under contract number G6RD-CT99-0179.

References and links

1.

D. Vogel, A. Schubert, W. Faust, R. Dudek, and B. Michel, “MicroDAC - a novel approach to measure in situ deformation fields of microscopic scale,” Microelectron. Reliab. 36, 1939–1942 (1996). [CrossRef]

2.

G. Vendroux, N. Schmidt, and W. G. Knauss, “Submicron deformation field measurements: Part 3. demonstration of deformation determinations,” Exp. Mech. 38, 154–160 (1998). [CrossRef]

3.

J. M. Schmitt, “OCT elastography: imaging microscopic deformation and strain of tissue,” Opt. Express 3, 199–211 (1998), http://www.opticsexpress.org/oearchive/source/5793.htm. [CrossRef] [PubMed]

4.

Y. Kobayashi, T. Takemori, N. Mukohzaka, N. Yoshida, and S. Fukishima, “Real-time velocity measurement by the use of a speckle-pattern correlation system that incorporates a ferroelectric liquid-crystal spatial light modulator,” Appl. Opt. 33, 2785–2794 (1994). [CrossRef] [PubMed]

5.

M. Sjödahl, “Electronic speckle photography: increased accuracy by nonintegral pixel shifting,” Appl. Opt. 33, 6667–6673 (1994). [CrossRef] [PubMed]

6.

M. Sjödahl, “Accuracy in electronic speckle photography,” Appl. Opt. 36, 2875–2885 (1997). [CrossRef] [PubMed]

7.

T. C. Chu, W. F. Ranson, M. A. Sutton, and W. H. Peters, “Applications of digital-image-correlation techniques to experimental mechanics,” Exp. Mech. 25, 232–244 (1985). [CrossRef]

8.

B. Han, “Recent advances of moiré and microscopic moiré interferometry for thermal deformation analyses of microelectronic devices,” Exp. Mech. 38, 278–288 (1998).

9.

J. W. Goodman, Statistical optics (Wiley, New York, 1985).

10.

J. Millman and C. C. Halkias, Integrated electronics (McGraw-Hill, Tokyo1971), Sect. 17–20.

11.

K. Swingler, Applying neural networks - a practical guide (Academic Press, London1996).

12.

J. Rogers, Object-oriented neural networks in C++ (Academic Press, San Diego1997), Chap. 5.

13.

R. C. Eberhart and R. W. Dobbins, Neural network PC tools - a practical guide (Academic Press, San Diego 1990), Chaps. 1014.

OCIS Codes
(100.2000) Image processing : Digital image processing
(180.0180) Microscopy : Microscopy

ToC Category:
Research Papers

History
Original Manuscript: January 17, 2001
Published: March 12, 2001

Citation
Mark Pitter, Chung Wah See, and Michael Somekh, "Subpixel microscopic deformation analysis using correlation and artificial neural networks," Opt. Express 8, 322-327 (2001)
http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-8-6-322


Sort:  Journal  |  Reset  

References

  1. D. Vogel, A. Schubert, W. Faust, R. Dudek and B. Michel, “MicroDAC - a novel approach to measure in situ deformation fields of microscopic scale,” Microelectron. Reliab. 36, 1939-1942 (1996). [CrossRef]
  2. G. Vendroux, N. Schmidt and W. G. Knauss, “Submicron deformation field measurements: Part 3. demonstration of deformation determinations,” Exp. Mech. 38, 154-160 (1998). [CrossRef]
  3. J. M. Schmitt, “OCT elastography: imaging microscopic deformation and strain of tissue,” Opt. Express 3, 199-211 (1998), http://www.opticsexpress.org/oearchive/source/5793.htm. [CrossRef] [PubMed]
  4. Y. Kobayashi, T. Takemori, N. Mukohzaka, N. Yoshida and S. Fukishima, “Real-time velocity measurement by the use of a speckle-pattern correlation system that incorporates a ferroelectric liquid-crystal spatial light modulator,” Appl. Opt. 33, 2785-2794 (1994). [CrossRef] [PubMed]
  5. M. Sjödahl, “Electronic speckle photography: increased accuracy by nonintegral pixel shifting,” Appl. Opt. 33, 6667-6673 (1994). [CrossRef] [PubMed]
  6. M. Sjödahl, “Accuracy in electronic speckle photography,” Appl. Opt. 36, 2875-2885 (1997). [CrossRef] [PubMed]
  7. T. C. Chu, W. F. Ranson, M. A. Sutton and W. H. Peters, “Applications of digital-image-correlation techniques to experimental mechanics,” Exp. Mech. 25, 232-244 (1985). [CrossRef]
  8. B. Han, “Recent advances of moiré and microscopic moiré interferometry for thermal deformation analyses of microelectronic devices,” Exp. Mech. 38, 278-288 (1998).
  9. J. W. Goodman, Statistical optics (Wiley, New York, 1985).
  10. J. Millman and C. C. Halkias, Integrated electronics (McGraw-Hill, Tokyo 1971), Sect. 17-20.
  11. K. Swingler, Applying neural networks - a practical guide (Academic Press, London 1996).
  12. J. Rogers, Object-oriented neural networks in C++ (Academic Press, San Diego 1997), Chap. 5.
  13. R. C. Eberhart and R. W. Dobbins, Neural network PC tools - a practical guide (Academic Press, San Diego 1990), Chaps. 10-14.

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

Figures

Fig. 1. Fig. 2. Fig. 3.
 
Fig. 4.
 

« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited