## Simultaneous multiple-emitter fitting for single molecule super-resolution imaging |

Biomedical Optics Express, Vol. 2, Issue 5, pp. 1377-1393 (2011)

http://dx.doi.org/10.1364/BOE.2.001377

Acrobat PDF (5471 KB)

### Abstract

Single molecule localization based super-resolution imaging techniques require repeated localization of many single emitters. We describe a method that uses the maximum likelihood estimator to localize multiple emitters simultaneously within a single, two-dimensional fitting sub-region, yielding an order of magnitude improvement in the tolerance of the analysis routine with regards to the single-frame active emitter density. Multiple-emitter fitting enables the overall performance of single-molecule super-resolution to be improved in one or more of several metrics that result in higher single-frame density of localized active emitters. For speed, the algorithm is implemented on Graphics Processing Unit (GPU) architecture, resulting in analysis times on the order of minutes. We show the performance of multiple emitter fitting as a function of the single-frame active emitter density. We describe the details of the algorithm that allow robust fitting, the details of the GPU implementation, and the other imaging processing steps required for the analysis of data sets.

© 2011 OSA

## 1. Introduction

*λ*/2NA or approximately 250 nm [1

1. S. W. Hell, “Far-field optical nanoscopy,” Science **316**, 1153–1158 (2007). [CrossRef] [PubMed]

5. L. Schermelleh, R. Heintzmann, and H. Leonhardt, “A guide to super-resolution fluorescence microscopy,” J. Cell Biol. **190**, 165–175 (2010). [CrossRef] [PubMed]

7. B. C. Lagerholm, L. Averett, G. E. Weinreb, K. Jacobson, and N. L. Thompson, “Analysis method for measuring submicroscopic distances with blinking quantum dots,” Biophys. J. **91**, 3050–3060 (2006). [CrossRef] [PubMed]

8. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science **313**, 1642–1645 (2006). [CrossRef] [PubMed]

9. S. T. Hess, T. P. K. Girirajan, and M. D. Mason, “Ultra-high resolution imaging by fluorescence photoactivation localization microscopy,” Biophys. J. **91**, 4258–4272 (2006). [CrossRef] [PubMed]

10. M. J. Rust, M. Bates, and X. W. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm),” Nat. Methods **3**, 793–795 (2006). [CrossRef] [PubMed]

4. G. Patterson, M. Davidson, S. Manley, and J. Lippincott-Schwartz, “Superresolution imaging using single-molecule localization,” Annu. Rev. Phys. Chem. **61**, 345–367 (2010). [CrossRef] [PubMed]

11. J. Vogelsang, C. Steinhauer, C. Forthmann, I. H. Stein, B. Person-Skegro, T. Cordes, and P. Tinnefeld, “Make them blink: probes for super-resolution microscopy,” Chemphyschem **11**, 2475–2490 (2010). [CrossRef] [PubMed]

12. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated-emission—stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. **19**, 780–782 (1994). [CrossRef] [PubMed]

13. S. Hell and E. H. K. Stelzer, “Fundamental improvement of resolution with a 4pi-confocal fluorescence microscope using 2-photon excitation,” Opt. Commun. **93**, 277–282 (1992). [CrossRef]

14. M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U. S. A. **102**, 13081–13086 (2005). [CrossRef] [PubMed]

^{6}emitters. To simplify and speed analysis, conventional analysis approaches only attempt to localize well separated, single emitter events and data that does not fit this model is rejected. Experimental conditions must then be optimized to give a single-frame active emitter density that makes best use of the data and yet minimizes acquisition time [15

15. A. R. Small, “Theoretical limits on errors and acquisition rates in localizing switchable fluorophores,” Biophys. J. **96**, L16–L18 (2009). [CrossRef] [PubMed]

*μ*m

^{−2}, more than 55% of 8

*σ*× 8

_{PSF}*σ*(

_{PSF}*σ*

_{PSF}= 127 nm) sub-regions contain 2 or more active emitters. Such nearby or overlapping emission patterns could result in a failure of the single emitter model and the data not being used in the SR image reconstruction. The distribution of the number of emitters found within these 8

*σ*× 8

_{PSF}*σ*sub-regions (

_{PSF}*σ*

_{PSF}= 127 nm) as a function of density is also shown in Fig. 1 and illustrates that with increasing active emitter density, isolated single-emitter events become rare and therefore a majority of the position estimates will get discarded due to an unacceptable fit to a single emitter model. It is clear that a multiple-emitter fitting approach would enable the analysis of images containing higher single-frame density of active emitters. Analysis of multiple emitters simultaneously in one sub-region does not necessarily impact the position uncertainties as visually overlapping emitters (around 100 nm between emitter centers) can be localized with similar uncertainties [16

16. S. Ram, E. S. Ward, and R. J. Ober, “Beyond rayleigh’s criterion: A resolution measure with application to single-molecule microscopy,” Proc. Natl. Acad. Sci. U.S.A. **103**, 4457–4462 (2006). [CrossRef] [PubMed]

17. J. Chao, S. Ram, E. S. Ward, and R. J. Ober, “A comparative study of high resolution microscopy imaging modalities using a three-dimensional resolution measure,” Opt. Express **17**, 24377–24402 (2009). [CrossRef]

15. A. R. Small, “Theoretical limits on errors and acquisition rates in localizing switchable fluorophores,” Biophys. J. **96**, L16–L18 (2009). [CrossRef] [PubMed]

21. M. P. Gordon, T. Ha, and P. R. Selvin, “Single-molecule high-resolution imaging with photobleaching,” Proc. Natl. Acad. Sci. U.S.A. **101**, 6462–6465 (2004). [CrossRef] [PubMed]

*N*emitters, where

*N*is varied from

*N*=1, to

*N*=

*N*

_{max}using a process that we will subsequently refer to as Multi-emitter Fitting Analysis (MFA). Based on the log-likelihood, a chi square distributed test statistic is used to either choose one model, or reject all fitting models. In this manuscript, we describe a procedure that allows robust application of the MFA, including model selection criteria, uncertainty calculations, and other procedures for analyzing a SM-SR data set and image reconstruction.

## 2. Theoretical basis for the multi-emitter fitting algorithm

### 2.1. Multiple emitter extension to the pixelized single emitter model

22. B. Zhang, J. Zerubia, and J. C. Olivo-Marin, “Gaussian approximations of fluorescence microscope point-spread function models,” Appl. Opt. **46**, 1819–1829 (2007). [CrossRef] [PubMed]

23. S. Stallinga and B. Rieger, “Accuracy of the gaussian point spread function model in 2d localization microscopy,” Opt. Express **18**, 24461–24476 (2010). [CrossRef] [PubMed]

*σ*

_{0}represents the standard deviation of the Gaussian.

*k*located at a position {

*x,y*} and assumed to rectangular, the expected number of photons in that pixel, which are emitted from a point object in focus, can be calculated by integrating Eq. (1) across the pixel assuming a square shaped pixel. This pixelized single emitter profile is given as: where

*μ*(

_{k}*x, y*) is the expected photon count for a given pixel ‘k’,

*I*

_{0}is the total emitted photon counts expected,

*b*

_{0}is the background and ΔE

*(*

_{x}*x, y*) and ΔE

*(*

_{y}*x, y*) are: where

*x*

_{0}and

*y*

_{0}are emitter positions.

*k*. The expected photon count for pixel

*k*,

*μ*(

_{k}*x, y*) generated by N emitters can then be calculated by summing over the total number of emitters

*N*and is defined as:

### 2.2. Maximum likelihood estimator

*θ*given the data

*D*is modeled as a photon counting process for each pixel, with the expected counts given by the multi-emitter model

*μ*defined in Eq. (4) and the observed counts

_{k}*d*. The maximum likelihood estimator (MLE) is used to estimate the emitter positions {

_{k}*x*,

_{i}*y*}....{

_{i}*x*,

_{N}*y*} and the background fluorescence rate

_{N}*b*

_{0}, giving

*θ̂*= {

*b*

_{0}

*, x*

_{1}

*, y*

_{1},...

*x*,

_{N}*y*}

_{N}*. To ensure robust estimation, we find that it is necessary to confine the intensity parameter*

^{T}*I*=

_{i}*I*

_{0}in Eq. (4), where

*I*

_{0}is obtained from independent measurements.

*θ*can be written for a Poisson noise model as follows [25

_{i}25. C. S. Smith, N. Joseph, B. Rieger, and K. A. Lidke, “Fast, single-molecule localization that achieves theoretically minimum uncertainty,” Nat. Methods **7**, 373–U52 (2010). [CrossRef] [PubMed]

*μ*(

*θ*) are identical in form to those from the single-emitter model and are given in our previous work [25

25. C. S. Smith, N. Joseph, B. Rieger, and K. A. Lidke, “Fast, single-molecule localization that achieves theoretically minimum uncertainty,” Nat. Methods **7**, 373–U52 (2010). [CrossRef] [PubMed]

## 3. The analysis procedure

*N*proposed emitters in a model of

*N*= 1 to

*N*

_{max}are found sequentially where the

*N*emitter model uses position information from the

*N*– 1 emitter model. We generate the p-value from a test statistic based on the log-likelihood ratio (LLR) to compare fits for each model. The model with the highest p-value is selected and the associated uncertainties and fits are determined based on a modified Fisher information matrix. The process is repeated for all frames and a reconstructed image is generated from the estimates by placing bivariate Gaussian shapes at the estimated locations using estimator uncertainties to build the bi-variate covariance matrix. Below we outline these steps in further detail.

### 3.1. Image pre-processing and segmentation

*I*, as follows: where uniform[

*image,q*] represents a uniform filter process with a square kernel size

*q*operating on the 2-D matrix

*image*. The uniform filter acts as a smoothing filter by reassigning the value of each pixel to the average pixel value within the square kernel centered at the pixel position. The analysis is not strongly dependent on the smoothing filter so the uniform filter is chosen for speed. Subtraction means a pixel-wise subtraction between results obtained for each filter process. The second filtering step is performed on the first filtered image

*A*

_{1}as follows: where max[

*image,q*] represents a maximum filter process used to obtain local maximum values within a square kernel size

*q*. Through this process, all pixels within a kernel take the maximum value within the kernel. These two filtered images

*A*

_{1}and

*A*

_{2}are then compared pixel-wise to identify regions of interest:

*A*

_{1}are identified in

*A*

_{3}. Sub-regions of size 6

*σ*× 6

_{PSF}*σ*that are centered at pixels where

_{PSF}*A*

_{3}= 1 are selected for further analysis.

### 3.2. Multi-emitter fitting analysis (MFA)

*N*= 1 model to a

*N*=

*N*

_{max}model. For the

*N*= 1 model, the center of mass of the sub-region is used as the initial position estimate. For the

*N*≠ 1, multi-emitter models, the

*N*– 1 position estimates found in the previous step are used as

*N*– 1 of the initial position estimates. The remaining initial position estimate is found by calculating the residuum image generated by a subtraction of the

*N*– 1 model (Eq. (4)) from the data in the sub-region. If the value of the maximum intensity pixel in the residuum image is low enough to assume that all emitters in the sub-region have been found, the analysis does not proceed further. Otherwise, from the residuum image, the last initial estimate is calculated from the position of the pixel with the maximum count value, giving {

*x*

_{def},

*y*

_{def}} and then is adjusted in a “Push&Pull” process to {

*x*

_{adj}

*, y*

_{adj}} = {

*x*

_{def}±

*σ*

_{PSF}/2,

*y*

_{def}±

*σ*

_{PSF}/2}. If {

*x*

_{def}

*, y*

_{def}} is within

*σ*

_{PSF}of the edge of the sub-region, that position is likely to correspond an emitter outside of the region, and the sign of the adjustment is such to move the adjusted position further away from the center of the sub-region. Otherwise, the sign of the adjustment is such to move the adjusted position towards the center of mass of the

*N*– 1 position estimates. This compensates for the effect that in a

*N*– 1 model of an underlying

*N*emitter system, the estimated positions of N-1 emitters are displaced such that after deflation, the position of the maximum value pixel is biased away from the actual position of that emitter. This effect is illustrated in Fig. 2(b). We found that the ”Push&Pull” adjustment of only one of the initial position estimates is sufficient to allow robust convergence. The initial estimates are then updated using a fixed number of iterations of Eq. (6). After obtaining estimates for each model, models with location estimates outside the fitting boundary, which is a 8

*σ*

_{PSF}× 8

*σ*

_{PSF}square region concentric with image sub-region (red box Fig. 2(b)–2(e)), are discarded. Models with positions estimates within the fitting boundary but outside the data sub-region (black region between red and yellow box in Fig. 2(b)–2(e)) are allowed since emitters located in this region will affect the data sub-region. The position and background estimates, along with their log-likelihood, are saved for each remaining model for a further model selection process.

### 3.3. Model selection

*K*– (2

*N*+ 1) degrees of freedom, where

*K*is the number of pixels in the sub-region and

*N*is the number of emitters in the model. where

*D*represents the sub-region data,

*θ̂*are the MLE estimates and

*L*(

*D|D*) gives the upper limit of likelihood of the data set with Poisson noise (when

*μ*=

_{k}*d*). The model is accepted if it has the maximum chi-square p-value of all models and passes the p-value threshold set by user. Considering that the variance of intensities in real or realistically simulated data would broaden the LLR distribution and thus result in a smaller p-value, typically a small p-value of 10

_{k}^{−3}to 10

^{−6}is used as the threshold in our analysis and is still sufficient to reject incorrect models and the un-converged fit. After obtaining the uncertainty for the position estimates, emitters with estimated positions near (within

*σ*

_{PSF}/2) or outside of the sub-region boundary are discarded. The parameters describing the remaining emitters are passed to the image reconstruction process.

### 3.4. Precision of the estimated parameters

16. S. Ram, E. S. Ward, and R. J. Ober, “Beyond rayleigh’s criterion: A resolution measure with application to single-molecule microscopy,” Proc. Natl. Acad. Sci. U.S.A. **103**, 4457–4462 (2006). [CrossRef] [PubMed]

25. C. S. Smith, N. Joseph, B. Rieger, and K. A. Lidke, “Fast, single-molecule localization that achieves theoretically minimum uncertainty,” Nat. Methods **7**, 373–U52 (2010). [CrossRef] [PubMed]

26. R. J. Ober, S. Ram, and E. S. Ward, “Localization accuracy in single-molecule microscopy,” Biophys. J. **86**, 1185–1200 (2004). [CrossRef] [PubMed]

27. P. Stoica and T. L. Marzetta, “Parameter estimation problems with singular information matrices,” IEEE Trans. Sig. Process. **49**, 87–90 (2001). [CrossRef]

*x*,

_{i}*y*} = {

_{i}*x*,

_{j}*y*}, and near this singular point, can not be used to correctly calculate estimator precision. We implemented a phenomenological correction to the Fisher information matrix by modifying the off diagonal terms that give rise to the singularity. Given our parameter set

_{j}*θ*= {

*b*

_{0},

*x*

_{1},

*y*

_{1},...

*x*,

_{N}*y*}

_{N}*, the corrections are given by:*

^{T}*σ*and

_{i}*σ*are the intermediate precision calculations obtained from

_{j}*F*(

*θ*) assuming

*A*= 0.

*F*(

*θ*), which we designate the modified Fisher Information matrix, replaces the original Fisher Information matrix in our precision calculation process, is non-singular at {

*x*,

_{i}*y*} = {

_{i}*x*,

_{j}*y*} and quickly converges to

_{j}*I*(

*θ*) once far from the point of singularity. Thus it provides reasonable precision estimates in the regions both near and far from the point of singularity.

### 3.5. Filtering and SR image reconstruction

*F*(

*θ*)

^{−1}and indicate the asymmetry of the position uncertainties that arise from the multi-emitter localization process.

## 4. Computational and experimental methods

### 4.1. Hardware and software implementation of analysis routines

29. NVIDIA, “Compute unified device architecture (CUDA),” http://www.nvidia.com/object/cuda_home.html (2007).

**7**, 373–U52 (2010). [CrossRef] [PubMed]

*F*(

*θ*) and its inversion, by LU decomposition with back substitute method [30], are implemented on the GPU executing with one thread per sub-region. The resulting uncertainties for each parameter are passed back to the CPU. The filtering of position estimates by sub-region position and their uncertainties is performed on the CPU. Reconstruction of the SR image is performed in a manner inverse to the sub-region selection. First, in the GPU, an up-sampled sub-region is generated that corresponds to each position estimate and its uncertainties. The bivariate Gaussian shapes for the position estimates are added to the sub-region. All generated up-sampled sub-regions are passed back to the CPU and assembled into a single up-sampled SR image.

### 4.2. Estimator precision and algorithm performance testing

*μ*= 800,

*σ*= 100. A background count rate of 5 count/pixel was added to the image, and then the image was corrupted with Poisson noise. After fitting these images using MFA with a target resolution of 20 nm or 50 nm, the localization fraction was calculated by taking the ratio between the number of correctly localized emitters which is defined as having a registered emitter position near the localized emitter within the target resolution and the total number of emitters in simulation. The error rate of the algorithm was obtained by calculating the ratio between the number of mis-localized emitters which is defined as having no actual emitter position near the localized emitter within the target resolution and the total number of emitters obtained from fitting.

### 4.3. Synthetic data generation

*ρ*

_{0}=5000

*μ*m

^{−2}and off rate

*k*

_{off}= 0.8 frame

^{−1}were used, with varied

*k*

_{on}to generate variations in active densities (

*ρ*

_{active}) according to: A blinking trace was generated for each emitter using the transition rates

*k*

_{on}and

*k*

_{off}for dark to active, and active to dark transitions respectively and were designed to emulate realistic photophysical properties. As in all of our simulations, the active emitters were represented as a 2D Gaussian shapes, with

*σ*=

*σ*

_{PSF}= 1.2 pixels (127 nm). To represent the experimentally observed variation in emitter brightness, for each emitter, the total expected photon count per frame was selected from a normal distribution with

*μ*= 800

*σ*= 100. Shown in Fig. 3 is the single frame intensity distribution of Alexa Fluor 647. A background count rate of 5 count/pixel was added to the image, and then the image was corrupted with Poisson noise. Calculation of the density of active emitters assumes a pixel size of 106 nm, which is the back-projected pixel size in the experimental system.

### 4.4. SM-SR imaging

#### 4.4.1. Cell culture

31. M. Heilemann, S. van de Linde, M. Schuttpelz, R. Kasper, B. Seefeldt, A. Mukherjee, P. Tinnefeld, and M. Sauer, “Subdiffraction-resolution fluorescence imaging with conventional fluorescent probes,” Angew. Chem. Int. Ed. **47**, 6172–6176 (2008). [CrossRef]

#### 4.4.2. Microscopy and data acquisition

*μ*m.

## 5. Results and discussion

### 5.1. Optimal sub-region size and *N*_{max}

_{max}

*σ*

_{PSF}to 8

*σ*

_{PSF}were evaluated in the aspects of both localization fraction and error rate that are defined in section 4.2. Small sub-regions tend to isolate individual emitters from one another better compared to larger sub-regions and thus results in sub-regions containing fewer emitters. However, the smaller area decreased the amount of information that could be used in fitting and thus the error rate increases compared to larger sub-regions. Large sub-regions provide more accurate estimates compared to a smaller subregion but the probability of introducing more emitters within or near the sub-region increases quadratically with the width of the square sub-region. We have tested our algorithm performance under different sub-region sizes, such as 4

*σ*

_{PSF}, 5

*σ*

_{PSF}, 6

*σ*

_{PSF}, 7

*σ*

_{PSF}, 8

*σ*

_{PSF}, various active emitter densities from 0.1

*μ*m

^{−2}to 10

*μ*m

^{−2}, various emitter intensities from 200 to 5000 and various intensity variance. After comparing these plots (data not shown), we found that sub-region size of 6

*σ*

_{PSF}shows the best compromise of error rates and localization fraction.

*N*values ranging from 1 to 8 were tested. Large

_{max}*N*tend to generate a more complex likelihood surface and thus the possibility for the estimator being stuck at a local minimum increases with

_{max}*N*. The complexity introduced by multi-emitter model results in higher error rates and thus

_{max}*N*was restricted to 5 in our analysis.

_{max}### 5.2. Uncertainty estimator performance

*θ*) of Eq. (11) and compared with observed standard deviations. Singularity of the Fisher Information matrix for the multi-component Gaussian model when 2 (or more) emitter centers that have a separation less than 100 nm results in a failure of the CRLB to correctly estimate the precision of the position estimation. This effect is demonstrated in Fig. 4(a). Figure 4(a) also shows that calculations based on F(

*θ*) gave a correct estimator precision (compared to the observed standard deviation of the estimates) in the regions both near and far from the point of singularity of the two emitter model, with only a small but conservative deviation below 50 nm. We also show the performance of F(

*θ*) based precision calculations for random configurations of multiple emitters by looking at the estimator accuracy, defined as the ratio of the precision calculated using F(

*θ*) to the observed standard deviation of the estimates. The cumulative distribution (the normalized integral of the histogram) of the estimator accuracy is shown in Fig. 4(b) and demonstrates that the estimator accuracy distribution (corresponding to the derivative the CDF) of is narrowly peaked around 1 for the 1–3 emitter models (ideal) and is conservative (reported precision is larger than observed standard deviation) on the 4 and 5 emitter models where the estimator accuracy distribution is peaked around 1.1 and 1.2 respectively.

### 5.3. Algorithm performance versus density and intensity distribution

*μ*m

^{2}in our microscope camera setup. By increasing the number of active emitters within the image, density increased from 0.01

*μ*m

^{−2}to 10

*μ*m

^{−2}. Both single (

*N*

_{max}= 1) to multi (

*N*

_{max}= 5) emitter fitting algorithm were performed on these data sets and localization fraction (defined in 4.2) were calculated.

*N*

_{max}= 1 to

*N*

_{max}= 5 is most significant at a densities higher than 1

*μ*m

^{−2}. We note that at high intensities with narrow intensity distribution (Figs. 5(e), 5(f)) the localization error improves, but the localization fraction does not. This is attributed to high sensitivity to model mismatches created by the fixed intensity assumption and emitters outside the fitting window.

### 5.4. Pattern simulation results

*μ*m

^{−2}and 6

*μ*m

^{−2}. These two sets of data were analyzed using

*N*

_{max}= 1 and

*N*

_{max}= 5. Results of the analysis are shown in Fig. 6c through Fig. 6f.

*N*

_{max}= 1 and

*N*

_{max}= 5 are similar. For

*N*

_{max}= 1 shown in Fig. 6(c), 12848 emitters were localized and accepted for use in the SR reconstruction, and for

*N*

_{max}= 5 shown in Fig. 6(d), 30354 emitters were accepted and used. In the high density case, shown in Fig. 6(e) and Fig. 6(f), there was nearly two orders of magnitude (519 versus 33580) more position estimations accepted and used in the reconstruction. As shown in the projections of the SR images, the

*N*

_{max}= 1 fitting performs better near the edges of the structures where the local active emitter density is smaller. It is interesting to note that at the low density,

*N*

_{max}= 5 fits almost 3 times more emitters than

*N*

_{max}= 1 case, and thus the pattern result shows a better resolved structure near the center and provides better resolution compared to

*N*

_{max}= 1 fitting result.

### 5.5. Algorithm speed

_{max}. Tests were performed on two set of data (data size: 128×128×1000) with densities 1

*μ*m

^{−2}and 5

*μ*m

^{−2}. Algorithm execution was divided into several major sections and the run times for each section were recorded. As shown in Table 1, the operation time for MFA

*N*

_{max}= 5 was 176 s for the 1

*μ*m

^{−2}case and 408 s for the 5

*μ*m

^{−2}case. When performing single emitter operation (

*N*

_{max}= 1), this run time decreased dramatically to 17 s and 30 s respectively. This dramatic difference is caused by the complexity introduced by fitting multiple emitters, such as fitting to multiple models, the deflation process, NR iteration on more parameters, the Fisher information modification and also a larger Fisher information matrix. However, the fraction of localization also dramatically increased when comparing single fitting results to multi fitting results as over 100 times more emitters were localized at a density of 5

*μ*m

^{−2}and almost 3 times more at a density of 1

*μ*m

^{−2}.

### 5.6. Imaging of actin structures

*N*

_{max}= 5) compared with single emitter analysis (

*N*

_{max}= 1). For samples with high labeling densities, such as those possible when using small molecule fluorescent probes such as Alexa Fluor 647 phalloidin, regions of interest that could be seen using conventional microscopy (Fig. 7(b)), may appear to be discontinuous when analyzed using

*N*

_{max}= 1 that can not process high active densities (Fig. 7(d)). By analyzing these data sets using MFA (

*N*

_{max}= 5), less events were discarded. The reconstructed SR image from

*N*

_{max}= 5 showed more continuous structures and ultimately, enabled finer detail of the underlying protein distributions to be revealed (Fig. 7(e)). It is shown in Fig. 7(c) that although the branching structures were resolved nicely using

*N*

_{max}= 1, structures toward the middle backbone can’t be resolved, because the backbone structure are composed of intercrossing actin fibers and thus possessed a higher local emitter density than isolated line structures. As shown in Fig. 7(e), MFA (

*N*

_{max}= 5) achieved to resolve the backbone structure better than single emitter fitting algorithm (

*N*

_{max}= 1).

## 6. Conclusion

*μ*m

^{−2}. This capability relaxes an important constraint in SM-SR, allowing an order of magnitude improvement in the speed of acquisition and/or the maximum supported duty cycle of the emitters. Although our approach is based on a maximum likelihood estimate, robust estimation of multiple emitter positions also requires strategies such as making good initial estimates, making accurate uncertainty estimates and the model selection and rejection algorithm. Higher density imaging allows shorter acquisition times, but results in more computational complexity in analysis. By implementing key analysis steps in GPU hardware, we show the MFA method can complete the analysis of real data sets on the time scale of minutes.

## References and links

1. | S. W. Hell, “Far-field optical nanoscopy,” Science |

2. | K. R. Chi, “Microscopy: ever-increasing resolution,” Nature |

3. | B. Huang, M. Bates, and X. W. Zhuang, “Super-resolution fluorescence microscopy,” Annu. Rev. Biochem. |

4. | G. Patterson, M. Davidson, S. Manley, and J. Lippincott-Schwartz, “Superresolution imaging using single-molecule localization,” Annu. Rev. Phys. Chem. |

5. | L. Schermelleh, R. Heintzmann, and H. Leonhardt, “A guide to super-resolution fluorescence microscopy,” J. Cell Biol. |

6. | K. A. Lidke, B. Rieger, T. M. Jovin, and R. Heintzmann, “Superresolution with quantum dots: enhanced localization in fluorescence microscopy by exploitation of quantum dot blinking,” Biophys. J. |

7. | B. C. Lagerholm, L. Averett, G. E. Weinreb, K. Jacobson, and N. L. Thompson, “Analysis method for measuring submicroscopic distances with blinking quantum dots,” Biophys. J. |

8. | E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science |

9. | S. T. Hess, T. P. K. Girirajan, and M. D. Mason, “Ultra-high resolution imaging by fluorescence photoactivation localization microscopy,” Biophys. J. |

10. | M. J. Rust, M. Bates, and X. W. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm),” Nat. Methods |

11. | J. Vogelsang, C. Steinhauer, C. Forthmann, I. H. Stein, B. Person-Skegro, T. Cordes, and P. Tinnefeld, “Make them blink: probes for super-resolution microscopy,” Chemphyschem |

12. | S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated-emission—stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. |

13. | S. Hell and E. H. K. Stelzer, “Fundamental improvement of resolution with a 4pi-confocal fluorescence microscope using 2-photon excitation,” Opt. Commun. |

14. | M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U. S. A. |

15. | A. R. Small, “Theoretical limits on errors and acquisition rates in localizing switchable fluorophores,” Biophys. J. |

16. | S. Ram, E. S. Ward, and R. J. Ober, “Beyond rayleigh’s criterion: A resolution measure with application to single-molecule microscopy,” Proc. Natl. Acad. Sci. U.S.A. |

17. | J. Chao, S. Ram, E. S. Ward, and R. J. Ober, “A comparative study of high resolution microscopy imaging modalities using a three-dimensional resolution measure,” Opt. Express |

18. | J. A. Högbom, “Aperture synthesis with a non-regular distribution of interferometer baselines,” Astron. Astrophys. Suppl. |

19. | A. Serge, N. Bertaux, H. Rigneault, and D. Marguet, “Dynamic multiple-target tracing to probe spatiotemporal cartography of cell membranes,” Nat. Methods |

20. | X. H. Qu, D. Wu, L. Mets, and N. F. Scherer, “Nanometer-localized multiple single-molecule fluorescence microscopy,” Proc. Natl. Acad. Sci. U.S.A. |

21. | M. P. Gordon, T. Ha, and P. R. Selvin, “Single-molecule high-resolution imaging with photobleaching,” Proc. Natl. Acad. Sci. U.S.A. |

22. | B. Zhang, J. Zerubia, and J. C. Olivo-Marin, “Gaussian approximations of fluorescence microscope point-spread function models,” Appl. Opt. |

23. | S. Stallinga and B. Rieger, “Accuracy of the gaussian point spread function model in 2d localization microscopy,” Opt. Express |

24. | A. Van den Bos and C. Ebooks, |

25. | C. S. Smith, N. Joseph, B. Rieger, and K. A. Lidke, “Fast, single-molecule localization that achieves theoretically minimum uncertainty,” Nat. Methods |

26. | R. J. Ober, S. Ram, and E. S. Ward, “Localization accuracy in single-molecule microscopy,” Biophys. J. |

27. | P. Stoica and T. L. Marzetta, “Parameter estimation problems with singular information matrices,” IEEE Trans. Sig. Process. |

28. | C. L. L. Hendriks, L. J. van Vliet, B. Rieger, G. M. P. van Kempen, and M. van Ginkel, “Dipimage: a scientific image processing toolbox for MATLAB,” Quantitative Imaging Group, Faculty of Applied Sciences, Delft University of Technology, Delft, The Netherlands (1999). |

29. | NVIDIA, “Compute unified device architecture (CUDA),” http://www.nvidia.com/object/cuda_home.html (2007). |

30. | W. H. Press, S. L. A. Teukolsky, B. N. P. Flannery, and W. M. T. Vetterling, |

31. | M. Heilemann, S. van de Linde, M. Schuttpelz, R. Kasper, B. Seefeldt, A. Mukherjee, P. Tinnefeld, and M. Sauer, “Subdiffraction-resolution fluorescence imaging with conventional fluorescent probes,” Angew. Chem. Int. Ed. |

**OCIS Codes**

(100.3010) Image processing : Image reconstruction techniques

(100.6640) Image processing : Superresolution

(180.2520) Microscopy : Fluorescence microscopy

**ToC Category:**

Microscopy

**History**

Original Manuscript: January 31, 2011

Revised Manuscript: April 9, 2011

Manuscript Accepted: April 14, 2011

Published: April 29, 2011

**Citation**

Fang Huang, Samantha L. Schwartz, Jason M. Byars, and Keith A. Lidke, "Simultaneous multiple-emitter fitting for single molecule super-resolution imaging," Biomed. Opt. Express **2**, 1377-1393 (2011)

http://www.opticsinfobase.org/boe/abstract.cfm?URI=boe-2-5-1377

Sort: Year | Journal | Reset

### References

- S. W. Hell, “Far-field optical nanoscopy,” Science 316, 1153–1158 (2007). [CrossRef] [PubMed]
- K. R. Chi, “Microscopy: ever-increasing resolution,” Nature 462, 675–678 (2009). [CrossRef] [PubMed]
- B. Huang, M. Bates, and X. W. Zhuang, “Super-resolution fluorescence microscopy,” Annu. Rev. Biochem. 78, 993–1016 (2009). [CrossRef] [PubMed]
- G. Patterson, M. Davidson, S. Manley, and J. Lippincott-Schwartz, “Superresolution imaging using single-molecule localization,” Annu. Rev. Phys. Chem. 61, 345–367 (2010). [CrossRef] [PubMed]
- L. Schermelleh, R. Heintzmann, and H. Leonhardt, “A guide to super-resolution fluorescence microscopy,” J. Cell Biol. 190, 165–175 (2010). [CrossRef] [PubMed]
- K. A. Lidke, B. Rieger, T. M. Jovin, and R. Heintzmann, “Superresolution with quantum dots: enhanced localization in fluorescence microscopy by exploitation of quantum dot blinking,” Biophys. J. 88, 346a–346a (2005).
- B. C. Lagerholm, L. Averett, G. E. Weinreb, K. Jacobson, and N. L. Thompson, “Analysis method for measuring submicroscopic distances with blinking quantum dots,” Biophys. J. 91, 3050–3060 (2006). [CrossRef] [PubMed]
- E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science 313, 1642–1645 (2006). [CrossRef] [PubMed]
- S. T. Hess, T. P. K. Girirajan, and M. D. Mason, “Ultra-high resolution imaging by fluorescence photoactivation localization microscopy,” Biophys. J. 91, 4258–4272 (2006). [CrossRef] [PubMed]
- M. J. Rust, M. Bates, and X. W. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm),” Nat. Methods 3, 793–795 (2006). [CrossRef] [PubMed]
- J. Vogelsang, C. Steinhauer, C. Forthmann, I. H. Stein, B. Person-Skegro, T. Cordes, and P. Tinnefeld, “Make them blink: probes for super-resolution microscopy,” Chemphyschem 11, 2475–2490 (2010). [CrossRef] [PubMed]
- S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated-emission—stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19, 780–782 (1994). [CrossRef] [PubMed]
- S. Hell and E. H. K. Stelzer, “Fundamental improvement of resolution with a 4pi-confocal fluorescence microscope using 2-photon excitation,” Opt. Commun. 93, 277–282 (1992). [CrossRef]
- M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U. S. A. 102, 13081–13086 (2005). [CrossRef] [PubMed]
- A. R. Small, “Theoretical limits on errors and acquisition rates in localizing switchable fluorophores,” Biophys. J. 96, L16–L18 (2009). [CrossRef] [PubMed]
- S. Ram, E. S. Ward, and R. J. Ober, “Beyond rayleigh’s criterion: A resolution measure with application to single-molecule microscopy,” Proc. Natl. Acad. Sci. U.S.A. 103, 4457–4462 (2006). [CrossRef] [PubMed]
- J. Chao, S. Ram, E. S. Ward, and R. J. Ober, “A comparative study of high resolution microscopy imaging modalities using a three-dimensional resolution measure,” Opt. Express 17, 24377–24402 (2009). [CrossRef]
- J. A. Högbom, “Aperture synthesis with a non-regular distribution of interferometer baselines,” Astron. Astrophys. Suppl. 15, 417 (1974).
- A. Serge, N. Bertaux, H. Rigneault, and D. Marguet, “Dynamic multiple-target tracing to probe spatiotemporal cartography of cell membranes,” Nat. Methods 5, 687–694 (2008). [CrossRef] [PubMed]
- X. H. Qu, D. Wu, L. Mets, and N. F. Scherer, “Nanometer-localized multiple single-molecule fluorescence microscopy,” Proc. Natl. Acad. Sci. U.S.A. 101, 11298–11303 (2004). [CrossRef] [PubMed]
- M. P. Gordon, T. Ha, and P. R. Selvin, “Single-molecule high-resolution imaging with photobleaching,” Proc. Natl. Acad. Sci. U.S.A. 101, 6462–6465 (2004). [CrossRef] [PubMed]
- B. Zhang, J. Zerubia, and J. C. Olivo-Marin, “Gaussian approximations of fluorescence microscope point-spread function models,” Appl. Opt. 46, 1819–1829 (2007). [CrossRef] [PubMed]
- S. Stallinga and B. Rieger, “Accuracy of the gaussian point spread function model in 2d localization microscopy,” Opt. Express 18, 24461–24476 (2010). [CrossRef] [PubMed]
- A. Van den Bos and C. Ebooks, Parameter Estimation for Scientists and Engineers (Wiley Online Library, 2007).
- C. S. Smith, N. Joseph, B. Rieger, and K. A. Lidke, “Fast, single-molecule localization that achieves theoretically minimum uncertainty,” Nat. Methods 7, 373–U52 (2010). [CrossRef] [PubMed]
- R. J. Ober, S. Ram, and E. S. Ward, “Localization accuracy in single-molecule microscopy,” Biophys. J. 86, 1185–1200 (2004). [CrossRef] [PubMed]
- P. Stoica and T. L. Marzetta, “Parameter estimation problems with singular information matrices,” IEEE Trans. Sig. Process. 49, 87–90 (2001). [CrossRef]
- C. L. L. Hendriks, L. J. van Vliet, B. Rieger, G. M. P. van Kempen, and M. van Ginkel, “Dipimage: a scientific image processing toolbox for MATLAB,” Quantitative Imaging Group, Faculty of Applied Sciences, Delft University of Technology, Delft, The Netherlands (1999).
- NVIDIA, “Compute unified device architecture (CUDA),” http://www.nvidia.com/object/cuda_home.html (2007).
- W. H. Press, S. L. A. Teukolsky, B. N. P. Flannery, and W. M. T. Vetterling, Numerical Recipes: FORTRAN (Cambridge University Press, 1990).
- M. Heilemann, S. van de Linde, M. Schuttpelz, R. Kasper, B. Seefeldt, A. Mukherjee, P. Tinnefeld, and M. Sauer, “Subdiffraction-resolution fluorescence imaging with conventional fluorescent probes,” Angew. Chem. Int. Ed. 47, 6172–6176 (2008). [CrossRef]

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.