SOFI is based on the spatio-temporal evaluation of the optical signal which stems from independently fluctuating emitters. This is done by taking a movie of the sample and subsequently calculating the temporal n
th order cumulant for each pixel. Plotting the value of the n
th order cumulant for almost any given set of time lags yields an image featuring a resolution enhancement of a factor n over the conventional image.
We previously demonstrated SOFI resolution enhancement of a factor of 2 in imaging applications. Here we report on an enhancement to the previously published SOFI algorithm, resulting in a linear scaling of the resolution enhancement with the cumulant order, i.e. the second-order cumulant results in a two-fold resolution enhancement. Furthermore, we show how spatio-temporal cross-cumulants can be advantageously used to find a good approximation of the Point Spread Function (PSF). The use of spatio-temporal cross-cumulants also leads to an up-sampled SOFI image with an effectively reduced pixel size. This in turn means that the systems magnification (nm / pixel) on array detectors (such as CCD-cameras) is not limiting the achievable resolution in SOFI as will be shown in simulations and experiments. In the following we will briefly summarize the underlying theory of SOFI. Then, we will develop the formalism which enables us to increase the apparent resolution of SOFI. Furthermore, we will demonstrate the advantages of spatio-temporal cumulants.
Mathematically, image formation using an optical system can be written as:
is the signal observed at position r
and time t
, and Eq. (1)
describes a convolution of the systems PSF U
) with the time-dependent emitter fluctuation functions fi
), where the index i
refers to the i
emitter. The summation index is extending over all N
emitters. Note that fi
) is not accounting for an emitter's movement but only for the time-dependent fluctuations of its emission, such as fluorescence intermittency. We assume that the emitters are immobile during image (movie) acquisition. Any motion (such as drift) during SOFI acquisition would lead to a blurred image in SOFI and might even result in negative image values.
) can be written as
where the product εi
) factorizes the fluctuations into molecular brightness εi
and a switching function si
) which takes values between zero and one. The Dirac delta function δ(r
) is accounting for the position and the assumed point-like nature of the emitters.
In SOFI, the n
th-order (auto-) cumulant of the detected signal fluctuations δF(r,t) = F(r,t) − 〈F(r,t)〉t is calculated for each pixel and subsequently the amplitude of the cumulant is assigned as the SOFI pixel’s value. The resulting image is called AC- resp. XC-SOFI image, depending whether it is derived from the auto- or cross-cumulant. The latter will be discussed later in this paper. For example, the second-order (auto-) cumulant AC
2 is given by:
The second line in Eq. (3)
is only true if the emitter fluctuations are independent from each other, which is a key ingredient to SOFI.
For the n
th-order (auto-) cumulant ACn, the following expression holds:
Where is a function composed of single-emitter correlation functions of orders up to n. The n
th-order cumulant will give raise to an SOFI image which is featuring the PSF of the original imaging system taken to the n
th power. Note the importance of an immobile sample. Assuming a Gaussian shaped PSF this translates into an apparent resolution enhancement of a factor n
1/2. However, using a simple reweighting scheme for the Optical Transfer function OTF and knowing that the support (supp[Ũ(k)]) of a real OTF is finite, the resolution enhancement can be further improved to a factor n as shown in the following scheme:
represents the OTF (Fourier-transformed PSF) of the imaging system,
is the OTF underlying the SOFI image.
is the Fourier-reweighted OTF which gives raise to the n
-fold resolution enhancement.
is the reweighting function, and α is a damping factor to prevent division by close-to-zero numbers. This reweighting is possible because the support of
is identical (this can be deduced by using the Titchmarsh convolution theorem [10
10. E. C. Titchmarsh, “The Zeros of Certain Integral Functions,” (1926), pp. 283–302.
]). The specific shape of the reweighting function is chosen in that way that the resulting PSF has the same functional form as the PSF
underlying the original intensity image. Note that even though this approach resembles a deconvolution, it is not inferring spatial frequencies which are higher than physically possible. The sole purpose of this reweighting is to transform the SOFI image in a way that the underlying SOFI PSF has the same functional form as the original PSF, but with smaller width. However if the Fourier reweighting scheme from Eq. (5)
is applied to conventional laser scanning confocal microscopy it will not yield any resolution enhancement information (assuming imaging with a perfect PSF; ‘perfect’ refers to a PSF whose OTF carries the maximum achievable spatial frequencies and the PSF is described by a diffraction limited airy disk. In this case the full scope of resolution is already displayed).
In essence, Eq. (5)
states that the resolution of SOFI images (or any images which are based on an n
-fold product of the PSF with itself, as occurring for example in 2-photon excitation laser scanning microscopy or tri-exciton imaging [11
11. S. Hennig, S. van de Linde, M. Heilemann, and M. Sauer, “Quantum dot triexciton imaging with three-dimensional subdiffraction resolution,” Nano Lett. 9(6), 2466–2470 (2009). [CrossRef] [PubMed]
]) could be increased up to a factor of n
-times. Taking into account that novel deconvolution algorithms for confocal and 2P microscopy are still being developed we expect that such advances could be applied beneficially to SOFI images as well due to the mathematical similarity of the problem (For a general review of deconvolution in microsopy we suggest the excellent review of Sibarita [12
12. J. B. Sibarita, “Deconvolution microscopy,” Adv. Biochem. Eng. Biotechnol. 95, 201–243 (2005). [PubMed]
]).Using the Fourier reweighting scheme, the SOFI image can be transformed into a high-resolution SOFI image
are the respective Fourier and inverse Fourier transforms. The performance of the Fourier-reweighting depends strongly on the precise knowledge of the PSF and its respective OTF.
We first evaluated the Fourier-reweighting scheme on simulated data set.
is calculated once using (i) the exact theoretical PSF (given by a Bessel function of the first kind divided by r
|) and (ii) an approximated Gaussian-shaped PSF which we estimated using cross-cumulants (as discussed later).Fig. 1
The effect of Fourier-reweighting on AC-SOFI images generated from a simulated data set. Upper panel:
Conventional AC-SOFI approach for orders 2, 3 and 4 resp. Lower panel:
Fourier-reweighted (using the Gaussian approximation of the PSF) AC-SOFI images featuring an increased resolution as compared to their untreated counterparts in the top panel. The AC-SOFI images shown have been generated using a Gaussian approximation for the PSF as derived from cross-cumulants as will be described later in this paper. Inset:
Relative FWHM of the original image PSF versus the recovered SOFI PSF as a function of the cumulant order. Lines represent the theoretical value, circles represent the values obtained by a 2D-Gauss fit of the emitter indicated by the white arrow in the original image. Blue
: Values obtained from the untreated AC-SOFI images from the upper panel generated using solely Eq. (4)
Values for Fourier-reweighted SOFI images following Eq. (5)
by using the exact PSF (i.e. certain Bessel function of the first kind). Black:
Values obtained by using a Gaussian approximation of the PSF as mentioned above in this caption. Scalebar: 1 µm.
shows the results for various orders of AC-SOFI images. As can be seen, the images show better resolution when Fourier-reweighting is applied. The analysis of the relative FWHM of the PSF shows that the theoretically expected and actually achieved values are in excellent agreement. The deviation from the theoretical values for increasing order is due to the diminished amplitudes of higher spatial frequencies of the OTF O
). In order to recover the full resolution at higher-orders, one would need to reduce the damping factor α, risking an unfavorable increase of the noise level and “ringing” artifacts. The quality of the Fourier-reweighting depends only weakly on the type of PSF (exact vs. Gaussian) which is used for calculating the weight function
. Even though the Gaussian PSF approximation (as shown in Fig. 1
) yields slightly lower resolution values, the resulting images exhibit slightly less ringing as compared to the exact PSF (data not shown). It should be noted that the shown simulations do not contain any kind of noise or background.
When applying the same strategy to noise-afflicted data sets, it turns out that the Fourier-reweighing is fairly robust (see Fig. 2
Fig. 2 The effect of noise on Fourier reweighted SOFI images. The simulation was done with a signal to background ratio of 3 and a signal intensity of 18 counts per time bin (‘Signal’ denotes the maximum intensity of the PSF). Using these relatively weak imaging parameters it was still possible to generate SOFI images and apply Fourier reweighting to them. In the upper panel we show how the quality of the Fourier reweighted second-order AC SOFI image improves when more frames are acquired. The same holds true for the third-order SOFI image (lower panel). However, we found that the fourth-order SOFI image could not satisfactorily be generated using these imaging parameters (due to high noise content already appearing in the fourth order AC SOFI image).
). Obviously the Fourier-reweighting scheme breaks down first for the higher-order SOFI images if noise becomes too strong ord the signal to background ratio is too low. The second-order SOFI images however can be recovered conveniently with full resolution even at relatively low signal to noise levels and relatively short simulation times (1000 frames).
The Fourier reweighting scheme described above therefore allows one to obtain SOFI superresolution that scales linearly with the cumulant order n (rather than square-root scaling).
Cross-Cumulants: PSF estimation and higher image sampling-frequency
When an image sequence (movie) is acquired by a CCD-camera (or any other array detector), circumjacent pixels carry information which can be exploited for SOFI using a temporal cross-cumulant approach. For the sake of simplicity, let us first consider a second-order SOFI image. By temporally cross-correlating two pixels and assuming a Gaussian-shaped PSF the following expression can be found [analogous to Eq. (3)
is the second-order cross-cumulant. We can gain two insights from Eq. (7)
1. The location of the resulting “cross-correlation pixel” lies in the geometric center of and :
2. This cross-correlation pixel is weighted by a PSF-shaped weight factor, which depends on the distance of and : , and therefore in the following is referred to as distance factor.
Applying the cross-correlation approach to directly neighboring pixels, virtual “cross-correlation” pixels which interpolate between neighboring pixels on the CCD can be generated (see (1.)). Thus, a fourfold higher density of pixels (2 × 2) compared to the pure auto-correlation approach can be achieved by cross-correlating vertically, horizontally and diagonally neighboring pixels of the movie sequence (see Fig. 3
Fig. 3 Schematic of various possibilities cross-cumulants can be used for. The squares represent pixels of the CCD-camera. (A) Second-order cross-correlation. By cross-correlating (indicated by arrows) two directly neighboring pixels (light blue squares) virtual, “cross-correlation” (red squares) pixels can be obtained. These pixels lie in between the physical camera pixels. (B). The same holds true for higher-order cross-cumulants. As shown on the example of the fourth order cross-cumulant. Using different combinations of the light blue pixels 16-times more virtual pixels can be generated. (C) Second-order correlation. The value for the red target pixel can be obtained by correlating various different pairs of pixels (as indicated by black and gray arrows). This way auto-correlations can be omitted completely.
). These cross-correlation pixels carry true information (in contrast to pixels generated by a simple interpolation of the auto-correlation (AC-) SOFI image). Note that not only directly neighboring pixels can be used to generate interleaving pixels, but also next-to-neighbor pixels and so on, as long as the size of the PSF is not exceeded. Note further, that it is even possible to generate an up-sampled (XC-) SOFI image while completely omitting auto-correlations (e.g. by cross-correlating pixels which are one pixel apart from each other. See Fig. 3
). The latter can be useful if one wants to generate SOFI images for the zero-time lag, which would be affected by shot-noise and camera read-out noise if one uses an autocorrelation approach.
One feature of the pixels which are generated by cross-correlation is that they exhibit an intensity which is weighted by the above mentioned distance factor. This leads to systematic intensity differences in the XC-SOFI image (see Fig. 4
Fig. 4 Comparison between distance factor corrected and uncorrected XC-SOFI images. (A) second-order XC-SOFI image not corrected for distance factor contributions. (B) Distance factor corrected second-order XC-SOFI image using relative variance minimization. Scalebar: 0.5 um.
). In order to compensate these intensity differences one has to counter-balance the distance factor for the cross-correlation pixels by multiplying these pixels with the inverse of the distance factor. In the following we propose a way to estimate the distance factor using the already acquired data set.In an XC-SOFI image we know that certain pixels carry the same distance factor. For example, every second pixel along x
will be a cross-correlation pixel which has been generated by using e.g. the left and right neighboring pixels, whose distances are always the same throughout the image. The same applies to every second pixel along y
, along the diagonal and to the autocorrelation pixels (where the distance is zero and thus the distance factor is one). Thus, we know that there are only four different values for the distance factor within the second-order XC-SOFI image and these values have to be applied to particular pixels only. By varying the intensity of these particular pixels with the same inverse distance factor we look for the XC-SOFI image which appears most “smooth”. This is done by minimizing the relative variance of the resulting image by varying a parameterized description of the Gaussian PSF [i.e. the distance factor, see (2.)] (see Fig. 4
). Since this fit involves only three fit parameters (width ωx
and orientation ϕ of the 2D Gauss) which are determined from a pool of Mx
pixels (where M
is the number of pixels along each axis of the up-sampled image) it can be considered very restrictive and accurate.
The described approach holds true also for higher-order XC-SOFI images (Fig. 3
). The general formula is then given by:
We demonstrate this on a simulation up to the fourth-order cross-cumulant. The pixels of a simulated data set (movie) were pooled in bigger pixels (1×1, 2×2, 3×3, and 4×4 pixels). The n
th-order XC-SOFI image was created from the n × n-times binned movie using the cross-cumulant approach as described above (i.e. the second-order XC-SOFI image was derived from the 2 × 2 binned data set, which yields 2 × 2 times more pixels). The resulting XC-SOFI image was than compared to the AC-SOFI image calculated from the original (not-binned) data set (1 × 1). Both SOFI images have therefore the same number of pixels, feature the same magnification (nm / pixel) and are almost identical. Differences arise from the Gaussian approximation of the PSF.
As can be seen in Fig. 5
Fig. 5 Estimation of the pixel recovery ability of XC-SOFI images. Upper panel: second order SOFI images. Lower panel: Fourth-order SOFI images. (A) resp. E): AC-SOFI image as obtained by a 2 × 2 resp. 4 × 4 binned data set. (B) resp. F): AC-SOFI image from the not-binned data set. (C) resp. G): XC-SOFI images obtained from the 2 × 2 rep. 4 × 4 binned data set using the cross-cumulant approach. (D) resp. H): Difference between B) resp. F) and C) resp. D). The simulation consists of 100,000 frames. Scalebar: 1 um.
, the differences between the normalized AC-SOFI images and the XC-images are not exceeding 0.4% for the second-order. Similar results were obtained for the third- and fourth-order SOFI images, though the mismatch was slightly larger (1.4% resp. 9%). The recovered FWHM of the approximated PSFs deviated less than 2% from the original PSFs’ FWHM for all cumulant-orders.
The decreased accuracy of the reconstruction can be assigned to the decreased oversampling of the PSF, but mainly to the order of the cumulant to be reconstructed. The latter refers to the fact that the statistics for higher-order cumulants have to stem from an increasingly longer data set in order to fully recover a SOFI image. When binning pixels, information gets pooled and therefore the data set has to provide longer time traces in order to recover this information. This can be seen from Fig. 5
where the fourth-order cumulant differences are biggest in regions of high emitter density and therefore the fluctuations of many emitters are pooled in a single pixel. This leads to an incomplete resolved fourth-order image (due to limited measurement time). Therefore the assumption for the cross-cumulant approach that the underlying SOFI image is fully resolved is violated and ill-compensated in these regions.
In the following simulation we explicitly show that the cross-correlation pixels indeed do carry information on the resolution. For this reason a different data set was analyzed, where the resolution enhancement could not be realized since the magnification / spatial oversampling was intentionally chosen to be too low (the effective pixel size of 160 nm was exceeding the obtainable resolution ~220 nm (second-order AC-SOFI)). Using the cross-correlation approach the effective pixel size can be reduced (80 nm/pixel) so it becomes sufficiently small to resolve all emitters. Figure 6
Fig. 6 Comparison of the AC-SOFI image and XC-SOFI image. Simulation of 7 emitters (emitting at: 800 nm) placed 260 nm apart and imaged (Numerical aperture NA 1.2) on a grid which has a magnification of (160 nm/pixel). Blue line: second-order AC-SOFI image. Red line: second-order XC-SOFI image. Gray line: SOFI image as would be it obtained for a infinite spatial sampling (i.e. pixel size → 0 nm). As can be seen the XC-SOFI image carries high resolution information, which would not have been revealed in the AC-SOFI image due to the too coarse grid of the imaging system. Vertical gray lines indicate the emitter positions. Stars represent values which have been generated by cross-correlation.
shows the result for the XC-approach versus the AC-approach. The emitter positions are marked to demonstrate that indeed the recovered pixels are carrying the higher resolution information.
To show that the above mentioned approaches (Fourier-reweighting and cross-cumulant calculation) work well for a real imaging application, we performed an experiment on fixed 3T3 fibroblasts whose tubulin network was labeled with infrared emitting quantum dots (QD800, life technologies, USA). The camera magnification was chosen to be 167 nm which on the one hand allows large field-of-view imaging, and on the other hand requires the cross-cumulant approach to realize the full resolution gain of a factor two by applying the Fourier-reweighting approach.
From the data set we estimated the shape of the PSF using the above described cross-cumulant approach, showing a slight ellipticity of 1.06 and a mean FWHM of 462 nm. This PSF was used in the subsequent Fourier-reweighting of the XC-SOFI image. The result can be seen in Fig. 7
Fig. 7 Tubulin network of a 3T3 fibroblast immuno-labeled with QD800 quantum dots. Top panel: A) original image taken from the average of 2000 frames of a wide-field microscope setup. White lines indicate locations where cross-section were taken for all images in the upper panel. B) XC-SOFI image featuring twice more pixels than the original image and a resolution enhancement of a factor of 2. C) Fourier-reweighted XC-SOFI image generated by using the PSF as obtained by the cross-cumulant approach. Also note that also this image has four times more pixels than the original image. Scalebar: 10 um. Lower panel: Cross-sections as taken from the upper panel. Black: interpolated original image, Blue: XC-SOFI image. Red: Fourier reweighted XC-SOFI image. a) PSF shrinks as the order increases. Lines indicating a Gauss fit and circles describe the data. The FWHM reflects the increased resolution enhancement: 1, 1.39 (~2) and 1.98 (~2) respectively b) A structure being resolved only in the Fourier-reweighted XC-SOFI image. The distance between the peaks is 220 nm (as indicated by the dashed black lines), which is approximatley 2x smaller than the Rayleigh limit for 800 nm emitting QDs and a NA 1.2 objective. c) “zoomed-out” cross-section giving a general impression of the improvement afforded by Fourier-reweighting.
. Cross-sections (taken as indicated by the white lines in the original image) are shown on the lower panel. It can be seen that the resolution of the Fourier-reweighted XC-SOFI image is increased as compared to the original image and also as compared to the conventional (unweighted) XC-SOFI image. Using infrared emission we could increase the resolution from ~400 nm as defined by the Rayleigh limit down to at least 220 nm.
We demonstrated a resolution enhancement of SOFI using a Fourier-reweighting scheme with a linear dependence of resolution vs. cumulant order. We also demonstrated that the use of spatio-temporal cross-cumulants holds many advantages over the auto- cumulant approach. The most obvious is the accurate description of the underlying PSF of the imaging system. The use of cross-cumulants enables the straight forward calculation of zero-time lag XC-SOFI images, eliminating shot-noise and camera read-out noise. Since most fluorescence intermittency phenomena occur on short time-scales and decay rapidly, the use of zero-time lag XC-SOFI images might prove beneficial in terms of higher signal to noise ratio.
The increased pixel number and information content afforded by XC-SOFI eliminates the need to change magnification (as long as slight oversampling of the PSF is guaranteed in the original image). No additional magnification in turns means that the fluorescent signal of a single emitter is not “diluted” on too many pixels, yielding a better signal-to-noise ratio. The magnification of commercial high NA microscopes is often designed to be in the 150-170 nm / pixel range. This magnification is sufficient to generate SOFI images with the resolution obtainable by any cumulant order while preserving the same field of view. Furthermore, relatively short acquisition times in SOFI (as compared to localization-based superresolution techniques) relax platform dependent issues such as sample drift during the measurement.