## FPGA implementation of a 32x32 autocorrelator array for analysis of fast image series |

Optics Express, Vol. 20, Issue 16, pp. 17767-17782 (2012)

http://dx.doi.org/10.1364/OE.20.017767

Acrobat PDF (4391 KB)

### Abstract

With the evolving technology in CMOS integration, new classes of 2D-imaging detectors have recently become available. In particular, single photon avalanche diode (SPAD) arrays allow detection of single photons at high acquisition rates (≥ 100kfps), which is about two orders of magnitude higher than with currently available cameras. Here we demonstrate the use of a SPAD array for imaging fluorescence correlation spectroscopy (imFCS), a tool to create 2D maps of the dynamics of fluorescent molecules inside living cells. Time-dependent fluorescence fluctuations, due to fluorophores entering and leaving the observed pixels, are evaluated by means of autocorrelation analysis. The multi-*τ* correlation algorithm is an appropriate choice, as it does not rely on the full data set to be held in memory. Thus, this algorithm can be efficiently implemented in custom logic. We describe a new implementation for massively parallel multi-*τ* correlation hardware. Our current implementation can calculate 1024 correlation functions at a resolution of 10*μ*s in real-time and therefore correlate real-time image streams from high speed single photon cameras with thousands of pixels.

© 2012 OSA

## 1. Introduction

1. D. Magde, E. L. Elson, and W. W. Webb, “Fluorescence correlation spectroscopy i: conceptual basis and theory,” Biopolymers **13**, 1–27 (1974). [CrossRef]

2. D. Magde, E. L. Elson, and W. W. Webb, “Fluorescence correlation spectroscopy. ii. an experimental realization,” Biopolymers **13**, 29–61 (1974). [CrossRef] [PubMed]

3. O. Krichevsky and G. Bonnet, “Fluorescence correlation spectroscopy: the technique and its applications,” Rep. Prog. Phys. **65**, 251–297 (2002). [CrossRef]

*I*(

*t*) inside a small observation volume (usually around 10

^{−15}l = 1

*μ*m

^{3}) is monitored. The fluctuations originate from particles entering and leaving the focus, or transitions between states having different quantum yields. Faster dynamics of the fluorescing particles also lead to faster fluctuations, which can be quantified by means of a temporal first-order autocorrelation function (ACF): The ACF usually contains features that are spread over several orders of magnitude in time (nanoseconds to seconds), with

*T*̃ being the runtime of the entire measurement.

*I*(

*t*) from one focal volume. Then the data is fed into a “correlator” (hardware or software component), which estimates the ACF over a certain dynamic range.

7. M. Wahl, I. Gregor, M. Patting, and J. Enderlein, “Fast calculation of fluorescence correlation data with asynchronous time-correlated single-photon counting,” Opt. Express **11**, 3583–3591 (2003). [CrossRef] [PubMed]

9. E. Schaub, “F2cor: fast 2-stage correlation algorithm for FCS and DLS,” Opt. Express **20**, 2184–2195 (2012). [CrossRef] [PubMed]

10. D. Magatti and F. Ferri, “Fast multi-tau real-time software correlator for dynamic light scattering,” Appl. Opt. **40**, 4011–4021 (2001). [CrossRef]

11. D. Magatti and F. Ferri, “25 ns software correlator for photon and fluorescence correlation spectroscopy,” Rev. Sci. Instrum. **74**, 1135–1144 (2003). [CrossRef]

*τ*algorithm are available also [12

12. M. Culbertson and D. Burden, “A distributed algorithm for multi-tau autocorrelation,” Rev. Sci. Instrum. **78**, 044102 (2007). [CrossRef] [PubMed]

13. B. Tieman, S. Narayanan, A. Sandy, and M. Sikorski, “Mpicorrelator: a parallel code for performing time correlations,” Nucl. Inst. Meth. A **649**, 240–242 (2011). [CrossRef]

19. G. Mocsar, B. Kreith, J. Buchholz, J. W. Krieger, J. Langowski, and G. Vamosi, “Note: multiplexed multiple-tau auto- and cross-correlators on a single field programmable gate array,” Rev. Sci. Instrum. **83**, 046101 (2012). [CrossRef] [PubMed]

20. M. Burkhardt and P. Schwille, “Electron multiplying ccd based detection for spatially resolved fluorescence correlation spectroscopy,” Opt. Express **14**, 5013–5020 (2006). [CrossRef] [PubMed]

21. R. A. Colyer, G. Scalia, I. Rech, A. Gulinatti, M. Ghioni, S. Cova, S. Weiss, and X. Michalet, “High-throughput FCS using an LCOS spatial light modulator and an 8 × 1 SPAD array,” Biomed. Opt. Express **1**, 1408–1431 (2010). [CrossRef]

23. G. Heuvelman, F. Erdel, M. Wachsmuth, and K. Rippe, “Analysis of protein mobilities and interactions in living cells by multifocal fluorescence fluctuation microscopy,” Eur. Biophys. J. **38**, 813–828 (2009). [CrossRef] [PubMed]

24. F. Bestvater, Z. Seghiri, M. S. Kang, N. Gröner, J. Y. Lee, I. Kang-Bin, and M. Wachsmuth, “EMCCD-based spectrally resolved fluorescence correlation spectroscopy,” Opt. Express **18**, 23818–23828 (2010). [CrossRef] [PubMed]

25. D. J. Needleman, Y. Xu, and T. J. Mitchison, “Pin-hole array correlation imaging: highly parallel fluorescence correlation spectroscopy,” Biophys. J. **96**, 5050–5059 (2009). [CrossRef] [PubMed]

26. B. Kannan, L. Guo, T. Sudhaharan, S. Ahmed, I. Maruyama, and T. Wohland, “Spatially resolved total internal reflection fluorescence correlation microscopy using an electron multiplying charge-coupled device camera,” Anal. Chem. **79**, 4463–4470 (2007). [CrossRef] [PubMed]

27. T. Wohland, X. Shi, J. Sankaran, and E. H. K. Stelzer, “Single plane illumination fluorescence correlation spectroscopy (SPIM-FCS) probes inhomogeneous three-dimensional environments,” Opt. Express **10**, 10627–10641 (2010). [CrossRef]

28. J. Capoulade, M. Wachsmuth, L. Hufnagel, and M. Knop, “Quantitative fluorescence imaging of protein diffusion and interaction in living cells,” Nat. Biotechnol. **29**, 835–839 (2011). [CrossRef] [PubMed]

28. J. Capoulade, M. Wachsmuth, L. Hufnagel, and M. Knop, “Quantitative fluorescence imaging of protein diffusion and interaction in living cells,” Nat. Biotechnol. **29**, 835–839 (2011). [CrossRef] [PubMed]

19. G. Mocsar, B. Kreith, J. Buchholz, J. W. Krieger, J. Langowski, and G. Vamosi, “Note: multiplexed multiple-tau auto- and cross-correlators on a single field programmable gate array,” Rev. Sci. Instrum. **83**, 046101 (2012). [CrossRef] [PubMed]

*τ*

_{min}...

*τ*

_{max}= 10

*μ*s...1s. The design could easily be adapted to different kinds of image sensors such as single photon avalanche diode (SPAD) arrays, customized scientific complementary metal oxide semiconductor (sCMOS) or electron multiplying charge-coupled device (EMCCD) cameras.

*Radhard2*as image sensor, which can be read at frame rates of 100kfps and above (for a detailed description of this sensor, please refer to section 2 and Ref. [29]). Smaller SPAD arrays have already been used for parallelized FCS on multi-focus confocal microscopes [30

30. M. Gösch, A. Serov, T. Anhut, T. Lasser, A. Rochas, P. Besse, R. Popovic, H. Blom, and R. Rigler, “Parallel single molecule detection with a fully integrated single-photon 2 × 2 CMOS detector array,” J. Biomed. Opt. **9**, 913 (2004). [CrossRef] [PubMed]

31. R. Colyer, G. Scalia, T. Kim, I. Rech, D. Resnati, S. Marangoni, M. Ghioni, S. Cova, S. Weiss, and X. Michalet, “High-throughput multispot single-molecule spectroscopy,” in “Proceedings-Society of Photo-Optical Instrumentation Engineers,”, vol. 7571 (NIH Public Access, 2010), vol. 7571, p. 75710G.

32. C. Veerappan, J. A. Richardson, R. J. Walker, D.-U. Li, M. W. Fishburn, Y. Maruyama, D. Stoppa, F. Borghetti, M. Gersbach, R. K. Henderson, and E. Charbon, “A 160x128 single-photon image sensor with on-pixel 55ps 10b time-to-digital converter.” in “ISSCC, IEEE International Solid-State Circuits Conference,” (IEEE, 2011), pp. 312–314.

## 2. Radhard2 SPAD array detector

*Radhard2*single photon avalanche diode array with 32 × 32 pixels as detector for our experiments [29]. The pixels are 30 × 30

*μ*m

^{2}of which only about 1.4% is active area (circular SPAD with 4

*μ*m diameter). The photon detection probability is ≈ 30% at a wavelength of 500nm. At room temperature the dark count rate is approximately 140Hz. After-pulsing probability is negligible at the used integration time of Δ

*t*

_{frame}= 10

*μ*s [33

33. C. Niclass, M. Sergio, and E. Charbon, “A single photon avalanche diode array fabricated in 0.35-μm CMOS and based on an event-driven readout for TCSPC experiments,” in “Proc. SPIE ,” **6372**, 63720S (2006). [CrossRef]

*μ*s. The design of the detector also allows the readout of subregions at higher speeds, e.g. a single line every 2.66

*μ*s/32 = 83ns. Frames read from the sensor contain 1bit of information per pixel (no photon or at least one photon in the last Δ

*t*

_{frame}).

## 3. Multi-*τ* hardware correlators

*T*and the integration time

*τ*

_{min}for one sample. When discretizing Eq. (1) with this intensity sequence, care has to be taken not to bias the normalization

34. K. Schätzel, “Noise on photon correlation data: I. autocorrelation functions,” Quantum Opt. **2**, 287–305 (1990). [CrossRef]

*τ*∈ ℕ (in units of

_{k}*τ*

_{min}, so

*τ*=

*τ*·

_{k}*τ*

_{min}). When the full sequence {

*I*}

_{n}

_{n}_{=0...}

_{T}_{−1}is available after the measurement, Eq. (3) may be evaluated directly for an arbitrary (also logarithmically spaced) set of lags

*τ*in software. This gives an unbiased estimation of the ACF (“direct correlation”). To implement our hardware correlator, we use the multi-

_{k}*τ*scheme introduced in Reference [35], which is also illustrated and compared to a linear implementation in Fig. 1. For the special case of linearly spaced lags

*τ*=

_{k}*k*, a simple hardware implementation exists, which is shown in Fig. 1(a). Each of the Δ

*τ*-blocks represents a delay of Δ

*τ*=

*τ*

_{min}, which can be implemented using a flip-flop [5] clocked with a frequency of 1/

*τ*

_{min}. Thus, in the final design the series of delay elements are structured as a shift register. After

*n*time-steps the input

*I*

_{n}_{−}

*has propagated to the*

_{k}*k*-th delay element (signal

*I*. So the

_{n}*k*-th channel accumulates

*G*(see Eq. (3)). For the normalization additional components that accumulate the input signal

_{k}*I*and

_{n}*I*

_{n}_{−}

*at different lags*

_{k}*k*(usually called monitor channels,

*M*

_{0}and

*M*in Eq. (3)) are implemented.

_{k}*τ*scheme uses a set of

*S*of these linear correlator blocks (Fig. 1(b,c)), with

*s*= 0,...,

*S*− 1. The input samples

*I*(

_{s,n}*n*is the same index as in Eq. (2)) are summed over increasingly long periods Δ

*n*=

*m*, with

^{s}*m*= 2 being the factor between the delay times of two subsequent blocks: with

*I*

_{0}

*=*

_{,n}*I*.

_{n}*P*linearly spaced lags where

*p*= 0...

*P*− 1.

*ĝ*

_{sym,multi-}

*(*

_{τ}*τ*). The advantage of this multi-

_{s,p}*τ*scheme is its simple implementation in hardware and a large dynamic time range with a reasonable number of channels. Its disadvantage is a systematic error introduced by averaging: As shown in Ref. [36

36. Z. Kojro, A. Riede, M. Schubert, and W. Grill, “Systematic and statistical errors in correlation estimators obtained from various digital correlators,” Rev. Sci. Instrum. **70**, 4487–4496 (1999). [CrossRef]

*ĝ*

_{sym,multi-}

*(*

_{τ}*τ*) equals the ideal correlation function

_{s,p}*g*(

*τ*·

_{s,p}*τ*

_{min}) (see Eq. (3)) convolved with a triangular kernel with width

*m*: where * denotes the convolution product and Λ(

^{s}*τ*, Δ

*τ*) = Δ

*τ*−

*|τ|*for |

*τ*| < Δ

*τ*and Λ(

*τ*, Δ

*τ*) = 0 for |

*τ*| ≥ Δ

*τ*, is the triangular shaped kernel.

## 4. Hardware design

### 4.1. Single-pixel correlator

*τ*correlator. This is possible by serial processing of the lag time channels, since the hardware in each channel is identical.

accumulator and delayed value of a channel from memory**L**oadfor memory access to complete**W**aitdelayed with global signal**M**ultiplymultiplication result to channel’s accumulator**A**ddcounter and new delayed value to memory**S**tore

*m*= 2) in the multi-

*τ*scheme, making the input data rate of block

*s*+1 half that of block

*s*. Hence we need to execute each block only half as often as its predecessor. Thus, a complete multi-

*τ*correlator can be executed in only twice the run-time Δ

*t*

_{lin}of a single linear correlator block: A key requirement is that Δ

*t*

_{lin}is at most half the integration time

*τ*

_{min}of the input signal

*I*.

_{n}*s*is only executed after its predecessor

*s*− 1 has been executed twice. A counter

*c*= 0, 1,... is incremented with every execution of any linear correlator block. The scheduler uses the following relations to determine which linear correlator block

*s*has to be executed at a given counter value

*c*(details see appendix): Figure 2(a) shows the solution of this relation for

*c*values from 0 to 31. In the binary representation of

*c*for a linear correlator block

*s*patterns are evident that can be used to implement the scheduler efficiently. As shown in Table 2 (for

*S*= 8 linear correlators), correlator block

*s*= 0 is executed whenever the last bit of

*c*is 0

_{b}, correlator

*s*= 1 is executed when the last two bits are 11

_{b}and so forth. This scheme uses only simple comparison operations.

*s*− 1 and

*s*, adder circuitry is inserted to sum up two subsequent input signal values

*I*

_{s}_{−1,}

_{n}_{−1}and

*I*

_{s}_{−1,}

*. This is done for both the delayed/local as well as the undelayed/global signal, while they are processed in the pipeline.*

_{n}### 4.2. Multi-pixel correlator

*c*and an accumulator for the local and the global input signals have to be saved. The latter are used for normalization and cross-correlation.

*n*= 32 pixels) of our SPAD array. To handle the full

_{y}*n*×

_{x}*n*array of pixels, we instantiate

_{y}*n*= 32 CorrPEs in parallel. An overview of this scheme is shown in Fig. 4. A “data acquisition” circuit communicates with the SPAD array and provides the image data for the correlators. As the image data is streamed out row by row, and each of the

_{x}*n*CorrPEs is only processing data from one specific context (i.e. a specific row), the remaining pixels have to be buffered row wise in 32 FIFOs (first-in first-out memory buffer) localized in external RAM.

_{x}### 4.3. ACF normalization

*τ*correlator with a monitor channel per lag (blue) and our estimation (magenta) can be seen in Fig. 5, where the data in (a) and (b) were obtained by correlating the input signal

*I*(

*t*) = 1 + sin(2

*πt*/(1.51 · 10

^{−4})) for which the exact ACF is known to be

*g*

^{(theoretical)}(

*τ*) = 1 + cos(2

*πτ*/(1.51·10

^{−4})) (time

*t*and lags

*τ*are unit free). The data in Fig. 5(c) was created by simulating a

*T*

_{sim}= 1s long FCS experiment with one diffusing species [38

38. The diffusion coefficient was *D* = 20*μ*m^{2}/s (corresponding to an intermediately sized protein in water), the simulation timestep of the random walk, as well as the minimum lag time were Δ*t*_{sim} = *τ*_{min} = 1*μ*s. There were around 1.2 particles in the effective measurement volume *V*_{eff} ≈ 0.4*μ*m^{3} on average.

39. T. Wocjan, J. Krieger, O. Krichevsky, and J. Langowski, “Dynamics of a fluorophore attached to superhelical DNA: FCS experiments simulated by brownian dynamics,” Phys. Chem. Chem. Phys. **11**, 10671–10681 (2009). [CrossRef]

*τ*correlators have an increased absolute error for longer lags, which is due to the averaging described in Eq. (6). This can be seen especially in the case of the sine wave signal. The multi-

*τ*estimates can still be used for FCS experiments, as here the ACFs usually decay to 1 (white noise) for large lag times, and thus the systematic error drops to zero again (for a detailed discussion of this, see e.g. Ref. [36

36. Z. Kojro, A. Riede, M. Schubert, and W. Grill, “Systematic and statistical errors in correlation estimators obtained from various digital correlators,” Rev. Sci. Instrum. **70**, 4487–4496 (1999). [CrossRef]

*τ*≳

*T*

_{sim}/10, multi-

*τ*correlators show additional systematic deviations from the theoretical curve and from the direct estimation, because the channels are not averaged over sufficiently many samples to yield reliable results. Here the multi-

*τ*implementation with multiple monitor channels performs better due to the better estimation of the normalization factor

*M*

_{τs,p}.

### 4.4. Crosscorrelation (CCF)

*I*

^{(}

^{x}^{)}(

*t*) and

*I*

^{(}

^{y}^{)}(

*t*) at the local

*J*

^{(l)}and global

*J*

^{(g)}inputs of the CorrPE (see Fig. 1 where both signals are tied to

*I*(

*t*) for ACF calculation). This changes the multi-

*τ*estimator Eq. (9) to: Here the normalization uses the monitors

*M*

_{0}=

*M*

^{(}

^{g}^{)}=

*M*

^{(}

^{l}^{)}.

## 5. Performance & implementation details

*P*= 8 channels within each of the

*S*= 14 linear correlator blocks.

*τ*correlator can be executed within twice the time needed for the first linear correlator block, so we devote half of the execution time to the first linear correlator and the rest to the remaining blocks. Therefore a new input sample can be accepted only once every 2Δ

*t*

_{lin}. Here Δ

*t*

_{lin}= 2

*P*+ 3cycles is the time needed to process a new input sample

*I*in the first linear correlator block. The 3 additional cycles are used for data hand over to the next block.

_{n}*t*

_{lin}for a single input signal. This is the minimum timespan between two subsequent samples, if no pixel multiplexing is used. Hence, our current FPGA platform can calculate 32 different ACFs or CCFs with a minimum lag time of 264ns, which is comparable to the 100ns designs presented in Ref. [14] and more recently in Ref. [19

19. G. Mocsar, B. Kreith, J. Buchholz, J. W. Krieger, J. Langowski, and G. Vamosi, “Note: multiplexed multiple-tau auto- and cross-correlators on a single field programmable gate array,” Rev. Sci. Instrum. **83**, 046101 (2012). [CrossRef] [PubMed]

*G*

_{τs,p}. The monitor channels are 32bits each. Data handover between consecutive blocks (accumulated local and global signals) is done via 16bit-wide memory areas, which is sufficient for

*S*≤ 16. Since in the later linear correlators the accumulated input signals

*I*are multiplied, the sums

_{s,n}*G*

_{τs,p}and also the

*I*can get relatively large and may not fit in the 32bit memory locations available. However, due to constant streaming of intermediate results to the host computer, counter overflows can be detected and corrected.

_{s,n}## 6. Benchmark experiments

*τ*correlator using the same data set. Both yielded exactly the same results. The comparison was also done using real experimental data from the SPAD array. Again, both results were identical.

41. K. Greger, J. Swoger, and E. H. K. Stelzer, “Basic building units and properties of a fluorescence single plane illumination microscope,” Rev. Sci. Instrum. **78**, 023705 (2007). [CrossRef] [PubMed]

*μ*m width (1/

*e*

^{2}half-width) by a Nikon Plan Fluor 10x/NA0.3 microscope objective and cylindrical lens of focal length

*f*= 100mm (CKX18-C, Newport Spectra-Physics GmbH, Darmstadt). A Nikon CFI Apo-W NIR 60x/NA1.0 water dipping objective and an achromatic lens with

*f*= 100mm (AC254-100-A-ML, Thorlabs GmbH, Dachau, Germany) are used to image the acquired fluorescence onto the SPAD array

*Radhard2*, with a 30× magnification. A 500nm long-pass filter (Edge Basic 488LP, Semrock, Rochester, USA) suppresses scattered light by about a factor of 10

^{−6}at 491nm. Figure 7 shows the results of an experiment with fluorescent microspheres of diameter 40nm (Invitrogen FluoSpheres YG carboxyl-modified, Life Technologies GmbH, Darmstadt). According to the manufacturer, each bead has a brightness which is equivalent to 350 fluorescein molecules. They were dissolved in water (1 : 1000 dilution from stock concentration

*c*

_{stock}≈ 2.4

*μ*M) and were mounted in a small sample bag made from 25

*μ*m thin transparent foil matching the refractive index of water (LUMOX FOLIE 25 M, SARSTEDT AG & Co, Nümbrecht). The sample was illuminated with a laser power of about 2.33mW, as measured behind the projection objective. This power is distributed over the light sheet of height 4mm and center 1/

*e*

^{2}-width of 2.5

*μ*m, which amounts to an intensity in the focus of about 23.3W/cm

^{2}.

*N*is the average particle number inside the focus and

*τ*is the diffusion decay time. The parameter

_{D}*γ*describes the aspect ratio of the Gaussian focus. The (unweighted) fits were performed using a Levenberg-Marquardt least squares fitting routine (lmfit [42

42. Joachim Wuttke: lmfit - a C/C++ routine for Levenberg-Marquardt minimization with wrapper for least-squares curve fitting, based on work by B. S. Garbow, K. E. Hillstrom, J. J. Moré, and S. Moshier. Version 3.2, retrieved on 2011-08-31 from http://www.messen-und-deuten.de/lmfit/.

43. QuickFit 3.0 can be downloaded free of charge from http://www.dkfz.de/Macromol/quickfit/. In addition to the fitting capabilities, it also contains software implementations of the correlators described in here.

*e*

^{2}half-width of

*w*= (0.5 ± 0.2)

_{xy}*μ*m and longitudinal half-width of

*w*= (0.8 ± 0.1)

_{z}*μ*m and therefore an aspect ratio of about

*γ*= (1.60 ± 0.72), which was fixed in the fits. The lateral and longitudinal widths fit the theoretically expected values for the setup, if the depth-selection due to the pinhole is taken into account. Also we assume a Gaussian detection probability distribution, as usually done in SPIM-FCS [27

27. T. Wohland, X. Shi, J. Sankaran, and E. H. K. Stelzer, “Single plane illumination fluorescence correlation spectroscopy (SPIM-FCS) probes inhomogeneous three-dimensional environments,” Opt. Express **10**, 10627–10641 (2010). [CrossRef]

28. J. Capoulade, M. Wachsmuth, L. Hufnagel, and M. Knop, “Quantitative fluorescence imaging of protein diffusion and interaction in living cells,” Nat. Biotechnol. **29**, 835–839 (2011). [CrossRef] [PubMed]

*τ*

_{D}= 10.2ms. From this we can calculate a diffusion coefficient of

*μ*m

^{2}/s measured with confocal FCS for the same sample. Due to the Gaussian nature of the light sheet, its width increases with the distance to the focal line. This leads to an increased number of detected particles

*N*at the edges of the field of view (see Fig. 7(f)). The average count rate during the measurement was 〈

*I*〉 = 2400Hz (see also Fig. 7(b) and (c)), which is well above the dark count rate (DCR) of the

*Radhard2*sensor of about 140Hz [29]. The overall background countrate (including the DCR) was

*I*

_{background}≈ 300Hz during the measurement, which leads to a correction of the measured particle number (as shown in Fig. 7) of

*N*

_{real}=

*N*

_{measured}/(1 +

*I*

_{background}/ 〈

*I*〉)

^{2}= 0.79·

*N*

_{measured}[44

44. S. T. Hess and W. W. Webb, “Focal volume optics and experimental artifacts in confocal fluorescence correlation spectroscopy,” Biophys. J. **83**, 2300–2317 (2002). [CrossRef] [PubMed]

*N*

_{real}) = 13.0 we get a molecular brightness of about 184Hz/particle, which is about a factor of 530 less than in the measurement on our confocal setup (

*NA*= 1.2), where the average excitation intensity in the focus was about a factor of 250 higher than in our SPIM setup. The difference can be explained by the higher NA, quantum efficiency and larger pinhole size of the confocal setup.

## 7. Conclusion

*τ*correlator design that can calculate 1024 correlation functions in real time at a minimum lag time of 10

*μ*s. To our knowledge this is the largest number of real time multi-

*τ*correlators implemented so far in a single device. The minimum lag time of 10

*μ*s in our design is longer than that of currently available hardware correlators (e.g. from ALV GmbH, Langen, Germany or correlator.com, Bridgewater, USA and Reference [11

11. D. Magatti and F. Ferri, “25 ns software correlator for photon and fluorescence correlation spectroscopy,” Rev. Sci. Instrum. **74**, 1135–1144 (2003). [CrossRef]

**83**, 046101 (2012). [CrossRef] [PubMed]

*τ*correlation scheme and for the multiplexed processing of all four input signals. Due to the utilization of a high level description of the hardware using the well established LabView software (National Instruments), their system is easy to adapt to different experimental requirements even for non-experts. But this approach also limits the overall performance in terms of resource consumption and speed of execution. Jakob et. al. proposed to use a single multiply-accumulate cell (MAC), to calculate all correlation channels in serial manner [14], the so called “virtual correlator architecture”. Our design further extends and optimizes both of these hardware reuse schemes: Only a single MAC per correlator is used, which allows us to implement 32 parallel correlators within a single Virtex-2 FPGA. These can then process up to 1024 pixels or input signals, using an advanced multiplexing scheme.

*μ*s time scale, which covers the range of motion of small molecules in solution and living cells. We presented several benchmark experiments that show the applicability and functionality of our correlator system. The low photon detection probability due to the small fill factor of our SPAD array will be addressed with next-generation sensors equipped with microlenses.

*T*̃ ≫ 1s) it is possible to output the accumulated global and delayed values after the last block. As the data rate is reduced by a factor of 2

^{14}, then the correlation can easily be processed on the host computer.

*μ*s. For a region of interest containing 64 pixels (2 lines), a frame rate of 102.4MHz/64 = 1.6MHz and therefore a minimal lag time of

*τ*

_{min}= 0.625

*μ*s can be achieved. As our current implementation accepts up to 16bit wide inputs, we can overcome the clipping due to the one bit counters within our sensor: The data acquisition logic will be extended to accumulate consecutive frames. A valid choice are 2bit counters, that allow the sum of three subsequent images. This results in a possible full frame integration time of 3.33

*μ*s, which is slightly above the sensor’s limit of (2.66

*μ*s). Newer FPGAs in combination with next-generation sensors, will allow us to increase the processing speed even further.

## Acknowledgments

## References and links

1. | D. Magde, E. L. Elson, and W. W. Webb, “Fluorescence correlation spectroscopy i: conceptual basis and theory,” Biopolymers |

2. | D. Magde, E. L. Elson, and W. W. Webb, “Fluorescence correlation spectroscopy. ii. an experimental realization,” Biopolymers |

3. | O. Krichevsky and G. Bonnet, “Fluorescence correlation spectroscopy: the technique and its applications,” Rep. Prog. Phys. |

4. | M. Engels, B. Hoppe, H. Meuth, and R. Peters, “A single chip 200 MHz digital correlation system for laser spectroscopy with 512 correlation channels,” in “ISCAS’99. Proceedings of the 1999 IEEE International Symposium on Circuits and Systems, 1999,”, vol. 5 (IEEE , 1999), vol. |

5. | B. Hoppe, H. Meuth, M. Engels, and R. Peters, “Design of digital correlation systems for low-intensity precision photon spectroscopic measurements,” in “IEEE Proceedings Circuits, Devices and Systems ,”, vol. |

6. | M. Engels, B. Hoppe, H. Meuth, and R. Peters, “Fast digital photon correlation system with high dynamic range,” in “Proceedings of the 13th Annual IEEE International ASIC/SOC Conference, 2000,” (IEEE, 2000), pp. 18–22. |

7. | M. Wahl, I. Gregor, M. Patting, and J. Enderlein, “Fast calculation of fluorescence correlation data with asynchronous time-correlated single-photon counting,” Opt. Express |

8. | T. Laurence, S. Fore, and T. Huser, “A fast, flexible algorithm for calculating correlations in fluorescence correlation spectroscopy,” Opt. Lett. |

9. | E. Schaub, “F2cor: fast 2-stage correlation algorithm for FCS and DLS,” Opt. Express |

10. | D. Magatti and F. Ferri, “Fast multi-tau real-time software correlator for dynamic light scattering,” Appl. Opt. |

11. | D. Magatti and F. Ferri, “25 ns software correlator for photon and fluorescence correlation spectroscopy,” Rev. Sci. Instrum. |

12. | M. Culbertson and D. Burden, “A distributed algorithm for multi-tau autocorrelation,” Rev. Sci. Instrum. |

13. | B. Tieman, S. Narayanan, A. Sandy, and M. Sikorski, “Mpicorrelator: a parallel code for performing time correlations,” Nucl. Inst. Meth. A |

14. | C. Jakob, A. Schwarzbacher, B. Hoppe, and R. Peters, “The development of a digital multichannel correlator system for light scattering experiments,” in “Irish Signals and Systems Conference, 2006. IET,” (IET, 2006), pp. 99–103. |

15. | C. Jakob, A. T. Schwarzbacher, B. Hoppe, and R. Peters, “A FPGA optimised digital real-time mutichannel correlator architecture,” in “10th Euromicro Conference on Digital System Design Architectures, Methods and Tools, 2007. DSD 2007,” (IEEE, 2007). |

16. | C. Jakob, A. Schwarzbacher, B. Hoppe, and R. Peters, “A multichannel digital real-time correlator as single FPGA implementation,” in “15th International Conference on Digital Signal Processing, 2007,” (2007), pp. 276–279. |

17. | Y. Yang, J. Shen, W. Liu, and Y. Cheng, “Digital real-time correlator implemented by field programmable gate array,” in “CISP’08. Congress on Image and Signal Processing, 2008,”, vol. 1 (IEEE, 2008), vol. 1, pp. 149–151. |

18. | W. Liu, J. Shen, and X. Sun, “Design of multiple-tau photon correlation system implemented by FPGA,” in “ICESS’08. International Conference on Embedded Software and Systems, 2008,” (IEEE, 2008), pp. 410–414. |

19. | G. Mocsar, B. Kreith, J. Buchholz, J. W. Krieger, J. Langowski, and G. Vamosi, “Note: multiplexed multiple-tau auto- and cross-correlators on a single field programmable gate array,” Rev. Sci. Instrum. |

20. | M. Burkhardt and P. Schwille, “Electron multiplying ccd based detection for spatially resolved fluorescence correlation spectroscopy,” Opt. Express |

21. | R. A. Colyer, G. Scalia, I. Rech, A. Gulinatti, M. Ghioni, S. Cova, S. Weiss, and X. Michalet, “High-throughput FCS using an LCOS spatial light modulator and an 8 × 1 SPAD array,” Biomed. Opt. Express |

22. | R. Colyer, G. Scalia, F. Villa, F. Guerrieri, S. Tisa, F. Zappa, S. Cova, S. Weiss, and X. Michalet, “Ultra high-throughput single molecule spectroscopy with a 1024 pixel SPAD,” in “Proc. SPIE ,” |

23. | G. Heuvelman, F. Erdel, M. Wachsmuth, and K. Rippe, “Analysis of protein mobilities and interactions in living cells by multifocal fluorescence fluctuation microscopy,” Eur. Biophys. J. |

24. | F. Bestvater, Z. Seghiri, M. S. Kang, N. Gröner, J. Y. Lee, I. Kang-Bin, and M. Wachsmuth, “EMCCD-based spectrally resolved fluorescence correlation spectroscopy,” Opt. Express |

25. | D. J. Needleman, Y. Xu, and T. J. Mitchison, “Pin-hole array correlation imaging: highly parallel fluorescence correlation spectroscopy,” Biophys. J. |

26. | B. Kannan, L. Guo, T. Sudhaharan, S. Ahmed, I. Maruyama, and T. Wohland, “Spatially resolved total internal reflection fluorescence correlation microscopy using an electron multiplying charge-coupled device camera,” Anal. Chem. |

27. | T. Wohland, X. Shi, J. Sankaran, and E. H. K. Stelzer, “Single plane illumination fluorescence correlation spectroscopy (SPIM-FCS) probes inhomogeneous three-dimensional environments,” Opt. Express |

28. | J. Capoulade, M. Wachsmuth, L. Hufnagel, and M. Knop, “Quantitative fluorescence imaging of protein diffusion and interaction in living cells,” Nat. Biotechnol. |

29. | L. Carrara, C. Niclass, N. Scheidegger, H. Shea, and E. Charbon, “A gamma, x-ray and high energy proton radiationtolerant CMOS image sensor for space applications,” in “ISSCC, IEEE International Solid-State Circuits Conference,” (2009), pp. 40–41. |

30. | M. Gösch, A. Serov, T. Anhut, T. Lasser, A. Rochas, P. Besse, R. Popovic, H. Blom, and R. Rigler, “Parallel single molecule detection with a fully integrated single-photon 2 × 2 CMOS detector array,” J. Biomed. Opt. |

31. | R. Colyer, G. Scalia, T. Kim, I. Rech, D. Resnati, S. Marangoni, M. Ghioni, S. Cova, S. Weiss, and X. Michalet, “High-throughput multispot single-molecule spectroscopy,” in “Proceedings-Society of Photo-Optical Instrumentation Engineers,”, vol. 7571 (NIH Public Access, 2010), vol. 7571, p. 75710G. |

32. | C. Veerappan, J. A. Richardson, R. J. Walker, D.-U. Li, M. W. Fishburn, Y. Maruyama, D. Stoppa, F. Borghetti, M. Gersbach, R. K. Henderson, and E. Charbon, “A 160x128 single-photon image sensor with on-pixel 55ps 10b time-to-digital converter.” in “ISSCC, IEEE International Solid-State Circuits Conference,” (IEEE, 2011), pp. 312–314. |

33. | C. Niclass, M. Sergio, and E. Charbon, “A single photon avalanche diode array fabricated in 0.35-μm CMOS and based on an event-driven readout for TCSPC experiments,” in “Proc. SPIE ,” |

34. | K. Schätzel, “Noise on photon correlation data: I. autocorrelation functions,” Quantum Opt. |

35. | K. Schätzel, “New concepts in correlator design,” Inst. Phys. Conf. Ser. |

36. | Z. Kojro, A. Riede, M. Schubert, and W. Grill, “Systematic and statistical errors in correlation estimators obtained from various digital correlators,” Rev. Sci. Instrum. |

37. | J. Sankaran, X. Shi, L. Ho, E. Stelzer, and T. Wohland, “ImFCS: a software for imaging FCS data analysis and visualization,” Opt. Express |

38. | The diffusion coefficient was |

39. | T. Wocjan, J. Krieger, O. Krichevsky, and J. Langowski, “Dynamics of a fluorophore attached to superhelical DNA: FCS experiments simulated by brownian dynamics,” Phys. Chem. Chem. Phys. |

40. | C. Niclass, C. Favi, T. Kluter, M. Gersbach, and E. Charbon, “A 128 × 128 single-photon imager with on-chip column-level 10b time-to-digital converter array capable of 97ps resolution,” in “ISSCC, IEEE International Solid-State Circuits Conference,” (IEEE, 2008), pp. 44–594. |

41. | K. Greger, J. Swoger, and E. H. K. Stelzer, “Basic building units and properties of a fluorescence single plane illumination microscope,” Rev. Sci. Instrum. |

42. | Joachim Wuttke: lmfit - a C/C++ routine for Levenberg-Marquardt minimization with wrapper for least-squares curve fitting, based on work by B. S. Garbow, K. E. Hillstrom, J. J. Moré, and S. Moshier. Version 3.2, retrieved on 2011-08-31 from http://www.messen-und-deuten.de/lmfit/. |

43. | QuickFit 3.0 can be downloaded free of charge from http://www.dkfz.de/Macromol/quickfit/. In addition to the fitting capabilities, it also contains software implementations of the correlators described in here. |

44. | S. T. Hess and W. W. Webb, “Focal volume optics and experimental artifacts in confocal fluorescence correlation spectroscopy,” Biophys. J. |

**OCIS Codes**

(040.0040) Detectors : Detectors

(040.1240) Detectors : Arrays

(040.1490) Detectors : Cameras

(100.4550) Image processing : Correlators

(180.2520) Microscopy : Fluorescence microscopy

(180.6900) Microscopy : Three-dimensional microscopy

(300.6280) Spectroscopy : Spectroscopy, fluorescence and luminescence

(040.1345) Detectors : Avalanche photodiodes (APDs)

**ToC Category:**

Detectors

**History**

Original Manuscript: March 22, 2012

Revised Manuscript: June 1, 2012

Manuscript Accepted: June 21, 2012

Published: July 20, 2012

**Virtual Issues**

Vol. 7, Iss. 9 *Virtual Journal for Biomedical Optics*

**Citation**

Jan Buchholz, Jan Wolfgang Krieger, Gábor Mocsár, Balázs Kreith, Edoardo Charbon, György Vámosi, Udo Kebschull, and Jörg Langowski, "FPGA implementation of a 32x32 autocorrelator array for analysis of fast image series," Opt. Express **20**, 17767-17782 (2012)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-20-16-17767

Sort: Year | Journal | Reset

### References

- D. Magde, E. L. Elson, and W. W. Webb, “Fluorescence correlation spectroscopy i: conceptual basis and theory,” Biopolymers13, 1–27 (1974). [CrossRef]
- D. Magde, E. L. Elson, and W. W. Webb, “Fluorescence correlation spectroscopy. ii. an experimental realization,” Biopolymers13, 29–61 (1974). [CrossRef] [PubMed]
- O. Krichevsky and G. Bonnet, “Fluorescence correlation spectroscopy: the technique and its applications,” Rep. Prog. Phys.65, 251–297 (2002). [CrossRef]
- M. Engels, B. Hoppe, H. Meuth, and R. Peters, “A single chip 200 MHz digital correlation system for laser spectroscopy with 512 correlation channels,” in “ISCAS’99. Proceedings of the 1999 IEEE International Symposium on Circuits and Systems, 1999,”, vol. 5 (IEEE, 1999), vol. 5, pp. 160–163.
- B. Hoppe, H. Meuth, M. Engels, and R. Peters, “Design of digital correlation systems for low-intensity precision photon spectroscopic measurements,” in “IEEE Proceedings Circuits, Devices and Systems,”, vol. 148 (IET, 2001), vol. 148, pp. 267–271.
- M. Engels, B. Hoppe, H. Meuth, and R. Peters, “Fast digital photon correlation system with high dynamic range,” in “Proceedings of the 13th Annual IEEE International ASIC/SOC Conference, 2000,” (IEEE, 2000), pp. 18–22.
- M. Wahl, I. Gregor, M. Patting, and J. Enderlein, “Fast calculation of fluorescence correlation data with asynchronous time-correlated single-photon counting,” Opt. Express11, 3583–3591 (2003). [CrossRef] [PubMed]
- T. Laurence, S. Fore, and T. Huser, “A fast, flexible algorithm for calculating correlations in fluorescence correlation spectroscopy,” Opt. Lett.31, 829–31 (2006). [CrossRef] [PubMed]
- E. Schaub, “F2cor: fast 2-stage correlation algorithm for FCS and DLS,” Opt. Express20, 2184–2195 (2012). [CrossRef] [PubMed]
- D. Magatti and F. Ferri, “Fast multi-tau real-time software correlator for dynamic light scattering,” Appl. Opt.40, 4011–4021 (2001). [CrossRef]
- D. Magatti and F. Ferri, “25 ns software correlator for photon and fluorescence correlation spectroscopy,” Rev. Sci. Instrum.74, 1135–1144 (2003). [CrossRef]
- M. Culbertson and D. Burden, “A distributed algorithm for multi-tau autocorrelation,” Rev. Sci. Instrum.78, 044102 (2007). [CrossRef] [PubMed]
- B. Tieman, S. Narayanan, A. Sandy, and M. Sikorski, “Mpicorrelator: a parallel code for performing time correlations,” Nucl. Inst. Meth. A649, 240–242 (2011). [CrossRef]
- C. Jakob, A. Schwarzbacher, B. Hoppe, and R. Peters, “The development of a digital multichannel correlator system for light scattering experiments,” in “Irish Signals and Systems Conference, 2006. IET,” (IET, 2006), pp. 99–103.
- C. Jakob, A. T. Schwarzbacher, B. Hoppe, and R. Peters, “A FPGA optimised digital real-time mutichannel correlator architecture,” in “10th Euromicro Conference on Digital System Design Architectures, Methods and Tools, 2007. DSD 2007,” (IEEE, 2007).
- C. Jakob, A. Schwarzbacher, B. Hoppe, and R. Peters, “A multichannel digital real-time correlator as single FPGA implementation,” in “15th International Conference on Digital Signal Processing, 2007,” (2007), pp. 276–279.
- Y. Yang, J. Shen, W. Liu, and Y. Cheng, “Digital real-time correlator implemented by field programmable gate array,” in “CISP’08. Congress on Image and Signal Processing, 2008,”, vol. 1 (IEEE, 2008), vol. 1, pp. 149–151.
- W. Liu, J. Shen, and X. Sun, “Design of multiple-tau photon correlation system implemented by FPGA,” in “ICESS’08. International Conference on Embedded Software and Systems, 2008,” (IEEE, 2008), pp. 410–414.
- G. Mocsar, B. Kreith, J. Buchholz, J. W. Krieger, J. Langowski, and G. Vamosi, “Note: multiplexed multiple-tau auto- and cross-correlators on a single field programmable gate array,” Rev. Sci. Instrum.83, 046101 (2012). [CrossRef] [PubMed]
- M. Burkhardt and P. Schwille, “Electron multiplying ccd based detection for spatially resolved fluorescence correlation spectroscopy,” Opt. Express14, 5013–5020 (2006). [CrossRef] [PubMed]
- R. A. Colyer, G. Scalia, I. Rech, A. Gulinatti, M. Ghioni, S. Cova, S. Weiss, and X. Michalet, “High-throughput FCS using an LCOS spatial light modulator and an 8 × 1 SPAD array,” Biomed. Opt. Express1, 1408–1431 (2010). [CrossRef]
- R. Colyer, G. Scalia, F. Villa, F. Guerrieri, S. Tisa, F. Zappa, S. Cova, S. Weiss, and X. Michalet, “Ultra high-throughput single molecule spectroscopy with a 1024 pixel SPAD,” in “Proc. SPIE,” 7905, 790503–1 (2011).
- G. Heuvelman, F. Erdel, M. Wachsmuth, and K. Rippe, “Analysis of protein mobilities and interactions in living cells by multifocal fluorescence fluctuation microscopy,” Eur. Biophys. J.38, 813–828 (2009). [CrossRef] [PubMed]
- F. Bestvater, Z. Seghiri, M. S. Kang, N. Gröner, J. Y. Lee, I. Kang-Bin, and M. Wachsmuth, “EMCCD-based spectrally resolved fluorescence correlation spectroscopy,” Opt. Express18, 23818–23828 (2010). [CrossRef] [PubMed]
- D. J. Needleman, Y. Xu, and T. J. Mitchison, “Pin-hole array correlation imaging: highly parallel fluorescence correlation spectroscopy,” Biophys. J.96, 5050–5059 (2009). [CrossRef] [PubMed]
- B. Kannan, L. Guo, T. Sudhaharan, S. Ahmed, I. Maruyama, and T. Wohland, “Spatially resolved total internal reflection fluorescence correlation microscopy using an electron multiplying charge-coupled device camera,” Anal. Chem.79, 4463–4470 (2007). [CrossRef] [PubMed]
- T. Wohland, X. Shi, J. Sankaran, and E. H. K. Stelzer, “Single plane illumination fluorescence correlation spectroscopy (SPIM-FCS) probes inhomogeneous three-dimensional environments,” Opt. Express10, 10627–10641 (2010). [CrossRef]
- J. Capoulade, M. Wachsmuth, L. Hufnagel, and M. Knop, “Quantitative fluorescence imaging of protein diffusion and interaction in living cells,” Nat. Biotechnol.29, 835–839 (2011). [CrossRef] [PubMed]
- L. Carrara, C. Niclass, N. Scheidegger, H. Shea, and E. Charbon, “A gamma, x-ray and high energy proton radiationtolerant CMOS image sensor for space applications,” in “ISSCC, IEEE International Solid-State Circuits Conference,” (2009), pp. 40–41.
- M. Gösch, A. Serov, T. Anhut, T. Lasser, A. Rochas, P. Besse, R. Popovic, H. Blom, and R. Rigler, “Parallel single molecule detection with a fully integrated single-photon 2 × 2 CMOS detector array,” J. Biomed. Opt.9, 913 (2004). [CrossRef] [PubMed]
- R. Colyer, G. Scalia, T. Kim, I. Rech, D. Resnati, S. Marangoni, M. Ghioni, S. Cova, S. Weiss, and X. Michalet, “High-throughput multispot single-molecule spectroscopy,” in “Proceedings-Society of Photo-Optical Instrumentation Engineers,”, vol. 7571 (NIH Public Access, 2010), vol. 7571, p. 75710G.
- C. Veerappan, J. A. Richardson, R. J. Walker, D.-U. Li, M. W. Fishburn, Y. Maruyama, D. Stoppa, F. Borghetti, M. Gersbach, R. K. Henderson, and E. Charbon, “A 160x128 single-photon image sensor with on-pixel 55ps 10b time-to-digital converter.” in “ISSCC, IEEE International Solid-State Circuits Conference,” (IEEE, 2011), pp. 312–314.
- C. Niclass, M. Sergio, and E. Charbon, “A single photon avalanche diode array fabricated in 0.35-μm CMOS and based on an event-driven readout for TCSPC experiments,” in “Proc. SPIE,” 6372, 63720S (2006). [CrossRef]
- K. Schätzel, “Noise on photon correlation data: I. autocorrelation functions,” Quantum Opt.2, 287–305 (1990). [CrossRef]
- K. Schätzel, “New concepts in correlator design,” Inst. Phys. Conf. Ser.77, 175–184 (1985).
- Z. Kojro, A. Riede, M. Schubert, and W. Grill, “Systematic and statistical errors in correlation estimators obtained from various digital correlators,” Rev. Sci. Instrum.70, 4487–4496 (1999). [CrossRef]
- J. Sankaran, X. Shi, L. Ho, E. Stelzer, and T. Wohland, “ImFCS: a software for imaging FCS data analysis and visualization,” Opt. Express18, 25468–25481 (2010). [CrossRef] [PubMed]
- The diffusion coefficient was D = 20μm2/s (corresponding to an intermediately sized protein in water), the simulation timestep of the random walk, as well as the minimum lag time were Δtsim = τmin = 1μs. There were around 1.2 particles in the effective measurement volume Veff ≈ 0.4μm3 on average.
- T. Wocjan, J. Krieger, O. Krichevsky, and J. Langowski, “Dynamics of a fluorophore attached to superhelical DNA: FCS experiments simulated by brownian dynamics,” Phys. Chem. Chem. Phys.11, 10671–10681 (2009). [CrossRef]
- C. Niclass, C. Favi, T. Kluter, M. Gersbach, and E. Charbon, “A 128 × 128 single-photon imager with on-chip column-level 10b time-to-digital converter array capable of 97ps resolution,” in “ISSCC, IEEE International Solid-State Circuits Conference,” (IEEE, 2008), pp. 44–594.
- K. Greger, J. Swoger, and E. H. K. Stelzer, “Basic building units and properties of a fluorescence single plane illumination microscope,” Rev. Sci. Instrum.78, 023705 (2007). [CrossRef] [PubMed]
- Joachim Wuttke: lmfit - a C/C++ routine for Levenberg-Marquardt minimization with wrapper for least-squares curve fitting, based on work by B. S. Garbow, K. E. Hillstrom, J. J. Moré, and S. Moshier. Version 3.2, retrieved on 2011-08-31 from http://www.messen-und-deuten.de/lmfit/ .
- QuickFit 3.0 can be downloaded free of charge from http://www.dkfz.de/Macromol/quickfit/ . In addition to the fitting capabilities, it also contains software implementations of the correlators described in here.
- S. T. Hess and W. W. Webb, “Focal volume optics and experimental artifacts in confocal fluorescence correlation spectroscopy,” Biophys. J.83, 2300–2317 (2002). [CrossRef] [PubMed]

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.