## Image reconstruction for structured-illumination microscopy with low signal level |

Optics Express, Vol. 22, Issue 7, pp. 8687-8702 (2014)

http://dx.doi.org/10.1364/OE.22.008687

Acrobat PDF (4151 KB)

### Abstract

We report a new image processing technique for the structured illumination microscopy designed to work with low signals, with the goal of reducing photobleaching and phototoxicity of the sample. Using a pre-filtering process to estimate experimental parameters and total variation as a constraint to reconstruct, we obtain two orders of magnitude of exposure reduction while maintaining the resolution improvement and image quality compared to a standard structured illumination microscopy. The algorithm is validated on both fixed and live cell data with results confirming that we can image more than 15x more time points compared to the standard technique.

© 2014 Optical Society of America

## 1. Introduction

1. T. A. Klar and S. W. Hell, “Subdiffraction resolution in far-field fluorescence microscopy,” Opt. Lett. **24**, 954–956 (1999). [CrossRef]

2. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at naometer resolution,” Science **313**, 1642–1645 (2006). [CrossRef] [PubMed]

3. M. G. Gustafsson, L. Shao, P. M. Carlton, C. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination,” Biophys J. **94**, 4957–4970 (2008). [CrossRef] [PubMed]

4. B. Huang, W. Wang, M. Bates, and X. Zhuang, “Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy,” Science **319**, 810–813 (2008). [CrossRef] [PubMed]

5. T. Dertinger, R. Colyer, R. Vogel, J. Enderlein, and S. Weiss, “Achieving increased resolution and more pixels with superresoltion optical fluctuation imaging (sofi),” Opt. Express **18**, 18875–18885 (2010). [CrossRef] [PubMed]

6. F. C. Zanacchi, Z. Lavagnino, M. P. Donnorso, A. Del Bue, L. Furia, M. Faretta, and A. Diaspro, “Live-cell 3D super-resolution imaging in thick biological samples,” Nat. Methods **8**, 1047–1049 (2011). [CrossRef]

7. L. M. Hirvonen, K. Wicker, O. Mandula, and R. Heintzmann, “Structured illumination microscopy of a living cell,” Eur. Biophys. J. **38**, 807–812 (2009). [CrossRef] [PubMed]

3. M. G. Gustafsson, L. Shao, P. M. Carlton, C. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination,” Biophys J. **94**, 4957–4970 (2008). [CrossRef] [PubMed]

8. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D: Nonlinear Phenomena **60**, 259–268 (1992). [CrossRef]

9. Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM J. Imaging Sci. **1**, 248–272 (2008). [CrossRef]

11. N. Dey, L. Blanc-Feraud, C. Zimmer, P. Roux, Z. Kam, J.-C. Olivo-Marin, and J. Zerubia, “Richardson-Lucy algorithm with total variation regularization for 3d confocal microscope deconvolution,” Microsc. Res. Tech. **69**, 260–266 (2006). [CrossRef] [PubMed]

3. M. G. Gustafsson, L. Shao, P. M. Carlton, C. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination,” Biophys J. **94**, 4957–4970 (2008). [CrossRef] [PubMed]

**94**, 4957–4970 (2008). [CrossRef] [PubMed]

**94**, 4957–4970 (2008). [CrossRef] [PubMed]

*S*is the estimate of the sample information in the Fourier space,

*O*is the optical transfer function for the

_{m}*m*order band,

^{th}*I*is the measured band for the

_{d,m}*m*order and

^{th}*d*direction, and

^{th}*a*is the scaling factor corresponding to

_{d,m}*m*order and

^{th}*d*direction. In the original equation,

^{th}*a*was absorbed in the optical transfer function but is explicitly expressed here. The superscript

_{d,m}^{*}implies the complex conjugate.

**p**

*is the spatial frequency of the*

_{d}*d*direction illumination pattern and itself is a vector.

^{th}*w*is the Wiener parameter and

*A*is an apodization function to suppress the ringing artifacts often associated with the Wiener filter. The experimental parameters such as

*a*and

_{d,m}**p**

*are estimated from the raw data. When the signal is high, we can follow standard steps to find the scaling factors and shift frequencies. However, when the signal is low, the image shot-noise and the electronic detector noise lowers the contrast of the interference pattern seen by the camera, thus making the estimation of the scaling factor difficult and usually resulting in a low amplitude estimate. In this paper, we suggest a new way of estimation which reduces the effect of the noise. Combining this new method of preprocessing and TV constrained image reconstruction yields much improved results with lower exposure levels, thus prolonging the lifetime of the fluorophores within the sample for study.*

_{d}## 2. Estimation of experimental parameters through pre-filtering

12. S. A. Shroff, J. R. Fienup, and D. R. Williams, “Phase-shift estimation in sinusoidally illuminated images for lateral superresolution,” J. Opt. Soc. Am. A **26**, 413–424 (2009). [CrossRef]

13. K. Wicker, O. Mandula, G. Best, R. Fiolka, and R. Heintzmann, “Phase optimization for strucutured illumination microscopy,” Opt. Express **21**, 2032–2049 (2013). [CrossRef] [PubMed]

**94**, 4957–4970 (2008). [CrossRef] [PubMed]

**k**and

**k**+

*m*

**p**

*are confined to the overlap area. Thus, the scaling factor between bands (central band and the side band),*

_{d}*a*, can be estimated through A more robust way to estimate the scaling factor is by using OTF compensation and averaging over the overlap areas, i.e., where the superscript

_{d,m}*ol*means overlap and

*T*= 10% and 0.1%, respectively, while the exposure time is

*t*= 5

*ms*for both cases. The illumination laser power at the sample plane is about 2.5

*mW*when the transmission filter is 100%, its wavelength is 488 nm and the center emission wavelength of the beads is 512 nm). The data was obtained with a commercial structured illumination microscope(

**D**elta

**V**ision

**O**MX, API-GE Healthcare, with an Olympus Plan Apo N 60x 1.42NA objective, the resolution is 120 nm in XY and 300 nm in Z). Figures 1(a) and 1(b) show the diffraction limited images which are the average of 5 phase-shifted images. The Fourier transforms of these images are the central bands, amplitudes of which at the

*f*= 0 plane are shown in Figs. 1(c) and 1(d) corresponding to high and low exposure respectively. We see that when the exposure is high, a gaussian-shaped signal spectrum is clearly visible (Fig. 1(c)). However when a low exposure is used, we can only see the bright stripes along the axes and a peak at the origin. In order to see where the noise aggregates, two cross-sections of the spectrum are computed. One is the radial average of the spectrum on the

_{z}*f*and

_{x}*f*axes while the other is radially averaged over areas other than the axes. The cross-sections of the central band for both high and low signal cases are shown in Figs. 1(e) and 1(f), respectively. In the plots, the cutoff frequencies are marked by the vertical dashed lines and the spectrum beyond this cutoff frequency is purely due to noise. We can see that for the high signal case, both cross-sections within the cutoff frequency are much higher than the noise level. In the low signal case, the on-axis cross-section is significantly higher than the noise but most of the non-axis cross-section is only slightly higher than the noise level. This means that most of the spectrum in the overlap area is low for the central band except areas near the origin and axes. Thus when a low exposure is used, we can only see the bright stripes along the axes and a peak at the origin (Fig. 1(d)). Clearly the noise spectrum is not uniform and it aggregates more at the low frequencies and region along the axes(Fig. 1(d)). With this kind of central band, the estimated scaling factor tends to be much lower than in the high signal case. In this example, the amplitudes of the scaling factors for the 2

_{y}*order bands of three directions are (0.37, 0.32, 0.40) for the high signal case while they are (0.07, 0.06, 0.09) for the low signal case. Using scaling factors this small in the reconstruction would enhance the noise and lower the contrast of the final reconstructed image.*

^{nd}*order bands of three directions are (0.17, 0.16, 0.18) at high signal levels, which are less than half of the values with the beads data even though both samples are thin and the data were taken with the same instrument. With our pre-filtering, these values are all closer to the scaling factor values for the beads data (∼ 0.4). With our estimated scaling factors, the various bands are properly OTF compensated in the Wiener method and the final image has a better contrast laterally and longitudinally (Fig. 2(c)). Note that the images shown here are plotted in the same scale and there is no subtraction or any other local processing. This means that our preprocessing of the data, namely filtering of the central band, helps in providing a better estimate of the scaling factors when the biological sample is complex. When the signal is low, the original estimate of the wave vector for the third direction of the illumination pattern is quite far from the value when the signal is high (high: (∼190, ∼49); low: (∼217, ∼31)). With our pre-processing, the estimate is much closer to the high signal value and the Wiener reconstructed image, while noisy due to the low signals, exhibits no periodic artifacts.*

^{nd}*k*, we empirically set it to be 0.125

*μm*

^{−1}in this paper.

## 3. Multi-channel TV constrained algorithm for SIM

9. Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM J. Imaging Sci. **1**, 248–272 (2008). [CrossRef]

11. N. Dey, L. Blanc-Feraud, C. Zimmer, P. Roux, Z. Kam, J.-C. Olivo-Marin, and J. Zerubia, “Richardson-Lucy algorithm with total variation regularization for 3d confocal microscope deconvolution,” Microsc. Res. Tech. **69**, 260–266 (2006). [CrossRef] [PubMed]

*s*is imaged by 15 channels. In each channel, the object is multiplied by a modulation function

*e*

^{j2πmpd · r}, where

**r**= (

*x*,

*y*,

*z*), before passing through the channel. The channel-specific transfer function is

*a*. Thus our modified TV constrained optimization problem can be formulated as follows: where

_{d,m}O_{m}*x*,

*y*,

*z*) and an integral of it will be the total variation of the object. ⊗ is the convolution operator,

*p*and

_{d,m}*i*are the inverse Fourier transforms of the optical transfer function

_{d,m}*O*and the band

_{m}*I*, respectively (they can be thought as the point spread function and image of the channel corresponding to direction

_{d,m}*d*and order

*m*). The first term inside of the integral in Eq. (6) is the TV constraint and the second term is the data fidelity term. The parameter

*μ*is the relative weight of these two terms during the optimization.

*l*1 term in the optimization function Eq. (6), the traditional gradient optimization method is very slow. A fast and convergence-guaranteed way to solve this optimization problem was developed recently through the Bregman variable splitting method [14

14. T. Goldstein and S. Osher, “The split Bregman method for L1-regularized problems,” SIAM J. Img. Sci. **2**, 323–343 (2009). [CrossRef]

**W**= (

*W*,

_{x}*W*,

_{y}*W*) ⇀ ∇

_{z}*s*. The difference between them,

**b**= (

*W*,

_{x}*W*,

_{y}*W*) − ∇

_{z}*s*, is accounted for by reinserting it back into the optimization problem during each iteration. Thus we can solve the optimization problem of Eq. (6) by iterating Here the parameter

*β*is positive and does not need to be large to ensure

**W**⇀ ∇

*s*[15

15. W. Yin and S. Osher, “Error forgetting of Bregman iteration,” J. Sci. Comput. **54**, 684–695 (2013). [CrossRef]

14. T. Goldstein and S. Osher, “The split Bregman method for L1-regularized problems,” SIAM J. Img. Sci. **2**, 323–343 (2009). [CrossRef]

*s*and

*W*can be further decoupled by solving them in two steps:

*s*

^{n+1}can be directly solved as: where the superscript

*T*means “conjugate transpose”. And Eq. (10) can be solved efficiently by the shrinkage operation [9

9. Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM J. Imaging Sci. **1**, 248–272 (2008). [CrossRef]

Algorithm (TV-SIM) |
---|

Initialization: s^{0} = 0, W^{0} = 0, b^{0} = 0 |

For n = 1, 2,... repeat until a stopping criterion is reached |

Update s according to Eq. (11)^{n} |

Update W according to Eq. (12)^{n} |

Update b according to Eq. (8).^{n} |

Output: s^{n} |

*s*,

^{n}**W**

*and*

^{n}**b**

*can be solved analytically, each iteration is computationally cheap. At the same time, due to the updating of the difference between ∇*

^{n}*s*and

**W**through application of Eq. (8), the convergence of the TV algorithm is achieved with only a few iterations [15

15. W. Yin and S. Osher, “Error forgetting of Bregman iteration,” J. Sci. Comput. **54**, 684–695 (2013). [CrossRef]

*s*=

*s*

^{n}^{+1}−

*s*. In this paper, we stop the algorithm after 20 iterations and usually the algorithm is close to convergence at this point. Regarding the parameters, smaller

^{n}*μ*or larger

*β*will give a smoother image with reduced resolution. We have used

*μ*= 1 × 10

^{6},

*β*= 100 for the TV method throughout this paper. The Wiener parameter

*w*= 10

^{−3}is used for both Wiener methods. A larger

*w*will give a more heavily smoothed image with reduced resolution.

## 4. Testing on the experimental beads data

## 5. Testing on fixed samples

## 6. Test on the live cell data

*μm*(scanned over z with step size 125 nm). The raw data were submitted to the standard Wiener and our TV reconstruction algorithms to obtain the overall results of the structured illumination microscopy. For different exposure settings, different cells were imaged.

16. F. Cichos, C. Von Borczyskowski, and M. Orrit, “Power-law intermittency of single emitters,” Curr. Opin. Colloid Interface Sci. **12**, 272–284 (2007). [CrossRef]

*I*(

*t*) and

*I*

_{0}are the intensities at time point

*t*and the beginning of the measurement respectively;

*t*and

_{A}*α*are parameters describing the intensity decay. In our experiments, we calculate the total intensity in the raw data and plot the decay of the intensity over time in Fig. 8 for both exposure levels. The curves were fitted with the power law and we found that

*t*= 1.331 and 1.113 for high and low exposures respectively,

_{A}*α*= 0.9805 and 0.1608 for high and low cases respectively. If we assume that the image quality at the 3

*time point in the high signal case (Fig. 6(b)) is minimally acceptable, the image at the 50*

^{rd}*time point in the low signal case is of similar quality. Thus we can conclude that for this sample, we have increased the number of usable time points by at least 15 times. However, the exact improvement will depend on the sample and on the photobleaching property of the fluorophores. For instance, if there are both very bright and very dim structures in the sample, the quality of the reconstructed image will be lower and we might expect faster disappearance of those dim structures over time than the bright structures.*

^{th}## 7. Discussion and summary

*et al*[17

17. L. Shao, P. Kner, E. H. Rego, and M. G. Gustafsson, “Super-resolution 3D microscopy of live whole cells using structured illumination,” Nat. Methods **8**, 1044–1046 (2011). [CrossRef] [PubMed]

**94**, 4957–4970 (2008). [CrossRef] [PubMed]

*i*7 processor and 12 GB of RAM. Although this time equates to only approximately 2 minutes to reconstruct a whole 3D volume per time point, we expect that the processing could be significantly sped up using a GPU to parallelize the processing.

## Acknowledgments

## References and links

1. | T. A. Klar and S. W. Hell, “Subdiffraction resolution in far-field fluorescence microscopy,” Opt. Lett. |

2. | E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at naometer resolution,” Science |

3. | M. G. Gustafsson, L. Shao, P. M. Carlton, C. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination,” Biophys J. |

4. | B. Huang, W. Wang, M. Bates, and X. Zhuang, “Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy,” Science |

5. | T. Dertinger, R. Colyer, R. Vogel, J. Enderlein, and S. Weiss, “Achieving increased resolution and more pixels with superresoltion optical fluctuation imaging (sofi),” Opt. Express |

6. | F. C. Zanacchi, Z. Lavagnino, M. P. Donnorso, A. Del Bue, L. Furia, M. Faretta, and A. Diaspro, “Live-cell 3D super-resolution imaging in thick biological samples,” Nat. Methods |

7. | L. M. Hirvonen, K. Wicker, O. Mandula, and R. Heintzmann, “Structured illumination microscopy of a living cell,” Eur. Biophys. J. |

8. | L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D: Nonlinear Phenomena |

9. | Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM J. Imaging Sci. |

10. | S. Setzer, G. Steidl, and T. Teuber, “Deblurring poissonian images by split Bregman techniques,” J. Vis. Commun. Image Representation |

11. | N. Dey, L. Blanc-Feraud, C. Zimmer, P. Roux, Z. Kam, J.-C. Olivo-Marin, and J. Zerubia, “Richardson-Lucy algorithm with total variation regularization for 3d confocal microscope deconvolution,” Microsc. Res. Tech. |

12. | S. A. Shroff, J. R. Fienup, and D. R. Williams, “Phase-shift estimation in sinusoidally illuminated images for lateral superresolution,” J. Opt. Soc. Am. A |

13. | K. Wicker, O. Mandula, G. Best, R. Fiolka, and R. Heintzmann, “Phase optimization for strucutured illumination microscopy,” Opt. Express |

14. | T. Goldstein and S. Osher, “The split Bregman method for L1-regularized problems,” SIAM J. Img. Sci. |

15. | W. Yin and S. Osher, “Error forgetting of Bregman iteration,” J. Sci. Comput. |

16. | F. Cichos, C. Von Borczyskowski, and M. Orrit, “Power-law intermittency of single emitters,” Curr. Opin. Colloid Interface Sci. |

17. | L. Shao, P. Kner, E. H. Rego, and M. G. Gustafsson, “Super-resolution 3D microscopy of live whole cells using structured illumination,” Nat. Methods |

**OCIS Codes**

(100.3010) Image processing : Image reconstruction techniques

(110.0180) Imaging systems : Microscopy

(180.2520) Microscopy : Fluorescence microscopy

(180.6900) Microscopy : Three-dimensional microscopy

(110.4155) Imaging systems : Multiframe image processing

**ToC Category:**

Microscopy

**History**

Original Manuscript: February 18, 2014

Revised Manuscript: March 20, 2014

Manuscript Accepted: March 23, 2014

Published: April 3, 2014

**Virtual Issues**

Vol. 9, Iss. 6 *Virtual Journal for Biomedical Optics*

**Citation**

Kaiqin Chu, Paul J. McMillan, Zachary J. Smith, Jie Yin, Jeniffer Atkins, Paul Goodwin, Sebastian Wachsmann-Hogiu, and Stephen Lane, "Image reconstruction for structured-illumination microscopy with low signal level," Opt. Express **22**, 8687-8702 (2014)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-22-7-8687

Sort: Year | Journal | Reset

### References

- T. A. Klar, S. W. Hell, “Subdiffraction resolution in far-field fluorescence microscopy,” Opt. Lett. 24, 954–956 (1999). [CrossRef]
- E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, H. F. Hess, “Imaging intracellular fluorescent proteins at naometer resolution,” Science 313, 1642–1645 (2006). [CrossRef] [PubMed]
- M. G. Gustafsson, L. Shao, P. M. Carlton, C. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, J. W. Sedat, “Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination,” Biophys J. 94, 4957–4970 (2008). [CrossRef] [PubMed]
- B. Huang, W. Wang, M. Bates, X. Zhuang, “Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy,” Science 319, 810–813 (2008). [CrossRef] [PubMed]
- T. Dertinger, R. Colyer, R. Vogel, J. Enderlein, S. Weiss, “Achieving increased resolution and more pixels with superresoltion optical fluctuation imaging (sofi),” Opt. Express 18, 18875–18885 (2010). [CrossRef] [PubMed]
- F. C. Zanacchi, Z. Lavagnino, M. P. Donnorso, A. Del Bue, L. Furia, M. Faretta, A. Diaspro, “Live-cell 3D super-resolution imaging in thick biological samples,” Nat. Methods 8, 1047–1049 (2011). [CrossRef]
- L. M. Hirvonen, K. Wicker, O. Mandula, R. Heintzmann, “Structured illumination microscopy of a living cell,” Eur. Biophys. J. 38, 807–812 (2009). [CrossRef] [PubMed]
- L. I. Rudin, S. Osher, E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D: Nonlinear Phenomena 60, 259–268 (1992). [CrossRef]
- Y. Wang, J. Yang, W. Yin, Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM J. Imaging Sci. 1, 248–272 (2008). [CrossRef]
- S. Setzer, G. Steidl, T. Teuber, “Deblurring poissonian images by split Bregman techniques,” J. Vis. Commun. Image Representation 21, 193–199 (2010). [CrossRef]
- N. Dey, L. Blanc-Feraud, C. Zimmer, P. Roux, Z. Kam, J.-C. Olivo-Marin, J. Zerubia, “Richardson-Lucy algorithm with total variation regularization for 3d confocal microscope deconvolution,” Microsc. Res. Tech. 69, 260–266 (2006). [CrossRef] [PubMed]
- S. A. Shroff, J. R. Fienup, D. R. Williams, “Phase-shift estimation in sinusoidally illuminated images for lateral superresolution,” J. Opt. Soc. Am. A 26, 413–424 (2009). [CrossRef]
- K. Wicker, O. Mandula, G. Best, R. Fiolka, R. Heintzmann, “Phase optimization for strucutured illumination microscopy,” Opt. Express 21, 2032–2049 (2013). [CrossRef] [PubMed]
- T. Goldstein, S. Osher, “The split Bregman method for L1-regularized problems,” SIAM J. Img. Sci. 2, 323–343 (2009). [CrossRef]
- W. Yin, S. Osher, “Error forgetting of Bregman iteration,” J. Sci. Comput. 54, 684–695 (2013). [CrossRef]
- F. Cichos, C. Von Borczyskowski, M. Orrit, “Power-law intermittency of single emitters,” Curr. Opin. Colloid Interface Sci. 12, 272–284 (2007). [CrossRef]
- L. Shao, P. Kner, E. H. Rego, M. G. Gustafsson, “Super-resolution 3D microscopy of live whole cells using structured illumination,” Nat. Methods 8, 1044–1046 (2011). [CrossRef] [PubMed]

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

### Supplementary Material

» Media 1: AVI (296 KB)

» Media 2: AVI (469 KB)

» Media 3: AVI (3735 KB)

» Media 4: AVI (2022 KB)

« Previous Article | Next Article »

OSA is a member of CrossRef.