Self-inspired learning for denoising live-cell super-resolution microscopy

Self-inspired learning for denoising live-cell super-resolution microscopy
  • Schermelleh, L. et al. Super-resolution microscopy demystified. Nat. Cell Biol. 21, 72–84 (2019).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Schermelleh, L. et al. Subdiffraction multicolor imaging of the nuclear periphery with 3D structured illumination microscopy. Science 320, 1332–1336 (2008).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Lawo, S., Hasegan, M., Gupta, G. D. & Pelletier, L. Subdiffraction imaging of centrosomes reveals higher-order organizational features of pericentriolar material. Nat. Cell Biol. 14, 1148–1158 (2012).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Szymborska, A. et al. Nuclear pore scaffold structure analyzed by super-resolution microscopy and particle averaging. Science 341, 655–658 (2013).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Xu, K., Zhong, G. & Zhuang, X. Actin, spectrin, and associated proteins form a periodic cytoskeletal structure in axons. Science 339, 452–456 (2013).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Mishin, A. & Lukyanov, K. Live-cell super-resolution fluorescence microscopy. Biochemistry 84, 19–31 (2019).

    CAS 

    Google Scholar 

  • Godin, A. G., Lounis, B. & Cognet, L. Super-resolution microscopy approaches for live cell imaging. Biophys. J. 107, 1777–1784 (2014).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Valli, J. et al. Seeing beyond the limit: a guide to choosing the right super-resolution microscopy technique. J. Biol. Chem. 297, 100791 (2021).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Luisier, F., Vonesch, C., Blu, T. & Unser, M. Fast interscale wavelet denoising of Poisson-corrupted images. Signal Process. 90, 415–427 (2010).

    Article 

    Google Scholar 

  • Huang, X. et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat. Biotechnol. 36, 451–459 (2018).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Mandracchia, B. et al. Fast and accurate sCMOS noise correction for fluorescence microscopy. Nat. Commun. 11, 94 (2020).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Weisong, Z. et al. Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy. Nat. Biotechnol. 40, 606–617 (2022).

    Article 

    Google Scholar 

  • Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Chen, J. et al. Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes. Nat. Methods 18, 678–687 (2021).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Fang, L. et al. Deep learning-based point-scanning super-resolution imaging. Nat. Methods 18, 406–416 (2021).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Lehtinen, J. et al. Noise2Noise: learning image restoration without clean data. In Proceedings of the 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 2965–2974 (PMLR, 2018).

  • Batson, J. & Royer, L. Noise2self: blind denoising by self-supervision. In Proceedings of the 36th International Conference on Machine Learning (eds Chaudhuri, K. & Salakhutdinov, R.) 524–533 (PMLR, 2019).

  • Krull, A., Buchholz, T. -O. & Jug, F. Noise2void—learning denoising from single noisy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2129–2137 (2019).

  • Eom, M. et al. Statistically unbiased prediction enables accurate denoising of voltage imaging data. Nat. Methods 20, 1581–1592 (2023).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Zhang, G. et al. Bio-friendly long-term subcellular dynamic recording by self-supervised image enhancement microscopy. Nat. Methods 20, 1957–1970 (2023).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Li, X. et al. Reinforcing neuron extraction and spike inference in calcium imaging using deep self-supervised denoising. Nat. Methods 18, 1395–1400 (2021).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Lecoq, J. et al. Removing independent noise in systems neuroscience data using DeepInterpolation. Nat. Methods 18, 1401–1408 (2021).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Qiao, C. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nat. Biotechnol. 41, 367–377 (2023).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Muller, C. B. & Enderlein, J. Image scanning microscopy. Phys. Rev. Lett. 104, 198101 (2010).

    Article 
    PubMed 

    Google Scholar 

  • Hayashi, S. & Okada, Y. Ultrafast superresolution fluorescence imaging with spinning disk confocal microscope optics. Mol. Biol. Cell 26, 1743–1751 (2015).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Hell, S. W. & Wichmann, J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Opt. Lett. 19, 780–782 (1994).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Vicidomini, G. et al. Sharper low-power STED nanoscopy by time gating. Nat. Methods 8, 571–573 (2011).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Sun, D.-E. et al. Click-ExM enables expansion microscopy for all biomolecules. Nat. Methods 18, 107–113 (2021).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Dertinger, T., Colyer, R., Iyer, G., Weiss, S. & Enderlein, J. Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI). Proc. Natl Acad. Sci. USA 106, 22287–22292 (2009).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Zhao, W. et al. Enhanced detection of fluorescence fluctuations for high-throughput super-resolution imaging. Nat. Photonics 17, 806–813 (2023).

    Article 
    CAS 

    Google Scholar 

  • Born, M. & Wolf, E. Principles of Optics, 7th Edn (Cambridge University Press, 1999).

  • Lequyer, J., Philip, R., Sharma, A., Hsu, W. -H. & Pelletier, L. A fast blind zero-shot denoiser. Nat. Mach. Intell. 4, 953–963 (2022).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Li, X. et al. Spatial redundancy transformer for self-supervised fluorescence image denoising. Nat. Comput. Sci. 3, 1067–1080 (2023).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Chen, X. et al. Self-supervised denoising for multimodal structured illumination microscopy enables long-term super-resolution live-cell imaging. PhotoniX 5, 1–22 (2024).

    Article 

    Google Scholar 

  • Stein, S. C., Huss, A., Hähnel, D., Gregor, I. & Enderlein, J. Fourier interpolation stochastic optical fluctuation imaging. Opt. Express 23, 16154–16163 (2015).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 234–241 (2015).

  • Yun, S. et al. Cutmix: regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF Conference on ICCV, 6023–6032 (2019).

  • Nieuwenhuizen, R. P. et al. Measuring image resolution in optical nanoscopy. Nat. Methods 10, 557–562 (2013).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).

    Article 
    PubMed 

    Google Scholar 

  • Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Adv. Neural Inf. Process. Syst. 30, 6402–6413 (2017).

    Google Scholar 

  • Prakash, M., Lalit, M., Tomancak, P., Krul, A. & Jug, F. Fully unsupervised probabilistic Noise2Void. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), 154–158 (2020).

  • Quan, Y., Chen, M., Pang, T. & Ji, H. Self2self with dropout: learning self-supervised denoising from single image. In Proceedings of the IEEE/CVF Conference on CVPR, 1890–1898 (2020).

  • Pang, T., Zheng, H., Quan, Y. & Ji, H. Recorrupted-to-recorrupted: unsupervised deep learning for image denoising. In Proceedings of the IEEE/CVF Conference on CVPR, 2043–2052 (2021).

  • Richardson, W. H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 (1972).

    Article 

    Google Scholar 

  • Lucy, L. B. An iterative technique for the rectification of observed distributions. Astron. J. 79, 745–754 (1974).

    Article 

    Google Scholar 

  • Guo, Y. et al. Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales. Cell 175, 1430–1442 (2018).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Lu, M. et al. The structure and global distribution of the endoplasmic reticulum network are actively regulated by lysosomes. Sci. Adv. 6, eabc7209 (2020).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Lu, M. et al. ERnet: a tool for the semantic segmentation and quantitative analysis of endoplasmic reticulum topology. Nat. Methods 20, 569–579 (2023).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Sekh, A. A. et al. Physics-based machine learning for subcellular segmentation in living cells. Nat. Mach. Intell. 3, 1071–1080 (2021).

    Article 

    Google Scholar 

  • Harke, B. et al. Resolution scaling in STED microscopy. Opt. Express 16, 4154–4162 (2008).

    Article 
    PubMed 

    Google Scholar 

  • Tortarolo, G. et al. Focus image scanning microscopy for sharp and gentle super-resolved microscopy. Nat. Commun. 13, 7723 (2022).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Liu, T. et al. Multi-color live-cell STED nanoscopy of mitochondria with a gentle inner membrane stain. Proc. Natl Acad. Sci. USA 119, e2215799119 (2022).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Wen, G. et al. High-fidelity structured illumination microscopy by point-spread-function engineering. Light Sci. Appl. 10, 70 (2021).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Qiao, C. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods 18, 194–202 (2021).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Zhang, Y. et al. Mitochondria determine the sequential propagation of the calcium macrodomains revealed by the super-resolution calcium lantern imaging. Sci. China Life Sci. 63, 1543–1551 (2020).

  • Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J. & Maier-Hein, K. H. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211 (2021).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Mirza, M. & Osindero, S. Conditional generative adversarial nets. Preprint at (2014).

  • Cao, H. et al. Swin-Unet: Unet-like pure transformer for medical image segmentation. In European Conference on Computer Vision, 205–218 (2022).

  • Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 448–456 (2015).

  • Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. Preprint at (2014).

  • Biggs, D. S. & Andrews, M. Acceleration of iterative image restoration algorithms. Appl. Opt. 36, 1766–1775 (1997).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Otsu, N. A threshold selection method from gray-level histograms. IEEE Trans. Syst. 9, 62–66 (1979).

    Google Scholar 

  • Ershov, D. et al. TrackMate 7: integrating state-of-the-art segmentation algorithms into tracking pipelines. Nat. Methods 19, 829–832 (2022).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Qian, H., Sheetz, M. P. & Elson, E. L. Single particle tracking. Analysis of diffusion and flow in two-dimensional systems. Biophys. J. 60, 910–921 (1991).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Ba, Q., Raghavan, G., Kiselyov, K. & Yang, G. Whole-cell scale dynamic organization of lysosomes revealed by spatial statistical analysis. Cell Rep. 23, 3591–3606 (2018).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Damenti, M., Coceano, G., Pennacchietti, F., Boden, A. & Testa, I. STED and parallelized RESOLFT optical nanoscopy of the tubular endoplasmic reticulum and its mitochondrial contacts in neuronal cells. Neurobiol. Dis. 155, 105361 (2021).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Zhao, W. et al. Quantitatively mapping local quality of super-resolution microscopy by rolling Fourier ring correlation. Light Sci. Appl. 12, 298 (2023).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Li, M. et al. LuckyProfiler: an ImageJ plug-in capable of quantifying FWHM resolution easily and effectively for super-resolution images. Biomed. Opt. Express 13, 4310–4325 (2022).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Lee, J. et al. Versatile phenotype-activated cell sorting. Sci. Adv. 6, eabb7438 (2020).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • Zhang, X. et al. Development of a reversibly switchable fluorescent protein for super-resolution optical fluctuation imaging (SOFI). ACS Nano 9, 2659–2667 (2015).

    Article 
    CAS 
    PubMed 

    Google Scholar 

  • Tillberg, P. et al. Protein-retention expansion microscopy of cells and tissues labeled using standard fluorescent proteins and antibodies. Nat. Biotechnol. 34, 987–992 (2016).

    Article 
    CAS 
    PubMed 
    PubMed Central 

    Google Scholar 

  • link