Spectrum-optimized direct image reconstruction of super-resolution structured illumination microscopy

Abstract

Super-resolution structured illumination microscopy (SR-SIM) has become a widely used nanoscopy technique for rapid, long-term, and multi-color imaging of live cells. Precise but troublesome determination of the illumination pattern parameters is a prerequisite for Wiener-deconvolution-based SR-SIM image reconstruction. Here, we present a direct reconstruction SIM algorithm (direct-SIM) with an initial spatial-domain reconstruction followed by frequency-domain spectrum optimization. Without any prior knowledge of illumination patterns and bypassing the artifact-sensitive Wiener deconvolution procedures, resolution-doubled SR images could be reconstructed by direct-SIM free of common artifacts, even for the raw images with large pattern variance in the field of view (FOV). Direct-SIM can be applied to previously difficult scenarios such as very sparse samples, periodic samples, very small FOV imaging, and stitched large FOV imaging.

Introduction

Structured illumination microscopy (SIM) stands out from the current super-resolution (SR) optical nanoscopy methodologies for its rapid multicolor acquisition, efficient photon budget, and compatibility with general fluorescent labeling protocols [1,2,3,4,5,6,7]. SIM can achieve resolution doubling compared to the Abbe diffraction limit with sinusoidal illumination patterns and post-processing algorithms. Currently, most SIM algorithms follow the prevalent linear Wiener deconvolution framework (hereafter referred to as Wiener-SIM) [1, 2], which involves the precise determination of illumination patterns, and complex Fourier-domain deconvolution. And dedicated calibration of the actual point spread function (PSF) is usually required to reconstruct high-quality SR images [8, 9]. However, small parameter errors in these three procedures have been shown to cause substantial artifacts [5, 8, 10]. In particular, determining the illumination patterns from the acquired data not only incurs a heavy computational burden, but is also extremely challenging in many imaging scenarios [8, 9]. If the samples are too sparse or have obvious periodic structures, current methods often fail to determine accurate pattern parameters [9, 11]. More critically, the actual imaging parameters, including the illumination patterns and the preferred PSF, are usually non-uniform over the field of view (FOV) [12]. However, Wiener-SIM assumes these parameters are constants in the FOV, often resulting in unreasonable recombined spectra and severe artifacts by the Wiener deconvolution [9, 12, 13].

To obtain high-fidelity SR-SIM images, a large body of research in the field of SIM imaging has long focused on satisfying the underlying assumptions of the Wiener-SIM architecture. These include improving the experimental acquisition [8, 14,15,16], determining the illumination patterns [2, 11, 17,18,19,20,21,22], optimizing the deconvolution models [19, 23, 24], and fine-tuning the algorithm parameters [25]. Previously, we developed high-fidelity linear and nonlinear SIM algorithms, HiFi-SIM [9] and HiFi-NL-SIM [13], respectively, which effectively remove the typical SIM artifacts by engineering the equivalent SIM PSF; however, they also require prior knowledge of the illumination patterns. Iterative deconvolution techniques [10, 26, 27] and deep learning (DL) approaches [28,29,30] also were proposed to reduce random non-continuous artifacts for low signal-to-noise ratio (SNR) data. However, the initial SIM images for iterative deconvolutions and the training datasets for DL methods still rely on Wiener-SIM algorithms.

Studies have also been conducted on adopting spatial domain reconstruction strategies to reduce the computing burden. SDR-SIM [31] and JSFR-SIM [32] achieved up to sevenfold and 80-fold increase in reconstruction speed, respectively, by transforming the main algorithm procedures into the spatial domain. However, constructing the structured coefficient matrixes still requires pre-estimation of the illumination patterns with conventional cross-correlation methods, so both approaches still face the same parameter estimation challenge as the Wiener-SIM architecture. SP-SIM allowed rapid reconstruction of SR images in the spatial domain without estimating the illumination patterns [33]; however, it has not been applied to biological specimen imaging, largely because the nonlinear reconstruction severely decreases the high frequencies of the image, eventually leading to abnormal contrast. To date, no method has achieved high-quality SR image reconstruction for classical SIM without prior knowledge about the illumination patterns or sample contents.

Here, we developed a direct reconstruction SIM algorithm (direct-SIM) that adopts a joint strategy of spatial-domain direct reconstruction and spectrum optimization. Direct-SIM avoids the illumination pattern estimation, and has a spatial resolution comparable to state-of-the-art Wiener-based SIM algorithms. Benefiting from the novel reconstruction mechanism with non-uniform illumination patterns, direct-SIM exhibits unique advantages in typical artifact suppression, and many imaging scenarios such as very sparse samples, apparently periodic samples, very small FOVs, and stitched large FOVs.

Method

The vast majority of SR-SIM algorithms follow the Wiener-SIM framework, which involves determining the illumination patterns, separating the 0^th-, and ± 1^st-order harmonics and shifting them back to the correct positions, and finally performing the Wiener deconvolution to obtain SR image. In sharp contrast, optical sectioning (OS)-SIM achieves enhanced OS capability by taking the root mean square (RMS) of the differences of raw data [34,35,36]. Adopting a similar RMS-form image model, SP-SIM allows rapid acquisition of SR images without prior estimation of illumination patterns [33]. Both RMS-based approaches essentially have only the ± 1^st-order harmonics contributing to the results; however, the former is mainly suitable for OS-SIM imaging with incoherent illumination, while the latter is a theoretical exploration for SR-SIM reconstruction and has not yet been applied to biological specimen imaging. Inspired by two pioneering works, we explored a new approach termed direct-SIM for directly reconstructing high-quality SR-SIM images, bypassing the requirement for prior knowledge of the illumination patterns and the artifact-sensitive Wiener deconvolution procedures. Specifically, we first proposed an improved spatial domain reconstruction model with RMS-form, in which all harmonic components contribute to the SR image (details in Supplementary Note S1 and Fig. S1):

$$\begin{aligned} SR_{initial} ({\varvec{r}}) &= \sum\nolimits_{d = 1}^{3} {SR_{d} ({\varvec{r}})} \\ &=\sum\nolimits_{d = 1}^{3} {\left\{ {\sqrt {\sum\nolimits_{n = 1}^{3} {\left[ {D_{d,n} ({\varvec{r}}) - \alpha \cdot D_{d,0} ({\varvec{r}})} \right]^{2} } } } \right\}} , \end{aligned}$$

(1)

where $SR_{d} ({\varvec{r}})$ is the initial SR image with the RMS-form model in the d-th orientation, $D_{d,n} ({\varvec{r}})$ is the raw data with $2\pi \left( {n - 1} \right)/3$ pattern phase step, and $D_{d,0} ({\varvec{r}}) = \sum\nolimits_{n = 1}^{3} {D_{d,n} ({\varvec{r}})} /3$ is the equivalent wide-field image; $\alpha \in (0.6,0.9)$ is an empirical constant weight.

Although Eq. (1) is very similar to the generally used OS-SIM equation [34], we found that the obtained image does include high-frequency components and extends the cut-off frequency. For that, we performed Fourier transform on the square of $SR_{d} ({\varvec{r}})$ (details in Supplementary Note S1):

$$\begin{gathered} F\left\{ {SR_{d} ({\varvec{r}})^{2} } \right\}({\varvec{k}}) \cong \frac{{3m_{d,ex}^{2} }}{2}\left[ {\widetilde{S}(\user2{k + k}_{d,ex} ) \cdot \widetilde{H}({\varvec{k}})} \right] \otimes \left[ {\widetilde{S}({\varvec{k}}{ - }{\varvec{k}}_{d,ex} ) \cdot \widetilde{H}({\varvec{k}})} \right]{ + } \hfill \\ \, 3\left( {1{ - }\alpha } \right)^{2} \left[ {\widetilde{S}({\varvec{k}}) \cdot \widetilde{H}({\varvec{k}})} \right] \otimes \left[ {\widetilde{S}({\varvec{k}}) \cdot \widetilde{H}({\varvec{k}})} \right], \hfill \\ \end{gathered}$$

(2)

where $F\left\{ \cdot \right\}$ and ${\mathbf{ \otimes }}$ represent the Fourier transform operator and convolution operator, respectively; $\widetilde{S}({\varvec{k}})$ is the sample spectrum; $\widetilde{H}({\varvec{k}})$ is the optical transfer function (OTF) of the microscope system; $m_{d,ex}$ and ${\varvec{k}}_{d,ex}$ denote the modulation depth and wave vector of the illumination pattern in the d-th orientation, respectively; $\widetilde{S}({\varvec{k}} \pm {\varvec{k}}_{d,ex} )$ is the ± 1^st-order spectra modulated by the illumination patterns. The first term is the convolution result of the -1^st- and + 1^st-order harmonics, containing the high-frequency information that improves the spatial resolution, as shown in Fig. 1a. Benefiting from the shift-integral and broadening properties of convolution operation, the effective spatial frequency can be extended to $(|{\varvec{k}}_{em} + {\varvec{k}}_{d,ex} | - 1)$ pixel without pre-separating and shifting any harmonic from raw data (Supplementary Fig. S1), where ${\varvec{k}}_{em}$ is the cutoff frequency of the microscope system. Of note, the maximum frequency support of the convolution results from Eqs. (1, 2) is $({2}{\varvec{k}}_{em} { - 1})$ pixel, but the actual resolution increase is mainly determined by $({\varvec{k}}_{em} { + }{\varvec{k}}_{d,ex} )$ (Supplementary Note S2 and Fig. S1). The second term on the right side of Eq. (2) is the self-convolution term of the 0^th-order harmonic, representing the low-resolution information (Supplementary Fig. S1). It is worth noting that SR image reconstructed only by the first term above may have obvious fracture features, due to the loss of some structures in the 0^th-order harmonic. To address this issue, direct-SIM introduces the low-frequency signal in the second term above through weight α to balance the structural continuity, spatial resolution, contrast and SNR of the reconstructed image (details in Supplementary Note S2 and Figs. S2-S4). Also, Eq. (2) shows that the initial phase of the structured pattern has no effect on the reconstruction results (Supplementary Note S1), while a high modulation factor $m_{d,ex}$ is crucial for high-quality reconstruction. As such, before executing Eq. (1), the OTF attenuation strategy [9, 18] and Richardson-Lucy (RL) deconvolution using a theoretically PSF [9, 13, 23] were first employed to remove the defocused background and enhance the modulation depths of raw SIM data (Step 1 in Fig. 1b).

Fig. 1

Superresolution mechanism and flow chart of direct-SIM. a The superresolution mechanism of spatial domain reconstruction of direct-SIM is equivalent to the convolution result of the separated -1^st- and + 1^st-order harmonics. Symbol $\otimes$ is the convolution operator. “Line” patterns in Argolight slide were imaged by DeltaVision OMX with 50 ms exposure. b The flow chart of direct-SIM consists of four steps: preprocessing [involving removal of background fluorescence, and enhancement of the equivalent modulation depths via RL deconvolution; in this case, the equivalent modulation depths are 0.312, 0.378, 0.495 (without preprocessing), and 0.572, 0.699, 0.772 (preprocessing)], spatial domain direct reconstruction of the initial image through the RMS-form SR model, nonlinear spectrum optimization to improve image contrast, and RL deconvolution to reconstruct the final SR image. Scale bars, 2 μm in (a), 3 μm in (b)

However, the relative amplitude of the Fourier spectrum of $SR_{initial} ({\varvec{r}})$ (denoted as: $\widetilde{SR}_{initial} ({\varvec{k}})$) in the low-frequency region is significantly higher than that in the high-frequency region, limiting optimal contrast and visual visibility (Step 2 in Fig. 1b; Supplementary Fig. S5b). To tackle the problem, a spectrum optimization was performed on $\widetilde{SR}_{initial} ({\varvec{k}})$ to yield a high-quality SR image. Considering that the imaging process of Eq. (1) can no longer be described by OTF [33, 34, 36], as an alternative, an effective transfer function $\widetilde{U}_{etf} ({\varvec{k}})$ with nonlinear response to $\widetilde{S}({\varvec{k}})$ was defined. Thus, $\widetilde{SR}_{initial} ({\varvec{k}}) \cong \widetilde{S}({\varvec{k}}) \cdot \widetilde{U}_{etf} ({\varvec{k}})$, where $\widetilde{U}_{etf} ({\varvec{k}})$ has a maximum frequency support of $(|\user2{k}_{em} + {\varvec{k}}_{d,ex} | - 1)$ pixel. Although the analytical solution of $\widetilde{U}_{etf} ({\varvec{k}})$ cannot be solved from Eqs. (1, 2), inspired by the concept of optimal OTF for bandwidth-limited imaging [37], an optimization function $\widetilde{W}({\varvec{k}})$ was designed for modulating $\widetilde{U}_{etf} ({\varvec{k}})$ to enhance the high-frequency relative amplitude of $\widetilde{SR}_{initial} ({\varvec{k}})$:

$$\widetilde{W}({\varvec{k}}) = 1{ - }\beta \cdot \sqrt {\widetilde{H}({\varvec{k}}) \otimes \widetilde{H}({\varvec{k}})} ,$$

(3)

where $\beta$ is an empirical constant for adjusting the strength of spectrum optimization (Supplementary Figs. S2-S3). Meanwhile, a Gaussian low-pass filter $\widetilde{G}({\varvec{k}})$ [9] was used to suppress the high-frequency noise (details in Supplementary Code, [38]). In addition, as an option, the OTF attenuation and multi-scale wavelet transform [39] were employed to remove the residual defocused background. With the above optimization, an SR image with improved contrast can be obtained (Step 3 in Fig. 1b; Supplementary Fig. S5b):

$$SR_{direct - SIM} ({\varvec{r}}) = F^{ - 1} \{ \widetilde{SR}_{initial} ({\varvec{k}}) \cdot \widetilde{W}({\varvec{k}}) \cdot \widetilde{G}({\varvec{k}})\} ({\varvec{r}}),$$

(4)

where $F^{ - 1} \left\{ \cdot \right\}$ represents the inverse Fourier transform.

Finally, RL deconvolution was performed on Eq. (4) using a theoretical PSF with twice the NA of wide-field imaging to further improve the quality of the final SR image (Step 4 in Fig. 1b; Supplementary Fig. S5b). For more details on the concept, design considerations, and implementation of direct-SIM, see Supplementary Notes S1-S3 and Supplementary Code [38]. In addition, see Supplementary Note S4 for a detailed discussion of the differences between OS-SIM and direct-SIM.

Results

Direct-SIM reconstructs resolution-doubled super-resolution image

As a proof-of-concept, we first characterized the super-resolving capability of direct-SIM. Fluorescent beads of 100-nm diameter (F8803, Thermo Fisher Scientific Inc., USA) were imaged by a home-built laser-interference SIM [9], and the acquired data were reconstructed by HiFi-SIM and direct-SIM, respectively. Many dense beads were indistinguishable in the wide-field image, appearing as clustered spots (Fig. 2a). As expected, the SR image obtained by direct-SIM is very similar to that obtained by HiFi-SIM, in which clustered beads were resolved as distinguishable individual beads (Fig. 2a-c). The full-width at half-maximum (FWHM) of the average profile of the individual beads by direct-SIM is 109.1 ± 5.2 nm (mean ± s.d.), corresponding to a 2.2 times resolution enhancement of the wide-field imaging (Fig. 2d). The image quality of direct-SIM was evaluated by SQUIRREL analysis plugin [40], showing a resolution-scaled Pearson coefficient (RSP) better than 0.93 (Fig. 2e), thereby validating the super-resolved capability of direct-SIM at high confidence.

Fig. 2

Quantitative validation of the super-resolved capability of direct-SIM. a Fluorescent beads of 100-nm diameter were imaged by our home-built 2D-SIM setup, and SR images were reconstructed using HiFi-SIM and direct-SIM. b Magnified images of the white-box region in (a). c Intensity profiles along the yellow line in (b). d Full-width at half-maxima (FWHMs) of the profiles of 10 beads in the images of widefield, HiFi-SIM, and direct-SIM. e Error map of direct-SIM obtained by SQUIRREL analysis. f “Line” patterns in Argolight slide were imaged by DeltaVision OMX with 50 ms exposure. The equivalent widefield image is shown on the top row, and the SR images were reconstructed by GE SoftWoRx, HiFi-SIM, and direct-SIM ($\alpha { = }0.8$), respectively. g Fourier spectra of the images in (f). h Magnified images of the white-box region in (f). i Intensity profiles along the red lines in (h), showing that two lines with 90 nm separation were resolved by direct-SIM. Scale bars, 2 μm in (a), 0.2 μm in (b), 3 μm in (f) and 1 μm in (h)

To quantitatively assess the enhanced resolution of direct-SIM, a line-pair pattern in a commercial standard slide (Argo-SIM, Argolight, France) was imaged by GE DeltaVision OMX, and the commercial SoftWoRx software and HiFi-SIM were employed as benchmark algorithms (Fig. 2f). As shown in Fig. 2f,g, direct-SIM achieved higher resolution improvement than SoftWoRx and HiFi-SIM. Two parallel lines separated by 90 nm were resolved by direct-SIM according to the Abbe criteria, as shown in the magnified images and cross-sectional plots in Fig. 2h,i, corresponding to a 2.1-fold increase in resolution while using an imaging objective lens with NA = 1.42 and an emission light of 527 nm. It should be pointed out that the spatial resolution of direct-SIM is affected by parameters α and β, and large α and β are conducive to obtaining high spatial resolution (Supplementary Figs. S2-S4), but may sacrifice a little continuity of structural features if the values are too large (Fig. 1a; Supplementary Note S2 and Fig. S4).

Finally, we explored biological specimen imaging with direct-SIM. As a typical biological sample, microtubules in fixed COS-7 cells expressing mEmerald-Tubulin-N-18 (plasmid #54,293, Addgene, USA) were imaged on the GE DeltaVision OMX, and the acquired data were reconstructed by HiFi-SIM and direct-SIM, respectively. As shown in Fig. 3a, direct-SIM demonstrates high-quality reconstruction comparable to HiFi-SIM. Magnified images and cross-section plots in Fig. 3b,c show that closely lying microtubules with a separation smaller than the optical diffraction limit (226 nm: NA = 1.42; Emission light = 527 nm) can be resolved consistently using direct-SIM. Overlay of direct-SIM image with HiFi-SIM image further confirms the consistency between direct-SIM and Wiener-based SIM reconstruction technique (Fig. 3d). The difference between direct-SIM and HiFi-SIM results mainly occurs in the regions with a strong defocused background, as noted in Fig. 3e. These differences are attributed to the low modulation depth in the corresponding region caused by the high background (Fig. 3f). Although direct-SIM does not involve estimating illumination patterns, the modulation depth essentially determines the SR image quality (Eqs. 1–2). The SQUIRREL error map again verifies that the reconstruction errors are mainly concentrated in the regions with a highly defocused background (Fig. 3g), and these signals are usually removed from the SR images because of the good OS ability of direct-SIM (Fig. 3a,e). To improve the SR image quality of high background region, we propose to reconstruct the region individually by appropriately reducing α and β (Supplementary Notes S3 and Fig. S3) [12]. Overall, these results show that direct-SIM is effective in resolving fine details in samples of acceptable quality.

Fig. 3

Improving imaging resolution of microtubule in COS-7 cells by direct-SIM. a Equivalent wide-field image is shown in the left, and the SR images reconstructed by HiFi-SIM and direct-SIM, respectively. Upper-right corner shows the corresponding reconstructed spectrum. b Magnified images of the white box in (a). c Intensity profiles along the white lines in (b). d Magnified images of the magenta box in (a), and the direct-SIM image (in green) is overlaid with the HiFi-SIM image (in magenta), for comparison. e Magnified images of the green box in (a), and the direct-SIM image (in green) is overlaid with the HiFi-SIM image (in magenta), for comparison. f Map of the illumination pattern modulation depth of different sub-regions (sub-block: 64 pixels*64 pixels) in the raw data was determined by the parameter estimation method of HiFi-SIM. g Error map of direct-SIM obtained by SQUIRREL analysis. Scale bars, 6 μm in (a), 0.5 μm in (b) and 2 μm in (d, e)

Direct-SIM achieves SR imaging with minimal artifacts

As it known that most SR-SIM algorithms follow the Wiener-SIM architecture, however, it is an ill-posed inverse problem prone to knotty artifacts [8,9,10]. The generalized Wiener deconvolution procedures not only involve troublesome spectrum separation, spectrum translation and Wiener filtering steps in the frequency domain, but is also highly susceptible to sidelobe artifacts related to OTF mismatch (Supplementary Fig. S7) [8, 9, 21], and periodic honeycomb artifacts related to defocused background [8, 9]. To the best of our knowledge, these two artifacts occur frequently in almost all Wiener-based SIM algorithms, but are difficult to eliminate reliably. HiFi-SIM effectively eliminates most SIM artifacts by optimizing the reconstructed OTF in frequency domain to approximate the ideal form of wide-field imaging OTF [9]. However, the two-step spectrum optimization function of HiFi-SIM requires knowledge of structured patterns, and residual artifacts may still appear if the estimated parameters are incorrect. Direct-SIM eliminates the possible artifacts by performing a simple spatial-domain reconstruction followed by spectrum optimization in frequency domain. Not only does it bypass the above artifact-sensitive procedures in Wiener-SIM architecture, but its spectrum optimization function is simpler than that of HiFi-SIM, since no knowledge of illumination patterns is required, as shown in Eq. (3).

To validate the capability of direct-SIM to suppress typical SIM artifacts, microtubules in live COS-7 cells expressing mEmerald-Tubulin-N-18 (plasmid #54,293, Addgene, USA) were imaged on a home-built setup, and the SR images were reconstructed with fairSIM, HiFi-SIM, SP-SIM, and direct-SIM, respectively (Fig. 4). Clearly, the SR image reconstructed by fairSIM contains serious sidelobe artifacts and honeycomb artifacts (Fig. 4a,e). With HiFi-SIM, most of the artifacts in Fig. 4a were effectively eliminated (Fig. 4b). Although SP-SIM does not require prior knowledge of the illumination patterns, its reconstructed spectrum exhibits an evident nonlinear growth in the low-frequency region (Fig. 4c: upper-right corner; Supplementary Fig. S5a), causing the SR image not only containing high background signals but also exhibit a very different visual visibility from those by fairSIM and HiFi-SIM (Fig. 4c,e). More intractable, some structural features in the SR image of SP-SIM become significantly discontinuous, as only ± 1^st-order components contribute to the reconstruction result. In contrast, direct-SIM overcomes the the challenge that relative amplitude of low-frequency regions in the spatial domain reconstruction model is much higher than that of high-frequency regions though preprocessing and spectrum optimization (Fig. 4d,e; Supplementary Fig. S5b). Moreover, the structural features of direct-SIM have good continuity due to the contribution of 0^th- and ± 1^st- order components. Overall, direct-SIM effectively alleviates the challenges encountered by the three benchmark approaches, obtaining an SR image with minimal artifacts.

Fig. 4

Suppression of typical SIM artifacts of microtubules in live COS-7 cells with direct-SIM. a Equivalent wide-field image (left), and the SR image reconstructed by fairSIM (right). b–d SR images reconstructed by HiFi-SIM, SP-SIM, and direct-SIM, respectively. Upper-right corner shows the corresponding reconstructed spectrum. e Magnified images of the white boxes in (a–d). Scale bars, 5 μm in (a-d) and 0.5 μm in (e)

Another critical factor hindering the high-fidelity imaging of traditional two-dimensional (2D)-SIM is the out-of-focus background. In practice, many organelle structures are primarily suitable for imaging with conventional 2D-SIM modality, as their relevant features are not close enough to the glass coverslip [41]. However, the 2D-SIM modality often suffers from strong background fluorescence and complex artifacts owing to the 'missing cone' of three-dimensional (3D) OTF [2, 5, 42, 43]. To demonstrate the ability of direct-SIM to remove defocused signals and related artifacts, mitochondrial cristae in live U2OS cells stained with MitoTracker™ Green (M7514, Thermo Fisher Scientific Inc., USA) were imaged on a home-built setup, and fairSIM and HiFi-SIM were employed as benchmark algorithms. As shown in Fig. 5a,b, mitochondrial data has high background fluorescence, so the mitochondrial cristae resolved by fairSIM were mixed with severe periodic honeycomb artifacts and residual out-of-focus signals, making it difficult to reliably distinguish between them. HiFi-SIM effectively reduces the typical artifacts in Fig. 5b through a two-step spectrum optimization, but this "Wiener filtering" optimization still causes a little artifact related to defocused signal under such condition (Fig. 5c). With direct-SIM, the honeycomb artifacts and residual background in Fig. 5b,c were effectively eliminated, achieving an SR image with better quality than those by fairSIM and HiFi-SIM (Fig. 5d). Further, another external dataset of microtubules in live COS-7 cells with high background fluorescence was employed for comparative testing, showing that the SR image reconstructed by direct-SIM was superior to those obtained by fairSIM and SP-SIM (Supplementary Fig. S8a-c). It also has comparable artifact suppression performance and higher spatial resolution compared with JSFR-SIM [32] and HiFi-SIM (Supplementary Fig. S8c,d). In a nutshell, direct-SIM could eliminate the typical SIM artifacts by the classical Wiener-SIM architectures.

Fig. 5

Imaging of mitochondrial cristae in live U2OS cells via direct-SIM. a Equivalent wide-field image. b–d SR images reconstructed by fairSIM, HiFi-SIM, and direct-SIM, respectively. Scale bar: 5 μm in (a-d)

Direct-SIM avoids the illumination pattern estimation

To date, most existing SIM algorithms require the determination of structured pattern parameters from raw data using the cross-correlation strategies [2, 9,10,11,12,13, 15,16,17,18,19,20,21,22,23,24,25]. Yet, to the best of our knowledge, the results inferred by the cross-correlation models are strongly affected by the sample contents, such as structural orientations, sample sparsity, modulation contrast, and SNR, as well as the modulation depth of the illumination patterns [8,9,10,11,12,13, 25]. Thus, in practice, the estimated pattern parameters are often unreliable in many imaging scenarios, such as very sparse samples, periodic samples, and very small imaging FOV, limiting the reliable application of SIM [9, 17, 44].

To illustrate this, sparsely distributed fluorescent beads of 100-nm diameter were imaged on a home-built setup (Fig. 6a), and fairSIM was employed to estimate the illumination patterns from acquired data. As a control, the illumination pattern period of the imaging system was calibrated with the same bead sample of medium distribution density. Compared with the calibrated values, the pattern periods in three directions estimated from the sparsely data had errors of 14.45%, 0.04%, and 16.00% (Fig. 6c), causing severe deformation of the round beads (Fig. 6a,d). In contrast, direct-SIM eliminated the pattern estimation dilemma, yielding a high-quality SR image (Fig. 6b,d). Similarly, for samples with evident periodicity, the cross-correlation strategies cannot easily distinguish between illumination pattern periods and sample periods [9, 11]. As shown in Fig. 6e, the cross-correlation map obtained from the periodic sample shows that the peak corresponding to the illumination pattern wavevector (green arrows in Fig. 6e) is mixed with the periodic peaks corresponding to sample structures (light blue profile in Fig. 6e), making it difficult for the cross-correlation method to distinguish them (blue arrows in Fig. 6e). Therefore, the pattern periods for directions D1 and D2 had errors of 15.09% and 3.71% (Fig. 6f), respectively, resulting in the fracture and dislocation of some local structures (Fig. 6g,h: FairSIM; Supplementary Fig. S9c). Intriguingly, by avoiding the illumination pattern estimation, direct-SIM obtained high-quality SR images (Fig. 6g-i; Supplementary Fig. S9). Furthermore, direct-SIM can avoid the manual assistance that most cross-correlation methods require to set a MASK to pre-locate the pattern periods (Fig. 6e; Supplementary Fig. S9b). In summary, direct-SIM could expand the application scenarios of SIM.

Fig. 6

Overcoming the parameter estimation obstacles encountered by the Wiener-based SIM algorithms with direct-SIM. a Equivalent wide-field image of 100-nm fluorescent beads (left), and the SR image reconstructed by fairSIM (right). b SR image reconstructed by direct-SIM. c Comparison of the pattern periods determined by fairSIM with the calibration values of the imaging system. d Magnified images of the green boxes in (a, b), and the direct-SIM image (in red) is overlaid with the equivalent wide-field image (in blue) and fairSIM image (in green), for comparison. e Cross-correlation map obtained by fairSIM. Symbol ${\varvec{k}}_{ex}$ denotes the illumination pattern wave vector, and ${\varvec{k}}_{em}$ denotes the cutoff frequency of the microscope system. f Comparison of the pattern periods determined by fairSIM with the calibration values of the imaging system. g Equivalent wide-field image of “grid” patterns in Argolight slide acquired on DeltaVision OMX (left), and the SR images reconstructed by fairSIM (middle) and direct-SIM (right), respectively. h Magnified images of the green box in (g), and the SR image of direct-SIM (in red) is overlaid with the equivalent wide-field image (in blue) and fairSIM image (in green), for comparison. i Intensity profiles along the magenta lines in (h). Scale bars, 1 μm in (a, b, h), 0.2 μm in (d) and 8 μm in (g)

Direct-SIM reconstructs SR images with non-uniform illumination patterns

Notably, the pattern parameters such as modulation depths determined from the raw data are uneven and have large variance in the FOV (Fig. 3f). However, Wiener-SIM architecture assumes that the illumination patterns remain constant over the entire FOV, and the results inferred from the raw data using a cross-association strategy are indeed uniform constants. As such, the estimated parameters are only approximate values of the actual pattern parameters, limiting the optimal reconstruction quality [12]. In direct-SIM, illumination pattern parameters do not explicitly appear during reconstruction and they only affect the final images locally. That is, direct-SIM adopts a reconstruction mechanism with non-uniform pattern parameters in the FOV, which is more consistent with the actual imaging conditions.

For an extreme demonstration, we flipped three regions along the horizontal (I), vertical (II), and 45° (III) orientations, to mimic large variances, for a set of endoplasmic reticulum (ER) data. Then the flipped raw data were reconstructed by HiFi-SIM and direct-SIM (Fig. 7a,b). As expected, the global pattern parameters by HiFi-SIM were mainly determined by the unrotated region, causing strong non-continuity artifacts in the flipped regions (Fig. 7a,c). In sharp contrast, direct-SIM was unaffected by the pattern differences caused by flipping (Fig. 7b,c), validating its reconstruction mechanism of only locally dependent pattern parameters.

Fig. 7

Direct-SIM reconstruction uses non-uniform illumination patterns instead of estimated parameters that are uniform across the entire FOV in the Wiener-SIM architecture. a Equivalent wide-field image of ER, and the SR image reconstructed by HiFi-SIM. b SR image reconstructed by direct-SIM. c Magnified images of the three flipped regions in (a, b). Raw data of ER was downloaded from Ref. [29]. Scale bars, 3 μm in (a, b) and 1 μm in (c)

To further demonstrate this unique superiority, a very small region (32 × 32 pixels) of caveolae in live U2OS cells expressing mEmerald-Caveolin vector (plasmid #54,025, Addgene, USA) was imaged on a home-built setup, and SR images were reconstructed by HiFi-SIM and direct-SIM, respectively (Fig. 8a). Because the imaging FOV is too small, there is large noise in the cross-correlation map, leading to suspicious pattern parameter estimation. As a result, the reconstructed spectrum of HiFi-SIM had the wrong combination, causing apparent artifacts (Fig. 8a: middle). While, direct-SIM can still reconstruct a high-quality SR image, in which adjacent caveolae can be easily distinguished (Fig. 8a: right). This enables the potential application of direct-SIM for ultrafast dynamic imaging of small regions of interest (ROIs) in live cells, reducing the high phototoxicity of full-field imaging [27, 41].

Fig. 8

Direct-SIM exhibits unique advantages in very small FOV imaging and large stitched FOV imaging scenarios. a Equivalent wide-field image of caveolae in live U2OS cells, and the SR images reconstructed by HiFi-SIM and direct-SIM, respectively. b Equivalent wide-field image of microtubules in live U2OS stained with Tubulin-Tracker Green (C2213S, Beyotime Biotechnology, China), and the SR images reconstructed by fairSIM, HiFi-SIM, and direct-SIM, respectively. Scale bars, 0.4 μm in (a) and 5 μm in (b)

At last, we applied direct-SIM to reconstruct the large-field images stitched from three sets of raw data of microtubules in live U2OS cells acquired on a home-built setup (Fig. 8b). Since the stitched raw data contains three different sets of illumination patterns, the globally unified pattern parameters estimated by fairSIM and HiFi-SIM from the stitched data cannot guarantee optimal reconstruction in such a large FOV, causing obvious artifacts (Fig. 8b: upper-right corner). Benefiting from the reconstruction mechanism with locally dependent illumination patterns, the SR image quality reconstructed by direct-SIM is superior to those reconstructed by fairSIM and HiFi-SIM (Fig. 8b: lower-right corner). In conclusion, direct-SIM could reconstruct high-quality SR images of raw data with large variances of the illumination patterns in the FOV or containing multiple sets of different structured patterns.

Conclusions

In this study, we have developed a new algorithm to directly reconstruct high-quality SR-SIM images (direct-SIM) with a joint reconstruction strategy of spatial-domain reconstruction and spectrum optimization. Direct-SIM provides a spatial resolution comparable to state-of-the-art Wiener-based SIM algorithms without needing any prior knowledge of the illumination patterns. We demonstrate that direct-SIM outperforms advanced benchmark algorithms including HiFi-SIM in many imaging scenarios where illumination pattern parameters are difficult to estimate accurately from raw data (Figs. 6, 8a, S9), acquisition data has non-uniform pattern parameters (Figs. 4, 7, 8b, S8), and some samples with high background such as mitochondria (Fig. 5). On the one hand, benefiting from the avoidance of the troublesome parameter estimation, direct-SIM guarantees the reliable application of SR-SIM where the precise illumination patterns are difficult to determine, such as the samples being too sparse or having obvious periodicity [45, 46], and the imaging FOV being very small. On the other hand, our approach circumvents the artifact-sensitive Wiener-filtering deconvolution procedure in the traditional Wiener-SIM architecture, thereby robustly suppressing the frequently occurring sidelobes artifacts and periodic honeycomb artifacts. This makes direct-SIM more user-friendly, especially for biologists and non-expert users who do not have extensive experience in SIM artifact discernment. More importantly, direct-SIM could achieve high-quality reconstruction of raw data containing multiple sets of different patterns, thanks to its reconstruction mechanism with non-uniform pattern parameters.

Although direct-SIM does not require the apparent determination of the illumination patterns from the raw data, the pattern parameters essentially determine the final SR image quality. As elaborated in Eqs. (1, 2), although the pattern wave vector ${\varvec{k}}_{d,ex}$ and the equivalent modulation depth $m_{d,ex}$ are implicit in the reconstruction process, ${\varvec{k}}_{d,ex}$ affects the spatial resolution, while $m_{d,ex}$ determines the weight of the high-frequency component in the reconstructed SR image. Theoretically, a larger $m_{d,ex}$ is conducive to obtaining high-quality SR images via direct-SIM (Fig. 3d-g). However, in practice, the equivalent modulation depth of the raw data is largely determined by the modulation depth of the illumination patterns, as well as the modulation contrast and SNR of the samples. So, well-aligned light paths, and illumination patterns with precise phase shift and minimal distortion [47, 48] are still crucial for high-quality reconstruction of direct-SIM. Raw data with high SNR and high modulation contrast is beneficial for high-quality reconstruction of direct-SIM. Whereas for samples with sub-optimal modulation contrast or low SNR, the equivalent modulation depth of raw data is usually low. As such, it may be challenging for direct-SIM to decode reliable high-resolution signals by using the spatial-domain reconstruction in Eq. (1). To further extend the wider applicability of direct-SIM, a systematic effort to adopt deep learning approaches, such as U-Net-SNR [28] and rDL-SIM [49], for improving the SNR and equivalent modulation depths of raw data is currently underway. Further sparse deconvolution [27] could be performed on the results of direct-SIM to yield better SR images of low SNR data.

As a nonlinear SR reconstruction technique, the transfer function of direct-SIM is different from that of the linear Wiener-SIM architecture, that is, the two reconstruction architectures modulate the sample features slightly differently. Nevertheless, we quantitatively demonstrated that, for the acquired data of acceptable quality, the SR features of direct-SIM are in good agreement with those of the Wiener-based SIM algorithms. Compared to Wiener-SIM and HiFi-SIM, the simplicity of our approach makes it about twice faster (Supplementary Fig. S10; Supplementary Table S1) and more suitable for real-time reconstruction with graphics processing unit (GPU) acceleration [32, 41]. At last, direct-SIM could be potentially extended to reconstruct high-quality SR images from single-layer 3D-SIM datasets [9, 18], speckle pattern illumination datasets [33], and digital holographic microscopy with structured illumination [50].