In this section, we present four SCLMs developed in our lab that not only exemplify the basic ideas of computational light microscopy presented in the previous section but also demonstrate their potential impact on the function and performance of microscopic instruments and the associated applications. They are shown in Fig. 3. These SCLMs are empowered by advanced computational microscopy techniques, including DHM, TIE, DPC, FPM, and lensless holography. Through proper optimization of optical modulation and information processing mechanisms, different imaging functions and performance suitable for different application scenarios can be achieved. The general design concepts, imaging functions, performance indexes, and intended biomedical applications are summarized in Fig. 4.
Digital holographic smart computational light microscope (DH-SCLM): DH-SCLM [Fig. 4(a)] is based on a classic computational microscopy approach — digital holography. By replacing the light source of a conventional microscope with a coherent laser and introducing an additional reference beam path, the invisible phase information can be modulated into the visible interference fringes, and then quantitatively reconstructed based on Fourier fringe analysis techniques. In DH-SCLM, by rational design of the optical system and associated reconstruction algorithms, we have overcome several inherent problems in holographic imaging, such as speckle noise, unreliability of phase unwrapping, phase aberration, and sensitivity to environmental perturbation. The resulting system has many unique features for micro-optics metrology and cell biology, including nanometric optical path length resolution, full-field measurement, real-time QPI, digital refocusing, automatic denoising, and aberration compensation. More details about DH-SCLM will be presented in “Paradigm 1: digital holographic smart computational light microscope (DH-SCLM)” section.
Multi-contrast quantitative phase smart computational light microscope (MQP-SCLM): MQP-SCLM [Fig. 4(b)] is a powerful multi-contrast computational phase microscope by combining illumination modulation (programmable LED illumination) with imaging optics coding (an ETL). The RGB programmable LED array gives the microscope additional flexibilities for illumination direction modulation and wavelength tuning. In addition, the ETL makes fast, telecentric, motion-free focal plane adjustment of the microscope system possible. The combination of two tunable optical devices allows MQP-SCLM to integrate seven different imaging modalities within a single microscopic system, including bright field, dark field, light field, Rheinberg optical staining, DPC, TIE quantitative phase microscopy, and FPM. In addition, the use of partially coherent LED illumination endows MQP-SCLM with higher phase imaging resolution (up to incoherent diffraction limit) and speckle-free image quality, which is extremely valuable for microscopic imaging of unlabeled biological cells and tissues. More details about DH-SCLM will be presented in “Paradigm 2: multi-contrast quantitative phase smart computational light microscope (MQP-SCLM)” section.
Miniaturized multi-contrast smart computational light microscope (MMC-SCLM): MMC-SCLM [Fig. 4(c)] is a miniaturized version of MQP-SCLM by removing the tunable lens and tailoring the optical system. The physical length of the optical imaging system is significantly shrunk by using a customized and non-standardized microscope objective lens, reducing the size of the entire microscope to 14 ×16.5×20cm3, making it compact, portable, and easily be fitted into a cell culture chamber or apply to telemedicine. In addition, the RGB programmable LED array also endow MMC-SCLM with the multi-contrast imaging ability of bright field, dark field, Rheinberg optical staining, DPC quantitative phase microscopy. More details about MMC-SCLM will be presented in “Paradigm 3: miniaturized multi-contrast smart computational light microscope (MMC-SCLM)” section.
High-throughput lensless smart computational light microscope (HTL-SCLM): HTL-SCLM [Fig. 4(d)] is a minimalist microscope by discarding all lenses and other bulky optical components of a conventional microscopic system. HTL-SCLM is composed of just a color LED and an image sensor so that it is small enough to fit in the palm of the hand. Based on multi-wavelength phase retrieval, this microscope offers high-throughput quantitative phase images with a pixel-super-resolved half-pitch imaging resolution of 870nm across a wide FOV of 29.8474 mm2. Such a powerful and miniaturized imaging device may offer a cost-effective tool for telemedicine applications or point-of-care diagnostics in locations where advanced laboratory facilities are unavailable. More details about HTL-SCLM will be presented in “Paradigm 4: high-throughput lensless smart computational light microscope (HTL-SCLM)” section.
We have developed specialized digital imaging microscope software for each system to unleash all features of our SCLMs. They enable users to acquire, process, and analyze images in multiple dimensions and over various timepoints. The optimized algorithm implementation and non-destructive image handling specially developed for SCLMs guarantee reproducible results for different application experiments. In addition, powerful analysis tools are developed to dynamically extract data from imagery for precise experiment results. The design architecture of the software will be introduced in “Computational light microscope: software development and hardware implementation” section. In each paradigm, we will describe the detailed features of the supporting software.
Paradigm 1: digital holographic smart computational light microscope (DH-SCLM)
Our first microscope is based on a very classical computational microscope approach, and it is called DH-SCLM. It is constructed as an off-axis interference system in which removing the zero-order and twin-image is generally trivial [42, 43, 51, 53, 201–204]. We optimally designed the optical system and associated reconstruction algorithms to overcome the inherent problems in holographic imaging such as speckle noise, reliability unwrapping, phase aberration, and sensitivity to environmental perturbation. In addition, several auxiliary functions, such as digital DIC (generate digital DIC image from recovered quantitative phase image), 2D quantitative phase map/3D pseudo phase profile display, morphological measurements, are integrated into the DH-SCLM system to provide powerful microscopic analysis tools for biological applications. In this subsection, we present the system structure, software design, and implementing algorithms of DH-SCLM. Diversified biological application experiments demonstrated that our DH-SCLM achieves a lateral resolution of 409nm (half-pitch imaging resolution with 40 × objective lens) and an axial resolution of 10nm for quantitative phase measurement.
Optical configuration and software
Figure 5(a) shows the physical diagram of our DH-SCLM system, the hardware configuration of our DH-SCLM is based on an off-axis interferometry system. As shown in Fig. 5(b), we adopt a laser with a central wavelength of 532nm, a spectral width of 1nm as the coherent illumination source. The emitted light is divided into two beams by the splitter prism to generate object light wave and reference light wave. In order to separate the twin images and the zero diffraction order automatically in the Fourier domain, these two waves interfere in the camera plane at an angle of 2.45 ° apart. Finally, a series of interference fringes are captured by a camera sensor (The Imaging Source, DMK 23U274, 1600 ×1200, with the pixel size of 4.4 μm), as shown in Fig. 5(c1). For different imaging applications, our DH-SCLM is equipped four objective lenses with magnification of 4 ×(0.13 NA), 10 ×(0.25 NA), 20 ×(0.45 NA), and 40 ×(0.65 NA), providing multi-scale imaging capabilities. The detailed performance indexes can be found in Fig. 4(a).
The supporting processing software is designed for DH-SCLM to perform numerical reconstruction of quantitative phase and multidimensional display. We defined four functional modules containing imaging setting, camera control, view display, and data post-analysis to implement flexible digital operations. The imaging setting module is used to determine the algorithm parameters for the phase reconstruction. First, the currently used objective lens should be selected to update the objective parameters of the phase reconstruction algorithm. Next, when the specimen cannot be focused by manual adjustment, digital refocusing is performed to obtain a high-precision in-focus image of the specimen by numerical propagation. In the phase reconstruction process, three phase unwrapping algorithms, i.e., DCT unwrapping [48], branch cut unwrapping [49], and quality-guided unwrapping [205], are provided for different observation specimens. In addition, this module also provides error correction and compensation capabilities. Spectrum centering, sub-pixel shift, and principal component analysis (PCA) algorithms, are flexibly worked to obtain highly robust reconstruction phases. The camera control module integrates the software development kit (SDK) for the camera, which allows adjustable settings of the brightness, contrast, exposure time, gain, and other parameters to meet the imaging requirements. View display module contains four view windows for presenting the results of the software running. We provide digital DIC display, 2D quantitative phase map, and 3D pseudo phase profile display to present the reconstruction results in high contrast. Data post-analysis module is designed to highlight or numerically analyze target regions or cells of interest for observers. On the one hand, the display contrast can be processed by histogram filtering to highlight the desired information of the specimen. On the other hand, the quantitative profile measurement can be performed for any cell to extract its quantitative data, establishing an accurate database for biological analysis. All implementation algorithms can be executed in real time so that the results will be presented on the display window without delay.
Principle and algorithms
∙Principle of DH-SCLM
DH-SCLM is an off-axis interferometry system in which separating the zero-order and ±1-order spectrums are generally trivial in the Fourier domain and one or the other can be removed through a bandpass filter. Assuming a reference wave with a relative angular shift θ, the reference light wave and object light wave interfere with each other, generating a hologram with the intensity distribution of:
$$ I_{h}(x,y)=|O|^{2}+|R|^{2}+|R|Oexp(-iksin\theta x)+|R|O^{*}exp(iksin\theta x) $$
(1)
where R(x,y) and O(x,y) denote the reference and object waves, respectively, ∗ denotes the complex conjugation. Taking Fourier transform on the hologram, the influence of the two phase factors exp(±iksinθx) can be interpreted as the translations of the spatial frequencies associated with the ±1-order spectrums from the center of the Fourier plane to higher frequencies domain. The spatial frequencies of the 0-order or direct-current-term (DC-term) are located in the center of the Fourier plane, producing the background distribution. While the spatial frequencies of the interference terms vary at different carrier frequencies, which are located symmetrically with respect to the center of the Fourier plane: −ksinθx for the +1-order image and ksinθx for -1-order image. This suggests that the +1-order or -1-order spectrum can be extracted from the hologram spectrum by selecting the appropriate bandpass filter, as shown in Step2 of Fig. 6. By performing inverse Fourier transform for the extracted subspectrum, the wrapped phase distribution of the specimen ϕw(x,y) can be obtained:
$$ \phi_{w}(x,y) =angle\{\mathscr{F}^{-1}\{\mathscr{F}\{I_{h}(x,y)\}\times H(u,v)\}\} $$
(2)
where \(\mathscr {F}\) and \(\mathscr {F}^{-1}\) denote the Fourier transform and its inverse transform. The H(u,v) is the filter function in Fourier domain, which should eliminate all spatial frequencies except those of the +1-order interference term wanted. At this time, the retrieved phase distribution ϕw(x,y) is wrapped to the values ranging between −π and π.
Phase unwrapping must be carried out in order to remove the discontinuities from their wrapped values and to obtain an estimate of the true continuous phase map. We achieve three phase unwrapping algorithms in our DH-SCLM, i.e., DCT unwrapping [48], branch cut unwrapping [49], and quality-guided unwrapping [205], as shown in Step5 of Fig. 6. They provide different imaging capabilities in algorithm complexity, noise robustness, imaging efficiency, etc. DCT unwrapping is appropriate for dynamic specimens due to its fewer computational time costs. The quality-guided unwrapping offers significantly improved imaging accuracy, which allows it to be used for high precision and high accuracy 3D quantitative phase reconstruction. The branch cut unwrapping method achieves a balance between imaging accuracy and resolution, thus it is suitable for general specimens.
After we obtain the quantitative phase of the specimen, it can be post-processed to form a digital DIC image, which displays the phase gradient of the specimen along an arbitrary shear direction:
$$ I_{DIC}(x,y)=B\left(1+cos \alpha \frac{d\phi}{dx}+sin \alpha \frac{d\phi}{dy}\right) $$
(3)
here B is a constant to generate a uniform background, α denotes the shear direction of the phase gradient. \(\frac {d\phi }{dx}\) and \(\frac {d\phi }{dy}\) are phase derivatives along x and y directions. In conventional DIC microscopes, the shear direction is often difficult to adjust, resulting in limited specimen phase gradient information. In our DH-SCLM, α can be digitally selected, allowing the system to be adjusted more flexibly and provide richer visualized information for the specimen.
∙ error correction algorithms
Due to the coherent nature of the laser sources, digital holograms of DHM are corrupted by a mixture of coherent speckle and incoherent noise, which can severely degrade the reconstruction quality [206–211]. Moreover, DHM is sensitive to mechanical vibrations and air fluctuations that typically affect any interferometric systems. These challenges restricted the wide usage of DHM in biological applications. In our DH-SCLM, we carefully addressed the inherent problems of DHM in speckle noise, system aberration, defocus error, and susceptibility by targeted compensation and correction algorithms on a case-by-case basis (which are displayed in Step3 and Step4 of Fig. 6). In the following presentation of this subsection, the compensation and correction algorithms used in our DH-SCLM will be described in detail.
In the coherent imaging systems, speckle noise results from the coherent nature of the light source, which may deteriorate the quality of the images/fringe patterns [212]. In our DH-SCLM, the speckle noise is suppressed by means of weighted average superposition processing. Specifically, the captured holograms over a very small time interval are superimposed and averaged with given weights. And the weights can be adjusted for different imaging speed requirements to optimize the degree of smoothing. In this method, the resolution of the processed hologram is preserved because any pixel is combined with the homologous pixels in the acquisition stack. In addition, an effective high-speed phase reconstruction scheme obtains a single frame reconstructed phase by averaging each acquired image with the adjacent images, then the final reconstruction rate remains consistent with the limit-frame rate of the camera.
The system aberrations of DHM generally include off-axis tilt aberrations and quadratic wavefront distortion, and they are compensated by different numerical approaches in our DH-SCLM. Off-axis tilt aberration roots in the incorrect selection of +1-order spectrum center, which introduces a linear phase factor in the virtual image terms. Thus, the spatial frequencies associated with the ±1-order spectrums are translated from the center of the Fourier plane to the higher frequencies domain. In our DH-SCLM, the off-axis tilt aberration is corrected by a simple spectrum centering procedure. Its realization begins with the 2D surface fitting for the extracted +1-order spectrum, whose maximum point coordinates are searched and locked as +1-order spectrum zero frequency. Then, the sub-pixel displacement of the spectrum is realized by multiplying the phase shift factor in the spatial domain, achieving the high accuracy of spectrum center alignment. Quadratic wavefront distortion is induced by the mismatch of interference optical paths because the objective lens is involved only in the object light wave, and it results in concentric circular patterns (introduced by the quadratic phase factor) in the reconstructed phase. PCA algorithm is a simple and effective numerical phase aberration compensation method to correct for both quadratic wavefront distortion and tilt aberration [213, 214]. It decomposes the phase map into a set of values of uncorrelated variables called principal components and then extracts the aberration terms from the first principal component. Finally, the estimated aberration is removed by subtracting the estimated aberrations from the reconstructed phase, only leaving the phase of the specimen.
Our DH-SCLM corrects the defocus error by seeking the precise defocus distance based on numerical propagation and the Laplace operator. As shown in Step4 of Fig. 6, the complex amplitude distribution of the extracted +1 spectrum is propagated to acquire the complex field distribution with defocus distance Δz. Then, the overall image clarity of the phase distribution at different defocus distances can be evaluated by the variance of the edge image extracted by the second-order Laplace operator. In order to locate the focus plane quickly, we adopt the dichotomy method to speed up the search. The post-processing digital focusing process is integrated into the software’s defocus correction module, and only one button is needed to get the precise defocus distance.
Experiments results
∙ error correction
In order to demonstrate the effectiveness of error correction of our DH-SCLM, we conducted several experiments on different specimens. Figure 7 displays the imaging results of pollen cells to compare the reconstructed phase without and with correcting off-axis tilt distortion and quadratic wavefront distortion. The recorded interference hologram can be seen in Fig. 7(a), and its spectrum is shown in Fig. 7(b). The incorrect centering of the zero frequency usually extracts shifted +1-order spectrum, as shown in the solid white box in Fig. 7(c). As a result, an extra low-frequency phase is attached in the phase of the specimen, overwhelming the phase information of the specimen [Fig. 7(d)]. In contrast, our correction method achieves the high-precision spectrum centering so that all phase information of the specimen is clearly visible in the aberration-corrected image [Fig. 7(e)]. Three areas of interest are extracted and amplified to compare the imaging performance without and with aberration correction. From Fig. 7(f1)-(f3) [without aberration correction] and Fig. 7(g1)-(g3) [with aberration correction], the specimen information is correctly recovered with aberration correction, providing a quantitative phase distribution that can be used for analysis.
HeLa cells were then imaged as specimens to compare the imaging results without and with defocus correction to verify the performance of the defocus correction. Figure 8(a) shows the reconstructed phase without defocus correction, which exhibits the serious detail loss of the reconstructed phase. The defocus distance obtained from the numerical propagation was applied for compensation of the defocus error, and the results shown in Fig. 8(b) can be obtained. The defocus artifacts are effectively suppressed so that detailed information of the subcellular structure is extracted correctly. We selected two enlarged areas of interest from Fig. 8(a) and (b), and then enlarged them in Fig. 8(c1), (d1), (e1), and (f1). It can be seen that in the defocus-uncorrected image, the phase artifacts distort the contour and internal structure of the specimen. Defocus correction compensates for the loss of resolution, provides clearer image detail information, and the vesicles and profile of the cell are clearly distinguished. Also, these reconstructed phases generate the phase gradient images of the specimen by digital DIC, and they are shown in Fig. 8(c2), (d2), (e2), and (f2).
∙ quantitative phase imaging and analysis
We demonstrated the imaging resolution of the DH-SCLM by the experiment on standard phase resolution targets (QPTTM, Benchmark Technologies Corporation, USA, RI n=1.52). This target contains seven sub-targets of varying heights (from 50∼350nm in 50nm increments), each of which is a USAF resolution target consisting of several steps with equal height. In this experiment, the structure with a height of 100nm was imaged by an objective lens with 10 ×, 0.25NA, and the reconstructed phase is shown in Fig. 9. Figure 9(a) shows the interference hologram captured by the camera sensor. The clear interference fringes can be observed from the magnified hologram shown in Fig. 9(d), which ensures high accuracy and SNR for the measured phase. According to the NA of the objective lens and illumination wavelength, the appropriate spectrum bandwidth filter is automatically selected to recover the quantitative phase of the specimen, and the result shown in Fig. 9(b) directly appeared on the software interface.
Interferometric holography provides the imaging resolution of the diffraction limit of the objective lens, corresponding to the half-pitch resolution of 1.064 μm (10 × objective lens, 0.25NA). We zoomed in on the high-resolution group in Fig. 9(b) to demonstrate the highest achievable resolution. It can be seen that our DH-SCLM system recovers the quantitative phase correctly and reaches its theoretical maximum resolution of Group 8, element 6 (half-pitch resolution of 1.096 μm). We then extracted the phase profile line and plotted the curve to quantitatively analyze the image resolution. As shown in Fig. 9(f), all the unit details of Group 8 are clearly resolved. Furthermore, the reconstructed phase in Fig. 9(b) can be used to achieve DIC imaging and obtain the phase gradient results of the specimen, as shown in Fig. 9(c). Compared with traditional DIC imaging, we can flexibly adjust the direction and degree of the phase gradient to achieve more comprehensive observation and analysis.
Morphological measurement and analysis is an important approach to early diagnosis of cancer because cancer cells often have obvious structural and morphological changes in the early stage [215–217]. Our DH-SCLM system can achieve high-precision, high-speed QPI, providing high-resolution quantitative morphological information of cancer cells. As shown in Fig. 10, the experiments on a static HeLa cell slide and living HeLa cells cultured in vitro were conducted in our DH-SCLM system. We used an objective lens with 20 ×, 0.4NA to imaging the specimen, and the corresponding highest theoretical half-pitch resolution is 665nm. As shown in Fig. 10(b), our system restored high-resolution detail of the HeLa cell correctly, which can also tbe used to generate the high-quality DIC result [Fig. 10(c)]. The microstructures of cells of Area1 and Area2 were extracted to analyze their subcellular morphology, as shown in Fig. 10(d1)-(d2), and 10(e1)-(e2). The complete morphology and subcellular structure of HeLa cells are clearly displayed, providing abundant data for quantitative research applications. In addition, the quantitative phase distribution can be further shown as a 3D pseudo-color rendering image in Fig. 10(f) so that the different phase distributions can be displayed more clearly. Then, we demonstrated the imaging capability of our DH-SCLM to cell growth in the culture dish. The experiment was carried out on HeLa cells cultured on Petri dishes in 10% fetal bovine serum and 90% Dulbecco’s modified eagle medium. Figure 10(g) displays the time-lapse results of the HeLa cell division process. These high-resolution phase images clearly reveal the cell morphological changes during different mitosis phases.
Paradigm 2: multi-contrast quantitative phase smart computational light microscope (MQP-SCLM)
Non-interferometric QPI is a new trend of the phase imaging to overcome the limitation of interferometric QPI approaches, providing the unique advantages of high quality image, high resolution, high robustness towards variant disturbance, and not needing phase unwrapping. Among the various non-interferometric QPI methods, the deterministic QPI methods often require only a few intensity images to achieve phase retrieval, which makes them superior for practical label-free imaging applications. We exhibited a revolutionary multi-contrast non-interferometric quantitative phase microscope containing seven imaging approaches for the first time, which was called MQP-SCLM. Instead of the traditional illumination source, a programmable LED array source is employed to flexibly switch the illumination patterns of bright field, dark field, light field, Reinberg optical staining, DPC, QPI (TIE-based), and FPM in a fixed microscope configuration. All imaging algorithms are integrated with a user interface by corresponding unified hardware platform and software framework. Furthermore, we step forward the widely spread biomedical applications with reducing significant expense and complexity of the related optical hardware and explore the potentially possible instrumentation techniques for MQP-SCLM. The investigations of various stained and unstained biological specimens using different types of objectives are presented in “Experiments” subsection, and results show the new possibility of widespread adoption of MQP-SCLM in the morphology study of cellular processes and the biomedical community.
Optical configuration and software
The physical diagram of MQP-SCLM is shown in Fig. 11(a), and Fig. 11(b)-(c) illustrate the photograph of key parts of the system hardware. Our system is equipped with an electronic platform whose displacement in all three directions x,y,z can be precisely adjusted by an external control handle [Fig. 11(b)] or software control. As depicted in Fig. 11(d), the optical path of MQP-SCLM is a standard 4-f optical system. The existing condenser-based illumination diagram is replaced with a programmable LED array, which is placed 54mm away from the specimen, and the center of the LED array is aligned with the optical axis of the microscope. The LED illumination array is a 64 ×64 pixels commercially available product with the pixel size of 2mm. Thus, this illumination module can provide an illumination NA up to 0.78 and sufficient illumination uniformity. The RGB LED array has central illumination wavelengths of 623nm, 530nm, and 470nm, and the brightness of a single LED is greater than 2000 cd/m2. The programmable LED array is controlled by a custom-designed field-programmable-gate-array-based (FPGA-based) controller (Altera EP4CE10E22C8N), providing a high LED illumination pattern refresh rate (>1kHz) and high display color-scale (8-bit grayscale for RGB channels). High refresh rate refresh frequency can effectively improve the uniformity of the bright field image and eliminate the image blurring caused by the camera global exposure or the light and dark stripes caused by the rolling exposure. The objective lens with several different NAs (4 ×0.1NA, 10 ×0.25NA, 20 ×0.4NA, 40 ×0.65NA, 60 ×0.8NA) and tube lens with different magnifications are equipped in MQP-SCLM for most biomedical observation and measurement applications. Moreover, the objective indicator is used to monitor the state of the objective, and all system parameters will be matched by the circuit board according to the last used objective lens. Unlike conventional systems with mechanical defocus adjustment, we adopt an ETL [as shown in Fig. 11(c)], which can be controlled by a hardware circuit board to realize the automatic control of defocus distance. It brings faster adjustment speed and higher focusing accuracy so that defocusing images required by TIE algorithm can be quickly and stably collected. A complementary metal oxide semiconductor (CMOS) camera (Basler, MED ace 5.1, 2448 ×2048, 2.2 μm pixel size) placed on top of the microscope is synchronized with the control board for image acquisition, and universal serial bus (USB) 3.0 is used to transfer data to the computer with the limited frame rate of 75FPS. When the objective is rotated, the FPGA reports the current objective position to the computer and sets the appropriate system parameters. The FPGA controller also provides a synchronization mechanism for the LED array and camera. It communicates with the computer via USB-to-serial and accepts command parameters from the computer, such as the radius of the display aperture, the color, and the position of the pattern. Next, it reads the bitmap of the display pattern stored in the on-chip random access memory (RAM) and then drives the programmable LED array to display the pattern at high speed. To simplify the microscope interface, we used a USB Hub to combine the USB of the camera and USB-to-serial into one interface. The bandwidth required for serial communication is extremely small, which does not affect the transmission of camera image data, ensuring that the camera works at the highest rate.
The supporting software of MQP-SCLM provides the capabilities of smart control, multi-contrast imaging, and powerful processing and analysis, enabling seven imaging functions to be implemented through the same software system. Four functional modules are defined to coordinate to easily control the system, providing users with high-quality observation images and accurate analysis data. First, the illumination module is used to flexibly adjust the LED illumination pattern to match different imaging modes. Common illumination adjustments such as pattern and color adjustments are provided to tailor the function precisely to the user’s requirements. Also, we defaulted the optimal illumination pattern for each imaging mode in advance to enable the software to acquire a high-quality image for observation. In the camera settings module, we redeveloped the camera’s SDK so that users can easily set the camera exposure time, white balance, gain, and other parameters. According to the used objective lens and illumination parameters, all camera parameters are also optimally preset in our software to provide superior images for display and subsequent computational reconstruction. The display view module is used to present all direct visualized observation images and post-recovery results. The multi-window display function is designed to monitor simultaneous multi-contrast images in a software interface for in-depth analysis of the specimen. In the smart processing and analysis module, our MQP-SCLM also provides important information on specimens interactively or fully automatically. With powerful image processing functions we can display specimen signals with even greater contrast or SNR. At the same time, besides size and volume information, the detailed data on cell physiological processes, can be extracted from the imaging result for quantitative analysis.
Principle and algorithms
∙ multi-contrast microscopic imaging
Phase contrast approaches, such as dark field imaging [81, 218], Rheinberg imaging [219, 220], DPC imaging [27, 29, 218, 221], which obtain diversified 2D structural information of the specimen, are established as widely and commonly configured label-free technique for microscope to visualize transparent specimens. However, all of these methods require spatially-modulated illuminations by introducing additional physical masks and optical filters before the condenser lens of the microscope. For example, a spider stop is inserted into the aperture focal plane of the condenser to create a hollow cone of illumination with larger NA compared to that of the objective lens, extracting high-resolution detailed information of the specimen. Computational light microscopy brought a revolution of the microscope in diversification, flexibility, and convenience of imaging methods. The active illumination control (LED or LCD) were introduced into the microscope system to replace the traditional aperture diaphragm, realizing so-called “computational illumination” that gains the flexibility to produce sophisticated illumination patterns or dynamically switchable illumination sources with no physically moving parts [81, 83, 84, 92, 109, 222, 223].
In a microscopic system, the NA of the objective lens is a very important parameter because it determines the maximum diffraction angle that the microscope can receive. In our MQP-SCLM, the illumination patterns corresponding to seven imaging approaches are accurately set according to the NA of the objective lens. Specifically, the illumination NA of ith LED can be calculated:
$$ NA^{i}_{ill}=\sin (\theta_{i}) {\textrm{ = }}\sin\left[arctan\left(\frac{{\sqrt {{{\left| {{x_{i}}\Delta d} \right|}^{2}} + {{\left| {{y_{i}}\Delta d} \right|}^{2}}} }}{h}\right)\right] $$
(4)
where Δd is the pixel pitch of LED array, xi and yi are location coordinates of LED units, and h is the distance between the LED array and specimen. And also, the angle of illumination θi=(θix,θiy) can be defined as θix = arctan(xiΔd/h),θiy = arctan(yiΔd/h).
As shown in Fig. 12(a), bright field imaging is the simplest illumination technique, and the bright field image corresponds to illumination by LEDs that lies within the cone of angles described by the objective NA, i.e., \(NA^{i}_{ill}\leq NA_{obj}\). Bright field imaging collects the most of directly transmitted light, thus, it is mostly suitable to visualize amplitude-contrast, e.g., absorptive structures of a specimen. While the dark field method excludes the unscattered beam from the image, and the dark field image is obtained by illuminating the specimen from angles beyond the angular acceptance of the objective (\(NA^{i}_{ill}>NA_{obj}\)). To achieve bright field and dark field imaging, we provide different LED patterns to match the NA of the objective lens. Besides, the Rheinberg illumination, known as optical staining with color patterns, can also be implemented by displaying different patterns on the central cone region (bright field) and the surrounding hollow cone region (dark field) of LED array simultaneously for more complicated color phase-contrast imaging. As a classic form of optical staining, Rheinberg illumination can render the normally colorless specimen with rich color against a contrasting background. Moreover, ZPC can be realized by displaying a ring-type pattern that matches the annular phase plate in the back focal plane of the phase contrast objective lens, and a phase-contrast image can be recorded without using any physical diaphragm.
DPC is a label-free phase contrast imaging based on asymmetric illumination, which only needs to collect several intensity images, achieving twice the imaging resolution of coherent imaging [27, 160, 224]. In contrast to DIC based on interferometric imaging, it provides a flexible, low-cost alternative which only need to replace the light source of commercial bright field microscopes with the LEDs. Our system realizes DPC imaging using the same system configuration. By controlling the LED to produce asymmetrical half-circular illumination patterns, the phase-contrast images of the specimen along the asymmetry axis are recorded. Then, a simple differential algorithm is used to calculate the phase gradient distribution of the specimen:
$$ I^{DPC}_{lr}(\mathbf{x})=\frac{I_{l}(\mathbf{x})-I_{r}(\mathbf{x})}{I_{l}(\mathbf{x})+I_{r}(\mathbf{x})} $$
(5)
where Il(x) and Ir(x) represent the captured images in the left and right illuminations, respectively. \(I^{DPC}_{lr}(\mathbf {x})\) is the phase gradient distribution of the specimen along the left-right shearing direction. By adjusting the illumination angle of the asymmetric axis, the phase gradient of the specimen along arbitrary shearing directions can be established. Furthermore, our previously published results have suggested that the optimized asymmetric annular pattern makes it impossible to enhance the contrast of phase gradient images [86].
∙ wide field and high resolution imaging based on FPM
FPM is a recently developed technique that overcomes the physical SBP limit of a bright field microscope by iteratively stitching together a number of variably illuminated, low-resolution intensity images in Fourier space [60, 194, 225, 226]. Instead of starting with a high-resolution objective lens and stitching together a larger FOV, FPM uses a low NA objective lens to take advantage of its innate wide FOV to recover high-resolution complex distribution (both amplitude and phase) of the specimen. As shown in Fig. 12(b), by invoking the LED array, each LED provides a plane wave of unique angle to illuminate the sample, recording a series of low-resolution images. In Fourier space, the sequential scanning illumination angles shift different amounts of high spatial frequency information into the low NA objective lens. These raw images are stitched together in Fourier space to generate a bandwidth-extended, high-resolution spectrum, thus achieving wide-field and high resolution imaging. For the problems in positional misalignment and noise that affect the accuracy of FPM reconstruction, we have proposed several effective correction and optimization methods [89, 227]. In our MQP-SCLM, the LED positional misalignment correction and adaptive step-size strategy are combined for FPM recovery processing to improve the imaging quality of FPM.
∙ quantitative phase imaging based on TIE
Without the need for a separate reference beam, non-interferometric QPI is another developed technique, which directly retrieves phase from intensity measurements [55–58, 61]. The phase retrieval from the recorded intensity is considered as solving a mathematical “inverse problem” [66, 67]. Such kind of problem can be solved by the deterministic algorithm and iterative algorithm [185, 228–232]. Generally, the deterministic algorithm recovers phase from several intensity measurements of the specimen under partially coherent illumination [71, 86, 229, 233–237], and it has higher stability and achieves twice the lateral resolution of the coherent diffraction limit [165, 176, 238, 239]. Defocusing acquisition is the effective method to produce an intensity image of the phase contrast of the specimen, and the developed QPI approach is called TIE. In essence, TIE is a second-order elliptic partial differential equation, outlining the quantitative relationship between the variation of intensity along the optical axis to the phase of the optical field at the plane perpendicular to the optical axis [70]:
$$ {-k\frac{\partial I(\mathbf{x},{z})}{\partial z}=\nabla \cdot \left[I(\mathbf{x},z)\nabla \phi(\mathbf{x})\right] } $$
(6)
where k is the wave number 2π/λ; x is the transverse coordinates x=(x,y); I(x,z) is intensity distribution at the plane located at the propagation distance z. Under the appropriate boundary condition (the Neumann boundary condition), this second-order elliptic partial differential equation is solved by FFT method [185, 240–242], thus the phase information of object can be easily obtained through intensity derivative on the in-focus plane [243–245].
Recently, another deconvolution solution is put forward based on the derivation of defocus phase transfer function (PTF) under weak phase approximation [231, 246], which lays a theoretical foundation for improving TIE imaging performance by optimizing illumination. In our published work, annular illumination has been proved to provide an optimal illumination scheme for TIE which enhances imaging contrast within the whole theoretical bandwidth of partially coherent imaging [246, 247]. In our MQP-SCLM system, we use annular illumination that matching the objective NAobj (NAill equals to NAobj) to collect three defocus images and recover quantitative phase distribution of specimens by one-step deconvolution [Fig. 12(c)]. The captured three images first are calculated to eliminate the background term, leaving only phase term [248, 249]:
$$ {\frac{I_{\Delta z}(\mathbf{u})-I_{-\Delta z}(\mathbf{u})}{4I_{0}(\mathbf{u})}=Im[PTF(\mathbf{u})\phi(\mathbf{u})] } $$
(7)
Then, the PTF of the system can be calculated by the following formula:
$$ {PTF(\mathbf{u})=\iint{S(\mathbf{u'})|P(\mathbf{u'})||P(\mathbf{u'}+\mathbf{u})|e^{ik\Delta z\left(-\sqrt{1-\lambda^{2}|\mathbf{u'}|^{2}}+\sqrt{1-\lambda^{2}|\mathbf{u}+\mathbf{u'}|^{2}}\right)}}d\mathbf{u'} } $$
(8)
Finally, quantitative phase information can be reconstructed by Fourier space de-convolution in one-step [234, 246, 248]:
$$ {\phi(\mathbf{x})=\mathscr{F}^{-1}{\left\{\frac{I_{\Delta z}(\mathbf{u})-I_{-\Delta z}(\mathbf{u})}{4I_{0}(\mathbf{u})}\frac{{{Im[PTF(\mathbf{u})]}}} {|{Im[PTF(\mathbf{u})]}|^{2}+\beta}\right\}}} $$
(9)
here β is the Tikhonov regularization term used to avoid excessive amplification of noise during deconvolution [250].
Experiments
∙ multi-contrast microscopic imaging
We first demonstrated the multi-contrast imaging capabilities of MQP-SCLM by realizing several classical microscope techniques such as bright field, dark field, ZPC, DPC, and Rheinberg optical staining. All these multi-contrast imaging modes can be achieved in a single-shot image by switching the illumination patterns. The experiments were conducted on the unstained single diatom microalgae cell and the unstained spiral algae (S68786, Fisher Scientific). A 40 ×, 0.65NA Plan microscope objective (40 ×0.45NA phase-contrast objective for ZPC) and a 10 ×, 0.25NA objective (10 ×0.25NA phase-contrast objective for ZPC) were used to observe the specimens. Figure 13 shows the multi-contrast imaging results of a small single diatom microalgae cell and spiral algae with different objective lenses.
As shown in Fig. 13(a)-(b), bright field images of two different algae specimens present the highly absorbing features (e.g., chloroplasts) while dark field images give more high-resolution details corresponding to the highly scattering information of diatom frustule shell. In DPC and ZPC, the experimental images provide more information about phase features (e.g., filaments and silica cell wall structures) and the internal structure is much better contrasted and can be well resolved, as illustrated in Fig. 13(c)-(e). Up-down and left-right phase-contrast images enhance the internal features from two shearing directions, and any desired direction of asymmetry can be used to highlight features of interest with the flexibility of the LED array patterning designed by the user through the software interface. In the Rheinberg optical staining imaging mode [Fig. 13(f)-(h)], we provide the experimental results of algae specimens with different color configurations. The central circle illumination pattern (bright field) determines the captured intensity background, while the outer ring illumination pattern (dark field) will stain the highly scattering details of the specimen with different colors. Not only the marginal contours of the diatom are visualized in a distinct manner, but the small pores on the frustule appear sharply demarcated in Rheinberg illuminations. High angle incident illumination will increase the contrast of small features of the specimen which corresponds to the high-frequency details in the spectrum domain. It is worth noting that in Rheinberg filter, the transmittance of the central circle should be much darker than the outer ring in order to enhance the overall contrast and the visibility of the detailed structure. Otherwise, the contrast of the dark field scattering component on the resultant image will be swamped out by the background signal.
Then, we employed a mosquito mouthpart specimen for multiplane refocusing under different contrast imaging modalities. In this experiment, the specimen was imaged by an objective lens of 10 ×0.25 NA, a tube lens of 0.25 ×. The ETL can be automatically controlled by the software to focus on different planes of the specimen. Figure 14 shows its refocusing stack images, and the main structures of the mouthpart are clear and distinguishable on four different focus planes. From bright field refocusing images in the first row, the mosquito head and compound eye provide relatively intensity contrast due to the strong absorption. The structures of antennae, maxillary palps, proboscis (include labella and labium), and labrum with high-resolution information is focused on different planes, and the features of large bushy antennae are marked with solid red circles and dash circles. Moreover, the dark field refocusing images enhance small details of antennae structure with highly scattering, and the hairy antennae feature with high resolution (marked with white circles) can be resolved axially. In the last row, the DPC images at different depths show the high-resolution phase features of mouthpart structure and provide phase gradient maps of antennae and maxillary palps across multiple axial slices (marked with white arrows and red arrows).
∙ wide field, high resolution imaging using FPM
We demonstrated the wide field, high resolution imaging capability of our MQP-SCLM for its intended high-throughput biomedical observation experimentally. The USAF target, human lung tissue section, and human kidney vessel cells were employed as the observation specimens to achieve FPM. We employed an objective lens with 4 ×, 0.1NA and tube lens with 0.5 ×, and collected a set of 225 low-resolution intensity images from different incident angles. Before the FPM reconstruction process, the brightness and the position of LED illumination have been calibrated. Figure 15 shows the FPM reconstructed results of these specimens. The full FOV raw low-resolution intensity image of the USAF resolution target is presented in Fig. 15(a), and the region of interest is marked and enlarged by a white box. Figure 15(b1) and (b2) present the recovered high-resolution intensity images of the corresponding central region of interest of the USAF target. As can be seen, the highest distinguishable resolution target elements in Fig. 15(b2) is Element 5 in Group 10 (corresponding to the half-pitch resolution of 274nm), and the intensity line-scan profile of Group 10 in Fig. 15(b3) shows the verified results. Moreover, we also provide high-resolution measuring specimens of the stained human lung tissue section and human kidney vessel cells. Figure 15(c) and (f) present the FOV of these two specimens, respectively. Figure 15(d1), (e1), (f1), (g1) and (d2), (e2), (f2), (g2) illustrate the comparisons of low-resolution and high-resolution image in the FOV of different enlarged regions. Significantly enhanced resolution in FPM reconstruction results demonstrates more detail in the pathological tissue than the conventional microscope.
∙ quantitative phase imaging using TIE
In order to validate the QPI of MQP-SCLM, two different specimens were used for the quantitative phase characterization. First, a quantitative phase target QPTTM with a height of 100nm was used for the quantitative investigation of achievable imaging resolution, and the reconstructed result is shown in Fig. 16(a1). As shown in Fig. 16(a2)-(a3), the resolution element Group 10 is enlarged and displayed, and the line profile of resolution elements in Group 10 is plotted in Fig. 16(a4). We can clearly distinguish these resolution bars up to Element 4, which means that achievable imaging resolution of MQP-SCLM under 0.4NA objective is 345nm.
Then, a standard plano-convex microlens array (SUSS, pitch 250 μm, fused silica, RI of 1.46 at 507nm) was measured using 0.25NA objective with 0.45 × tube lens. Figure 16(b1) shows the reconstructed quantitative phase. Overall, the distribution of the lens array is clearly reflected in the quantitative phase map. In order to assess the accuracy of the phase measurement, the measured thickness profiles of microlens taken from the solid yellow square in Fig. 16(b1) and theoretical profile line are compared quantitatively in Fig. 16(b3), showing a reasonable agreement. The pitch of microlens is measured to be 252 μm, which is slightly larger than the nominal pitch of 250 μm. The height of the measured microlens is about 25 μm, which is consistent with the nominal value. The small discrepancy may be attributed to the imperfect boundary condition in the phase reconstruction process and the imperfection profile of each lens on the microlens array.
Additionally, the reconstructed quantitative phases of unstained biological specimens are provided to demonstrate the imaging ability of MQP-SCLM on clusters of unstained cells distributed on a glass specimen slide. In this experiment, only three defocus intensity images were captured with 50ms exposure time, enabling the high-speed QPI of MQP-SCLM. Figure 17 gives a comparison of the QPI performances under conventional circular illumination and our optimized annular illumination apertures (with the same objective lens of 40 ×, 0.65 NA). As shown in Fig. 17(a) (conventional circular illumination aperture) and 17(c) (our optimized annular illumination aperture), due to the poor transfer response of the PTF at the central low frequency and high frequency approaching 2NAobj, the low-frequency cloud noise is included in the reconstructed phase in Fig. 17(a) while the high-resolution details are not reconstructed clearly. These deficiencies are avoided under annular illumination aperture, and the reconstructed phase exhibits high robustness and enhanced resolution. Two enlarged sub-areas are selected by dotted rectangular shape to compare phase details. As can be seen in Fig. 17(b1)-(b2) and (d1)-(d2), the morphological shape and internal details of the HeLa cells are clearly visible, and the tiny phase features (e.g., grains and fibers) are well resolved, demonstrating the near-diffraction-limited lateral resolution and high-frequency features in the cell can still be distinguished.
Further, we imaged unstained MCF-7 breast cancer cell and HepG2 living carcinoma cells fixed in formaldehyde solution using the 20 ×0.4NA and 40 ×0.65NA objective lens with 1 × tube lens, respectively. Figure 18(a) is the full FOV quantitative phase image of MCF-7 cell, and we enlarge four sub-regions of the reconstructed phase in Fig. 18(b1)-(b4) to highlight our high quality phase reconstructions. Subcellular tiny grains feature (nuclei and organelle) are correctly reconstructed, and features with higher RI values in the cytoplasm appear brighter contrast in quantitative phase image are reconstructed. As shown in Fig. 18(c), the experimental image of HepG2 also gives the high-resolution quantitative phase distribution. The fine structures and subcellular structures are also visible and resolved. The recovered quantitative phase distributions of cellular features enable the calculation of dry and buoyant mass, sphericity, and other morphometric descriptors used for cell profiling.
Paradigm 3: miniaturized multi-contrast smart computational light microscope (MMC-SCLM)
With telemedicine applications, microscope is required to be as compact and light-weight as possible to provide cost-effective solution for health care especially in the developing world where medical facilities and infrastructure are extremely limited. Computational imaging allows microscope to tailor the optical system to become miniaturized and portable. We developed a miniaturized, low-cost, and portable multi-contrast unlabeled microscope, which is named as MMC-SCLM. In this system, a highly integrated optical path using a miniaturization lens is adopted to replace the complex optical system of traditional microscopes, thus reducing the size of the whole microscope to 14×16.5×20cm3. We are compatible with five imaging modes in this system to produce multi-contrast images of bright field, dark field, Rheinberg optical staining, DPC, QPI (DPC-based) for living cells. All imaging modes can be flexibly switched and operated through our designed software. In addition, cell analyzing functions are also added to our system, which can realize cell counter and 3D profile measurement. In this subsection, we demonstrated the multi-contrast imaging and QPI capabilities of MMC-SCLM by multiple observation and detection experiments with different specimens.
Optical configuration and software
Figure 19(a) displays the physical diagram of our MMC-SCLM, and it is designed as the inverted microscope system. As shown in Fig. 19(b), a commercial full-color LED array with a unit size of 2mm is placed at the top of the entire system, and its center aligned with the optical axis of the microscope system. The complex optical path of traditional microscopes is replaced by a miniaturization lens with NA of 0.14 (6.4 ×), a focal length of 4.25mm, a lens distortion within 1%. This miniaturization lens can image the specimen between its one and two times the focal length directly onto the CCD target. Thus, the working distance of the optical system was reduced to less than 20mm. At the bottom of the whole system, a high-performance color CCD industrial camera (the imaging source DFK 23U445, 1280 ×960, pixel size of 3.75 μm) is used to capture the intensity image for observation and other recovery algorithms. The command interaction between the software system and hardware system is implemented by an independently designed hardware control circuit board, and it is embedded into the back of the microscope, as shown in Fig. 19(c).
Figure 19(d) displays the interactive flow of the whole system. The hardware circuit board is the core of the control, which accepts instructions from the software operating system and controls the hardware to implement response operations. All control instructions are performed through a set serial communication protocol within 1ms. It should be noted that due to the LED array lighting at a certain refresh frequency, the camera exposure needs to be synchronized with the light source to ensure the uniformity and accuracy of the whole image exposure. Therefore, USB 3.0 data transmission protocol is adopted in image acquisition, connecting with the external trigger of the camera to realize the rapidity and stability of image acquisition and transmission. Under such a hardware configuration, the size of the entire microscope system was reduced to 14×16.5×20cm3.
In order to ensure the implementation of multi-contrast imaging and smart analysis, we developed an operating software for MMC-SCLM. Similar to the MQP-SCLM system, the operating software is designed as four modules to realize illumination control, camera setting adjustment, result display, and smart processing and analysis. A detailed description of each module can be found in the “Optical configuration and software” subsection in “Paradigm 2: multi-contrast quantitative phase smart computational light microscope (MQP-SCLM)”. These modules are managed by flexible layout and collaborate for diverse imaging and data analysis. As a result, the specimen can be observed in five imaging modes, including bright field, dark field, DPC, Rheinberg optical staining, and QPI (based-DPC), get continuous data.
Principle and algorithms
∙ multi-contrast microscopic imaging
MMC-SCLM achieves five imaging approaches, including bright field, dark field, DPC, Rheinberg optical staining, and QPI (based-DPC), as shown in Fig. 20. As described in “Principle and algorithms” subsection in “Paradigm 2: multi-contrast quantitative phase smart computational light microscope (MQP-SCLM)”, bright field imaging requires the illumination NAill of all lit LED units are less than NAobj (transmitted light), and whereas dark field imaging requires all NAill are greater than NAobj (diffraction or scattered light). Rheinberg imaging realizes optical staining through a combination of different colors for bright field and dark field illumination to mark different spatial frequency information of specimens, enhancing the imaging contrast.
MMC-SCLM is also capable of phase gradient imaging by DPC. As can be seen from Step1 and Step2 in Fig. 20(b), DPC records the phase contrast images of the specimen by asymmetric illumination, and the phase gradient information of the specimen is further obtained by a simple differential calculation of Eq. (5). However, the compact optical system has a limited distance between the LED and the sensor, which may lead to uneven background under asymmetric illumination. Especially in the DPC-based QPI method, such background unevenness will cause serious reconstruction errors. In order to avoid the imaging error caused by the uneven background, we adopt a correction and compensation method to improve the imaging quality and phase accuracy while ensuring the imaging speed of the DPC imaging microscope system.
The background correction and compensation method is based on PCA and plane fitting. Performing PCA on the recorded phase gradient image and then extracting the first principal component and its corresponding coefficient to fit the plane distribution as image background. The fitted background is then removed from the captured image to obtain the corrected image. When removing the background distribution, a uniform intensity distribution with the fitting background means should be added to the recorded image to compensate for the phase gradient.
∙ QPI based on DPC
Asymmetric illumination is one of the key approaches to produce phase contrast intensity, and the derived QPI method is called DPC QPI. This method was first proposed to establish a linear relationship between the image intensity and the specimen’s phase gradient [85, 86, 228]. Subsequently, slow-varying object approximation and weak object approximation are introduced to simplify imaging process, thus two quantitative phase inversion solvers were developed [228]. Under slow-varying specimen approximation, the phase gradient transfer function (PGTF), which relates the phase gradient and the captured intensity in the sense of a look-up table, was extracted from the partially coherent transfer function. Thus, the measurement of phase gradient along two orthogonal directions allows retrieval of phase distribution [85, 251]. Under weak phase approximation, PTF, which establishes a quantitative relationship between the object phase and the captured intensity, was derived by linearizing partial coherent imaging process [61, 165]. A one-step deconvolution solver was developed to recover the quantitative phase of the object in this approximation. In our recent advances, annular illumination whose illumination NAill equal to the objective NAobj and brightness varies with angle as a cosine function, maximizes the response at almost all frequencies of the theoretical bandwidth of the entire partially coherent imaging (from 0 to \(\frac {{2NA}_{obj}}{\lambda }\)) [86, 165]. Such an annular illumination is used in our MMC-SCLM to optimize the imaging resolution and imaging contrast. In this case, the PTF is solved by:
$$\begin{array}{*{20}l} {PTF_{lr}(\mathbf{u})=\frac{\iint{S_{lr}(\mathbf{u_{j}})\left[{P^{*}(\mathbf{u_{j}})P(\mathbf{u}+\mathbf{u_{j}})-P(\mathbf{u_{j}})P^{*}(\mathbf{u}-\mathbf{u_{j}})}\right]}d^{2}\mathbf{u_{j}}}{\iint{|S_{lr}(\mathbf{u_{j}})||P(\mathbf{u_{j}})|^{2}}d^{2}\mathbf{u_{j}}}} \end{array} $$
(10)
here Slr(uj) is the illumination function, uj is the frequency shift corresponding to the illumination angle. P(u) is the pupil function of the objective lens, which is a low-pass filter with cut-off frequency of \(\frac {NA_{obj}}{\lambda }\). In our system, the PTF is solved and saved in the local folder of the operating software. When recovering the quantitative phase of the specimen in our system, only the PTF is read to implement one-step deconvolution with phase gradient images, as shown in Step4 in Fig. 20(b) [160, 250]:
$$ {\phi(\mathbf{x})=\mathscr{F}^{-1}{\left\{\frac{\sum_{i}\left[{{PTF_{i}^{*}(\mathbf{u}){\cdot}I^{DPC}_{i}(\mathbf{u})}{}}\right]} {\sum_{i}|{PTF_{i}^{*}(\mathbf{u})}|^{2}+\beta}\right\}}} $$
(11)
where ∗ represents the complex conjugate item, and i denotes the asymmetric axis direction of illumination. β is the Tikhonov regularization term used to avoid excessive amplification of noise during deconvolution.
Experiments
∙ multi-contrast microscopic imaging
To demonstrate the multi-contrast microscopic imaging capability of MMC-SCLM, we observed different specimens under bright field, dark field, rainbow dark field, and Rheinberg optical staining imaging. Figure 21 shows the multi-contrast visualized results on the specimens of bird feather, spirogyra communis, lily anther, ant, and dicot. As we can see from the first row, the bird feather can be observed clearly under bright field imaging due to its strong absorption. The scattered light imaging of the dark field provides more detailed information, and the structure of the feather rachis and barbules are clearly displayed. For arbitrary specimens, such as spirogyra communis, lily anther, ant, and dicot, the bright field and dark field imaging compensate each other to acquire the overall structure and high-resolution details of the specimen. In the third column, we show the imaging results of these specimens under rainbow dark field imaging. The spatial structures along different directions are marked with variable colors, which provides intuitive images for distinguishing wispy structures in the specimen. Rheinberg imaging achieves optical staining for different structures of the specimen by flexible, customized illumination pattern (color and shape) to highlight the high-resolution details of the specimen, and the results are shown in the last column. The integration of these imaging methods allows for a diverse view of the specimen without replacing any hardware.
We also employed several of the above specimens to verify the imaging effectiveness of MMC-SCLM for DPC. Figure 22 displays the observation results of bird feather, spirogyra communis, and ant under DPC imaging without and with removing tilt phase gradient. In the blue dotted box on the left, we show the uncorrected DPC results along the left-right and up-down shearing directions. The inhomogeneity of the illumination introduces an undesired tilted phase gradient with respect to the shear direction over the phase gradient of the specimen. The PCA method was executed to remove the tilted phase gradient, and their resulting phase gradients are shown in the red dotted box on the right, demonstrating improvement in smoothness compared with the raw phase gradients. The sub-structural features, as well as the thin borders of the specimens, can be clearly observed without any tilted background perceivable.
In addition, multi-contrast imaging of MMC-SCLM allows the comparison of specimen details through flexible sub-window segmentation displays or multi-window full FOV tiled display. As shown in Fig. 23, we show several examples of multi-contrast display views of our system. Figure 23(a) is the stitching display view of two contrasts imaging results on maize slitting, that is, the left and right (or the upper and lower) sub-regions display the visualized results from different imaging modes. In the same view of the software interface, two phase-contrast images (Rheinberg optical staining imaging in the left sub-region and dark field imaging in the right sub-region) are presented to distinguish different visualization information of a specimen. Both images can be displayed and saved in full view to compare the microstructure detail of any sub-region, as shown in Fig. 23(b1) and (b2). Furthermore, the full FOV can be split into more sub-areas to display more phase contrast observations simultaneously. As shown in Fig. 23(c), four images of the pathology specimen are displayed on the same interface forming a full FOV image so that significant differences can be found from one observation. Alternatively, multiple observations can also be displayed simultaneously on the interface at full FOV size, as shown in Fig. 23(d1)-(d4). These multi-contrast observation tools effectively avoid the instrument modification, complex observation recording and comparison required by conventional multi-contrast imaging, providing a simple, straightforward, and efficient observation means.
∙ quantitative phase imaging using DPC
We validated the fast and highly robust quantitative phase reconstruction of MMC-SCLM through several experiments on unlabeled specimens. First, to verify the achievable imaging resolution of our MMC-SCLM, we measured a pure phase resolution target QPTTM with the height of 100nm, and the experimental results are shown in Fig. 24. Figure 24(a) shows the observed image of this specimen in bright field microscope, which has a weak contrast and is almost invisible due to its weak absorption. The phase gradient images after background correction (the up-down shearing direction) are shown in Fig. 24(a2), and it is used to perform deconvolution to avoid the low-frequency tilt phase error. Figure 24(c) displays the reconstructed phase. The quantitative profile along the red line is then extracted and plotted in Fig. 24(d) to reveal the highest achievable resolution. It can be seen that Group 9, Element 1 can be recovered correctly, which indicates the theoretical realizable resolution of our MMC-SCLM is 970nm (half-pitch resolution). Then, we measured a 250 μm pitch plano-convex quartz microlens array (SUSS MicroOptics, lens diameter 240 μm, hex-packed) by our MMC-SCLM. A microlens element in a solid white frame of Fig. 24(e) is selected as the target for quantitative phase measurement, and the reconstructed quantitative phase is shown in Fig. 24(f). The accuracy of this reconstructed phase is demonstrated by quantitatively comparing their differences to the nominal values. As shown in Fig. 24(g), the profile of the reconstructed phase is basically consistent with the nominal value (except that the measured value of 19.52 μm at the extreme value is slightly smaller than the standard value of 20 μm). In addition, the validated quantitative phase can be displayed with a 2D quantitative phase map or a 3D pseudo-color phase profile in our MMC-SCLM to provide a high-contrast view of the 3D structure.
Due to the advantages of miniaturization, low cost, and portability of MMC-SCLM, it provides a powerful means for telemedicine diagnostic tests, which yields immediate acquisition of information on an individual’s condition to facilitate treatment decisions or further extensive testing, offering the advantages of widening accessibility to diagnosis, minimal sample volumes, reduced costs, and rapid analysis times. We demonstrated the imaging capability of MMC-SCLM by imaging a specimen of HeLa cells prepared in situ, and the experimental results are shown in Fig. 25. Figure 25(a) shows the reconstructed quantitative phase of unlabeled HeLa cells. Figure 25(b) and (c) exhibit the observed images in dark field and phase gradient image in DPC. From these results, dark field and DPC give conspicuous phase-contrast information of the unlabeled sample, while QPI further provides quantitative, high-resolution details. In addition, the unlabeled cell samples can be presented in a high-contrast pseudo-color profile, providing a direct visualized 3D structure of the sample, as shown in Fig. 25(e). Furthermore, the cell counter was developed in MMC-SCLM as an auxiliary function, and the result is shown in Fig. 25(d). It can be seen that the exact number of the HeLa cells in the full FOV is accurately counted as 120.
Paradigm 4: high-throughput lensless smart computational light microscope (HTL-SCLM)
Computational imaging allows the microscope to be simplified as much as possible or even remove the necessary objective lens of conventional microscopes. Such a lensless microscope structure maintains the imaging FOV and provides pixel super-resolution specimen information by implementing computational reconstruction algorithms on its acquired images. In this paradigm, we exhibited a lensless QPI microscope, termed HTL-SCLM, which only contains a camera sensor and illumination source, providing a portable, low-cost, stable, and high-throughput imaging tool. A multi-wavelength illumination scheme is utilized to record three raw images without mechanical moving adjustment. According to diffraction propagation theory, these images can be converted to multi-depth images at three defocusing distances. GS iteration is then performed to recover the high-resolution information of the specimen across full FOV of the sensor size [66, 67]. High-throughput experimental results for different specimens demonstrated that our lensless microscope achieves the high-throughput imaging with a wide FOV of 29.8474 mm2 and imaging resolution (half-pitch imaging resolution) of 870 nm.
Optical configuration and software
Figure 26 shows the schematic diagram of our HTL-SCLM. It does not contain any moveable mechanisms, and the two essential components are a color LED unit and a monochrome CMOS sensor, as shown in Fig. 26(a) and (b). Thus, the size of the whole imaging system was shrink to one-tenth of the conventional microscope. The LED is employed as a coherent illumination source instead of a laser source to produce approximately spatially coherent illumination with the center wavelength of 631nm (red), 522nm (green), and 465nm (blue) (∼10nm bandwidth, ∼100μm size). The use of such a LED not only effectively suppresses the speckle noise and distortions caused by parasitic reflections in the optical setup but also provides a simple and cost-effective implementation for lensless microscope. In our HTL-SCLM setup, the LED unit is placed at 80mm away from the specimen plane to sequentially produce red, green, and blue light to illuminate the sample. An industrial camera (The Imaging Source, DMK-24UJ003, 3872 ×2764) with a pixel size of 1.67μm is used to capture images, thus it can achieve a full FOV of 29.8474 mm2. The specimen could be placed as close to the sensor surface as possible [typically less than 1mm, see Fig. 26(d)] so that the spatial incoherence gating effect will minimize for high-resolution phase imaging. Under the above configurations, the lensless system is integrated in a size of 130×80×75mm3.
The operating software of HTL-SCLM provides a friendly interface and versatile functions, which can be started quickly without complicated training for users. Four modules were developed to implement the algorithm preset, camera control, view display, and data analysis. Algorithm preset module must be performed prior to reconstruction to seek the accurate defocus distance and to select the appropriate image acquisition mode and reconstruction mode. Due to the absence of the objective lens, it is difficult for a lensless microscope to acquire an in-focus image. We overcome this problem by estimating the defocus distance through numerical propagation. In the subsequent reconstruction, the estimated defocus distance will be automatically applied to compensate for the phase artifacts caused by defocus and improve the resolution of phase reconstruction. The image acquisition settings provide two options for reconstruction, i.e., single image acquisition mode for one reconstruction and continuous image acquisition mode for multiple reconstructions. The single image acquisition mode captures only three images needed for single quantitative phase reconstruction and is suitable for a static specimen. The continuous acquisition mode, on the other hand, continues to acquire images to perform quantitative phase reconstruction, providing time-lapse imaging results of dynamic specimens. In reconstruction mode settings, two modes of the region of interest reconstruction and full FOV reconstruction can be selected alternatively. Current window reconstruction is required when the reconstruction result needs to be previewed, while full-field reconstruction should be selected after all parameters have been properly set. The other three modules are similar to several of the systems mentioned above. The camera control module enables flexible adjustment of camera parameters, such as frame rate, exposure time, gain. The view display module is used to display the visualized results and reconstructed phases directly in the software interface with 2D phase map, 3D pseudo phase profile, providing high-throughput imaging data. Besides, the data analysis module can afford quantitative and statistical analysis data by analysis tools of cell counting, cell profile analysis, automatic digital refocusing function. Combined with this operating software, our HTL-SCLM is a new generation of label-free high-throughput imaging tool, enabling smart and automatic microscopic imaging.
Principle and algorithms
Lensless holography is a promising high-throughput QPI method, which bypasses the intrinsical trade-off between the spatial resolution and FOV of conventional microscopes [170, 252–256]. It removes all lenses of the microscopic optical path, thus the light passing through the specimen is directly received with a wide FOV of the sensor surface. However, by use of a small object-to-sensor distance, the collection NA at the detection plane approaches 1, resulting in the imaging resolution limited by the sensor pixel size. Several pixel-super-resolution algorithms were developed to overcome this limitation in which a smaller effective pixel-size is synthesized from a series of low-resolution images captured by sub-pixel shifting or wavelength scanning through specific computational algorithms [182, 257–260]. However, all these methods require extra controllable mechanical devices or extra wavelength calibration and dispersion compensation processes, resulting in a significant increase in the complexity and cost of the whole system.
We combined phase retrieval (TIE-based) and pixel super-resolution (GS iteration) in our system to deliver super-resolved reconstructions with significantly reduced the number of raw measurements. The LED is used to achieve multi-wavelength illumination in a cost-effective manner, thus three intensity images with different wavelengths (red, green, blue) are captured. According to diffraction propagation theory, wavelength and propagation distance always appear in pairs, which means that a change in wavelength demonstrates an equivalent effect on the field propagation as a change in propagation distance [261, 262]. Thus, the captured images under multi-wavelength illuminations can be converted to multi-depth images, as shown in Fig. 27(a).
First, the phase retrieved by the TIE is performed to give an initial input for the GS iterative phase refinement. The green wavelength is set as the normalized wavelength because it has forward and backward propagation defocusing images. Then, the longitudinal intensity derivative ∂I/∂z can be estimated through the following equation:
$$ \frac{{\partial I}}{{\partial z}} = \frac{{{{\left({\Delta {z_{b}}} \right)}^{2}}\left({{I_{r}} - {I_{g}}} \right) - {{\left({\Delta {z_{r}}} \right)}^{2}}\left({{I_{b}} - {I_{g}}} \right)}}{{\Delta {z_{r}}\Delta {z_{b}}\left({\Delta {z_{b}} - \Delta {z_{r}}} \right)}} $$
(12)
where Δzr,b=zr,b−zg are equivalent propagation distances from the red/blue channel image to the green channel image. The form of wavelength and defocus distance of the same complex amplitude distribution can be expressed as \(\frac {z_{r}}{\lambda _{r}}=\frac {z_{g}}{\lambda _{g}}=\frac {z_{b}}{\lambda _{b}}\). Thus, once three diffraction patterns at different illumination wavelengths (red, green, blue) are captured, the effective defocus distances of the red and blue channel images are converted according to zr,b = zgλr,b/λg. It should be noted that there is a tiny distance between the red, green, blue units of a LED, the recorded intensity under different wavelength illumination appears translation deviation. In order to address this problem, Ir and Ib are registered with the reference image Ig via computing the upsampled cross correlation through the FFT method and locating its peak [263]. We use the FFT-based solver to restore the phase distribution of the specimen due to its simplicity, efficiency, and applicability for the case of non-uniform intensity distribution [185], and the phase can be presented as:
$$ \phi \left(\mathbf{x} \right)=-k_{g}{{\nabla }^{-2}}\nabla \cdot \left[ {{I}_{g}^{-1}}\left(\mathbf{x} \right)\nabla {{\nabla }^{-2}}\frac{\partial I\left(\mathbf{x} \right)}{\partial z} \right] $$
(13)
where kg indicates wave number (kg=2π/λg), Ig is the intensity with green wavelength, and x=(x,y) is the lateral 2D spatial coordinate. ∇−2 is the inverse Laplacian operator, denoting a simplified representation of the solution to the corresponding Poisson equation.
GS iterative phase retrieval algorithm starts from the retrieved phase of TIE, then the complex field is propagated back and forth between different planes, replacing the amplitude with the measurement amplitude \(\sqrt {{I}_{r,g,b}}\) until the whole process converges, as shown in Fig. 27(d). In order to avoid the convergence of the algorithm stagnates after early iterations or even get trapped into local optima, the incremental gradient method can be used for the iteration process to improve the stability and robustness of the reconstruction towards noise by adopting the adaptive step-size strategy [227]. Finally, the complex amplitude retrieved from iterative refinement is propagated back to the object plane to produce the image back into focus, leading to sharp in-focus intensity and phase with all out-of-focus blurring effectively removed. Similarly, the reconstructed phase can also be used to generate digital DIC results showing the phase gradient image of the specimen.
Experiment results
∙ high-throughput quantitative phase imaging
In order to verify the imaging resolution of our HTL-SCLM, a phase resolution target QPTTM with the height of 350nm was measured as a specimen, and the experimental results are shown in Fig. 28. Figure 28(a) shows the intensity pattern directly captured by the sensor. Though the object to sensor distance is relatively small, the defocus blurring can be easily observed in the magnified regions of the raw images. The precise defocus distance can be determined by the digital refocusing function of the software and applied for iterative reconstruction. After the TIE phase retrieval and several iterations phase refinement, the reconstructed quantitative phase shown in Fig. 28(d) can be obtained. Compared to the severe defocus artifacts in direct image acquisition, the reconstructed complex field leads to a sharp in-focus image with all defocus blurring effectively removed. The small group corresponding to the blue box in Fig. 28(d) is enlarged to display the distinguishable resolution element. As shown in Fig. 28(e), our imaging system correctly recovered the unit of Group 9 Element 2, which corresponds to the highest achievable resolution of 870nm (half-pitch resolution), exceeding the resolution-limited by the pixel size 1.67μm. The phase values along the red line are then extracted and plotted in Fig. 28(f) to quantify the lateral resolution, which gives a direct demonstration of the achievable resolution of our HTL-SCLM system. In addition, this high-resolution quantitative phase can be further used to generate phase gradient images of the specimen along an arbitrary shearing direction by digital DIC, as shown in Fig. 28(c).
Our HTL-SCLM achieves both a wide FOV and high imaging resolution, which provides an effective means for high-throughput imaging of label-free cells. To demonstrate the imaging performance of our HTL-SCLM system on biological specimens, we conducted an experiment on unlabeled HeLa cells. Figure 29(a) shows the reconstructed quantitative phase of HeLa cells with a full FOV of 29.8474 mm2. A region of interest [within the yellow dashed box in Fig. 29(a)] is selected and enlarged for quantitative analysis of the cells, as shown in Fig. 29(b). Our system allows users to extract the quantitative data of the specimen (along the red line) and drawing its profile, thus we can get quantitative data on the morphology of cells. Any area within the wide FOV can be magnified for further observation and detection of cells. As shown in Fig. 29(c1)-(c4) and 29(d1)-(d4), we extracted four sub-areas of reconstructed phase marked by four blue dashed boxes to display their enlarged versions and phase gradient images. It can be seen that the structure details of HeLa cells can be clearly observed. Due to the advantages of the quantitative phase in high contrast and high resolution, it also provides an effective data source for cell counting. From Fig. 29(a), a total of 8707 cells are presented in the full FOV.
∙ digital refocusing
In the absence of imaging optics, we can not acquire in-focus intensity images of the specimen in lensless microscope. To overcome this problem, an indispensable step for image reconstruction is to back propagate the acquired wavefront from sensor plane to object plane to get a sharp in-focus image. We employed the soybeans stem crosscut and dicot root to demonstrate the reconstruction results with and without digital refocusing, and the reconstructed intensities are displayed in Fig. 30. Compare Fig. 30(b1), (d1) and 30(b2), (d2), the reconstructed results with digital refocusing have clearer edges and details than those of without digital defocusing. This flexible ability to digitally adjust the ‘z’ parameter significantly extends depth-of-field (DOF) of our platform compared to other imaging systems where conventional microscope objectives are used.