Smart computational light microscopes (SCLMs) of smart computational imaging laboratory (SCILab)

Abstract

Computational microscopy, as a subfield of computational imaging, combines optical manipulation and image algorithmic reconstruction to recover multi-dimensional microscopic images or information of micro-objects. In recent years, the revolution in light-emitting diodes (LEDs), low-cost consumer image sensors, modern digital computers, and smartphones provide fertile opportunities for the rapid development of computational microscopy. Consequently, diverse forms of computational microscopy have been invented, including digital holographic microscopy (DHM), transport of intensity equation (TIE), differential phase contrast (DPC) microscopy, lens-free on-chip holography, and Fourier ptychographic microscopy (FPM). These computational microscopy techniques not only provide high-resolution, label-free, quantitative phase imaging capability but also decipher new and advanced biomedical research and industrial applications. Nevertheless, most computational microscopy techniques are still at an early stage of “proof of concept” or “proof of prototype” (based on commercially available microscope platforms). Translating those concepts to stand-alone optical instruments for practical use is an essential step for the promotion and adoption of computational microscopy by the wider bio-medicine, industry, and education community. In this paper, we present four smart computational light microscopes (SCLMs) developed by our laboratory, i.e., smart computational imaging laboratory (SCILab) of Nanjing University of Science and Technology (NJUST), China. These microscopes are empowered by advanced computational microscopy techniques, including digital holography, TIE, DPC, lensless holography, and FPM, which not only enables multi-modal contrast-enhanced observations for unstained specimens, but also can recover their three-dimensional profiles quantitatively. We introduce their basic principles, hardware configurations, reconstruction algorithms, and software design, quantify their imaging performance, and illustrate their typical applications for cell analysis, medical diagnosis, and microlens characterization.

Introduction

The optical microscope is one of the most significant inventions in the history of humankind that witnessed the fundamental revolution in biomedicine, chemistry, material science, electronics, and other various fields of scientific society. During its four centuries of development, it has evolved from a simple single lens device of van Leeuwenhoek’s day into a sophisticated instrument capable of observing micro-scale objects with high resolution ranging from the scale between molecular machinery (nanometer) and individual cells (micrometer), and now has become an essential visualization tool in forefront researches and industrial applications [1].

One of the greatest challenges in light microscopy is “contrast” [Fig. 1]. In conventional bright field microscopy, image contrast is generated based on scattering and absorbing of the incident light. However, most biological specimens are inherently weakly absorptive and scattering, thus are almost transparent in a conventional bright field microscope. A number of approaches for increasing contrast have been developed, among which staining and fluorescence are possibly the most far-reaching development since the inception of the light microscope [Fig. 1]. In the twentieth century, various fluorescence modalities, e.g., fluorescence microscopy [2, 3], confocal microscopy [4–6], total internal reflection fluorescence [7, 8], two/multi-photon microscopy [9, 10], and light-sheet microscopy [11, 12], were developed that allow to detect very small signals and to reveal structural and functional properties of a specimen with high specificity. In addition, based on the inherent optical sectioning capability of confocal microscopy and light-sheet microscopy, the fluorescence specimen can be imaged slice-by-slice to build a three-dimensional (3D) reconstruction of biomolecules inside the specimen [5, 13]. The advent of new fluorescent molecular probes and single-molecule detection schemes opened up the possibility to observe the behavior of individual biomolecules sequentially without ensemble averaging [14, 15]. The Abbe diffraction limit, thought unbreakable for over one hundred years, has been circumvented by ever more inventive far-field super-resolution fluorescence techniques, including stimulated emission depletion microscopy (STED) [16, 17], (fluorescence) photo-activated localization microscopy (PALM) [18, 19], stochastic optical reconstruction microscopy (STORM) [20, 21], and structured illumination microscopy (SIM) [22, 23]. Despite the immense success of fluorescence microscopy, this technique still suffers from several fundamental limitations. First, the exogenous contrast agents could induce negative effects on cellular functions like cell viability, proliferation, differentiation capacity. In addition, the photobleaching and phototoxicity of fluorescent agents prevent long-term observation of live cells [24]. Finally, fluorescence techniques are ill-suited for imaging non-fluorescent specimens or visualizing cellular components that cannot be fluorescently tagged (such as small intracellular lipids).

Fig. 1

The motivation and development trend of light microscopy for enhancing “contrast” – from qualitative to quantitative, from interferometric to non-interferometric

An alternative contrast mechanism is to exploit phase (rather than amplitude) changes in the incident light [Fig. 1]. In the 1930s, Zernike developed phase contrast microscopy (PCM), in which the image contrast is produced by shifting the relative phase between the unperturbed background light and the scattered light by a quarter of a wavelength [25]. PCM represents a major advance in endogenous (intrinsic) contrast imaging, as it significantly improves the contrast of transparent biological cells and tissues without staining or tagging [26–28]. An alternative to PCM is differential interference contrast (DIC) invented by Nomarski, which is based on the principle of polarization beam-splitting and shear interferometry [29]. It records the phase contrast proportional to the phase gradient of the specimen along the shear direction, giving the appearance of an illuminated relief map highlighting the optical density variation of a specimen [30–32]. Nowadays, PCM and DIC have developed into standard microscopic techniques for imaging unstained biological specimens with exceptional ability to reveal cell boundaries and subcellular organelles of unstained biological specimens that were nearly invisible before. However, the resultant phase-contrast image is an intensity distribution, in which the phase information is coupled nonlinearly, and the associated optical thickness, dry mass density, and refractive index (RI) of the specimen cannot be retrieved quantitatively.

“Computational light microscopy” is a new terminology that emerged in the past decade. It can be considered as a branch of “computational imaging” or understood as an extension and application of “computational imaging” technology in the field of light microscopy [33, 34]. Unlike conventional microscopy, in which an image is formed based on direct point-to-point intensity sampling, computational light microscopy provides a new way to overcome many physical limitations of microscopic imaging systems by indirect image reconstruction based on an ingenious integration of optical manipulation and image processing. Although the term terminology has not been around for long, the idea of “computational imaging” has been permeating the field of light microscopy for several decades, which arguably developed in parallel with “quantitative phase imaging” (QPI) techniques [35–38]. It is not difficult to understand because the essence of computational imaging is to characterize the optical field mathematically and then manipulate it digitally, while “phase” is one of the most important components of an optical field (strictly speaking, the monochromatic coherent optical field) which cannot be accessed directly.

The advent of computational light microscopy could be attributed to two revolutionary inventions: first, the invention of laser marked the first time that such highly controlled light became available as a physical medium in optical interferometry and holography to measure the tiny optical path difference induced by the specimens, opening up new possibilities for phase measurement [39]. Second, the invention of charged coupled device (CCD) cameras represented another important milestone: the compatibility of light with electricity, making computational storage, access, analysis, and transmission of captured data possible [40, 41]. Computer-based digital signal processing can be combined to automate the quantitative phase determination, eliminating the inconvenience associated with the manual, labor-intensive, time-consuming evaluation of the fringe patterns. In such a context, digital holographic microscopy (DHM) has emerged [42–45], which is essentially a digitalized version of optical holography applied to the field of light microscopy. By superimposing an additional coherent reference beam with the original object beam, the invisible phase information can be encoded into the fringe pattern, which is recorded as a digital image by a camera [46]. Fringe analysis, numerical propagation, and phase unwrapping algorithms are then performed to demodulate the quantitative phase information of the specimen [47–49]. Due to its unique advantages and flexibilities of digital recording and numerical reconstruction, DHM (including its variants) has made remarkable progress in the past two decades and now has become one of the most classic and well-established computational light microscopies and QPI approaches [46, 50–54].

In the early 21st century, computational light microscopy has developed very rapidly and made fruitful achievements, which closely followed the exponential growth in the power of digital computers with image processing capabilities. Various wavefront sensing and phase retrieval algorithms were introduced to the field of computational microscopy, leading to a series of non-interferometric QPI approaches [55–61]. They can be divided into three categories: wavefront sensing, iterative methods and deterministic methods. Wavefront sensing is a major branch of non-interferometric phase measurement techniques, such as Shack-Hartmann wavefront sensor [55, 62], pyramid wavefront sensor [56, 63, 64], and model-based wavefront sensor [58, 65]. They reconstructed the wavefront (phase) distribution based on the principle that the integrated gradient of the wavefront across the light-field modulator is proportional to the displacement of the centroid relative to the reference regular spacing spot array (ideal aberration-free case). The iterative methods often consider phase retrieval as an optimization problem. Gerchberg-Saxton (GS) is the most well-known iterative phase retrieval algorithms in which the phase distribution within an object is found that is consistent with both an intensity image of it and with its far-field diffraction pattern [66, 67]. Ptychography is another iterative phase retrieval approach, which adopts a limited-size illumination probe to move sequentially across an extended object, collecting the coherent diffraction patterns [68, 69]. With sufficient data redundancy, both methods reconstruct the wavefront information (including amplitude and phase) of the sample iteratively by alternating projection. In contrast to iterative methods, the deterministic phase retrieval method uses propagation to recover quantitative phase directly, in a non-iterative manner. Transport-of-intensity equation (TIE) is arguably the first phase retrieval technique proposed following this idea. It allows quantitative phase reconstruction from a series of intensity measurements taken at different propagation distances [70, 71]. Such data- or computation-intensive algorithms can be efficiently managed for processing in real-time by existing computer modern multicore and Graphic Processing Unit (GPU) processors. In particular, TIE has been increasingly adopted as a promising QPI tool due to its unique advantages over interferometric and iterative phase retrieval techniques: it is non-interferometric, non-iterative, phase-unwrapping-free, easy to implement, and more importantly, compatible with the built-in Köhler illumination of a conventional bright field microscope [59, 72]. During the same period, researchers engaged in interferometric QPI were increasingly aware of the importance of using low coherence illumination for improving the resolution and image quality, making considerable efforts to reduce the dependence of illumination coherence, as represented by spatial light interference microscopy (SLIM) [73, 74], white-light diffraction phase microscopy (wDPM) [75, 76], quadriwave lateral shearing interferometry (QWLSI) [77, 78], and τ interferometry [79].

More recently, a further remarkable extension of computational light microscopy was achieved by the applications of programmable light sources, e.g., wavelength-tunable lasers [80] and light-emitting diode (LED) array [81] and tunable optical elements [e.g., electrically tunable lens (ETL) [82], liquid crystal display (LCD) [83], and digital mirror device (DMD) [84]], complementing the digital processing with a matching optical modulation capability. Such devices not only provide additional flexibilities of active illumination and aperture control to realize multi-modal microscopy but result in a bunch of new computational light microscopy approaches. Differential phase contrast (DPC) microscopy is an alternative non-interferometric QPI approach based on asymmetric illumination [85, 86]. It recovers high-resolution quantitative phase from intensity differences captured with different illumination patterns created by programmable an LED array. Fourier ptychographic microscopy (FPM) is a computational microscopy technique for high space-bandwidth product [SBP, both high-resolution and large field-of-view (FOV)] imaging by combining the advantages of ptychography, synthetic aperture, and iterative phase retrieval [60, 87–89]. In FPM, the specimen is successively illuminated by plane waves from different angles with a programmable LED array, and the corresponding low-resolution intensity images are synthesized in the Fourier domain to reconstruct a high-resolution wide-field complex (both amplitude and phase) image. Light field microscopy is a motion-free single-shot 3D microscopy technique, which is achieved by inserting a microlens array in the intermediate image plane just before the camera sensor, allowing for recording a four-dimensional (4D) light field containing both the intensity and angular distribution of all rays [90–92]. During the reconstruction phase, ray-tracing techniques can be used to reconstruct synthetic photographs, estimate depth, and change focus or viewing perspectives. Programmable aperture microscopy is a multi-modal microscopy technique by integrating a programmable light modulation device [e.g., spatial light modulator (SLM) [93, 94] or programmable LCD [92]] into a conventional wide-field microscope. By selectively modulating the light distribution at the objective aperture plane, numerous imaging modalities, such as bright field imaging, dark field imaging, DPC imaging, QPI, multi-perspective imaging, and full-resolution light field imaging to be achieved and switched rapidly, without requiring specialized hardware and any physically moving parts. Optical diffraction tomographic (ODT) microscopy, which combines QPI with the computed tomography (CT) technique, is high-resolution “true 3D” phase imaging technique by recovering the volumetric 3D RI distribution of a weakly scattering specimen [95, 96]. By rotating the object or changing the illumination directions, a set of quantitative phase distributions is obtained to synthesize a 3D tomographic image in the Fourier space with the Fourier slice theorem [97–99] or Wolf’s Fourier diffraction theorem [100–102]. Structured illumination microscopy is a super-resolution microscopy technique based on Moiré effect, i.e., the specimen is excited by a known spatially structured pattern of light to generate interference patterns contain information beyond the diffraction limit [22, 23]. As the illumination pattern with high spatial frequency is close to the diffraction limit of the imaging system, the spatial information encoded by the Moiré fringes increases up to a factor of 2, resulting in up to a twofold gain in the resolution of the specimen.

In addition, the revolution in low-cost consumer image sensors, modern micro-processors, and smartphones allowing computational light microscopy imaging systems to be designed in a miniaturized, lens-free, and cost-effective manner [103–109]. Such low-cost, portable microscopes may offer a promising solution for telemedicine and point-of-care diagnostics in resource-limited environments. Finally, it has to been mentioned that computational light microscopy is currently undergoing a fundamental transition due to the advent and development of artificial intelligence (AI) technologies, especially deep learning techniques [110, 111]. Deep neural networks can “learn” to approximate solutions to inverse problems in computational imaging and microscopy from the captured data with unprecedented performance [112–114]. Within a few short years, deep-learning-based techniques, as evidenced by the ever-increasing and respectable number of publications, has been gaining increasing attention and demonstrating promising performance in various computational microscopy applications, including DHM [115–117], TIE phase microscopy [118, 119], DPC [120, 121], lens-free on-chip holography [112, 122], FPM [123–125], light-field microscopy [126–128], and SIM [129, 130].

To data, computational light microscopy has developed into an interdisciplinary field of fundamental optics, optical engineering, light microscopy, image processing, and biophotonics. With the further emergence and development of new imaging principles (e.g., quantum correlation imaging [131, 132] and imaging through scattering media [133–135]), new optical modulation devices (e.g., diffractive optical elements [136, 137], photonic integrated devices [138, 139], metamaterials [140, 141], metasurfaces [142–144], and metalenses [145–147]), and new computational mechanisms (e.g., AI [148, 149], quantum computing [150], and all-optical neural networks [151, 152]), computational light microscopy now is ready to enter an explosive phase of development. Nevertheless, we have to admit that in this field, these flamboyant concepts are not yet matched by mature technologies and engineering applications. Computational optical metrology is not able to (at least not yet) shaken the position of traditional microscopic imaging techniques, failing to trigger expected technological revolutions. One of the main reasons for this paradox is that most computational microscopy techniques are still at an early stage of “proof of concept” or “proof of prototype” (based on commercially available microscope platform). Translating those concepts to stand-alone optical instruments for practical use is an essential step for the promotion and adoption of computational microscopy by the wider bio-medicine, industry, and education community.

Please note that this paper may be somewhat different from others in this field. The authors of this article come from the Smart Computational Imaging Laboratory (SCILab) of Nanjing University of Science and Technology (NJUST), China. As the name suggests, our laboratory is devoted to exploring new science and technologies in the field of computational imaging, especially computational light microscopy. More importantly, we have also been working on the engineering and instrumentation of related technologies. Through nearly a decade of efforts, we developed four pioneering microscopes based on the principle of “computational imaging”, which we call “smart computational light microscopes (SCLMs)”. These SCLMs are empowered by advanced computational microscopy techniques, including DHM, TIE, DPC, lensless holography, and FPM, which not only enable multi-modal observation for unstained specimens, but also can recover their 3D shapes quantitatively. We introduce their basic principles, hardware configurations, reconstruction algorithms, and software design, quantify their imaging performance, and illustrate their typical applications. Although some of these instruments are not yet considered a completely commercial product, we believe that a detailed description of basic principles and technical details of these instruments will certainly bring some “distinct” knowledge and “unique” information to this field, which may provide useful insights into the future development of computational light microscopy, especially in the perspective of engineering applications.

Are you ready for the remaining content? Just a friendly reminder that this article is a bit long. The remainder content is organized as follows. In “Computational light microscopy: concept” section, we start with the concepts related to computational microscopy, illustrating the basic principles of computational imaging, including forward image formation model, optical manipulation mechanism, and inverse reconstruction algorithm. In “Computational light microscopy: paradigms” section, we introduce four SCLMs we have developed in four paradigms, and their working principles, system configurations, imaging algorithms, and typical applications are presented in detail. Then, some additional information with respect to the “engineering”, including software, hardware, and optical designs of SCLMs, are provided in “Computational light microscope: software development and hardware implementation” section. Finally, the challenging problems, as well as future directions of computational light microscopy, are outlined in “Conclusion and discussion” section.

Computational light microscopy: concept

A typical light microscope consists of three main components: light source, imaging system, and light detector. In conventional microscope, illumination emitted from the light source interacts with the specimen, producing absorption, scattering, or fluorescence. The resulting light from the object is collected by the optical imaging system and focused onto the image sensor, which is finally discretely sampled by the image sensor and recorded as a magnified digital image [Fig. 2]. Limited by the intensity-only detection, single-view observation, two-dimensional (2D) planar recording, spectral integration, diffraction limits, optical aberrations, etc., this “point-to-point” direct imaging mechanism generally results in a severe loss of high-dimensional information about the object. In addition, traditional microscopes are often thought of as an organic combination of precision optics and mechanics. However, increasingly complex, bulky, and expensive optical lenses and mechanical systems have created the stereotype of the microscope as a complex, clunky, difficult-to-maintain behemoth, limiting its use in many scenarios where miniaturized and portable devices are required.

Fig. 2

Comparison of basic imaging concepts of conventional light microscopy and computational light microscopy

Unlike conventional microscopy, computational light microscopy is based on an indirect imaging scheme: “optical modulation, then image acquisition, and finally information demodulation” [Fig. 2]. It provides a new way to overcome many physical limitations of traditional light microscopy by ingeniously integrating optical modulation with image processing. The hardware (illumination, optics, photodetector) and the image processing algorithm are jointly designed and optimized, creating a kind of ‘hybrid optical-digital microscopy mechanism’ [Fig. 2]. More specifically, in computational light microscopy, there are three important aspects that need to be carefully considered:

(1) Forward image formation model

In computational light microscopy, a forward mathematical model describing the image formation process is established with all its parameters, which is essential for the subsequent image processing and analysis. The general physical imaging problem is electromagnetic in its nature. In 1873, Abbe derived the first comprehensive theoretical description of optical image formation [153]. The image intensity is interpreted by the interference of the coherent waves, which are formed by emission or diffraction at the object, and transmitted and transformed by the optical imaging system. Since the time of Abbe, the complexity of optical systems has increased considerably, and the level of knowledge about them has improved a great deal. In computational microscopy systems, the use of laser or LED light sources makes it necessary to consider the questions of coherence [154, 155]. Imaging at high numerical apertures, or the use of crystal materials, generates polarization effects [156]. Digital image detection with pixelated image sensors involves the consideration of sampled images [157, 158]. The physical model of microscopy imaging aims, whenever possible, to provide a linear system description. In the most convenient description, the effective light source is described by incoherent source points, and the object is illuminated by an assembly of incoherent plane waves [159]. Diffraction is often taken into account by the thin object approximation (2D specimen) [60], weak object approximation [160], slowly-varying approximation [85]. In Table 1, we summarize the physical analysis model involved in the image formation and the mathematical model considered for the computational microscopy. These theoretical descriptions have been developed or adapted from other disciplines of optics.

Table 1 Standard physical analysis model of computational light microscopy

(2) Optical modulation mechanism

Optical modulation is the core step in computational light microscopy, which is also the key element that distinguishes computational imaging from traditional digital image processing. According to the system component being modulated, optical modulation can be further divided into four types, i.e., illumination modulation, imaging system modulation, specimen modulation, and detector modulation. The common methods of optical modulation are also listed in the bottom row of Table 1.

With these optical modulation approaches in hand, the crux of problems of this stage lies in how to control and manipulate the flexible parameters in the forward image formation model so that the desired specimen information can be optimally encoded into and decoded from the optical field with the highest possible precision and accuracy. In addition, regarding different types of applications, specific imaging indicators, such as spatial resolution, temporal resolution, FOV, measurement accuracy, and signal-to-noise ratio (SNR), are also essential factors to be considered. The relation between specific imaging indicators and optical modulation approaches should be analyzed by controlled variable method (i.e., change only one adjustable parameter of the illumination, imaging system, and detector), and explore the corresponding influence). And finally, the optical modulation scheme should be carefully designed so that the key imaging indicators for a specific application are fully optimized. For example, the spatial resolution (mainly including lateral/axial resolution and de-pixelation) is associated with the illumination wavelength, illumination angle, objective numerical aperture (NA), and the pixel size of the detector. For phase retrieval, illumination wavelength [164, 182], angle [85, 86], aperture modulation [82, 92], sensor defocus [176] can all produce phase contrast that allows for non-interferometric quantitative phase recovery. Therefore, if one wants to design a quantitative phase microscopy system for high-resolution single-cell-level label-free imaging, all these factors should be considered comprehensively to develop an appropriate optical modulation scheme.

(3) Inverse reconstruction algorithm

Finally, the forward image formation model is inverted (by solving the corresponding inverse problem) to reconstruct the specimen image and, importantly, additional high dimensional information, such as phase, spectrum, polarization, optical field, coherence, RI, and 3D profile, which cannot be directly acquired using traditional methods. Such an inverse problem is usually solved by deterministic [70, 71] and iterative methods [66, 67, 183].

∙ For inverse problems that can be linearized by invoking certain approximations, they can be solved deterministically by the matrix inversion, deconvolution, and solving partial differential equations, etc. For example, the phase retrieval problem under the paraxial approximation can be linearized by TIE [70, 71]. It can be conveniently solved by multigrid or fast-Fourier transform (FFT)/discrete cosine transform (DCT) based Poisson solvers [184–186]. Under the 1st Born or Rytvo approximation, the inverse scattering problems in ODT can also be solved linearly based on Fourier diffraction theorem with simple Fourier domain filling [96, 98, 100, 187] or 3D deconvolution algorithm [188–190].

∙ For inverse problems that cannot be linearized (e.g., non-paraxial condition, generally phase retrieval problems including ptychography and Fourier ptychography), iterative solutions based on alternating projection and gradient descent are often employed [60, 68, 191, 192]. In general, the solution obtained by linearizable inverse problems based on more restrictive assumptions can be used as the initial value for solving the corresponding nonlinear version under more general conditions, allowing faster convergence and the prospect of extrication from local minimums [193, 194]. Advanced numerical reconstruction algorithms like compressive sensing and deep learning are well suited to solve such nonlinear optimization problems by introducing sparse representation [195–197] and Low-Rank Regularization [198, 199], or directly establishes a “pseudo nonlinear forward model” from the raw measurement to the desired specimen information in a data-driven manner [200]. $\mathscr {l1}$

Computational light microscopy: paradigms

In this section, we present four SCLMs developed in our lab that not only exemplify the basic ideas of computational light microscopy presented in the previous section but also demonstrate their potential impact on the function and performance of microscopic instruments and the associated applications. They are shown in Fig. 3. These SCLMs are empowered by advanced computational microscopy techniques, including DHM, TIE, DPC, FPM, and lensless holography. Through proper optimization of optical modulation and information processing mechanisms, different imaging functions and performance suitable for different application scenarios can be achieved. The general design concepts, imaging functions, performance indexes, and intended biomedical applications are summarized in Fig. 4.

Fig. 3

Four SCLMs developed by our SCILab based on the principle of “computational imaging”. From left to right are MQP-SCLM, DH-SCLM, MMC-SCLM, and HTL-SCLM

Fig. 4

Design concepts, imaging functions, performance indexes, and intended biomedical applications of four SCLMs

Digital holographic smart computational light microscope (DH-SCLM): DH-SCLM [Fig. 4(a)] is based on a classic computational microscopy approach — digital holography. By replacing the light source of a conventional microscope with a coherent laser and introducing an additional reference beam path, the invisible phase information can be modulated into the visible interference fringes, and then quantitatively reconstructed based on Fourier fringe analysis techniques. In DH-SCLM, by rational design of the optical system and associated reconstruction algorithms, we have overcome several inherent problems in holographic imaging, such as speckle noise, unreliability of phase unwrapping, phase aberration, and sensitivity to environmental perturbation. The resulting system has many unique features for micro-optics metrology and cell biology, including nanometric optical path length resolution, full-field measurement, real-time QPI, digital refocusing, automatic denoising, and aberration compensation. More details about DH-SCLM will be presented in “Paradigm 1: digital holographic smart computational light microscope (DH-SCLM)” section.

Multi-contrast quantitative phase smart computational light microscope (MQP-SCLM): MQP-SCLM [Fig. 4(b)] is a powerful multi-contrast computational phase microscope by combining illumination modulation (programmable LED illumination) with imaging optics coding (an ETL). The RGB programmable LED array gives the microscope additional flexibilities for illumination direction modulation and wavelength tuning. In addition, the ETL makes fast, telecentric, motion-free focal plane adjustment of the microscope system possible. The combination of two tunable optical devices allows MQP-SCLM to integrate seven different imaging modalities within a single microscopic system, including bright field, dark field, light field, Rheinberg optical staining, DPC, TIE quantitative phase microscopy, and FPM. In addition, the use of partially coherent LED illumination endows MQP-SCLM with higher phase imaging resolution (up to incoherent diffraction limit) and speckle-free image quality, which is extremely valuable for microscopic imaging of unlabeled biological cells and tissues. More details about DH-SCLM will be presented in “Paradigm 2: multi-contrast quantitative phase smart computational light microscope (MQP-SCLM)” section.

Miniaturized multi-contrast smart computational light microscope (MMC-SCLM): MMC-SCLM [Fig. 4(c)] is a miniaturized version of MQP-SCLM by removing the tunable lens and tailoring the optical system. The physical length of the optical imaging system is significantly shrunk by using a customized and non-standardized microscope objective lens, reducing the size of the entire microscope to 14 ×16.5×20cm³, making it compact, portable, and easily be fitted into a cell culture chamber or apply to telemedicine. In addition, the RGB programmable LED array also endow MMC-SCLM with the multi-contrast imaging ability of bright field, dark field, Rheinberg optical staining, DPC quantitative phase microscopy. More details about MMC-SCLM will be presented in “Paradigm 3: miniaturized multi-contrast smart computational light microscope (MMC-SCLM)” section.

High-throughput lensless smart computational light microscope (HTL-SCLM): HTL-SCLM [Fig. 4(d)] is a minimalist microscope by discarding all lenses and other bulky optical components of a conventional microscopic system. HTL-SCLM is composed of just a color LED and an image sensor so that it is small enough to fit in the palm of the hand. Based on multi-wavelength phase retrieval, this microscope offers high-throughput quantitative phase images with a pixel-super-resolved half-pitch imaging resolution of 870nm across a wide FOV of 29.8474 mm². Such a powerful and miniaturized imaging device may offer a cost-effective tool for telemedicine applications or point-of-care diagnostics in locations where advanced laboratory facilities are unavailable. More details about HTL-SCLM will be presented in “Paradigm 4: high-throughput lensless smart computational light microscope (HTL-SCLM)” section.

We have developed specialized digital imaging microscope software for each system to unleash all features of our SCLMs. They enable users to acquire, process, and analyze images in multiple dimensions and over various timepoints. The optimized algorithm implementation and non-destructive image handling specially developed for SCLMs guarantee reproducible results for different application experiments. In addition, powerful analysis tools are developed to dynamically extract data from imagery for precise experiment results. The design architecture of the software will be introduced in “Computational light microscope: software development and hardware implementation” section. In each paradigm, we will describe the detailed features of the supporting software.

Paradigm 1: digital holographic smart computational light microscope (DH-SCLM)

Our first microscope is based on a very classical computational microscope approach, and it is called DH-SCLM. It is constructed as an off-axis interference system in which removing the zero-order and twin-image is generally trivial [42, 43, 51, 53, 201–204]. We optimally designed the optical system and associated reconstruction algorithms to overcome the inherent problems in holographic imaging such as speckle noise, reliability unwrapping, phase aberration, and sensitivity to environmental perturbation. In addition, several auxiliary functions, such as digital DIC (generate digital DIC image from recovered quantitative phase image), 2D quantitative phase map/3D pseudo phase profile display, morphological measurements, are integrated into the DH-SCLM system to provide powerful microscopic analysis tools for biological applications. In this subsection, we present the system structure, software design, and implementing algorithms of DH-SCLM. Diversified biological application experiments demonstrated that our DH-SCLM achieves a lateral resolution of 409nm (half-pitch imaging resolution with 40 × objective lens) and an axial resolution of 10nm for quantitative phase measurement.

Optical configuration and software

Figure 5(a) shows the physical diagram of our DH-SCLM system, the hardware configuration of our DH-SCLM is based on an off-axis interferometry system. As shown in Fig. 5(b), we adopt a laser with a central wavelength of 532nm, a spectral width of 1nm as the coherent illumination source. The emitted light is divided into two beams by the splitter prism to generate object light wave and reference light wave. In order to separate the twin images and the zero diffraction order automatically in the Fourier domain, these two waves interfere in the camera plane at an angle of 2.45 ° apart. Finally, a series of interference fringes are captured by a camera sensor (The Imaging Source, DMK 23U274, 1600 ×1200, with the pixel size of 4.4 μm), as shown in Fig. 5(c1). For different imaging applications, our DH-SCLM is equipped four objective lenses with magnification of 4 ×(0.13 NA), 10 ×(0.25 NA), 20 ×(0.45 NA), and 40 ×(0.65 NA), providing multi-scale imaging capabilities. The detailed performance indexes can be found in Fig. 4(a).

Fig. 5

Schematic diagram of our DH-SCLM. (a) The hardware configuration of DH-SCLM. (b) Interference optical path. (c1) Recorded interference hologram. (c2) Recovered quantitative phase distribution

The supporting processing software is designed for DH-SCLM to perform numerical reconstruction of quantitative phase and multidimensional display. We defined four functional modules containing imaging setting, camera control, view display, and data post-analysis to implement flexible digital operations. The imaging setting module is used to determine the algorithm parameters for the phase reconstruction. First, the currently used objective lens should be selected to update the objective parameters of the phase reconstruction algorithm. Next, when the specimen cannot be focused by manual adjustment, digital refocusing is performed to obtain a high-precision in-focus image of the specimen by numerical propagation. In the phase reconstruction process, three phase unwrapping algorithms, i.e., DCT unwrapping [48], branch cut unwrapping [49], and quality-guided unwrapping [205], are provided for different observation specimens. In addition, this module also provides error correction and compensation capabilities. Spectrum centering, sub-pixel shift, and principal component analysis (PCA) algorithms, are flexibly worked to obtain highly robust reconstruction phases. The camera control module integrates the software development kit (SDK) for the camera, which allows adjustable settings of the brightness, contrast, exposure time, gain, and other parameters to meet the imaging requirements. View display module contains four view windows for presenting the results of the software running. We provide digital DIC display, 2D quantitative phase map, and 3D pseudo phase profile display to present the reconstruction results in high contrast. Data post-analysis module is designed to highlight or numerically analyze target regions or cells of interest for observers. On the one hand, the display contrast can be processed by histogram filtering to highlight the desired information of the specimen. On the other hand, the quantitative profile measurement can be performed for any cell to extract its quantitative data, establishing an accurate database for biological analysis. All implementation algorithms can be executed in real time so that the results will be presented on the display window without delay.

Principle and algorithms

∙Principle of DH-SCLM

DH-SCLM is an off-axis interferometry system in which separating the zero-order and ±1-order spectrums are generally trivial in the Fourier domain and one or the other can be removed through a bandpass filter. Assuming a reference wave with a relative angular shift θ, the reference light wave and object light wave interfere with each other, generating a hologram with the intensity distribution of:

$$ I_{h}(x,y)=|O|^{2}+|R|^{2}+|R|Oexp(-iksin\theta x)+|R|O^{*}exp(iksin\theta x) $$

(1)

where R(x,y) and O(x,y) denote the reference and object waves, respectively, ∗ denotes the complex conjugation. Taking Fourier transform on the hologram, the influence of the two phase factors exp(±iksinθx) can be interpreted as the translations of the spatial frequencies associated with the ±1-order spectrums from the center of the Fourier plane to higher frequencies domain. The spatial frequencies of the 0-order or direct-current-term (DC-term) are located in the center of the Fourier plane, producing the background distribution. While the spatial frequencies of the interference terms vary at different carrier frequencies, which are located symmetrically with respect to the center of the Fourier plane: −ksinθx for the +1-order image and ksinθx for -1-order image. This suggests that the +1-order or -1-order spectrum can be extracted from the hologram spectrum by selecting the appropriate bandpass filter, as shown in Step2 of Fig. 6. By performing inverse Fourier transform for the extracted subspectrum, the wrapped phase distribution of the specimen ϕ_w(x,y) can be obtained:

$$ \phi_{w}(x,y) =angle\{\mathscr{F}^{-1}\{\mathscr{F}\{I_{h}(x,y)\}\times H(u,v)\}\} $$

(2)

Fig. 6

Technical roadmap of DH-SCLM

where $\mathscr {F}$ and $\mathscr {F}^{-1}$ denote the Fourier transform and its inverse transform. The H(u,v) is the filter function in Fourier domain, which should eliminate all spatial frequencies except those of the +1-order interference term wanted. At this time, the retrieved phase distribution ϕ_w(x,y) is wrapped to the values ranging between −π and π.

Phase unwrapping must be carried out in order to remove the discontinuities from their wrapped values and to obtain an estimate of the true continuous phase map. We achieve three phase unwrapping algorithms in our DH-SCLM, i.e., DCT unwrapping [48], branch cut unwrapping [49], and quality-guided unwrapping [205], as shown in Step5 of Fig. 6. They provide different imaging capabilities in algorithm complexity, noise robustness, imaging efficiency, etc. DCT unwrapping is appropriate for dynamic specimens due to its fewer computational time costs. The quality-guided unwrapping offers significantly improved imaging accuracy, which allows it to be used for high precision and high accuracy 3D quantitative phase reconstruction. The branch cut unwrapping method achieves a balance between imaging accuracy and resolution, thus it is suitable for general specimens.

After we obtain the quantitative phase of the specimen, it can be post-processed to form a digital DIC image, which displays the phase gradient of the specimen along an arbitrary shear direction:

$$ I_{DIC}(x,y)=B\left(1+cos \alpha \frac{d\phi}{dx}+sin \alpha \frac{d\phi}{dy}\right) $$

(3)

here B is a constant to generate a uniform background, α denotes the shear direction of the phase gradient. $\frac {d\phi }{dx}$ and $\frac {d\phi }{dy}$ are phase derivatives along x and y directions. In conventional DIC microscopes, the shear direction is often difficult to adjust, resulting in limited specimen phase gradient information. In our DH-SCLM, α can be digitally selected, allowing the system to be adjusted more flexibly and provide richer visualized information for the specimen.

∙ error correction algorithms

Due to the coherent nature of the laser sources, digital holograms of DHM are corrupted by a mixture of coherent speckle and incoherent noise, which can severely degrade the reconstruction quality [206–211]. Moreover, DHM is sensitive to mechanical vibrations and air fluctuations that typically affect any interferometric systems. These challenges restricted the wide usage of DHM in biological applications. In our DH-SCLM, we carefully addressed the inherent problems of DHM in speckle noise, system aberration, defocus error, and susceptibility by targeted compensation and correction algorithms on a case-by-case basis (which are displayed in Step3 and Step4 of Fig. 6). In the following presentation of this subsection, the compensation and correction algorithms used in our DH-SCLM will be described in detail.

In the coherent imaging systems, speckle noise results from the coherent nature of the light source, which may deteriorate the quality of the images/fringe patterns [212]. In our DH-SCLM, the speckle noise is suppressed by means of weighted average superposition processing. Specifically, the captured holograms over a very small time interval are superimposed and averaged with given weights. And the weights can be adjusted for different imaging speed requirements to optimize the degree of smoothing. In this method, the resolution of the processed hologram is preserved because any pixel is combined with the homologous pixels in the acquisition stack. In addition, an effective high-speed phase reconstruction scheme obtains a single frame reconstructed phase by averaging each acquired image with the adjacent images, then the final reconstruction rate remains consistent with the limit-frame rate of the camera.

The system aberrations of DHM generally include off-axis tilt aberrations and quadratic wavefront distortion, and they are compensated by different numerical approaches in our DH-SCLM. Off-axis tilt aberration roots in the incorrect selection of +1-order spectrum center, which introduces a linear phase factor in the virtual image terms. Thus, the spatial frequencies associated with the ±1-order spectrums are translated from the center of the Fourier plane to the higher frequencies domain. In our DH-SCLM, the off-axis tilt aberration is corrected by a simple spectrum centering procedure. Its realization begins with the 2D surface fitting for the extracted +1-order spectrum, whose maximum point coordinates are searched and locked as +1-order spectrum zero frequency. Then, the sub-pixel displacement of the spectrum is realized by multiplying the phase shift factor in the spatial domain, achieving the high accuracy of spectrum center alignment. Quadratic wavefront distortion is induced by the mismatch of interference optical paths because the objective lens is involved only in the object light wave, and it results in concentric circular patterns (introduced by the quadratic phase factor) in the reconstructed phase. PCA algorithm is a simple and effective numerical phase aberration compensation method to correct for both quadratic wavefront distortion and tilt aberration [213, 214]. It decomposes the phase map into a set of values of uncorrelated variables called principal components and then extracts the aberration terms from the first principal component. Finally, the estimated aberration is removed by subtracting the estimated aberrations from the reconstructed phase, only leaving the phase of the specimen.

Our DH-SCLM corrects the defocus error by seeking the precise defocus distance based on numerical propagation and the Laplace operator. As shown in Step4 of Fig. 6, the complex amplitude distribution of the extracted +1 spectrum is propagated to acquire the complex field distribution with defocus distance Δz. Then, the overall image clarity of the phase distribution at different defocus distances can be evaluated by the variance of the edge image extracted by the second-order Laplace operator. In order to locate the focus plane quickly, we adopt the dichotomy method to speed up the search. The post-processing digital focusing process is integrated into the software’s defocus correction module, and only one button is needed to get the precise defocus distance.

Experiments results

∙ error correction

In order to demonstrate the effectiveness of error correction of our DH-SCLM, we conducted several experiments on different specimens. Figure 7 displays the imaging results of pollen cells to compare the reconstructed phase without and with correcting off-axis tilt distortion and quadratic wavefront distortion. The recorded interference hologram can be seen in Fig. 7(a), and its spectrum is shown in Fig. 7(b). The incorrect centering of the zero frequency usually extracts shifted +1-order spectrum, as shown in the solid white box in Fig. 7(c). As a result, an extra low-frequency phase is attached in the phase of the specimen, overwhelming the phase information of the specimen [Fig. 7(d)]. In contrast, our correction method achieves the high-precision spectrum centering so that all phase information of the specimen is clearly visible in the aberration-corrected image [Fig. 7(e)]. Three areas of interest are extracted and amplified to compare the imaging performance without and with aberration correction. From Fig. 7(f1)-(f3) [without aberration correction] and Fig. 7(g1)-(g3) [with aberration correction], the specimen information is correctly recovered with aberration correction, providing a quantitative phase distribution that can be used for analysis.

Fig. 7

Comparison of reconstructed phase without and with correcting the off-axis tilt aberration and quadratic wavefront aberration. (a) Recorded hologram. (b) Spectrum of the hologram. (c) Extracted +1-order spectrum. (d) Reconstructed phase without correcting the off-axis tilt aberration and quadratic wavefront aberration. (e) Reconstructed phase with correcting the off-axis tilt aberration and quadratic wavefront aberration. (f1)-(f3) The selected subregions in (d). (g1)-(g3) The selected subregions in (e)

HeLa cells were then imaged as specimens to compare the imaging results without and with defocus correction to verify the performance of the defocus correction. Figure 8(a) shows the reconstructed phase without defocus correction, which exhibits the serious detail loss of the reconstructed phase. The defocus distance obtained from the numerical propagation was applied for compensation of the defocus error, and the results shown in Fig. 8(b) can be obtained. The defocus artifacts are effectively suppressed so that detailed information of the subcellular structure is extracted correctly. We selected two enlarged areas of interest from Fig. 8(a) and (b), and then enlarged them in Fig. 8(c1), (d1), (e1), and (f1). It can be seen that in the defocus-uncorrected image, the phase artifacts distort the contour and internal structure of the specimen. Defocus correction compensates for the loss of resolution, provides clearer image detail information, and the vesicles and profile of the cell are clearly distinguished. Also, these reconstructed phases generate the phase gradient images of the specimen by digital DIC, and they are shown in Fig. 8(c2), (d2), (e2), and (f2).

Fig. 8

Comparison of reconstructed phase without and with correcting defocus aberration. (a) Reconstructed phase without correcting defocus aberration. (b) Reconstructed phase with correcting defocus aberration. (c1), (e1) Selected enlarged regions of interest of (a) [without correcting defocus aberration]. (c2), (e2) Digital DIC results corresponding to quantitative phase in (c1), (e1) [without correcting defocus aberration]. (d1), (f1) Selected enlarged regions of interest of (b) [with correcting defocus aberration]. (d2), (f2) Digital DIC results corresponding to quantitative phase in (d1), (f1) [with correcting defocus aberration]

∙ quantitative phase imaging and analysis

We demonstrated the imaging resolution of the DH-SCLM by the experiment on standard phase resolution targets (QPT^TM, Benchmark Technologies Corporation, USA, RI n=1.52). This target contains seven sub-targets of varying heights (from 50∼350nm in 50nm increments), each of which is a USAF resolution target consisting of several steps with equal height. In this experiment, the structure with a height of 100nm was imaged by an objective lens with 10 ×, 0.25NA, and the reconstructed phase is shown in Fig. 9. Figure 9(a) shows the interference hologram captured by the camera sensor. The clear interference fringes can be observed from the magnified hologram shown in Fig. 9(d), which ensures high accuracy and SNR for the measured phase. According to the NA of the objective lens and illumination wavelength, the appropriate spectrum bandwidth filter is automatically selected to recover the quantitative phase of the specimen, and the result shown in Fig. 9(b) directly appeared on the software interface.

Fig. 9

Experiment for imaging resolution verification by standard phase resolution targets [Quantitative Phase Microscopy Target (QPT^TM), Benchmark Technologies Corporation, USA]. (a) The captured hologram. (b) Reconstructed quantitative phase. (c) The phase gradient image generated from reconstructed quantitative phase by digital DIC method. (d) The enlarged sub-region of interest of the reconstructed quantitative phase in (a). (e) The enlarged sub-region of interest of the reconstructed quantitative phase in (b). (f) The quantitative profile of quantitative phase along the red line in (e)

Interferometric holography provides the imaging resolution of the diffraction limit of the objective lens, corresponding to the half-pitch resolution of 1.064 μm (10 × objective lens, 0.25NA). We zoomed in on the high-resolution group in Fig. 9(b) to demonstrate the highest achievable resolution. It can be seen that our DH-SCLM system recovers the quantitative phase correctly and reaches its theoretical maximum resolution of Group 8, element 6 (half-pitch resolution of 1.096 μm). We then extracted the phase profile line and plotted the curve to quantitatively analyze the image resolution. As shown in Fig. 9(f), all the unit details of Group 8 are clearly resolved. Furthermore, the reconstructed phase in Fig. 9(b) can be used to achieve DIC imaging and obtain the phase gradient results of the specimen, as shown in Fig. 9(c). Compared with traditional DIC imaging, we can flexibly adjust the direction and degree of the phase gradient to achieve more comprehensive observation and analysis.

Morphological measurement and analysis is an important approach to early diagnosis of cancer because cancer cells often have obvious structural and morphological changes in the early stage [215–217]. Our DH-SCLM system can achieve high-precision, high-speed QPI, providing high-resolution quantitative morphological information of cancer cells. As shown in Fig. 10, the experiments on a static HeLa cell slide and living HeLa cells cultured in vitro were conducted in our DH-SCLM system. We used an objective lens with 20 ×, 0.4NA to imaging the specimen, and the corresponding highest theoretical half-pitch resolution is 665nm. As shown in Fig. 10(b), our system restored high-resolution detail of the HeLa cell correctly, which can also tbe used to generate the high-quality DIC result [Fig. 10(c)]. The microstructures of cells of Area1 and Area2 were extracted to analyze their subcellular morphology, as shown in Fig. 10(d1)-(d2), and 10(e1)-(e2). The complete morphology and subcellular structure of HeLa cells are clearly displayed, providing abundant data for quantitative research applications. In addition, the quantitative phase distribution can be further shown as a 3D pseudo-color rendering image in Fig. 10(f) so that the different phase distributions can be displayed more clearly. Then, we demonstrated the imaging capability of our DH-SCLM to cell growth in the culture dish. The experiment was carried out on HeLa cells cultured on Petri dishes in 10% fetal bovine serum and 90% Dulbecco’s modified eagle medium. Figure 10(g) displays the time-lapse results of the HeLa cell division process. These high-resolution phase images clearly reveal the cell morphological changes during different mitosis phases.

Fig. 10

Quantitative phase of HeLa (human cervical cancer) cells. (a) A recorded hologram. (b) Quantitative phase reconstructed by DH-SCLM. (c) Phase gradient images generated from quantitative phase by digital DIC method. (d1), (e1) The sub-regions of interest of quantitative phase in (b). (d2), (e2) The sub-regions of interest of phase gradient images in (c). (f) 3D pseudo-color rendering image of the reconstructed quantitative phase. (g) The time-lapse video of dynamic division of HeLa cells in vitro

Paradigm 2: multi-contrast quantitative phase smart computational light microscope (MQP-SCLM)

Non-interferometric QPI is a new trend of the phase imaging to overcome the limitation of interferometric QPI approaches, providing the unique advantages of high quality image, high resolution, high robustness towards variant disturbance, and not needing phase unwrapping. Among the various non-interferometric QPI methods, the deterministic QPI methods often require only a few intensity images to achieve phase retrieval, which makes them superior for practical label-free imaging applications. We exhibited a revolutionary multi-contrast non-interferometric quantitative phase microscope containing seven imaging approaches for the first time, which was called MQP-SCLM. Instead of the traditional illumination source, a programmable LED array source is employed to flexibly switch the illumination patterns of bright field, dark field, light field, Reinberg optical staining, DPC, QPI (TIE-based), and FPM in a fixed microscope configuration. All imaging algorithms are integrated with a user interface by corresponding unified hardware platform and software framework. Furthermore, we step forward the widely spread biomedical applications with reducing significant expense and complexity of the related optical hardware and explore the potentially possible instrumentation techniques for MQP-SCLM. The investigations of various stained and unstained biological specimens using different types of objectives are presented in “Experiments” subsection, and results show the new possibility of widespread adoption of MQP-SCLM in the morphology study of cellular processes and the biomedical community.

Optical configuration and software

The physical diagram of MQP-SCLM is shown in Fig. 11(a), and Fig. 11(b)-(c) illustrate the photograph of key parts of the system hardware. Our system is equipped with an electronic platform whose displacement in all three directions x,y,z can be precisely adjusted by an external control handle [Fig. 11(b)] or software control. As depicted in Fig. 11(d), the optical path of MQP-SCLM is a standard 4-f optical system. The existing condenser-based illumination diagram is replaced with a programmable LED array, which is placed 54mm away from the specimen, and the center of the LED array is aligned with the optical axis of the microscope. The LED illumination array is a 64 ×64 pixels commercially available product with the pixel size of 2mm. Thus, this illumination module can provide an illumination NA up to 0.78 and sufficient illumination uniformity. The RGB LED array has central illumination wavelengths of 623nm, 530nm, and 470nm, and the brightness of a single LED is greater than 2000 cd/m². The programmable LED array is controlled by a custom-designed field-programmable-gate-array-based (FPGA-based) controller (Altera EP4CE10E22C8N), providing a high LED illumination pattern refresh rate (>1kHz) and high display color-scale (8-bit grayscale for RGB channels). High refresh rate refresh frequency can effectively improve the uniformity of the bright field image and eliminate the image blurring caused by the camera global exposure or the light and dark stripes caused by the rolling exposure. The objective lens with several different NAs (4 ×0.1NA, 10 ×0.25NA, 20 ×0.4NA, 40 ×0.65NA, 60 ×0.8NA) and tube lens with different magnifications are equipped in MQP-SCLM for most biomedical observation and measurement applications. Moreover, the objective indicator is used to monitor the state of the objective, and all system parameters will be matched by the circuit board according to the last used objective lens. Unlike conventional systems with mechanical defocus adjustment, we adopt an ETL [as shown in Fig. 11(c)], which can be controlled by a hardware circuit board to realize the automatic control of defocus distance. It brings faster adjustment speed and higher focusing accuracy so that defocusing images required by TIE algorithm can be quickly and stably collected. A complementary metal oxide semiconductor (CMOS) camera (Basler, MED ace 5.1, 2448 ×2048, 2.2 μm pixel size) placed on top of the microscope is synchronized with the control board for image acquisition, and universal serial bus (USB) 3.0 is used to transfer data to the computer with the limited frame rate of 75FPS. When the objective is rotated, the FPGA reports the current objective position to the computer and sets the appropriate system parameters. The FPGA controller also provides a synchronization mechanism for the LED array and camera. It communicates with the computer via USB-to-serial and accepts command parameters from the computer, such as the radius of the display aperture, the color, and the position of the pattern. Next, it reads the bitmap of the display pattern stored in the on-chip random access memory (RAM) and then drives the programmable LED array to display the pattern at high speed. To simplify the microscope interface, we used a USB Hub to combine the USB of the camera and USB-to-serial into one interface. The bandwidth required for serial communication is extremely small, which does not affect the transmission of camera image data, ensuring that the camera works at the highest rate.

Fig. 11

Schematic diagram of MQP-SCLM. (a) Physical diagram of MQP-SCLM. (b) External control handle for precise adjustment of the displacement of the electronic platform in all three x,y,z directions. (c) ETL for automatic control of defocusing distance. (d) Optical path diagram of the MQP-SCLM

The supporting software of MQP-SCLM provides the capabilities of smart control, multi-contrast imaging, and powerful processing and analysis, enabling seven imaging functions to be implemented through the same software system. Four functional modules are defined to coordinate to easily control the system, providing users with high-quality observation images and accurate analysis data. First, the illumination module is used to flexibly adjust the LED illumination pattern to match different imaging modes. Common illumination adjustments such as pattern and color adjustments are provided to tailor the function precisely to the user’s requirements. Also, we defaulted the optimal illumination pattern for each imaging mode in advance to enable the software to acquire a high-quality image for observation. In the camera settings module, we redeveloped the camera’s SDK so that users can easily set the camera exposure time, white balance, gain, and other parameters. According to the used objective lens and illumination parameters, all camera parameters are also optimally preset in our software to provide superior images for display and subsequent computational reconstruction. The display view module is used to present all direct visualized observation images and post-recovery results. The multi-window display function is designed to monitor simultaneous multi-contrast images in a software interface for in-depth analysis of the specimen. In the smart processing and analysis module, our MQP-SCLM also provides important information on specimens interactively or fully automatically. With powerful image processing functions we can display specimen signals with even greater contrast or SNR. At the same time, besides size and volume information, the detailed data on cell physiological processes, can be extracted from the imaging result for quantitative analysis.

Principle and algorithms

∙ multi-contrast microscopic imaging

Phase contrast approaches, such as dark field imaging [81, 218], Rheinberg imaging [219, 220], DPC imaging [27, 29, 218, 221], which obtain diversified 2D structural information of the specimen, are established as widely and commonly configured label-free technique for microscope to visualize transparent specimens. However, all of these methods require spatially-modulated illuminations by introducing additional physical masks and optical filters before the condenser lens of the microscope. For example, a spider stop is inserted into the aperture focal plane of the condenser to create a hollow cone of illumination with larger NA compared to that of the objective lens, extracting high-resolution detailed information of the specimen. Computational light microscopy brought a revolution of the microscope in diversification, flexibility, and convenience of imaging methods. The active illumination control (LED or LCD) were introduced into the microscope system to replace the traditional aperture diaphragm, realizing so-called “computational illumination” that gains the flexibility to produce sophisticated illumination patterns or dynamically switchable illumination sources with no physically moving parts [81, 83, 84, 92, 109, 222, 223].

In a microscopic system, the NA of the objective lens is a very important parameter because it determines the maximum diffraction angle that the microscope can receive. In our MQP-SCLM, the illumination patterns corresponding to seven imaging approaches are accurately set according to the NA of the objective lens. Specifically, the illumination NA of ith LED can be calculated:

$$ NA^{i}_{ill}=\sin (\theta_{i}) {\textrm{ = }}\sin\left[arctan\left(\frac{{\sqrt {{{\left| {{x_{i}}\Delta d} \right|}^{2}} + {{\left| {{y_{i}}\Delta d} \right|}^{2}}} }}{h}\right)\right] $$

(4)

where Δd is the pixel pitch of LED array, x_i and y_i are location coordinates of LED units, and h is the distance between the LED array and specimen. And also, the angle of illumination θ_i=(θ_ix,θ_iy) can be defined as θ_ix = arctan(x_iΔd/h),θ_iy = arctan(y_iΔd/h).

As shown in Fig. 12(a), bright field imaging is the simplest illumination technique, and the bright field image corresponds to illumination by LEDs that lies within the cone of angles described by the objective NA, i.e., $NA^{i}_{ill}\leq NA_{obj}$. Bright field imaging collects the most of directly transmitted light, thus, it is mostly suitable to visualize amplitude-contrast, e.g., absorptive structures of a specimen. While the dark field method excludes the unscattered beam from the image, and the dark field image is obtained by illuminating the specimen from angles beyond the angular acceptance of the objective ($NA^{i}_{ill}>NA_{obj}$). To achieve bright field and dark field imaging, we provide different LED patterns to match the NA of the objective lens. Besides, the Rheinberg illumination, known as optical staining with color patterns, can also be implemented by displaying different patterns on the central cone region (bright field) and the surrounding hollow cone region (dark field) of LED array simultaneously for more complicated color phase-contrast imaging. As a classic form of optical staining, Rheinberg illumination can render the normally colorless specimen with rich color against a contrasting background. Moreover, ZPC can be realized by displaying a ring-type pattern that matches the annular phase plate in the back focal plane of the phase contrast objective lens, and a phase-contrast image can be recorded without using any physical diaphragm.

Fig. 12

Technical roadmap for MQP-SCLM. (a) Multi-contrast imaging including bright field, dark field, Rheinberg optical staining, DPC, and ZPC. (b) High-throughput imaging based on FPM. (c) QPI based on TIE

DPC is a label-free phase contrast imaging based on asymmetric illumination, which only needs to collect several intensity images, achieving twice the imaging resolution of coherent imaging [27, 160, 224]. In contrast to DIC based on interferometric imaging, it provides a flexible, low-cost alternative which only need to replace the light source of commercial bright field microscopes with the LEDs. Our system realizes DPC imaging using the same system configuration. By controlling the LED to produce asymmetrical half-circular illumination patterns, the phase-contrast images of the specimen along the asymmetry axis are recorded. Then, a simple differential algorithm is used to calculate the phase gradient distribution of the specimen:

$$ I^{DPC}_{lr}(\mathbf{x})=\frac{I_{l}(\mathbf{x})-I_{r}(\mathbf{x})}{I_{l}(\mathbf{x})+I_{r}(\mathbf{x})} $$

(5)

where I_l(x) and I_r(x) represent the captured images in the left and right illuminations, respectively. $I^{DPC}_{lr}(\mathbf {x})$ is the phase gradient distribution of the specimen along the left-right shearing direction. By adjusting the illumination angle of the asymmetric axis, the phase gradient of the specimen along arbitrary shearing directions can be established. Furthermore, our previously published results have suggested that the optimized asymmetric annular pattern makes it impossible to enhance the contrast of phase gradient images [86].

∙ wide field and high resolution imaging based on FPM

FPM is a recently developed technique that overcomes the physical SBP limit of a bright field microscope by iteratively stitching together a number of variably illuminated, low-resolution intensity images in Fourier space [60, 194, 225, 226]. Instead of starting with a high-resolution objective lens and stitching together a larger FOV, FPM uses a low NA objective lens to take advantage of its innate wide FOV to recover high-resolution complex distribution (both amplitude and phase) of the specimen. As shown in Fig. 12(b), by invoking the LED array, each LED provides a plane wave of unique angle to illuminate the sample, recording a series of low-resolution images. In Fourier space, the sequential scanning illumination angles shift different amounts of high spatial frequency information into the low NA objective lens. These raw images are stitched together in Fourier space to generate a bandwidth-extended, high-resolution spectrum, thus achieving wide-field and high resolution imaging. For the problems in positional misalignment and noise that affect the accuracy of FPM reconstruction, we have proposed several effective correction and optimization methods [89, 227]. In our MQP-SCLM, the LED positional misalignment correction and adaptive step-size strategy are combined for FPM recovery processing to improve the imaging quality of FPM.

∙ quantitative phase imaging based on TIE

Without the need for a separate reference beam, non-interferometric QPI is another developed technique, which directly retrieves phase from intensity measurements [55–58, 61]. The phase retrieval from the recorded intensity is considered as solving a mathematical “inverse problem” [66, 67]. Such kind of problem can be solved by the deterministic algorithm and iterative algorithm [185, 228–232]. Generally, the deterministic algorithm recovers phase from several intensity measurements of the specimen under partially coherent illumination [71, 86, 229, 233–237], and it has higher stability and achieves twice the lateral resolution of the coherent diffraction limit [165, 176, 238, 239]. Defocusing acquisition is the effective method to produce an intensity image of the phase contrast of the specimen, and the developed QPI approach is called TIE. In essence, TIE is a second-order elliptic partial differential equation, outlining the quantitative relationship between the variation of intensity along the optical axis to the phase of the optical field at the plane perpendicular to the optical axis [70]:

$$ {-k\frac{\partial I(\mathbf{x},{z})}{\partial z}=\nabla \cdot \left[I(\mathbf{x},z)\nabla \phi(\mathbf{x})\right] } $$

(6)

where k is the wave number 2π/λ; x is the transverse coordinates x=(x,y); I(x,z) is intensity distribution at the plane located at the propagation distance z. Under the appropriate boundary condition (the Neumann boundary condition), this second-order elliptic partial differential equation is solved by FFT method [185, 240–242], thus the phase information of object can be easily obtained through intensity derivative on the in-focus plane [243–245].

Recently, another deconvolution solution is put forward based on the derivation of defocus phase transfer function (PTF) under weak phase approximation [231, 246], which lays a theoretical foundation for improving TIE imaging performance by optimizing illumination. In our published work, annular illumination has been proved to provide an optimal illumination scheme for TIE which enhances imaging contrast within the whole theoretical bandwidth of partially coherent imaging [246, 247]. In our MQP-SCLM system, we use annular illumination that matching the objective NA_obj (NA_ill equals to NA_obj) to collect three defocus images and recover quantitative phase distribution of specimens by one-step deconvolution [Fig. 12(c)]. The captured three images first are calculated to eliminate the background term, leaving only phase term [248, 249]:

$$ {\frac{I_{\Delta z}(\mathbf{u})-I_{-\Delta z}(\mathbf{u})}{4I_{0}(\mathbf{u})}=Im[PTF(\mathbf{u})\phi(\mathbf{u})] } $$

(7)

Then, the PTF of the system can be calculated by the following formula:

$$ {PTF(\mathbf{u})=\iint{S(\mathbf{u'})|P(\mathbf{u'})||P(\mathbf{u'}+\mathbf{u})|e^{ik\Delta z\left(-\sqrt{1-\lambda^{2}|\mathbf{u'}|^{2}}+\sqrt{1-\lambda^{2}|\mathbf{u}+\mathbf{u'}|^{2}}\right)}}d\mathbf{u'} } $$

(8)

Finally, quantitative phase information can be reconstructed by Fourier space de-convolution in one-step [234, 246, 248]:

$$ {\phi(\mathbf{x})=\mathscr{F}^{-1}{\left\{\frac{I_{\Delta z}(\mathbf{u})-I_{-\Delta z}(\mathbf{u})}{4I_{0}(\mathbf{u})}\frac{{{Im[PTF(\mathbf{u})]}}} {|{Im[PTF(\mathbf{u})]}|^{2}+\beta}\right\}}} $$

(9)

here β is the Tikhonov regularization term used to avoid excessive amplification of noise during deconvolution [250].

Experiments

∙ multi-contrast microscopic imaging

We first demonstrated the multi-contrast imaging capabilities of MQP-SCLM by realizing several classical microscope techniques such as bright field, dark field, ZPC, DPC, and Rheinberg optical staining. All these multi-contrast imaging modes can be achieved in a single-shot image by switching the illumination patterns. The experiments were conducted on the unstained single diatom microalgae cell and the unstained spiral algae (S68786, Fisher Scientific). A 40 ×, 0.65NA Plan microscope objective (40 ×0.45NA phase-contrast objective for ZPC) and a 10 ×, 0.25NA objective (10 ×0.25NA phase-contrast objective for ZPC) were used to observe the specimens. Figure 13 shows the multi-contrast imaging results of a small single diatom microalgae cell and spiral algae with different objective lenses.

Fig. 13

Multi-contrast microscopic imaging results of a small single diatom microalgae cell and spiral algae with different objective lenses, including bright field, dark field, DPC, ZPC, Rheinberg optical staining. (a), (b) Observation results under bright field and dark field imaging. (c), (d) Observation results under DPC imaging along the left-right and up-down shearing directions. (e) Observation results under ZPC. (f)-(h) Observation results under Rheinberg optical staining imaging with different color configurations

As shown in Fig. 13(a)-(b), bright field images of two different algae specimens present the highly absorbing features (e.g., chloroplasts) while dark field images give more high-resolution details corresponding to the highly scattering information of diatom frustule shell. In DPC and ZPC, the experimental images provide more information about phase features (e.g., filaments and silica cell wall structures) and the internal structure is much better contrasted and can be well resolved, as illustrated in Fig. 13(c)-(e). Up-down and left-right phase-contrast images enhance the internal features from two shearing directions, and any desired direction of asymmetry can be used to highlight features of interest with the flexibility of the LED array patterning designed by the user through the software interface. In the Rheinberg optical staining imaging mode [Fig. 13(f)-(h)], we provide the experimental results of algae specimens with different color configurations. The central circle illumination pattern (bright field) determines the captured intensity background, while the outer ring illumination pattern (dark field) will stain the highly scattering details of the specimen with different colors. Not only the marginal contours of the diatom are visualized in a distinct manner, but the small pores on the frustule appear sharply demarcated in Rheinberg illuminations. High angle incident illumination will increase the contrast of small features of the specimen which corresponds to the high-frequency details in the spectrum domain. It is worth noting that in Rheinberg filter, the transmittance of the central circle should be much darker than the outer ring in order to enhance the overall contrast and the visibility of the detailed structure. Otherwise, the contrast of the dark field scattering component on the resultant image will be swamped out by the background signal.

Then, we employed a mosquito mouthpart specimen for multiplane refocusing under different contrast imaging modalities. In this experiment, the specimen was imaged by an objective lens of 10 ×0.25 NA, a tube lens of 0.25 ×. The ETL can be automatically controlled by the software to focus on different planes of the specimen. Figure 14 shows its refocusing stack images, and the main structures of the mouthpart are clear and distinguishable on four different focus planes. From bright field refocusing images in the first row, the mosquito head and compound eye provide relatively intensity contrast due to the strong absorption. The structures of antennae, maxillary palps, proboscis (include labella and labium), and labrum with high-resolution information is focused on different planes, and the features of large bushy antennae are marked with solid red circles and dash circles. Moreover, the dark field refocusing images enhance small details of antennae structure with highly scattering, and the hairy antennae feature with high resolution (marked with white circles) can be resolved axially. In the last row, the DPC images at different depths show the high-resolution phase features of mouthpart structure and provide phase gradient maps of antennae and maxillary palps across multiple axial slices (marked with white arrows and red arrows).

Fig. 14

Refocusing results of mosquito mouthpart specimen in bright field, dark field, DPC at z=−60μm, z=−20μm, z=20μm, z=60μm

∙ wide field, high resolution imaging using FPM

We demonstrated the wide field, high resolution imaging capability of our MQP-SCLM for its intended high-throughput biomedical observation experimentally. The USAF target, human lung tissue section, and human kidney vessel cells were employed as the observation specimens to achieve FPM. We employed an objective lens with 4 ×, 0.1NA and tube lens with 0.5 ×, and collected a set of 225 low-resolution intensity images from different incident angles. Before the FPM reconstruction process, the brightness and the position of LED illumination have been calibrated. Figure 15 shows the FPM reconstructed results of these specimens. The full FOV raw low-resolution intensity image of the USAF resolution target is presented in Fig. 15(a), and the region of interest is marked and enlarged by a white box. Figure 15(b1) and (b2) present the recovered high-resolution intensity images of the corresponding central region of interest of the USAF target. As can be seen, the highest distinguishable resolution target elements in Fig. 15(b2) is Element 5 in Group 10 (corresponding to the half-pitch resolution of 274nm), and the intensity line-scan profile of Group 10 in Fig. 15(b3) shows the verified results. Moreover, we also provide high-resolution measuring specimens of the stained human lung tissue section and human kidney vessel cells. Figure 15(c) and (f) present the FOV of these two specimens, respectively. Figure 15(d1), (e1), (f1), (g1) and (d2), (e2), (f2), (g2) illustrate the comparisons of low-resolution and high-resolution image in the FOV of different enlarged regions. Significantly enhanced resolution in FPM reconstruction results demonstrates more detail in the pathological tissue than the conventional microscope.

Fig. 15

High-throughput (wide field and high resolution) imaging results of USAF absorption target and stained tissue sections (human lung tissue section and human kidney vessel cells) reconstructed by FPM. (a) Full FOV low-resolution image of USAF target under bright field. (b1), (b2) The sub-region images of reconstructed high-resolution image. (b3) The intensity quantitative profile of the red line in (b2). (c), (f) Full FOV low-resolution image of stained human lung tissue section and human kidney vessel cells under bright field. (d1), (e1), (g1), (h1) The sub-region images of low-resolution image. (d2), (e2), (g2), (h2) The sub-region images of high-resolution image reconstructed by FPM

∙ quantitative phase imaging using TIE

In order to validate the QPI of MQP-SCLM, two different specimens were used for the quantitative phase characterization. First, a quantitative phase target QPT^TM with a height of 100nm was used for the quantitative investigation of achievable imaging resolution, and the reconstructed result is shown in Fig. 16(a1). As shown in Fig. 16(a2)-(a3), the resolution element Group 10 is enlarged and displayed, and the line profile of resolution elements in Group 10 is plotted in Fig. 16(a4). We can clearly distinguish these resolution bars up to Element 4, which means that achievable imaging resolution of MQP-SCLM under 0.4NA objective is 345nm.

Fig. 16

QPI experiment results on standard phase resolution targets QPT^TM and microlens array. (a1)-(a3) Reconstructed quantitative phase and its sub-region images. (a4) The quantitative profile along the red line in (a3). (b1) Reconstructed quantitative phase of the microlens array. (b2) The quantitative profile along the red line in (b1). (b3) Comparison of quantitative profile of reconstructed phase and nominal profile

Then, a standard plano-convex microlens array (SUSS, pitch 250 μm, fused silica, RI of 1.46 at 507nm) was measured using 0.25NA objective with 0.45 × tube lens. Figure 16(b1) shows the reconstructed quantitative phase. Overall, the distribution of the lens array is clearly reflected in the quantitative phase map. In order to assess the accuracy of the phase measurement, the measured thickness profiles of microlens taken from the solid yellow square in Fig. 16(b1) and theoretical profile line are compared quantitatively in Fig. 16(b3), showing a reasonable agreement. The pitch of microlens is measured to be 252 μm, which is slightly larger than the nominal pitch of 250 μm. The height of the measured microlens is about 25 μm, which is consistent with the nominal value. The small discrepancy may be attributed to the imperfect boundary condition in the phase reconstruction process and the imperfection profile of each lens on the microlens array.

Additionally, the reconstructed quantitative phases of unstained biological specimens are provided to demonstrate the imaging ability of MQP-SCLM on clusters of unstained cells distributed on a glass specimen slide. In this experiment, only three defocus intensity images were captured with 50ms exposure time, enabling the high-speed QPI of MQP-SCLM. Figure 17 gives a comparison of the QPI performances under conventional circular illumination and our optimized annular illumination apertures (with the same objective lens of 40 ×, 0.65 NA). As shown in Fig. 17(a) (conventional circular illumination aperture) and 17(c) (our optimized annular illumination aperture), due to the poor transfer response of the PTF at the central low frequency and high frequency approaching 2NA_obj, the low-frequency cloud noise is included in the reconstructed phase in Fig. 17(a) while the high-resolution details are not reconstructed clearly. These deficiencies are avoided under annular illumination aperture, and the reconstructed phase exhibits high robustness and enhanced resolution. Two enlarged sub-areas are selected by dotted rectangular shape to compare phase details. As can be seen in Fig. 17(b1)-(b2) and (d1)-(d2), the morphological shape and internal details of the HeLa cells are clearly visible, and the tiny phase features (e.g., grains and fibers) are well resolved, demonstrating the near-diffraction-limited lateral resolution and high-frequency features in the cell can still be distinguished.

Fig. 17

The comparison of reconstructed quantitative phase of HeLa cell under circle and annular illumination apertures. (a) The reconstructed phase of HeLa cells under circle illumination aperture. (b1)-(b2) The enlarged images of the sub-region of Area1 and Area2 in (a). (c) The reconstructed quantitative phase of HeLa cells under annular illumination aperture. (d1)-(d2) The enlarged images of the subregion of Area1 and Area2 in (c)

Further, we imaged unstained MCF-7 breast cancer cell and HepG2 living carcinoma cells fixed in formaldehyde solution using the 20 ×0.4NA and 40 ×0.65NA objective lens with 1 × tube lens, respectively. Figure 18(a) is the full FOV quantitative phase image of MCF-7 cell, and we enlarge four sub-regions of the reconstructed phase in Fig. 18(b1)-(b4) to highlight our high quality phase reconstructions. Subcellular tiny grains feature (nuclei and organelle) are correctly reconstructed, and features with higher RI values in the cytoplasm appear brighter contrast in quantitative phase image are reconstructed. As shown in Fig. 18(c), the experimental image of HepG2 also gives the high-resolution quantitative phase distribution. The fine structures and subcellular structures are also visible and resolved. The recovered quantitative phase distributions of cellular features enable the calculation of dry and buoyant mass, sphericity, and other morphometric descriptors used for cell profiling.

Fig. 18

The reconstructed quantitative phase of unstained MCF-7 Breast Cancer Cell and HepG2 living carcinoma cells. (a) Full FOV quantitative phase image of MCF-7 cell. (b1)-(b4) Enlarged images of the sub-region of interest in the red, wathet, blue, and yellow boxes. (c) The reconstructed quantitative phase of HepG2

Paradigm 3: miniaturized multi-contrast smart computational light microscope (MMC-SCLM)

With telemedicine applications, microscope is required to be as compact and light-weight as possible to provide cost-effective solution for health care especially in the developing world where medical facilities and infrastructure are extremely limited. Computational imaging allows microscope to tailor the optical system to become miniaturized and portable. We developed a miniaturized, low-cost, and portable multi-contrast unlabeled microscope, which is named as MMC-SCLM. In this system, a highly integrated optical path using a miniaturization lens is adopted to replace the complex optical system of traditional microscopes, thus reducing the size of the whole microscope to 14×16.5×20cm³. We are compatible with five imaging modes in this system to produce multi-contrast images of bright field, dark field, Rheinberg optical staining, DPC, QPI (DPC-based) for living cells. All imaging modes can be flexibly switched and operated through our designed software. In addition, cell analyzing functions are also added to our system, which can realize cell counter and 3D profile measurement. In this subsection, we demonstrated the multi-contrast imaging and QPI capabilities of MMC-SCLM by multiple observation and detection experiments with different specimens.

Optical configuration and software

Figure 19(a) displays the physical diagram of our MMC-SCLM, and it is designed as the inverted microscope system. As shown in Fig. 19(b), a commercial full-color LED array with a unit size of 2mm is placed at the top of the entire system, and its center aligned with the optical axis of the microscope system. The complex optical path of traditional microscopes is replaced by a miniaturization lens with NA of 0.14 (6.4 ×), a focal length of 4.25mm, a lens distortion within 1%. This miniaturization lens can image the specimen between its one and two times the focal length directly onto the CCD target. Thus, the working distance of the optical system was reduced to less than 20mm. At the bottom of the whole system, a high-performance color CCD industrial camera (the imaging source DFK 23U445, 1280 ×960, pixel size of 3.75 μm) is used to capture the intensity image for observation and other recovery algorithms. The command interaction between the software system and hardware system is implemented by an independently designed hardware control circuit board, and it is embedded into the back of the microscope, as shown in Fig. 19(c).

Fig. 19

Schematic diagram of MMC-SCLM. (a) Physical diagram of MMC-SCLM. (b), (c) Internal structure diagrams from different angles. (d) Hardware system interactive control diagram

Figure 19(d) displays the interactive flow of the whole system. The hardware circuit board is the core of the control, which accepts instructions from the software operating system and controls the hardware to implement response operations. All control instructions are performed through a set serial communication protocol within 1ms. It should be noted that due to the LED array lighting at a certain refresh frequency, the camera exposure needs to be synchronized with the light source to ensure the uniformity and accuracy of the whole image exposure. Therefore, USB 3.0 data transmission protocol is adopted in image acquisition, connecting with the external trigger of the camera to realize the rapidity and stability of image acquisition and transmission. Under such a hardware configuration, the size of the entire microscope system was reduced to 14×16.5×20cm³.

In order to ensure the implementation of multi-contrast imaging and smart analysis, we developed an operating software for MMC-SCLM. Similar to the MQP-SCLM system, the operating software is designed as four modules to realize illumination control, camera setting adjustment, result display, and smart processing and analysis. A detailed description of each module can be found in the “Optical configuration and software” subsection in “Paradigm 2: multi-contrast quantitative phase smart computational light microscope (MQP-SCLM)”. These modules are managed by flexible layout and collaborate for diverse imaging and data analysis. As a result, the specimen can be observed in five imaging modes, including bright field, dark field, DPC, Rheinberg optical staining, and QPI (based-DPC), get continuous data.

Principle and algorithms

∙ multi-contrast microscopic imaging

MMC-SCLM achieves five imaging approaches, including bright field, dark field, DPC, Rheinberg optical staining, and QPI (based-DPC), as shown in Fig. 20. As described in “Principle and algorithms” subsection in “Paradigm 2: multi-contrast quantitative phase smart computational light microscope (MQP-SCLM)”, bright field imaging requires the illumination NA_ill of all lit LED units are less than NA_obj (transmitted light), and whereas dark field imaging requires all NA_ill are greater than NA_obj (diffraction or scattered light). Rheinberg imaging realizes optical staining through a combination of different colors for bright field and dark field illumination to mark different spatial frequency information of specimens, enhancing the imaging contrast.

Fig. 20

Technical roadmap of MMC-SCLM. (a) Multi-contrast imaging including bright field, dark field, DPC, and Rheinberg optical staining. (b) QPI based on DPC

MMC-SCLM is also capable of phase gradient imaging by DPC. As can be seen from Step1 and Step2 in Fig. 20(b), DPC records the phase contrast images of the specimen by asymmetric illumination, and the phase gradient information of the specimen is further obtained by a simple differential calculation of Eq. (5). However, the compact optical system has a limited distance between the LED and the sensor, which may lead to uneven background under asymmetric illumination. Especially in the DPC-based QPI method, such background unevenness will cause serious reconstruction errors. In order to avoid the imaging error caused by the uneven background, we adopt a correction and compensation method to improve the imaging quality and phase accuracy while ensuring the imaging speed of the DPC imaging microscope system.

The background correction and compensation method is based on PCA and plane fitting. Performing PCA on the recorded phase gradient image and then extracting the first principal component and its corresponding coefficient to fit the plane distribution as image background. The fitted background is then removed from the captured image to obtain the corrected image. When removing the background distribution, a uniform intensity distribution with the fitting background means should be added to the recorded image to compensate for the phase gradient.

∙ QPI based on DPC

Asymmetric illumination is one of the key approaches to produce phase contrast intensity, and the derived QPI method is called DPC QPI. This method was first proposed to establish a linear relationship between the image intensity and the specimen’s phase gradient [85, 86, 228]. Subsequently, slow-varying object approximation and weak object approximation are introduced to simplify imaging process, thus two quantitative phase inversion solvers were developed [228]. Under slow-varying specimen approximation, the phase gradient transfer function (PGTF), which relates the phase gradient and the captured intensity in the sense of a look-up table, was extracted from the partially coherent transfer function. Thus, the measurement of phase gradient along two orthogonal directions allows retrieval of phase distribution [85, 251]. Under weak phase approximation, PTF, which establishes a quantitative relationship between the object phase and the captured intensity, was derived by linearizing partial coherent imaging process [61, 165]. A one-step deconvolution solver was developed to recover the quantitative phase of the object in this approximation. In our recent advances, annular illumination whose illumination NA_ill equal to the objective NA_obj and brightness varies with angle as a cosine function, maximizes the response at almost all frequencies of the theoretical bandwidth of the entire partially coherent imaging (from 0 to $\frac {{2NA}_{obj}}{\lambda }$) [86, 165]. Such an annular illumination is used in our MMC-SCLM to optimize the imaging resolution and imaging contrast. In this case, the PTF is solved by:

$$\begin{array}{*{20}l} {PTF_{lr}(\mathbf{u})=\frac{\iint{S_{lr}(\mathbf{u_{j}})\left[{P^{*}(\mathbf{u_{j}})P(\mathbf{u}+\mathbf{u_{j}})-P(\mathbf{u_{j}})P^{*}(\mathbf{u}-\mathbf{u_{j}})}\right]}d^{2}\mathbf{u_{j}}}{\iint{|S_{lr}(\mathbf{u_{j}})||P(\mathbf{u_{j}})|^{2}}d^{2}\mathbf{u_{j}}}} \end{array} $$

(10)

here S_lr(u_j) is the illumination function, u_j is the frequency shift corresponding to the illumination angle. P(u) is the pupil function of the objective lens, which is a low-pass filter with cut-off frequency of $\frac {NA_{obj}}{\lambda }$. In our system, the PTF is solved and saved in the local folder of the operating software. When recovering the quantitative phase of the specimen in our system, only the PTF is read to implement one-step deconvolution with phase gradient images, as shown in Step4 in Fig. 20(b) [160, 250]:

$$ {\phi(\mathbf{x})=\mathscr{F}^{-1}{\left\{\frac{\sum_{i}\left[{{PTF_{i}^{*}(\mathbf{u}){\cdot}I^{DPC}_{i}(\mathbf{u})}{}}\right]} {\sum_{i}|{PTF_{i}^{*}(\mathbf{u})}|^{2}+\beta}\right\}}} $$

(11)

where ^∗ represents the complex conjugate item, and i denotes the asymmetric axis direction of illumination. β is the Tikhonov regularization term used to avoid excessive amplification of noise during deconvolution.

Experiments

∙ multi-contrast microscopic imaging

To demonstrate the multi-contrast microscopic imaging capability of MMC-SCLM, we observed different specimens under bright field, dark field, rainbow dark field, and Rheinberg optical staining imaging. Figure 21 shows the multi-contrast visualized results on the specimens of bird feather, spirogyra communis, lily anther, ant, and dicot. As we can see from the first row, the bird feather can be observed clearly under bright field imaging due to its strong absorption. The scattered light imaging of the dark field provides more detailed information, and the structure of the feather rachis and barbules are clearly displayed. For arbitrary specimens, such as spirogyra communis, lily anther, ant, and dicot, the bright field and dark field imaging compensate each other to acquire the overall structure and high-resolution details of the specimen. In the third column, we show the imaging results of these specimens under rainbow dark field imaging. The spatial structures along different directions are marked with variable colors, which provides intuitive images for distinguishing wispy structures in the specimen. Rheinberg imaging achieves optical staining for different structures of the specimen by flexible, customized illumination pattern (color and shape) to highlight the high-resolution details of the specimen, and the results are shown in the last column. The integration of these imaging methods allows for a diverse view of the specimen without replacing any hardware.

Fig. 21

The muli-contrast visualization results of bird feather, spirogyra communis, lily anther, ant, and dicot under bright field, dark field, rainbow dark field and Rheinberg optical staining imaging

We also employed several of the above specimens to verify the imaging effectiveness of MMC-SCLM for DPC. Figure 22 displays the observation results of bird feather, spirogyra communis, and ant under DPC imaging without and with removing tilt phase gradient. In the blue dotted box on the left, we show the uncorrected DPC results along the left-right and up-down shearing directions. The inhomogeneity of the illumination introduces an undesired tilted phase gradient with respect to the shear direction over the phase gradient of the specimen. The PCA method was executed to remove the tilted phase gradient, and their resulting phase gradients are shown in the red dotted box on the right, demonstrating improvement in smoothness compared with the raw phase gradients. The sub-structural features, as well as the thin borders of the specimens, can be clearly observed without any tilted background perceivable.

Fig. 22

Comparison of phase gradient results of bird feather, spirogyra communis, and ant under DPC imaging without and with removing tilt error

In addition, multi-contrast imaging of MMC-SCLM allows the comparison of specimen details through flexible sub-window segmentation displays or multi-window full FOV tiled display. As shown in Fig. 23, we show several examples of multi-contrast display views of our system. Figure 23(a) is the stitching display view of two contrasts imaging results on maize slitting, that is, the left and right (or the upper and lower) sub-regions display the visualized results from different imaging modes. In the same view of the software interface, two phase-contrast images (Rheinberg optical staining imaging in the left sub-region and dark field imaging in the right sub-region) are presented to distinguish different visualization information of a specimen. Both images can be displayed and saved in full view to compare the microstructure detail of any sub-region, as shown in Fig. 23(b1) and (b2). Furthermore, the full FOV can be split into more sub-areas to display more phase contrast observations simultaneously. As shown in Fig. 23(c), four images of the pathology specimen are displayed on the same interface forming a full FOV image so that significant differences can be found from one observation. Alternatively, multiple observations can also be displayed simultaneously on the interface at full FOV size, as shown in Fig. 23(d1)-(d4). These multi-contrast observation tools effectively avoid the instrument modification, complex observation recording and comparison required by conventional multi-contrast imaging, providing a simple, straightforward, and efficient observation means.

Fig. 23

Multi-contrast visualization results through sub-window segmentation displays or multi-window full FOV tiled display. (a) Stitching display view of two contrasts imaging results on maize slitting. (b1), (b2) The enlarged images of the Rheinberg and dark field results of the region of interest within the white dotted box in (a). (c) Stitching display view of four contrast imaging results on pathology specimen. (d1)-(d4) Four full FOV of multi-contrast images on pollen specimen

∙ quantitative phase imaging using DPC

We validated the fast and highly robust quantitative phase reconstruction of MMC-SCLM through several experiments on unlabeled specimens. First, to verify the achievable imaging resolution of our MMC-SCLM, we measured a pure phase resolution target QPT^TM with the height of 100nm, and the experimental results are shown in Fig. 24. Figure 24(a) shows the observed image of this specimen in bright field microscope, which has a weak contrast and is almost invisible due to its weak absorption. The phase gradient images after background correction (the up-down shearing direction) are shown in Fig. 24(a2), and it is used to perform deconvolution to avoid the low-frequency tilt phase error. Figure 24(c) displays the reconstructed phase. The quantitative profile along the red line is then extracted and plotted in Fig. 24(d) to reveal the highest achievable resolution. It can be seen that Group 9, Element 1 can be recovered correctly, which indicates the theoretical realizable resolution of our MMC-SCLM is 970nm (half-pitch resolution). Then, we measured a 250 μm pitch plano-convex quartz microlens array (SUSS MicroOptics, lens diameter 240 μm, hex-packed) by our MMC-SCLM. A microlens element in a solid white frame of Fig. 24(e) is selected as the target for quantitative phase measurement, and the reconstructed quantitative phase is shown in Fig. 24(f). The accuracy of this reconstructed phase is demonstrated by quantitatively comparing their differences to the nominal values. As shown in Fig. 24(g), the profile of the reconstructed phase is basically consistent with the nominal value (except that the measured value of 19.52 μm at the extreme value is slightly smaller than the standard value of 20 μm). In addition, the validated quantitative phase can be displayed with a 2D quantitative phase map or a 3D pseudo-color phase profile in our MMC-SCLM to provide a high-contrast view of the 3D structure.

Fig. 24

Experiment for imaging resolution verification by standard phase resolution targets QPT^TM and microlens array. (a) Bright field image of QPT^TM. (b) Phase gradient image obtained by DPC. (c) Reconstructed quantitative phase. (d) Quantitative profile along the red line in (c). (e) Bright field image of the microlens array. (f) Reconstructed quantitative phase of a selected microlens element. (g) Comparison of quantitative profile of reconstructed phase and nominal profile, and 3D pseudo-color phase profile of the selected microlens element

Due to the advantages of miniaturization, low cost, and portability of MMC-SCLM, it provides a powerful means for telemedicine diagnostic tests, which yields immediate acquisition of information on an individual’s condition to facilitate treatment decisions or further extensive testing, offering the advantages of widening accessibility to diagnosis, minimal sample volumes, reduced costs, and rapid analysis times. We demonstrated the imaging capability of MMC-SCLM by imaging a specimen of HeLa cells prepared in situ, and the experimental results are shown in Fig. 25. Figure 25(a) shows the reconstructed quantitative phase of unlabeled HeLa cells. Figure 25(b) and (c) exhibit the observed images in dark field and phase gradient image in DPC. From these results, dark field and DPC give conspicuous phase-contrast information of the unlabeled sample, while QPI further provides quantitative, high-resolution details. In addition, the unlabeled cell samples can be presented in a high-contrast pseudo-color profile, providing a direct visualized 3D structure of the sample, as shown in Fig. 25(e). Furthermore, the cell counter was developed in MMC-SCLM as an auxiliary function, and the result is shown in Fig. 25(d). It can be seen that the exact number of the HeLa cells in the full FOV is accurately counted as 120.

Fig. 25

Reconstruction quantitative phase of HeLa cells. (a) Reconstruction quantitative phase and quantitative profile of HeLa cells. (b) Dark field visualization image. (c) Phase gradient image under DPC imaging. (d) Cell counting display (120 cells in full FOV). (e) 3D pseudo-color phase profile display of HeLa cells

Paradigm 4: high-throughput lensless smart computational light microscope (HTL-SCLM)

Computational imaging allows the microscope to be simplified as much as possible or even remove the necessary objective lens of conventional microscopes. Such a lensless microscope structure maintains the imaging FOV and provides pixel super-resolution specimen information by implementing computational reconstruction algorithms on its acquired images. In this paradigm, we exhibited a lensless QPI microscope, termed HTL-SCLM, which only contains a camera sensor and illumination source, providing a portable, low-cost, stable, and high-throughput imaging tool. A multi-wavelength illumination scheme is utilized to record three raw images without mechanical moving adjustment. According to diffraction propagation theory, these images can be converted to multi-depth images at three defocusing distances. GS iteration is then performed to recover the high-resolution information of the specimen across full FOV of the sensor size [66, 67]. High-throughput experimental results for different specimens demonstrated that our lensless microscope achieves the high-throughput imaging with a wide FOV of 29.8474 mm² and imaging resolution (half-pitch imaging resolution) of 870 nm.

Optical configuration and software

Figure 26 shows the schematic diagram of our HTL-SCLM. It does not contain any moveable mechanisms, and the two essential components are a color LED unit and a monochrome CMOS sensor, as shown in Fig. 26(a) and (b). Thus, the size of the whole imaging system was shrink to one-tenth of the conventional microscope. The LED is employed as a coherent illumination source instead of a laser source to produce approximately spatially coherent illumination with the center wavelength of 631nm (red), 522nm (green), and 465nm (blue) (∼10nm bandwidth, ∼100μm size). The use of such a LED not only effectively suppresses the speckle noise and distortions caused by parasitic reflections in the optical setup but also provides a simple and cost-effective implementation for lensless microscope. In our HTL-SCLM setup, the LED unit is placed at 80mm away from the specimen plane to sequentially produce red, green, and blue light to illuminate the sample. An industrial camera (The Imaging Source, DMK-24UJ003, 3872 ×2764) with a pixel size of 1.67μm is used to capture images, thus it can achieve a full FOV of 29.8474 mm². The specimen could be placed as close to the sensor surface as possible [typically less than 1mm, see Fig. 26(d)] so that the spatial incoherence gating effect will minimize for high-resolution phase imaging. Under the above configurations, the lensless system is integrated in a size of 130×80×75mm³.

Fig. 26

Schematic diagram of HTL-SCLM system. (a) Physical diagram of HTL-SCLM system. (b) Hardware system interactive control diagram. (c) Multi-wavelength illuminations generated by a color LED unit. (d) Stage and CMOS sensor

The operating software of HTL-SCLM provides a friendly interface and versatile functions, which can be started quickly without complicated training for users. Four modules were developed to implement the algorithm preset, camera control, view display, and data analysis. Algorithm preset module must be performed prior to reconstruction to seek the accurate defocus distance and to select the appropriate image acquisition mode and reconstruction mode. Due to the absence of the objective lens, it is difficult for a lensless microscope to acquire an in-focus image. We overcome this problem by estimating the defocus distance through numerical propagation. In the subsequent reconstruction, the estimated defocus distance will be automatically applied to compensate for the phase artifacts caused by defocus and improve the resolution of phase reconstruction. The image acquisition settings provide two options for reconstruction, i.e., single image acquisition mode for one reconstruction and continuous image acquisition mode for multiple reconstructions. The single image acquisition mode captures only three images needed for single quantitative phase reconstruction and is suitable for a static specimen. The continuous acquisition mode, on the other hand, continues to acquire images to perform quantitative phase reconstruction, providing time-lapse imaging results of dynamic specimens. In reconstruction mode settings, two modes of the region of interest reconstruction and full FOV reconstruction can be selected alternatively. Current window reconstruction is required when the reconstruction result needs to be previewed, while full-field reconstruction should be selected after all parameters have been properly set. The other three modules are similar to several of the systems mentioned above. The camera control module enables flexible adjustment of camera parameters, such as frame rate, exposure time, gain. The view display module is used to display the visualized results and reconstructed phases directly in the software interface with 2D phase map, 3D pseudo phase profile, providing high-throughput imaging data. Besides, the data analysis module can afford quantitative and statistical analysis data by analysis tools of cell counting, cell profile analysis, automatic digital refocusing function. Combined with this operating software, our HTL-SCLM is a new generation of label-free high-throughput imaging tool, enabling smart and automatic microscopic imaging.

Principle and algorithms

Lensless holography is a promising high-throughput QPI method, which bypasses the intrinsical trade-off between the spatial resolution and FOV of conventional microscopes [170, 252–256]. It removes all lenses of the microscopic optical path, thus the light passing through the specimen is directly received with a wide FOV of the sensor surface. However, by use of a small object-to-sensor distance, the collection NA at the detection plane approaches 1, resulting in the imaging resolution limited by the sensor pixel size. Several pixel-super-resolution algorithms were developed to overcome this limitation in which a smaller effective pixel-size is synthesized from a series of low-resolution images captured by sub-pixel shifting or wavelength scanning through specific computational algorithms [182, 257–260]. However, all these methods require extra controllable mechanical devices or extra wavelength calibration and dispersion compensation processes, resulting in a significant increase in the complexity and cost of the whole system.

We combined phase retrieval (TIE-based) and pixel super-resolution (GS iteration) in our system to deliver super-resolved reconstructions with significantly reduced the number of raw measurements. The LED is used to achieve multi-wavelength illumination in a cost-effective manner, thus three intensity images with different wavelengths (red, green, blue) are captured. According to diffraction propagation theory, wavelength and propagation distance always appear in pairs, which means that a change in wavelength demonstrates an equivalent effect on the field propagation as a change in propagation distance [261, 262]. Thus, the captured images under multi-wavelength illuminations can be converted to multi-depth images, as shown in Fig. 27(a).

Fig. 27

Technical roadmap of HTL-SCLM with multi-wavelength illumination

First, the phase retrieved by the TIE is performed to give an initial input for the GS iterative phase refinement. The green wavelength is set as the normalized wavelength because it has forward and backward propagation defocusing images. Then, the longitudinal intensity derivative ∂I/∂z can be estimated through the following equation:

$$ \frac{{\partial I}}{{\partial z}} = \frac{{{{\left({\Delta {z_{b}}} \right)}^{2}}\left({{I_{r}} - {I_{g}}} \right) - {{\left({\Delta {z_{r}}} \right)}^{2}}\left({{I_{b}} - {I_{g}}} \right)}}{{\Delta {z_{r}}\Delta {z_{b}}\left({\Delta {z_{b}} - \Delta {z_{r}}} \right)}} $$

(12)

where Δz_r,b=z_r,b−z_g are equivalent propagation distances from the red/blue channel image to the green channel image. The form of wavelength and defocus distance of the same complex amplitude distribution can be expressed as $\frac {z_{r}}{\lambda _{r}}=\frac {z_{g}}{\lambda _{g}}=\frac {z_{b}}{\lambda _{b}}$. Thus, once three diffraction patterns at different illumination wavelengths (red, green, blue) are captured, the effective defocus distances of the red and blue channel images are converted according to z_r,b = z_gλ_r,b/λ_g. It should be noted that there is a tiny distance between the red, green, blue units of a LED, the recorded intensity under different wavelength illumination appears translation deviation. In order to address this problem, I_r and I_b are registered with the reference image I_g via computing the upsampled cross correlation through the FFT method and locating its peak [263]. We use the FFT-based solver to restore the phase distribution of the specimen due to its simplicity, efficiency, and applicability for the case of non-uniform intensity distribution [185], and the phase can be presented as:

$$ \phi \left(\mathbf{x} \right)=-k_{g}{{\nabla }^{-2}}\nabla \cdot \left[ {{I}_{g}^{-1}}\left(\mathbf{x} \right)\nabla {{\nabla }^{-2}}\frac{\partial I\left(\mathbf{x} \right)}{\partial z} \right] $$

(13)

where k_g indicates wave number (k_g=2π/λ_g), I_g is the intensity with green wavelength, and x=(x,y) is the lateral 2D spatial coordinate. ∇⁻² is the inverse Laplacian operator, denoting a simplified representation of the solution to the corresponding Poisson equation.

GS iterative phase retrieval algorithm starts from the retrieved phase of TIE, then the complex field is propagated back and forth between different planes, replacing the amplitude with the measurement amplitude $\sqrt {{I}_{r,g,b}}$ until the whole process converges, as shown in Fig. 27(d). In order to avoid the convergence of the algorithm stagnates after early iterations or even get trapped into local optima, the incremental gradient method can be used for the iteration process to improve the stability and robustness of the reconstruction towards noise by adopting the adaptive step-size strategy [227]. Finally, the complex amplitude retrieved from iterative refinement is propagated back to the object plane to produce the image back into focus, leading to sharp in-focus intensity and phase with all out-of-focus blurring effectively removed. Similarly, the reconstructed phase can also be used to generate digital DIC results showing the phase gradient image of the specimen.

Experiment results

∙ high-throughput quantitative phase imaging

In order to verify the imaging resolution of our HTL-SCLM, a phase resolution target QPT^TM with the height of 350nm was measured as a specimen, and the experimental results are shown in Fig. 28. Figure 28(a) shows the intensity pattern directly captured by the sensor. Though the object to sensor distance is relatively small, the defocus blurring can be easily observed in the magnified regions of the raw images. The precise defocus distance can be determined by the digital refocusing function of the software and applied for iterative reconstruction. After the TIE phase retrieval and several iterations phase refinement, the reconstructed quantitative phase shown in Fig. 28(d) can be obtained. Compared to the severe defocus artifacts in direct image acquisition, the reconstructed complex field leads to a sharp in-focus image with all defocus blurring effectively removed. The small group corresponding to the blue box in Fig. 28(d) is enlarged to display the distinguishable resolution element. As shown in Fig. 28(e), our imaging system correctly recovered the unit of Group 9 Element 2, which corresponds to the highest achievable resolution of 870nm (half-pitch resolution), exceeding the resolution-limited by the pixel size 1.67μm. The phase values along the red line are then extracted and plotted in Fig. 28(f) to quantify the lateral resolution, which gives a direct demonstration of the achievable resolution of our HTL-SCLM system. In addition, this high-resolution quantitative phase can be further used to generate phase gradient images of the specimen along an arbitrary shearing direction by digital DIC, as shown in Fig. 28(c).

Fig. 28

Experiment for imaging verification by standard phase resolution targets QPT^TM. (a) Original captured image. (b) The enlarged image of the sub-region of interest in original captured image. (c) Phase gradient image generated from reconstructed quantitative phases by digital DIC method. (d), (e) Reconstructed quantitative phase and the enlarged phase of the sub-region. (f) Quantitative phase profile along the red line in (e)

Our HTL-SCLM achieves both a wide FOV and high imaging resolution, which provides an effective means for high-throughput imaging of label-free cells. To demonstrate the imaging performance of our HTL-SCLM system on biological specimens, we conducted an experiment on unlabeled HeLa cells. Figure 29(a) shows the reconstructed quantitative phase of HeLa cells with a full FOV of 29.8474 mm². A region of interest [within the yellow dashed box in Fig. 29(a)] is selected and enlarged for quantitative analysis of the cells, as shown in Fig. 29(b). Our system allows users to extract the quantitative data of the specimen (along the red line) and drawing its profile, thus we can get quantitative data on the morphology of cells. Any area within the wide FOV can be magnified for further observation and detection of cells. As shown in Fig. 29(c1)-(c4) and 29(d1)-(d4), we extracted four sub-areas of reconstructed phase marked by four blue dashed boxes to display their enlarged versions and phase gradient images. It can be seen that the structure details of HeLa cells can be clearly observed. Due to the advantages of the quantitative phase in high contrast and high resolution, it also provides an effective data source for cell counting. From Fig. 29(a), a total of 8707 cells are presented in the full FOV.

Fig. 29

High-throughput reconstructed quantitative phase of HeLa cells. (a) Reconstructed quantitative phase of HeLa cells in full FOV of 29.8474 mm². (b) The enlarged image of Area5 in (a) and the quantitative profile of cells. (c1)-(c4) The enlarged images of Area1, Area2, Area3, Area4 in (a). (d1)-(d4) Phase gradient images of Area1, Area2, Area3, Area4 generated from reconstructed quantitative phase by digital DIC method

∙ digital refocusing

In the absence of imaging optics, we can not acquire in-focus intensity images of the specimen in lensless microscope. To overcome this problem, an indispensable step for image reconstruction is to back propagate the acquired wavefront from sensor plane to object plane to get a sharp in-focus image. We employed the soybeans stem crosscut and dicot root to demonstrate the reconstruction results with and without digital refocusing, and the reconstructed intensities are displayed in Fig. 30. Compare Fig. 30(b1), (d1) and 30(b2), (d2), the reconstructed results with digital refocusing have clearer edges and details than those of without digital defocusing. This flexible ability to digitally adjust the ‘z’ parameter significantly extends depth-of-field (DOF) of our platform compared to other imaging systems where conventional microscope objectives are used.

Fig. 30

Comparison of reconstructed intensities of soybeans stem crosscut and dicot root without and with digital refocusing. (a) The raw captured diffraction image over the full FOV of soybeans stem crosscut without digital refocusing. (b1) The sub-region image of the raw captured image in (a). (b2) The sub-region image of the reconstructed intensity of soybeans stem crosscut with digital refocusing. (c) The raw captured diffraction image of dicot root over the full FOV without digital refocusing. (d1) The sub-region image of the raw captured image in (c). (d2) The sub-region image of the reconstructed intensity of dicot root with digital refocusing

Computational light microscope: software development and hardware implementation

Software development

Microscope software of SCLMs combines microscope, digital camera and accessories into one fully integrated solution. With an intuitive user interface, it guides the user through any workflow, whether fast image acquisition, computational reconstruction, or sophisticated analysis.

∙ software architectural pattern

We exploit the Model-View-ViewModel (MVVM) architectural pattern (as shown in Fig. 31) to separate data algorithm and display view, thus the low coupling allows us to modify either of these independently without recompiling other module code. ViewModel realizes bidirectional binding of the data algorithm and display view, and it can catch operation changes of the view interface through data monitoring and then inform the data algorithm to respond. Besides, it can inform the interface view to update when the data changes. User settings are transmitted to the Model as input of the ViewModel through the event loop, and corresponding algorithm instructions are performed to achieve serial port transmission, data readout, phase calculation, cell segmentation, graphical user interface (GUI) thread, etc. The resulting algorithm results and data are saved in memory and are transmitted as updated data to the ViewModel and notice View to display the corresponding results.

Fig. 31

Model-View-ViewModel (MVVM) framework mode

∙ software development framework

The Windows operating system C++ graphical user interface application development framework QT is used to build the user interface, at the same time combined with the Open Source Computer Vision Library (OpenCV) for algorithm implementation and image processing and the Open Graphics Library (OpenGL) for 2D/3D rendering graphics. When running the software, the human-computer interaction can be realized through the interface operations, and the information interaction between the software and the algorithm is implemented through the signal slot function. The software refreshes the calculation data in a continuous event loop, thus the high-speed refresh enables the software system to shoot and execute algorithms in real-time. In addition, GPU, which is a highly parallel, multi-threaded, multi-core processor with powerful computing power and very high memory bandwidth, is adopted to perform many time-consuming calculations, such as unwrapping, discrete Fourier transform, etc.

The GPU accelerator is implemented through the language editing model of the Compute Unified Device Architecture (CUDA), which is a parallel computing platform and programming model. Figure 32 shows the schematic diagram of calculation process of CUDA. It provides programming interfaces for other programming languages, including C, C++, Java, Python, etc. The organization of threads in CUDA is in the form of blocks, each of which can have up to 1024 threads, with several blocks forming a grid. In the actual execution, the CUDA device assigns blocks to different streaming multiprocessors (SM) for parallel computation. From Fig. 32, all threads in a block will distribute into the basic execution unit of the wrap. In CUDA, the size of the wrap is specified as 32, which indicates 32 threads are grouped into a single warp to be executed together. And all threads in a warp execute the same instructions but process different data. In SM, multiple streaming process (SP) is employed to execute different threads in parallel. In order to take full advantage of the parallel performance of the GPU to maximize acceleration of algorithms, our software adopted some optimization strategies: (a) excessive data transfer between host and device is avoided as much as possible; (b) the number of threads in a block is set to an integer multiple of 32; (c) shared memory is preferred for communication between threads in the same block. Besides, to make it compatible with non-NVIDIA GPU devices, the software also uses Open Computing Language (OpenCL) to achieve parallel acceleration of the algorithm.

Fig. 32

The schematic diagram of calculation process of Compute Unified Device Architecture (CUDA)

∙ software functional modules

All softwares of our four SCLMs are designed to be the same user interface style. The software of MQP-SCLM is taken as an example to introduce their functional modules. As shown in Fig. 33, we marked the functional modules of the interface using a numerical sequence, and they will be introduced in the following. ① The imaging module provides the flexibility to switch between multiple imaging modes. The different imaging approaches are switched by selecting different buttons to achieve bright field, dark field, DPC, QPI, etc., while the parameter adjustment interface of the illumination module changes in response. ② The objective selection module is used to indicate the currently used objective and to provide the imaging algorithm with the accurate parameters of the objective lens. Once the objective is selected, its parameters will be automatically rewritten and transmitted to the imaging algorithm. ③ The illumination setting module automatically switches with the conversion of the imaging modes. According to the illumination requirements of different imaging modes, several customized illumination adjustments are set in the panel. The microscope illumination parameters (such as illumination aperture, illumination color, etc.) can be flexibly adjusted with one button. ④ The middle of the interface is the image display module that shows the results of microscopic imaging in real-time and provides measurement and editing functions. It can realize multimodal imaging display, that is, simultaneously display a plurality of different sub-windows whose image source from different imaging modes, and can be controlled to display dynamically or statically. Each sub-window can be independently controlled to realize real-time display, image saving, profile extraction, image interception display, and other functions. ⑤ The diverse adjustment and post-processing functions are included in the software to assist in denoising or enhancing the display of the imaging results. ⑥ When the software starts, the software automatically identifies the connected serial port and opens it to realize the interactive control between the software command and the hardware FPGA. ⑦ After selecting the imaging mode, the rightmost of the interface shows the camera parameter setting docking window, which provides basic parameter settings such as exposure, white balance, and external triggering. It also supports the automatic loading of camera parameters and resets. ⑧ Some auxiliary detection functions, such as drawing quantitative phase profile, cell counting, are placed in the lower right corner of the software interface.

Fig. 33

Software interface, module, and functions

Hardware design implementation

Drive control circuit

The computational microscopes acquire images with different imaging modes by precisely adjusting the illumination patterns and further reconstruct the required information of the specimen based on specific imaging algorithms. To provide high-quality record images, our SCLMs employ small-size, high-power programmable full-color LED as illumination sources to obtain uniform images under a short exposure. Normally, we need to implement various structured patterns as the light source and control the camera to capture the images under certain illumination situations in an orderly manner.

To realize the goal mentioned above, it is inseparable from designing a general-purpose control unit for SCLMs. The control unit needs to drive the LED array to generate specific illumination patterns and ensure that functions such as defocus control, image transmission, and microscope system parameter settings can be stably and collaboratively performed. The composition framework of the circuit is shown in Fig. 34. The printed circuit board (PCB) adopts a double-sided and four-layer design. The power interface is connected to a 5V DC power and is responsible for the entire circuit’s power supply. The logic circuit part includes the USB Hub circuit and the FPGA circuit.

Fig. 34

The control unit for SCLMs. (a) The components of the control unit PCB. (b) The resistor transistor logic circuit for LED array driving. (c) Control sequence of an LED array

The USB Hub circuit’s function is to simplify the link between the microscope and the host computer, and all the parts of the microscope can be controlled through the PCB by only a USB cable. The USB hub has three sub-interfaces on the control unit side. The first sub-interface is connected to the FPGA through a USB-to-serial module and transmits commands based on the serial port protocol. The second sub-interface is connected to an ETL drive to adjust the defocus distance in TIE-associated applications. The third sub-interface is directly connected to an industrial camera, which builds an image transmission link from the camera to the host computer.

The FPGA works as the central controller of the PCB, and its logic function is implemented based on the Verilog language. Once the power is turned on, the FPGA automatically reads the configuration data stored in a flash and renders the FPGA to build a register transfer level (RTL) circuit architecture as shown in Fig. 34(b). The input of the framework contains the objective switch, serial command receiver, and the system clock. The framework’s output contains a serial command sender, a camera trigger, and an LED driver. The internal circuit runs the following functions: decode the serial command sent by the host computer based on a self-defined protocol, generate the pattern for LED array to display, synchronously trigger capture signal for the camera, drive the LED array display generated pattern according to the signal sequence diagram in Fig. 34(c), and switch the objective according to the user’s command. Each module operates in the same clock domain, executes in parallel, and responds in real-time.

Telecentric axial scanning system

In the MQP-SCLM, the ETL is used to construct a telecentric axial scanning system to automatically control the defocus distance to acquire images at multi planes. Essentially, the telecentric axial scanning system is a telecentric relay optical path, which consists of a pair of relay lenses L1, L2, and ETL. It is located after the primary imaging plane of the infinite correction light microscope system to switch the focal length quickly without mechanical adjustment. As shown in Fig. 35(b), the ETL is located in the Fourier plane of L1 and L2 to form 4-f imaging system. The advantage of the 4-f configuration is that this arrangement does not produce optical scaling, that is, the scanning of the focal plane position of the object can be realized by changing the focal length of the ETL. From Fig. 35(c), L1 relays the back focal plane of the objective lens to the through aperture of the ETL to adjust the focal length, and then L2 restores the final image to the secondary imaging plane.

Fig. 35

Optical design of the MQP-SCLM. (a) Optical path of the MQP-SCLM. (b) General light tracing diagram of the system. (c) Light tracing diagram of the system

In the hardware configuration, we employ an ETL (Optotune AG, EL-16-40-TC) with a diopter adjustable range from -2D to 3D and an aperture of 16mm to achieve bidirectional defocus. It is driven by the ADN8810 programmable precision current source embedded, which is embedded with a precision 12-bit digital to analog convertor (DAC), one precision V/A converter, and a high precision current output. Such a current drive circuit is compatible with the control circuit. The ADN8810 source is connected to the computer through a serial interface, and its current is controlled by the computer. Due to the capability of backhaul scanning, there is no step in the current control process. The ETL can switch to the next axial position only with a response time of 5ms. Thus, high precision and fast axial scanning can be realized by strictly calibrating the current/diopter.

Conclusion and discussion

By now, we believe the readers should have a more in-depth understanding of “computational light microscopy”. Of course, only through written text and figures, one may not be able to perceive the charm of these computational light microscopes comprehensively. Therefore, at the end of this article, we further provide a supplementary video (see supplementary information) to visualize the principles and configurations of these four microscopes, and meanwhile, to demonstrate their functions and performance more intuitively. By dragging the sliding bar on the software control panel, one can flexibly adjust the focus of the digital holographic microscope through the numerical refocusing algorithm. With a simple mouse click, subcellular details almost invisible in bright field illumination can be clearly visualized in dark field imaging mode. By rotating the rainbow ring on the software control panel, the specimen is optically stained in brilliant colors by the iridescent Rheinberg illumination. Under the rapid axial scanning of the electronically tunable lens, the high-resolution phase distribution of unstained cells is accurately reproduced by the annular-illumination TIE, and their 3D morphology instantly appears on the screen. By moving the light source pattern on the illumination panel, the spine and burr on the fly’s wings can be observed from different perspectives in the light field imaging mode. Under the sequential illumination of the tricolor LEDs, tens of thousands of unstained HeLa cells are clearly revealed in a FOV of 30mm², just like infinite stars dotting the vast night sky. We are sure one will be impressed with the functions and performance of these microscopes through the video demonstration. The marriage between computational imaging and light microscopy brings unprecedented enhancements in their functions and performance, such as flexible multi-mode imaging, non-interferometric quantitative phase imaging, and high-throughput giga-pixel imaging capabilities that were never achieved before. These smart computational optical microscopies are expected to provide high-resolution, high-sensitivity, and convenient label-free tools for life sciences, medical diagnosis, drug development, pathology, industrial inspection, etc. The lensless microscope may provide a convenient and inexpensive POCT tool in resource-limited areas, significantly reducing the threshold of medical diagnosis. However, it has to be acknowledged that computational light microscopy still faces a number of problems and challenges that need to be solved. Due to space limitations here, we list the four most critical aspects below:

• (1) Difficulties in accurate image formation modeling In a microscopic system, the image field is generally formed as electromagnetic waves or a wavefront, transmitted by an optical system, which can be accurately described by wave optics. However, a full wave optical treatment of physical image formation is rarely possible. The high dimensions of specimen information and the complexity of optical systems typically do not allow for a rigorous solution of Maxwell’s equations. Therefore, the image formation is typically modeled based on certain levels of approximations [251, 264]. However, such an approximated formulation may not be able to cover all the effects that take place. Moreover, there are various measurement uncertainties (noise, vibration, etc.) that can affect the image formation process, leading to additional model mismatches. Even with full knowledge of these influential factors, one may still face the challenge that a simplified model may not be able to produce accurate results, while a more realistic model may involve a large number of system parameters, making their solution an impossible task.

• (2) Bottleneck in optical modulation device Although termed as “computational” light microscopy, the recent and rapid progress of this area has been driven by the advent and development of “physical” light modulation devices. While currently available programmable optics, such as LCDs, DMDs, and LEDs, have made flexible spatial/angular light modulation possible, the “tunable parameters” they have enabled so far are only the tip of the iceberg of all possible light parameters that can be modulated. Further development of computational light microscopy relies on more sophisticated optical modulation mechanisms to acquire multi-parameter, multi-dimensional data, which also requires more sophisticated and powerful optical modulation devices. However, most of the existing light modulation devices can only realize single-parameter, low-dimensional optical modulation and cannot realize high-dimensional multi-parameter joint modulation (such as amplitude-phase-wavelength-polarization joint modulation). Their optical modulation quality factors (e.g., resolution, contrast, fill factor, diffraction efficiency, response speed, linearity, etc.) is also limited by the current semiconductor manufacturing process. In addition, the size, weight, and cost associated with the optical modulation devices [e.g., liquid crystal on silicon (LCOS) and DMD] prevent their wide adoption in many applications. Recent rapid progress in optical metasurfaces provides a thin and light-weight alternative to conventional bulky optical elements by manipulating light scattering via resonant nanostructures [265, 266]. They are designed as a new type of optical modulation device that allowing to modify different characteristics of light such as its wavefront, polarization, intensity, or spectrum. However, the inherent diffractive nature of metalenses induces severe chromatic aberrations when imaging under broadband illumination. Despite several attempts at realizing achromatic and dispersion-engineered metasurfaces, the operation bandwidths, sizes, and numerical apertures of such devices are still quite limited. As a result, there is still a long way for metasurface optical components toward industrial realization.

• (3) Fidelity of image reconstruction algorithms In computational light microscopy, the final image is indirectly reconstructed from the raw intensity measurements using computational reconstruction algorithms, which raises questions about the validity and reliability of reconstructed images. Experimental results among the literature reveal that there is still a gap in image quality and fidelity between the images reconstructed by computational imaging algorithms and those captured directly by conventional microscopes. The fundamental reason lies in the fact that computational light microscopy requires that the mathematical model of the optical system can describe the actual physical imaging process in a realistic and comprehensive manner. If the model fails to reflect the properties and complexities of the optical system realistically, computational light microscopy may not yield valid and reliable reconstruction results. This issue may be critical in many applications such as industrial inspection, medical diagnosis, and scientific research: it is not only necessary to get a good-looking result, but also need to make sure that the result is accurate, reliable, repeatable, and traceable. Consequently, when designing a computational imaging algorithm, the establishment conditions of the forward image formation model, the complexity of the inverse reconstruction algorithm, the sensitivity of the measurement process to noise and environmental perturbations, and the level of introduced artifacts need to be considered in a comprehensive manner.

  Recently, the rapid development of deep learning technology has opened a new window for the development of computational light microscopy [110, 113, 115, 122, 267]. Different from the traditional method of “forward modeling and then solving the inverse problem”, deep learning directly establishes a “pseudo forward model” from the image to the desired specimen information, taking the original images as the “network input” and the expected information as the “network output”. Consequently, the obstacle of “solving nonlinear ill-posed inverse problems” can be bypassed, and information extraction and reconstruction can be realized directly. However, the success of the deep learning technology depends on the “common” features learned from a large number of training samples, which may lead to poor results when facing “rare samples”. Though everyone hopes that deep learning approaches can produce a provably correct solution, no one can guarantee at present, at least not yet. Therefore, when we embrace the great success of the current deep learning technology, we should keep a clear and rational attitude towards its applications in “high-risk” scenarios.

• (4) Challenges in data volume and computing power As the saying goes, “there is no free lunch”, there are two sides to everything, and the same is true for computational light microscopy. The improvement in function and performance that can be achieved with computational light microscopy often comes at the cost of complicated system hardware, extensive additional data acquisition, and complex and time-consuming data processing. From the perspective of information theory, computational light microscopy follows the law of information conservation. The post-processing algorithm is essentially a kind of information extraction and re-integration, with the gain in additional information and imaging performance coming from the diversity and multi-dimensionality of the acquired data. Many of the current problems in computational light microscopy are essentially high-dimensional information retrieval from a large number of low-dimensional raw measurements. The continuous pursuit of high resolution, deep penetration, wide FOV, and fast imaging speed in high-dimensional computational light microscopy methods such as 3D tomography, 4D coherent retrieval, and full-resolution light-field imaging, has produced an overwhelming volume of data, exceeding the ability of the imaging system and computing platforms for data generation, capture, processing, analysis, and storage rapidly and efficiently. Emerging scalable storage and computing architectures, including parallel computing [268, 269], distributed computing [270, 271], and cloud computing [272, 273], may provide promising solutions for tackling these challenging problems in data volume and computing power.

Additional file 1: Supplementary video.

Can computational light microscopy overturn the traditional microscopic imaging mode of “What You See Is What You Get (WYSIWYG)” and open a new door for future light microscopy? Even if we cannot say that this is an inevitable development of light microscopy, at least from our bold attempt shown here, the results are auspicious, and we are fully confident about its future. We expect that through the joint efforts of academia and industry, in the near future, computational light microscopy will play an indispensable role in life science, biomedicine, and industrial applications, producing more substantial and significant impacts on the surrounding areas.