Skip to main content

Continuous optical zoom microscope with extended depth of field and 3D reconstruction

Abstract

Microscope such as fluorescence microscope, confocal microscope and two-photon microscope plays an important role in life science, laser processing and other fields. However, most microscopes only have discrete zoom rates. In this paper, a continuous optical zoom microscope with extended depth of field and 3D reconstruction is demonstrated for the first time. It consists of a zoom objective lens, a microscope holder, an adjustable three-dimensional object stage, an Abbe condenser and an LED light source. The zoom objective lens is composed of several liquid lenses and solid lenses. By adjusting the applied voltage to the liquid lens, the proposed microscope can achieve a large continuous magnification from 10? to 60?. Moreover, an improved shape from focus (SFF) algorithm and image fusion algorithm are designed for 3D reproduction. Based on the liquid lenses, the axial focusing position can be adjusted to obtain images with different depths, and then the extended depth of field and 3D reconstruction can be realized. Our experimental results demonstrate the feasibility of the proposed microscope. The proposed microscope is expected to be applied in the fields of pathological diagnosis, biological detection, etc.

Introduction

Microscopes are one of the most basic types of equipment in the fields of biomedicine, micro-manufacturing, industrial inspection and other fields of science and technology 1,2,3,4,5,6,7,8,9. The traditional optical microscope employs discretely fixed focus objectives to change the magnification. The switching of the magnification requires manual or electronic rotation of the objective lenses, which inevitably brings about sample vibration and has a long response time.

In recent years, liquid lenses have been widely used in optical imaging systems due to the advantages such as integration, adaptive zoom and low consumption 10,11,12,13,14,15,16,17,18,19,20,21. Especially, a series of liquid lenses are applied in microscopes or microscopic imaging systems to achieve the functions such as axial scanning 22,23,24,25,26, aberration correction2728, autofocusing 29,30,31, etc. 32,33,34,35,36. In 2016, a zoom microscope objective using electrowetting lenses was designed, of which the magnification can be just tuned from ~ 7.8× to ~ 13.2× but the resolution was limited 37. In 2021, a continuous optical zoom microscopy imaging system with axial depth scanning was designed. Although the magnification was enhanced to 22.2×, continuous optical zoom with high magnification was still difficult to achieve 38.

Moreover, a microscope with extended depth of field and 3D reconstruction can acquire a large range of clear images at one time and obtain the 3D morphology of the samples 39,40,41. This kind of microscope can be used not only for pathological diagnosis, but also for industrial inspection to judge the appearance defects of the samples. In 2018, researchers proposed a monocular wide-field optical microscopy with extended depth of field for accurate 3D reconstruction of microsamples 39. The proposed system has a high relative measurement accuracy of 1.9%. However, the reconstruction process involves rotating the mirror through 360°, therefore it takes a long time to acquire the images. In 2018, researchers proposed a nonmechanical and multi-view 3D measurement microscope for workpieces with large slopes and complex geometry 40. The proposed microscope is lateral shift-free, has no light intensity variation and can obtain an accuracy of 2.4 μm. But this microscope only works at a fixed magnification and the reconstructed samples cannot be observed in detail. Therefore, there is an urgent need for a microscope that can quickly acquire images with different depths of field and has a continuous zoom function.

To the best of our knowledge, a continuous optical zoom microscope with extended depth of field and 3D reconstruction is demonstrated for the first time. The proposed microscope mainly consists of a zoom objective lens and an adjustable three-dimensional object stage. The zoom objective lens is composed of several liquid lenses and solid lenses. By adjusting the voltage to the liquid lenses and using the improved image algorithm, the proposed microscope can realize large range continuous zoom imaging as well as extended depth of field. Moreover, 3D reconstruction can also be achieved without moving object stage or objective lens. The key novelties of the proposed microscope include: (1) high-resolution imaging with large continuous magnification from 10× to 60× is realized, so the sample in richer details can be observed; (2) the proposed microscope can carry out both zoom extended depth of field and 3D reconstruction only employing liquid lenses, which has not been reported before; (3) the improved shape from focus (SFF) algorithm and image fusion algorithm are designed to cooperate with the adjusting mode of the liquid lenses to realize fast and high quality imaging. Our experiments demonstrate the feasibility of the design and the proposed microscope can be applied in the fields of biological detection, laser processing, and chemical detection.

Structure and method

Structure of the proposed microscope

Figure 1(a) shows the schematic structure of the proposed microscope. The proposed microscope consists of a zoom objective lens, a microscope holder, an adjustable three-dimensional object stage, an Abbe condenser and an LED light source. The zoom objective lens is used to magnify the microstructure of the real sample. The image magnified by the zoom objective lens is captured with a high-resolution complementary metal–oxide semiconductor (CMOS) camera.

Fig. 1
figure 1

Schematic structure of the (a) proposed microscope and (b) the zoom objective lens

Method of the secondary transfer zoom objective lens

The zoom optical system can change focal length while keeping the image plane stationary. The most common way is the mechanical compensation zoom system which changes the focal lengths of the system by changing the distance among the solid lenses of the zoom group. Nevertheless, in this work, the adaptive zoom is achieved by employing the liquid lenses instead of adjusting the relative displacement between the solid lenses. Each liquid lens consists of conductive liquid and non-conductive liquid and is actuated by electrowetting effect. The liquid lens cavity is coated with a dielectric layer and a hydrophobic layer. The dielectric layer ensures that the liquid lens is not broken down by the external voltage. The hydrophobic layer ensures that the conductive liquid has a large initial contact angle. The contact angle between the conductive liquid and the cavity can be modulated by an external voltage. The change of contact angle results in the change of the curvature of the liquid-liquid interface. According to the Young-Lippmann equation, the relationship between the contact angle θ1 and the external voltage U can be described as follows 42:

$${cos}{\theta }_{1}={cos}{\theta }_{0}+\frac{{U}^{2}\epsilon }{2D{\gamma }_{12}}$$
(1)

where θ0 is the initial contact angle without applied voltage, ε is the dielectric constant of the dielectric layer, D is the thickness of the dielectric layer and γ12 is the surface tension between the conductive liquid and the non-conductive liquid.

In the process of operation, the liquid lenses not only have to withstand a wide range of focal power, but also ensure high magnification. However, a liquid lens with large aperture is difficult to fabricate due to the limitation of the contact angle. The small aperture of the liquid lens limits the transmission of light in the microscopic system, resulting in the inability to achieve large zoom ratio and large numerical aperture simultaneously. In order to solve this problem, we propose an optical design method called secondary transfer design. For details, the zoom objective lens is innovatively designed with two parts: the front zoom group and the rear zoom group. Figure 1(b) shows the sectional views of the optical design and mechanical elements. The arrangement structures of the front and rear zoom groups both consist of a front fixed part, a zoom part and a rear fixed part. The fixed part is composed of solid lenses and the zoom part is composed of liquid lenses. The front zoom group is used to enhance the magnification of the proposed microscope, and the rear zoom group bears most of the focal power of the proposed microscope. In the microscopic observation process, the sample is magnified by the front zoom group and imaged into the relay image plane, which is also the sample plane of the rear zoom group. Then, the relay image is captured by the proposed microscope after secondary amplification by the rear zoom group. The magnification of the proposed microscope β can be expressed by the following equations:

$$\beta ={\beta }_{f}\times {\beta }_{c}$$
(2)
$${\beta }_{f}=(1+{\varPhi }_{f}\times u{)}^{-1}$$
(3)
$${\beta }_{c}=(1+{\varPhi }_{c}({\varPhi }_{f}+{u}^{-1}{)}^{-1}-{\varPhi }_{f}\times d{)}^{-1}$$
(4)

where βf and βc are the magnifications of the front zoom group and the rear zoom group, respectively, Φf and Φc are the focal power of the front zoom group and the rear zoom group, respectively, u is the working distance of the zoom objective lens, and d is the central distance between the equivalent principal plane of the front and rear zoom groups. The liquid lenses can not only play a significant role of the zoom function, but also keep the image plane fixed during the zoom process, which means u and d remain constant. The magnification of the proposed microscope is only related to the focal power of the front and rear zoom groups. The focal length of the front zoom group is expressed as follows:

$${\varPhi }_{c}={\varPhi }_{1}+{\varPhi }_{2}+{\varPhi }_{3}-{d}_{1}{\varPhi }_{1}{\varPhi }_{2}-{d}_{1}{\varPhi }_{1}{\varPhi }_{3}-{d}_{2}{\varPhi }_{2}{\varPhi }_{3}+{d}_{1}{d}_{2}{\varPhi }_{1}{\varPhi }_{2}{\varPhi }_{3}$$
(5)

where Φ1 is the focal power of the front fixed part, Φ2 is the focal power of the zoom part, Φ3 is the focal power of the rear fixed part, d1 is the distance between the front fixed part and the zoom part, and d2 is the distance between the zoom part and the rear fixed part. The focal power of the proposed microscope undertaken by the solid lenses remains unchanged, and the distance between the zoom part and fixed part remains the same. The expression of the focal power of the rear zoom group is consistent with that of the front zoom group. Therefore, the magnification of the proposed microscope can be modulated by changing the focal lengths of one or more liquid lenses in the front and rear zoom groups.

Methods of the extended depth of field and 3D reconstruction

The depth of field of an optical microscope is inversely proportional to the resolution. The acquisition of high-resolution images will inevitably lead to a reduction in the depth of field. When the thickness of the sample is greater than the depth of field, a complete image of the sample cannot be obtained. The traditional methods generally rely on the mechanical movement of the object stage to obtain images at different depths of the sample. Then, image fusion is used to expand the depth of field, which presents the complete morphology of the sample 43,44,45. In this work, the proposed microscope can scan the axial depth of the sample by modulating the focal length of the liquid lens.

Adjusting the voltage of a single liquid lens for axial scanning results in a change in magnification. The synergistic adjustment of multiple liquid lenses in the proposed microscope enables higher accuracy of depth scanning while maintaining a constant magnification. Then, the image fusion algorithm based on the Laplacian pyramid is applied to fuse the scanned images with different depths into an all-focus image. The algorithm decomposes each source image into n layer sub-images. This process can be expressed as follows:

$$\left\{\begin{array}{c}{L}_{i}={G}_{i}-UP\left({G}_{i+1}\right)\otimes {g}_{5\times 5}(1\le i\le n-1)\\ {L}_{n}={G}_{n}\end{array}\right.$$
(6)

where Girepresents the sub-image of the ith layer of the Gauss pyramid, Li represents the sub-image of the ith layer of the Laplacian pyramid, UP(Gi+1) represents the upsampling operation on the sub-image of the (i + 1)th layer on the Gaussian pyramid, ς denotes the convolution operation, and g5 × 5 indicates the Gaussian convolution kernel with the window size of 5 × 5. Next, the sub-images of each layer are fused according to the fusion rule of the modified Laplacian (ML):

$$M{L}_{i,k}(x,y)=\left.\left|2{L}_{i,k}(x,y)-{L}_{i,k}(x-step,y)-{L}_{i,k}(x+step,y)\right.\right|+\left.\left|2{L}_{i,k}(x,y)-{L}_{i,k}(x,y-step)-{L}_{i,k}(x,y+step)\right.\right|$$
(7)
$${L}_{i}(x,y)={L}_{m,i}(x,y)$$
(8)

where MLi,k(x, y) represents the ML of the ith layer image on the Laplacian pyramid, k represents the number of frames in the image, 0 ≤ x < row, 0 ≤ y < column, ‘step’ represents the window size of the ML operator, and m is equal to the number of the image frames where MLi,k(x, y) is the maximum value. Appropriately increasing the window size can suppress noise interference and improve the fusion image effect. Finally, the fused image with large depth of field is reconstructed by the reverse process.

According to the above source images at different focal depths, the shape from focus (SFF) algorithm is improved to reconstruct the 3D structure of the sample. The schematic of the 3D reconstruction algorithm is shown in Fig. 2. When the microscope is focused at different depth planes, focused portion of the sample can be obtained. Defocus image is equivalent to the low-pass filtering for focused image. The larger the defocus degree is, the more high-frequency components are filtered out. Therefore, the defocus degree can be calculated by using an operator sensitive to high-frequency components. In this work, we use sum-modified-Laplacian (SML) as the focus evaluation operator to process the source image I(x, y), and select the appropriate size step according to the texture of the sample under test:

$$F(i,j)={\sum }_{x=i+N}^{i+N}{\sum }_{y=j-N}^{j+N}ML(x,y)forML(x,y)\ge T$$
(9)

where ML(x, y) represents the ML of the source image I(x,y), F(i,j) is the focus evaluation operator, N determines the window size used to calculate the focus measure, and T is the custom threshold. The functional relationship between the pixel focus degree Fi,j(k) at the coordinate (i,j) and the number of image frame k satisfies the Gaussian distribution. The peak value of Fi,j(k) and two points near the peak value are Gaussian interpolated to obtain more accurate depth values. To sum up, according to the focus information of the image, the depth information Z corresponding to each coordinate point (x,y) in the sample can be obtained. By using the full-focus image obtained by the mentioned above image fusion algorithm as the texture image, the 3D model of the sample can be acquired.

Fig. 2
figure 2

Schematic of the 3D reconstruction algorithm

Fig. 3
figure 3

Simulation results of the zoom objective lens at the magnifications of 10×, 20×, 30×, 40×, 50×, and 60×

Simulation, experiments and results

In order to verify the feasibility of the proposed microscope, optical experiments are carried out. The mechanical structure of the proposed microscope and the solid lenses are fabricated by DaHeng New Epoch Technology Inc. The CMOS used in the experiments is the type of BFS-U3-32S4C-C produced by FLIR, CAN. The size of the CMOS is 1/1.8ʺ and the pixel size is 3.45 μm. The highest resolution is 2048 pixels×1536 pixels. The liquid lenses are produced by Corning, US with the type of Arctic-39N0, whose useful aperture is 3.9 mm and the optical power can be changed from − 15 D(m− 1) to 20.1 D(m− 1). The maximum numerical aperture of Abbe condenser is 1.25.

Simulation of the zoom objective lens

Using the commercial software Zemax, we simulate the front and rear zoom groups respectively. According to the design requirements, each component of the initial structure is normalized, and the focal length and interval of each component are reasonably allocated. In order to obtain a perfect combination, two cooperating optimization operands are constructed to accomplish the following purposes: (1) the angle of emergence in the front zoom group is consistent with the angle of incidence in the rear zoom group; (2) the exit pupil of the front zoom group is at the same position as the entrance pupil of the rear zoom group; (3) the aperture angle of the emergent ray in the front zoom group is smaller than the aperture angle of the incident ray in the rear zoom group. These conditions ensure that the front and rear zoom groups are stitched together perfectly.

After splicing the front and the rear zoom groups, a new merit function is constructed to optimize the radii of the seven liquid lenses, which aims to get the optimized solutions of different magnifications in Zemax. The 3D layout diagram of the proposed microscope obtained by the final simulation is shown in Fig. 3. The simulation results show that the magnification of the zoom objective lens can be tuned from ~ 10× to ~ 60×. We simulated the modulation transfer function (MTF) and the spot diagram at the magnifications of 10×, 30× and 60×, as shown in Fig. 4(a)-4(c).

Fig. 4
figure 4

MTF and spot diagrams of the zoom objective lens at the magnifications of (a) 10×, (b) 30× and (c) 60×

The MTF is given by three wavelengths of 0.486 μm, 0.587 μm, and 0.656 μm, respectively. The black line represents the MTF at diffraction-limited resolution. Axis, T and S represent the zero, tangential and sagittal fields of view, respectively. We plotted the relationship between resolution limit (dropping to 10% of maximum MTF) and magnification based on the simulation results, as shown in Fig. 5. As can be seen from the simulation results, with the decrease of magnification, the resolution limit of the center field of view (FOV) decreases slowly, while the resolution limit of the edge field of view decreases significantly. During the continuous zoom process, the spatial frequency can reach 400 lp/mm when MTF > 0.1 (the center field) and the spatial frequency can reach 150 lp/mm when MTF > 0.1 (the edge field), which mean that the zoom objective lens has high resolution. The root mean square (RMS) radius is smaller than the airy disk radius, which indicates that the zoom objective lens has reasonable high image quality. The limitation of the aperture size of the liquid lens prevents edge light from passing through the zoom objective lens, thereby reducing the spatial frequency of other fields. Aberrations can be balanced by increasing the aperture size of the liquid lens.

Fig. 5
figure 5

Relationship between resolution limit (dropping to 10% of maximum MTF) and magnification. The blue curve represents the central FOV resolution. The red curve represents the edge FOV resolution

Experiments of the proposed microscope

The proposed microscope is fabricated to evaluate the optical performance, as shown in Fig. 6(a). The microscope bracket is made of aluminum alloy by lathe milling. The magnification adjustment knob is installed to manually adjust the magnification. Below the proposed microscope is the three-dimensional object stage with a movement accuracy of 100 μm in the x and y directions and 1 μm in the z direction. The LED light source is installed on the bottom of the proposed microscope to provide transmissive light illumination from ~ 400 nm to ~ 760 nm.

The zoom objective lens is assembled by the front and rear zoom groups, as shown in Fig. 6(b). The front zoom group consists of four liquid lenses, three single glass lenses and a doublet lens. The rear zoom group consists of three liquid lenses, three single glass lenses and a doublet lens. They are tightened by built-in screw threads. The length and diameter of the zoom objective lens are ~ 220 mm and 25 mm, respectively. The effective imaging aperture of the commercial liquid lens is ~ 3.9 mm. The semi-diameter of the single and doublet lens is 5 mm. The space ring at the top ensures that the zoom objective lens can be fixed to the mechanical frame. A retractable compression ring structure is added in front of the first surface to prevent damage from contact with the sample.

Fig. 6
figure 6

Prototype of (a) the proposed microscope and (b) the zoom objective lens

We measure the focal length of the electrowetting liquid lens at different applied voltages. The negative focal length range of the liquid lens is (-∞, -67 mm) and the positive focal length range is (55 mm, +∞). According to the optimized solution of the liquid lens curvature in Zemax, we obtain the voltages applied to each liquid lens at different magnifications. Figure 7. shows the relationship between the magnification and the applied voltage. The magnification range is divided into high magnification area and low magnification area with a boundary of 30×. There are two liquid lenses which play the main role of adjusting magnification in the high and low magnification regions respectively. The other liquid lenses play the role of correcting aberrations and maintaining image plane stability. In addition, the transition from low to high magnification can be achieved by coordinating magnification with seven liquid lenses.

Fig. 7
figure 7

Voltages applied to seven liquid lenses at different magnifications. (LL: Liquid lens)

After carefully measuring, the focal length of the zoom objective lens can be changed from − 0.2 mm to -1.2 mm and the magnifications can be adjusted from ~ 10× to ~ 60×. To guarantee the continuous optical zoom of the proposed microscope, 100 sets of voltage values corresponding to 100 sets of magnifications between ~ 10× and ~ 60× are preset. We can adjust the magnification through the magnification adjustment knob on the microscope bracket or the self-developed liquid lens driving software.

In the first experiment, a resolution test chart (GCG-020602) is used to measure the resolution of the proposed microscope. The resolution test pattern conforms to the MIL-STD-150 A standard, set by US Air Force in 1951. The maximum resolution is 645 lp/mm and the minimum linewidth is 0.78 μm (Element-3, Group-9). During the continuous optical zoom process, we capture images at six magnifications of 10×, 20×, 30×, 40×, 50× and 60×, respectively, as shown in Fig. 8(a)-8(f). At the magnification of 10×, the fourth element of the resolution target Group-8 can be clearly captured and the modulation curve of normalized intensity is shown in Fig. 8(g), indicating that the resolution can reach more than 362 lp/mm. With the increase in magnification, the resolution of the proposed microscope is improved continuously. When the magnification is increased to 60×, the first element of the resolution target Group-9 can be clearly captured and the modulation curve of normalized intensity is shown in Fig. 8(h), indicating that the resolution is increased to more than 512 lp/mm. Different exposure times can cause small differences in image brightness. The exposure time of the COMS is modulated according to the light intensity required by different magnifications to ensure the consistency of brightness during continuous zoom. The slight difference in resolution between the pictures may be caused by the numerical aperture (NA) difference during continuous zoom. The experiment proves that the proposed microscope maintains excellent resolution throughout the continuous optical zoom process.

Fig. 8
figure 8

Captured images of the resolution target. (a) Magnification 10×. (b) Magnification 20×. (c) Magnification 30×. (d) Magnification 40×. (e) Magnification 50×. (f) Magnification 60×. The scale bars in these pictures are 20 μm. (g) The normalized intensity distribution of the fourth element of the Group-8 at 10×. (h) The normalized intensity distribution of the first element of the Group-9 at 60×

To further verify the practical applications of the proposed microscope in biomedicine, the cross section of “ascaris ovaries” is used as the observation sample. To achieve the continuous optical zoom, the liquid lenses are driven according to the voltages provided in Fig. 9. The dynamic response video of the image capture process is included in Media-1 (Additional file 1). Unlike traditional microscopes where target loss is caused by switching objective lenses of different magnifications, the sample remains clearly in the center of the field during continuous zoom. The response rate of the magnification switch is ~ 50 ms, which achieves real-time zoom observation in a real sense. The increase in magnification results in a smaller field of view, which reduces the brightness of the picture. Therefore, the exposure time of the CMOS camera is customized to keep the brightness of the images consistent. Six images at various magnifications are selected to facilitate observation, as shown in Fig. 9(a)-9(f). As continuously zoom in, the “ascaris ovaries” is gradually enlarged and the image remains sharp at each magnification. The liquid lens frame acts as the field aperture in the proposed microscope, which explains the low brightness of the edge field of view in Fig. 9(a). This issue can be ameliorated by using the liquid lens with larger aperture.

Fig. 9
figure 9

Captured images of the sample “ascaris ovaries”. (a) Magnification 10×. (b) Magnification 20×. (c) Magnification 30×. (d) Magnification 40×. (e) Magnification 50×. (e) Magnification 60×. The scale bars in these pictures are 20 μm

Unlike the traditional microscope, which results in sample loss due to switching magnification, the proposed microscope can always keep the observed sample in the visual field during zoom. This reduces the time delay associated with switching magnification to find the sample again. During capturing image process, the working distance does not need to be tuned, which reduces the vibration of the sample. The response time for switching magnification is measured to be ~ 50 ms, which is determined by our self-developed liquid lens drive board. Therefore, the proposed microscope can greatly improve the accuracy and real-time performance of microscopic observation.

Experiments of the extended depth of field and 3D reconstruction

As the magnification of the proposed microscope increases, the numerical aperture increases while the depth of field decreases. For a sample whose depth is greater than the depth of field, only part of the image can be focused and obtained. By adjusting the voltage to one certain liquid lens, the proposed microscope can carry out axial depth scanning without changing the magnification. Experiments show that different liquid lenses can be adjusted to achieve the same scanning depth, but the corresponding axial scanning accuracy is different. Therefore, a sample with depth difference by using 3D print technology is made to carry out a calibration. The stepped sample made of translucent photosensitive resin material has a height difference of 100 μm between each step. Experiments prove that adjusting the voltage of one liquid lens can achieve axial depth scanning, which will result in a change in magnification. One focused state needs at least one liquid lens to maintain constant magnification and another liquid lens to correct aberration. In addition, different liquid lenses can achieve different axial scanning ranges with the same voltage, which means that different axial scanning accuracies can be achieved by adjusting the same voltage of different liquid lenses. Table 1 shows the three groups of voltages corresponding to scanning a 100 μm axial depth under the magnification of ~ 30×.

Table 1 Voltages corresponding to scanning 100 μm axial depth

In the second experiment, a “spirogyra communis” sample is used to demonstrate the results of extended depth of field. Due to the inherent depth of field limitations of the proposed microscope, it is impossible to capture an individual “spirogyra communis” in one image. Under the magnification of ~ 10×, we perform continuous axial scans from the bottom to the top of the “spirogyra communis” sample by adjusting the driving voltage to the liquid lenses. Figure 10(a) shows the images of six different depths selected from the axial depth scan. The images are fused by means of the Laplacian pyramid image fusion algorithm. The fusion result is shown in Fig. 10(b). The proposed microscope also has the ability to extend the depth of field as the magnification changes continuously. The extended depth of field of the same sample at the magnification of 60× is shown in Fig. 10(c)-10(d). The comparison results show that all areas of the fused image are clearly focused.

In order to objectively evaluate the effectiveness of the fusion algorithm, the average gradient and standard deviation are selected to quantitatively evaluate the fusion images. Table 2 shows the data comparison of different depth images and fusion images at the magnification of 60×. The average gradient can be sensitive to the ability of the image to express the contrast of the details. The larger the average gradient of the fused image, the greater the contrast of tiny details will be shown in the image. The more obvious the texture is, the higher the resolution of the fused image is. The standard deviation reflects the degree of dispersion between the pixel value and the mean value of the image. The larger standard deviation indicates that the fusion image has a higher level of information richness.

Table 2 Data comparison between different depth images and fusion images
Fig. 10
figure 10

Extended depth of field images of the sample “spirogyra communis”. (a) Image sequences of the “spirogyra communis” from the bottom to the top at the magnification of 10×. (b)Result of the “spirogyra communis” image fusion at the magnification of 10×. (c) Image sequences of the “spirogyra communis” from the bottom to the top at the magnification of 60×. (d) Result of the “spirogyra communis” image fusion at the magnification of 60×. The scale bars in these pictures are 20 μm

The sample observed by an optical microscope at high magnification using transmission illumination is usually sliced. In the sample “spirogyra communis”, the depth information of the transparent part cannot be accurately judged. Therefore, threshold processing is used to extract the “spirogyra communis” tract from the image which has been fused before. The SFF algorithm mentioned above is used to calculate the depth information of the “spirogyra communis” at the magnification of 10×. Then the depth information is converted into a false color map, as shown in Fig. 11(a). The low-texture and high-exposure areas of the sample will lead to errors in the depth calculation. Because the depth calculation process is based on sharpness, errors can also be introduced by the inadequacy of the resolution. The extended depth of field image can provide texture and color information for the 3D model. The three-dimensional view map is shown in Fig. 11(b). By the same procedure, we obtained the false color map and the three-dimensional view map of the image captured at the magnification of 60×, as shown in Fig. 11(c)-11(d).

Fig. 11
figure 11

3D reconstruction results of the “spirogyra communis” intestinal. (a) False color map at the magnification of 10×. (b) Three-dimensional view map at the magnification of 10×. (d) False color map at the magnification of 60×. (e) Three-dimensional view map at the magnification of 60×

Discussion

As the chromatic aberration cannot be quantified according to the images directly, the relationship between the chromatic aberration and the magnification is given according to the simulation results of the zoom objective lens, as shown in Table 3. The spherical aberration and chromatic aberration at the magnification of 40×, 50× and 60× are significantly higher than those at the magnification of 10×, 20× and 30×, which also explains why the aberration is always obvious at high magnification in the images taken from the proposed microscope.

Table 3 Variation of the chromatic aberration with the magnification

The value of the focus evaluation operator has the characteristic of Gaussian distribution. Using this feature, the noise points which do not conform to the Gaussian distribution can be removed. This is where our algorithm differs from the ordinary SFF algorithms. As shown in Fig. 12, the focusing measure value varies with the image order. The black line family indicates pixels that do not satisfy the Gaussian distribution. The red line family represents pixels that satisfy the Gaussian distribution. We can find that the black line family is concentrated in the region of low focus value (0 ~ 0.2). Part of the black curve has two peaks. This is because in the low-texture region, the image texture does not change significantly in the out-of-focus and focus state, and the focus measure fails. In our work, we use root mean square error (RMSE) to quantify how far the focusing measure deviates from the Gaussian curve. According to the empirical value 0.07 ~ 0.095 obtained from the test, the noise point can be effectively removed

Fig. 12
figure 12

Family of curves satisfying and dissatisfying Gaussian distributions

As shown in Fig. 13(a), the white part is the point removed as noise. Figure 13(b) shows the depth map after removing the noise. According to the continuity of the sample surface, the depth value at the noise point is estimated by using the surrounding depth value reasonably. The removed noise points are filled with the estimated depth values, resulting in an complete depth map as shown in Fig. 13(c). We use this method to remove the Gaussian noise from the depth map, which will eventually result in better visual perception.

Fig. 13
figure 13

Process of obtaining a complete depth map. (a) Noise mask diagram, (b) Depth map with noise removed, (c) Complete depth map recovered using the estimated depth values. The scale bars in these pictures are 10 μm

The area with weak sample texture has a smaller focusing measure. And the background of the sample has this characteristic. Therefore, the slowly changing noise, such as the background of the sample, can be removed by setting an appropriate threshold. In addition to the thresholding method, the RMSE method described above can be used to remove such noise. Figure 14(a)-14(f) show the depth maps when the thresholds are 3, 7, 13, 16, 20, and 25, respectively. As can be seen from Fig. 14(a), the depth information of the “spirogyra communis” sample is presented completely and continuously. As the threshold increases, certain sample information will be lost, as shown in Fig. 14(b)-14(d). When the threshold is raised to more than 20, a large amount of depth information is filtered out, and the “spirogyra communis” samples cannot be reconstructed correctly, as shown in Fig. 14(e)-14(f). In this work, according to the window size of the ML operator and the strength of the sample texture, T = 3 is chosen as the appropriate threshold.

Fig. 14
figure 14

Effect of different custom threshold T values on the deep fusion of samples. (a) T = 3, (b) T = 7, (c) T = 13, (d) T = 16, (e) T = 20 and (f) T = 25. The scale bars in these pictures are 10 μm

Conclusion

In this paper, we report a continuous optical zoom microscope with extended depth of field and 3D reconstruction. By adjusting the applied voltages to the liquid lenses, the proposed microscope can achieve a large continuous magnification from ~ 10× to ~ 60× with a response time of ~ 50 ms. The experiments demonstrate that with the change of magnification, the resolution of the proposed microscope can reach more than 512 lp/mm during the zoom process. Under the premise of a fixed working distance, the proposed microscope can perform axial depth scanning to realize extended depth of field and 3D reconstruction. The proposed microscope can be widely applied in the fields of biological detection, chemical detection, etc.

Data Availability

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Abbreviations

SFF:

shape from focus.

CMOS:

complementary metal–oxide semiconductor.

ML:

modified Laplacian.

SML:

sum-modified-Laplacian.

MTF:

modulation transfer function.

RMS:

root mean square.

LL:

liquid lens.

RMSE:

root mean square error.

References

  1. Pasqualini FS, Agarwal A, O’Connor BB, Liu Q, Parker KK. Traction force microscopy of engineered cardiac tissues. PLoS ONE. 2018;13:e0194706.

    Article  Google Scholar 

  2. Tanaka N, Kanatani S, Tomer R, Sahlgren C, Kronqvist P, Kaczynska D, Louhivuori L, Kis L, Lindh C, Mitura P. Whole-tissue biopsy phenotyping of three-dimensional tumours reveals patterns of cancer heterogeneity. Nat Biomedical Eng. 2017;1:796–806.

    Article  Google Scholar 

  3. Wassie AT, Zhao Y, Boyden ES. Expansion microscopy: principles and uses in biological research. Nat Methods. 2019;16:33–41.

    Article  Google Scholar 

  4. Sun HY, Wang SL, Hu XB, Liu HJ, Zhou XY, Huang J, Cheng XL, Sun F, Liu YB, Liu D. Detection of surface defects and subsurface defects of polished optics with multisensor image fusion. PhotoniX. 2022;3:6.

    Article  Google Scholar 

  5. Kühnemund M, Wei QS, Darai E, Wang YJ, Hernández-Neuta I, Yang Z, Tseng D, Ahlford A, Mathot L, Sjöblom T, Ozcan A, Nilsson M. Targeted DNA sequencing and in situ mutation analysis using mobile phone microscopy. Nat Commun. 2017;8:13913.

    Article  Google Scholar 

  6. Fan Y, Li J, Lu L, Sun JS, Hu Y, Zhang JL, Li ZS, Shen Q, Wang BW, Zhang RN, Chen Q, Zuo C. Smart computational light microscopes (SCLMs) of smart computational imaging laboratory (SCILab). PhotoniX. 2021;2:19.

    Article  Google Scholar 

  7. Zhou Z, Huang JF, Li X, Gao XJ, Chen ZY, Jiao ZF, Zhang ZF, Luo QM, Fu L. Adaptive optical microscopy via virtual-imaging-assisted wavefront sensing for high-resolution tissue imaging. PhotoniX. 2022;3:1–20.

    Article  Google Scholar 

  8. Huang Z, Memmolo P, Ferraro P, Cao L. Dual-plane coupled phase retrieval for non-prior holographic imaging. PhotoniX 3, 3, (2022).

  9. Li Y, Shen C, Tan J, Wen X, Sun M, Huang G, Liu S, Liu Z. Fast quantitative phase imaging based on Kramers-Kronig relations in space domain. Opt Express. 2021;29:41067–80.

    Article  Google Scholar 

  10. Ren H, Wu ST. Variable-focus liquid lens by changing aperture. Appl Phys Lett. 2005;86:211107.

    Article  Google Scholar 

  11. Ren H, Wu ST. Variable-focus liquid lens. Opt Express. 2007;15:5931–6.

    Article  Google Scholar 

  12. Ren H, Xianyu HQ, Xu S, Wu ST. Adaptive dielectric liquid lens. Opt Express. 2008;16:14954–60.

    Article  Google Scholar 

  13. Zhan T, Xiong JH, Zou JY, Wu ST. Multifocal displays: review and prospect. PhotoniX. 2020;1:10.

    Article  Google Scholar 

  14. Hack MA, Tewes W, Xie QG, Datt C, Harth K, Harting J, Snoeijer JH. Self-similar liquid lens coalescence. Phys Rev Lett. 2020;124:194502.

    Article  Google Scholar 

  15. Dai B, Jiao Z, Zheng L, Bachman H, Fu YF, Wan XJ, Zhang YL, Huang Y, Han XD, Zhao CL, Huang T, Zhuang SL, Zhang DW: Colour compound lenses for a portable fluorescence microscope. Light: Science & Applications 8, 75 (2019).

  16. Cheng Y, Cao J, Hao Q, Xiao YQ, Zhang FH, Xia WZ, Zhang KY, Yu HY. A novel denoising method for improving the performance of full-waveform LiDAR using differential optical path. Remote Sens. 2017;9:1109.

    Article  Google Scholar 

  17. Tang B, Meng CZ, Zhuang L, Groenewold J, Qian YY, Sun ZQ, Liu XL, Gao J, Zhou GF. Field-induced wettability gradients for no-loss transport of oil droplets on slippery surfaces. ACS Appl Mater Interfaces. 2020;12:38723–9.

    Article  Google Scholar 

  18. Song XM, Zhang HX, Li DY, Jin QW, Jia DG, Liu TG, Wang C. Liquid lens with large focal length tunability fabricated in a polyvinyl chloride/dibutyl phthalate gel tube. Langmuir. 2019;36:1430–6.

    Article  Google Scholar 

  19. Kong M, Chen D, Chen X, Liang ZC, Zhao R, Xu EM. Research of the human eye model with variable-focus liquid lens. Microfluidics & Nanofluidics. 2017;21:40.

    Article  Google Scholar 

  20. Wang D, Liu C, Shen C, Xing Y, Wang QH. Holographic capture and projection system of real object based on tunable zoom lens. PhotoniX. 2020;1:6.

    Article  Google Scholar 

  21. Wang ZYH, Liu YZ, Gong CY, Yuan ZY, Shen L, Chang PX, Liu K, Xu TH, Jiang JF, Chen YC, Liu TG. Liquid crystal-amplified optofluidic biosensor for ultra-highly sensitive and stable protein assay. PhotoniX. 2021;2:18.

    Article  Google Scholar 

  22. Lan G, Mauger TF, Li G. Design of high-performance adaptive objective lens with large optical depth scanning range for ultrabroad near infrared microscopic imaging. Biomedical Opt Express. 2015;6:3362–77.

    Article  Google Scholar 

  23. Murali S, Thompson KP, Rolland JP. Three-dimensional adaptive microscopy using embedded liquid lens. Opt Lett. 2009;34:145–7.

    Article  Google Scholar 

  24. Kanhere A, Lin G, Jiang H. Remote axial tuning in microscopy utilizing hydrogel-driven tunable liquid lens. J Microelectromech Syst. 2016;25:304–10.

    Article  Google Scholar 

  25. Thériault G, Koninck YD, McCarthy N. Extended depth of field microscopy for rapid volumetric two-photon imaging. Opt Express. 2013;21:10095–104.

    Article  Google Scholar 

  26. Qu YF, Hu YB. Analysis of axial scanning range and magnification variation in wide-field microscope for measurement using an electrically tunable lens. Microsc Res Tech. 2019;82:101–13.

    Article  Google Scholar 

  27. Furieri T, Ancora D, Calisesi G, Morara S, Bassi A, Bonora S. Aberration measurement and correction on a large field of view in fluorescence microscopy. Biomedical Opt Express. 2022;13:262–73.

    Article  Google Scholar 

  28. Bonora S, Jian Y, Zhang P, Zam A, Pugh EN Jr, Zawadzki RJ, Sarunic MV. Wavefront correction and high-resolution in vivo OCT imaging with an objective integrated multi-actuator adaptive lens. Opt Express. 2015;23:21931.

    Article  Google Scholar 

  29. Wang ZJ, Lei L, Yao BL, Cai YN, Liang YS, Yang YL, Yang XL, Li H, Xiong DX. Compact multi-band fluorescent microscope with an electrically tunable lens for autofocusing. Biomedical Opt Express. 2015;6:4353–64.

    Article  Google Scholar 

  30. Qu YF, Zhu SY, Ping Z. A self-adaptive and nonmechanical motion autofocusing system for optical microscopes. Microscopy Res Technique. 2016;79:1112.

    Article  Google Scholar 

  31. Tehrani KF, Latchoumane CV, Southern WM, Pendleton EG, Mortensen LJ. Five-dimensional two-photon volumetric microscopy of in-vivo dynamic activities using liquid lens remote focusing. Biomedical Opt Express. 2019;10:3591.

    Article  Google Scholar 

  32. Ma HG, Cheng ZW, Xiong ZY, Yang KD. S. H.: Fast controllable confocal focus photoacoustic microscopy using a synchronous zoom opto-sono objective. Opt Lett. 2019;44:1880–3.

    Article  Google Scholar 

  33. Kuang FL, Yuan RY, Wang QH, Li L. Large zooming range adaptive microscope employing tunable objective and eyepiece. Sci Rep. 2020;10:14644.

    Article  Google Scholar 

  34. Song C, Yang Y, Tu X, Chen Z, Lin C. A smartphone-based fluorescence microscope with hydraulically driven optofluidic lens for quantification of glucose. IEEE Sens J. 2021;21:1229–35.

    Article  Google Scholar 

  35. Xi L, Song CL, Jiang HB. Confocal photoacoustic microscopy using a single multifunctional lens. Opt Lett. 2014;39:3328–31.

    Article  Google Scholar 

  36. Dean JL, Hirsa AH. Performance of a microscope with an embedded oscillating pinned-contact liquid lens. Appl Opt. 2015;54:8228–34.

    Article  Google Scholar 

  37. Li L, Wang D, Liu C, Wang QH. Zoom microscope objective using electrowetting lenses. Opt Express. 2016;24:2931–40.

    Article  Google Scholar 

  38. Jiang Z, Wang D, Zheng Y, Liu C, Wang QH. Continuous optical zoom microscopy imaging system based on liquid lens. Opt Express. 2021;29:20322–35.

    Article  Google Scholar 

  39. Qu YF, Zhang P, Hu YB. 3D measurements of micro-objects based on monocular wide-field optical microscopy with extended depth of field. Microscopy Res Technique. 2018;81:1434–42.

    Article  Google Scholar 

  40. Qu YF, Hu YB, Zhang P. Nonmechanical and multiview 3D measurement microscope for workpiece with large slope and complex geometry. J Microsc. 2018;272:123–35.

    Article  Google Scholar 

  41. Llavador A, Scrofani G, Saavedra G, Martinez-Corral M. Large depth-of-field integral microscopy by use of a liquid lens. Sensors. 2018;18:3383.

    Article  Google Scholar 

  42. Mugele F, Baret JC. Electrowetting: from basics to applications. J Phys: Condens Matter. 2005;17:705–74.

    Google Scholar 

  43. Nayar SK, Nakagawa Y. Shape from focus. IEEE Trans Pattern Anal Mach Intell. 1989;16:824–31.

    Article  Google Scholar 

  44. Tian YZ, Hu HJ, Cui HY, Yang SC, Qi J, Xu ZM, Li L. Three-dimensional surface microtopography recovery from a multifocus image sequence using an omnidirectional modified Laplacian operator with adaptive window size. Appl Opt. 2017;56:6300–10.

    Article  Google Scholar 

  45. Toet A. Image fusion by a ratio of low-pass pyramid. Pattern Recognit Lett. 1989;9:245–53.

    Article  MATH  Google Scholar 

Download references

Acknowledgements

We would like to thank Nanofabrication facility in Beihang Nano for technique consultation.

Funding

This work is supported by the National Natural Science Foundation of China under Grant Nos: 61927809 and 62175006. Beijing Natural Science Foundation under Grant No: 4222069.

Author information

Authors and Affiliations

Authors

Contributions

C. Liu and Z Jiang conceived the initial idea and performed the experiments. X. Wang, Y. Zheng and Y. W. Zheng analyzed the data. Q. H. Wang discussed the results and supervised the project.

Corresponding author

Correspondence to Qiong-Hua Wang.

Ethics declarations

Ethical approval and Consent to participate

Not applicable.

Consent for publication

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Additional file 1

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, C., Jiang, Z., Wang, X. et al. Continuous optical zoom microscope with extended depth of field and 3D reconstruction. PhotoniX 3, 20 (2022). https://doi.org/10.1186/s43074-022-00066-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43074-022-00066-0