Skip to main content

Harnessing disordered photonics via multi-task learning towards intelligent four-dimensional light field sensors


The complete description of a continuous-wave light field includes its four fundamental properties: wavelength, polarization, phase and amplitude. However, the simultaneous measurement of a multi-dimensional light field of such four degrees of freedom is challenging in conventional optical systems requiring a cascade of dispersive and polarization elements. In this work, we demonstrate a disordered-photonics-assisted intelligent four-dimensional light field sensor. This is achieved by discovering that the speckle patterns, generated from light scattering in a disordered medium, are intrinsically sensitive to a high-dimension light field given their high structural degrees of freedom. Further, the multi-task-learning deep neural network is leveraged to process the single-shot light-field-encoded speckle images free from any prior knowledge of the complex disordered structures and realizes the high-accuracy recognition of full-Stokes vector, multiple orbital angular momentum (OAM), wavelength and power. The proof-of-concept study shows that the states space of four-dimensional light field spanning as high as 1680=4 (multiple-OAM) \(\times\)2 (OAM power spectra) \(\times\)15 (multiple-wavelength) \(\times\)14 (polarizations) can be well recognized with high accuracy in the chip-integrated sensor. Our work provides a novel paradigm for the design of optical sensors for high-dimension light fields, which can be widely applied in optical communication, holography, and imaging.


Complex light field generation, modulation and detection are the cornerstone of modern optics and photonics research. The rise of metamaterials and metasurfaces using artificial subwavelength structures to tailor light-matter interactions has created unprecedented capability of light field manipulation either in the free space or on-chip circuits [1,2,3,4,5]. In particular, metasurface conventionally fabricated on an ultra-thin interface via nanofabrications, modulates the wavefront of light through spatially varying geometric structures [6, 7]. Since the optical response of a metasurface is engineered locally addressing large degrees of freedom, it has become a mainstay platform for harnessing the intrinsic high-dimension of light, such as amplitude, phase, polarization and wavelength, etc [8,9,10]. In contrast to the fast development of light field manipulation, the integrated optical detection systems for multi-dimensional light fields are only emerging recently [11,12,13], although they have important applications in optical communications, biomedical imaging and autonomous vehicles [12].

The conventional photodetectors only measure light intensity, and the intrinsic high-dimension of light collapse in the simply photo-electric conversion process. By leveraging the optical response of functional materials/microstructures, the novel integrated light-field sensors are developed, capable of resolving light polarization [14, 15], wavelength [16], incident angle [17] or orbital angular momentum (OAM) [18], etc. On the other hand, the direct integration of metasurface with photodetectors (arrays) for high-dimension light detection is attracting great attention [11, 19,20,21,22,23]. Nevertheless, so far these multi-dimensional light-field sensors require sophisticated designs and precise fabrications, and nearly all of them are optimized for no more than three featured properties of light [12,13,14, 16, 19,20,21,22,23], which are fundamentally limited by their internal degrees of freedom in the light transmission and detection systems [24,25,26]. Besides, the signal cross-talks in the measurements of high-dimension light due to the wave dynamics rapidly increase as the increase of detection freedom, and this challenges the conventional deterministic mapping or demultiplexing method based on the physics-driven models.

Light scattering by disordered media is a ubiquitous phenomenon that can be seen everywhere in life, such as sunset glows. These disordered structures of high internal degrees of freedom can support numerous transmitting channels of light. Although the light field can be evidently scrambled by the multiple scattering in disordered media, the optical information is not lost and thus contained in a speckle pattern since the scattering is elastic [27,28,29]. Basically, the input light field can be recovered from these speckle patterns if the transmission matrix is determined. However, when the system becomes large and complex (multiple scattering) and is further coupled with the high-dimension light, it will be challenging to solve the inverse problem from the conventional transmission matrix method [29]. In this work, we propose the concept of integrated four-dimensional light field sensors by harnessing light scattering in disordered structures via multi-task-learning (MTL) deep neural network (DNN) [30, 31] as illustrated in Fig. 1. The disordered nematic liquid crystals (NLC) forming schlieren textures via self-assembly provide more degrees of freedom of transmitting channels for light and generate speckle patterns. We reveal theoretically and experimentally that these speckle patterns are intrinsically correlated to the input high-dimension light field, i.e., polarization, wavelength, OAM and power, from the structural similarity (SSIM) index [32]. Furthermore, by leveraging a trained MTL-DNN, we demonstrate an intelligent four-dimensional light field sensor capable of simultaneously extracting the full-Stokes parameters, multiple-wavelength, multiple-OAM and optical power from the single-shot speckle patterns by a CMOS image chip. This data-driven and deep learning method overcomes the shortcomings of conventional light field decoding systems based on physical theories or numerical modelings since the signal processing can be automatically tracked to the optimal sensing data via a user-defined cost function [13, 33].

Fig. 1
figure 1

Schematic of integrated four-dimensional light field sensors via disordered nematic liquid crystal (NLC) medium, imaging chip and deep learning. The input high-dimensional light field of various wavelength, polarization, orbital angular momentum (OAM) and power, are scattered and encoded by a disordered NLC film, forming diversified speckle patterns and being captured in a CMOS imaging chip. The speckle image data can be regarded as states in a n-dimension vector space. The multi-task-learning (MTL) deep neural network (DNN) as a decoder maps the measured vector to the high-dimension space of light field including OAM, wavelength, power and polarization, through a user-define cost function

Results and discussion

Properties of speckle patterns correlated to high-dimension of light

The explicit calculation of light propagation and scattering in disordered NLC film is complex and can not provide clear physics intuition (see Methods). Here we use a simplified model of square NLC lattices of random director distributions (Fig. 2a), and calculate the far-field speckles as shown in Fig. 2b (Supplementary Section 2). The lattice constant of 20 \(\upmu\)m is determined from the autocorrelation function of measured NLC director distributions (Supplementary Fig. S1). We implement SSIM index (Supplementary Section 3) to quantitatively evaluate the correlations of the speckle patterns to high-dimension light field [32]. The larger SSIM difference of speckles among different input light fields indicates higher optical-field sensitivity of disordered media.

Fig. 2
figure 2

Principles of light field interactions with disordered NLC medium. a Simplified model composed of randomly LC directors arranged in a square lattice. Inset: Schematic of in-plane director angle \(\theta\) relating to the x-axis. b Simulated in-plane random distributions of NLC directors (left panel), and their far-field diffracted speckle patterns (right panel) with Gaussian beam input (waist radius of 1 mm). The ordinary refraction index and extraordinary refraction index of NLC medium are set as 1.517 and 1.741, respectively; and the NLC film thickness is set as 20 \(\upmu\)m. The scale bar is 2 mm. c Calculated structural similarity (SSIM) index of speckle patterns for various linear-polarization light input in a range of disorder strength (\(\varepsilon\)). d Calculated SSIM contrast for various disorder parameter (\(\varepsilon\)) under linear-polarization light input. e,f Simulated and experimental results of SSIM index of speckle patterns for different polarized light input (e) and different OAM-state input (f). The light wavelength is fixed as 660 nm. g Simulated and experimental results of SSIM for speckle patterns under different light wavelength input

We first explore the influence of disorder strength on light-field sensing. The simulated NLC film is characterized by the disorder parameter \(\varepsilon \ (0<\varepsilon \le 1)\), and it indicates the range of the random rotation angle of NLC director \(\theta _{\varepsilon }(x,y)\), which obeys a uniform distribution function \(f_{\varepsilon }(\theta )=\{1/\varepsilon \pi , 0<\theta \le \varepsilon \pi ; 0,\) otherwise} [34] (Supplementary Fig. S2). In Fig. 2c, we show the calculated SSIM of speckles as a function of the linear polarization angle of light under different disorder parameters. When \(\varepsilon =0\), the orientation of the NLC directors is homogeneous, hence the element acts as a uniform medium and cannot respond to light polarization via SSIM. By incrementally increasing the disorder, the similarity between simulated speckle patterns is decreased gradually in a certain range of polarization angle, since the wavefront of input light is spoiled. The SSIM contrast, defined as the maximal SSIM change in a parameter space, is monotonously increased with the disorder strength and is greatest near the critical point (\(\varepsilon =1\)), as shown in Fig. 2d. A similar trend of SSIM on the disorder strength is found for OAM and wavelength parameters (Supplementary Fig. S3 and S4). We note that the fabricated NLC film approximately corresponds to a fully disordered structure (\(\varepsilon \approx 1\), Supplementary Fig. S1).

Figure 2e shows the simulated (upper panel) and measured (lower panel) SSIM of speckle patterns under different polarized light incidences. The SSIM of speckles corresponding to a polarized light input is minimal when its linear polarization angle is 90 deg compared to the reference polarization state of 0 deg. The SSIM fluctuates with the change of input polarization states because the similarity among the speckles is fundamentally related to the distance of polarization states on Poincaré sphere (Fig. S5). Figure 2f shows the statistical response of speckle patterns with different OAM modes input, i.e. topological charge \(l=\pm 1,\pm 3,\pm 5\). Since the helical-phase front of OAM modes is easily distorted by the anisotropic inhomogeneity of NLC film, SSIM rapidly decreases with the change of OAM modes numerically and experimentally. As for the light wavelength dependency, numerical results in Fig. 2g (upper panel) show that SSIM is monotonously decreased with the increase of wavelength, which is consistent with experimental measurements. Basically, the local phase retardation for transporting light in NLC medium can be written as \(2\pi (n_o-n_e)d/\lambda\), where no and ne are the ordinary index and extraordinary index, respectively; d is the film thickness, \(\lambda\) is the light wavelength. Consequently, the far-field diffracting patterns shaped by disordered NLC medium depend on the input light polarization, phase distribution, wavelength and power theoretically.

Overall, from the SSIM index evaluations, the speckle patterns from a disordered NLC film are strongly correlated to the input light field properties. Nevertheless, it would be challenging to differentiate a multi-dimensional light field of multiple properties, by simply statistically analyzing the speckle patterns. First, the statistical features, such as SSIM index, can neither have single-valued mapping nor a monotonous relation to a single optical property of light. Regarding the infinite number of states for polarizations and OAM of light, their corresponding relations to SSIM make it impracticable to conduct the pre-calibrations of SSIM as indicated from Fig. 2e, f. Second, the random nature of NLC director distributions introduces strong coupling and mixing effects for the multi-dimensional light field input. For example, the multi-wavelength and multi-OAM states are constructively encoded in the diffracting patterns, and thus the deterministic one-to-more mapping simply from SSIM index is nearly impossible.

Methodology of MTL algorithms

To tackle these challenges, here we use deep learning methods to process the as-imaged speckle patterns in no need of knowing the physical properties of devices in advance, supporting a transferable architecture for the multi-dimensional light field sensing, as shown in Fig. 1. We adopt a hard-parameter sharing MTL-DNN architecture (Supplementary Section 6), which consists of a shared encoder and two task-specific heads. A multi-layer convolutional neural network (CNN) plays the role of shared encoder to extract general feature maps from the speckles, branching out into two independent multi-layer fully-connected layers for classification and regression, respectively. The classification head is implemented to recognize the discrete values of OAM modes and wavelengths, while the regression one predicts the continuous values of the power and polarization states of light.

It is well known that the form of loss function has a great impact on the power of training DNN models. In this study, we introduce the idea of homoscedastic uncertainty to adaptively decide the values of weights. The loss function of an MTL-DNN is constructed as the weighted linear sum of the losses for each individual task, which is sensitive to each task via the settings of weights [35]:

$$\begin{aligned} \mathcal {L}\left( \sigma _1,\sigma _2\right) =\frac{1}{2\sigma _1^2}\mathcal {L}_1+\frac{1}{{\sigma }_2^2}\mathcal {L}_2+\log {\sigma _1}+\log {\sigma _2} \end{aligned}$$

where the regression loss function \(\mathcal {L}_1\) and the classification one \(\mathcal {L}_2\) are weighted by observation noise parameters \(\sigma _1\) and \(\sigma _2\), respectively; \(\sigma _1\) and \(\sigma _2\) are used to capture relative confidence between tasks and tune the weights of the single-task losses during training processes (Supplementary Section 6). Since the relative confidence reflects the homoscedastic uncertainty of prediction outputs for each task, the program tends to suppress the effect of the task with high uncertainty, but increase the weight of the task with low uncertainty. This strategy makes the learning process more balanced as well as effective.

In the following sections, we separately optimize the MTL-DNN model for three intelligent sensing functions, i.e., full-Stokes vector sensing, multiple-OAM sensing and four-dimensional light field sensing.

Characterizing the performance of intelligent full-Stokes polarimetry

Polarization is an intrinsic characteristic of electromagnetic waves that describes the geometric path traced by the wave’s electric field vector [36]. The conventional polarization measurement in free space is mainly based on the division-of-amplitude or division-of-time approach [20, 23]. The Stokes vector is widely used to describe the polarization behavior of light, mainly because polarization ellipse is an amplitude description of polarized light and cannot be directly measured [37]. In this section, we demonstrate intelligent full-Stokes polarimetry by leveraging DNN. Theoretically, when a coherent light beam passes through the disordered NLC film, its polarization state and optical power are implicitly encoded in the speckle patterns. Because of the continuous values of the Stokes vector (polarization state) and power, we employ the regression head of the MTL-DNN model to decipher the optical properties of unknown light beams. The predicted label vector \((S_0, S_1, S_2, S_3)^T\) of four Stokes parameters are constructed, where \(S_1\), \(S_2\) and \(S_3\) are obtained at the output layer of the regression head, and \(S_0\) is calculated by the equation \(S_0=(S_1^2+S_2^2+S_3^2)^{1/2}\) considering the coherent light input. The datasets of captured 20,000 speckle images are divided into three subsets, i.e., training set (80%), validation set (10%) and testing set (10%). After training procedures, we plot the normalized Stokes vector for different input/predicted polarizations of various optical power as shown in Fig. 3a. Figure 3b further plots the projected light polarization of test results onto the \(S_1/S_0-S_2/S_0\), \(S_2/S_0-S_3/S_0\) and \(S_1/S_0-S_3/S_0\) planes for better clarity and error analysis. The results show that the predicted Stokes vectors are in good agreement with the input parameters. Figure 3c shows the predicted optical power \(S_0\) as a function of the input values, and the average error for the predicted power of light compared to the measurement is about 5%. Figure 3d shows the relationship between the loss function and the number of training epochs. With the increase in the number of iterations for the training dataset, the loss is decreased significantly and converged to around 0.005. We define the error on the Stokes vector reconstruction as \(\sigma =\textrm{arccos}(({S_1S^{'}_1+S_2S^{'}_2+ S_3S^{'}_3})/({S_0S^{'}_0}))\), where \(S_i\) and \(S^{'}_i\) (i=0,1,2,3) are input and predicted Stokes vector [23]. It is clear that \(\sigma\) corresponds to the angle between the input and the reconstructed Stokes vector on the Poincaré sphere. Therefore, the proposed intelligent full-Stokes polarimetry is capable of reconstructing the light polarization with a high accuracy and the average deviation angle \(\left\langle {\sigma }\right\rangle\) is as small as \(7^{\circ }\), which is comparable to the state-of-the-art results of computational polarimetry [15, 23].

Fig. 3
figure 3

Full-Stokes polarimetry by leveraging DNN. a Polarization states from the DNN predicted results (solid circles) and their corresponding input values (hollow circles) are plotted in the Poincaré sphere. The symbols’ colors differentiate the optical power for the input and predicted states. b The results from (a) are projected onto the normalized \(S_1-S_2\), \(S_1-S_3\) and \(S_2-S_3\) planes for better clarity and analysis. c Predicted power (\(S_0\)) as a function of incidence power. The red area corresponds to a 5% error in power measurements. d The loss function versus epochs for the training and testing datasets

Characterizing performance of intelligent multiple-OAM detection

Light with helical phase wavefront featuring exp(\(il\varphi\)) (where l and \(\varphi\) represent the topological charge and azimuthal angle, respectively) carries OAM, which has boosted advanced optical applications in communications, holography encryption, tweezers, high-order quantum entanglement and nonlinear optics [38,39,40,41,42]. The measurement of the topological charge of the OAM beam is a nontrivial task for many optical applications. The conventional measurements of topological charge are based on the deterministic interference and diffraction properties of OAM modes, which often require a precise alignment of the optical path and are difficult for compact integration [39, 43]. Recently, the speckle-based deep learning methods for OAM mode classification are being developed, and they have found applications in the OAM-multiplexed data transmission [44,45,46,47,48]. However, it is still challenging to realize simultaneous recognition of multiple-OAM states and their power spectra of different OAM channels, mainly limited by the single-task learning configuration, although the measurement of multiple-OAM spectra is crucial for high-capacity information processing systems [49, 50].

Here we demonstrate a novel MTL-based intelligent multiple-OAM sensing system to classify the multiple-OAM states and estimate their power spectra as shown in Fig. 4a. The major components of MTL-DNN consist of a four-layer CNN and two independently task-specific fully-connected layers (FCs) for the classification of OAM modes and the regression of their power spectra. In essence, the feature maps extracted from the input speckle images via the four-layer CNN are shared by the task-specific FCs; meanwhile, the task-specific FCs are used for learning a nonlinear mapping from features to individual targets. Several dropout layers are embedded between every two three-layer FCs to prevent overfitting and improve the generalization ability.

Fig. 4
figure 4

Intelligent multiple-OAM states sensing. a Schematic of multiple-OAM state detection empowered by MTL-DNN. The multiple-OAM state composing three weighted OAM modes is produced via a superposed phase plate. The multiple-OAM state is scattered by the disordered NLC device, and the generated speckle images are input to the MTL-DNN. The multiple-OAM states and their power spectra are recognized by two independently task-specific fully-connected layers. b The loss function on the training epochs numbers for train and test datasets of multiple-OAM states: (i) regression loss function \(\mathcal {L}_1\); (ii) classification loss function \(\mathcal {L}_2\) in semi-log axis; (iii) total loss function \(\mathcal {L}\). c The confusion matrix of 20 multiple-OAM states labeled 1-20, reporting accuracy of 100%. The right panel lists the OAM mode compositions for the different multiple-OAM states. d Four typical input and predicted spectra of multiple-OAM states. The predicted results agree reasonably well with the input spectra

Experimentally, the calculated phase profile is loaded into a spatial light modulator (SLM), and the diffracted multiple-OAM beam (Supplementary Section 5, Supplementary Fig. S7) is then launched to the disordered NLC medium to generate speckle patterns in the far-field regime. To reduce the performance degradation caused by discrete phase pattern and phase-only modulation on generated collinearly multiple-OAM beams, we design OAM base with a state interval of 2 for proof-of-concept demonstration [51]. We recorded 25,000 speckle images covering 20 different multiple-OAM states along with various power spectra. In the training stage, different multiple-OAM light field information and their corresponding speckle images are used to train the MTL-DNN model. Labeled samples are randomly split into training, validation and testing subsets in the ratio of 8:1:1. A MTL loss function is well-defined to adaptively balance the weights of two tasks during the training procedure. After the optimization process, the network is able to recover the multiple-OAM states information directly from a single speckle pattern. From Fig. 4b, it shows that the loss function of the regression and classification tasks converge to \(\sim 3\times 10^{-3}\) and \(\sim 1\times 10^{-8}\), respectively. And thus the total loss function of the MLT-DNN for multiple-OAM recognition is \(\sim\)0.02. Figure 4c shows the truth table for multiple-OAM states classification, and the recognition accuracy of twenty kinds of multiple-OAM states reaches 100%. In addition, Fig. 4d shows a comparison between the reconstructed OAM power spectra and the input results. It can be found that all the components of predicted results agree well with the input values. As far as we know, the proposed intelligent light-field sensor is for the first time demonstrated to realize the multiple-OAM spectra resolution, which will stimulate applications in OAM-based holography, imaging and communications.

Intelligent single-shot reconstruction of four-dimensional light fields via integrated sensors

In this section, we discuss the potential of using disordered NLC film and MTL-DNN to achieve the complete single-shot reconstruction of realistic four-dimensional light fields including polarization, multiple-OAM, multiple-wavelength, and power. The disordered NLC medium of high degrees of freedom provides rich scattering channels to the high dimension of light, which lays the foundation for direct single-shot recognition. We assemble the disordered liquid crystal cell and commercial imaging chip for proof-of-concept demonstration, as shown in Fig. 5a. The high-dimension light fields generated from the home-made optical system (Fig. 5b, see Methods) are directly imaged and decoded in the integrated sensors. We would like to note that the integrated sensing scheme significantly reduces the light propagation distance of the millimeter-scale and influence the diffracted speckle patterns compared with that of long-distance diffraction of tens of centimeter-scale (Supplementary Fig. S9), while their predicted results via the MTL-DNN are comparable (Supplementary Fig. S10). The decoding of high-dimensional properties of light from the single-shot speckles is not a trivial task considering the strong linear and nonlinear coupling in the light scattering and detection process. To cope with this difficulty, we continue to use the MTL-DNN model for previous multiple-OAM detection while making slight modifications in its output layers to meet specific requirements. We apply linear extrapolation and impose random noise on the expanded datasets. The signal-to-noise ratio (SNR) is defined as \(\text {SNR}=-10\lg {({\mu }/{I_s})}\), where \(\mu\) and Is represent the variance of noise and intensity of original measured image. Here, random noise with a SNR of 5-15 dB is imposed to generate a total of 120,000 speckle images based on the original 336 measured images. The extrapolation is appropriate based on two methods: (1) The speckle images are scaled linearly with the excitation power; (2) Polychromatic speckles are considered as a linear superposition of monochromatic speckles. These techniques are widely used in machine learning studies when a larger training set was desired [15].

Fig. 5
figure 5

Demonstration of integrated four-dimensional light field sensors. a Photograph of integrated light field sensors composing a liquid crystal cell and imaging chip. The inset is the crossed-polarized optical image of the disordered liquid crystal film, showing schlieren textures. The black arrows are the polarizer and analyzer directions. The scale bar is 150 \(\upmu\)m. b Schematic of the experimental setup. SLM: spatial light modulator, AOTF: acoustic-optical tunable filter, SC laser: super-continuum laser. c,d The typical input (left panel) and predicted results (right panel) of multiple-OAM modes in combinations of four polarizations (\(P_1\), \(P_4\), \(P_7\), \(P_9\)) at the wavelength of (c) 650 nm and (d) 660 nm. The labeled light wavelength is predicted by the MTL-DNN rather than predetermined. e,f The typical input (left panel) and predicted results (right panel) for the poly-chromatic light field in combinations of four polarizations (\(P_1\), \(P_4\), \(P_7\), \(P_9\)) for (e) multiple-OAM state 1 and (f) multiple-OAM state 2. These polarization states \(P_i\) are illustrated in Fig. 2e. The input power spectra of multiple-OAM state 1 are \(l=-1\) (20%), 1 (50%), 3 (30%); the input power spectra of multiple-OAM state 2 are \(l=-3\) (50%), -1 (30%), 3 (20%). The labeled multiple-OAM state is predicted by the MTL-DNN rather than predetermined. g The reconstruction error of polarization, power, wavelength and multiple-OAM on the signal-to-noise ratio (SNR)

After training MTL-DNN model, the loss function stops at \(\sim\)0.08 with classification loss of \(\sim\)0.03 and regression loss of \(\sim\)5\(\times 10^{-3}\) (Fig. S8). Using the trained model, the input multi-dimensional light fields can be completely recovered from single-shot speckle images. Figure 5c-f show the comparison of input and predicted results of 16 typical states of high-dimension light fields composing different combinations of polarization, multiple-OAM, multiple-wavelength and intensity, which clearly indicate the high recognition accuracy of the deep learning network. Since it is not intuitive to simultaneously illustrate the predicted results of such four properties of light, we have to fix one property of light and plot the remaining dimensions. To further validate the effectiveness of the proposed intelligent sensor, the mean squared error (MSE) between the input value and predicted value are implemented. The reconstruction error of polarization, power, wavelength and multiple-OAM as a function of SNR for high-dimension light is shown in Fig. 5g. The results indicate that the light-field sensor can realize high-dimension light field detection with high fidelity at a relatively high noise level (SNR \(\ge\)10 dB). In the current proof-of-concept study, the states space of a four-dimensional light field is as high as 1680, spanning by 4 multiple-OAMs, 2 OAM-power-spectra and 15 single-/multiple-wavelength and 14 polarizations, which can be well recognized with high accuracy. The states space can be readily extended (Supplementary Section 8) if more experimental data are included in the training network. As far as we know, this is the first demonstration of complete light field construction of four universal degrees of freedom, i.e., wavelength, polarization, OAM, and power via an integrated sensor, which provides a new paradigm shift in the complex light-field manipulations and detections.


In summary, we have demonstrated an integrated intelligent four-dimensional light field sensor enabled by disordered anisotropic medium and deep learning. Our results unambiguously show that the complete information of light including wavelength, polarization, OAM, and power can be decoded simultaneously via a single-shot speckle image from the disordered anisotropic medium. Compared with the conventional light field sensing systems by cascading numerous bulky opto-electro-mechanical elements along the optical path, this developed concept of the intelligent optical sensor provides a disruptive technology for the ultimate multi-dimensional light field detection. The current disordered NLC film is fabricated via self-assembly, and we envision that for practical applications, the photo-patterning technique can be implemented to generate arbitrarily-designed mesostructured NLC film with high repeatability [52, 53]. On the other hand, the developed speckle-based light field-sensing configuration can also be transferred to the multimode fiber scheme [46, 54, 55], and the extended degrees of freedom for high-dimension light encoding and decoding with ultrahigh data capacity can be realized. Furthermore, it is anticipated to construct a photonic neural network to implement machine learning, paving the way for on-chip all-optical light field sensing.


Device fabrications

The nematic liquid crystal (NLC) film (E7, Jiangsu Hecheng Display Technology) is sandwiched between two glass plates coated with indium-tin-oxide. The film thickness is controlled by spacers with a thickness of 20 \(\upmu\)m.

Optical measurements

The spatial director distributions of the NLC film is measured by a commercial birefringence imaging microscope (Crystalent-50, Ningcui). The multi-dimensional light field is generated by a home-built optical system, which consists of a wavelength-tunable module, a phase modulation module and polarization tuning module. The wavelength-tunable module is a super-continuum laser (YSL SC-Pro) filtered by an acoustic-optical tunable filter (AOTF, YSL AOTF-Pro bandwidths \(\sim\)4 nm). The spatial light modulator (SLM, UPOlabs HDSLM80R-Plus) is used to control the phase wavefront of the beam. The polarization state of light is modulated by a \(\lambda /2\) plate and a \(\lambda /4\) plate. The diffracted multi-dimensional light field in disordered NLC film is captured by a home-built imaging system (MIchrome 5 Pro, Shanghai Taizi Technology).

Light propagation in anisotropic disordered medium

We consider the Helmholtz equation that governs light propagation for the electric field \(\vec {E}_\omega (\vec {r})\):

$$\begin{aligned} -\nabla \times \nabla \times \vec {E}_\omega (\vec {r})+k^2_0 (\varvec{\varepsilon }^r+\varvec{\varepsilon }^s(\vec {r})) \vec {E}_\omega (\vec {r})=0 \end{aligned}$$

where \(k_0\) is the vacuum wave number and dielectric function \(\varvec{\varepsilon }(\vec {r})=\varvec{\varepsilon }^r+\varvec{\varepsilon }^s(\vec {r})\) is embedded in an infinite homogeneous reference medium \(\varvec{\varepsilon }^r\). When an incident field \(\vec {E}^0_\omega\) is considered, the scattering problem can be treated by setting up equivalent relations for the incident field that satisfies: \(-\nabla \times \nabla \times \vec {E}^0_\omega +k^2_0 \varvec{\varepsilon }^r \vec {E}^0_\omega =0\) in the homogeneous and source-free reference system. Introducing the operators \(\varvec{L}\), \(\varvec{e}_r\), and \(\varvec{e}_s\) for simplifying the notation \(-\nabla \times \nabla \times\), \(k^2_0 \varvec{\varepsilon }^r\), and \(k^2_0 \varvec{\varepsilon }^s(\vec {r})\), respectively, we can rewrite the Eq. 2 as:

$$\begin{aligned} (\varvec{L}+\varvec{e}_r+\varvec{e}_s)\vec {E}_\omega (\vec {r})=0 \end{aligned}$$

The incident light field \(\vec {E}^0_\omega\) in the homogeneous and source-free reference system satisfies:

$$\begin{aligned} (\varvec{L}+\varvec{e}_r)\vec {E}^0_\omega =0 \end{aligned}$$

The Green’s tensor \(\varvec{G}(\vec {r}, \vec {r}^{\prime }, \omega )\) describes the impulse response of a given physical system. This dyadic tensor is closely related to the field susceptibility of the system. The Green’s function \(\varvec{G}\) associated with the complete system is defined by

$$\begin{aligned} (\varvec{L}+\varvec{e}_r+\varvec{e}_s)\varvec{G}=\delta (\vec {r}-\vec {r}^{\prime })\varvec{I} \end{aligned}$$

where \(\varvec{I}\) is the unit tensor. By setting Eq. 3 equal to Eq. 4 and using Eq. 5, we obtain the following formula:

$$\begin{aligned} \vec {E}_\omega (\vec {r})=\vec {E}^0_\omega -\varvec{G}\varvec{e}_s\vec {E}^0_\omega \end{aligned}$$

We define the generalized propagator as \(\vec {K}(\vec {r},\vec {r}^{\prime },\omega )=1-\varvec{G}\varvec{e}_s\). Thus the scattered field of Eq. 6 in the \(\vec {r}\) representation becomes [56]:

$$\begin{aligned} \vec {E}_\omega (\vec {r})=\int d \vec {r}^{\prime } \vec {K}(\vec {r},\vec {r}^{\prime },\omega ) \vec {E}^0_\omega (\vec {r}^{\prime }) \end{aligned}$$

Data preparation

The color images collected by the camera will be converted into gray values according to the formula \(I(x,y) = 0.3I_R(x,y) +0.59I_G(x,y)+ 0.11I_B(x,y)\), where \(I_R\), \(I_G\), and \(I_B\) are red-, green-, and blue-channel of images, respectively. The converted and subsampled (408 \(\times\) 342 pixels) images are imported into the deep-learning network for training.

Availability of data and materials

All the data and codes are available from the corresponding author upon reasonable request.



orbital angular momentum




deep neural network


disordered nematic liquid crystals


structural similarity


convolutional neural network


fully-connected layer


spatial light modulator


signal-to-noise ratio


mean squared error


acoustic-optical tunable filter


  1. Liu Y, Zhang X. Metamaterials: a new frontier of science and technology. Chem Soc Rev. 2011;40(5):2494–507.

    Google Scholar 

  2. Chen H, Chan CT, Sheng P. Transformation optics and metamaterials. Nat Mater. 2010;9(5):387–96.

    Google Scholar 

  3. Yu N, Capasso F. Flat optics with designer metasurfaces. Nat Mater. 2014;13(2):139–50.

    Google Scholar 

  4. Meng Y, Chen Y, Lu L, Ding Y, Cusano A, Fan JA, et al. Optical meta-waveguides for integrated photonics and beyond. Light Sci Appl. 2021;10:235.

    Google Scholar 

  5. Sun S, He Q, Hao J, Xiao S, Zhou L. Electromagnetic metasurfaces: physics and applications. Adv Opt Photon. 2019;11(2):380–479.

    Google Scholar 

  6. Yu N, Genevet P, Kats MA, Aieta F, Tetienne JP, Capasso F, et al. Light propagation with phase discontinuities: generalized laws of reflection and refraction. Science. 2011;334(6054):333–7.

    Google Scholar 

  7. Sun S, He Q, Xiao S, Xu Q, Li X, Zhou L. Gradient-index meta-surfaces as a bridge linking propagating waves and surface waves. Nat Mater. 2012;11(5):426–31.

    Google Scholar 

  8. Song Q, Liu X, Qiu CW, Genevet P. Vectorial metasurface holography. Appl Phys Rev. 2022;9(1):011311.

    Google Scholar 

  9. Chen HT, Taylor AJ, Yu N. A review of metasurfaces: physics and applications. Rep Prog Phys. 2016;79(7):076401.

    Google Scholar 

  10. Li G, Zhang S, Zentgraf T. Nonlinear photonic metasurfaces. Nat Rev Mater. 2017;2:17010.

    Google Scholar 

  11. Chen WT, Zhu AY, Capasso F. Flat optics with dispersion-engineered metasurfaces. Nat Rev Mater. 2020;5(8):604–20.

    Google Scholar 

  12. Qu Y, Yi S, Yang L, Yu Z. Multimodal light-sensing pixel arrays. Appl Phys Lett. 2022;121(4):040501.

    Google Scholar 

  13. Yuan S, Ma C, Fetaya E, Mueller T, Naveh D, Zhang F, et al. Geometric deep optical sensing. Science. 2023;379(6637):eade1220.

    Google Scholar 

  14. Xiong Y, Wang Y, Zhu R, Xu H, Wu C, Chen J, et al. Twisted black phosphorus–based van der Waals stacks for fiber-integrated polarimeters. Sci Adv. 2022;8(18):eabo0375.

    Google Scholar 

  15. Ma C, Yuan S, Cheung P, Watanabe K, Taniguchi T, Zhang F, et al. Intelligent infrared sensing enabled by tunable moiré quantum geometry. Nature. 2022;604(7905):266–72.

    Google Scholar 

  16. Yang Z, Albrow-Owen T, Cai W, Hasan T. Miniaturization of optical spectrometers. Science. 2021;371(6528):eabe0722.

    Google Scholar 

  17. Yi S, Zhou M, Yu Z, Fan P, Behdad N, Lin D, et al. Subwavelength angle-sensing photodetectors inspired by directional hearing in small animals. Nat Nanotechnol. 2018;13(12):1143–7.

    Google Scholar 

  18. Ji Z, Liu W, Krylyuk S, Fan X, Zhang Z, Pan A, et al. Photocurrent detection of the orbital angular momentum of light. Science. 2020;368(6492):763–7.

    MathSciNet  MATH  Google Scholar 

  19. Tittl A, Leitis A, Liu M, Yesilkoy F, Choi DY, Neshev DN, et al. Imaging-based molecular barcoding with pixelated dielectric metasurfaces. Science. 2018;360(6393):1105–9.

    MathSciNet  MATH  Google Scholar 

  20. Wei J, Xu C, Dong B, Qiu CW, Lee C. Mid-infrared semimetal polarization detectors with configurable polarity transition. Nat Photon. 2021;15(8):614–21.

    Google Scholar 

  21. Wang Z, Yi S, Chen A, Zhou M, Luk TS, James A, et al. Single-shot on-chip spectral sensors based on photonic crystal slabs. Nat Commun. 2019;10:1020.

    Google Scholar 

  22. Pors A, Nielsen MG, Bozhevolnyi SI. Plasmonic metagratings for simultaneous determination of Stokes parameters. Optica. 2015;2(8):716–23.

    Google Scholar 

  23. Ni Y, Chen C, Wen S, Xue X, Sun L, Yang Y. Computational spectropolarimetry with a tunable liquid crystal metasurface. eLight. 2022;2:23.

    Google Scholar 

  24. Lukosz W. Optical systems with resolving powers exceeding the classical limit. J Opt Soc Am. 1966;56(11):1463–71.

    Google Scholar 

  25. Di Francia GT. Degrees of freedom of an image. J Opt Soc Am A. 1969;59(7):799–804.

    Google Scholar 

  26. Zuo C, Chen Q. Exploiting optical degrees of freedom for information multiplexing in diffractive neural networks. Light Sci Appl. 2022;11:208.

    Google Scholar 

  27. Bellini T, Clark NA, Degiorgio V, Mantegazza F, Natale G. Light-scattering measurement of the nematic correlation length in a liquid crystal with quenched disorder. Phys Rev E. 1998;57:2996–3006.

    Article  Google Scholar 

  28. Wiersma DS. Disordered photonics. Nat Photon. 2013;7(3):188–96.

    Google Scholar 

  29. Gigan S. Imaging and computing with disorder. Nat Phys. 2022;18(9):980–5.

    Google Scholar 

  30. Vandenhende S, Georgoulis S, Van Gansbeke W, Proesmans M, Dai D, Van Gool L. Multi-task learning for dense prediction tasks: A survey. IEEE Trans Pattern Anal Mach Intell. 2021;44(7):3614–33.

    Google Scholar 

  31. Thung KH, Wee CY. A brief review on multi-task learning. Multimed Tools Appl. 2018;77(22):29705–25.

    Google Scholar 

  32. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13(4):600–12.

    Google Scholar 

  33. Ballard Z, Brown C, Madni AM, Ozcan A. Machine learning and computation-enabled intelligent sensor design. Nat Mach Intell. 2021;3(7):556–65.

    Google Scholar 

  34. Maguid E, Yannai M, Faerman A, Yulevich I, Kleiner V, Hasman E. Disorder-induced optical transition from spin Hall to random Rashba effect. Science. 2017;358(6369):1411–5.

    MathSciNet  MATH  Google Scholar 

  35. Kendall A, Gal Y, Cipolla R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: Proc. IEEE Conf. Comput. Vis. Pattern Recog. 2018. p. 7482–7491.

  36. Rubin NA, Shi Z, Capasso F. Polarization in diffractive optics and metasurfaces. Adv Opt Photon. 2021;13(4):836–970.

    Google Scholar 

  37. Schaefer B, Collett E, Smyth R, Barrett D, Fraher B. Measuring the Stokes polarization parameters. Am J Phys. 2007;75(2):163–8.

    Google Scholar 

  38. Wang J, Yang JY, Fazal IM, Ahmed N, Yan Y, Huang H, et al. Terabit free-space data transmission employing orbital angular momentum multiplexing. Nat Photon. 2012;6(7):488–96.

    Google Scholar 

  39. Shen Y, Wang X, Xie Z, Min C, Fu X, Liu Q, et al. Optical vortices 30 years on: OAM manipulation from topological charge to multiple singularities. Light Sci Appl. 2019;8:90.

    Google Scholar 

  40. Padgett M, Bowman R. Tweezers with a twist. Nat Photon. 2011;5(6):343–8.

    Google Scholar 

  41. Fang X, Ren H, Gu M. Orbital angular momentum holography for high-security encryption. Nat Photon. 2020;14(2):102–8.

    Google Scholar 

  42. Ouyang X, Xu Y, Xian M, Feng Z, Zhu L, Cao Y, et al. Synthetic helical dichroism for six-dimensional optical orbital angular momentum multiplexing. Nat Photon. 2021;15(12):901–7.

    Google Scholar 

  43. Fu S, Zhai Y, Zhang J, Liu X, Song R, Zhou H, Gao C. Universal orbital angular momentum spectrum analyzer for beams. PhotoniX. 2020;1(19):1–12.

  44. Raskatla V, Singh B, Patil S, Kumar V, Singh R. Speckle-based deep learning approach for classification of orbital angular momentum modes. J Opt Soc Am A. 2022;39(4):759–65.

    Google Scholar 

  45. Liu Y, Zhang Z, Yu P, Wu Y, Wang Z, Li Y, et al. Learning-enabled recovering scattered data from twisted light transmitted through a long standard multimode fiber. Appl Phys Lett. 2022;120(13):131101.

    Google Scholar 

  46. Zhao Q, Yu PP, Liu YF, Wang ZQ, Li YM, Gong L. Light field imaging through a single multimode fiber for OAM-multiplexed data transmission. Appl Phys Lett. 2020;116(18):181101.

    Google Scholar 

  47. Wang J, Fu S, Shang Z, Hai L, Gao C. Adjusted EfficientNet for the diagnostic of orbital angular momentum spectrum. Opt Lett. 2022;47(6):1419–22.

    Google Scholar 

  48. Fu S, Shang Z, Hai L, Huang L, Lv Y, Gao C. Orbital angular momentum comb generation from azimuthal binary phases. Adv Photon Nexus. 2022;1(1):016003.

    Google Scholar 

  49. Fang X, Ren H, Li K, Luan H, Hua Y, Zhang Q, et al. Nanophotonic manipulation of optical angular momentum for high-dimensional information optics. Adv Opt Photon. 2021;13(4):772–833.

    Google Scholar 

  50. Willner AE, Huang H, Yan Y, Ren Y, Ahmed N, Xie G, et al. Optical communications using orbital angular momentum beams. Adv Opt Photon. 2015;7(1):66–106.

    Google Scholar 

  51. Du J, Wang J. Design of on-chip N-fold orbital angular momentum multicasting using V-shaped antenna array. Sci Rep. 2015;5:9662.

    Google Scholar 

  52. Ma L-L, Li C-Y, Pan J-T, Ji Y-E, Jiang C, Zheng R, Wang Z-Y, Wang Y, Li B-X, Lu Y-Q. Self-assembled liquid crystal architectures for soft matter photonics. Light Sci Appl. 2022;11:270.

    Google Scholar 

  53. Guo Y, Jiang M, Peng C, Sun K, Yaroshchuk O, Lavrentovich O, Wei Q-H. High-resolution and high-throughput plasmonic photopatterning of complex molecular orientations in liquid crystals. Adv Mater. 2016;28(12):2353–8.

    Google Scholar 

  54. Liu Z, Wang L, Meng Y, He T, He S, Yang Y, et al. All-fiber high-speed image detection enabled by deep learning. Nat Commun. 2022;13:1433.

  55. Tang P, Zheng K, Yuan W, Pan T, Xu Y, Fu S, et al. Learning to transmit images through optical speckle of a multimode fiber with high fidelity. Appl Phys Lett. 2022;121(8):081107.

    Google Scholar 

  56. Martin OJF, Girard C, Dereux A. Generalized Field Propagator for Electromagnetic Scattering and Light Confinement. Phys Rev Lett. 1995;74:526–9.

    Article  Google Scholar 

Download references


The authors thank Prof. Jun Yi for experimental assistance and Mr. Ke Cheng for his help with the preparation of illustration.


This project was supported by National Key Research and Development Program of China (2022YFA1203700), Natural Science Foundation of Fujian Province of China (2023J06011), National Natural Science Foundation of China (Grants No. 12274357, 62005231, 62075186, 62005164), Fundamental Research Funds for the Central Universities (20720210045), Guangdong Basic and Applied Basic Research Foundation (2021A1515012199), Shanghai Rising-Star Program (20QA1404100), Science and Technology Commission of Shanghai Municipality (Grant No. 21DZ1100500), Shanghai Municipal Science and Technology Major Project, Shanghai Frontiers Science Center Program (2021-2025 No. 20) and Zhangjiang National Innovation Demonstration Zone (Grant No. ZJ2019-ZD-005).

Author information

Authors and Affiliations



S.-k. Z. and Z.-h. Z. contributed equally to this work. S.-k. Z. and S.-s. C. fabricated the device, S.-k. Z. performed the experiment, S.-k. Z., Z.-h. Z. and J.-h. C. built the theoretical model and algorithms, S.-k. Z., Z.-h. Z., X. F., and J.-h. C. analyzed the results, J.-h C. conceived the idea and co-supervised the project with L.-J. C. and X. F. All authors contributed to the discussion, analyzed the data, and wrote the manuscript.

Corresponding authors

Correspondence to Lu-Jian Chen, Xinyuan Fang or Jin-hui Chen.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Including Supplementary Section 1-8, Fig.S1-11 and Table S1. The Supplementary Information is organized as follows. In Section 1, the optical characterizations and measurements of disordered nematic liquid crystal (NLC) is discussed. It includes multi-dimensional light field generation, speckle pattern detection and the NLC director distribution measurement. In Section 2, we introduce the numerical modeling of the speckles diffracted from disordered NLC medium. In Section 3, structural similarity (SSIM) index to evaluate the similarity of speckle patterns are discussed. In Section 4, analysis of diffraction patterns from uniform to full disorder for high-dimension light based on SSIM index are discussed. In Section 5, the algorithm for generation multiple-OAM modes are discussed. In Section 6, MTL-DNN structure and training processes are explicitly discussed. In Section 7, The influence of distance between camera and disordered NLC film is discussed. In Section 8, we discuss fundamental limit of intelligent light field sensor.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, Sk., Zheng, Zh., Meng, W. et al. Harnessing disordered photonics via multi-task learning towards intelligent four-dimensional light field sensors. PhotoniX 4, 26 (2023).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: