 Research
 Open access
 Published:
Harnessing disordered photonics via multitask learning towards intelligent fourdimensional light field sensors
PhotoniX volumeÂ 4, ArticleÂ number:Â 26 (2023)
Abstract
The complete description of a continuouswave light field includes its four fundamental properties: wavelength, polarization, phase and amplitude. However, the simultaneous measurement of a multidimensional light field of such four degrees of freedom is challenging in conventional optical systems requiring a cascade of dispersive and polarization elements. In this work, we demonstrate a disorderedphotonicsassisted intelligent fourdimensional light field sensor. This is achieved by discovering that the speckle patterns, generated from light scattering in a disordered medium, are intrinsically sensitive to a highdimension light field given their high structural degrees of freedom. Further, the multitasklearning deep neural network is leveraged to process the singleshot lightfieldencoded speckle images free from any prior knowledge of the complex disordered structures and realizes the highaccuracy recognition of fullStokes vector, multiple orbital angular momentum (OAM), wavelength and power. The proofofconcept study shows that the states space of fourdimensional light field spanning as high as 1680=4 (multipleOAM) \(\times\)2 (OAM power spectra) \(\times\)15 (multiplewavelength) \(\times\)14 (polarizations) can be well recognized with high accuracy in the chipintegrated sensor. Our work provides a novel paradigm for the design of optical sensors for highdimension light fields, which can be widely applied in optical communication, holography, and imaging.
Introduction
Complex light field generation, modulation and detection are the cornerstone of modern optics and photonics research. The rise of metamaterials and metasurfaces using artificial subwavelength structures to tailor lightmatter interactions has created unprecedented capability of light field manipulation either in the free space or onchip circuits [1,2,3,4,5]. In particular, metasurface conventionally fabricated on an ultrathin interface via nanofabrications, modulates the wavefront of light through spatially varying geometric structures [6, 7]. Since the optical response of a metasurface is engineered locally addressing large degrees of freedom, it has become a mainstay platform for harnessing the intrinsic highdimension of light, such as amplitude, phase, polarization and wavelength, etc [8,9,10]. In contrast to the fast development of light field manipulation, the integrated optical detection systems for multidimensional light fields are only emerging recently [11,12,13], although they have important applications in optical communications, biomedical imaging and autonomous vehicles [12].
The conventional photodetectors only measure light intensity, and the intrinsic highdimension of light collapse in the simply photoelectric conversion process. By leveraging the optical response of functional materials/microstructures, the novel integrated lightfield sensors are developed, capable of resolving light polarization [14, 15], wavelength [16], incident angle [17] or orbital angular momentum (OAM) [18], etc. On the other hand, the direct integration of metasurface with photodetectors (arrays) for highdimension light detection is attracting great attention [11, 19,20,21,22,23]. Nevertheless, so far these multidimensional lightfield sensors require sophisticated designs and precise fabrications, and nearly all of them are optimized for no more than three featured properties of light [12,13,14, 16, 19,20,21,22,23], which are fundamentally limited by their internal degrees of freedom in the light transmission and detection systems [24,25,26]. Besides, the signal crosstalks in the measurements of highdimension light due to the wave dynamics rapidly increase as the increase of detection freedom, and this challenges the conventional deterministic mapping or demultiplexing method based on the physicsdriven models.
Light scattering by disordered media is a ubiquitous phenomenon that can be seen everywhere in life, such as sunset glows. These disordered structures of high internal degrees of freedom can support numerous transmitting channels of light. Although the light field can be evidently scrambled by the multiple scattering in disordered media, the optical information is not lost and thus contained in a speckle pattern since the scattering is elastic [27,28,29]. Basically, the input light field can be recovered from these speckle patterns if the transmission matrix is determined. However, when the system becomes large and complex (multiple scattering) and is further coupled with the highdimension light, it will be challenging to solve the inverse problem from the conventional transmission matrix method [29]. In this work, we propose the concept of integrated fourdimensional light field sensors by harnessing light scattering in disordered structures via multitasklearning (MTL) deep neural network (DNN) [30, 31] as illustrated in Fig. 1. The disordered nematic liquid crystals (NLC) forming schlieren textures via selfassembly provide more degrees of freedom of transmitting channels for light and generate speckle patterns. We reveal theoretically and experimentally that these speckle patterns are intrinsically correlated to the input highdimension light field, i.e., polarization, wavelength, OAM and power, from the structural similarity (SSIM) index [32]. Furthermore, by leveraging a trained MTLDNN, we demonstrate an intelligent fourdimensional light field sensor capable of simultaneously extracting the fullStokes parameters, multiplewavelength, multipleOAM and optical power from the singleshot speckle patterns by a CMOS image chip. This datadriven and deep learning method overcomes the shortcomings of conventional light field decoding systems based on physical theories or numerical modelings since the signal processing can be automatically tracked to the optimal sensing data via a userdefined cost function [13, 33].
Results and discussion
Properties of speckle patterns correlated to highdimension of light
The explicit calculation of light propagation and scattering in disordered NLC film is complex and can not provide clear physics intuition (see Methods). Here we use a simplified model of square NLC lattices of random director distributions (Fig. 2a), and calculate the farfield speckles as shown in Fig. 2b (Supplementary Section 2). The lattice constant of 20 \(\upmu\)m is determined from the autocorrelation function of measured NLC director distributions (Supplementary Fig. S1). We implement SSIM index (Supplementary Section 3) to quantitatively evaluate the correlations of the speckle patterns to highdimension light field [32]. The larger SSIM difference of speckles among different input light fields indicates higher opticalfield sensitivity of disordered media.
We first explore the influence of disorder strength on lightfield sensing. The simulated NLC film is characterized by the disorder parameter \(\varepsilon \ (0<\varepsilon \le 1)\), and it indicates the range of the random rotation angle of NLC director \(\theta _{\varepsilon }(x,y)\), which obeys a uniform distribution function \(f_{\varepsilon }(\theta )=\{1/\varepsilon \pi , 0<\theta \le \varepsilon \pi ; 0,\) otherwise} [34] (Supplementary Fig. S2). In Fig. 2c, we show the calculated SSIM of speckles as a function of the linear polarization angle of light under different disorder parameters. When \(\varepsilon =0\), the orientation of the NLC directors is homogeneous, hence the element acts as a uniform medium and cannot respond to light polarization via SSIM. By incrementally increasing the disorder, the similarity between simulated speckle patterns is decreased gradually in a certain range of polarization angle, since the wavefront of input light is spoiled. The SSIM contrast, defined as the maximal SSIM change in a parameter space, is monotonously increased with the disorder strength and is greatest near the critical point (\(\varepsilon =1\)), as shown in Fig. 2d. A similar trend of SSIM on the disorder strength is found for OAM and wavelength parameters (Supplementary Fig. S3 and S4). We note that the fabricated NLC film approximately corresponds to a fully disordered structure (\(\varepsilon \approx 1\), Supplementary Fig. S1).
Figure 2e shows the simulated (upper panel) and measured (lower panel) SSIM of speckle patterns under different polarized light incidences. The SSIM of speckles corresponding to a polarized light input is minimal when its linear polarization angle is 90 deg compared to the reference polarization state of 0 deg. The SSIM fluctuates with the change of input polarization states because the similarity among the speckles is fundamentally related to the distance of polarization states on PoincarÃ© sphere (Fig. S5). Figure 2f shows the statistical response of speckle patterns with different OAM modes input, i.e. topological charge \(l=\pm 1,\pm 3,\pm 5\). Since the helicalphase front of OAM modes is easily distorted by the anisotropic inhomogeneity of NLC film, SSIM rapidly decreases with the change of OAM modes numerically and experimentally. As for the light wavelength dependency, numerical results in Fig. 2g (upper panel) show that SSIM is monotonously decreased with the increase of wavelength, which is consistent with experimental measurements. Basically, the local phase retardation for transporting light in NLC medium can be written as \(2\pi (n_on_e)d/\lambda\), where n_{o} and n_{e} are the ordinary index and extraordinary index, respectively; d is the film thickness, \(\lambda\) is the light wavelength. Consequently, the farfield diffracting patterns shaped by disordered NLC medium depend on the input light polarization, phase distribution, wavelength and power theoretically.
Overall, from the SSIM index evaluations, the speckle patterns from a disordered NLC film are strongly correlated to the input light field properties. Nevertheless, it would be challenging to differentiate a multidimensional light field of multiple properties, by simply statistically analyzing the speckle patterns. First, the statistical features, such as SSIM index, can neither have singlevalued mapping nor a monotonous relation to a single optical property of light. Regarding the infinite number of states for polarizations and OAM of light, their corresponding relations to SSIM make it impracticable to conduct the precalibrations of SSIM as indicated from Fig. 2e, f. Second, the random nature of NLC director distributions introduces strong coupling and mixing effects for the multidimensional light field input. For example, the multiwavelength and multiOAM states are constructively encoded in the diffracting patterns, and thus the deterministic onetomore mapping simply from SSIM index is nearly impossible.
Methodology of MTL algorithms
To tackle these challenges, here we use deep learning methods to process the asimaged speckle patterns in no need of knowing the physical properties of devices in advance, supporting a transferable architecture for the multidimensional light field sensing, as shown in Fig. 1. We adopt a hardparameter sharing MTLDNN architecture (Supplementary Section 6), which consists of a shared encoder and two taskspecific heads. A multilayer convolutional neural network (CNN) plays the role of shared encoder to extract general feature maps from the speckles, branching out into two independent multilayer fullyconnected layers for classification and regression, respectively. The classification head is implemented to recognize the discrete values of OAM modes and wavelengths, while the regression one predicts the continuous values of the power and polarization states of light.
It is well known that the form of loss function has a great impact on the power of training DNN models. In this study, we introduce the idea of homoscedastic uncertainty to adaptively decide the values of weights. The loss function of an MTLDNN is constructed as the weighted linear sum of the losses for each individual task, which is sensitive to each task via the settings of weights [35]:
where the regression loss function \(\mathcal {L}_1\) and the classification one \(\mathcal {L}_2\) are weighted by observation noise parameters \(\sigma _1\) and \(\sigma _2\), respectively; \(\sigma _1\) and \(\sigma _2\) are used to capture relative confidence between tasks and tune the weights of the singletask losses during training processes (Supplementary Section 6). Since the relative confidence reflects the homoscedastic uncertainty of prediction outputs for each task, the program tends to suppress the effect of the task with high uncertainty, but increase the weight of the task with low uncertainty. This strategy makes the learning process more balanced as well as effective.
In the following sections, we separately optimize the MTLDNN model for three intelligent sensing functions, i.e., fullStokes vector sensing, multipleOAM sensing and fourdimensional light field sensing.
Characterizing the performance of intelligent fullStokes polarimetry
Polarization is an intrinsic characteristic of electromagnetic waves that describes the geometric path traced by the waveâ€™s electric field vector [36]. The conventional polarization measurement in free space is mainly based on the divisionofamplitude or divisionoftime approach [20, 23]. The Stokes vector is widely used to describe the polarization behavior of light, mainly because polarization ellipse is an amplitude description of polarized light and cannot be directly measured [37]. In this section, we demonstrate intelligent fullStokes polarimetry by leveraging DNN. Theoretically, when a coherent light beam passes through the disordered NLC film, its polarization state and optical power are implicitly encoded in the speckle patterns. Because of the continuous values of the Stokes vector (polarization state) and power, we employ the regression head of the MTLDNN model to decipher the optical properties of unknown light beams. The predicted label vector \((S_0, S_1, S_2, S_3)^T\) of four Stokes parameters are constructed, where \(S_1\), \(S_2\) and \(S_3\) are obtained at the output layer of the regression head, and \(S_0\) is calculated by the equation \(S_0=(S_1^2+S_2^2+S_3^2)^{1/2}\) considering the coherent light input. The datasets of captured 20,000 speckle images are divided into three subsets, i.e., training set (80%), validation set (10%) and testing set (10%). After training procedures, we plot the normalized Stokes vector for different input/predicted polarizations of various optical power as shown in Fig. 3a. Figure 3b further plots the projected light polarization of test results onto the \(S_1/S_0S_2/S_0\), \(S_2/S_0S_3/S_0\) and \(S_1/S_0S_3/S_0\) planes for better clarity and error analysis. The results show that the predicted Stokes vectors are in good agreement with the input parameters. Figure 3c shows the predicted optical power \(S_0\) as a function of the input values, and the average error for the predicted power of light compared to the measurement is about 5%. Figure 3d shows the relationship between the loss function and the number of training epochs. With the increase in the number of iterations for the training dataset, the loss is decreased significantly and converged to around 0.005. We define the error on the Stokes vector reconstruction as \(\sigma =\textrm{arccos}(({S_1S^{'}_1+S_2S^{'}_2+ S_3S^{'}_3})/({S_0S^{'}_0}))\), where \(S_i\) and \(S^{'}_i\) (i=0,1,2,3) are input and predicted Stokes vector [23]. It is clear that \(\sigma\) corresponds to the angle between the input and the reconstructed Stokes vector on the PoincarÃ© sphere. Therefore, the proposed intelligent fullStokes polarimetry is capable of reconstructing the light polarization with a high accuracy and the average deviation angle \(\left\langle {\sigma }\right\rangle\) is as small as \(7^{\circ }\), which is comparable to the stateoftheart results of computational polarimetry [15, 23].
Characterizing performance of intelligent multipleOAM detection
Light with helical phase wavefront featuring exp(\(il\varphi\)) (where l and \(\varphi\) represent the topological charge and azimuthal angle, respectively) carries OAM, which has boosted advanced optical applications in communications, holography encryption, tweezers, highorder quantum entanglement and nonlinear optics [38,39,40,41,42]. The measurement of the topological charge of the OAM beam is a nontrivial task for many optical applications. The conventional measurements of topological charge are based on the deterministic interference and diffraction properties of OAM modes, which often require a precise alignment of the optical path and are difficult for compact integration [39, 43]. Recently, the specklebased deep learning methods for OAM mode classification are being developed, and they have found applications in the OAMmultiplexed data transmission [44,45,46,47,48]. However, it is still challenging to realize simultaneous recognition of multipleOAM states and their power spectra of different OAM channels, mainly limited by the singletask learning configuration, although the measurement of multipleOAM spectra is crucial for highcapacity information processing systems [49, 50].
Here we demonstrate a novel MTLbased intelligent multipleOAM sensing system to classify the multipleOAM states and estimate their power spectra as shown in Fig. 4a. The major components of MTLDNN consist of a fourlayer CNN and two independently taskspecific fullyconnected layers (FCs) for the classification of OAM modes and the regression of their power spectra. In essence, the feature maps extracted from the input speckle images via the fourlayer CNN are shared by the taskspecific FCs; meanwhile, the taskspecific FCs are used for learning a nonlinear mapping from features to individual targets. Several dropout layers are embedded between every twoÂ threelayer FCs to prevent overfitting and improve the generalization ability.
Experimentally, the calculated phase profile is loaded into a spatial light modulator (SLM), and the diffracted multipleOAM beam (Supplementary Section 5, Supplementary Fig. S7) is then launched to the disordered NLC medium to generate speckle patterns in the farfield regime. To reduce the performance degradation caused by discrete phase pattern and phaseonly modulation on generated collinearly multipleOAM beams, we design OAM base with a state interval of 2 for proofofconcept demonstration [51]. We recorded 25,000 speckle images covering 20 different multipleOAM states along with various power spectra. In the training stage, different multipleOAM light field information and their corresponding speckle images are used to train the MTLDNN model. Labeled samples are randomly split into training, validation and testing subsets in the ratio of 8:1:1. A MTL loss function is welldefined to adaptively balance the weights of two tasks during the training procedure. After the optimization process, the network is able to recover the multipleOAM states information directly from a single speckle pattern. From Fig. 4b, it shows that the loss function of the regression and classification tasks converge to \(\sim 3\times 10^{3}\) and \(\sim 1\times 10^{8}\), respectively. And thus the total loss function of the MLTDNN for multipleOAM recognition is \(\sim\)0.02. Figure 4c shows the truth table for multipleOAM states classification, and the recognition accuracy of twenty kinds of multipleOAM states reaches 100%. In addition, Fig. 4d shows a comparison between the reconstructed OAM power spectra and the input results. It can be found that all the components of predicted results agree well with the input values. As far as we know, the proposed intelligent lightfield sensor is for the first time demonstrated to realize the multipleOAM spectra resolution, which will stimulate applications in OAMbased holography, imaging and communications.
Intelligent singleshot reconstruction of fourdimensional light fields via integrated sensors
In this section, we discuss the potential of using disordered NLC film and MTLDNN to achieve the complete singleshot reconstruction of realistic fourdimensional light fields including polarization, multipleOAM, multiplewavelength, and power. The disordered NLC medium of high degrees of freedom provides rich scattering channels to the high dimension of light, which lays the foundation for direct singleshot recognition. We assemble the disordered liquid crystal cell and commercial imaging chip for proofofconcept demonstration, as shown in Fig. 5a. The highdimension light fields generated from the homemade optical system (Fig. 5b, see Methods) are directly imaged and decoded in the integrated sensors. We would like to note that the integrated sensing scheme significantly reduces the light propagation distance of the millimeterscale and influence the diffracted speckle patterns compared with that of longdistance diffraction of tens of centimeterscale (Supplementary Fig. S9), while their predicted results via the MTLDNN are comparable (Supplementary Fig. S10). The decoding of highdimensional properties of light from the singleshot speckles is not a trivial task considering the strong linear and nonlinear coupling in the light scattering and detection process. To cope with this difficulty, we continue to use the MTLDNN model for previous multipleOAM detection while making slight modifications in its output layers to meet specific requirements. We apply linear extrapolation and impose random noise on the expanded datasets. The signaltonoise ratio (SNR) is defined as \(\text {SNR}=10\lg {({\mu }/{I_s})}\), where \(\mu\) and I_{s} represent the variance of noise and intensity of original measured image. Here, random noise with a SNR of 515 dB is imposed to generate a total of 120,000 speckle images based on the original 336 measured images. The extrapolation is appropriate based on two methods: (1) The speckle images are scaled linearly with the excitation power; (2) Polychromatic speckles are considered as a linear superposition of monochromatic speckles. These techniques are widely used in machine learning studies when a larger training set was desired [15].
After training MTLDNN model, the loss function stops at \(\sim\)0.08 with classification loss of \(\sim\)0.03 and regression loss of \(\sim\)5\(\times 10^{3}\) (Fig. S8). Using the trained model, the input multidimensional light fields can be completely recovered from singleshot speckle images. FigureÂ 5cf show the comparison of input and predicted results of 16 typical states of highdimension light fields composing different combinations of polarization, multipleOAM, multiplewavelength and intensity, which clearly indicate the high recognition accuracy of the deep learning network. Since it is not intuitive to simultaneously illustrate the predicted results of such four properties of light, we have to fix one property of light and plot the remaining dimensions. To further validate the effectiveness of the proposed intelligent sensor, the mean squared error (MSE) between the input value and predicted value are implemented. The reconstruction error of polarization, power, wavelength and multipleOAM as a function of SNR for highdimension light is shown in Fig. 5g. The results indicate that the lightfield sensor can realize highdimension light field detection with high fidelity at a relatively high noise level (SNR \(\ge\)10 dB). In the current proofofconcept study, the states space of a fourdimensional light field is as high as 1680, spanning by 4 multipleOAMs, 2 OAMpowerspectra and 15 single/multiplewavelength and 14 polarizations, which can be well recognized with high accuracy. The states space can be readily extended (Supplementary Section 8) if more experimental data are included in the training network. As far as we know, this is the first demonstration of complete light field construction of four universal degrees of freedom, i.e., wavelength, polarization, OAM, and power via an integrated sensor, which provides a new paradigm shift in the complex lightfield manipulations and detections.
Conclusions
In summary, we have demonstrated an integrated intelligent fourdimensional light field sensor enabled by disordered anisotropic medium and deep learning. Our results unambiguously show that the complete information of light including wavelength, polarization, OAM, and power can be decoded simultaneously via a singleshot speckle image from the disordered anisotropic medium. Compared with the conventional light field sensing systems by cascading numerous bulky optoelectromechanical elements along the optical path, this developed concept of the intelligent optical sensor provides a disruptive technology for the ultimate multidimensional light field detection. The current disordered NLC film is fabricated via selfassembly, and we envision that for practical applications, the photopatterning technique can be implemented to generate arbitrarilydesigned mesostructured NLC film with high repeatability [52, 53]. On the other hand, the developed specklebased light fieldsensing configuration can also be transferred to the multimode fiber scheme [46, 54, 55], and the extended degrees of freedom for highdimension light encoding and decoding with ultrahigh data capacity can be realized. Furthermore, it is anticipated to construct a photonic neural network to implement machine learning, paving the way for onchip alloptical light field sensing.
Methods
Device fabrications
The nematic liquid crystal (NLC) film (E7, Jiangsu Hecheng Display Technology) is sandwiched between two glass plates coated with indiumtinoxide. The film thickness is controlled by spacers with a thickness of 20 \(\upmu\)m.
Optical measurements
The spatial director distributions of the NLC film is measured by a commercial birefringence imaging microscope (Crystalent50, Ningcui). The multidimensional light field is generated by a homebuilt optical system, which consists of a wavelengthtunable module, a phase modulation module and polarization tuning module. The wavelengthtunable module is a supercontinuum laser (YSL SCPro) filtered by an acousticoptical tunable filter (AOTF, YSL AOTFPro bandwidths \(\sim\)4 nm). The spatial light modulator (SLM, UPOlabs HDSLM80RPlus) is used to control the phase wavefront of the beam. The polarization state of light is modulated by a \(\lambda /2\) plate and a \(\lambda /4\) plate. The diffracted multidimensional light field in disordered NLC film is captured by a homebuilt imaging system (MIchrome 5 Pro, Shanghai Taizi Technology).
Light propagation in anisotropic disordered medium
We consider the Helmholtz equation that governs light propagation for the electric field \(\vec {E}_\omega (\vec {r})\):
where \(k_0\) is the vacuum wave number and dielectric function \(\varvec{\varepsilon }(\vec {r})=\varvec{\varepsilon }^r+\varvec{\varepsilon }^s(\vec {r})\) is embedded in an infinite homogeneous reference medium \(\varvec{\varepsilon }^r\). When an incident field \(\vec {E}^0_\omega\) is considered, the scattering problem can be treated by setting up equivalent relations for the incident field that satisfies: \(\nabla \times \nabla \times \vec {E}^0_\omega +k^2_0 \varvec{\varepsilon }^r \vec {E}^0_\omega =0\) in the homogeneous and sourcefree reference system. Introducing the operators \(\varvec{L}\), \(\varvec{e}_r\), and \(\varvec{e}_s\) for simplifying the notation \(\nabla \times \nabla \times\), \(k^2_0 \varvec{\varepsilon }^r\), and \(k^2_0 \varvec{\varepsilon }^s(\vec {r})\), respectively, we can rewrite the Eq. 2 as:
The incident light field \(\vec {E}^0_\omega\) in the homogeneous and sourcefree reference system satisfies:
The Greenâ€™s tensor \(\varvec{G}(\vec {r}, \vec {r}^{\prime }, \omega )\) describes the impulse response of a given physical system. This dyadic tensor is closely related to the field susceptibility of the system. The Greenâ€™s function \(\varvec{G}\) associated with the complete system is defined by
where \(\varvec{I}\) is the unit tensor. By setting Eq. 3 equal to Eq. 4 and using Eq. 5, we obtain the following formula:
We define the generalized propagator as \(\vec {K}(\vec {r},\vec {r}^{\prime },\omega )=1\varvec{G}\varvec{e}_s\). Thus the scattered field of Eq. 6 in the \(\vec {r}\) representation becomes [56]:
Data preparation
The color images collected by the camera will be converted into gray values according to the formula \(I(x,y) = 0.3I_R(x,y) +0.59I_G(x,y)+ 0.11I_B(x,y)\), where \(I_R\), \(I_G\), and \(I_B\) are red, green, and bluechannel of images, respectively. The converted and subsampled (408 \(\times\) 342 pixels) images are imported into the deeplearning network for training.
Availability of data and materials
All the data and codes are available from the corresponding author upon reasonable request.
Abbreviations
 OAM:

orbital angular momentum
 MTL:

multitasklearning
 DNN:

deep neural network
 NLC:

disordered nematic liquid crystals
 SSIM:

structural similarity
 CNN:

convolutional neural network
 FCs:

fullyconnected layer
 SLM:

spatial light modulator
 SNR:

signaltonoise ratio
 MSE:

mean squared error
 AOTF:

acousticoptical tunable filter
References
Liu Y, Zhang X. Metamaterials: a new frontier of science and technology. Chem Soc Rev. 2011;40(5):2494â€“507.
Chen H, Chan CT, Sheng P. Transformation optics and metamaterials. Nat Mater. 2010;9(5):387â€“96.
Yu N, Capasso F. Flat optics with designer metasurfaces. Nat Mater. 2014;13(2):139â€“50.
Meng Y, Chen Y, Lu L, Ding Y, Cusano A, Fan JA, et al. Optical metawaveguides for integrated photonics and beyond. Light Sci Appl. 2021;10:235.
Sun S, He Q, Hao J, Xiao S, Zhou L. Electromagnetic metasurfaces: physics and applications. Adv Opt Photon. 2019;11(2):380â€“479.
Yu N, Genevet P, Kats MA, Aieta F, Tetienne JP, Capasso F, et al. Light propagation with phase discontinuities: generalized laws of reflection and refraction. Science. 2011;334(6054):333â€“7.
Sun S, He Q, Xiao S, Xu Q, Li X, Zhou L. Gradientindex metasurfaces as a bridge linking propagating waves and surface waves. Nat Mater. 2012;11(5):426â€“31.
Song Q, Liu X, Qiu CW, Genevet P. Vectorial metasurface holography. Appl Phys Rev. 2022;9(1):011311.
Chen HT, Taylor AJ, Yu N. A review of metasurfaces: physics and applications. Rep Prog Phys. 2016;79(7):076401.
Li G, Zhang S, Zentgraf T. Nonlinear photonic metasurfaces. Nat Rev Mater. 2017;2:17010.
Chen WT, Zhu AY, Capasso F. Flat optics with dispersionengineered metasurfaces. Nat Rev Mater. 2020;5(8):604â€“20.
Qu Y, Yi S, Yang L, Yu Z. Multimodal lightsensing pixel arrays. Appl Phys Lett. 2022;121(4):040501.
Yuan S, Ma C, Fetaya E, Mueller T, Naveh D, Zhang F, et al. Geometric deep optical sensing. Science. 2023;379(6637):eade1220.
Xiong Y, Wang Y, Zhu R, Xu H, Wu C, Chen J, et al. Twisted black phosphorusâ€“based van der Waals stacks for fiberintegrated polarimeters. Sci Adv. 2022;8(18):eabo0375.
Ma C, Yuan S, Cheung P, Watanabe K, Taniguchi T, Zhang F, et al. Intelligent infrared sensing enabled by tunable moirÃ© quantum geometry. Nature. 2022;604(7905):266â€“72.
Yang Z, AlbrowOwen T, Cai W, Hasan T. Miniaturization of optical spectrometers. Science. 2021;371(6528):eabe0722.
Yi S, Zhou M, Yu Z, Fan P, Behdad N, Lin D, et al. Subwavelength anglesensing photodetectors inspired by directional hearing in small animals. Nat Nanotechnol. 2018;13(12):1143â€“7.
Ji Z, Liu W, Krylyuk S, Fan X, Zhang Z, Pan A, et al. Photocurrent detection of the orbital angular momentum of light. Science. 2020;368(6492):763â€“7.
Tittl A, Leitis A, Liu M, Yesilkoy F, Choi DY, Neshev DN, et al. Imagingbased molecular barcoding with pixelated dielectric metasurfaces. Science. 2018;360(6393):1105â€“9.
Wei J, Xu C, Dong B, Qiu CW, Lee C. Midinfrared semimetal polarization detectors with configurable polarity transition. Nat Photon. 2021;15(8):614â€“21.
Wang Z, Yi S, Chen A, Zhou M, Luk TS, James A, et al. Singleshot onchip spectral sensors based on photonic crystal slabs. Nat Commun. 2019;10:1020.
Pors A, Nielsen MG, Bozhevolnyi SI. Plasmonic metagratings for simultaneous determination of Stokes parameters. Optica. 2015;2(8):716â€“23.
Ni Y, Chen C, Wen S, Xue X, Sun L, Yang Y. Computational spectropolarimetry with a tunable liquid crystal metasurface. eLight. 2022;2:23.
Lukosz W. Optical systems with resolving powers exceeding the classical limit. J Opt Soc Am. 1966;56(11):1463â€“71.
Di Francia GT. Degrees of freedom of an image. J Opt Soc Am A. 1969;59(7):799â€“804.
Zuo C, Chen Q. Exploiting optical degrees of freedom for information multiplexing in diffractive neural networks. Light Sci Appl. 2022;11:208.
Bellini T, Clark NA, Degiorgio V, Mantegazza F, Natale G. Lightscattering measurement of the nematic correlation length in a liquid crystal with quenched disorder. Phys Rev E. 1998;57:2996â€“3006. https://doi.org/10.1103/PhysRevE.57.2996.
Wiersma DS. Disordered photonics. Nat Photon. 2013;7(3):188â€“96.
Gigan S. Imaging and computing with disorder. Nat Phys. 2022;18(9):980â€“5.
Vandenhende S, Georgoulis S, Van Gansbeke W, Proesmans M, Dai D, Van Gool L. Multitask learning for dense prediction tasks: A survey. IEEE Trans Pattern Anal Mach Intell. 2021;44(7):3614â€“33.
Thung KH, Wee CY. A brief review on multitask learning. Multimed Tools Appl. 2018;77(22):29705â€“25.
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13(4):600â€“12.
Ballard Z, Brown C, Madni AM, Ozcan A. Machine learning and computationenabled intelligent sensor design. Nat Mach Intell. 2021;3(7):556â€“65.
Maguid E, Yannai M, Faerman A, Yulevich I, Kleiner V, Hasman E. Disorderinduced optical transition from spin Hall to random Rashba effect. Science. 2017;358(6369):1411â€“5.
Kendall A, Gal Y, Cipolla R. Multitask learning using uncertainty to weigh losses for scene geometry and semantics. In: Proc. IEEE Conf. Comput. Vis. Pattern Recog. 2018. p. 7482â€“7491.
Rubin NA, Shi Z, Capasso F. Polarization in diffractive optics and metasurfaces. Adv Opt Photon. 2021;13(4):836â€“970.
Schaefer B, Collett E, Smyth R, Barrett D, Fraher B. Measuring the Stokes polarization parameters. Am J Phys. 2007;75(2):163â€“8.
Wang J, Yang JY, Fazal IM, Ahmed N, Yan Y, Huang H, et al. Terabit freespace data transmission employing orbital angular momentum multiplexing. Nat Photon. 2012;6(7):488â€“96.
Shen Y, Wang X, Xie Z, Min C, Fu X, Liu Q, et al. Optical vortices 30 years on: OAM manipulation from topological charge to multiple singularities. Light Sci Appl. 2019;8:90.
Padgett M, Bowman R. Tweezers with a twist. Nat Photon. 2011;5(6):343â€“8.
Fang X, Ren H, Gu M. Orbital angular momentum holography for highsecurity encryption. Nat Photon. 2020;14(2):102â€“8.
Ouyang X, Xu Y, Xian M, Feng Z, Zhu L, Cao Y, et al. Synthetic helical dichroism for sixdimensional optical orbital angular momentum multiplexing. Nat Photon. 2021;15(12):901â€“7.
Fu S, Zhai Y, Zhang J, Liu X, Song R, Zhou H, Gao C. Universal orbital angular momentum spectrum analyzer for beams. PhotoniX. 2020;1(19):1â€“12.
Raskatla V, Singh B, Patil S, Kumar V, Singh R. Specklebased deep learning approach for classification of orbital angular momentum modes. J Opt Soc Am A. 2022;39(4):759â€“65.
Liu Y, Zhang Z, Yu P, Wu Y, Wang Z, Li Y, et al. Learningenabled recovering scattered data from twisted light transmitted through a long standard multimode fiber. Appl Phys Lett. 2022;120(13):131101.
Zhao Q, Yu PP, Liu YF, Wang ZQ, Li YM, Gong L. Light field imaging through a single multimode fiber for OAMmultiplexed data transmission. Appl Phys Lett. 2020;116(18):181101.
Wang J, Fu S, Shang Z, Hai L, Gao C. Adjusted EfficientNet for the diagnostic of orbital angular momentum spectrum. Opt Lett. 2022;47(6):1419â€“22.
Fu S, Shang Z, Hai L, Huang L, Lv Y, Gao C. Orbital angular momentum comb generation from azimuthal binary phases. Adv Photon Nexus. 2022;1(1):016003.
Fang X, Ren H, Li K, Luan H, Hua Y, Zhang Q, et al. Nanophotonic manipulation of optical angular momentum for highdimensional information optics. Adv Opt Photon. 2021;13(4):772â€“833.
Willner AE, Huang H, Yan Y, Ren Y, Ahmed N, Xie G, et al. Optical communications using orbital angular momentum beams. Adv Opt Photon. 2015;7(1):66â€“106.
Du J, Wang J. Design of onchip Nfold orbital angular momentum multicasting using Vshaped antenna array. Sci Rep. 2015;5:9662.
Ma LL, Li CY, Pan JT, Ji YE, Jiang C, Zheng R, Wang ZY, Wang Y, Li BX, Lu YQ. Selfassembled liquid crystal architectures for soft matter photonics. Light Sci Appl. 2022;11:270.
Guo Y, Jiang M, Peng C, Sun K, Yaroshchuk O, Lavrentovich O, Wei QH. Highresolution and highthroughput plasmonic photopatterning of complex molecular orientations in liquid crystals. Adv Mater. 2016;28(12):2353â€“8.
Liu Z, Wang L, Meng Y, He T, He S, Yang Y, etÂ al. Allfiber highspeed image detection enabled by deep learning. Nat Commun. 2022;13:1433.
Tang P, Zheng K, Yuan W, Pan T, Xu Y, Fu S, et al. Learning to transmit images through optical speckle of a multimode fiber with high fidelity. Appl Phys Lett. 2022;121(8):081107.
Martin OJF, Girard C, Dereux A. Generalized Field Propagator for Electromagnetic Scattering and Light Confinement. Phys Rev Lett. 1995;74:526â€“9. https://doi.org/10.1103/PhysRevLett.74.526.
Acknowledgements
The authors thank Prof. Jun Yi for experimental assistance and Mr. Ke Cheng for his help with the preparation of illustration.
Funding
This project was supported by National Key Research and Development Program of China (2022YFA1203700), Natural Science Foundation of Fujian Province of China (2023J06011), National Natural Science Foundation of China (Grants No. 12274357, 62005231, 62075186, 62005164), Fundamental Research Funds for the Central Universities (20720210045), Guangdong Basic and Applied Basic Research Foundation (2021A1515012199), Shanghai RisingStar Program (20QA1404100), Science and Technology Commission of Shanghai Municipality (Grant No. 21DZ1100500), Shanghai Municipal Science and Technology Major Project, Shanghai Frontiers Science Center Program (20212025 No. 20) and Zhangjiang National Innovation Demonstration Zone (Grant No. ZJ2019ZD005).
Author information
Authors and Affiliations
Contributions
S.k. Z. and Z.h. Z. contributed equally to this work. S.k. Z. and S.s. C. fabricated the device, S.k. Z. performed the experiment, S.k. Z., Z.h. Z. and J.h. C. built the theoretical model and algorithms, S.k. Z., Z.h. Z., X. F., and J.h. C. analyzed the results, J.h C. conceived the idea and cosupervised the project with L.J. C. and X. F. All authors contributed to the discussion, analyzed the data, and wrote the manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisherâ€™s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Additional file 1.
Including Supplementary Section 18, Fig.S111 and Table S1. The Supplementary Information is organized as follows. In Section 1, the optical characterizations and measurements of disordered nematic liquid crystal (NLC) is discussed. It includes multidimensional light field generation, speckle pattern detection and the NLC director distribution measurement. In Section 2, we introduce the numerical modeling of the speckles diffracted from disordered NLC medium. In Section 3, structural similarity (SSIM) index to evaluate the similarity of speckle patterns are discussed. In Section 4, analysis of diffraction patterns from uniform to full disorder for highdimension light based on SSIM index are discussed. In Section 5, the algorithm for generation multipleOAM modes are discussed. In Section 6, MTLDNN structure and training processes are explicitly discussed. In Section 7, The influence of distance between camera and disordered NLC film is discussed. In Section 8, we discuss fundamental limit of intelligent light field sensor.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Zhu, Sk., Zheng, Zh., Meng, W. et al. Harnessing disordered photonics via multitask learning towards intelligent fourdimensional light field sensors. PhotoniX 4, 26 (2023). https://doi.org/10.1186/s43074023001027
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1186/s43074023001027