3D polarimetric integral imaging in low illumination conditions

We overview a previously reported three-dimensional (3D) polarimetric integral imaging method and algorithms for extracting 3D polarimetric information in low light environment. 3D integral imaging reconstruction algorithm is first performed to the originally captured two-dimensional (2D) polarimetric images. The signal-to-noise ratio (SNR) of the 3D reconstructed polarimetric image is enhanced comparing with the 2D images. The Stokes polarization parameters are measured and applied for the calculation of the 3D volumetric degree of polarization (DoP) image of the scene. Statistical analysis on the 3D DoP can extract the polarimetric properties of the scene. Experimental results verified the proposed method out performs the conventional 2D polarimetric imaging in low illumination environment.


INTRODUCTION
Conventional imaging technologies can record the light intensity to provide information of the scene. In order to reveal additional information, multidimensional optical imaging and sensing technologies and systems [1] have be researched for the past decades. Among various degree of freedoms, the three-dimensional (3D) imaging and polarimetric imaging have drawn attentions by worldwide research groups.
As one of the promising 3D sensing and imaging technologies, integral imaging (InIm) [2] has been proposed by Lippman about one hundred years ago. A conventional integral imaging system obtains information of a 3D scene from multiple perspectives by placing a lenslet array in front of a 2D image sensor. The 3D reconstruction process is the reverse of the 3D sensing by illuminating the captured multi-perspective 2D images, also named as elemental images, in front of a lenslet array to form real 3D images in the imaging space [3] [4]. In addition, the computational reconstruction is further developed with volumetric 3D images corresponding to a series of in-focused planes. Nowadays, integral imaging has been widely applied for 3D display, object recognition, augmented reality, biomedical applications, sensing and imaging under degraded conditions, [5]- [8]etc.
As a fundamental property in electromagnetic fields, the polarization state of light may represent the physical and optical properties of material surface [9]. Polarimetric imaging can provide a visual extension for conventional imagery, and it has the advantage to reveal additional information of the object of interest comparing with the intensity-based imaging technologies. The potential applications of polarimetric imaging include object recognition, material inspection and classification for remote sensing, security and manufacture [10]- [12].
In this overview paper, we present a recently reported approach for polarimetric imaging under low illumination conditions using integral imaging [16]. With the scheme of synthetic aperture integral imaging [17], 3D information under low light environment is captured by a moving polarimetric image sensor, which includes a camera attached with a linear polarizer, to obtain multi-perspective 2D polarimetric images. Our approach will first reduce the camera dark offset from each 2D image, then the 3D computational reconstruction is performed to further mitigate the effect of camera read noise and extend the polarimetric information into 3D space. The generated Stokes polarimetric images are further utilized to generate 3D Degree of Polarization (DoP) image enhanced signal-to-noise ratio (SNR). Finally, total variation (TV) denoising algorithm is applied to the 3D DoP image. Comparisional experiments show that the DoP image with the proposed approach can reveal accurate polarimetric properties of the scene under low illumination conditions. However, the conventional 2D / 3D polarimetric imaging approaches false on it.

Polarimetric integral imaging and Degree of Polarization
To create the polarimetric image, the Stokes polarization parameters (Si, i=[0, 1, 2, 3]) need to be obtained [14][16]: where E0x and E0y are the instantaneous (time average) amplitudes of the x and y components in electric field, δ is the instantaneous phase of plane wave. The Degree of Polarization (DoP) can be derived by the Stokes polarization parameters: where DoP is between [0, 1]. Furthermore, we can decompose the DoP into (1), the Stokes parameters can be measured by the captured polarimetric images [I α° and I α°, π/2 ]. I α° represents the linear polarizer with an angle of α°, which corresponding to the x axis for measuring the linear polarimetric components. I α°, /2 represents that a quarter waveplate attached to the polarizer for measuring the circular polarimetric component (S3). A total of six sets of polarimetric images are required for polarimetric imaging. Integral imaging based 3D polarimetric imaging has been presented in [14]. However, under low illumination conditions, the conventional imaging approaches may not be able to reveal the accurate information, because the images are read noise dominated. In the proposed method, we apply synthetic aperture integral imaging [17] for polarimetric 3D imaging by replacing the lenslet array with a moving camera. As shown in Fig. 1(a)(c), a moving camera attached with a linear polarizer and quarter waveplate are used.

Remove camera dark offset and 3D reconstruction
The camera dark offset is due to the electrical system of the digital camera. When we preform 3D imaging under low illumination conditions, the signal intensity is very weak and camera dark offset may lead the image with a relatively low dynamic range. To reduce the effect from the camera dark offset, we first reduce the offset noise from the captured 2D elemental images. The camera dark offset can be measured by setting the camera sensor with a minimum exposure time and a maximum f-stop of camera to capture bias frames. The measured offset is named as a bias image, which can be obtained by averaging over 100 single bias frames: = 1⁄ ∑ , =1 ( = 1,2, … , ; ≥ 100) where biasi is the i-th bias frame. With the reduction of the camera dark offset, the camera read noise will dominate in the processed polarimetric images. The perspective 2D polarimetric images ,° with read noise can be expressed as , = , + , where , is the (m, n)-th ideal elemental image, is the additive read noise. Note that the image may also contain dark current noise, because the camera exposure time is not zero. However, the dark current noise is much smaller than read noise. To reduce the read noise, SAII reconstruction is applied to the polarimetric elemental images without the dark offset noise. The SAII reconstruction can be expressed as: where (x, y) is the pixel index of a 2D image, Z is the 3D reconstructed depth, g is the focal length of the camera. , and (cx, cy) and (px, py) are the sensor size and period between adjacent sensors, respectively.

Statistic approach for extracting polarimetric information
We proposed a statistic approach for extracting the polarimetric information from 3D DoP image. Based on Eq.1, the Stokes parameter matrices ( ) indicate the intensity differences between orthogonal polarimetric images. In low illumination condition, the images are read noise dominated, and the read noise follows zero mean Gaussian distribution In the 3D DoP image, the statistic distribution of the Stokes parameters in (i) the out-of-focus depth and (ii) the nonpolarimetric in focus material area will follow Chi distribution [~N (0, 2 )]. Furthermore, if the material only with linear polarized property, which is common in the nature, Xz will follow the Rayleigh distribution. However, Xz will not follow a Chi distribution when the Stokes parameters are measured from an in-focus depth of a polarimetric material [~N (μ, 2 ), μ≠0]. The depth information of the polarimetric materials can be measured by the relative entropy of intensity distributions in the 3D DoP images along depth. The relative entropy is calculated by applying the Kullback-Leibler Divergence.

EXPERIMENTAL RESULTS
We performed indoor low illumination 3D polarimetric integral imaging experiments to verify the proposed method. In the experiments, the digital sensor used in the experiments is with a resolution of 2048 pixels (H) × 2048 pixels (V), and its sensor pixel size is 6.5 μm (H) × 6.5 μm (V). As shown in Fig. 2, the 3D scene includes a mannequin, and a linear polarizer sheet was attached on it. SAII based polarimetric imaging and reconstruction were performed with a total of 49 [7 (H) × 7(V)] perspectives, the pitch between adjacent perspectives is 30 mm in both directions. Figs. 2(a) illustrate the original 2D elemental image with offset noise and the corresponding reconstructed images at different focus distances. The corresponding data with reduced offset noise is shown in Fig. 2(b). As discussed in Section 2.2, the camera dark offset is subtracted from the captured 2D images. 100 bias frames were recorded with a minimum camera exposure time of 3ms and a maximum f-stop. In addition, we estimate the number of photons falling on the image sensor by using the derived image without camera offset. The flow for photon estimation is first convert the intensity image to electrons by the conversion gain of the camera (conversion coefficient), then we divide the converted electrons with the quantum efficiency (QE) to achieve the photons / pixel recorded by the sensor.
In the experiment, the estimated number of photons per pixel falling on the image sensor is 1.24. We further calculate the SNR by selecting identical areas on both the 2D and 3D images. The SNR can be derived as: where , are the mean value of object and background areas, respectively. 2 , 2 are the variance of object and background areas, respectively. The data are extracted from the red and green boxes shown in Fig.2.    Fig. 3c (ii)]. The histograms of the random variables derived from the 3D un-normalized DoP is illustrated in Figure 4(a)-(b) for depths of 3.2 meters and 5.2 meters, respectively. Fig. 4(c) depicts the KL Divergence curve between the polarimetric and non-polarimetric areas along the depth range.

CONCLUSION
In this letter, we have demonstrated a previously reported approach for effectively extracting 3D polarimetric information under low illumination conditions. 3D Stokes polarization parameters are obtained by the integral imaging technology and the DoP images are calculated without the effect of camera offset. Total variation algorithm is further applied to enhance the DoP image quality. We also developed a statistical approach to analyze 3D polarimetric properties in the scene. Experimental results verified that the proposed approach can extract 3D polarimetric information in low light conditions, and it outperforms the 2D polarimetric imaging which performed poorly in these conditions.