Enhancing Intrinsic Image Decomposition with Transformer and Laplacian Pyramid Network

IID has made great progress. In 1971, Land and McCann [1] proposed the Retinex Theory. The Retinex Theory, a portmanteau of “retina” and "cortex," suggests that color perception is not solely determined by the wavelength of light entering the eyes. Instead, it proposes that the brain compares and processes the light reflected from different parts of a scene to comprehend the color and brightness consistently. This process helps in maintaining color constancy under varying shading conditions. The Retinex theory differentiates between shading and reflectance components in an image by analyzing gradients [1, 2]. Barrow et al. [3] explored the idea of decomposing an image into its intrinsic components. Eq. (1) encapsulates this idea, where I represents the observed image:

$I=R * S$             (1)

R is the reflectance image, indicating the intrinsic color properties of the objects in the scene, independent of the illumination conditions; S is the shading image, which contains the incident illumination and the shape information. Reflectance images and shading images have great significance in other research fields. Reflectance has been found to enhance various image processing and computer vision tasks. In particular, it can greatly improve the performance of semantic segmentation [4], help find attempts to spoof face recognition systems [5], lead to more in-depth facial intrinsic analysis [6], and make photo editing better [7]. On the other hand, shading is particularly useful in extracting shape information from images (shape from shading) [8], assisting in relighting tasks for more realistic lighting effects [9], and playing a critical role in 3D reconstruction processes [10]. Each of these applications leverages the unique properties of reflectance and shading to improve the quality and accuracy of the results. Ma et al. [11] comprehensively examine various approaches and methodologies developed over the years for separating the reflectance and illumination components of an image. It likely covers the theoretical underpinnings of the technique, evaluates different algorithms, and discusses their applications and limitations.

It can be known from Eq. (1) that the IID is ill-posed. Many solutions were proposed by researchers to address this problem. Before deep learning and neural networks became mainstream, IID mainly relied on some traditional computer vision techniques. These methods are usually based on the physical properties of the image, heuristic rules, or statistical models [12-16]. These methods also have problems. Different scenes may require different physical constraints, and selecting or adjusting these constraints can be difficult without prior knowledge about the scene. In dynamic or changing environments, such as outdoor scenes or changing shading conditions, physical constraints may need to be adjusted in real time, which increases the complexity of the algorithm. Deep learning models can better generalize to different images and scenes after being trained on large and diverse data, rather than being limited to specific physical conditions or constraints. Deep learning and neural networks have developed vigorously in recent years [16, 17]. In this field of IID, there are two main methods: unsupervised learning [18, 19] and supervised learning [20-24]. Unsupervised learning does not rely on labeled training data. It learns image decomposition by exploring the intrinsic structure of the data. Supervised learning relies on labeled training data. These labels guide the learning process to ensure that the decomposition results match the expected output.

In this paper, we propose IID using a TLPNet. First, the application of Transformer in this field of image processing is gradually increasing. On the one hand, Transformer is able to capture global dependencies through the self-attention mechanism, and even regions that are far apart in the image can directly influence each other. On the other hand, the Transformer architecture performs well when handling large-scale data sets, especially when large amounts of training data are available. As model size increases, transformers are often better able to utilize the additional data to improve performance. In addition, we also use the Laplacian pyramid to improve the effect of the shading image. Laplacian pyramid by representing images at different scales. This method allows the shading image to extract image features from coarse to fine. The Laplacian pyramid can be used to enhance the details of an image, such as by sharpening edges and textures. In summary, our contributions are as follows:

●We apply Transformer to the encoder of IID, which can better extract features from the image. Using transformers can often make better use of the extra data to improve performance.

●When the Laplacian pyramid is used to process shading images, its multiscale representation is used to improve image features step by step from big to small, especially by sharpening edges and textures. This makes the shading image better overall and better at capturing shading details.

●Multiple loss functions, which include mean square error (MSE) and cosine similarity error (CSE), are applied in our TLPNet. The loss of reflectance image, shading image, and reconstruction image have different weights.

3.1 Architecture network

Currently, there are two different network structures in deep learning for IID. One is the parallel structure, with the reflectance image and shading image using the same encoder to extract image features but using different decoders to generate the reflectance image and shading image. Another one is serial structure [22], which uses two encoders and decoders to reconstruct the reflectance image and the shading image.

11.png

Figure 1. Architecture network of TLPNet

As shown in Figure 1, we propose TLPNet, which adopts a similar structure to CasQNet. The reason we use a serial network structure is that the reflection image generated by the Transformer network is better. First, we use a Transformer encoder to extract image features; it generates a reflectance image from a natural image. Second, we use the outcome of the quotient of the natural image and the reflectance image to splice the natural image. We use the splicing outcome to extract image features to get the shading image.

3.2 Transformer for reflectance

TRNet is designed for constructing reflection images from nature images. The architecture is shown in Figure 2. First of all, we use a transformer encoder to exact the features from the nature image. Transformer encoders can catch more features from nature images than other neural networks. Second, we down-sample the reflectance image. We get the reflectance images of 1/2, 1/4, 1/8, and 1/16 sizes to prepare for obtaining the shading image. Figure 3 has more detail about the Transformer encoder and decoder. The Transformer block in Figure 3 is shown in Figure 4.

2.png

Figure 2. Architecture network of TRNet

3.png

Figure 3. Architecture network of transformer encoder and decoder

4.png

Figure 4. Architecture network of transformer block

3.3 Laplacian pyramid for shading

In the process of predicting shading images, we first use Eq. (2) to perform quotient operations on original images and reflectance images of different sizes. Since the predicted reflection image is not completely accurate, this will cause the edges of the shading map to be unsmooth and the texture to be unclear. We call this result the temporary shading image. In order to solve the problems raised above, we applied the Laplacian Pyramid to our model. Laplacian Pyramid is a technique used in image processing, mainly for multi-scale decomposition of images. It is a further development based on the Gaussian Pyramid and is used to capture details in images in greater detail. Before extracting shading features, we concatenate the temporary shading map and the Laplacian map of corresponding sizes. This can better preserve important information in the image. Figure 5 also shows the process of shading the image from coarse to fine. In Figure 5, Ri (i from 2 to 5) is the different size of reflectance image, R size is the same to the nature image; Di (i from 0 to 4) is the different size of temporary shading image; and L is the result of the different sizes of Laplacian pyramid. The highest level of the shading image is as follows:

$S=I / R$    (2)

$S=S_k+U p\left(S_{k+1}\right), k=0,1,2,3$      (3)

S4 contains the global layout of the shading map at the image pyramid. By iteratively computing Eq. (3) with the order of k = 3 → 2 → 1 → 0, S is computed as the final shading image.

5.png

Figure 5. Architecture network of LPSNet

3.4 Loss function

The trainable parameters of the TLIID network are optimized based on three loss functions, which include L_R, L_S, and L_C, as follows:

$L_t=\left\{\begin{array}{cc}L_R & \text { epoch }<20 \\ L_S & 20 \leq \text { е р о } h<40 \\ \lambda_R L_R+\lambda_S L_S+\lambda_I L_C & 40 \leq \text { е р } \text { с } h<60\end{array}\right.$                (4)

The loss function of $L_R$ is designed for TRNet, $L_S$ is designed for LPSNet. Also, we design $L_C$ for the product of the predicting $\mathrm{R}$ and $\mathrm{S}$. In training process, the parameters are empirically set as: $\lambda_R=1.0 ; \lambda_S=1.0 ; \lambda_I=0.25$; We can conclude from $\mathrm{I}=\mathrm{R} * \mathrm{~S}, \mathrm{R}$ and $\mathrm{S}$ are two parts of $\mathrm{I}, \mathrm{R}$ and $\mathrm{S}$ perform multiplication operations, resulting in $\lambda_I=0.5 * 0.5$.

The effect of reflection images is an important section on IID. The reflectance loss function is the most important part to impact reflection images. Therefore, we use $L_{R m s e}$ and $L_{R c s e}$ to compose the reflectance loss, which are defined as follows:

$L_R=L_{R m s e}+L_{\text {Rcse }}$           (5)

$L_{R m s e}=\operatorname{MSE}(R, \hat{R})=\frac{1}{n} \sum_{i=0}^n\left(R_i-\widehat{R_l}\right)^2$        (6)

$L_{c s e}(R, \hat{R})=\frac{R * \hat{R}}{\|R\|\|\hat{R}\|}=\frac{\sum_{i=0}^n\left(R_i * \widehat{R}_l\right)}{\sqrt{\sum_{i=0}^n R_i^2} * \sqrt{\sum_{i=0}^n \widehat{R}_l^2}}$   (7)

$L_{R c s e}=1-\frac{L_{c s e}(R, \hat{R})+1}{2}$           (8)

where, $R_i$ represents the actual point values and $\widehat{R}_l$ represents the predicted point values. For each data point, subtract the predicted value from the actual value. $\mathrm{n}$ is the quantity of data points. Cos Similarity Error (CSE) $\in[-1,1]$, In order to achieve the best effect when $L_{\text {Rcse }}$ is 0 . We did some processing on the CSE in Eq. (8).

For the LPSNet, the loss function is composed of MSE and CSE for predicting shading images. The shading loss is as follows:

$L_S=L_{S m s e}+L_{S c s e}$               (9)

$L_{S m s e}=\operatorname{MSE}(S, \hat{S})=\frac{1}{n} \sum_{i=0}^n\left(S_i-\widehat{S_l}\right)^2$            (10)

$L_{c s e}(S, \hat{S})=\frac{S * \hat{S}}{\|S\|\|\hat{S}\|}=\frac{\sum_{i=0}^n\left(S_i * \widehat{S}_l\right)}{\sqrt{\sum_{i=0}^n S_i^2} * \sqrt{\sum_{i=0}^n \widehat{S}_l^2}}$      (11)

$L_{S c s e}=1-\frac{L_{c s e}(S, \hat{S})+1}{2}$       (12)

where, $L_{\text {Smse }}$ is the MSE of shading image, $L_{\text {Scse }}$ is the CSE of the shading image. $\mathrm{S}$ is the ground-truth image. $\hat{S}$ is the estimated shading image.

According to $\hat{I}=\hat{R} * \hat{S}$, we estimate $\hat{I}$ by using $\hat{R}$ and $\hat{S}$. Reconstruction image loss functions are also composed of MSE and CSE. The reconstruction image loss as follows:

$L_I=L_{\text {Imse }}+L_{\text {Icse }}$                   (13)

$L_{\text {Imse }}=\operatorname{MSE}(I, \hat{I})=\frac{1}{n} \sum_{i=0}^n\left(I_i-\widehat{I}_l\right)^2$                   (14)

$L_{\text {Icse }}(I, \hat{I})=\frac{I * \hat{I}}{\|I\|\|\hat{I}\|}=\frac{\sum_{i=0}^n\left(I_i * \widehat{I}\right)}{\sqrt{\sum_{i=0}^n I_i^2} * \sqrt{\sum_{i=0}^n \widehat{I}_l^2}}$           (15)

$L_{\text {Icse }}=1-\frac{L_{c s e}(I, \hat{I})+1}{2}$            (16)

where, $L_{\text {Imse }}$ is the MSE of the reconstruction image, $L_{\text {Icse }}$ is the CSE of the reconstruction image.

The loss function was designed using MSE and CSE. MSE can compare pixel values directly. It is sensitive to noise and outliers. CSE focuses on directional similarity rather than size. CSE is more in line with human perceptions of image similarity.

In this work, we present a novel model called TLPNet for IID. TLPNet is a hybrid transformer for reflectance and a Laplacian Pyramid for shading. We train both sub-nets jointly using our proposed special weight loss function. The TLPNet estimated reflectance image and shading image are substantially more consistent than the state-of-the art baselines, both locally and globally. The result image of different algorithms on the ShapeNet dataset is shown in Figure 7. From Figure 7, the first row is the clean image from the ShapeNet dataset. For each sample, R is the estimated reflectance and S is the estimated shading. Ranjit is the algorithm from the study [45]. Carega is the algorithm from the study [46].

Nevertheless, there is still future work to consider. Our model struggles to accurately estimate reflectance and shading in DSSIM. Besides, as the training data mainly consists of indoor scenes, it seems harder to deal with outdoor scenes. Further research could focus on improving high-quality reconstruction in these two aspects. If we have more datasets under water, we would apply the method in the study [47] to get reflectance and shading images under water.

7.png

Figure 7. The result image of different algorithms on the ShapeNet dataset