Hybrid Deep Learning Approach for Low-Light Image Enhancement Based on Attention-Guided Residual Networks

In nighttime urban road surveillance images, both bright areas under streetlights and dark regions in the shadows coexist. When traditional methods directly enhance the entire image, overexposure in the bright regions and blurry details in the dark areas often occur. To address the coupled nature of illumination and detail information in low-light images, this paper proposes decomposing the original image into illumination and reflectance maps, incorporating the design of residual modules. By using a decomposition network to separate lighting information from detail information, targeted processing can be achieved. The illumination map focuses on adjusting light distribution, preventing detail loss due to overall brightness enhancement, while the reflectance map focuses on preserving object textures, ensuring that details are not drowned out during the illumination adjustment. The introduction of residual modules effectively alleviates the gradient vanishing problem during deep network training, improving the decomposition accuracy.

To address the issues of uneven noise distribution, color bias, and significant differences in light intensity in low-light images, this paper designs a collaborative recovery network and adjustment network embedded with multi-scale attention modules. For low-light medical images, for example, the image may not only be overall dark due to insufficient exposure but also have local spots due to device noise, while fine textures in the lesion areas need to be preserved accurately. The multi-scale attention module in the recovery network adaptively focuses on key regions: when suppressing noise, higher attention weights are assigned to denser spot areas to strengthen denoising; during color correction, color repair is enhanced in regions with color bias. The adjustment network can then finely tune the illumination according to different scene requirements, such as appropriately increasing the brightness of the bone area in X-ray images to highlight structures, while maintaining a soft light around the surrounding soft tissue areas to avoid overexposure. The final fusion step combines the processed reflectance and illumination maps, ensuring clear details while achieving natural overall light balance, resolving the contradiction in traditional methods where “denoising results in detail loss” and “brightness enhancement leads to color distortion.”

2.1 Image decomposition

The design of the image decomposition subtask primarily follows the core idea of the Retinex theory, which states that an image can be decomposed into reflectance and illumination components. The reflectance component is determined by the intrinsic properties of objects and is unaffected by fluctuations in lighting intensity. For example, in nighttime alley surveillance images, the brick texture of walls, the contours of trash bins, and other detailed information belong to the reflectance component. Even with changes in light intensity, the inherent characteristics of these objects should remain stable. The illumination component, on the other hand, is determined by the distribution of light and is independent of the objects themselves, such as the strong light in areas directly illuminated by streetlights or the shadows formed by buildings in the same scene. It only reflects the intensity and distribution of light. Based on this, decomposing a low-light image into these two components allows for targeted enhancement: the reflectance component focuses on detail preservation and optimization, avoiding distortion during lighting adjustments; the illumination component focuses on light balancing and correction, preventing local overexposure or underexposure due to overall processing, thereby laying a foundation for subsequent precise enhancement. Let T represent the original image, U represent the illumination map, and E represent the reflectance map. The matrix multiplication is denoted by ∘, and the expression is:

$T=U \circ E$           (1)

To enhance the feature extraction ability during the image decomposition phase, this subtask adopts the U-Net architecture, which fuses low-level and high-level features through skip connections. The contracting path uses 3×3 convolutions, Rectified Linear Unit (ReLU) activation functions, and 2×2 max-pooling operations, with the channel count doubling at each downsampling step to capture more abstract features. For example, when processing low-light indoor scene images, the contracting path gradually extracts high-level features such as the overall outline of furniture and the light-shadow transition on walls. After downsampling, three consecutive residual blocks are introduced to retain low-level color information, edge lines, and other detail features, avoiding feature loss during deep network training. The upsampling stage halves the number of channels and fuses the high-resolution low-level features from the contracting path with the high-level features obtained from upsampling through skip connections. Finally, a 3×3 convolution and sigmoid function output the three-channel reflectance map and the single-channel illumination map, ensuring that the decomposition result contains both global illumination information and local details. Figure 1 shows the image decomposition process.

1.png

Figure 1. Image decomposition process

The special structure of the residual block further optimizes the decomposition performance. It uses a combination of two 1×1 convolution kernels and one 3×3 convolution kernel, rather than the conventional two 3×3 convolution kernels, significantly reducing computational costs. For example, when processing low-light night scene images with complex light sources, the input features are first reduced in dimension by the 1×1 convolution layer to decrease feature dimensions and reduce computation. Then, the 3×3 convolution layer extracts key features, such as the spectral characteristics of different light sources and the diffusion range of light. Finally, the 1×1 convolution layer restores the dimensions, significantly reducing the number of parameters and computation time while ensuring feature extraction accuracy. Figure 2 shows the residual block structure.

2.png

Figure 2. Residual block structure

2.2 Illumination information processing

In the illumination component processing subtask, the enhancement network performs feature extraction through six convolutional activation layers, a convolutional layer, and a sigmoid layer to fuse features. Skip connections are introduced to retain low-level information, ensuring the network optimization efficiency while enhancing image contrast. The six convolutional activation layers progressively delve into the illumination features in the illumination map. For example, when processing low-light indoor living room images, the earlier convolution layers capture basic lighting distribution features, such as direct lighting areas and shadows cast by furniture, while later layers extract more complex lighting gradients and lighting interactions. Through hierarchical feature extraction, precise illumination details can be captured. The convolutional layer and sigmoid layer fusion operation integrates multi-dimensional illumination features into a unified illumination adjustment mode, avoiding processing efficiency decline due to feature redundancy. Skip connections merge the original illumination map information into the final convolution layer, effectively preserving low-level illumination details and preventing distortion of lighting features in the deep network process. For instance, when enhancing the brightness of a dark corner in a living room, it prevents the loss of light and shadow contours of baseboards due to excessive adjustments, ensuring that the contrast is enhanced while maintaining natural lighting changes and complete details. Figure 3 shows the illumination information processing flow.

3.png

Figure 3. Illumination information processing flow

To adapt to the complex and dynamic lighting conditions in real-world scenarios, this subtask uses the ratio of illumination intensity between the normal light image and the low-light image as the input to the enhancement network. The enhancement ratio can be set by the user, significantly improving the flexibility of the algorithm. For example, when processing nighttime road surveillance images under different lighting conditions: in a scene with a faint moonlit sky, the illumination ratio between the normal light and low-light image is small, allowing the network to perform a mild brightness enhancement to avoid excessive reflection on the road surface; in a heavy rain nighttime scene with severe light scattering, the ratio is larger, enabling the network to enhance the brightness adjustment more strongly to ensure clear visibility of road markings. At the same time, users can flexibly set the enhancement ratio according to actual needs. For instance, when identifying distant vehicle license plates, the lighting ratio can be increased to improve the brightness of distant areas; when capturing pedestrians in close-up shots, the ratio can be lowered to avoid overexposure of faces. Let U_g represent the illumination intensity of the normal light, U₁ represent the illumination intensity of the low-light image, and the illumination intensity ratio can be calculated using the following formula:

$\beta=\frac{U_g}{U_1}$           (2)

2.3 Detail information processing

The core of the reflectance component processing subtask is to coordinate the enhancement of weak-light areas and suppress the degradation interference of dark areas, avoiding detail loss or insufficient smoothing caused by a dominant single task. The reflectance component carries the intrinsic details of objects, which, in low-light scenes, are often blurred in weak-light areas, while dark areas are prone to degradation interference such as noise and blurring. If the focus is solely on removing degradation, subtle features in the weak-light areas may be overly smoothed, leading to detail loss. On the other hand, if the weak-light areas are simply enhanced, the noise in the dark areas may be amplified, causing distortion. Therefore, this subtask treats removing degradation and preserving detail information as a collaborative task. For example, when processing a low-light indoor image, for the text on the book covers on a bookshelf, the clarity must be enhanced while simultaneously suppressing the noise-induced artifacts in the shadows of the bookshelf, such as wrinkles in paper. By dynamically balancing the relationship between these tasks, the reflectance component can both present rich details and maintain visual smoothness.

4.png

Figure 4. Detail information processing flow

The recovery network uses ResNet as the backbone network, with an encoder-decoder structure and attention modules to achieve precise processing of the reflectance component. The encoder extracts features at different resolutions through multiple rounds of convolution and downsampling, increasing the number of channels to capture more complex feature patterns. The decoder restores the resolution via upsampling and uses skip connections to concatenate the feature maps of corresponding layers from the encoder, fusing high-resolution low-level details with abstract high-level features, preventing information loss during feature transmission. During the downsampling process, embedded illumination attention blocks and multi-scale attention blocks further optimize the processing effect. The illumination attention block guides the network to focus on severely degraded regions and enhances their feature weights, while the multi-scale attention block extracts features at different scales to assist in color correction and detail restoration. For example, when processing low-light remote sensing images of crop areas, the leaf texture features extracted by the encoder are restored by the decoder, and through skip connections, the fine structures of the leaf veins are preserved. The attention modules enhance the color authenticity of the weak-light leaf areas while suppressing the noise interference in the dark soil areas. The final output is a high-quality reflectance component. Figure 4 illustrates the detail information processing flow.

Specifically, in this paper, the setting of the illumination attention block in the reflectance component processing subtask focuses on dynamically capturing the image illumination distribution to achieve differentiated processing for regions with different degradation complexities, so as to solve the problem that uniform processing under low-light conditions easily leads to overexposure and halo artifacts. In low-light scenarios, the degradation degree of the reflectance component varies significantly with the illumination intensity: for example, in nighttime intersection images, the road surface area directly illuminated by street lamps has relatively sufficient illumination, and the degradation of the reflectance component is mainly mild noise; while the sidewalk area far from the light source has extremely weak illumination, and the reflectance component not only has blurred details but also severe noise and color deviation; the semi-shadow area near traffic lights exhibits local blurring and edge distortion caused by uneven illumination. If the same processing strategy is applied to these regions, enhancing weakly lit areas may cause overexposure in strongly lit areas, while suppressing degradation in strongly lit areas may aggravate detail loss in weakly lit areas. The illumination attention block captures the illumination intensity distribution of each region accurately by reducing the dimensionality of the illumination features through a 1×1 convolution layer and sigmoid activation, then multiplying element-wise with the reflectance component. Regions with high weights receive stronger detail enhancement and noise suppression, while regions with low weights receive weakened processing to avoid overexposure, achieving adaptive regulation of “enhance where needed.” This mechanism ensures both the detail restoration of reflectance components in weakly lit areas and the prevention of halo artifacts caused by over-processing in strongly lit areas, ultimately achieving optimal balance between global illumination and local degradation removal, providing high-quality basic features for subsequent image fusion.

The setting of the multi-scale feature extraction module in the multi-scale attention block aims to solve the problem that traditional single max-pooling layers cannot retain sufficient contextual information, thereby more comprehensively capturing details of the reflectance component at different scales. When traditional methods use a single max-pooling, small-scale details may be lost or large-scale structures may be ignored. This module processes the illumination-attention-corrected reflectance feature map in four ways: no pooling, single pooling, double pooling, and original input, and assigns different weight coefficients: no pooling retains the finest local details, single pooling captures medium-scale features, double pooling extracts global structural information, and the original input serves as a reference baseline. Each pooled feature map undergoes convolutional dimensionality reduction and deconvolutional restoration to ensure fusion of features at the same dimension across scales. For example, when processing low-light rural road images, this module can simultaneously preserve the details of small stones on the road surface, the winding direction of the road, and the overall contour of surrounding fields. Through continuous extraction across multiple downsampling stages, the module enriches feature information at different granularity levels and expands the network receptive field, allowing effective capture of features in the reflectance component from fine textures to macro structures.

The combination of the Convolutional Block Attention Module (CBAM) module with multi-scale feature extraction in the multi-scale attention block can enrich feature diversity while accurately filtering out useless features, improving the specificity and effectiveness of reflectance component processing. The CBAM module integrates channel attention and spatial attention mechanisms: channel attention identifies feature channels crucial for the enhancement task and assigns high weights; spatial attention focuses on valuable spatial regions and suppresses background noise regions. The multi-scale attention block encodes global contextual information in four ways, and combined with CBAM, can perform feature selection at each scale. For example, when processing low-light shopping mall window images, multi-scale feature extraction covers the reflective highlights on window glass, label text on products inside the window, and surface texture of the products. The CBAM module strengthens product color and texture channels via channel attention and focuses on product regions via spatial attention while weakening interference from glass reflections. Finally, while retaining the diversity of useful features, redundant information is removed, making the key features of objects in the reflectance component more prominent, providing a high-quality basis for subsequent enhancement. Figure 5 shows the architecture of the multi-scale attention module.

5.png

Figure 5. Multi-scale attention module architecture

2.4 Loss function

The overall network loss achieves collaborative optimization of the three sub-networks through a linear combination of decomposition loss loss_fz, recovery loss loss_fv, and adjustment loss loss_u, ensuring that the objectives of each sub-task are consistent with the overall research objective. For example, when processing low-light shopping mall surveillance images, the decomposition loss ensures accurate separation of product textures and lighting distribution; the recovery loss ensures that the text details of product labels remain clear after denoising; the adjustment loss constrains the enhanced brightness of chandelier areas to always be higher than the shadow areas of the shelves. By minimizing the overall loss, each sub-network is no longer optimized in isolation. Improvement in decomposition accuracy provides more reliable inputs for recovery and adjustment, and the optimization of recovery and adjustment in turn guides the decomposition process to better meet practical requirements, ultimately achieving comprehensive enhancement of low-light images in terms of detail preservation, noise suppression, and illumination balance, thus accomplishing the research objectives. The overall loss function can be expressed as:

$loss = loss_{f z}+\eta_1 loss_{f v}+\eta_2 loss_u$          (1)

In the above equations, the decomposition loss is designed to address the ill-posed nature of the image decomposition into reflectance and illumination components. It uses multi-dimensional constraints to ensure the rationality of the decomposition results. The reflectance loss, based on the Retinex theory, constrains the reflectance component to remain stable under different lighting conditions. For example, in low-light images of the same scene taken on a cloudy day or at dusk, the reflectance component of the building's wall texture, window outlines, and other details should remain consistent, unaffected by the intensity of the lighting, avoiding distortion of the object’s inherent properties caused by decomposition bias. The illumination component smoothness loss targets the illumination distribution characteristics, requiring the smooth transition in areas with gradual lighting changes in the original image to also be smooth in the illumination map, preventing abrupt lighting shifts. The reconstruction loss ensures that the re-synthesized image from the decomposed reflectance and illumination components closely matches the original image. For example, after decomposing a low-light street scene image, the re-synthesized image should retain key details such as the location of streetlights and the general outline of trees, avoiding information loss or distortion during decomposition. These three components, through linear combination, provide comprehensive constraints for image decomposition. Specifically, let the reflectance components of the low-light image and the normal light image be represented by E₁ and E_g, and the L₂ norm constraint loss be represented by $\left\|\|_2\right.$. The reflectance loss (loss_ed) expression is:

$loss_{e d}=\left\|E_1-E_g\right\|_2^2$             (4)

Let the low-light-normal light image pairs be represented by T₁ and T_g, and the illumination components of the low-light image and normal light image be represented by U₁ and U_g. The first-order operators in the horizontal and vertical directions are represented by ∇, and a fixed constant by γ. The illumination component smoothness loss expression is:

$loss_{f u}=\left\|\frac{\nabla U_1}{M A X\left(\left|\nabla T_1\right|, \gamma\right)}\right\|_1+\left\|\frac{\nabla U_g}{M A X\left(\left|\nabla T_g\right|, \gamma\right)}\right\|_1$              (5)

In addition, it is necessary to ensure that the re-synthesized images from the decomposed E₁, E_g, U₁, and U_g (T^-₁ and T^-_g) are similar to the original image pairs T₁ and T_g. The reconstruction loss expression is:

$loss_{e z}=\left\|T_1-E_1 U_1\right\|_1+\left\|T_g-E_g U_g\right\|_1$          (6)

Let the weight coefficients be represented by $\omega, \lambda$, and $\delta$. The decomposition loss expression is:

$l o s s_{d c}=\omega l o s s_{e d}+\lambda l o s s_{f u}+\delta l o s s_{e z}$             (7)

The recovery loss is designed to focus on ensuring the overall structural stability of the denoised reflectance component, and it achieves precise constraints through the collaborative effects of mean squared error (MSE) loss (loss_JF) and structural similarity loss (loss_JG). loss_JF calculates the pixel difference between the reflectance component before and after denoising, suppressing detail loss caused by excessive denoising. For example, when processing low-light leaf images, it can prevent the leaf texture from becoming blurred due to denoising. loss_JG focuses on the overall structural similarity of the reflectance component, constraining the denoising process so that the macro shape of the object is not destroyed, such as ensuring that the contour of branches after denoising is consistent with the branch orientation in the original reflectance component. For instance, in low-light license plate images, the recovery loss ensures that, while removing noise from the character areas, the edges, strokes, and structure of the characters are highly consistent with the original reflectance component, achieving denoising without damaging key details, thus providing reliable reflectance features for subsequent image recognition. The recovery loss expression is:

$loss_{f v}=loss_{J F}+loss_{J G}$              (8)

The adjustment loss is designed to ensure that the enhanced illumination component maintains the same lighting distribution trend as the original image, avoiding chaotic lighting logic. For example, in a low-light indoor scene, the window area in the original image is brighter than the wall corner due to natural light illumination. The adjustment loss constrains the enhanced illumination map to preserve this distribution relationship. After enhancement, the brightness of the window area will be significantly increased, while the wall corner will be appropriately brightened, rather than having the wall corner brightness surpass that of the window, which would be unrealistic. This constraint ensures that the lighting enhancement follows the actual light propagation laws in real scenes. For example, in nighttime road images, the enhanced brightness of the streetlight illuminated areas will always be higher than the shadow areas, ensuring that the enhanced image looks more natural visually, avoiding the sense of disharmony caused by a disordered lighting distribution. The specific expression is:

$l o s s_u=\left\|\bar{U}-U_g\right\|_2^2+\left\||\nabla \bar{U}|-\left|\nabla U_g\right|\right\|_2^2$           (9)