Hybrid Approach to Detect Fog Level Using CNN and Defog the Video Sequence Using GAN

A hybrid technique is designed to enhance visibility in video scenes affected by fog. This method addresses the specific challenges posed by varying fog intensities, which can impair vision and reduce the quality of video content, by combining CNN with GAN shown in Figure 1. In the first stage, a CNN-based system accurately detects and classifies fog levels, assessing its intensity to enable adaptive analysis. A GAN-based defogging model then removes the fog while preserving essential scene details, ensuring minimal distortion and improving the visibility of critical elements. This hybrid approach provides a more customized defogging process and outperforms conventional methods by delivering outstanding results in real-time. With applications in environmental surveillance, monitoring, and autonomous vehicles, this study demonstrates how proposed system can enhance video quality in adverse weather conditions, increasing the safety and reliability of systems dependent on visual data.

1.png

Figure 1. Proposed architecture

3.1 Problem formulation

The problem of fog detection and removal in video sequences can be formulated as an optimization problem aimed at enhancing visual clarity by reducing fog-induced noise while preserving essential scene details. Given a video sequence $V=\left\{F_1, F_2, \ldots, F_x\right\}$ consisting of x frames $F_x$ impacted by varying levels of fog, the task involves two main objectives: detecting the fog level $L\left(F_x\right)$ for each frame and restoring a clear version $\widehat{F}_x$ with minimized distortion.

Fog Level Detection: Let $C N N_\theta$ parameterized by $\theta$. The fog level $L\left(F_x\right)$ is predicted as:

$L\left(F_x\right)=C N N_\theta\left(F_x\right)$                         (1)

where, $L\left(F_x\right) \in\{$Low, Medium, High$\}$ is the categorical label indicating fog density. The CNN model is optimized to minimize the classification error for fog levels.

Defogging: Let $G A N_{\emptyset}$, where includes both the generator and discriminator networks. The generator aims to produce a defogged video $\widehat{F}_x=G_{\emptyset}\left(F_x, L\left(F_x\right)\right)$ that closely approximates a fog-free video. The GAN objective function combines a reconstruction loss $L_{r e c}$ to ensure similarity to clear videos and an adversarial loss $L_{a d v}$ to encourage realistic outputs:

$\begin{gathered}\min _{\emptyset} \max _D L_{G A N}\left(D, G_{\emptyset}\right)=E\left[\log D\left(F_x\right)\right]+E\left[\log \left(1-D\left(G_{\emptyset}\left(F_x, L\left(F_x\right)\right)\right]\right.\right.\end{gathered}$                              (2)

where, $D$ is the discriminator, and $G_{\emptyset}$ seeks to minimize this loss to produce clear, defogged frames. The overall goal is to enhance video clarity in a computationally efficient manner, enabling real-time processing by balancing accuracy and processing speed.

3.2 Dataset description

The study's dataset consists of 1,000 video sequences with a total of around 100,000 frames that have been specially selected for fog identification and removal. With a pixel resolution of 1920×1080, each video offers a high-definition video that is necessary for thorough fog investigation and defogging procedures. To guarantee the model's resilience in a range of environments, the dataset comprises a variety of environmental contexts including residential properties suburban, rural, and highway sectors. To accurately replicate real-world situations, it also takes into account changes in climate (obvious, moderate fog, and thick fog) and illumination (daytime, evening, and low light). The CNN algorithm can precisely identify and categorize fog density since every video is tagged with fog levels of intensity designated as low, medium, or high. Video files are saved in MP4 format, however, to facilitate processing convenience, each of the frames is saved in JPG format. This dataset provides a thorough basis for assessing the proposed CNN-GAN strategy under dynamic and demanding environmental situations shown in Table 1. It is specifically designed to enable both fog-level identification and defogging activities. Sample data supports precise fog identification and defogging effectiveness under a variety of scenarios by providing crucial information for CNN-GAN model evaluation and training shown in Table 2 and Figure 2. 

Table 1. Dataset description

Table 2. Sample data

2.png

Figure 2. Sample data

3.3 Pre-processing

Pre-processing is essential for preparing foggy video sequences for effective fog level detection and defogging. Frame Extraction: Each video sequence $V=\left\{F_1, F_2, \ldots, F_{x y}\right\}$ is first decomposed into individual frames for isolated processing. Let $V_x$ be the $x^{\text {th }}$ video in the dataset. Frame extraction can be represented as:

$V_x=\left\{F_{x 1}, F_{x 2}, \ldots, F_{x y}\right\}$                        (3)

where, $F_{x y}$ denotes the yth frame of video $V_x$.

Gray-Scale Conversion: To simplify processing, each frame $F_{x y}$ is converted from RGB color space to gray scale. This reduces computational complexity and focuses on the luminance needed for fog detection. The gray-scale value $G(i$, $j$ ) at pixel location $(i, j)$ is calculated as:

$\begin{gathered}G(i, j)=0.2989 R(i, j)+0.5870-G(i, j)+0.1140-B(i, j)\end{gathered}$                            (4)

where, $R(i, j), G(i, j)$, and $B(i, j)$, are the red, green, and blue channel intensities at pixel $(x, y)$.

Histogram Equalization: Foggy videos tend to have low contrast, so histogram equalization is applied to enhance contrast. Given the Cumulative Distribution Function (CDF) of the videos gray-scale histogram H, the equalized intensity $X_{e q}$ at each pixel (i, j) is:

$X_{e q}(i, j)=\frac{H(G(i, j))-\min (H)}{\max (H)-\min (H)} \times 255$                        (5)

where, $H(G(i, j))$ maps the original gray-level pixel values to the enhanced range.

Fog Density Normalization: To handle varied lighting conditions, intensity normalization can be applied. This adjusts pixel values to a standardized range, which helps in reducing the influence of lighting variations. Let $X(i, j)$ be the pixel intensity at $(i, j)$, normalized to the range $[0,1]$ as follows:

$X_{n o r m}(i, j)=\frac{X(i, j)-\min (X)}{\max (X)-\min (X)}$                              (6)

Noise Reduction (Smoothing): To further reduce unwanted noise, a Gaussian filter $G_\sigma$ with standard deviation o is applied. For each pixel (i, j) in the frame, the smoothed intensity S(i, j) is:

$S(i, j)=\sum_{u=-k}^k \sum_{v=-k}^k G_\sigma(u, v), X(i+u, j+v)$                            (7)

where, $G_\sigma(u, v)=\frac{1}{2 \pi \sigma^2} e^{-\frac{u^2+v^2}{2 \sigma^2}}$ and k is the kernel radius.

Data Augmentation: To create a more robust model, frames can be augmented by rotating, flipping, or adjusting brightness, which helps the model generalize better. Let A represent an augmentation transformation; then, each frame can be augmented as:

$F_{x y}^{a u g}=A\left(F_{x y}\right)$                         (8)

where, A includes operations like rotation, flipping, and brightness adjustments.

3.4 Detect fog level of a video sequence using CNN

The network's encoder portion is in charge of analysing the incoming video and identifying pertinent characteristics. It is made up of many dropout layers and convolutional layers (CONV1, CONV2, CONV3, etc.).

Convolutional Layers: Extract characteristics at various levels of abstraction by applying filters to the input video. Patterns such as edges, textures in particular, and forms are captured by the filters. The filters get more intricate and higher-level characteristics go further into the architecture of the network.

Dropout Layers: Throughout training, these layers randomly deactivate a subset of neurons. It enhances the model's capacity for generalizations and helps avoid over fitting. The encoder increases the number of channel features while gradually decreasing the input videos physical dimensions. Enables deeper layers of the system to catch more intricate characteristics. The original video is recreated in the decoder part using the encoded characteristics obtained from the encoder.

Deconvolutional Layers: Gradually increase the spatial dimensions by up sampling the encoder's map of characteristics.

Estimating Modules: These modules forecast the depth data for every pixel using the upsampled mappings of features. Activation processes and convolutional layers of information are usually included.

Concat Layers: These layers join the encoder's matching characteristic maps with the upsampled characteristic maps. As a result, the decoder can accurately estimate depth by utilizing both low-level and high-level characteristics.

Dropout Layers: Employed to enhance the model's capacity for generalizations and avoid excessive fitting.

Figure 3 flowchart explains as follows:

3.png

Figure 3. Detect fog level of a video sequence using CNN

Input Video: It serves as the network's first input.

Encoder: The encoder gradually extracts information from the video by processing it through a number of convolution and dropout stages.

Decoder: This device upsamples the encoded characteristics using deconvolutional stages. The decoder uses concatenated to add the relevant encoder characteristics at every expanding level.

Depth Estimate: Using the upsampled characteristics as a basis, the estimate modules of the decoder forecast the depth information needed for every pixel.

Output: The network's final output consists of the predicted depths map and the reassembled video.

This process involves designing a CNN model that accepts individual frames as input and outputs the fog level (e.g., Low, Medium, High).

Input Frame Representation: Let $F_{x y}$ represent the $\mathrm{y}^{\text {th }}$ frame of the $\mathrm{x}^{\text {th }}$ video sequence, where $F_{x y} \in R^{H \times W \times C}$, with height H , width W , and color channels C . Each frame is preprocessed (e.g., converted to gray scale, normalized, etc.) before being input into the CNN model.

CNN Architecture for Feature Extraction: A CNN model consists of multiple layers, including convolutional, pooling, and fully connected layers, which extract features relevant to fog detection. Let: $f_k^l$ represent the k -th feature map in the 1-th convolutional layer, $w_k^l$ be the filter (weight matrix) for $f_k^l$, and $b_k^l$ the bias term. The output of each convolutional layer $f_k^l$ for a given pixel ( $\mathrm{i}, \mathrm{j}$ ) in the frame is:

$f_k^l(i, j)=\sigma\left(\sum_{m=1}^M w_{k, m}^l * f_k^l(i, j)+b_k^l\right)$                           (9)

where, M is the number of input feature maps to layer l, and $\sigma$ is the activation function, typically ReLU (Rectified Linear Unit) for non-linearity:

$\sigma(z)=\max (0, z)$                          (10)

Pooling Layer: After convolution, pooling layers reduce the spatial dimensions to extract dominant features and reduce computational complexity. If P is the pooling operation (e.g., max pooling), the pooled feature map $p_k^l(i, j)$ is:

$p_k^l(i, j)=P\left(f_k^l(i, j)\right)$                           (11)

Fog Level Classification: After passing through multiple convolutional and pooling layers, the extracted features are flattened and fed into fully connected layers. Let $z_y$ denote the output of the final fully connected layer before the softmax activation. For a fog level classification into K categories (e.g., Low, Medium, High fog levels), the probability $P\left(j=k \mid F_{x y}\right)$ that frame $F_{x y}$, belongs to class k is computed as:

$P\left(j=k \mid F_{x y}\right)=\frac{\exp \left(z_k\right)}{\sum_{k^{\prime}=1}^K \exp \left(z_k\right)}$                             (12)

Loss Function: The CNN is trained to minimize the categorical cross-entropy loss function, which measures the discrepancy between the predicted fog level and the actual fog level. For a training sample ($F_{x y}, j$) with true fog level y and predicted probabilities $P\left(j=k \mid F_{x y}\right)$, the loss L is:

$L=-\sum_{k=1}^k j_k \log \left(P\left(j-k \mid F_{x y}\right)\right)$                          (13)

where, j_k is a one-hot encoded vector indicating the true class.

Prediction: After training, for each new frame $F_{x y}$, the model predicts the fog level $\hat{J}$ by selecting the class with the highest probability:

$\hat{J}=\arg \max _k P\left(j=k \mid F_{x y}\right)$                            (14)

This CNN-based process allows for effective fog level detection across frames can then be used as input to defogging models or further analysis.

3.5 Defog the video sequence using GAN

Figure 4 displays the general overview of the proposed methods for improving readability in hazy and foggy videos. The proposed structure has solved collectively make it unique. For the first time, fog is eliminated from videos using GAN. The median filter is used to eliminate noise from videos during the pre-processing stage in order to get high-quality output. A nonlinear filter, the median filter eliminates noise from photographs deteriorated by bad weather, including haze and fog. In processing videos, a variety of filters are employed. The median filter eliminates noise while maintaining video color details, borders, and smoothness. The median filter's primary purpose is to reduce computation time. GAN is utilized following pre-processing. The generator networks and discriminator networks are the two systems that make up the proposed GAN. The generating network's objective is to produce haze- and fog-free videos by using input haze and fog videos. The generator network in the proposed technique estimates natural light and transmissions immediately. The produced video and the beginning fog and haze-free video are distinguished by the discriminator networks. The propagation map and sunlight from the atmosphere are immediately estimated by the generator system. Three phases make up the generator networking: scene radiation, distribution map, and environmental light.

4.png

Figure 4. GAN for dehazing or defogging

Transmission Map: Figure 5 depicts the layout of the generating system. These layers were utilized by the generator network to determine their transmission mapping. Pooling and upsampling layers are utilized after every convolutional layer has been applied. The last layer is a fully linked layer that builds a representation by combining characteristics and the output of earlier levels.

Environmental Light: Estimating the atmosphere's light A is the goal of the atmospheric light element. as seen in Figure 5. A 7 × 7 convolution filtering with a 3 × 3 kernel size makes up the over-sampling layer.

739f125c752e1e89baaa7f76002d170e.png

Figure 5. GAN generator architecture

Scene Radiance: It is to recover the scene brightness following the estimation of ambient light and transmission map. Combining the ambient light is the goal of scene brightness.

Figure 6 depicts the structure of the discriminator system. Distinguishing between the generated video and the original fog-free video is the aim of the discriminator networks. Convolutional neural network sequential normalization is the discriminating fundamental function, and its final layer consists of the Leaky Relu and sigmoid functions. The initial fog-free video is ultimately distinguished and reconstructed by the discrimination.

6.png

Figure 6. Discriminator network architecture

To defog a video sequence, GAN can be used to enhance each frame by reducing the impact of fog, improving visual clarity.

GAN Framework: A GAN framework for video defogging aims to train a generator G that outputs a defogged frame $\widehat{F}_{x y}$ from a foggy frame $F_{x y}$, while a discriminator D leams to distinguish between real (ground-truth) fog-free frames and generated defogged frames.

The GAN objective can be represented as:

$\begin{aligned} & \min _G \max _D E_{F_{\text {real }} \sim P_{\text {data }}\left(F_{\text {real }}\right)}\left[\log D\left(F_{\text {real }}\right)\right]+E_{F_{\text {fog }} \sim P_{\text {data }}\left(F_{\text {fog }}\right)}\left[\log D\left(G\left(F_{\text {fog }}\right)\right)\right]\end{aligned}$                        (15)

where, $F_{\text {real }}$ is the ground-truth fog-free frame, $F_{f o g}$ is the foggy frame, $G\left(F_{f o g}\right)=\widehat{F}_{x y}$ is the generated defogged frame.

Generator Network (G): The generator G takes a foggy frame $F_{x y}$ as input and attempts to generate a defogged version $\widehat{F}_{x y}$ that resembles a clear frame. For each pixel (i, j) in the frame, the generator's output can be expressed as:

$\widehat{F}_{x y}(i, j)=G\left(F_{x y}(i, j) ; \theta_G\right)$,                           (16)

where, $\theta_G$ represents the parameters of the generator. G is typically built using convolutional layers with a U-Net or ResNet architecture, which is adept at retaining and restoring fine video details.

Discriminator Network (D): The discriminator $D$ attempts to classify whether a given frame is a real fog-free frame $F_{\text {real }}$ or a generated defogged frame $\widehat{F}_{x y}$. The discriminator outputs a probability $D(F)$ for each input frame $F$, where:

$D(F)=\sigma(W \cdot F+b)$,                         (17)

with W as the weight matrix, b as the bias, and a representing the sigmoid function for binary classification (real vs. fake).

Loss Functions: The training process includes a combination of adversarial and perceptual losses to ensure that G generates visually clear, realistic defogged frames.

Adversarial Loss: The adversarial loss for the generator G is based on the discriminator's probability for generated frames. The generator aims to "fool" the discriminator by minimizing:

$L_{a d v}(G)=-E_{F_{f o g} \sim P_{d a t a}\left(F_{f o g}\right)}\left[\log D\left(G\left(F_{f o g}\right)\right)\right]$                            (18)

Encouraging G to produce defogged frames that the discriminator cannot distinguish from real fog-free frames.

Content Loss (e.g., L1 or L2): To ensure that the defogged frame is structurally similar to the ground truth, a content loss $L_{\text {content }}$ is used. The L1 content loss between generated $\widehat{F}_{x y}$ and real frame $F_{\text {real }}$ is:

$L_{\text {content }}(G)=E_{F_{\text {fog }}, F_{\text {real }}}\left[\left\|F_{\text {real }}-G\left(F_{\text {fog }}\right)\right\|_1\right]$                       (19)

Perceptual Loss (optional): Perceptual loss can help retain high-level features by using a pre- trained model $\Phi$(e.g., VGG) to measure feature-level differences:

$\begin{aligned} L_{\text {perceptual }}(G)= & E_{F_{\text {fog }}, F_{\text {real }}}\left[\| \Phi\left(F_{\text {real }}\right)\right. \left.-\Phi G\left(F_{\text {fog }}\right) \|_2\right]\end{aligned}$                        (20)

The total loss for G is a combination of these components:

$\begin{aligned} L_G=\lambda_{\text {adv }} L_{\text {adv }}(G) & +\lambda_{\text {content }} L_{\text {content }}(G)+\lambda_{\text {perceptual }} L_{\text {perceptual }}(G),\end{aligned}$                         (21)

where, $\lambda_{\text {adv }}, \lambda_{\text {content }}$, and $\lambda_{\text {perceptual }}$ are weights that control the contribution of each term.

Defogging Process: During inference, the generator $G$ takes a foggy frame $F_{x y}$ from the video sequence and produces a defogged frame $\widehat{F}_{x y}$ :

$\widehat{F}_{x y}=G\left(F_{x y} ; \theta_G\right)$                          (22)

This process is applied frame-by-frame across the video sequence, resulting in a defogged video. This GAN-based framework allows for high-quality defogging by leveraging adversarial training and loss functions that ensure both visual clarity and structural accuracy in defogged video sequences.