Real-Time Fire and Smoke Detection and Classification of Video Frames in Smart Cities Based on You Only Look Once Version 10 Model

Our perspectives on Smart cities are radically changing sustainability, safety and urbanization. As smart cities become more prevalent worldwide, protecting citizens and their possessions becomes increasingly important [1]. Consequently, the application of traditional sensor-based fire detection is restricted to indoor environments. Video fire detection provides a plethora of visual fire information and eliminates distance limitations. It is really exciting that visual fire detection is being used in applications. Video flame detection and video smoke detection are two types of fire detection methods that utilize video technology, each based on the specific elements being detected [2]. A method used color information to derive flame outlines and provided flame identification in the spatiotemporal domain. A method for identifying fires using grayscale images taken in tunnels was implemented [3]. 

It is crucial to have a dependable and effective early fire detection system. Due to the growing trend of urbanization and an increased awareness of the importance of safety, it has become the top priority in smart cities. Early detection of containment and fire can reduce property damage and save lives. However, this project must address problems such as the unpredictable nature of fire, the requirement for continuous surveillance, and the enormous amounts of data produced by smart cities [4]. To identify fires early, scientists and planners have built vision-based detectors of fire (VFDs) in addition to fire sensors that react to temperature, gas, solid, sound, or flame [5]. The earliest way of detecting fire was manual inspection, which involves assigning individuals to periodically check possible fire spots. However, as civilization has evolved, the drawbacks of manual inspection—such as sluggish reaction times and hazards in the early stages of fires—have grown more obvious, rendering it inappropriate for the demands of the modern world [6]. Closed-circuit television (CCTV) has been widely installed in buildings and public areas as a result of urbanization, making it possible to monitor fire threats at a reasonable cost. Fire detection techniques may potentially be a part of these monitoring systems [7]. Among one-stage algorithms, the YOLO series is notable for its straightforward architecture, high accuracy, and ability to perform real-time detection [8]. Ghost convolutions were utilized to lower the computational load imposed by the second, tiny object identification layer that was added to the YOLOv5 network. The network was able to improve its ability to detect small things, at a comparatively minimal computing cost, like dim flames [9]. The main structure of the YOLOv3 Darknet-53 is found to have width, resolution, and depth mismatches. They replaced Darknet-53 with an improved Efficient Net and proposed an improved model based on YOLOv3 with faster performance, fewer parameters, and greater detection capabilities [10]. The YOLOv8 network serves as the foundation for the proposed methodology for creating a ship fire detection network. The researchers employed channel and spatial reconstruction convolutions and an extra object-detection layer to decrease duplicate characteristics [11]. These modules could make better the capacity to recognize little items. However, the second detection layer's detection capability was constrained by the harsh hardware conditions aboard the ship [12]. Bu Utilizing the network of Convolutional Neural Network (CNN), simple linear iterative clustering (SLIC) for segmentation of smoke picture, and density-based spatial clustering of applications with noise (DBSCAN) that may attain faster observation. However, due to its low rate of FPR, which mention worthy model allergy, the offer procedure demanded further more development [13]. Although deep learning methods for smoke detection have shown a lot of potential, using edge equipment for detection has three drawbacks. In the first place, the large amount of parameters and expensive requirements of hardware for huge network models make it strenuous to deploy them for workable activities and meet real-time smoke detection needs. Second, current lightweight samples can discover smoke faster under the similar situations as models with a large network, but their detection accuracy is often significantly worse. When recognizing things with thin characteristics, such as smoke, the fusion of features is usually insufficient, this lowers the detection accuracy. The accuracy and detection speed are therefore performing at disparate levels, which is a problem. Third, during the initial statuses of fire of the forest, a type of smoke called "small smoke" is produced. It is characterized by its thinness and small volume. It cannot effectively extract information because thin and little smoke can only extract a limited number of characteristics. It is additional challenging to discover than regular smoke, which has formerly taken on shape and is vulnerable to disturbances like lens flaws. This boosts the issue of UAVs taking noisy pictures midst missions' discovery, which could lead to missed discovery. An enhanced detection of fire technique for smart metropolises focused on the You Only Look Once version 10 (YOLOv10) model was suggested in order to address the main problems associated with earlier methods involving deep learning. This approach produced the following advantages.

1. Improvement in accuracy: The proposed approach may improve the accuracy of fire detection in intelligent cities when compared to traditional methods. This might be achieved by utilizing the advantages of deep learning algorithms, such as YOLOv10, to recognize fire-specific traits that might be challenging to recognize using traditional image processing methods.

2. Real-time recognition of object: The YOLOv10 algorithm is well known for its speed and ability to identify objects. Therefore, the recommended approach is perfect for smart city applications where prompt fire detection is essential.

3. Reduced false alarms: By using deep learning to comprehend fire-specific characteristics, the proposed method may be able to reduce false alarms, which are common in traditional fire detection systems. It is possible to minimize false alarm expenses and prevent needless emergency reactions.

4. Effectiveness of Cost: Since the suggested method may be used with inexpensive cameras and technologies, it may be less expensive than traditional fire detection methods since it doesn't require pricey fire detection equipment.

5. Large dataset: This methodology utilized a huge dataset that includes three various kinds of smoke and fire scenarios in addition to usual situations involving more people, unlike other approaches that rely on a small number of datasets. The collection includes real-world images and videos collected from different sources. The collection covers a wide range of fire scenarios, such as tiny and huge fires, inside and outdoor flames, and high and low light levels. A deep neural network extracts crucial information from huge datasets in order to produce accurate predictions and reduce overfitting.

6. Research on deep learning for fire and smoke warning has been ongoing, with target detection algorithms in particular demonstrating encouraging outcomes. Based on the You Only Look Once (YOLOv8) model, a high-precision, lightweight improvement is developed to improve the model's capacity to detect smoke and fire in different environments. It uses partial convolutions to streamline the model and adds an attention block to acquire cross-space learning capabilities. Additionally, a variation of the neck network is used to achieve bidirectional feature fusion. The improved model achieved 77.8% precision and 65.3% recall [14].

The key contribution of this article is YOLOv10, which improves efficiency and accuracy in real-time, end-to-end object identification, especially by removing non-maximum suppression (NMS) and optimizing model design. It accomplishes this contribution through a number of significant improvements, including a rank-guided block architecture, spatial-channel decoupled down sampling, lightweight classification heads with depth-wise separable convolutions, and a dual assignment technique for NMS-free training. Thanks to these developments, YOLOv10 can now attain cutting-edge performance with less computational overhead, which makes it appropriate for a variety of real-time applications. The remaining sections of the search consist of related works, methodology, Experiments and results, and conclusion.

Three primary steps make up the recommended fire and smoke detection methodology: data collection and preprocessing in the first stage, feature selection using the Speeded-Up Robust Features (SURF) detector in the second stage, and model training and detection using YOLOv10 in the third stage. The specifics of the proposed system were illustrated in Figure 1.

Figure 1. Proposed intelligent smoke and intelligent system fire and smoke detection (ISFD)

3.1 Dataset gathering and resizing

A bespoke dataset containing both fire incidents and non-fire events that visually resemble fire was developed in order to efficiently train and assess the fire detection system. The method of preparing the dataset included the following crucial steps:

1. Collection of Data 

Video clips of non-fire, smoke, and fire were gathered from a variety of public sources, including as YouTube, open-source repositories, Kaggle datasets of fire detection, and personal fire films. The video clip shows a variety of situations, such as industrial flames, forest fires, domestic fires, and intricate backgrounds enhanced with lighting effects.

2. Augmentation of Data 

A variety method of augmentation the data were applied to every frame in order to increase dataset diversity and strengthen the model's capacity for generalization. 

These methods involve flipping both horizontally and vertically, rotating (±15–30 degrees), and adjusting brightness and contrast (±20%). Scaling methods like zooming in and out by up to ±10% were also used. By simulating variations in illumination, camera angles, and environmental factors, these adjustments strengthen the trained model's resilience.

3. Resizing 

Before being fed into the model, images or frames are reduced to standard dimensions, such as 224 × 224. In order to ensure uniform processing and performance across the neural network, this preserves input size homogeneity.

4. Normalization 

Faster convergence during training and increased model accuracy are made possible by normalizing the image pixel values. By scaling the numbers to a range of 0 to 1, the model is able to handle the data more quickly.

5. Reduction of Noise 

Noise in raw photos can reduce model performance. To overcome this, a technique of median filtering is employed to smooth the images and remove extraneous noise, resulting in more accurate input for the detection system.

6. Balancing of Data 

Equal representation of non-fire and fire photographs in the dataset is necessary for the model to be trained effectively. Data balance approaches are used to guarantee that the training data is not biased toward a single class, hence boosting the model's capacity to generalize.

Impulse noise sometimes referred to as noise of 'salt-and-pepper', can be eliminated effectively via utilized median filter. YOLOv10 utilizes a mix of sophisticated mechanisms of spatial channel, loss functions of shape-aware, and deep learning on extensive datasets with variety situations of illumination differentiate among fire and lighting of background (like headlights, sun glare or streetlights). YOLOv10 examines the texture, motion, and behavior of flickering flames over time.

Through an enhanced architecture of (NMS-free) ("Non-Maximum Suppression") that increases speed of real-time inference and enhances extraction of feature to handle occlusions and complex backdrops, YOLOv10 tackles high-density movements of pedestrian and interference of detection of fire. Consistent dual-label assignment for dense scenes and specialized feature fusion techniques that distinguish pertinent items from background noise are its key superiority in these appointments.

3.2 Feature extraction

In order to elicit the interest points (features) from the input image and capture pertinent data at different scales, the proposed methodology utilized a SURF detector with the YOLOv10 backbone network. A robust detector for local features, the SURF technique is an extractor used in many computer vision applications, including object identification and 3D reconstruction. SURF is a better approach that works well for real-time implementation. The SURF algorithm's vector descriptor was called to detect interest locations, and this process was founded on the assumption of a scale space [18].

1. Integral Images

An integral image can be utilized to quickly calculate Hessian's determinant blob detector, which is one of the integer approximations used in the SURF technique. Box-kind convolution filters may be calculated quickly thanks to it. Equation 1 provides an illustration of the invasion of an integral image (I_Σ (x)) at (x = (x, y)T).

$I_{\Sigma}=\sum_{i=0}^{i<x} \sum_{i=0}^{j<y} I(i, j)$   (1)

2. Hessian Matrix-Based Interest Points

Because of its high accuracy performance, a detector can rely on the Hessian matrix. At locations where the determinant was at its maximum, it recognized a blob as a structure. Eq. (2) was used to determine the matrix of Hessian (Ή(x, σ)) in (x) at scale (σ) given a point (x = (x; y)) in an image (I).

$'\mathrm{H}(\mathrm{x}, \sigma)=\left[\begin{array}{cc}L_{x x}(x, \sigma) & L_{x y}(x, \sigma) \\ L_x y(x, \sigma) & L_{y y}(x, \sigma)\end{array}\right]$   (2)

The Gaussian second-order derivative (∂2/∂x2g (σ)) in the image (I) at a point (x) is convolved by Lxx (x, σ), and Gaussians are optimal for scale space analysis. It must be clipped and discretized in the program, as shown in Figure 2, on the left side.

Figure 2. Left to right Gaussian second order partial derivative for Lxy (xy-direction), Lyy (y- direction) and, respectively [18]

The blob restraint map is calculated at the lowest scale with the (9 × 9) box filters shown in Figure 2, which approximate a Gaussian distribution with σ = 1.2 [18].

3. Scale Space Representation

In order to compare the points of interest that must be established at different scales-not least because of the correspondences that are frequently searched for-scale spaces are typically represented as a pyramid of an image. The photos are repeatedly subsampled for greater level pyramid achievement after being smoothed using a Gaussian technique. It is not necessary to apply the same filter repeatedly to the results of a previously filtered layer since integral pictures and box filters are used. Instead, box filters of any size are applied to the main image at precisely the same speed, even in parallel. Consequently, a scale space was examined by upscaling the filter's size rather than repeatedly shrinking an image's size, as seen in Figure 3. The result of the (9 × 9) filter is regarded as the first scale layer and can be compared to the scale (s = 1.2). The following layers were obtained by filtering the image using ever larger masks while accounting for the discrete type of integral images and the structure of the specific filters [19].

Figure 3. Instead of reducing iteratively the size of an image left, the use of integral-images permits the upscaling of the filter at fixed cost right [18]

4. Interest Point Localization

In order to locate points of interest in a picture and through scales, a non-maximum suppression in the (3 × 3 × 3) neighborhood was used. Then, within the scale, the Hessian matrix determinant maxima are interpolated. In this case, the scale difference between the first layers of each octave is rather substantial, making the interpolation of a scale space especially important. Using the Fast-Hessian detector, a paradigm of the identified places of interest is shown in Figure 4 [19].

A hybrid model (SURF+YOLOv10) that aims to improve detection quality on small or sparse objects while frequently decreasing the overall inference speed is typically created by adding SURF to the YOLO family or by combining similar handcrafted, robust techniques for feature elicitation with deep learning. 

Combining these strategies has the following key effects:

1. Considerably better for detection of small objects: SURF approach enhancement aids in the identification of fine-grained details, which are essential for spotting small, far-off, or hazy targets.
2. Improved understanding of context: Adding strong local characteristics enables better classification in difficult or cluttered backgrounds, even while YOLO excels at capturing high-level semantic information.
3. Greater Precision on Particular Datasets: Combining YOLO with conventional detection of feature SURF helps maintain high mean Average Precision (mAP) in pictures of rare situations in photography.

3.3 Model training and detection using You Only Look Once version 10

Deep learning approaches outperform conventional methods using object detection for choosing fire detection models. YOLOv10 was used in the suggested system. A model designed for real-time object detection aims to identify and classify fire in images [20]. The effectiveness of YOLOv10 is influenced by the tradeoff between accuracy and speed, as well as the capabilities of the hardware used. Training is carried out utilizing labeled datasets that include both fire and non-fire instances. Cross-validation is used to ensure model reliability.

The following procedures are taken to aggregate the detection outputs from YOLO10 utilizing a decision fusion mechanism:

Procedure 1: The system initiates a fire warning if the model independently detects fire with confidence over a predetermined level. 

Procedure 2: Temporal consistency is evaluated over multiple frames to prevent false alarms caused by fleeting false positives. 

Procedure 3: Alerts are communicated by a message or notification module in the surveillance system.

A backbone, a Path Aggregation Network (PAN), and prediction heads make up YOLOv10's architecture, which is seen in Figure 5. Features from the input image are extracted by the backbone, and the PAN efficiently gathers and disperses those features across the network. For classification and regression (bounding box) tasks, YOLOv10 presents dual label assignments with two prediction heads: the one-to-many head and the one-to-one head. To enhance prediction accuracy across various scales and scenarios, a Consistent Match Metric is utilized to refine label assignment.

Figure 5. The structure of You Only Look Once version 10 (YOLOv10) [21]

For fire detection quality, the proposed methodology calculates the F1-score recall, and precision. To compute the precision of a proposed method, it was utilized in Eq. (3) [22]. To compute the recall of the proposed method, it was utilized in Eq. (4). Also, to compute the F1-score of the proposed method, it was utilized in Eq. (5) [23].

Precision $=\frac{T P}{T P+F P}$   (3)

Recall $=\frac{T P}{T P+F N}$   (4)

$F 1-$ Score $=\frac{\text { PrecisionxRecall }}{\text { Precision }+ \text { Recall }} \times 2$   (5)

True Positive (TP): The number of cases that are correctly categorized as positive.

False Positive (False Positive - FP): The number of cases that are incorrectly categorized as positive (when the actual status is negative).

True Negative (True Negative - TN): The number of cases that were correctly categorized as negative.

False Negative (False Negative - FN): The number of cases that are incorrectly categorized as negative (when the actual state is positive).

The classification of things in the pictures—fire, smoke, and others—was done using labeling software. The txt format was used to save labels.

As indicated by the ratio in Table 3, the data set was split at random into training, validation, and testing sets.

Table 3. Details of the dataset

In order to meet real-time necessity, YOLOv10 utilized. The two main measures are mAP@0.5 and mAP@0.5:0.95. In order to ensure that the model correctly locates minor sources of fire, mAP50-95 offers a tighter estimation than mAP50, which analyzes discovery at a 50% threshold of "intersection over union (IoU)". Acceleration of hardware (like "NVIDIA GPUs") to test speed of inference ("FPS") and tiny object recognition performance in the detection of fire via customized estimation scripts.

Performance Metrics (PM) score is the weighted average of the recall and accuracy percentages. Consequently, this score considers both false negatives and false positives. Accuracy is sometimes difficult to understand, even though PM is further widespread than precision. When the costs of false positives and false negatives are equal, accuracy performs well.

The accuracy, recall, and F1-score standard metrics were used to assess the proposed system. Furthermore, the dataset's size was taken into account for comparison, while earlier techniques took either smoke or fire into consideration. We also made a comparison between our system and the most advanced fire detection systems available today, which include a deep learning-based system, an enhanced YOLOv6-based approach [25], a real-time video fire/smoke detection system based on YOLOv2 [26], an enhanced YOLOv5-based smoke detection model [27], a deep learning-based approach for detection for fire detection in smart city environments [28], and a YOLOv3-based approach [29].

It is preferable to consider recall in addition to accuracy if the costs of false positives and false negatives are different. Precision is defined as the proportion of correctly anticipated observations to all predicted positive outcomes. The complete performance outcomes of the suggested model can be shown in Figure 10. While the smoke and fire and other detection results using the proposed approach including precision, recall and F1-score are also shown in Figure 11.

Figure 10. The complete performance measurements outcomes of the suggested model

Figure 11. The proposed model's F1-score, precision, and recall model

The systems' performance comparison on the test dataset is displayed in Table 4 and Figure 12. The efficacy of the suggested system in identifying fires in real-world situations was demonstrated by its superior performance over the current methods in terms of precision, recall, and F1-score. The dataset consists of 5426 photos in the collection. Furthermore, on the other hand, our suggested model has an elevated rate accuracy of 95.9% throughout every class with 98.4% of recall and F1-score was 0.971% for detecting fire, smoke, and other things. We credit YOLO v10's enhanced speed and accuracy over earlier iterations of YOLO and additional deep learning-oriented systems of fire detection for our system's better performance. The Comparison of fire detecting systems' performances values can be listed in Table 4 and the histogram of these values can be shown in Figure 12.

Table 4. The performance of the systems for the detection of fire is compared

Figure 12. Comparison of fire, smoke and other detecting systems' performances

An error occurred in calculating the F1-score of the proposed model due to calculation mistakes. A range of smoke, human outlines and fire are included in this dataset; these include small and large flames, outdoor fire and indoor fire, high and nadir light levels, and typical fire-free environments. The number of images is 5,426. All the images contain fires, but some of the images have smoke with the fire, and some have fire, smoke, and other things such as cars near the fire or smoke. Data augmentation is a two-edged sword in ML since it can add errors for classification and skew sample distributions. Although expansion techniques are intended to improve generalization, their overuse or improper use might undermine the presumption that enlarged data maintains its original classification, leading to distortion. However, because the growth was not enormous, there was no major distortion or addressing error in our suggested system.

Although precise times for "SURF+YOLOv10" are less common than "YOLOv10 only," the outcomes of combining these approaches in a minor increase in computational effort.

The proposed system focused on flame reflections and smoke detection under varying lighting and on nighttime fire detection, and the detection accuracy was good in dealing with highly complex conditions.

Fires of intelligent city can have disastrous results, putting residents' lives in jeopardy and causing property damage. The accuracy and speed limitations of conventional fire detection methods make real-time fire detection challenging. This study presents the ISFD system, a highly advanced intelligent fire and smoke detection system built on the YOLOv10 algorithm for real-time fire detection and employing deep learning capabilities. When compared to traditional fire detection systems, the ISFD methodology can reduce false alarm rates; increase the detection accuracy of smoke, fire, and other, and save money. It can help with early fire detection and containment, preserving life and minimizing environmental and property harm. As indicated in Table 4, the ISFD methodology can save money, decrease false alarms, and improve fire detection accuracy when contrast to conventional systems for detection of fire. Other interesting features of intelligent cities, such as leaks of gas or floods, can be discovered by extending it. To identify and locate fires in real time, the suggested method makes use of a deep neural network that has been trained on a sizable collection of fire photos. Our proposed model has an excellent rate of accuracy 95.9% for every class with 98.4% recall and an F1-score of 0.971% for detecting fire, smoke, and other things.