Human Activity Recognition Using Thermal Videos in Low Light: A Comparative Analysis

The section also examines different Convolutional Neural Networks (CNNs) that play a crucial role in the research. Finally, the discussion extends to the mobilenetV2 model, which is essential for the ultimate goal of the classification of human activities. Initially, by describing the specific video datasets selected for this research, detailing their characteristics and relevance to the study. Following this, it involves exploring the range of human activity detection models used, highlighting their functionalities and how they contribute to the detection process [24].

The focus then shifts to various CNNs, demonstrating their architecture and their importance in extracting meaningful features from the video data. Finally, the section investigates the use of the Mobilenet V2 model. The Mobilenet V2 model is critical for achieving the goal of human activity classification, as it effectively captures the temporal dynamics of the activities. This thorough examination of the datasets, object detection models, CNNs, and the Mobilenet V2 model provides a solid foundation for understanding the methodologies applied in this research. The block diagram of Human action recognition is displayed in Figure 3.

3.jpg

Figure 3. Human action recognition block diagram

2.1 Overview of the dataset

This paper utilizes a collection of images acquired with the thermal imager FLIR C5. The thermal imager FLIR C5 is a compact, professional-grade thermal camera designed for easy use in various applications, including building inspections, HVAC/R, electrical inspections, and mechanical troubleshooting. It features a 160 × 120 (19,200 pixels) thermal resolution, providing clear and detailed thermal images. FLIR’s patented MSX technology enhances image clarity by adding visible light details to thermal images, making it easier to identify problem areas. The camera can measure temperatures from -20°C to 400°C (-4°F to 752°F), making it adaptable to a variety of thermal inspection requirements. The C5 includes Wi-Fi capability, allowing for quick sharing of images and data via the FLIR Ignite™ cloud service, facilitating easy documentation and reporting. Alongside the thermal camera, the C5 has a 5-megapixel visual camera for capturing standard photos, useful for creating comprehensive inspection reports. The device features a 3.5-inch touchscreen for intuitive navigation and operation.

4.jpg

Figure 4. FLIR C5 thermal imaging camera [34]

Designed to withstand tough work environments, the FLIR C5 has an IP54 rating for protection against dust and water. Its pocket-sized design makes it convenient to carry around and use in tight spaces [25]. The dataset consists of videos featuring a single individual performing various actions such as walking, running, duck walking, and crawling. These videos were recorded under low-contrast lighting conditions, typically after sunset or at night, to simulate challenging environments. The backdrops in these videos are varied yet static, providing a consistent context for the actions being performed. A total of 1000 videos were captured from distance 10 to 30 feet with different angles, with each action (walking, running, duckwalking, and crawling) represented by 250 videos. Figure 4 shows the FLIR C5 thermal imaging camera used in this work. Each video clip captures approximately 15 seconds of activity, ensuring sufficient data for training and analysis. This dataset includes four distinct activities, offering a diverse range of motion patterns for the study. Preprocessing involves cleaning the data by handling missing values, noise, and outliers; segmenting the continuous data into smaller segments or windows; and extracting meaningful features from each segment, such as mean, standard deviation, and entropy. Features include height to width ratio and the velocity.

Figure 5 shows some captured images of FLIR C5 thermal imager camera. Once the features are extracted, they are used to represent the activities performed by individuals. The extracted features are then represented as vectors in a high-dimensional feature space. Each vector represents a particular activity instance, and its dimensions correspond to the extracted features [26]. If the height to width ratio is less than the threshold value the person is either duck walking or crawling and if the height to width ratio is greater than threshold value person is standing. SVMs are trained using labeled activity data, with each activity instance associated with a label indicating its activity class, and they aim to find the hyper plane that best separates the feature vectors of different activity classes in the feature space by exploiting the margin between the classes while minimizing classification errors. Kernel functions, such as linear, sigmoid, radial basis function (RBF) and polynomial are used to transfer the input feature space to a space with more dimensions where the data might be linearly separable, with the choice of kernel depending on the characteristics of the data and problem domain. Once trained, the SVM can classify new activity instances by determining which side of the hyperplane they fall on, with the decision boundary separating different activity classes. Metrics like accuracy, precision, recall, F1-score, and confusion matrix are used to assess the performance of the SVM model, while cross-validation methods like k-fold cross-validation are frequently employed to prevent overfitting and gauge the model's capacity for generalization [27].

5.jpg

Figure 5. Captured images of FLIR C5 thermal imager camera

2.2 Method

This section provides an in-depth discussion on various object detection models employed for human detection in videos. One of the key components utilized is the NVIDIA GeForce RTX 3080, a high-performance graphics card from the RTX 30 series family, known for its exceptional processing power and speed, which significantly enhances the efficiency of object detection tasks [28]. Among the object detection models discussed is MobileNetV2. MobileNetV2 is highlighted for its lightweight architecture and speed, making it ideal for real-time applications. Additionally, the section covers the use of Support Vector Machine (SVM) for classification tasks within the detection process. SVM is a powerful algorithm that contributes to the precise classification of detected objects, enhancing the overall reliability of the detection system [29]. By integrating these advanced object detection models and CNNs, the section underscores the comprehensive approach taken to achieve high-accuracy human detection in videos. The combination of robust hardware like the NVIDIA GeForce RTX 3080 and sophisticated algorithms like MobileNetV2 illustrates the meticulous effort to optimize both detection speed and accuracy. This thorough examination provides a clear understanding of the methodologies and technologies employed in this research for effective human activity detection and classification.

Thresholding Algorithm: Detection of human activity using a Thresholding algorithm involves setting specific thresholds to classify activities based on sensor data. This method is simpler compared to machine learning approaches and works well for distinguishing between different activities when the signal characteristics are distinct. Detecting human activity using a Thresholding algorithm involves setting specific thresholds based on sensor data characteristics to classify different activities.

6.jpg

Figure 6. Human detection (Example of duck/crawling)

This method is straightforward and efficient for real-time applications but may struggle with complex activities and noisy data. Figure 6 shows human detection (Example of duck/crawing) by using Thresholding algorithm.

Support Vector Machine (SVM): A popular supervised machine learning approach for both regression and classification applications is the Support Vector Machine (SVM). It is a strong and adaptable tool. Finding the best border or hyper-plane to divide data points into distinct classes within a dataset is the main goal of a Support Vector Machine (SVM). This optimal boundary maximizes the margin between the classes, thereby ensuring a clear distinction between them. To accurately categorize various physical activities from a set of data, Support Vector Machines (SVM) are used in human activity detection. This process entails multiple processes.

The SVM classifier’s parameter settings are essential for optimizing human activity recognition. Key parameters include the kernel function, penalty factor (C), and gamma [30].

Kernel Function: The kernel transforms input data into a higher-dimensional space to enable linear separability. The linear kernel is ideal for high-dimensional, linearly separable data. The RBF kernel is preferred for non-linear data, capturing complex patterns well, making it popular for activity recognition. Polynomial and sigmoid kernels are less common due to higher complexity. The selection process typically starts with a linear kernel, progressing to RBF or polynomial if needed, with cross-validation to evaluate each.

Penalty Factor (C): The C parameter balances low training error with a simple decision boundary. A high C narrows the margin and reduces misclassification but risks overfitting if there is noise. A low C creates a broader margin, promoting generalization. The tuning process begins with a moderate C value and uses grid search and cross-validation to find an optimal balance.

Gamma (for RBF and Polynomial Kernels): Gamma influences individual training samples’ effect on the boundary. A low gamma gives a smoother boundary, fitting well for simple data. A high gamma allows the boundary to capture finer details, which may lead to overfitting. Cross-validation helps select a gamma that balances detail retention with generalization.

General Principles: Cross-validation is essential for robust parameter tuning, along with grid or random search for systematic exploration. Accuracy, precision, recall, and F1-score are standard metrics used to evaluate the model’s suitability for human activity recognition. These strategies allow SVM to adapt well to data features, achieving high recognition accuracy without overfitting.

Figure 7 shows human detection (Example of walking) using SVM. The following are the main steps in this process: gathering data, extracting features, preprocessing the data, training the model, evaluating the model, and making predictions. The first stage in identifying human activity is data collection. Numerous sensors, including accelerometers, gyroscopes, cameras, and wearable technology, may provide this data. The sensors record the gestures and motions of people engaging in a variety of activities, including as walking, running, and duck walking and crawling. The next stage after data collection is to identify relevant features in the unprocessed sensor data. Time-domain, frequency-domain, and statistical features are all used in feature extraction. The following stage is data processing, which entails a number of procedures including labeling, segmentation, and normalization to get the data ready for SVM model training [26]. Following that, selecting a kernel function and splitting the data are steps in the training of an SVM model. After training, the SVM model's performance is evaluated on the testing set.

7.jpg

Figure 7. Human detection (Example of walking)

Figure 8 outlines a process for creating and evaluating a human activity classification model using video data. Here's a detailed explanation of each step is given.

(1) Original Data Samples (200 videos): The process begins with a dataset consisting of 200 video samples.

(2) Feature Extraction (h/w and Velocity): Features are extracted from the video samples. These features include height-to-width ratio (h/w) and velocity, which are key indicators of human activity.

(3) Extracted Features: The extracted features are processed to make the data more manageable and improving the efficiency of the subsequent steps.

(4) Division of Samples into Testing and Training Samples: There are two sections to the dataset:

(5) Training Data Samples: A portion of the data is used to train the classification model.

(6) Testing Data Samples: Another portion of the data is set aside to test the performance of the trained model.

8.jpg

Figure 8. Flow Chart using SVM algorithm

(7) SVM (Support Vector Machine): The training data samples are used to train a (SVM) Support Vector Machine, a type of machine learning algorithm that is effective for classification tasks.

(8) Trained Classification Model: The output of the SVM training process is a trained classification model that can classify new data based on the learned features.

(9) Comparative Analysis: By comparing the trained model's predictions, the model's performance is assessed on the testing data samples against the actual classifications.

(10) Performance Evaluation: The results of the comparative analysis are used to evaluate the performance of the classification model. This step determines how well the model performs in terms of accuracy, precision, recall, and other metrics.

(11) Desired Results (Class 1, 2, 3, 4): The ultimate goal is to achieve desired classification results, categorizing the human activities into predefined classes (e.g., Class 1, Class 2, Class 3, Class 4).

MobileNET: Image segmentation is a fundamental task in computer vision, where images are divided into distinct, meaningful regions based on shared visual features like color, intensity, or texture. This method can identify damaged areas in CT scans by isolating specific regions through segmentation. The development of Convolutional Neural Networks (CNNs) has greatly advanced image segmentation capabilities, especially with architectures like MobileNetV2.

MobileNetV2 begins with a fully convolutional layer containing 32 filters, followed by 19 residual bottleneck layers, which streamline feature extraction. To improve resilience in low-precision computations, ReLU6 is used as the activation function. This architecture maintains a consistent structure with a 3x3 kernel size, using dropout and batch normalization to optimize training. An expansion factor of 6 is applied in most layers, which increases the number of channels in the feature maps, enabling more expressive intermediate representations while keeping the network compact [31].

MobileNetV2 is particularly well-suited for human activity recognition (HAR) and similar applications due to its unique design components: inverted residuals, linear bottlenecks, and depthwise separable convolutions. Inverted residuals are a key feature that contrasts with traditional residual blocks by transforming the input through a “wide-to-narrow-to-wide” structure. The initial 1×1 convolution expands the feature space, a 3×3 depthwise convolution processes it while retaining spatial details, and a final 1×1 convolution compresses it back to its original size. This design allows MobileNetV2 to capture intricate spatial relationships and subtle motion patterns critical for differentiating between similar activities like walking and running. Linear bottlenecks enhance this design by preserving detailed information within compressed layers. Conventional convolutions apply non-linear activations that can obscure low-dimensional information, which is especially problematic in applications requiring subtle distinctions, such as HAR. MobileNetV2 replaces this with a linear activation at the end of bottleneck layers, preserving fine-grained details in motion and posture, ultimately improving recognition accuracy. Depthwise separable convolutions further enhance MobileNetV2's efficiency, a core element carried over from MobileNetV1 [32].

9.jpg

Figure 9. MobileNet computer vision model

Here, each filter operates independently on input channels before combining the results in a separate 1x1 convolution. This significantly reduces computational costs and the number of parameters, making the network compact and suitable for real-time applications.

Figure 9 gives the architectural structure of Mobilenet model. MobileNetV2, an open-source CNN developed by Google, is well-optimized for embedded applications with constrained resources. It combines speed and accuracy, making it highly effective for tasks like HAR, where it can accurately and efficiently process image data on mobile or low-power devices, ensuring that complex human activities are detected swiftly and reliably.

2.3 Flow chart and algorithm of using human tracking and future location estimation

Figure 10 illustrates the flowchart for a human tracking and future location estimation algorithm. This flowchart illustrates a process for analyzing video input to classify human motion based on certain criteria. Step-by-step explanation of each stage is presented here.

(1) Input Video: The process begins with an input video that needs to be analyzed.

(2) Video Deframing: The input video is divided into individual frames for further analysis.

(3) Motion Estimation: Movement is estimated from the sequence of frames. This step involves detecting changes between consecutive frames to identify motion.

(4) Plotting Bounding Box: A bounding box is plotted around the detected motion. The bounding box typically outlines the moving object (e.g., a person).

(5) Centroid Locating: The centroid (center point) of the bounding box is located. This helps in tracking the movement of the object.

(6) Height to Width Ratio: The height-to-width ratio of the bounding box is calculated to help distinguish between different types of motion:

If the ratio is less than 1.4, the motion is classified as a crawl or duck.

If the ratio is greater than 1.4, the motion is classified as a walk or run.

10.jpg

Figure 10. Flow chart using human tracking and future location estimation algorithm

(7) Spatial Displacement: For objects classified as crawling or ducking, spatial displacement is measured to further refine the classification:

If the maximum velocity is less than 19, the motion is classified as a crawl or duck.

For objects classified as walking or running, spatial displacement is also measured:

If the maximum velocity is less than 22, the motion is classified as slow (walking).

If the maximum velocity is 22 or more, the motion is classified as a run.

The flowchart in Figure 10 uses these criteria (height-to-width ratio and spatial displacement/velocity) to classify the detected motion into categories such as crawling, ducking, walking, and running.

For human tracking future location estimation algorithm.

Location of object is detected in few initial consecutive frames.

Consider that p1(x1,y1), p2(x2,y2), p3(x3,y3) are locations of object detected in consecutive frames F1, F2, and F3.

S1 is line segment between point p1 and p2. And S2 is line segment between poin p2 and p3. These locations are connected together with straight lines.

Suppose video is of n FPS (frames per second).

Time difference between two frames (Eq.(1))

$T f=\frac{1}{F P S}$      (1)

Length of each line signifies special distance travelled by object in between two consecutive frames. Velocity of object travelling can be detected by using mathematical formula. Differences between slope angles of line segments are also calculated and will be stored in arrays.

Length of Segment S1 (as shown in Eq. (2))

$\mathrm{L}(\mathrm{S} 1)=\sqrt{(\mathrm{x} 1-\mathrm{x} 2)^2-(\mathrm{y} 1-\mathrm{y} 2)^2}$        (2)

Length of Segment S2 (as shown in Eq.(3))

$L(S 2) \sqrt{(x 2-x 3)^2-(y 2-y 3)^2}$        (3)

Velocity of object travel between point p1 and p2 is Vs1 (as shown in Eq. (4)) and that for between point2 and point3 is Vs2 (as shown in Eq. (5)).

$\mathrm{Vs} 1=\mathrm{L}(\mathrm{S} 1) / \mathrm{Tf}=\mathrm{FPS} \times \sqrt{\left[(x 1-x 2)^2+(y 1-y 2)^2\right]}$       (4)

$L \mathrm{Vs} 2=\mathrm{L}(\mathrm{S} 2) / \mathrm{Tf}=\mathrm{FPS} * \sqrt{(x 2-x 3)^2-(y 2-y 3)^2}$              (5)

Further angle can be estimated to predict future location:

Angle of line segment S1 (as shown in Eq. (6))

$\mathrm{A}(\mathrm{S} 1)=\tan ^{-1}\left[\left(y_2-y_1\right) /\left(x_2-x_1\right)\right]$          (6)

Angle of line segment S2 (as shown in Eq. (7))

$\mathrm{A}(\mathrm{S} 2)=\tan ^{-1}\left[\left(y_3-y_2\right) /\left(x_3-x_2\right)\right]$         (7)

Angle difference is shown in Eq. (8).

$\begin{gathered}\text { Angle difference }=\tan ^{-1}\left[\left(y_3-y_2\right) /\left(x_3-x_2\right)\right]-\tan ^{-1}\left[\left(y_2-y_1\right) /\left(x_2-x_1\right)\right]\end{gathered}$          (8)

Due to motion inertia, angle difference found above is likely to be continued for line segment between frames F3 and F4.

So, using angle difference and velocity of previous line segment’s new location of moving object can be predicted.

Knowing velocity of object and radius of curvature of travel path of object, future locations of object will be estimated.

MobileNetV2 demonstrates strong recognition performance in interference scenarios like occlusion and rapid motion due to its optimized architecture for handling complex patterns in real-time. The model’s depth wise separable convolutions allow it to capture fine details from spatial data, enabling accurate activity recognition even when parts of the body are obscured. Its inverted residual structure retains essential features from partially visible regions, allowing the network to infer activity based on available information without needing a complete view. This makes MobileNetV2 more effective than simpler models, such as Thresholding or SVM, which may miss or misclassify occluded activities due to limited feature extraction.

The lightweight structure and efficient parameters of MobileNetV2 also make it ideal for handling rapid motion in real-time. Rapid movements can blur frames, creating challenges for detection algorithms. MobileNetV2’s layers process these frames quickly while capturing subtle spatial patterns, allowing it to keep up with fast activity changes. The model captures essential motion cues, balancing accuracy with speed, making it suitable for dynamic activities like running, jumping, or abrupt posture changes.

Additionally, the linear bottleneck layers in MobileNetV2 improve generalization, enabling the model to distinguish between the subject and complex backgrounds, which is essential during occlusion and high-speed movements. This architecture minimizes noise and adapts well to varying environments, enhancing accuracy under challenging conditions.