Real-Time Deep Learning-Driven Surveillance with Spatiotemporal Feature Extraction for Detection of Anomalous Human Behavior Across Dynamic Environments

The development of automated human activity monitoring through Closed-Circuit Television (CCTV) is in high demand for the safety and security of individuals and their belongings. The detection of anomalous human behavior is a challenging task due to varying environmental conditions in public and private places. The method for detecting suspicious human activities in residential and public areas addresses the need for increased safety and security of individuals [1]. With the increase in CCTV installation at residential and public places, there is a significant opportunity to use this infrastructure to implement proactive safety measures. However, the traditional security methods hampered the expected security by a lack of real-time monitoring, resulting in missed possibilities for timely intervention during suspicious activities like loitering, aggressive behavior, and unauthorized access [2]. Current methods of detecting suspicious activities and public safety have relied mainly on manual monitoring, which is time-consuming, highly vulnerable to human mistake and delayed responses.

The existing automated techniques frequently failed to give real-time alerts and relied on computationally heavy and expensive frameworks [2]. Recent security measures in this regard have led to very rapid growth of surveillance systems to improve security. Most of the surveillance systems characterize human activities as the ‘normal’ or the ‘abnormal’. The traditional modes of surveillance are based on manual observation and become complex and require significant human interaction hence they are time-consuming and likely to contain errors. Surveillance systems have evolved significantly with advancements in deep learning, enabling real-time detection of anomalous human behavior. However, existing approaches often falter in dynamic and unstructured environments where factors like occlusion, lighting variability, and complex human-object interactions challenge model performance. To address such challenges, new system is proposed using deep learning techniques.

Spatiotemporal feature extraction has emerged as a promising direction, leveraging spatial and temporal dimensions for deeper context understanding. Despite its potential, its integration into real-time systems remains underexplored, particularly in dynamic, high-traffic scenarios. Convolutional, neural Networks or CNNs are among the most successful models, especially in the area of computer vision, particularly image and video. It provides the solution on major research gap of improving robustness and accuracy with ensemble-based of surveillance systems under varying environmental conditions. Firstly, extracted frames are passed through an Inception Network which is a convolutional neural network (CNN) model designed to extract important features from the input images (frames). It identifies relevant patterns in the frame like objects, textures, and other details. Secondly, it uses Bidirectional Long Short-Term Memory (BiLSTM) networks which are specialized for processing sequences, making them ideal for analyzing the temporal dynamics of the video, such as detecting actions or events that unfold over time. BiLSTM’s superior ability to recognize patterns makes it suitable to the human activity classification problem. But if they are incorporated into CNNs one can generate highly effective automatic systems for the constant analysis of voluminous video and identification of prohibited activities. To improve the temporal feature extraction process, an attention mechanism to the BiLSTM network was added.

The proposed system aims to implement a deep leaning based technique (CNN+RNN) to address the task of detecting malicious behaviors of humans in surveillance videos. This proposed model provides more efficient anomalous activity detection and provides an efficient solution for improving the existing security systems in public spaces, transport structures, and other premises. This model focuses on spatiotemporal dependencies to identify human movements and activities.

The first section introduced the basic concept of human activities' classification and the methods developed earlier. Second section provides a comprehensive overview of the existing solutions and the challenges which motivated the development of the proposed system. Third section provides the materials and methodology used in the model. The next sections are validation, results discussion and conclusion.

The proposed architecture is based on convolutional and recurrent layers to classify suspicious human activities from surveillance videos for safety and security of individuals and their belongings.

3.1 Datasets

The dataset used in this study is a selected subset of the KTH Human Motion Dataset, a well-known standard for human activity recognition tasks [22]. The dataset used in our experiment is divided into two categories: violence and nonviolence, with a total of 2000 video clips in.avi format. Each video clip is very brief, lasting several seconds, and sampled at a frame rate of 25 frames per second (fps), making it ideal for evaluating spatiotemporal patterns in human movements.

The Violence folder contains 1000 video clips that primarily feature situations of physical aggressiveness, such as boxing and street fighting. These violent sequences feature a variety of fighting actions, including hand-to-hand combat (fists and other physical altercations) that resembles real-life street violence. These videos use various angles and surroundings to replicate real-life violent scenarios. The emphasis on fistfights, particularly boxing, gives a clear and regulated set of aggressive movements, making this subset suitable for training models to recognize violent activity.

The Non-Violence folder contains 500 clips of common human activities like walking, jogging, and other non-aggressive actions. These clips were chosen to contrast with the violent activities by showing peaceful, regular behaviors. These activities, like as walking and jogging, are simple and consistent, allowing models to learn key traits that distinguish nonviolent from violent behavior. The sample data classes images are shown in Figure 1 and Figure 2, represent sample images of group activities.

1.png

Figure 1. Sample data classes images

2.png

Figure 2. Group of various activities

3.2 System architecture

The proposed system architecture is shown in Figure 3. This model is mainly divided in two phases. The first phase is designed with deep learning architecture that included Inception V3 network which eliminates the high-level features of the images and simplify the information process and used for spatial feature extraction. A Second phase is designed with Bidirectional Long Short-Term Memory (BiLSTM) networks which specialized for capturing the temporal dynamics of the videos. The model also includes an attention mechanism, which focuses on the most significant spatiotemporal variables for violence detection.

3.png

Figure 3. Video classification architecture

3.3 Data acquisition and preprocessing

The first step involves capturing video data from surveillance cameras that are fixed at various parts of the environment that have been deemed open space, transportation means, and other places. This video data becomes the input of the system, as further illustrated in the next videos on this page.

3.3.1 Preprocessing steps include:

Frame extraction: a process of partitioning the quantized footage into sections coming from the video frames.

Normalization: standardization, which involves bringing pixel values into a standard range of 0 to 1, is another commonly used feature pre-processing technique.

Data augmentation: augmentation involves flipping, rotation, scaling, cropping, and others to make the training data set more diverse and thereby minimize overfitting.

3.4 Spatial feature extraction using Inception V3

The extracted frames are passed through an Inception Network as illustrated in Figure 4, which is a convolutional neural network (CNN) model designed to extract important features from the input images (frames). Inception V3, a convolutional neural network pre-trained on the ImageNet dataset, to extract spatial features. Inception V3 is well-known for efficiently extracting detailed spatial patterns from images, thanks to its unique architecture, which includes inception modules that handle multi-scale feature extraction. This was crucial for comprehending the appearance and context of each frame.

For this task, ImageNet weights were loaded while excluding the top (completely linked) layers of the Inception V3 model. The remaining convolutional layers were frozen during training to avoid fine-tuning, allowing the model to concentrate on temporal dynamics without overfitting to spatial features.

It identifies relevant patterns in the frame like objects, textures, and other details.

4.png

Figure 4. Inception V3 model

This network outputs Transfer Values which is greater than 15, which are essentially a set of learned features that describe the content of each frame. This process loops back and extract new features so that classification process will be more accurate. After getting transfer value as per our threshold, these features are grouped together into single pattern. The inception network organizes the features from multiple frames into a sequence that captures temporal information across the video. Figure 4 illustrates the extraction of frames and identify the patterns and the convert them into the values which show the contents of frame.

3.5 Temporal feature extraction using BiLSTM

The grouped transfer values gathered from the output of Inception V3 network are fed into an BiLSTM network.

The temporal feature exactions important parameter in identifying anomalous human behavior across dynamic environments. To model the temporal evolution of actions across frames, a BiLSTM network, as illustrated in Figure 5, was employed. BiLSTM’s are very useful in video analysis because they can process data in both forward and backward directions, capturing temporal dependencies from the past and future. This is critical for identifying behaviors such as punches, kicks, and movements that lead to violent confrontations.

5.png

Figure 5. BiLSTM network

Forward LSTM: It processes the sequence in a forward direction, capturing dependencies from past to future. The forward LSTM can be denoted by the Eq. (1) where $\overrightarrow{\mathrm{h}_{\mathrm{t}}}$ represents the hidden states at time step t.

$\overrightarrow{\mathrm{h}_{\mathrm{t}}}=\operatorname{LSTM}\left(\mathrm{x}_{\mathrm{t}}, \overrightarrow{\mathrm{h}_{\mathrm{t}-1}}\right)$          (1)

Backward LSTM: It processes the sequence in a backward direction, capturing dependencies from future to past. The Eq. (2) denotes $\overleftarrow{\mathrm{h}_{\mathrm{t}}}$ which represents the hidden states at time step t for backward LSTM.

$\overleftarrow{\mathrm{h}_{\mathrm{t}}}=\operatorname{LSTM}\left(\mathrm{x}_{\mathrm{t}}, \overleftarrow{\mathrm{h}_{\mathrm{t}+1}}\right)$          (2)

Concatenation: The outputs of the forward and backward LSTMs at each time step is combined, forming a richer representation as shown in Eq. (3).

$\mathrm{h}_{\mathrm{t}}=\left[\overrightarrow{\mathrm{h}_{\mathrm{t}}} ; \overleftarrow{\mathrm{h}_{\mathrm{t}}}\right]$          (3)

By processing the sequence in both directions, BiLSTM can utilize knowledge from both past and future contexts, which is particularly effective for detecting temporal features as it requires context from both directions. The model with its parameters and values can be observed in the Table 1. The model is built using a sequential API, indicating that layers are added sequentially one after another.

Table 1. Model specific configuration

The BiLSTM networks is recurring neural network model which is capable of remembering previous states, making them useful for tasks that require learning patterns over longer time intervals. These characteristic of BiLSTM make it ideal for analyzing the temporal dynamics of the video, such as detecting actions or events that unfold over time. The BiLSTM analyzed the feature maps created by the Inception V3 model for each frame in a sequential manner, learning movement patterns and changes across frames. This temporal study was critical in distinguishing between violent activities, which frequently feature quick, powerful motions, and nonviolent actions, which are more consistent and less abrupt.

3.6 Attention mechanism

To improve the temporal feature extraction process, an attention mechanism to the BiLSTM network was added [23]. The attention layer was designed to direct the model's attention to the most relevant frames in the sequence, giving higher weight to frames that provided more critical information for recognizing violent behaviors. For example, in a boxing battle, frames with punches are more essential than frames with combatants moving into position. This selective weighting approach helps the model prioritize crucial moments, resulting in better performance.

3.7 Compatibility and benefits of ensemble modeling

The proposed system is designed and developed to capture the abnormal or anomalies human behaviour from CCTV video clips. So, there is a need of capturing video sequences and compare to recently captured video clips to make differentiate between normal or abnormal activities. This can be achieved with capturing spatial features from video frames and compare through temporal pattern to classify human activities. The Inception network superiors in extracting spatial features (what the person is doing in each frame), while the BiLSTM network efficient to handle temporal understanding (how the person’s behaviour changes over time). Combining these two strengths allows the system to capture both spatial and temporal aspects of suspicious behaviour [23, 24].

By the ensemble of two networks with attention mechanism, the proposed model can generalize better across different behaviours and scenarios. The Inception V3 Network (CNN) ensures that visual features are captured accurately, while the BiLSTM support to capture generalize behaviour patterns over time, making it better at detecting anomalies in human behaviour.

Finally, the model included a fully connected (Dense) layer with a sigmoid activation function that produced a probability score for binary classification: 1 for violent and 0 for nonviolent video clips. This score showed the model's confidence in its prediction, allowing for a clear separation of the two classes [25, 26].