Deep Learning Based Multistage Approach for Anomaly Detection

In today's demanding world, automated video surveillance has become a crucial concern, playing a pivotal role in both individual and national security. Video from surveillance cameras can be analyzed manually, semi-automatically, or entirely automatically to find anomalies or unexpected events. Conventional surveillance systems are entirely manual and reliant on people, demanding the workforce's continuous observation and attention to analyze events or behaviors in order to determine whether the recorded activities are suspicious or unusual. Multiple video feeds must be observed by the human operators at once, which are laborious and extremely hard on their attention span, resulting in poor performance. Security officers typically use a display screen attached to a video camera to watch unusual events such as traffic accidents, fires, explosions, robberies, stampedes, etc. Human inspection of surveillance footage is not optimal, though, it requires prolonged periods of focus. As a result, an automated system is needed to identify and detect anomaly or anomalous activity.

The primary aim of automated video surveillance lies in distinguishing between normal and abnormal activities. Monitoring devices are deployed to continuously observe activities in the public spaces and identify any behavior that deviates from the normal behavior. The core objective is to promptly detect unusual human behavior, particularly in in circumstances where stopping crime and mitigating security issues require quick action. The UCF Crime dataset provides a comprehensive list of thirteen anomalous acts, including Abuse, Arrest, Arson, Assault, Burglary, Explosion, Fight, Road Accident, Robbery, Shooting, Stealing, Shoplifting, and Vandalism. Efficient tracking systems that can autonomously identifying suspicious conduct are crucial for maintaining safety. To achieve this objective, an in-depth study of human activity recognition is imperative. This involves gaining a thorough understanding of the unique characteristics of each action.

The rest of the paper is structured as follows to provide a comprehensive overview of the work. In section II related work portraits, the various methodologies used similar field. Section III describe the present methodology. Section IV elaborates the experimentation and its results. Section V discussion on performance parameters and its analysis. The last section VI is about conclusion and future scope.

The process for video-based recognition of anomalies using LSTM and convolutional neural networks is shown in Figure 1. Acquisition of datasets, extracting the videos and frames from datasets are its components. Various pre-processing techniques are like Histogram of Gradients (HoG) is applied. Filtered images are obtained and applied as an input to multistage approach. The step-wise flow of research work is shown in Figure 2.

1.png

Figure 1. The block diagram of CNN and LSTM combination model for anomalous activity detection

2.png

Figure 2. The stepwise flow of work

3.1 UCF crime database

The previous datasets used for anomaly detection were quite small in size, Sultani et al. [11] created the UCF-Crime Dataset. It contains unprocessed surveillance videos capturing 13 common real-world crime scenarios which include theft, shoplifting, vandalism, fighting, robbery, explosion, and accidents. The activities included are namely Abuse, Arrest, Arson, Assault, Burglary, Explosion, Fight, Road Accident, Robbery, Shooting, Stealing, Shoplifting, and Vandalism. The UCF crime dataset consist of 128 hours of footage comprising 1900 videos, each with 7247 average frames. These videos are evenly divided in two categories: 950 are ordinary recordings and the other 950 are uncut real-world surveillance recordings. With 70 percentages of the videos typically running for about four minutes, this dataset is particularly valuable for addressing the challenge of video categorization.

To achieve the primary objective of the system, it is essential to have wide variety anomalous activities. This requirement is effectively met by the UCF Crime dataset which includes 13 distinct and relevant actions, providing a comprehensive foundation for anomaly detection.

3.2 Multistage approach with layered CNN-LSTM

CNN is a very efficient algorithm which gives good results and it verified by different models developed [12-15]. When CNN and LSTM are combined into one model, it becomes possible to extract features from spatial data (which CNN handles well) and capture temporal dependencies in sequences (which LSTM handles well). The layered architecture has two distinct layers.

CNN Layers: CNN layers are usually the first layers that are used to extract spatial information from the input data. Pooling and convolutional procedures are possible components of these layers. Figure 3 shows the working architecture of CNN model.

3.png

Figure 3. CNN model architecture

The input image maintains a consistent size of 224×224×3 throughout the process. It is then processed through a series of convolutional layers stacked together, with ReLU (Rectified Linear Unit) serving as the activation function for each layer, involving 4,096 nodes. Through the application of max-pooling techniques, the output feature vectors' dimensions are reduced in the CNN, improving the representation's computational efficiency and manageability. In the dense layer, the Softmax activation function is employed, which is crucial for making the final classification decisions. This layer contains approximately 1,500 nodes. For a detailed overview of the implemented CNN model, refer to Table 1, which provides a summary of each layer and the associated parameter count. The convolutional layer extracts a collection of spatial features which are then pass on to the long short term memory layers for further processing. A collection of spatial features is what these layers produce, and these are subsequently sent to the LSTM layers.

Table 1. Layer-wise summary of CNN model

Layers of LSTM: A typical LSTM model architecture is shown in Figure 4. Temporal dependencies are captured by the LSTM layers through their sequential processing of the spatial information. Understanding long-range dependencies in the data is made possible by the internal memory state that the LSTM layers maintain. Regression, classification, and other tasks can be performed with the final output of the LSTM layers [16-18].

4.png

Figure 4. The LSTM model architecture

LSTM architectures typically incorporate three fundamental types of logical gates that serve as the building blocks of the system. The essential elements within an LSTM are h_t and c_t, which play a crucial role in its functioning. The following Figure 5 elaborates all equations which are used to evaluate and determine the two units, h_t and c_t.

The σ is the sigmoidal function, φ is the hyperbolic tangent, ⊗ represents the product with the value of the gate and the weights of the matrix denoted by W_ij.

The LSTM model summary is provided in Table 2.

5.png

Figure 5. The LSTM model with elements and equations

Table 2. Layer-wise summary of LSTM model

CNN-LSTM multistage architecture is explained in Figure 6. Initially, the process involves extracting frames from videos, setting a threshold of 40 frames. This guarantees that the input of the video is fully standardized into frames, and the later processing is done by selecting these 40 frames. Using eighty percent of the videos in the database, the CNN model is trained. Then, the features and components of the video are extracted using the competent typical CNN architecture. A total of 25088 features are garnered from each frame, resulting in 40 sets of 25088 features. These are subsequently fed into the Long Short-Term Memory (LSTM), where each vector is processed step-by-step. The output was finally recovered from the model's dense layer.

6.png

Figure 6. System architecture

Performance parameter like accuracy, specificity, recall, precision etc. are considered for the performance analysis of the present work.

The detail analysis is discussed here.

5.1 Performance metrics

The evaluation metrics to analyse the effectiveness of the implemented model is discussed below.

Accuracy: To determine the percentage of values that were successfully classified, accuracy has been used. It indicates the number of times the used classifier is accurate. It is calculated by dividing the total values by the sum of all true values.

$\mathrm {Accuracy}=(\mathrm{TP}+\mathrm{TN}) /(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN})$             (1)

Precision: The ability of the model to accurately classify positive values is computed using precision. It is calculated by dividing the total number of projected positive values by the true positives.

$\mathrm{Precision}=\mathrm{TP} /(\mathrm{TP}+\mathrm{FP})$               (2)

Recall: The model's predictive capacity for positive values is computed using it. "What is the frequency of the model's correct positive value predictions?" It is the true positives divided by the total number of actual positive values.

$\mathrm{Recall}=\mathrm{TP} /(\mathrm{TP}+\mathrm{FN})$              (3)

F1-Score: It is the precision and recall harmonic mean. When precision and recall must be considered simultaneously, it can be helpful.

$\begin{gathered}\text { F1score }=2 *(\text { precision } * \text { recall }) /(\text { precision }+ \text { recall })\end{gathered}$              (4)

The ratio of accurate negative predictions to the total number of negatives is used to calculate specificity. Another name for it is true negative rate (TNR). Specificity ranges from 0 to 1, with 1 being the finest.

$\begin{gathered}\mathrm{TNR} / \text { Specificity }=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}} \\ \mathrm{FPR}=\mathrm{FP} /(\mathrm{TN}+\mathrm{FP})\end{gathered}$            (5)

The performance parameters precision, recall, F1 score and accuracy can be better understood by the graphical representation. Also, the class wise comparison for the single parameter can be easily done by Figure 8 and Figure 9.

8.png

Figure 8. Class wise performance parameters

9.png

Figure 9. Class wise line graph for precision, recall, accuracy and F1 score

5.2 Training and validation accuracy and loss curves

The training and validation loss values are crucial for assessing model performance as they provide insight into how the model's learning evolves over epochs. These metrics help diagnose issues related to underfitting or overfitting and guide the selection of optimal model weights for inference.

Training and Validation Loss are vital parameters for defining the model performance, as discussed below.

Training Loss: Represents how well the model fits the training data. A decrease in training loss over epochs indicates that the model is learning and improving its performance on the training set.

Validation Loss: Reflects how well the model generalizes to unseen data. A decreasing validation loss suggests that the model is also performing well on the validation set, indicating good generalization.

The loss values are typically plotted against the number of epochs to visualize learning performance. The Figure 10 depicts the training and validation accuracy curves for 2 and 10 epochs. It also shows that accuracy is increased as the number of epoch increases. It indicates that the model’s performance improves as it continues to train. The Figure 11 shows the training and validation loss curves for 2 and 10 epochs. It reveals that the loss function decreases as the number of epochs increases, indicating that the model becomes better at minimizing errors over time.

By analyzing these curves, it can be determined:

(1) Learning Trends: A decreasing loss and increasing accuracy with more epochs generally indicate effective learning.

(2) Optimal Epochs: The epoch where the validation loss stabilizes or starts to increase can be chosen to avoid overfitting. This epoch represents the point where the model weights should be used for inference.

Monitoring these metrics helps in selecting the best model configuration and preventing potential issues such as overfitting or underfitting.

10.png

Figure 10. Training and validation accuracy for 2 and 10 epoch

11.png

Figure 11. Training and validation loss for 2 and 10 epoch

5.3 Confusion matrix

A confusion matrix provides a comprehensive overview of the performance of a classification model by displaying a table layout of the prediction verses actual outcome. The confusion matrix shows individual class-wise accuracy, through which one can identify the percentage accuracy of all 12 classes which is shown in Figure 12.

The experimental results of the CNN-LSTM model are derived on a most challenging dataset that is UCF Crime dataset. The other publicly available databases are having single viewed, similar background, typical angle videos but in UCF crime dataset there is a great variety of actions and videos for single action. Other datasets commonly used basic actions whereas UCF dataset comprises crime activities happened at real time grounds and captured by CCTVs.

The developed system faced main challenge during experimentation is the processing time required for model performance. This is very crucial thing which effects on the overall performance on the video surveillance system. The response time required foe the system must be as small as possible so that the corrective actions can be taken in consideration with the anomaly detected through proposed system.

12.png

Figure 12. Confusion matrix for 12 activity recognition