A Robust Multi Descriptor Fusion with One-Class CNN for Detecting Anomalies in Video Surveillance

Video Anomaly Detection (VAD) has cemented itself as an indispensable task within the realms of computer vision and surveillance [1]. The principal objective of VAD is the automated differentiation of abnormal actions or behaviors within surveillance footage. Anomalies of interest span a broad spectrum, from acts of theft and vandalism to emergent situations such as fires or accidents. The criticality of VAD is anchored in its utility for bolstering public safety, fortifying security measures, and preempting criminal activity. The capability of VAD systems to identify and flag anomalous events in real-time confers upon security operatives the advantage of prompt response to potential threats, thereby upholding public safety [1]. Moreover, the scope of VAD transcends public security, finding relevance in domains such as industrial monitoring, traffic management, and medical diagnostics, where the early detection of irregularities is paramount. In light of the proliferation of video surveillance systems across a plethora of public spaces-airports, train stations, shopping centers, and more-the demand for automated systems capable of vigilant anomaly detection has surged. The traditional approach of manually sifting through surveillance footage is not only labor-intensive and costly but also prone to reliability issues, highlighting the imperative for sophisticated automated anomaly detection systems. These systems are paramount for the realization of surveillance that is both efficient and effective [2].

The significance of Video Anomaly Detection (VAD) in the field of video analytics has escalated, driven by its potential for the automatic identification of abnormal occurrences and potential hazards within public spaces [3]. The automated analysis of surveillance footage stands as a cornerstone for activity tracking and recognition, a function that becomes critical in environments where human surveillance is limited. Despite this, the precise detection of anomalies within video streams presents a considerable challenge, particularly due to the complexity involved when subjects traverse the fields of view of multiple cameras [4]. VAD systems have been extensively implemented on a global scale, augmenting public safety by enabling the detection of irregular activities such as altercations, theft, vehicular mishaps, and other criminal acts [5]. The urgency for robust and effective VAD methodologies has intensified in parallel with the expanding reliance on automated surveillance systems. Illustrative examples of these applications are depicted in Figure 1, showcasing frames from various datasets that are analyzed within this study.

In the realm of video surveillance, the efficacy of anomaly detection systems is paramount for ensuring public safety and security. Figure 1 illustrates eight instances of anomalies identified in various surveillance scenarios, each highlighting the critical nature of such systems [6]. Examples a and b, sourced from the UCSD dataset [7], depict anomalies in the form of a truck inappropriately traversing a footpath and a pedestrian straying onto a lawn, respectively. Examples c and d, from the Avenue dataset [8], portray a person engaged in the act of throwing an object and another individual carrying a suspicious bag. The fifth instance, example e from the MDVD dataset [9], captures a vehicle parked incorrectly, while example f from the same dataset illustrates an altercation between individuals. The final examples, g and h, extracted from the ShanghaiTech dataset [10], present a person seizing a suitcase and traffic moving on a pavement, respectively.

The process of anomaly detection is underpinned by the critical role of feature selection and extraction. It is through the discernment of features such as motion, texture, shape, and color that relevant information is culled from video frames. The ability of anomaly detection algorithms to distinguish between normal and abnormal behavior hinges on the robustness of these features. Techniques such as optical flow, histogram of oriented gradients (HOG), local binary patterns (LBP), and deep neural networks (DNNs) are employed for feature extraction. Thus, the judicious selection and accurate extraction of features are indispensable to the performance of Video Anomaly Detection systems, directly influencing their accuracy and efficiency.

Variability in the dimensionality of Video Anomaly Detection (VAD) datasets is often observed, influenced by the nature of the dataset and the plethora of features extracted from video frames. High-dimensional data are typified in VAD datasets, reflective of the extensive volume of frames and their extracted features. The number of features, contingent upon the extraction methodology employed and the video's inherent attributes, can significantly differ. Furthermore, certain datasets may exhibit increased dimensionality due to the complexity or heterogeneity of the anomalies they contain. Therefore, the dimensionality is a pivotal factor in the selection and application of feature extraction and dimensionality reduction techniques.

1.png

Figure 1. An illustration of a single surveillance video frame [6]

2.png

Figure 2. Implementation of proposed work

In the current investigation, a novel approach employing a multi-descriptor fusion alongside a One-Class Convolutional Neural Network (OC-CNN) [11] has been introduced for the purpose of anomaly detection within video surveillance frameworks. The OC-CNN, a variant of the Convolutional Autoencoder (CAE), is exclusively trained on regular samples, enabling it to reconstruct the normal patterns present in the input data. The objective of the OC-CNN is to encapsulate the quintessential structure of normal patterns, thereby utilizing this knowledge to identify anomalies. A fused feature descriptor, combining Histogram of Oriented Gradients (HOG) [12] and Local Binary Patterns (LBP) [13], has been employed to detect anomalous events in surveillance footage effectively. Termed HOG-LBP, this descriptor amalgamates gradient and textural features to garner robust and distinctive feature representations. Specifically, HOG encapsulates the image's gradient information, while LBP is responsible for textural detail. The synergy of these descriptors enhances the robustness and discriminative power of the feature representation, beneficial for anomaly detection. The methodology presented in this paper unfolds in two distinct phases, as delineated in Figure 2. The training phase incorporates the HOG-LBP with the OC-CNN to foster the learning of normal patterns. The ensuing sections of the paper are organized as follows: Section II delves into the related work, providing a comprehensive background. A detailed exposition of the HOG-LBP feature descriptor is presented in Section III. Section IV delineates the implementation details, and the paper concludes with Section V, summarizing the study's findings.

Multiple Feature Descriptors are commonly used in VAD to capture different aspects of the video data. This approach involves combining multiple types of features to obtain a more robust and comprehensive representation of the video data. Examples of feature descriptors that can be combined include motion features, texture features, shape features, and color features. By combining Multiple Feature Descriptors, the resulting feature vector can capture a more diverse set of characteristics, allowing for better detection of anomalies in complex and dynamic scenes. The fusion of feature descriptors can be done at different levels, such as early fusion where features are concatenated before classification, or late fusion where multiple classifiers are used to combine the results of each feature descriptor.

HOG and LBP are two prevalent feature descriptors considered in computer vision and image processing tasks, including VAD (Figure 3 and Figure 4). HOG is a feature descriptor that calculates the gradient orientation and magnitude in local image regions. The gradient orientations are quantized into histograms to create a feature vector representation of the local image region. HOG has been shown to be effective in detecting human shapes and movements in surveillance videos, which can be useful for detecting abnormal events. LBP is a texture descriptor that compares the intensity values of pixels with their surrounding pixels to represent the local structure of an image. The LBP operator outputs a binary code that represents the local texture pattern of the image. LBP has been applied in VAD to capture texture variations and irregularities in the video frames, which can be indicative of abnormal events. Both HOG and LBP have been used as single feature descriptors or combined with other feature descriptors to improve the performance of VAD systems. Though, it is significant to note that the effectiveness of these feature descriptors can vary depending on the specific characteristics of the video dataset and the anomaly types that are being detected. HOG and LBP are both feature descriptors used for image and video analysis.

The mathematical form of HOG can be represented as follows:

• Image gradient computation: Compute the gradients of the picture in the x and y direction using Sobel operators.
• Histogram computation: Divide the image into small cells (typically 8×8 or 16×16 pixels) and calculate the gradient magnitude and orientation for each pixel. Compute the histogram of orientations for each cell.
• Block normalization: Combine the histograms of neighboring cells into blocks (typically 2×2 or 3×3 cells) and normalize the block by L2-norm to reduce the influence of illumination variations.

3.png

Figure 3. HOG descriptor

The mathematical form of LBP can be represented as follows:

• Image patch extraction: Extract a square patch from the image centered around each pixel.
• Thresholding: Compare the concentration of each pixel in the patch to the strength of the central pixel. Assign a binary value of 1 if the pixel value is >= the central pixel value, and 0 otherwise.
• Pattern coding: Convert the binary pattern into a decimal number and create a histogram of the frequency of occurrence of each pattern in the image.

$\operatorname{LBP}\left(g p_x, g p_y\right) \sum_{p=0}^{p-1} S(g p-g c) \times 2^p$

jie_tu_20231225205148.png

Figure 4. Evaluation of LBP

Overall, both HOG and LBP are powerful and effective feature descriptors for VAD, with their mathematical forms providing a clear framework for understanding their computation.

A framework for VAD based on DL is called One-Class CNN. It is designed to recognise any abnormalities that depart from the regular patterns it has trained to recognise in a video. One-Class CNN only needs normal data for training, in contrast to conventional anomaly detection techniques that need labelled data for both normal and abnormal occurrences. One-Class CNN has the ability to simulate the typical patterns of the data during training and recognise any deviations from those patterns during testing. One-Class CNN extracts spatiotemporal properties from video input using a 3D convolutional neural network (CNN). By reducing a reconstruction error between the input frames and the model's output frames, the model learns to recognise the typical patterns of the data by ingesting video frames. While testing, the model looks for abnormalities by comparing the input frames' reconstruction error to a predetermined threshold. The model marks the input frames as anomalous if the error rises over the threshold. One-Class CNN has shown good results in a range of anomaly detection applications, including seeing unusual occurrences in surveillance footage, spotting flaws in production methods, and spotting abnormalities in x-ray pictures (Figure 5).

Using OC-CNN, features are extracted from the input frames and converted into a lower-dimensional representation using the encoder component of the network (Figure 6). The network's decoder component is then utilized to rebuild the input frames using the features that were retrieved. Only normal samples are utilized to train the network during the training phase, and the loss function is created to reduce the discrepancy between the reconstructed frames and the original normal frames. The frame is labelled as an anomaly if the difference is greater than a certain limit. The benefit of OC-CNN is that it can identify abnormalities unsupervisedly and does not need any previous knowledge of them. In numerous benchmark datasets, it has been shown that OC-CNN performs better than other cutting-edge approaches for detecting video anomalies. Its processing efficiency makes it appropriate for real-time applications.

5.png

Figure 5. Decision boundary classifier with One-Class

6.png

Figure 6. Neural networks with One-Class

Some potential future directions for research on VAD using HOG-LBP and One-Class CNN could include:

• Investigating the use of other feature extraction methods in combination with HOG and LBP, such as DL-based methods like recurrent neural networks (RNNs).
• Exploring the use of more advanced techniques for anomaly detection, such as generative models like variational autoencoders (VAEs) or adversarial networks.
• Conducting experiments on larger and more diverse datasets to evaluate the generalization capabilities of the proposed approach.
• Addressing the issue of class imbalance in anomaly detection by exploring techniques such as oversampling or undersampling, or using more advanced methods like cost-sensitive learning or ensemble methods.
• Developing methods for real-time or near real-time anomaly detection in videos, which would require optimizing for computational efficiency and minimizing the latency between video frames.
• Investigating the use of transfer learning, where models trained on one dataset are fine-tuned on another dataset, to improve the performance of the anomaly detection system.