Video Trajectory-based Event Recognition using Hidden Markov Models. Reconnaissance D’Événements Vidéos par L’Analyse de Trajectoires À L’Aide de Modèles de Markov

We address the problem of dynamic event recognition in videos. This is motivated by increasing needs for contentbased exploitation of video footage, as encouraged in numerous applications, e.g., retrieving video sequences in large TV archives, creating automatic video summarization of sport TV programs, or detecting specific actions or activities in video-surveillance. It implies to tackle the well-known semantic gap between computed low-level features and high-level concepts. Considering 2D trajectories is attractive since they form computable image features which capture elaborated spatio-temporal information on the viewed actions. Methods for tracking moving objects in an image sequence are now available to get reliable enough 2D trajectories in various situations. These trajectories are given as a set of consecutive positions (x, y) in the image plane over time. If they are embedded in an appropriate modeling framework, high-level information on the dynamic scene can then be reachable.

We aim at designing a general trajectory classification method that does not exploit strong a priori information on the scene structure, the camera set-up, the 3D object motions, while taking into account both the trajectory shape (geometrical information related to the type of motion and to variations in the motion direction) and the speed changesof the moving object on its trajectory (dynamics-related information). Appropriate local differential features combining curvature and motion magnitude are defined and robustly computed on the motion trajectories.

Moreover, these features are not affected by the location of the trajectory in the image plane (invariance to translation), by its direction in the image plane (invariance to rotation) and by the distance of the viewed action to the camera (invariance to scale), and may allow comparison of trajectories from different cameras. A robust enough non-parametric feature extraction framework is also proposed since local differential features computed on the extracted trajectories are prone to be noise corrupted.

To efficiently process the invariant trajectory characterization, probabilistic networks, and more specifically hidden Markov models (HMM) are used since the inherent properties of this modeling help taking into account the temporal evolution of the spatio-temporal information contained in the trajectories. Classical HMM, relying on Gaussian mixture modes (GMM), are designed to model data of sufficient sizes. Hence they may fail treating small trajectories with only few dozens of observations. An original HMM modeling, based on a uniform quantization of the observation space and dealing efficiently with small trajectories, is proposed, and an efficient HMM state number selection is also developed. To compare trajectories, a similarity measure is defined based on the Rabiner distance between HMM, and used to process the video event retrieval task.

By considering a feature having those invariances and characteristics (considering both the trajectory shapes and speed evolutions), we consider the trajectory as a dynamical pattern while other methods consider the trajectories as attached to the camera point of view. We have compared our approach with other methods to put forward the properties (spatio-temporal modeling and efficient processing of small trajectories) of the developed method. Methods relying on feature histogram comparisons, on HMM/GMM modeling and on support vector machine (SVM) were considered. A set of comparative experiments on real videos (especially Formula One and ski TV program) with classification ground truth has been conducted and showed that the proposed method supplies accurate results and offers better performances than other methods.

Résumé

Nous présentons une méthode originale de classification de trajectoires dans des séquences vidéos pour la reconnaissance d’événements dynamiques. Les Modèles de Markov Cachés (MMC) sont utilisés afin de représenter chaque trajectoire et d’évaluer leurs similarités. Nous avons pu valider notre méthode en la comparant à plusieurs autres méthodes telles que la comparaison d’histogrammes, une méthode utilisant les Séparateurs à Vaste Marge (SVM) ainsi qu’une méthode de MMC utilisant des modélisations par mélanges de gaussiennes. Des descripteurs appropriés, invariants à la translation, à la rotation ainsi qu’au facteur d’échelle sont calculés sur les trajectoires, puis exploités dans une représentation par MMC. Une méthode statistique est également proposée pour le choix du nombre d’états pour la modélisation par MMC choisie. Nous avons testé notre méthode sur deux ensembles de trajectoires issues de vidéos (respectivement de Formule 1 et de ski) obtenues par une méthode de suivi dans des vidéos de sport.