Dynamic Tactical Image Recognition and Analysis in Football Matches Using Convolutional Neural Networks

2.1 Problem description

The positions of players in a football match constantly change on the field, influenced by various factors, including the players' own movement, interactions within and outside the field, and changes in the camera's viewpoint. Due to the high dynamics of football matches and complex tactical changes, a single motion model is difficult to adapt to all scenarios. Therefore, this paper proposes a motion prediction method that combines the Kalman filter with the Enhanced Correlation Coefficient (ECC) image registration method, aimed at overcoming this challenge. First, for predicting the movement trajectory of individual players, the Kalman filter can model each player’s motion state and predict based on historical position data, addressing the rapid movement and position changes between players. As for the deviations between video frames caused by camera movement, the ECC image registration method is used to calculate the affine vector of consecutive frames, adjusting the differences between frames to reduce the impact of camera motion and ensure the accuracy of the prediction results.

Specifically, for tactical image analysis in football matches, player movement is not merely simple linear or rigid motion; it often includes complex non-rigid motions, such as acceleration, deceleration, and turning. These motion characteristics need to be processed through refined modeling and data association methods. With high-quality detection and high frame rates, data association methods based on Intersection over Union (IoU) can effectively achieve precise tracking of target players. However, under low frame rates or in complex scenarios, more accurate prediction and adjustment of the target player's movement position are required. By combining the Kalman filter for predicting non-rigid motion and ECC for aligning rigid motion, precise multi-target motion prediction can be achieved in dynamic football scenarios. Assume that the 8-dimensional motion state [z_a,z_b,x,g,n_za,n_zb,n_x,n_g] is represented by t, where it includes the target player's center coordinates [z_a,z_b], aspect ratio x, height g, and the rate of change of these four variables [n_za,n_zb,n_x,n_g]. The motion covariance is represented by W, the prior covariance by O, and the ECC model by WA. The model can be established using the following equation:

${{t}_{s+1}}=WA\left( D{{t}_{t}} \right),{{O}_{s+1}}=D{{O}_{s}}{{D}^{S}}+W$                   (1)

Assume that the affine vector calculated by ECC is represented by Q, the affine vector of the static frame by E, the identity matrix by U, and the zero matrix by P. The intensity of the camera movement is defined by the following formula:

${{U}_{z}}=1-\frac{Q\times E}{{{\left\| Q \right\|}_{2}}\times {{\left\| E \right\|}_{2}}},E=\left[ U;0 \right]$                  (2)

Based on the above definitions, the Kalman filter can be adjusted by changing the state transition matrix. Assume that the adjusted state transition matrix is represented by D_z, and the original time step of the Kalman filter is represented by f_s. Then, the state of the target player at frame s+1 is:

$\begin{align}  & {{T}_{s+1}}=WA\left( {{D}_{z}}{{f}_{s}} \right),{{D}_{z}}=\left[ \begin{matrix}   U\left( {{f}_{s}}+{{U}_{z}} \right) & U  \\   0 & U  \\\end{matrix} \right] \\ & {{O}_{s+1}}=x{{D}_{z}}{{O}_{s}}D_{z}^{S}+W \\\end{align}$                  (3)

After obtaining T_s+1, the vector [z_a,z_b,x,g] of the position part can be further extracted, and the predicted bounding box of the target player in the dynamic tactical image frame of the football match can be calculated.

2.2 Network architecture

For the multi-target motion prediction problem in dynamic tactical image frames of football matches, this study designs a network architecture based on Faster-RCNN. By integrating CNNs and a motion prediction module, the network achieves precise location prediction for multiple football players. Specifically, the network architecture combines ResNet50 and the Feature Pyramid Network (FPN) as the backbone to fully utilize feature maps at different scales, thus improving the accuracy of target player detection and motion trajectory prediction, as shown in Figure 1. On this basis, the network further introduces three functional modules: the regression head, the classification head, and the ReID head. The regression head refines the target player's bounding box and outputs the precise player position. The classification head determines whether the image region is a target player or background, ensuring the accuracy of detection. The ReID head extracts the appearance feature vector of each target player for effective association and identification across consecutive frames. This structure not only effectively handles multiple target players in dynamic scenes but also enables long-term tracking and association between target players.

1.png

Figure 1. Basic backbone network structure of the constructed model

2.3 Training process

In the training process of the multi-target motion prediction algorithm for dynamic tactical image frames in football matches, this study simulates a real multi-target tracking task. By combining target player detection and motion prediction models, the training data's representativeness and diversity are enhanced. Unlike traditional methods that train based solely on detection tasks, this study adopts a strategy that generates supplementary training samples by predicting the target player's position through a motion model. During the training phase, N consecutive frames are randomly selected from the training dataset as input. The tracking state of the target player is initialized using the true label from the first frame, including the player's position, velocity, and historical ROI feature set. Due to the rapid movement and occlusion of target players in football matches, a single target player detection model cannot handle all scenarios. Therefore, it is necessary to combine the motion model to predict the target player's position, ensuring the continuity and stability of tracking. Specifically, the target player's position is initialized using a Kalman filter, and historical ROI features are input into the ReID head to obtain the appearance features of each target player. The specific process flow is shown in Figure 2.

2.png

Figure 2. Process flow of ROI features input into the ReID Head

When predicting the next frame of dynamic tactical image frames in the football match, the losses for three parts are computed using the ground truth (GT) label: the loss from the Region Proposal Network (RPN) denoted as M^mpz_EOV and M^zmt_EOV; the loss from the regression model denoted as M^mpz_{R_H} and M^zmt_{C_H}; and the loss used to train the ReID head denoted as M_ME. Assuming the factors controlling the influence of different sub-losses on the target player function are represented by η₁, η₂, and η₃, the overall target player loss function for the constructed model is as follows:

${ LOSS }=\eta_1\left(M_{E O V}^{m p z}+M_{E O V}^{{zmt }}\right) \\ +\eta_1\left(M_{R_{-} H}^{m p z}+M_{C_{-} H}^{ {zmt }}\right)+\eta_3 M_{M E}$               (4)

In multi-target motion prediction for football matches, considering the fast movement and frequent occlusions of players on the field, the RPN loss function needs to ensure that the network can adapt to the mutual influences between target players and the rapid positional changes of target players. To meet this demand, in addition to using traditional IoU threshold-based positive and negative sample judgments, the RPN loss function needs to account for the specificity of motion prediction. For instance, in some fast-moving scenarios, the predicted position of the target player may differ significantly from the real position. In such cases, introducing supplementary samples based on the motion model can enhance the RPN’s adaptability in these complex scenarios. Additionally, since the motion trajectories of target players in football matches are highly nonlinear, the model must not only correctly identify the bounding box of the target player but also predict the potential position of the target player in future frames.

Specifically, for each anchor point, we assign labels based on the IoU between the anchor point and the target player's bounding box. Positive anchor boxes are of two types: (1) the one with the highest IoU with a specific target player's label bounding box; (2) the ones with IoU greater than 0.7 with any target player's label bounding box. Negative samples are those anchor boxes whose IoU with all target player's bounding boxes is below 0.3. As with the standard operation of Faster-RCNN, any anchor point that is neither a positive sample nor a negative sample is ignored and not included in the loss calculation. Let the index of the anchor point be represented by u, the prediction probability of the anchor box containing the target player be represented by O_u, the vector of the four parameterized coordinates of the predicted bounding box be represented by s_u, and the coordinates of the ground truth box associated with the positive anchor be represented by s^*_u. The log loss between the target player and non-target player classes is represented by M_zmt, and the loss function expression is as follows:

Figure 3. Illustration of the supplementary sample generation process

$M_{EOV}^{zmt}=\frac{1}{V}\sum\nolimits_{u}{{{M}_{zmt}}\left( {{O}_{u}},O_{u}^{*} \right)}$                    (5)

$M_{EOV}^{zmt}=\frac{1}{{{V}_{RE}}}\sum\nolimits_{u}{{{O}^{*}}_{u}{{M}_{RE}}\left( {{s}_{u}},s_{u}^{*} \right)}$                    (6)

Considering the rapid movement and potential occlusion of target players in football matches, the regression model's loss function needs to not only accurately predict the target player's position but also handle the complexity related to movement. Therefore, we construct training samples by combining the regions proposed by RPN and the predicted bounding boxes generated by the motion prediction module. During training, we first use the Ninter interpolation method to generate interpolated bounding boxes, which are created in such a way that they include both the true target player positions and are close to the motion prediction results. This allows us to generate more positive samples, especially when the number of target players is small, avoiding the problem of insufficient negative samples leading to data imbalance. The generation of supplementary samples proceeds through four steps: bounding box interpolation, negative sample generation, random scaling and shifting, and sample filtering. The process flow is illustrated in Figure 3. Each potential positive sample bounding box is trained alongside multiple negative samples to ensure that the model learns effective target player localization under various conditions.

To further enhance the regression model's performance, the regression loss function needs to account for both the balance between positive and negative samples and the accuracy of target player prediction. In dynamic football match scenes, players' movement trajectories are usually nonlinear and highly time-varying, requiring the regression model to not only consider the current frame's target player position but also predict the target player's future position. Therefore, in the loss function calculation, in addition to the traditional regression error, an error metric between the motion prediction results and the actual position is also introduced. Specifically, for each positive sample, the regression network predicts the target player's position and calculates the deviation between the predicted bounding box and the true target player. If the IoU value between the predicted box and the true target player box is below a certain threshold, it is considered a regression failure, and the error is penalized through the loss function. Additionally, the regression model must handle interactions and occlusions between target players, especially in cases of overlapping or alternating occlusions, ensuring that the model maintains stable prediction performance in a multi-target environment. Let the number of samples be represented by V_SAM, the number of positive samples be represented by V_{P_S}, and the probability of the category of target players (football players or background) within the sample bounding box be represented by O_u. The loss function expression is as follows:

$M_{Z\_H}^{zmt}=\frac{1}{{{V}_{SAM}}\sum\nolimits_{u}{{{M}_{zmt}}\left( {{O}_{u}},O_{u}^{*} \right)}}$                    (7)

$M_{R\_H}^{zmt}=\frac{1}{{{V}_{P\_S}}\sum\nolimits_{u}{{{O}^{\text{*}}}_{u}{{M}_{RE}}\left( {{s}_{u}},s_{u}^{*} \right)}}$                    (8)

In the multi-target motion prediction of dynamic tactical image frames in football matches, the design and training of the ReID head are crucial for accurately identifying and distinguishing players. To improve the stability and consistency of the model in recognizing the appearance features of the same player across different frames, this study adopts a metric loss function to optimize the extraction of appearance features in the ReID task. During training, by simulating the tracking process of target players, the appearance feature vectors are obtained from the true position of the target player in each frame, and these features are used to enhance the similarity of appearance features of the same target player across different time frames, thus strengthening their temporal correlation. This approach effectively reduces the potential issue of inconsistency in appearance features caused by insufficient random sampling in the training data. If the appearance features lack sufficient temporal correlation, the model is prone to identity switching during inference, meaning it could incorrectly identify different players as the same player or assign the same player to different identities, leading to a decrease in tracking stability and accuracy.

To further enhance the performance of the ReID head, this study adopts the idea of metric loss and makes certain adjustments. Specifically, in addition to using position data, the proposed scheme in this chapter computes the metric loss solely based on the appearance feature vector of the players, without involving position information. This is because, in a football match, the position data may become unreliable due to rapid player movements, positional changes, and occlusions. Therefore, this study focuses on judging identity consistency through the appearance features of the players. By extracting the appearance features of the target player from the current frame and comparing them with the appearance features from historical frames, the model can further encode the target player's appearance feature sequence using a Bidirectional Recurrent Neural Network (Bi-RNN). These continuous feature sequences are processed by two independent Bi-RNNs, producing phase-specific hidden layer representations, which are then converted into a soft assignment matrix through a fully connected layer. This ensures that the appearance features of each target player maintain temporal consistency, reducing the risk of misidentification. Finally, the soft assignment matrix generated through a sigmoid activation function optimizes the accurate maintenance of player identities, improving the stability and accuracy of tracking each player in multi-target motion prediction.

Specifically, let the appearance feature of the target player in the current frame extracted from the target player's labeled bounding box be represented by d_xoo. The historical appearance features of the target player and the average historical appearance feature computed by the ReID head are represented by d_{H_A}. Let the cosine distance between X and Y be denoted as COS(X, Y). The following equation gives the calculation of the feature distance between the current frame's target player appearance feature and the saved average historical appearance feature:

$DI{{S}_{xoo}}=\frac{1}{2}\left( 1-COS\left( {{d}_{xoo}},{{d}_{H\_A}} \right) \right)$                    (9)

In multi-target motion prediction for dynamic tactical image frames in football matches, the metric loss function of the ReID head mainly consists of the Multi-Target Tracking Accuracy (FMA) and Multi-Target Tracking Precision (FMP). These two indicators help optimize the model's target tracking performance in complex match environments. FMA evaluates the overall quality of the tracking results, taking into account false positives, missed detections, and identity switches. FMP measures the tracking accuracy of the target, calculating the error between the predicted position of the target in the current frame and the actual position. Let the approximate representations of false positives, missed detections, and identity switches be \mathop{DO}, \mathop{DV}, and \mathop{UFT}, respectively. The matching target players are denoted by L, the binary assignment matrix corresponding to the distance matrix F_D-A is represented by Y^SO, and the weight factor controlling the proportion is represented by ε_ME. The following formula gives the calculation:

$FMA=1-\frac{\overset{\sim}{\mathop{DO}}\,+\overset{\sim}{\mathop{DV}}\,+{{\varepsilon }_{ME}}\overset{\sim}{\mathop{UFT}}\,}{L}$                       (10)

$FMP=1-\frac{{{\left\| {{F}_{D\_A}}\otimes {{Y}^{SO}} \right\|}_{1}}}{{{\left\| {{Y}^{SO}} \right\|}_{0}}}$                  (11)

To effectively compute these metrics, this study constructs matrices Z^S and Z^Z by adding a row or column to the soft assignment matrix, respectively filling it with a threshold σ and performing the softmax operation, further refining the matching relationship between targets.

$\overset{\sim}{\mathop{DO}}\,=\sum\nolimits_{{{V}_{TA}}}{Z_{{{V}_{TA}},{{V}_{TA}}+1}^{e}},\overset{\sim}{\mathop{DV}}\,=\sum\nolimits_{{{V}_{TA}}}{Z_{{{V}_{TA}},{{V}_{TA}}+1}^{z}}$                   (12)

In the calculation of FMA and FMP, the method in this paper avoids saving the binary assignment matrix of each frame, instead directly utilizing the target matching relationship between the previous frame and the current frame to simplify the calculation of IDS. The core strategy here is that by only considering the targets that exist in both the current frame and the previous frame, the computational complexity is reduced, and real-time performance is improved. Especially in football matches, player position changes and rapid movements may lead to occlusions and mismatches. Therefore, by directly calculating the target matching status for each frame, the temporal consistency and accuracy of the ReID head are ensured. In the specific implementation, this study uses the calculation formulas for false positives, missed detections, and identity switches, further optimizing the matching of target appearance features and positions during the target tracking process, thereby improving overall tracking accuracy.

When calculating the distance matrix F_D-A, this paper only considers the target players that exist in both the current frame and the previous frame of the dynamic tactical image of the football match, and the order of the target players in F_D-A and X^~_D-A correspond to each other. Let the L₁ normalization of the flattened matrix be denoted by ||•||₁. The matrix of size V_TA×V_TA, with diagonal elements being 0 and other elements being 1, is denoted by U^-_VT. The simplified calculation formula is:

$U\tilde{F}T=\left\| Z{{_{1:}^{z}}_{{{V}_{TA}},1:{{V}_{TA}}}}{{\left. \otimes {{{\bar{U}}}_{{{V}_{TA}}}} \right\|}_{1}} \right.$                    (13)

Finally, the weight factor η_ME for controlling the sub-loss ratio is represented, and the metric loss is assumed as follows:

${{M}_{ME}}=\left( 1-FMA \right)+{{\eta }_{ME}}\left( 1-FMP \right)$                     (14)

The overall training strategy of the proposed network model is shown in Figure 4.

Figure 4. The training strategy of the proposed network model

From the data of the line chart on the left side of Figure 5, it can be seen that under the zone defense tactic, the cumulative number of defenders increases significantly over time, eventually approaching 400. In contrast, the number of defenders under the full-field pressing tactic remains relatively low and fluctuates less. The image on the right shows the tactical scenes, with different colored boxes marking different players, representing the results of tracking management. This indicates that in the zone defense tactic, the dynamic changes of players are more frequent, and the number of defenders gradually increases. This is because the tactic requires constant adjustments to defensive zones and player positioning based on the ball's location and the offensive team's changes. Therefore, multi-target motion prediction and tracking management based on CNNs face greater challenges. On the other hand, the full-field pressing tactic is relatively stable, with little change in the number of defenders, making the difficulty of tracking and prediction relatively low. Hence, the proposed method shows different levels of adaptability to various tactical scenarios with varying complexity. It is more adept at handling relatively stable full-field pressing tactics, while there is still room for optimization in dealing with the more complex and dynamic zone defense tactics.

From the multiple comparison experiment data in Table 1, it can be seen that the proposed method outperforms significantly in multi-target tracking accuracy (48.9%), main target tracking rate (28.9%), and main target missed rate (23.5%). Compared to other methods, the CNN-based multi-target tracking and motion prediction method presented in this study shows a substantial improvement in both accuracy and robustness. For instance, compared to Gradient Boosting Tree (36.2%) and Regularized Particle Filter (37.5%), the multi-target tracking accuracy of the proposed method has improved by about 12-13 percentage points. At the same time, the false alarm count and false negative count are also reduced compared to other methods. The proposed method has 2895 false alarms and 22368 false negatives, which is significantly better than methods such as Optical Flow Method (false alarm count 7263, false negative count 25146), demonstrating the reliability and efficiency of the method in practical applications.

Figure 5. Comparison of test results on different tactics

Table 1. Comparison on public sports event video screenshot dataset

Table 2. Comparison on dataset provided by professional sports data company

Table 3. Comparison on specific scene dataset

Table 4. Ablation analysis results comparison