Dance Motion Recognition and Fine-Grained Structural Parsing via Spatio-Temporal Attention Graph Convolution and Motion Primitive Learning

2.1 Overall framework overview

The dance motion recognition and structural parsing framework proposed in this paper adopts an end-to-end integrated architecture, which is divided into three clearly structured and collaboratively working stages, sequentially realizing skeleton sequence extraction, enhanced spatio-temporal feature modeling, and multi-task parallel output. The complete process from input to parsing results can be accomplished without manual intervention. The framework is based on skeleton sequence extraction. A lightweight pose estimation network is used to extract human joint coordinates from input RGB videos and construct temporal skeleton sequences. This module operates independently in the preprocessing stage and its parameters are fixed, providing a stable and less interference-prone input foundation for subsequent feature modeling. The enhanced spatio-temporal graph convolutional network (ST-GCN) serves as the core module of the entire framework, responsible for hierarchical spatio-temporal feature modeling of the skeleton sequence. By integrating innovative attention mechanisms and multi-scale temporal convolution, it fully captures the global spatio-temporal dependencies and rhythm variation characteristics of dance movements. The motion primitive learning and multi-task parallel output module serves as the goal-oriented part of the framework. Based on the spatio-temporal features extracted by the backbone network, it realizes fine-grained decomposition of motion primitives and synchronous output of action categories, frame-level motion units, and motion boundaries. The three components work collaboratively, which not only ensures high precision of motion recognition, but also achieves interpretable parsing of dance motion structure, forming a closed-loop optimization from feature extraction to result output. Figure 1 shows the end-to-end framework of dance motion recognition and structural parsing.

Figure 1. End-to-end framework of dance motion recognition and structural parsing

2.2 Human skeleton sequence extraction

To reduce the interference of irrelevant factors such as background and illumination on subsequent feature modeling [16], and to make the model focus on human dance motion itself, this paper adopts HRNet-W32 as the pose estimation backbone network to extract human key joint coordinates from input RGB video frames and construct temporal skeleton sequences. The network can stably estimate the 2D coordinates of 18 human joints in each frame. If it is necessary to further enrich motion description to improve feature representation capability, it can be extended to 3D coordinates through monocular depth estimation or 3D pose estimation networks. Finally, the temporal skeleton sequence X∈$\text{R}$^T×V×C is obtained, where T is the temporal length of the skeleton sequence, V=18 is the number of human joints, and C is the coordinate dimension with C = 2 or 3. The core optimization of the pose estimation module in this paper is that it operates independently in the preprocessing stage and all parameters are fixed, and it does not participate in the training process of subsequent enhanced spatio-temporal graph convolution, motion primitive learning, and other innovation modules. This design can effectively ensure the stability and consistency of skeleton sequence extraction, avoid the interference of parameter fluctuations of the pose estimation module on the training effect of subsequent innovation modules, and ensure that the performance improvement of subsequent modules can be accurately attributed to their own structural innovation and design optimization, providing a reliable input basis for the performance evaluation of the entire framework.

2.3 Enhanced spatio-temporal graph convolutional network

2.3.1 Overall network structure

The core design objective of the enhanced spatio-temporal graph convolution backbone network is to break through the limitations of traditional spatio-temporal graph convolution in dance motion modeling, and to achieve efficient and hierarchical extraction of global spatial relationships and temporal dynamic features of dance movements. Figure 2 shows the internal details of the enhanced spatio-temporal graph convolution block. The network constructs a spatial graph based on the natural joint connection relationships of the human body, taking each joint as a graph node and the physiological connections between joints as edges, ensuring that the spatial graph conforms to the inherent structural characteristics of human motion. On this basis, the spatial graphs of consecutive frames are extended along the temporal dimension to form a spatio-temporal graph, enabling the network to simultaneously capture the spatial positional relationships of joints and the temporal motion trends across frames. The network adopts a stacked spatio-temporal graph convolution block structure. Each convolution block integrates a spatio-temporal feature fusion mechanism. Through stacking multiple blocks, hierarchical extraction from shallow local motion features to deep global spatio-temporal features is achieved, effectively mining the complex spatio-temporal dependency relationships in dance movements. The network input is the skeleton sequence X extracted in Section 2.2. After multiple rounds of spatio-temporal graph convolution and feature transformation, the output is a spatio-temporal feature map F ∈ $\text{R}$^T×D, where T is the temporal length of the sequence and D is the feature dimension. This feature map will serve as the core input for subsequent motion primitive learning and multi-task output, providing high-precision feature support for dance motion recognition and structural parsing.

Figure 2. Internal details of the enhanced spatio-temporal graph convolution block

2.3.2 Multi-head self-attention graph convolution

The core of multi-head self-attention graph convolution lies in breaking through the inherent dependence of traditional graph convolution on predefined adjacency matrices [17]. Through the deep integration of attention mechanisms and graph convolution, adaptive learning of dependency weights between joints is achieved, thereby effectively capturing the coordinated motion characteristics of long-range joints in dance movements. Traditional graph convolution can only aggregate local neighbor information based on fixed human joint connection relationships, and cannot adapt to the complex global spatial correlations in dance movements, resulting in insufficient modeling capability for cross-part coordinated motion. To address this limitation, the multi-head self-attention graph convolution designed in this paper dynamically learns the correlation weights between joint pairs by introducing an attention mechanism, while retaining the prior of human structure, achieving collaborative optimization of local features and global features. This design is also the core point that distinguishes it from existing improved graph convolution methods.

The specific implementation of this module revolves around adaptive weight learning and feature fusion. The core steps and related formulas are as follows: first, linear projection is performed on the input joint features. Through three independent linear layers, each joint feature is mapped to a query matrix Q, a key matrix K, and a value matrix V, respectively. The dimensions of all three are ${{\text{R}}^{\text{T }\!\!\times\!\!\text{ V }\!\!\times\!\!\text{ }{{\text{D}}_{\text{k}}}}}$, where T is the temporal length of the sequence, V is the number of joints, and D_k is the feature dimension of a single attention head. To improve the diversity and comprehensiveness of feature capture, Q, K, and V are divided into 4 attention heads. This number has been verified as the optimal configuration through multiple comparative experiments, achieving a balance between computational complexity and feature representation capability. Each attention head independently calculates the dependency weights between joints, and the calculation formula is $\text{A=softmax(Q}{{\text{K}}^{\text{T}}}\text{/}\sqrt{{{\text{D}}_{\text{k}}}}\text{)}$, where $\sqrt{{{\text{D}}_{\text{k}}}}$ is used for scale normalization to avoid extreme attention weights caused by excessively high dimensions, ensuring the rationality of weight distribution. Then, the attention weight A is multiplied with the value matrix V for weighted summation to obtain the single-head attention output O_i=A_iV_i, where i is the index of the attention head. To achieve the fusion of global attention features and local structural features, the outputs of the 4 attention heads are concatenated along the feature dimension and mapped through a linear layer to obtain the global attention feature O_att. It is then fused with the traditional graph convolution output O_gcn through weighted fusion. The fusion formula is O=αO_att+(1−α)O_gcn, where α∈[0,1] is a learnable fusion coefficient, used to adaptively adjust the proportion of global attention features and local structural features, ensuring that the network can capture global coordinated motion while not losing the inherent structural information of the human body.

2.3.3 Temporal pyramid dilated convolution

The core of temporal pyramid dilated convolution lies in addressing the inherent characteristic of varying rhythms in dance movements. It breaks through the limitation of fixed receptive fields in traditional temporal convolution. Through a multi-scale dilated convolution parallel architecture, it achieves simultaneous capture of short-term instantaneous actions and long-term sustained postures without increasing the number of parameters and computational cost, constructing a temporal feature modeling mechanism that adapts to the diversity of dance motion rhythms. Dance movements exhibit obvious differences in speed. Instantaneous force-exerting actions and long-term sustained postures together form a complete dance motion pattern. Traditional single-scale temporal convolution can only cover a fixed temporal receptive field and cannot simultaneously consider the effective extraction of both types of motion features [18, 19], resulting in insufficient completeness and robustness of temporal feature modeling. To address this, this paper designs temporal pyramid dilated convolution. Through an innovative multi-dilation-rate parallel structure and feature fusion strategy, efficient mining of multi-scale temporal features is achieved. This is also the core point that distinguishes this module from existing improved temporal convolution methods.

The specific implementation of this module revolves around multi-scale temporal feature extraction and efficient fusion. The core technical details and related formulas are as follows: first, a temporal pyramid structure is constructed. In each temporal convolution layer, three 1D dilated convolutions with different dilation rates are set in parallel, with the dilation rate set d∈{1,2,3}, and the kernel size uniformly set to 3. The receptive field of dilated convolution is calculated as RF=(k−1)×d+1, where k is the kernel size and d is the dilation rate. Substituting the parameters, the receptive fields of the three convolution branches are 3, 5, and 7 frames, corresponding to short-term, mid-term, and long-term temporal feature capture, respectively. For the input spatio-temporal feature F_in∈$\text{R}$^T×D, the outputs of the three dilated convolution branches are represented as F₁=Conv1D_d=1(F_in), F₂=Conv1D_d=2(F_in), F₃=Conv1D_d=3(F_in), where Conv1D_d$~$denotes a 1D dilated convolution with dilation rate d. Its core advantage lies in expanding the receptive field by inserting zero padding into the convolution kernel, without increasing the convolution kernel parameters and computational cost. Then, the outputs of the three branches are concatenated along the channel dimension to obtain the concatenated feature F_concat∈$\text{R}$^T×3D. A 1×1 convolution is then applied for channel compression to output the final multi-scale temporal feature F_time∈$\text{R}$^T×D, with the calculation formula F_time=Conv1D_1×1(F_concat). This step not only achieves organic fusion of multi-scale features, but also ensures consistency between the feature dimension and the input of subsequent modules, while further controlling computational complexity.

2.4 Motion primitive learning and structural parsing

2.4.1 Motion primitive dictionary design

Figure 3 shows the complete process of motion primitive dictionary learning and fine-grained semantic matching. The core of motion primitive dictionary design lies in constructing learnable primitives and adaptive key segment extraction, breaking through the limitations of semantic ambiguity and lack of targeted decomposition in traditional action structure parsing, and realizing fine-grained semantic decomposition and interpretable modeling of dance movements. The continuity of dance movements and the ambiguity of semantic units make it difficult to directly perform structured parsing on continuous frame features, while fixed primitive dictionaries cannot adapt to the semantic differences of different dance movements and cannot capture distinctive basic motion units. To address this, this paper designs a learnable motion primitive dictionary, combined with temporal attention pooling to achieve adaptive extraction of key motion segments and accurate primitive assignment, constructing a parsing mechanism that can dynamically adapt to the semantics of dance movements. This is also the core point that distinguishes this module from existing action decomposition methods.

Figure 3. Process of motion primitive dictionary learning and fine-grained semantic matching

The specific implementation of this module revolves around three core steps: key segment extraction, learnable primitive dictionary construction, and primitive assignment. The core technical details and related formulas are as follows: first, key motion segment extraction is performed. The input is the spatio-temporal feature map F∈$\text{R}$^T×D output by the enhanced spatio-temporal graph convolution backbone network. Through a temporal attention pooling layer, key frame segments in the motion sequence are adaptively mined to avoid the influence of redundant frames on parsing accuracy. The temporal attention weight is calculated using the Softmax activation function, with the formula a_t=softmax(W_aF_t+b_a), where F_t is the feature vector of the t-th frame, W_a and b_a are learnable parameters, and a_t is the attention weight of the t-th frame, used to measure the importance of this frame in the motion structure. Based on the attention weights, the top K key frame segments with the highest weights are selected to form the key segment representation S∈$\text{R}$^K×D, where K=16 is determined through multiple experimental optimizations, achieving a balance between key information retention and computational complexity. Then, a learnable motion primitive dictionary D∈$\text{R}$^M×D is constructed, where M=32 is the number of primitives, and each primitive corresponds to a basic dance motion unit with clear semantics. The dictionary parameters are jointly trained with the entire network, enabling dynamic learning of semantic features of dance movements and adaptation to motion differences of different dance styles. Finally, primitive assignment is performed by calculating the cosine similarity between each key segment and each primitive in the dictionary. The similarity calculation formula is:

$\text{sim(}{{\text{S}}_{\text{i}}}\text{,}{{\text{D}}_{\text{j}}}\text{)=}\frac{{{\text{S}}_{\text{i}}}\text{.}{{\text{D}}_{\text{j}}}}{\|{{\text{S}}_{\text{i}}}\|\|{{\text{D}}_{\text{j}}}\|}$                  (1)

where, S_i is the i-th key segment and D_j is the j-th primitive in the dictionary. The similarity is normalized through a soft attention mechanism to obtain the primitive assignment probability α_ij=softmax(sim(S_i,D_j)). Finally, the continuous dance motion is decomposed into a primitive sequence of length K: $\Pi$=[argmax_jα_1j,argmax_jα_2j,…,argmax_jα_Kj].

2.4.2 Primitive dictionary optimization and reconstruction loss constraint

The core of primitive dictionary optimization and reconstruction loss constraint lies in solving the problems of non-compact semantics and insufficient representativeness during the training process of the learnable primitive dictionary through targeted loss design and joint optimization mechanism, ensuring that the primitives can accurately capture the core semantic features of dance movements. The performance of the learnable primitive dictionary directly determines the accuracy of action structure parsing. Without effective constraints, problems such as semantic overlap, redundancy, or deviation from the essence of motion may occur during training, leading to unreasonable parsing results. To address this, this paper designs a reconstruction loss and constructs a joint optimization mechanism, forcing the primitives to compactly represent motion semantics through loss constraints, while achieving collaborative optimization of the primitive dictionary and the overall network. This is also the core of this module.

To achieve the above objective, this paper introduces the reconstruction loss L_rec, which uses L2 distance to measure the difference between the original key segment features and the primitive combination features. The core formula is:

${{\text{L}}_{\text{rec}}}\text{=}\frac{\text{1}}{\text{K}}\underset{\text{i=1}}{\overset{\text{K}}{\mathop \sum }}\,\|{{\text{S}}_{\text{i}}}\text{-}\underset{\text{j=1}}{\overset{\text{M}}{\mathop \sum }}\,{{\text{ }\!\!\alpha\!\!\text{ }}_{\text{ij}}}{{\text{D}}_{\text{j}}}\|_{\text{2}}^{\text{2}}$                            (2)

where, S_i is the i-th key segment feature, α_ij is the assignment probability of the i-th segment to the j-th primitive, D_j is the j-th primitive in the dictionary, K is the number of key segments, and M is the number of primitives. This formula forces the primitive dictionary to learn features that can compactly represent motion semantics by calculating the Euclidean distance between the original segments and the weighted combination of primitives, avoiding primitive redundancy and semantic ambiguity. The averaging operation ensures the stability of loss calculation and avoids the interference of individual abnormal segments on overall optimization. During optimization, the primitive dictionary and other network parameters are jointly trained through backpropagation. The reconstruction loss is jointly constrained with subsequent multi-task loss terms, which not only ensures that the primitives can accurately match the semantic features of key segments, but also ensures that the update of the primitive dictionary is coordinated with spatio-temporal feature modeling and multi-task output, ultimately enabling the primitives to have clear semantic discriminability and strong representativeness, providing reliable semantic support for fine-grained structural parsing of dance movements.

2.4.3 Frame-level motion unit prediction

The core of frame-level motion unit prediction lies in constructing a lightweight temporal mapping mechanism to achieve accurate conversion from primitive sequences to frame-level motion unit labels, solving the semantic connection problem between primitive segments and continuous frames, while taking into account both prediction accuracy and computational efficiency. Based on the primitive sequence Π∈$\text{R}$^K obtained in Section 2.4.1, it is first mapped to a temporal feature Π′∈$\text{R}$^T with the same length as the input skeleton sequence through temporal linear interpolation, ensuring that primitive semantics can evenly cover the entire motion sequence. The interpolation formula is:

$\text{ }\!\!\Pi\!\!\text{ }{{\text{ }\!\!'\!\!\text{ }}_{\text{t}}}\text{=}\underset{\text{i=1}}{\overset{\text{K}}{\mathop \sum }}\,{{\text{ }\!\!\Pi\!\!\text{ }}_{\text{i}}}\cdot {{\text{w}}_{\text{t,i}}}$                         (3)

where, w_t,i is the temporal weight between the t-th frame and the i-th primitive segment, which is adaptively assigned based on the temporal distance between frames and segments. Then, a lightweight temporal convolutional network is introduced, which consists of two layers of 1D convolution, batch normalization, and ReLU activation functions. This design avoids computational redundancy caused by complex networks while effectively capturing the temporal correlation of motion units between frames. Its forward propagation formula is F_seg=ReLU(BN(Conv1D(Π′))), where Conv1D denotes the 1D convolution operation and BN denotes the batch normalization layer. Finally, the Softmax activation function is applied to normalize the features, outputting the frame-level motion unit prediction probability p_seg∈$\text{R}$^T×M, where M is the number of motion unit categories. This prediction result directly achieves accurate localization of motion units for each frame, completing fine-grained frame-level parsing of dance movements. At the same time, the lightweight design enables efficient inference and is compatible with the performance requirements of the entire end-to-end framework.

2.5 Multi-task output head and joint optimization mechanism

2.5.1 Three parallel output branch design

The core of the three parallel output branches lies in constructing a collaborative output mechanism that shares the spatio-temporal features of the backbone network, breaking through the limitations of independent branches and low feature utilization in traditional multi-task design, and realizing the integrated advancement of dance action category recognition, frame-level motion unit prediction, and motion boundary regression. This design not only ensures the specific accuracy of each task, but also improves the collaboration of multi-task outputs. The three branches share the spatio-temporal feature F∈$\text{R}$^T×D output by the enhanced spatio-temporal graph convolution backbone network, without the need for separate feature extraction, effectively reducing computational complexity, while enabling the outputs of each branch to have inherent semantic correlation, laying the foundation for subsequent joint optimization. Figure 4 shows the three parallel output branches and the joint loss function constraints.

Figure 4. Three parallel output branches and joint loss function constraints