A Vision-Based Gesture Recognition and Student Engagement Assessment Model for Interactive Educational Environments

2.1 Method framework

In educational interactive scenarios, for short-term motion features with clear instructional semantics such as raising hands, waving, and classroom demonstration gestures, the gesture recognition method proposed in this paper achieves fine-grained modeling of multi-time-scale sub-gesture actions by constructing an adaptive temporal segmentation attention module and a dynamic information fusion module. Considering the characteristics of high frequency and short duration of gesture interactions in classroom environments, the joint coordinates of the hand skeleton sequence are first transformed into high-dimensional feature vectors through a linear embedding layer to retain spatial structure information. Then, using the adaptive temporal segmentation attention module, the input feature sequence is segmented at multiple scales in the temporal dimension based on knowledge of short-term motion, capturing intra-frame joint associations and inter-frame motion trajectories of local sub-actions through different time windows. This module learns the correlation weights between different joints in adjacent frames and dynamically focuses on key semantic action segments in educational scenarios, avoiding the general treatment of entire action sequences in traditional models. Meanwhile, the dynamic information fusion module adaptively fuses the short-term features extracted by branches of different time scales through weight vector learning, which not only preserves the detailed features of individual gestures but also enhances the robustness of the model against interference factors such as complex classroom lighting and multi-student occlusion, providing high-precision basic gesture recognition results for subsequent engagement assessment. Figure 1 shows the framework of the gesture recognition model in educational interactive scenarios.

1.png

Figure 1. Framework of the gesture recognition model in educational interactive scenarios

In response to long-duration gesture interactions that may occur in educational scenarios, the method captures long-term correlations between different sub-action segments through a multi-scale temporal convolution feature fusion module. This module performs hierarchical processing on the encoded skeleton feature sequences using multi-scale temporal convolution kernels to learn both short-distance action transitions and long-distance temporal dependencies. In the L-layer stacked encoder modules, each layer achieves progressive spatiotemporal feature extraction through the cascade of the adaptive temporal segmentation attention module and the multi-scale temporal convolution feature fusion module: the former focuses on fine modeling of local sub-actions, while the latter captures semantic correlations of cross-period action segments. Finally, after global average pooling and classification layers, the model can accurately recognize gesture categories with specific instructional intentions in educational scenarios. Through interpretable temporal segmentation attention weights and multi-scale convolution feature mapping, the gesture recognition process is transformed into a human-understandable decision logic of "sub-action - time scale - semantic label," solving the problem of insufficient interpretability of traditional models for complex gesture sequences in classroom environments.

2.2 Linear embedding layer

In educational interactive scenarios, gestures serve as an important medium of communication between students and teachers, and their core features are composed of spatial coordinate sequences of hand joints. For example, the dynamic position changes of shoulder, elbow, and wrist joints when raising a hand, or the motion trajectory of the arm when waving. The linear embedding layer proposed in this paper targets such skeleton sequences composed of a small number of joints and discards the traditional visual Transformer’s approach of dividing images into fixed-size patches, directly performing linear mapping on the 3D coordinates of each joint. Specifically, through a learnable weight matrix, the coordinate vector of each joint is mapped into a d-dimensional feature vector, thereby converting the original skeleton sequence into a feature vector sequence. This joint-based direct mapping method not only retains the spatial structure information of the hand skeleton but also provides basic representations for the model to capture subtle differences in educational gestures through transformation into high-dimensional feature space. Suppose the feature vector sequence obtained after linear mapping is denoted by $\tilde{C}_0$. The frame number, number of joints, and number of feature channels are denoted by S, N, and Z, respectively. The 2D convolution with kernel size 1×1 is denoted by CONV2D_(1×1). The input feature vector sequence obtained by linear mapping for each joint is:

$\tilde{C}_0={CONV} 2 D_{(1 \times 1)}\left(A_{S E Q}\right) \in \Re^{S \times N \times Z}$                  (1)

Although the feature vector sequence HHH obtained through linear mapping contains the spatial coordinate information of joints, it lacks two key dimensions: first, the spatial hierarchical relationship of joints in the skeleton sequence; second, the temporal order information of gesture actions. In response to the temporal dependency of gestures in educational scenarios—for example, the "raising hand to ask a question" action involves continuous stages of "lifting arm → fixing wrist → opening palm"—this paper adopts the sine-cosine positional encoding method consistent with Transformer to generate encoded vectors containing time and spatial position information, denoted as PE. Among them, spatial position encoding is generated through the hierarchical index of joints in the skeleton tree, and temporal position encoding is calculated based on the frame index of the action sequence. Suppose the position of each feature vector in the input sequence is denoted by o, and the specific index of the positional encoding vector is denoted by u∈[0,1,…,Z/2], the specific formula is:

$O R(o, 2 u)=S I N\left(o / 10000^{2 u / Z}\right)$                   (2)

$O R(o, 2 u+1)={COS}\left(o / 10000^{2 u / Z}\right)$                   (3)

By adding the positional encoding vector to the feature vector after linear embedding, the model is not only provided with spatiotemporal coordinate references for gesture actions but also enhances the semantic attributes of gestures in educational scenarios. For example, through temporal encoding, the model can distinguish the engagement differences between "raising hand quickly" and "raising hand slowly"; through spatial encoding, it can recognize the action type differences between "waving with one hand" and "gesturing with both hands."

In educational interactive scenarios, gesture recognition needs to meet the requirements of real-time performance and robustness. For example, in class, teachers need to instantly capture students' hand-raising gestures to adjust the teaching pace. The design of the linear embedding layer is optimized in two aspects to adapt to this demand: First, the coordinates of joints are directly linearly mapped, avoiding the computational redundancy caused by traditional image patch segmentation, significantly reducing the input dimension and computational complexity of the model, laying the foundation for real-time processing. Second, with the supplementation of spatiotemporal information from positional encoding, the model can, when dealing with complex scenes with multiple students participating simultaneously, distinguish the gesture trajectories of different individuals through the spatial position encoding of joints, and capture the temporal correlation of continuous gestures of the same student through temporal encoding. The input feature C_IN after being processed by the linear embedding layer not only preserves the physical properties of educational gestures but also injects spatiotemporal coordinates consistent with the semantics of teaching scenarios, providing structured input for the subsequent encoder modules to model gesture sequences with clear educational intentions such as "raise hand - ask question" and "wave hand - speak," ultimately achieving accurate recognition and semantic analysis of diverse gestures in classroom interaction. The specific input feature formula is:

$C_{I N}=\tilde{C}_{I N}+O R \in \Re^{S \times N \times Z}$                     (4)

2.3 Inter-segment self-attention mechanism

In educational interactive scenarios, the semantics of gestures are usually composed of multiple sub-actions with clear instructional intentions arranged in temporal order. For example, the "raise hand to speak" action can be decomposed into three sub-action stages: "arm lifting → palm turning outward → holding still," with each stage corresponding to a specific joint co-movement pattern. The inter-segment self-attention mechanism targets this type of temporally structured feature. By dividing the input feature sequence into fixed-length, non-overlapping temporal segments along the time dimension, it limits self-attention computation within local temporal windows, thereby explicitly modeling the interaction relationships between joints within sub-actions. Figure 2 shows the architecture of the inter-segment self-attention mechanism module. Taking the commonly seen "waving to signal" gesture in the classroom as an example, this mechanism focuses within a single time segment on the dynamic correlation between the angle changes of the shoulder and elbow joints and the trajectory of the wrist joint during arm swinging. Through query, key, and value interaction aggregation, it captures the motion coordination between different joints within the same sub-action stage. For instance, when the arm swings forward, the model will automatically enhance the weight correlation between the shoulder joint rotation angle and the wrist joint displacement speed, suppressing interference from irrelevant joints, thereby accurately extracting the feature representation of the core sub-action "arm swinging." Specifically, suppose the length of each temporal segment is denoted by v, and the total number of joints in each temporal segment is N_v=N×v, then the input feature C_IN is divided along the time dimension into S_v non-overlapping temporal segments of equal length:

$C_{I N} \in \Re^{S \times N \times Z} \rightarrow \Re^{S_1 \times v \times N \times Z} \rightarrow \Re^{S_1 \times N_v \times Z}$                  (5)

2.png

Assuming that the linear mapping matrices for Q, K, and V are represented by W_Q, W_K, and W_V, respectively, Q, K, and V are obtained by applying linear mapping to C_IN as follows:

$Q, K, V=W_Q C_{I N}, W_K C_{I N}, W_V C_{I N}$                   (6)

Then, the output features can be obtained by multiplying the attention weight matrix—calculated from the correlation between Q and K—with N. Suppose the output features of the self-attention mechanism operation are denoted by C_AT, the hyperbolic tangent function is denoted by tanh, gradient stability during training is adjusted by (Z)^1/2, and the attention weight matrix calculated from the similarity between Q and K is denoted by F, the formula is:

$F=\tanh \left(Q K^S / \sqrt{Z}\right)$                  (7)

In view of the inherent hinge-joint connection characteristics of the hand skeleton in educational gestures, the inter-segment self-attention mechanism introduces a learnable topological parameter matrix O to model the physical connection relationships of hand joints. For example, in the "heart gesture," the fingertip contact between the thumb and index finger relies on the spatial position constraint between them. The matrix O can automatically learn this prior structural information through training, allowing the model to prioritize the coordinate interaction between thumb and index finger joints when processing this sub-action. Specifically, during the calculation of attention weights, the feature interaction matrix F of joints is fused with the topological matrix O through weighted addition, where β is a learnable parameter, dynamically balancing the contributions of data-driven features and domain knowledge. The output feature expression of the self-attention mechanism operation is:

$C_{A T}=V(\beta \times F+O)$                (8)

Considering the diversity of gestures in educational interaction, the inter-segment self-attention mechanism adopts a multi-head attention strategy, dividing the input features into multiple subspaces for parallel processing, with each head focusing on different dimensions of sub-action features. For example, the first head may specifically capture fine movements of the fingers, while the second head focuses on the macro movement of the arms. Through the concatenation of multi-head results, the model can retain both the detailed features and the overall shape of gestures. In complex classroom scenarios with multiple students participating simultaneously, the multi-head mechanism can also distinguish the gesture trajectories of different individuals by assigning weights through different heads. For example, in the spatially overlapping area where the student on the left is "raising hand" and the student on the right is "waving," the model can use the attention weights of specific heads to focus on their respective joint combinations. Furthermore, the fully connected layer at the end of the mechanism further performs nonlinear transformation on the multi-head output to adapt to gesture feature variations caused by lighting changes and occlusion in educational scenarios. For example, when a student's raised hand is partially occluded by a book, the fully connected layer can learn the historical trajectory of joint movement to complete the missing spatial information, thereby improving recognition robustness under complex conditions. Suppose the matrix concatenation along the feature channel dimension is denoted by CONCAT, and the output feature of the g-th attention group is denoted by C^g_ATT. Finally, the output feature C_ATT is obtained through concatenation:

$C_{A T T}={CONCAT}\left(C_{A T T}^1, C_{A T T}^2, \ldots, C_{A T T}^3\right)$                (9)

To enhance the fitting capability of the network, the inter-segment self-attention mechanism finally adopts a fully connected layer for information processing. Suppose the 1×1 convolution operation is denoted as CONV2D_(1×1), its output features are obtained by the following formula:

$C_{D J_{-} A T T}=\operatorname{CONV} 2 D_{(1 \times 1)}\left(C_{A T T}\right)$                 (10)

Although the inter-segment self-attention mechanism achieves explicit modeling of sub-actions through fixed time segments, the duration of sub-actions in educational gestures varies significantly. A single fixed segment length may lead to the fragmentation of long-duration sub-action features. Therefore, in educational scenario applications, the mechanism introduces a domain knowledge-guided segment length adaptation strategy: Based on prior classroom observations, the average duration of common gesture sub-actions is used as a baseline parameter for segment division, while allowing the model to dynamically adjust segment boundaries during training according to specific gesture types.

2.4 Adaptive temporal segmentation attention module

In educational interactive scenarios, the time dimension of gestures presents significant variability: instantaneous gestures such as "raise hand to ask a question" require capturing sudden joint trajectory changes during high-speed motion, while procedural gestures such as "demonstration of experimental operations" contain temporal combinations of multi-stage sub-actions like "grasping tool → adjusting gesture → demonstrating action." Traditional fixed-time-segment attention mechanisms struggle to balance feature extraction for both types of gestures. Short segments may fragment the continuity of procedural gestures, while long segments may include irrelevant frames for instantaneous gestures. To address this, the adaptive temporal segmentation attention module adopts parallel multi-branch modeling. According to different gesture motion speeds and stage features, three temporal scale branches—short, medium, and long—are designed to respectively focus on high-frequency features of instantaneous actions and temporal dependencies of long-duration actions. For example, when recognizing "group discussion gesture combinations," the short branch captures "joint velocity spikes at gesture switching moments," while the long branch models "spatial position transitions between consecutive gestures," thereby preserving the temporal semantic structure of educational gestures. The module architecture is shown in Figure 3.

The module splits the input skeleton feature sequence C_IN along the channel dimension into J parallel branches, with each branch processing Z/J dimensional features and corresponding to time segments of different lengths. Suppose the split operation along the channel dimension is denoted by SPLIT, then:

$C_{I N}^1, C_{I N}^2, \ldots, C_{I N}^u, \ldots, C_{I N}^J={SPLIT}\left(C_{I N}\right)$                   (11)

Suppose the output features of the u-th branch are denoted by C^u_{DJ_ATT}, and the inter-segment self-attention mechanism using time segment length u is denoted by DJ_ATT_v=u. C_IN is input to different branches for short-time motion feature modeling:

$C_{D J_{-} A T T}^u=D J_{-} A T T_{v=u}\left(C_{I N}^u\right)$                   (12)

3.png

Figure 3. Architecture of the adaptive temporal segmentation attention module

This design brings two advantages. First, targeted feature extraction: Each branch independently models sub-actions at specific temporal scales through the inter-segment self-attention mechanism. For example, the short branch performs local attention computation on a 0.5-second segment of a "quick hand raise" gesture, enhancing the weight correlation of instantaneous joint coordination between the shoulder and wrist; the long branch models a 4-second segment of a "slow demonstration gesture," capturing the smooth variation pattern of palm rotation angle over time. Second, computational complexity optimization: since the channel dimension of each branch is reduced to 1/J, the computational cost of the attention mechanism is reduced from O(S²NZ) of a single-branch scheme to O(S²N(Z/J)²), with a total computation of only 1/J² of the original scheme. This feature is highly suitable for real-time classroom interaction requirements. In scenarios with multiple students raising hands simultaneously, the model can process multiple skeleton data streams rapidly on ordinary computing devices, avoiding gesture recognition lag caused by computational delay.

4.png

Figure 4. Feature fusion diagram of the module

To address the problem of feature fusion across different temporal scales, the module introduces a dynamic information fusion mechanism, which achieves semantic integration of multi-branch features through three stages: "feature selection - weight allocation - adaptive fusion." Figure 4 provides a diagram of the feature fusion process.

(1) Global representation generation: First, the outputs of the J branches are summed and then compressed into a compact global vector via global average pooling to extract the overall motion trend of the educational gesture.

(2) Dynamic weight computation: Using a multi-layer perceptron (MLP) and Softmax layer, weight vectors for each branch are generated based on the global representation. When the input is a "hand raise" gesture, the model automatically assigns higher weights to the short branch and lower weights to the long branch. When processing an "experimental demonstration gesture," the long branch weights increase significantly to capture the temporal association of multi-stage actions. Suppose the weight vector of the u-th branch is denoted by Q_u, then the expression for the weight vector of each set of features is:

$Q=M L P\left(G A P\left(\sum_{u=1}^J Z_{D J_{-} A T T}^u\right)\right) \in \Re^{J Z \times 1 \times 1}$                     (13)

$Q_1, Q_2, \ldots, Q_J={SPLIT}(Q) \in \Re^{Z \times 1 \times 1}$                   (14)

(3) Multi-scale feature fusion: The output features of each inter-segment self-attention mechanism module are multiplied and summed with their corresponding weight vectors to obtain the final output feature:

$C_{DTRH}=\sum_{u=1}^J Q_u \otimes C_{D J_{-} A T T}^u$                      (15)

Through feature fusion, the model can retain both short-term details such as "slightly spreading fingers" and long-term context such as "arm movement trajectory." This dynamic weighting mechanism is particularly suitable for the requirement of "gesture semantics changing with teaching context" in educational scenarios. For the same "waving" gesture, in the contexts of "class end signal" and "group discussion guidance," the model automatically adjusts branch weights to generate feature representations consistent with scenario semantics.

To address interferences such as lighting variation and partial occlusion that may occur in classroom environments, the module introduces residual connections at the output stage to directly add the original input features with the fused high-level features. By introducing residual connections, the basic motion information of non-occluded joints is effectively retained, avoiding recognition bias caused by local feature loss. For example, when the wrist joint is occluded, residual connections can assist in inferring the spatial position of the wrist through the historical motion trajectory of the shoulder and elbow, improving the robustness of gesture recognition in complex environments. The output feature expression of the adaptive temporal segmentation attention module with residual connection is:

$\begin{aligned} C_{Z S_{-} A T T}=D T R H \left(C_{D J_{-} A T T}^1, C_{D J_{-} A T T}^2, \ldots, C_{D J_{-} A T T}^u, \ldots, C_{D J_{-} A T T}^J\right)+C_{I N}\end{aligned}$                  (16)

The adaptive temporal segmentation attention module constructs a mapping channel from hand skeleton motion to educational gesture semantics through a three-layer architecture of "multi-scale branch modeling → dynamic weight fusion → residual enhancement." For the teacher side, it can parse students’ instantaneous interactive behaviors such as "raise hand to ask questions" and "wave to express doubt" in real time, providing accurate references for classroom rhythm adjustment; for the student side, it can capture complex participation behaviors such as "experimental operation gesture sequence" and "group discussion gesture combination," providing fine-grained features for personalized learning analysis; for educational technology systems, its lightweight design and dynamic adaptability meet the requirements of real-time and robustness in multimodal data processing in smart classrooms, becoming a core component connecting underlying skeleton data and higher-level participation evaluation.

2.5 Multi-scale temporal convolution feature fusion module

In educational interactive scenarios, the semantic understanding of gestures not only relies on the accurate recognition of individual sub-actions but also requires capturing the temporal correlations between different sub-actions and the long-term coordinated motion features of hand joints. Although traditional short-term sub-action modeling methods can capture joint interactions within local time segments, they fail to model long-range action dependencies across segments and cannot effectively integrate overall skeleton motion with the dynamic relationship of local joints. To this end, the multi-scale temporal convolution feature fusion module combines multi-scale temporal convolution and skeleton feature fusion mechanisms to compensate for the shortcomings of short-term sub-action modeling and achieve deep analysis of long-term correlations of educational gestures. The module architecture is shown in Figure 5.

5.png

Figure 5. Architecture of the multi-scale temporal convolution feature fusion module

The module first performs multi-branch temporal modeling on the input features A_IN, using six different temporal operation branches to capture multi-scale temporal dependencies:

(1) Basic feature enhancement: 1×1 convolution adjusts channel weights to achieve cross-channel fusion of joint features, enhancing the feature expression of key joints in educational gestures. For example, in the "heart-shaped gesture," 1×1 convolution can enhance the feature weights of the thumb and index finger joints while suppressing interference from irrelevant joints.

(2) Salient frame capture: 3×1 max pooling layer extracts key frame features in the time series, suitable for identifying peak frames of arm lift in the "raise hand" gesture or extreme points of arm swing in the "wave" gesture, avoiding redundancy dilution of core motion features.

(3) Multi-scale temporal modeling: Four 3×1 dilated convolutions with different dilation rates are used to respectively capture short-distance action transitions, medium-distance action connections, and long-distance action dependencies. For example, in the "experimental operation gesture sequence," branches with small dilation factors capture the rapid closure of finger joints during "grasping the tool," while branches with large dilation factors model the angle stability of wrist joints during "sustained gripping."

All branch outputs are concatenated along the channel dimension and fused through a 1×1 convolution to form an intermediate representation containing multi-scale temporal information, which not only retains short-term details such as "micro finger movements" but also extracts long-term trends such as "arm movement trajectories," providing rich temporal feature input for subsequent skeleton feature fusion. Specifically, suppose the feature of the u-th joint is denoted as A_u, and the number of joints is denoted as N. Global hand skeleton features containing global information can be obtained by applying average pooling along the joint dimension on A_IN:

$A_T=\frac{1}{N} \sum_{u=1}^N A_u$                  (17)

Suppose the output features of the six branches are denoted as A₁,A₂,…,A₆, the concatenation operation along the channel dimension is denoted as CONCAT, and the 1×1 convolution operation is denoted as CONV2D. The multi-scale temporal convolution computation process is:

$A_{M T C}={CONV} 2 D_{(1 \times 1)}\left({CONCAT}\left(A_1, A_2, \ldots, A_6\right)\right)$                    (18)

6.png

Figure 6. Schematic diagram of module dynamic information fusion

To address the correlation between overall skeleton motion and local joint points in educational gestures, the module introduces skeleton-joint feature fusion guided by dynamic weights. The schematic is shown in Figure 6. First, average pooling over the joint dimension is performed to generate skeleton features containing global information of all joints, representing the overall motion pattern of the gesture. Furthermore, the input features and skeleton features are both dimensionally reduced via 1×1 convolution, and the fusion coefficient between the two is calculated through a learnable weight vector to obtain the output feature. For example, in the "raise hand" gesture, this mechanism enhances the feature weights of dominant joints such as the shoulder and elbow, while integrating the vertical motion trend of the entire skeleton to avoid recognition bias caused by occlusion of local joints. In "group discussion gestures," guided by skeleton features, the model can capture spatial position transitions between different gestures. Suppose the learnable weight vector is denoted as Q_k, then the output feature expression of the multi-scale temporal convolution feature fusion module is:

$A_{O U T}=M T C\left(A_{I N}\right)+M T C\left(A_T\right) Q_k$                   (19)

This fusion mechanism breaks the traditional method of modeling individual joints in isolation and establishes a dynamic relationship between local joint features and global skeleton motion, especially suitable for complex gestures in educational scenarios that require multi-joint coordination.

From the normalized confusion matrix in Figure 7, it can be observed that the diagonal elements generally present high values. The prediction accuracy of original class 61 is 1, class 65 is 0.94, class 70 is 0.75, class 75 is 0.7, and class 80 is 0.98. The off-diagonal elements are all relatively low, for example, the misclassification probability of 61 as 65 is only 0.01, 65 as 66 is 0.06, and 70 as 69 or 71 is both less than 0.1. This indicates that the gesture recognition model has excellent classification performance across categories, with high recognition accuracy for common classroom gestures and low inter-class confusion. Experimental data verifies the effectiveness of the image processing algorithm proposed in the paper: by optimizing the algorithm, not only the dependence on special devices is reduced, but also the accuracy and real-time performance meet practical standards. The recognition of core classroom gestures is almost error-free, providing reliable basic data for subsequent engagement evaluation.

7.png

Figure 7. Normalized confusion matrix

Table 1. Ablation experiment of inter-segment self-attention mechanism module

Time Segment Length	Training Set	Test Set
1	88.5	92.5
2	88.3	93.4
3	91.5	94.8
4	87.4	93.2
5	86.3	93.5
6	86.5	93.7

Table 4. Accuracy comparison of different methods on SCB-dataset and HaGRID datasets

Table 5. Accuracy comparison of different methods on HandPose-v3 dataset

From the data in Table 4, the test accuracy of the proposed method on SCB-dataset and HaGRID datasets is 96.8% and 92.5%, respectively, significantly better than similar methods. On the SCB-dataset, the test accuracy of the proposed method is higher than GCN-based methods such as CKGCN, and the computation and parameter size are balanced, reflecting high-precision recognition capability for classroom gestures. On the HaGRID dataset, the 92.5% test accuracy also outperforms most comparison methods, verifying the algorithm's generalization to gestures in different scenarios. Through improvements in image preprocessing, feature extraction, and pattern recognition, the proposed method achieves low device dependency, high accuracy, and real-time performance, providing reliable gesture feature input for engagement evaluation. From Table 5, it is seen that the proposed method achieves a Top-1 accuracy of 94.5% and a MeanTop1 accuracy of 92.5% on the HandPose-v3 dataset, significantly outperforming GCN, CNN, and Transformer-based methods. This indicates that the image processing algorithm proposed in this paper achieves high-precision recognition of classroom gestures by improving image preprocessing, feature extraction, and pattern recognition, without the need for special equipment and meeting real-time requirements.

From the data in Table 6, it can be seen that the proposed method achieves a Top1 accuracy of 87.9% on the ChaLearnLAP-IsoGD dataset using skeleton modality, significantly higher than similar CNN and Transformer methods. This result indicates that by utilizing the skeleton modality combined with improved image preprocessing, feature extraction, and pattern recognition strategies, the proposed algorithm has better recognition ability for common gestures such as raising hands and waving in general classroom environments. For example, the skeleton modality can more stably capture the spatiotemporal dynamics of gestures, and can still maintain high accuracy even in classroom scenes with changing light or complex backgrounds, and does not require special equipment, meeting real-time requirements. This validates the effectiveness of the proposed recognition method in complex educational interaction scenarios and provides a reliable gesture feature basis for subsequent engagement evaluation, ensuring the accuracy of quantitative analysis.

Table 6. Accuracy comparison of different methods on ChaLearn LAP IsoGD dataset

Based on the high accuracy of gesture recognition, the engagement evaluation model can perform deep fusion analysis of multidimensional gesture features with teaching contexts. The dynamic weight strategy makes the evaluation results more aligned with actual classroom scenarios. For example, in experimental operation classes, the weight of “operation demonstration gesture” can be raised to 0.5 to highlight practical engagement. Gesture duration can be captured through skeleton modality to achieve fine-grained quantification of students’ engagement status. For example, if a student frequently raises their hand in class, consistently gives likes, and seldom shrugs, a calculated engagement score can be obtained by combining the weights of different teaching segments. This result reflects that the student has a high level of active participation, positive emotional investment, and low confusion, providing teachers with personalized teaching suggestions. For students with high confusion levels, targeted tutoring plans can be designed; for students with insufficient active participation, more interactive tasks can be added. The teaching rhythm can be adjusted according to the emotional feedback score to improve overall engagement.