Computer Vision-Based Analysis of Teacher–Student Interaction Behaviors in Dynamic English Classroom Environments

As a representative complex interactive setting, the dynamic classroom environment comprises multiple functional zones, including the lecture area, student seating area, and group discussion area. Teacher–student interaction behaviors within such environments are often characterized by cross-regional and multimodal features. Teachers may turn to face students while writing on the board or engage in close-range communication during in-class supervision, whereas students may participate through actions such as raising hands, asking questions, or engaging in collaborative group work. A single-camera perspective can capture only a partial view of the classroom scene, frequently resulting in the loss of critical visual cues—such as students’ micro-expressions and group-level gestural coordination—thus limiting the ability to reconstruct the dynamic interaction chain of "teacher instruction–student feedback–bidirectional regulation" typical of English classroom discourse. However, in a computer vision–based multiview acquisition strategy, cameras are deployed at various locations, such as the classroom ceiling and side walls, to synchronously capture multiple visual dimensions, including frontal views of the teacher's instructional posture, lateral views of student reactions, and panoramic activity trajectories. This setup enables comprehensive 3D visual coverage of teacher–student interaction behaviors. The resulting heterogeneous and multisource visual data serve as the essential foundation for accurately identifying complex behavioral patterns in English classrooms, such as question–answer exchanges, group discussions, and emotional engagement. Multiview fusion techniques function as the critical technological bridge for transforming fragmented visual inputs into coherent behavioral representations.

2.1 Model architecture design

A method for recognizing teacher–student interaction behaviors in English classroom environments based on multiview fusion was proposed. The overall model architecture is illustrated in Figure 1.

1.png

Figure 1. Overall architecture of the multiview fusion-based recognition model for teacher–student interaction behaviors in English classroom environments

In dynamic classroom environments, a multiview computer vision data acquisition setup was implemented by deploying panoramic ceiling-mounted cameras, close-up side-view lenses at the lectern, and wide-angle cameras on the classroom walls. These devices were used to synchronously capture multiview video streams that comprehensively cover teacher–student interaction scenarios. To ensure temporal and spatial coherence, spatiotemporal alignment techniques were applied during preprocessing to synchronize the multisource visual data across different camera perspectives. Following alignment, the multiview fusion module performs cross-view modeling using a feature-level fusion strategy. This process involves integrating 2D pose features—such as OpenPose keypoint coordinates—with appearance features (e.g., clothing color, teaching aids), as well as scene-context features (e.g., blackboard content, seating layout). For example, during a teaching aid demonstration, fine-grained hand movements are captured by the lectern-side camera, while the panoramic view records collective student gaze directions and head postures. The fusion module employs an attention mechanism to adaptively weight salient features from each view, thereby generating a comprehensive visual representation that encodes spatial positioning, coordinated body movements, and semantic scene context. This approach effectively resolves the issue of interaction detail loss caused by blind spots in single-view visual data.

After the completion of spatial multiview feature fusion, a multi-scale temporal attention mechanism was introduced to model the long-range temporal dynamics of video segments. This mechanism is specifically designed to capture the sequential nature of interaction behaviors in English classrooms, such as the staged action chain observed during question–answer segments (e.g., “teacher poses a question – student raises hand – student stands to respond”). To implement this, continuous video streams were segmented into clips of varying temporal resolutions, including short (e.g., 1-second), medium (e.g., 10-second), and long (e.g., full-lesson) segments. A bidirectional temporal difference network was then employed to compute motion differences across segments, enabling the identification of key frames associated with significant interaction events, such as sudden student movements (e.g., standing up) or abrupt changes in teacher gestures. Subsequently, an attention mechanism-based temporal weighting map was generated, allowing the model to focus selectively on critical interaction intervals—such as turn-taking moments in dialogue or instances of emotional resonance—while suppressing the influence of irrelevant routine actions, such as page-turning or blackboard cleaning. In group discussion scenarios, for example, the mechanism captures continuous interaction sequences such as “teacher approaches discussion group – leans in to listen – nods in response.” These temporally correlated multiview fusion features were enhanced through attention-based weight reinforcement and were ultimately integrated with segment-level motion features, resulting in a dynamic motion representation that encodes long-term temporal dependencies.

2.2 Multiview fusion module

Due to the spatiotemporal complexity of teacher–student interaction behaviors in dynamic English classroom environments—such as the motion trajectories of teachers delivering instruction while walking and the postural shifts of students engaged in multidirectional interactions—traditional video analysis methods that rely solely on height–width planar modeling have proven inadequate. These conventional approaches often fail to capture the deeper coupling between temporal (S) and spatial dimensions. Dynamic behaviors, including horizontal teacher movement from the lectern to student groups and vertical actions such as students standing to respond, are frequently segmented into isolated spatial frame sequences when analyzed on a single plane. As a result, temporal dependency information tends to be lost. To address this limitation, the proposed multiview fusion module introduces two novel analytical dimensions: height–time (G×S) and width–time (Q×S). Together with the traditional spatial plane, these dimensions constitute a 3D modeling framework. Within this framework, triaxial spatiotemporal convolutional kernels—3×1×1, 1×3×1, and 1×1×3—are employed to capture dynamic features along the temporal, horizontal, and vertical axes, respectively. This design overcomes the limitations of conventional methods in mining temporal dynamics, establishing a technical pathway for accurately decoding the spatiotemporal correlations among actions, motion trajectories, and classroom scenes within English instruction. The architecture of the multiview fusion module is illustrated in Figure 2.

2.png

Figure 2. Architecture of the multiview fusion module

The module is designed based on an input feature tensor A∈R^Z×S×G×Q, which is initially divided along the channel dimension into two branches: the raw activation branch A¹∈R^βZ×S×G×Q, and the multiview modeling branch A²∈R^{(1-β)Z×S×G×Q}. The parameter β controls the proportion between raw information retention and spatiotemporal modeling information, thereby optimizing the trade-off between feature completeness and computational efficiency. For A², multiview modeling is achieved through three directional channel convolutions:

Temporal convolution (kernel size: 3×1×1): This operation slides along the temporal axis S, capturing sequential frame-to-frame changes in behaviors such as the evolution of teacher gestures or the dynamic shifts in student facial expressions. This enables the modeling of temporal dependencies in staged interaction patterns such as “questioning–thinking–responding.”

Horizontal convolution (kernel size: 1×3×1): This operation focuses on variations along the width axis Q, facilitating the analysis of horizontal trajectories such as teacher walking paths or lateral student group movements during collaborative tasks.

Vertical convolution (kernel size: 1×1×3): This operation targets motion along the height axis G, capturing key posture changes, including student transitions between sitting and standing, or vertical body movements during teacher blackboard writing.

The outputs of the three directional convolutions—denoted as P_S, P_G, and P_Q—were fused via weighted summation across dimensions to generate a high-order representation that integrates multiview spatiotemporal information. This representation was subsequently concatenated with the raw activation branch A², resulting in a composite feature representation that preserves both local detail and global dynamic structure. Let z, s, a, and b represent the indices along the channel, temporal, height, and width dimensions, respectively. The convolution kernels used for modeling the G-Q, Q-S, and G-S views are denoted by J^S, J^G, and J^Q. The expressions for the outputs of the three directional convolutions are given by:

$P_S=\sum_u J_{z, u}^S \Pi A_{z, s+u, g, q}^1$           (1)

$P_G=\sum_u J_{z, u}^G \Pi A_{z, s, g+u, q}^1$                (2)

$P_Q=\sum_u J_{z, u}^Q \Pi A_{z, s, h, q+u}^1$            (3)

Let σ denote the activation function, and let the activated feature map be represented as P¹∈R^βZ×S×G×Q. The corresponding weights for each view are denoted as α_S, α_G, and α_Q, respectively. The fusion of P_S, P_G, and P_Q was performed using the following equation:

$P^1=\sigma\left(\alpha_S \cdot P_S+\alpha_G \cdot P_G+\alpha_Q \cdot P_Q\right)$             (4)

In this study, α_S=α_G=α_Q=1 was set. Finally, the outputs were concatenated along the channel dimension—denoted as concat—to produce the output of the multiview fusion module:

$D_u=\operatorname{concat}\left(A^2, P^1\right)$        (5)

where, D_u∈R ^{V×Z×S×G×Q}. In the context of English classroom interaction analysis, the triaxial modeling capabilities of the proposed module yield substantial advantages. For instance, when a teacher engages in a compound behavior involving both teaching aid demonstration and blackboard explanation in the lectern area, the width–time convolution captures the temporal correlation between horizontal movement and blackboard content updates. Simultaneously, the height–time convolution identifies vertical motion trajectories such as the raising or lowering of instructional tools, while the original spatial-plane convolution retains spatial relations, including fine-grained details of the teaching aid and the collective gaze direction of the students. The fusion of these three components enables precise semantic parsing of multistage interaction events, such as “teacher points to blackboard vocabulary → students engage in collective repetition → teacher nods in affirmation.” In group discussion scenarios, the horizontal convolution can be used to track the teacher's movement between groups, the vertical convolution detects height changes as students stand to speak, and the temporal convolution captures gesture exchanges and text-passing sequences, along with their temporal intervals. Together, these fused features construct a comprehensive feature vector that encodes spatial positioning, action sequencing, and interaction rhythm.

2.3 Motion feature extraction

Given the temporal dynamics inherent in teacher–student interaction behaviors within dynamic English classroom environments—such as multi-stage action chains in classroom questioning or sustained collaborative patterns during group discussions—traditional video analysis methods that rely on direct computation of long-range frame differences are prone to introducing noise and often overlook the temporal dependencies within intermediate steps. This can result in distorted representations of motion patterns associated with complex interaction behaviors. To address these limitations, the motion feature extraction module was designed around a “neighboring segment temporal-difference modeling” strategy. A multi-scale receptive field (MF) mechanism was introduced to capture fine-grained motion information and to mitigate discontinuities in long-term motion representation. The architecture of the motion feature extraction module is illustrated in Figure 3. Specifically, the input to this module consisted of the fused feature map sequence D_u, obtained from the multiview fusion module. The continuous video was first segmented into equal-length clips. Motion differentials between adjacent segments—such as frame-wise optical flow fields and pose keypoint displacements—were then computed to generate a short-term motion difference matrix. This approach avoids the computational complexity and redundancy associated with long-range frame comparisons, while precisely focusing on the dynamic progression of interaction sequences such as “question – response – feedback,” thereby providing a temporal characteristic basis for the frequent occurrence of verbal exchanges accompanied by coordinated physical gestures in English classrooms.

3.png

Figure 3. Architecture of the motion feature extraction module

A multi-receptive field strategy based on a three-branch parallel architecture was adopted within the module to perform multi-scale feature extraction on the motion difference matrix. This was achieved using dilated convolutions with varying dilation rates, enabling multi-level capture of fine-grained motion information under a shared-weights and lightweight design.

Small receptive field branch (dilation rate = 1): This branch focuses on capturing instantaneous frame-to-frame motion changes, such as subtle finger tremors during teaching aid manipulation or student micro-expressions like blinking. These features help preserve local interaction behavior details.

Medium receptive field branch (dilation rate = 2): Designed to cover a moderate temporal span of approximately 5–10 frames, this branch is suitable for analyzing the temporal correlations of staged action sequences, such as “teacher walks toward student → pauses for explanation” or “student raises hand → stands to answer.”

Large receptive field branch (dilation rate = 3): This branch processes long temporal sequences extending beyond 20 frames, focusing on the interaction rhythms of an entire lesson. Examples include the initiation–peak–resolution cycle of group discussions and fluctuations in instructional tempo over the course.

4.png

Figure 4. Architecture of the multi-receptive field module

The architecture of the multi-receptive field module is shown in Figure 4. Motion features extracted by each branch were temporally aligned through zero-padding and then concatenated to form a composite motion feature vector. This vector integrates instantaneous motion details, mid-range stage transitions, and long-range interaction patterns. Subsequently, an attention mechanism was introduced to generate an attention map over the long-term motion features. This mechanism selectively amplifies features corresponding to key interactive moments—such as physical contact between teacher and student or eye-gaze alignment—while suppressing irrelevant motions such as equipment shifts or student posture adjustments. In this way, a transformation is achieved from raw motion signals to semantically enriched features.

Formally, the temporal alignment difference between video segments D^e_u is denoted by T(D^e_u,D^e_u+1), and is defined as:

$T\left(D_{u,}^e, D_{u+1}^e\right)=J_2 * D_{u+1}^e-D_u^e$          (6)

The preliminary long-term motion information extracted from video segment u via the multi-receptive field module was then used to compute the attention map, denoted by L(D^e_u,D^e_u+1), where:

$L\left(D_u^e, D_{u+1}^e\right)=\frac{1}{3} \sum_{k=1}^3 F k\left(\bar{T}\left(D_u^e, D_{u+1}^e\right)\right)$            (7)

Finally, the attention map for long-term motion features within each segment was computed via residual connection, expressed as:

$K\left(D_u^e, D_{u+1}^e\right)=\operatorname{SIG}\left(J_3\left(\operatorname{MAX}\left(L\left(D_u^e, D_{u+1}^e\right)\right)\right)\right)$            (8)

$L\left(D_u^e, D_{u+1}^e\right)=\frac{1}{2}\left[K\left(D_u^e, D_{u+1}^e\right)+K\left(D_{u+1}^e, D_u^e\right)\right]$               (9)

In complex interactive scenes of English classroom environments, the residual connection architecture is employed to effectively mitigate gradient vanishing issues encountered in the training of deep networks, thereby ensuring the stability of multi-stage feature extraction. For example, in analyzing the compound action sequence of "blackboard writing followed by turning to pose a question," the small receptive field branch captures fine wrist movements during writing; the medium receptive field branch models torso posture changes during the turning action; and the large receptive field branch integrates the entire temporal span from the beginning of the writing to the end of the questioning behavior. Through residual connection, these features are complementarily fused, avoiding fragmentation caused by single-scale modeling. In group discussion scenarios involving simultaneous participation by multiple students, the attention map dynamically focuses on the standing actions and gestural interactions of active speakers, while also retaining the spatial trajectory of the teacher's supervisory movement. This yields a multidimensional motion representation encompassing spatial positioning, temporal sequencing, and interaction roles.

To address the technical challenges of recognizing teacher–student interaction behaviors in dynamic English classroom environments, an end-to-end method integrating multiview spatial modeling with fine-grained temporal dynamic analysis was proposed. Two core modules were developed to support this framework. By deploying multiview cameras to capture classroom video data, channel separation and triaxial spatiotemporal convolution were employed to achieve cross-view feature fusion, thereby resolving the issue of interaction detail loss caused by single-view blind spots. To address the temporal dynamics of English classroom interactions, a multi-receptive field strategy based on temporal difference computation between adjacent segments was designed. A three-branch parallel structure was constructed using dilated convolutions, enabling the model to capture instantaneous micro-movements, staged action sequences, and long-term interaction patterns. An attention mechanism was further integrated to enhance the weight of features associated with key interactive moments while suppressing noise from irrelevant scenes. The use of residual connections ensured stable training of the deep network and enabled efficient transformation from raw motion signals to semantically enriched features.

This research is tightly aligned with the multidirectional interaction characteristics of English classroom environments. Through multiview fusion, full spatial coverage of the classroom was achieved, allowing for the first time the spatiotemporal modeling of cross-perspective teacher mobility and multidirectional student feedback. The motion feature extraction module, combining adjacent segment difference calculation with multi-receptive fields, effectively addressed issues inherent in conventional methods such as high noise in long-range frame computations and disrupted temporal dependencies. Using hierarchical feature extraction across small, medium, and large receptive fields, the model was capable of capturing instantaneous features such as subtle changes in student mouth movement during repetition, as well as modeling long-term patterns such as the overall rhythm of a lesson. This provided a robust hierarchical feature representation of frequent language–gesture synchronized behaviors in educational scenarios. Ablation experiments confirmed that this mechanism improved test accuracy by 1.4% to 2.2%. These findings offer critical technical support for intelligent classroom analysis systems and enable automated semantic interpretation of teacher–student interactions, laying a foundation for applications such as instructional evaluation and structured retrieval of classroom video content. Compared to generic video analysis models, the proposed approach demonstrated superior performance in context-specific interaction recognition within educational environments, underscoring its substantial practical value.

However, several limitations in this study remain to be addressed. First, the current model relies on synchronized acquisition from multiple cameras, which introduces considerable complexity in hardware deployment and demands high spatial–temporal alignment accuracy. As a result, full adaptation to low-cost, single-camera classroom environments has not yet been achieved. Second, in scenarios involving severe occlusion, complex lighting conditions, or unstructured interactions, the model's ability to focus on critical features may be weakened, resulting in fluctuations in recognition accuracy. Third, the model training still depends on large-scale, manually annotated datasets specific to educational contexts, limiting generalization performance under low-resource or small-sample conditions. Future efforts may focus on developing lightweight multiview fusion approaches based on self-supervised learning, thereby reducing dependence on hardware. Enhancements in spatial information modeling under monocular settings should also be pursued to improve deployment flexibility. The integration of non-visual modalities such as audio and text is encouraged to construct cross-modal interaction models that couple linguistic and visual features, thereby enabling more comprehensive interpretation of compound behaviors such as spoken instructions accompanied by gestural demonstrations. Additionally, domain adaptation techniques could be explored to enhance model generalization across diverse classroom environments and instructional models.