Multimodal Image Processing and Learning Behavior Pattern Visualization for Educational Management

The overall architecture of the multimodal image feature integration model for educational management is based on an encoder–decoder framework. It aims to construct feature representations that can reveal learning behavior patterns by collaboratively analyzing synchronously collected RGB images capturing student postures and interactions, depth images quantifying spatial positions and distances, and thermal infrared images reflecting body temperature distribution and attention fluctuations in classroom scenes. The model architecture is shown in Figure 1. The model first uses three parallel encoders based on VGG16 to extract five-level pyramid features {D^F_u,D^N_u,D^S_u}⁵_u=1 of the three modalities, where deep features contain semantic behavior information, and shallow features retain spatial details. To fuse multimodal behavior representations, each feature level introduces a tri-modal feature fusion module, which integrates visual appearance, spatial relationship, and physiological response features through a cross-modal attention mechanism, generating fused features {D^SD_u}⁵_u=1. For example, the association enhancement between abnormal thermal infrared regions and head-down postures in RGB images can be achieved.

To further strengthen multi-scale behavioral context, the model designs a neighboring-layer feature enhancement module, which aggregates tri-modal fusion outputs of adjacent levels through a top-down path, and captures cross-scale dependencies from local actions to global scene layout by using dilated convolution and spatial pyramid pooling, generating enhanced decoding features D^VMR_u. In particular, to optimize the discriminability of behavior patterns, the model concatenates the semantic abstract features D^VMR₄ from the deepest layer of the encoder with the fine-grained features D^VMR₁ from the shallowest layer, obtaining joint features D^VMR_1,4, which are used to guide the decoding process in the multi-level cascaded feature integration module. This module progressively fuses behavioral features at different resolutions through gated recurrent units, avoiding detail loss while suppressing background interference. Finally, the decoder block reconstructs spatial resolution and outputs a high-quality learning behavior saliency map T. This saliency map not only highlights key behavioral regions but also provides quantitative evidence for educational managers through feature back-projection interpretability mapping, supporting decision-making applications such as classroom engagement evaluation and teaching strategy adjustment.

2.1 Tri-modal feature fusion

The core goal of the tri-modal feature fusion module for educational management is to construct more robust learning behavior representations by complementary enhancement of appearance behavior information from RGB images (such as students raising hands or writing posture), spatial relationship information from depth images (such as relative distance between students and podium/peers), and physiological engagement information from thermal infrared images (such as attention concentration implied by facial temperature changes). The module architecture is shown in Figure 2. The module first performs input calibration on the same-level features {D^F_u, D^N_u, D^S_u} extracted by three parallel encoders. Specifically, each modality feature is independently input into a sequence consisting of a Squeeze-and-Excitation block and a Convolution–Batch Normalization–ReLU block. The SE block generates channel weights through global average pooling and fully connected layers to achieve dynamic calibration. For example, in analyzing a group discussion scene, the weight of the depth modality may be increased; while in recognizing individual reading behavior, the importance of the RGB modality may be enhanced. The calibrated features {D^SD_N, D^SD_F, D^SD_S}are concatenated along the channel dimension to form the preliminary fused feature D^SD_CAT, laying the foundation for subsequent deep mixing. Specifically, the operation of the SE module is denoted as d_TR(·), the convolution layer as ZYE(·), and the concatenation operation as [ , , ], then:

$\left\{\begin{array}{l}D_N^{S D}=Z Y E\left(d_{T R}\left(D_u^N\right)\right) \\ D_F^{S D}=Z Y E\left(d_{T R}\left(D_u^F\right)\right) \\ D_S^{S D}=Z Y E\left(d_{T R}\left(D_u^S\right)\right) \\ D_{C A T}^{S D}=\left[D_N^{S D}, D_F^{S D}, D_S^{S D}\right]\end{array}\right.$              (1)

1.png

Figure 1. Overall architecture of the multimodal image feature integration model for educational management

2.png

3.png

After completing the first round of spatial reweighting, the module adds the two weighted types of features and inputs them into the second spatial attention block CA2. At this stage, channel max pooling operation is adopted to further refine the strongest response across channels at each spatial location. This strategy helps to emphasize the most discriminative behavioral cues at each location. For example, a certain location may appear as a hand-raising posture in the RGB channel and as a slight temperature increase in the thermal infrared channel; max pooling can strengthen such cross-modal consistent key evidence. The final enhanced feature D^VMR_u not only fuses complementary information from adjacent levels but also achieves precise focus on key behavioral regions through a two-stage spatial attention mechanism. This enhanced feature significantly improves the quality of decoder reconstructed feature maps, enabling the final generated visualization saliency map to clearly and reliably highlight patterns of great value to educational management, such as “actively interacting student groups” and “individuals with dispersed attention,” directly serving core research goals such as classroom engagement evaluation and teaching strategy optimization. Specifically, assuming the channel max pooling operation is denoted as φ(·), the process can be described as follows:

$\left\{\begin{array}{l}D_{S U M}^{V F}=\left(D_{S U M}^{V M R}(\mid) F_{u+1}\right) \otimes T Q \\ D_u^{V M R}=\phi\left(D_{S U M}^{V F}\right) \otimes D_{S U M}^{V F}\end{array}\right.$                 (4)

2.3 Multi-level cascaded feature integration module

In multimodal learning behavior analysis, it is crucial to effectively distinguish “foreground” regions representing key learning behaviors such as teacher–student interaction and group collaboration from irrelevant classroom environment “background.” For this purpose, we design a multi-level cascaded feature integration module, aiming to intelligently aggregate features from different decoder levels in a “from global to detail” manner. The core inputs of this module are two key elements: first, the joint feature D^VMR_1,4 formed by concatenating the deepest semantic feature D^VMR₄ (encoding global patterns of “what behavior”) and the shallowest detail feature D^VMR₁ (encoding subtle actions of “how behavior”); second, the multi-resolution features {F₅,F₄,F₃,F₂} generated during the decoding process. As a powerful guiding signal, the joint feature D^VMR_1,4 itself has already integrated broad-spectrum information from behavioral semantics to spatial details and will be used to guide the weighting and fusion process of all subsequent decoder features, ensuring that the final generated behavior saliency map can both accurately locate behavior subjects and clearly outline behavioral details.

This module progressively integrates features through four cascaded convolution blocks Conv-Bi. Each Conv-Bi block contains a Convolution–Batch Normalization–ReLU block, a Convolution–Batch Normalization block, a Sigmoid activation function, and a convolution layer, jointly forming a precise feature selection and enhancement unit. Assuming Conv-Bu is denoted as φ(·), the specific process can be defined as follows:

$\left\{\begin{array}{l}D_{u+1}^{L F}=\left[U P_{\times 2}\left(C O\left(F_{u+1}\right)\right), D_{u+1}^L\right] \\ D_u^L=C O_{-} B_u\left(D_{u+1}^{L F}, D_{u+1}^L\right)\end{array}, u=1,2,3,4\right.$                 (5)

Taking Conv-B₄, which processes the highest semantic level feature, as an example: first, the decoder feature F₅ is convolved and upsampled to compress its channel number to 1. This operation aims to extract the most essential behavioral semantic information of this level while reducing computational complexity. Then, the compressed feature is concatenated with the guiding feature D^VMR_1,4 to form the fusion input D^LF₄. Inside Conv-B₄, FMD5 undergoes feature transformation successively through the CBR block and the CB block, then a spatial weight map is generated through the Sigmoid function. This weight map clearly identifies regions considered highly related to learning behaviors under the joint guidance of high-level semantics and low-level details. Finally, by multiplication of the weight map with the guiding feature D^VMR_1,4 and residual connection, the enhanced feature D^L₄ is generated, which has deeply fused the global context of F₅. The entire process can be expressed as:

$D_4^L=C O\left(S I G\left(Z Y\left(Z Y E\left(D_5^{L F}\right)\right)\right) \otimes D_5^L \oplus D_5^L\right)$                   (6)

This process is repeated step by step downward, such as Conv-B₃ processing D^L₄ and F₄, and so on. Like an information filter, under the guidance of D^VMR_1,4, each step reinforces behavior-related multi-scale cues and suppresses background noise. Ultimately, a highly pure and progressively restored resolution feature map is output, laying a solid foundation for generating high-quality saliency maps that can accurately highlight key learning behaviors such as “raising hand,” “turning,” and “gathering.”

2.4 Loss function

To ensure that each level in the decoder generates saliency representations related to learning behaviors and to alleviate the gradient vanishing problem for faster convergence, the model introduces a deep supervision mechanism. This mechanism connects a convolutional prediction head to the four intermediate decoder blocks, except for the final output layer, to obtain multiple intermediate predictions at different resolutions. The losses between these intermediate predictions and the ground truth are calculated separately, with the core purpose of providing explicit gradient guidance for different depths of the network: deep supervision forces the network to focus on the overall semantic structure of behavior patterns at early stages, such as identifying regions where “group discussion” occurs, while shallow supervision constrains the network to recover clear spatial boundaries and details of behavior subjects, such as accurately delineating the contours of individuals participating in the discussion. This coarse-to-fine multi-scale supervision strategy effectively ensures that the model optimizes each step of learning behavior feature extraction and reconstruction in the correct direction, enhancing the model’s ability to capture multi-scale behaviors from global interactions to individual micro-actions in complex classroom scenes and improving training stability.

To comprehensively optimize the quality of saliency maps, making them not only pixel-accurate but also structurally complete, thereby providing reliable analysis for educational managers, we design a joint loss function. This function consists of three components: binary cross-entropy (BCE) loss, IOU loss, and SSIM loss, which constrain the prediction results from different dimensions. BCE loss is responsible for pixel-level classification, minimizing the error between predicted values and ground truth at each pixel, ensuring that the model can clearly distinguish foreground pixels representing key learning behaviors from background pixels, which is the basis for accurate localization of behavior regions. IOU loss optimizes at the region level, directly maximizing the overlap between predicted behavior regions and ground truth regions. Its sensitivity to region shape helps the model detect the entire behavior subject completely rather than in fragmented pieces, which is crucial for accurately quantifying the scope of behavior occurrence. SSIM loss focuses on evaluating structural similarity differences between the predicted map and ground truth map. It penalizes outputs that may have acceptable pixel accuracy but distorted structures, ensuring that the generated behavior saliency maps have good visual fidelity and structural authenticity. This greatly improves the reliability and interpretability of subsequent behavior pattern visualization and manual judgment based on saliency maps. Specifically, assuming BCE loss, IOU loss, and SSIM loss are denoted as loss^Y_u, loss^Y_u, and loss^T_u, respectively, the model loss function is calculated as follows:

$\left\{\begin{array}{l}{LOSS}=\sum_{u=1}^5\left({loss}_u\right) \\ {loss}_u={loss}_u^Y+{loss}_u^U+{loss}_u^T\end{array}\right.$                   (7)

Assuming the ground truth map is denoted by HS, the predicted saliency map by T, and the coordinates of each pixel in the image by (a, b), the BCE loss in the above equation is expressed as:

${loss}_u^Y=-\sum_{(a, b)}[H S(a, b) \log (T(a, b))+(1-H S(a, b)) \log (1-T(a, b))]$               (8)

The IOU loss is expressed as:

$l o s s_u^U=1-\frac{\sum_{(a, b)} T(a, b) H S(a, b)}{\sum_{(a, b)}[T(a, b)+H S(a, b)-T(a, b)+H S(a, b)]}$                 (9)

Let the image patches cropped from the saliency map T and ground truth map HS be O_T=O^k_T and O_H=O^k_H, respectively. The mean and standard deviation of O_T and O_H are ω_OT, ω_OH and δ_OT, δ_OH, and their covariance is δ_OTOH. The SSIM loss is defined as:

${loss}_u^T=1-\frac{\left(2 \omega_{O_T} \omega_{O_H}+Z_1\right)\left(2 \delta \omega_{O_T O_H}+Z_2\right)}{\left(\omega_{O_T}^2 \omega_{O_H}^2+Z_1\right)\left(\delta_{O_T}^2 \delta_{O_H}^2+Z_2\right)}$                   (10)

The collaboration of these three losses jointly drives the model to generate learning behavior saliency maps with high precision, high completeness, and high structural fidelity, providing a solid data foundation for educational managers to perform quantitative analysis and make decisions.

Using only general saliency metrics is insufficient to demonstrate the model's practical value in the educational field. Behavior analysis specific metrics must be combined to comprehensively evaluate its performance on real tasks. From the average of the ten experimental groups in Table 2, the model performs excellently on general saliency metrics: S-Measure is 0.9385, MAE is 0.0023, and maxF is 0.9210, ensuring that the generated behavior heatmaps are visually accurate and clear. More importantly, on behavior analysis specific metrics, the model achieves behavior localization accuracy of 0.853, behavior classification accuracy of 0.782, and interaction event detection rate of 0.911. This indicates that the model can not only highlight salient regions at the pixel level but also precisely associate these regions with specific learning behaviors and effectively capture complex interactive events such as “teacher-student Q&A.” The high performance of both specific and general metrics demonstrates that the model successfully establishes strong associations between low-level visual features and high-level behavior semantics. It can be concluded that the proposed model retains excellent pixel-level saliency detection capabilities while showing high effectiveness and practical value in core educational management tasks, namely learning behavior localization, recognition, and interaction analysis, significantly surpassing performance measured solely by general metrics.

Table 3. Statistical significance test (p-values) of the proposed model and baseline methods on educational behavior analysis dataset

Furthermore, this paper statistically confirms that the performance advantage of the proposed model in educational behavior analysis tasks is not accidental, but represents a substantial improvement with statistical significance. Analyzing the data in the Table 3, compared with the general video behavior recognition model I3D, the proposed model achieves extremely low p-values on all general and specific metrics (all far below 0.01), indicating that the model's performance is comprehensive and significantly superior to models relying only on RGB information. Second, in comparison with the efficient skeleton-based model PoseC3D, although the differences in general metrics such as S-Measure are not statistically significant (p>0.05), reflecting PoseC3D’s capability in extracting abstract spatiotemporal features, the proposed model exhibits overwhelming advantages in the three core educational analysis metrics: behavior localization accuracy (p=4.4e^-6), classification accuracy (p=9.8e^-5), and interaction event detection rate (p=3.2e^-7). This strongly demonstrates that generating behavior heatmaps directly via multi-modal fusion far exceeds skeleton-based simplified representations in understanding and parsing complex classroom interaction semantics. Finally, compared with the same-type multi-modal model HWSI, our model also shows statistically significant improvements across all metrics.

To quantitatively evaluate the contribution of each core component of the proposed model to the final educational behavior analysis performance, ablation experiments were conducted. Analyzing the data in Table 4, removing the tri-modal feature fusion module leads to the most obvious decline in all metrics, especially behavior localization accuracy and interaction event detection rate, proving that deep fusion of RGB, depth, and thermal infrared information is the cornerstone of accurately understanding learning behaviors. Second, removing deep supervision training causes a comprehensive and significant performance degradation, highlighting the critical role of this strategy in guiding the network to learn multi-scale behavior features. Moreover, although removing adjacent layer feature enhancement or multi-level cascade feature integration has relatively minor impacts on general metrics, their negative effects on behavior-specific metrics are clearly visible, indicating that these modules mainly optimize semantic understanding and spatial precision of behaviors. It can be concluded that each core component of the proposed model contributes positively and irreplaceably to improving educational behavior analysis performance, with tri-modal feature fusion and deep supervision training being key guarantees of overall performance, fully validating the design of the model as an organic whole.

Table 4. Ablation study of core components of the proposed model on educational behavior analysis dataset (Mean performance)

Table 5. Performance comparison of different feature fusion strategies on educational behavior analysis task

Fusion strategy is the core determinant of multi-modal model performance. To demonstrate the superiority of the proposed tri-modal feature fusion module in capturing complex correlations of educational behaviors, feature fusion strategy comparison experiments were conducted. The data in the Table 5 clearly shows a stepwise improvement in all evaluation metrics from simple “feature addition” to “attention-based fusion,” and further to the proposed tri-modal feature fusion module. In particular, the tri-modal feature fusion module shows the most obvious advantages in behavior classification accuracy and interaction event detection rate, indicating that its cross-modal attention and multi-scale mixing mechanism can more effectively mine complementary information between modalities, thereby enabling more accurate judgment of complex behavior states such as “distracted” or “focused.” It can be concluded that the proposed feature fusion strategy significantly outperforms traditional fusion methods in decoding learning behavior semantics and is a key innovation for improving the overall system’s behavior analysis accuracy.

Table 6. Effects of different modal combinations on educational behavior analysis performance

To further verify that RGB, depth, and thermal infrared tri-modal information are indispensable for educational behavior analysis tasks, experiments on the effects of modal combination inputs on educational behavior analysis performance were conducted. The analysis data in Table 6 shows that when the RGB modality is missing, the model cannot obtain rich appearance texture and semantic information, causing the behavior classification accuracy to drop sharply to 0.655, indicating that appearance information is the basis for behavior recognition. Second, when the thermal infrared modality is missing, the model struggles to capture physiological thermal signals related to attention and emotion, leading to the worst performance in interaction event detection rate, highlighting the importance of physiological information for understanding classroom interaction quality. It is noteworthy that the R-T combination is already close to the complete model in various metrics, especially performing well on general metrics, indicating that the combination of appearance and physiological signals can largely infer behaviors. However, the complete model maintains leading performance on all behavior-specific metrics, especially the further improvement of interaction event detection rate sensitive to spatial relationships, proving that depth information provides precise spatial relations necessary to clarify complex interactions. It can be concluded that in educational behavior analysis, RGB, depth, and thermal modalities respectively provide irreplaceable appearance, spatial, and physiological cues, and together form a comprehensive and robust basis for behavior understanding. The absence of any modality leads to a significant decline in analysis performance, thereby validating the necessity of the tri-modal design adopted in this paper.