Adaptive Enhancement Strategy for Multimodal Image Fusion in Behavior Monitoring for Remote Education Environments

In remote education scenarios, the acquisition of learners’ behavior data faces significant challenges of environmental heterogeneity and modality diversity. From the spatial dimension, students may study in different scenarios such as at home, in study rooms, or outdoors, where the lighting conditions, background complexity, and device deployment methods vary greatly, resulting in single-modal images being unable to stably capture key behavioral features. For example, under low light, RGB images may suffer from noise and blurring, and in complex backgrounds, the edge information of human postures may be obscured. Although the introduction of multimodal images can compensate for the shortcomings of single modality, the resolution differences, spatiotemporal alignment deviations, and semantic information complementarity among different modality data often lead to feature conflicts or information redundancy when directly fused, making it difficult to form a complete characterization of learning behavior. In addition, the large-scale application of remote education results in a scarcity of labeled data, and students’ behavior patterns dynamically change with the learning stage. Therefore, there is an urgent need for a multimodal image fusion and data enhancement strategy that can be dynamically optimized according to real-time scenarios to generate high-quality training data and inference inputs that meet the requirements of behavior monitoring.

The adaptive enhancement algorithm of multimodal image fusion data for remote education behavior monitoring proposed in this paper achieves adaptive enhancement starting from the efficient extraction and deep fusion of multimodal features. Figure 1 shows the proposed algorithm architecture. Aiming at the heterogeneous characteristics of multimodal images such as RGB, depth, and infrared in remote education scenarios, the designed multimodal feature online generation module dynamically captures key behavioral cues under different modalities through hierarchical convolution and attention mechanisms. For example, the color texture features of facial micro-expressions and hand gestures in RGB images, the spatial positional relationships of limbs in depth images, and the body temperature distribution and motion thermal imaging trajectories in infrared images. These multi-dimensional features are mapped into the latent variable space of the diffusion model, which can break the semantic barriers between modalities and generate synthetic data that conforms to the distribution of remote education scenarios. On this basis, the enhanced gate-controlled self-attention module adaptively allocates weights to selectively fuse learning behavior-related features and suppress complex background noise, thereby retaining high-discriminative behavior feature attributes. This generative modeling not only solves the defects of the original multimodal data in terms of spatiotemporal alignment and resolution differences, but also generates high-quality fused images with realistic scenario distribution through the noise elimination of the diffusion process, providing richer training materials for behavior monitoring models.

1.png

Figure 1. Algorithm architecture of adaptive enhancement of multimodal image fusion data for remote education behavior monitoring

2.png

Figure 2. Diagram of adaptive enhancement process of multimodal image data

Based on the generated multimodal fused images, the data adaptive enhancement method is guided by the core requirements of remote education behavior monitoring and constructs a closed-loop mechanism of “feature retention - label alignment - strategy dynamic adjustment”. Firstly, semantic labeling is performed on the synthetic images through pseudo-label generation technology to ensure that the target behavioral features in the new sample images are strictly aligned with the label space of the monitoring task, avoiding annotation bias caused by feature distortion in traditional data augmentation. Secondly, according to the monitoring focus of different teaching stages, an adaptive enhancement strategy library is designed to dynamically select operations such as illumination adjustment, scale transformation, and local feature enhancement. For example, in infrared-RGB fused images generated under low light conditions, the contrast of hand operation areas is specifically enhanced, and background clutter interference is weakened; in depth images of multi-person learning scenarios, the spatial coordinate information of human skeleton joints is highlighted. This task-driven enhancement method not only expands the diversity of data samples but also avoids the destruction of key behavior features by general enhancement methods, so that the enhanced data can accurately match the input requirements of the behavior monitoring model, ultimately improving the detection accuracy and generalization ability of the model in complex remote education environments. Figure 2 shows the diagram of the adaptive enhancement process of multimodal image data.

2.1 Algorithm framework

The adaptive enhancement algorithm proposed in this paper is based on the diffusion model and constructs a three-level processing architecture of “feature extraction - cross-modal fusion - scene generation”, specifically adapted to the complex characteristics of remote education multimodal images. Firstly, the modality feature online generation module uses lightweight convolutional networks and attention mechanisms to automatically parse multi-dimensional inputs such as color textures of RGB images, spatial coordinates of depth images, and thermal radiation distributions of infrared images, without manually preset feature extraction rules, significantly reducing the algorithm’s dependence on specific devices or scenarios. While inheriting the prior knowledge of the stable diffusion model, the backbone network weights are frozen to retain its strong image generation capability. At the same time, a customized feature fusion module for remote education scenarios is improved: in the feature preprocessing stage, a text encoder is used to convert the description of the teaching scenario into a semantic feature sequence, which is input into the U-shaped network together with image modality features. The multi-level attention feature fusion units and enhanced gate-controlled self-attention modules designed inside the network can dynamically capture the behavior-associated features between different modalities. For example, aligning the hand motion area in RGB images with the three-dimensional coordinates in depth images enhances the feature expression of key behaviors such as “writing” and “clicking on the screen”. Finally, the high-dimensional fused features are sampled through a variational encoder to generate new images that retain the behavioral attributes of the original scenario and conform to real-world distribution, ensuring that the generated data is deeply consistent with remote education scenarios in terms of semantics and geometric structure.

Based on the generation of high-quality multimodal fused images, the algorithm constructs an adaptive data augmentation process of “generation - labeling - diversified enhancement” directly serving the behavior monitoring task. Firstly, the behavior labels of the original samples are directly mapped to the newly generated images through pseudo-label mapping technology. By utilizing the object consistency between the generated image and the original image, the precise alignment of labels and target features is ensured, avoiding the label noise caused by semantic distortion in traditional data augmentation. Secondly, by introducing scene prompts and random parameter control, each generated image produces reasonable variations in non-critical features such as target appearance and environmental conditions while retaining the core attributes required for behavior monitoring. This task-oriented enhancement strategy not only expands the diversity of data samples but also avoids the damage to behavior-discriminative features caused by general enhancement methods through a feature retention mechanism. Finally, the generated multimodal images and pseudo-labeled data together constitute an enhanced dataset, providing richer and more representative training samples for behavior monitoring models, effectively improving the detection accuracy and robustness of the model in actual remote education environments.

2.2 Online generation module for multimodal features

The online generation module for multimodal features realizes the automatic parsing and structured expression of multimodal features in remote education scenarios through three parallel channels, primarily addressing the limitations of traditional methods that rely on manual feature engineering. Figure 3 shows the architecture of the online generation module for multimodal features. Channel 1 performs pixel-level semantic modeling on RGB images based on semantic segmentation technology, subdividing entities in the remote education scene into 150 category labels and generating semantic masks containing target categories and positional information. This fine-grained semantic description can not only accurately locate the key carriers of learning behaviors but also provide explicit category constraints for the subsequent generation process, avoiding irrelevant background interference in behavior monitoring. Channel 2 introduces a monocular depth estimation algorithm to extract the depth value of each pixel from the 2D image and diffuse it to each channel, constructing a depth feature map that implies 3D spatial relationships. Combined with the semantic mask, this feature can accurately depict behavioral-related geometric attributes such as the spatial angle of the student’s sitting posture and depth changes in the operation trajectory, compensating for the lack of spatial information in the single RGB modality and providing cross-dimensional positional association basis for multimodal fusion.

3.png

Figure 3. Architecture of the online generation module for multimodal features

Channel 3 adopts the BLIP algorithm to achieve vision-language cross-modal alignment, converting remote education scene images into natural language descriptions and generating textual feature sequences containing environmental conditions, target behaviors, and interaction objects. This design breaks through the limitation of traditional generation models relying on fixed textual labels, enabling the algorithm to flexibly adjust the generation direction according to dynamic scene requirements. The output features of the three major channels form a complement in the subsequent processing: semantic segmentation provides category information of "what the target is", depth estimation clarifies "where the target is" in spatial coordinates, and textual descriptions supplement "what is done in which scene" in contextual semantics. These three jointly form a multimodal feature vector, which is input into the U-Net of the diffusion model for cross-modal fusion. Specifically, assuming that the descriptors of scene target quantity, target color, target category, and actions are represented by Q_l, Q_x, Q_zx, and Q_n respectively, the basic structure of the text input corresponding to each original remote education scene is:

$Q_v+Q_x+Q_{z x}+Q_n$           (1)

This multi-dimensional feature decoupling mechanism not only reduces the algorithm’s strict dependence on the preprocessed data format but also retains the core elements required for behavior monitoring in remote education through automated feature generation. Target category, spatial position, and contextual semantics lay the feature foundation for subsequently generating high-quality fused images and adapting to behavior monitoring tasks.

2.3 Feature preprocessing

The core of feature preprocessing is to convert heterogeneous information such as textual semantics, spatial depth, and target categories in remote education scenarios into a unified feature space capable of efficient interaction. The key principle lies in modality decoupling and structured alignment. Figure 4 shows the architecture of the feature preprocessing module. For text input, the algorithm inherits the OpenCLIP module of the stable diffusion model and maps the scene description text into a high-dimensional feature sequence g^z= [g^z₁,g^z₂ ,…,g^z_v] through contrastive learning. This sequence not only retains entity concepts such as "laboratory" and "microscope" but also captures the contextual association of behavior verbs like "operation", providing semantic guidance for the generation process. For semantic features t and depth features f, a spatial sampler composed of multiple layers of 4×4 convolution is used for downsampling, converting different modal images into feature noise maps adapted to the original image size, solving the alignment difficulty caused by resolution differences in multimodal data. Through such structured processing, the algorithm transforms the original multimodal input into a set of target-feature pairs h =[(t₁, ,f₁), (t₂,f₂), …, (t_v , f_v)], where each element corresponds to an independent target in the remote education scenario, such as students, teaching aids, or interactive behavior regions, clearly labeling their category semantics and spatial location, and providing fine-grained feature anchors for subsequent cross-modal fusion. Assuming that the downsampling of semantic and depth inputs is represented by d_t(·) and d_f(·), and the channel-wise concatenation of inputs is represented by O_CAT(·,·), the original image input is u, random noise is v_e, the encoding of these using an autoencoder is represented by D_XR(·), the sampling method of the diffusion model is T(·), and the resulting image noise is g^u, then:

$g^h=O_{C A T}\left(d_t(t), d_f(f)\right)$         (2)

$g^u=T\left(D_{X R}(u)+v_e\right)$        (3)

4.png

Figure 4. Architecture of the feature preprocessing module

After completing modal alignment, the algorithm uses a lightweight ConvNeXt-T network module to perform deep semantic mining of multimodal features, avoiding the computational redundancy of traditional heavy networks and adapting to the possible lightweight deployment needs in remote education. This module extracts local details and global structures of the target layer by layer through hierarchical convolution operations, especially enhancing the feature response of key regions for behavior monitoring. At the same time, a linear module is introduced to embed positional encoding information, converting pixel coordinates in 2D images into learnable positional vectors to ensure that the generation model retains strict spatial positional associations when processing multimodal features. For example, binding the semantic label of the “mouse click” action with the depth coordinates of the screen area to avoid target position shift or semantic misalignment during the generation process. Finally, the text feature sequence g^z and the modality-aligned features h jointly form the network input, forming a three-level preprocessing system of "scene semantic guidance – target feature alignment – spatial position constraint". Assuming that the aligned input is represented by a, the ConvNeXt-T network module and linear module are represented by V_ZV(·) and V_M respectively, and the preset positional encoding is o, the high-dimensional feature sequence after preprocessing such as size adjustment is g^r, then:

$O(a)=V_M\left(V_{Z V}(a)+o\right)$          (4)

$g^r=\left[O\left(h_t\right), O\left(h_r\right)\right]$               (5)      

The above operations not only solve the interaction inefficiency caused by multimodal data heterogeneity but also provide structured and high-quality input from the feature level by retaining the core features required for behavior monitoring, supporting the subsequent diffusion model to generate high-fidelity fused images and perform adaptive data enhancement, and ensuring the understanding accuracy of behavior monitoring models for complex remote education scenarios.

2.4 Feature fusion

Figure 5 shows the architecture of the feature fusion network and image generation module. The enhanced gated self-attention module constructs a three-level feature fusion system of "semantic–depth–spatial" within the U-Net, dynamically adjusting the interaction weights of multimodal information through learnable parameters, ensuring the consistency between scene structure and behavioral semantics in remote education scenarios. The module architecture is shown in Figure 6. The module first introduces the learnable parameters: modality fusion coefficient η initialized to 0 and conditional information weight β, which respectively control the fusion strength between latent features and conditional inputs such as textual semantics and depth coordinates. The first layer of the gated self-attention module performs cross-modal alignment between the category labels generated by semantic segmentation and the scene semantics from text descriptions through a cross-attention mechanism, extracting latent features rich in behavioral semantics and enhancing the model’s semantic understanding of "what the target is" and "what is being done". The second layer of the gated self-attention module introduces spatial coordinate information obtained from monocular depth estimation and performs layer-by-layer fusion of depth features with visual semantic features through residual connections, generating fused features 2 with precise positional constraints. For example, the semantic label of the “mouse click” action is bound with the depth coordinates of the screen region, ensuring that the spatial relationship between the hand position and the operation interface in the generated image conforms to real physical logic. Assuming that the visual feature representation of the scene is denoted by n=[n₁, n₂,…,n_V], the self-attention network and cross-attention network are denoted by V_TX and V_ZX, and the visual feature selector is denoted by S_t(·). The expression of the gated self-attention module is given by:

$n=V_{Z X}\left(n+S_t\left(V_{T X}\left(\left\lfloor n, g_t^r\right\rfloor\right)\right), z\right)$               (6)

5.png

Figure 5. Architecture of the feature fusion network and image generation module

The gated self-attention module is given by the following expression:

$n=n+\beta \cdot \tanh (\eta) \cdot V_{Z X}\left(S_t\left(V_{T X}\left(\left\lfloor n, g_t^r\right\rfloor\right)\right), z\right)$           (7)

6.png

Figure 6. Architecture of the enhanced gated attention module

The above hierarchical fusion mechanism avoids the over-reliance on a single modality in traditional attention models, enabling the network to adaptively balance semantic integrity and spatial structural stability, providing a foundational support for generating high-quality multimodal images that retain key behavioral features.

Compared with traditional diffusion models, the enhanced gated self-attention module constructs a finer feature control loop by introducing additional learnable parameters ϕ and residual structures. During feature transmission, residual connections not only retain the low-level features of the original modality but also progressively refine the spatial correspondence between semantic and depth features through multiple cross-attention computations. For example, in handling collaborative learning scenes involving multiple individuals, the module can accurately align the semantic labels of different students’ limb movements with their respective 3D coordinates, avoiding target position misalignment or action semantic conflicts in the generated image. Through optimizing the parameter ϕ via the objective function, the algorithm can dynamically adjust attention weights, ensuring that the fused feature vector contains both rich behavior-discriminative information and maintains overall spatial consistency in the remote education scenario. Assuming that the parameters obtained through uniform sampling in the sampling time set are denoted by s, and the VAE is denoted by d_(ϕ,ϕ’), then the objective function expression for the newly introduced learnable parameter ϕ is:

$\frac{M I N}{\phi^{\prime}} \angle=(C, \gamma) \sim V(0, U)_s\left[\left\|\gamma-d_{\left(\phi, \phi^{\prime}\right)}(c, s, U)\right\|_2^2\right]$           (8)

Through the above controllable feature fusion strategy, the issue of feature conflict caused by differences in viewpoints and mismatched resolutions in multimodal images is effectively resolved. This lays a foundation for the subsequent variational encoder to generate new images with real behavioral attributes, ultimately ensuring that the data-augmented samples can accurately reflect complex behavioral patterns in remote education and improve the detection accuracy of behavior monitoring models in tasks such as posture estimation and action recognition.

This paper addressed the core problems of “quality defects” and “insufficient diversity” in multimodal image data for remote education behavior monitoring, and proposes a multimodal image generation algorithm based on diffusion model and a data adaptive enhancement method. It constructs a technical loop of "feature decoupling–deep fusion–scenario adaptation". By designing a multimodal feature online generation module and an enhanced gated self-attention module, layered fusion of RGB, depth, infrared, and scene semantics is achieved. The generated multimodal images significantly outperform traditional methods in pixel fidelity, structural consistency, and semantic completeness. The data adaptive enhancement strategy aligns pseudo-labels and dynamically adjusts parameters, enabling the behavior monitoring model to break through performance bottlenecks in tasks like target detection and pose estimation under complex scenarios.

This research breaks through the limitation of traditional multimodal fusion relying on handcrafted features or a single deep learning architecture, combining the generation capability of diffusion models with the cross-modal attention mechanism of ECSA module, realizing full-process automation from “feature extraction–fusion–enhancement,” and providing a data-driven intelligent solution for remote education behavior monitoring.

By constructing gradient datasets for three core scenarios of remote education, the algorithm's robustness under challenging conditions such as sudden lighting change, occlusion, and sparse labels is verified. The generated data cover extreme cases not included in original scenes, filling the data gap in practical applications. By improving behavior monitoring accuracy, it provides real-time basis for personalized teaching intervention and promotes the transformation of remote education from “extensive content delivery” to “precision behavior empowerment.” The related technology can be transferred to fields such as remote medical monitoring and industrial remote operation and maintenance, showing cross-domain application potential.

There are still two areas for improvement in the current research: (1) The iterative sampling process of the diffusion model and the multi-layer attention mechanism of the ECSA module led to long training and inference times, and lack real-time performance on mobile or large-scale concurrent scenarios. Future work can explore lightweight diffusion models or introduce model distillation techniques to improve efficiency while maintaining generation quality. (2) The research mainly focuses on the fusion of visual modality and text semantics, and has not fully utilized multidimensional data such as audio and physiological signals. The modeling capability of "high-level behavior semantics" is still limited. Future work can expand input modality types, combine graph neural networks or temporal modeling to capture time dependencies and contextual associations of behaviors, and construct a full-dimensional multimodal behavior monitoring system.

In summary, this paper provides an innovative technical path for processing multimodal data in remote education, with its core value lying in improving behavior monitoring accuracy through data enhancement, thereby feeding back into teaching decision-making. Future research should continue to explore efficiency optimization and modality expansion, promoting the transition of technology from “laboratory validation” to “large-scale educational scenario deployment.” From the perspective of educational theory, this study enhanced behavior monitoring accuracy through data augmentation techniques, fundamentally aiming to provide more precise “learning behavior profiles” to support personalized instruction. According to constructivist learning theory, student interaction behaviors in remote education serve as external manifestations of their cognitive construction processes. The integration of data-driven precise monitoring with educational theory offers empirical support for building a closed-loop instructional system encompassing data collection – behavior analysis – strategy adaptation. Ultimately, this enables a paradigm shift from experience-driven teaching to data-intelligent teaching.