Analyzing and Optimizing Virtual Reality Classroom Scenarios: A Deep Learning Approach

In VR classroom environments, scenes often contain a variety of complex visual elements, such as diverse teaching materials, simulated objects, and learner interactions. These elements can easily lead to information confusion and loss of details in traditional segmentation methods. Particularly under conditions of changing lighting, perspective shifts, and instances of blurring or occlusion in the scene, effective information extraction becomes notably challenging. In response to these practical issues in the analysis and optimization of VR classroom scenes, this paper introduces an attention method based on multi-pooling compression excitation. The multi-pooling structure enhances the network's ability to capture features of varying scales. The compression excitation mechanism, through weighted allocation, intensifies focus on pertinent features while suppressing irrelevant information, thus improving the precision in recognizing educational content and learner interactions during the segmentation task. Moreover, this method demonstrates superiority in removing or reducing visual noise that may stem from VR technology itself, ensuring that the model maintains high segmentation performance under less than ideal visual conditions. This implies that the method proposed in this paper maintains stable segmentation effects, whether in dimly lit scenes or against complex backgrounds, providing more accurate and robust scene analysis capabilities for VR classrooms.

Figure 1 illustrates the structure of the multi-pooling compression excitation module. The core design of the proposed module lies in the parallel use of global max pooling and global average pooling layers, which capture different types of global information. The global max pooling layer highlights the most significant features, namely the salient signals in the scene, while the global average pooling layer provides statistical information of the feature mappings, reflecting the overall distribution. These two types of information are then dimensionally reduced through their respective fully connected layers, compressing the features and reducing the number of parameters, thereby also enhancing the model's generalization ability. Subsequently, the ReLU activation layer is employed to introduce non-linearity, enhancing the network's capability to express and learn more complex non-linear relationships between features. Following this, the features output from the aforementioned two paths are further encoded through individual FC layers, making the features more compact while retaining necessary information. Then, these two features are fused through an addition operation, effectively combining the global information extracted by the max and average values, offering a more comprehensive perspective of the features. Finally, the fused features are activated through a Sigmoid activation function, yielding weights between 0 and 1, which represent the importance of each channel. These weights, when multiplied channel-wise with the input features, enable recalibration of the original features, thereby emphasizing beneficial information and suppressing insignificant signals.

1.png

Figure 1. Structure of the multi-pooling compression excitation module

Initially, the input feature map is processed through the global max pooling and global average pooling layers, capturing the salient and overall statistical information of the scene's features, respectively. Subsequently, these global pieces of information are compressed through their respective fully connected layers, aiming to reduce dimensions, thereby decreasing the model's parameters and computational load, and preparing compact feature representations for the subsequent excitation step. Assuming the compressed features are represented by D_MS and D_AS, the number of channels in feature D is denoted by Z, and the height and width of the features are represented by G and Q, the two-dimensional input feature in the z-th channel, and the compressed features post-global max pooling and global average pooling are denoted by D^z, D^z_MS, and D^z_AS, respectively. The computational formulas are as follows:

$D_{M S}^z=\underset{u \in G, k \in Q}{M A X}\left(D^z(u, k)\right)$                        (1)

$D_{A S}^z=\frac{1}{G \times Q} \sum_{u=0}^G \sum_{k=0}^Q D^z(u, k)$                        (2)

The compressed features D^z_MS and D^z_AS are passed through a ReLU activation layer to introduce non-linearity, enabling the model to capture more complex feature relationships. Subsequently, these features are further encoded through a fully connected layer, generating an independent weight value for each feature channel. Suppose the weight matrices for DZ₁ and DZ₂ are represented by Q₁ $\in$ R^Z*Z/e and Q₂ $\in$ R^Z*Z/e, respectively. The ReLU activation function is denoted by σ, and the process is expressed as follows:

$D_{M E X}^{\prime}=Q_2 \sigma\left(Q_1 D_{M S}\right)$                     (3)

$D_{A E X}^{\prime}=Q_2 \sigma\left(Q_1 L D_{A S}\right)$                     (4)

The two sets of channel weights obtained in the previous step, D'_MEX and D'_AEX, are added together to yield the aggregated feature D'_EX, integrating the global information captured by both maximization and average pooling strategies. The aggregated feature is then processed through a Sigmoid activation function, outputting the final weights for each channel. These weights, ranging between 0 and 1, represent the relative importance of each channel to the task. Assuming the Sigmoid activation function is denoted by σ, the final expression for the excited feature D'_EX is as follows:

$D_{E X}=\delta\left(D_{E X}^{\prime}\right)=\delta\left(D_{M E X}^{\prime}+D_{A E X}^{\prime}\right)$                       (5)

Suppose the input variable value is represented by c, and the final function output value is denoted by δ(c). The specific formula for the Sigmoid activation function is:

$\delta(c)=\frac{1}{1-e^{-c}}$                    (6)

Finally, the weights output by the Sigmoid function are multiplied channel-wise with the original input feature map, achieving feature recalibration. This step enhances focus on useful features while suppressing unimportant or interfering information, thereby maintaining the original feature structure while highlighting the expression of key information in the analysis of VR classroom scenes. Assuming the excited one-dimensional feature is represented by D_EX, and the recalibrated feature is denoted by D^~, which is the product of D_EX and the original feature D in the corresponding channels, the expression is as follows:

$\tilde{D}=D_{E X} \cdot D$                      (7)

Figure 2 displays the overall network structure that incorporates the multi-pooling compression excitation module. This paper provides solutions to a series of practical problems encountered in the analysis and optimization of VR classroom scenes by introducing a feature dehazing branch structure with an "enhance-refine-subtract" enhancement module, along with a feature distillation module incorporating an attention mechanism. Initially, the dehazing branch structure effectively addresses the issue of visual information loss in VR environments caused by simulated fog effects. The "enhance" step improves the contrast and clarity of features, thereby enhancing their recognizability; the "refine" step further precisely adjusts these features, ensuring the preservation of details; the "subtract" step effectively removes the scattering and color degradation effects caused by the fog, restoring the essential characteristics of the scene. Subsequently, the integrated feature distillation module, through its attention mechanism, further focuses and optimizes key features, ensuring effective integration of features in multi-task learning and enhancing the model's ability to parse complex scenes in semantic segmentation tasks. Figure 3 demonstrates the backbone network structure incorporating the feature dehazing branch and the feature distillation module.

2.png

Figure 2. Network structure incorporating multi-pooling compression excitation module

3.png

Figure 3. Backbone network incorporating feature dehazing branch and feature distillation module

The feature dehazing branch structure first upsamples deep features through two stride-2 transposed convolution modules. This process not only increases the spatial resolution of the feature map but also aligns the number of feature channels with corresponding stages in the decoder, laying the foundation for feature fusion. Subsequently, these upsampled features are fused with the jump connection features from the encoder. This step utilizes the rich edge and texture information contained in shallow features to complement deep features, thereby obtaining a more comprehensive feature representation. Further, the fused features are input into the "Enhance-Refine-Subtract" enhancement module, the core function of which is to improve the quality and distinction of features. The output fog-free features are significantly enhanced in quality through structured processing in the "enhance," "refine," and "subtract" components. Assuming the enhanced image after v iterations is represented by K^{^}_v, the image deblurring method by h(.), and the input blurred image by U, the following formula represents the expression of the enhancement module:

$\hat{K}_{v+1}=h\left(U+\hat{K}_v\right)-\hat{K}_v$                      (8)

Assuming the enhanced feature at stage v is represented by k_v, the shallow feature obtained from the encoder at stage v by u_v, the 2x upsampling operation by ↑₂, and the feature refinement operation unit by H(.), the module can also be represented as:

$k_v=H\left(u_v+\left(k_{v+1}\right) \uparrow_2\right)-\left(k_{v+1}\right) \uparrow_2$                         (9)

The feature distillation module with an attention mechanism introduced in this paper is an efficient multi-task learning strategy. It enhances the performance of the final task by synthesizing feature information generated from different tasks. This module's structural feature lies in its ability to operate within a multi-dimensional feature space, concatenating the fog-free features from the feature dehazing branch with the refined segmentation features from the decoder. In this process, the attention mechanism plays a key role by learning the importance distribution of these features, selectively strengthening information beneficial to the final task while suppressing irrelevant or interfering information. In the feature distillation module, the specific role of the attention mechanism is reflected in its weight distribution among different features. These weights are not randomly assigned but learned through the network, reflecting the contribution of each feature to the final task. The attention mechanism dynamically adjusts the information flow between features, enabling the network to focus more on information useful for tasks like analyzing VR classroom scenes.

Suppose the intermediate feature of the target task is represented by D^u₁, the attention feature map by T^u, the intermediate feature of the auxiliary task by D^u₂, and the distilled feature by D^p. Multiplying D^u₂ with T^u at the element level to obtain the weighted feature and adding it to D^u₁ yields D^p. The following formula represents the feature distillation module process expression:

$D^p=D_1^u+T^u \cdot D_2^u=D_1^u+\delta\left(Q D_1^u\right) \cdot D_2^u$                  (10)

From Table 1, it is evident that the ablation study results of the VR classroom scene segmentation model demonstrate the performance comparison under different model configurations. MIoU, an important metric, is used to measure the accuracy of model segmentation; a higher value indicates better segmentation. GFLOPs and Parameters represent the computational complexity and size of the model, respectively. As shown, the baseline model serves as the control group, with an MIoU of 71.2%, a parameter size of 31.254MB, and computational complexity of 95.26GFLOPs. This was the starting point for subsequent improvement. The addition of Multi-Pooling Compression Module 1 to the baseline model resulted in a slight increase in MIoU to 71.5% and a slight increase in parameter number and computation complexity, and this indicates that this module has some positive impact on model performance, but the improvement is limited. After adding the multi-pooling compression excitation module 2, the MIoU instead decreases to 70.5%, and the amount of computation decreases slightly, this means that module 2 is not as good as module 1 in terms of performance improvement, and even introduces additional disturbances. Later, continuing to add the multi-pooling compression excitation module 3, the MIoU improves to 71.7%, which exceeds the performance of the baseline model and the first two modules, while the increase in the number of parameters and the amount of computation is very small, and this indicates that module 3 can enhance model performance while keeping a low computational cost. The introduction of the feature de-hazing branching structure further improves the MIoU to 71.6% with a slight increase in the number of parameters but a decrease in the computational effort, and this indicates that the de-hazing branching structure can enhance the model's ability to handle scene details while having little impact on computational efficiency. Finally, incorporating the Feature Distillation Module significantly raised the MIoU to 73.1%, the highest among all configurations. Although there was an increase in parameters and computational complexity, the magnitude of performance improvement justifies this increase. In summary, the model proposed in this paper effectively improved the accuracy of VR classroom scene segmentation by introducing multi-pooling compression modules and feature dehazing branch structures, and further significantly enhanced segmentation performance through the feature distillation module. These results validate the effectiveness of the methods presented in this paper, particularly the key role of the feature distillation module in enhancing performance. Even with a slight increase in parameters and computational costs, more accurate segmentation results were achieved.

Table 2 shows the segmentation performance comparison of different models in two different VR classroom scenes. These data are represented in the form of MIoU percentage, a commonly used metric for evaluating image segmentation quality. Higher MIoU values typically indicate more precise segmentation results. In Scene 1, HR-Net's MIoU is 28.9%, suggesting poor performance or lack of assessment in Scene 2. FDA performs at 22.4% in Scene 1 and improves in Scene 2, reaching 38.9%, indicating better suitability or effectiveness of the FDA model in Scene 2. For DANN, the MIoU values are 45.3% and 61.2% in the two scenes, respectively, performing better in Scene 2 but not as well as MF-Net and RTF-Net in Scene 1. MF-Net shows good performance in both scenes with MIoU values of 48.7% and 64.5%, being the first model to perform well in both scenes. RTF-Net's MIoU in Scene 1 is 48.9%, slightly higher than MF-Net, but it drops to 58.6% in Scene 2, lower than both DANN and MF-Net. BMFF-Net has a slightly lower performance in Scene 1 at 44.6%, but significantly improves in Scene 2 to 71.2%, surpassing all other models. In Scene 1, this paper's model has an MIoU of 48.7%, the same as MF-Net, while in Scene 2, it leads all other models with an MIoU of 72.1%. It can be concluded that this paper's model has the highest segmentation accuracy in Scene 2 (72.1% MIoU), highlighting its robustness and efficiency in complex scenarios. Although not the absolute leader in Scene 1 (matching MF-Net), considering the significant performance improvement in Scene 2, it can be inferred that this paper's model has good versatility and adaptability.

6.png

Figure 6. PSNR convergence of the baseline model and the proposed model during training

Figure 6 depicts the Peak Signal-to-Noise Ratio (PSNR) values of the baseline model and the proposed model at every 25 epochs during the training process. PSNR is a commonly used quality evaluation metric in image processing, often employed to assess the quality of image restoration or image compression. A higher PSNR value indicates less distortion and better image quality. From the figure, it is evident that at the initial stage, this paper's model starts with a PSNR of 13, while the baseline model is at 11, demonstrating superior performance of the proposed model from the outset. During the mid-phase of training, both the proposed model and the baseline model exhibit a steady increase in PSNR, but the proposed model consistently maintains a lead over the baseline model. For example, at epoch 100, the proposed model reaches 32.2, while the baseline model is at 31.1. In the later stages of training, the PSNR values of both models tend to stabilize. The proposed model reaches 34.2 at epoch 200, compared to 33 for the baseline model. Not only does the proposed model maintain its leading position, but the margin of lead also increases in the later stages of training. Observing the overall trend, the proposed model shows a more stable and continuous upward trend in PSNR growth, while the baseline model experiences slight fluctuations at certain periods, such as a decrease followed by an increase in PSNR between epochs 150 to 175. The following conclusions can be drawn: throughout the training process, the proposed model demonstrates superior performance, with not only a higher starting point for PSNR but also a more stable growth trend and faster convergence rate. In the later stages of training, the proposed model achieves higher PSNR values than the baseline model, indicating significant improvements in image quality, especially in the later phases of training. Therefore, it can be emphasized that the proposed model is effective and superior in image processing tasks, particularly suitable for applications requiring high PSNR.

Table 3 shows the ablation study results of the VR classroom scene classification optimization model, considering the impact of spatial scale information extraction and multi-scale global information enhancement techniques. Accuracy (Acc) is a standard metric for evaluating the performance of classification models, and the table presents the accuracies for two different training ratios (20% and 50%). As can be known from the data given in the table, without spatial scale information extraction and multi-scale global information enhancement, the model achieved an accuracy of 94.12% at 20% training ratio, and 95.68% accuracy at 50% training ratio, acting as a baseline of performance comparison. In case that spatial scale information extraction is adopted, there is an improvement in accuracy for both training ratios (reaching 95.63% and 96.12% respectively), and this indicates that even by solely extracting spatial scale information, the model's classification performance has been improved indeed. With the simultaneous application of both spatial scale information extraction and multi-scale global information enhancement, the model's performance further increases, reaching accuracy of 96.35% and 96.98%. This demonstrates the effectiveness of combining these two techniques in enhancing the model's classification capability. The conclusion is that by integrating the Transformer architecture, the model effectively extracts multi-scale spatial information and deeply integrates this information through global information enhancement techniques, thereby significantly improving the classification accuracy of VR classroom scenes. The ablation study results indicate that the introduction of each technology positively contributes to model performance, and the combination of both technologies yields the best performance.

Table 4 provides the classification results of different VR classroom scene classification optimization models in various scenes and training ratios, represented by accuracy (%). Each column corresponds to a different scene and training set ratio, and each row represents a specific classification model. From the table, it is clear that the SSGA-E model performs best in Scene 2 with an 80% training ratio, achieving an accuracy of 97.87%. VGG11 offers data across all scenes and training ratios, showing relatively stable performance, but it's not the highest in most cases. ViT reaches an accuracy of 97.25% in Scene 2 with an 80% training ratio, also a strong contender. SA-Gate provides data in all scenes but generally falls short of the proposed model in accuracy. D-CNNs reach 96.32% accuracy in Scene 2 with an 80% training ratio but underperform compared to the proposed model in other scenes and ratios. SF-CNN excels in Scene 2 with an 80% training ratio, matching SSGA-E at 97.87% accuracy, but still falls behind this paper's model in other scenarios and ratios. ViT-21k shows impressive performance in Scene 1 with a 50% training ratio at 96.24% accuracy but lacks data for Scene 2 and Scene 3 at a 50% training ratio. The proposed model provides data across all scenes and training ratios and achieves the highest accuracy in most cases. Particularly in Scene 2 with an 80% training ratio, it reaches an accuracy of 98.88%, the highest among all models. It can be concluded that the proposed model demonstrates competitive or optimal performance in all scenes and training ratios. Especially in Scene 2 with an 80% training ratio, achieving an accuracy of 98.88% is the highest among all models. Additionally, its performance is consistently stable in other scenes and ratios, almost always maintaining top-level performance. This indicates that the proposed model has strong generalization capabilities and efficient classification performance.

Table 5 provides performance data of different VR classroom scene classification optimization models in various scenes, assessed through Accuracy (Acc) and Intersection over Union (IoU) metrics. The table reveals that in the "Historical Recreation" scene, the VGG11 model showed the highest accuracy (99.5%) and IoU (97%). The proposed model also had high accuracy (98.7%) and IoU (96.8%) in this scene, nearly matching the best performance. In the "Geographical Exploration" scene, VGG11 again exhibited high accuracy and IoU, but proposed model model slightly underperformed with an accuracy of 89.3% and IoU of 83.6%. In the "Space Simulation" scene, the ViT model stood out with 85.6% accuracy and 65.6% IoU.

The proposed model model had lower performance in this scene, with an accuracy of 71.3% and IoU of 64.5%. In the "Underwater World" scene, the proposed model performed similarly to ViT, with 72.3% accuracy and 58.9% IoU. In the "Art Gallery" scene, all models generally showed decreased performance, with SF-CNN performing best and the proposed model at a moderate level. For other scenes such as "Science Laboratory," "Natural Disaster Simulation," "Digital Visualization," and "Language Immersion," the proposed model consistently showed relatively high accuracy and IoU across all scenes, notably in the "Natural Disaster Simulation" scene, with an accuracy of 55.8% and IoU of 11.2%, significantly higher than other models. Overall, while this paper's model was not always the best in individual scenes, it demonstrated robustness and consistency across multiple different scenes, particularly in more challenging scene classification tasks.