Pei Wang | Peng Li | Maria C. Cuntapay*
© 2022 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
During the learning process, students should keep positive emotions like delightfulness, cheerfulness, joy, and enthusiasm, and try to realize happy learning. This would improve learning efficiency and learning effect. However, most of the existing models cannot recognize negative discrete emotions and dimensional emotions, such as indignation and sadness, of students in classroom learning. To solve the problem, this paper deeply explores the recognition of student emotions in classroom learning based on image processing. Specifically, the authors designed a recognition model of emotional state of classroom learning, modeled the discrete emotions of students in classroom learning, and discussed the relationship between learning effect, response efficacy and stimulus. Furthermore, the generation process of student emotions in classroom learning was analyzed, and the recognition model of emotional state of classroom learning was finalized based on ResNet18. In addition, the recognition model was optimized by introducing the local importance pooling and adding a recalibration module. The experimental results verify the effectiveness of the constructed model.
image processing, student emotions in learning, emotion recognition
Online learning refers to the teaching and learning in an online virtual classroom [1-8]. Without being limited by time, place and space, it fully respects the individuality of students, and stimulates the motivation of learning, making it easier to achieve one-to-one communication between students and the teacher [9-13]. However, the teacher and students, and each student and his/her peers, are separated by space in most of the time of online learning. Hence, students must have a certain self-learning ability in online learning, where the classroom atmosphere is weak. Those failing to keep up with the learning rhythm is easy to generate negative emotions, such as worry, pain, distress, indignation, and indifference [14, 15].
Positive emotions can improve the function of the human body, and contribute to the development of intelligence. In the process of learning, especially online learning, students should keep positive emotions like delightfulness, cheerfulness, joy, and enthusiasm, and try to realize happy learning. This would improve learning efficiency and learning effect [16-20]. Therefore, monitoring student emotions in classroom learning is an important means to assist teachers in online teaching, and classroom learning emotions directly affect teaching and learning effects [21-23].
Putra and Arifin [24] built a real-time facial emotion recognition system, allowing teachers to monitor student emotions through classroom activities. The system should be reliable enough when running on mid-range computer specs. Bulut Özek [25] provided a system that integrates student emotion recognition with a learning management system. The aim is to disclose how the integration affects learner motivation and academic performance. A total of 103 engineering students were selected for the experiments, and divided into three groups: control group (face-to-face education), test group I (learning management system with emotional recognition), and test group II (learning management system). The results show that the academic performance and motivation of test group I were significantly better than those of test group II and the control group. Liu et al. [26] adopted a deep neural network (DNN) to classify the collected emotional electroencephalogram (EEG) data, and obtained the emotional state of college students according to the classification results. Considering that different features represent different information of the original data, the various EEG features were extracted to express the original EEG data as comprehensively as possible. Next, multiple features were fused with the integration technology of automatic learning models. After that, the fused features were imported to the DNN to obtain the final classification results. The video-based analysis of student classroom behaviors focuses on a single type of behavioral features, failing to fully reflect the classroom behaviors of students. In response to the above problems, Huang and Zhang [27] introduced a face recognition algorithm based on student classroom videos, and specified its implementation process. After improving the hybrid face detection model based on traditional models, they proposed a neural network algorithm for student expression recognition based on visual transducers. The experimental results show that the algorithm based on students’ classroom videos can effectively detect students' attention and emotional state in the classroom.
The negative impact of emotions on learning can be mitigated through the emotional regulation of students' classroom learning in the form of teacher-student interaction and student-student interaction, according to the emotional state of students in classroom learning. However, most of the existing models cannot recognize negative discrete emotions and dimensional emotions, such as indignation and sadness, of students in classroom learning. Some scholars tried to recognize emotions of samples with strongly invasive physiological signals. Their methods are too costly to implement, and difficult to promote, and may hinder students' real emotional expression and learning emotional experience in classroom learning. To solve the problem, this paper designs a student emotion recognition model for classroom learning based on image processing. Section 2 devises a recognition model of emotional state of classroom learning, models the discrete emotions of students in classroom learning, and discusses the relationship between learning effect, response efficacy and stimulus. In addition, the generation process of student emotions in classroom learning was analyzed, and the recognition model of emotional state of classroom learning was finalized based on ResNet18. Section 3 optimizes the recognition model by introducing the local importance pooling and adding a recalibration module. The experimental results verify the effectiveness of the constructed model.
The first step to study students’ discrete emotions in classroom learning is to choose a representation model (Figure 1). This paper discretizes students’ discrete emotions in classroom learning into eight basic emotions, namely, joy, appreciation, terror, surprise, grief, distain, fury, and caution.
This paper classifies the student emotions in classroom learning that affect the learning effect into two categories: discrete emotions and dimensional emotions. Moreover, the authors differentiated the effects of student emotions in classroom learning on aspects like learning enthusiasm, classroom cooperation, duration, task completion, knowledge expansion, and skill levels. Specifically, the learning effect of students in online classroom is the best, when the emotional stimulation is moderate, and the response efficacy is moderate to high (Figure 2).
Figure 3 explains the generation process of student emotions in classroom learning. It can be roughly divided into four phases: classroom experience, knowledge cognition, physical response, and psychological emotions. To improve the accuracy of students' emotional state recognition in classroom learning, this paper chooses to fuse students' facial expression features at different levels.
Figure 1. Discrete emotion model of students in classroom learning
Figure 2. Relationship between learning effect, response efficacy, and stimulus
Figure 3. Generation process of student emotions in classroom learning
Figure 4. Framework of emotional state recognition model for classroom learning
In convolutional neural networks (CNNs), there is a difference in the feature resolutions extracted by shallow and deep networks. The former features have more detailed information but weaker semantics, resulting in a poor overall presentation effect. The latter features have fewer details, but strong semantics. In this paper, the two types of features are fused, and the local and global features of the facial expressions of students participating in online learning are extracted at the same time, which is conducive to the recognition of their emotional states in classroom learning.
With ResNet18 as the basic framework, this paper constructs the emotional state recognition model for classroom learning (Figure 4), and integrates local and global features based on three blocks. An attention module was added to extract more effective facial expression features of students.
It is assumed that the input feature map of facial expressions of online learning students is of the size D×F×Q; the number of channels is D; the height of the feature map is F; the width of the feature map is q. By the value of D, the facial expression features of online learning students are divided into H feature groups, Ψ={a1, a2.., .an}, which satisfies ai $\in$ RD/H, n=F×Q. Then, the features in group Ψ are subject to global average pooling. The eigenvector h can be obtained by:
$h=G_{h o}(\Psi)=\frac{1}{n} \sum_{i=1}^n a_i$ (1)
Let ai be the original feature information in the feature map of facial expressions of online learning students; h be the semantic information in group Ψ. Then, the attention coefficient di of each feature can be calculated as the dot product between ai and h:
$d_i=h \cdot a_i$ (2)
The attention coefficient di can effectively measure the similarity between h and ai. Next, di is normalized to reduce the influence of the attention coefficient difference between image samples over the execution effect of the attention module. Let λd and εd denote the mean and standard deviation of di, respectively; d*i denote the normalized value of di; ρ denote a constant. Then, we have:
$\lambda_d=\frac{1}{n} \sum_j^n d_j$ (3)
$\varepsilon_d^2=\frac{1}{n} \sum_j^n\left(d_j-\lambda_d\right)^2$ (4)
$d_i^*=\frac{d_i-\lambda_d}{\varepsilon_d+\rho}$ (5)
In addition, hyperparameters α and φ are introduced to move and scale di*:
$\varphi_i=\alpha d_i^*+\phi$ (6)
The ϕi is activated by the sigmoid function. The enhanced eigenvector ai* can be obtained as the dot product between the original feature information in the feature map of facial expressions of online learning students and ϕi:
$a_i^*=a_i \cdot \varepsilon\left(\varphi_i\right)$ (7)
Then, the enhanced Ψ can be expressed as:
$\Psi^*=\left\{a_1^* \cdots_n\right\}, a_i^* \in R^{D / H}, n=F \times Q$ (8)
The pooling operation, which can further reduce the size of feature maps, is extremely important to the processing of the images on student facial expressions. To prevent the network from losing important facial features of students, this paper improves the residual block and down sampling layer of ResNet18. traditional strided convolution usually includeS two steps: convolution with stride of 1 and down sampling. Let PO be the pooling function; EN be the input feature map. Then, we have:
$P O(E N)_{a, b}=\left\{\begin{array}{c}1, \text { if } a \text { and } b \text { are both multiples of } r \\ 0, \text { otherwise }\end{array}\right.$ (9)
The above formula shows that the strided convolution only pays attention to the fixed positions in the sliding window of average pooling, while ignoring other positions. This will lead to the loss of important feature information of the expressions in the key areas of the student faces. As a result, the proposed emotional state recognition model for classroom learning will be less accuracy in recognizing the facial expressions of students in online learning.
To fully extract the facial expression features of students in online learning, this paper introduces the local importance pooling operation. Through the convolution of logarithmic data module, the feature weight of the input feature map for the facial expressions of students in online learning is generated, and locally normalized. As for the feature loss of down sampling, the problem is solved by automatic reidentification of features during the pooling process.
Let SC be the post-pooling output feature map; (a, b) is the sliding window position corresponding to the input feature map; (a’, b’) is the position of the corresponding output; (Δa, Δb) be the relative position within the sliding window; Θ be the kernel in the sliding window. Then, the pooling operation can be optimized by:
$S C_{a^{\prime}, b^{\prime}}=\frac{\sum_{(\Delta a, \Delta b) \in \Theta} \quad P O(E N)_{a+\Delta a, b+\Delta b} \quad \quad E N_{a+\Delta a, b+\Delta b}}{\sum_{(\Delta a, \Delta b) \in \Theta} \quad P O(E N)_{a+\Delta a, b+\Delta b}}$ (10)
To generate the importance feature map, tiny fully convolutional networks are constructed to characterize the importance function PO in the local importance pooling. To prevent the loss of features during down sampling, the features are automatically reidentified by learning the larger WD(EN) value at the relative position by the PO-based logarithmic module WD. Composed of only one 1×1 convolutional layer, this module is good at learning the feature map from the facial expressions of students in online learning. Normally, the feature mapping and activation of feature maps in different channels do not obey the normal distribution. To solve the problem, this paper performs affine instance normalization on WD, and then rescales the feature map based on learnable affine parameters. To ensure the nonnegativity of weights, the following operations are performed on the logarithmic module WD:
$P O(E N)=\exp (W D(E N))$ (11)
To sum up, the local importance pooling can be expressed as:
$S C_{a^{\prime}, b^{\prime}}=$$\frac{\sum_{(\Delta a, \Delta b)\in\Theta} \quad P O(E N)_{a+\Delta a, b+\Delta b} \quad\exp \quad(W D(E N))\quad_{a+\Delta a, b+\Delta b}}{\sum_{(\Delta a, \Delta b)\in\Theta} \quad\quad\exp (W D(E N))\quad_{a+\Delta a, b+\Delta b}}$ (12)
Different students have different facial expression features in online learning. These differences will affect how our model extracts facial expression features, and in turn hinder the recognition and classification of emotional states in the subsequent classroom learning. To recognize the emotional states in classroom learning more accurately, it is necessary to enable the model learn more discriminative feature information, that is, to increase the feature weights of key parts in the input facial expression images of online learners.
Figure 5. Structure of recalibration module
The additive recalibration module can quantify the relative importance of different facial expression features, and further realize the adaptive calibration of feature weights, so that the constructed model ignores useless features and pays attention to effective features. The recalibration module includes two sub-modules, style pooling and style integration (Figure 5).
The style pooling can be further split into average pooling and standard deviation pooling. Let AÎRM×D×F×Q be the input feature map for the facial features of online learners; λmd be the output of average pooling; aNC be the original input feature map. Then, the average pooling can be expressed as:
$\lambda_{m d}=\frac{1}{F Q} \sum_{f=1}^F \sum_{q=1}^Q a_{N C}$ (13)
The output εmd of standard deviation pooling can be obtained by:
$\varepsilon_{m d}=\sqrt{\frac{1}{F Q} \sum_{f=1}^F \sum_{q=1}^Q\left(a_{N C}-\lambda_{m d}\right)^2}$ (14)
The eigenvector emd generated by style pooling can be expressed as:
$e_{m d}=\left[\lambda_{m d}, \varepsilon_{m d}\right]$ (15)
In the style integration submodule, the channel fully connected layer is represented by TQL, the batch normalization layer is represented by PG, and Sigmoid is the activation function. Firstly, emd is imported into the sub-module. The specific process of the module is given by:
$c_{m d}=q_d \cdot e_{m d}$ (16)
The output eigenvector λ(c) d of averaging can be expressed as:
$\lambda_d^{(c)}=\frac{1}{M} \sum_{m=1}^M c_{m d}$ (17)
The output eigenvector ε(c) of taking standard deviation can be expressed as:
$\varepsilon_d^{(c)}=\sqrt{\frac{1}{M} \sum_{m=1}^M\left(c_{m d}-\lambda_d^{(c)}\right)^2}$ (18)
Let cmd* be the output eigenvector of batch normalization; α, ωÎRD be the affine transform parameters. Then, we have:
${c_{m d}}^*=\alpha_d\left(\frac{c_{m d}-\lambda_d^{(c)}}{\varepsilon_d^{(c)}}\right)+\omega_d$ (19)
The output eigenvector hmd processed by activation function can be obtained by:
$h_{m d}=\frac{1}{1+t^{-\hat{c}_{m d}}}$ (20)
The final output feature map can be calculated by:
$A_{m d}{ }^*=h_{m d} \cdot A_{m d}$ (21)
Figure 6. Experimental scheme
To acquire the data on classroom learning emotions of online learners, this paper designs a scheme for an 1h-long online learning experiment. As shown in Figure 6, each student enters the classroom for about 15mins, and take a brief 2min-long rest. Then, he/she receives one of the eight situational teaching scenarios in the scheme. In this way, eight different basic learning emotions are triggered, namely, joy, appreciation, terror, surprise, grief, distain, fury, and caution.
The effectiveness of our model was demonstrated by comparing its recognition performance with that of two reference models: the optical flow extraction of movement features (OF), and hidden Markov model (HMM). Accuracy and F1-score were selected as the performance metrics, considering the class imbalance between the sample set of student expression images, which come from different courses. The performance of the three models is compared in Table 1 and Figure 7.
Table 1. Emotions recognized by different models
|
Model |
Sample set 1 |
Sample set 2 |
||
|
Accuracy |
F1-score |
Accuracy |
F1-score |
|
|
OF |
0.8147 |
0.8127 |
0.7958 |
0.7748 |
|
HMM |
0.8263 |
0.8912 |
0.8471 |
0.8063 |
|
Our model |
0.9274 |
0.9368 |
0.8869 |
0.8981 |
As shown in Figures 7 (a) and (b), our model achieved better accuracy and F1-score than the two reference models, indicating its superior performance over the other models. On sample set 1, the accuracy and F1-score of our model were 12.18% and 2.89% higher than those of OF, respectively, and 13.24% and 4.86% higher than those of HMM, respectively. On sample set 2, the accuracy and F1-score of our model were 10.27% and 4.48% higher than those of OF, respectively, and 13.72% and 10.24% higher than those of HMM, respectively. The results indicate that our model improves the utilization efficiency of the input facial images of online learners, and verifies the effectiveness of the improvement of the model.
Figure 8 shows the emotions recognized based on facial expression images of students in classroom learning. Subgraphs a and b list the clustering of stimulus scores and response efficiency scores of student emotions in classroom learning, as well as the clustering of dominance scores and response efficiency scores. The clustering results show that eight kinds of emotions, namely, joy, appreciation, terror, surprise, grief, distain, fury, and caution were generated in the experiments, and the scores of each online learning video clip were summarized. It can be seen that all the results belonged to two 2D spaces: stimulus-response efficacy and response efficacy-dominance.
(a)
(b)
Figure 7. Emotions recognized by different models
Figure 9 presents the predicted intensities of the eight emotions, aiming to more visually display the positive correlation between the intensity of each kind of emotion and the stimulating time of situation teaching. The intensities of the eight emotions were estimated by our model, and illustrated as curves. The x-axis and y-axis are the number of frames that an emotion lasts, and the emotional intensity. It can be seen that the intensity of each type of emotion increased with the number of frames. The result agrees with the reality, and further demonstrates the effectiveness of our model.
(a)
(b)
Figure 8. Emotions recognized based on facial expression images
Figure 9. Predicted intensities of the eight emotions
Based on image processing, this paper deeply examines the recognition of student emotions in classroom learning. Firstly, a recognition model of emotional state of classroom learning was designed, the discrete emotions of students in classroom learning were modeled, and the relationship between learning effect, response efficacy and stimulus were discussed. After that, the generation process of student emotions in classroom learning was analyzed, and the recognition model of emotional state of classroom learning was finalized based on ResNet18. In addition, the recognition model was optimized by introducing the local importance pooling and adding a recalibration module. Moreover, the authors prepared an experimental scheme for recognizing student emotions in classroom learning, and compared the recognition performance of our model and two reference models. The experimental results verify the effectiveness of the constructed model. Further, the emotions recognized based on the facial feature images of students in classroom teaching were displayed, reflecting that all the results belonged to two 2D spaces: stimulus-response efficacy and response efficacy-dominance. In the end, the predicted intensities of the eight emotions were summarized, aiming to more visually display the positive correlation between the intensity of each kind of emotion and the stimulating time of situation teaching. The findings further confirm the validity of our model.
[1] Malaj, E.G., Ibro, M., Marinova, G. (2022). Experimental setup for Hardware Encryption logic evaluation for e-learning. In 2022 29th International Conference on Systems, Signals and Image Processing (IWSSIP), pp. 1-4. https://doi.org/10.1109/IWSSIP55020.2022.9854421
[2] Vapiwala, F., Pandita, D. (2022). Strategies for effective use of gamification technology in e-learning and e-assessment. In 2022 7th International Conference on Business and Industrial Research (ICBIR), pp. 596-601. https://doi.org/10.1109/ICBIR54589.2022.9786495
[3] Kulkarni, T., Kabra, M., Shankarmani, R. (2019). User profiling based recommendation system for e-learning. In 2019 IEEE 16th India Council International Conference (INDICON), pp. 1-4. https://doi.org/10.1109/INDICON47234.2019.9028982
[4] Alameen, A., Dhupia, B. (2019). Implementing adaptive e-learning conceptual model: A survey and comparison with open source LMS. International Journal of Emerging Technologies in Learning (iJET), 14(21): 28-45. https://doi.org/10.3991/ijet.v14i21.11030
[5] Bennani, S., Maalel, A., Ben Ghezala, H. (2022). Adaptive gamification in E-learning: A literature review and future challenges. Computer Applications in Engineering Education, 30(2): 628-642. https://doi.org/10.1002/cae.22477
[6] El Ouesdadi, N., Sara, R. (2021). E learning and the new pedagogical practices of Moroccan teachers. In Proceedings of the 4th International Conference on Networking, Information Systems & Security, 1-7. https://doi.org/10.1145/3454127.3456590
[7] El Omari, A., Tridane, M., Belaaouad, S. (2022). Teacher e-training and student e-learning during the period of confinement caused by COVID-19 in case of Morocco. International Journal of Advanced Computer Science and Applications, 13(2): 814-818. https://doi.org/10.14569/IJACSA.2022.0130294
[8] Ung, L.L., Labadin, J., Mohamad, F.S. (2022). Computational thinking for teachers: Development of a localised E-learning system. Computers & Education, 177: 104379. https://doi.org/10.1016/j.compedu.2021.104379
[9] Giannakos, M.N., Mikalef, P., Pappas, I.O. (2021). Systematic literature review of E-learning capabilities to enhance organizational learning. Information Systems Frontiers, 24: 619-635. https://doi.org/10.1007/s10796-020-10097-2
[10] Vanitha, V., Krishnan, P., Elakkiya, R. (2019). Collaborative optimization algorithm for learning path construction in E-learning. Computers & Electrical Engineering, 77: 325-338. https://doi.org/10.1016/j.compeleceng.2019.06.016
[11] Sharma, P., Esengönül, M., Khanal, S.R., Khanal, T.T., Filipe, V., Reis, M.J. (2018). Student concentration evaluation index in an e-learning context using facial emotion analysis. In International Conference on Technology and Innovation in Learning, Teaching and Education, pp. 529-538. https://doi.org/10.1007/978-3-030-20954-4_40
[12] Imani, M., Montazer, G.A. (2019). A survey of emotion recognition methods with emphasis on E-Learning environments. Journal of Network and Computer Applications, 147: 102423. https://doi.org/10.1016/j.jnca.2019.102423
[13] van der Haar, D.T. (2019). Student emotion recognition in computer science education: A blessing or curse. In International Conference on Human-Computer Interaction, pp. 301-311. https://doi.org/10.1007/978-3-030-21814-0_23
[14] Karanian, B.A., Taajamaa, V.M., Parlier, C.A., Eskandari, M. (2019). Provoked emotion in student stories of motivation reveal gendered perceptions of what it means to be innovative in engineering. In 2019 ASEE Annual Conference & Exposition. https://doi.org/10.18260/1-2--33213
[15] Demirkol, D., Seneler, C. (2019). Evaluation of student information system (SIS) in terms of user emotion, performance and perceived usability: A Turkish university case (an empirical study). Procedia Computer Science, 158: 1033-1051. https://doi.org/10.1016/j.procs.2019.09.145
[16] Georgescu, M.I., Duţǎ, G.E., Ionescu, R.T. (2022). Teacher–student training and triplet loss to reduce the effect of drastic face occlusion. Machine Vision and Applications, 33(1): 1-19. https://doi.org/10.1007/s00138-021-01270-x
[17] Wang, B., Al Barakati, A., Hasan, H. (2022). Characteristics of mathematical statistics model of student emotion in college physical education. Applied Mathematics and Nonlinear Sciences, 7(1): 841-850.
[18] Cui, Y., Wang, S., Zhao, R. (2021). Machine learning-based student emotion recognition for business English class. International Journal of Emerging Technologies in Learning (iJET), 16(12): 94-107. https://doi.org/10.3991/ijet.v16i12.23313
[19] He, J., Peng, L., Sun, B., Yu, L., Guo, M. (2021). Dual multi-task network with bridge-temporal-attention for student emotion recognition via classroom video. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. https://doi.org/10.1109/IJCNN52387.2021.9533471
[20] Gkontzis, A.F., Karachristos, C.V., Panagiotakopoulos, C.T., Stavropoulos, E.C., Verykios, V.S. (2017). Sentiment analysis to track emotion and polarity in student fora. In Proceedings of the 21st Pan-Hellenic Conference on Informatics, pp. 1-6. https://doi.org/10.1145/3139367.3139389
[21] Yu, J.Y., Bai, X.J. (2021). Analysis of classroom learning behaviors based on internet of things and image processing. Traitement du Signal, 38(3): 845-851. https://doi.org/10.18280/ts.380331
[22] AL-Ayash, A., Kane, R.T., Smith, D., Green-Armytage, P. (2016). The influence of color on student emotion, heart rate, and performance in learning environments. Color Research & Application, 41(2): 196-205. https://doi.org/10.1002/col.21949
[23] Ayeche, F., Alti, A. (2021). Facial expressions recognition based on Delaunay triangulation of landmark and machine learning. Traitement du Signal, 38(6): 1575-1586. https://doi.org/10.18280/ts.380602
[24] Putra, W.B., Arifin, F. (2019). Real-time emotion recognition system to monitor student’s mood in a classroom. In Journal of Physics: Conference Series, 1413(1): 012021. https://doi.org/10.1088/1742-6596/1413/1/012021
[25] Bulut Özek, M. (2018). The effects of merging student emotion recognition with learning management systems on learners’ motivation and academic achievements. Computer Applications in Engineering Education, 26(5): 1862-1872. https://doi.org/10.1002/cae.22000
[26] Liu, L., Ji, Y., Gao, Y., Li, T., Xu, W. (2022). A data-driven adaptive emotion recognition model for college students using an improved multifeature deep neural network technology. Computational Intelligence and Neuroscience, 2022: Article ID 1343358. https://doi.org/10.1155/2022/1343358
[27] Huang, D., Zhang, W. (2021). Research on learning state based on students’ attitude and emotion in class learning. Scientific Programming, 2021. Article ID 9944176. https://doi.org/10.1155/2021/9944176
