Classroom Video Image Emotion Analysis Method for Online Teaching Quality Evaluation

Classroom teaching is a process of emotional communication between teachers and students. Whether online or offline, facial expressions are more explicit behaviors that can reflect the emotions of teachers and students than body movements and speech behaviors [1-5]. It is more direct and effective to capture the emotional changes of teachers and students and classroom emotions through facial expressions. In recent years, with the development of facial expression recognition technology, it has become an inevitable trend to promote the application of image processing in online teaching classroom emotion analysis [6-14].

Classroom emotion is an important dimension to evaluate classroom effect. From a micro perspective, online teaching quality evaluation based on image emotion analysis can help teachers obtain students’ classroom responses and judge the classroom atmosphere [15-19]. From a macro perspective, online teaching quality evaluation based on image emotion analysis has promoted the development of online teaching supported by facial expression recognition technology, making online teaching methods more abundant and diversified. Therefore, online teaching quality evaluation based on image emotion analysis has extremely important exploration value and application value.

Online teaching has the advantage of not being limited by location and space, but it also has some disadvantages. Wu and Huwan [20] analyzed and modeled the facial expression features of learners in the situations collected and extracted by the students of the project group, and integrated the time consumption, space occupation and classification effect of feature vectors generated by the improved model. The literature [21] aims to verify the application effect of online automatic evaluation system in English translation teaching, and understand students’ satisfaction with different feedback methods. Using the methods of questionnaire and comparative experiment, this paper makes a comparative analysis of three feedback methods: teacher feedback, online automatic feedback and teacher feedback combined with online automatic feedback. The study will help teachers better understand the deficiencies in translation teaching, so that they can take effective measures to solve these problems in the follow-up teaching process and improve the teaching effect. Compared with other models, facial expression plays a crucial role in emotion recognition. In this work, Saxena et al. [22] proposed an integrated network, which can use depth learning technology to recognize the level of emotional intensity in facial images in real time. Research on facial expression cognition based on expression intensity level is very useful in medical care, collaboration, Industry 4.0 and other applications. This work suggests using two other important parameters, namely, valence and emotion intensity, to enhance emotion recognition. This helps the machine to make a better automatic response to emotions. The proposed architecture achieves 99.1% and 99.11% accuracy for the price model and discrete model of offline image data, respectively. With the progress of machine learning, facial expression recognition (FER) has been greatly improved, which is mainly due to the development of deep learning methods for automatic facial expression classification. In the literature [23], the performance evaluation of the pre-training open source model for emotion facial expression recognition was conducted for images with different facial expression intensities. A total of 1,512 video frames from the ADFES-BIV dataset were submitted to five pre-training machine learning models for performance evaluation. The results show that the tested pre-training machine learning model is very sensitive to changes in facial expression intensity. In addition, a large recall variation rate was observed between different facial expressions.

According to the current research situation at home and abroad, the online teaching quality evaluation based on image emotion analysis is still in the stage of theoretical research and practical exploration, with low accuracy of expression recognition, unclear emotional scheme for evaluating online teaching, and low applicability of expression recognition model. In order to solve the above problems, this paper studies the method of classroom video image emotion analysis for online teaching quality evaluation. First, in the second part of this paper, the classroom video image emotion analysis task is divided into facial expression recognition task and facial feature point location task, and multi-task learning is carried out to achieve real-time switching between the two tasks for different types of input. In the third part of the article, this paper proposes to use the tag attention mechanism to deeply mine the key areas of the face in the classroom video image, and always maintain the compactness of the distribution of the center sample of the classroom video image and its neighborhood samples in the feature space. Finally, the fourth part of the article analyzes the online classroom teaching emotion based on the expression activity of teachers and students in the online classroom, and evaluates the online teaching quality from the side. The experimental results verify the validity of the model.

When part of the classroom video image is occluded, if the facial expression recognition model constructed can still recognize the corresponding facial expression category, it shows that the reserved area of the classroom video image is more important for facial expression recognition at this time. In addition, because there are large intra-class differences in facial expression recognition, the distribution of facial expression features extracted from the model in the class is relatively decentralized. For this reason, this paper proposes to use the tag attention mechanism to deeply mine the key areas of the face in the classroom video image, and always maintain the compactness of the distribution of the center sample and its neighborhood samples in the feature space of the classroom video image, so as to complete the correction of the feature distribution of the center sample. The structure of the model is shown in Figure 3.

The facial expression recognition model uses KL divergence and JS divergence to measure the difference between the two distributions. The formula is as follows:

$S U(E \| W)=\sum E(a) \cdot \log \frac{E(a)}{W(a)}$              (6)

$J S(E \| W)=\frac{1}{2} K L\left(E(a) \| \frac{E(a)+W(a)}{2}\right)$ $+\frac{1}{2} K L\left(W(a) \| \frac{E(a)+W(a)}{2}\right)$        (7)

Because the reserved classroom video image area is the key area in the task of facial expression recognition. If the parameters of a certain location area of the facial expression recognition model are updated violently, then the information of the location area is considered to be more important for the recognition performance of the model. Therefore, this paper constructs an importance matrix to reflect the importance of each position on the image. Specifically, each sample is adaptively calculated based on the sample labels of classroom video images nd gradient of model parameters.

3.png

Figure 3. Schematic diagram of model structure

The input image sample SR={(a1, b1)}, ..., (am, bm)} is provided, including m classroom video images and corresponding digital tags. The digital tag of image a_i is represented by bi∈{1,..., d}, and the number of expression categories is represented by d. Supposing that the value of the d_th position of the classification layer in the facial expression recognition model is represented by POS^d, and the feature mapping of the l_th channel of the feature map in a certain convolution layer is set. The corresponding POS^bi is set according to a_i and b_i, and the importance weight x^bi_l of each channel on X through the gradient between POS^bi and X^lis calculated. The calculation formula is as follows:

$x_l^{b_i}=\frac{1}{v \times u} \sum_{i=1}^v \sum_{j=1}^u \frac{\partial P O S^{b_i}}{\partial X^l}$             (8)  

x^bi_l can be seen as the sum of gradients at each pixel in X^l, which indicates the importance of X^l to x^bi_l. Next, the linear weighting operation of the image is performed:

$C V I=\sum_{l=1}^{S U} x_l^{b_i} X^l$      (9)

It is supposed that the channel number of X is represented by SU, then CVI∈R^v×u. In order to retain the pixels at the key positions in the image, this paper sets the threshold value of filtering the pixels at the unimportant positions in the CVI of classroom video images. Because X^l is the facial expression feature extracted from the deeper network of the model, the following activation operations are carried out to CVI in this paper, and the activation function used is ReLU function:

$C V I_{\operatorname{ReLU}}=\operatorname{ReLU}\left(C V I-\frac{1}{v \times u} \sum_{i=1}^v \sum_{j=1}^u C V I_{i j}\right)$    (10)

Finally, CVI_ReLU is normalized and sampled until an image with the same size as the original image is obtained. Assuming that the importance matrix is represented by N∈R^f*q, the function Max() is used to calculate the maximum value. The input characteristic graph is bilinear interpolated and represented by BI( ), then:

$I M=B I\left(\frac{M a p_{\mathrm{Re} L U}}{\operatorname{Max}\left(M a p_{\mathrm{Re} L U}\right)}, f, q\right)$     (11)

Based on the acquired importance matrix IM, the pixels of the classroom video image sample a_iare processed as follows:

$\bar{a}_i=I M \cdot a_i$     (12)

For processed image ā_i, since the value range of all elements in the importance matrix IM is [0, 1], so the degree of retention of pixels at partial position of ā_i that is not 0 will correspond to the size of all elements in IM. Because the labels of ā_i and a_i are the same, so the cross entropy loss function CE can be calculated based on the same label:

$C E_{C L}=C E\left(a_i, b_i\right)+C E\left(\tilde{a}_i, b_i\right)$     (13)

An ideal facial expression recognition model expects that when an image with partial occluded area is input network, the facial expression feature information extracted by the model is basically consistent with the feature information extracted when the original image sample is input, so it can be considered that the constructed model has a good performance in filtering redundant information in the image sample. Following the reference domain adaptive algorithm idea, this algorithm takes the original image a_i as the source domain, and the image ā_i  with area occluded is taken as the target domain, and the distance between the source domain and the target domain in the feature space is shortened based on the maximum mean difference. Let the data after occluding some pixels be represented by SR^*={(a^_₁,b₁),...,(a^__m,b_m)}, and the mapping function is represented by ψ(), then the distance between the source domain and the target domain can be calculated by the following formula:

$\operatorname{DIS}\left(S R, S R^*\right)=\left\|\frac{1}{m} \sum_{i=1}^m \psi\left(a_i\right)-\frac{1}{m} \sum_{i=1}^m \bar{a}_j\right\|_F^2$      (14)

There are different forms of ψ() for different tasks, and the above equation can be expanded by adopting kernel function technique:

$D I S\left(S R, S R^*\right)$ $=\left\|\begin{array}{l}\frac{1}{m^2} \sum_{i=1}^m \sum_{i^*}^m l\left(a_i, a_i^*\right) \\ -\frac{1}{m^2} \sum_{i=1}^m \sum_{j=1}^m l\left(a_i, \bar{a}_j\right)+\frac{1}{m^2} \sum_{j=1}^m \sum_{j^*}^m l\left(\bar{a}_j, \bar{a}_j^*\right)\end{array}\right\|_F$     (15)

In order to obtain the distribution difference between the source domain and the target domain in the feature space, this paper measures the distance between them in the kernel space based on the maximum mean difference. It is assumed that the depth information corresponding to ā_i and a_i is represented by ḡ_i and g_i, then the following distribution loss DL_DIS is as follows:

$D L_{D I S}=D I S\left(g, \bar{g}_i\right)$          (16)

Because the emotions of teachers and students in online classes are complex and changeable, it is not appropriate to use traditional facial expression labels. In addition, for facial expression recognition tasks with large intra class differences, it is necessary to properly constrain the feature distribution in their feature space. In view of the above problems, this paper takes certain measures from the level of classroom video image samples. Considering the original image and the image with partial occluded area, the algorithm in this chapter ensures that the key area information of the neighborhood image is consistent with the information of the corresponding area in the original image by finding the neighborhood of each center classroom video image sample.

The cascade features [g_i, ḡ_i] of g_i and ḡ_i are clustered , and n neighborhood images {a^-1_i,…,aⁿ_i} is found for classroom video image sample a_i. Because this paper comprehensively considers the global and local information of facial expression, the feature information of neighborhood images in local areas is ensured to have certain similarity. Next, n neighborhood images are sent into the built facial expression recognition model. The feature {g_i¹,...,g_iⁿ} extracted from the model gives a confidence degree to each classroom video image sample in the neighborhood by measuring the distance between g_i and {g_i¹,...,g_iⁿ}, that is, the feasibility of the feature of neighborhood image aⁿ_i to modify the feature of a_i, which is represented by δ^j_i and calculated by the following formula:

$\delta_i^j=p^{-\frac{\left\|g_i-g_i^j\right\|\quad^2}{2^* \varepsilon^2}}$     (17)

KL divergence is used to measure the distribution difference between two classroom video image samples. Assuming that the probability distribution of classroom video image sample a_i generated by the facial expression recognition model constructed is represented by E_i, and the probability distribution of the jt_h sample in the neighborhood of classroom video image sample a_i is represented by E^j_i, then there is the following modified loss expression:

$C L_{A M}=\frac{1}{m} \sum_{i=1}^m \sum_{j=1}^n \delta_i^j \bullet K L\left(E_i^j, E_i\right)$      (18)

Through the correction loss of the construction, the facial expression recognition model constructed avoids the negative impact of the label problem in the optimization process, while maintaining the compactness of the intra class distribution. Assuming that the super parameters used to control the loss ratio are represented by γ and α, to sum up, the following formula gives the expression of the total loss function of the model training:

$L O S S=C E_{C L}+\gamma \cdot D L_{D I S}+\alpha \cdot C L_{A M}$      (19)

In order to test the model, 1,250 online classroom video images were selected in this paper. Each image was set with emotion tags including pleasure, stability, anxiety, helplessness, frustration, anger, disgust, fear, sadness, and surprise, etc. Figure 4 shows the training and testing of the constructed model characterized by three indicators: test accuracy, training loss and test loss. It can be seen from the figure that the output of the model is basically stable after 20,000 iterations, and the recognition accuracy can reach 72.4%.

4.png

Figure 4. Training and testing of the constructed model

5.png

Figure 5. Comparison of test accuracy rates of different models

In order to verify that the model constructed in this paper has more effective facial expression recognition capability than the traditional convolutional neural network model, the test results of this model and the convolutional neural network expression recognition model are compared. Figure 5 shows the test accuracy comparison results of the two models. It can be seen from the figure that the convergence speed of the model in this paper is faster and the recognition accuracy is higher. This verifies the advantages of the model in high-quality facial expression feature extraction. Table 1 shows the test accuracy confusion matrix of different models. The facial expression recognition accuracy of frustration category is the highest, while that of anxiety category is the lowest.

In order to analyze the impact of the constructed model on the intra class distribution of facial expression recognition tasks, this paper visualizes the facial expression features generated by the model training in this paper, and the visualization results are shown in Figure 6. It is obvious from Figure 6 that the interval of different categories in the feature space embedded in the model is obvious and the distribution within the category is relatively compact.

Facial feature point localization is an auxiliary task of facial expression recognition. In order to verify that the model constructed in this paper has a good performance in facial feature point locating, in this paper, the average detection error of selected eyes, nose, and two corners of mouth is taken as the performance evaluation index. Figure 7 shows the average detection error comparison of facial feature point locating. It can be seen from the figure that compared with the traditional convolutional neural network model, the traditional multitask convolutional neural network and the Retinaface model, the feature point positioning performance of the model in this paper is more ideal, which verifies the practicability of its online classroom video image facial expression recognition.

Table 1. Confusion matrix of test accuracy of different models

Table 2. Online teaching quality evaluation results of different models

Table 3. Manual evaluation results of online teaching quality

6.png

Figure 6. Facial expression feature distribution

7.png

Figure 7. Comparison of average detection errors of face feature point positioning

The traditional convolutional neural network model and the model constructed in this paper are used for online teaching emotion evaluation experiment. In this paper, three real online teaching videos are selected and input into the trained models. Table 2 shows the online teaching quality evaluation results of different models.

After many tests and experiments, this paper sets the weight of teachers’ expression activity and students’ expression activity as 0.6 and 0.4, and the threshold values as 0.4 and 0.6. Table 3 shows the results of manual evaluation of online teaching quality, which provides a reference for the experiment.

It can be seen from Table 2 and Table 3 that the traditional convolutional neural network model and the model constructed in this paper distinguish that there are certain differences between the classroom emotional activity of teachers and students in the three video clips and the online classroom quality evaluation results, but the evaluation results of the model constructed in this paper are more consistent with the results of manual evaluation, reflecting that the actual effect of classroom emotional performance is relatively ideal. Due to the correction loss of the construction, the model avoids the negative impact of the label problem in the optimization process, while maintaining the compactness of the intra class distribution. Therefore, the model constructed has better performance in online teaching emotional evaluation, and has certain feasibility and application significance.