A Multi-Task Learning Framework for Character Face Recognition and Emotion Analysis in Television Programs

2.1 Model framework

The proposed face recognition method for characters in television programs based on multi-task learning adopts an end-to-end encoder-decoder structure as the core framework, aiming to improve recognition performance in complex television media scenarios through feature sharing and collaborative optimization between tasks. The method framework is shown in Figure 1. In the encoding stage, the model introduces a channel attention mechanism to dynamically mine discriminative features important for recognition tasks, such as facial contours and texture information of key regions like eyes and mouth, in response to problems such as uneven lighting, diverse poses, and local occlusions that may occur in character faces in television frames. This is achieved by enhancing the feature expression capability of key channels and suppressing interference from irrelevant background noise. In the decoding stage, a top-down feature aggregation process is designed to progressively fuse high-level semantic features with low-level detailed features, thereby preserving overall facial structure information while capturing local subtle feature differences. Specifically, the model shares encoder-decoder module parameters to achieve deep coupling between the face recognition task and auxiliary tasks, enabling mutual guidance in the feature extraction process among different tasks, forming a multidimensional representation of character faces and enhancing the model’s generalization ability in complex scenarios.

1.png

Figure 1. Framework of face recognition method for characters in television programs based on multi-task learning

To address the temporal characteristics of television media video streams, the model further embeds a spatio-temporal attention module to aggregate cross-frame contextual features. This module analyzes the motion trajectory and appearance changes of character faces in a continuous S-frame video clip to capture temporal dependencies such as dynamic evolution of facial expressions and head movements, effectively resolving recognition ambiguity caused by pose mutations or occlusions in single-frame images. The model input consists of video clips and image sets containing S frames, and the output is the corresponding scale facial saliency mask maps. By jointly optimizing the multi-task loss function of the face recognition task and auxiliary tasks, accurate extraction of character identity features is achieved. Specifically, the model input is the video clip Z_n={U_u∈R ^G×Q×3}^S_u=1 and the image set Z_U={U_u∈R ^G×Q×3}^S_u=1. The output is the same scale saliency mask maps E_n={T_u∈R^G×Q×3}^S_u=1 and E_U={T_u∈R ^G×Q×3}^S_u=1. The proposed model not only fully utilizes the knowledge correlation between tasks in multi-task learning but also enhances the continuous tracking and recognition ability of dynamic character faces in television programs through spatio-temporal feature modeling, providing high-precision fundamental data support for the subsequent emotion analysis framework.

2.2 Salient feature extraction module

The salient feature extraction module in the model uses ResNet as the backbone network, aiming to precisely capture key recognition cues of character faces in television programs through hierarchical feature extraction and channel-level attention mechanisms. The architecture is shown in Figure 2. Considering that character faces in television frames often face challenges such as low resolution, motion blur, and complex lighting, the deep convolution structure of ResNet can extract multi-level feature maps {D^k_u} layer by layer, from low-level features of edge textures to high-level semantic representations. The k-th level feature map is spatially downsampled to G/2^k×Q/2^k to expand the receptive field, and the F-channel dimension corresponds to feature encodings of different semantics. The model initializes backbone network parameters with the ImageNet pre-trained model, quickly adapting to television media scenarios through transfer learning. At the same time, a low-dimensional feature mining strategy is used to retain and enhance detailed features output from shallow layers, providing richer basic information for the subsequent channel attention mechanism.

In the channel attention mechanism processing stage, the module captures the global average response and peak response of the feature map Fji in the channel dimension through global average pooling and max pooling paths, namely D^k_u,AVG and D^k_u,MAX, to generate channel descriptors with different statistical properties. The two descriptors are processed by a shared multilayer perceptron (MLP), and then added element-wise to generate the channel attention vector L^k_u, which has the same dimensionality Z as the number of channels in the feature map. Each element corresponds to the importance weight of one channel. This vector adaptively adjusts the weights of each channel to selectively enhance low-level features: higher weights are assigned to channels that contain key facial regions such as eyes, nose, and mouth, while suppressing the interference from irrelevant channels such as background noise and non-face regions. Assuming the Sigmoid non-linear activation function is denoted as δ( ), and the MLP weights are q₀∈R ^(Z/e)×Z and q₁∈R ^(Z/e)×Z, where e is the reduction ratio. The specific computation is as follows:

$L_u^k=\delta\left(\operatorname{MLP}\left(\operatorname{AvgPool}\left(D_u^k\right)\right)+\operatorname{MLP}\left(\operatorname{MaxPool}\left(D_u^k\right)\right)\right)=\delta\left(q_1\left(q_0\left(D_{u, A V G}^k\right)\right)+q_1\left(q_0\left(D_{u, M A X}^k\right)\right)\right)$             (1)

The channel attention vector L^k_u∈R ^1×1×Z is used to enhance the low-level features. Assuming channel-wise tensor multiplication is denoted as ⊗, the enhancement process is expressed as:

$D_u^{k^{\prime}}=L_u^k \otimes D_u^k$             (2)

Similarly, for the image set Z_U={U_j∈R^G×Q×3}^S_j=1, the same processing steps are used to generate the salient features D^k_u'. In particular, in low-light scenes, the channel attention mechanism can enhance channels sensitive to brightness and contrast to improve the distinguishability of dark facial details; in side-face or occlusion scenarios, it focuses on feature channels corresponding to contours and visible organs to ensure effective extraction of key recognition information. Through this dynamic feature selection mechanism, the salient feature extraction module can provide more discriminative inputs for subsequent multi-task learning, significantly improving the accuracy of face recognition in complex television media scenarios.

2.png

Figure 2. Architecture of salient feature extraction module

2.3 Spatio-temporal attention module

The spatio-temporal attention module of the model addresses the temporal dynamics and complex background interference problems in video sequences of television programs. It enhances the model's continuous recognition capability of character faces by constructing cross-frame spatio-temporal correlations and focusing on salient target regions. The architecture is shown in Figure 3. Different from traditional non-local networks which perform full association computation on Query, Key, and Value, this module constructs spatio-temporal dependencies using only Key and Value, reducing computational complexity. At the same time, through the TopJ operation, the J most relevant attention maps to the current frame are selected to precisely capture motion trajectories and appearance consistency features of character faces in the time dimension. Specifically, the module takes the video sequence features D⁴' output by the deep network as input, which contains S frames of high-level semantic features with a resolution of G/16×Q/16 and Z channels. Two independent convolution modules are used to generate Key and Value features respectively, constructing an inter-frame correlation map X, where each element represents the spatio-temporal correlation between frame t and frame s. The Key and Value query results Θ and φ are expressed as:

$\begin{aligned} & \Theta=C O N V_{1 \times 1}\left(D^{4^{\prime}}\right) \in \Re^{(S G Q / 16 / 16) \times Z} \\ & \phi=C O N V_{1 \times 1}\left(D^{4^{\prime}}\right) \in \Re^{Z \times(S G Q / 16 / 16)}\end{aligned}$             (3)

Assuming tensor multiplication is denoted by *, the expression for constructing correlation map X is as follows:

$X=\Theta^* \Xi \in \Re^{(S G Q / 16 / 16) \times(S G Q / 16 / 16)}$              (4)

The above operation can effectively capture temporal clues such as pose changes and expression evolution of character faces in continuous frames, avoiding interference from irrelevant motion in background regions.

3.png

Figure 3. Spatio-temporal attention module architecture

In the process of correlation map computation, the module first generates the intermediate map C₁ through matrix operations. Then, the TopJ strategy is used to select the J most relevant frames at each time point as references, generating the TopJ map C_TOP. Weighted aggregation is used to obtain intermediate maps C₂ and C₃, and finally generate J attention masks Ψ. These masks can adaptively enhance the spatio-temporal consistency features of character face regions and suppress background noise interference. In particular, when a character face is partially occluded in a certain frame, the spatio-temporal attention module can use the TopJ operation to select the corresponding region features from preceding or succeeding clear frames to compensate for missing information in the current frame. When processing blurry frames caused by fast motion, the module focuses on facial features in adjacent clear frames to maintain stability of recognition features. Specifically, assuming reshape operation is represented by REsha pe( ), tensor mean operation by Mean(·), and normalization coefficient by V, the calculation formulas for intermediate map C₁, TopJ map C_TOP, intermediate maps C₂, C₃, and attention masks Ψ are as follows:

$\begin{aligned} & C_1=\operatorname{REshape}(X) \in \Re^{(S G Q / 16 / 16) \times S \times(G Q / 16 / 16)} \\ & C_{T O P}=\operatorname{Top} J\left(C_1\right) \in \Re^{(S G Q / 16 / 16) \times S \times J} \\ & C_2=\operatorname{Mean}\left(C_{M A X}\right) \in \Re^{(S G Q / 16 / 16) \times 1 \times J} \\ & C_3=\operatorname{SoftMax}\left(\frac{1}{\sqrt{V}} R E\left(C_2\right)\right) \in \Re^{S \times(G Q / 16 / 16) \times J} \\ & \Psi=\operatorname{REshape}\left(C_3\right) \in \Re^{S \times(G / 16) \times(Q / 16) \times J}\end{aligned}$             (5)

The adopted Softmax(·) operation is expressed as:

 $\operatorname{SoftMax}\left(c_u\right)=\frac{r^{c_u}}{\sum_{z=1}^Z r^{c_z}}$             (6)

To reduce background interference, this paper proposes to use J attention masks Ψ to enhance the feature map D^4’. Assuming tensor multiplication along the spatial dimension is denoted by ⊗, and convolution with kernel size 1×1 is denoted by CONV_1×1, and the j-th component of Ψ is denoted by Ψ_j. The specific calculation process is as follows:

$B_j^4=C O N V_{1 \times 1}\left(\Psi_j \otimes D^{4^{\prime}}\right) \in \Re^{S \times G / 16 \times Q / 16 \times Z / J}$             (7)

Finally, spatio-temporal features B⁴ are obtained by concatenation along the channel dimension.

2.4 Decoder module

The proposed decoder module is designed to meet the dual demand for detailed features and semantic information in character face recognition in television programs. It designs dual decoding branches for image and video, and implements knowledge transfer and complementarity among tasks through parameter sharing mechanisms. Considering the possible low resolution and motion blur in character faces in television frames, shallow network outputs retain detailed features such as facial edges, textures, and skin color, while deep features such as D⁴'/B⁴ contain high-level information such as overall facial structure and pose semantics. The decoder module fuses deep semantic features and shallow detail features across layers through shared-parameter image and video decoding branches, avoiding feature bias from a single branch while reducing model complexity through parameter sharing, and enhancing the ability to extract common features from different input modalities in multi-task learning.

In the feature fusion process, the decoder module adopts the strategy of “deep features guiding shallow features”, progressively refining contextual information to generate high-precision saliency maps. For the video decoding branch, the deep feature G⁴ gradually restores spatial resolution through upsampling, and is concatenated or added element-wise with shallow features to inject global structure information into local detail features. Similarly, the image decoding branch uses deep features D⁴' to guide the detail optimization of shallow features, ensuring semantic-level consistency in edge contours of facial organs and texture variations. This top-down feature aggregation process can effectively solve the problem of feature discontinuity caused by resolution variation and shot switching in television media scenarios. Specifically, assuming the upsampling unit is denoted by Upscale(·), and the residual unit by RES(·), the k-th level saliency map output of the u-th frame in the video branch is denoted by G^k_u, and that of the u-th image in the image branch is denoted by T^k_u. The following formula gives the calculation for progressively refining contextual information in the feature fusion process:

$\begin{aligned} & G^3=\operatorname{RES}\left(D^{3^{\prime}}\right)+\operatorname{RES}\left(\operatorname{Upscale}\left(G^4\right)\right) \in \Re^{S \times G / 8 \times Q / 8 \times Z} \\ & G^2=\operatorname{RES}\left(D^{2^{\prime}}\right)+\operatorname{RES}\left(\operatorname{Upscale}\left(G^3\right)\right)  +\operatorname{RES}\left(\operatorname{Upscale}\left(G^4\right)\right) \in \Re^{T \times G / 4 \times Q / 4 \times Z} \\ & G^1=\operatorname{RES}\left(D^{1^{\prime}}\right)+\operatorname{RES}\left(\operatorname{Upscale}\left(G^2\right)\right)  +\operatorname{RES}\left(\operatorname{Upscale}\left(G^4\right)\right) \in \Re^{S \times G / 2 \times Q / 2 \times Z}\end{aligned}$              (8)

In particular, when processing side-face sequences, the pose semantics provided by deep features can assist shallow features in restoring occluded facial contour details. In low-light scenes, semantic-level face region localization can guide shallow features to enhance contrast-sensitive channels and improve recognition of dark texture details. Through progressive feature fusion and contextual refinement, the saliency maps output by the decoder module can precisely locate character face regions while retaining local discriminative features crucial for recognition tasks, providing feature representations with rich hierarchy and strong robustness for the final face recognition decision in the multi-task learning framework.

2.5 Multi-task loss

The proposed multi-task loss function addresses the complex requirements of character face recognition in television programs. Through joint optimization of the image and video saliency target detection tasks and the core recognition task, a hierarchical supervision system is constructed to ensure that the model achieves a balance between feature sharing and task specificity. Considering that television media data includes both single-frame images and continuous video segments, the model designs dual-task outputs for the image branch and the video branch to generate saliency masks at the corresponding scales. The loss function is based on binary cross-entropy (BCE), performing pixel-level supervision on the saliency map to force the model to precisely locate facial regions and suppress background noise interference. Specifically, the image branch loss LOSS_TPF and video branch loss LOSS_NTPF respectively calculate the BCE between the predicted mask and the ground truth facial mask, ensuring the segmentation accuracy of facial regions under two input modalities and providing high-quality region of interest (ROI) features for subsequent recognition tasks. Assuming the weighting coefficient is represented by β, the loss function is defined as:

$L O S S=L O S S_{N T P F}+\beta L O S S_{T P F}$               (9)

Assuming the ground truth of the saliency map is H∈{0,1}, and the prediction is T^k_u∈[0,1], the BCE loss is defined as:

$\operatorname{LOSS}_{Y Z R}(H, T)=H \cdot L O G T+(1-H) L O G(1-T)$               (10)

At the level of multi-task joint optimization, the loss function further integrates the identity classification loss of the face recognition task, forming a dual supervision mechanism of “saliency target detection + identity recognition”. For the image branch, the model performs identity classification on the deep features of the facial ROI extracted based on the saliency map segmentation. The video branch combines the temporal features output by the spatio-temporal attention module to realize dynamic identity recognition by aggregating cross-frame facial representations. The total loss is defined by weighted summation as shown in the following formula, which is used to balance the optimization priorities between region localization and identity recognition:

$\begin{aligned} & L O S S=\sum_{u=1}^S \sum_{k=1}^4 \alpha_{u k} \operatorname{LOSS}_{Y Z R}\left(H_{N T P F}, T_u^k\right)  +\beta \sum_{u=1}^S \sum_{k=1}^4 \alpha_{u k} \operatorname{LOSS}_{Y Z R}\left(H_{T P F}, T_u^k\right)\end{aligned}$               (11)

Facing challenges such as occlusion and blur common in television media scenarios, the loss function strengthens the model's robust localization capability of incomplete facial regions through saliency map supervision, and at the same time drives the model to mine discriminative deep features through identity classification loss, avoiding degradation of recognition features caused by overly simplified saliency detection tasks. The multi-task collaborative optimization mechanism adopted by the model not only provides accurate pre-processing information for the recognition task via saliency detection, but also feeds back high-level semantics from the recognition task to the feature extraction module, forming a closed-loop supervision of “bottom-up localization—top-down recognition”, ultimately improving the comprehensive recognition performance of the model in complex television media environments.

As can be clearly seen from the data in Table 1, the performance differences of different loss functions on television media-related datasets are significant. On the known dataset FePh, the BCE loss function (83.98) slightly outperforms the proposed loss function (83.56), but in the dynamic video dataset CMU-MOSEI and the self-built video set, the advantage of the proposed method is obvious: on CMU-MOSEI, the proposed loss function (58.69) is higher than BCE (57.25), and in the self-built video set (56.98) it significantly outperforms the image saliency prediction loss function (55.48) and BCE (55.21). Under the self-built image+video set, the proposed loss function (43.56) improves by 1.31 compared to BCE (42.25), and nearly 9 compared to the video saliency prediction loss function (34.58), demonstrating strong fusion capability for multimodal data.

Further analysis shows that the video saliency prediction loss function performs the worst in dynamic video scenarios due to the lack of effective modeling of temporal features; although the image saliency prediction loss function (61.58) performs excellently on static images (FePh 83.26), its performance drops sharply on dynamic video and hybrid data, exposing its insufficient adaptability to dynamic scenes. In contrast, the proposed loss function, through the multi-task learning framework, shares spatiotemporal and appearance features between face recognition and emotion analysis tasks, achieving stable and improved performance on dynamic video and hybrid data, proving that multi-task learning effectively enhances the model's robustness in complex television media scenarios.

The ablation study data in Table 2 clearly demonstrate the critical impact of the three core modules (salient feature extraction, spatio-temporal attention, and decoding modules) on model performance. When only the salient feature extraction module is enabled, the model performs reasonably on the static image dataset FePh (83.26), but performs poorly on the dynamic video dataset CMU-MOSEI (43.25) and the hybrid dataset (81.56), exposing the insufficient adaptability of single feature extraction to dynamic scenes. As the spatio-temporal attention module and the decoding module are gradually added, the performance on the dynamic dataset CMU-MOSEI first decreases and then increases, indicating that spatio-temporal modeling must be combined with salient features to be effective. Ultimately, when all three modules are enabled, performance on all datasets reaches optimal values: FePh improves by 2.97, CMU-MOSEI increases by 8, and the hybrid dataset improves by 2.06. This process proves that module collaboration under the multi-task learning framework is the core mechanism for overcoming the complexity of television media scenarios.

Specifically, in the CMU-MOSEI dynamic video dataset, the collaboration of the spatio-temporal attention module and the decoding module increases performance from 42.58 to 51.23, an increase of 20.3%, verifying the strong adaptability of temporal features and multi-task fusion to dynamic scenarios. In the hybrid dataset, when all modules are enabled (83.62), it improves by 2.06 compared to the single module case (81.56), indicating that the model integrates static and dynamic features through multi-task learning, achieving cross-modal generalization and solving the problem of performance fragmentation under multimodal data in traditional methods.

The data in Table 3 visually presents the effect of fixed weights and learnable parameters on the performance of the algorithm in face recognition. In the static dataset FePh, the fixed weight of 0.3 achieves a peak value of 88.97, while the learnable parameters are slightly lower. However, on the dynamic dataset CMU-MOSEI, the learnable parameters are all higher than the fixed values, reflecting stronger adaptability to dynamic scenes. In the self-built image+video hybrid dataset, the learnable parameter β (43.56) shows more stable performance on dynamic data than the fixed values and avoids the fluctuations of fixed weights under multimodal data. This indicates that the weight adaptive mechanism under the multi-task learning framework can automatically optimize the task-shared weight allocation according to the complex scenes of television media, breaking through the performance bottleneck of fixed weights in dynamic and multimodal scenarios.

Further analysis shows that fixed weights are locally optimal in static scenes like FePh, but perform poorly in dynamic scenes like CMU-MOSEI and multimodal scenarios, exposing their lack of adaptability to complex contexts. Learnable parameters dynamically adjust the weight distribution between face recognition and auxiliary tasks via backpropagation in multi-task learning, achieving stable performance improvements on CMU-MOSEI and hybrid datasets. For example, in the hybrid dataset, β = 43.56, although lower than the local peak value of the fixed setting (51.23), it is more robust in dynamic frame processing, proving that multi-task learning can capture the complementary relationship between spatiotemporal dynamics and appearance features, enhancing the generalization ability of the model in complex television media scenarios.

Table 1. Impact of different loss functions on different datasets

Table 2. Ablation study of the three core modules on different datasets

Table 3. Effect of fixed and adaptive weight vectors on algorithm face recognition performance

The data in Table 4 visually presents the performance differences between the proposed method and comparison algorithms on television media datasets. In the static dataset FePh, the proposed method achieves an accuracy of 61.23%, which is 8.87% higher than SST-ResNet (52.36%), and outperforms methods such as DAN (56.97%) and STACNN (57.82%), proving the effectiveness of multi-task learning in extracting static facial features. In the CMU-MOSEI dynamic dataset, the proposed method achieves a mean rate of 51.28% and accuracy of 62.31%, which are 10.03% and 6.62% higher than Baseline (41.25% / 55.69%) respectively, significantly surpassing dynamic modeling methods such as TCN (43.69% / 56.34%), highlighting strong adaptability to dynamic television media scenarios (such as facial expression changes, camera switching).

Table 4. Comparison of face recognition methods on known datasets

Table 5. Cross-dataset evaluation results of different face recognition methods

Further analysis shows that in the FePh data, the proposed method enhances the discriminability of static image features by integrating appearance features from face recognition and emotion analysis via multi-task learning. In the CMU-MOSEI dynamic data, the spatiotemporal attention module in the multi-task learning effectively captures facial motion trajectories across frames, addressing the limitations of traditional dynamic models in multimodal feature fusion. For example, in scenes where guests turn their heads during interviews, the proposed method can accurately recognize the facial identity in each frame through inter-task feature sharing, while the comparison algorithms suffer accuracy decline on dynamic frames due to lack of spatiotemporal-emotion collaboration.

The data in Table 5 visually presents the significant advantages of the proposed method in cross-dataset scenarios. In the self-built image+video hybrid dataset, the proposed method achieves an accuracy of 92.86%, far surpassing comparison methods such as ST-ATTNet (82.36%) and MCTransformer (84.52%), indicating its outstanding fusion capability for multimodal data. In the cross-dataset evaluation between FePh and CMU-MOSEI, the proposed method significantly outperforms baseline methods in both Scheme 1 (FePh → CMU-MOSEI, 51.23%) and Scheme 2 (CMU-MOSEI → FePh, 82.51%), demonstrating strong generalization ability to different television scenarios.

Specifically, the self-built hybrid dataset simulates multimodal features in practical television media applications. The proposed method integrates the spatiotemporal and appearance features of face recognition and emotion analysis through multi-task learning, achieving efficient feature transfer across modalities. For example, in news programs with static covers and dynamic main segments, the model utilizes shared features between tasks to accurately recognize cross-scene facial identities, solving the problem of performance collapse of traditional methods in hybrid data. In Scheme 2, the proposed method (82.51%) improves by 3.55% over the baseline (78.96%), verifying that multi-task learning enhances static feature representations, ensuring stable transfer between static and dynamic scenes.

Figure 5 presents the dynamic emotional trajectory of characters in television programs, with time on the horizontal axis and emotion categories on the vertical axis. In the curve, the emotion rapidly switches from “happy (3)” to “neutral (7)” and “surprised (6)” within frames 0-200, then transitions to “sad (4)” after stabilizing at “neutral (7)” from frame 200-400, reflecting the framework’s precise capture of instantaneous expression changes (e.g., surprised) and sustained emotional states (e.g., neutral). This dynamic tracking capability originates from the core design of the emotion analysis framework proposed in this paper: based on face recognition results, it integrates the spatiotemporal attention module and appearance feature extraction via multi-task learning. For example, recognition of “surprised (6)” relies on dynamic changes like pupil dilation and eyebrow raising. The framework aggregates features across consecutive frames through the spatiotemporal attention module to sensitively respond to such instantaneous emotions. Meanwhile, recognition of “sad (4)” combines appearance features like drooping facial contours and dim eye expressions with temporally low emotional features, verifying the effectiveness of multi-dimensional feature fusion.

Compared with traditional emotion analysis methods, the proposed framework has significant advantages in temporal emotion tracking. For instance, at the emotion mutation around frame 200, traditional methods may misclassify it as neutral due to a lack of temporal modeling, while the proposed framework accurately identifies it by capturing inter-frame facial motion differences through the spatiotemporal attention module. This capability is critical for television media scenarios, where character emotions often change dynamically with program content. The temporal tracking ability of the framework ensures real-time and accurate emotion analysis, providing essential data support for both program producers and viewers.

5.png

Figure 5. Temporal variation of character emotions in television programs

6-1.png	6-2.png
(a) FePh dataset	(b) CMU-MOSEI dataset
6-3.png	6-4.png
(c) Self-built video dataset	(d) Self-built image+video dataset

Figure 6. Confusion matrix results of the proposed method on different datasets

The confusion matrix in Figure 6 clearly presents the emotion classification performance of the proposed method on different datasets. In the FePh dataset (1), the recall rate for the "happy" class reaches as high as 95.61% and 92.50% for the "neutral" class, showing excellent recognition ability for positive and stable emotions. In the CMU-MOSEI dataset (2), "happy" is 86.00% and "sad" is 85.40%, reflecting the classification accuracy for complex emotions in dynamic video scenes. In the self-built video dataset (3), "happy" reaches 93.15%, and "sad" is 58.90%, validating the adaptability to dynamic content in television media. In the self-built image+video hybrid dataset (4), "happy" is 77.64% and "sad" is 64.64%, demonstrating robustness under multimodal data.

Comparing the datasets, the model generally achieves high recall rates for high-frequency emotions, such as "neutral" at 92.50% in FePh and "happy" at 93.15% in the self-built video dataset. This is attributed to the feature sharing between face recognition and emotion analysis in multi-task learning, which enhances the extraction of appearance stability and dynamic features. For low-frequency emotions, the model reduces cross-class misclassification through the fusion of spatiotemporal attention and appearance features. For example, in FePh, "sad" is misclassified as "happy" at a rate of 2.09%, but in the CMU-MOSEI dynamic dataset, "sad" is correctly recognized at 59.40%, proving the crucial role of spatiotemporal modeling in low-frequency emotion detection. This capability ensures accurate detection of implicit emotions in television programs, overcoming the limitations of traditional methods.

In summary, the experimental results show that the proposed emotion analysis framework integrates multi-dimensional features of face recognition and emotion analysis through multi-task learning, achieving high precision and strong robustness in emotion classification across multiple television media scenarios. Whether on static, dynamic, or multimodal data, the framework can effectively recognize both high-frequency and low-frequency emotions, providing core technical support for emotion analysis in television programs. This fully validates its practicality and effectiveness in the television media field and highlights the unique advantages of the multi-task learning framework based on face recognition results in emotion analysis.