Modeling Internal Nonverbal Communication Patterns and Predicting Team Performance via Video-Based Behavioral Representation Learning

2.1 Overview of the research framework

An end-to-end, multi-task, multimodal, multi-granularity CNN framework is developed with the objective of integrating multimodal video data, accurately decoding latent nonverbal communication patterns, and achieving efficient team performance prediction. The overall architecture comprises four core components: multi-granularity feature extraction, latent variable decoding of nonverbal communication, dual-task output, and multi-task optimization. The central processing pipeline is structured below. Multimodal video data, consisting of RGB video and skeletal data are provided as inputs. Individual- and team-level skeletal features, together with frame- and clip-level RGB features, are extracted through the proposed network architecture. Latent variables representing core nonverbal communication patterns are subsequently decoded at intermediate layers. Finally, a dual-output layer is employed to simultaneously generate team performance predictions and reconstructed nonverbal behavior representations. To strengthen the intrinsic associations between features and the ultimate objective of performance prediction, a multi-task joint optimization strategy is adopted. The primary task corresponds to team performance prediction, with an associated loss function denoted as L_main, while the auxiliary task corresponds to self-supervised reconstruction of nonverbal behaviors, with an associated loss function denoted as L_aux. The overall loss function is defined as:

$\text{L= }\!\!\lambda\!\!\text{ }{{\text{L}}_{\text{main}}}\text{+(1- }\!\!\lambda\!\!\text{ )}{{\text{L}}_{\text{aux}}}$    (1)

where, λ is set to 0.7 and determined via cross-validation to balance the relative priorities of the primary and auxiliary tasks. This ensures that the latent variables are capable of capturing essential nonverbal communication information while effectively serving the team performance prediction task.

2.2 Dataset construction and preprocessing

Well-established collaborative task paradigms from organizational behavior research were adopted to ensure the authenticity of the collected nonverbal communication data. The task set encompassed creative decision-making scenarios (the winter survival task), problem-solving scenarios (the moon survival task), and execution-oriented scenarios involving real corporate project collaboration. Data collection was conducted across both laboratory-simulated collaboration settings and authentic corporate meeting room environments. A total of 62 teams were recruited, with team sizes ranging from 3 to 8 members. Team compositions spanned three major industry sectors—technology, education, and finance—yielding a cumulative effective data duration of 128 hours. During data acquisition, multimodal information was synchronously recorded, including RGB video at a resolution of 1080p and 30 frames per second (fps), three-dimensional skeletal data comprising 25 keypoints extracted via MediaPipe, and auxiliary audio recordings retained as backup signals. To preserve the ecological validity of the dataset, task designs were closely aligned with real-world organizational workflows. Standardized project requirement documents and realistic time constraints were incorporated, and team compositions were structured to reflect typical corporate role distributions, including leaders, executors, and coordinators. The data collection process adhered to a minimal-intervention principle; concealed camera setups were employed to mitigate behavioral distortion and to ensure the naturalistic expression of nonverbal communication.

A multidimensional performance labeling system was constructed based on the input-process-output framework, and a triangulation approach was employed to ensure label reliability. The labeling scheme comprised three core dimensions. The task performance dimension was defined using objective indicators, including task completion rate, decision accuracy, and execution efficiency. The team vitality dimension integrated objective and subjective measures, encompassing the variance in member speaking frequency and the mean level of emotional positivity derived from facial expression recognition. The member satisfaction dimension was obtained through subjective assessments, including post-task peer evaluations of collaboration quality and communication fluency, as well as self-reported satisfaction scores provided by participants. Labels across all dimensions were integrated using a weighted averaging method to produce a composite performance score normalized to the [0,1] interval. Weight assignments were determined by organizational behavior experts according to the relative contribution of each dimension to overall team performance. The annotation process was jointly conducted by three organizational behavior experts and two technical engineers. Inter-annotator reliability was evaluated using Krippendorff’s α, with all coefficients exceeding 0.88, thereby satisfying established reliability standards for empirical research.

During skeletal data preprocessing, median filtering was applied for noise reduction, and missing frames were completed via linear interpolation. Individual skeletal sequences and team-level spatial position matrices were subsequently constructed to meet model input requirements. For RGB video preprocessing, frame sampling at 15 fps was first performed to balance data volume and computational efficiency. Face and body regions were detected using the You Only Look Once version 8 (YOLOv8) model, followed by region-of-interest cropping to remove background interference and to focus on core interaction areas. To enhance model generalization and to accommodate auxiliary task training, targeted data augmentation strategies were implemented, including random frame flipping, temporal sequence shuffling, and adjustments to brightness and contrast. All preprocessing procedures were designed to preserve the integrity and authenticity of the original nonverbal communication characteristics.

2.3 Detailed design of the proposed model

2.3.1 Multi-granularity feature extraction module

The primary objective of the multi-granularity feature extraction module is to accommodate the hierarchical structure of team nonverbal communication by capturing core representations of individual-team-level features and frame- and clip-level features from both skeletal and RGB modalities, providing fine-grained representations for subsequent interaction modeling and team performance prediction. For the skeletal branch, a dual-granularity design is adopted. The individual skeletal keypoint subnetwork is constructed based on Spatial-Temporal Graph Temporal Convolutional Networks (S-GTCNs). The input consists of temporal sequences with length T, each containing 25 three-dimensional skeletal keypoints. Through the combined application of temporal convolutions and graph convolutions, dynamic body motion patterns of individual members are extracted. The output is an individual-level feature matrix F_ind of dimensionality T × d, where the feature dimensionality d is set to 256 to balance representational capacity and computational efficiency.

Figure 1. Temporal feature extraction module for team nonverbal communication in the RGB modality

The team interaction skeletal subnetwork is designed to capture interaction-related features among team members. A graph structure is constructed based on pairwise spatial distances between members, in which nodes correspond to individual team members and edge weights are defined as the inverse of inter-member spatial distances, such that closer proximity corresponds to stronger interaction association. Feature aggregation is performed over this graph using a GCN, enabling effective capturing of team-level interaction patterns, including bodily coordination and spatial orientation consistency. The resulting team-level interaction representation is denoted as F_team, matching the dimensionality (T × d) of the individual-level features and achieving hierarchical coverage of skeletal representations from individual actions to collective team interactions.

The RGB branch likewise adopts a dual-granularity design to jointly capture static visual attributes and temporal dynamics. The frame-level feature subnetwork is constructed based on the I3D architecture. The input consists of T frames of three-channel RGB images. Through three-dimensional convolution operations, frame-level visual features are extracted, encompassing static nonverbal cues such as facial expressions and body postures. The resulting output is a frame-level feature representation F_frame of dimensionality T × d. The clip-level temporal subnetwork is implemented using a three-layer Temporal Convolutional Network (TCN), with the dilation factor set to 2 to expand the receptive field. The frame-level feature sequence is provided as input, and temporal convolutions are applied to capture behavioral evolution across consecutive frames, thereby extracting temporal dynamics of nonverbal communication. The final output is a clip-level temporal feature representation F_seq of dimensionality T × d, completing multi-granularity feature extraction in the RGB modality from static frames to dynamic temporal segments. Figure 1 provides an overview of the RGB-based team nonverbal feature extraction module. RGB video streams from team interactions are processed through three steps: dividing into temporal segments, extracting frame-level nonverbal features, and applying clip-level temporal enhancement. This produces multi-level representations of team nonverbal communication in the RGB modality.

2.3.2 Cross-member attention mechanism

The primary function of the cross-member attention mechanism is to quantify the dynamic association strength of nonverbal behaviors among team members, thereby amplifying interaction features relevant to performance while suppressing interference from behaviorally irrelevant individuals. To achieve this objective, preliminary feature fusion is first performed for each member. Specifically, individual skeletal features F_{ind_i}^t and frame-level visual features F_{frame_i}^t are concatenated along the feature dimension, yielding a composite feature vector f_i^t for member i at time step t. The resulting feature dimensionality is two-dimensional, preserving essential individual behavioral information while providing a comprehensive feature basis for computing interaction dependencies among team members.

The computation of inter-member attention weights is formulated using a scaled dot-product attention mechanism. The attention weight $\alpha _{ij}^{t}$, representing the influence of member i on member j at time t, is defined as:

$\alpha _{ij}^{t}=softmax\left( \frac{f_{i}^{t}\cdot {{W}_{a}}\cdot {{\left( f_{i}^{t} \right)}^{T}}}{\sqrt{2d}}+{{b}_{a}} \right)$    (2)

where, W_a denotes a learnable weight matrix of dimensionality 2d × 2d used to model feature associations between members, b_a represents a bias term, and (2d)^1/2 serves as a scaling factor to mitigate gradient vanishing issues induced by increasing feature dimensionality. The softmax function normalizes weights to the [0,1] interval, ensuring that their sum equals unity and enabling quantification of relative interaction strengths.

To balance the contributions of individual features and interaction-driven features, residual connections are incorporated into the cross-member feature aggregation process. The equation is as follows:

$f_{att,i}^{t}\text{=}\underset{\text{j=1}}{\overset{\text{M}}{\mathop \sum }}\,\text{ }\!\!\alpha\!\!\text{ }_{ij}^{t}f_{j}^{t}\text{+ }\!\!\gamma\!\!\text{ }f_{i}^{t}$     (3)

where, M denotes the number of team members, and γ is set to 0.5 as the residual weighting coefficient. This design ensures that critical individual behavioral characteristics are retained while inter-member interaction dependencies are effectively emphasized. Through the dynamic learning of attention weights, the proposed mechanism adaptively focuses on task-critical interaction behaviors, such as gestural coordination during creative discussions and gaze convergence during decision-making phases. As a result, more targeted and interaction-aware representations are produced for subsequent feature fusion.

2.3.3 Multi-granularity feature fusion

The primary objective of multi-granularity feature fusion is to integrate four complementary feature dimensions across skeletal and RGB modalities, thereby generating a comprehensive representation that preserves hierarchical completeness while emphasizing interaction relevance. This integrated representation serves as a high-quality input for subsequent decoding of latent nonverbal communication variables. An adaptive weighted fusion strategy is adopted in place of fixed-weight fusion, enabling the model to automatically learn dynamic feature weights that align with the requirements of team performance prediction. Weight allocation is designed to be positively correlated with each feature’s predictive contribution to performance outcomes.

The fusion process is formulated as follows:

$F_{\text{fusion}}^{t}={{w}_{1}}\cdot F_{\text{ind}}^{t}+{{w}_{2}}\cdot F_{\text{team}}^{t}+{{w}_{3}}\cdot F_{\text{frame}}^{t}+{{w}_{4}}\cdot F_{\text{seq}}^{t}$    (4)

where, w₁, w₂, w₃, and w₄ denote the dynamic weights associated with individual skeletal features, team interaction features, frame-level visual features, and clip-level temporal features, respectively. Each weight is learned via a sigmoid activation function according to:

${{w}_{k}}=\operatorname{sigmoid}\left( {{W}_{w}}\cdot F_{k}^{t}+{{b}_{w}} \right)$    (5)

where, W_w represents a learnable weight matrix of dimensionality d × d that captures the relationship between each feature type and the performance prediction objective, while b_w denotes a bias term. The sigmoid function constrains weight values to the [0,1] interval, allowing each feature’s contribution to be quantitatively interpreted while enabling adaptive normalization across feature dimensions.

A key advantage of this fusion strategy lies in its ability to dynamically adjust feature importance across task types and collaboration phases. For example, during brainstorming stages of creative tasks, higher weights are automatically assigned to clip-level temporal features and cross-member interaction features, whereas during execution-oriented stages, increased emphasis is placed on individual skeletal features and frame-level visual features. Through this adaptive mechanism, the fused representation F_fusion^t is guided to focus on context-specific nonverbal communication cues, effectively leveraging the complementary strengths of multimodal and multi-granularity features. This design establishes a robust foundation for latent variable decoding and subsequent team performance prediction.

2.3.4 Latent variable decoding of nonverbal communication

The latent variable decoding module for nonverbal communication serves as the central intermediate layer connecting multi-granularity fused features to the team performance prediction task. Its primary objective is to automatically distill low-dimensional latent variables that characterize team nonverbal communication patterns from high-dimensional fused representations, thereby avoiding subjective bias introduced by expert-defined constructs and enabling data-driven quantification of communication patterns. The dimensionality of the latent variables directly affects representational capacity and interpretability. Based on extensive cross-validation experiments, the latent dimensionality k is set to 64, a configuration that sufficiently captures essential communication information while mitigating overfitting and preserving interpretability.

Latent variable decoding is implemented using two fully connected layers. The latent representation at time step t, denoted as Z^t, is computed as:

${{Z}^{t}}=\operatorname{ReLU}\left( {{W}_{z}}\cdot F_{\text{fusion}}^{t}+{{b}_{z}} \right)$    (6)

where, W_z represents a learnable weight matrix of dimensionality d × k, and b_z denotes a k-dimensional bias term. The ReLU activation function introduces nonlinearity to enhance the model’s capacity to represent complex communication patterns. The resulting latent variable matrix Z consists of T × k dimensions that correspond to core nonverbal communication factors automatically learned by the model, such as bodily synchrony, balance of attention allocation, and emotional positivity. These dimensions are not predefined and are entirely induced from data, allowing adaptive capture of task-specific key communication characteristics across diverse team collaboration scenarios.

To promote independence and interpretability across latent dimensions and to prevent information redundancy, an L2 regularization constraint is incorporated during the decoding process. The regularization term is defined as λ_z||Z||₂², where λ_z is set to 1e^-5. By penalizing the L2 norm of the latent variables, this constraint encourages the learning of sparse and relatively independent communication dimensions, thereby clarifying the semantic meaning of each latent factor, supporting subsequent interpretability analyses and further enhancing model generalization capability.

2.3.5 Multi-task output head design

The multi-task output head is designed under a joint optimization framework comprising a primary task and an auxiliary task. The primary task focuses on team performance prediction, while the auxiliary task constrains latent variable quality through self-supervised reconstruction of nonverbal behaviors. The collaborative optimization of these tasks is intended to jointly enhance prediction accuracy and feature representation capability.

The primary task, corresponding to the team performance prediction head, receives two types of inputs: the temporal latent variable features Z and team attribute features A. Team attribute features include the number of members, task type, and role composition. After one-hot encoding and normalization, the attribute feature dimensionality is denoted as m. These attributes are jointly fed with the latent variable features to complement critical non-behavioral information. To capture the dynamic evolution of communication patterns across task phases, a bidirectional Long Short-Term Memory (BiLSTM) layer is employed for temporal modeling. Through the combined operation of forward and backward LSTM units, temporal dependencies are effectively extracted. The resulting temporally fused feature representation is denoted as H of dimensionality T × 2k and is computed as:

$H={BiLSTM}(Z)$    (7)

To emphasize the differential contribution of critical task phases to overall performance, temporal attention-weighted pooling is applied to H. Temporal attention weights β_t are obtained via softmax normalization:

${{\beta }_{t}}=\operatorname{softmax}\left( {{W}_{\beta }}{{H}^{t}}+{{b}_{\beta }} \right)$    (8)

where, W_β denotes a learnable weight matrix of dimensionality 2k × 1, and b_β represents a bias term. The features obtained after weighted pooling are passed through a fully connected layer followed by a sigmoid activation function to produce a composite team performance score P in the range [0,1].

$P=\operatorname{sigmoid}\left( {{W}_{p}}\cdot \left( \sum\limits_{\text{t}=1}^{\text{T}}{{{\beta }_{\text{t}}}}{{H}^{t}} \right)+{{b}_{p}} \right)$    (9)

The loss function for the primary task adopts the mean squared error (MSE) formulation, which is well suited for continuous-valued performance labels. The primary task loss is defined as:

${{L}_{\text{main}}}=\frac{1}{N}\sum\limits_{i=1}^{N}{{{\left\| {{P}_{i}}-{{Y}_{i}} \right\|}^{2}}}$    (10)

where, N denotes the number of samples, and Y_i represents the ground-truth performance label.

The auxiliary task corresponds to a self-supervised nonverbal behavior reconstruction head, whose primary objective is to constrain the latent variable representation Z to preserve essential communication information by reconstructing original nonverbal behavior features. This design enhances the robustness and effectiveness of feature representations. The reconstruction targets include individual skeletal features F_ind and frame-level RGB features F_frame, as these features directly reflect fundamental nonverbal behavioral cues. The reconstruction network is composed of two transposed convolutional layers followed by a fully connected layer. Transposed convolutions are employed to restore spatial feature structures, while the fully connected layer is used to ensure dimensional alignment. The network outputs reconstructed features ${{\hat{F}}_{ind}}$ and ${{\hat{F}}_{\text{frame}}}$. To improve robustness to outliers, the auxiliary task employs an L1 loss function, defined as:

${{L}_{\text{aux }}}=\frac{1}{N\cdot T}\left( {{\left\| {{\widehat{\mathbf{F}}}_{ind}}-{{\mathbf{F}}_{ind}} \right\|}_{1}}+{{\left\| {{\widehat{\mathbf{F}}}_{\text{frame}}}-{{\mathbf{F}}_{\text{frame}}} \right\|}_{1}} \right)$    (11)

The overall loss function of the model is defined as a weighted sum of the primary task loss and the auxiliary task loss, and is computed as follows:

${{\text{L}}_{\text{total}}}\text{= }\!\!\lambda\!\!\text{ }{{\text{L}}_{\text{main}}}\text{+(1- }\!\!\lambda\!\!\text{ )}{{\text{L}}_{\text{aux}}}$    (12)

where, λ denotes the balancing coefficient between the two tasks. The value is determined via grid search and is set to 0.7. This configuration preserves the central objective of team performance prediction while enabling the auxiliary task to effectively regularize latent variable quality, thereby achieving coordinated optimization of both tasks.

2.4 Model training and optimization details

Model training was conducted using the AdamW optimizer to achieve efficient convergence while incorporating parameter regularization. The weight decay coefficient was set to 1e^-5, effectively mitigating overfitting by suppressing excessive parameter growth. The initial learning rate was configured as 1e^-4, and a cosine annealing scheduling strategy was employed. Under this strategy, the learning rate was decayed to one-tenth of its current value every ten training epochs, thereby preserving exploratory capacity during early training stages while enabling gradual stabilization and convergence in later phases. During training, the batch size was set to 8, balancing GPU memory constraints with gradient estimation stability. The total number of training epochs was fixed at 100, supplemented by an early stopping mechanism. Training was terminated when the mean absolute error (MAE) on the validation set exhibited no improvement for 15 consecutive epochs, thereby preventing redundant iterations and reducing the risk of overfitting. Parameter initialization was performed using the Xavier uniform distribution scheme to ensure consistent variance across layer inputs and outputs, while all bias terms were initialized to zero to provide a stable starting point for training. With respect to regularization, in addition to the L2 constraint applied within the latent variable decoding module, dropout layers with a probability of 0.3 were introduced in fully connected layers and feature fusion modules. By randomly deactivating a subset of neurons during training, model generalization capability was further enhanced, ensuring robust performance across diverse datasets and scenarios. An overview of the proposed framework for team nonverbal communication decoding and performance prediction is illustrated in Figure 2.

Figure 2. Schematic overview of the proposed framework for team nonverbal communication decoding and performance prediction

This study addressed the longstanding challenge in organizational management of objectively quantifying team collaboration states by integrating computational behavioral science with organizational behavior theory. An end-to-end, multi-task, multimodal, multi-granularity CNN framework was proposed, through which nonverbal communication patterns were automatically induced and team performance was accurately predicted via the coordinated design of multi-granularity feature extraction, cross-member attention mechanisms, and latent variable decoding. The results demonstrated that the proposed model achieved significantly higher performance prediction accuracy than existing baseline methods on both self-constructed and public datasets, with all three core hypotheses quantitatively validated. Ablation experiments and SHAP-based analyses further confirmed the necessity of the key architectural components and the representational superiority of the learned latent variables. The contributions of this study are manifested across three dimensions. At the technical level, the application boundary of video-based behavioral representation learning is extended to multi-agent team interaction scenarios. At the theoretical level, objective quantitative evidence is provided for coordination theory and team diversity theory within organizational behavior research. At the practical level, an implementable technological solution is offered for team collaboration diagnosis and performance prediction in organizational settings.

Despite these advances, several limitations remain. The dataset coverage is primarily concentrated in technology-oriented industries, and emerging work settings such as remote collaboration have not yet been incorporated. Audio modality information has not been integrated, potentially omitting complementary nonverbal cues such as prosody and vocal tone. In addition, the proposed framework has not yet been deployed as an operational management tool, and validation in real corporate environments is still lacking. Future research may be advanced along three directions. First, the multimodal fusion framework may be extended to incorporate audio signals and physiological data in order to enrich the dimensionality of communication features. Second, more advanced temporal modeling strategies may be introduced to enable real-time tracking and prediction of team performance. Third, lightweight enterprise collaboration analysis tools may be developed and deployed in real organizational contexts to validate practical value, while further expanding dataset diversity across scenarios and industries to enhance generalizability and applied relevance.