Fusing Multi-Channel Perceptual Features for Visual Communication Image Representation Learning and Interpretability Analysis

2.1 Problem formalization

The multimodal semantic features of visual communication images stem from three core information carriers. First, the formal definitions of each modality’s features are provided. Let the visual semantic features of the input visual communication image be V$\in$R^H×W×C, where H and W are the height and width of the image, and $C$ is the number of visual feature channels, encoded by convolutional neural networks or visual Transformers, encompassing low-level textures, color distribution, and high-level semantic information; the text semantic features T$\in$R^L×D correspond to the embedded textual content in the image, where L is the text sequence length, and D is the text encoding dimension, generated by pre-trained language models, merging textual semantics with visual layout attributes; the design metadata semantic features M$\in$R^K are low-dimensional structured vectors, where K is the metadata dimension, containing design attributes such as layout type, color tone, etc. The three types of modality features are integrated into a multimodal input X={V,T,M}, and subsequent fusion and representation learning are based on this multimodal input.

The core goal of CERL is to learn a structured representation R from the multimodal input X, and decouple it into task-related causal factors ${{R}_{c}}$ and task-independent style factors ${{R}_{s}}$, i.e., $R=\left( {{R}_{c}},{{R}_{s}} \right)$. This goal can be formalized through two key constraints: First, the causal factors must retain the core information of task decisions, satisfying $P\left( Y\mid {{R}_{c}},{{R}_{s}} \right)=P\left( Y\mid {{R}_{c}} \right)$, which indicates that the task label $Y$ is determined only by the causal factor; second, the causal factors must remain invariant under style interventions, i.e., $P\left( {{R}_{c}}\mid \text{do}\left( {{R}_{s}}=r_{s}^{\text{ }\!\!'\!\!\text{ }} \right) \right)=P\left( {{R}_{c}} \right)$, where do( ) is an intervention operation and $r_{s}^{\text{ }\!\!'\!\!\text{ }}$ is any style factor value. This constraint ensures the robustness of the representation to style changes. At the same time, the model must output concept attribution weights α$\in$R^N and generative explanation examples S, where the former quantifies the contribution of each concept to task decisions, and the latter visually presents the core concepts, jointly supporting the interpretability analysis.

To rigorously characterize the above causal relationships, a SCM M=⟨U,V,F,P(U)⟩ is introduced as the theoretical basis. Here, $U$ is the set of exogenous variables, representing unobservable noise and confounding factors; $V=\left\{ X,{{R}_{c}},{{R}_{s}},Y \right\}$ is the set of endogenous variables, including multimodal input, the two types of factors, and task labels; $F=\left\{ {{f}_{X}},{{f}_{{{R}_{c}}}},{{f}_{{{R}_{s}}}},{{f}_{Y}} \right\}$ is the set of causal mechanisms, which defines the mappings $U\to X$, $X\to {{R}_{c}}$, $X\to {{R}_{s}}$, and ${{R}_{c}}\to Y$; $P\left( U \right)$ is the prior probability distribution of exogenous variables. The causal flow between variables satisfies: the multimodal input $X$ is a common cause of ${{R}_{c}}$ and ${{R}_{s}}$, while ${{R}_{s}}$ has no direct causal link with $Y$. This structure provides a strict theoretical boundary for subsequent causal intervention and factor decoupling.

2.2 CERL framework and CERL paradigm

To address the issues of blind fusion, inadequate robustness, and lack of interpretability in multimodal semantic fusion of visual communication images, this paper proposes the CERL paradigm. The core idea of this paradigm is to guide structured domain knowledge injection, constrain the process with causal reasoning, and aim for interpretability, systematically realizing the paradigm shift from traditional associative learning to causal understanding. This paradigm breaks through the limitations of existing methods, which only rely on data statistical associations, by transforming visual communication domain expertise into computable conceptual prototypes, providing directed guidance for multimodal fusion and avoiding irrelevant feature interactions. At the same time, it constrains the representation learning process with causal reasoning to ensure that the model captures task-core causal logic rather than superficial style associations. Finally, through a full-link interpretable system, it transforms the learning results into domain-understandable knowledge, achieving a virtuous cycle of "learning - explanation - validation," aligning with the deep needs of the design field for causal understanding and practical guidance.

Figure 1. The ECPCF

The ECPCF is the specific implementation carrier of the CERL paradigm. It adopts a hierarchical modular design, clearly distinguishing the information transmission and logical control paths by using solid lines to mark the data flow and dashed lines to mark the control flow. The core innovative modules are highlighted with differentiated colors to emphasize design priorities. The framework follows a five-phase closed-loop logic, with each phase progressing incrementally and providing mutual feedback: the Multimodal Semantic Feature Preprocessing and Alignment phase standardizes and cross-modally calibrates the three types of features (visual, textual, and design metadata), providing a high-quality feature foundation for subsequent fusion; the Construction of Domain-Specific Concept Prototype Memory Bank phase structurally injects domain knowledge into the model, generating a concept prototype set with domain discriminative power to provide directional guidance for the fusion process; the concept-guided multimodal semantic fusion phase uses concept prototypes as intermediaries to achieve precise interaction and structured fusion of multimodal features, generating fusion features rich in domain semantics; the SCM-based Causal Intervention and Representation Learning phase decouples the fusion features into causal and style factors through factor decomposition and intervention operations, ensuring the robustness and causal validity of the representations; the Task Prediction and Bidirectional Explainability Analysis phase completes downstream task prediction based on causal factors, while also outputting interpretable results through retrospective attribution and generative explanation. The explanation information can be fed back to the concept prototype memory bank for optimization, forming a complete closed loop. This framework achieves the organic unity of multimodal semantic fusion and CERL through the collaborative function of all modules, fully embodying the guiding-constraint-target core logic of the CERL paradigm. Figure 1 intuitively demonstrates the ECPCF.

2.3 Multimodal semantic feature preprocessing and alignment

The core of multimodal semantic feature preprocessing is to precisely extract the core information of each modality and complete the preliminary standardization to provide high-quality input for subsequent fusion. The visual semantic features adopt a fusion strategy of low-level and high-level features: low-level features include the HSV histogram, LBP texture descriptor, and spatial layout matrix. The HSV histogram quantizes the image color space into 16×16×16 intervals, generating a 4096-dimensional vector. The LBP texture descriptor uses a 3 × 3 neighborhood calculation to generate a 256-dimensional vector. The spatial layout matrix divides the image into 16 × 16 grids and generates a 256-dimensional vector by calculating the average pixel values within each grid. The three features are concatenated to obtain the low-level visual feature V_low$\in$R⁴⁶⁰⁸; high-level semantic features are generated by encoding with a pre-trained Swin Transformer. The input image is normalized to 224 × 224 and passed through the Swin-Tiny model to output a 768-dimensional feature vector V_high$\in$R⁷⁶⁸. The final visual semantic feature V is obtained by concatenating V_low and V_high after layer normalization (LN): V=LN([V_low;V_high])$\in$R⁵³⁷⁶. The text semantic features adopt joint encoding of semantics and visual attributes: the text content is encoded by the BERT-base model to obtain a 768-dimensional semantic vector T_sem$\in$R⁷⁶⁸, and the text’s visual attributes are one-hot encoded and normalized to generate a 64-dimensional vector T_vis$\in$R⁶⁴. The final text feature T is obtained through attention-weighted fusion: T=α·T_sem+(1−α)·T_vis, where α$\in$[0,1] is the attention weight learned adaptively based on feature correlation. The design metadata semantic features are encoded structurally: the objective design labels are one-hot encoded to generate a 128-dimensional vector, and the subjective user perception feedback is generated by statistical heatmap peak positions and score distributions to form a 32-dimensional feature. The two are concatenated and standardized to obtain M$\in$R160.

To solve the semantic gap caused by the heterogeneity of multimodal features, a cross-modal semantic calibration alignment strategy based on contrastive learning is used. The core idea is to map features from different modalities into a unified semantic space through concept consistency contrastive loss. Let the multimodal feature set of the i-th sample in a batch be {V_i,T_i,M_i}, where the positive sample pair corresponds to different modality features of the same image, and the negative sample pair corresponds to either same-modality or cross-modality features from different images. The concept consistency contrastive loss L_cc$~$is defined as:

${{\text{L}}_{\text{cc}}}\text{=-}\frac{\text{1}}{\text{3N}}\underset{\text{i=1}}{\overset{\text{N}}{\mathop \sum }}\,\underset{{{\text{F}}_{\text{i}}}\in \text{ }\!\!\{\!\!\text{ }{{\text{V}}_{\text{i}}}\text{,}{{\text{T}}_{\text{i}}}\text{,}{{\text{M}}_{\text{i}}}\text{ }\!\!\}\!\!\text{ ,F}_{\text{i}}^{\text{+}}\in \text{ }\!\!\{\!\!\text{ }{{\text{V}}_{\text{i}}}\text{,}{{\text{T}}_{\text{i}}}\text{,}{{\text{M}}_{\text{i}}}\text{ }\!\!\}\!\!\text{ ,}{{\text{F}}_{\text{i}}}\ne \text{F}_{\text{i}}^{\text{+}}}{\mathop \sum }\, \\ \text{log}\frac{\text{exp(sim(}{{\text{F}}_{\text{i}}}\text{,F}_{\text{i}}^{\text{+}}\text{)/ }\!\!\tau\!\!\text{ )}}{\mathop{\sum }_{{{\text{F}}_{\text{j}}}\in {{\text{N}}_{\text{i}}}}\text{exp(sim(}{{\text{F}}_{\text{i}}}\text{,}{{\text{F}}_{\text{j}}}\text{)/ }\!\!\tau\!\!\text{ )}}$                   (1)

where, N is the batch size, $\text{F}_{\text{i}}^{\text{+}}$ is the positive sample feature, N_i is the set of negative sample features, sim(,) is the cosine similarity function, and τ is the temperature parameter. The optimization objective of this loss is to minimize the distance between different modality features of the same image while maximizing the distance between features from different images. Additionally, the introduction of domain concept prior constraints in the similarity calculation ensures that the aligned features fit the semantic logic of the visual communication domain, laying the foundation for subsequent concept-guided fusion. Figure 2 shows the process of multimodal semantic feature preprocessing and concept prototype memory bank construction.

Figure 2. Multimodal semantic feature preprocessing and concept prototype memory bank construction process

2.4 Visual communication domain-specific concept prototype memory bank construction

The core of constructing the visual communication domain-specific concept prototype memory bank is to build a structured concept system that fits domain needs and generate learnable concept prototypes. The concept set definition follows both the consensus of domain experts and the practical requirements of design, covering three core categories: information transmission, aesthetic perception, and user experience. Information transmission concepts focus on the core information delivery function of design, including information hierarchy clarity, theme prominence, and text readability; aesthetic perception concepts rely on visual design aesthetics principles, covering color harmony, layout balance, and style consistency; user experience concepts relate to audience perception feedback, including visual appeal, information retrieval efficiency, and emotional resonance. The concept screening process is completed collaboratively by three visual communication domain experts and two computer vision researchers, with multiple rounds of discussion to eliminate ambiguous concepts. Ultimately, 60 core concepts form the concept set, ensuring the domain adaptability, discriminative power, and operability of the concepts.

Concept prototype initialization is completed based on an expert-labeled sample set, using a strategy combining clustering and feature averaging. First, an expert-labeled sample set is constructed. For each concept c_k (k = 1,2,...,K, K = 60), the expert selects 100 representative visual communication image samples to form the labeled set ${{\text{S}}_{{{\text{c}}_{\text{k}}}}}$; for each sample, the aligned multimodal semantic features are extracted. The K-means clustering algorithm is used to cluster the feature set corresponding to ${{\text{S}}_{{{\text{c}}_{\text{k}}}}}$, and the initial concept prototype p_k is obtained by averaging the features. The initialization formula for the concept prototype vector set P=[p₁,p₂,...,p_K]$\in$R^D×K is:

${{\text{p}}_{\text{k}}}\text{=}\frac{\text{1}}{\text{ }\!\!|\!\!\text{ }{{\text{S}}_{{{\text{c}}_{\text{k}}}}}\text{ }\!\!|\!\!\text{ }}\underset{\text{x}\in {{\text{S}}_{{{\text{c}}_{\text{k}}}}}}{\mathop \sum }\,\text{Feat(x)}$                      (2)

where, Feat(x) denotes the multimodal semantic features of sample x, and |${{\text{S}}_{{{\text{c}}_{\text{k}}}}}$| is the size of the concept c_k labeled sample set. This initialization method ensures that the prototypes can accurately capture the core feature distribution of similar concepts.

To make the concept prototypes adaptive to data distribution and dynamically optimized, a prototype-sample feature matching loss is designed to enable dynamic updating of the memory bank. During training, for each sample's multimodal feature f_i in the batch, the cosine similarity sim(f_i,p_k) between the sample and each concept prototype p_k is calculated. The top-3 concept prototypes most matching the sample are selected to form the matching prototype set. The prototype-sample feature matching loss L_pm is defined as:

${{\text{L}}_{\text{pm}}}\text{=}\frac{\text{1}}{\text{N}}\underset{\text{i=1}}{\overset{\text{N}}{\mathop \sum }}\,\text{(1-}\underset{\text{k}\in {{\text{K}}_{\text{i}}}}{\mathop{\text{max}}}\,\text{sim(}{{\text{f}}_{\text{i}}}\text{,}{{\text{p}}_{\text{k}}}\text{))}$                   (3)

where, N is the batch size, and K_i is the matching prototype set for sample i. This loss minimizes the distance between the sample features and matching prototypes, and during training, each iteration updates the concept prototypes based on the current batch features:

${{\text{p}}_{\text{k}}}\leftarrow \text{ }\!\!\eta\!\!\text{ }\cdot {{\text{p}}_{\text{k}}}\text{+(1- }\!\!\eta\!\!\text{ )}\cdot \frac{\text{1}}{\text{ }\!\!|\!\!\text{ }{{\text{B}}_{\text{k}}}\text{ }\!\!|\!\!\text{ }}\underset{{{\text{f}}_{\text{i}}}\in {{\text{B}}_{\text{k}}}}{\mathop \sum }\,{{\text{f}}_{\text{i}}}$                   (4)

where, η$\in$(0,1) is the update weight, and B_k is the set of sample features in the batch that match prototype p_k, ensuring that the prototypes dynamically adapt to the data distribution during training and improve the precision of concept-guided fusion.

2.5 Concept-guided multimodal semantic fusion

Concept-guided multimodal semantic fusion uses domain-specific concept prototypes as intermediaries and achieves precise interaction of multimodal features through a cross-attention mechanism. The core idea is to avoid ineffective fusion by relying on domain concepts, enhancing the semantic relevance of features to the domain. First, feature-concept prototype matching is performed. For the preprocessed and aligned visual, textual, and design metadata features V$\in$R^D, T$\in$R^D, M$\in$R^D, the cosine similarity with each prototype in the concept prototype set P$\in$R^D×K is calculated, and the top-5 most similar concepts are selected to form the active concept set ${{\text{P}}_{\text{act}}}\text{=}\left[ \text{p}_{\text{1}}^{\text{act}}\text{, ... ,p}_{\text{5}}^{\text{act}} \right]$. The cosine similarity calculation formula is:

$\text{sim}(f, p_k) = \frac{f \cdot p_k}{ \| f \|_2 \cdot \| p_k \|_2 }$               (5)

where, f is any modality feature, and p_k is the k-th concept prototype. This process ensures that only domain concepts related to the current sample's semantics are activated, providing directional guidance for subsequent fusion.

The concept-guided cross-attention mechanism is constructed based on the active concept set to achieve cross-modal feature interaction. The core idea is to calculate the attention weights between modalities through concept prototypes. First, the modality features are associated with the active concept prototypes, yielding modality-concept associated features V_rel=V·P_act, T_rel=T·P_act, M_rel=M·P_act. Then, using modality-concept associated features as a bridge, the cross-modal attention weights between visual and text, visual and metadata, and text and metadata are calculated. Taking the visual-text attention weight calculation as an example:

${{\text{A}}_{\text{V}\to \text{T}}}\text{=softmax}\left( \frac{{{\text{V}}_{\text{rel}}}\cdot \text{T}_{\text{rel}}^{\top }}{\sqrt{{{\text{D}}_{\text{act}}}}} \right)$                   (6)

where, D_act=5 is the number of active concepts, and the attention weight A_V→T quantifies the guidance weight of visual features on text features. Similarly, other modality attention weights A_T→V, A_V→M, etc., can be obtained. Finally, the fused feature F is obtained by concatenating the weighted features from each modality:

$\text{F=LN}\left( \text{ }\!\![\!\!\text{ }{{\text{A}}_{\text{T}\to \text{V}}}\text{T+}{{\text{A}}_{\text{M}\to \text{V}}}\text{M+V,}{{\text{A}}_{\text{V}\to \text{T}}}\text{V+}{{\text{A}}_{\text{M}\to \text{T}}}\text{M+T }\!\!]\!\!\text{ } \right)$                  (7)

where, LN ensures stable feature distribution. This fusion method achieves precise semantic alignment and interaction of multimodal features through concept mediation.

To ensure semantic consistency between the fused features and domain concepts, a concept consistency loss L_con is defined to constrain the fusion process. The average cosine similarity between the fused feature F and the active concept set P_act is computed, and the loss function is defined as:

${{\text{L}}_{\text{con}}}\text{=1-}\frac{\text{1}}{\text{5}}\underset{\text{k=1}}{\overset{\text{5}}{\mathop \sum }}\,\text{sim(F,p}_{\text{k}}^{\text{act}}\text{)}$                 (8)

The optimization objective of this loss is to maximize the semantic similarity between the fused features and the active concepts, forcing the fused features to encode domain-related information and enhance their structure and explainability. During training, the concept consistency loss is jointly optimized with the prototype-sample matching loss and task loss to ensure that the fused features meet both task requirements and domain concept constraints. Figure 3 presents the complete process of concept-guided multimodal semantic fusion.

Figure 3. Concept-guided multimodal semantic fusion process

2.6 Causal intervention and representation learning based on SCM

Based on the structure causal model defined earlier, this section formalizes the dependencies between variables using causal graphs and constructs causal constraints based on intervention theory to provide strict theoretical support for representation learning. The causal graph clearly depicts the causal flow between exogenous variables, multimodal inputs, causal factors, style factors, and task labels: multimodal inputs directly drive the generation of causal and style factors, task labels are only determined by causal factors, and style factors have no direct causal relation with task labels. Exogenous variables provide noise disturbances to the endogenous variables. Based on this structure, the intervention operation do(R_s=rₛ') is defined, meaning fixing the causal factor R_c while replacing the style factor R_s with any value rₛ'. Combining the stability assumption of the causal mechanism in SCM, the core theoretical guarantee under intervention is derived:

$\text{P(Y }\!\!|\!\!\text{ do(}{{\text{R}}_{\text{s}}}\text{=r}{{\text{ }\!\!'\!\!\text{ }}_{\text{s}}}\text{),}{{\text{R}}_{\text{c}}}\text{)=P(Y }\!\!|\!\!\text{ }{{\text{R}}_{\text{c}}}\text{)}$                 (9)

The derivation process is as follows: since the causal mechanism of Y is only determined by R_c, the intervention only changes the value of R without disrupting the causal mechanism from R_c to Y, so the conditional probability remains unchanged. This conclusion provides a theoretical basis for representation robustness, meaning that as long as the learned representation can accurately decouple R_c and R_s, task predictions will remain stable during style changes.

Variational inference is used to perform factorization of the fused feature F, decoupling it into task-relevant causal factors R_c$\in$ℝ^Dc and task-independent style factors R_s$\in$ℝ^Ds, where D_c+D_s=D and D is the dimension of the fused features. A variational encoder q_φ(R_c,R_s|F) is introduced to approximate the posterior distribution, aiming to approach the true posterior P(R_c,R_s|F). The prior distribution is defined as p(R_c,R_s)=p(R_c)p(R_s), assuming that R_c and R_s are independent. Based on the variational inference principle, the evidence lower bound (ELBO) is maximized to optimize the encoder parameters. The ELBO objective function is derived as:

$\text{logP(F)}\ge {{\text{E}}_{{{\text{q}}_{\phi }}\text{(}{{\text{R}}_{\text{c}}}\text{,}{{\text{R}}_{\text{s}}}\text{ }\!\!|\!\!\text{ F)}}}\text{ }\!\![\!\!\text{ logP(F }\!\!|\!\!\text{ }{{\text{R}}_{\text{c}}}\text{,}{{\text{R}}_{\text{s}}}\text{) }\!\!]\!\!\text{ } \text{-KL(}{{\text{q}}_{\phi }}\text{(}{{\text{R}}_{\text{c}}}\text{,}{{\text{R}}_{\text{s}}}\text{ }\!\!|\!\!\text{ F)}\parallel \text{p(}{{\text{R}}_{\text{c}}}\text{)p(}{{\text{R}}_{\text{s}}}\text{))}$                (10)

where, the first term is the reconstruction loss, which constrains the decoder to accurately reconstruct the fused features from the decoupled factors. The second term is the KL divergence, which ensures that the approximate posterior approximates the prior distribution and ensures the effectiveness of factor decoupling. A decoder p_θ(F|R_c,R_s) is introduced to implement feature reconstruction. The final factorization is completed by jointly optimizing the ELBO, resulting in decoupled representations (R_c,R_s).

To strengthen the factor decoupling effect and verify the validity of the intervention theory, a counterfactual training strategy is designed, integrating the do operation into the training process. Counterfactual samples are generated based on the principle "causal factors remain unchanged, style factors are replaced": for the decoupled factors $\text{(R}_{\text{c}}^{\text{i}}\text{,R}_{\text{s}}^{\text{j}}\text{)}$ of the original sample, a style factor $\text{R}_{\text{s}}^{\text{j}}$ from another sample in the batch is randomly selected as a replacement, generating a counterfactual factor pair $\text{(R}_{\text{c}}^{\text{i}}\text{,R}_{\text{s}}^{\text{j}}\text{)}$, and then the counterfactual fused feature ${{\text{F}}^{\text{cf}}}$=p_θ(F|$\text{R}_{\text{c}}^{\text{i}}\text{,R}_{\text{s}}^{\text{j}}$) is generated through the decoder. The generation process must satisfy the constraint: the cosine similarity between the counterfactual factor pair $\text{R}_{\text{c}}^{\text{i}}$ and the original $\text{R}_{\text{c}}^{\text{i}}$ should not be lower than 0.95, ensuring that the causal factor is unaffected. To constrain the consistency of model predictions, a counterfactual consistency loss is defined:

${{\text{L}}_{\text{cf}}}\text{=}\frac{\text{1}}{\text{N}}\underset{\text{i=1}}{\overset{\text{N}}{\mathop \sum }}\,\|\text{Pred(}{{\text{F}}^{\text{i}}}\text{)-Pred(}{{\text{F}}^{\text{cf,i}}}\text{)}\|_{\text{2}}^{\text{2}}$                     (11)

where, Pred( ) represents the model's task prediction output, and N is the batch size. This loss minimizes the prediction difference between the original and counterfactual samples, forcing the model to rely solely on causal factors for decision-making, further strengthening the task relevance of R_c and the irrelevance of R_s, ultimately improving representation robustness and causal validity. During training, this loss is optimized together with the variational inference ELBO and concept consistency loss, forming a complete causal constraint system. Figure 4 shows the principles of causal intervention and representation learning based on SCM.

Figure 4. Causal intervention and representation learning based on SCM