Cross-Modal Product Image Retrieval for E-Commerce Recommendation Systems via Deep Learning

In the domain of natural images, cross-modal text-image retrieval has been advanced through techniques such as self-attention, enabling long-range semantic association and key information extraction. However, essential distinctions exist between product and natural images. Product images are required to emphasize “purchasability features,” such as fabric texture in apparel or functional interfaces in home appliances, and are often embedded in complex backgrounds. Traditional approaches struggle to isolate core product features from such redundant visual information. Simultaneously, textual descriptions in e-commerce incorporate both marketing-driven and function-oriented expressions. For instance, phrases like “slimming dress” convey not only an effect-based descriptor but also implicit category information. Existing models are easily distracted by non-informative tokens such as “bestseller” or “hot item” and fail to focus on purchase-critical attributes such as “waist-cinching” or “chiffon.” More critically, the primary objective of e-commerce recommendation lies in facilitating purchase decisions, which necessitates accurate mapping between latent user needs and product characteristics. Most existing cross-modal models are designed for general-purpose retrieval tasks and have not been optimized for the demand-attribute alignment required in recommendation scenarios. This underscores the necessity for task-specific approaches tailored to e-commerce recommendation systems.

To address the specific demands of e-commerce recommendation, a dedicated cross-modal product image retrieval method was proposed. This approach is guided by the principle of “adapting to product-specific characteristics and serving recommendation objectives,” and a full-chain optimization framework was constructed based on self-attention mechanisms, as illustrated in Figure 1. In the visual feature extraction stage, product images were divided into sub-image patches, and self-attention mechanisms were introduced. Attention weights were directed—under the guidance of class-supervised signals—toward discriminative regions such as collars and cuffs, while positional encoding was applied to preserve the structural relationships within the product. This design facilitates the differentiation of visually similar products that share the same style but differ in fine-grained details, a challenge that frequently arises in e-commerce scenarios.

In the textual feature extraction stage, semantic masking was combined with self-attention mechanisms. Non-essential expressions such as “recommended” or “bestseller” were first masked. Subsequently, semantic associations—such as those between “slim fit” and “tailoring”—were captured through self-attention, while representations of attributes that strongly influence purchasing decisions, such as “fabric” and “size,” were reinforced. This process ensures that the extracted textual features remain closely aligned with the user's actual intent. During the cross-modal fusion stage, a bidirectional mapping between “visual sub-blocks” and “textual keywords” was constructed using self-attention-based mechanisms. For instance, when the query “durable running shoes” is entered, the model is designed to increase the attention weight linking the visual sub-block representing a “rubber sole” to the keyword “durable.” This enables accurate alignment between product characteristics and user demands. As a result, the recommendation system is supported by a three-tiered matching rationale comprising image detail, textual demand, and purchase intent. This architecture ensures that the recommendations not only correspond to the product's appearance but also align with the user’s practical usage scenarios. The overall model architecture for the proposed cross-modal product image retrieval method in e-commerce recommendation is presented in Figure 2.

Figure 2. Model architecture of the cross-modal product image retrieval method for e-commerce recommendation

2.1 Global-local visual feature extraction module

The global-local visual feature extraction module, guided by class supervision, was designed to address the core requirement of accurately interpreting visual product attributes in e-commerce recommendation scenarios. In such contexts, visual features extracted from product images are directly associated with user purchasing decisions. A shirt may be selected based on a local detail such as “embroidered cuffs,” while a pair of trousers may be dismissed due to a global characteristic like a “loose overall fit.” However, due to the variability of shooting angles, small targets such as buttons and zippers, as well as densely distributed targets such as printed patterns, are often blurred in product images. Moreover, cluttered backgrounds introduced by accessories or props frequently interfere with the extraction of core product features. Product category information serves as a critical anchor for interpreting such features. For example, “dress” images should prompt attention to the neckline rather than the sole. Therefore, the proposed module was designed to jointly enable global orientation, local detail focus, and category-guided feature prioritization, providing a robust visual representation tailored to the requirements of cross-modal matching in e-commerce recommendation systems.

The global feature extraction module was built upon the Residual Neural Network (ResNet) and was optimized using class-supervised signals to enhance the detection of salient category-specific features, which is essential in scenarios where product categories strongly influence purchasing intent. The architecture incorporates interactive connections between shallow and deep features: shallow features preserve basic information such as edges and color, while deep features capture abstract semantics. This interaction mitigates the limitations of single features. Class supervision was introduced by converting product category labels into supervisory signals that guide the feature extraction process. This encourages the model to focus on category-relevant core attributes. For instance, in the case of the “sneakers” category, increased attention is allocated to the “sole pattern” and “lacing system,” whereas unrelated elements such as “floor texture” are downweighted. As shown in the formulation below, the input image is transformed by the ResNet architecture to produce the global feature representation D_H. The class supervision signal adjusts the model parameters ϕ through a loss function, ensuring that D_H includes not only the global contour but also category-relevant core features, providing support for recognizing functional attributes of interest when users query terms like “sneakers.” The feature extraction process using the ResNet model is denoted as L_RES, and the corresponding expression is:

$D_H=L_{R E S}(U ; \varphi)$              (1)

The local feature extraction module was constructed based on the Vision Transformer (ViT) architecture to address the challenge of representing small and densely packed visual targets in product images—an essential factor in e-commerce, where “fine details determine purchase decisions.” In such images, local features often constitute critical decision-making elements. However, conventional convolutional methods are prone to detail loss due to the limited receptive field. ViT addresses this limitation by segmenting an image into O×O patches and transforming them into sequential inputs. Through the multi-head attention mechanism, long-range dependencies across image patches can be captured. For instance, the spatial relationship between the “left chest pocket” and “right sleeve logo” on a shirt can be simultaneously modeled. Additionally, multi-layer perceptrons (MLPs) and layer normalization enhance the stability of fine-grained representations. To mitigate the effects of variable camera angles, ViT's sequence-based modeling inherently ignores absolute position interference and instead emphasizes relative spatial relationships among patches. This enables the precise extraction of fine-grained features of small and densely distributed targets, thereby supporting the recommendation system in aligning with users’ attention to fine detail. Denoting the multi-head attention layer as MSA, the MLP as MLP, and layer normalization as LN, the internal processing process of ViT is formulated as follows:

$\begin{aligned} & d_0=\left[a_{C L S} ; a_1 A ; \ldots ; a_v A\right]+A^{P O S} \\ & A \in \Re^{\left(O^2 \cdot Z\right) \times G}, A^{P O S} \in \Re^{(V+1) \times G}\end{aligned}$                   (2)

$d_f^{\prime}={MSA}\left(L N\left(d_{f-1}\right)\right)+d_{f-1}, f=1 \ldots F$                 (3)

$d_f=M L P\left(L N\left(d_f^{\prime}\right)\right)+d_f^{\prime}, f=1 \ldots F$                 (4)

$D_M=L N\left(d_F^0\right)$             (5)

The integration of global and local features represents a critical stage in adapting the module output to e-commerce recommendation tasks. This fusion enables both “macro-level category orientation” and “micro-level detail support.” The global feature representation D_H carries salient, category-dominant features, while the local feature representation D_M encodes fine-grained visual details. During integration at the terminal stage of the visual pipeline, these features are not simply concatenated; rather, feature fusion is performed through weighted allocation, thereby emphasizing attributes that are most relevant to user intent. For instance, when the query is “warm down jacket,” the weight of “loftiness” in the global feature is increased. Conversely, when durability is the focus—as in “durable down jacket”—the weight of “stitching quality” in the local feature is enhanced. The resulting fused representation retains both category-level attributes and detail-level distinctions, providing a rich and accurate visual semantic foundation for cross-modal alignment. This ultimately enables the recommendation system to infer not only the target product category but also the specific product details that users value. The integration of D_H and D_M is defined by the following expression:

$D_N=D_H \oplus D_M$                 (6)

2.2 Textual feature extraction module

Textual descriptions in e-commerce are characterized by low semantic density and high informational focus. These descriptions typically consist of product category terms, key attribute phrases, and marketing-related vocabulary. Semantically relevant information—particularly that which strongly influences purchasing decisions—is concentrated in category and attribute terms, resulting in an overall semantic complexity significantly lower than that of natural language text. To accommodate these characteristics, a textual feature extraction module based on dominant semantic masking was designed. This module enables precise semantic parsing of product descriptions in e-commerce recommendation scenarios. The first six layers of the Bidirectional Encoder Representations from Transformers (BERT) model were employed to construct the core of the textual feature extraction module. These layers, comprising multi-head attention, MLPs, and layer normalization components, are sufficient for capturing shallow semantic associations such as “material-functionality” while avoiding the redundant computation associated with deeper network layers. This structure aligns with the computational efficiency requirements of real-time recommendation systems. The final six layers of BERT were reserved for subsequent cross-modal fusion, ensuring that textual features can interact deeply with visual representations. This design lays the foundation for accurate alignment between “product image details” and “textual attributes.” A representative example of dominant semantic masking is illustrated in Figure 3. Specifically, a product description S is first tokenized and encoded into a sequence of word and positional vectors. Let the input sequence length be denoted by v, and let the dimensionality of each word vector be represented by f. The resulting representations can be formalized as:

$S=\left[n_1, n_2, \ldots, n_v\right] \in E^{v \times f}$                (7)

$D_S=L_{B E R T}(S)$                 (8)

3.jpg

Figure 3. Example of dominant semantic masking

This module enhances the extraction of core semantics in e-commerce texts through a combined strategy of targeted masking and random masking, thereby addressing issues commonly observed in traditional textual feature extraction—namely, interference from marketing terms and the dilution of critical semantic content. In e-commerce descriptions, category terms such as “sneakers” and attribute terms such as “non-slip sole” or “breathable mesh” serve as essential semantic bridges connecting product features to user needs. In contrast, marketing expressions such as “hot sale” or “limited time” constitute non-essential information. The module first applies random masking to 15% of the input tokens, which preserves the model’s ability to comprehend overall textual semantics. On this basis, category terms and key attribute terms are forcibly tagged and masked with higher probability or subjected to random replacement. For example, in the phrase “non-slip sneakers, limited-time discount,” the terms “non-slip” and “sneakers” are preferentially masked to compel the model to strengthen its semantic encoding of these critical elements during training. This approach not only prevents the model from overfitting to non-informative marketing content but also leverages a “deletion-prediction” learning mechanism to deepen the representation of key semantics. As a result, the extracted textual features are more accurately aligned with genuine user intent.

Collaborative representation between the textual and visual feature extraction modules was achieved through the reuse of class supervision information, thereby enhancing consistency in cross-modal matching within e-commerce recommendation settings. The visual feature extraction module is guided to emphasize category-relevant visual features through class supervision. Correspondingly, the textual feature extraction module employs targeted masking of category terms to direct semantic alignment toward the same category space. For instance, when the visual module emphasizes neckline designs in the “dress” category, the masking of terms such as “dress” and “V-neck” in the textual module increases their semantic attention weights, facilitating alignment between textual and visual features at both the category and detail levels. This coordinated mechanism effectively mitigates the issue of cross-modal matching deviation among items within the same category—a common challenge in e-commerce environments. Meanwhile, the residual robustness introduced by random masking enables the model to tolerate irregularities in user-generated descriptions. Even when input text is irregular or imprecise, core semantic elements can still be accurately extracted, thus providing the recommendation system with a reliable link between textual intent and visual features.

2.3 Cross-extraction module

The cross-extraction module, which integrates visual and textual semantic information, was designed to address the core challenge of visual and textual semantic misalignment in e-commerce scenarios. A feature calibration architecture, constructed from the final six layers of the BERT model, was established and kept independent from the inference pipeline, bridging the correlation gap between visual and textual features in the context of e-commerce recommendation. Although visual and textual features may reference the same product attribute, independently extracted features are prone to a mismatch of "seemingly unrelated but semantically related" due to modal differences. For example, the visual representation of a “metallic zipper’s gloss” and the textual description “durable hardware” may be intrinsically related but appear distant in the feature space, leading to false mismatches. This module focuses solely on semantic calibration and does not participate in the final retrieval inference. As such, it avoids interference with the primary inference path while leveraging the deep semantic modeling capability of BERT’s final six layers to uncover latent correlations between “visual details” and “textual attributes,” thereby providing foundational support for feature alignment in the recommendation system.

The cross-extraction module comprises two submodules. The similarity computation submodule L_SIM is responsible for precisely measuring the correlation between visual and textual features, thereby reinforcing the e-commerce-specific logic of “intra-category aggregation and inter-category separation” in the feature distribution. This submodule employs vector similarity measures, such as cosine distance, to directly compare visual features D_N with textual features D_S. For instance, for a query like “non-slip running shoes,” the cosine similarity is computed between the visual feature representing the “rubber outsole texture” and the textual feature corresponding to “non-slip,” ensuring that highly correlated features are located closer in the feature space. In addition, a triplet loss function was introduced to optimize the parameters. A triplet structure—consisting of a target product, a similar product from the same category, and a product from a different category—was constructed. The loss function minimizes the distance between the target and the similar product while maximizing the distance from the dissimilar product. This encourages the feature space to satisfy the dual demands of distinguishing categories and differentiating styles typical in e-commerce scenarios. As a result, the risk of erroneously recommending, for example, “leather shoes” when a user searches for “running shoes” is effectively minimized.

The vision-language fusion submodule L_MUL employs a dual-loss optimization strategy to enhance semantic interaction between modal features in e-commerce settings, addressing the challenge of achieving precise alignment between visual details and textual attributes. The masked language loss extends the dominant semantic masking strategy introduced in the textual module. Core attribute terms—such as “linen” or “non-slip”—are intentionally masked in the text, and the model is required to predict the masked tokens by leveraging visual features. For instance, when the textual token “linen” is masked, the model is prompted to infer this label based on visual cues such as the “loose fiber texture” extracted from the image, enhancing the mapping between visual fine-grained features and textual product attributes. In parallel, the image-text matching loss focuses on the compatibility judgment of product image-text pairs. For positive samples, such as a floral dress image paired with the phrase “floral long dress,” the model is trained to produce a high probability of correspondence. For negative samples, such as pairing the same image with “solid short skirt,” the model is optimized to yield a low correspondence score, ensuring that the system can distinguish whether an image and a text description refer to the same product. Through joint optimization of these two losses, visual features include not only what the product looks like but also which textual attributes it corresponds to. Simultaneously, textual features convey not only what attributes are described but also how they visually manifest. This ensures that the final recommendation aligns both with the product’s physical appearance and with the user's expressed intent through textual queries. The cross-extraction module can be formally expressed as:

$L_{C R O S S}=\left\{L_{S I M}, L_{M U L}\right\}$               (9)

2.4 Multi-objective loss function optimization

The multi-objective loss function optimization was designed with the core goal of achieving precise cross-modal feature alignment in e-commerce recommendation scenarios. Through the joint optimization of four distinct loss functions, model parameters were constrained from multiple dimensions, addressing complex optimization requirements that cannot be adequately resolved by a single loss function. In e-commerce settings, the alignment between product images and textual descriptions must extend beyond surface-level “appearance-to-description” correspondence to include deeper “function-to-demand” associations. A single loss function often optimizes only one dimension of model performance, whereas a multi-objective loss design enables comprehensive improvements through the combined application of the triplet loss LOSS_SY (enhancing feature discriminability), the class-supervised cross-entropy loss LOSS_JCS (improving category-specific feature salience), the masked language loss LOSS_YM (strengthening semantic association), and the image-text matching loss LOSS_PP (optimizing matching accuracy). Through their integration, a comprehensive objective can be achieved—ensuring retrieval accuracy, category clarity, semantic alignment, and robust matching performance—thereby meeting the multidimensional requirements of cross-modal matching in e-commerce recommendation systems.

The triplet loss serves as the core loss function, designed to improve cross-modal feature discriminability through bidirectional retrieval constraints. This approach supports the nuanced distinction of visually or semantically similar products—a frequent necessity in e-commerce environments. The optimization is carried out in two directions: When retrieving text based on image features, the feature distance between the query image and non-matching texts is maximized, while the distance to the corresponding text is minimized. Conversely, when retrieving images based on textual features, the distance to mismatched images is increased, and the distance to relevant images is reduced. This bidirectional constraint addresses the challenge of distinguishing between products with similar overall appearances but different attributes. For instance, when the user query is “loose-fit jeans,” the model is guided by the triplet loss to strengthen the alignment between the visual representation of a “loose silhouette” and the corresponding textual description, while increasing the distance to the feature representation of “skinny jeans.” This prevents the recommendation of visually similar but functionally mismatched products, thereby ensuring that retrieval results directly reflect user intent. Let the margin parameter that enlarges the separation between positive and negative pairs be denoted by β. Let COS() represent the similarity score obtained from the cross-modal similarity computation module. The negative text corresponding to an image is represented as S', and the negative image corresponding to a text S is denoted as U'. The formal expression is:

${LOSS}_{S Y}=\sum_{S^{\prime}}\left[\beta-{COS}(U, S)+{COS}\left(U, S^{\prime}\right)\right]+\sum_{U^{\prime}}\left[\beta-{COS}(U, S)+{COS}\left(U^{\prime}, S\right)\right]$               (10)

The class-supervised cross-entropy loss LOSS_JCS and the masked language loss LOSS_YM were jointly designed to enhance the semantic association between category and attribute from the visual and textual modalities, respectively. This design aligns with user behavior patterns observed in e-commerce, wherein category guides decision-making and attribute determines selection. The class-supervised cross-entropy loss introduces category labels to guide the global visual feature extraction module, encouraging the generation of discriminative category-specific representations. For instance, in the case of the “sneakers” category, the loss function constrains the model to emphasize core visual features such as “sole pattern” and “collar height,” ensuring that the resulting visual representation satisfies the foundational requirement of correct category identification for downstream recommendation. LOSS_YM optimizes the prediction of masked core attribute terms within e-commerce product texts using image features. For example, if an image exhibits “linen-like texture,” the loss function compels the model to accurately predict the masked token “linen,” thereby minimizing prediction error and reinforcing the correspondence between local visual features and textual attribute terms. This mechanism ensures that textual representations more accurately reflect the actual characteristics of the product, addressing common disjunctions between image and text in product listings. Let b_u denote the ground-truth category of an image, and let o_u represent the predicted category probability distribution output by the model. Then the expression is as follows:

${LOSS}_{J C S}=-\sum_{u=1}^V b_u \log \left(o_u\right)+\left(1-b_u\right) \log \left(1-o_u\right)$                (11)

Let the masked textual vector be denoted by S^, and let o^MA(U,S^) represent the predicted probability distribution of the masked text conditioned on the visual features. Then the expression is as follows:

${LOSS}_{Y M}=R_{\left(U, S^{\wedge}\right) \sim F} G\left(b^{M A}, o^{M A},\left(U, S^{\wedge}\right)\right)$                (12)

The image-text matching loss LOSS_PP was designed to optimize the overall compatibility of image-text pairs through a binary classification framework. This approach enhances the model’s ability to discern the consistency between marketing descriptions and actual product visuals, a frequent concern in e-commerce platforms where image-text mismatches are common. Under this formulation, image-text matching is treated as a binary classification task. A fully connected layer was employed to output a matching probability, which was then constrained using cross-entropy loss. When an image and its corresponding text description are semantically aligned, the model is encouraged to assign a matching probability close to 1. Conversely, when the image-text pair is contradictory, the probability is pushed toward 0. This optimization enables the model to autonomously filter out “misleading promotional content” and ensures that recommendations are based solely on authentically aligned features. Moreover, LOSS_PP complements LOSS_SY. While the former optimizes overall compatibility, the latter refines inter-feature distances. The combination of these two losses guarantees both the correctness of individual image-text matches and the discriminative power across different pairs, thereby achieving a retrieval performance that is both accurate and comprehensive—a critical requirement for effective e-commerce recommendation. The image-text matching loss generates the matching probability of an image-text pair, denoted by o^MT(U,S), by introducing a fully connected layer. The overall optimization objective remains the minimization of cross-entropy loss, formally expressed as:

${LOSS}_{P P}=R_{(U, S) \sim F} G\left(b^{M A}, o^{M A},(U, S)\right)$                  (13)

2.5 Algorithmic workflow

The algorithm training workflow was designed with the core objective of adapting to the unique characteristics of e-commerce data while strengthening cross-modal collaboration. Model performance was improved through a phased optimization process. The initial stage involved data preprocessing tailored to the properties of e-commerce data. For image data, background removal, angle normalization, and illumination enhancement were performed to ensure that visual input remains focused on the product itself. For textual data, non-essential marketing expressions such as “bestseller” or “hot sale” were filtered out. Only core elements—typically consisting of category and attribute terms—were retained. Synonym replacement was applied to enhance dataset diversity. Following preprocessing, the refined dataset entered the iterative training phase. Product images were processed through the visual feature extraction module to generate global features D_H and local features D_M, which were then fused into the comprehensive visual representation D_N. Simultaneously, the textual feature extraction module encoded the textual input into the representation D_S. During training, four loss functions operated in coordination to guide the model. The triplet loss ensured that products with the same style and attributes are positioned closer in the shared feature space. The class-supervised cross-entropy loss reinforced the salience of category-specific features. The masked language loss promoted fine-grained image-text alignment through the masked attribute terms. The image-text matching loss enabled the model to filter mismatched image-text pairs. The cross-extraction module indirectly optimized the single-modality modules. Through iterative learning, model parameters converged to an optimal configuration that enabled precise recognition of correlations between product characteristics and user intent in e-commerce settings. Figure 4 illustrates the conceptual architecture of the cross-modal product image retrieval model for e-commerce recommendation, while the complete training workflow is depicted in Figure 5.

The inference process was designed to satisfy the dual demands of real-time responsiveness and high-precision retrieval in e-commerce recommendation scenarios. To this end, a feature retrieval database tailored to e-commerce products was constructed in advance. Specifically, the pre-trained visual module was employed to generate FV using the product images, while the corresponding product description was processed by the textual module to generate D_S. All feature vectors were stored in association with their respective product IDs. Vector quantization techniques were applied to compress the feature dimensionality, thereby achieving a balance between storage cost and retrieval speed. This design was implemented to address the large-scale retrieval demands associated with e-commerce platforms containing tens of millions of products, while ensuring that feature comparison latency remains constrained within the millisecond range. Upon user query initiation, the inference pipeline was activated. The input image generated D_N using the visual module, while the textual input generated D_S via the textual module. The similarity computation module then compared the query features with the corresponding modal features stored in the database. The top V most similar candidates were selected based on the cosine distances. In parallel, e-commerce recommendation logic was incorporated to prioritize the retrieval of products with high sales volume and favorable user ratings, thereby ensuring that the final results not only satisfy feature-level matching but also align with user purchasing preferences.

Figure 4. Conceptual architecture of the proposed cross-modal product image retrieval model for e-commerce recommendation