Deep Feature Fusion Network: Cross-Guided Convolutional Neural Network–Transformer Feature Fusion for Plant Disease Image Classification

Plant diseases have been one of the most important constraints to crop production, accounting for an estimated annual loss of 20–40% of global crop yields [1]. Accurate and timely disease diagnosis is the first step to conduct targeted treatment, reduce unnecessary use of chemical pesticides, and maintain the health of the food supply chain. However, conventional disease diagnosis still relies on manual visual inspection by agronomists, which is time-consuming, labor-intensive, and subject to human error, especially when symptoms are slight, appear at an early stage, or are visually similar among different diseases.

Although deep learning has made significant progress in automatic plant disease recognition and benchmark datasets such as PlantVillage [2] allow supervised models to achieve or even surpass expert performance under controlled conditions, fine-grained classification remains challenging because models must capture both local texture abnormalities and global structural patterns. Single-backbone convolutional neural networks (CNNs) suffer from high inter-class similarity [3], but vision transformers (ViTs) [4] require large-scale training data and therefore degrade when training data is limited. In this study, the term fine-grained refers to the discrimination of visually similar disease categories (and their healthy counterparts) within the same crop species, and is evaluated through standard class-level recognition on PlantVillage, PlantDoc, and IP102, rather than through a dedicated, formally defined fine-grained taxonomic benchmark.

Recent works such as Swin Transformer [5] and multi-ViTs [6] have shown that hierarchical Transformer architectures can be scaled up efficiently. However, the question of how to effectively fuse the complementary CNN and Transformer features for fine-grained plant disease recognition remains unexplored. In this study, we propose Deep Feature Fusion Network (DFF-Net), which is built upon three novelties:

(a) A dual-stream architecture combining EfficientNet-B4 (CNN branch) and Swin Transformer-Base (Transformer branch) to simultaneously capture local texture features and global context.

(b) A Cross-Guided Feature Aggregation (CGFA) module that fuses both streams using bidirectional channel-attention gates, enabling each stream to guide the other toward disease-discriminative representations.

(c) The Hierarchical Symptom-Aware Module (HSAM) that relates fine-grained, small-scale symptom cues to large-scale structural changes across feature pyramid levels.

Our proposed DFF-Net has achieved the highest top-1 accuracy of 98.6% and 84.3% in the PlantVillage [2] and PlantDoc [7] benchmarks, respectively, and this is better than all the methods reported in the literature. Extensive ablation studies are performed to demonstrate the effectiveness of each of the proposed components. The rest of this study is organized as follows: Section 2 provides a literature review, Section 3 provides a detailed description of the proposed DFF-Net architecture, Section 4 describes the experimental setup and results, Section 5 discusses the results elaborately, and Section 6 concludes the study and provides future directions.

3.1 Architecture overview

The DFF-Net model processes the input image $x\in {{\mathbb{R}}^{\left\{ \text{H }\!\!~\!\!\text{ }\times \text{ }\!\!~\!\!\text{ W }\!\!~\!\!\text{ }\times \text{ }\!\!~\!\!\text{ }3 \right\}}}$ through two parallel branches, i.e., (1) a CNN branch based on EfficientNet-B4 to extract multi-scale local texture and color features, and (2) Swin Transformer branch to capture hierarchical global context across the whole leaf, and the feature maps from both branches are first aligned in their channel dimensions and then fused using the proposed CGFA module, and refined using the HSAM, and finally input to a classification head to obtain the disease prediction (Figure 1).

tu_pian_17.jpg

3.2 Convolutional neural network branch: EfficientNet-B4 backbone

The CNN branch employs an EfficientNet-B4 [12] pre-trained on ImageNet-22k, and it produces features at multiple scales: H/4, H/8, H/16, and H/32 with channel numbers 24, 48, 120, and 272, respectively. We take the last feature map $F_{\text{cnn}}^{4}$ of resolution (H/32) × (W/32) × 272 and transform it to dimensionality d = 512 to match the other branches through a 1 × 1 convolutional layer, batch normalization (BN), and rectified linear unit (ReLU), as illustrated in Eq. (1), because this transformation is necessary for the processing steps:

$\hat{F}_{\mathrm{cnn}}=\mathrm{BN}\left(\operatorname{ReLU}\left(W_c \cdot F_{\mathrm{cnn}}^4\right)\right)$          (1)

where, ${{\hat{F}}_{\text{cnn}}}\in {{\mathbb{R}}^{\left\{ \left( H/32 \right)\times \left( W/32 \right)\times 512 \right\}}}$, W_c is the weight matrix of the 1 × 1 projection layer. Because of the compound scaling of EfficientNet, the CNN branch can efficiently learn the fine-grained texture information, which is crucial to distinguish subtle visual cues, e.g., necrotic lesions, chlorotic halos, and sporulation patterns.

3.3 Transformer branch: Hierarchical Swin backbone

The Transformer branch is based on Swin Transformer-Base [5] pre-trained on ImageNet-22k. The input image is first split into non-overlapping patches, and then these patches are grouped into windows of size M × M, where M is 7 by default, and local self-attention is computed within each window. To enable information exchange across different windows, therefore capturing global context without introducing quadratic computational cost, Swin adopts a shifted-window approach in alternate layers, which shifts the window partition by a fixed offset before applying self-attention.

Each Swin Transformer block applies two sequential operations, as shown in Eqs. (2) and (3):

${{z}_{{{l}'}}}=W-MSA\left( LN\left( {{z}_{\left\{ l-1 \right\}}} \right) \right)+{{z}_{\left\{ l-1 \right\}}}$         (2)

${{z}_{l}}=FFN\left( LN\left( {{z}_{{{l}'}}} \right) \right)+{{z}_{{{l}'}}}$         (3)

where, z_l denotes the token sequence at layer l, and W-MSA stands for window-based multi-head self-attention. LN denotes layer normalization, and FFN denotes a feed-forward network with Gaussian error linear unit (GELU). By patch merging, the Swin Transformer builds up a 4-stage feature pyramid, where the channels are 96, 192, 384, and 768 for stages 1 to 4. We adopt the stage-4 feature map $F_{\text{tr}}^{4}$ as our input feature, which has a size of (H/32) × (W/32) and 768 channels. It is transformed into a common embedding size d = 512 via a linear projection (see Eq. (4)).

${{\hat{F}}_{\text{tr}}}=\text{LN}\left( {{W}_{t}}\cdot F_{\text{tr}}^{4} \right)$            (4)

where,$\text{ }\!\!~\!\!\text{ }{{\hat{F}}_{\text{tr}}}\in {{\mathbb{R}}^{\left\{ \left( H/32 \right)~\times ~\left( W/32 \right)~\times ~512 \right\}}}$.

3.4 Cross-Guided Feature Aggregation module

The CGFA module fuses F̂_cnn and F̂_tr through bidirectional channel-attention gates. Unlike independent squeeze-and-excitation (SE)-style recalibration [19], CGFA conditions each branch's attention weights on the partner branch's global-pooled representation, enabling guided mutual enhancement.

Let g_cnn = GAP(F̂_cnn) $\in ~$ℝ⁵¹² and g_tr = GAP(F̂_tr) $\in $ ℝ⁵¹² be the global average-pooled descriptors. The cross-attention gates are computed as shown in Eqs. (5) and (6):

${{\alpha }_{\text{cnn}}}=\sigma \left( \text{W}{{2}_{\text{c}}}\cdot \delta \left( \text{W}{{1}_{\text{c}}}\cdot \left[ {{g}_{\text{cnn}}};{{g}_{\text{tr}}} \right] \right) \right)$           (5)

${{\alpha }_{\text{tr}}}=\sigma \left( \text{W}{{2}_{\text{t}}}\cdot \delta \left( \text{W}{{1}_{\text{t}}}\cdot \left[ {{g}_{\text{tr}}};{{g}_{\text{cnn}}} \right] \right) \right)$          (6)

where [·;·] denotes channel-wise concatenation, δ is the ReLU activation, σ is the sigmoid function, W1 $\in ~$ℝ^{{1024 × 256}} and W2 $\in $ ℝ^{{256 × 512}} are learned weight matrices. The gated features are obtained by channel-wise scaling, as in Eqs. (7) and (8):

${{\tilde{F}}_{\text{cnn}}}={{\hat{F}}_{\text{cnn}}}\otimes {{\alpha }_{\text{cnn}}}$           (7)

${{\tilde{F}}_{\text{tr}}}={{\hat{F}}_{\text{tr}}}\otimes {{\alpha }_{\text{tr}}}$         (8)

The operator $\otimes $ denotes channel-wise multiplication (broadcasting over spatial dimensions). The fused representation is obtained by element-wise addition followed by a 3 × 3 depth-wise separable convolution for local context refinement, as expressed in Eq. (9):

${{F}_{\text{fused}}}=\text{DWCon}{{\text{v}}_{\left\{ 3~\times ~3 \right\}}}\left( {{{\tilde{F}}}_{\text{cnn}}}+{{{\tilde{F}}}_{\text{tr}}} \right)$           (9)

3.5 Hierarchical Symptom-Aware Module

Plant disease symptoms manifest at different spatial scales: some appear as small, high-frequency texture anomalies that are only visible at fine resolutions (e.g., localized lesions, spots, and discoloration), while others involve large-scale changes to the overall leaf shape and structure that are better captured at coarse resolutions. HSAM explicitly models such multi-scale dependencies by incorporating intermediate feature maps from both branches across the feature hierarchy.

For each pair of corresponding stage features ($F_{s}^{\text{cnn}}$, $F_{s}^{tr}$) at stage s $\in $ {2, 3, 4}, HSAM applies a cross-scale attention operation defined in Eq. (10):

${{H}_{s}}=CA\left( F_{s}^{\text{cnn}},F_{s}^{tr} \right)\oplus Up\left( {{H}_{\left\{ s+1 \right\}}} \right)\text{ }\!\!~\!\!\text{ }\left( \text{for }\!\!~\!\!\text{ }s=4,3,2 \right)$            (10)

The CA denotes the channel attention applied to the features of both branches and represents the element-wise summation operation after spatial alignment. Up denotes the 2-times expansion operation by bilinear upsampling; H₄ is set to F_fused at the lowest level. The final representation, H₂, fuses the symptom clues at all scales and is fed into the classification head.

3.6 Classification head and loss function

The classification head applies global average pooling to H₂, yielding a descriptor v $\in $ ℝ⁵¹². This is passed through a dropout layer (p = 0.4) and a fully connected layer to produce logits for C disease classes. The training objective combines standard cross-entropy with label-smoothing regularization, as formulated in Eq. (11):

$L={{L}_{\text{CE}}}+\lambda \cdot {{L}_{\text{LabelSmooth}}}$           (11)

where, $\lambda =0.5,~~\varepsilon =0.1.$ L_CE denotes cross-entropy loss, L_LabelSmooth is the label-smoothing regularization term with smoothing factor ε = 0.1, and λ = 0.5 is the balancing coefficient. Label smoothing prevents over-confident predictions on visually ambiguous disease categories.

The experiments validate our main hypothesis: the complementary CNN and Transformer features, when fused using guided cross-attention, yield consistently higher accuracy than either stream alone, as observed across all three benchmark datasets (Section 4). The CGFA module's bidirectional conditioning (Eqs. (5)–(6)) allows the CNN branch to concentrate its local texture extraction in regions which the Transformer branch has globally found important, effectively implementing a lightweight but very effective form of visual attention guidance.

Eq. (10) bridges fine-grained, small-scale symptom cues and large-scale structural changes, akin to the way expert agronomists jointly examine subtle local anomalies and the overall condition of the leaf to diagnose plant diseases. Huo et al. [18] similarly showed that hierarchical multi-scale feature fusion is central to top-performing deep learning models for medical image classification, and our results demonstrate that this observation applies to plant pathology as well, with multi-scale feature integration proving to be an effective strategy specifically for fine-grained visual disease classification.

To this end, we can view the CGFA module as a learned feature selection technique that occurs jointly in the CNN and Transformer feature spaces, while conceptually similar to metaheuristic-based feature selection techniques explored in other medical and biological image classification tasks [16, 17]. Our framework differs from this technique in that the selection is done end-to-end through gradient-based optimization rather than external search, and our results are also in agreement with the general consensus in the literature: methods that selectively use features, either through metaheuristic search [16, 17] or attention-driven fusion, outperform naive feature concatenation.

Limitations. Our approach has several limitations; firstly, our method takes about 45 ms per image on an RTX 3080 (single NVIDIA RTX 3080 GPU, separate from the four-GPU RTX 3060 training cluster; batch size of 1; 16-bit floating-point (FP16) mixed-precision inference; 384 × 384 input resolution; measured after a 20-iteration GPU warm-up; the reported time includes image pre-processing but excludes data-loading I/O). It may not be applicable to low-power edge devices or large-scale real-time systems because it requires significant computational resources. Secondly, on IP102 [20], the accuracy drops to 76.4%, indicating that extremely imbalanced classes and high visual diversity are still challenging problems that our architecture fails to address, thus highlighting the need for further improvement. Lastly, the model processes images independently without considering temporal information, which could provide insights into disease progression over time, therefore limiting its ability to capture dynamic changes. Future work. In future work, we plan to explore knowledge distillation for a mobile-friendly variant of DFF-Net, and we will also investigate semi-supervised variants utilizing massive amounts of unlabeled field images. These directions are motivated by promising outcomes of semi-supervised learning and knowledge-distillation techniques reported in the broader computer vision literature.