Content-Based Remote Sensing Image Retrieval Based on Two-Way Feature Representation Using ResNet50 and Modified Multiscale Local Ternary Pattern

Content-based image retrieval (CBIR) is the process of retrieving an image from a large dataset based on its visual content, rather than relying on keywords and metadata. CBIR uses spatial information, color, texture, and shape features for image retrieval [1]. It is essential in various medical image retrieval systems, e-commerce applications, digital art archives, satellite and remote sensing, digital libraries, and security and surveillance [2]. CBIR for remote sensing images is very challenging, as Remote sensing generates a considerable volume of aerial, satellite, and drone-captured images daily. Traditional CBIR depends on timestamps, manually annotated descriptions, and location tags. These methods are often inconsistent, time-consuming, unreliable, and subjective, failing to adequately represent the visual content of images [3]. Thus, CBIR is crucial for enabling efficient, automated, and precise retrieval of remote sensing images based on actual visual content, rather than relying on tags and text-based annotations. The necessity of CBIR for remote sensing is due to the higher complexity of remote sensing images. The CBIR is widely utilized in various remote sensing applications, including environmental monitoring and climate change, disaster management, agricultural monitoring, land use and planning, as well as military and security applications [4]. The CBIR for remote sensing is useful for automating image analysis, handling larger-scale datasets, retrieving geospatial data, and supporting multispectral and hyperspectral data [5].

Various automated machine learning (ML) and deep learning (DL) schemes have been proposed for the CBIR to enhance the effectiveness of multimedia content retrieval [6-10]. Yuan et al. [11] proposed a Cross-modal Remote Sensing Text-Image Retrieval approach utilizing GaLR for image-text retrieval, assessed on the Remote Sensing Image Captioning Dataset (RSICD) and Remote Sensing Image–Text Matching Dataset (RSITMD) datasets. This approach enhances retrieval accuracy to 31.41% in RSITMD by dynamically integrating global and local features, thus improving retrieval precision. Nonetheless, excessive redundancy in local features may occasionally result in diminished performance. Cheng et al. [12] proposed a deep semantic alignment network (DSAN) for cross-modal retrieval, which was tested on multiple captioned datasets, including UC Merced Land Use-Captions and NWPU- RESISC45-Captions. By enhancing cross-modal retrieval through semantic alignment, it achieves more than 90% classification accuracy on the UCM dataset. Despite its high performance, it struggles with lower accuracy in visually similar categories, which may limit its effectiveness. Zhang et al. [13] developed a triplet non-local neural network with dual-anchor triplet loss and applied it to UCM dataset, the aerial image dataset, and the PatternNet dataset. This approach significantly improves retrieval performance, particularly on the UCM dataset, by leveraging better discriminative learning. However, its accuracy is limited when dealing with images that have simple backgrounds, which may reduce its applicability in specific scenarios.

Sumbul et al. [14] proposed a deep metric learning approach using triplet selection for multilabel CBIR, tested on IRS-BigEarthNet and UC Merced datasets. Efficient triplet selection enhances multilabel image retrieval while reducing training complexity. However, selecting optimal triplets remains a challenge, especially for large-scale datasets. Mi et al. [15] introduced a knowledge-aware cross-modal retrieval (KCR) framework utilizing external knowledge graphs for retrieval, evaluated on two standard remote sensing benchmarks. The method outperforms state-of-the-art retrieval techniques by mitigating the semantic gap through the use of knowledge graphs. Nevertheless, its reliance on external knowledge graphs may introduce bias, which can affect the consistency of retrieval results. Al Rahhal et al. [16] presented a multilingual Transformer-based retrieval method for text-to-image matching, which was tested on the RSITMD, RSICD, and UCM datasets. This model improves retrieval performance by 11% compared to previous approaches, supporting multilingual queries and thereby increasing its versatility. However, its performance varies depending on dataset size and language complexity, which could impact its usability across different applications.

Song et al. [17] proposed an asymmetric hash code learning (AHCL) technique for remote sensing image retrieval, which was applied to three publicly available datasets. It achieves the highest mean average precision (MAP) across multiple datasets, reducing computational costs through efficient hashing while maintaining retrieval accuracy. However, optimal performance requires precise tuning of hash functions, a process that can be resource-intensive. Zhang et al. [18] presented deep multi-similarity hashing (DMsH) for remote sensing image (RSI) retrieval by combining spatial information and hashing information to mitigate noise. It provided a mean average precision of 0.97 for the UCM dataset. Maurya et al. [19] suggested an adaptive DL-based model for the RSI retrieval to minimize the limitations of poor retrieval, lack of flexibility, and lower efficiency. They analyzed the effectiveness of various transfer learning models on the UCM dataset for image retrieval. It achieved a superior accuracy of 95.07% for VGG19, compared to 93% for VGG16 and 91% for ResNet, in a 21-class classification task. However, the higher number of trainable parameters in the VGG19, VGG16, and ResNet models leads to an increase in the computational complexity of the system. It limits its deployment on systems with limited resources.

In recent years, Chembian et al. [20] proposed the k-means pelican optimized algorithm (KMPOA) for the CBRSIR, aiming to reduce the search space and enhance the retrieval of RSIs. It used the RSIs’ grey-level cooccurrence matrix (GLCM), color moments, LBP, and ResNet-18 features. Neighborhood component analysis (NCA) is used to select the most salient features. The KMPOA achieved a 96.29% accuracy for the ARIAL image dataset using CBRSIR. Hou et al. [21] explored an efficient and secure CBRSIR based on CNN. It used feature dimension reduction using mapping-based spectral hashing with spectral rotation (SHSR) and clustering to enhance retrieval accuracy. Further, the asymmetric scalar product-preserving encryption scheme is utilized to protect the image during retrieval. It provided better authenticity, security, and retrieval rate. However, the system’s effectiveness is challenging because of the CNN’s intricate structure and parameter optimization. Sudha and Aji [22] proposed an optimized label propagation network (OLPNet) for hierarchical CBRSIR. The system utilized a support vector machine and a relevance vector machine to boost the label distribution of the RSIs’ high-dimensional and complex features. The complexity in the framework results in higher time complexity, which limits the system's real-time deployment. Yang et al. [23] and Zhang et al. [24] proposed remote sensing text image retrieval based on descriptive text. Retrieving the RSIs based on the text inputs is challenging, as there can be a disparity between the metadata and the actual RSI content. Therefore, it is necessary to focus on the actual image content depicted using color, textural properties, and shape attributes [25, 26].

From the extensive survey of the various RSI retrieval schemes, it is observed that the outcomes of the existing systems are challenging because of the following research gaps:

• Poor spatial connectivity between the local and global representation of the RSIs, which leads to poor feature depiction.
• Higher computational complexity due to complex DL frameworks utilized for CBIR.
• Higher retrieval time due to the use of hashing for retrieval.
• Lower retrieval accuracy and MAP for higher classes.
• The metadata is stored with the RSIs, which is often irrelevant and depends upon the manual tagging and metadata provision to the content.

Existing CBRSIR systems face limitations, including weak spatial connectivity between local and global features, high computational complexity, long retrieval times, reduced accuracy for large-class datasets, and reliance on irrelevant, manually tagged metadata. To overcome these challenges, this work proposes a hybrid CBIR framework that combines deep learning and traditional descriptors, enabling richer feature representation, faster retrieval, and higher accuracy, particularly for complex and large-scale RSI datasets. The main contributions of the article are summarized as follows:

• Two-way feature representation of the remote sensing images using ResNet50, novel modified adaptive multi-scale LTP (MALTP), and histogram of oriented gradient (HOG). Here, ResNet50 is utilized to enhance the hierarchical and multilevel feature depiction of remote sensing images. The proposed MALTP is utilized to provide the fine texture details and better spatial correlation of the images, and an HOG is employed to depict the shape of the images.
• The features are embedded using novel block based discrete wavelet transform (BBDWT) into the RSIs to minimize the feature extraction time during the retrieval process and reduce the complexity of mapping metadata with images.
• The suggested CBIR uses a K-Nearest Neighbor (KNN) classifier based on the feature-level fusion of the ResNet50, HOG, and MALTP features to improve retrieval accuracy.

The remainder of this article is structured as follows. Section 2 details the methodology, Section 3 outlines feature embedding block based discrete wavelet transform technique, Section 4 presents the experimental results and corresponding discussion. Finally, Section 5 presents the conclusions and outlines the future direction for improving the proposed work.

The Discrete Wavelet Transform (DWT)-based data hiding helps to retain the structural and perceptual quality of the RSIs. The features are embedded in the RSIs as the blind watermarking techniques without disturbing the visual, perceptual, and structural quality of the RSIs. The embedded features are extracted during the retrieval process to compare the features of the stored images with those of the query images. This feature embedding helps to minimize the retrieval time of the CBRSIR. The original RSI is divided into blocks of smaller blocks, each with m×m pixels, to preserve the local texture and shape properties of the RSIs. The RSI is divided into three levels, utilizing a Daubechies-2 (Db2) filter to embed the feature vector (message) within the RSI, ensuring robustness and imperceptibility. The DWT decomposes the host RSI into four sub-bands: LL (Low-Low), LH (Low-High), HL (High-Low), and HH (High-High). At the first level, the image undergoes DWT decomposition, resulting in LL1, HL1, LH1, and HH1 sub-bands. The LL1 sub-band is further decomposed at the second level, yielding LL2, HL2, LH2, and HH2 sub-bands. Furthermore, third-level DWT decomposition provides subbands such as LL3, HL3, LH3, and HH3, enabling the creation of a hierarchical structure of frequency components. Using Db2 as the wavelet basis ensures better smoothness and frequency localization, making the data embedding less visible while maintaining robustness against attacks like compression and noise. The features are typically embedded in the LL sub-bands of the third-level decomposition, as they contain less important edge information and offer a good trade-off between robustness and imperceptibility. After embedding, an Inverse DWT is applied to reconstruct the watermarked image. This approach ensures that the features remain hidden from the human eye while being resilient to common image-processing attacks. Eq. (10) shows the embedding process. We have considered a block size of 2×2 to retain the subband’s overall structural and local perceptual quality.

$W M^{\prime}=(k \times L L 3)+(q \times w m)$  (10)

where, WM^' denotes the feature embedded image, k and q denote scaling factors, LL3 suggests the low-frequency element of the level 3 decomposed components of the original RSI, and wm means feature vector. The watermark image and the decomposed components of the cover after being multiplied by k are added. The scaling factor q is crucial in DWT-based feature embedding, as it balances imperceptibility and robustness. A higher q increases robustness against attacks but may introduce visible distortions, while a lower q preserves image quality but makes the feature vector more vulnerable. Typically, q ranges between 0.1 and 0.5, depending on the application and image characteristics. A value of 0.25 is chosen as the optimal trade-off, ensuring the features remain perceptually invisible while providing sufficient resistance against common attacks, such as geometric attacks, filtering, and noise. This value is particularly effective in RSIs, where preserving visual fidelity is essential while maintaining feature vector preservation under various transformations.

During retrieval, features are extracted from RSIs by applying the reverse process of feature embedding and are then compared with the features of the query image.