Adaptive Graph Convolution Deep Learning Model for Multilingual Hand Sign Recognition with Texture and Frequency-Based Features

Many works have been carried out in the area of sign language recognition due to this perpetual quest for an effective and efficient method of recognizing hand gestures. However, opportunities still abound in delivering results more accurate and robust than the existing approaches can muster. Recent research proposals attempt new techniques to clearly detect and analyze complex hand movements so that a two-way communication system involving somebody who is hard-of-hearing or deaf on one end but does not know sign language on the other can understand them. The related works are classified into three main categories for systematic review:

2.1 Studies related to the preprocessing stage

These works focus on the initial stages of developing grayscale and normalized images to enhance quality and maintain consistency. Common techniques include converting the image into grayscale, applying histogram equalization, Gaussian Blur, resizing the image as per requirement, and Normalization.

The main objective of preprocessing is to normalize abnormal input conditions and eliminate irrelevant variations that may badly affect learning stages later on. Several related papers to this stage have been analyzed for this purpose.

Ahmed et al. [2] developed an American Sign Language static image recognizer of 29 classes in the year 2025. The pre-processing steps on each frame are grayscale conversion, resizing to 224 × 224 pixels, normalization of the range 0–255 to [0,1], and noise removal by Gaussian/median filters. Three Tamura texture (descriptors- coarseness, contrast, directionality) are fused with raw image data at input level and fed into a pruned ResNet-50 backbone. Sequential Feature Selection further refines these features while tuning Generalized Additive Model (GAM) classifier using tenfold cross validation scheme. Average accuracy over all letters is about 96.7%.

Authors emphasize that this shallow Tamura–ResNet50–GAM hybrid comes close to YOLOv3-like deep detectors yet low resource hungry, however real time deployment plus generalization outside ASL dataset remain open issues.

In 2024, Abd Al-Latief et al. [3] first converted the RGB frames into grayscale, enhances the local contrast through histogram equalization, reduces noise, and then applies a contour-based segmentation method to crop out the hand region; finally resizes the cropped region to 50×50 for classification.

It thereafter applies WAR-Strategy meta-heuristic that prunes features and fine-tunes six classical ML classifiers-in achieving accuracies of 93.11–100% on American, Arabic, and Malaysian static-image datasets with training times reduced to 0.038–10 s across these dataset languages-with sub second inference on some models as clear strengths but is still limited to isolated frames with conventional ML pipeline scalability ceiling inherited.

2.2 Studies related to the feature extraction stage

Research in this category aims to identify and extract the most discriminative features from hand gestures, either through handcrafted descriptors or learned representations. There are many previous studies within this stage, the feature extraction stage.

In 2023, Park et al. [4] employed a combination of 2D-FFTand convolutional neural networks (CNNs) to address complex hand gesture input and noise caused by the external environment. 2D FFT have been used to convert time data (image) into frequency domain then applied normalization and resizing. The method’s reliance on specialized radar hardware is an additional cost and a five-word vocabulary limits broader sign-language applicability.

In 2023, Ahmed [5] introduces a hybrid Tamura–SIFT pipeline for texture features. The Tamura descriptors provide perceptual global cues (coarseness, contrast, directionality), and SIFT injects scale- and rotation-invariant key-points that represent fine local structure. Feature-level concatenation forms a composite vector that carries global statistics united with local gradients. It beats every single descriptor under changing illumination, scale, and view by a good margin in standard benchmarks. The richer signature improves retrieval, segmentation, and object-recognition accuracy across multiple datasets. The major limitation of the method is its high computational cost which restrains real-time deployment.

2.3 Studies related to the classification stage

This group covers studies that design and evaluate models for mapping gesture inputs to particular sign language classes. Models run the gamut from baseline SVMs and k-NNs up to state-of-the-art deep learning models, including CNNs and even Graph Neural Networks (GCNs). A couple of papers also venture hybrid models toward accuracy and generalization enhancement.

In 2024, Miah et al. [6] proposed a two-stream GCAR model in which an effective combination of spatial and temporal information is achieved through GCN, sep-TCN, and channel attention. The architecture produces very high accuracies on big largescale datasets such as WLASL and PSL because it represents full-body dynamics and was designed to effectively manage joint discontinuity drawbacks. However, results remain modest for ASLLVD and this cannot be applied in real-time due to the complexity of the model.

In that year, a three-layer GCN with successive residual connections applied to 21-landmark hand graphs attains 99.1% accuracy on the ASL-Alphabet dataset. The model is lightweight, enjoys stable gradients, and resists over-fitting, but it speaks only to static letters, ignores temporal dynamics, and generalizes poorly outside of America where the landmark detector typically does not work [7].

In 2023, Vasudevan et al. [8] introduced WaveMesh superpixels and WavePool pooling as inputs to SplineCNNs that outperform SLIC on MNIST, Fashion-MNIST, and CIFAR-10. The method has strengths in adaptive multiscale graph construction but suffers from computationally expensive wavelet preprocessing per image, requires tuning for each image, and has not been tested on skeleton-based sign-language data.

In 2022, Han et al. [9] proposed Vision GNN (ViG) as an image backbone representing each patch as a graph node and by alternating Grapher and FeedForward blocks to overcome the over-smoothing problem. ViG-S achieves 82.1 % Top-1 on ImageNet. Its strength lies in scalable patch-level graph reasoning that flexibly models spatial context. However, the use of fixed k-NN graph construction and the computational overhead of graph building, along with larger model variants, may limit its suitability for real-time sign-language recognition, which requires fine-grained hand-joint modeling and temporal information.

Figure 4 presents the research framework of the proposed system. It summarizes the complete pipeline, starting from image preprocessing and feature extraction, followed by k-NN graph construction and data splitting, then adaptive GCN training and testing. Finally, the model is evaluated using standard performance metrics, including accuracy, precision, recall, and F1-score.

This study hereby develops an adaptive GCN deep learning model which uses the skeleton keypoints extracted for one particular word as input. The model makes a GCN network and inserts layers of CNN, GRU, and LSTM within the internal structure of the network utilizing their properties to come up with a generalized powerful model that could be used for sign language recognition. To validate this proposed model, two comprehensive datasets covering both American Sign Language (ASL) and ArSL were used to test how versatile and broadly applicable it is. This framework presents the system under five stages: preprocessing stage, features extraction stage, training stage, testing stage, and evaluation stage shown in Figure 4.

Figure 4. Research framework

4.1 Preprocessing

Preprocessing of the hand sign image dataset involves steps meant to improve visual quality, reduce computational requirements, and provide favorable conditions for extracting features. It starts with grayscale conversion whereby RGB images are transformed into single channel presentations. This reduces data dimensionality and the cost of computation while maintaining shape and texture clues that are very relevant towards achieving recognition accuracy. Next, histogram equalization is applied to enhance contrast through better utilization of the full range by setting pixel intensities to be more evenly spread out; detail is increased on an image surface and minute variations supporting pattern recognition become apparent. Images are normalized whereby pixel values get scaled within a standard range (most commonly between 0-1) so as to keep consistency across the entire dataset thus enabling steady model training.

The pictures are then run with a Gaussian blur, which takes away high sounds and smooths out unwanted looks, making it less likely that noise will cause mistakes when getting features. A step that comes right after normalizing keeps the same level of data spread after this smoothing. Changing the size of the image helps to make all examples the same size, allowing for consistent input handling and quick calculation during both training and testing. Another round of normalization makes sure data is steady before going into the feature-getting and deep learning steps, thus making the whole finding system better accurate strong and general.

4.2 Features extraction

The feature extraction becomes the core of a proposed framework for sign language recognition because it describes how the most salient and informative features are drawn out from preprocessed images of hand signs. Three complementary extraction techniques-PCA, Fast Fourier Transform (FFT), and Tamura texture descriptors-are used in this work to provide varied viewpoints on data. As the first step, PCA is applied as a feature-extraction transform to generate uncorrelated, energy-ordered scores that serve to enrich representation. Only the first ten principal components have been retained in this study as they contain most of the discriminative information and thus represent the strongest relevant features necessary for classification. Next, FFT will be applied such that image information will be converted from the spatial domain into its frequency domain. Periodic structures and subtle variations are well captured in their frequency domains which can never be seen from a spatial perspective. This step helps much when periodic textures need recognition and minute details regarding gestures are required. Tamura features encode those texture characteristics in terms of a human perceptual attribute with dimensions such as coarseness, contrast, and directionality. Such subjective intake makes the model more sensitive to fine changes in shape and texture. Normalization ensures equal participation for all features at this and subsequent stages, particularly when a feature set does not overly dominate results due to scale differentials. After normalization has been completed, these features from the PCA, FFT, and Tamura methods are appended together and stored within CSV files combining such miscellaneous features into a single representation will guarantee very wide and strong coverage over all possible visual and structural characteristics that exist within the dataset. Once the properties of each image have been saved as a vector, the KNN algorithm runs between every two vectors to find the edges among the nodes- hence, images features vectors in this chunk by measuring their degree of closeness so that they can be inputted into GCNs layers which require data structured as a graph of nodes and edges. The consolidated feature dataset is split into 70% training data and 30% testing data to comprehensively assess and validate how well the model generalizes, i.e., on unseen scenarios and datasets how reliable and effective recognition systems might turn out to be.

4.3 Graph construction procedure (K-Nearest Neighbors graph per mini-batch)

A separate graph is constructed for each mini-batch (chunk) of the dataset in order to provide a graph structure for subsequent graph convolution operations. The mini-batch size is 32 and each image features vector represnts anode in the graph. The resulting graph for each batch contains exactly 32 node. The graph is constructed using a k-nearest neighbor rule with K=2 under the Euclidean distance, meaning that each node is connected to its two closest neighbors in the feature space. Because the neighborhood relation is computed independently for each node, the resulting k-NN adjacency is generally asymmetric and the graph is directed. The outgoing degree of every node is fixed at 3 outgoing edges (two edges to the nearest neighbors plus one self-loop), whereas the incoming degree is not fixed and varies depending on how frequently a node is selected as a nearest neighbor by other nodes.

4.4 Classification stage

All important features are extracted in previse stage to use in the current stage. Where the classification stage is central to the effectiveness of the proposed methodology, transforming refined, high-dimensional feature vectors into accurate and interpretable class predictions. To this end, we designed a structured yet adaptable framework integrating graph-theoretic insights with advanced deep neural network architectures. This careful integration allows the model to capitalize on both spatial and temporal complexities present in multilingual hand gesture datasets, thus enhancing generalizability and accuracy across different languages, particularly Arabic and English.

4.4.1 Training stage: Adaptive Graph Convolutional representation

A more elaborate graph-based recognition pipeline where a k-nearest-neighbor (k-NN) graph injects relational structure into otherwise independent handcrafted descriptors. In this setup, feature vectors become nodes and edges represent proximity under some distance metric, typically the Euclidean distance as a special case of Minkowski. Then, information for one node embedding can be propagated by stacking Graph Convolutional Network (GCN) layers such that each embedding will have not only its own attributes but also those of the immediate local neighborhood.

Each GCN layer works by building a k-nearest neighbor (k-NN) graph. The single feature vectors shown as nodes in the graph, while edges show the closeness-based similarity among these vectors. Flexible behavior is done through changing graph updates, where joining links change during training to best show natural data structures across Arabic and English datasets. Figure 5 shows an adaptive GCNs model layers. The Adaptive GCNs layers are penetrated sequentially, meaning that the extracted features forms from each layer serve as an entry point to the next layer.

Figure 5. Adaptive Graph Convolutional Networks (GCNs) model

The Adaptive GCNs model architecture that progressively transforms hand-sign feature vectors into a robust representation suitable for accurate multi-class recognition across heterogeneous datasets. First, an adaptive neighborhood structure is introduced by constructing a k-nearest-neighbor graph, where each feature vector is treated as a node and edges encode proximity-based similarity. Two consecutive GCN layers then perform neighborhood aggregation using the normalized adjacency matrix. This stage injects relational context into otherwise independent descriptors and enables the model to learn similarity-consistent patterns by propagating information across nearby nodes, thereby improving robustness to intra-class variation and reducing sensitivity to dataset-specific capture conditions.

Next, a stack of convolution–activation–pooling blocks refines the graph-enriched embedding's by learning higher-level, locally compositional patterns while max-pooling compresses the feature maps and increases invariance to minor perturbations. A recurrent stage, implemented as a GRU followed by an LSTM, is subsequently applied to model sequential dependencies within the learned representations and to capture both short- and long-range correlations, which further stabilizes recognition under variability and improves generalization. Finally, the resulting representation is flattened and passed through fully connected layers to produce class probabilities. Overall, by combining relational learning (GCN), hierarchical pattern abstraction (CNN), and dependency modeling (GRU/LSTM), the proposed architecture enhances feature robustness and is explicitly designed to generalize effectively across multiple sign-language datasets rather than overfitting to a single dataset distribution.

4.4.2 Testing stage

In this stage, the trained adaptive GCN model undergoes evaluation using the test subset (30% of the dataset, as per the provided document).

4.5 Evaluation stage

Precision, recall, and F1-score measures are also calculated in Figure 4 explain the results of these measures for the proposed model. It had the highest values over all these measures.

Accuracy measures the overall correctness of the model's predictions and is calculated as the proportion of correctly classified instances (both true positives and true negatives) relative to the total number of predictions. It is formally defined in Eq. (13), utilizing the standard classification components: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) [20].

$Accuracy =\frac{T P+T N}{T P+T N+F P+F N}$                          (13)

The ratio of True Positives to All Positives is known as precision. It is formally defined in Eq. (14) [21].

$precision=\frac{T P}{T P+E P}$                          (14)

The Recall (R) is the measure of the model correctly identifying True Positives. It is formally defined in Eq. (15) [22].

$Recall=\frac{T P}{T P+E N}$                        (15)

The weighted average of Precision and Recall is known as F1-score. It is calculated through Eq. (16) [23].

$F1-score =\frac{2 . { Precision } * { Recall }}{{ Precision }+ { Recall }}$                  (16)