ViT-FESEL: A Multi-Stage Vision Transformer-Based Feature Extraction and Selection Framework for Osteoporosis Classification from Knee X-Ray Images

2.1 Dataset

In this study, the Multi-class Knee Osteoporosis X-ray Dataset [14], available as open source, was used to automatically classify knee joint osteoporosis. The dataset consists of 1,573 knee X-ray images from different individuals and is a multi-class dataset divided into three classes: clinically normal, osteopenia, and osteoporosis (Figure 1). In this study, within the scope of binary classification, the osteopenia and osteoporosis classes were combined into a single pathological class, resulting in a total of 793 images; the normal class consists of 780 images. The images do not contain any manual annotation or regional labeling; they reflect structural changes in bone mineral density and differences in trabecular patterns, incorporating the variation in resolution, contrast, and imaging conditions encountered in actual clinical practice. In this respect, the dataset provides a highly representative and realistic resource for modeling osteoporotic changes specific to the knee region. Within the scope of the study, the dataset was used in multiple ways to evaluate different clinical decision scenarios. In binary classification experiments, the osteopenia and osteoporosis classes were combined into a single pathological class, and classification was performed against the normal class. In the ordinal classification scenario, three classes were preserved to model the natural ordinal structure of disease severity. In ensemble-based experiments, the entire dataset was used to combine representations obtained from different ViT backbones. All experiments were conducted using a k-fold cross-validation scheme that preserved the class distribution; feature selection and model training were performed only on training subsets to prevent information leakage into the test data.

1.png

Figure 1. Examples of the dataset

The X-ray images in the dataset exhibit variability in resolution and contrast levels, reflecting the imaging conditions commonly encountered in clinical practice. The images do not contain any manual markings or regional annotations, and are organized solely by class labels. This characteristic enables the dataset to be directly applicable to both traditional machine learning and deep learning–based approaches. Furthermore, the multi-class nature of the dataset provides a suitable basis for studies aimed not only at identifying advanced-stage osteoporosis, but also at distinguishing early-stage bone loss conditions such as osteopenia.

2.2 Vision Transformer-Based Feature Extraction and Selection

In this study, a novel and comprehensive method, ViT-FESEL, is proposed for osteoporosis classification from knee X-ray images. The main motivation behind ViT-FESEL is that disease-specific findings in musculoskeletal radiographs are not limited to local tissue characteristics but are also strongly associated with the overall morphological structure of bone and global contextual relationships. In this regard, unlike convolutional neural network (CNN)–based architectures with limited receptive fields, ViT architectures leverage global self-attention, enabling more effective modeling of long-range dependencies across different image regions.

The proposed ViT-FESEL framework is a multi-branch architecture designed to address the problem from multiple clinical decision-making perspectives. This structure consists of three main components:

• Branch-A aims to perform binary classification by extracting ViT-based deep representations, applying feature selection, and employing classical machine learning classifiers.
• Branch-B focuses on ViT-based ordinal classification by explicitly accounting for the natural ordering of osteoporosis classes.
• Branch-C provides a multi-scale, more generalizable ensemble-based decision mechanism by fusing complementary representations from different ViT variants.

This multi-branch design reduces dependency on a single architecture while allowing each branch to model different levels of information. Overall, ViT-FESEL integrates transformer-based representation learning, dimensional refinement through feature selection, and task-specific classification strategies within a unified framework. By decoupling representation learning from decision-making processes and formulating the method to be adaptable to binary, ordinal, and ensemble-based scenarios, the proposed approach aims to deliver both high classification performance and clinically meaningful, stable outputs.   

2.2.1 Vision Transformers–based architecture

In this study, the ViT architecture is employed during the deep representation learning stage. The primary reason for preferring ViT architectures is their ability to model images not only through local spatial windows, as in convolution-based networks, but also through global contextual relationships. In knee X-ray images, osteoporosis-related findings are not restricted to local bone tissue alterations; rather, they are closely associated with the overall bone structure and the relationships among different anatomical regions. Therefore, the self-attention mechanism in ViT offers a significant advantage for capturing such structural dependencies. 

The ViT architecture divides the input image into fixed-size patches and treats these patches as a sequence [15-18]. Each patch is transformed into a fixed-dimensional embedding vector via a linear projection, which is then used as an input token to the transformer architecture [19]. To support the classification task, an additional learnable [CLS] token is appended to the sequence. This token aggregates information from all patches across the transformer layers and forms a global representation of the image [20, 21]. 

Since transformer architectures do not inherently encode spatial order, positional embeddings are used to encode patch locations within the image [22]. Positional embeddings are added to each patch token, enabling the model to learn the relative positions of anatomical structures in knee X-ray images [23]. Consequently, the ViT model generates representations that account for both content information and spatial arrangement. The fundamental building blocks of the ViT architecture are transformer encoder blocks, composed of Multi-Head Self-Attention and fully connected feed-forward networks [24]. The self-attention mechanism allows each patch to simultaneously attend to all other patches, enabling the model to capture correlations between distant anatomical regions and to more effectively learn global structural patterns associated with osteoporosis. Residual connections and layer normalization employed within the encoder blocks facilitate stable training of the deep architecture [25, 26].

In this study, ViT variants with different capacities and patch sizes are employed to enhance the model’s multi-scale representation learning capability. Smaller patch sizes enable a more detailed capture of local tissue characteristics, whereas larger patch sizes emphasize the image's global structural organization. This design choice lays the foundation for constructing rich and complementary representation spaces that are subsequently exploited by the feature selection and ensemble strategies within the ViT-FESEL framework.  

Finally, the [CLS] token representation from the transformer encoder layers is treated as a high-level summary feature vector of the image. This representation is utilized for different purposes across the branches of the ViT-FESEL method. In the binary and ensemble-based classification branches, this vector is forwarded as input to the subsequent feature selection and classical classification stages. In contrast, within the ordinal classification branch, it is directly connected to the ordinal learning head, enabling end-to-end learning. 

2.2.2 ViT-based feature extraction and feature selection (FESEL)

ViT architectures can generate high-dimensional, rich representations from input images. However, not all components of these representations are equally discriminative for the classification task. In medical X-ray images, data-related noise, variations in imaging conditions, and subtle inter-class differences may lead to redundancy and low-discriminative dimensions in the learned embedding space. This issue can adversely affect the model’s generalization performance, particularly in limited-data scenarios. For this reason, ViT-based feature extraction within the ViT-FESEL framework is supported by a systematic feature selection stage (FESEL).

In this study, the [CLS] token representation obtained from the ViT backbone for each knee X-ray image is regarded as a high-level summary feature vector of the image. This vector is extracted from the final encoder block of the ViT model and encodes the image's global structural information. Rather than being directly fed into the classifier, this deep representation vector is subjected to a feature selection process to improve classification performance and control model complexity. 

The primary objective of the feature selection process is to identify, from the high-dimensional representation space learned by ViT, the subspace that best reflects inter-class separability and is most beneficial for generalization. Accordingly, the FESEL stage is designed with two main goals:

• preserving features with high discriminative power among classes, and
• eliminating features that contribute little to the classification process or contain noise. 

This strategy reduces computational cost and mitigates the risk of overfitting. 

Mathematically, the feature vector extracted from the ViT backbone can be denoted as f∈RD. In the FESEL stage, this vector is mapped to a lower-dimensional subspace and represented as f∗∈Rd with d≪D. This transformation is not merely a dimensionality reduction operation, but rather a selection of features that contain discriminative information. The selected feature subset is determined solely from the training data in each cross-validation fold, thereby preventing data leakage into the test set. In the FESEL stage, a two-step dimensionality reduction is applied: in the first step, MI-based selection retains the top 512 most informative features from the high-dimensional ViT representation vector; in the second step, NCA reduces this to a final feature dimensionality of d = 128. These settings were kept constant across all backbone architectures and cross-validation folds. 

The integration of the FESEL mechanism with ViT-based representations establishes a hybrid architecture that effectively combines the strengths of deep learning and classical machine learning approaches. While the ViT model learns complex, high-level visual patterns, the FESEL stage selects the most informative components of these patterns for classification, thereby enabling classical classifiers to learn more effectively. This decoupling enhances the modularity of the ViT-FESEL framework and allows different classification strategies to be evaluated on the same representation space.

The refined representation vectors obtained after feature selection are provided as inputs to classical machine learning–based classifiers within the Branch-A and Branch-C components of the ViT-FESEL method. By combining the strong representation-learning capabilities of deep models with the flexibility of classical algorithms for decision boundary formation, this structure aims to deliver more stable, generalizable classification performance for osteoporosis detection in knee X-ray images.

2.2.3 Branch-A: ViT-FESEL–based binary classification approach

Branch-A forms the core and reference pipeline of the proposed ViT-FESEL method and is designed for scenarios in which osteoporosis status is treated as a binary classification problem (presence/absence) based on knee X-ray images. The main objective of this branch is to demonstrate the extent to which ViT-based deep representations, systematically refined through feature selection and used in conjunction with classical machine learning classifiers, can produce effective, stable, and generalizable results. In this context, Branch-A presents both the simplest implementation of the proposed framework and provides a strong basis for comparison for more complex ordinal and ensemble-based scenarios. As the first step in this pipeline, each input X-ray image sequence is processed by the selected ViT backbone, yielding a high-dimensional representation vector that captures the image's global structural features and potentially related contextual patterns. However, such ViT-based representations, due to their high dimensionality and limited discriminative components, can lead to performance fluctuations and overfitting risks when used directly in the classification stage. Therefore, in Branch-A, the ViT-based representation learning process is supported by a feature selection (FESEL) stage. 

In this pipeline, as a first step, each input X-ray image is processed by the selected ViT backbone, yielding a high-dimensional representation vector that captures the image's global structural features and potentially associated contextual patterns with osteoporosis. However, such ViT-based representations, due to their high dimensionality and limited discriminative components, can lead to performance fluctuations and overfitting risks when used directly in the classification stage. Therefore, in Branch-A, the ViT-based representation learning process is supported by a feature selection (FESEL) stage.

In the FESEL phase, deep representation vectors generated by ViT are analyzed using only training data, and the feature subset that best distinguishes between classes is determined. In this study, Mutual Information (MI) and Neighborhood Components Analysis (NCA) methods were used within the FESEL mechanism. MI-based feature selection aims to identify the most informative components for classification by measuring the statistical dependence of each feature on class labels, while the NCA method offers a feature weighting and selection strategy that maximizes the distinction between classes by considering the neighborhood relationships between examples. Thanks to these two approaches, features that are strongly related to class information and that define decision boundaries more clearly are retained, while redundant or noisy dimensions are eliminated. The feature selection process was performed independently for each cross-validation fold, thus preventing information leakage to the test data.

The more compact, discriminative feature space obtained through feature selection enables classical machine learning classifiers to learn decision boundaries more stably. This study focuses on methods that can effectively model nonlinear decision boundaries and perform well in high-dimensional feature spaces; in this context, XGBoost and RBF-kernel Support Vector Machines (SVM-RBF) were used. These classifiers can effectively distinguish between the presence and absence of osteoporosis using refined deep representations generated within the ViT-FESEL framework.

Within the scope of Branch-A, model performance was evaluated using multiple metrics that account for class imbalance. These metrics include Accuracy, macro-averaged precision (macro-Precision), macro-Recall, macro-F1 score, Balanced Accuracy, and ROC-AUC, as appropriate. This multifaceted evaluation strategy ensures a comprehensive examination of not only overall accuracy but also the model's discriminative power for each class. In conclusion, Branch-A demonstrates high discriminative power and stable performance in the binary osteoporosis classification problem when ViT-based representation learning is combined with MI and NCA-based feature selection; it also provides a robust and meaningful reference for the Branch-B and Branch-C architectures. 

2.2.4 Branch-B: ViT-based ordinal classification approach

Osteoporosis assessment is not treated as a binary decision problem (“present” or “absent”) in many clinical scenarios. Instead, it is evaluated using classes that represent the disease's severity levels and have a natural ordering (e.g., normal, mild, moderate, severe). Such problems are defined as situations where there is an ordinal (sequential) relationship between classes. However, softmax-based models used in classical multi-class classification approaches ignore this sequential relationship among classes and treat all classes as equally distant from one another. This situation can lead to clinically meaningful errors not being sufficiently distinguished. To overcome this limitation, a separate pipeline, Branch-B, based on an ordinal learning approach, was designed within the ViT-FESEL framework. 

The primary objective of Branch-B is to produce clinically more meaningful and consistent predictions by directly modeling the natural hierarchical structure of osteoporosis classes, enabling the learning of relative relationships between classes. In this context, an ordinal classification head based on CORAL (Consistent Rank Logits) has been integrated onto the ViT backbone [27, 28]. The CORAL approach addresses a problem with K ordered classes by converting it into K − 1 binary decision problems; each decision aims to predict whether the relevant example is above a certain class threshold. Thanks to this structure, the model can consistently learn not only class labels but also the ordinal relationships between classes. Thus, an error between “mild” and “moderate” osteoporosis, for example, can be less penalized than an error between “normal” and “severe” osteoporosis; this provides a more clinically meaningful assessment. 

In Branch-B, as in the previous branch, a high-level deep representation is obtained from the input sequence of X-ray images via the ViT backbone. The [CLS] token representation from the final encoder block of ViT serves as a summary of the image's global structural features and contextual information. However, unlike Branch-A and Branch-C, no MI or NCA-based feature selection is applied to these representations in Branch-B. This choice is deliberate and based on methodological reasoning. Since the fundamental goal in ordinal learning is to preserve the continuous and relative-intensity relationship between classes, aggressive feature selection steps aimed at increasing discriminability can break this ordered structure and lead to a loss of intensity continuity. Therefore, in Branch-B, the goal is to directly integrate ordinal information into the deep representation learning process, and ViT-based representations are connected to the end-to-end CORAL header.

When evaluating model performance under Branch-B, evaluation metrics specific to ordinal problems were used in addition to the classic accuracy metric. In particular, the Quadratic Weighted Kappa (QWK) metric was chosen to assess the extent to which model predictions align with the actual class rankings. The QWK metric weights errors by considering the ordinal distance between classes, thereby providing a clinically more meaningful performance assessment. In addition, macro-averaged precision, recall, and F1 scores were also reported to account for the effects of class imbalance. This multifaceted evaluation approach enables a comprehensive analysis of ordinal classification performance. 

As a result, Branch-B, as the component of the ViT-FESEL method that addresses the scenario closest to clinical reality, provides an assessment that considers not only the presence of osteoporosis but also its severity. While Branch-A aims for high discriminative power through MI and NCA-based feature selection, Branch-B deliberately omits this step to preserve ordinal continuity. In this respect, Branch-B complements the binary classification approach and demonstrates the flexibility and comprehensive adaptability of the proposed framework to different diagnostic requirements. 

2.2.5 Branch-C: Multi–vision transformer–based ensemble approach

Although ViT architectures have strong representation learning capabilities, when trained with a specific patch size or model capacity, they can become more sensitive to certain patterns in the image compared to others. Particularly in knee X-ray images, osteoporosis findings become more pronounced in some cases with local bone tissue and trabecular structure changes, and in some cases with the overall morphological structure and global arrangement of the bone. This situation can lead to models based on a single ViT variant exhibiting unstable or variable performance in certain cases. In this study, to overcome this limitation, a multi-ViT-based ensemble approach called Branch-C is proposed within the ViT-FESEL framework.

The fundamental idea behind Branch-C is based on the assumption that ViT variants with different patch sizes and model capacities can learn complementary and multi-scale representations from the image. Smaller patch sizes focus on local bone tissue and fine structural details, while larger patch sizes more effectively represent the overall morphological arrangement and global structural relationships of the image. Similarly, ViT models with different depths and parameter counts can capture osteoporosis-related patterns at different levels of abstraction. Therefore, in Branch-C, multiple ViT variants are used in parallel to create a rich, diversified, and complementary representation space.

In this context, high-dimensional deep representation vectors are extracted independently for each input sequence X-ray image from different ViT backbones. These representations are then structured to preserve the information provided by each ViT model and subsequently combined into a single unified representation vector through feature-level concatenation. This process enables the model to learn on a multi-scale and multi-perspective feature space. However, directly combining representations obtained from different ViT variants significantly increases the dimension of the feature space, which can lead to higher computational costs and overfitting risk. 

Therefore, in Branch-C, a two-stage feature selection (FESEL) mechanism was applied on the combined ensemble representation vectors. In the first stage of the FESEL process, the statistical dependency of each feature with the class labels was measured using the MI method, and features that carried weak information for classification, were noisy, or redundant were eliminated. This MI-based pre-selection stage ensures the creation of a more compact and informative subspace in the high-dimensional ensemble representation space. Subsequently, NCA was applied to this feature subset selected by MI. NCA selects features that maximize class separation by considering inter-example neighborhood relationships and contribute to learning clearer decision boundaries. Thanks to this two-stage FESEL approach, complementary and discriminative features from different ViT variants are preserved, while components that make only a limited contribution to classification are systematically eliminated. For MI-based feature selection, the `mutual_info_classif` function from the scikit-learn library was used, and the continuous prediction method was preferred (discrete_features=False). This choice stems from the fact that ViT-based representation vectors contain continuous-valued features. During the MI phase, the top 512 features with the highest statistical dependence were selected. NCA, on the other hand, was implemented using scikit-learn’s Neighborhood Components Analysis. NCA is a direct metric learning approach that does not involve a traditional k-neighborhood parameter; therefore, the k value is not relevant in this method. The number of NCA components was set to n_components=128, and the maximum number of iterations to max_iter=200; these values were selected through preliminary experiments. 

The refined ensemble representation vectors obtained after feature selection are used in the final classification stage within the Branch-C framework. In this study, XGBoost and RBF-kernel Support Vector Machines (SVM-RBF), which can effectively model non-linear decision boundaries, were preferred. These classifiers can effectively learn complex, high-dimensional decision spaces by combining information from multiple ViT models. The primary goal of Branch-C is to provide a more stable, balanced, and generalizable performance across different data samples and classes rather than achieving the highest score with a single model.  

As a result, Branch-C, as the top-level integration component of the ViT-FESEL method, provides a holistic decision mechanism that refines complementary representations from multiple ViT architectures via MI- and NCA-based feature selection, and combines them with powerful classical classifiers. This structure aims to achieve more robust, consistent, and clinically reliable osteoporosis classification results compared to approaches based on a single ViT backbone, strongly supporting the practical clinical application potential of the proposed framework. 

2.2.6 Training procedure, implementation details, and overall workflow

Within the proposed ViT-FESEL framework, all experiments are structured according to principles of fair comparison and generalizability. A unified training and evaluation protocol is adopted for all branches (Branch-A, Branch-B, and Branch-C), enabling consistent comparison of different architectural configurations and classification strategies.  

ViT backbones are initialized using weights pre-trained on large-scale natural image datasets. This choice facilitates faster convergence and more stable representation learning, which is particularly important given the limited size of medical imaging datasets. During training, the final layers of ViT models are updated task-specifically, while limited or progressive fine-tuning is applied to earlier layers to mitigate overfitting. All experiments are conducted using a k-fold cross-validation scheme. For each fold, training, validation, and test splits are strictly maintained, and both feature selection (FESEL) and model learning processes are performed exclusively on the corresponding training subset. This design prevents information leakage from the test data and increases the reliability of the reported performance metrics. 

At the feature selection stage, high-dimensional ViT representation vectors are refined to retain only the most discriminative components for classification. The representations obtained after feature selection are used as inputs to classical machine learning classifiers in Branch A and Branch C. Hyperparameters of the classifiers employed in the classification stage are optimized using the training data and are applied consistently across all folds.  

Model performance is evaluated using multiple metrics that account for class imbalance and clinical decision-making requirements. For the binary and ensemble-based branches, metrics such as accuracy, macro-averaged precision, recall, F1-score, balanced accuracy, and ROC-AUC are reported. For the ordinal classification branch, the QWK metric, which measures the degree of agreement between predicted and true ordinal labels, is additionally employed. This multi-dimensional evaluation strategy enables analysis not only of overall performance, but also of inter-class separability and clinically meaningful error patterns [29].

2.png

Figure 2. Graphical abstract of Vision Transformer-Based Feature Extraction and Selection