Multimodal Approach for Non-Invasive Blood Glucose Estimation Using Fingertip Video

Prior work has focused on deriving BGL through non-invasive means, driven by the growing demand for painless and user-friendly monitoring solutions. These techniques were developed as alternatives to traditional finger-prick methods, which are invasive and often uncomfortable for users. Broadly, NI glucose estimation techniques can be categorized into two primary groups: sensor-driven methods and spectroscopy-based approaches.

Prasad et al. [13] introduced an IoT-enabled system for random blood glucose estimation using Photoacoustic Spectroscopy (PAS) signals analyzed with a Shallow Dense Neural Network (SDNN). With 105 subjects, their system achieved Root Mean Square Error (RMSE) = 2.86 mg/dL and Mean Absolute Relative Difference (MARD) = 8.49%. Kumar et al. [14] employed NIR spectroscopy with a 940 nm LED sensor, where ensemble regressors achieved a coefficient of determination (R²) = 0.921 and RMSE = 27.36 mg/dL from 611 measurements. Song et al. [15] proposed Multi-Scale Fusion (RBANet), a deep learning model integrating wearable physiological data (blood volume pulse, exploratory data analysis, heart rate, accelerometry) with nutrition information, reporting Mean Square Error (MSE) = 0.22 mmol/L and 96.75% accuracy for hyperglycemia detection. Chellamani et al. [16] used PPG signals with a Deep Sparse Capsule Network (DSCNet), obtaining R² = 0.98 and Mean Absolute Percentage Error (MAPE) = 3.02%. Further, Piao et al. [17] combined Graph Attention Networks with Gated Recurrent Units (GRUs) to model multivariate signals from Empatica devices, achieving RMSE ≈ 19.86 mg/dL. Mazgouti et al. [18] presented a hybrid Long Short-Term Memory (LSTM)–XGBoost model for Type 1 diabetes prediction, where CGM data from 12 patients yielded RMSEs of 7.97–10.93 mg/dL with R² up to 0.98. Fathimal et al. [19] designed a dual-wavelength NIR optical setup (940 nm and 1050 nm) with polynomial regression, reporting MAPE = 5.99%. Jian et al. [20] employed dual-wavelength PPG signals (660 nm, 880 nm), where Random Forest achieved MARD = 5.15% and R = 0.93.

Despite significant progress in sensor-based NI BGL estimation, many existing methods depend on bulky, costly, or specialized hardware and focus predominantly on signal-level processing. An alternative line of research has explored video-based methods, leveraging fingertip or facial recordings to derive PPG signals. Golap et al. [11] analyzed fingertip recordings with 850 nm LEDs and smartphone cameras, extracting 48 features; Multigene Genetic Programming achieved R² = 0.881. Nie et al. [8] proposed non-contact imaging photoplethysmography (IPPG) from facial NIR recordings, where Random Forest Regression reported MAE = 1.72 mmol/L. Sridevi et al. [21] explored Quantum Machine Learning with NIR-illuminated fingertip videos, achieving 89.30% accuracy. Chinchanikar and Dale [22] processed fingertip videos in Red, Green and Blue (RGB) and Hue, Saturation and Value (HSV) color spaces, where XGBoost yielded R² = 0.89 (RGB) and 0.84 (HSV). Table 1 summarizes representative NI glucose estimation studies, highlighting methods, devices, dataset sizes, and reported performance.

Table 1. Summary of NI BGL estimation studies

Fingertip videos capture changes in light intensity caused by volumetric variations in blood during systole and diastole. In most existing video-based approaches for NI BGL, these recordings were processed by averaging pixel intensity values over selected regions of interest to extract the PPG signal. While this reduction facilitates interpretation and physiological analysis, it inherently compresses complex spatial–temporal data (width × height × time) into a single time-series waveform, potentially discarding rich spatial and spatiotemporal information that may hold subtle yet valuable cues for accurate glucose estimation. Although the potential of deep learning to capture such complex patterns has been demonstrated in related applications, such as hemoglobin level estimation [12], its role in blood glucose prediction remains underexplored. This highlights the need for hybrid frameworks that integrate physiological signal features with deep visual representations to achieve robust, NI BGL prediction.

At the core of this study, PPG waveforms were extracted from fingertip videos, from which 46 handcrafted features were computed. In parallel, ResNet-50 was applied to each video frame, and after a mean pooling operation, 2,048 deep features per video were obtained. Three experimental models were developed to evaluate the predictive capability of different feature representations: (i) handcrafted PPG features, (ii) ResNet-50–based deep features, and (iii) a combined multimodal feature set integrating both handcrafted and deep representations. Models trained on individual feature sets provided useful insights, but the combined approach consistently outperformed the others, highlighting the complementary nature of handcrafted physiological descriptors and deep visual embeddings.

To establish a robust framework for NI BGL estimation, two primary feature sets were generated from fingertip videos: handcrafted features derived from PPG signals, and deep features extracted using the pre-trained ResNet-50 model. Additionally, a hybrid feature set was created by combining these two feature sets, leveraging the complementary strengths of physiological descriptors and deep visual representations for improved predictive performance.

4.1 PPG signal acquisition and characteristic feature extraction

Figure 4 depicts the overall framework used for clean PPG signal extraction from fingertip videos. For the extraction of the PPG signal, the red channel was selected as it exhibited the highest pixel intensity among the RGB channels.

tu_pian_4.png

Figure 4. Overview of the PPG signal processing pipeline

To ensure signal quality and remove potential distortions, the initial 3 seconds and concluding 2 seconds of each video were excluded. This resulted in 300 usable frames per subject. To identify the ROI, K-means clustering was employed on the video frames, segmenting pixel values into separate groups.

Based on the clustering outcome, a 500 × 500 pixel area spanning rows 750 to 1250 and columns 0 to 500 was selected as the ROI for computing the mean intensity of the red channel. The mean red channel values across the defined region were used to derive the unprocessed PPG signal. This process is described by Eq. (1).

$PPG\left( t \right)=\frac{1}{MN}\underset{i=1}{\overset{M}{\mathop \sum }}\,\text{ }\!\!~\!\!\text{ }\underset{j=1}{\overset{N}{\mathop \sum }}\,{{I}_{red}}\left( i,j,t \right)$             (1)

where, M and N indicate the number of rows and columns in the region of focus, and I_red(i, j, t) corresponds to the intensity of the pixel located at (i, j) at time t. To ensure precise PPG signal analysis, preprocessing was applied to 10-second videos (300 frames). A Butterworth band-pass filter (0.5–4 Hz) was applied to remove motion artifacts and baseline drift while preserving the physiological frequency range of heart rates (30–240 bpm). The Butterworth design was chosen over other filters (e.g., Chebyshev, elliptic) because it provides a maximally flat frequency response in the passband, ensuring minimal signal distortion—an important factor for maintaining the integrity of the PPG waveform morphology. Once the raw PPG signal was filtered, peaks were identified using a peak detection method. A single PPG cycle with the most distinct systolic peak was identified.

Figure 5 shows the block diagram of the PPG signal processing and feature derivation applied to the filtered PPG signal. The process began with the application of a peak detection algorithm to identify individual PPG pulses within the signal. Among these, the PPG cycle corresponding to the maximum peak amplitude was selected as the representative waveform, as it was assumed to be the least affected by noise and the most physiologically relevant.

tu_pian_1.png

tu_pian_4.png

tu_pian_2.png

tu_pian_5.png

tu_pian_6.png

tu_pian_7.png

Figure 5. Schematic representation of the PPG signal processing and feature extraction steps [22]

The first and second derivatives of the selected PPG waveform were then computed to capture the rate of change and acceleration in signal morphology. Additionally, the Fast Fourier Transform (FFT) was applied to extract the frequency-domain characteristics of the waveform. From the original waveform, its derivatives, and its frequency representation, a total of 46 features were extracted as shown in Table 2. Let h denote the derived handcrafted feature vector, as given in Eq. (2).

$h=\left[ {{h}_{1}},{{h}_{2}},..{{h}_{46}} \right]\in {{R}^{46}}$           (2)

Table 2. Handcrafted features

4.2 Deep feature extraction from video frames using ResNet

To preserve the spatial and textural information that is often lost during the transformation of fingertip videos into a 1D PPG signal, pretrained ResNet-50 was employed as a deep feature extractor. Unlike handcrafted features that rely solely on signal morphology, ResNet-50 focuses on learning spatiotemporal variations in light absorption, which may be indicative of blood volume changes linked to glucose levels. Its ability to learn complex representations from raw visual data makes it a powerful tool for enhancing model performance in NI BGL estimation.

ResNet-50 is a deep CNN consisting of 50 layers, renowned for its use of residual connections that facilitate the training of very deep architectures without performance degradation. These shortcut connections address the vanishing gradient issue and enable efficient learning of both low- and high-level features. The architecture consists of multiple convolutional blocks, identity mappings, and batch normalization layers. ResNet-50 is widely used for feature extraction in image-based tasks due to its robustness and strong generalization capability. Figure 6 shows the block diagram illustrating the process of feature extraction from fingertip videos.

tu_pian_7.png

Figure 6. Feature extraction pipeline from fingertip video via ResNet-50

In the proposed work, fingertip video data for each subject was processed by extracting frames between the 3rd and 13th seconds, resulting in a consistent temporal segment of approximately 300 frames per video. The frames were scaled to 224 × 224 pixels and standardized using the predefined ImageNet mean and standard deviation values [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225], respectively, to ensure alignment with the ResNet-50 framework.

To derive features, the final fully connected classification layer of the ResNet-50 framework was removed. Let f_ResNet be a function that represents the ResNet-50 model’s feature extraction process and is given by Eq. (3).

${{f}_{ResNet}}={{R}^{224\times 224\times 1}}={{R}^{2048}}$             (3)

For each frame I_t, deep feature vector is given by Eq. (4).

${{x}_{t}}={{f}_{ResNet}}\left( {{I}_{t}} \right)={{R}^{2048}}$             (4)

For t = 1, 2, 3, ...T, where, T = 300, x_t represents features from one frame and I_t is the t^th red frame of the video. This resulted in a feature matrix for the entire video and is given by Eq. (5).

$X=\left[ {{x}_{1}},{{x}_{2}},...,{{x}_{T}} \right]\in {{R}^{2048\times T}}$           (5)

This configuration enabled access to the 2048-dimensional feature vector from the penultimate layer. The model was executed in inference mode using PyTorch with GPU acceleration where available. Each processed frame was passed through the network, and the resulting features were stored in separate files for each video to facilitate modular analysis. Processing 243 fingertip videos through ResNet-50 for deep feature extraction required approximately 2 hours and 32 minutes.

To convert the frame-level feature matrix X $\in $ R^2048×T into a single, fixed-length representation for each video, temporal mean pooling was applied across all frame-level features. This operation averaged the ResNet embeddings over 300 frames, resulting in a 2048-dimensional vector that compactly captured the aggregated spatiotemporal information of the entire video, as expressed in Eq. (6). By averaging out transient variations, this strategy reduces frame-level noise and motion artefacts while emphasizing stable spatial patterns that are more likely to reflect underlying physiological changes. Mean pooling was employed to obtain a compact and robust feature representation suitable for tree-based regression models. It reduces the influence of outlier frames that may adversely affect performance in max pooling. Furthermore, mean pooling lowers dimensionality and computational cost, improving the sample-to-parameter ratio and enhancing the robustness of subsequent regression modeling.

$\bar{x}=\frac{1}{T} \sum_{t=1}^T x_t \in R^{2048}$            (6)

The resulting pooled vector x̄ captures the average spatiotemporal characteristics of the video, serving as its compact deep representation. The aggregated dataset was compiled into a unified CSV file, with each row corresponding to one video sample.

Thus, two distinct feature sets were derived: the first comprising 46 handcrafted PPG features and the second consisting of 2048 deep features extracted using ResNet-50. A third hybrid feature set was then constructed by concatenating the handcrafted and deep features, resulting in a total of 2094 features. These three feature sets were subsequently employed to train and evaluate three experimental models: one based solely on handcrafted PPG features, one utilizing ResNet-50 features, and one leveraging the combined hybrid feature set.

A total of 243 subjects, including both diabetic and non-diabetic individuals, participated in this study, with ages ranging from 18 to 88 years. For each subject, a 15-second fingertip video was recorded using a mobile camera paired with a NIR illumination unit to improve signal fidelity. From each video, a PPG waveform was extracted, from which 46 handcrafted features were computed. Additionally, 2048 deep features were extracted from video frames using ResNet-50 CNN. A third hybrid feature set was then constructed by combining the handcrafted and deep features, resulting in a 2094-dimensional dataset per subject.

Feature importance-based selection was employed to prioritize the most informative variables within the feature space. This approach ensured that the most relevant information was preserved while reducing redundancy among correlated features. By retaining variables that contributed most strongly to predictive performance, the method provided a refined and balanced feature set for subsequent model training. R² quantifies the proportion of the target variable's variance explained by the predictions, thereby indicating its degree of fit. In contrast, MAE measures the mean of the absolute deviation among actual and predicted glucose values, serving as an intuitive metric of prediction error that remains stable in the presence of outliers. The corresponding mathematical definitions are provided in Eqs. (8) and (9).

$MAE=\frac{1}{n}\underset{i=1}{\overset{n}{\mathop \sum }}\,\left| {{y}_{gact}}~-{{y}_{gpred}} \right|$          (8)

${{R}^{2}}=1-\frac{\mathop{\sum }_{i=1}^{n}{{({{y}_{gact}}-{{y}_{gpred}})}^{2}}}{\mathop{\sum }_{i=1}^{n}{{({{y}_{gact}}-{{y}_{mean\_of\_gact}})}^{2}}}$        (9)

where, y_gact: Measured BGL, y_gpred: Algorithm-generated BGL, y_{mean_of_gact}: Average measured BGL.

An initial 80:20 train-test split was carried out to evaluate model generalization on unseen data. A stratified 5-fold CV was subsequently applied to the training set by dividing it into five equal subsets. In each iteration, four folds were employed for training, while the remaining fold was used for validation. This process helped assess the model’s robustness, reduce variance from individual splits, and ensure more reliable performance estimation.

To predict BGL, several ensemble regression models were evaluated, including RFR, GBR, XGBoost, and CatBoost. Each model was trained on an optimized subset of features, obtained by applying an importance-based feature selection method to the PPG dataset, the deep feature dataset, and the hybrid dataset. The number of selected features varied across datasets and regression models, reflecting the differing contributions of features to predictive performance, reducing redundancy, and the varying sensitivity of models to feature relevance.

Figures 7(a) and (b) present the prediction performance of the evaluated regression models on the test dataset in terms of R² and MAE, respectively. CatBoost and Gradient Boosting achieved the highest predictive accuracy when trained on hybrid features (R² = 0.88 and 0.91, MAE = 14.49 and 13.72, respectively), while XGBoost and Random Forest showed moderate performance (R² = 0.80 – 0.85). Models trained solely on handcrafted PPG features had lower predictive power (R² = 0.52 – 0.83), with ResNet-50 features yielding intermediate performance. These results highlight the complementary nature of hybrid features: deep embeddings provide rich abstract representations, whereas handcrafted descriptors preserve physiologically interpretable signal properties. 5-fold cross-validation further confirmed these observations. Hybrid features maintained superior generalization performance across all models. Figures 8(a) and (b) illustrate the R² and MAE values obtained after 5-fold cross-validation, highlighting the comparative performance of the three feature sets. CatBoost achieved the highest R2 of 0.89 and lowest MAE of 15.13 on hybrid features, followed by Gradient Boosting (R² = 0.79, MAE = 19.14). XGBoost and Random Forest achieved R² = 0.70 – 0.76 and MAE = 18.96 – 25.02. The superior performance of CatBoost can be attributed to its strong ability to model intricate nonlinear interactions among heterogeneous features, its inherent robustness to outliers, and its efficient handling of high-dimensional hybrid feature spaces, thereby making it particularly suitable for reliable blood glucose level prediction from fingertip video-derived data.

tu_pian_9.png

(a)

tu_pian_8.png

(b)

Figure 7. Comparison of (a) R² and (b) MAE values for different regression models across PPG features, ResNet-50 deep features, and the hybrid feature set

tu_pian_10.png

(a)

tu_pian_11.png

(b)

Figure 8. Comparative (a) R² and (b) MAE values of regression models trained on PPG, ResNet-50, and hybrid feature sets under 5-fold cross-validation

To enhance statistical rigor, all experiments were repeated ten times using different random seeds. The plots in Figures 7 and 8 illustrate the distribution of model performance across runs, showing mean values along with 95% confidence intervals. This approach mitigates the effect of random initialization and sampling variations on the reported results. Among all ensemble regressors, CatBoost consistently achieved the lowest MAE and the highest R2 across repeated trials, confirming its robustness and stability. A paired t-test was conducted to examine whether the observed improvement of CatBoost over other models was statistically significant. The analysis revealed that CatBoost’s MAE was significantly lower than those of Gradient Boosting (p < 0.05), XGBoost (p < 0.01), Bagging Regressor (p < 0.01), and Random Forest (p < 0.01), confirming that its superior performance is statistically meaningful.

Error analysis revealed that the largest prediction errors were primarily associated with extreme glucose values and suboptimal acquisition conditions, such as low ambient lighting or slight finger motion. Variability in demographics, including age, BMI, and finger thickness, also contributed to error differences, with younger subjects with thinner fingers exhibiting slightly lower errors due to higher PPG signal quality. While the models demonstrated robust performance, several limitations may affect generalizability. Skin tone can influence PPG signal contrast, BMI and vascular health may affect peripheral blood flow, and environmental factors such as lighting and motion artifacts can introduce additional noise. Furthermore, tree-based feature importance may not fully capture complex feature interactions, and the study population was limited.

In addition to predictive performance, the computational efficiency of the evaluated models was assessed to consider real-time deployment feasibility. Inference time per sample was measured on a standard workstation with an Intel i7 CPU and 16 GB RAM. CatBoost and Gradient Boosting required approximately 5–7 ms per sample, whereas XGBoost and Random Forest required 6–9 ms per sample.

Overall, integrating handcrafted PPG and deep ResNet-50 features consistently reduced prediction error and enhanced generalization across all models. Figure 9 summarizes the final hyperparameter settings for each regression model.

tu_pian_12.png

Figure 9. Hyperparameter configurations for evaluated regression models

In terms of computational efficiency, all tree-based ensemble models demonstrated sufficiently low inference times for near real-time blood glucose estimation from fingertip videos. CatBoost combines high predictive accuracy with efficient inference, making it a strong candidate for practical deployment in mobile or embedded applications. Although training with hybrid features required a higher computational cost due to increased dimensionality, this is a one-time process and does not affect real-time prediction. Overall, the results highlight that integrating handcrafted PPG features with deep ResNet-50 representations enhances both predictive accuracy and error minimization, establishing hybrid features as a robust strategy for non-invasive BGL estimation from fingertip video data. Figure 9 presents the final hyperparameter settings used for each ensemble regression model.

tu_pian_13.png

(a)

tu_pian_14.png

(b)

Figure 10. (a) CatBoost regression and (b) Bland–Altman agreement analysis for the hybrid dataset

Figure 10(a) presents the regression and (b) Bland–Altman plots for the CatBoost model trained on the hybrid feature set, evaluated on the 20% test dataset comprising 49 subjects. The regression plot shows a strong correlation between predicted and reference BGL, indicating high predictive accuracy. In the Bland–Altman analysis, 45 out of 49 points (91.84%) were located within the 95% limits of agreement (Region 1, considered acceptable), while 4 points (8.16%) fell outside these limits (Region 2), suggesting minor deviations. These findings further validate the clinical reliability of the model, with a large majority of predictions exhibiting acceptable agreement with reference values.

Table 3 presents a performance comparison between the proposed approach and multiple previously reported contact and non-contact methods for estimating BGL using video data.

Table 3. Evaluation of the proposed method against current video-based BGL prediction techniques

All the methods listed above estimate BGL by extracting PPG signals from video data. Although Sridevi et al. [21] and Haque et al. [24] reported higher R² values (0.89 – 0.90), their studies involved relatively small subject counts (136 and 93, respectively). The proposed work achieved comparable predictive performance (R² = 0.88; average CV R² = 0.89) on a larger cohort of 243 subjects, demonstrating improved generalizability. Unlike these approaches, the proposed method incorporates deep features extracted directly from fingertip video frames using a pre-trained ResNet-50 CNN. By fusing handcrafted PPG features with high-level CNN-derived representations, the model leverages both physiological domain knowledge and abstract visual patterns. Specifically, ResNet-50 captures spatiotemporal variations in light absorption across video frames, reflecting subtle blood volume changes associated with glucose levels—information often lost during conventional signal extraction.