Non-Invasive Tongue-Based HCI System Using Deep Learning for Microgesture Detection

Interaction (HCI) is an interdisciplinary field of study that investigates the dynamics of human interaction with computers. It incorporates various disciplines, including but not limited to, human factors, ergonomics, and psychology. HCI research probes diverse aspects that influence the user experience, such as technology perception, decision-making processes, and interactive visualizations.

As sensor technology continues to evolve and finds integration in an increasing number of applications, the need for user-friendly interactions becomes paramount. HCI systems have been developed to facilitate intuitive and straightforward interactions between human users and computers or other intelligent devices. By offering a user-centric interface, these systems strive to deliver an interaction experience that is both natural and accessible to the user.

The considerable advancements in sensor technology, coupled with its integration into wearable devices, have given rise to a novel interaction paradigm referred to as 'micro-interaction'. Micro-interactions can be defined as non-verbal communication methods employed in interactions with computers, smartwatches, and similar devices.

The improvement of micro-interactions with wearable devices has profound implications for user experience. Microgestures can serve as an alternative input method particularly beneficial for individuals with limited or no limb mobility, including those with quadriplegia. Moreover, for typical users, enhanced micro-interactions with wearable devices could result in a more seamless and intuitive experience.

Interest in HCI systems has surged, given their potential to significantly enhance user experience across various domains. Applications of HCI span a wide array of fields, such as medicine, education, cybersecurity, the military, and gaming, among others. The potential impact and utility of HCI systems underscore the importance of continuous research and development in this area. One of the human-computer interaction applications in the medical field is augmented and alternative communication or AAC. An AAC system aims to support individuals with speech or language impediments by providing various assistive tools, such as gestures and pictures, to facilitate effective communication. However, technical problems and poor interfaces caused to limit the adoption of such a system. To overcome these limitations, we can apply SCI system Approaches to Improve AAC systems [1, 2].

Another medical application that uses the HCI system is a human-in-the-loop system. The system is where humans oversee and control the operation, while the system provides feedback and assists humans in carrying out tasks. As the system can be utilized to help individuals with physical disabilities, as well as such systems can be utilized health monitoring and disease diagnosis [3-14].

Another application for HCI systems in developing assistive technologies for people with disabilities is the tongue computer interface or TCI. Such a system helps people with complete tetraplegia or quadriplegia to control electronic devices using tongue gestures. The main idea behind TCI is that cranial nerves innervate the tongue. Thus, it does not get affect by spinal or cord injuries. As a result, it can serve as an alternative means of Control [15-18].

In the education field HCI system is implemented by designing and developing interactive learning experiences. Therefore, HCI system applications in education can vary from interactive whiteboards that facilitate controlling collaborative learning to virtual reality systems that simulate real-world experiences for immersive learning.

As online learning platforms become more popular, improving the student experience during courses is important. Therefore, implementing HCI systems principles when designing an e-learning website can improve the student experience by optimizing user interaction and creating a tailored system that's created for users' need and preferences, moreover, by applying HCI principles lead to increased motivation and engagement of students, resulting in an improvement, learning experience [19-28].

The security research community acknowledged that human behavior is one of the main factors of security failure, where it is common to refer to humans as the weakest link in the security chain. Therefore, designers should consider user behavior to create and improve security systems. Therefore, researchers apply the knowledge and techniques of the HCI system to mitigate and resolve the issue and cyber security field. We utilize HCI systems main points when designing and developing security applications. This includes user authentication systems, intrusion detection systems, security awareness, training, and tools.

In the military application, we can utilize the principles to design and develop interactive systems that can help prove the effectiveness and efficiency of the military operation. An example of a military SCI system is comment on Control systems that enables real-time to distinguish making virtual training and unmanned aerial systems that provide remote surveillance [29-33].

Moreover, finally, we can utilize HCI principles to design and develop games, enhancing the gaming experience or design and engaging game mechanisms. There is a wide range of SCI system applications in gaming, including gesture paste, Control that enables the player to interact using hand gestures, and virtual r reality systems that immerse the player in the game environment. Moreover, by using intelligent gaming, billions, we can understand player motivation and create a tailored gaming experience that can be done by analyzing different aspects of user behavior, such as different gameplay modes or how humans interact with non-player characters [34-41].

The increasing influence of HCI systems has given rise to a novel domain known as tongue-based HCI systems, which serve as an alternative input method for individuals with limited or no limb mobility. The development of these systems presents significant challenges. The selection of appropriate sensors and sensor locations is necessary to detect tongue gestures accurately.

In this section, we viewed a brief introduction, and the next sections are organized as follows: Literature review, Methodology, Result and Discussion, and finally, Conclusions.

Our system was designed with a primary goal: to be user-friendly without requiring any dental retainers or other invasive methods that may be intrusive or obtrusive. Therefore, we aimed to design a lightweight and small system suitable for everyday use or as an assistive input technology for people with limited or no limb movement without compromising accuracy. Therefore, one of the critical aspects of designing the system is: Selecting a suitable sensor, choosing the sensor position which helps us achieve our vision for the system, and selecting the gestures.

Regarding the available sensing technologies, such as textile pressure, X-band Doppler, and EEG /EMG sensors, these sensors can be bulky or obtrusive, which aligns differently from our vision for the system. Therefore, we selected an accelerometer since the sensor is small, lightweight, and can be concealed easily. Therefore, it serves our vision for a non-obtrusive non-invasive, and non-intrusive system without compromising accuracy.

Moreover, when designing the system, we must choose a suitable location for the sensor. Our system vision indicates that the forehead, tongue tip, and mouth are not valid positions for the sensor to detect tongue gestures. As a result, we have selected the lower jaw on the Genoglasses muscle (on the skin) as the location for the accelerometer. This location has been shown to accurately detect tongue gestures without being invasive or obtrusive.

To complete the system design, we had to choose the appropriate gestures. Hence, we carried out a series of experiments for the selection process. We picked out five tongue movements that can be done without opening the mouth, allowing the system to be used in public areas.

After shaping our design, we conduct machine learning, coupled with time domain and frequency domain data pre-processing and deep learning classification model to classify tongue gestures and compare the result of the two methods.

3.1 Gesture selection

Gestures are a method of nonverbal communication that is powerful for expressing intentions and meanings without using spoken language. In human-computer interaction, gestures can be used as an input method for controlling smart devices. Since tongue gestures are not a standard input method for users, we need to consider various factors, such as the anatomical features of the tongue, the ease of execution for users with varying abilities, and the system's accuracy in recognizing the gestures. Moreover, researchers also consider users' preferences for specific gestures, which can influence the system's overall usability and user experience. For example, in a study by Chen, Victor, et al., taxonomies were used to describe the formation and use of micro-mouth gestures. Twenty mouth gestures were identified as suitable for everyday software application activities [49]. Therefore, in this study, we examined multiple tongue gestures as an alternative input method for our system. One of the essential points when choosing tongue gestures is simplicity, ease of memorizing and selecting gestures that do not interfere with common tongue movement.

Further, we selected five gestures as our final selected gestures aligned with our goals for the system. The chosen gestures consist of an idle gesture and two variations of vertical tongue movements performed at different speeds. These movements occur when the mouth is closed and the tongue touches the upper and lower jaws. Additionally, two gestures involve the tongue moving swiftly or slowly from left to right, contacting the interior gums. By carefully selecting these gestures, we created an effective and user-friendly tongue-based HCI system that can provide individuals with disabilities, injuries, or other limitations with a new input method for interacting with smart devices.

3.2 Designing systems and conducting experiments

In this section, we discuss the system design and experiment process.

One of the main features we aimed to achieve in this study is to design a small, lightweight system that can be used in public without the social awkwardness of a bulky device. Therefore, in our design, we selected the 3×5 mm² ADXL 345 accelerometer as a sensing unit since it checks all the criteria we need in our design. The sensing unit is attached to the lower jaw on the Genoglasses muscle (on the skin), making the system non-obtrusive and non-invasive to use in public. The sensing unit is attached to a 29×58 mm² ESP32 microcontroller that processes the data from the sensing unit. After that, the collected data were stored on the computer.

We collected data through experiments on one test subject in a lab environment to minimize the noise. The sensing unit was attached to the participant's lower jaw on the Genoglasses muscle, using a gentle, double side tape, which provided a secure and none of crucify and non-invasive connection. We designed the sensing unit to collect at the sampling rate of 100 data better second. The experiment lasted for 100 seconds, resulting in a sizable dataset of 500,000 records where each gesture consisted of 100,000 records.

3.3 Data classification

‏In our study, the design system relies on tongue gesture classification to distinguish multiple gestures as an alternative input method. Therefore, we utilized machine learning, deep learning models, and feature extraction methods to achieve the highest accuracy.

We use different supervised machine learning algorithms to achieve our goal, including Support Vector Machine, K-Nearest Neighbor, Decision tree, Random Forest, and Convolution neural network. Given their demonstrated efficacy in prior research, these machine learning and deep learning models have exhibited notable classification accuracy when applied to the task of classifying tongue-based HCI systems.

Deep learning models significantly impact HCI systems by learning convolution neural networks or CNN HCI systems can extract meaningful patterns and insights from complex data, enabling more advanced and intuitive interactions between humans and computers [50].

In the initial step, the dataset preprocessed using time and frequency domain methods to extract intricate relationships at a higher level of abstraction. Following the data preprocessing stage, the dataset is partitioned into a 70% training subset and a 30% testing subset. This division ensures an appropriate allocation of data for model training and evaluation.

During the training phase, the machine learning algorithm receives the training records and corresponding class labels, enabling the model to discern the underlying patterns and relationships that differentiate each class. Through this process, the model acquires the ability to classify new records in the test dataset, even those belonging to the Unknown class, facilitating robust and comprehensive classification.

One of the supervised machine learning algorithms used in the study is K-Nearest Neighbor, a model usually used for classification and regression problems. This model relies on measuring the distance between the new input record from the test dataset and all the points in the training dataset. Further, to identify the nearest K neighbor, we use the Euclidean distance metric to calculate the nearest neighbor. The k value in this model represents the number of neighbors considered in the classification process, as shown in Algorithm 1.

Algorithm 1. KNN Algorithm

Another popular machine-learning technique is Support Vector Machine (SVM), a supervised machine-learning algorithm often used to solve classification and regression problems. This model aims to find the ideal hyperplane to divide different classes. Moreover, this model aims to maximize the margin between classes, which is determined by measuring the distance between the nearest Datapoint in each class and the hyperplane. Furthermore, the hyperplane chooses to categorize brand-new data points according to their properties correctly. The SVM model utilizes the kernel function to map the data points into the high dimensional feature space, allowing the hyperplane to distinguish between different classes, as shown in Algorithm 2.

Algorithm 2. SVM algorithm

The Decision Tree model is a common type of supervised machine learning algorithm. It uses a tree-like structure to classify new data points by making decisions based on the potential outcomes. Decision nodes are used to make decisions, and they have multiple branches that lead to output in the form of leaf nodes, as shown in Algorithm 3.

Algorithm 3. Decision tree algorithm

This study uses another popular supervised machine learning algorithm called a Random Forest. Random forest is an effective machine learning algorithm that tackles classification and regression problems. This model utilizes multiple decision trees, improving generalizability and prediction accuracy. The main principle of random forest is to construct several decision trees where each is trained on the selection of training Dataset, then aggregate their production. Therefore, this method gets beyond some of the drawbacks of the individual decision trees, including bias or overfitting, and generates more solid and trustworthy outcomes, as shown in Algorithm 4.

Algorithm 4. Random forest algorithm

Each machine learning model employed in this study exhibits distinct strengths and weaknesses. For instance, K-Nearest Neighbors (KNN) demonstrates effective handling of multi-class classification problems, yet its performance may be impacted by the curse of dimensionality when confronted with high-dimensional data. Support Vector Machines (SVM), on the other hand, possess the capacity to handle both linear and non-linear relationships between features and target variables through the use of diverse kernel functions. Nonetheless, the selection of an appropriate kernel function and the tuning of associated hyperparameters present challenges. Decision trees are adept at capturing non-linear relationships, but their sensitivity to minor variations in the training data can result in divergent tree structures. In contrast, Random Forests mitigate overfitting concerns by leveraging ensemble learning, yet their application to imbalanced datasets may introduce bias. In the results and discussion section, it becomes evident that the Random Forest model demonstrates a significant classification accuracy across various preprocessing methods.

Lastly, we utilized the convolution neural network (CNN), a deep learning model with impressive results in computer vision applications and partial image recognition. Moreover, researchers also utilized it in the time series classification of problems. In this model, the input time series data is transformed into two dimensions image-like structure where time is shown on one axis, while the values in the time series are representative on the other axis. Then the CNN model uses a sliding kernel across the time series data, set to apply filters and extract pertinent features. Further, the model uses a pulling layer to reduce the dimensionality of the relevant features, as shown in Algorithm 5.

Algorithm 5. CNN algorithm