Enhancing Human-Computer Interaction: A Comprehensive Analysis of Assistive Virtual Keyboard Technologies

Recent scholarly discourse on virtual keyboards has predominantly concentrated on input system optimization strategies. It has been observed that a considerable portion of this research has prioritized enhancing system performance, which involves augmenting text entry rates and reducing error margins, often at the expense of exploring multi-modal approaches. In an effort to elucidate the advancements in visual input systems, particularly in the realm of virtual keyboards, the existing literature has been segmented into three distinct categories as follows.

2.1 Studies on utilizing body gestures

In the domain of alternative input methods, several studies have explored the utilization of body gestures, focusing particularly on hand and tongue movements, as a means to facilitate user interaction with virtual keyboards. One notable approach involves leveraging the agility and precision of the tongue as a substitute for hand-based control over mouse and keyboard functions. Struijk et al. [10] proposed an inductive intraoral tongue interface comprising 18 sensors, designed to control an assistive robotic arm. This interface includes an external central unit that receives data from the tongue-based input and functions as a wireless mouse or keyboard when connected to a computer.

Additionally, Hedeshy et al. [11] introduced an innovative hands-free text input method through visual cues and humming. They developed and evaluated two design options: HumHum and Hummer. In the former, text input is initiated and concluded with brief hums marking each syllable, while the latter interprets continuous humming as a marker for the start and end of a word. This method utilized an RDE NT-USB microphone for capturing the humming sounds and a Tobii 4C eye-tracker with a 90 Hz tracking frequency for gaze detection.

Conversely, Tathe et al. [12] proposed a system aimed at identifying hand gestures to emulate mouse and keyboard functionalities. This system included the use of various hand gestures to manipulate mouse cursor movements and emulate clicking actions. Specifically, it explored the feasibility of using a single-finger gesture for letter selection on the keyboard and a four-finger motion for navigational purposes. The pupil tracking technique employed was derived from the TrackEye software developed by Zafer Savas.

2.2 Studies on eye-tracking techniques

Exploration in the field of eye-tracking techniques has yielded various methodologies for enhancing gaze-based interaction with virtual keyboards. The subsequent analysis highlights key contributions in this domain:

Meena et al. [4] proposed an array of dwell-free and dwell-based approaches, incorporating multisensory elements, for a gaze-based keyboard simulation tailored to the Hindi language. This approach allows for item selection via gaze-tracking, with options such as asynchronous surface electromyography, soft switches, or gesture recognition for making selections. Furthermore, a probabilistic model adjusts dwell time based on the likelihood of letter selection informed by previous choices [13]. Advancements in machine learning have facilitated the development of the i-Riter [14], designed to enable individuals with paralysis to type on a screen using only their gaze. Lystbµk et al. [6] introduced a gaze-based virtual keyboard allowing direct access to letters with a single command and proposed a USB mouse switch for left-click functions.

Cecotti et al. [15] devised a simulated keyboard with ten primary commands, correlating to essential keystrokes and enabling the creation of 74 different characters using an eye-tracker. This interface reduces the spatial proximity of commands within the graphical user interface (GUI) and operates based on a tree selection mechanism, allowing letter selection from two tiers with minimal commands.

The critical aspect of gaze-tracking lies in the accurate determination of pupil or iris position. Iris recognition is crucial in various applications, with several regression-based methods available for gaze position estimation post-iris center identification [16]. Most eye-tracking techniques involve iris recognition challenges associated with circle analysis [17]. A machine-learning approach for eye identification in a dynamic environment, utilizing a Viola-Jones face detector [18]. The WiViK [19] represents a virtual keyboard offering standard QWERTY functionality, operated via an eye-gaze pointing device. Hearty Ladder [20] developed a Japanese syllabary eye-gaze-controlled typing system, considering the frequency of Japanese vowels in the Roman alphabet for designing its key input interface.

Cecotti et al. [21] introduced a novel multimodal, multiscript gaze-based virtual keyboard, adaptable for Bangla, Latin, or Devanagari scripts. This GUI can be modified to suit different scripts within the same layout, with compatibility for mouse, touch display, mouth switch, and gaze recognition inputs. The keyboard features 40 keys, including an eraser, English text types, mathematical numbers, and various Latin symbols, along with a spacebar for form typing and textual suggestions at the top.

Lastly, Islam et al. [5] presented an eye gaze-controlled on-screen keyboard, operable through eye blinks. This keyboard includes 40 keys for English fonts, a delete key, numeric, and various Latin symbols, with keys illuminated sequentially or in a counterclockwise fashion.

2.3 Studies on keyboard layout algorithms

This segment of the literature review delves into the research focused on keyboard layout algorithms, with an aim to enhance typing speed and accuracy. The following studies are noteworthy in this context:

2.3.1 Dwell-free gaze-controlled typing systems

A novel approach integrating a dwell-free method for intermediate letters and a dwell-time strategy for initiating and concluding words was introduced by Cheat and Wongsaisuwan [22]. This method employs the concept of edit distance, which is calculated based on the replacement cost and influenced by the weights of adjacent characters. This calculation aims to determine the optimal weight that ensures high accuracy in typing. Additionally, the study encompasses an array of previous dwell-free gaze-controlled typing systems. These include EyePoint [23], Gazing with pEyes [24], StarGazer [25], Eyeboard [26], Eyeboard++ [27], Filteryedping [28], and EyeSwipe [29]. Each of these systems represents a unique contribution to the field, demonstrating the diverse approaches and innovations in dwell-free gaze-controlled typing technologies.

2.3.2 Dwell-duration gaze-controlled typing systems

The following studies have focused on dwell-duration gaze-controlled typing systems, which aim to optimize key selection efficiency and accuracy:

A study introduced a method for dynamically adjusting key dwell duration based on key placement and selection [30]. Subsequently, Tamilselvi and Sarguna [3] developed a multimodal virtual keyboard with 30 characters and evaluated performance variations under three conditions: direct eye-tracking for command selection, mouse-based target pointing, and switch-based selection.

Cheat and Wongsaisuwan [31] proposed an on-screen keyboard using the Tobii EyeX Controller. Their method integrated dwell-free and dwell-time strategies in a swipe gaze-based keyboard, incorporating a storage system for previously typed words to prioritize intended phrases. The initial and final letters are activated using dwell-time, switching to dwell-free mode thereafter.

Pedrosa et al. [32] described cascading dwell gaze-typing, dynamically reducing dwell time for likely keys based on their probability of being selected next and their position on the keyboard.

Meena et al. [33] presented a method for optimizing the arrangement of displayed objects in a gaze-controlled, tree-based panel selection infrastructure. This Hindi-specific virtual keyboard was designed around a menu with ten commands, accessing 88 character sets and various text editing functions.

Pradeepmon et al. [34] introduced S-finger typing layouts, employing quadratic assignment project (QAP) optimization algorithms for creating keyboard layouts optimized for typing speed and finger mobility. Sandnes et al. [35] recommended increasing dwell time for the first element in scan sequences to reduce input errors on scanning keyboards.

The C-QWERTY arrangement [36] was designed for circular smartwatches, featuring keys in a circular layout along the screen edge, supporting both finger touch and finger interaction. Whittington and Dogan [37] proposed the Smart Power Chair (SoS), which employed an appropriate set of independent and interoperable technologies that were networked over time to meet the specified overall aim of improving the quality of life for individuals with disabilities.

Herthel and Subramanian [38] worked on optimal single-finger keyboard layouts for mobile phones. Kumar et al. [39] introduced TAGSwipe, a bi-modal swiping method combining touch and gaze motions for word selection on virtual keyboards.

Onsorodi and Korhan [8] used a genetic algorithm to design an ergonomic keyboard, reducing typing time and fatigue by considering the frequency of the most commonly used English words. The T18 keyboard by De Rosa et al. [40], inspired by the T9 system, features 18 keys in a QWERTY pattern, selecting dictionary-based characters for each keypress. Benabid Najjar et al. [41] optimized key placement for Arabic keyboards using eye-tracking, focusing on minimizing overall travel distance while typing specific sentences.

A Japanese syllabary interface [42] organized letters on a pentagonal chart, considering the consonant-vowel structure of the Japanese sound system. Alshudukhi and Alshaibani [9] proposed an optimized Arabic keyboard layout to reduce typing time using statistical measures. Caligari et al. [43] investigated the impact of age on eye-typing capabilities, finding a decline in typing speed and an increase in errors with age. Finally, Mohsin and Oday [44] utilized a virtual keyboard as a defense against attacks, transforming Arabic or English letters from ASCOII values to a point-based system.

2.3.3 Modern algorithms-controlled typing systems

This section explores contemporary approaches in typing systems, emphasizing the integration of modern algorithms to enhance user experience and accuracy:

Shen et al. [45] introduced Adapti-Keyboard, a mid-air gesture keyboard optimized for augmented reality (AR) environments. This system focuses on improving user experience by addressing articulation inaccuracy. Adapti-Keyboard employs an adaptable layout size optimization, allowing for a more versatile keyboard with an expanded design and operating area. This customization considers users' motor skills and preferences, facilitating a more effective adjustment of the keyboard's layout to suit individual user needs.

In contrast, Turner et al. [46] investigated the usability and typing performance of two QWERTY keyboard layouts on different smartphone sizes. Their findings indicated a slower typing speed with a curved keyboard on a smaller phone (15 WPM) and a faster rate with a standard keyboard on a larger phone (24 WPM). The curved keyboard consistently resulted in higher error rates, regardless of phone size. Nivasch and Azaria [47] developed the MKLOGA method, which combines deep learning with a genetic algorithm-based approach to enhance keyboard layouts. This method significantly improves the efficiency of genetic algorithms through an advanced crossover process. MKLOGA demonstrates superiority over previous layouts and shows potential for designing layouts in languages other than English.

Khan and Deb [48] utilized the NSGA-II method to generate a Pareto collection of keyboard layouts, ranging from the most economical to the most ergonomically advantageous. This collection can facilitate a gradual transition from the prevalent QWERTY layout to configurations better suited for specific applications. The NSGA-II algorithm, paired with an EMO approach, optimizes keyboard layout to maximize typing efficiency using a distance metric.

Additionally, Kafaee et al. [49] examined the evolution of keyboard layout design, applying the Collingridge dilemma to the transition from typewriters to brain-computer interfaces. This study provides a quantitative analysis of the dilemma, offering a deeper understanding of the challenges in technological evolution. The investigation into the QWERTY keyboard underscores the potential of human-technology co-construction theory in comprehending technological advancements.

In summary, research in keyboard layout design has varied in focus, from eye-tracking and mid-air gestures to advanced algorithmic solutions. While most studies concentrated on the English language, significant efforts were made to develop keyboards in other languages, including Arabic [9], Hindi [4], Latin, Bangla, Devanagari [5], Japanese [20], and others, reflecting the global application of these technologies.

4.1 Results

The results of previous research were compared in three aspects. First, the results of the performance measures were compared with the previous results in an accurate numerical manner. Second, there was a comparison in terms of the system's usefulness by taking extra measures. Finally, the participants had a significant influence on the results obtained through the experiments, so it was necessary to compare their features with respect to gender, level of education, age, and other factors.

4.1.1 Performance evaluation

Several performance evaluations of the virtual keyboard were carried out using the following:

(1) Character per minute

The method below was used to compute the number of characters typed per minute for various participants in several trials [1, 5].

$\mathrm{CPM}=\frac{\text { No. of character }}{\text { Time taken }} \times 60$           (1)

(2) Word per minute (WPM)

WPM calculates the number of words typed per minute as a measure of text entry rate. Using the following formula, the WPM was determined [1, 5].

$\mathrm{WPM}=\frac{\text { No. of word }}{\text { Time taken }} \times 60$           (2)

(3) Total error rate

This is equal to the sum of the corrected error rate (CER) and the not corrected error rate (NCER) [51].

$\mathrm{TER}=\frac{C E R}{N C E R} \times 100$        (3)

(4) Information transfer rate

The ITR is divided and measured at the basic letter level (ITR letter) and command level (ITRcom). Since it is based on the letters that are formed and displayed on the screen, the ITR at the letter level is known as the ITR letter, and at the command level, it is known as the ITRcom due to the fact that it depends on the generated instructions in the GUI. The following is the definition of the ITR [4, 52]:

ITRcom $=\log _2\left(M_{\text {com }}\right) \frac{N_{\text {com }}}{T}$      (4)

ITRletter $=\log _2\left(M_{\text {letter }}\right) \frac{N_{\text {letter }}}{T}$      (5)

where, the user's total number of commands needed to type Nletter characters is denoted by the value Ncom. T is the entire amount of time required to create Ncom or type all Nletter.

(5) Keystrokes per character (KSPC)

The average number of keystrokes per character is known as the KSPC [40].

$K S P C=\frac{\text { No. of keystrokes }}{\text { Character }}$        (6)

4.1.2 System usability scales

To evaluate the usability of virtual keyboards, prior research has predominantly employed two distinct scales:

(1) System Usability Scale (SUS)

The SUS is a ten-item attitude Likert-type measure offering a comprehensive view of users' emotional evaluations. Each item is rated on a five-point scale, with each part capable of receiving a score between 0 and 4. SUS scores range from 0 to 100, with higher scores indicating greater usability. This scale assesses the effectiveness, efficiency, and satisfaction of a system, taking into consideration the context of its usage [4].

(2) NASA Task Load Index (NASA-TLX)

The NASA-TLX is a widely-used, subjective, multimodal assessment tool that measures perceived workload to evaluate the effectiveness and/or other performance characteristics of a task, system, or team. Recognized for its reliability in assessing user workload, the NASA-TLX scale ranges from 0 to 100, with lower scores denoting superior performance. This index was specifically employed to determine the workload experienced by users while utilizing the virtual keyboard application [4].

(3) Participants

Participant characteristics significantly impact trial outcomes. Table 1 in the study juxtaposes participant characteristics with those in previous studies. These characteristics include:

•Both healthy volunteers and individuals with disabilities.

•A typical age range of participants was between 20 and 35 years, although some studies involved a broader age spectrum, including young, middle-aged, and older volunteers.

•Participants comprised employees, general public members, and university students.

•Individuals with normal vision, those wearing glasses, and contact lens users.

•Participants were positioned at a distance of 50 to 80 cm from the computer screen.

•Participants were required to have medium to high English proficiency or native language ability, and similar levels of experience with smartphones and virtual keyboards.

•Usage of eye-tracking communication devices and multimodal input devices, alongside eye-tracking algorithms like Tobii technology.

Table 1. Comparing previous studies of several performance evaluation and system usability scales of the participants

Table 2. A comparison of personal characteristics of the participants from previous studies

Table 3. A comparison of technical characteristics of the participants from previous studies

4.2 Discussion

The analysis of Table 2 reveals that the 'Characters per Minute' criterion is paramount in assessing the efficacy of virtual keyboard creation techniques, particularly in terms of typing speed. This is closely followed by the 'Information Transfer Rate'. Conversely, many studies prioritize the 'Keystrokes per Character' metric over the 'Total Error Rate', underscoring the importance of understanding error magnitude in these systems.

The fluctuating results in Table 2 can be attributed to the diverse characteristics of the participants. Notably, recent research indicates an upward trend in the 'Characters per Minute' criterion, suggesting that contemporary methods are enhancing typing speed.

Additionally, the metric of 'Words per Minute' is considered secondary to 'Characters per Minute'. This is because the definition of a 'word' varies based on the number of letters it contains, leading to variability in measurement precision.

Recent years have seen the development of system usability scales based on participant feedback, focusing on two key aspects: typing speed and user comfort. The choice of input devices interfacing with the keyboard significantly influences these outcomes, especially in terms of speed and accuracy in key selection.

Table 1 highlights that the factors such as age, health condition, and education level greatly impact results. Optimal outcomes are often observed in participants who are in good health, middle-aged, and highly educated.

Table 3 suggests a preference among researchers for participants with good eyesight, as the use of eyeglasses can adversely affect results. Participation of individuals with disabilities has been limited, primarily due to the complexities and specialized techniques required for their inclusion in such studies.

The employment of eye-tracking equipment demonstrates a direct correlation between the scale of vision and distance and the accuracy of key selection from the virtual keyboard. In contrast, visual and distance parameters are less critical when using alternative input methods like fingers, tongues, or EEG devices.