Güneş Ekim* | Ayten Atasoy | Nuri İkizler
© 2021 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
Motor neuron patients such as paralysis, locking syndrome, and amyotrophic lateral sclerosis can see and hear what is happening in their environment, but cannot communicate with their environment. It is very important for these patients, who do not have any physical function other than eye movements, to be able to express their needs, feelings and thoughts. Therefore, to express the thoughts, needs and feelings of these patients, a system that converts eye-blink signals to speech was developed in this study. The main purpose of the designed system is high accuracy, low cost, high speed and independence from environmental factors. Undoubtedly, it is also very important that it causes as little discomfort to the patient as possible. Morse-coded signals generated by voluntary eye-blinks and the single-channel wireless NeuroSky MindWave Mobile device eliminates the need for cost-increasing equipment such as a camera or eye tracker and environmental factors such as light. With the use of Dynamic Time Warping (DTW), an algorithm which works at high speed and high accuracy at the time domain and does not require any training process has been implemented. In this way, the recorded speech was performed with a quite impressive accuracy.
amyotrophic lateral sclerosis, dynamic time warping, eye-blink detection, eye-blink to speech
Diseases such as lock-in syndrome, stroke, and amyotrophic lateral sclerosis (ALS) cause progressive damage to nerve cells. The movements of people with these diseases are restricted or they remain completely immobile, they cannot control any part of their body except eye movement and eye-blink. Due to the loss of control of all voluntary muscles, these people are unable to speak, write messages, and therefore unable to express their thoughts and feelings to the people around them. In this case, the most appropriate and perhaps the only way for communication seems to be using eye movements. In the studies made until today, it is common to use eye-blinks especially for control purposes. For this purpose, eye-blinks are detected by electroencephalography (EEG), electrooculogram (EOG) and image-based systems, so that it becomes possible for ALS patients and immobile patients with severe physical disabilities to use some devices themselves [1].
EEG signals can be measured easily and it causes no discomfort to the person whose brain activity is being recorded. In addition, they are highly preferred due to their lower cost and risk. Voluntary eye-blinks from the EEG signals has been used in tasks such as activating the home lighting system [2, 3], controlling an external device such as a computer or a wheelchair [4]. Detection of eye-related events such as opening eyes, closing eyes, eye-blinking and eye movements [3, 5-7] and smart lock to be opened remotely by detecting a certain number of eye-blink signals [8] are remarkable studies.
There are studies in which wheelchair control is performed using eye-blinks obtained from EOG signals [9, 10]. EOG signals can be detected with surface electrodes placed under, above, right and left of the eye. Electrodes affixed to the eye area pose no risk to the person being recorded. However, the constant presence of 5-6 electrodes around both eyes may cause discomfort to the person. In addition, the narrow frequency band of the EOG signal (0.1 - 10 Hz) makes it difficult to obtain the signal.
For the same purpose, in a patient bed control system developed by using image processing technology, the voluntary eye-blink movements taken from patient were transformed into a command and then a bed or a call system connected to it were controlled [11].
In the studies made to convert the eye-blink to speech, although there is no study that exactly matches the system implemented, two studies that show similarities stand out. In the first of these studies, Soman and Murthy present the design and development of a Brain Computer Interface (BCI) based system for generation of synthesized speech, which works on eye-blinks detected from the EEG signals of the user. This system enables patients to communicate by selecting the desired options from a configured list by performing eye-blinks. It is stated that the system gives an offline accuracy of 95%, average across users [12]. Akram et al., designed a P300-BCI spelling system, which they call the T9 interface. They used BrainAmp MR - 32-channel EEG device for data acquisition. Word entry speed is on average 1.67 minutes [13]. Another study is a low-cost device design by Mukherjee and Chatterjee that detects eye-blinks from the patient and converts them into Morse code which is a universally accepted communication code [14].
Other studies on eye-based text entry systems are methods based on choosing by eye movements the characters of the word that the person wants to write or say. Since such systems are usually performed using the on-screen keyboard, it is also necessary to consider the keyboard layout. The writing process is performed by following the eyes of the person with an eye tracking device and analyzing them with a computer software [15]. Models such as GazeTalk [16] and WiViK [17] are systems based on looking at the desired letter and making a choice by emphasizing that letter. Alternatively, there are also systems that use a switch such as eye-blink and horizontal or vertical movements of the eye to select the focused letter. MacKenzie and Ashtiani have implemented a scanning keyboard that they call Blinkwrite, which uses eye-blink for text entry and works with an eye tracking device [18]. Ülkütaş has developed a computer based eye-writing system by using EOG based on coding letters, numbers and some punctuation marks with eyes. She used six small diameter surface electrodes to detect EOG signals generated during eye movement. The maximum coding speed achieved in trials on a single person is 17.5 words per minute (wpm) and an accurate coding rate of 92.5% was obtained [19]. In another system, an electrooculagram based virtual keyboard was developed by using the EOG signals obtained from only eye movements. The writing speed of the system is 2.5 words per minute [20]. The study of Miniotas et al. is based on the principle of selecting characters from the screen using the virtual keyboard. The hardware used for the experiment consisted of two PCs and the head-mounted eye tracking system from Sensomotoric Instruments, Inc. For this technique, the average text entry rate is 8.58 words per minute, and the error rate is about 3% [21]. In a similar study using a camera eye tracking device, Majaranta et al. achieved 9.89 words per minute and 98.8% accuracy [22]. Preferring to look at the on-screen keyboard and blink as the text input method, Zhang et al used cell phones, webcams, and digital cameras for eye tracking. Average input rate is 20 letters per minute, accuracy rate is 99.8% [23].
In studies where text entry was made by drawing with eyes, Tobii eye tracker was used in the system developed by Porta and Turina, and the average text entry rate was reported as 6.8 words per minute [24]. In a similar study using the 4-electrode EOG device, Lee et al. obtained 87.38% accuracy [25].
The first point that draws attention in the limited number of studies conducted to convert eye-blink to speech is the acquisition of eye-blink signals. The ways that can be followed in detecting eye-blinks are determined. It is possible to detect eye-blink signals using the EEG, EOG, electromyography (EMG), IR sensors or camera systems. Even the way these signals are obtained occurs a significant cost difference between systems. In Table 1, in the literature studies the equipment used, the method applied, how many characters can be written per minute, character recognition percentages and approximate costs are given.
The aim of this study is to develop a low-cost, fast and highly accurate system compared to previous studies, so that it can be purchased by every patient. Study presents the design and implementation of a system that uses Morse coded eye-blink signals to create words and then converts these words to the speech. The system will be very useful for patients who cannot communicate due to motor neuron disorders, to be able to ask for food, drink or medicine, to explain any of their needs such as bathroom, toilet, and to communicate with other people around them.
Table 1. Comparison of the studies in the literature on visual text entry systems
|
Researcher |
Hardware |
Method |
Cost ($) |
Speed |
Accuracy (%) |
|
Mukherjee and Chatterjee (2015) |
IR sensor |
Eye-blink Write |
20-25 |
3 seconds average for a letter |
Unspecified |
|
Soman and Murthy (2015) |
Emotiv Epoc 14 electrodes |
Select text with eye-blink Convert selected text to speech |
700-750 |
Unspecified |
95 |
|
MacKenzie and Ashtiani (2011) |
EyeTech Digital Systems TM3 eye tracker and two monitors |
Eye-blink Scanning keyboard |
High |
4.8 wpm |
97 |
|
Özbek Ülkütaş (2015) |
Two-channel EOG 6 electrodes |
Eye movement Write |
250-300 |
17.5 wpm |
92.5 |
|
Miniotas et al. (2003) |
Two PCs and the head-mounted eye tracking system |
Eye movement Virtual keyboard |
High |
8.58 wpm |
96.6 |
|
Majaranta et al. (2007) |
iView X RED-III eye tracking device and two computers |
Eye movement Virtual keyboard |
High |
9.89 wpm |
98.8 |
|
Porta and Turina (2008) |
Tobii eye tracker device |
Drawing with the eye |
20,000 |
6.8 wpm |
Unspecified |
|
Usakli and Gurkan (2009) |
Two-channel EOG 5 electrodes |
Eye movement Write |
250-300 |
The word SU was written in 24.5 seconds. |
95 |
|
Zhang et al. (2018) |
Mobile phones, webcams and digital cameras |
Screen keyboard Eye movement Eye-blink |
Low |
20 letters per minute |
99.8 |
|
Akram et al. (2015) |
BrainAmp MR 32-channel |
P300-based spelling |
High |
Average time of 1.67 min per word |
Unspecified |
|
Lee et al. (2016) |
Two-channel EOG 4 electrodes and a laptop computer |
Drawing with the eye |
Low |
Unspecified |
87.38 |
In this study, a system has been developed for people with severe motor disabilities who have lost the ability to speak. This system converts patient’s eye-blinks to the speech in order to communicate with people around them. The block diagram of the developed system with this aim is given in Figure 1 and the flow diagram of the developed algorithm is given in Figure 2. The processing of the system has been examined as subtitles, respectively.
Figure 1. Block diagram of developed system
2.1 Data acquisition
The single-channel wireless NeuroSky MindWave Mobile device is wirelessly transmits the electrical activity from the brain to a computer or phone via bluetooth, and is quite cheap device compared to its peers. It is used extensively in simple electronics applications. It consists of a headset, an ear clip and a sensor arm. The EEG electrode is located above the eye, on the sensor arm resting on the forehead [26]. It has a dry active sensor technology that eliminates the need for gel during electrode placement. The sampling rate of this device with a single channel, reliable measurement and powered by one AAA battery is 512 Hz. The electrode of the device is in position FP1, which is present in the 10-20 electrode placement arrangement, and the reference and ground electrode are located on the clip attached to the left ear (A1). Figure 3 shows the single channel wireless NeuroSky MindWave Mobile device and the electrode layout. Thanks to its electrode position, it is a suitable device for use in applications related to eye movements [27]. In this study, the reasons why this device is preferred in data acquisition that is the first step in processing biological signals, are as follows. The device does not cause any discomfort to the person using the device during both the perception of the signal and the wireless data transmission, it is very affordable cost, and its electrode location and type are convenient.
Figure 2. Flow diagram of developed algorithm
2.2 Morse coded signal input
Eye-blinking is the continuous, mostly periodic and symmetrical short-term opening and closing movement of the eyelid. Undoubtedly, it is possible to create different coding techniques using these movements. The universality of Morse Alphabet [28] has played an important role in the choice of coding type.
Figure 3. (a) Parts of Neurosky MindWave Mobile device (b) FP1 and reference electrodes positions of Neurosky MindWave Mobile device (c) Recording session with the system
Table 2. Morse codes of some letters and generating these letters with eye-blinks using Morse codes
|
Letter |
Code |
Eye-blink Correspondence of Letters |
|
E |
• |
Single eye-blink |
|
N |
-• |
Double eye-blink + in-letter space + Single eye-blink |
|
O |
--- |
Double eye-blink + in-letter space + Double eye-blink + in-letter space + Double eye-blink |
|
P |
•--• |
Single eye-blink + in-letter space + Double eye-blink + in-letter space + Double eye-blink + in-letter space + Single eye-blink |
Figure 4. (a) Signal of letters E generated by eye-blinks using Morse codes (b) Signal of letters N generated by eye-blinks using Morse codes (c) Signal of letters O generated by eye-blinks using Morse codes (d) Signal of letters P generated by eye-blinks using Morse codes
In Table 2, Morse codes belonging to the letters E, N, O and P of the word “OPEN” and how these letters were formed with eye-blinks are shown. In this alphabet, which was created by Samuel Morse in 1835, short lines are represented by single eye-blink and long lines are represented by double eye-blink. One eye-blink can be described as both eyes closing and opening together once, and double eye-blink can also be described as both eyes are closing and opening twice, one after the other.
The signals belonging to the letters E, N, O and P given in Table 2 are shown in Figure 4.
2.3 Eye-blink analysis
There are no restrictions for single and double eye-blink durations. While word entries were made with eye-blinks, subjects were asked to leave a short waiting duration at the end of each letter, just as when communicating with the Morse Alphabet. This short waiting duration between the letters is called in-letter space. On the other hand, subjects determined the duration between single and double eye-blink by themselves. However, the length of this period must be less than the duration which is at the end of each letter. This is the only restriction of the system during data entry. The graphic of the “OPEN” word recorded in this way is shown in Figure 5.
In Figure 5; T1, T2, T3 and T4 expressed as;
T1: Single eye-blink duration
T2: Double eye-blink duration
T3: In-letter duration
T4: Space duration between letters
When Figure 4 and Figure 5 are examined, in the X(n) data signal taken from all subjects;
$T_{1}<T_{2}$ and $T_{3}<T_{4}$ (1)
conditions are provided. These expressions can be written as;
$T_{2}=\alpha \cdot T_{1}$ and $T_{4}=\beta \cdot T_{3}$ (2)
$\alpha$ and $\beta$ values are average values calculated from data signals of subjects.
Figure 5. Graphic of the word OPEN created by eye-blink
In the system given in Figure 1, eye-blink moments are detected as follows. When Figure 4 and Figure 5 are examined, a negative peak following the positive peak is observed in the eye-blinking form. If the positive peak in the signal is denoted by Xpos(k) and the negative peak by Xneg(k);
$\left|X_{\text {pos }}(k)\right|<\left|X_{\text {neg }}(k)\right|$ (3)
inequality is provided for positive and negative peaks formed at every eye-blink.
$|X(n)|>|X(n+1)|$
$|X(n)|>|X(n-1)|$
$|X(n-1)|=|X(n+1)|$ (4)
In a X(n) data set, the peak values are defined by the relations in Eq. (4). In order to determine only negative peaks above a certain threshold voltage level, E to indicate the threshold level in Volts;
$|X(n)|>E$ (5)
Eq. (5) can be added to the equations in (4) for the system designed.
Xneg has same array length with X data signal, but it has values that provide the Eqns. (4) and (5). If the data signal X(n) for the array value of n does not provide these equations, then $X_{n e g}(n)=0$. The first $X_{n e g}(n)>0$ in Xneg that provides these four correlations in the data set will be found as Xneg(1). The value of n will also give the location of the first negative peak in the time axis. The number $X_{\text {neg }}(n)>0$ in Xneg gives how many eye-blinks are made within a letter. In other words, it gives the result of how many eye-blinks occur between two T4 duration.
$\begin{aligned} &|X(n)|>|X(n+1)| \\ X_{n e g}(n)=1, &|X(n)|>|X(n-1)| \\ &|X(n-1)|=|X(n+1)| \\ & \quad \text { and }|X(n)|>E \\ X_{n e g}(n)=0, &|X(n)|<|X(n-1)| \\ &|X(n-1)| \neq|X(n+1)| \\ & \quad \text { or }|X(n)|<E \end{aligned}$ (6)
$L_{N}=T_{4}+1$ (7)
LN indicates the number of letters in a word. It is possible to express the relationship between LN and T4 with Eq. (7). For example, it can be easily seen in Figure 5 that while T4 = 3 for the word “OPEN”, LN = 4.
With the determination of the number of letters in a word, we have a signal sequence consisting of single or double eye-blinks. The biggest problem observed at this point is that there is no possibility of any particular time synchronization for single or double eye-blinks between subjects. For example, it is not known that which eye-blink combinations of the 6 eye-blinks in O, which is the first letter of the word OPEN given in Figure 5. Does it consist of 6 single eye-blinks, 3 double eye-blinks, or a different combination? This is the most important problem to be solved in order to get the correct results. Obviously, the moment of time at which eye-blinks occurs is a factor that directly affects the system success. The system must not be affected by shifts in the time axis. At this point, using the W(n) rectangular windowing process and comparing the resulting window output with a previously known window output seems to be the most logical solution. The initial n value of the function W(n) given in Eq. (8) can be shown as a relation derived from the experimental results.
$W_{\text {start }}=X_{n e g}(1)-D_{N G}$ (8)
Here; DNG, is average sample difference between Xneg and Xpos.
$T_{2}<W_{L}<T_{2}+T_{3}$ sample (9)
must be selected. This will allow T1 or T2 to fit comfortably within a window duration. Here; WL is window length.
2.4 Dynamic time warping algorithm
The Dynamic Time Warping (DTW) algorithm used in this proposed system is an algorithm that is used to measure the similarity between two signals that differ in time or speed in time series analysis. Since it is a template-based method, the similarity between the known template is found by comparing the time sequence with an unknown time sequence. Creating a matching path between two templates is the most important part of comparing two templates. This determines which points match and will be used to calculate the distance between them [29, 30].
Let the two time series X(i) = [x1 x2 x3… xi] and Y(j) = [y1 y2 y3… yj] given for the DTW algorithm. i indicates the number of elements in the first array, j indicates the number of elements in the second array. The algorithm calculates the sum of the distances between two series from point (x1, y1) to point (xi, yj). Many functions can be used for this purpose. The most used ones are Euclidean function and difference square function given in Eq. (10) and Eq. (11).
$D(i, j)=\left|x_{i}-y_{j}\right|$ (10)
$D(i, j)=\left(x_{i}-y_{j}\right)^{2}$ (11)
The reason why DTW algorithm is preferred is that it is not affected by the shifts in time as can be understood from its definition. Eye-blink signals can be started at any t moment of time, whether single or double. It is not possible to establish a synchronization between subjects in terms of onset or eye-blinking time. Even for the same subject, it is impossible to expect the same onset time and the same durations for the eye-blink in repetitions.
The start of the W(n) window is the value of Wstart. Shifting the window through time (Wshift) is carried out according to the conditions in Eq. (12).
$\begin{array}{ll}\left|D_{1}\right|<\left|D_{2}\right|, & W_{\text {shift }}=W_{L} \\ \left|D_{1}\right|>\left|D_{2}\right|, & W_{\text {shift }}=2 W_{L}\end{array}$ (12)
Here;
D1: Euclidean distance between single eye-blink template and windowing data.
D2: Euclidean distance between double eye-blink template and windowing data.
For example, if there is a single eye-blink signal in the window taken after the first eye-blink signal, the DTW algorithm will match it with a single eye-blink signal from single and double eye-blink template signals and will give the similarity rate as low Euclidean distance. In this case, it will appear that the first eye-blink is single and the same procedures will be repeated at the second eye-blink. If the Euclidean distance at the first eye-blink signal is low for double eye-blinking, this time the windowing process will skip the second eye-blink and continue with the third eye-blink. In this way, by analyzing the eye-blinks in a letter duration, the Morse code equivalents of the eye-blinks will be completed for that letter. The word will show up with the completion of the letters one by one with this algorithm.
2.5 Text-to-speech synthesis
As the last step in this system, 500 words used in daily life were recorded. The experiments were carried out by asking the subjects to choose their eye-blink inputs from the words from these audio recordings. The resulting word was voiced by appropriate recording from these speech databases. The experiments were also tried by using the speech database in the Windows operating system and resulting word was voiced from this database without data set limitation.
Two values are very important when entering words with Morse code. The first is to determine the T4 moments in the signal. The second is the Xneg number. The α and β values specified in Eq. (1) and Eq. (2) change for each of the subjects at data entry. When all data entry values shown for 10 subjects in Table 3 are analyzed; It is seen that it converges to α = 2 and β = 2.2 values.
Between the letters, they left about 2.2 times more space in the letter than after single or double blinking. Average of experimental data; T1=500 (max) sample, T2=1000 (max) sample, T3=1000 (average) sample, and T4=2200 (min) sample are values that converge around. As stated in Eq. (3), |Xpos| is 1300-1500 µV, |Xneg| is 2000-2100 µV value range. In all experiments, it has been observed that the amplitude level of the positive peak is considerably lower than the amplitude level of the negative peak. Therefore, considering the noise effect, the presence of negative peaks more accurately indicates the presence of an eye-blink. It has been observed that positive peaks formed 300-400 samples earlier than negative peaks. Initially, a shift of DNG = 500 samples is taken. It is ensured that there are both negative and positive peaks within a window period. Thus, the similarity of the two templates to be compared in the DTW algorithm has been found more accurately.
Figure 6 (a) shows two signals with a single blink, in (b) these two signals are shown on the same axis set, and in (c), the aligned signals obtained from these original signals are shown. The first signal is the single blink pattern signal in the DTW algorithm, the other is the single blink signal that the subjects made at any given moment. As can be seen from the figure, there is a time shift between the two signals. The two signals do not contain the same number of sample values. When the time shifts were aligned using the DTW algorithm, the value of the Euclidean distance between single blink signals was calculated as 20.137.
Table 3. T1, T2, T3, T4, | Xpos |, | Xneg | and DNG values for 10 subjects
|
Subject |
T1 (Sample) |
T2 (Sample) |
T3 (Sample) |
T4 (Sample) |
|Xpos| (µV) |
|Xneg| (µV) |
DNG (Sample) |
|
1 2 3 4 5 6 7 8 9 10 |
490 485 493 475 488 491 465 481 480 491 |
993 975 983 976 989 988 952 961 990 985 |
1010 995 995 992 1005 1002 975 1012 995 1002 |
2212 2256 2213 2202 2401 2207 2387 2276 2209 2235 |
1315 1398 1301 1489 1445 1400 1398 1395 1495 1325 |
2050 2014 2095 2095 2084 2052 2041 2020 2101 2074 |
313 395 325 345 412 486 366 399 452 305 |
Figure 6. (a) Two signals with a single eye-blink (b) Indication of the signals in (a) on the same axis set (c) Aligned signals obtained from the original signals in (b)
Figure 7. (a) Two signals with a double eye-blink (b) Indication of the signals in (a) on the same axis set (c) Aligned signals obtained from the original signals in (b)
Figure 8. (a) Signals with a single eye-blink and a double eye-blink (b) Indication of the signals in (a) on the same axis set (c) Aligned signals obtained from the original signals in (b)
Figure 7 (a) shows two signals with double blinking, in (b) these two signals are shown on the same axis set and in (c), the aligned signals obtained from these signals are shown. The first signal is the double-blink pattern signal in the DTW algorithm, and the other is the double-blink signal made by the subjects. As can be seen from the figure, there is a time shift between the two signals. When the time shifts are aligned using the DTW algorithm, the value of the Euclidean distance between the double blink signals is calculated as 27,792.
Figure 8 (a) shows two signals containing single and double blinking, in (b) these two signals are shown on the same axis set and in (c) the aligned signals obtained from these signals are shown. As can be seen from the figure, when these two different signals are aligned using the DTW algorithm, the value of the Euclidean distance between the single and double blink signals is calculated as 172,014.
As a result, the value of the Euclidean distance between similar signals is considerably lower than the value of the Euclidean distance between different signals. The value found as a result of the comparison clearly indicates that the current test signal is single or double blinking. It is possible to create the Morse codes of all the eye-blinks in the letter by determining whether a blink is odd or even with the windowing placed at the beginning of each blink in the letter.
In order to determine the text entry speed and accuracy value of the system, signals were received from 25 healthy subjects aged 20-55 years. A total of 400 records, 40 each, were taken from these subjects for 16 expressions used in daily life. The subjects were first asked to write "TEA" with blinking eyes. All subjects wrote the word with 100% accuracy within an average of 17 seconds, and then the word was translated into speech. In the attempts made to increase the speed of text entry, there was no change as a result of writing the correct word. The main point here is the value of β. If this value does not fall below 2.2, the system certainly does not make any mistake. When the data entry speed is increased, which is very difficult for patients, there will be no problems as long as the waiting period is not reduced. Both the waiting times at the end of the letter and word are statistical values which taken from the persons who entered data. β value is adaptive and can be easily changed for the patient who will use the system.
In the experiments, it has been calculated that an average time between 1 and 13 s is required for one letter. For example, the letter E is the character that is typed in the fastest time. Because it is coded with a single point in Morse Alphabet. The significant lengthening of the entry time for some letters is due to the large number of characters in the letter code and the waiting time between these characters. In Table 4, the words and sentences frequently used in daily life and the experimental results of entry time are given.
In addition to the experimental results given in Table 4, each subject was asked to randomly select 10 out of 500 sound recordings. When these Morse coded signals were given to the system, only 2 inputs are voiced incorrectly in 250 entry. 99.2% accuracy rate was achieved.
In this study; a system has been developed that converts the thoughts of motor neuron patients such as paralysis, lock-in syndrome and amyotrophic lateral sclerosis, from eye blink signals to speech.
Eleven different studies in the literature on visual text entry systems is given in Table 1. Cost, speed and high precision are most effective parameters to compare these studies. In addition, the number of words that can be used in system can also be added to these parameters. With these parameters, the analysis of the system is possible as follows.
The most important advantage of this system is that has an unlimited database with a very high accuracy rate and low cost. The designed system has achieved nearly 100% success in all testings by both person-dependent and person-independent data as seen in Table 4. In rare cases where the $\beta<2.2$, 0.1% -0.2% inaccuracy has been observed in the system. In addition, it does not have any word restrictions and word database. Therefore, the conversion of every word to speech is the biggest advantage of the system besides its simplicity and high precision.
The disadvantage of this system is the length of the window to be used at the beginning of each eye-blink and the waiting times at the end of the letters. In this study, an average window time was taken by statistically examining different people eye-blink durations. However, these values can be easily adjusted according to the user using the system.
This system has a distinct advantage in terms of cost and accuracy over eye tracking systems made in this field or even introduced to the market. The total cost of the system is the sum of the prices of a Neurosky mindwave device and a computer. The cost of the system can be reduced considerably with a single-channel EEG device and mobile phone software instead of a Neurosky device and a computer. The text entry rate per minute is variable according to Morse code and your blinking ability. However, it is seen that in Table 4, it is above the average when compared to other systems. Other advantages include the ability to work even in a dark environment, no need to constantly look at the screen in a concentrated manner, no discomfort to the patient due to the single number of electrodes used, and no risk of cataract formation in long-term use. However, it is true that no system can replace the sound production of a healthy person. Continuous blinking is a troublesome process. Considering that this system is designed to fulfill the basic needs and desires of the unhealty person, this discomfort can be ignored.
Table 4. Text entry time and accuracy value of expressions frequently used in daily life
|
Word / Phrase |
Entry time (s) |
Repetitions of each expression (25 subject) |
Inaccurate Output Speech |
Accuracy (%) |
|
NO TEA YES OPEN WANT SURE THINK HELP ME THANK YOU NOT YET I GOT IT I AM GOOD I THINK SO I HOPE SO I KNEW IT I CAN’T WAIT |
19.5 17 30 41.5 34.5 37.5 48 64.5 95 57 56.5 82 85.5 84 64 87 |
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 |
- - - 1 1 1 2 1 2 3 3 3 2 2 1 3 |
100 100 100 99.9 99.9 99.9 99.8 99.9 99.8 99.7 99.7 99.7 99.8 99.8 99.9 99.7 |
Although it is developed for the medical sector, it can be considered as an alternative option for military communication in very noisy environments or which environments needs absolute silence.
To summarize the above analysis, this study is at the top with its high accuracy rate compared to the other systems in Table 1. Its cost is quite affordable among others. In terms of speed, it is at an average level among its peers.
[1] Jr. Pinheiro, C.G., Naves, E.L., Pino, P., Losson, E., Andrade, A.O., Bourhis, G. (2011). Alternative communication systems for people with severe motor disabilities: A survey. Biomedical Engineering Online, 10(1): 1-28. http://doi.org/10.1186/1475-925X-10-31
[2] Wahy, N., Mansor, W. (2010). EEG based home lighting system. In 2010 International Conference on Computer Applications and Industrial Electronics, pp. 379-381. http://doi.org/10.1109/ICCAIE.2010.5735107
[3] Abd Rani, M.S., Mansor, W. (2009). Detection of eye blinks from EEG signals for home lighting system activation. In 2009 6th International Symposium on Mechatronics and Its Applications, pp. 1-4. http://doi.org/10.1109/ISMA.2009.5164828
[4] Gupta, S.S., Soman, S., Raj, P.G., Prakash, R., Sailaja, S., Borgohain, R. (2012). Detecting eye movements in EEG for controlling devices. In 2012 IEEE International Conference on Computational Intelligence and Cybernetics (CyberneticsCom), pp. 69-73. http://doi.org/10.1109/CyberneticsCom.2012.6381619
[5] Chambayil, B., Singla, R., Jha, R. (2010). EEG eye blink classification using neural network. In Proceedings of the World Congress on Engineering, pp. 2-5.
[6] Singla, R., Chambayil, B., Khosla, A., Santosh, J. (2011). Comparison of SVM and ANN for classification of eye events in EEG. Journal of Biomedical Science and Engineering, 4(1): 62-69. http://doi.org/10.4236/jbise.2011.41008
[7] Rihana, S., Damien, P., Moujaess, T. (2013). EEG-eye blink detection system for brain computer interface. In Converging Clinical and Engineering Research on Neurorehabilitation (pp. 603-608). Springer, Berlin, Heidelberg. http://doi.org/10.1007/978-3-642-34546-3_98
[8] Güner, M. (2019). Biomedical signal-based control system design. M.Sc. thesis. Department of Electronics and Communications Engineering, Yildiz Technical University, Istanbul, Turkey.
[9] Afandi, J. (2019). EOG-Controlled OMNI-Directional Wheelchair. M.Sc. thesis. Department of Biomedical Engineering, İzmir Katip Çelebi University, Izmir, Turkey.
[10] Bhuyain, M.F., Shawon, M.A.U.K., Sakib, N., Faruk, T., Islam, M.K., Salim, K.M. (2019). Design and development of an EOG-based system to control electric wheelchair for people suffering from quadriplegia or quadriparesis. In 2019 International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST), pp. 460-465. http://doi.org/10.1109/ICREST.2019.8644378
[11] Kağıtçı, A. (2019). For patient with end-stage amyotrophic lateral sclerosis (ALS) patients, intelligent bed control due to simultaneous patient eye movement. M.Sc. thesis. Department of Mechatronics Engineering, Selçuk University, Konya, Turkey.
[12] Soman, S., Murthy, B.K. (2015). Using brain computer interface for synthesized speech communication for the physically disabled. Procedia Computer Science, 46: 292-298. http://doi.org/10.1016/j.procs.2015.02.023
[13] Akram, F., Han, S.M., Kim, T.S. (2015). An efficient word typing P300-BCI system using a modified T9 interface and random forest classifier. Computers in Biology and Medicine, 56: 30-36. http://doi.org/10.1016/j.compbiomed.2014.10.021
[14] Mukherjee, K., Chatterjee, D. (2015). Augmentative and alternative communication device based on eye-blink detection and conversion to Morse-code to aid paralyzed individuals. In 2015 International Conference on Communication, Information & Computing Technology (ICCICT), pp. 1-5. http://doi.org/10.1109/ICCICT.2015.7045754
[15] Majaranta, P., Räihä, K.J. (2002). Twenty years of eye typing: Systems and design issues. In Proceedings of the 2002 Symposium on Eye Tracking Research & Applications, pp. 15-22. http://doi.org/10.1145/507072.507076
[16] Hansen, J.P., Hansen, D.W., Johansen, A.S. (2001). Bringing Gaze-based Interaction Back to Basics in Universal Access In HCI, C. Stephanidis.
[17] Shein, F., Hamann, G., Brownlow, N., Treviranus, J., Milner, M., Parnes, P. (1991). WiViK: A visual keyboard for Windows 3.0. In Proceedings of the 14th Annual Conference of RESNA, pp. 160-162.
[18] MacKenzie, I.S., Ashtiani, B. (2011). BlinkWrite: efficient text entry using eye blinks. Universal Access in the Information Society, 10(1): 69-80.
[19] Özbek Ülkütaş, H. (2015). Development of computer based eye-writing system by using EOG. M.Sc. thesis. Department of Biomedical Engineering, Başkent University, Ankara, Turkey.
[20] Usakli, A.B., Gurkan, S. (2009). Design of a novel efficient human–computer interface: An electrooculagram based virtual keyboard. IEEE Transactions on Instrumentation and Measurement, 59(8): 2099-2108. http://doi.org/10.1109/TIM.2009.2030923
[21] Miniotas, D., Spakov, O., Evreinov, G.E. (2003). Symbol creator: An alternative eye-based text entry technique with low demand for screen space. In INTERACT.
[22] Majaranta, P., Räihä, K.J. (2007). Text entry by gaze: Utilizing eye-tracking. Text Entry Systems: Mobility, Accessibility, Universality, 175-187. http://doi.org/10.1016/B978-012373591-1/50009-7
[23] Zhang, C., Yao, R., Cai, J. (2018). Efficient eye typing with 9-direction gaze estimation. Multimedia Tools and Applications, 77(15): 19679-19696. https://doi.org/10.1007/s11042-017-5426-y
[24] Porta, M., Turina, M. (2008). Eye-S: A full-screen input modality for pure eye-based communication. In Proceedings of the 2008 Symposium on Eye Tracking Research & Applications, pp. 27-34. http://doi.org/10.1145/1344471.1344477
[25] Lee, K.R., Chang, W.D., Kim, S., Im, C.H. (2016). Real-time “eye-writing” recognition using electrooculogram. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(1): 37-48. http://doi.org/10.1109/TNSRE.2016.2542524
[26] Lotte, F., Bougrain, L., Clerc, M. (1999). Electroencephalography (EEG)-based brain-computer interfaces. Wiley Encyclopedia of Electrical and Electronics Engineering, 1-20. http://doi.org/10.1002/047134608X.W8278
[27] https://store.neurosky.com/pages/mindwave, accessed on 10 October 2020.
[28] Carron, L.P. (1991). Morse code: the essential language (No. 69). Amer Radio Relay League.
[29] Kruskall, J.B., Liberman, M. (1983). The symmetric time warping algorithm: From continuous to discrete. Time Warps, String Edits and Macromolecules.
[30] Zhou, N., Wang, J., Wang, Q. (2016). A novel estimation method of metering errors of electric energy based on membership cloud and dynamic time warping. IEEE Transactions on Smart Grid, 8(3): 1318-1329. https://doi.org/10.1109/tsg.2016.2619375
