Fazia Sbargoud | Mohamed Djeha* | Mohamed Guiatni | Noureddine Ababou
© 2019 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
The electromyography (EMG) and electroencephalography (EEG) are two frequently used modalities of bio-signals in the field of bio-robotics. However, it is insufficient to use a single isolated modality, due to the presence of artifacts, the lack of information, etc. To solve the problem, this paper proposes an EEG/EMG data fusion method that take advantages of both signals and overcome their drawbacks to achieve accurate identification of movements. The two types of bio-signals were preprocessed through wavelet packets transform (WPT) and classified by the artificial neural network (ANN). Then, the belief theory was introduced to allow for model uncertainty and imprecision, which adapts to the ambiguities and conflicts between sources. Experimental results show that the proposed EEG/EMG data fusion method outperformed the strategies based on only one modality. The research findings provide new impetus to bio-robotic applications.
wavelet packet transform, artificial neural networks, belief theory, data fusion, hand movement identification, electro-physiological signals, electromyography, electroencephalography
Current advances in bio-signal sensors, data acquisition, embedded systems and processing techniques contribute to the integration of physiological signals in a wide variety of clinical and non-clinical settings like medical diagnosis [1], biorobotics [2], brain-computer interfaces (BCI) or brain-machine interfaces (BMI) [3], biometrics [4]. These bio-signals use different modalities: the electrocardiography (ECG), the electromyography (EMG), the electrooculography (EOG), the electrocorticography (ECoG), the electroencephalography (EEG), the positron emission tomography (PET), the magnetic resonance imaging (MRI), the functional MRI (fMRI), and the diffusion tensor imaging (DTI).
Due to a large number of artifacts affecting the measurement and to the lack of information in a single signal; the use of an isolated modality can often be insufficient. Add to that, the source’s heterogeneous character and modalities acquisition are challenging issues. In order to overcome these limitations, data fusion allows to combine complementary properties of several single modality methods aiming to separately improve each of them [5]. Fusion can use complex structured raw data and noticeable features giving a normalized workspace easing the synergetic relationship that improves decision-making processes. This field of research has been named in different ways, for instance, multimodal fusion, sensor data fusion, mixture of experts, decision fusion, classifier combiners, and multiway signal processing.
Among the different modalities, EMG is one of the most used in bio-robotic applications due to the fact that it reflects directly the muscle activity of the user following the human motion intention. Numerous applications use EMG-based control method such as intelligent wheelchairs [6-8], prosthetics [9, 10], and exoskeletons-orthoses [11-13]. Meanwhile, EMG-based control approaches have some limitations due to their dependence on the user (the generated signal in the case of a paralyzed person is abnormal) and on the target application.
Besides, the EEG-based control methods involve also a lot of attention in the bio-robotic applications. These methods decode the user’s brain signals to control robots such as prosthetics [14, 15], exoskeletons, orthoses [16-18] and wheelchairs [19, 20]. Unfortunately, the use of an isolated EEG signal is not fully recognized in the bio-robotic domain because of the low reliability and data transfer rate [5]. In addition, the artifacts in raw EEG recordings, such as EMG, ECG and power signal, disturb the EEG information. The combination of both signal sources might be a promising approach allowing to take advantages of each signal and reduce their disadvantages. Lalitharatne et al. [5] have presented a review on hybrid/fusion EEG-EMG interfaces proposing important design features. They have demonstrated the interesting potential for hybrid EEG-EMG-based control approaches to be used in bio-robotic applications.
However, it has been remarked that there is still significant need to improve their effectiveness and to eventually bring these technologies from laboratories to real life. Among the various fusion techniques, cited in the literature, few of them have been used for EMG/EEG fusion. For instance, Leeb et al. [21, 22] explored the parallel use of EEG and EMG modalities. The user is equipped with an EEG signal recording headset with 16 electrodes and 4 electrodes for the acquisition of EMG signals (two on each arm). The data fusion step is performed after the classification of each source separately. Two fusion methods are compared in the context of muscle fatigue. It has been found that the Bayesian fusion method is more robust than the average classification results. Xie et al. [23] aimed to identify the movement intention of patients with perceptual-motor dysfunction using EEG and EMG signals. They proposed a combined algorithm based on Local Mean Decomposition (LMD) and multiscale entropy (MSE) as features. A pattern recognition model was designed based on the Extreme Learning Machine technique (ELM). Sherwani and Kumar [24] propose an algorithm for fusion of EEG and EMG signals for the detection of walking intention. The used approach for the fusion of the two signals is based on the two classifiers outputs weighting. The efficiency of data fusion has been evaluated in the context of simulated muscle fatigue. Tang et al. [25] used the EMG and EEG data acquisition methods of a lower limb rehabilitation system for patients with stroke. The wavelet transform is used to analyze the time and frequency domain, which provides a good feature vector for the dynamic analysis and motion recognition of EMG signals. However, no fusion technique is used. The authors used only the coherence of EEG and EMG signals to extract the activation signals as the control side of rehabilitation.
To the best of our knowledge, this paper describes the first integration of the Belief Theory [26] for EMG/EEG data fusion. This fusion approach aims to take advantages of both EMG and EEG signals to overcome the drawbacks of each one in order to improve the interpretation of the user intention. The most frequently cited advantages of this theory are its capacity to represent ignorance, and that does not need prior probabilities. The theory of evidence proposed by G. Shafer is an extension of Dempster’s idea using the Bayesian probabilities [26]. This theory is commonly referred to as Dempster-Shafer Theory (DST). It has been used in a variety of perceptual activities including sensor fusion [27-29], scene interpretation [30, 31], target recognition [32], and verification [33, 34]. We propose the integration of the Belief theory for multi-sensor fusion of physiological signals (EEG and EMG). This theory allows to model uncertainty and imprecision of information. It also takes into account ambiguities and conflicts between sources. We aim to decode three movements based on the EEG and the EMG signals. The EEG signals can compensate the missing EMG signals. Moreover, in the case of exoskeletons control, some required muscles or nerves for the EMG signals might be disconnected or paralyzed, leading the EEG signals to compensate them.
This paper is organized as follows: Section 2 presents the used materials for signal’s recording and describes the used experimental protocol for the dataset construction. Section 3 introduces the signal acquisition techniques and data processing, feature extraction and preliminary classification. Section 4 introduces the mathematical formalism of the belief theory and its application on the EEG and the EMG data fusion. Section 5 discusses the achieved results.
Before describing of the data acquisition methodology, we specify that we aim to decode the targeted movements:
Figure 1. Considered hand movement
Figure 2. Raw EEG signal (blue), filtered EEG signal (black) and segmentation window (red)
2.1 EEG signal acquisition
An EEG acquisition headset NeuroSky MindWave headset has been used. We found it more suitable (easier to connect and wear) and affordable (low cost). This device has a single dry electrode placed on the front left eye at the Fp1 position according to the 10/20 system. The reference electrode is placed below the left ear. The headset contains Neurosky Thinkgear technology that measures the EEG signal at a sampling frequency of 512 Hz, then transmits data using a dynamic link library (under both C++ and Matlab software) to the computer via Bluetooth wireless link. A recording sample of the EEG signal is plotted in Figure 2.
2.2 EMG signal acquisition
For the acquisition of the EMG signals, it is necessary to identify the responsible muscles for pronation, supination and hand closure movements. Based on the muscle’s palpation, three muscles are identified, namely the flexor digitorum, quadratus pronator and supinator. However, only two EMG signals are exploited and for each signal acquisition, three gelled surface electrodes (non-invasive technique) are used: two electrodes for differential measurement and a reference one. The EMG signal is recorded via a DSpace 1104 acquisition and control board at a sampling rate of 1KHz. A recording sample of the EMG signal is plotted in Figure 3.
Figure 3. Raw EMG signal (blue), filtered EMG signal (red)
2.3 Movement detection algorithm
A continuous analysis of a recorded EMG signals is necessary to detect any hand movement. The thresholding method is the simplest one for the movement detection. This method is based on performing a comparison with respect to a threshold that is calculated online for one second during which the user keeps a neutral position. The thresholds of the two EMG signals are calculated as follows:
$S_{E M G 1}=\mu_{E M G 1}+J_{E M G 1} \sigma_{E M G 1}$ (1)
$S_{E M G 2}=\mu_{E M G 2}+J_{E M G 2} \sigma_{E M G 2}$ (2)
where, µ and σ are the mean and standard envelope’s deviation during the period of inactivity, and J is a constant defined empirically as: JEMG1=JEMG2=1.2.
2.4 Dataset building
In order to build our dataset, 13 persons aged around 23 years (5 women and 8 men) have participated in our experiments. The experimental paradigm is defined as follows:
- In order to acquire the best signal; special skin preparation such as hair removal, proper gel concentration and prevention of sweat accumulation were followed. Different precautions may also be necessary (respect of the optimized distance between the electronic acquisition system and the electric power source).
- The participant makes three hand movements: closing, pronation, and supination;
- The movement must be short and strong enough.
The experiment is conducted 10 times for each movement.
3.1 Signals preprocessing
EEG and EMG signals are highly contaminated by environmental interferences (such as the 50 Hz power line electromagnetic interference) which have to be removed. EEG signals are centered then filtered by a third order Butterworth low pass filter. EMG signals were pre-amplified using an instrumentation amplifier with a gain of 5, then filtered by a second order stop band analogue filter with a cut-off frequencies of 60 Hz and 160 Hz to minimize the influence of the power line and artifacts. Finally, signals are amplified 318 times in order to make them exploitable. So, the total amplification gain is 1,590.
3.2 Feature extraction using DWT and WPT
EMG signals are non-stationary in nature. The most suitable way to study these signals is to use time-frequency methods such as the wavelet decomposition. This method was developed by Mallat in 1980 [36] for data compression in image coding. Wavelet decomposition represents the information extracted from the signal in both time and frequency domain based on applied transformations for a unique mother wavelet.
Indeed, the mother wavelet gets scaling and translation 4 operations represented by $\mathbf{a} \in \mathbb{R}^{+}$ and $b \in \mathbb{R}$ parameters as follows:
$\psi_{a, b}(t)=\frac{1}{\sqrt{a}} \psi\left(\frac{t-b}{a}\right)$ (3)
The mathematical formalism of continues wavelet transform applied to a finite energy signal s(t) is represented by the Eq. (4). It is defined as a correlation between the signal s(t) and a family of function $\psi_{(a, b)}$ (extracted from unique mother wavelet $\psi$ ).
$W_{S}(a, b, \psi)=\frac{1}{\sqrt{a}} \int s(t) \psi^{*}\left(\frac{t-b}{a}\right) d t$ (4)
where, $\psi^{*}(.)$ is the complex conjugate of $\psi(.)$. In the Discrete wavelet transform (DWT) we replace the coefficients a and b by the discrete version, and the signal is decomposed using the high pass filter and the low pass filter. The outputs of this filters are referred to as the coefficients of the Approximation (A) and details (D), respectively [37]. Wavelet packets Transform (WPT) were introduced by Coifman and Wickerhauser. WPT appears as an extension of wavelet analysis (WA) where the signal is decomposed into approximation and detail. In the case of WPT, adding to decomposing the approximations of each level, the details are also decomposed into appropriate approximations and details [38] (Figure 4). The signal’s decomposition (Details and Approximations) to level nlevel, generates a decomposition tree of the original signal giving 2nlevel nodes, represented by a number of coefficients for its appropriate filter (see Figure 4).
Figure 4. Five level decomposition using discrete wavelet packet for EEG signal
The energy calculation corresponding to each node, is calculated as follows, [39]:
$E_{n_{\text {node}}, n_{\text {level}}}=\sum_{k=1}^{k=n_{\text {coef}}}\left|d_{n_{\text {node}}, n_{\text {level}}}(k)\right|^{2}$ (5)
where, $n_{\text {node}}$ is the node position at the $n_{\text {level}} ^{t h}$ level and $n_{\text {coef}}$ is the number of its coefficients [40].
In our work, the decomposition level is selected to be 5 and the mother wavelet is 4th order Daubechies (db4). This wavelet is smooth and orthogonal and has a shape that looks similar to the patterns localized in the signal, [41].
3.3 Classification using ANN
The use of Artificial Neural Network (ANN) is common for pattern recognition, classification and identification. Rosenblatt [42] studied and implemented a classifier containing a unique layer of neurons. Subsequently, Multi-Layer Perceptron (MLP) appeared to improve the efficiency of this classifier. The idea is to predict the output by adjusting weights multiplied by the inputs to be evaluated by continuous functions that activate or deactivate the neuron. The typical structure of an ANN consists of three layers: the input layer where each neuron receives an element of the input vector. A hidden layer containing several neurons defines the complexity of the network and finally an output layer where each neuron represents a class. The network topology chosen was the feed-forward variety with one input layer. The inputs of ANNs are the extracted wavelet coefficients. For EEG signal classification, the ANN has 38 input neurons, one hidden layer with 25 neurons and three output neurons corresponding to the three targeted movements. However, for EMG signal classification, the ANN has 32 input neurons and 15 neurons in the hidden layer. The sigmoid activation function and Levenberg-Marquardt algorithm learning method are used.
4.1 Mathematical formalism of the belief theory
Several theories are available regarding information processing depending on mathematical formalism. The most known is the probability theory, which has shortcomings when knowledge is incomplete, uncertain or imprecise. For this reason, several theories appeared since 1960s, namely the fuzzy subset that has been developed by Zadeh and the belief theory initially introduced by Dempster [26, 43] and developed by Shafer [44] in a new mathematical formalism. This theory is based on the modeling of a belief in an event, expressed as a mass function allowing a good representation of knowledge [45]. The advantage of this theory is the realized compromise between incertitude and imprecision. In the belief theory, a discernment framework $\Theta$ represents all possible states of system and is defined as: $\Theta=\left\{\theta_{1}, \theta_{2}, \ldots, \theta_{N}\right\}$.
Dempster Shafer theory assigns a belief mass m to each element A of $2^{\Theta}$. The elementary mass function $m_{j}$ associated to the source $S_{j}$ (sensor, classifier...) is defined as follows:
$m: 2^{\Theta} \rightarrow[0,1]$ (6)
and
$\sum_{A \in 2^\Theta} m(A)=1$ (7)
The difference between probability theory and belief function theory is the fact that element A may be the union of more than one decision. This particularity makes it possible to model uncertainties.
4.2 Combination rules of mass functions
Implementation of a combination rule, using the obtained mass sets from each information source, provides a combined mass set. Among the combination rules, we mention the following:
(1) Dempster-Shafer orthogonal combination rule: This rule is initially introduced by Dempster [46], and modified by Shafer [44] given for every $A \in 2^{\Theta}$ by:
$m(A)=\left(m_{1} \oplus m_{2} \ldots \oplus m_{m}\right)(A)$$=\frac{1}{1-k} \sum_{B_{1} \cap \ldots \cap B_{m}}=A \prod_{j=1}^{m} m_{j}\left(B_{j}\right)$ (8)
and $m(\phi)=0$ . This rule combination is interesting in close world hypothesis. k is the factor that can eliminate the conflict. It is defined as follows:
$k=\sum_{B_{1} \cap \ldots \cap B_{m}}=\phi \prod_{j=1}^{m} m_{j}\left(B_{j}\right)$ (9)
(2) Disjunctive combination rule: This combination rule is given by considering unions not intersections.
$m(A)=\left(m_{1} \oplus m_{2} \ldots \oplus m_{m}\right)(A)$$=\sum_{B \cup_{1} \ldots \cup B_{m}}=A \prod_{j=1}^{m} m_{j}\left(B_{j}\right)$ (10)
(3) Smets combination rule: It is a non-normalized version of Dempster-Shafer combination rule. This rule may be applied in the case of the open world [47].
$m(A)=\sum_{B \cup_{1} \ldots \cup B_{m}}=A \prod_{j=1}^{m} m_{j}\left(B_{j}\right)$ (11)
and
$m(\phi)=\sum_{B \cup_{1} \ldots \cup B_{m}=\phi} \prod_{j=1}^{m} m_{j}\left(B_{j}\right)$
$m(\phi)=\sum_{B \cup_{1} \ldots \cup B_{m}}=\phi \prod_{j=1}^{m} m_{j}\left(B_{j}\right)$ (12)
(4) Yager combination rule: Yager [48] proposed a closed world model where the conflict measure is assigned to the total discernment framework $\Theta$ The conflict is thus transformed into ignorance. We get for every $A \in 2^{\Theta}$:
$m(A)=\sum_{B \cup_{1} \ldots \cup B_{m}}=A\prod_{j=1}^{m} m_{j}\left(B_{j}\right)$ (13)
and the mass of $\Theta$ is:
$m(\Theta)=1-\sum_{B \cup_{1} \ldots \cup B_{m}}=A \prod_{j=1}^{m} m_{j}\left(B_{j}\right)$ (14)
(5) Murphy combination rule: It consists of an arithmetic average of the mass functions. We get for every $A \in 2^{\Theta}$:
$m(A)=\frac{1}{n} \sum_{i=1}^{n} m_{i}(A)$ (15)
(6) Dubois and Prade combination rule: Dubois and Prade’s combination rule admits that two sources are reliable when they are not in conflict. The Dubois and Prade rule represent a reasonable compromise between accuracy and reliability [49]. We get for every $A \in 2^{\Theta}$ (combination of two sources):
$m(A)=\sum_{B, C \in 2^{\Theta} \atop B \cap C=A \neq \phi} m_{1}(B) m_{2}(C)+\sum_{B, C \in 2^{\Theta} \atop B \cup C=A\&B\cap C = \phi}m_{1}(B) m_{2}(C)$ (16)
(7) PCR combination rule: PCR (Proportional Conflict Redistribution) uses $S_{j}$ sources and masses obtained by the Dempster-Shafer rule before normalization. It consists in redistributing the conflict mass in proportion to the valid combinations [50]. The rule is defined [51] as:
$m(A)=\sum_{B, C \in 2^{\Theta} \atop B \cap C=A \neq \phi} m_{1}(B) m_{2}(C)+\sum_{B, C \in 2^{\Theta} \atop B \cap A=\phi} \frac{m_{1}^{2}(A) m_{2}(B)}{m_{1}(A)+m_{2}(B)}+\frac{m_{2}^{2}(A) m_{1}(B)}{m_{2}(A)+m_{1}(B)}$ (17)
4.3 Decision rules
After combining the mass functions of the different sources, the decision rules must provide an optimal choice of singleton $\theta_{i}$ , and among them we find:
(1) Maximum plausibility: Based on the choice of the singleton $\theta_{i}$ which gives the maximum plausibility. For the observation x we decide $\theta_{i}$ for:
$\operatorname{Pl}\left(\theta_{i}\right)=\max _{i \leq k \leq n} P l\left(\theta_{k}\right)(x)$ (18)
This criterion corresponds to the optimistic search for a solution, since it retains the upper limit of the confidence interval.
(2) Maximum credibility: The second criterion consists in choosing the singleton $\theta_{i}$ giving the maximum belief. For the observation x we decide $\theta_{i}$ for:
$\operatorname{Cr}\left(\theta_{i}\right)=\max _{i \leq k \leq n} \operatorname{Cr}\left(\theta_{k}\right)(x)$ (19)
The use of this criterion corresponds to the pessimistic search for a solution, since it only considers cases for which there is no uncertainty, and which are contained within the lower limit of the confidence interval.
(3) Maximum pignistic probability: This rule was proposed by Smets [52, 53] for the decision in the context of the belief functions. Smets proposes to define a particular probability distribution called pignistics in order to make the decision. This pignistic probability is obtained by distributing the mass m(A) equally among the elements of A. Thus, for any decision $\theta_{i}$, the pignistic probability is defined by:
$\operatorname{bet}\left(\theta_{i}\right)=\sum_{A \in 2\Theta, \theta_{i} \in A} {} \frac{m(A)}{|A|(1-m(\phi))}$ (20)
With $|A|$ is the cardinal of A. The criterion of the maximum pignistic probability is equivalent of deciding $\theta_{i}$ for the observation x if :
$\operatorname{bet}\left(\theta_{i}\right)=\max _{i \leq k \leq n} \operatorname{bet}\left(\theta_{k}\right)(x)$ (21)
In this section, we propose to apply the theory of evidence for EMG and EEG signals. First, the classification results of both type of signals are given separately. Thereafter, full details of the proposed fusion approach will be presented. The purpose is to identify the desired movements of a user according to three new classes corresponding to the following directions: ”front”, ”right”, ”left”. This is done using EMG signals issued from the hand movements (close hand, supination and pronation), and EEG signals generated by the eye movement in three directions (up, right, left). To calculate the efficiency rate of the corresponding classification, the $F_{\text {measure}}$ was chosen because it represents a combination of accuracy and recall, such as:
Precision $=\frac{N_{c a i}}{N_{a i}}$ (22)
$\operatorname{Recall}=\frac{N_{c a i}}{N_{i}}$ (23)
where, $N_{i}$ represents the number of vectors of the class i, $N_{a i}$ represents the number of vectors assigned to the class i and $N_{c a i}$ represents the number of vectors correctly assigned to the class i.
$F_{\text {measure}}=2 \frac{\text {Precision } \times \text {Recall}}{\text {Precision}+\text {Recall}}$ (24)
The arithmetic average value of $F_{\text {measure}}$ is calculated.
Table 1. Confusion matrix of the EEG signal classification
|
Output Target |
Up |
Left |
Right |
Accuracy |
|
Up |
34 |
15 |
2 |
66.7 |
|
Left |
25 |
39 |
25 |
43.8 |
|
Right |
6 |
11 |
38 |
69.1 |
|
Accuracy (%) |
52.3 |
60.0 |
58.5 |
56.9 |
|
Criteria Classes |
Up |
Left |
Right |
|
Precision (%) |
66.7 |
43.8 |
69.1 |
|
Recall (%) |
52.3 |
60 |
58.5 |
|
Fmeasure (%) |
58.63 |
51.95 |
63.36 |
|
Output Target |
Close hand |
Pronation |
Supination |
Accuracy (%) |
|
Close hand |
32 |
6 |
6 |
72.7 |
|
Pronation |
0 |
23 |
17 |
57.5 |
|
Supination |
3 |
6 |
12 |
57.1 |
|
Accuracy (%) |
91.4 |
65.7 |
34.3 |
63.8 |
|
Output Target |
Close hand |
Pronation |
Supination |
|
Precision (%) |
72.7 |
57.5 |
57.1 |
|
Recall (%) |
91.4 |
65.7 |
34.3 |
|
Fmeasure (%) |
80.98 |
61.33 |
42.86 |
5.1 EEG signal classification
Using the wavelet transform’s coefficients as feature vector (38 coefficients), the classification results are obtained on a test base as shown in Table 1. Table 2 represents the Fmeasurecalculated for each class. The mean value of Fmeasure of EEG classification is found to be 57.98%.
5.2 EMG signal classification
Using energies of nodes tree of WPD at level 4 as characteristic vector (32 values), the classification results for EMG signals are represented in Table 3. Table 4 presents the Fmeasure calculated for each class. The mean value of Fmeasure for EMG signal classification is 61.72%.
The basic idea behind a hybrid EEG-EMG based-control interface is the signals fusion of both EEG and EMG in the seven control method. this fusion depends on several factors like a specific application and the user’s abilities.
5.3 EMG and EEG data fusion
The classification accuracy is not sufficient for EEG and EMG signals separately. As showed in Table 1, there is a confusion between Up and Right movement classes, and for EMG signal as showed in Table 3 there is a confusion between Pronation and Supination movement classes. This confusion and incertitude are mainly due to a small number of electrodes used for the signal measurement. This situation may be adapted to the concept of using a belief theory for the EEG and EMG data fusion as shown in Figure 5.
Figure 5. EMG - EEG data fusion scheme
The first step to apply this theory is the definition of a discernment framework $\Theta$ as follows:
$\Theta=\{\text { Front, Right, Left }\}$ (25)
We use two sources $S_{E E G}$ and $S_{E M G}$. The confusion situation existing between the classes (Up and Right) and (Pronation and Supination), may be used for the incertitude modeling by the definition of focal element of the mass functions $m_{E E G}$ and $m_{E M G}$ as:
$m_{E E G}(A) \neq 0$ with $A=\{U p, \text { Right }\}$ or $A=\{L e f t\}$ (26)
$\begin{aligned} m_{E M G}(A) & \neq 0 \text { with } A=\{\text { close hand }\} \\ \text { or } \mathrm{A} &=\{\text { Pronation, Supination }\} \end{aligned}$ (27)
For the mass function assignment, we chose to use of the modified classifiers results. The first one is related to the classification of EEG signal considering two great classes:
{Up, Right} and {Left} and the second one counts two classes: {close hand} and {Pronation, Supination}. The mass assignment for the two classes of EEG and EMG source is calculated using the following equation:
$m_{s}\left(c_{i}\right)=\frac{\text { classifications }\left(c_{i}\right)}{\text { classifications }\left(c_{i}\right)+\text { classifications }\left(c_{j}\right)}$ (28)
Table 5. Focal elements intersection of masses function
|
$\boldsymbol{S}_{E E G} / \boldsymbol{S}_{E M G}$ |
{Close hand } |
{Pronation, Supination} |
|
{Up, Right} |
{Front} |
{Right} |
|
{Left} |
ϕ |
{Left} |
Table 6. Fmeasure of EMG and EEG data fusion
|
Cr (%)/ Rule |
D-S, Yager , D-P, Murphy, Smet, Disj |
PCR |
|
Fmeasure |
68.24 |
68.18 |
|
Enhancement |
10.56 |
10.47 |
Table 5 gives the combination of the focal elements describing below, which determines the singletons defined in discernment framework $\Theta$. The data fusion is realized by combining the mass functions using different combination rules described in section 5 and the decision step is done using one decision criteria such as: maximum of plausibility or maximum of credibility or maximum of pignistic probability. In our work we compare the use of nine combination rules and as decision criteria the maximum of plausibility. Table 6 shows the results of data fusion using as evaluation the mean factor. The classification of EMG and EEG signals before the fusion gives a performance of 61.0% and 57.78%, respectively.
The fusion is performed using different conjunctive and disjunctive combination rules. There has been a 10.56% enhancement in performance, particularly for D-S, Yager and D-F rules. For the different rules, the improvement is constant for the used decision criterion, because the intersection of the focal elements of the two mass functions generate singletons.
This paper proposed the combination of two sources of electrophysiological signals namely the EEG and the EMG. Signal processing and classification techniques have been used in order to prepare data for a preliminary classification. Belief Theory based approach has been used for the fusion of the EEG and EMG signals. This approach has the ability to mathematically model any data uncertainty and imprecision. This theory is based on a specific mass functions formulation according to the target application. In this framework, EEG and the EMG signals are separately insufficient for movement identification leading to confusions between the classes. The power of a belief theory is to give the ability of modeling the uncertainty through the definition of the focal elements containing grouping classes.
Despite the fact that a single low cost EEG electrode has been used for the combination with the EMG electrodes, the proposed approach has improved the movement classification rate by more than 10%. The present developed method is mainly interesting in a large number of applications where a lake of information is a barrier for the efficiency rate of movement control applied in robotic systems. Future researches will focus on the fusion of data from multi-channels EEG acquisition devices and multi-EMG electrodes. The use of Deep Learning algorithms combined with the Believe Theory could be explored for any specific desired tasks. To achieve this objective, it is essential to increase the practicality of algorithms by enhancing both their accuracy and computational effectiveness/complexity. It is also critical to find efficient algorithms those satisfy the time and memory requirements for practical usage of EEG and EMG based classification in clinical settings such as bio-prosthesis and orthosis control, and abnormal functional corticomuscular coupling detection.
|
a |
dimensionless scale factor |
|
A, B, C |
dimensionless elements of 2Θ |
|
b |
time shift, s |
|
bet |
pignistic probability |
|
Cr |
credibility |
|
\({{d}_{{{n}_{node}},{{n}_{level}}}}(k)\) |
kth coefficient of a node at ${n}_{node}$,${n}_{level}$ |
|
E |
energy, J |
|
Fmeasure |
dimensionless weighted harmonic mean of precision and recall |
|
J |
dimensionless constant |
|
k |
dimensionless conflict factor |
|
m |
dimensionless belief mass |
|
Nai |
dimensionless number of vectors assigned to the class i |
|
Ncai |
dimensionless number of vectors correctly assigned to the class i |
|
Ni |
dimensionless number of vectors of the class i |
|
${n}_{level}$ |
dimensionless level of wavelet decomposition |
|
${n}_{node}$ |
dimensionless node position at the $n_{\text {level}}$ th level |
|
${n}_{coef}$ |
dimensionless number of coefficients of the $n_{\text {level}}$ th level |
|
s(t) |
finite energy signal, volt |
|
Pl |
plausibility |
|
S |
source |
|
SEMG |
threshold of EMG signal, volt |
|
t |
time, s |
|
Ws |
wavelet transform applied to a finite energy signal s(t) |
|
x |
observation |
| Greek symbols | |
|
$\mu$ |
mean of envelope, volt |
|
$\sigma$ |
standard deviation of envelope, volt |
|
$\phi$ |
empty set |
|
$\Psi$ |
mother wavelet |
|
$\Theta$ |
discernment framework |
|
$\theta_{i}$ |
Singleton |
[1] Das, S., Chowdhury, S.R., Saha, H. (2012). Accuracy enhancement in a fuzzy expert decision making system through appropriate determination of membership functions and its application in a medical diagnostic decision making system. Journal of Medical Systems, 36(3): 1607-1620. http://dx.doi.org/10.1007/s10916-010-9623-8
[2] AL-Quraishi, M., Elamvazuthi, I., Daud, S.A., Parasuraman, S., Borboni A. (2018). EEG-based control for upper and lower limb exoskeletons and prostheses: A systematic review. Sensors, 18(10): 3342. http://dx.doi.org/10.3390/s18103342
[3] Lebedev, M.A., Nicolelis, M.A. (2017). Brain-machine interfaces: From basic science to neuroprostheses and neurorehabilitation. Physiological Reviews, 97(2): 767-837. https://doi.org/10.1152/physrev.00027.2016
[4] Zapata, J., Duque, C., Rojas-Idarraga, Y., Gonzalez, M., Guzmàn, J., Botero, M.B. (2017). Data fusion applied to biometric identification-a review. In Colombian Conference on Computing, Springer), pp. 721-733. http://dx.doi.org/10.1007/978-3-319-66562-7_51
[5] Lalitharatne, T.D., Teramoto, K., Hayashi, Y., Kiguchi, K. (2013). Towards hybrid EEG-EMG-based control approaches to be used in bio-robotics applications: Current status, challenges and future directions. Paladyn, Journal of Behavioral Robotics, 4(2): 147-154. http://dx.doi.org/10.2478/pjbr-2013-0009
[6] Oonishi, Y., Oh, S., Hori, Y. (2010). A new control method for power assisted wheelchair based on the surface myoelectric signal. IEEE Transactions on Industrial Electronics, 57(9): 3191-3196. http://dx.doi.org/10.1109/TIE.2010.2051931
[7] Ishii, C., Konishi, R. (2016). A control of electric wheelchair using an EMG based on degree of muscular activity. In Digital System Design (DSD), Euromicro Conference on. IEEE, pp. 567-574. http://dx.doi.org/10.1109/DSD.2016.19
[8] Mahendran, R. (2014). EMG signal based control of an intelligent wheelchair. in Communications and Signal Processing (ICCSP), International Conference on. IEEE), pp. 1267-1272. http://dx.doi.org/10.1109/ICCSP.2014.6950055
[9] Dimitrov, V., Cebry, N., Őnal, C., Padır, T. (2015). Towards user-centered design of a robotic prosthetic hand with EMG control interfaces. In Proceedings of the 8th ACM International Conference on Pervasive Technologies Related to Assistive Environments. ACM, p. 14. http://dx.doi.org/10.1145/2769493.2769536
[10] Resnik, L., Huang, H.H., Winslow, A., Crouch, D.L., Zhang, F., Wolk, N. (2018). Evaluation of EMG pattern recognition for upper limb prosthesis control: A case study in comparison with direct myoelectric control. Journal of Neuro Engineering and Rehabilitation, 15(1): 23-23. http://dx.doi.org/10.1186/s12984-018-0361-3
[11] Leonardis, D., Barsotti, M., Loconsole, C., Solazzi, M., Troncossi, M., Mazzotti, C., Castelli, V.P., Procopio, C., Lamola, G., Chisari C., Bergamasco, M., Frisoli, A. (2015). An EMG-controlled robotic hand exoskeleton for bilateral rehabilitation. IEEE Transactions on Haptics, 8(2): 140-151. http://dx.doi.org/10.1109/TOH.2015.2417570
[12] Yao, S., Zhuang, Y., Li, Z., Song, R. (2018). Adaptive admittance control for an ankle exoskeleton using an EMG-driven musculoskeletal model. Frontiers in Neurorobotics, (12): 16. http://dx.doi.org/10.3389/fnbot.2018.00016
[13] Siu, H.C., Arenas, A.M., Sun, T., Stirling, L.A. (2018). Implementation of a surface electromyography-based upper extremity exoskeleton controller using learning from demonstration. Sensors, 18(2): 467. http://dx.doi.org/10.3390/s18020467
[14] Beyrouthy, T., Al Kork, S., Korbane, J.A., Abouelela, M. (2017). EEG mind controlled smart prosthetic arm-a comprehensive study. Advances in Science, Technology and Engineering Systems Journal, 2(3): 891-899. http://dx.doi.org/10.25046/aj0203111
[15] Bandara, D., Arata, J., Kiguchi, K. (2018). Towards control of a transhumeral prosthesis with EEG signals. Bioengineering, 5(2): E26 http://dx.doi.org/10.3390/bioenginee ring5020026
[16] He, Y., Eguren, D., Azorn, J.M., Grossman, R.G., Luu, T.P., Contreras-Vidal, J.L. (2018). Brain machine interfaces for controlling lower-limb powered robotic systems. Journal of Neural Engineering, 15(2): 021004. http://dx.doi.org/10.1088/1741-2552/aaa8c0
[17] AL-Quraishi, M.S., Elamvazuthi, I., Daud, S., Parasuraman, S., Borboni, A. (2018). EEG-based control for upper and lower limb exoskeletons and prostheses: A systematic review. Sensors, 18(10): 3342. http://dx.doi.org/10.3390/s18103342
[18] Crea, S., Nann, M., Trigili, E., Cordella, F., Baldoni, A., Badesa, F.J., Catalàn, J.M., Zollo, L., Vitiello, N., Aracil, N.G., Soekadar, S.R. (2018). Feasibility and safety of shared EEG/EOG and vision-guided autonomous whole-arm exoskeleton control to perform activities of daily living. Scientific Reports, 8(1): 10823. http://dx.doi.org/10.1038/s41598-018-29091-5
[19] Swee, S.K., Kiang, K.D.T., You, L.Z. (2016). EEG controlled wheelchair. MATEC Web of Conferences, 51: 1-9. http://dx.doi.org/10.1051/matecconf/20165102011
[20] Kim, K.T., Suk, H.I., Lee, S.W. (2018). Commanding a brain-controlled wheelchair using steady-state somatosensory evoked potentials. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(3): 654-665. http://dx.doi.org/10.1109/IEMBS.2010.5626233
[21] Leeb, R., Sagha, H., Chavarriaga, R., Millan, J.D.R. (2010). Multimodal fusion of muscle and brain signals for a hybrid-BCI. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4343-4346. http://dx.doi.org/10.1109/IEMBS.2010.5626233
[22] Leeb, R., Sagha, H., Chavarriaga, R., Millàn, J.d.R. (2011). A hybrid brain-computer interface based on the fusion of electroencephalographic and electromyographic activities. Journal of Neural Engineering, 8(2). http://dx.doi.org/10.1088/1741-2560/8/2/025011
[23] Xie, P., Chen, X., Ma, P., Lia, X., Su, Y. (2013). Identification method of human movement intention based on the fusion feature of EEG and EMG. In World Congress on Engineering, London, U.K, (II), pp. 4343-6.
[24] Sherwani, K.I., Kumar, N. (2016). Fusion of EEG and EMG signals for gait intent detection. MMU J. Manag. Tech, (1): 50-55.
[25] Tang, H., Li, G., Sun, Y., Jiang, G., Kong, J., Ju, Z., Jiang, D. (2017). Rehabilitation training for leg based on EEG-EMG fusion. Intelligent Robotics and Applications, 10462: 517-527. http://dx.doi.org/10.1007/978-3-319-65289-4_49
[26] Dempster, A.P. (1967). Upper and lower probabilities induced by a multivalued mapping. The Annals of Mathematical Statistics, 38(2): 325-339. http://dx.doi.org/10.1214/aoms/1177698950
[27] Murphy, R.R. (1998). Dempster-shafer theory for sensor fusion in autonomous mobile robots. IEEE Transactions on Robotics and Automation, 14(2): 197-206. http://dx.doi.org/10.1109/70.681240
[28] Wu, H., Siegel, M., Stiefelhagen, R., Yang, J. (2002). Sensor fusion using dempster-shafer theory [for context-aware hci]. In Instrumentation and Measurement Technology Conference, Proceedings of the 19th IEEE, (1): 7-12. http://dx.doi.org/10.1109/IMTC.2002.1006807
[29] Chen, C., Jafari, R., Kehtarnavaz, N. (2015). Improving human action recognition using fusion of depth camera and inertial sensors. IEEE Transactions on Human-Machine Systems, 45(1): 51-61. http://dx.doi.org/10.1109/THMS.2014.2362520
[30] Dìaz-Màs, L., Muñoz-Salinas, R., Madrid-Cuevas, F.J., Medina-Carnicer, R. (2010). Shape from silhouette using dempster-shafer theory. Pattern Recognition, 43(6): 2119-2131. http://dx.doi.org/10.1016/j.patcog.2010.01.001
[31] Luo, H., Liu, C., Wu, C., Guo, X. (2018). Urban change detection based on dempster-shafer theory for multitemporal very high-resolution imagery. Remote Sensing, 10(7): 980. http://dx.doi.org/10.3390/rs10070980
[32] Muñoz-Salinas, R., Medina-Carnicer, R., Madrid-Cuevas, F.J., Carmona-Poyato, A. (2009). Multi-camera people tracking using evidential filters. International Journal of Approximate Reasoning, 50(5): 732-749. http://dx.doi.org/10.1016/j.ijar.2009.02.001
[33] Jiang, W., Xie, C., Zhuang, M., Tang, Y. (2017). Failure mode and effects analysis based on a novel fuzzy evidential method. Applied Soft Computing, 57: 672-683. http://dx.doi.org/10.1016/j.asoc.2017.04.008
[34] Hou, D., He, H., Huang, P., Zhang, G., Loaiciga, H. (2013). Detection of water-quality contamination events based on multi-sensor fusion using an extended dempster-shafer method. Measurement Science and Technology, 24(5): 801-818. http://dx.doi.org/10.1088/0957-0233/24/5/055801
[35] Djeha, M., Sbargoud, F., Guiatni, M., Fellah, K., Ababou, N. (2017). A combined EEG and EOG signals based wheelchair control in virtual environment. In 5th International Conference on Electrical Engineering - Boumerdes (ICEE-B), pp. 1-6. http://dx.doi.org/10.1109/ICEE-B.2017.8192087
[36] Mallat, S.G. (1989). A theory for multiresolution signal decomposition: The wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7): 674-693. http://dx.doi.org/10.1109/34.192463
[37] Gargour, C., Gabrea, M., Ramachandran, V., Lina, J.M. (2009). A short introduction to wavelets and their applications. IEEE Circuits and Systems Magazine, 9(2): 57-68. http://dx.doi.org/10.1109/MCAS.2009.932556
[38] Blanco, S., Figliola, A., Quiroga, R.Q., Rosso, O., Serrano, E. (1998). Time frequency analysis of electroencephalogram series. iii. Wavelet packets and information cost function. Physical Review E, 57(1): 932. http://dx.doi.org/10.1103/ PhysRev E.57.932
[39] Subasi, A. (2012). Classification of EMG signals using combined features and soft computing techniques. Applied Soft Computing, 12(8): 2188-2198. http://dx.doi.org/10.1016/j.asoc.2012.03.035
[40] Akansu, A.N., Haddad, P.A., Haddad, R.A., Haddad, P.R. (2001). Multiresolution Signal Decomposition: Transforms, Subbands, and Wavelets.
[41] Wali, M.K., Muruagappan, M., Ahmad, R. (2013). Wavelet packet transform based driver distraction level classification using EEG. Mathematical Problems in Engineering, 2013: 1-10. http://dx.doi.org/10.1155/2013/297587
[42] Rosenblatt, F. (1961). Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Cornell Aeronautical Lab Inc Buffalo NY, Tech. Rep. AD0256582.
[43] Dempster, A.P. (2008). A generalization of bayesian inference. in Classic works of the dempster-shafer theory of belief functions. Journal of the Royal Statistical Society. Series B (Methodological), 30(2): 73-104. http://dx.doi.org/10.1007/978-3-540-44792-4_4
[44] Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press, 42.
[45] Lefebvre, E. (2007). Advances and Challenges in Multisensor Data and Information Processing. IOS Press, 8.
[46] Dempster, A.P. (2008). Upper and lower probabilities induced by a multivalued mapping. Classic Works of the Dempster-Shafer Theory of Belief Functions. Springer, 57-72. http://dx.doi.org/10.1007/978-3-540-44792-4_3
[47] Smets, P. (2000). Data fusion in the transferable belief model. In Information Fusion, FUSION 2000. Proceedings of the Third International Conference on. IEEE, pp. PS21-PS33. http://dx.doi.org/10.1109/IFIC.2000.862713
[48] Yager, R.R. (1987). On the dempster-shafer framework and new combination rules. Information Sciences, 41(2): 93-137. http://dx.doi.org/10.1016/0020-0255(87)90007-7
[49] Dubois, D., Fargier, H., Prade, H., Perny, P. (2002). Qualitative decision theory. Journal of the ACM, 49(4): 455-495. http://dx.doi.org/10.1145/581771.581772
[50] Scholte, K.A., van Norden, W.L. (2009). Applying the PCR6 rule of combination in real time classification systems. In Information Fusion, FUSION’09. 12th International Conference on. IEEE, pp. 1665-1672.
[51] Sebbak, F., Benhammadi, F. Chibani, A., Amirat, Y., Mokhtari, A. (2013). New evidence combination rules for activity recognition in smart home. In Information Fusion (FUSION), 16th International Conference on. IEEE, pp. 46-52.
[52] Smets, P., Kennes, R. (1994). The transferable belief model. Artificial Intelligence, 66(2): 191-234. http://dx.doi.org/10.1016/0004-3702(94)90026-4
[53] Smets, P. (1990). Constructing the pignistic probability function in a context of uncertainty. Uncertainty in Artificial Intelligence, (89): 29-40. http://dx.doi.org/10.1016/B978-0-444-88738-2.50010-5
