DOI:10.32604/cmc.2022.018318
| Computers, Materials & Continua DOI:10.32604/cmc.2022.018318 | |
| Article |
Hemodynamic Response Detection Using Integrated EEG-fNIRS-VPA for BCI
1National University of Sciences and Technology (NUST), Islamabad, Pakistan
2Intelligent Robotics Lab, National Center of Artificial Intelligence, National University of Sciences and Technology (NUST), Islamabad, Pakistan
3Department of Mechatronics Engineering, Air University, Islamabad, Pakistan
4Institute of Space Technology, Islamabad, Pakistan
*Corresponding Author: M. Jawad Khan. Email: jawad.khan@smme.nust.edu.pk
Received: 04 March 2021; Accepted: 21 April 2021
Abstract: For BCI systems, it is important to have an accurate and less complex architecture to control a device with enhanced accuracy. In this paper, a novel methodology for more accurate detection of the hemodynamic response has been developed using a multimodal brain-computer interface (BCI). An integrated classifier has been developed for achieving better classification accuracy using two modalities. An integrated EEG-fNIRS-based vector-phase analysis (VPA) has been conducted. An open-source dataset collected at the Technische Universität Berlin, including simultaneous electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals of 26 healthy participants during n-back tests, has been used for this research. Instrumental and physiological noise removal has been done using preprocessing techniques followed by individually detecting activity in both modalities. With resting state threshold circle, VPA has been used to detect a hemodynamic response in fNIRS signals, whereas phase plots for EEG signals have been constructed using Hilbert Transform to detect the activity in each trial. Multiple threshold circles are drawn in the vector plane, where each circle is drawn after task completion in each trial of EEG signal. Finally, both processes are integrated into one vector-phase plot to get combined detection of hemodynamic response for activity. Results of this study illustrate that the combined EEG-fNIRS VPA yields considerably higher average classification accuracy, that is 91.35%, as compared to other classifiers such as support vector machine (SVM), convolutional neural networks (CNN), deep neural networks (DNN) and VPA (with dual-threshold circles) with classification accuracies 82%, 89%, 87% and 86% respectively. Outcomes of this research demonstrate that improved classification performance can be feasibly achieved using multimodal VPA for EEG-fNIRS hybrid data.
Keywords: EEG-fNIRS hybrid BCI; vector-phase analysis; hemodynamic response detection
A brain-computer interface (BCI) is a pathway for communication between brain thoughts and computer to achieve hardware control, without any dependence on channels like nerves and muscles. [1,2]. The main purpose of a BCI is to equip physically impaired people, especially with motor disabilities, with the facility to communicate with the help of their brain signals [3,4]. A BCI is used to detect and interpret brain signals to control the devices [5]. A BCI helps users develop an interface between their brain and peripheral devices without any kind of physical movement [6–9]. Different neuroimaging modalities measurbrain activity via different aspects of brain signals [10]. Various assistive rehabilitative devices have been controlled using different types of BCI systems [11], such as electroencephalography (EEG) [12,13] and functional near-infrared spectroscopy (fNIRS) [14,15] etc.
EEG and fNIRS are two of the significant non-invasive modalities. Being portable, cost effective and less noisy, counts as the major benefits of these modalities [6,16]. EEG is a signal formed by the field potential generated due to the collective and synchronous action of neurons. As a non-invasive BCI, voltage fluctuations can be recorded with the help of electrodes placed on the scalp [17]. EEG is one of the modalities which are mostly used nowadays for research [18,19]. EEG has its own strengths and drawbacks. For example, EEG possesses good temporal resolution (~0.05 s), whereas its spatial resolution is poor (~10 mm) [4]. Localization of activity in the brain is inaccurate due to poor spatial resolution of EEG [20]. EEG is also more sensitive to motion artifacts compared to fNIRS [21,22].
fNIRS is one of the emerging BCIs which records brain activity as blood oxygen level changes. It uses near-infrared-range light of wavelength 650~1000 nm to estimate the variations in the concentration of oxygenated hemoglobin (∆HbO) and deoxygenated hemoglobin (∆HbR) [23]. fNIRS also has its strengths and drawbacks. For example, fNIRS’s temporal resolution (~1 s) is just moderate, whereas it offers reasonably better spatial resolution(~5 mm) than EEG [4].
A hybrid BCI system is usually comprised of two BCIs. It can also be composed of at least one BCI system and another system (ECG and EMG etc.) [24,25]. It can also have one brain signal and a non-brain signal as its inputs. A hybrid BCI is expected to achieve better performance and classification accuracy than other conventional systems [26]. EEG and fNIRS measure the complementary characteristics of brain signals, i.e., electrophysiological and hemodynamic aspects, so a hybrid BCI integrates more information producing better results than using individual modalities [27–30].
Accuracy is one of the major concerns of most researchers to evaluate brain-computer machines. There is a need to develop a hybrid EEG-fNIRS architecture that can enhance the accuracy for better performance and control of devices. Thus, to further improve the accuracy, in this study, we have devised a novel methodology using an open-source meta-dataset comprising of simultaneous EEG and fNIRS data of 26 healthy subjects, integrated at Technische Universität Berlin is available online (http://doc.ml.tu-berlin.de/simultaneous_EEG_NIRS/). The n-back test has been carried out on all these participants (more explanation in next section) [31].
The rest of this study is organized as follows. Section 2 investigates the literature work and presents a review on the techniques/methods for early response detection using EEG. Section 3 provides details of the dataset and discusses the preprocessing process. Section 4 describes the design details of the proposed method. Section 5 presents the experimental results. Section 6 outlines the conclusion and discusses the limitations of the proposed method.
2 Literature Review and Related Work
Vector-phase analysis (VPA) displays the trajectory formed as a result of oxy-hemoglobin (ΔHbO) and deoxy-hemoglobin (ΔHbR) concentration changes [32]. Magnitude and angle, calculated using ΔHbO and ΔHbR, are used to construct a two-dimensional vector plane [33]. This plane is divided into eight phases for the classification of hemodynamic response [34]. A threshold circle is plotted on the vector plane to detect brain activity [35]. This method has already been used for neuronal activation detection [32], initial dip detection in hemodynamic response [32,36–38], reduction of delay in initial dip detection [35], oxygen level detection in prefrontal cortex [39] and determining the brain region of interest for BCI [40]. VPA, with dual-threshold circles, has been used for early hemodynamic response detection using EEG. The second threshold circle has been drawn using ΔHbO and ΔHbR magnitudes during the time span when a noticeable EEG activity has been sensed. During this time window, highest EEG power has been used as a criterion to select the corresponding ∆HbO and ∆HbR magnitudes, which are then further used to determine the magnitude of the second circle. The accuracy reported with this technique is 86% [41].
Different researches have been carried out on the dataset selected for this research, and efforts have been made to improve the classification accuracy [31]. Studies have been done on this dataset with the implementation of machine learning techniques and deep learning algorithms. Event-related potential (ERP) analysis has been conducted on this dataset in 2018, for which the average accuracy turned out to be 76.5% ± 8%) [31]. SVM and CNN were applied for 7 out of 26 subjects in 2020 to classify mental workload. The average classification accuracy using SVM is 70.64%, and that for CNN is 83.42% [42]. Then in a comprehensive study of 2020, SVM and CNN were applied to hybrid EEG-fNIRS dataset of the n-back task for 26 subjects, resulting in average classification accuracy of 82% and 89%, respectively [43]. In another research of 2020, DNN was employed for 26 subjects using a hybrid dataset and for the n-back task, an average accuracy of 87% was achieved [5]. There is still room for improvement in the classification accuracy for which we have devised a novel methodology.
In this paper, we propose a novel modified multimodal VPA methodology to detect activity in hemodynamic response. We have used state-of-the-art hybrid BCI (EEG-fNIRS) data for the n-back test for the presented methodology. Complete data has been preprocessed using conventional ways to make it noise-free. Initially, both the modalities have been dealt with individually. Hilbert transform has been applied to EEG signals to get the required magnitude and phase values to construct polar plots of all the trials. Then, activity is detected using these polar plots.
Similarly, VPA has been applied to fNIRS signals to construct vector-based phase plot for hemodynamic response detection with a resting state threshold circle as a detection criterion. Finally, an integrated multimodal VPA has been designed with multiple threshold circles, based on the activity completion of each EEG signal trial, to achieve better detection of hemodynamic response. This proposed design yield 91.35% average classification accuracy, which is significantly higher than other techniques mentioned previously.
An open-source dataset has been used for this research (http://doc.ml.tu-berlin.de/simultaneous https://_EEG_NIRS/) [31]. Data has been collected at Technische Universität Berlin. Twenty-six subjects, with an average age of almost 26.1 ± 3.5 years, who participated in this data collection were healthy right-handed people. Nine of them were males, and 17 were females [31]. None of them possessed any mental, neuronal, or brain-related disorder. Written consent was taken from all the subjects after informing them about the complete experimental paradigm [31].
Each participant was provided with a comfortable armchair to sit in front of a 24’ LCD screen. Distance between the person’s eyes and the screen was 1.2 m. The right armrest had numeric keypad buttons (number 7 and 8) fixed to it. All participants were directed to look at the screen and try to abstain from moving their body. This experiment comprised of three types of tasks (n-back tasks, discrimination/selection response tasks and word generation tasks) with three sessions each. For this study, we have used the n-back task dataset [31].
This dataset of the n-back test was comprised of three sessions for every subject as shown in Fig. 1, where every session had nine series of three types, i.e., 0-, 2- and 3-back tasks, in a counterbalanced order (i.e., 0 → 2 → 3 → 2 → 3 → 0 → 3 → 0 → 2). Every series consisted of 2 s instruction time, displaying the type of series (0-, 2- and 3-back), followed by a 40 s task period, 1 s time for “STOP” word and a 20 s rest period. Hence, each series was composed of total of 63 s [31].
Figure 1: Experimental paradigm for n-back task
A short beep of 250 ms was used to signify the person about the beginning and end of every task period. A fixation cross was shown on the screen for the rest period. Every task period consisted of twenty trials, each of 2 s. In every trial, a random one-digit number was displayed on the screen for 0.5 s, followed by a fixation cross for 1.5 s. For the 0-back test, participants pressed either the number 7 button for a ‘target’ digit or number 8 button for a ‘non-target’ digit. In the case of 2- and 3-back tasks, participants were instructed to press the ‘target’ button, number 7, if presently shown digit matched the 2 or 3 preceding numbers respectively, otherwise the ‘non-target’ button 8. For each type of n-back task, a total of 180 trials were carried out (3 session
Total three sessions were conducted with 9 series each, and every series was comprised of initial 2 s of instruction about the type of task (0-, 2- and 3-back), the 40 s of the task, 1 s of “STOP” word shown and 20 s of rest. The task period had 20 trials in it, and each trial of a total of 2 s consisted of 0.5 s of digit display and 1.5 s of fixation cross display [31].
3.4 Data Acquisition and Channel Configuration
fNIRS and EEG signals were recorded in parallel. EEG data were acquired at the sampling frequency of 200 Hz using a multichannel BrainAmp EEG amplifier (Brain Products GmbH, Gilching, Germany). According to the international 10–5 system, thirty electrodes were attached to a flexible fabric cap (EASYCAP GmbH, Herrsching am Ammersee, Germany) as shown in Fig. 2 (Fp1, Fp2, AFF5h, AFF6h, AFz, F1, F2, FC1, FC2, FC5, FC6, Cz, C3, C4, T7, T8, CP1, CP2, CP5, CP6, Pz, P3, P4, P7, P8, POz, O1, O2, TP9 (reference) and TP10 (ground)). Electrooculogram (EOG) was also measured using an EEG amplifier. EOG was also recorded at the same sampling frequency as EEG, with the help of two vertical and two horizontal electrodes [31]. Out of all these channels, seven frontal channels (Fp1, Fp2, AFF5h, AFF6h, AFz, F1, F2) were used for this study. Frontal channels have been chosen because n-back task is a cognitive task, and its activity signals are expected to appear in the frontal cortex. The frontal region has been indicated in Fig. 2.
Figure 2: Channel configuration of EEG and fNIRS. Blue circles denote the EEG channels, whereas red circles denote the fNIRS channels [31]. The frontal region (4 cm
fNIRS data was recorded at the sampling frequency of 10Hz with a NIRScout (NIRx Medizintechnik GmbH, Berlin, Germany). Sixteen sources and sixteen detectors were attached at frontal (sixteen channels around AFz, AF3, AF4, AF7 and AF8), parietal (four channels each around P3 and P4), motor (four channels each around C3 and C4), and occipital (four channels around POz) areas. An adjoining source-detector pair sets up an fNIRS channel. A configuration of a total of 36 channels was formed. The fNIRS channels were configured according to the international 10–5 system around AFpz, AFp3, AFp4, AFp7, AFp8, AF1, AF2, AF7, AF8, AF5h, AF6h, AFFz, AFF3h, AFF4h, AFF5, AFF6, FCC3, FCC4, C3h, C4h, C5h, C6h, CCP3, CCP4, CPP3, CPP4, P3h, P4h, P5h, P6h, PPOz, PPO3, PPO4, PO1, PO2, and POOz as shown in Fig. 2. For this configuration, the distance between source and detector was set to 30 mm for every channel. Fig. 2 displays the frontal channels’ configuration for EEG and fNIRS [31]. Blue circles are indicating the EEG channel locations, whereas red circles are denoting the fNIRS channels.
Twelve frontal channels (i.e., AF1, AF2, AFF5, AFF6, AFFz, AFpz, AFp3, AFp4, AF5h, AF6h, AFF3h, AFF4h) were used for this study based on the activity signal appearance in their hemodynamic response, as can be seen clearly from their VPA diagrams for all types of tasks (0-, 2- and 3-back) in Fig. 3.
Figure 3: VPA Plots for fNIRS frontal channels for all 3 tasks (0-, 2- and 3-back tasks). The selected channel are the ones encaptured by dotted line boxes. Channels 1 (AF7), 2 (AFF5) and 3 (AFp7) are shown in the figure. Remaining channels are shown in the figure provided in supplementary material
Before using the data for any technique, we have preprocessed the signals to get the best possible results. For EEG data, since the fundamental frequencies for this data were lying in the alpha (α) (8–13 Hz) and theta (θ) (4–8 Hz) bands, so, to remove the noise and to remain within the interested frequency bands, a bandpass filter (5th order, Butterworth filter) of 0.1–15 Hz has been applied to these signals to achieve the optimum outcomes. fNIRS data was already in the form of ∆HbO and ∆HbR concentration changes, so there was no need to apply Beer–Lambert law. A low pass filter with cut off frequency 0.2 Hz (6th order, Butterworth filter) followed by a high pass filter with cutoff frequency 0.01 Hz has been applied to the data to achieve signal within the frequency range of 0.01–0.2 Hz as the fundamental frequencies for this data were present in this band [31]. The intention behind applying these filters to fNIRS data was also to remove the instrumental and physiological noise present in the data.
4 Proposed Integrated Vector Phase Analysis
In this research, Hilbert transform (HT) has been used to calculate the imaginary component of EEG signals along with their phases and magnitudes. Polar plot construction for each trial of each series of all EEG signals is then achieved to detect activity.
For an EEG signal x(t), the imaginary component y(t) can be calculated using HT [44] as follows:
Then the analytical signal corresponding to x(t) can be written as:
where x(t) and y(t) are complex conjugates of each other and the magnitude |z(t)| and phase
and
Outcomes of HT are used to construct the polar plots of EEG signal trials to indicate activity. Mean values are calculated, for both (x and y) coordinates using the complete trajectory in the phase plot, as meanx and meany respectively. In this research, we have set the criterion that if meanx is greater than 0, it is considered the occurrence of activity (more explanation in the next section with results).
The vector-phase analysis is a technique that can be used to detect the hemodynamic response by using just the two components, ∆HbO and ∆HbR, of fNIRS signals [14] In this method, there is a vector plane which is based on two orthogonal axes with ∆HbO values at x-axis and ∆HbR values at the y-axis. This plane is divided into eight phases [37,39,45,46] by getting two more axes in the plane. When the ∆HbO and ∆HbR plane is rotated counterclockwise by 45°, the other two axes, i.e., ∆HBT (total hemoglobin) and ∆COE (cerebral oxygen exchange), come into existence. ∆HBT and ∆COE can be defined as:
The magnitude and phase of a vector v = (∆HbO, ∆HbR) expressed in this vector plane can be calculated as follows:
The eight phases of this vector plane are as shown in Fig. 4.
Figure 4: Vector-phase plot configuration displaying 8 phases. The Black dotted circle is the threshold circle for the detection of activity [47]
A threshold circle is drawn based on the maximum value of the rest period in a signal. If the trajectory of ∆HbO and ∆HbR crosses this threshold circle, this indicates activity. Magnitude values less than this threshold circle are considered as resting-state [35,38,34]. Initial dip and hemodynamic response can be detected in these eight phases. Phase (1–5) are there for initial dip detection whereas, phase (6–8) are there for the detection of the hemodynamic signal [38].
4.3 Proposed Modified Multimodal (EEG-fNIRS) Vector-based Phase Analysis:
In this study, a modified form of vector-based phase analysis has been proposed. For this study, we have used just ∆HbO and ∆HbR to keep it simple. According to our proposed method, the threshold circle has been drawn for task detection at the mean value of the resting period based on the reason that if an activity occurs, then its magnitude should exceed this mean value at least. This can be claimed based on the ideal ∆HbO and ∆HbR signals, as shown in Fig. 5. As shown in Fig. 5, when the activity starts to occur, the value of HbO increases rapidly, making the overall magnitude considerably more significant than the mean value of baseline. This mean value threshold circle can be used to detect the detection ofto detect presence of activity in a series, as shown in Fig. 3.
Figure 5: Ideal HbO/HbR signals constructed using two gamma function
So, the radius of the threshold circle can be calculated as
A vector-phase diagram based on both EEG and fNIRS activity detection has been proposed in this design. So, for that purpose, we draw a circle for each trial activity completion in EEG signal. As the activity can be detected earlier in EEG signal than fNIRS signal [41], it has been deduced that if we draw a circle for the detected activity completion in each trial of EEG signal, then ∆HbO and ∆HbR trajectory is expected to cross that circle, if the activity is also detected in fNIRS signal. It has been claimed that if fNIRS signal trajectory crosses the EEG-based circle of a trial, then the activity is considered as detected in hemodynamic response for that trial, too. There are 20 trials in each series of EEG signal, as can be seen in Fig. 1. EEG-based circle, for each trial, is drawn in a way that when ith trial activity (i = 1, 2, 3,…, 20) is completed at time ti, then values of ∆HbO and ∆HbR at ti are used to calculate the magnitude |pi| of the circle. So, circle magnitude |pi| for ith trial can be calculated as
Figure 6: Flow chart for adopted methodology
Figure 7: The ideal trajectory for modified VPA. (a) Ideal cHRF convolved with 20 impulses to form dHRF depicting 20 trials. (b) Ideal trajectory of HbO and HbR for modified VPA crossing all 20 circles one by one
If any trail is detected through phase plot of EEG signal or modified VPA, then it is considered the presence of activity. The flowchart for the proposed methodology is shown in Fig. 6. The proposed scheme can be depicted using ideal signals for both EEG and fNIRS, as shown in Fig. 7.
4.4 Ideal Trajectory for Proposed Modified Multimodal VPA:
For this novel technique, we have used two gamma function to construct the ideal trajectory of ∆HbO and ∆HbR [41] as shown in Fig. 7. Convolution of a canonical hemodynamic response function (cHRF) (i.e., H(k)) with the stimulus S(k) is called designed hemodynamic response function (dHRF). The cHRF is constructed using the linear combination of two gamma variant functions as follows:
where α1 represents the amplitude, τi and ϕi (i = 1, 2) are for the tuning of shape and scale, respectively and α2 represents the ratio of response to undershoot.
The dHRF can be mathematically stated as follows:
where S(k) is an impulse stimulus for each trial indicating rest and activity as
There were 20 trials for each series for this experiment, so we have convolved 20 impulses with ideal cHRF. Then it has been used to construct the modified VPA as mentioned in a previous section. This approach for ideal trajectory has been depicted in Fig. 7.
5.1 Hilbert Transform for Activity Detection in EEG Signals:
In this novel methodology, data from selected channels of subject one was initially filtered to retain signals only in the 0.1–15 Hz frequency range. Then data from all channels were averaged out to construct one average signal. After that, HT is used first to calculate the imaginary component of the average signal using Eq. (1), as shown in Fig. 8.
Figure 8: HT (Eq. (1)) used to construct the imaginary component of the average EEG signal
Figure 9: (a) Twenty trials for series 1 of session 1 for subject 1 (b) Average activity signal of 20 trials (c) Phase plot of average activity signal (d) Simultaneous phase plot for rest and activity
For the 1st series of session 1, which is a 3-back task, 20 trials were averaged out, and the activity portion was detected, as shown in Fig. 9. Then phase plot for the average activity signal was constructed and compared with the phase plot of the rest signal. It can be seen from the simultaneous phase plot of activity and rest, in Fig. 9, that the activity is contained in the right side of the plane, indicating the x-coordinate of its centre value as greater than 0. This proves our claim for the criterion of activity detection in EEG signal that the mean should be greater than 0 (as mentioned in the previous section).
Next, we have implemented the same scheme for all the trials of series 1 as shown in Fig. 10. Here too, we can see that the trajectories for all trials are contained on the right side of the plane with meanx > 0, indicating the presence of activity.
Figure 10: (a) Selection of activity portion in 5 trials for 1st series of 1st session for subject 1 is shown here. The remaining trials can be seen in supplementary material (b) Phase plots for five trials using HT
5.2 Modified VPA for Hemodynamic Response Detection
For every subject, fNIRS signals of all selected channels are preprocessed and then averaged to get an average signal. As mentioned in the previous section, the conventional VPA plots were constructed, for series 1 of 1st session for Subject 1, with threshold circle having radius r, calculated using Eq. (9), at the mean value of resting-state as depicted in Fig. 11a. After that EEG-based circles were constructed for 20 trials with radii calculated using Eq. (10). As shown in Fig. 11b, activity is indicated when the ∆HbO and ∆HbR trajectory crosses that trial circle. When the color of the trajectory turns green from red, it indicates that its magnitude is lesser than |pi|, whereas when the trajectory color turns red from green, it shows that its magnitude is greater than |pi|, indicating the presence of activity. If the activity is either detected in the EEG phase plot or in hemodynamic response, it is considered the activity.
For depicting the channels’ activation, brain maps have been constructed. For this purpose, five brain maps of each series (0-, 2- and 3-back tasks) for two subjects have been created, as shown in Fig. 12. For VPA construction with multiple circles, EEG channel closest to each fNIRS channel has been selected. As we are working on the frontal region of the brain, so we have selected 7 frontal EEG channels. EEG channels selected corresponding to fNIRS channels are reported in Tab. 1. The difference has been calculated for the radii of 4 trials (i.e., trial no. 5, 10, 15, 20) and rest period circles, with the radius of the baseline circle individually. Using these differences, brain maps have been constructed. This method can be mathematically stated as:
Lij is the difference of trial, i circle radius with baseline circle radius for each channel j. For now, we have taken i = 5, 10, 15,&, 20 for constructing four maps, and the 5th map is constructed based on the difference of rest period circle radius with baseline circle radius as stated below:
Five maps are constructed for all three types of series (0-, 2- and 3-back tasks), as shown in Fig. 12. The presence of red color shows the highest level of activation at a brain region.
Figure 11: (a) ∆HbO and ∆HbR trajectory for 1st series of 1st session for subject one signal with threshold circle at mean of resting state. (b) EEG-based circles for 20 trials are drawn. Trajectory color turning green from red indicates its magnitude lesser than |pi|, whereas trajectory color turning red from green shows that its magnitude is greater than |pi|, indicating the detection of activity in hemodynamic response
Figure 12: 5 Brain Maps of 0-back test for subject 1. Remaining brain maps are provided in the supplementary material
5.4 Average Classification Accuracy
After using this novel classifier for all the 3 sessions for all subjects, the classification accuracy for every series has been calculated. For each subject’s average signal, all types of tasks (0-, 2- and 3-back) were performed nine times each. The overall accuracy for this novel classifier, i.e., 91.35%, is reported and compared with the average classification accuracies for SVM, CNN and DNN in Tab. 2 [5,43].
Using this novel methodology, we have achieved a relatively higher average classification accuracy than other reported techniques used for this dataset and VPA with dual-threshold circles. As it can be seen from Fig. 13, the accuracy of our classifier, i.e., 91.35%, surpassed the average accuracies of VPA with dual circles [41], SVM, CNN, DNN [5,42,43] and ERP analysis [31] that are 86%, 82%, 89%, 87% and 76% respectively. The average command generation time calculated for this dataset is 2 ± 0.1 s.
Many researches have been carried out up till now to improve the classification accuracy using hybrid BCI [30,48,49]. We have used an open-source simultaneous EEG-fNIRS dataset integrated at Technische Universität Berlin [31]. n-back data for EEG and fNIRS has been used to design our novel classifier. Work had been done on this dataset previously to enhance the performance accuracy. Techniques such as SVM, DNN and CNN had been implemented on n-back data, and their accuracies were reported to be 82%, 87% and 89%, respectively [5,42,43]. VPA has been used for designing our classifier but in a modified form. An approach using VPA has already been implemented using dual-threshold circles, where the first circle was the conventional resting-state threshold circle, and the second circle was an EEG-based circle drawn at the highest power of the EEG activity window. Classification accuracy using this technique was reported to be 86% [41]. Intending to further improve the classification accuracy of the dataset used, we have proposed a design where modified multimodal VPA with multiple EEG-based circles has been implemented. Average command generation time has also been calculated for this dataset which came out to be 2 ± 0.1 s. To the best of the authors’ knowledge, this novel classifier has achieved a relatively higher average classification accuracy, i.e., 91.35%, as reported in Fig. 13.
Figure 13: The bar chart displays the comparison of average classification accuracy of different techniques used for the multimodal data set [31] and VPA with dual-threshold circles
One of the advantages of this proposed classifier is that it uses VPA to channel fNIRS signals. After rejecting the inactive channels, we are averaging the selected channels’ signals for each subject. Therefore, inactive channels do not reduce signal activation, hence improving the performance, making it more accurate to detect the activity in hemodynamic response.
Another advantage of this methodology is that it uses HT in a different way to construct phase plots of EEG signal trials to indicate the occurrence of activity, which is an easy and feasible method. Detection of activity in EEG separately further enhances the performance of our classifier by increasing the average classification accuracy.
Another benefit of this classifier is that it does not require any training like other conventional machine learning and deep learning classifiers because it is a trajectory-based approach with EEG trials-based multiple circles.
For this research, a considerably larger dataset [31] of 26 people has been used to design this classifier compared to the dataset of 3 people used for VPA with dual threshold circles [41]. This further strengthens the validation of the average classification accuracy achieved using our classifier.
In this study, channel activation has also been highlighted using brain maps constructed in a relatively different way than other conventional ways like t-score [41] and z-score [50]. Trial wise brain maps have been constructed to show the presence of activity in different brain regions at different stages. The brain maps are constructed based on the difference of magnitudes of different trials’ circles with the magnitude of baseline circle in the vector-phase diagram.
A limitation in this research is that activity in a time span is considered as detected if its occurrence is indicated in either EEG signal or multimodal VPA trajectory. A false positive detection can result in some false detection of activity. To further improve the classifier, research can be carried out to overcome this shortcoming. In our proposed methodology, simple preprocessing techniques have been used, such as low pass, bandpass and high pass filters. The presence of artifacts is still possible in the signals and can affect the resting state circle of the vector-phase diagram. So, to further improve the performance of this technique, advanced preprocessing techniques and artifact rejection algorithms are desirable. Moreover, in this research, a comparison between gender-based accuracy has not been conducted, so this investigation can also be carried out to indicate whether the accuracy gets affected by gender or not.
In this study, a novel methodology has been proposed for enhancing average classification accuracy using hybrid BCI (EEG-fNIRS). For this research, a hybrid (EEG-fNIRS) dataset for n-back tasks, collected at Technische Universität Berlin was used. Hilbert transform was used to construct phase plots for activity detection in EEG trials. A modified multimodal VPA was designed with multiple threshold circles, drawn at the completion time of each trial activity in EEG signals, using ∆HbO and ∆HbR magnitudes. If the ∆HbO and ∆HbR trajectory crossed the EEG-activity-based threshold circle in the time span of each trial, then activity was considered as detected. Thus, a modified multimodal (EEG-fNIRS) VPA was used as a classifier to get the combined accuracy to detect activity. The collective accuracy achieved using this novel classifier was 91.35%, relatively higher than other conventional classifiers, i.e., SVM, CNN and DNN. The average command generation time computed using this technique was 2 ± 0.1 s. This research provides a step forward in improving the classification accuracy of state-of-the-art hybrid EEG-fNIRS BCI systems.
Acknowledgement: This research work was supported by National University of Sciences and Technology, Pakistan.
Funding Statement: National University of Sciences and Technology supported the research.
Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.
1. H. Ghonchi, M. Fateh, V. Abolghasemi, S. Ferdowsi and M. Rezvani, “Deep recurrent-convolutional neural network for classification of simultaneous EEG-fNIRS signals,” IET Signal Processing, vol. 14, no. 3, pp. 142–153, 2020. [Google Scholar]
2. F. Riaz, A. Hassan, S. Rehman, I. Niazi, M. Jochumsen et al., “Processing movement related cortical potentials in EEG signals for identification of slow and fast movements,” in 2014 36th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society, Chicago, Illinois, USA, IEEE, pp. 4908–4911, 2014. [Google Scholar]
3. A. J. Karran, T. Demazure, P. M. Leger, E. Labonte-LeMoyne, S. Senecal et al., “Toward a hybrid passive BCI for the modulation of sustained attention using EEG and fNIRS,” Frontiers in Human Neuroscience, vol. 13, pp. 393, 2019. [Google Scholar]
4. L. F. Nicolas-Alonso and J. Gomez-Gil, “Brain computer interfaces, a review,” Sensors, vol. 12, no. 2, pp. 1211–1279, 2012. [Google Scholar]
5. M. Saadati, J. Nelson and H. Ayaz, “Multimodal fNIRS-EEG classification using deep learning algorithms for brain-computer interfaces purposes,” in Int. Conf. on Applied Human Factors and Ergonomics, Cham, Springer, pp. 209–220, 2020. [Google Scholar]
6. K. S. Hong and M. J. Khan, “Hybrid brain-computer interface techniques for improved classification accuracy and increased number of commands: A review,” Frontiers in Neurorobotics, vol. 11, pp. 35, 2017. [Google Scholar]
7. N. Naseer and K. S. Hong, “fNIRS-based brain-computer interfaces: A review,” Frontiers in Human Neuroscience, vol. 9, pp. 1–15, 2015. [Google Scholar]
8. M. Hasan, M. U. Khan and D. Mishra, “A computationally efficient method for hybrid EEG-fNIRS BCI based on the Pearson correlation,” BioMed Research International, vol. 2020, no. 3, pp. 1–13, 2020. [Google Scholar]
9. B. Allison, T. Luth, D. Valbuena, A. Teymourian, I. Volosyak et al., “BCI demographics: How many (and what kinds of) people can use an SSVEP BCI?,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 18, no. 2, pp. 107–116, 2010. [Google Scholar]
10. J. Kwon, J. Shin and C. H. Im, “Toward a compact hybrid brain-computer interface (BCIPerformance evaluation of multi-class hybrid EEG-fNIRS BCIs with limited number of channels,” PLOS ONE, vol. 15, no. 3, pp. e0230491, 2020. [Google Scholar]
11. P. Verma, A. Heilinger, P. Reitner, J. Grünwald, C. Guger et al., “Performance investigation of brain-computer interfaces that combine EEG and fNIRS for motor imagery tasks,” in IEEE Int. Conf. on Systems, Man and Cybernetics, Bari, Italy, IEEE, pp. 259–263, 2019. [Google Scholar]
12. M. S. Al-Quraishi, I. Elamvazuthi, S. A. Daud, S. Parasuraman and A. Borboni, “EEG-based control for upper and lower limb exoskeletons and prostheses: A systematic review,” Sensors, vol. 18, no. 10, pp. 3342, 2018. [Google Scholar]
13. S. Al Kork, T. Beyrouthy, A. Kork, J. A. Korbane and M. Abouelela, “EEG mind controlled smart prosthetic arm—A comprehensive study,” in 2016 IEEE Int. Conf. on Emerging Technologies and Innovative Business Practices for the Transformation of Societies, Mauritius, IEEE, pp. 404–409, 2016. [Google Scholar]
14. R. A. Khan, N. Naseer, N. K. Qureshi, F. M. Noori, H. Nazeer et al., “FNIRS-based neurorobotic interface for gait rehabilitation,” Journal of Neuroengineering and Rehabilation, vol. 15, no. 1, pp. 1–17, 2018. [Google Scholar]
15. N. Naseer and K. S. Hong, “Classification of functional near-infrared spectroscopy signals corresponding to the right- and left-wrist motor imagery for development of a brain-computer,” Neuroscience Letters, vol. 553, pp. 84–89, 2013. [Google Scholar]
16. M. J. Khan, K. S. Hong, N. Naseer and M. R. Bhutta, “A hybrid EEG-fNIRS BCI: Motor imagery for EEG and mental arithmetic for FNIRS,” in 14th Int. Conf. on Control, Automation and Systems, Seoul, Korea, IEEE, pp. 275–278, 2014. [Google Scholar]
17. F. Riaz, A. Hassan, S. Rehman, I. K. Niazi and K. Dremstrup, “EMD-based temporal and spectral features for the classification of EEG signals using supervised learning,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 24, no. 1, pp. 28–35, 2015. [Google Scholar]
18. A. Chiarelli, P. Croce, A. Merla and F. Zappasodi, “Deep learning for hybrid EEG-fNIRS brain-computer interface: Application to motor imagery classification related content,” Journal of Neural Engineering, vol. 15, no. 3, pp. 36028, 2018. [Google Scholar]
19. A. P. Buccino, H. O. Keles and A. Omurtag, “Hybrid EEG-FNIRS asynchronous brain-computer interface for multiple motor tasks,” PLOS ONE, vol. 11, no. 1, pp. e0146610, 2016. [Google Scholar]
20. M. Jawad Khan, M. J. Hong and K. S. Hong, “Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface,” Frontiers in Human Neuroscience, vol. 8, pp. 244, 2014. [Google Scholar]
21. J. Shin, K. Müller, C. Schmitz, D. W. Kim and H. J. Hwang, “Evaluation of a compact hybrid brain-computer interface system,” BioMed Research International, vol. 2017, no. 1, pp. 1–11, 2017. [Google Scholar]
22. F. Robertson, T. S. Douglas and E. M. Meintjes, “Motion artifact removal for functional near infrared spectroscopy: A comparison of methods,” IEEE Transactions on Biomedical Engineering, vol. 57, no. 6, pp. 1377–1387, 2010. [Google Scholar]
23. S. Tak and J. C. Ye, “Statistical analysis of fNIRS data: A comprehensive review,” Neuroimage, vol. 85, no. 1, pp. 72–91, 2014. [Google Scholar]
24. J. Shin, D. W. Kim, K. R. Müller and H. J. Hwang, “Improvement of information transfer rates using a Hybrid EEG-NIRS brain-computer interface with a short trial length: Offline and pseudo-online analyses,” Sensors, vol. 18, pp. 1827, 2018. [Google Scholar]
25. S. Lee, Y. Shin, A. Kumar, M. Kim and H. N. Lee, “Dry electrode-based fully isolated EEG/FNIRS hybrid brain-monitoring system,” IEEE Transactions on Biomedical Engineering, vol. 66, no. 4, pp. 1055–1068, 2018. [Google Scholar]
26. G. Pfurtscheller, B. Z. Allison, C. Brunner, G. Bauernfeind, T. Solis-Escalante et al., “The hybrid BCI,” Frontiers in Neuroscience, vol. 4, pp. 42, 2010. [Google Scholar]
27. Y. Sun, H. Ayaz and A. N. Akansu, “Multimodal affective state assessment using FNIRS + EEG and spontaneous facial expression,” Brain Sciences, vol. 10, no. 2, pp. 85, 2020. [Google Scholar]
28. Y. Liu, H. Ayaz and P. A. Shewokis, “Multisubject ‘learning’ for mental workload classification using concurrent EEG, FNIRS, and physiological measures,” Frontiers in Human Neuroscience, vol. 11, pp. 389, 2017. [Google Scholar]
29. R. Li, T. Potter, W. Huang and Y. Zhang, “Enhancing performance of a hybrid EEG-FNIRS system using channel selection and early temporal features,” Frontiers in Human Neuroscience, vol. 11, pp. 462, 2017. [Google Scholar]
30. S. K. Piper, A. Krueger, S. P. Koch, J. Mehnert, C. Habermehl et al., “A wearable multi-channel fNIRS system for brain imaging in freely moving subjects,” Neuroimage, vol. 85, no. Suppl. 1, pp. 64–71, 2014. [Google Scholar]
31. J. Shin, A. Von Lühmann, D. W. Kim, J. Mehnert, H. J. Hwang et al., “Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset,” Scientific Data, vol. 5, no. 1, pp. 1–16, 2018. [Google Scholar]
32. A. Zafar and K. S. Hong, “Neuronal activation detection using vector phase analysis with dual threshold circles: A functional near-infrared spectroscopy study,” International Journal of Neural Systems, vol. 28, no. 10, pp. 1850031, 2018. [Google Scholar]
33. H. Nazeer, N. Naseer, R. A. Khan, F. M. Noori, N. K. Qureshi et al., “Enhancing classification accuracy of FNIRS-BCI using features acquired from vector-based phase analysis,” Journal of Neural Engineering, vol. 17, no. 5, pp. 56025, 2020. [Google Scholar]
34. T. Kato, Vector-based approach for the detection of initial dips using functional near-infrared spectroscopy. in Neuroimaging-Structure, Function and Mind. London, UK: IntechOpen, 2019. [Google Scholar]
35. K. S. Hong and N. Naseer, “Reduction of delay in detecting initial dips from functional near-infrared spectroscopy signals using vector-based phase analysis,” International Journal of Neural System, vol. 26, no. 3, pp. 1650012, 2016. [Google Scholar]
36. A. Zafar and K. S. Hong, “Detection and classification of three-class initial dips from prefrontal cortex,” Biomedical Optics Express, vol. 8, no. 1, pp. 367–383, 2017. [Google Scholar]
37. K. Yoshino and T. Kato, “Vector-based phase classification of initial dips during word listening using near-infrared spectroscopy,” Neuroreport, vol. 23, no. 16, pp. 947–951, 2012. [Google Scholar]
38. K. Hong and A. Zafar, “Existence of initial dip for BCI: An illusion or reality,” Frontiers in Neurorobotics, vol. 12, pp. 69, 2018. [Google Scholar]
39. M. Sano, S. Sano, N. Oka, K. Yoshino and T. Kato, “Increased oxygen load in the prefrontal cortex from mouth breathing: A vector-based near-infrared spectroscopy study,” Neuroreport, vol. 24, no. 17, pp. 935–940, 2013. [Google Scholar]
40. K. S. Hong, M. J. Khan and M. J. Hong, “Feature extraction and classification methods for hybrid fNIRS-EEG brain-computer interfaces,” Frontiers in Human Neuroscience, vol. 12, pp. 246, 2018. [Google Scholar]
41. M. J. Khan, U. Ghafoor and K. S. Hong, “Early detection of hemodynamic responses using EEG: A hybrid EEG-FNIRS study,” Frontiers in Human Neuroscience, vol. 12, pp. 479, 2018. [Google Scholar]
42. U. Asgher, K. Khalil, Y. Ayaz, R. Ahmed and M. J. Khan, “Classification of mental workload (MWL) using support vector machines (SVM) and convolutional neural networks (CNN),” in 3rd Int. Conf. on Computing, Mathematics and Engineering Technologies, Sukkur, Pakistan, IEEE, pp. 1–6, 2020. [Google Scholar]
43. M. Saadati, J. Nelson and H. Ayaz, “Convolutional neural network for hybrid FNIRS-EEG mental workload classification,” in Advances in Intelligent Systems and Computing. vol. 953. Berlin, Germany: Springer, pp. 221–232, 2020. [Google Scholar]
44. W. De Clercq, P. Lemmerling, W. Van Paesschen and S. Van Huffel, “Characterization of interictal and ictal scalp EEG signals with the Hilbert transform,” in Proc. of the 25th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No. 03CH37439Cancun, Mexico, IEEE, vol. 3, pp. 2459–2462, 2003. [Google Scholar]
45. K. Yoshino, N. Oka, K. Yamamoto, H. Takahashi and T. Kato, “Correlation of prefrontal cortical activation with changing vehicle speeds in actual driving: A vector-based functional near-infrared spectroscopy study,” Frontiers in Human Neuroscience, vol. 7, pp. 895, 2013. [Google Scholar]
46. N. Oka, K. Yoshino, K. Yamamoto, H. Takahashi, H. S. Li et al., “Greater activity in the frontal cortex on left curves: A vector-based fNIRS study of left and right curve driving,” PLOS ONE, vol. 10, no. 5, pp. e0127594, 2015. [Google Scholar]
47. H. Nazeer, N. Naseer, R. A. Khan, F. M. Noori, N. K. Qureshi et al., “Enhancing classification accuracy of FNIRS-BCI using features acquired from vector-based phase analysis,” Journal of Neural Engineering, vol. 17, no. 5, pp. 56025, 2020. [Google Scholar]
48. S. Fazli, J. Mehnert, J. Steinbrink, G. Curio and A. Villringer, “Enhanced performance by a hybrid NIRS-EEG brain computer interface,” Neuroimage, vol. 59, no. 1, pp. 519–529, 2012. [Google Scholar]
49. B. Koo, H. G. Lee, Y. Nam, H. Kang, C. S. Koh et al., “A hybrid NIRS-EEG system for self-paced brain computer interface with online motor imagery,” Journal of Neuroscience Methods, vol. 244, pp. 26–32, 2015. [Google Scholar]
50. H. Matsuda, S. Mizumura, T. Nagao, T. Ota, T. Iizuka et al., “An easy Z-score imaging system for discrimination between very early Alzheimer’s disease and controls using brain perfusion SPECT in a multicentre study,” Nuclear Medicine Communications, vol. 28, no. 3, pp. 199–205, 2021. [Google Scholar]
 | This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |