Nine States HCI using Electrooculogram and Neural Networks

—Human Computer Interface (HCI) translate biosignals to control external devices like computers wheelchairs, mouse and keyboard. This paper presents the feasibility of creating one such HCI using Electroooculography. Electrooculography is a technique of measuring the potential difference between the cornea and retina of the eye. Most of the EOG based HCI have focused on two to four and six states, this study focuses on increasing the possible states of the HCI to nine states. Two new eye movements were proposed. The proposed reference power technique was applied to extract the features from twenty subjects. Layered recurrent neural networks were used to classify the different EOG eye movement task signals. Experimental results validate the feasibility of using eleven different eye movements EOG signals for designing nine states HCI. From the result it was proved that feasibility of designing a nine states HCI is possible using reference power features with Layered Recurrent Neural Network.


II. RELATED WORKS
In the past decade, many EOG based HCI researches have been performed. Some of the related works have been explained below Ki-Hong Kim et al, designed an alternative human interface system for enabling people with severe motor disabilities to control an assistive wheelchair. Signals are acquired from four task using four electrode system. Five subjects were participated in the experiment ranging from 25 to 52 years in age. All the subjects have a severe loss of motor function caused by traffic or hiking accidents. Acquired signals are amplified by ADS8343 to extract the features using linear prediction coefficients and its entropy techniques. For estimating time-varying linear prediction coefficients parameters, a signal of 0.5 s, sampled at a rate of 512 Hz, is first divided into short segments, generally called frames, and linear prediction coefficients are then extracted from each frame. The feature vector set obtained in this way represents a time course of the linear prediction coefficients that describes the time-varying characteristics of the signal. A time of 30 to 50 ms was used as the frame length, and the overlap rate of frames was 50%. Hidden Markov Model is used to classify the four different task signals. The control signals are applied to digital signal processor kit (TMS320-C6711B) to validate and test the wheelchair. The size of this system is 12 × 11 × 5.5 cm, and the control commands decoded in the DSP, which are associated with particular directions such as left, right, forward, or backward, are transferred to the 5-switch module of the power-wheelchair via the RS232C port. The overall correct classification rate occurring in the five subjects was around 97.2% [13]. Masaki Nakanishi et al, identify voluntary eye blink detection method using electrooculogram for controlling brain computer interface using normal blink, double blink, and wink from vertical and horizontal EOG signals collected from eight subjects using five electrode systems. Acquired EOG signals are first preprocessed to extract 0.53-15Hz signals using finite impulse response filter. Preprocessed signals are applied to vertical peak amplitude, horizontal peak amplitude and maximum cross correlation techniques to extract the features. Eye blinks were classified by support vector machine technique is used to classify the feature extracted EOG signals. As the result of simulations, an average accuracy of 97.28% was obtained using this method. In addition, the best accuracy for voluntary eye blinks was obtained for Wink with an accuracy of 97.69 % and the average misclassification rate of 0.43% for REST was obtained [14].Fuming Fang et al, develop an efficient means of communication for those who cannot move muscles of the body except eyes using EOG to control speech synthesizer based on T 3 WFST decoder which supports live decoding at the end of the input. Signals are acquired for five tasks namely up, down, left, right and center from two subjects with six electrode system using six channel. Speech recognition technique based on hidden markov model was proposed to recognition the eye movements. Experimental result shows that 96.1% of recognition accuracy is obtained for five classes of eye actions by user dependent system using six channels [15].
Human Machine Interface based on EOG proposed by Kenji Yamagishi, Junichi Hori, and MichioMiyakawa, implements communication device by controlling both horizontal and vertical eye movements with three electrode using two channel systems for nine tasks namely up, down, right, left, up right, up left, down right and down left) and one selection. Acquired signals are sampled at 100 Hz and amplified with Ac coupling in order to reduce the unnecessary drift. Obtained signals are pre processed with band pass filter in the range of 0.53Hz and 5 Hz. EOG peaks are detected using upper and lower thresholds of the vertical channel V (V1 and V2) and the upper and lower thresholds of the horizontal channel H (H1 and H2). These thresholds should be determined beforehand. When the EOG potential exceeds one of these thresholds, the output assumes ON. The setup time is the time interval between the first and second peak of EOG signals. All thresholds automatically switch to OFF after intention is output. The optimum values for the thresholds and the setup time are determined by trial experiments. The thresholds and the setup time of EOG signals should be used for distinguishing the intentional signals from artifacts such as eye blink and ordinary saccade eye movements. Eight directional cursor movements and one selected operation were realized by logically combining the two detected channel signals based on threshold setting specific to the individuals. Eye movements operate cursor in eight directions and a selection is indicated by a voluntary eye blink. A quick and simplified selection can be done by cursor movements alone, without the need for automatic scanning selection. As experimental results using a projected screen keyboard, processing speed were improved to 12.1 letters/min with an accuracy of 90.4% [16]. Sherenby et al, introduced a new method for detecting the eye gaze electrical signals resulted from human eye's saccadic motion using EOG to control several applications to help handicapped persons such as eye mouse, and eye document reader. High gain differential amplifier with 1000 to 5000 gain value, high pass filter with cut off frequency 0.1 Hz, another amplifier possesses an amplification factor of 10:50, low pass filter with cut off frequency at 100 Hz and five Ag-AgCl floating metal body surface electrodes are used in this research. Signals are acquired from five subjects. Data are collected in real time from human eye movement tasks using Lab VIEW software. Acquired signals are integrated with three integration methods they are curve, trapezoidal rule and Mid-ordinate rule to work in real time. Multipoint detection technique is applied to extract the features. The features are trained and tested using neural network classifiers. The three above methods give reasonable results.
But multipoint detection techniques results are the best among the other to carry out eye computer interface application in this research. Finally four different software and hardware applications are designed to help disabled peoples [17]. Yingxi Chen and Wyatt S Newman discussed and design EOG based gaze controlled robotic arm system for human arm. EOG signal is acquired from five volunteers for fixation, saccades and blinks. In this experiment, disposable EOG electrodes are used and a pair of CED 1902 amplifier modules (high resolution, good CMRR) is used to amplify electrode signals. The amplifiers included options for gains, biases and filtering. The amplified analog EOG signals were sampled at 2 kHz and were processed to reconstruct the subject's eye gaze. Experiments showed that the gaze vector was within the angular range of ±50° horizontally and ±30° vertically, the recorded EOG signals were almost proportional to the eye gaze displacements. Additional experiments showed that cross-talk effects were stronger at higher and lower elevation angles. Incoming data is processed within a moving window of a duration threshold (100 -200 ms). Preprocessed signals are applied to dispersion threshold and duration threshold using position based scheme and velocity based scheme. Therefore, the dispersion threshold and time threshold should be adjusted and tested carefully for a specific application. Also, Amplitude of various blinks through unconscious blinks/blink reflex, normal blinks, strong blinks and intentionally strong blinks is defined in this research. Neural Network classifier is used to classify the signals. Experimental results show that position based controller outperforms the velocity based controller using dispersion method. The output from the neural network is interfaced to a Kawasaki JS-10 robot for its evaluation [18]. Wissel.T and Palaniappan, R, designed Human-Computer Interfaces based on Electrooculogram signal acquired from five subjects using five electrode systems for developing an virtual keyboard. EOG signals were sampled with 256 Hz, passed to a cascade of two digital, low-order Bessel filters acting as a bandpass with the lower cut-off frequency at 1 Hz and the higher at 5 Hz. The filter type was chosen due to its linear phase characteristic in the pass band without any ripples in the amplitude response. Acquired signals are applied to Euclidean distance, FFT and wavelet decomposition for extracting features. ANN and LDA method is used to classify the EOG signals.
The experiment was conducted for six task, namely right, left, up, down and long blink and short blink with an average accuracy of 94.90% using DWT and LDA [19]. Aungskan et al, implemented eight states HCI using two channel EOG systems to extract the eight eye movements using instrumentation amplifier (Mobi6-6b) with a sampling rate of 128 Hz, amplifier, with a gain of 19.5 and filter of 1 to 500 Hz bandwidth were set for the recording system to convert analog signal to digital signals. The EOG data were recorded from three normal subjects with 8 directional eye movements. Each movement was held for 2 s and it was performed five times throughout a trial. Fifteen data sets were obtained from each directional movement. A robust classification algorithm based on onset analysis, first derivative technique and threshold analysis were proposed by the authors. Based on the optimal threshold values and conditions, the result showed that classification accuracy reached 100% for three subjects during testing [20]. Most of the research on EOG based HCI focuses only on limited states varying from two to six. This study investigates the possibility of increasing the number of states for HCI to nine states using controllable eye movements. Sixteen eye movements are studied to find the suitable eye movements for designing the HCI.
III. ELECTROOCULOGRAPHY EOG is the technique of sensing eye movements by recording the cornea-retinal potential that exists between the front and the back of the human eye. To measure eye movements, pair of electrodes is typically placed above and below the eye and to the left and right of the eyes. The eye acts as a dipole in which the anterior pole is positive and posterior pole is negative. Human eyeball can be considered as a spherical battery with the center of the cornea as positive and the retina as negative. The micro currents flow radially from the positive pole to the negative pole of the battery through the conductive tissue in the path. These currents generate the standing potentials around the eye, and the micro potentials (EOG) can be detected from the skin electrodes pasted on the surface of the canthus. The potential of the EOG varies from 50 to 3500µV. 20µV changes are seen for changes in each degree of eye movement. [21]- [22].
Eye position and motion are controlled by six muscles in each eye. The six muscles are Medial Rectus (MR), Lateral Rectus (LR), Superior Rectus (SR), Inferior Rectus (IR), Superior Oblique (SO) and Inferior Oblique (IO). MR muscles perform the movement of moving the eye inward, toward the nose. LR moves the eye outward, away from the nose. SR muscles primarily moves the eye upward, secondarily rotates the top of the eye toward the nose, tertiarily moves the eye inward. IR muscles primarily execute the eye downward, secondarily rotate the top of the eye away from the nose, tertiarily moves the eye inward. SO primarily make rotates the top of the eye toward the nose, secondarily moves the eye downward, tertiarily moves the eye outward and IO muscles mostly rotate the top of the eye away from the nose. Secondarily moves the eye upward, tertiarily moves the eye outward. Each movement that elevates or depresses needs the participation of a minimum of 2 muscles of the axis of the orbit and also the muscles visual axis. The primary function of the four rectus muscles, namely SR, MR and IR, LR are used to control the eye movements from left to right and up and down. Top and bottom rotations are controlled by the SO and IO. These six tiny muscles that surround the eye and control its movements are known as the extra ocular muscles [22]. In this study, however Right (R), Left (L), Upright (UR), Downright (DR), Upleft (UL), Downleft (DL), Rapid Movement (RM), Lateral Movement (LM) are considered as events and Open (O), Close (C), Stare (S) is considered as a non event because most subjects have difficult in voluntarily controlling blinks for a quick duration of 80ms.
IV. MATERIALS AND METHODS Sixteen eye movements were studied and it was found that eleven eye movements could be voluntarily controlled, the remaining five eye movements were difficult to be controlled by all subjects. From the chosen eleven movements eight movements are considered as events and three are taken as non events to design a nine state HCI. The protocols for the nine states are given below: Right: Subject is requested to move both the eyes synchronously and symmetrically in the right direction to achieve this moment. LR and MR muscles are involved in this task.
Left: Subject is asked to move both the eyes synchronously and symmetrically in the left direction. MR and LR muscles are responsible for this movement.
Up Right: Subject is told to move both the eyes synchronously and symmetrically in the upper right direction to complete the task. SR and IO muscles are in charge of this movement.
Down Right: Subject is instructed to move both the eyes synchronously and symmetrically in the down right direction. IR and SO muscles are accountable for this task.
Up Left: Subject is requested to move both the eyes synchronously and symmetrically in the upside left direction. IO and SR muscles are occupied with this task.
Down Left: Subject is initiating to move both the eyes synchronously and symmetrically in the down left direction. SO and IR muscles are engaged in this movement.
Rapid Movement: Rapidly moving both the eyes from left to right and right to left are called rapid movement. The subject is requested to move both the eyes synchronously and symmetrically in the same direction quickly and repeatedly. MR and LR muscles are responsible for this task.
Lateral Movement: Lateral movement is achieved by moving both eyes slowly from left to right or vice versa. The subject is told to move both the eyes synchronously and symmetrically in the same direction slowly and repeatedly. MR and LR muscles are involved in this task.

Open: The external, visible portion of the organ called eyelids are open slowly together to focus is called open. Subject is instructed to open both the eyes slowly together. SR and IR muscles are engaged in this movement.
Close: The eyelids is closed slowly together to cut off the focus is called close. Subject is requested to close both the eyes slowly together to achieve this task.SR and IR muscles are involved in this movement.
Stare: Subject is instructed to maintain the visual gaze on a single location to complete the task. SR and IR muscles are implicated in this movement.
A preliminary study [23] on the spectral distribution of the eleven eye movements were made which revealed that the eight movements chosen as events were found to have frequency components in the range of 5-8 Hz while the non event eye movements frequency components in the range of 6-9 Hz as shown in Table 1. A) Signal Acquisition EOG signals of the eight events and three nonevents were acquired using a two channel AD Instrument Biosignal amplifier. Five gold plated, cup shaped electrodes were placed above and below the right eye and on the left eye and rightside and left side of the eye as shown in Fig.1. All subjects who participated in the experiments are university students and staff aged between 21 and 44 years and voluntarily participated in the studies. It was ensured that all subjects were healthy and free from illness during the acquisition. The subjects were asked to execute eleven different eye movement tasks while remaining in a totally inactive state. Subjects were seated in a comfortable chair in front of marked wall and requested not to make any overt movement during data acquisition. Subjects were given the eleven eye movement tasks to be executed by moving their eyes as per the protocol given for each task. The EOG signals are sampled at 100Hz [24]. During signal acquisition a notch filter was applied to remove the 50Hz power line artifacts. EOG signals evoked by all the eleven tasks stated above were recorded from twenty subjects. Each recording trial lasts for two seconds. Ten trials were recorded for each task. Subjects were given a break of five minutes between trials and data were collected in two sessions, each session has five trials per task. All trials for a single subject were conducted on the same day. For each subject, a data set consisting of 110 sets (11 tasks x 10 trials per task) of EOG signals was formulated. The data sets of all the subjects were combined and a master data set consisting of 2200 trials of EOG signals was formulated.

B) Preprocessing and Feature Extraction
A spectral distribution analysis done on the signals showed that the eight movements chosen as events were found to have frequency components in the range of 5-8 Hz while the non event eye movements frequency components in the range of 6-9 Hz more info is discussed in our previous study. The raw EOG signals have to be further processed to isolate the event and the non event frequency range. Since artifacts due to EEG, EMG of the facial muscles, position of the electrodes, head and facial moments, lighting conditions and blinks in the signal can be removed by a bandpass filter. Eight frequency bands are extracted using a Chebyshev bandpass filters by split the signal in the range of two Hz to filter the noisy data. The eight frequency ranges are (0.1-2) Hz, (2-4) Hz, (4-6) Hz, (6-8) Hz, (8-10) Hz, (10-12) Hz, (12)(13)(14) Hz, (14)(15)(16) Hz [24]. The feature extraction method proposed was the reference power technique which states that the difference between reference signal X t and task signal E x is summed, squared and performed logarithmic transform to the band power data S ∑ X t E x (1) R 20 * log S (2) Where S is the sum of the difference between two signals and R is the power density of the signal. Sixteen features are extracted for each task per trial. The features are extracted for ten such trials for each task. 110 data samples for one subject are obtained. The feature sets obtained from the above feature extraction methods are individually applied to neural networks as input features to classify the signals into eleven eye movements.

C) Signal Classification
Layered recurrent neural networks are used to classify the signals. A layered recurrent neural network (LRNN) is a dynamic network, which creates an internal state of the network that allows it to exhibit dynamic maps between input and output. When the output of a neuron is feedback into a neuron in earlier layer, the output of that neuron is a function of both inputs from previous layer that existed for one cycle of calculation. Each input units are fully connected to hidden units and output units. Unlike feed forward neural networks, LRNN can use their internal memory to process arbitrary sequences of inputs [24].
The LRNN is trained using Levenberg back propagation training algorithm because it finds a solution even if it starts very far off the final minimum. The training and testing samples are normalized between 0 to 1 using a binary normalization algorithm to fit the data within unity of 1. Out of the 110 samples 75% of the data is used in the training of the network. The network is modelled using sixteen input neurons, eight hidden neurons and four output neurons to identify the events and non event eye movements. The learning rate is chosen as 0.0001.
Training is conducted until the average error falls below 0.001 or reaches maximum iteration limit of 1000 and testing error tolerance is fixed at 0.1 [25]- [26]. Fig.2. Shows the architecture of the LRNN used in this study.

V. EXPERIMENTAL RESULTS
The classification performance of the LRNN network model using reference power features for the eleven eye movement tasks is shown in Table 2. Table 2 comprises of mean testing time, training time, maximum, minimum, mean classification accuracy and standard deviation. From the Table 2, it is observed that maximum classification accuracy of 94.55% for reference power features and minimum classification accuracy of 84.55% was obtained. The result shows that the maximum mean classification accuracy of 94.12% for reference power features and the minimum mean classification accuracy of 86.08% were achieved. The highest mean classification rate was observed for S9 of 91.78% and the mean average performance of the LRNN is 90.74% using reference power features. The mean training time and testing time for the network was varied from 46.46 to 46.80 seconds and 0.58 to 0.79 seconds and also standard deviation varies from 1.76 to 2.49.From Table 2, it is observed that the performance of the LRNN network model using reference power features is comparatively better in terms of classification accuracy. The classification accuracy for reference power features using LRNN is shown in fig.2. Second highest mean classification accuracy was obtained for subject4 of 91.00% and minimum classification accuracy of 90.27% was achieved for subject5 using same features. From the result it can be concluded that reference power features with LRNN network model is more suitable for identifying and recognizing EOG signals for eleven different eye movement tasks.
Offline analysis of the signals for recognition of events and non events were conducted to assess the performance of the nine states HCI. The performance of the nine state HCI system designed for each subject is verified through a single trail analysis using LRNN for reference power features are shown in Table3. From the  Table 3, it is observed that for subject 8 the acceptance rate was high at a mean of 90% for events and 85% for non events using reference power and subject5 the acceptance rate was low at a mean of 70% for events and 75% for non events using same features. From the single trail analysis, it is evident that eighty percentages of the signals have a recognition rate of eight and above for some subjects like S2, S3, S4, S5, S15 and S16 the recognition rates were not appreciable from the analysis we were able to conclude that more training data is required to improve the recognition accuracy of the events. From the result it is observed that feasibility of designing a nine state HCI is possible for some subjects using reference power features for LRNN, while some of the subjects like S2, S3, S4, S5, S15 and S16 the mean accuracy of nine states HCI was around 80% only so it requires more training data. However further training of the subjects could provide improved classification performance.
From Table 3, it is observed that the performance of the LRNN network model using reference power features is comparatively better. From the result it can be concluded that reference power features with LRNN network model are more suitable for identifying and recognizing EOG signals for eleven different eye movement tasks.

VI. CONCLUSION
In this paper a nine state HCI system design using EOG signals is proposed. EOG signal data for eight events and three non events are collected from 20 subjects. Events and non events are assigned based on the voluntary and non voluntary eye movements possible by all subjects. Two new eye movements were proposed as events, namely the rapid movement and the lateral movement in addition to the six movements used in the literature. Signal features are extracted using an algorithm to determine the covariance between the movement signal and reference signal. Eye blinking is used as the reference signal. A layered recurrent neural network is used to identify the events and non events which recorded a classification performance of 91.78%. Offline analysis of individual signals revealed the feasibility of designing the nine state HCI using EOG signals; however verification of the system with real time experimentation id required which will be the focus of our future research. It is also observed that more training is required for some subjects on movements like rapid and lateral movements.