Method for Psychological Status Estimation by Gaze Location Monitoring Using Eye-Based Human- Computer Interaction

Method for psychological status estimation by gaze location monitoring using Eye-Based Human-Computer Interaction: EBHCI is proposed. Through the experiment with English book reading of e-learning content, relation between psychological status and the distance between the correct location of English sentence reading points and the corresponding location derived from EBHCI is clarified. Psych9ological status is estimated from peak alpha frequency derived from eeg signals. It is concluded that psychological status can be estimated with gaze location monitoring.


INTRODUCTION
These experiments are aimed for measuring and monitoring the student attention.The student attention monitoring will be useful for a teacher evaluates the learning process under e-learning system.In the e-learning system, the interaction between teacher and student is limited.The occurred interaction is only under the content in the e-learning system.Hence, the student monitoring is being investigated to obtain the best way to monitor and evaluate the performance of a student.
Computer key-in system by human eyes only (just by sight) is proposed [1], [2].The system allows key-in when student looks at the desired key (for a while or with blink) in the screen keyboard displayed onto computer screen.Also blink detection accuracy had to be improved [3], [4].Meanwhile, influence due to students' head pose, different cornea curvature for each student, illumination conditions, background conditions, reflected image (environmental image) on students' eyes, eyelashes affecting to pupil center detection, un-intentional blink, etc. are eliminated for gaze detection accuracy improvement [5], [6].On the other hands, the system is applied for communication aid, having meal aid, electric wheel chair control, content access aid (e-learning, e-comic, ebook), phoning aid, Internet access aid (including Web search), TV watching aid, radio listening aid, and so on [7]- [17].
The method for key-in accuracy improvement with moving screen keyboard is also proposed [18].Only thing student has to do is looking at one of the following five directions, center, top, bottom, left and right so that key-in accuracy is remarkably improved (100% perfect) and student can use the system in a relax situation.
The following section describes the proposed method for psychological status estimation by gaze location monitoring using EBHCI followed by some experiments with English book reading of e-learning content.Then conclusion is described together with some discussions.

A. System Configuration
System configuration of the proposed method is shown in Figure 1.We create a software consisting function as follow: (1) Tracking the eye, (2) Estimating the user sight, (3) Sight trajectory and blinking logger, and (4) EEG signal logger.In Figure 1, it explains the methodology of our investigation.In our methodology, user/student access/use the e-learning content using personal computer.The used e-learning content consist of teacher voice (we recorded it using recorder and storage it as voice data) and the text shown on the screen.These two content are connected each other.The saying voice has same timeline with the text, so student could follow the teacher voice (hearing using ear phone) while he read the text on screen.
The concept of our this experiment is how we could make a relation between the eye behavior to student attention, so we would measure the student attention by synchronizing between hearing voice and the student sight/gaze to text on the screen.We compare between the trajectory sight points on the text with the true one.We make assumption of the relation between student attentions and sight point as if student do not looking at the displayed text while the teacher voice speaking at, we judge it as student do not focus to the content.Otherwise, if student could follow the teacher voice by looking at on the same text, we judge it as student focus to the content.The difference between true trajectory text points and user sight trajectory point will be evaluated to determine whether the student is focus to the content or not.Also, we measure the happened blinking manually through the saved video file saved while experiment is conducted.We count the number of blinking and record it by time frame.www.ijacsa.thesai.orgSystem configuration of the proposed psychological status estimation with EBHCI We measure the brain activity of student simultaneously.We record the raw EEG data by using Neurosky mindset EEG sensor.This EEG signal will represent the brain condition of student whether student under the stress condition or not.Such stress condition could be measure by using Peak Alpha Frequency of EEG signal.

B. Hardware Configuration
Major specifications of the proposed system including the eeg sensor used are as follows,

1) Measures:
We record the EEG signal by using NeuroSky Mindset EEG sensor and OpenVibe software.By using the openvibe software, we could record the EEG signal and process it to peak alpha frequency.
• Raw-Brainwaves • EEG power spectrums (Alpha, Beta, etc.) • eSense meter for Attention and Meditation • Quality of data being collected (can be used to detect poor contact and whether the device is off the head) 2) Signal and EEG: • Maximum signal input range: 1mV pk-pk • Hardware filter range: 3Hz to 100Hz

C. Eye Tracking and Line of Sight Estimation
We use the eye tracking method utilizing adaptive threshold and pupil knowledge which is robust for various users and illumination changes.Our system utilizes IR camera mounted on glass to allow user`s movement.Pupil knowledge such as shape, size, location, and motion are used.This knowledge works based on the knowledge priority.Pupil appearance such as size, color, and shape is used as first priority.
When this step fails, then pupil is estimated based on its location as second priority.If all steps failed, then we estimate pupil based on its motion as last priority.The eye tracking block diagram is shown in Figure 2. We develop our software under C++ Visual Studio 2005 and OpenCv Image processing Library which can be downloaded as free on their website.Our system utilize infrared web camera NetCowBow DC-NCR 131 as real time data input.This camera has benefit when illumination changes.By using 7 IR LED, this camera will adjust the illumination.This will cause the input image has minimum influence against illumination changes.The camera is mounted on the glass.
Outlook of the eeg sensor is shown in Figure 3.Meanwhile, example of the raw eeg signal and its spectral data as well as frequency component is shown in Figure 4 www.ijacsa.thesai.org

D. Peak Alpha Frequency
Peak Alpha Frequency: PAF is measured using NeuroSky and OpenVibe software.Peak Alpha Frequency: PAF of eeg signals 1 represent how relax do students during learning processes with e-learning contents [19].
In order to connect NeuroSky with OpenVibe, we installed OpenVibe v.7.First, OpenVibe is connected with neurosky by using openvibe acquisition server as depicted in Figure 5.After acquisition server is opened and start the connection, the streamed EEG data were ready for acquiring.We use openvibe designer to create scenario and play it.The used scenario for PAF measurement is depicted in Figure 6 and 7.
Connect between acquisition server and designer tool As depicted in Figure 6, we acquire streamed EEG data by using Acquisition client box.This box will take data from openvibe acquisition server.After streamed EEG data were taken, we use signal display box to display the signal as depicted in Figure 7. Also we use GDF file writer to save the streamed EEG data into file.After we save the streamed EEG data, next is selection of alpha band and show its spectrum frequency.We use temporal filter box (Filter method is set to Butterworth, filter type is set to Band pass, filter order is set to 4, Low cut frequency is set to 6 Hz, High cut frequency is set to 15 Hz, and Pass band ripple is set to 0.5 dB).After selection of alpha range, next is convert to power spectrum.We use spectral analysis (FFT operator) to convert this signal into power spectrum.The output of power spectrum then is saved into CSV file by using CSV File writer box.The power spectrum display is shown in Figure 8 and streamed EEG signal is shown in Figure 9.
After we got alpha spectrum, next is determination of peak alpha.Next calculation was done by using Math Lab.We determine Peak Alpha Frequency by using command line of Math Lab and output will be saved into excel file as follow,  The average of PAF of resting condition is 9.695 Hz and reading condition is 8.902 Hz.It is make sense that according to Klimesch journal that PAF will decrease when the task demand is increase.In the experiment data, PAF decrease about 0.793 Hz.

E. Gaze Location Estimation
In Figure 12, it is shown gaze location distribution.This distribution was taken while user was looking at 6 keys serially.While user looking at these key, we recorded the gaze result and created its distribution.This figure shows that at key 5, the gaze result has more scattered than others.It was caused by the position of eye at this key little bit occluded and it made the gaze result become unstable (the instability become increase).Example of 3D distribution of gaze location estimated for the case of the aforementioned six targeted keys III.EXPERIMENTS

A. Method of the Experiment
The experiment is conducted by the following steps,

1) User sit on the prepared seat 2) User wears the glasses camera sensor 3) User uses ear phone to hear the sound instruction 4) User wear the Electroencephalograph (EEG, Neurosky mind set)
5) All the sensor above were wear simultaneously 6) Take a calibration of the sight estimation system 7) Our system play the recorded voice representing the text on the screen 8) Student has to read the text in the same time with the hear audio voice from ear phone 9) We record the trajectory of user gaze, blinking, and the EEG signal 10) We analyze the data Figure 14 shows the English book reading used for the experiment.Students read the English book by character by character, by word by word, by sentence by sentence, and by page by page.The location at which content creator would like students to look at is indicated with black circle while the location at which students is looking at is also indicated on display with green circle.Therefore, the distance between two locations can be calculated by frame by frame.In the meantime, eeg signal is acquired.Thus relation between psychological status derived from eeg signal and the distance between two locations results in estimation of psychological status with gaze movement.

B. Modification of eeg Sensor
The neurosky (eeg sensor) is replaced to the eye glasses as shown in Figure 15.The two positions of electrodes are used in the experiment to measure the difference between two points of head (one point is at center of two eyes and another one is at forehead).The main goal of this experiment is for measuring signal to noise ratio (SNR).Unfortunately, the experiment was done using two computers separately.Due to this, the recorded signals are difficult to compared directly using raw signal.Fortunately, we already know that the glasses will be used for measuring the brain load activity.Therefore, I just made comparison in output of Peak Alpha Frequency (PAF).We could judge whether two points of electrodes have similar or same output through the PAF output.If the two points of electrodes have similar output, we could see it through the shape of PAF signal.Otherwise, if the two points of electrodes are different (noise is too high), we will see the different shape of PAF signal.The output signal of PAF is shown in Figure 16.The output of PAF signals has different shape.It means that the placement of electrode at center of eyes is different with the placement of electrode at the forehead.So, my conclusion is the modification of neurosky to neuroglasses sensor has to put the sensor in the forehead (same point/location with the original neurosky sensor).

C. Results from the Experiment
We record three kind of data: (1) Blinking, (2) Sight trajectory point, and (3) EEG signal.By using these data, we would investigate the relationship/correlation between eye behaviors to user attention level.

Blinking
We count the happened blinking manually (from the saved experiment video).The blinking data by valuating the closed eye as "1" and opened eye as "0" we present the blinking data as shown in Figure 17.
In the data above, it shows that increasing the time (in frame, 4 frame per second), the frequency of blinking occurrence increases.It shows that the blinking has a linear correlation to student attention.Such situation happen caused by student become tired follows the content under long duration and he need take a rest a while through closing the eye.After taking a rest (by closing the eye), student becomes fresh again to follow the content.To analyze student attention using the sight, we calculate the error by measuring the distance between the sight trajectory points and the real point (tracked manually).It is shown in Figure 18.The time frame in Figure 18 is 8 frames per seconds.From the figure above, we can see that on the frame above to 2800, the frequency of error increase.It is caused by student was looking at different point with the true one (the true one collected manually and used it as the true reference).Also, we investigate the correlation between student attentions monitoring to EEG signal.We acquired the EEG signal while student using e-learning system and calculate the peak alpha frequency.The PAF data is shown in Figure 19.
In the figure above, it is not clearly when student focus to content or not.It may happen because a human has ability to recovery his brain performance by doing physiological action such as blinking, taking a rest for a while, etc.As shown in Figure 20, the distance between two locations is sometime significant when the student take a long time to read and to understand the words.At that time PAF and alpha frequency component derived from eeg signal is decreasing results in student feels frustration.
IV. CONCLUSION Method for psychological status estimation by gaze location monitoring using Eye-Based Human-Computer Interaction: EBHCI is proposed.Through the experiment with English book reading of e-learning content, relation between psychological status and the distance between the correct location of English sentence reading points and the corresponding location derived from EBHCI is clarified.Psych9ological status is estimated from peak alpha frequency derived from eeg signals.It is concluded that psychological status can be estimated with gaze location monitoring.
From the investigation data, we can conclude that for monitoring the student attention to e-learning content, the blinking is most dominant parameter that could be used to determine whether a student focus to content or not.The second dominant parameter is sight trajectory point although the high performance of sight estimation method has to be used to avoid the error/noise.The last dominant (less dominant) parameter is peak alpha frequency of EEG signal.

Fig. 1 .
Fig.1.System configuration of the proposed psychological status estimation with EBHCI

Fig. 2 .
Fig.2.Eye tracking and line of sight estimation

Fig. 14 .
Fig.14.Example of reading types of e-learning contents

Fig. 18 .
Fig.18.The difference of Sight trajectory points to the true one

Fig. 19 .
Fig.19.Student attention measurement using EEG Figure20shows the example of the location at which content creator would students like to look at and the location at which students are looking at one page of English reading book.

Fig. 20 .
Fig.20.example of the location at which content creator would students like to look at and the location at which students are looking at one page of English reading book