Relations between Psychological Status and Eye Movements

—Relations between psychological status and eye movements are found through experiments with readings of different types of documents as well as playing games. Psychological status can be monitored with Electroencephalogram: EEG sensor while eye movements can be monitored with Near Infrared: NIR cameras with NIR Light Emission Diode: LED. EEG signals are suffred from noises while eye movement can be acquired without any influence from nise. Therefore, psychlogical status can be monitored with eye movement detection instead of EEG signal acquisition if there is relation between both. Through the experiments, it is found strong relation between both. In particular, relation between the number of rapid changes of line of sight directions and relatively high frequency components of EEG signals is found. It is also found that the number of rapid eye movement is counted when the users are reading the documents. The rapid eye movement is defined as 10 degrees of look angle difference for one second. Not only when the users change the lines in the document, but also when the users feel a difficulty for reading words in the document, the users' line of sight direction moves rapidly.


INTRODUCTION
Nowadays, the eye-based HCI has been widely used to assist not only handicap person but also for a normal person.In handicap person, especially paraplegic, they use eye-based HCI for helping them to self-sufficient in the daily life such as input text to compute [1], communication aids [2], controlling wheelchair [3], [4] , fetch a meal on table using robot arm [5], etc.The eye key-in system has been developed by many researchers [1].The commercial available system provided by Tobii Tracker Company has been used by many researchers for developing text input, customer interest estimator on the business market, etc. [6].Technology has been successful in rehabilitating paraplegics` personal lives.Prof Stephen Hawking, who was diagnosed with Athe proposedotrophic Lateral Sclerosis (ALS), uses an electronic voice synthesizer to help him communicate with others [7].By typing the text through aid of a predictive text entry system, approximating his voice, he is able to make coherent speech and present at conferences.To give another example, a paraplegic patient wearing a head-mounted camera is able to draw figures, lines, and play computer games [1].Clearly, through the use of assistive technology, handicapped people are able to do feats on par with non-handicapped people.
The published papers discussing eye-based HCI system are categorized into: (1) vision-based and (2) bio-potential-based.The vision-based method utilized camera to capture an image and estimate the user sight.The key issue here is how the method/system could be deal with environment changing.Lighting changing, user movement, various types of user, etc have to cooperate with the system.The vision-based system could be explained as follow.
Eye mouse based on sight is developed [8].The method searches and tracks the eye by using projection of difference left-right eye after it is success to detect the face.It obtains left and right direction that could be used for controlling mouse pointer without upward and downward directions.They implemented the method for controlling application such as "Block Escape" game and spelling program.Eye mouse by utilizing sight obtained from pupil location and detected by using Haar Classifier (OpenCv function) is developed [9].It used blinking for replacing left click of mouse event.The system enabling it to control mouse by utilizing face movement and blinking is developed [10].The Adaboost face detection method detects center position of face and tracks it by using Lucas-Kanade Optical flow.The user used the resulted location for controlling mouse cursor and will execute left mouse click by detecting the blinking.Human-computer interface by integrating eye and head position monitoring devices is developed [11].It controlled the interface based on sight and blinking.The user command could be translated by system via sight and blinking.Also, they modified calibration method for reducing visual angle between the center of target and the intersection point (derived by sight).They reported that the modification allowed 108 or higher number of command blocks to be displayed on 14 inch monitor.Also, they reported that it has hit rate of 98% when viewed at the distance of 500mm.For triggering, the blinking invokes commands.
The bio-potential-based method estimated user behavior (eye behavior) by measuring user's bio-potential.The biopotential measurement instrument such as Electrooculograph (EOG), Electroencephalograph (EEG), etc could be used for measuring eye behaviors.The example of bio-potential-based has been applied in application of electric wheelchair controlled using EOG analyzed user eye movement via electrodes directly on the eye to obtain horizontal and vertical eye-muscle activity.www.ijarai.thesai.orgSignal recognition analyzed Omni directional eye movement patterns [12].I search pupil location on an eye image by using the proposed method that has been published [13].I estimate the sight by converting the obtained position of pupil to sight angle.After the sight angle is estimated, I control the mouse cursor by using this sight.
In the following section, eye movement detection method is described followed by psychological status monitoring with EEG sensor.Then relations between eye movement and psychological status are described followed by conclusion with discussions.

A. Hardware Configuration
The system utilizes Infrared Camera NetCowBoy DC NCR-131 to acquire user image.I modified the position of 8 IR LED for adjusting illumination and obtaining stable image even illumination of environment changes as shown in Fig. 1.The use of IR camera will solve problem of illumination changes.I put the camera on user's glasses.Distance between camera and eye is 2 cm.I use Netbook Asus Eee PC 1002 HA with Intel Atom N270 CPU (1.6 GHz), 1GB Memory, 160 GB SATA HDD, and have small screen display 10 inch as main processing device.The software is developed under C++ language of Visual Studio 2005 and OpenCv, image processing library, which can be downloaded as free at their website.The hardware configuration is shown in Fig. 2.

B. Eye Detection
The eye detection is handled by adaptive thresholds and pupil knowledge.I do not need the tracking method because the camera is on the user's glasses, so the next eye location has the same position as the previous one.Pupil location is detected by using pupil knowledge such as color, size, shape, sequential location, and movement.In pupil detection, pupil location is estimated as stated earlier using pupil knowledge extracted using an adaptive threshold to separate the pupil from other eye components.The threshold value T is 0.27% below the mean μ of eye image I and is derived from adjusting illumination intensity of 150 lux.The proposed threshold is suitable only for this condition but enables the camera to automatically adjust illumination as it changes.
Output from the adaptive threshold is black pixels representing the pupil in the image.To eliminate noise, I use a median filter.Widely adaptive threshold output is divided into three categories: (1) case 1, in which noise free black pixels clearly represent the pupil, (2) case 2, in which noise appears and is the same size and shape as pupil, and (3) case 3, when no pupil properties can be used to locate pupil.Cases are shown in Fig. 3. Fig. 3. Three cases in pupil detection that to be solved by using pupil knowledge www.ijarai.thesai.orgOnce adaptive threshold output is classified, I estimate pupil location in three steps based on pupil knowledge.In case 1, the pupil is easily located by shape and size, even with noise in the image.In case 2, noise appears as almost the same size and shape as the pupil, e.g., when the adaptive threshold cannot be separated from other eye components such as the eyelid or the corner of the eye.To minimize these problems I recorded pupil movement histories assuming that the true pupil is closest to its previous location.
Reasonable pupil location P(t) is within the surrounding of previous location P(t−1) with area C. In case 3, when features cannot be found to locate the pupil as when the user is looking from the corner of the eye, I estimate pupil location based on the movement.

C. EEG Sensor
The EEG sensor used for getting EEG signal from users' forehead in the experiment is Brain Catcher BCG-1S manufactured by TECHNOS Co. Ltd.Outlook of the sensor is shown in Fig. 4. The sensor consists of two subsystems, EEG sensor and EEG signal receiver.Major specification is shown in Table 1.

A. Eye detection Accuracy
In order to measure performance of pupil detection, I involved five different users who have different race and nationality (Indonesian, Japanese, Sri Lankan, and Vietnamese).The uses of many samples will prove that the proposed method work when it is used for all types of users.The eye movement data was collected from each user while was making several eye movement.
Eye images of three Indonesian show that even though images were taken from same country, each person has different eye shape.The number of one of the Indonesian images 882 samples.One of them has slated eye and the other two Indonesians have narrow eye and clear pupil.The number of samples of these two Indonesian is 552 and 668, respectively.The collected data from Sri Lankan consists of the number of images is 828 samples.His skin color is black and eyelid is thick.Collected data from Japanese consists of the number of images is 665 samples.His skin color is bright and eye is slant.The collected data from Vietnamese is almost same of the other five.This experiment evaluates pupil detection accuracy and variance against different user by counting the success sample and the fail one.After accuracy of pupil detection has been counted, I compared the proposed method with adaptive threshold method and also template matching method.The compared adaptive threshold method uses combination between adaptive threshold itself and connected labeling method.Also, another compared method uses pupil template as reference and matched with the pupil images.The accuracy of the proposed pupil detection method against different users is shown in TABLE II. .The result data show that the proposed pupil detection method is accurate and robust against different users with variance value is 16.27.Line of sight direction is detected with NIR camera and NIR LED.Therefore, eye color and skin color are not matter for the eye movement detections.

B. Relation between Eye Movements and EEG Signals
Four Japanese users are selected for the experiment of eye movement detections and EEG signal acquisitions.Through the experiment, two documents (simple document 1 and complicated document 2 which are shown in Fig. 5) are read by the users.It is expected that brain is in very active for the complicated document while users are in relax status for the simple document.
(a)Simple document 1 Simple document in Japanese is written in "Hiragana" character only.The content of the document is story for children.Therefore, it is very easy to read. 2 Complicated document in Japanese is written in "Kanji" and "Hiragana" characters.The content of the document is text book for psychology for university students.Therefore, it is rather difficult to read.It is found that the frequency components for the simple document are much lower than those for the complicated document through a comparison between Fig. 7 and Fig. 9.The time variation is not so large.Also, there is not so large discrepancy among four users except user C. The user C is good at reading documents and has a plenty of knowledge and experiences on the contents of the documents.
From time to time, frequency component is getting high when users are reading complicated portions of the documents while is getting down when users are reading simple portions of the documents as shown in Fig. 7 and Fig. 9.
On the other hand, the number of rapid eye movement is counted when the users are reading the documents.The rapid eye movement is defined as 10 degrees of look angle difference for one second.Not only when the users change the lines in the document, but also when the users feel a difficulty for reading words in the document, the users' look direction moves rapidly.The number of the rapid eye movements is shown in Fig. 10.The timing of the rapid eye movements is synchronized to raising the EEG frequency component.Namely, when the users feel a difficulty of reading the documents and line changes of the documents, EEG frequency components are raising and eye moves rapidly as shown in Fig. 11 (Red circle indicates such moments).It is found that the number of rapid eye movement is counted when the users are reading the documents.The rapid eye movement is defined as 10 degrees of look angle difference for one second.Not only when the users change the lines in the document, but also when the users feel a difficulty for reading words in the document, the users' look direction moves rapidly.It is also found that the timing of the rapid eye movements is synchronized to raising the EEG frequency component.Namely, when the users feel a difficulty of reading the documents and line changes of the documents, EEG frequency components are raising and eye moves rapidly.Future research works are required for typical psychological status monitoring, hatred, beloved, joy, anger, happy, surprise etc. with eye movement detection.Then the patients without mimic muscle function can represent their www.ijarai.thesai.orgemotions.Also, psychological status monitored with the proposed eye movement detection is useful for triage of victims when they evacuate from disaster occurred areas to safe areas.Because vital sign which consist heart rate, blood pressure, the number of breath, body temperature and copiousness is required for triage.If patients in hospitals, group homes, etc. wear the glass with eye movement detection function and physical status monitoring sensor and network system for transmission of these data, then the patients can be saved their life because their status are known together with their location and attitude.

Fig. 5 .
Fig. 5. Documents used for the experiment Fig.6 shows EEG signals when the users read the simple document.Superimposed EEG signals for four users are shown in Fig.7 (The simple document).Meanwhile, Fig.8shows those when the users read the complicated document.

Fig. 10 .
Fig. 10.The number of rapid eye movements

TABLE II .
ACCURACY OF THE PROPOSED PUPIL DETECTION METHOD AGAINST DIFFERENT USERS, THIS TABLE SHOWS THAT THE PROPOSED METHOD IS ROBUST AGAINST DIFFERENT USER AND ACCURATE