SmileToPhone : A Mobile Phone System for Quadriplegic Users Controlled by EEG Signals

Quadriplegic people are unable to use mobile devices without the aid of other persons which can be devastating for them both socially and economically. This has motivated many researchers to propose hardware and software solutions that operate as intermediates between the impaired users and their devices: accessibility switches, joysticks and head movements. However, the efficiency of these tools is limited in some conditions. To alleviate this problem, we propose to exploit electroencephalographic signals captured via an adequate headset. More precisely, the user is asked to perform a facial expression that will be recognized by the system through the analysis of the EEG signals. Several facial expressions are offered and each one corresponds to a command wirelessly sent to the mobile device and executed. This Brain Computer Interface based system is called SmileToPhone. It enables the quadriplegic patients to use their smartphones in an easy way with a minimum of effort and with respect to studied Human-Computer-Interaction requirements. The system includes the main functionalities of a smartphone such as making calls and sending messages. The evaluation of the system usability showed that most of the time, users were able to use the different functionalities of the system in an easy way. The current results are encouraging and motivating to add more features to the system. Keywords—Quadriplegia; EEG; facial expression; BCI system; HCI


I. INTRODUCTION
Mobile devices like tablets and smartphones are transforming our life by the emerging of new technologies and mobile applications offering new possibilities for communicating, working, shopping, etc.However, people suffering from disabilities particularly due to a Spinal Cord Injury (SCI), find themselves unable to follow this flow of technologies in continuous progress, which can be devastating for a person both socially and economically.Furthermore, a study reveals that the most common age of injury is 19 years and that a large percentage of spinal cord injury patients are under 30 years old (except in Japan where the majority of the patients are over the age of 50 years) [1].Physical difficulties, to mobility and use of basic technology yield to the exclusion of many people from participation in society, especially during this period of life between the age of 19 and 30.Hence the need for a system that allows mobility impaired persons to benefit from the available technologies and services likewise healthy people.Several applications are proposed in the literature that aim to help mobility impaired users to make phone calls [2], [3], use computers [4], [5], play games [6], prepare the meal and other functions [7].The key idea is to use a hardware operating as an interface between the user and the device to be manipulated.The interface could be a joystick that the user moves in different directions using one finger [4], [7] or using his lips [5] in order to navigate or select a functionality on the device.Accessibility switches were also exploited in the Tecla product to transmit commands to a smartphone or a tablet via a Bluetooth connection by using the user's hand or a finger [4].Sip and puff sensors allow the user to puff for clicking and selecting a functionality.In addition to a lip position sensor, a push switch and voice commands are exploited in the Quadstick product for playing games [6].A different idea for moving a cursor and selecting the desired item on an android device is the one implemented in Sesame application [3].It consists of tracking the head movements through the camera of the device, recognizing them using computer vision algorithms, and associating each movement to a defined action on the screen.Applications based on the Brain Computer Interface (BCI) are also proposed to help mobility impaired persons using their mobile devices: the idea consists of analyzing the brain signals to recognize the action to be executed on the device [2], [7].
In the present work, we are interested in developing a mobile phone system to people suffering from a special spinal cord injury which is Quadriplegia (also called Tetraplegia).According to the severity of the injury, quadriplegia yields to varying levels of functional loss in the neck, trunk, and upper and lower limbs [8], whereas quadriplegic patients have a full control of the head and the facial organs.As a consequence, the use of materials such as joysticks and push switches is not appropriate for our target users.Furthermore, puffing may be tiring; in addition to the fact that it requires a wired connection to the device.A number of requirements should be accounted for when designing a mobile phone system for quadriplegic patients.For instance, a physical movement from other than the head and the face of the user are discouraged and even not possible.Besides, in order to ensure a maximum level of usability of the system, it is preferred that the material used for transmitting the commands to the mobile device be wirelessly connected.These requirements are perfectly satisfied in Sesame application [3].However, it presents some limitations restricting its use to some conditions: since the head movements are captured via the camera of the device, it is very sensitive to the brightness level present in the room.Hence, the sesame phone should be in a well lit room without being exposed to a light source.This compromises the comfort of the user when he needs to be within a slightly bright room and restricts the usage of the phone in some areas, especially when the user is out of home and has no control on the lightning level.Another issue is that the unlocking of the phone is performed using the voice, by recognizing the sentence 'Open sesame'.The recognition may fail when the user is in a noisy environment.Neurophone [2] is another phone system that satisfies the aforementioned requirements.It is a BCI based system that exploits the P300 brain potential to select the photo of the contact that the user wants to call.The idea of the Neurophone application is to sequentially flash in a random order the photos stored in the address book contacts.When the flashed photo corresponds to the contact to call, a P300 potential is evoked by generating a peak after a stimulus.Although the idea of using brain signals to send commands to the phone is interesting and ensures flexibility to the user, the P300 depends on the levels of attention and arousal [9].In addition, a more accurate way that does not require a prior training stage and allows the understanding of the user's intent, is to interpret his facial expressions through his brain signals.Furthermore, the Neurophone application restricts the phone calls to the contacts stored in the address book and whose photos are available.Given the high degree of autonomy offered by BCI technology and the success it achieved through several available systems [2], [7], we resort to the exploitation of the brain signals to manipulate the proposed mobile phone system.More precisely, the brain signals are used to recognize a facial expression performed by the user, which is then translated to an action to be executed on the mobile device.Our choice of the analysis of the brain signals is motivated by their accuracy and the quasi real-time of their processing; whereas the use of the camera to capture the facial expression followed by an analysis step based on computer vision algorithms is compromised by the lightning of the room as mentioned earlier.
The contribution of our mobile phone application, named SmileToPhone (referring to the smiling facial expression), is not restricted to only phone calls from the contacts of the address book, but also includes dialing a phone number, performing an emergency call (by dialing a number or selecting a predefined number), reading and writing messages, setting alarms and also adjusting some settings regarding the way in which the commands are sent to the device.It also includes a fault management module offering the possibility to the user to reset his inputs in case of error, and allowing an additional flexibility to the application.The remainder of the paper is organized as follows.Section II describes the proposed system by detailing the process of brain signals acquisition, the system features and the HCI requirements specific to the quadriplegic people and taken into consideration in the design phase.The evaluation results of the system usability are presented in Section III.Finally, conclusions and future work are drawn in Section IV.

II. PROPOSED MOBILE PHONE SYSTEM FOR
QUADRIPLEGIC USERS The SmileToPhone system consists of two main parts: the first part aims to analyze the brain signals in order to recognize the facial expression performed by the user.The second part is an Android application installed on the patient's smartphone that interprets the facial expression as a function to be executed.The high level architecture of SmileToPhone system is illustrated in Figure 1.

A. Brain signals acquisition and analysis
Thanks to the interactions between billions of neurons present in the brain, people are able to think, move, feel emotions, and more.All these feelings and thoughts start in the brain and are transmitted through neurons to other neurons or other types of cells such as muscles, via electrical signals.The electrical activities of the neurons emerge in the brain surface and thus can be captured by placing electrodes on standard positions on the scalp according to the 10-20 international system [8].The recording of the electrical activity of the neurons is called electroencephalography (EEG).Several cap-like devices composed of electrodes and allowing the acquisition of the EEG signals exist [10].They differ by their external appearance, the number of electrodes, their applicability (medical or non-medical use), cost, and other characteristics.
Taking into account the features of the proposed system and the targeted users, some constraints regarding the choice of the EEG headset should be accounted for.In one hand, the cost of the headset should not be expensive and its placement should be relatively easy and does not require a training stage.In the other hand, the acquisition and the interpretation of the signals should ensure a minimum of accuracy that allows a satisfying level of the system usability.Several low-cost EEG devices are commercially available in the market.A survey of most of them along with a comparison are conducted in [11], where the Emotiv Epoc headset [12] was evaluated as the most usable low-cost device.More precisely, a comparison between the Emotiv Epoc headset and the Neurosky headset was conducted in several works, confirming the outperforming of the former one [11], [13].The Emotiv Epoc headset has 14 electrodes located on AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4 positions as shown in Figure 2. Eight of these EEG sensors are positioned around the frontal and prefrontal lobes to collect and record signals from facial muscles and eyes.Once the brain signals are collected, they are processed in order to extract the relevant features allowing to recognize the facial expression performed by the user.It is worth pointing out that a Software Development Kit (SDK) for research is available along with the Emotiv Epoc headset and Fig. 2. Positions of the electrodes in the EPOC headset [15] offers the processing of signals, which is mainly composed of the following 3 stages: 1) Preprocessing: The aim of this stage is to make the acquired brain signals suitable for analysis by amplifying them and removing the electrical noise to enhance their quality.The signals are then digitized.2) Feature extraction: In this stage, suitable features helping to recognize the user's facial expression are extracted from the digitized brain signal samples.3) Features classification: This step can also be denoted as the translation algorithm; it is comprised mainly of a signal translation procedure that converts the set of brain signal features into a set of output signals to control a device.This translation is accomplished using conventional classification procedures [14].
Once the facial expression is identified, a command is associated to it in order to control the mobile phone device.

B. Command identification
The system can recognize up to 12 facial expressions including smile, left wink, right wink, blink, raised eyebrows (surprise) and some others.We associate some of these facial expressions to specific commands that allow the functioning of the desired feature in the mobile phone.The main commands consist of: • Unlocking the phone, • Selecting an icon, • Moving up/down to navigate through icons.
A facial expression is attributed by default to each of these commands: smiling to unlock the phone and to select an icon, winking left to move up and winking right to move down.As will be explained later in the paragraph II-D, the keypad is required to be simple with large icons.Consequently, the keypad of our application (for the dialing function) and the icons organization are designed to be vertical.Moving through icons is only in up/down directions, as shown in Figure 3.A minimum number of facial expressions is exploited in order to facilitate their use and memorization by the quadriplegic.However, it is to be noted that the user has the possibility to customize the facial expressions associated to the commands through the function 'Settings' of our application.The remaining features offered by SmileToPhone application are described below.

C. System features
The application includes the following main functions as shown in the use case diagram (see Figure 4); the first function is the Emergency call; it allows the patient to ask for help through a predefined phone number or a new one that he dials.The second one is the Call function; it helps the patient to call any number from his contacts or enter a new number by using a keypad appropriate for the mobility impaired users.The requirements related to the design of the keypad and more generally, the Human-Computer-Interaction aspects will be discussed in the paragraph II-D.The third function is the Message function: by using it, the patient can read his messages and write a new message with a special keyboard.As a fourth available feature, the user has the possibility to set an alarm.
Another important feature of the SmileToPhone system is that it supports a fault management module allowing the user to reset his entry after an error.
It is worth noting that our system is designed in such a way it can be easily extended to support additional functionalities without altering to the existing implementation.
All the features are presented to the user with respect to Human-Computer-Interaction (HCI) requirements defined in [16] and described in the following.

D. HCI user requirements
The HCI of the proposed system is based on a study conducted in [16] on 11 participants suffering from mobility impairments.The participants were men and women of different ages and professions.The study aimed to observe how the mobility impaired users interact with computers and mobile devices and what are the limitations they face.A questionnaire was also addressed.Some of the findings of the study are listed below and are taken into consideration in the implementation.
• Graphic icons should be large enough to be easily manipulated by users suffering from quadriplegia.
• The text should be clear.
• It should be easy to read the interface at some distance that allows operation from the wheelchair.• The screen should be vertically positioned.
As can be seen in Figure 5, the screens of the Smile-ToPhone system are vertically positioned, with large icons and clear text.The list of contacts also appears in a vertical direction.The keypad used for dialing a number is simple, clear and easy to move up and down through it (by right winking and left winking respectively).

III. USABILITY EVALUATION
In order to evaluate the usability of the proposed system, a usability study is conducted in which five healthy participants were asked to perform a set of tasks.It was not possible to make the study on quadriplegics.The focus was on the main features of the proposed system, which are: make an emergency call, make a call and send a message.
In the emergency call task, participants have to select the emergency call icon from the home interface and make an emergency call.Two sub-tasks are considered in this task; making an emergency call when a number is already saved in the emergency call list and making an emergency call with a new phone number.In the make call task, participants are invited to select the make call icon in order to be able to make a call.Also in this task, two sub-tasks are considered; making a call to a phone number from the list of contacts and making a call by entering a new number.In the send message task, participants have to send a message in two ways; by selecting a predefined message and by writing a new message.
As results, it was observed that the executions of the different commands using the corresponding facial expressions were instantaneous, except for some isolated cases where a user had to perform a facial expression twice in order for the related command to be executed.

IV. CONCLUSIONS AND FUTURE WORK
In this paper, we were interested in facilitating the social integration of users suffering from quadriplegia.We proposed a mobile system that allows them to use the smartphones effectively.Taking into account that physical movements are discouraged and sometimes not possible for most of the quadriplegic patients, the facial expressions were exploited to control the smartphone.In that way, quadriplegics can use their smartphones with a minimum effort.For that sake, the mobile application consists of five main functionalities; make an emergency call, make a call, send message, customize the facial expressions and set an alarm.HCI requirements have been taken into account when designing the system.As a future work, the aim will concern the adding of more functionalities to the system allowing the full control of the smartphone by quadriplegics.