Real-time Facial Emotion Detection using Support Vector Machines

There have been continuous researches in the field of emotion detection through faces of biological species in the last few decades. This was further fuelled by the rise of artificial intelligence which has added a new paradigm to its ongoing research. This paper discusses the role of one of the artificial intelligence techniques, Support vector machines for efficient emotion detection. This study comprised of experiments conducted on Java platform by using libsvm. The coordinates of vital points of a face have been used for training the SVM network which finally led to proper identification of various emotions appearing on a human face. KeywordsEmotion-Detection;Facial-expressions; libsvm; Support Vector Machines; Facial action coding system(FACS)


I. Introduction
Human face is one of the most important source for knowing a person's behavior, personality and cognitive state.They are the perfect keys to unlock an individual's identity.Various researchers have tried to define emotion as a person's reflection.Keith Oatley and Jennifer M. Jenkins [1] said about the definition of the emotion as following: 1)An emotion is usually caused by a person consciously or unconsciously evaluating an event as relevant to a concern (a goal) that is important; the emotion is felt as positive when a concern is advanced and negative when a concern is impeded.2) The core of an emotion is readiness to act and the prompting of plans; an emotion gives priority for one or a few kinds of action to which it gives a sense of urgency -so it can interrupt, or compete with, alternative mental processes or actions.Different types of readiness create different outline relation-ships with other.An emotion is usually experienced as a distinctive type of mental state, sometimes accompanied or followed by bodily changes, expressions, actions.Knowing about other's emotion greatly affects our interpersonal communication providing important communicative cues such as our level of interest, our desire to take a speaking t u r n and c o n t i n u o u s feedback signaling understanding of the information conveyed [3].Facial expression constitutes 55 percent of the effect of a communicated message [9] and is hence a major modality in human communication.The literature of expression analysis system classifies emotions through t w o m a i n s c h e m a s .Ekman [2] proposed the first, which is also the most widely used schema.This schema says that are six 'basic' emotions, namely anger, disgust, fear, joy, sorrow and surprise, from which more complex combinations can be constructed and which are easily identified (by humans) in facial expressions.FACS, Facial action coding system is the other one, which is more accurate and precise but complex [4].Here every expression is coded as a combination of 44 facial movements called 'action units'.It is followed by decomposition of every expression into individual action units by FACS experts.[5].An automatic FACS system is still a distant dream.Because of this limitation most of the emotion detection system uses the first schema of basic emotions.
In this study we have also used the first schema for facial emotion detection that uses an artificial intelligence technique, support vector machines.SVMs view the classification problem as a quadratic optimization problem.This technique has successfully been applied to standard classification tasks, such as text classification [6,7] and medical diagnosis [8].The user manually marks the most vital points on the face of an image which acts as an input the SVM's for its training.After training, once we apply an image as an input, it efficiently recognizes the underlying emotion.The remainder of this paper is organized as follows: next section gives an overview of the SVM's and its role as an emotion detection tool, which will be followed by discussion of libsvm.Then we outline the overall design of our approach to emotion detection.And finally we conclude with future work and conclusion.

II. Overview
The overall project can be divided into four major components a)Importing an image and processing it so as to provide it as an input to SVM b) Incorporating SVM into the application c) Train SVM d) Present the results of the classification output by the SVM in a meaningful manner to the user.
While for the first two issues we have used LIBSVM toolkit, the latter two are application specific.The description of LIBSVM tool would be discussed in coming section Here we take the image of a face and ask the user to mark 22 features of face.User needs to manually mark these features and then the coordinates of these features are stored for further analysis.Data extraction is done by SVM while for classifying expression we have used Ekman approach, which states that there are six 'basic' emotions, namely anger, disgust, fear, joy, sorrow and surprise, from which more complex combinations can be constructed.

III. LIBSVM
Implementing SVM can be a tedious job; thankfully there are freely available softwares whose goal is to help users to easily use SVM as a tool.LIBSVM is integrated software for support vector classification, (C-SVC, nu-SVC), regression (epsilon-SVR, nu-SVR) and distribution estimation (one-class SVM).It supports multi-class classification.LIBSVM provides a simple interface where users can easily link it with their own programs.The current release (Version 2.85, November 2007) of LIBSVM is available.The package includes the source code of the library in C++ and Java, and a simple program for scaling training data.

IV. Support Vector Machines
During the training phase of machine learning algorithms, a model from the received input is build.And finally it provides a hypothesis function that can be used to predict result for the further input data.There has been a great increase in the use of SVM's for high performance pattern classification.

V. Simulation results
In this section we demonstrate the working of the proposed facial emotion detection tool for two basic emotions which can be further extended to other 6 basic emotions.This tool has been constructed on the java platform using various predefined functions of java Swing.The tool starts with the below given applet: This is followed by manual marking of the various vital points on the face with the tool.
Figure 4. Vital co-ordinates marked on a 'Happy' face [11] This same is repeated for the different classes available for the above mentioned six emotions (anger, disgust, fear, sorrow and surprise).Figure 6.Vital co-ordibates marked on a 'Sad' face [11] After this the SVM is trained by pressing the 'Train' button of the applet.This process leads to the formation of hyper planes as shown in figure.
Figure 7. Formation of hyper-planes when the SVM is 'Trained' [11] Once it gets trained, our network is ready for emotion detection.Now as soon as we gave an image as an input and pressed the 'Predict' button, it gave the correct prediction about the emotion of the image.The experiment carried out by us in the initial phases.Further tests are being carried out to find the accuracy of our proposed emotion detection tool.

VI. Conclusion
We have presented some progressing experiments for classifying emotional facial gestures using still images.We have used SVM classifiers.Its test results have proved to be acceptable.We have worked on our image database by manually extracting the prominent facial features from the image and henceforth training the SVM's for classification of each emotion.Since the direct transition between certain states of the face may happen rather rarely, thus information retrieved from more than one immediately succeeding frames of video sequences can increase on the accuracy.It will be the next step in our research work.

VII. Future Work
The possible extensions that can be made to the application are that it can be made to work with a greater number of and complex emotion categories.Could for instance train the SVM with 'happy, sad, angry, and surprised' and then query it with a new sample belonging to one of these four categories (multi-class classification).This is more difficult than binary classification.Moreover there is also room for experimenting with different versions of Support Vector Machines.There are various optimizations of the basic SVM, suited for different applications.Evaluating and comparing the performance of different versions can be tried.Moreover, analysis results here can be made compatible with emotion animation modeling, so that it will be simple enough to or combine the developed system into Human Computer Interaction applications, where a user can interact with an emotionally responding e-buddy or virtual character.It is expected that the results can further be worked upon by incorporating the inputs from a 2D image data into a fullfledged 3D model of the face.

Figure 1 .
Figure 1.Emotion detection using SVM For detecting emotion of images Pantic & Rothkrantz [10] defined three core problems-a) Face detection in an image or image sequence, b) Facial expression data extraction and c) Facial expression classification.For still images, it is assumed that the images are of faces.This solves the face detection problem.If the images have other body parts included, then we may crop the face from the image.Facial expression data extraction can be achieved by making an application work in a manner similar to that of FaceTracker[5].Here we take the image of a face and ask the user to mark 22 features of face.User needs to manually mark these features and then the coordinates of these features are stored for further analysis.Data extraction is done by SVM while for classifying expression we have used Ekman approach, which states that there are six 'basic' emotions, namely anger, disgust, fear, joy, sorrow and surprise, from which more complex combinations can be constructed.

Figure 2 .Figure 3 .
Figure 2. Start-applet of the Emotion detection using SVM tool

Figure 5 .
Figure 5. Test image for the Facial Expression of a 'Sad' face(JAFFE Database[11])