Augmented Reality Adapted Book (AREmotion) Design as Emotional Expression Recognition Media for Children with Autistic Spectrum Disorders (ASD)

One of the Autism Spectrum Disorder (ASD) characteristics is their difficulty understanding other people's emotions. Their lack of skill of understanding emotion includes expression and appropriate emotional response for a certain situation. This paper proposes an adapted book that helps therapists and parents guide ASD children to learn facial emotional expression. The adapted book combined with video, animation, and Augmented Reality increases children with ASD at recognizing emotional expression. This research uses UserCentered Design (UCD) approach to design the AR Adapted Book application and design the social story to be observable in school and family environment. Based on usability testing tried on the application, the result shows that AREmotion has an average score of 82.73 percent, and the learnability aspect has the highest score of 86.7 percent. This preliminary usability testing proves that the design of the AREmotion application is ready to use in real implementation for children with ASD. Keywords—Autism Spectrum Disorder (ASD); adapted book; Augmented Reality (AR); User-Centered Design (UCD)


I. INTRODUCTION
Recognizing facial emotional expression is an important part of daily life to build an excellent interpersonal relationship [1]. Yet, not all people, in general, can easily recognize other people's facial emotional expressions. People with ASD (Autism Spectrum Disorder) have difficulties recognizing facial expressions [2]. Autism is a developmental and neurobiological disorder that has powerful effects on children's growth and influences their communication skills and social interaction in daily life [3]. People with ASD spreads all over the world, and the people with ASD total number is not small. The World Health Organization (WHO) stated that one of 160 children is a child with Autism Spectrum Disorder [4]. This estimate is the average result of a population in a certain period in all studies that have been carried out. Since the time of their growth, children with ASD have a problem in some social interaction area, i.e. (1) The development of their interpersonal relationship is blocked. (2) The spontaneous willingness to share hobbies and interests is lacking. (3) A significant disruption in body language, eye gazing, and facial expression when they do social interaction [5]. There is bad impact if ASD children do not study about facial emotional expression. They will grow up with lack of empathy and they find difficulty to give appropriate reaction to other people emotional expression. To avoid those problems, a therapy is needed for ASD children so they can studying recognize other people expression in early stage.
A therapy, treatment, and guidance performed by therapist, parents, shadow teacher is needed to train the children with ASD to be able to learn how to interact socially. Parents and professionals, especially in Indonesia, stated that they need more coordination to give education, therapy, and care to children with ASD [6]. Children spent most of their time with family. This situation puts parents and family in the crucial role to treat the children with ASD very carefully. The treatment should be done correctly as every growth of children with ASD could inhibit the children growth phase. As a result, the children are in a high risk to be isolated from society. The isolation is due to their studying ability, lack of communication skills, and social stigma of children with ASD [7]. To overcome the described problem, a medium is needed to help therapists and parents to be able to take care of ASD Children in correct manner, especially so that the therapy for ASD children becomes more easily yet effective in treating the children.
There are various methods can be used to conduct a therapy for children with ASD. One of the therapies is using the sketch from drawing or art. Sketch art is a psychotherapy medium for children in general [8]. In case either normal children or children with ASD have difficulties or disruption, sketch art can be used so they can communicate in a nonverbal manner [9]. In general, therapists for ASD use a flashcard with sketch art or an illustration book as a supporting medium in learning and recognizing facial emotional expressions.
Previously, various technology has been developed to increase the quality of education and learning process of children with ASD. The research entitled The Impact of Technology on People with ASD [10] stated that there are four categories in learning with technology for people with ASD. The categories are conceptual skill (cover the subtopics: language, mathematics, colors, money, science, and programming), practical skill (subtopics: health, transportation, and daily life), social skill (subtopics: interpersonal relationship, communication, emotion) and general skill. The researcher explained that the most recent examination focused on utilizing technology to increase children's social skills with ASD. The study also emphasized that user needs, experience, and comfort are very important considerations in developing technology for children with ASD. The method for developing development designs based on user needs is User Centered Design (UCD).
One technology incorporated used in children ASD therapy is Augmented Reality (AR). Augmented Reality is a technology that visualizes virtual objects generated by a computer, and the virtual object resides in a real-world environment to give an interactive experience to the user. The virtual object consists of 2D or 3D objects, video, animation, and audio. Augmented Reality technology can be applied to many fields such as entertainment, advertising, education, and medicine [11]. Using AR in education, students will experience new things and give more exciting and interactive ways during the learning process [12]. However there are a lot of research about AR technology with positive findings that explained on systematic review that Augmented Reality Technology for ASD patients show that 56 % social skill learning using AR [13]. The study also state that social skills, cognitive skills, and communication skills can be improved by using AR-based learning. Most of AR learning implementation is carried out on children and adolescents with ASD on age range of 3-14 years old.
Previous research carried out innovation in interactive media by combining game books and Augmented Reality on tablet device to learn expression to children with ASD [14]. User with ASD confronted with choosing expression that match with scenario story that presented according the real environment, the game book was developed with mobile based technology. The author believed that the game book application would improve social skills of ASD children. The limitation of this research was they did not explain how to develop the application and social story on the game book, and it was not explained either how significant the used of application for the development of ASD children.
Previous research discussed ARSFM (Augmented Reality Self Facial Modeling) [15]. This research aimed the goal to help children with ASD to improve their social skills. The ARSFM used a mask with facial emotional expression to train their ability to recognize facial expressions. The result from the research, there was a significant difference between baseline and follow-up lines. The AR technology is implemented on a personal computer, not free to move while using the application. Furthermore the research using ARVMS (Augmented Reality Based Video Modeling Storybook), to teach social skills for children with ASD by understanding the expression and emotions of other people in the social environment [16]. ARVMS learning system has 2 layers : the first one is static image layer, and the second one contains videos. The system has 20 videos that focus on social interaction. These studies also confirmed that augmented reality can help children with ASD to improve their social skills through expression. Several limitations in this study undeveloped technology based on User-Centered Design principles in order to comply the requirement of ASD children learning process. The research also did not include how they started make the design from early stage and there is no assessment of the application using usability testing after the application was developed.
The current study proposed an Augmented Reality Adapted Book that combines static sketch in flashcards and adapted books as a tool for children with ASD. Each of these two combination tools is a common tool used by the therapist to help ASD children learn facial emotional expression. This research started by conducting an observation. The process of designing the adapted book's story and the application uses the User-Centered Design (UCD) approach. This technique will assist in the development process, to ensure the design meets the needs of children with ASD. After that, ASD's therapists will evaluate it by assessing the usability level of the developed application.
The hypothesis in this research is augmented Reality with virtual objects, i.e., animation, audio, and video, can help ASD children increase their ability to recognize facial emotional expressions. Comfortability and enjoyment experience in using the application will be aimed by using the User-Centered Design (UCD) principle to develop a technology on a handheld device. This research also hopes to help parents and therapists guide children with ASD in learning facial emotional expression. Therefore, people with ASD need special attention and special treatment to become people who can recognize and distinguish other people's facial emotional expressions.

II. MATERIALS AND METHODS
To comply with a request of ASD children's facial emotional recognition application developments, it should be designed as its requirement list. End user and stake-holder's experience are one of the important part so the product agrees with the requirement, the goals, and learning process for ASD children. Three special needs teachers was involved in the process of iteration to made AREmotion application. The iteration is based on User Centered Design, and from this step the research gets feedback from end user or stakeholder. The feedbacks influence the design so a user friendly application can be developed for ASD children [19]. Indirectly, UCD is a continual process a combination of direct identification, development, and brainstorming processes by involving several people or professionals from several disciplines to produce a product or media according to user needs [17].
To be more precise, this research used UCD for design and develop the AR application. There are four steps insist: 1) Specify the context of use to store user information, 2) Specify user requirements to gets information and start the product modification by inputting feedback that collected from previous step, 3) Design and development, 4) Evaluation of the design as an assessment in order to improve the design, so end user can comfortably interact with the application and the application can help them learning something. The methodology diagram is presented in Fig. 1.
In the development process of the AREmotion application for children with autism, 2 literacy sessions were carried out. This iteration process applies to the process of making stories in the adapted book (ARBook) and the development of the AREmotion application as a whole accompanied by a special needs teacher. In the evaluation phase, usability testing was carried out, which was carried out on the therapist for children with autism. Evaluation is an effort to find out whether the 633 | P a g e www.ijacsa.thesai.org overall design of the AR application developed in this study is easy enough before being used/implemented for special users, namely children with ASD.

A. Understanding and Specify of Context of use
In the first phase, to understand and specify the context of use, direct observation is made on a Special Needs School. The teachers at the school were asked and interviewed to get some information about ASD children. Other information collected from the interview is: a general picture of ASD children's activity, their social interaction with school peers, and learning methods and therapy applied for them. The collected data will be used as a reference when designing the story and illustration on the adapted book. The information gathered from the interview, explained in Table I.   TABLE I. INFORMATION ABOUT ASD CHILDREN ACCORDING TO SPECIAL NEEDS TEACHER

1.
In the studying process, ASD children often need a familiar guide (parents/teacher/therapist/shadow teacher) 2. ASD children experience difficulties in understanding the abstract concept of social interaction, i.e., facial emotion recognition.

3.
It is often hard for ASD children to focus on their studies.

4.
Therapy and learning methods for ASD children rarely use an updated technology.

5.
ASD children are more interested in visual objects such as pictures, photos, and videos.

B. Understanding and Specify user Requirement
After getting information about therapy and the learning process, the next step is modeling the actor's behavior, as explained before using the case diagram. The diagram in Fig 2 shows that the main actor in this research is children with ASD. The children will interact with the application in four different use cases. The first use case is viewing the animation through Augmented Reality. The second is viewing the video footage menu for human emotion and expression. The third is finding the multiple-choice to answer the problem on ARBook through AR marker. And the last is checking the notification 'True' or 'False' on their answer through augmented Reality.

III. DESIGN AND DEVELOPING OF AREMOTION APPLICATION
A. Design of Adapted Book (ARBook) 1) Design of the story of adapted book: The first design of storybooks that the researcher used is related to the book titled "A guide to producing written information in easy read" [18]. This book suggests using simple words, clear fonts, and sentences with less than 15 words, using pictures that are easy to understand and support the target sentences, and the images are made in the possible largest size.
After showing the first storybook's design to the ASD special needs teachers, they suggested dividing each expression into different categories. Every category contained some actions (in this field, the researcher made a limit in 3 actions in every category), and in 1 action would consists of 5 to 6 sentences.
In the second step, there are 18 accumulated stories divided into three parts. Each part contains six human expressions: happy, sad, angry, scary, disgust, and shocked. The therapist suggested a choice of word in every sentence the researcher made. Then, in the last step in developing the adapted book, the researcher formulated a flow to make a story for Autistic Spectrum Disorder (ASD). Here are the lists: • Illustration picture must be in sequence • Instead of abstract words, the explanation uses a concrete word. Take, for example, a dark sky, and there is a dark sky on the illustration.
• Avoid using the word like "want" and "will." • Show familiar story for children 634 | P a g e www.ijacsa.thesai.org • Use variable character, such as curly hair, straight hair, bright-dark skin, to show there are many ethnics and race (SARA) in Indonesia.
After arranging the sentences, an illustration made from each of the sentences and dividing a page of the Augmented Reality adapted the book into three layout sections, as shown in Fig. 3.   Fig. 3. Three-Section Layout on AR Adapted Book.
2) Illustration design: The adapted book illustration was made sequentially according to the formula list in Section III (A). Each page on the adapted book is divided into three layout sections, i.e., illustrative layout, sentence or narrative layout, marker layout. The illustrative layout contains pictures that visually explain the sentences beneath it on the narrative layout and marker layout allocated for the camera to detect the marker on that area.
3) Marker ARBook design: There are 18 different markers on each page designed on an Augmented Reality adapted book (ARBook). Every marker was made to suit facial expression on the adapted book pages. The marker can show the multiplechoice answer to the problem story in the adapted book.

B. Design of Augmented Reality Object on ARAnimasi 1) ARAnimasi's character design:
A digital sketch was used in the process of designing and editing the animation character. The character was made and grouped into two categories, male character, and female character. When the animation showed, the upper body of each character will pop out. The illustration described in Fig. 4. While designing the sketch, it focused on facial features such as the shape of the eyes, eyebrow, and mouth which is described in Fig. 5. The shape of the facial feature is used to differentiate facial emotional character on each animation sketch.  2) ARAnimasi's marker design: On the ARAnimasi menu, there is 12 flashcard-shaped marker choice with emotion animation object on augmented Reality. The ARAnimasi marker consists of two gender groups, male and female. Each group contains six emotional expression animations (angry, sad, happy, afraid, nauseous, and surprised). The design of the ARAnimasi's marker can be seen in Fig. 6.

A. Tools and Technology
The development of the AREmotion application using Unity and Vuforia. Vuforia was chosen because this platform provides Augmented Reality SDK for image recognition ability to recognize the marker [19]. Every marker in the AREmotion application is saved in the Vuforia account. The engine that is used in the AREmotion application, Unity3D, all of the process is in PC. The result of the Unity3D engine is in the form of a smartphone application with Android systems.
Every animation and character in the ARAnimasi menu was drawn in Adobe Photoshop and Crazy Talk Animator. Wavepad was used for recording and balancing audio. The Adobe Premiere was used for the final rendering of the animation video. AREmotion application was used in the smartphone. The mobile device specification used was 2 GB RAM, 16GB internal memory, 16 MP camera to detect the markers, and the operating system Android 6.0 (Marshmallow).

B. System Overview
AREmotion is a combinate application of adapted book and mobile Augmented Reality, which runs in smartphone with platform Android. AREmotion divided into three menus. They are video, ARAnimasi, and ARBook. The first menu is "video", consists of video footages with kinds of expression gesture like sad, happy, angry, afraid, disgust and shock. The second menu is "ARAnimasi", consists of a marker flashcard with six child expressions (for girls and boys) in AR is shown in Fig. 8(left). The third menu is "ARBook", which consists of questions in the form of a story/storybook, with the answers in AR is shown in. 8(right). The AR adopted book's physical form is printed in the size 21cm x 16cm, as shown in Fig. 7. There are two menus in the book that is ARAnimasi and ARbook.

C. Features Overview
In this study, there are two designs Augmented Reality, ARAnimasi dan ARBook. Fig. 8 shows an example of the ARAnimasi. Fig. 9 shows a menu containing 5 seconds animation video supported by clean audio to express human emotion. For example, there is the sound of someone's crying in the video. It is to show the children the expression of sadness.  Fig. 10 shows an example of the ARBook menu design when the smartphone camera is ready to detect the marker. While Fig. 11 shows, there is a multiple-choice answer organized as a UI AR button with slight animation from the detected marker. Animation of the answer pops out for 3 seconds, and it played automatically when the smartphone camera detects the marker. Fig. 12 shows the design of augmented Reality that pops out when the user chooses the correct answer and wrong answer.

V. RESULT AND ANALYSIS
The usability of ARemotion is tested on three therapists as the respondent. The therapist tested the usability by experiencing the AREmotion end fill out the questionnaire. Usability is a method that aims to measure whether the application media that has been built is in accordance with user needs. Usability testing can verify whether the applied design has worked well because there is an interaction how the user can cognitively assess the questions on the questionnaire, as the implementation of the interaction with the designed application [20].
There are 17 statements related to five categories of usability in the questionnaire, i.e., efficiency, learnability, errors, memorability, and satisfaction [21]. Each statement in the questionnaire is evaluated by the respondent using the Likert Scale. The respondent gives marks or rates from 1 to 5 to each statement [22]. Table II shows the Likert Scale which describes the symmetrical scale assessment of the respondent's level of agreement or disagreement [20], and Table III shows the questionnaire and all respondents's rates.  In my opinion, children with ASD will not be making many mistakes when they use AREmotion (Augmented Reality Adapted Book) 0 0 2 1 0 10 13.
I think when children make some mistake in using the application, they will easily correct it. Based on the result in Table III, the sum of each category will be calculated. Table IV and Table V show the calculation  process.   TABLE IV. THE TOTALLING SORE FOR EACH CATEGORY   No  Category  1  2  3  4  5

VI. CONCLUSION
This paper focused on analyzing the design and developing the application augmented Reality adapted book (AREmotion) for children with Autism Spectrum Disorder (ASD). The primary participant in this research is therapist and ASD children. The therapist was involved in designing the application using the User-Centered Design (UCD) approach.
The adapted book's story design was based on a simple consecutively event concept explained in 2-5 sentences and illustration embedded in each sentence. There are three main menus on AREmotion, i.e., ARAnimasi consists of emotional expression animation video in augmented Reality, video emotion menu consists of a group of human emotional expression footage video, and the last menu is ARBook that consists of story problem series about emotional expression for ASD children includes the answer in augmented Reality.
In this initial research work, the application's design is tested on the therapist to en-sure that this application is efficient, easy, and understandable before children with ASD use it. Usability testing on AREmotion has 82.73%, and the highest score among all categories is learnability, with an 86.7% score. The result shows that the AREmotion application can be used as educational media for children with ASD when learning about facial emotional expression. The next step of this research is evaluating the usage of AREmotion to help children with ASD recognize emotional expression through this innovation. 637 | P a g e www.ijacsa.thesai.org