Evaluation on the Effects of 2D Animation as a Verbal Apraxia Therapy for Children with Verbal Apraxia of Speech

—This article presents an evaluation of 2D Animation of Learning Numbers and Letters for Children with Verbal Apraxia. The developed application provides some knowledge and encourage children with verbal apraxia to learn and know about numbers and letters. An experimental testing was conducted to evaluate the usability of the developed application, aimed as a therapy for the children who suffer this apraxia across all age. Five important evaluation components such as learnability, usability, accessibility, functionality, and effectiveness were included in this testing to investigate the user engagement and satisfaction of the proposed medical and educational learning system. Online questionnaires were distributed as a method to collect user testing outputs. A total of 33 respondents from multimedia designers, practitioners, psychologists, and parents were involved in this survey. The results of the testing indicate that majority of respondents are satisfied with the outcomes of the 2D animation video. The results presented may facilitate improvements in the teaching syllabus for students with speech and language disorder and produce a great visual animation treatment to the users.


I. INTRODUCTION
Childhood Apraxia of Speech (CAS) is a speech condition in which a child's brain struggles to coordinate the complex oral motions required to create sounds into syllables, syllables into words, and words into phrases. An evaluation of the child's expressive and receptive language abilities is required as part of a diagnosis of childhood apraxia of speech; many children with this disorder have language deficiencies. This article focuses on the testing phase of the 2D animation of Learning Numbers and Letters for Children with Verbal Apraxia. The objective of this paper is to present a comprehensive usability evaluation of visual animation named "2D animation of Learning Numbers and Letters for Children with Verbal Apraxia". There are five evaluation components included in the testing to evaluate the user engagement and satisfaction of the 2D animation learning system. It is envisaged the proposed product would assist in providing a visual therapy to patients with Apraxia of Speech in the form of 2D animated video. The contribution of this study is obvious as the resulting outcomes can be capitalised as guidelines to increase an awareness about Verbal Apraxia to the public.

II. LITERATURE REVIEW / PREVIOUS WORK
There have been several studies in the literature reporting on the usage of animation, particularly in teaching and learning purpose [1][2][3]. In addition, there are some products which are related to the proposed visual animation for verbal apraxia of speech. For example, Speech Therapy Songs Animation by Barefoot Books [4] and Alphabets and Numbers Verbal Therapy by Terapi Autisme Online [5]. However, these products are limited for general, autism and spectrum disorder use. Besides, these applications are not interactive and the syllabus is outdated.
Hence, a 2D Animation of Learning Numbers and Letters for Children with Verbal Apraxia [6] is developed. This application is focusing on visual therapy to patients with Apraxia of Speech using an animation element. The comprehensive explanation on the design and development phases of the application; 2D Animation of Learning Numbers and Letters for Children with Verbal Apraxia is presented in [6]. The comparison of the existing products with the proposed application is also presented in [6]. Fig. 1, Fig. 2     Usability is described as a product's or service's ability to provide maximum satisfaction, efficiency, and effectiveness when utilised by various types of users as described in [7]. The usability has been investigated by many researchers in [8][9][10][11][12][13]. In recent years, there has been an increasing amount of literature on usability testing [14][15].
This paper presents an extension of the work reported earlier in [6] with emphasis on the evaluation of the proposed application. Also, this paper elaborated five components which are used as evaluation components for the system testing purpose; they are learnability, usability, accessibility, functionality, and effectiveness.

III. METHODOLOGY
The methodology section presents about the testing and outcomes. The test plan comprising of the test user, test schedule, test strategy and test implementation and eventually test results are elaborated in this section.
In this research work, the main target user is children who suffers from Verbal Apraxia of Speech from the age of 5-10 in Autism Therapy Center in Edufa Pekanbaru, Indonesia. The practitioner who guides the children will also be the target user as they will be observing the children's behavior while checking the content of the application to make sure it is according to the children's method of study. Table I shows the details of respondents that involved in this testing.

1) Multimedia expert:
A multimedia expert is a specialist in the field of multimedia and information technology who has extensive knowledge and expertise. Graphic Designer, Animator, Videographer, and Lecturer were chosen to take the test. This test is performed at the end of the development process and before the product is released. They will test the application with a focus on interface, interactivity, design, integration of multimedia elements, and content layouts in the application.
2) Subject matter expert: A subject matter expert is an individual that has a specialties and background of medical that help improve the application before and after the testing method. The practitioners, therapists, and psychologists were chosen to take the test. They will independently test the application and give an input about the application with medical terms and medical point of view.
3) Parents: The children with autistic and verbal apraxia of speech are the main target user of this research work. Since the testing was conducted online, the device setup and supervision were assisted by their parents. This category is addressed to the parents, who will explain the questionnaire to their children with their own way. Basically, this category is answered by their children assisted by the role of parents.
During the testing phase, test implementation will outline how the testing will be implemented to a certain target user. The relationship between the test description and the test data is carried out in accordance with the test strategy.

A. Test Description
The total number of respondents for the testing procedure is 33 people. After receiving an explanation of the research work, each respondent will conduct the testing independently. They are required to answer all of the questionnaire questions. Each of the respondent is required to respond and provide comments regarding the application in their perspective.

B. Test Data
The test data for the user testing will be explained in Table  II, while Table III, Table IV and Table V represent the respondents of subject matter expert, multimedia expert and parents, respectively.

C. Chart Results and Analysis
Diagrams and charts will be displayed in this analysis based on the results of the overview and testing measures. This is an examination diagram of the assessment testing that was conducted. A few charts have been created based on the information obtained from the testing results to summarize the outcome of the assessment.

1) Multimedia expert:
Eleven multimedia experts, including animators, graphic designers, videographers, filmmakers, lecturers, and multimedia students, took part in the testing phase. The multimedia experts were asked to assess the usage of the multimedia features in the animation video, which include content, audio, video, and interface design. The collected data will be analyzed and compiled into a bar chart. a) Chart of Learnability for Multimedia Expert: Fig. 4 shows the testing results from multimedia expert for the learnability aspect. For the information and messages outcome, we can see that 63.6% of multimedia experts strongly agree that the information and message in animation are simple and straightforward, this is because animation is aimed at children, who do not like watching long videos that make them feel bored.
There are about 72.7% of multimedia experts who strongly agree that the content of the animation is very easy to understand because the animation displays eye-catching and easy-to-understand visuals for the viewers.
It is indicated that there are 54.5% experts agreed that the instructions in the video can guide users because the instructions are easy to follow and understandable, but parental assistance is needed to be able to help children to always follow the instructions until the end. Fig. 5 shows the testing results from multimedia expert for the usability aspect. Although 45.5% agree, 9.1% disagree that the text and graphics are readable and clear because some experts feel that these graphics need to be improved, so that they are easily seen by the viewer's eyes.

b) Chart of Usability for Multimedia Expert:
More than 50% of experts agree that users do not need help with instructions outside of the video because the instructions in the video itself are very clear to understand and follow.
As illustrated in Fig. 5, the finding revealed that 54.5% of experts agree that the use of animation gains a better understanding of the content of the animated video because of the interaction of the host and the movement of the animation to the viewer.  c) Chart of Effectiveness for Multimedia Expert: Fig. 6 shows the testing results from multimedia expert for the effectiveness aspect. Fig. 6 shows more than half experts agree that integration between multimedia elements can help users obtain information effectively because multimedia instruments work together to help deliver information.
About 72.7% of experts agree that the layout of the content increases the effectiveness in conveying information as illustrated in Fig. 6 because the arrangement in the preparation of the layout is considered by the creators before being applied to the video. Fig. 6 highlight 72.7% of experts agree that the information has an impact on the user as stated in Fig. 12 because they believe that the instructions and interactions will help children in verbal apraxia therapy through visual interactions. Fig. 7 shows the testing results from multimedia expert for the accessibility aspect. Fig. 7 clarifies that although seven respondents agree that audio used is suitable, it turns out that there is one respondent who thinks that the audio used is not clear to hear because indeed the device in using audio recording does not use a special microphone for voice over, but only an ordinary cellphone headset.

d) Chart of Accessibility for Multimedia Expert:
Surprisingly two experts believe that the graphics used are not too attractive. This is because there is still a lot of potential that can be developed from the animation, such as designs that are more attractive to children than the existing ones. Fig. 7 shows that almost as many as a quarter of experts believe that the use of color in this animation is appropriate and attractive because the purpose of this animation is for children, so the animation must be made as interesting as possible.  e) Chart of Functionality for Multimedia Expert: Fig. 8 shows the testing results from multimedia expert for the accessibility aspect. Fig. 8 illustrates that there are four experts who strongly agree, and four others also agree with the statement that the animation is smooth and can be seen clearly. This is because according to their proposal, the use of visual aspects is very critical in making an animation video.
This study indicates that 63.6% experts agree that the user will be able to catch the message from the host quickly as presents in Fig. 8, this is because of the influence of good delivery and interaction from the host which makes the message will be conveyed well to the users.
Summary for Multimedia Expert responses are summarises in Table VI. f) Suggestion for Improvement from Multimedia Expert: Based on the questionnaire to the multimedia experts, below are the suggestion for improvement: i. Font for animation (numbers and alphabets) can be improved if the font used is a more familiar font. For example, the number 0 looks like a dot, because the hole is not so visible. The shape of the numbers/letters are not clear. ii. The main problem regarding handwritten text animation could be improved a lot so that the kids understand how they should write it, especially given with the font choice that should be suitable enough for using that effect. iii. Regarding about the sound quality coming from the speaker, it would be better to use a camera instead of using laptop or personal computer due to the sound quality would be bad for the kids to listen. With an alternative way to get the sound right, using a phone for sound recording while using a camera for visual recording will make the quality better, including for getting a good interactive feeling. iv. The animation wise could be better by not making it the same all the time, but should be at least different, no matter how simple it is. v. The animation could be better by not making it the same all the time, but should be at least different, no matter how simple it is. vi. The video is good and informative, the quality is fine, but the audio can be improved, maybe by using more prepared tools, because audio is one of the important instruments. vii. Instead of using numbers and letters animations that uses bold and thick type of font, the application might seek other font type alternatives that gives a clearer identification and distinguish between each alphabet/number. This suggestion is based on the personal difficulty to identify zero due to the unclear shape of font and animation flow. Hence, other appropriate font type and animation could be suggested to give audience a clearer understanding.
2) Subject matter expert: There are 16 subject matter experts, including practitioners, therapists, psychologists, lecturers, and psychology students who took part in the testing phase. The subject matter expert was asked to assess the content of the animation from the medical point of view. Then the collected data will be analyzed and compiled into a bar chart.
a) Chart of Learnability for Subject Matter Expert: Fig.  9 shows the testing results from subject matter expert for the learnability aspect. The finding highlights 56.3% experts agree that the user will be able to catch the message from the host quickly as displayed in Fig. 9, this is because of the influence of good delivery and interaction from the host which means the message will be conveyed well to the users.  Fig. 9 also shows that our finding revealed that most of the respondents agreed that the application animation and content are easily understandable.
There are ten experts agreed that the instructions in the video were clear to follow as stated in Fig. 20 because the instructions from the host were very clear and delivered straight to the point without wasting time. Fig.  10 shows the testing results from subject matter expert for the flexibility aspect. The finding provides evidence that eight experts agree that the content provided can promote positive behavior between children and parents as mentioned in Fig.  10. It is because it takes good cooperation and synergy between them to be able to apply the knowledge learned from the effects after watching the animated video. Fig. 10 also presents that all experts agree that this content about learning numbers and letters is very suitable for users such as children because in general at this age the syllabus is used in the field of education in schools is general. In addition, Fig. 10 stated that half of the experts agree and even the rest strongly agree that this animation will improve understanding of the content presented because children need content that is appropriate for their age such as numbers, letters, objects, colors, and so on.  Vol. 13, No. 7, 2022 This study indicated that eight out of 16 experts strongly agree that this application provides a complete information about alphabets and numbers as mentioned in Fig. 10. This kind of syllabus is very easy to understand and remember because it will always be used in everyday life by children who watch this animated video. Fig. 10 shows about 50% of the experts will use this application for verbal speech therapy in the future, because they believe they will be able to use it easily and without special instructions. Fig.  11 shows the testing results from subject matter expert for the effectiveness aspect. This study indicates that nine experts strongly agree that the integration of the use of multimedia elements in the animation helps users get information effectively as stated in Fig. 11 because they have used such a good, appropriate, and quite interactive elements and instruments.  Fig. 11 presents that our finding revealed that 75% or almost all of the experts agree that the layout of this content increases the effectiveness of delivering information. This is because the arrangement is in accordance with the needs of users, namely the placement of letters, numbers, and images that are not too flashy and the selection of colors that are not monotonous makes it easy for children to remember shapes and colors different in their memory.

c) Chart of Effectiveness for Subject Matter Expert:
There are nine out of 16 experts who were satisfied, even six of them were very satisfied with this animated content as mentioned in Fig. 11. They believed it would be a good therapy for children with verbal apraxia, enabling them to learn from the animated visual interactions of the video. Fig. 11 displays eight experts agree and five others strongly agree that this application is very relevant and believed to be useful for humans in the fields of health and education, especially for verbal speech therapy patients.

d) Suggestion for Improvement from Subject Matter
Expert: Based on the questionnaire provided to the subject matter expert, below are the suggestion for improvement: i. Videos are made more colorful to help children visually receive learning. ii. Made suitable for children to be able to guess the numbers or letters that have been learned. iii. The material is good, only in the selection of fonts for letters, otherwise use a simple font (without an accent of beauty) so that it is easier for children to imitate. iv. Children with special needs require less time in learning activities. A long enough duration can interfere with the child's concentration in receiving the material presented. v. Children with disorders need more visual acuity so that animated animations can be added that can provoke cognitive thinking in children to accept learning. vi. The background used is trying to be more dynamic, so as not to disturb the audience's focus on the back sound. vii. This video is quite interesting, where the media used in learning is technology. As we know that children with autism disorder like new things that are not too monotonous. With interesting videos full of colors and animations, it can increase children's motivation in understanding the material presented. However, this video has a long enough duration which can interfere with the concentration of children in receiving learning. The suggestion is that this video can be divided into two sessions, where the children can first be given a break and have a moment to rest in receiving the next material. 3) Parents: There are six parents of children with verbal apraxia took part in the testing phase. They assist the testing phase to run well by explaining the questionnaire to their children with their own way. a) Chart of Learnability for Parents: Fig. 12 shows the testing results from parents for the learnability aspect. Fig. 12 shows most parents agree and the rest strongly agree that this animation is able to attract the attention of their children because most of them acknowledge that their children are interested in interesting audio as well as cute and easy to remember animated pictures.
Half of the parents agree and the other half strongly agree that this animation is easy for their child to understand as illustrated in Fig. 12, because it contains clear instructions and keeps them focused.
There are four parents who agree and two others who strongly agree that the instructions in this video are clear for their children to follow as represented in Fig. 12 because the instructions and directions from the host are very clear with clear words and voices for the user who follows. Fig. 12 interprets about 66.7% of parents (4 respondents) agree that learning with animation really helps their children for better understanding of numbers and letters. It is more interesting and attractive for children, as children like and are happy when there are moving pictures that resemble shapes and have eyes image to teach them how to memorize letters and numbers. Fig. 13 shows the testing results from parents for the effectiveness aspect. Fig.  13 depicts that four out of six chose "strongly agree" that animated interactions involve their children as a whole from the beginning of learning to completion of learning, so that children are 100% involved in the learning process through the animated visuals.

b) Chart of Effectiveness for Parents:
The finding highlight 100% of parents agree that this application has a good impact on their children as shows in Fig.  13. It makes children happy with the animations given and able to repeat songs and read letters and numbers, also making them easily remember the concepts learned. Fig. 13 illustrates that our finding revealed that 66.7% of parents agree this application can promote positive behavior in children because it can make children to be more focused on something, be patient in seeing things and understand things slowly and well.
Also, Fig. 13 shows that four people agree and two others do not really agree with the statement that by using 2D animation is more effective than watching other learning videos. It is because not all children like animation, some only like the audio or the visual, not both. c) Chart of Accessibility for Parents: Fig. 14 shows the testing results from parents for the accessibility aspect. There are five people who agree that the layout of the content increases the effectiveness in delivering information as stated in Fig. 14 because behind a neat layout, a structured pattern will appear so that the video goes according to the specified plan.   This study indicates that 66.7% of parents agree this application can be used easily, without written instructions as mentioned in Figure 14. This animation ran smoothly and without issue. The host and the character along with the transitions both flows well to the end. The sounds and text are both clear and smooth.  but can talk about shapes, colors, animals, food and others. iv. Quite good, and quite interesting. The child is happy with the video as the child watched it three times because the child really liked the song. v. Do not make the video too long, suggested for two or three minutes. The rest is interesting and educational. vi. The child may be a little annoyed with a rather long duration but enjoys.

IV. DISCUSSION
It is recommended that further improvement be undertaken such as the professional recording equipment where the equipment like voice recorders and cameras play an important role in overcoming visual and hearing problems. It is suggested to use proper equipment which can help solve the audio problem. Besides that, duration of the video also needs to be shortened which will make the audience feel less bored. Thus, dividing the topics into separate video parts and not putting them in the same video will help solve the problem. In addition, the other contents or topics of the animation should be added like types of colors, shapes, plants, human body, and other topics which is still in line with the children's learning syllabus.
The contribution of this study is obvious as the resulting outcomes can be capitalised as guidelines as verbal apraxia therapy for children with verbal apraxia of speech. The findings of this study will help demonstrate the aid for children with verbal apraxia of speech in their learning. This research work promotes children and society by providing therapy for children with verbal apraxia. It can enhance the way to conduct therapy with a new method for kids, and hopefully can be used for adults too. Furthermore, this animation includes essential knowledge that is simple to comprehend for children. This animation also provides an excellent view for children to learn about topics, such as numbers and alphabets. This application encourages children to listen to and comprehend the messages, as well as to memorize them.

V. CONCLUSION
The purpose of the current study was to determine the usability of the proposed application; 2D animation for Learning Numbers and Alphabets. One of the more significant findings to emerge from this study is that the proposed application received positive feedbacks from the target users. These findings suggest that in general the application is accepted and greatly contributes to the verbal apraxia children. The animation has been successfully developed and run for the target users; children with verbal apraxia. This research work also meets the objectives, which is to develop an animation video as a therapeutic solution for children with verbal apraxia, and to evaluate the effectiveness of this animation. It is very useful and could be an alternative for therapists, instructors, doctors, and parents who have children who suffer verbal apraxia, for them to learn things by a new learning method with fun and entertainment. The children are entertained by the visuals of the animation, which motivates them to learn and study new things; if this application is developed carefully with more research in accordance with the criteria recommended by medical experts, multimedia experts, and parents, this research work will have a major impact on science and education if developed carefully. The 2D animation application was developed for the society and can reach out other scopes like social, community and even science and technology.