EDUGXQ: User Experience Instrument for Educational Games’ Evaluation

A significant increase in research on educational computer games in recent years has proven that the demand for educational games has increased as well. However, production of incompatible educational games not only cost wastage of money but also energy and time for game designers and game developers. To produce a suitable educational game, it is important to understand the user’s need as well as the educational need. Therefore, this study aims to develop a User Experience (UX) framework for educational games (EDUGX) based on UX elements and psychometrically validate a new instrument, EDUGX questionnaire (EDUGXQ) that is appropriate to evaluate educational games. Based on literature review, six main UX elements were identified which are Flow, Immersion, Player Context, Game Usability, Game System and Learnability to construct the framework. In this paper, we first discussed the development process of EDUGX framework followed by EDUGXQ. This study will also review and discuss several UX questionnaires for educational games in UX design evaluation which at the same time supports the framework’s elements to develop the EDUGXQ.


I. INTRODUCTION
The era of the gaming world has grown extensively as the recent internet trend report shows that the number of gamers from 1995 to 2017 has reached tremendous height, with an increase of 2.6 billion gamers in 2017 versus 100 million in 1995 with a global gaming revenue of $100 billion in 2016 [1]. This sudden rise of computer games usage as a favorite past time activity has increased the attention in using educational games to help in skill acquisition, behavior change, attitude and learning [2] and created a significant increase in research on educational computer games in recent years. Problems such as lack of game content or learning content often occur when designing a computer game with learning purposes [3]. Nevertheless, there is lack of agreement among game aficionados in deciding what features are essentials to the creation of successful or good games [4].
Since it's still a new knowledge and in its inarguable phase, the game developer considered the field of educational game as new and risky. Moreover, the designing and developing of educational computer game would require a large budget, resources and special skills. According to [5], there are also needs to have an appropriate level of educational accuracy to suit the learning subject and the classroom environment. Poor and irrelevant design of a product may resulted in unwanted or wasted production, money and energy. This clearly shows that educational games production can be a difficult process.
To understand the complicated process of developing educational games, the conception of evaluation is required. An evaluation is required as it allows describing a framework which will be verified by third parties in different perspectives and the results of the evaluation can assist the developer or designer to pinpoint the errors or negative points of the game. Even with a growing body of evidence on the efficacy of computer games for learning, evaluation is often incomplete, biased, poorly designed, if not absent [6]. In addition, developing and evaluating games for specific purposes whether for behavioral change or learning is a very challenging enterprise.
Currently, most of the evaluation or testing of computer games during the development stage are done with technical testing such as bug testing and usability testing. This testing ensures that computer games can perform as per their technical requirement and game functionality. However, when a player uses computer games, it becomes more than technical or functional aspects. Game which consist of the noninstrumental qualities are more important to be evaluated compared to other digital systems [7]. Non-instrumental qualities can be defined as quality features of an interactive system that address user's needs that go beyond goals, tasks and their efficient achievement [8]. These qualities involved the user's personal preferences and emotions. Therefore, for this purpose an UX evaluation is needed to identify the user's preferences and responses [9] as each of the player's experience is unique.
This unique experience can be evaluated if the right UX elements are measured [9]. Hence, an UX evaluation framework is especially important for predicting, understanding and reasoning about procedures of UX [10], which indirectly help the game designers to evaluate their games. By producing a framework on the UX evaluation for educational games, it helps the future educational games designers to understand the needs of their users and indirectly helps in developing productive educational games with costeffective learning design and game development.
Thus, in this study, the key elements for UX evaluation will be identified and tested to identify if the games contain the elements to become a successful game. User Experience Evaluation for Educational Games (EDUGX) framework was developed to support UX evaluation for educational games www.ijacsa.thesai.org followed by the development of the gaming scale EDUGX questionnaire (EDUGXQ) that is based on the EDUGX framework. This section includes the background of the present research, research approach and its aim. Section 2 discusses the related works in the process of EDUGX framework followed by section 3 which focuses on the construction of EDUGXQ and discusses the expert review on the proposed EDUGX and EDGXQ. Followed by the discussion on the findings. The last section is the conclusions and future works of this paper.

II. RELATED WORKS
User experience is focused on the interactions between products and people, and the experience that resulted in certain contexts of use [15]. This experience can be evaluated if suitable elements are measured. To identify the suitable elements of UX evaluation for educational games, the previous models/frameworks of UX are reviewed. The literature has indicated that many elements can be used for computer games evaluation. However, only six main elements are used by most of the researchers for UX evaluation as presented in Table 1.
Based on the six elements reviewed earlier, all of these elements are identified as suitable for UX evaluation for educational games. Flow, immersion, game usability, game function/system, player context and learnability are added to the proposed conceptual framework. These six elements support each other to find the right balance of the user experience and educational games. The suitable sub-elements of each of the elements are also identified based on the earlier review. The following are the elements and their subelements.

A. Flow
The template Flow state engages a person in a goaldirected, complex challenge merely for the pleasure of dealing with the challenge [11]. As for flow, goal clarity is an important sub-element to achieve the goal of the educational game. Meanwhile, sub-element feedback enables players to understand their status of the game in real time. Sub-element challenge ensures that the players do not lose their interest of the game by providing an appropriate level of challenge according to the player's skill level. Another sub-element under flow is concentration which ensures that the game quickly grabs the players' attention and maintain their focus throughout the game [12]. Additionally, sense of control towards the game is also important to create a fun gameplay experience [13].
Flow is a state that needs to be sustained with supporting sub-elements such as sense of control, sufficient feedback and clear goals to ensure the players are fully immersed in the game's challenges [14]. In educational games, concentration is important to ensure that the players receive the knowledge behind all the gameplay. Thus, clear goals, feedback, playability, control, challenge and concentration are listed under Flow element for this study.

B. Immersion
Immersion is a state that makes players believe that they are in the game content and directly involved in the game world [15]. In addition, [16] have employed grounded theory to explore game immersion and suggested that immersion is involved in three stages: engagement, engrossment, and total immersion, respectively using Game immersion questionnaires (GIQ).
Most immersive gaming experiences were of the engrossing or engaging variety with total immersion reserved for the most intense periods and even restricted to a short period within a longer playing session [17]. The lowest level of involvement is engagement. Engagement is dependent on the gamer's willingness to invest time, attention and effort in the game meanwhile the engrossment level affect players by involving player's emotions. In the last stage, total immersion will cut off the player from reality into the game world [18].
A recent study involving immersion, engagement and flow in game-based learning shows that both the challenge of the game and being skilled in the game had a positive effect on both being engaged and immersed in the game. In this case, challenge which is a sub-element of flow acts as a strong predictor of learning outcomes [19]. However, the study shows that immersion did not have a significant relationship with perceived learning and suggested that different types of game that involve sensory or imaginative immersion can be tested besides challenge-based immersion game. [20] defined the "usability" as the possibility and the ability to have contact with a product in terms of satisfaction, efficiency, learnability, errors and memorability for older adults. Usability plays a significant role to make sure that the user can accomplish the goal of the product effectively and efficiently at an optimal time [21]. One of the objectives of ISO 9241-11 is to ensure that the satisfaction component of usability contains features of user experience [22]. According to [23], usability can be broken down into five sub-elements; Operability, Understandability, Learnability, Attractiveness, and Satisfaction.

C. Game Usability
When an immersive state is achieved, it could help to overcome other usability issues [24]. Besides that, the usability scores may also be positively or negatively affected when a game delivers experiences of immersion and flow [25]. Usability also supports the UX evaluation as good usability, a useful artefact and an engaging task (challenges that the game provides) create conditions for a good educational experience [26]. As for usability, the sub-elements of attractiveness, understandability, satisfaction and operability are included as all of these sub-elements play an important role in supporting the main element accordingly.

D. Game System
Based on the models proposed by [27][28][29], the game experience or player experience involved three layers of interaction: game system, player and context. The quality of the game system is important as player's experience is shaped when a user interacts with the game system. The game system plays an important part to ensure that the game can be played www.ijacsa.thesai.org without any hiccup. A hiccup in gameplay can ruin the concentration of a player thus indirectly affect the immersion and flow experiences towards a game.
Some of the methodologies for assessing game system are soak testing, open beta-testing, localization testing, unit testing, compatibility testing, regression testing, bug tracking, stress testing and gameplay metrics [27]. In a learning environment, a system should be able to support both sides of communication (teacher and students). Therefore, the engine should be able to guide the process and command the games to activate their adaptation mechanisms to fit certain requirements every time they are run [30]. The device and functional level of a game system must be tested by each of the game development team to ensure it is working correctly.

E. Player Context
Player context can be explained as an interaction between the contextual gameplay experience formed and a player in a given temporal, social, spatial context [27]. As for the player context, many features can be measured to evaluate the user experience. User background, time, and culture are some of the important aspects to be considered. Culture reflects the environment that a player resides thus game related to the player's culture or environment can be more reachable. User background can also help the game designer to get some ideas of what the players prefer in a game and the level of the player's gaming skills. A game that relates more to the reality of the players can connect easily with the player. Thus, player's background needs to be identified.
As for time, it is unpredictable how the players will react to a game after a certain period. However, for an educational game, time cannot be considered as reliable since a subject used for the learning process will be acquired for a certain short period only. For example, for computer science students who use an educational game to learn a basic programming language, they will only use that game for that particular period or in certain semester. For the following semester, a new batch of students will join the class and use the same educational game. Besides, different educational game will have different learning objectives (learning content) that will only be used for a limited time. Therefore, sub-element of time can be ignored by this player content element.

F. Learnability
The main idea of a game is to have fun meanwhile learning is always hard, and making people learn through games can ruin the fun [31]. Thus, the right balance of learning and fun should always be the important principles of educational games. If the fun elements control over the learning goals, then the real objective of educational games will fail. Moreover, the aim of an educational game should be related to the learning goals of the game [26].
Without the right amount of educational content in the game, the real objective of the educational game will not be achieved. Hence, it is important to ensure that the learnability element is included in the user experience evaluation for educational games. The sub-elements of knowledge improvement, learning goals and learning content are included in learnability. Knowledge improvement is one of the key points of educational games as the objective of educational games is to ensure that knowledge of a particular subject is improved. It is considered as aspects of an enjoyable experience and an important criterion in evaluating educational games. Learning content and learning goals also support learnability to achieve the goals.

III. METHODOLOGY
Based on the review, the proposed EDUGX is modelled. Table 1 shows the list of all sub elements and the sources to model the EDUGX framework.
Sub elements from EDUGX framework will be used to develop the questionnaire tool (EDUGXQ) for validation and an initial expert validation will be carried out on the framework and questionnaire before the data collection process.

A. Identification Of EDUGX Framework
This UX framework was developed based on six elements which are Flow, Immersion, Player Context, Game System, Game Usability and Learnability. Questionnaires used by previous researchers for evaluation of user experience and the elements involved in each of the questionnaires are listed in Appendix A. A total of seven instruments that are used for educational games evaluation are reviewed. For each instrument, the elements involved are identified and crosschecked with the elements in the proposed EDUGX.

B. EDUGX Questionnaire (EDUGXQ) Development
Based on the review of the existing questionnaires, UGALCO, PLEXQ, GIQ, Game Engagement Questionnaire and Gameflow questionnaires are selected for the development of the EDUGX questionnaire since these questionnaires focused on UX in games. To develop the required questionnaire, the elements involved in the six main elements of EDUGX are picked from the selected questionnaires. As for the game system element which is focused on measurement done by the game development team, a few questions will be added to understand the level of functionality from the user perceptions [27,29]. Once the elements needed for the EDUGX questionnaire development are identified, applicable questions are then selected. This is to avoid long or repeated questions.
Questionnaire related to the six elements on the proposed EDUGX framework was selected with a total of 99 questions. Closed questions with multiple tick boxes, yes/no choices and ranking with 5-point Likert scales were applied as per existing questionnaire for a quicker and easier approach. The selected questionnaire items for the proposed EDUGX questionnaire (EDUGXQ) is listed and EDUGXQ will be finalized once experts' reviews suggestion and comments are obtained.

C. Expert Review
In this section, the proposed EDUGX framework and EDUGX questionnaire are shown to the UX and game industry experts for validation. The framework and questionnaire development will then be finalized. The objective of the expert review is to avoid any big changes to the EDUGX framework or questionnaire after the data collection phase. Once the experts validate the framework and questionnaire, any changes or update will be done according to their suggestion.
An official email request was sent to a total of eight experts (five academicians and three game developers). From the eight experts, four academicians in the field of User Experience and one game developer from the game industry agreed to be in this research expert panel. Following that, their details were taken, and a set of expert review form was sent to them by mail and also face to face meeting, according to their preferences. This expert review form consists of five main sections: a) Introduction: Include demographic and research work information. e) Overall Comments/Suggestions: Experts are required to answer five questions on the framework and a questionnaire based on their opinion.

IV. FINDINGS
Based on the result obtained from the experts, the validating process was done to make changes and discard any irrelevant items. The common comment among the experts was to change the negative statements to positive statements to avoid confusion since it's hard to evaluate negative statements. Other comments were, to make simpler and clearer questions/wordings. Validity ensures that the questions being asked allows for valid inferences to be made. Since this study involves students and educational game, the type of validity is selected based on this category. There are four types of validity in educational research which are criterion-related validity, construct validity, content validity and face validity [42]. This study will use content validity to ensure that the items in the questionnaire that addressed each of EDUGX elements will be evaluated.
Content validity has been defined as the extent to which an instrument has an appropriate sample of factors for the questionnaire being considered [43]. The content validity index (CVI) was used to measure content validity, which is 'based on experts' rating of item relevance [44]. Based on the 4-point scale, the rating of 1 and 2 is considered as not agreed as relevant items and the rating of 3 and 4 is considered as agreed items to be relevant. Thus, an item level CVI (I-CVI) is used to calculate the degree of agreement among the expert panelists.
The I-CVI score was computed for each item by adding the number of experts who rated an item either 3 or 4 and dividing it by 5. This number shows the proportion of experts who agreed that the item is relevant. If all five experts rate an item as either 3 or 4 in relevance, the item CVI will be 1.00. According to [45], CVI value of 0.78 or higher from more than three experts can be considered as good content validity. Thus, for this study, the items with CVI greater than 0.8 were included in the final EDUGXQ. All of the items were listed to identify the CVI value as shown in Appendix B.
From a total of 99 questions, 24 questions were removed from the initial EDUGXQ and sentence structure of the remaining items were also restructured based on feedbacks received from the experts. The revised EDUGXQ contained 75 items of 5-point Likert scale with multiple choices options, categorized under five thematic domains namely (1) Flow; (2) Immersion; (3) Player Content; (4) Game Usability and (5) Learnability. www.ijacsa.thesai.org As for the EDUGX framework, based on the suggestions from the experts, the game system element is combined under game usability thus the number of elements are reduced from six to five. The experts (R1, R4 and R5) suggested that it will be better to have fewer elements to evaluate. Therefore, based on the review and analysis, a proposed EDUGX framework is illustrated in Figure 1.

V. CONCLUSIONS AND FUTURE WORKS
This study was carried out as it is applicable to the present as well as future requirements of suitable educational game industry and contributes the knowledge to user experience evaluation field. As mentioned in this study, user experience plays an important role to make the products of the games more efficient and reliable. By depending on the studies in the literature review and experts' review, an UX evaluation framework of the educational game (EDUGX) which consist of five main elements Flow, Immersion, Player Context, Usability and Learnability was developed. A suitable User Experience (UX) tool for an educational game will ease the game designers work as well as contribute to effective educational games. Thus, through this paper, some reviews are done on previously define User Experience (UX) tool which was based on EDUGX framework elements to identify the most approachable tool to evaluate UX design for educational games (EDUGXQ). As for future work, EDUGXQ will be used in a real experiment to evaluate the UX in an educational game through a process of mixed method research design. www.ijacsa.thesai.org