Design and Evaluation of an Engagement Framework for e-Learning Gamification

Recently, gamification in education software development to improve student engagement and performance has become prevalent. Gamification is used to counter attrition and dropout issues in e-learning. A handful of methods are presented for the gamification of e-learning systems in the literature. However, the e-learning gamification methods proposed in the literature lacked consistency. The number and types of game elements used in the methods are varied. In addition, there is a lack of an engagement framework that can be used in applying game elements to e-learning systems. Therefore, this paper provides insights into gamification and how it is used in e-learning systems. Then, the study proposes and evaluates an engagement framework that can be used to guide developers on how to add game elements to e-learning to improve student engagement and performance. The framework consists of three components: game elements, learning activities, and engagement factors components. Two experts evaluate the engagement framework via a semi-structured interview. The evaluation results indicate that developers can efficiently and effectively use the framework to gamify e-learning systems for improved student engagement and performance. Keywords—e-learning; learning activities; gamification; game elements; engagement framework


I. INTRODUCTION
Student engagement plays an essential role in minimising student attrition and dropout rates. However, it also represents one of the main challenges in teaching and learning, especially in online education or e-learning. The level of engagement is influenced not only by how lessons are delivered but by the students' readiness and enthusiasm to engage with their studies. Hence sustaining student engagement is a daily struggle that leaves higher education lecturers in a quandary because the students' energy into immersing themselves in their courses plays a key role in their engagement [1].
Several studies have proposed the use of gamification to improve student engagement [2,3]. Gamification adds game elements to e-learning systems to engage students and achieve the desired learning behaviours and outcomes [4,5]. The researchers in the domain have proposed a wide array of gamification guidelines in different research areas. Most of the guidelines provide strategies for gamifying other systems, including business systems like Facebook, Twitter, etc. [6,7]. Likewise, gamification strategies proposed in education intends to improve student performance, student outcome, or student engagement. Yet, the strategy adds game elements to educational systems in an ad-hoc manner. The wide variety of e-learning tools utilised in education, such as Moodle, Adaptweb, etc., makes the gamification process vague and unclear to developers [7,8].
However, the literature has not adequately investigated an engagement framework that can help educational software developers gamify e-learning systems. Therefore, this paper aims to propose an engagement framework that can address the lack of methods or standards for the gamification of e-learning systems. The gamified e-learning system ensures students become engaged in their learning, which will indirectly lead to minimising the number of dropouts and attrition, which is a suffered by most higher education institutions (HEIs).
Subsequently, the framework is evaluated by submitting the framework and a document that explains the usage to experts. The experts assessed the selected game elements and their completeness. The experts were asked to suggest any additional game elements apart from those commonly used elements selected by the authors. Furthermore, the experts were asked to comment on the learning activities included in the framework for the gamification of e-learning systems. In this light, the experts were requested to evaluate the framework for its usability and whether the mapping between the game elements, learning activities, and engagement factors are easy to understand and can improve student engagement in elearning systems. The overall result of the evaluation was positive. They agreed with the authors that the selected game elements were enough and needed no additional elements. Similarly, the experts approved that the learning activities were enough and could represent learning activities required for elearning systems. The experts, as a result, agreed with the usability of the proposed engagement framework of the gamification in e-learning systems to improve engagement of higher education students.
The rest of the paper is organised as follows: Section II provides an overview of related work, Section III briefly describes learning management systems (LMSs), Section IV explains the reasons for attrition and dropouts in learning, Section V introduces the proposed engagement framework, Sections VI evaluates the framework, and VII concludes the paper.

II. RELATED WORK
Based on the literature on gamification in education, several studies have employed game elements in e-learning systems for various purposes. Some have used gamification to improve student outcomes (performance), while others have utilised gamification to improve student engagement [1,9,10]. Given the scope of the current research, only those works that focus on gamification in e-learning systems for student engagement are reviewed here.
In [11], the researchers proposed a model for use in the gamification of e-learning systems for better user engagement. Several commonly used game elements were selected in gamifying e-learning systems. However, the researchers did not implement the model to investigate whether it improves user engagement.
The researchers in [12] also proposed an e-learning gamification model, but they further used it in the gamification of a newly developed e-learning platform. The model was validated via experimentation, where a group of students used the platform. Based on the data collected on the students' experiences with the gamified platform, the researchers were able to realise the extent to which the model was able to improve student engagement.
In another study [13], the researchers investigated the influence that a gamified learning environment may have on student learning achievements when used in conjunction with social media. The experiment showed that there was improved user engagement. However, in this study, the employment of gamification did not involve the use of an e-learning platform.
In [14], the researchers investigated which game elements could improve user engagement when added to LMSs. In addition, they explored the effects that the deployment of gamified e-learning platforms may have on users. In [15], the researchers assessed the level of user engagement when users in higher education institutions used a gamified environment. However, the researchers failed to identify which game elements improved which engagement factors. Moreover, they could not determine which game element needed to be used in which e-learning activity for improved user engagement.
In [16], the researchers explained the influence of gamification in online discussion systems on user engagement. The results showed that gamified online discussion platforms have a positive impact on user engagement. Furthermore, they also revealed that some technical issues and classmate behaviours were impediments to user engagement.
In another study [17], the researchers examined the impact of gamified e-assessments on user engagement. The researchers studied the effect of gamification on one learning activity, such as discussion and assessment, respectively. In other words, learning activities such as assignments and learning material were not studied in the research.

III. LEARNING MANAGEMENT SYSTEM
In the context of HEIs, an LMS can be defined as "a serverbased or cloud-based software program containing information about users, course and content which provides a place to learn and teach without depending on the time and space boundaries" [22]. An LMS can also include applications and web-based technologies that can be utilised by HEIs and students to access, plan, implement, supplement, monitor, and assess learning [23]. Furthermore, an LMS can provide a discussion platform to facilitate communication among the students [23]. Different LMSs are commercially available, some of the most well-known are Moodle, Canvas, Blackboard, and Desire2learn. These LMSs may employ standard components for implementing learning activities. Learning management systems can hold course contents, provide discussion and chat forums, administer e-quizzes and e-assignments, create activity logs, summarise grades, to schedule, provide calendar reminders, and store multimedia files.
Hence, it can be concluded that LMSs can play a significant role in creating, distributing, tracking, and managing the learning activities widely used in HEIs to provide an innovative technology-based teaching and learning experience. In addition, LMS platforms are believed to have the potential to expand reach, reduce cost, and improve the quality of education, and thereby help HEIs to meet the demands of a growing student population. Thus, LMSs can be considered to be one of the numerous positive outcomes of digitising conventional campus teaching and learning functions.
However, despite the growing use of LMS solutions in higher education, researchers have also found some disadvantages associated with their service. Among these, lack of user engagement, poor intention to use, and low satisfaction are the major drawbacks cited in the literature. Other issues raised concerning LMS usage include ease of use, usage behaviour, expected performance, and attitude towards usage. These are essential factors that require the immediate attention of the research community. They have not been exhausted enough in terms of how they relate to the user performance and impediments that need to be handled to improve LMS usage. www.ijacsa.thesai.org

IV. ATTRITION AND DROPOUTS IN E-LEARNING
The attrition rate in HEIs has become a noticeable problem since e-learning systems have been adopted. Although there is no concrete evidence as yet, according to the available literature, there are strong indications from the higher education sector that student attrition is higher because of elearning systems [3,24]. Also, somewhat ironically, although many institutions have implemented e-learning to meet learners' needs, a large percentage of learners do not complete e-learning courses [3,24].
Some studies have taken time on covering various studies that have looked at the problem of high attrition in e-learning [25,26]. Such studies have identified that the factors that cause attrition in e-learning are related to the students' characteristics and the design of the e-learning system. To date, the literature has focused primarily on investigating the student-related factors [26,27], and less research has been dedicated to assessing whether there is a link between attrition and dropout rates and e-learning system designs. Moreover, the studies that do exist do not provide any overarching solutions [26,27].
Nevertheless, it remains crucial to develop strategies to reduce the attrition rate in e-learning from an economic and a quality perspective. High attrition rates harm the financial performance and academic reputation of HEIs. Therefore, some studies have proposed student-centred learning as a solution for reducing attrition. In [28], the researchers studied the relationship between poor course design in e-learning and attrition. The results of their analysis indicated that course workload was a significant barrier and caused student dropout. In another study [29], the same researchers highlighted several strategies that could be used to overcome attrition. The learning systems should include good course design with greater flexibility to provide activities, structured course formats, time management support, syllabus quizzes, a feedback strategy, and collaboration between students and lecturers.
The technical aspect is believed to help the readiness of a student to accept online learning can improve their success in e-learning. For example, in a previous study [30], it was found that student success in online programs depends on the level of student engagement with information and communication technologies.
It can be argued that student success in e-learning has more to do with the student's relationship with technology. Technology's role in a student's life can determine how engaged they will behave in an e-learning environment [31]. It is essential for gamification comes into play. The researchers in [32] argued that gamified e-learning courses might build confidence and engage students in a course if they were familiar with the technology. Therefore, in the following paragraphs, student engagement factors are discussed.
In the literature, little has been said about the theoretical connections between gamification and engagement. However, in her book, "Reality is Broken", Jane McGonical has argued that games are not just for entertainment; but rather, the skills developed during games can be useful for solving real-life problems [33]. In other words, gamification can promote the engagement of users and help them to solve problems. For example, a study on gamification in the business domain showed that customers became engaged by combining the power of games with business strategies [34]. Moreover, in their book, "Total Engagement", Leighton and Baron explored the idea of using gamification to improve enthusiasm for work among employees [35]. That is to say that gamification can be used to engage people in the workplace. Furthermore, in the context of education, the authors of 24 peer-reviewed empirical studies on gamification found that most of the investigations yielded positive results when they applied gamification to improve learners' engagement [36].

V. PROPOSED ENGAGEMENT FRAMEWORK
In the literature, studies have used different game elements in e-learning for student engagement [37]. Hence, there is no consistency or consensus on which type of game element influences student engagement. However, from the literature review, several game elements are commonly used in such studies. As shown in Table II, these game elements are points, progress bars, levels, badges, Leaderboards, dashboards, content unlocking, teams, avatars, and timers. These ten game elements are therefore used in the proposed engagement framework. Descriptions of the selected game elements are provided in Table III.
The literature also shows that different learning activities in e-learning systems have been gamified for various purposes [24,38,39]. e-Learning systems offer several learning activities, but some are less frequently used in LMSs. Therefore, only those learning activities that may directly impact student engagement have been included in the proposed framework, namely, learning materials, assessment, assignment, and discussion.

Game element Description
Points Points indicate the level of student achievement in various learning activities (e.g., quizzes, examinations, assignments, and discussion fora in e-learning systems [35].

Levels
Levels usually correspond to different modules or chapters of a course composed of activities such as course material, assignments, and assessments.

Badges
Virtual badges are given to students when they have completed specific activities, such as quizzes, assignments, and examinations.

Leaderboard
The Leaderboard is shown at the system level in a scoreboard that displays students' results based on the number of points and badges earned [11,71].

Dashboard
The dashboard provides almost immediate feedback by presenting a summary of all the activities that have been completed by the student and those that the student has not yet completed. It also helps the student identify his or her expected performance outcomes by displaying analyses of the activities they have done on the e-learning system.

Progress bar
The function of the progress bar is to track and display how much progress a student has made in an e-learning activity. Based on the proposed engagement framework, almost all learning activities are expected to have a specific progress bar.

Avatar
When a student uses an avatar, it can make them feel safer as it helps to maintain privacy by hiding their identity and activities from others.

Teams
Teams are associated with specific assignments, and teams that complete their tasks on time are rewarded with badges, points etc.

Content unlocking
Content unlocking refers to moving up a level, where students proceed to a new level when they complete predetermined requirements.

Timer
The timer is used for quizzes and examinations and counts down the hours, minutes and seconds until the time limit that has been set for the completion of quizzes and examinations has been reached.
The learning materials activity refers to all the courserelated documents submitted to and distributed by teachers in the e-learning system. The assessment activity involves using all of the exercises created by teachers for students' practical work and examinations and quizzes that are administered to test the students' level of knowledge acquisition, which are delivered through the e-learning platform [17]. The assignment activity allows students to submit assignments directly into the e-learning system [40]. Finally, the discussion activity in an elearning platform enables students to engage in higher-order thinking and active learning and to have a social presence outside of the classroom environment [41].
Several engagement factors have been studied in the literature on gamification [17,42,43]. Behavioural, emotional, and cognitive factors were selected for inclusion in the proposed engagement framework. The main reason for choosing these three factors is that they are all widely used in the gamification of e-learning systems to investigate whether there is improvement in student engagement. There are also specific indicators that need to be used when studying factors [17,43]. The factors and their corresponding indicators are shown in Table IV. The proposed engagement framework is shown in Fig. 1. The framework consists of three elements or gamification components, learning activities, and student engagement factors. The game elements that influence certain learning activities are grouped and mapped to those activities.

VI. EVALUATION OF THE FRAMEWORK
The proposed framework underwent a process of validation and evaluation to determine whether it could solve the issue of the lack of a standard for the gamification of e-learning systems. Generally, an assessment of a framework involves two activities: 1) Conducting a formative evaluation to assess the framework in design and development stage to find ways of improving it. 2) Conducting a summative evaluation to assess the framework after a prototype has been developed. This study uses the former type.
The evaluation was based on the views of experts. Gathering such views is necessary because it enables the researcher to draw upon knowledge that could not have been obtained otherwise through an analysis of the literature. Several evaluation methods have been used by previous studies that have proposed artefacts for the information systems. For this study, it was considered that the semi-structured interview method was the best option for the evaluation, which is in line with several other studies [44].
As the framework's design was based on an analysis of previous studies, it might be considered as self-reflection. So, to remove subjectivity from the process, a semi-structured interview was conducted. The experts who participated in the evaluation were selected based on their experience in teaching and the use of e-learning systems. Experts with a minimum of www.ijacsa.thesai.org 10 years of teaching experience volunteered to participate in the evaluation. At the time of the interview, they were practising lecturers in higher education. The interview questionnaire was administered to the experts using Google Forms.
The game elements and learning activities in the framework and the engagement indicators selected to measure student engagement were evaluated by the experts for their suitability and completeness. The questions related to these aspects were as follows: Q1. Do you agree that the customisation of the game elements for specific e-learning activities in the engagement framework will increase student engagement when using the elearning platform?
Q2. Are there any missing game elements that you think would be important for gamifying the e-learning activities platform to increase student engagement? If yes, please state the game elements.
Q3. Are there any missing e-learning activities you think would be necessary for e-learning course platforms in higher education? If yes, please state the e-learning course activities.
Based on the literature, three engagement factors and their associated indicators for measuring the level of student engagement were included in the proposed framework:  Behavioural: participation, persistence, collaboration, independent learning.
The question that was posed to the experts in respect of these indicators was as follows: Q4. Are these indicators adequate to measure student engagement in the e-learning activities platform?
Based on their feedback, overall, the evaluation of the experts was positive. Both experts (EX1 and EX2) unanimously agreed on the suitability of the game elements that were mapped to each of the learning activities in the framework and that these game elements would improve students' engagement in e-learning systems. EX1 also commented that customisation allows students to collaborate more friendly. Regarding the game elements' completeness, both experts indicated that the identified elements were enough to engage students in e-learning systems. Also, commenting on the completeness of the e-learning activities, both experts agreed that there were no missing activities and enough for the e-learning environment. Lastly, the experts agreed that the engagement indicators provided in the framework were adequate for measuring student engagement. Furthermore, EX1 commented that the indicators satisfied the Saudi Arabia national qualification framework.
The framework was also evaluated for its limitations, usefulness, ease of use, and clarity for utilisation in the gamification of e-learning systems. The questions that were posed in this regard were as follows: Q5. From an expert evaluation perspective, are there any limitations to the framework? If any, please list the restrictions.
Q6. On a scale of 1 (worst) to 5 (best), how would you rate the framework's usefulness?
Q7. On a scale of 1 (worst) to 5 (best), how would you rate the ease of use of the framework?
Q8. On a scale of 1 (worst) to 5 (best), how would you rate the clarity of the framework?
Regarding Q5, which asked about the framework's limitations, the experts agreed that there were none. Likewise, to optimise the evaluation result of the framework, the output of the analysis of the usefulness, easiness of use, and clarity is shown in Table V.

VII. DISCUSSION
The findings of the evaluation of the framework indicate that the experts are satisfied with the components of the framework and the elements and learning activities provided. The findings suggest that the framework is easy and clear to follow by e-learning developers when gamifying e-learning systems. The study showed that the gamified e-learning system using the framework is enough to improve student engagement. The result is in line with numerous studies, as reviewed in the related work, showing improvements in student engagement through gamification.
Moreover, this study reveals that engagement factors proposed in the framework, as commented by the experts, can be used to investigate whether a gamified e-learning system developed with the help of the framework can improve student engagement.

VIII. CONCLUSION
This paper proposed an engagement framework that can be used to develop gamified e-learning systems. The framework consists of three components: game elements, learning activities and engagement factors. The framework uses commonly used game elements and learning activities because it may become cumbersome for developers if more elements and learning activities are employed. The expert evaluation of the proposed framework showed that the framework is useful, easy to follow, and provides clear steps for the gamification of e-learning systems. Therefore, this framework is a value add for developers to stand as a reference that can be used for the gamification of e-learning systems.
In the future, an e-learning system will be developed where the selected game element will be added to it as per the directions and recommendations are given in the framework. Consequently, an experiment will be conducted where the gamified e-learning platform is exposed to a group of students to study their level of engagement.