Study of Gamification Effectiveness in Online e-Learning Systems

—Online distance e-learning systems allow introducing innovative methods in pedagogy, along with studying their effectiveness. Assessing the system effectiveness is based on analyzing the log files to track the studying time, the number of connections, and earned game bonus points. This study is based on an example of the online application for practical foreign language speaking skills training between random users, which select the role of a teacher or a student on their own. The main features of the developed system include pre-defined synchronized teaching and learning materials displayed for both participants, along with user motivation by means of gamification. The actual percentage of successful connects between specifically unmotivated and unfamiliar with each other users was measured. The obtained result can be used for gauging the developed system success and the proposed teaching methodology in general.


INTRODUCTION
The paper describes newly developed open educational resource for learning foreign languages from native speakers, called i2istudy. Learning is achieved by using pre-defined educational materials through live interaction between the users acting as the teacher and the student. This is why the system is called i2istudy, "eye to eye", utilizing the peer-topeer principle, based on the patented technology [1]. This technology allows learning basics of a foreign language from scratch, or enhance foreign language proficiency in a short period of time [2].
Open educational resources (OER) have recently become quite popular in the area of computer assisted language learning [3]. Currently there are several educational services on the market with a considerable amount of OERs that provide an opportunity to learn foreign languages (livemocha.com, www.learn-english-online.org, www.duolingo.com). [4,5,6] Most of these systems are automated, i.e. don't provide live human interactions. These systems can be divided into two categories: autonomous and social. Autonomous methods offer tasks, which are checked or monitored in accordance with the algorithms set up within the system as tests and quizzes, etc. Social methods allow direct or indirect interaction with real people, including communication and checking assignments, etc. (www.facebook.com). Such systems have been used in language learning [7,8,9,10] also attempted to integrate computer-assisted language learning systems into the educational process.
Communication culture is formed as a result of live human speech interactions. Speech interaction is characterized by audio messages exchange between humans. Speech activity consists of the two aspects: language and speech. Together, they transform into the four types of speech activity, combined into two groups: Receptive, perception-oriented types of speech activities, such as reading and listening; Productive types of speech activities, focused on information production, such as speaking and writing.
Verbal means of communication are formed by all kinds of speech activity, which can be further developed and applied during foreign language learning. Thus, communication and training with representatives of other cultures is essential. An important quality for voice communication interaction is the "social" character of the learner, who's open for the dialogue and ready to participate in various discussions and debates. Learning a foreign language is also associated with acquiring the knowledge of other cultures, which is impossible without speech communication and knowledge of the linguistic and cultural features. Necessary properties of applications, which can provide language communication practice, are:  The possibility of audio or visual contact, chosen by the training participants;  Teaching methods, including conversation scenarios, allowing participants to actually start a conversation and keep it within the specified time on a given topic;  Motivation of the participants.
Skype is currently the most common and popular tool for distance learning of foreign languages. While Skype was not (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 6, No. 2, 2015 72 | P a g e www.ijacsa.thesai.org specifically designed for this purpose, it provides live audio and video connection between the participants. Thousands of small agencies and individuals offer distance language learning through Skype. The query for "English via Skype" in Google.com gives over 43 million results, with similar numbers when searched in Spanish, Russian and other languages. However, Skype does not allow finding people willing to teach or learn languages. It does not provide teaching or study materials, and does not track the time spent teaching or learning foreign languages. [11] However, Skype and other systems of cooperative joint learning provide invaluable engagement, which plays a crucial role in learning. As noted by Clark and Mayer [12]: "all learning requires engagement", regardless of the delivery media. Zhang et al., [13] also suggested that increased student engagement can improve learning outcomes, such as problem solving and critical thinking skills. In the review article, Fredricks, Blumenfeld and Paris [14], defined engagement by its multifaceted nature: "Behavioural engagement draws on the idea of participation; it includes involvement in academic and social or extracurricular activities. Emotional engagement encompasses positive and negative reactions to teachers, classmates, academics, and school and is presumed to create ties to an object and influence willingness to do the work. Finally, cognitive engagement draws on the idea of mental investment; it incorporates thoughtfulness and willingness to exert the effort necessary to comprehend complex ideas and master difficult skills." Fredricks, Blumenfeld and Paris [14], also claimed that the focus on behavior, emotion, and cognition, within the concept of engagement, may provide a richer characterization of learning. The authors reminded that a robust body of research addresses each of the components separately, but pointed out these dimensions of engagement had not been studied in conjunction. Thus, emotions aid communication process substantially [15,16].
Computer training system was developed, called i2istudy, in the form of a game to implement all three necessary components of spoken communication for training foreign language skills. This game consists of the computer-aided casual conversation with native speakers. For example, English-speaking users can learn Spanish from Spanishspeaking users, and visa versa. The i2istudy allows native speakers to teach others without knowing how to teach and without knowing foreign languages. In other words, i2istudy allows all native speakers, not necessarily professional teachers, to teach their native language in a collegial network game setting [17,18].
The main feature of the system consists of providing a common space with educational materials, including specifically designed lesson plans, which are simple and understandable step-by-step educational materials aiding communication. The platform, which allows live audio-video communication, is built into the web interface, based on the popular Web real time communications (WebRTC) technology.
Motivation is achieved by attracting a large number of users available online at the same time, always allowing to find a companion, along with gamification. Gamification is based on utilizing game elements in design and motivation principles in non-game situations [19]. It this case it is necessary to stimulate users to spend more time in the system to achieve the needed quantity and volume of practical skills, based on the modern principles realized in the e-learning systems [20]. The users should also be motivated to return to the system on the regular basis, the so called retention cycle [21]. The developed application has the following gamification methods: 1) Time banking. When user acts as a student by taking lessons, virtual system currency in minutes is spent from the user account. One minute of learning is debited from the account, while one minute of teaching is credited to the account. Thus, the user acting as a teacher earns minutes, and the same user spends minutes as a student. In this way all users participate in the virtual economy. Users are motivated to earn minutes, pushing the user to periodically assume the role of a teacher. Each user gets 30 minutes in the system as a part of the registration process. If all minutes are spent in the account, the system does not allow to study, but offers to teach to earn more minutes. (Accumulated minutes are shown in figure 1) The implemented time banking goal is to motivate users to teach in addition to learning [22,23].
2) Sequential lessons presentation. Most computer games utilize this gamification principle when the next game level becomes available after previous level has been completed. www.ijacsa.thesai.org For example, Figure 3 shows all lessons, but only a limited number of them is available. New lessons become available as the user goes through the previous lessons. Moreover, there is a grade displayed for each passed lesson as a single, dual or triple star, reflecting how well the student passed the test at the end of each lesson. Sequential opening of the lessons in batches intrigues the user to find out what's coming next, and boosts user engagement. Besides, explicit visibility of the grade encourages user to retake lessons with poor grades.
3) Achievements and badges. Figures 1 and 2 show user "achievements and badges". The user acquires nominal status, presented as an achievement, for learning and teaching in the system. The user gets status notifications by email, while other users also see these "achievements and badges", and can select their learning partner based on this information. Basic list of "achievements and badges" includes: "The First-grader; The Middle-schooler; Highschool student". For short these are presented by the first two letters of the achievement, displayed in the corresponding language next to the user name, and are called badges. Shortened badges are used to save the space in the list, and will be replaced with medals in the future for better visibility. The goal here is to motivate users to receive awards as an external evaluation, thus motivating users to come back  (Figures 1 and 2). Additionally, there is an option to report indecent user behavior to the system moderator. However, this option is a part of system moderation, rather than gamification.

II. RESEARCH RESULTS AND DISCUSSION
It is necessary to evaluate the effectiveness of the mechanisms implemented in the application designed to improve verbal communication skills. Assessment is based on measuring:  The increase in the number of application users, allowing to estimate the demand for service and implemented principles;  The number of users willing to "teach", allowing to estimate the required number of people willing to teach and indirectly assess the suitability of the developed scenarios and gamification means of motivation;  The time spent in the application as a parameter to a large extent characterizing the main development goal -the ability of users to establish long-term audio-video communication to obtain practical foreign language communication skills. The main research objective was to determine whether unfamiliar with each other www.ijacsa.thesai.org people, one acting as the "teacher", and the other acting as the "student", can take lessons together following a given scenario provided by the platform.
All experiments were conducted in the system with 40,000 registered users and 1,000-1,500 daily active users (DAU).
One of the main assumptions was whether unfamiliar with each other people, who met in the system for the first time, could communicate and to learn foreign languages together. The research objective was to find out if specifically unmotivated individuals without special training could choose a "teacher" or a "student" among the users currently available online in the system, send an invitation to study or to learn, establish audio-video connection and follow provided lesson scenario using the WebRTC face-to-face communication.
Initial system users, interested in practicing foreign language skills, registered in the system as a result of advertising in the Facebook social network. The ad suggested registering online and learning foreign languages for free, in exchange for teaching native language. This ad was displayed in Spanish and English-speaking countries, Germany and Russia. Besides, some users registered as a result of existing user invitations (users invited by the existing users). There was no additional information provided about the system, no verbal commentaries, or explanations were provided to the participants.
As a result, 39,729 users registered in the system in 6 months. 28,180 users indicated that they want to learn English, 8 After registration users were asked to select a role of a teacher or a student and locate a potential learning partner to engage in the dialogue, based on the step-by-step methodology presented by the system. Each phrase is presented to the student with the corresponding prompts and translation if needed, and to the teacher with corresponding comments to aid the teaching process. The users see and hear each other in real time, while working with the synchronized teaching materials, and can type messages to each other in chat [24,25].
Users could see other users in the system, along with their attributes, such as native language, the country of residence, the role of a "teacher" or a "student" in the system, along with the button to initiate a joint study session.
If a "teacher" accepts "student's" request, or visa versa, audio-video connection is established, where both participants can see and hear each other, along with the common filed with the synchronized study materials with the corresponding prompts for the teacher and the student. Besides, the system tracks the connection time for billing purposes in game currency, as seen in Figure 4.
About 20% of all 40,000 registered users participated in the experiment. The rest were shy to speak with strangers, or decided not to spend their time. Some users failed to configure their microphone and the web camera needed for the real time audio-video connection, or their browser did not support Web RTC [26] www.ijacsa.thesai.org As a result of the conducted experiments it was established that two users, previously unfamiliar with each other, and met in the developed application for the first time could carry on a conversation following the suggested scenario, helping each other to learn foreign languages. Moreover, some users did not have a common language to use for communication. Average connection time was 11.94 min (189,207 min or 3,153 hours), divided by the number of successful connections (15,842). Any kind of interaction interruption was taken into account, including closing the browser or turning off the computer, or successfully finishing the lesson materials. Table 3 shows the number of successful connections and the connection duration in min. Regardless of the fact that the average connection time is not very long, the experiment showed that two unfamiliar and unprepared users can carry on a conversation in a foreign language for quite long. Besides, the average connection time continued to increase with the number of registered users, and reached 14.35 min in August 2014. Moreover, the most loyal and active users became apparent, spending hours learning and teaching, and even repeating the same lessons. Table 4 shows the most active users, along with the time spent learning or teaching in min. The users registered as a result of advertising and conducted lessons either as a teacher or a student, learned the system interface on their own, without any special training. There were users were not specifically recruited to conduct initial proof of concept experiments. The users accepted roles or teacher and student. The numbers of both types of user roles are listed in Table 5. The corresponding ratio of 6.4 "teacher" users to 10.6 "student" users indicates that an average user is not afraid to play the role of a teacher. Thus, the following experimental results were obtained: The idea and the proposed form of training, based on the developed platform and teaching methods are in demand, indicated by the growth dynamics in terms of the number of users without motivational advertising.
Developed communication scenarios and lessons allows to overcome psychological barriers of communicating with strangers, both as a "student" and as a "teacher". A significant percentage of users who want to act as teachers for the proposed method is revealed. This means that teaching staff is not required.
The introduced gamification adequately motivates users. A significant percentage of users have been returning to the application for further studies, demonstrating the effectiveness of developed tools and ideas.

III. CONCLUSIONS
Conducted study presents the new methodology to assess gamification tools in the e-learning systems. The e-learning system not only allows to conduct quality training, but presents and opportunity for statistical analysis of different parameters, contained in the log files, to assess the effectiveness of technical and pedagogical tools. Application popularity with users and grows of the number of users both act as assessment for the system motivation elements and tools.
Developed application is of interest for the majority of users, and allows to maintain a prolonged dialogue between the users in a given language. This definitely allows developing speech communication skills in a foreign language. Regardless of the stereotype that quality foreign language education can only be provided by the professional teacher, the developed system demonstrates that it is also convenient for users to study together. In this case, similar to the teaching materials presented in a text book, or interactive recorded media, professional teacher is recruited to develop teaching materials, while users can use these materials for training and practice. However, it is more interesting and encouraging doing this with other users, since the social effect also gets employed. Based on the conducted experiments, users not only spend more time in the system, but invite their friends to join them.