Evaluating Learning Management System based on PACMAD Usability Model: Brighten Mobile Application

—During the pre-COVID-19 pandemic, mobile learning is just an optional or a supplementary module in learning process. However, when the pandemic hit the world in the middle of 2020, a large number of students were forced to move from traditional learning process to online learning. This has become a critical issue especially for new online learners. Usability of a mobile learning application is important in ensuring that learners are able to learn efficiently and effectively with ease. This study evaluates the usability of the Brighten mobile application; a Moodle-based Learning Management System (LMS) which is currently used by all Universiti Tenaga Nasional’s students. The evaluation is based on People at the Center of Mobile Application Development (PACMAD). The results indicate that Brighten mobile application is acceptable in terms of usability’s effectiveness, efficiency, learnability, memorability and error-tolerance. Learners’ satisfaction level shows a “marginally acceptable” result based on the SUS Adjective Rating Scale and the result for cognitive load shows that the highest cognitive load was in terms of the performance factor.


I. INTRODUCTION
Mobile learning is the process of learning that allows learners to obtain learning materials anytime and anywhere, using mobile devices, such as mobile phones and tablets. Before the COVID-19 pandemic hits the world in 2020, mobile learning has been used as complimentary learning resources for the traditional in-class teaching [1]. However, the pandemic forced most of the students across the world to rely more on online learning. This can be seen in the increased number of online learning users in Malaysia up from 9.5% to 20.8% in 2020 [2]. In 2021, mobile learning has been seen as an increasingly popular learning method as is it able to improve and make learning easier for students around the world [3].
One of the methods of applying mobile learning technique is through a Learning Management System (LMS). LMS is an application that is used for administering e-learning practices, i.e. planning, implementing and accessing the learning and development programs [4]. One of the most widely used LMS is the Modular Object-Oriented Developmental Learning Environment (Moodle), which is an open-source software that allow personalization of its learning environment [5]. Brighten mobile application is a customized Moodle for students of Universiti Tenaga Nasional (UNITEN) [6]. It is widely used for online and blended learning in UNITEN. During the peak of COVID-19 pandemic, Brighten has become the main source of learning delivery process in UNITEN. Usability of a system is when the system can be used by it intended users, in a specified context of use, to achieve goals effectively, efficiently and satisfyingly [7]. Usability testing of applications on mobile device needs to consider challenges such as small screen size [8] [9][10], restricted input [11] and design issues [12] [13]. People At the Center of Mobile Application Development (PACMAD) Usability Model is a usability model that is developed specifically for measuring mobile application performance based on the seven usability attributes by Harisson, Flood and Duce [14]. It takes into consideration on attributes, which are normally neglected by the other usability models when applied to mobile devices. PACMAD focuses on the effectiveness, efficiency, satisfaction, learnability, memorability, errors and cognitive load factors.
Despite LMS being widely used since 1960s, there are a lot of problems in terms of its usability. These usability problems include inconsistency in design, issues with navigational links and search functions, inappropriate contents and difficult to be use [15]. These issues are very crucial as they might interrupt the effective process of knowledge transfer [16].
The objective of this research is to evaluate the usability of Brighten mobile application using PACMAD usability model. In order to evaluate the usability of the Brighten mobile application, the research questions can be divided into three parts, which are: RQ1: What is the effectiveness, efficiency, learnability, memorability and errors of the Brighten Mobile Application? RQ2: What is the satisfaction level of the Brighten Mobile Application's user after using the application? RQ3: What is the cognitive load of the Brighten Mobile Application's user while using the application?
The remainder of this paper is structured as follows: Section 2 discusses relevant works related to mobile learning and usability testing. This is then followed by Section 3, which explains the details of the methodology used in the experiment. Section 4 discusses the result of the experiment. Finally, in Section 5 provides the conclusions and suggestions for future work.
Yayasan Canselor UNITEN (YCU) Research Grant 2021 www.ijacsa.thesai.org II. LITERATURE REVIEW Mobile learning or sometimes known as m-learning is simply defined as a learning process, in terms of pedagogy and education [17], that takes place through mobile devices. Using the mobile technology, mobile learning can be accessed from any location at any time [18] . Ozdamli and Cavus mentioned that this learning method should be ubiquitous, portable, blended, private, interactive, collaborative and instant [19].
Mobile learning is performed through a mobile phone, a tablet, a Personal Digital Assistance (PDA), an iPod, a palmtop or any special ubiquitous handheld devices. Although laptops and notebooks are portable devices, they are not considered as a mobile learning device as they are much bigger and heavier [20].
Mobile learning is applied through mobile medium such as SMS/MMS, email, message boards, forums, blogs and video conferencing. El-Sofany and El-Haggar divided these teaching tools into three categories, which are social networks (such as Facebook, Twitter, Blogs and Youtube EDU), web-based platform (such as Rapid Cycle Evaluation Coach, TED-ed and Moodle) and Internet of Things (such as Smart classroom environment device, attendance system and real-time feedback on lecture quality) [21].
As mobile learning is the extension of e-learning, it inherits all the advantages of e-learning such as supporting distance learning and enhancing student-centered learning [22]. Visual learners gain benefits from mobile learning as compared to learning through textbooks. There are improvements in the communication process between teachers and learners through mobile learning environment. Learners can control their learning process and pace. This leads to efficient learning [23] and positively motivates them to learn [24]. LMS is a system that enables educators to administer, build, track, maintain, update and report information related to a learning program [25] [26]. It supports online and offline discussions, formative and summative evaluations and practical-related contents [27]. It assists the learning process in an e-learning environment and can be divided into two (2) types; proprietary LMS and open source LMS [26]. Moodle, Open edX and Chamilo are among the most popular open source LMS [28]. Among the advantage of Moodle compared to the other system is that it can be used by users from different platforms (Windows, Mac, UNIX and Linux) without modification. It supports learning management activity through learning materials, videos, discussion and forums, chat and assessments [26].
Brighten is a customized Moodle that is currently being used in Universiti Tenaga Nasional, Malaysia. It was first implemented in June 2021 to replace the earlier Moodle system. The customization of Moodle into Brighten was done after an informal study conducted on the data usage and data redundancy of the -older‖ Moodle LMS. The main interface of the Brighten application is shown in Fig. 1.
Based on International Organization for Standardization (ISO), the evaluation of a system's effectiveness, efficiency and satisfaction determines the usability of a system [30]. ISO also mentioned that there are three (3) factors that need to be considered when evaluating the usability of a system; user, goal and context of use. Nielsen's usability model consists of efficiency, satisfaction, learnability, memorability and errors [31].
Despite both of the usability models being widely used, it does not fulfill the context of used when it comes to mobile applications. Hence, PACMAD was introduced to overcome the limitation of the common usability models. PACMAD looks into the context of use for mobile application, such as mobile context, connectivity issues, small screen size, different display resolution, limited processing capability and power and different data entry method [14]. It combines the attributes from both ISO and Nielsen's usability model and it is designed specifically for mobile application [32]. PACMAD has seven attributes. These attributes are efficiency, effectiveness, learnability, memorability, error, satisfaction and cognitive load. This usability model includes cognitive load as one of the measuring attributes as its main contribution [33]. These attributes are very important for applications that run on mobile devices as mobile devices have different task setting and size limitation compared to desktop PC [34].
Based on these studies, we can conclude that PACMAD is usability model that is created specifically for mobile application. It evaluates the cognitive load of a user. Cognitive load is among the most important factor, which needs to be focused upon when knowledge transfer process takes place. In the current situation where some of the learning process needed to be done online, it is very crucial to have a mobile learning application that can assists in teaching, and not a burden to learners. As such, based on the results of this usability testing, the Brighten mobile application can be further improved.   Fig. 2.
The execution of the experiment was done online, through MS-Teams. This was done online, because at that period Malaysia was under the Movement Control Order (MCO) due to COVID-19 pandemic. MCO restricted the mobility of the participants as well as the researchers.

A. Participants
77 participants were involved in the usability testing. However, 3 of the students pulls out in the middle of the experiment. As such their data are not counted in the result.
All participants are undergraduate students from six different bachelor degree programs (at UNITEN), taking Human Computer Interaction (HCI) subject. Age group, gender, program of study, year of study, mobile OS are the demographics characteristics that have been included in this study as shown in Table I. From 77 participants, more than half of the participants (54%) are between 21 to 23 years old, most of them are male (61%) and almost 80% of the participants are second year students. 61% of the participants are Android users.
Participants were asked on their daily frequency mobile phone usage. A large group of the participants spent between 6 to 12 hours using mobile phone daily as shown in Fig. 3.
Generally, the participants used their mobile phone for communication (WhatsApp, Telegram, Line and Signal), social media (Facebook, Instagram, Twitter and TikTok), mobile learning (Brighten and MS-Teams) and playing games as shown in Fig. 4.
The average students' spending time on mobile learning is shown in Fig. 5. Most participants spent between 1 to 3 hours on mobile phone utilizing mobile learning where they attended lectures, read and download notes, doing quizzes and assignments.

B. Result
The results obtained are based on all the seven elements in PACMAD effectiveness, efficiency, learnability, memorability, error-tolerance, satisfaction and cognitive load.

1) Effectiveness:
The result of effectiveness depends on the ability of the participants to complete the tasks given. Table II shows that 98% of the participants managed to complete the tasks in Round 1, 100% in Round 2 and 99% in Round 3. Some of the participants failed in completing the tasks in Round 1 and Round 3 which is either due to network problem or system error that leads to failure in downloading the files.
2) Efficiency: Efficiency is calculated based on the speed and accuracy of the participant in completing the tasks measured using the task completion time, as indicated in Table III. It shows the task per second calculation that demonstrate the number of tasks completed in a second. Overall, the efficiency of the system increases with the number of rounds.
3) Learnability: Learnability measures how easy a task is for the users to accomplish it when they first time encounter the interface. It also measures the number of repetitions that the users take to become efficient at that task. Learnability is measured by the time taken for the participants to finish a task and the number of rounds they need in learning on using the system. According to the Nielsen Norman Group, the same task needs to be repeated until the time taken for the participants to finish the task started to plateau. However, in this research, we have fixed the number of rounds to be 3, as mentioned in the Methodology Section.
Several of the participants were eliminated from the calculation as they were facing some technical problem when completing the task. Only 64 results from the participants are included in the calculation for Task 2 and Task 3. The average time required by the participants for each task in each round is shown in Fig. 6.
It can be seen that the time taken for each task is decreasing when it is being repeated for the second and third time.

4) Memorability:
Memorability is about how easy is for the users to reestablish their skills after a long time of no use. Comparing task completion task 1, 2 and 3 on Round 1, Round 2 and Round 3. Based on the task completion on Table II, Round 2 performed the best by 100% had completed the task. Whereas, in Round 1 less than Round 2 which are 98%. Therefore, the task had been increased in Round 3 which are 99% successful completing the task. Mostly student can perform and not give up completing the task. For Round 1, most student spend most time to understand the task given and some students do not understand the task given. But then, they can do the task smoothly for the next round. Issues that affect the performance of the participants are unresponsive application and unstable connection. 5) Error-tolerance: Error is measured by calculating the number of errors that are done by the participants in completing the task. Among the error listed are; wrongly entered an input, click the wrong pageneed to go to sitemap/menu, click the wrong pageneed to press back button, understanding error (doing the wrong task) and press submit button when some questions were left unanswered. Each of the error will be counted including the repetitive errors. If the error faced by the participants is not in the list, participants are required to write the error and how it affects their process in completing the task.
As can be seen in Fig. 7, the number of errors for each of the task can be seem to be decreasing as the number of rounds increasing. The global errors are obtained by dividing the number of errors for a particular round with the number of tasks. Table IV shows the global errors obtained in Round 1, Round 2 and Round 3.

6) Satisfaction
: SUS questionnaire is used to measure user satisfaction of the apps and the results are shown in Fig. 8. The average score from all respondents is calculated and the results were converted to SUS adjective rating.
The average score from all participants is 61.13. Thus, the result falls in the 'OK' range based on the SUS Adjective Rating Scale.
The participants explained that they are still confused with the "downloading" system during the execution of the task. They are not sure whether they actually have downloaded the file or not. Besides that, the difference between Brighten App via mobile phones and desktop/laptop in terms of UI design also can lead to several confusion and complication. Overall, Brighten App is still acceptable to participant's perceptions of usability value but it needs a bit of improvement in terms of downloading files, IU design, etc.

7) Cognitive load:
The level of cognitive load involves while using the application is measured to determine the cognitive processing needs. NASA Task Load Index (TLX) is used to assess work load on five 7-point scales. Increments of high, medium and low instrument consists of five dimensions (one question associated with each dimension) was used to determine the cognitive load of students in performing the given tasks (refer Table V.). The specific dimensions determined the activity's contribution to the cognitive workload and measured using a Likert Scale that range from 1-Very Low and 7-Very High. The overall cognitive workload that the participant experiences is calculated by adding up all the scores and then the average is calculated by dividing the total score with the six different dimensions. The higher score indicated the higher the cognitive workload that the participant experienced.
The highest response is rating 5 with 34% for the performance dimension with the average rating of 5.5, indicating high cognitive load for this dimension. The next average rating of 3.9 for the temporal demand dimension followed by effort, mental, frustration and physical dimension (average scores 3.7, 3.6, 3.4 and 3.2 respectively).

8) Comments from participants:
Most of the participants managed to complete the tasks without any critical issues. However, some of them has provided some comments and issues on Brighten application that they faced while implementing the tasks. The comments and issues are categorized into five categories. The comments are shown in Fig. 9.    a The output from these questions will be measured using Likert Scale that range from 1-Very Low and 7-Very High Fig. 9. Comments from Participants.
The most common problem-faced is regarding the difficulties in finding items or functions in the application. In the message sending task, the participants are having problem in finding the send message function or the receiver's name. The application requires the exact full name of the receiver in order to ensure that the message is send to the right person. As the application allows the message to be sent to any user in the university, there is a possibility that the message will be sent to the wrong person.
In the downloading file task, the participants are having problem in finding the location of the files that have been downloaded. The application neither allows the participants to choose the location for the file to be saved nor informing the participants on the location of the saved file. www.ijacsa.thesai.org Some of the participants are having issues in navigating the application. Among the issues that slow down page navigation are; pages kept on reloading and the location of the button is not at common location. Most of these participants find that the mobile version of the application is not as friendly as the web version.
Several participants find that the color template used in the application is too pale and it is difficult to distinguish the sections. This can be seen in Fig. 10. Slow response and no task feedback are issues faced by 15 of the participants, where they find that it takes sometimes for the application to response despite good internet connection. In the task where the participants are required to answer quiz questions, they complained that there is no timer to indicate the time available for them. They find that this is very demotivating.

V. DISCUSSION
From the result, it can be seen that Brighten application passed all of the usability testing with minimum issues. The result of the effectiveness, efficiency, learnability, memorability and errors of the Brighten Mobile Application are acceptable where the learners have no major issues in implementing the tasks.
The average SUS score is 61.13, as mentioned in the previous section. This raw SUS score is below the average of common raw SUS threshold which is 68. This result is deemed as marginally acceptable.
In terms of cognitive load, the participants seemed to have a high cognitive load on performance dimension. This shows that the learners find it quite difficult to successfully accomplishing the given tasks.
In the comment section, some of the participants did mention that they find that the Brighten application on mobile device is difficult to be used compared to the web version. The participants are having problems in finding some of the menu buttons and users.
One of the main issues that interrupted the participants from completing the task is the network issue. Network issues not only affecting the usability testing result but also the communication between the researcher and the participants.
In this experiment, the number of errors is counted by the participants themselves. To ensure a more reliable result is being obtained, it is suggested that for the experiment to be conducted in face-to-face mode where the researcher can assist the participants in counting the number of errors that they encountered. This can also ensure that the testing is not being interrupted by network issue.

VI. CONCLUSION
This study evaluates the usability of the Brighten application based on PACMAD usability model where it evaluates the application in terms of effectiveness, efficiency, learnability, memorability, error-tolerance, satisfaction and cognitive load. On the surface, the result of the effectiveness, efficiency, learnability, memorability, error, satisfaction of the Brighten mobile application are acceptable with minor issues.
Using SUS and NASA-TLX shows that there are problems in using the application. However, most of the issues are being detailed out in the comment from the participants sections. It was observed that the issues that prevent full satisfaction from the learners and burden the cognitive of the learners comes from the navigation and user interface design.
Future work can be carried out by developing a guideline/framework for mobile learning user interface design that will increase learners' satisfaction and improve their cognitive load.