Unmoderated Remote Usability Testing: An Approach during Covid-19 Pandemic

Online Nurse Test for Indonesian Nurse Competency (ONT UKNI) is a mobile application that was developed to help increase the success rate of nurse competency test participants. By using this application, users can learn more about the materials tested and conduct try out as a competency test simulation. However, ONT UKNI has not yet passed adequate testing stages, especially in terms of User Interface/User Experience (UI/UX). The Covid-19 pandemic situation presents challenges in the UI/UX testing process. Testing process which is ideally carried out face-to-face with respondents to get further insight, have to be carried out using another approach following the new normal protocol. This study aims to test the usability of UI/UX with an unmoderated remote testing approach on ONT UKNI application using a USE questionnaire. The test was performed using 26 respondents and all were nursing profession students of Universitas Muhammadiyah Yogyakarta. Respondents performed 8 tasks on ONT UKNI and answered set of questionnaire that will be tabulated and analyzed. The results indicate that usefulness, ease of learning, and satisfaction variables get the Very Good category while the ease of use variable gets the Good category. Overall, usability testing using an unmoderated remote testing approach can be carried out and able to provide information about areas where users are satisfied with ONT UKNI application. However, some areas still have room for improvement such as better UI design and implementation of gamification. Keywords—Mobile learning; nurse competency test; unmoderated remote testing; usability


I. INTRODUCTION
Mobile-based learning media is not a new thing in an effort to improve the quality of educational outcomes. Rapid growth of mobile technology and its application gradually replace the role of computer [1]. The author in [2] conducted research on the effectiveness of using Android-based learning media for biology subjects in high school students. From the research conducted, it is known that the learning media has a positive impact. Android-based learning media increases interest and motivation in learning which indirectly increases the effectiveness of learning outcomes. Android-based learning media provides several advantages such as attractive designs, both in terms of images, colors, and writing. Learning media is also easy to use independently either at school or outside of school. In another study conducted by [3], it is known that school students are easier to accept learning materials in digital form than in written or oral form. Digital learning media such as Android-based learning media application have several advantages, such as ease of use, can be used for learning anywhere, and can be used offline [4]. This shows that mobile applications are increasingly popular for use in education area.
Online Nurse Test for Indonesian Nurse Competency (ONT UKNI) is a mobile application that was developed and aimed to increasing the success rate of nurse competency test participants [5]. By using this application, users -in this case are nursing profession students-can learn about the materials tested at competency test and conduct try out as competency test simulations. However, ONT UKNI has not passed adequate testing stages, especially in terms of user experience or User Interface / User Experience (UI/UX). This is important because user experience determines the success or failure of a product [6]. If product's usability cannot satisfy user, it will hinder the overall quality of the application. Thus developer should consider usability when design an application so it will meet its purpose [7]. Furthermore, [8] added that the mobile app needs to go through a thorough testing phase due to several factors such as the limited screen size. For this reason, it is necessary to conduct user-oriented testing to ensure feasibility and user experience.
Usability testing can be used to test feasibility and user experience [9]. The author in [10] conducted research related to usability testing on the online guardianship application STMIK AMIK Bandung by using a USE questionnaire as a research instrument. Although USE questionnaire lack of evidence in reliability and validity, it provides insight for researchers in terms of usefulness, satisfaction, and ease of use [11] [12]. The author in [13] conducted usability testing research on YouTube websites among Malaysian teenagers using the same 3 criteria, namely usefulness, satisfaction, and ease of use. Apart from the USE questionnaire, there are also several other usability testing instruments, such as the User Experience Questionnaire (UEQ) and the System Usability Scale (SUS) [14] [15]. The author in [16] performed usability testing on the MyTelkomsel mobile application using 5 criteria, namely: learnability, efficiency, memorability, error, and satisfaction. The data collection technique used was a survey with SUS instrument, observation, and direct interviews. Based on this research, it can be seen that the interview technique helped researchers to get further insight and find solutions to the problems that respondents complains about the MyTelkomsel application. The author in [17] also used a survey and interview techniques in usability testing with certain variables, such as demographic information, using experience, ease of use, and usefulness.
The Covid-19 pandemic situation presents challenges in the UI/UX testing process. The author in [18] even stated that 283 | P a g e www.ijacsa.thesai.org based on a study conducted by researchers from Harvard University, Covid-19 protocols such as physical distancing will last until 2022. This conditions certainly become obstacles and challenges in various sectors, including the software testing sector. UI/UX testing, which ideally be carried out in-person with respondents to get optimum feedback, should be carried out using another approach following the new normal protocol policy [19] [20]. This can be overcome by conducting UI/UX testing with an unmoderated remote usability testing approach [21]. Unmoderated remote usability testing offers flexibility within time and distance constraints.
Based on the explanation above, the researcher decided to conduct usability testing on ONT UKNI application using an unmoderated remote testing approach. Unmoderated testing is a test where respondents try the product being tested and provide an assessment without being accompanied by a moderator [22]. The moderator creates a test scenario for respondents to follow who will then provide an assessment. Remote testing approach can solve problems related to distance since respondents can be located anywhere [23] [24]. Thus, remote testing considered suitable to be used considering physical distancing protocol during Covid-19 pandemic situation. USE questionnaire instrument was used to obtain qualitative data then processed further to determine user experience in terms of usefulness, satisfaction, ease of use, and ease of learning. This research contributes to theoretical and practical basis of using USE questionnaire in unmoderated remote testing and encourages professionals to adopt this approach.

II. ONT UKNI BACKGROUND
Nursing education is one area that can benefited of Android applications as a learning media for the preparation of the Indonesian Nurses Competency Test (UKNI). UKNI is a process to determine whether or not someone is eligible to become a nurse in Indonesia [25]. The author in [26] stated that the graduation rate in 2018 was 57.1% with 26,208 graduates, in 2019 it was 58.6% with 29,240 graduates, and in 2020 it was 54.4% with 23,663 graduates. This figure is quite low considering the need for the nursing profession in Indonesia is quite high. Indonesian Central Statistics Agency predicts that the need for nurses in Indonesia is 48,253,428 nurses with a ratio of 180 nurses per 100,000 population [27]. The need for an additional number of nurses in Indonesia in 2019 is predicted to be 372,534 nurses [28]. This is in line with the statement of the Professor of the Faculty of Nursing Universitas Indonesia, Achir Yani Hamid, who stated that the number of Indonesian nursing personnel was only 60% of the total population where the world standard was 80% [29].
The author in [30] stated that the obstacles faced by UKNI participants were lack of focus and lack of time to prepare, considering that most UKNI participants were students who were practicing the nursing profession. Another reason was the participants' ignorance about the UKNI concept, especially regarding the UKNI test grid. The author in [31], [32], and [33] analyze the factors that influence UKNI graduation, including try out, Grade Point Average, learning style, and learning motivation.
ONT UKNI was developed to overcome problems that have been stated before. It is a mobile application which has features such as learning material and try out simulation as shown in Fig. 1. Learning materials and try out given have been adapted to the material being tested at UKNI to provide real experience regarding nurse competency test.

III. METHODOLOGY
This study used a four-step procedure. Fig. 2 shows the research methodology used in this study. The authors start from literature review, requirement analysis, usability testing and data tabulation, and analysis.

A. Literature Review
The authors examine several theories sourced from books, reputable online sources, or journals. Several theories used in 284 | P a g e www.ijacsa.thesai.org this research include usability testing, USE questionnaire, testing, population, sample, data tabulation. This stage also aims to explore and find references related to previous studies.

B. Requirement Analysis
Research instruments and variables will be determined in this phase. The research instrument used to perform usability testing on a number of respondents. Sample size of respondents were taken from the population who are potential users of ONT UKNI application. Respondents will be asked to perform several steps contained in the test scenario that has been given by the author. Respondents were given 30 days to try the ONT UKNI application with the aim of getting as much user experience as possible. After that the respondents will fill out a questionnaire provided by the author.

C. Usability Test and Data Tabulation
At this stage the respondents will fill out an online assessment questionnaire based on the experience they have gained while using ONT UKNI application. The usability testing process is carried out remotely using Google Forms and is not moderated by researchers. Calculation of the value of each variable used in usability testing is carried out. Each variable used will be categorized based on class intervals.

D. Analysis
Usability test and data tabulation results were analyzed to find out how the results of each research variable were used. Recommendations regarding applications were analyzed to give ONT UKNI developer insight.

A. Requirement Analysis 1) Research instruments and variables:
At this stage the authors determine the instruments and research variables used. The assessment scale used in the questionnaire is Likert scale with descriptive statements converted into numerical values as in Table I. USE questionnaire developed by Arnold M. Lund in 2001 was used as a test instrument [12]. USE questionnaire consists of 30 statements which are divided into four variables of assessment variables, namely: usefulness, ease of use, ease of learning, and satisfaction. Usability instrument has been modified according to the scope of the research as shown in Tables II to V.   TABLE I. LIKERT SCALE ASSESSMENT CRITERIA

Statement Mark
Strongly agree 5 2) Research population: The population in this study were prospective users of ONT UKNI application, namely students of the nursing profession at the Universitas Muhammadiyah Yogyakarta who were at PKU Yogyakarta Hospital. This population was chosen because UKNI is a test for professional students to get nurse certification. Thus respondents were in a preparation for UKNI. From the entire population of nursing profession students of Universitas Muhammadiyah Yogyakarta who were interns at PKU, several samples taken to become respondents in usability testing.

U5
ONT UKNI application makes the learning process to deal with UKNI easier to do.

U6
ONT UKNI application saves me time in preparation for UKNI.

U7
ONT UKNI application fulfills my needs in preparation for UKNI.

U8
ONT UKNI app did a lot of what I expected with regards to UKNI preparation.

EU1
ONT UKNI application is easy to use.

EU2
ONT UKNI application is practical to use.

EU3
ONT UKNI application is user friendly.

EU4
ONT UKNI application requires a few steps when I want to use a feature in the application.

EU5
This application is very flexible.

EU6
Using ONT UKNI application does not require a lot of effort.

EU7
I can use ONT UKNI application without requiring written instructions or guidance.
EU8 I did not find any inconsistencies in ONT UKNI application.

EU9
Users who regularly or only occasionally use ONT UKNI application will love this application.
EU10 I can solve errors in using ONT UKNI application easily.
EU11 I managed to use ONT UKNI app every time.

S3
Using ONT UKNI application is very fun.

S4
ONT UKNI app worked as I expected.

S5
ONT UKNI application has a nice display.

S6
I feel the need to use ONT UKNI application in preparation for UKNI.

S7
ONT UKNI application is convenient to use in preparing for UKNI.

3) Sample size:
The sampling technique used is simple random sampling where each member of the population has the same opportunity to become a respondent. The size of sample is determined using the Slovin formula approach as described as in (1). The population size of nursing profession students at Muhammadiyah Yogyakarta University who are at PKU Yogyakarta Hospital is 27 students. The error rate used in the search for the sample size is 5%. By using (1), it can be calculated the sample size used in the usability test. The sample size used was 25.292 respondents which were rounded up to 26 respondents.

27
1+27 (0,05) 2 = 25,292 4) Testing scenario: Testing is carried out without moderation and remotely considering the Covid-19 pandemic situation. Remote testing is possible in the presence of a test scenario [34]. Test scenario consists of 8 steps in a task-based form and aims to ensure that respondents have tried and understood the features of ONT UKNI application. Table VI shows the scenarios that will be carried out by respondents remotely without moderation. Test scenario is made as simple as possible to make it easier for respondents to follow and does not require any clarification from researchers [24] [35]. 26 respondents in this study carried out a number of activities following the scenario given by the research team. Respondents were given 14 days to try ONT UKNI application and carry out the given scenario and try ONT UKNI application. After 14 days, the respondent will assess the usability of ONT UKNI application using the provided instrument.

B. Usability Testing and Data Tabulations
Tables VII to X shows post-usability test results that were tabulated and compiled. From the test results, it can be calculated the value of each variable used. Equation (2)   To be able to determine the category of assessment class based on known variable values, a class interval size is needed [36]. Equation (3) shows the formula used to calculate the length of the class interval.  (3), an assessment table of the results of the usability test analysis can be made which is then used to determine the category of each variable. Table XI shows the length of the interval for each category. Table XII shows the results of the analysis carried out. The usability variable gets a value of 4.21 and is included in the good category class interval. The ease of use variable gets a value of 4.16 and is included in the good class interval. The ease of learning variable got a value of 4.29 and was included in the good class interval. The satisfaction variable gets a value of 4.20 and is included in the very good class interval. Usefulness variable gets a value of 4.21 and is included in the Very Good category. ONT UKNI application has learning and tries out features that are in accordance with the needs of application users who are students of the nursing profession in preparation for joining UKNI. In addition, ONT UKNI application has discussion feature so that users will better understand the questions and exact answers given which are very useful, efficient, and can increase user productivity in the preparation for UKNI. One aspect that needs to be considered is the aspect of the responsiveness. Application responsiveness is one of the factors that cam further increase user productivity and efficiency in using applications [7].
Ease of use variable gets a value of 4.16 and falls into the Good category. Button design and layout is easy to reach. It has minimalist feature by having 2 main buttons for the main features, namely the learning feature and the try out feature. Users can also use the application even though it is not accompanied by a user guide and at any time. Consistent selection of text and colors also helps users. However, there are still some areas that could be improved. Color selection and interface design are 2 factors that have the potential to be improved to provide better interaction and user experience [37].
Ease of learning variable gets a value of 4.29 and was included in the Very Good category. It shows that ONT UKNI easy for user to understand. It is intuitive enough for first time user to follow. The application has a consistent design which helps user to get familiar faster which is an important aspect in ease of learning [38]. The placement of buttons accompanied by text makes it easier for users in the learning process to use this application. In addition, the selection of clear and contrasting colors between the buttons, written text, and background also helps the use of ONT UKNI application.
Satisfaction variable gets a value of 4.20 and is included in the Very Good category. ONT UKNI application was built with the aim of helping students of the nursing profession in dealing with UKNI. Based on the test results, it can be seen that users feel confident to use ONT UKNI application in preparation for UKNI. Application design that is attractive, easy to use, and easy to learn is also a factor that determines user satisfaction [39].

V. CONCLUSION AND FUTURE WORK
This study was aimed to conduct usability testing on ONT UKNI application. The test has been carried out on 26 respondents who are nursing profession students of Universitas Muhammadiyah Yogyakarta who were interns at PKU Yogyakarta Hospital. The test was conducted using a USE questionnaire remotely and without any moderation from the research team. USE questionnaire gives insight on 4 variables, namely usefulness, ease of use, ease of learning, and satisfaction. The usability variable gets a value of 4.21 and is included in the Very Good category. The ease of use variable gets a value of 4.16 and falls into the Good category. The ease of learning variable got a value of 4.29 and was included in the Very Good category. The satisfaction variable gets a value of 4.20 and is included in the Very Good category. However, there are several areas to improve the user experience of ONT UKNI application, such as color selection, application interface design, and implementation of gamification into ONT UKNI application. Overall, usability testing with unmoderated remote testing approach using USE questionnaire can be done and able to provide information about areas that users are satisfied with or areas that need improvement.
Future work should focus on improving ONT UKNI application by redesign UI and implementing gamification. Further test with intervention using quasi experiment also possible to be conducted to see whether ONT UKNI application really meets its purpose, which is to help nursing profession student preparing for UKNI.