Evaluating the Effectiveness and Usability of AR-based OSH Application: HazHunt

—This study investigates the effectiveness and usability level of an augmented reality (AR) application called HazHunt to improve occupational safety and health (OSH) training. Previous research shows that AR has been growing in popularity as an innovative tool to enhance hazard identification courses. HazHunt, a marker-based AR app, was first developed using Vuforia software with OSH experts' guidance. Then, two online sessions of hazard identification course were conducted, where the experimental group's (EG) training was enhanced with the implementation of HazHunt. Analysis shows that the EG scores better (mean = 13.82, s = 3.38, n = 22) than the CG (mean = 13.41, s = 2.15, n = 22) in the post-quiz, but this difference is statistically non-significant, with t (21) = 0.48 and one-tail p = 0.32. Reduced Instructional Motivation Survey (RIMMS) shows that EG participants obtained higher confidence levels among the Attention, relevance, confidence, satisfaction (ARCS) factors in learning motivation. The System Usability Scale (SUS) score of HazHunt recorded the maximum count of 'Good' rating (mean = 78.41, n = 8). It is concluded that HazHunt has positive impacts on enhancing OSH training in terms of effectiveness and motivational impact. HazHunt also scored a high SUS score among the EG.


I. INTRODUCTION
Augmented reality (AR) has been proven to play a vital role among the nine pillars of Industrial Revolution 4.0 (IR4) in enhancing occupational safety and health (OSH) activities [1]. Among the activities include rehabilitation [2] and the innovation of serious games for safety training [3]. Particularly, OSH training has evolved tremendously over the years, but most organizations still rely upon traditional way to impart the knowledge. Today, complex working infrastructure present more hazardous environment for the safety and well-being of workers. Thus, AR is a relevant tool that has the capacity to enhance conventional training, suitable for current trend.
The effectiveness of using AR in improving academic performances has been proven for many teaching-learning process [4]. AR have been one of the many popular technologies, used by various institutions as the attractive and interactive elements provide positive effects towards learning performances. However, the overall effectiveness of AR-based technology intervention for professional training was reported to have a small effect on the outcome [5]. Interestingly, AR improved the overall effectiveness of vocational training for a large size effect [6]. This indicates the potential of deploying applications built with AR technology for OSH related training may produce better effects in achieving training outcomes as the first issue highlighted in this study.
Motivational impact is the second issue addressed when training is conducted conventionally [7]. Conventional training has seen a worse decline in providing sufficient motivational impact, especially in this pandemic era [8]. To overcome this, other IR4 technology such as virtual reality has been implemented for the sole purpose of boosting the motivation among learners [9]. In this study, the technological tool of interest, chosen to enhance motivation is AR [10], as AR is proven to be able to increase motivational impact as opposed to the conventional training delivery methodology [11].
Consequently, today's era perceived many information and communications technology (ICT) tools developed to be used in teaching-learning for various benefits, which requires the usability level of such tools to be properly examined [12], as the third issue for this study. In easy terms, usability level refers to the indicator on a specific user in a specific context, the ability to utilize the tool in achieving a goal effectively, efficiently, and satisfactorily. Simply developing a mobile app is insufficient to conclude its contribution towards the users in said fields, which is the reason to evaluate the usability level.
Therefore, based on the three issues elaborated, the following research questions are established: 1) What is the effect of implementing AR technology tool towards academic performances in OSH training?
2) What is the motivational impact of deploying the AR tool as a part of the OSH training?
3) What is the usability level of the developed AR tool?

II. BACKGROUND STUDY
Based on the research questions stated, there is a necessity to study several important areas. These important areas include the importance of conducting OSH training among organizational members in the organization, the effect as well as motivational impact of conducting training conventionally, and the relevant usage of AR as an innovative tool that had been reported to improvise the content delivery of OSH training in recent times.

A. Importance of OSH Training
In general, training is important because it is a precondition to ensure employees can perform their job effectively and www.ijacsa.thesai.org efficiently and, in the context of OSH, to ensure they can perform a job safely on top of being effective and efficient. To perform a job safely, the employees must equip themselves with sufficient, suitable OSH knowledge and skills, including the knowledge of hazards and how to identify them, the skills to assess risks related to the job, and the identification of suitable control measures reduce the risks.
Clarke & Flitcroft [13] studied the impact of training intervention over 24 months period on a sample of 10 companies and found that the interventions had a significant impact on company safety culture and productivity and even reduced workplace accidents by 22% on average. They also reported a significant increase in employee motivation and safety participation. The study demonstrated that training could be one of the important factors in creating a positive OSH culture within organizations.
In addition to the inherent characteristics of training that makes it important to have a safe workplace, the provision of training is also part of the organization's legal duty stipulated in many statutory laws around the world. For instance, in the United Kingdom, the Health and Safety at Work etc. Act (1974) [14] requires the employer to provide whatever training as is necessary to ensure, so far as is reasonably practicable, the health and safety at work of their employees. Iteration of similar requirements can also be found in the United States Occupational Safety and Health Act (1970) [15] and its Standards, Australia's Work Health & Safety Act (2011) [16], India's Occupational Safety, Health and Working Conditions Code (2020) [17], Canada Labour Code (1985) [18], Japanese Industrial Safety and Health Act (1972) [15], France's Code Du Travail (Labour Code, 2015) [19] and Germany's Arbeitsschutzgesetz (Occupational Health and Safety Act, 1996) [20].
The importance of OSH training is further compounded by the establishment of government-owned or government-linked OSH training centres such as OSHA Training Institute in the United States, OSH Training Centre of Hong Kong, The Occupational Safety Training Institute of Korea and Malaysian National Institute of Occupational Safety and Health, among others.

B. Effect and Motivation of Conventional OSH Training
Conventional OSH training commonly refers to a method of transferring OSH knowledge from the instructor (or trainer) to the training participants using a presentation-based lecture over one to several days' duration in a classroom setting. The variability of training delivery methods will depend on the syllabus, time allocation, the experience of instructors, the setting and location of the classroom and availability of training equipment and material. At its worst, conventional OSH training involves dry lectures coupled with PowerPoint printouts given as handouts and the use of unnecessarily long and unstimulating videos with content unrelated to the participant's learning needs [21]. Despite the many potential weaknesses and variability of conventional training, it remained popular as ever, as evidenced by the Association of Talent Development's State of The Industry report (2019) [22], citing 40% of an organization's hours were spent in a traditional classroom setting.
According to Casey, Turner, Hu & Bancroft [23], two important concepts that are relevant for OSH conventional training effectiveness are training engagement and training transfer. Training transfer is the application of learned skills, generalization to work scenarios and maintenance over time. A recent study by Aziz & Osman [24] highlighted that 98.3% of respondents used what they learned in training at their respective workplaces after training completion. The respondent of the study comprised of those who attended the Malaysian National Institute of Occupational Safety and Health (NIOSH) conventional OSH training courses.
On the other hand, training engagement involves the trainee's engagement with the training during its delivery. Safety training engagement, according to the authors, is the combination of optimal cognitive, emotional and behavioral activity that drives a trainee's motivation to learn. However, there is little research done to answer the question of how engagement can be increased and which elements of engagement are crucial within the context of different types of safety training [23].

C. AR as an Innovative Tool for OSH Training
The impact of AR technology has been studied in various industrial and educational settings, such as the real estate and construction industry [25]. While AR has been applied to several applications-OSH training is one novel application. For example, Kamal et al. found that the AR-based application has increased OSH training participants' active learning behavior, engagement, and interest [26]. It has also been established that AR can aid employees in identifying possible risks and correcting them before they cause actual damage. Sports, architecture, entertainment, and health training [27] have benefited from AR-based training. AR-based training is projected to grow in popularity and functionality greatly in the next years-aligned with the explosion of mobile technology [28].
In general, AR-based training provides several possible benefits and downsides. When developing or conducting a training programme, it is critical to consider these elements to ensure that everyone participating is optimally matched for the training opportunity. For example, Vignali et al. [29] developed Wearable Augmented Reality for Employee Safety in Manufacturing Systems (W-Artemys). They intended to provide new technology assistance for enhancing employee safety while performing machine operations using AR. Additionally, it is critical to assess the performance of ARbased training tools to ensure that workers retain and implement the material. If these goals are accomplished, ARbased training may be an effective method of equipping employees with the information and skills.
Immersive environments based on AR and VR have been effectively used to train personnel in many high-risk sectors [30]. Here AR may be a helpful tool for OSH training for various reasons. First, it can aid workers in comprehending complicated procedures. Second, it can serve as a visual depiction of potentially dangerous circumstances, reinforcing safety messaging. For example, Laciok et al. [31] designed a scenario for a work-related accident using the XVR software environment. Thirdly, it can assist employees in identifying www.ijacsa.thesai.org and resolving dangers prior to the occurrence of real damage. Fourth, AR may be utilized to deliver hands-on training tailored to specific industrial or safety needs.
However, the fundamental challenge is that applications of AR to OSH are still in their infancy [32]. For example, current AR-based training systems lack the scalability to support complicated training requirements. In addition, they may not give adequate augmented visuals to offer cognitive support or support for educational techniques. Verily, the influence of AR on OSH training is dependent on the way it is integrated into the curriculum. Aromaa et al. [33] stressed that there is a need for more interactive virtual and augmented OSH training material.
Therefore, there are a few possible ways to integrate AR into OSH training. This includes bringing a serious game and gamification approach to offer personalized learning interaction. For example, Holtkamp et al. [34] integrate AR and serious gaming to train workers wearing the right protective equipment and how to properly use ladders on their job site. In addition, more and more new AR-based instructional technique is required to motivate trainees and students alike to learn OHS [35].
However, assessing and evaluating the success of online training can be challenging. For example, some individuals may find AR-based OSH training to be tedious or ineffective. This may result in the loss of critical information or skills that employees require to do their jobs. If employees are disengaged with the training, they may be less likely to retain and implement the material. All these possible disadvantages may result in diminished learning and retention. To tackle these challenges, Tarallo et al. [36] created a mobile solution targeted at streamlining and speeding up the flow of training content concerning safety-related issues among safety managers, workers, and casual users.
These disadvantages may be offset by the benefits of ARbased training, such as its accessibility and ease of use [37]. In general, both online and enhanced OSH trainings are projected to gain popularity soon. The primary issue will be to ensure that these training alternatives are meaningful and available to workers in various scenarios and locales.

D. AR Enhanced OSH Training
Indeed, AR has the potential to improve OSH training outcomes by increasing trainee motivation. For example, Vigoroso et al. [38] prove that AR-based training games can boost training effectiveness. In their study, they proved that machine operators' abilities and safety understanding has also been enhanced. The learning process may be further sped up by incorporating learners. While AR has been implemented in a few sectors, it has yet to be implemented in OSH training.
Therefore, an enhanced OSH training is proposed by deploying an AR-based application called HazHunt. This enhanced training suggests the following hypotheses to be tested where the dependent variables are the effectiveness of implementing HazHunt in the enhanced OSH training, the impact of HazHunt towards learners' motivation, and the usability level of HazHunt application.

III. METHODOLOGY
In overall, the aim of this study is to enhance the conventional OSH training by implementing AR technology. So, the AR-based application, HazHunt was firstly developed. Then, to achieve the three ROs, two sessions of short OSH training course was held. Department of Occupational Safety and Health (DOSH) officers were appointed as the trainers for the short courses. After the HazHunt prototype is developed, an advertisement targeting university staff and students is launched to find potential participants. Due to the pandemic, the courses had to be delivered online via Google Meet. Registered participants are separated into CG and EG. The first session is delivered conventionally to a control group (CG), followed by the second session to an experimental group (EG). HazHunt was deployed in the second session to EG. For the purpose of data collection, RO1 is measured with mechanical and chemical hazard identification post-quiz, RO2 is measured with Reduced Instructional Materials Motivation Survey (RIMMS), and RO3 is measured with System Usability Scale (SUS), developed by John Brooke [39].

A. Phase I: Development of HazHunt
HazHunt is a marker-based AR application. It is developed using Vuforia software, with the assistance of OSH expert. Fig.  1 shows the HazHunt main menu interface. HazHunt contains AR embedded hazard pictograms to be scanned with the app to access the elaborations and quizzes. Fig. 2 shows the pictogram buttons to access specific hazards. These pictograms are the AR markers to trigger the elaboration of each hazard type, consisting of a video and description. Fig. 3 is an example of HazHunt in action, used to trigger the "skull and crossbones" elaboration by scanning its pictogram.

B. Phase II: Short Courses for CG
In the first session, the participants selected as the CG attended the course by conventional training method (online) as in Fig. 4. By the end of the session, a question and answer (Q&A) slot was conducted as shown in Fig. 5. Later, participants answered the post-quiz and the RIMMS.

C. Phase III: HazHunt Implementation for EG
The group of participants selected as EG underwent the second session (online). The trainers involved were prepared with the knowledge and ability to use HazHunt. The usage of HazHunt was integrated into the training in delivering related content. Prior to the session, participants were provided with a HazHunt apk file to be downloaded and installed on their smart devices. During the session, the trainers instructed the EG to scan the marker that appeared on the shared screen using HazHunt, as shown in Fig. 6. In addition to the learning materials, Fig. 7 shows the HazHunt quiz to enrich the training with AR contents. At the end of the session, the EG answered the post-quiz, the RIMMS and the SUS.

A. Post-Quiz
The post-quiz consists of 18 questions derived from the topics covered in the course, which are corrosion, explosion, fire, general, health, oxidizer, and toxicity hazards. The postquiz is analyzed to test the first hypothesis: www.ijacsa.thesai.org H 1 : The implementation of HazHunt has a significant effect on trainees' learning effectiveness. Table I shows the descriptive statistics of the post-quiz results from both groups. The post-quiz results of EG were hypothesized to be greater than CG's results. Based on the results, the EG has higher mean scores but also with higher standard deviation. This means the EG scores better than CG but vary larger among the participants, which shows that the HazHunt implementation does influence the learning effectiveness for the participants. This difference, however, was statistically non-significant, with t (21) = 0.48, p = 0.32 (one-tail), as can be observed from Table II.

B. RIMMS
The Reduced Instructional Motivation Survey (RIMMS) is a very useful tool to measure the learning motivations of the participants [40] to discern the differences between the CG and EG. The RIMMS is modified in conjunction with the training courses implementation. This survey is used to test the second hypothesis: H 2 : The enhanced training using HazHunt has a positive impact on the motivation of trainees. Table III shows the descriptive statistics for CG, whereas Table IV shows the descriptive statistics for EG. The means from statements 1 to 3, statements 4 to 6, statements 7 to 9, and statements 10 to 12 contribute to the mean for attention, relevance, confidence, and satisfaction factors, respectively, as shown in Table V. Individually, the EG scores a higher mean in statements 2, 6, 7, and 11. However, EG participants only scored higher in the confidence factor in the ARCS model for learning motivation, collectively. Nonetheless, the enhanced training using HazHunt has a slightly higher positive impact on the participants' motivation to learn.

C. SUS Score
The SUS questionnaire is used to acquire the usability level [41] of HazHunt, distributed only to the participants in EG.
The third hypothesis to be tested is: The HazHunt app is perceived to have a high usability level by the trainees. Table VI shows the descriptive statistics of the SUS scores for HazHunt. The scores range from 0 to 100, where HazHunt obtained a mean of 78.41 across 22 data collected, with the minimum and maximum scores being 62.50 and 95.00, respectively. According to the results shown in Table VII, the maximum scores with the count of 8 were recorded in the 'Good' rating, which means most of the scores were between 68 to 80.3 range. The second-highest scores count was recorded in 'Excellent' rating with the count of 10 participants to rate HazHunt above the 80.3 scores. None of the scores was equal to 68 under the 'Okay' and 'Poor' rating, but the 'Awful' rating consisted of 4 counts for the scores below 51. There might be several interpretations that can be perceived from these scores. However, factors non-related with HazHunt usability may have affected how participants provided the feedback in this questionnaire. Given that 81.8% (18 out of 22) participants provided above average for HazHunt SUS scores, HazHunt is perceived to have a high usability level.

V. DISCUSSION
The HazHunt app is developed in the hope of enhancing the benefits of OSH training by applying AR-based technology. The analysis is done in the previous section shows positive results in terms of participants' feedback, but the differences are not huge or significant between CG and EG. However, there may be several inferences that can be made from the analysis done as follows.

A. Online Training Environment
Online learning decreases a certain amount of enthusiasm among participants [42]. Furthermore, an online learning environment could have required the participants to master the usage of various hardware and software to get the most out of the training learning outcome [43]. Online training delivery is supposed to encourage active learning as the participants need to adapt to technology simultaneously looking at the course contents [44]. However, those who lack experience in virtual learning might face various technical issues before and during the training [45]. Additionally, the lack of interaction may have caused the participants to experience the feeling of being alienated or isolated [46]. These could have led the EG participant's HazHunt-enhanced training experience to be affected negatively.

B. Technical Inconveniences
Another common factor that may lead to bad feedback is the occurrence of the technical inconvenience with participants being alone with no physical assistance could be provided, which includes when handling AR technology [47]. In addition, throughout the online learning session, connectivity issues also occurred among participants, which could disrupt the AR demonstration [48], causing a bad impression of the technology and HazHunt. As of date, HazHunt is only available for Android users, and some device or operating software-specific bugs may hinder certain participant learning processes [49]. Although only a small percentage, some users among the participants might have viewed AR as an entertainment tool and rather perceived the technology lack the suitability to be used as a part of professional training [50].

C. Teaching Presence
For some people, teaching presence is required to ensure that the learning process happens as certain participants feel the need to have the trainer be physically closed to assist them to grow as the course progresses [51]. It is true that online training may replace face to face sessions in the cognitive area. Still, it could not replace meaningful interaction between trainer to participants and among peers [52], which caused the HazHunt enhanced training was not able to empower participants significantly.

VI. CONCLUSION
The purpose of this study was to determine the efficacy and usefulness of HazHunt, an AR-based application that was developed to enhance conventional OSH training. The markerbased AR app was built with the assistance of OSH expert using Vuforia. The enhancement of OSH training using HazHunt has been demonstrated to have a good influence on trainees' learning process. Based on the findings, AR technology has a positive effect towards academic performances and motivational impact of OSH training. Furthermore, the AR tool is shown to have a good usability level among the trainees, making the enhanced OSH training to have a new and meaningful experience.