Evaluating M-Learning in Saudi Arabia Universities using Concerns-Based Adoption Model Level of use Framework

Numerous studies have evaluated aspects of mlearning use in Saudi Arabia, mostly focused on technology use and its impact on students, or technology challenges and promises. Few studies have explored features of m-learning use and engagement among university faculty members. This paper presents a new methodology for evaluating the status of mlearning from faculty members’ perspectives in Saudi Arabia by investigating level of use using Concerns-Based Adoption Model framework. Concerns-Based Adoption Model is well established in the United States of America and in research investigating innovation adoption in education, including recent efforts in the Middle East (Jordan and Saudi Arabia). The outcome of such research, including this study, promotes better use and engagement with m-learning and provides a better understanding of advantages, disadvantages and barriers. The outcomes of this research study can reflect positively on universities’ status in the future and help in reforming policies and practices for developing the use of m-learning in Saudi Arabia. Keywords—Concern Based Adoption Model (CBAM); evaluation; M-learning; Saudi Universities; level of use; mobile phone


I. INTRODUCTION
Interest in m-learning in the Kingdom of Saudi Arabia (KSA) has grown immensely in recent years as a result of the rapid increase in mobile technologies, wireless networks, and the capabilities of today's mobile devices, facilitated by massive infrastructure and educational investment.The Saudi government is highly concerned to diversify the national economy and reduce oil dependency, and establishing a knowledge-based service economy is a key to this.Thus, more interest has been targeted to investing in IT and m-learning projects for Saudi educational institutions [1].These range from simple SMS services for individuals and groups to complex operations that related to managing students and distributing learning materials.
Studies have revealed that positive attitudes among students towards m-learning improve adoption of and engagement with educational technologies and m-learning can promote the learning process to shift from a teacher-centric model to a learner-centric approach.Moreover, different research studies have been oriented towards investigating m-learning in terms of adoption, benefits, obstacles and future development needs in KSA [2].However, few studies have considered issues related to faculty members' use of educational technologies, and defining their effects, impacts and challenges for future enhancements in use and policies.The exact engagement with m-learning by faculty members in KSA and level of use is thus unknown in Saudi m-learning literature, although different tools and frameworks are present for outlining and defining engagement levels [3].
The Concern Based Adoption Model (CBAM) is a framework known for its uses in investigating instructors' practices and providing a formative and summative evaluation methodology.CBAM consists of three different tools used for assessment, each focusing on a specific dimension with specific characteristics and strengths as a tool.The first tool is known as "Stages of Concern" (SoC), which is a quantitative questionnaire measuring users' feelings towards and innovation.The second tool is known as "Levels of Use" (LoU) which is an open-ended questionnaire based on a predefined matrix for measuring instructors' actions in eight behavioral categories against the variety of use.The last tool in this framework is known as "Innovation Configuration" (IC), which is a map of verbal description of the components of an innovation that describes the actions related to each component with differences on evaluating the actions, from "poor" to "ideal" [4].
CBAM tools can be used separately if needed in order to investigate different dimensions of m-learning adoption, or they can be used together sequentially, starting from SoC and ending with IC, to have a wider scope of understanding through assessing, monitoring and understanding different aspects of the implementation process related to m-learning.The CBAM tools provide a consistent and coherent taxonomy to describe emotional and behavioral domains with respect to the innovation used.CBAM has been widely deployed to study m-learning contexts in the United Stat of America (USA), Canada, Australia and more recently in the Middle East, specifically Jordan [4], [5].
This research is the first study to use the CBAM Levels of Use (CBAM-LoU) model to assess the use of m-learning in KSA.It is believed that the use of CBAM-LoU will provide a better understanding for m-learning implementation in KSA, which will promote better use, engagement and policies in the future.The next section presents CBAM-LoU and explains the use of this tool.www.ijacsa.thesai.org The structure of this paper is as follows.Section II discusses the levels CBAM.Section III describes the research methodology.Section IV presents the data analysis and results.Section V gives further discussion of the findings and Section VI concludes the paper.

II. CBAM LEVELS OF USE (CBAM-LOU)
CBAM-LoU focuses on behavior and does not pay any attention towards attitudes, emotions and feelings.Moreover, it does not concentrate or investigate the quality of the researched innovation.It focuses on the behavior of a user or group and classifies them into eight different levels relating to use of the innovation [6].The first three levels (0-II) are for the non-use of innovation, and the remaining five levels (III-VI) are for users using the innovation.CBAM-LoU has decision points that are triggered when users shift from one level of use to another.Table I shows the level of use with respect for each decision points used [6], [7].Each presented level has seven different categories of behavioral indicators [7], [8], as shown in Table II.Participants form a routine pattern of use.

Level IVA Routine
Participants' use of the innovation is stable and few changes are made over the period of use.Few efforts are made to develop or improve the use or its consequences.

Decision Point D-2
Participants adjust the use of the innovation according to formal or informal evaluation to improve expected benefits.

Level IVB Refinement
Participants vary the use of the innovation to increase benefits within the direct scope of influence.Differences are based on understanding of short-and long-term consequences.
Decision Point E Participants starts change in the usage of the innovation for the advantage of learners and in coordination with other colleagues to enhance the benefits

Level V Integration
Participants bring their efforts with each other for related activities in order to reach collective effect within the scope of influence.Decision Point F Participants starts exploring different modifications to the innovation they are currently involved with.

Level VI Renewal
Participants re-asses the quality of interaction and use of the innovation, searching for possibilities of performing main modifications or alternatives, in order to bring better impacts of the innovation to their practices, students and the system.
CBAM-LoU outlines each participant in only one level, unlike CBAM's SoC that defines users in different stages.In addition, the results of CBAM-LoU evaluation must be perceived as development status and not as summative end state [8].The previously presented categories for each LoU are all concerned with action performed by participants except for the knowledge category, which deals with observable behaviors, dealing with understanding about the innovation and its use and effects.
The used categories represent subparts for each LoU that define each LoU through presenting more in depth description of a part of LoU, thus it makes it possible to have different data points for determining LoU.Each presented category can be evaluated.
Distinctly and the groupings of ratings can be used to define the overall LoU [9].

Knowledge
Participants know how to use the innovation and its characteristics and consequences of use.Knowledge is cognitive and related to the use of the innovation, not to feelings or attitudes.

Acquiring Information
Participants ask for information about the innovation in different ways, such as asking about resources related to the innovation, persons, agencies and materials.

Sharing
Participants discuss the innovation use with other colleagues, and share plans, thoughts, resources, outcomes and challenges related to the use of the innovation.

Assessing
Participants explore the possible or actual use of the innovation, through mental evaluation or actual collection and analysis of data.

Planning
Participants make short-and long-plans during innovation adoption, such as scheduling, aligning resources, activities and coordinating the use of the innovation.

Status Reporting
Participants report their personal stand at the current time in regard to the use of the innovation.

Performing
Participants perform actions and activities related to the innovation.www.ijacsa.thesai.orgCBAM-LoU reliability, validity and internal consistency have been demonstrated by different research studies for use with more than eleven educational tools [9]- [11].The research sample consisted of 347 faculty members who agreed to participate in this research study, recruited by email from six different universities.A total of 119 faculty members were defined as users for m-learning, and 228 were classified as non-users.All participation was voluntary and fully informed consent was obtained.Interviews were performed in different settings and locations chosen by the participants and with respect for each university, as shown in Table III.
Classification distributed among different stages based on the derived results.The following table shows the categories and stages that are defined by CBAM-LoU tool [1], [13].
In terms of the gathered data, descriptive analysis was undertaken using SPSS V22 in order to define descriptive numerical data (frequencies) related to the LoU for all participants.The following section presents the output of this research study.

A. Participants using M-learning
Table IV shows the results of 119 faculty members using m-learning from six different universities in KSA.
The results indicate that participants are classified as mechanical use of m-learning according to the behavioral indications of CBAM-LoU.As noted previously, LoU is different from CBAM's Stages of Concern in that it defines only one level with the highest percent shown from the results [7], [8], [12].

B. Participants Not using M-learning
Table V shows the results for 228 participants not using mlearning in Saudi universities according to the CBAM-LoU matrix [7], [9].www.ijacsa.thesai.orgBased on CBAM-LoU, the results for non-users of mlearning are classified as orientation based on the behavioral indications provided by LoU methodology.

V. CBAM-LOU RESULT'S DISCUSSION
This section discusses the results related to the status of using m-learning in Saudi universities based on the results shown in Tables IV and V.

A. CBAM's LoU-Results Discussion (Using M-Learning)
As stated before that LoU is based on semi-structured interview and it has 8 questions for participants that were used to outline their level of uses based on the predefined categories.
The following discussion will be related to the results shown in (Table IV) with respect for the CBAM's LoU categories.

1) Knowledge category
In this category the following question was used: (What are the strengths and challenges of m-learning according to you, and have you tried to overcome challenges?).The results had two answers that identified participants as mechanical and routine use (86% and 14%, respectively).This category had different answers, and the majority of participants agreed that the main strength of m-learning was its availability and easy access anytime, anywhere.Moreover, they agreed that many used applications facilitated collaborative learning and enhanced learners' engagement with the materials.In addition, the majority agreed that m-learning encourages self-paced learning and it addresses different learning styles (e.g.reading text, watching videos, listening to podcasts, working with interactive material and researching on the internet).On the other hand, the majority of participants agreed on some challenges related to m-learning, such as the connectivity in many cases can be a major challenge for m-learning and the challenge of screen size limitation.

Knowledge
Furthermore, the mentioned device compatibility issues with some technologies and finally they agreed that using mobile phones for learning can cause distractions due to phone calls, instant messaging and other applications.In terms of overcoming challenges, the majority of participants agreed that faculty members must be trained to create content suitable for mobile devices and to be educated on using m-learning strategies.

2) Acquiring information
The following question was used for this category: (At the present time are you looking for information related to mlearning, and if so, for what kind and for what purpose?).The answers identified mechanical use and routine use (96% and 6%, respectively).Mechanical use participants agreed that they are not currently seeking additional information related to mlearning, as they are overwhelmed by the currently available applications and tools.On the other hand, routine use participants agreed that they are seeking different information related to m-learning, especially those used for creating microlearning objects to be used and distributed as learning content.

3) Sharing
In this category the following question was used: (Do you discuss m-learning with colleagues, and what do you tell them?).The results included non-use (66%), mechanical use (25%), routine use (6%) and refinement use (3%).
The non-use group agreed that they do not share any information related to m-learning, while the group defined as mechanical use agreed that they share some basic information related to some common tools and applications that are widely used in their institutions.The routine use group agreed that they share information about uses of some applications and its www.ijacsa.thesai.orgcommon uses among their close colleagues, while the refinement group agreed that they share the uses of m-learning tools and applications among different friends and on social media networks related to education and technological use, and they try to educate others on the benefits of using those applications.

4) Assessing
This category had two broad questions: (Based on your experience what is the effect of m-learning, how do you determine this, are you performing any formal or informal evaluation for using m-learning, and have you received feedback from students or colleagues, and what have you done with the information you got from the assessment?); and (Have you recently changed your use of m-learning, what, why and how recently are you considering making any changes?).The results from this category and the used questions came into four different results: non-use (96%), mechanical use (15%), routine use (3%) and integration use (2%).
The results show that the majority of participants in Saudi universities are not concerned with assessing their use of mlearning.This practice gives an indication for the need to educate faculty members in KSA about the importance of assessing their use of m-learning and its effect on future implementations and pedagogy in the educational sector.The mechanical use group agreed that they do some basic nonformal assessment for some features and uses of mobile applications and tools.The routine group agreed that they are performing informal assessment for their use of m-learning.The informal assessment is based on their daily use and discussion with other colleagues.
The integration group participants agreed that they are performing formal and informal assessment for their use of mlearning.The informal assessment is based on their discussion with other colleagues about the features and their impact on learning.On the other hand, the formal assessment is based on students' performance and output from participating and engaging with m-learning tools and applications.

5) Planning
In this category the following question was used: (What plans do you have for m-learning use?).This category had four different results: mechanical use (50%), non-use (26%), routine use (18%) and refinement (7%).The results show that half of participants are mechanical users with no real plans for future use of m-learning.All participants in this group showed no real plans for future use and enhancements.The non-use showed no interest in making any plans for future use of m-learning and mentioned that no plans are being made.The routine use group agreed that they are making small plans for enhancing their use and engagement with tools and applications used for educational purposes.The refinement group agreed that they are having serious plans for exploring different tools that are being used within the educational context, in KSA and abroad.

6) Status reporting
This category had the following question: (Are you coordinating your work with other colleagues out of your institution on the use of m-learning, and have you made any changes on m-learning based on your coordination?).The majority of participants agreed on not coordinating or working with others, thus they are classified as non-use (91%), with 5% of routine use participants showing some coordination with their colleagues outside their institutions using social media channels and educational groups.Mechanical use (4%) participants showed little and shallow coordination with some colleagues in different universities.The routine use group mentioned that they have made some changes related to mlearning use of tools and applications in that they have discovered better tools through reporting their use of existing tools and applications to others.

7) Performing
In this category the following question was used: (Are you planning to make major changes or to replace m-learning use at this time?).Three different groups emerged from answers: nonuse (49%), mechanical use (34%) and routine use (18%).The first groups' answers stated that no planning had been made to change or replace m-learning.The second group (mechanical use) agreed that they have plans to change and enhance their use of m-learning through adopting different applications and tools and that they need to consider the use of m-learning with different subjects being taught and to encourage students on more interaction.
The routine use group agreed that they are seriously considering making major changes in their use and interaction with m-learning through exploring different tools used in education by different foreign institutions worldwide.Moreover, they agreed that they need to enhance the collaboration level among different faculty members in different universities to share experiences and knowledge.In terms of replacing m-learning, all participants agreed that mlearning is the latest technology being used by them and their institutions and no other technology is present that can act as substitute to m-learning.
The previous results for all categories were shown that the interaction level for participants with m-learning ranges from basic to intermediate.Moreover, it shows that the interaction level with m-learning is defined as mechanical use, as participants' efforts are based on short-range, day-to-day use of m-learning with little time for reflection.As stated before, mechanical use is related with participants directing their efforts to become proficient at tasks required for using mlearning, and their use is generally considered fragmented and shallow.
On the other hand, a smaller group of participants showed more interest and engagement with m-learning activities, and this group was defined as routine use.The routine use group showed different and simple changes in their ongoing use, and they provided some efforts and views to improve their use of m-learning.For some categories, participants' attitude was classified as non-use, in the categories sharing, assessing, planning, status reporting and performing.
The results of CBAM-LoU for participants in this study were identified as mechanical, as CBAM identifies one level of use only.The next section explains the results related to participants who did not use m-learning and discusses their status based on CBAM methodology.www.ijacsa.thesai.org

B. CBAM's LoU-Results Discussion (Not using M-Learning)
The second group of 228 participants found not using mlearning has been interviewed using CBAM-LoU matrix, and this section will provide the discussion for their results with respect for LoU categories.

1) Knowledge category
For this category the following two questions were used: (Have you decided to use m-learning in the future, if so, when?); and (How do you describe m-learning as you see it?).The results indicated non-use (23%) and orientation (77%).The non-use groups were found to be uninterested in adopting m-learning as part of their educational activities and routine.They did not show any positivity for adopting m-learning in the future, nor did they provide satisfactory description of mlearning as they focused on the hardware aspects and some of its challenges rather than tools and services.
The orientation group showed positive attitudes towards adopting m-learning within their educational activities, and they agreed on using m-learning in the near future.They managed to describe m-learning in satisfactory manner, as they focused on the mobility, tools and applications that can be used in education.Moreover, it was found that they have not been using m-learning because of high responsibilities and being busy with many educational tasks and activities.In addition, they believed that m-learning will add more responsibilities that they cannot adopt; especially as such practices are not being recognized or encouraged by their institutions or educational policies.

2) Acquiring information category
In this category the following question was used: (Are you currently searching for any information related to m-learning, and if so, for what kind and for what purpose?).The answers for this category indicated non-use (32%) and orientation (68%).The non-use participants showed no interest in knowing more about m-learning and the wide selection of services it can offer.They stated that they are satisfied with the traditional approach or the current knowledge they have about m-learning.
On the other hand, the orientation group showed a positive attitude for learning more about m-learning and the services it offers related to content creation, sharing resources and providing collaboration among users.Moreover, they were interested in knowing about tools and services that can support their teaching, with the focus on minimizing efforts and fatigue inherent in traditional approaches of teaching.

3) Sharing category
In this category the following question was used: (Do you share information about m-learning with others, and what do you share?).The answers for this category indicated non-use (84%) and orientation (16%).The non-use group agreed that they do not discuss or share any information related to mlearning use, tools, services and applications.On the other hand, a small percent of the orientation group showed positive attitudes as they discussed and shared information with other users.The shared information was related to the perceived benefits of m-learning, tools and services that are used in education and the possibility of saving efforts using m-learning technology.

4) Assessing category
For this category two questions were used: (What are the assets and flaws of m-learning for your situation?);and (What questions do you ask related to m-learning use, giving examples if possible?).The results indicated non-use (87%) and orientation (22%).The non-use group did not specify or identify any assets related to m-learning, but they focused on the flaws they mentioned, such as that m-learning is timeconsuming and not beneficial compared with the traditional approach and activities, or even if compared with internet use through personal computers.
In terms of the questions asked related to m-learning use, the most frequent questions were related to the benefits of mlearning on educational activities, such as how m-learning supports different pedagogies, assessments and user engagements.On the other hand, the orientation group recognized some benefits for using m-learning as they mentioned it might help in learning more about different technologies and bridging the gap between Saudi and foreign universities.Moreover, they mentioned that using m-learning can provide the same benefits of e-learning.In terms of flaws, they mentioned the lack of Arabic digitized resources, tool and services, and the absence of consensus of specific tools and services in education are another main flow.
In terms of questions and concerns related to m-learning, the main concerns were related to possibilities of supporting mlearning with policies and recognition for the efforts.The possibilities of providing tools in Arabic language were the main concern for many faculty members KSA.

5) Planning category
The following question was used with this category: (Do you have any plans or preparations for m-learning adoption?).The results indicated non-use (86%) and orientation (14%).The majority of participants mentioned that they do not have plans or preparations for m-learning use, especially as most of the tools are presented in non-Arabic languages and there is no agreement on using specific tools in educational settings in KSA or having a policy for such use.On the other hand, the orientation group mentioned that they need to have training on the use and benefits of m-learning, and they need better management skills for their traditional educational activities.Moreover, they focused on the need for support by their institutions to be able to have serious plans of use and adoption.

6) Status reporting category
For this category the following question was used: (Where do you see yourself in relation to the use of m-learning?).The answers indicated non-use (83%) and orientation (17%).The non-use group agreed that they are currently not using mlearning as they are satisfied with traditional learning and with the use of e-learning through Personal Computers (PCs).On the other hand, the orientation group agreed that they are interested in m-learning but there are some serious obstacles for adopting such services and practice.The challenges are related to policies, time management, language and content creation.www.ijacsa.thesai.orgInvestigating participants not using m-learning gave better understanding for the challenges facing m-learning adoption in Saudi universities.Different challenges that act as barriers for m-learning use were identified, and it was found that eliminating those challenges will enhance the adoption by the groups defined as orientation by CBAM-LoU.However, there will be a group that resists the change and favors traditional approaches only, seeing the obstacles and neglecting the opportunities for enhancing educational outcomes.

VI. CONCLUSION
The importance of evaluating the use and interaction with educational tools, applications or systems is essential to provide better quality of education and to form and reform appropriate policies.Different evaluation tools are used to evaluate the interaction with m-learning.This research deployed CBAM-LoU tool, as CBAM is a well-used framework for evaluating educational tools within educational settings.Moreover, CBAM-LoU managed to provide different views on the evaluation results through the matrix it provides.This research study managed to evaluate the use of m-learning in six different Saudi universities with 347 participants, using an evaluation methodology based on using semi-structured interviews.
The evaluation sample was divided into two groups, users and non-users of m-learning.The first group consisted of 119 participants, and the results from CBAM-LoU identified them as mechanical use, which defines participants as having their focus on short-term daily engagement with m-learning with little time for reproduction.Their efforts are largely focused on learning important issues related to the use of the educational tool, thus their use is fragmented and shallow.In terms of challenges related to that group, it was found that using mlearning adds more responsibilities for managing their tasks and activities.Moreover, it was found that their practice of mlearning use is based on self-efforts and less is related to university's support and policies.
The categories that had challenges and which need more attention from Saudi management and officials in universities are sharing, accessing, planning, status reporting and performing.On the other hand, the second group consisted of 223 participants who are not using m-learning.The CBAM-LoU matrix with semi-structured interviews was performed and the results identified some participants as the non-use category.This category classifies users as having little or no knowledge of using m-learning, and participants are not showing interest or action towards becoming involved.
In addition, it was found that participants are favoring the traditional approach of teaching and the use of e-learning using personal computers.According to CBAM-LoU, participants showed the lowest impact in the categories sharing, assessing, planning and status reporting.
These results give a good indication about the level of use and the status of m-learning in Saudi universities.The information from this research study can be used to promote better practice and engagement with m-learning through the defined strengths and challenges, and through planning for better training and policy settings by university officials and decision makers.

Fig. 1 .
Fig. 1.Branching technique used to define level of use category.III.RESEARCH METHODOLOGY This research methodology is based on the specifications of CBAM-LoU tool applied to faculty members in Saudi universities concerning the use of m-learning.The evaluation of participants was based on focused interviews that used a branching technique based on the output obtained from each participant.The used branching technique is presented in Fig 1.

TABLE I .
CBAM LEVEL OF USE CATEGORIES AND DECISION POINTS

TABLE III .
PARTICIPANT GROUPS AND UNIVERSITIES

TABLE V .
CBAM LOU RESULTS FOR (NOT USING M-LEARNING)