Instrument Development for Measuring the Acceptance of UC & C : A Content Validity Study

Studies on the acceptance of Unified Communications and Collaboration (UC&C) tools such as instant messaging and video conferencing have been around for some time. Adoption and acceptance of UC&C tools and services has boosted productivity and improved communications by integrating voice, video and data into one platform. UC&C also allows collaboration by enabling users to interact with each other in different media. However, their acceptance rate by individual in developing nations has been low. It is hypothesized that the factors that contribute to acceptance are developed based on two underlying theories. The first is that of diffusion of innovation and the other is that of service dominant logic. Items is constructed based on eight constructs which are relative advantage, compatibility, ease of use, trialability, observability, improved service, value co-creation capacity, and coordination efficiency. In order to validate the items, content validity ratios are calculated on a set of questionnaire. The ratios will determine which items should be included or removed from the questionnaire. The paper concludes with a discussion on the implications of the findings from the experts’ evaluation and also from the content validity ratios. The new items are used in designing the survey instrument to measure the acceptance of UC&C. Keywords—Acceptance model; content validity; diffusion of innovation; service science; unified communications and collaboration


I. INTRODUCTION
Unified Communication and Collaboration (UC&C) which has been introduced in over a decade is a popular new technology used by organizations for communications purposes.Previous studies [1]- [3] state that UC&C services are used in organizations to ease user involvement in communication activities.However, organizations still lack knowledge about the availability of UC&C in providing effective communications [4], [5].Pérez et al. [3] also have pointed out that practitioners and scholars need to develop more understanding of the factors which can contribute or hampers successful adoption of UC&C.
According to Amily and Zaidi [6], the marketing strategies of communication technology should be focused on understanding the needs, trends and lifestyles of their potential users.Besides, the involvement of customer in the technologybased service is important for successful service delivery [7].Therefore, there is a need to discover factors that influence the acceptance of UC&C in organizations.In a previous study by Emy Salfarina Alias et al. [8], eight constructs that influence the acceptance of UC&C services has been identified.Items derived from these constructs are then developed into an assessment instrument for UC&C services users.Prior to administering the instrument, it has to go through a validation process.There are various ways to do this, one of which is via feedbacks from expert panels [9].Accordingly, in this paper, the content validity of the assessment instrument of the UC&C services is evaluated by using by Lawshe's technique.This paper is structured as follows: In Section II, a discussion of the related works is given.Section III describes the research method followed by results and analysis in Section IV.Section V discusses the findings and the limitations of the study.Section VI concludes the paper with suggestions for future research.

II. RELATED WORKS
Content validity refers to the ability of selected survey instruments to adequately reflect the characteristics of the measured constructs [10].Also, the purpose of content validity is to determine the extent to which dimensions and concept elements are clearly explained [11].Item verification function, on the other hand, examines whether it is sufficient to measure each construct [12].According to Saunders et al. [13] comments and suggestions from experts would help establish content validity in order to make necessary amendments before pilot testing.
The development of the instrument needs to go through the validity process of the content to ensure that constructed constructions are legitimate, clear and reflect its contents [14].Content validity can be implemented qualitatively or quantitatively as shown in Table I.
According to Nor'ashikin Ali et al. [17], qualitative analysis is difficult to interpret and the results obtained are less accurate because the questionnaire usually involves a large number of items.Meanwhile, Allahyari et al. [14] believes quantitative analysis is a better solution for content validity.Quantitative methods using Lawshe techniques are selected for this study because of its practical method.Based on the research conducted by Tojib and Sugianto [18], the content validity ratio (CVR) calculated using Lawshe techniques is more practical, convenient and saves time especially during the evaluation process.The CVR uses binomial distribution and also prepares tables to determine the values to be followed in calculations based on the number of experts involved [19].In addition, CVR calculations are also suitable for use in studies involving a small number of experts.www.ijacsa.thesai.orgContent validity ratio (CVR) [15] This method involves questionnaires assessed by a group of experts using a three points Likert scale to assess every construct.
Comment space is provided so that expert panels can provide their additional views.
The number of experts is not determined and usually depends on the suitability of the study.The CVR calculation is based on acceptance criteria set by Lawshe [15].

2
Content validity ratio (CVR) [14] This method is similar with the Lawshe method; it used a five Likert scale to assess each construct.The CVR calculation is based on acceptance criteria set by Allahyari et al. [14]. 3 Content validity index (CVI) [16] This method involves the assessment of constructs by experts panel based on the scale of four which is "1 = irrelevant", "2 = somewhat relevant", "3 = relevant ", and" 4 = very relevant ".The number of expert panels is between three and ten.
The content validity index (CVI) is not considered in this study because the scale 4 is not universal and can cause doubt [18].According to Nor'ashikin Ali et al. [17] CVI uses normal distribution that can cause inconsistencies and is less suitable for a small number of expert panels.Hence, this paper discusses step by step the process involved in the development of the instrument and subsequently validates the content by adapting the steps proposed by Nor'ashikin Ali et al. [17] using the CVR technique introduced by Lawshe [15].

A. Research Design
Questionnaire is an effective data collection tool or mechanism for researchers when conducting a survey [20].The data collected must be up-to-date, uniform and adhere to prescribed sampling procedures [11].Prior to the questionnaire development process, a thorough understanding of the important concepts in the study must be acquired via an in depth literature review [22].
The development of a questionnaire in this study adapted the procedure introduced by Mackenzie et al. [23] which originally contained 10 steps, but summarized by combining several steps into only six steps to suit the three key processes conceptualization, development process, and validity as shown in Fig. 1.
This study uses a structured and closed questionnaire that respondents are required to read questions/statements and choose the answers available.This type of questionnaire is chosen because it is easy to administer because the questions and answers are available to enable respondents to respond easily and accurately.The questionnaire should facilitate the respondents to answer questions.The approach to the questionnaire of this study is fundamental to the guidelines proposed by Creswell [24] which is from public-shaped questions to special forms, does not involve sensitive issues and not burden respondents.The physical design of a good question can help to increase the understanding of the respondents and then increase the rate of return and the accuracy of the answers.As suggested by Fallis [25], question designs need to be clear, easy to read and not confusing format.The questionnaire administration method should be established early either by correspondence, email, telephone or face-to-face interview.

B. Research Procedure
The research procedure can be broken into six steps.The steps are explained as follows Step 1: Develop a conceptual definition of the constructs.
Previous study has identified the conceptual framework in evaluating UC&C services through literature review.The conceptual model was developed for measuring the factors that could influence the acceptance of UC&C [8].The framework has identified eight factors that could contribute to the acceptance of UC&C services.They are relative advantage, compatibility, ease of use, trialability, observability, improved service, value co-created capacity and coordination efficiency.All of these factors have been identified through the literature by combining two theories which are diffusion of innovation theory (DOI) and service dominant logic (SDL) from service science.DOI is chosen because UC&C services is seen to be an evolving technology that is progressing to meet current demand in communication and collaboration.It is thus interesting to view UC&C as a diffusion, hence the use of DOI.Complementing the use of DOI, SDL has been chosen because, SDL encompass the concepts of value in use and value co-creation which are crucial concepts that enable users to extract the value from UC&C services.Using these two theories as the basis, a conceptual framework for the acceptance of UC&C was developed.The framework is made up of eight factors or constructs.Each of the constructs are then elaborated into items.Here, a total of 49 items were identified.
Generating concept definitions for the involved constructs is important to ensure that constructs can represent the concepts being studied and the constructs are unique [26].Development of the questionnaire can be done through the reuse of existing constructs obtained from past studies, but amendments need to be made to conform to the context of the study [21], [27].According to Kitchenham and Pfleeger [28], the advantages of reapplying existing constructs are: (i) existing constructs have undergone validity and reliability tests; and (ii) comparison of findings of current and past studies can be done.Table II shows the list of definitions for the eight constructs involved in the study.www.ijacsa.thesai.orgProcess of Instrument Development.

TABLE II. THE DEFINITION OF CONSTRUCT
No. Factor Definition Source

Relative advantage
The degree to which an innovation is Perceived as better than the idea it supersedes. [29]

Compatible
The degree to which an innovation is perceived As being consistent with the existing values, past experiences, and needs of potential adopters. [29] 3.
Ease of use The degree to which an innovation is perceived as Easy to understand and use. [29]

Trialability
The degree to which an innovation may be experimented with on a limited basis. [29]

Observability
The degree to which the results of an innovation Are visible to others. [29] 6 Improved Service The degree to which service is improved or new service is created from the introduced innovation. [8] 7.
Value Co-Created Capacity The degree to which value cocreation is enabled or allowed in the organizations. [8]

Coordination Efficiency
The degree to which the institutions or institutional arrangements coordinates value co-creation in the organizations. [8] Step 2: Generate the Items to Represent the Construct Podsakoff et al. [30] asserts that the selection of indicators is necessarily based on indicators with high weighting value in a construct because, low weighting value indicates that the construct is not correctly measured (no validity exists).Hair et al. [31] states that in order to provide stability to the constructs, each construct should have at least three indicators.Overall, this study involved 49 indicators (initial recommendation) to measure eight constructs as shown in Table III.
Step 3: Specify the Measurement Scale There are various opinions on the number of optimal measurement scales such as even numbers (2, 4, 6, 8 or 10) or odd (3, 5, 7 or 9).For an odd numbered scale, the numbers in the middle represent the neutral, uncertain, unknown or irrelevant choices.Whereas the even numbered scale shows the respondents are forced to choose the answer either positive or negative [28].An odd numbered scale (scale 3 and 5) can match the model accurately over the even numbered scale.Hence, the survey questionnaire uses three Likert scale which are (1) not necessary, (2) useful but not essential and (3) essential.
Step 4: Solicit Expert Participation Experts expressing the desire to participate were sent a package that contains: a) cover letter, b) a content review form.Participants are given 20 days to complete the form and send it back to the researcher.The cover letter explains the rationale for conducting the research and also indicates the confidentiality of the responses.www.ijacsa.thesai.org[32]- [35] Compatibility

COM1 -COM6
Describes your acceptance of the suitability of UC&C services within your organization.
[32]- [35] Ease of use EU1 -EU4 Describe your acceptance of UC&C service capabilities that are easy to use and understand.
[35], [38] Improved service IS1 -IS5 Describes the perception of UC&C service could improve and combine the services to create new service. [39] Value cocreated capacity

VCC1 -VCC5
Describes the perception of the organization is allowed value co-creation to extract the value of UC&C.

COOR1 -COOR5
Describes the perception of UC&C services are allowed your organization to coordinate value cocreation.

ACC1 -ACC6
Describes the perception of the UC&C adoption in your organizations.
[45], [46] Invitation to the panel of experts for the purpose of obtaining consent engagement in content validity sessions is performed individually via official email and telephone.Email is accompanied by an official letter.Once the expert agrees to be involved, a date is set for the interview sessions.The validity session content by all experts takes two to three weeks to be completed and transcribed.
Step 5: Selection of the Participation 1) Expert Panel Selection: Nor'ashikin Ali et al. [17] recommends the selected panel of experts to be involved and who are experienced in the same domain and have expertise in the development of the instrument (whether from academic or professional).There are some opinions in determining the number of experts.Lawshe [15] suggested that the panel of experts be composed of at least four people while Allahyari et al. [14] suggested 8 to 16 people.Nor'ashikin Ali et al. [17] on the contrary think that the expert panel should consist of 2 to 20 people.
In this study, seven experts were identified for content validity sessions.The selected specialist consists of academics, experienced professionals in the implementation of UC&C and consultants from industry sectors and Public Sector agencies.Table IV indicates the profiles of experts.Selection criteria are based on panel experience and involvement in relevant areas and fulfilling one or more of the following criteria:  Knowledgeable and experienced in the field of service science.
 Experienced in the development and implementation of UC&C in the private or public sector.
 Knowledgeable in theory, statistical or construct measurement.

IV. RESULT AND ANALYSIS
Validity means the extent to which the measurements (indicators) used are able to represent the concept correctly and also the extent to which the selected indicators correspond to the construct [47].Content validity is also known as expert confirmation as it is performed by a group of professional panels or experts in the related field [21].The validity study described in this paper adapts the process introduced by Lawshe [15] and is also used in the study of [14], [17], [48].It is based on quantitative methods using Likert 3 scale which is (1) not necessary, (2) useful but not essential and (3) essential.Expert panel is required to evaluate and validate the significance of the indicators based these given scales of 1 to 3.
Based on the feedback from the expert panel, the consensus among experts is measured by the calculation of CVR [15], which is analyzed using Microsoft Excel software.Table V shows that the minimum Lawshe's CVR value.While, in this study, the calculation of CVR is adapted from Nor'ashikin Ali et al. [17], the answers "2" and "3" are considered relevant while the answers "1" are irrelevant.The formula used is:  Ne = The number of experts who gave the relevant answer "2 = Agree" and "3 = Strongly Agree"  N = Total number of experts www.ijacsa.thesai.org This equation is described below:  If everyone on the panel of experts indicated the answer "3 = essential", then, the CVR value is 1.00 (all agreed).
 If more than half (> 50%), but less than everyone (<100%) on the panel of experts indicated the answer "3 = essential", then, the CVR value is positive, ranging between 0.00 and 0.99.
 If less than half (< 50%) of the panel of experts indicated the answer "3 = essential", then, the CVR value is negative:

V. DISCUSSION
The process of developing a detailed instrument (step by step) can be improved by understanding of the researcher and can be used as a guide to build a questionnaire instrument.It can also serve as a guide for future researchers in their specific fields.The development of a complete and orderly instrument can improve the quality of research management while producing good and reliable results.Thus, weak instruments are not capable of generating high quality outputs, thus www.ijacsa.thesai.orggenerating dubious findings.The valid and reliable questionnaire can be used to assess the level of acceptance of UC&C services in organizations.The results from the panel of experts yielded the following results.A total of 47 items fulfilled the condition of CVR.After the CVR calculations were done, two items were marked up for deletion as they failed to meet the set criteria.Item RA6 was rejected because one of the experts commented that the item has the same meaning with item RA3.Item COOR3 was rejected because it is irrelevant based on the comment from an IT service provider expert.The item is related to the provision of UC&C training for users.Since not all respondents for a future survey will come from IT related fields, the item is still considered important, and it is possible, after further investigation to still retain the item code COOR3.Besides this, three experts suggested minor revisions regarding the clarity or wording of the items, and those revisions were incorporated into the instrument.
Even though the study had carefully selected its experts, more insights can be obtained and the study can be further improved by including more experts from more diverse fields.

VI. CONCLUSION
This paper has highlighted a method that can be used to validate the contents of a survey instrument that was constructed to investigate the factors that drives the adoption of UC&C.A total of 49 items were evaluated which resulted in two items being rejected.In this study, it is shown that the CVR approach is able to differentiate the opinion of experts clearly and easily.All the 47 items that were refined would proceed to a pilot testing by distributing the questionnaire to the targeted respondents.Through the pilot study, the items will then be subjected to more statistical tests in order to confirm that they are reliable and valid to be used in the main study.

TABLE IV .
PROFILES OF EXPERTS

TABLE V
TableVIshows the final results of CVR calculations.Based on the calculation, there are two indicators that are rejected because they returned the value of 0.71.Hence, 47 indicators will be included in the final questionnaire.

TABLE VI .
CVR CALCULATION RESULT FOR EACH INDICATOR