Information System Evaluation based on Multi-Criteria Decision Making : A Comparison of Two Sectors

In this article, our purpose is to introduce the results of a new approach to assess the information system success. It is based on the DeLone and McLean model and was applied on two domains. The chosen domains are banking sector being the most customer of information technology and construction industry as the least computer-intensive sector. The work methodology used to evaluate the information system performance is a combined approach of the two most popular multi-criteria decision making techniques: AHP and TOPSIS. Based on the results of this technique applied on studied sectors, we can obtain a horizontal comparison at the sector level and optimize the choice of the best system. Keywords—Information system success; multi criteria decision; AHP and TOPSIS methods; criteria


INTRODUCTION
The emergence of computing and communication technology commonly known as information technology and the relatively cheap of hardware, the managers of organizations invest massively on information system (IS).
IS [1] is a set of data, hardware, software to treat data and procedure to help personnel establish the several objectives of organization.
The literature is rich with different models to assess information system [2] that offer the possibility to highlight the importance of evaluation in this field.
Concurrence, economic evolution and technological progress impose organizations to invest heavily on information system regardless their domain.
In this work, we will take two paradoxes sectors on terms of information system use.Banking sector being the most customer of IS, which has a primordial role at the internal and external environment.The role of banking information system is more complex considering the nature of products and services that offer also the requirements of these customers.
In contrary, construction industry sector who do not generally appear to appreciate the positive influence and changes that an IS provide.In general, construction sector cannot headily be compared to the banking sector in regards to adoption of information systems.Some major reasons for such situation are [3]: the nature of construction industry, the traditionalism of construction and low level of investments into research and development within this sector.This paper present the results of a new technique developed to assess information system success being tested on two sectors.The proposed technique has adopted the D&M success model ( 2003) [4] and used two famous multi criteria decision making approaches; AHP and TOPSIS [5], [6] to evaluate the IS success.As a contribution of this work, this article present a benchmark study of the two studies sectors on information system success based on the same work methodology and comparing the results.

II. INFORMATION SYSTEM SUCCESS MODELS
In the last two decades, many models have been cited to evaluate the IS and to explain the dimensions that makes IS successful [4], [7]- [12].The majority of these models have been validated empirically by researchers in several domains.

A. The DeLone and McLean Model
The evaluation of IS success is among the most delicate areas, because of the complexity and the multidimensional aspect of IS.Among the first attempt at solving this problem that of D&M in 1992 [8].However, this first model has undergone several criticisms which pushed the authors to update their model [4].
The original (Fig. 1) and updated (Fig. 2) DeLone and McLean IS success models are the most cited and used models in the field of IS evaluation [13].
Based on empirical studies, suggestions and tests, DeLone and McLean have updated the first model.That is way this updated model is the most adopted and cited version in the literature review in the field on IS assessment.

III. MCDM
Multiple criteria decision-making (MCDM), is a part of operations research [15], its objective is to facilitate decision facing problems involving multiple criteria.The complexity of such as problem is the lack of a single solution, that way it is necessary to use maker's preferences to obtain the best solution from a collection of alternatives under a number of criteria and even sub-criteria [16].

A. Analytical Hierarchy Process (AHP)
AHP is an analytical technique based on a hierarchy process [18], [19].The principal is to decompose a problem into hierarchies of goals, criteria (sub-criteria) and alternatives (Fig. 3).
AHP is classified as one of the the most cited approach.It has the possibility to treat both tangibles and non-tangibles criteria.

B. Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)
The TOPSIS method was developed in 1981 [20], the basic concept is to choose the best alternative (Fig. 4) depending on closest and most distance respectively to positive ideal solution and negative ideal solution.

A. Purpose of Study
This study provide a comparison between IS success in two sectors and presents the results of a framework based on the MCDM approaches for ranking these information systems.Our work methodology was applied on two sectors: banking sector [21] and construction industry sector [22].Firstly, D&M model (2003) was adopted to construct the analytical hierarchy process; the six dimensions are considered as main criteria and sub-criteria were taken from literature.Using AHP method the weights of criteria (sub-criteria) are obtained.Then, TOPSIS method was applied to rank information systems.For the purpose of testing and verifying the proposed framework on each sector, we were done an online questionnaire to collect data.

B. Studied Sectors
As mentioned, this study compare the results of a framework applied on two sectors, in the following sections we will explain the particularities of each sector.
1) Banking Sector: Banks are the first companies that have invested heavily in the technologies of information and communication.The banking sector is one of the sectors most risk-sensitive [23] with this multitude of risk types, the IS is considered both as a risk factor and a tool for identifying, evaluating and monitoring risks.
Banking IS is generally subject to stricter rules [24]: The availability of the IS even after malfunctions, the security of data especially the confidentiality of customer information, archiving and data traceability to facilitate internal control and audit and finally the integrity of data.
A last particularity of the banking IS is the opening to the outside and that it is to be functional 24 hours a day, banks offer today sales and purchase transactions, consultation or even transaction from Internet.

2) Construction Industry Sector:
The construction industry sector use information systems [25] to enhance organizational flexibility, improve decision making capability, reduce project completion time and cost and to present an ideal schedule for the factory construction work.In general way, it used to achieve project mission objectives within specified constraints that construction industry knows.Despite, the construction industry is one of the sector that doesn't use heavily IS [26].
The following figure presents the basic idea of the study (Fig. 5):

C. Research Methodology
The evaluation procedure consists of six main steps as summarized in the following figure (Fig. 6): Step 2: Identify sub-criteria; Step 3: Construct the structural hierarchy; establish goal which is in our case the evaluation of IS success, identify the alternatives which are the methods that change the preliminary condition into preferred condition and select the main criteria (sub-criteria); www.ijacsa.thesai.org Step 4: Calculate the weights of each criterion using AHP; Step 5: Apply the TOPSIS method; Step 6: Achieve the final ranking results.
The description of each step will be given in the following sections.

A. Implementation of AHP Method
The AHP hierarchy for decision making in this paper as shown in Table I is constitute of six main criteria which are the several dimensions of the updated model (D&M 2003) and sub-criteria were inspired from literature.
This study utilized a questionnaire survey to collect data from the different decision makers.According to the steps defined in Section III-A, (Fig. 7) display the pairwise comparison matrix using ( 1) and (Fig. 8) shows the normalized decision matrix which is calculated using (2).In this part, we choose to only present results of banking sector.Then the priority weights are calculated using ( 2 The same calculation steps were followed to determine the weights of the sub-criteria.We chose not to present them so as not to load our article by several tables.These results were used for the TOPSIS method.To calculate , we used ( 4), ( 5), and ( 6): =6.09 (CI) and (CR) are calculated through ( 7) and ( 8), respectively (for RI=1.24, according to table below (Table II)): CI=0.018 and CR= 0.014 CR=0.014<0.10, it is accepted.

B. Implementation of TOPSIS Method
The second MCDM methods we used in this work is TOPSIS.AHP weighted scores are used by TOPSIS for each sector's alternative to obtained performance ranks of systems.
The five banking companies are referenced: , , , and the companies operating in the construction industry sector are referenced as: , , , , .
The different steps of the TOPSIS method were followed applying equations.In the final step, using (16), the relative closeness to ideal solution is calculated and the companies were ranked as shown in Tables III and IV.According to the value of , the alternative having highest closeness coefficient in bank is alternative 4 with =0,551 and in construction industry is alternative 3 with =0,771.These alternatives are selected as the two best companies among studied alternatives in terms of information system performance.

C. Elaboration of a Prototype
This work is based on a proposed framework to evaluate the IS success, to implement it we propose a prototype that consists of three interconnected parts (Fig. 9)  The different actors which in our case are the decision makers to specify the weights of each criterion (subcriterion) to evaluate the IS.
 The user interface that allows adding details of criterion to assess the studied system.
 Functional part that simulates the different steps of the two used approaches AHP and TOPSIS.
 Data Base to store the various data that will be used in the next steps of evaluation. 1) User Interface: After authentification throught a login and a password, the decision marker goes on to enter the data of the evaluation.
2) Functional part: The role of this part is to rank the different alternatives using AHP and TOPSIS methods according to the criteria chosen from the DeLone and McLean model (2003).The two used MCDM in this work requires a very important number of calculations specially when the number of criteria, sub-criteria and alternatives increases; for this reason, we present a prototype of software that implement our methodology process, it is developed in java language under Netbeans platform.The main interfaces will be presented in the following paragraphs.
The first step after authentication, the analyst fills the pairwise comparison matrices which represent several criteria.(Fig. 10 and 11) shows interfaces that lead the analyst to input the values of matrices respecting Saaty's pairwise comparison scale of AHP method.(Fig. 12) provides the weights of main and sub criteria using AHP equations.
The second step of the proposed prototype aims to implement the TOPSIS method, the first interface of this phase, lead the user to input the values of pairwise comparison of matrices which contain criteria and sub-criteria.(Fig. 13 and 14) shows the application of (10) (11) (12) (13) ( 14) and (15) based on the data of alternatives input in by the user to make calculations.
Finally, we display the results of alternatives ranking based on the calculations of the distance between positive and negative ideal solutions.Fig. 15 shows the list of the alternatives selected by the maker decision which can be printed.3) Data Base: to store data's evaluation.We chose to work with SQL Lite because the recorded information is not large.We used a SQLite database given the reduced size of the data.

D. Discussions
In the context of the application of MCDM methods, main criteria are chosen from The D&M model and sub-criteria from the literature to applied our research methodology.From the results, it can be observed that alternatives and consistently perform better than the rest.It should be noted that the weights of criteria differ between banking sector and construction industry sector.
Banks characterized by the nature of their operations and services that explain the high value of weights of system quality and information quality (Fig. 12) At the opposite of construction industry sector who rarely uses an information system which is clear by the weight of criteria that are very close.(Fig. 16) shows the separation measures of each alternative (5 alternatives of each sector) from positive ideal solution, it's clear that values of bank's alternatives are close this can be explained by the large use of IS within banks.In Construction industry, we observe a big difference between the values there are companies that use the IS and others that do not use it, in this sector the IS is not used in an equitable way.The same remark can be done concerning the separation measures of each alternative from negative ideal solution (Fig. 17   The relative of each alternative confirm the previous remarks (Fig. 18), bank's alternatives have close values due to the high use of information system within banks.Construction industry companies are very distant in terms of closeness coefficient values; we have company that use heavily IS and the others who use the IS partially.
We conclude that the proposed methodology and the hierarchical presentation of criteria can be used for IS evaluation no matter which sector is studied.The method evaluate IS independently of input values of the two used MCDM.

VI. CONCLUSION
Determining the success of information system and evaluating them is crucial for the sector's development.In this study, we were presented an application of our work methodology based on a hybrid MCDM process (AHP and TOPSIS methods).This methodology is tested on two sectors; banking and construction industry.
Five companies are chosen from each sector to select the most performing IS.Furthermore, a prototype of software implementing the work methodology is proposed.AHP method is used to determine the weights of main criteria and subcriteria and TOPSIS method is utilized for ranking alternatives.

Fig. 2 .
Fig. 2. Updated DeLone and McLean IS success model.For the reasons mentioned before, in this work, our research methodology is based on the updated IS success model which is composed by six interrelated dimensions as shown in the figure at the top.

Fig. 3 .Stage 7 :
Fig. 3. Analytical hierarchy process tree.In mathematical way, the basic AHP equations are as follow [19]: Stage 1: Decomposes the initial problem into hierarchical presentation (goal, criteria and alternatives) Stage 2: Develop the pairwise comparison: =1 when i=j and =1/

Fig. 6 .
Fig. 6.Steps of evaluation procedure.Step 1: Identify the evaluation criteria using the information system success model (DeLone & McLean 2003);

TABLE IV .
RANKING OF COMPANIES ON CONSTRUCTION INDUSTRY