The Group Decision Support System to Evaluate the ICT Project Performance Using the Hybrid Method of AHP , TOPSIS and Copeland Score

This paper proposed a concept of the Group Decision Support System (GDSS) to evaluate the performance of Information and Communications Technology (ICT) Projects in Indonesian regional government agencies to overcome any possible inconsistencies which may occur in a decision-making process. By considering the aspect of the applicable legislation, decision makers involved to provide an assessment and evaluation of the ICT project implementation in regional government agencies consisted of Executing Parties of Government Institutions, ICT Management Work Units, Business Process Owner Units, and Society, represented by DPRD (Regional People’s Representative Assembly). The contributions of those decision makers in the said model were in the form of preferences to evaluate the ICT project-related alternatives based on the predetermined criteria for the method of Multiple Criteria Decision Making (MCDM). This research presented a GDSS framework integrating the Methods of Analytic Hierarchy Process (AHP), Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) and Copeland Score. The AHP method was used to generate values for the criteria used as input in the calculation process of the TOPSIS method. Results of the TOPSIS calculation generated a project rank indicated by each decision maker, and to combine the different preferences of these decision makers, the Copeland Score method was used as one of the voting methods to determine the best project rank of all the ranks indicated by the desicion makers. Keyword—GDSS; ICT; MCDM; AHP; TOPSIS; Copeland Score; Decision Maker


INTRODUCTION
The main advantage which this Multiple Criteria Decision Making (MCDM) offers is its ability to provide decisionmaking processes through the analysis of complex problems, aggregation of the criteria used in evaluation processes, the possibility of making the right decision, and the scope for decision makers to participate actively in the decision-making processes [1].
Several research in ICT project performance evaluationrelated decision making employed this MCDM method [1][2][3] [4].Selection of effective and efficient projectsis crucial for every organizationas the decision-making processes to assess the feasibility of a certain project are extremely complex.The research was conducted by employing the methods of AHP and Moora as the research approaches [1].
To cope with uncertainties and obscurity found in humans' subjective perceptionsin decision making processes, a Fuzzy Multi-criteria Decision-Making (FMCDM) based evaluation method was applied to measure the performance of the software development projects [2].What constitutes a problem in the MCDM is that it is the decision maker (DM) who have to choose which one is the best alternative that meets the criteria.Generally, it is not easy to an alternative that meets all the criteria simultaneously and thus a compromise solution was preferred.The problem's complexity may increase ifa number of DMs do not have the same perception relating to the existing alternatives.The VIKOR-based ranking method was proposed to identify such a compromise solution.This method used the suitable value for the alternative assessment with unquantifiable criteria, especially if the evaluation was undertaken based on the aspect of linguistics.Kazemi et al [3] offers a project supervision method in order that such projects are consistent with the strategic objectives.The initial step to diminish the risk of project failure is to choose an optimum project with the MCDM approach using AHP and TOPSIS methods.In another model, Linear Programming (LP) and MCDM for decision making were applied in the priority project selection evaluation based on a number of predetermined criteria [4].The analysis results indicated that MCDM can be used for evaluating project performance.
In Indonesian government agencies, especially in regional governments, there is a type of report called LAKIP, which is the Performance Accountability Report of Government Agencies, which serves as an instrument for measuring www.ijacsa.thesai.orgperformance of related agencies with regard to the extent of the successful implementation of their programs/activities.Unfortunately, this type of measurement is undertaken on a general basis with a variety of variables used, not specific to ICT.In another research, Ishak [5] examines the effectiveness of performance assessments in each SKPD (the Local Apparatus Work Unit).By using the analysis method based on a variety of data sources, it was concluded that the accountability of Indonesian governments remained focusing merely on financial management, while in the daily reality such financial information failed to answer public curiosity about government accountability and thus an appropriate measuring tool to measure performance of SKPD is necessary.Consequently, e-Government projects need to be evaluated to determine causes of the resulting changes, deficiencies, and irregularities [6].
This paper described a GDSS for ICT project performance evaluation in regional government agencies.This GDSS was used as a tool for decision makers to expand their capabilities, but not as a substitute for their judgment.Broadly speaking, this paper consists of several sections.The first one presents a brief overview of AHP, TOPSIS and Copeland Score.Then, the methodology, i.e. the measures to apply the hybrid method is described by also providing examples on the ways itwas implemented.In the final section, findings of the research that had been conducted are concluded.
Unlike the previous research, in addition to GDSS implementation using the hybrid method, the assessment criteria used were the ones that can be used for the assessment in any categories of ICT projects, not just limited to software and hardware related ICT projects.Moreover, to determine the assessment criteriato be used it is necessary to take into account the technical and managerial aspects in order to accommodate all the DMs.

II. THE OVERVIEW OF MULTI-CRITERIA DECISION MAKING
Based on the number of criteria used, decision related issues can be divided into two categories, namely singlecriterion decisions and multi-criteria decisions.The Multi Criteria Decision Making (MCDM) is defined as a decisionmaking method to determine the best alternative of various alternatives based on certain criteria [7].This MCDM is divided into Multi Objective Decision Making (MODM) dan Multi Attribute Decision Making (MADM) [8].

A. The Analytical Hierarchy Process (AHP)
AHP is a decision support model developed by Thomas L. Saaty.This decision support model will describe complicated multi-factor or multi-criteria problems in a hierarchy [9] [10].A hierarchy is defined as a representation of a complex problem in a multi-level structure where the first level is a goal, followed by the levels of factors, criteria, sub-criteria, and so on downwards with alternatives as the lowest level.A hierarchy helps to untangle a complex problem into groups which later are organized into a hierarchical form so that the problem itself will appear more structured and systematic.AHP has its own advantages as it has the ability to perform analyses in a simultaneous and integrated manner of the criteria, both qualitative and quantitative ones.Basically, the steps in the AHP method consist of: a) Defining the hierarchical structure of a problem The problem is decomposed into a hierarchical tree illustrating the relationship between the problem, the criteria and the alternative solutions.

b) Undertaking a weighting process of the criteria at each level of the hierarchy
At this stage, all the criteria in each level of the hierarchy are measured in terms of their relative importance compared with the other criteria.It can be done using Saaty's weighting standards with a scale ranging from 1 to 9 and its opposite.The scale used can be changed using other values in accordance with the condition of cases to resolve.Information about the scale used by Saaty can be seen in Table 1.The median of Criteriaiand j is between two adjacent decision values opposite ( ai,j = 1/ai,j) Criterion I has a higher importance value than Criterion j, thus Criterion j has an opposite value Based on the values of those criteria, the pairwise comparison matrix A can be formulated as follows: [ ] A i,j refers tothe element of Matrix A in the i th row and the j th column .

c) Calculatingthe criteria weightingand weighting consistency
At this stage, the weighting priority is calculated by looking for the eigenvector value of matrix A through the following processes:  SquareMatrix A. The value of the element of Matrix A 2 is determined using the following formula: ∑ ai,k refers tothe element of Matrix A in the i th row and the k th columnand ak,j refers tothe element of Matrix A in the k th row and the j th column.www.ijacsa.thesai.org  Add up the whole elements of each rom inMatrixA 2  until Matrix B is generated using the following formula: ∑ b i refers tothe element of Matrix B in the i th row.Matrix B is arranged by Element b i in the following pattern: Add up the whole elements of Matrix B using the following formula: normalization is undertaken to Matrix B to obtain its eigenvector value.This eigenvector value of Matrix B is described in the form of Matrix E as follows: e i refers tothe element of Matrix E in the i th row.
 Those three processes above are performed repeatedly and at the end of each iteration, the differential of the eigenvector values of Matrix E is calculated using the previous eigenvector values of Matrix E until an amount whose value is close to zero is generated.Matrix E obtained in the last step indicates the criteria priority indicated by the eigenvector value coefficient.

d) Calculating the alternative weighting
In this stage, alternative weighting is performed for each criterion in the pairwise comparison matrix.The process to undertake such alternative weighting is similar to that performed to calculate criteria weighting.

e) Showing the order of alternatives under consideration and selecting the alternatives
In this stage, the eigenvector values obtained in the alternative weighting for each criterionand the eigenvector values generated from the criteria weighting are calculated.This is done to determine the alternative chosen from all the available alternatives.

f) Repeating Steps c, d and e for the whole levels of the hierarchy g) Calculating the eigenvector value of each pairwise comparison matrix
Eigenvector values are the score for each element.This step aims to synthesize the element priority from the lowest hierarchy level to the goal attainment.

h) Examining the consistency of the hierarchy. If the value is greater than 10 percent, it means that the judgment data assessment should be revised B. Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)
The Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) is developed based on the concept that the best selected alternative should not only has the shortest distance from the positive ideal solution, but it also has the longest distance from the negative ideal solution [11].Generally, TOPSIS procedures are given in the following steps: a) Calculating normalization values TOPSIS requires performance rating of eachalternative of Ai in each normalized criterion of Cj, namely: (7) Description of the symbols: rij refers to the normalization value of each alternative (i) compared with criterion (j) where i=1,2,...,m; and j = 1,2,...,n.xij refers to a value of an alternative (i) compared with criterion(j) where i=1,2,...,m; and j = 1,2,...,n.

b) Calculating weighted normalization values
After calculating the normalization values, the next step is to calculate weighted normalization values by multiplying the value of each alternative in the normalization matrix by thescore given by decision makers.The following equation used is: Description of the symbols:  The positive ideal solution (A+) is obtained by searching the maximum value of the weighted normalization value (yij) if the attributeis the benefit attribute and the minimum value of the weighted normalization value (yij) if the attribute is the cost attribute.
 The negative ideal solution (A-) is obtained by searching the minimum value of the weighted normalization value (yij) if the attribute is the benefit attribute and themaximum value of the weighted normalization value (yij) if the attribute is the cost attribute.

c) Calculating the distance between each alternativeand either the positive ideal solution or the negative ideal solution
The distance between Alternative Ai andthe positive ideal solution is formulated as: The distance between Alternative Ai andthe negative ideal solution is formulated as: Description of the symbols:  The distance between Alternative Ai and the positive ideal solution (yj+) represented by the symbol Di+ is derived from the square root of the total values of each alternative obtained and the weighted normalization value for each alternative (yij) minus the positive ideal solution (yi+) and then squared.
 The distance between Alternative Ai and the negativeideal solution (yj-) represented by the symbol Di-is derived from the square root of the total values of each alternative obtained and the weighted normalization value for each alternative (yij) minus the negative ideal solution (yi-) and then squared.

d) Determiningthe proximity value of each alternative towards the ideal solutions (preference)
The preference valuefor each alternative (Vi) is given as follows: (15) Description of the symbols:  Vi (the preference value for each alternative) is obtained from the value of the distance between Alternative Ai and the negative ideal solution (Di-) divided by the total value of the distance between Alternative Ai and the negative ideal solution (Di-) plus the sum of the value of the distance between Alternative Ai and the negative ideal solution (Di+).
 The value of Vi which is greater indicates that Alternative Ai is preferred.

C. The Copeland Score
One of the common problems in the GDSS is the way to aggregate decision makers'opinions in order to make an appropriate decision.The methods of group decision-making (especially those related to MCDM) will usually experience problems if each decision maker gives their individual preference [12].In general, the GDSS consists of two stages in, namely stimulating decision makers' preferences separately and performing group aggregation towards any preferences given.
Among the tools used in the aggregation of group-based decision making is voting.Voting is defined as an act to select the most frequently appearing value among the selected alternatives [13].
Copeland score is one of the voting methods whose technique is based on a subtraction of the frequency of winning minus the defeat frequency of a pairwise comparison [13].Another research [14] describes the way the voting method of Copeland Score accommodates decision makers'score based on their respective level of expertise.The example of how to determine pairwise comparisons in the Copeland Score method is presented in Figure 1.B, C, D).This pairwise comparison is completedone by one and applies for the overall items of choices.
In the first rowrelating to the pairwise comparison between A and B (see the Table of Preference Profiles), 45% of the population prefers A to B; in the second row, 40% of the population prefers B to A; in the third row, 15% of the population prefers B toA.This implies that there is 45% of the population that prefers A to B, while the remaining 55% (the total number of the population that prefers b) of the population prefers B to A. Thus, B is chosen as the winner of a pairwise comparison between A to B. Comparisons are also made to other candidatesas described above.
The Pairwise Contest Table shows that as the winner, Option Aappears 2x (twice).Option B appears 3x (three times).Option D appears 1x (once), and Option C does not appear.

According to the Pairwise Contest Table, it is revealed that
Option A has two chances of winning over C and D, and a chance of losing to B. To determine whether Option A is the best choice or not, a subtraction of the frequency of winning minus the defeat frequency is performed.The Table of the Voting Results shows that Option B has the highest frequency.Based on the frequency, it is decided that Alternative B wins.

A. Classification of the Types of ICT Projects
The projects in the regional government agencies belonging to ICT projects are [15]:

B. The Implemented GDSS Method
Evaluation of ICT projects designedwas a model of Multiple-Criteria Decision Making (MCDM) using the methods in Multi-Attribute Decision Making (MADM).Determination of the best alternative among a number of alternatives was done based on several predetermined criteria.The scoring criteria to evaluate ICT projects were a compilation of project management concepts in general [16], ISO/IEC 15939 concerning how to measure software [17] and benchmarks that can be used to measure computer performance [18].Table 2 describes decision makers along with the parameters and criteria used in the evaluation of ICT projects.Evaluation of ICT projects in government agencies requires assessments undertaken by the Executing Parties of Government Institutions, ICT Management Work Units, Business Process Owner Units, and Society.Stakeholders of the ICT management as a group of decision makers have specified the assessment criteria based on performance indicators according to the duties and functions.Such performance assessments employed both qualitative and quantitative criteria, where the qualitative criteria used linguistic variables.These linguistic variables referred to variables whose values are indicated in the forms of words or sentences in natural or artificial language [19].
Then, to draw the conclusion relating to the ICT project results attained, the performance assessment scale based on the existing criteria was used.The measurement scale was developed based on the consideration of each decision maker.Table 3 presents the assessment scoring scale used in this research.

C. Scoring for each criteria is elucidated as follows:  Project Schedule
Based on the criteria of the project schedule timeliness, the percentage between the planned project scheduleand the actual project schedule [20].The ability to provide the agreed scope of duties concerning costs, hours of work, laboratories and travel expenses.Based on the percentage between the committed (baseline) and expected costs (actual + forecast) [20].
In this criteria category, the scoring used several linguistic variables, namely: Very Large, Fairly Large, Large, Less Large, and Not Large.

 Project Risks
These refer to the arising impacts of the risks, which are defined as follows [20]:  Critical: If this risk occurs,a project will fail.The minimum requirements of the project cannot be met.www.ijacsa.thesai.orgij i ij r w y   Serious: If this risk occurs, a project will encounter increases in terms of the costs/ schedule.The minimum requirements of the project that are acceptable can be met while the secondary requirements cannot. Moderate: If this risk occurs, a project will encounter increases in terms of the costs/ schedule.The minimum requirements of the project that are acceptable and a few of the secondary requirements can be met. Minor: If this risk occurs, a project will encounter slight increases in terms of the costs/ schedule.The minimum requirements of the project that are acceptable and some of the secondary requirements can be met. Ignored: If this risk occurs, it will not affect a project.All the requirements can be met. Project Performance, Project Effectiveness and Project User Satisfaction In this criteria category, the scoring used several linguistic variables, namely: Very Good, Fairly Good, Good, Less Good, and Not Good.
Each has its own performance assessment criteria indicated in a measurement scale.-w i refersto the score of each criteria (generated from AHP scoring) -r ij refers to the normalization value of each alternativewherei=1,2,...,m; and j = 1,2,...,n d.Identifying positive and negative ideal solutions (TOPSIS) e. Calculating the distance between each alternative and the positiveand negative ideal solutions (TOPSIS) f.Determining the proximity value of each alternative towards the ideal solution (preference) (TOPSIS) g.Repeating steps a to f for each Decision Maker h.Ranking all the DMs (TOPSIS-Copeland Score)

a) Performing criteria scoring(AHP)
The first step was to create a pairwise comparison matrix of criteria for DM1, followed by scoring the criteria.Then,the total value of a ij for each pairwise comparison matrix column was calculated as shown in Table 4.    e) Calculating the distance between each alternative and the positive and negative ideal solutions (TOPSIS) f) Determining the proximity value of each alternative towards the ideal solution (preference) (TOPSIS) g) Repeating steps a to f for each Decision Maker After the scoring process had been completed by each DM (DM1, DM2, DM3 and DM4), the following results of project ranking presented in Table 9 were obtained.The following are the stages ofvoting results for the best ICT projects:

 Preference Profile
The preference profile presented in Table 10 shows that there are a total of ten ICT project alternatives (namely P1, P2, P3, P4, P5, P6, P7, P8, P9, and P10).Each decision maker in the process of decision making had their own score which had been determined according to their respective expertise and competence.The score for DM1 was equal to 0.1, the score for DM2 was equal to 0.4, and the scoresfor DM3 and DM4 were equal to 0.2.

 Performing pairwise contests
A pairwise contest is defined as a paired-comparison process comparing one candidate (alternative) to the other candidates, which was performed by: -Displaying alternative contests in pairs.For example, P1 is compared with P2, P1 is compared with P3, and so on.Similarly, P2 is compared with P3, P2 is compared with P3, and so on.This pairwise comparisons are taken one at a time and done to all the options until P9 is compared with p10.-Searching for the winner of the comparisons (contests) of each paired alternative.-For example, in the contest of the pairwise comparison between P1 and P2 (see Table 11), the winner is P1 as in DM1, P1 ranks 1 st while P2 ranks 9 th so that the winner in DM1 is P1.In DM2, P1 ranks 10 th while P2 ranks 6 th , and as a resultP2 wins in DM2.In DM3, P1 ranks 2 nd while P2 ranks 7 th and consequently the winner in DM3 is P1.Lastly, in DM4,P1 ranks 1 st while P2 ranks 8 th and thus the winner in DM4 is P1.These imply that the rank of P1 is three times higher than the rank of P2, after calculating the scores of each DM it is revealed that the total score for P1 is equal to 0.1 + 0.3 + 0.2 = 0.6 while for P2 is equal to 0.4.Thus, P1 wins. Calculating the voting results The voting results present the results of voting (score) for each candidate afterpairwise contests, based on the following stages: -Calculating the frequency of winningof the candidates (alternatives) which had been compared using the pairwise contest.-Calculating the defeat frequency of the candidates (alternatives) which had been compared using the pairwise contest.-Calculating the differential betweenthe frequency of winningand the defeat frequency of the candidates (alternatives) which had been compared -Presenting the frequency differential obtained as a score for each candidate.The voting results are presented in order according to the ranking of the frequency-of-winning scores from the highest to the lowest one for the four DMs, which can be seen in Table 12.

V. CONCLUSION
This paper offered a hybrid method in MCDM to evaluate ICT projects in Indonesian regional government agencies based on the concept of Group Decision Support Systems (GDSS).
The GDSS concept can overcome any possible inconsistencies which may occur in decision making as it makes decisions based on the mathematical calculation model.Contributionsof the decision makers in the model were in the fom of preferences for choosing ICT Project alternatives based on predetermined criteria using the hybrid method of AHP, TOPSIS and Copeland Score.Based on the implementation examples, projects with the best rank were produced from the assessment undertaken by all DMs, namely Projects 1, 3 and 10 which had the same performance while Project 5 had the worst performance.
Our next research will focus on the development of a webbased prototype to implement the proposed model.The prototype developed attempts to provide an answer to the problems relating to ICT project performance evaluation in regional government agencies.

Fig. 1 .Figure 1
Fig. 1.Determination of pairwise comparisons in the Copeland Score method Figure 1 above presents three tables, namely theTable of Preference Profiles, the Pairwise Contest Table, and the Table of Voting Results.The Table of Preference Profiles suggests that there are four options, namely A, B, C, and D. 45% of the population prefers A to D, B, and C (see the first row of the Table of Preference Profiles).The Pairwise Contest Table indicates that an option (for exampleA) is compared with all the available options (B, C, D).This pairwise comparison is completedone by one and applies for the overall items of choices.

TABLE I
Table of Preference Profiles, the Pairwise Contest Table, and the Table of Voting Results.The Table of Preference Profiles suggests that there are four options, namely A, B, C, and D. 45% of the population prefers A to D, B, and C (see the first row of the Table of Preference Profiles).The Pairwise Contest Table indicates that an option (for exampleA) is compared with all the available options (

TABLE III .
ASSESSMENT SCORING FOR CRITERIA PERFORMANCE

TABLE IV .
THE PAIRWISE MATRIX FOR THE CRITERIA OF DM1

TABLE V
Based on the dataon the evaluation results given by DM1 on the criteria for each ICT project alternative, the following data on assessment results presented in Table6are obtained.

TABLE VI
The scoring of normalized values for DM1/ Matrix Y presented in Table8was obtained from the multiplication of the normalized value of each criterion in Table7by the normalized scoring for DM1/ Matrix R in Table6.www.ijacsa.thesai.org

TABLE IX .
RANKING OF ALL THE DMS