Comparative Study from Several Business Cases and Methodologies for ICT Project Evaluation

Achieving high competitive advantage through Information and Communication Technologies (ICT) has never been easy without proper management and appropriate utilization of ICT resources. Therefore, the statistics suggested that ICT project failures are very common in the organization due to several reasons; it fails to deliver the required objectives of investment, inaccurate budget planning, lack of risk management plan and time overrun are some basic reasons for an ICT project’s failure. To overcome these issues, recently ICT decision makers are emphasizing more on ICT project’s evaluation rather than investment. The practitioner broadly categorized the evaluation techniques in post and pre evaluation methods, which is further divided into measuring the return from financial and non-financial perspectives. The main purpose of this paper is to provide a comparative analysis on ICT investment’s evaluation, their categories based on pre and post evaluation. Thus, the paper offers an extensive literature review that can help ICT decision makers and organizations to better select the evaluation techniques available, where integration of multiple techniques can further improve this process. Keywords—ICT Investment, Evaluation of ICT Investment; Multi-Dimensional Approaches; Multi-Criteria Approaches; Financial Approaches


INTRODUCTION
Several studies have been conducted to discuss the issue of measuring ICT impact on organizational performance. These studies have proposed many methods, amongst which the most widely used are techniques which can evaluate the cost benefits with limited factors, or portfolio management methods which are specially constructed to provide preevaluation for new investments. Despite the fact that the research conducted in this study, is purely based on postevaluation, it is additionally important to measure the nonfinancial values produced from particular ICT projects. Another technique to support ICT investment in an efficient way is to develop a proposal through an ICT portfolio management method. It"s a pre-evaluation method, which enables analyzing several possible alternatives using graphical methods. The optimal project can be selected by visually comparing and analyzing projects [1]. The proposal-based techniques focus on business and technology domains and they make possible analysis using risk and sensitivity analysis based on previously implemented projects [2]. Yet another technique is the multi-dimensional approach which addresses assessment criteria differently from other methods, and is considered an important development in the field of measuring ICT investment [3]. The technique is basically an approach to evaluate direct and indirect benefits (business and technology) an organization can achieve post-implementation. Under this category several attempts have been made for measuring benefits where business and technology are inter-linked [4]- [8]. These techniques enable the ICT decision makers to measure the impact of ICT projects in a context-dependent manner.
To ensure successful value delivery investment, an ICT decision-maker needs to manage and focus on post implementation effects of ICT projects. Recognizing different categories of investment and associated objectives can help to evaluate it properly. However, each objective should define the key performance indicators and metrics, where the changes that occur in those metrics as determined by analysis are the real achievements after project implementation. This paper demonstrates the theoretical aspects around the ICT investment evaluation, further leading to the findings and gaps in this field. The next section highlights the current trends of ICT investment and ICT productivity paradox. After that the discussion of evaluation approaches and techniques used to measure ICT investments. This paper concludes at the barriers found in evaluating ICT investments. Finally, the paper concludes with discussion of overall findings collected through literature review. The extracted findings will lead to develop research framework and the construction of research methods for proposing framework based on the highlighted gaps found in previous studies.

A. ICT Investment Evaluation
The potential list of objectives from ICT investments, as discussed in the previous section, increased the organization"s perception to invest more in ICT. The extensive implications of ICT inside and outside the organization make its evaluation even more complex [9]. Evaluation is important to defend the large ICT investment in realizing the real impact of ICT on business performance [10], in developing future business strategies [11], improving the decision making process [12], and finally, assessing the business values generated from ICT investments [4].
Regardless of extensive literature, proposed approaches and techniques discussed in this paper, ICT investment evaluation is a complex and debatable issue due to measuring the impact and assessing benefits, most of the time being intangible [13]. Several techniques exist, but none of them are universally adopted as every method depends on the situation, www.ijacsa.thesai.org objectives and output of the particular organization [14]. Due to its disperse implications on business strategies, operational processes and cost reduction, the researcher demonstrates it as a complex task [4]. However, the evolution of assessing an ICT investment"s impact on enterprises is still growing with high demands from higher management to develop and enhance the monetary aspects of their ICT resources [15]. The literature suggests that ICT is still a big pillar for supporting production and business processes which can create a large difference in economic sectors [16]. Therefore, innovative ideas are still in support to overcome the difficulties and issues in measuring the impact of ICT investments on businesses [17]- [19].
The subsequent sections comprise to answer some basic questions in order to elaborate the ICT investment such as its importance, what to evaluate from ICT investments, different kinds of approaches and techniques used in measuring ICT investments and finally, highlighted issues in measuring ICT investments.

II. EVALUATION APPROACHES
ICT investment evaluation is a complex process for the organization to implement due to disperse impact of investment, multi-criteria decisions and the measurement process [20]. Reviewing the literature, numerous approaches proposed for evaluating ICT investments were found [4], [5], [17], [19], [21]. Researchers categorized ICT investment evaluation methodologies with different perspectives such as; financial, non-financial, multi-criteria dimensions found in literature review [3], [22]- [24]. This section discusses the wide ranges of evaluation approaches and methods developed with different characteristics as depicted in Fig. 1. Broadly, the approaches distinguished are based on the pre and post evaluation which implicitly indicates the time duration of the evaluation phase [25]. Pre-evaluation or predictive based approaches, also known as ex-ante or prospective approach, are for building ICT investment proposals to predict future achievable benefits and net incomes [26]. On the other side, some organizations developed the ex-post or retrospective approach for ICT investment"s post implementation"s measurement and potential ICT impacts achieved based on the organizational objectives [27]. Furthermore, approaches range from objective to subjective [28] or financial to non-financials factors [29]. Based on the literature review covered in this study, the following sub section involves discussion about evaluation approaches and techniques in detail.

A. Traditional Financial Approaches
Numerous financial approaches are developed for measuring investments in order to know the return from the investment. Individual or combinations of several financial methods have been used in researches for assessing the investment"s return [30]- [33]. In this section, specially talking about financial approaches has been used in measuring investments. Return on Investment (ROI) is considered to be the most commonly used method for calculating return from investment [29]. ROI is a financial model calculate ratio between the gained profit and invested amount for a given period of time of project using the following formula: stment CostOfInve stment CostOfInve vestment GainFromIn ROI ) (   [34] Where CostOfInvestment and GainFromInvestment, considering as total cost applied and total profit return respectively. There is a range of ROI methods, strategies, software and tools used to measure ICT investments. Using the track of ROI, several other related models have been proposed by researchers such as, Social Return on Investment (SROI) developed by Roberts Enterprise Development Fund and reworked by others [27], [35], [36] and Performance Reference Model (PRM) developed by US Federal Enterprise Architecture Program Management Office [37]. In addition, [38] proposed in his research a new ROI idea based on measuring ROI in infrastructure.
Another common method for assessing financial return based on a company"s assets is called Return on Assets (ROA), a method used for finding out the utilization of company"s assets in order to get the status of a company"s earning profit and if it is quantifying the assets or not. In [39] (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 7, No. 6, 2016 422 | P a g e www.ijacsa.thesai.org definition, ROA shows a signal of how profitable a company is related to its total assets. It is calculated by using the following formula: s TotalAsset NetIncome ROA  [39] Where NetIncome is the amount which remains after subtracting all cost of raw materials, expenditures etc., it"s also known as net profit. TotalAssets is associated with cost of property, resources, materials, and all types of belongings while ROI calculates the return on a whole invested amount, but ROA is specially used for analyzing and measuring that. Earning profit is justifying the company"s total assets. Individually, ROA doesn"t provide a comprehensive measuring of the investment.
These financial methods provide an analysis very quickly by using few parameters and fixed formulas. However, applying individually can be used for different purposes such as financial return, feasibility reports, ranking between multiple projects, etc., but has been criticized due to several reasons. Using traditional financial approaches such as ROI and ROA, does not always provide a complete understanding and attainment and cannot be considered as a tool for comprehensive evaluation of ICT investments [4]. ROI only is not feasible for measuring return on investments due to its uncertainty, invisibility and difficult decision process in ICT investments [43]. [44] discussed the tangible and intangible benefits that can be achieved from ICT investment, describing that ROI provides a limited analysis for measuring return and is only used to measure hard or tangible benefits. It is now well understood that these financial approaches are not good enough to support measuring ICT investments in other nonfinancial perspectives such as; strategic benefits and indirect cost analysis [29], [45], [46]. Taxonomy of some common financial approaches designed by the author is presented in Table 1.

B.
Portfolio Management Techniques As discussed, the limitations in the financial approaches move the thinking of decision makers in finding alternatives for comprehensive evaluation techniques. An alternative which can deal with non-financial benefits include financial return as well. Portfolio management techniques are a kind of approach used for building pre-investment systematic project specific metrics to target the expected list of benefits to be achieved from investments as discussed in a different article found in literature review [1], [19], [24], [47] . Giga Information Group is one of the pioneers in this area developed by IT portfolio management to analyze and plan IT decisions prior to the investment [48]. It works through building proposals by analyzing the possible impact of IT on business strategies and business operations. The techniques are based on different phases during the discussion of the IT portfolio including; IT/business alignment, resource management, IT performance and reporting and etc., [48].
IT portfolio management is a kind of application used for managing and planning investments in an organized manner for all activities, processes and projects in the organization [2]. In the same way, Ross and Beath proposed IT portfolio framework, using four kinds of investment types to cover the impact of IT investment on different types of assets and services in phases [49]. It"s a prospective approach that helps manage a case study by aligning IT and business strategies. The quadrants used in this approach are defined as; transformation investment, renewal investment, process improvement investment, and experiments investment. The purpose is to allocate a proper budget plan for each category while impact assessment can be more specific in this way [49].
An ICT project portfolio management technique helps in selecting the optimal project that can provide good return from the investment, but still has some limitations due to its complex selection procedure. Portfolio analysis offers an approach to prioritized multiple project investments based on different factors such as; maximization, balance, strategic alignment and resource balancing, which highlight the complexities involved in portfolio management techniques [50], selection of the project based on the feedback and historical financial data for any product. The forecast analysis for selecting a project doesn"t assure the optimal return on investment. Finally, IT portfolio management is a well-known technique, but is practically not accepted and implemented in -an analytic approach used for examining a new business investment -the analyzed benefits of business related investments/pro jects summed together, and subtracted from the required cost -addition with other non-financial approaches -inclusion of more aspects of post evaluation www.ijacsa.thesai.org the organization on a regular basis [51]. A summary of ICT portfolio management techniques is illustrated in Table 2.

C. Multi-Criteria Approaches
Multi-criteria approaches are another kind of approach that can be used for assessing ICT projects pre and post evaluation purposes. Information Economics is a method which is considered a grandfather in measuring weight and rate of IT value management approaches [3]. It"s highly intensive and implemented in such a complex environment such are those running in the federal government. It is used for defining business cases by providing different financial analysis such as; enhanced ROI, IRR and NPV. In addition, it also works for analyzing non-financial business impacts such as; economic impact, business domain assessment, and technology domain assessment [53]. Douglas W. Hubbard proposed an enhanced version of the multi-criteria approach as Applied Information Economics by including several other factors such as; option theory, actuarial science, measurement science, etc., [54]. It also helped measure the difference between the current state and expected value achievement. Total Economic Impact [55] is another kind of multi-criteria technique proposed by Giga.
Multi-criteria approaches provide analysis better than traditional financial approaches. A combination of financial and non-financial factors makes it a strong selection for measuring ICT investments. As shown in Table 3, most of them help in selecting optimal investments from multiple projects prospectively. Post evaluation and addition of intangible factors associated with ICT and business domain can enhance the capability of assessing the ICT investment"s impact on business.

D. Multi-Dimensional Approaches
This field of study has shown remarkable improvement as many scholars have presented multi-dimension approaches, including financial and non-financial factors. Obviously, most of the investments are connected with financial values, but the ICT impact is over organizational, operational, strategic and service values are important as well as shown in Fig. 2. [1]. Several frameworks were proposed in order to cover a maximum list of values based on the organizational investment"s objectives. Australian Government Information Management Office proposed a tool for measuring demand and value assessment methodologies [26]. This technique helped plan well before the investments through outlining the business case with estimated values to be achieved. Financial values, risk analysis, social and governance values are the major variables used in this method.
ICT is not just a tool for keeping and promoting organizational changes. To know its impact other than cost benefit analysis, there is a need of measurement tools for assessing productivity and customer satisfaction [56]. The IT Governance Institute (IGI) developed a framework work to ensure that IT investments fully managed and generated expected values to all stakeholders. This is an ex-ante approach developed to help the organization in defining ITbusiness cases based on value governance, portfolio management and investment management [57]. Measuring the value of ICT investments is strongly connected with different variables such as; business processes and work practices which ultimately can be assessed through productivity increase and by reducing cost [58]. In the same way, the Performance Reference Model (PRM) designed a framework to learn the performance and manage the ICT portfolio in a better way. The framework was specially built to measure the ICT investment alignment with strategic objectives [37]. The summary of multi-dimensional approaches extracted from literature review, with detailed descriptions for each one provided in Table 4.   TABLE II. TAXONOMY OF MULTI-CRITERIA APPROACHES www.ijacsa.thesai.org Multi dimension approaches are considered better evaluation approaches as compared to traditional financial approaches [4]. The point is to correctly identify the impact of ICT based on the organizational objectives for which investment is allocated. This study also focused on multi dimension approaches for measuring the ICT investment based on the investment"s objectives linked with direct and indirect benefits and finally, to assess the business values associated with ICT investments. Discussion for each of the selected researching we are emphasizing in this research is discussed in section 2.4, related work.

III. ISSUES IN ICT INVESTMENT EVALUATION
During the last two decades, several discussions and developments have been presented by various scholars and practitioners in regards to ICT investment evaluation. Some evaluation models have been developed for a specific organization"s structure [27], [57], [59], [61], [62] while others have only considered limited factors for the evaluation process [34], [40] which is not suitable for the comprehensive evaluation process of ICT investment. As discussed earlier, ICT investments involve different objectives and reasons, such as mandatory IT, strategic IT, transformational IT, and new infrastructures [63], [64]. [65] described several reasons for ICT project failure, including the lack of proper evaluation techniques being applied to measure ICT investments. Therefore, there is still a need for general purpose, comprehensive and easy-to implement evaluation models and methodologies based on financial and non-financial impact factors which need to be evaluated concurrently.
Moreover, the ICT investment evaluation performs in several transitions, such as on the IT transformation phase [66] and IT implementation and development phases [67]. Based on the review of previous literatures and research, the overall process of ICT investment evaluation is not an easy task due to its scattered impact and implementation for different resources. As discussed by [29], an organization measuring return on ICT investments is complex and requires a thorough understanding and knowledge of both (i) processes involved in the business and (ii) the environment in which they are running. Therefore, it is essential to know the relationships among the risks, benefits, and costs of ICT investments as well as multidimensional environmental factors, including societal, organizational, and institutional issues [29].
ICT evaluation involves several stages and branches to be evaluated simultaneously during the allocated evaluation period. In growing organizations, investments in ICT comprise of application software, system software, programming languages, communication, hardware [68] equipment, services, and other technologies [69]. Although the technological aspect in the evaluation process is the major phase, this cannot ignore the management, services and product aspects of the investment. It is easier to calculate and measure investments using variables like cost and expected financial return, but risks, benefits, and services create many hurdles in the evaluation [70]. Although it is significant to quantify IT asset"s value, especially in IT service-related contexts [38] to improve customer satisfaction, the measurement of ICT investments is still progressing by building and developing models and methodologies to support organizational decisions. The following issues are involved in measuring ICT investments, as illustrated by [71]:  In IT investments, there are lots of benefits of an intangible nature.
 IT investments sometimes lead to long-term benefits.
 Sometimes benefits of IT investment are indirect; these can be evaluated by using several complex matrixes and factors.
 The theories and techniques available are somewhat unsuitable for understanding and capturing the business value of IT.
The literature review supports that the complete process of evaluation of ICT investments is still under progress which requires a thorough understanding of multi-dimensional factors where the business value and cost return may exist. [29] suggests that if any organization calculating returns on investment from ICT investments may be considered a complex process and needs thorough understanding and knowledge of all kind of processes involved in the business and environment where they are running. There are different kinds of models developed together with multiple aspects, but decision makers are still looking for completed and reliable methods to assess their decisions. [20] stated that to assist the ICT decision makers, there are no dependable, reliable, and optimal techniques, which can evaluate their decisions as well as investments.

IV. CONCLUSION
This paper highlighted the importance of ICT projects in an organization to better run their business process, to get competitive advantage and reasons for using technology oriented business model for more benefits. However, the literature review suggests that ICT project can fail due to several reasons. Therefore, using of proper pre-evaluation techniques help decision makers to plan well, and post evaluation may provide the analysis on utilization of ICT resources. www.ijacsa.thesai.org ICT projects evaluation is a complex and multidimensional process needs extra efforts, integration of different dimensions and techniques based on organizational requirements. The research findings suggest that ICT investment is not self-governing projects, all of the investments are to support business process and structure of the organization to run in more efficient way. Combination of business and technology is the real motto of current organizational scenario. The better way of measurement can lead the organization towards best utilization of ICT resources. The list of measuring techniques showed that it is a multipart procedure divided into pre and post evaluation which is further divided into several others methods. The number of researches presented in this paper increased the importance of this research area, where practitioners and academicians both are still progressing for standard method or tool to evaluate the ICT project. In future, evaluation of ICT projects using real case studies through different kinds of measuring techniques discussed in this paper can improve the idea of ICT investment evaluation.