General Characteristics and Common Practices for ICT Projects : Evaluation Perspective

In today’s business world, organizations are more dependent on Information and Communication Technologies (ICT) resources. Cloud services, communication services and software services are most common resources, enterprises are spending large amount. To install new services and upgrade existing services, ICT project are essential part of organization’s business strategies. Researchers highlighted the real problem for the organization is to initiate new ICT projects and its evaluation after implementation. This research investigated the common approaches organizations using to start with ICT projects and how to evaluate its impact on after implementation. For this, we have extracted the number of steps with the help of literature review. To validate those steps, six case studies are selected for collecting the samples. The findings of this study elaborate that every ICT project has list of objectives i.e. strategic, informational, IT infrastructure and others. Furthermore, the results highlight that organizations believe on both financial and non-financial evaluation methods based on the type of organization i.e. public or private. Moreover, measurement process applied on project wise, monthly and yearly bases. Importantly, we have found that currently outsourcing plays significant role in success of ICT projects. The results of this study can be helpful for the organization to understand the type of ICT investments, approaches and possible impact on the organizations goals. Keywords—ICT project; ICT evaluation; measurement process; case studies; common practices


I. INTRODUCTION
Information and communication technologies (ICT) are considered one of the main pillars in building business architecture in any organization.The ICT has been categorized in different perspectives such as; hardware and software resources [1], IT infrastructure [2], cloud computing [3], information management tools [4] and different kinds of information systems [5]- [7].In order to improve the services and business processes, organizations intend to plan, build and implement different kinds of ICT projects every year.Based on the report published by Gartner 1 the ICT spending has been reached to billions of US Dollars.Although, the main purpose of organization to spend large amount of ICT investment is to improve employee productivity [8], customer satisfaction [6], [9], enhance data management [10] and last but not the least, to align ICT resources with business strategies [11].
Due to this large investment on ICT resources, measuring the performance and underutilization of ICT resources are 1 www.gartner.commajor concern of the enterprises.This research is actually highlights the common practices for ICT investment and measurement process practicing in an organization.In addition, how to measure the benefits from already implemented ICT projects.The first part of this study is to highlight the ICT project investment process.Normally, ICT investment are implemented based on list of objectives such as; to improve information management process [10] enhance the scalability of IT infrastructure [12], [13] to enhance business structure/process transformational phase [14] or to increase the transactional capabilities [11].Whereas single ICT projects can help to achieve one or many objectives based on its implementation.
Second major aim of this study is to investigate the common approaches enterprise following to measure those kinds of investments.Evaluation of ICT project's impact on business values/benefits is a complex process and involves multiple stages [15].The vast literature review suggested number of approaches, techniques, and phases required to accomplish this task.Each approach has its own characteristics and context to be used, which is based on the type of investment and list of expected benefits for which a firm has initiated in the ICT project.Due to complexities involve during evaluation process, organizations are still struggling to know the optimal technique for ICT project post evaluation.Therefore, the idea in this study is to investigate the current status/procedures of organizations, following for ICT project implementation and evaluation.To do this, number of questions extracted from literature review related with ICT projects investment and evaluation.Furthermore, the questions have been asked from major representative of the selected organizations.The extraction procedure of the topics and details of case studies are presented in Section 2. In addition, Section 3 discusses the major findings, limitations and analysis on collected data.

II. ICT INVESTMENT AND MEASUREMENT PROCEDURE EXTRACTED FROM LITERATURE REVIEW CONFIRMED THROUGH CASE STUDIES
Researchers elaborated ICT investment and evaluation in different ways.There are number of researcher published already supported the idea of using case studies for ICT investment and measurement the post implementation benefits [4], [10], [11], [14], [16].This study highlights the organization's point of view in finding out the objectives behind every ICT investment and how organization evaluates the performance of those projects.This section categorized in eight sub-sections, whereas each sub-section aimed upon the www.ijacsa.thesai.orgmajor topics extracted from literature review and further asked in case studies.Section A discussed about case study characteristics and participants overview.An overview of ICT usage and motivation toward projects in the case studies presented in Section B. Section C elaborate the common practices for ICT investments and evaluation found in case studies.Generally, ICT investments are consisting of different kinds of input and output resources as extracted from literature, discussed in Section D, while Section E, described the major objectives behind ICT project investment.Moreover, Section F talks about types of measurement methods examined from case studies.The case studies are following which kind of evaluation approach inspected in Section G. Rather they are outsourcing their investment and evaluation using third party assistance discussed in Section H.

A. Case Study Characteristics
The discussion of the results begins with the summary of the six case studies from Saudi region, selected in this research.It especially helps us to understand the participation of each case study, number of employees, and their experiences in ICT field.In order to ascertain the appropriateness and guarantee of the data sources as well as participants from each case studies.The participant were majorly selected for this study are top management, IT project director, supervisors, software developers, IT executives, IT project managers, IT project team members, IT and business users.The number of questionnaire received with the percentage of whole, associating from each case study as depicted in Table I.The sample size is better than the size used in related studies which was 143 [10].Also the sample size meeting with the condition explained by [17] that the minimum of 5 questionnaires are essential for each variable.
Table I categorizes the ratio of participants under each case who have recorded their responses.It highlights the high responses collected from airline and University is 20.5% and 17.9%, respectively.While other four cases fall around 13.8% to 16.4% which altogether counted as fair average of responses per case.The variation in collected responses collection was based on data-collection timing and other circumstances inside the organization.In order to gauge the size of the organization, a question was asked about the approximate number of employees working at an organization, as demonstrated in Table I.It appears that a large number of responding firms (38.4%) have the largest number of employees as compared with all six case studies.This indicates that the number of participants taken the part in this research belonged to the company, which has more than 1000 employees.On the other side, 13.8% of the participants associated from the bank that have lowest number of employees as compare with all other companies selected in this research.The contribution in this research was essential, especially from participants who have good experience in ICT.Therefore, a question was included to obtain the information about their experiences in this field.It is apparent from the Table II that 43.32% of participants have the experience of between 6 and 10 years, while 30.3% were part of this field since around five and less years.The indicators of experience highlight the significance of the collected data, as 69.7% of participants have been associated with the ICT field.

B. Motivation towards ICT Project's Investments
Recent developments in ICT have heightened the organizational vision to invest more and get more benefits in terms of profit, intact values to the business, and eventually more customers.As from the literature review, researchers have found the usage of ICT common in almost every organization.In [18], highlights the major spending in ICT categorized by sectors.We tried to confirm those sectors of ICT from the companies, whether they are investing and using those resources or not.Table III proves the findings of literature review, where all selected companies have been investing and using different types of resources mentioned in the table.The advanced technologies always put pressure on the companies to invest more and keep updated the ICT resources, all of the participants agreed upon it.They refused to provide the particular amount for each sector; rather they provided their answers in three categories -High,‖ -Medium,‖ and -Low‖, which highlights how frequently they invest in different categories of ICT resources.www.ijacsa.thesai.orgIt is evident from the table that different companies have different objectives for ICT projects, as mentioned earlier.Airlines, university, and banks are investing in a high amount for data center systems, as their businesses are more dependent on data integration and validation.On the other side, food industry, water and electric, and telecommunication are investing in a high amount of purchasing devices to support their business processes.Data integration, improved communication, and many others are the significant purposes for which companies are investing on ICT [19].Based on the ICT investment and their related tangible and intangible benefits, the organization faces internal and external pressure to invest in a particular sector, where each sector can provide a selected list of benefits, as mentioned in Table IV Receiving more benefits is another motivational factor investigated from case studies, which keep creating stress (internal or external) on the organization and highlight reasons for investing on ICT.The result is based more generally on which types of benefits are based on ICT investments in a particular organization.The telecommunication company, which mainly considers ICT services providers, received the most benefits out of it.On the other side, -training‖ is the least interested item in the case studies that are associated with ICT investment.
Table IV highlights for what purposes these organizations invest more generally.The list of intangible and tangible benefits they can achieve from ICT projects.For example, technology update is very common tangible benefit which organization needs to update their software and license every year.For this they can feel pressure from internal (employees) or external (customers).On the other side intangible benefits such as process efficiency is required when system are running out of order several time during transactions.Most of the organization agreed that for getting those types of updates we always feel pressure from their employees, partners, collaborators and customers as well.

C. ICT Project's Investments: Common Practices in Case Studies
An investigation was conducted to explore the common practice in case studies for each ICT project's investment they want to implement.The findings in Table V can be used as a base for drawing general conclusions on the ICT project investment practices in the investigated organization.The analysis reveals that most of the organization is invested mostly based on projects, where a project can for strategic, informational, or other purpose [11] and summarized in Table VII.Specifically, the studied cases invested in three kinds of resources technology, human and relationship resources as described in [20], although all of them are critical for investing in technological resources.
The investment needs to be evaluated to find out the return from the investment, financial and nonfinancial types of methods used for evaluation in ICT investment in different cases.University, as being a government organization, is not supportive for measuring financial returns.Altogether, organizations are using different types of approaches, where ROI is being used most except in University.On the other side, airlines, banks, telecommunications, and university are using multidimensional methods as well for measuring business values or nonfinancial factors as a measuring return from the investments.As extracted from literature review post implementation measurement fall in two different categories; first return on investment (ROI) using non-financial benefits [10], [21]- [23], while second one is by evaluating using nonfinancial measuring factors [4], [11], [14].It is understood from the case studies that post-multidimensional (for measuring nonfinancial return) and ROI (for measuring financial return) are the most common approach organization for using ICT evaluation.In addition, airline and telecommunication also mentioned the approach of building proposals using the pre-evaluation method such as IT portfolio management.www.ijacsa.thesai.org

D. ICT Project's Investment: Input Resources and Output/objectives
ICT investment is about implementation of resources for a particular sector to support the enterprises.Referring to [20] the input resources are of three types; human, technology and relationship, while each input resource generates multiple output [20].Table VI is the combination of input resources and corresponding output generated from those resources investigated from case studies.All of the studied companies are investing in human and technology type of input resources to improve their company's performance and technical skill for their employees.The integration of technology and human resources create IT business value for the organization.Relationship resources are defined as those factors that can increase the business value of the firm.Corresponding on this point, airlines, University, banks, and telecommunication are the companies who are measuring business values from nonfinancial perspectives.The major factors used for evaluating relationship resources are accountability, leadership, and organizational learning.In human resources, none of the organization is relating to ICT investment for supporting business understanding factor.Most of the participants do not relate ICT investment with business understanding perspectives.
The two approaches defined for measuring ICT investment are based on cost return (financial factors) and business value return (nonfinancial factors).All studied case studies are invested in human and technology resources.Technical resources can be evaluated using return cost factors, but human resources need to be evaluated though nonfinancial factors such as IT staff reduction, employee involvement, etc. Table VI indicates that all organizations are investing for human recourses such as for improving technical skills, but it was shown, as in Table V, that only airlines, University, banks, and telecommunication are using non-financial approaches for evaluating their investment.Food Industry and water & electric needs to employ such nonfinancial techniques for evaluating their human resources more properly.

E. ICT Projects Objectives
ICT projects based on different types of objectives to accomplish particular goals are based on organizational requirements.The literature review suggested that ICT practitioners have placed them into eight categories [10], [11].The current study consolidated those objectives for basic requirements while measuring ICT investments examined using case studies, as illustrated in Table VII.Competitive advantage and aligning of ICT strategy with business strategies are common objectives, which fall under the category of strategic types of investment.Most of the organizations relate ICT investments with their strategic objectives.Another objective investigated is informational type of investment on which all case studies regularly invest.The purpose of this investment is to enable the data to be accessed easily and faster.Information accuracy is the fundamental requirements of each case study examined.IT infrastructure is the most common term, which normally considers a pure ICT investment rather than some scholars, who have excluded it from the list of dimensions, which can create business value of ICT [10].For the purpose of reducing operating and communication cost, airlines, food industry, water & electric, and banks are doing ICT investments.They mentioned that installing advanced technologies, web servers, and data storage servers help to reduce operating and communication costs.This type of investment helps them to reduce work load and time consumption while performing different types of activities.While University and telecommunication did not mention this type of investment, they do regular updates to support transactional benefits as well.Most of the case studies did not provide enough evidence for ICT investment's impact on organizational types of benefits.They are not specifically investing for getting benefits for organizational support such as work pattern, empowerment, and building common vision.Organizational learning, business understanding, and creating common vision are not really supported by investing on ICT, which most of the participants agreed upon.Only airlines and banks mentioned investment for organizational learning to increase employee morale and satisfaction toward business goals of the organization.Moreover, based on collected evidence, most of the time, telecommunication offers an investment plan based on a new service or enhancing previous services for their customers and employees.In summary, the case studies offer different types of aforementioned investments to support their business objectives and goals.They have shown their high interest toward ICT, thus having a positive impact on different types of category for creating business value.

F. ICT Project's Evaluation Status: Common Practices in Case Studies
ICT project's evaluation for large enterprises is an iterative process.There are different types of measurement they are following based on the evaluation requirements.Refining the findings from literature review the evaluation categories are known as financial and non-financial which further divide depending upon the time when organization are evaluating it before or after ICT project investment [21], [24].The combination of type of measurement, time of measurement, and how frequently they are evaluating their ICT investment at cases studied are depicted in Table VIII.
As per the discussion, measuring through financial factors is the most commonly used technique we have found in case studies.University, which is a government organization, is not interested in financial returns of any investment.The budgeting and funding allocated per year by government, which is the main reason they are not acquiring return on investment (ROI) from any ICT projects.University is sometimes for nonfinancial measuring methods using feedback forms or by using other surveying methods, they examine the thinking and views of the stakeholders for any particular ICT services already offered.Airlines, banks, and telecommunication are also performed nonfinancial measurement factors using different factors such as knowing the competitive advantage in the market through information retrieval, workflow, and process efficiency.
Organizations believe in ex ante (pre) and ex post (post) evaluation based on criteria of measurement.From the literature review, we have extracted different types of measurement techniques offering pre-and post-evaluation [22], [25]- [27].ROI is most commonly used method for measuring return on investment considered post-evaluation methods for assessing hard benefits or financial returns, to know exactly the cost and benefit ratio.Pre-and post-evaluation are common in each case study, while pre-evaluation is time-consuming and assessment is based on previous documents or reports and predicting the future.Risk measurements and possible outcomes discussed in pre-evaluation process during the planning phase of every new ICT project.On the other side, post-evaluation is based on the results and outcomes achieved already.For the given time period, proper measuring factors can provide the analysis and achieving objectives from the invested ICT project.Food Industry and water & electric case studies have mentioned investing on technology resources (Table VI), which can easily be assessed using financial methods to know the return of each ICT project.Some of the resources such as human resource have to employ techniques that can measure hard benefits as well.Airlines, banks, and telecommunication based their objectives believing in pre-and post-evaluation using financial and nonfinancial factors.This means that each of their investment has provided risk assessments and expected cost, which ultimately generates expected benefits in the form of return cost and business values.
The frequency of the ICT project measurement is investigated in the organization, where most organizations evaluate their investments project-wise.As airline, food industry, water & electric, and bank investing is particularly based on the ICT project with lists of objectives and input resources.It has been explored from the literature review and acquired through case studies that it makes it easier to measure ICT project return project-wise, as organizations are well known about objectives of investment, resources, and expected outcomes.Their mapping can provide enough evidence for measuring financial return as well as business values.Apart from an University-use yearly basis approach for ICT evaluation, where the return is measured using nonfinancial factors such as quality of University, faculty member, and course surveys, feedback for e-services facilities is provided to different stakeholders.

G. ICT Project Evaluation Approaches
Four major approaches used for evaluating ICT investments extracted from literature review are traditional financial [21], [28], IT portfolio management [29], [30], multicriteria [31], [32], and multidimensional [11], [15], [27], [33].The most common financial techniques such as ROI and ROA are methods used to measure return on post-investment.It will provide the analysis using variable spending amount versus return amount after a given period of time.Portfolio management is the kind of approach used for building a preassessed plan for new IT investments.This technique is used to provide multiple options for new investment while mapping the investment objectives with expected list of outcomes.On the other side, multi-criteria (mostly pre-assessment), and multi-dimensional (mostly post-assessment), which has similarities in the sense of nonfinancial factors.
In addition to the previous discussion section, Table IX indicates investigation on case studies in order to understand their priorities in using ICT evaluation approaches.The evidence from the case studies highlights the most common approach they follow is the financial approach.This kind of method can provide some part of analysis for particular investment to get only a return amount from the investment, but it cannot provide comprehensive analysis regarding business values as criticized by ICT practitioners.Most case studies also rely on a financial approach for specific reasons of knowing profit ratio; otherwise, all of the studies use other approaches as well.
The portfolio management approach used by airlines and telecommunication for some specific ICT projects is joint venture with an outsource agency (Table X) such as ICT services for cargo and catering in airlines and transmission and routing services in telecommunication.Those case studies indicate that the portfolio management approach from an outside consulting agency is feasible where partners are involved.University is a government profile, most of the time depending on multi-criteria and multidimensional approaches to get the value of university assets and resources from a stakeholder's point of view.Food Industry and water & electric are exceptional in this list by using financial approaches mostly for measuring return on their ICT projects.www.ijacsa.thesai.org

H. Third Party Assistance in Investment and Evaluation in Case Studies
Innovation in technologies is happening every day; companies that cannot cope with these changes cannot survive in the market.Organizations have to keep the pace of technology change and regularly upgrade requirements of users and market as well.ICT project outsourcing is based on several reasons; it may be because of low storage so hire cloud computing service and relative cost of outsourcing is cheaper than in-house building, and also time flexibility [34].The companies can outsource any ICT service due to any reason as discussed above.
The investigated case studies explore that every company, other than in-house building and maintenance, also believe in outsourcing ICT projects as shown in Table X.Notably, some companies mentioned their contract such as in airline system catering and cargo outsources, where all ICT services related to those departments are outsourced, too.In university learning management system, course registration, and other administrative system are outsourced.They discussed saving their time and cost, and it's better than in-house building.For evaluation purposes, airlines typically hire consultancy agencies to evaluate their ICT service performance; they also use help for building proposals for new investments using third-party assistance.On the other side, university typically use an in-house evaluation process; they use some tools and survey instruments for measuring services most of the time at the end of each calendar year.Food Industry, water & electric, and banks and telecommunication take assistance from third parties to outsource systems for their daily routine work.Telecommunications involve collaborative work to boost their services and improve their channels of data communication.Cloud services and data banks are outsourced by food industry and water & electric, while banks have taken assistance for their online services.As far as evaluation is concerned, only telecommunication asked for third-party assistance to help in evaluating some services while others not.Outsourcing of ICT investments and evaluation can be a better choice for the organizations because it can typically save on time and cost.Most companies seek third-party assistance due to lack of in-house expertise and resource availability.There are some disadvantages for outsourcing, which is the risk of data exposed to outside sources.Switching from one vendor to another can have some complexities and loss of money.Outsourcing will also allow outsiders to learn business models and processes of a particular organization, which can be risky for them.Pre-evaluation planning and assessment approach can be feasible to be outsourced because new development can progress under the supervision of professional ICT experts and consultants offering solutions to different business problems.

III. DISCUSSION
Investigation into the above-mentioned studies had drawn conclusions on measuring business value of ICT projects; thus, investment and evaluation can be realized.Two ICT investment approaches are common practices in six case studies; the companies are investing project-wise associated with the list of objectives or up-gradation on a regular basis using a time frame.In both ways, the finding suggests that, behind every investment, there are specific objectives.Four major objectives strategic, informational, transactional, and IT infrastructure are the main reasons associated with ICT investment.ICT investment can have an impact on those types of a firm's objectives.This study showed that ICT has a significant role and usage in all case studies.The major sectors found in this study are data center systems, IT services, communication services, and IT devices.The majority of respondents said that the investment was motivated by internal pressure while airlines, banks, and telecommunication also highlighted the exceptional cases where external pressure stresses them to invest on ICT resources.Internal matters such as improving access to the information, data server, web servers, and maintaining data warehouses, and external matters such as suppliers, customers, partners, and online transactions are basic motivation to invest more on ICT resources.www.ijacsa.thesai.org The most popular resources companies acquire in this study are technology resources such as hardware, software, ICT applications, and databases where firms invest.Human resources related with ICT are another indication for which the proper budget is allocated to provide technical and support facilities.The relationship and integration between ICT and human resources are the most common method that can generate ICT business values.The most popular approach for measuring ICT investment in our study is a traditional financial approach.Every organization except University uses financial approaches to measure return on investment for specific ICT resources or projects.In the category of a nonfinancial approach, multidimensional practices are used in the case studies.The process of the nonfinancial approach is based on the objective of the investment.Thorough understanding of objectives and expected outcomes can elaborate measuring factors.Time management, process efficiency using process life cycle, technical skill, employee productivity, response time, competitive advantage are major factors used in a multidimensional evaluation approach.
The time and approach of evaluation is critical for examining the case studies.With respect to the time of evaluation, the two common approaches investigated from the case studies are pre-and post.Pre-evaluation and postevaluation have different purposes.Pre-evaluation, which is mostly used in the planning phase, is where companies try to build a portfolio of new investment with risk analysis, the list of objectives, and expected outcome of the investment.The portfolio provides alternative options to select the optimal and best characterized project.The preliminary work helps them to identify outcomes, and it helps them to link project objectives.All participants agreed that planning and portfolio building can take long time for pre-assessment, where urgent implementation of ICT projects cannot afford this type of preevaluation.
On the other side, post-assessment methods help a business to identify the list of benefits achieved after applying an ICT project.The multidimensional post-evaluation method, which is most commonly investigated in the case studies, is measured through several stages.Participants highlighted several questions that need to be investigated in the multidimensional post-evaluation method.It is time-consuming but depends more on initial investigation; however, this approach is better for knowing actual values (nonfinancial) acknowledged by the receivers.The outsourcing of ICT projects is an attractive option in today's turbulent business environment.All case studies include private and government organizations to support their choice for outsourcing the ICT projects as per requirements.It will allow them to activate service on time to accomplish the goal of particular strategy.The approval depends on ICT decision-makers and executive-level management.Accordingly, some organizations hire outside consultants or assessment firms to evaluation their projects.The disadvantages have also been mentioned from all participants are in regards to privacy and data exposed to an outside source, which is an ultimate risk for the organization.

IV. CONCLUSION
In this research, researchers have reviewed the most common approaches used for ICT project's investment and evaluation extracted from literature review and investigated in six companies.Based on the similarities, the investment type is categorized as project based (strategic, informational, and so on) and regular upgrading on specific time.The most common input resources are based on technology and human and relationship resources.Finding the most common practices for ICT project evaluation type is essential, which is investigated in case studies as financial and nonfinancial.The investment, which can be measured before and after implementation of the project, depends on the time and requirements.Initial investigation is important to know the objectives, expected outcomes, measuring factors, stakeholders, and time duration for evaluation.Cost and business values are the common factors involved during assessment.In addition to the most common financial methods, ROI and ROA, the multidimensional post-implementation measuring approach is the most common practice in the case studies used for assessing nonfinancial ICT business values.Based on the findings of this research the list and steps of the evaluation processes can be developed in future which is lacking in this research.Using the expected framework in future, the companies can evaluate the performance of their ICT project based on the list of objectives defined in this research.

TABLE IV .
GENERAL PURPOSES OF ICT INVESTMENTS IN CASE STUDIES

TABLE V .
ICT PROJECT'S COMMON PRACTICES IN CASE STUDIES

TABLE VI .
ICT PROJECT'S INVESTMENT -INPUT RESOURCES AND OUTPUT/OBJECTIVES IN CASE STUDIES

TABLE X .
THIRD PARTY ASSISTANCE FOR INVESTMENT AND EVALUATION IN CASE STUDIES