Supplier Qualification Model (SQM): A Quantitative Model for Supplier Agreements Evaluation

Recently software outsourcing has increasingly widespread due to the valuable economical and technical benefits it introduced to the software development industry. Where the software development organizations adopt a third party to acquire a software project component (product, service). In the acquisition, process companies rely on the CMMI supplier agreement management (SAM) process area to select the potential supplier. Potential suppliers (vendors) are carefully selected through a dedicated process to ensure the delivery of high-quality and reliable services. Most of the published work in the context of how to evaluate and select the right supplier is based on a normal process with plain steps, nevertheless, no literature was reported to evaluate suppliers in a measurable way and select the potentials depending on a quantitative model. The purpose of this paper is to propose a practical quantitative model called the Supplier Qualification Model that enables the organizations to easily evaluate and select the potential suppliers through a measurable approach depends on monitoring and executing the SLAs of the SAM. The proposed model has been verified by implementing it through building an extension for one of the worldwide leading Agile management platforms according to Gartner (Microsoft Team Foundation Server). Multiple versions of the extension were implemented to target the major versions of Microsoft Team Foundation Server and validated by using them in 426 worldwide companies. This proves the suitability of the model to be used. Keywords—Agile practices; vendor selection; CMMI; outsourcing; software acquisition; supplier agreement management; supplier selection; supplier evaluation


I. INTRODUCTION
In the last decade, the software development markets became more competitive and demanding, where it came to be crucial for organizations to invest more effort to improve their software processes to meet specific requirements. As a result, they had to follow a quality model to improve their software development process, increase their capability and maturity level, and become a benchmark for their competitors [1].
The Capability Maturity Model Integration (CMMI) Product Suite that was resealed in January 2002 paved the way for the organizations to improve their processes for acquisition, development, and sustainability [2]. Therefore, a large number of software development organizations (according to the CMMI Institute, 5000 businesses in 49 countries all over the world, and 1900 appraisal were conducted [3]) embark on it to improve their performance, develop higher-quality software, meet stakeholders satisfaction and achieve external validation [4].
As the software market mandate not only a product of high quality but also time and cost control [1], the organizations found that working Agile based on the CMMI model would allow continuous improvement and help them to reach the required maturity level [5].
Agile is a traditional software development approach that was created in 2001 [6], and its methodologies have gained widespread acceptance among a large number of organizations as a quality-focused and highly collaborative mechanism to manage software development and improve the delivery process [7]. Agility is based on the idea that high-quality software can be developed by following a set of rules that allows continuous product improvement and testing depending on rapid feedback and testing [8].
Williams and Cockburn [9] defined the Agile team at the beginning to be as small as 50 people or fewer. However, Duka [10] showed that the larger teams had adopted Agile methodologies over the waterfall methods due to the great benefits of Agile practices. As well, in 2010 studies in the Agile journal [10] showed that 88% of companies with some of them have more than 10.000 employees are adopting Agile.
Different studies [11][12][13] were conducted to display the great benefits of working agile from the perspective of productivity and time management which leads at the end to customer satisfaction. They showed that agility can increase productivity to about 88%, improve cost efficiency by about 26%, and 41% time to market. Also, the 12 th annual state of the Agile report [14] revealed that working Agile, 71 % manage change priorities, 66% improve the project visibility, gives 65% better business and IT alignment, speed the project delivery 62%, and 61% increase the team productivity. Besides, the Standish Group Chaos Report 2018 [15] results exhibited that working agile gives higher success and lower failure rates when compared to projects adopting the waterfall method (see Fig. 1 and Fig. 2). 445 | P a g e www.ijacsa.thesai.org Another reason that many organizations adopt Agile practices is that it permits the so-called distributed development (software outsourcing) [16], where an organization selects a third-party service provider (supplier) to execute a part of a software development project. The strategy of software outsourcing brought enormous benefits over the in-house development, where it reduced the operating costs, saved time, and gave immediate access to talented and higherlevel IT professionals [17,18]. Watts Humphrey [19] implied that the quality of the software development process determines the quality of the produced software, and in the case of outsourced products the quality of software acquired from suppliers. This means that the selection process of the suppliers should be done according to specific and carefully calculated criteria.
Since the process to manage and control outsourced development projects is complicated and faces obstacles as inhouse projects. Agile organizations had to adopt a specific process to manage the relationship between them and the service provider to be able to identify and select the potential suppliers. For an Agile organization that needs to be compliant with the CMMI supplier agreement management (SAM) process area, it must demonstrate an explicit commitment to establish a process-based confirmation through people and should verify the execution of the process and validate the outcome of process execution through a measured result [2]. EM Soares et al. [20] mapped the relationship between the supplier agreement management process area and agile practices, where they developed a catalog of the best practices using the concepts of the agile methodologies to manage the software acquisition process in the Supplier Agreement Management (SAM) process area of CMMI-DEV. Neither EM Soares nor other studies in the literature to the best of our knowledge was reported to introduce a quantitative method that helps the organizations to differentiate between software development suppliers, to select the potential ones based on measured data, and to monitor the execution of service level agreements established between the stakeholders and service providers according to CMMI-DEV v.2 specific goals of SAM process area.  The main goal of this work is to propose a reference model that can be used to evaluate the supplier agreements quantitatively. And help the organizations to identify and select the potential suppliers. In the coming sections of this paper, a background about the supplier agreement management (SAM) process area and the selection process of potential suppliers is presented in a detailed manner. Then an introduction to the proposed model, its mathematical profile, and how it is in the evaluation process of suppliers is introduced. As well as, clarification of why the SQM model will be helpful for organizations to select the potential suppliers is demonstrated. Afterward, the verification and validation of the SQM model were discussed. Finally, a vision for further work in the future is proposed.

II. BACKGROUND AND RELATED WORK
Supplier agreement management (SAM) is a CMMI process area at maturity level 2, it falls in the project management category and aims to manage the acquisition process of products and services from suppliers [21]. SAM is applicable only when organizations deal with a third party (supplier) to acquire products or services for software development. The specific goals of this process area were formulated to determine the acquisition type, to select suppliers, to establish and maintain supplier agreements, to execute the service level agreements (SLAs), to monitor the processes of the selected supplier, to accept the acquired product, and to ensure the successful delivery of products.
The supplier selection process comes directly after the identification of the acquisition type required from the vendor/supplier. Suppliers are selected based on an evaluation of their ability to meet specific predefined requirements and founded criteria. In the evaluation criteria, it is crucial to define the critical factors for the project such as costs, the geographical location of the supplier, quality services, supplier's performance record, prior experience, etc. to be able to identify and select the potential suppliers, then to evaluate their proposals [20,22,23]. Afterward, the risks associated with each proposed supplier and his ability to perform work must be assessed through the evaluation of their prior experience and prior performance [22]. 446 | P a g e www.ijacsa.thesai.org Selecting the right supplier who can deliver a high-quality and reliable product at the desired time and within the defined budget is not an easy task, and unfortunately, not much literature was dedicated to define a measurable method with quantitative terms to select potential suppliers. Nevertheless, authors always report the evaluation and selection process of suppliers in a plain way as mentioned above, for example, G.O'regan [24] reported that in the identification process of suppliers, organizations may search or depend on recommendations from colleagues and previous work relationships to select the candidate suppliers. Then an evaluation team will be responsible to rate each one of them according to the founded criteria. Afterward, a shortlist of potential suppliers will present their proposals followed by a Q & A session, where the final decision will be made depending on this discussion.
Chaudhary et al. [25] stated that candidate suppliers can be chosen based on their capability or experience in delivering the desired product or services. After selecting the potential candidates, the final selection will be performed depending on their proposals, and the advantages and the disadvantages of each in the light of the factors of the predefined criteria.
EM Soares et al. [20] developed a catalog of the best practices to manage the software acquisition process by employing agile methodologies. They build up their work depending on the framework activities of Furtado [26], where they showed that these activities are compatible with the specific goals of the SAM process area in CMMI-DEV. They used this catalog to define finely the steps to evaluate and select suppliers based on the agile practices using the formal alternate evaluation method. Where they made a survey and collected information about suppliers to have a list, then they analyzed these data to limit this list to the most suitable candidates. Afterward, the potentials were selected depending on a checklist of the project requirements and the features of the supplier. Finally, the supplier who meets all the requirements or is closest to them will be selected.
Although these references give detailed and precise ways to evaluate and select suppliers, to the best of our knowledge, no study was reported in the literature to propose a method or model that facilitates the differentiation and evaluation of suppliers and the risk associated with each one of them in a measured way.
Since the identification of potential suppliers and the risk associated with each one of them is considered a crucial role in the success of a software acquisition process, the present study proposes a model that facilitates the evaluation and estimates the risk associated with each proposed supplier.
The developed model calculates the supplier's qualification through a quantitative score (SQScore) based on the established service level agreements (SLAs), enables organizations to use this quantitative score (SQScore) to distinguish between suppliers depending on their historical accumulative SQScores.

A. The Need for a Qualification Model for Selecting the Right Supplier
According to the CMMI SAM process area reference guide, many factors control the selection process of suppliers such as budget (money), quality, project business value, supplier reliability, compliance, the risk of failure, supplier experience, and previous work [21].
In the light of the above factors, when an organization must select a supplier from a list of professional suppliers, before the final selection, a set of questions should be answered to define the potential (right) supplier: • Which supplier could deliver better business value when delivering the project?
• Does the supplier have experience working with the organization?
• What is the rate of associated failure of each supplier?
• What is the percentage of failure of each supplier?
• What is the risk of assigning the project to each supplier?
The supplier qualification model (SQM) will help the organizations to answer these questions, not only to select the supplier but also to monitor the selected supplier during the project execution for a better future qualification process.

B. Idea behind the SQM Model
According to SAM specific goal (SAM.SG.1), Establish Supplier Agreements [27], organizations established some service level agreements (SLAs). Each particular SLA mentioned in the supplier agreement has terms and conditions that suppliers should fulfill to satisfy this SLA. For example, an organization can establish a service level agreement to determine the level of quality for a specific number of works (Wn), the maximum amount of budget for a set of specified features, and/or the level of skilled resources that can handle certain types of work.
The proposed SQM is considering these factors depending on a set of SLAs execution measures. When the supplier fails to satisfy the terms and conditions of an agreed SLA, this is termed as a violation, hence the total number of violations is given as (Vn), while the percentile of deviation from the original goal of the SLA refers to the violation percentage (Vp). Since the degree of importance of an established SLA varies depending on the nature of the SLA and the outstanding project, in the present model, we were concerned to measure two SLA weights, (i) the compliance weight (ComplianceWt) which identifies how much compliant is a supplier with all the founded SLAs, and (ii) the risk weight (RiskWt) to estimate the associated risk of the supplier.
Organizations can control the weights of the SLAs, where in some cases they can decide to give a low score for any violation of a certain SLA regardless of the percentage of the violation, while for other SLAs the percentage of failure is 447 | P a g e www.ijacsa.thesai.org more important. Since the two weights are mutually exclusive, the summation of both weights per SLA should be 100 points of weight.

C. Mathematical Profile of the SQM Model
The values of the ComplianceWt and the RiskWt are defined according to specific rules related to the project. This could differ from one SLA to another. As an example, for some SLAs, if the cost of failure to comply with a given SLA is very high, the compliance weight (CompilanceWt) should have a high value. Nevertheless, for other SLAs, the cost of failure will not be as high as long as the percentage of failure is low, Hence the SLA should have a high-risk weight (RiskWt) and low-Compliance weight (ComplianceWt) as will be explained in the following example.
If we assume that an organization is assigning a project to Supplier 1 and that 8 SLAs were established with each SLA has a number of work items (Wn). Each SLA has designated specific RiskWt points and ComplianceWt points (please note that the summation of each pair is 100). Each SLA has been violated a number of times (Vn), each time the violation is incurred with a specific percentage of violation (Vp). These values are depicted in the following set of matrices.  (1) moreover, the index (i) is, The SLA Risk Weight Index (SlaRiskWI) and the SLA Compliance Weight Index (SLACWI) that measure the importance of each SLA relative to the whole project can be calculated using equations (3) and (4). For both weight indices, the total value of each is 100.
Using the above equations (3 & 4), for (I = 3) the SlaRiskWI 3 , which is the SLA risk percentage for the fourth SLA out of all risk weights for the whole project) will be 22.989%, and the SlaCWI 3 , which is the SLA compliance percentage of the fourth SLA out of all compliance weights for the whole project is 18.805%. Then using the violation percentage (Vp), the Violation Severity (VS) can be calculated as follows: Dividing the violation severity (VS) by the number of work gives the average violation percentage (Average Compliance) for all work assigned to the supplier (see equation (6)).

1) Evaluating Risk vs. Compliance: While using Average
Compliance is helpful in many cases, using this measure only for qualifying suppliers, could be misleading. That is why we need in some cases to have an additional measure that considers risk associated with each supplier. For instance, If we assumed a case study of two suppliers, supplier A and supplier B, have the same violation severity, and supplier B has fewer number of violations than supplier A., In this case, the number of violations of supplier B (who has fewer violations) will indicate how much risk is associated with this supplier. For more clarification let us assume that both suppliers are assigning five projects and their violation percentages were as follows:  (5) and (6), the violation severity of both suppliers is 150, and their average compliance is 30 respectively. This means that both suppliers have the same failure percentage for the whole assigned work.
For supplier A, since the violation percentage of each project is small, the risk is therefore low. On the other hand, for supplier B, nevertheless, he assigned three projects successfully, the associated risk would be high because of the high violation percentage of the last two elements of violated projects array (75,75).
Based on the numbers of the original example of (5) & (6), the average violation percentage of violated work (which is the AverageRisk) will be calculated by dividing the violation severity (VS) by the number of violations (see equation 7).
Equations 6 and 7 calculate the average violation percentage for the whole assigned and violated work. Nevertheless, the success factor of not making compliance failures, and not incurring risk factors should be rewarded for each supplier. Therefore, the success value must be considered by calculating the average compliance success, and the average risk success as follows: 2) Calculating SQScore for each SLA: The Compliance Score for each SLA (SlaComSc) is calculated by subtracting the product of the AverageCompliance i and SlaCWI i of the violations, from the product of AverageComplianceSuccess i and SlaCWI i (see equation (10)). Based on this equation, when the supplier achieves successes more than failures, the SlaComSc i will be a matrix of positive numbers otherwise, it will be negative numbers. Similarly, the Risk Score for each SLA (SlaComSc) is calculated using equation 11.
3) Calculating SQScore: Therefore, the Supplier Compliance Score for all SLAs and the Supplier Risk Score for all SLAs, per each supplier, will be calculated from the sum of the SlaComSc and the sum of the SlaRiskSc respectively as seen in equations 12 and 13.
Finally, the SQScore of the supplier will be the average of both SlaRiskSc and SlaComSc.
Equation 14 shows that the supplier scored a success rate of 76.274. Now we can say that using this model organizations will be able to differentiate easily between suppliers based on their historical projects.

4) Evaluating suppliers:
If we assume that an organization has the following historical SQScore for six different suppliers: The organization needs to categorize these suppliers into three ranges a high, medium, and low depending on their SQScores. Hence we have to calculate the standard deviation. To do so, firstly the mean is calculated by dividing the sum of suppliers score by the number of suppliers: If n = index of the supplier:  26 12 -14 -31 - 9 16 Afterward, we get the variance from the sum of the square of each difference, then dividing by the number of suppliers.
Now the Standard Deviation (σ) is the square root of the variance.
Adding equations 15, 16, and 17 we get: Using the standard deviation and the mean, suppliers 2, 3, 5, and 6 can be categorized as average suppliers (the period of values between the ∓ ), while supplier 1 is above average, and supplier 4 is below average as illustrated in Fig. 3.

A. Model Verification
To verify the applicability of the proposed SQ Model, an extension tool has been built to automate applying the proposed SQM for software teams. In this study, the extension is based on the Microsoft Team Foundation Server (TFS), which is one of the worldwide leading software engineering platforms according to Gartner [28].
The verification process was executed in two steps, (i) check the possibility of extending an agile tool (in this study is the TFS) to calculate the total SQscore of a supplier and of each established SLA using the proposed model, (ii) verify the possibility of extending the TFS client (Visual Studio) that allows organizations to create SLAs based on the SQ model, and monitor and track the SLA execution and the supplier SQ score.
The TFS Scrum process template provides organizations with the artifacts, processes, and workflows which are necessary to adopt the scrum method. The proposed TFS SLA Server Extension adds the following artifacts to the TFS Scrum process (i) SLA Configuration, (ii) SLA Violation Work-Items. The SLA Configuration will store the Service Level Agreements information such as SLA Risk Weight, SLA compliance weight, SLA threshold, and SLA percentage (see Fig. 4). Nevertheless, the SLA violation will store the violation information such as the violation percentage. Organizations can establish and maintain Service Level Agreements by creating as many SLA configuration items as needed to define flexible rules such as work deadline information and relation with other work. The SLA configurations are implemented dynamically to allow customizations that fit many different business scenarios. For example: • Escalating work that in-progress for more than n number of hours/days.
• Escalating work with deadline configured on fields.
• Escalating all work related to a specific feature, epic, or product backlog items.
• Escalating work that takes more than n% of parents.
• Escalating assignment of specific work to a specific resource with a specific persona.
Once the organization created and activated SLAs for a particular project, the SLA is executed, and the SQscore is calculated by the TFS extension. Hence, the organization can monitor and track the execution and satisfaction of the SLAs.
The TFS SLA Client extension introduces a new capability to Visual Studio which allows organizations to monitor and track SLAs and measures the SQScore of a specific Supplier. 450 | P a g e www.ijacsa.thesai.org If Project X (see Fig. 5) with 8 SLAs has been established and executed by the supplier; each SLA has been violated several times, the extension calculates the Supplier Qualification Model Score by calculating the SQScore for each SLA using the model described above.
For example, SLA no. 320 has been violated 6 times. Therefore the Risk score is 4.44, the Compliance score is 13.69, and the SQScore is 9.1, and the supplier SQScore for the project is 76, as shown in (Fig. 5).

B. Model Validation
The TFS SLA Server and TFS SLA Client extensions have been published to Microsoft Visual Studio Gallery [29] and have scored 426 (for both versions) usages for the TFS SLA Server and 614 (for both versions) usages for the TFS SLA Client (check Ref. [29]) as illustrated in Fig. 6.
The SLA has been used by many organizations in the public and government sectors in multiple countries, which proofs the validity of the proposed SQM Model.

V. FUTURE WORK
The proposed model is an efficient suit for an organization that wants to qualify their vendors and categorize them into categories. It works effectively for the organizations that have vendors with previous data, nevertheless, in the case of new vendors, organizations will not have previous data, every organization will have its own SLAs. To help overcome this problem, future research can define and unify a set of common SLAs for vendors who want to get appraised and qualified by the proposed SQM. A global system that uses predefined SLAs should be created to give appraisal and SQScore for each vendor, therefore organizations could use this global SQScore to validate vendors.

VI. CONCLUSION
Evaluating and selecting service suppliers is a hard task especially that up to now, no method or model was reported to evaluate suppliers in a quantified manner according to the CMMI's SAM process area and its corresponding specific goals. The present work has proposed a quantitative model to help teams and organizations to evaluate and select potential suppliers based on the previous history of each one of them. The proposed model has introduced some measures to evaluate the success and failure of each supplier of being compliant with the agreed SLAs. Besides, it presented some other measures to evaluate the risk of dealing with each supplier. Taking these two factors in consideration, a set of mathematical equations are used to come up with the SQScore which could help in categorizing the suppliers into three categories (low, medium, and high performing suppliers) based on the average value of the SQScores of all suppliers and the corresponding Standard Deviation. This quantitative model has been verified by implementing it in a practical extension tool. Furthermore, it has been validated by using this tool in about 600 companies which proves it is suitable.