An Expert Comparison of Accreditation Support Tools for the Undergraduate Computing Programs

Realizing continuous quality improvement within educational programs is a challenging task. However, there exist various assessment tools and models that help in this regard. This paper explores the features and capabilities of three major international accreditation support tools and compares their strengths and weaknesses. The investigated tools include EvalTools, CLOSO, and WEAVEonline. Two education quality experts performed a thorough comparison of the three tools across a range of criteria including coverage of the continuous quality improvement cycle, usability of the system, learning curve of faculty, data entry, data protection and privacy, among others. The paper highlights the advantages offered by each tool and identifies the gaps in respect to the continuous quality improvement cycle. Keywords—Component; accreditation support tools; continuous quality improvement; undergraduate programs; assessment; student outcomes; software


I. INTRODUCTION
Maintaining accreditation standards within educational programs is a major endeavor for international universities today to ensure competitiveness and credibility of their quality [6].Moreover, the assessment and accreditation of a certain educational program are two intertwined processes and are usually concurrent.Acquiring accreditation often means that the educational program must meet and satisfy certain requirements and criteria established by a dedicated world accreditation body [11].For instance, the accreditation of engineering and computing programs by ABET necessitates the fulfilment of nine different criteria concerning student, student outcomes, continuous improvement, faculty, and curriculum, to name a few [1].Realizing and monitoring accreditation standards manually is a cumbersome and complex procedure where evidences must be collected and documented throughout the assessment life cycle [19].Therefore, more universities and educational institutions are working towards developing automated software solutions to govern and facilitate the assessment and accreditation processes [20].
However, three major challenges hinder the exploitation of such tools.Firstly, the literature reports on a limited number of accreditation support tools which necessitates further research.Secondly, the modules, components and functions of these tools are still not well understood.Thirdly, selecting or developing an appropriate accreditation tool is not a straightforward task and requires the definition of appropriate requirements.The herein paper investigates the details of three world accreditation tools with the aim of empowering education quality experts to recognize and select the most appropriate tool that suits their needs and context of use.Overall, this research contributes in the following ways.
 Provide a detailed review of the capabilities and functionalities of three modern accreditation tools for academic programs, namely EvalTools [2], CLOSO [3], WEAVEonline [4].
 Identify the strengths and weaknesses of each accreditation tool.The findings will enable decision makers in educational institutions to select the most appropriate tool that fulfill their needs, requirements, and resources.
The remainder of this paper is organized as follows.Section two briefly reviews the models and tools supporting the continuous quality improvement (CQI) cycle and assessment of student outcomes.Section three presents the procedure of the comparison and the selected tools.Section four presents the features of the tools.Section five compares the three accreditation tools.

II. RELATED WORKS
Managing and promoting quality within education has become a top priority for educational establishments and decision makers [6].This process, however, is time consuming and resourceful.The benefits of total quality management are well documented in the literature, comprising the provision of better services, satisfaction of students, maintenance of a competitive edge, and demonstration of high accountability [6].There exist different quality models for achieving excellence in non-profit contexts such as the European Foundation for Quality Management (EFQM) and Malcolm Baldrige National Quality Award (MBNQA).Comparing the two models revealed that the EFQM model outperforms the MBNQA model in respect to satisfying the core principles of total quality management such as leadership, customer focus, process management and continuous improvement [9].
At the heart of quality management in education is the continuous improvement [21].A longitudinal study of an Irish university demonstrated the need for external reviews, internal expertise, as well as training and senior management commitment as key elements for the successful realization of continuous improvement in higher education institutions [7].A recent improvement framework describes the ingredients for www.ijacsa.thesai.orgtransforming knowledge into practice to achieve sustainable change and improvement in education [5].Particularly, the framework encompasses three questions addressing the aims, improvement teams and workforce, and ways for change to result in improvement.These questions are then tested using the plan-do-study-act cycle to verify whether a proposed change indeed results in an improvement.
Outcome based education (OBE) is the new trend in teaching and learning where more focus is placed on delivering the knowledge, skills, and attitudes that learners should gain at the time of graduation [23].However, achieving outcomebased education is laborious and time consuming.Therefore, [19] suggest the use of ICT to overcome the drawbacks of outcome-based education and facilitate continuous improvement.Indeed, various accreditation software applications are implemented and reported in the literature.ACAT is a web-based tool that facilitates the collection of data and generation of assessment reports as requested by ABET [12].Overall ACAT encompasses four modules namely the administration, data entry, report generation, and authentication of access module.Similarly, a web-based assessment tool was developed in [13] to automatically collect assessment data, evaluate the student attainment levels and monitor the execution of remedial improvement actions for ABETprograms.The portal includes three key components namely a course module, monitoring module, and assessment module [16].The portal streamlines the efforts of different stakeholders of the assessment cycle.Moreover, the tool enables the direct mapping of course outcomes to program outcomes which enables the identification of deficiencies within the course contents.
In [15], OBACIS is introduced as a framework to fulfill the requirements of Canadian Engineering Accreditation Board (CEAB) and subsequently enhanced to expedite the reporting phase of assessment documents and resources [18].In fact, the tool automates the creation of course information sheets by combining three modules, namely an office application for grading and OBE reports, a web application for faculty templates, and a Windows application for program and assessment reports.All modules communicate with a central database of multiple resources.The authors expect that the tool could save up to 70% of the required assessment efforts.To improve the attainment level of student outcomes by weak student groups, [20] suggest the inclusion of a learning analytics' module.
Other sophisticated accreditation support tools employ the multi-agent paradigm.For example, [17] utilized software agents to check the needs and status of the institution's program and accordingly provide advice and generate assessment reports.In [10], a case study is undertaken to showcase the benefits of exploiting the Internet of Things, through a dedicated web service, to enhance the educational learning process.Other frameworks that aimed at instilling quality within education introduced the use of six sigma to develop quality management systems [8].A Moodle tool based on the socio formative competencies model is suggested in [22] where instructors can design and include assessments according to the competency level of their students.Satisfaction scores of the tool were favorable.
In relation to our research, [14] utilized CLOSO, which is developed in Visual Basic .NET, to demonstrate the assessment of course outcomes and student outcomes in an automatic fashion in a bid to reduce instructors' time and efforts.Finally, EvalTools is detailed in [21] as a web-based tool for implementing specific performance indicators at the course level rather than the program level to fulfil the requirements of ABET.The tool applies a weighted algorithm to measure different learning levels (e.g.beginner, intermediate and master) across three learning domains (e.g.cognitive, affective, and psychomotor).

III. EVALUATION PROCEDURE AND SELECTED TOOLS
Two education quality experts who have more than 5 years of experience in the domain of local and international accreditation of various computing and engineering programs performed an-in-depth analysis of three accreditation tools.The tools that were selected for the expert evaluation included EvalTools, CLOSO, and WEAVEonline.Each expert went independently through the tools and investigated their various functionalities in detail documenting the key functionalities, reports generated by the tools, and strengths and weaknesses.The results were then discussed and integrated as summarized in the results' section.

A. EvalTools
EvalTools is a complete online learning management and comprehensive assessment tool developed by Makteam [2], as depicted in Fig 1 .It integrates three features mainly: learning management (e.g.attendance, lessons, assignments, gradebook … etc.), learning outcomes assessment (e.g.direct and indirect assessment of learning outcomes), and academic administration (e.g.dean office, department teaching, … etc.).Until today, EvalTools has been used by few universities to attain the ABET accreditation of undergraduate engineering and science programs.Until today, CLOSO has been used by several universities to attain the international accreditation of science and engineering programs, specifically the ABET accreditation.

C. WEAVEonline
WEAVEonline is a web-based tool for managing the assessment and evaluation of educational and non-educational outputs (e.g.outcomes or objectives) within various academic or non-academic units to assist universities and academic programs in the management of accreditation, evaluation, planning and quality improvement processes [4], as shown in Fig 3.This tool is designed to store evaluation and assessment data that ultimately improve the student learning and quality of services offered by universities and academic programs in various forms.WEAVEonline mainly focuses on the continuous improvement of all academic programs, administrative and student affairs units, and other services provided.In addition, this tool supports the implementation of improvement plans.

A. EvalTools
A careful inspection of EvalTools demonstrated that there are four types of users who can access the different modules of EvalTools as detailed below.
 Administrator: the admin has full access and total privileges to all tool modules related to all academic programs of a faculty;  Program Coordinator: the coordinator has access to the system modules that are related to the assessment of an academic program within a faculty;  Faculty Members: instructors have limited access to the functions and modules of the system that are related to their courses and reports within a specific academic program; and  Students: students have access to only the learning management module of their current and past courses.
Table I summarises the main functionalities and features made accessible to the different types of users of EvalTools.www.ijacsa.thesai.orgAdd students grades to the system directly through a table or an imported excel file Upload and download graded assignments for students Manage the grade book and specify the way grades are calculated for students with their percentages Assign grades to students for each assignment, as part of a table or as an imported excel file View the gradebook containing all students' grades as a table or excel file Data entry is integral part of the continuous quality improvement where the stakeholders are required to enter different inputs into the accreditation tool to monitor the assessment cycle.The major data entry tasks performed by the administrator in EvalTools include creating new users of the tool, entering data related to the program assessment such as the student outcomes and performance indicators, creating a curriculum map and setting up the relevant indirect assessment surveys.The faculty member however is responsible about entering courses' information and uploading related materials (e.g.lecture slides and assignments), inserting students' grades, uploading graded assessments, and writing the reflections and actions about the performance of their students.Students, on the other hand, upload their assignments' answers to EvalTools and rate the quality of their courses by completing different types of surveys (e.g.course, senior, field experience etc.).
EvalTools automatically generates, based on the inputted data, a wealth of reports and documents that are considered useful to both the student learning and program assessment.Table II summarizes nine main reports that are created by EvalTools along with their inner sections.

B. CLOSO
The users of CLOSO can be classified into administrators or instructors.The administrators have full access to all tool's modules and functions that are related to the program assessment.However, the instructors have limited access to the functions and modules that are related to the assessment of their courses.Table III summarises the main functionalities and features made accessible to the different types of CLOSO users.www.ijacsa.thesai.org The major data entry activities and tasks that need to be completed by CLOSO administrator include setting options related to the list of student outcomes, terminologies, and target satisfaction criterion, inputting data about the departments and courses offered in the programs and setting up the coursesstudent outcomes mapping (i.e. the curriculum map).However, the instructors are required to input course information (e.g.syllabus, CLOS, and topics) and students' names, to write the assessment questions to measure the attainment of student outcomes, and to upload the assessment evidences (e.g.students' answers) as PDFs or images.Obviously, data entry in CLOSO is less complex than EvalTools.Similarly, CLOSO generates nine reports that serve the accreditation process and assessment of student learning outcomes as listed in Table IV.

C. WEAVEonline
The roles that can be created using WEAVEonline include the administrator, entity administrator, entity-write and review, entity-review only, approver, group owner, map owner, external reviewer, and data uploader.Add and update the results of the assessment results for an assessment cycle www.ijacsa.thesai.orgApprover Add, update or delete an action plan along with details such as start and end dates, status of plan, priorities, and budget.

Add or delete an analysis of assessment data for a cycle
View the different courses / experiences, outcomes / objectives, courses and learning outcomes Establish the mapping between courses and different learning outcomes and learning experiences Nine types of reports are generated by WEAVEonline along with their sections as summarized in Table VI.Interestingly, these reports go beyond the assessment of learning outcomes and include the assessment of key performance indicators (KPIs).The reports also clearly link the KPIS to the mission and objectives of the institution.Unlike EvalTools and CLOSO, WEAVEonline places more emphasis on improving the strategy of the educational institution and its units.

Mapping
This report displays the entities that are using curriculum mapping in the assessment.

D. The Comparison Matrix
Table VII contrasts the three accreditation tools EvalTools, CLOSO, and WEAVEonline, and allocates a rating to each tool against 36 comparison criteria, with the following rating scheme:  Yes: means that the criterion has been fully satisfied;  No: means that the criterion has not be met at all; and  Partial: means that the tool partially complies to the specified criterion V. DISCUSSION OF THE STRENGTHS AND WEAKNESSES Based on the extensive review and analysis carried out with the three accreditation support tools, various strengths and drawbacks were extracted and are highlighted below.
EvalTools complies with almost a complete continuous quality improvement cycle and addresses many aspects of the CQI phases, ranging from planning, implementation, reporting, and closing the loop (i.e.implementation of improvement actions).EvalTools can be flexibly applied to satisfy the requirements of differing assessment philosophies and accreditation bodies including the use of simple generic performance indicators (i.e.ABET assessment philosophy) and complex specific performance indicators (i.e. at the course level) to assess the programs' quality and achievement levels of the students.Moreover, EvalTools uses an adjustable model for scoring and calculating the achievement levels of the performance indicators and course outcomes.The model can be changed as per the needs and goals of each educational program.EvalTools implements a strong reporting functionality and provides a myriad of reports including direct assessment (e.g.performance indicators measurement using examinations) results and indirect assessment (e.g.course level surveys) results utilizing graphs and charts whenever possible.The reports are presented at different levels, a fine-grained level (e.g.student or course) and holistic level (e.g.program) which enables the monitoring of student performance accurately.EvalTools enforces the mapping of program student outcomes to the courses, courses' outcomes and performance indicators creating a solid and clear curriculum map.EvalTools empowers the design and administration of multiple surveys to collect feedback from various stakeholders of the program (e.g.students, faculty, employers … etc.) to support the indirect assessment of the program continuously.
On the negative side, EvalTools is a rather complex and intimidating system for the inexperienced faculty members; for instance, data entry requires multiple and redundant clicks.
Moreover, EvalTools introduces a steep learning curve to operate its modules and to get accustomed to its full features.It does not provide any manual or user guide for beginners.This usually leads to mistakes in the entry of data (e.g.mapping of COs and PIs) and comprehension of generated reports.EvalTools is designed mainly to satisfy criterion four of ABET (i.e.Continuous Improvement).The remaining criteria of ABET (e.g.students, faculty … etc.) are not addressed by the tool which means that more work is needed for the accreditation.Whilst EvalTools covers to a large extent the assessment cycle, it does adequately cover the planning phase of the continuous quality improvement.Performance wise, EvalTools often suffers from constant delays and server crashes especially during peak times and concurrent access.EvalTools does not implement strong privacy and security measures as it is easy to access the profiles of other EvalTools users.
CLOSO empowers customization; for example, the terminology can be modified by the administrator to suit any academic program, department or university.CLOSO is built on top of the ABET assessment model.Therefore, it organizes the documentations required by ABET in neat way and makes them ready to print through a click of a button.In particular, it prepares criterion four and five of the self-study report of ABET in a professional manner.CLOSO covers the planning phase of CQI reasonably well as the assessment plan can be created through a set of dropdown menus with pre-defined options reducing the need to write an assessment plan from scratch.CLOSO uses graphs to aid understanding of the performance of the students; for example, the scores input form displays various interesting charts.CLOSO also provides a set of readymade and easy to answer faculty and student surveys to assess the quality of educational programs.Moreover, it empowers the preparation and implementation of an improvement plan along with the necessary evidences.Overall, CLOSO supports a sustainable assessment philosophy that can www.ijacsa.thesai.orgbe managed easily by the program coordinators and course instructors.
As drawbacks, only two types of users (i.e.administrator and instructors) can use CLOSO.However, educational programs usually include coordinators and students.The assessment plan offered by CLOSO is quite rigid which means that policy makers cannot introduce major changes to the existing assessment plan.Accreditation work is typically organized through various committees; however, CLOSO does not support the creation and management of committees and work groups.CLOSO does not support student advising based on the identified weaknesses from the assessment process.The surveys used for indirect assessment cannot be customized to fit the needs of the program.Although CLOSO encourages instructors to reflect on their teaching by specifying the limitations and potential improvements, these reflections are performed at the course level instead of the program level which results in an incomplete improvement plan.
Finally, WEAVEonline can be used for the outcome assessment at all levels (e.g.university, colleges, academic programs, administrative units, etc.) through the establishment of units.WEAVEonline supports the creation of a plan for the continuous development of all units of the university based on the achievement levels of outcomes and objectives, being either educational or non-educational.In fact, WEAVEonline is used by numerous American universities to manage the evaluation and assessment processes to ensure the continuous development.One interesting feature of WEAVEonline is that it allows monitoring the data entry completion rate of various stakeholders through the Data Entry Status (DES) Reports.Moreover, the tool generates rich reports such as the full evaluation report, audit reports, data entry status reports, analysis questions reports, planning and budget reports, personal reports, and so forth.There is a review and approval feature for all parts of the assessment and planning that need to be approved by the decision makers, enabling a good level of control and accountability.The possibility of identifying the responsible person / group for the implementation of any improvement plan is done electronically and the implementation of such action plans can be monitored through the tool.WEAVEonline provides the external reviewers access to all reports and assessment evidences.
As limitations, WEAVEonline does not allow the implementation of assessment methods (e.g.exams, quizzes, etc.) directly nor their mapping to the student learning outcomes.Instead, assessments must be carried out independently and only then the results can be inserted into the tool.Moreover, the administration of surveys to collect indirect assessment data is not possible via the tool.Although the surveys can be selected as an assessment method in WEAVEonline, its administration and analysis must be performed externally.Students cannot view the results of their attainment level of the student outcomes using the tool.Although quantitative data can be entered in WEAVEonline, the improvement plans rely heavily on the descriptive analysis.

VI. CONCLUSIONS
This paper provides detailed insights about the functions and features of three world accreditation support tools for educational programs.Developing or selecting an appropriate assessment tool that satisfies the accreditation requirements is a challenging job and this research aims to empower decision makers to make an informed decision about the tool to adopt.
The expert analysis, carried out by two education quality reviewers, included a comprehensive review of the main functions, data entry activities, and generated reports of the selected accreditation tools.The results showed that no single tool covers all phases of the continuous quality improvement.EvalTools is feature rich with a strong reporting mechanism; however, it is relatively time consuming to operate and less user friendly.CLOSO covers criterion four and five of ABET and generates the respective ready to use reports.However, it does not support group work and collaboration and is limited by a rigid assessment plan.WEAVEonline supports the assessment of educational and non-educational outputs and outcomes and uses a tracking component to monitor the progress of assessment tasks and improvement action plans.However, WEAVEonline lacks a survey management module to collect indirect assessment data and requires the preparation of assessments externally of the tool.
Future research plans will focus on inferring and proposing an architecture for an accreditation support system that combines the strengths of all three tools discussed in this paper.The architecture will be translated into a functional prototype that will be subsequently subjected to a user testing.

Fig. 1 .
Fig. 1.EvalTools Admin Inteface.B. CLOSO CLOSO is an accreditation tool developed by a research team in the United States in 2009 [3], as shown in Fig 2. This
Assign and add members to program committees and remove them as needed Access and view the courses syllabi of the program View rating of teaching performance of the program View rating of individual faculty of the program Download the program teaching rating report and faculty teaching rating report as word files Assign students to advisors through the advising system View and monitor the performance of all students of the program through the advising system View course assessment reports (indirect assessment achieved through surveys) for all courses of a program in different formats (tables and excel) Administrator, Coordinator, Faculty View and track progress of the faculty portfolios including their past and current courses Write own courses' description, and update the various sections of the courses View, add and edit existing course assignments and homeworks, along with their learning domains / levels Upload oe download teaching materials (e.g.lecture notes, research papers etc) in different formats Import existing course descriptions from a set of templates Export course descriptions into a word file Create or import rubrics for student assessment View and download students' submitted assignments Publish and share notes by the faculty with their students View or export the results of the indirect assessments (i.e.surveys) as tables, word or excel files View the various elements of the program assessment including program educational objectives, student outcomes, and performance indicators, and their mappings View or export the elements of the program assessment including program educational objectives, student outcomes, and performance indicators, and their mappings as a word and excel file View or export program evaluation and executive summaries reports that are related to student outcomes evaluation and performance indicators evaluation View or download the objective evidence folders containing all evidences (direct and indirect assessments) that justify the performance levels for the student outcomes and performance indicators as word Add, update or delete reviews and improvement actions, about the program and courses, as a result of reflection about the achievement levels of student outcomes and performance indicators View action item matrix about the improvements and responsible entities that need to be taken care of in relation to the courses and program www.ijacsa.thesai.orgGenerate and download curriculum outcome matrix that shows the links between course outcomes and specific student outcomes for a particular program View the committees of which he is a member and its members, documents and tasks Communicate with the members of designated committees directly using EvalTools Upload the relevant documents (e.g.minutes of meetings) to the designated committees Monitor and advise allocated students on their academic performance and provide regular feedback View and download course assessment reports (indirect assessment achieved through surveys) for their courses in different formats (tables and excel) Student Browse past and current courses to view their information (descriptions, assessment methods … etc.) Download the lecture materials /assignments Monitor academic achievement against a set of course outcomes and performance indicators per assessment Submit answers for a particular assignment View grades for various types of assessments Access students advising information through the student advising functionality Assess the quality of current courses by responding to various types of surveys

TABLE .
Manage or delete existing user profiles (e.g.update details, user types, access control … etc.) Create, manage and send automatic emails to the various users of the system Set up and manage the program assessment elements including adding and mapping PEOS, SOs, and PIs Set up, update and administer the surveys (e.g.course, exit, senior, employer, alumni, field experience … etc) used for the indirect assessment of programs Administrator, Coordinator Create, edit, and delete committees at the faculty and program level as needed

TABLE .
This report provides a summary of the student evaluation of the courses using a dedicated survey that enquires about the quality of various aspects of the courses, field training and learning experience in general.The questions focus on the quality of the course contents, outcomes, teaching and assessment methods, and faculty.These results represent the backbone of the indirect assessment of the quality of the courses and program.The report includes the following items:This report provides a set of graphs that indicate the achievement levels of student outcomes of a program as well as the achievement levels of their performance indicators.The report includes the following items:This report lists all reflection and action items of the program.These actions are usually planned for the upcoming term and are related to various courses of the program.The report includes the following items: Various assessments of the course (e.g.homeworks, quizzes, exams, etc.)Faculty Course Assessment Report (FCAR)This is considered the most important and comprehensive report of EvalTools at the course level as it gives an overview of the performance of students during the term, with strengths and weaknesses clearly highlighted in relation to the student outcomes and performance indicators.It also includes the reflections and improvement actions planned for the next term.Graphs are used throughout the report to aid understanding.The report includes these sections: www.ijacsa.thesai.orgThis is a detailed report covering all student outcomes of a program.The report summarizes the review of the program including the reflections and actions about students' performance and achievement levels.These are linked to the scores of the performance indicators (out of 5).The report includes the following items:This report details the performance indicators under each student outcome and summarizes the achievement levels of each performance indicator along with a classification of the level of learning.Students are distributed within a table across the achievement thresholds (E, A, M, U).The report includes the following items:

TABLE .
This report help organizes the objective evidences of the students' assessments into easy to read folders.All direct assessments that are used to measure the attainment of the student outcomes are collected in this report.
Table V summarizes the main functions of WEAVEonline.

TABLE .
Manage (add, update and delete) entity trees and their types (academic, administrative … etc.) Add and update information about new, present and past assessment cycles Add and update the information of the beneficiary e.g.university such as University name … etc.Create and add strategic plans for the institution, faculties, and programs Add and edit key information about the curriculum, and priorities of the institution and programs Add and edit the local news of the university Add, update, and delete the mission statement, goals, outcomes and objectives of the beneficiary Add and update the analytical questions which could be used in the assessment summary report Add, update and delete assessment methods and threshold for measuring the attainment of objectives and outcomes

TABLE .
VII. THE COMPARISON OF THE ACCREDITATION SUPPORT TOOLS