A Rule-based Approach toward Automating the Assessments of Academic Curriculum Mapping

Curriculum mapping is the blueprint of a successful academic program. It is progressively utilised in higher education as a monitoring tool in the current age of standard-based regulations and empowers program leaders and course instructors to align their curricula for the offered courses and the corresponding learning outcomes. It is often depicted by a two-dimensional matrix expressing the relationship between the students learning outcomes (i.e., SOs) and the courses. However, its mapping remains a challenging exercise, even for experienced program leaders. The complexity stems from the fact that mistakes are prone to happen during the mapping, and program leaders need to be aware of the rules and the acceptable practices of curriculum-effective mapping. Besides, it is not straightforward to spot contradictions in the SO-course mappings. Consequently, this paper aims to tackle these challenges by investigating effectivemapping rules from existing curriculum mappings, which allows one to inspect the SO-course mappings, discover inefficiencies, and provide suggestions for improving the curriculum mapping. We identify the main mapping criteria and propose a rule-based algorithm for curriculum matrix assessments. This algorithm is implemented in an online application and evaluated using a userbased experiment, relying on curriculum mapping experts. The findings have shedded light on the promise of our approach. Keywords—Rule based algorithm; curriculum mapping; student learning outcomes; program outcomes; assessment; curriculum matrix


I. INTRODUCTION
Curriculum mapping is the blueprint of a successful academic program. It is progressively utilised in higher education as a monitoring tool in the current age of standardbased regulations and empowers program leaders and course instructors to align their curricula for the offered courses and the corresponding learning outcomes [1]. Curriculum mapping approaches are implemented in academic institutions in a bid to improve the quality of education [2], culture of involvement, and cooperation. Such approaches can assist students in attaining the learning outcomes of educational programs more effectively and highlight the deficiencies within these programs [3], [4]. Curriculum mapping refers to the process of regulating program learning outcomes with the offered courses to recognize or highlight educational discrepancies, inefficiencies, as well as misalignments with a perspective program outcome [5]. The results include improvements to the overall consistency of course-program mappings and acceptable coverage of the program student outcomes (i.e. SOs) [6]. Moreover, curriculum mapping explores how well and to what extent a course instructor might have represented the contents to cover the academic requirements described by the student learning outcomes [7]. Curriculum mapping is often documented for every semester, throughout all subjects, and by each lecturer directly once the class's material is designed.
Despite its evident benefits, curriculum mapping remains a challenging exercise, even for experienced program leaders. The complexity stems from the fact that mistakes can be easily made during the mapping. Moreover, program leaders need to be aware of the rules and the acceptable practices of curriculum-effective mapping. Furthermore, manual mapping involves course instructors, which makes it resource intensive. Besides, it is not straightforward to spot contradictions in the SO-course mappings [8]. Consequently, this paper aims to tackle these challenges by investigating effective-mapping rules from existing curriculum mappings, which allows one to inspect the SO-course mappings, discover inefficiencies, and provide suggestions for improving the curriculum mapping. This paper focuses on developing an effective relationship between the courses and program learning outcomes, represented as a two-dimensional matrix, in an automated model. Quality assurance and accreditation agencies mandate that curriculum courses must be consistent with the program's educational goals and clearly state that curriculum mapping should concentrate on the fulfillment of each program/student outcome. More in detail, this paper is motivated by the following two research questions: (Question One) what are the best practices to achieve effective curriculum mapping in educational programs? and (Question Two) how can we implement these practices as part of a rational mapping solution?. In this paper, aiming at assessing the curriculum matrix of academic programs besides giving a set of recommendations for improving course-alignments, we make two contributions based on our observations on several existing curriculum mappings.
• We identify the main rules (criteria) for assessing curriculum matrix based on studying several well-defined curriculum mappings. • We propose a rule-based algorithm for curriculum matrix assessments. This algorithm is implemented and encapsulated in a web-based application that is publicly available online. • We report on a used-based evaluation (i.e., based on curriculum mapping experts) for validating our proposal.
The remainder of the paper is divided into four sections. Section II represents the related works, shedding light on the well-used mapping techniques for curriculum evaluation. Section III introduces our rule-based algorithm for assessing the curriculum mapping, besides discussing the derivation of these rules from the existing curriculum matrix. Section IV describes the proposed online assessment tool. Section V presents and discusses the conducted user-study for evaluating our proposal, and finally, Section VI concludes the paper with directions for future research.

II. RELATED WORK
In this section, we review several types of relevant work, categorized into three aspects. First, we discuss a few strategic design approaches for course-alignments with program outcomes in two-dimensional matrix representation. Analyzing these approaches allowed us to derive the core mapping rules (or criteria). In the second category, we focus particularly on analytic mapping techniques for curriculum evaluation. Finally, since this paper attempts to assess the quality of a given curriculum matrix along with its set of program outcomes, we review the closely related approaches that focus on alignment's quality between courses and program outcomes.

A. Strategic Design for Creating a Curriculum Matrix
In [9], the authors investigate how curriculums have developed in Estonia since the late 90s where the government took the initiative to reform national school curriculums. A comparison is made between their curriculum and that of Great Britain and Latvia while focusing on the historical and theoretical aspects. Plaza et al. [10] proposed a curriculum mapping approach, including program evaluation. In their article, the authors represent the association between the graphical curriculum maps of students and teachers concerning the ranking of the relative emphasis of each domain, indicating the adjustment between the curricula intended/delivered and obtained [10].
Spencer, et al. [11] have presented curriculum mapping as an embedded tool with graduate capabilities. In this article, the authors discussed a method for collecting, analyzing, and presenting current teaching and graduate capacity evaluation information. Their conceptual approach promotes interactive curriculum development exercise, whereas the resulting graphs continue providing conceptual representations of present procedures and measurements of where curriculum redesign should be positioned. Lacerda and Sepel [12] research curriculum perceptions through educators collect information regarding their experience and come up with ideas that will positively change the curriculum. Their results identified that post-critical theories were more accepted than critical theories, whereas the results regarding traditional theories were not clear enough. Nevertheless, it was heavily linked with the organization of classroom practices.
Linden, et al. [13] describe the influence of curriculum theories in higher education while concentrating on their historical and conceptual roots. Results show that adapting modern higher education environments requires identifying the differences between normative and critical curriculum while keeping in mind the curricula' influence. Roth and Thom [14] clarify the differences in curriculum theory over the past decade or two; they claim that number of curriculum topics studied has increased along with methodological and theoretical expansions. Their research indicates that curriculum inquiry is becoming more widespread and that there is potential for a cross-cultural attitude. Furthermore, it explores the relationship between the perception of the structure of the field and curriculum inquiry.
Sweden implemented the new curriculum in 2011, which was investigated by [15]. This investigation indicates that the teachers' curriculum agency identifies three spaces in an educational classroom: interactive space, collective space, and individual space. The teachers' feedback displayed clear evidence that the change in the curriculum required a greater level of content assessment, whereas the teachers can more easily assess abilities as they may be required to get involved with the students directly. Ghaderia [16] study ideas regarding peace-based curriculum, in a time where they believe peace is crucial for global stabilization with the technological warfare advancements. They believe that through merging post-modern theories and liberal theories (i.e., by merging the similarities and differences between the two), the ultimate peace-based curriculum can be achieved.

B. Analytic Mapping Techniques for Curriculum Evaluation
Treadwell, et al. [17] explores the launch of curriculum mapping on the web-based interactive learning opportunities, objectives, and outcome platform (LOOOP) via interviewing 30 lecturers concerning their experience with this curriculum mapping. Overall, the results showed that although the participants did not immediately master the use of this system and did face some slow development issues, except that they believed it was ultimately beneficial to the learning program and the students within the course due to several reasons, including; communication, usability, and transparency.
According to George Mason University's Health Administration Program experience, Perlin [18] examined the curriculum mapping for a program evaluation. Their examination helped in establishing a framework with the merging of an enhanced analytic process and a mapping exercise. This framework was initiated by identifying the analytic and technical methods by investigating the mapping process used within the university and its overall course objectives. As a result, setbacks were identified and pointed out so that it can be adjusted in the future, and users are aware of how to overcome these hurdles. Therefore users of this curriculummapping program will play a role in upgrading the quality of the university graduate program.
Avella, et al. [19] claim that although the use of learning analytics in higher education is very promising, except that one of the flaws associated with the immediate introduction of learning analytics in this sector is that the users are not able to make use of this system effectively or to its maximum potential. Therefore, the researchers studied the techniques that learning analytics implement such as visual data analytic techniques, relationship mining, and so on. Consequently, the benefits and the challenges of learning analytics were identified and listed according to its use in higher education, therefore educators can make use of this implemented system successfully and effectively which in turn would improve the quality of teaching within higher education.
Pat Hutchings [20] has published a report on the alignment of educational outcomes and practices. This research explores and disseminates ways in which educational programs and organizations could use evaluation information effectively to inform and enhance academic education and interact indirectly with decision-makers. Yates and Millar [21] discover the physics curriculum, in particular within Australian universities and schools. Physics was specifically chosen as it is a science that must adapt to change and update as time goes by. Their focus was to investigate whether or not the curriculum can be logically derived depending on the discipline.
However, Ghaith Al-Eyd1 et al. [22] claim that the use of curriculum mapping in a new medical school help provide a better understanding of the system as compared to other medical schools as well as the educational environment. Their research identified which factors benefitted from the use of their curriculum mapping; it mainly involved improving the level of organization and providing more evident learning outcomes that demonstrate the links between the entire course and outcomes.

C. Quality of Course-Outcomes Alignments
In 2019, Buker and Niklason [23] presented an improved curriculum mapping model. They tried to include some essential standards to help develop an assessment process. The recommended approaches involve assessing the program's mission and evaluating the course outcomes based on the criteria. Uchiyama and Radin [24] published an article on curriculum mapping in higher education. They have represented curriculum mapping as a mechanism that generates a visualization of the curriculum based on real-time data to improve the quality of education.
Lam and Tsui [25], published an article on the examination of alignments in curriculum mapping. This research proposes that curriculum mapping can be a helpful tool for assessing how the students approved learning outcomes are aligned with the classes provided by the academic faculty. M Jacobsen, et al [26] presented the article on action research for graduate program improvements. They have presented continuous program improvement techniques derived from actions and suggestions, which emerged from a year-long faculty-led, institutionally approved evaluation curriculum of academic programs. Jacobsen, et al. [27] has identified several factors that contribute to improving the program at hand. These factors include interconnecting the courses efficiently and effectively, providing a greater level of ethical support to students within the graduate program, etc. This research has helped identify the strengths of this graduate program and methods to fix its flaws.
The original contributions of this paper that differ from all previous works lies in examining various curriculum-mappings for rules extraction in addition to our proposed rule-based algorithm for mapping assessments. We will discuss these contributions in the next section.

III. ASSESSING CURRICULUM MATRIX
In this section, we present our rule-based procedure for assessing the curriculum mapping based on students learning outcomes. Given an academic program consisting of a set of outcomes, the proposed procedure assesses its two dimensional matrix that links the top-level outcomes with the low-level course-outcomes. More significantly, our procedure can help to give insights into curriculum mappings design, pinpointing the main criteria used for quality measurements. In order to automate our procedure, we have studied many different curriculum mappings, attempting to extract common rules (i.e., rules potentially applied during the development of the mappings) as a bottom-up approach. To this end, we consider a three-phase methodology to achieve the objectives of this paper: • Identify the existing curriculum mapping criteria and requirements. • Propose (and give an implicit implementation) of an algorithm to measure the quality of the curriculum mapping. • Evaluate the effectiveness of the proposed procedure based on surveys conducted with curriculum experts (discussed in the next section).
For the first phase, the Faculty of Computer and Information Systems (FCIS) at the Islamic University of Madinah was selected as a research site with a four-year history of curriculum mappings. In addition, we assume standard mapping letters (I, R and E) when designing curriculum mappings (i.e., link courses to program learning outcomes). These mapping letters are broadly employed in various academic accreditation agencies (e.g., ABET, AACSB, NCAAA) with the following common purposes: • Introduced (I): It means that students are not expected to be proficient with the content or expertise. Learning activities here focus on basic knowledge, comprehension, skills, and competencies at an entry-level (typically assigned in first to second-year courses). • Reinforced (R): Expecting students to have a necessary amount of knowledge and understanding of the content or talents. Learning activities concentrate on enhancing and strengthening knowledge, skills, and expanding complexity (typically assigned in Second, Third, Fourth-year courses). • Emphasized (E): Here, students are expected to have a robust as well as a sophisticated understanding, expertise, or competency base. Instructional and learning activities focus on using the content or skills in multiple contexts and at multiple levels of complexity (capstone courses). Figure 1 exemplifies a representation of the curriculum matrix, which illustrates different issues, such that not all PLOs are properly aligned with courses in accordance with the standard rules for creating a robust mapping. In a nutshell, to address such issues, the following designing steps are suggested to obtain a proper curriculum mapping that may assist in getting more accurate learning-outcome measurements.
1) Learning Outcomes: The learning outcomes at the program level, as well as the course level, must be defined. The learning outcomes must be consistent with the objectives, goals, and mission of the program. 2) Mapping: the learning outcomes of the courses must be aligned with the program learning outcomes. The prerequisite courses should be taken into consideration to ensure cumulative and consecutive learning achievement of students.
3) Alignment Criteria: a set of criteria should be applied to build curriculum mapping. For instance, the logical order of the alignment levels (I, R, E), the number of courses aligned to each program, etc. 4) Involvement of stakeholders: the faculty members, other educational expertise, and the curriculum developer should be involved in developing the curriculum mapping.
? … X X  In Table I, we illustrate a curriculum matrix as a model that visually represents the alignment between program learning outcomes and the curriculum courses. Broadly speaking, there exist four recommend rules of thump to apply for measuring such a curriculum matrix's quality. These rules are derived from the best quality assurance practices as well as from the well-known accreditation agencies [28], [29], [30]. More in detail, the first rule suggests that all course outcomes (with I, R, E) should be aligned to at least three courses [28]. While the second rule focuses on not to align many courses to a particular outcome. Here, the recommended number of courses covering a particular outcome is 3-5 [29]. The third rule suggests attaining the learning outcomes in a logical order, which empowers students to gradually progress their attainments from the first level to greater experience levels. (I, R, E) [30]. The fourth rule suggests having (I) in only the first and second academic levels, which typically should cover all introductory courses. Similarly, (R) is recommended to cover all the mid-advanced courses (i.e., from the third academic level to the last level), and (E) for only capstone courses such as the final senior project [30]. In this paper, we consider these four abstracted rules as the core criteria for assessing a given curriculum matrix using our implemented algorithm, introduced explicitly in algorithm 1. This section presents our proposed helper web-based application tool for automating the assessments of a given academic program. It is publicly available at (http://iudss.com/). The curriculum mapping feature is implemented based on algorithm 1, which provides different runtime figures/charts to explore the efficiency of the mapping at both program and student levels. We imagine that giving insights about the generated figures can help in understanding the overall learning outcomes and efficiency of the educational program at hand. In a nutshell, our tool produces several dynamic charts with the percentage of the mapping quality. An example of these charts is illustrated in Figure 2. Here, a distinct color scheme is used to signify the mapping issues, relying on the identified validating rules.

V. PRELIMINARY EMPIRICAL RESULTS AND DISCUSSION
A straightforward technique to gauge the efficacy of designing a curriculum mapping is to consider various sources, such as students, universities, disciplines, and higher education institutions. Almost every stakeholder has their expectations as to how the curriculum impacts current and future outcomes. In this research, we investigate through a simple user-study (i.e., conducted with curriculum mapping experts) the validity (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 11, No. 12, 2020 of our proposed method compared with various practices for designing curriculum mapping. At the end of this section, we will identify the main benefits that one can gain by having well-defined curriculum mapping.
We have extracted the opinions of curriculum mapping experts, academic professionals, and senior faculties through questionnaires. Commonly, there are many data-collection methods to consider; however, this paper's authors decided to implement a comprehensive questionnaire as it is the most natural and reliable method. This method provides valuable information regarding the experience of the participants on a specific system and measures their level of satisfaction [31], [32], [33]. We have gathered seven experts for completing our questionnaire that consists of ten questions. The first question aims to identify the expertise field of the involved experts, see Table II. The remaining nine questions concentrate on curriculum mapping, scaled as follows; very satisfied, satisfied, neutral, dissatisfied, and finally very dissatisfied.
Most of the experts involved in this questionnaire are curriculum experts (42.9%), whereas the remaining experts are academic professionals (28.6%) and senior faculty members (28.6%). Their valuable experience and knowledge allowed them to analyze how helpful the intelligent mapping curricu-lum evaluated was. More than half claimed that they were very satisfied with what they were provided with (42.9% and 14.3%, respectively). None of the experts were very dissatisfied; however, 14.3% were dissatisfied with how helpful the intelligent mapping curriculum was, whereas 28.6% of the experts took a neutral standpoint. In the third part of the questionnaire, we asked the experts ... Do you think curriculum mapping will be helpful for the satisfaction of learning outcomes? and only 14.3% of them were dissatisfied and another 14.3% were neutral. The remaining experts were either very satisfied or satisfied (42.9% and 28.6% respectively).
The expert's opinions regarding whether or not intelligent curriculum mapping will evaluate the weaknesses of curriculum mappings is as follows; none of the experts showed a negative view in regards to this question, but only 14.3% of them were neutral and the remaining showed a positive attitude towards this question. The fifth question states ... Do you think that the curriculum mapping could visibly allow the curriculum and illustrate the connections between the courses and the student's learning outcomes?. The results of this question varied, 57.1% were very satisfied, 28.6% were neutral and the remaining 14.3% were dissatisfied. The next question discusses whether intelligent curriculum planning will generate the recommendations for improvements in curriculum mapping. Once again, there was no negative responses to this question, but 14.3% were neutral and the remaining had a positive response (42.9% for both very satisfied and satisfied). Question seven debates the transparency of intelligent curriculum mapping; 42.9% were very satisfied, 28.6% were satisfied and the remaining were neutral. Curriculum Expert Expert 7 Academic Professional

Q2:
How helpful is the intelligent curriculum mapping provided to you? Q3: Do you think the curriculum mapping will be helpful for the satisfaction of learning outcomes? Q4: Do you think intelligent curriculum mapping will evaluate the weaknesses of curriculum mappings? Q5: Do you think that the curriculum mapping could visibly allow the curriculum and illustrate the connections between the courses and the student's learning outcomes? Q6: Do you think that the Intelligent curriculum mapping will generate recommendations for improvements in curriculum mapping? Q7: Do you think that the Intelligent curriculum mapping will be transparent? Q8: Do you think that the Intelligent curriculum mapping will align all student learning outcomes fairly? Q9: Do you think the Intelligent curriculum mapping provided the right amount of theoretical and practical experience? Q10: Please set your level of satisfaction for the intelligent curriculum mapping?
The eighth question states ... Do you think that the intelligent curriculum planning will align all student learning outcomes fairly. Here, most experts showed a positive reaction to this question, where 42.9% were very satisfied and another 42.9% were satisfied, the remaining 14.3% took a neutral standpoint. The next question inquires the experts about their opinions regarding the level of theoretical and practical experience of the intelligent curriculum mapping; 28.6% were very satisfied, 42.9% claimed they were satisfied and the remaining 28.6% had a neutral viewpoint. Finally, the results of the level of satisfaction of the experts were gathered, which showed that there were no negative responses, 14.3% were neutral and the remaining experts all had a positive outlook (85.8% of the experts were satisfied or very satisfied) on the level of satisfaction for the intelligent curriculum mapping. In Table II, we present the results of each question for each expert besides the calculated average for each question.
To summing up our observations, we list the main benefits of having an accurate representation of curriculum mapping, as follows: • Ensuring the consistency of courses along with course outcomes. • Effectively applied improvement techniques such as continuous, constant, standardized, and iterative. • Help for continuous academic learning (consistency or incorporation). • Revise and analyze learning outcomes).
• Enabling the professionalism of the academic staff (able to share the learning process). • Discussing transparency problems (specifications, evaluation of programs, student support, and optimization of program outcomes), • Criteria for quality assurance. With consideration for the program's curriculum, each course will need to create a plan to evaluate student learning outcomes.  The exhaustive assessment plan involves the learning outcomes, the process used to evaluate each outcome, the benchmark of each strategy, the person responsible for collecting the information, the information gathering regularity, the responsibility for interpreting the results and detecting modifications, as well as measuring enhancements, illustrated in Table III and Figure 3. We assume a total of 14 students' learning outcomes are divided into Disciplinary Knowledge, Critical Thinking, Communication, Research Skills, and Ethical Reasoning for the required courses [34]. According to the results of this study, most of the experts are satisfied with the proposed approach, see the results of the opinions of the experts in Table II. The curriculum review would be routine, but the number of variables, like input data and the facilities available, depends upon whether the entire curriculum is evaluated. Even so, the institutions must be aware of the need for a large-scale assessment and strategy to conduct the