Performance Assessment and Analysis of Blended Learning in IT Education: A Longitudinal Study in Saudi Electronic University

Blended learning is a new educational model that binds traditional face-to-face learning with application of modern tools and technologies. This helps in retaining the positive features of the traditional learning while allowing students to realize potential of modern technologies. In blended learning, student perceptions and satisfaction plays a key role. Longitudinal studies can help identify patterns of these perceptions and expectations to evolve blended learning with changing times and technologies. In this paper, a longitudinal study has been carried out with the students and the faculty of Saudi Electronic University to identify major drivers and their role in shaping student perceptions and satisfaction. The results of this longitudinal study have been validated, and their subsequent comparisons are ascertained with the application of a decision tree based data mining technique. Based on the analysis and the findings of this study, the paper presents recommendations to improve blended learning experience and enhance the effectiveness of the teaching pedagogies developed consequently. Keywords—Blended learning; educational data; information technology; longitudinal study; data mining; decision tree


I. INTRODUCTION
Modern learning methods and various pedagogies to impart education have undergone a lot of enhancements during the last five decades [1]. Today, education is not confined to merely traditional classroom based learning. With the advancements in technology, new and versatile learning models have also evolved. Since the advent of the 21st century, integration of synchronous or asynchronous learning technologies have enabled educators to deliver education with innovative approaches using technology. Online learning found a great acceptance among academia for its flexibility and global reach to impart education where traditional infrastructure was hard to be provided. However, it is also a fact that there are certain unique aspects of traditional education, such as human connection, social interaction, spontaneity and personal attention; that cannot be substituted by any online learning approach. There has been always a need to find a learning environment that can combine positive aspects of both traditional and online learning approaches, while avoiding negative aspects associated with both. Blended learning was primarily introduced to achieve this goal.
Blended learning, as a concept, is a convergence of electronic-learning (e-learning) approach and face-to-face learning. It has been regarded as a new paradigm in modern education. The concept of blended learning emerged near the dawn of 21st century and soon found great acceptance amongst all levels of education. Today blended learning is being used at both the elementary and the higher education levels. The core feature of blending learning is its ability to incorporate technology while retaining features of face-to-face education. Today, blended learning is being adopted and studied as effective means of learning in all parts of the world.
For a successful blended learning model, achieving positive student perception plays a crucial role. The students are probably the most critical stakeholders in any academic system, and thus, their level of satisfaction directly reflects on quality of the education model. It is imperative in blended learning to know the level of adoption of technology among students and its measure of achieving students' learning goals. Being in a continuously changing environment of technology, practitioners of blended learning need to be constantly aware of student's perceptions, expectations and demands from their learning facilities, whether be traditional face-to-face or online. This interesting combination of traditional as well as technology driven education model therefore succeeds in captivating students' attention. Therefore, the current research in blended learning points towards the need for constant and up-to-date data collection regarding students' perception, with blended learning model under practice. However, there is still very little work done in the form of concrete studies to identify the role of various technological factors in students' satisfaction and meeting their goals. There is a need for concrete studies which observe student perceptions over a period of time to model their behavior and map it with significant features of blended learning.
Longitudinal studies are a very useful research method that involve continuous incremental data collection and its observation to find underlying models and patterns for any scientific phenomenon. These studies have proven to be effectively useful in many scientific domains such as environmental studies, biological studies and social sciences. Blended learning, as an area of research, can benefit greatly by application of longitudinal studies methods on student data to explore its effectiveness. Since such studies span over considerable period of time, their effectiveness in incorporating various relevant factors overpasses other similar research methods. Longitudinal studies are also a very good means of gauging the effectiveness of blended learning by successfully identifying factors which may be overlooked otherwise. This can help in identifying changes required to better cope with the student expectations and demands. This can also help in evolving learning model with changing technology and other platforms.
Saudi Electronic University (SEU) was established in 2011 as a premier institution of blended learning in Kingdom of Saudi Arabia with the aim to encourage provision of uniform quality education across all the regions of the kingdom. For this purpose, university was tasked to use blended learning as its medium of instruction. During all this time, a valuable amount of data about the student learning, performance, satisfaction, quality drivers and several other benchmarks has been collected and archived. In this paper, we have used a segment of that data to perform a longitudinal study determining student perception of blended learning model adopted in SEU. The data sample consists of information gleaned from more than 478 surveys of 243 students and 16 teachers over a period of four consecutive semesters from 2016 to 2018. The purpose of this study is to identify major quality drivers in positive student perception with blended learning. This study helps in understanding evolution of perception during time under study, which can further assist in shaping the learning model according to these perceptions and aspirations.
The remaining of this paper is organized as follows. After this brief introduction, literature review is presented in section 2. After literature review the data collection process is presented in section 3, followed by discussion and analysis of results in section 4, and decision tree based results validation in section 5. In the end, conclusion and future recommendations are presented in sections 6 and 7 respectively.

II. LITERATURE REVIEW
No academic model can be deemed successful if it fails to meet student perceptions of quality of learning and effectiveness of learning model. Students are the most critical stakeholders in academic systems. Success or failure of a model depends greatly on perception and acceptance of students. Quality of learning for students has been an area of interest for researchers since 1970s, which has helped in refining our concepts about changing dynamics of learning [2]. In [3] the factors that influence quality of learning are proposed, such as approach to learning, course material and student perceptions as shown in Fig. 1 Blended learning as a medium of instruction has been around for more than two decades now. According to a study conducted in 2007, more than 45% undergraduate institutions in USA had adopted Blended learning as early as 2004. In 2001, assessment of effectiveness of blended learning had also begun. An evaluation on use of blended learning in a module at Master program at Cardiff University is presented by Banks [4]. A framework for blended learning is proposed in [5] which could be used to identify most suitable material for education at higher education level. In addition, the problem of creating effective number of assessments and its relation to blended learning environment has been introduced in [6] and [7], whereby, a data mining approach is implanted over educational data to predict the effect of the total number of assessment on student performance.
Pérez et al. [8] showed that applying blended learning can effectively reduce student attrition rate and increase their grasp of concepts resulting in better academic grades. Blended learning can be viewed as a combination of traditional face-toface learning with e-learning [9]. Using distributed learning as a medium of education allows efficient interaction between faculty and students across different locations while retaining features of traditional face-to-face learning. In addition, it maintains a physical contact essential for effective and immediate guidance. There are definitions of blended learning that focus on percentage of time allocated to both face-to-face as well as distributed/online learning. For example, in [10] Bernard et al. proposed equal proportion (50%) contribution to both face to face and distributed learning. In [11], Yen and Lee emphasize that "blended learning, thoughtfully combining the best elements of online and face-to-face education, is likely to emerge as the predominant teaching model of the future". Blended learning provides a personalized and adaptive learning approach to students that can be easily customized to suit the unique need of different students based on their unique characteristics and learning styles [12]. The blended course design involves thoughtful integration of various course delivery methods, learning principles, and instructional technologies which can provide the learners with a flexible, autonomic, and situated learning environment. Thus, blended learning is defined in [13] to be on way to becoming the new norm in higher education learning environments. A significant difference in success rate of students between blended learning environment and traditional environment, with higher success rate in blended learning, is presented in [14]. The authors attribute this higher success of blended learning to its ability in integrating face-to-face teaching that features the presence of an instructor, and e-learning with flexibility and accessibility in learning process. This course delivery method gives students opportunities to share and control learning, and to adapt to different learning context and situations.
Driscoll [15] proposed that blended learning could be explained as a combination of four approaches, namely, (i) Application of online and Information technology to achieve academic goals, (ii) Use of pedagogical approaches to improve learning outcomes, (iii) Amalgamation of information technology with traditional learning, and (iv) A mix of instructional methodology with actual job tasks. This study is 78 | P a g e www.ijacsa.thesai.org (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 11, No. 6, 2020 very significant since it establishes the core competencies and scope of any successful blended learning model.
Student engagement is considered a fundamental and critical aspect of blended learning. It plays a crucial role in success of teaching model by contributing significantly in factors such as grades, persistence, and college completion [16]. By applying various modes of instructions (face to face as well as distributed), student's motivation to engage in self learning increases [17]. In [18], Dringus and Seagull consider student engagement to be the most critical factor for penetration of blended learning. Student engagement can involve many and often diverse factors, ranging from effort and persistence to learning motivation and involvement [19]. Self-report methods have been used effectively in domain of blended learning to measure student engagement. The data collected using this approach has been used to evaluate the blended learning model and study the relationship between student engagement and other important academic outcomes [20].
Longitudinal studies that intend to measure student engagement by capturing immediate student experience throughout blended learning, have been advocated as a means of effective quality enhancement mechanism in recent times [21], [22]. Longitudinal studies involve multiple measurements over time to model effectiveness of a process and to identify changes over the course of time [23]. These studies not only help in collecting specific data linked to activities that motivate student's performance in sphere of blended learning, but also provide researchers with ability to link these activities with driving factors and their relative influence. Another advantage of applying longitudinal studies is to have more transparency in data because of its multi-interval nature, since it is collected regularly over a period of time.

III. DATA FOR LONGITUDINAL STUDY
The student data collected for experimentation in this study is derived from Saudi Electronic University (SEU) systems. Various systems are in operation in SEU for academic purposes including Blackboard (Learning Management System (LMS)), Banner (registration system), and Attendance System etc. Details of data collected for analysis and experimentation can be summarized as following.
The dataset used consist of two parts. The first part is collected from surveys that were done on 243 different students having a total of 478 surveys and 16 faculty members over four consequent semesters of undergraduate students in IT program from 2016 to 2018. The second part of the dataset comprises of students' extracted data from the registration system (i.e. the Banner), along with students' data from Blackboard LMS.
In this study, three different experiments are conducted in order to measure multiple factors. The first part concentrates on students' perspective, measuring the following factors: 1) Effectiveness of Learning programming language in blended learning environment.
2) Effectiveness of applying practical labs for students versus having the same course without labs (Last two semester labs have been applied for programming courses).
3) Effectiveness of updating and modifying courses content.
4) Effectiveness of the total number of assignments per course.
5) Effectiveness of online quizzes 6) Effectiveness of participating on forums. 7) Effectiveness and easiness of using IT tools in Blended Learning System. 8) Rate of Satisfaction with IT Systems used for Blended Learning.
9) The second part of the study concentrates on faculty members and their satisfaction level for providing programming courses in blended learning environments 10) Faculty members' satisfactions with student performance in programming courses. 11) Faculty members' impression of applying practical labs. 12) Faculty members' satisfactions on updating and modifying courses contents.
14) Faculty members' satisfactions on online quizzes 15) Faculty members' satisfactions on forums feedback to students.
16) Effectiveness and Easiness of using IT tools in blended learning system. 17) Rate of satisfaction with IT systems used for blended learning.
The third part of the study implements a decision tree on students' data to ensure the confidence of the analysis and results.

IV. RESULTS AND ANALYSIS
In the following subsections, we present some of the salient findings of the study from all the three perspectives that we contemplated.

A. Student Feedback and Assessment
In this part of the study, we choose four courses in programming field, namely, (i) Computer Programming, (ii) Advanced Computer Programming, (iii) Web Technologies and (iv) Mobile Application Development. Major reason to select these course courses for experimental purposes is their applied nature that would require intensive collaboration between faculty and students on one hand, and offer an excellent opportunity to demonstrate effectiveness of IT systems in learning on the other hand. Subsequently, the survey was conducted after execution of each of these courses during the period of data collection. These four selected courses were then presented to the students in a blended learning environment through LMS comprising the academic and assessment resources.
Students answered an anonymous survey, which consisted of 32 questions in an online form after completion of each course. The total number of collected surveys were 478. The students selected to participate in survey comprised of various cross-sections of participants. Student were from different age groups, as diverse as from 18 to 39 years. In addition, they belonged to different academic backgrounds such as high school, bachelors or diplomas in different areas of specializations. All these parameters were taken into consideration in building the students' performance prediction model. These surveys were conducted for five consecutive semesters in order to make sure that as many environmental factors as possible could be taken into consideration that could affect the outcome of the study. The resultant was a longitudinal study which test the following parameters: 1) P1: Effectiveness of learning programming language in blended learning environment.
2) P2: Effectiveness of applying practical labs for students versus having the same course without labs (Last two semester labs have been applied for programming courses) 3) P3: Effectiveness of updating and modifying courses content. These eight parameters are being used almost universally to measure the effectiveness of academic resources, assessment and application of knowledge. As seen in Fig. 2, for the first parameter we have more than 79% of students with great acceptance for programming courses in the blended learning environment. In addition, 84% of the students have approved the practical labs.
For the effectiveness of content update, we have around 73% of students. This shows that no real update on the content has been done and they feel no effectiveness of applied changes on the content. This depicts an effective need for constant review and upgrade of course contents in a blended learning environment. This is one of the significantly worrisome areas of blended learning, which shows that continuous administrative oversight is necessary to update the academic resources with changing environment. For the acceptance of a large number of applied assignments in blended learning environment, it has been shown from the results that around 83% of the students consider the number of assessments (assignments, quizzes, discussions and projects) to be excessive and are in favor of reducing these. Of all assessments results, it is evident that there is a good acceptance of online Quizzes, whereby, around 74% of students approve that. On the other hand, forums participation shows average acceptance from students for participation. Table II, more than 75% of the students have shown satisfaction with the first three parameters P1, P2 and P5. This indicates the higher acceptance from the students for programing courses, practical labs and having online quizzes in the blended learning environment.

1) Analysis of student response: As is evident from
On the other hand, for parameters P3, P4 and P6, most of students have natural response highlighting the need for updates and changes to course contents, number of assignments and forums, but not having high effectiveness on students.
Major areas of concern can be observed for parameters P7 and P8 where 30% have expressed facing difficulties in using IT tools as well as dissatisfaction with Blended Learning environment. Interestingly this is at par (30%) with students expressing greater acceptance and satisfaction of using IT systems in blended learning. One more observation during survey was about demographic variance and acceptance of blended learning. It is found that most of the students who faced difficulties are between 33 and 39 years old. Meanwhile, students who accepted blended learning tools are mostly under 30 years old. This shows that demographics can play a significant role in the success or failure of blended learning systems and should be considered as a significant parameter. Fig. 3 show the results for the six parameters over the five semesters. The first figure (P1) indicates that student level of acceptance for blended learning environment is increasing over semesters, where the total number of students who agree and strongly agree is increasing in last three semesters.  For the second parameter, which measures the effectiveness of practical labs, at the first two semesters there appears to be a general resistance in applying the labs (P2). However, over a period of time, students are more eager to attend labs.
The third, fourth and sixth figures (P3, P4, and P5) show the effectiveness of course content updates, number of assignments and participation in forums, respectively. These patterns show that most of the students do not agree on these parameters, while the level of effectiveness is increasing over semesters with very low rate. In addition, the fifth figure, that measures the effectiveness of using online quizzes, shows a great impact and acceptance of students and it increases over semesters.
A very interesting observation is made about last two parameters. The seventh and eighth parameters start with high disagreement from students in initial levels of their education. However, it shows that over the semesters, students get more involved in blended learning environment and using different IT systems applied in this environment. In the last two semesters, the number of disagreed students is reduced, while more students found it easy to use IT tools and more satisfied with blended learning environment. This shows that student perception of blended learning changes as their expertise in working with systems involved improves. It also shows that in order to make blended learning universally effective, novice students need to be presented with more opportunities to interact with systems on experimental basis.

B. Faculty Feedback and Assessment
In this part, the survey were gathered over five semesters from teaching staff and faculty. The total number of staff members participating in the survey were 16, with 102 surveys. The faculty feedback about effectiveness and perception of blended learning systems can naturally vary to a great degree from students due to their expertise and knowledge. However, it can provide a unique opportunity to identify converging and conflicting factors between students and faculty. 1) P1: Faculty members satisfactions with student performance in programming courses.
3) P3: Faculty members satisfactions on updating and modifying courses contents. 4) P4: Faculty members satisfactions on predesigned assignments. 5) P5: Faculty members satisfactions on online quizzes 6) P6: Faculty members satisfactions on forums feedback to students. 7) P7: Effectiveness and Easiness of using IT tools in Blended Learning System. 8) P8: Rate of Satisfaction with IT Systems used for Blended Learning. As is evident from Faculty members' results shows a great acceptance of students' performance, applying practical labs, modifying courses contents, quizzes and forums. While about 84% of the faculty members are not satisfied with the currently designed assignments and advise to redesign new assignments for students. This result is very useful and reflect the same disagreement from the students against the current assignments. Table III shows the number of faculty and staff surveyed over a period of five semesters, whereas, Table IV and Fig. 4 presents the percentage of the staff responses on different parameters. 81 | P a g e www.ijacsa.thesai.org  Table IV shows feedback from faculty to various parameters being assessed during this study. It shows that most staff members have high confidence on parameters P1, P2, P3, P5 and P7. Faculty has shown a need to have more meaningful practical component in combination with conventional blended learning based teaching for programming courses. It was also observed that faculty showed greater confidence in change rate of course contents and online quizzes as compared to students. This can be attributed to active participation from faculty in actual revision and update process. In addition, the IT tools offered in blended learning were found to be very effective and easy to use by faculty members.

1) Analysis for faculty response:
The biggest area in concern was discovered about predesigned assignments and assessment tools. Faculty showed a greater need to redesign and prepare custom assignments. For parameters P6 and P8, staff members have neutral response for the number of forums applied in each course. In addition, staff members show average satisfaction of current IT systems used, and provided suggestions for improving these systems.

C. Comparative Analysis of Student and Faculty Response
One important aspect of our study was to find areas of convergence and divergence between assessments made by faculty and students. The combined assessment is shown in Fig. 5. Fig. 5 presents a comprehensive and interesting assessment where it can be observed that there is a great similarity between perceptions of both stakeholders for parameters P1, P2, P4, P5 and P6. The results show that high similarity is attained between the disagreement and agreement percentage for both the staff and the students for these parameters.
However, P3 is one parameter that indicates that staff members highly agree with the current changes and updates every semester on the course contents while students not agree on that. In addition, P7 shows the similar results which indicate that staff members found it easy and very effective to make use of different IT tools in blended learning environment. On the contrary, students responses do not depict that they agree on this. The last parameter shows the level of satisfaction to IT systems applied in blended learning environment. IT refers staff members have more neutral responses, while students show higher dissatisfaction, and suggest improvement to the currently applied IT systems.
These results show that even after the application of blended learning systems for a long time now, two major stakeholders in any academic system can differ greatly in their perception about effectiveness of the model. Our proposed decision tree based system, as described and explained in subsequent sections, is an attempt to present a uniform set of conditions that can help in ensuring uniformity of perception amongst both the students and the faculty alike.

V. DECISION TREE BASED RESULTS VALIDATION
In order to prove the results from the first two parts and ensure the confidence of the analysis, students data is extracted from the registration system (i.e. Banner), along with students data from the LMS. The extracted dataset represents all students participated in the survey enrolled in Information Technology program over five semesters. Data contains seven attributes as show in Table V.
In order to remove any errors and clean the data, a preprocessing phase is incorporated into the proposed scheme. This also removes any errors pertaining to the entry of the data in addition to the irrelevant attributes. The main objective of this approach is to find a relation between students' course GPA and the total number of assignments, quizzes and forums participation. Therefore, J48 classifier is used on the extracted and prepared dataset to provide a decision tree. The dataset, additionally, goes through a second step of preprocessing to convert the data into suitable format for the decision tree algorithm. Table VI shows the discretization rules applied on the dataset. 82 | P a g e www.ijacsa.thesai.org  The data for the decision tress is divided into 66.6% for the training and 33.3% for the testing. The experimental results demonstrate that 78.6 % instances are correctly classified, while the incorrectly classified instances are 21.2%. Fig. 6 demonstrates the evolved decision tree structure. The results show that 92.2% of students, who solved more than five assignments during each course, got a higher GPA (over 2.0) and classified in stable stats during the IT program. While 88.8% of students who have less than 2 assignments, 2 quizzes and 2-forums participation are in critical status, with a GPA of less than 2.0.
This tree depicts very helpful results for the decision makers and for the academic advisors that could guide students during their enrollment in the IT program. In addition, we found that the age and gender are not critical attributes for students' status during the program. Table VII shows the following performance measures for each of the predicted classes: • The True Positive (TP) rate: a measure for ratio where the model correctly predicts the positive class.
• The False Positive (FP) rate: measure for ratio where the model incorrectly predicts the positive class.
• Precision: a measure of how precise/accurate your model is out of those predicted positive, how many of them are actual positive.

= +
• Recall: calculates how many of the Actual Positives our model capture through labeling it as Positive.   In the second experiment, the results of applying practical labs and its influence on students' performance in programming courses is investigated. The same dataset, as was used in the previous experiment, is applied with some different attributes. These selected attributes are listed in Table VIII. Initially, a preprocessing phase is applied on the dataset to remove the null values. J48 classifier is then used on the extracted and prepared dataset to provide a decision tree. The same discretization process, as presented in Table VI, is applied with the addition of a new rule for the new attribute Lab Attendance as presented in Table IX. The resultant decision tree (Fig. 7) imparts a very important indication for the effectiveness of practical labs. As is evident that 91% of the students are less than 24 years old and attended less than two labs are in critical status. While, an average of 87% of male and female students are between 24 years and 38 years and attended more than 2 labs, have GPA over 2.0. In addition, for students over 38 years old, there is no direct impact from practical labs. Table X presents the discretization rule for the above mentioned investigation.

VI. RECOMMENDATIONS AND GUIDELINES
This section introduce and summarize some suggestions for improving the effectiveness of blended learning environment based on students and staff members' responses which are summarized as follows: Students have shown enhanced grasp of subject matter when subjected to practical work. Increasing practical labs for programming courses is preferable, as it would increase the success rate in these courses.
The need for formal course review was highlighted through both the student and the faculty responses. Course contents should be updated every semester and staff members should optimize application of various IT tools in blended learning for assessments and evaluation.
The number of home based assessments need to be revisited. The overall number of assessments in each course should be revisited and redesigned if necessary.
Online forums must be restructured in order to attract students and make it more effective during the learning process.
University should introduce training courses for both students and faculty on how to make best use of different IT systems applied in the Blended Learning environment.
The IT systems can be enhanced to be more user friendly and displaying course contents in attractive way to students, such as to increase the overall experience of blended learning.

VII. CONCLUSION
Blended learning combines good practices of traditional face to face learning and online learning. Blended learning relies heavily on identifying and meeting student perceptions for its success and evolution. It is very important to collect and analyze student data to meet this objective. In this paper a longitudinal study has been carried out on student data from Saudi Electronic University. This study has later been validated by comparing results with a decision tree based model of student as well as faculty responses. The results of this study show that both students and faculty have shown greater confidence in application of blended learning for education processes. However, the study also shows greater need for constant evolution and improvement in course content and assessment tools. In future, the authors intend to explore advanced heuristics based classifiers, such as Genetic Programming and Deep Learning, in order to exploit hidden dependencies in the solution domain which are often overlooked by traditional classifiers.