Knowledge Level Assessment in e-Learning Systems Using Machine Learning and User Activity Analysis

Electronic Learning has been one of the foremost trends in education so far. Such importance draws the attention to an important shift in the educational paradigm. Due to the complexity of the evolving paradigm, the prospective dynamics of learning require an evolution of knowledge delivery and evaluation. This research work tries to put in hand a futuristic design of an autonomous and intelligent e-Learning system. In which machine learning and user activity analysis play the role of an automatic evaluator for the knowledge level. It is important to assess the knowledge level in order to adapt content presentation and to have more realistic evaluation of online learners. Several classification algorithms are applied to predict the knowledge level of the learners and the corresponding results are reported. Furthermore, this research proposes a modern design of a dynamic learning environment that goes along the most recent trends in e-Learning. The experimental results illustrate an overall performance superiority of a support vector machine model in evaluating the knowledge levels; having 98.6% of correctly classified instances with 0.0069 mean absolute error. Keywords—Concept Maps; Multi-Class Classification; Machine Learning; Electronic Learning; Activity Analysis


I. INTRODUCTION
Incorporating the technology in the learning process is complex, but at the same time may shift the paradigm of learning and training.Authors in literature [1], [2] argue that e-Learning offers a balance between the technology enablers and the acceptance issue.According to a twenty-year chronology of e-Learning evolution, Rosenberg in [1] made available an important road map for keeping the e-learning in a sustainable and continuous growth.
Automatic knowledge evaluation is an important component of online learning systems due to the nature of such systems, as the absence of direct contact between instructors and learners.On the other hand, content presentation needs to be tailored according to personal knowledge levels in some contexts.Furthermore, commonly used examinations and assignments may not reflect the actual knowledge level precisely in the context of automatic evaluation.Such issues regarding online learners evaluation drags the attention to a more comprehensive evaluation scheme.For example the time spent by online users in reading the content and the progress may give reasonable indications of the knowledge level.e-Learning as a system of knowledge dissemination and evaluation has been assessed by various researchers [3]- [8].Part of the assessment tackled the usability of such systems using machine learning [3], acceptance [4], the effect of student satisfaction on the learning process [5], and the integration of social networks [8].It is clear that the dynamics of an e-Learning system may affect the learning process in both the positive or negative directions.
Despite the bottlenecks and diversity of learners, it is still a promising trend that is expanding over time to shift towards electronic means of learning and evaluation [9].Furthermore, mobile e-learning is expected to be a major trend according to the statistical study in [6] that investigated the behaviors for mobile learning.
Going further in developing futuristic e-learning platforms and approaches requires intensive analysis of more than an exam paper or in class assignment.Complex and wide spread learning activities ask for an intervention of artificial intelligence to reveal hidden or cluttered online activities.Data mining techniques represent an example of the computer intervention to address different e-Learning issues [10].
The behaviors of the learners in electronic learning systems are collected by different means in [6], for example the mobile device sensors, touch screen activity monitor, and mini keyboard input.Further collections of some behaviors are gathered and measured in [11], then a classification scheme is used to evaluate the users knowledge.This research work tries to find the best algorithm among different machine learning approaches for classification, in order to recommend a suitable autonomous evaluator for the user's knowledge level in dynamic e-Learning systems.The outcomes of this research are expected to aid in building an efficient architecture of future dynamic e-Learning environments.Furthermore, it would illustrate different performance considerations of the classification approaches in the problem domain.The main objective in brief is to recommend an automatic evaluator of learners, which takes into account the user's activity beside the various forms of examinations.This document is structured as follows: Section II highlights the related work the related work that employed machine learning in building evaluation systems.Section III presents a proposed architecture of an evaluation system in dynamic e-Learning environments, Section IV describes the used data set and the evaluation metrics for the classification algorithms.The empirical results of the classification algorithms and the comparisons are discussed in Section V. Then concluding with a summary of the remarks and possible future work.www.ijacsa.thesai.org

II. RELATED WORK
The use of computer intelligence in user-adaptable e-Learning environments has been addressed and examined by several researchers.
An interesting approach is presented in [11] for modeling the knowledge of e-Learning users under different domains.The approach is composed of a generic domain object model, user modeling, weight adjusting method, and a classification algorithm.The authors claim a successful implementation of a robust, domain independent and efficient approach that relies on Bayesian Network and Nearest Neighbor classification algorithms.Consequently, the authors present an intuitive activity-based evaluation approach for adaptive web-based learning environments.
One of the possible dimensions of an e-Learning system is the adaptation to the knowledge level of the user.The authors in [12] extended an IEEE reference model for Learning Technology Systems Architecture (LTSA) [13] to include knowledge level adaptability.The proposed extension alongside the LTSA form an adaptable web-based environment that relies on the users behavior as one of the evaluation parameters.
Automatic evaluation of online learning assignments is discussed in [14] as the authors highlight its importance and applicability.The authors reflected on the promising outcomes of an automatic assessment tool, based on Latent Semantic Analysis, that has been examined in a specific online learning environment.Using automatic scoring of an assignment gives the learners an instant access to their performance anytime, without the need for direct supervision by an instructor.Such automation makes it more flexible for learners to schedule their time and to work on their weak points.
In [15] the authors focused on the learners' requirements to achieve a flexible web-based learning system.The proposed evaluation methods rely on introducing the semantic web and cognitive maps to the LTSA and Sharable Content Object Reference Model (SCORM) [16].The analysis of semantic web content and the cognitive model of the students is discussed by different researchers in the learning and evaluation of learners knowledge [17]- [22].For example, intelligent agents are supposed to enhance self assessment and feedbacks in e-Learning environments that are based on concept maps [22].

III. THE PROPOSED FRAMEWORK
The proposed framework for the automatic assessment of knowledge levels tries to consolidate the efforts of several researchers and standards [11]- [13], [16].In addition, it includes the idea of mobile learning [2], [6] and flexible e-Learning environments.Furthermore, knowledge representation was inspired from the work of [17], [20], [22] which illustrates the role of "Concept Maps [20]" approach in learning and assessment.Fig. 1 represents an abstract architecture of the proposed e-Learning system.
The main aspects of the proposed framework are:

A. User Experience
The e-Learning users have the ability to access the educational system via different web-enabled means with disregard Fig. 1: Proposed e-Learning System Architecture to time schedules.It is meant to be flexible and adaptable learning environment to the users' lifestyle.The objective of device-independent suitability is to offer a wide range of options to those unable to learn and work under strict time schedules, for example employees, housewives, travelers, and learners with high possibility of relocation.Further advantages include the opportunity of global reach or possible placeindependent collaboration between content providers.
The ultimate goal of any learning process is to deliver the proper amount of knowledge and to accurately judge the attainment of the goals.The authors in [23] investigated the possibility of tracking the user attention for higher level of engagement.In the absence of human control, tracking the user environment and providing a natural user interface would solve the reliability issue of such autonomous systems.Moreover, language teaching and practical training needs incorporating the users in a different manner, for example voice/motion recognition and evaluation.

B. Cloud-Based Environment
Cloud-based e-learning systems [24] offer a wide range of opportunities, for example competitive cost accompanied by a high level of scalability.The proposed secure cloud-based environment offers a number of services for rich user experience.Concurrently, there is a subset of services (Intelligent Agents) dedicated to record the user activities.The collection of the activity logs and the examination results of each user will be stored in a repository.Later on, the repository will be used in evaluating the knowledge level of the user.
Two main consideration are important here that are the design of the content and the delivery method.In [25] the authors highlighted one of the issues related to the acceptance of digital newspaper reading over paper-based version.Though, the delivery system of the digital content requires a significant value proposition in order to have higher rate of acceptance.www.ijacsa.thesai.org

C. Incorporating Concept Maps
Concept Maps have been highlighted recently by several researchers in the domain of automatic assessment of learners' knowledge [26]- [28].In the theme of distance learning and self assessment, Concept Maps have various advantages over traditional approaches.The maps of the concepts will be an important tool in order to understand the level of knowledge for each system user.
The contribution is expected to highly facilitate a modern and efficient autonomous method of evaluation.A simple map consists of concepts denoted by circles in Fig. 2 while the underlying relations are denoted by annotated links.Also, it is possible to use concept maps in conjunction with rich media objects for a higher level of content enrichment.

D. Generating Activity Reports
A summary of the user activity includes several derived variables that represent the activity, concept maps assessment, and examination results.Each variable is a result of a preprocessing step that produces a representative value of the corresponding activity.Part of the variables are to be produced by an approach similar to the work in [11].Pre-processing step determines which specific attributes per user will be processed by the consecutive classification step.Main pre-processing aspects are as follows: • Feature extraction and selection from the user activity log (for example average time spent per study objective).
• Categorizing the extracted features.
• Concept map analysis.
• Normalizing the results of the exam sets.
• Attention level (for example average time of focus).
This part of the system is essential and would need a thorough investigation in order to come up with a robust and efficient processing algorithms.One of the main concerns is to find linear and straightforward mappings between the inputs and the computed output values [29].

E. Knowledge Level Assessment
The ultimate goal of the proposed system is to provide an accurate and automatic evaluation of the knowledge level by analyzing the variables in the activity report.An effective Machine Learning method for classification provides the desired outcomes using the user activity model.However, there are various classification algorithms that vary in performance and require a solid experimental methodology for selecting the proper algorithm.
Two possible configurations need to be considered before building the classification models, in terms of domain knowledge dependence, to be either generic or specific.Proper selection of the configuration will result in a classifier for a specific domain, or alternatively a generic classifier.However, this research relies on a specific data set in a specific domain in building the classification models.On which several classification algorithms show promising results towards fulfilling the requirements of knowledge evaluation.

IV. DATA DESCRIPTION AND EVALUATION METRICS
The machine learning classification models are built and evaluated in this research using the user knowledge modeling data set 1 for an e-Learning environment [11].The data set represents the user's model and divided into three main sections that are the individual behavioral attributes, the exam performance attributes, and the knowledge level as target attribute.There are 403 instances from which 145 instances are used for model validation and the rest are used for creating the models.Table I shows the five input variables and the output variable description, Fig. 3 illustrates the distribution of the training instances, and Fig. 4 illustrates the distribution of the test instances.Several algorithms in this research attempted to classify the "test" instances using the developed model, where building the model is based on the "training" instances only.Fig. 5 illustrates the "Confusion or Contingency Matrix"; which is a visual representation of the classified (predicted) instances confronted with the actual instances [30].The performance evaluation of the applied algorithms relies on different measures over the classified instances.The classification's "confusion", alternatively called classification error, is illustrated in the rows holding actual classes and columns holding the predicted classes.The number of correctly classified instances resides diagonally in True Positive (TP) and True Negative (TN).TN is the "Actual Correct" classified as "True" while TN is the "Actual Incorrect" classified as "False".Better performing algorithm will have the highest number of TP and TN.Incorrectly classified will be in False Positive (FP) and False Negative (FN) which are "Actual www.ijacsa.thesai.orgCorrect" classified as "False" and "Actual Incorrect" classified as "True" respectively.The total number of the count in the all the cells (A, B, C, and D) is equal to the total number of the input instances.
Relying on the confusion matrix it is possible to formulate several performance indicators that give better understanding of the results.The empirical results of this research are used to report the following performance measure: • Total Accuracy (Correctly Classified Instances) • Recall (alternatively TP Rate , Sensitivity, Hit Rate, or Type II Error) • Precision (Confidence or Type I Error) • F-Measure • Area Under Receiver Operating Characteristics Curve (ROC) [31] [30] is a statistical measure that helps in comparing the performance of the classifiers.Such measure indicates the probability of a classification algorithm in giving higher rank for positive value over negative values, in other words it confronts the TP rate to the TN rate.
The implemented algorithms create a model using the training data set and consequently use the test data set for validation.Table II shows the tuned parameters of the applied algorithms.The classification results of the test data set illustrate a significant performance superiority of SVM, MLP, and SL in terms of total accuracy.Incorrectly classified instances are 1.39% of the total prediction results for the best models.Referring to total accuracy of all the models in Fig. 6, It is clear that Naive based algorithms are unable to exceed 88% of total accuracy, while Tree-based algorithms are slightly better.Fig. 6: Total accuracy of the built models over test dataset Further analysis of the error rate is illustrated in Fig. 7, in which the SVM model demonstrates the lowest error margin for all the measures.Followed by MLP and SL respectively.Tree-based and Naive-based algorithms show several variations among the different measures, which makes it more difficult to judge which one has an overall better error margin.
The analysis of the confusion matrix for all the classes is summarized and illustrated in Fig. 8, in which the weighted average of all the measures for all the classes represents the overall performance of the algorithms.III.The predictions of the SVM-based model were highly accurate, however the algorithms seems to be clustered into three main groups.The first group has SVM, MLP, and SL; the second group includes Random Trees; and the third group includes Naive-based algorithms.Consequently, there is a relation between the general type of the classification algorithm and its performance.

VI. CONCLUSIONS AND FUTURE WORK
This research introduced a number of enhancements to dynamic e-Learning systems in terms of knowledge delivery and evaluation.It proposed an abstract framework of futuristic learning systems with more focus on the use of machine learning for evaluation.The analysis of various user activities in such dynamic environments leads to a comprehensive evaluation scheme, that is not biased and improves commonly used evaluation methods.In literature there exists a tremendous amount of attempts to empirically prove the accuracy of agentbased evaluation and knowledge dissemination over traditional approaches.Combining the research efforts found in literature and several standards led to the formulation of a common base for the prospective learning systems, at the same time acts as a technological enabler for the e-Learning trends.SVM models over-perform several classification models in terms of stability and error rates, therefore SVM-based models can play an important role in the evaluation of knowledge levels.MLP and SL models show a competitive performance, while Naive and Tree based models do not show a significant capability.
Future work needs to investigate the importance of input variables to the classification method.Possibly, importanceweighted input variables would increase the prediction accu-racy as it would mimic the evaluation behavior of humanbased methods.Dimensionality reduction and the tradeoff between domain knowledge independence and model generalization is an issue that is not covered in this research work.Furthermore, the adaptability of the delivered content to the users according to knowledge level in serving educational programs and training as well.It would be also possible to use the dynamic learning environment in corporate learning and workers evaluation.Looking further for a totally autonomous and adaptive e-Learning system in the future.

Fig. 9 ,
Fig. 9, 10, 11, and 12 confront the classification results drawn from the confusion matrix for the four classes.Where the variations in the performance of each algorithm for the different classes is clear.Also it is apparent that MLP, SL, and SVM are less tolerant to the differences between the classes.By looking at all the performance measures, it is evident that SVM-based classification models over-perform the other

TABLE I :
Input and Output Variables VariableDescription STGThe degree of study time for goal object materials SCGThe degree of repetition number of user for goal object materials STRThe degree of study time of user for related objects with goal object LPRThe exam performance of user for related objects with goal object PEGThe exam performance of user for goal objects UNSThe knowledge level of user (Output Classes) Very Low (Beginner) Low (Intermediate) Middle (Expert) High (Advanced)

TABLE II :
Tuned parameters of the algorithms

TABLE III :
Weighted average of the performance measures