Hybrid Machine Learning-Based Approach for Anomaly Detection using Apache Spark

Abstract: Over the past few decades, the volume of data has increased significantly in both scientific institutions and universities, with a large number of students enrolled and a high volume of related data. Furthermore, network traffic has increased with post-pandemic and the use of online learning. Therefore, processing network traffic data is a complex and challenging task that increases the possibility of intrusions and anomalies. Traditional security systems cannot deal with such high-speed and big data traffic. Real-time anomaly detection should be able to process data as quickly as possible to detect abnormal and malicious data. This paper proposes a hybrid approach consisting of supervised and unsupervised learning for anomaly detection based on the big data engine Apache Spark. Initially, the k-means algorithm was implemented in Sparks MLlib for clustering network traffic, then for each cluster, K-nearest neighbors algorithm (KNN) was implemented for classification and anomaly detection. The proposed model was trained and validated against a real dataset from Ibn Zohr University. The results indicate that the proposed model outperformed other well-known algorithms in detecting anomalies based on the aforementioned dataset. The experimental results show that the proposed hybrid approach can reach up to 99.94 % accuracy using the k-fold cross-validation method in the complete dataset with all 48 features.

Hanane Chliah, Amal Battou, Maryem Ait el hadj and Adil Laoufi. “Hybrid Machine Learning-Based Approach for Anomaly Detection using Apache Spark”. International Journal of Advanced Computer Science and Applications (IJACSA) 14.4 (2023). http://dx.doi.org/10.14569/IJACSA.2023.0140496

@article{Chliah2023,
title = {Hybrid Machine Learning-Based Approach for Anomaly Detection using Apache Spark},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2023.0140496},
url = {http://dx.doi.org/10.14569/IJACSA.2023.0140496},
year = {2023},
publisher = {The Science and Information Organization},
volume = {14},
number = {4},
author = {Hanane Chliah and Amal Battou and Maryem Ait el hadj and Adil Laoufi}
}