Hybrid Machine Learning-Based Approach for Anomaly Detection using Apache Spark

Over the past few decades, the volume of data has increased significantly in both scientific institutions and universities, with a large number of students enrolled and a high volume of related data. Furthermore, network traffic has increased with post-pandemic and the use of online learning. Therefore, processing network traffic data is a complex and challenging task that increases the possibility of intrusions and anomalies. Traditional security systems cannot deal with such high-speed and big data traffic. Real-time anomaly detection should be able to process data as quickly as possible to detect abnormal and malicious data. This paper proposes a hybrid approach consisting of supervised and unsupervised learning for anomaly detection based on the big data engine Apache Spark. Initially, the k-means algorithm was implemented in Sparks MLlib for clustering network traffic, then for each cluster, K-nearest neighbors algorithm (KNN) was implemented for classification and anomaly detection. The proposed model was trained and validated against a real dataset from Ibn Zohr University. The results indicate that the proposed model outperformed other well-known algorithms in detecting anomalies based on the aforementioned dataset. The experimental results show that the proposed hybrid approach can reach up to 99.94 % accuracy using the k-fold cross-validation method in the complete dataset with all 48 features.

@article{Chliah2023,
  title     = {Hybrid Machine Learning-Based Approach for Anomaly Detection using Apache Spark},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {14},
  number    = {4},
  year      = {2023},
  publisher = {The Science and Information Organization},
  author    = {Hanane Chliah and Amal Battou and Maryem Ait el hadj and Adil Laoufi},
  doi       = {10.14569/IJACSA.2023.0140496},
  url       = {https://doi.org/10.14569/IJACSA.2023.0140496}
}