A Novel High Dimensional and High Speed Data Streams Algorithm: HSDStream

This paper presents a novel high speed clustering scheme for high-dimensional data stream. Data stream clustering has gained importance in different applications, for example, network monitoring, intrusion detection, and real-time sensing. High dimensional stream data is inherently more complex when used for clustering because the evolving nature of the stream data and high dimensionality make it non-trivial. In order to tackle this problem, projected subspace within the high dimensions and limited window sized data per unit of time are used for clustering purpose. We propose a High Speed and Dimensions data stream clustering scheme (HSDStream) which employs exponential mov-ing averages to reduce the size of the memory and speed up the processing of projected subspace data stream. It works in three steps: i) initialization, ii) real-time maintenance of core and outlier micro-clusters, and iii) on-demand offline generation of the final clusters. The proposed algorithm is tested against high dimensional density-based projected clustering (HDDStream) for cluster purity, memory usage, and the cluster sensitivity. Experi-mental results are obtained for corrected KDD intrusion detection dataset. These results show that HSDStream outperforms the HDDStream in all performance metrics, especially, the memory usage and the processing speed.

@article{Ahmed2016,
  title     = {A Novel High Dimensional and High Speed Data Streams Algorithm: HSDStream},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {7},
  number    = {9},
  year      = {2016},
  publisher = {The Science and Information Organization},
  author    = {Irshad Ahmed and Irfan Ahmed and Waseem Shahzad},
  doi       = {10.14569/IJACSA.2016.070952},
  url       = {https://doi.org/10.14569/IJACSA.2016.070952}
}