Application of Big Data Mining System Integrating Spectral Clustering Algorithm and Apache Spark Framework

Spectral clustering algorithm is a highly effective clustering algorithm with broad application prospects in data mining. To improve the efficient data processing capability of big data mining systems, a big data mining system that integrates spectral clustering algorithm and Apache Spark framework is proposed. It is applied in the big data mining system by combining Hdoop, Spark framework, and spectral clustering algorithm. The research results indicated that after 300 iterations of spectral clustering algorithm, the error value tended to stabilize and drops to 0.123. In different datasets, different error values were displayed, indicating that spectral clustering algorithm had better performance in discrete data processing and smaller testing errors. The minimum time consumed by the comparative system was 37.83 seconds, the maximum time was 55.26 seconds, and the average time was 51.65 seconds. The minimum time consumed by the research system was 18.93 seconds, the maximum time consumed was 32.22 seconds, and the average time consumed was 28.14 seconds. Compared with the comparative system, the research system consumed less time, trained faster, and was more conducive to shortening the clustering running time. The algorithm framework and system raised in the research have good operational efficiency and clustering ability in data mining processing, which promotes the reliability and development of big data mining systems.

@article{Guo2025,
  title     = {Application of Big Data Mining System Integrating Spectral Clustering Algorithm and Apache Spark Framework},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {16},
  number    = {1},
  year      = {2025},
  publisher = {The Science and Information Organization},
  author    = {Yuansheng Guo},
  doi       = {10.14569/IJACSA.2025.0160165},
  url       = {https://doi.org/10.14569/IJACSA.2025.0160165}
}