Deep Autoencoder and Deep Q-Network-Based Feature Learning with Multi-Head Residual Attention DNN for Cyberattack Classification

As society’s digital infrastructure grows, so do the threats. We cannot deny internet access, so we need to safeguard ourselves against cyber threats. Every day, new threats emerge. The most conventional systems classify cyberattacks in a closed set; that is, training and testing use the same set of attack behaviors. In the real world, we cannot train our system on all attack patterns because attack behavior evolves daily. To address this issue, this study proposes open-set cyberattack classification, in which we train our system on a limited set of labeled attack types and, during testing, use a larger unlabeled data sample that includes both known and unknown attack patterns. Our framework integrates a Deep Autoencoder (DAE)-based latent representation, an Explainable Deep Q-Network (Explainable DQN) for feature selection, and a Multi-Head Residual Attention Deep Neural Network (MHRA-DNN) for final classification. The DAE transforms high-dimensional network traffic into a low-dimensional latent feature representation. This reduces redundancy while preserving important feature characteristics. The Explainable DQN selects the features that contribute most to the model’s performance. It also improves explainability by highlighting the most influential features. The selected features are then processed by the MHRA-DNN. It captures multiple feature combinations and strengthens learning through attention mechanisms and residual connections. We evaluate the proposed framework on the benchmark cybersecurity datasets UNSW-NB15, APA-DDoS, and CICIDS2017. We set up an open-set environment that involves known and unknown attack classes. The proposed model performs strongly across accuracy, precision, recall, F1-score, specificity, MCC, and NPV. In addition, parameter tuning improves model stability, and SHAP-based analysis provides evidence of feature contributions to model performance. Overall, the proposed model offers an adaptive, explainable, and robust solution to cyberattack classification in an open-set environment.

@article{Roy2026,
  title     = {Deep Autoencoder and Deep Q-Network-Based Feature Learning with Multi-Head Residual Attention DNN for Cyberattack Classification},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {17},
  number    = {8},
  year      = {2026},
  publisher = {The Science and Information Organization},
  author    = {Dharani Kanta Roy and Hemanta Kumar Kalita},
  doi       = {10.14569/IJACSA.2026.0170883},
  url       = {https://doi.org/10.14569/IJACSA.2026.0170883}
}