A Leakage-Aware and Reproducible Evaluation Framework for Predictive Maintenance Classification

Abstract: Predictive maintenance classification is widely used to support industrial maintenance planning; however, reported model performance is often influenced by evaluation practices that allow unintended information leakage between training and testing data, resulting in optimistic and difficult-to-reproduce estimates. This study examines predictive maintenance classification from the perspective of evaluation design, with a specific focus on quantifying the impact of leakage on performance assessment. A leakage-aware and fully reproducible evaluation protocol is implemented on the AI4I 2020 dataset, which exhibits severe class imbalance representative of practical industrial conditions. A comparative analysis between leakage-prone and leakage-aware evaluation settings shows that leakage-prone configurations can inflate AUC estimates by up to 8–9 percentage points, demonstrating the substantial influence of evaluation design on reported performance. Logistic Regression, Random Forest, and Gradient Boosting models are evaluated using stratified five-fold cross-validation with strictly fold-wise isolated preprocessing. While tree-based models achieved strong discriminative performance (mean AUC = 0.966 and 0.971), recall remained substantially lower than specificity, highlighting the persistent challenge of minority-class detection. The findings demonstrate that evaluation configuration, rather than model architecture alone, can significantly influence performance interpretation and lead to misleading conclusions when leakage is not controlled. This work provides a transparent and reproducible framework for reliable empirical evaluation in predictive maintenance research.

Abdulrahman M. Qahtani. “A Leakage-Aware and Reproducible Evaluation Framework for Predictive Maintenance Classification”. International Journal of Advanced Computer Science and Applications (IJACSA) 17.3 (2026). http://dx.doi.org/10.14569/IJACSA.2026.0170383

@article{Qahtani2026,
title = {A Leakage-Aware and Reproducible Evaluation Framework for Predictive Maintenance Classification},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2026.0170383},
url = {http://dx.doi.org/10.14569/IJACSA.2026.0170383},
year = {2026},
publisher = {The Science and Information Organization},
volume = {17},
number = {3},
author = {Abdulrahman M. Qahtani}
}