Student-Disjoint Cross-Module Benchmarking for Calibrated Early Warning in Online Learning

Early-warning models in online learning are commonly evaluated with random data partitions, even though deployment often requires transfer to a course that supplied no training data. This study establishes a leakage-resistant benchmark for cross-module risk prediction using the Open University Learning Analytics Dataset. The analysis covered 32,593 enrollments from 28,785 learners, seven modules, 22 module presentations, and 10,655,280 daily virtual learning environment interaction records. Predictions were produced after 7, 14, and 28 days for learners still active at each cutoff. Logistic regression, random forest, and histogram gradient boosting were evaluated by leaving one entire module out and removing every learner who also appeared in the held-out module from training. Probability calibration was learned only from student-grouped out-of-fold predictions, using the same nested procedure in both validation designs. Random student-disjoint cross-validation yielded an area under the receiver operating characteristic curve (AUROC) of 0.716, whereas held-out-module evaluation yielded an AUROC of 0.661, revealing substantial contextual optimism. Under the fixed hyperparameter configurations used in this benchmark, logistic regression achieved higher cross-module performance than both nonlinear models. A virtual-learning-environment-only specification preserved discrimination while improving precision-recall performance and Brier score relative to adding assessment-submission features. Calibration nevertheless varied markedly across modules, indicating that global post-hoc calibration did not eliminate course-level distribution shift. The benchmark shows that student independence, course separation, feature availability, and probability quality must be evaluated jointly before an early-warning model is considered transferable.

@article{Lim2026,
  title     = {Student-Disjoint Cross-Module Benchmarking for Calibrated Early Warning in Online Learning},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {17},
  number    = {9},
  year      = {2026},
  publisher = {The Science and Information Organization},
  author    = {Mika Lim},
  doi       = {10.14569/IJACSA.2026.0170958},
  url       = {https://doi.org/10.14569/IJACSA.2026.0170958}
}