Deployer-Side XAI Instrumentation for Regulated AI: A Clinical Case Study in ICL Sizing

Regulated AI creates a monitoring problem for deployers who must organise human oversight, log-retention and post-market surveillance while often having access only to the prediction interface. This study specifies a deployer-side XAI instrumentation protocol for the output→action boundary, where a model output becomes a reason for action. The protocol reorganises KernelSHAP, nearest-neighbour envelope checks, bounded perturbation, and rank-order stability into a per-decision evidence record computed from predict() calls. We instantiate the protocol in a clinical case of phakic Implantable Collamer Lens sizing, using a 55-eye held-out cohort and an Extra Trees regressor for post-operative vault prediction. The record contains five signals: score_margin, constraint_enforcement, envelope_validity, decision_robustness, and record_integrity, plus two cohort-level oversight aggregates. The case study shows how the same record can support decision-time human oversight, later audit and post-market surveillance under the EU AI Act and the Medical Devices Regulation.

@article{Urrutia-Onate2026,
  title     = {Deployer-Side XAI Instrumentation for Regulated AI: A Clinical Case Study in ICL Sizing},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {17},
  number    = {6},
  year      = {2026},
  publisher = {The Science and Information Organization},
  author    = {Dorleta Urrutia-Onate and Enrique Onieva and Asier Perallos},
  doi       = {10.14569/IJACSA.2026.0170605},
  url       = {https://doi.org/10.14569/IJACSA.2026.0170605}
}