Token-Level PII Detection with Symbolic, Sequential, and Transformer-Based Ensemble Models

The rapid increase in unstructured digital information has led to an urgent demand for effective systems for safeguarding Personally Identifiable Information (PII) across multiple sectors and application domains. Existing single-model approaches frequently fail to resolve entity-type ambiguity in unstructured text, particularly when a token's PII status is context-dependent rather than syntactically predictable. This study presents EnsemblePII, a weighted voting ensemble that combines rule-based patterns, dictionary matching, Conditional Random Fields, Bi-LSTM sequence models, and transformer-based token classification. The ensemble applies a class-specific weighted voting strategy in which each model's per-entity influence is proportional to its per-class F1-score on a held-out validation set. The approach is assessed by using the ai4privacy/pii-masking-43k data set and the Mendeley Synthetic Financial Documents data set. EnsemblePII achieves a weighted F1 of 0.9749 on the general-purpose HF test set, marginally exceeding the strongest individual component (DistilBERT, F1 = 0.9744) on the in-distribution evaluation, and outperforming a published entity-span-level hybrid baseline by over 6 percentage points on the multi-class token-level task. On the Mendeley financial test set, Bi-LSTM and DistilBERT achieve F1 scores of 0.9959 and 0.9703, respectively, while the ensemble records 0.8433, revealing calibration sensitivity to entity classes absent from the weight calibration corpus. The results indicate that ensemble-based PII detection can improve in-distribution performance, but stable cross-domain generalization requires domain-aware calibration of ensemble weights; the DistilBERT component achieves the highest average F1 (0.9724) across both test sets, underscoring the value of transformer-based models for cross-domain PII detection.

@article{Alshamrani2026,
  title     = {Token-Level PII Detection with Symbolic, Sequential, and Transformer-Based Ensemble Models},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {17},
  number    = {6},
  year      = {2026},
  publisher = {The Science and Information Organization},
  author    = {Hessah Abdullah Alshamrani and Mona Alnahari},
  doi       = {10.14569/IJACSA.2026.0170653},
  url       = {https://doi.org/10.14569/IJACSA.2026.0170653}
}