Facebook pixel tracking

The Science and Information (SAI) Organization publishes open-access peer-reviewed journals in computer science and artificial intelligence.

Contact Info
Website thesai.org
Follow Us
Contact Info
Follow Us
Research Article | Open Access |

Beyond Conventional ASR Metrics: Evaluating Semantic Preservation in Low-Resource Speech Recognition

Author 1: Nurul Emilia Diyana Abdul Malik Author 2: Mohd Izzuddin Mohd Tamrin Author 3: Airul Azizan Zainudin Author 4: Muhamad Ariff Ibrahim Author 5: Mohd Zulfaezal Che Azemin
International Journal of Advanced Computer Science and Applications (IJACSA) · Vol. 17, No. 10 · Published 2026

DOI: 10.14569/IJACSA.2026.0171021

Abstract

Word error rate (WER) and character error rate (CER) quantify lexical discrepancies in automatic speech recognition (ASR) but do not directly indicate whether the intended message remains intact. This exploratory study compared lexical accuracy with human-rated preservation of essential meaning in Malay-English code-switched conversational speech. Four complete inference pipelines were evaluated on 30 purposively selected windows from one recording (826 reference words): Mesolitica Malaysian Whisper Large-v3-Turbo-v3 (Pipeline 1), OpenAI Whisper Large-v3-Turbo through faster-whisper (Pipeline 2), Qwen3-ASR-1.7B with a combined Malay-English contextual prompt (Pipeline 3), and the same Qwen3-ASR model without that prompt (Pipeline 4). Three evaluators independently assessed 120 reference-hypothesis pairs; a two-of-three majority constituted the semantic endpoint. Pipeline 4 had the lowest WER (29.18%) and CER (20.60%), whereas Pipeline 1 had the highest observed majority-rated semantic-preservation proportion (53.3%, versus 36.7% for Pipeline 4). The overall pipeline effect was significant for WER (Friedman chi-square (3) = 15.64, p = 0.0013) and CER (chi-square (3) = 18.17, p < 0.001), but not for semantic preservation (Cochran's Q = 5.57, p = 0.134); no paired semantic contrast remained significant after Holm adjustment. The Pipeline 1–Pipeline 4 semantic difference was 16.7 percentage points (paired-window bootstrap 95% confidence interval: −6.7 to 40.0 percentage points). Overall Fleiss' kappa was 0.529. Different descriptive rankings were therefore observed, but this single-recording sample does not establish significant differences in semantic performance or general model superiority. These findings support further evaluation of human semantic judgments alongside lexical metrics, with independent validation across recordings, speakers, and inference implementations.

Keywords

How to Cite this Article

Malik, N. E. D. A., Tamrin, M. I. M., Zainudin, A. A., Ibrahim, M. A., & Azemin, M. Z. C. (2026). Beyond Conventional ASR Metrics: Evaluating Semantic Preservation in Low-Resource Speech Recognition. International Journal of Advanced Computer Science and Applications, 17(10). https://doi.org/10.14569/IJACSA.2026.0171021

Malik, Nurul Emilia Diyana Abdul, et al.. "Beyond Conventional ASR Metrics: Evaluating Semantic Preservation in Low-Resource Speech Recognition." International Journal of Advanced Computer Science and Applications, vol. 17, no. 10, 2026, https://doi.org/10.14569/IJACSA.2026.0171021.

@article{Malik2026,
  title     = {Beyond Conventional ASR Metrics: Evaluating Semantic Preservation in Low-Resource Speech Recognition},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {17},
  number    = {10},
  year      = {2026},
  publisher = {The Science and Information Organization},
  author    = {Nurul Emilia Diyana Abdul Malik and Mohd Izzuddin Mohd Tamrin and Airul Azizan Zainudin and Muhamad Ariff Ibrahim and Mohd Zulfaezal Che Azemin},
  doi       = {10.14569/IJACSA.2026.0171021},
  url       = {https://doi.org/10.14569/IJACSA.2026.0171021}
}

Open Access — licensed under a Creative Commons Attribution 4.0 International License. Unrestricted use, distribution, and reproduction in any medium, even commercially, as long as the original work is properly cited.

Call for Papers - Important Dates

Volume 17 No. 12, December 2026
Paper Submission Due November 25, 2026
Review Notification December 15, 2026
Publication Date December 31, 2026