A Human-Centered Evaluation of AI-Generated Guidance: Integrated Statistical and Machine Learning Analysis with a Risk Framework for High-Stakes Domains

Abstract: The increasing use of large language models (LLMs) in domains requiring interpretation and judgment has raised critical questions about trust, reliability, and account-ability, particularly in contexts where decisions carry significant consequences. While prior work has focused primarily on improving system performance, limited attention has been given to how users evaluate and interact with AI-generated guidance in real-world, high-stakes settings. This paper addresses this gap through a large-scale empirical investigation of public perceptions of AI-generated religious guidance in Saudi Arabia. The analysis is based on survey data collected from 572 participants and combines quantitative statistical methods with a machine learning-based pipeline for analyzing open-ended responses. The quantitative component examines patterns in trust, perceived risk, privacy concerns, credibility, and user practices, while the qualitative component employs embedding-based clustering using Bidirectional Encoder Representations from Transformers (BERT), Uniform Manifold Approximation and Projection (UMAP), and Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), followed by expert interpretation to derive structured parameters. The results indicate a cautious and conditional engagement with AI systems, characterized by moderate usage, low levels of trust, and strong concerns regarding reliability and source credibility. Users frequently verify AI-generated outputs and demonstrate a preference for human expert validation, particularly in complex or sensitive cases. Building on these insights, the study introduces a layered taxonomy of perceived risks spanning epistemic, reasoning, interactional, and institutional dimensions, providing a structured analytical framework for understanding how technical limitations translate into broader behavioural and governance challenges. These results highlight the importance of aligning AI system design with user expectations, emphasizing transparency, verifiability, and human oversight. The proposed taxonomy and analytical framework provide a foundation for future research and contribute to the development of governance approaches for AI systems deployed in high-stakes interpretive domains.

Omar Al-Turki, Felwah Alqahtani, Eman Alqahtani, Sarah Alswedani, Sami Alshmrany and Rashid Mehmood. “A Human-Centered Evaluation of AI-Generated Guidance: Integrated Statistical and Machine Learning Analysis with a Risk Framework for High-Stakes Domains”. International Journal of Advanced Computer Science and Applications (IJACSA) 17.5 (2026). http://dx.doi.org/10.14569/IJACSA.2026.0170591

@article{Al-Turki2026,
title = {A Human-Centered Evaluation of AI-Generated Guidance: Integrated Statistical and Machine Learning Analysis with a Risk Framework for High-Stakes Domains},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2026.0170591},
url = {http://dx.doi.org/10.14569/IJACSA.2026.0170591},
year = {2026},
publisher = {The Science and Information Organization},
volume = {17},
number = {5},
author = {Omar Al-Turki and Felwah Alqahtani and Eman Alqahtani and Sarah Alswedani and Sami Alshmrany and Rashid Mehmood}
}