Pedagogical Mediation Through Prompt Engineering: An Expert Evaluation of AI-Generated Feedback on Islamic-Integrated EFL Argumentative Writing

Abstract: This research tested whether prompt engineering could act as a type of pedagogical mediation to enhance the quality of AI-generated student feedback on EFL students' argumentative essays in an Islamic education system. An initial pool of eight expert raters (four primary raters and four inter-raters) scored the AI-generated feedback from Claude Sonnet 4 using 12 systematically developed prompts to elicit feedback. Raters were asked to score feedback produced by these prompts across four areas of evaluation: pedagogy, linguistics, Islamic content, and AI reliability. The highest rated configuration was Prompt 4 (Feedback-only sequencing, English Lecturer Persona), with a mean rating of 31.00 (out of 35) in all categories. A Friedman test showed there were statistically significant differences among the four evaluative categories, χ²(3) = 30.077, p < .001. Additionally, inter-rater reliabilities were high for each of the possible pairs of raters (r = .89 -.96). Overall, this research suggests that prompt engineering is a potentially viable method of pedagogical mediation, allowing educators to develop more culturally responsive and pedagogically relevant AI-generated feedback systems for Islamic EFL higher education settings.

Sari Dewi Noviyanti, Rudi Hartono, Hendi Pratama and Seful Bahri. “Pedagogical Mediation Through Prompt Engineering: An Expert Evaluation of AI-Generated Feedback on Islamic-Integrated EFL Argumentative Writing”. International Journal of Advanced Computer Science and Applications (IJACSA) 17.3 (2026). http://dx.doi.org/10.14569/IJACSA.2026.0170340

@article{Noviyanti2026,
title = {Pedagogical Mediation Through Prompt Engineering: An Expert Evaluation of AI-Generated Feedback on Islamic-Integrated EFL Argumentative Writing},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2026.0170340},
url = {http://dx.doi.org/10.14569/IJACSA.2026.0170340},
year = {2026},
publisher = {The Science and Information Organization},
volume = {17},
number = {3},
author = {Sari Dewi Noviyanti and Rudi Hartono and Hendi Pratama and Seful Bahri}
}