Pedagogical Mediation Through Prompt Engineering: An Expert Evaluation of AI-Generated Feedback on Islamic-Integrated EFL Argumentative Writing

This research tested whether prompt engineering could act as a type of pedagogical mediation to enhance the quality of AI-generated student feedback on EFL students' argumentative essays in an Islamic education system. An initial pool of eight expert raters (four primary raters and four inter-raters) scored the AI-generated feedback from Claude Sonnet 4 using 12 systematically developed prompts to elicit feedback. Raters were asked to score feedback produced by these prompts across four areas of evaluation: pedagogy, linguistics, Islamic content, and AI reliability. The highest rated configuration was Prompt 4 (Feedback-only sequencing, English Lecturer Persona), with a mean rating of 31.00 (out of 35) in all categories. A Friedman test showed there were statistically significant differences among the four evaluative categories, χ²(3) = 30.077, p < .001. Additionally, inter-rater reliabilities were high for each of the possible pairs of raters (r = .89 -.96). Overall, this research suggests that prompt engineering is a potentially viable method of pedagogical mediation, allowing educators to develop more culturally responsive and pedagogically relevant AI-generated feedback systems for Islamic EFL higher education settings.

@article{Noviyanti2026,
  title     = {Pedagogical Mediation Through Prompt Engineering: An Expert Evaluation of AI-Generated Feedback on Islamic-Integrated EFL Argumentative Writing},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {17},
  number    = {3},
  year      = {2026},
  publisher = {The Science and Information Organization},
  author    = {Sari Dewi Noviyanti and Rudi Hartono and Hendi Pratama and Seful Bahri},
  doi       = {10.14569/IJACSA.2026.0170340},
  url       = {https://doi.org/10.14569/IJACSA.2026.0170340}
}