Cognitively Aligned Assessment Item Generation with Open-Source LLMs: A Comprehensive Evaluation on LearningQ

Abstract: Automated generation of high-quality educational assessment items is still difficult, especially when it comes to higher-order cognitive skills. Although Large Language Models (LLMs) show promise, their structural validity and cognitive alignment are limited. This study systematically evaluates fine-tuning open-source LLMs using an enriched LearningQ dataset that includes Bloom’s cognitive labels and evidence. The results show a clear performance contrast. Qwen2.5-3B-Instruct displays the best semantic reasoning, while Llama-3.2-3B shows better structural adherence, achieving a 94.9% validity rate and full compliance against answer leakage while maintaining high question validity. In contrast, older encoder-decoder models like FLAN-T5-XL do not generate valid questions. The study finds that small- to medium-sized instruction-tuned models, backed by strong data engineering, are successful at developing scalable, cognitively well-aligned assessment items.

Mahmoud Badry, Walaa Medhat, Shereen A. Taie and Asmaa Hashem Sweidan. “Cognitively Aligned Assessment Item Generation with Open-Source LLMs: A Comprehensive Evaluation on LearningQ”. International Journal of Advanced Computer Science and Applications (IJACSA) 17.5 (2026). http://dx.doi.org/10.14569/IJACSA.2026.0170589

@article{Badry2026,
title = {Cognitively Aligned Assessment Item Generation with Open-Source LLMs: A Comprehensive Evaluation on LearningQ},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2026.0170589},
url = {http://dx.doi.org/10.14569/IJACSA.2026.0170589},
year = {2026},
publisher = {The Science and Information Organization},
volume = {17},
number = {5},
author = {Mahmoud Badry and Walaa Medhat and Shereen A. Taie and Asmaa Hashem Sweidan}
}