Cross-Domain Evaluation of Large Language Models for Abstractive Text Summarization: An Empirical Perspective

Abstract: Large Language Models (LLMs) have demon-strated remarkable capabilities in generating human-like text; however, their effectiveness in abstractive summarization across diverse domains remains underexplored. This study conducts a comprehensive evaluation of six open source LLMs across four datasets: CNN / Daily Mail and NewsRoom (news), SAMSum (dialogue) and ArXiv (scientific) using zero shot and in-context learning techniques. Performance was assessed using ROUGE and BERTScore metrics, and inference time was measured to examine the trade-off between accuracy and efficiency. For long documents, a sentence-based chunking strategy is introduced to overcome context limitations. Results reveal that in-context learning consistently enhances summarization quality, and chunking improves performance on long scientific texts. The model performance varies according to architecture, scale, prompt design, and dataset characteristics. The qualitative analysis further demonstrates that the top-performing models produce summaries that are coherent, informative, and contextually aligned with human-written references, despite occasional lexical divergence or factual omissions. These findings provide practical insights into designing instruction-based summarization systems using open-source LLMs.

Walid Mohamed Aly, Taysir Hassan A. Soliman and Amr Mohamed AbdelAziz. “Cross-Domain Evaluation of Large Language Models for Abstractive Text Summarization: An Empirical Perspective”. International Journal of Advanced Computer Science and Applications (IJACSA) 16.6 (2025). http://dx.doi.org/10.14569/IJACSA.2025.0160695

@article{Aly2025,
title = {Cross-Domain Evaluation of Large Language Models for Abstractive Text Summarization: An Empirical Perspective},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2025.0160695},
url = {http://dx.doi.org/10.14569/IJACSA.2025.0160695},
year = {2025},
publisher = {The Science and Information Organization},
volume = {16},
number = {6},
author = {Walid Mohamed Aly and Taysir Hassan A. Soliman and Amr Mohamed AbdelAziz}
}