FEM-KP: A Functional Evaluation Metric for Keyphrase Prediction Models

Abstract: Keyphrase prediction models are among the natural language processing (NLP) tasks that have improved their performance with transformers and large language models (LLMs). Instead of extracting present keyphrases in the text, these models also generate absent keyphrases. This improvement has led to significant challenges in the evaluation process of these models, which rely on metrics that compare the predicted keyphrases with the reference keyphrases. To measure the performance of these models, several evaluation metrics such as F1-score, ROUGE-L, and BertScore have been used. However, they often prioritize lexical similarity over semantic usefulness. Consequently, the functional usefulness of keyphrases in document representation is not evaluated during the evaluation process, which leads to inconsistencies in the evaluation results. Therefore, in this paper we propose a functional evaluation metric for Keyphrase prediction models (FEM-KP), a new evaluation metric that uses a two-track approach where track (A) evaluates the performance of the model to generate keyphrases capable of constructing a document summary, while track (B) measures the ability of these phrases to retrieve the document. We evaluated the performance of four keyphrase prediction models using current evaluation metrics and FEM-KP across the Inspec, KP20k, and Krapivin datasets. The experimental results showed that FEM-KP is the only evaluation system that maintained a consistent performance ranking regardless of document length or dataset complexity. In contrast, other metrics showed inversions in ranking. These results confirm that FEM-KP is a robust, reliable, and domain-independent evaluation metric for evaluating the performance of keyphrase prediction systems.

Lahbib Ajallouda and Ahmed Zellou. “FEM-KP: A Functional Evaluation Metric for Keyphrase Prediction Models”. International Journal of Advanced Computer Science and Applications (IJACSA) 17.4 (2026). http://dx.doi.org/10.14569/IJACSA.2026.0170426

@article{Ajallouda2026,
title = {FEM-KP: A Functional Evaluation Metric for Keyphrase Prediction Models},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2026.0170426},
url = {http://dx.doi.org/10.14569/IJACSA.2026.0170426},
year = {2026},
publisher = {The Science and Information Organization},
volume = {17},
number = {4},
author = {Lahbib Ajallouda and Ahmed Zellou}
}