An Empirical Analysis of BERT Embedding for Automated Essay Scoring

Automated Essay Scoring (AES) is one of the most challenging problems in Natural Language Processing (NLP). The significant challenges include the length of the essay, the presence of spelling mistakes affecting the quality of the essay and representing essay in terms of relevant features for the efficient scoring of essays. In this work, we present a comparative empirical analysis of Automatic Essay Scoring (AES) models based on combinations of various feature sets. We use 30-manually extracted features, 300-word2vec representation, and 768-word embedding features using BERT model and forms different combinations for evaluating the performance of AES models. We formulate an automated essay scoring problem as a rescaled regression problem and quantized classification problem. We analyzed the performance of AES models for different combinations. We compared them against the existing ensemble approaches in terms of Kappa Statistics and Accuracy for rescaled regression problem and quantized classification problem respectively. A combination of 30-manually extracted features, 300-word2vec representation, and 768-word embedding features using BERT model results up to 77.2 ± 1.7 of Kappa statistics for rescaled regression problem and 75.2 ± 1.0 of accuracy value for Quantized Classification problem using a benchmark dataset consisting of about 12,000 essays divided into eight groups. The reporting results provide directions to the researchers in the field to use manually extracted features along with deep encoded features for developing a more reliable AES model.

@article{Beseiso2020,
  title     = {An Empirical Analysis of BERT Embedding for Automated Essay Scoring},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {11},
  number    = {10},
  year      = {2020},
  publisher = {The Science and Information Organization},
  author    = {Majdi Beseiso and Saleh Alzahrani},
  doi       = {10.14569/IJACSA.2020.0111027},
  url       = {https://doi.org/10.14569/IJACSA.2020.0111027}
}