SpanBERT-based Multilayer Fusion Model for Extractive Reading Comprehension

Extractive reading comprehension is a prominent research topic in machine reading comprehension, which aims to predict the correct answer from the given context. Pre-trained models have recently shown considerable effectiveness in this area. However, during the training process, most existing models face the problem of semantic information loss. To address this problem, this paper proposes a model based on the SpanBERT pre-trained model to predict answers using a multi-layer fusion method. Both the outputs of the intermediate layer and the prediction layer of the transformer are fused to perform answer prediction, thereby improving the model's performance. The proposed model achieves F1 scores of 92.54%, 84.02%, 80.86%, 71.32%, and EM scores of 86.27%, 81.25%, 69.10%, 56.42% on the SQuAD1.1, SQuAD2.0, Natural Questions and NewsQA datasets, respectively. Experimental results show that our model outperforms a number of existing models and has excellent performance.

@article{Zhang2024,
  title     = {SpanBERT-based Multilayer Fusion Model for Extractive Reading Comprehension},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {15},
  number    = {1},
  year      = {2024},
  publisher = {The Science and Information Organization},
  author    = {Pu Zhang and Lei He and Deng Xi},
  doi       = {10.14569/IJACSA.2024.0150149},
  url       = {https://doi.org/10.14569/IJACSA.2024.0150149}
}