Statistical Language Model-based Analysis of English Corpora and Literature

Despite widespread use of statistical language models in language processing, their ability to process natural languages is not advanced and they struggle to effectively capture linguistic information. Furthermore, there is a lack of automatic processing models in the field of natural language processing. In order to address these issues, and Improve the processing ability of statistical language models for English language a statistical language model optimization algorithm has been proposed. This algorithm is based on an improved resorting algorithm and is specifically applied to process English literary texts. Experimental results indicate that the proposed algorithm outperforms the N-gram algorithm in a majority of texts, with a maximum accuracy improvement of 14.5%. Additionally, in terms of the grammar analysis model, there is a high level of consistency between the model's scoring and the expert manpower scoring, as reflected by a correlation coefficient of 0.7893. This high level of consistency between the grammar analysis model and expert analysis results holds significant importance for the advancement of natural language processing.

@article{Chai2023,
  title     = {Statistical Language Model-based Analysis of English Corpora and Literature},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {14},
  number    = {9},
  year      = {2023},
  publisher = {The Science and Information Organization},
  author    = {Wenwen Chai},
  doi       = {10.14569/IJACSA.2023.0140995},
  url       = {https://doi.org/10.14569/IJACSA.2023.0140995}
}