Urdu Word Segmentation using Machine Learning Approaches

Word Segmentation is considered a basic NLP task and in diverse NLP areas, it plays a significant role. The main areas which can be benefited from Word segmentation are IR, POS, NER, sentiment analysis, etc. Urdu Word Segmentation is a challenging task. There can be a number of reasons but Space Insertion Problem and Space Omission Problems are the major ones. Compared to Urdu, the tools and resources developed for word segmentation of English and English like other western languages have record-setting performance. Some languages provide a clear indication for words just like English which having space or capitalization of the first character in a word. But there are many languages which do not have proper delimitation in between words e.g. Thai, Lao, Urdu, etc. The objective of this research work is to present a machine learning based approach for Urdu word segmentation. We adopted the use of conditional random fields (CRF) to achieve the subject task. Some other challenges faced in Urdu text are compound words and reduplicated words. In this paper, we tried to overcome such challenges in Urdu text by machine learning methodology.

@article{Khan2018,
  title     = {Urdu Word Segmentation using Machine Learning Approaches},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {9},
  number    = {6},
  year      = {2018},
  publisher = {The Science and Information Organization},
  author    = {Sadiq Nawaz Khan and Khairullah Khan and Wahab Khan and Asfandyar Khan and Fazali Subhan and Aman Ullah Khan and Burhan Ullah},
  doi       = {10.14569/IJACSA.2018.090628},
  url       = {https://doi.org/10.14569/IJACSA.2018.090628}
}