A Multimodal Data Scraping Tool for Collecting Authentic Islamic Text Datasets

Making decisions based on accurate knowledge is agreed upon to provide ample opportunities in different walks of life. Machine learning and natural language processing (NLP) systems, such as Large Language Models, may use unrecognized sources of Islamic content to fuel their predictive models, which could often lead to incorrect judgments and rulings. This article presents the development of an automated method with four distinct algorithms for text extraction from static websites, dynamic websites, YouTube videos with transcripts, and for speech-to-text conversion from videos without transcripts, particularly targeting Islamic knowledge text. The tool is tested by collecting a reliable Islamic knowledge dataset from authentic sources in Saudi Arabia. We scraped Islamic content in Arabic from text websites of prominent scholars and YouTube channels administered by five authorized agencies in Saudi Arabia. These agencies include the general authority for the affairs of the grand mosque and the prophet’s mosque and charitable foundations in Saudi Arabia. For websites, text data were scraped using Python tools for static and dynamic web scraping such as Beautiful Soup and Selenium. For YouTube channels, data were scraped from existing transcripts or transcribed using automatic speech recognition tools. The final Islamic content dataset comprises 31225 records from regulated sources. Our Islamic knowledge dataset can be used to develop accurate Islamic question answering, AI chatbots and other NLP systems.

@article{Namoun2024,
  title     = {A Multimodal Data Scraping Tool for Collecting Authentic Islamic Text Datasets},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {15},
  number    = {12},
  year      = {2024},
  publisher = {The Science and Information Organization},
  author    = {Abdallah Namoun and Mohammad Ali Humayun and Waqas Nawaz},
  doi       = {10.14569/IJACSA.2024.0151224},
  url       = {https://doi.org/10.14569/IJACSA.2024.0151224}
}