A Deep Learning Approach for Nepali Image Captioning and Speech Generation

This article introduces a novel approach for Image-to-Speech generation that aims in converting images into textual captions along with spoken descriptions in Nepali Language using deep learning techniques. By leveraging computer vision and natural language processing, the system analyzes images, extracts features, generates human-readable captions, and produces intelligible speech output. The experimentation utilizes state-of-the-art transformer architecture for image caption generation complemented by ResNet and EfficientNet as feature extractors. BLEU score is used as an evaluation metric for generated captions. The BLEU scores obtained for BLEU-1, BLEU-2, BLEU-3, and BLEU-4 n-grams are 0.4852, 0.2952, 0.181, and 0.113, respectively. Pretrained HifiGaN(vocoder) and Tacotorn2 are used for text to speech synthesis. The proposed approach contributes to the underexplored domain of Nepali-language AI applications, aiming to improve accessibility and technological inclusivity for the Nepali-speaking population.

@article{Sharma2025,
  title     = {A Deep Learning Approach for Nepali Image Captioning and Speech Generation},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {16},
  number    = {2},
  year      = {2025},
  publisher = {The Science and Information Organization},
  author    = {Sagar Sharma and Samikshya Chapagain and Sachin Acharya and Sanjeeb Prasad Panday},
  doi       = {10.14569/IJACSA.2025.01602124},
  url       = {https://doi.org/10.14569/IJACSA.2025.01602124}
}