Multimodal Cognitive Mapping Framework for Context-Aware Figurative Language Understanding

Learning figurative language, including idioms, metaphors, and similes, remains challenging due to subtle cultural, contextual, and multimodal cues that cannot be inferred from literal meanings alone. Traditional unimodal and text-only approaches, such as CLS-BERT, LaBSE, and mUSE, often fail to capture these deeper semantic patterns, resulting in reduced accuracy and limited cultural generalization. This study introduces a context-aware multimodal learning framework that integrates textual embeddings from a Graph-Enhanced Transformer (HCGT) with visual embeddings from CLIP, fused through a graph-based cross-modal attention mechanism, and refined using a cognitive mapping layer. This architecture models human-like semantic reasoning by aligning literal and figurative senses across modalities while maintaining conceptual structure through graph-driven representation learning. Experiments conducted on idiom, metaphor, simile, and multimodal meme datasets include preprocessing steps such as text cleaning, tokenization, image normalization, and label standardization. The framework achieves an accuracy of 90%, surpassing state-of-the-art text-only transformer baselines by 3–4%. Explainable AI tools, including attention heatmaps and SHAP values, validate the interpretability of the model by highlighting influential textual tokens and visual regions. The results confirm that integrating multimodal embeddings with cognitive mapping substantially enhances performance, interpretability, and cultural sensitivity in figurative language understanding.

@article{Gudipati2025,
  title     = {Multimodal Cognitive Mapping Framework for Context-Aware Figurative Language Understanding},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {16},
  number    = {11},
  year      = {2025},
  publisher = {The Science and Information Organization},
  author    = {R. Swathi Gudipati and Neena PC and K. Ezhilmathi and M. Durairaj and S. Farhad and Elangovan Muniyandy and Padmashree V},
  doi       = {10.14569/IJACSA.2025.0161164},
  url       = {https://doi.org/10.14569/IJACSA.2025.0161164}
}