Audio-Visual Multimodal Deepfake Detection Leveraging Emotional Recognition

Recently, there has been a significant reliance on the Internet. This creates a fertile environment for various risks, including fraud, privacy violations, and theft. The most common and dangerous risks at present are known as deepfakes. The development of deepfake technologies relies on advancements in artificial intelligence. Deepfake content can greatly affect privacy and security, posing a significant risk to many fields. Therefore, recent research has focused on mechanisms to detect real content from fake content. These mechanisms are classified into two main types: single-modal and multimodal detection. It is worth noting that the widespread deepfake technology has recently become more complex. This may hinder traditional single-mode detection methods in detecting video clips. In this study, we designed an effective multimodal fusion mechanism that integrates pre-trained audio, visual, and textual features. Our framework is based on three considerations: audio features, visual features, and emotion recognition. Emotion recognition focuses on three considerations: audio emotion, facial emotion, and sentiment of speech. We take advantage of the sentiment of speech to ensure there is consistency between audio and visual emotion with the meaning of words. As we achieved, the sentiment of speech makes our model more accurate and robust than when we used the audio-visual emotion inconsistency measures only. In our experiment, we used the FakeAVCeleb dataset, and we achieved 95.24% accuracy, proving our assumption of the impact of the sentiment of speech, the emotion of audio tone, and facial expressions to detect deepfakes.

@article{Alsaeedi2025,
  title     = {Audio-Visual Multimodal Deepfake Detection Leveraging Emotional Recognition},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {16},
  number    = {6},
  year      = {2025},
  publisher = {The Science and Information Organization},
  author    = {Alaa Alsaeedi and Amal AlMansour and Amani Jamal},
  doi       = {10.14569/IJACSA.2025.0160622},
  url       = {https://doi.org/10.14569/IJACSA.2025.0160622}
}