Abstract: In the emotion recognition of conversations, the complementary relationship between the context information and multimodal data cannot be fully exploited. This results in insufficient comprehensiveness and accuracy in emotion recognition. To address these challenges, this paper proposed a Multimodal Interactive Temporal Graph Conversation Understanding model (MITG-CU) composed of textual, audio and visual modalities. Firstly, the pre-extracted textual, audio, and visual features are adopted as the input of the Transformer, and the attention mechanism is utilized to capture the cross-modal context correlation information. Furthermore, structural relationships and temporal dependencies between utterances are captured through a local-level relational temporal graph module. Inter-modal interaction weights are dynamically adjusted by a global-level pairwise cross-modal interaction mechanism. By integrating two complementary hierarchical structures, a hierarchical multimodal information fusion was achieved, and at the same time, the model’s adapt-ability to complex conversation scenarios was enhanced. Finally, feature fusion is carried out by using the gating mechanism and sentiment classification is conducted. Experimental results demonstrated that the proposed model outperforms six common baseline methods across metrics including accuracy, precision, recall, and F1-score. Especially in Weighted-F1 and Accuracy have improved by 0.28 % and 0.39 % respectively, which confirmed the effectiveness of the model.

Qian Xing, Yaqin Qiu, Minglu Chi, Xuewei Li and Changyi Gao, “MITG-CU: Multimodal Interaction Temporal Graphs Approach for Conversational Emotion Recognition” International Journal of Advanced Computer Science and Applications(IJACSA), 16(6), 2025. http://dx.doi.org/10.14569/IJACSA.2025.0160694

@article{Xing2025,
title = {MITG-CU: Multimodal Interaction Temporal Graphs Approach for Conversational Emotion Recognition},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2025.0160694},
url = {http://dx.doi.org/10.14569/IJACSA.2025.0160694},
year = {2025},
publisher = {The Science and Information Organization},
volume = {16},
number = {6},
author = {Qian Xing and Yaqin Qiu and Minglu Chi and Xuewei Li and Changyi Gao}
}