TGMoE: A Text Guided Mixture-of-Experts Model for Multimodal Sentiment Analysis

Multimodal sentiment analysis seeks to determine the sentiment polarity of targets by integrating diverse data types, including text, visual, and audio modalities. However, during the process of multimodal data fusion, existing methods often fail to adequately analyze the sentimental relationships between different modalities and overlook the varying contributions of different modalities to sentiment analysis results. To address this issue, we propose a Text Guided Mixture-of-Experts (TGMoE) Model for Multimodal Sentiment Analysis. Based on the varying contributions of different modalities to sentiment analysis, this model introduces a text guided cross-modal attention mechanism that fuses text separately with visual and audio modalities, leveraging attention to capture interactions between these modalities and effectively enrich the text modality with supplementary information from the visual and audio data. Additionally, by employing a sparsely gated mixture of expert layers, the TGMoE model constructs multiple expert networks to simultaneously learn sentiment information, enhancing the nonlinear representation capability of multimodal features. This approach makes multimodal features more distinguishable concerning sentiment, thereby improving the accuracy of sentiment polarity judgments. The experimental results on the publicly available multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI show that the TGMoE model outperforms most existing multimodal sentiment analysis models and can effectively improve the performance of sentiment analysis.

@article{Zhao2024,
  title     = {TGMoE: A Text Guided Mixture-of-Experts Model for Multimodal Sentiment Analysis},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {15},
  number    = {8},
  year      = {2024},
  publisher = {The Science and Information Organization},
  author    = {Xueliang Zhao and Mingyang Wang and Yingchun Tan and Xianjie Wang},
  doi       = {10.14569/IJACSA.2024.01508119},
  url       = {https://doi.org/10.14569/IJACSA.2024.01508119}
}