Enhancing Emotion Prediction in Multimedia Content Through Multi-Task Learning

This study presents a robust multimodal emotion analysis model aimed at improving emotion prediction in film and television communication. Addressing challenges in modal fusion and data association, the model integrates a Transformer-based framework with multi-task learning to capture emotional associations and temporal features across various modalities. It overcomes the limitations of single-modal labels by incorporating multi-task learning, and is tested on the Cmumosi dataset using both classification and regression tasks. The model achieves strong performance, with an average absolute error of 0.70, a Pearson correlation coefficient of 0.82, and an accuracy of 47.1% in a seven-class task. In a two-class task, it achieves an accuracy and F1 score of 88.4%. Predictions for specific video segments are highly consistent with actual labels, with predicted scores of 2.15 and 1.4. This research offers a new approach to multimodal emotion analysis, providing valuable insights for film and television content creation and setting the foundation for further advancements in this area.

@article{Fan2025,
  title     = {Enhancing Emotion Prediction in Multimedia Content Through Multi-Task Learning},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {16},
  number    = {2},
  year      = {2025},
  publisher = {The Science and Information Organization},
  author    = {Wan Fan},
  doi       = {10.14569/IJACSA.2025.01602118},
  url       = {https://doi.org/10.14569/IJACSA.2025.01602118}
}