Emotion Recognition on Multimodal with Deep Learning and Ensemble

Emotion Recognition on multimodal dataset is a difficult task, which is one of the most important tasks in topics like Human Computer Interaction (HCI). This paper presents a multimodal approach for emotion recognition on dataset MELD. The dataset contains three modalities, audio, text, and facial features. In this research, only audio and text features will be experimented on. For audio data, the raw audio is converted into MFCC as an input to a bidirectional LSTM, which will be built to perform emotion classification. On the other hand, BERT will be used to tokenize the text data as an input to the text model. To classify the emotion in text data, a Bidirectional LSTM will be built. And finally, the voting ensemble method will be implemented to combine the result from two modalities. The model will be evaluated using F1-score and confusion matrix. The unimodal audio model achieved 41.69% of F1-score, while the unimodal text model achieved 47.29% of F1-score, and the voting ensemble model achieved 47.47% of F1-score. To conclude this research, this paper also discussed future works, which involved how to build and improve deep learning models and combine them with ensemble model for better performance in emotion recognition tasks in multimodal dataset.

@article{Dharma2022,
  title     = {Emotion Recognition on Multimodal with Deep Learning and Ensemble},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {13},
  number    = {12},
  year      = {2022},
  publisher = {The Science and Information Organization},
  author    = {David Adi Dharma and Amalia Zahra},
  doi       = {10.14569/IJACSA.2022.0131278},
  url       = {https://doi.org/10.14569/IJACSA.2022.0131278}
}