Emotional Speech Transfer on Demand based on Contextual Information and Generative Models: A Case Study

The automated generation of speech audio that closely resembles human emotional speech has garnered signif-icant attention from the society and the engineering academia. This attention is due to its diverse applications, including au-diobooks, podcasts, and the development of empathetic home assistants. In the scope of this study, it is introduced a novel approach to emotional speech transfer utilizing generative models and a selected emotional target desired for the output speech. The natural speech has been extended with contextual information data related with emotional speech cues. The generative models used for pursuing this task are a variational autoencoder model and a conditional generative adversarial network model. In this case study, an input voice audio, a desired utterance, and user-selected emotional cues, are used to produce emotionally expressive speech audio, transferring an ordinary speech audio with added contextual cues, into a happy emotional speech audio by a variational autoencoder model. The model try to reproduce in the ordinary speech, the emotion present in the emotional contextual cues used for training. The results show that, the proposed unsupervised VAE model with custom dataset for generating emotional data reach an MSE lower than 0.010 and an SSIM almost reaching the 0.70, while most of the values are greater than 0.60, respect to the input data and the generated data. CGAN and VAE models when generating new emotional data on demand, show a certain degree of success in the evaluation of an emotion classifier that determines the similarity with real emotional audios.

@article{Porco2023,
  title     = {Emotional Speech Transfer on Demand based on Contextual Information and Generative Models: A Case Study},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {14},
  number    = {11},
  year      = {2023},
  publisher = {The Science and Information Organization},
  author    = {Andrea Veronica Porco and Kang Dongshik},
  doi       = {10.14569/IJACSA.2023.01411139},
  url       = {https://doi.org/10.14569/IJACSA.2023.01411139}
}