iEmoTTS: Toward Robust Cross-Speaker Emotion Transfer and Control for Speech Synthesis based on Disentanglement between Prosody and Timbre
Abstract
The capability of generating speech with specific type of emotion is desired for many applications of human-computer interaction. Cross-speaker emotion transfer is a common approach to generating emotional speech when speech with emotion labels from target speakers is not available for model training. This paper presents a novel cross-speaker emotion transfer system, named iEmoTTS. The system is composed of an emotion encoder, a prosody predictor, and a timbre encoder. The emotion encoder extracts the identity of emotion type as well as the respective emotion intensity from the mel-spectrogram of input speech. The emotion intensity is measured by the posterior probability that the input utterance carries that emotion. The prosody predictor is used to provide prosodic features for emotion transfer. The timber encoder provides timbre-related information for the system. Unlike many other studies which focus on disentangling speaker and style factors of speech, the iEmoTTS is designed to achieve cross-speaker emotion transfer via disentanglement between prosody and timbre. Prosody is considered as the main carrier of emotion-related speech characteristics and timbre accounts for the essential characteristics for speaker identification. Zero-shot emotion transfer, meaning that speech of target speakers are not seen in model training, is also realized with iEmoTTS. Extensive experiments of subjective evaluation have been carried out. The results demonstrate the effectiveness of iEmoTTS as compared with other recently proposed systems of cross-speaker emotion transfer. It is shown that iEmoTTS can produce speech with designated emotion type and controllable emotion intensity. With appropriate information bottleneck capacity, iEmoTTS is able to effectively transfer emotion information to a new speaker. Audio samples are publicly available\footnote{https://patrick-g-zhang.github.io/iemotts/}.
Audio Samples
1. Some real recordings are presented in this part.
2. The utterance of "sad" emotion is from corpus Multi-S60-E3, while the utterances of other emotions are from corpus Child-S1-E6.
3. One target speaker from corpus VA-S2.
4. One target speaker from corpus Read-S40.
Audio Samples of Different Emotion Types
Amazement
Anger
Disgust
Fear
Happiness
Poorness
Sadness
Audio Samples of one Target Speakers from VA-S2
Audio Samples of one Target Speaker from Read-S40
Performance Evaluation
These examples are sampled from the subjective evaluation set for Table II in the paper.
Emotion
iEmoTTS
Trans-CLN
Trans-Pros
Amazement
Effectiveness of Disentanglement
These examples are sampled from the subjective evaluation set for Table III in the paper.
Emotion
iEmoTTS
iEmoTTS-NP
iEmoTTS-SE
iEmoTTS-SENP
Amazement
Emotion Intensity Control
These examples are sampled from the subjective evaluation set for Table IV in the paper.
Source Speaker
Emotion
System
Low
Moderate
High
Target Speaker
Emotion
System
Low
Moderate
High
Zero-shot Cross-speaker Emotion Transfer
These examples are sampled from the subjective evaluation set for Table V in the paper.
Emotion
IB-full
IB-large
IB-default
IB-small
Effectiveness of Emotion Encoder
These examples are sampled from the subjective evaluation set for Table VI in the paper.
Emotion
iEmoTTS
iEMOTTS-SER
iEMOTTS-WoEI
Effectiveness of Semi-Supervised Strategy
These examples are sampled from the subjective evaluation set for Table VII in the paper.