LITESING: TOWARDS FAST, LIGHTWEIGHT AND EXPRESSIVE SINGING VOICE SYNTHESIS

Xiaobin Zhuang, Tao Jiang, Szu-Yu Chou, Bin Wu, Peng Hu, Simon Lui
Tencent Music Entertainment(TME)

Abstract

LiteSing proposed in this paper is a high-quality singing voice synthesis (SVS) system, which is fast, lightweight and expressive. This model mainly stacks several non-autoregressive WaveNet blocks in the encoder and decoder under a generative adversarial architecture, predicts expressive conditions from the musical score, and generates acoustic features from full conditions. The full conditions used in this model consist of spectrogram energy, voiced/unvoiced (V/UV) decision and dynamic pitch curve, which are proved related to the expressiveness. We predict the pitch and the timbre features respectively, avoiding the interdependence between these two features. Instead of neural network vocoders, a parametric WORLD vocoder is employed in the end for the pitch curve consistency. Experiment results show that LiteSing outperforms the baseline model using feed-forward Transformer by 1.386 times faster on inference speed, 15 times smaller on training parameters number, and almost the same MOS on sound quality. Through an A/B test, LiteSing achieves 67.3% preference rate over baseline in expressiveness, which suggests the advantage of LiteSing over the other compared models.

arch

Results

For each singing voice synthesis system, we prepare 40 audio samples and ask 6 listeners to evaluate each sample in three aspects, such as sound quality, pronunciation accuracy, and expressiveness. The mean opinion score (MOS) is averaged over all the samples in three aspects.

arch

To compare the improvement in expressiveness, we have an A/B test for evaluating the expressiveness of LiteSing and FastSpeech2 in two aspects: pitch curve and dynamic energy. It shows that 67.3% supports LiteSing, while only 18.3% supports FastSpeech2. When we feed the full conditions extracted from reference recordings to LiteSing, experiment results show that 79.4% supports LiteSing while only 10.1% supports FastSpeech2. arch

Singing Voice Samples

* Note: All audio samples (in 32kHz sampling rate) are in Mandrin Chinese

Singer: kg_kun (male, 48 minutes, used in this paper)


Recording Reconstruction LiteSing(proposed) NPSS FastSpeech2 LiteSing-MSE