The Chinese University of Hong Kong, Shenzhen

Sing voice conversion (SVC) is to convert singing voice to our desired targets. There are three common SVC tasks:

Here are some examples:

Source:

Target:

Jiawei Li. Use more chest resonance for increasing your singing’s power. Bilibili, 2022.

Source:

Reference:

Target (source singer's content + reference singer's timbre):

Chao Wang, et al. Towards High-Fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding. InterSpeech 2022.

Source (singing voice):

Reference (speech):

Target (source singer's content + reference speaker's timbre):

Heyang Xue, et al. Learn2Sing 2.0: Diffusion and Mutual Information-Based Target Speaker SVS by Learning from Singing Teacher. InterSpeech 2022.

Ya Yue. Arrangements for 姐就是女王 as different singers' styles. Bilibili, 2022.

Magic of Tuner. Bilibili, 2022.

Jiawei Li. Use more chest resonance for increasing your singing’s power. Bilibili, 2022.

Peter Bence. Beat it (piano cover). Bilibili, 2022.

[1] Zhiyan Duan, et al. The NUS sung and spoken lyrics corpus: A quantitative comparison of singing and speech. APSIPA 2013.
[2] Yu Wang, et al. Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis. InterSpeech 2022.
[3] Lichao Zhang, et al. M4Singer: a Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus. NeurIPS 2022.

WORLD [1] is a classical vocoder in Digital Signal Processing (DSP). It supposes there are three acoustic components of an audio: Fundamental frequency (F0), Spectral envelope (SP), and Aperiodic parameter (AP). It can be considered not only an extractor, which can extract F0, SP, and AP efficiently, but also a synthesizer, which can synthesis an audio by incorporating the three.

Compared to the recent neural-based vocoders (such as WaveNet, WaveRNN, Hifi-GAN, Diffwave, etc.), WORLD is more like a white box and is more controllable and manipulable, while its synthesis quality is worse. We can easily adjust the input parameters to change the synthesized audios. For example, given such a male singing voice:

We can keep the F0 unchanged, and convert it to a robot-like voice:

Or, we can also increase the F0 two times, divide the SP by 1.2, and convert it to a female-like voice:

The official code of WORLD has been released here. You can play it, and manipulate the voice as you like!

Paradigm of the conversion framework

The conversion framework of the figure above is proposed by [1]. There are two main modules of it, Content Extractor and Singer-specific Synthesizer. Given a source audio, firstly the Content Extractor is aim to extract content features (i.e., singer independent features) from the audio. Then, the Singer-specific Synthesizer is designed to inject the singer dependent features for the synthesis, so that the target audio can be able to capture the singer's characteristics.

The authors of [1] assume that among the three components of WORLD (i.e., F0, AP, and SP), only SP is singer dependent and should be modeled by the Singer-specific Synthesizer, while the other two can be considered pure content features. Based on that, we can utilize the following two stages to conduct any-to-one conversion:

[1] Xin Chen, et al. Singing Voice Conversion with Non-parallel Data. IEEE MIPR 2019.

The experiment setting is many-to-one conversion. Specifically, we consider Opencpop (which is a single singer dataset) as target singer and use M4Singer (which is a 19-singer dataset) as source singers.

We adopt a python wrapper of WORLD to extract F0, AP, and SP, and to synthesis audios. We use diffsptk to transform between SP and MCEP. We utilize the last layer encoder's output of Whisper as the content features (which is 1024d). During training stage, we use a 6-layer Transformer to train the mapping from whisper features to 40d MCEP features.

Xueyao Zhang. Singing Voice Conversion Tutorial. RMSnow's Blog. Jan 2023. https://www.zhangxueyao.com/data/SVC/tutorial.html.

Or

Design: Xueyao Zhang
Last updated:       #Views: (2022/7/6~)