Encoder Factorization for Capturing Dialect and Articulation Level in End-to-End Speech Synthesis
| Project/Area Number |
19K24372
|
|---|
| Research Category |
Grant-in-Aid for Research Activity Start-up
|
| Allocation Type | Multi-year Fund |
|---|
| Review Section |
1002:Human informatics, applied informatics and related fields
|
|---|
| Research Institution | National Institute of Informatics |
|---|
Principal Investigator |
Cooper Erica 国立情報学研究所, コンテンツ科学研究系, 特任研究員 (30843156)
|
|---|
| Project Period (FY) |
2019-08-30 – 2021-03-31
|
| Project Status |
Granted (Fiscal Year 2019)
|
| Budget Amount *help |
¥2,860,000 (Direct Cost: ¥2,200,000、Indirect Cost: ¥660,000)
Fiscal Year 2020: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Fiscal Year 2019: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
|
|---|
| Keywords | speech synthesis / speaker modeling / deep learning / dialect modeling / articulation |
|---|
| Outline of Research at the Start |
Synthesizing speech in a variety of speaker voices and styles has long been a goal in speech research. Recent advances in speech synthesis have resulted in very natural-sounding synthetic speech. Current approaches to modeling multiple speakers in synthetic speech have resulted in high similarity to the different speakers, but fail to capture characteristics such as dialect and level of articulation. We aim to determine whether including models of dialect and level of articulation in synthetic speech systems can successfully capture these aspects of speech.
|
| Outline of Annual Research Achievements |
Synthesizing speech in a variety of speaker voices and styles has long been a goal in speech research. Recent advances in speech synthesis have resulted in very natural-sounding synthetic speech. Current approaches to modeling multiple speakers in synthetic speech have resulted in high similarity to the different speakers, but fail to capture characteristics such as dialect and level of articulation. We aim to determine whether including models of dialect and level of articulation in synthetic speech systems can successfully capture these aspects of speech.
In this past year, we implemented multi-speaker capability for a Tacotron-based TTS system using speaker embeddings obtained from a separately-trained speaker model. We conducted experiments to determine what kind of speaker embeddings work best for synthesis in the voice of a speaker who was unseen during TTS training, and we found that Learnable Dictionary Encoding (LDE) based speaker representations worked well, based on a large-scale crowdsourced listening test. We also found that there is a gap between seen and unseen speaker similarity, so we are currently exploring data augmentation approaches for TTS training, as well as dialect modeling using dialect embeddings similar to our speaker embeddings.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
According to our original research plan, we have created and evaluated TTS systems using many different types of speaker embeddings, and we have also completed experiments training dialect models and using embeddings obtained from these models in TTS. We have conducted large-scale listening tests to evaluate our synthesized speech for quality and similarity to the target speaker. We have also begun experiments to further improve speaker similarity by data augmentation, both by creating artificial training data in similar ways that have been shown to improve speech and speaker recognition, and also by including other sources of data that are not typically used for TTS (such as training data for speech recognition), and compensating for differences in recording conditions by including channel information during training.
|
| Strategy for Future Research Activity |
Since we obtained better speaker similarity in TTS by using embeddings from improved speaker models, we can next explore matching aspects of speaking style such as level of articulation. However, we still find that there is room for improvement in matching the voices of speakers who were not seen during training, so we will also continue exploring data augmentation approaches to further improve speaker similarity, and we will also continue investigating the reasons why overfitting to seen speakers occurs.
|
Report
(1 results)
Research Products
(4 results)
