Speech privacy protection by high-quality, invertible, and extendable speech anonymization and de-anonymization
| Project/Area Number |
21K17775
|
|---|
| Research Category |
Grant-in-Aid for Early-Career Scientists
|
| Allocation Type | Multi-year Fund |
|---|
| Review Section |
Basic Section 61010:Perceptual information processing-related
|
|---|
| Research Institution | National Institute of Informatics |
|---|
Principal Investigator |
Wang Xin 国立情報学研究所, コンテンツ科学研究系, 特任助教 (60843141)
|
|---|
| Project Period (FY) |
2021-04-01 – 2024-03-31
|
| Project Status |
Granted (Fiscal Year 2022)
|
| Budget Amount *help |
¥4,550,000 (Direct Cost: ¥3,500,000、Indirect Cost: ¥1,050,000)
Fiscal Year 2023: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000)
Fiscal Year 2022: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000)
Fiscal Year 2021: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
|
|---|
| Keywords | speech privacy / speaker anonymization / speech waveform modeling / neural network / deep learning |
|---|
| Outline of Research at the Start |
Human speech contains not only verbal contents but also private information about the speaker such as the speaker identity. This proposal is on protecting the speaker’s privacy in speech data for two scenarios: 1) Speech anonymization: when the speaker shares the speech data in untrusted cyberspace, this speech data should be protected so that the audience can understand the speech but cannot infer who the speaker is; 2) Speech de-anonymization: when the speaker further shares the speech data to trusted audience, the original natural speech can be reconstructed from protected version.
|
| Outline of Annual Research Achievements |
The second year's work consists of three parts: Part 1) Based on the previous year's work, the second VoicePrivacy challenge was organized by us and other universities. We defined new evaluation frameworks and conducted solid evaluations. In addition to many findings, we found that the new baseline, which was the research outcome of the previous year, outperformed the legacy baseline. We also saw submissions that outperformed the new baseline, which indicates the advancement of the research field brought by the VoicePrivacy challenge.
Part 2) Based on the framework of the voice privacy challenge, we did a deep analysis of the common approaches to generating anonymized speaker identity representation (i.e., pseudo speaker embedding). Through a large-scale experiment, we identified good strategies to choose and assign the pseudo-speaker, including random gender selection and utterance-level anonymization. We also found that a simple percentile-based pitch conversion reduced the risk against the strongest (Semi-Informed) attacker. These findings were published in a top IEEE journal.
Part 3) We followed the research plan and extended the language-independent speaker anonymization framework. Although the framework is language-independent, its performance degrades when processing unseen languages. We found that using multilingual training data for the waveform generator was helpful. We also proposed a correlation-alignment-based strategy to alleviate channel mismatch. Additionally, we extended the framework to hide gender information. Both works were published in top conferences.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
The efforts of the VoicePrivacy Challenge 2022 produced good outcomes. The challenge attracted 43 registered teams from 17 countries, which led to 16 successful submissions. We also organized a special session in the Interspeech 2022 satellite workshop and had presentations from participants and ourselves. The results are released on VoicePrivacy Challenge's official website: https://www.voiceprivacychallenge.org/results-2022/.
The experimental study analyzing the shortcomings and optimal strategy for speaker anonymization under (Part 2 of the research outcome) was published in a top IEEE journal.
We followed the research plan and investigated the language-independent speaker anonymization framework (Part 3 of the research outcome), and the work was accepted by the Interspeech 2022 conference (CORE rank A) and ICASSP 2023 conference (CORE rank B).
|
| Strategy for Future Research Activity |
Following the research plan made in the previous year, we will work on the language-independent speaker anonymization framework. Although it performs well in different languages (research outcome of 1st year) and other speaker attributes (Part 3 of the research outcome), there are issues left: 1) The quality of the anonymized voice is still inferior to the natural voice. Findings from the research outcome (Part 2) indicate that the selection-based generate pseudo speaker embedding is one bottleneck. We plan to investigate generative approaches for better performance. 2) The optimization of the speaker anonymization framework lacks a solid mathematical description. We plan to derive a unified mathematical description to consider multiple goals of the optimization and improve the current framework accordingly.
The final year research plan also includes work on the VoicePrivacy Challenge series: 1) post-challenge analysis on VoicePrivacy Challenge 2022 and how the progress of the research field has been made since the previous challenge. 2) whether stronger attacker models can recognize the speaker identity in the anonymized speech waveforms.
|
Report
(2 results)
Research Products
(20 results)
-
-
-
-
-
[Journal Article] The VoicePrivacy 2020 Challenge: Results and findings2022
Author(s)
Natalia Tomashenko, Xin Wang, Emmanuel Vincent, Jose Patino, Brij Mohan Lal Srivastava, Paul-Gauthier No?, Andreas Nautsch, Nicholas Evans, Junichi Yamagishi, Benjamin O’Brien, Ana?s Chanclu, Jean-Fran?ois Bonastre, Massimiliano Todisco, Mohamed Maouche
-
Journal Title
Computer Speech & Language
Volume: 74
Pages: 101362-101362
DOI
Related Report
Peer Reviewed / Open Access / Int'l Joint Research
-
-
-
-
-
-
-
-
-
[Presentation] Benchmarking and challenges in security and privacy for voice biometrics2021
Author(s)
Jean-Francois Bonastre, Hector Delgado, Nicholas Evans, Tomi Kinnunen, Kong Aik Lee, Xuechen Liu, Andreas Nautsch, Paul-Gauthier NoE, Jose Patino, Md Sahidullah, Brij Mohan Lal Srivastava, Massimiliano Todisco, Natalia Tomashenko, Emmanuel Vincent, Xin Wang, Junichi Yamagishi
Organizer
2021 ISCA Symposium on Security and Privacy in Speech Communication
Related Report
Int'l Joint Research
-
-
-
-
-
-
