Speech synthesis with communicative prosody driven by the impressions of output lexicons

• Project/Area Number: 18300063
• Research Category: Grant-in-Aid for Scientific Research (B)
• Allocation Type: Single-year Grants
• Section: 一般
• Research Field: Perception information processing/Intelligent robotics
• Research Institution: Waseda University
• Principal Investigator: SAGISAKA Yoshinori Waseda University, 理工学術院, 教授 (70339737)
• Co-Investigator(Kenkyū-buntansha): 小林 哲則 早稲田大学, 理工学部, 教授 (30162001) ; 誉田 雅彰 (誉田 雅章) 早稲田大学, スポーツ科学学術院, 教授 (90367095)
• Co-Investigator(Renkei-kenkyūsha): KOBAYASHI Tetsunori 早稲田大学, 理工学術院, 教授 (30162001) ; NONDA Masaaki 早稲田大学, スポーツ科学学術院, 教授 (90367095)
• Project Period (FY): 2006 – 2009
• Project Status: Completed (Fiscal Year 2009)
• Budget Amount: ¥18,280,000 (Direct Cost: ¥14,800,000、Indirect Cost: ¥3,480,000); Fiscal Year 2009: ¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000); Fiscal Year 2008: ¥5,330,000 (Direct Cost: ¥4,100,000、Indirect Cost: ¥1,230,000); Fiscal Year 2007: ¥5,330,000 (Direct Cost: ¥4,100,000、Indirect Cost: ¥1,230,000); Fiscal Year 2006: ¥3,200,000 (Direct Cost: ¥3,200,000)
• Keywords: 音声情報処理 / 音声合成 / 韻律制御 / 対話音声 / イントネーション / 基本周波数 / パラ言語情報

Research Abstract

A scheme for communicative prosody generation was proposed to synthesize speech needed for conversational purposes. Using the correlation between communicative prosody and impression attributes of lexicons constituting output, the proposed scheme enables prosody control for conversational speech output. Perceptual experiments showed the superiority of the speech synthesized with the proposed communicative prosody to the conventional one with reading style prosody. Further application to Chinese and English speech synthesis and the reverse technology of impression extraction from speech clarified the usefulness of the proposed approach.

Research Products

• [Journal Article] Analysis on paralinguistic prosody control in perceptual impression space using multiple dimensional scaling (2009)
  • Author(s): Y. Greenberg, N. Shibuya, M. Tsuzaki, H. Kato, Y. Sagisaka
  • Journal Title: Speech Communication Vol.51No.7 Pages: 585-593
  • Peer Reviewed
• [Journal Article] Analysis on paralinguistic prosody control in perceptual impression space using multiple dimensional scaling (2009)
  • Author(s): Y.Greenberg, N.Shibuya, M.Tsuzaki, H.Kato, Y.Sagisaka
  • Journal Title: Speech Communication Vol.51 No.7 Pages: 585-593
  • Peer Reviewed
• [Journal Article] Towards Computing Phonetics (2008)
  • Author(s): Yoshinori Sagisaka
  • Journal Title: 中国語音学報 Vol.1 Pages: 23-37
  • Peer Reviewed
• [Journal Article] Towards Computing Phonetics (2008)
  • Author(s): Y. SAGISAKA
  • Journal Title: 中国語音学報 Vol.1 Pages: 23-37
  • Peer Reviewed
• [Journal Article] Towards Computing Phonetics (2008)
  • Author(s): Yoshinori SAGISAKA
  • Journal Title: 中国語音学報(Zhogguo Yuyin Xuebao) 第1巻 Pages: 23-37
  • Peer Reviewed
• [Journal Article] A trial of communicative prosody generation based on control characteristic of one word utterance observed in real conversational speech (2006)
  • Author(s): Y.Greenberg, N.Shibuya, M.Tsuzaki, H.Kato, Y.Sagisaka
  • Journal Title: Proc. Speech prosody 2006 Pages: 37-40
• [Journal Article] Towards Computing Phonetics (2006)
  • Author(s): Y.Nagasaki
  • Journal Title: Proc. The 7th Phonetic Conference of China and International forum on Phonetic Frontiers
• [Journal Article] 音声学研究の輝かしい展開を求めて-数理モデルからの提案- (2006)
  • Author(s): 勾坂芳典
  • Journal Title: 日本音声学会創立80周年記念フォーラム発表論文集
• [Journal Article] 印象表現によるパラ言語情報を用いた韻律制御 (2006)
  • Author(s): 李 克, グリーンバーグ陽子, 渋谷 渚, 勾坂芳典
  • Journal Title: 日本音響学会2006年秋季研究発表会論文集 Pages: 233-234
• [Presentation] Computing prosody variations (2009)
  • Author(s): Y.Sagisaka, et al.
  • Organizer: International Workshop on Spoken Language Prosody
  • Place of Presentation: インドコルカタ(カルカッタ)
  • Year and Date: 2009-11-26
• [Presentation] Communicative prosody generation using language common features provided by input lexicons (2009)
  • Author(s): Y.Greenberg, M.Tsuzaki, H.Kato, Y.Sagisaka
  • Organizer: Synposium on Natural Language Pricessing
  • Place of Presentation: タイバンコク
  • Year and Date: 2009-10-21
• [Presentation] Studies on speech timing control and its application to the 2nd language learning (2009)
  • Author(s): Yoshinori Sagisaka, Hiroaki Kato, Minoru Tsuzaki, Shizuka Nakamura
  • Organizer: International Conference on Computer Applications
  • Place of Presentation: Yangon Myanmar
  • Year and Date: 2009-02-27
• [Presentation] Computing prosody variations (2009)
  • Author(s): Y. Sagisaka, H. Kato, M. Tsuzaki, Y. Greenberg S. Nakamura, C. Hansakunbuntheung
  • Organizer: Proc. IWSLPR (CDROM)
  • Place of Presentation: Kalkota(India)
• [Presentation] Communicative prosody generation using language common features provided by input lexicons (2009)
  • Author(s): Y. Greenberg, M. Tsuzaki, H. Kato, Y. Sagisaka
  • Organizer: Proc. SNLP
  • Place of Presentation: Bangkok(Thailand)
• [Presentation] Corpus-based speechsyn thesis from reading speech to communicative speech (2008)
  • Author(s): Y. Sagisaka
  • Organizer: ISCA Workshop onSpoken Language Technologies forUnder-resourced languages
  • Place of Presentation: Hanoi
  • Year and Date: 2008-06-05
• [Presentation] Corpus-based speech synthesis from reading speech to communicative speech (2008)
  • Author(s): Yoshinori Sagisaka
  • Organizer: ISCA Workshop on Spoken Language Technologies for Unde-resouroed languages
  • Place of Presentation: Hanoi Vietnum
  • Year and Date: 2008-05-06
• [Presentation] Communicative prosody processing for synthesis and recognition of para-linguistic information (2008)
  • Author(s): Y. Sagisaka, Y. Greenberg, K. Li, M. Zhu, M. Tsuzaki and H. Kato
  • Organizer: ICCA 2008
  • Place of Presentation: Yangon
  • Year and Date: 2008-02-14
• [Presentation] Corpus-based speech synthesis from reading speech to communicative speech (2008)
  • Author(s): Y. Sagisaka
  • Organizer: The first International Workshop on Spoken Languages Technologies for Under-resourced languages
  • Place of Presentation: Hanoi(Vietnam)
• [Presentation] Communicative prosody processing for synthesis and recognition of para-linguistic information (2008)
  • Author(s): Y. Sagisaka, Y. Greenberg, K. Li, M. Zhu, M. Tsuzaki and H. Kato
  • Organizer: International Conference on Computer Applications
  • Place of Presentation: Yangon
• [Presentation] Automatic extraction of paralinguistic information from communicative speech (2007)
  • Author(s): M. Zhu, K. Li, Y. Greenberg, Y. Sagisaka
  • Organizer: Proc. the 7th Symposium on Natural Language Processing 2007
  • Place of Presentation: Pattaya(Thailand)
• [Presentation] Inter-language prosodic style modification experiment using word impression vector for communicative speech generation (2007)
  • Author(s): K. Li, Y. Greenberg, Y. Sagisaka
  • Organizer: Proc. Interspeech 2007
  • Place of Presentation: Vietri sul Male(Italy)
• [Presentation] Prosody Generation for Communicative Speech Synthesis (2007)
  • Author(s): Y. Sagisaka
  • Organizer: Proc. Taiwan-Japan Joint Workshop on Speech Science and Technologies
  • Place of Presentation: 台北(台湾)
• [Presentation] 自然発話の韻律情報に基づく聴覚印象の自動抽出 (2007)
  • Author(s): 朱明朝, 李克, グリーンバーグ陽子, 匂坂芳典
  • Organizer: 日本音響学会2007年秋季研究発表会講演
  • Place of Presentation: 山梨
• [Presentation] 印象表現ベクトルに基づく言語間韻律変換 (2007)
  • Author(s): 李克, グリーンバーグ陽子, 匂坂芳典
  • Organizer: 日本音響学会2007年秋季研究発表会講演
  • Place of Presentation: 山梨
• [Presentation] Automatic extraction of paralinguistic information from communicative speech (2007)
  • Author(s): Mingzhao Zhu, Ke Li, Y. Greenberg and Y. Sagisaka
  • Organizer: Proc. the 7th Symposium on Natural Language Processing
  • Place of Presentation: Pattaya
• [Presentation] Inter-language prosodic style modification experiment using word impression vectorfor communicative speech generation (2007)
  • Author(s): K. Li, Y. Greenberg and Y. Sagisaka
  • Organizer: Proc. Interspeech
  • Place of Presentation: Antwerp
• [Presentation] Prosody Generation for Communicative Speech Synthesis (2007)
  • Author(s): Yoshinori Sagisaka
  • Organizer: Taiwan-Japan Joint Workshop on Speech Science and Technologies
  • Place of Presentation: 台北
• [Presentation] Automatic extraction of paralinguistic information from communicative speech (2007)
  • Author(s): Mingzhao Zhu, Ke Li, Yoko Greenberg and Yoshinori Sagisaka
  • Organizer: SNLP 2007
  • Place of Presentation: Pattaya
• [Presentation] 印象表現ベクトルに基づく言語間韻律変換 (2007)
  • Author(s): 李 克, グリーンバーグ 陽子, 匂坂 芳典
  • Organizer: 日本音響学会2007年秋季研究発表会
  • Place of Presentation: 山梨
• [Presentation] 自然発話の韻律情報に基づく聴覚印象の自動抽出 (2007)
  • Author(s): 朱 明朝, 李 克, グリーンバーグ 陽子, 匂坂 芳典
  • Organizer: 日本音響学会2007年秋季研究発表会
  • Place of Presentation: 山梨
• [Presentation] Inter-language prosodic style modification experiment using word impression vector for communicative speech generation (2007)
  • Author(s): Ke Li, Yoko Greenberg and Yoshinori Sagisaka
  • Organizer: Interspeech 2007
  • Place of Presentation: Antwerp
• [Presentation] Towards Computing Phonetics (2006)
  • Author(s): Y. Sagisaka
  • Organizer: Proc. The 7th Phonetic Conference of China and International forum on Phonetic Frontiers.
  • Place of Presentation: 北京(中国)
• [Presentation] A trial of communicative prosody generation based on control characteristic of one word utterance observed in real conversational speech (2006)
  • Author(s): Y. Greenberg, N. Shibuya, M. Tsuzaki, H. Kato, Y. Sagisaka
  • Organizer: Proc. Speech prosody
  • Place of Presentation: Dresden(Germany)
• [Presentation] 印象表現によるパラ言語情報を用いた韻律制御 (2006)
  • Author(s): 李克, グリーンバーグ陽子, 渋谷渚, 匂坂芳典
  • Organizer: 2006年秋季日本音響学会講演
  • Place of Presentation: 金沢
• [Presentation] 音声研究の輝かしい展開を求めて-数理モデルからの提案- (2006)
  • Author(s): 匂坂芳典
  • Organizer: 日本音声学会創立80周年記念フォーラム資料
  • Place of Presentation: 東京
• [Presentation] A trial of communicative prosody generation basedon control characteristic of one word utterance observed in real conversational speech (2006)
  • Author(s): Y.Greenberg, N.Shibuya, M.Tsuzaki, H.Kato, Y.Sagisaka
  • Organizer: Proc. Speech prosody
  • Place of Presentation: Dresden
• [Presentation] Synthesis and Recognition of Communicative Prosody
  • Author(s): Y. Sagisaka, Y. Greenberg, K. Li, M. Zhu, M. Tsuzaki, H. Kato
  • Organizer: The 8th Phonetic Conference of China and International Symposium on Phonetic Frontiers 2008
  • Place of Presentation: 北京
• [Presentation] Communicative prosody processing for synthesis and recognition of para-linguistic information
  • Author(s): Y. Sagisaka, Y. Greenberg, K. Li, M. Zhu, M. Tsuzaki, H. Kato
  • Organizer: ICCA 2008
  • Place of Presentation: Yangon(Myanmar)
• [Book] On the analysis of F0 control characteristics of nonverbal utterances and its application to communicative prosody generation (2007)
  • Author(s): K. Li, Y. Greenberg, N. Shibuya, N. Campbell, Y. Sagisaka
  • Publisher: IOS Press
• [Book] in NATO Security through Science Series E : Human and Societal Dynamics Vol.8 The Fundamentals of Verbal and Non-verbal Communication and the Biometric Issue (2007)
  • Author(s): K.Li , Y Greenberg, N.Shibuya, N.Campbell, Y.Sagisaka
  • Publisher: IOS Press On the analysis of F0 control characteristics of nonverbal utterances and its application to communicative prosody generation
• [Book] NATO Security through Science Series E: Human and Societal Dynamics Vol.8 The Fundamentals of Verbal and Non-verbal Communication and the Biometric Issue (2007)
  • Author(s): K. Li, Y.Greenberg, N.Shibuya, N.Campbell, Y.Sagisaka
  • Publisher: IOS Press
• [Book] 語彙情報を用いた会話韻律生成について (2006)
  • Author(s): 匂坂芳典, グリーンバーグ陽子, 山下琢美
  • Publisher: くろしお出版
• [Book] 「文法と音声V」所載「語彙情報を用いた会話韻律生成について」 (2006)
  • Author(s): 匂坂、グリンバーグ、山下
  • Publisher: くろしお出版