End-to-End Model for Task-Independent Speech Understanding and Dialogue

• Project/Area Number: 20H00602
• Research Category: Grant-in-Aid for Scientific Research (A)
• Allocation Type: Single-year Grants
• Section: 一般
• Review Section: Medium-sized Section 61:Human informatics and related fields
• Research Institution: Kyoto University
• Principal Investigator: 河原 達也 京都大学, 情報学研究科, 教授 (00234104)
• Co-Investigator(Kenkyū-buntansha): 井上 昂治 京都大学, 情報学研究科, 助教 (10838684) ; 吉井 和佳 京都大学, 情報学研究科, 准教授 (20510001)
• Project Period (FY): 2020-04-01 – 2024-03-31
• Project Status: Granted (Fiscal Year 2023)
• Budget Amount: ¥44,720,000 (Direct Cost: ¥34,400,000、Indirect Cost: ¥10,320,000); Fiscal Year 2023: ¥9,620,000 (Direct Cost: ¥7,400,000、Indirect Cost: ¥2,220,000); Fiscal Year 2022: ¥12,220,000 (Direct Cost: ¥9,400,000、Indirect Cost: ¥2,820,000); Fiscal Year 2021: ¥12,220,000 (Direct Cost: ¥9,400,000、Indirect Cost: ¥2,820,000); Fiscal Year 2020: ¥10,660,000 (Direct Cost: ¥8,200,000、Indirect Cost: ¥2,460,000)
• Keywords: 音声理解 / 音声対話 / 音声認識 / End-to-Endモデル

Outline of Research at the Start

人間どうしが行うような音声コミュニケーションにおいて、相手の意図・概念・感情を理解し、応答するためのモデルを研究する。音声から理解さらには相槌生成を行う系と、理解結果に応じて適切な知識・モデルを用いて応答生成を行う系のEnd-to-Endモデル化を行う。これにより、音声認識誤りの影響と音声に含まれるニュアンスや感情などの情報を考慮して、インタラクションを行うシステムを実現する。傾聴・カウンセリングや就職面接などを対象として、モデル化及び対話システムのロボットによる実装を行う。これにより、人間のコミュニケーションスキルの解明と実現を目指す。

Outline of Annual Research Achievements

End-to-Endモデルに基づく汎用的な音声理解・対話に関して、音声認識の高度化の観点と対話生成の観点から各々以下の研究を実施した。
まず、人間どうしの自然な話し言葉音声から、フィラーや言い誤りの削除・句読点や脱落した助詞の挿入・口語的な表現の修正などの適宜必要な編集を行いながら、直接可読性の高い書き言葉スタイルの文を出力するEnd-to-Endモデルを設計・構築した。その際に、音声に忠実な書き起こしを疑似的に復元してEnd-to-Endモデルの学習を補助する手法と、句読点位置を手がかりとした音声区分化手法も併せて提案し、各々の効果を示した。衆議院審議音声を用いた評価実験により、提案手法は音声認識とテキストベースの話し言葉スタイル変換を組み合わせたカスケード型のアプローチより高精度かつ高速に会議録テキストを生成できることを確認した。
次に、ユーザの入力発話からシステムの応答を生成するEnd-to-End(Seq-to-Seq)モデルにおいて、感情認識を統合するとともに、応答から入力発話を復元するモデルも統合学習することで、文脈理解と感情認識を伴った応答生成の実現を図った。感情認識と検索型の応答を組み合わせることで、共感的な対話が実現できることを確認した。
さらに、音声認識モデルにおける自己教師付き学習の導入についても検討し、音声認識と言語認識・ドメイン認識を同時にEnd-to-Endモデルで行い、かつ後者の認識結果を利用することで、音声認識の精度が改善されることを示した。

Current Status of Research Progress

2: Research has progressed on the whole more than it was originally planned.

Reason

音声認識・理解及び対話システムにおいて、多面的に研究を実施し、着実に成果を挙げることができた。

Strategy for Future Research Activity

要素技術をさらに発展させるとともに、システムとして統合していく。

Research Products

• [Journal Article] End-to-End Generation of Written-style Transcript of Speech from Parliamentary Meetings (2023)
  • Author(s): Mimura Masato、Kawahara Tatsuya
  • Journal Title: Journal of Natural Language Processing Volume: 30 Issue: 1 Pages: 88-124
  • DOI: 10.5715/jnlp.30.88
  • ISSN: 1340-7619, 2185-8314
  • Peer Reviewed / Open Access
• [Journal Article] TriECCC: Trilingual Corpus of the Extraordinary Chambers in the Courts of Cambodia for Speech Recognition and Translation Studies (2022)
  • Author(s): Soky Kak、Mimura Masato、Kawahara Tatsuya、Chu Chenhui、Li Sheng、Ding Chenchen、Sam Sethserey
  • Journal Title: International Journal of Asian Language Processing Volume: 31 Issue: 03n04 Pages: 1-21
  • DOI: 10.1142/s2717554522500072
  • Peer Reviewed / Open Access
• [Journal Article] Synthesizing waveform sequence-to-sequence to augment training data for sequence-to-sequence speech recognition (2021)
  • Author(s): S.Ueno, M.Mimura, S.Sakai, and T.Kawahara
  • Journal Title: Acoustical Science and Technology Volume: 42 Issue: 6 Pages: 333-343
  • DOI: 10.1250/ast.42.333
  • NAID: 130008110355
  • ISSN: 0369-4232, 1346-3969, 1347-5177
  • Year and Date: 2021-11-01
  • Peer Reviewed / Open Access
• [Journal Article] Alignment knowledge distillation for online streaming attention-based speech recognition (2021)
  • Author(s): H.Inaguma and T.Kawahara
  • Journal Title: IEEE/ACM Trans. Audio, Speech & Language Process Volume: Vol.29 Pages: 1-15
  • DOI: 10.1109/taslp.2021.3133217
  • Peer Reviewed / Open Access
• [Presentation] Fusing multiple bandwidth spectrograms for improving speech enhancement. (2022)
  • Author(s): H.Shi, Y.Shu, L.Wang, J.Dang, and T.Kawahara.
  • Organizer: APSIPA ASC
  • Int'l Joint Research
• [Presentation] Subband-based spectrogram fusion for speech enhancement by combining mapping and masking approaches. (2022)
  • Author(s): H.Shi, L.Wang, S.Li, J.Dang, and T.Kawahara.
  • Organizer: APSIPA ASC
  • Int'l Joint Research
• [Presentation] Non-autoregressive error correction for CTC-based ASR with phone-conditioned masked LM. (2022)
  • Author(s): H.Futami, H.Inaguma, S.Ueno, M.Mimura, S.Sakai, and T.Kawahara.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] End-to-end speech-to-punctuated-text recognition. (2022)
  • Author(s): J.Nozaki, T.Kawahara, K.Ishizuka, and T.Hashimoto.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] Leveraging simultaneous translation for enhancing transcription of low-resource language via cross attention mechanism. (2022)
  • Author(s): K.Soky, S.Li, M.Mimura, C.Chu, and T.Kawahara.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] Monaural speech enhancement based on spectrogram decomposition for convolutional neural network-sensitive feature extraction. (2022)
  • Author(s): H.Shi, L.Wang, S.Li, J.Dang, and T.Kawahara.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] Selective multi-task learning for speech emotion recognition using corpora of different styles. (2022)
  • Author(s): H.Zhang, M.Mimura, T.Kawahara, and K.Ishizuka.
  • Organizer: IEEE-ICASSP
  • Int'l Joint Research
• [Presentation] Phone-informed refinement of synthesized mel spectrogram for data augmentation in speech recognition. (2022)
  • Author(s): S.Ueno and T.Kawahara.
  • Organizer: IEEE-ICASSP
  • Int'l Joint Research
• [Presentation] An end-to-end model from speech to clean transcript for parliamentary meetings (2021)
  • Author(s): M.Mimura, S.Sakai, and T.Kawahara
  • Organizer: APSIPA ASC
  • Int'l Joint Research
• [Presentation] ASR rescoring and confidence estimation with ELECTRA (2021)
  • Author(s): H.Futami, H.Inaguma, M.Mimura, S.Sakai, and T.Kawahara
  • Organizer: IEEE Workshop Automatic Speech Recognition & Understanding (ASRU)
  • Int'l Joint Research
• [Presentation] Data augmentation for ASR using TTS via a discrete representation (2021)
  • Author(s): S.Ueno, M.Mimura, S.Sakai, and T.Kawahara
  • Organizer: IEEE Workshop Automatic Speech Recognition & Understanding (ASRU)
  • Int'l Joint Research
• [Presentation] VAD-free streaming hybrid CTC/Attention ASR for unsegmented recording (2021)
  • Author(s): H.Inaguma, M.Mimura, and T.Kawahara
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] StableEmit: Selection probability discount for reducing emission latency of streaming monotonic attention ASR (2021)
  • Author(s): H.Inaguma, M.Mimura, and T.Kawahara
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] Multi-referenced training for dialogue response generation (2021)
  • Author(s): T.Zhao and T.Kawahara
  • Organizer: SIGdial Meeting Discourse & Dialogue
  • Int'l Joint Research
• [Presentation] Response generation to out-of-database questions for example-based dialogue systems. (2020)
  • Author(s): S.Isonishi, K.Inoue, D.Lala, K.Takanashi, and T.Kawahara.
  • Organizer: Int'l Workshop Spoken Dialogue Systems (IWSDS)
  • Int'l Joint Research
• [Presentation] End-to-end speech emotion recognition combined with acoustic-to-word ASR model. (2020)
  • Author(s): H.Feng, S.Ueno, and T.Kawahara.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] End-to-end speech-to-dialog-act recognition. (2020)
  • Author(s): T.V.Dang, T.Zhao, S.Ueno, H.Inaguma, and T.Kawahara.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] Topic-relevant response generation using optimal transport for an open-domain dialog system. (2020)
  • Author(s): S.Zhang, T.Zhao, and T.Kawahara.
  • Organizer: COLING
  • Int'l Joint Research
• [Presentation] Distilling the knowledge of BERT for sequence-to-sequence ASR. (2020)
  • Author(s): H.Futami, H.Inaguma, S.Ueno, M.Mimura, S.Sakai, and T.Kawahara.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] CTC-synchronous training for monotonic attention model. (2020)
  • Author(s): H.Inaguma, M.Mimura, and T.Kawahara.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Presentation] Enhancing monotonic multihead attention for streaming ASR. (2020)
  • Author(s): H.Inaguma, M.Mimura, and T.Kawahara.
  • Organizer: INTERSPEECH
  • Int'l Joint Research
• [Book] 音声（下） (2022)
  • Author(s): 日本音響学会、岩野 公司、河原 達也、篠田 浩一、伊藤 彰則、増村 亮、小川 哲司、駒谷 和範
  • Total Pages: 208
  • Publisher: コロナ社
  • ISBN: 9784339013672
• [Book] 音声対話システム (2022)
  • Author(s): 井上 昂治、河原 達也
  • Total Pages: 272
  • Publisher: オーム社
  • ISBN: 9784274229541