A study on a response generation method based on simultaneous generation of speech and physical expression for conversational AI

2021 Fiscal Year Final Research Report

• Project/Area Number: 20K19903
• Research Category: Grant-in-Aid for Early-Career Scientists
• Allocation Type: Multi-year Fund
• Review Section: Basic Section 61060:Kansei informatics-related
• Research Institution: NTT Communication Science Laboratories
• Principal Investigator: Chiba Yuya 日本電信電話株式会社NTTコミュニケーション科学基礎研究所, 協創情報研究部, 研究員 (30780936)
• Project Period (FY): 2020-04-01 – 2022-03-31
• Keywords: 音声対話システム / マルチモーダル情報処理 / 応答生成

Outline of Final Research Achievements

This study constructed a spoken response generation method using linguistic and prosodic information of the user's utterance based on the neural conversational model, which is actively studied for dialogue systems. Our experiments confirmed that the proposed method can produce F0 sequences that are closer to natural speech than the baseline. Then, our research group expanded the spoken response generation model to a multimodal response generation model, that can consider the facial expression control signals. Experimental results suggested that the performance of the model can be improved by considering multimodal information. Additionally, we also proposed a response timing estimation model based on the dialogue context encoder and the continuous LSTM. We have presented six papers at domestic conferences and workshops, four papers at international conferences, and applied for one patent.

Free Research Field

対話システム

Academic Significance and Societal Importance of the Research Achievements

本課題では，近年盛んに研究されているニューラルベースの応答生成技術が言語情報だけでなく韻律や表情といった非言語情報も扱えること，また人間のコミュニケーションにおける社会的な現象を考慮できる可能性があることを示した．加えて，そのような非言語ベースの応答生成において効果的にモデルを学習するためのデータ拡張手法，自然なタイミング・間での応答を実現する応答タイミング推定手法も提案し，それぞれ一定の効果が得られた．これらの検討より，非言語情報を取り入れた対話システムの応答生成研究における有益な知見を提供できたと考える．本研究の成果は今後ますます重要性を増す対話システムの自然性の向上に寄与するものである．