• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Multimodal Dialogue System with Knowledge-based User Adaptive Dialogue Management

Research Project

Project/Area Number 19H04159
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypeSingle-year Grants
Section一般
Review Section Basic Section 61020:Human interface and interaction-related
Research InstitutionSeikei University

Principal Investigator

Nakano Yukiko  成蹊大学, 理工学部, 教授 (40422505)

Co-Investigator(Kenkyū-buntansha) 坂戸 達陽  成蹊大学, 理工学部, 助教 (10780679)
高瀬 裕  成蹊大学, 理工学部, 助教 (60744000)
Project Period (FY) 2019-04-01 – 2023-03-31
Project Status Completed (Fiscal Year 2022)
Budget Amount *help
¥17,030,000 (Direct Cost: ¥13,100,000、Indirect Cost: ¥3,930,000)
Fiscal Year 2022: ¥3,380,000 (Direct Cost: ¥2,600,000、Indirect Cost: ¥780,000)
Fiscal Year 2021: ¥4,160,000 (Direct Cost: ¥3,200,000、Indirect Cost: ¥960,000)
Fiscal Year 2020: ¥4,810,000 (Direct Cost: ¥3,700,000、Indirect Cost: ¥1,110,000)
Fiscal Year 2019: ¥4,680,000 (Direct Cost: ¥3,600,000、Indirect Cost: ¥1,080,000)
Keywordsインタビュー対話システム / 大規模知識 / 質問生成 / ジェスチャ生成 / 対話システム / 意味内容 / 生成モデル / マルチモーダルモデル / ユーザ適応 / 会話参加者特性 / ジェスチャ / 性格特性 / マルチモーダル理解 / 会話エージェント / マルチモーダル / 話題埋め込み / 知識埋め込み / ジェスチャ形態決定 / 個人適応
Outline of Research at the Start

AIスピーカーやチャットボット等,機械との音声対話が身近になりつつある.しかし現在,雑談対話といわれる,オープンドメインの対話システムは,個人に適応した知的なコミュニケーションができるレベルには至っていない.本研究では,大規模な知識ベースとコーパス学習モデルを融合し,オープンドメインの個人適応型対話システムの実現を目指す.また,ユーザ応答のマルチモーダルな解析を行うことにより,ユーザとの対話を通して,ユーザモデルを獲得する方法を確立する.さらに,姿を変えても,一貫したユーザ適応性を保証する会話エージェントシステムを実現するために,エージェントマイグレーションのプラットフォームを開発する.

Outline of Final Research Achievements

With the goal of developing a system that acquires users' preferences through dialogue, we proposed an interview dialogue system that asks users' food preferences through conversations. In order to realize meaningful and intelligent dialogues, we focus on generating questions using large-scale knowledge and learning semantic content of the interviewer's questions. This allows to generate a rich variety of dialogues and more appropriate questions than previous methods.

Academic Significance and Societal Importance of the Research Achievements

対話システムが実用化され,社会に普及し始める中,個人適応型のサービスを提供するためには,対話システムが自動的にユーザの特性を獲得することが必要となる.本研究は,食に関する嗜好獲得に焦点を当て,ユーザがシステムとの対話を楽しむ中で,システムがユーザの嗜好を獲得するインタビュー対話システムを開発した.本研究は,「知識」と「文脈」という対話制御には不可欠である要素に焦点を当て,最新の深層学習の利点を活かしつつ,深層学習モデルに基づく対話生成手法ではあまり注目されていなかった「発話の意味」を重視した手法を提案している.

Report

(5 results)
  • 2022 Annual Research Report   Final Research Report ( PDF )
  • 2021 Annual Research Report
  • 2020 Annual Research Report
  • 2019 Annual Research Report
  • Research Products

    (36 results)

All 2023 2022 2021 2020 2019 Other

All Int'l Joint Research (3 results) Journal Article (6 results) (of which Peer Reviewed: 6 results,  Open Access: 3 results) Presentation (24 results) (of which Int'l Joint Research: 11 results,  Invited: 1 results) Remarks (3 results)

  • [Int'l Joint Research] ISN-CNRS/CNRS-ISIR(フランス)

    • Related Report
      2022 Annual Research Report
  • [Int'l Joint Research] Augsburg university(ドイツ)

    • Related Report
      2022 Annual Research Report
  • [Int'l Joint Research] CMU(米国)

    • Related Report
      2019 Annual Research Report
  • [Journal Article] Estimating and Visualizing Persuasiveness of Participants in Group Discussions2023

    • Author(s)
      Ito Atsushi、Nakano Yukiko I.、Nihei Fumio、Sakato Tatsuya、Ishii Ryo、Fukayama Atsushi、Nakamura Takao
    • Journal Title

      情報処理学会論文誌

      Volume: 64

    • Related Report
      2022 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Generating Agent’s Gestures that Express Personality Traits2021

    • Author(s)
      中野 有紀子、大山 真央、二瓶 芙巳雄、東中 竜一郎、石井 亮
    • Journal Title

      The Transactions of Human Interface Society

      Volume: 23 Issue: 2 Pages: 153-164

    • DOI

      10.11184/his.23.2_153

    • NAID

      130008043055

    • ISSN
      1344-7262, 2186-8271
    • Year and Date
      2021-05-25
    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Question Generation in Dialogue Systems based on Embedding Representation of Knowledge and Topics: Towards Food Preference Interview Systems2021

    • Author(s)
      Zeng Jie、Nakano Yukiko
    • Journal Title

      Journal of Natural Language Processing

      Volume: 28 Issue: 2 Pages: 598-631

    • DOI

      10.5715/jnlp.28.598

    • NAID

      130008052563

    • ISSN
      1340-7619, 2185-8314
    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] 多人数遠隔コミュニケーションにおける肯定的感情表出支援の効果と支援適用タイミングの決定2021

    • Author(s)
      二瓶芙巳雄,田口和佳奈,中野有紀子,深澤伸一,赤津裕子
    • Journal Title

      情報処理学会論文誌

      Volume: 62 Pages: 761-771

    • NAID

      170000184350

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Validating the Effectiveness of a Meeting Summarizing Browser that Implements an Important Utterance Detection Model based on Multimodal Information2020

    • Author(s)
      二瓶 芙巳雄、中野 有紀子
    • Journal Title

      The Transactions of Human Interface Society

      Volume: 22 Issue: 2 Pages: 137-150

    • DOI

      10.11184/his.22.2_137

    • NAID

      130007846089

    • ISSN
      1344-7262, 2186-8271
    • Year and Date
      2020-05-25
    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Exploring Methods for Predicting Important Utterances Contributing to Meeting Summarization2019

    • Author(s)
      Nihei, F. and Nakano, Y.I.
    • Journal Title

      Multimodal Technologies and Interaction

      Volume: 3 Issue: 3 Pages: 50-50

    • DOI

      10.3390/mti3030050

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Presentation] Semantic Content Prediction for Generating Interviewing Dialogues to Elicit Users' Food Preferences2022

    • Author(s)
      Jie Zeng, Tatsuya Sakato, and Yukiko I. Nakano
    • Organizer
      Proceedings of the Second Workshop on When Creative AI Meets Conversational AI, in the 29th International Conference on Computational Linguistics (COLING2022)
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Detecting Change Talk in Motivational Interviewing using Verbal and Facial Information.2022

    • Author(s)
      Yukiko I. Nakano, Eri Hirose, Tatsuya Sakato, Shogo Okada, and Jean-Claude Martin
    • Organizer
      the 2022 International Conference on Multimodal Interaction (ICMI '22) [Best Paper Award Runner-up] https://doi.org/10.1145/3536221.3556607
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] ガウス混合モデルを用いたHuman-Agent Interactionにおけるユーザの表情分析2022

    • Author(s)
      坂戸達陽,ZENG Jie,中野有紀子
    • Organizer
      第21回情報科学技術フォーラム (FIT2022)2022/8/30
    • Related Report
      2022 Annual Research Report
  • [Presentation] グループディスカッションにおける説得力の推定のためのマルチパーティモデル2022

    • Author(s)
      伊藤温志,坂戸達陽,中野有紀子,二瓶芙巳雄,石井亮,深山篤,中村高雄
    • Organizer
      2022年度人工知能学会全国大会(第36回) 2022/6/1
    • Related Report
      2022 Annual Research Report
  • [Presentation] Predicting Persuasiveness of Participants in Multiparty Conversations2022

    • Author(s)
      Atsushi Ito, Yukiko I. Nakano, Fumio Nihei, Tatsuya Sakato, Ryo Ishii, Atsushi Fukayama, and Takao Nakamura
    • Organizer
      27th International Conference on Intelligent User Interfaces (IUI '22 Companion).Association for Computing Machinery, New York, NY, USA, 85-88. DOI:https://doi.org/10.1145/3490100.3516466
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 社会的信号処理に基づくコミュニケーション支援2022

    • Author(s)
      中野有紀子
    • Organizer
      日本動機づけ面接協会(JAMI)第10回大会,2022/3/12.(招待講演)
    • Related Report
      2021 Annual Research Report
    • Invited
  • [Presentation] マルチモーダル情報に基づく相槌的応答と表情強度の推定2021

    • Author(s)
      上野諒祐,坂戸達陽,中野有紀子
    • Organizer
      2021年度人工知能学会全国大会(第35回)E2-OS-5b-01,2021/6/8. 【全国大会優秀賞】
    • Related Report
      2021 Annual Research Report
  • [Presentation] グループディスカッションにおける説得力の推定2021

    • Author(s)
      伊藤温志,坂戸達陽,中野有紀子
    • Organizer
      2021年度人工知能学会全国大会(第35回)3E2-OS-5b-02,2021/6/8
    • Related Report
      2021 Annual Research Report
  • [Presentation] Estimating the Intensity of Facial Expressions Accompanying Feedback Responses in Multiparty Video-Mediated Communication2020

    • Author(s)
      Ryosuke Ueno, Yukiko I. Nakano, Jie Zeng, and Fumio Nihei
    • Organizer
      the 2020 International Conference on Multimodal Interaction (ICMI '20). Association for Computing Machinery, New York, NY, USA, pp. 144-152. DOI:https://doi.org/10.1145/3382507.3418878
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Impact of Personality on Nonverbal Behavior Generation2020

    • Author(s)
      Ryo Ishii, Chaitanya Ahuja, Yukiko I. Nakano, and Louis-Philippe Morency
    • Organizer
      the 20th ACM International Conference on Intelligent Virtual Agents (IVA '20).DOI:https://doi.org/10.1145/3383652.3423908
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Style transfer for co-speech gesture animation: A multi-speaker conditional mixture approach2020

    • Author(s)
      C. Ahuja, D. W. Lee, Y. I. Nakano, and L.-P. Morency
    • Organizer
      European Conference on Computer Vision(ECCV,2020)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 対象物のイメージに基づく図像的ジェスチャの形状推定2020

    • Author(s)
      二瓶 芙巳雄,中野 有紀子,東中 竜一郎,石井 亮
    • Organizer
      第19回情報科学技術フォーラム(FIT2020) (2020年9月1-3日 オンライン開催).【FIT論文賞】
    • Related Report
      2020 Annual Research Report
  • [Presentation] 知識と話題の埋め込みに基づく質問生成機能を有する料理嗜好インタビューシステムの評価2020

    • Author(s)
      曽 傑,中野 有紀子
    • Organizer
      2020年度人工知能学会全国大会(第34回)2H6-GS-9-02,2020/6/10
    • Related Report
      2020 Annual Research Report
  • [Presentation] マルチモーダル情報に基づく学習者のエンゲージメント推定2020

    • Author(s)
      内山 和也,中野 有紀子
    • Organizer
      2020年度人工知能学会全国大会(第34回)3M5-GS-12-03,2020/6/11
    • Related Report
      2020 Annual Research Report
  • [Presentation] マルチモーダル情報に基づく重要発言推定モデルを搭載したマルチモーダル議論要約ブラウザの評価2020

    • Author(s)
      二瓶 芙巳雄,中野 有紀子
    • Organizer
      2020年度人工知能学会全国大会(第34回)4E2-OS-19a-02,2020/6/12
    • Related Report
      2020 Annual Research Report
  • [Presentation] マルチモーダル情報に基づく応答と表情の自動生成2020

    • Author(s)
      上野 諒祐,曽 傑,中野 有紀子
    • Organizer
      2020年度人工知能学会全国大会(第34回)4E2-OS-19a-04,2020/6/12
    • Related Report
      2020 Annual Research Report
  • [Presentation] 音声・表情の非言語シグナルに基づく多人数遠隔コミュニケーション支援方法2020

    • Author(s)
      田口 和佳奈,中野 有紀子,二瓶 芙巳雄,深澤 伸一,赤津 裕子
    • Organizer
      2020年度人工知能学会全国大会(第34回)4E2-OS-19a-05,2020/6/12
    • Related Report
      2020 Annual Research Report
  • [Presentation] Exploiting a Large-scale Knowledge Graph for Question Generation in Food Preference Interview Systems2020

    • Author(s)
      Jie Zeng and Yukiko I. Nakano
    • Organizer
      the 25th International Conference on Intelligent User Interfaces Companion (ACM IUI’20 Companion)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] A Multimodal Meeting Browser that Implements an Important Utterance Detection Model based on Multimodal Information2020

    • Author(s)
      Fumio Nihei and Yukiko I. Nakano
    • Organizer
      the 25th International Conference on Intelligent User Interfaces Companion (ACM IUI’20 Companion)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Determining Iconic Gesture Forms based on Entity Image Representation2019

    • Author(s)
      Fumio Nihei, Yukiko I. Nakano, Ryuichiro Higashinaka, Ryo Ishii
    • Organizer
      ICMI 2019: 419-425
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Multimodal BigFive Personality Trait Analysis Using Communication Skill Indices and Multiple Discussion Types Dataset2019

    • Author(s)
      Candy Olivia Mawalim, Shogo Okada, Yukiko I. Nakano, Masashi Unoki
    • Organizer
      HCI (13) : 370-383
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Presenting Low-Accuracy Information of Emotion Recognition Enhances Human Awareness Performance2019

    • Author(s)
      Shinichi Fukasawa, Hiroko Akatsu, Wakana Taguchi, Fumio Nihei, Yukiko I. Nakano
    • Organizer
      HCI (4) 2019: 415-424
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] マルチモーダル特徴量を用いた談話セグメントの検出2019

    • Author(s)
      冨山 健, 二瓶 芙巳雄, 高瀬 裕, 中野 有紀子
    • Organizer
      2019年度人工知能学会全国大会(第33回), 4F3-OS-11b-02, 新潟県新潟市, 2019/6/7
    • Related Report
      2019 Annual Research Report
  • [Presentation] 知識と話題の埋め込み表現に基づく応答生成-料理嗜好インタビューシステムに向けてー2019

    • Author(s)
      曽 傑, 高瀬 裕, 中野 有紀子
    • Organizer
      2019年度人工知能学会全国大会(第33回), 2L3-J-9-01, 新潟県新潟市, 2019/6/5. 【全国大会優秀賞】
    • Related Report
      2019 Annual Research Report
  • [Remarks] 成蹊大学知的インタフェース研究室

    • URL

      https://iui.ci.seikei.ac.jp/

    • Related Report
      2022 Annual Research Report 2021 Annual Research Report
  • [Remarks] 成蹊大学 知的インタフェース研究室

    • URL

      http://iui.ci.seikei.ac.jp/achievement-2/

    • Related Report
      2020 Annual Research Report
  • [Remarks]

    • URL

      http://iui.ci.seikei.ac.jp/

    • Related Report
      2019 Annual Research Report

URL: 

Published: 2019-04-18   Modified: 2024-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi