• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Development of the responsive model for a virtual listener agent by using multimodal information

Research Project

Project/Area Number 16K00289
Research Category

Grant-in-Aid for Scientific Research (C)

Allocation TypeMulti-year Fund
Section一般
Research Field Human interface and interaction
Research InstitutionInstitute of Physical and Chemical Research (2018)
Ritsumeikan University (2016-2017)

Principal Investigator

Huang Hung-Hsuan  国立研究開発法人理化学研究所, 革新知能統合研究センター, 研究員 (00572950)

Co-Investigator(Kenkyū-buntansha) 林 勇吾  立命館大学, 総合心理学部, 准教授 (60437085)
Project Period (FY) 2016-04-01 – 2019-03-31
Project Status Completed (Fiscal Year 2018)
Budget Amount *help
¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000)
Fiscal Year 2018: ¥1,300,000 (Direct Cost: ¥1,000,000、Indirect Cost: ¥300,000)
Fiscal Year 2017: ¥1,300,000 (Direct Cost: ¥1,000,000、Indirect Cost: ¥300,000)
Fiscal Year 2016: ¥1,820,000 (Direct Cost: ¥1,400,000、Indirect Cost: ¥420,000)
Keywords会話エージェント / 高齢者支援 / 対話システム / マルチモーダルインタラクション / 会話状態推定 / 機械学習 / 深層学習 / 人工知能 / コミュニケーション / 傾聴 / 感情モデル
Outline of Final Research Achievements

This project aims the realization of an active listening agent which can support human volunteers in the domain of mental care older generation. During the project period, we worked on three tasks related to this topic. 1. The development of an automatic estimation model of the speaker’s engagement state. 2. The development of the nonverbal behavior generation model of the agent. 3. The agent platform the connect the developed models and drive a workable virtual listener agent. These tasks were done with a data-driven method. We conducted two subject experiment to collect multimodal data corpus and used machine learning methods to develop the models.

Academic Significance and Societal Importance of the Research Achievements

利用者に対しての癒し効果を狙って利用者から話しかければ,意図を持たず単純な反応を適当に返す市販のペットロボットや研究用ロボットが提案されている.しかし,言語・非言語チャンネルの両方が重要な対面対話である傾聴会話を扱い,それが行えるシステムはまだ存在していない.本研究は,人間同士による会話実験を通じて傾聴会話の参加者の具体的な言動の関係を分析し,モデル化,実用システムの開発まで行おうとするものである.傾聴という難しいタスクに踏み込み,コミュニケーションのメカニズムを解明するという科学的な意義と実際に動作するエージェントシステムの働きにより利用者の心的ケアを目指す工学的な意義の両方をもつ.

Report

(4 results)
  • 2018 Annual Research Report   Final Research Report ( PDF )
  • 2017 Research-status Report
  • 2016 Research-status Report
  • Research Products

    (38 results)

All 2019 2018 2017 2016

All Journal Article (3 results) (of which Peer Reviewed: 3 results,  Open Access: 2 results,  Acknowledgement Compliant: 1 results) Presentation (35 results) (of which Int'l Joint Research: 18 results,  Invited: 2 results)

  • [Journal Article] Tuning a conversation strategy for interactive recommendations in a chatbot setting2019

    • Author(s)
      Yuichiro Ikemoto, Varit Asawavetvutt, Kazuhiro Kuwabara and Hung-Hsuan Huang
    • Journal Title

      Journal of Information and Telecommunication

      Volume: 3 Issue: 2 Pages: 180-195

    • DOI

      10.1080/24751839.2018.1544818

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Analysis on the Participants' Functional Roles and their Transitions in Group Discussion Conversation2018

    • Author(s)
      張 琪、黄 宏軒、木村 清也、岡田 将吾、大田 直樹、桑原 和宏
    • Journal Title

      The Transactions of Human Interface Society

      Volume: 20 Issue: 1 Pages: 31-44

    • DOI

      10.11184/his.20.1_31

    • NAID

      130006401342

    • ISSN
      1344-7262, 2186-8271
    • Related Report
      2017 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] 傾聴エージェントの実現に向けた傾聴会話参加者の自他評価による数値化モデルの提案2016

    • Author(s)
      黄 宏軒,澁澤紗優美,林 勇吾,川越恭二
    • Journal Title

      ヒューマンインタフェース学会論文誌

      Volume: 18 Pages: 373-384

    • NAID

      130007671640

    • Related Report
      2016 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Presentation] Detection of Student Teacher's Intention using Multimodal Features in a Virtual Classroom2019

    • Author(s)
      Masato Fukuda, Hung-Hsuan Huang, and Toyoaki Nishida
    • Organizer
      11th International Conference on Agents and Artificial Intelligence (ICAART 2019)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Toward RNN based Micro Non-verbal Behavior Generation for Virtual Listener Agents2019

    • Author(s)
      Hung-Hsuan Huang, Masato Fukuda, Toyoaki Nishida
    • Organizer
      HCI International 2019
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Story Envisioning Framework for Visualized Collective Storytelling in Conversation2019

    • Author(s)
      Qiang Zhang, Maryam Mirzaei, Hung-Hsuan Huang, Toyoaki Nishida
    • Organizer
      HCI International 2019
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Development of a Platform of RNN Driven Multimodal Interaction for Embodied Conversational Agents2019

    • Author(s)
      Hung-Hsuan Huang, Masato Fukuda, Toyoaki Nishida
    • Organizer
      19th ACM International Conference on Intelligent Virtual Agents (IVA 2019)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Multimodal Assessment on Teaching Skills in a Virtual Rehearsal Environment2019

    • Author(s)
      Masato Fukuda, Hung-Hsuan Huang, Kazuhiro Kuwabara, and Toyoaki Nishida
    • Organizer
      19th ACM International Conference on Intelligent Virtual Agents (IVA 2019)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] マルチモーダル特徴量を用いた教員志望者の授業スキル評価モデルの提案2019

    • Author(s)
      福田匡人,黄 宏軒,桑 原和宏,西田豊明
    • Organizer
      HAIシンポジウム2018
    • Related Report
      2018 Annual Research Report
  • [Presentation] グループディスカッション参加ロボットの頭部動作の印象評価2019

    • Author(s)
      木村清也,黄 宏軒, 桑原和宏,西田豊明
    • Organizer
      HAIシンポジウム2018
    • Related Report
      2018 Annual Research Report
  • [Presentation] 回帰型ニューラルネットを用いた会話エージェントのアイドリング動作生成の検討2019

    • Author(s)
      黄 宏軒,福田匡人,西 田豊明
    • Organizer
      第33回人工知能学会全国大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] Conversational Informatics -Virtual Agents and their Applications2019

    • Author(s)
      Hung-Hsuan Huang
    • Organizer
      SOCRATES Meeting
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Proposal of a Multimodal Framework for Generating Robot's Spontaneous Attention Directions and Nods in Group Discussion2018

    • Author(s)
      Hung-Hsuan Huang, Seiya Kimura, Kazuhiro Kuwabara, Toyoaki Nishida
    • Organizer
      AI for Multimodal Human Robot Interaction (AI-MHRI) Workshop, IJCAI-ECAI 2018
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Estimating Speaker's Engagement from Non-verbal Features based on an Active Listening Corpus2018

    • Author(s)
      Lei Zhang, Hung-Hsuan Huang, Kazuhiro Kuwabara
    • Organizer
      HCI International 2018
    • Related Report
      2018 Annual Research Report 2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] Adopting Functional Roles for Improving Participants' Communication Skill in Group Discussion Conversation2018

    • Author(s)
      Hung-Hsuan Huang, Qi Zhang, Shogo Okada, Kazuhiro Kuwabara, Toyoaki Nishida
    • Organizer
      Workshop on Group Interaction Frontiers in Technology (GIFT 2018), 20th ACM International Conference on Multimodal Interaction (ICMI 2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] FACSvatar: An Open Source Modular Framework for Real-Time FACS based Facial Animation2018

    • Author(s)
      Stef van der Struijk, Hung-Hsuan Huang, Maryam Sadat Mirzaei, and Toyoaki Nishida
    • Organizer
      18th ACM International Conference on Intelligent Virtual Agents (IVA 2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Proposal of a Multi-purpose and Modular Virtual Classroom Framework for Teacher Training2018

    • Author(s)
      Masato Fukuda, Hung-Hsuan Huang, Kazuhiro Kuwabara, and Toyoaki Nishida
    • Organizer
      18th ACM International Conference on Intelligent Virtual Agents (IVA 2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Integration of DNN Generated Spontaneous Reactions with a Generic Multimodal Framework for Embodied Conversational Agents2018

    • Author(s)
      Hung-Hsuan Huang, Masato Fukuda, Stef van der Struijk, and Toyoaki Nishida
    • Organizer
      6th International Conference on Human-Agent Interaction (HAI 2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Investigation of Class Atmosphere Cognition in a VR Classroom2018

    • Author(s)
      Masato Fukuda, Hung-Hsuan Huang, Kazuhiro Kuwabara, and Toyoaki Nishida
    • Organizer
      6th International Conference on Human-Agent Interaction (HAI 2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 仮想学級の雰囲気表現を用いた教員志望者の指導訓練環境の検討2018

    • Author(s)
      福田 匡人,黄 宏軒, 桑原 和宏
    • Organizer
      第32回人工知能学会全国大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] グループディスカッション参加ロボットの頭部動作決定機構の検討2018

    • Author(s)
      木村 清也,黄 宏軒, 桑原 和宏
    • Organizer
      第32回人工知能学会全国大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 仮想学級におけるマルチモーダル特徴量を用いた教員志望者の授業意図の検出2018

    • Author(s)
      福田 匡人,黄 宏軒, 西田豊明
    • Organizer
      HCGシンポジウム2018
    • Related Report
      2018 Annual Research Report
  • [Presentation] コミュニケーション支援エージェントのための多人数会話マルチモーダル解析2018

    • Author(s)
      黄 宏軒
    • Organizer
      電子情報通信学会ヒューマンプローブ研究会(HPB)第19回研究会
    • Related Report
      2018 Annual Research Report
    • Invited
  • [Presentation] 仮想学級の雰囲気表現を用いた教員志望者の指導訓練環境の検討2018

    • Author(s)
      福田 匡人、黄 宏軒、桑原 和宏
    • Organizer
      第32回人工知能学会全国大会
    • Related Report
      2017 Research-status Report
  • [Presentation] グループディスカッション参加ロボットの頭部動作決定機構の検討2018

    • Author(s)
      木村 清也、黄 宏軒、桑原 和宏
    • Organizer
      第32回人工知能学会全国大会
    • Related Report
      2017 Research-status Report
  • [Presentation] Toward a Supporting System of Communication Skill: The Influence of Functional Roles of Participants in Group Discussion2017

    • Author(s)
      Qi Zhang, Hung-Hsuan Huang, Seiya Kimura, Shogo Okada, Yuki Hayashi, Yutaka Takase, Yukiko Nakano, Naoki Ohta, Kazuhiro Kuwabara
    • Organizer
      HCI International 2017
    • Place of Presentation
      Vancouver, Canada
    • Year and Date
      2017-07-12
    • Related Report
      2016 Research-status Report
    • Int'l Joint Research
  • [Presentation] グループディスカッション参加者の役割に基づいた会話状況とコミュニケーション能力の分析2017

    • Author(s)
      張 琪,木村清也,黄 宏軒,岡田将吾,林 佑樹,高瀬 裕,中野有紀子,桑原和宏
    • Organizer
      第31回人工知能学会全国大会
    • Place of Presentation
      ウインクあいち(愛知県・名古屋市)
    • Year and Date
      2017-05-24
    • Related Report
      2016 Research-status Report
  • [Presentation] グループディスカッションに参加するロボットにおけるアテンション対象モデルの提案2017

    • Author(s)
      木村清也,張 琪,黄 宏軒,岡田将吾,林 佑樹,高瀬 裕,中野有紀子,大田直樹,桑原和宏
    • Organizer
      第31回人工知能学会全国大会
    • Place of Presentation
      ウインクあいち(愛知県・名古屋市)
    • Year and Date
      2017-05-24
    • Related Report
      2016 Research-status Report
  • [Presentation] 教員志願者の練習システムにおける仮想学級の雰囲気モデルの提案2017

    • Author(s)
      福田匡人,黄 宏軒,桑原和宏,大田直樹
    • Organizer
      電子情報通信学会・ヒューマンコミュニケーション基礎研究会
    • Place of Presentation
      なみきスクウェア(福岡県・北九州市)
    • Year and Date
      2017-01-27
    • Related Report
      2016 Research-status Report
  • [Presentation] Proposal of a Parameterized Atmosphere Generation Model in a Virtual Classroom2017

    • Author(s)
      Masato Fukuda, Hung-Hsuan Huang, Naoki Ohta, Kazuhiro Kuwabara
    • Organizer
      5th International Conference on Human-Agent Interaction (HAI 2017)
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] Proposal of a Model to Determine the Attention Target for an Agent in Group Discussion with Non-verbal Features2017

    • Author(s)
      Seiya Kimura, Hung-Hsuan Huang, Qi Zhang, Shogo Okada, Naoki Ohta, Kazuhiro Kuwabara
    • Organizer
      5th International Conference on Human-Agent Interaction (HAI 2017)
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] グループディスカッションに参加するロボットのための頭部動作モデルの検討2017

    • Author(s)
      木村清也,黄 宏軒,岡田将吾,大田直樹,桑原和宏
    • Organizer
      HAIシンポジウム2017
    • Related Report
      2017 Research-status Report
  • [Presentation] 実験結果に基づく仮想学級雰囲気生成モデルの提案2017

    • Author(s)
      福田匡人,黄 宏軒,大田直樹,桑原和宏
    • Organizer
      HAIシンポジウム2017
    • Related Report
      2017 Research-status Report
  • [Presentation] 仮想学級システムに向けての自動評価基準の検討2017

    • Author(s)
      淀川 大,木下遼祐,黄 宏軒,大田直樹,桑原和宏
    • Organizer
      電子情報通信学会ヒューマンコミュニケーション基礎研究会
    • Related Report
      2017 Research-status Report
  • [Presentation] グループディスカッションにおける参加者役割とコミュニケーション能力印象評価の関連性分析2016

    • Author(s)
      張 琪,木村清也,黄 宏軒,岡田将吾,林 佑樹,高瀬 裕,中野有紀子,大田直樹,桑原和宏
    • Organizer
      HAIシンポジウム 2016
    • Place of Presentation
      東京大学駒場キャンパス(東京都・目黒区)
    • Year and Date
      2016-12-03
    • Related Report
      2016 Research-status Report
  • [Presentation] Development of a Virtual Classroom for High School Teacher Training2016

    • Author(s)
      Hung-Hsuan Huang, Yuki Ida, Kohei Yamaguchi, Kyoji Kawagoe
    • Organizer
      16th International Conference on Intelligent Virtual Agents (IVA 2016)
    • Place of Presentation
      Los Angeles, USA
    • Year and Date
      2016-09-21
    • Related Report
      2016 Research-status Report
    • Int'l Joint Research
  • [Presentation] Composing the Atmosphere of a Virtual Classroom with a Group of Student Agents2016

    • Author(s)
      Masato Fukuda, Hung-Hsuan Huang, Naoki Ohta, Kazuhiro Kuwabara
    • Organizer
      16th International Conference on Intelligent Virtual Agents (IVA 2016)
    • Place of Presentation
      Los Angeles, USA
    • Year and Date
      2016-09-21
    • Related Report
      2016 Research-status Report
    • Int'l Joint Research
  • [Presentation] 仮想学級による教員採用試験のための場面指導練習システムの開発2016

    • Author(s)
      福田匡人,冠野哲也,黄 宏軒,桑原和宏
    • Organizer
      第30回人工知能学会全国大会
    • Place of Presentation
      北九州国際会議場(福岡県・北九州市)
    • Year and Date
      2016-06-07
    • Related Report
      2016 Research-status Report

URL: 

Published: 2016-04-21   Modified: 2020-03-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi