Development of Assistive Technology for Sustaining Dialogue with Voice-Calling Robot to Monitor Older Adults at Bedside

• Project Area: Studies on intelligent systems for dialogue toward the human-machine symbiotic society
• Project/Area Number: 22H04872
• Research Category: Grant-in-Aid for Scientific Research on Innovative Areas (Research in a proposed research area)
• Allocation Type: Single-year Grants
• Review Section: Complex systems
• Research Institution: Institute of Physical and Chemical Research
• Principal Investigator: 大武 美保子 国立研究開発法人理化学研究所, 革新知能統合研究センター, チームリーダー (10361544)
• Project Period (FY): 2022-04-01 – 2024-03-31
• Project Status: Granted (Fiscal Year 2023)
• Budget Amount: ¥22,100,000 (Direct Cost: ¥17,000,000、Indirect Cost: ¥5,100,000); Fiscal Year 2023: ¥11,050,000 (Direct Cost: ¥8,500,000、Indirect Cost: ¥2,550,000); Fiscal Year 2022: ¥11,050,000 (Direct Cost: ¥8,500,000、Indirect Cost: ¥2,550,000)
• Keywords: 高齢者 / 健康 / 対話 / 支援技術 / ロボット

Outline of Research at the Start

本研究の目的は、高齢者の生活環境における心身の健康維持および増進を目指して、在宅のベッドサイドを中心とする生活環境内において設置する声掛けロボットの対話継続機能を向上するための技術を提案し、実証することである。具体的には、シナリオ対話技術に立脚して、以下の２つの手法を確立することを目指す：１)個人の特性アセスメント結果に基づく対話内容の生成および制御手法、２)関係構築による対話継続手法ならびにそれに基づく心身健康状態の推定手法。数か月間、対話を継続するために、個人の特性に応じた対話制御を行うことが可能な対話システムを開発する点に学術的な特色がある。

Outline of Annual Research Achievements

2022年度は、研究項目1,2について研究を進めた。
1 シナリオベース対話における間合いの推定：高齢者の健康を見守り育むことを目的とする，見守り声掛けロボットは，あらかじめ定めたシナリオに沿って声掛けする．利用者の発話の間合いに基づいて，発話終了か，息継ぎかを判別する．発話の間合いには個人差が大きいことから，発話終了か息継ぎかを判別するタスクを，強化学習ベースで行った場合と，あらかじめ収集したデータに基づいて閾値を設定して行った場合を比較した．調査の結果，人間がシステムに合わせようとする傾向がある場合には，閾値ベースの方がよく，そうではない傾向がある場合には，強化学習ベースで行った場合の方がよいという示唆が得られた．
2 グループ会話の司会進行におけるロボットの受容：グループ会話を司会進行し，発話量に応じて話者交代を促す，グループ会話を見守り声掛けするグループ会話支援ロボットを開発している．司会進行と話者交代自体は，音声のみで行うことも可能であり，ロボットが身体性を有する効果については，明らかでなかった．そこで，ロボット，音声のみ，人間が司会した場合とで，参加者の感じ方などがどのように異なるかを解析した．実験自体は，コロナ禍前に行った．発話が少ない人に当てる，発話が多い人を止めるなど，人間の司会者にとっても難しいタスクを，ロボットおよび音声により自動化することは，有効であることが明らかになった．一方，発話量が多い人にとって，ロボットよりは，音声に発話を止められる方が，不快さが少ないとの知見が得られた．

Current Status of Research Progress

2: Research has progressed on the whole more than it was originally planned.

Reason

本研究の目的に即して、1)個人の特性に応じた対話制御手法、2)人間とロボットの関係構築に向けたロボットの受容について、それぞれ知見が得られたため。

Strategy for Future Research Activity

対話ロボットを、実験室の他、高齢者宅、および、高齢者施設において利用評価する実験を行い、その時の反応を取得し、ロボットの利用が人間に与える影響を検討する計画である。

Research Products

• [Int'l Joint Research] University of Friberg(ドイツ)
• [Int'l Joint Research] INRIA(フランス)
• [Int'l Joint Research] Nicholas Copernicus University(ポーランド)
• [Journal Article] Mild cognitive impairment prediction and cognitive score regression in the elderly using EEG topological data analysis and machine learning with awareness assessed in affective reminiscent paradigm (2024)
  • Author(s): Tomasz M. Rutkowski,Tomasz Komendzinski,Mihoko Otake-Matsuura
  • Journal Title: Frontiers in Aging Neuroscience Volume: 15 Pages: 1294139-1294139
  • DOI: 10.3389/fnagi.2023.1294139
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Home-Based Cognitive Intervention for Healthy Older Adults Through Asking Robots Questions: Randomized Controlled Trial (2024)
  • Author(s): Seiki Tokunaga, Takuya Sekiguchi, Kumi Watanabe Miura, Hikaru Sugimoto, Masato S. Abe, Kazuhiro Tamura, Taishiro Kishimoto, Takashi Kudo, Mihoko Otake-Matsuura
  • Journal Title: JMIR Aging Volume: 7 Pages: e47229-e47229
  • DOI: 10.2196/47229
  • Peer Reviewed / Open Access
• [Journal Article] Word-producing brain: Contribution of the left anterior middle temporal gyrus to word production patterns in spoken language (2023)
  • Author(s): Sugimoto Hikaru、Abe Masato S.、Otake-Matsuura Mihoko
  • Journal Title: Brain and Language Volume: 238 Pages: 105233-105233
  • DOI: 10.1016/j.bandl.2023.105233
  • Peer Reviewed / Open Access
• [Journal Article] Effect of home-based group conversation intervention using smartphone application on cognitive health and psychological well-being of older adults with subjective cognitive concerns in Japan: a randomized controlled trial protocol (2023)
  • Author(s): Miura Kumi Watanabe、Tokunaga Seiki、Sekiguchi Takuya、Sugimoto Hikaru、Otake-Matsuura Mihoko
  • Journal Title: Frontiers in Psychology Volume: 14 Pages: 1114790-1114790
  • DOI: 10.3389/fpsyg.2023.1114790
  • Peer Reviewed / Open Access
• [Journal Article] Time-orientations of older adults in group conversations and their association with memory functioning (2023)
  • Author(s): Sekiguchi, T., Sugimoto, H., Tokunaga, S., and Otake-Matsuura, M.
  • Journal Title: Current Psychology Volume: 43 Issue: 7 Pages: 5854-5867
  • DOI: 10.1007/s12144-023-04545-w
  • Peer Reviewed / Open Access
• [Journal Article] Novel EEG Network Neuroscience Features for Dementia and Awareness Level Estimation in Passive BCI Framework (2023)
  • Author(s): Tomasz M. Rutkowski,Tomasz Komendzinski,Mihoko Otake-Matsuura
  • Journal Title: Proceedings of the 10th International Brain-Computer Interface Meeting. BCI Society; 2023 Volume: -
  • Open Access / Int'l Joint Research
• [Journal Article] Machine learning approach for early onset dementia neurobiomarker using EEG network topology features (2023)
  • Author(s): Rutkowski, T. M., Abe, M. S., Komendzinski, T., Sugimoto, H., Narebski, S., and Otake-Matsuura, M.
  • Journal Title: Frontiers in Human Neuroscience Volume: 17 Pages: 1155194-1155194
  • DOI: 10.3389/fnhum.2023.1155194
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] AI, Robotics, and Clinical Research for Innovative Dementia Interventions: A Japanese-German Collaboration (2023)
  • Author(s): Simon Stock,Mihoko Otake,Ingo Kilimann,Markus Schinle,Kazumi Kumagai,Sascha Weimar,Marius Gerdes,Florian Mazura,Wilhelm Stork
  • Journal Title: International German-Japanese Workshop on Digital Dementia Intervention and Prevention Strategies in the Age of AI Volume: -
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Eye-tracking paradigms for the assessment of mild cognitive impairment: a systematic review (2023)
  • Author(s): Alexandra Wolf,Kornkanok Tripanpitak,Satoshi Umeda,Mihoko Otake-Matsuura
  • Journal Title: Frontiers in Psychology Volume: 14 Pages: 1197567-1197567
  • DOI: 10.3389/fpsyg.2023.1197567
  • Peer Reviewed / Open Access
• [Journal Article] Multi-modal EEG/fNIRS/ET Data Collection and Cloud Storage System Prototype for the Elderly Home-based Dementia Monitoring (2023)
  • Author(s): Takumi Kodama,Tomasz M. Rutkowski,Tomasz Komendzinski,Youichi M Ito,Mihoko Otake-Matsuura
  • Journal Title: 2023 45th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC): IEEE Press; 2023. Volume: -
  • Int'l Joint Research
• [Journal Article] Mild Cognitive Impairment Detection with Machine Learning and Topological Data Analysis Applied to EEG Time-series in Facial Emotion Oddball Paradigm - Objective Neurobiomarker for Early Onset Dementia Prediction Approach (2023)
  • Author(s): Tomasz M. Rutkowski,Masato S. Abe,Hikaru Sugimoto,Mihoko Otake-Matsuura
  • Journal Title: 2023 45th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC): IEEE Press; 2023. Volume: -
  • Open Access
• [Journal Article] Topological Data Analysis Applied to EEG for Objective Early Onset Dementia Neurobiomarker (2023)
  • Author(s): Tomasz M. Rutkowski,Masato S. Abe,Hikaru Sugimoto,Tomasz Komendzinski,Mihoko Otake-Matsuura
  • Journal Title: TDA Week 2023 Volume: -
  • Int'l Joint Research
• [Journal Article] Tract-Based Spatial Statistics Analysis of Diffusion Tensor Imaging in Older Adults After the PICMOR Intervention Program: A Pilot Study (2022)
  • Author(s): Sugimoto Hikaru、Otake-Matsuura Mihoko
  • Journal Title: Frontiers in Aging Neuroscience Volume: 14 Pages: 867417-867417
  • DOI: 10.3389/fnagi.2022.867417
  • Peer Reviewed / Open Access
• [Journal Article] Dementia Digital Neuro-biomarker Study from Theta-band EEG Fluctuation Analysis in Facial and Emotional Identification Short-term Memory Oddball Paradigm (2022)
  • Author(s): Rutkowski TM, Abe MS, Tokunaga S, Komendzinski T, Otake-Matsuura M.
  • Journal Title: 2022 44th Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC) Volume: - Pages: 4056-4059
  • DOI: 10.1109/embc48229.2022.9871991
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Passive BCI Oddball Paradigm for Dementia Digital Neuro-biomarker Elucidation from Attended and Inhibited ERPs Utilizing Information Geometry Classification Approaches. (2022)
  • Author(s): Rutkowski TM, Abe MS, Tokunaga S, Sugimoto H, Komendzinski T, Otake-Matsuura M.
  • Journal Title: 2022 IEEE International Conference on Systems, Man and Cybernetics (SMC) Volume: - Pages: 2657-2662
  • DOI: 10.1109/smc53654.2022.9945159
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Cognitive behavioral assistive technology (CBAT) as AI for super aged society (2022)
  • Author(s): M. Otake-Matsuura, M. S. Abe, T. Sekiguchi, S. Tokunaga, H. Sugimoto, T. M. Rutkowski, K. W. Miura
  • Journal Title: Gerontechnology Volume: 21 Issue: s Pages: 1-1
  • DOI: 10.4017/gt.2022.21.s.713.opp3
  • Peer Reviewed / Open Access
• [Journal Article] A summary report through home experiment on a dialogue system for older adults (2022)
  • Author(s): S. Tokunaga, K. Tamura, M. Otake-Matsuura
  • Journal Title: Gerontechnology Volume: 21 Issue: s Pages: 1-1
  • DOI: 10.4017/gt.2022.21.s.497.opp4
  • Peer Reviewed / Open Access
• [Journal Article] Development of RobotHub: Integration of external system to group conversation system for older adults (2022)
  • Author(s): S. Tokunaga, K. Fukumori, K. Tamura, K. Inoue, T. Kawahara, M. Otake-Matsuura
  • Journal Title: Gerontechnology Volume: 21 Issue: s Pages: 1-1
  • DOI: 10.4017/gt.2022.21.s.510.opp4
  • Peer Reviewed / Open Access
• [Journal Article] Voice Over Body? Older Adults’ Reactions to Robot and Voice Assistant Facilitators of Group Conversation (2022)
  • Author(s): Katie Seaborn, Takuya Sekiguchi, Seiki Tokunaga, Norihisa P. Miyake, Mihoko Otake-Matsuura
  • Journal Title: International Journal of Social Robotics Volume: 15 Issue: 2 Pages: 143-163
  • DOI: 10.1007/s12369-022-00925-7
  • Peer Reviewed / Open Access
• [Journal Article] Annotation System for Dialogue Datasets of Older Adult’s with Photos and Storytelling (2022)
  • Author(s): Tokunaga, S., Takata, S., Tamura, K., Otake-Matsuura, M.
  • Journal Title: 2022 Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS&ISIS) Volume: - Pages: 447-452
  • DOI: 10.1007/978-3-031-21047-1_41
  • ISBN: 9783031210464, 9783031210471
  • Peer Reviewed / Open Access
• [Journal Article] Cross-cultural Evaluation of Dementia Passive BCI Neuro-biomarker Candidates (2022)
  • Author(s): T. M. Rutkowski, S. Narebski, P. Bekier, T. Komendzinski, H. Sugimoto and M. Otake-Matsuura
  • Journal Title: 2022 Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS&ISIS) Volume: - Pages: 1-1
  • DOI: 10.1109/scisisis55246.2022.10002105
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Association between social comparison orientation and hippocampal properties in older adults: A multimodal MRI study (2022)
  • Author(s): Sugimoto Hikaru、Sekiguchi Takuya、Otake-Matsuura Mihoko
  • Journal Title: Social Neuroscience Volume: 17 Issue: 6 Pages: 544-557
  • DOI: 10.1080/17470919.2023.2166580
  • Peer Reviewed / Open Access
• [Presentation] 認知症予防を支援するロボット (2024)
  • Author(s): 大武美保子
  • Organizer: RRI勉強会
  • Invited
• [Presentation] 今からできる認知症の予防 (2024)
  • Author(s): 大武美保子
  • Organizer: 岸和田市認知症ケア（KDC）報告会
  • Invited
• [Presentation] Embodied Intelligence in Conversations and its Appliction to Cognitive Intervention and Prediction (2024)
  • Author(s): Mihoko Otake-Matsuura
  • Organizer: The 4th Embodied Intelligence Conference
  • Int'l Joint Research / Invited
• [Presentation] 高齢者の認知的健康を見守り育むAI (2023)
  • Author(s): 大武美保子
  • Organizer: AIと文化シンポジウム「人の行動変容におけるAIの役割」
  • Invited
• [Presentation] AI for cognitive intervention and prediction of cognitive decline (2023)
  • Author(s): Mihoko Otake-Matsuura
  • Organizer: International German-Japanese Workshop on Digital Dementia Intervention and Prevention Strategies in the Age of AI
  • Int'l Joint Research / Invited
• [Presentation] ありたい未来を共に考え行動を促すAIロボット (2023)
  • Author(s): 大武美保子
  • Organizer: 日本ロボット学会学術講演会2023
• [Presentation] AI and robots for cognitive intervention and prediction of cognitive decline (2023)
  • Author(s): Mihoko Otake-Matsuura
  • Organizer: 6th EU-Japan Workshop on Neurorobotics: Healthcare and aging in the era of AI
  • Int'l Joint Research
• [Presentation] 認知症予防とAI (2023)
  • Author(s): 大武美保子
  • Organizer: 第12回日本認知症予防学会学術集会
  • Invited
• [Presentation] Conversation Assistive Technology: Face to Face, Dialogue System, and Remote Communication (2023)
  • Author(s): Mihoko Otake-Matsuura
  • Organizer: Second International Workshop on Complex Systems Science and Health Neuroscience
  • Int'l Joint Research
• [Presentation] Network Neuroscience Dementia Neurobiomarker with ML Support (2023)
  • Author(s): Tomasz M.RUTKOWSKI
  • Organizer: Second International Workshop on Complex Systems Science and Health Neuroscience
  • Int'l Joint Research
• [Presentation] Predictive EEG Biomarkers for Cognitive Impairment (2023)
  • Author(s): Kornkanok TRIPANPITAK
  • Organizer: Second International Workshop on Complex Systems Science and Health Neuroscience
  • Int'l Joint Research
• [Presentation] Leverage the potential of data collection and utilization to foster cognitively healthy aging (2023)
  • Author(s): Mihoko Otake-Matsuura, Mike Martin
  • Organizer: International Data Week, SciDataCon 2023
  • Int'l Joint Research
• [Presentation] Contribution of the left anterior middle temporal gyrus to word production patterns in spoken language (2022)
  • Author(s): Sugimoto, H., Abe, M.S., and Otake-Matsuura, M.
  • Organizer: 29th annual meeting of the Cognitive Neuroscience Society (CNS)
  • Int'l Joint Research
• [Presentation] 高齢者見守り声掛けロボットのシナリオベース対話中のユーザ発話終了予測に基づく話者交替タイミングの決定 (2022)
  • Author(s): 熊谷 和実, 徳永 清輝, 三宅 徳久, 田村 和弘, 水内 郁夫, 大武 美保子
  • Organizer: 2022年度人工知能学会全国大会（第36回）
• [Presentation] ロボット声掛け音声の個人適合に向けた対話中の感情推定結果の利用手法の検討 (2022)
  • Author(s): 熊谷 和実, 徳永 清輝, 三宅 徳久, 田村 和弘, 福森 航輔, 水内 郁夫, 大武 美保子
  • Organizer: LIFE2022
• [Presentation] Neural basis underlying production of a variety of words in conversations (2022)
  • Author(s): Sugimoto, H., Abe, M.S., and Otake-Matsuura, M.
  • Organizer: Constructive Approaches to Co-Creative Communication at the Joint Conference on Language Evolution, 8
  • Int'l Joint Research
• [Presentation] Robots Using “Aizuchi” in Online Group Conversation (2022)
  • Author(s): KOBUKI Sota, SEABORN Katie, TOKUNAGA Seik, FUKUMORI Kosuke, HIDAKA Shun, TAMURA Kazuhiro, INOUE Koji, KAWAHARA Tatsuya, OTAKE-MASTUURA Mihoko
  • Organizer: 第40回 日本ロボット学会学術講演会
• [Book] テクノロジー・ロードマップ 2024-2033 全産業編 (2023)
  • Author(s): 出川 通 ほか95名
  • Total Pages: 602
  • Publisher: 日経BP
  • ISBN: 9784296203031
• [Book] テクノロジー・ロードマップ2023-2032 全産業編 (2022)
  • Author(s): 出川 通 ほか100名
  • Total Pages: 608
  • Publisher: 日経BP
  • ISBN: 9784296200474
• [Remarks] 認知行動支援技術チーム
  • URL: https://aip.riken.jp/labs/goalorient_tech/cogn_behav_assist_tech/