Robotic System for Encouraging Elderly Dementia Patients to Communicate with Emotional Estimation and Monitoring Functions

• Project/Area Number: 26540131
• Research Category: Grant-in-Aid for Challenging Exploratory Research
• Allocation Type: Multi-year Fund
• Research Field: Intelligent robotics
• Research Institution: Saitama University
• Principal Investigator: KUNO Yoshinori 埼玉大学, 理工学研究科, 教授 (10252595)
• Co-Investigator(Kenkyū-buntansha): 小林 貴訓 埼玉大学, 理工学研究科, 准教授 (20466692)
• Project Period (FY): 2014-04-01 – 2017-03-31
• Project Status: Completed (Fiscal Year 2016)
• Budget Amount: ¥3,510,000 (Direct Cost: ¥2,700,000、Indirect Cost: ¥810,000); Fiscal Year 2016: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000); Fiscal Year 2015: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000); Fiscal Year 2014: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
• Keywords: コンピュータビジョン / 知能ロボット / ヒューマンロボットインタラクション / 認知症 / 感情認識 / 対話システム / 画像、文章、音声認識等 / 知能ロボティックス / ユーザインタフェース

Outline of Final Research Achievements

We have worked toward realizing a system to encourage elderly dementia patients living alone to actively communicate. We have developed a video communication system that enables the elderly to talk with their family members just by pushing a simple button. The system is also equipped with monitoring sensors that can send alarm email messages to family members if it senses a lack of activity in the elderly person’s room. In addition, a small robot is equipped at the elderly person’s side and can talk with the elderly person whenever his/her family members are unavailable to talk. We have also developed a method for recognizing emotions via facial expressions and estimating heart rate from facial video images.　We have obtained promising results through experiments that we can estimate the elderly’s emotional status from the information obtained by our method.

Research Products

• [Presentation] Detecting Inner Emotions from Video Based Heart Rate Sensing (2017)
  • Author(s): K. Das, S Ali, K. Otsu, H. Fukuda, A. Lam, Y. Kobayashi, and Y. Kuno
  • Organizer: International Conference on Intelligent Computing (ICIC2017)
  • Place of Presentation: Liverpool John Moores University (Liverpool, イギリス)
  • Year and Date: 2017-08-07
  • Int'l Joint Research
• [Presentation] 環境変化に頑健なビデオ映像による心拍数計測手法 (2017)
  • Author(s): 大津耕陽，倉橋知己，Das Keya Tilottoma, 福田悠人，Antony Lam, 小林貴訓，久野義徳
  • Organizer: 画像センシングシンポジウム(SSII2017)
  • Place of Presentation: パシフィコ横浜（神奈川県横浜市）
  • Year and Date: 2017-06-07
• [Presentation] 玄関での日常会話を援用した徘徊抑止システムの提案 (2017)
  • Author(s): 長嶺 洋佑，大津 耕陽，福田 悠人，小林 貴訓，久野 義徳
  • Organizer: 2017年電子情報通信学会総合大会
  • Place of Presentation: 名城大学天白キャンパス（愛知県名古屋市）
  • Year and Date: 2017-03-22
• [Presentation] Towards Estimating Heart Rates from Video under Low Light (2016)
  • Author(s): A. Lam and Y. Kuno, .
  • Organizer: International Conference on Visual Computing (ISVC2016)
  • Place of Presentation: MGM Grand Resort and Casino (Las Vegas, アメリカ合衆国)
  • Year and Date: 2016-12-12
  • Int'l Joint Research
• [Presentation] 遠隔ビデオ通話機能を備えた自律追従型見守りロボット (2016)
  • Author(s): Sidra Tariq，福田悠人，小林貴訓，久野義徳
  • Organizer: HAIシンポジウム
  • Place of Presentation: 東京大学駒場キャンパス（東京都目黒区）
  • Year and Date: 2016-12-03
• [Presentation] Remote Monitoring and Communication System with a Doll-like Robot for the Elderly (2016)
  • Author(s): K. Otsu, H. Fukuda, Y. Kobayashi, and Y. Kuno
  • Organizer: The 42nd Annual Conference of IEEE Industrial Electronics Society (IECON2016)
  • Place of Presentation: Palazzo dei Congressi (Firenze, イタリア)
  • Year and Date: 2016-10-24
  • Int'l Joint Research
• [Presentation] テレビ電話と連携した人形型デバイスを用いた高齢者の遠隔見守りシステム (2016)
  • Author(s): 大津耕陽，松田 成，福田悠人，小林貴訓，久野義徳
  • Organizer: 画像センシングシンポジウム(SSII2016)
  • Place of Presentation: パシフィコ横浜（神奈川県横浜市）
  • Year and Date: 2016-06-08
• [Presentation] テレビ電話を介した視聴覚情報によるロボットの遠隔操作 (2016)
  • Author(s): 水村育美, 鈴木亮太, 小林貴訓, 久野義徳
  • Organizer: 2016年電子情報通信学会総合大会
  • Place of Presentation: 九州大学伊都キャンパス（福岡県福岡市）
  • Year and Date: 2016-03-15
• [Presentation] 対話状況の観察に基づく高齢者遠隔見守りシステム (2016)
  • Author(s): 大津耕陽, 小林貴訓, 久野義徳
  • Organizer: 2016年電子情報通信学会総合大会
  • Place of Presentation: 九州大学伊都キャンパス（福岡県福岡市）
  • Year and Date: 2016-03-15
• [Presentation] Robust Heart Rate Measurement from Video Using Select Random Patches (2015)
  • Author(s): A. Lam and Y. Kuno
  • Organizer: International Conference on Computer Vision (ICCV2015)
  • Place of Presentation: Centro Parque Convention Center (Santiago, Chile)
  • Year and Date: 2015-12-11
  • Int'l Joint Research
• [Presentation] 高齢者の状態センシングに基づく遠隔見守りシステム (2015)
  • Author(s): 大津耕陽，小林貴訓，久野義徳
  • Organizer: 第20回パターン計測シンポジウム
  • Place of Presentation: 湖邸滋びわこクラブ（滋賀県大津市）
  • Year and Date: 2015-11-19
• [Presentation] 認知症介護支援のためのキーワード検出に基づく対話システム (2015)
  • Author(s): 堀江 直人，小林 貴訓，久野 義徳
  • Organizer: 第20回パターン計測シンポジウム
  • Place of Presentation: 湖邸滋びわこクラブ（滋賀県大津市）
  • Year and Date: 2015-11-19
• [Presentation] Remote Communication Support System as Communication Catalyst for Dementia Care (2015)
  • Author(s): S. Goto, Y. Matsuda, T. Kikugawa, Y. Kobayashi, and Y. Kuno
  • Organizer: The 41st Annual Conference of the IEEE Industrial Electronics Society
  • Place of Presentation: パシフィコ横浜（神奈川県横浜市）
  • Year and Date: 2015-11-09
  • Int'l Joint Research
• [Presentation] Toward a Robot System Supporting Communication between People with Dementia and Their Relatives (2015)
  • Author(s): Y. Kuno, S. Goto, Y. Matsuda, T. Kikugawa, A. Lam and Y. Kobayashi
  • Organizer: International Conference on Intelligent Robots and Systems (IROS2015)
  • Place of Presentation: Congress Center Hamburg (Hamburg, Germany)
  • Year and Date: 2015-09-28
  • Int'l Joint Research
• [Presentation] Facial Expression Recognition Based on Hybrid Approach (2015)
  • Author(s): M.A, Mannan, A. Lam, Y. Kobayashi, and Y. Kuno
  • Organizer: International Conference on Intelligent Computing (ICIC2015)
  • Place of Presentation: Fu Jian Hai Lian Business Hotel (Fuzhou, China)
  • Year and Date: 2015-08-20
  • Int'l Joint Research
• [Presentation] 認知症介護を支援する遠隔コミュニケーションシステム (2015)
  • Author(s): 松田成, 小林貴訓, 久野義徳
  • Organizer: 第21回画像センシングシンポジウム(SSII2015)
  • Place of Presentation: パシフィコ横浜（神奈川県横浜市）
  • Year and Date: 2015-06-10
• [Presentation] Robotic Systems Based on Sociological Interaction Analysis (2015)
  • Author(s): Yoshinori Kuno
  • Organizer: The 2nd International Conference on Electrical Engineering and Information & Communication Technology (iCEEiCT2015)
  • Place of Presentation: Jahangirnagar University (Dhaka, Bangladesh)
  • Year and Date: 2015-05-21
  • Int'l Joint Research / Invited
• [Presentation] エージェント対話システムのための高速カメラを用いた表情認識 (2015)
  • Author(s): 秋山 俊貴、小林 貴訓、久野 義徳
  • Organizer: 2015年電子情報通信学会総合大会
  • Place of Presentation: 立命館大学びわこ・くさつキャンパス（滋賀県草津市）
  • Year and Date: 2015-03-10 – 2015-03-13
• [Presentation] ロボットを用いた遠隔コミュニケーションシステム (2015)
  • Author(s): 菊川 俊樹、小林 貴訓、久野 義徳
  • Organizer: 2015年電子情報通信学会総合大会
  • Place of Presentation: 立命館大学びわこ・くさつキャンパス（滋賀県草津市）
  • Year and Date: 2015-03-10 – 2015-03-13
• [Presentation] Assistive Robots Based on Observation of Humans (2015)
  • Author(s): Yoshinori Kuno
  • Organizer: 21st Japan-Korea Joint Workshop on Frontiers of Computer Vision (FCV2015)
  • Place of Presentation: Shinan Beach Hotel （Mokpo, 大韓民国)
  • Year and Date: 2015-01-28 – 2015-01-30
  • Invited
• [Presentation] Assistive Robots Based on Sociological Interaction Analysis (2014)
  • Author(s): Yoshinori Kuno
  • Organizer: 9th International Forum on Strategic Technology(IFOST2014)
  • Place of Presentation: Long Beach Hotel (Cox’s Bazar, バングラデシュ)
  • Year and Date: 2014-10-21 – 2014-10-23
  • Invited
• [Patent(Industrial Property Rights)] テレビ電話によるロボットの遠隔制御システム (2016)
  • Inventor(s): 小林貴訓, 久野義徳
  • Industrial Property Rights Holder: 埼玉大学
  • Industrial Property Rights Type: 特許
  • Industrial Property Number: 2016-030525
  • Filing Date: 2016-02-21
• [Patent(Industrial Property Rights)] 心拍に連動する周期的変動の計測装置及び計測方法 (2015)
  • Inventor(s): Antony Lam, 久野義徳
  • Industrial Property Rights Holder: 埼玉大学
  • Industrial Property Rights Type: 特許
  • Industrial Property Number: 2015-228248
  • Filing Date: 2015-11-22
• [Patent(Industrial Property Rights)] 見守りシステム (2014)
  • Inventor(s): 久野 義徳、小林 貴訓、杉山 一英、興津 勝彦
  • Industrial Property Rights Holder: 久野 義徳、小林 貴訓、杉山 一英、興津 勝彦
  • Industrial Property Rights Type: 特許
  • Industrial Property Number: 2014-234061
  • Filing Date: 2014-11-18