Development of BMIs using facial images and sounds for the smooth communication with persons with disabilities

• Project/Area Number: 18K17667
• Research Category: Grant-in-Aid for Early-Career Scientists
• Allocation Type: Multi-year Fund
• Review Section: Basic Section 59010:Rehabilitation science-related
• Research Institution: Kagawa National College of Technology (2020); Chiba University (2018-2019)
• Principal Investigator: Onishi Akinari 香川高等専門学校, 電子システム工学科, 助教 (20747969)
• Project Period (FY): 2018-04-01 – 2021-03-31
• Project Status: Completed (Fiscal Year 2020)
• Budget Amount: ¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000); Fiscal Year 2020: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000); Fiscal Year 2019: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000); Fiscal Year 2018: ¥2,730,000 (Direct Cost: ¥2,100,000、Indirect Cost: ¥630,000)
• Keywords: BMI / BCI / 脳波 / P300 / 視聴覚刺激 / 顔 / 音声 / ブレイン-マシン・インタフェース(BMI) / ユーザインタフェース / ディープラーニング / 畳み込みニューラルネットワーク(CNN)

Outline of Final Research Achievements

Brain-machine interface (BMI) converts brain signals into control commands for external devices. A recent study indicated performance improvement of the audiovisual BMI. However, is it caused by the integration effect of multimodal stimuli? In addition, is it also influenced by the facial images or affective stimuli? This study proposed a gaze-independent BMI that has facial images and artificial voice, and conducted an experiment that was controlled the stimulus intensity. The results implied that the performance improvement caused by the audiovisual stimuli was due to the integration effect of the stimuli. However, affective audiovisual stimuli did not show significant improvement. The online classification accuracy of the BMI was 85.71±11.50%.

Academic Significance and Societal Importance of the Research Achievements

BMIは手足が不自由な方が福祉機器を制御するための新しい手法として着目されている．従来のBMIは視線の制御により性能が左右されるものであったため視線を自由に動かせない患者に適さない問題があった．本研究では視線の移動を必要としないBMIに視聴覚刺激を導入し，それにより精度が向上することを明らかにした．本研究で開発したBMIを用いることで手足が不自由で，かつ視線を自由に動かすことが難しい方が使えるようなBMIを開発することが可能となり，国民の生活の質の向上に貢献すると考える．

Research Products

• [Journal Article] Convolutional Neural Network Transfer Learning Applied to the Affective Auditory P300-Based BCI (2020)
  • Author(s): Onishi Akinari、Chiba University 1-33 Yayoicho, Inage-ku, Chiba-shi, Chiba 263-8522, Japan、National Institute of Technology, Kagawa College 551 Kohda, Takuma-cho, Mitoyo-shi, Kagawa 769-1192, Japan
  • Journal Title: Journal of Robotics and Mechatronics Volume: 32 Issue: 4 Pages: 731-737
  • DOI: 10.20965/jrm.2020.p0731
  • NAID: 130007888802
  • ISSN: 0915-3942, 1883-8049
  • Year and Date: 2020-08-20
  • Peer Reviewed / Open Access
• [Journal Article] How Does the Degree of Valence Influence Affective Auditory P300-Based BCIs? (2019)
  • Author(s): Onishi Akinari、Nakagawa Seiji
  • Journal Title: Frontiers in Neuroscience Volume: 13 Pages: 1-8
  • DOI: 10.3389/fnins.2019.00045
  • Peer Reviewed / Open Access
• [Presentation] 脳波による走行ロボットの遠隔制御に関する研究 (2021)
  • Author(s): 森岡大介，大西章也
  • Organizer: 第26回高専シンポジウムオンライン
• [Presentation] Comparison of Classifiers for the Transfer Learning of Affective Auditory P300-Based BCIs (2019)
  • Author(s): Onishi Akinari, Nakagawa Seiji
  • Organizer: 41th Annual International Conference of the IEEE Engineering in Medicine and Biology Society
  • Int'l Joint Research
• [Presentation] Within- and Between-Subject Classification of the Affective Auditory P300-based Brain-Computer Interface (2019)
  • Author(s): Onishi Akinari, Nakagawa Seiji
  • Organizer: International Symposium on Info Comm and Mechatronics Technology in Bio-Medical and Healthcare Application (IS 3T-in-3A)
  • Int'l Joint Research
• [Presentation] Ensemble Convoluted Feature Extraction for Affective Auditory P300 Brain-Computer Interfaces (2018)
  • Author(s): A. Onishi, S. Nakagawa
  • Organizer: The 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society
  • Int'l Joint Research