2018 Fiscal Year Annual Research Report
Multimodal Deep Learning Framework for Intelligent Brain Computer Interface System
| Project/Area Number |
17K13279
|
|---|
| Research Institution | Advanced Telecommunications Research Institute International |
|---|
Principal Investigator |
Penaloza C. 株式会社国際電気通信基礎技術研究所, 石黒浩特別研究所, 研究員 (80753532)
|
|---|
| Project Period (FY) |
2017-04-01 – 2019-03-31
|
| Keywords | Brain Machine Interface / Brain Computer Interface / Robot Arm / EEG |
|---|
| Outline of Annual Research Achievements |
We developed a BMI system that incorporates a multimodal approach to learn the correlation of the context of a task, visual sensory data, and the brain data. The platform to test this system consisted of a human-like robotic arm controlled with a BMI that has context-aware capabilities in order to complement BMI-based commands and increase the number of actions that it can perform with the same BMI-based command. The proposed arm can be activated (i.e. grasp action) with a non-invasive EEG-based BMI when the human operator imagines the action. Since there are different ways that the SRL can perform the action (i.e. different grasping configurations) depending on the context (i.e. type of the object), we provided vision capabilities to the SRL so it can recognize the context and optimize its behavior in order to match the user intention.
Moreover, we proposed a method to decode visual representations of the objects from brain data towards improving robot arm grasp configurations. More specifically, we recorded EEG data during an object-grasping experimen and developed a multimodal representation of the encoded brain data and object image Given this representation, the objective was to reconstruct the image given that only half of the image (the brain data encoding) was provided. To achieve this goal, we developed a deep stacked convolutional autoencoder that learned a noise-free joint manifold of brain data encoding and object image. After training, the autoencoder was able to reconstruct the missing part of the object image given that only brain data encoding was provided
|
