2020 Fiscal Year Annual Research Report
Deep Learning the Human Mind - Recognising and Augmenting Cognitive Performance Fluctuations
| Project/Area Number |
18H03278
|
|---|
| Research Institution | Keio University |
|---|
Principal Investigator |
クンツェ カイ 慶應義塾大学, メディアデザイン研究科(日吉), 教授 (00648040)
|
|---|
| Project Period (FY) |
2018-04-01 – 2022-03-31
|
| Keywords | cognitive fluctuations / mental fatigue detection / Virtual Reality / daytime sleepiness / HCI / computer integration |
|---|
| Outline of Annual Research Achievements |
We shifted the recognition of cognitive fluctuations from real world situations (lectures, meetings, conferences, performances) to mostly online and virtual reality applications due to the pandemic. We can detect cognitive fluctuations related to cybersickness (mental fatigue) in VR. We collected a larger scale data set in virtual environments recording motions and oxyhemoglobin change using FNRIS.
We show that in-ear and wrist temperature can determine daytime sleepiness. The datasets and models are in preparation to be open-sourced (some already used from collaborators). The concept of “Deep Learning the Mind” is a core contribution for a future direction of human computer interaction coined “Human Computer Integration” (CHI paper and SIGCHI special interest group on the topic).
The modeling effort is a bit behind in terms of user independent models (as we cannot get that many users for the experiments), yet given to the rest of the project proceeding better than expected and the additional output, this is not a major issue. User-dependent models work also better than expected.
|
| Current Status of Research Progress |
Current Status of Research Progress
1: Research has progressed more than it was originally planned.
Reason
Even though we encountered trouble in dataset recordings, as the large scale experiments (except the first at UbiComp 2019) could not be executed ,we are on track with the experiments shifting to individual fluctuations (daytime sleepiness and alertness modeling for the everyday use case and cybersickness/mental fatigue for virtual environments). Also we contributed a larger open benchmark for VR locomotion and a novel concept for the future research directions in Human Computer Interaction.
The modeling effort is a bit behind in terms of user independent models (as we cannot get that many users for the experiments), yet given to the rest of the project proceeding better than expected and the additional output, this is not a major issue. User-dependent models work also better than expected.
Dagstuhl Seminar for Cognitive Augmentation is accepted and will be held December 2022.
|
| Strategy for Future Research Activity |
Dataset / Model /Source Code Release Preparations. We will make the datasets for open benchmark for VR locomotion methods open to other researchers (cognitive evaluation of different locomotion methods) and extend its validity to other fields. As well as the daily fatigue recordings.
Evaluate Cognitive State Models on Novel Hardware (Physiological Sensing). Additionally, we move towards using a novel physiological sensing platform (wrist based, recording EDA, heart rate and temperature). We will apply the cognitive state detection models to this new sensing platform.
We will record several datasets (already in plan and started) focusing on attention in the classroom setting.
Finish the attention modeling for 2021 and work on a major journal publications as well as other ways of dissemination the project results. Goal: release an analysis system to detect attention fluctuations in real world remote lecture scenarios. As mentioned the Dagstuhl Seminar for Cognitive Augmentation is accepted and will be held December 2022. This is will be a perfect opportunity to disseminate the project results.
|
