• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Deep Learning the Human Mind - Recognising and Augmenting Cognitive Performance Fluctuations

Research Project

Project/Area Number 18H03278
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypeSingle-year Grants
Section一般
Review Section Basic Section 61020:Human interface and interaction-related
Research InstitutionKeio University

Principal Investigator

Kunze Kai  慶應義塾大学, メディアデザイン研究科(日吉), 教授 (00648040)

Project Period (FY) 2018-04-01 – 2022-03-31
Project Status Completed (Fiscal Year 2021)
Budget Amount *help
¥16,640,000 (Direct Cost: ¥12,800,000、Indirect Cost: ¥3,840,000)
Fiscal Year 2021: ¥3,380,000 (Direct Cost: ¥2,600,000、Indirect Cost: ¥780,000)
Fiscal Year 2020: ¥4,030,000 (Direct Cost: ¥3,100,000、Indirect Cost: ¥930,000)
Fiscal Year 2019: ¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2018: ¥4,940,000 (Direct Cost: ¥3,800,000、Indirect Cost: ¥1,140,000)
KeywordsCognitive Augmentation / Wearable Computing / Adaptable Interfaces / Cognitive Fluctuations / Virtual Reality / Remote Work / Liveness / Lectures / cognitive fluctuations / mental fatigue detection / daytime sleepiness / HCI / computer integration / Behavior Analysis / Cognitive Recognition / hci / cognitive recognition / behavior modeling / ヒューマンインターフェース / ヒューマンインタフェース / ウェアラブル機器 / ヒューマンインターフェイス
Outline of Final Research Achievements

This project provides tools to quantify and manage properties of our mind better using physiological sensing (from sleep, over cognitive load, towards problems related to motion/cyber-sickness and accessibility issues). In a first step, we implemented several methods to assess cognitive fluctuations using unobtrusive physiological sensing and deep learning/machine learning techniques. Highlights include the detection of sleep onset, sleep quality, alertness (while driving), and cognitive load changes. In a second step, we implemented interactions to improve or manage these cognitive fluctuations, highlights include a nap assistant, Increased situational awareness while driving, and a reduction in motion- cyber sickness.
We also developed two novel paradigms: Cognitive Augmentation and Human Computer Integration, for implementing systems that are more integrated with the human body. Overall, we present significant contributions to the human computer interaction and related fields.

Academic Significance and Societal Importance of the Research Achievements

Our project introduces the novel paradigm of cognitive augmentation. We enable a tighter feedback loop taking the user’s real time cognition model into account. Making systems more accessible. Accessibility is a core concept emphasized in nearly all 17 of the UN’s SDGs.

Report

(5 results)
  • 2021 Annual Research Report   Final Research Report ( PDF )
  • 2020 Annual Research Report
  • 2019 Annual Research Report
  • 2018 Annual Research Report
  • Research Products

    (27 results)

All 2022 2021 2020 2019 2018

All Journal Article (18 results) (of which Int'l Joint Research: 15 results,  Peer Reviewed: 17 results) Presentation (8 results) (of which Int'l Joint Research: 2 results,  Invited: 5 results) Funded Workshop (1 results)

  • [Journal Article] ThermalDrive - Towards Situation Awareness over Thermal Feedback in Automated Driving Scenarios2022

    • Author(s)
      Meng Xiaru、Han Jiawen、Chernyshov George、Ragozin Kirill、Kunze Kai
    • Journal Title

      IIUI '22 Companion: 27th International Conference on Intelligent User Interfaces

      Volume: 1 Pages: 1-4

    • DOI

      10.1145/3490100.3516453

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] NapWell: An EOG-based Sleep Assistant Exploring the Effects of Virtual Reality on Sleep Onset2021

    • Author(s)
      Pai Yun Suen、Bait Marsel L.、Lee Juyoung、Xu Jingjing、Peiris Roshan L、Woo Woontack、Billinghurst Mark、Kunze Kai
    • Journal Title

      Virtual Reality

      Volume: 1 Issue: 2 Pages: 1-15

    • DOI

      10.1007/s10055-021-00571-w

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Tunnel Vision - Dynamic Peripheral Vision Blocking Glasses for Reducing Motion Sickness Symptoms2021

    • Author(s)
      Zhang Qing、Yamamura Hiroo、Baldauf Holger、Zheng Dingding、Chen Kanyu、Yamaoka Junichi、Kunze Kai
    • Journal Title

      Proceedings of the International Symposium on Wearable Computing

      Volume: 1 Pages: 1-8

    • DOI

      10.1145/3460421.3478824

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] ThermoQuest - A Wearable Head Mounted Display to Augment Realities with Thermal Feedback2021

    • Author(s)
      Ragozin Kirill、Meng Xiaru、Lalintha Peiris Roshan、Wolf Katrin、Chernyshov George、Kunze Kai
    • Journal Title

      Proceedings of the International Conference on Mobile and Ubiquitous Multimedia

      Volume: 1 Pages: 1-4

    • DOI

      10.1145/3490632.3490649

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] JINSense: Repurposing Electrooculography Sensors on Smart Glass for Midair Gesture and Context Sensing2021

    • Author(s)
      Yeo Hui-Shyong、Lee Juyoung、Woo Woontack、Koike Hideki、Quigley Aaron J、Kunze Kai
    • Journal Title

      Extended Abstracts of the 2021 CHI Conference

      Volume: 1 Pages: 1-4

    • DOI

      10.1145/3411763.3451741

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] 25 Years of ISWC: Time Flies When You’re Having Fun2021

    • Author(s)
      Martin Tom、Starner Thad、Siewiorek Dan、Kunze Kai、Van Laerhoven Kristof、Amft Oliver、Laerhoven Kristof Van
    • Journal Title

      IEEE Pervasive Computing

      Volume: 20 Issue: 3 Pages: 72-78

    • DOI

      10.1109/mprv.2021.3094898

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] HemodynamicVR - Adapting the User’s Field Of View duringVirtual Reality Locomotion Tasks to Reduce Cybersickness usingWearable Functional Near-Infrared Spectroscopy2021

    • Author(s)
      HirooYamamura, HolgerBaldauf, KaiKunze
    • Journal Title

      Augmented Humans

      Volume: 1 Pages: 1-5

    • DOI

      10.1145/3458709.3458994

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Next Steps in Human-Computer Integration2020

    • Author(s)
      Florian ‘Floyd’ Mueller, Pedro Lopes, Paul Strohmaier, Kai Kunze et. al.
    • Journal Title

      CHI 2020 Human Factors in Computing Systems

      Volume: 1 Pages: 1-15

    • DOI

      10.1145/3313831.3376242

    • Related Report
      2020 Annual Research Report 2019 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] GenVibe: Exploration of Interactive Generation of PersonalVibrotactile Patterns2020

    • Author(s)
      Pescara Erik、Dreschner Florian、Marky Karola、Kunze Kai、Beigl Michael
    • Journal Title

      Augmented Humans International Conference

      Volume: 1 Pages: 1-9

    • DOI

      10.1145/3384657.3384794

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Pleasant Locomotion--Towards Reducing Cybersickness using fNIRS during Walking Events in VR2020

    • Author(s)
      HirooYamamura, HolgerBaldauf, KaiKunze
    • Journal Title

      Adjunct Publication of the 33rd Annual ACM Symposium on User Interface Software and Technology.

      Volume: 1 Pages: 1-4

    • DOI

      10.1145/3379350.3416184

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Face Commands - User-Defined Facial Gestures for Smart Glasses2020

    • Author(s)
      Masai Katsutoshi、Kunze Kai、Sakamoto Daisuke、Sugiura Yuta、Sugimoto Maki
    • Journal Title

      ISMAR

      Volume: 1 Pages: 1-13

    • DOI

      10.1109/ismar50242.2020.00064

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] MazeRunVR: An Open Benchmark for VR Locomotion Performance, Preference and Sickness in the Wild2020

    • Author(s)
      Ragozin Kirill、Kunze Kai、Marky Karola、Pai Yun Suen
    • Journal Title

      CHI Extended Abstracts

      Volume: 1 Pages: 1-6

    • DOI

      10.1145/3334480.3383035

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Sleepy watch2020

    • Author(s)
      Bao Jie、Han Jiawen、Kato Akira、Kunze Kai
    • Journal Title

      UbiComp Adjunct

      Volume: 1 Pages: 1-4

    • DOI

      10.1145/3410530.3414415

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Blink as you sync uncovering eye and nod synchrony in conversation using wearable sensing2019

    • Author(s)
      Aman Gupta, Finn L Strivens, Benjamin Tag, Kai Kunze and Jamie A Ward
    • Journal Title

      International Symposium on Wearable Computers

      Volume: 3 Pages: 1-6

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Eyewear 2019 third workshop on eyewear computing - focus: social interactions2019

    • Author(s)
      Benjamin Tag, Jamie A Ward, Yuji Uema and Kai Kunze
    • Journal Title

      UbiComp Adjunct Proceedings

      Volume: 2 Pages: 1-3

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] EOG Glasses an Eyewear Platform for Cognitive and Social Interaction Assessments in the Wild2019

    • Author(s)
      George Chernyshov, Kirill Ragozin, Benjamin Tag and Kai Kunze
    • Journal Title

      International Conference on Human-Computer Interaction with Mobile Devices and Services

      Volume: 4 Pages: 1-5

    • DOI

      10.1145/3338286.3344418

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Demo: http://eyewear.pro An Open Platform to Record and Analyze Large Scale Data Sets from Smart Eyewear2019

    • Author(s)
      George Chernyshov, Kai Kunze, Benjamin Tag, Jamie A Ward and Yuji Uema
    • Journal Title

      UbiComp Adjunct

      Volume: 2 Pages: 1-2

    • DOI

      10.1145/3341162.3343790

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Assessing hands-free interactions for VR using eye gaze and electromyography2018

    • Author(s)
      Yun Suen Pai, Tilman Dingler, Kai Kunze
    • Journal Title

      Virtual Reality

      Volume: 12 Issue: 2 Pages: 1-13

    • DOI

      10.1007/s10055-018-0371-2

    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Augmenting the Human Mind using Physiological Sensing and Deep Learning2021

    • Author(s)
      Kai Kunze
    • Organizer
      SIGHInt: Special Interest Group for Human-Computer Integration
    • Related Report
      2021 Annual Research Report
    • Invited
  • [Presentation] Deep Learning for Mind and Body2021

    • Author(s)
      Kai Kunze
    • Organizer
      Body As Starting Point 4: Inbodied Interaction Design for Health Ownership.
    • Related Report
      2021 Annual Research Report
    • Invited
  • [Presentation] Eyewear to understand cognitive fluctuations2021

    • Author(s)
      Kai Kunze
    • Organizer
      Eyewear 2021 The Forth Workshop on Eyewear Computing
    • Related Report
      2021 Annual Research Report
    • Invited
  • [Presentation] Vibrothermal Haptics of a Smart Fidget Surface for Remote Workers2021

    • Author(s)
      Kai Kunze
    • Organizer
      Poster Session: UbiComp Conference
    • Related Report
      2021 Annual Research Report
  • [Presentation] Augmenting the Human Mind2021

    • Author(s)
      Kai Kunze
    • Organizer
      Neurolive Symposium
    • Related Report
      2020 Annual Research Report
    • Invited
  • [Presentation] Measuring Cognitive Fluctuations2020

    • Author(s)
      Kai Kunze
    • Organizer
      Insorcing Workshop collocated with CHI
    • Related Report
      2020 Annual Research Report
    • Invited
  • [Presentation] Virtual Gaze: Exploring Use of Gaze As Rich Interaction Method with Virtual Agent in Interactive Virtual Reality Content2018

    • Author(s)
      Kevin, Stevanus and Pai, Yun Suen and Kunze, Kai
    • Organizer
      Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] UbiTrain: Leveraging the Physical and Virtual Environment for Ubiquitous Sports Training2018

    • Author(s)
      Pai, Yun Suen and Isogai, Megumi and Kunze, Kai and Nakao, Takuro and Kimata, Hideaki
    • Organizer
      roceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Funded Workshop] Eyewear 2019 third workshop on eyewear computing collocated with UbiComp2019

    • Related Report
      2019 Annual Research Report

URL: 

Published: 2018-04-23   Modified: 2023-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi