Discovery of Conversational Errors by Comparison between Turn Taking and Brain Activity during Remote Collaboration

• Project/Area Number: 26730079
• Research Category: Grant-in-Aid for Young Scientists (B)
• Allocation Type: Multi-year Fund
• Research Field: Cognitive science
• Research Institution: Nihon University
• Principal Investigator: OYAMA Katsunori 日本大学, 工学部, 准教授 (50615606)
• Research Collaborator: Chang Carl K. Iowa State University, Department of Computer Science, 教授 ; WATANABE Hiroyuki 日本大学, 工学部, 教授 (40147658)
• Project Period (FY): 2014-04-01 – 2016-03-31
• Project Status: Completed (Fiscal Year 2015)
• Budget Amount: ¥3,770,000 (Direct Cost: ¥2,900,000、Indirect Cost: ¥870,000); Fiscal Year 2015: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000); Fiscal Year 2014: ¥2,600,000 (Direct Cost: ¥2,000,000、Indirect Cost: ¥600,000)
• Keywords: EEG / NIRS / BCI / 自己組織化マップ / 意思疎通 / 暗算課題 / 記憶課題 / クラスの種類 / EEGとNIRSの同時計測 / 対話課題 / 意思疎通の失敗

Outline of Final Research Achievements

This study conducted the analysis on conversational errors, i.e., falling into silence for more than 1 minute as undesirable situation in collaborative environments.
1) This study compared between temporal changes in frequency of turn taking and variation of brain waves during the exercise of pair programming. From the experimental result, variation of brain waves is larger at the point the transition from active conversation to falling into silence. In other words, the effects on brain activity by conversational errors between the collaborators can be visualized by the index, variation of brain waves.
2) This study also conducted the experiment of mental arithmetic task by simultaneous EEG and NIRS monitoring, and classified the brain states using Self-Organizing Maps (SOMs). From the experimental result, our discussion concluded that not only the states of calculating and resting, but also the states of eye-closing, listening and writing are classifiable.

Research Products

• [Int'l Joint Research] Iowa State University/Oakland University(米国)
• [Presentation] Hierarchical Self-Organizing Maps of NIRS and EEG Signals for Recognition of Brain States (2016)
  • Author(s): Katsunori Oyama, Kaoru Sakatani, Hua Ming, Carl K. Chang
  • Organizer: 14th International Conference on Smart homes, Assistive Technologies, Robotics and Health telematics (ICOST)
  • Place of Presentation: Wuhan, China
  • Year and Date: 2016-05-25
  • Int'l Joint Research
• [Presentation] 暗算課題における脳波と脳血流の特徴量の比較 (2015)
  • Author(s): 大山勝徳, 西園敏弘, 酒谷薫
  • Organizer: 第49回日本生体医工学会東北支部大会
  • Place of Presentation: 日本大学工学部 次世代工学技術研究センター
  • Year and Date: 2015-11-21
• [Presentation] Temporal Comparison between NIRS and EEG Signals during Mental Arithmetic Task Evaluated by Self-Organizing Map (2015)
  • Author(s): Katsunori Oyama, Kaoru Sakatani
  • Organizer: 43rd Annual Meeting of the International Society on Oxygen Transport to Tissue (ISOTT)
  • Place of Presentation: Wuhan, China
  • Year and Date: 2015-07-11
  • Int'l Joint Research
• [Presentation] Visualization of turn-taking and mental workload in collaborative working environment (2014)
  • Author(s): Katsunori Oyama, Hiroyuki Watanabe and Atsushi Takeuchi
  • Organizer: IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD)
  • Place of Presentation: Hsinchu, Taiwan
  • Year and Date: 2014-05-21 – 2014-05-23