Study on the hierarchy of cross-modal processing using functional MRI and Deep Learning

2021 Fiscal Year Final Research Report

• Project/Area Number: 19K20390
• Research Category: Grant-in-Aid for Early-Career Scientists
• Allocation Type: Multi-year Fund
• Review Section: Basic Section 61060:Kansei informatics-related
• Research Institution: National Institute for Physiological Sciences
• Principal Investigator: PHAM QUANG TRUNG 生理学研究所, システム脳科学研究領域, 特任研究員 (60837722)
• Project Period (FY): 2019-04-01 – 2022-03-31
• Keywords: cognitive science / somatosensory / haptics / perception

Outline of Final Research Achievements

In this study, I proposed a hypothesis where the fragments of haptic information are integrated into visual imagery at the intraparietal sulcus (IPS) and lateral occipital cortex (LOC) before passing to the visual ventral pathway for object recognition. For examining that hypothesis, I designed an experiment where the participants try to recognize the 3x5 dot-digits (from 0 to 9) from visual and haptic stimuli. During the experiment, I captured their brain signals using an MRI scanner. Behavioral analysis revealed that most participants can recognize the dot-digit above the chance level. Their visual performance is better than haptic performance. The fMRI analysis confirmed the involvements of IPS and LOC, and the dorsolateral prefrontal cortex (dlPFC) which is related to working-memory. I performed a representational analysis and found similar representations for both haptic and visual information at the IPS and LOC.

Free Research Field

computational neuroscience

Academic Significance and Societal Importance of the Research Achievements

The results of this study extend our current understanding of the cross-modal processing of haptic and visual information in the brain. In the future, I expected these results to become hint for building a more efficient robotics system that mimics human brain.