• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Data-mining of shitsukan representation using brain, image, and text data

Planned

Project AreaUnderstanding human recognition of material properties for innovation in SHITSUKAN science and technology
Project/Area Number 15H05920
Research Category

Grant-in-Aid for Scientific Research on Innovative Areas (Research in a proposed research area)

Allocation TypeSingle-year Grants
Review Section Complex systems
Research InstitutionAdvanced Telecommunications Research Institute International

Principal Investigator

Kamitani Yukiyasu  株式会社国際電気通信基礎技術研究所, 脳情報通信総合研究所, 研究室長 (50418513)

Project Period (FY) 2015-06-29 – 2020-03-31
Project Status Completed (Fiscal Year 2019)
Budget Amount *help
¥57,850,000 (Direct Cost: ¥44,500,000、Indirect Cost: ¥13,350,000)
Fiscal Year 2019: ¥11,310,000 (Direct Cost: ¥8,700,000、Indirect Cost: ¥2,610,000)
Fiscal Year 2018: ¥11,050,000 (Direct Cost: ¥8,500,000、Indirect Cost: ¥2,550,000)
Fiscal Year 2017: ¥11,050,000 (Direct Cost: ¥8,500,000、Indirect Cost: ¥2,550,000)
Fiscal Year 2016: ¥11,830,000 (Direct Cost: ¥9,100,000、Indirect Cost: ¥2,730,000)
Fiscal Year 2015: ¥12,610,000 (Direct Cost: ¥9,700,000、Indirect Cost: ¥2,910,000)
Keywordsブレイン・デコーディング / 視覚 / ブレイン・マシン・インターフェース / fMRI / 深層ニューラルネットワーク / 質感 / 脳情報デコーディング / 認知科学 / 認知神経科学 / 機械学習
Outline of Final Research Achievements

We established a data-mining-based approach to discover and utilize the representation of shitsukan information in the human brain. We developed a method to predict deep neural network (DNN) features of an image from the induced brain activity pattern (brain-to-DNN signal translation), and found hierarchical homology between the brain and DNN. We also developed methods for reconstructing the viewed image using DNN features predicted from brain activity, and succeeded in re-creating texture and material images only from the brain activity measured while they were viewed.

Academic Significance and Societal Importance of the Research Achievements

本プロジェクトの成果は、質感を含む知覚内容を今までにない精度で脳活動から解読できることを実証するとともに、脳-DNN間の信号変換技術にもとづくブレイン・デコーディングのポテンシャルを示している。同様のアプローチは、画像だけでなく、動画や音声、テキストなど、さまざまな刺激に対する脳活動からの情報解読に応用可能で、ブレイン-マシン・インターフェイスの技術の開発にも貢献することが期待される。また、脳との階層的相同性を定量化するBHスコアは、今後の脳型AI開発の指針となると考えられる。

Report

(6 results)
  • 2019 Annual Research Report   Final Research Report ( PDF )
  • 2018 Annual Research Report
  • 2017 Annual Research Report
  • 2016 Annual Research Report
  • 2015 Annual Research Report

Research Products

(43 results)

All 2019 2018 2017 2016 2015 Other

All Journal Article (12 results) (of which Peer Reviewed: 12 results,  Open Access: 12 results,  Acknowledgement Compliant: 5 results) Presentation (29 results) (of which Int'l Joint Research: 20 results,  Invited: 14 results) Remarks (2 results)

  • [Journal Article] End-to-End Deep Image Reconstruction From Human Brain Activity2019

    • Author(s)
      Shen Guohua、Dwivedi Kshitij、Majima Kei、Horikawa Tomoyasu、Kamitani Yukiyasu
    • Journal Title

      Frontiers in Computational Neuroscience

      Volume: 13 Pages: 1-11

    • DOI

      10.3389/fncom.2019.00021

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Deep image reconstruction from human brain activity2019

    • Author(s)
      Shen Guohua、Horikawa Tomoyasu、Majima Kei、Kamitani Yukiyasu
    • Journal Title

      PLOS Computational Biology

      Volume: 15 Pages: 1-23

    • DOI

      10.1371/journal.pcbi.1006633

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Characterization of deep neural network features by decodability from human brain activity2019

    • Author(s)
      Horikawa Tomoyasu、Aoki Shuntaro C.、Tsukamoto Mitsuaki、Kamitani Yukiyasu
    • Journal Title

      Scientific Data

      Volume: 6 Pages: 1-12

    • DOI

      10.1038/sdata.2019.12

    • NAID

      120006713554

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Sparse Ordinal Logistic Regression and Its Application to Brain Decoding2018

    • Author(s)
      Emi Satake, Kei Majima, Shuntaro C. Aoki, Yukiyasu Kamitani
    • Journal Title

      Frontiers in Neuroinformatics

      Volume: 12 Pages: 1-10

    • DOI

      10.3389/fninf.2018.00051

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Sharpening of Hierarchical Visual Feature Representations of Blurred Images2018

    • Author(s)
      Mohamed Abdelhack, Yukiyasu Kamitani
    • Journal Title

      eNeuro

      Volume: 5 Pages: 1-15

    • DOI

      10.1523/eneuro.0443-17.2018

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Heterogeneous Redistribution of Facial Subcategory Information Within and Outside the Face-Selective Domain in Primate Inferior Temporal Cortex2018

    • Author(s)
      Miyakawa Naohisa、Majima Kei、Sawahata Hirohito、Kawasaki Keisuke、Matsuo Takeshi、Kotake Naoki、Suzuki Takafumi、Kamitani Yukiyasu、Hasegawa Isao
    • Journal Title

      CEREBRAL CORTEX

      Volume: 28 Pages: 1416-1431

    • DOI

      10.1093/cercor/bhx342

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Generic decoding of seen and imagined objects using hierarchical visual features2017

    • Author(s)
      Horikawa Tomoyasu、Kamitani Yukiyasu
    • Journal Title

      Nature Communications

      Volume: 8 Pages: 15037-15037

    • DOI

      10.1038/ncomms15037

    • NAID

      120006305983

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Generic decoding of seen and imagined objects using hierarchical visual features2017

    • Author(s)
      Horikawa T, Kamitani Y
    • Journal Title

      Nature communications

      Volume: 印刷中

    • NAID

      120006305983

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Open Access / Acknowledgement Compliant
  • [Journal Article] Hierarchical neural representations of dreamed objects revealed by brain decoding with deep neural network features2017

    • Author(s)
      Horikawa T, Kamitani Y
    • Journal Title

      Frontiers in computational neuroscience

      Volume: 11: 4.

    • DOI

      10.3389/fncom.2017.00004

    • NAID

      120005980907

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Open Access / Acknowledgement Compliant
  • [Journal Article] Associative-memory representations emerge as shared spatial patterns of theta activity spanning the primate temporal cortex2016

    • Author(s)
      Kiyoshi Nakahara, Ken Adachi, Keisuke Kawasaki, Takeshi Matsuo, Hirohito Sawahata, Kei Majima, Masaki Takeda, Sayaka Sugiyama, Ryota Nakata, Atsuhiko Iijima, Hisashi Tanigawa, Takafumi Suzuki, Yukiyasu Kamitani, Isao Hasegawa
    • Journal Title

      Natu Commun.

      Volume: 未定 Pages: 11827-11827

    • DOI

      10.1038/ncomms11827

    • NAID

      120005822769

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Open Access / Acknowledgement Compliant
  • [Journal Article] Induced sensorimotor brain plasticity controls pain in phantom limb patients2016

    • Author(s)
      Yanagisawa T, Fukuma R, Seymour B, Hosomi K, Kishima H, Shimizu T, Yokoi H, Hirata M, Yoshimine T, Kamintani Y, Saitoh Y
    • Journal Title

      Nature communications

      Volume: 7 Pages: 13209-13209

    • DOI

      10.1038/ncomms13209

    • NAID

      120006323740

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Open Access / Acknowledgement Compliant
  • [Journal Article] BrainLiner: A Neuroinformatics Platform for Sharing Time-Aligned Brain-Behavior Data2016

    • Author(s)
      Takemiya M, Majima K, Tsukamoto M and Kamitani Y
    • Journal Title

      Frontiers in Neuroinformatics

      Volume: 10 Pages: 3-3

    • DOI

      10.3389/fninf.2016.00003

    • NAID

      120005947013

    • Related Report
      2015 Annual Research Report
    • Peer Reviewed / Open Access / Acknowledgement Compliant
  • [Presentation] Attention biases neural representations of hierarchical visual features2019

    • Author(s)
      Tomoyasu HORIKAWA, Yukiyasu KAMITANI
    • Organizer
      Cognitive Computational Neuroscience(CCN2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 心的イメージの脳情報デコーディング2019

    • Author(s)
      堀川 友慈
    • Organizer
      日本生理人類学会「感性・脳科学研究部会&睡眠研究部会」合同会合
    • Related Report
      2019 Annual Research Report
    • Invited
  • [Presentation] Inter-individual deep image reconstruction2019

    • Author(s)
      Tomoyasu HORIKAWA, Yukiyasu KAMITANI
    • Organizer
      15th Asea-Pacific Conference on Vision (APCV2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Decoding image motion using deep neural network features2019

    • Author(s)
      Ken SHIRAKAWA, Yukiyasu KAMITANI
    • Organizer
      15th Asia-Pacific Conference in Vision (APCV2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Reconstructing visual and subjective experience from the brain2019

    • Author(s)
      Yukiyasu KAMITANI
    • Organizer
      15th Asia-Pacific Conference on Vision(APCV2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Brain hierarchy score: Which deep neural network most matches the human visual hierarchy?2019

    • Author(s)
      Soma NONAKA, Shuntaro C. Aoki, Kei MAJIMA, Yukiyasu KAMITANI
    • Organizer
      第42回日本神経科学大会(Neuro2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Framework for predicting probability from brain activity2019

    • Author(s)
      Shirin VAFAEI, Kei MAJIMA,
    • Organizer
      第42回日本神経科学大会(Neuro2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Dissonant imaginary2019

    • Author(s)
      Yukiyasu KAMITANI, Daito MANABE
    • Organizer
      Sonar2019 Music, Creativity & Technology
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Neural decoding of what's not there from human brain activity2019

    • Author(s)
      Tomoyasu HORIKAWA
    • Organizer
      A Workshop on Hallucinations,Dreams,Imagination,and Virtual Reality,Experiencing What's Not There
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Brain decoding of auditory-induced cortical activity with deep neural network features2018

    • Author(s)
      Jongyun Park, Tomoyasu Horikawa, Kei Majima,Yukiyasu Kamitani
    • Organizer
      2018 Annual Meeting of Korean Society for Cognitive Science
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Deep image reconstruction from the human brain2018

    • Author(s)
      Yukiyasu Kamitani
    • Organizer
      2018 Annual Meeting of Korean Society for Cognitive Science
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Sparse ordinal logistic regression and its application to brain decoding2018

    • Author(s)
      Emi Satake, Kei Majima, Shuntaro C Aoki, Yukiyasu Kamitani
    • Organizer
      第41回日本神経科学大会
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Decoding of depth information from human brain activity2018

    • Author(s)
      Akira Murakami, Kshitij Dwivedi, Yukiyasu Kamitani
    • Organizer
      第41回日本神経科学大会
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 深層ニューラルネットワークを介した心的イメージの解読2018

    • Author(s)
      堀川友慈
    • Organizer
      第29回生理科学実験技術トレーニングコース
    • Related Report
      2018 Annual Research Report
    • Invited
  • [Presentation] Decoding of seen and imagined contents from the human brain via deep neural network representation2018

    • Author(s)
      Tomoyasu Horikawa
    • Organizer
      第28回日本神経回路学会全国大会(JNNS2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] 深層ニューラルネットワークと脳の相同性とその応用2018

    • Author(s)
      神谷之康
    • Organizer
      第2回ヒト脳イメージング研究会
    • Related Report
      2018 Annual Research Report
    • Invited
  • [Presentation] Brain-DNN homology and its applications2018

    • Author(s)
      Yukiyasu Kamitani
    • Organizer
      第28回日本神経回路学会全国大会(JNNS2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Deep image reconstruction from the human brain2018

    • Author(s)
      Yukiyasu Kamitani
    • Organizer
      The 5th CiNet Conference
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Top-down modulation in human visual cortex revealed by brain decoding with deep neural network representation2018

    • Author(s)
      Mohamed Abdelhack, Yukiyasu Kamitani
    • Organizer
      The 5th CiNet Conference
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Shared hierarchical neural representations between perception, imagery, and dreaming2017

    • Author(s)
      Horikawa T
    • Organizer
      計算論的神経科学に関するNICT-NSF連携ワークショップ
    • Place of Presentation
      NICT脳情報通信融合研究センター(大阪府)
    • Year and Date
      2017-01-17
    • Related Report
      2016 Annual Research Report
    • Invited
  • [Presentation] Deep image reconstruction from human brain activity2017

    • Author(s)
      Guohua SHEN, Tomoyasu HORIKAWA, Kei MAJIMA, Yukiyasu KAMITANI
    • Organizer
      Annual Conference on Cognitive Computational Neuroscience(CCN)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Induction of sensorimotor plasticity altered pain in phantom limb patients with instructions to not move the phantom hand2017

    • Author(s)
      Takufumi YANAGISAWA, Ryohei FUKUMA, Ben SEYMOUR, Kouichi HOSOMI, Haruhiko KISHIMA, Masahiro SHIBATA, Hiroshi YOKOI, Toshiki YOSHIMINE,Yukiyasu KAMITANI, Youichi SAITOH
    • Organizer
      第40回日本神経科学大会(Neuroscience2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Top-down neural processing that supplements missing image features revealed by brain decoding with deep neural network representation2017

    • Author(s)
      Mohamed ABDELHACK, Yukiyasu KAMITANI
    • Organizer
      Vision Sciences Society 17th Annual Meeting(VSS2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Brain decoding via deep neural network feature representation2016

    • Author(s)
      Horikawa T
    • Organizer
      第39回日本神経科学大会(Neuroscience2016)
    • Place of Presentation
      パシフィコ横浜(神奈川県)
    • Year and Date
      2016-07-20
    • Related Report
      2016 Annual Research Report
  • [Presentation] Generic decoding of seen and imagined objects using features of deep neural networks2016

    • Author(s)
      Horikawa T
    • Organizer
      Vision Science Society 16th Annual Meeting(VSS2016)
    • Place of Presentation
      Flolida(USA)
    • Year and Date
      2016-05-13
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 視覚的夢内容のブレイン・デコーディング2016

    • Author(s)
      堀川 友慈
    • Organizer
      第18回日本ヒト脳機能マッピング学会
    • Place of Presentation
      京都大学桂キャンパス 船井哲良記念講堂(京都府)
    • Year and Date
      2016-03-07
    • Related Report
      2015 Annual Research Report
    • Invited
  • [Presentation] Brain decoding of visual dream contents2016

    • Author(s)
      Tomoyasu HORIKAWA
    • Organizer
      脳と心のメカニズム第16回冬のワークショップ
    • Place of Presentation
      ルスツリゾート(北海道)
    • Year and Date
      2016-01-06
    • Related Report
      2015 Annual Research Report
    • Invited
  • [Presentation] 脳情報デコーディング-脳から心を読む技術2015

    • Author(s)
      神谷 之康
    • Organizer
      第33回大阪科学賞表彰式・記念講演
    • Place of Presentation
      大阪科学技術センター8階大ホール(大阪府)
    • Year and Date
      2015-10-28
    • Related Report
      2015 Annual Research Report
    • Invited
  • [Presentation] 脳画像から心を読む2015

    • Author(s)
      神谷 之康
    • Organizer
      第18回画像の認識・理解シンポジウム(MIRU2015)
    • Place of Presentation
      ホテル阪急エキスポパーク(大阪府)
    • Year and Date
      2015-07-27
    • Related Report
      2015 Annual Research Report
    • Invited
  • [Remarks] Github: Generic object decoding

    • URL

      https://github.com/KamitaniLab/GenericObjectDecoding

    • Related Report
      2016 Annual Research Report
  • [Remarks] Github: Brain Decoder Toolbox 2

    • URL

      https://github.com/KamitaniLab/BrainDecoderToolbox2

    • Related Report
      2016 Annual Research Report

URL: 

Published: 2015-07-02   Modified: 2021-02-19  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi