• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Computer Vision for Tangible and Intangible Deep Shitsukan

Planned Research

Project AreaAnalysis and synthesis of deep SHITSUKAN information in the real world
Project/Area Number 20H05951
Research Category

Grant-in-Aid for Transformative Research Areas (A)

Allocation TypeSingle-year Grants
Review Section Transformative Research Areas, Section (IV)
Research InstitutionKyoto University

Principal Investigator

Nishino Ko  京都大学, 情報学研究科, 教授 (60814754)

Co-Investigator(Kenkyū-buntansha) 延原 章平  京都工芸繊維大学, 情報工学・人間科学系, 教授 (00423020)
鄭 銀強  東京大学, 大学院情報理工学系研究科, 教授 (30756896)
Project Period (FY) 2020-11-19 – 2025-03-31
Project Status Completed (Fiscal Year 2024)
Budget Amount *help
¥116,090,000 (Direct Cost: ¥89,300,000、Indirect Cost: ¥26,790,000)
Fiscal Year 2024: ¥18,200,000 (Direct Cost: ¥14,000,000、Indirect Cost: ¥4,200,000)
Fiscal Year 2023: ¥20,930,000 (Direct Cost: ¥16,100,000、Indirect Cost: ¥4,830,000)
Fiscal Year 2022: ¥20,930,000 (Direct Cost: ¥16,100,000、Indirect Cost: ¥4,830,000)
Fiscal Year 2021: ¥19,890,000 (Direct Cost: ¥15,300,000、Indirect Cost: ¥4,590,000)
Fiscal Year 2020: ¥36,140,000 (Direct Cost: ¥27,800,000、Indirect Cost: ¥8,340,000)
Keywords知覚情報処理 / 視覚情報処理 / 知能情報処理 / コンピュータビジョン / コンピュテーショナルフォトグラフィ / 画像情報処理 / 人間計測 / 深奥質感 / 知能情報学
Outline of Research at the Start

人間の「目測」の計算機による実現を目指し、単に物を見た目から認識するという現在の計算機視覚の限界を突破し、一見明らかではないが、視覚情報から抽出されうる物体や環境にまつわる属性を推定できるようにする。また、人間が暗黙に視覚より得ている物、人や場に対して働きかける上で有用な属性情報の顕在化を目指す。特に、「物」「人」さらにそれらが配置され形成する「場」を考え、それぞれについて視覚から抽出しうる人間の行動の決定に有用な質感にまつわる情報を網羅的に推定することを目指す。これら計算機視覚を用いた深奥質感の知覚、表現、そして利用の実現を通し、人間の深奥質感知覚機構の解明および操作にも資する。

Outline of Final Research Achievements

Based on the premise that the interaction of people with things, other people, and the space as well as the information necessary to plan and execute actions in the real world lie at the core of Shitsukan, i.e., Deep Shitsukan we seek, we aimed to establish foundations of computational visual perception that can understand tangible instantiations as well as intangible clues of Shitsukan from visual information through research aimed at recovering physical properties and conditions of things, capturing signals reflecting the internal states and intentions of people, and reconstructing spaces and their affordance. This has led to a number of research results consisting of novel algorithms and data that enable the extraction of essential physical quantities and emergent behaviors of people from visual information. These results have been published widely in top international conferences and journals.

Academic Significance and Societal Importance of the Research Achievements

物体などの物理量を視覚情報から明示的に復元し、シーンを構成する素材を認識し、人をただ観察するだけでその視線行動や指差しなどの意図発現を捉え、環境の精緻な三次元復元を行い、その中の人物の姿勢を正確に求め、人間も知覚できない方法で表面構成などの物の質を捉えることができる計算機視覚を実現することにより、肉眼でも明らかな質感ばかりでなく、実体はないが感じ取られる質感を定量化する技術基盤を確立した。これらの成果を通し、我々の実世界を彩る質感の本質に迫る手段を広く実現することにより、より豊かな計算機と人間の共生基盤を樹立したばかりではなく、人間の質感認知機構の解明に資すると期待できる。

Report

(6 results)
  • 2024 Annual Research Report   Final Research Report ( PDF )
  • 2023 Annual Research Report
  • 2022 Annual Research Report
  • 2021 Annual Research Report
  • 2020 Annual Research Report
  • Research Products

    (36 results)

All 2025 2024 2023 2022 2021 Other

All Int'l Joint Research (3 results) Journal Article (6 results) (of which Peer Reviewed: 6 results) Presentation (20 results) (of which Int'l Joint Research: 20 results) Remarks (5 results) Patent(Industrial Property Rights) (2 results)

  • [Int'l Joint Research] ENPC(フランス)

    • Related Report
      2024 Annual Research Report
  • [Int'l Joint Research] Simon Fraser University(カナダ)

    • Related Report
      2023 Annual Research Report
  • [Int'l Joint Research] Rutgers University(米国)

    • Related Report
      2020 Annual Research Report
  • [Journal Article] Extrinsic Camera Calibration From a Moving Person2022

    • Author(s)
      Sang-Eun Lee, Keisuke Shibata, Soma Nonaka, Shohei Nobuhara, Ko Nishino
    • Journal Title

      IEEE Robotics and Automation Letters

      Volume: 7(4) Issue: 4 Pages: 10344-10351

    • DOI

      10.1109/lra.2022.3192629

    • Related Report
      2022 Annual Research Report
    • Peer Reviewed
  • [Journal Article] ViewBirdiformer: Learning to recover ground-plane crowd trajectories and ego-motion from a single ego-centric view2022

    • Author(s)
      Mai Nishimura, Shohei Nobuhara, Ko Nishino
    • Journal Title

      IEEE Robotics and Automation Letters

      Volume: 1(8) Issue: 1 Pages: 368-375

    • DOI

      10.1109/lra.2022.3221335

    • Related Report
      2022 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Non-Rigid Shape From Water2021

    • Author(s)
      Meng-Yu Jennifer Kuo, Ryo Kawahara , Shohei Nobuhara , Ko Nishino
    • Journal Title

      IEEE Transactions on Pattern Analysis and Machine Intelligence

      Volume: 7(43) Issue: 7 Pages: 2220-2232

    • DOI

      10.1109/tpami.2021.3075450

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Invertible Neural BRDF for Object Inverse Rendering2021

    • Author(s)
      Zhe Chen, Shohei Nobuhara, Ko Nishino
    • Journal Title

      IEEE Transactions on Pattern Analysis and Machine Intelligence

      Volume: online first Issue: 12 Pages: 1-16

    • DOI

      10.1109/tpami.2021.3129537

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Human object interaction detection with missing objects2021

    • Author(s)
      Kaen Kogashi, Yang Wu, Shohei Nobuhara, Ko Nishino
    • Journal Title

      Image and Vision Computing

      Volume: 113 Pages: 109-109

    • DOI

      10.1016/j.imavis.2021.104262

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Surface Normals and Shape From Water2021

    • Author(s)
      Meng-Yu Jennifer Kuo, Satoshi Murai, Ryo Kawahara, Shohei Nobuhara, Ko Nishino
    • Journal Title

      IEEE Transactions on Pattern Analysis and Machine Intelligence

      Volume: online first Issue: 12 Pages: 1-12

    • DOI

      10.1109/tpami.2021.3121963

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Presentation] HeatFormer: A Neural Optimizer for Multiview Human Mesh Recovery2025

    • Author(s)
      Yuto Matsubara, Ko Nishino
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR’25
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] MAtCha Gaussians: Atlas of Charts for High-Quality Geometry and Photorealism From Sparse View2025

    • Author(s)
      Antoine Guedon, Tomoki Ichikawa, Kohei Yamashita, Ko Nishino
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR’25
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] DeepShaRM: Multi-View Shape and Reflectance Map Recovery Under Unknown Lighting2024

    • Author(s)
      Kohei Yamashita, Shohei Nobuhara, Ko Nishino
    • Organizer
      International Conference on 3D Vision (3DV), 2024
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Fooling Polarization-Based Vision using Locally Controllable Polarizing Projection2024

    • Author(s)
      Zhuoxiao Li, Zhihang Zhong, Shohei Nobuhara, Ko Nishino, Yinqiang Zheng
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR’24
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] SPIDeRS: Structured Polarization for Invisible Depth and Reflectance Sensing2024

    • Author(s)
      Tomoki Ichikawa, Shohei Nobuhara, Ko Nishino
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR’24
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance2024

    • Author(s)
      Yuto Enyo, Ko Nishino
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR’24
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] KFD-NeRF: Rethinking Dynamic NeRF with Kalman Filter2024

    • Author(s)
      Yifan Zhan, Zhuoxiao Li, Muyao Niu, Zhihang Zhong, Shohei Nobuhara, Ko Nishino, Yinqiang Zheng
    • Organizer
      IEEE/CVF European Conference on Computer Vision ECCV’24
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Camera Height Doesn’t Change: Unsupervised Training for Metric Monocular Road-Scene Depth Estimation2024

    • Author(s)
      Genki Kinoshita, Ko Nishino
    • Organizer
      IEEE/CVF European Conference on Computer Vision ECCV’24
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Correspondences of the Third Kind: Camera Pose Estimation from Object Reflection2024

    • Author(s)
      Kohei Yamashita, Vincent Lepetit, K. Nishino
    • Organizer
      IEEE/CVF European Conference on Computer Vision ECCV’24
    • Related Report
      2024 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Fresnel Microfacet BRDF: Unification of Polari-Radiometric Surface-Body Reflection2023

    • Author(s)
      Tomoki Ichikawa, Yhoshiki Fukao, Shohei Nobuhara, Ko Nishino
    • Organizer
      IEEE/CVF International Conference on Computer Vision
    • Related Report
      2023 Annual Research Report
    • Int'l Joint Research
  • [Presentation] DeePoint: Visual Pointing Recognition and Direction Estimation2023

    • Author(s)
      Shu Nakamura, Yasutomo Kawanishi, Shohei Nobuhara, Ko Nishino
    • Organizer
      IEEE/CVF International Conference on Computer Vision
    • Related Report
      2023 Annual Research Report
    • Int'l Joint Research
  • [Presentation] NeRFrac: Neural Radiance Fields through Refractive Surface2023

    • Author(s)
      Yifan Zhan, Shohei Nobuhara, Ko Nishino, and Yinqiang Zheng
    • Organizer
      IEEE/CVF International Conference on Computer Vision
    • Related Report
      2023 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Multimodal Material Segmentation2022

    • Author(s)
      Yupeng Liang, Ryosuke Wakaki, Shohei Nobuhara, Ko Nishino
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition
    • Related Report
      2022 Annual Research Report 2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Dynamic 3D Gaze from Afar: Deep Gaze Estimation from Temporal Eye-Head-Body Coordination2022

    • Author(s)
      Soma Nonaka, Shohei Nobuhara, Ko Nishino
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition
    • Related Report
      2022 Annual Research Report 2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] BlindSpotNet: Seeing Where We Cannot See2022

    • Author(s)
      Taichi Fukuda, Kotaro Hasegawa, Shinya Ishizaki, Shohei Nobuhara, Ko Nishino
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] RGB Road Scene Material Segmentation2022

    • Author(s)
      Sudong Cai, Ryosuke Wakaki, Shohei Nobuhara, Ko Nishino
    • Organizer
      Asian Conference on Computer Vision
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] nLMVS-Net: Deep Non-Lambertian Multi-View Stereo2022

    • Author(s)
      Kohei Yamashita, Yuto Enyo, Shohei Nobuhara, Ko Nishino,
    • Organizer
      IEEE/CVF Winter Conference on Applications of Computer Vision
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Polarimetric Normal Stereo2021

    • Author(s)
      Yoshiki Fukao, Ryo Kawahara, Shohei Nobuhara, Ko Nishino
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Shape from Sky: Polarimetric Normal Recovery Under The Sky2021

    • Author(s)
      Tomoki Ichikawa, Matthew Purri, Ryo Kawahara, Shohei Nobuhara, Kristin Dana, Ko Nishino,
    • Organizer
      IEEE/CVF Conference on Computer Vision and Pattern Recognition
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Human-Object Interaction Detection with Missing Objects2021

    • Author(s)
      Kaen Kogashi, Yang Wu, Shohei Nobuhara, Ko Nishino
    • Organizer
      International Conference on Machine Vision Applications
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Remarks] Diffusion Reflectance Map

    • URL

      https://vision.ist.i.kyoto-u.ac.jp/research/drm/

    • Related Report
      2024 Annual Research Report
  • [Remarks] Correspondences of the Third Kind

    • URL

      https://vision.ist.i.kyoto-u.ac.jp/research/3rdcorr/

    • Related Report
      2024 Annual Research Report
  • [Remarks] Multistable Shape from Shading

    • URL

      https://vision.ist.i.kyoto-u.ac.jp/research/mssfs/

    • Related Report
      2024 Annual Research Report
  • [Remarks] HeatFormer

    • URL

      https://vision.ist.i.kyoto-u.ac.jp/research/heatformer/

    • Related Report
      2024 Annual Research Report
  • [Remarks] MAtCha Gaussians

    • URL

      https://anttwo.github.io/matcha/

    • Related Report
      2024 Annual Research Report
  • [Patent(Industrial Property Rights)] 情報処理装置、情報処理方法、情報処理プログラム、死角推定モデルプログラム、自動運転システムおよび運転教習システム2022

    • Inventor(s)
      西野恒、延原章平、福田太一、長谷川浩太郎、石崎慎弥
    • Industrial Property Rights Holder
      西野恒、延原章平、福田太一、長谷川浩太郎、石崎慎弥
    • Industrial Property Rights Type
      特許
    • Filing Date
      2022
    • Related Report
      2022 Annual Research Report
  • [Patent(Industrial Property Rights)] 情報処理装置、情報処理方法および情報処理プログラム2021

    • Inventor(s)
      西野恒、延原章平、川原僚、深尾圭貴
    • Industrial Property Rights Holder
      西野恒、延原章平、川原僚、深尾圭貴
    • Industrial Property Rights Type
      特許
    • Industrial Property Number
      2021-091519
    • Filing Date
      2021
    • Related Report
      2020 Annual Research Report

URL: 

Published: 2020-11-24   Modified: 2026-01-16  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi