• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Detailed 3D Reconstruction of Whole Environment Including Shape, Illumination, and Reflectance

Research Project

Project/Area Number 17H00744
Research Category

Grant-in-Aid for Scientific Research (A)

Allocation TypeSingle-year Grants
Section一般
Research Field Perceptual information processing
Research InstitutionTokyo Institute of Technology

Principal Investigator

Okutomi Masatoshi  東京工業大学, 工学院, 教授 (00262303)

Co-Investigator(Kenkyū-buntansha) 岡部 孝弘  九州工業大学, 大学院情報工学研究院, 教授 (00396904)
鳥居 秋彦  東京工業大学, 工学院, 助教 (20585179)
Project Period (FY) 2017-04-01 – 2021-03-31
Project Status Completed (Fiscal Year 2021)
Budget Amount *help
¥44,330,000 (Direct Cost: ¥34,100,000、Indirect Cost: ¥10,230,000)
Fiscal Year 2020: ¥11,180,000 (Direct Cost: ¥8,600,000、Indirect Cost: ¥2,580,000)
Fiscal Year 2019: ¥12,480,000 (Direct Cost: ¥9,600,000、Indirect Cost: ¥2,880,000)
Fiscal Year 2018: ¥12,480,000 (Direct Cost: ¥9,600,000、Indirect Cost: ¥2,880,000)
Fiscal Year 2017: ¥8,190,000 (Direct Cost: ¥6,300,000、Indirect Cost: ¥1,890,000)
Keywordsマルチビューインバースレンダリング / 3次元復元 / 光源推定 / 反射率推定 / コンピュータビジョン / コンピュータービジョン
Outline of Final Research Achievements

In this study, we have developed novel multi-view inverse rendering (MVIR) methods called Spectral MVIR and Polarimetric MVIR, where detailed 3D shape, illumination, and reflectance of the scene can be estimated simultaneously from multi-view input images.
Spectral MVIR enables us to estimate detailed 3D shape and precise spectral reflectance of the scene using an off-the-shelf LED bulb/projector and a standard RGB camera. Polarimetric MVIR enables us to acquire more detailed 3D shape compared with standard MVIR by exploiting the polarization information of reflected light obtained using a polarization camera.
These achievements have been published in major journals or conferences, such as The Visual Computer, ECCV, and ICCP.

Academic Significance and Societal Importance of the Research Achievements

画像を用いた3次元復元は、コンピュータビジョンの中核を担う研究課題である。本研究では、これまで独立に発展を続けてきた幾何学的3次元復元および光学的3次元復元を融合し、シーンの3D形状・光源分布・反射率を同時推定するマルチビューインバースレンダリング (MVIR)手法を確立・発展させた点に学術的な意義がある。
MVIRは画像に含まれる幾何学的および光学的特徴を最大限活かし、シーン全体の高精度な復元を実現する手法であるため、常に高度化が求められている3次元計測技術において、多岐にわたる分野での応用が期待される。

Report

(5 results)
  • 2021 Final Research Report ( PDF )
  • 2020 Annual Research Report
  • 2019 Annual Research Report
  • 2018 Annual Research Report
  • 2017 Annual Research Report
  • Research Products

    (53 results)

All 2022 2021 2020 2019 2018 2017 Other

All Journal Article (7 results) (of which Int'l Joint Research: 3 results,  Peer Reviewed: 7 results,  Open Access: 3 results) Presentation (45 results) (of which Int'l Joint Research: 22 results,  Invited: 3 results) Remarks (1 results)

  • [Journal Article] Pro-Cam SSfM: projector-camera system for structure and spectral reflectance from motion2022

    • Author(s)
      Li Chunyu、Monno Yusuke、Okutomi Masatoshi
    • Journal Title

      The Visual Computer

      Volume: -

    • DOI

      10.1007/s00371-022-02434-0

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Long-Term Visual Localization Revisited2022

    • Author(s)
      Toft Carl、Maddern Will、Torii Akihiko、Hammarstrand Lars、Stenborg Erik、Safari Daniel、Okutomi Masatoshi、Pollefeys Marc、Sivic Josef、Pajdla Tomas、Kahl Fredrik、Sattler Torsten
    • Journal Title

      IEEE Transactions on Pattern Analysis and Machine Intelligence

      Volume: 44 Issue: 4 Pages: 2074-2088

    • DOI

      10.1109/tpami.2020.3032010

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Stomach 3D Reconstruction Using Virtual Chromoendoscopic Images2021

    • Author(s)
      Widya Aji Resindra、Monno Yusuke、Okutomi Masatoshi、Suzuki Sho、Gotoda Takuji、Miki Kenji
    • Journal Title

      IEEE Journal of Translational Engineering in Health and Medicine

      Volume: 9 Pages: 1-11

    • DOI

      10.1109/jtehm.2021.3062226

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Whole Stomach 3D Reconstruction and Frame Localization from Monocular Endoscope Video2019

    • Author(s)
      Widya Aji Resindra、Monno Yusuke、Okutomi Masatoshi、Suzuki Sho、Gotoda Takuji、Miki Kenji
    • Journal Title

      IEEE Journal of Translational Engineering in Health and Medicine

      Volume: 7 Pages: 1-10

    • DOI

      10.1109/jtehm.2019.2946802

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization?2019

    • Author(s)
      Torii Akihiko、Taira Hajime、Sivic Josef、Pollefeys Marc、Okutomi Masatoshi、Pajdla Tomas、Sattler Torsten
    • Journal Title

      IEEE Transactions on Pattern Analysis and Machine Intelligence

      Volume: - Issue: 3 Pages: 814-829

    • DOI

      10.1109/tpami.2019.2941876

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] InLoc: Indoor Visual Localization with Dense Matching and View Synthesis2019

    • Author(s)
      Taira Hajime、Okutomi Masatoshi、Sattler Torsten、Cimpoi Mircea、Pollefeys Marc、Sivic Josef、Pajdla Tomas、Torii Akihiko
    • Journal Title

      IEEE Transactions on Pattern Analysis and Machine Intelligence

      Volume: - Issue: 4 Pages: 1293-1307

    • DOI

      10.1109/tpami.2019.2952114

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Structure from motion using dense CNN features with keypoint relocalization2018

    • Author(s)
      Widya Aji Resindra、Torii Akihiko、Okutomi Masatoshi
    • Journal Title

      IPSJ Transactions on Computer Vision and Applications

      Volume: 10 Issue: 1

    • DOI

      10.1186/s41074-018-0042-y

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Presentation] Online Illumination Planning for Shadow-Robust Photometric Stereo2022

    • Author(s)
      Hirochika Tanikawa, Ryo Kawahara, and Takahiro Okabe
    • Organizer
      the 28th International Workshop on Frontiers of Computer Vision (IW-FCV2022)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Self-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth2022

    • Author(s)
      Zijie Jiang, Hajime Taira, Naoyuki Miyashita and Masatoshi Okutomi
    • Organizer
      IEEE International Conference on Robotics and Automation (ICRA2022)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Deep Hyperspectral-Depth Reconstruction Using Single Color-Dot Projection2022

    • Author(s)
      Chunyu Li, Yusuke Monno and Masatoshi Okutomi
    • Organizer
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR2022)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] カラー画像と推定深度画像の Multi-Layer Fusion に基づく自己教師あり学習による Ego-Motion 推定2022

    • Author(s)
      蒋 子傑, 田平 創, 宮下尚之, 奥富正敏
    • Organizer
      第28回画像センシングシンポジウム(SSII2022)
    • Related Report
      2020 Annual Research Report
  • [Presentation] マルチビューインバースレンダリングによる高精細な3D形状と分光反射率の同時推定2022

    • Author(s)
      李 淳雨, 紋野雄介, 奥富正敏
    • Organizer
      第28回画像センシングシンポジウム(SSII2022)
    • Related Report
      2020 Annual Research Report
  • [Presentation] Self-supervised monocular depth estimation in gastroendoscopy using GAN-augmented images2021

    • Author(s)
      Aji Resindra Widya, Yusuke Monno, Masatoshi Okutomi, Sho Suzuki, Takuji Gotoda and Kenji Miki
    • Organizer
      SPIE Medical Imaging
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] VIO-Aided Structure from Motion Under Challenging Environments2021

    • Author(s)
      Zijie Jiang, Hajime Taira, Naoyuki Miyashita and Masatoshi Okutomi
    • Organizer
      the 22nd IEEE International Conference on Industrial Technology (ICIT2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Spectral MVIR: Joint Reconstruction of 3D Shape and Spectral Reflectance2021

    • Author(s)
      Chunyu Li, Yusuke Monno and Masatoshi Okutomi
    • Organizer
      IEEE International Conference on Computational Photography (ICCP2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Polarimetric MVIR: カラー偏光画像を用いたマルチビューインバースレンダリングによる高精細3次元復元2021

    • Author(s)
      趙 金雨, 紋野雄介, 奥富正敏
    • Organizer
      第27回画像センシングシンポジウム(SSII2021)
    • Related Report
      2020 Annual Research Report
  • [Presentation] Video-Based Camera Localization Using Anchor View Detection and Recursive 3D Reconstruction2021

    • Author(s)
      Hajime Taira, Koki Onbe, Naoyuki Miyashita and Masatoshi Okutomi
    • Organizer
      the 17th International Conference on Machine Vision Applications (MVA2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Illumination Planning for Measuring Per-Pixel Surface Roughness2021

    • Author(s)
      Kota Arieda and Takahiro Okabe
    • Organizer
      the 17th International Conference on Machine Vision Applications (MVA2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Learning-Based Depth and Pose Estimation for Monocular Endoscope with Loss Generalization2021

    • Author(s)
      Aji Resindra Widya, Yusuke Monno, Masatoshi Okutomi, Sho Suzuki, Takuji Gotoda and Kenji Miki
    • Organizer
      the 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Polarimetric Multi-view Inverse Rendering2020

    • Author(s)
      Jinyu Zhao, Yusuke Monno and Masatoshi Okutomi
    • Organizer
      16th European Conference on Computer Vision (ECCV2020)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Pro-Cam SSfM: Projector-Camera System for Structure and Spectral Reflectance from Motion2020

    • Author(s)
      Chunyu Li, Yusuke Monno, Hironori Hidaka and Masatoshi Okutomi
    • Organizer
      第23回画像の認識・理解シンポジウム (MIRU2020)
    • Related Report
      2020 Annual Research Report
    • Invited
  • [Presentation] 3D Model Reconstruction of Whole Stomach from Standard Endoscope Video2020

    • Author(s)
      Sho Suzuki, Kenji Miki, Takuji Gotoda, Aji Resindra Widya, Yusuke Monno, Masatoshi Okutomi
    • Organizer
      the International Digestive Disease Forum (IDDF2020)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Spectral Reflectance Estimation Using Projector with Unknown Spectral Power Distribution2020

    • Author(s)
      Hironori Hidaka, Yusuke Monno and Masatoshi Okutomi
    • Organizer
      the Twenty-eighth Color and Imaging Conference (CIC28)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Inverse Lighting from Cast Shadows Under Unknown Radiometric Response Function2020

    • Author(s)
      Takuto Nakashima, Ryo Matsuoka, Takahiro Okabe
    • Organizer
      International Workshop on Frontiers of Computer Vision (IW-FCV2020)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 内視鏡画像を基にした胃の3次元モデル表示法の開発2020

    • Author(s)
      鈴木翔, 三木健司, 後藤田卓志, Widya Aji Resindra, 紋野雄介, 奥富 正敏
    • Organizer
      第16回日本消化管学会総会学術集会 ワークショップ11
    • Related Report
      2019 Annual Research Report
  • [Presentation] カメラ応答関数未知の影に基づくインバースライティング2020

    • Author(s)
      中島拓人, 松岡諒, 岡部孝弘
    • Organizer
      電子情報通信学会技術研究報告, PRMU2019-77
    • Related Report
      2019 Annual Research Report
  • [Presentation] 3D Pipe Network Reconstruction Based on Structure from Motion with Incremental Conic Shape Detection and Cylindrical Constraint2020

    • Author(s)
      Sho Kagami, Hajime Taira, Naoyuki Miyashita, Akihiko Torii and Masatoshi Okutomi
    • Organizer
      29th IEEE International Symposium on Industrial Electronics (ISIE2020)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 内視鏡動画像からの胃の3次元形状復元2020

    • Author(s)
      紋野雄介, Widya Aji Resindra, 奥富正敏, 鈴木翔, 後藤田卓志, 三木健司
    • Organizer
      第26回画像センシングシンポジウム(SSII2020)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Pro-Cam SSfM:汎用プロジェクタとカメラを用いた分光3D計測2020

    • Author(s)
      李淳雨, 紋野雄介, 日高宏紀, 奥富正敏
    • Organizer
      第26回画像センシングシンポジウム(SSII2020)
    • Related Report
      2019 Annual Research Report
  • [Presentation] ステレオカメラのセルフキャリブレーションに対する解析と2フレーム手法の提案2020

    • Author(s)
      大石慎太郎, 奥富正敏
    • Organizer
      第26回画像センシングシンポジウム(SSII2020)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Stomach 3D Reconstruction Based on Virtual Chromoendoscopic Image Generation2020

    • Author(s)
      Aji Resindra Widya, Yusuke Monno, Masatoshi Okutomi, Sho Suzuki, Takuji Gotoda and Kenji Miki
    • Organizer
      42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC2020)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 大規模屋内環境における3Dマップを用いた自己位置推定2019

    • Author(s)
      田平創, Torsten Sattler, Josef Sivic, Tomas Pajdla, 鳥居秋彦, 奥富正敏
    • Organizer
      第25回画像センシングシンポジウム(SSII2019)
    • Related Report
      2019 Annual Research Report 2018 Annual Research Report
  • [Presentation] カメラ応答関数未知の単一画像からの影に基づく光源分布推定2019

    • Author(s)
      中島拓人, 松岡諒, 岡部孝弘
    • Organizer
      第22回画像の認識・理解シンポジウム (MIRU2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] 3D Reconstruction of Whole Stomach from Endoscope Video Using Structure-from-Motion2019

    • Author(s)
      Aji Resindra Widya, Yusuke Monno, Kosuke Imahori, Masatoshi Okutomi, Sho Suzuki, Takuji Gotoda and Kenji Miki
    • Organizer
      41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Is This The Right Place? Geometric-Semantic Pose Verification for Indoor Visual Localization2019

    • Author(s)
      Hajime Taira, Ignacio Rocco, Jiri Sedlar, Masatoshi Okutomi, Josef Sivic, Tomas Pajdla, Torsten Sattler and Akihiko Torii
    • Organizer
      IEEE International Conference on Computer Vision (ICCV2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Pro-Cam SSfM: Projector-Camera System for Structure and Spectral Reflectance from Motion2019

    • Author(s)
      Chunyu Li, Yusuke Monno, Hironori Hidaka and Masatoshi Okutomi
    • Organizer
      IEEE International Conference on Computer Vision (ICCV2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 内視鏡画像を基にした胃の3次元モデル表示法の開発2019

    • Author(s)
      鈴木翔, 三木健司, 後藤田卓志, Widya Aji Resindra, 紋野雄介, 今堀公介, 奥富 正敏
    • Organizer
      第27回日本消化器関連学会週間
    • Related Report
      2019 Annual Research Report
  • [Presentation] ライトフィールド履歴の類似度に基づくシーンの法線推定2019

    • Author(s)
      黒木武, 岡部孝弘
    • Organizer
      2019年3月情報処理学会コンピュータビジョンとイメージメディア研究会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 屈折と吸収に基づく多波長画像からの水面・水中物体の形状復元2019

    • Author(s)
      中嶋倫太郎, 岡部孝弘
    • Organizer
      2019年3月情報処理学会コンピュータビジョンとイメージメディア研究会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 外観検査のための半透明物体の表面粗さ推定2019

    • Author(s)
      有枝航汰, 岡部孝弘
    • Organizer
      情報処理学会 第81回全国大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 参照物体を用いた燐光物体の法線推定2019

    • Author(s)
      村上時代, 日高大地, 岡部孝弘
    • Organizer
      情報処理学会 第81回全国大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 3D Reconstruction of Whole Stomach from Endoscope Video Using Structure-from-Motion2019

    • Author(s)
      Aji Resindra Widya, Yusuke Monno, Kosuke Imahori, Masatoshi Okutomi, Sho Suzuki, Takuji Gotoda and Kenji Miki
    • Organizer
      41th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC2019)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 影に基づく光源分布とカメラ応答関数の同時推定2018

    • Author(s)
      中島拓人, 岡部孝弘
    • Organizer
      2018年5月情報処理学会コンピュータビジョンとイメージメディア研究会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 大規模visual localization の実用化に向けた評価用データセットの作成2018

    • Author(s)
      田平創, 荻野凌, 岩田健太郎, Torsten Sattler, Josef Sivic, Tomas Pajdla, 鳥居秋彦, 奥富正敏
    • Organizer
      第24回画像センシングシンポジウム(SSII2018)
    • Related Report
      2018 Annual Research Report
  • [Presentation] 2時刻間のカメラ運動推定を伴うステレオセルフキャリブレーション2018

    • Author(s)
      洞山慶太, 鳥居秋彦, 奥富正敏
    • Organizer
      第24回画像センシングシンポジウム(SSII2018)
    • Related Report
      2018 Annual Research Report
  • [Presentation] 汎用のカメラとプロジェクターを用いたキャリブレーションの不要な高精度3次元計測2018

    • Author(s)
      李淳雨, 鳥居秋彦, 奥富正敏
    • Organizer
      第24回画像センシングシンポジウム(SSII2018)
    • Related Report
      2018 Annual Research Report
  • [Presentation] InLoc: Indoor Visual Localization with Dense Matching and View Synthesis2018

    • Author(s)
      Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Mark Pollefeys, Josef Sivic, Tomas Pajdla and Akihiko Torii
    • Organizer
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Robust, Precise, and Calibration-Free Shape Acquisition with an Off-the-Shelf Camera and Projector2018

    • Author(s)
      Chunyu Li, Akihiko Torii and Masatoshi Okutomi
    • Organizer
      IEEE International Conference on Consumer Electronics (ICCE2018)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] InLoc: Indoor visual localization with dense matching and view synthesis2018

    • Author(s)
      Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Josef Sivic, Mark Pollefeys, Tomas Pajdla and Akihiko Torii
    • Organizer
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR2018)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Multi-View Inverse Renderingによる高精細な3次元復元2017

    • Author(s)
      鳥居秋彦, 金杞昌, 奥富正敏
    • Organizer
      第23回画像センシングシンポジウム(SSII2017)
    • Related Report
      2017 Annual Research Report
  • [Presentation] Multi-View Inverse Rendering under Arbitrary Illumination and Albedo2017

    • Author(s)
      Kichang Kim, Akihiko Torii and Masatoshi Okutomi
    • Organizer
      第20回画像の認識・理解シンポジウム(MIRU2017)
    • Related Report
      2017 Annual Research Report
    • Invited
  • [Presentation] 3Dシーンの復元 --より精密に、より大規模に、より完全な情報を--2017

    • Author(s)
      奥富正敏
    • Organizer
      3Dレーザスキャニング&イメージングシンポジウム 2017
    • Related Report
      2017 Annual Research Report
    • Invited
  • [Remarks] 奥富研究室ホームページ

    • URL

      http://www.ok.sc.e.titech.ac.jp/index-j.shtml

    • Related Report
      2020 Annual Research Report 2019 Annual Research Report 2018 Annual Research Report 2017 Annual Research Report

URL: 

Published: 2017-04-28   Modified: 2023-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi