• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

3D modeling using an RGBD camera in the mobile environment and its application to mixed reality

Research Project

Project/Area Number 16H02851
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypeSingle-year Grants
Section一般
Research Field Perceptual information processing
Research InstitutionNational Institute of Informatics

Principal Investigator

Sugimoto Akihiro  国立情報学研究所, コンテンツ科学研究系, 教授 (30314256)

Co-Investigator(Kenkyū-buntansha) 大石 岳史  東京大学, 生産技術研究所, 准教授 (80569509)
Research Collaborator PAJDLA tomas  
THOMAS diego  
Project Period (FY) 2016-04-01 – 2019-03-31
Project Status Completed (Fiscal Year 2018)
Budget Amount *help
¥15,990,000 (Direct Cost: ¥12,300,000、Indirect Cost: ¥3,690,000)
Fiscal Year 2018: ¥6,240,000 (Direct Cost: ¥4,800,000、Indirect Cost: ¥1,440,000)
Fiscal Year 2017: ¥5,330,000 (Direct Cost: ¥4,100,000、Indirect Cost: ¥1,230,000)
Fiscal Year 2016: ¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000)
Keywords3次元モデル / 複合現実感 / モバイル環境 / コンピュータビジョン / 3次元モデル
Outline of Final Research Achievements

We aimed at developing a method for constructing high fidelity 3D models in real time with less memory consumption from streaming input data captured by an RGBD camera in the mobile environment. We developed a parametric surface representation for dense 3D modeling, a model fitting method, robust and accurate depth fusion, and so on. As a result, we succeeded in accurately modeling large-scale indoor scenes in real time with cheap memory consumption. We could also show the effectiveness of our developed methods for mixed reality in the outdoor scenes.

Academic Significance and Societal Importance of the Research Achievements

省メモリでリアルタイムに3次元モデルを高精度に生成する技術は、スマートフォンの写真撮影や映像撮影の要領で、エンドユーザがだれでも思い立った時にその場で、簡単にリアルタイムで3次元モデルを生成し、それを手軽に楽しめる環境を提供するものである。通信技術と連携することで、訪問が困難な博物館に保管された展示物や未公開遺産のモデル化・伝送を通じた文化遺産の3次元情報発信、遠隔講義おける教材提供の充実、3次元地図データ生成などへの応用が見込まれる。

Report

(4 results)
  • 2018 Annual Research Report   Final Research Report ( PDF )
  • 2017 Annual Research Report
  • 2016 Annual Research Report
  • Research Products

    (48 results)

All 2019 2018 2017 2016 Other

All Int'l Joint Research (5 results) Journal Article (18 results) (of which Int'l Joint Research: 5 results,  Peer Reviewed: 18 results,  Open Access: 2 results,  Acknowledgement Compliant: 3 results) Presentation (25 results) (of which Int'l Joint Research: 24 results,  Invited: 1 results)

  • [Int'l Joint Research] Czech Technical University(チェコ)

    • Related Report
      2018 Annual Research Report
  • [Int'l Joint Research] National Tsing Hua University(台湾)

    • Related Report
      2018 Annual Research Report
  • [Int'l Joint Research] University College London(英国)

    • Related Report
      2017 Annual Research Report
  • [Int'l Joint Research] トリノ大(イタリア)

    • Related Report
      2017 Annual Research Report
  • [Int'l Joint Research] チェコ工科大(チェコ)

    • Related Report
      2016 Annual Research Report
  • [Journal Article] Semantic Instance Meets Salient Object: Study on Video Semantic Salient Instance Segmentation2019

    • Author(s)
      T.-N. Le and A. Sugimoto
    • Journal Title

      Proc. of IEEE Winter Conference on Applications of Computer Vision

      Volume: 1 Pages: 1779-1788

    • DOI

      10.1109/wacv.2019.00194

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Journal Article] SegmentedFusion: 3D Human Body Reconstruction using Stitched Bounding Boxes2018

    • Author(s)
      Y. Shih-Hsuan, D. Thomas, A. Sugimoto, S.-H. Lai, and R. Taniguchi
    • Journal Title

      Proc. of The 6th International Conference on 3D Vision

      Volume: 1 Pages: 190-198

    • DOI

      10.1109/3dv.2018.00031

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Video Salient Object Detection Using Spatiotemporal Deep Features2018

    • Author(s)
      T.-N. Le and A. Sugimoto
    • Journal Title

      IEEE Trans. on Image Processing

      Volume: 27 Issue: 10 Pages: 5002-5015

    • DOI

      10.1109/tip.2018.2849860

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Linear Solution to the Minimal Absolute Pose Rolling Shutter Problem2018

    • Author(s)
      Zuzana Kukelova, Cenek Albl, Akihiro Sugimoto, and Tomas Pajdla
    • Journal Title

      Proc. of Asian Conference on Computer Vision

      Volume: 1

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Occlusion Handling using Semantic Segmentation and Visibility-Based Rendering for Mixed Reality2018

    • Author(s)
      M. Roxas, T. Hori, T. Fukiage, Y. Okamoto, T. Oishi
    • Journal Title

      Proc. of 24th ACM Symposium on Virtual Reality Software and Technology

      Volume: 1

    • DOI

      10.1145/3281505.3281546

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Journal Article] LiDAR and Camera Calibration using Motion Estimated by Sensor Fusion Odometry2018

    • Author(s)
      R. Ishikawa, T. Oishi, K. Ikeuchi
    • Journal Title

      Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems

      Volume: 1 Pages: 7342-7349

    • DOI

      10.1109/iros.2018.8593360

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Real-Time Simultaneous 3D Reconstruction and Optical Flow Estimation2018

    • Author(s)
      M. Roxas and T. Oishi
    • Journal Title

      IEEE Winter Conference on Applications of Computer Vision (WACV)

      Volume: 1

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Fast 3D Point Cloud Segmentation using Supervoxels with Geometry and Color for 3D Scene Understanding2017

    • Author(s)
      F. Verdoja, D. Thomas, and A. Sugimoto
    • Journal Title

      Proc. of IEEE International Conference on Multimedia and Expo (ICME2017)

      Volume: 1 Pages: 1285-1290

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Deeply Supervised 3D Recurrent FCN for Salient Object Detection in Videos2017

    • Author(s)
      T.-N. Le, and A. Sugimoto
    • Journal Title

      Proc. of British Machine Vision Conference (BMVC2017)

      Volume: 1

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Efficiently Updating Feasible Regions for Fitting Discrete Polynomial Curve2017

    • Author(s)
      F. Sekiya and A. Sugimoto
    • Journal Title

      Proc. of 20th International Conference on Discrete Geometry for Computer Imagery (DGCI2017)

      Volume: LNCS 10502 Pages: 254-266

    • DOI

      10.1007/978-3-319-66272-5_21

    • ISBN
      9783319662718, 9783319662725
    • Related Report
      2017 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Synthesis of Environment Maps for Mixed Reality2017

    • Author(s)
      D. R. Walton, D. Thomas, A. Steed, and A. Sugimoto
    • Journal Title

      Proc. of IEEE International Symposium on Mixed and Augmented Reality (ISMAR2017)

      Volume: 1 Pages: 72-81

    • DOI

      10.1109/ismar.2017.24

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] On Properties of Analytical Approximation of Morphological Discretization for 2D curve and 3D surface2017

    • Author(s)
      F. Sekiya and A. Sugimoto
    • Journal Title

      Mathematical Morphology: Theory and Applications

      Volume: 2 Pages: 25-34

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Parametric Surface Representation with Bump Image for Dense 3D Modeling Using an RBG-D Camera2017

    • Author(s)
      D. Thomas and A. Sugimoto
    • Journal Title

      International Journal of Computer Vision

      Volume: 印刷中 Issue: 2 Pages: 206-225

    • DOI

      10.1007/s11263-016-0969-3

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] Modeling large-scale indoor scenes with rigid fragments using RGB-D cameras2017

    • Author(s)
      D. Thomas and A. Sugimoto
    • Journal Title

      Computer Vision and Image Understanding

      Volume: 印刷中 Pages: 103-116

    • DOI

      10.1016/j.cviu.2016.11.008

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Degeneracies in Rolling Shutter SfM2016

    • Author(s)
      C. Albl, A. Sugimoto, and T. Pajdla
    • Journal Title

      Proc. of European Conference on Computer Vision (ECCV2016)

      Volume: Part V (LNCS Vol. 9909) Pages: 36-51

    • DOI

      10.1007/978-3-319-46454-1_3

    • ISBN
      9783319464534, 9783319464541
    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Room Reconstruction from a Single Spherical Image by Higher-order Energy Minimization2016

    • Author(s)
      K. Fukano, Y. Mochizuki, S. Iizuka, E. Simo-Sera, A. Sugimoto, and H. Ishikawa
    • Journal Title

      Proc. of International Conference on Pattern Recognition (ICPR2016)

      Volume: 1 Pages: 1768-1773

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed
  • [Journal Article] A 3D Reconstruction with High Density and Accuracy using Laser Profiler and Camera Fusion System on a Rover2016

    • Author(s)
      R. Ishikawa, M. Roxas, Y. Sato, T. Oishi, T. Masuda, K. Ikeuchi
    • Journal Title

      Proc. International Conference on 3D Vision (3DV2016)

      Volume: 1 Pages: 620-628

    • DOI

      10.1109/3dv.2016.70

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] Outdoor Omnidirectional Video Completion via Depth Estimation by Motion Analysis2016

    • Author(s)
      C. Morales, S. Ono, Y. Okamoto, M. Roxas, T. Oishi and K. Ikeuchi
    • Journal Title

      Proc. of International Conference on Pattern Recognition (ICPR2016)

      Volume: 1 Pages: 3934-3939

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed
  • [Presentation] Semantic Instance Meets Salient Object: Study on Video Semantic Salient Instance Segmentation2019

    • Author(s)
      T.-N. Le and A. Sugimoto
    • Organizer
      IEEE Winter Conference on Applications of Computer Vision (WACV2019)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] SegmentedFusion: 3D Human Body Reconstruction using Stitched Bounding Boxes2018

    • Author(s)
      Y. Shih-Hsuan, D. Thomas, A. Sugimoto, S.-H. Lai, and R. Taniguchi
    • Organizer
      The 6th International Conference on 3D Vision (3DV2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Linear Solution to the Minimal Absolute Pose Rolling Shutter Problem2018

    • Author(s)
      Zuzana Kukelova, Cenek Albl, Akihiro Sugimoto, and Tomas Pajdla
    • Organizer
      Asian Conference on Computer Vision (ACCV2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Occlusion Handling using Semantic Segmentation and Visibility-Based Rendering for Mixed Reality2018

    • Author(s)
      M. Roxas, T. Hori, T. Fukiage, Y. Okamoto, T. Oishi
    • Organizer
      ACM Symposium on Virtual Reality Software and Technology
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] LiDAR and Camera Calibration using Motion Estimated by Sensor Fusion Odometry2018

    • Author(s)
      R. Ishikawa, T. Oishi, K. Ikeuchi
    • Organizer
      IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Interpolation and Extrapolation of LiDAR Data using Camera Images2018

    • Author(s)
      A. Hirata, R. Ishikawa, M. Roxas, T. Oishi
    • Organizer
      The 13th International Workshop on Robust Computer Vision
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 3D Vision for Mobility2018

    • Author(s)
      T. Oishi
    • Organizer
      2nd International MIS workshop on 3D Vision and Robotics
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Real-Time Simultaneous 3D Reconstruction and Optical Flow Estimation2018

    • Author(s)
      M. Roxas and T. Oishi
    • Organizer
      IEEE Winter Conference on Applications of Computer Vision (WACV)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Real-Time Simultaneous 3D Reconstruction and Optical Flow Estimation2018

    • Author(s)
      M. Roxas, T. Oishi
    • Organizer
      The 12th International Workshop on Robust Computer Vision
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Robust Visual Localization using Effective Feature Points in Dense 3D Map2018

    • Author(s)
      Y. Ike, R. Menandro, Y. Okamoto, T. Oishi
    • Organizer
      The 12th International Workshop on Robust Computer Vision
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Fast 3D Point Cloud Segmentation using Supervoxels with Geometry and Color for 3D Scene Understanding2017

    • Author(s)
      F. Verdoja, D. Thomas, and A. Sugimoto
    • Organizer
      IEEE International Conference on Multimedia and Expo (ICME2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Deeply Supervised 3D Recurrent FCN for Salient Object Detection in Videos2017

    • Author(s)
      T.-N. Le, and A. Sugimoto
    • Organizer
      ritish Machine Vision Conference (BMVC2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Efficiently Updating Feasible Regions for Fitting Discrete Polynomial Curve2017

    • Author(s)
      F. Sekiya and A. Sugimoto
    • Organizer
      The 20th International Conference on Discrete Geometry for Computer Imagery (DGCI2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Synthesis of Environment Maps for Mixed Reality2017

    • Author(s)
      D. R. Walton, D. Thomas, A. Steed, and A. Sugimoto
    • Organizer
      IEEE International Symposium on Mixed and Augmented Reality (ISMAR2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Accurate Position and Pose Estimation for Mobile Platforms using Dense 3D Model2017

    • Author(s)
      T. Oishi
    • Organizer
      Korea-Japan Workshop on Robotics and Information Technology for Better Quality of Life
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 追跡に効果的な特徴量による頑健な位置合わせ2017

    • Author(s)
      池友太, 岡本泰英, 大石岳史
    • Organizer
      第53回 複合現実感研究会
    • Related Report
      2017 Annual Research Report
  • [Presentation] Variational Optical Flow with 3D Smoothness Constraint for a Single Moving Camera2016

    • Author(s)
      Menandro Roxas, Tomoki Hori, and Takeshi Oishi
    • Organizer
      The 11th International Workshop on Robust Computer Vision
    • Place of Presentation
      デジョン(韓国)
    • Year and Date
      2016-12-17
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Panoramic Video Completion from Depth Recovery by Pixel Motion Assessment2016

    • Author(s)
      Carlos Morales, Shintaro Ono, Yasuhide Okamoto, Menandro Roxas, Takeshi Oishi and Katsushi Ikeuchi
    • Organizer
      The 11th International Workshop on Robust Computer Vision
    • Place of Presentation
      デジョン(韓国)
    • Year and Date
      2016-12-17
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] A Sensor Fusion 3D Reconstruction System using Depth-based Triangulation and Multimodal Registration2016

    • Author(s)
      Ryoichi Ishikawa, Menandro Roxas, Yoshihiro Sato, Takeshi Oishi, Takeshi Masuda, Katsushi Ikeuchi
    • Organizer
      The 11th International Workshop on Robust Computer Vision
    • Place of Presentation
      デジョン(韓国)
    • Year and Date
      2016-12-17
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Occlusion Handling by Semantic Segmentation and Transparency Blending for Outdoor Mixed Reality2016

    • Author(s)
      Tomoki Hori, Taiki Fukiage, Roxas Menandro, Yasuhide Okamato and Takeshi Oishi
    • Organizer
      The 11th International Workshop on Robust Computer Vision
    • Place of Presentation
      デジョン(韓国)
    • Year and Date
      2016-12-17
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Robust Motion Estimation for MR Mobility System with Multiple Panoramic Cameras2016

    • Author(s)
      Yasuhide Okamoto, Kosuke Fujimoto, Takeshi Oishi and Katsushi Ikeuchi
    • Organizer
      The 11th International Workshop on Robust Computer Vision
    • Place of Presentation
      デジョン(韓国)
    • Year and Date
      2016-12-17
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Room Reconstruction from a Single Spherical Image by Higher-order Energy Minimization2016

    • Author(s)
      K. Fukano, Y. Mochizuki, S. Iizuka, E. Simo-Sera, A. Sugimoto, and H. Ishikawa
    • Organizer
      International Conference on Pattern Recognition
    • Place of Presentation
      カンクン(メキシコ)
    • Year and Date
      2016-12-04
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Outdoor Omnidirectional Video Completion via Depth Estimation by Motion Analysis2016

    • Author(s)
      C. Morales, S. Ono, Y. Okamoto, M. Roxas, T. Oishi and K. Ikeuchi
    • Organizer
      International Conference on Pattern Recognition
    • Place of Presentation
      カンクン(メキシコ)
    • Year and Date
      2016-12-04
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] A 3D Reconstruction with High Density and Accuracy using Laser Profiler and Camera Fusion System on a Rover2016

    • Author(s)
      R. Ishikawa, M. Roxas, Y. Sato, T. Oishi, T. Masuda, K. Ikeuchi
    • Organizer
      International Conference on 3D Vision
    • Place of Presentation
      スタンフォード(アメリカ)
    • Year and Date
      2016-10-25
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Degeneracies in Rolling Shutter SfM2016

    • Author(s)
      C. Albl, A. Sugimoto, and T. Pajdla
    • Organizer
      European Conference on Computer Vision
    • Place of Presentation
      アムステルダム(オランダ)
    • Year and Date
      2016-10-08
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research

URL: 

Published: 2016-04-21   Modified: 2023-03-16  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi