• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Unifying multiple RGB and depth cameras for real-time large-scale dynamic 3D modeling with unmanned micro aerial vehicles

Research Project

Project/Area Number 19K20297
Research Category

Grant-in-Aid for Early-Career Scientists

Allocation TypeMulti-year Fund
Review Section Basic Section 61010:Perceptual information processing-related
Research InstitutionKyushu University

Principal Investigator

Thomas Diego  九州大学, システム情報科学研究院, 助教 (10804651)

Project Period (FY) 2019-04-01 – 2021-03-31
Project Status Completed (Fiscal Year 2020)
Budget Amount *help
¥4,160,000 (Direct Cost: ¥3,200,000、Indirect Cost: ¥960,000)
Fiscal Year 2020: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Fiscal Year 2019: ¥2,730,000 (Direct Cost: ¥2,100,000、Indirect Cost: ¥630,000)
KeywordsRGB-D SLAM / Aerial 3D capture / Dynamic scene / Aritificial intelligence / Deep neural network / 3D human body / Aerial drone / outdoor scene / 3D reconstruction / sensor fusion / RGB―D SLAM / Unmanned vehicle / dynamic modeling / real-time / unmanned vehicle
Outline of Research at the Start

Various sensors from unmanned aerial vehicles will be unified to reconstruct large-scale outdoor dynamic 3D scenes in real-time.
(1)Compact RGB and depth cameras will be fused to incrementally reconstruct outdoor scenes in a coarse-to-fine strategy.
(2)NURBS surfaces will maintain the large-scale evolving 3D model while displacement mapping will generate detailed 3D geometry.
(3)Physical and semantic constraints in the scene will be used to enforce at all time a globally consistent 3D reconstruction with loops closed and minimal registration errors.

Outline of Final Research Achievements

The goal of this project was to develop a real-time system for 3D reconstruction of dynamic scenes with unmanned micro aerial vehicles.
We proposed a new method for robust and accurate fusion of depth images without increasing computational speed. We also proposed a method that can handle dynamic scenes like a moving person. This was achieved by jointly optimizing non rigid motion and geometry. To handle situations when depth information is not available, we proposed a solution for 3D shape estimation from a single RGB image. We focused on the case of the human body and proposed a new deep neural network to reconstruct detailed shapes of humans wearing loose clothes from single RGB images.
We proposed a new 3D scanning system equipped on a consumer-grade aerial drone that can capture live sequences of RGB-D data. our proposed system consists of a minicomputer powered by a portable battery and an RGB-D camera. We shared material and code and captured real world data with our system.

Academic Significance and Societal Importance of the Research Achievements

単一の画像から人体の緻密で詳細な3Dモデルを作成する最初の方法の1つを提案しました。 コンピュータビジョンのトップ国際会議で私たちの仕事を発表し、コードを一般に公開しました。 制御されていない環境での3Dシーン再構築の新しい可能性を開きます。
このシステムを消費者向けの空中ドローンに簡単に装備するためのソリューションと、それを制御するソフトウェアを紹介します。 実世界のデータをキャプチャし、システムを使用していくつかの最先端のRGB-DSLAM技術を評価しました。 空中3Dスキャンとマッピングの研究開発を後押しできるように、すべてのデータとコードをコミュニティで公開しました。

Report

(3 results)
  • 2020 Annual Research Report   Final Research Report ( PDF )
  • 2019 Research-status Report
  • Research Products

    (11 results)

All 2020 2019 Other

All Int'l Joint Research (3 results) Journal Article (3 results) (of which Int'l Joint Research: 3 results,  Peer Reviewed: 3 results) Presentation (4 results) (of which Int'l Joint Research: 3 results,  Invited: 1 results) Remarks (1 results)

  • [Int'l Joint Research] Gustave Eiffel University(フランス)

    • Related Report
      2020 Annual Research Report
  • [Int'l Joint Research] Standford University(米国)

    • Related Report
      2020 Annual Research Report
  • [Int'l Joint Research] Gustave Eiffel University(フランス)

    • Related Report
      2019 Research-status Report
  • [Journal Article] TetraTSDF: 3D Human Reconstruction From a Single Image With a Tetrahedral Outer Shell2020

    • Author(s)
      Hayato Onizuka, Zehra Hayirci, Diego Thomas, Akihiro Sugimoto, Hideaki Uchiyama, Rin-ichiro Taniguchi;
    • Journal Title

      the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

      Volume: 1 Pages: 6011-6020

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Mobile Photometric Stereo Combined with SLAM for Dense 3D Reconstruction2019

    • Author(s)
      Maxence Remy; Hideaki Uchiyama; Hiroshi Kawasaki; Diego Thomas; Vincent Nozick; Hideo Saito
    • Journal Title

      International Conference on 3D Vision

      Volume: - Pages: 574-582

    • DOI

      10.1109/3dv.2019.00044

    • Related Report
      2019 Research-status Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction2019

    • Author(s)
      Maxence Remy、Uchiyama Hideaki、Kawasaki Hiroshi、Thomas Diego、Nozick Vincent、Saito Hideo
    • Journal Title

      2019 International Conference on 3D Vision (3DV)

      Volume: 0 Pages: 574-582

    • DOI

      10.1109/3dv.2019.00069

    • Related Report
      2019 Research-status Report
    • Peer Reviewed / Int'l Joint Research
  • [Presentation] TetraTSDF: 3D Human Reconstruction From a Single Image With a Tetrahedral Outer Shell2020

    • Author(s)
      Hayato Onizuka, Zehra Hayirci, Diego Thomas
    • Organizer
      the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Merging SLAM and photometric stereo for 3D reconstruction with a moving camera2019

    • Author(s)
      Maxence Remy, Hideo Saito, Hideaki Uchiyama, Hiroshi Kawasaki, Vincent Nozick, Diego Thomas
    • Organizer
      25th International Workshop on Frontiers of Computer Vision
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] Regression of 3D human body shapes from a single image2019

    • Author(s)
      Hayato Onizuka, Diego Thomas, Zehra Hayirci, Akihiro Sugimoto, Hideaki Uchiyama, Rin-ichiro
    • Organizer
      Machine Perception and Robotics 2019
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] 3D human body reconstruction using RGB-D camera2019

    • Author(s)
      Diego Thomas
    • Organizer
      Asia Pacific Society for Computing and Information Technology 2019 Annual Meeting (APSCIT 2019 Annual Meeting)
    • Related Report
      2019 Research-status Report
    • Invited
  • [Remarks] Computer Vision & Graphics Laboratory

    • URL

      https://www.cvg.ait.kyushu-u.ac.jp

    • Related Report
      2020 Annual Research Report

URL: 

Published: 2019-04-18   Modified: 2022-01-27  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi