2019 Fiscal Year Research-status Report
Unifying multiple RGB and depth cameras for real-time large-scale dynamic 3D modeling with unmanned micro aerial vehicles
| Project/Area Number |
19K20297
|
|---|
| Research Institution | Kyushu University |
|---|
Principal Investigator |
THOMAS DIEGO 九州大学, システム情報科学研究院, 助教 (10804651)
|
|---|
| Project Period (FY) |
2019-04-01 – 2021-03-31
|
| Keywords | RGB―D SLAM / Unmanned vehicle / sensor fusion / dynamic modeling / real-time |
|---|
| Outline of Annual Research Achievements |
I investigated techniques for 3D sensing from 2D videos. Our proposed method for reconstructing dense 3D mesh from mobile phones was published at the international conference on 3D vision 3DV2019.
I investigated theory-supported depth fusion methods for a unified 3D mapping framework that does not depend on the type of 3D sensor used. We proposed to formulate the depth fusion problem into the variational message passing framework. Our proposed method for robust real time depth measurement fusion was published at the international conference on 3D vision 3DV2019.
I pursued research on 3D shape estimation from a single color image. The results of this research have been accepted for publication at the international conference on Computer Vision and Pattern Recognition CVPR 2020.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
It is necessary to combine standard color cameras with depth sensors in a unified 3D reconstruction framework for outside 3D mapping. Our results (published at 3DV 2019) have shown promising possibilities for 3D mapping when using 2D videos.
We proposed a new theory supported depth fusion method that is robust and only require a statistical characteristic of the sensors. Our results (published at 3DV2019) unlock the possibility to merge different sensors into a unified 3D mapping framework.
In some specific situations such as the 3D reconstruction of well-known objects like the human body, deep learning has proven to be a powerful tool. We proposed a new method and demonstrated (in our paper accepted at CVPR2020) impressive performances on publicly available datasets.
|
| Strategy for Future Research Activity |
The remaining tasks to achieve these goals are three-fold:
(1)Fusion of RGB and depth sensors embedded on a micro aerial vehicle (MAV). Four RGB cameras will be mounted in a rig fashion around the MAV and all data will be fused by using a unified framework based on our proposed Variational Message Passing depth fusion method.
(2)Different objects in the scene will be segmented out and one surface will be fitted to each object. When new data is captured, surface deformations will be estimated and the 3D model will be updated and enlarged if necessary.
(3)We will extract semantic information for each reconstructed object. We will evaluate the performance of our system with real-world experiments on large scale indoor-outdoor dynamic 3D scene reconstruction in the campus of the university.
|
| Causes of Carryover |
We need to build an actual prototype and evaluate our proposed system. We are going to buy a MAV onto which we will embed 4 RGB-D cameras (the Intel RealSense cameras). We already have the RGB-D cameras in the laboratory, so we only need to buy the MAV.
To build the prototype, calibrate the cameras and manipulate the drone is a tedious task that requires some implementation skills. Therefore, we will hire an internship student for 4 months to help build the prototype. The objectives of the internship will be: (1) design the prototype; (2) calibrate the cameras; (3) capture raw data.
|
