Unifying multiple RGB and depth cameras for real-time large-scale dynamic 3D modeling with unmanned micro aerial vehicles
| Project/Area Number |
19K20297
|
|---|
| Research Category |
Grant-in-Aid for Early-Career Scientists
|
| Allocation Type | Multi-year Fund |
|---|
| Review Section |
Basic Section 61010:Perceptual information processing-related
|
|---|
| Research Institution | Kyushu University |
|---|
Principal Investigator |
THOMAS DIEGO 九州大学, システム情報科学研究院, 助教 (10804651)
|
|---|
| Project Period (FY) |
2019-04-01 – 2021-03-31
|
| Project Status |
Granted (Fiscal Year 2019)
|
| Budget Amount *help |
¥4,160,000 (Direct Cost: ¥3,200,000、Indirect Cost: ¥960,000)
Fiscal Year 2020: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Fiscal Year 2019: ¥2,730,000 (Direct Cost: ¥2,100,000、Indirect Cost: ¥630,000)
|
|---|
| Keywords | RGB―D SLAM / Unmanned vehicle / sensor fusion / dynamic modeling / real-time / RGB-D SLAM / unmanned vehicle |
|---|
| Outline of Research at the Start |
Various sensors from unmanned aerial vehicles will be unified to reconstruct large-scale outdoor dynamic 3D scenes in real-time.
(1)Compact RGB and depth cameras will be fused to incrementally reconstruct outdoor scenes in a coarse-to-fine strategy.
(2)NURBS surfaces will maintain the large-scale evolving 3D model while displacement mapping will generate detailed 3D geometry.
(3)Physical and semantic constraints in the scene will be used to enforce at all time a globally consistent 3D reconstruction with loops closed and minimal registration errors.
|
| Outline of Annual Research Achievements |
I investigated techniques for 3D sensing from 2D videos. Our proposed method for reconstructing dense 3D mesh from mobile phones was published at the international conference on 3D vision 3DV2019.
I investigated theory-supported depth fusion methods for a unified 3D mapping framework that does not depend on the type of 3D sensor used. We proposed to formulate the depth fusion problem into the variational message passing framework. Our proposed method for robust real time depth measurement fusion was published at the international conference on 3D vision 3DV2019.
I pursued research on 3D shape estimation from a single color image. The results of this research have been accepted for publication at the international conference on Computer Vision and Pattern Recognition CVPR 2020.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
It is necessary to combine standard color cameras with depth sensors in a unified 3D reconstruction framework for outside 3D mapping. Our results (published at 3DV 2019) have shown promising possibilities for 3D mapping when using 2D videos.
We proposed a new theory supported depth fusion method that is robust and only require a statistical characteristic of the sensors. Our results (published at 3DV2019) unlock the possibility to merge different sensors into a unified 3D mapping framework.
In some specific situations such as the 3D reconstruction of well-known objects like the human body, deep learning has proven to be a powerful tool. We proposed a new method and demonstrated (in our paper accepted at CVPR2020) impressive performances on publicly available datasets.
|
| Strategy for Future Research Activity |
The remaining tasks to achieve these goals are three-fold:
(1)Fusion of RGB and depth sensors embedded on a micro aerial vehicle (MAV). Four RGB cameras will be mounted in a rig fashion around the MAV and all data will be fused by using a unified framework based on our proposed Variational Message Passing depth fusion method.
(2)Different objects in the scene will be segmented out and one surface will be fitted to each object. When new data is captured, surface deformations will be estimated and the 3D model will be updated and enlarged if necessary.
(3)We will extract semantic information for each reconstructed object. We will evaluate the performance of our system with real-world experiments on large scale indoor-outdoor dynamic 3D scene reconstruction in the campus of the university.
|
Report
(1 results)
Research Products
(6 results)
