| Project/Area Number |
23K28129
|
|---|
| Project/Area Number (Other) |
23H03439 (2023)
|
|---|
| Research Category |
Grant-in-Aid for Scientific Research (B)
|
| Allocation Type | Multi-year Fund (2024)
Single-year Grants (2023) |
|---|
| Section | 一般 |
|---|
| Review Section |
Basic Section 61020:Human interface and interaction-related
Basic Section 62040:Entertainment and game informatics-related
Sections That Are Subject to Joint Review: Basic Section61020:Human interface and interaction-related , Basic Section62040:Entertainment and game informatics-related
|
|---|
| Research Institution | Kyushu University |
|---|
Principal Investigator |
THOMAS DIEGO 九州大学, システム情報科学研究院, 准教授 (10804651)
|
|---|
| Co-Investigator(Kenkyū-buntansha) |
鍛冶 静雄 九州大学, マス・フォア・インダストリ研究所, 教授 (00509656)
古賀 靖子 九州大学, 人間環境学研究院, 准教授 (60225399)
川崎 洋 九州大学, システム情報科学研究院, 教授 (80361393)
落合 啓之 九州大学, マス・フォア・インダストリ研究所, 教授 (90214163)
|
|---|
| Project Period (FY) |
2024-04-01 – 2026-03-31
|
| Project Status |
Granted (Fiscal Year 2024)
|
| Budget Amount *help |
¥18,590,000 (Direct Cost: ¥14,300,000、Indirect Cost: ¥4,290,000)
Fiscal Year 2025: ¥5,200,000 (Direct Cost: ¥4,000,000、Indirect Cost: ¥1,200,000)
Fiscal Year 2024: ¥7,800,000 (Direct Cost: ¥6,000,000、Indirect Cost: ¥1,800,000)
Fiscal Year 2023: ¥5,590,000 (Direct Cost: ¥4,300,000、Indirect Cost: ¥1,290,000)
|
|---|
| Keywords | Digital Humans / Differentiable rendering / 3D reconstruction / Neural Surface / Motion generation / デジタルヒューマン / 4D capture / Immersive VR / AI-based 3D modeling / Animatable Avatar / Deep learning / Weak supervision / Digital humans |
|---|
| Outline of Research at the Start |
The core idea is to leverage RGB-D data to achieve realism of digital humans. We propose:
(1) Weakly supervised method to learn 3D shape from in-the-wild RGB-D data.
(2) Real-time animation model to learn motion from RGB-D videos.
(3) Data-driven method to learn semantically appropriate body movements.
|
| Outline of Annual Research Achievements |
About learning detailed clothed humans 3D models, we have developed a multi-view 3D reconstruction technique that can achieve millimeter-level accuracy in minutes and synthesize state-of-the-art novel viewpoints in real time (SIGGRAPH ASIA 2024). About 3D human body reconstruction from sparse views we have proposed a novel deep learning-based approach to the using weak supervision via 2D normal maps (oral presentation at WACV 2025). We have also developed an alternative discretization strategy with the Centroidal Voronoi Tesselation (CVT), which allows high 3D reconstruction quality (oral at WACV 2025 and Nagano award at MIRU 2024). About human motion synthesis, we have proposed a method of training using only existing datasets that do not contain scene information (award at MIRU 2024).
|
| Current Status of Research Progress |
Current Status of Research Progress
1: Research has progressed more than it was originally planned.
Reason
Our research was presented at top-tier international conferences, showcasing our findings through oral presentations. At the 27th Meeting on Image Recognition and Understanding (MIRU2024), our work received significant recognition, winning both the Nagano Award for Best Paper and the MIRU Excellent Paper Award for Second Best Paper. These prestigious accolades highlight the innovation, technical excellence, and impact of our research in the field of image recognition and understanding. Our achievements at MIRU2024 reflect the dedication and collaborative effort invested in advancing the frontiers of this domain.
|
| Strategy for Future Research Activity |
This year we will develop cutting-edge technology to reduce computational costs in 3D reconstruction through adaptive tessellation, capture texture and material properties for photorealistic rendering, and populate 3D scenes with animated human bodies for interactive VR deployment.
(1-C) In order to address the objective of instantaneous 3D reconstruction, the applicant will propose adaptive tessellation of the 3D space to reduce computational cost while maintaining level of details.
(2-C) Texture and material properties of skin and clothes will be captured to enable photorealistic rendering of animations.
(3-C) 3D scenes will be populated with animated 3D human bodies that interact with the scene in a semantically correct manner. The system will be deployed on head mounted display VR device.
|
