A new data-driven approach to bring humanity into virtual worlds with computer vision

Research Project

Project/Area Number	23K28129
Project/Area Number (Other)	23H03439 (2023)
Research Category	Grant-in-Aid for Scientific Research (B)
Allocation Type	Multi-year Fund (2024) Single-year Grants (2023)
Section	一般
Review Section	Basic Section 61020:Human interface and interaction-related Basic Section 62040:Entertainment and game informatics-related Sections That Are Subject to Joint Review: Basic Section61020:Human interface and interaction-related , Basic Section62040:Entertainment and game informatics-related
Research Institution	Kyushu University
Principal Investigator	THOMAS DIEGO 九州大学, システム情報科学研究院, 准教授 (10804651)
Co-Investigator(Kenkyū-buntansha)	鍛冶静雄九州大学, マス・フォア・インダストリ研究所, 教授 (00509656) 古賀靖子九州大学, 人間環境学研究院, 准教授 (60225399) 川崎洋九州大学, システム情報科学研究院, 教授 (80361393) 落合啓之九州大学, マス・フォア・インダストリ研究所, 教授 (90214163)
Project Period (FY)	2023-04-01 – 2026-03-31
Project Status	Granted (Fiscal Year 2024)
Budget Amount *help	¥18,590,000 (Direct Cost: ¥14,300,000、Indirect Cost: ¥4,290,000) Fiscal Year 2025: ¥5,200,000 (Direct Cost: ¥4,000,000、Indirect Cost: ¥1,200,000) Fiscal Year 2024: ¥7,800,000 (Direct Cost: ¥6,000,000、Indirect Cost: ¥1,800,000) Fiscal Year 2023: ¥5,590,000 (Direct Cost: ¥4,300,000、Indirect Cost: ¥1,290,000)
Keywords	デジタルヒューマン / 4D capture / Immersive VR / Differentiable rendering / AI-based 3D modeling / Digital humans
Outline of Research at the Start	The core idea is to leverage RGB-D data to achieve realism of digital humans. We propose: (1) Weakly supervised method to learn 3D shape from in-the-wild RGB-D data. (2) Real-time animation model to learn motion from RGB-D videos. (3) Data-driven method to learn semantically appropriate body movements.