Human behavior sensing by integrated analysis of first, second, and third-person point of view videos

2023 Fiscal Year Final Research Report

• Project/Area Number: 20H04205
• Research Category: Grant-in-Aid for Scientific Research (B)
• Allocation Type: Single-year Grants
• Section: 一般
• Review Section: Basic Section 61010:Perceptual information processing-related
• Research Institution: The University of Tokyo
• Principal Investigator: Sato Yoichi 東京大学, 生産技術研究所, 教授 (70302627)
• Co-Investigator(Kenkyū-buntansha): 菅野 裕介 東京大学, 生産技術研究所, 准教授 (10593585)
• Project Period (FY): 2020-04-01 – 2024-03-31
• Keywords: 一人称視点映像解析 / 人物行動理解

Outline of Final Research Achievements

In this project, we conducted research on the sensing of human behavior by integrated analysis of videos from different viewpoints (self-viewpoint, other viewpoint, and fixed viewpoint) in four aspects: 1. Feature learning for comprehensive analysis of multiple viewpoint videos by self-supervised learning, 2. Spatio-temporal alignment of videos from different viewpoints, 3. Human behavior understanding by integration of self-viewpoint and fixed viewpoint videos, and 4. Interaction analysis by integration of self-viewpoint and other viewpoint videos.

Free Research Field

computer vision

Academic Significance and Societal Importance of the Research Achievements

映像に基づく人物行動理解技術は、様々なアプリケーションドメインにおいて書くことが出来ない重要な基盤技術であり、これまでもコンピュータビジョンの分野において活発に研究が進められてきた。これまでは、固定視点映像、自己視点映像、他者視点映像という異なる視点映像が別々に扱われていたのに対し、本研究課題では、他に先駆けて固定視点・自己視点・他者視点という異なる視点映像の統合解析に取り組み、映像に基づく人物行動理解の高度化に貢献する成果を得ることができた。