Audio-Visual Integration to Target Recognition by Drone Audition

2019 Fiscal Year Final Research Report

• Project/Area Number: 17K00365
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Research Field: Intelligent robotics
• Research Institution: Kumamoto University
• Principal Investigator: Kumon Makoto 熊本大学, 大学院先端科学研究部(工), 准教授 (70332864)
• Co-Investigator(Kenkyū-buntansha): 中臺 一博 東京工業大学, 工学院, 特任教授 (70436715)
• Project Period (FY): 2017-04-01 – 2020-03-31
• Keywords: ロボット聴覚 / 視聴覚統合 / センサフュージョン / ドローン聴覚

Outline of Final Research Achievements

In this study, it is considered to recognize targets on the ground from drones with microphones. The target acoustic signal obtained at the drone is generally significantly distorted by the ego-noise, and, hence, it is difficult to recognize the target only by acoustic signals. This study aims to develop the technology to compensate this difficulty by incorporating visual sensor information.
Acoustic features that contain pauses is fused with visual features that are normally provided sequentially where it is not trivial to associate the visual information with the acoustic target.
Based on the developed methods, it is shown that audio-visual integration improves the audio target recognition under noisy situation, and as an example, three-dimensional position estimation of moving plural targets by the drone with microphones was achieved.

Free Research Field

ロボット工学

Academic Significance and Societal Importance of the Research Achievements

本来音によって特徴づけられる音源について、一定の条件の下で画像という異なるセンサ情報（外見の分からない音源）を通じて認識する出来るようになったことを通じ、様々なセンサ情報を自律的に統合する方向へと展開が可能で意義があると考えている。異常検知や、防犯等、社会一般で必要とされる技術としても利用可能である。