Smart drone audition: A search and rescue drone system that listens and communicates

Research Project

Project/Area Number	22KF0141
Project/Area Number (Other)	22F22769 (2022)
Research Category	Grant-in-Aid for JSPS Fellows
Allocation Type	Multi-year Fund (2023) Single-year Grants (2022)
Section	外国
Review Section	Basic Section 20020:Robotics and intelligent system-related
Research Institution	Tokyo Institute of Technology
Principal Investigator	中臺一博東京工業大学, 工学院, 教授 (70436715)
Co-Investigator(Kenkyū-buntansha)	YEN BENJAMIN 東京工業大学, 工学院, 外国人特別研究員
Project Period (FY)	2023-03-08 – 2025-03-31
Project Status	Granted (Fiscal Year 2023)
Budget Amount *help	¥2,200,000 (Direct Cost: ¥2,200,000) Fiscal Year 2024: ¥600,000 (Direct Cost: ¥600,000) Fiscal Year 2023: ¥1,100,000 (Direct Cost: ¥1,100,000) Fiscal Year 2022: ¥500,000 (Direct Cost: ¥500,000)
Keywords	ドローン聴覚 / 音響信号処理 / 深層学習
Outline of Research at the Start	This research aims to develop a drone intended for search and rescue. Equipped with "ears", the drone "listens" to an audible target and, in turn, promotes effective communication with them. "Listening" includes recognising, locating and recording the target sound "clearly" (i.e. free of noise from the drone's rotors and surroundings) under harsh, noisy environments, typical in search and rescue.
Outline of Annual Research Achievements	Dr. Benjamin Yen has integrated some of his research at the University of Auckland into related drone audition projects at the Tokyo Institute of Technology. Specifically, he incorporated techniques from his work in sound source enhancement to improve sound source tracking using multiple drones. This has yielded a conference paper submission to the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems. The paper is currently under review. We have submitted a journal article for drone audition to the IEEE/ACM Transactions on Audio Speech and Language Processing. Specifically, we proposed a noise covariance matrix estimation method that can be applied to sound source enhancement algorithms for drone audition. While this research is developed during Dr. Benjamin Yen's time at the University of Auckland, it is entirely within the research scope of the JSPS postdoctoral fellowship proposal and is a topic that will be further developed during this fellowship. The paper is currently under major revision. We have carried out a major outdoor experiment to collect the necessary data to perform performance evaluation for various drone audition algorithms. This includes accurate audio recordings of drone noise and a moving target sound source. In addition, accurate GPS data of the target sound source's movement is also recorded. These measurements will be helpful in evaluating sound source tracking performance from the algorithms proposed. We have planned a range of research topics related to the research theme to be researched by ourselves and new B4/masters students.
Current Status of Research Progress	Current Status of Research Progress 2: Research has progressed on the whole more than it was originally planned. Reason The first fiscal year mainly involves discovering the relationships between the acoustic transfer function of a controlled (quiet) outdoor environment and a search and rescue environment. Naturally, this involves outdoor acoustic measurements to capture the acoustical influence of the environment towards a flying drone that is performing audition. In this fiscal year, we have successfully carried out outdoor experiments. While the environmental conditions do not strictly represent a search and rescue scenario, we have captured data under quiet and noisy conditions. Naturally, the equipment necessary for the experiments was also purchased as planned following the KAKENHI proposal. Unfortunately, we could not make significant progress in the algorithm development side to discover the acoustic transfer functions of search and rescue environments. However, we have made progress in other research topics as part of the general research theme. This includes the ongoing research in sound source enhancement and sound source tracking for drones, which has yielded a number of paper submissions(1 journal article and two international conference papers). The topic related to acoustic transfer functions is also planned for research in this fiscal year (see Plans for the research scheme). Overall, the research is progressing as planned.
Strategy for Future Research Activity	During the first fiscal year, we have planned a number of topics/directions, briefly summarised as follows: Using rotor noise as an acoustic reference to estimate the distance between the drone and the geometric surface. Using rotor noise as a reference to estimate the acoustic properties or acoustic transfer function of the drone's surrounding environment. Practical evaluation of sound source tracking using multiple drones based on the real-time sound source location. Upcoming B4/masters students will carry out a number of the abovementioned topics while Dr. Benjamin Yen acts as a mentor/advisor. We also note that research on sound source enhancement and localisation is ongoing. Due to practical constraints, such as difficulty in setting up a full-size drone swarm system that can simultaneously record audio and track sound sources in real-time, as well as tight restrictions for drone piloting in Japan, we intend to utilise small-sized drone swarms to perform experiments related to topics involving multiple drones prior to evaluation using full-sized drones. We have investigated a suitable drone system and have completed the purchase. The collective of these research topics, if successful, allows pathways for integration to build a smart drone auditioning system.