2022 Fiscal Year Annual Research Report
A Unified Computational Model for Audio-Visual Recognition of Human Social Interaction
| Project/Area Number |
20K19833
|
|---|
| Research Institution | Institute of Physical and Chemical Research |
|---|
Principal Investigator |
Nugraha Aditya 国立研究開発法人理化学研究所, 革新知能統合研究センター, 研究員 (60858025)
|
|---|
| Project Period (FY) |
2020-04-01 – 2023-03-31
|
| Keywords | Audio-visual processing / Smart glasses / Adaptive system / Blind source separation / Speech enhancement / Speech recognition / Neural spatial model / Normalizing flow |
|---|
| Outline of Annual Research Achievements |
This study aims to formulate a probabilistic computational model of audio-visual information processing for understanding verbal communication in human social interactions. The model is based on the probabilistic local Gaussian model for a multichannel audio signal, which uses spectral parameters to portray source characteristics, and spatial parameters to represent source and sensor locations in an environment. Initially, we assumed the availability of high-resolution recordings using stationary sensors (microphones and cameras) as in roundtable discussion scenarios. However, we decided to shift our focus to tackling issues in recordings captured by non-stationary sensors of head-worn smartglasses. The users naturally move their heads and bodies in real-world scenarios, especially when interacting with multiple persons in a group. Thus, dealing with the recordings requires a highly adaptive system that is robust to noise and reverberation. In FY2020, we developed multiple deep spectral models, including one based on the speaker and phone disentanglement. In FY2021, we worked on deep spatial models, including one based on an integration of normalizing flow and state-of-the-art joint diagonalization techniques for spatial covariance matrices, and started to incorporate visual aspects into our audio-visual information processing. In FY2022, we continued working on adaptive visually-informed audio signal processing, in which probable speaker locations govern the spatial parameter optimization of audio source separation or speech enhancement for speech recognition purposes.
|
