Cocktail party effect procesor using audio-visual sensor fusion technique

Research Project

Project/Area Number	08455194
Research Category	Grant-in-Aid for Scientific Research (B)
Allocation Type	Single-year Grants
Section	一般
Research Field	計測・制御工学
Research Institution	The University of Electro-Communications
Principal Investigator	TAKAHASHI Kota The University of Electro-Communications, Associate Professor, 電気通信学部, 助教授 (10188005)
Project Period (FY)	1996 – 1997
Project Status	Completed (Fiscal Year 1997)
Budget Amount *help	¥8,200,000 (Direct Cost: ¥8,200,000) Fiscal Year 1997: ¥4,800,000 (Direct Cost: ¥4,800,000) Fiscal Year 1996: ¥3,400,000 (Direct Cost: ¥3,400,000)
Keywords	image processing / parallel processing / FPGA
Research Abstract	The term "Cocktail party effect" means biological special ability to extract an important signal from other ambient noises. In this research, the cocktail party effect is focused on in order to construct a new type of sensing system. Two dimensional array of logic elements for sensor fusion, i.e., two dimensional sensor fusion system which integrate audio signals and sequential images on time-space coordinates, is proposed in this research. Summaries of the research are as followas. A small prototype system wiht an array of 8x8, which had been made as preliminary system, was extended to a larger system with an array of 24x16 inthis year. Results of experiments using the revised system demonstraed the advantages of the concept of the new system, especially in the case of multiplexed data. Automatic pitch tracking algorithm was also developed in this year, which was used as a preliminary step of sound source localization. Using this algorithm, improvements in the source localization subsystem were demonstrated experimentally. The other subsystem is an image processing part. In this part, a new algorithm using feature points was developed, which wa based on the theory of independent component analysis. Using this type of feature points, improvements in the stereo vision subsystem were demonstrated experimentally. As the consequence of these results, the advantages of the concept of two dimensional logic elements for sensor fusion on time-space coordinates were shown. In the future, the system will be revised in order to evaluate the efficiency of the algorithm more quantitatively.