Acoustic signal processing for real world captioning system via see-through head mounted display

2017 Fiscal Year Final Research Report

• Project/Area Number: 15K20886
• Research Category: Grant-in-Aid for Young Scientists (B)
• Allocation Type: Multi-year Fund
• Research Field: Human interface and interaction; Perceptual information processing
• Research Institution: University of Tsukuba
• Principal Investigator: Zempo Keiichi 筑波大学, システム情報系, 助教 (70725712)
• Project Period (FY): 2015-04-01 – 2018-03-31
• Keywords: ウエアラブル機器 / 聴覚障害支援 / 音響情報処理 / ヒューマンインターフェイス / 感覚代行システム

Outline of Final Research Achievements

The aim of this research is to realize a system that performs natural perceptual substitution to realize information guarantee for people with hearing impairment. In this research, we have realized a real world captioning system that presents subtitles by watching face-to-face dialogue by adding microphone and subtitle presentation application that can handle directionality to a see-through head mounted display (HMD) .
Since ordinary HMD has a limit on the number of microphone input terminals, we developed a microphone array that can handle the beam pattern even with single channel input and developed a signal processing technique for that. For the captioning, we automatically detected the speaker and constructed an AR system that can see conversation information like a speech balloon from the face.

Free Research Field

知覚情報処理