A study for displaying timing between verbal and non-verbal information for a real-time speech-to-caption system

Research Project

Project/Area Number	23501096
Research Category	Grant-in-Aid for Scientific Research (C)
Allocation Type	Multi-year Fund
Section	一般
Research Field	Educational technology
Research Institution	Tsukuba University of Technology
Principal Investigator	KUROKI Hayato 筑波技術大学, 産業技術学部, 教授 (00345159)
Co-Investigator(Renkei-kenkyūsha)	IFUKUBE Tohru 東京大学, 高齢社会総合研究機構, 名誉教授 (70002102) NAKANO Satoko 広島大学, アクセシビリティセンター, 特任講師 (20359665)
Project Period (FY)	2011-04-28 – 2015-03-31
Project Status	Completed (Fiscal Year 2014)
Budget Amount *help	¥4,940,000 (Direct Cost: ¥3,800,000、Indirect Cost: ¥1,140,000) Fiscal Year 2014: ¥1,040,000 (Direct Cost: ¥800,000、Indirect Cost: ¥240,000) Fiscal Year 2013: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000) Fiscal Year 2012: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000) Fiscal Year 2011: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000)
Keywords	ヒューマン・インタフェース / 障害者支援 / 高等教育支援 / 教育工学 / 認知科学
Outline of Final Research Achievements	This study aimed to investigate the suitable ways of the display display sequence and display timing between captions that have speech recognition errors and the speaker's face movement images for our developing real-time speech-to-caption system for the hearing-impaired. The results showed that the sequence displaying the caption before the speaker's face image improved the comprehension of the captions. The sequence displaying both simultaneously showed an improvement of only a few percent higher than that of the question sentence, and the sequence displaying the speaker's face image before the caption showed almost no change. In addition, the sequence displaying the caption 1 second before the speaker's face showed the most significant improvement of all the conditions in the hearing-impaired.