A study of automatic English caption generation for Japanese lecture speech

2017 Fiscal Year Final Research Report

• Project/Area Number: 25280062
• Research Category: Grant-in-Aid for Scientific Research (B)
• Allocation Type: Partial Multi-year Fund
• Section: 一般
• Research Field: Perceptual information processing
• Research Institution: Toyohashi University of Technology
• Principal Investigator: nakagawa seiichi 豊橋技術科学大学, リーディング大学院教育推進機構, 特命教授 (20115893)
• Co-Investigator(Kenkyū-buntansha): 秋葉 友良 豊橋技術科学大学, 工学(系)研究科(研究院), 准教授 (00356346) ; 井佐原 均 豊橋技術科学大学, 情報メディア基盤センター, 教授 (20358881) ; 山本 一公 中部大学, 工学部, 准教授 (40324230) ; 土屋 雅稔 豊橋技術科学大学, 情報メディア基盤センター, 准教授 (70378256)
• Project Period (FY): 2013-04-01 – 2018-03-31
• Keywords: 講義の字幕化 / 講義音声 / 日本語講義 / 音声認識 / 日英翻訳 / 英日翻訳 / 音声翻訳

Outline of Final Research Achievements

First of all, we developed Japanese and English lecture speech recognition systems based on DNN. Next, we studied on English to Japanese translation for lecture speech. We transcribed English lectures by using plural English recognition systems and translated to Japanese based on a statistical machine translation system (SMT). Then, we selected the best translation from plural translation candidates by using a re-scoring technique based on language models and improved the translation result. For attacking mis-recognition errors, we simulated speech recognizers and used them as the training data for the SMT.
By 1-year extension of the research period, we developed a neural machine translation system (NMT), and combine SMT and NMT. By re-scoring based on back translation for these translation candidates, we improved remarkably the translation performance.
Finally, we applied the above techniques to the translation from Japanese lecture speech to English caption.

Free Research Field

音声言語情報処理