Audio-visual speech corpus for evaluating speech recognition performance in noisy environments

• Project/Area Number: 19700163
• Research Category: Grant-in-Aid for Young Scientists (B)
• Allocation Type: Single-year Grants
• Research Field: Perception information processing/Intelligent robotics
• Research Institution: Nagoya University
• Principal Investigator: MIYAJIMA Chiyomi Nagoya University, 大学院・情報科学研究科, 助教 (90335092)
• Project Period (FY): 2007 – 2009
• Project Status: Completed (Fiscal Year 2009)
• Budget Amount: ¥3,843,644 (Direct Cost: ¥3,325,880、Indirect Cost: ¥517,764); Fiscal Year 2009: ¥813,644 (Direct Cost: ¥625,880、Indirect Cost: ¥187,764); Fiscal Year 2008: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000); Fiscal Year 2007: ¥1,600,000 (Direct Cost: ¥1,600,000)
• Keywords: バイモーダル音声認識 / データベース / 雑音環境 / 車内雑音 / 雑音下音声認識 / 音声認識性能評価 / 近赤外映像 / 自動車内雑音 / 主成分分析 / オプティカルフロー

Research Abstract

Audio-visual speech data are collected in a silent room and a vehicle for developing an audio-visual speech corpus which is used for evaluating speech recognition performance in noisy environments, especially in in-car environments. Acoustic noise and gamma values of images are used for simulating in-car environments over the recorded data in the silent room. Baseline audio and visual features and an integration method are calibrated in some experimental evaluations. The corpus will be open to the public along with database manuals for research purposes.

Research Products

• [Journal Article] CENSREC-1-C: An evaluation framework for voice activity detection under noisy environments (2009)
  • Author(s): N. Kitaoka, T. Yamada, S. Tsuge, C. Miyajima, K. Yamamoto, T. Nishiura, M. Nakayama, Y. Denda, M. Fujimoto, T. Takiguchi, S. Tamura, S. Matsuda, T. Ogawa, S. Kuroiwa, K. Takeda, S. Nakamura
  • Journal Title: Acoustical Science and Technology Pages: 363-371
  • NAID: 10025992968
  • Peer Reviewed
• [Journal Article] CENSREC-1-C : An evaluation framework for voice activity detection under noisy environments (2009)
  • Author(s): N.Kitaoka, T.Yamada, S.Tsuge, C.Miyajima, K.Yamamoto, T.Nishiura, M.Nakayama, Y.Denda, M.Fujimoto, T.Takiguchi, S.Tamura, S.Matsuda, T.Ogawa, S.Kuroiwa, K.Takeda, S.Nakamura
  • Journal Title: Acoustical Science and Technology 30 Pages: 363-371
  • NAID: 10025992968
  • Peer Reviewed
• [Journal Article] 音声と画像の統合によるドライバの発話区間検出 (2008)
  • Author(s): 二宮芳樹, 坂義秀, 前野俊希, 根木大輔, 宮島千代美, 森健策, 北坂孝幸, 末永康仁
  • Journal Title: 映像情報メディア学会誌 vol.62,no.3 Pages: 435-441
  • NAID: 110006855164
  • Peer Reviewed
• [Journal Article] 多様な音響環境下における音声認識システム利用時のデータ収集システム (2007)
  • Author(s): 原直, 宮島千代美, 伊藤克亘, 武田一哉
  • Journal Title: 電子情報通信学会論文誌 vol.J90-D,no.10 Pages: 1115-1123
  • NAID: 110007380588
  • Peer Reviewed
• [Journal Article] 音声と画像の統合によるドライバの発話区間検出 (2007)
  • Author(s): 二宮芳樹, 坂義秀, 前野俊希, 根木大輔, 宮島千代美, 森健策, 北坂孝幸, 末永康仁
  • Journal Title: 映像情報メディア学会誌 vol.62 Pages: 435-441
  • NAID: 110006855164
  • Peer Reviewed
• [Journal Article] 多様な音響環境下における音声認識システム利用時のデータ収集システム (2007)
  • Author(s): 原直, 宮島千代美, 伊藤克亘, 武田一哉
  • Journal Title: 電子情報通信学会論文誌 vol.J90-D Pages: 1115-1123
  • NAID: 110007380588
  • Peer Reviewed
• [Presentation] CENSREC-1-AV:マルチモーダル音声認識コーパスの構築 (2010)
  • Author(s): 田村哲嗣, 宮島千代美, 北岡教英, 武田一哉, 山田武志, 滝口哲也, 柘植覚, 山本一公, 西浦敬信, 中山雅人, 傳田遊亀, 藤本雅清, 松田繁樹小川哲司, 黒岩眞吾, 中村哲
  • Organizer: 2010年日本音響学会春季研究発表会
  • Place of Presentation: 電気通信大学(東京都)
  • Year and Date: 2010-03-08
• [Presentation] CENSREC- 1-AV:マルチモーダル音声認識コーパスの構築 (2010)
  • Author(s): 田村哲嗣, 宮島千代美, 北岡教英, 武田一哉, 山田武志, 滝口哲也, 柘植覚, 山本一公, 西浦敬信, 中山雅人, 傳田遊亀, 藤本雅清, 松田繁樹, 小川哲司, 黒岩眞吾, 中村哲
  • Organizer: 2010年日本音響学会春季研究発表会
  • Place of Presentation: 調布市
• [Presentation] 複数音響モデルからの最適選択による音声認識 (2009)
  • Author(s): 伊藤新, 原直, 宮島千代美, 北岡教英, 武田一哉
  • Organizer: 2009年電気関係学会東海支部連合大会
  • Place of Presentation: 愛知工業大学(愛知県)
  • Year and Date: 2009-09-10
• [Presentation] 自動車運転コーパスにおける行動観測信号の統合と利用 (2009)
  • Author(s): 武田一哉, 尾崎晃, マルタルーカス, 西脇由博, 宮島千代美, 北岡教英
  • Organizer: 2009年マルチメディア,分散,協調とモバイルシンポジウム
  • Place of Presentation: 杉乃井ホテル(大分県)
  • Year and Date: 2009-07-08
• [Presentation] 複数音響モデルからの最適選択による音声認識 (2009)
  • Author(s): 伊藤新, 原直, 宮島千代美, 北岡教英, 武田一哉
  • Organizer: 2009年電気関係学会東海支部連合大会
  • Place of Presentation: 豊田市
• [Presentation] 動車運転コーパスにおける行動観測信号の統合と利用 (2009)
  • Author(s): 武田一哉, 尾崎晃, マルタルーカス, 西脇由博, 宮島千代美, 北岡教英
  • Organizer: 2009年マルチメディア, 分散, 協調とモバイルシンポジウム
  • Place of Presentation: 別府市
• [Presentation] CENSREC-AV : Evaluation frameworks for audio-visual speech recognition (2008)
  • Author(s): S. Tamura, C. Miyajima, N. Kitaoka, S. Hayamizu, K. Takeda
  • Organizer: International. Conference on Auditory and Visual Speech Processing
  • Place of Presentation: Tangalooma, Australia
  • Year and Date: 2008-09-27
• [Presentation] CENSREC-4 : Development of evaluation framework for distant-talking speech recognition under reverberant environments (2008)
  • Author(s): M. Nakayama, T. Nishiura, Y. Denda, N. Kitaoka, K. Yamamoto, T. Yamada, S. Tsuge, C. Miyajima, M. Fujimoto, T. Takiguchi, S. Tamura, T. Ogawa, S. Matsuda, S. Kuroiwa, K. Takeda, S. Nakamura
  • Organizer: International Conference on Spoken Language Processing
  • Place of Presentation: Brisbane, Australia
  • Year and Date: 2008-09-24
• [Presentation] Multi-modal real-world driving data collection, transcription, and integration using Bayesian network (2008)
  • Author(s): L. Malta, P. Angkititrakul, C. Mivaiima, K. Takeda
  • Organizer: Intelligent Vehicles Symposium
  • Place of Presentation: Marrakech, Morocco
  • Year and Date: 2008-06-05
• [Presentation] CENSREC-4 : Development of evaluation framework for distant-talking speech recognition under reverberant environments (2008)
  • Author(s): T. Nishiura, M. Nakayama, Y. Denda, N. Kitaoka, K. Yamamoto, T. Yamada, S. Tsuge, C. Miyajima, M. Fujimoto, T. Takiguchi, S. Tamura, S. Kuroiwa, K. Takeda, and S. Nakamura
  • Organizer: Language Resources and Evaluation Conference
  • Place of Presentation: Marrakech, Morocco
  • Year and Date: 2008-05-29
• [Presentation] In-car speech data collection along with various multimodal signals (2008)
  • Author(s): A. Ozaki, S. Hara, T. Kusakawa, C. Miyajima, T. Nishino, N. Kitaoka, K. Itou, K. Takeda
  • Organizer: Language Resources and Evaluation Conference
  • Place of Presentation: Marrakech, Morocco
  • Year and Date: 2008-05-28
• [Presentation] CENSREC-AV: Evaluation frameworks for audio- visual speech recognition (2008)
  • Author(s): S. Tamura, C. Miyajima, N. Kitaoka, S. Hayamizu, K. Takeda
  • Organizer: 2008 International Conference on Auditory and Visual Speech Processing
  • Place of Presentation: オーストラリア
• [Presentation] CENSREC-4: Development of evaluation framework for distant- talking speech recognition under reverberant environments (2008)
  • Author(s): M. Nakayama, T. Nishiura, Y. Denda, N. Kitaoka, K. Yamamoto, T. Yamada, S. Tsuge, C. Miyajima, M. Fujimoto, T. Takiguchi, S. Tamura, T. Ogawa, S. Matsuda, S. Kuroiwa, K. Takeda, S. Nakamura
  • Organizer: 2008 International Conference on Spoken Language Processing
  • Place of Presentation: オーストラリア
• [Presentation] In-car speech data collection along with various multimodal signals (2008)
  • Author(s): L. Malta, P. Angkititrakul, C. Miyajima, K. Takeda
  • Organizer: 2008 IEEE Intelligent Vehicles Symposium
  • Place of Presentation: オランダ
• [Presentation] In-car speech data collection along with various multimodal signals (2008)
  • Author(s): A. Ozaki, S. Hara, T. Kusakawa, C. Miyajima, T. Nishino, N. Kitaoka, K. Itou, K. Takeda
  • Organizer: 2008 Language Resources and Evaluation Conference
  • Place of Presentation: モロッコ
• [Presentation] CENSREC- 4: Development of evaluation framework for distant-talking speech recognition under reverberant environments (2008)
  • Author(s): T. Nishiura, M. Nakayama, Y. Denda, N. Kitaoka, K. Yamamoto, T. Yamada, S. Tsuge, C. Miyajima, M. Fujimoto, T. Takiguchi, S. Tamura, S. Kuroiwa, K. Takeda, S. Nakamura
  • Organizer: 2008 Language Resources and Evaluation Conference
  • Place of Presentation: モロッコ
• [Presentation] Development of VAD evaluation framework CENSREC- 1-C and investigation of relationship between VAD and speech recognition performance (2007)
  • Author(s): N. Kitaoka, K. Yamamoto, T. Kusamizu, S. Nakagawa, T. Yamada, S. Tsuge, C. Miyajima, T. Nishiura, M. Nakayama, Y. Denda, M. Fujimoto, T. Takiguchi, S. Tamura, S. Kuroiwa, K. Takeda, S. Nakamura
  • Organizer: 2007 IEEE workshop on Automatic Speech Recognition and Understanding
  • Place of Presentation: 京都市
• [Presentation] On-going data collection for driving behavior signal (2007)
  • Author(s): C. Miyajima, T. Kusakawa, T. Nishino, N. Kitaoka, K. Itou, K. Takeda
  • Organizer: 2007 Biennial on DSP for in-Vehicle and Mobile Systems
  • Place of Presentation: トルコ
• [Presentation] Development of VAD evaluation framework CENSREC-1-C and investigation of relationship between VAD and speech recognition perfor mance (2007)
  • Author(s): N. Kitaoka, K. Yamamoto, T. Kusamizu, S. Nakagawa, T. Yamada, S. Tsuge, C. Miyajima, T. Nishiura, M. Nakayama, Y. Denda, M. Fujimoto, T. Takiguchi, S. Tamura, S. Kuroiwa, K. Takeda, and S. Nakamura
  • Organizer: Proc. IEEE workshop on Automatic Speech Recognition and Understanding
  • Place of Presentation: Kyoto, Japan
• [Presentation] On-going data collection for driving behavior signal (2007)
  • Author(s): C. Miyajima, T. Kusakawa, T. Nishino, N. Kitaoka, K. Itou, and K. Takeda,
  • Organizer: Proc. 2007 Biennial on DSP for in-Vehicle and Mobile Systems
  • Place of Presentation: Istanbul, Turkey
• [Book] Multimodal Speech Corpora for Robust Japanese Speech Recognition in Noisy Environments(S. Itahashi and C.Y. Tseng eds., Computer Processing ofAsian Spoken Languages, Section 4.9(3)) (2010)
  • Author(s): S. Tamura, C. Miyajima
  • Total Pages: 5
• [Book] Computer Processing of Asian Spoken Languages (Section 4.10)(S. Itahashi, C.Y. Tseng eds., Multimodal Speech Corpora for Robust Japanese Speech Recognition in Noisy Environments) (2010)
  • Author(s): M.Tamura, C.Mivajima
  • Total Pages: 5
  • Publisher: Japanese Writer's House