Study of audio-visual integration process revealed by eye-tracking experiment with the "off-screen' sound presentation.

• Project/Area Number: 26330311
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Research Field: Kansei informatics
• Research Institution: Miyagi University
• Principal Investigator: Kayahara Takuro 宮城大学, 事業構想学群(部), 教授 (00345026)
• Project Period (FY): 2014-04-01 – 2018-03-31
• Project Status: Completed (Fiscal Year 2017)
• Budget Amount: ¥4,680,000 (Direct Cost: ¥3,600,000、Indirect Cost: ¥1,080,000); Fiscal Year 2016: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000); Fiscal Year 2015: ¥3,380,000 (Direct Cost: ¥2,600,000、Indirect Cost: ¥780,000); Fiscal Year 2014: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000)
• Keywords: シーン認識 / 視聴覚統合 / 視線計測 / サウンドトラック / オフスクリーン音 / マルチモーダル / 視聴覚情報統合 / アニメーション / 意味解釈 / 映画

Outline of Final Research Achievements

In this study, whether the sound induce subjects' gaze and affect meaning of the scene was examined when the subjects was presented with audio-visual stimulus, including the condition in which the sound source is outside of scene ("off-screen" sound condition). The data from eye-track recording revealed that the sound induce subjects' gaze and affect meaning of the scene. Interestingly, in off-screen sound condition, the sound didn't induce subjects' gaze to specific visual object in the scene, but, made meaning of the scene read by subjects more rich and complex.

Research Products

• [Presentation] 心理学におけるワークショップの古くて新しい可能性 ～19世紀知覚研究とエンタテイメントメディアの温故知新～ (2017)
  • Author(s): 茅原拓朗
  • Organizer: 日本基礎心理学会第36回大会
  • Invited
• [Presentation] "Sound track" brings new approaches of the psychological study of audio-visual interaction and a new track of outreach for psychology. (2016)
  • Author(s): Takuro Kayahara
  • Organizer: 31st International Congress of Psychology (ICP2016)
  • Place of Presentation: Yokohama, Japan
  • Int'l Joint Research / Invited
• [Presentation] 映像を伝える音 (2015)
  • Author(s): 茅原拓朗
  • Organizer: 日本基礎心理学会公開シンポジウム
  • Place of Presentation: 慶應大学三田キャンパス（東京都港区）
  • Year and Date: 2015-10-25
  • Invited