Involvement-in-conversation type CALL system

• Project/Area Number: 15K02738
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Research Field: Foreign language education
• Research Institution: Doshisha University
• Principal Investigator: Yamamoto Seiichi 同志社大学, 研究開発推進機構, 嘱託研究員 (20374100)
• Co-Investigator(Kenkyū-buntansha): 馬田 一郎 国立研究開発法人情報通信研究機構, ユニバーサルコミュニケーション研究所多感覚・評価研究室, 研究員 (40374110)
• Project Period (FY): 2015-04-01 – 2020-03-31
• Project Status: Discontinued (Fiscal Year 2019)
• Budget Amount: ¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000); Fiscal Year 2019: ¥520,000 (Direct Cost: ¥400,000、Indirect Cost: ¥120,000); Fiscal Year 2018: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000); Fiscal Year 2017: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000); Fiscal Year 2016: ¥1,040,000 (Direct Cost: ¥800,000、Indirect Cost: ¥240,000); Fiscal Year 2015: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000)
• Keywords: 音声認識 / CALL / 自然言語処理 / 第二言語

Outline of Final Research Achievements

This research proposed a joining-in-type robot-assisted language learning (JIT-RALL) system as a computer-assisted language learning (CALL) systems with unique features that invites a learner to join in English conversations with two robots. The learner is encouraged to obtain communicative skill in English through implicit learning from dialogues between the robot and tutoring from a teacher robot. Experimental results comparing the ratios that the learners use appropriate expression before and after training with the JIT-RALL system demonstrated that the learners used appropriate expressions significantly more after the training. The experiment results demonstrated that the combination of tutoring and implicit learning implemented in the JIT-RALL system is effective for the learner to facilitate proceduralization of their declarative knowledge and develop their spontaneous productive skills in using grammatical patterns.

Academic Significance and Societal Importance of the Research Achievements

近年のグローバル化を背景として，国際的共通語としての英語でのコミュニケーション能力の習得は一層その重要性を増しており，学習者と対話を行い，発音以外に文法，語彙等のコミュニケーション能力全般の向上を支援するシステムである音声対話型 CALL (DB-CALL)の研究開発が急がれている社会環境で，広範囲な話題を対象としてimplicit learningとtutoringを効果的に組み合わせた音声対話型 CALLシステムを提案し，その実現可能性を実証した意義は大きい．

Research Products

• [Journal Article] Learning Effect of Implicit Learning in Joining-in-type Robot-assisted Language Learning System (2019)
  • Author(s): Khalifa AlBara、Kato Tsuneo、Yamamoto Seiichi
  • Journal Title: International Journal of Emerging Technologies in Learning (iJET) Volume: 14 Issue: 02 Pages: 105-123
  • DOI: 10.3991/ijet.v14i02.9212
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Classification of Utterances Based on Multiple BLEU Scores for Translation-game-type CALL Systems (2018)
  • Author(s): R. Kuwa, T. Kato, S. Yamamoto
  • Journal Title: IEICE Trans. INF. & SYST Volume: E101-D Pages: 750-757
  • NAID: 130006414103
  • Peer Reviewed / Open Access
• [Journal Article] Difference in Eye Gaze for Floor Apportionment in Native- and Second- language Conversations (2017)
  • Author(s): K. Ijuin, I. Umata, T. Kato, S. Yamamoto
  • Journal Title: Journal of Nonverbal Behavior Volume: 42 Issue: 1 Pages: 113-128
  • DOI: 10.1007/s10919-017-0262-3
  • Peer Reviewed / Open Access
• [Journal Article] Content-Aware Image Retargeting Incorporated with Letterboxing (2017)
  • Author(s): X. Wang, T. Kato, S. Yamamoto
  • Journal Title: IEICE Transactions on Information and Systems Volume: E100.D Issue: 4 Pages: 865-873
  • DOI: 10.1587/transinf.2016EDP7395
  • NAID: 130005529868
  • ISSN: 0916-8532, 1745-1361
  • Peer Reviewed / Open Access / Acknowledgement Compliant
• [Journal Article] Multimodal corpus of multiparty conversations in L1 and L2 languages and findings obtained from it (2015)
  • Author(s): Yamamoto, S., Taguchi, K., Ijuin, K., Umata, I., and Nishida, M.
  • Journal Title: Language Resources and Evaluation Volume: 49 Issue: 4 Pages: 857-882
  • DOI: 10.1007/s10579-015-9299-2
  • Peer Reviewed / Open Access / Acknowledgement Compliant
• [Journal Article] Speech Recognition of English by Japanese using Lexicon Represented by Multiple Reduced Phoneme Sets (2015)
  • Author(s): X. Wang, S. Yamamoto
  • Journal Title: Trans. IEICE Volume: E98-D Pages: 2271-2279
  • NAID: 130005112316
  • Peer Reviewed
• [Presentation] Eye-gaze in Social Robot Interactions - Grounding of Information and Eye-gaze Patterns (2019)
  • Author(s): Koki Ijuin, Kristiina Jokinen, Tsuneo Kato, Seiichi Yamamoto
  • Organizer: JSAI2019
• [Presentation] Gaze from and toward the Silent Third Participant in a Triadic Conversation (2019)
  • Author(s): Ichiro Umata, Koki Ijuin, Tsuneo Kato, Seiichi Yamamoto
  • Organizer: HCII2019
  • Int'l Joint Research / Invited
• [Presentation] Joining-in-type RALLシステムにおける視線動作と会話の理解度との関係についての分析 (2019)
  • Author(s): 藤尾昇平、伊集院幸輝、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合全国大会
• [Presentation] 眼球動作情報の有無における母語と第二言語での注視行動の比較 (2019)
  • Author(s): 大野善彦、伊集院幸輝、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合全国大会
• [Presentation] Comparison on Effect of Eye Gaze Activities between Human-human and Human-robot Conversations in Second-Language (2018)
  • Author(s): Koki Ijuin, Shohe Fujio, AlBara Khalifa, Tsuneo Kato, Seiichi Yamamoto
  • Organizer: AI-MHR
  • Int'l Joint Research
• [Presentation] Floor Apportionment and Mutual Gazes in Native and Second-Language Conversation (2018)
  • Author(s): Ichiro Umata, Koki Ijuin, Tsuneo Kato, Seiichi Yamamoto
  • Organizer: ICMI2018
  • Int'l Joint Research
• [Presentation] 注視が次発話者の選択に及ぼす影響に関する母語と第二言語会話での比較分析 (2018)
  • Author(s): 山下昴人、伊集院幸輝、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合全国大会
• [Presentation] 人型ロボットを用いた第二言語学習支援システムにおける繰り返し訓練における評価 (2018)
  • Author(s): 尾坂和哉、三田村悠作、AlBara Khalifa、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合全国大会
• [Presentation] Joining-in-type RALLシステムにおける視線動作の計測と分析 (2018)
  • Author(s): 藤尾昇平、伊集院幸輝、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合全国大会
• [Presentation] 人型ロボットを用いたダニ言語学習支援システムにおける質問の繰り返しによる表現学習の分析 (2017)
  • Author(s): 石田充、AlBara Khalifa、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合大会
  • Place of Presentation: 名城大学、愛知県名古屋市
  • Year and Date: 2017-03-22
• [Presentation] 母語と第二言語会話の会話構造の差異における視線動作の相違分析 (2017)
  • Author(s): 堀内保大、伊集院幸輝、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合大会
  • Place of Presentation: 名城大学、愛知県名古屋市
  • Year and Date: 2017-03-22
• [Presentation] 音声対話型CALLシステムのためのBLEUスコアに基づく学習者発話の分類 (2017)
  • Author(s): 桑怜子、加藤恒夫、山本誠一
  • Organizer: 日本音響学会春季研究発表会
  • Place of Presentation: 明治大学、神奈川県川崎市
  • Year and Date: 2017-03-15
• [Presentation] Measuring Effect of Repetitive Questioning and Implicit Learning with Joining-in Type Robot Assisted Language Learning System (2017)
  • Author(s): A. Khalifa, T. Kato, S. Yamamoto
  • Organizer: 7th ISCA Workshop on Speech and Language Technology in Education
  • Int'l Joint Research
• [Presentation] Factor Analysis of Gazing Activities in Native and Second Language Conversations (2017)
  • Author(s): I. Umata, K. Ijuin, S. Yamamoto
  • Organizer: 21st Workshop on the Semantics and Pragmatics of Dialogue
  • Int'l Joint Research
• [Presentation] Comparison of Effect of Speaker's Eye Gaze on Selection of Next Speaker between Native- and Second-language Conversations (2017)
  • Author(s): K. Ijuin, T. Yamashita, T. Kato, S. Yamamoto
  • Organizer: Grounding Language Understanding GLU2017
  • Int'l Joint Research
• [Presentation] 注視が次発話者の選択に及ぼす影響に関する母語と第二言語会話での比較分析 (2017)
  • Author(s): 山下昴人、伊集院幸輝、加藤恒夫、山本誠一
  • Organizer: ＦＩＴ２０１７
• [Presentation] Effect of Music Experience on Perception of Temo Change (2016)
  • Author(s): M. Yamashita, M. Yanagida, I. Umata, T. Kato, S. Yamamoto
  • Organizer: Joint Meeting of ASA and ASJ
  • Place of Presentation: Honolulu, USA
  • Year and Date: 2016-11-28
  • Int'l Joint Research
• [Presentation] Features of Learner Corpsu Collected with Joining-in Type Robot Assisted Language Learning System (2016)
  • Author(s): M. Ishida, A. Khalifa, T. Kato, S. Yamamoto
  • Organizer: Oriental COCOSDA
  • Place of Presentation: Bali, Indonesia
  • Year and Date: 2016-10-26
  • Int'l Joint Research
• [Presentation] Classification of Utterance Acceptability Based on BLEU scores for Dialogue-based CALL Systems (2016)
  • Author(s): R. Kuwa, X. Wang, T. Kato, S. Yamamoto
  • Organizer: TSD2016
  • Place of Presentation: Burno, Czech
  • Year and Date: 2016-09-12
  • Int'l Joint Research
• [Presentation] F0 Countour Analysis Based on Empirical Mode Decomposition for DNN acoustic modeling in Mandarin Speech Recognition (2016)
  • Author(s): X. Wang, X. Li, H. Kawai, S. Yamamoto
  • Organizer: Interspeech2016
  • Place of Presentation: San Francisco, USA
  • Year and Date: 2016-09-08
  • Int'l Joint Research
• [Presentation] Phoneme Set Design Considering Integrated Acoustic and Linguistic Features of Second Language Speech (2016)
  • Author(s): X. Wang, T. Kato, S. Yamamoto
  • Organizer: Interspeech2016
  • Place of Presentation: San Francisco, USA
  • Year and Date: 2016-09-08
  • Int'l Joint Research
• [Presentation] Effects of the mode of tempo change on perception of tempo change (2016)
  • Author(s): M. Yanagida, I. Umata, S. Yamamoto
  • Organizer: ICA2016
  • Place of Presentation: Buenos Aires, Argentina
  • Year and Date: 2016-09-05
  • Int'l Joint Research
• [Presentation] Joining-in-type Humanoid Robot Assited Language Learning Systems" (2016)
  • Author(s): A. Khalifa, T. Kato, S. Yamamoto
  • Organizer: LREC
  • Place of Presentation: Portoroz, Slovenia
  • Year and Date: 2016-05-23
  • Int'l Joint Research
• [Presentation] Quantitative Analysis of Gazes and Grounding Acts in L1 and L2 Conversations (2016)
  • Author(s): I. Umata, K. Ijuin, M. Ishida, M. Takeuchi, S. Yamamoto
  • Organizer: LREC
  • Place of Presentation: Portoroz, Slovenia
  • Year and Date: 2016-05-23
  • Int'l Joint Research
• [Presentation] 人型ロボットを用いた言語学習支援システムにおけるユーザ発話の音声特徴の分析 (2016)
  • Author(s): 小此木謙一、石田充、谷添友哉、AlBara Khalifa、 馬田一郎、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合大会
  • Place of Presentation: 九州大学
  • Year and Date: 2016-03-15
• [Presentation] 3 人会話におけるInteractive Alignment の特徴分析 (2016)
  • Author(s): 谷添友哉，石田充、小此木謙一、AlBara Khalifa，馬田一郎、加藤恒夫、山本誠一
  • Organizer: 電子情報通信学会総合大会
  • Place of Presentation: 九州大学
  • Year and Date: 2016-03-15
• [Presentation] Towards Computer-Assisted Language Learning with Robots, Wikipedia and CogInfoCom (2015)
  • Author(s): G. Wilcock, S. Yamamoto
  • Organizer: IEEE Conf. on Cognitive Infocommunications
  • Place of Presentation: Gyor, Hungary
  • Year and Date: 2015-10-19
  • Int'l Joint Research
• [Presentation] Quantitative analyses of Gaze Activity during Silence: Comaprison between Native-language and Second-language Conversations (2015)
  • Author(s): I. Umata, T. Tanizoe, K. Ijuin, S. Yamamoto
  • Organizer: EPACogsci
  • Place of Presentation: Turin, Italy
  • Year and Date: 2015-09-25
  • Int'l Joint Research
• [Presentation] 第二言語での会話構造が視線動作に及ぼす影響についての分析 (2015)
  • Author(s): 伊集院幸輝，石田充，堀内保大，馬田一郎，山本誠一
  • Organizer: ＦＩＴ
  • Place of Presentation: 愛媛大学
  • Year and Date: 2015-09-15
• [Presentation] Eye Gaze Analyses in L1 and L2 Conversations: Difference in Interaction Structure (2015)
  • Author(s): K. Ijuin, Y. Horiuchi, I. Umata, S. Yamamoto
  • Organizer: Text Speech and Dialogue
  • Place of Presentation: Plzen, Czech
  • Year and Date: 2015-09-14
  • Int'l Joint Research
• [Presentation] Second Language Speech Recognition using Multiple-Pass Decoding with Lexicon Represented by Multiple Reduced Phoneme Sets (2015)
  • Author(s): Second Language Speech Recognition using Multiple-Pass Decoding with
  • Organizer: Interspeech
  • Place of Presentation: Dresden, Germany
  • Year and Date: 2015-09-06
  • Int'l Joint Research