Developing a learner-adaptive captioning system to improve second language listening
| Project/Area Number |
17K02925
|
|---|
| Research Category |
Grant-in-Aid for Scientific Research (C)
|
| Allocation Type | Multi-year Fund |
|---|
| Section | 一般 |
|---|
| Research Field |
Foreign language education
|
|---|
| Research Institution | Institute of Physical and Chemical Research (2019)
Kyoto University (2017-2018) |
|---|
Principal Investigator |
MESHGI Kourosh 国立研究開発法人理化学研究所, 革新知能統合研究センター, 研究員 (80774835)
|
|---|
| Co-Investigator(Kenkyū-buntansha) |
Mirzaei Maryam 国立研究開発法人理化学研究所, 革新知能統合研究センター, 特別研究員 (10810509)
|
|---|
| Project Period (FY) |
2017-04-01 – 2020-03-31
|
| Project Status |
Completed (Fiscal Year 2019)
|
| Budget Amount *help |
¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000)
Fiscal Year 2019: ¥390,000 (Direct Cost: ¥300,000、Indirect Cost: ¥90,000)
Fiscal Year 2018: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000)
Fiscal Year 2017: ¥2,470,000 (Direct Cost: ¥1,900,000、Indirect Cost: ¥570,000)
|
|---|
| Keywords | PSC / Surprisal Model / L2 listening / ASR error analysis / Listening Difficulty Idx / Instantaneous Hints / Adaptive Captions / Individualized Cues / Cognitive Load / Lexical Complexity / Syntactic Complexity / NLP / Learner adaptation / Online machine learning / Active learning / Data-driven / automatic scaffolding / CALL / ASR / Listening Comprehension |
|---|
| Outline of Final Research Achievements |
This project aimed at developing a user-adaptive caption system to realize an individualized platform for training second language listening skills. In this caption, words are synchronized by the utterance, and only difficult words are shown in the caption. The difficulty of the words is determined using word frequency and specificity, speech rate, lexical and syntactic complexity, and automatic speech recognition (ASR) errors as the indicators of acoustic/speech difficulty. ASR errors have been analyzed, and identified patterns were evaluated by experiments to ensure that they cause difficulties for L2 listeners. A listening difficulty index has been proposed using these factors. Additionally, a data-driven difficulty detector is built on top of these factors. In the proposed caption, learners can improve the word selection by feedback (adaptation), and timely and useful cues are embedded in the caption for ambiguous and difficult words/phrases (hints).
|
| Academic Significance and Societal Importance of the Research Achievements |
Improved listening skill of Japanese learners of English; Popular among university students and teachers; Facilitated training listening skill and using authentic material such as TED talks; Introduced a novel type of captioning method in Japan.
|
Report
(4 results)
Research Products
(4 results)
