2021 Fiscal Year Research-status Report
Cross-disciplinary approach to prosody-based automatic speech processing and its application to computer-assisted language teaching
| Project/Area Number |
20K00838
|
|---|
| Research Institution | The University of Aizu |
|---|
Principal Investigator |
Pyshkin Evgeny 会津大学, コンピュータ理工学部, 上級准教授 (50794088)
|
|---|
| Co-Investigator(Kenkyū-buntansha) |
Mozgovoy Maxim 会津大学, コンピュータ理工学部, 准教授 (60571776)
BLAKE John 会津大学, コンピュータ理工学部, 上級准教授 (80635954)
|
|---|
| Project Period (FY) |
2020-04-01 – 2023-03-31
|
| Keywords | CAPT / L2 education / mobile technology / multimodal feedback / speech processing |
|---|
| Outline of Annual Research Achievements |
We worked on adopting StudyIntonation CAPT tools to the case of tonal languages (e.g. Vietnamese, with a complexity of tones characterized by pitch, length, contour melody, and intensity). While assessing of our CAPT environment, we studied approaches to support tailored and meaningful feedback with evaluative, instructive, and actionable components using the metrics of the distance between the graphs, based on a dynamic time warping algorithm assuring tempo invariant estimation. We implemented a number of additional features including pitch graph contours with multiple attempts cross-check, pitch graph segmentation and segmented visualization (particularly important for tonal languages), and designed an interface enabling the exercises on attitudinal intonation training.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
In its current implementation, our environment and the corresponding mobile tools provide an authentic speech context with real-time pitch graphs of speakers, perform learners’ speech recording, display model speakers and learners pitch graphs to output a contrastive feedback and calculate speech quality measures aimed at improving the learners’ speech progressively. By offering the activities different ways of perception, the system addresses a problem of supporting a diversity of user learning styles. We continue extensive research on prosodic synchronization between speakers as well as longitudinal pronunciation assessment that would allow for quantitative evaluation by means of a dynamical modeling technique of cross-recurrence quantification analysis.
|
| Strategy for Future Research Activity |
According to independent reviews and our own experience, the current system is promising in its goal to serve as a mobile-assisted pronunciation training tool for classroom and individual learning purposes, however, there are still open issues to be addressed in the further studies and application releases. Such issues include providing more specific contrastive consistent feedback to users, so that to enable better problem segmentation, easier user self-correction, as well as clear positioning of the approach in scope of relevant pedagogical models.
|
| Causes of Carryover |
Due to COVID-19 restriction we could not arrange our expenses for travel and workshop organization, that is why they need to be transferred to
the next fiscal year with the same usage plan as it was in 2020-2021.
|
