2023 Fiscal Year Annual Research Report
Linking Vision and Language through Computational Modelling
| Project/Area Number |
19K12733
|
|---|
| Research Institution | Kobe City University of Foreign Studies |
|---|
Principal Investigator |
CHANG Franklin 神戸市外国語大学, 外国語学部, 教授 (60827343)
|
|---|
| Project Period (FY) |
2019-04-01 – 2024-03-31
|
| Keywords | action understanding / deep learning / Japanese verbs |
|---|
| Outline of Annual Research Achievements |
Published a paper describing experiments and a computational model for action understanding. Videos of simple actions (e.g., climbing a wall) were created in a 3D video game engine and Japanese adults and children asked to describe these scenes. Then we developed a deep learning model that learned about actions by tracking the multiple body parts of the animated figures in the videos. This information was then paired with the Japanese verbs that were used to describe the videos and the model could learn to produce Japanese verbs from the video. It could also use the endstate information in the visual scene to select past tense or present progressive. The model made predictions that were tested in a final experiment.
|
