2019 Fiscal Year Research-status Report
Linking Vision and Language through Computational Modelling
Project/Area Number |
19K12733
|
Research Institution | Kobe City University of Foreign Studies |
Principal Investigator |
CHANG Franklin 神戸市外国語大学, 英米学科, 准教授 (60827343)
|
Project Period (FY) |
2019-04-01 – 2024-03-31
|
Keywords | Vision / Language / Syntax / Deep Learning / Computational model / Eyetracking |
Outline of Annual Research Achievements |
I have developed a deep learning model that is trained on text and we find that the model is unable to generalize syntax in a human-like way. We hypothesize that one reason that humans have this ability is due to visual representations. I have developed simulations of various visual events in a game engine and use this input with a deep learning model that can generalize syntactically in a way that mirrors human behavior. I have submitted an abstract about this work to an international conference in Berlin called Architectures and Mechanisms for Language Processing. With my student, I have also published a paper about an experiment on multiple object tracking in vision which will be part of the modelling work in this grant. I have also submitted another paper on related work looking at how that task is related to relative clause production.
|
Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
Overall the computational modelling work is progressing well. I am having some trouble with my deep learning system, so I am purchasing a deep learning machine that should work better/faster. Due to the coronavirus, I have decided to reduce the use of human experiments, since it will be difficult to collect data. Instead, I will do more computational modelling work.
|
Strategy for Future Research Activity |
I have started to write a paper about the present computational modelling work. When the new deep learning machine arrives, it might allow more powerful models to be built, so I will examine the possibilities at that point.
|
Causes of Carryover |
Funds were originally going to be used for an eye-tracker, but due to the cost of a good system, I decided to wait till the next year to purchase it. But now with the coronavirus, I have decided to move towards computational modelling work.
|