Linking Vision and Language through Computational Modelling

2020 Fiscal Year Research-status Report

• Project/Area Number: 19K12733
• Research Institution: Kobe City University of Foreign Studies
• Principal Investigator: CHANG Franklin 神戸市外国語大学, 英米学科, 教授 (60827343)
• Project Period (FY): 2019-04-01 – 2024-03-31
• Keywords: Vision / Language / Learning / Syntax / Computational model / Deep Learning

Outline of Annual Research Achievements

Work on the deep learning model continues, and I have started to work on experiments which test some of the basic learning assumptions of the deep learning model. Deep learning works by making a prediction and generating error, which is then back-propagated through the network to change the knowledge in the system to improve the ability to make predictions. While error-based learning is one of the most successful approaches for computational learning systems, there is relatively little experimental evidence for it. In the present experiments, I am testing whether prediction error influences syntax-related learning. Participants read prime sentences one word at a time on a computer screen, and this allows them to generate a prediction after each work. Some of the sentences have ungrammatical structures, which create prediction error, and these sentences should cause greater adaptation of syntactic structures. We have done several pilot studies so far and found a range of unexpected factors are influencing our results (e.g., length of phrases, kanji vs kana). We are slowly trying to improve the study so that it will be a strong test of whether error is involved in learning.

I have submitted a paper on a multiple object tracking production study about relative clauses which is related to the modelling work in this grant.

Current Status of Research Progress

2: Research has progressed on the whole more than it was originally planned.

Reason

The modelling work is progressing slowly as I have started to do experiments to test the learning assumptions of the model. I will focus on both modelling and experiments this year.

Strategy for Future Research Activity

I think the experiments will have some publishable results by summer and I will start to write up some papers on those studies.
Since I have finished submitting the experimental papers that will be used for the modelling work, I will start to work more on getting the modelling work finished.

Causes of Carryover

I have decided to not buy an eye-tracker and instead use the money for on-line experiments and computational work. So far we have done several pilot studies and are working toward the main study. This should continue into the next year.