2019 Fiscal Year Research-status Report
Neural Machine Translation Based on Bilingual Resources Extracted from Multimodal Data
| Project/Area Number |
19K20343
|
|---|
| Research Institution | Osaka University |
|---|
Principal Investigator |
チョ シンキ 大阪大学, データビリティフロンティア機構, 特任助教(常勤) (70784891)
|
|---|
| Project Period (FY) |
2019-04-01 – 2022-03-31
|
| Keywords | 機械翻訳 / マルチモーダル |
|---|
| Outline of Annual Research Achievements |
Machine translation (MT) is crucial to promote globalization. In Japan, because of the rapid increase of foreign tourists and the host of the 2020 Tokyo Olympic Games, translation needs are rapidly growing, making MT indispensable. In MT, the translation knowledge is acquired from parallel corpora (sentence-aligned bilingual texts). However, domain specific parallel corpora are usually scarce or nonexistent in most languages, and thus MT performs poorly in such scenarios. Improving MT quality in this low resource scenario is a challenging unsolved problem.
To address this problem, we aim to extract parallel data from multimodal data consisting of images and multilingual description text, which is widely available from the web and social media and studying neural MT (NMT) using the extracted parallel data. In FY 2019, we mainly studied the following towards our goal.
1. NMT with semantic image regions. We developed a multimodal NMT system with double attention that attends to both words and semantic image regions and published our work at NL 241 and YANS 2019.
2. Cross-lingual visual grounding. To identify parallel data from multimodal data, we studied a cross-lingual visual grounding model on both English and French data. This work is under submission.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
We planed to study the following items on image caption datasets in FY 2019 and finished them as scheduled:
1. MT with parallel sentences and image/region representation fusion.
2. Visual grounding to identify corresponding image regions given phrases as queries.
|
| Strategy for Future Research Activity |
1. Improve our mutlimodal MT with parallel sentences: we plan to integrate visual grounding supervision and word-region interaction into our model to further improve the performance.
2. Mutlimodal MT with comparable sentences: we plan to study this problem on Japanese-English image caption datasets by integrating our techniques on mutlimodal MT with parallel sentences and cross-lingual visual grounding.
|
| Causes of Carryover |
理由：新型コロナウイルスにより予定していた出張がキャンセルされたためでした。
使用計画：
新型コロナウイルスにより次年度の国際会議の多くがオンライン開催になっています。そこで出張費はかかりませんが、次年度使用額をACL 2020やEMNLP 2020などの国際会議のオンライン参加費として使う予定です。当該年度以降分として請求した助成金は元々の予定通り使います。
|
Research Products
(11 results)
