• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Zero-shot machine translation using multimodal deep encoder-decoder networks

Research Project

Project/Area Number 16H05872
Research Category

Grant-in-Aid for Young Scientists (A)

Allocation TypeSingle-year Grants
Research Field Intelligent informatics
Research InstitutionThe University of Tokyo

Principal Investigator

Nakayama Hideki  東京大学, 大学院情報理工学系研究科, 准教授 (00643305)

Project Period (FY) 2016-04-01 – 2019-03-31
Project Status Completed (Fiscal Year 2018)
Budget Amount *help
¥23,400,000 (Direct Cost: ¥18,000,000、Indirect Cost: ¥5,400,000)
Fiscal Year 2018: ¥6,890,000 (Direct Cost: ¥5,300,000、Indirect Cost: ¥1,590,000)
Fiscal Year 2017: ¥6,890,000 (Direct Cost: ¥5,300,000、Indirect Cost: ¥1,590,000)
Fiscal Year 2016: ¥9,620,000 (Direct Cost: ¥7,400,000、Indirect Cost: ¥2,220,000)
Keywords機械翻訳 / ゼロショット学習 / マルチモーダル / 画像認識 / ニューラルネットワーク / 表現学習 / 自然言語処理 / デコーディング / 量子化 / 教師なし学習 / 深層学習 / データ圧縮 / マルチモーダル学習
Outline of Final Research Achievements

In this research, we have developed a zero-shot machine translation method which can be trained only with monolingual image-text data, without the help of parallel text corpus. This method is realized by the idea of using images as a hub to align texts in different languages. Moreover, we have improved the method in many aspects to enhance its practicality such as output diversification and speeding up. These results are accepted at many top-level international conferences such as ACL and ICLR, and awarded the best paper awards twice at the NLP domestic conference.

Academic Significance and Societal Importance of the Research Achievements

機械翻訳はより一層の技術革新が強く求められているアプリケーションであるが、現在の一般的なアプローチにおいては、学習に用いる対訳テキストコーパスの量が性能向上の鍵となる。しかしながら、実際には同一内容を複数言語で記述したテキストドキュメントは少なく、GAFA等一部の巨大企業にデータを独占されているのが現状である。本研究で提案するアプローチでは、誰でも比較的容易に入手可能な画像付き単一言語ドキュメントのみからの学習を実現するものであり、学術的にも独創的な試みであると同時に、機械翻訳の民主化に貢献しうる点で社会的意義も大きいものであると考える。

Report

(4 results)
  • 2018 Annual Research Report   Final Research Report ( PDF )
  • 2017 Annual Research Report
  • 2016 Annual Research Report
  • Research Products

    (38 results)

All 2019 2018 2017 2016 Other

All Journal Article (13 results) (of which Peer Reviewed: 13 results,  Open Access: 7 results,  Acknowledgement Compliant: 1 results) Presentation (23 results) (of which Int'l Joint Research: 13 results,  Invited: 3 results) Remarks (2 results)

  • [Journal Article] Semantic Aware Attention Based Deep Object Co-segmentation2019

    • Author(s)
      Hong Chen, Yifei Huang, Hideki Nakayama
    • Journal Title

      Proceedings of the 14th Asian Conference on Computer Vision (ACCV)

      Volume: 印刷中

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Augmenting Image Question Answering Dataset by Exploiting Image Captions2018

    • Author(s)
      Masashi Yokota, Hideki Nakayama
    • Journal Title

      Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC)

      Volume: - Pages: 2753-2757

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Compressing Word Embeddings via Deep Compositional Code Learning2018

    • Author(s)
      Raphael Shu, Hideki Nakayama
    • Journal Title

      Proceedings of the 6th International Conference on Learning Representations (ICLR)

      Volume: -

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Coherence Modeling Improves Implicit Discourse Relation Recognition2018

    • Author(s)
      Noriki Nishida, Hideki Nakayama
    • Journal Title

      Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL)

      Volume: - Pages: 344-349

    • DOI

      10.18653/v1/w18-5040

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Improving Beam Search by Removing Monotonic Constraint for Neural Machine Translation2018

    • Author(s)
      Raphael Shu, Hideki Nakayama
    • Journal Title

      Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)

      Volume: 2 Pages: 339-344

    • DOI

      10.18653/v1/p18-2054

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Augmenting Image Question Answering Dataset by Exploiting Image Captions2018

    • Author(s)
      Masashi Yokota and Hideki Nakayama
    • Journal Title

      Proceedings of International Conference on Language Resources and Evaluation (LREC)

      Volume: 印刷中

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Compressing Word Embeddings via Deep Compositional Code Learning2018

    • Author(s)
      Raphael Shu and Hideki Nakayama
    • Journal Title

      Proceedings of International Conference on Learning Representations (ICLR)

      Volume: 印刷中

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Zero-resource Machine Translation by Multimodal Encoder-Decoder Network with Multimedia Pivot2017

    • Author(s)
      Hideki Nakayama and Noriki Nishida
    • Journal Title

      Machine Translation

      Volume: 31 Issue: 1-2 Pages: 49-64

    • DOI

      10.1007/s10590-017-9197-z

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation2017

    • Author(s)
      Raphael Shu and Hideki Nakayama
    • Journal Title

      Proceedings of the First Workshop on Neural Machine Translation

      Volume: - Pages: 1-10

    • DOI

      10.18653/v1/w17-3201

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Word Ordering as Unsupervised Learning Towards Syntactically Plausible Word Representations2017

    • Author(s)
      Noriki Nishida and Hideki Nakayama
    • Journal Title

      Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJCNLP)

      Volume: 1 Pages: 70-79

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Bag of Local Convolutional Triplets for Script Identification in Scene Text2017

    • Author(s)
      Jan Zdenek and Hideki Nakayama
    • Journal Title

      Proceedings of 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)

      Volume: - Pages: 369-375

    • DOI

      10.1109/icdar.2017.68

    • Related Report
      2017 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Efficient Two-Step Middle-Level Part Feature Extraction for Fine-Grained Visual Categorization2016

    • Author(s)
      Hideki Nakayama, Tomoya Tsuda
    • Journal Title

      IEICE Transactions on Information and Systems

      Volume: E99.D Issue: 6 Pages: 1626-1634

    • DOI

      10.1587/transinf.2015EDP7358

    • NAID

      130005154735

    • ISSN
      0916-8532, 1745-1361
    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] Annotation Order Matters: Recurrent Image Annotator for Arbitrary Length Image Tagging2016

    • Author(s)
      Jiren Jin, Hideki Nakayama
    • Journal Title

      Proceedings of International Conference on Pattern Recognition

      Volume: -

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed
  • [Presentation] 談話構成素とその文脈による教師なし談話構成素構造解析2019

    • Author(s)
      西田典起, 中山英樹
    • Organizer
      言語処理学会第25回年次大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] Generating Syntactically Diverse Translations with Syntactic Codes2019

    • Author(s)
      朱中元, 中山英樹
    • Organizer
      言語処理学会第25回年次大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] Augmenting Image Question Answering Dataset by Exploiting Image Captions2018

    • Author(s)
      Masashi Yokota, Hideki Nakayama
    • Organizer
      The Eleventh International Conference on Language Resources and Evaluation (LREC)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Compressing Word Embeddings via Deep Compositional Code Learning2018

    • Author(s)
      Raphael Shu, Hideki Nakayama
    • Organizer
      The 6th International Conference on Learning Representations (ICLR)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Coherence Modeling Improves Implicit Discourse Relation Recognition2018

    • Author(s)
      Noriki Nishida, Hideki Nakayama
    • Organizer
      The 19th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Improving Beam Search by Removing Monotonic Constraint for Neural Machine Translation2018

    • Author(s)
      Raphael Shu, Hideki Nakayama
    • Organizer
      The 56th Annual Meeting of the Association for Computational Linguistics (ACL)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Semantic Aware Attention Based Deep Object Co-segmentation2018

    • Author(s)
      Hong Chen, Yifei Huang, Hideki Nakayama
    • Organizer
      The 14th Asian Conference on Computer Vision (ACCV)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] マルチモーダル深層学習の発展2018

    • Author(s)
      中山英樹
    • Organizer
      第20回インタラクティブ情報アクセスと可視化マイニング研究会(SIG-AM)
    • Related Report
      2018 Annual Research Report
    • Invited
  • [Presentation] テキストの局所一貫性に基づく半教師あり暗黙的談話関係認識2018

    • Author(s)
      西田典起
    • Organizer
      言語処理学会年次大会
    • Related Report
      2017 Annual Research Report
  • [Presentation] 深層コード学習による単語分散表現の圧縮2018

    • Author(s)
      朱中元
    • Organizer
      言語処理学会年次大会
    • Related Report
      2017 Annual Research Report
  • [Presentation] Augmenting Image Question Answering Dataset by Exploiting Image Captions2018

    • Author(s)
      Masashi Yokota
    • Organizer
      International Conference on Language Resources and Evaluation (LREC)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Compressing Word Embeddings via Deep Compositional Code Learning2018

    • Author(s)
      Raphael Shu
    • Organizer
      International Conference on Learning Representations (ICLR)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation2017

    • Author(s)
      Raphael Shu
    • Organizer
      First Workshop on Neural Machine Translation
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Word Ordering as Unsupervised Learning Towards Syntactically Plausible Word Representations2017

    • Author(s)
      Noriki Nishida
    • Organizer
      International Joint Conference on Natural Language Processing (IJCNLP)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Bag of Local Convolutional Triplets for Script Identification in Scene Text2017

    • Author(s)
      Jan Zdenek
    • Organizer
      International Conference on Document Analysis and Recognition (ICDAR)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Pivot-based Multimodality Integration for Unsupervised Cross-domain Machine Intelligence2017

    • Author(s)
      Hideki Nakayama
    • Organizer
      International Symposium on Research and Education of Computational Science (RECS)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] 文脈を考慮したアテンションメカニズムの計算量の削減2017

    • Author(s)
      朱中元
    • Organizer
      人工知能学会全国大会
    • Related Report
      2017 Annual Research Report
  • [Presentation] Script Identification using Bag-of-Words with Entropy-weighted Patches2017

    • Author(s)
      ズデニェク・ヤン
    • Organizer
      人工知能学会全国大会
    • Related Report
      2017 Annual Research Report
  • [Presentation] シーングラフを用いた質問文生成によるデータ拡張の手法2017

    • Author(s)
      横田匡史
    • Organizer
      人工知能学会全国大会
    • Related Report
      2017 Annual Research Report
  • [Presentation] Learning Syntactically Plausible Word Representations by Solving Word Ordering2017

    • Author(s)
      西田典起
    • Organizer
      人工知能学会全国大会
    • Related Report
      2017 Annual Research Report
  • [Presentation] Annotation Order Matters: Recurrent Image Annotator for Arbitrary Length Image Tagging2016

    • Author(s)
      Jiren Jin, Hideki Nakayama
    • Organizer
      International Conference on Pattern Recognition (ICPR)
    • Place of Presentation
      カンクン(メキシコ)
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Pivot-based multimodality integration for cross-media machine intelligence2016

    • Author(s)
      Hideki Nakayama
    • Organizer
      CEMS Topical Meeting on Soft Robotics
    • Place of Presentation
      理化学研究所(埼玉県和光市)
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Recurrent image annotator2016

    • Author(s)
      Jiren Jin, Hideki Nakayama
    • Organizer
      第19回画像の認識・理解シンポジウム
    • Place of Presentation
      アクトシティ浜松(静岡県浜松市)
    • Related Report
      2016 Annual Research Report
  • [Remarks] 言語とビジョンの融合に関わる研究成果

    • URL

      http://www.nlab.ci.i.u-tokyo.ac.jp/projects/vision_and_language.html

    • Related Report
      2018 Annual Research Report 2017 Annual Research Report
  • [Remarks] 自然言語処理に関わる研究成果

    • URL

      http://www.nlab.ci.i.u-tokyo.ac.jp/projects/nlp.html

    • Related Report
      2018 Annual Research Report 2017 Annual Research Report

URL: 

Published: 2016-04-21   Modified: 2020-03-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi