Development of a Method to Combine Deep Learning and Symbolic Reasoning and its Application to Machine Reading Comprehension

2021 Fiscal Year Final Research Report

• Project/Area Number: 20K23314
• Research Category: Grant-in-Aid for Research Activity Start-up
• Allocation Type: Multi-year Fund
• Review Section: 1001:Information science, computer engineering, and related fields
• Research Institution: Tohoku University
• Principal Investigator: Yoshikawa Masashi 東北大学, タフ・サイバーフィジカルAI研究センター, 助教 (80883470)
• Project Period (FY): 2020-09-11 – 2022-03-31
• Keywords: 自然言語処理 / 深層学習 / 文章読解 / 機械学習 / 記号推論 / 数量推論

Outline of Final Research Achievements

We believe the key to advance further the state of deep learning (DL)-based natural language processing (NLP) is combining its technologies with those of symbolic reasoning. By the combination, it will be possible to develop an AI system that is (1) more data-efficient, and (2) more robust to variations of texts (e.g., domains), with (3) the more visible inference process essential to symbols. However, as a major key to the success of DL is end-to-end training using backpropagation, it is challenging to incorporate discrete symbolic function within a neural network. In this project, we propose a simple extension of the Gumbel-Softmax trick to overcome the problem and enable gradient-based learning of such a neuro-symbolic system.
Using the method, we tackle numerical reasoning, which is an unsolved problem in DL-based NLP. We approach this problem by incorporating an arithmetic calculator layer within a DL-based reasoning model.

Free Research Field

自然言語処理

Academic Significance and Societal Importance of the Research Achievements

深層学習言語モデルの高い推論能力が喧伝される一方で、少し入力を操作するだけで思わぬ挙動を示すなど脆弱性も分野内でよく知られている。記号推論の仕組みを外部知識として組み込むことで、深層モデルのブラックボックス性を軽減でき、推論過程の透明性のみならず、期待される推論過程を教え込むことでより頑健なシステムを構築することが可能になる。
数量は言語中に頻出である一方で自然言語処理では見過ごされがちである。金融文章の解析など応用にも直結する課題であるが、本研究の取り組みにより数量処理に大きな前進をもたらすことが期待される。