Enhancement of handwritten mathematical expression recognition through establishment of multi-dimensional structural analysis

2022 Fiscal Year Final Research Report

• Project/Area Number: 19H01117
• Research Category: Grant-in-Aid for Scientific Research (A)
• Allocation Type: Single-year Grants
• Section: 一般
• Review Section: Medium-sized Section 61:Human informatics and related fields
• Research Institution: Tokyo University of Agriculture and Technology
• Principal Investigator: Nakagawa Masaki 東京農工大学, 学内共同利用施設等, 特任教授 (10126295)
• Co-Investigator(Kenkyū-buntansha): NGUYENTUAN CUONG 東京農工大学, 工学(系)研究科(研究院), 特任助教 (10814246)
• Project Period (FY): 2019-04-01 – 2023-03-31
• Keywords: 手書き数式認識 / 構造解析 / 文脈処理 / 深層ニューラルネットワーク / アテンション

Outline of Final Research Achievements

This research aimed to recognize handwritten mathematical expressions, where ambiguities occurs in symbol segmentation, symbol identification, and positional relationship identification. Context processing is difficult unlike natural languages due to the lack of redundancy. This research clarified that it is superior to learn geometric context implicitly by Deep Neural Networks and evaluate language context explicitly by a language model. Specifically, an encoder/decoder model that pays attention to multiple substructures, semi-supervised learning that compensates for the lack of labeled learning patterns, and weighting the evaluation function with language context improved recognition performance. Furthermore, a method was proposed to artificially generate learning patterns, which were availed for an international recognition contest.

Free Research Field

人間情報学

Academic Significance and Societal Importance of the Research Achievements

手書き数式認識は，構造解析的パターン認識の典型的問題であり，本研究成果は，その進展を阻んできたノイズを含む多次元構造解析の曖昧性解消に資する．また，本研究により，手書きによる数式解答の答合せや，数式解答の自動採点・採点支援の土台ができた．自動採点は，受験者が自分の答案に対する採点を確認し，誤採点があれば照会できる仕組みを前提に，機械が手書き認識と採点を行って受験者に返す．採点結果を即時に受験者に返して復習を促すことができ，採点の労力を軽減できる．採点支援には，解答を事前に機械採点やクラスタリングし，また，採点の注意点を明確にすることで，採点者による誤った採点やばらつきを防止できる．