Research on multimodal formula input UI and mathematics learning support environment

2023 Fiscal Year Final Research Report

• Project/Area Number: 20K12117
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Review Section: Basic Section 62030:Learning support system-related
• Research Institution: Mukogawa Women's University
• Principal Investigator: FUKUI Tetsuo 武庫川女子大学, 社会情報学部, 教授 (70218890)
• Project Period (FY): 2020-04-01 – 2024-03-31
• Keywords: マルチモーダル学習 / 音声入出力 / 数式入力インタフェース / 数学文書エディタ / 知的学習支援システム / 数学eラーニング

Outline of Final Research Achievements

We have realized mathematical formula input by voice by introducing voice recognition technology to the previous math input UI that uses AI predictive conversion from mathematically ambiguous character strings. Adding this UI, we have developed a multimodal formula input UI such as keyboard, touch screen, and voice input that corresponds to a variety of user situations. Furthermore, we have developed a mathematics integrated document editor for creating mathematics e-learning materials that implements this UI, and released it as open source. Additionally, in order to verify this system, we created a prototype mathematics learning support environment that included learning materials based on the creation of function graphs, and conducted classes with approximately 60 high school students, which resulted in a high level of satisfaction. These results were presented at academic conferences and the developed system was made available on our public server.

Free Research Field

情報科学

Academic Significance and Societal Importance of the Research Achievements

本研究のAIによる数式曖昧文字列予測変換は，数式の音声の読み上げ文字と親和性が高く，低コストで中学数学レベルの数式の音声認識が実現できることを示した。しかし，数式は似たような発音が多く100 %の認識率を目指すには限界がある。本研究では，音声のみではなく，キーボード・タッチスクリーン入力と組み合わせたマルチモーダルな数式入力UIを実現しており，誤認や入力ミスに対して修正のしやすさが向上したことにある。社会的意義として，マルチモーダルなUIを備えた数学文書エディタを公開したことによって，教師による教材作成支援だけでなく，多様な場面に対応した数学学習支援環境を提供できるようになった。