| 研究課題/領域番号 |
23K11757
|
|---|
| 研究種目 |
基盤研究(C)
|
| 配分区分 | 基金 |
|---|
| 応募区分 | 一般 |
|---|
| 審査区分 |
小区分90020:図書館情報学および人文社会情報学関連
|
|---|
| 研究機関 | 北海道大学 |
|---|
研究代表者 |
大林 明彦 北海道大学, 産学・地域協働推進機構, 教授 (80798124)
|
|---|
| 研究分担者 |
RZEPKA Rafal 北海道大学, 情報科学研究院, 助教 (80396316)
|
|---|
| 研究期間 (年度) |
2023-04-01 – 2026-03-31
|
| 研究課題ステータス |
交付 (2023年度)
|
| 配分額 *注記 |
4,030千円 (直接経費: 3,100千円、間接経費: 930千円)
2025年度: 910千円 (直接経費: 700千円、間接経費: 210千円)
2024年度: 1,300千円 (直接経費: 1,000千円、間接経費: 300千円)
2023年度: 1,820千円 (直接経費: 1,400千円、間接経費: 420千円)
|
|---|
| キーワード | export control / dialog system / question answering / trade security education / 該非判定 / エキスパートシステム / 対話システム / テキスト分類 / 質疑応答 |
|---|
| 研究開始時の研究の概要 |
本研究では、輸出管理と人工知能の専門家が協働して、研究者自身が研究領域の機微度を容易にチェックできるユーザフレンドリーなガイダンスシステムの提供を目指す。本システムは、輸出管理の法令文を計算処理可能な形式に変換し、研究者から提供される情報から自動的に機微度を推論する。また不足する情報は、外部知識により機械学習で強化されたチャットボットとの対話により補完する。本システムにより、専門知識を持たない者でも輸出管理を正しく運用し、機微技術の流出を防止することが可能となる。本研究で開発したガイダンスシステムは、無償提供とし、オープンソースとして自由に拡張できるものとする。
|
| 研究実績の概要 |
For the first year of the grant we have planned to concentrate on developing an ontology for teaching our dialog system rules underneath trade security to be able to add explanation abilities to the existing system. However, when we were experimenting with matching masked language models like RoBERTa to help with retrieving graph nodes, OpenAI has released Chat-GPT, a large language model surpassing previous applications of natural language processing. This forced us to start broad tests to see if this closed model as well as other open-source ones can replace modules we have developed so far. We have performed experiments with GPT-3.4 and GPT-4 and have observed that they were trained on vast export control-related texts which helps them to answer questions about regulations in Japanese language. However, the expert evaluation showed several problems. For example, even the top (at the time) commercial model (GPT-4) hallucinated names of the regulations, cited ones that exist but not in Japan, etc. We shared our experimental findings in a international conference publication. We have tested models trained on Japanese language directly, but the performance was poor.
|
| 現在までの達成度 (区分) |
現在までの達成度 (区分)
3: やや遅れている
理由
One one hand, the sudden technological jump in natural language processing caused an unexpected turn in our plans, as we had to assimilate the new trend and perform experiments which have not been planned. As the grant topic is rare, waiting for somebody to do the testing of newest GPTs on export control-related topics was not a promising idea, we performed experiments ourselves. On the other hand, we have learned the latest trends in prompting and few-shot learning, and employed a basic RAG (Retrieval-Augmented Generation) algorithm. This delay used for learning will possibly pay off in the next year as we are now able to extend our ideas for creating an export control ontology in more automatic manner. We also now have a strong baseline model for experimenting with explanations and teaching in a more user-friendly manner because large language models thrive in making quizzes and assessing users’ knowledge. Nevertheless, we must carry out very careful tests to avoid hallucination and utilizing the newest and safest RAGs equipped with fact-checking capabilities.
|
| 今後の研究の推進方策 |
As the export control-related data is scarce, methods like fine tuning are relatively difficult to implement, we started trials with RAG (Retrieval-Augmented Generation). Searching (retrieval) module seems to improve finding related regulations, but generation causes errors and does not help with explaining. Our basic hypothesis that ontological knowledge graph can be useful with explaining dangers and understanding what users do not know, stays unchanged. However, new opportunities have appeared - short semantic relations can be now populated with large language models, they also can be used as examples in few-shot learning approaches and in fine-tuning which requires many more data. Combining rule-based trustful methods with masked language models and latest GPTs can bring new opportunities to develop our system faster than planned. The system can be also extended sooner when it comes to the educational side of the chatbot. If the RAG’s generation shows hallucinations, we will keep the retrieval module and concentrate on refining results via interaction with the user.
|
