| 研究課題/領域番号 |
23K11757
|
|---|
| 研究機関 | 北海道大学 |
|---|
研究代表者 |
大林 明彦 北海道大学, 産学・地域協働推進機構, 教授 (80798124)
|
|---|
| 研究分担者 |
RZEPKA Rafal 北海道大学, 情報科学研究院, 助教 (80396316)
|
|---|
| 研究期間 (年度) |
2023-04-01 – 2026-03-31
|
| キーワード | export control / dialog system / question answering / trade security education |
|---|
| 研究実績の概要 |
For the first year of the grant we have planned to concentrate on developing an ontology for teaching our dialog system rules underneath trade security to be able to add explanation abilities to the existing system. However, when we were experimenting with matching masked language models like RoBERTa to help with retrieving graph nodes, OpenAI has released Chat-GPT, a large language model surpassing previous applications of natural language processing. This forced us to start broad tests to see if this closed model as well as other open-source ones can replace modules we have developed so far. We have performed experiments with GPT-3.4 and GPT-4 and have observed that they were trained on vast export control-related texts which helps them to answer questions about regulations in Japanese language. However, the expert evaluation showed several problems. For example, even the top (at the time) commercial model (GPT-4) hallucinated names of the regulations, cited ones that exist but not in Japan, etc. We shared our experimental findings in a international conference publication. We have tested models trained on Japanese language directly, but the performance was poor.
|
| 現在までの達成度 (区分) |
現在までの達成度 (区分)
3: やや遅れている
理由
One one hand, the sudden technological jump in natural language processing caused an unexpected turn in our plans, as we had to assimilate the new trend and perform experiments which have not been planned. As the grant topic is rare, waiting for somebody to do the testing of newest GPTs on export control-related topics was not a promising idea, we performed experiments ourselves. On the other hand, we have learned the latest trends in prompting and few-shot learning, and employed a basic RAG (Retrieval-Augmented Generation) algorithm. This delay used for learning will possibly pay off in the next year as we are now able to extend our ideas for creating an export control ontology in more automatic manner. We also now have a strong baseline model for experimenting with explanations and teaching in a more user-friendly manner because large language models thrive in making quizzes and assessing users’ knowledge. Nevertheless, we must carry out very careful tests to avoid hallucination and utilizing the newest and safest RAGs equipped with fact-checking capabilities.
|
| 今後の研究の推進方策 |
As the export control-related data is scarce, methods like fine tuning are relatively difficult to implement, we started trials with RAG (Retrieval-Augmented Generation). Searching (retrieval) module seems to improve finding related regulations, but generation causes errors and does not help with explaining. Our basic hypothesis that ontological knowledge graph can be useful with explaining dangers and understanding what users do not know, stays unchanged. However, new opportunities have appeared - short semantic relations can be now populated with large language models, they also can be used as examples in few-shot learning approaches and in fine-tuning which requires many more data. Combining rule-based trustful methods with masked language models and latest GPTs can bring new opportunities to develop our system faster than planned. The system can be also extended sooner when it comes to the educational side of the chatbot. If the RAG’s generation shows hallucinations, we will keep the retrieval module and concentrate on refining results via interaction with the user.
|
| 次年度使用額が生じた理由 |
Because of the appearance of a new powerful technology, we had to revise our plans regarding deployment and experimentation. As we needed to learn it from scratch, we had smaller output and saved grant money on travel and hiring a programmer. The new technology in its basic form is easier to program, therefore most of the experiments we managed to do ourselves. The cost of using GPT models was smaller than expected letting us abandon our plan to add a GPU to our working machine where our system is deployed. However, it may become necessary for developing a faster RAG algorithm. Also, except increasing the number of presentations, we have arranged meetings with export control experts in Japan. For this reason we want to use the save grant money in the next fiscal year.
|
