2021 Fiscal Year Research-status Report
Development of Interactive Advisory System for Security Export Control
| Project/Area Number |
20K12556
|
|---|
| Research Institution | Hokkaido University |
|---|
Principal Investigator |
大林 明彦 北海道大学, 産学・地域協働推進機構, 教授 (80798124)
|
|---|
| Co-Investigator(Kenkyū-buntansha) |
RZEPKA Rafal 北海道大学, 情報科学研究院, 助教 (80396316)
|
|---|
| Project Period (FY) |
2020-04-01 – 2023-03-31
|
| Keywords | 該非判定 / エキスパートシステム / 対話システム / テキスト分類 / 質疑応答 |
|---|
| Outline of Annual Research Achievements |
During the second year of our project we extended our question-answer database for the conversational expert system for supporting non-expert in deciding if their research or goods for export are require governmental permission. We also created a synonym dataset of chemical substances based on CAS numbers (by hiring a worker with the grant funds) to improve mapping terms in queries to the keywords in our database. This lead to enriching triplets for our knowledge graph which will now contains concepts as "SynonymOf", "HasCAS", "HasDefinition", "AddressHasText" or "IsControlledBy". We experimented with visualizing these concepts to have a better insight into connections among terms used in the Controlled Goods/Technologies Matrix Table. This should improve the process of searching legal texts for answers to the user’s queries. We published our findings from experiments with question answering and presented them during an international conference. It appeared that traditional and modern learning approaches (LDA-Based Ranker, BERT Ranker, SQuAD model-based QA and GPT-2-based Answer Generation) achieved worse results than keyword matching based on the glossary published as an index for the legal text regarding trade security issues. This convinced us to shift the weight from automatic to half-automatic in our approach for searching paragraphs related to user queries. However, there is still a necessity for using similarity calculation in order to deal with terms which are used in queries but are not included in the glossary used for keyword matching.
|
| Current Status of Research Progress |
Current Status of Research Progress
3: Progress in research has been slightly delayed.
Reason
Although the deep learning-based approach appeared less useful as we expected for question answering, we did not give it up entirely and started testing XLM-RoBERTa zero-shot learning to see if it can help to automatically categorize answer types (e.g. "Controlled" or "Not Controlled") and question intents (e.g. "Yes/No Question", "Asking for Confirmation"). Both task did not yield satisfactory results (question intent 43.02%, answer type 63.95% of correct predictions) showing that we need more QA data to switch to the few-shot approach. Mechanizing the process of referencing other parts of documents or excluding them appeared more difficult as expected which led to the slower progress that we anticipated. For this reason we rethought our approach and moved to the rule-based one. We also began experimenting with classifying possible relations which are not explicitly derivable from the legal texts. For example, it is possible to automatically classify matrix category ("biological weapons", "nuclear power" or "missiles") regarding an unknown term in a query, but it would be more useful for generating explanations upon recognition if the input is related to danger or not. We managed to create a LINE version of our expert dialog system (currently working on the Slack platform), however university security restrictions do not allow access, therefore we are currently working on minimizing the system to be stored on an external server. By doing so, we are going to make our final system more accessible to users as LINE platform is being used more widely in Japan than Slack.
|
| Strategy for Future Research Activity |
According to our findings from the previous year, the most important step to finalize the dialog expert system is to shift the weight from the automatic reasoning within the ontology to manually crafted rules considering answer search algorithm. Referencing other parts of documents and including or excluding them is an easy task for a human, but tricky for computers. While in English hierarchical coordinate structure is relatively simple, in the case of Japanese legal texts there are specific coordinates which must be dealt with. If the manual approach does not resolve this problem, we plan to implement Yamakoshi et al.’s method for combining traditional methods with predictive capabilities of neural language models.
Because the system is expected to give exact and trustful answers, we will hire workers not only for programming the rules but also for thorough testing. This is required for free input dialog systems because the system must also automatically decide if the query is related to the topic of trade security. We will also implement an algorithm for automatic calculation of answer credibility, because it is crucial to advise users to contact the human expert if the system is not sure in its reply or is not able to reply at all.
As the terms dataset we use originates in a copyrighted materials, we also plan to automatize the discovery process of name entities in order to publish our system as open source. The final version will be described with the experimental results in a journal publication.
|
| Causes of Carryover |
今年度は国際学会発表をするも、コロナ禍の影響でオンライン発表であったため、支出は低調。次年度はKES2022等での国際学会発表を予定している。
|
Research Products
(2 results)
