2021 Fiscal Year Research-status Report
Knowledge-Base-Grounded Language Models
| Project/Area Number |
21K17814
|
|---|
| Research Institution | Institute of Physical and Chemical Research |
|---|
Principal Investigator |
HEINZERLING BENJAMIN 国立研究開発法人理化学研究所, 革新知能統合研究センター, 特別研究員 (50846491)
|
|---|
| Project Period (FY) |
2021-04-01 – 2023-03-31
|
| Keywords | language model / knowledge base / world knowledge / grounding |
|---|
| Outline of Annual Research Achievements |
The goal of the first year of this grant was the training and analysis of knowledge-base-grounded language model.
While not published yet, a prototype model that combines symbolic information from the knowledge base and textual information has been implemented. The next step is to evaluate this prototype both intrinsically by analyzing its internal representations, and extrinsically via knowledge-intensive downstream applications.
Towards the goal of analyzing internal representations of language models, one published case study analyzed the form in which entity and concept knowledge is stored in pretrained language models, finding strong evidence of local storage, i.e., such knowledge is often encoded in only a small number of neurons.
|
| Current Status of Research Progress |
Current Status of Research Progress
3: Progress in research has been slightly delayed.
Reason
During development of the knowledge-base grounded language model, which was planned to be fully completed during the first year of the funding period, the need for an intrinsic evaluation method was identified, as this would allow faster evaluation, which, in turn, will speed up development overall.
While the originally planned evaluation on knowledge-intensive downstream task remains the main evaluation method, some unforeseen time had to be spent to develop a novel method for the intrinsic evaluation and analysis of world knowledge in language models.
|
| Strategy for Future Research Activity |
Most of the first year of the funding period was spent prototyping and testing different architectures for the knowledge-base grounded language model, as well as developing a novel evaluation method. The plan for the next year of the funding period is as to wrap up both of these started sub-goals, i.e., publish the work on the novel intrinsic evaluation of world knowledge in language models, and then to scale the current prototype model to a full scale pretrained language model.
|
| Causes of Carryover |
As described in the progress report, the training of the the full-scale model, which was originally planned for the first year is now planned for the second year, which in turn means, that contrary to the original plan, costs for using the ABCI cluster have not been incurred during the first year. Furthermore, due to the continuation of the COVID-19 pandemic, no conference travel was undertaken.
The amount will be used in order to pay for computing costs on the ABCI cluster and to cover conference travel costs, should they occur.
|
