Temporal knowledge supervision for pre-training tranfer learning models
Publicly Offered Research
| Project Area | Chronogenesis: how the mind generates time |
|---|
| Project/Area Number |
21H00308
|
|---|
| Research Category |
Grant-in-Aid for Scientific Research on Innovative Areas (Research in a proposed research area)
|
| Allocation Type | Single-year Grants |
|---|
| Review Section |
Complex systems
|
|---|
| Research Institution | Kyoto University |
|---|
Principal Investigator |
程 飛 京都大学, 情報学研究科, 特定助教 (70801570)
|
|---|
| Project Period (FY) |
2021-04-01 – 2023-03-31
|
| Project Status |
Completed (Fiscal Year 2022)
|
| Budget Amount *help |
¥5,200,000 (Direct Cost: ¥4,000,000、Indirect Cost: ¥1,200,000)
Fiscal Year 2022: ¥2,600,000 (Direct Cost: ¥2,000,000、Indirect Cost: ¥600,000)
Fiscal Year 2021: ¥2,600,000 (Direct Cost: ¥2,000,000、Indirect Cost: ¥600,000)
|
|---|
| Keywords | Temporal Reasoning / Commonsense Reasoning / Large Language Model / Weak Supervision / temporal knowledge / deep neural networks / transfer learning / temporal reasoning / knowledge pre-training / contrastive learning / Temporal Knowledge / Natural Language / Neural Networks / Deep Learning |
|---|
| Outline of Research at the Start |
We design a series of empirical experiments to investigate the feasibility of exploiting temporal knowledge as supervision during pretraining. We plan to develop sufficent computing environments to accelarate training progress. We believe the temporal-aware representations will play an important role towards better understanding of time. Several extended topics can be explored, such as how human brain recognizes duration, frequency scales in languages.
|
| Outline of Annual Research Achievements |
Large language models (LLMs) often lack the ability to reason numerical and temporal knowledge such as how long an event lasts, how frequent it is, etc. Enhancing off-the-shelf LLMs' reasoning capability with large-scale weak supervision becomes a crucial topic. We relieved the reliance on human annotation and propose a bimodal voting strategy to obtain high-quality semi temporal knowledge. We re-train off-the-shelf LLMs on semi-supervision and observe significant improvement in temporal commonsense reasoning. We also explore a novel approach for identifying semantic relations (including temporal relations) between two events by revealing the labels of the most similar training examples. We have several papers accepted by the top AI conferences (EMNLP, EACL) and domestic conferences.
|
| Research Progress Status |
令和4年度が最終年度であるため、記入しない。
|
| Strategy for Future Research Activity |
令和4年度が最終年度であるため、記入しない。
|
Report
(2 results)
Research Products
(13 results)
