2021 Fiscal Year Annual Research Report
Temporal knowledge supervision for pre-training tranfer learning models
Publicly Offered Research
| Project Area | Chronogenesis: how the mind generates time |
|---|
| Project/Area Number |
21H00308
|
|---|
| Research Institution | Kyoto University |
|---|
Principal Investigator |
程 飛 京都大学, 情報学研究科, 特定助教 (70801570)
|
|---|
| Project Period (FY) |
2021-04-01 – 2023-03-31
|
| Keywords | temporal reasoning / transfer learning / knowledge pre-training / contrastive learning |
|---|
| Outline of Annual Research Achievements |
Reasoning over temporal knowledge relevant to the events mentioned in the text can help us understand when the events begin, how long they last, how frequent they are, etc. In this year:
We obtain significant progress on the research topic last year: event `duration' as the supervision for pre-training. Our paper is accepted by the 13th International Conference on Language Resources and Evaluation (LREC 2022).
We explore another research line of using distant supervion of existing human-annotated data for pre-training to improve the task of temporal relation extraction. The experimental results suggest our de-noise contrastive learning method outperforms state-of-the-art methods. We present it in the domestic conference NLP2022 and submit the furter to the international conference.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
According to the plan, we started the project by doing a detailed survey of the related task definition of temporal information extraction, previous approaches, and existing corpora.
We bought several Graphics Processing Unit (GPU) servers for conducting a series of transfer learning pre-training experiments.
We manage to leverage event `duration' as pre-training supervision for improving the task of temporal commonsense question answering. Our method significantly outperforms other pre-training methods in various evaluation metrics.
The paper has been accepted by the international conference on Language Resources and Evaluation (LREC 2022).
|
| Strategy for Future Research Activity |
Although transfer learning models (ELMO, GPT, BERT, etc.) achieve remarkable impacts on the NLP community, temporal knowledge can hardly be learned from context explicitly.
In the last year, we managed to leverage existing human-annotated event `duration' knowledge as pre-training supervision to improve the temporal commonsense question answering task.
In this year, we plan to go further to explore the feasibility of extracting various temporal supervision ( including duration, date, frequency, etc.) from large-scale raw corpora for pre-training transfer learning models. We assume this research can be beneficial to broad downstream tasks.
|
Research Products
(3 results)
