Adversarial Training for Robust and Generalizable Natural Language Processing
| Project/Area Number |
21K17802
|
|---|
| Research Category |
Grant-in-Aid for Early-Career Scientists
|
| Allocation Type | Multi-year Fund |
|---|
| Review Section |
Basic Section 61030:Intelligent informatics-related
|
|---|
| Research Institution | Nara Institute of Science and Technology (2023)
Ochanomizu University (2021-2022) |
|---|
Principal Investigator |
KANASHIRO・PEREIRA LIS・WEIJI (KANASHIROPEREIRA LISWEIJI) 奈良先端科学技術大学院大学, 先端科学技術研究科, 特任助教 (50896579)
|
|---|
| Project Period (FY) |
2021-04-01 – 2025-03-31
|
| Project Status |
Granted (Fiscal Year 2023)
|
| Budget Amount *help |
¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2022: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000)
Fiscal Year 2021: ¥3,120,000 (Direct Cost: ¥2,400,000、Indirect Cost: ¥720,000)
|
|---|
| Keywords | adversarial training / nlp / NLP / machine learning / robustness / deep learning / language model |
|---|
| Outline of Research at the Start |
In this research, we aim to improve the generalization and robustness of pre-trained language models on downstream NLP tasks by adopting adversarial training. Adversarial training has a great potential to improve model robustness and generalization, as shown by recent works. Moreover, adversarial training works as an online data augmentation method and can help improve model performance on low-resource scenarios. It can also help improve model performance without increasing the model size, which is helpful in scenarios where computational resources are limited.
|
| Outline of Annual Research Achievements |
We have so far accomplished most of the proposed research questions from the initial proposal during the execution of the project. We have shown that applying perturbations to other layers of the network improves current adversarial training methods for natural language processing (NLP). Besides applying perturbations at the embedding level, and exploring applying perturbations to other layers of the model or a combination of layers and performing a comparison of these variations. Similarity, we have shown that multi-task learning also improves current adversarial training methods for NLP. We have also applied our models to Japanese NLP tasks and achieved similar improvements, showing that our models are language-agnostic.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
As for the remaining research question: using prior knowledge that can guide the algorithm to generate better perturbations, we have made advancements by performing several experiments and we have a draft under submission at the moment.
|
| Strategy for Future Research Activity |
As for this last year, we plan to gather all results obtained into a major publication, such as a journal. In addition, with the rapid progress and releases of language models, such as ChatGPT, we plan to also apply our proposed models to such language models, and verify if they can further improve the performance of such models. Current research has shown that even models such as ChatGPT are susceptible to adversarial attacks and can have their performance degraded by them. From these results, we plan to prepare and submit another draft to a major conference.
|
Report
(3 results)
Research Products
(8 results)
