2023 Fiscal Year Research-status Report
EmplifAI: Development of an Empathetic AI Chatbot to Support Cancer Follow-up Care
| Project/Area Number |
23K17007
|
|---|
| Research Institution | Nara Institute of Science and Technology |
|---|
Principal Investigator |
SHE WanJou 奈良先端科学技術大学院大学, 先端科学技術研究科, 特任助教 (40967410)
|
|---|
| Project Period (FY) |
2023-04-01 – 2026-03-31
|
| Keywords | Empathy / Large Language Model / Chatbot / Cancer Care |
|---|
| Outline of Annual Research Achievements |
My first-year research aims to develop an empathetic chatbot model for providing emotionally sensitive medical advice to cancer patients. I'm fortunate to utilize emerging Large Language Models (LLMs) like ChatGPT, llama-2, and mixtral, which have significantly enhanced my original study plan with groundbreaking generative AI technology.
In summary, I've collected 280 data points on emotionally challenged medical situations and 4230 data points of empathetic dialogues to address diverse patient emotions through crowdsourcing. I've also conducted interviews with eight cancer professionals to guide empathetic chatbot development. Additionally, I've published three chatbot papers in top HCI venues (CHI and IJHCI) and secured an additional chatbot grant from the JR West-Relief Foundation.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
I'm currently exploring the best methods to fine-tune a language model or construct our Retrieval Augmented Generation (RAG) framework. After receiving predominantly positive yet cautious feedback from our interviews with medical professionals, I've decided to prioritize implementing safety mechanisms in the development of our chatbot prototype. Additionally, I have relocated to Kyoto Institute of Technology and will be submitting the ethical application through the new institution. I aim to publish the two datasets as open-access resources for the Japanese Natural Language Processing research community. The interview data will be analyzed using a thematic analysis approach and submitted to the Journal of Medical Internet Research (JMIR).
|
| Strategy for Future Research Activity |
After successfully developing the language model candidates and integrating them into a chatbot user interface, my next step is to engage doctors in evaluating the safety of the chatbot's responses. The criteria for assessing the safety and performance of the model will include: Scientific Consensus, Potential Harm, Evidence of Miscomprehension, Evidence of Incorrect Retrieval, Evidence of Flawed Reasoning, Inappropriate or Incorrect Content, Helpfulness of Responses, Bedside Manner, and Empathy Level. I aim to enlist up to 5 doctors to evaluate the answers generated by multiple chatbot candidates. These criteria will guide me in selecting the safest and most effective model to advance to the final stage in the third year.
|
| Causes of Carryover |
One of the conference papers will be presented after April 2024, requiring budget allocation for the conference presentation. The event is scheduled to take place in the U.S. this May, and considering the lower rate of the yen to US dollar, I've decided to reallocate some budget from the first year to cover the additional travel expenses.
|
