Nugget-based Automatic Evaluation of Task-oriented dialogues

• Project/Area Number: 17H01830
• Research Category: Grant-in-Aid for Scientific Research (B)
• Allocation Type: Single-year Grants
• Section: 一般
• Research Field: Library and information science/Humanistic social informatics
• Research Institution: Waseda University
• Principal Investigator: Sakai Tetsuya 早稲田大学, 理工学術院, 教授 (80723519)
• Project Period (FY): 2017-04-01 – 2021-03-31
• Project Status: Completed (Fiscal Year 2020)
• Budget Amount: ¥14,690,000 (Direct Cost: ¥11,300,000、Indirect Cost: ¥3,390,000); Fiscal Year 2020: ¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000); Fiscal Year 2019: ¥2,860,000 (Direct Cost: ¥2,200,000、Indirect Cost: ¥660,000); Fiscal Year 2018: ¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000); Fiscal Year 2017: ¥3,250,000 (Direct Cost: ¥2,500,000、Indirect Cost: ¥750,000)
• Keywords: 対話 / ヘルプデスク / ナゲット / ordinal quantification / データセット / 評価指標 / 評価 / 自然言語 / 情報アクセス / 情報検索 / 人工知能

Outline of Final Research Achievements

We served as task organisers of NTCIR (NII Testbeds and Community for Information access Research), an international evaluation forum, and constructed the DCH-2 dialogue data set. Our paper on an early version of this data set received the excellent paper runner-up award at WebDB Forum 2018. We also designed an evaluation measure for a task that requires systems to estimate the gold distribution of dialogue quality scores and showed its effectiveness. The work has been accepted as a full paper at ACL-IJCNLP 2021, a top conference in natural language processing.

Academic Significance and Societal Importance of the Research Achievements

我々が構築した対話データセットDCH-2は利用許諾書にサインするだけで誰でも利用できる。元データは中国語であるが全てに人手による英訳を付与しているため汎用性が高い。さらに、我々が提案した対話品質スコアの分布を推定するタスクのための評価指標は、ordinal quantificationと呼ばれる形態のすべてのタスク（「ネガティブ」「ニュートラル」「ポジティブ」のような順序尺度上の分布を扱うタスク）に適用可能である。

Research Products

• [Int'l Joint Research] Naver Corporation(韓国)
• [Int'l Joint Research] 清華大学/華為技術/今日頭條(中国)
• [Journal Article] Towards Automatic Evaluation of Customer-Helpdesk Dialogues (2018)
  • Author(s): Zhaohao Zeng, Cheng Luo, Lifeng Shang, Hang Li, Tetsuya Sakai
  • Journal Title: Journal of Information Processing Volume: 26 Issue: 0 Pages: 768-778
  • DOI: 10.2197/ipsjjip.26.768
  • NAID: 130007511649
  • ISSN: 1882-6652
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Presentation] RSLNV at the NTCIR-15 Dialogue Evaluation (DialEval-1) Task (2020)
  • Author(s): Ting Cao, Fan Zhang, Haoxiang Shi, Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, Injae Lee, Kyungduk Kim, and Inho Kang
  • Organizer: NTCIR-15
  • Int'l Joint Research
• [Presentation] SKYMN at the NTCIR-15 DialEval-1 Task (2020)
  • Author(s): Junjie Wang, Yuxiang Zhang, Tetsuya Sakai, and Hayato Yamana
  • Organizer: NTCIR-15
  • Int'l Joint Research
• [Presentation] Overview of the NTCIR-15 Dialogue Evaluation (DialEval-1) Task (2020)
  • Author(s): Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, and Inho Kang
  • Organizer: NTCIR-15
  • Int'l Joint Research
• [Presentation] SLSTC at the NTCIR-14 STC-3 Dialogue Quality and Nugget Detection Subtasks (2019)
  • Author(s): Sosuke Kato and Rikiya Suzuki and Zhaohao Zeng and Tetsuya Sakai
  • Organizer: NTCIR-15
  • Int'l Joint Research
• [Presentation] Overview of the NTCIR-14 Short Text Conversation Task: Dialogue Quality and Nugget Detection Subtasks (2019)
  • Author(s): Zhaohao Zeng and Sosuke Kato and Tetsuya Sakai
  • Organizer: NTCIR-15
  • Int'l Joint Research
• [Presentation] Comparing Two BinnedProbability Distributions for Information Access Evaluation (2018)
  • Author(s): Tetsuya Sakai
  • Organizer: ACM SIGIR 2018
  • Int'l Joint Research
• [Presentation] Test Collections and Measures for Evaluating Customer-Helpdesk Dialogues (2017)
  • Author(s): Zeng, Z., Luo, C., Shang, L., Li, H., and Sakai, T.
  • Organizer: The 8th International Workshop on Evaluating Information Access (EVIA 2017)
  • Int'l Joint Research
• [Presentation] Towards Automatic Evaluation of Multi-Turn Dialogues: A Task Design that Leverages Inherently Subjective Annotations (2017)
  • Author(s): Sakai, T.
  • Organizer: The 8th International Workshop on Evaluating Information Access (EVIA 2017)
  • Int'l Joint Research
• [Remarks] NTCIR Dialogue Evaluation Task
  • URL: http://sakailab.com/dialeval1/
• [Remarks] ヘルプデスク対話データセットDCH-1 (中国語、一部英訳あり)
  • URL: http://waseda.box.com/DCH-0-1
• [Remarks] NTCIR-14 Short Text Conversation 3 タスク ホームページ
  • URL: http://sakailab.com/ntcir14stc3/