• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Nugget-based Automatic Evaluation of Task-oriented dialogues

Research Project

Project/Area Number 17H01830
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypeSingle-year Grants
Section一般
Research Field Library and information science/Humanistic social informatics
Research InstitutionWaseda University

Principal Investigator

Sakai Tetsuya  早稲田大学, 理工学術院, 教授 (80723519)

Project Period (FY) 2017-04-01 – 2021-03-31
Project Status Completed (Fiscal Year 2020)
Budget Amount *help
¥14,690,000 (Direct Cost: ¥11,300,000、Indirect Cost: ¥3,390,000)
Fiscal Year 2020: ¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2019: ¥2,860,000 (Direct Cost: ¥2,200,000、Indirect Cost: ¥660,000)
Fiscal Year 2018: ¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2017: ¥3,250,000 (Direct Cost: ¥2,500,000、Indirect Cost: ¥750,000)
Keywords対話 / ヘルプデスク / ナゲット / ordinal quantification / データセット / 評価指標 / 評価 / 自然言語 / 情報アクセス / 情報検索 / 人工知能
Outline of Final Research Achievements

We served as task organisers of NTCIR (NII Testbeds and Community for Information access Research), an international evaluation forum, and constructed the DCH-2 dialogue data set. Our paper on an early version of this data set received the excellent paper runner-up award at WebDB Forum 2018. We also designed an evaluation measure for a task that requires systems to estimate the gold distribution of dialogue quality scores and showed its effectiveness. The work has been accepted as a full paper at ACL-IJCNLP 2021, a top conference in natural language processing.

Academic Significance and Societal Importance of the Research Achievements

我々が構築した対話データセットDCH-2は利用許諾書にサインするだけで誰でも利用できる。元データは中国語であるが全てに人手による英訳を付与しているため汎用性が高い。さらに、我々が提案した対話品質スコアの分布を推定するタスクのための評価指標は、ordinal quantificationと呼ばれる形態のすべてのタスク(「ネガティブ」「ニュートラル」「ポジティブ」のような順序尺度上の分布を扱うタスク)に適用可能である。

Report

(5 results)
  • 2020 Annual Research Report   Final Research Report ( PDF )
  • 2019 Annual Research Report
  • 2018 Annual Research Report
  • 2017 Annual Research Report
  • Research Products

    (14 results)

All 2020 2019 2018 2017 Other

All Int'l Joint Research (2 results) Journal Article (1 results) (of which Int'l Joint Research: 1 results,  Peer Reviewed: 1 results,  Open Access: 1 results) Presentation (8 results) (of which Int'l Joint Research: 8 results) Remarks (3 results)

  • [Int'l Joint Research] Naver Corporation(韓国)

    • Related Report
      2020 Annual Research Report
  • [Int'l Joint Research] 清華大学/華為技術/今日頭條(中国)

    • Related Report
      2017 Annual Research Report
  • [Journal Article] Towards Automatic Evaluation of Customer-Helpdesk Dialogues2018

    • Author(s)
      Zhaohao Zeng, Cheng Luo, Lifeng Shang, Hang Li, Tetsuya Sakai
    • Journal Title

      Journal of Information Processing

      Volume: 26 Issue: 0 Pages: 768-778

    • DOI

      10.2197/ipsjjip.26.768

    • NAID

      130007511649

    • ISSN
      1882-6652
    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access / Int'l Joint Research
  • [Presentation] RSLNV at the NTCIR-15 Dialogue Evaluation (DialEval-1) Task2020

    • Author(s)
      Ting Cao, Fan Zhang, Haoxiang Shi, Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, Injae Lee, Kyungduk Kim, and Inho Kang
    • Organizer
      NTCIR-15
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] SKYMN at the NTCIR-15 DialEval-1 Task2020

    • Author(s)
      Junjie Wang, Yuxiang Zhang, Tetsuya Sakai, and Hayato Yamana
    • Organizer
      NTCIR-15
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Overview of the NTCIR-15 Dialogue Evaluation (DialEval-1) Task2020

    • Author(s)
      Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, and Inho Kang
    • Organizer
      NTCIR-15
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] SLSTC at the NTCIR-14 STC-3 Dialogue Quality and Nugget Detection Subtasks2019

    • Author(s)
      Sosuke Kato and Rikiya Suzuki and Zhaohao Zeng and Tetsuya Sakai
    • Organizer
      NTCIR-15
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Overview of the NTCIR-14 Short Text Conversation Task: Dialogue Quality and Nugget Detection Subtasks2019

    • Author(s)
      Zhaohao Zeng and Sosuke Kato and Tetsuya Sakai
    • Organizer
      NTCIR-15
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Comparing Two BinnedProbability Distributions for Information Access Evaluation2018

    • Author(s)
      Tetsuya Sakai
    • Organizer
      ACM SIGIR 2018
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Test Collections and Measures for Evaluating Customer-Helpdesk Dialogues2017

    • Author(s)
      Zeng, Z., Luo, C., Shang, L., Li, H., and Sakai, T.
    • Organizer
      The 8th International Workshop on Evaluating Information Access (EVIA 2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Towards Automatic Evaluation of Multi-Turn Dialogues: A Task Design that Leverages Inherently Subjective Annotations2017

    • Author(s)
      Sakai, T.
    • Organizer
      The 8th International Workshop on Evaluating Information Access (EVIA 2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Remarks] NTCIR Dialogue Evaluation Task

    • URL

      http://sakailab.com/dialeval1/

    • Related Report
      2020 Annual Research Report
  • [Remarks] ヘルプデスク対話データセットDCH-1 (中国語、一部英訳あり)

    • URL

      http://waseda.box.com/DCH-0-1

    • Related Report
      2017 Annual Research Report
  • [Remarks] NTCIR-14 Short Text Conversation 3 タスク ホームページ

    • URL

      http://sakailab.com/ntcir14stc3/

    • Related Report
      2017 Annual Research Report

URL: 

Published: 2017-04-28   Modified: 2022-01-27  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi