Nugget-based Automatic Evaluation of Task-oriented dialogues
| Project/Area Number |
17H01830
|
|---|
| Research Category |
Grant-in-Aid for Scientific Research (B)
|
| Allocation Type | Single-year Grants |
|---|
| Section | 一般 |
|---|
| Research Field |
Library and information science/Humanistic social informatics
|
|---|
| Research Institution | Waseda University |
|---|
Principal Investigator |
Sakai Tetsuya 早稲田大学, 理工学術院, 教授 (80723519)
|
|---|
| Project Period (FY) |
2017-04-01 – 2021-03-31
|
| Project Status |
Completed (Fiscal Year 2020)
|
| Budget Amount *help |
¥14,690,000 (Direct Cost: ¥11,300,000、Indirect Cost: ¥3,390,000)
Fiscal Year 2020: ¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2019: ¥2,860,000 (Direct Cost: ¥2,200,000、Indirect Cost: ¥660,000)
Fiscal Year 2018: ¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2017: ¥3,250,000 (Direct Cost: ¥2,500,000、Indirect Cost: ¥750,000)
|
|---|
| Keywords | 対話 / ヘルプデスク / ナゲット / ordinal quantification / データセット / 評価指標 / 評価 / 自然言語 / 情報アクセス / 情報検索 / 人工知能 |
|---|
| Outline of Final Research Achievements |
We served as task organisers of NTCIR (NII Testbeds and Community for Information access Research), an international evaluation forum, and constructed the DCH-2 dialogue data set. Our paper on an early version of this data set received the excellent paper runner-up award at WebDB Forum 2018. We also designed an evaluation measure for a task that requires systems to estimate the gold distribution of dialogue quality scores and showed its effectiveness. The work has been accepted as a full paper at ACL-IJCNLP 2021, a top conference in natural language processing.
|
| Academic Significance and Societal Importance of the Research Achievements |
我々が構築した対話データセットDCH-2は利用許諾書にサインするだけで誰でも利用できる。元データは中国語であるが全てに人手による英訳を付与しているため汎用性が高い。さらに、我々が提案した対話品質スコアの分布を推定するタスクのための評価指標は、ordinal quantificationと呼ばれる形態のすべてのタスク（「ネガティブ」「ニュートラル」「ポジティブ」のような順序尺度上の分布を扱うタスク）に適用可能である。
|
Report
(5 results)
Research Products
(14 results)
