| 研究実績の概要 |
ヘルプデスク対話データセットversion 0の公開 (http://waseda.box.com/DCH-0-1) と、NTCIR-14 Short Text Conversation Task (STC-3) (http://sakailab.com/ntcir14stc3/) の対話サブタスクの設計および評価指標の検討を行い、以下の学会発表を行った。
・Zeng, Z. et al.: Test Collections and Measures for Evaluating Customer-Helpdesk Dialogues, EVIA 2017, pp.1-9, 査読あり, 2017.
概要: We address the problem of evaluating textual, task-oriented dialogues between the customer and the helpdesk, such as those that take the form of online chats. As an initial step towards evaluating automatic helpdesk dialogue systems, we have constructed a test collection comprising 3,700 real Customer-Helpdesk multiturn dialogues by mining Weibo, a major Chinese social media. We have made our test collection DCH-1 publicly available for research purposes.
・Sakai, T: Towards Automatic Evaluation of Multi-Turn Dialogues: A Task Design that Leverages Inherently Subjective Annotations, EVIA 2017, pp.24-30, 査読あり, 2017.概要: This paper proposes a design of a shared task whose ultimate goal is automatic evaluation of multi-turn, dyadic, textual helpdesk dialogues. The proposed task takes the form of an offline evaluation, where participating systems are given a dialogue as input, and output at least one of the following: (1) an estimated distribution of the annotators’ quality ratings for that dialogue; and (2) an estimated distribution of the annotators’ nugget type labels for each utterance block (i.e., a maximal sequence of consecutive posts by the same utterer) in that dialogue.
|
