View The Document

Accession Number:

AD1157726

Title:

A Semi-automated Evaluation Metric for Dialogue Model Coherence

Author(s):

• Gandhe, Sudeep
• Traum, David

Author Organization(s):

• UNIVERSITY OF SOUTHERN CALIFORNIA LOS ANGELES

Report Date:

2016-04-01

Abstract:

We propose a new metric, Voted Appropriateness, which can be used to automatically evaluate dialogue policy decisions, once some wizard data has been collected. We show that this metric outperforms a previously proposed metric Weak agreement. We also present a taxonomy for dialogue model evaluation schemas, and orient our new metric within this taxonomy.

Document Type:

Conference:

Journal:

Pages:

9

DOI:

10.1007/978-3-319-21834-2_19

File Size:

0.28MB

Contracts:

Grants:

Descriptors:

• computational linguistics
• language
• natural language processing
• linguistics
• dialogue systems
• electronic mail
• machine translation
• natural languages
• agreements
• computational science
• automatic
• cognitive systems engineering
• computer languages
• judgment
• learning
• machine learning
• reinforcement learning

Identifiers:

• Automatic Evaluation
• Virtual Human
• Dialogue Model
• Context Evaluation

SubjectCategory:

• Mathematical and Computer Sciences

Communities of Interest:

• Autonomy

Distribution Statement:

Approved For Public Release

View The Document