Accession Number:

AD1107783

Title:

Testing the Accuracy of Automated Classification Systems Using Only Expert Ratings That Are Less Accurate Than the System

Descriptive Note:

Technical Report

Corporate Author:

MITRE CORP MCLEAN VA MCLEAN

Personal Author(s):

Report Date:

2014-01-01

Pagination or Media Count:

28.0

Abstract:

A method is presented to estimate the accuracy of automated classification systems using only expert ratings that may be substantially less accurate than the systems being evaluated. The estimation method begins with multiple expert ratings on test cases, uses the level of inter-rater agreement to estimate rater accuracy, uses Bayesian updating based on estimated rater accuracy to estimate a ground truth probability for each classification, and then estimates system accuracy by comparing the relative frequency that the system agrees with the most probable classification at different probability levels. A simulation analysis provides evidence that the method is robust and yields reasonable estimates of system accuracy under diverse and predictable conditions.

Subject Categories:

  • Information Science

Distribution Statement:

APPROVED FOR PUBLIC RELEASE