Accession Number:

AD1000128

Title:

Modeling Temporal Crowd Work Quality with Limited Supervision

Descriptive Note:

Conference Paper

Corporate Author:

University of Texas at Austin Austin

Report Date:

2015-11-11

Pagination or Media Count:

10.0

Abstract:

While recent work has shown that a workers performancecan be more accurately modeled by temporal correlation intask performance, a fundamental challenge remains in theneed for expert gold labels to evaluate a workers performance.To solve this problem, we explore two methods of utilizinglimited gold labels, initial training and periodic updating.Furthermore, we present a novel way of learning a predictionmodel in the absence of gold labels with uncertainty awarelearning and soft-label updating. Our experiment witha real crowdsourcing dataset demonstrates that periodic updatingtends to show better performance than initial trainingwhen the number of gold labels are very limited 25.

Subject Categories:

Distribution Statement:

APPROVED FOR PUBLIC RELEASE