Modeling Temporal Crowd Work Quality with Limited Supervision
University of Texas at Austin Austin
Pagination or Media Count:
While recent work has shown that a workers performancecan be more accurately modeled by temporal correlation intask performance, a fundamental challenge remains in theneed for expert gold labels to evaluate a workers performance.To solve this problem, we explore two methods of utilizinglimited gold labels, initial training and periodic updating.Furthermore, we present a novel way of learning a predictionmodel in the absence of gold labels with uncertainty awarelearning and soft-label updating. Our experiment witha real crowdsourcing dataset demonstrates that periodic updatingtends to show better performance than initial trainingwhen the number of gold labels are very limited 25.