Using Hybrid Methods for Relevance Assessment in TREC Crowd'12
IOWA UNIV IOWA CITY
Pagination or Media Count:
The University of Iowa UIowaS submitted three runs to the TRAT subtask of the 2012 TREC Crowdsourcing track. The task objective was to evaluate approaches to crowdsourcing high quality relevance judgments for a text document collection. We used this as an opportunity to examine three hybrid combination of human-based and machine-based approaches while simultaneously limiting time and cost. We create a training set from topics, which were previously assessed for relevance on the same document set, and use this training set to build strategies. We apply machine approaches, including clustering, to order documents for each topic, and then ask crowdworkers to provide relevance judgments for a subset of documents. One of our runs provides the best logistic average misclassification LAM rates of all submitted TRAT runs.
- Information Science