Million Query Track 2008 Overview
MASSACHUSETTS UNIV AMHERST DEPT OF COMPUTER SCIENCE
Pagination or Media Count:
The Million Query 1MQ track ran for the second time in TREC 2008. The track is designed to serve two purposes first, it is an exploration of ad-hoc retrieval over a large set of queries and a large collection of documents second, it investigates questions of system evaluation, in particular whether it is better to evaluate using many shallow judgments or fewer thorough judgments. As with the 2007 track ACA07, participants ran 10,000 queries against a collection of 25 million documents. The 2008 track differed in the following ways 1. Queries were assigned to one of four categories. 2. Each query was assigned a target of 8, 16, 32, 64, or 128 judgments. 3. Assessors could judge documents not relevant but reasonable. Section 1 describes how the corpus and queries were selected, the query classes, details of the submission formats, and a brief description of each submitted run. Section 2 provides an overview of the judging process, including a sketch of how it alternated between two methods for selecting the small set of documents to be judged. Sections 3.1 and 3.2 provide an overview of those two selection methods, developed at UMass and NEU, respectively. In Section 4 we present statistics collected during the judging process, including the total number of queries judged, how many judgments were served by each approach, and so on, along with the overall results of the track. We present additional results and analysis in Section 5.
- Information Science
- Computer Programming and Software
- Statistics and Probability