Exploring Interactive Relevance Feedback With a Two-Pass Study Design
MARYLAND UNIV COLLEGE PARK INST FOR ADVANCED COMPUTER STUDIES
Pagination or Media Count:
Interactive query refinement is widely believed to improve the effectiveness of ranked retrieval but it can be difficult to leverage existing batch evaluation frameworks to quantify the relative benefits of alternative interaction designs. This paper uses the new two-pass interaction design of the Text Retrieval Conferences High Accuracy Retrieval from Documents HARD track to explore the design space for cluster-based interactive relevance feedback. Two sites contributed two techniques for cluster formation and three techniques for cluster labeling. The effectiveness of each technique was compared with lower bounds based on blind relevance feedback, and with upper bounds found with oracle-based techniques. The clustering techniques were found to yield potential benefits, but the automatically constructed cluster labels were found not to support sufficiently accurate cluster selection. Elicitation of a desired cluster descriptor was found to significantly improve the effectiveness of a subsequent retrieval pass. These results indicate that the affordable two-pass study design used in the HARD track can yield useful insights to guide future design decisions.
- Information Science