Lucene for n-grams using the ClueWeb Collection

reportActive / Technical Report | Accession Number: ADA517732 | Open PDF

Abstract:

The ARSC team made modifications to the Apache Lucene engine to accommodate go words, taken from the Google Gigaword vocabulary of n-grams. Indexing the Category B subset of the ClueWeb collection was accomplished by a divide and conquer method, working across the separate ClueWeb subsets for 1, 2 and 3-grams.

Author(s):
Newby, Gregory B. ; Fallen, Chris ; Mccormick, Kylie
Author Organization(s):
ALASKA UNIV ANCHORAGE ARTIC REGION SUPERCOMPUTING CENTER
Descriptive Note:
Conference paper
Supplementary Note:
Presented at the Text REtrieval Conference (TREC 2009, 18th) held in Gaithersburg, Maryland on 17-20 November 2009. Published in the Proceedings of the Text REtrieval Conference (TREC 2009, 18th), 2009. The conference was co-sponsored by the National Institute of Standards and Technology (NIST), the Defense Advanced Research Projects Agency (DARPA), and the Advanced Research and Development Activity (ARDA).
Pagination:
0008

Security Markings

DOCUMENT & CONTEXTUAL SUMMARY

Distribution:
Approved For Public Release
Distribution Statement:
Approved For Public Release; Distribution Is Unlimited.

RECORD

Collection: TR
Identifying Numbers
Monitor Series:
DARPA
 Subject Terms
Joint Capability Areas:
JCA_8_Building Partnerships
Communities of Interest:
No COI(s) Identified
Descriptor(s):
*INDEX TERMS, *INFORMATION SCIENCES, *INFORMATION RETRIEVAL, *INTERNET, INTERNET BROWSERS, NATURAL LANGUAGE, INDEXES, SOCIAL COMMUNICATION, SYMPOSIA, REPRINTS, SUPERCOMPUTERS
Field(s)/Group(s):
Information Science, Computer Programming and Software, Test Facilities, Equipment and Methods, Linguistics
Keyword(s):
*SEARCH ENGINES(INTERNET), *N GRAMS, *LUCENE SEARCH ENGINE, *WEB(INTERNET), ARSC(ARTIC REGION SUPERCOMPUTING CENTER), APACHE LUCENE SEARCH ENGINE, CLUEWEB DATABASE, GIGAWORDS, GOOGLE, GO WORDS, GO LISTS
Report Date:
2009 Nov 01
Creation Date:
2010 Apr 28