University of Massachusetts: MUC-3 Test Results and Analysis
MASSACHUSETTS UNIV AMHERST DEPT OF COMPUTER AND INFORMATION SCIENCE
Pagination or Media Count:
We believe that the score reports we obtained for TST2 provide an accurate assessment of our systems capabilities insofar as they are consistent with the results of our own internal tests conducted near the end of phase 2. The required TSTh score reports indicate that our system achieved the highest combined scores for recall 51 and precision 62 as well as the highest recall score of all the MUC-3 systems under the official MATCHEDMISSING scoring profile. We ran one optional test in addition to the required test for TST2. The optional run differs from the required run in only one respect, an alteration to our consolidation module. The consolidation module contains all procedures that translate parser output into target template instantiations. The complete consolidation module includes a case-based reasoning CBR component that makes predictions about the target output based on a portion of the development corpus. For our optional run, we executed a modified version of consolidation that does not include this CBR component. We predicted that the absence of the CBR component would pull recall down but push precision up looking at MATCHEDMISSING only. This trade off prediction was confirmed by the required and optional TST2 score reports. Please consult Appendix F for our required and optional test score summaries.