An Evaluation of Polyweighting in Domain-Referenced Testing
Final rept. Oct 88-Sep 91,
NAVY PERSONNEL RESEARCH AND DEVELOPMENT CENTER SAN DIEGO CA
Pagination or Media Count:
This technical note describes an empirical evaluation of a polychotomous item scoring procedure developed by the first author. This new scoring procedure polyweighting assigns an empirically-derived scoring weight to each possible response to a test question. An examinees polyscore is equal to the mean of the scoring weights of the response categories chosen by the examinee. In this research, polyweighting was applied to test data obtained from 1, 1 00 resident physicians who had completed a 200-item medical certification test. Using the 200 items as an item bank, the authors assembled 20 short 10-, 20-, 30-, 40-item assessment tests and used both proportion-correct scores and polyscores from these short tests to predict each physicians score on the 200- item certification test. For all 20 assessment tests, polyweighting resulted in higher cross-validated internal-consistency reliability coefficient-CL and domain validity. The observed increases in reliability corresponded to a mean increase in test length of 28. Over all 20 tests, the mean increase in domain validity was .075. The minimum increase in domain validity was .052.
- Humanities and History
- Military Forces and Organizations