Enhancing the Validity of Rating-Based Tests
Technical Report,01 Sep 2015,30 Nov 2018
ARMY RESEARCH INST FOR THE BEHAVIORAL AND SOCIAL SCIENCES FORT BELVOIR VA FORT BELVOIR
Pagination or Media Count:
Profile similarity metrics PSMs can be computed for rating-based judgment tests, personality scales, and biodata inventories to supplement conventional measures and enhance scale validity. These metrics quantify shape, the correlation between a respondents rating profile and the scoring key scatter, respondent tendency to use more or less of the available rating scale elevation, respondent tendency to systematically provide high or low ratings and delta, respondent tendency to provide high or low ratings relative to the key. Analyses conducted for three projects confirmed theoretical expectations that PSMs can be used to accurately model distance score variance and increment the validity of distance scores against performance outcomes. Project 1 utilized three judgment tests and demonstrated that shape and delta metrics predicted supervisor performance ratings R .33, while elevation and shape metrics predicted career intent R .25. Project 2 utilized conventional personality scales and showed that PSMs provided incremental validity beyond distance scores against performance outcomes and documented the stability of the validity gains using an independent cross sample. Project 3 evaluated the use of PSMs to score experimental 9-point personality in addition to conventional 5 point personality scales. Project 3 analyses demonstrated that PSMs provided incremental validity against performance outcomes beyond distance scoring for the combined personality battery R .54 vs. R .47. The third project also documented construct validity between overlapping constructs for the 5-point and 9-point scales. These results redefine validity expectations for personalityjudgment constructs and demonstrate the efficacy of PSMs procedures to broaden the scope of psychological domains for which accurate measurement is possible. The U.S. Army Research Institute for the Behavioral and Social Sciences ARI supported this research project.
- Personnel Management and Labor Relations