Generalizability Theory: An Assessment of Its Relevance to the Air Force Job Performance Measurement Project
Interim technical paper May 1986-Oct 1987
UNIVERSAL ENERGY SYSTEMS INC DAYTON OH
Pagination or Media Count:
Generalizability theory is a method for estimating the dependability of scores over various conditions of measurement. In contrast to classical test theory, which permits the investigation of only one error source at a time, generalizability theory is multifaceted and allows the researcher to propose and simultaneously investigate multiple sources of error. The applicability of generalizability theory to the Air Forces ongoing Job Performance Measurement JPM Project is reviewed in this paper. It was concluded that generalizability theory would be relevant to the JPM Project because it is a sophisticated and efficient method of investigating reliability, a fundamental property of any measurement instrument. Next, generalizability theory is illustrated by applying it to data collected from Air Force jet engine mechanics. The question of interest was whether performance scores were generalizable consistent over different rating sources incumbents, supervisors, or peers, rating forms one of four forms used in the project, or specific items included on any one form. The results indicated that scores were generalizable over both forms and items within forms. However, scores were not generalizable over rating sources. Sources tended to differentially rank ratees, depending on the specific form used. When scores were averaged over rating sources, the resulting mean scores were generalized if at least two rating forms were used. Implications of the results and the applicability of generalizability theory for the Job Performance Measurement Project are discussed.
- Statistics and Probability