Performance Test Objectivity: A Comparison of Rater Accuracy and Reliability Using Three Observation Forms.

The study examined two variables that may influence the consistency and accuracy of raters judgments in evaluating job performance 1 the precision with which behaviors to be observed and evaluated are specified on a performance observation form and 2 the level of proficiency of the rater at the task being evaluated. Videotapes were prepared that depicted both passing and failing performances in the use of two types of electronic test equipments. The videotapes were observed in two experiments by raters who used a structured, semistructured, or unstructured performance observation form. Rater skill level was determined by the score the rater obtained on a performance test that consisted of the same types of electrical measurement problems as shown on the videotape. For both experiments, the presence of at least some structure in a performance observation form produced more accurate and reliable evaluations of job task performance than did a form with no structure. Within the range of rater skill tested, results showed that the level of skill proficiency that raters have with a particular type of electronic test equipment is largely independent of their ability to judge accurately and consistently the performance of others in using the same equipment. Author