Test Reliability and Experimental Power,
NAVAL AEROSPACE MEDICAL RESEARCH LAB PENSACOLA FL
Pagination or Media Count:
The reliability coefficient, pxx, has long been accepted as an index of the stability, repeatability, and precision of psychological tests. Because pxx measures the proportion of the variance in a set of scores attributable to variation among individuals, values of pxx are sometimes compared to justify using particular tests in studies of individual differences. Values of pxx are also sometimes compared to justify using particular tests in experimental research. The latter practice is usually justified by arguing that larger values of pxx imply greater measurement precision and, therefore, potentially greater sensitivity to experimental treatments. That argument is not generally correct because the individual variation measured by pxx is frequently confounded with measurement error in the denominators of significance tests. The effects of this confounding lead to paradoxical situations in which reliability, as measured by pxx, may be inversely related or unrelated to experimental precision, as measured-by the reciprocal of experimental error. Because the power of an experiment increases with precision, as just defined, conditions that invert or negate the relationship between pxx and precision also invert or negate the relationship between pxx and power. These considerations do not mean that the reliability coefficient is necessarily irrelevant to experimental research. Because experimental designs differ in the degree to which they are influenced by individual variation, a consideration of the value of pxx a specific test yields will sometimes provide information about the best design in which to use that test. Statistics, Psychometrics, Reliability, Power.