Navy Promotion Testing Research: Item Exposure Analyses

This report describes work conducted by PDRI to assist the Navy in studying item exposure and other exam development policies related to Navy-Wide Advancement Examinations (NWAE), one of several factors used in the Navy Enlisted Advancement System to determine advancement to ranks E-4 through E-7. We discuss current practices and recommendations regarding ways to ensure that item and exam development processes are fair, valid, and credible. The work consisted of three components: (1) A focused literature review on item exposure; (2) analyses of archival data to assess how item exposure across multiple administrations affected item performance; and (3) a description of best practices and recommendations on exam development policies concerning item exposure, random and randomized equivalent exams, parallel items, SJTs and other measures. This project underscores the need to carefully, consistently analyze item performance and ensure that trends in data do not change in unexplainable ways. If shifts are identified, tools are available to investigate what caused those changes, along with a variety of solutions. However, the application of appropriate tools is not always straightforward, and may have unanticipated secondary effects. Actions taken must be considered in the context of the overall program to ensure fair, valid and accurate testing.