When is Analysis Sufficient? A Study of how Professional Intelligence Analysts Judge Rigor
OHIO STATE UNIV COLUMBUS INDUSTRIAL AND SYSTEMS ENGINEERING
Pagination or Media Count:
The proliferation of data accessibility has exacerbated the risk of shallowness in information analysis, making it increasingly difficult to tell when analysis is sufficient for making decisions or changing plans, even as it becomes increasingly easy to find seemingly relevant data. In addressing the risk of shallow analysis, the concept of rigor emerges as an approach for coping with this fundamental uncertainty-motivating the need to better define and understand analytical rigor. The concept of rigor is explored in this thesis through a study that asks how professional analysts decide when there is sufficient rigor in an analytic process. Nine professional intelligence analysts participated in a scenario walkthrough in which they critiqued the analysis processes of two junior analysts-one representing a high-rigor analysis process and the other a low-rigor process. In the study, participants assumed the role of analyst supervisor, deciding if these analyses were of sufficient rigor to send to a decision maker-a fundamental judgment task characterized as the Supervisors Dilemma. This study design validated and refined the Elicitation by Critiquing methodology, also developing the Liquified Natural Gas Scenario, based on security issues that challenge safety analyses, as a cognitive case for exploring themes in information analysis. This research identified three general findings on rigor in information analysis. First, it found that process insight influenced judgments of rigor. Second, it found that while similar cues were used in forming assessments of rigor, the way in which those cues were interpreted as indicating rigor tended to be more varied. Third, the results of the study suggest a revised definition of analytical rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation.
- Personnel Management and Labor Relations