Sample Size Requirements and Considerations for Models to Assess Human-Machine System Performance
Technical Report,02 Sep 2013,01 Aug 2019
AIR FORCE INSTITUTE OF TECHNOLOGY WRIGHT-PATTERSON AFB OH WRIGHT-PATTERSON AFB United States
Pagination or Media Count:
Hierarchical Linear Models HLMs, also known as multi-level models, are an extension of multiple regression analysis and can aid in the understanding of human and machine workloads of a system. These models allow for prediction and testing in systems with hierarchies of two or more levels. The complex interrelated variability of these multi-level models exists in operational settings, such as the Air Force Distributed Common Ground System Full Motion Video AF DCGS FMV community which is composed of individuals Level-1, groups Level-2, units Level-3, and organizations Level-4. Through the development of sample size requirements and considerations for multi-level models, this research determined necessary requirements and strategies to assess human-machine system performance based requests for statistical testing and evaluation. This research compares sample size recommendations to previous Level-2 HLM recommendations and extends recommendations to Level-3 and Level-4 HLMs based on varying effect sizes, level predictor variance, and error variance scenarios. Depending on the application, results demonstrate that sample size requirements may be smaller than what literature previously reported. Further, when sample size requirements cannot be met, this research develops and assesses re-sampling methods as a means to augment small samples for estimation. The operational community, DCGS FMV, has a small population of 1,000 and limited access to analysts. This research provides distributions based on Subject Matter Expert SME opinion for re-sampling methods to the DCGS FMV Level-3 and Level-4 HLMs. These findings provide a foundation on which sample size recommendations and extrapolation techniques for HLM are made. Such recommendations and techniques with SME input provide simulation based HLM assessments that can give initial recommendations with limited associated cost, analyst and researcher time, and resources.
- Human Factors Engineering and Man Machine Systems
- Statistics and Probability