Experimental Validation of Probability Predictions,

A common type of simulation used in military operations research estimates the probability of a particular occurrence, such as detection of a submarine. Because of the constraints on operational exercises which examine these events, there is seldom sufficient replication for traditional validation. The purpose of this paper is to derive the methodology for using classical test statistics in the validation of these simulation models, using a small, varied data base. Two specific types of experiments are considered Bernoulli trials and trucated Bernoulli trials. For each type of experiment, tests are suggested which examine overall bias of predicted probabilities, the variance of the experimental sequence, correlation of the predicted probabilities with observed successes, and, where adequate data is available, agreement over the range of probability predictions. Author