TEASG Step 3 Report on APB Step 3 Test, Evaluation, and Analysis Process

A subcommittee of the Test, Evaluation, and Assessment Support Group was formed to address issues and procedures relating to the Advanced Processor Build Step 3 test and analysis procedures. This examination was prompted by the recent APB-99 Step 4 test, wherein a number of system deficiencies were noted with little or no perceived forewarning from the APB-99 Step 3 evaluation. This report discusses the subcommittees assessment of the overall Step 3 process and its implementation. The report provides conclusions regarding the implementation of the APB Step 3 mechanism and recommendations for the modification of Step 3 procedures so that more effective functional and integrated system testing can be attained. The failure to adequately address the Step 3 procedural issues raised here can increase performance risk associated with each new generation APB sonar system build. A majority of the system deficiencies highlighted during the APB-99 Step 4 sea test were either observed during prior lab tests or could not be tested during Step 3 lab tests due to a lack of testing capabilities. Nonetheless, the appearance of these problems was not sufficiently disseminated to the general community in a timely fashion. The most significant reason for this was that the primary focus of APB-99 testing was to enable a stable and fully functional system in preparation for the Step 4 sea test. This distracted the test team from assessing functional performance. Recommendations are presented to help correct some of the shortcomings of the testing implementation. These include minimizing system modifications and retuning during the Step 3 test increasing communication between the testing groups and other support groups and developers providing a more focused approach to Step 3 testing and extending the Step 3 test period to provide a find, fix, repair, and retest phase that examines the impact of post-test modifications.