For every weapons system being developed, the U.S. Department of Defense (DOD) must make a critical decision: Should the system go forward to full-scale production? The answer to that question may involve not only tens of billions of dollars but also the nation's security and military capabilities. In the milestone process used by DOD to answer the basic acquisition question, one component near the end of the process is operational testing, to determine if a system meets the requirements for effectiveness and suitability in realistic battlefield settings. Problems discovered at this stage can cause significant production delays and can necessitate costly system redesign. This book examines the milestone process, as well as the DOD's entire approach to testing and evaluating defense systems. It brings to the topic of defense acquisition the application of scientific statistical principles and practices.

The U.S. Army Test and Evaluation Command (ATEC) is responsible for the operational testing and evaluation of Army systems in development. ATEC requested that the National Research Council form the Panel on Operational Test Design and Evaluation of the Interim Armored Vehicle (Stryker) to explore three issues concerning the initial operation test plans for the Stryker/Interim Brigade Combat Team (IBCT). First, the panel was asked to examine the measures selected to assess the performance and effectiveness of the Stryker/IBCT in comparison both to requirements and to the baseline system. Second, the panel was asked to review the test design for the Stryker/IBCT initial operational test to see whether it is consistent with best practices. Third, the panel was asked to identify the advantages and disadvantages of techniques for combining operational test data with data from other sources and types of use. In this report the panel presents findings, conclusions, and recommendations pertaining to the first two issues: measures of performance and effectiveness, and test design. The panel intends to prepare a second report that discusses techniques for combining information.