The Coyle Report

March 2000

On February 14, the Defense Department's office of operational test and evaluation (DOT&E) released its annual report covering the operational and live-fire testing of 161 military systems, including the limited national missile defense (NMD) program. Philip Coyle, director of the office of operational test and evaluation, reported that "undue pressure has been placed on the [NMD] program to meet an artificial decision point in the development process." He observed that this approach "has historically resulted in a negative effect on virtually every troubled DoD development program." With President Clinton scheduled to make a decision in July on whether to deploy the missile defense, Coyle recommended a thorough analysis of the next intercept test—currently planned for mid-May—to inform the Pentagon's deployment readiness review (DRR) in June, which will, in turn, guide Clinton's decision. (See news story.)

Below are selected sections of Coyle's report. The full text can be found online at www.dote.osd.mil/reports/FY99/index.html. The NMD program is under the section "Other Defense Programs."

TEST & EVALUATION ASSESSMENT

Despite the revised program, the aggressive schedule established for the NMD Program presents a major challenge. The NMD program will have to compress the work of 10 to 12 years into 8 or less years. As a result, many of the design and T&E [test and evaluation] activities will be performed concurrently. Program delays also caused the conduct of IFT-3 [integrated flight test] to slip to October 1999. This represents almost a 20-month slip over the last two years and demonstrates an extremely high-risk schedule. Additionally, the failure of IFT-4 to achieve an intercept may result in a further setback to the NMD schedule. The revised program has alleviated some of the long-term risk by deferring and staging the decision process as described earlier. However, since the DRR date has not been deferred, undue pressure has been placed on the program to meet an artificial decision point in the development process. The DRR will be a "come as you are" type of review which will examine the maturity and potential of the program at that point. This is driving the program to be "schedule" rather than "event" driven. This pattern has historically resulted in a negative effect on virtually every troubled DoD [Department of Defense] development program. In spite of this intense pressure, the program manager is doing an excellent job in trying to efficiently and effectively manage the preparation for the DRR and ultimately the deployment.

The complex operating characteristics and environments of the NMD T&E Program make it necessary to plan and conduct IFTs that are restricted in scope. DRR information based on a few flight tests with immature elements will be limited. Although IFT-3 was an important test in ballistic missile defense and demonstrated a new technology, it had significant limitations to operational realism, as noted throughout this report.

Due to the restrictions on realistic operational flight testing, the T&E program will rely heavily on ground testing and the execution of simulations for assessing the maturity and performance of the NMD system concept. The LIDS [LSI (lead system integrator) integrated distributed simulation] model development is proceeding much slower than planned. It is extremely doubtful that the model will be completed in time to allow for a rigorous system analysis for the DRR, resulting in limited analysis. A "beta" version of the software is promised to be ready by the end of February 2000. Service Operational Test Agencies may have to rely on alternative low fidelity models to assess the potential system effectiveness.

The FY98 [fiscal year] DOT&E Annual Report identified a number of risks that could have significant impact on the NMD T&E program's ability to test, analyze, and evaluate system performance. The degree to which those risk areas have changed from the last reporting cycle are addressed below:

• Limited Pre-DRR system-level testing: Only three intercept flight tests are planned before the DRR. Furthermore, the IFT-5 configuration will differ from the Capability-1 system; it uses prototype and surrogate sensors and a surrogate GBI [ground-based interceptor] booster stack. Nothing in the program alleviates this system maturity or schedule issues. Since IFT-3 was not conducted until October 2, 1999 and IFT-4 failed to achieve intercept, the schedule risk is increasing. On a positive note, while stretching out the program does not increase or decrease the number of pre- or post-DRR flight tests, it does allow more opportunity for operational testing a more mature system prior to fielding.

• Limited engagement conditions: Flight test launches from California and interceptors from Kwajalein Missile Range [KMR], along with safety constraints, place significant limitations on achieving realistic geometry and closing velocities. This area is unchanged. The geometry of an intercept of a missile launched from Vandenberg AFB [air force base], CA, presents an easily detectable, large, then decreasing radar return signal to the surrogate early warning radar used to support the flight tests. The mid-range tracking coordinates of the target RV [reentry vehicle] are provided by a beacon transmitter on the RV or through a GPS [global positioning system] receiver on the RV relayed to the ground. Pre-launch Weapon Task Plans for the interceptor are created using these data sources. This approach is acceptable for early developmental testing, but it does not suitably stress the NMD system in a realistic enough manner to support acquisition decisions. Additionally, the intercept velocities that are safely permitted during testing are on the low end of what might occur in a real ICBM attack. This limits the operational realism and engagement conditions.

• GBI booster testing: The NMD T&E program makes use of a surrogate launch vehicle, the Payload Launch Vehicle [PLV], for all flight tests until IFT-7. The program restructure has not affected this limitation. The objective booster contract was awarded in July 1998, and first delivery will not occur until after the FY00 DRR. Lack of IFT data without the objective GBI capability (e.g., larger burnout velocity than the Payload Launch Vehicle) before the DRR will limit the GBI evaluation. Since the date of the DRR is not being changed, the evaluation will not have the benefit of data from intercept flight tests using the new booster. However, the risk of limited GBI booster testing has been mitigated somewhat by the scheduling of two Boost Vehicle Tests before the DRR. These tests will evaluate the performance of the booster with an emulated EKV [exoatmospheric kill vehicle] package added to the front end of the missile. However, IFT-7 is the first integrated system test against a target that makes use of the objective booster. The mitigating factor in this risk area is that the weapon decision will not be made until 2003.

• Limitations of ground testing: The ISTC [integrated system test capability] will be the major source of data generated from ground testing. This area has been improved somewhat through the incorporation of common scenarios from one IGT [integrated ground test] to the next. This will allow the tracking of progress in the ISTC development. However, test articles used to represent NMD elements in the testbed will still have minimal verification or validation in time for the DRR. Additionally, the validation process is not linked directly to flight test scenarios, since the IGTs use actual Element processors versus the surrogate Elements that currently support flight testing. The risk in this area should be reduced in the post-DRR timeframe, as the program embarks on an aggressive, comprehensive end-to-end hardware-in-the-loop [HWIL] effort. However, it is imperative that the hardware-in-the-loop program focus its initial efforts on the EKV.

• Target suite: The NMD T&E program is building a target suite that, while an adequate representation of one or two RVs, may not be representative of threat penetration aids, booster, or post-boost vehicles. Use of the large balloon in the target complex has some value, but continued use should be reevaluated for future flight tests. Test targets of the current program do not represent the complete "design-to" threat space and are not representative of the full sensor requirements spectrum (e.g., discrimination requirements). Much of this limitation, however, is attributable to the lack of information surrounding the real threat. As the knowledge of the threat evolves, the risk in this area should decrease slightly. However, specific details on threat characteristics are rarely readily accessible.

• Multiple target testing: NMD system performance against multiple targets is still not currently planned for demonstration in the flight test program. There are, however, plans to begin construction on two silos at KMR, which can be employed to do flight testing against multiple targets. The focus in this area is to use validated simulations to evaluate multiple simultaneous target engagement.

• BMC3 [battle management/command, control and communications] interoperability testing: The BMC3 to Commander-In-Chief interface inside Cheyenne Mountain will not be tested prior to the DRR. Little has changed in the pre-DRR timeframe under the current program. Build Increment-1, the first significant BMC3 release, will not be available until 2QFY00 [quarter fiscal year], providing very little time to be fully evaluated by the June 2000 DRR. The revised deployment schedule does reduce risk in this area, however, by providing more time for post-DRR BMC3/Cheyenne Mountain Operations Center integration and testing. Additionally, the decision on whether to initiate the integration has been deferred to the 2QFY01 Defense Acquisition Board.

• Spare test articles: The previous TEMP [test and evaluation master plan] identified a lack of spare test articles due to a resource allocation trade-off. Current program planning uses a rolling spare concept in which the test target for the immediate future test flight serves as the backup for the current flight test. This approach will mitigate the spare target problem; however, the spare test article issue also applies to the interceptor and EKV, where test failures have major schedule impacts.

• Limitations of ground lethality testing: Currently, there is no ground test facility capable of propelling EKVs or their full-scale replicas against targets at the closing velocities expected for NMD intercepts. These closing velocities will exceed 7 kilometers per second (KPS) and in some cases will even exceed 10 KPS. Existing full-scale sled track facilities have only approached 3 KPS. Additionally, propelling a non-aerodynamic structure, such as the EKV, down a sled track through an atmosphere at the operational velocities involves special considerations. Holloman High Speed Sled Track is working on measures to achieve much higher velocities approaching Mach 10 (approx. 3.5 KPS), still much lower than tactical intercepts. If this work is successful, the lethality test data to support DRR will still have to be collected from light-gas-gun tests of reduced-scale replicas of EKV surrogates and targets at the lower-end (six kilometers per second or less) of the intercept velocity spectrum, with hydrocode simulations for the higher velocities.

• Programmatic Issues: The LSI contractor has taken time to overcome the inertia of bringing the program up to full speed. The Government's System Evaluation Plan was supposed to be replaced by a LSI generated System Verification Plan (SVP). The LSI has now determined that the SVP is not sufficient to evaluate the program for the DRR, and is developing a System Analysis Plan that will provide the roadmap for DRR assessment. The High Fidelity System Simulation, which was to be the fast running, system performance, digital simulation for assessing many scenarios throughout the threat space, has been abandoned in favor of Boeing's LIDS model. It now appears that LIDS is at high risk of being delivered in time to allow for a robust system evaluation for the DRR or will have a reduced functionality and only allow for minimal evaluation.

• Logistics Support (New concern): Mathematical predictions for the Element reliability and availability goals that are needed to satisfy operational requirements are extraordinarily high. These requirements may be either unachievable or necessitate extensive spare parts supplies or intense maintenance efforts.

VALUE ADDED

DOT&E has been a significant contributor through the IPT [integrated planning team] process to formulate the NMD T&E program on practically a daily basis. We have been one of the principal stimuli to the JPO's [joint program office] plan to develop a comprehensive integrated HWIL effort. This will enable an effective and efficient ground testing capability, which will significantly reduce the risk of successful flight testing.

At DOT&E's recommendation, the JPO is proposing to alter the 2003 weapon decision to seek low rate rather than full-rate production authorization. It will permit dedicated LFT&E [live fire test and evaluation] flight tests to be performed with production representative EKVs and allow the IOT&E [initial operational test and evaluation] to be conducted prior to full-rate production. This will reduce the risk of prematurely committing to the production in large quantities of interceptors that may not have sufficient lethality to defeat threat RVs.

Many of DOT&E's concerns and recommendations have likewise been independently captured in the second Welch panel report.

RECOMMENDATIONS

The DRR is currently firmly scheduled in June 2000 rather than after completion of the analysis of IFT-5. This is a strongly "schedule driven" (vice "event driven") approach, thereby placing unrealistic pressure on the JPO. IFT-5 will be the first intercept attempt with all NMD elements integrated except the booster. DOT&E is recommending that preparations for the DRR allow time for a thorough analysis of the IFT-5 test data in order to inform the DRR decision, especially in light of the failure of IFT-4 to intercept the target. This would provide a clear technical understanding of the results and avoid forcing the DRR before the analysis is complete.

Several factors drive the need for an improved hardware-in-the-loop approach. They include the failed IFT-4 intercept, the role of the large balloon in supporting an intercept and speculation on the EKV's ability to discriminate countermeasures. DOT&E strongly recommends an intensive effort to develop a flexible, comprehensive hardware-in-the-loop facility that presents a high fidelity representation of the threat target for designing and testing of the EKV.

Current Schedule of Major Flight Test Milestones

Source: Director of Operational Test and Evaluation

*Editor's Note: Capability-1 calls for 20-100 interceptors and one X-band radar to defend against a "few, simple" warheads. Capability-2, incorporating additional X-band radars, would defend against a "few, more sophisticated" warheads. Capability-3 would add 25 interceptors to the first site and 125 interceptors at a second site to defend against "many, more sophisticated" warheads.