Bad Idea: Relying on Bad Schedule Estimates to Promote Agile Programs
A group of reformers in both Congress and the executive branch have a common diagnosis of the Department of Defense’s (DoD) weapon system acquisition process: it is too slow and rigid. The DoD News coverage of Undersecretary for Research and Engineering Mike Griffin’s comments this past April succinctly summarize this concern:
“The Chinese love our acquisition system, they are the biggest fans of our acquisition system that there could possibly be; they certainly don’t want us to change it,” Michael D. Griffin said at the ninth Defense Programs conference, hosted by McAleese and Associates and Credit Suisse.
Griffin pointed out DoD takes about 16.5 years to go from statement of need to initial operating capability. The United States used to operate in the time frame that China now operates in — two to three years, he said.
As is covered in greater detail in CSIS’s 2018 Acquisition Trends report, those seeking greater speed in the acquisition process are concerned that U.S. military technology superiority is eroding. Proponents argue that this erosion can be traced both to diffusion of technology due to advances in a globally interconnected commercial sector and specific investments by competitor nations intended to counter U.S. strengths.
Parts of these arguments can be traced back to the Third Offset strategy, led by Obama-era Deputy Defense Secretary Bob Work, which identified a range of areas of human-computer interaction as areas to invest in at the margin in order to achieve the next generation of U.S. technology superiority. That focus broadened under President Donald Trump to include less software-oriented technologies such as hypersonics. Congress further expanded the demand for new technologies to include a reorganization of the parts of DoD responsible for acquisition, an increasing emphasis on prototypes, and greater support for new funding even though it required a series of deals to work past Budget Control Act budget caps.
Congress has taken explicit inspiration from the agile development approach employed by many commercial software producers. Rather than incrementally produce complex pieces of software over multiple years, this approach emphasizes creating the minimum viable product to get feedback from users and iterate designs over a series of sprints. Software is widely seen as an area in which the rate of development in both the commercial and open source spheres significantly outpaces that occurring within the government sector. Commercial smartphones may leverage government pioneered Global Positioning System (GPS) technology, but they rapidly outpaced DoD efforts on software-driven radios during the same era. On the civilian side, the initial failure of Healthcare.gov showed limitations in government software acquisition in areas where cutting edge military technology was not necessary. The 2018 National Defense Authorization Act, among other provisions on software, included pilot programs in the use of agile development approaches and commissioned a year-long study of software methods from the Defense Innovation Board.
Can this drive for agility and speed succeed? To answer this question, it helps to look at the prior generation of acquisition reform efforts exemplified by the Weapon System Acquisition Reform Act (WSARA) from Congress and the earlier rounds of DoD’s Better Buying Power (BBP) initiatives. Then, as now, reform efforts involved a range of objectives that were sometimes in conflict with one another, but the headline concern was generating cost savings and reducing overruns in major programs. The ongoing nature of defense reform efforts lead some to believe that it is a Sisyphean task, doomed to fail. However, the last round of reform did yield early positive results. The Government Accountability Office (GAO) reported on growing reductions in cost overruns in the major defense acquisition program portfolio, and CSIS analysis of overall contracting trends found similar results. Acquisition reform is a regularly re-occurring process not because it always fails but because priorities of those in power shift due to internal factors, such as the state and age of the U.S. inventory, and to external factors, such as global strategic and economic developments.
The comparison to prior acquisition reform offers hope but also highlights a significant risk in efforts to increase speed: schedule estimates for major weapon systems are quite bad. When the aforementioned GAO report found that the time to deliver a capability increased by only a month in 2017, that represented an improvement over the prior two years. For the 61 programs they analyzed initiated prior to 2010, the average delay was 35 months. Why are the estimates so bad? An analysis by David Tate at the Institute for Defense Analysis sheds light on this and reveals a few important surprises. First, he finds that the time between program initiation, somewhere between milestone A and B, to initial operating capacity has increased in recent decades. Second, he finds that delays have increased for some commodities and that programs that were estimated to deliver faster than relevant historical averages experienced greater delays.
A key aspect of the WSARA and BBP reforms were improvements in the quality of cost estimates. Better cost estimates do not inherently reduce costs, but they are a way to enable better informed trade-offs within and across different programs. The present set of reforms are not taking similar steps, due in no small part to the fact that these reforms tend to reduce process rather than fortify it. In this environment, relying on scheduling estimates, rather than other metrics, to encourage program agility is a bad idea and one likely to promote perverse outcomes.
At present, the Section 804 Middle Tier/Rapid Prototyping Authority has largely dodged this problem because most programs that have announced their use of the authority are sticking to the “rapid prototyping pathway” and avoiding the more ambitious “rapid fielding pathway” that mandates complete fielding within five years. For those seeking to expand the use of such rapid fielding authorities, focusing on program characteristics rather than fine-tuning schedule-based criteria may provide a more stable foundation. Such a change could benefit both those seeking to avoid the abuse of new authorities and those hesitant to use the new authorities due to concerns that even programs that rapidly reach initial operating capability may not qualify unless they are completely deployed within five years.
David Tate provides a list of program characteristics and examples that have been proven to actually accelerate cycle time. Most straightforward are those that do not seek to leap ahead in technology, such as non-development items using existing commercial or overseas systems as well as certain forms of upgrade and integration involving mature technologies. Fast leap ahead systems could include new systems with agile methods; new systems with limited requirements; new systems whose critical technology have been demonstrated; and new modular systems operating within existing bounds. These criteria still require judgment and are not appropriate for all systems, which is part of the point. Not all leap ahead technologies can be implemented swiftly and agilely. Finally, while the specific steps to take remain quite controversial, Congress has its eye on the ball when focusing on software questions. Software and software-intensive hardware systems are where the acquisitions of today are most different from that of prior decades and agile processes are most proven.
(U.S. Air National Guard photo by Master Sgt. Matt Hecht)