This essay is adapted from Defense Budgeting for a Safer World: The Experts Speak, a new publication by the Hoover Institution Press.
To examine the challenges faced by America’s eighteen intelligence agencies requires posing a crucial question: how much does money matter? The short answer is that nobody really knows, but it’s probably less than we think. The US intelligence community failed to adapt to the rising terrorist threat in the 1990s, when intelligence budgets were cut dramatically after the Cold War, but the intelligence community is also struggling to adapt to the technological age today, when intelligence budgets have never been higher. When budget scarcity and abundance lead to the same suboptimal outcome, something more systematic is likely at work. Its name is organizational pathologies.
To be sure, higher spending certainly can help shift intelligence priorities and deliver new capabilities. And reduced spending can hurt. But I find that organizational features of intelligence agencies are often silent but deadly killers of innovation. Agency structures, cultures, and career incentives critically shape what is valued, what gets done, and how well. Unless these organizational features are aligned more rapidly with the threat landscape, intelligence agencies will struggle to deliver timely insights to policy makers no matter how much funding they have.
It’s worth underscoring that studying anything in intelligence is tricky business because the public record is so incomplete. It’s hard to identify the causal factors that lead to success or failure when failures are often public but successes are often secret. The impact of budgeting decisions is especially challenging. Intelligence spending is so highly classified that until 2007, with rare exceptions, even the topline total intelligence budget remained secret. As a result, for years, expert analysts have estimated intelligence spending over time based on breadcrumbs of data from declassified reports and remarks by government officials.
Since 2007, total intelligence spending in two major categories has been released annually. These are the National Intelligence Program (NIP), which covers programs, projects, and activities of the intelligence community, and the Military Intelligence Program (MIP), which covers intelligence activities of military departments and agencies in the Defense Department that support tactical US military operations. In FY2022, the NIP was $65.7 billion, and the MIP was $24.1 billion, for a total intelligence budget of $89.8 billion. Yet even this aggregate figure is incomplete. It excludes other specific intelligence gathering programs in cabinet departments and agencies (such as Homeland Security) as well as military programs that include intelligence but have a different primary purpose—such as the MQ-9 Reaper unmanned aerial strike platform.
More important, declassified intelligence budgets do not provide meaningful data to assess whether the eighteen agencies of the US intelligence community are deploying their resources against the right priorities, particularly as the threat landscape changes. How much does the US government spend by intelligence agency, activity, or capability? How much is spent on understanding and countering nation-state actors like China and Russia versus transnational terrorists, the proliferation of weapons of mass destruction, or cyber threats? Is the intelligence community dedicating sufficient resources to attracting and retaining the right STEM talent? We don’t know. There are, of course, good national security reasons for not making this kind of information publicly available. My point is that analyzing the efficiency or effectiveness of intelligence spending from the outside is an exercise in speculation. Humility is in order.
Here’s what we do know: In broad-brush terms, spending for US intelligence increased significantly during the Cold War, declined by approximately 20 percent during the 1990s, and skyrocketed after 9/11. A 1994 House report showing spending trends from 1965 to 1994 describes Cold War spending as experiencing “tremendous real growth” over thirty years.
The Soviet Union’s collapse in 1991 brought dramatic reductions to intelligence and defense budgets, which lawmakers dubbed a “peace dividend.” Some, including Senator Daniel Patrick Moynihan, argued that the CIA should be abolished, because it was no longer needed. According to former director of national intelligence James R. Clapper, in the 1990s, the IC experienced a 23 percent budget reduction, creating a “damaging downward spiral.” Director of Central Intelligence George Tenet told the 9/11 Commission that during the 1990s, the entire IC lost 25 percent of its workforce, the CIA suffered a 16 percent workforce decline, and the agency’s budget declined by 18 percent in real terms. “This loss of manpower was devastating,” noted Tenet, “particularly in our two most manpower intensive activities: all-source analysis and human source collection. By the mid-1990s, recruitment of new CIA analysts and case officers had come to a virtual halt. NSA [National Security Agency] was hiring no new technologists during the greatest information technology change in our lifetimes.” The real picture was even worse than these numbers suggest, because personnel reductions were made through voluntary attrition rather than targeted cuts to retain top talent, weed out poor performers, or ensure key skill sets and geographic and functional areas were well covered.
Declining budgets undoubtedly made it difficult for the intelligence community to adapt to the rising terrorist threat in the years before the September 11, 2001, terrorist attacks. Yet evidence suggests there’s much more to the story than shrinking resources. I find that the roots of the failure to prevent 9/11 lay in broader, deeper organizational weaknesses in US intelligence agencies that had surprisingly little to do with funding. Throughout the 1990s, even as America’s spy agencies warned of the growing terrorist danger, they remained stuck in their Cold War posture, operating with organizational structures, cultures, and career incentives that offered little chance of stopping Al-Qaeda from committing the worst terrorist attack in American history. My five-year examination of thousands of pages of declassified documents and interviews with seventy-five current and former intelligence and government officials found that the CIA and FBI had twenty-three opportunities to penetrate and possibly stop the 9/11 plot. Organizational weaknesses led to failure every time.
The 9/11 wake-up call
After September 11, for example, it became clear that the CIA had never been in the habit of watchlisting suspected Al-Qaeda terrorists before. For more than forty years, the agency and the rest of the intelligence community had operated with Cold War priorities, procedures, and thinking, all of which had little need to ensure dangerous foreign terrorists stayed out of the United States. Before 9/11, there was a watchlisting program in name but not in practice: there was no formal training, no clear process, and no priority placed on it.
On August 23, 2001, just nineteen days before the 9/11 attacks, the CIA finally told the FBI that Al-Qaeda operatives Khalid al-Mihdhar and Nawaf al-Hazmi were probably in the United States and needed to be found. The FBI responded by putting the search for these two suspected terrorists at the bottom of its priority list, designated “routine.” The “nationwide” hunt was the focus of just one of the bureau’s fifty-six US field offices. And it was assigned to a junior agent who had just finished his rookie year and had never led this kind of investigation before.
Organizational pathologies, not individual screwups, were to blame. The bureau dedicated just one office to what should have been a nationwide search because the FBI had always been a decentralized organization where each field office operated largely autonomously. Putting one office on each case made sense for catching criminals after the fact and tailoring priorities to local law enforcement needs. It was a poor organizational setup for collecting and coordinating intelligence about future national security threats to the nation as a whole. Promotion incentives reflected this culture. Handing the search to a junior agent wasn’t a mistake; it was how things were supposed to work. Convictions made careers, so finding two potential terrorists who hadn’t yet committed a crime and might never do anything illegal went to one of the office’s least experienced investigators, because it was one of the least desirable jobs.
The two Al-Qaeda operatives didn’t need secret identities or clever schemes. They just needed the CIA and the FBI to operate as usual.
Emerging technologies
Twenty years later, intelligence agencies face a much more favorable budgetary environment, and yet they are struggling again to adapt to a shifting geopolitical landscape—driven this time by emerging technologies that are disrupting every facet of the intelligence enterprise. Despite record spending over the past two decades, intelligence agencies are losing their relative advantage.
The US intelligence budget has increased dramatically since 9/11, jumping from an estimated $60.37 billion in fiscal year 2001 to $89.8 billion in fiscal year 2022 in constant 2022 dollars—an increase of 49 percent over twenty years. The Congressional Research Service estimates that intelligence spending quadrupled from 1980 to 2010 in real terms.
In policy areas, from health care to K–12 education to defense, increased government spending has not translated into better results. Similarly, bigger defense budgets have not translated into better military readiness. The United States is estimated to spend more than the following nine countries in terms of defense budgets combined, and yet China’s relative defense advantages are growing. In many areas, American taxpayers seem to be getting less bang for their buck.
Increased intelligence spending after 9/11 produced arguably better outcomes than in these other policy areas, enabling the US to prosecute the war on terror and defend the homeland to great effect. Changes included creating the National Counterterrorism Center and the Office of the Director of National Intelligence, an expanded drone program, and tighter integration between intelligence and military counterterrorism operations. As a result of these and other measures, the United States has not suffered another major catastrophic terrorist attack on American soil.
However, the dramatic infusion of counterterrorism funding also ended up hard-wiring the bureaucracy to fight the last war. Great-power competition, not transnational terrorism, now tops the threat list. And emerging technologies are transforming both the future and how intelligence agencies go about understanding it. This is a moment of reckoning for American spy agencies. And it reveals the paradox of plenty: surging budgets led to wide-scale changes, but by the time US intelligence agencies mastered the Al-Qaeda problem, Al-Qaeda wasn’t the problem anymore.
Rivers of data
Never before has the world stood at the cusp of so many technologies transforming so much so fast. Artificial intelligence (AI) is disrupting nearly every industry and changing how wars are fought—automating everything from logistics to cyber defenses to unmanned fighter jets that can overwhelm defenses with swarms and maneuver faster and better than human pilots. Russian president Vladimir Putin has declared that whoever leads AI development “will become the ruler of the world,” and China has made no secret of its plans to lead the world in AI by 2030.
That’s not all. Advances in quantum computing could eventually unlock the encryption protecting nearly all the world’s data. Synthetic biology enables scientists to engineer living organisms with the potential for revolutionary improvements in food production, medicine, data storage, and weapons of war.
Perhaps most important from an American national security perspective is that nearly all of today’s emerging technologies are invented outside the government, made available to the world, and have widespread applications for commerce and conflict. That’s new: nuclear technology was born secret and stayed that way, limiting the proliferation of the world’s most dangerous weapons. Today’s technologies are born open.
Companies and investors have to weigh their economic interests against the national interest in new ways. Meanwhile, US intelligence agencies are struggling to adopt critical new technologies from the outside and move at the speed of invention instead of the pace of bureaucracy. Increasingly, private-sector leaders have responsibilities they don’t want, and government leaders want capabilities they don’t have. Power isn’t just shifting abroad. Power is shifting at home.
All these forces unleashed by emerging technologies create a moment of reckoning for America’s intelligence agencies. Intelligence leaders know that their success in the twenty-first century hinges on adapting to a world of more threats, more speed, more data, more customers, and more competitors, and they have been working hard to get there. But if I’m right, Washington cannot address its present intelligence challenges by throwing more money at existing agencies. Instead, developing US intelligence capabilities for the tech age requires something new: a dedicated, open-source intelligence agency focused on combing through unclassified data and discerning what it all means. If history is any guide, the agencies, processes, and cultures that got us here will not get us there.
