Ideas

American Spy Agencies Are Struggling in the Age of Data

Massive amounts of data are revolutionizing the work of intelligence—and not necessarily in a good way.
Collage of images of scientist looking at model of spy plane old computer new satellite and soldier on smartphone
Photo-Illustration: Sam Whitney; Getty Images

This story is adapted from Spies, Lies, and Algorithms: The History and Future of American Intelligence, by Amy B. Zegart.

We’ve seen technological advances before. But never have we seen the convergence of so many new technologies changing so much so fast. This moment is challenging American intelligence agencies in three profound ways.

First, technological breakthroughs are transforming the threat landscape by generating new uncertainties and empowering new adversaries. During the Cold War, America had one principal enemy: the Soviet Union. The Cold War was a dangerous time, but it was simpler. America’s top intelligence priority was clear. Every foreign policy decision was viewed through the lens of “What would Moscow think?”

Now, a wide array of bad actors is leveraging technology to threaten across vast distances. China is launching massive cyberattacks to steal American intellectual property and building space weapons to cut off US military satellite communications before the fighting ever starts. Russia is using Facebook, Twitter, and other social media platforms to wage information warfare. Three dozen countries have autonomous combat drones and at least nine have already used them. Terrorist groups are using online video games to recruit followers and Google Earth to plan their attacks. Despots in developing nations are employing high tech repression tools. Weak states and non-state actors can inflict massive disruption, destruction, and deception with the click of a mouse.

For most of history, power and geography provided security. The strong threatened the weak, not the other way around. Oceans protected countries from one another, and distance mattered. Not anymore. In this era, the United States is simultaneously powerful and vulnerable to a head-spinning number of dangers, all moving at the speed of networks. It’s a far cry from the plodding pace of Soviet five-year plans from a few decades ago.

The second challenge of the digital age involves data. Intelligence is a sense-making enterprise. Agencies like the CIA gather and analyze information to help policymakers understand the present and anticipate the future. Intelligence isn’t always right. But it beats the best alternatives: guesswork, opinion, and gut feel.

In the old days, spy agencies in a handful of powerful countries dominated the collection and analysis of information. They were the only organizations with the resources and know-how to build billion-dollar satellites, make and break sophisticated codes, and collect information at scale. In 2001, the National Security Agency (NSA) intercepted about 200 million foreign emails, phone calls, and other signals a day. Few countries or companies could come close.

Now, data is democratizing, and American spy agencies are struggling to keep up. More than half the world is online, conducting 5 billion Google searches each day. Cell phone users are recording and posting events in real time—turning everyone into intelligence collectors, whether they know it or not. Anyone with an internet connection can access Google Earth satellite imagery, identify people using facial recognition software, and track events on Twitter.

On January 6, 2021, when pro-Trump rioters violently attacked the US Capitol to prevent congressional certification of the 2020 presidential election, causing the deaths of five people, online sleuths immediately started mining images and video posted on social media to help law enforcement agencies identify the perpetrators. One anonymous college student even created a website called Faces of the Riot. Using widely available facial detection software, the student scanned hundreds of videos and thousands of pictures shared by rioters and others on the social media site Parler and extracted images of those who may have been involved in the Capitol siege.

Most Popular

The sheer volume of online data today is so staggering, it’s hard to comprehend: In 2019, internet users posted 500 million tweets, sent 294 billion emails, and posted 350 million photos on Facebook every day. Some estimate that the amount of information on Earth is doubling every two years.

This kind of publicly available information is called open source intelligence, and it is becoming increasingly valuable. When US Navy SEALs conducted their secret nighttime raid on Osama bin Laden’s Pakistani compound, Pakistan’s military didn’t detect a thing. But a local information technology consultant named Sohaib Athar did. Hearing strange noises, he took to Twitter. “Helicopter hovering above Abbottabad at 1 a.m. (is a rare event),” he posted. Athar ended up live tweeting the operation, including reporting when an explosion shook his windows.

Similarly, when Russia invaded Ukraine in 2014, the best evidence did not come from spies or secretly intercepted communications. It came from selfies: time-stamped photos taken by Russian soldiers and posted on social media with Ukrainian highway signs in the background. Social media has become so important, even the consoles at America’s underground nuclear command center display Twitter feeds alongside classified information feeds.

That’s not all. Commercial firms worldwide are launching hundreds of small satellites every year, offering low-cost eyes in the sky to anyone who wants them. Some satellite sensors have resolutions so sharp they can detect manhole covers from space. Others can capture images at night, in cloudy weather, or through dense vegetation and camouflage. And constellations of cheap, small satellites are offering something new: faster revisit rates over the same location to detect changes over time. Already, commercial imagery and machine learning tools are enabling some of my Stanford colleagues to analyze North Korea’s trade relationship with China by counting the number of trucks crossing the border in hundreds of images over the past five years. Commercial imagery is becoming so valuable that the National Reconnaissance Office, the American agency that builds and operates spy satellites, is spending $300 million a year to buy it rather than just building satellites of its own.

In short, data volume and accessibility are revolutionizing sensemaking. The intelligence playing field is leveling—and not in a good way. Intelligence collectors are everywhere, and government spy agencies are drowning in data. This is a radical new world and intelligence agencies are struggling to adapt to it. While secrets once conferred a huge advantage, today open source information increasingly does. Intelligence used to be a race for insight where great powers were the only ones with the capabilities to access secrets. Now everyone is racing for insight and the internet gives them tools to do it. Secrets still matter, but whoever can harness all this data better and faster will win.

The third challenge posed by emerging technologies strikes at the heart of espionage: secrecy. Until now, American spy agencies didn’t have to interact much with outsiders, and they didn’t want to. The intelligence mission meant gathering secrets so we knew more about adversaries than they knew about us, and keeping how we gathered secrets a secret too.

Most Popular

Walk into CIA headquarters and you feel it. There’s a gleaming white marble Memorial Wall covered with more than 100 stars, each denoting an intelligence officer who died in the line of duty. A Book of Honor records their names, except for 40 entries that have only blank lines. For these CIA officers, service remains classified even in death.

Balancing secrecy and openness is an age-old struggle. Secrecy is vital for protecting intelligence sources and collection methods, as well as securing advantage. Openness is vital for ensuring democratic accountability. Too much secrecy invites abuse. Too much transparency makes intelligence ineffective.

In the digital age, however, secrecy is bringing greater risk because emerging technologies are blurring nearly all the old boundaries of geopolitics. Increasingly, national security requires intelligence agencies to engage the outside world, not stand apart from it.

It used to be that adversaries threatened from abroad and we could see them coming; military mobilization took time. Now they can attack privately owned critical infrastructure like power grids and financial systems in cyberspace—anytime, from anywhere, without crossing a border or firing a shot. In the 20th century, economics and security politics were separate spheres because the Soviet-bloc command economies were never part of the global trading order. In the 21st century, economics and security politics have become tightly intertwined because of global supply chains and dramatic advances in dual-use technologies like AI that offer game-changing commercial and military applications. Until now, intelligence agencies focused on understanding foreign governments and terrorist groups. Today they also have to understand American tech giants and startups—and how malign actors can use our own inventions against us.

Securing advantage in this new world means that intelligence agencies must find new ways to work with private sector companies to combat online threats and harness commercial technological advances. They must engage the universe of open source data to capture the power of its insights. And they must serve a broader array of intelligence customers outside of government to defend the nation.

These days, the National Security Agency isn’t the only big data behemoth. Amazon, Apple, Facebook, Google, and Microsoft are too. Although some companies have declared they will never use their technology for weapons, the reality is their technology already is a weapon: Hackers are attacking computer networks through Gmail phishing schemes and Microsoft coding vulnerabilities, terrorists are livestreaming attacks, and malign actors have turned social media platforms like Twitter and Facebook into disinformation superhighways that undermine democracy from within. American intelligence agencies have to find better ways to access relevant threat information held by these and other companies without jeopardizing civil liberties or firms’ commercial success.

Intelligence agencies need the private sector more for innovation now too. Analyzing massive troves of data, for example, will increasingly depend on AI tools. Technological advances (like the internet) used to start in government and then migrate to the commercial sector. Now that process is reversed, with breakthroughs coming from large companies like Google and Nvidia and from startups like Ginko Bioworks and Dataminr. Instead of developing technologies in-house, spy agencies now have to spot and adopt them rapidly from outside. That requires talent as well as technology, and the private sector is cornering the labor market too, offering compensation packages and cutting-edge computing facilities that are hard for government agencies (or universities) to match. Companies have been hiring away so many top AI professors (41 AI faculty left academia in 2018 alone), experts are worried there won’t be enough left to teach the next generation of students.

Most Popular

Engagement and collaboration with the private sector don’t come easily. Distrust of American spy agencies has a long history with some dark chapters. In the 1970s, revelations that intelligence agencies had been spying on Americans, infiltrating dissident groups, and assassinating foreign leaders prompted outcries and congressional oversight reforms. More recent controversies include CIA drone strikes and secret NSA surveillance programs revealed by a former agency contractor named Edward Snowden in 2013.

In the summer of 2014, a year after the Snowden revelations hit the press, I held a cyber boot camp for congressional staffers that included a visit to a major Silicon Valley tech company. As we filed into the conference room, the tension was palpable. One tech executive told the group he viewed the US government just like China’s People’s Liberation Army—as an adversary that needed to be stopped from surreptitiously penetrating his systems. Jaws dropped. An intelligence committee staffer rushed outside to call the boss and relay the news: They had a lot more repair work to do. NSA’s surveillance programs had been authorized, but in the eyes of tech executives, they had broken faith by secretly gathering customer data and making companies look weak, complicit, or both.

Intelligence agencies are still working hard to rebuild that trust.

Excerpted from Spies, Lies, and Algorithms: The History and Future of American Intelligence by Amy B. Zegart. Copyright © 2022 by Amy B. Zegart. Published by Princeton University Press and reprinted here by permission of the author.

More Great WIRED Stories