NSA XKeyscore — Search Engine for Global Internet

Overview

In the summer of 2013, when Edward Snowden began releasing classified NSA documents to journalists, the revelations came in waves. First there was PRISM, the program that collected data from major internet companies. Then there were the phone metadata programs. Then the encryption-busting programs. Each disclosure was shocking in its own right. But it was XKeyscore — revealed by Glenn Greenwald in the Guardian on July 31, 2013 — that may have been the most viscerally unsettling, because it answered a question that most people had never quite dared to ask: can the government actually read my emails?

The answer, it turned out, was yes. Not in some theoretical, we-could-if-we-really-wanted-to sense. Yes as in: an NSA analyst could sit down at a terminal, type in your email address, and read your messages. They could search your browsing history. They could read your Facebook chats. They could pull up your Google searches. And according to Snowden and the leaked training materials, they could do all of this without getting a warrant, without getting a court order, and in many cases without getting any prior authorization at all beyond filling out an on-screen form.

XKeyscore was not a surveillance program in the traditional sense — it was more like a search engine. The surveillance had already happened. The data had already been collected through the NSA’s global network of fiber-optic taps, partnerships with telecommunications companies, and arrangements with allied intelligence services. XKeyscore was the tool that made all of that collected data searchable, queryable, and immediately accessible to thousands of NSA analysts worldwide.

This theory is classified as confirmed — XKeyscore’s existence, capabilities, and operational scope were documented in classified NSA training materials published by multiple news organizations and have not been disputed by the U.S. government, which has instead argued about the legal authorities and oversight mechanisms governing its use.

Origins & History

The Evolution of NSA Signals Intelligence

To understand XKeyscore’s significance, you need to understand the NSA’s transformation in the early twenty-first century. During the Cold War, signals intelligence (SIGINT) was primarily about intercepting specific, targeted communications — tapping a particular phone line, monitoring a specific radio frequency, decrypting a particular diplomatic cable. The analyst knew who they were looking for and directed collection resources at that target.

The rise of the internet changed everything. Communications that had once traveled through discrete, targetable channels now flowed through a vast, interconnected web of fiber-optic cables, internet exchange points, and wireless networks. The sheer volume of data was staggering — and growing exponentially. The traditional SIGINT model of targeting specific communications became increasingly inadequate in a world where the target might use any number of email addresses, messaging platforms, and devices.

The NSA’s response, developed through programs beginning in the early 2000s, was to shift from targeted collection to bulk collection: intercept as much data as possible, store it, and then search it later. XKeyscore was the tool that made the “search it later” part work.

Development and Deployment (2007-2008)

XKeyscore was developed as part of the NSA’s broader effort to create a unified system for processing and searching the enormous volumes of internet data flowing through its collection infrastructure. According to the leaked training materials, the system became operational around 2007-2008.

The system was designed with a distributed architecture: rather than storing all collected data in a single central database, XKeyscore deployed servers at approximately 150 sites around the world, including NSA facilities, Five Eyes partner sites (in the UK, Canada, Australia, and New Zealand), and other locations. Each site processed and stored data from its local collection sources, but an analyst at any site could query the entire network.

The training materials described XKeyscore as operating in three tiers of data:

Full content: The actual text of emails, web pages, chat sessions, and other communications. Due to the enormous volume, full content was typically stored for only three to five days before being overwritten.

Metadata: Information about communications — who communicated with whom, when, from where, using what device and software — without the actual content. Metadata was stored for 30 days or longer.

Session data: Records of individual internet sessions — websites visited, searches conducted, files downloaded — associated with specific user identifiers. This data was also stored for varying periods.

How the System Worked

The XKeyscore training slides, published by the Guardian and subsequently by other news organizations including Der Spiegel and the Sydney Morning Herald, depicted a remarkably user-friendly interface for what was essentially the world’s most invasive search engine.

An analyst could search by any number of selectors: an email address, a phone number, a name, an IP address, a language, a geographic location, or even specific keywords or phrases used in communications. The system would return matching results from across its global network of servers.

One training slide showed an example search for all encrypted internet sessions (those using HTTPS/SSL) from a particular country — a search that would identify users attempting to protect their privacy, effectively targeting people for the security precautions they took.

Another slide described how XKeyscore could track a user’s browsing activity in near-real-time, showing every website they visited and every search they conducted. A third showed how the system could pull up the content of specific emails, including attachments.

The Authorization Question

The most explosive aspect of the XKeyscore revelations was the question of authorization. According to the leaked training materials and Snowden’s testimony, an NSA analyst could conduct many XKeyscore searches without prior judicial or supervisory authorization. The analyst was required to fill out an on-screen form justifying the search, but the justification was reviewed after the fact, not before it. This was “analyst self-certification” — the analyst authorized their own search.

Edward Snowden described the implications in stark terms during his initial interview with Glenn Greenwald: “I, sitting at my desk, certainly had the authorities to wiretap anyone, from you or your accountant to a federal judge to even the president, if I had a personal email.”

The NSA pushed back on this characterization, saying that XKeyscore access was subject to multiple layers of oversight, that analysts were required to have valid foreign intelligence justifications for their searches, and that searches involving U.S. persons were subject to additional restrictions. But the training materials themselves suggested that the practical barriers to misuse were low — an analyst needed only to assert a foreign intelligence purpose, and the assertion was reviewed only after the search had already been conducted.

Five Eyes and International Sharing

XKeyscore was not used by the NSA alone. Leaked documents revealed that the system was shared with the NSA’s Five Eyes partners: GCHQ (United Kingdom), GCSB (New Zealand), ASD (Australia), and CSE (Canada). Documents published by Der Spiegel showed that Germany’s BND intelligence service also had access to XKeyscore under an agreement with the NSA.

This international dimension raised additional concerns. Citizens of Five Eyes countries whose own domestic laws prohibited mass surveillance by their own governments could still have their communications collected and searched by a partner agency using XKeyscore. The arrangement created what critics described as an intelligence-sharing loophole: governments that could not legally spy on their own citizens could outsource the spying to allies and then access the results.

The Scale of Collection

The training materials included statistics that, even by the NSA’s own standards, were breathtaking. According to one slide, XKeyscore collected approximately 20 terabytes of data per day at one site alone. Across the global network, the collection volume was far larger. The system was described as having contributed to the capture of over 300 terrorists — a statistic the NSA used to justify the program’s scope.

The sheer volume of data created its own limitations. Because XKeyscore could store full content for only a few days, an analyst had to search within that window or rely on metadata to reconstruct communications. This led to discussions within the NSA about expanding storage capacity — documents referenced the agency’s efforts to build massive new data centers, including the Utah Data Center in Bluffdale, which became operational in 2014 with an estimated storage capacity of several exabytes.

Key Claims

• XKeyscore allows the NSA to search the internet activity of nearly any person worldwide: An analyst can query emails, chats, browsing history, and social media using simple selectors. Confirmed by leaked training materials.

• Searches can be conducted without prior authorization: An analyst can conduct many XKeyscore searches by self-certifying a foreign intelligence purpose, without obtaining a warrant or prior supervisory approval. Confirmed by training materials; disputed by NSA as to the extent of oversight.

• The system is shared with Five Eyes partners: Intelligence agencies in the UK, Canada, Australia, New Zealand, and Germany have access to XKeyscore. Confirmed by leaked documents.

• XKeyscore enables mass surveillance, not targeted surveillance: The system searches databases created by bulk collection, making it a tool of mass surveillance regardless of whether individual searches are targeted. Confirmed by the nature of the system’s architecture; debated as a matter of policy framing.

• There is insufficient oversight to prevent abuse: The combination of vast capabilities, minimal authorization requirements, and after-the-fact review creates conditions ripe for abuse. Debated; no large-scale pattern of individual analyst abuse has been documented, but the structural vulnerability is acknowledged.

Evidence

The evidence for XKeyscore’s existence and capabilities comes almost entirely from the Snowden documents:

• Classified NSA training slides describing XKeyscore’s capabilities, interface, and data sources, published by the Guardian, Der Spiegel, Sydney Morning Herald, and other outlets
• Edward Snowden’s testimony to journalists, the European Parliament, and in various interviews
• Supporting documents describing the collection programs (PRISM, UPSTREAM, etc.) that feed data into XKeyscore
• Documents describing Five Eyes partner access to XKeyscore
• The NSA’s own responses, which disputed certain characterizations but did not deny the system’s existence

The U.S. government has not publicly released a detailed description of XKeyscore, nor has it confirmed or denied specific capabilities described in the leaked documents. However, the government has not claimed the documents are fabricated, and its public statements have focused on defending the legal authorities and oversight mechanisms rather than disputing the system’s technical capabilities.

Cultural Impact

XKeyscore’s revelation was perhaps the single most impactful of the Snowden disclosures in terms of public perception. While programs like PRISM and the phone metadata collection were more legally and politically complex, XKeyscore was viscerally comprehensible: the government can read your emails. That simplicity gave it enormous rhetorical power.

The disclosure contributed to a global reassessment of digital privacy. In the European Union, the revelations accelerated the development of the General Data Protection Regulation (GDPR), adopted in 2016, which established strict rules about data collection and processing. In the United States, the revelations led to the USA FREEDOM Act of 2015, which reformed some aspects of NSA surveillance (primarily the phone metadata program) but left Section 702 — the legal authority underlying much of the collection that feeds XKeyscore — largely intact.

The XKeyscore revelations also influenced the technology industry’s approach to encryption and data security. The knowledge that the NSA could search through internet traffic at will accelerated the adoption of end-to-end encryption by messaging platforms, the implementation of HTTPS as a default for websites, and the development of privacy-focused technologies.

For the broader public, XKeyscore became a symbol of the surveillance state — the concrete embodiment of the abstract fear that the government is watching. “They have a search engine for your life” became a shorthand for the scope of NSA surveillance, and the program’s name entered the lexicon of digital rights activism.

In Popular Culture

XKeyscore and the broader Snowden revelations have been extensively covered in documentaries, films, and literature. Laura Poitras’s Oscar-winning 2014 documentary Citizenfour documented Snowden’s disclosure in real time and included discussion of XKeyscore and other programs. Oliver Stone’s 2016 film Snowden, starring Joseph Gordon-Levitt, dramatized Snowden’s career at the NSA and included depictions of surveillance capabilities based on XKeyscore. Glenn Greenwald’s 2014 book No Place to Hide provided detailed analysis of XKeyscore and other programs based on the Snowden documents. The system has been referenced in television series including John Oliver’s Last Week Tonight (in a memorable interview with Snowden himself), Mr. Robot, and Homeland.

Timeline

Sources & Further Reading

• Greenwald, Glenn. “XKeyscore: NSA Tool Collects ‘Nearly Everything a User Does on the Internet.’” The Guardian, July 31, 2013.
• Greenwald, Glenn. No Place to Hide: Edward Snowden, the NSA, and the U.S. Surveillance State. Metropolitan Books, 2014.
• Poitras, Laura, director. Citizenfour. Praxis Films, 2014.
• Harding, Luke. The Snowden Files: The Inside Story of the World’s Most Wanted Man. Guardian Faber, 2014.
• Bamford, James. “The NSA Is Building the Country’s Biggest Spy Center.” Wired, March 15, 2012.
• Der Spiegel staff. “NSA’s XKeyscore Gives Broad Surveillance Powers.” Der Spiegel, July 2013.
• Snowden, Edward. Permanent Record. Metropolitan Books, 2019.
• Stone, Oliver, director. Snowden. Open Road Films, 2016.
• Savage, Charlie. Power Wars: Inside Obama’s Post-9/11 Presidency. Little, Brown, 2015.

Related Theories

• NSA Warrantless Wiretapping — The pre-Snowden domestic surveillance program that laid the groundwork
• NSA Encryption Backdoors — The NSA’s program to weaken the encryption that might have protected communications from XKeyscore
• Five Eyes Surveillance — The international intelligence-sharing alliance that shared access to XKeyscore