Current and former US government officials have been pointing to the terror attacks in Paris as justification for mass surveillance programs. CIA Director John Brennan accused privacy advocates of “hand-wringing” that has made “our ability collectively internationally to find these terrorists much more challenging”. Former US National Security Agency and CIA director Michael Hayden said, “In the wake of Paris, a big stack of metadata doesn’t seem to be the scariest thing in the room.”

Ultimately, it’s impossible to know just how successful sweeping surveillance has been, since much of the work is secret. But what has been disclosed so far suggests the programs have been of limited value. Here’s a roundup of what we know.

An internal review of the Bush administration’s warrantless programme – called Stellarwind – found it resulted in few useful leads from 2001-2004, and none after that. New York Times reporter Charlie Savage obtained the findings through a Freedom of Information Act lawsuit and published them in his new book, Power Wars: Inside Obama’s Post–9/11 Presidency:
[The FBI general counsel] defined as useful those [leads] that made a substantive contribution to identifying a terrorist, or identifying a potential confidential informant. Just 1.2% of them fit that category. In 2006, she conducted a comprehensive study of all the leads generated from the content basket of Stellarwind between March 2004 and January 2006 and discovered that zero of those had been useful.

In an endnote, Savage added:
The programme was generating numerous tips to the FBI about suspicious phone numbers and e-mail addresses, and it was the job of the FBI field offices to pursue those leads and scrutinize the people behind them. (The tips were so frequent and such a waste of time that the field offices reported back, in frustration, “You’re sending us garbage.”)

In 2013, the US President’s Review Group on Intelligence and Communications Technologies analyzed terrorism cases from 2001 on, and determined that the NSA’s bulk collection of phone records “was not essential to preventing attacks”. According to the group’s report,
In at least 48 instances, traditional surveillance warrants obtained from the Foreign Intelligence Surveillance Court were used to obtain evidence through intercepts of phone calls and e-mails, said the researchers, whose results are in an online database.

More than half of the cases were initiated as a result of traditional investigative tools. The most common was a community or family tip to the authorities. Other methods included the use of informants, a suspicious-activity report filed by a business or community member to the FBI, or information turned up in investigations of non-terrorism cases.

Another 2014 report by the nonprofit New America Foundation echoed those conclusions. It described the government claims about the success of surveillance programs in the wake of the 9/11 attacks as “overblown and even misleading.”
An in-depth analysis of 225 individuals recruited by al-Qaeda or a like-minded group or inspired by al-Qaeda’s ideology, and charged in the United States with an act of terrorism since 9/11, demonstrates that traditional investigative methods, such as the use of informants, tips from local communities, and targeted intelligence operations, provided the initial impetus for investigations in the majority of cases, while the contribution of NSA’s bulk surveillance programs to these cases was minimal.

Edward Snowden’s leaks about the scope of the NSA’s surveillance system in the summer of 2013 put government officials on the defensive. Many US politicians and media outlets echoed the agency’s claim that it had successfully thwarted more than 50 terror attacks. ProPublica examined the claim and found “no evidence that the oft-cited figure is accurate”.
It’s impossible to assess the role NSA surveillance played in the 54 cases because, while the agency has provided a full list to Congress, it remains classified.

The NSA has publicly discussed four cases, and just one in which surveillance made a significant difference. That case involved a San Diego taxi driver named Basaaly Moalin, who sent $8,500 to the Somali terrorist group al-Shabab. But even the details of that case are murky. From the Washington Post:
In 2009, an FBI field intelligence group assessed that Moalin’s support for al-Shabab was not ideological. Rather, according to an FBI document provided to his defense team, Moalin probably sent money to an al-Shabab leader out of “tribal affiliation” and to “promote his own status” with tribal elders.

Also in the months after the Snowden revelations, the Justice Department said publicly that it had used warrantless wiretapping to gather evidence in a criminal case against another terrorist sympathizer, which fueled ongoing debates over the constitutionality of those methods. From the New York Times:
Prosecutors filed such a notice late Friday in the case of Jamshid Muhtorov, who was charged in Colorado in January 2012 with providing material support to the Islamic Jihad Union, a designated terrorist organization based in Uzbekistan.

Mr. Muhtorov is accused of planning to travel abroad to join the militants and has pleaded not guilty. A criminal complaint against him showed that much of the government’s case was based on intercepted e-mails and phone calls.

Local US police departments have also acknowledged the limitations of mass surveillance, as Boston Police Commissioner Ed Davis did after the Boston Marathon bombings in 2013. Federal authorities had received Russian intelligence reports about bomber Tamerlan Tsarnaev, but had not shared this information with authorities in Massachusetts or Boston. During a US House Homeland Security Committee hearing, Davis said,
“There’s no computer that’s going to spit out a terrorist’s name. It’s the community being involved in the conversation and being appropriately open to communicating with law enforcement when something awry is identified. That really needs to happen and should be our first step.”

This article was first published on ProPublica.