NSA files – live coverage of all developments and reaction

This live blog will continue here.

That's it from us for this week. We'll be back on Monday on a new blog.

Britain's most senior prosecutor has launched a robust defence of journalists who break the law pursuing investigations that have a genuine public interest, write Zoe Williams and Nick Hopkins.

Legal guidelines had been drafted, he said, to protect reporters.

Keir Starmer, the director of public prosecutions (DPP), insisted it "would be very unhealthy if you had a situation where a journalist felt that they needed to go to their lawyer before they pursued any lead or asked any question".

In an interview with the Guardian, Starmer added: "We've got to recognise that in the course of journalism, journalists will rub up against the criminal law and that is why, in our guidelines, we took the approach that we would assess where there was evidence of a criminal offence, whether the public interest in what the journalist was trying to achieve outweighed the overall criminality" ...

Without mentioning Snowden, Starmer made clear he drafted guidelines specifically to allow journalists to pursue difficult stories without fear of prosecution, so long as a public interest threshold had been reached. He acknowledged there were "potentially many offences that journalists could commit in the course of their business" but said guidelines were there to offer them protection.

Starmer also joined lawyers, academics and senior members of the intelligence community who have said the laws governing Britain's intelligence agencies need to reviewed in the light of Snowden's revelations about mass surveillance programmes run by GCHQ and its US counterpart, the National Security Agency.

The Intelligence and Security Act was passed in 1994, and the Regulation of Investigatory Powers Act in 2000 – long before some recent technological capabilities became available. Starmer said:

I think there's a growing recognition that the legislation in place needs to be looked at again to see whether it works well in the current environment … some of the old laws should be looked at again. I think most people accept that it is necessary to have some surveillance in a democratic society. I think most people accept that it's important to have limits and clear safeguards on that.

On Germany’s Jung & Naiv YouTube show, investigative journalist Jeremy Scahill has spoken about his collaboration with Glenn Greenwald and filmmaker Laura Poitras on a new news website funded by eBay founder Pierre Omidyar.

Scahill described this as a “serious investigative journalism site”. He said:

All of us work with different media outlets, but we were trying to think of a place where we could do more quick response journalism, to put things out in more detail, to try to democratise the process by which we reveal some of the revelations from the NSA and from other reporting that we’re doing.

He said of Omidyar:

He’s not hiring us. We’re not employees. This is something we’re doing as a collaborative project, and we’re going to be bringing on board some really exciting young journalists and also some veteran journalists to work with us.

He said they were going to produce some “really hard-hitting journalism” together.

Asked what he wanted to change about journalism today, Scahill said:

One of the reasons why we’re really excited about working with Pierre is that he said that he didn’t want to have a top-down model of editorial process, where editors are telling reporters what to do.

So we’re going to develop more of a horizontal model, where editors are supporting the work of the journalists, but that it’s going to be a journalism-driven website. We’re not just trying to fill positions with people, we’re going to bring people on board based on a proven track record of great journalism, and trying to create a space for them where they can do that journalism without being hindered by bureaucratic institutions and processes.

In this no-boss model, what happens if one of you “fucks up”, interviewer Tilo Jung asked. Scahill joked:

We have a policy where we’re going to elect … Glenn is going to be the interior minister, Glenn Greenwald, and he’s going to have a secret intelligence service that’s going to be investigating me and Laura. No. No, no, we’ll work it out.

“We ultimately want this to be a public service,” he said, and “we’re going to be looking for partnerships around the world, and journalists who do investigative reporting”.

He praised WikiLeaks as “incredible” and “extraordinary”.

A big part of journalism going forward is going to be finding sources of independent information, including from whistleblowers and people who want to leak documents that they think are in the public interest … and we’re going to continue to do that.

Asked about his own experience of having information about his own project leaked, he said:

The main thing was that we wanted to be able to present to people the whole vision for what we were trying to do. So we were in the process of discussing that, and the language of it, and charting out a vision, and then it gets picked up in the press, and then you’re forced to kind of fast forward, and say ‘OK, we’ll put this statement out, and give people a sense of what we’re doing’.

In the Nation, Jesselyn Radack, formerly of the US justice department and now with an organisation that protects whistleblowers, writes about being one of the four American advocates for whistleblowers who presented Edward Snowden with an award recently. She says that the four of them left their cellphones and laptops in the US because they knew Washington “could geo-locate our whereabouts and find Snowden”, although she says she does not know where he is.

She writes of the NSA leaker:

Given the extraordinary circumstances and pressure he’s under, Snowden is doing remarkably well. He’s warm and engaged, greeting us with long embraces. His is well-grounded, centered, and has a quick sense of humor, darkly joking that if he were a spy, Russia treats its spies much better than leaving them trapped in the Sheremetyevo transit zone for over a month. He is brilliant, humble and idealistic—in the best sense of the word.

And she adds:

I can say with certainty: Edward Snowden is not being controlled by the Russians, or anyone for that matter. He is fiercely independent and makes his own decisions, leaving him perplexed and understandably frustrated by the continuous insinuations that he is giving the Russians information. He ticks off abundant evidence to the contrary. First, he points out, he didn’t destroy his life to become a Russian asset. Second, he’s in Russia only because of the United States, which revoked his passport while he was en route to Latin America. Third, WikiLeaks journalist Sarah Harrison has been by his side the whole time, in part to bear witness to the fact that he is not engaged in spying activity. Fourth, it is obvious that he chose to give information about NSA’s secret dragnet surveillance to the US people, not foreign adversaries. Fifth, and perhaps most significantly considering the contrary narrative promulgated in the United States, he has not had access to the information he revealed since he left Hong Kong.

Britain needs to have a full public debate about the scale of internet surveillance to give confidence that state powers are not being abused, the world's five biggest internet companies have told MPs. Rowena Mason has the full story here.

In a joint memo, Facebook, Microsoft, Google, Yahoo! and Twitter have called for the UK government to allow greater transparency about requests for them to hand over data on their users ...

"We recommend that requests for user data made by the UK government are made as transparent as possible," they say. "Each of our companies already publishes a transparency report and, as public concern grows around the world about the scale of digital surveillance, we believe that greater transparency is important in encouraging a full public debate and maintaining confidence that powers are not being abused." ...

Their comments echo calls for more transparency about state data requests in the US, where there has been a storm of political debate about the revelations and a series of follow-up investigations that have forced the Obama administration to consider reforms. Internet companies have been engaged in that US debate.

Wired reports that the American Civil Liberties Union has filed a lawsuit against the US justice department, calling on it to reveal which defendants have been the target of the warrantless phone or email surveillance that was revealed in 2005.

Wired reports:

The suit (.pdf) said that the Justice Department’s failure to “release responsive records is of particular concern because the request relates to a highly controversial surveillance authority whose wisdom, effectiveness, and legality is a matter of intense and ongoing public debate.”

The American pressure group the Committee to Protect Journalists has issued a statement of concern regarding David Cameron's call for a select committee to investigate the Guardian (a call answered by the home affairs committee).

Nina Ognianova of the CPJ said:

Prime minister Cameron suggested today [Wednesday] that the Guardian has damaged national security or broken the law, but he provided no evidence whatsoever to support these claims. Today's statements and the suggestion that the Guardian be criminally investigated are deeply chilling.

A Guardian leader column today points out the differences in the debate over the NSA files in the UK and the US.

There are practically no voices in Congress calling for the Washington Post or New York Times to be investigated. It is simply not an issue – just as prior restraint on publication would be unimaginable in the US. The former general counsel of the NSA, Stewart Baker, expressed it thus in a recent debate in New York: "Snowden violated the law, but once he's given it to the reporters, the reporters are protected." It would be unthinkable for a president to suggest that the NYT should be investigated by Congress under the rubric of "counter-terrorism". This current administration has a poor record in regard to aggressively prosecuting whistleblowers, but it is virtually unheard of for espionage legislation to be deployed against reporters. American lawmakers do not confuse terrorism with journalism. Congress does still act as a check on the executive.

Meanwhile, Hazel Blears, the senior Labour member of parliament's intelligence and security committee, has said it is right that a debate is under way in Britain over the powers of the security services, adding that the committee’s inquiry into agencies' powerful new capabilities will go wherever the evidence takes it.

And last night it was reported that the director of the National Security Agency, Keith Alexander, and his deputy, Chris Inglis, are expected to depart in the coming months. Reuters described the development as one that could give Barack Obama “a chance to reshape the eavesdropping agency”.

While both men are leaving voluntarily, the dual vacancies give Obama an opportunity both to install new leadership following Snowden's revelations and to decide whether the NSA and Cyber Command should have separate leaders.

Cyber Command, which has grown significantly in recent years, has the authority to engage in both defensive and offensive operations in cyberspace. Many NSA veterans argue that having the same person lead the spy agency and Cyber Command diminishes the emphasis on the NSA's work and its unique capabilities.

Good morning.

The New York Times has carried out an interview with Edward Snowden via encrypted email, in which the NSA leaker says:

 He left all the leaked documents behind when he flew from Hong Kong to Moscow. He says he did not take the documents because that “wouldn’t serve the public interest”.

There was a “0% chance” the Russians or Chinese have received any of the documents. He says he targeted China when at the NSA, taught a course in Chinese “cybercounterintelligence” and knew how to keep the documents secure from them. He says he had had access to “every target, every active operation” mounted by the NSA against China, adding:

If that was compromised, NSA would have set the table on fire from slamming it so many times in denouncing the damage it had caused. Yet NSA has not offered a single example of damage from the leaks. They haven’t said boo about it except ‘we think,’ ‘maybe,’ ‘have to assume’ from anonymous and former officials. Not ‘China is going dark.’ Not ‘the Chinese military has shut us out.’

He gave the documents to the Guardian’s Glenn Greenwald and Ewen MacAskill and filmmaker Laura Poitras, but had been reluctant to say so before now to avoid exposing them to greater scrutiny.

He decided to become a whistleblower because he had no faith in the internal reporting mechanisms of the US government. “You have to report wrongdoing to those most responsible for it,” he said. He decided to act after reading about the NSA’s warrantless wiretapping programme during the Bush administration.

He never considered defecting while in Hong Kong or Russia, where he has been granted a year’s asylum.

His main objection to the NSA’s mass surveillance programmes was that they were being carried out in secret. “The secret continuance of these programmes represents a far greater danger than their disclosure,” he says adding:

So long as there’s broad support amongst a people, it can be argued there’s a level of legitimacy even to the most invasive and morally wrong programme, as it was an informed and willing decision. However, programmes that are implemented in secret, out of public oversight, lack that legitimacy, and that’s a problem. It also represents a dangerous normalisation of ‘governing in the dark,’ where decisions with enormous public impact occur without any public input.

The impact of his leaks was bigger than he anticipated.

My colleague Ed Pilkington has more here. We'll have all today's news and reaction on the NSA leaks here throughout the day.

That's it from us for today. We'll be back with more tomorrow.

Departing Guardian journalist Glenn Greenwald has responded on Twitter to Julian Smith MP's calls for the paper to be prosecuted (see earlier).

New European rules aimed at curbing questionable transfers of data from EU countries to the US are being finalised in Brussels in the first concrete reaction to the Edward Snowden disclosures on US and British mass surveillance of digital communications, reports Ian Traynor in Brussels.

Regulations on European data protection standards are expected to pass the European parliament committee stage on Monday after the various political groupings agreed on a new compromise draft following two years of gridlock on the issue.

The draft would make it harder for the big US internet servers and social media providers to transfer European data to third countries, subject them to EU law rather than secret American court orders, and authorise swingeing fines possibly running into the billions for the first time for not complying with the new rules.

"As parliamentarians, as politicians, as governments we have lost control over our intelligence services. We have to get it back again," said Jan Philipp Albrecht, the German Greens MEP who is steering the data protection regulation through the parliament ...

But the proposed rules remain riddled with loopholes for intelligence services to exploit, MEPs admit.

Read the full story here.

As Patrick Wintour reports, there will be a debate on the Guardian and the Snowden files on Tuesday afternoon in Westminster Hall for 30 minutes.

Tory MP Julian Smith has expanded on his calls for the Guardian to be prosecuted over the Snowden stories, saying in the Commons that this should happen if the paper does not hand over a decryption code that would enable the security services to read the intelligence files seized from David Miranda. He said:

The editor of the Guardian recently boasted online that he was taking precautions to prevent UK security services having access to the files of vital national security that he had sent out of the remit of the UK court to the New York Times. Security services are still trying to decrypt these files, which the Independent newspaper described as 'highly detailed and a threat to national security'. Can we have a statement to reassure the House that the Guardian will be asked for a decryption key, and if not forthcoming, action will be taken?

My colleague Andrew Sparrow reports that Tom Brake, the Lib Dem deputy leader of the Commons, said Smith would be able to pursue this matter further in a debate next Tuesday.

But Labour's David Winnick defended the Guardian, saying it was "receiving threats and smears ... for publishing details which is not in any way a threat to the security of our country, information which the public have a right to know".

I have just spoken to Julian Huppert, the Lib Dem MP who is one of the three pushing for a parliamentary debate on oversight of the security and intelligence services.

I asked him what made him want such a debate.

I think the information that has been revealed by the Guardian has shown just how much the security services have been doing which the public have been unaware of.

Of course it’s not up to us to know all the details … but the principles and the frameworks should be discussed and broadly accepted by the public.

He said that was in the interests of the intelligence and security services as well as the public.

What did he think of the push from other politicians for an investigation or even a prosecution of the Guardian over the leaks?

I haven’t seen anything that the Guardian has published that to my mind is a criminal offence. I don’t see the need to look into it.

He added that “there should be discussions about these issues”.

What was his view of Edward Snowden’s role?

I think he had to make a very difficult decision when he felt conscious of what had happened. The solution isn’t for individual whistleblowers to feel this pressure; there should be a clear framework in which people can operate [in order to draw attention to things they disagree with].

I put it to him that it seemed strange that the government had pushed so hard for the communications data bill when it seemed now that GCHQ already had broader powers than the bill would have granted it.

I am very alarmed that it does seem as though the committee that investigated the communications data bill were misinformed. We were deliberately not told material that would have been greatly helpful to us.

He said of Lord Blencathra, who led the formal inquiry into the bill and told the Guardian he condemned the fact his committee was never told about GCHQ's existing mass surveillance capabilities: “I completely agree with him.”

The bill is currently stalled after the intervention of Nick Clegg.

Huppert said he, Dominic Raab and Tom Watson would appear at the backbench business committee on Tuesday to make the case for such a debate, and that the committee typically made its decision by the next morning.

Free-speech pressure group Article 19 has posted a statement in response to David Cameron's call for a select committee to investigate the Guardian over the Snowden files.

The group – named after the article in the Universal Declaration of Human Rights recognising the right to freedom of expression – says the resulting home affairs select committee review "could have dangerous consequences for media freedom".

Article 19 says it continues to be "alarmed" about the "sustained pressure" being applied to the Guardian over its surveillance stories. Executive director Thomas Hughes says:

The Guardian’s reporting of the Snowden disclosures has facilitated a much needed public debate about blanket surveillance by national security agencies without appropriate oversight.

It has also exposed the violation of fundamental human rights of millions of people in the UK and around the world. This is exactly the sort of the debate that is necessary in a healthy democratic society.

We are deeply concerned that the investigation by the home affairs committee could have a chilling effect on press freedom in the UK and beyond - encouraging editors and journalists to exercise self-censorship when dealing with controversial issues.

We urge politicians and the authorities to seriously consider the long term implications of their actions, which may erode freedom of expression in the UK.

Click here for a recap of all the Guardian's surveillance scoops based on Edward Snowden's files, plus background, explainers and all the key documents.

Nick Pickles of campaign group Big Brother Watch has tweeted in support of the MPs' move.

Guardian deputy editor Paul Johnson tweets that the cross-party group of MPs Julian Huppert (Lib Dem), Tom Watson (Labour), and Dominic Raab (Con) have called for a parliamentary debate on oversight of the security and intelligence services.

More details soon ...

A Washington Post exclusive explains the link between the NSA and CIA, which means that drone strikes can target suspected terrorists based on details about references to their location in emails.

The story centres around Osama Bin Laden associate Hassan Ghul, whose whereabouts were detailed in an email from his wife picked up by the NSA.

In the search for targets, the NSA has draped a surveillance blanket over dozens of square miles of northwest Pakistan. In Ghul’s case, the agency deployed an arsenal of cyber-espionage tools, secretly seizing control of laptops, siphoning audio files and other messages, and tracking radio transmissions to determine where Ghul might “bed down.”

The e-mail from Ghul’s wife “about her current living conditions” contained enough detail to confirm the coordinates of that household, according to a document summarizing the mission. “This information enabled a capture/kill operation against an individual believed to be Hassan Ghul on October 1,” it said.

The newspaper has redacted details of CIA and NSA drone missions at the request of security services, it said.

You can watch the whole episode of Call Clegg here.

Clegg was asked whether he was pleased that the Guardian "is going to be investigated for the betrayal of publishing the spy secrets".

He replied:

I don't know what Keth Vaz's select committee [the home affairs committee] is doing.

Last week I said to you I think anything that helps terrorists and other people to learn more about the technical methods used by our agencies to keep us safe and so allow them to do more harm to us, that is a very bad thing.

Having said that I think it is entirely legitimate to ask ourselves whether the oversight arrangements, the way in which we make sure that the agencies who by definition have to work in secret, do so in a way which is accountable, because if you don't have proper accountability, in a way that the public trusts and understands – because quite a lot of the accountability mechanisms you've got at the moment are ... very much kind of Westminster Village stuff – the problem then is that people then start losing faith in the whole system.

Then of course there's the bigger issue, which is a debate which is happening here and on the other side of the Atlantic, and I saw that myself a couple of weeks ago in Washington, which is just how these big, new, powerful technologies are used both by security agencies and indeed by people who wish to do us harm.

My colleagues have written up this story in full here.

On his weekly phone-in show on LBC radio, deputy prime minister Nick Clegg was asked if he was glad the Guardian was going to be investigated by the home affairs select committee over the Snowden leaks.

Clegg said that last week he said that publishing details about how intelligence works could be harmful. But there was also a need for proper oversight of the intelligence services, he said.

More details soon ...

Good morning.

 Last night it was announced that the British parliament’s intelligence and security committee is to scrutinise the scale of mass surveillance undertaken by the UK’s spy agencies. The ISC’s inquiry will be formally launched today. The committee – which oversees GCHQ, MI5 and MI6 – will confirm that the investigation is a response to concerns raised by the Edward SNowden leaks. As my colleagues report:

Sir Malcolm Rifkind, the committee chair, said "an informed and proper debate was needed". One Whitehall source described the investigation as "a public inquiry in all but name" …

In a change from its usual protocol, the normally secretive committee also announced that part of its inquiry would be held in public.

It will also take written evidence from interested groups and the public, as well as assessing secret material supplied by the intelligence agencies. The Guardian will also consider submitting evidence.

Conceding that public concerns had to be addressed, Rifkind, a former foreign secretary, added: "There is a balance to be found between our individual right to privacy and our collective right to security."

This inquiry is separate to a Commons home affairs select committee investigation into whether the Guardian endangered national security and potentially broke the law through its revelations. This will be part of a broader investigation into counterterrorism. This came after David Cameron told the Commons yesterday:

The plain fact is that what has happened has damaged national security and in many ways the Guardian themselves admitted that when they agreed, when asked politely by my national security adviser and cabinet secretary, to destroy the files they had, they went ahead and destroyed those files.

So they know that what they're dealing with is dangerous for national security. I think it's up to select committees in this house if they want to examine this issue and make further recommendations.

A spokesperson for the Guardian said:

The prime minister is wrong to say the Guardian destroyed computer files because we agreed our reporting was damaging. We destroyed the computers because the government said it would use the full force of the law to prevent a newspaper from publishing anything about the NSA or GCHQ.

That is called 'prior restraint' and it is unthinkable in the US, where the New York Times and Washington Post have been widely applauded – along with the Guardian – for reporting on the Snowden files. That reporting has so far led to a presidential review and three proposed bills before Congress.

Here Richard Norton-Taylor and Ian Cobain provide 10 reasons not to trust claims that national security is being threatened by leaks.

These include the authorities' reactions to the Spycatcher case, WikiLeaks, Zircon – and the existence of GCHQ itself.

Ever since they were set up more than a hundred years ago Britain's security and intelligence agencies have been accused of using the excuse of "national security" to suppress information. Whenever information has been disclosed against their will, through leaks or whistleblowers, they have claimed security has been jeopardised.

Agencies are said to have consistently used this argument to protect themselves from embarrassment and to suppress evidence of information relating to a wide range of subjects, from government waste to involvement in torture.

Ministerial claims that the publication of reports based on NSA and GCHQ documentation undermined national security prompted a scathing response from United Nations experts on freedom of expression and human rights.

That's it from us for the day. We'll be back tomorrow.

Liam Fox, the former defence secretary, has written to select committee chairmen asking them to investigate what damage the Guardian's revelations about GCHQ and the National Security Agency may have caused, Andrew Sparrow reports.

Over 20 congressional bills aim to address the crisis of confidence in NSA surveillance, writes Yochai Benkler.

With Patriot Act author and Republican Congressman Jim Sensenbrenner working with Vermont Democratic Senator Patrick Leahy on a bipartisan proposal to put the NSA's metadata program "out of business", we face two fundamentally different paths on the future of government surveillance.

One, pursued by the intelligence establishment, wants to normalize and perpetuate its dragnet surveillance program with as minimal cosmetic adjustments as necessary to mollify a concerned public. The other challenges the very concept that dragnet surveillance can be a stable part of a privacy-respecting system of limited government.

As the Washington Examiner reports, a woman averted a mugging by telling the would-be thief she was an intern for the NSA:

I told him that the NSA could track the phone within minutes, and it could cause possible problems for him.

The mugger "just looked at me and ran away", she added.

The woman actually works for a not-for-profit organisation.

As internet news website the Daily Dot reports, the presiding judge of the US’s Foreign Intelligence Surveillance Court, has written to Senate judiciary committee chairman Patrick Leahy to try to explain the statistic that the Fisa court approves more than 99% of government surveillance requests. He wrote:

[The 99% figure] reflects only the number of final applications submitted to and acted on by the court. These statistics do not reflect the fact that many applications are altered prior to final submission or even withheld from final submission entirely, often after an indication that a judge would not approve them.

The Dot points out:

In June, President Barack Obama offered a similar analysis of the 99% figure, telling Charlie Rose that “folks don’t go with a query unless they’ve got a pretty good suspicion".

Here is Patrick Wintour's story on David Cameron's comments encouraging one of the Commons select committees to investigate whether the Guardian has broken the law or damaged national security by publishing secrets leaked by the National Security Agency whistleblower Edward Snowden. He writes:

The prime minister's spokesman refused to elaborate on what Cameron meant by the issue of the Guardian disclosures being examined by a select committee.

There are as many as four committees that might take up David Cameron's suggestion, including the culture select committee, the home affairs select committee, the defence select committee and the intelligence and security select committee.

The ISC largely meets in private but is due soon to meet the leaders of the spy agencies in public and it is certain that the issues raised by the Guardian, including the impact on national security, will be discussed.

Cameron did not follow calls by the backbench Tory MP Julian Smith for the police to prosecute.

At prime minister's questions just now, former shadow defence secretary Liam Fox asked David Cameron for an assessment of whether the Guardian's Snowden/NSA leaks have damaged national security. The prime minister replied:

I think the plain fact is that what has happened has damaged national security and in many ways the Guardian themselves admitted that when asked politely by my national security adviser and cabinet secretary to destroy the files they had, they went ahead and destroyed those files.

So they know that what they’re dealing with is dangerous for national security. I think it’s up to select committees in this house if they want to examine this issue and make further recommendations.

The Guardian's decision to destroy computer hard drives was taken after a threat of legal action by the government that could have stopped reporting on the extent of American and British government surveillance revealed by the documents. The editor of the Guardian, Alan Rusbridger, had earlier informed government officials that other copies of the files existed outside the country and that the Guardian was neither the sole recipient nor steward of the files leaked by Snowden. But the government insisted that the material be either destroyed or surrendered. As Julian Borger reported in August:

The Guardian's lawyers believed the government might either seek an injunction under the law of confidence, a catch-all statute that covers any unauthorised possession of confidential material, or start criminal proceedings under the Official Secrets Act.

Either brought with it the risk that the Guardian's reporting would be frozen everywhere and that the newspaper would be forced to hand over material.

"I explained to British authorities that there were other copies in America and Brazil so they wouldn't be achieving anything," Rusbridger said. "But once it was obvious that they would be going to law I preferred to destroy our copy rather than hand it back to them or allow the courts to freeze our reporting."

Fox said the Guardian had been guilty of double standards for exposing the scandal of phone hacking by newspapers and yet going on to publish secrets from the NSA taken by Snowden.

My colleague Helen Davidson in Sydney has been looking into the implications for Australia of yesterday's Washington Post revelations that the National Security Agency is harvesting hundreds of millions of contact lists from email and instant messaging accounts around the world. She writes:

More than 250m address books are gathered every year, collected “on the fly” as they are transmitted.

On a single day used as an example, the address book haul included 444,743 from Yahoo, 105,068 from Hotmail, 82,857 from Facebook, 33,697 from Gmail and 22,881 from other providers. A third of them came from an Australian Defence Signals Directorate (DSD) search.

A case study of the email account of an Iranian target was mined by the DSD on behalf of the NSA.

Good morning and welcome back to the Guardian’s live blog on all the latest developments resulting from Edward Snowden’s leaks to the Guardian on US and UK surveillance. Here are today’s headlines:

Former Labour cabinet minister Nick Brown has warned that GCHQ and Britain's other intelligence agencies appear to be undertaking mass surveillance without parliament's consent because the coalition failed to get the communications data bill – the so-called "snoopers' charter" – passed into law after Liberal Democrat opposition, reports Rowena Mason.

The communications data bill – dubbed the "snoopers' charter" by critics – would have given GCHQ, MI5 and MI6 much greater powers to gather and save information about people's internet activities but it was shelved in the spring amid Lib Dem fears that it intruded too much into privacy.

Brown, a Labour MP, said that it "looks very much like this is what is happening anyway, with or without parliament's consent" under GCHQ's secret Tempora programme, which was revealed by the Guardian in July in reports based on files leaked by Snowden. Tempora allows GCHQ to harvest, store and analyse millions of phone calls, emails and search engine queries by tapping the transatlantic cables that carry internet traffic …

On Tuesday night, two other Liberal Democrat members of the joint committee also questioned why the Home Office did not reveal the extent of GCHQ's spying capabilities during the committee's inquiry, which concluded the bill carried a risk of "trampling on the privacy of citizens". Lord Strasburger, a businessman, said nothing was mentioned about Tempora during two private "no holds barred sessions with the Home Office".

 Here Alan Travis looks back at the parliamentary scrutiny of the communications data bill. The bill is now stalled but, as Travis reports, reports of its death are likely to prove exaggerated. He quotes Strasburger as saying: "I sat on the committee and the Home Office misled parliament by concealing that they were already doing what the bill would have permitted."

 Theresa May, the home secretary, told the home affairs select committee there was nothing in the Snowden files that changed the case for new laws giving the security services more powers to monitor the internet. She also described the Guardian's publication of the material as "damaging to the public interest" and repeatedly rejected the need for a debate on oversight of the intelligence agencies. Afterwards, it emerged that Julian Smith, a Conservative MP, had written to the Metropolitan police asking the force to investigate whether the Guardian has broken the law by communicating intelligence information obtained from Snowden. The MP wrote to Sir Bernard Hogan-Howe to ask if offences had been committed under Section 58A of the Terrorism Act 2000 or the Official Secrets Act.

A Guardian spokesman said:

The high public interest in the stories we have responsibly published is evidenced by the debates, presidential review and proposed legislative reforms in the US Congress, throughout Europe and in Westminster. We're surprised that, once again, it is being proposed that terror legislation should be used against journalists.

Jill Abramson, executive editor of the New York Times, mounted a defence of the ability of journalists at her own paper and at the Guardian to publish public interest stories based on the files leaked by Snowden. "I think … that those articles are very much in the public interest and inform the public," she said on BBC's Newsnight. On claims by MI5 director general Andrew Parker that newspaper reports were causing "enormous damage" to the fight against terrorism, Abramson said that there had been no proof of actual harm to security.

In a leader column, the Guardian argues that comments by four public figures have illustrated the security services’ lack of accountability. Former cabinet minister Chris Huhne said the National Security Council was never briefed on the Prism or Tempora surveillance programmes. Lord Macdonald, the former director of public prosecutions, said parliamentary oversight was “sickly” and intelligence and security committee chair Sir Malcolm Rifkind badly compromised. And former ministers Lord Blencathra and Nick Brown revealed that the committee scrutinising the government's abandoned data communications bill was never informed about GCHQ's existing and far more sweeping mass surveillance capabilities.

Here, therefore, is the pressing problem that faces Britain's political leaders. Our security services practice mass collection of communications metadata which, as an NSA official admits, "tells you everything about somebody's life", terrorist or not. Neither the cabinet nor the parliamentary oversight, nor the legislative committee looking into snooping laws, has provided real accountability over such sweeping activity. In this, as in other respects, the security services enjoy a degree of legal and operational autonomy that exceeds what many MPs and ministers, if they knew about it, would judge appropriate. The head of MI5 says such discussion helps the nation's enemies. That is untrue. It is the unregulated surveillance that poses a threat to the nation, along with the threat from our enemies. Parliament urgently needs to exert a proper grip and to find a better balance. Starting now.

Glenn Greenwald, the journalist who broke a string of stories about widespread electronic surveillance by the National Security Agency based on files leaked by whistleblower Edward Snowden, has announced that he is leaving the Guardian.

We’ll have all this and more throughout the day today.

Pando Daily writes that one of the most disturbing Snowden revelations was that the NSA had worked with tech companies to build vulnerabilities into commercial encryption products that it could later exploit; something akin to cryogenics.

A technology called quantum key distribution protects against that, and Swiss startup ID Quantique has just received $5.6m in funding to pursue it.

Standard encryption works by (in essence) locking and unlocking information using a algorithmically generated key. QKD uses light protons:

The first part is the same: Data is encrypted using an algorithm. But then the data itself is encoded on a light particle known as a photon. Because photons are smaller than atoms, they behave in some pretty crazy ways. For example, you can “entangle” two photons so their properties correlate with one another. A change to one photon (which can occur as easily as by someone observing it) will cause a change in the other photon, even if the two are a universe apart.

After entanglement occurs, the sender transmits the first photon through a fiber cable to the receiver. If anyone has measured or even observed the photon in transit, it will have altered one of the properties of photon no. 1, like its spin or its polarization. And as a result, entangled photon no. 2, with its correlated properties, would change as well, alerting the individuals that the message had been observed by a third party between point A and point B.

This obviously makes no sense. Einstein called it “spooky action at a distance.”Richard Feynman said, “If you think you understand quantum theory, you don’t understand quantum theory.”

Researchers have previously managed to send QKD 150 miles, but after that the photons began to "decohere", and ID Quantique is working to extend that range. Its technology has already been used as part of Battelle's first commercial QKD network in the US.

The victory of the NSA over existing encryption standards ... was a political one, as the agency used its influence to muscle technology companies into setting up backdoors. In other words, we can use the most unbreakable, state-of-the-art encryption techniques we want, but if the gatekeepers of this technology succumb to government pressure, then all the spooky science in the world can’t help us.

Much nearer to home that the NSA is the threat from commercial surveillance products available for smartphones, according to Bloomberg. San Francisco-based Lacoon Mobile Security examined data traffic on the Middle Eastern network of a European operator and found 600 phones had spyware installed.

These are products that someone has actively installed on a device to monitor text, calls and even eavesdrop on calls in real time. Half the devices used a product called SpyToMobile; Bloomberg says that in the context of corporate espionage spyware is a growing problem.

For its part, SpyToMobile is actually using the NSA spy scandal in its sales pitch, although the company said it does not share data with the NSA. At least, not knowingly.

WA Today in Australia has looked at the Washington Post stats and worked out that 300,000 contact lists were harvested by Australian authorities in one day - on behalf of the NSA.

Australia's NSA counterpart — the Defence Signals Directorate (now the Australian Signals Directorate) — collected 311,113 address books as part of the program on a single day, naming it as the designated "DS" code in the leaked file. Another code, AUC, also attributed to Australia, appears in another document.

Canberra has yet to confirm Australia is the culprit, and the author of the report has himself left open the possibility of the DS prefix belonging to another country, however Australia has already been revealed to have close ties with US and other international intelligence agencies. In late August, for example, Fairfax Media revealed that the Australian Signals Directorate was in a partnership with British, American and Singaporean intelligence agencies to tap undersea fibre-optic telecommunications cables that link Asia, the Middle East and Europe and carry much of Australia's international phone and internet traffic.

Each day, the presentation said, the NSA collects contacts from an estimated 500,000 buddy lists on live-chat services as well as from the "inbox" displays of web-based email accounts.

Obama, in June 2013, saying that Congress is "fully apprised" of the NSA's activity and that only non-American citizens are having their communications monitored.

A helpful and very succinct recap from the tech blog TechCrunch today, following up the Washington Post story on email list harvesting. The NSA lacks congressional authority to gather address information.

To get around that lack of a mandate, the NSA has agreements with non-US telcos and works with other, non-U.S. intelligence groups. So to get its hands on even more information, the NSA avoids the constraints of its provided oversight and legal boundaries, by going to alternative sources of the data that it wants.

How private are you feeling, TechCrunch asks?

The phone metadata program knows who you called, when, and for how long. PRISM can force your private information out of major Internet companies. XKeyscore can read your email, and tracks most of what you do online. And the above program circumvents Congressional oversight by collecting more data on U.S. citizens by merely executing that collection abroad.

Facebook told TechCrunch a statement, which reflects founder Mark Zuckerberg's statement that the US government needs to be more proactive in communicating what it is doing.

As we have said many times, we believe that while governments have an important responsibility to keep people safe, it is possible to do so while also being transparent. We strongly encourage all governments to provide greater transparency about their efforts aimed at keeping the public safe, and we will continue to be aggressive advocates for greater disclosure.

Lavabit is opening for 96 hours to allow users to download their email archive. The 'window' is open until 19 October (1AM BST) and involves a password change.

Lavabit founder Ladar Levison was fined $10,000 for resisting against attempts by US authorities to access data of its users, and appealed against the court order.

First, the government is bereft of any statutory authority to command the production of Lavabit's private keys … Second, the Fourth Amendment independently prohibited what the government did here … Finally, the grand jury subpoena was oppressive, unduly burdensome, and ought to have been quashed.

No result for that as yet.

The NSA has been gathering hundreds of millions of email address books, according to the Washington Post, enabling them to map the relationships of millions of people - including US citizens.

During a single day last year, the NSA’s Special Source Operations branch collected 444,743 e-mail address books from Yahoo, 105,068 from Hotmail, 82,857 from Facebook, 33,697 from Gmail and 22,881 from unspecified other providers, according to an internal NSA PowerPoint presentation. Those figures, described as a typical daily intake in the document, correspond to a rate of more than 250 million a year.

Each day, the presentation said, the NSA collects contacts from an estimated 500,000 buddy lists on live-chat services as well as from the inbox displays of web-based e-mail accounts.

Telecommunications companies are complicit in the collection of this data, the report states, because the NSA is tapping facilities along major internet data routes. NSA director Keith Alexander is quoted as saying the agency "needs the haystack to find the needle".

Controversy around the data collection includes concerns about the potential for linking or identifying groups of people around religious, political or personal details. The NSA has not been authorised to collect lists in bulk.

Senior intelligence officials said it would be illegal to do so from facilities in the United States. The agency avoids the restrictions in the Foreign Intelligence Surveillance Act by intercepting contact lists from access points “all over the world,” one official said, speaking on the condition of anonymity to discuss the classified program. “None of those are on U.S. territory.”

The assumption is that anyone whose information passes through "overseas collection apparatus" is not a US person - according to an official.

In case you missed yesterday's This Week interview with WikiLeaks founder Julian Assange, he rejected the idea that the NSA/Snowden leaks put the public at threat, saying:

Every time the press embarrasses the security establishment, shows they have been acting unlawfully, against what they have said to Congress or to the media, they trot out this old canard, that some speculative harm sometime in the future might happen, when we’re discussing harm that is happening right now, as a result of these abusive programmes.

Assange also said that the west in general is “getting pretty close in the practical elements” of a totalitarian regime.

It’s a threat to US democracy and to democracy more broadly in the west to have a surveillance apparatus on every single person that would have been the dream of East Germany.

What type of place is western democracy going to be? Is it going to be a place with a collapsing rule of law, with mass surveillance of entire populations? The west is becoming a place where the best and the brightest, who keep the government, hold the government to account, are ending up in asylum or in exile in other countries. We’ve seen that before with dictatorships in Latin America, with the Soviet Union, and it’s time it stops.

That's it from me for today. Join us again here tomorrow.

The former director of public prosecutions in the UK, Lord Macdonald, has criticised the head of MI5 for using "foolish self-serving rhetoric" to resist legitimate calls for Britain's intelligence agencies to face more scrutiny in the face of revelations about their surveillance capabilities.

Writing in the Guardian, Macdonald was responding to Sir Andrew Parker's claim last week that stories about GCHQ's mass surveillance programmes had been "a gift to terrorists", and that there was no need for more rigorous oversight.

Macdonald also said that Sir Malcolm Rifkind, the chair of the intelligence services committee, which is supposed to provide parliamentary scrutiny of the agencies, was "badly compromised" and appeared too close to the organisations he was supposed to be holding to account. Rifkind was in charge of GCHQ when he was foreign secretary and publically supported Parker's position.

In his piece, Macdonald writes:

Worst of all has been the argument, heavily deployed in recent days, including by Sir Malcolm himself, that any more daylight than we currently enjoy simply assists the nation's enemies. Andrew Parker, the new director general of MI5, should be slower to employ this foolish, self-serving rhetoric that naively raises a perfectly legitimate question: how should we ensure that those privileged to be granted special powers to intrude into everything that is private serve a real public interest, rather than the dangerously false god of securitisation for its own sake?

Macdonald also criticises the "sickly character" of parliamentary oversight of the security services, writing: "An intelligence and security committee that goes into brief private session, only to emerge blinking into the daylight with protestations of apparent fealty to the security services, is a poor substitute for grown-up scrutiny."

On Arstechnica, Cyrus Farivar analyses which email providers are the most secure, in Germany and America. His conclusion? "Protecting email is hard."

Similarly, in Germany, Deutsche Telekom is trying to reach agreement with other internet providers to make sure any data transmitted domestically does not leave German borders. Thomas Kremer of the telecoms company suggested the measure could be rolled out across Europe. "In a next step, this initiative could be expanded to the Schengen area," he said, referring to the 26-country European agreement, of which Britain is not a part.

Controversy over the Snowden revelations has been particularly marked in Germany, where mass state snooping in the former communist east by the Stasi secret police casts a long shadow.

Dilma Rousseff, the Brazilian president, who has been one of the fiercest international critics of the US for its surveillance programmes, announced on Sunday that her government is planning to create a secure email system to try to protect official communications from US spies. "We need more security on our messages to prevent possible espionage," Rousseff said on Twitter.

In a statement released on Friday, the Foreign Intelligence Surveillance Court announced it had approved a renewal of what it called "the court authorization requiring the production of certain telephony metadata" – one of the programmes revealed by Edward Snowden. The office decided to announce the court decision "in light of the significant and continuing public interest in the telephony metadata collection programme". It was unclear for how long the renewal had been granted, but the announcement said the Obama administration was reviewing whether to declassify this court order. Other details of the programme were declassified and disclosed in the wake of Snowden's revelations.

On Arstechnica, Sean Vitka writes about the case of Basaaly Moalin, which John C. Inglis, deputy director of the National Security Agency has said was convicted of material support for terrorism partly on the basis of NSA surveillance.

Moalin is calling for a new trial, Vitka reports, “arguing that evidence collected about him under the government’s recently disclosed dragnet telephone surveillance program violated his constitutional and statutory rights”. Vitka writes:

The government’s response (PDF), filed on September 30th, is a heavily redacted opposition arguing that when law enforcement can monitor one person’s information without a warrant, it can monitor everyone’s information, “regardless of the collection’s expanse.” Notably, the government is also arguing that no one other than the company that provided the information—including the defendant in this case—has the right to challenge this disclosure in court.

Glenn Greenwald, one of the Guardian's key reporters of the Snowden scoops, has written a piece discussing Chris Blackhurst's column asking "Who am I to disbelieve [the security services]?" (See earlier.) Greenwald notes:

That this mentality condemns - and would render outlawed - most of the worthwhile investigative journalism over the last several decades never seems to occur to good journalistic servants like Blackhurst. National security state officials also decreed that it would "not be in the public interest" to report on the Pentagon Papers, or the My Lai massacre, or the network of CIA black sites in which detainees were tortured, or the NSA warrantless eavesdropping program, or the documents negating claims of Iraqi WMDs, or a whole litany of waste, corruption and illegality that once bore the "top secret" label. Indeed, one of the best reporters in the UK, Duncan Campbell, works for Blackhurst's newspaper, and he was arrested and prosecuted by the UK government in the 1970s for the "crime" of disclosing the existence of the GCHQ. When Blackhurst sees Campbell in the hallways, does he ask him: "who are you to have decided on your own to disclose that which UK officials had told you should remain concealed?"

Chris Blackhurst, the former editor of the Independent, has joined other press opponents of the Guardian in saying he would not have published Snowden’s leaked files.

In a comment piece for the Independent, Blackhurst – now group content director for his newspaper – writes:

My problem with publishing is twofold. First, try as I might, I cannot get that excited about it. With the Snowden leaks I find myself speculating – as I did with Julian Assange and WikiLeaks – as to whether I am getting too old and losing the plot as a journalist. But, as with WikiLeaks, will someone please put the boasts about size and volume on one side and tell me: where is the story? If it’s that the security services monitor emails and phone calls, and use internet searches to track down terrorists and would-be terrorists – including, I now read, something called the “dark net” – I cannot get wound up about it.

His second issue was this:

If the security services insist something is contrary to the public interest, and might harm their operations, who am I (despite my grounding from Watergate onwards) to disbelieve them?

He adds:

In August, this paper also received information from the Snowden files. We did not publish much of the information we were given because the Government, in the shape of a Defence Advisory Notice or “DA” notice, asked us to desist, in the interests of national security. Several times in my career, I’ve been served with a DA notice. On each occasion, I confess, I’ve not published. Does that make me a coward and an establishment lackey? Or responsible and sensible?

In August, two months after Blackhurst had stood down as editor, the paper published a story that began:

Britain runs a secret internet-monitoring station in the Middle East to intercept and process vast quantities of emails, telephone calls and web traffic on behalf of Western intelligence agencies, The Independent has learnt. The station is able to tap into and extract data from the underwater fibre-optic cables passing through the region.

This was where the story said the information had come from:

The Independent is not revealing the precise location of the station but information on its activities was contained in the leaked documents obtained from the NSA by Edward Snowden … Information about the project was contained in 50,000 GCHQ documents that Mr Snowden downloaded during 2012. Many of them came from an internal Wikipedia-style information site called GC-Wiki. Unlike the public Wikipedia, GCHQ’s wiki was generally classified Top Secret or above.

But Snowden said he was not the source for this story. “I have never spoken with, worked with, or provided any journalistic materials to the Independent,” he said.

We “need” the press to reveal secrets such as those the Guardian reported from the Edward Snowden leaks, Boris Johnson has written in his regular column for the Daily Telegraph.

In a piece attacking proposals to regulate the press, the Conservative mayor of London sets himself at odds with David Cameron when he writes:

We need someone to tell us that we are all being spied on by the American security services – that strikes me as being an invaluable bit of news, if hardly surprising. And if papers are genuinely at risk of compromising our national security by their revelations, then we have the D-notice system – to which all editors subscribe – to keep them in order.

View image in fullscreen

Good morning and welcome back to the Guardian’s live coverage of the fallout from former NSA contractor Edward Snowden’s revelations about US and UK surveillance. Here are today’s headlines:

 The Law Society is considering issuing new guidance to solicitors across England and Wales amid growing concern that the government's mass online surveillance operations are undermining their ability to take legal cases against the state, report Matthew Taylor and Nick Hopkins.

Lawyers representing people who make serious complaints against the police, army or security services fear the industrial-scale collection of email and phone messages revealed by the Guardian over the past four months is threatening the confidential relationship between them and their clients, jeopardising a crucial plank of the criminal justice system.

"These are absolutely fundamental issues," said Shamik Dutta, from Bhatt Murphy lawyers in London. "The NSA revelations are having a chilling effect on the way a crucial part of the justice system operates. Individuals who are making serious allegations of wrongdoing against the state are becoming increasingly concerned about whether the information they share with their lawyers will remain confidential."

 GCHQ is probably intercepting legally privileged communications between lawyers and their clients, according to a detailed claim filed on behalf of eight Libyans involved in politically sensitive compensation battles with the UK, reports Owen Bowcott.

The accusation has been lodged with Britain's most secret court, the investigatory powers tribunal (IPT), which examines complaints about the intelligence services and government use of covert surveillance. Most of its hearings are in private.

The allegation has emerged in the wake of the Guardian's revelations about extensive monitoring by GCHQ of the internet and telephone calls, chiefly through its Tempora programme.

The editor of the New York Times, Jill Abramson, has confirmed that senior British officials attempted to persuade her to hand over secret documents leaked by the former National Security Agency contractor Edward Snowden, reports Ed Pilkington in New York.

Giving the newspaper's first official comments on the incident, Abramson said that she was approached by the UK embassy in Washington after it was announced that the New York Times was collaborating with the Guardian to explore some of the files disclosed by Snowden. Among the files are several relating to the activities of GCHQ, the agency responsible for signals interception in the UK.

"They were hopeful that we would relinquish any material that we might be reporting on, relating to Edward Snowden. Needless to say I considered what they told me, and said no," Abramson told the Guardian in an interview to mark the International Herald Tribune's relaunch as the International New York Times.

Read the full interview here.

 Edward Snowden has said that the mass surveillance programmes used by the US to tap into phone and internet connections around the world are making people less safe.

In short video clips posted by the WikiLeaks website on Friday, Snowden said that the NSA's mass surveillance, which he disclosed before fleeing to Russia, "puts us at risk of coming into conflict with our own government". View the video clips below:

The Guardian's critics say journalists cannot be trusted to judge what may damage national security, writes investigative journalist Nick Davies. But the press’s track record shows it to be more trustworthy than politicians or spooks, he says, also making a point about the proposed communications data bill, which is now stalled:

It's instructive to look back from our vantage point, post-Snowden, to consider what was happening only two years ago when the government attempted to introduce new legislation which came to be known as the snooper's charter. If the oversight politicians are as well-informed as they claim, they must have known that this was in part a cynical attempt to create retrospective legal cover for surveillance tools that were already secretly being used, but they said nothing. And when parliament refused to pass that law, clearly indicating that there was no democratic mandate for those tools, they still stayed silent.

And a Guardian leader column asks whether politicians can be trusted to supervise the security services. 

Do those who have oversight really understand today's technologies? And are they themselves free of personal inclinations? The former home secretary Jack Straw has been critical of the Guardian's reporting. But his own life and career raise their own questions. In his autobiography there is a revealing passage in which MI5 tells him in 1974 that he and his family have been under surveillance since Mr Straw was a teenager. They questioned him about his sexuality. They believed (probably falsely) that his sister was a communist. MI5 had notes of a pub conversation he had had with a "politically suspect" friend years earlier. Many people would find this deeply disturbing. Not Mr Straw, who was "neither surprised nor shocked".

Is it reassuring for the rest of us that Mr Straw went on to have responsibility for the work of our spy agencies as home and foreign secretary? Mr Straw accepted the unreliable intelligence assessments which led us into war in Iraq. He faces troubling questions (and civil proceedings) about rendition. He backed secret courts. He supported blocking the Serious Fraud Office inquiry into corruption at British Aerospace. Quis custodiet? The answer, for many people, wouldn't be Jack Straw. Which is merely illustrative of a question which won't go away.

We’ll have live coverage of all today’s developments here throughout the day. You can read last week’s blog here.