Two years ago, GCHQ’s annual sports day took place on Wednesday, 15 June at the Civil Service Sports Club in London. A mixed six-a-side football tournament was the centrepiece of the day, with matches kicking off at 11am sharp.
The event was a jolly for those routinely cooped up in the agency’s distinctive doughnut-shaped headquarters in Cheltenham, and they were furnished with six pages of rules and regulations to ensure fair play.
"Each team MUST field at least ONE lady player at all times," the note said. "Appropriate footwear shall be worn. No crocs, sandals or flip-flops will be allowed. The wearing of shin-pads is COMPULSORY."
Of all the highly classified documents about GCHQ revealed by the whistleblower Edward Snowden, this has to be one of the least sensitive. But it offers a glimpse into the world of the 6,100 people crammed into the open-plan and underground offices at GCHQ; the fact there is a sports day at all reveals something about the agency which most people outside their bubble could not appreciate.
Last year, GCHQ organised trips to Disneyland in Paris, and its sailing club took part in an offshore regatta at Cowes. It has a chess club, cake sales, regular pub quiz nights and an internal puzzle newsletter called Kryptos. A member of Stonewall since last year, GCHQ has its own ‘Pride’ group for staff who are lesbian, gay, bisexual and transgender. There is even a paranormal organisation. Describing itself as "GCHQ’s ghost-hunting group", it is open to staff and their partners "whether they are sceptics or believers" for visits to "reputedly haunted properties".
Staff date themselves on the internal directory, "GCWiki", by their "internet age", a measure of how many years they have been adept on the web.
They make friends during annual family open days, or via messages on the agency’s internal version of MySpace, which they have called SpySpace.
Colleagues are likely to find people cut from the same cloth. The agency’s 2010/11 recruitment guide says GCHQ needs high-calibre technologists and mathematicians familiar with the complex algorithms that power the internet. It has room for a sprinkling of accountants and librarians. Classicists need not apply.
Nobody at Cheltenham is particularly well paid, compared to the private sector at least – a junior analyst might earn £25,000. "We can offer a fantastic mission but we can’t compete with [private sector] salaries," one briefing note lamented.
In a world of its own, GCHQ is a complex, secret community, which is tightly bound by its location outside the capital, the nature of its people, and the secrecy in which their work has to be done.
When it was built in 2002 the "doughnut" was the biggest construction project in Europe. It is now home to a parallel world – one that mirrors the society around it while being set apart by high walls of secrecy and the vastly superior technology concealed within.
Today this intensely private organisation is under a spotlight it has never had to face before, as its methods and practices come under unprecedented scrutiny, thanks to the release of files that would otherwise have been locked away for another 30 years.
Snowden wanted to reveal the extent of the surveillance activities being undertaken by GCHQ and its American equivalent, the National Security Agency (NSA), and the stories published by the Guardian have certainly done that.
Before the 30-year-old analyst turned whistleblower, only a few people outside GCHQ had heard of "Tempora", the programme that gives the agency access to the fibre-optic cables which carry the world’s phone calls and web traffic; only they knew it had developed an ingenious way of storing this material for up to 30 days.
Only those in the intelligence community had heard of "Prism", another initiative that has given the NSA – and GCHQ too – access to millions of emails and live chat conversations held by the world’s major internet companies, including Google, Facebook, Microsoft and Apple.
Teams of analysts at GCHQ now have the authority and the technical capacity to tap directly into the nervous system of the 21st century and peer into the lives of others. Dig deeper into the drily worded, acronym-filled files, and there are other insights about the challenges faced by GCHQ, and its own anxieties about meeting them.
And while politicians, including the prime minister and William Hague, the foreign secretary, have led the defence of the agency from the questions posed by the recent revelations, the papers show the agency is not always at ease with itself. There is understandable concern about being left behind by technology, and a desire to drive itself on so it can continue to feed high-quality intelligence to the "customers".
They include the government, the domestic security service, MI5, and the Secret Intelligence Service, MI6.
But the "customer" the agency frets about most is the NSA. In numerous papers, GCHQ reveals its need to keep the Americans happy, and how it regards this as an overriding priority.
It is not hard to see why; the Guardian has discovered GCHQ receives tens of millions of pounds from the NSA every year, money it has come to rely upon to build and maintain its collecting and decoding capabilities. In turn, the US expects a service, and, potentially, access to a range of programmes, such as Tempora.
Those campaigners and academics who fear the agencies are too close, and suspect they do each other’s "dirty work", will probably be alarmed by the explicit nature of the quid-pro-quo arrangements.
Though there is evident excitement within GCHQ that new responsibilities in recent years have made it Britain’s pre-eminent intelligence agency, it has been accompanied by occasional pauses for reflection, and worry that the agency cannot cope with those demands.
In an internal document published in August last year, one of GCHQ’s most senior officers set out his fears. The officer, one of the team responsible for managing the Tempora project, used a power-point presentation to explain to colleagues the far-reaching way GCHQ’s "mission role had changed".
He reminded his team that new techniques had given it access to vast amounts of new data or "light" – emails, phone calls and Skype conversations garnered from internet cables. But the officer was obviously disconcerted.
"Over the last five years, GCHQ’s access to ‘light’ [has] increased by 7,000%," he explained. The amount of the material being analysed and processed had increased by 3,000%, he said – another startling admission.
"GCHQ is breaking new ground and in doing so, testing our systems and processes to the full. Our challenge today is to achieve success against tomorrow’s demands starting from yesterday’s capability."
But he warned the agency was ill-equipped to do this: "The complexity of our mission has evolved to the point where existing mission management capability is no longer fit for purpose."
New threats, new enemies, new challenges - the rise and rise of GCHQ
Perhaps it isn’t surprising such concerns have been raised in private around the corridors of GCHQ. Over the past decade, the agency’s portfolio has evolved into something barely recognisable to its most celebrated alumni – the Nazi code-breakers of Bletchley Park.
GCHQ’s core business was always the "gathering intelligence based on intercepted communications". It still does this, but the days of putting "clips on copper wires" to hear phone conversations are long gone.
The world has embraced –computers, tablets and mobile phones, and the need to find valuable information amid vast amounts of digital traffic created by them has become more difficult.
GCHQ has been tasked with finding the solutions, mindful that the potential rewards are high; never before has the agency had the opportunity to build such a complete record of someone’s life through their texts, conversations, emails and search records.
The use of cyberspace by criminal networks and other states to attack government departments and British businesses has opened a new dimension of silent warfare. With its technological and computing background, GCHQ has been told to defend the nation – and to develop the means for counter-attack.
Once a niche area, this is regarded by Downing Street as a "Tier One" national security priority because of the damage being done to the UK economy, and the danger of British defence secrets being stolen by stealth through sophisticated hacking attacks.
The pressure on the agency to deliver on all these fronts was made clear in GCHQ’s corporate plan for 2009, the first year in which Sir Iain Lobban was director. In his foreword, he warned colleagues the agency had to do more.
"This needs to be the year when we achieve real traction with our internet age transformation so we can continue to deliver in the future what HMG [Her Majesty’s Government] and our allies have come to expect of us.
"Over the last five years we have seen GCHQ change from being simply an intelligence producer into a genuine operational partner for the military and civilian customers."
The report added: "Put simply, HMG expects value from GCHQ which at least matches the £1bn a year that is being invested in us each year."
With so much now resting on the agency, its influence has spread across Whitehall. GCHQ now has liaison officers working inside MI5, MI6 and the Soca, the serious and organised crime agency. It takes the lion’s share of the £1.9bn budget for Britain’s intelligence services, and has a staff that is more than twice the size of the combined workforces of MI5 and MI6.
GCHQ also has a hefty presence in the Cabinet Office, which is responsible for setting the UK’s cyber security strategy. Defending the nation is the Cabinet Office’s priority – but from whom?
In an internal report in 2010, GCHQ described with remarkable candour the threats posed to the UK from cyberspace, eschewing the mealy-mouthed formulations adopted by ministers – and Lobban – in public.
The government has consistently maintained it is too difficult to point the finger at any particular country when it comes to cyber attacks. The 33-page report written by GCHQ’s Cyber Security Operations Centre makes a nonsense of that. Beijing is to blame, it says.
"China has a capable and very wide-ranging cyber programme targeting the full spectrum of governmental, military, and commercial targets. The Chinese mount a large number of relatively unsophisticated attacks, often using publicly known vulnerabilities and have successfully compromised networks globally.
"This assessment is based only on the attacks that have been detected, and does not preclude more sophisticated and targeted attacks from China."
The report adds: "Allegations of Chinese involvement in cyber attacks are unlikely to deter China from carrying out similar attacks in future, or from censoring its population’s access to the internet.
"China is a major player in the global telecommunications market. In addition to the threat of industrial espionage to sustain this position, there is an inherent risk of Chinese equipment being used for intelligence purposes.
"Chinese industrial espionage comprises the single greatest threat to US technology … Various UK companies have also been targeted and large amounts of data have been lost."
Insisting that British interests are "under sustained attack", the paper is highly critical of Russia. It says Moscow "operates a sophisticated, mature and successful cyber programme, using an extensive global internet-based infrastructure".
"The programme employs a wide variety of malicious software, and poses a significant threat to UK networks.
"Targeting of UK government departments is assessed to be a priority for Russia, and is likely to be ongoing. Governments, industry and academic institutions across a range of sectors have been targeted. Russia is judged to pose a threat to UK communications in a variety of countries, and UK data may be at risk due to compromises of networks outside of UK control."
But the document says the UK has started a fightback of sorts.
"The UK is developing and testing offensive cyber capabilities, although policy is not yet in place to underpin all potential opportunities."
In June, 2010, a new "Cyber Development Centre" was opened at GCHQ’s –sister base in Bude, north Cornwall, which is a hub for the analysis of intercepted satellite and internet traffic. Since then, GCHQ and the Ministry of Defence have been working on a secret programme to provide the military with new cyber weapons "to achieve desired effects in the run-up to or during a –conflict", another file explained.
In a top secret memo, the director of cyber operations at GCHQ told colleagues the agency needed to turn cyber into "another capability alongside air, sea and land forces." The coalition committed £650m to cyber security during the 2010 strategic defence and security review ; defence sources have told the Guardian half of this cash went to GCHQ to give the UK attacking capabilities against other states.
Mastering mobile phones - 'any time, any place anywhere'
While cyber warfare is regarded as one of the agency’s top priorities, it has two other pressing technological challenges that are treated with equal significance.
The rise of smartphones and the widespread use of encryption by internet service providers to protect the privacy of web users has also caused them great anxiety.
Much of GCHQ’s focus over the past 20 years has been snooping on people’s desktop and laptop computers. But now people are using mobiles and tablets as mini-computers; they can do internet searches wherever there is a 3G signal, and use a range of apps for –communicating with friends and colleagues.
The agencies have had to rethink what they do because it is much more difficult to target information from hand-held devices, particularly when so much email traffic is now routinely encrypted to stop hackers finding out what you are saying, doing, and paying for.
GCHQ realised the gravity of the problem; it could spend millions of pounds developing systems to sweep up and store vast amounts of information from these devices, but the value of this database would "weaken over time" if the material was encrypted, a briefing explained. The agency’s many and varied sifting and analysis tools – which search for key words, names and patterns – would become increasingly redundant.
A GCHQ internal review for 2011/12, said: "The two major technology risks that GCHQ has to face next year are the spread of ubiquitous encryption on the internet and the explosion in the use of smartphones as mobile internet devices. Over time both of these technologies could have significant effect on our current trade craft."
In July last year, GCHQ circulated a document which put the challenge in context. The agency estimated that by 2015, 90 per cent of all internet traffic would come from mobile phones, and that there were already 100 million smart phones being used across the world. The mobile, said GCHQ, was the "most prolific customer product ever invented."
"Mobiles are already far more than just phones and will continue to evolve in the future. GCHQ is playing catch-up. Our exploitation of mobiles is fragmented, uncoordinated." The document revealed GCHQ had launched a new "mobile" project which was designed to "exploit mobile devices".
"Not just collecting voice and SMS and geo-locating phone," the document said, "but getting intelligence from all the extra functionality that iPhones and BlackBerrys offer. Mobile is all about staying one step ahead of how our targets are sharing info on the go."
That seems to be a reference not only to the calls people make, but the emails they send, the internet searches they make, and the messages they post on social media sites. The agency said it needed a comprehensive approach that would "deliver more than a quick fix. We need a solution that will adapt and grow as mobile phones evolve".
Another paper encouraged analysts to see the exponential rise of mobile technology as an opportunity for GCHQ, given the richness of the intelligence on offer. "The world is going mobile. Mobile telephony has already overtaken landlines … Google Apps already has over 30 million users. This is good news. It allows us to exploit the mobile advantage."
The end-goal was ambitious. GCHQ said it wanted to be able to "exploit any phone, anywhere, any time".
Was this legal? By 8 February, 2011, GCHQ was reporting internally that "legal assurances [were] now believed to be good", though it is unclear what this refers to.
The mobile project has now been absorbed into the over-arching "Mastering the Internet" programme (MTI), which was established by GCHQ to allow it to capture as much information as possible from the world wide web.
The Guardian has seen documents which make clear "a surge" of mobile activity by the agency over the past three years has helped it make up lost ground. Papers said GCHQ was now capable of "attacking" hundreds of apps, and a "mobile capability map" from June last year stated the agency had found ways of looking at the search patterns, emails and conversations on many commonly used phone services.
In July last year, the head of the MTI programme congratulated the analysts working on the top secret project.
"This is a big step towards us developing capabilities to exploit mobile opportunities," he said. "Please pass on our thanks to all the team…"
GCHQ's most demanding customer: the NSA
There is one recurring worry that appears to infuse all of GCHQ’s work, and is mentioned in numerous documents; the need to satisfy the demands of the NSA. This is an anxiety that has been developing since the early years of the second world war. In 1940 and with Britain’s position desperate, Winston Churchill sent Lord Lothian, his ambassador to the US, to negotiate with President Roosevelt over ways the countries could share intelligence.
Churchill offered secret details of Britain’s latest developments in radar and other scientific fields; in return he asked from the Americans for "certain information of a technical nature". These tentative exchanges evolved into a complex intelligence sharing relationship. In 1947, the UK and USA signed an agreement, together with Canada, Australia and New Zealand which was initially known only as the "secret treaty".
The deal gave birth to an acronym – UKUSA – and an extraordinary deal among these English-speaking nations. Each would devote its cryptological resources to different regions of the world. The alliance is also known as Five Eyes. The relationship is not completely lopsided; thanks to its colonial history, the UK has been able to offer the NSA access to a network of listening stations across the globe, in places such as Hong Kong.
One of its most important outposts is RAF Menwith Hill in Harrogate, north Yorkshire, a British military base in name alone. Since the 1950s, "MHS" has been the NSA’s satellite surveillance and interception hub in Europe and it remains much coveted by Washington. But the UK’s leverage has to be applied with care.
Nobody in Whitehall wants to risk a repeat of the calamity of 1973 – when President Richard Nixon ordered an end to intelligence sharing with Britain, having taken a dim view of Prime Minister Edward Heath’s cosiness to Europe, and his haughty attitude to the US. Britain’s spy agencies were horrified when the taps were turned off, and it took a change of government in both countries for relations to get back on an even keel.
In the decades since then, the NSA has become the world’s biggest intelligence-gathering organisation, a position that has left its allies uncomfortably reliant on its financial and technical resources.
The Snowden files reveal just how much the UK is in hoc to the US.
There are details of payments made to GCHQ by the NSA, and vignettes which point to the UK’s need to do America’s bidding or risk losing favour with the senior partner.
The UK’s comparatively weak oversight regime, and the flexibility of British laws are referred to as "selling points" to Washington – a clear –indication, perhaps, that the US expects –Britain to help in areas it finds too difficult because of America’s own rules and regulations. At the very least, it suggests the UK trades on the weak oversight regime that British ministers have been fighting so hard to defend.
In December 2009, GCHQ, published a strategy document which highlighted the agency’s most significant concerns for the future. Chief among them was this: "US perceptions of the UK/USA partnership diminish, leading to loss of access, and/or reduction in investment in accesses of interest to the UK."
The money spent by the NSA would seem to reinforce some critics’ fears that GCHQ has become, to some extent, a subsidiary of the American intelligence giant. The papers show the NSA has been making regular payments to GCHQ for a number of years. In 2009, the NSA gave the agency £22.9m.
The following year, the GCHQ mid-year review acknowledged it was receiving £118m of external funding, "mostly from the Home Office, –Ministry of Defence and the NSA". The NSA contribution amounted to £39.9m, which included £4m to support GCHQ’s work for Nato forces in Afghanistan, and £17.2m for the UK’s Mastering the Internet project.
America also contributed £15.5m to "radically enhance the infrastructure at Bude" in north Cornwall – one of GCHQ’s five sister bases. "Securing external NSA funding for Bude has protected the [core] budget," the GCHQ document explained.
This paper acknowledged GCHQ is dependent on the "NSA’s continued investment in key technologies of interest to the UK". The "2011/12 investment portfolio" was also specific; it said external funding would reach £150.7m, and £34.7m would come from the NSA.
The cash injections appear to give the US considerable influence over what programmes GCHQ intends to pursue.
"The portfolio will spend money supplied by the NSA and UK government departments against agreed requirements," the paper explains.
No wonder, perhaps, that the NSA makes continual demands, which the British agency struggles to meet. In 2010, GCHQ admitted the Americans had "raised a number of issues with regards to meeting NSA’s minimum expectations … we are still short of the full NSA ask".
It added: "The NSA ask is not static and retaining ‘equability’ will remain a challenge for the near future."
On one decryption project, GCHQ feared if it failed to "deliver" it would "diminish NSA’s confidence in GCHQ’s ability to meet minimum NSA requirements".
The following year, GCHQ was obviously keen to avoid finding itself in the same position. Senior officers reassured the agency’s investment board – which signs off where money will be spent – that "the portfolio includes an appropriate level of contribution as viewed from the NSA perspective".
The NSA is also a demanding customer of GCHQ’s foreign capabilities, particularly in the Middle East. A blueprint for the future of Cyprus – a hub for military intelligence-gathering across the region – said this had to "remain resourced and equipped … to maintain healthy relationships with USA customers".
The papers show the NSA pays half the costs of one of Britain’s Cyprus-based surveillance systems, and that GCHQ has been worried that recent cuts in spending have been affecting its –ability to serve up the right amount of quality intelligence to the US.
"This is not sustainable if numbers reduce further, and reflects badly on our commitments to the NSA," said a senior manager writing in April last year.
The American money came at a –crucial time for GCHQ. In 2010, it –suffered what it called a £60m "super cut" in funding as part of the coalition government’s efforts to bring down Whitehall spending. The agency had taken steps to mitigate the loss, by reduced costs on a number of programmes.
"We feared that due to the current state there would be a risk that the money supply may be reduced at short-notice," a report said. "This proved to be the case with government cuts in year to GCHQ’s budgets."
"Simply doing everything we do today more efficiently will come nowhere near close to bridging the funding gap," another report explained.
"To do more of the new it’s now vital that we cease doing some of the old, making firm decisions on activities which no longer produce sufficient value."
When GCHQ is able to provide the Americans with useful material, or access to intelligence they did not have, the British are cock-a-hoop. This was particularly obvious when GCHQ started tapping traffic from undersea internet cables, which is referred to as Special Source Events in the documents, or by the codename Tempora.
In 2011, the agency boasted that sharing this database with the Americans highlighted "the unique contribution we are now making to the NSA in providing insights into some of their highest priority targets".
GCHQ also boasted that it had given the NSA 36% of all the raw information the British had intercepted from computers the agency was monitoring. The intelligence had been "forwarded to NSA", the document explained.
It added: "We can now interchange 100% of GCHQ End Point Projects with NSA."
This suggests the NSA potentially has access to all the sifted and refined intelligence gathered by GCHQ.
The agency was also keen to take credit for the role it played providing information to the NSA over two attempted terrorist attacks in the US.
The first involved the man who tried to blow up an airliner bound for Detroit on Christmas Day, 2009. The suspect, Umar Farouk Abdulmutallab, hid the device in his underpants, and has since been jailed for life.
Five months later on 1 May, 2010, there was an attempted car-bombing in New York’s Times Square. Faizal –Shahzad, a 30-year-old Pakistani-born man who became a US citizen in 2009, has since been jailed.
In the mid-year review for 2010/11, GCHQ proclaimed: "Our partners have felt the impact of our capability too, with NSA in particular, delighted by our unique contributions against the Times Square and Detroit bombers."
What those contributions were is not explained. We know the NSA is forbidden from spying on American citizens; in the case of Shahzad, this question remains – was GCHQ doing it for them?
Keeping the US happy to remain the UK's no1 priority
Two years ago, GCHQ set out in a –colour-coded diagram its priorities for the coming years, and described its relationship with the US in terms of what it could offer, and hope to get in return.
It declared the UK/USA agreement "remains our pivotal partnership" and that Britain’s help to the Americans has been "rewarded".
"Through our work with the NSA, other US government departments acknowledge our contribution to their cyber agenda. We are recognised and rewarded for having consistently punched above our weight across those areas that matter most to the US."
It added: "Sharing of data between GCHQ and NSA has increased significantly. Technology and roadmaps align well. We both accept and accommodate NSA’s different way of working."
That might be a reference to the tighter legal and regulatory regime in which the NSA has to operate. This includes abiding by the Foreign Intelligence Service Act (FISA) and scrutiny by FISA courts. Not so in the UK, and the document explained this is another area where GCHQ can help the US.
"We are less constrained by NSA’s concerns about compliance," it said. "We handle mutual operational compliance issues readily."
Despite ministers insisting Britain’s intelligence agencies are tightly controlled and regulated, the comparative looseness of this regime is regarded as an important "selling point" for Washington.
In the UK, GCHQ needs only to get the authority of a minister to win approval for mass surveillance under a little-known section of the Regulatory Investigatory Powers Act 2000.
In July 2010, this advantage was made explicit in a GCHQ strategy document which described "a desired end state" for what it could offer the NSA within three years.
"In 2013, we will have access to and make readily available, data of the highest possible value to facilitate cyber operations. We will have exploited to the full our unique selling points of geography, partnerships, the UK’s legal regime and our skilled workforce."
Britain, GCHQ admitted, is reliant on this compliant legal framework "for continued access" to the undersea internet cables that carry "UK transit traffic at scale".
Undercover and under the spotlight
Most of the staff at GCHQ are so involved in their own projects they may not always have had time to contemplate all the repercussions of what they are being tasked to do.
One document drafted by analysts involved in an encryption team noted: "We’re good at solving the technical challenges, chipping away at hard problems over a period of time. We are less good at communicating scale of the problem to others. Perception [is] that we are in our own little world…"
And so they are. It is a world of decoding and cake sales, programming and pub quizzes.
There is no hint of politics in the documents, though there appears to be a strong sense of engagement in the world beyond British shores.
In 2009, several employees organised a "ride and stride" to raise money to feed the Palestinian population in Gaza, where they described the humanitarian situation as "extremely dire".
Staff cycled, ran or walked a total of 2,230 miles – the distance from Cheltenham to Gaza – to help Palestinians "survive extreme deprivation".
But those who join this elite club are under no illusions about the secrets they are expected to keep.
There are protocols on the clearing of desks at the end of the day, with particularly sensitive documents being locked in special cupboards, the keys to which are then stored in other reinforced –lockers which can only be opened by following a set of complex instructions.
The strictures accompany GCHQ workers lucky enough to travel abroad on the job. Analysts sent to a global congress on mobile technology in Spain three years ago went undercover as "trainers".
Getting in to the most interesting talks however took some ingenuity.
"As an ‘exhibitor pass’ holder I had to invade the personal space of one of the door ladies to distract her from scanning my badge," one of the analysts reported back.
The British delegates reported they were not the only spies undercover that weekend. "Interestingly enough we spotted several NSA’ers who recognised us but were also using some light cover so we kept our distance."
And, of course, anyone who works for GCHQ can never talk about work to anyone outside the circle of trust.
In a prescient footnote to the guide published two years ago, the author said: "GCHQ has been in the media spotlight a fair bit recently and you may get some individuals asking you about aspects (for example, they may get you to comment on stories around GCHQ being asked to monitor all emails). Under no circumstances enter into any dialogue around these topics."