Bernard Madoff: Wall Street's Wizard of Oz

As they swapped investment stories at the golf course in Palm Beach, Bernard Madoff's delighted clients sometimes joked that if he was a fraud, he'd take down half the world with him.

In this, at least, they were perceptive. The fallout from the news that the celebrated Wall Street investor was behind an alleged $50 billion (£32.7 billion) swindle – a "Ponzi" scheme in which he paid early investors with the money put in by later ones – has swept through the worlds of international business, moneyed society and philanthropy. Potentially thousands of investors, including hedge funds, banks and wealthy individuals, have had huge sums wiped out at a stroke in what could be the biggest fraud in Wall Street history.

A Jewish charity in Boston has shut down and sacked its staff, a group of hedge funds in Connecticut has said it is closing ("I'm wiped out", said its chairman simply) and banks as far afield as Paris, Zurich and Tokyo are anxiously calculating the cost of their exposure to Bernie Madoff. Geneva-based banks and investment funds alone are estimated to have lost more than £2.5 billion. Even Nicola Horlick, the so-called "superwoman" of the City of London, has been hit, her Bramdean Investment firm reportedly investing nearly £21 million with Mr Madoff. Like others, she could only splutter her fury over the weekend at the "systemic failure" of US regulators in letting him get away with it for years.

She may have a point. It appears that Mr Madoff's Manhattan-based investment advisory business was never inspected by regulators after he subjected it to oversight two years ago. Mr Madoff, 70, was a generous political donor and had advised the Securities and Exchange Commission how to regulate markets. According to Bloomberg News, for whatever reason, the SEC hadn't examined his books since he registered the unit in September 2006.

A dozen SEC inspectors are now poring over his records, trying to find out what went wrong. There are plenty of theories around already but the bottom line is that Mr Madoff fooled a lot of people who should have known better. Supposedly "sophisticated investors" included Fred Wilpon, the main owner of the New York Mets baseball team; Norman Braman, a big Florida car dealer who once owned the Philadelphia Eagles American football club and Ezra Merkin, the chairman of General Motors' financing arm. Carl Shapiro, a Boston philanthropist and former women's clothing magnate, has lost at least $145 million (£95 million), it was reported yesterday. The alleged fraud had "swept up some of the most prominent and wealthy Americans, along with many people who thought they were embarking on a comfortable retirement and have now been left destitute," said Brad Friedman, a lawyer involved in representing more than 30 investors with losses they believe could total more than $1 billion (£655 million).

The big names, it appears, would have been among an A-list of investors that Mr Madoff used as a marketing tool to pull in others. He hired unofficial agents and fostered an aura of exclusivity by targeting wealthy country clubs across the US with "invitation only" policies. Refusing to take some people on as investors only added to his appeal – Barbara Fox, president of a Manhattan estate agency, told the New York Times she "literally begged him" to take her on but he turned her down.

At lunch or on the links, members would talk of him in hushed, admiring tones – Magic Mr Madoff and his Amazing 10 per cent Returns, even in a bear market. In clubs such as the Palm Beach Country Club in Florida and the Old Oaks in Purchase, New York, some called him "the Jewish bond" because his fund always managed to pay eight to 12 per cent every year. He also had unofficial agents – friends and colleagues – helping him link up with the wealthy of Dallas, Chicago, Boston and Minneapolis. In Minneapolis, investors from two country clubs alone are thought to have sunk $100 million (£65 million) into his operation.

Even his agents invested heavily. Richard Spring, a former securities analyst from Boca Raton, told the Wall Street Journal he had about $11 million (£7 million) – 95 per cent of his net worth – invested with Mr Madoff. "That's how much I believed in him," he said. Mr Madoff never wanted people to give him too much money straight away. "Bernie would tell me, 'Let them start small, and if they're happy the first year or two, they can put it more,' "he said.

It helped enormously that many of these individual investors were just like Mr Madoff – rich, elderly Jewish New Yorkers who divided their time between the city and Florida. It helped, too, that he and his wife, Ruth, were well-liked on that social circuit. Keen golfers, Mr Madoff belonged to half a dozen clubs and were known as a friendly, an understated couple who were not interested in social climbing. Perhaps they felt they didn't need to – the Madoffs have at least three homes including a $9 million (£5.8 million) duplex apartment close to Central Park on Manhattan's Upper East Side, another in Roslyn, New York, and a beachfront house in Montauk, Long Island. There may still be a yacht moored somewhere down in the Bahamas. He has a 55-ft fishing boat called Bull on which he loved to take friends and staff.

Mr Madoff, a native New Yorker who went into finance aged 22 with $5,000 (£3,200) raised from a summer job as a beach lifeguard, was a prominent and generous member of the city's Jewish community. He set up a charitable foundation and was a pillar, too, of Wall Street, where he became chairman of the Nasdaq stock exchange.

There was, as countless hapless investors have testified, little about him to arouse suspicion and so much to inspire trust. The lucky few who were allowed to invest with him believed they were being let in on a big secret – which was half right. The secret was that there was no Midas touch to the Madoff investment strategy. His magic, according to the prosecution line, was more that of the Wizard of Oz – pull back the curtain and you would find a little man in a tiny office above his legitimate, art-laden investment firm where there was no computer but a lot of chits written down on pieces of paper.

Ponzi schemes – named after a notorious 1920s American fraudster – can only last as long as there is enough new investment to pay investors who want to withdraw their funds from the scheme. From what prosecutors say Mr Madoff told colleagues, his grip on the operation finally came loose earlier this month when investors asked to redeem $7 billion (£4.6 billion) and he struggled to find it.

Prosecutors have yet to say where all the money went and how much, if any, is left – Did he spend it? Did he lose it on the markets? Is it under his bed? Anything seems possible in this case. Nor will they spell out exactly how long it lasted although they have said it was years and some have speculated that it may have been decades. However, it certainly unravelled fast.

His arrest at 8.30am last Thursday by two FBI agents who turned up at his Manhattan apartment came just hours after he had come clean about everything to his sons, Andrew and Mark, said one of their lawyers. Mark, 42, ran the proprietary trading business and Andrew, 40, was one of its directors. Their lawyers insist that neither, who had worked their lives in the family business, had any knowledge of what their father was doing.

According to prosecutors, he had told two staff – understood to be his sons – that he was "finished", that his business was "just one big lie" and "basically, a giant Ponzi scheme" that had been insolvent for years.

Charged with a single count of securities fraud, as well as a separate civil fraud lawsuit, he has been freed on $10 million (£6.5 million) bail.

As the blame game deepens over responsibility for the mess, it has emerged that Mr Madoff has weathered several investigations. In 1992, he faced regulatory scrutiny after the SEC sued two Florida accountants for allegedly raising $441 million (£288 million) while selling unregistered securities. The SEC determined that the money had been accounted for and didn't accuse Mr Madoff of wrongdoing.

And his uncanny ability to give a steady 10 per cent or so return to investors had raised eyebrows at least once before. But hedge fund managers never like to discuss their strategies and Mr Madoff's reticence – "It's a proprietary strategy. I can't go into it in great deal", he said seven years ago – was never seen as that unusual.