AdWords is the core of the business of Google, delivering billions of data-targeted ads to browsers every year. Most of the ads are legitimate, but not all of them, and it’s Google’s job to tell which is which. Last year, that meant kicking 224 million ads off the network. The majority were run-of-the-mill web fraud: counterfeit handbags, thinly veiled phishing schemes, the kind of thing any decent spam filter would take out. But in 2010, in the midst of reworking the ad-filtering model, something strange happened. The new model was flagging a lot of otherwise innocuous ads for used cars. Most of the bad products were counterfeit goods, and, as AdWords engineering director David Baker says, "We'd never heard of a counterfeit car." Had they trained AdWords into anti-car prejudice? Was the model simply broken?

The answer turned out to be even stranger. They were real cars, but they weren't really for sale. Scammers were taking pictures of cars on the street, and when a hapless customer showed up a few days later offering money, they'd steal the car and hand it over. By the time the mark realized he had purchased stolen goods, the sellers were long gone, taking his money with them. It's a lucrative scam, and in China, a well-known one — but to anyone looking at the ads, it just looks like one more crop of used-car ads.

For those who study fraud in China, on the other hand, this is far from surprising. "These people are very professional," says Dahui Li, an information systems expert at the University of Minnesota who specializes in Chinese online fraud. In the case of the car scam, he says the offline component is the most important part, as a way to assure skeptical customers that the sale is legit. "Chinese people want to see the product before they pay for it," Li says. "They have to see the car." So the criminal element developed a scheme that could show it to them.

According to Li, the larger problem is the Chinese financial system, which requires every bank-to-bank transaction to be routed through the central government’s banking authority. As a result, anti-fraud measures are usually slower than criminals. Stopping a payment could take as long as three days, by which time the money is usually unrecoverable. Since customers can't trust the banks to stop fraud, they're left to fend for themselves. On Taobao, the Chinese equivalent of eBay, Li sees this effect driving customers to have longer conversations before they trust a seller, testing out every offer to make sure it's legitimate. In other markets, it boils down to common-sense habits like holding the product in your hands before you buy it. And as the sales move offline, so does the fraud. Li describes scams that create fake websites, designed to fool users into thinking they’re on Taobao or another moderated e-commerce site. Even when police are able to follow the credit card trail, criminals are able to stay a few steps ahead. "Until the payment model is fixed, you’re always going to see these scams," Li says.

The surprising thing is how much this supposedly offline fraud caught the attention of Google’s AdWords cops. The team is only looking for fraudulent offers made on the AdWords network, but scams like the car-swapping trick blur the lines between the crimes that happen on the network and the crimes that take place after the fact. More importantly, AdWords’ crime-spotting tools don’t distinguish between online and offline scams. They’re just looking for suspicious behavior. Like many Google products, AdWords' quality control is managed by a massive machine-learning algorithm, similar to the PageRank function that powers Google Search. "There’s no one thing or even a handful of things," says Baker. "It’s thousands of pieces of information in aggregate." That includes IP address, account age, and links to past accounts. After millions of iterations, it’s hard to trace the red flags back to any group of factors, which makes it even harder for bad guys to game the system.

More importantly, it doesn’t take human prejudice into account. Baker and his team weren’t looking for cars or car thieves. But the algorithm saw a pattern of quick buys from new accounts, tied together with larger and more subtle patterns, and deduced something was up. It’s not an airtight system: more than a few valid accounts have had their orders delayed while the team checked them out. But in this case, it was able to reach across continents to suss out a scheme its engineers had never even imagined. Cultural differences could fool the humans, but they couldn’t fool the machine.

At the same time, once Google found the scammers, they couldn’t do any more than kick them off AdWords. The Google team will sometimes consult law enforcement in the fight against fraud, but the final action always falls to the authorities. And when the fraud itself can be plead down to simply taking pictures of someone else's car, stopping the bad guys can be difficult. It makes sense; Google isn’t in the crime-fighting business. But as they try to keep scams from spilling onto their network, they’re getting awfully close.