Starting this fall, you'll have a new reason to trust the information you find on Wikipedia: An optional feature called "WikiTrust" will color code every word of the encyclopedia based on the reliability of its author and the length of time it has persisted on the page.
More than 60 million people visit the free, open-access encyclopedia each month, searching for knowledge on 12 million pages in 260 languages. But despite its popularity, Wikipedia has long suffered criticism from those who say it's not reliable. Because anyone with an internet connection can contribute, the site is subject to vandalism, bias and misinformation. And edits are anonymous, so there's no easy way to separate credible information from fake content created by vandals.
Now, researchers from the Wiki Lab at the University of California, Santa Cruz have created a system__ __to help users know when to trust Wikipedia—and when to reach for that dusty Encyclopedia Britannica on the shelf. Called WikiTrust, the program assigns a color code to newly edited text using an algorithm that calculates author reputation from the lifespan of their past contributions. It’s based on a simple concept: The longer information persists on the page, the more accurate it's likely to be.
Text from questionable sources starts out with a bright orange background, while text from trusted authors gets a lighter shade. As more people view and edit the new text, it gradually gains more "trust" and turns from orange to white.
"They’ve hit on the fundamentally Darwinian nature of Wikipedia,” said Wikipedia software developer and neuroscientist Virgil Griffith of the California Institute of Technology, who was not involved in the project. “Everyone’s injecting random crap into Wikipedia, and what people agree with more often sticks around. Crap that people don’t like goes away.”
WikiTrust has been available as a MediaWiki extension since November 2008, meaning that anyone who runs a wiki site has been free to download the code and add the feature to their site. Now, the WikiMedia Foundation, the non-profit organization that manages Wikipedia, has decided to demo the WikiTrust feature on the entire encyclopedia, according to Jay Walsh, head of communications. Registered Wikipedia users will be able to click on a "trust info" tab and view the color-coded text, and the researchers expect the gadget to be ready sometime this fall.
"Online collaboration is becoming more and more central to the way in which knowledge is created and assembled worldwide," said computer scientist Luca de Alfaro, who runs the UCSC Wiki Lab and led the WikiTrust project. "There are more and more services that simply cannot exist without some notion of user reputation and trust in the content."
De Alfaro first came up with the idea for a wiki reputation system when he became frustrated with the amount of vandalism on his own wiki site. “I started to think there has to be some way to give people an incentive to behave in a more productive way,” he said.
Collaborative websites such as Amazon.com and eBay already have reputation systems based on user ratings. Many people proposed creating a similar system for Wikipedia, but de Alfaro feared that user-generated ratings might upset Wikipedia’s collaborative atmosphere. He also didn’t want to create more work for editors. “If something works as well as Wikipedia,” de Alfaro said, “you think very hard before proposing to modify it in such a way that everybody has to give comments on everybody else.”
Since Wikipedia already keeps track of every revision, de Alfaro realized he could use that data to create a reputation system independent of human input. “Machines should work for humans and not the other way around,” he said. “So if you can get information without bothering people, via clever algorithms, this is much better.”
The Wiki Lab built its trust tool around the principle that Wikipedia pages tend to improve over time, or at least to move toward consensus. You can measure an author’s trustworthiness by looking at how long his or her edits persist over time, said UCSC graduate student Bo Adler, who developed WikiTrust with de Alfaro and graduate student Ian Pye. “When you add something to Wikipedia and it lasts a long time, you did a good job,” Adler said. “If it gets erased right away, you did a bad job.”
Based on an person’s past contributions, WikiTrust computes a reputation score between zero and nine. When someone makes an edit, the background behind the new text gets shaded orange depending on their reputation: the brighter the orange, the less “trust” the text has. Then when another author edits the page, they essentially vote on the new text. If they like the edit, they’ll keep it, and if not, they’ll revert it. Text that persists will become less orange over time, as more editors give their votes of approval.
“We try to predict when things are going to be deleted,” Adler said. “We want words that are going to be deleted to have a low trust, and words that are not going to be deleted to have a high trust.”
But some critics think there may be hurdles to running the trust tool over the entire site. “This isn't a trivial web architecture design and implementation issue,” said computer scientist Ed Chi of the Palo Alto Research Group, who studies Wikipedia and social cognition. Since WikiTrust assigns a reputation score to every word in every article, running the program in real time will demand significant processing power and several terabytes of extra disk space.
But Wiki Lab researchers say they're already working on making the program more efficient. Using the first version of WikiTrust, it took a regular computer 20 days to process five years of Wikipedia revision data. The latest edition cuts that time to four days, and it can calculate trust ratings for 30 to 40 revisions per second. “That’s on a single machine,” Adler said. “So it’s very practical for us to keep up with Wikipedia.”
In addition, because the product hasn't been tested in large numbers of real users, Chi said he's not convinced that users will find it helpful. Although his reasearch group had similar ideas for a reputation system in Wikipedia, Chi said they decided not to pursue the iea. "I wasn't sure how well it really would work with ordinary users, in terms of their increased ability to detect trust problems and its interference with actual reading tasks," he said.
The Wiki Lab researchers also worried about their product detracting from the Wikipedia experience, so they designed it to be as unobtrusive as possible. Because too much orange text would turn people off, they balanced the need to flag questionable text with the need to keep the page readable. They also hid the gadget in a tab at the top of the screen, so if you don’t want to bother with trust ratings, you don’t have to click on the “trust info” tab.
And don’t go hunting for your own orange ratings: The team decided not to display user reputation to avoid discouraging new users. “Even if you’re a wonderful biologist,” de Alfaro said, “if you haven’t written very much at all on Wikipedia, your reputation will be low.”
WikiTrust can detect most types of questionable content. But when asked whether his gadget measures "truth" on Wikipedia, de Alfaro hesitated. WikiTrust determines trustworthiness based on how many people agree with a particular passage of text, he said, but majority approval doesn’t guarantee truth. “If 20 people are all biased in one way, our tool does not know it,” de Alfaro said. “Our tool can simply measure consensus.”
Adler offers a hypothetical example. “What if Wikipedia was dominated by Nazis?” he said. “Whatever you say about the Holocaust, they’re going to revert you, and then other people are going to come and support those edits rather than your edits.” In that case, WikiTrust would start flagging your Holocaust content as unreliable—no matter how accurate it was.
Trial by consensus sounds sketchy, but majority opinion has nearly always dictated society's definition of truth. A 15th century encyclopedia would have insisted that the sun revolves around the earth. The 1911 edition of the Encyclopedia Britannica asserted that bacteria causes the flu, since viruses hadn’t been discovered yet. So perhaps it’s not a question of whether to trust consensus. Rather, whose consensus do you want to trust: a handful of experts, or thousands of anonymous internet users and a clever computer algorithm?
UPDATE: At the time of publication, the WikiMedia Foundation had confirmed with us that they planned to demo WikiTrust, but could not be reached for further comment. Now they have contacted us and wish to add the following statement, emphasizing that the WikiTrust extension will be optional, and like all additions to Wikipedia, will undergo a community process of testing and evaluation. Here is what they have to say:
"WikiTrust is one of thousands of extensions available within the MediaWiki library. As with other extensions, including flagged revisions, which has been the subject of quite a bit of attention over the past few weeks, extensions are tested and evaluated by members of the volunteer community before being implemented on any of the Foundation's projects, including Wikipedia. Many of these extensions are part of ongoing research by outside developers and volunteers who want to look at ways of improving the quality of content on Wikipedia and other wikis using MediaWiki. When WikiTrust makes its way through testing and analysis it may be made available as an optional tool that Wikipedia users can activate through their user settings. That timeframe has still not been set. The Foundation is also looking at introducing a number of visible trust/quality metric tools, which may include tools familiar to many users, including 'rate this article' tools on Wikipedia pages. These enhancements would be introduced in the spirit of letting readers and editors better understand which articles, facts, or edits need to be reviewed for quality and accuracy."
Image 1: Flickr/bastique. Image 2: Screenshot of a Wikipedia page using WikiTrust; questionable text appears orange. Courtesy UCSC Wiki Lab.
*This story was adapted from a feature by Hadley Leggett published in Science Notes, the annual publication of the UCSC Science Communication Program.
*
See Also:
- Wikipedia Faces Growing Pains
- Fox Claims Wikipedia Whitewashes Obama's Past — Update
- See Who's Editing Wikipedia - Diebold, the CIA, a Campaign
- Encyclopedia Smackdown: Wikipedia Founder Jimmy 'Jimbo' Wales vs ...
- Jan. 15, 2001: Enter Wikipedia, for Better and Worse
- Wikipedia Teaches NIH Scientists Wiki Culture
Follow us on Twitter @wiredscience, and on Facebook.
