Everybody who has been involved in scientific research knows that it can be very difficult to publish negative results. Knocking out a gene and finding a severely altered mouse (and thereby confirming the gene's importance) can net you a paper in a high-profile journal; knocking it out and seeing nothing can make it really difficult to publish anything. Some researchers have now performed a geographic analysis of positive and negative results, and they found that the rate of positive results is highest in states with the most productive (and hence the most competitive) research environments.
The difficulty of publishing negative results, those that indicate a hypothesis that doesn't hold water in light of relevant data, can cause significant problems for the scientific community. If the information isn't out there, researchers might waste time chasing after experiments that are predicated on these hypotheses, or ignore alternative ideas. So ensuring that negative data eventually reaches the scientific community is essential.
The authors of the new study proposed their own hypothesis on the publication of negative results: researchers in a competitive environment, who are most sensitive to the "publish or perish" mentality that prevails in the sciences, would be less likely to publish papers that describe negative results.
To test the idea, they obtained data from the National Science Foundation on the number of researchers per capita in each state, and then randomly selected research papers that contained the phrase "test* the hypothes*". Those papers were characterized as either confirming (positive result) or rejecting (negative result) the hypothesis. To link the papers to the geographic data, the researchers used the address of the corresponding author, who is responsible for getting the paper to the journal and answering any further inquiries on it.
The end conclusion of the analysis is that "those based in US states where researchers publish more papers per capita were significantly more likely to report positive results, independently of their discipline." In other words, as local competition increases, the fraction of papers that confirmed a hypothesis went up.
The authors looked at a number of factors that could confound the effect—the total number of PhDs per capita, total publication output per state, and R&D expenditure per state—and found no correlation. So, they feel that their hypothesis is well supported by the data that's available. Which makes it seem like the publish-or-perish mentality is seriously distorting the scientific record.
Caveats galore
But even the authors recognize a few factors that they can't control could be distorting their own results. For example, they recognize that it's possible that the most competitive research environments produce more perceptive scientists, who are better at choosing the correct hypothesis to test. The authors also acknowledge that "We cannot exclude the possibility that authors in more productive states simply tend to write the sentence 'test the hypothesis" more often when they get positive results."
That latter possibility is probably more significant than it sounds. In many cases, it's relatively easy to take a set of negative results from a hypothesis-driven experiment and present them as purely descriptive, an indication of how a given experimental system behaves. If that's done, the word "hypothesis" wouldn't appear at all, and the authors of this analysis wouldn't have ever seen the paper.
Another situation that may have a similar impact is when experiments are done to discriminate between two competing hypotheses. The work is likely to confirm one of them—if it's easier to get the work published by focusing on that and downplaying the negative alternative, who wouldn't take that approach?
Based on these possibilities, we could propose an alternate hypothesis: researchers in competitive environments are better at presenting their results in a way that's likely to get them published. The data presented here is consistent with that hypothesis as well.
None of this is to say that negative results aren't a problem for the scientific community. The authors noted that they're such a widely recognized issue that scientists have developed terms for common ways of dealing with them. (The "file-drawer effect," for when researchers put the data aside and then forget about it, and "HARKing," which is shorthand for "Hypothesizing After the Results are Known.") The publish-or-perish attitude comes with its own set of problems, as it tends to deemphasize science education skills, and creates the sort of high-pressure atmosphere that has led to scientific fraud.
But, even though both of these issues are problematic, it's not clear whether the new data is sufficient to link them. The authors themselves raise some significant questions about their interpretations without even going into some of the basic issues with the data—the corresponding author may not always reside at the site where much of the work was performed, and a US state is generally not a fine-grained measure of a research environment.
Fortunately, the authors are careful enough to say that the data supports their hypothesis, rather than confirm it. And, in the process, they neatly avoid the possibility that their own paper would qualify for the selection criteria of any follow-up research they do.
PLoS ONE, 2010. DOI: 10.1371/journal.pone.0010271 (About DOIs).
reader comments
40