Psychology research: hopeless case or pioneering field?

Ever since Brian Nosek and his colleagues first set up the Reproducibility Project in 2011 many psychologists have been twitchy. The aim of the project was simple: recruit an army of experimenters, with the aim of trying to reproduce results from 100 articles published in three well-established journals in 2008. Why would anyone want to do that? Well, because in psychology, as in many other fields, there have been growing concerns that scientific results are often not reproducible, but there was no hard evidence as to how serious a problem it was.

A notable feature of the study was the care taken to liaise with authors of the original studies to ensure methods and materials were comparable. Also the researchers adopted transparent and audited methods for depositing data and summarising results. In addition, all replication studies had to have a sample size large enough to give convincing results – in many cases these were substantially larger than in the original study.

As it turned out, only 36 per cent of studies replicated, in the sense that they found a statistically significant result in the same direction as the original study - substantially less than the 90 per cent that would be expected, given the statistics used. Furthermore, it was shown that this was not explained by some results ‘just missing’ significance.

So, what do we do with this result? Some of the media are responding just in the way that many people feared. The Independent, for instance, headlined its story on the study “A lot of psychology research really is just ‘psycho-babble’”. Researchers are well aware that psychology has had to work hard to overcome a reputation as a ‘soft’ subject, and critics will be delighted to crow over the findings of low levels of reproducibility. However, the crowing might die down when it is pointed out that problems with reproducibility have been raising alarm bells many other areas of science, including some much ‘harder’ subjects. In fact doing a study to quantify reproducibility, it could be argued that Nosek and colleagues are acting as pioneers, documenting a problem so that we can tackle it by developing new approaches in scientific method.

The key issue, of course, is why is reproducibility so much lower than it should be, and what can we do to remedy it. We can learn much from other subject areas that have had their own problems with reproducibility: especially clinical medicine and genetics.

Since 2005, if you want to do a clinical trial of a drug, you are expected to register in advance a protocol explaining what you plan to do, what data will be gathered, how participants will be recruited and how data will be analysed. Why? Quite simply, this prevents researchers from tinkering with their methods or analyses after collecting the data, and so distorting findings in favour of a positive result. There has been growing interest in adopting a similar approach in psychology, and this should lead to a higher rate of reproducibility. The main challenge when introducing such a change is to ensure it does not have unanticipated negative consequences, for instance by stifling creativity or increasing bureaucracy. My personal view is that the advantages will greatly outweigh the disadvantages, and I look forward to seeing how this approach develops in the next few years.

Another field we can learn from is genetics, which went through a bad period around ten to fifteen years ago, when numerous papers were published reporting associations between genetic variants and various traits or diseases, only for it to turn out that they did not replicate in another sample. The problem was that researchers were looking for effects in very large arrays of genes, and so spurious results were very likely. The field has now moved to a point where a finding will only be publishable if it can be shown to replicate in another sample. This has had the interesting effect of forcing researchers to collaborate, rather than competing with each other to be first in the race to a new result. Insistence on replication prior to publication would be feasible in some areas of psychology, but in some fields, this is less realistic, as studies may take years to do and can be extremely expensive. The solution there seems to be to take (albeit to a less extreme degree) the approach adopted in nuclear physics: form teams to work together so that studies can be done with adequate samples. This also has the potential advantage that team-working can ensure a wide range of expertise is available. In some areas, we may need to abandon the current model whereby a principal investigator with a small team publishes numerous small studies, and instead move to a model where a larger group tackles an important question over a longer time period, but does then arrive at a solid, conclusive result.

As Christie Aschwangen has argued: “Science isn’t broken. It’s just a hell of a lot harder than we give it credit for”. Rather than despairing at the findings of Nosek and colleagues, we should take them as a starting point for considering how we can revitalise science, including psychology, by devising new ways of working that will help overcome the challenges of low reproducibility.