After the Commons science and technology committee report this week, and the stupidity of "we bring you both sides" media coverage, you are bored with homeopathy. So am I, but it gives a simple window into the wider disasters in medicine.
Homeopathy is a small sector of the pharmaceutical industry, a few sugar pill companies worth a couple of billion pounds a year in Europe. Overall, trials show their pills perform no better than placebo. All claims to the contrary are rubbish, but rubbish tolerated by plenty of MPs, huge swaths of the media, a fair few GPs and, most worryingly, the Medicines and Healthcare Regulatory Authority, who permit homeopathy pill companies to list diseases they say they can treat on the bottle, with no requirement that they provide evidence their treatments work.
This doesn't only tell you a story about homeopathy; this tells you how doctors, politicians, the media and regulators deal with the issue of evidence in medicine.
How closely do the great and the good, for example, scrutinise the promotional material for medical drugs? The latest paper looking at this question is published this month. Researchers in Holland went through the world's biggest medical journals – the Journal of the American Medical Association, Lancet, the New England Journal of Medicine, and so on – between 2003 and 2005. Adverts were included, once each, if they made a claim about the effect of a drug. For all the claims, they checked the references, found the trials referred to, and gave them out to easily exploited assessors: 250 medical students who'd just finished their evidence-based medicine teaching.
Each student independently assessed two trials, and associated adverts, following a questionnaire and a well-established scoring system to assess quality of trials. Scores were given for factors including:
Whether the method of randomly assigning patients to one treatment or another was adequate, and clearly described.
Whether patients could know which treatment they were getting.
Whether drop-outs were appropriately included in the analysis, and so on.
These are good measures of whether a trial is a fair test of a treatment.
By now you will rightly be worrying that medical students – although cheap and easy to come by – are not reliable raters, so you will be pleased to hear that each trial was scored by between two and six students, and any discrepancy reviewed by a panel of four academics.
The results were abysmal. Only half of the claims in the adverts were supported by the specific trials referenced and, of all the trials, only 55% got a score of "high quality". Overall, only 39.2% of these adverts referenced a high-quality trial which supported their claim.
This is not the first time such a study has been conducted. Villanueva and colleagues, in 2003, published a paper in the Lancet assessing claims for cardiac medication adverts in six Spanish medical journals: of the 102 references they could trace, 44% did not support the promotional statement. Similar results have been found in psychiatric drug adverts, and in the field of rheumatology.
To offset any suggestion that I am cherry-picking, a review in the Public Library of Science's open access journal PLoS One found 24 similar studies, and overall only 67% of the claims in adverts were supported by a systematic review, a meta-analysis or a randomised control trial.
Quacks see shortcomings in medicine as justification for their own dubious behaviour, but the horror is this: homeopathy is the tip of the iceberg. It is the simplest story of how bad things are, how dumb doctors and politicians can be about evidence, and how lame regulation has become. But it is only the most obvious illustration of the fearsome depths into which these problems extend. We are in very big trouble.