(Guest blogger Simon McGrath focuses on an important topic and stumbling block for ME/CFS – replication studies (or the lack thereof). The XMRV story presents a somewhat unusual theme; a subject receiving enough study that a consensus (at least to date) has been reached and in relative rapid fashion but a recent blog found a significant number of research efforts are never or are haphazardly followed up on. – Cort)
Replication
Replication of important findings is essential for progress in CFS research (and other research too)
A recent review by Harvard professor Anthony Komaroff cited over 150 findings of either biological abnormalities or evidence of chronic infection in CFS patients, yet almost every one was different, with very few replications. Which is disappointing given that replication of findings is the Gold Standard of scientific research. A good number of the cited reports were from the 1990s, suggesting there had been no follow-up or replication for over a decade. The apparent lack of follow-up of promising findings is probably a significant factor in why so little progress has been made in understanding the pathophysiology of CFS.
The Komaroff review quite rightly pointed out that the evidence was consistent with chronic infection being a cause of some cases of CFS, but also acknowledged the limitations of the data.
The titles and abstracts of cited papers in that review (as well as those in other papers) often contain words like ‘possible’, ‘preliminary’, ‘exploratory’, ‘suggests’ and ‘need confirmation’ – but rarely contain words like ‘validated’, ‘confirmed’ or ‘replicated’, Suzanne Vernon has pithily described this as a ‘one and done’ approach to research. In fields such as biochemistry there are, of course, many preliminary and provisional findings in recent papers – it’s the nature of science – but there are also a respectable number of confirmations (or refutations) of original findings.
It’s only by confirming interesting findings that there can be progress: each validated finding is a building block that other researchers can use to make further progress. Of course, CFS is phenomenally challenging to study, but without rigorously following up the leads that do exist to either prove or discount them, it’s going to be tough to get anywhere.
CFS studies also tend to be quite small, making findings less reliable and less heeded. To a large extent this is because CFS research is woefully underfunded and well-defined CFS cohorts are expensive to assemble. Nevertheless, many, if not most, studies conclude that their findings need replicating on larger samples – and that replication rarely comes.
Other Roadblocks
A look at the last three months of PubMed citations suggested another problem the ME/CFS research field faces…
Small Study Size
Small study size is another almost common problem. Researchers often dismiss studies as unreliable if they have small sample sizes and they’re generally tagged with a ‘preliminary’ label – meaning the data is nothing more than suggestive. Poorly funded ME/CFS researchers, however, often have no choice than to do the best they can with the funds they have.
A recent autoimmune study had 15 patients and an amino acid study contained 11. Some decently sized biological studies did appear; a Japanese infrared spectrum study (n=57), two PHANU studies on natural killer cells (n=28, , a prognosis study (n=47), a blood pressure study (n=68) and an inflammation study (n=144)) all had decent or good numbers of patients
But contrast these with size and quantity of the behavioral studies over the last couple of months; three CBT trials had 63, 105, 120, 123 and 135 patients, a stress management study had 117, a pscyhosocial study had 84, a ‘psychological aspects’ study 103, a ‘disease-related worries’ study had 140, and a co-morbid psychiatric diagnoses study had 640 patients. All of these studies, not surprisingly, appearing in easily accessible electronic format. They had abundant funding and they will get read.
Its easy, given this pattern, to see why ME/CFS fights an uphill battle against a psychological interpretation. Physiologically based studies are not only dwarfed by not only by the number of behaviorally oriented studies but by their sheer size.
Back to Replication
Replication is key, and not just in CFS research – the CFS scientific literature holds a vast array of unconfirmed findings that anyone can pick from to support just about any hypothesis about the illness – leading to confusion and another reason for researchers in more well-defined disorders to discount findings in ME/CFS or to apply the ‘wastebasket’ term to them.
What’s needed to get beyond the ‘wastebasket trap’ is robust replication: repeating the findings on an independent sample of patients and controls, with a substantial sample size, and, ideally, by an independent research group.
More CFS research funding would clearly help, but there’s no guarantee it would lead to more replications, at least not judging by the experience in other, better funded fields.
The lack of replication is a hot topic across life-science research, particularly since John Ioannidis’s landmark paper ‘Why Most Published Research Findings are False’. A recent Nature article noted that a pharmaceutical company’s attempt to replicate 53 published findings of promising drugs managed to fully replicate just 6 (11%) of the findings. Similar work in Germany, at Bayer, found only 25% of preclinical studies could be validated to the point they were worth pursuing. There is even more concern around studies where there are many, many possible links that could arise by chance, as with genetic association studies that look at thousands of genes. One study found that of 166 reported associations between gene variants and common diseases studied 3 or more times, only 6 were replicated consistently.
Gene Expression Studies Illustrate the Need for Replication
The same problem of false positives applies to gene expression studies, which have thrown up many putative associations between expression of specific genes and CFS, but, as yet, no proof. Jonathan Kerr authored several of these studies and he also bravely published research that discounted them too. His PLoS One replication paper last year concluded that, “many of the previously identified reporter genes are study-specific and thus cannot be used as a broad CFS diagnostic”. The problem is that when you look at the expression thousands of different genes it’s highly likely that some of them will appear to correlate with CFS just by chance. If the correlation is by chance, rather than real, it is unlikely to reappear in a replication on an independent sample, which is just what Kerr’s PLoS One study found.
Biomarker Mirage?
A similar problem applies to biomarkers, which have frequently been ‘found’ over the years. One reason so many biological abnormalities appear to identify CFS is that authors look at the results first, then choose a threshold level for the biomarker that best discriminate between controls and patients in their particular sample. This post-hoc tweaking doesn’t mean a whole lot until the finding has been replicated on an independent sample of patients and controls, using the same thresholds as the original study. Again, replication helps to separate the real effects from the statistical artefacts.
A more robust approach is to divide patients into separate exploratory and validation samples, effectively building a replication into the study itself. This approach is often used in gene association studies and was also used in a study from Julia Newton: the diagnostic criterion for autonomic dysfunction in CFS patients established in the phase one (n=40) was shown in an independent phase 2 sample (n=30) to have a positive predictive value of 0.96. This is a much better tactic, though with relatively small patient numbers it still needs confirmation in a larger sample.
A Real, Replicated Bio-abnormality: Natural Killer Cell Functioning
The one shining example of a robustly replicated bio-abnormality is lower natural killer cell cytotoxicity (NKCC) in CFS patients. Natural killer cells protect the body by killing invading organisms and it’s this ability to kill other cells that is reduced in CFS patients. Lower NKCC has been detected at least since 1987, and was most convincingly replicated by Fletcher and Klimas (n=176, p<0.0005 vs controls). There was also a later confirming paper by Brenu, who has now shown these differences are sustained vs controls over time. However, there is still substantial overlap on NKCC levels between individual CFS patients and healthy controls, and as the Komaroff review pointed out, it’s not clear if these findings are specific to CFS – so this bioabnormality may not prove to be such a good biomarker.
Replication Problems with Psychological CFS Research
We’ve focused on biological research but the same lack of replication applies to much psychosocial research too. For instance, some studies have reported spectacular results for CBT for CFS e.g. a study by Deale found physical functioning improved dramatically and another study found 23-44% of CFS patients made a full recovery. Yet a meta-analysis, and the large PACE trial, found only modest gains, and even these self-reported improvements may not be real. But this situation is typical of clinical trials in many fields: small early studies often find impressive results that are not replicated in larger more robust studies, or meta-analyses that look at the data from many studies together.
Reasons Not to Replicate (Why the Low Rate of Replication?)
Most scientists naturally want to pursue their own ideas, so spending time and limited funds on replication studies will never be popular, particularly as journals are often reluctant to publish replications on the grounds they say are not ‘original’ contributions to science. And there may, in fact, be many failed replication attempts that are simply never published, in what is known as the ‘File Drawer Effect’: researchers don’t bother writing up and publishing negative results. Psych File Drawer has been set up to encourage and publish replication attempts in Experimental Psychology. Maybe CFS needs its own File Drawer project. Perhaps there have been unsuccessful – and unpublished – replication attempts of some of the many findings reported in the Komaroff review. If so, other researchers need to know.
Also, replication studies are not a good way to make friends, as XMRV has shown. ‘You’re Wrong’ never goes down well, whether aimed at the original study or the replication attempts. The Reproducibility Project is a bold move to systematically measure reproducibility by attempting to replicate all experiments published during 2008 in three different psychology journals. Brian Nosek, a professor of psychology who helps run the project, was warned not to go ahead “because psychology is under threat and this could look bad”, but pressed ahead anyway – so there may be some very nervous authors out there.
Promising Future?
Over the past two decades I’ve been illthere have been many seemingly-exciting new discoveries in CFS, but little real progress. What’s encouraging now is that there appears to be a substantial and important shift towards more replication studies in CFS research.
The finding that CFS patients have lower NKCC has already been robustly replicated. The three other most important findings in recent years are probably XMRV, Rituximab and the Lights’ gene expression on moderate exercise studies. There have already been numerous attempts to replicate the XMRV findings, with the Lipkin’s ‘Mother of All Replication Studies’ due to report soon. Fluge & Mella, the authors of the amazing but small Rituximab study, are planning a large multi-centre trial to attempt replication of their initial findings, while Dr’s Enlander and Bell are planning a replication study in the US too. And the Lights are being funded by the NIH to validate the results of their original study with new patients. Hopefully attempts to replicate important findings will become the norm in CFS research.
A new focus on replication of promising findings won’t guarantee success, but is likely to help greatly as research will be increasingly focused on meaningful rather than simply intriguing findings.
(Simon McGrath took a biochemistry degree in the eighties planning to go into research, but then decided to work for a charity instead. He got ill in 1994. Between then and now he’s been hospitalised, bedbound for several years and is currently just largely housebound. He posts fairly regularly on the forums under a psuedonym.)
Support Phoenix Rising
(Why not support Ph0enix Rising with a $5 , $10, $15, $20 or more monthly ‘subscription’. Click on the Subscribe button and look on the right hand side of the page for options :))