Publication bias has inflated efficacy claims of psychotherapy incl CBT 4 depression

[Dolphin]

I wonder how much of an issue this is in the ME/CFS field?

Free full text at: http://bjp.rcpsych.org/content/196/3/173.long or http://bjp.rcpsych.org/content/196/3/173.full.pdf

Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias.

Br J Psychiatry. 2010 Mar;196(3):173-8.

Cuijpers P, Smit F, Bohlmeijer E, Hollon SD, Andersson G.

Source
Department of Clinical Psychology, VU University Amsterdam, Van der Boechorststraat 1, Amsterdam, The Netherlands. p.cuijpers@psy.vu.nl

Abstract

BACKGROUND:
It is not clear whether the effects of cognitive-behavioural therapy and other psychotherapies have been overestimated because of publication bias.

AIMS:
To examine indicators of publication bias in randomised controlled trials of psychotherapy for adult depression.

METHOD:
We examined effect sizes of 117 trials with 175 comparisons between psychotherapy and control conditions. As indicators of publication bias we examined funnel plots, calculated adjusted effect sizes after publication had been taken into account using Duval & Tweedie's procedure, and tested the symmetry of the funnel plots using the Begg & Mazumdar rank correlation test and Egger's test.

RESULTS:
The mean effect size was 0.67, which was reduced after adjustment for publication bias to 0.42 (51 imputed studies). Both Begg & Mazumbar's test and Egger's test were highly significant (P<0.001).

CONCLUSIONS:
The effects of psychotherapy for adult depression seem to be overestimated considerably because of publication bias.

PMID: 20194536 [PubMed - indexed for MEDLINE]

 

[anciendaze]

One of the side-effects of the mental illness that struck George III and precipitated the regency crisis of 1788 was a reexamination of the effectiveness of methods of treatment. You can see a depiction of those treatments in the movie, "The Madness of George III".

A skeptical doctor (whose name escapes me at the moment) made a study of admissions to madhouses. He concluded that 1/3 of those admitted later returned to their previous lives, 1/3 improved, but remained confined, and 1/3 remained unchanged.

Taking this as the null hypothesis for treatment of mental illness you might conclude that 33% would recover regardless of treatment.

 

[Enid]

It's about time the psychology/psychiatric fraternity took a hard look at their so-called scientific findings.

 

[Sean]

As I recall, one of the claims made by the psych school of ME/CFS is that the results of their use of CBT etc are comparable to other disorders that these therapies are used for. This is supposed to help prove that they are effective for ME/CFS, and even that this supports their long standing claim that ME/CFS is a primarily mental disorder. But which I have always thought really just proves how ineffective they are, for both ME/CFS and the other disorders they are used for.

In other words, this whole branch of medicine is riddled with problems of over reliance on subjective assessment, and publication bias. This may help explain why they are so reluctant to bring the ME/CFS psychs to heel, as it will expose the deeper generic methodological problems cutting across all of psychiatry/psychology.

 

[biophile]

Dolphin wrote: Publication bias has inflated efficacy claims of psychotherapy incl CBT 4 depression. I wonder how much of an issue this is in the ME/CFS field? Free full text at: http://bjp.rcpsych.org/content/196/3/173.long or http://bjp.rcpsych.org/content/196/3/173.full.pdf .

Good find. An interesting reminder from the discussion section:

"The overall mean effect size of psychotherapy was 0.67, which corresponds with a number needed to treat (NNT) of 2.75.[30] After adjustment for publication bias the effect size was reduced to 0.42, which corresponds to an NNT of 4.27."

[30] = "Size of treatment effects and their importance to clinical research and practice." | Kraemer HC, Kupfer DJ. | Biol Psychiatry. 2006 Jun 1;59(11):990-6. Epub 2005 Dec 20. | http://www.ncbi.nlm.nih.gov/pubmed/16368078

So a NNT of 7-8 for CBT/GET in PACE would therefore be roughly about d=0.3? Perhaps I should take a look at the Lancet paper again and try to calculate d more directly.

Enid wrote: It's about time the psychology/psychiatric fraternity took a hard look at their so-called scientific findings.

I have noticed a trend that when meta-analyses tackle the issue of psychological factors in illness/disease, the results usually end up sobering the hype. The symbol for much of the biopsychosocial CFS research and commentary should be the gyroscope, the rapid persistent spinning is what keeps holding up an otherwise unstable device due to the angular momentum! Although the biological research hasn't been perfect either.

Sean wrote:

As I recall, one of the claims made by the psych school of ME/CFS is that the results of their use of CBT etc are comparable to other disorders that these therapies are used for. This is supposed to help prove that they are effective for ME/CFS, and even that this supports their long standing claim that ME/CFS is a primarily mental disorder. But which I have always thought really just proves how ineffective they are, for both ME/CFS and the other disorders they are used for.

In other words, this whole branch of medicine is riddled with problems of over reliance on subjective assessment, and publication bias. This may help explain why they are so reluctant to bring the ME/CFS psychs to heel, as it will expose the deeper generic methodological problems cutting across all of psychiatry/psychology.

I don't remember all the details now but I looked into that claim once before and (unsurprisingly) found that in general CBT for CFS is not as effective as CBT for depression (maybe about 50-60% as effective at face value?). So just another CFS assfact? At best without methodological considerations, the former has generally a small to moderate effect and the latter has generally a moderate to large effect. There are rare outliers in both examples reporting much greater success. Except now of course, according to the above mentioned Cuijpers et al 2010 paper posted by Dolphin, when factoring in indications of publication bias, CBT for depression generally does not have large effect but a moderate one. IIRC, systematic reviews into CBT for CFS have also noted possible publication bias, but there aren't as many studies to review and the issue hasn't been thoroughly investigated.

 

[anciendaze]

During the period when I felt my problem must have a psychological basis I talked to a number of people who had recovered from serious psychological problems. Many credited a particular therapist for their recovery. Others were less enthusiastic about the therapist, but mentioned a friend who had remained in communication when others avoided them. Later, questioning still others who gave their therapist credit, I discovered that virtually all those who recovered had such a reliable friend. This stuck in my mind because I was astonished at the range of treatment strategies, including contradictory techniques, offered as psychological therapies.

This raised a question in my mind which I have been unable to answer. One control you are unlikely to see in research on psychological therapy is the involvement of people with empathy, but without any professional training. This might expose the true value of that professional training.

 

[Sean]

One control you are unlikely to see in research on psychological therapy is the involvement of people with empathy, but without any professional training. This might expose the true value of that professional training.

Pretty sure something like this has actually been done with CBT (not for ME/CFS specifically), and the results were not good for the highly trained professional experts, the amateurs did almost as well. So you will not be hearing much about this result from the pros.

Far as I can recall the literature, the only consistent factor in psycho-therapeutic response is the quality of the relationship between therapist and patient, not something that can be taught or subject to formulas or methodologies. The particular mode of therapy is virtually irrelevant. CBT just happens to be the flavour of the decade. It only has the "most evidence" because it has the most studies done. It all gets a bit circular after that.

 

[Esther12]

While the downsides of the over-promotion of CBT for things like depression are less worrying, I think that there are a lot of similarities to the problems found with CFS. There's also no need to deal with the 'they just don't like psychology' side of things.

 

[Esther12]

Looks like the same group with a new paper and a similar finding.

Open access: http://www.ncbi.nlm.nih.gov/pubmed/25062429

Is there an excess of significant findings in published studies of psychotherapy for depression?
Flint J1, Cuijpers P2, Horder J3, Koole SL2, Munafò MR4.
Author information
Abstract
BACKGROUND:
Many studies have examined the efficacy of psychotherapy for major depressive disorder (MDD) but publication bias against null results may exist in this literature. However, to date, the presence of an excess of significant findings in this literature has not been explicitly tested.

METHOD:
We used a database of 1344 articles on the psychological treatment of depression, identified through systematic search in PubMed, PsycINFO, EMBASE and the Cochrane database of randomized trials. From these we identified 149 studies eligible for inclusion that provided 212 comparisons. We tested for an excess of significant findings using the method developed by Ioannidis and Trikalinos (2007), and compared the distribution of p values in this literature with the distribution in the antidepressant literature, where publication bias is known to be operating.

RESULTS:
The average statistical power to detect the effect size indicated by the meta-analysis was 49%. A total of 123 comparisons (58%) reported a statistically significant difference between treatment and control groups, but on the basis of the average power observed, we would only have expected 104 (i.e. 49%) to do so. There was therefore evidence of an excess of significance in this literature (p = 0.010). Similar results were obtained when these analyses were restricted to studies including a cognitive behavioural therapy (CBT) arm. Finally, the distribution of p values for psychotherapy studies resembled that for published antidepressant studies, where publication bias against null results has already been established.

CONCLUSIONS:
The small average size of individual psychotherapy studies is only sufficient to detect large effects. Our results indicate an excess of significant findings relative to what would be expected, given the average statistical power of studies of psychotherapy for major depression.