by Michael D. Anestis, M.S.
Generally speaking, I like to keep a positive tone on PBB. The focus here is on providing readers with concrete information on mental illness and the research demonstrating the utility of different forms of treatment. Every once in a while, however, I come across a study that makes me want to fling my laptop against the wall and go on a horrifically nerdy rant about the virtues of empirical data, scientific-mindedness, and quality experimental designs. These rants would center on the frustratingly common presence of misinformation and the people whose efforts prevent the world from developing a clearer understanding of mental illness and the effective treatments that are available. Despite my temptation however, I want to keep things civil and instead explain why the study I will describe did not arrive particularly valid conclusions and how, in making the comments they did, the authors have contributed to the perpetuation of misinformation on psychotherapy.
Today's focus is on a study published by William Stiles, Michael Barkham, John Mellor-Clark, and Janice Connell (2008) in the journal Psychological Medicine. Bear in mind that, while I will be openly critical of the study itself as well as the resulting conclusions, this is not meant as a personal attack. I do not know these individuals, nor do I question their intentions or intelligence. I simply disagree with their work and feel compelled to present an alternative view of their findings. In the study, the authors wanted to examine whether clients in a primary care setting benefited more from one particular form of treatment. Specifically, they were interested in comparing cognitive-behavioral therapy (CBT), psychodynamic therapy, and client centered therapy to one another as stand-alone therapies as well as comparing them to the combination of each therapy and another therapeutic approache (e.g., art therapy, supportive therapy).
The motivation for this study was to test the Dodo Bird Hypothesis. Named to reflect the Lewis Carroll line from Alice in Wonderland, "everybody has won, and all must have prizes," the Dodo Bird Hypothesis purports that all therapeutic interventions produce equivalent outcomes, with an average finding of generally positive results. Perhaps in another post I will describe all of the details of the study upon which this premise is based, but suffice to say that the conclusion is highly controversial and based, in large part, on many of the flaws that I will describe from the Stiles et al. (2008) study.
Okay...on to the study itself. Over a three-year period from 2002 through 2005, the authors collected data from 12,162 patients who received psychological treatment through a variety of primary-care facilities in the United Kingdom. This, quite obviously, is an impressively large sample, although this reflects only the clients who stayed through all of treatment and completed both pre and post-treatment measures (the total number of eligible participants was 33,587). The therapist for each client indicated which type(s) of therapy was used throughout treatment. Each participant was administered the CORE-OM, a self-report measure comprised of 34 items assessing a variety of characteristics of well-being (e.g., depression, anxiety, trauma, general functioning, close relationships, social relationships, risk to self), with items rated on a 0-4 scale. On this measure, a concept such as depression is thus measured through a single or small selection of items rather than a number of items that directly measure all of the symptoms of the syndrome. Additionally, the measure yields a single outcome variable, meaning that risk to self and social relationships, for example, are seen as equivalent even though it is clearly more important for a therapeutic intervention to decrease risk of death by suicide than to improve the client's social life (which is not to dismiss the importance of strong relationships). Click on the CORE-OM link above for a more detailed description and an opportunity to download the measure itself.
As the authors anticipated, all forms of therapies, whether the therapist used a single approach (e.g., CBT) or a combination (e.g., CBT plus art therapy) yielded an overall positive result with respect to improvement on the CORE-OM. Additionally, consistent with the authors' hypotheses, no differences were found between groups on overall outcomes. The authors looked at this data and concluded that the Dodo Bird hypothesis was, in fact, validated. Now let me explain why this conclusion was faulty.
Who was treated?
One pivotal question in any treatment study is who is being treated. Are clients coming in for the treatment of depression? Panic disorder? Borderline personality disorder? An autism spectrum disorder? In this study, there was no answer to this question. No diagnoses were reported and, if any were given, it is unclear how those diagnostic decisions were made. So, what we are looking at here is a study that wanted to see whether any one treatment is particularly good at treating a disparate set of outcomes in all clients, whether or not they are presenting with diagnoses with any overlap whatsoever.
Let me give a non-psychological example to clarify what I mean. Let's say you wanted to see if chemotherapy or Icy-Hot is better. Not better at one particular thing, mind you. Just better. Then, let's assume we recruit a sample of people with different ailments. Some of these individuals have cancer, some have sore muscles, and some were recently fired and reported feeling depressed. In the end, neither chemotherapy nor Icy-Hot proves to be better than the other at treating everything in everyone. Does this mean that they are equivalent? Of course not. Chemotherapy is designed to be part of the treatment of many forms of cancer whereas Icy-Hot is good in treating sore muscles. Neither of the two is particularly useful in treating depression. Quite clearly, some treatments are designed to perform particular functions and not others, at least not without modifications to fit the different presenting problem.
Back to this study. Participants in this sample presented with varying amounts of symptoms of depression, anxiety, psychosis, personality disorders, learning difficulties, eating disorders, physical problems, interpersonal problems, and other variables. The authors did not report mean levels of any of these variables, nor did they examine whether particular treatments outperformed other treatments for specific disorders. In fact, we don't know what percentage of the sample - if any - actually met criteria for any DSM-IV-TR mental illnesses prior to or after treatment. As such, we have no idea whether any of these treatments were better than any others at helping clients no longer meet diagnostic criteria. Fortunately, while that data is not available in this study, it is widely available in other published accounts (read our CBT articles, for example), as well as the APA Division 12 website detailing the empirical status of specific treatments with respect to specific disorders. In other words, if you click on that APA link, you can see the degree to which each of these treatments helps address specific problems rather than looking at the results of a broader question akin to the chemotherapy versus Icy-Hot scenario outlined above.
Medication
It is only mentioned in one line in the method section, but a substantial number of participants in Stiles et al (2008) study reported taking psychotropic medication (e.g., antidepressants) at the onset of therapy. No effort was made to have clients cease using their medication and no control group existed. In other words, there is no reason to believe that medication did not account for a substantial amount of the treatment effects, completely confounding the results.
Interpreting equivalence
Even considering the information mentioned above, there is utility in examining these effects in primary care settings as the authors did in this study. After all, primary care settings afford us an opportunity to see to what degree results from research translate to "real world" practice. Now, in prior PBB articles, we have detailed multiple studies demonstrating that findings in research settings do, in fact, translate to "real world" practice (a phrase that, in an of itself, is annoyingly condescending and reflects an absurd view of the value of research). In fact, Craig Bryan of the United States Air Force even contributed an article demonstrating incredible results for the use of empirically supported treatments in a primary care setting. Nonetheless, let's consider the authors' conclusions here.
The authors saw that there was no difference between groups and assumed that this reflects a lack of true difference in the utility of the treatments. Nowhere in the paper, however, did they mention the possibility that the equivalence might reflect a flaw in the implementation of therapy in primary care settings. In other words, given that this sample did not bother to even report diagnoses, never mind the procedures through which diagnostic decisions were made, it seems more than reasonable to wonder whether the therapists were adequately trained in the use of any of these treatments and effectively followed the proper protocols, basing their treatment choice on data reflecting that the treatment held promise for the presenting problem. We can look at that and say "well, that's the real world so we have to just accept it" or we can look at that and say "wow...in the real world, we are administering these procedures so poorly that we can not produce the proper results. We might need to change how we do things in primary care." Think of it this way: when we read a book and don't understand it, it could mean that the book was written poorly or...alternatively...it might simply reflect that we did not understand a perfectly well written book. Given that an absurdly large amount of research has shown that different treatments perform better than others in the treatment of specific mental illnesses, it seems a bit odd for the authors to look at this data and assume the problem was the treatments themselves.
As an example of how conclusions like this can be problematic, consider a recent meta-analytic review comparing short-term psychodynamic therapy to CBT in the treatment of depression (Leichsenring et al., 2004). The authors looked at nine studies and found no difference between the two treatment approaches. Here again, the authors reported that the Dodo bird hypothesis had been supported, but yet again, the authors also looked at all disorders at the same time and, perhaps even more importantly, provided no data on the degree to which the therapists in the studies properly administered therapy. In response, Sunil Bhar and Aaron Beck (2009) examined those same 9 studies and found that not a single one provided adequate information to determine whether treatment protocols were properly administered. Of the studies that provided at least minimal data on this point, a substantial number met criteria for "inadequate" administration of treatment protocols. Flawed research that makes jarring conclusions completely contradictory to mountains of evidence are harmful. They are not simply incorrect and frustrating for a clinician and researcher such as myself - they actually perpetuate misinformation and serve as an obstacle between those who need help and those who provide the right form of treatment.
Yeah, yeah...but the research is biased because it is all about CBT
This retort is problematic for a variety of reasons. First, the research is not all about CBT. Dialectical behavior therapy (DBT) has been shown to be the treatment of choice for borderline personality disorder. Interpersonal psychotherapy has been shown to be effective in the treatment of depression and bulimia nervosa. We even presented some preliminary evidence for the utility of time-limited manualized psychodynamic therapy in the treatment of generalized anxiety disorder in an earlier PBB article. There are plenty of other examples of times when CBT is not the most well supported or only well supported treatment option for a disorder, including several instances in which pharmacological interventions are a better or equivalently strong choice.
Secondly, while there is more research on CBT, that is because folks who believe in the utility of CBT are more inclined to test their hypotheses rather than assuming their philosophy is best and citing anecdotal evidence to support their case. Nothing is preventing people from researching other approaches. There is no conspiracy to prop up CBT and shun other approaches. If the data show strong results for other approaches, that is fantastic, as it would mean that there are more options available. In the meantime, however, it is entirely unethical to make assumptions without evidence and it is absurd to dismiss the evidence supporting CBT as a superior choice for a variety of mental illnesses because there is so much of it. Publishing data with enormous confounds and then glossing over those weaknesses is simply unacceptable.
If you would like to learn more about empirically supported treatment, we recommend the following resources, all of which are available through our online store:
A Guide to Treatments that Work
by Peter Nathan and Jack Gorman
Feeling Good: The New Mood Therapy
by David Burns
Cognitive Therapy of Depression
by Aaron Beck, John Rush, Brian Shaw, and Gary Emery
Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University.
