by Michael D. Anestis, M.S.
In the past, we have devoted a substantial amount of attention to the ways in which we determine whether or not a particular form of therapy for a particular mental illness has established empirical support (evidence that it actually leads to improved outcomes). We have also spent (and will continue to spend) a significant amount of time discussing and rebutting the arguments put forth by some researchers that, in fact, all forms of psychotherapy work equally well for all disorders, a perspective known as the "dodo bird hypothesis." I encourage you to share your thoughts on those articles and to add your voice to that important discussion, but today I would like to shift gears a little bit and discuss what I believe to be one of the most valuable options we have available for clarifying this debate: comparative treatment studies.
Unfortunately, more often than not, research that examines the efficacy (how well does a treatment work in a highly controlled research setting) and effectiveness (how well does a treatment work in a "real world" setting) of psychotherapy compares one treatment to a waitlist or a "treatment as usual" condition, in which a mix of techniques are used in a generally informal and unstructured manner. These studies provide us with valuable information in that they demonstrate that one particular form of therapy is better than nothing (or almost nothing) in the treatment of a specific diagnosis or set of diagnoses. What these trials do not tell us, however, is whether that particular form of treatment is more or less useful than another form of treatment that also outperformed a waitlist or treatment as usual condition.
Because most treatment trials compare one form of treatment to treatment as usual or no treatment at all, we are left in a situation in which we must be creative in developing a hierarchy of treatments to guide clinicians in determining how to address a particular client's presenting problem(s). Sometimes this is easy, as only one treatment or set of treatments has any support whatsoever. Other times, it is easy because some comparative treatment trials do exist and those trials tell us which treatments tend to produce the best results. Other times, however, we are left without such clear cut evidence and must turn to statistical methods theorized to be capable of answering the question even in the absence of trials that specifically test the hypotheses in question. In a meta-analysis, a series of trials are combined and the outcomes are compared to one another to tell us, on average, how powerful the findings are one way or another. Think of it this way: in a regular study, a group of individuals each supply data and we then analyze the entire group together to see what the overall trend was. In a meta-analysis on the other hand, a group of studies are combined, so in this sense, the studies serve the role of individuals. Here's where we have a problem though. In a regular study, each participant does the same thing. In other words, folks in the treatment group all receive the same treatment and, if there are differences between those people, we control for those differences and examine whether they impact results. In a meta-analysis, this is not the case. Authors utilize a degree of subjectivity in selecting which studies to include and then they make comparisons across studies that examined different questions with different outcome variables. So, in order to determine whether cognitive behavioral therapy (CBT) is more effective in treating generalized anxiety disorder (GAD) than is psychodynamic therapy, for instance, the author might compare how much better CBT did than treatment as usual for GAD to how much better psychodynamic therapy did than treatment as usual for GAD. Because the two studies likely used different outcomes, the authors will combine different variables and create their own outcome based upon their own definition, regardless of how similar those outcomes might actually be. If one or two trials exist that directly compare the two treatments to one another, these might left out of the meta-analysis (even if they demonstrated that one is better than the other) or they might be combined with studies in which the treatments were not directly compared, resulting in a confused mix of data.
For you sports fans out there, think of it this way:
Imagine there are two undefeated teams at the end of a season and we are trying to determine who should be declared the champion. Unfortunately, there is no championship game scheduled, so we simply have to name a champion. Team A and Team B both played 10 games and won all of them. During the course of the season, Team A and Team B only played one similar opponent: Team C. One way we could determine the champion would be to say "who beat Team C by more points?" For a variety of reasons, this is problematic. Maybe weather conditions were different in those two games. Maybe Team C played particularly well or particularly poorly in one game, such that the game itself was not representative of what normally happens. Regardless, you can likely see why simply comparing how Team A and Team B did against Team C is not a satisfactory way of determining the winner.
Let's bring this point back to research on psychotherapy. If Team A is CBT, Team B is psychodynamic therapy, and Team C is treatment as usual, the parallel starts to become more clear. Meta-analysis - the tool used by dodo bird hypothesis proponents such as Bruce Wampold - is often forced to rely on comparisons in which they can only consider whether Team A vs. Team C was more of a massacre than was Team B vs. Team C. The resulting findings are thus not particularly informative. They frequently find that many forms of psychotherapy are better than nothing (or treatment as usual) and fail to find meaningful differences between approaches.
You might read that and think "maybe they find that because there really are no meaningful differences." That's a great question and, in a number of prior articles, we have explained in detail why that is actually not the case (click here for an example). When two active treatments are compared to one another, the results very often reveal stark differences in results (both short and long term). Using our CBT vs psychodynamic therapy for GAD example, consider a study we discussed back in July. The authors compared these two treatments and found that both resulted in significant improvement across a wide variety of outcome measures. This is good news for both treatments, but the analyses did not end there. The results also indicated that CBT resulted in significantly greater improvements in depression symptoms, trait anxiety symptoms, and worry, the three sets of symptoms that are most characteristic of GAD. In other words, both treatments had solid results, but CBT was substantially better at treating the core problems associated with the condition. Now, this is only one study, so the results should be kept in perspective; however, this type of study tells us so much more than does a meta-analysis in which the two treatments in question are never actually compared in the same sample using the same agreed upon outcome measures. This study does not say that CBT for GAD is perfect and psychodynamic therapy for GAD is useless, but it does give us reason to consider prioritizing CBT as the frontline treatment relative to psychodynamic therapy for this particular disorder, with those who don't respond well to CBT potentially switching to a psychodynamic approach later in therapy.
So what's the solution? Sadly, it's not as simple as one would hope. A well-run randomly controlled trial in which the therapists are well trained in the therapies in question, treatment is administered properly, fidelity to the treatment is assessed rigorously, and assessments of improvement are made by individuals who are unaware of which form of treatment each participant received are expensive and time-consuming. The long-term answer is, of course, to prioritize the initiation of studies capable of answering these questions in exactly this manner. The short-term answer, on the other hand, seems to be very different. Some studies like this exist, but they are under-publicized. Attention needs to be drawn towards them and away from results based upon non-direct comparisons. Additionally, some of these studies are poorly done and the conclusions of the authors overstep the bounds of what the data actually are capable of telling us, so the publicity needs to be accurate and to clearly explain to people how conclusions were drawn and why certain results might be open to question. We certainly try to do this here on PBB, but more and more sources need to become available for this same service. Quite often, doctors speak about their findings in a convincing but entirely inaccurate manner and it is difficult for readers/listeners to know what that is the case unless they are well versed in data analysis and have access to the studies themselves. There needs to be an effort made by scientifically-minded individuals to present this information to the public in a clear and honest manner and in a way that allows the public to fact check claims. With respect to PBB, you can do this by clicking on our references page and looking up the studies we discuss. Due to copyright and intellectual property infringement concerns, we can not make those studies freely accessible through PBB, but you can often find them at no cost online through a university or library computer network and we always welcome discussion of the data in our comments section.
Perhaps in the end, as more comparative treatment studies are conducted, we will find that most therapies are, in fact, equivalent for most disorders, although the evidence we have thus far definitively contradicts that as a likely result. A more likely outcome is that we will find equivalence in some treatments (as we have for CBT and interpersonal psychotherapy for depression) and not for others (e.g., the CBT versus psychodynamic therapy for GAD example). Some clinicians and proponents of particular forms of therapy will continue to contend that research is not capable of answering these questions, but their point will continue to ring hollow. If somebody believes that a particular form of therapy is effective for a particular diagnosis or range of diagnoses, it is beholden upon them to produce objective evidence to back up their assertions. Yes, this is difficult to do, but sometimes important things are difficult and people simply need to make the effort necessary to accomplish their goal rather than expecting people to simply take their word for it or to rely on subjective testimonials that may or may not represent the general pattern.
************
If you would like to learn more about the concept of empirically supported treatments for mental illness, we recommend the following book, which is available through our online store for scientifically-based psychological resources:
- A Guide to Treatments that Work
by Peter Nathan and Jack Gorman
Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University
Subscribe in a reader
