"Everybody has won and all must have prizes" - The quote from Alice in Wonderland that inspired the phrase "Dodo Bird Hypothesis"
The Dodo Bird hypothesis, the argument that all treatments for mental illness produce equivalent results, is based upon a series of meta-analyses conducted over the past several decades. These studies involve a number of rather important flaws and, in the coming weeks, we have guest articles planned that will cover these flaws and provide data that support an entirely different conclusion: that particular treatments perform particularly well for particular diagnoses. In the meantime, if for no other reason than I am enjoying the conversation on this matter and excited to see it continue without interruption, I would like to discuss a paper published in 1997 by Paul Crits-Christoph of the University of Pennsylvania in Psychological Bulletin. Importantly, the goal here is not to convince you of one point through compelling language or to question the motives or intelligence of those who believe differently than those of us at PBB. Instead, we hope to provide you with a full description of our side and point you towards data that will allow you to make an informed decision one way or the other. On a side note, from time to time you might see the Dodo Bird hypothesis referred to as the "Dodo Bird verdict" (including in the title of the Crits-Christoph article). Given the substantial amount of disagreement on the matter and the compelling evidence that it is a fallacious conclusion, I choose to stick with "hypothesis."
Crits-Christoph's paper was not an empirical study per se, but rather a rebuttal to the work of Wampold and colleagues (1997), who published what has become one of the most revered scholarly works for individuals who oppose the movement for empirically supported treatments (ESTs). Crits-Christpoh (1997) offered a critical examination of the data in Wampold et al's (1997) study in an effort to determine the degree to which people were justified in arriving at particular conclusions. Wampold et al (1997) conducted a meta-analysis of studies over the previous several decades in which multiple treatments were compared to one another in an effort to determine to what degree treatments tend to produce equivalent results. Their data revealed that, when averaging across treatments and across studies, results were, in fact, equivalent. Despite admitting that it would be "unwarranted to conclude from this study that all therapies are equally effective with all disorders," (p.210), Wampold and colleagues nonetheless felt justified in saying that the findings had "profound implications for research and practice." (p.211). Undoubtedly, the findings have, in fact, had a profound impact on many individuals who do not believe in ESTs; however, the question remains whether the conviction on the part of such individuals is justified given the nature of Wampold et al's (1997) actual work. In other words, if we look at the data rather than listening to the advocates for the study's conclusions, can we find a logical path from results to subsequent beliefs?
Now, before going any further, it is important that everyone understand who the author of the critique actually is. Paul Crits-Christoph is a prolific and well-respected researcher who also happens to be a proponent of psychodynamic theory and therapy (among other approaches). Why is that important? Well, quite often, when we discuss this topic, a popular retort is that ESTs are simply an effort to force everyone to engage in cognitive behavioral therapy (CBT). A quick look at the source of the criticism here should be enough to deflate that argument completely. Supporters of ESTs are not attempting to force everyone practice CBT. Instead, they are attempting determine which therapies produce valid and reliable change on a number of specific outcome measures and maintain those benefits at long term follow-up. In other words, they simply demand accountability and a devotion to scientific methods. CBT has been the subject of more research than other treatment modalities, but the wealth of support for its impact on numerous diagnoses and the dearth of research on many other approaches (or combinations of approaches in "eclectic" practices) does not represent a weakness on the part of CBT or a conspiracy on the part of the scientific community to exaggerate the impact of one particular approach. Everyone is free to research their own approach and report the results.
In my coverage of Crits-Christoph's (1997) rebuttal to Wampold et al (1997), I will not cover every single point in the paper, so I highly recommend reading the original articles to ensure that you are able to consider the entirety of both papers (click on our References button to see the entire citations).
Methodological Problems with Wampold et al (1997)
One of the primary critiques that Crits-Christoph (1997) and subsequent authors have voiced regarding the Wampold et al (1997) meta-analysis was the original authors' decision to average across all outcomes within each study. In a given study, researchers measure a wide number of outcomes. For instance, in a study on depression, a researcher may be interested in changes in depression symptoms, anxiety symptoms, personality features, occupational functioning, overall quality of life, interpersonal functioning, and many other variables. Generally speaking, however, participants in such studies are not presenting for struggles in each of these areas. Instead, there is generally a particularly problematic area (primary outcome) and a number of additional areas in which improvement would be welcome (secondary outcomes). Building off of this further, most ESTs are designed to address particular problems - specifically, they aim to help clients reduce their symptoms of particular mental illnesses and help them prevent those symptoms from returning in the future. When a treatment does not successfully impact all other outcomes, the secondary outcomes are then addressed in subsequent treatment should the client desire it. When researchers present their data, they discuss each outcome separately, focusing particularly upon its ability to address the primary outcome, but also attending to the secondary outcomes.
In the Wampold et al (1997) study, the authors averaged the results across all outcomes to form one variable. In doing this, the authors were essentially stating that a treatment for depression that effectively reduces symptoms of depression but does not change a secondary outcome is equal to a depression treatment that does not reduce symptoms of depression, but does impact a secondary outcome. Now, without question, secondary outcomes are important and it would be unwise to focus solely on symptom reduction, but assuming equality across outcomes is not a reasonable way to broaden our focus. It simply creates a false equality that is likely to artificially influence results of a study like Wampold et al (1997).
Did Wampold et al (1997) truly confirm the Dodo Bird hypothesis?
Of the 114 studies that Wampold et al (1997) chose to include in their analyses (and keep in mind, that when conducting a meta-analysis, the authors choose which studies to include, based upon criteria they themselves typically develop), 51 (approximately 45%) included participants seeking help for an actual mental illness listed in the DSM-IV. In other words, in a meta-analysis attempting to determine the degree to which different treatments impact mental illness, 55% of the studies that were analyzed did not actually involve the treatment of mental illness. Furthermore, 35% of the articles utilized undergraduates as the participants in their samples - most of whom were recruited specifically for the study itself. Given the tendency for opponents of ESTs to say that the results of scientific inquiry are irrelevant because they somehow do not translate to "real world" clinical practice (despite overwhelming evidence to the contrary), it seems a bit problematic to place substantial weight on the conclusions of a study that, in large part, looked at the impact of various therapies on undergraduates who did not meet criteria for any DSM-IV mental illness.
Taking this a step further, 57 (50%) of the studies involved the treatment of anxiety, not exactly a well-rounded look at all mental illnesses. Even more startling, Crits-Christoph counted 79 studies (69%) that compared individual forms of CBT (e.g., cognitive therapy, desensitization, exposure, relaxation, skills training, assertion training) to one another. In other words, in a study that so many individuals use to justify the claim that all forms of therapy work equally well in the treatment of all mental illnesses, the authors, in large part, simply found that different forms of CBT work equally well in the treatment of anxiety, often in undergraduate samples. Now, keep in mind that some studies more recent than Wampold et al (1997) have found that different forms of CBT actually produce different effects for particular anxiety disorders; however, even if that were not the case, the implications of this study remain questionable. How can we assume that, because different forms of CBT work equally well for anxiety, all treatments work equally well for all diagnoses?
What about the studies that were not comparing different forms of CBT to one another? Crits-Christoph counted 29 (25%) out of the 114 studies that involved comparisons of two therapies (at least one of which was not a form of CBT) with a sample that was not comprised entirely of undergraduates. In at least 14 of those 29 studies, meaningful differences were, in fact, found between the treatments and the effect sizes of those differences were almost universally large (see the original paper for detailed descriptions of these studies and their outcomes). Now, certainly there were studies in the sample that did legitimately compare two different forms of therapy in a clinical sample and found equivalent results. Nobody is arguing that this never happens. The problem is, by including such a massive number of comparisons of different forms of CBT for one set of diagnoses, Wampold et al (1997) deflated the differences between different forms of therapy, thereby clouding his results.
Taking this point even further, it is important to consider how little representation was given to a number of popular forms of therapy. Within the entire sample of 114 studies, only four examined psychodynamic therapy, only three examined child or adolescent treatment, and none looked at family therapy. So, despite the fact that many popular treatments received zero or almost zero attention in the study, many individuals have somehow arrived at the conclusion that Wampold et al (1997) vindicated a number of popular therapies that lack empirical support. Now, again, remember that Crist-Christoph himself is a proponent of psychodynamic therapy. He has conducted substantial research on the matter and I highly recommend that readers consult his work. That being said, a scientifically-minded therapist - even one who believes in psychodynamic approaches - can not look at these results and honestly conclude that they tell us anything about the relative merits of psychodynamic therapy across disorders in comparison to other approaches. The conclusions of Crits-Christoph's study represent impressive scientific integrity.
Summary
Crits-Christoph's (1997) critique should not be taken as proof of the utility of ESTs anymore than the Wampold et al (1997) study should be taken as proof of the veracity of the Dodo Bird hypothesis. Instead, it should be seen as a reason for toning down the conclusions based upon Wampold et al (1997) and justification for consulting the mountains of evidence that contradict the conclusions of that study. We have covered a large number of such studies on PBB and will continue to do so as time goes on. If you want to look at the primary source documents yourself and draw informed conclusions based upon the actual data, you can find the complete citations on our references page. If you can not gain access to the papers, I will gladly help you figure out a way to change that.
In the past couple weeks, we have spent a fairly substantial amount of time addressing this and similar issues. We are doing this because the publication of the Baker et al (2009) study and Sharon Begley's Newsweek article have prompted a lot of discussion, much of which seems to be based upon misguided assumptions, including misinterpreting or simply not actually looking at the data in Wampold et al's (1997) study. As we continue to discuss these matters, it is important that we all actually consult the research that we cite. Sometimes, what the authors say in the conclusion or what colleagues say about a paper they read does not actually match what the data tell us. This is, in fact, one of the many benefits of science: a devotion to its principles keeps us from exaggerating our findings, blindly following a particular theory, or dismissing the beliefs of others without empirical evidence to justify our action. I suspect that, as you actually look at the data, you will conclude that the Dodo Bird hypothesis should share the fate of its namesake.
If you would like to learn more about empirically supported treatments for mental illness, we recommend the following items, all of which are available through our online store of scientifically-based psychological resources:
- A Guide to Treatments that Work
by Peter Nathan and Jack Gorman
- Clinical Handbook of Psychological Disorders, Fourth Edition: A Step-by-Step Treatment Manual
by David Barlow
-
Science and Pseudoscience in Clinical Psychology
by Scott Lilienfeld, Steven Lynn, and Jeffrey Lohr
- Cognitive Therapy Techniques: A Practitioner's Guide
by Robert Leahy
Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University
Subscribe in a reader
