by Michael D. Anestis, M.S.
Today's topic is one I've sat on for a while because I suspect it will ruffle some feathers and I wanted to get all of my thoughts organized so that I could write this article tactfully and in a manner that focuses on evidence rather than theory. In late 2009, an article written by Jonathan Shedler of the University of Denver and set to be published this year in the American Psychologist, started making the rounds on listservs and internet forums (Shedler appears to have made the PDF available on his profile, so click on his name above to access the full article for yourself). Numerous people contacted me via email with copies of the document as well. The focus of the article is a review of the literature on the degree to which psychodynamic therapy is an efficacious and effective treatment for mental illnesses as determined by empirical research.
Historically speaking, the perception of psychodynamic therapy has been that it completely avoids research and is practiced only by non-scientifically-minded individuals. This perception is not entirely fair and the Shedler article does a good job of addressing this issue. In the text, Shedler notes that earlier generations of psychoanalysts and psychodynamically oriented psychologists did, in fact, embody an anti-research bias that led them to be discredited amongst the scientific community. Because of this, changes in the psychodynamic approach went largely unnoticed and the original ideas of Sigmund Freud remained associated with the core teachings of the treatment. As it turns out, a fair number of researchers in this most recent generation have worked hard to change this aspect of psychodynamic culture, which is a development that should be applauded by all.
All of that being said, the Shedler article itself, which was hailed as a remarkable moment for psychodynamic theorists and proof that it is not only a valid and empirically supported approach, but in fact is superior to most others (including cognitive behavioral therapy), is full of a number of shockingly profound flaws that render the vast majority of the conclusions drawn by its enthusiasts quite empty (to be fair, Shedler himself tempered his enthusiasm quite a bit in the text). My goal today is to explain what flaws I see in the paper and what evidence I have to support my claims. As always, I welcome a civil discussions amongst both like-minded individuals and those who vehemently disagree with my conclusions. To accomplish this, I will walk you through the different sections of the paper and organize my responses according to that material.
Distinctive Features of Psychodynamic Technique
In this section of the paper, Shedler explains the general approaches used in psychodynamic therapy as it is today rather than as it was in the days of Freud. Although I find myself generally a bit put off by the vagueness inherent in many of the concepts embodied in this theory, the bottom line is that this is a fairly rich description of techniques and a good resource for readers interested in learning more about the theory/therapeutic approach.
How Effective is Psychotherapy in General?
Here, Shedler discusses the voluminous research conducted over generations indicating that psychotherapy, in its various forms, tends to be helpful for individuals with mental illnesses. This point would be difficult to argue against at this point.
How Effective is Psychodynamic Psychotherapy?
At this point, Shedler begins to build his case for the efficacy and effectiveness of psychodynamic therapy. He cites a number of meta-analyses (studies that combine the results of multiple other studies) indicating that, when compared to no treatment at all, psychodynamic therapy produces strong results in the treatment of mental illness. Here is where I start to diverge with Shedler and the article's enthusiasts. As I have mentioned several times before (including earlier this week), I find meta-analyses troubling for a number of reasons. First of all, they often combine very different groups and treatments into a single category in an illogical way that is certain to mask effects and produce misleading results. They also tend to group all outcome variables together in an arbitrary manner that, again, is likely to mask results and overlook important differences between groups on specific outcomes (e.g., depression symptoms versus panic symptoms). Perhaps most importantly, however, is that it is up to the author of a meta-analysis to choose which studies to include. More often than not, researchers try to be fair, but our own biases can lead us to develop inclusion and exclusion rules (and to assess the degree to which a study fits those rules) in such a manner as to increase the odds that we will find the results we want. Case in point: Shedler cites a meta-analysis conducted by Leichsenring, Rabung, and Leibing (2004) in which the authors claimed to demonstrate that psychodynamic therapy has strong empirical support for a number of disorders. This meta-analysis had its strengths in that it only included randomly controlled trials (RCTs) and did not lump interpersonal psychotherapy together with psychodyanmic therapy as some prior meta-analyses had done; however, it also had its clear weaknesses. First of all, the studies included in this meta-analyses included a wide range of diagnoses so, rather than having a large sample to analyze for a particular set of clients, we were left with a cluttered collection of underpowered studies on widely varying individuals. In a situation like that, the question being answered by the data is essentially meaningless. A clearer weakness, however, was the failure to reference contradictory findings in more than a passing line. Specifically, in a 1991 review published in the Journal of Consulting and Clinical Psychology, Svartberg and Stiles found a very different set of results. Short term psychodynamic therapy (STPP) was superior to no treatment at post-treatment, but inferior to alternative psychotherapies (CBT, supportive psychotherapy, etc...) at post-treatment and particularly at 1-year follow-up. This inferiority to alternative treatments was particularly pronounced in the treatment of depression and was strongest when STPP was compared to CBT. Even more startling, however, was that the authors found that, as the quality of the studies improved, the degree to which STPP was better than no treatment at all decreased substantially. So, the less bias and the greater the scientific rigor, the less useful STPP actually appears.
My point above is not that there is no evidence supporting STPP, but rather that it is dangerous to accept the conclusions of a meta-analysis without looking at the actual studies involved (or not involved). Meta-analyses are not objective answers and their conclusions are no less prone to exaggeration and error than those of single study comparative research. I would argue that these risks are substantially higher because so few people actually examine the studies referenced in a meta-analysis, choosing instead to take the meta-results at face value.
Shedler next called attention to a meta-analysis conducted by Leichsenring and Leibing (2003) published in the American Journal of Psychiatry that compared psychodynamic therapy to CBT in the treatment of personality disorders. Here again, the claim was that both treatments were effective, but that psychodynamic therapy had a larger effect size. Putting aside the fact that grouping personality disorders together as though they are a single entity that responds to the same treatment is highly questionable and a practice devoid of any research support, let's take a closer look at what the Leichsenring and Leibing (2003) meta-analysis actually showed us. 8 out of the 11 studies included that compared psychodynamic therapy to a control condition were naturalistic studies rather than RCTs (as compared to 3 of 8 for CBT). Only three studies included in the analyses actually involved a direct comparison of psychodynamic therapy to CBT. Fortunately, all three were RCT's; however, the results of these three studies, again the only ones that actually compare these two treatments to one another, were not at all supportive of the authors' overall conclusions.
In one study, conducted by Liberman and Eckman (1981), "insight oriented therapy" was compared to behavior therapy in the treatment of patients with multiple suicide attempts (no diagnostic information was given here, so it is unclear why this was included as a treatment of personality disorders, by the way). At two-year follow-up, the behavior therapy group exhibited fewer suicide attempts and less suicidal ideation than the group that received the psychodynamically oriented treatment.
In the second study, conducted by Hardy and colleagues (1995), individuals with depression with or without a comorbid cluster C personality disorder were randomized to receive either CBT or psychodynamic therapy. At one year follow-up, individuals who received psychodynamic therapy and originally had both depression and a personality disorder were significantly more depressed than were individuals who did not have a comorbid diagnosis. Those who received CBT saw drastic reductions in depression regardless of personality disorder status. In other words, CBT outperformed psychodynamic therapy for more severe and comorbid cases, which is a bit ironic given that one of the central criticisms levied against CBT and other manualized treatments by psychodynamically oriented psychologists is that they are supposedly inferior in treating such individuals.
In the third study that directly compared CBT to psychodynamic therapy, Woody and colleagues (1985) looked at the treatment of individuals with various combinations of depression, opiate dependence, and antisocial personality disorder. The authors did not report results for each individual treatment, although they indicated that results from earlier phases of treatment revealed no significant differences. This sounds promising except that the authors were looking for group differences between four groups with between 13 and 17 individuals. In order for a statistical analysis to show a significant difference between groups of this size, one treatment would have to absolutely destroy the other one in terms of impact. In other words, a lack of significant finding here does not prove that the treatments are equal anymore than looking at your results on a few random questions on a 300 question exam reveals how well you did on that exam.
So, in the only three trials that directly compared CBT to psychodynamic therapy in the treatment of PDs, two showed CBT to be better and the other at most showed equivalence (although no actual results on that front were given and it was so severely underpowered that its findings on this point would be impossible to interpret anyway). Also, remember that only three of the studies comparing psychodynamic therapy to a control condition were RCTs and the earlier results we discussed indicated that the more well-designed the study is, the less psychodynamic therapy outperforms a condition in which clients receive no treatment at all. Nonetheless, Shedler cited this as a prime example of psychodynamic therapy having robust support in the treatment of personality disorders and a greater effect size than CBT. Starting to see why meta-analyses can be dangerous yet?
From here, Shedler shifted focus to a meta-analysis conducted by Leichsenring and Rabung (2008) that supposedly demonstrated that long term psychotherapy (at least 50 sessions) is more effective than short term psychotherapy. A paper that is in the final stages of review addresses the remarkable shortcomings of this paper and, once it is published, will be discussed here at length.
Psychodynamic Process in Other Therapies
At this point, Shedler shifts his focus to an entirely different point: his belief that psychodynamic principles are actually applied in other therapies and that these principles explain why those therapies work. Essentially, he cited a number of studies in which individuals unaware of whether psychodynamic therapy or CBT was being conducted rated therapy transcripts on the degree to which they used components from each type of treatment. They found that the use of psychodynamic principles predicted greater treatment outcome in both forms of therapy and the use of CBT principles was unrelated to outcome in either approach.
There are a number of ways in which this is fairly ridiculous. First of all, it is impossible to be blind to treatment type when you are reading the transcript of one of these forms of therapy. If for no other reason than the jargon used in each therapy is completely different, it would be completely apparent to the assessor what was being done. As such, allegiance effects completely invalidate this finding. Even more problematic, however, was how they assessed this. They rated three principles:
- Therapeutic alliance
- Implementing the cognitive model - e.g., addressing distorted cognitions
- Experiencing - this one is the ridiculous one and the one said to be a measure of psychodynamic principles. It was explained as follows: "At the lower stages, the client talks about events, ideas, or others; refers to the self but without expressing emotions, or expressed emotions but only as they relate to external circumstances. At higher stages, the client focuses directly on emotions and thoughts about self, engages in an exploration of his or her inner experience, and gains awareness of previously implicit feelings and meanings. The highest state refers to an ongoing process of in-depth self-understanding."
So...in that last variable, the raters assessed whether the client look at their thoughts and environment, identified how they related to their emotions and view of themselves, and gained awareness of their own emotions and thoughts. How is that inconsistent with what happens in CBT and, more importantly, how is that unique to psychodynamic therapy? Where in there is the requirement that things be open-ended? Where is the focus on unconscious conflicts? Quite frankly, this is akin to saying that CBT emphasizes getting better so, when a client improves, that is a measure of whether or not they have implemented the cognitive model and, if a client happens to be in psychodynamic therapy and they get better, this means that they really got CBT. The degree to which this argument lacks scientific validity is staggering. Additionally, if the implementation of psychodynamic principles predicts outcome, why does CBT outperform psychodynamic therapy as it does in the studies mentioned above? Does this mean that CBT therapists are better at psychodynamic therapy than psychodynamic therapists are? Additionally, if there is enough variability within a CBT study on the degree to which cognitive principles were adhered to, that means that the study itself was not a valid representation of CBT. In other words, the only way that successful therapy can occur in this model - low usage of CBT principles and high usage of psychodynamic principles - is if CBT is administered improperly. As such, the only way that this could be found would be by looking at inadequate studies. Furthermore, Shedler himself stated that "qualitative analyses of the verbatim sessions transcripts suggest that the poorer outcomes associated with cognitive interventions were due to implementation of the cognitive treatment model in dogmatic, rigidly insensitive ways by certain of the therapists." So...all this really tells us is that, when CBT is conducted poorly, it's not great. All right then.
The Flight of the Dodo
Shedler then shifted gears to the discussion of therapeutic equivalence. I'm not going to waste your time by critiquing the dodo bird hypothesis yet again in this already long post. Yesterday's post and several others provide detailed information as to why the dodo bird hypothesis is an erroneous claim based upon gross misinterpretations of highly flawed data.
That being said, Shedler did mention something worth discussing in this section: the Shedler-Westen Assessment Procedure (SWAP), created by himself and Drew Westen, a vocal opponent of empirically supported treatments for mental illness. This scale, they claim, can tap into the "inner capacities and resources that psychotherapy may develop." How does it do this, you ask? Does it read the brains of the client? Does it teach the client to explain their unconscious feelings and beliefs? Nope. For the low cost of $145 per year for individuals, $295 per year for agencies, and $495 per year for funded research, the SWAP asks the therapist him or herself to rate these things for the patient. The therapist is thus entrusted to evaluate such things as "has an active and satisfying sex life" and "has moral and ethical standards and strives to live up to them" for the client. For some reason, the authors believe that the clinicians have more insight into this and are not the least bit motivated to rate clients in such a way that would make it seem as though they are performing well at their jobs whereas self-report measures (which are, to be fair, also flawed) result in nothing but bias (and results that favor other forms of therapy of course). The authors claim that there is extensive data supporting the utility of this measure; however, this research consists of a single case study and a comparative study in which psychoanalysis was compared to....more psychoanalysis.
One of the reasons the authors favor the SWAP is that they believe symptom reduction is not a valid goal. This is a legitimate belief in that one would obviously hope that a client's life in general improves as a result of psychotherapy; however, this is also a talking point that can be and often is easily distorted. If a client comes in with panic disorder and treatment removes panic attacks but does not have a huge impact on their interpersonal relationships (which may or may not have been an issue in the first place), that's an imperfect but good outcome. The panic attacks in panic disorder are crippling and their removal is no small feat. It would be better if it improved relationships too; however, symptoms are of greater immediate concern as they are associated with worse outcomes (e.g., death by suicide). If one therapy is shown to be equally effective at reducing symptoms but is superior at reducing secondary variables relative to another treatment, that treatment is the better choice. The implication in the Shedler paper is that psychodynamic therapy does that; however, the evidence does not support that in any way whatsoever.
Conclusion
Ultimately, the Shedler article did provide some useful insights. It clearly defines many of the principles of modern psychodynamic approaches, it emphasizes the need for empirical research and an increase in the quality of psychodynamic research in particular, and it promotes the idea of treating as many aspects of a client's life as he or she needs. That being said, the vast majority of the empirical evidence provided was based upon meta-analyses that were flawed to such an extent that their publication is somewhat mystifying to me. However, because the results of those meta-analyses were cited as strong, others have come to believe that this is the case and that message has traveled quickly in a massive game of telephone in which the original message (the actual outcomes of the data examined in those meta-analyses) has been completely changed to suit the needs of those repeating it.
My goal here is not to dismiss psychodynamic therapy as useless. In a prior PBB article, we discussed some fairly strong evidence that it can be a decent alternative for generalized anxiety disorder if CBT fails (CBT outperformed psychodynamic therapy on measures of worry, depression, and trait anxiety). The goal, instead, is to temper claims that the Shedler article and the meta-analyses it cites are somehow proof of the superiority of psychodynamic approaches or sufficient for the approach to be considered evidence-based.
************
If you would like to learn more about the topics discussed on Psychotherapy Brown Bag, we hope you will consult our online store of scientifically-based psychological resources.
Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University
Subscribe in a reader
