by Michael D. Anestis, M.S.
For the past couple weeks, I have spent a substantial amount of time thinking about some of the core ideas of PBB and the messages I am trying to get across when I write these articles. Quite often, the focus of my writing involves describing the results from a specific study. My thought in doing this is that, by writing about this material in a more accessible way and disseminating it as widely as possible, perhaps a greater number of people can learn accurate information about mental illness and psychotherapy rather than absorbing the misinformation that is so prevalent. That being said, in my interactions with readers and colleagues, I have frequently come across an obstacle in this goal: confusion about the different types of research that exist, what those types of research are capable of telling us, and the degree to which any of that information actually has tangible implications in every day clinical care. Today, my hope is to clarify those points while discussing a recently published study by Daniel van Igen, Stacy, Freiheit, and Christopher Vye (2009) examining the degree to which cognitive-behavioral treatments for anxiety disorders are successful in clinical settings relative to laboratory settings. Joye and I have touched on some of these topics in prior articles, but by more directly addressing them in a single post, I hope to bring an added sense of clarity to the matter.
How do we determine if a treatment "works?"
This particular question is a point of contention and is central to everything I want to discuss today. Given the amount of time we spend discussing empirically supported treatments (ESTs) on PBB, it makes sense that we should make an effort to explain precisely what we mean by "support." Generally speaking, there are two types of studies that examine the degree to which a particular type of treatment is useful: efficacy studies and effectiveness studies. Efficacy studies, which often take the form of randomized controlled trials (RCTs), examine the degree to which a particular form of treatment works under highly controlled conditions. These studies often occur at universities and exclude participants with multiple diagnoses or high risk of suicide and demand that the therapists stick to treatment protocols and utilized treatment manuals. Effectiveness studies, on the other hand, examine the degree to which a particular treatment works in more common clinical settings and, as such, are less restrictive with respect to who can participate and how closely the clinicians must follow a manual.
One of the most common critiques of research and ESTs is that they rely heavily upon efficacy studies and, as such, they are based upon samples that do not resemble the folks who show up at clinics in need of treatment. This concern is legitimate, but it also problematic in its own right. In the next section of this article, I would like to discuss the relative value of each form of research.
Efficacy studies - what's the point?
If efficacy studies are so different from clinical practice, what is their use? Actually, they offer numerous, highly valuable pieces of insight. At first glance, it might seem odd that efficacy studies make an effort to control so many aspects of the environment when, in every day practice, we do not have nearly that much control. They do this, however, in order to be good and thorough scientists.
Why exclude people?
As you might have seen in my "bad science" article recently, one of my main critiques of some studies is that they examine the wrong question. They want to know if one treatment solves everything for everyone. The answer to that question is no, but the question itself is inane. The correct question is whether there are particular treatments that work particularly well for specific diagnoses. After all, we take different medications depending upon which physical ailment we have, right? In efficacy studies, the authors exclude people with multiple diagnoses - even though it is more common for an individual to meet criteria for multiple diagnoses than to only meet for one - because they want to see the degree to which their treatment directly impacts specific symptoms and to remove the possibility that other variables (e.g., the presence of other symptoms) account for the relative value of the treatment. This is not to say that it is not important to see if a treatment works for clients with multiple diagnoses. Instead, it simply points toward the importance of first establishing that the treatment can accomplish its purpose under controlled conditions.
Waitlist/Control condition?!
It might seem weird or even unethical to have a waitlist condition in which some of the people in a study are not receiving treatment. Keep in mind, more often than not, these same individuals do receive treatment but are simply delayed because, by the time they registered, the therapists all had full case loads in the study. Individuals on waitlists are not ignored and left in dangerous situations. Still, this does not explain why they are such an important part of efficacy studies. Think about depression. By nature, depression is episodic - meaning it is there, then it is gone, then it comes back, then it is gone again. On average, a depressive episode left to its own devices will last approximately nine months. That being said, it does not tend to last forever, meaning the simple passage of time can lead to decreased symptoms. Herein lies the value of the waitlist or control condition. In order to make sure that people are improving because of the treatment and not simply the passage of time, researchers much be able to compare people who receive treatment to people who do not over an extended period of time.
What about common factors?
Common factors - characteristics shared by all or most therapies - are often thought of by people less involved in research as the true mechanism of change in therapy. They might refer to the therapeutic alliance, catharsis, or some other similar concept and say that it does not matter what type of therapy they practice as long as these variables are attended to properly. Joye has written a number of great posts on this topic, but I bring it up because, here again, efficacy studies go a long way to providing us with empirical evidence that can counter simple beliefs based on anecdotes and experience. In these studies, a specific treatment (e.g., cognitive-behavioral therapy) can be compared to other treatments and the researchers can control for common factors to see whether they actually account for therapeutic change. Given that efficacy studies frequently indicate that numerous therapies produce positive results, but certain approaches are better than others, pinning all of our hopes on common factors seems ill advised. Now, this type of analysis can be done in effectiveness studies as well, but nonetheless, it is a source of value in efficacy studies.
Manuals and treatment fidelity? Are we robots or therapists?
Efficacy studies take great care to ensure that therapists faithfully follow treatment protocols and use manuals. Why? Because doing so ensures that each therapist is doing the same thing, so when we compare outcomes, we are not comparing a heterogeneous mixture of different treatment approaches, but rather a very clear system of treatment. In this way, again, we can be sure that what we think is causing change is actually what is causing change.
Effectiveness studies - are they the answer?
Effectiveness studies, which look at treatment outcomes in standard everyday clinical practice, have become a common method of study and they offer significant value. After all, they give us a sense of whether treatments can be immediately implemented into clinical care settings and still produce meaningful results with clients who present with a multitude of comorbid diagnoses. At the same time, they are not perfect, and it is important to understand their limitations.
The good
As I said above, effectiveness studies answer the question - do these empirically supported treatments only work when we control the environment so tightly that the sample bears no resemblance to actual clientele at a clinic? Fortunately, the results indicate that laboratory findings are very successful in everyday practice (see our past article on CBT for anxiety disorders in clinical settings). These studies also help keep non-academic clinicians involved in research, thereby expanding its reach and helping to close the research-practice gap.
The bad
Unfortunately, these studies are also full of their own unique weaknesses. First of all, quite often the therapists do not follow the specific treatment protocol or manual. Now, given that effectiveness studies have shown similar results as efficacy studies, an argument could be made that this means treatment fidelity and manuals are not necessary. Remember, however, Bhar and Beck (2009) among others have found that, in studies in which therapists adhere to treatment protocol, results are stronger. So, positive results can be possible with improvisation; but on average the results are not as strong. Some might say that deviation from protocol simply represents standard everyday practice. My response, however, is that it does not have to. Just because people choose to dismiss protocol does not mean that they can not change that decision, either on their own or through regulations that mandate such shifts in practice. Manuals and protocols do not remove all flexibility from treatment or turn therapists into robots and ignoring evidence without justification shifts the priority of treatment away from the client's well being and onto the clinician's need to adhere to a particular philosophy.
Effectiveness studies (and efficacy studies) are also plagued by small sample sizes. When you have a small sample size in your study, you can not statistically detect differences between groups even if they are there. In other words, without enough participants, you can not answer your own question. Nonetheless, studies with small samples are published all the time and, because statistical power is a bit of an esoteric, nerdy topic, nobody talks about the fact that these studies can not actually do their job. This is particularly problematic in effectiveness studies because, when no difference is found between two groups, folks then use the underpowered study to justify a false belief that, no matter what they do, they will get the same results in the end.
If we were to ignore effectiveness studies, we could not legitimately speak to the degree to which laboratory findings generalize to clinics. If we relied entirely upon effectiveness studies, however, we would be left unable to determine the degree to which variables other than the treatments themselves account for changes in clients.
Okay Anestis...enough with the rambling. Give me some evidence
A PBB post generally isn't a PBB post if it does not discuss at least one published study, so let me wrap this up with a quick summary of an effectiveness trial, just to emphasize the point that, no matter what you hear some people telling you, CBT is not something that only works in a lab. Van Ingen and colleagues (2009) analyzed a number of effectiveness studies that examined the degree to which CBT for anxiety disorders is successful in everyday clinical practice. In order for a study to be included in their analyses, the sample had to be non-university based, patients had to be referred for therapy rather than research, clients had to present with heterogeneous problems, the therapists involved in the study had to maintain a regular caseload, treatment could not be manualized, and treatment fidelity could not be monitored. Now, as I'll explain in a moment, I think some of these criteria are ridiculous, but let's just go with it for a moment and discuss the results.
The authors found that CBT was associated with significant reductions in anxiety symptoms and that those gains were maintained at follow-up (average follow-up time was one year post-treatment). Similarly, the authors found that CBT was associated with significant reductions in depressive symptoms and that those gains were also maintained at follow-up. So, even when the restrictions associated with efficacy studies are removed, CBT successfully reduces symptoms of mental illness in individuals with anxiety disorders.
Keep in mind, CBT was not compared to any other treatment or a waitlist here, so the degree to which it works relative to other approaches (or no approach at all) can not be inferred from these results. That being said, the results are still highly promising.
My final point on this study, however, involves the criteria based upon which studies could or could not be included. I fear that our willingness to accept that manuals and treatment fidelity are not a part of everyday practice represents a harmful trend. Simply because people do not adhere to these ideas does not mean that they should not. Ultimately, this is an empirical question. We can absolutely test whether the use of manuals and an adherence to treatment protocols is important. Thus far, what evidence is out there supports the idea that they are.
If you would like to learn more about empirically supported treatments, the following represents a small sampling of resources we recommend. Many others can be found through our online store.
A Guide to Treatments that Work
by Peter Nathan and Jack Gorman
Cognitive Therapy of Depression
by Aaron Beck, John Rush, Brian Shaw, and Gary Emery
Clinical Handbook of Psychological Disorders, Fourth Edition: A Step-by-Step Treatment Manual
by David Barlow
Skills Training Manual for Treating Borderline Personality Disorder
by Marsha Linehan
Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University
Subscribe in a reader
