The empirical status of acceptance and commitment therapy (ACT): A conversation amongst two prominent psychologists

by Michael D. Anestis, M.S.

Back in November, we covered a series of commentaries written by Brandon Gaudiano of Brown University and Lars-Goran Ost of Stockholm University regarding the degree to which acceptance and commitment therapy (ACT) had developed an evidence base indicating that it is, in fact, empirically supported as a treatment for a number of mental illnesses (click here to read that article).  Gaudiano made the case that ACT, a newer therapeutic approach still developing its research foundation, had in fact met that threshold even as newer, more large scale studies are being funded to test the approach more rigorously.  Ost, on the other hand, contended that ACT had not yet been subject to the same level of research as older, more well-established approaches such as cognitive behavioral therapy (CBT) and, as such, could not yet be considered empirically supported.

What made their back-and-forth so interesting - much like the similar type of conversation we covered that went on between Siev and colleagues on one hand and Wampold and colleagues on the other - is that it allowed readers to read the thoughts on both sides of the argument, consider the evidence for both positions, and arrive at their own informed opinions.  These types of published conversations are rare, however, as the process of publishing is time-consuming and leads to unnatural gaps and interference in ordinary conversations.  That being said, I was thrilled when, last week, Dr.Gaudiano emailed me with a copy of a new response he has written to Ost in an attempt to continue the conversation.  Rather than wait for the peer review process to take its course - a process that is necessary for the evaluation of studies, but not for the rigorous debate of empirical data - he thought it made sense to take advantage of social media and the internet and publish his commentary here instead.  His hope is that Ost will respond in turn and my hope is that, if he does, he will consider posting his response here for all of our readers to view.  In doing so, we can bring an otherwise distant conversation into the present and render it interactive, with all readers capable of sharing their thoughts and contributing to the debate.  I suppose this is a tech-savvy populist psychology moment in a sense.

Please do note that, by posting Dr.Gaudiano's response, I am not endorsing one side over the other.  That being said, he makes some excellent points in this commentary and I look forward to hearing your thoughts.  Of course, Joye and I also want to express our sincere gratitude to Dr.Gaudiano for thinking to contact us with this and for allowing us to post the material on PBB for everyone to read.  What follows is the text of his reply and, again, I highly recommend reading our initial coverage (linked above) of their conversation so that you will have context for what you are about to read here:

************

Recently, I (Gaudiano, 2009) criticized the methodology used by Öst (2008) when he attempted to create a matched sample of traditional cognitive behavior therapy (CBT) studies to compare with acceptance and commitment therapy (ACT) studies for evaluating their relative methodological rigor. My reanalysis clearly demonstrated that Öst did not create a matched sample of studies as he had claimed. He erroneously reported that there was not a significant difference in grant support for the ACT and CBT studies used in his analyses. After correcting a few errors in Öst’s data, it became clear that the CBT studies were significantly more likely to have been supported by grants compared with the ACT studies (80% vs 38%, respectively). In addition, the average amount of grant support for funded ACT versus CBT studies clearly favored the latter (by a factor of over 5 to 1). Furthermore, I showed that the ACT and CBT studies also were mismatched in terms of treatment population. All the CBT studies chosen by Öst were in depression or anxiety disorder populations, whereas only 38% of the ACT studies were conducted in similar groups. In contrast, many of the ACT trials were in difficult-to-treat populations, including inpatient psychosis, drug dependence, and borderline personality disorder. Öst’s inadvertent matching of apples and oranges appeared to be the result of his questionable decision to develop a CBT comparison sample solely by choosing studies that appeared in the same journal ± one year of a corresponding ACT study. However, Öst was not able to adhere to even this minimal matching criterion fully because he could not identify a corresponding CBT study for 38% of ACT studies, and instead completed his sample by selecting CBT studies appearing in top-tier clinical psychology journals (e.g., Journal of Consulting and Clinical Psychology). To put it another way: Öst’s methodology would be analogous to matching a white, 14-year old female with a 58-year old African-American male in a study because both are human beings who happened to shop at the same department store in the same month. Since Öst’s stated goal was to compare the relative methodological rigor of ACT and CBT studies, a legitimately matched sample was necessary. Clearly the CBT and ACT studies were not actually matched as Öst had previously claimed and I believed that it was important to point this out.

In his reply to my critique, Öst (2009) now appears to concede that the studies were indeed mismatched, but then inexplicably goes on to conclude that this is largely irrelevant to the interpretation of his findings. Öst also makes several additional claims about ACT research that I believe require further clarification. I will respond to Öst’s most recent statements and also discuss the larger, and from my perspective, more important issues raised in this discussion regarding the evaluation of psychotherapy outcome research.

In my original commentary (Gaudiano, 2009), I agreed with Öst (2009) that early ACT studies had methodological limitations that should be considered carefully when attempting to interpret and generalize their results. We also agreed that more research was needed to replicate and extend the results of many of these initial ACT studies. However, I further pointed out that many of the methodological limitations noted by Öst are typical of any newer treatment approach in the earlier stages of development. It requires considerable time and money to conduct the type of methodologically stringent clinical trials with all the bells and whistles that Öst so strongly values. Öst asserts: “Thus, a relatively new treatment like ACT should not be expected to ‘invent the wheel again’ but learn from the way outcomes research is done in relatively older treatments like CBT, thus avoiding to make a lot of the methodological mistakes made previously” (p. 1071). However, this view is neither practical nor based in the realities of conducting psychotherapy outcome research at the present time. The normal 5-year National Institutes of Health (NIH) R01 grant to conduct a methodologically stringent psychotherapy trial has a multi-million dollar budget and employs a small army of personnel to accomplish the task. Thus, funding agencies such as the NIH do not provide large grants to study a treatment in this fashion until enough preliminary data from pilot studies can be presented demonstrating the potential promise of the treatment. ACT is now at the stage that it is beginning to amass the type of data that will make funding larger studies more compelling to such agencies and thus the kinds of “gold standard” trials that Öst desires will be forthcoming. At that point, he can legitimately compare properly matched ACT and CBT studies if he desires.

Öst (2009) also argues: “However, it does not take large grants to include a fairly representative sample, to use outcome measure with good psychometric data, to assign participants to conditions in a sound way, to use more than one therapist, to have therapists with proper training/experience in ACT, and to control for concomitant treatments.” First, I would point out that the issue is not merely one of “large” grants as Öst puts it. In contrast, the issue was no grant funding in 62% of the ACT studies evaluated by Öst, several of which were student dissertations. Most importantly, Öst’s argument appears to fall victim to the compared to what problem. In other words, Öst fails to properly acknowledge that his analyses compared the relative methodological characteristics of a subgroup of CBT and ACT studies. One group of studies was well funded and the other was not. It takes considerable financial resources for advertising and recruiting subjects, using blinded and expert raters to conduct assessment interviews, hiring several experienced therapists to provide the treatment, enrolling a large enough sample to use advanced randomization procedures, and providing subjects with active and credible comparison treatments. In fact, all of these features require relatively large amounts of money, so it is unclear what Öst means when he asserts otherwise. Anyone who conducts this type of research is well aware of the costs involved, especially as the treatment under investigation becomes more lengthy and complicated, and when the populations are more difficult-to-treat.

Öst (2008) originally criticized the use of treatment as usual (TAU) comparison conditions in some of the ACT studies because he argued that they were not sufficiently rigorous. I pointed out that studies of more severely ill populations use TAU conditions more often than studies of depression or anxiety disorders (Gaudiano, 2009) because the former are typically adjunctive treatments. Öst (2009) then replied by noting that several CBT studies of psychosis have used active comparison treatments in the past. However, again he misses the point. First, I should note that Öst did not actually include any CBT for psychosis studies in his “matched” sample, even though two of the ACT studies were in this population. More importantly, the issue is not that some CBT for psychosis studies have used active treatment comparisons. In fact, several ACT studies also have used active comparison conditions. The issue is that Öst was comparing the relative use of TAU conditions in ACT versus CBT studies. Because the ACT studies were in populations where TAU conditions are more typically used, and this is the case with previous CBT trials in this area as well, Öst’s mismatched sample resulted in another spurious finding.

I also criticized Öst (2008) for implying that ACT studies are not legitimate if they do not use current Diagnostic and Statistical Manual (DSM) criteria for defining the sample. Öst (2009) now claims that he did not suggest that DSM criteria are the only acceptable method of defining a sample, and that he considered other diagnostic tests (e.g., blood tests) when appropriate. However, the diagnostic rating question used by Öst (2008) actually specifies that a “good” rating requires that the “diagnosis was assessed with structured interview by a trained interviewer and adequate inter-rater reliability was demonstrated (e.g., kappa coefficient)” (p. 315). This clearly implies that a psychiatric diagnostic assessment device such as the Structured Clinical Interview for DSM be used, and these types of clinical interviews are based on DSM or International Classification of Diseases criteria. If Öst actually rated this question in a different way than it was written, then it still remains unclear why he scored my ACT study (Gaudiano & Herbert, 2006) as “poor” on this question. We did not define the sample based on clinical interviews of DSM criteria, but instead required that patients have verified hallucinations or delusions based on symptom ratings from the standardized Brief Psychiatric Rating Scale (BPRS). We also reported acceptable inter-rater reliability for the BPRS in the study. Additionally, I discovered that Öst scored the “handling of attrition” rating incorrectly for my study. This item was scored “0” but should have been “2” according to the stated criteria for this rating (i.e., “No attrition, or proportions of attrition are described, dropout analysis is performed, and results are presented as intent-to-treat analysis,” Öst, 2008, p. 317). Such discrepancies would appear to further undermine confidence in the reliability and validity of Öst’s ratings.

Öst (2009) further claims that I misunderstood the nature of matching procedures when I noted that individual CBT and ACT studies appeared to be mismatched on important variables. However, I described these individual studies merely for example purposes in order to provide the reader with some insight into the systematic problems with Öst’s matching procedure. To reiterate, I reported that 100% of CBT studies used patients with emotional disorders but only 38% of the ACT studies were in similar populations. This was certainly not proper matching.

Öst (2009) points out that the mediation analyses conducted in previous ACT studies also have methodological limitations. He further emphasizes that statistical mediation does not necessarily prove the existence of specific mechanisms of action for a therapy. Again, I agree with Öst in general about these caveats; however, he misses the larger point. ACT has done a better job to date at demonstrating statistical mediation than CBT studies, even though there are many more CBT studies and they have been conducted over a much longer period of time (see Hayes, Luoma, Bond, Masuda, & Lillis, 2006 for a review). As Öst was originally comparing the methodological rigor of ACT studies relative to CBT studies, so I was making the point that the strength of the evidence for statistical mediation at the moment favors ACT relative to CBT. For example, only 1/13 or 8% of the studies in Öst’s CBT comparison sample reported a forthcoming mediation analysis, whereas 10/13 or 77% of the ACT studies have already published mediation analyses or reported that such analyses are forthcoming (S. C. Hayes, personal communication, December 14, 2009). Öst then goes on to argue that this point is irrelevant because statistical mediation is not part of the American Psychological Association (APA) Task Force’s empirically supported treatment (EST) criteria (Chambless & Ollendick, 2001). True, but demonstrating statistical mediation of outcomes through changes in the processes targeted by the therapy is now recognized as an important part of the evaluation of the evidence for different psychotherapies (Kazdin, 2008). My point was that Öst’s focus on horse race trial methodology alone did not reflect a sophisticated analysis of psychotherapy outcome research as currently understood.

Öst (2008) concluded that, based on the EST Task Force criteria, ACT was not “empirically supported.” When I then pointed out that ACT is now officially listed on the Task Force’s web site as having “moderately strong” empirical support in the treatment of depression, Öst (2009) inexplicably continues to reject this fact. Herein lies the problem posed by individual researchers who take it upon themselves to be the arbiters of EST status. This process is designed to be done by a committee of experts based on impartial peer review.

Finally, Öst (2009) writes: “Any proponent of a new treatment that wants to claim that his/her treatment is empirically supported must accept that their treatment studies are evaluated against the criteria the same way as established treatment are” (1073). There is no disagreement between us here. To be clear: Currently, traditional CBT has a larger base of research support than ACT, and ACT proponents acknowledge this. As the ACT web site clearly states: “We recommend ACT on an experimental basis with any problem that fits the underlying model (e.g., the problem appears to involve cognitive fusion, or experiential avoidance, or a lack of clarity of values, and resulting inactivity, inflexibility, and ineffectiveness) provided it is used with systematic evaluation and there is a good reason not to use existing ESTs first (e.g., if they have already failed; client rejects their use).” Öst is arguing a straw man here. In conclusion, it is the convergence of evidence that is most important to making evaluations of psychotherapy, like all other determinations in science. Furthermore, a multi-method empirical strategy is necessary that combines basic research, process research, and outcome research for evaluating modern psychotherapies (Kazdin, 2008). ACT researchers have outlined how they are addressing each of these areas (Hayes, Levin, Plumb, Boulanger, & Pistorello, in press; Hayes et al., 2006; Vilardaga, Hayes, Levin, & Muto, 2009). Attempting to use overly simplistic checklists focused on one aspect of a body of research does not provide particularly meaningful or useful criteria for evaluating the overall empirical support and proper use of a psychotherapy. The EST criteria have their place but can only be considered one component of true evidencebased practice (Spring, 2007). The field is increasingly moving toward more flexible treatment guidelines based on emerging research that can better help clinicians make informed treatment decisions using ever changing bodies of evidence (Herbert & Gaudiano, 2005). Focusing solely on large-scale horse race trial methodology paints a distorted picture of the empirical support or nonsupport for a treatment. This is a mistake that psychotherapy researchers have made in the past and it is time to move beyond this approach. As Öst did not create a proper matched comparison group and thus could not draw the conclusions he intended, he would have been better served by simply arguing for the type of psychotherapy research program he believes is best.

************

There are many things to like about this piece.  Putting aside the overall interesting idea of bringing conversations like this into real-time by placing them on websites with established communities and thereby fostering a broad conversation, Gaudiano presented a very even-handed view of ACT rather than overstating the implications of the literature.  This was best summarized early in the final paragraph, when he explicitly stated that the research support for ACT is, at this time, less substantial than that of other empirically supported treatments (ESTs) and, as such, should be considered as a strong alternative only when other EST's have failed or are not desired by the client.  That being said, he argues that empirical support for ACT in this developmental stage of its existence is very promising and should more rigorous trials prove to support its efficacy and effectiveness, it should move freely to the front of the line along with other EST's for the diagnoses shown to be impacted in such trials.

So...I'm interested in hearing your thoughts on a number of issues.  First of all, what do you think of Gaudiano's points in this article?  Do you think that ACT should be considered empirically supported only when it has been tested in the most rigorous manner possible or do you think that, because ACT is a new and developing treatment still raising the funds required to run massive studies, should the preliminary data be taken into consideration?  Secondly, what do you think the potential value is (if any) of moving discussions like this from published journals to sites like PBB?  What might the implications be for such a transition?

As always, if you are interested in learning more about the topics discussed on PBB, including ACT, we hope that you will consult our online store of scientifically-based psychological resources.

Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University.