by Michael D. Anestis, M.S.
Amidst the many articles Joye and I have written sharing findings from various psychological studies has been a scattering of articles critically examining what some believe to be unsubstantiated claims regarding both general psychological phenomena (e.g., does suicide risk assessment actually increase suicide risk?) and specific forms of treatments (e.g., equine-assisted psychotherapy; EAP). The goal of those articles is not to criticize anyone personally nor to discourage people from investigating these phenomena. In fact, the opposite intention is very much in play: we hope to alert readers to the lack of research on particular topics (or the presence of uninspiring research) and to encourage interested parties to empirically investigate the issues. All that being said, our recent article on EAP sparked some interesting conversations, both in the comment section and on other websites, regarding how newer treatments are evaluated and what can be done to encourage innovation while discouraging the proliferation of treatments that lack adequate empirical support.
This morning, I received a timely email from James Coyne, co-author of the November 2009 PBB featured article, alerting me to a back-and-forth exchange published in the current issue of Behaviour Research and Therapy between Brandon Gaudiano of Brown University and Lars-Goran Ost of Stockholm University. In Gaudiano's (2009) article, he critiques a paper published by Ost in 2008 in which he concluded that studies on Acceptance and Commitment Therapy (ACT) have not been methodologically strong relative to studies investigating Cognitive Behavioral Therapy (CBT) and that ACT studies have not fulfilled the requirements outlined by Division 12 of the American Psychological Association (APA) for a psychological treatment to be considered empirically supported for any particular diagnosis (also see Chambless & Ollendick, 2001 for a description of these criteria). Ost's (2009) article serves as a rebuttal of Gaudiano's critique.
Today, I would like to discuss some of the issues these two authors considered in their articles, although I have no intention of coming down on one side or the other regarding their particular argument. Instead, I want to discuss the very important question of how we should evaluate emerging treatments to ensure that innovation is pursued, but untested treatments are not practiced in a manner inconsistent with their research foundation. As a little background, let's first consider the nature of the original Ost (2008) article that inspired this back-and-forth. In that study, Ost examined 13 randomly controlled trials (RCTs) of ACT. Additionally, he "matched" each of these studies with an RCT of CBT published in the same journal within one year of the publication of each corresponding ACT study. The point of this exercise was to examine the methodological quality of RCTs of these two treatments published in identically prestigious journals during the same era of research. Ost (2008) reported that the ACT studies had significantly lower average scores on a number of variables, including:
- Representativeness of the sample
- Reliability of diagnosis
- Reliability and validity of outcome measures
- Assignment to treatments
- Number of therapists
- Training/experience of therapists
- Assessment of treatment fidelity
- Control over concomitant treatments (e.g., medication use)
Gaudiano's (2009) criticism of Ost's (2008) original work centered on two key points (although he raised other issues as well). First, CBT studies receive a greater average amount of grant funding relative to ACT studies. Further, he argued that the increased funding enables CBT researchers to use more advanced methodologies in their studies. Second, ACT is a newer form of treatment that is still in the process of establishing its empirical support. As such, some of the studies are best thought of as pilot studies intended to provide a rationale for funding agencies considering whether or not to provide grant money for larger scale future investigations of the treatment (and, in fact, he pointed out that one such grant funded study is now underway). Furthermore, he pointed out that APA Division 12 currently lists ACT as "probably efficacious" in the treatment of depression. Importantly, Gaudiano was a co-author of one of the ACT studies considered by Ost (2008) as well as one of the CBT studies, so he was not approaching this from the perspective of an individual in favor of one treatment and against another, but rather as a researcher with a different perspective on the state of ACT research and the expectations of new treatments with respect to the type of data that must be collected in order to establish empirical support.
Ost's (2009) rebuttal pointed out that, although less funding can have an obvious impact on establishing the reliability of diagnosis and the degree of treatment fidelity, as doing this requires independent observers to join the team and that costs money, the funding issue would have no impact on the use of fairly representative samples, outcome measures with well-established psychometric properties, proper assignment of participants to treatment, the use of multiple therapists with training in the conditions involved in the study, and the control of concomitant treatments. Additionally, although reliability of diagnosis can be impacted, the establishment of diagnoses does not hinge upon heavy funding. In other words, funding is not irrelevant by any stretch, but it also does not explain all of the methodological differences between the studies evaluated in the original Ost (2008) study. Ost (2009) maintained his original position, that ACT has not established itself as empirically supported for any diagnosis and indicated that he only found two ACT studies on depression that were available at the time of the original reports and he believes those studies are methodologically weak and actually did not establish differences between ACT and the comparison approach in treating depression.
The one area in which these two researchers agreed is that both proponents and critics of a particular treatment should be held to an equal standard. Where disagreement exists, however, is in determining whether treatments should be held to the highest standard when they are still establishing initial efficacy.
The main issue
Ultimately, when we step away from the specific argument between Gaudiano and Ost about CBT and ACT, the central issue to consider is whether the rules applied to already established treatments should be applied to novel treatments at the earliest stage of their research development. In other words, CBT trials that do not involve the highest methodological rigor, at this point, would generally (although not universally) be laughed out of the room. This is because, over decades of research, the basic efficacy of CBT for a number of specific diagnoses has been well-established and research methods have been improved such that we no longer need basic info on CBT but rather more rigorous data that demonstrates specific effects across a number of populations in diverse settings. Trials like that, no doubt, cost money, and researchers aiming to investigate CBT have a mountain of evidence to rely upon when proposing such work to funding agencies. When a new treatment is developed, no such foundation exists, meaning that funding is hard to come by, at least to some extent. The question thus becomes, in the absence of funding, given that certain components of ideal studies become impossible, what should still be required of work investigating the new therapy?
This issue is relevant to our discussion of equine assisted psychotherapy (EAP), which caused a bit of a stir and prompted some strong reactions, both positive and negative. The point of that article was essentially that, because no RCTs have demonstrated that EAP is, in fact, efficacious or effective in the treatment of any specific mental illnesses, the treatment should not be practiced or marketed as though it has (the same argument was put forth in our discussions of dolphin assisted therapy, wilderness therapy, and thought field therapy). This, of course, raises the question of how to establish that a treatment does work. After all, in the absence of data indicating that it is useful, it still might be just that. Ost's side of the back-and-forth seems to maintain that, once an established treatment for a particular diagnosis has demonstrated its effects through rigorous studies, a new treatment seeking to be considered empirically supported for that same diagnosis needs to be tested just as rigorously, even at the beginning. In other words, if the treatment wants to be treated equally, it needs to be put through equally strenuous tests and thereby produce equally compelling data. Gaudiano's side of the back-and-forth, on the other hand, seems to indicate that, when a treatment is new and grant funding is hard to come by, evaluation should be a bit more lenient so as to allow the treatment to establish a more readily accumulated research base.
Personally, I see reason in both sides, but I think that is because neither one addresses the entire question. How we research a treatment is different than how we practice and market it. In my opinion, if a novel treatment needs to establish some results through less rigorous study because funding is difficult to attain, that makes sense, as long as those results are not used as anything other than justification for future, more rigorous investigations to agencies considering funding future research. In order for a treatment to be introduced into the marketplace as a form of health care capable of attaining specific effects for particular mental illnesses, I believe that the treatment approach must be subject to the highest of methodological standards. This is not to say that I side with Ost on the ACT vs CBT argument, but I do agree with him in principle regarding many of the basic requirements set forth in order for a treatment to be considered empirically supported.
Where do you stand on these issues? Should clinicians be able to practice forms of therapy that lack empirical evidence established through methodologically strong studies? Should the standards of research requirements be relaxed for newer treatments and, if so, should there be limitations in how those treatments are used before further, more methodologically strong support is attained? Independent of your answers to those questions, do you think there should be restrictions on the marketing of treatments without empirical support and, with or without such restrictions, do you think it is beholden upon proponents of empirically supported treatments to better market their findings to potential consumers so as to increase awareness of the treatments available and the research supporting their use? Remember, this is not a discussion of ACT versus CBT but rather a broader discussion of how the arguments put forth by Gaudiano and Ost apply to all novel mental health treatments.
********************
If you would like to learn more about empirically supported treatments, we recommend the following items, all of which are available through our online store of scientifically-based psychological resources:
- A Guide to Treatments that Work
by Peter Nathan and Jack Gorman
-
Clinical Handbook of Psychological Disorders, Fourth Edition: A Step-by-Step Treatment Manual
by David Barlow
Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University
Subscribe in a reader
