by Michael D. Anestis, M.S.
I came across a great study thanks to Laura Collins and her blog about family-based treatment for adolescent anorexia nervosa. In this study, conducted by Geoffrey Munro (2010) of Townson University and published in the Journal of Applied Social Psychology, the author took a look at how our view of science can be shaped by the degree to which scientific evidence supports our views. As you might imagine, the prospect that our belief in science hinges upon science backing up our philosophy is a disturbing one to me, as I believe firmly in the importance of using data to drive our understanding of mental illness, even when the data discomfirm my expectations.
Munro's manuscript actually involved two different studies that built upon one another. In the first study, 84 undergraduates were asked to what degree they believed in the stereotype that homosexuals are mentally ill (to be fair, the study was unclear whether this meant that homosexuality itself represents a mental illness or whether homosexual individuals are more likely to have a mental illness, but this is irrelevant to the outcome). The participants were randomized into one of three conditions. In one condition, they read a series of supposed summaries of scientific studies indicating that the stereotype was valid. In another condition, they read a series of supposed summaries of scientific studies indicating that the stereotype was invalid. In the control condition, no mention was made of sexual orientation. After reading the summaries, the participants were asked to indicate to what degree science is capable of answering the questions asked in the summarized studies. Additionally, they were asked to what degree science was capable of answering five unrelated issues (e.g., the existence of clairvoyance, the effectiveness of spanking, the impact of viewing violent television on behavior).
Before discussing the results, I have one substantial issue with the analytical procedure undertaken by the author. Even though belief in the stereotype regarding homosexuality was assessed on a dimensional scale (e.g., to what extent do you believe this?), the author then broke the individuals into dichotomous groups (e.g., do you believe or do you not believe?). In other words, even though some individuals indicated a slight belief or disbelief and others indicated a strong belief or disbelief, the strength of belief was deleted from the analyses and simplified to simply represent belief or disbelief. Additionally, any participants who indicated they were neutral or unsure (a rating of zero on a scale ranging from -4 to +4) were deleted from the analyses all together. One of the first lessons I learned in my first graduate level stats class is not to dichotomize continuous variables, so I'm a little confused by this decision. Doing this vastly diminishes the amount of information you can obtain and can obscure results in a spurious manner.
Anyway, the findings from this first study were very interesting. Individuals who read summaries of evidence that disconfirmed their beliefs (regardless of whether their belief was that the stereotype was valid or invalid) were significantly more likely to say that science is incapable of answering the question than were individuals who read summaries that confirmed their beliefs or individuals in the control condition. In other words, when confronted with purported scientific evidence that painted a different picture than what they believed, these individuals were increasingly likely to simply dismiss science as a valid approach to answering the issue. This, of course, is a completely unscientific way of thinking. Either you believe in data or not - that belief can not hinge upon whether or not you like the results. Now, data can be bad, whether due to bad methodology, inappropriate analyses, or another valid reason, but the quality of a particular study has no bearing on the utility of science as a whole and the participants in this study were not given an opportunity to read such details, so they could not be the basis for their rejection of the findings.
In this same study, the authors did not find a significant difference across groups on the likelihood of believing science to be capable of answering the questions mentioned earlier that were unrelated to homosexuality. This prompted them to conduct a second study to further explore the issue. In study 2, 93 students were assigned to the experimental condition (no control group was used this time). The methods were identical to study 1, except that the authors also asked about what type of information individuals would seek to help solidify their beliefs on the death penalty. As was the case in study 1, individuals who read summaries that disconfirmed their beliefs were more likely to state that science was incapable of answering the question. This time, the disconfirming group was also more likely to indicate that science was incapable of answering the unrelated questions. Those who read confirming evidence indicated that they would be more likely to seek out scientific than nonscientific information about the death penalty whereas those who read disconfirming evidence were equally likely to seek our nonscientific evidence as scientific evidence.
All of this brings me to an important question: why am I discussing this on PBB, which is focused on mental health? First of all, I am not doing so to prompt a conversation about homosexuality and mental illness. Instead, I am writing about this because it is highly relevant to research on psychotherapy (and those who have read some of the more heated conversations between myself and some readers have seen this issue come to a head on this very site. Click here or here for an example). There is a tendency on the part of individuals who do not believe in the utility of empirically supported treatments for mental illness to claim that scientific studies simply are not capable of measuring the variables that would otherwise either prove that all treatments work equally well or that their favored therapy is, in fact, highly effective despite data indicating something different. They make arguments that might sound valid at first glance (e.g., "how do you measure suffering and health?" "How can you discount the happiness on the faces of my clients after we utilize alternative approaches to therapy?"). These arguments fall flat, however, when science is made accessible and the methods and results are explained in a way that they can be understood for what they are. Certainly, there are variables we have not yet measured and others that we have but have only partially captured; however, to focus on them and assign them superior value is to discount mountains of evidence that empirically supported treatments - a perpetually evolving list of treatments shown to be the most effective for the greatest number of people for particular conditions - are life-saving, cost-effective options for individuals suffering from mental illness.
It is so easy to dismiss things that go against our preconceived notions. It is highly difficult to revise our strongly held beliefs in the face of discounting evidence. Considering the facts and adjusting our beliefs as needed is imperative, however, otherwise we are left with a situation in which the psychologist who makes the most emotionally compelling case for their approach will garner the greatest following, regardless of whether his or her approach is the best option for the most people. Science is imperfect, but it controls for our emotional responses and intractable biases and it provides a blueprint for how it arrived at its conclusions, thereby offering opponents an opportunity to base their counter-argument on facts rather than philosophies.
I highly value passionate beliefs, but like Karl Popper and Paul Meehl, I believe them to be the starting point of something better: reliable facts based upon reality rather than our wishes which, unfortunately, do not always align with one another. In the field of clinical psychology, the decisions we make impact the lives of individuals suffering greatly and we need to ensure that our conclusions benefit them rather than simply satisfying our own philosophical predisposition.
One final note before I sign off on this one and open things up for comments: some of you might read this and think "didn't Mike exhibit these same logical errors in criticizing other studies like the Shedler (2010) article?" I'm certainly that I have made logical errors in the face of evidence that has not supported what I anticipated finding, but on this site, I have been sure to base all of my criticisms on precise pieces of data. In other words, rather than saying that something is invalid because it simply is, I have stated the exact methodological and/or data analytic error that contributed to its weakness. Doing so is an important component of the scientific process, as it is only through challenges to existing studies that better studies are developed and more accurate information is obtained.
************
If you would like to learn more about the topics discussed on PBB, we hope you will consult our online store for scientifically-based psychological resources.
Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University and an incoming resident at the University of Mississippi Medical Center.
Subscribe in a reader
