Comments on What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?

Mike Anestis commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-19T19:45:53Z

Dave -

I just spent some time on your site. I had it opened in a tab for around a week and finally found some time today and am so glad that I did.

Your article on the place of emotion in cognitive theory was fascinating and I hope other readers will take the time to check it out.

Dave Johnson commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-12T20:37:23Z

I just cross posted my comment on my blog here: http://www.dare-to-dream.us/archives/2009/07/finding_meaning_in_research.php The links are good there. I thought I'd check back in case you said more, and there it was! If you wish to comment, you could do so there with a TypePad ID. Thanks!

I agree, Wood was testing a hypothesis based on Cognitive theory. So I guess her sweeping assumption that immediately reported lower self-esteem was evidence of "harm", was equivalent to my sweeping statement that her research wasn't based on theory. Wood's article isn't published yet. I say more on the linked article.

Mike Anestis commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-08T15:31:51Z

Dave -

Sorry it took me a few days to respond to your comment. Before I do, a quick note: the tiny URLs to the article you wrote did not work for me, so I have been unable to read your articles. Do you want to take another stab at tiny URLs? You're also welcome to just posted the entire link, even if it's a long and messy one. I'm interested in reading your articles, so hopefully you'll post them again.

I know of the Wood et al (2009) study you mentioned. I am, admittedly, only familiar with it in the context of discussions I have had about the results with colleagues, so I might be misspeaking here. That being said, couldn't an argument be made that the interpretations are grounded in cognitive theory? My understanding of these findings was that they were essentially testing the theory that individuals are uncomfortable with information that is incompatible with their own beliefs. That being said, because depressed individuals - or individuals with low self-esteem in this case - tend to view themselves negatively, positive self-statements represent a contradiction of their own beliefs and consequently result in a negative response in the absence of evidence that convinces them otherwise (e.g., an experiment testing hypothesized response of peers, with the client delineating evidence for and against the belief that the peer responded negatively to the interaction).

Again, I have not read the article, so I have no idea if the authors stated their findings within that context. Additionally, I like your hypothesis that the immediate negative results represented the first stage of a shifting response pattern. Longitudinal measurements of responses indicating whether the degree to which individuals responded negatively across time would help answer that question.

I'm not familiar with the tree of knowledge theory, but am curious. I agree that, quite often, researchers fail to infuse theory into their design and interpretation. Although, at the time, I found them painful, graduate courses on research design that stressed the importance of theory have proven useful in this regard. I do think that a good number of researchers do include strong theory in their research (e.g., Joiner's research on the interpersonal-psychological theory of suicidal behavior, Nock and Prinstein's work on the functional model of non-suicidal self-injury, Linehan's work on the biosocial model of borderline personality disorder), but clearly there are many researchers who look at data without theory and, as such, are ultimately making post hoc interpretations that may not generalize outside of that particular sample.

Anyway, I suppose my long winded point is that I truly wish that all individuals in the field of clinical psychology, whether they ultimately adopt a science based perspective, received rigorous training in this type of material. If they did, I think the conversations would be elevated in quality along with research and clinical work. I think that the sheer number of paths to the mental health profession in terms of training approach, rigorousness of training, and exposure to various approaches is so vast that it makes such an outcome unlikely, unfortunately...at least in the short term.

Thanks for the comment! I look forward to trying those links again.

Seesaw commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-06T05:07:06Z

Mike,

This has indeed been a pleasure. I too thought this was a very productive discussion, and I was glad of the opportunity to be confronted on my viewpoints, as I was able to clarify those viewpoints to myself, and, hopefully, to others in the process.

Regarding the Lilienfeld article, as I've said before, I am very much in favor of all the measurement that can be done. Everything contributes to the discussion, and certainly Lilienfeld highlights certain possible harms of treatments.

Now the tricky part is in the implications of such an article on treatment choices by clinicians. To what extent should clinicians be bound by a statement of harms that is limited to those based solely on *this kind of evidence* (highly replicable, etc.)? What about harms that are less easily measured? And how are these harms measured against benefits that are also less easily measured, and so not respected by the "evidence-based" movement? And finally, how much leeway do we give clinicians in overriding what is generalized in favor of an individualized treatment (to help the "outliers" you refer to in your article)?

I think that less measurable harms and benefits are critical to the clinician's analysis, and that individualized treatment is also key. I believe the clinician should certainly be cognizant of Lilienfeld-style harms. However, just as you believe that we should try harder to create better measures rather than abandoning certain phenomena as hard-to-measure, I believe that ultimately, the uniqueness of each individual case is more important than group averages, and that these pitfalls should not mean that the clinician gives up on tricky treatments. Instead, the clinician must strive to be ever more sensitive to the nuances of the individual case. In other words, I think the harmful outcomes are often (though NOT always--I'm sure there are treatments that are unequivocally harmful from any point of view) more a result of insensitivity to the individual case and inadequately developed empathetic and reflective capabilities than to an ignorance of evidence-based treatments.

A good example of this is recovered-memory techniques. From what I understand, there are genuine cases of long-forgotten abuse that are remembered. In other cases, false memories are recovered. So this is a tricky treatment that requires discretion and sensitivity to the individual case, not blanket avoidance. Unfortunately, these case-specific peculiarities and whether a therapist possesses the requisite sensitivity may be hard to measure in an EST-style test. We have to fall back on other kinds of evidence (i.e. the less reliable kind).

Hope that makes sense. :)

---

Dave,

Thanks for the pointer to the Tree of Knowledge. I think I'll have to study it more carefully before I have any thoughts, but it's certainly intriguing.

Dave Johnson commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-05T21:58:39Z

I have very much enjoyed the above exchange and find myself frequently thinking of the implications of our approach to research and how it contributes to our understanding of psychology.
Not only can our interpretations affect how we see and use a research finding, but the assumptions we bring to the research affects our choice of hypothesis and measurement target. Wood et al.(2009) (http://tinyurl.com/pyx4bl) pre-publication manuscript has gotten much press inappropriately proclaiming the positive affirmations may in fact harm those those most in need, those with low self-esteem. As I stated in an article I wrote(http://tinyurl.com/my869xout) these conclusions, there was I believe an error in one of the basic assumptions of the research. Wood and her colleagues assumed negative feelings after affirmations demonstrated harm. A review of basic theory might have captured what I believe was actually happening, the subjects were beginning a process of extinguishing there conditioned negative emotional response.

It seems researchers have drifted away from embedding their investigations in theory. Few authors seem willing to delve into the grand theoretical formulations as a basis for their research. For that reason, it's difficult to apply the results to much more than the specifics of the research setting. You've been discussing intuition as it impacts research. I think theory serves as a check on intuition.

I think one of the most important recent grand psychological theories was Henriques "Tree of Knowledge" (http://tinyurl.com/ooel74), yet I've caught little written about it since 2002 other than my humble attempt (http://tinyurl.com/q3jpps). I think this model provides us with a framework for these sorts of discussions. The link between psychology's investigation of the mind and interpreting the meaning of behavior (The Justification Hypothesis) is where data meets intuition, where research interfaces with theory. Ever since studying psychometrics I've integrated the concept of validity and reliability into my thinking about the theoretical interpretation of data. Reliable data (predictive validity) set in a meaningful context (content and construct validity), give us an opportunity to further our understanding of the meaning of human behavior in it's cultural context (construct validity). Yet I've never seen the concepts applied outside of psychometrics where they certainly seem to belong. Perhaps its again related to researchers reluctance to bringing a theoretical discussion to their research.

Have there been other frameworks that could be useful here, or might the one I suggest suffice?

Mike Anestis commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-05T19:27:06Z

Thanks to you as well! This is, by far, the most enjoyable, congenial conversation I've ever had on this topic and I think, as a result, it has also been the most productive. Although both of us feel incredibly strongly about very different positions on an important topic, I think our back and forth provides a strong resource for readers to start formulating their own opinions. I suspect neither of us is setting out to change the other's opinion per se, but I think we've done a good thing by providing a record of some thoughtful arguments for two different perspectives.

I truly hope other readers who maintain a different perspective than Psychotherapy Brown Bag will follow in your footsteps and engage with us like this. The more information that we can put out there in a single resource for readers, the better able they will be to make informed decisions. We do have an agenda in the sense that we want to provide readers with a place to find easily digestible information on the use of science and data in clinical psychology, but the central goal of the site is to open a dialogue and engage multiple perspectives.

At this point, I don't think it would be worthwhile to go through your latest thoughtful reply and detail disagreements because we have covered our positions quite well, but if I change my position on that later I'll add another comment.

I do have one question for you though because I'm curious what you think about another related topic. In an earlier article, we covered iatrogenic treatments and summarized Lilienfeld's (2007) description of treatments that have been shown to do harm. Do you see value in measurements in order to point out harmful treatments (as opposed to our current discussion on determining which treatment offers the most benefits)?

Seesaw commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-05T18:24:24Z

I'm really enjoying this debate. Thanks for putting so much time into it!

1. I'm basically in agreement that precise measurements and peer review are the most reliable method we have of drawing conclusions. I'm just pointing out that they too depend critically on theories and conceptualizations that are themselves not in the data; these theories and concepts come from intuition. There is no escape from that reality. Even with multi-method studies, each of those methods has to be chosen as having a valid relationship to the phenomenal reality; that choice is intuitive. The conditions of the experiment as being valid or not are also assessed by peers on the basis of their intuition.

What you're really saying is that there is a method of applying group intuition in a structured way that is superior to individual intuition; I say it's superior in its reliability but not necessarily in the places it can go.

The literature example is actually very apropos of therapy. We can measure things that contribute to life, like the heartbeat, as we can measure things to contribute to great literature, but they are simply not enough.

Literature, like human speech and action, has meaning. Meaning is not something that can be fully broken down into reliable measures, and this is especially true for beautiful meanings (the kind in great literature).

Mental health refers to a certain kind of functioning of human speech and action. Human speech and action have meaning. Similarly, their meaning may not be something that can be fully broken down into reliable measures, and that is especially true for beautiful meanings (the kind that mentally healthy people produce).

Is not the definition of mental health analogous to a definition of what is aesthetically pleasing (or, if you will, not too displeasing) in certain dimensions of human speech and action? Is that not the source of much of the controversy about what constitutes a mental illness?

2. You ask: if ESTs are not as effective as non-ESTs, then according to whom and what metric? Well, according to people who disagree that what is important is measured using the EST metrics.

So let's continue with our depression example. Suppose one believes that depression cannot be reduced to depressive symptoms as measured by objective and reliable measures + the possibility of the recurrence of those symptoms. Suppose one believes that depressive symptoms are in fact a pain reaction pointing to something more fundamentally wrong.

Let's say that what is more fundamentally wrong is not a positive symptom but an absence of human potential -- a lack of motivation or of positive emotional capacity that might, but for the disorder, have otherwise existed.

An EST might hide the disorder's immediate manifestation without curing the unfulfilled potential. So how to find whether the actual disorder is cured? Well, perhaps it can be found by less-reliable measurements, like the therapist's empathetic reaction to the patient.

What's the evidence that all this is true? One can only refer to other therapists' empathetic reactions and to other subjective assessments, PLUS a measure of interpretation and intuition.

But of course the EST proponent would reject those accounts as unreliable and biased. He would prefer his own more reliable assessments, which seem to leave out the essence of the matter according to the empathy-based people.

Evidence cannot resolve this dilemma, since its interpretation depends on normative assumptions that are not in themselves in the evidence -- that is, is depression a matter of depressive symptoms, or of unfulfilled human potential? Also, evidence cannot prove a negative (the absence of a unfulfilled potential). Nor can you rerun someone's life to see if they would have been more motivated or creative or whatever under a different therapy. You simply can't control all the variables.

The assumptions you begin with, the Kuhnian paradigm, color the import of all data collected, and therefore the preference for one kind of evidence or another is itself a matter of intuition and not of evidence.

3. I think we're in agreement that measurement is a great thing. I simply believe that, much as I'd like it to be otherwise, not everything can be measured at this point at the highest levels of reliability. Yet I believe these are the things which are the most important.

Mike Anestis commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-05T16:13:04Z

1. Yes, interpretation of data involves a moderate level of subjectivity in that you are measuring the degree to which depressive symptoms, for example (and I'm choosing to keep my hypotheticals within the confines of psychology related topics so as to explain how they actually have real world applicability), are decreased as measured by an assessment tool that works within a particular conceptualization of depression. The thing is, when we do this, we don't only do that. Nobody is basing empirical conclusions on one measure and most do not base them on one measurement type. If they do, they are making larger, less reliable assumptions. Utilizing a multi-method, multi-informant method of measurement provides a broader picture of the degree to which overall mental health is being addressed. Is this completely devoid of subjectivity? Of course not. Is it significantly more reliable than intuition alone? Absolutely. Is there any evidence that these reliable measures of mental health are providing faulty support for empirically supported treatments? Absolutely not. There is simply the claim that such a possibility exists, just as potential third variables always exist in any analysis, thereby necessitating caution in overly strong interpretations of data.

This brings me to the peer-review issue. My point in mentioning peer-review is not that it removes all interpretation of data. My point is that reviewers are careful to demand that authors temper their interpretations of data, offering only tentative interpretations that reflect a *rejection of the null* not *proof* of any conclusive reality.

In terms of the literature example - I just don't see the place for it in this discussion. Using generalities and abstractions does not add clarity to a discussion of the use of empirical data in clinical psychology. Literature is not psychology. If the greatness of literature can not be measured, that has nothing to do with whether or not the variables relevant to therapy can be measured. They are two entirely separate entities and the human tendency to draw conclusions based on things that have no bearing on one another - such as literature and clinical psychology - is one of the many reasons why relying on our own intuition alone without any attempt to empirically measure relevant phenomena is a faulty approach. "Life" can not be measured, but does that mean we can not adequately measure things that contribute to life, such as heartbeat, breath, etc...?

2. Again I don't agree with your analogy in this section. Comparing the relief of depression symptoms in depression to the relief of pain in cancer is not a proper comparison. Relief of cancer symptoms would be the proper comparison and, yes, if we cause a person to show no discernable symptoms of cancer and can demonstrate that there is a low likelihood of relapse, I would consider that a cure for cancer and a cause for great hope and I would prefer such a measurement over a doctor merely concluding, based on his beliefs, that cancer is no longer an issue.

The idea that ESTs only work to reduce "what they measure" seems to imply two things. First, that non-ESTs somehow reduce things ESTs are not capable of reducing. Second, that ESTs are leaving clients feeling discontent with therapy and lacking something important. To this, yet again, I reply: according to who and to what metric? Sometimes, in my discussions with people who have an alternative perspective, I get the distinct impression that the other individual believes that I and other proponents of ESTs have no experience of clinical work and, as such, can not grasp what happens in therapy. This is, in fact, far from the truth in most cases. In my experience, a metric upon which I place very little emphasis, clients experience ESTs as a life-changing experience that reduces all aspects of what we refer to as mental health. If a client sees a reduction in some symptoms but is still discontent, further diagnostic interviews will take place and additional therapy will be performed in order to ensure that whatever is missing is addressed. Nobody is kicked out of therapy because a therapist claims that the data demands such action take place. If a measure of depressive symptoms does not lead to a sense on the part of the client that his or her life is what they want it to be, further measurement takes place. This does not invalidate the measure of depression, but rather the choice of metrics in the beginning. Fortunately, further assessments can take place. Errors in measurement, unlike errors that involve no such measurement, can thus be tested and improved upon systematically.

3. I like your point about the choice between Type I and Type II errors being an issue of intuition regarding the relative importance of false positives versus false negatives. Again though, I am not saying that evidence speaks for itself, but rather allows everyone to speak together in the same language regarding the same variables. As such, it removes the "magical insight" element of the therapist's role and, instead, provides a metric by which he or she can make comparisons to others, thereby accessing a wealth of information to inform his or her interpretations. Without that data, there is information to pool from, but the therapist must make assumptions about his or her ability to accurately remember and weight that data - a risky proposition given the human tendency to falter in that area. There is simply no reason to believe that what transpires in therapy without measurement somehow allows better access to and treatment of any area of mental health.

Personally, I think it is worth the effort to create measures that will help us improve mental health care. I think this just as I think it is worth the effort to develop better surgical procedures, medical treatments for physical illness, curricula capable of improving school performance and learning, and so many other things. There is a finite amount of energy in this world. Science dictates that we put that energy towards progress, challenging what is believed by testing it and making incremental improvements.

Seesaw commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-05T01:40:53Z

Hi there,

1. Well, just as a starting point, intuition is very much behind every thought and move in science, from induction on up. There is no validating a conclusion "outside of" intuition. We may only choose to constrain ourselves with certain rules regarding reliability because *our intuition itself* so dictates.

As you point out, whether a useful protocol has been followed, what conclusions are drawn from the data, and so forth, is a matter of interpretation, and thus subjectivity and intuition.

Peer review only helps validate WITHIN a particular set of intuited assumptions. Peer review does not tell you whether that system itself is valid. Thus you have paradigm shifts every so often, per Kuhn. These paradigm shifts are not in some magical way necessitated by the data; they are choices scientists make in response to intuition -- in response to what they view as the more beautiful (parsimonious, flexible, useful, etc.) way of handling the data.

That said, I think you are correct in saying that the question is whether we are in a world where the key is more often in the dark or not. You claim that "evidence-based" approaches are going to have a low base rate of keys in the dark because they investigate past phenomena.

Actually, they do not. They investigate a particular operationalized construct that measures the past phenomenon. The question is: what is (significant about) that phenomenon? Is the construct a good proxy for the past phenomenon? And that is exactly what is at issue in this debate.

And bearing on this question, another hypothetical: suppose we wanted to figure out what made great literature great. We could analyze every "objective," replicable criterion possible: we could count syllables and words, analyze frequency distributions and look to geographical settings, character names, grammatical structures, and even structures at the story level.

These measures would be moderately informative.

But you know what? The greatness does not reside in any of them. The beauty of the literature resides in something aesthetic that is very hard to put into precise, replicable measurements. Nevertheless, "it" exists. And it is the most important thing about that literature. Until such day as objective approaches beyond our imagination are created, they would fail to capture what is truly significant.

And if this is so for literature, must it not be infinitely more so for the creator of that literature, the human mind?

2. So some emotional data can be measured, and some cannot be, and you ask what is the evidence that what we cannot measure is indeed important.

Once again, it is not evidence that tells us something is important; importance by definition is not a property of evidence (at least not in your sense of that word) but a function of our intuition, our interpretation.

My intuition tells me that the deep, complex emotions and thoughts that rage within us all are important. They are, indeed, at the crux of what great literature, art, and philosophy have always worked to understand. Any good psychology which seeks to comprehend the human condition, therefore, must encompass these "ineffables," as you put it. Highly reliable measures often cannot, and so they are inadequate.

As far as confidentiality, I don't mean purely ethical constraints or that clients would hesitate for fear of public revelation to talk about their innermost secrets. I mean that clients would be INCAPABLE of simply regurgitating what is deep within them except in the situation of deep comfort and intimacy generated by certain therapeutic relationships. It is perhaps that kind of relationship that makes certain reactions pour forth. They cannot necessarily be called on, or even remembered, on demand. That makes gold-star replicability close to impossible.

Next, whether and to what extent mental health can be broken down into simpler variables is indeed a serious question. ESTs work, all right -- they work at changing the variables they measure. But IS what they measure indeed mental health? THAT is the question. Not everyone agrees that mere relief of depressive symptoms is the cure to depression anymore than people agree that medicine which relieves the pain of cancer cures cancer (pain relief is necessary, but not sufficient). Not everyone agrees that because someone answers that they're "very happy" on some Likert scale that that indeed means that they are happy in a meaningful sense. Not everyone agrees that someone who is asymptomatic post-treatment but perhaps has lost some hard-to-pinpoint zest is actually better off. And so on.

These questions are not sideshows, but are central to treatment, and point to contours of mental health that, like the greatness of literature, are inherently vague and very hard to measure. And that leads to...

3. I certainly agree with creating better measures in the abstract, but there is limited time and energy in this world. Whether we find more value in trying to create better measures, in trying to use existing highly-reliable measures to investigate what may be investigated, or whether to use less-reliable measures to investigate yet other perhaps more significant things is a matter of judgment.

Much like the choice in measures between committing Type I and Type II errors, this all comes down to...intuition. Evidence does not speak its own meaning and cannot solve the problem.

Mike Anestis commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-04T18:21:04Z

This is, indeed, a very fun intellectual back and forth and, again, I appreciate you coming forward with an alternative perspective, as it allows for a fuller, richer examination of a topic, both for us and for readers. The goal of Psychotherapy Brown Bag is to present strong accurate information to help readers make informed decisions, not to indoctrinate anyone into a certain perspective, so this is very helpful. We are, of course, strong believers in the use of science and data, but we welcome discussions on this sort for so many reasons.

Although this is, without doubt, the type of discussion in which neither side is likely to alter their stance, I'd like to reply in turn with some thoughts on your thoughtful comment.

1. I agree with you that intuition drives the type of data that we gather. This is, in fact, what I was trying to get at when I quoted Karl Popper at the end of the article. Like any scientist, I agree with Popper that, while intuition drives the initial investigation, it can not be the basis for any conclusions. If it is, we are using pure guesswork without any framework within which to assess the validity of our conclusions. Certainly data is imperfect, but like I said before, imperfection is true of any assessment measure, including intuition and personal experience. Is there a degree of subjectivity in how we interpret data? That depends. Some people certainly take liberty with their interpretations of data; however, the peer-review process is in place to minimize this and, regardless, the data is spelled out in every journal article, ready to be challenged and improved upon by individuals who disagree with the author(s) conclusions. In this sense, much of the subjectivity inherent in intuition is removed and multiple individuals are able to interpret the same thing.

Your key example, while interesting is as much a criticism of your own perspective as mine. It represents a generalization and the type of story that hints at a point but does not directly address the question at hand. Are there specific example of evidenced-based procedures representing a narrow focus that leads to an inability to measure the key variables? If so, are those representative of the norm or are they representative of the error inherent in any form of investigation? Additionally, while a blind, intuition-based search may help you find the key outside the light on those occasions when the key is in such a location, such an approach is certain to miss out on many occasions when it is in the light, which the data driven approach will not do. Evidence-based approaches do not shine the light blindly - they do so in a manner consistent with past behaviors, thereby ensuring that there is a very low base rate of key-in-darkness scenarios. In that sense, the question becomes whether the occasional moment when intuition helps the exception is worth all the times when intuition overlooks the norm. In other words, is the one big win worth all the repeated losses?

2. I agree that subjective emotional data can be difficult to articulate. At the same time, much rich emotional data is in fact, quite measurable. As such, the question becomes whether data we can not measure is as valuable given the results produced by utilizing data that can, in fact, be measured. What is gained by focusing on the ineffable and what evidence is there that such an approach offers incremental validity?

Confidentiality is not a problem. Data can always be gathered anonymously. Certainly not every client would agree to that, but many do, particularly when a strong rationale is provided for the procedures, indicating that such research is used to ensure the best treatment is being provided.

I do not agree with your conclusion that breaking mental health down into measurable chunks might not work. If that were the case, we would not have so many empirically supported treatments (ESTs) that produce amazing results for a host of mental illness both in lab and "real world" settings. The question is not whether EST's work - such a question ignores so much evidence as to be difficult to even comprehend - but rather whether non-EST's work as well. They might...we just don't know.

3. I absolutely agree that there is a tricky balance between reliability and validity. The shift of the DSM in DSM-III to a more behavioral approach to symptom descriptions represented an emphasis on reliability that, at times (e.g., psychopathy and antisocial personality disorder) may have negatively impacted validity. The response though, in my opinion, should not be to de-emphasize reliability, but rather to improve our measures to make them increasingly valid. Psychopathy research represents the ideal on this front, as Lilienfeld, Patrick, and others have gone to great lengths to develop valid and reliable measures of psychopathy now, thus allowing it to return to the forefront of clinical research.

If our measures are flawed, we must create better measures, not abandon them altogether.

I'm not saying that there are never times when an individual's intuition will prove highly valuable, whether based on experience or some other factor. The "broken leg" phenomenon is a great example of that (e.g., using past behavior to predict when an individual will go to a movie is a great algorithm except when the individual breaks his or her leg, thus rendering the algorithm less valuable). The problem is that, when we focus on these exceptional moments, the value of intuition can become inflated in our minds and we can thus become prone to errors, biases, and misinterpretations of stimuli due to our own human nature.

Again - very much enjoying this conversation! It's fun to have a non-confrontational discussion on this topic.

Seesaw commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-04T17:04:10Z

Thanks for your thoughtful comment, Mike. It is always a pleasure to have an earnest and intelligent debate.

A few points:

1. Intuition is not a substitute for data so much as it guides what kinds of data we gather (i.e. the hypotheses we view as interesting and thus choose to test) and what kinds of conclusions we draw from that data. This is inevitably true even in an "evidence-based" paradigm. There is no such thing as data free from intuition.

If I choose to focus on hypotheses for which replicable, measurable data is harder to gather because I think these hypotheses are more interesting, that is a trade-off I make based on my intuition of what is valuable.

It's a lot like the old story about the man who looks for his key under the lamplight even though he dropped it 20 feet away in the dark. It's easier to look under the light (more precise, more reliable, easier to measure) -- but that's not where the key is going to be found. Indeed, he is better off groping in the dark randomly than searching in the light systematically. Of course, a flashlight is best if available, but it isn't always available.

2. Why might data be hard to measure? I can think of many reasons.

Extremely rich, subjective data on emotions and feelings may be very hard to articulate for the person experiencing them. They may be complex, contradictory, and follow one another so rapidly that they are hard to keep track of. Nevertheless, the clinician may be able to follow these emotions empathetically, and summarize his own impressions qualitatively.

Much data in therapeutic sessions is confidential. It cannot be reliably re-measured by someone else, since its disclosure is based on a long-term relationship of trust unique to a particular client and therapist.

Precise measurement intrinsically must break down the phenomena into chunks that can be measured. For certain phenomena - perhaps for something like mental health - this breakdown process may not work. The overall picture may be more than the sum of its parts. One may have a choice between crude measurement of the big picture and more precise but misleading measurement of the constituent parts.

3. Just to be clear, I am not inveighing against measurement. Everyone claims to be measuring things. It is the *precision* and *replicability* of the measurement that must be traded off against other things. Any doctor who does something based on intuition will say that it is based on measured data - the qualitatively measured data of his own experience.

Mike Anestis commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-04T15:00:15Z

Seesaw -

If the wrong thing is being measured, why not measure the correct thing instead of abandoning measurement? Is it because you believe the correct thing can not be measured? If so, why not?

It seems as though your conclusion is based on intuition, but you provide no logical reason why that must be so. If a client is coming in with specific symptoms, why would a direct measurement of the degree to which those symptoms are impacted be a measurement of the wrong thing?

Our instruments may be imperfect, but our intuition is as well. As such, the imperfection of measurement certainly can not be a reason why intuition is preferable. If your point is that imperfection makes something useless in psychology, then we can not use intuition either. In fact, we can use nothing - seems like a bad deal. At least, in the case of measurement, we are up front and clear about how we arrive at our conclusions, thus allowing for alternative perspectives to test that conclusion rather than simply declaring our gut to be true and asking clients to depend upon that feeling in order to help them through a difficult and potentially life threatening time. In fact, by not using measurement, we seem to be setting up clinicians to not be held accountable for their decisions and performance, allowing them to simply be persuasive in their argument that everything is working perfectly.

Control over situations is limited in every field. This is not unique to psychology. It is true in the medical, financial, and business worlds as well as many others, and yet psychology seems at times to hold a monopoly on the idea of being "beyond measurement." Perhaps that is true, but I am yet to hear a compelling case to support that idea.

The problem with the final statement in your comment is that, in order to know if we are providing evidence about the wrong phenomenon, we would have to measure the right one. Without evidence, there is no evaluation. So, the question is not whether there is evidence, but rather how strong your evidence is. Your logic assumes intuition is best, but provides no rationale other than a claim based on intuition that intuition alone can do the job. That is circular logic - my intuition says I should use my intuition, and, intuitively, I thus conclude intuition is the most logical. What if two very experienced, well-trained therapists arrive at two different conclusions based on intuition? How do we settle the argument? How do we determine who was right?

In clinical psychology, just like everywhere else in the world, performance can be evaluated, phenomena can be measured (or are not particularly useful tools), and deductive logic provides more valid and reliable conclusions.

I appreciate your decision to post your comment here. Given your alternative perspective, I suspect that must have been somewhat difficult and I hope you choose to continue this discussion. Readers benefit from alternative perspectives.

Seesaw commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-04T05:33:35Z

All the statistics and data in the world don't help you if you believe that the wrong thing is being measured. The data do not tell you if they're data about the right phenomenon. That is a matter of intuition.

And if our instruments are imperfect and our control of situations limited (which they both in psychology very much are), one might legitimately prefer somewhat weaker evidence about the right phenomenon than stronger evidence about the wrong one. That's the crux of the matter.

Mike Anestis commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-02T19:54:50Z

Thanks for providing those links, David!

David W. Ballard, PsyD, MBA commented on 'What do we mean when we say "evidence" in clinical psychology and why do scientists favor its use in evaluating different forms of psychotherapy?'

2009-07-02T19:16:36Z

For those who are interested, the American Psychological Association's Policy Statement on Evidence-Based Practice in Psychology is available online at http://www2.apa.org/practice/ebpstatement.pdf and the full report from our 2005 Presidential Task Force on Evidence-Based Practice is online at http://www.apa.org/practice/ebpreport.pdf