by Michael D. Anestis, M.S.
I'm stepping a little bit outside of my comfort zone today in order to talk about a story that relates directly to a recurring topic on PBB: how meta-analysis can be an incredibly powerful and useful tool...but also how it can be a remarkably misleading one capable of advancing an agenda while ignoring the facts. The article I'm discussing today involves a meta-analysis conducted by Dr.Joseph Wax and published in the American Journal of Obstetrics and Gynecology that examines the outcomes of homebirths versus births that take place in a hospital. A friend sent it my way and I thought the story of how the studies in the analysis were chosen and the narrative that those particular studies created was a worthwhile topic for this site.
As I write this please keep a few points in mind. First and foremost, I am not an M.D., so although I'll be commenting on an article that has to do with medical procedures, my comments are not meant to be guidance for how people should go about handling their medical decisions. I'm simply commenting on the data. Secondly, my point is not to champion one birth option or to vilify another. I certainly have opinions about things, but my purpose here is simply to discuss how we as scientists sometimes lose sight of the proper way to conduct research and ultimately make decisions that further our ideas at the expense of helping the public understand reality.
If you would like to read about the study yourself, I'd recommend clicking here. You'll be taken to an article that quite obviously is not unbiased, but it discusses the situation well and provides a number of links that will help you find other resources discussing these results. As a quick reminder for those of you who might not have ever heard of "meta-analysis" before, this is a type of analysis that looks at the results of multiple studies combined. Ordinarily, in a single study, we look at the results of a group of individuals and examine trends across those people or between groups of people within the sample. In this approach, individual studies take the place of individual people. There are a lot of promising aspects of this approach: it allows for a summary of results across the work of several researchers; if done in a particular way, it can increase sample size and statistical power; and it can keep people from looking only at one study in drawing general conclusions. Unfortunately, there are also a number of recurrent problems: authors decide which studies to include in their meta-analyses, which allows for misleading summaries; sometimes only severely underpowered studies are included in the meta-analysis; and oftentimes the authors of the meta-analysis combine different measures into a single outcome or combine highly different groups of people (e.g., individuals with different diagnoses) in an arbitrary manner based upon absolutely no empirical evidence indicating that such decisions are valid.
So, what is the problem with this meta-analysis that has me and so many others frustrated? Let's start with the results. Home births were associated with fewer complications for mothers (e.g., lacerations), fewer premature births, fewer low birth weight births, and a number of other important outcomes. Pivotally, however, home births were associated with a statistically significantly greater number of infant deaths between birth and 28 days. This, quite obviously, is important information to consider and the concept of conducting a meta-analysis on this issue is a reasonable one. The problem, however, as is so often the case, rests in the data that were selected for this study.
In this case, the primary issue with the data is that studies were included that did not differentiate between whether the births that took place at particular locations took place there intentionally. In other words, did the mother intend to give birth to her child at home or did something happen that resulted in an unplanned homebirth? Additionally, no information was given about whether or not home births were attended by a properly trained midwife. You might look at those distinctions and consider them trivial. After all, don't we all think of hospitals as the safe place to be? Isn't this finding consistent with what a lot of people would expect? It might be, but that is, in fact, the root of the problem. Our expectations do not align properly with reality and data. Hospitals are great and the world is a better place because obstetricians exist and perform miraculous procedures on babies and mothers in need. The thing is, however, countless studies conducted across the world that do, in fact, include all of the information left out of Dr.Wax's work, demonstrate that there is no difference in infant mortality rate between planned home births attended to by properly trained midwives and planned hospital births (click here for a description of one such work). In other words, Dr.Wax made misleading selections of studies and created a narrative that is highly consistent with the beliefs of people who have not had an opportunity to familiarize themselves with the research on this matter and, as a result, people - many of whom have no background in reading and interpreting research on their own - were left to hear his interpretations of other peoples' data and to decide for themselves based on nothing but their own intsincts whether or not he was right. If you did not know much about this topic and a doctor told you something was risky for your child, how easy do you think it would be for you to question that?
Anyway, as I said before, I am not an M.D. and I am not trying to convince anyone that home births are the way to go. Instead, all I am trying to do is provide another reminder that, although data are essential and must guide our decisions, it is also important for us to understand whether what an author says about the data is actually consistent with what the data say about reality. What we end up with far too often is a game of telephone and, by the time the message reaches a consumer in need of health care, it has such little resemblance to reality that there is essentially no chance of an accurate education.
My take home message here also is not that meta-analyses are inherently bad. Instead, my point is that meta-analyses are inherently dangerous. Why are they dangerous? Consider this: my first semester of graduate school, Chris Schatschneider told myself and my classmates taking his statistics course that, when he reads a study, he often reads only the method and results or at least spends most of his time on those sections. In unison, we all gasped. Who would want to just read the numbers? The author summarizes things so neatly and cleanly for us in the discussion and the abstract! The things is, over the years I have come to appreciate Dr.Schatschneider's point more and more. The method and results tell us what happened in the study. The abstract and discussion just tell us what the author said about what happened in the study (or what he or she wished/thought happened in the study). The former represents facts and the latter represents something much messier. In the case of meta-analysis, this point goes one giant step further: the original method and results are not even included in the manuscript. Instead, we see summaries of the results of articles previously published in other journals and we are left to assume that they represented valid choices for the analysis (and that better options were not left out) and that they were well conducted studies. Unless we go back and look at those studies, which almost nobody ever does, we are essentially reading one gigantic discussion section. That is undoubtedly convenient and, oftentimes, the meta-anslysis is fantastic...but sometimes it is not and a message then gets proliferated that serves only to obstruct science and mislead the public. If you want to read about an example of this directly relevant to clinical psychology, click here.
Ultimately, my hope here is simply that readers will continue to appreciate that, although we must use data to guide us, it is equally important that we be certain we actually know where those data are coming from, how they were collected, and what the results are actually telling us.
************
If you would like to learn more about this or other topics discussed on PBB, we hope you will consult our online store of scientifically-based mental health resources.
Mike Anestis is a resident at the University of Mississippi Medical Center and a doctoral candidate in the clinical psychology department at Florida State University.
Subscribe in a reader
Recent Comments