The Secret Lair

This article marks the beginning of a periodic new series here in The Lair: The Secret Journal Club. Journal clubs are a grand tradition in medicine, whereby physicians and students gather to critique a selection of journal articles – not merely read them and take what they say at face value, but also to dissect their techniques and message to better understand if their conclusions are valid and applicable to what we do as doctors. As previously noted, I contend that much of media reporting on medical articles is plagued with oversimplification and sensationalism, which leads to news stories that are just plain wrong. Although there are some barriers, anyone regardless of their training can approach medical articles and get some reasonable understanding from them, especially when you know what to look for.

To that end, each edition of The Secret Journal Club will tackle a medical journal article and briefly look at what it says, what it’s biases are, and whether or not it’s conclusions are valid. We’ll divide each discussion up in to six sections, four of which mirror the format typical of such articles: Background, Methods, Results, and Discussion. Since the things that bookend the article – the nature of the journal itself, the affiliations of authors, and where the money comes from for the study, for example – are also critical and often overlooked, we’ll make note of those as well. Last but not least, we’ll give a critique of both the article itself and both the context and the media reporting surrounding it.

My hope is that, whenever possible, the full text of the article will be freely available for you to read; in many cases, such as today’s selection, this isn’t the case without a subscription.¹ In those cases, I will at least send you to the abstract of the article, and would encourage you to seek the full text out through your local library or other source if your interest is piqued.

Let’s do this.

Bookends – The article in question today is Active Albuterol or Placebo, Sham Acupuncture, or No Treatment in Asthma, which was published in the July 14, 2011, edition of the New England Journal of Medicine. The abstract can be read here, and the full article is sadly accessible only with a subscription.

Most people have heard of the NEJM, and it is considered to be the highest impact medical journal in the world, meaning that articles published on it’s hallowed pages are the most cited by other articles in the medical literature. This means that they win the popularity contest, but this doesn’t guarantee that the conclusions their studies come to are always valid, as we are about to see.

Interestingly, only two of the eight authors are actual medical doctors. The senior author is Ted Kaptchuk, who is not a physician but carries the title of Doctor of Oriental Medicine, and specializes in researching the placebo effect. This is important to note in terms of the agenda of the trial. It is also important to note that since the placebo effect as seen in clinical trials is really an artifact of the trial process itself, this doesn’t seem like a thing that particularly needs researching, unless one believes magical things about it… which it seems apparent that Kaptchuk does.

The study is funded by the National Institutes of Health’s NCCAM branch: the National Center for Complimentary and Alternative Medicine. NCCAM has a track record of publishing good studies on so-called CAM modalities. They also have a track record of including researchers who aren’t so much interested in doing good science to figure out if CAM modalities work, but instead already believe that said modalities are effective despite evidence to the contrary, and so end up spending taxpayer money on studies to try and justify those beliefs. All that noted, as grant sourced money through the NIH, this is pretty much as unbiased of a funding source as one can get.

Background – The stated premise of the study was to see if in asthma, placebo therapies could result in objective improvements in lung function when compared to active medication. The authors attempt to redefine what the placebo effect actually is in this section of the article, including mixing up the two different scientific and therapeutic meanings of the phrase:

Placebo effects (i.e., benefits resulting from simulated treatment or the experience of receiving care) are reported to improve signs and symptoms of many diseases in clinical trials and in clinical practice. On this basis, the accepted standards for clinical-trial design specify that the effects of active treatment should ideally be compared with the effects of placebo. Despite this common practice, it is unclear whether placebo effects observed in clinical trials (or those that presumably occur in clinical care) influence both objective and subjective outcomes and whether placebo effects differ from the natural course of disease or regression to the mean.

This automatically raises concern about their agenda here – this basic premise of the article makes no rational sense. Asthma is a disease where the airways of the lungs constrict and become inflamed, which can be objectively measured with airflow testing called spirometry and can be treated with medications to ease the constriction (bronchodilators, such as inhaled albuterol) and inflammation (such as inhaled steroids). Inert, placebo therapies won’t fix these problems, and so would not be expected to make an asthma patient’s pulmonary function tests better… unless one were to believe magical things about the placebo effect.

Methods – The study was done over a two year period, and was a randomized, double-blinded, crossover pilot study. This means that in the hierarchy of evidence-based medicine, it is fairly solid – participants were randomly assigned to therapies, and both they and the people administering the therapies did not know if they were active or inert. The description of this being a pilot study implies it is a small, proof of concept study that could pave the way for larger studies in the future.

The active medication used in the trial was inhaled albuterol – a cheap, easily available and highly effective bronchodilator. One placebo intervention they used were inhalers containing no albuterol but otherwise looking and functioning identically. A second placebo intervention was that of sham acupuncture, which is a very interesting choice. “Real” acupuncture itself is considered to be effective therapy by some, although good medical evidence has repeatedly demonstrated that it works no better than placebo interventions. Even for acupuncture proponents, it is accepted that doing acupuncture “wrong” (either with needles inserted incorrectly, or using needles with retractable sleeves, similar to how a prop knife works) is a highly ceremonial but inert process.

The study screened 79 patients who met the American Thoracic Society criteria for mild to moderate asthma, and had a stable regimen of asthma medications for at least 4 weeks prior to the trial. The goal was to find people who could have their bronchodilator medications safely stopped for a period of time, but who’s lung function would worsen just enough so that the worsening could be detected by spirometry testing and then fixed by a treatment of albuterol. The specific value looked at on spirometry was the FEV1, or the amount of air a patient can forcibly bow out from their lungs in one second. 46 patient met this criteria by having their short acting bronchodilators stopped for 8 hours and long acting ones stopped for 24, having their FEV1 tested, then given an albuterol treatment and having their FEV1 improve at least 12% after the treatment.

Of those 46 patients, half (49%) were only on a short acting bronchodilator, a quarter (23%) had inhaled steroids on top of that, and the remainder had both of those plus a long acting bronchodilator or other medication. Since only the bronchodilators were stopped for the study interventions, that means that half of the patients stayed on their usual steroids for asthma therapy, which could easily have affected any results. Additionally, 62% of the patients were white, while in the general population asthma disproportionally affects those of African or Hispanic descent, meaning that the patient sample is not representative of the demographics of asthma in the real world.

The study patients then underwent four different interventions: treatment with an albuterol inhaler, treatment with a placebo inhaler, a sham acupuncture session, or no active therapy. Each patient went though a random series of the four interventions three separate times, for a total of twelve interventions each. Prior to the intervention, the patients stopped any bronchodilators they were taking as they did for the screening procedure. After undergoing the intervention, they had spirometry done to determine the change in their FEV1, as well as a questionnaire that asked whether they felt any improvement on a 1-10 scale, and to guess whether they had received effective treatment or placebo.

One major pet peeve of mine is when authors choose to include certain aspects of their study in an “online appendix”, as opposed to in the article itself. Sometimes, this can be justified in terms of space considerations or having extra data that is only peripherally related to the main point of the trial. That noted, in this case key things such as the specific sham acupuncture technique used and the criteria for patient selection were left out from the body of the text, which is inexcusable from my perspective.

Results – There were several results that were very striking. The first is that of the 46 patients randomized into the study, 7 (15%) dropped out. The striking thing about this is not that such a proportion of people left the study – this is a common occurence in any clinical trial – but rather that this is only mentioned in one of the figures and not at all in the text of the article; it is noted in the appendix that these patients were not included in the final analysis. Even there, there is no way to tell what the baseline treatment was of the dropouts, how far they made it in the study, or why they dropped out.

The next result of note was that among the four interventions, only the active treatment of albuterol caused a meaningful improvement in lung function, as measured by an increase of 20% of the measured FEV1. Both placebo interventions and the no treatment arm saw a much smaller increase of about 7%, and the difference between these and the albuterol result was statistically significant.

The last interesting result was that with the three intervention arms, patients reporting statistically similar levels of feeling better of 45-50% on the scale used in the study. These were all statistically significantly higher than the arm that received no intervention, but it is important to note that even that arm showed a 21% improvement.

Discussion – The discussion section of this article is fascinating, because it mixes sober assessment of the trial with frankly weird statements that seem to be spinning their results to support the beliefs of the authors about placebos. The most confusing part of this is when they describe the difference between the two placebo interventions and the no treatment arm. In the Background, they specifically describe the placebo effect as relating to “simulated treatment or the experience of receiving care”. However, in this section, they take great care to state that the no-intervention arm was not a placebo intervention, despite the fact that by their own definition, the no treatment arm still underwent the experience of receiving care and so involves the placebo effect. In particular, they state:

For the objective physiological outcome (change in FEV1), there was a powerful medication effect (drug vs. placebo) but no placebo effect (no difference between placebo and the no-intervention control). For the subjective outcome, the placebo effects were equivalent to the drug effect, and all were greater than the no-intervention effect.

This is a spin-laded interpretation. For the change in FEV1, there was a 7% improvement in the three non-albuterol groups, all of which were technically placebo groups, so there was an observed “placebo effect”. There is no such thing as a “no-intervention effect”, which the authors seem to have fabricated as a concept because it suits their beliefs about how placebos work.

In their summation, the authors do distill out the key finding of objective versus subjective improvement, which is the most important clinical take-home lesson. Then, their ruminations go off the rails again:

Many patients with asthma have symptoms that remain uncontrolled, and the discrepancy between objective pulmonary function and patients’ self-reports noted in this study suggests that subjective improvement in asthma should be interpreted with caution and that objective outcomes should be more heavily relied on for optimal asthma care. Indeed, although improvement in objective measures of lung function would be expected to correlate with subjective measures, our study suggests that in clinical trials, reliance solely on subjective outcomes may be inherently unreliable, since they may be significantly influenced by placebo effects. However, even though objective physiological measures (e.g., FEV1) are important, other outcomes such as emergency room visits and quality-of-life metrics may be more clinically relevant to patients and physicians. Although placebos remain an essential component of clinical trials to validate objective findings, assessment of the course of the disease without treatment, if medically appropriate, is essential in the evaluation of patient-reported outcomes.

This is a worrisome and telling series of statements. Here, they recognize that how asthma patients feel do not necessarily correlate with how well their lungs are doing, which is measured by the FEV1. The obvious conclusion is that the most important thing to rely on in assessing asthmatic patients are objective measures of lung function, because those will tell us how sick a patient truly is and how close they may be to a life-threatening asthma attack. Instead, the study authors suggest that these objective measures aren’t as important to patients as how they feel, so we should focus on things that assess the less-reliable indicator of how sick they may be, which is how they are feeling.

Um, what?

My Critique – This is simultaneously a great and horrible article that’s going straight into my teaching file.

It is great because it has a well-executed design that very clearly shows that albuterol helps fix the lungs of asthmatics while inert interventions really don’t, and that at the same time, patients receiving those inert interventions can still subjectively feel better after receiving them, sometimes dramatically. Often, physicians rely on asthma patients reporting their symptoms as the main measure of their clinical status, and this is good evidence to suggest this is unreliable. We need to be tracking objective measures of how well the lungs are doing instead. While spirometry to objectively assess someone’s FEV1 is not easily accessible for everyone, the simpler and inexpensive peak flow meter is a device that patients can use to track their lung function objectively at home, and this is something that all patients with asthma should do.

It is horrible both for some aspects of the study design, and the conclusions the authors draw. In terms of demographics, the patients are not a good sample of the genetic backgrounds of those who tend to develop asthma, especially severe disease (more often in people of African or Hispanic origin as opposed to Caucasian). A chunk of the patients left the study and aren’t accounted for in the text of the article. The authors want to redefine the placebo effect as being improvements seen with specific inert treatments only, that can somehow be extrapolated directly from a trial into clinical practice. Furthermore, they seem to suggest that making patients feel better with placebos is more important than actually fixing what’s wrong with them by using interventions that are proven to work.

I’m not sure about the rest of you, but as both a physician and someone who’s had asthma since 4 years of age, I think the focus of asthma management should be on making sure people’s lungs are working right and preventing hospitalization and death by using what works – objective measures, and effective medications. The fact that our taxpayer dollars are being spent by people who think otherwise is a bit of a problem, especially as they consider this to be a pilot study for future work in the same vein. The media has advertised this study as evidence that placebos are “powerful”, with the implications that doctors should start treating people with inert therapies to make them feel better. That’s both unethical and bad medicine.

Reference: Wechsler ME, Kelley JM, Boyd IO, Dutile S, Marigowda G, Kirsch I, Israel E, & Kaptchuk TJ (2011). Active Albuterol or Placebo, Sham Acupuncture, or No Intervention in Asthma. The New England Journal of Medicine, 365 (2), 119-26 PMID: 21751905

1. While most journals aren’t at the stage of being open access yet – especially because the people who publish them are interested in making money from them – most major journals will make specific important of seminal articles available for free online. [↩]

Tags: asthma, John Cmar, NEJM, New England Journal of Medicine, placebos, sham acupuncture, the placebo effect, The Secret Journal Club

This entry was posted on Thursday, August 18th, 2011 at 8:00 am and is filed under Monologuing, Science. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.