Inkwell: Authors and Artists

    
There are some stupid, aloppy statements in Gelman's blog post. I
don't think it's going to be productive to engage him.
  

    
I think he was getting at the mismatch between the way we study
disease and its treatment--mostly through the double-blind
controlled trial-- and the kind of disease that Julie was writing
about. He is correct about this, and as time goes on, more diseases
will surface that are not suitable to the clinical trial--because
their symptoms are heterogeneous (as in CFS/ME), or because the
pathways that lead to the symptoms are heterogeneous (as in
depression) or because the major symptoms are subjective, and so on.
Thedouble-blind controlled study is aa historical, rather than a
scientific development, after all, and there's no reason to think
it's a good way to assay every disease. With these mismatches on the
rise, history may be  in the process of correcting itself.

One thing in that article that bothered me. He desribes the clinical
trial by saying "you gather a bunch of patients and randomly assign
some to a control group, which receives no treatment, and some to an
experimental group, which does." But this is not accurate. Generally
the control group receives treatment, but it is placebo treatment.
This is not the same as no treatment, not even close. And especially
with disorders like CFS/ME, placebo responses can be very strong.
I'm pretty surprised the New Yorker let that through, but then again
I can say from personal experience that the fact checking on the
website is nowhere near as robust as the fact checking for theprint
version.
  

    
Yeah, that's a startling and substantive error.
  

    
I did email him. Here's the exchange.

>
> Prof. Gelman--I'm catching up with your critique of clinical
trials, which I think is well taken. But with me you're preaching to
the converted, and there's an error in your New Yorker piece that is
not going to help you with the unconverted. Clinical trials in
general do not compare a treatment group to a no-treatment group.
They compare a treatment group to a placebo group. The treatment
effect is not the change in the treatment group minus the change in
the no-treatment group, but rather treatment minus placebo. That
means that placebo effects are part of every treatment effect.
>
> Two implications for your well-taken critique: 1. The clinical
trial assumes that the placebo effect is noise, and is designed to
filter it out. Obviously a problem if indeed there is a difference
between placebo and no treatment (which there is), because it means
that an entire category of potential treatment is simply being used
as a stalking horse rather than being investigated. 2. Clinical
trials in general don't have a no-treatment condition, which means
that the actual signal, of medication, placebo or both, can't really
be discerned. That is a crucial weakness in an experimental design
that is the foundation of medical research. 


His response:

Hi, Gary, thanks for the note.  Good point:  "control" can include a
placebo or indeed any alternative treatment.  I don't think I can
easily correct on New Yorker site but I can correct on blog.
Andrew


I think he misses the point. "Control," in the context of clinical
trials nearly always involves a placebo treatment. Sometimes it
involves a known drug.  It never, that I know of, includes an
alternative treatment. I did not bother responding. 

But the larger point (his and mine) remains: conditions like Julie's
do not lend themselves well to the current regime.
  

    
I read it as "alternative" in the sense of "some other drug," not 
"alternative" as in some new agey treatment or whatever. That is the 
basic meaning of alternative, I think.
  

    
Thanks for writing to him, Gary.

His review was an odd experience for me. Of course, it's delightful
to be positively reviewed in the New Yorker! But it's very odd to
have the PACE trial held up as an example of the limitations of
randomized control trials, when the real problem here isn't that,
but that it was so dreadfully done. 

I just posted this comment on Gelman's site:

Hi Andrew,

Thanks so much for this review! Three points:

1. There was an important error in the New Yorker version. It says:
One of their chief objections is that the pace research team, while
collecting its data, changed the main metric of “recovery” from
objective measures, such as fitness and employment status, to
subjective ones, such as the patients’ self-evaluations over time. 

The primary outcomes were always the subjective ones, so this isn't
accurate. However, they weakened their thresholds on the subjective
measures for "recovery" (as well as for every other benchmark in the
trial) enormously -- so much so that (as you point out in your
longer version) patients could get worse on fatigue and physical
function and still be considered "recovered." When the objective
measures didn't improve, the researchers dismissed them, even though
they themselves had chosen them.

2. You make the point that RCTs tell you less when you're looking at
heterogeneous groups. But in the PACE trial, the researchers chose a
broad definition of ME/CFS that exacerbated this problem, including
a far more diverse population than was necessary or appropriate.

While it's true that ME/CFS (or SEID) may well be a cluster of
related illnesses, it's also the case that definition really
matters. The Mayo Clinic definition that you offer is pretty much
the worst one out there -- it negates itself, giving a medical
explanation by demanding that there is no medical explanation. The
one the PACE trial used isn't quite that bad, but it's close,
requiring only six months of disabling fatigue as the primary
symptom. That can include a significant percentage of depressed
people, which is most likely part of why the treatments appeared
effective -- increasing activity is a good treatment for depression.
And that definition doesn't require the hallmark symptom of the
disease, exercise intolerance (as the term SEID emphasizes). 

Though you se the term SEID, you didn't use the SEID definition,
which requires exercise intolerance (the hallmark of the disease)
along with fatigue, unrefreshing sleep, and either cognitive
problems or difficulties regulating heart rate or blood pressure
when standing. The SEID definition is designed to be simple for
clinicians to use and able to catch all patients with the disease,
at the cost of including some others. That's useful for clinical
practice, but not so useful for research. The best current
definition for research (IMO) is the Canadian Consensus Criteria,
which is more restrictive and also more complex:
http://me-pedia.org/wiki/Canadian_Consensus_Criteria. It adds
requirements for pain and neuroimmune symptoms.

Even among patients who meet the Canadian Consensus Criteria, there
is likely heterogeneity, but far, far less so than simply looking at
really tired folks.

The PACE investigators and their intellectual heirs persist in using
very broad definitions to this day.

3. Finally, I just wanted to point out that actually you do have
access to a chunk of the data. Patients managed to get the data
relevant to evaluating recovery by the original defintion, which
they made publicly available here:
https://sites.google.com/site/pacefoir/pace-ipd_foia-qmul-2014-f73.xlsx?attred
irects=0.

The data showed that accordingly to the original definition of
recovery, the researchers had null results. That is, neither CBT nor
GET led a statistically significant number of patients to "recover."
  

    
Gary, I'm trying to make sure I understand your point. Are you
essentially saying that placebo effects can be valuable?
  

    
Yes, and the clinical trial design implicitly endorses that view by
assuming that placebo responses are part of every response to a
treatment. But, perversely, the design also creates the impression
that placebo treatments are not valuable by treating them as noise.
Which often means researchers are on to the next drug rather than
trying to figure out why the placebo group improved.


The cynical view is that the reason for our relative ignorance about
placebo effects is that there is no money in sugar pills. But that's
only part of the explanation. It's also the case that modern medical
research, and especially the clinical trial, is extremely ill-suited
to placebo treatments. The farther away you get from diseases with
clear molecular causes that are remedied by treatments aimed at
them, the less valuable that approach is. 
  

    
Fwiw, One study showed that the placebo effect is weaker with
quantity syndrome treatments in typical:
https://www.ncbi.nlm.nih.gov/m/pubmed/15784798/
  

    
That is an interesting study. And it's not unlike other
meta-analyses of placebo effects: they tend to indicate that placebo
responses are not as strong as individual studies make them out to
be. And in general, finding a placebo signal has proven to be
difficult. I'm not sure what this means, but it isn't entirely clear
thata it means that placebo effects are weak. 

REading between the lines here, Julie, I wonder whether you have a
sense that if there were a strong placebo response in CFS/ME, that
this would somehow deligitimize the diagnosis or the people  who
have it.
  

    
Not necessarily. But I do think it's interesting that there's a
general assumption that placebo effects in ME/CFS would be
particularly strong, even though the evidence doesn't support that.
Why did you say that?
  

    
Julie will be interviewed on Your Call on KALW today at 10am.
  

    
(streaming live at <http://www.kalw.org> - 10-11am Pacific
  

    
Apologies for being late to close this discussion and thank Julie
Rehmeyer and everyone who participated! 
  

    
Thanks so much, everyone! It was a great conversation.

And by the way, the interview is available here:
<http://kalw.org/post/your-call-julie-rehmeyers-long-road-recovery-chronic-fati
gue-syndrome#stream/0>