Convalescent Plasma: Look Before You Leap

In the last few days I have been wondering how Michael Joyner and Arturo Casadevall have been feeling.

Joyner and Casadevall are the first and senior authors, respectively, of the report, “Effect of Convalescent Plasma on Mortality among Hospitalized Patients with COVID-19: Initial Three Month Experience,” posted on Medrxiv on August 12. The preprint server allows researchers to make their work public before peer review, and has become a popular source for producers and consumers of evolving research on the SARS-CoV-2 pandemic.

The report describes the experience of an Expanded Access Program, providing convalescent plasma from recovered COVID-19 patients to roughly 35,000 hospitalized patients over 3 months. All received plasma transfusions, so patients receiving the product could not be compared with those who did not get it.

Despite the lack of any comparison between patients who did and did not receive plasma transfusions, this report has formed the basis for the FDA’s Emergency Use Authorization for convalescent plasma to treat COVID-19 patients. It’s the report that was embarrassingly misinterpreted by the commissioner of the FDA, Stephen Hahn, and by administration officials. And it is the report that may well have spelled the death knell for controlled trials that were hoping to give us a convincing answer as to whether and how well the treatment actually works.

It is hard to imagine that the report’s authors actually intended for all of this to happen. Yet, by doing what medical researchers do all the time, they played into the hands of an administration hungry for anything that would show they were making progress against an intractable virus that has killed at least 200,000 Americans.

To understand how this unfolded and why it bodes so badly for the future, we need to examine the story that the report lays out. The report itself is mostly reasonably transparent, at least about the patients involved (roughly 35,000 over three months from April to July) and how sick they were (52% ICU and 27% on ventilators). The authors obviously recognized that they were reporting an experience rather than a randomized study, and that there was no comparison group since all included patients received at least one unit of plasma. But they still concluded that their findings provided “signatures of efficacy for convalescent plasma.”

What were these “signatures of efficacy”? According to the report and the FDA announcement, there were two: that patients given the product earlier in the course of their illness and those with high antibody levels in their donated plasma were less likely to die than those given plasma later or whose transfusions had low antibody levels.

More specifically, 8.7% of those given plasma within the first three days following their COVID-19 diagnosis died within a week, whereas 11.9% of those given plasma four days or later died. And, of the few patients (roughly one out of ten) for whom information on antibody titer could be retrieved, 8.9% of those in the top 20% and 13.7% of those in the bottom 20% died within a week. Notwithstanding the modest absolute numbers, the relative differences are clinically significant.

At first glance, these results might seem somewhat convincing.

I mean, if you wait to get plasma you do worse, and if your plasma is antibody rich you do better. Surely this must mean plasma works!

Not so fast.

There are major issues with these results that go beyond the (utterly justified) criticism that the data doesn’t come close to the established standard for evidence about novel treatments, the randomized trial. Indeed, the report’s authors lacked not only a randomized comparison between a treated and an untreated group, they were missing the untreated group entirely. So, they presented two alternative comparisons.

Each is problematic for its own reason.

The question of whether it is better to treat a patient earlier rather than later is universally challenging. In prostate cancer, a major controversy has raged for years about whether high-risk prostate cancer cases should be offered additional (adjuvant) radiation treatment right after primary surgery, or wait until their cancer relapses. A similar question occurs in breast cancer. But we can’t just compare patients given radiation in the adjuvant setting with patients given radiation for recurrent disease. They are simply not the same; indeed, some of the patients treated early would never have progressed to relapse even in the absence of radiation therapy.

In general, patients treated early in their disease course are a mix of those who would and would not be eligible for treatment later. In the case of COVID-19, patients at day 2 include those who would make it to day 5 or 10 without treatment and also those who would not! So, the analysis in the report that compares the patients treated within three days against those treated at four days or later is ill conceived. In the end, the results are basically useless.

Alright, you say, but what about the plasma titer comparison? The authors go to great pains to note that the titer was not known at the time of treatment and even use the term “pseudo-randomized” to describe this comparison.

I am going to leave aside the fact that the antibody titer analysis only included about 10% of the sample, and that it is therefore unclear how representative this group might have been, because there is a bigger issue here. This is the matter of how the high- (and low-) titer groups were defined.

There is no established threshold to define high- versus low-antibody titer for SARS-CoV-2. The authors chose the top 20% to be the high-titer, and the bottom 20% to be the low-titer group. This seems reasonable until you realize that they could have chosen the highest and lowest 25%. Or the highest and lowest 33%! Maybe they even tried these thresholds but presented just the results provided. We don’t know. Regardless however, the results show only that survival was improved in the high-titer group compared with the low-titer group. We don’t know if those who received low-titer samples did any better than if they had not been treated at all. In fact, there are real concerns about whether low-titer plasma might induce a harmful effect, referred to as antibody-dependent enhancement (ADE) of  coronavirus infection.

At best, we can only conclude from the titer comparison that high-titer plasma, amounting to approximately one-fifth of the plasma samples, might possibly constitute an effective treatment. There is absolutely no signal of efficacy for low-titer samples. But the FDA’s Emergency Use Authorization actually emphasizes that it covers all plasma samples, reflecting the report’s conclusions which also do not differentiate.

I don’t have a crystal ball and I can’t read the minds of the report’s authors in terms of what other comparisons they may have considered, or what they might be working on now to deliver more reliable answers. But their choices and their conclusions thus far are entirely consistent with an academic culture that rewards the finding of significant, positive effects associated with new treatments. It is a culture that encourages positive spin about novel medical interventions and overstatement of marginal clinical results.

In normal times, we might take the admittedly cautious but still overstated conclusion of benefit in this report with a grain of salt and modulate our expectations of effectiveness in practice accordingly. Doctors wouldn’t rush off and change prescribing habits instantly on this thin body of evidence – not until they could see a clear and convincing difference between patients who got convalescent plasma, and patients who didn’t.

Clearly, these are not normal times. Scientists need to do whatever they can to make sure that their science is not misinterpreted, exaggerated, or even hijacked to make a political point. I sure hope that Drs Joyner and Casadevall are alarmed that their research featured so prominently in what can only be considered a dress rehearsal for the vaccine study results that are expected later this fall.