Getting it Right: Walking the Diagnostic and Serologic Testing Tightrope

Last week I felt like I was coming down with something. My throat hurt and I had a bit of a dry cough.

A dreadful feeling set in. Obviously, I had COVID-19.

My mother and her partner, both in their 70s, are currently living with me in Seattle. My friend’s son had recently returned from New York. A few days earlier, I had chatted briefly with him (outside, keeping my distance) not realizing he would be diagnosed shortly thereafter. These factors made me eligible for testing through my employer, Fred Hutch.

Thankfully, I was negative.

I never thought that my test might have been a false negative, that I might actually be harboring the virus and still capable of spreading the virus to those around me. No, in this case I had confidence in the accuracy of the result. I heaved a sigh of relief and returned to my socially distanced life. When my symptoms cleared up quickly, it seemed to confirm that the test had been accurate.

But many people who get tested for COVID-19, and are told they do not have it, are not so lucky. It has been estimated that as many as 30 percent of patients with negative test results are “false negatives.” That means they actually are infected, and able to infect others.

Many of us trust the tests we take, and believe the yes-or-no results they return. But we need to be more curious.

In the fraught deliberations about the optimal path to re-opening the US, one theme has emerged louder and clearer than any other: testing, testing, testing. In practically every forum and on practically every stage a consensus has emerged that if widespread rapid testing is not made readily available, and easily accessible, the US cannot reopen for the business of everyday life. But it’s not that simple.

There are two main types of tests. Diagnostic tests are designed to tell you whether you have the virus: are you sick now? Antibody tests, also called serologic tests, are designed to tell you whether you had the virus in the past: were you sick and are you now immune?

It is not altogether clear which test is the magic bullet that will return our lives to normal. Right now in Wuhan, both are being used; diagnostic tests to find existing asymptomatic cases and serologic tests to learn how much of the population still remains susceptible.

There are at least 95 antibody tests and the number keeps going up. Yet, we have little information about how well they work. This is most certainly not front and center in press releases and advertisements.

We need to be much more curious about this.

No test is completely accurate. There are false negative results, meaning that the test is negative, but you are sick. There are also false positive results, meaning that the test is positive, but you are just fine. And some tests have much better performance than others; their error rates are lower.

Development of new tests involves rigorous assessment of these two types of testing errors, by examining the test results for people known to have the disease and known to be disease-free. The process of actually creating the test involves a lot of tweaking and refining to produce error rates that are acceptable. It’s a little like having a custom jacket tailored to your size and build. The jacket fits you well, but how will it fit the average person?  

To assess this, the test performance needs to be confirmed, or validated, in a whole new sample, with no tweaking allowed. This is like seeing how well your tailored jacket, sitting in a boutique, fits folks coming in off the street. Many of the new antibody test have not been validated. This means that the error rates cited (if you can find them) are likely to be too low, making the test performance seem overly optimistic.

Different types of testing errors have different consequences. Sometimes people worry more about a false negative than a false positive. Women going for their annual mammogram are far more concerned about a false negative – not finding a silent tumor – than a false positive – having an unnecessary biopsy for something that looked suspicious but was actually nothing.

What are the consequences of testing errors in the case of COVID-19 tests?

For diagnostic tests, a false negative test has two potentially fatal consequences. First, it means that an infected person can continue in principle to keep spreading the virus; second it may delay the opportunity to offer potentially life-saving treatment. A false-positive test, on the other hand, is less consequential; it will increase the likelihood of self-quarantine for a time. Thus, for diagnostic testing, it is clear that we need tests that are highly sensitive, meaning that they almost never miss a case of the virus.

Interestingly, the opposite is true for antibody tests. Let’s first note that these tests are really only useful for asymptomatic individuals who do not know whether they had COVID-19 in the past. A false negative antibody test means that a person who has had the virus and recovered gets told they never had it. This is unfortunate for the individual who is immune and does not know it, and will certainly put a damper on their lifestyle. If the opportunity exists to do serial testing, then requiring more than one negative test result could bring down the fraction of people so affected.

In the case of antibody tests, the real problem is the false positives. The false positives will be told that they have antibodies but in fact they don’t. They will likely resume life with a vengeance (I know I would), distancing themselves from the memories of social distancing, potentially exposing themselves to becoming infected and then spreading the virus to others. So, in the case of antibody tests, we really need to focus on making sure that the false positive error rate is exceedingly low.

How low is low enough? It turns out that we can actually answer this question, but it requires knowing what fraction of the population is still susceptible. Even in Wuhan, only 2-3 per 100 people tested had antibodies, so the vast majority were still susceptible. In the state of Washington, the early center of the US outbreak, the fraction of people with antibodies is closer to 1 per 1,000. This means that even if the false positive rate is very low at 1%, 9.9 people per 1,000 tested will be wrongly informed that they have antibodies; scaling up, this would amount to 9,900 per million people tested going about their lives under the mistaken impression that they were immune. It is easy to see how this could provide sparks for new clusters of infection in the absence of a vaccine or other means of eradicating the virus.  So, when the prevalence of true cases is low, the antibody test has to have a vanishingly small false positive rate.

When we think about getting a test, particularly for a disease that scares us, like COVID-19 or mammography, we naturally think about sensitivity – if I have it, will the test detect it?  We want to know that there will be no false negatives. But false positives can have enormous costs, not only for individuals but for society. A study of women getting mammograms estimated that, over ten years, at least half of women screened annually received at least one false-positive recall. Many went on to have unnecessary biopsies, with all the pain, cost, and anxiety these create.

There are many other questions about the new antibody tests, including the extent to which they actually measure immunity. This only reinforces the message: Be curious; Trust less.

I should have heeded my own advice when I went for my own COVID-19 diagnostic test. It’s human nature to breathe a sigh of relief, like I did with that negative result. But it’s not too late to be skeptical. I’ll do better when I get tested for antibodies one day. First, I’ll ask which test is being used, and whether it is from a reputable source. Then I’ll read the fine print about the testing error rates.