COVID-19 Models: What Makes Them Tick?

As the COVID-19 pandemic unfolds, questions about its likely course are much on our minds.

How long will it last? How bad will it get? And are we doing enough to flatten the curve?

These questions are not about the past, but about the future. Models are now frequently cited in public by elected leaders to inform expectations and justify policy decisions.

But the models themselves are poorly understood. To many, the models appear to be black boxes which somehow combine biology, mathematics, and behavioral factors to miraculously produce precise projections of disease and its outcomes.

But all models are not created equal. There are key differences between the models that are currently being used to guide the national conversation — differences that should be understood by the public.

To illustrate just how different models can be, consider two of the most publicly cited COVID-19 models – one from Imperial College London (ICL), and another from the Institute for Health Metrics and Evaluation (IHME).

Both models project what would happen under actions to mitigate transmission, such as physical distancing over a period of time. Both models produce similar looking graphs that show infections, and deaths, increasing and subsequently declining, over the course of the pandemic. But their innards could not be more different.

The ICL model is a mechanistic model. This means it replicates the person-to-person process by which disease is transmitted between individuals as they interact within their households, schools and communities.

Remember SimCity, the video game in which players designed cities, populated them with simulated people, and let them run under given budgets and social policies? The ICL model is SimCity plus an infectious disease. The “city” is the country or location being modeled.

Now, imagine trying to model a country like the UK with almost 68 million people, or a state like Colorado with some urban pockets of density along with large geographic expanses with different economic and behavioral patterns. Building a model of this nature requires a dizzying array of data including the population age structure and density, travel and commuting patterns, and school and workplace sizes. The ICL model sources all of these and more.

Once the population infrastructure is specified, the infection is added to the mix. Here’s where the new — and shifting — biological understanding of COVID-19 comes in. Modelers input its incubation period and transmission rate and generate anticipated growth in cases over time. These trigger further outcomes based on assumed rates of hospitalization, ICU needs, and other inputs  including fatality and recovery rates. Interventions to mitigate infection alter the model’s settings in multiple directions; for example, school closures eliminate interactions associated with school attendance, but increase household transmission risks. As more data becomes available, whether it’s new biological learning about the transmission rate, the possibility of re-infection, or compliance with physical distancing – these new data can further refine the model over time.

In contrast, the IHME model is an empirical model. An empirical model does not attempt to capture the mechanistic process by which disease spreads. It doesn’t get that deep into the details. Instead, it uses a mathematical formula to summarize the pattern of deaths in a location that has already crested the wave of the pandemic, and extrapolates it to other locations. Empirical models of COVID-19 have looked at the data from China and Italy to inform their projections of what is likely to happen in other countries. In terms of the SimCity analogy, the empirical model is about what happens in the game rather than how it happens. 

To model a new location, the IHME model squeezes or stretches the pattern of deaths so that its projection matches whatever data has accumulated there. Mitigating interventions can change the height of the peak or its timing. The model assumes that mitigation will have a similar effect on deaths as in other locations. Then, it projects further outcomes like hospitalizations or ICU beds based on estimates of the fraction of deaths among patients hospitalized or in intensive care.

The original IHME model was based on the experience of Wuhan, but updates have incorporated data from parts of Spain and Italy. While the model continues to evolve, it remains reliant on a specific form for the pattern of rise and fall in deaths that is data — rather than process — based. The IHME model cannot explicitly incorporate new learning about transmission rates or the possibility of re-infection. To the extent that these are reflected in the data, they are already accounted for. Similarly, the model cannot change behavior patterns to match a target level or duration of social distancing; it is just not that granular.

It is natural to ask which approach is to be preferred — a model that is more mechanistic or a model that is more empirical. 

A more salient question is whether we can trust any of the models to deliver accurate, quantitative predictions of the future course of the pandemic.  

All models simplify reality, make assumptions, and require reliable data. But different models depend on different input data streams. Thus, misestimated disease transmission rates will reduce the accuracy of the ICL model, while under-reported death rates will reduce the reliability of the IHME model.

At this point we might find ourselves feeling that all model predictions of the future of COVID-19 are likely missing the mark. But if we expect models to accurately predict numbers of cases, hospitalizations and deaths, we may be asking too much. No single place’s experience is the same as another, and responses — by individuals and countries — vary significantly. Even as we use the models to plan for the next weeks and months, we need to recognize what they can and cannot do. Knowing what goes into each model is the first step. 

Ruth Etzioni has spent the last twenty years developing and comparing models of cancer prevention, detection and treatment. Thanks to Roman Gulati and Eli Etzioni for their help with this report.