America's Coronavirus Data Debacle Before anyone declares that it is possible to predict that a country can safely open its economy by a given date because of what the models show—they better know how the models were built and be familiar with their driving assumptions. by Amitai Etzioni and Ruth Etzioni

Is it plat already?” millions of Americans learned to ask, as we live and die by statistical models. We are not exaggerating when we say “live and die,” because if the models on which the White House and governors rely on are faulty, if they predict a recovery before there is one, restrictions will be prematurely lifted, and many deaths that could have been averted, will take place.   

The leading scientist on the pandemic, Dr. Anthony Fauci put it succinctly, “models are only as good as the assumptions you put into them.” One tends to zoom over this short line but it speaks legions. It suggests that the outcomes of the vital statistical models vary a great deal not only on the basis of the evidence that one puts into them, the way one puts meat into a grinder, but also depending on the kind of grinder one chooses. Before anyone declares that it is possible to predict that a country can safely open its economy by a given date because of what the models show—they better know how the models were built and be familiar with their driving assumptions. The reason the White House model is unduly optimistic is because it assumed that the data China provided was reliable. Even later, as data from other nations were added, the model continued to greatly underestimate the numbers of those who will be infected and of those who perished. (On April 17, the Chinese upped their official tally of the deaths in Wuhan by a staggering 50 percent.) 

As a tool for policy guidance, the model the White House mainly relies on is held, to put it very gently, not in high regard among statisticians with experience modeling other epidemics like AIDS and Ebola. It was fashioned by the University of Washington’s Institute for Health Metrics and Evaluation (IHME). This model was one reason Trump has maintained that the governors were asking for many more ventilators, protective gear, and medications than they would need.  

As the Washington Post reported “Leaders in one state said Trump administration officials have used IHME’s numbers to push back and in some cases deny their requests for equipment and help. Officials in that state cited emails and documents in which federal officials highlight IHME projections as evidence the state needs thousands fewer ventilators and beds than the state’s models project.” Dylan George, who helped the Obama White House develop models to guide its Ebola response in 2014 added, that “As a result, you have every state trying to create their own models to anticipate their needs.”  

To gain a sense of the issues at hand, we must go into the statistical weeds. On April 2, IHME predicted that cumulative coronavirus deaths could reach as high as approximately 178,000. This prediction was reduced on April 5 to about 136,000. On April 8, this figure was scaled down to 60,145 (with the upper range cut, to about 126,000). Thus, in a few days, “the model changed its predictions by more than 33 percent.” Even the notorious weather forecasts are more reliable. 

Aside from being a poor predictor, a strong case can be made that the kind of model IHME provides and that the White House draws on, is the wrong kind of model to begin with. The IHME model takes the pattern of data of what happened in some countries, tweaks it to match the accumulating U.S. data, and uses it to project what will happen in the future. It does not examine what will be the outcome if various specific conditions would vary. For instance, if there would be more or less social distancing, then various levels of testing and of contract tracing. We need such a polity exploring model rather than a mere one that seeks to predict the future on the bases of past occurrences. The British Empirical College is an example of such a polity model; there are many others.  

Meanwhile, those of us who skipped the course on statistics in college, or did not have a chance to look up our notes recently, sadly better take all predictions about the course of the pandemic, especially those coming from the White House, with the biggest grain of salt we can find. And, to the extent that we have leeway, maintain virus safeguards longer than the headlines of the next days will suggest. 

Amitai Etzioni is a university professor and professor of international affairs at The George Washington University. His latest book, Reclaiming Patriotism, was published by the University of Virginia Press in 2019 and is available for download without charge.