I made a website to guess tomorrow’s Covid-19 case count

"People have over- and underguesstimated the number of new cases, but not by much. There were only a few occasions where the actual case count didn’t fall within one standard deviation of the mean of guesses (represented by the whiskers around blue bars on the plot). Granted, the daily standard deviation tends to be large (on the order of a few thousand), but still, I’m impressed."

The author almost calls this out but there is a natural range that the data is expected to fall into given the previous data. (Standard deviation across the whole time series isn't actually that helpful). This leads us to what should be everyone's first baseline. Last-time-step prediction. If your prediction is simply the previous true value how good are your predictions? The mean absolute error over some time window is a decent place to start when you're measuring quality.

This predict-the-last-timestep model is often surprisingly good. If your prediction horizon is really only one timestep then its easy to fool yourself that your model is doing well. This is why models are usually judged on how quickly the diverge from the true data over as long a time horizon as possible. If your model is capturing underlying dynamics then it should be better at predicting long time horizons.

There are, of course, many ways to then start building up better and better models for use as baselines, and even relatively simple models (random forest over a moving window for example) can produce pretty good predictors. I'd love to see how humans do in compared to those over medium to long term predictors of covid case counts!

Predicting next day's cases doesn't seem like a good use for the wisdom of crowds. It's too easy, since there is so much inertia. (Basically all the errors here are from people not taking into account the biases from day of week). And the problem with predicting further into the future for a site like this would be that it will delay gratification for too long: I can wait until tomorrow to find out how well I did on today's predictions, but I probably don't care to come back in two weeks.

You came up with a clickbait title for a blog post?