Forecasting s-curves is hard
constancecrozier.com
constancecrozier.com
Smoothing helps. Estimating error helps. Domain knowledge helps. Experience in applying the model helps. There are techniques other than the one demonstrated in the article. Sometimes those help. There are also professionals who create models which are used to support the bias of the modeler or the modelers employer (i.e. cherry picking).
Politicians and pundits more often than not take the results of these models and draw conclusions which are unwarranted or at least highly uncertain without mentioning the uncertainty or only giving it lip service.
suggesting that only an "expert" can possibly fit the data correctly is the wrong conclusion
Of course in many cases there are external bounds that are similarly useful, for example when you know the maxima (e.g. full population size). Then the question to the early data is simply whether a fit to an exponential is "better" than successively higher order polynomials.
Currently, this forcasting information seems not particularly helpful at all for fitting epidemiological data for Covid19 since nowhere do we have data for a case where strict distancing hasn't been an effect of high death and hospitalization rates well before 50% of the population would provide any effective immunity.
On the other hand, I realize most experts carry the baggage of unrecognized assumptions in their toolkit, which need to be challenged (at the first principles level) by people outside the domain who don't carry the same baggage.
The ideal situation would be to leverage the expertise of both epidemiologists and non-epidemiologists on a common platform (that is not Twitter) and have them check and build on top each other's work.
That’s one of the reasons why some of the governors in the US are getting accolades as their approach to communication contrasts against other political figures with a more political focused approach.
Capturing and interpreting risk appropriately is more important than being right. The various models scoped the problem, but it’s hard to model things like a complete breakdown of federal response.
[1] https://www.amazon.com/Superforecasting-Science-Prediction-P...
[1]https://en.wikipedia.org/wiki/The_Good_Judgment_Project
[2]https://en.wikipedia.org/wiki/Intelligence_Advanced_Research...
I participated in the initial part of the project, figured out the winning strategy very quickly, and then dropped my participation immediately. How to be "super at forecasting" in Tetlock's eyes? Have lots of spare time in the middle of the day when professional are at work, because that is when questions get posted, and easy points can get made. (To win a prize that is less than the hourly rate of a professional.) Hmm, no wonder that the conclusion was that people with no expertise but plenty of spare time can beat the pros...
https://twitter.com/outa_joseph/status/1250592983659024385?s...
(Sorry. I take it you would like them to be complete garbage.)
There are two main ways in which a model can give bad results.
1. It can be a fundamentally bad model. E.g., Kelvin had a perfectly reasonable model of the long-term evolution of the temperature of the earth from which he deduced that it couldn't be more than a few hundred million years old, probably less. The problem was that he didn't know about radioactivity, which turns out to make a tremendous difference.
2. It can be a reasonable model but the parameters can be hard to estimate accurately, and this can lead to inaccurate predictions. In some cases, small errors in the parameters lead to large errors in the predictions. For instance, if you try to forecast the weather a year ahead then you will get all the details wrong.
This article is about the second kind of problem, and that's the one that shows the most dramatic degradation as you look at longer and longer intervals.
Different properties of the model output can show very different sensitivities to the inputs. For instance, suppose you make a model of the solar system and try to predict where the planets will be ten million years from now. I expect (though I haven't actually done the relevant calculations) you will find that tiny errors in your measurements of the planets' present positions and momenta will lead to very large errors in estimating how far around their orbits the planets are, and probably also in estimating the precession of the orbits. But I bet that almost every simulation run will put the planets' distances from the sun at about the same values, which will be about the same as they are now, and I would bet very heavily that those distances will be a pretty good match for reality. (Barring utterly out-of-model events like the human race getting technologically advanced enough to move planets around and finding it useful to do so.)
Or suppose you make a model of the weather and try to predict things five years from now. The model will do very poorly at predicting which days will be rainy in (say) New York, and if you run it many times with tiny changes in its parameters it will give completely different predictions. But it will very consistently tell you that in New York it'll be warmer in July than in January, and it will almost certainly be right; it will do pretty well at predicting rainfall in the Sahara Desert, and it will almost certainly be right about that. Some things behave chaotically, some things not so much.
So, what about those climate models? Weather is chaotic: small variations quickly lead to huge changes. Climate is much less so: to whatever extent the models are fundamentally accurate, small changes to the parameters tend to produce small changes to outputs like global mean temperature ten years later.
How fundamentally accurate are they? They seem to be pretty decent; when you go back and look at the predictions of old models for what are now past years, they do quite well. (To predict the climate you need to predict human behaviour, because human behaviour affects the climate. Some of the predictions of human behaviour that went into the models were wrong. So you can get a sort of halfway house by running an old model but using now-historical information about things like how much CO2 we would put into the atmosphere. Unsurprisingly, this improves their performance relative to just using the old models as they stand.) See e.g. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019... for some analysis.
So: no, the sort of effects described in the article don't mean that climate models are complete garbage. The other sort of problem (fundamental errors in the model itself as opposed to its parameters) could make them garbage, but the evidence available so far suggests that even decades-old models were pretty good in this respect.
The right answer is that, just like COVID models, of course the climate models will turn out to be completely wrong in the exact details, in the scale, in the timeframe, and possibly even in the endpoints.
COVID is actually supremely easier to predict than climate. The two fundamental problems with most COVID models are that 1) the positive test counts don’t mean what you think they do, and 2) because of that, everything that is based on positive test counts being relatively meaningful is therefore totally wrong.
The climate models, since they were created by humans and not God, will likewise have almost innumerable errors which will cause all sorts of problems both minor and major with their predictions. This is universally true with all models of complex systems, and therefore should be totally uncontroversial, unfortunately the models have become like sacred cows.
Personally, just like I was with COVID, I am confident that paradigm-shifting events will intercede that will totally invalidate the model one way or another.
For COVID the paradigm shift was that the majority of cases are asymptomatic. Most people are still coming to terms with this. I heard it recently best described to think of SARS-CoV-2 as actually two viruses. One, much less prevalent, is serious and deadly. The other, much more prevalent, is mild to the point of being unremarkable. Getting one confers immunity to both. This is, I stress, a mental model, not a scientific explanation of what’s actually happening (which does not presently exist).
The point is that the things that make models wrong are often “external” to the model, the unknowables. For Kelvin it was radioactivity. His model was total “sound” just like the climate models are “sound” in terms of everything that is presently knowable. But complex systems, particularly ones with humans in the loop, have a way of surfacing curveballs the model did not, and could not, anticipate which have outsized effects in the endpoints.
The lesson with climate, just as it is being shown with COVID, is that dramatic action to address the problem (“flatten the curve”) with almost incomprehensible direct costs does not pay off.
With climate, just like with COVID, a less blunt instrument will achieve a better outcome for orders of magnitude less cost.
The sad part is that the top 2% couldn’t give a fuck about the cost either way. It’s the bottom 75% which will suffer from terrible policy.
Surely the lesson from Covid-19 is "early intervention works". Compare for eg the number of deaths in Japan/Australia/South Korea with Italy/US/UK.
Humans have a tendency to see the thing in front of them and claim, I did X, and then Y happened, so X must have helped cause Y. When in fact no such thing has been proven.
I think there are a lot of things different about those examples you gave, and absolutely no scientific way to draw a conclusion about a particular intervention having a particular effect.
Here’s an interesting comparison between Australia and Canada for example, who both responded similarly but have ended up in markedly different situations. [1]
So instead I would offer a different framework for the discussion.
In the end there are only two simple questions with COVID. 1) When we hit the endpoint, what is the total prevalence in the population, and is it lower than the natural saturation point?
So my guess for #1 is that at the endpoint, we will see that basically everyone who could have gotten it did get it. To put it another way, I suspect only the highest risk population taking the most extraordinary precautions will end up not having antibodies to SARS-CoV-2, everyone else at some point being naturally exposed.
In other words, I assume we will reach whatever the top of the S curve for COVID would naturally be based on the biological nature of the virus. Essentially, I am assuming that vaccines will not play a large role, because it will have played out before they become available, and/or because we don’t have any coronavirus vaccines and it’s probably harder to make one than we think.
2) How long do we stretch things out to get to that endpoint, and therefore, how many lives are saved through increased availability of medical care and, on the flip side, how much economic cost is incurred?
My hope here—based on some very preliminary data—is that we are actually much further along than the testing might indicate (e.g. 5% overall prevalence and much higher in certain metros, vs 0.2%)
From a health care perspective I think that the only effective treatments we have today are minimally invasive and extremely scalable — oxygen therapy and frequent repositioning. We are learning that ventilators have proven to be basically lethal to COVID patients, therefore increased availability and usage of ventilators presumably just increases the death rate. There have been no controlled studies either way at this point so no one can say for sure.
At this point because we have no effective cure or treatment, and because initial antibody testing seems to indicate COVID is extremely widespread despite the shutdown, I am very skeptical of the current approach paying off both in terms of delaying spread & saving lives through higher availability of care.
[1] - https://www.theglobeandmail.com/opinion/article-canada-and-a...
Well no - Australia required quarantine of all returning overseas travellers from March 15, Canada advised self isolation from March 25. That's a huge difference.
Again - correct, thoughtful interventions worked.
> Essentially, I am assuming that vaccines will not play a large role, because it will have played out before they become available, and/or because we don’t have any coronavirus vaccines and it’s probably harder to make one than we think.
I've seen the "we don’t have any coronavirus vaccines" meme too. People don't seem to realise we've only had one serious coronavirus before (SARS) and it isn't anywhere as nearly contagious. There just hasn't been much work done on coronavirus vaccines before.
We do however have anti-viral treatments (eg Tamilflu), and it's likely we'll either get a working vaccine or a anti-viral drug that provides some relief in the next 12-18 months.
> My hope here—based on some very preliminary data—is that we are actually much further along than the testing might indicate (e.g. 5% overall prevalence and much higher in certain metros, vs 0.2%)
Either way, it's a long way off the required 80% for herd immunity.
> We are learning that ventilators have proven to be basically lethal to COVID patients
This is absolutely untrue. There have been people sharing data showing that the mortality rate of people on ventilators is higher than those who aren't, but that is because ventilators are only used on people who are closer to death.
It's true that in some cases oxygen is sufficient for some people.
The point is a careful and thoughtful examination shows that a country’s geographic, social, and economic “natural affordances” are largely responsible for the differing COVID case loads, much less than the effective dollar amount “spent” on interventions. Spending on an intervention includes the economic hit as well as the direct cost. E.g. Australia was able to get a lot more bang for their buck because they’re an island and didn’t have a large segment of their population traveling in hotspots.
I think we agree on the vaccines. The point isn’t there’s something specific about coronavirus. It’s not a “meme” that we don’t have a coronavirus vaccine. We actually don’t have one. That just means we might not have as much experience ramping up a new one as we do, for example, with influenza, and there may be things we need to learn along the way.
A working vaccine or treatment in 12-18 months is a long way off.
If we are anywhere near 5% prevalence that means we are a lot closer to 65% (when herd immunity starts being noticing beneficial) than you might think, since it is non-linear.
Regarding ventilators, even in the general (non-COVID) case if you find yourself on one you’re about 50/50 not going to walk away from it. But with COVID it appears ventilators exacerbate the swelling in the lungs and in NY 80%+ of ventilated patients have died. [1]. That’s what I mean by “proven to be basically lethal”. We know the vast majority of COVID patients on ventilators die, far more than just the average ventilator death rate for pneumonia, using the same O2 saturation markers.
So doctors are moving away from ventilators and no longer using the typical O2 saturation cutoffs for ventilating because interestingly patients with lower O2 levels are not in the typical level of distress. It seems the CO2 is still getting out even though the oxygen is not getting in... more like hypoxia.
> It's true that in some cases oxygen is sufficient for some people.
We can agree on more than that. Luckily the fact is that oxygen therapy is sufficient for the vast majority of cases. Just do the math on hospitalizations and deaths in NY with the known variable that 80% that were ventilated died. QED the vast majority were not ventilated.
[1] - https://www.nytimes.com/aponline/2020/04/08/health/ap-us-med...
I was referring to the idea that is going around that "we don’t have any coronavirus vaccines" is somehow significant and means it is especially hard to make one.
> Regarding ventilators
I think the NY Times article is good, but it's important to note that this is evolving and treatments are changing. But saying "some doctors say they're trying to keep patients off ventilators as long as possible, and turning to other techniques instead" is very different to "ventilators have proven to be basically lethal to COVID patients".
That makes ventilation an absolutely last ditch effort. It means if the ~60,000 ventilators already in US hospitals before this started were each used just once ever to ventilate a COVID patient, you would expect 48,000 of those patients to die.
I’m sure this also has a pretty significant effect on the ICU utilization rate as a percentage of hospitalizations.
And this is besides the fact that early serological testing points to the overall hospitalization rate being significantly lower than 10%.
You seem to believe that the ventilators are causing deaths. There's plenty of evidence to show that once Covid-19 takes hold it is particularly viscous because of the number of things that it does to the body.
Some of the things that go wrong in some cases include: lung damage (obviously), kidney damage, liver damage, strokes, encephalitis, seizures, heart damage, meningitis, bloody diarrhea etc[1].
I agree the death date of people on ventilators are horrifying. But I don't think there is much evidence that ventilators are causing that rather than being administered to people who already have serious symptoms.
[1] https://www.sciencemag.org/news/2020/04/how-does-coronavirus...
All we have is anecdotal reports, plus the fact that significantly more COVID patients die on ventilation than the average pneumonia patient who is [invasively] ventilated.
Since doctors were ventilating patients at the same O2 saturation points as usual, we can say at least that something is markedly different with COVID induced ARDS from typical ARDS that makes invasive ventilation particularly insufficient, if not actively harmful.
> Leaving modeling up to the professionals is completely the wrong lesson.
I don’t understand what you’re saying. Nobody is advocating that amateurs be precluded from modeling to their heart’s content.
So what are you advocating? One should discard models from professionals? One should prefer models from non-experts? One should crowdsource models? One should make policy decisions based on gut instinct?
I think what GP means is, leaving it upto professionals alone is a recipe for disaster because of the inherent corruption, greed, biases, and prejudices at play.
> So what are you advocating?
Not speaking for GP, but simpler models are more likely to be a good approximation and more than enough, anyway [0]? Simplicity helps because it stops the pundits and politicians from sort of running away with it?
[0] Newton's theory of gravitation has been supplanted by Einstein's theory of relativity and yet Newton's theory remains generally "empirically adequate". Indeed, Newton's theory generally has excellent predictive power. Yet Newton's theory is not an approximation of Einstein's theory. For illustration, consider an apple falling down from a tree. Under Newton's theory, the apple falls because Earth exerts a force on the apple—what is called "the force of gravity". Under Einstein's theory, Earth does not exert any force on the apple. Hence, Newton's theory might be regarded as being, in some sense, completely wrong but extremely useful (The usefulness of Newton's theory comes partly from being vastly simpler, both mathematically and computationally, than Einstein's theory). https://en.wikipedia.org/wiki/All_models_are_wrong
You and I must be thinking of different professionals.
At any rate, here are two very very very simple models of this epidemic: a) cases will grow linearly, b) cases will grow exponentially.
Super simple. extremely different outcome. Now, one still has to make policy decisions.
I think the solution is not fewer professionals, but more and better professionals.
Exponential processes don't exist in nature, they always break into some other curve at some point.
The entire point is that forecasting exponentials is easy but becomes badly wrong at some point and that point is difficult to forecast
Epidemics are (more) properly modelled by the Gompertz function[1], and the position of the "transition" between the exponential portion and non-exponential points it the important point.
That is exactly what the person you are replying to is saying ("they always break into some other curve at some point.").
An epi curve is created from observations and shows new infections per day.
The Gompertz function is a mathematical formula and can be used to estimate total infections up to a date.
However, if you take the epi curve and plot the aggregate number of infections you'll find a curve that the Gompertz function approximates quite well.
Not sure about you, but I suspect they're thinking of professional politicians and the professionals politicians hire.
That's the kind of bias he was probably thinking of.
Newton's model captures the essence but isn't the eternal truth. Relativity proved that. Today, singularities have been proven to exist that point to missing pieces in Einstein's theory.
The point is, simpler models are a good approximation even though they don't hold true under scrutiny, so, at what point do we stop obsessing over details and complex models professionals come up with or the policy-makers seek because surely there's law of diminishing returns at play here?
It's not so much that fitting exponential or S-curves is hard as much as even a very good fit is likely to have very significant error bars.
The hard parts about modeling the current situation involve the poor and limited data. The dynamics of the system are highly variable and unobserved and thus require clever tricks to reduce uncertainty by expert knowledge, clever alternative signals, and leveraging other models.
So ultimately, it's just not the forecasting but the response. People rarely handle uncertainty very well and a situation with uncertain exponential growth ends up with exponential uncertainty. Making policy decisions where the predicted outcome spans orders of magnitude is tremendously challenging.
"Cuomo claimed to need 40k ventilators after the lockdown was put in place. He ended up needing only a fraction of that. Obviously, without the lockdown, there would likely be more needed. Of course, modeling has errors, but this is an extremely large error, and one that has direct policy implications." (1)
My response was that it wasn't "a large error" but what only 6 more days of exponential growth would have made still insufficient:
https://news.ycombinator.com/item?id=22911465
Also as an illustration of the "uncertainty" one can see the shaded areas clearly here:
https://covid19.healthdata.org/united-states-of-america
As I write this, the model on that page was last time updated based on the data up to the 15th of April, only 4 days ago, and its best estimate for number of deaths in USA on 19th of April was 37,056. Also as I check the https://www.worldometers.info/coronavirus/country/us/ it's already 40,423. But the 95% error area for 19th is 39,721-56,108. It's that hard. Also at this moment the projection for 1st of June is 60,262 (34,063-140,106) and we can compare all these values after some later updates, e.g. in 6 and 12 days.
1) careful observers know that there is one more talked about person who is used the same pseudo-argument before.
June 01, 2020 projection: 67,444 (48,058-122,203)
Today's number of deaths in the U.S., according to worldometers: 47,808. Today is April 23.
June 01, 2020 projection: 73,621 (56,562-128,167)
Total number of deaths in the U.S., worldometers: 56,803. Today is April 28.
June 01, 2020 projection: 112,073 (91,586-155,417)
Total number of deaths in the U.S., worldometers: 74,114. Today is May 06.
I was correct in my predictions of the disease coming to North America and the stages it would entail but I was wrong about the timeline because so as it happened it happened so fast.
What we do now know is that it changes VERY fast so you can't wait for your ICUs to fill up or a lot of people will die... you have to act early. And if it came once this fast... it can again, if our guard goes down (and it will).
This is really important to understand, and unfortunately often overlooked, especially by economists, who often work with good data and solid mathematical background but no prior domain knowledge and no reference to literature of the given field.
this disease defies the models, youth is unaffected etc.
If they were prepared well then they would not insist on imposing the exact same rules for NYC as for rural NY State. The two places could not be any more different.
no person under 18 has died of the disease in Italy, 3% or less of cases in population under 20 etc. many people don't show symptoms but are quite infectious etc
which disease is like that? none actually.
Pressure to obfuscate the impact of the epidemic where way higher then.
The IHME model - one of the most cited models - explicitly factored this into their model.
Here, for reference is the IHME model preprint: https://www.medrxiv.org/content/10.1101/2020.03.27.20043752v...
The parameterization of social distancing is on pages 3-4. They considered four coarse categories of interventions, with the total effect given by a ad hoc score of 1, 0.67, 0.334, or 0.
It should be blindingly obvious that can be improved upon with more data and work. Moreover, the effects of social distancing aren't even constant--compliance certainly varies from place to place and over time (enforcement changes, people get restless).
This suggests that social distancing happened 1-2 weeks sooner than their model anticipated. This will give you a much smaller number.
What is the policy you believe should have been followed instead? Are you saying that social distancing should have been abandoned? Why do you believe that this would result in fewer deaths?
People were voluntarily taking precaution well before our government bureaucracies enforced it, and it was available in public datasets. IHME just didn't look hard enough, apparently. Despite what the media keeps pushing by finding pockets of idiots, people aren't stupid. The most at-risk populations realize this and tend to shelter in place on their own and take precautions.
de Blasio on March 11th telling people to go out: https://ny.eater.com/2020/3/11/21175497/coronavirus-nyc-rest...
OpenTable data showing a voluntary reduction of restaurant activity despite de Blasio's encouragements: https://ibb.co/r2R9xnT
How does one evaluate the OpenTable data and feed it into the model they were using to estimate the amount of social distancing?
https://twitter.com/boriquagato/status/1251943418860728320
I'm saying that lengthy lockdowns, shelter-in-place, and shutting down non-storefront businesses did not provide much value, if any, and possibly had a negative effect.
The IHME trended into 95%+ CI territory over half the time. So when posters here say "well the variance was high," that's accounted for in the confidence intervals, which they also missed on wildly.
The models justified harsh authoritarian action, and the actual data way, way undershot it. Popular sentiment is "oh well, it saved lives at least" but few, if any, are looking at the acute - and more importantly, chronic - economic costs of these policies that may or may not have even helped beyond just telling people what the risks were.
>> How does one evaluate the OpenTable data and feed it into the model they were using to estimate the amount of social distancing?
Pretty simple; it was clear that news of COVID-19 alone was enough to cause people to stop going to restaurants and start basic social distancing protocols on their own without government mandating it.
EDIT: Shockingly, a bunch of downvotes without explanation are forthcoming.
I'm interested in them.
The chronic effects of shutting down the economy are vast. People think money can be regenerated; it can. But what Americans have seemingly forgotten is the past where American autoworkers feared Japanese efficiency and where the Japanese economy looked to crush ours. Today, the Japanese salaryman works at least 50% more hours and his purchasing power is less than his counterpart in the United States (just look at patio11's posts on it).
https://www.kalzumeus.com/2014/11/07/doing-business-in-japan...
It's not a popular take because of recency bias (the same thing happened on 9/11 with the PATRIOT Act and always doe when outliers happen), but the data will be pretty clear in the end.
I would have panicked less if I had read the text of this hacker news story, instead.
Does this perhaps explain why we are so bad at calling the top of economic bubbles and similar phenomena? Maybe there literally is not enough information until the very end. It's not that we're dumb. It's that we can't do the mathematically impossible.
I have a sense that this might be a very important and profound principle that might explain a lot of seemingly irrational behaviors.
Thus, any bubble that does manage to grow to significant size before bursting is necessarily "unforeseen". (of course because of that thing with the monkeys, the keyboards, and online message boards, the will always be plenty of people that did see it coming, but I would wait to buy their book until they do it a second time).
I know it's always trendy to hate on economists (some people have taken that idea all the way to creating cryptocurrencies and reinventing economics along the way). But comparing, say, the 2008 crisis with the 1920 or even the 1970s, I can't shake the feeling that maybe economists have become slightly better over time. The gold standard fandom that was all the rage for a while essentially rests on the idea that interventions by central banks are worse than doing nothing, and the evidence now seems overwhelming that we can do better than this (admittedly low) benchmark.
They can be hard to do, but you must absolutely reconcile them for your effort to be fruitful. Until they are separated, your intuition or your mathematics is wrong, and you can't know which is.
It's harder than just fitting a curve, but that's kind of the point. If you want actionable predictions, it is harder.
But the coefficients aren't fixed. Coefficients change drastically due to public policy, individual actions in response to news and social media, culture of local communities, degree of compliance with public policy, travel between regions with different rates of infection, etc.
So you're not fitting a curve to the data, you're modeling dynamic human actions which are not nearly as easy to forecast.
Since we're on HN, I'll plug a question I've had for a long time:
Do IPOs/liquidity events/exits/etc all happen when the right price has been determined; ie when we can see what size the company will be, and it is no longer useful to capitalize it further? When all growth paths have been explored. Is that time the inflexion point?
What does the conversation look like with the VCs when they see the inflexion point?
Sorry, but I hear a lot of people saying this and it's driving me crazy. S-curves are S-curves from the beginning, not exponentials. It can be useful to use an exponential growth model at the beginning of the curve for short-term forecasting, but these two models will diverge dramatically at the S-curve inflection point.
Not that we shouldn't plan as if exponential growth will occur in a crisis like we're in now, but many people I know don't understand these dynamics and it has lead to a lot of undue panic.
For curves of infection rates I don't doubt your verity.
Differentiable once means, by definition, approximable everywhere by a line at small enough scale.
Of course you knew what you meant, but, for anyone who's confused, the containment goes the other way.
Imagine a sine wave, except when you look at it at 1000x magnification it's a sin + cos. It looks smooth, and at x = π you think that the derivative will be -1 but in fact it's 0 because you don't have a sine crossover there you have a cosine trough.
Except at 1000000 times magnification (ie another 1000) the cosine curve that forms that apparent sine curve is itself a sine curve. So everything is switched again.
f(x) is something like sin(x)+cos(ax)/a+sin(a^2x)/a^2+cos(a^3x)/a^3+ ... (sin(x.a^j)/a^j+cos(x.a^j+1)/a^j+1+ ...
for a=some arbitrary large number. Something like that, I'm a bit rusty, sorry.
At whatever scale you look at the curve the derivative is always wrong: you zoom in on the sine, at the peak it's got a cosine, so the d/dx is -1; but zoom in and the cosine has a sine at the crossing point, so the d/dx is 0; but zoom in and ...
The curve is provably smooth, it's sinuses all the way down, but nowhere can you tell the derivative as it's fractal???
That's what I had in mind.
Anyway, I thought their might be clever curves of that type.
> At whatever scale you look at the curve the derivative is always wrong: you zoom in on the sine, at the peak it's got a cosine, so the d/dx is -1; but zoom in and the cosine has a sine at the crossing point, so the d/dx is 0; but zoom in and ...
This is pretty much the definition of something not being differentiable. "Differentiable" means that the approximations to the derivative (i.e., difference quotients) converge to some fixed value as the scale they're measured at approaches the infinitely small.
You might be interested in the Weierstrass function, which seems to be the sort of thing you're getting at with your idea: https://en.wikipedia.org/wiki/Weierstrass_function . Continuous everywhere, but differentiable nowhere.
Edit: the specific function you wrote down is not differentiable (at least not everywhere). For example, at x=0, its derivative, if it had one, should be cos(0) + cos (a * 0) + cos (a^2 * 0) + ... , but that series clearly diverges.
Have you studied fractal dimensions formally, might I know the background you're speaking from?
Yes, I'm imagining, as a first example, something akin to a Weierstrass function but without the discontinuities.
So the series for the derivative is 0 + 1 + 0 + 1 + ..., which doesn’t converge.
Not sure what this has to do with fractal dimensions. This is a simple question of definitions. The word “smooth”, in math, literally implies, by definition, that the function is everywhere approximable by lines.
If you don’t think it does, can you state the formal definition of “smooth” that you’re using?
The Weierstrass function doesn’t have discontinuities, btw.
Non-linear fitting is VERY hard to do well. And with pathetic data like now, it is doomed.
Some of them are basically exponential at the beginning (the logistic curve). This makes perfect sense in modelling an infectious disease - initially, all contacts are susceptible. Thus you have exponential growth.
Others, for example the CDF of a normal (which is often confused with the logistic), is not exponential at the beginning, but decays much faster (the derivative is exp(-x^2)).
Arctan decays much slower (derivative is 1/(1+x^2)).
So, S-curves are sort of linear in the middle, but sort of exponential or exp(-x^2) or 1/(1+x^2) or hyperbolic at the ends, and which one of them it is tells us a lot about their behaviour.
Note also that without any social distancing or other measures and a sustained Rt of, say, 3, the inflection point will only occur once about a third of the population are infected, and thus it makes perfect sense to model it exponentially while still being below 5%, say.
To answer this question, you need to ask the question: what reason is the company going public?
For example, in the dot com boom, companies went public with a business plan (almost). They were clearly looking for funding to expand on their business and were nowhere near the inflection point.
In contrast, with WeWork, SoftBank was pretty clearly looking to unload radioactive garbage on the public.
So the answer is: it depends.
So the answer is: it depends.
Ye olde pump n dump
This is interesting! Could you expand on this a bit? Why is the length of the interval bound by the ratio of the data error over the derivative?
https://joelonsql.shinyapps.io/coronalyzer/ https://github.com/joelonsql/coronalyzer
"Sweden FHM" is the default country, which is a different data source, it's using data from the Folkhälsomyndigheten FHM (Swedish Public Health Agency), which is adjusted by death date and not reporting date as the John Hopkins data is.
That means that there is huge relative error initially (we have 10 infections +/- 100), and very little relative error eventually (we have 1000000 infections +/- 100).
I assume the forecasts would be better if the error were multiplicative (in other words, with standard deviation proportional to the current value).
However, I think the main point stands: the forecasts get much better once one approaches the inflection point.
This I tried: - Weibull Gamma distributions, but it was impossible to find good parameters for the distributions without exploding. It would only work if I put in an additional parameter saying 99% of the population wouldnt get it. It would come up with good shapes of growth but the predictions were far too below actual values in the future.
- Logistic curves. usually great for countries that already ran up the curve but terrible for ones still in the exponential phase (as the article states). Also kind of useless for countries that didnt even begin their journey up the curve.
- light gbm: good for predicting the next day but terrible many days out. It seems other counties curves do not help that much
- SARIMAX: really good but later predictions would explode, like showing 4 million deaths in france, etc.
I tried to get around these by ensembling them together, but overall I did very poorly at predicting coronvavirus. I still want to get better at this, so if anyone has any good suggestions, please share. Also you can check out what other kagglers have done as well
https://www.kaggle.com/c/covid19-global-forecasting-week-1 https://www.kaggle.com/c/covid19-global-forecasting-week-2
You could do a regularized stratified model [0], for example, in the case where you have plenty of data in one country but not in others, or similar cases which incorporate dynamics (log-log models of growth are also pretty good). Overall, I think it's just too hard of a problem as it is very ill-posed—even if we knew the exact dynamics, any small perturbation in the data samples significantly changes the best fit (in other words, the dynamics of the system are 'chaotic').
-----
The OP makes the basic point that epidemic logistic/S-curves like Covid look like exponential growth for a good portion of their trajectory and so will behave like exponential for a reasonable period in the future. And the data is inherently noisy. Trying to get more than that is counter-productive. Basically, the bulk of the data is pointing to massive death later, the factor stopping that is factors external to the data we have now.
Basically, we know the epidemic is ramping now, we know there's some limit to how far it can go but we don't really know that limit 'till we're close. And trying to get more clever than this results in really bad claims (like guessing we're peaking and abandoning caution).
I'd also add testing results have been unreliable based on real world limits to testing (limits to number of tests, general screw-ups). Death data is more reliable than testing data but delayed by infections taking time to kill people.
The "quick and dirty" modeling of Tomas Pueyo has been more successful than clever actions.
See: https://medium.com/@tomaspueyo/coronavirus-act-today-or-peop...
In contrast, https://covid-19.direct has good data but too clever algorithm for spotting peaks. If you browse their data, you get used to garbage "forecasts" of where the peak is.
https://medium.com/@tomaspueyo/coronavirus-act-today-or-peop...
Could you write a model that will tell you how fast your car is going, just by using math? Or are there innumerable latent variables which define a state space so large that the only meaningful prediction you could make is a short-term linear extrapolation?
I mean, what would make anyone think that there is any constant parameter set for a model that would predict COVID-19 case evolution, when governments and the population are taking active measures to combat spread, such as perhaps the most dramatic shift in the physical interconnectivity of human society in history? It's fundamentally impossible to predict something that is changing in an unpredictable manner.
The reason why we can make meaningful inferences about the accuracy of weather models is because we have a good understanding of the underlying physics which decribes the time evolution of the system. However, the combination of the chaotic dynamics of that system and the limitations of simulation yields accurate short-term probablistic predictions and inaccurate long-term probabilistic predictions.
SARIMAX, in contrast, doens't know if it's predicting shampoo sales, $GOOG, or epidemiology. Although I am curious what your exogenous regressors and lag/differencing/seasonal orders were, if you'd care to share. Did you fit them via an information criterion approach?
no it does not. it just so happens that the solution of certain differential equations produces an s-shape. it has nothing to do with having only 3 parameters. Very sophisticated models with many of parameters and conditions can produce this shape.
A separate set of 3 parameters also parameterizes the SIR compartmental model (https://en.wikipedia.org/wiki/Compartmental_models_in_epidem...), the solution of which also looks like a logistic curve. But this is a model-based (dynamics-based) solution whereas one may be interested in just fitting the logistic model based on the assumption that it's going to be logistic.
Which is great if your velocity on the project retains a normal distribution over time. But missing requirements, bad risk analysis, and team dynamic changes can make for a long velocity tail.
Which means the last 10% of the project accounts for the other 90% of the project time.
If however you have managed to do the important work first, you drop less important features and ship 95% of the planned features on time.
The bell curve drops to zero with exp(-x^2), that is extremely fast.
The logistic distribution (derivative of the S-curve described here) drops to zero with exp(-|x|), that is exponentially, but not as fast as the normal distribution.
> For technological changes, can the final level-off be reasonably estimated?
It helps a lot if it's 0% or 100% :) Given that, you get a decent long-term fit, after about half the time to plateau.
I'd argue it's a matter of calculating confidence intervals for all of your fitted parameters and displaying joint confidence intervals if they display a large degree of interdependence/nonlinearity, as well as showing the fitted residuals beneath the main plot so it's easily apparent what issues the fitted function might have, such as high leverage, or overfitting. The prediction made will be from the best-fit parameters, but the contributions of other possible models can be used to infer a distribution of potential outcomes. A parameter sensitivity analysis wouldn't hurt either.
Then you'll immediately know if you're looking at (or about to publish) junk.
Not sure what you mean by infinitely many solutions, but this is not true in general. A silly example is
y'(t) = Ct,
where, for most distributions, you would only need a few points to get both C and the initial condition to reasonably high accuracy (~O(sqrt(n))). More complicated examples exist that have much more interesting dynamics, but their general trajectories are just not as sensitive/chaotic (w.r.t. the initial parameters).
> If those constants are wrong, the whole model is essentially useless.
I think what makes this hard is not if they're wrong, but rather, being even just a tiny bit wrong makes the whole future prediction change drastically. In other words, two possible inferred parameters which are statistically indistinguishable given our current observations will yield incredibly different outcomes under many of these models.
> It is a matter of keeping constants up to date in light of new information.
Indeed! :)
The upper limit of the curve is 1 - 1/R0. If you use mitigation and effective reproductive number the number is 1 - 1/Rt, where Rt varies with mitigation effort.
What is missing from this article is the relationship between S-Curves and Bell Curves. We can use the Rules of Thumb associated with the Normal Distribution to think about peak growth rate and standard deviations.
The health data.org curve fitting is a decent Fermi Estimate based on observed data. Models are always wrong but sometimes they are useful and I hope we start to discuss the underlying key assumptions used in each case rather than focusing on their imperfect predictive power.
that’s not the right intuition. you could model a logistic curve with just 3 points if the error in those measurements tended toward infinity. the further apart the points are, the less tight the error bars need to be.
the problem with real-world modeling/curve-fitting is that measurements are super noisy and the errors in them are significant.
This is the final plot: https://i.imgur.com/o54t0Ts.png. You could make a good guess from the data how many people it will affect in the lifetime per country by continuing the same pattern till it reaches x axis.
4 points are needed in my universe.
But there is some good news: The data used can vary, and in some cases good projections can be easier to make. Can see, e.g., for COVID-19 the recent
https://news.ycombinator.com/item?id=22898015
https://news.ycombinator.com/item?id=22897967
https://news.ycombinator.com/item?id=22900104
https://news.ycombinator.com/item?id=22902667
In the third one of those we have that the projection from a FedEx case is the solution to the first order ordinary differential equation initial value problem
y'(t) = k y(t) (b - y(t))
There for data we used y(0) and b. Then we guessed at k. Had we used values of y for the past month, we could have picked a better, likely fairly good, value for k.
Lesson: Fitting an S curve does not have to be terribly bad.
The key here is the b: The S curve of the solution is the logistic curve, and it rises to be asymptotic to b from below. Knowing b helps a LOT! When have b, are no longer doing a projection or extrapolation but nearly just an interpolation -- much better.
For FedEx, the b was the capacity of the fleet. For COVID-19 the b would be the population needed for herd immunity (from recovering from the virus, from therapeutics that confer immunity, and a vaccine that confers immunity).
Knowing b makes the fitting much easier/better. To know b, likely need to look at the real situation, e.g., population of candidate smart phone users, candidate TV set owners, market potential of FedEx (as it was planned at the time), or population needed for herd immunity for the people in some relatively isolated geographic area.
Then in TeX source code, the solution is
y(t) = { y(0) b e^{bkt} \over y(0) \big ( e^{bkt} - 1 \big ) + b}
Can also use a continuous time discrete state space Markov process subordinated to a Poisson process. Here's how that works:
Have some states, right, they are discrete. For FedEx, that would be (i) the number of customers talking about the service and (ii) the number of target customers listening. Then the time to the next customer is much like the time to the next click of a Geiger counter, that is, has exponential distribution, that is, is the time of the next arrival in a Poisson arrival process (e.g., the time of the next arrival at the Google Web site). So at this arrival, the process moves to a new state where we have 1 more current customer and 1 less target customer. Then start again to get the next new customer.
The Markov assumption is that the past and future of the process are conditionally independent given the present state; so that justifies our getting to the next state using only the current state -- given the current state, for predicting the future, everything before that is irrelevant.
What is a Markov process, what satisfies that the Markov assumption, can depend on what we select for the state -- roughly the more we have in the state, the closer we are to Markov. In particular, if we take the whole past history of the process as the state, IIRC every process is Markov. But Markov helps in something like the FedEx application since that state is so simple.
We get to use continuous time since the time to the next change of state is from a Poisson process whose arrival times are the continuum -- that is, we don't have to make time discrete although it is true that the history of the process (one sample path) has state changes only at discrete times.
So, for state change, and for some positive integer n we have some n possible states, then for i, j = 1, 2, ..., n, we can have some p(i,j) which is the probability of jumping from state i to state j, that is, we have an n x n matrix of transition probabilities.
[p(i,j) is the conditional probability of entering state j given that the last state was i.]
For two jumps, square that matrix. Now there is a lot of pretty math -- get some limits and eigenvectors of states, etc. Actually fairly generally there is a closed form solution to the process. Alas, often in practice that closed form is useless because the n and the n x n are so large, maybe n^2 in the trillions. E.g., in a problem I solved for war at sea, there were Red weapons, Blue weapons, on each side some number types and some number of weapons of that type. The states were the combinatorial explosion. Then there were the one on one Red-Blue encounters where one died, the other died, both died, or neither died. The time to an encounter was the next arrival of Poisson processes, also Poisson. Well, that was an example where there was a closed form solution but n and n x n were wildly too large for the closed form solution but running off, say, 500 sample paths via Monte-Carlo was easy to program and fast for the computer. So, sure the software reported the average of the 500 sample paths. On a PC today, my software would be done before could get finger off the mouse button or the Enter key.
This approach is fairly general. And since what I did included attack submarines, SSBN submarines, anti-submarine destroyer ships, long range airplanes, etc., there should be no difficulty building such a model for COVID-19 that included babies, grade school kids, ..., nursing home residents, people at home, people working nearly alone on farms, ....
Back to S curves, IIRC dropping out of the math for the n x n matrix and its powers is an S curve. So, in a broad range of cases, always get an S curve although a different curve depending on, yes, the p(i,j) and the initial state. Uh, when no one is left sick, the Markov process handles that as an absorbing state -- once get there, don't leave.
For the n in the billions, the n x n is really a biggie. So, for the submarine problem I did,
J. Keilson, Green's Function Methods in Probability Theory.
asked "How can you possibly fathom that enormous state space?". That is a good question, and my answer was: "After, say, 5 days, the number of SSBNs left is a random variable. It is bounded. So it has finite variance. So, both the strong and weak laws of large numbers apply. So, run off 500 sample paths, average them, and get the expectation within a gnat's ass nearly all the time. Intuitively, Monte Carlo puts the effort where the action is.". Keilson was offended by "gnat's ass" but liked the math and approved my work for the US Navy. That question and answer are good to keep in mind.
There is more in, say,
Erhan Çinlar, Introduction to Stochastic Processes, ISBN 0-13-498089-1, Prentice-Hall, Englewood Cliffs, NJ, 1975.
For why the arrival times have exponential distribution and why we get a Poisson process, Çinlar has a nice simple, intuitive, useful axiomatic derivation. There is more via the renewal theorem in
William Feller, An Introduction to Probability Theory and Its Applications, Second Edition, Volume II, ISBN 0-471-25709-5, John Wiley & Sons, New York, 1971.
https://en.wikipedia.org/wiki/Compartmental_models_in_epidem...
First cut it appears that the differential equation I gave is an SIR model except with R = 0, i.e., once are a customer of FedEx then remain one.
At one point article finds the logistic curve as I did -- maybe for the same situation.
The article does touch on stochastic models, but I didn't see discussion of a Markov assumption or Poisson process for time to the next infection.
There is an unclear reference to operator spectral radius, and maybe that is related to the eigenvalues, vectors I mentioned.
Whatever, especially with the Wikipedia reference, it appears that we are a step or two beyond the OP.
The work I did and described here I did decades ago and is not at all related to the math of my startup now. So, here I was able to describe some of my old work, but I'm doing other things now.
For the differential equation I gave, the solution as I derived it needed only ordinary calculus and not the additional techniques of differential equations. I just did the derivation in a hurry at FedEx just as a little calculus exercise. I didn't consult a differential equations book. I discovered that the solution was the logistic curve only by accident years later.
The Wikipedia article is nice.
Statistical epidemiology is quite heavily invested in stochastic differential equation. Wikipedia would hardly suffice as an academic, up to date and an exhaustive bibliography.
Ah, been there, done that, got the T-shirt, and doing other things now!
BTW you mention Prof. Bertsekas a few times, did you ever run into Pof. Tsitsiklis ? He has done some work in the area of epidemics around 2015. Mentioning just in case you knw him
The left tail of the derivative of a sigmoid also looks like an exponential.
This article is about estimating the more realistic s-curve in the real world. The point it: You cannot get a sense of the end by observing the beginning... even in the super idealized case of the data coming from a perfect s-curve, let alone the real world. Its a great article.
Take tech adoption for example. Often incorrectly represented as s-curves, they are the result of the inherent cost/benefit of the technology combined with the stateful diffusion process of communication and inherently human resistance to change. And being inherently stateful, you can have chaotic influences in that diffusion process. For example, the idea of microservice architecture had an absolutely massive diffusion jump the moment Amazon sent out that now-famous email mandating adoption. It wasn't linear, it wasn't exponential...it was a discrete step, and a very large one at that. These are everywhere too, because communication doesn't propogate like bacteria grows, it propagates due to extremely non-linear levels of influence. Bill Smith, 45 year old mid-level programmer for a tiny Midwest bank, will never have the tech-adoption influence levels of a Steve Jobs or Alan May or Linus Torvalds.
A better option for modeling would be to use Monte Carlo methods or systems methods. Something that acknowledges the inherent statefulness of the process.
You're taking from two completely different levels of abstraction here. Tech adoption rather obviously happens in an S-shaped curve, at least sometimes. See the article for examples. And of course that shape, and its exact parameters, are the result of some underlying processes.
These two things aren't contradictory. Outside air temperature follows a roughly sinusoid curve. It's the result of the earth turning and therefore alternating between night and day. But it's still sinusoid.
And the S-curve does acknowledge state, or it would just be exponential.
Yes, there are different approaches to disease modelling, such as agent- and rule-base simulations. Unfortunately, they tend to be really bad because we just don't enough data to satisfactorily simulate societies at the level necessary for this application.
One thing that can immediately disprove mathematical sigmoid modeling: curve symmetry. If the early adoption exponential growth is not exactly the same shape as the late adoption slowdown, then you don't have a sigmoid function.
And that's the problem: if you're using a single function to model the result of two (or more) separate processes with distinct mechanics and parameters, you're going to have the same exact pitfalls as you would trying to model a bimodal process with a single probability distribution. Namely that they might fit well with interpolative methods, but completely fail with extrapolative methods (like forecasting!).