Hacker News Reader

Comment by bkohlmann | Hacker News Reader

While the article obliquely makes reference to this, I wonder if this is somewhat similar to the "birthday" problem in statistics. Any given person has ~1 in 365 chance of being born on a particular day. Yet, if you have 30 people in a room, there is a greater than 50 percent chance that at least one PAIR has the same birthday. Given the hundreds of cities (thousands?) in the US that may be affected by a flood, it's actually quite likely that at least one of them could have a pretty high probability of experiencing one in 500 chance of something each year. I haven't run the numbers, of course, but we're looking at Houston in isolation - it needs to be seen in the broader context of all possible cities. We're focusing on the individual probability of Houston having this improbable string of events, rather than looking at Houston only being the city that experienced this phenomenon.

Of course, the "once in 500 years flood" may also be an inaccurate probability as well, but if its not, then I'm not surprised this is happening SOMEWHERE.

View on HN

A 500 year flood event is very likely new data that tells us our prediction it was a 1 in 500 year event was wrong.

  http://engineering.curiouscatblog.net/2008/07/13/500-year-floods/

It is of course possible, it was actually a once in 500 year event. I just believe it is much more likely our previous conclusion was faulty. Our prediction of 500 year floods is not very good, we rely on way less than 500 years of data. Also in most places where this happens the massive changes to the environment (roads, cities, paved over wetlands, constrained rivers...) are not factored in well at all. Add to that global climate change and 500 year flood estimates are likely still poor today. The birthday problem doesn't relate to us getting new data that changes what we used to know. A 500 year flood probably is new data (that gives us a strong indication our previous belief was wrong).

This is part of the reason few private insurers offer flood insurance. They understand they there isn't enough information to properly price the risk. The historic data is spotty, we don't understand the geography well enough, you have to understand how construction and urbanization affects water absorption over long distances. If everyone who claimed to know what a 500-year storm looked like had to pay out every time one happened, we might have better models.

No, that isn't why. I worked in insurance. Not homeowners insurance, but I had a certificate and processed claims.

The reason is that most human settlements occur in flood plains because we cannot live without water. We drink it. We bath in it. We irrigate crops with it. Flood plains have fertile soil. We use rivers and oceans for essential cargo transit.

Insurance is about risk management. It is a form of betting. And there is no bet here because there is no question of if it will flood. The question is only when will it flood?

That's a fool's bet to say "I will pay you X amount of money if it floods" when it is guaranteed to flood sooner or later. That amounts to charity, not insurance.

Hey, I work in insurance too - as a non-life actuary for the last almost 8-9 years. Your description is not at all how insurance works, and insurance is not a form of betting or gambling but really distinctly different. What it's really about is just risk pooling; my share of the risk pool is smaller than if I would keep the risk to myself.

Insurance companies don't mind insuring risks that they know will happen at some point, e.g. most property policies will have some claims, and life term contracts (as have been stated below) of course will have a claim at some point (unless the policy is lapsed). However, they have an idea of how often and how costly these claims will be, and through risk pooling diversification, this is lower (per policy) than the cost to the individual. Thus there is an incentive to buy the policy, and an incentive to sell it - as the difference can be made as profit to the insurer.

In addition there's a timing element to insurance: insurers take in premiums "now" for claims that will be paid out "later", so they can invest the money in the meantime. Large insurance companies may have $200bn investment portfolios.

EDIT: So the point is that flood insurance can of course be sold by private companies, however they know the risk is too high and won't offer competitive premiums. If we only had private flood insurance in the US, in the long term this would lead to people having to move to places with cheaper flood insurance. In this way it actively promotes people moving away from risky places (which I think is a good thing) - but it makes it difficult for people in the short term.

You're both right. Cost pooling and betting are two ways of thinking of the same thing. If anything, betting is closer to marginal thinking than cost pooling, but the two are not exclusive viewpoints.

I find this a little confusing, because one can also purchase life insurance, and mors certa, hora incerta.

A lot of life insurance is term insurance. So the bet is "will you die during the years that your policy is in force?" Whole life is much, much more expensive.

Could you then have the same kind of structures for flood insurance policies? I had to pay every year to renew my renter's insurance policy, it only covered losses during the course of that year, and the insurance carrier had discretion to adjust the premiums or not to renew it.

In the US, flood insurance is provided by the government. And the premiums still jump insanely sometimes.

It gets handled this way for the same reason our government provides welfare et al: it makes no sense as a business, but the cost to the nation to do nothing is a bigger problem.

There is some private flood insurance these days, but I don't know who offers it, in what locations, what the premiums are, and what kinds of exclusions there are. Since it's a new thing, I think the claim that it'll be vastly better than NFIP is invisible hand waving. Nevertheless... http://www.insurancejournal.com/magazines/features/2017/07/1...

I think the bigger issue is that there is an inherent incompatibility between free markets (or strict laissez faire, non-intervention of any kind) and democracy. Of course people, in the millions, are going to say "help us" and direct it at their various layers of government, and punish those who don't at the election booth.

Therefore it stands to reason in major floods like this, that everyone is going to get some kind of relief even if they didn't have flood insurance. What I'm not sure of is whether the insured get 100% payouts and those not insured get partial payouts? What's the incentive to have flood insurance, except in smaller, localized, 50 or 100 year floods?

This is asset destruction and the only way to properly handle it is through savings. So it's either made compulsory or you do end up with something of a free loader problem. Whether that free loader problem is a real problem, I don't know.

How could term insurance for homes be possible? Homes are (despite some fluctuations) relatively good stores of value that you want to pass on to later generations. Comparatively, once a typical human retires from working, he isn't producing much of insurable value (and presumably, his offspring are capable of achieving financial independence).

Homeowners insurance is all term insurance, the contract only covers a short period of time. At the end of the contract, the policy holder doesn't have anything.

I had to look it up to confirm: "death is certain, its hour is uncertain" (latin)

That's not the problem. Private insurance does not offer flood insurance because A) it tends to all be due at the same time requiring a very large fund. B) It also promotes people living in flood prone areas. C) It's very hard to asses risks. Combined and you end up with bankrupt insurance companies.

The industry also feels B makes selling flood insurance a bad idea. That said, there are Private policies you can buy in some areas.

PS: Insurance companies like frequent small scale random events like car accidents, because they are easy to plan for.

I don't think point B is valid. Following your reasoning, the industry would also think that health insurances are a bad idea because they would promote an unhealthy livestyle.

That's one of the arguments for high deductibles. The reverse promotes frivolous trips to the doctor.

Being sick is it's own deterrent. However, people like living near water and it's often cheaper in the short term.

> it tends to all be due at the same time requiring a very large fund.

i don't quite see that.

if the flood insurer covered a broad variety of geographically dispersed areas across North America, or even just the US, would that be true?

The company would have to be huge to pull it off. One event like what happened in Houston could easily wipe out a big chunk of any fund they could amass. That kind of diversity could also be a problem if there are shared waterways, think back to the Mississippi/Missouri flood from 1993. I'm seeing 15 billion as the damages there, which is a large percentage of the "cash" pile Apple is claimed to have.

https://en.m.wikipedia.org/wiki/Great_Flood_of_1993

Edit: fix an extraneous wording mistake.

What does this have to do with Apple? Is that just a reference to them hoarding large amounts of cash or are they in the reinsurance business?

I believe it's a reference to the fact that even large companies, Apple is $800b, don't have all that cash on hand. To convert $15b into cash takes time. Disaster victims can't exactly pay for hotels or buy foods with bonds or stock certificates.

btw, Apple has far more than $15 billion in its cash pile. it's over $250 billion by some accounts

https://www.cnbc.com/2017/05/02/apples-cash-hoard-swells-to-...

I was using them as an example of a large company that would be reasonably familiar to the audience here.

That's a fool's bet to say "I will pay you X amount of money if it floods" when it is guaranteed to flood sooner or later. That amounts to charity, not insurance.

A statement with profound implications in other areas of insurance, notably health insurance. You're insuring against expensive losses that will almost certainly happen at some point... which suggests that traditional insurance isn't the right framework for solving the problem at hand.

I have said that repeatedly on HN. Obamacare is a terrible idea and we need another solution.

well. ok. but, i don't see placing the blame on Obamacare. i mean that was not the origin of the concept of health insurance.

Yes, but health insurance worked differently before Obamacare. Insurance companies simply did not cover chronically ill people or people with preexisting conditions, thus they were able to arrange a "bet." Obamacare removes the element of a "bet" from the system.

Just like the government provides fire protection in the form of fire departments, the government needs to provide health care. What we are doing currently is broken.

Structuring healthcare as an insurance doesn't work because pretty much everyone needs it at some point. It needs to be structured like a tax, because the rate at which you pay in has no correspondance with how much you need to get back out, but in aggregate the total sum paid matches the total care provided. That's why the successful healthcare systems in the world are typically single payer systems. However, such systems are politically untenable in the U.S. and so you get obamacare. I'm sure if it was within the overton window of acceptable political thought obamacare would have been a single payer system.

> Structuring healthcare as an insurance doesn't work because pretty much everyone needs it at some point. It needs to be structured like a tax, because the rate at which you pay in has no correspondance with how much you need to get back out, but in aggregate the total sum paid matches the total care provided.

you've just outlined a money-losing system run by the government.

the reason private health insurance systems fail is not merely because everyone needs it at some point. it's because everyone uses much more of it than they've paid for with their premiums. (which is why, pre-Obamacare, insurance companies used to reject applicants with pre-existing conditions and why they put lifetime caps on benefits paid.)

but a government run single-payer system is not magic. the extra cash must come from somewhere.

one way a single-payer system could obtain the additional funds needed is by taking cash from some other government source (which is exactly what Obamacare does when it provides refundable tax credits to help low income people pay their otherwise unaffordable health insurance premiums.) but there's no end to the amount of money that could require.

another thing the government can do is limit services to patients, i.e. take a certain amount of decision making authority, by law, away from individuals, especially old individuals who are very sick. ("i'm sorry, but we're not paying for that new chemotherapy. it's hospice care for you.")

in addition, a single-payer system can also uniformly limit the prices health care providers, pharma companies, hospitals, etc can charge. (i.e take some decision making authority away from that sector of the economy, again, by law.)

I live in a single payer system country and it's nothing like what you describe.

First, how it is paid. It's text book socialism. People pay into it based on ability and receive benefits based on need. It doesn't run at a loss.

Second, the level of care. Basically the government puts a lower bound on health care and uses taxation to do it. You're guaranteed a basic level of care regardless of your situation. You can opt for private insurance that stacks on top of that. A basic level of treatment is covered by the government, but anything experimental or exorbitant you will likely have to pay yourself, unless you have the aforementioned gold-plated insurance. Insurance is much cheaper because it only needs to cover unusual care, but of course I do pay a lot of taxes.

It's not different from a private system, because you can't receive unlimited care, but unlike a private system you're always guaranteed a minimum care.

that sounds a lot like what i described. it avoids being a money-losing system by limiting how much care people can receive.

If I'm not mistaken, they also had yearly or life-time limits, removing the "losing" part of the betting from the system. Insurances are not bets, not more than other commercial undertakings. They are, for the insured, hedges against bets. The bet is: I will not get sick with something that will ruin me financially. Or: My house will not burn down. You pay a steady, comparably small (to the risk) amount of money. Still the total amount of money paid by everybody is bigger than the total amount of pay-outs.

In the case of these massive flooding, it's hard to make the pool big enough. That's why reinsurance companies exist, which can mix the risk pool of different types and locals.

Insurances are not bets

Origin story for Lloyd's of London:

The patrons bet, for example, on whether Admiral John Byng would be shot for his incompetence in a naval battle with the French. He was.

The gentlemen of Lloyd's would have had no qualms about taking my bet on my own life.

Edward Lloyd realised his customers were as thirsty for information to fuel their bets as they were for coffee, and began to assemble a network of informants and a newsletter full of information about foreign ports, tides, and the comings and goings of ships.

His newsletter became known as Lloyd's List.

Lloyd's coffee house hosted ship auctions, and gatherings of sea captains who would share stories.

If someone wished to insure a ship, that could be done too: a contract would be drawn up, and the insurer would sign his name underneath - hence the term "underwriter". It became hard to say quite where coffee-house gambling ended and formal insurance began.

http://www.bbc.com/news/business-38905963

https://news.ycombinator.com/item?id=13911451#13912524

> they also had yearly or life-time limits

Usually, both. Plus, if you had high expenses they'd actively search for an (even unrelated) pre-existing condition as a pretext to cancel your coverage (recission).

Obamacare is not a terrible idea, it's a transitional idea as we're already doing this with insurance; it's an evolution of the system in the right direction, it's not a final solution. Add a national exchange, offer a public option, bam medicade for all.

bam medicade for all.

That is called single payer, government provided coverage. I would support that.

That is not remotely what Obamacare does. Obamacare requires private insurance to cover people with pre-existing conditions and it requires everyone to buy private insurance. It is busted as all fuck and I would like to see it go die in a fire.

I am aware what we were doing before Obamacare sucks and we need a real solution and to not simply go back to that. But this is not a real solution.

Of course it's not what Obamacare does, you can't go from a private insurance market to a single payer government system in one single step. Obamacare is the transition between the two.

> Obamacare requires private insurance to cover people with pre-existing conditions and it requires everyone to buy private insurance.

So does single payer, we just call "buying" taxes and we replace insurers with government. You're still forced to pay and the insurer is still forced to provide for everyone regardless of pre-existing conditions. So it would seem you're simply against the name Obamacare.

> But this is not a real solution.

It's not supposed to be, everyone isn't blind, they see that single payer is the solution, but you can't simply declare the private insurance industry we already have dead in one fell swoop. Obamacare or something just like Obamacare is a necessary step to get to single payer. We need to get everyone into a public option, and then have that option slowly kill off private insurance by simply operating cheaper than they can until the public option is pretty much what is insuring everyone, at that point, it's effectively single payer.

Huh? Why?

Just charge more money for your insurance. Boom! Profitable business.

The people who most need healthcare tend to have low incomes because they are chronically ill. So, no, that Boom! fundamentally does not work.

> That's a fool's bet to say "I will pay you X amount of money if it floods" when it is guaranteed to flood sooner or later. That amounts to charity, not insurance.

The bet is more like, "I will pay you X amount of money if your damages from flooding exceed your deductible within the next year." It's not at all given that it will flood at all this year, or that your losses will exceed the deductible.

It's true that it would require substantial diversification or re-insurance, since flooding is a clustering event.

You wouldn’t be provided with a fixed lump sum, but rather your costs would be covered up to a limit.

So it’s about determining the expected value of losses for a given risk. You’re right that given a long enough time, there is a near-certain chance of a flood, but the question is more about how much that’s likely to cost.

That is entirely insurance.

That reasoning fails in the face of there existing earthquake insurance in the Bay Area.

Major earthquakes are less frequent than major flooding. If the time frame is large enough and the timing uncertain enough, it can still make sense as a bet.

Frequency only changes premiums, car insurance is much more likely to pay out than earthquake insurance per year. That just means it costs more money per year relative to the payout.

Well, insurers will often cover you for water damage from burst pipes, but not from flooding. That's because the damage from burst pipes happens at a predictable trickle of people over the course of a year, but damage from flooding happens to tens of thousands of people on the same day. So the amount of money you have to set aside for the two cases is wildly different. Side note: This is also why your life insurance policy is useless in the case of nuclear war :)

I don't understand this. Even if they can't assess the risk in any one property, can't they estimate the total amount of flood damage in (say) the US each year, and spread the risk across thousands or millions of properties?

If the answer to that is that only people in areas vulnerable to flooding would buy insurance, and the companies would have to pay out too much - well, that means the insurance buyers are pricing the risks better, and the insurance company should hire some of them.

To re-emphasise what the poster before you said - not only could the model be wrong, but even if it was right once before, you have to keep up the model against all the construction changes in the area that alter the watershed, as well as account for climate change contributions.

more likely their model is wrong. it reminds me of fukushima: https://ml.berkeley.edu/blog/2017/07/13/tutorial-4/

Their model is wrong, but it's because it assumes climate is stationary. Climatologists are actively debating what the models should look like when global warming is taken into account:

https://i.imgur.com/HLxUBim.png

edit: not sure why I'm being downvoted. I can only assume the downvotes are political in nature, which is a shame.

That's not such a great reference...

A screen cap of a discussion forum is a poor reference indeed...

One of the things that is expected with global warming is heavier rain as the seas get warmer. This 500 year rain calculation is looking at the past for prediction but the future is different.

Does 1 in 500 come from assuming a normal distribution? Seems you would also need to accommodate random events like "tropical storm parks itself off the coast" which is more like a poisson process.

The other problem is that these definitions are time independent - they treat the probability of each event as a roll of die, when in fact each previous event impacts the probability of future events, by impacting surface features, moving centres of production, driving further energy expenditure and altering haloclines and other such variables.

It's a positive feedback process.

Your comment goes on to show that human intution aligns with an assumption that things are distributed with a thin tail (for example Gaussian like). On the other hand if the phenomena has a fat tail (Pareto for example) then runs of such 'extremes' are not unusual.

What is interesting is that one of the motivations (funding wise) for understanding fat tailed distribution, extreme value distributions were indeed flooding and dyke failure events.

https://en.wikipedia.org/wiki/Extreme_value_theory#Univariat...

> It is of course possible, it was actually a once in 500 year event. I just believe it is much more likely our previous conclusion was faulty.

It seems it could go either way to me, and the way to get a good indication of which is most likely is to look at look at much smaller timeframe models and assess if they are changing over time, and how much. It's entirely possible it's both, but it seems we have a good way of gauging the relative importance of changing climate norms so we should at look at that before discounting it as noise in the bigger problem of poor models in general.

>A 500 year flood event is very likely new data that tells us our prediction it was a 1 in 500 year event was wrong.

Sounds like a use for Baye's Theorem.

yea its almost like the underlying model is... warming?

Good point, but the chance of a given city having a 500-year flood in three given successive years is only (0.002)³ = 8 × 10⁻⁹. So even if we allowed for 100 U.S. cities, 10 different natural disasters and 10 different three-year periods we're still talking about a probability of less than 0.01%.

And it's not like Houston is the only city having 100 year+ floods regularly.

yes, it's almost as if the climate is changing

Or our assumptions about climate / wheather were wrong.

...the assumption being that the risks are separate. This is clearly false - you are not rolling dice here, the probabilities of the same disaster at geographically "close" cities are linked.

That is true but it's an error in the unproblematic direction: fixing it makes the argument even stronger.

That is, if the probability for each city flooding is 0.002 but the different cities are positively correlated, then the probability of at least one city flooding is lower than it would be if they were uncorrelated. (Because the probability of no cities flooding is higher, because of positive correlation.)

Having lived in Houston, this isn't a measure of the flood but a measure of how terrible the infrastructure is. They have terrible flood management systems but are "rated" to handle everything but a "100" year flood... yet Houston gets 100 year floods every year- multiple ones- and has for 40 years.

It's a scam involving the ratings much like the mooody's ratings scam of the last decade IMNSHO

I understand your argument, but you can't compare floods to the birthday problem. Unlike randomly-sampled people's birthdays, floods in US cities are not independent random variables.

For instance, chances of 500-year floods in two Texas cities is not 1 out of 500*500 in a year. If they're geographically similar and physically close, it should be closer to 1 out of 500.

In 2 Texas cities, sure, but in every U.S. city? There are going to be several that end up getting a few 500-year floods... and several others that don't flood as expected in several 1000 years.

In many US cities. Recall that during hurricane sandy, pretty much any landfill or low lying area was flooded in the NY metro area.

That means dozens of cities and towns with 7% of the US population.

>> Of course, the "once in 500 years flood" may also be an inaccurate probability as well

I suspect that's it. Of course I may be falling into the same birthday paradox here, but a similar thing happened when it flooded where I lived now. The x in "x-year Flood" kept increasing (I'm inclinced to blame that on sensationalizing news for ratings), but people kept comparing it to the 3 or 4 similar floods that happened in the last hundred years. It would seem the probably has gone up due to climate change, or that we're noticing the floods more now because there's more damage to be caused to more things that have been built up, and we have better records and better news transmission, but it certainly seems that can objectively say they're no longer 1-in-500 year flood regardless of the reason.

>. It would seem the probably has gone up due to climate change

That probably has a small reason to do with the increase, but your next line highlights the issue.

> or that we're noticing the floods more now because there's more damage to be caused to more things that have been built up

No, not exactly. The more stuff you build, the faster water runs off. Unincorporated land has a pretty high water slowing capability. It takes more time for water to run through tall grass, and puddles, and log jams, and loose dirt. All things that occur naturally. Humans build houses that have all the water run into gutters, that run into a well cut lawn and down a sidewalk. Once it hits the road it's like a water expressway. It can travel quickly to a local stream. A stream that likely has a concrete lining so the water moves swiftly to a lake or reservoir in hours rather than days.

As a place develops new flood plans must be studied every 5 to 10 years. A new development of 100 hours up the watershed from you can mean your risk of flooding increases dramatically. Places like Houston have had 10,000's of houses build in the surrounding areas meaning what was a 1 in 100 year flood could be as low as a 1 in 10 year flood. All this stuff being built is not taking into account its effect on downstream rainfall runoff.

I actually hadn't thought about that aspect at all. I was thinking more along the lines of 'more development -> more stuff to get destroyed', but that's kinda backwards anyway because a flood would probably wipe out a bigger portion of infrastructure hundreds of years ago than it does now.

But faster run-off sounds like a good thing to prevent the flood - are you saying that more development upstream means more flooding downstream because all the run-off arrives with much lower latency and higher bandwidth, to use IT terms?

Yes, lower latency is a pretty good example.

Also people tend to drain marshes and any other shallow stagnant waterways to avoid mosquitoes having breeding places. It can save us from terrible diseases, but it can make flooding worse.

An additional on flooding:

When Rainfall * Area >= SoilAbsorption(See note) + WetlandAbsorption + ArtificialDrainage the end result is flooding.

Essentially the left hand side of the inequality has been slightly increasing because of climate change, but we're chipping away constantly at the right side of the equation in three ways.

(1) More and more soils in American metro areas have been covered by impervious surfaces for development. (2) More wetlands are being destroyed for development. (3) Artificial drainage is being under-built. Drainage is generally built for 1 in 100 year events but the rate of urbanization is increasing the magnitude of flooding. (Drainage is only a problem for the developer that hasn't sold what they're building.)

Note: Soil has a finite capacity for water infiltration.

There is one additional thing to note. Soil has a certain sponge like quality to absorb water. So a runoff curve number was created to "predict runoff or infiltration"

The tables on the wiki pages shows "curve numbers" for different types of soils. These numbers are based on experimental data which is then used to model runoff. (Or used to be. Its been 20 years since I've done runoff modeling, so maybe things have changed)

https://en.wikipedia.org/wiki/Runoff_curve_number

Sounds like an effective, cheap solution would be water butts on every gutter.

Up to a point. To continue the bandwidth analogy, having a buffer is cheap - unless the input link fills it faster than the output link can handle, for a prolonged period. The buffer gives you some extra time for processing, but when it fills, you are back to step 1.

There is always considerable political pressure to low-ball the estimates. That is hinted at in this article, and it happened in New York, where the first post-Sandy flood zone maps provoked widespread opposition from those in the newly-expanded zones, and they were rapidly scaled back. There is, of course, some uncertainty in the estimation, but it seems likely that the official maps will be, at best, based on the most favorable possible interpretation of the data.

Mother nature does not respect grandfather clauses.

Because, crucially, your mortgage may require you to buy insurance in flood plains. And it can be quite expensive.

There's gonna be lots of screaming when homeowners in Houston realize how much flood insurance is going to cost them in the future. We all got a preview of this in coastal Florida.

Pretty much everyone knows it's a good idea to get flood insurance in Houston.. We still get lots of localized downpours, floodplane or no, which can easily overwhelm local drainage, etc. Whether they actually do it or not is another idea

I am aware that flood insurance itself is subject to government influence - e.g. the National Flood Insurance Program, and possibly state regulation (even in Texas) - but I don't know the details. Does it result in premiums being affected by official flood-zone designations?

As you appear to be in or near Houston, I hope things are OK with you and your family.

non homeowner, and haven't spoken to many, so no clear inisght.

all well here, many people I know or are aware of are affected, thanks.

From what I understand of the article, the "1 in 500 years" measure is based on the estimated probability of a flood on a given year in a local area. So the probability of a "1 in 500 year" flood in Houston is independent of the probability of a "1 in 500 year" flood for another city.

Right what the parent comment is saying is that look at a map of the US, how many major cities are there. The odds of two "1 in 500 year" floods happening consecutively are low for any individual city. But if you have enough cities it will happen somewhere. And then you write the article about it.

It's kind of analogous to hitting a golf ball into an open field and then exclaiming "Of all the blades of grass, the ball landed on this one!" It's only interesting if it goes in the hole with the flag, otherwise you have a sort of selection bias going.

That makes sense, I was confused since the Birthday problem is more about the pigeonhole principle rather than larger samples being more likely to have individually unlikely outcomes.

But for this case, it would still be very unlikely. For any arbitrary 3-year span, the probability of consecutively getting a "1 in 500 year" flood for 3 years is (1/500)^3. If we have N cities, then the probability of none of these cities having 3 consecutive floods is (1-(1/500)^3)^N. For even a vast overestimate such as N=20000, it is still a significantly improbable event. Of course, this doesn't account for "3 consecutive years or more within some year range" and it is a gross simplification, but I think there are probably better explanations than selection bias, such as the inaccuracy of the model or the fact that these events might be temporally dependent on each other.

My understanding is that by definition, a 500 year flood has a probability of 1/500 per year. But I live at the top of a hill in the Midwest, so a 500 year flood in my locale has a much different severity than a 500 year flood in Houston.

A more accurate term for what you describe would be a clustering illusion[1], and it certainly could be a valid explanation (among other possibilities) for the observation made in the article.

[1] https://en.wikipedia.org/wiki/Clustering_illusion

Yes, or as with nuclear power, it could be that the base assumption regarding safety just was wrong.

Birthday paradox relies on non-independent events. I'd assume floods should be relatively independent between different cities, meaning that the number of 500-year flood events in the U.S. would follow a binomial distribution (https://en.wikipedia.org/wiki/Binomial_distribution).

p = 1/500 years = 0.002

For 100 cities, the probability of at least one city having a 500-year flood is 18.14%.

For a 3-year run:

p = (1/500)^3 = 8e-9

For 100 cities, the probability is 8e-7 of at least one success. For 1000 cities, the probability is still only 8e-6.

Wolfram alpha helps in calculating PDFs and CDFs: http://www.wolframalpha.com/input/?i=binomial+distribution+n...

This is actually a really interesting problem because depending on how you set it up, you'll get very different answers. To illustrate, notice the difference between these questions:

1) Suppose I flip a coin 3 times and repeat that experiment on 10 different occasions. What's the probability that I get all heads at least once?

2) Suppose I flip a coin 30 times (same total number of flips). What's the probability that I get at least 1 string of at least 3 heads?

(1) is more like the binomial calculation you've set up, but I think (2) is closer to the question we want to be asking.

This is important because, as it turns out, (2) is much more likely than (1). You can tell intuitively that there are more sequences of coin flips that would satisfy (2). For example "THH HTT ..." would satisfy (2) but not (1).

So how should we pose the hurricane question? I propose: What's the probability that in a 100 year period, across 100 cities, there will be at least 1 city that experiences a 500-year flood for at least 3 contiguous years?

Solving this analytically is non-trivial, but it's easy to simulate:

  // Scala

  val rand = scala.util.Random
  def coinFlip(p: Double): Boolean = rand.nextDouble <= p

  // Simulates 100 years and returns true if there's a contiguous string 
  // of three events with p=1/500.
  val targetEvent = Seq(true,true,true)
  def threeIn100Years = Seq
    .fill(100)(coinFlip(1.0/500))
    .containsSlice(targetEvent)
  
  // Simulate this 100,000,000 times
  val simulations = 100000000
  val count = Stream
    .fill(simulations)(threeIn100Years)
    .foldLeft(0){case (acc, cur) => if(cur){acc + 1} else {acc}}

  // Estimated probability for a single city.
  val p = count/simulations.toDouble
  
  // The probability it will happen *at least* once in a population of
  // 100 cities.
  // This calculates the probability if *won't* happen 100 times, and
  // then takes the complement, giving us the final probability
  1 - Math.pow(1 - p, 100)

The final answer I get is 9.5E-5, so still very unlikely, but 100x more likely than the binomial calculation.

How does the birthday paradox rely on non-independent events?

I guess what I meant wasn't "non-independent events". However, there is still a fundamental difference. The probability of a 500-year flood happening next year is still 0.2%. With a birthday paradox, the filling of birthdays is what causes the explosion in probability. Assuming no collisions so far, the 20th person you add to the room has a 19/365 chance of colliding, while the 2nd person only has a 1/365 chance.

I'm not sure how to succinctly phrase this idea. Maybe "non-constant probabilities"?

In the terms of an example:

If you put 400 people in a room, the odds that no two of them share a birthday have gone all the way down to zero percent.

No matter how many independent cities you measure flooding in, you can never achieve a perfect 0 or 1 probability of (whatever), because the cities are independent.

We don't define "interesting" flooding in one city by reference to what happened in another city, so this situation is not described by the birthday paradox.

> Yet, if you have 30 people in a room, there is a greater than 50 percent chance that at least one PAIR has the same birthday.

Not quite the same because Huston is not the only city that has experienced extreme climate these two years, many other cities also have (maybe not 500 years flood, but close).

So that would be like: one pair is born the same day, and 20 other people in the room are born a week around that day. Hardly a coincidence.

It's more likely that there was a party about 9 months and X years before that day. (I'm not sure how far we can go with this silly birthday analogy:)

I think that this statistic only makes sense if it is specific to the location. Otherwise, it is useless for planning purposes. If it is specific to the location then the birthday paradox does not apply.

Interesting note, this is Houstons third 500-year flood in three years [1].

[1] -https://www.washingtonpost.com/news/wonk/wp/2017/08/29/houst...

> it's actually quite likely that at least one of them could have a pretty high probability of experiencing one in 500 chance of something each year.

This is the key point here, and it's called the fallacy of multiple endpoints. If there are 500 observable cities, then somewhere among them should occur a "500-year event" every year. It's just that nobody ever notices the big bulk of the population that hasn't yet had any 500-year event in its recorded history.

How do you count "floods"? If there is one hurricane that causes floods in 3 cities, is that 1 flood, or 3?

If it's 1 - which is what I think is meant when saying "a 1 in XXX years flood" - then your comparison doesn't hold, because the number of floods isn't dependent on the number of cities, but on the distribution of cities on the land, and specifically in flood-prone areas.

IMO you need to look at something other than city limits. Cities like Houston and LA are huge cities geographically. IMO you need to look at regional and per capita impacts.

No it's not the same thing. The 500 was a Houston stat, not a US overall one.

Ahem, no - your birthdays example can't apply here. When you increase the number of people in the room, each one of them benefits from every new person added, in terms of probability of having been born on the same day.

In the flood case, a flood in California and one in Miami don't really increase the chance of another flood simply because they have happened.

> Of course, the "once in 500 years flood" may also be an inaccurate probability as well

Or rather, the odds of that kind of flood increase a bit each year.

Nobody notices the 500yr events that happen in the middle of nowhere.

With water? Unlikely. Water doesn't just drop and disappear on the same spot in the middle of nowhere, it tends to move downstream, into places that are in the middle of somewhere - as we have seen with the Mississippi watershed: first Out In Nowhere, IN, then ORCS, then Lake Ponchartrain. A 500y water event in nowhere land would still propagate; the speed of water release from the drop point (sorry for not using the correct jargon) would mean the difference between a 10y event downstream or a 500y event on the go.

its the gamblers fallacy black 17 has not come up for ages so its more likely to come up - looks like the journalist does not understand probability

It's more analogous to getting black 17 six times in a row. It is possible, but it's weird enough to justify thinking that you've misunderstood the game, or the game has changed.

Except there's a very good ability to know precisely the odds of what the ball landing on black 17 are.

It's a very complicated question to determine what a 1/500 odds of flooding are.

So, it's very possible that this isn't the gambler's fallacy, but rather the model itself is inaccurate.

or the massive building growth without building the required infrastructure

really voted down for pointing out probability 101 I would have expected better mathematical knowledge on HN

Probability 101 begins with a qualifier: "Assuming independent events..."

This can mean two things: a) a set of events that is evidently possible, but increasingly improbable b) events that are not independent

The "Birthday Problem" has a simple explanation. Something like nearly 45% of people are born in the Summer months. Ensure that you get people of different ages and your chances of finding someone with similar/same birthday goes up enormously.

No, the birthday problem still works if you consider that the probability of being born on a particular day is uniform.

It's related to the fact that the probability of two people in a room having a birthday is more closely related to the number of pairs of people in the room (which in turn grows as O(n^2)), than to the number of people in the room.

A detailed proof/explanation can be looked up easily with Google, if you have a bit of mathematical maturity.

For those who down voted me, you do not know your sample set or your test subjects. Go to the empirical data sources. Human behavior certainly affects the real probabilities. As of 2015 in the US [0], a person is more likely to be born in July, August or September. They're also more likely to be born during the middle of the work week rather than the weekends. Draw your own conclusions from the data.

[0] https://www.cdc.gov/nchs/data/nvsr/nvsr66/nvsr66_01_tables.p...

Your information is interesting, but your inference is incorrect.

The Birthday Paradox has absolutely nothing to do with procreation, and everything to do with statistical probability.

Probability is just a model in lieu of the actual data with many built-in assumptions. Probability is a shortcut finding, collecting or generating data. My point is, if you have the data or a very large faction thereof, use it. Don't guess using probabilities.

Right, but it's really not about birthdays at all.

It's specifically about coincidences in a randomly distributed set.

It just says "birthday" in the common name because that's a relatable, if imperfect, example.

The birthday problem still exists even assuming that everyone has an equal chance of being born on any day of the year. It's a phenomenon of the statistical likelihood that given a large enough group of people, the likelihood that any two will have the same birthday goes to almost certainty very rapidly.

Thats....not how that works. At all.

Is this true? Which months are considered "summer"? Northern Hemisphere or Southern or both? This can easily turn into a "40% of sick days are on Monday or Friday" type fallacy.

But storms aren't people with randomly distributed cities to flood so you don't get the combinatorial effect. It's not like there are a fixed number of 500 year storms every 3 years that have to flood a uniformly distributed list of cities. The floods are supposedly independent (even if our estimates of likelihood are bad) so there's no binomial coeeficient.

The simple calculation is P=1-(1-1/500^3)^N and the probability is less than 1:10000 assuming independence and even given 6000 cities of a million people (the entire world population is in a similar city).

If it was birthday like (every possible 500 year storm for 3 years is in a room choosing out of 100 random million person cities), the probablility would be reasonable. You can use a Poisson distribution to estimate the 3 person birthday problem. P=1-exp(-3choose100/500^2)=48%

Actually, in the US it is approximately 1 in 250,000 /assuming independence which is a bit of an assumption/. Obviously, nature doesn't know what a year is so it is totally possible that big storms come in batches.

Assuming 1 in 500 year floods are independent, note the ~300 cities in the US [0]. The fact that Huston is a big city doesn't seem relevant to me, because the storm doesn't care. Much like picking a specific number plate, it had to land somewhere.

The chance of there being a city in the US which has a 1 in 500 year flood is therefore 60% (300/500) each year.

The chance of that city having another flood of similar magnitude in Year 2 is 1/500, because the odds are independent and we started monitoring the city in Year 1. The chance of it flooding again the third year brings the odds down to 1/(500 * 500) = 1 in 250,000.

[0] https://en.wikipedia.org/wiki/List_of_United_States_cities_b...

Reply on news.ycombinator.com