Hacker News Reader

Comment by feral | Hacker News Reader

Disagree: even in your formulation, no 'quick jump' is needed.

Your statement A:"The longer something has been around, the longer it will be around" is not the opposite of B:"the shorter something has been around, the shorter it will be around".

Rather they mean the same thing. 'longer' and 'shorter' here are just English language ways of referring to the same time t that an object has been around.

If someone tells you "the longer a distance is, the more time it takes to walk it", that is exactly the same as "the shorter a distance is, the less time it takes to walk it"; there's no logical leap there.

I could conceive a rule that says "archeological artefacts are likely to be around for a long time" and it'd be a mistake to conclude that this means that non-archelogical artefacts will only be around for a short time.

But that doesn't seem to be how the Lindy effect is formulated, either on Wikipedia or on your post, so there doesn't seem to be an error in applying it to new things.

View on HN

The real meaning of the Lindy effect is:

   the more something has been continued to be around due to be continuously and repeatedly selected from a pool of similar other somethings, the longer it will likely continue to be so.

Because this is a statistical effect (longer lived things are drawn from a pool of things, some of which are not long lived), you cannot invert it trivially.

If there is no selection process, then the Lindy effect is either meaningless, or decomposes to an assertion that the current thing is the only way to do something.

Ok, so let's say discuss your formulation instead:

Why can't we 'invert' it, just because it's statistical effect? Yes, in your formulation, some of the new things in the pool will go on to live a long time, while others will be selected out.

But so what? We are talking about the expected lifetime of an item in the pool, conditioned only on it's age. There's no fundamental problem making a statement that this expected lifetime is short for new things, even if some fraction of those new things will last a long time, right?

After all, we don't know any one individual item that's been around a long time will last a lot longer. We only know we expect it to. Because even long lived items have finite lifetime, hence they'll eventually die (and when they do it'll be really surprising, because they've been around so long; but it will happen eventually.)

And so the statement is always talking about expected lifetime, whether for items that have already lasted a long or short time.

(Hence I still don't think there's really any logical 'inversion' here.)

> There's no fundamental problem making a statement that this expected lifetime is short for new things, even if some fraction of those new things will last a long time, right?

Everything that's new in the pool might be better than everything that's old.

All you can say about the old stuff is that it was better (by some metric(s)) than anything it had to compete with so far.

But you can't say anything about the new stuff. Sure, statistically it is likely that it will some blend of bad, middling and good, but you don't actually know the mix, or which term describes which items, until after the selection process (i.e. time) has taken place.

Absolutely there are real situations where the new stuff is going to last longer than the old stuff, even the old stuff that's been through a selection process. E.g. modern manufacturing techniques have improved overall longevity of all 2022 models.

But I think you are outside the Lindy effect model at that point.

To put this in the example of the original post: The author says visual studio code is expected to last less time than VIM.

You could counter by saying: "hey, maybe, uh, the rise of Product Management as a discipline has meant that modern software overall will have longer lifetimes, and hence it's not fair to guess that VIM will outlive VSCode".

And that'd be a fine position. But imo the right way to frame that isn't "the author did an incorrect logical inversion of the Lindy effect model"; rather it would be "I don't think the Lindy effect model applies to this domain".

(No one is saying the Lindy model is universal.)

I guess you could say you want to apply it only within a given year of software; so, we're happy to look backwards and apply the Lindy model to software written in 2011, but we've no idea how to think about the lifespan of software written in 2022, and aren't allowed make any inferences from software written before 2022.

That's fine, but that's an additional constraint we've added, is outside the Lindy model, and, really, we're in "all models are wrong, some are useful" territory here, where I'd ask "is it really useful to throw away all that previous data? Wouldn't it be a better starting point to use the lifetimes of previous years as at least a prior?" And if you grant that, then I think theres no logical error here.

>Everything that's new in the pool might be better than everything that's old.

It "might", but the empirical observation behind the Lindy effect points that this is unlikely (if we take "better" to mean "more fit to live and grow old and still used").

Sure, we haven't seen the new things develop yet. But we have seen that most new things dont survice time: the things that survive are a small subset of each "new things" (say, vi and emacs, and not one of 100+ 70s programming editors).

If you sampled from a pool of now defunct software projects, or all software projects that were made in some year, you could make some estimate of the survival probability of software projects that are current. But if you can't draw an Kaplan-Meier curve, it's unclear to me how you would assert there exists a survival function that could be inverted.

I’m talking out of ignorance her, so please educate me: if you take into account the black swan theory both kaplan meier and Lindy effect can’t say anything about anything, no?

The point of the black swan is you can't model it. You can't predict COVID-19. You can't predict Russia renouncing it's debt, leading to the collapse of LTCM. You can't predict Hurricane Katrina hitting just so as to push feet of water into Lake Pontchartrain. You can't predict the the 1989 Loma Prieta earthquake. You can know everything there is to know about epidemics, finance, weather, and seismology, and you still can't predict those events. So don't. Do you best to build robust systems, looks for places with excessive efficiencies, and plan mitigation strategies in advance.

Quite funny that the concept did not appear to apply to Lindy's restaurant itself which operated from 1921 to 1969 - shutting down 5 years after the term was coined in 1964.

https://en.wikipedia.org/wiki/Lindy%27s

Exactly. Suppose a piece of software remains around for another year with probability p, and for the sake of this example, that p is constant.

Then if the software has been around for one year, the expected value of p is 50%. But if the software has been around for ten years, the expected value of p jumps to 0.5^(1/10) ≈ 93.3%.

In this way, if a piece of software has been around for longer, then it has a greater chance of sticking around. In fact, the expected number of years it has left is indeed equal to the number of years it has already been around, as stated in the article.

In practice this mechanism is more complicated, as all software is influenced by a changing environment, but this same idea is still at the core.

> Then if the software has been around for one year, the expected value of p is 50%. But if the software has been around for ten years, the expected value of p jumps to 0.5^(1/10) ≈ 93.3%.

This reasoning is incorrect. You have to take the distribution of p into account.

In a world where almost every software has p=0.5 the software which has been around for 10 years is likely to have been lucky and not to have higher p.

Old things that are still around are generally longer lived things. New things may or may not be longer lived. This means that if you sample old things that are still around, they are more likely on average to be longer lived than new things, because the short lived new things have not been weeded out yet.

It's like how the music of the 80s or whatever decade seems better when reviewing, as you can just listen to the albums which stood the test of time. And you forget about all the trash music.

But one can think of a counter example where all the new music are very high quality and will stay around for a long time (not saying this is likely to happen). Then the statement "the shorter something has been around, the shorter it will be around" is false.

Make sense ! If you squint just enough you have also evolution there :)

>If someone tells you "the longer a distance is, the more time it takes to walk it", that is exactly the same as "the shorter a distance is, the less time it takes to walk it"; there's no logical leap there.

that's comparing A->B with A->B. comparing A->B with B->A would yield "the less time a distance takes to walk, the shorter that distance must be"

> that's comparing A->B with A->B

Yes, that's the point. feral (your parent comment) has accurately observed that Etheryte (your grandparent comment) has mislabeled A->B as B->A. But feral is completely correct that "the more time something has already been around, the longer its future expected lifespan is" is exactly the same claim as "the less time something has already been around, the shorter its future expected lifespan is". Etheryte is making a pretty bizarre error; he seems to be under the impression that "a < b" is the opposite of "b > a".

The logical contrapositive to A is A' “if something dies off soon, there's a good chance it was a recent fad“. A makes sense, and A' makes equal sense, but B is claiming stuff about new tools for which we have no way of guessing the future.

Your logic is wrong. You could go to the Wikipedia page for contraposition and learn about this, rather than arguing in comments.

To use your own example: You have to take a long time to walk a long distance. That doesn't mean you have to take a short time to walk a short distance.

I think you are taking the Lindy effect too seriously.

You basically only know one thing about something, which is how long it has been around. The things that come to life and die that you observe will be, on average, in the middle of their life.

So if something has been around for five years, given no more information, your best bet is that it'll be around for another five years.

One year? Another one year.

Hopefully this explains well why the Lindy effect also says that something that's been around for a shorter period of time is more likely to disappear sooner.

But this is not the only thing you observe. You can see many more things about those new products than simply the amount of time they have been around.

You can see Notion come out, and go "wow, I find this good and I think it will keep being around and good for a while". Like the GP said, the fact that it is young doesn't mean it is doomed, or even that it is not good. The Lindy effect says nothing in that direction.

> go "wow, I find this good and I think it will keep being around and good for a while".

Sure, you can have a much richer model of the world than the Lindy model provides.

>fact that it is young doesn't mean it is doomed, or even that it is not good. The Lindy effect says nothing in that direction.

No; it says that young things are less likely to last than old things. If the average expectancy is small, sure it doesn't guarantee any one thing is doomed (some aren't) but it absolutely does tell you that most of them aren't going to last long.

Imagine a friend tells you that they know someone who is planning to become a professional rock star when they leave school.

They are probably not going to make it, because most people who try don't. You can't be sure, because some people do become rockstars. But it's not an error to be sceptical of their chances.

If you hear they are still gigging after 3 years, even if they haven't made it yet, you are probably a tiny bit less sceptical.

That all makes sense, right?

The fact that "most of them aren't going to last long" is not enough to deduce that any specific one of them is not going to last long. All things that are old were once new, and survived. Again, the implication does not go that way and cannot be used even as a rule of thumb. It does not hold.

If you tell your rock star friend to go back to school, you are helping him minimize risk. The chances are that he won't make it, but that's not a consequence of the Lindy effect. It's because surviving in the rocking world is difficult as it requires that people pick your music repeatedly over other available bands, which is the requirement for the Lindy effect to apply, not its consequence.

More importantly, if you tell every rocker to stop and go back to school because their chances of succeeding are low, you end up with no rock bands. Because all rock bands have to start out being new.

Believing that most new rockers won't become big is fine. Believing that any rocker that is new won't make it is wrong, in fact you can be certain that some will.

Yes, however we are reasoning probabilistically here.

We're not making absolute statements about either new or old things.

We're not saying any one new thing definitely won't last another 10 years, the same way we're not saying any one old thing will definitely last another 10 years.

The whole discussion is about what things are likely to do.

There's a place for formal logic in discussions of probabilistic models, but it can also confuse people.

> Believing that any rocker that is new won't make it is wrong, in fact you can be certain that some will.

If you were made to bet money repeatedly on whether a new rocker or an old rocker would still be a rocker in 5 years, you would end up with more money if you always picked the old rocker. This means that older rockers last relatively longer and new rockers last relatively shorter ("on average" / "by expectation" / "probabilistically"); that's literally the same statement.

That the underlying mechanism might be gradual removal doesn't change this, and doesn't matter.

You know, fair enough. I understand what Etheryte at the top of this thread is saying: Assuming that "most of the new tools won't be around in 10 years" is correct. Assuming "none of the new tools will be around, only Vi will remain" is incorrect. Betting on other tools make sense, and giving up on all new tools would fulfill the prophecy at everyone's detriment.

I got the sense that the author is trying to say the latter, however what they literally said is more like the former: Visual Studio Code, specifically, is likely not to be around in 30 years. I can't argue with that.

> But this is not the only thing you observe.

Yes it is, in this context.

When you take into account more factors, then sure you can get a better prediction. But then it's not the Lindy effect anymore.

I'm talking about the Lindy effect only. Nothing else.

First sentence on the Wikipedia page:

> The Lindy effect (also known as Lindy's Law[1]) is a theorized phenomenon by which the future life expectancy of some non-perishable things, like a technology or an idea, is proportional to their current age.

Proportional.

But it does, if you assume constant speed. (I almost wrote that in my comment but decided it was too obviously implied to mention.)

So if we assume that's implicit in my example, how does the Wikipedia page on the contrapositive help now? (It doesn't, as my argument isn't relying on any property of it.)

So are you saying Pareto distributions follow linear relationships? Are you sure about that?

I would read up on the topic if I were you. Statistics can be unintuitive sometimes, and basing your argument on supposition can be unreliable. I'm not sure about the correct answer here, but most folks get e.g. the Monty Hall problem incorrect if they rely on their intuition.

For a product at T = 0, why would it be optimal to assume its lifespan would be T = 0 as well, rather than the average age of all products?

>saying Pareto distributions follow linear relationships

No; not sure where you got that from.

>For a product at T = 0, why would it be optimal to assume its lifespan would be T = 0 as well, rather than the average age of all products?

I'm not saying it would be. (The original blog might, but that's irrelevant to my posts here.)

If you knew the average lifespan of all products, that would be your best estimate of the lifespan of a new product. (You can't use average age naively without thinking about right censoring.)

If the average lifespan of a product was short, then the average lifespan of a new product would be short. As the product aged, it's expected lifespan would increase.

I.e. the longer something has been around the longer it will be around, or, equivalently, the shorter it has been around the shorter it will be around (both obviously taking about expected times.) Make sense now?

> I would read up on the topic if I were you. Statistics can be unintuitive sometimes, and basing your argument on supposition can be unreliable. I'm not sure about the correct answer here, but most folks get e.g. the Monty Hall problem incorrect if they rely on their intuition.

But this is a stupid-obvious problem. The Lindy effect is easy to write out in formalisms:

    age(x) > age(y) ⟶ life_expectancy(x) > life_expectancy(y)

Here's the claim that Etheryte says is the opposite of that:

    age(x) < age(y) ⟶ life_expectancy(x) < life_expectancy(y)

It is hopefully obvious that these two claims are identical, not opposites. But if it isn't, consider that we can rewrite the first one like so:

    age(y) < age(x) ⟶ life_expectancy(y) < life_expectancy(x)

Statistics aren't relevant to the question in any way; it doesn't matter whether they can be unintuitive sometimes.

There is no reason to assume content speed. Initial statement does not assume constant speed.

I mean, the 'initial statement' we're talking about here was my hypothetical example, right?; just pretend I decided not to leave out "assuming constant speed" after I first wrote it, and we're good? Unless you are referring to a different statement in which case we're all mixed up and have exceeded the carrying capacity of HN threads :)

You have to take a long time to walk a long distance. Regardless of whether your speed is constant.

The statement is A=>B. A implies B e.g. long life, implies longer life remaining.

The moment you negate A, both positive B and negative B, satisfy implication.

I.e. you can't claim shorter life implies shorter life remaining. For that to hold you need equivalency, not implication.

Here is an example. Rain implies streets are wet.

Does no rain implies streets are dry? No. There could be a flood or street cleaning, or a pipe burst.

You have somehow failed to understand what is being claimed.

The Lindy effect says that entities with longer realized lifespans have longer expected future lifespans.

In other words, if one thing has been around for 5 years, and another thing has been around for 3 years, then we know three things:

1: The first thing's expected future lifespan is f(5) years.

2: The second thing's expected future lifespan is f(3) years.

3: f(5) is greater than f(3).

Etheryte claims, in a gross error, that this does not imply that the expected future lifespan of shorter-lived things is shorter than that of longer-lived things. This is ridiculous; the claim Etheryte denies is a simple restatement of the Lindy effect. For our two objects of ages 5 and 3 years, the "new" claim would tell us the following three things:

1': The expected future lifespan of the first thing is f(5) years.

2': The expected future lifespan of the second thing is f(3) years.

3': f(3) is less than f(5).

But 1' is exactly the same claim as 1, 2' is exactly the same claim as 2, and 3' is exactly the same claim as 3. No claim has been negated, only repeated.

I salute you, Thaumasiotes, great explanation.

Ok. But then you just proved Lindy effect is at best probabilistic, and most likely survivorship bias.

Because it doesn't hold invariant to time.

It's not hard to make a counterexample. E.g. when Windows 3.11 existed for several years and Microsoft published Windows 95.

If we were to travel back then and use Lindy effect we would get wrong predictions.

Because it's such a simple heuristic it only demonstrates one way effect.

> But then you just proved Lindy effect is at best probabilistic

Obviously? Look at the statement of the Lindy effect:

> The Lindy effect proposes, simplified, that the longer something has been around, the longer it will probably stay around still.

What do you think the word "probably" means?

>Does no rain implies streets are dry?

It would, if the only thing that could make the streets wet is rain; ie if we were in a limited model where that was true.

We're talking in the context of a limited model in this thread. My point was that the statements are equivalent in this model.

Alternatively:

Imagine we were talking about the size of a pizza and how big a dinner it will lead to.

A blog says "wider diameter pizzas generally lead to bigger meals. Narrower diameter pizzas generally lead to smaller meals".

The first comment here says "just because wider pizzas implies bigger means doesn't mean narrower pizzas implies smaller meals, the blog has made a logical error!".

I reply "in the context of roughly round pizzas the two statements are the same".

You then give the example about the rain and streets. Hopefully it's clear that your general logical point isn't relevant to the pizza discussion, where there is a relationship between width and area.

Similarly, in the Lindy model, which we are actually discussing, there is a relationship between age and expected lifetime. That relationship is more complex than the pizza one, which confuses things, but hopefully the pizza example makes it clear why there isn't a general logical error here.

> It would, if the only thing that could make the streets wet is rain; ie if we were in a limited model where that was true

We are in a limited model. Many factors affect software lifespan.

It's a heuristic at best.

I remember a rule in audio speakers which sounds similar to Lindy effect: bad distortion (>=10%) are bad sounding speakers, but the opposite of that isn't true (e.g 0.001% distortion can still be a bad speaker)

Reply on news.ycombinator.com