Hacker News Reader

Comment by The_suffocated | Hacker News Reader

The research methodology in this blog post is fundamentally flawed. The author only counts how many people move from X to Y, but he doesn't count how many of them do not move at all. The whole diagonal of his (sample) transition matrix are actually missing values, but he treats them as zeroes. This greatly distorts the equilibrium distribution. As a result, he misinterprets each equilibrium probability as the "future popularity" of a language as well, when it at best only represents the future popularity of a language among those who constantly switch their languages.

View on HN

Author here. You are absolutely right. As I mentioned in the notes, I think this matters a bit less than it might seem like (the stationary distribution does not change if you add a diagonal matrix) but clearly some languages will have a higher propensity for people to stay.

I think this flaw is even smaller than the issue of using Google statistics to infer transition probabilities. It's just a shitty proxy, at best.

At the end of the day, there's a lot of assumptions going into this analysis. I hope I didn't make it seem more serious than I meant it to be – it's really just a fun project and kind of a joke not to meant taken seriously.

That being said, I think the conclusions are at least "directionally" correct. They might be off by a factor of 2x or 5x or even 10x, but the stationary distribution exhibits an even bigger spread (multiple orders of magnitude) so I suspect the final ranking is still "roughly" correct (with a very liberal definition of "rough")

The thing I like most about this post is that it's falsifiable. We will know in ten years whether C and Java are still popular, and whether Go succeeds in the sense this data suggests. So thank you for being concrete and clear, even if it's all in fun and other people don't like it :)

> whether Go succeeds in the sense this data suggests

An interesting thing about this methodology is that it is extremely sensitive to the age of a language. It's possible to switch from an old language to a new language, but not the other way around -- so if you happen to do your measurements after a language has had some uptake but before it's been around for long enough that people have built significant projects on it and subsequently gotten sick of it, the future distribution by this method can only be 100% New Language. (Because sometimes people switch to New Language, but no one ever switches away.)

Actually, to predict the future distribution of language use, you also need to know the rate of people moving from nothing ("I just had a brilliant idea!") to each language. If everyone eventually transitions to Go, but everyone starts in Ruby, then the division of market share between Go and Ruby depends in part on how frequently people start new projects.

The sorted stochastic matrix shows that C contradicts your assumption that it's not possible to switch from a new language to an old one. Or, at least, it shows that portions of new language code are occasionally rewritten in C.

What's missing from the matrix is "no language" language or null language. That is, a column and row that represents people who start projects from scratch in a given language.

I agree that the analysis makes it abundantly clear people move to older languages, but the question is what new projects are started in, and how many projects represent new versus transitioned projects.

This analysis is interesting, and gives a rough idea of what people are moving from and to when they decide to do that, but not necessarily popularity.

What the author is indexing in the end isn't really predicted overall language use, it's predicted transition target frequency.

I have defined "new language" in my comment as one so new that no significant projects exist in that language, not as one which is newer than C but still arbitrarily old.

> if you happen to do your measurements after a language has had some uptake but before it's been around for long enough that people have built significant projects on it and subsequently gotten sick of it

By this definition, it is not possible to switch from a new language to anything.

It's stated as a binary, but really this defines a continuum of newness, and the metric of the OP is very sensitive to it.

I didn't interpret the results in this post as "predicting the future distribution of language use." Rather, I interpreted the ranking as an indicator of a qualitative trend in that distribution. I think the author also made this very clear.

And of course, all things trend toward newness, so your objection there seems more about time or human psychology than the methodology of the post.

From the post:

> I took the stochastic matrix sorted by the future popularity of the language (as predicted by the first eigenvector).

Emphasis in original.

Nevertheless, it seems quite obvious the author did not literally interpret the eigenvector as "x% of future projects will be written in Go." Rather, the conclusions he drew were along the lines of "Oh wow look Go is on top, C and Java are still relevant."

I reserve the right to respond to what people say. Commenting on the accuracy of a label is worthwhile regardless of whether the label was meant to be precise or loose.

This is a great point. It's a model that predicts something about the future. I could have backtested it on historical Google stats to figure out if it's a good model :)

In an unrelated note: interested in doing a guest post over at Math ∩ Programming? I see you've got lots of cool stuff with high-dimensional NN :)

That bit about the stationary distribution not changing if you add a diagonal matrix sounds completely wrong to me. Let me see if I understand what you mean. Given a matrix M with non-negative entries (and no row of just zeros), let S(M) denote the stochastic matrix you get by normalizing each row of M. You are saying that if M is any matrix and D is a diagonal matrix with non-negative entries then S(M) and S(M+D) have the same stationary distribution?

Moreover data is collected over the entire history. A matrix is a linear operator from time step T_i to T_i+1. By conflating all historical observations into one matrix it definitely is not an ordinary transition matrix.

That apart from the fact that it is questionable that it can be represented by an operator that is finite and linear.

It's more likely a stochastic process (infinite matrix) with births and deaths.

I would be surprised if it became true. :-)

That was my immediate thought upon reading. A little more accuracy in the description would be helpful. This should be presented as: "If this aggregated data reflected a constant across time, then we can see where language usage would end up in the 'long run distribution'." But the jump matrix shown here will change with time. Most likely, if search results could be binned up by time period, the time dependence of the distribution represented by the eigenvector(s) would be somewhat interesting to watch, even if not remotely predictive. Could animate that or contour plot it.... exercise for the author... :)

Right. Both the matrix S and the identity matrix will project the stationary distribution onto itself. So any linear combination of them will project the stationary distribution onto itself. Let me know if I'm saying something really stupid

Oh, you meant "multiple of the identity matrix" when you said "diagonal matrix"?

Doesn't that mean you're assuming the number of people staying with any language X is the same regardless of X?

[[1 1] [1 1]] and [[10 1] [1 1]] will have different stationary distributions, the values on the diagonal will likely be different than a multiple of the identity matrix.

Another thing which seems at least as important:

How many projects are _started_ in a language, and how many _die_?

Given p.e. Java - It may seem that there is a huge flow to go. However, if there are enough new projects started in java, then the number of java projects might still rise faster than the number of go projects.

And I share my Go reservations with you, given the whole error-handling (or lack thereof) philosophy as well as information emerging that it may require 100 lines of Go to do roughly the same amount of work as 20 lines of Elixir or Haskell, according to one example at https://medium.com/unbabel-dev/a-tale-of-three-kings-e0be17a...

(Although I concluded the Haskell-Elixir equivalency myself based on functional semantics)

Go is verbose. There is a lot of thought behind that, but that is an intentional design aspect of the language. Personally I would not use gin, either net/http, gorilla/mux, or httprouter are solid choices

What is the empirically-determined advantage of verbosity, then?

A lot of it boils down to making it easier for developers to work with each other, rather than any technological benefit.

The less magic that happens and the more code that is commonly used by all developers, the easier it is for others to read your code and understand what it does. Rob Pike and others have some interesting talks and blog posts on this

Also, the likelihood of people blogging about a language change is not the same across languages.

Some crowds (Go and nodejs enthusiasts) are notoriously vocal due to the hype.

Finally, it's common practice for companies to have their marketing department to pay "media marketing specialist" to advertise for products (their language) by posting on forums.

Some languages and frameworks are simply too new to have a significant number of people moving away from them. Go and Vue fit that case. MySQL looks genuine, even if surprising.

Yeah, many languages will be almost entirely silent in transition. COBOL, FORTRAN, any proprietary language. And I would guess those transitions are usually to the "boring" languages: Java, C#, etc.

I wonder if the data could be normalized against usage statistics somehow. Maybe not perfectly effectively, but at least better than as-is, precisely because the number of people "switching" is so much smaller than the number using a given language.

Normalization would be extremely helpful indeed. C, C#, C++, Java, and Python are all very popular languages and have large values in both dimensions, which isn't particularly useful.

Presumably also on a time series. People used to switch from Ruby to Node.js, now they switch to ???

Agree. Depending on what one wants to measure, giving every sample equal weight can be potentially wrong.

If you mean to account for the larger variances of rarer events, that would be difficult. Perhaps some bootstrap sampling methods may help, but we need a true statistician here.

It would be interesting if the results could be controlled for time. Age of the language, year to year switch rates, etc.

At the bottom he writes, "the stationary distribution is actually independent of adding a constant diagonal (identity) matrix," but I'm not sure how that could be true (intuitively it doesn't make sense, but I don't know the math)

edit: A identity matrix wouldn't affect the stationary distribution, but if you had the actual "stay" probabilities they wouldn't all be the same, and thus not an identity matrix at all.

Adding a constant diagonal matrix indeed would not affect the equilibrium distribution. Mathematically, if the original transition matrix is A and you add a multiple of the identity matrix to it, then after normalisation by row sums, the new transition matrix becomes tA+(1-t)I for some 0<=t<=1. Since the Perron vector x of A corresponds to the eigenvalue 1, it must also be an eigenvector of tA+(1-t)I, because (tA+(1-t)I)x=tAx+(1-t)x=tx+(1-t)x=x. Hence the stationary distribution remains unchanged.

But as you said, the stationary distribution does change if the missing diagonal is not constant. And there is no reason to believe it is constant in the first place. In the end, what the author measures is still different from what he thinks he measures.

I agree with you, but your last line is wrong: the footnotes make it clear the author knows the omission.

This is similar to people measuring popularity of a library or language by stackoverflow question counts.

Sure, you can't get a lot of questions for an obscure library that on one uses, but the mere fact that tons of people post questions about it doesn't mean it's good or popular. It might mean lots of beginners want to use it because it's been marketed and hyped as easy to use. (Angular comes to mind).

It might also mean that it's confusing and poorly designed, leading many, many of its users to ask questions about it.

Exactly! (That's what I intended to say but I just realized that I didn't emphasize the point).

Or that the official documentation is poor.

The real issue is that he is not counting the projects that start from zero ('why did I do this project in php') and the ones that end ('why did I finish this project in Haskell'). If he adds this data to it's matrix somehow then it would be much more interesting.

> when it at best only represents the future popularity of a language among those who constantly switch their languages.

This is literally in the title

>The eigenvector of "Why we moved from language X to language Y"

The parent's point is that it is a very poor proxy for future language popularity.

I took this whole post as very tongue-in-cheek. No one should really be using this to predict the future. Instead, it was a fun data-analysis exercise.

I wouldn't call that a "flaw" so much as an "entirely different question".

No it's definitely a flaw. The analysis doesn't model what he thinks it models. He claims that it doesn't matter by (correctly) pointing out that adding a diagonal doesn't change anything, and that hides the modelling assumption he makes:

That there is the same absolute number of people choosing to stay within every language studied.

His analysis would only be correct if the number of people who choose to stay with Java at any moment is equal to the number of people who choose to stay with Rust. That is absurd and hence the eigenvector is meaningless.

Doesn't exactly matter much given that this is just a bit of fun on a blog post though. :P

Doesn't it make sense if you ask: Given that someone will change the language they're using (and writes a public blog post about it), which are they most likely to change to?

It's not unreasonable to use that as a proxy for industry trends. I recall reading about manufacturing jobs which, sure, might have lots of factories not changing, but the ones that do change _definitely_ opt for more automation with fewer workers. That's still a trend worth thinking about.

> but he doesn't count how many of them do not move at all. The whole diagonal of his (sample) transition matrix are actually missing values, but he treats them as zeroes

Great point. It's interesting to think about how exactly this could bias the results -- would it bias it in favor of languages that developers tend to initially not choose for their project?

I do see that as very consistent with C, C++, and Java being up there. For new projects, developers love to choose anything but those, but then they find themselves gravitating towards them when the project gets bigger and practical concerns intrude

I assumed it was a joke.

Great. We'll wait for you to come up with a better research then.

Reply on news.ycombinator.com