What is going on with measures of programming language popularity? | Hacker News Reader

What is going on with measures of programming language popularity?

techcrunch.com

112 pointstrapatsas86 comments

The article omits the Redmonk ranking [1], which is imho much better than TIOBE. The Redmonk approach uses github and stackexchange as complementary data sources, and shows a two dimensional main result.

The top 20 in the Redmonk ranking are

  1 JavaScript
  2 Java
  3 Python
  4 PHP
  5 C#
  6 C++
  7 CSS
  8 Ruby
  9 C
  9 Objective-C
  11 Swift
  12 Scala
  12 Shell
  14 Go
  14 R
  16 TypeScript
  17 PowerShell
  18 Perl
  19 Haskell
  20 Lua

Considering how different languages occupy different niches, the relative ranking withing niches seems reasonable. And ranking across niches scales according to size of the niche.

Client-side web scripting is a huge niche, hence javascript is in the top spot. General purpose desktop/server applications are another huge niche, served by Java, C#, C++, C, Scala, Go (in that order). And so on.

It seems pretty sane, and the results have been stable enough to interpret the rise of upcoming languages and see the resilience of top contenders.

[1] https://redmonk.com/sogrady/2018/08/10/language-rankings-6-1...

What??! No Nim?!

Nim is shown in the graph on the bottom left, at approximately X = 5, Y = 13.

Thanks, I was looking at the top 20 rankings in the post, not the site. I see it now.

Pretty sure I wrote this already at some point, but as a 9 year user who has not really starred anything in the last 8 years I still find this methodology flawed. Probably not as flawed as the other two, ok. Also there are some ecosystems where people don't just really use stackexchange, so those are also way off. Could be huge, could be nil.

Also, StackExchange doesn’t measure popularity, but popularity AND difficulty. All else equal, the harder language will have more SO activity.

I'm not convinced that SE activity measures difficulty. But it certainly is affected by much more than just popularity. I could imagine it measuring the total API surface, as you can ask more questions about larger subjects.

It's certainly a deeply flawed measure, but then pure search engine hits like TIOBE are likely much worse.

To be clear, I don't think it measures difficulty, but difficulty is necessarily a driver of the metric. A complex language is going to generate more questions than a small, intuitive language.

Specifically, complexity will generate more questions than difficulty.

No, it can’t by definition. Complexity is a type of difficulty. :)

It is also skewed towards what slightly-beyond-beginners use; many true beginners haven't discovered it, many experienced people more easily find the same answers without posting.

It's very good of measuring documentation quality. As in, the better your documentation, the less activity there will be.

With Stack Overflow most people measure number of new questions - but that's a partial measure that can lead to wrong conclusions.

How to measure number of pageviews instead? I have a solution!

- https://towardsdatascience.com/these-are-the-real-stack-over...

It would be interesting to see this combined with the github commits dataset on bigquery and compared to the redmonk ranking. The redmonk ranking is slightly different from the state of the octoverse ranking.

the Redmonk ranking [1], which is imho much better than TIOBE

What I think many fans of any sort of language popularity survey miss is that there are huge swathes of the industry that simply don’t actively participate in things like SO and GH. All of these results should be qualified as “amongst webdevs” (sorry “full stack engineers”).

The TIOBE index is hot garbage, and always has been.

Even so, this article is also pretty bad. Nowhere does the author define what he means by "popularity", which is one of the key problems all of these popularity contests suffer.

If we say "Language X is more popular than language Y?" What does that mean? Answers could be any of:

* Total extant corpus of X is larger than Y.

* Number of people who know X is larger than Y.

* Amount of open source code in X is greater than Y.

* Number of people currently writing X (how much?) is greater than Y.

* Number of people who want to be writing X is greater than Y.

* Number of jobs available for writing X is greater than Y.

* Number of people talking about X is greater than Y.

These are all wildly different metrics but all have reasonable claims to represent "popularity" and/or are what some of these rankings claim to show.

To do anything useful, you really need to know what problem the reader is trying to solve and pick a metric that helps that problem. Is the reader trying to decide what language to learn to find a job today? To get ahead of the curve and be an expert in five years? To discover a new exciting language? To choose a language to use for a large, conservative project? A small ambitious one?

I think the most useful popularity vector of a language is probably "What proportion of good programmers I interview will know a language" that vector would influence what language a company might chose to build a new product in in order to ensure they can find good developers to maintain it.

Actually a better question might be "What proportion of good programmers I interview will know a language in five years".

That's going to vary wildly depending on who you interview, which is going to depend on what sort of thing your company does. A game studio is going to get people that know C and Lua with a fringe group that are excited about Rust, while a big enterprisey company is going to get Java and C# developers with a fringe group that are excited about F#. (Maybe these examples are all wrong, but you get the idea.)

I absolutely agree, yea, and it's locality based... so up here in Vancouver it's really easy to find people familiar with AWS and system programmer types (Java/C++) along with the strong web development scene adding a lot of availability to react, node, a little bit of remaining angular.

But within that market it's further divided, if you're looking for non-game related Objective-C developers you'd probably be in trouble and so on...

It really is a super complex question, but that's the sort of ranking vector I'd be more interested as opposed to "How quickly do stackoverflow questions get accepted answers?" and I am under no illusion that it'd be super hard to measure.

This is a false aggregate: https://blog.shortbar.com/false-aggregates-571745b2d483

If I'm writing a game I care about what's most popular with games. If I'm writing an enterprise reservation system, I care about that.

These languages really live in completely different universes and I think it's time we acknowledge that. The one-size-fits-all survey of this sort feels like a holdover of a time when software development was far more homogeneous.

It's yet another example of how the entire profession is getting more specialized. You used to be a "programmer". Now you're a data engineer, ops engineer, front-end engineer, back-end/full-stack engineer, ios/android/react, system/OS programmer, etc. These are becoming entirely different jobs with little crossover between them, just like family law vs. criminal law vs. estate law vs. corporate law.

https://blog.shortbar.com/the-end-of-the-country-developer-7...

> You used to be a "programmer". Now you're a data engineer, ops engineer, front-end engineer, back-end/full-stack engineer, ios/android/react, system/OS programmer, etc.

I still wonder when there's actually going to be official training in these specialties, that counts as credentials when seeking employment. As far as I know (and I could be wrong), traditional education tends to be somewhat general.

Which specialty you're qualified for seems to have more to do with what job you happened to hold when the specialization split began, rather than something you deliberately decided to study.

(Okay, I'll admit that there are training courses in all manner of things, but they don't seem like a formal course of study you can actually put on a resume.)

You can totally do a MOOC in any of those things and put it on your CV

Which tends to be a signal for being unqualified

What would you consider to be “official” enough then? All EdX MOOCs are backed by real institutions, for example.

Or maybe we're starting to act like a real profession, adopting the idea of "continuing education" they have in every other field from law to dietetics?

Just a thought

I've been thinking about this recently.

The split started to happen about 5 years ago and it's really picked up velocity in the last year or two, to where it's getting hard to "jump" if you want to change. It's one of those slow, long-term things that's easy to miss if you aren't paying attention.

I think Udacity has a good model for this. I'm a generalist dev who has let curiosity drive my career more than anything else and feel I'm paying the penalty for it. Most places seem to want 5+ years specialization in whatever you're applying for if you're coming in at the senior level.

It's getting harder asking people to trust you if your resume isn't properly "tracked" into one specialty. I think that's a huge loss in terms of diversity and cross-skilling but perhaps necessary given the immense amount of sheer knowledge there is in each area, e.g. the entire iOS or Android API set.

As has always been. Academia is a trailing indicator for the job market, except in a very small group of innovative fields.

This is just a return to the norm.

There can also be local factors at work if you’re looking for a job. I had a job doing Elixir/Erlang (obviously not something people are writing everywhere), and did an analysis of job listings for languages in my city.

I ended up finding C#/.NET was listed on 33/50 recent job listings, Java was on 30/50 (some jobs had multiple languages listed), and other languages cropped up between 1-20 times or so.

So in my market, if I want a job where I don’t relocate (with a 3 year old and a baby and a wife who chooses not to work non-remote SV jobs are out), I’d better get to learning C#.

What's going on is that there are many equally-valid definitions of "popularity". Do you mean number of developers who's heard of a language? Who uses a language? Who likes a language? Or do you mean a language that's most widely deployed, for some definition of deployed?

The question you need to ask is what decision is going to be impacted as the result of knowing the "popularity" of some language. The answer to that question will guide you towards the right metric you should be looking at.

Indeed! Keep-up-with-the-Jonesism sends people into circles of craziness and self-fulfilling prophecies.

From https://www.xach.com/naggum/articles/3141310154691952@naggum... "if you aren't using today's fad language in the very latest version of the IDE, you'll be left behind. aaaugh! but it's good that some people run like they are scared out of their wits. if they suddenly disappear over the edge of a cliff, a good number of people will notice in time and _not_ follow them. those are the ones that matter."

That’s similar to asking: which species is the most successful one? By which measure? Bio-mass? Space occupied?

Ooh, that's an interesting question. I'd have to go with highest total energy consumed through their metabolism and activities. Those with the highest chemical impact on the planet.

Please don’t backward engineer the metric so that we will win ;-)

Yeah, factoring in activities certainly gives us the advantage. I suspect most people have an idea of success more along the lines of mass of other species consumed. That seems unfair to plants though.

I feel some measure of chemical impact of the species to be the most level playing field, and that relative total energy consumption is close to the core of the question.

I would go with the highest biomass :-). Whoever weights most in total wins.

EDIT: According to https://en.wikipedia.org/wiki/Biomass_(ecology)#Ocean_biomas... it's earthworms.

Is a giant sequoia tree, the largest of which was able to accumulate ~1350 tons over 2000 years, more successful than the quantity of worms that lived under it during that time? A continual measurement per unit time seems more fair to me than instant current weight at any given time. Maybe if it was possible to measure the biomass generated over the course of the year for each species. I'd think that's directly related to their energy consumption.

I would go with the time fitness takes to bounce back after an environmental change...

So us, then bacteria, and so on.

Why is that even the right direction of the axis? Couldn't you argue that the less energy you need for all your species' activities the better, if all else is equal?

Because energy is a limited resource, ability to extract a higher share of it implies competitive success. Also, it's possible to convert and directly compare the energy potential lost to the environment through the various mechanisms for metabolism and activity.

I don't really understand how you can measure popularity without doing a census-like survey, reaching out to every developer possible in a coordinated effort.

Awhile back, I took a statistics class that explained how polling everyone by calling everyone with a phone landline resulted in a misrepresented sample.

The reason: not everyone has a phone landline. And those with phone landlines had a tendency to belong to specific demographics.

So...if you survey everyone using Github, and not every developer uses Github, are you not prone to the same fallacy?

One example off the top of my head are the billions of lines of Cobol code that are certainly not on GitHub :) .

I'm not sure if Cobol programmers use StackExchange (I guess they do).

How about available jobs? I suspect a lot of people are trying to determine which language to invest their time in learning.

Another relevant measure would be ALL jobs, not just available. Some people may not care if a million people are using language X if they're not getting paid for it.

Seems like a good suggestion

Then, in the web sphere you'd have to focus on PHP (in Germany). But then again, looking at the details, you will realize that those job offers ask for developers who are fluent in: php, JavaScript, css, mysql, Photoshop (!), wordpress (!), etc. Reads like a job offer from the beginning of 2000.

But the sheer numbers cannot be ignored: Java / c# >> php >> Python >> Ruby/Rails >>>> Clojure... (Just one example)

Now you'd have to go into the details again to answer the question if you'd want to work at the problem set that is described in the job offer (how sophisticated, what kind of domain, how challenging,...)

> TIOBE measures the sheer quantity of search engine hits. PYPL measures how often language tutorials are Googled.

Popularity by who searches for language help like Java or C/C++ is mostly because those are more difficult languages with bigger, historical sprawl and frameworks.

Just using searching for popularity is skewed because people search more so for things they don't know, than ones they do, and harder languages will probably be more searched. Same with tutorials, it only shows what people are learning. However it is somewhat valid in that even languages you use daily you end up searching for solutions and information on docs, community, etc.

The github and repo stats probably give a bigger picture of popularity combined with searching and surveys.

Side note: Personally I think everyone should learn C/C++ and maybe a functional language or a dynamic language like Python in addition to their main languages. Learning C/C++ and building in it is closer to the metal, has memory management and every language you learn after is less difficult. I do love C++ for game development but it helps with learning all other languages as well as all other languages difficulty is downhill from it, and is especially great for memory management, stack/heap understanding, value/reference understanding and C/C++ is empowering in the power/speed of the platforms. To this day still most apps are built with C/C++ under the hood whether directly, exported to or in a virtual machine that is built in it. For highly performant apps/code/systems C/C++ still are king.

C++: it feels so good when the beating stops

Maybe object Pascal or rust for the less masochistic?

I’m ok with C if need be.

My day job mostly involves C code, which lives in a company repository; I only rarely search for C specific stuff. Hobby projects - work on OSS in C++/Python, or learning Rust - must result in an order of magnitude more of my language-specific searching and public committing, even though it represents a small fraction of my programming time.

So, for me, searching is mainly related to the learning about a language, not so much using a language. This makes me wonder if the integral over time of search volumes might be a useful measure of language use?

Article underestimates how much iOS app legacy Objective-C code there is; even Apple is a far ways off from replacing it all with Swift.

I think you meant "underestimate".

Good catch.

As a Rubyist I'm pleased with the Github ranking, but how much of that is tilted by Github being a Ruby platform initially colonized by Rubyists?

It's unclear how important it is today, but historically analyses that used Github heavily favored Ruby: https://www.benfrederickson.com/ranking-programming-language....

I'd expect that Github's scope/growth/ubiquity has mostly smoothed out that bump.

And GitHub rankings don't factor in things like the fact that java programmers may be under represented, because they tend to be older with enough experience and contacts to get work - without needing to show a GitHub project (plus if they have kids, they just don't have the time)

I'm in my 30's and one of the younger people in my department. I work in Engineering. I don't use Github I do not know anyone personally who uses it. I think it is probably a generational thing.

Search engine result as well I'd be suspicious of. If I am searching a programming language it is something I don't use very often thus why I am looking up the information. The languages I use the most often I have stack of books on my desk rarely need to look anything up with a search engine.

I don't put a lot of stock into TIOBE and similar lists because I'm not really sure how that information helps. It seems to me that these lists are measuring ubiquity rather than popularity, which I define as being well-liked. I guess there is value in that, but I'm far more interested in seeing a ranked list of well-liked languages and the reasoning behind those opinions. Slant isn't perfect, but it offers me the sort of pros vs. cons lists that I find much more valuable when selecting a language.

Job prospects/salary is another ranking I'm interested in. For example, I know I can get a JavaScript job on any corner, but is that ubiquity going to cost me in salary? Can I earn more by mastering something like Rust, Elixir, or Erlang?

I used to do a language popularity thing, langpop.

One of the takeaways is that there are different metrics and none of them is perfect.

So I grabbed what I could, combined them, and even had a JS thing where you could weight them differently.

Stuff I looked at: raw search, source code, books, job advertisements.

I've used the language popularity indexes to show MBA's that my preferred tool chain isn't some weird obscure thing that I just discovered on the Internet. The numbers were good enough for that purpose, and of course they didn't question the underlying methodology. Perhaps the more useful message for them is that there are in fact more than one or two popular languages, and that they are not all commercial products.

The big problem is all the major indexes are driven by search engine measures, and those themselves are highly sensitive to any change in the underlying search engine methodology itself. So a small change in Google's ranking algorithm will see a language move up or down dozens of places.

I would tend to focus more on job ads personally because those are at least linked to "real intent" to use a language for something tangible. You don't put it in an ad because something is controversial or had a lot of news lately etc. which can all lead to things being Googled a lot or getting a spurt of search activity.

Github's index is nice in that it is based on actual code, but then it's also heavily biased by what is open source and therefore doesn't fully reflect industry use of languages (which is why I think it deviates from other indexes to put Javascript and Python a bit higher, and less emphasis on say Java and C#).

There is an excellent book on such things that I highly recommend: How to lie with statistics

Problems like this are part of why we have sayings like:

Measure twice, cut once.

GIGO (Garbage in, garbage out)

Some saying about measuring to extreme precision, then "cutting with an axe."

> Measure twice, cut once.

I always thought that had to do with carpentry and woodworking, not statistics.

Same problem space to my mind: applying math to the real world in some meaningful, useful fashion.

Instead of popularity indices, I'd like to see the percentage that a language+framework takes in applications of a certain type, such as desktop applications, client-side web applications, server applications, data crunching, embedded, etc. That way you can spot trends or find alternative languages for a certain task.

Another interesting metric would be the number of open source libraries for the language that have had commits within the last 3 months and whose major semantic version is 1.0 or higher (or some other way to weed out unfinished libraries), sortable by license.

As Mark Twain eloquently put it...there are lies, damned lies and statistics.

JavaScript Is ambiguous these days. How much percentage of the JavaScript popularity is from the backend language Nodejs and how much of it from the client side(html, react, angular, etc)?

Why the distinction? It's the same language.

i guess it depends on what you are looking for?

in the frontend world, it really is only Javascript, so I don't think we need a popularity contest there. and it's in use by practically all the websites in the world as a dynamic renderer.

but when comparing a language like Java, PHP, Node, Python, I really don't want to see the results of frontend Javascript skewing the results. otherwise, one would just assume Javascript is the most popular backend language, but that is not the case, as stated in the article.

article states that one of the rankings had Java as #1 and Javascript as #8. but on the other, JS was overwhelmingly #1 (because it combined both frontend and backend).

you need distinction because frontend and backend are like apples and oranges.

Tiobe's faulty ranking method isn't even reliable. Three years ago Apache Groovy was in the top 20, but six months ago it wasn't even in the top 50. Now it's supposedly rising rapidly again. And this isn't the first time this has happened with Groovy -- it was also in the top 20 in Oct 2013 but three months later out of the top 50. I suspect Tiobe's rankings are being gamed somehow.

Github ranking by open pull request to me feels the best approach if what you are interested in is trends of language adoption.

Measuring total corpus would be measuring past popularity and may miss trends. Search metrics on the other hand may actually overrate newer languages which may be searched more compared to more mature languages. Opened pull requests sounds like a better metric overall.

I have absolutely no doubts that the most used language is JavaScript - considering nearly every website in existence relies on it.

So it's really just a question of methodology. When TIOBE says popularity they must not be talking about usage rates, or if they are perhaps they are specifically sampling some subset of enterprise?

> I have absolutely no doubts that the most used language is JavaScript - considering nearly every website in existence relies on it.

You should have doubts because every website you look at is on a device that most likely has tons of parts written in C.

Pretty much any piece of hardware you run your browser on, irrespective of whether you're looking at a website that has javascript code or not, has some C code running to support your javascript VM, browser, and operating system.

Whether you're on a computer, cell phone, gaming console, IoT device, you name it... you're most likely executing some C code quite frequently. And that's whether you're using a browser or not.

C is still the go-to language (no pun intended) for pretty much all drivers (with certain parts in assembly as needed) of any piece of hardware.

Right, but even that comes down to what measure of popularity you're using.

E.g., how many people are writing C, vs how many people are using things written in C.

Absolutely. That's why I reacted to the "most used" criterion.

TIOBE is, AFAICT, using a single web search engine (Google) and searching for a string representing the language like "C programming" or "Ruby programming", and taking a count of results. This favors languages that have a lot written about them due to age, or student interest, or whatever leads people to write a web page about a language.

It'd be interesting, and no more or less valid, to see how many paper books and ebooks have been sold on each.

GitHub takes count of the projects on its own site in each language, but that microcosm may not represent other places. Anecdotally, my employer runs private git repos on private servers, and our ratios don't match GitHub's.

Another interesting metric would be number of job openings. Another would be average pay for a developer mainly using a particular language. Another would be asking people in a scientific poll which languages they use and which ones they enjoy using.

> I have absolutely no doubts that the most used language is JavaScript - considering nearly every website in existence relies on it.

If you count by total lines of code running in production, then the most likely winner is probably C or COBOL (decent chance your paycheck relies on large COBOL applications working correctly). If you count it by lines of code times number of running instances, then I'm really hard-pressed to pick any other language of C because of its sheer prevalence in things such as cars. If you count by number of programmers, I'm not sure what the winner ends up being.

Even crufty old languages like FORTRAN or COBOL have surprisingly wide usage. Sure, they may be completely absent in user-facing applications, but it's easy to forget that such applications are only a small portion of applications. JS itself would probably end up somewhere in the bottom half of the top 10 in most metrics of "top languages," as it has very little presence outside of user-facing applications.

Both R and numpy have had some of their internals written in Fortran, iirc, so the amount of Fortran getting executed may actually be going up. The amount of people programming in Fortran is probably not, though.

> If you count by total lines of code running in production, then the most likely winner is probably C or COBOL

By lines of production, it's definitely Javascript. COBOL doesn't come close. What was a lot of developer time in the 80s is dwarfed by the number of people around the world working on websites alone (much less this Node.js silliness).

More Javascript is written every day than is produced in C, all year due to lower barriers to entry, lower cost to fail, easier to debug, less (possible) side effects, and number of developers.

> it has very little presence outside of user-facing applications.

That's where most code ends up getting written. SMH

Your first mistake is assuming that COBOL code was written in the 80s and untouched since then. People are still adding to COBOL code even today, and the language itself is still evolving (the newest COBOL spec was in 2014).

> That's where most code ends up getting written.

That's where most of the easy code to find and count gets written. But there is much more code that's outside of user-facing applications. Operating systems and drivers are relatively small, but have a much higher fraction of utilization since you're always using them. Embedded code is massive--not just the obvious things such as your car or airplanes, but also things like the microcontroller in your disk drive, the bluetooth controller for your headphones, your refrigerator, your microwave. There's the business applications: billing, payroll, corporate intranet, not to mention things such as scheduling the automated builds and tests or the management of releases for all of those applications. It's easy to forget that all of that stuff exists if you never work on it, and it's especially easy to think that it can't be that large.

If you avoid double-counting libraries (which is how I interpret the first number), then applications become a lot smaller because so much of applications is hooking together libraries. This is particularly true of non-app-y JS. By contrast, a lot of the business backend application logic (particularly for things such as payroll) is essentially 100% custom-implemented, so a much smaller fraction of it is common libraries. On pretty much any other metric of code use, though, such applications become far smaller in importance.

It may be true that Javascript is the most used language for "websites", but the percentage of programmers who work on websites is tiny.

Take any big company, go to their jobs site and browse through tech jobs to get a rough sense of what portion involve building websites with Javascript.

As one random example, I just glanced at Bank of America (http://careers.bankofamerica.com/search-jobs.aspx), a $300bn company that employees 200k people, and if I look at "Technology", I see pages of Java/ .Net/ Hadoop roles before I get to the first UI/ website/ frontend one.

> I have absolutely no doubts that the most used language is JavaScript - considering nearly every website in existence relies on it.

Just about everything around you that uses electricity is programmed with something other than JavaScript, including everything that runs JavaScript.

Depends on what you mean by "most used". Would you include copy&paste duplicates? I doubt that JavaScript remains #1 when you filter these out.

"The data doesn't match my personal preconceptions, so the methodology must be broken" Uhm yeah, nah.

The Stack Overflow developer survey is another data set that I trust.

https://insights.stackoverflow.com/survey/2018#technology

Nor is github a perfect indicator. Some languages promote their own open-source repository hubs, such as https://gitlab.common-lisp.net.

I dont think these measurements are any good or true reflection of a languages usage in the industry, some languages go totally under the radar, like clojure/clojurescript.

Reply on news.ycombinator.com