Hacker News Reader

> What, then, is a good way to compare languages?

Digging into details and yielding very specific conclusions.

Let’s take the example benchmarks.

The first one, Fibonacci, can only be properly assessed by inspecting the assembly. It is an analysis of compiler output, not of the language, not even of compiler optimizations.

It therefore only measures the CPU speed of function calls, and one compiler outputs a different set of assembly instructions:

• Go (go tool objdump fib) is all like “CMPQ JBE(label(morestack)) … LEAL(-1,cx) MOVL(cx) CALL(fib) …×3 ADDL(-2,cx) MOVL(cx) CALL(fib) MOVL ADDL …×3 RET label(morestack) CALL(morestack) JMP”.

• Crystal (crystal build --emit=asm fib.cr) is all like “subl(1) …×6 callq(fib) …×1 subl(2) …×7 callq(fib) movl addl …×6 addq retq” for the recursive condition.

So the conclusion is that Go outputs less instructions, but has to maintain its segmented stacks (useful for goroutines, its lightweight threading system) on every call, while Crystal uses the standard assembly calling system with little overhead beyond light dynamic typechecks.

So there is a language-motivated difference: calls in Go are a tiny bit more expensive, but built-in lightweight threads consume little memory because their stack can increase dynamically according to use[0].

The second benchmark, HTTP servers, has everything to do with the implementation of the default library, and nothing to do with either the language or the compiler. And I presume Crystal is backed by an optimized C library for TCP[1] while Go is probably a full reimplementation of TCP and HTTP.

[0]: https://blog.cloudflare.com/how-stacks-are-handled-in-go/

[1]: https://github.com/crystal-lang/crystal/blob/master/src/sock...

> Go outputs less instructions, but has to maintain its segmented stacks

I think they dropped segmented stacks in 2014: https://en.wikipedia.org/wiki/Go_(programming_language)#Vers...

The stack can still grow, but contiguously. I'm not sure if there are any instructions required to monitor the stack size and grow if it needed.

> CMPQ JBE(label(morestack)) …

^^ It's still the preamble for most functions :)

> And I presume Crystal is backed by an optimized C library for TCP[1]

On what basis are you presuming that?

What we did (and continue to do) in the TechEmpower Framework Benchmarks [1] is publish a set of requirements for our test types, provide a small set of example implementations on mainstream frameworks, and then solicit input and contributions from the open source community. Fast forward to today, and we'll be at 5,000 pull requests processed soon, and receive many from framework maintainers or advocates. Other test implementations are maintained by broader fans of languages.

Obviously there are an unbounded set of ways people can test the boundaries or contribute implementations that aren't in the spirit of the project. We've had to deal with several contributions that required push back or removal. Since we run our benchmarks project in our spare time, we rely heavily on the community to help us keep an eye on implementations, and also help determine where to set boundaries when they require clarification. For example, is pipelining in the Postgres driver acceptable given we disallowed batch querying? We ultimately decided it was acceptable because it's leaving the ergonomics of application development unchanged.

I feel that overall, the project has been effective in large part because the community has done a good job of self-policing.

[1] https://www.techempower.com/benchmarks/

I think this kind of language comparison is usually done to prove a point rather than in a dispassionate search for real truth.

I've never seen a situation where a team genuinely looks at the available languages for their project and chooses one based purely on performance. Even projects that are very performance-sensitive usually have other criteria (like not being able to use a GC language).

In most situations, there are a number of competing criteria for choosing a language. Team familiarity, deployment environment, availability of necessary libraries, compatability with other code in the organisation, compilation time, making maximum use of available resources. All of these, and many more, are usually as important as raw speed.

For a "real" comparison, there would probably need to be a lot more code being executed, and specific tests for specific environments - standalone web server? microservice? CLI? desktop app? mobile app? All of them will have different needs and preferences for performance.

I don't believe in microbenchmarks like this anymore, I find more value in the typical rewrite experience reports (with the usual caveat of not placing any gains only on the language, but also on the fact that the domain problem is better known so even a rewrite in the same language would have provided some gains)

There are different projects that allow people to compare languages in real scenarios:

- Good old TodoMVC (http://todomvc.com/)

- HNPWA (https://hnpwa.com/), that compares PWAs for accessing the HN api

- RealWorld (https://github.com/gothinkster/realworld) that does the same but for both frontend and backend. The API between both is standardized, so you must be able to mix and match any frontend (including mobile) with any backend. The idea is to build a Medium clone, so it's much closer to a real use case than a todo list

More than the language, the framework and the architecture can play a role in speed. If I cared enough about relative performance, I'd build a client that adheres to the API spec of RealWorld and simulate a few thousand clients on a few thousands posts, all reading and writing posts and comments, and see how well they compare under load. That would be much more telling than comparing who can produce the nth Fibonacci number the fastest.

The problem is that solutions optimised for benchmarks are usually non-idiomatic. If the solution doesn't represent "real code", then is there any real value? An interesting benchmarking approach would be to take solutions to the same problem from many different programmers at different skill levels and gather metrics from them. I think that would give a much more interesting view into a language's performance characteristics.

> If the solution doesn't represent "real code", then is there any real value?

When "idiomatic" performs so poorly that the software is unfit for purpose, does that represent "real code" ?

> … solutions to the same problem from many different programmers at different skill levels…

How could we know programs weren't written to make some language look bad?

> When "idiomatic" performs so poorly that the software is unfit for purpose, does that represent "real code" ?

I'm confused about what argument you're making here. I assume you aren't suggesting that suboptimal performance on a benchmark implies that the software is unfit for purpose? Can you elaborate?

I'm suggesting that "idiomatic" code in real software can perform so poorly that the real software is unfit for purpose.

Hence "idiomatic" may represent not "real code" but "naive code".

Fair enough, but I don't see how that relates to the conversation. Is that intended to rebut the idea that it's useful to know about the performance of idiomatic software?

You don't seem to have said "it's useful to know about the performance of idiomatic software".

I’m not the one you were originally responding to but it seemed pretty clear that that’s what they were getting at.

It seems clear that they conflated "idiomatic" code with "real code" into a subjective dismissal.

Do you intend to provide some support for your idea that "it's useful to know about the performance of idiomatic software"?

> It seems clear that they conflated "idiomatic" code with "real code" into a subjective dismissal.

No, they're dismissing it as useful for answering the questions we tend to care about when evaluating the performance of various programming languages. One such question is "If I choose a given language, how far will I be able to go with just idiomatic code before I need to invest in expensive optimizations (most of the expense here is that we're spending time optimizing instead of iterating on features)?". "Real code" is a function of this concern, so measuring it doesn't help us answer this question (i.e., if a language performs poorly on this question, "real code" will be more optimized and less idiomatic). Further, microbenchmarks don't measure "real code" either they maximally optimized code, which is what the OP was complaining about.

> If the solution doesn't represent "real code", then is there any real value?

When you're actually aiming for performance, in my experience code stops looking like "real code" pretty quickly in the hotspot, so benchmarks are representative in that way.

I mean, for example media codecs "written in C" are typically highly vectorised, architecture-specific ASM kernels with C around it for handling the flow.

Fair benchmarking is really hard. It is much better to compare languages on features, safety, programmer efficiency etc. But you can't boil that down to a single number.

The most fair benchmarks I've seen are to implement strictly numerical algorithms, but that's not very userful because you're basically testing a small piece of compiler/VM optimization at that point. It also leads to unidiomatic code because it's faster, e.g. "C in Java".

The most useful benchmarks I've seen are exercises where programmers of different languages try their best to create as fast a program in their language as possible. It allows you to see idiomatic v.s. fast code in that language. Downside is you're testing for a large part which contributors you've managed to catch.

Fair benchmarking is really hard! But I appreciate this style, because it shows two basic "real world" examples that you can extrapolate/estimate the results of. If Crystal is compiling to smaller binaries that run faster in a basic test without a lot of optimization and configuration tweaking, you can imagine that that relationship will hold true in larger codebases as well. Also very important is that the style, verbosity, and clarity of both examples showcase the differences in the languages on the programmer's side.

> * If Crystal is compiling to smaller binaries that run faster in a basic test without a lot of optimization and configuration tweaking, you can imagine that that relationship will hold true in larger codebases as well.*

Why would that be the case? I see no correlation. The two programs in the article were little more than glorified "hello, world" samples.

For all we know, Crystal might accumulate binary code cruft much quicker than Go as the project grows.

> After all, they are not apples-oranges comparison

And more: Them are the MOST useful comparisons. Compare 2 oranges is not as useful than compare an apple and orange. You have MORE useful differences and more insights on what make them similar.

In the case of speed:

I compare a bike and a car. Is unfair? yes. But, if the GOAL is how fast is go to the next city, is totally worth to know that the car is better.

However, if the goal is what is the most ENVIRONMENT FRIENDLY why to travel, the bike win.

Comparing 2 very different vehicles is far more useful.

---

What is important in benchmarks? To make clear WHAT is the goal to be measured. And not shy away to hammer if some choice become clearly superior.

> compare a bike and a car. Is unfair? yes. But, if the GOAL is how fast is go to the next city, is totally worth to know that the car is better.

The problem with these benchmarks is they have a propensity to not make a fair comparison. For example, sometimes they try to compare underperforming naive implementations in one language with expertly tuned implementations in another. Sometimes a blatantly underperforming implementation is used. Sometimes results are turned upside down by flipping compiler flags.

And sometimes photofinish results that fall within the standard deviation are sold as this absolute proof that language A outperforms language B.

Let's look at these benchmarks to see how these things are spun to sound like objective assertions are derived from meaningless data. Does it really matter if someone could use language A to shave off about 1MB from build artifacts? Are we really expected to take a 1.5s/5% difference in a cherry picked synthetic benchmark as meaningful? Is anyone in their right mind expected to drive their decisions on their tech stack on two cherry-picked tricks posted on a blog?

This is why no one takes these stunts seriously.

Never trust a benchmark you didn't falsify yourself.

The corollary is to write your own benchmark (i.e. something that remotely matches your use case).

Also it is almost always more interesting why something is slower/faster than something else than the raw numbers.

> For majority of programmers, the speed doesn't quite matter.

Sure. But if that code is going to be ran a lot, please try to keep the number of required nuclear power plants down.

Also don't forget about phone and laptop battery life either.

I think the problem is languages are very versatile and there's almost always several ways of doing things, so nailing down what constitutes a fair comparison is always contentious.

To use your analogy, I'd saying comparing two languages is more like comparing a woodworking workshop to a metal working workshop. There's a lot of similarities (they both have hammers) and there's a lot of overlap in functionality (you could make a little box in both workshops). But how do you make a sensible comparison between the two shops? Just saying 'you can build this little box slightly faster in the woodworking shop' doesn't really tell you much at all. They're tools that can be applied to a wide range of tasks, and unless you have a specific task you're interested in (e.g. I want to be able to process my log files as quickly as possible) it's hard to make any general statement about one set of tools being better or worse than another.

> What is a good way to compare 2 languages. Say Python vs. C?

The old 0install “language replacement” shootout is definitely the one I remember best.

It’s really outdated now (and oddly enough not always for the better) but:

* it evaluated multiple dimensions / criteria

* the criteria were very explicitly about relevance to the tester

* but their testing was quite objective

* and it really was quite extensive

I guess it depends on your use-case, but personally I'm interested in ballpark estimates of idiomatic code because that's sort of the performance space I occupy. I bucket pretty heftily (interpreted ruby, JITted Lua, Javascript, C) and that's enough. The specific details aren't important.

A more complete web benchmark might be one which connects to the database, does some queries, and renders the result. Then we'd look at what percentage of the time is spent in the runtime of each language, and apply Amdahl's law to see if switching is cost effective.

There isn't any easy answer.

The problem is that yes, they are an apples to oranges comparison. You may use them for the same problems, but different languages lead to different styles, different trade-offs, and even different ways of thinking.

As an extreme example, a slow language may make it easier to do some high-level optimization that you would never be able to do on a fast language, getting a much faster program. And we have no reliable way to measure or communicate that kind of thing.

Not exactly a "benchmark" but I think projects like ToDoMVC [1] - where the same functionality is implemented across different frameworks serves as an interesting and useful way to see the real differences.

This of course still misses how long it took to implement, how easily changes can be made, etc.

1 - http://todomvc.com/

> What, then, is a good way to compare languages?

With genuine interest and curiosity.

With the understanding that any conclusions will be provisional.

With incredible focus on the context that matters to you.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

The point of most such caveats is that for any non-trivial benchmark, implementation details will inevitably be different enough that it's unlikely a single benchmark result will give you much information in general about how to compare the two languages. But the other thing I'd say is that speed of execution is only one of many, many factors to consider when choosing a programming language for a project, and it's very often not a very important one.

A golang expert isn't going to flip to crystal based off of one benchmark.. Humans don't work that way. The first reactions to a post like this always involves golang fans trying to reconstruct the evidence into a reality that is convenient and proportional to their time invested into golang.

The results do say something about golang and crystal. Without more data we cannot confirm anything, therefore the following possibility remains an open question:

Crystal may be generally faster and more efficient than golang for most use cases.

To not acknowledge this possibility or to fully embrace it as an absolute is a form of bias.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/

People do whatever they can to "win" on these benchmarks, so the resulting code is often not idiomatic

The recommendation is to write only idiomatic code that implements a certain algorithm without tricks: https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...

Did you actually see programs that do not follow the guidelines or you're just stating this as what you would expect?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Templating Lua here is probably borderline really.

Seems like those Lua fasta programs should be removed.

What someone else considers to be "idiomatic", you might not consider to be "idiomatic".