Hacker News Reader

Comment by eiathom | Hacker News Reader

It's interesting to imagine the head scratching going on in this thread. All these clever people, stumped. 'How can this be!???'. Quite the post, bravo OP.

But seriously, other comments have this nailed. This 'benchmark' isn't comparing apples with apples. There are too many quirks, optimisations and design decisions (of a compiler) that have not being taken into account.

Also, as others have repeated, this isn't indicative of actual usage - real world usage. In that regard, this type of 'benchmark' isn't scientific at all, and worse, is misleading a naive reader.

View on HN

While it's not (necessarily) indicative of the languages as a whole, it looks pretty apples to apples to me.

These are very simple programs with no fancy tricks going on.

It's comparing a granny smith apple to a mcintosh apple by hitting them with a baseball bat and inferring the texture of the apple from that. It uses them in a way that almost nobody does in real life, and is a vague proxy to only one of several important factors. If you use this test as a basis for deciding which you would like, you're going to have a bad time, whereas you'll be fine if you merely look at what other people like.

Is the "way" it's using things affecting performance significantly?

If not then it's a pretty good proxy to one or two dimensions of performance, and a mix of similar benchmarks could paint a good fraction of a reasonable picture.

If you want to buy a bushel of apples and hit them with an array of objects at varying velocities and then try to construct a model that can use the resulting splatter patterns to infer the texture of the apple as experienced by humans, you're welcome to do so.

Personally, I'm going to bite the apple.

I do not accept your analogy.

This program is a perfectly reasonable "bite". It's not doing weird stuff.

The program is a useless toy that nobody would write and expect to be performant in real life. I can write an optimized version of the algorithm that prints the exact same output (barring edge cases) without looping. With this new algorithm (really just a single expression), the C program is 500x faster than before and the python script is 5000x faster than before. So python is only 10x slower than C in this example (and I'd imagine that this is measuring more overhead and less arithmetic).

Presumably the C compiler has a relative advantage at optimizing the looped algorithm vs the arithmetic algorithm. How much? Is C 100x faster than python, or was the compiler able to get closer to the non-looping algorithm than python? How close? Is it something about the arithmetic or the looping that's natively faster in C? What part of it was optimized by the compiler? Since nobody does large loops in python, would python be more comparable if we introduced a C-backed library like numpy?

What other objects do you think we should bash this apple with?

Do you think a slightly more complex piece of math that can't be replaced with an equation would have significantly different performance from the benchmark?

I don't. So the simple math just makes things easier, it doesn't harm results.

Your suggestion to skip the math entirely is the apple-smash.

> Since nobody does large loops in python

Oh I doubt that.

Numpy would be fine as a different entry.

I don't know whether or not a more complex piece of math would change the outcome, so I would err on the side of not making assumptions until I had done more benchmarks and investigated the inner workings more. You probably know more about C than I do and might be able to make that assumption more confidently, but I think most people know less than either of us, and will read too much into a single statistic without context.

Nobody who cares about performance enough to look at a benchmark like this would do large loops in pure python.

> Nobody who cares about performance enough to look at a benchmark like this would do large loops in pure python.

Because they already know it would be very slow, and putting specific numbers on that is helpful for them and for non-veterans too. Even with some wiggle room for "microbenchmarks are weird".

I already know that python is generally slower than C. This microbenchmark tells me that python is 100x slower than C, but only in these specific circumstances that wouldn't happen in the real world, but may or may not be similar enough to the real world to infer that C could be 10^(2±1) faster, but only for a small portion of someone's program, and only if they don't use one of the libraries that they probably do use.

Personally, I find it misleading for the results of a benchmark to be posted when it's not made clear up front what the benchmark is measuring. At minimum I'd want a disclaimer that mentions at least one of the first two points mentioned in the top comment https://news.ycombinator.com/item?id=42250205

You keep acting like this code is super far from the real world, but it isn't. It's representative of basic number crunching just fine.

The one serious caveat is that someone that already knows CPython is godawful at this will switch to numpy, but that reinforces what the benchmark says.

None of the things in the post are going to change the order of magnitude here.

Nearly 25 years ago, nested loops and fibs.

https://web.archive.org/web/20010424150558/http://www.bagley...

https://web.archive.org/web/20010124092800/http://www.bagley...

It's been a long time since the benchmarks game showed pico-benchmarks

https://benchmarksgame-team.pages.debian.net/benchmarksgame/

Reply on news.ycombinator.com