Yet Another Language Speed Test – Counting Primes
bjpelc.wordpress.com
bjpelc.wordpress.com
There's no C++ features really being used here besides true/false (which basically gets treated as 1 and 0 in most compilers[1]) so there shouldn't be any difference between the two - even the generated code should be identical. The C++ code even uses printf instead of cout, which is one area where I would expect a difference. Thus I'm surprised by the large difference between C and C++ at the low end.
[1] On x86 and other ISAs with flags, booleans can be returned in them, which is an optimisation that I haven't seen compilers do (yet) on x86. It's common on embedded ISAs like 8051 though.
You could only return bools in the flags for static functions that don't have their address taken, because this would break the established API. It'd also generally only be useful if the function result is immediately used in a test.
This is also easy to see by looking at the second derivative of the graphs. When n is large enough to outweigh startup time, the graph should be linear. The areas where the second derivative is non-zero indicate that start-up time is affecting results.
You can see that all the languages have some kind of startup time impacting results. Java in particular has the same performance as C/C++ under this test, but has a significantly larger startup time.
Really experiments like this should split the results to explicitly show these two aspects of performance. Measure startup time and measure the time running the actual test code separately. Though that might be more challenging on JIT'd languages.
https://bjpelc.wordpress.com/2015/01/13/yet-another-language...
Notably, it should be mentioned that writing idiomatic
Python and Ruby results in much slower code than that
used here. Ranges bad. While loops good.
But unless you're doing computationally expensive operations, I don't see the benefit of these speed tests except to provide ammunition for flame wars. What good is it for your code to run 50% faster if you spend 300% more time creating, debugging, and maintaining it?
There's also the issue that among languages like Python, standard libraries (like multiprocessing[0]) make parallelizing code trivial (even with/despite the dreaded global interpreter lock). I rewrote some of the code the author uses and on a smaller number (6710886 instead of 67108864) I'm seeing ~1/3 the time to run compared to his baseline with 4 worker processes (27.469517s vs 1:25.373176). I haven't looked into what might be keeping it from being a full 4x faster, but my hunch is that it's a combination of other things going on in my laptop and the undeniable difference in execution speed in cPython (specifically, cPython3.4).
One might make the argument that this is a wildly unfair comparison, but my bigger-picture argument is that a language that abstracts things like parallelization to the point that one can invoke such features in a line or two of added & changed code is probably worth factoring into an analysis, if we're even interested in analyzing languages in this way.
If we're going to write C-like code and run it as Python, should we be surprised when it doesn't live up to those expectations?
[0]: https://docs.python.org/3.4/library/multiprocessing.html
[] ~/projects/jruby $ time jruby primes.rb
Number of primes in the interval [0,67108864]: 3957809
real 6m26.536s
user 6m43.472s
sys 0m4.967s
[] ~/projects/jruby $ time rvm ruby-2.2 do ruby primes.rb
Number of primes in the interval [0,67108864]: 3957809
real 11m17.610s
user 11m16.564s
sys 0m0.738sApproximations to this computation are even more fun: Asymptotically, this function tends to x/ln(x), or more accurately to the integral 1/ln(x) dx from x=2 to n, whose asymptotic expansion is x/ln(x) + 1!x/(ln(x))^2 + 2!x/(ln(x))^3 + ...
However, after swapping out the Float only capable square root from Swift Foundation for the C sqrt function, leaving the rest of the code identical, the execution time on the Swift code went down to 35 seconds, basically identical.
I have to wonder if Python, Ruby, and PHP just have a Float only capable square root function, or handle numbers with much higher precision than necessary by the test.
double sqrt(double)Overall, be very careful of micro-benchmarks like these. The conclusions that you can draw from them are tenuous at best, and usually have little bearing on the performance of an actual program.
This looks like an opportunity for the Ruby interpreter to optimise: if the range is excessively large, it could decide itself to loop over the bounds rather than instantiating the entire range.
I have no illusion that this is definite test of speed of languages.
I could see myself compile with Cython here and there if the performance gains would be meaningful but I do not want to write Cython.
However, as I understand, Py3's type hints are derived from a previously available module (which I think also worked with Py2). So why would they add compatibility for that now? Why not just write your own parser that reads types annotations and then outputs Cython-equivalent variables declartions?
There are also some Cython specific types that you won't find, for example the functions can be either cdef or cpdef, depending if you plan to use them from C only, or C and Python.
I still see value in learning Cython as a complement to Python, for the cases where you need to get "closer to the metal". It's not so much more, and it helps a lot in performance.
The link: https://www.youtube.com/watch?v=4LlCqlMJr3E&t=18m50s
The whole talk is valuable, of course, but it might not be your thing if you're not into numerical coding.
I did a straight port of C to lua, and ran it with luajit.
The C code ran in 49.901s.
The luajit code ran in 14m16.547s.
C code was compiled with -02 -std=c11 -lm (version 4.8.1, mingw).
luajit.exe was 2.0.3, static build, using VS2012 x64
"The luajit code ran in 856.547s."
The graph is log-log which can be a bit tricky to read but that puts it in the same ballpark as Ruby and PHP (assuming the same factor of difference between the two, after scaling your C vs luajit results to match the given C value of ~40s, luajit becomes 687s.)
Edit: fixed luajit version
If you see anything wrong with it, or odd, feel free to share.
I'm still investigating what is happening to make the run so slow, so if you can find something wrong in my code, that would help.
Replacing math.mod(n, i) with (n % i) gives roughly 9.4x performance.
EDIT: luajit version was LuaJIT 2.0.4 on Mac OSX
* Oops, I had cheated and used a for loop instead of while. The LuaJIT time was the same but the others are 10.5x, 12.58x, 15.63x.
$ math
In[1]:= AbsoluteTiming[PrimePi[67108864]]
Out[1]= {0.009661, 3957809}
on the other hand:
combining the first 2 ifs into if (n <= 2) { if (n < 2) return 0; else return 1}, would eliminate an unnecessary if statement check that will be false for most values.
>If n < two: return false.
> If number is equal to two: return true.
> If n (>2) is even: return false
Further, your suggested optimization is terrible. Even if it turns out that's slightly faster (which I'm not convinced it is: did you actually compile that code and look at the generated assembly or profile it?) there are much bigger wins to be had by using a better overall algorithm, such as the sieve of Eratosthenes, which was mentioned in the article if you had read it.
I know I'm being a grumpy old man here, but if you decide to comment on articles you either didn't read or didn't understand, you've only yourself to blame.
When you benchmark, it's important to know what you are doing or you'll arrive to the wrong conclusions because of unknowns you aren't aware of.
Writing some code to generate performance numbers and jumping to conclusions without understanding what influenced the numbers is unproductive to say the least. These conclusions influence a lot of code that will be written in the future.
If you cannot tell me where the bottleneck is (what's saturated), then you are better off not doing anything.
Thank you, Captain Obvious, for that very simplified explanation of junior undergrad systems architecture. That would all be very important if branch prediction misses were your biggest problem. Maybe look up the Sieve of Eratosthenes people keep mentioning because hey, people keep mentioning it for a reason!
Arguing about branch misses in this algorithm is like arguing about branch misses in a bogosort implementation. The correct optimization isn't to remove branch misses, it's to stop using bogosort and use a reasonable sorting algorithm.
I'll charitably believe that you read the article, but if you're still complaining that the code is unoptimized when the author clearly said it was unoptimized then you didn't understand the article. The article isn't about code optimization, it's about gathering performance metrics for naively-written code. It's profiling the language, not the code.
> If you cannot tell me where the bottleneck is (what's saturated), then you are better off not doing anything.
I'll tell you one thing for sure: the bottleneck isn't branch prediction misses.
EDIT: Just to tack on more reasons you're wrong here: 1) You don't actually know that what you did removed any possible branch prediction misses without looking at the generated assembly, and 2) you could probably eliminate more branches using a switch-case statement (although again you'd have to look at the generated assembly to see if that's the case).
It's excluding compiler time, and it's only outputting a single number. As such, there's nothing preventing any of the optimizing compilers from optimizing the entire program to essentially print("pi<num> = <num>"). Ditto, time to load interpreters is included, whereas time to load compilers is excluded.
Also, note that the output is different in format for the different languages.
On a different note: he mentions PyPy in passing but doesn't post benchmark numbers. It'd be interesting to see how PyPy fares, especially with the slower-in-Python idiomatic version.
That has clearly not happened here. If it did, you would see a flat line regardless of n. Nevertheless it is a good idea to make it e.g. take n as a commandline parameter, even if only to make testing different values easier.
Ditto, time to load interpreters is included, whereas time to load compilers is excluded.
You don't have to load the compiler every single time you run the program, whereas that isn't the case with an interpreter.
Also, note that the output is different in format for the different languages.
It's performing a single output operation at the very end of the computation. Any difference in that would be completely lost in the noise as n increases and the computation time dominates.
On a different note: he mentions PyPy in passing but doesn't post benchmark numbers
PyPy is there between JavaScript and Ruby:
Who cares?
This covers real world cases. If an optimizing compiler could actually figure out to optimize those programs to a printf() call, that would be an impressive level of optimization and I'd want to know about it (note that no optimizing compiler achieved that level of optimization). Likewise, I don't know about you, but when I run interpreted code, I often start by loading an interpreter.
If you want to profile a different use case, do it, but that doesn't mean profiling these cases is "completely and totally meaningless".