Hacker News Reader

Comment by alberth | Hacker News Reader

How does PyPy 2.0 compares to LuaJIT 2.0 [1] with regards to memory usage and performance?

I ask since they have similar goals for their respective languages.

There is very limited data for non-artificial and non-numeric benchmarks for Lua and Python, so treat it all with a grain of salt.

PyPy mostly cares about large scale programs (thanks to a large quantity of Python libraries it's not that hard), while LuaJIT usually cares for smaller stuff.

Last time I tried to run richards (which is the largest example I found for both Lua and Python), PyPy was able to eventually outperform LuaJIT slightly, given enough iterations, however LuaJIT warmup time was very very impressive and it was much faster at the beginning.

From a very unscientific perspective I would expect LuaJIT to mostly outperform PyPy, especially on stuff where it's feasible to compare, like by implementing small numeric kernels in both Lua and Python, but I haven't measured a whole lot in this area. LuaJIT definitely outperforms PyPy on the computer language shootout benchmarks. I would also expect the field to be more leveled if you implement say a webserver in both, caring about performance, but this is really on the guessing side.

> PyPy was able to eventually outperform LuaJIT slightly, given enough iterations

Is this benchmark data-intensive? One known weak point of LuaJIT is its GC, which is slated for a rewrite: http://wiki.luajit.org/New-Garbage-Collector

If "richards" allocates lots of objects that accumulate and need to be GC'd, this could explain why PyPy wins eventually.

"[Richards] simulates the task dispatcher in the kernel of an operating system."

http://labs.oracle.com/people/mario/java_benchmarking/richar...

That doesn't answer the question of whether the benchmark is written in a way that generates lots of objects that need to be GC'd.

I'd be very surprised if PyPy beat LuaJIT for anything that was numerical or otherwise had a fixed working set.

[Richards] > Is this benchmark data-intensive?

A draft paper where C F Bolz and I look at this is here: http://tratt.net/laurie/research/publications/papers/bolz_tr...

You can also download the full experimental suite and check out the particular benchmark(s) yourself. In my opinion, fijal has provided an accurate summary of the situation as we currently understand it.

> PyPy was able to eventually outperform LuaJIT slightly, given enough iterations

It would be awesome to save the state of the JIT. It would also be awesome to export the type inference information.

The problem really is Python import semantics. As soon as you can provide me with a way to say "this imported code is identical to the one you loaded before", I'm fine with providing a way to save the JIT state. Note that "identical" means also module globals, state of sys.modules etc. etc. In short, the assumptions are all the same.

> As soon as you can provide me with a way to say "this imported code is identical to the one you loaded before", I'm fine with providing a way to save the JIT state.

How about an SHA2 hash of the source files?

That would not be sufficient. Python modules can execute arbitrary code at import time, so the behaviour could depend on a number of external factors, such as environment variables, other files, current time, response from a remote service (?!), etc...

That would not be sufficient in some corner cases. I think for a competent group running web application servers, which knows it needs to control such external factors, it could well be enough.

but this is not the attitude you have to have when writing compilers. from the compiler standpoint, it has to be correct, even in corner cases (which are not even as corner as you would like them to be).

A competent group can leave a dormant process that communicates over pipe for example and then the warmup issue disappears (it does not really appear for long running web application servers to start with)

Have you given up on automatic handling being feasible? It seems like the performance wins could be enough to justify something like a module-level constant explicitly stating that you're not doing metaprogramming, monkey-patching, etc. and allowing implementors to optimize as desired.

<rant>

Oh, well ...

The Lua port of Richards I've seen is naive and nobody within his right mind would write Lua code like that. As has been pointed out elsewhere, Richards is a very atypical benchmark and different implementation styles have very different performance characteristics.

In fact, PyPy wins ONLY on Richards by 1.2x and ONLY for a specific iteration count. But LuaJIT wins substantially on ALL OTHER BENCHMARKS where something is actually compiled (i.e. not regex stuff). Your paper clearly shows that in the appendix, but no matter what, you're still pointing out that one little result.

And, even worse, you're trying to draw conclusions from this outlier. I have absolutely no idea how you could possibly come to the conclusion that this makes PyPy more suitable for 'large scale' programs? And that LuaJIT is not? That's not just unscientific -- it's dishonest and disparaging.

Also, constantly repeating that 'hand-crafted JIT compilers' are somehow inferior to PyPy's overly complex approach is getting really old by now. LuaJIT runs on more architectures and more platforms than PyPy ever will. And it's vastly ahead in terms of optimizations. Now, please compare the timeframe all of this was implemented and the size of our teams. Ok, so what was the better deal, again?

And, no, the semantics of Lua and Python are not that far apart as you'd like everyone to believe. Dispatch is more complex in Python, but that can be taken care of in a central place in the code. Most library code can be written in terms of lower-level code. And so on. There are simple solutions for all of that. And none of that warrants a clumsy meta-tracing approach.

</rant>

I know myself that I had a high regard for LuaJIT before benchmarking, and an even higher regard for it afterwards. Richards is an oldish benchmark, and none of the subsequent versions (Lua, Python, Converge) is exactly "native", so I'm not sure that any of those languages has an advantage. It's a datapoint, no more, no less. As to which VM would be faster on "large" programs - well, personally I agree with your implication that it's a matter of conjecture, and we'll probably never know because it's hard to write large cross-language benchmarks.

I do, however, think you might be a little harsh on meta-tracing. It's an interesting approach, in no small part because it allows VMs to be written for languages other than Python. LuaJIT can be retargeted too (I've been impressed by Thomas Schilling's work), but I think it would be fair to say that it's harder work than writing a new RPython interpreter. In other words, IMHO, there is plenty of scope for both LuaJIT and RPython/PyPy to co-exist happily alongside each other.

Hi Mike.

Seriously, I would not try to say LuaJIT is worse than PyPy, ever. I picked richards, because it's the best benchmark from the set I know, while still being not a very good one. I'm all for saying "this is a bad benchmark".

I think you're putting words into my mouth. I did not say PyPy is better for large scale programs, I said we have more focus and I derive it from what we use as PyPy benchmarks. Since it's impossible to compare, because we lack programs we don't actually know, but my gut feeling would be that LuaJIT would be slightly faster on something large.

The problem with Python (and why we choose our approach instead of hand crafting the JIT) is that Python is a very large language, with lots of builtin modules and types interfering in very unobvious ways, while Lua is not. This is why we decided it makes more sense to go that way, we probably wouldn't do it for Lua. People tried to write a JIT by hand for Python and failed so far, I'm however fine with the explanation we're just too dumb to do that.

Overall I find your answer way too agressive - I didn't claim PyPy is better, I mostly pointed out to things I know and added it's very unscientific and that scientific approach is very hard to get. Also your attack on our meta-tracing is unfounded - Python is just a very very large language with strange corner cases and we have to somehow deal with it. Feel free to implement it differently, but I'm happy with our approach.

You did say that LuaJIT "usually cares for smaller stuff," implying that it isn't optimal (or as optimal as PyPy) for larger stuff. I could see how someone who had worked for years to optimize LuaJIT for all cases could find that disparaging. It sounds like you're writing off LuaJIT as only meant for toy programs.

Well. the luaJIT benchmark page does not contain any large benchmarks, while it does contain computer language shootout ones which are small. There might be many many legitimate reasons for that, but it does imply that LuaJIT cares for small stuff more, since all the public benchmarks are small.

Reply on news.ycombinator.com