Hacker News Reader

Comment by adamt | Hacker News Reader

Whilst I applaud the efforts of PyPy, I've always been disappointed with the performance benefit it's given me with real world code.

Specifically:

* PyPy Dict performance is very slow (slower than Cpython). A lot of the code I need to write that is cpu-intensive python code processes data (from stuff like logs) into various data structures that use dicts. Pypy is rarely more than 10% faster and sometimes slower.

* The memory management/GC can be bad. I've seen the same code that runs fine on cpython end up using excessive amounts of memory (and causing out of memory issues etc) with PyPy. Again - this is normally involving complicated data-structures.

On about 10 occasions now I've had CPU bound Python tasks, then I've tried to use with PyPy and never had > 20% performance improvements. Which is a big contrast with the benchmarks.

Is it just me? Or have other people had similar experiences?

[edit: typos]

View on HN

The benchmarks are a little bit misleading for some types of code. They do not measure single run code well. In these cases pypy doesn't do as well as when the JIT has warmed up.

The interpreter is slower than the CPython one. For code which doesn't JIT well, it will go slower with pypy. The firefox JS engine now has a fast interpreter, a quick JIT, and a more optimizing one. This means the baseline performance is better in CPython, so if pypy manages to make their interpreter better baseline performance will get better.

The memory management in CPython is much more predictable and deterministic since it is using reference counting and not GC. The pypy GC may be faster in many circumstances however. IO in pypy can be slower sometimes if the GC is unhappy with the way you're doing it. Especially if you're leaving files open, or reading in different sized chunks of data.

The new pypy release speeds up a few different types of real world code however. XML processing, DB processing, and stackless async code (eventlet, and greenlet) are three areas where pypy has improved with this release. Numpy based code is another area where pypy has gotten better with this release.

In short, pypy still has many areas where it could improve... but with each release is getting better at more types of real code :)

For run-once, and latency or memory sensitive code the benchmarks may be a bit misleading.

"The firefox JS engine now has a fast interpreter, a quick JIT, and a more optimizing one. This means the baseline performance is better in CPython"

Huh?

From what I understand, there are three stages:

- interpreted

- If run 'enough' times, the code is JIT compiled (fast compilation)

- If run 'enough' times again, it is JIT compiled with the most optimizing compiler (slow compilation)

The question is what the firefox JS interpreter has to do with baseline performance in CPython.

I've used PyPy in production with success. The task involved implementing a worker which needs to process Wikipedia data.

http://rz.scale-it.pl/2013/02/18/wikipedia_processing._PyPy_...

Also I'm using it with my web applications.

Have you used pypy recently? I've found that memory usage in particular is much better as of around 1.9, compared to previous releases. Still worse than CPython, for sure, but some of my code is around 10x faster under pypy (all depends on what I'm doing, though, for sure; this stuff is numerically heavy).

I've had exactly the same problem. I have had tree-manipluation AI problems which run for > 30 minutes (seems like an ideal thing for pypy, before I rewrite them in C++). pypy is almost always slower, and never more than about 15% faster, whereas a simple line-by-line C++ rewrite can be 20x faster.

Please help PyPy by providing an example benchmark. The project is test and data driven, and the more data it has the better it can be

Out of curiosity, was your python code relying on dicts or using structured classes ? It looks like Pypy is better at optimizing classes than dictionaries. Which might explain to some degree why people are not seeing the perf improvement they expect. A lot of existing python code rely on dict manipulation which gives decent perf on CPython where classes would play nicer with Pypy.

Python is a vast language. It's very hard to know upfront what sort of patterns people use - if you don't talk to us, don't post stuff on the bug tracker, don't do anything - it's your own fault. PyPy is known to speed up real world code to various degrees - sometimes 10x sometimes not at all, but it all really depends. We would be happy to help you with your problem, but if the only thing you do is to complain on hackernews, well, too bad, we can't help you.

I am sympathetic because PyPy is very good and is improving fast. but...

That doesn't change how the PyPy project tends to represent itself, which almost always comes across as something like "6x speedup for everything (excluding JIT warmup)". If you want everyone to adopt PyPy instead of CPython then it is part of your job to find the cases where PyPy is not actually faster rather than saying it is the user's fault. And it is not your job to select only benchmarks which tell the story you want.

If the difference between interpreters is nuanced then that nuance should be expressed so that people can make intelligent decisions rather than dismissing one or another interpreter as "slow".

FWIW, it's trivial to get benchmarks into their comparisons, provided they aren't microbenchmarks, at least in my experience. Saying they pick benchmarks that are favourable is untrue: the majority were added because of PyPy performing badly on them, and they've improved as a result of being included thus making them mostly benchmarks PyPy does well at.

I have at least a 10 fold increase. In my project I have many lists of integer tuples and I need to do a lot of comparison/arithmetic on that.

Reply on news.ycombinator.com