Comment by kbuck | Hacker News Reader

Comment by kbuck | Hacker News Reader

Parent Full thread

I actually experimented a while ago by running a long-running Twisted-based daemon on top of PyPy to see if I could squeeze more speed out. PyPy did indeed vastly increase the speed versus the plain Python version, but once I discovered that Twisted was using select/poll by default and switched it to epoll, my performance issues with the original CPython version were gone (and PyPy couldn't use Twisted's epoll at the time).

Another major issue was that running the daemon under PyPy used about 5 times the memory that the CPython version did. This was a really old version of PyPy, though, so they have probably fixed some of this memory greediness.

What version of twisted and os you were using? I'm asking because all latest Twisted releases are using epoll by default.

It's worth noting that PyPy also supports epoll (and kqueue), and has for a few versions.

I remember looking at that, but Twisted's epoll reactor was a C extension at the time. It looks like Twisted 12.1.0 switched to using the epoll provided by the Python base library, but that was released about a year after I was originally installing this daemon (and I was installing everything from apt, so add another year to the age of the packages I got).

This was with Twisted 10.1.0 and Python 2.6.6. I remember being in extreme disbelief when I found that it was using select/poll instead of epoll (who does that, especially when they already have epoll support?). I ended up writing this:

  for reactor in ["epoll", "kq"]:
      try:
          rn = reactor + "reactor"
          getattr(__import__("twisted.internet", fromlist=[rn], rn).install()
          print "Auto-selecting reactor: " + reactor
          break
      except ImportError:
          pass

A couple of years back when I was deploying twisted using Debian packages, it was not using epoll by default.

Yep, this was on a Debian 6.0.2 install, with packages from apt.

Reply on news.ycombinator.com