Not clear to me why you would change the architecture of your system just to use more CPU cores. Can't they just run # of cores Token Servers and each physical server and load balance across them from their frontends?
This looks like a great use of gevent and zeromq. Anyone interested in author's PowerHose library (https://github.com/mozilla-services/powerhose), might also want to check out the protocol patterns documented by the zeromq guys, http://rfc.zeromq.org. Specifically the Worker Manager Protocol might be relevant to this use case.
I'm curious as to why he didn't experiment with the multiprocessing module and its Queue class. It'd be interesting to see the performance difference between it and a ZeroMQ solution for message passing to multiple workers.
I've seen comparisons of raw message passing, but none that involve actual processing of the messages.
I don't understand why the author didn't just release the interpreter lock before going into the C routines? See Py_BeginAllowThreads