Hacker News Reader

This question is more suitable for SO than HN.

Given a four core CPU, if each request is lasting 30 seconds without I/O, then each request is going to use 1 core. On a 4-core CPU, you would only be able to serve 4 requests within a 30 second span. What is the request doing for 30 seconds? How much of it is actual CPU-bound work and what amount is I/O.

View on HN

The request is harvesting data from Facebook. For users with a tremendous amount of data, batch requests must be made, so it can take a while sometimes.

I can't say authoritatively for Python, but Twisted is probably what you need here (I have used both Node and EventMachine to do things like this, and I'm pretty sure Twisted is similar). The event reactor model will let you defer the request while it's doing things like performing external IO, so your application is freed up to handle other requests in the meantime.

Reply on news.ycombinator.com