Hacker News Reader

Comment by nurettin | Hacker News Reader

>> spin up a bunch of worker processes as EC2 spot instances that check for the next PENDING task

Does that mean workers poll the database?

Yup, but they do so only once per shard (well, twice, once at the beginning and once at the end). If you've got a big job where each shard takes a few minutes to process and a hundred or so workers, the DB gets about 1 req/sec, which is well within the capabilities of Postgres.

We had an "interesting" bug where we inadvertently ended up polling postgres advisory locks around 6 million times per minute from a cluster of 200 servers. It added about 20% load to the database, enough to be anomalous but not enough to outright trip any alarms. The overall database performance was fine.

(That's 100,000 req/sec)

Was "the database" a single node, or multiple systems?

I would lean towards maintaining a simple list on redis and letting the workers pop the next job from it whenever they are done.

That works fine too. The advantage of Postgres is that the master is a single point of failure, and if Postgres goes down all the state is persistent and you can just bring it back up again. Also, ad-hoc query tools are a little better for Postgres, which gives you a really simple interface to inspect the status of the job and the partial results.

This architecture originally evolved from an Amazon-SQS based queue system, where I'd stick the jobs in SQS and pop them off. Then I realized that I needed the DB anyway for my reducer-substitute, and if I'm writing SQL/ORM code anyway it was simpler to use the DB to manage the queue than to use SQS, and it gave me some other nifty abilities like being able to record timing & size stats with each shard, being able to query progress while the job is going, and being able to manually restart shards after correcting the errors that led to their failure. If you're running dozens-to-hundreds of workers, the cost of a t2 RDS instance (that's all you need for 1 QPS) is a rounding error, and a lot cheaper than EMR would be.

To me, the message queue is simply a substitute for polling the DB, not a way of persisting message status. That is a job for the DB.

Postgres provides LISTEN/NOTIFY for precisely this sort of use case.

https://www.postgresql.org/docs/9.1/sql-notify.html

Yes, any idea about why isn't it used in the original article above?

It is used in the article.

do you mean that there should be something like a controller that can spin up workers and also get rid of those that hanged (or retry) etc?

Reply on news.ycombinator.com