Hacker News Reader

Comment by envolt | Hacker News Reader

Can you shed a bit more light on this -

> job scheduling systems that could be replaced by a postgres table

I've never used Postgres in my life ever.

I think the point is to use a DB as a message broker. Schedule a job by adding a row to a table. Have services running that secure a lock on a row, run the task, and then update/remove the job from the DB.

Django-Q (the multiprocessing task queue framework, annoyingly named the same as Django Q objects), allows this as one of the configurable brokers. https://django-q.readthedocs.io/en/latest/brokers.html#djang...

+1. And I think specifically the SKIP LOCKED Postgres feature is relevant for this -https://tnishimura.github.io/articles/queues-in-postgresql/

Oh, I hadn't noticed that. I'm not sure how Django Q manages with a non-postgres DB, since it doesn't mention that as a requirement.

I have used nowait before to batch scheduled jobs together for a slow API call.

Why would you pick django-q over Celery, which also supports a database backend?

We've had a lot of mysterious "it just stops working" issues with Celery, and their # of open tickets is approaching scary levels.

Moving everything to rq, and so far so good.

Are you sure that changing your implementation didn't fix a bug?

I've noticed that a lot of "dependency" problems on "complex systems" are usually programmer implementation problems. Changing the implementation forces you to review the original implementation (or throw it out) and fix the bug that caused you to change the implementation in the first place.

We’ve had exactly the opposite experience. RQ was an absolute nightmare in every aspect across two companies I’ve worked at. And when you need something that Celery supports that RQ does not, you are in a pickle.

I wish you the best, but I cannot help but think you will regret that decision.

Also, Kiwi.com gave a talk[1] about their experiences with RQ and how/why they migrated to Celery at EuroPython last week. It's worth a watch.

1. https://youtu.be/SUQYuA9AJT0?t=7546

Have never experienced this and we sold a huge number of our app using celery. Based on other replies one should pause, take a step back and consider if their code or design is to blame..

Funnily enough, we also use this approach in Google. There's at least once chapter in the first (the unactionable) SRE book about this.

As long as I want to avoid very edgy race condition cases (low throughput), any DB will work right?

Either Mongo, MySQL or List in Redis.

Is there anything sepcific about Postgres when it comes to this? (except SKIP TABLES, as mentioned in other comments)

I said postgres because it's just all around good at a bunch of stuff. But I've used redis lists in the past to great success. Redis streams might be even better but they weren't ready the last time I needed a queue.

rq ("redis queue") for python does this with, obviously, redis.

API is pretty small and comprehensible.

It can't. Not easily.

Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse.

There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes)

However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines

What Jon Richards said. Also see https://github.com/collectiveidea/delayed_job which is a popular DB-based scheduling library for Ruby.

Reply on news.ycombinator.com