Hacker News Reader

Comment by latchkey | Hacker News Reader

If this was part of reducing operational overhead, why not implement something functionally like GCP Cloud Tasks [0]?

Since this is part of Rails, all you would need to do is implement regular http endpoints, no need for workers/listeners. Submit a "job" to the queue (which itself is just a POST) and the message details: the endpoint and some data to POST to said endpoint.

The queue "server" processes the list of jobs, hits the specified endpoint and when it gets a 200 response, it deletes it. Otherwise, it just keeps retrying.

[0] https://cloud.google.com/tasks/docs

View on HN

How does this reduce operational overhead? You still need a queue and a worker to dispatch the tasks to your api.

The end user doesn't need to implement the queue or the worker. They just need to implement the http api for receiving POST messages. The queue "server" is effectively just a http client that reads messages from the database and sends them off.

That makes sense as a general pattern, but probably less so in the context of rails which is typically a self contained monolith. It would be adding another hop, more indirection, and more complexity. It would introduce new problems like the need to segment your "real" api from your worker api for the purpose of load isolation.

If it is a self contained monolith, that's perfect for this. You have a long running process, which scans a database for new work and then posts to localhost.

We use to do this by making use of the cloudtasker gem.

However we've recently just migrated to Sidekiq. We regularly found that the time to enqueue a job with cloud tasks was anywhere from 200-800ms. We often have workflows from users where they need do something that would involve a batch of say 100+ jobs. The time to process each job was minuscule, but actually enqueuing them took a long time.

Time to enqueue with Sidekiq / redis was brought down to under 20ms. We can also make use of bulk enqueuing and enqueue all 100+ jobs in one redis call.

However when we were first starting out, it was a godsend. Made our infrastructure significantly more easy to manage and not have to worry.

Cloudtasker is a wrapper around the actual GCP Cloud Tasks service, that's not what I'm talking about here. What I'm talking about is the implementation detail of Solid Queue itself.

Why would GCP Cloud Tasks be part of the default libraries? If it’s that simple surely a community maintained gem would be fine?

To clarify: I'm talking about implementing the core functionality of GCP Cloud Tasks, not using that specific product.

Reply on news.ycombinator.com