Hacker News Reader

Comment by mperham | Hacker News Reader

I have dozens of customers doing 1b+ jobs/day. The world is bigger than you think.

I'm not sure that's a particularly fair conclusion to make Mike.

1bn a day is order 10k/s. if you the creator of Sidekiq know "dozens" of people hitting that rates, if anything you're supporting arcticfox's point that these systems are quite rare.

No shade intended. I have been a very grateful Sidekiq Enterprise customer in past Ruby jobs.

It's a piece of mind thing. Everywhere I worked did a lot of stuff on the background. The db is already so full on a normal Rails application that adding more reads doesn't make sense to me when Sidekiq works so well. You never have to worry about Sidekiq throughput, or when you do is to be careful to not overload the db because Sidekiq is so fast.

Rails is already on a slow language, I don't understand why we want to overload even more with slow things.

Having both available is a huge win for the Rails ecosystem.

If you’re just starting or have a small project Solid Queue is invaluable and reduces costs and friction.

If and when you need more performance or power Sidekiq is still available. Both run in the context of your Rails app so moving jobs from one to the other shouldn’t be terribly hard.

Yes. But as with all things Rails I do wish the default performance could be pushed a bit. So say Solid Queue could work for 90% of project instead of 70 - 80%.

May be ZJIT could help.

Exactly, it's not an either/or situation. I've worked on a codebase that did 3-4k/s jobs through sidekiq and in this case sidekiq is obviously the best option (although we also used a go sidekiq client for certain tasks that didn't need the rails app context) but using sidekiq from the beginning is premature optimization. You can easily start with Solid Queue and switch over to sidekiq when you actually have data that tells you that it'll potentially become a bottleneck. Premature optimization is a complexity cost.

Dozens is just the number I've talked to who have told me their scale. I'm sure there's a lot more.

And ultimately the precise number is not as important as the fact that it *is normal* for midsized Ruby apps to operate at this level of scale. The more you start to use the background job pattern, the more you tend to create. "A Job for every ProductVariant record" can easily create 10-20m jobs.

You want tools that scale to the size of "success". That would be a good marketing tagline for Sidekiq.

Reply on news.ycombinator.com