How does Sidekiq really work?
dansvetlov.me
dansvetlov.me
Would also be interesting to compare this to GoodJob or basecamp's Solid Queue which work with relational databases instead of redis (and might be more specialised for use with Rails/ActiveJob).
Really? Where can I read more about it?
If so, why? Do you have trouble hiring?
If not, what did you switch to and why?
Wondering what folks are up to.
I think the pattern of a monolith that's very easy to rapidly develop on, plus specialist services around the edge that might be high performance/security/etc and are written in languages like Rust is a pretty good strategy. Personally, having seen the speed of development on a Django project, it's hard to justify much else for the boring CRUD core of a service.
[0] https://twitter.com/dhh/status/1701299614148919301/quotes
This looks like a web service framework, rather than a web application framework like Rails/Django. Like it's closer to Sinatra/Express than Rails -
The speed of Rails comes with rapid development & strong conventions leaving the important stuff (like business logic and building something) to you.
Once you get to a size where scale is an issue with Rails, you'll have a startup that's working - it's a better problem to have than a startup that has perfect fast code but no customers.
I think a part of the reason why it has become "boring" is that everything's already there and so you don't read every week about a new shiny thing like in the good ol' RailCasts days.
and still got some value out of it. Rails + Railscasts was truly a exceptional combination
People will knock the performance almost out of habit but there are major internet scale applications out there running this stack. Its horizontally scalable like many other web stacks and I would put developer happiness and ease of use as more important than some theoretical spreadsheet benchmarks.
Many developers just get bored and want to play with something new and shiny. Our industry has spent countless hours reinventing the wheel. I'm not saying there should be one tech stack to rule them all, just that the time spent inventing new frameworks for Resume Driven Development is kinda ridiculous.
I've been playing around with Ktor lately, and while a lot of it still depends on DSL's, it feels less magical and more like regular code, and I can easily step into the types/source of each DSL.
This is the worst part for me. Even though I've been using Rails for years, I still stumble on this.
Otherwise Django is quite good too and very similar philosophy to rails and ruby, but I like Ruby just a bit more in term of productivity.
Personally, if it's my own side project, I'm partial to Elixir and Phoenix, but definitely harder to hire for.
Would I use it for a new app, yes. But it is very opinionated, and the tools take a while to get working nicely.
Rails is magic, both in good and bad ways. The good generally outweighs the bad. And it has things (like Sidekiq) that I typically need in a project, and I don't have to waste time putting it together and can worry about getting stuff done in my developer role, and code quality in my team lead role. I do use Rust in some performance critical stuff - I used to use C++, but Rust just feels good to write, like Ruby. We have typescript on the front-end, but everything's React and React just never clicked for me in the "this feels natural to write" sense, so I don't venture into that realm often.
We don't have trouble hiring, other than the usual "not enough money in the budget" problems.
For example, you don’t need sidekiq or Redis as Elixir/Erlang is a concurrent, distributed platform. This simplifies operations and is cheaper to run.
This is simply wrong. You need a Sidekiq alternative (There is Exq which is protocol compatible with Sidekiq, others like Oban are available as well) because Erlang processes are not durable, nor do you get retry, concurrency control, etc. Everyone loves to claim Erlang is distributed and you can connect them so Redis is useless? but this is rarely utilized in the context of the web server. I have been using Elixir for more than 5+ years now, and I have never seen a valid use case for connecting two nodes.
In reality, the tech stack of the Elixir web app is mostly similar, but it makes life a lot easier if you ever have to deal with any kind of concurrency. Making concurrent requests is as simple as doing a map over a list and they just work, unlike other languages where you have to double-guess whether the library you use is thread-safe.
Have you never worked with websockets, LiveView, caching, OTP, etc? Those require distributed Elixir.
And okay, need durability? Oban - which runs without Redis and kicks the crap out of Sidekiq.
> Why?
To build fast. Our team knew Ruby well. Many of us worked at GitHub and trusted we could solve the problems ahead of us with it.
> Do you have trouble hiring?
No issues at all.
Fortunately, I don't have to choose either, because Phoenix exists.
I would and do use it for new projects as well. The ecosystem is excellent, Rails itself is adding new/useful features all the time. I don't know what would happen if our app scaled to crazy huge traffic loads, I assume we would adapt. Right now we're around 70k monthly active users but a year ago that was double and we didn't really have any performance issues. We've been able to use Sidekiq and Kafka to ingest huge loads of data on the backend without needing to do anything too crazy.
So yeah I don't really understand why Rails has to prove itself over and over again - it works, it keeps improving still after all this time. Its great.
Rails, Sidekiq, Postgres, Redis, boring technologies powering thousands of customers.
It's the ultimate batteries included framework. It provides high-quality, sane functionality for nearly everything. There are lots of high-quality plugins that drop in without much functionality. And, many companies write Ruby libraries for integrating with their APIs.
I enjoy trying things out on side projects, but almost always find myself spending way too much time trying to figure out how to do something Rails provides out of the box.
Great Functionality:
* Basic REST framework. This part isn't unique, but it works just fine.
* Data modeling with all of the lifecycle and validation stuff you need.
* Database migrations. I'm always amazed by how many modern tools just ignore this part
* Queues and Jobs
* Tests that work out of the box.
* Dev/Test/Production as a 1st class mindset
* Binary/Blob Storage out of the box.
* Email support (though I don't see most people use the built in)
* Pretty darn good security defaults
* View engine (though, I mostly rip it out to write a SPA)
* Caching
Amazing 3rd Party Functionality:
* Drop in Admin panels that mostly just work (ActiveAdmin)
* Devise for authentication. Includes all of the lifecycle basics (forgot password, reset password, email confirmation, etc, etc)
* Several authorization options. Not strictly necessary, but helpful.
* Scheduled/Cron jobs
Complaints:
* Convention over configuration can make it _extremely_ hard to understand or debug things when you don't understand the convention.
* Some people complain about computational speed. It's valid. In my opinion, though if you're at the scale where it matters, you have a really good problem to solve.
* Incompatibilities between gems can be annoying and challenging to resolve.
That's not strictly true. You can use them just fine if there's a good reason to do it. For example this erb file works as expected:
<% require 'socket' %>
<%= Socket.gethostname %>
When I saw he created a language agnostic sibling "Faktory", I started recommending it to all my non-Ruby colleagues looking for a background job processor.
TSR was an old simple trick in achieving a likeliness of non-preemptive multitasking on an operating system that didn't have any multitasking, but only one single active process.
Sidekiq (or Celery or nearly any task runner system) are designed for multitasking operating systems and they all keep the master process running, spawning/forking worker processes (or threads or whatever primitives they use for concurrency) as needed. They don't use CPU interrupts to stay active (IRQs aren't particularly accessible to user space), as most queue systems don't exactly allow it - typically they have a sleep-poll-repeat cycle. Though if the queue system allows it they may sleep waiting for I/O (which, in turn, indeed, could be triggered by an IRQ), but that's the closest resemblance I can find - and it's quite a stretch.