Mike Perham, Creator of Sidekiq: From Employment to Independence
codecodeship.com
codecodeship.com
> I still have 0 employees and don't plan to hire. I tune my business processes to run as lean as possible: Most of my customers are on credit card so their payments are automatic. The gem servers take about one day of maintenance per year. I can't really outsource much of my support work because it is so technical and specialized.
> My gem server is a $6 droplet on DigitalOcean. Because gems are just static files, that little droplet can handle millions of requests per day with just Apache. Oh, and I run three servers in parallel for failover purposes for a grand total of $18/mo.
I wonder what makes Sidekiq special? I don't imagine any equivalent product (background job processing) in the other languages is raking in that much, if making any revenue in the first place.
Mike has built a system that's rock solid, and he doesn't twiddle around with it willy nilly. He keeps it super stable.
The people paying for it (I'm one of those people) are paying for the reliability, in addition to the features.
Mike's also managed to find a real sweet-spot with the features he holds out for Pro/Enterprise. The stratification is just right to nudge you up, but it always feels like you're getting a good deal when you pay.
Time, know-how, and long term dedication are the key ingredients from my perspective. Maybe he's working 10-20 hours now, but there's support and development every day for the past 11 years.
> I don't imagine any equivalent product (background job processing) in the other languages is raking in that much, if making any revenue in the first place.
I guarantee, there are background job processors in other languages making revenue.
He was so understated that I missed it lol
I had no idea until someone just hinted me that this was self-referential... LOL, apologies for the now-embarrassing other comment I made LOL (I only know you as "sorentwo", whoops!)
[0] https://www.indiehackers.com/podcast/016-mike-perham-of-side...
I'm closer to $10m than $1m in annual revenue now.
My take on Sidekiq's secret sauce: a job system is a distributed system. Most of Sidekiq's commercial features are available as OSS gems but the complexity sneaks up on you as you integrate 3-6 of those features together. Building your own almost always leads to a worse system than the mature, well-debugged system which I have curated.
- Do you have any outside hired help? I see you still have no employees, but do you contract with anyone to do customer support as an example?
- How many approx customers do you have these days? $10M revenues = 10,000 customers. Is that roughly correct, if so - wahoo, congrats.
Genuine kudos to you, you should be an inspiration to any indie hacker.
Yeah, I had a Sidekiq question on StackOverflow and Mike Perham answered himself couple of days later.
I quit my job to go full in with EmailEngine exactly one year ago, when MRR for EmailEngine was $500, now it is $3,900 and steadily growing. I do work full time on it, but the result is async - the work I put into EmailEngine today brings me income sometime later, and the recurring income I receive today is unrelated to any effort I put into the company right now.
Any other great ideas?
I googled “emailengine” and it’s wasn’t super obvious which was your business since so much paid ads exist for that search.
TL;DR Nylas is hosted service that does way more than just emails. Also, it syncs all emails on monitored accounts to their DB, so when you make an API request, it hits their DB. EmailEngine is self-hosted and runs directly against the mail servers, so when you run an API request, the request ends up in the mail server. Both approaches have their pros and cons. Other than that, there aren't any other good solutions than building everything yourself by using any IMAP library.
Edit, looks like you've answered in the other thread, thanks.
Do you have any concern that a customer is going to look under the covers and re-implement your product? I would guess the overwhelming majority of customers are going to be happy to just consume the API and go about their own business, but some products attract copycats, and handing out code / build artifacts makes that easier.
The tradeoff you've made makes a ton of sense if you're not concerned about copycats.
Granted, it still does kind of suck to implement even these three, but I'm sure it would likely suck less if I were part of a bigger enterprise who could afford Nylas for example, which I looked into but seemed too enterprisey for my needs.
Edit, I just took a look at your FAQ and saw the price was 695 Euros a month. It is strange to not put this pricing on a prominent pricing tab I believe, rather than making me click a link in the FAQ to figure it out. Any reason why it's set up like that?
There are no limits on how many EmailEngine instances you can run or how many email accounts you connect. So, I don't differentiate between customers at all. I have single-person startups and large enterprises paying the exact same amount. Maybe not be the wisest option potential-revenue-wise, but this approach has worked well for me so far.
If you're worried about copying the API design, well, the implementation is the hard part, not necessarily the API design, which the Google vs Oracle lawsuit also showed, interestingly enough.
My thinking has always been that those who try to hack the license validation stuff and replace the missing build pipeline were never going to be my customers in the first place, so every second I would spend on them is a wasted effort.
[1] https://github.com/postalsys/emailengine [2] https://imapflow.com/
This is why I love the ruby community, so much sense :)
It doesn't have to be like this. Your community tried to be better once. Have you forgotten?
https://jasonfleetwoodboldt.com/courses/stepping-up-rails/ma...
OK, maybe it's only, say, 96% of Ruby users, or 100% of Rails users at least
It costs nothing to actually put down a reason for being upset, versus having totally generic downputs. Very few things are truly rotten to the core, most bad things have some bad aspects but could be much better if ___. If something is rotten to the core the central articles of faith for why that's so should be declared.
Plenty of Ruby folks have gone on to do great friendly Rails ish JS works. Ember.js is very batteries included, tries to show that yes all the potential is there to pave a nice well integrated happy-path system. To assume misery seems unreasonable; many have done fine.
I think we have a real obligation to do better to ourselves & one another than to foster shallow prejudices. Trump in his April 11th Tucker Carlson interview was asked why he thought Dems weren't worried about nuclear weapons. "That's because they don't understand life. That's because they don't understand what it is that you have to understand." This is an irrefutable claim. There's no point to start discussing here, no possibility that the other side might change or have some nuances. Let's not do this. Let's rise to higher places where we take real appraising concerned looks at things, rather than just dismiss stuff out of hand. We don't even have to be nice, but let's at least strive to be somewhat useful.
> Context also matters. Conversationally Mike's words could be an amusing wink & grin quip. I can see that. On paper, & seeing it repeated with the same reckless unnuanced antipathy, it lacks the personal connection & feels indicative of a general attitude situation that is quite prevalent.
"Don't get so upset it's just a joke" is sometimes indeed a legit statement, sometimes we do take things too seriously & need to be reminded. But when the light jabs keep coming, when there's so rarely an attempt to ever bring both parties into the laughing, when it's always jokes at expense, the action whatever it's intent becomes part of a larger behavior that is more than a lighthearted jab.
How many folks would stand up and say, no, there is no pattern of behavior about JS receiving short shift put downs? Do folks think there is no issue? How above board Vs how trashy do we think we've gotten with how programmer culture especially those parts that don't like it treats JS? What other parts of programmer culture endure repeated joke making at their expense today?
If you identify with JavaScript so strongly that you'll go over to message boards and rend your garments when you see people making fun of it, and you consider it a big problem that you have to do this so often... maybe consider whether there are some faults in JavaScript that lead it to attract a steady stream of mockery?
> What other parts of programmer culture endure repeated joke making at their expense today?
All of them. Programmers are an unpopular group.
I don't go out of my way to start this discussion every time. But sometimes I do. Because it's super notable to me how asymmetric it is. If there was more evidence of programmers dropping little nasty grams about any old topic, I could see it as cultural. That's why I tried to survey the room. It's remarkable to how persistently nasty people are to the most popular language on the planet. It feels decidedly unhacker like to bully me into silence over this search for meaning & inquiry.
I'd like an example of programmers persistently being mean & denigrating to other programmer spaces or languages. That was the context of the rest of my line of questioning, and implied by this question too. The popular/mainstream perception of programmers is unimportant & distracting, disinformation that misleads the discussion; why do some programmers enjoy dropping little vacuous nasty gram jabs at JS, and is this pattern repeated broadly against any other targets?
Are we concluding that JS is the only bad programming thing, thus no one else gets mocked? Maybe JS is a magnet given how popular it is? Or do we think the casual barbs really do go every way & I'm just missing it?
Actually, it does cost something- time and effort- but thanks to the latest AI, I can cut some of that out right now:
As an Elixir enthusiast, I'd like to point out some technical criticisms of JavaScript and its ecosystem, particularly in comparison to Elixir's approach to dependency management and community.
A) Dependency bloat: The JavaScript ecosystem, especially when working with Node.js, often suffers from excessive dependencies. Even when you require just a few dependencies for your project, you may end up pulling in a large number of indirect dependencies. This can lead to increased complexity, slower build times, a larger attack surface for potential security vulnerabilities, and an ever-growing workload spent applying security fixes that have nothing to directly do with the purpose of the web app.
In contrast, Elixir's dependency management system, powered by Mix and Hex, encourages a more conservative approach. Elixir libraries are often designed with minimal dependencies, and the community values self-contained, focused libraries.
B) Security concerns: The extensive use of third-party packages in JavaScript projects can expose your application to security risks. When you pull in a large number of packages, you become responsible for ensuring that all of those dependencies are up-to-date and secure.
Elixir's ecosystem tends to have fewer dependencies, reducing the potential attack surface. Additionally, Elixir's focus on immutability and the actor model can help minimize the impact of security vulnerabilities.
C) Lack of native concurrency and parallelism: JavaScript is single-threaded by design, which can limit its ability to take full advantage of multi-core systems. While Node.js introduced asynchronous I/O to help mitigate this issue, it can still be challenging to build highly concurrent applications in JavaScript.
Elixir, built on the Erlang VM (BEAM), provides excellent support for concurrency and parallelism through lightweight processes, message-passing, and the actor model. This allows Elixir applications to scale across multiple cores and handle a large number of simultaneous connections more efficiently than most JavaScript applications.
D) Callback hell and async/await complexity: JavaScript has traditionally relied on callbacks for asynchronous programming, which can lead to a phenomenon known as "callback hell" when dealing with nested callbacks. While the introduction of async/await has improved this situation, it can still be cumbersome and confusing for developers.
Elixir offers a cleaner approach to concurrency with its process-based model and message-passing mechanism, enabling easier-to-read and maintainable code.
E) Mutable data- One of the key differences between Elixir and JavaScript is the way they handle data. Elixir enforces immutability, meaning that once a data structure is created, it cannot be modified. Instead, operations on data structures in Elixir return new versions of the data, leaving the original untouched. This approach offers several advantages compared to JavaScript's mutable data structures:
1) Predictability and easier reasoning: Immutability in Elixir makes it easier to reason about and understand your code. Since data structures cannot be changed once created, you don't have to worry about unintended side effects or data being altered in unexpected ways. This results in more predictable and maintainable code.
2) Concurrency safety: Elixir's immutable data structures simplify concurrent programming by eliminating the need for locks, mutexes, or other synchronization primitives. This is because multiple processes can safely read and share the same data without the risk of data corruption due to concurrent modifications. In JavaScript, managing shared mutable data in concurrent scenarios can be error-prone and challenging.
3) Functional programming: Immutability is a core principle of functional programming, and Elixir is heavily influenced by this paradigm. Functional programming promotes the use of pure functions that do not produce side effects, which can make code more modular, reusable, and easier to test. While JavaScript supports functional programming concepts, its mutable data structures can make it more challenging to adhere to functional programming principles.
4) Reduced cognitive load: Immutable data in Elixir means that developers don't have to keep as much track of changing state as they read or write code. This can make it easier to understand the flow of data and logic in your application, leading to a more pleasant and efficient development experience and ultimately, fewer bugs generated per LOC.
The Elixir community tends to be more focused on stability, long-term support, and collaboration. This can result in a more consistent and cohesive ecosystem, where tools and libraries are more likely to work together seamlessly.
In conclusion, while JavaScript has its strengths and has undoubtedly played a vital role in the growth of web development, its dependency management and ecosystem can be seen as less than ideal from an Elixir enthusiast's perspective. Elixir's dependency management system, focus on concurrency, and community values offer a more robust, sane and secure alternative.
(How's that for "actually putting down a reason" as to why Javascript is terrible? Hey, you asked!)
I also think a huge amount of these tend to be pretty typical conservative-developer overconcern (see Yegge's Notes From The Mystery Machine Bus https://gist.github.com/cornchz/3313150), governing yourself on fear, that is probably not really as fully warranted or deserved.
But I'm also a huge fan of wild & fun & diversity. I also love that we try & do other things, that programming languages are not a Last Man Standing situation, & that we have great technical ecosystems with caring trying folks finding new ways to move forward & progress like Elixer. I look forward to seeing what lessons there prove really valuable (and perhaps aping them)! It's a choice in JS world but I do use immutable.js, for example, and it's great & as described as an advantage above. Ditto for some fp tools, but often I will mutate & use effectual/imperative styles, I often find there are significant performance or understandabity wins.
I would say, yes, the cost of talking vs snarking may be real, but it doesn't have to be this large & well examined a list as you've madd. I don't like JS, the amount of dependencies terrifies me. I don't like JS, it's too fragmented. I'd rather use Elixer, I like having a good actor model underpinning my ecosystem. Sharing some sort of tidbit can help calibrate where a person is & turn destructive contagious negativity into healthy discourse.
I don't actually hate JS. It's possible to write great JS. (And honestly, things like Elm do exist, and Svelte looks pretty amazing, but these also have their cost.) The failing for me is that there's nothing stopping anyone, especially a library I might decide to depend on, from writing bad JS. This was the same problem I had with Ruby. We spent all this time aiming for thread safety in our app (a whole team spent a lot of time on it) and it turns out the 2 culprits were 2 libs we depended on, because Ruby simply makes no guarantees about thread safety (and cannot, because it's mutable, thus making forking either expensive or dangerous). That and probably a few other things burned me enough to make me say "no more" and I suddenly loved the idea of a language where "thread safety" is completely a given and impossible to violate and you don't have to mutex-lock everything.
I don't think FP will ever get popular, unfortunately, unless we NEVER teach ANYONE ANYTHING but FP, because the people that pay the cost of learning the FP way are paying it because of all the ways they got burned by the non-FP (OOP, imperative, mutable) ways...
If you ever get there yourself, you have my hat-tip. And truly, it is worth the journey.
Regardless, it's probably better if we leave room for little jokes with each other.
I wish they felt like they had some punchlines though. No one ever bothers to make the JS world laugh along. We know it's wild here (Come play! So much fun freedom!). It comes off more like a beat down.
Context also matters. Conversationally Mike's words could be an amusing wink & grin quip. I can see that. On paper, & seeing it repeated with the same reckless unnuanced antipathy, it lacks the personal connection & feels indicative of a general attitude situation that is quite prevalent.
Again I think there's plenty of valid negativities in js, but looking at the distribution of where folks fall on the alignment chart when they talk about JS issues, it concerns me how lopsidedly & with what casual acceptance folks tend towards the scariest boxes of the chart. To me we are all in the challenge of making computing better together, & we can help ourselves by helping others.
1. When deploying a Rails app, the JS asset compilation is always the slowest part and is the most likely to break.
It doesn’t help that Rails has made a complete mess of JS assets, which I wrote about at https://fly.io/ruby-dispatch/making-sense-of-rails-assets/
2. For people who had to ship a Rails API + JS SPA, their workflow felt slow, brittle, and cumbersome. It wasn’t their imagination either—testing the stack required integration tests, which are always slow. Maintaining an HTTP API to talk to the SPA is additional effort that’s not needed, which Hotwired has demonstrated clearly.
I still think Rails has a lot of fuel left in its tank, thanks to Hotwired and the big companies behind it, but I agree the “Rails is the most productive framework” gets way overplayed. It was def “most productive” 18 years ago, but most other modern frameworks took notice, caught up, and have even surpassed Rails in a lot of ways.
The Ruby runtime leaves a lot to desire when you compare it to runtimes like Elixir/BEAM, Go, etc. I also think Rails has a terrible view layer, but most folks don’t quite understand that that means yet. This is something I’m working on at the moment.
Mike has always struck me as a great guy, really the kind of person who you're happy to know is building one of your favorite tools. In a world of starving open source contributors and mega corps throwing around weight, his success with Sidekiq stands out. It makes me very happy to see.
But for people who are interested in alternatives, I'd also suggest Good Job (runs on Postgresql).
This makes me curious about either
(a) conflict-of-interest rules at The Clymb that may or may not have governed the DTO directing the company to use his own separately-owned (commercial?) software.
(b) negotiating private ownership of software written for The Clymb.
One of those must have been relevant?
I never charged The Clymb to use the software. Previously they had been using delayed_job and were suffering badly from scale issues due to the use of MySQL as their queue store.
I've been using sidekiq now for almost a decade and had the pleasure to meet Mike at a Ruby conf many years ago.
Instead, we shell over the negligible monthly fees, spend no time thinking about it, and move on to solve real problems.
There are enough of us that it probably simply doesn’t matter.
I don't see the problem in having that kind of business model, it still allows the community to thrive and offers entreprises a way to have premium support.
Plus it allows him to invest more time in maintaining the free version.
In sidekiq without super_fetch (a paid feature), any jobs in progress when a worker crashes are lost forever. If a worker merely encounters an exception the job will be put back on the queue and retried but a crash means the job is lost.
Again, no problem paying for Pro, but I would prefer a little more transparency on how big a gap that is.
I have in the past monitored how many jobs were lost and, although a small percentage, it was still recurring thing.
Over the past week there were 2 jobs that would have been lost if not for superfetch.
It's not a ton, but it's not zero. And when it comes to data durability the difference between zero and not zero is usually all that matters.
Edit for additional color: One of the most common crashes we'll see is OutOfMemory. We run in a containerized environment and if a rogue job uses too much memory (or a deploy drastically changes our memory footprint) the container will be killed. In that scenario, the job is not placed back into the queue. SuperFetch is able to recover them, albeit with really lose guarantees around "when".
50,000,000 * 7 = 350,000,000
2 / 350,000,000 = 0.000000005714286
1 - (2 / 350,000,000) = 0.999999994285714 = 99.999999%
> It's not a ton, but it's not zero. And when it comes to data durability the difference between zero and not zero is usually all that matters.
If your system isn't resilient to 2 in 350,000,000 jobs failing I think there is something wrong with your system.
It's not reliability we're talking about, it's about durability. For reference, S3 has eleven 9s of durability.
Every major queuing system solves this problem. RabbitMQ uses unacknowledged messages which are pinned to a tcp connection, so when that connection drops before acknowledging them they get picked up by another worker. SQS uses visibility timeouts, where if the message hasn't been successfully processed within a time frame it's made available to other workers. Sidekiq free edition chooses not to solve it. And that's a fine stance for a free product, but just one I wish was made clearer.
I think it's fair to assume that something backed by Redis is not durable by default because that's not what Redis is known for, whereas the other options you listed are known for their resiliency and durability. I wouldn't view Sidekiq as a similar product to RabbitMQ and SQS.
Also, Sidekiq Pro uses more advanced Redis features to enable super_fetch lending to the assumption that by default Redis is not durable: https://www.bigbinary.com/blog/increase-reliability-of-backg....
OOM kills are particularly pernicious as they can get into a vicious cycle of retry-killed-retry loops. The individual job causing the OOM isn't that important (we will identify it, log it and noop it), it's the blast radius effect on other sidekiq threads (we use up to 20 threads on some of our workers), so you want to be able to recover and re-run any jobs that are innocent victims of a misbehaving job.
This doesn't happen at a high rate, but it happens more than zero times per week for us. We pay for Sidekiq Pro and have superfetch enabled so we are protected. If we didn't do so we'd need to create some additional infra to detect jobs that were never properly run and re-run them.
I'm still confused about what you're saying though. You're saying that the language of "enhanced reliability" doesn't reflect losing 2 jobs over about 50*7 million (from your other comment)?
And that if you didn't pay for the service, you'd have to add some checks to make up for this?
That all seems incredibly reasonable to me.
[1] https://gitlab.com/gitlab-org/ruby/gems/sidekiq-reliable-fet...
[2] https://redis.io/commands/rpoplpush/#pattern-reliable-queue
It’s hard to get this right though. No matter where the line gets drawn, free users will complain that they don’t get everything for free.
I guess you can say that hardware issues on your host aren't under your control, but it's under your control to find a host that doesn't have these issues. And not even a full-on ACID database is going to be 100% reliable if you yank the power cord at the wrong moment.
> it's under your control to find a host that doesn't have these issues
All hosts will have these issues, the only question is how often. If you need 100% consistency, then you can't use the free Sidekiq. Personally, I've never needed Sidekiq pro (as these kinds of crashes are extremely rare). But this will depend on your scale and use case.
> And not even a full-on ACID database is going to be 100% reliable if you yank the power cord at the wrong moment
This is only true if there's bugs in the DB, or some underlying disk corruption happens. The whole point of an ACID database is that they're atomic, durable, and consistent, even in the worst case scenario. If a power failure corrupted my SQL database I would feel very betrayed by the database.
I take your point that at a certain scale, hardware failure is inevitable, but if you’re running that many servers, you can afford sidekiq’s enterprise plan. It’s not something that will realistically happen if you’re just running like 20 instances on AWS. It’s perfectly reasonable to charge extra for something only large organizations with huge infrastructure budgets need.
I would say that queued jobs being lost is different from an in-flight transaction being auto-rolled-back, but it's not a super important distinction. Like others have said, I think Sidekiq really nailed the free vs premium features and its success is evidence of that.
No thanks.
Here's Resque literally using `lpop` which is destructive and will lose jobs.
https://github.com/resque/resque/blob/7623b8dfbdd0a07eb04b19...
Great point, and thanks for chiming in. I wonder if containerization has made this more painful (due to cgroups and OOMs). The comments here are basically some people saying it's never been a problem for them and some people saying they encounter it a lot (in containerized environments) and have had to add mitigations.
Either way, my observation is a lot of people not paying for Sidekiq Pro should. I hope you can agree with that.
"If I had asked people what they wanted, they would have said faster horses." or whatever the quote is.
There are tons of businesses selling products that are ideas brought to life from scratching their own itch or simply a desire to make something and put it out there.
Personally, I am one of those people who started a business based on an idea with no validation before launching it.
I built it in its entirety, then went to places and people who I expected would want it and low-and-behold, I am making a living doing it.
I'd bet money that the number of businesses which fail because they boil down to "a solution in search of a problem" is vastly larger than the number of businesses that succeeded despite performing "no validation".
That said, "making something" and "starting a business" are two different things. I would challenge you to point out where in the post he argues against making something, especially for the reasons you mention.
> I am one of those people who started a business based on an idea with no validation before launching it. I built it in its entirety, then went to places and people who I expected would want it and low-and-behold, I am making a living doing it.
Consider the possibility that you're in the minority there, and that you succeeded despite performing no validation.
> Sorry, this is bogus.
So if it doesn't apply to you personally, or in all cases, then you dismiss it as "bogus", full stop? Are you in the habit of doing this often?
Also, for future reference, it's "lo-and-behold", not "low-and-behold". [1]
The author states up directly, "Don’t make a website or an app. Don’t build a system".
Starting a business doesn't have to follow any real "formula". It can be a step in the beginning, middle, or end of a process. Yes, you should have the ingredients for a cake before you bake it, but you don't need to find someone who will eat your cake before you bake one or offer it for sale and you sure as shit don't need to "find real people whose problem you can solve. You listen deeply to find their dream scenario."
> Consider the possibility that you're in the minority there, and that you succeeded despite performing no validation, rather than because of it.
While I never claimed it was due to not performing validation, we can clarify that yes, it is not due to this, it is despite not doing it. Validation itself can be found in successful sales or other means, it does not need to be done pre-market and there are many examples of this, aside from the horses quote I provided.
And yes, I, in the same way the author declares it, declare it as bogus because these rules do not need to be followed in the way the author claims.
The post is riddled with questionable content, IMO, made to hit the wannapreneur market.
I don't dare tell someone how to start a business. It's their business, it's their journey, they should do it how they want.
I don't even necessarily disagree with your point that without being free those things wouldn't have taken off but we need to find a way to strike a balance in the developer community.
Sidekiq having a free version and an enterprise version walks an okay middle line imo.
So many communities across the web rely on people putting in their spare hours for free just to enjoy things. Whether it's spreadsheets in Eve, Addons and Weak Auras in WoW, forum analysis posts, or whatever goes on in the depths of pvpoke, so much free labor underpins massive parts of the world today.
I would love something that I could donate x money to per month and then based on usage, have it dole out to all the content providers with perhaps a minimum per month. It just seems daunting to do that as a) not a crypto scheme and b.) across all the various creator landscapes.
[0]https://www.polygon.com/2018/9/25/17901552/world-of-warcraft...