"You Don't Need Kafka, Just Use Postgres" Considered Harmful
morling.dev
morling.dev
There are plenty of choices between PSQL & Kafka. It's not like you take one step north and you're in the "oh no you better know what you're doing" territory.
Postgres land is a comfy place filled with transactions across all your data at once, one backup solution that you (hopefully) have had running for months or years and has been thoroughly tested, and ACID compliance. You have a single host, probably, which means that you are neither Available nor Partion-tolerant, but at least you are Consistent.
The moment you expand beyond a single database host you now have a distributed system, and woe unto you if you don't understand what that means.
You can work around this, but to the best of my knowledge you can't have consistency (between your existing Postgres database and your separate queue or event log) and simplicity.
But with PSQL you have even less built on top for a use case it wasn't meant for. Now you are in the "you better know what you're doing" territory.
When people use Redis/whatever, are you surprised that speed is just a side benefit and ergonomics is what they're looking for? Those are the same ergonomics you'd have to rebuild, as opposed to not build at all because it's already there.
If you want to rebuild Redis in PSQL you are very much in the "you better know what you're doing" territory, and you're also very much in the "are you sure this is your core business value" territory as you rediscover how much nice stuff was packed into the Redis ecosystem. And how am I supposed to uncover the surface or ergonomics of your Redis replacement?
Redis is fine as-is and I'm not suggesting anyone should rebuild the whole thing in Postgres, only that if they want a specific thing that Redis can do and they already use Postgres, then they might be able to do the same thing in Postgres with less effort and thus avoid losing atomicity and consistency and any other properties of Postgres that they find desirable.
[0] Which was mentioned by the original "You Don't Need Kafka" article
Do I really need to discuss why BullMQ is non-trivial software, or Redis Streams with consumer groups?
And I asked but you didn't answer. How am I supposed to discover the ergonomics of your custom Redis or BullMQ replacement? Do I read your hand-written docs?
I'm not suggesting that BullMQ and Redis are trivial and that you should rewrite them yourself, I'm telling you that Postgres has multiple existing implementations of a message queue already.
And frankly, the fact that something is non-trivial does not automatically mean you can't or shouldn't rewrite it, only that you need a good reason. Even if there was no existing Postgres message queue (there are many), atomicity is a good enough reason on its own. Imagine if you could avoid idempotency issues when calling third party APIs until you had more than ten engineering teams! Well, you can! Just don't add more than one method of storing online data to your architecture.
Take the approach that appeals to you and feel happy about it without big open source telling you "you're holding it wrong!"
Nailed it. I read the original post earlier this week and was very impressed with its technical detail. But the point of the the post was incongruent with the post's title. But the post got way more attention because of that title.
But if you think about the effort it took to write that post, the title was a really good bet on ROI.
> Nailed it.
Worked for the confluent marketing fluff as well.
Trying to explain the distinction between an event streaming platform and a distributed message queue to your enterprise architect is an exercise that no one should have to go through.
If you have to explain that distinction, are you really speaking to an "enterprise architect" ?
Disclaimer: I'm the author of this post.
> Managed services make running Kafka a very uneventful experience (pun intended) and should be the first choice
Confluent, you say?
Good idea; this is stated in the bio on my web site, but I've just added the same info again to the end of the post.
It might be worth adding a more direct call-out to posts like this one. Many may not go as far as reading the Bio page. That may be on them technically speaking, but still.
In any case, thank you for writing and sharing your considered opinion!
The author is a an employee for Confluent/Kafka so because his paycheck and equity grant depends on it and CFLT stock price, obviously whatever he writes is going to be heavily slanted in favor of Kafka. This isn't something that is a footnote at the bottom, it should be right up at the front.
I think that shouldn't matter but I still have a lot to disagree with the article.
feels like overengineering has become the standard for some people, and I quite dislike it personally.
Example: Someone writes a software that could use something simple like SQLite, and they switched to Postgres for performance reasons. Now unless what Kafka beings is the core reason they switched to Postgres not pulling in another dependency and adding a nother piece to the puzzle, can be a total legitimate engineering decision. And that renders the "considered harmful" utterly ridiculous.
Use a system like Kafka if you need what it brings (a distributed event streaming platform). If that isn't what you need or a very simple postgres solition suffices, go for that. Maybe you need event streaming but distributing it is overkill. Maybe you just need some sort of queue. Who knows? Not the author of this post.
Indeed that is the point I am trying to make in the article. Postgres oftentimes absolutely is the right tool to use, and oftentimes it's not. The thing I'm advocating to be wary of is "if all you have is a hammer...". This is to what "considered harmful" refers.
LinkedIn have moved onto Northguard... but no GitHub yet