>small scale kafka, though. It's conceptually great to have everything work off of logs, but kafka does add a non trivial operational burden.
does something like that exist ???
does something like that exist ???
Writing a distributed, scalable system is really hard, and beyond the API, that is the real value for kafka
It's a weekend project to demonstrate the concept, maybe a few weeks to really flesh it out and iron out quirks. I imagine if you're willing to use sqlite as a backend for persistence, it gets a bit easier.
You have employees - you try to get and retain the best talent you can. However, every human has strengths and weaknesses, and these may not all be fully visible to you.
Rolling your own vs buying off the shelf is a gamble on future outages.
Will a third-party support and fix the issue, or have a strong community that can help you work through the issue?
If your best engineer builds something that works for long enough to become entrenched, but then carks it, will your best engineering talent be able to resolve the issue? If your rockstar quits, does the team have to pick through the halls of Cthulhu? Does your organisational ignorance of kernel networking suddenly become painfully apparent?
Remember, you need to be twice as clever to debug your code than to write it...
Every single person I know who's done this says it was a fantastic decision, and the "eventually I'll have to migrate to X" never came.
Having an event sourced system doesn't make much sense unless you maintain messages from the start of the system. You can snapshot state and resume in order to quickly rebuild from a known good state. That doesn't help if there was a logic error corrupting every state from the start, and a full rebuild is required.
I'm unsure how redis streams behave with regard to cache eviction, nor am I familiar enough with rabbitmq to comment on it's behavior. It's been 10 years since I used either, and at the time neither were good solutions for a log based system.