Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”
One reason why a "logs are all you need" solution may fail: untrusted-log-as-injection[1].
Check those SBOM, and don't forget to include their CICD pipelines[2].
[1] https://news.ycombinator.com/item?id=48315440
[2] https://github.com/jqwik-team/jqwik/issues/708#issuecomment-...
Different workflows should probably go in different buckets or "topics" for clarity. Since it's distributed, the system must guarantee that the log items are stored in the same ordering ("offsets") among the nodes.
Not a bad way to do things.
A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.
Google "Log-structured merge trees" if you want an interesting read.
"Sockets are all you need for durable workflows" and then finally "Kernel primitives are all you need for durable workflows."
But seriously, part of being a professional is using the right tool for the job.
I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects. It doesn't seem like there are alternatives at the high end but instead the question is when other technologies (that are easier to manage, require less compute, etc.) max out.
Yes
> I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects.
It's not gonna win or die on merit.
If I were to sit here and propose that you mutate bank accounts in-place (which I'm more or less doing by analogy right now over on https://news.ycombinator.com/item?id=48339103), and you just need safe enough locking technology to do so, I'd be immediately and rightfully shouted down that no one does that and you write down the money transfers and derive the final balances from that.
Manage any other kind of state, however, by appending state-changes and rolling them into a derived state, and then you're chasing fads and doing resume-driven development. So I'm skeptical about the future of this way of doing things.