Hacker News Reader

Comment by capmorganbih | Hacker News Reader

Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts

View on HN

Can you explain what your issue is with Kafka? What makes it not reliable enough?

Kafka is good. The problem is we don’t have a dedicated person to manage it so sometimes we have kafka outages

There’s a whole conference in Vegas next week that want to sell you something.

So basically, you want to build a write-ahead-log before writing data to Kafka, and I think you're underestimating the effort to implement a WAL.

If you don't have a person that can manage Kafka, you almost definitely don't have the person to maintain a WAL.

Just a suggestion, you can learn about automq (a message queue that has rewritten the storage layer based on Kafka), which has done a lot in automatic recovery and self-balancing, maybe it will help you. If you don't need it, you can ignore this comment.

You could but its not purpose built for that. You'd probably be happier using some other memory mapped file format for that.

Could you please suggest java library for this if you know one?

https://github.com/OpenHFT/Chronicle-Queue?tab=readme-ov-fil...

crazy idea, but SQLite

Write your data to a MognoDB database. Then use the Kafka Mongo connector to pull data from MongoDB to Kafka.

Reply on news.ycombinator.com