Walrus – high performance distributed message streaming engine (Rust)

Hey HN! recently I've been working on Walrus, a distributed message streaming engine that combines high-performance log storage with Raft-based coordination.

What makes it different:

- Segment-based sharding with automatic load balancing – Topics split into ~1M entry segments, leadership rotates round-robin on rollover. No manual partition management. - Lease-based write fencing – Only the segment leader can write. Leases sync from Raft metadata every 100ms, preventing split-brain without coordination on the data path. - Sealed segment reads – Old segments stay on the original leader after rollover. No data movement, reads scale with replicas. - Simple TCP protocol – Connect to any node, auto-forwarding to the right leader. Commands are just PUT topic payload and GET topic.

Performance: The underlying storage engine hits ~1.2M writes/sec (unsynced) and ~5K writes/sec (fsynced), competitive with Kafka and faster than RocksDB in benchmarks.

Correctness: Includes a TLA+ spec verified with TLC covering write fencing, rollover mechanics, and cursor advancement.

The repo has both the distributed system (distributed-walrus/) and the standalone storage engine library (walrus-rust on crates.io).

Would love feedback on the architecture, especially around the lease synchronization approach and sealed segment design!