What did you find most difficult about the implementation?
It took a while to get the building blocks together: the log file, which can be traversed, read, and rolled back/truncated (use a kv store for this, trust me), and the transport mechanism to deal with sending complex objects and handling errors and timeouts.
Tracking terms, states (leader, follower, candidate), rejecting out of date terms, voting, elections -- it's all very complex. You'll have to build some really, really good multi-threaded tests that simulate out-of-order and dropped messages, then really pound away at the system. And good logging to find that odd corner case that keeps coming up and throwing mysterious messages.
I haven't even tried membership changes yet.
Order is important here - is this inherent in your key?
I did two implementations, one with plain text files and one with a kv store. For plain text files, I had to maintain file offsets of the records so I could traverse backward and truncate records that had to be replaced, and then break the files into segments so I could retire segments that are no longer needed. It had all kinds of little bugs.
Then I reimplemented using a kv store, and it took a tiny fraction of the time. And surprisingly had better performance.
Just create a good interface at the start, use the kv store underneath, and then later if you don't want the dependency swap it out for the more complicated code.
Do you host diep.io on cloudflare, heroku, linode etc.? or do you have own servers?
Because I'm interested about developing something.