It looks like you can get away with it and let your service(s) have downtime for more than 5 hours if little to no-one is paying for your service or there are almost no users using it.
Their stock price being 66% down since IPO, also suggests that not only Hashicorp's valuation was extremely inflated, but it also seems that they are in (actual) decline.
Their tech seemed good initially so i thought everyone going to need the multi cloud stuff.
But after using it and encountering how brittle it was, and easily our nomad cluster would go down without any way to recover, even though the major point of a cluster is high availability, I can see that the tech is just very brittle and the value of the stock doesn’t surprise me.
I’m relieved that I followed my instinct and chose not to deploy it in production even after we invested so much money in setting it up.
I really hope I’m wrong about the stock and the tech.
I like the founders and they’re clearly very smart but the company has grown too fast and likely let quality go down.
https://web.archive.org/web/20230614095444/https://twitter.c...
for my curiosity, was in Nomad or Consul that fell over? My experience with etcd leads me to suspect it was actually a consul fire, but since I (thankfully) have never run Nomad I don't know first hand about its dragons
Essentially, about once a week the raft pings between the nodes would result in no response from the primary, so then the cluster would assume it lost the node and try to hold an election to pick the new leader and get stuck in a loop cuz it kept indefinitely trying to ask the leader for it’s vote.
I thought, surely this software isn’t that stupid. But it was.
The recovery documentation was to hand-generate a peers.json file. Seriously? In 2013 when you have a million ways to do auto discovery? Including in your own software? it couldn’t just auto heal?
I managed a MongoDB cluster ten years ago and never had a single issue like this. I could routinely take nodes down and bring them up and the cluster healed perfectly.