A Foolish Consistency: Consul at Fly.io

Can't speak for current experience, but we've used it ca early 2016 - late 2017 on ~100 VMs on GCP and it performed flawlessly. No autoscaling though.

you autoscale the consensus system?

They are, and we do have those issues; we address them somewhat towards the end of the post, with São Paolo and Tokyo --- but also with São Paolo and São Paolo: a VM that pops up in GRU has to post to Consul in IAD before other GRU nodes "see" it in Consul. We've addressed that problem by "hinting", sending service discovery updates over NATS that usually beat out Consul, and then reconciling the two event sources.

As I understand it, the reasonable-person thing to do here is to have federated Consul clusters in different regions. We run up against the problem that we're not hosting one application globally, but rather a zillion applications; every node needs to know about every instance of every application. So things like losing the global KV store are problematic for us.

TBH that's sound advice for pretty much every distributed system with a consensus protocol: high latency = slow consensus = slow change propagation.

(HashiCorp Co-Founder)

First, I love this blog post and I tweeted about it, so please don’t interpret any of my feedback here negatively.

> If fly.io found the upper bound on a Consul scale out, what do you think a reasonable threshold looks like for a smaller system?

I don’t know the exact scale of Fly.io’s Consul usage, but I would imagine they’re far, far from the upper bound of Consul scale out. We have documented some exact scale numbers here[1]. And of course these are only the customers we can talk about.

I didn’t read this post as talking about scale limits. Instead, it discusses the tradeoffs of certain Consul deployment patterns, and considers whether their particular usage of Consul is the best way to use a system with the properties that Consul has. And this is a really important question that everyone should be asking about all the software they use! I appreciate Fly sharing their approach.

To answer your second part (what is a reasonable threshold), we have documented recommended hardware requirements and scale limits here: https://learn.hashicorp.com/tutorials/consul/reference-archi... This is the same information we share with our paying customers.

[1]: https://www.hashicorp.com/case-studies

Agreeing with mitchellh's sibling comment. This post doesn't seem to discuss a scaling problem really (there's an aside about an issue with n squared messaging, but it was fixed, so no big deal). The point of the discussion seems to be that consensus isn't really needed for their use case, and it makes things complicated, so maybe get rid of it.

Really, a node doesn't want a consensus on what services are available, it wants to know what services are available to it. Hopefully that's the same globally, but waiting for global consensus means sending more traffic to the wrong place than if you had full mesh announcements (probably not great at scale) or announcements sent to a designated node(s) in each datacenter (possibly gossiped from there to each node, if desired, could be looked up on demand from the monitor node). Of course, that works only if you're ok assuming that a service reachable by one node in a datacenter is reachable by all nodes in a datacenter.

Anyway, you still may want to deal with requests that were sent to the right place, but are received by the wrong place, because the right place changed beyond the observation window for the routing decision. (This is also touched on in the article).

Distributed systems are fun and exciting! And, there was another post today noticing that even a single computer is really a distributed system (and they have been for quite a while)

It's thousands of teams with consistency around the entire world, updated and read in real time as the control plane of a CDN.

I didn't go into detail about Raft because it isn't the point of the article, and because I don't want people checking out because they feel like they need to grok distributed consensus to understand what we're saying. If that means we lose people with strong opinions about Raft, I'll accept the tradeoff.

Also wondered this - Pretty sure it is unless you’re using K3s et al which swap it out for SQLite.

Definitely still etcd in there.

You could deploy your own: https://github.com/google/seesaw | how-to: https://render.com/blog/how-to-build-an-anycast-network

Or have someone else deploy it for you: https://www.vultr.com/docs/configuring-bgp-on-vultr/ | https://netactuate.com/anycast/

All big cloud providers have anycast load balancer offerings. In my experience, Fly's load balancer is wayy more simpler than those to use (because it is the only way to load balance, like with Cloudflare).

You can bring your own AS quite easily at a couple of places such as vultr. I heard TCP works surprisingly well over anycast as routes are more static than one would fear.

Note that in the case of fly you don't directly get traffic to an anycast IP but through a reverse proxy.

I could imagine that they replicate connection states to all pops so that traffic that arrives in the wrong pop can be rerouted.

It's really common for CDNs. Definitely for DNS and almost always for http. Anycast routing.

In the past people were concerned more about route flapping disrupting TCP connections. It seems that people aren't worried about that anymore. Google does it.

There are multiple reasons, some simple and some subtle. We don't totally have the bin-packing problem Nomad is designed to solve. We do have "users have fussy and meticulously specified ideas of how they want their apps to deploy", which is not really what Nomad is designed for; Nomad is designed to decouple your pool of compute from your applications, and to make it low-drama to deploy new applications to that pool of compute. But our users come with some level of drama built-in; it's an impedance mismatch.

I like Nomad a lot. I think it's a better fit for what we're doing than Consul is (they're both great, but I think Consul is the bigger sticking point). You'd get different opinions from different people on our team, but I think Nomad's going to be around for awhile. But there are things we very much want to ship that we can't ship on Nomad.

I would 100% use Nomad again if I had a somewhat complicated app I needed to deploy and couldn't use Fly.io for. (I'd use Consul too, for that matter.)

Per the sibling comment: we are indeed writing a new orchestration system in Go, called `flyd`.

> Rewriting a Rust program in Go is sacrilege, we know, but Go had the Consul libraries we needed, and, long term, that Go server is going to end up baked into our orchestration code, which is already in Go.

This line from the post indicates to me that they're writing it themselves (since they wouldn't be worrying about being baked into orchestration code if it's Consul and Nomad). Still, I'd also be interested in what they're doing.

Believe it or not, pretty much every cloud that runs full stack apps has a similar proxy / service discovery layer. Even if they only let you run in one region.

If you run your app on your own servers with your own load balancers and terminate your own TLS, you avoid a lot of this infrastructure. If you use anyone with a "cloud" load balancer, you get almost exactly what we've described here.

Complexity is a legitimate reason to manage your own server. A single server with a single database and nothing between you and the users avoids a lot of complexity.

I don’t have a take on if it’s a good fit for single region hosting (though the tooling makes that tremendously simple).

I will say that I think the days of single region deployments are coming to an end, even for “the little guys”. Regulation all over the world is demanding your application be region aware and the nature of the internet means your users will come from those jurisdictions.

If anything we need simple web stacks that acknowledge this and hosts (like fly) and data tiers that enable it trivially.

Sorry for the tangent-y rant.

When a VM comes up, we write a service to consul and simultaneously broadcast an identical NATS message. Nodes that are in a happy place get the NATS message, update their state, and then reconcile against Consul when Consul catches up.

NATS has the benefit of being p2p, so when a VM comes up in Sydney, the other Sydney nodes see:

- The nats message in about 5ms

- Time passes (call it 1-10s when things are working well)

- Consul data includes new entry

We also use "hints" to broadcast things that we can afford to never see. Load on VMs is an example. When a VM hits its max concurrency, we broadcast a NATS message so other nodes can take that into account for load balancing.