Hacker News Reader

Comment by mark-ruwt | Hacker News Reader

If maximum uptime is critical to the business, your infrastructure should be cross-provider.

I've been running three providers as peers (DO, Linode, Vultr) as a one-man shop for years, and I sleep better at night knowing that no one intern can fatfinger code that takes me offline.

View on HN

At that Point wouldn't hosting it yourself running on something like vmware vsphere be a simpler option? At least you would have a nice hardware abstraction and a consistent api to build your tooling on.

I hear you. For me, the abstraction is the Linux distro. Build scripts abstract out creating a clean, secure box before installing any custom software, so regardless of the provider, every machine is exactly the same.

All those are VPS providers, it is hard when you use other services of vendors

Agreed! My take is that you don't have to, you shouldn't.

What kind of business do you run? Can you explain a bit about how your systems in different providers are connected?

Surely! Our business is mainly an API that B-to-B customers consume. The strategy is to create identical "pods" in different cities across different providers on identical distros. Between Vultr, Linode, and DO, that's 20+ cities you could place a pod in the States alone. Each pod has a proxy up front, a database slave, and a pair of app server and cache machines.

Ignoring tweaks for international customers, each proxy is in an A record round-robin with health checks via Route 53. US-based requests get forwarded to one of the pods, and the proxy either handles the request with local servers, or points to servers in another pod if it has servers that are down. If any pod has a power outage or goes down for any reason, Route 53 automatically pulls the entire pod out of the rotation. If an entire provider goes dark, all of the pods get pulled out of the rotation, but all the others keep running.

This is very cool. Where can I learn more about stuff like this, and what are the prerequisites for learning something like this? I have a BSc in CS and understand OSes and programming languages pretty well.

Posts on http://highscalability.com can get a bit hardcore, but there's a lot of great knowledge there.

Thanks a lot for your help!

Your domain registrar can probably still hose you. Where's your DNS hosted?

Very true! I have a script that automatically commits any DNS changes to source control, so if Route 53 bites it I could quickly move somewhere else, but you're right, on the off chance my registrar decides to vanish, there'll be some panicking.

you really can't compare switching between generic vps providers from switching between the big cloud providers that provide lots of more useful services.

It is a very different model, but my take is that you shouldn't be building anything on one provider that you couldn't easily move over to another. Provider lock-in is scary.

Reply on news.ycombinator.com