Cheap Kubernetes Cluster on AWS with Kubeadm
github.com
github.com
One of the things about Kubernetes I like most is the likelihood that it reduces the barrier to entry for "PaaS land" (and resultingly "cloud land") for providers like Hetzner (see OVH's recently announced k8s offering[0]).
Once these "baremetal" providers get in the managed (by k8s) game, I'm certain companies will start to spring up and offer/chip away at cloud provider offerings like S3/RDS/etc -- they'll just connect to your k8s infrastructure, and bring their kubernetes-compatible know-how. This is going to cause prices to plummet, as it will make the tiers of value added services more distinct -- i.e. "a platform to run things" on vs. "really good managed databases".
10 from failures (2 or 3 of which directly affected me),
10 emails stating when planned maintenance of certain resources will occur
1 email to let me know of the spectre and meltdown vulnerabilities.
So far i'm pretty happy customer, and sending business their way and migrating client's sites to their cloud offerings.
I've been using Hetzner for roughly 2 years and am a pretty happy customer.
I can say though that the latency is pretty bad when compared to local options -- their server are pretty geographically far from where I and others who use some of my apps access from, and I get ~200ms of latency that I can't do much about. I've looked a traceroute and it's not their network but rather some points in between.
With these shared core machines, I wonder what the performance reliability guarantees available and how it compares to the dedicated core machines.
If you're hosting a website (or a database backing one) there is going to be a natural kind of ebb and flow as traffic comes in, gets processed, and answered. Since you're not utilizing a large amount of CPU consistently this actually fits really well into the shared core model - that's exactly why they can offer it. Even if a subset of requests take twice as long to fulfill this is usually not even noticeable to the user.
On the other hand if you're routinely running a large ETL process, resizing video or images, or any of dozens of other things I'm sure you can think of that are using a lot of CPU for prolonged periods you're going to notice it.
Specifically I have two situations I've run into: My TeamCity build agent will be unpredictable for longer builds - one might finish in 5 minutes, the next might take 10 - and when I had a Windows Server running as an Amazon Workspace alternative (RDP in, run Visual Studio, etc.) things like building and debugging an app were noticeably slower than if I did it locally or on my dedicated server (even with similar specs).
So if CPU usage isn't normally your bottleneck or you're scaling horizontally and CPU performance isn't as important it's a great option that will save you quite a bit. If your workload is very CPU sensitive you probably shouldn't be using a VM anyway and should look into a more dedicated infrastructure, but obviously there is also a middle ground to be had...
The interesting question comes when people start implementing hardware hypervisors and what is the risk profile there.
Security, at the end of the day, isn't about what is secure and what isn't. If you want to be secure, don't get on the internet. Everything else is a exercise in risk tradeoffs and mitigation.
If I was doing anything with PII, cc#'s or any other data I never want to touch I wouldn't use shared hardware without hard thought on it.
For example, they'll typically be on secret mailing lists and aware of security vulnerabilities weeks before you know about them.
Most of the smaller cloud providers seem to run a similar model, I guess it's worth it unless you need predictability.
kubehost was designed for this purpose on GKE: https://github.com/GoogleContainerTools/kubehost
https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dn...
Not sure why you'd ever want to do this, given that GKE or DO will always be cheaper and AWS's core services aren't all that special, but as a thought experiment it's interesting.
[1] https://www.radishlogic.com/kubernetes/running-minikube-in-a...
But even in the "rich" EU Avergae monthly wages can be only 1,000 USD, making this a tenth of a day's work, and we don't have to go much further afield to see 6 bucks becoming a significant chunk of a workers day.
So thank you, some kids somewhere will be able to afford to develop skills because you started penny pinching.
cheers
I have a couple of silly sites that I've run on App Engine for many years, using its generous free tier. If they were on Digital Ocean I'd have shut them down by now, as I eventually did for a Go search engine that I wasn't using much.
If it's only a hobby project with a small cluster, the Fargate costs could very well be under $10.
https://aws.amazon.com/fargate/pricing/
Assume that you are actually _doing something_ with the cluster, then there will be charges for vCPU and memory. The pricing on this page indicates that if you only need your containers for 10 minutes a day, every day, then you will pay $1.23 for the month at 2GB/1vCPU. That's pretty modest usage at a pretty modest cost.
Sure, you can do it, but... you may need to re-architect your product to take advantage of transient workers. (Now I think we all should do that, but that's another conversation...)
Compared to DigitalOcean's $10/mo (single-worker, managed) cluster, which provides one full-time node with the same specs, that you can slice-and-dice to run as many tasks as you can fit in 2GB of memory and a single vCPU. Now it becomes clearer that Fargate is priced at a premium. If I'm doing the math right, you'll pay $177.12 for that same task space with your Fargate cluster.
If your hobby project needs to run for more than a couple of hours a month, you will usually pay a lot more with Fargate.
> For example, your service uses 5 ECS Tasks, running for 10 minutes (600 seconds) every day for a month (30 days) where each ECS Task uses 1 vCPU and 2GB memory.
5 workers, at 10 minutes each. That's still $35 / $10 = 350% of the cost, and they're not full-time. The point was that Fargate is not priced to be the cheapest option, unless your product is designed specifically with Fargate in mind.
(And ok, not specifically Fargate, really any architecture that permits the workers to come and go, but mostly go.)
Fargate is a tool for a job, and in the suite of Amazon's offerings that includes Lambda, S3, CloudFront, you really can make some cool stuff for cheap, if you pay attention to what kind of resources it needs, and if it _really needs_ those resources, or maybe there's a cheaper offering that is a better fit, or at least that could do the job just as well.
Don't get me wrong! But if I want to take, say, my Ruby on Rails app and host it somewhere without reinventing my whole stack from scratch, I definitely won't use Lambda because any Rails app just won't fit the model very well, and I won't use Fargate because it's [N] integer times as expensive as comparable offerings.
If Fargate was capable of auto-scaling tasks to zero workers during periods of inactivity while remaining mostly responsive, like Lambda "cold start" vs "warm start", then it would be a much easier sell for me.
My take was, that hobby projects probably don't run 24/7, and at some point, you pay while not using them.
Riff is really a Function-as-a-Service library, built on Knative, but Knative can run any arbitrary workloads. Riff will soon have support for Ruby again (I promise, I'm working on it[1]) and Knative has this capability now[2]
So, there's a convergence coming and it's going to make a big difference in this cost/benefit trade-off. You need a cluster to run your Knative workloads on, and that cluster has to be persistent and always available to make the magic trick work.
But say you have a small to medium enterprise that runs 75 custom apps, and any 10-30 of them could be in use at any given time. You'd like to take advantage of office hours and turn things off when they're not in use, but you can't guarantee that nobody is going to need some app, some time after 5pm.
The cluster itself can autoscale up and down to a smaller or larger size, depending on how many balls are in the air. Your apps remain responsive even at nighttime, when your cluster footprint is a tiny fraction of what supports the company's daytime operations.
(By "I'm working on it" I really mean, they've made it as easy as they possibly can for new runtimes to be added with their buildpack v3 framework. I riffed on Ruby back in v0.0.7 and it was possible to reuse the work in v0.1.0, after they ported their stack over to Knative. Now in v0.2.0, the work I did on my Ruby invoker is not wholly reusable, but I'm hoping it will be a pretty smooth transition.)
And you didn't say you're using Ruby, but the point again is that anyone can add support for any language, and it's no big deal. I'm basically nobody here, and I can do it...
So that may sound a little crazy, but keep in mind that there are also K8s "virtual node" solutions in the works or already out there. So binpacking your containers into nodes could soon be a thing of the past, and as clunky as what I'm proposing sounds, it may not always be that way.
Sure, AWS could do it with Fargate too, and it might turn out to be just as good, but right now that's speculative. This is pretty much stuff that is all out there now. It's just parts that need to be put together.
[1]: https://github.com/projectriff/riff/issues/1093
[2]: https://github.com/knative/docs/tree/master/serving/samples/...
It’s a bit more work though, which is the trade-off, but since the last few versions kubeadm makes it really easy to spin up clusters.
Personally I opt for complete flexibility.
Specifically kops (by default) does not enable authentication on the kubelet, meaning any attacker who gets access to one container in your cluster is very likely to be able to compromise the whole thing.
I have a comment somewhere in my history that explains in a bit more detail: https://news.ycombinator.com/item?id=18660503
that's it... hostPort mode, hostNetwork, and enable in DaemonSet mode. Now all of your nodes are load balancers for ingress, and you don't need any ELBs. This is not a recommended configuration because something has to point DNS at your nodes, and the nodes are really not designed to be permanent.
If you autoscale, or scale your cluster manually, your DNS needs to be updated to keep up with that. You may be able to find a way to automate that, but DNS has limitations related to TTL, such that if you are doing this too frequently, visitors to your cluster are likely to experience issues.
But if your nodes never come and go, this is a pretty good way to run a cluster and keep it on the cheap. If the traffic you want to balance is not HTTP then ingress won't help you (for now?), but the configuration for HAproxy will be similar.