Kubernetes 1.20
kubernetes.io
kubernetes.io
I mostly consult with small startups, so I've seen the damage these ideas can do. Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed.
If anyone is interested in a case study of excess devops development, please see "High Availability is not compatible with an MVP, because an MVP is about fast iteration"
http://www.smashcompany.com/technology/high-availability-is-...
But the design (functional architecture, APIs, etc.) of Kubernetes makes good sense for small scale as well.
There's nothing fundamentally stopping someone from creating an "embedded Kubernetes" that exposes all the same resource APIs, but which is just a management interface for e.g. a JVM, or an Erlang node, where all those APIs are just controlling in-memory abstractions existing inside one userland process. Presenting itself as a "cluster" with one fixed node pool, with one fixed node, with one fixed type of container runtime which is just "spawn an actor-thread in this process."
I don't disagree that people are pursuing the enterprise k8s stuff when they aren't at the right scale to need it — but I think it's important to realize that they're often doing that because they see the design of k8s as fundamentally sound, providing sensible abstractions and management interfaces that neither plain OS process APIs, nor Docker's APIs, nor IaaS-level control-plane APIs have great equivalents for. It's that design they're after; the HA stuff is just a "nice-to-have" that they end up adding on, because all the guides they read (that are written for enterprise integrators) insist it's a great idea.
(Tangent: if Docker Inc. would get off their collective butts and do Docker Swarm over again, but this time with k8s-compatible APIs, I'm sure it'd become the most-used "Kubernetes" overnight. Most devs just want a regular Docker host, that they can `kubectl apply -f` against.)
not sure, but with 1$ million on budget, you can run a pretty pretty big gke cluster. btw. we run on e2-standard-2 and pay ~500 per month (would be a little bit more without commited usage). 6 nodes + 2 managed cloud sql databases+several load balancers.
really no idea how one can pay a million even in a year. that would not be a small startup...
you can probably have like 9x e2-standard-32 (32vcpu+128gb) + postgres ha 16vcpu/256gb mem/500gb ssd for like 10,867.33 per month without traffic (which might be another 5000) so with traffic you would only pay 190407.96 per year?!
Edit: even without gke, k3s is way simplier than most "deployment scripts" or whatever. it's always a matter if you want to have ops people or if gke's self managing is enough.
Edit2: so many lols at the blog post. I mean who the fuck spends 7000 at a colo for 40.0000 users. my macbook could probably serve that. also "and then script those older Virtual Machines with some mix of Ansible and Terraform (assuming you want to be in the cloud). " is a fucking terrible advice, been there, done that and it soon is a clusterfuck of stuff cobbeled together.
> I’ve a good friend at a startup who found a co-lo data center in a small college town in Virginia, where they were able to build out a massive MongoDB database, sharded across 4 massive servers, each server holding 256 gigabytes of RAM, 1 terabyte of RAM in total. They also had a Cassandra database and several servers for their front-end. They were paying $7,000 a month for this.
Like, if you're paying AWS to host MongoDB on four EC2 machines with 256gb of RAM each and no autoscaling/load balancing, you can spend a lot of money. But... why would you do that? Porting terrible architectures to cloud VMs with no rearchitecting costs a boatload, but... don't do that?
You misread the article. The startup with the $1 million devops ended up with 40,000 users. The startup with the co-lo was a different startup, which in fact had a very large user base.
I might be biased, because my first commercial use of kubernetes was to reduce infrastructure spend and optimize the hell of what we have. From my PoV, the "big company" narrative is just what happens when the big company notices the cost of spinning separate VMs for everything.
However Kubernetes does have a lot to offer, especially with the easy of hosted Kubernetes solutions these days. Helm makes installing tools and dependencies you may need almost trivial.
One of the nice things about Kubernetes is that it is relatively easy to scale from a small deployment to a larger HA deployment when (and if) you need to.
> Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed.
This sounds like more of a management, requirements, and overbuilding problem, rather than a Kubernetes problem. Even without Kubernetes, if they wanted an HA devops set up, they still would have built one. Honestly without Kubernetes it could have ended up more expensive.
Upgrades that get in bad states, ya, I've had to deal with it before, and it definitely sucks. The "atomic" goal that Helm 3 tries for is better in my opinion, but still has rough edges.
What have you been using as an alternative that doesn't require you to write your own manifests for every dependency?
We found a lot of success also with Kustomize. I did not enjoy the ksonnet detour as much - it was nice to override a prototype instead of having to screw with yaml templating, but jsonnet is not a good language.
Helm 3 had some designs on a lua API, which I thought was a promising idea. I've been out of the loop since moving on to a company that is back in EC2-land (which really just means that another team manages the thing that approximates Kubernetes...).
I agree with you about an addon manager, but then something has to be the source of truth for that, and the "KubeApps" type apps were all using Helm under the hood anyways.
I played around with Kustomize for a bit at one point, but honestly had a bit of difficulty understanding how it loaded and applied the different "layers". I've been looking more recently at it as a "make some quick customizations to this rendered helm chart" tool as of late.
This seems to be a big problem with container orchestrators these days. I was deploying something to ECS and while 90% of the config could be accomplished with environment variables, there are still a few cases where it was expected that I would provide files. On a local machine with Docker and some config mgmt like Chef or Ansible, no problem - template out those files and volume mount them in. In ECS? No easy to way to do it without baking your own container or using EFS, which for one config option seemed like major fuckin' overkill. Ended up bastardizing the entrypoint with some judicious base64'd bullshit and got it going without either, but christ, how fucking hard would it be to just provide a native mechanism for templating config files out?
For applications I built myself, I would often just roll the possible configs for us as files in the container, and you pick one with a single env var (realistically this was like "dev", "beta", "prod"). Not great, but meant I could avoid the clusterfuck that is specifying all my AWS configuration in env vars on a per deploy basis.
ConfigMaps could sort of help with this, but I found what you found - having to base64 a config file as a field in a ConfigMap and make sure it gets mounted correctly just seems pretty baroque for configuring something with a file. And at the time, ConfigMap changes didn't even cause a redeploy, so your pods would for their lifetime keep on using a stale config!
I'm not sure how familiar you are with AWS, EKS, Kubernetes, or Docker in general and I don't know of a guide that encompasses all these technologies.
My best advice would be:
1) Get a little toy Docker container running locally with a http server that responds with "Hello world"
2) Try and deploy it to as a pod to a local Kubernetes cluster (Docker desktop has one built in or you can try deploying Minikube if you're feeling adventurous)
3) Add an ingress and service to your cluster, so you can access your server from a browser
4) Expand your cluster by installing something like Postgres on it and have your toy app query it and print out a query result in response to requests
5) Try passing your toy app the password to login to Postgres as a Kubernetes Secret via an environmental variable
6) Start exploring EKS specific deployment documentation such as the links below.
7) Start learning about helm
https://docs.aws.amazon.com/eks/latest/userguide/create-clus...
https://docs.aws.amazon.com/eks/latest/userguide/sample-depl...
https://learn.hashicorp.com/tutorials/terraform/eks (Once your comfortable with everything else)
K8s may not be the best option for every project, but my original point was that choosing it ought not blow out the budget or delivery window of a project. Maybe, you might shave a week or two off a greenfield project by going with something like Elastic Beanstalk, but in the scheme of things that's a rounding error.
To put things in perspective, a typical deployment for a small app requires 3-4 short configuration files specific to K8s plus your Docker container.
Furthermore, you'd generally still use the cloud managed services like Elastic Cache and RDS when developing K8s applications. I was merely using a Postgres as an example in my mini-not-quite-a-tutorial before.
So yeah, if Elastic Beanstalk is working for you and you think the tradeoff between flexibility and simplicity is right for your project, then keep doing what you're doing.
Or do you mean a small team should not use unmanaged k8s?
My company is in a place where I think k8s would be really beneficial. We are at the 40k VMs running 1000 different types of apps. We suffer from tremendous wasted resources as a result.
The company started before k8s (and even docker) were things. So getting us there seems like a real daunting (yet worth it) task.
So, we are left with sort of a chicken/egg problem. It would have been unreasonable to start with k8s, yet now it is a huge cost to switch to it (even though it will save us millions).
Further, we've got a fairly deep data processing pipeline. It's not unusual for us to see high loads on early parts of the system 1 hour and 1 hour later seeing high loads on the later parts of the system.
Unfortunately, we don't currently have a way to scale out and reassign resources. That means we've built out for our peaks even though the majority of the time that hardware is sleeping.
We could spend the time building out a scaling system (right now, our resources are all on prem). However, why do that when we can instead take advantage of k8s already built in scaling capabilities?
So, technically not k8s that will be saving us money, but rather implementing autoscaling. However, the solution with k8s is really nice.
Tooling has come along way as well.
But the learning curve is knarly and I picked up a few tricks from companies I worked at.
Moving from AWS to Google Cloud to bare metal can save massive amount of money.
Knowing what I know now, I think if you are successful something like k8s is required to scale and you are running non trivial apps. For trivial stuff I'd go serverless plus some kind of managed app platform as much as possible.
For non trivial, there are stripped down k8s distributions out there that would work and let you work in portable containers. I'd think pretty hard about something like Rancher to get started, and 'graduate' to GKE / EKS / AKS when you hit scale with at least 100's of VM's. You need to keep it in containers to plausibly make the switch though and not require a rewrite later. I think Rancher deploys on all the major and smaller cloud vendors, is small enough to run on your desktop, still do development, and adheres to the API standards you'll care about later.
Have ansible fire up a docker container on your host.
That's it, done. You don't need Kubernetes.
Using EC2 segregates everything nicely out of the box. You don't need to be concerned about resource contention or figuring out how to size instances/pods. A container going nuts and filling up your disk won't cause a bunch of other services to experience errors.
While convenient for deploys, the biggest issue with Kubernetes is that you introduce an entire new layer of technology that you need expertise in.
If you're running your own EC2 instances anyway (which seems likely, since spot pricing is amazing compared to managed stuff) you now have to know how autoscaling works, plus a VPC, plus Kubernetes. The way Kubernetes interacts with cloud providers (at least AWS) adds a ton of complexity.
Also, if you have a decent build process for AMIs you can insulate yourself from container registry outages by baking the images into an AMI. There's something really powerful about this approach. If you build an AMI properly, that thing will run _forever_ with a minimal set of external service dependencies.
I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
This is borderline trivial. If you build an AMI, you update a launch template with the new AMI and use instance refresh (https://aws.amazon.com/blogs/compute/introducing-instance-re...) to replace instances. Rolling back is the same thing, you just set it back to the previous AMI. Instance refresh took way too long to become a thing, but it was not even very difficult before that. List instances in your autoscaling group, mark them unhealthy one by one.
Now, you'll probably want a couple other safety checks in your deployment process to make sure it all goes smoothly. But you're realistically looking at what, a couple of days of work to have something built to your specifications that can deploy any of your services without introducing the added complexity of Kubernetes?
> I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
The harm is not that the deployments are hard. In fact, they're quite nice. The problem is that small teams don't have the bandwidth to keep up with the relentless release cadence. They may not have the expertise to get the upgrades done confidently.
There is also the fact that, when something does go wrong, you've got a whole lot of complex Kube stuff to churn through.
None of that applies to managed Kube, but then you're paying extra for Kube every month. By contrast, you could have just paid extra one time for a relatively simple deployment script in CI.
About the only real issue I encountered was with major API change involving ingresses, when the cluster would continue running, but we had to update our manifests that were hamfistedly done back in 1.4 days.
What you mean is probably without vanilla k8s
Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with.
The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on).
Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting.
Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers.
That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
AWS EKS seems to support major versions six or seven months later. Support for 1.17 was announced 2020-07-13, 1.18 was announced 2020-10-13. I'm guessing we can plan for 1.20 support on EKS around June of 2021. No judgement, just an observation.
When you build something, you need it to work until _you_ decide to futz with it again.
You can even turn your devbox into k8s via minikube.
It’s a great abstraction