Do I Need Kubernetes?
mbird.biz
mbird.biz
I think this is the key take away for many startups. Get it so you:
1. Have a single-command way to bring up your entire backend
2. That command can run on your dev machine
3. You document all of the settings for your containers and deps
The only thing you must be critical of is bashing people over the head to make everything you run in your company run in this environment you've created. No special deployments. Establish 1 method with 1 tool and go from there.
Every company I've worked at I've brought a single command `docker-compose up -d` workflow and people were amazed by the productivity gains. No more remembering ports (container_name.local.company.com), no more chasing down configuration files and examples, no more talking to shared DBs, etc.
1. S3/Azure Blob Store
2. DocumentDB/CosmoDB/Cloud Atlas
3. Aurora/A for PG DB
4. ElastiCache/some complex name in azure
If there was ever a case that I needed to ever use a real - full vendor lockin - solution I would hide it behind an API of my own and configure that API to store data locally. For example, the Azure Blob Store local container is completely broken for any sensible usage and I've been on an github issue thread for ~3 months waiting for them to resolve the issue (you're forced to link your netns to that container to talk to it as the application clients only works over 127.0.0.1). So essentially I have the following settings:
- SERVICE_STORAGE_DRIVER=azure/fs
- SERVICE_FS_LOCATION=/data
- SERVICE_AZURE_CONTAINER=....
- SERVICE_AZURE_....=....
NixOS, however, I will not recommend as your dev environment. Use a regular distro and use NixOS for your containers.
I've also done deploys just using managed services like ECS/Fargate or Heroku where we just build an artifact and push it to the host with a script. A stack that's a load balancer, stateless app server then a DB and/or file store can be provisioned manually once then not really worried about for a long time.
Ops gives devs "disposable environments" to do their testing in. Basically, GitHub + Jenkins + Terraform + AWS. I've used this to stand up real infrastructure every time a PR is opened, and pipelines run against the infra. Code and infra match each other because they both exist in the same branch/repo (monorepo; you can of course do multi-repo, just takes more coordination). It's all destroyed as soon as the PR is closed, but you can also keep it up to do dev work against. You can also keep one copy of the latest master branch up at all times (the "dev" or "cert" or "test" infrastructure) as a shared test environment. Downside: you have to have access to the network (which is in AWS, so that's not so difficult). Upside: the dev environment always mimics production, devs don't need to do anything to stand it up.
Option 2: A bunch of mocks, and scripts to stand up stuff in Docker, using docker-compose or something else. This becomes a bit of a problem to manage with lots of devs, though, and docker-compose is not a good model for production deployments.
We used to be able clone 400 VMs in 1 hour, sanitize all customer data and refresh a full dev environment from m prod in an hour. TBs of data.
But let's all stand around and claim k8s is not your flavor of the day solution to a very old problem, but one that caries a huge amount of complexity and so much network effry that it scares the pants off most people.
Yeah, docker compose is great for running distributed apps locally when everything “just works”. The problem is that it also impedes development when the compose setup isn’t fully optimized.
I work on dockerized rails apps, and whenever a new dependency is added, it can take 15 minutes to rebuild the image, which completely breaks my flow. Docker also creates a ton of data bloat locally, requiring me to run docker system prune pretty regularly. Setting up a dependency cache isn’t completely straightforward either.
I still think docker compose is worth it, but it’s a deceptively complex beast to get right, and most companies I’ve worked for don’t take the time to get it absolutely right. There’s definitely a cost to using it that people tend to overlook.
Long term I'm planning on moving these builds to bazel and taking advantage of incremental builds for my entire monorepo. I'm in the final stages and just validating some stuff is in place before I do that. In the short term I've mitigated this by adding in heavy unit test coverage. Backend devs can basically run the unit tests for most services and are confident the service is working as expected and also enforces any/all security/auth checks. When developing I don't even start up the containers. I boot them up once at the end just to check my work.
Then just have a different Docker-compose config for prod
1. My applications have no dynamic "linking"
2. The containers are distroless
3. Services can be built independently
4. I run the same exact sha in dev/staging/prod
The better solution for me is looking into a build system that can support my different language and tooling choices which happens to be things like Blaze (Bazel/Pants/Buck/Please).
You could try making your own base image that installs your gems. Eg, copy your Dockerfile to Dockerfile.base. Remove everything after the bundle install, and build to say app-base:latest.
Then change the FROM in your Dockerfile to app-base, and remove any OS package installs. Keep the bundle install; it will use the layer from app-base if unchanged, or do an incremental install if not.
This will reduce the amount of data bloat too.
You could try using guard-bundler or something similar to further streamline the process when adding gems
Personally, I found having multiple intermediate dockerfiles more confusing than helpful, so having an easy to follow, centralised build made life a lot easier for me, especially if you produce multiple images that are pretty similar at the base.
Just remember to have a process to rebuild your base image at least once a month so you pick up latest security fixes etc.
I assume this is due to docker downloading the related image and running the container.
If this is the case, isn't this orders of magnitude faster than adding the dependency manually. And aren't the gains are even more significant when we factor the fact that most of the cases dev and prod environments are identical? Or am I missing something?
Yes
> If this is the case, isn't this orders of magnitude faster than adding the dependency manually.
No. When developing a rails server running locally, you just need to install the new dependency via bundle install (usually takes just a few seconds) and then your server is ready to run again. With docker, the image needs to be entirely rebuilt, which is a much slower process, even with good caching. The difference is that docker treats the build as ephemeral and willing to be thrown away and rebuilt, as opposed to running directly, which simply augments the existing environment. It’s a much, much slower process to rebuild via docker.
It's worth noting that bundle uses a Gemfile, so it's not installing each Gem as a seperate step.
I wasn’t involved in the setup but adding a new gem or upgrading one is very quick. We use a mounted volume to hold the gems as seen here: https://github.com/concord-consortium/rigse/blob/master/rail...
1. The syntax is very complex
2. It's just a wrapper around shell command execution
3. There's no single standard everyone fully implements
How is this any more complex than any programming language? Take javascript. Do I want to use =, ==, or ===?
> I cannot test if my Makefile will run on a developer's mac if I don't have a mac myself and even then it's a crap shoot (do they have homebrew? is their PATH correct? etc).
Containers suffer from these kinds of problems as well. For example, if your ip tables are not set up just so, you get no network access from inside your container.
Or you can just make everyone wrap their build tools in make.
Make is not a "wrapper" it's a "translucent layer on top of" since it doesn't hide the deps on the underlying tools.
When I do `docker build -t ... something` or `bazel build //something` I don't need to care if it's node, C++, Java, etc. I just need to know that it is a thing that I want to build. Make cannot deliver that to you as you need all of your system toolchains installed and you need to make sure your makefile works on every OS/system config.
For me, that's kinda the point. If (when!) the underlying layers break, I can open up the Makefile and see exactly what was called, without too much abstraction. It's not there to provide complex build options, it's to provide aliases to the commonly used commands in a platform agnostic way.
Most of my make commands are simply aliases for one or more commands, which are often something like `docker run --rm -it project_runner_image some_command`
There's some real problems with Makefiles in the form of tabs as the prefix, running everything as a shell, weird platform specific versions, etc - but I've yet to see anything else which is as small, as universally available, with such a low barrier to entry. Most of the other options I've seen so far (and I'll be the first to admit I haven't looked very hard) seemed to either be overly complex for simple dev setups, or bound to a particular language.
I agree that blaze is the better option, but all the benefits it provides comes with a greater up-front cost, where its not necessarily possible to drop-in on a project and run with. Also notable is that blaze is inspired by and is a replacement for make. So in a lot of ways, make is a poor mans blaze now.
Just learn it then? Running make commands is trivial. Writing basic make targets is trivial too.
[1] Yes, our Makefiles sometimes call docker/npm/ant/maven/gradle/cargo under the hood. Its been worth it, in my experience. One syntax on all projects to set up the workspace, refresh dependencies, deploy to dev env, clean, etc.
Actually I find most answers either in official documentation or on StackExchange sites. And I much prefer digging in mailing lists over digging in GitHub issues. You access both over web anyway.
1. sets up automatic port forwards to my host system
2. handles ingresses by binding port :80 on my system
3. provisions PVs for you when you create PVCs and respects retention policies
4. Allows you to select from a monitoring stack and only require that you supply configurations for dashboards/alerts that can be 1:1 applicable to production so devs can debug dashboards/alerts locally.
`k3d` is targeting itself as being Rancher's kubernetes implementation so if you buy into all of the Rancher tooling it'd be a great option.
The command is supposed to be: 'k3d image import --cluster myCluster image:tag', but it fails with "Failed to start exec process in node 'k3d-myCluster-tools'"
On kind it works fine with 'kind load docker-image --name myCluster image:tag'.
I'm using WSL2.
This means that you need to build your app using Docker, and then import it to k3d/kind environment, thus needing local registry (not enabled by default), which means that images are duplicated.
We do a lot of integration with third parties which is essentially pull down some data and write it into a microservice responsible for keeping that data. Each of these integrations is owned by a separate engineer because it takes up a lot of time to deal with integration calls, planning, etc and we need to be language flexible (some people only give you a .dll or .jar, no choices).
I'm assuming people building these tools don't use a local development workflow because: their systems are too big to run on a single machine (Googlers) or don't work on large projects (open source side projects).
The clostest thing to what you and I are looking for, though, is skaffold and tilt.dev.
That said, a self-contained system in the context of a container orchestration framework, docker-compose, kubernetes, etc, facilitates an number of benefits, not the least of which is the ease of scaling engineering staff.
For example, in typical organizations that claim to have modern development practices, there are static dev, test, and staging environments for a given service. Obviously these are subject to config skew, etc. and must be time-shared between folks. When it's a simple task to create an environment per engineer for each purpose or any other, that can accelerate development tremendously.
There are other benefits around resilience in production, but that's another topic entirely.
We've got multiple apps deployed various ways, everything from pseudo-automated CI/CD (auto build but manual deploy) to fully manual deployment directly out of our IDE. Nothing set up within Docker containers yet except for a few proofs of concept.
Another approach, from a previous startup, who mainly/only used PHP was to volume mount the code you were editing into the containers that were running and to expose an xdebug port for our IDEs to connect to. This was very hacky and error prone but when it worked it worked.
Something that I plan to do in the future is to setup some incremental build system that can build all of my software and use that to build my containers. Whenever these outputs change I'll have them redeploy the container in docker-compose or minikube. This way I get very fast builds, very small containers, and much fewer hacks.
Most of the issues seem to be with Postgres DB connections, but it makes starting up our backend painfully slow...
Edit: Some tuning options also exist - https://docs.docker.com/docker-for-mac/osxfs-caching/
HN uses * for formatting so it's been replaced with {STAR}
I good reason not to use kubernetes though is if you know your app is probably never going to scale, or if it's the kind of thing that can scale very well on a single machine, or if it's not primarily based on http communication. (I wouldn't write a real time game server in kubernetes, for instance, because I doubt it'd really help with a primarily UDP workload that is likely going to be attached to one server)
And in the case of Fargate, Heroku etc you're paying significantly more than if you had made use of Spot instances or shopped around for a cheaper vendor.
So I'll worry about that price when I get to at least 10 servers
But you can just use cloud providers for their hardware and not needlessly tie yourself to their software. For example you can use a managed K8s solution like EKS but then have all of your monitoring, logging, databases etc all be self-hosted.
And it's not just about cost but also about being able to take advantage of other cloud provider's unique strengths or being truly resilient to outages.
If you’re limiting yourself to using only easily portable services, you’re not going to be taking advantage of any cloud provider’s unique strengths anyway. So you might as well just pick one.
One thing that I think is non-obvious: there are easy ways to set up a kubernetes cluster, it's just that there's so many options out there that the easier solutions tend to get lost in the noise. If I had to manage kubernetes nodes manually I'd probably never do it, but setting up something like AKS/EKS/GKS -- they all have their own issues, but they're pretty easy to get started with.
Serious question, as I don't have any experience of those, can you scale based on a custom metric? It's probably the one feature of k8s I appreciate the most.
None of this means I have any answers, I’m afraid.
Suggesting that you can only scale with Kubernetes just isn't true. Our tech stack originally ran on AWS in multi-regions. Then we moved to GCP. The architecture was the same for both:
- public facing Load Balancer
- instance groups attached to the LB
- add health checks so unresponsive instances get killed
- add scaling rules based on whatever metric you want (AWS's offering was far better than GCP's)
- choose your instance type(s).
And for the record, we handle ~15 billion HTTP requests per day (traffic is seasonal). Our busiest data centre is us-east, which has a max of 50 instances (16 vCPU, 31GB RAM), and we rarely need them all. You can also mix spot and non-spot instances too, so we know we have a least one or two machines that GCP won't kill.
Rolling out updates involves building a debian package, building a new VM image, and then clicking a 'deploy' button (GCP makes this easier than AWS).
It all started with Ruby. Ruby's syntactic sugar inspired the "syntactic sugar" of tooling, primarily Bundler and Rspec. Tooling, for what felt like the first time, became a first class citizen. Ruby's tooling made Heroku possible: ie, reproducible builds across; dev, testing, staging and production environments. Heroku's success was based on the primitives of the Twelve-Factor App[1]. The 12 factors (and therefore Heroku) were fundamentally designed around the already-old lightweight virtualisation technology of LXC. The success of Heroku paved the way for Docker. The success of Docker created the world in which Kubernetes makes sense.
To be blunt: if you don't understand the relevance of Kubernetes, or whether it's relevant to you, you don't understand the benefits of the 12 factors in their broadest sense. The 12 factors are much, much more than just "How To Deploy On Heroku".
Copypasting the 12 factors:
I. Codebase
One codebase tracked in revision control, many deploys
II. Dependencies
Explicitly declare and isolate dependencies
III. Config
Store config in the environment
IV. Backing services
Treat backing services as attached resources
V. Build, release, run
Strictly separate build and run stages
VI. Processes
Execute the app as one or more stateless processes
VII. Port binding
Export services via port binding
VIII. Concurrency
Scale out via the process model
IX. Disposability
Maximize robustness with fast startup and graceful shutdown
X. Dev/prod parity
Keep development, staging, and production as similar as possible
XI. Logs
Treat logs as event streams
XII. Admin processes
Run admin/management tasks as one-off processes
> Store config in the environment
How do you pass hierarchical config? In JSON, YAML, TOML, etc., it's easy to group env vars, but how do you do that with env vars? LOGGING__HANDLER__FORMAT, LOGGING__HANDLER__ARGS__ARG1, LOGGING__HANDLER__ARGS__ARG2 ? If so, that looks positively awful. If the solution is passing a YAML or JSON string in the env variable, that sounds even worse than having a config file that the app reads.
>> it’s easy to mistakenly check in a config file to the repo
Add it to the ignore file. Someone would really have to force commit it to get into version control. Plus, even if it's stored in env vars, you're going to commit the values somewhere be it your Ansible secrets, SaltStack yml, Chef or Puppet repo, whatever.
To be fair, that's the only point I contend with. The rest is very reasonable.
Spring Boot approaches that problem via `SPRING_APPLICATION_JSON` (and likely `SPRING_APPLICATION_YAML` but I haven't personally tried it)
containers:
- name: web
image: whatever
env:
- name: SPRING_APPLICATION_JSON
# language=json
value: |
{"spring": {"logging": {"level": "debug"}}}The fact that there is public interest in these things _maybe_ you can attribute in some way to ruby but not the existence of them IMO.
If you don't know it, then learn it first. Then you can properly evaluate whether it suits your requirements or not.
There's huge value in choosing something that your team already knows + is familiar with
I'd totally advocate for learning something new, but solving a business problem probably isn't the best time for this :)
I think the article gave a great overview though, and is perhaps a good way to decide what to learn next
"Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."
Well, there's a kubernetes version, which is that any sufficiently advanced deployment system will ultimately be half assed version of kubernetes.
...back in 2015 ;)
Now I'm completely in favor of hosting anything you can cram into kubernetes in kubernetes, even though kubernetes is more complex than most other infra tools, most of the time there is only one way to do things(configmaps for config, PV allocations for storage etc..) So if you understand kubernetes, it's easier to get the larger picture about the infrastructure even if you know nothing about the application stack.
[1] https://github.com/cloudfoundry-incubator/kubecf [2] https://github.com/cloudfoundry/cf-for-k8s
The scaling, and potential savings is a cloud feature, not a feature of Kubernetes.
In my experience hosting services in VMs on-prem I was able to achieve roughly 30% efficiency across 3k instances and hundreds of nodes. In my experience hosting microservices on k8s, I was able to achieve 80% efficiency across hundreds of nodes.
Both were the result of a great deal of work to optimize efficiency. In this case, I use the word "efficiency" to refer to a blend of CPU and memory utilization.
This only relates to hardware cost - tooling impact can provide cost improvements if it enhances productivity (not always clear for k8s case)
If I would have a cloud hosted k8s cluster and don't configure any auto scaling, the situation would be equivalent to me buying x number of physical machines and hosting them as worker nodes on prem. No ability to scale to "bursty traffic".
If you have a relatively simple deployment structure (one big application with 10-20 services), maybe you don't need to add the docker skill to what you're already doing.
If you're in the process of rebuilding your applications and you've decided cloud native is the way to go, then containers are pointless.
I'd argue if you don't have the skills and you don't want to pay someone to do this work full-time, you might put your developers in a position to fail.
Based on my 30+ years of building software, I see containers as a dead end. It may help you out in the short term, prove to your CTO that you're using "well-known" technology, but in the end, cloud native is going to replace everything. And before you say "but we can't be in the cloud," then you should know that cloud-native development like Lambdas can be done on-prem as well.
I'm positive this post will get negative votes. That's fine. I like tilting at windmills.
My problem is that eventually all apps will be rewritten...and it's probable they will be in some kind of Serverless manner.
Containers still have some separation of concerns issues as well as maintaining state or sharing data.
My default would be Serverless with a plan B for Containers and a plan C for VM's.
This is the concern I have with k8s. All this complexity introduces operational and security concerns, while adding more work to do before you can just deploy business value (compared to launching on standard auto-scaling cloud instances)
If you want to go all out you can also grab an operator to manage rolling out databases for you (postgres [2], mongo, etc).
A lot of the complexity people bump into with kube is really poorly planned out tools like Istio that have way too many features, a very overly complex mode of operation (out of the box it breaks CronJobs!!!), and very sub-standard community documentation. If you avoid Istio, or anything that injects sidecars and initcontainers, you'll find the experience enjoyable.
[0] - https://clouddocs.f5.com/containers/v2/kubernetes/
How much Kubernetes-adjacent code you actually need to adopt (and therefore, how much risk you take on) depends from project to project and organization to organization.
It's kind of like an API gateway with traditional microservice instances. If you have DNS and load balancers pushing requests directly to your services, you might wonder why you would ever need such a thing. Until you do.
You get all of this with serverless (be it with managed services or FaaS), with reduced (albeit not zero) complexity compared to K8s.
It's fine for certain use cases but you get the benefits of both worlds by just using a managed K8s service like AWS EKS.
1) Are you on AWS? Then you don't need Kubernetes. Use Fargate.
2) Are you on Google Cloud? Then you don't need Kubernetes. Use Cloud Run.
3) Are you on Azure? Then you don't need Kubernetes. Use Azure Container Instances.
4) Are you on a PaaS like Heroku? Then you don't need Kubernetes.
5) Are you on a random VPC provider / bare metal machines? You could probably still do without Kubernetes using Docker Swarm (apparently it's not dead!), Nomad, Mesos DC/OS, or a standard Linux box and systemd (or some other process or cluster scheduler).
6) Do you need to solve the bin-packing problem? Do you need to self-host a service mesh of microservices in multiple colocated regions? Do you need a fully automated redundant fault-tolerant network of disposable nodes to constantly reschedule different versions of applications with stringent RBACs, scheduled tasks, dynamic resource allocation, and do you have about a million dollars to spend on building and maintaining it all? Then you need Kubernetes.
Also at this point there is a ton of stuff out there that assumes you are deploying to kubernetes and makes that the path of least resistance.
To overcome those limitations, you can use EC2 Container Instances with ECS. And instance auto scaling can now be automated for you via ECS Capacity Providers.
I think you're overstating the investment necessary to overcome the initial complication of Kubernetes and also understating the benefit of being on a platform with a massive and thriving community behind it.
As an example, in a prior role, there were a set of data engineers that would receive data in the form of MS SQL server backups from which these engineers would need to query and transform data on an exploratory rather than production basis. Certainly one could use an "undifferentiated" service from a cloud provider, but it was also a roughly 5-minute process for me to use the rather high quality helm chart and docker image commonly available to stand up a new service for the benefit of each engineer that had the need.
The process of creating that automation necessary to deploy the helm chart and restore the backups took approximately one hour and could be repeated ad nauseum in the aforementioned 5-minute time period.
There are many, many other examples of this. Want a data-science platform complete out of the box with no vendor lock-in, how about data8.org? The list goes on.
The trouble is, A/C is a complex system. There are moving pieces with specialized oils that oxidize and break down over time. There's a sealed system of pressurized gas. There's a pump, clutch, coils, fans, filter, thermostat, drain, belt, and electronics, Any of those parts could fail in a number of ways. Just to inspect it you need a custom gauge set, a tank of R134a, and a vacuum pump.
Etcd is about as complex as an A/C system. That is one of a dozen components of a Kubernetes system, before we get into custom integrations, which you will need about another dozen of.
The million dollars is to pay for everything needed to set up and maintain all of that, create the custom integrations that do not come turn-key from the community, create the custom integrations the community doesn't even have, integrate it with your development and deployment systems, business requirements, application-specific needs, and so on.
A million is an average. You can get away with less, just like you can get away with pumping a pre-pressurized A/C system with extra coolant: if you're lucky it won't break. When it does, I hope you have either a lot of time, or a lot of money to pay a consultant.
But it's part of other systems too, making the overall thing a system of systems. Interactions between systems of systems are complex and cause unexpected behavior. At some point you will run into an error in K8s that you can't resolve that will require you to debug Etcd. And "Bob" help you if the database gets corrupt or overwritten, or incompatible versions of software screw up what's in the database, etc. (My original analogy was inaccurate... Etcd is more like the engine than just A/C, because if it stops working, everything stops working)
Do you know what happens if an Etcd certificate's SAN field does not include domain names but only IP addresses? The client requests HelloInfo with an empty ServerName so it doesn't trigger a TLS reload on handshake, making it more difficult to replace expired certs. That is a single random quirk in a single component of this software which underpins all of Kubernetes. I cannot sit here and explain every single reason why Etcd is complex; it must suffice to say that the software just is complex, and that this reality means that while it may sometimes be simple for some people to operate, it will definitely not always be simple to operate, and there will come a time that the true cost will emerge.
Now, most people don't need to pay for that high cost of complexity. They can use a SaaS/PaaS product like AWS ECS/Fargate or others, where somebody at some other company is dealing with the cost of complexity for you. All you have to do at that point is run some API calls and everything just works. Not only is it easier, it's immensely cheaper, less time-consuming, and more reliable.
...But you might not even need ECS! There's a lot of work just to get a simple PoC up on Fargate with an NLB, ACM cert, RDS instance, security groups, VPCs, cluster, service, task, etc. Compare that to just spinning up a micro instance and running MariaDB and a Python on it, and the latter you can have done in 20 minutes. If you can avoid complexity and still meet your SLOs, do that.
Point being, you're asking the wrong guy.
Edit: I see you wrote "car's AC". My answer is pretty much the same.
I think ^^^ the above is the main thing driving adoption. It's a real social engineering problem. Many engineers like to work with the latest and greatest.
As a freelancer I visit quite a number of enterprise companies, think: Banks, Insurance, Airports, and they are all making the switch or are full-on invested in living in K8s by default. If it does not run in or was not made with K8s in mind: It will not be used/bought.
Another thing I'm noticing with smaller companies: If you start fresh, you choose k8s. Which mens all other stuff is already slowly dying by virtue of not beeing chosen.
Developers want/expect it, sysadmins see the benefits from day one, and companies see the potential gains of using less cloud resources and a platform that could potentially run in multiple clouds for the first time ever.
K8s, openebs, prometheus/grafana, loki, kustomize, github actions. This is truly where "it's at" at the moment.
We currently run everything on mesos/marathon, but are looking to switch away from it. K8s is kinda the “default” option, and is potentially appealing to some potential acquirers and investors.
But I never really see k8s being talked about in that context of “physical hardware that’s on prem, but not on MY prem.” Is there a reason for that? If we go with k8s is it going to bite us? Does anyone have experience with something like that they could share?
It helps abstract out everything in the stack below your app and easier conceptually on everybody; now the clients can train in k8s and use same set of tooling like prometheus/grafana etc that usually go with k8s, same or similar RBAC access etc.
OTOH realistic expectations need to be set; not because it's k8s there's going to be no problems or the learning/adaptation won't come without some pain. I suggest writing down some standard procedures for your clients like upgrades etc, pick same set of tooling for all of them (like same dashboard, same logging/monitoring/alerting etc) as a way to homeganize ("standardize") all of them.
Feel free to email me.
Kubernetes provides a leaky abstraction above the underlying hardware - the storage and networking are going to be different depending on who is maintaining the Kubernetes cluster. Kubernetes's strength is that it acknowledges the leakiness of the abstraction and makes it explicit. If your customer uses a specific networking and storage provider, Kubernetes makes it easier for you to say (or not) that you have certified your product for those networking and storage providers, and here's what the manifests look like, because there's a standard way of configuring the application to work with that networking (CNI, which powers the standard Service as well as maybe NetworkPolicies) and storage (CSI, specifically StorageClass) provider.
If you just provide Docker images, or VM appliances, then Murphy promises you that you're going to get frustrated support calls from customers saying "your application is slow and we don't understand why." Good luck then.
Try using eg. Ubuntu and some kind of centralised management tool, like Salt, and install k8s. For better control, use Flux for storing your k8s configuration (deployments, configs, etc) in Git. I believe it would be good for your sanity.
Else your k8s objects will be susceptible to someone doing a klutz and "whoops" your applications are gone, real gone...
But k8s introduced non trivial amount of complexity and its own bugs and maintenance cost, meaning new separate engineers to maintain and develop just k8s tooling. And we had to rewrite a lot of legacy code. But the trade off is much better for a big project. Cutting downtimes by an order of magnitude and being able to boast it to the board - apparently priceless :) .
Following the kubeadm getting started guide on the kubernetes.io site can get you an 'ha', 'production ready' going in a couple hours. Most of it is pretty mechanical, and only needs a couple key decisions, mainly your networking plugin. Generally the most popular ones have instructions as part of the getting started guide, making the process straight forward.
Where it quickly becomes difficult is after this step. You have a cluster ready to serve workloads, but it has no storage, no ingress/external load balancer.
Storage can be as simple as NFS volumes (you don't even need a provider for this, but you should use one anyway). Rook/Ceph will work, but now you've just taken on two complex technologies instead of one.
Without an external load balancer of some sort, you will have trouble getting traffic into your cluster, and it likely won't be actually HA. You can use MetalLB for this, or appliances. If you're just starting out though, you can totally get away with setting up CNAME aliases in DNS to your nodes in a round robin type fashion. It won't be HA, but it will work, and is simple and straight forward.
Ingress is pretty easy to setup for the most part. Usually just applying an available manifest with a tweak or two. If you go the CNAME route, you will need an ingress setup so you can serve http/https on standard ports without too many issues.
If you do all these things, then you have a real deal cluster. Things like ingresses are recommended even if you're running in the cloud, so you may find that you're not all that far off from what you might find there.
Overall, the biggest trouble is all the choices you need to make. If you're starting out, maybe read up on two or three of the most popular choices for each step, and then just pick one. Anything that exists entirely within the cluster can usually be expressed purely as source controlled manifests, and kubeadm deployments can be simple shell scripts if you don't make them do everything (i.e. only support one container driver, not all of them).
One major caveat; If you screw up your network layer, you basically have to start over. This isn't strictly true, but it's the one where you are often better off starting over when you need to make fundamental changes to your network setup (like podCIDR and serviceCIDR or your network plugin). Pretty much everything else can be made to work with multiple setups at once, or you just need to delete and redeploy that component.
- Do you have such high traffic that you need a distributed system?
- Will a unified framework solve all your distribution problems?
- Do you really need high availability?
- Can you swallow the cost of high availability?
- Can you handle the insane complexity of Kubernetes at a reasonable cost?
You should not start asking questions about "Pods, Ingress" or anything Kubernetes specific, those are just implementation details.
For me, this point is akin to asking "Can you handle the insane complexity of Linux at a reasonable cost?"
And what I mean by that is: If noone exists on your company who can administer it, then you shouldn't do it.
Not sure if it is a vendor lock-in. If a customer wanted to move from ECS to kubernetes, they will need to migrate the manifests, roles/rolebinding etc. That is a small effort as long as they do not have to rewrite the code.
Basically, I think that a team/product knows when the time has come in which the infrastructure has grown in complexity so much for it to need something like kubernetes to orchestrate it. If there are doubts, then whatever current setup is in place* is probably still enough and kubernetes is beyond what the team requires.
I am very proud of the one time I managed to convince both my then tech lead and project manager, in one of my past jobs, to move away from kubernetes into a simpler architecture leveraging docker, compose and PaaS.
* Hopefully one using docker and compose or similar, as mentioned in the article.
Once that feels too little, I would start looking at docker, and only when docker feels again too little, to kubernetes.
So in essence, for 95% of the apps/people, the answer is no.
Can anyone suggest a good migration guide from Docker Swarm -> Kubernetes?
At a recent employer, we moved cold turkey from Swarm to Kubernetes (K8s in the rest of this reply). But we did this one microservice at a time. We didn't want our resulting K8s solution to be compromised by a misguided attempt to foist Swarm concept into the K8s way. Probably the biggest decision to make is whether to manage the cluster from scratch (not recommended), by using kops to deploy on a cloud platform, or use a cloud-native solution like EKS on AWS. After that -- here's a good guide to help with the differences in configuring the services[1]
[1] https://kubernetes.io/docs/tasks/configure-pod-container/tra...
See also: http://dowebsitesneedtolookexactlythesameineverybrowser.com/
Betteridge’s law of headlines:
Any headline that ends in a question mark can be answered by the word no.
⌘ https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
As somebody who's pretty skeptical about onboarding the big lump of complexity that is k8s, I really appreciated the information in the article.
Interestingly, it was pg who got it going on HN: https://news.ycombinator.com/item?id=4092880
I once used the joke about Mr. Betteridge retiring and got a reply pointing out that he hasn't: https://news.ycombinator.com/item?id=10393754.
And then he goes on to throw buzzwords. Yet another poorly written article as is now the norm for Kubetnetes.