Cost wise it was cheaper for a while but now RedHat are bumping up licensing costs so now I think is about the same costs.
Overall it seems like a waste of time, but has been interesting.
Cost wise it was cheaper for a while but now RedHat are bumping up licensing costs so now I think is about the same costs.
Overall it seems like a waste of time, but has been interesting.
- Builds with fixed dependencies that never change. Rollback is easy
- Easy deployment of a prod environment on a local machine
- Fast deployment
- Easy automation (use version X with config Y)
With Kubernetes (or other derivates like Openshift) you get:
- Auto scaling
- Fail over
- Better resource usage if multiple environments are executed
- Abstraction of infrastructure
- Zero downtime deployment (biggest point for my company, we deploy >3 times per week)
There are applications that do not need Kubernetes or even containers, but is this list really nothing oO?
I can imagine that if you use Kubernetes just like a classic cluster it could seem like an unnecesarry added complexity but you gain a lot of things.
> Builds with fixed dependencies that never change. Rollback is easy
Any good build system already did this, such as Bazel, or a Gemfile.lock. We'd just snapshot AMIs to keep OS dependencies fixed... which is what Docker images effectively do. If you re-docker-build the same Dockerfile, it's not like you get the same result of "apt-get install libxml" the next time either.
> Easy deployment of a prod environment on a local machine
How containers are deployed varies wildly between prod and the local machine. All the things that were hard before are still hard. Things like secrets and external dependencies still usually vary.
If prod is a kubernetes environment, getting a suitable k8s environment setup locally sucks, especially since it will probably have a different ingress controller, load balancer setup, storage classes available, resource requests, etc. If prod is kubernetes and local is docker-compose, that honestly seems like just as much work to create a second way to run the stack than just using a bash script + "npm start" or "bundle exec rails server" or whatever.
Either way, it's not really a prod environment. It's hard to run identical-to-prod environments locally, and those problems are related to secrets and clouds and such, not due to the lack of containers, in my experience.
> Fast deployment
In my experience, containers haven't sped up deployment. Let's say you use ubuntu for your host and container's OS. Before containers, this meant you had to download one version of libssl ever, and that was it. If there was an update to libz, that didn't require a new download of libssl. After containers, if you build your container for app1 last week, and your container for app2 today, the "FROM ubuntu" likely resolves to a different image. Both your apps now have different "ubuntu" layers, which probably have the same version of libssl, but deduplication of downloads only happens if the whole layer is identical.
In essence, we went from downloading 1 copy of libssl (for the host OS only) to 3 copies (host OS + 2 containers w/ different ubuntu bases), and there's no deduplication.
That by itself seems like it has to be slower since there's an inherent increase in network bandwidth that has to happen. Even if you have a shared base image, you're at least doubling the downloads of libssl since before you could use the host's copy only.
All the items you listed under k8s are things I had before it, excluding "Abstraction of infrastructure". Frankly, if you have a well-made load balancer, it's hard not to have zero-downtime deployments and auto-scaling.
This is a good solution, but I would not call it easier.
Using docker container feels like installing an app on my smartphone. I choose the version and it will always work like I build it at date x without an additional system. Works for every programming language with every dependency out of the box. Python, Java, Javascript, GO, Ocaml, C, ...
> How containers are deployed varies wildly between prod and the local machine
I just brought a product of my company to Kubernetes.
Run helm upgrade --install . -f dev-values.yaml for dev
Run helm upgrade --install . -f prod-values.yaml for prod (of course you need the secrets there. Jenkins has them).
My laptop does run an environment with all components of the prod env. Something like email and sap services are of course mocked, but everything else? All on my machine. Why not?
I can spin up a new test environment for customers with new settings on the same day.
> Both your apps now have different "ubuntu" layers
We use a base image that does change not that often. Even if: no problem, the registry is connected via 1000 MBit/s and zero-downtime deployment does its magic so I don't even notice if it takes one or two minutes.
Another thing: my node (or VM) libs and the libs of my software should not be connected in any way (at least for me). I want to patch my nodes and my software independently. Different software should also not be bound to libs of another software.
> All the items you listed under k8s are things I had before it
- How do you easily scale up? Including starting new machines and spinning down machines that are no longer needed
- How are multiple software parts executed on one host?
- How do you do fail over?
I know that everything can be done without Kubernetes. With enough time and money one can create large systems that do this.
I spun up a new Kubernetes cluster and ported our product (already containerized) on the cluster in about three months.
Really: I also love the classic dev ops and have a proxmox server at home, but Kubernetes just solves many problems at once in a short time.
AWS autoscaling groups + cloudwatch for adding and removing machines + checking them into load balancers is something that has worked for longer than K8s has been a thing.
> How are multiple software parts executed on one host?
systemd units, or for more resource hungry things, multiple autoscaling groups.
The overhead of running the kubelet on each host + etcd cluster + apiserver means that I still end up with fewer hosts if I just run each component on every single host vs scaling different deployments independently.
It is true that kubernetes might be more resource efficient in some combination of nodes and software, but at under 10 servers, I've always found the overhead of the etcd cluster + apiserver + kubelet to dwarf any savings from not just running 10 copies of my software.
> How do you do fail over?
The AWS-managed load balancer can fail over based on health checks failing, metrics, or I can add/remove servers from it manually. You can also do DNS health checks, or add a layer of haproxy/nginx/whatever if you want.
It's not like k8s has some magic ability to fail over under the hood. It's just using k8s service objects (probably LoadBalancer type), which does the same thing.
In the end you are bound to AWS and have to reinvent many things when moving to another cloud provider.
How long did it take to set this up?
> AWS autoscaling groups + cloudwatch for adding and removing machines
Does this also work for multiple applications on one host?
> means that I still end up with fewer hosts if I just run each component
I don't know how your environments looks like but we used one VM for every environment. One environment needs about 20 GB max so we have to use 32 GB RAM VMs and waste quite some resources.
With k8s I have two beefy 64 GB nodes that host 6 environments.
This also speeds up the execution as 70% of time the environments have a low load and the beefy nodes have more CPU cores available than smaller VMs.
Most of the time we can also throw some Jenkins and other test jobs on the cluster for free (low priority deployments).
> Overhead
Yes there is definitely some overhead. 2 GB RAM for the master and 700 MB RAM on every node.
We choose larger nodes (8 CPUs and 64 GB RAM) so the overhead is not that great in comparison.
We do gain savings from k8s (about 15 to 30 %).
> k8s magic
You are right, there is no magic sauce. k8s just packs a nice package of things I otherwise need to do externally. Sure the external logic worked well in the past, but now I get many features for a rather low price without committing my company to a provider to hard.
The "vendor specific implementation" is the loadbalancer and the autoscaling group. If you want to switch to baremetal servers, or some other cloud, getting a loadbalancer setup with similar properties is easy. Every major offering (GCE, Azure, HAProxy on baremetal, etc) will handle healthchecks and dynamically adding/removing servers.
Autoscaling servers based on metrics is admittedly more provider specific, but kubernetes doesn't solve that either. If I have k8s on 5 bare metal hosts, there's no cloud-agnostic way for k8s to magically launch a 6th host. You need to solve the same problem in either case. The k8s cloud autoscaler exists, and so do similar features for most clouds.
Kubernetes feels like even more lock-in than I have. If you were unlucky enough to run your services on docker swarm, mesos, triton, or various other container-running abstractions, then you'll have had to deal with a painful migration to kubernetes (since each of those technologies clearly 'lost'). Moving off kubernetes to another abstraction that runs containers (like mesos or such) is much more painful than moving from an AWS load balancer to a GCE load balancer or a bare-metal L5 load balancer.
In that sense, I feel like I have less lock-in than the typical K8s setup.
> [the rest of your comment]
It sounds like in your situation, k8s is working decently well for you. Congrats.
My point is not that k8s is always bad or using plain VMs + a load balancer is inherently superior, but that neither is clearly better, and both have benefits in different situations. In yours, k8s seems fine. In some other situations, k8s is a net negative.
When someone writes "fixed dependencies" I read "developers can more easily add more bloat before the cardhouse tumbles". That happens for example when the "fixed dependencies" are upgraded.
I am miserable having to touch all this junk. I feel a project is right when I can just git clone it (a few megabytes of data at most) and am left with a self contained repo that was written with minimal dependencies (optimally stored in-tree), and that can be easily built in seconds with a simple shell script on any reasonably modern system.
The bare bones way takes a good amount of initial work, but mostly it's a learning experience. Once one understands a few principles of writing portable software, I'm sure it saves a huge amount of time compared to adding all these shells of junk.
--
Oh yeah, I have zero experience about integrating with Kubernetes or whatever. I've been a small time user of Jenkins and CircleCI (unvoluntarily), and when I don't have to set it up and it actually works, it's alright and can help where the developer maybe lacks a bit of discipline (build all targets, run all the tests).
But, I doubt these technologies are a replacement for an ergonomic build environment (with simple python build script or even a crude Makefile). Is incremental building a thing on any of this CI pipelines? Because one thing I want is building really really fast, and it's already way too much overhead if I have to go through a git commit to check this stuff. Don't even think about requiring a full rebuild or Docker image build just to get some quick feedback on a code change.
If my prod environment disappears over night I want to be able to restore everything as fast as possible.
Otherwise my boss will be very unhappy and I will be promoted to customer :D.
> I feel a project is right when I can just git clone it (a few megabytes of data at most)
I don't know your experience but especially small projects often have some difficulties installing all dependencies correctly. The fun starts when you don't run a widely supported distro like Ubuntu.
If I want to run an old version of the production software I pull the image version X and execute it.
I know that everything in the package has the right version and works like it used to.
Tests have been executed and the version in the image has proven to meet the standards back then.
> But, I doubt these technologies are a replacement for an ergonomic build environment
If you build software for larger customer bases you will most likely encounter some kind of clustering.
Most large companies have already adapted this way. For personal and small projects it certainly is overkill.
> get some quick feedback on a code change
...wait, do you deploy to prod without tests and no direct way to role back the changes?
- Builds with fixed dependencies that never change. Rollback is easy -> what about VMs?
- Easy deployment of a prod environment on a local machine -> yep, that's a nice touch, the only valid point for me!
- Fast deployment -> lol no, Im faster with VMs.
- Easy automation (use version X with config Y) -> valid for VMs and baremetal too
With Kubernetes (or other derivates like Openshift) you get:
- Auto scaling -> you can get it with VMs too
- Fail over -> you can get it with VMs too
- Better resource usage if multiple environments are executed -> you can get it with VMs too
- Abstraction of infrastructure -> Should I really write it?
- Zero downtime deployment (biggest point for my company, we deploy >3 times per week) -> We do on some specific DC (government style) and we release 10-15 times and day with Bare metal servers and ansible
There are applications that do not need Kubernetes or even containers, but is this list really nothing oO? -> None of the arguments convinced me
I can imagine that if you use Kubernetes just like a classic cluster it could seem like an unnecesarry added complexity but you gain a lot of things. -> yes, extra cost and extra skills needed
I've seen more success from organizations running smaller K3s or K8s clusters (if they need the orchestration) or just running small apps via Docker/Docker Compose separately, using a CI system (even as simple as GitHub Actions) to manage deployments.
I wasn't around when it was introduced so can't talk to the initial complexity/pain but the current team responsible for it is surprisingly small.
The responsibility for everyday use has been delegated to all teams which gives them a capability to build and deploy frequently. That's a real enabler.
Most of the things that make this work are not really technology based, good collaboration, sharing knowledge and practices and being able to ask and get help quickly but I do think Openshift itself isn't bad at all. It does appear to provide a very decent build, deploy and run platform.
Was there particular problems, or just "all bad"?