Kubernetes 1.18

All this knowledge is still relevant. Kubernetes is an extra layer on top of all the classical technologies out there. It is not a replacement - besides some fancy experiments, it doesn't replace GNU/Linux.

If you're going to run a Kubernetes cluster, you still have to know all the "classical" networking, GNU/Linux system administration and architecture, etc. Having a fancy task scheduler for containers, and a tool that sets up clever network bridges (or whatever your preferred CNI flavor does), and a bunch of other nice extras does not remove the requirement to know what's underneath.

And in any non-standard situation, there are chances that you'll have to build things from scratch. For example, I do want to run a K8s cluster over a cjdns overlay network (which doesn't have IPv4, so Flannel or Weave won't work). Haven't figured this out yet.

And if you're not setting up and managing the infrastructure - it's about the same as it had always been. Just a different user interface to manage your deployments/storage/networking.

> Managed solutions make K8s easy to use

I would disagree. "Using" is the same (that's the whole point of K8s). "Install and maintain" is easier, but only in a sense that you don't do this. ;)

K8S doesn't eliminate the workflow for "deployments, upgrades, monitoring, etc.." it just black boxes them. It also assumes out of the gate that everything needs to be able to do HA, scale for 1,000,000 instances/s etc...

Over and over and over people show examples (I'm guilty too) of running internet scale applications on a single load balanced system with no containers, orchestration or anything.

So please stop preaching this as something for general computing applications - it's killing me cause I've got people above me, up my ass about why I haven't moved everything to Kubernetes yet.

No, it isn't. It's extra complexity most don't need.

https://zwischenzugs.com/2019/03/25/aws-vs-k8s-is-the-new-wi...

However I do agree on vendor lock-in. At least to a degree. KVM and Xen didn't really prevent this for cloud environments.

I do think the main takeaway from a lot of this is the way software is designed. It is becoming more self-contained. Docker in many respects is like a static binary. FatJARs are a similar approach. Also Go in general seems to go that path.

What Kubernetes really does is providing an agreed upon standard for infrastructure, similar to what Docker gave for software packages.

They enabled concepts like Unikernels to at least become interesting, because they smoothed the way for the thinking of software that should be self-contained.

I think the future really is one where Kubernetes and Docker are just annoying overhead, where we find it odd how something "emulates" them, just how terminal emulators emulate... well, terminals.

We are in a feedback-loop where we put a more and more tight corset on the software we develop. First there were compute clouds, where developers learned it's bad to keep state around, then there was Docker, then Kubernetes where certain best practices, that have been best practices for a long time "forced" them to be followed more and more, especially because whoever provides your infrastructure and the developer are able to agree on the interface.

Docker and Kubernetes are standards due to their dominance, similar to Internet Explorer back in the days and Chrome today. As of now there is only minimal written specifications. Most of it is standardized by the implementation. Hopefully this will change some day to stabilize the interface and give opportunities for competing implementations, so more innovation can happen outside the boundaries of these projects, allowing for competition.

Maybe this has a positive influence on complexity as well.

To me this advice is only useful for tiny startups running a handful of web servers.

Although I would argue that you need to know what trade offs you are making if you have the right use-case (multiple containers you need to orchestrate, preferably across multiple machines) and you are not using it or a similar tool. There are lots of best-practices and features you get out of the box, that you would have to implement yourself.

You get:

* Deployments and updates (rolling if you so wish)

* Secret management

* Configuration management

* Health Checks

* Load balancing

* Resource limits

* Logging

And so on(not even going into stateful here), but you get the picture. Whatever you don't get out of the box, you can easily add. Want Prometheus? That's an easy helm install away.

Almost every system starts out by being 'simple'. The question is, it going to _stay_ simple? If so, sure, you can docker run your container and forget about it.

It's running and is more hands off than without it. I'm using a managed digital ocean cluster. I no longer have to worry about patching the OS as it's all handled for me. I also don't have to worry about having a server with a bunch of specialized packages installed, although I suppose only using containers could have gotten me that far.

I haven't had a ton of headaches. So, I guess people's experiences may differ.

It's interesting to me that K8s always draws out the "you probably don't need it" comments.

However, if I use a managed k8s service, I probably don't have to think about any of that. I can focus on the metrics of my application, and not the cluster itself. At least, that's how I think it should work. I haven't used k8s in a while.

2. Same applies to dedicate cores for pretty much same reasons

Having said that if you really really want quota but don’t want shit tail latency I suggest setting cfs_quota_period to under 5ms via kubelet flag

The symptom primarily seems to manifest that you get heavy throttling even though you haven't actually gotten to your cpu limit yet.

If you're just seeing heavy throttling AND its pegged at the limit, you haven't necessarily hit the issue and should raise the limit first and observe.

Also don't forget to eliminate other possibilities. We initially thought we were experiencing this issue and later discovered the app was just memory constrained and extremely heavy GC during certain operations was causing the throttling.

At the petabyte scale there will probably be huge cost savings to using your own private cloud on your own hardware.

The portability argument boils down to saying you are not boxed in. If things get bad enough you can move. This is a big long-term advantage for businesses because it means you can correct mistakes or adjust to changing business conditions. That's what most people who run companies are really looking for.

And it's not mutually exclusive. I've seen companies working very cost efficiently by using dedicated hardware for 99% of the workload and AWS or GCP for any experiments where they wanted to try out new features.

I get the feeling people glance over the documentation and assume that you MUST use every single feature. Most of the features are only there when you need them for supporting advanced use.

Where does Nix/Guix manage dynamic binpacking of applications to servers (preferably without a blood sacrifice on the altar of "infinite recursin")?

Where do they handle nicely (de)provisioning and attachment/detachment of special resources, whether those are filesystems, block devices, any kind of special device etc.?

How about integrating various special networking resources like load balancers?

No, NixOps and related do not handle it well, as they need static assignment upfront and can't dynamically adjust as the system runs. Even at my small scale, all those features above are crucial, and k8s let's me not worry about problems most of the time (Rarely we have a black swan, that pretty much always ends up being "not k8s fault that network died or nginx got hung")

Nix is great to build the image and manage changes to dependencies.

Don’t do what I did: googling and going through random top hits. Most of these are 1-off blog articles that revolve around “install Helm and then do these 6 things” or “just kubectl apply random.link.com/some-script”.

Doing that just leads to tons of confusion and anger.

My recommendation: Suck it up and read through the official kubernetes docs. Their docs aren’t written in a way to easily explain core concepts unfortunately. However, slogging through it will give you some initial exposure to concepts and will let you know where to go back to later when your making mental connections.

Next, look for k8s tutorials from Digital Ocean and Linode. In my opinion, they’re the best written guides for demonstrating how to get from A to B.

You’ll start running through those guides and be referencing back to the official docs. Gradually bridges will form on your head and you’ll get an intermediate, functional level of competence.

If you grok all that, then you are ready to grok the K8s Container Runtime Interface. And, if you really grokked it, you understand that Docker is besides the point, isolation and namespaces are optional, and so forth. All k8s demands from the "container runtime" is that a thing has a name and a defined lifecycle.

Once you understand how k8s works at the pod and node level then it should be perfectly obvious how it works at higher levels. That's why I really want to see someone write the bottom-up guide!

Start here: https://kubernetes.io/docs/tutorials/ Look at all the diagrams in the tutorials but don't bother with the interactive stuff.

Then learn the more detailed concepts here: https://kubernetes.io/docs/concepts/

Once you understand the lingo and the general idea of what the different functions are supposed to do, just copy some example deployments and try to get something of your own working.

I've found this pretty good starting point. Keep in mind the book is still a work in progress. Note, I haven't actually run k8s in production, but this book has helped me get something up and running in VM's pretty quickly using Ansible on top of kubeadm.