Will Kubernetes Collapse Under the Weight of Its Complexity?
influxdata.com
influxdata.com
The industry is full of engineers who are experts in weirdly named "technologies" (which are really just products and libraries) but have no idea how the actual technologies (e.g. TCP/IP, file systems, memory hierarchy etc.) work. I don't know what to think when I meet engineers who know how to setup an ELB on AWS but don't quite understand what a socket is...
On one hand I definitely agree that it's good to know lower level technologies and that's something I always ask people in the interviews. I think it's important because I know it.
On the other hand there is no end on how low you can go in the technology stack. Do you need to know how sockets works underneath? Low level network protocols? Do you need to know how hardware works because it probably influenced decisions that were made on the low level of the software stack? I know what socket is, but maybe the only reason I know it is because I'm old and I started coding at the time when raw sockets was the only way to implement network communication. When we have modern libraries and frameworks do we really need low level knowledge?
Are you a storage expert? Do not ignore how storage hardware is designed (even on a historical perspective) and frequently think about the system calls it is your responsibility to implement.
Are you developing a sidecar proxy? Be fluent in low level network programming, and be aware of the interface you are offering to the application.
Are you a manager? Focus on your own manager and objectives, but do not forget to empathically look at your team members.
And a frontend sitebuilder doesen't have to know anything about transistors, FinFET, system calls, and only the most curious would know about reflow and painting calls in a browser's rendering engine. Sure, the quest for performance always pushes people toward the metal, but from a website's perspective that's just the browser, and maybe TCP window tuning. (And sometimes someone finds that DNS was slow, but sometimes it was a bad BGP constellation, and there was nothing to do really. Or maybe the load balancer - or the backend - was overloaded because there was mutex contention in the kernel - or in the DB behind the backend, but that's again rather far from the usual sphere of investigation for "Frontend Engineers".)
The imporant point would be to be able to go deeper, if there's a need. Or go higher, more abstract. That's engineering (or software design), recognizing, analyzing, and solving problems.
I can't really blame the people who get by just knowing parts of AWS. They make great money by being a Cloud Consultant. It simply shows the low hanging fruits on the market. And how bad the demand side is when it comes to picking the experts.
You meant, there is a very strong demand ?
And yes, there's a demand for these specific skills, because big inefficient companies are willing to pay for just this, because their in house people are usually overworked or simply not allowed to do it themselves without an expert on site (or on call, or on skype).
And of course AWS is a big jungle of buzzwords itself, without clear documentation (and that's of course very important for them, they don't want to give away trade secrets, nor it should matter for the guarantees they offer, but their own custom nomenclature and sort of arbitrary abstractions over the standard Xen/KVM/qemu, VLAN, VXLAN, etc stuff makes it very hard to know exactly what's going on), so no surprise that there's demand for this. But as others said, sometimes these consultants have rather narrow skills.
And to add a bit more to this, generalists are few, and always quickly disappear from the market, so the holes/niches are plugged by whoever happens to be around.
Finally, I don't exactly blame these corps, it's just what makes sense nowadays. Need for extremely short turn around, instant risk-elimination, ASAP ROI, all point to "experts" and "consultants". No point in getting someone who understands the full stack top-to-bottom, when the project is in the initial phase and you just need EC2 and a DB on RDS.
Of course I get more satisfaction in "feeling" a "deep" and "true" technical folk.
I think these are never contradictory, at least I think software jobs allow for the flexibility to keep working on high-yield things while learning something for depth on the side. (Or vice-versa, if you live off your side job, and do/learn something deeper/longterm.)
Especially with network, virtualization, security, distributed systems (DB, storage), etc. the ideas that are trendy now are spin-offs of older things. (Docker is basically LXC with a sane UI/UX, which is just chroot done better, a'la BSD jails, which is just Solaris zones, which is just IBM mainframe partition shit. Similarly everything on the network, like overlay networking, VXLAN, Geneve, OpenFlow, etc. are just natural layering abstractions and applied trade offs that were considered silly a decade ago, or were simply not needed a decade ago. Now that north-south traffic is dwarfed by east-west traffic in every system [this means that north is the client, south is your "DB", and east-west just means services in your system, and since people nowadays use a lot more complicated systems, the inter-component = intra-system communication is huge, compared to the few MBs sent to the client].)
>Now that north-south traffic is dwarfed by east-west >traffic in every system [this means that north is the >client, south is your "DB", and east-west just means >services in your system, and since people nowadays use a >lot more complicated systems, the inter-component = >intra-system communication is huge, compared to the few >MBs sent to the client].)
Never thought about it, in those terms.
There have been a number of cases where I've had to rely on the ability to debug binaries directly at -O3, or to resort to Wireshark to get the dose of reality needed to challenge my [our] flawed mental model. If I hadn't been able to do those things, I'd probably still be there, pondering those defects, if not in body [due to declaring outright failure], at least in spirit.
More than tools they are products nowadays :-(
If you are running k8s, you should run a proper overlay network first.
I would question the assumption of having "modern" libraries, or at least rephrase it in terms of "performance" and "reliability":
Do we really have performant libraries (do they at least saturate the hardware)?
When it comes to networking, I think the answer is going to be more and more, no, we do not have performant libraries and software is being outpaced by hardware. Hardware is sitting idle thanks to our libraries.
Do we really have reliable libraries?
When it comes to storage, again I think the answer is no. For example, Linux RAID has long suffered from a plethora of faulty logic (pick a random disk sector as the canonical sector, split-brain etc.) and of course Linux RAID is not at the right layer to even have a chance of getting this right. And yet even ZFS lacks algorithms to detect or prevent split-brain. It's possible to detect (with 2 disks) and possible to prevent (with 3 or more disks), by taking quorum from the longest partially ordered set in the topology, but even ZFS (which is still awesome) is not reliable in this regard.
When it comes to networking, even TCP is not that reliable but has huge issues with bufferbloat. For example, a single person uploading a Gmail attachment on a 10mbps line will break the Internet for a second or two for everyone at your office (try it sometime - get everyone running "ping" and then upload an attachment). This is why Apple's background software update uses LEDBAT. Most libraries are not using LEDBAT or similar but are trusting TCP's congestion algorithms to do the right thing.
Sockets are generally accepted as the standard. If you want to stream data to another computer, that's pretty much the only way to do it. The solution has been decided already.
A lot of these frameworks live in a space that's still evolving. We, as a community of developers, haven't agreed on the 1 best way to do it. As such, there are multiple offerings. In these cases, I feel it is definitely necessary to now what's underneath, in order to better evaluate ones options.
Not completely true. I think it's possible to have a complete simplified image of a computer in ones head while working on the newest version of your web app, going into depth in this area where problems arise.
And usually it's the other way around than you might think. Sockets you learn once and can probably even apply in 40 years. The current hip web framework you can't even use anymore in the same company in 2-5 years.
Would you complain that you can't be a doctor without significant training or a lawyer without understanding the law? At some point, knowledge is required. Distributed applications require certain primitives and those can either be provided by Kubernetes or built from scratch, but they need to be understood to be used. There's no shortcut to that.
People tend to say Kubernetes is complicated because they don't understand the problems it's solving in the first place. If they did, it would be rather straightforward or they would recognize they don't need it in the first place.
That may be deliberate resume-padding. Same as most Hadoop implementations.
unless you have a single server with no availability your system is probably not "simple".
* what do you use for master election
* what do you use to deploy your stuff
* what do you use on the tcp level for HA (keepalived?)
* what do you use to keep your system up to date
these would be my first questions about your system. and if you would use k8s all of them could be answered with "kubernetes"
Not trying to be snide, but so far I haven't found k8s to be much more than primitives with which to build those things.
I've also only begun working with it recently, so I do not have much experience, and any guidance would be much appreciated.
what do you use to scale your databases:
databases can be scaled with statefulsets there is a good library that helps with that: github.com/zalando/spilo it is actually way easier than scaling over vms (especially on bare metal) since 1.10 you can even use block storage with local storage, so no need for a storage system. so basically k8s is just a good layer to scale such things, before we used patroni without k8s and it is way harder to actually do service discovery (more on that later)
what do you use for master election:
master election can be done with configmap or endpoints (actually github.com/zalando/spilo uses github.com/zalando/patroni for running postgresql and does master election over endpoints or configmap annotations) actually etcd also exists when you use k8s (actually it's the hardest thing to install when you deploy k8s)
what do you use to deploy your stuff:
well deployment/apps are extremly easy to use and do zero downtime deployments when you just change your image version
what do you use on the tcp level for HA (keepalived?): some folks at google created metallb: https://metallb.universe.tf/ it's actually a load balancer on top of k8s that is so simple to use (just run https://metallb.universe.tf/tutorial/layer2/ and it will work)
what do you use to keep your system up to date: well that's more tricky however ContainerLinux (formerly CoreOS) (not related to k8s) works very well for keeping the os up to date. you can even use https://github.com/coreos/container-linux-update-operator to only update one node at the time in a k8s env. however it also works without k8s, but you still need to run mostly container software on ContainerLinux.
If you are new to k8s and want to know more about "easy" to use on bare-metal stuff you should look into ContainerLinux ignition and kubeadm, it's really simple to get up and running from scratch, it's mostly just creating a nice cloud-config and run kubeadm once on the nodes you wanna install.
And it's good for deploying big systems with lots of moving parts (such a OpenStack, or any proprietary system, let's say a big e-commerce site).
And even if the problem is straightforward (just run these docker containers, just do this rolling upgrade, just do setup these nginx load balancers, just add these to monitoring), all of these are a chore, especially if done right. And config managers always fail at the premise of apply changes to a "known state" (or they spend an eternity checking the state), so why not fuse actual monitoring with config management, and a lot of nice concepts fall out automatically, like Deployments. No longer do you have to disable the alerting part of monitoring, or mark certain parts of the telemetry time series as irrelevant for SLA accounting because we were deploying (or cobble together scripts to handle this for you), and then evolve the whole system as the underlying components (the things that are being deployed) change.
Sure, k8s is not there yet, as the problem is big, complex, and hard to get right even in parts, but a lot better already than the simple one-man-army homegrown solutions. (IMHO)
I keep meeting people who list say Postgres as a skill and when probed they admit they just clicked a button in Amazon RDS and that’s all they know about it. At some point it crosses the line into outright deception. This is the other side of the coin on why interviewing is so broken too.
I once applied for a job writing tools for embedded programmers that said they want experience in Angular, React, SPAs whatever. I'm working in embedded systems, so obviously 99.9% of people haven't even touched web apps, never mind the modern technologies. I actually got to the interview (despite no deception), because I knew someone working there. They said they actually don't have a single web app and are doing everything in C# (on desktop), and were thinking about making some web apps.
They still don't have any web apps.
Who would trust a doctor who proclaims his expertise in terms of brand names: "I've got ten years of experience in Cipro and Amoxil. Lately I've been prescribing Zithromax too! It's exciting to write that big Z. I really like Pfizer's community, their Slack is so helpful."
Whether we like it or not, software engineering is becoming a trade. Trades aren't inherently low skill or high skill.
What you say about trades is true, but then we need to start making the distinction between software engineers and software technicians. I doubt that will go down well in an industry where everyone wants to be a senior software engineer, a tech lead or higher...
Maybe that would be understanding the organs, the underlying physical processes those organs regulate, the chemicals that are released as parts of those processes, and the effects and composition of those chemicals, and if it is possible to induce similar effects through replacement chemicals in the way of medication.
Biology is not much different, though there schooling seems to do a better job at exposing the student to all the layers we know about.
I only wish that Informatics Technicial would be more prepared. And that Software Engineers/Licenciates would really be worth their salt.
On the other hand, someone who already worked with 10 Languages, 7 webframeworks and 5 deployment/config management tools plus showing good craftmanship has enough experience to learn one more of these quickly. Only for a short term gig, the learning/being productive relation could be disproportional.
I‘ve experienced it being habdled rather the latter than the former way and don’t remember not getting a job for having not yet worked with one of these tools, but bei g chosen for experience and quality work.
At every level of abstraction they is a boundary that people may not choose (or have time) to pass.
I'm not saying everyone should be forced to learn about every layer, but these knowledge boundaries exist everywhere, and I'm not sure what can be done to improve the situation...
They do write code but the technical job, per-se, is not that skilled.
Please don't miss the Cloud Native Trail Map, which is the front side: https://twitter.com/dankohn1/status/999727252135972864
What we discovered is that this meant they understood kubectl pretty well, and could write the yaml files. He didn't understand anything under the hood, so we had to re-discover everything.
And, ultimately, if you want to get a feel for how complex k8s really is, try and use their documentation beyond the very basics. It's contradictory, incomplete, and behind by a version of two sometimes. It was fun building a set of automated interactions with kubectl and its cousins from the web documentation, only to find it doesn't work because the actual binaries expect different arguments.
Or the API libraries if you're feeling particularly brave: a set of these horribly complex, dynamically generated files. Oh, some of them will be alpha, others beta, others beta2, etc. The version depends on the function you're using. Have fun!
Kube is very clearly in the expansion phase of its lifecycle but I doubt it will stay this way over the long run.
Additionally, I think the fact that the overall categories are pretty well chosen is a sign of how decently understood the macro problems are and that there's a good reason to hope that you'll end in a place with 1-2 dominant players in each.
Then it became a commodity and now you can just use postgresql for a basic web app without knowing much about dbs.
Start adding scale and availability requirements in the mix, and you're back in need of specialized knowledge on indices, query planning, disk access patterns and db admin in general.
Same for cloud providers imho: you can throw an app online quicker than ever before, but as things get more complex you will still need an understanding of infrastructure, network, availability, security etc.
It's exactly the same with Kubernetes. It's just clustering software that ties multiple servers together to give you an PaaS-like workspace to run containers, but there are 1000s of details you can use if you need them to build much more intelligence into operations. If you don't need them, you can just run a single container and still benefit from simple declarative deployments and automated oversight.
Yes you can write a dissertation with a ton of support from Word to make you life easier, but doing simple things is simple.
It sounds like that’s what missing from kubectl. Even for a small start it takes a lot of knowledge.
To continue the Word analogy, that sounds like LaTeX. It’s very powerful, but no normal person is going to get started for the first time very fast for basic tasks. Certainly not compared to Word.
The Rails analogy from the article seems very apt.
kubectl run appname --image=yourimage
kubectl apply -f yourfile.yaml
Kubernetes is not going to be simpler or easier than the software that it's designed to run on top of itself, but it's not that hard anymore either. The installers work well, there are several distros with varying capabilities, or you can just use the public clouds like most where it's 1 button.
Plugging in numbers for my $1000 synology, which runs a half dozen dockers, a vm, and is off-site backed up (for $120/yr), it tells me that I’ll pay $135/month to run comparable kubernetes at google.
Even if you pretend my $75/month broadband connection is only used by the synology, and include power costs, the synology still wins by $10’s/month.
(I included 3TB of storage for kubernetes, and 6TB usable for synology, with 3TB used, and sized for the high-memory 4 core machine, since dram is the synology bottleneck, even with a low wattage cpu. The synology is intentionally over-sized and is basically idle, so “but scalability!” isn’t a valid complaint.)
Also, the 9’s I’ve observed were trouncing amazon for a while, though they may be catching up, thanks to comcast...
But if you want to roll your own, then you can, it's FOSS after all. kubeadm works very well.
Comparing Synology with anything Google is silly, but useful. You can rent dedicated machines easily. Leaseweb is nice. And then you can build your infrastructure for cheaper.
Yes, different things are different levels of complexity, and using a managed service means paying someone else to handle it.
E.g. over years of using Etcd. I've come to the conclusion that none of the uses I've actually used it for were necessary, and so I've generally stopped using it. I'll use it again if I come across instances where consistency is sufficiently critical and the system needs to be distributed, but that's a rare situation to be in. E.g. I've seen lots of systems try to use it to distribute configuration data. Most of the time this doesn't need to be consistent, as long as you can determine whether or not it is (and so take outdated instances out of rotation, or not put them back in rotation). Same for load balancer configs, for example.
It's gone to the point where whenever someone mentions "distributed database" alarm bells goes off in my head. 9 out of 10 times it's a sign of someone over-engineering a system and building in unnecessary complexity.
What I want out of a system to manage my containers is something that reduces complexity of deployment and operation, not something that introduces more complexity. So far I've seen nothing of Kubernetes that indicates it fits that bill. Maybe it does on the very high end. My largest systems have "only" been in the few hundred containers/VM range over dozens of servers in 4-5 physical locations (on premises/co-location, managed and public cloud). For that scale, I've personally found Kubernetes overkill and adding too much complexity.
Maybe that'll change some day as it matures, but I'm not in a rush to complicate my stacks.
You have 1000 machines. Each runs 1000 containers with an average runtime of 1 hour per container.
Design a scheduler that runs on a single machine with a cold spare. For failover, assume you have access to a reliable RDBMS or relible NFS server.
My main point in all of this is that concepts must be understood to run large distributed applications. If you don't have such an app, nothing applies. If you do, then you have a choice of tools to use, and one of them is Kubernetes which offers a great foundation and set of features. There are other tools like Nomad, Chef or your own code and scripts - it doesn't matter what you use as long as you understand what you're doing.
Or just remain a bit more strict and look at typewriters. Word is a typewriter. People look at typewriters and sort of understand them after pushing one-two buttons. Yes, there are hidden gems and details and a long road from mechanical typewriters to Word and laser printers (e.g. https://www.youtube.com/watch?v=bRCNenhcvpw ), but people are a lot more intuitive about these pieces of hardware and software, because they are simply updates on existing tech.
Distributed systems is a new, and hitherto an explicitly unknown concept for many people.
And with minikube you can set up k8s as easily as MS Office. (Or you can use Typhoon https://typhoon.psdn.io .. )
And it's trivial to package up all this into a website, like Office 365 and GKE https://cloud.google.com/kubernetes-engine/ .
Or you can run it on your own PC with minikube and Virtualbox/KVM/qemu, and similarly you can install Office locally. But it does not make sense for one and it increasingly makes less for the other too.
K8s is also not that complicated after you grok it. It's also pretty easy to get started for simple cases. It's just that you need to switch your mindset a bit; but the benefits from the mental switch are non-negligble.
Writing (or code-generating) a bunch of YAML configs is the easiest part about K8s. Diagnosing why in the world it misbehaves in some weird manner is where the real complexity is.
Of course, this does not apply when you're not a system administrator, but just an user of someone's else (Google) K8s.
With GitLab you can deploy to k8s directly. Like Word and let's say SharePoint.
Also, there are a lot of folks doing this commoditization and packaging up k8s in various ways to make it more usable for a wider audience.
You're comparing k8s to Word instead of Windows kernel. Or maybe instead of .NET.
I'm a single-person ops team for my startup. Granted, I MAY be an exceptional learner and not realize it, but I mostly consider myself an under-achieving stoner.
I chose Kubernetes as our platform a few years ago and it's been absolutely wonderful and only getting better. Every once in a while I do a thought experiment with my co-workers to ensure Kubernetes is still the product for us. It is. We all love it and it makes my job easier. And it's really not that complicated.
Let's go ahead and list a few things Kubernetes Provides at the cost of writing a few YAML files and provisioning some docker containers in our CI/CD stack:
- Service discovery and health checks
- Zero downtime upgrades with rollouts and rollbacks
- Horizontal auto-scaling ( VM AND container )
- Configuration management and secret storage
- Immutable infrastructure
- Automatic SSL provisioning and routing via kube-lego and GCE ingress controller
- Log aggregation and monitoring via Heapster
- Cloud-provider agnostic configurations
- A consistent and approachable API to implement required features onto.
- ( With Google GKE ) Automatic security updates with a set maintenance window.
- Easy deployments with Kubernetes Keel
There are more benefits but these are the ones that come off of the top of my head.
ALSO, it's an opinionated framework that makes doing things "The Right Way" easy and intuitive to do.
In my TINY organization's reckoning, K8 actually makes our lives EASIER rather than more complicated and is considered one of the best decisions we ever made. It gets out of our way and lets us work on the fun stuff.
> In the case of applications that simply don’t have the scale problem, they usually don’t need the added complexity of a self-healing, horizontally scalable system.
If you build on a swamp your castle is going to sink into the swamp. Don't obsess over scale, but plan to scale, even if it's just "We can throw more servers at it."
Self-healing? The author is correct: Probably not necessary.
But scaling goes both ways: Our start-up is counting pennies and saving money wherever we can. Having an auto-scaling pod and node cluster makes so much sense. Our cluster automatically scales up and down based on load, saving us thousands of dollars a month.
> The problem I see with Kubernetes is that the cognitive load in the early parts of a project are simply too high. The number of things you need to create and worry about are a barrier to starting a project and iterating on it quickly. In the early days of a project, feature velocity and iteration speed are the most important factors. I view the Heroku model as the ideal development model. You have a managed hosted Postgres database and you just git push to get new code deployed and out the door. There’s very little to think about. It may not scale to infinity and it may get expensive, but you can worry about those things once you’ve actually got a hit on your hands.
I disagree with the assertion that starting out is difficult ( Or, that it is more difficult than other solutions ), and heroku is EXPENSIVE. With GKE, you are paying for the VMs and little more and get a BUCKET-LOAD more features.
What is this nebulous "complicated part" of setting up kubernetes? You literally hit a button and it creates your cluster.
You can have an application ready to go with three objects: a service, an ingress, and a deployment. I find Heroku to be of a similar complexity with less flexibility.
I get EXCELLENT monitoring and log aggregation by running `helm install datadog` and providing my key. I get a good-enough rabbitmq cluster by calling `helm install rabbitmq`. I get automatically provisioned SSL by calling `helm install kube-lego.`
So I suppose my response would be: No, I don't think it will. I think the complexity is over stated.
This is because you don't manage K8s and don't deal with this complexity. Google does.
Now, imagine a case where suddenly one pod can't ping another, but packets flow okay in reverse direction. The cost suddenly goes up from "writing a few YAML files" to "debugging CNI".
> What is this nebulous "complicated part" of setting up kubernetes? You literally hit a button and it creates your cluster.
That is, if you want to depend on other companies to provide your infrastructure. Which is perfectly reasonable business decision, but the opposite (self-managed bare-metal colocated servers) is also perfectly reasonable.
But if you're maintaining a fleet of independent sites, Kubernetes' scheduling can make sense, despite the inherent complexity (TBH, you're going to have a similar level of complexity managing the same kind of scale with any other tool).
I’m constantly trying to figure out what it’s hiding from the Kubernetes layer or what it is being manipulated to provide its behaviour.
I personally wouldn’t recommend Openshift->Kubernetes but the other way round would be a better approach once you know you need the additional functionality.
(Edit: fix typo)
Once you know what tools to ignore, and build scripts around the ones you need, it's very powerful.
This line of thinking is like faulting the golang stdlib for having a lot of useful stuff in it.
It's a toolbox. The job of choosing the right tool is on you.
One is organized and sensible. The other is a mess.
Just because they are both technically capable of the same things doesn’t mean we shouldn’t hold ourselves to the higher standard.
(Haven’t used Kubernetes, just going with the analogy)
I think you'd be surprised how quickly you can understand > 80% of it with just an hour spent reading the docs.
IMO it's a bit like Java for the cloud, the write once run anywhere bit (same caveats, but the win is still there). It intermediates the differences between cloud platforms. That alone makes it strategically worthwhile even if it was overcomplicated. But I don't think it is. Its core control loop is simple in principle, and the concepts and extension points fall out pretty naturally from it. It is a well-designed system at core and can support improvements over time without accruing cruft indefinitely, for example. Part of that makes it look like a bag of decoupled tools. But that's a strength of the system in the longer term.
I was simply commenting on the “it’s a complex thing so it had to be complex” line of reasoning. It’s faulty, you can make complex thing easier with good design.
Same with kubeadm. It's pretty okay for a test cluster, but it can't even do a HA setup out of the box. That's an absolute must-have if you have a project big and serious enough to warrant using kubernetes.
Disagree, it's a tool designed to abstract away the complexity of deployment and operations, simplicity of use/management sound like should be a design goal.
It will probably get there but for now it's hard to disagree with the "complex" label.
As a developer one wants to spin up a system to work on, then work on it, then push results to some repo. And this loop simply isn't possible (yet?).
Also what the author didn't mention is that even the vanilla k8s stuff is already super complex. Let's assume you manage to setup a cluster somewhere and it continues to work for more than 2 days (rarely seen in real world work environments). Then you are faced with deploying your hello world app to work on. Just for a simple single-server nginx deployment with no files and no config, you already need to understand mutliple objects: deployments, replicasets, pods, containers, nodes, hosts, services, nodePorts, port-forwards, maybe even ingresses and controllers.
That means until you feel well, even in a perfect environment, you need several days or weeks. And the documentation is not really helping you there. Yes, it's better than most enterprise grade documentation out there, but still it assumes a lot of stuff upfront. For instance, why should a developer even know what an ingress is and that it might be something he needs?
Combine that huge learning overhead with nearly impossible network debugging, beta level stability, and the low possibility to "just use it", then you have a system that most developers will never touch.
Docker itself might survive though, and one of the good things of this CNCF world is that other alternatives to docker also get a chance to improve on the existing system.
Now I can't find a way to get the docker kubeconfig back, resetting the cluster, reinstalling kubernetes etc does nothing, and now that I have uninstalled and reinstalled docker, kubernetes doesn't even come up anymore. I'm also not allowed to share debug information with the Docker company. No real help from Docker developers without debug info. Because they can't tell me how docker generates a kubeconfig or where it writes its logs without seeing my diagnostics output...
Not the vanilla perfect world use case is what makes your software but how well your software manages edge cases.
huh? What does that mean? Maybe that means the environment is broken, not the software you want to use/work on.
> For instance, why should a developer even know what an ingress is and that it might be something he needs?
Then probably that dev shouldn't work with k8s. At all.
If the dev wants/needs a hello world on a domain/IP, then they need a web hosting service provider. (ghost.org) Or a PaaS (Heroku), or they can spin up a VM on DigitalOcean and follow any of the thousand Ubuntu Nginx Website Hosting tutorials on HowToForge.
If they already have dozens of VMs, scores of containers, and they are fighting with monitoring and config management, then they might need k8s.
And a lot of folks do need this level of infrastructure and infrastructure management (automation, abstraction, standardization, etc).
> huh? What does that mean? Maybe that means the environment is broken, not the software you want to use/work on.
That is what the word "Enterprise IT" means. If you ever work in a company that makes more than a million USD per year, you will find an imperfect network in a nearly unknown state, with proxies and firewalls making your life hard, and automatic reconfiguration tools/scripts/antivirus-software resetting everything to "not working" the moment you take your eyes of the config files.
People and Software who really want to make money in Enterprise need to be able to handle that somehow. If you develop software on your macbook in an environment with the complexity of a Starbucks Wifi nobody can actually use your software in Enterprise.
Btw. did I mention that Windows+Outlook+Lync is the high standard of Enterprise laptops? Forget Enterprise users if you are not developing ON Windows.
k8s is a project, a lot of people find it useful. A lot of big corps have IT R&D groups (basically all Fortune 500 have), and they have their own test network. Or they test on their own rack, or on their own cloud, or on their own AWS account.
I think I don't really understand what your belief with regards to k8s is, but I'm interested, so could you give some details?
iterating on docker-compose might be easier/faster locally, then pushing to a dev k8s cluster for testing in ci/cd
curious if people are using k8s locally and why
Yes, if you look at all the possible parts, and at the current monolithic codebase, there's a lot of complexity. It also supports umpteen cloud providers, volume providers, networking stacks, etc., and comes with a whole swathe of bootstrapping tools for various environments (e.g. AWS).
But if you strip it down, Kubernetes is "simple": There's a consistent object store made out of JSON structures, and then there's a bunch of controllers listening to changes to that store to make stuff real. That is the core. Everything is, in principle, controllers mediating between the data model and the real world. Very elegant and orthogonal.
You also have an API, a scheduler, and a thing called Kubelet that runs on each node to manage containers and report node-specific metrics. And of course you have Docker, though with 1.10 you can more easily run dockerless via containerd, which is a great thing indeed.
The complexity comes from the operational part, when the pieces come together. And as you say, there's not really any way around it.
If you distill k8s down to this model alone, k8s becomes nothing but a pattern that has existed for decades. Maintaining "desired state" and "operational state" as separate things is not new.
And of course "nothing but a pattern" is nonsense. Pre-container systems like Puppet and Chef -- which are also, vaguely, based on converging real state towards desired state -- are firmly rooted in the traditional Unix model of mutable boxes. You can't implement a consistent reconciliation loop if your state can't be cleanly encapsulated (as with containers).
I can solve most of these problems with a simple Linux + systemd in a much more readable way and am finished much quicker, even now that I am using k8s full time for 2 years.
So, no. What it represents is the complexity of the community having multiple sources of requirements, and the complexity of the landscape with everybody trying to make a claim with their name+logo without investing too much into a full stack answer.
That being said, if you don’t need any of those things you probably don’t need Kubernetes either.
only used internally and not in a scaling fashion
> load balancing
External component that is hooked into k8s in some way
> autoscaling
External component that is hooked into k8s in some way
> automated rolling deployments
I doubt that many people are at a point yet where they can reliably do that. There are basic assumptions baked into that feature like that the k8s cluster survives until the next update comes out.
Real-life example : https://www.reddit.com/r/devops/comments/8byasq/is_kubernete...
Paraphrasing for discussion: Poster: A rails project with deployed to 6 servers running in production currently.
Poster: During the asset compilation process, the servers often freeze.
Poster: I need to manually remove servers from the load balancer and deploy one by one.
Poster:I looked a lot into kubernetes and production containerization lately, and as far as I read it, it should solve the deployment and uptime issues. I imagine it'd be a lot easier to just switch containers instead of deploying with capistrano. I also really like the self-healing capabilities a lot.
So, he/she hopes that Kubernetes will magically solve his problem(asset compilation freezes the server). I suppose in his/her mind, Kubernetes is the snake oil.
Things that he/she failed to put thought into (and rather got revved up about Kubernetes):
* Could I setup CI with a script that will perform the asset compilation once on one server and just rsync the final result to the prod servers?
* Could I spend a couple of hours understanding the asset compilation process and find out why it freezes the server?
* Could I learn more about load balancing, rolling deploys?
I think this is the real problem in the tech field. People are running after shiny tools and hope to through tools at their problems all the while ignoring the basics.
In this particular case, I think if they had a grey beard sysadmin who was grumpy to the devs, and enforced a strict release process, everyone would've been happier.
It reminds me a little of in the 00's everyone thought their company should write its own CMS. I think we're in danger of every writing their own PaaS.
This is why things like Deis and Cloud Foundry exist. Most app developers should not have to understand the full depth and breadth of Kubernetes.
This exists, its called helm, which in fact delivers the productivity gains the author is looking for.
>Maybe the combination of operators and Helm charts covers this, but I don’t think that will cut it. Then we’re forcing developers to learn about two other things in addition to Kubernetes. Even if it’s just increasing the vocabulary and installing a new command line tool, it’s extra effort and thought. These things need to be first-class citizens and part of the Kubernetes out-of-the-box experience. They need to be accessible via kubectl.
I mean, why should I install multiple libraries? Why can't glibc wash my car? Will glibc collapse under the weight of it's own complexity -- even though it can't wash my car like I want it to?
Don't buy the FUD. There's a lot of it about. K8s commoditises cloud providers. It's a strategic weapon against AWS lock-in.
With few exceptions, those cookie cutter examples are not ideal for running production applications; they're decent starting points for learning how they could be built, just like Helm could be (if it were a little less "here's a 1,000 line Kubernetes configuration file in a box, good luck!"-ish).
As an SRE / Ops person it is my take that Kubernetes addresses many of the complexity your startup should be worrying about, but doesn't because "it is not a feature".
Yes, it is massively complex. That is because Ops is massively complex.
K8s appears more open than Android, is not largely controlled by one corporation, doesn’t come crippled with vendor services, etc.
And having no „end users“ (app developers) on a conference about tools that should serve exactly these is an interesting observation to investigate further.
Having to install one mire tool to get ready for production apps installed with helm in one command is not asked too much, though.
Then, slighty unrelated, but it comes to my mind:
i wonder if this happy path thing works with influx, where the author is working.
Can i have a simple single command and install everything i need to look at logs from my app and db server, see most important performance stats and http/ip access logs, geaphical as well as with notifications if certain, easily to be entered ( and in case of cou, io, ram and diskspace reasonable defaults like 80% or so) thresholds are met?
Can i do that with only the free open source tools as the author expects it from the k8s ecosystem? Or do i get it when buying influx‘s professional service?
So maybe it’s the job of, and an opportunity for, commercial companies to develop and sell such simplifying tools. At some place, developers time to develop all these things must be paid for - if millions of developers just use the perfectly polished open source tools - and a high percentage doesn’t even help in development with big reports, not to think of patches, what are the developers going to live from while doing the polishing/ simplifying?
The Kubernetes website (1) currently claims that it supports clusters of up to 5000 nodes, which is a decent amount but not enough to avoid having multiple clusters. Does anyone have experience operating multiple production clusters in a single territory as partitions for scaling reasons? What's the experience like?
The vast majority of Kuberentes users probably won’t even need a 500 node cluster, let alone a 5000 node cluster. It makes sense for them to prioritize optimizing for other things like developer experience to help address some of the concerns mentioned in this article over supporting even larger clusters right now.
in case there was confusion
Also Kubernetes does support federation for cross-cluster deployments (now named multi-cluster). Some cloud services even support ingress load balancing across these, or you can do that part yourself by simple sticking with the same ports, but it all works fine today. Nothing custom needed.
That is pretty scalable.
An operator should never have to "kick" a service. It should repair itself, except for the occasional hardware replacement if one is working with bare metal. And for anything that's being sold as a product, as opposed to an internal tool, I think 99.9% availability should be the minimum.
But I don't know enough about Kubernetes to say whether it's overkill at the scale of just a few servers.
Additionally, resilience is in the eye of the beholder: might mean AP for a service that needs to be up at the cost of consistency or CP for a service that detects when it can't achieve quorum and fail gratefully.
Have you ever been a sysadmin? There are very few services that don’t need a kick every now and then.
The Cloud Foundry community has started exploring a switch from their own container management system to K8s. If that becomes real, CF would "just" become a nice user interface on top of k8s. The right move imho.
Related to this, SUSE and IBM have already released distributions of CF that run on Kubernetes.
I'm biased (I work on one of these) but I really think this is the most expedient way to enable PaaS features on K8s.
In particular, you can start off with Azure AppService, AWS Elastic Beanstalk, Google AppEngine. You could also go serverless. All these approaches allow rapid development and deployment with low ops overhead, and they'll actually scale and heal well. Ultimately, the services are doing the k8s type of stuff for you. To state that inversely, running kubernetes is like trying to run your own PaaS. (When put that way, it sounds dubious that so many people are trying to jump into k8s, but I'm not an expert on the $$ economics of devops.)
The next gen evolution of the cloud platforms could really take this migration from PaaS to IaaS to a whole new level than it's at right now.
Secondly, aren’t solutions like Helm supposed to take away the need for scaffolding? The problem with scaffolding is staying up to date with the new templates and usually results in the deployments not being updated anymore.
Additionally, I have to say that getting started with K8s was quite easy because we already had experience with Docker. OpenShift was similar and has source2build which is very convenient. So I don’t percieve K8s to be hard to start to use. To use it ‘correctly’ and use all potential, yes that is harder but that holds for any product.
My current client work has recently shifted to using k8. I took the time to get minikube working locally to get a better understanding. It definitely helped, but I find the layers of abstraction hard to grok after not working with it for a while.
I can see the value the tool offers, but I get the feeling it's supposed to be reserved for higher degrees of scale than the average 2-8 node app.
Black box is how I feel about it sometimes. Hopefully I'll get more one on one time with it in the future. It seems like a really cool technology.
The value imho could be in being able to package distributed applications and deploy across cloud providers or on prem, seamlessly.
I don't think this is true though short of doing a lot of effort to abstract access to a gcp/aws/azure managed service (say, a db), which is probably a bad idea.
If you take that away, then a lot of the replication, autoscaling, load balancing, failover etc. can be implemented using cloud providers without having to manage the complexity of k8s.
Hope to be proven wrong here.
The project has been listening: https://github.com/kubernetes/community/blob/master/sig-apps...
Rook sort of does this. You deploy a Rook operator, then just one other kubectl command to get an object store, database, shared filesystem, etc...
https://blog.rook.io/rooks-framework-for-cloud-native-storag...
Coming from someone who gave a talk at Kubecon I'm very surprised to read something like this. Maybe I'm the one with the misunderstanding, but I'm going to try and refute the things this article said/is implying.
1. Kubernetes is complex
This is kind of right, but it's also kind of not -- Kubernetes is essentially complex, given that it encourages write-once solutions to all the problems it faces. Here are the pieces that make a basic Kubernetes "cluster":
- apiserver => you send commands to this to change/query cluster state
- controller-manager => works to make your ensure that the cluster is in the state you want it to be (making workloads replicate/restart/etc)
- scheduler => figures out where to put workloads
- kubelet => runs containers -- one on each node that can do work
- kube-proxy => maintains the routing infrastructure necessary to enable containers on any node to hit a container on another one.
All of those pieces are needed -- the only concession I would make is that they could all be in the same daemon (one executable), but that's actually worse at scale, and harder to debug -- all of these services can produce a lot of logs.
2. Application developers can't use kubernetes as it is
Application developers can use kubernetes as it is. Learning to write a kubernetes resource definition is not any harder than figuring out the conventions and configuration you have to write for Heroku, or AWS ElasticBeanstalk, or AWS ECS. In fact, I would argue that it's simpler.
We've touched on another problem here -- the competitor for kubernetes is not SSH, it's not heroku -- it's tools like CloudFormation/ECS. I don't know if you've CloudFormation, but it's kind of a clusterfuck, hard to set up quite right, and the dynamic yaml approach they've taken is enough rope for one clever developer to hang you and your whole team with.
Bold prediction, but I think AWS is going to abandon CloudFormation and ECS in favor of Kubernetes resources once it stabilizes.
OK, let's say you disagreed with everything I've said up until this point -- at the very least, you can deploy tools like the following to your kubernetes cluster:
https://gitkube.sh => heroku workflow
https://helm.sh => cloud-formation/elastic-beanstalk workflow (with kubernetes primitives)
And presto, you have a completely different interface to your cluster, WITHOUT changing anything fundamental underneath.
3. Developers who only focus on the application-level are the goal
Why would you even want this? Not only is it basically impossible to hide the underlying infrastructure so well that the application developer doesn't have to know about it, it's arguably not even a good idea.
Take session management -- if you want to handle it in the context of more than one frontend running at a time, you generally outsource that state to a cache like redis. An application developer who grew up in this imaginary world where app developers never touch infrastructure is not who I want solving this issue, assuming there isn't a qualified ops person. If you needed to optimize even further, app-local caches could be deployed, but this requires knowledge of "sticky sessions" -- this very much is a deployment/infrastructure specific question, again, that app-only developer is just about useless here.
I'm no hiring manager but the desire to stay an "application" developer who only worries about that part of the stack when the "application" as a whole is so much more would be a red flag for me. Even if you were delivering a desktop application, the developer who worries about underlying OS-specific enhancements (for example knowing how to optimize the app for MacOS) is the one I want, the one I want to pay the big bucks for.
4. It's hard to deploy the usual app+backing store+caching+worker pool structure
The author touches on this a little bit with the "maybe operators and helm charts solve this", and that's exactly what the operator pattern (Custom Resource Definitions, AKA CRDs, plus custom controllers) were meant to solve -- now you can actually give declarative specifications of what you want your Redis/Postgres/Celery/whatever cluster to look like, and `kubectl apply`, and the platform handles it. There's arguably no difference here between how you'd use this and a tool like heroku.
Also, for the record you can trivially extend `kubectl`: https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plug...
This is already a thing, sort of, called hyperkube: https://github.com/kubernetes/kubernetes/blob/master/cluster...
The caveat is that each daemon has to be started separately still
I didn't include it due to that caveat, and the general feeling that the processes really are meant to be started separately. I haven't seen anyone try and run these processes with some sort of supervisor but it seems like that's not "the way" and wouldn't really even offer any benefits.
I don’t really understand what this sentiment is about.
It’s a really useful orchestrator, in some cases. In other circumstances, it’s unnecessary complexity.
https://kubernetes.io/blog/2015/04/borg-predecessor-to-kuber...
Borg and Container Engine:
https://www.quora.com/Does-Google-use-the-Open-Source-Kubern...
If you understand the implications of running N replicas of a command on <= N machines in the cloud, borg does exactly what you want. Most of the people I met at Google complaining about borg just didn't understand how a program gets executed at all ... borg was the least of their problems.
Also borg is easier to pronounce.
It’s amazing that the hosted option 100% mirrors the self hosted option which mirrors dev.
As far as I can tell, the aim with Kubernetes was never to replace Borg at Google -- Google is far too invested in Borg, and it would take a considerable engineering effort to migrate away from it. Rather, developers at Google saw an opportunity to create an open source version based on what they had learned and help the world along in adopting the same engineering principles as Google has long practiced. Not all altruistic notions, of course -- Google benefits from the commoditization of containers indirectly, by undermining competitors such as AWS (where containers are still not well-supported) and making their own cloud the best fit for Kubernetes.
Google does run stuff on Kubernetes, via GKE. As I understand it, new products are encouraged to run on GCP. I don't know how many applications they run, however. Maybe someone from Google can comment.
Despite having 15+ years of *nix experience, including internals, having a track record of building large scalable infra and knowing a few different programming languages, what happened to me was this: I was getting filtered out because I didn't have Docker and Kubernetes and even (at one point) Cloudformation and/or Terraform. No problem - I learned those things (minus Kubernetes, so far) quite quickly. Much more quickly than the grueling trial-by-fire years of Unix administration. I like to know how things work, not just how to use them.
So if you wonder and worry about the state of enterprise IT some days, look no further than hiring managers themselves, who will pick a 25 year-old who writes YAML for some abstraction-of-an-abstraction system that does infrastructure under the hood, infrastructure that people kind of don't really try very hard to understand. After all, it's disposable thanks to infrastructure-as-code, right?
How do I know this? Well, I've seen shop after shop that's suffered a spaghetti infrastructure, using all the latest and greatest, from AWS and Kubernetes and Docker and other abstraction layers above AWS. And what happens is that it gets so complex that no one knows what's really going on, and at the very least two common symptoms arise: people are terrified during releases and they take hours, with many people on a call together very late at night; they spend a fortune on extra instances (in the case of AWS) because they haven't properly worked out environment separations (they had trouble keeping them the same, or one of many other problems).
A talented dev manager I used to work with used to complain that they had trouble hiring people who knew Javascript well, but they had expert after expert of some fancy JS framework try to interview, unable to answer the fundamentals-types of questions. I think it's similar with enterprise infrastructure.
I don't know what to say. I hope things go full swing and people who know how things work under the hood can charge consulting dollars for fixing the fuckups. It's not enough to know YAML, you also need to have wisdom in maintaining complex infrastructure, understand the delicate balance between change and stability, and be able to troubleshoot when it goes wrong WITHOUT just 'rinsing and repeating' where you learn absolutely no lessons at all.
[edit:] One theory for all of this. Some of the big shops (Google, FB, Netflix, etc.) did it right, and now everyone is trying to copy the style of infrastructure management, except doesn't have the talent or wisdom to do it well.
In like 90% of the cases when someone used Kubernetes, Docker Swarm would have easily sufficed.
- Swarm does not have namespaces, a swarm cannot be divided into multiple independent ones.
- Swarm CE does not have per-operator Auth
And
- k8s has more moving parts
- has more complicated config files
To me swarm is to k8s like nginx is to apache. The former is lean and easy to setup, the latter has all the bells and whistles for enterprisey, service provider scenarios.
Docker Swarm is much more deserving of this kind of cynicism -- a weak, badly designed solution forced on users by a company that's realizing their invention has been commoditized and is no longer a platform they control. Swarm was redesigned at one point to work more like Kubernetes because they realized it was a much saner model.
Kubernetes has more complexity, but it does scale down to single-node clusters.
Docker Swarm might be incomplete and missing quite a few features, and it won't scale to thousands of nodes with dozens of independent apps, but the use-cases I've seen are far from that. A single engineer can basically get a "good-enough" moderately scaled system going.
If you have Google-scale problems with Google-caliber engineers and SREs backing you, use Kubernetes. Otherwise, using something else (Docker Swarm "just worked" for the cases I've seen) is easier.
The core of Kubernetes is super simple and all the hard parts are hidden away on actually setting up and maintaining kubernetes the hard way on bare metal machines. Odds are, if you're doing that, you have the resources to take some time to dive deeper into how it works.
I've done multiple single engineer Kubernetes setups that are working on production today. So far I've had only a few problems with it. I know it's not a huge data range, but I'm not the smartest person in the world (or a true infrastructure guy) and still found it easy to work with.
Swarm has always been a rushed afterthought IMO. Although I have way less experience with it than k8s, so I'm biased on this.
I moved to Swarm a year ago and have had a grand total of 0 issues. It can do everything I need.
The ingress story in kubernetes is bring-your-own and usually people run their clusters behind Google or AWS LB which are supported as ingress in k8s. Running k8s on metal is a super daunting task. Choosing your network plugin is another task and usually you have to install a different system service on each of your nodes - Swarm has this built in.
And lastly the Compose file format. 10 lines of Swarm compose file literally translate into multiple dozens of separate k8s yml file. Creating this in k8s is super complex in itself.
Swarm is really pretty nice for someone setting up a few dozen nodes and services.
In my experience it's even like this: I have never seen any working kubernetes app. But I deployed my own hello world docker swarm app in 2 days. (And that is a fact, not an opinion. The opinion part is up to you.)
A penguin walks into a bar, goes to the counter, and asks the bartender, "Have you seen my brother?"
The bartender says, "I don't know. What does he look like?"
Betteridge’s Law still applies.
Betteridge's law of headlines is an adage that states: "Any headline that ends in a question mark can be answered by the word no." It is named after Ian Betteridge, a British technology journalist, although the principle is much older. As with similar "laws" (e.g., Murphy's law), it is intended to be humorous rather than the literal truth.