Containers Are Not the Future
linkedin.com
linkedin.com
To me, the rise of containers has been about packaging software alongside all its dependencies in a standalone package that is (to varying degrees) isolated from the underlying host. In that sense, lightweight VMs are an evolution of the container model we already use.
The author fails to explain how these lightweight VMs would be orchestrated. Maybe Kubernetes sucks, but if something more secure comes along, is that a failure of the container approach, or is it an evolution?
All in all, this feels like someone writing a post saying "The aircraft created by the Wright Brothers was terrible, therefore powered flight was a failure". That, an attempt to sell something by bashing popular techs and millennial coders.
Someone else mentioned this in a different comment, but there was very much a link to firecracker in the article.
That - and all the other many tools that you would deploy vms to the public cloud today of which there are numerous production quality options out there.
Lastly, this is precisely the point I was trying to address with the "I'm honestly curious - what are the alternatives?" paragraph.
As mentioned in the article - the base unit of compute on public cloud is the virtual machine so there are quite a few tools out there to deploy those. Some microvm implementations out there won't boot straight off AWS/GCE but many including Nanos do. It takes ~2min to deploy && boot most applications we are running on GCE - AWS is a bit longer (but I could see that changing in the future). Note - in this case you are actually uploading the entire image - if you are just spinning up/down than it's much much faster.
I think for our general use case we are looking at people with needs that are using vms or containers today (eg: not necessarily those that want plaid speed boot-times). That's not really something we are currently optimizing for. For something like a database there would be no desire to spin up/down so fast. Also, many frameworks with heavy language vms such as JVM or ruby take a long time to boot regardless so trying to run those in a 5-10ms boot-time doesn't make a ton of sense.
Having said that, there most definitely are very very interesting projects like LightVM, http://cnp.neclab.eu/projects/lightvm/ that do target low latency and NFV would be a great use-case for projects like that.
That's why containers are a security disaster.
And they also make it impossible to do system administration with reasonable resources.
There are two directions that most organizations can go from containers, and they have opposite answers to this question.
The first direction is back to a monolithic application running in a VM. Orchestration no longer exists as a problem. What else do we get? Development is easier. Debugging is easier. We have literally decades of experience figuring out how to make it work.
Very, very few of the people reading this post are solving problems harder than ones I was solving decades ago using that kind of stack. Other than the fact that your resume won't look as good, there is zero reason not to do this.
The other is, as the post's author recommends, even further into isolating the app. Down to something even simpler than a container. In which case you will need a solution for orchestration, but it won't let you leverage a single running VM to a whole cluster. Limiting the attack area will be good for security.
As a bonus things that container folks say work, but don't, will work. For example autoscaling fails in adtech with containers fails hard because the time that it takes to boot up new instances exceeds the time that you are getting flooded by your ad. But if your unikernel boots fast enough, now autoscaling does work.
Now there is a third use case. And that is organizations like Google whose codebase is so big that they can't run it monolithically on a single machine. Containers solve that in a distributed environment by letting them run only some of their code on each machine. There is a hidden cost that RPCs over a network are inherently an order of magnitude more expensive (in time, latency, and resources) than a function call inside of a running application. So you order an order of magnitude more hardware, and containers are the sweet spot for you.
But that problem is of a kind that very few organizations have. And making it actually work takes a lot of thought and organization that very few in the containerization world seem to understand. Certainly if you read a few blog posts, install Docker and Kubernetes, then start deploying through Helm, I am willing to bet large sums of money that you are almost certainly making rookie mistakes.
Of course it does, VMware makes a boatload of money with VM orchestration. Or you setup OpenStack which is free but a nightmare in itself.
You do not want a hand-rolled KVM virtualization stack. I've been there, I've seen the horrors.
At that scale, orchestration is simply not a challenge. We wrote a deployment script to roll out code changes and life was good.
2020 hardware is not worse than 2004 hardware. And while that business eventually failed, it was not due to technology. (And it made a lot of money before it failed.) As I said, most here are not doing something harder now than we did then. The same class of solution works.
As the old saying goes, KISS. Know when it works, and when it doesn't. As another saying says, YAGNIY. (You Ain't Gonna Need It Yet.) Deal with the limits of being monolithic after you encounter them. Don't complicate your life to tackle problems that don't exist.
fwiw, Stackoverflow runs on a similarly sized/deployed stack.
But that is because Stackoverflow is fine with their employees running a cheap, un-complex system that works (I can't remember the last SO outage!), and their employees are also happy with this.
Others, especially startups? Gotta chase these hipster credentials and use the free AWS/Azure/GCP credits somehow... and the employees don't want boring old school stuff, they want the newest hipster crap that's barely alpha tested, but in production. Utterly ridiculous. Most of them would be fine with three sub-$10k AMD servers and a bit of work by a seasoned Linux sys-admin. But eww. Linux management. Not cool/hip.
They do NOT need a stack of daily traffic analytics hardware. As I said above, remote RPCs take an order of magnitude more resources than function calls. If you are used to using various distributed tools to do analytics, you are used to requiring an order of magnitude more hardware than a simpler solution. And if there are inefficiencies in your solution, it is easy for a second order of magnitude to sneak in. Therefore your experience is going to seriously mislead you about the likely limits of a simpler solution.
Making this concrete, if your job requires a Hadoop cluster of less than 50 machines, odds are good that it could be run just fine on a single machine with a different technology choice. But if you are fully utilizing hundreds of machines in your cluster, then you actually need to be distributed. (Though you could achieve considerable savings for some of your jobs could be run in a more efficient architecture. Amazon in particular makes excellent use of this kind of hybrid architecture in places.)
Google is making excellent decisions for the scale that they operate on. You just are unlikely to operate at a scale that makes their decisions make much sense for you.
Stackoverflow doesn't need this kind of shit and frankly, no one else does. The GDPR was a first start in the right direction and I'm seriously hoping coronavirus has at least one positive side and eliminates those fraudsters who survived the GDPR wave.
Great example!
Far too often we introduce complexity to manage complexity. Then we fail to manage the complexity that we created, and conclude that we need even more complexity!
Taking the simple approach seems impossible and utterly scary. But it works more often than most would believe.
That's not to say that things like containers are useless. I've seen a lot of benefits from developers working on containerization because it forces them to think about making their code reproducible and automating their development and delivery models. That has huge benefits from a security perspective, even if the end result is that you take the artifacts produced and rsync them onto a remote VM in the datacenter and restart services. There's lots of good to be gained there—I just wish people would quit trying to forklift everything into containers (and thence into the cloud) when they don't have to.
1) automatic restarts of services when they fail: yes, this is possible with systemd but it's a nightmare to set up compared to a simple HEALTHCHECK command in Docker or an appropriate config section in k8s/rancher.
2) dependency chains (e.g. a java app that depends on mysql database and an external elasticsearch): again, this is possible with systemd, but way easier in k8s.
3) hardware/OS maintenance: in ye olde days, the admin would have to manually shut down services, copy data to new servers, start services there and then pray everything would work to prepare for a large-ish OS update or a longer hardware maintenance without having too much downtime (or buy more expensive software versions such as JIRA/Confluence DC). With k8s/rancher, assuming shared NFS storage, it's as easy as draining and cordoning the node, minimal downtime (essentially the time the software needs to stop/start) and no network/firewall/LB surprises along the way.
4) related to 3), in the cloud, assuming that services have horizontal scalability in mind (which most popular PHP environments have), it's relatively easy and straightforward to use autoscaling to save costs on one side but also prevent your site from going belly-up in case of HN/Reddit mentions. With manual deployments? Ouch.
5) Software upgrades or the creation of stage environments are so much easier, assuming well-written Docker images and software. Again let's take JIRA and Confluence - manually installed it's a lot of fumbling around to upgrade an instance, but with Docker? As easy as incrementing the version tag in the k8s config and 5min later it's updated.
After working with and securing large bare-metal and monolithic virtual host environments for way too long, there's got to be a better way to deploy services and update systems than the snowflake hell some monolithic approaches seem to bring.
Perhaps there's a middle way between both worlds that has yet to gain any momentum.
You mean like vmdk or qcow2 images did for nearly a decade before containers rose?
The tone is also very arrogant and condescending. Not a great way to win mind share in this industry.
1) Packaging 2) Isolation 3) Secure isolation
Containers have dong a good job of packaging and isolating applications. This is has been a boon to development. Particularly with testing and deploying applications in a more or less consistent state and managing those applications. Note that these benefits have nothing to do with security.
Containers have done a poor job of being secure against malicious actors but that doesn't mean the isolation is useless. From a security point of view you just have to treat it the same as if it was running outside the container(s).
One day we'll hopefully get something better from a security POV but that doesn't mean packaging and isolation are worthless in themselves, even if they don't add any security. The problem comes when people assume containers add more security than they do.
Most recently Google themselves wrote about this just a few weeks ago:
https://cloud.google.com/blog/products/containers-kubernetes...
but the issues have been present forever:
https://capsule8.com/blog/oomypod-nothin-to-cri-o-bout/
https://www.zdnet.com/article/doomsday-docker-security-hole-...
https://www.zdnet.com/article/alpine-linux-docker-images-shi...
I find it very irritating and deceptive that the author wildly equivocates between containers and Kubernetes almost every sentence. And it's incredibly condescending that they suggest people buying the container Kool-Aid are simply too young to have done development and operations on regular servers/VMs, "duh!" Or that they're just trying to pad their resumes. No, I've had plenty of years of experience doing that, and I find containers far simpler to develop with, deploy, maintain, and update. And I've never used Kubernetes.
As most others have said, this comes across as extremely patronizing and demeaning marketing FUD from a company so certain that they're obviously right and almost everyone else is mind-numbingly wrong that even if that's actually the case, no one is ever going to believe them; and very justifiably. This is not how you do marketing, or write prose or communicate in general.
The conflation of containers and K8s is pretty damning. I personally believe k8s is an early realization of orchestration and will be superseded by better ones. Building yet another orchestration system is a cottage industry at the moment and I've noticed that the existence of k8s hasn't impeded it.
And "spin up a linux box" is deeply naive. It is the habit of spinning up linux instances that leave companies and other institutions with huge collections of neglected, mis-configured, vulnerable linux VMs spread all over the planet with 0.1% average utilization.
Developers popularized containers, independent of the cloud or the troubles of operations. Containers solve tough development and packaging problems and have value whether the ultimate deployment abstraction is 'docker' or some microVM system. The most likely use case for whatever microVM product one might imagine is running containers.
* As for microvms running containers -- in this case Nanos doesn't run Linux so it can't run containers.
In the 1980s a "server" was a big expensive special snowflake. Each server had a "system administrator." The system administrator would "install" software "on" the machine.
I quoted all those words to highlight no longer valid concepts.
Boxes are ephemeral now. Systems have more than one administrator, sometimes belonging to a different organization or department. Applications are not installed "on" the system in the sense that they root into it and modify its state. If they do this makes them a terrible hassle to manage. Applications should be ephemeral things that can float freely between machines and store their state in some kind of distributed or remote data store or database, or at least can be easily migrated between hosts. Everything is mobile. Everything is commodity. Everything is liquid.
OSes are just not designed this way, so we have an ugly hack called a container that basically amounts to "let's tar up images of operating systems and just treat them like giant static binaries." Onto this is bolted a whole array of other ugly hacks to make that easy to deal with, but the concept is fundamentally ugly as hell.
Things have changed dramatically since the 1970s (Unix) and even since 1991 (Linux) yet we are still using the same constructs.
Containers do the first part but not the second. Classical OSes do the second part but not the first.
Here's what I would do if I were working on this problem:
(1) Shared libraries can be linked by name and version constraints or by specific cryptographic hash of the library. The former uses some kind of name resolution system to get a cryptographic hash from that name and version constraint. This is not terribly unlike NPM or modern Go module versioning. It would allow sharing of resources without security or "DLL hell" type issues. The OS lazily grabs libraries by cryptographic hash off a configurable list of sources that could include URLs, IPFS, BitTorrent, peers on the same LAN, etc. (1) Could also perhaps be extended to include binaries, but this would require more thought.
(2) Kill the concept of installing software "on" a machine, but in a finer grained way than containers. Segment applications and services the way users are segmented: each app or service has a home directory and a UID/GID. Get rid of /bin, /usr/bin, etc. in favor of some system infrastructure for finding binaries where they live. MacOS has some of these things but isn't consistent about them.
(3) Lock down the system a lot harder in terms of different users being able to see anything outside things they own. For example a user should not be able to "find /" at all. This would be similar to containers but a little simpler.
(4) Extend namespacing and user/group ownership to network identifiers like IP addresses and other things. I think you could do this with existing cgroups plus some user land support.
(5) Get rid of system level service management (except for truly system level services) in favor of per-user services. Each user can run their own services/daemons that bind to IPs/ports in their own namespace.
(6) Kill 'root' in favor of a hierarchical administrative permission system. This may require some new stuff in the kernel or maybe cgroups could be leveraged to implement it.
(7) Kill privileged ports. This stupid restriction is utterly antequated and serves no purpose on modern systems or networks, and it makes a ton of stuff needlessly insecure and complex by mandating 'root' to e.g. bind port 80. I wrote a rant on this years ago: http://adamierymenko.com/ports.html
(8) Implement process hibernate/restore and migration as a standard feature in the kernel (if it's not somewhere in there already... not sure). That way the 'box' can be rendered absolutely ephemeral and entire runtimes can be hot migrated between boxes.
(9) Replace numeric UID/GIDs with GUIDs or even better hashes of public keys to allow some transparent tie-in to some kind of IAM infrastructure. It's possible that this could be done by having a local mapping to numeric UIDs/GIDs to avoid having to recompile the universe and refactor code that expects UID/GID to fit in 'int' or 'long'. This way a user can log into any machine anywhere and still be the same user and all that UID/GID synchronization or LDAP or whatever trash can die.
I think that would be a start.
Some of these concepts have shown up in various places: MacOS, iOS, Android, Mosix/OpenMosix, Beowulf, various container virtualization stacks, etc. I have never seen them in one place or integrated nicely.
"Grab random libraries and never update them again" leads to the same security disaster of containers.
It's crucial to have large communities AKA Linux distributions that ensure that a set of libraries at given versions work well together and receive security backports!
3, 4, 5, 6, 8: you just described seccomp, symlinks, and tools the already exists.
The problem here is not about technology. Doing better packaging and modularization was possible for decades but goes against some commercial interests.
Making shared libraries and other upstream dependencies actually scale is an open problem in practical computer science / systems engineering.
The fact that commercial interests often have a vested interest in things sucking is a problem all over the place.
Have you seen or heard of any alternatives? I mean, software getting more and more complicated, maybe even more complicated than necessary, however running apps natively has its own pitfalls (we've all been there). It's like choosing necessary evil and if there is a better way of doing things we should evaluate it and talk louder about it
https://en.wikipedia.org/wiki/NixOS
Unfortunately these tend to be overly complex and have too steep of a learning curve, and they don't address all the issues.
The closest is probably apps on mobile, but again that doesn't address all the issues and comes with a whole set of restrictions that are very problematic outside small single-user devices.
[1] https://nixos.org/nixos/nix-pills/our-first-derivation.html#...
Seems like it would align pretty well with your vision
I recently wrote about this here: https://jonathan.bergknoff.com/journal/run-more-stuff-in-doc...
Something that really bugs me about a lot of criticisms of containers is that often you get two groups of people agreeing that containers are bad for opposite reasons: one group is fine with the old-world model, and one group that thinks containers don't go far enough in squeezing out the old-world model and replacing it with something better. It makes me worried that it's very easy for the audience to think the conclusion is "containers bad, old-world plain linux server good", which is detrimental to the second group's goal.
I'm not sure what's stopping people from developing tools to make the whole containerization process invisible to developers who don't need that much control. I'm sure stuff like that already exists/is being worked on, and I don't know why it shouldn't work.
That’s why I had to stop reading this article. They repeatedly conflate k8s with containers. From the users perspective, one is an extraordinarily complex orchestration framework, and the other is an incredibly simple packaging system.
Containers don't solve that issue, they simply allow you to ignore it. Not having to struggle to keep up with dependencies is the other side of the "containers are a security nightmare" coin.
I once returned to a company after several years of being away. Of course nobody had bothered to update the dependencies in the old build. When I was working to fix and update the old build somebody informed me that the problem had already been "solved" by another team, and directed to me speak with that team's lead.
They managed to get a new build with slightly updated dependencies working inside a container. They showed me the new build inside the Docker container. I asked some questions, they fumbled around to do a demonstration. It was clear that they hadn't solved anything. Instead of one problem, we now had two problems, as the new build environment couldn't be updated any easier than the old one.
But it simply can't be denied that productivity will usually win over X, where X is security, technical debt minimization, etc. And the improved productivity comes from unlinking dependencies, particularly process dependencies more than actual software dependencies.
I still prefer to package and bundle my projects, open source or internally, using RPM and Deb, or just the classic `make install prefix=/usr/local/foo` (support for the latter, with traditional GNU Make targets[1] and variables[2], makes building RPM and Deb infinitely easier). I know people will end up installing my projects into containers, but it's necessary all the same to ensure it's easier to keep those containers up-to-date.
[1] https://www.gnu.org/software/make/manual/html_node/Standard-...
[2] https://www.gnu.org/software/make/manual/html_node/Directory...
We ended up pivoting to running a lightweight linux, based on Alpine and orchestrating everything using Kubernetes and Virtual Kubelet [1]. Shameless Plug! Pods are isolated on their own virtual machines that are booted for the pod, the underlying OS is rock solid and gives users all the great tools, bells and whistles you'd expect from a linux based system. Fewer surprises, easier development. We actually open sourced the system today.
In 2017 we measured the restart time in of our unikernel images in AWS to be 22 seconds, all that time was waiting for Zen (2017... no KVM yet) to get around to getting to the place where we could run our image. So for our use case, the boot time didn't actually matter, it was far overshadowed by everything else happening under the hood.
I should say: Unikernels do have their advantages and should be used in areas that can exploit those advantages: Fast boot, low attack surface, way better performance for some workloads. We had trouble finding the specialized customers in the cloud that needed any of a unikernel's positives so badly that they would take on a unikernel's shortcomings.
This reads like a marketing piece ("CEO at NanoVMs") by someone who is betting on the frustration of Developers and Operators with the pace at which "DevOps tech" is moving.
Maybe MicroVMs are the future (whatever that means) but this is not the way to push it.
I don't know if Dark (my startup trying to solve this) is going to take over, but it's pretty clear that the current complexity is unsustainable.
This goes double if recession/layoffs/etc is happening due to COVID. People are going to need to spend a lot less on engineering, and the amount being spent on engineers that maintain infra is an obvious starting point. I would expect people to switch to much simpler infrastructures, and reduce the complexities of their stacks by necessity.
I'm very disappointed with the fact that now one cares about simplicity nowadays. Everyone gets excited by k8s, then gets excited about tools around it (k3s, kops, helm, etc, etc) but no one understands that those are incredible complex tools with high cognitive overload.
This is 2020. Why do I have to think about 'containers' and 'nodes', why do I have to wait 30 mins while my pipeline will build image and trigger k8s to redeploy the things, why do I have bother with tons of YAML?
10 years back we've got Heroku and surprizingly no one tries to disrupt the space of infrastructure simplifying then. Everyone just tries to get things more and more complex.
The Dark ecosystem is perfect example of what backend should be. I don't agree that we need new language though, but at least Dark tries to solve complexity. While others just bring more and more leaky abstractions to the table.
Even your comment doesn't received any response. Why? People don't like to get they code just to work without all the infrastructure burden? I really really don't get it.
Does that mean there can't be improvements? Of course not. If you're working on simplifying things even further then great. But I'm concerned with your ability to execute if you believe Docker/k8s have made things more complicated because that signals to me you may not realize where the complexities lie.
Meanwhile I'd love to hear about your ideas to make things less complex.
TL;DR -> I see what you are saying, but there are methods to make your infrastructure consistent and keep the manual drift away that do not require k8s-level complexity. Both for monoliths and microservices.
Because we talk of the processes of yore, I will use a simple application architecture in the form of a web site that clients access that CNAMEs to a load balancer.
The load balancer distributes traffic to web servers, which each make back-end connections to a database.
This could also be load balancer to hosts that just run single-host instances of docker CE/EE as well - microservices can be done through this setup as well. If they are designed correctly, they shouldn't "need" to all be in one cluster, after all, these are durable microservices, not sharded bits of a monolith, right? :)
You have 4 basic tools, which with not a lot of thought can be used as a foundational platform for many, many workloads.
Aside from your codebase, you have the following (and/or their competitors)
Hashicorp Terraform
JetBrains TeamCity
Octopus Deploy
With a build server, a deployment platform, and a configuration management platform, you can get 98% of 'web scale' (all the scale you are really gonna get + plenty of runway for all but the FAANGs among us) for WAY less daily cognitive load. This applies to the DevOps team, the developers, the support cannon fodder (note if this is too true, and you are in leadership - please help this by empowering your cannon fodder), the DBAs, all of 'em.
1. Programmers check in their code to source
2. TeamCity runs unit/integration/regression tests. Pass? Publish to nuget feed for Octopus
3. Terraform sets up automated DB, web server, load balancer infrastructure for the servers, configured for whatever your company's definition of 'production server' is
4. Octopus deploy takes the newest build artifact from TC nuget package and deploys it to those servers.
In the above process, if you need to scale up:
1. modify the count of web servers in terraform
2. Terraform apply
3. wait for TF to complete its provisioning of new servers
4. wait for Octopus to recognize the new servers as deployment targets for your code (in this scenario, you write TF such that it will install the Octopus agent and set deployment target)
5. Deploy your code to the new servers
6. Have Terraform reconfigure the load balancer to expect these web servers to work but start in a disabled state
7. the load balancer will detect they are now passing health checks, enable them, and poof, they are serving traffic.
You have infrastructure as code, you have CICD, you have scalability for miles, and not a docker registry, helm chart, istio policy, or rancher needed. Best of all, your helpdesk, intern developers, even non-tech managers can cognitively follow the process.
Need to patch? Cattle, not pets says:
1. Terraform: reconfigure load balancer to drain connections on the boxes to patch
2. Terraform: reconfigure lb to remove from LB when done
3. Terraform: destroy the boxes
4. Terraform: spin me up same number of new boxes. Using packer or a fresh AMI, the new box will be patched, then have TF set configs, install etc. as done with the original boxes
5. Octopus: deploy code to all known web servers
6. TF: add boxes to load balancer
PS: sorry this grew into a wannabe blog post with handwave set to maximum.
"Conclusion Containers are clearly not the future. You will eventually need to migrate completely off of containers and kubernetes and we are here to help."
While this does not negate the core idea, it summarizes the motivation and bias behind the arguments.
Unixes got the abstraction boundary not-quite-right, by excluding networking and filesystems from the concept of process.
Containers are a correction to this.
i.e. not quite as catchy, but:
"Containers: the present is what the past should have been."
* Windows abstract screens, Files abstract storage Sockets abstract connections, Threads abstract CPUs, etc.
This is one of a number of issues that make containers hard to deal with.
(We also see an issue where containers interact badly with the OOM killer, leading the entire system to hang because /proc/*/cmdline cannot be read at all.)
If you had a small army of SRE and seriously senior devs / devops you could easily manage containers.
If you are just stuck on running containers and k8s I suggest you leverage their resources and get to GKE
But then releasing k8s might just have been the plan to get people into GKE instead of AWS
Who am I to speculate on googles marketing acumen ?
The kernel doesn't have a concept of containers, but rather of kernel namespaces. For example, it's possible for a process to create a new namespace that is the same as that of a default process, except with a new file system namespace — i.e., a chroot.
Hopefully we'll see these tools updated to better understand kernel namespaces. In the meantime, if you need to interact with containers, use the nsenter [1] tool. For example, it allows you to enter the same PID namespace as the container, but still use the host file system.
For example, run to lsof against a container's PID 1 (corresponding to the host PID 1234):
$ nsenter -p 1234 lsof -p 1
I think the post-POSIX world will have to look somewhat like the IOS model of isolation, something like the NIX level of packaging, and something like the Docker model of running. A shared filesystem where certain information is held in certain locations in the filesystem is just a nightmare to manage, but it is currently not possible to replace it.
Unfortunately, that's where we drew the line on "startup innovation tokens" - maybe round 2. :)
isn't that the basis of like, every modern operating system? Haven't we been working on getting it right since the '70s?
I know one developer who is obviously planning to go to Amazon because every single application he works on immediately starts with looking at which AWS services he can glue together. I'm to the point where I want to create and AWS bingo card and when hes on a conference call, pass it around to see if anyone wins by the time he describes some solution set.
"Well I'm starting with [AWS service here] but it doesn't support [major, normally functional breaking requirement], so I'll use [this crazy set of JS NPM packages] to do the work in between"
Wish people would develop solutions, not resumes.
And last time I checked Kubernetes basically washes its hands of the problem, and maybe makes it worse by adding layers that obscure the security properties.
I worked on a Kube-like project from 2010-2015 and I was definitely horrified by the state of containers in the Linux kernel. Security is basically what made me give up on the project.
Undoubtedly there are risks to having the flexibility that a full OS provides, but the benefits to troubleshooting, incident-response, resource-sharing, and just plain operational flexibility are huge. Until there's a good story for managing unikernel-based microVMs at scale, they aren't going to make a dent in container-world.
I suspect the path we'll actually see pursued is more along the lines of Bottlerocket (https://github.com/bottlerocket-os/bottlerocket), which is basically the Linux kernel plus a containerd API that works with Kubernetes but doesn't expose nearly as much surface area.
Most software sold to large organizations are tire fires/clusterfucks, Kris is just saying up front that complex software is hard.
Everyone knows reasonably complex software tends to be a clusterfuck -- if you're making purchasing decisions based on a vendor telling you their software does all these things and it's a perfectly engineered masterpiece then ...
Kubernetes isn't directly responsible for the sins of container runtimes and vice versa. Lightweight VMs solve some of these and I don't think it's controversial at this point to say so.
I may have reservations about the design of Kubernetes but I don't need self-serving conspiracy amongst the designers to hold those reservations.
Disclosure: I work for VMware, we do a fair amount of Kubernetes stuff, including lightweight VMs.
Container orchestration is not the first silver bullet that has grown a large following that makes money off of it, whether reasonable or not. I do have a question though. Say you have a bunch of microvms processing your workload or serving dynamic pages. How are you going to manage your fleet of MicroVMs? Would you perhaps write a microvm orchestrator of some sort?
Got a huge kick out of the first comment on the linked page:
> You know what you call a "micro-VM that's purpose-built to run one application" if you're a Unix person?
> A process.
> And now we've come full circle. Everything old is new again!
It's not impolite to point out impoliteness.
Public cloud like AWS and GCP does all the networking, volume management and everything else for you - for free. That's arguably one of the reasons for using it in the first place. There is no need to put something on top. That's kinda the whole point here.
A lot of older unix abstractions have been heavily broken when we all started running tens, hundreds, or thousands of vms on the public cloud. Users are relegated to IAM. Most databases can't even fit on a single server. Many webapps are load balanced to begin with.
The classic monolithic 'server' named Mars or Jupiter outgrew itself cause we all were consuming so much software. So part of what we are asking here is why do we have 2 layers of linux? We have the underlying one for the hypervisor but all these legacy abstractions are still present in the guest.
a) I don't necessarily agree with that statement - witness HashiCorp's recent raise:
https://www.sdxcentral.com/articles/news/hashicorp-scores- 175m-funding-round-5b-valuation/2020/03/
b) The vast majority of container users run their workloads on public clouds like AWS and GCP which run on top of virtual machines to begin with.
I'm not stating you need to use terraform (I definitely don't) - what I'm saying is that you can use the underlying orchestration provided by the cloud of choice - be it the networking or the volumes. There's no need to replicate that on top as container workloads do. When people who speak of kubernetes/container complexity - this is what they are complaining about.
If you look at a lot of the configuration mgmt software out there - chef/puppet/ansible/salt/terraform/etc. it all involves configuring a linux system whereas with unikernels it's literally just one application and instead of having to plumb your network together or ensure that your workloads or 'stateless' vs 'stateful' you just use the underlying cloud facilities. That's the magic. It actively removes complexity from the provisioning.
In the video of someone breaking into a Kubernetes cluster, they start off with restricted kubectl access. Okay, so restricted kubectl access isn't a strong security boundary. I don't think it's a big deal to a lot of people's use-cases. I didn't know it was a feature; the lack of a strong feature like that is only a dealbreaker to certain kinds of setups. It doesn't seem necessary for sandboxing different services apart from each other; it's for giving different sysadmins/developers access to deploying to different parts of the cluster. I'm going to make a wild guess that the issue isn't core to and unfixable in Kubernetes, because the person in the video shows a way to plug the issue. Summarizing this as calling Kubernetes security a clusterfuck is way over the top.
I could believe that the author's NanoVM project is even better at security out of the box, but I find the way they present the argument as overblown, and they don't introduce NanoVM for comparison until the end, so I think the arguments are doubly misleading because most people are going to be comparing containers to the classic plain multi-tenant Linux server model, and Docker and Kubernetes seem like at least an improvement to that in security.
OK.
I feel as though the author is only thinking about Kubernetes as an application platform. However, you can build platforms on top of Kubernetes (knative, Istio, etc) which solve many of the issues the author has brought up.
When I saw and read about unikernels a few years ago I knew they'd be the future. I've explained several times in the past: why did we create containers instead of working hard to make VMs better, lighter, faster? Now the time has come.
K8s will be dead in 3-4 years.
It's something the article doesn't address. Containers used to be a slam dunk, so I'm not sure I'd say that we were wrong _at the time_. But HW hypervisors evolved, so now I can see HW hypervisors like Nitro and Firecracker being compelling for some* workloads (e.g., to give the end user a dedicated kernel where they can do anything, including run any BPF program, run PGO kernels, etc.)
* I'm not sure yet whether it's "some" or "most".
Containers are just a solution to a problem. I’ve been around long enough to know that containers aren’t novel either.
Novelty is the future. It always is.
Sure, I can take a look at these microvms, but if they're not as convenient to run multiple projects simultaneously—I'm a freelancer and have my own hobby projects—then I see no reason to switch.
After all, I used to use vagrant, but spinning out containers is so much faster.
https://share.getcloudapp.com/JruWePeK
Scrolling pushes state to your history, and it's so poorly done that hitting back doesn't even restore previous scroll positions (I'm assuming that was the intent).
Containers were never meant to be "secure" and Kubernetes has nothing to do with security in general. Application level containerization solves 3 problems: 1- repeatable builds 2- software distribution 3- infrastructure mutability
Docker solves brilliantly all these problems: we can build software in controlled environments that we call images and the product is an image itself, that we can share by means of software catalogs called registries and that we can deploy the same way we can play CDs in a computer, without modifying the host system. Security was never meant to be a concern, it was never part of the plan, at least no more than running a process in an operating system. And containment systems are just a smart way to launch a process, that's it. They can't replace security functionalities beyond what is already provided by host OS.
Is this a fault? Would you blame a hammer for not being able to unscrew a cork?
NanoVMs CEO spends some time also making fun of "stripped down systems", accusing them of recklessly "ripping out components of a system that simply don't belong in the virtualized world of 2020", namely users, SSH logins and desktops. Again, there is a big misunderstanding here. Linux processes expect some facilities to be present at runtime, namely system calls to retrieve Process IDs and user IDs. But these facilities were merely retrofitted in containment systems to avoid processes crash upon syscall invocations. As all retrofitted functionalities, their implementation is barely beyond the dummy level. Therefore, we have PID 1 everywhere, no SSH logins and dummy users.
Now, Kubernetes. K8s is just a scheduler, an orchestrator. It has nothing to do with security, it is mainly concerned with resource optimal usage and container placement. And it is a fundamental component in microservice world, otherwise we would be paralized by operations at each deployment. "You spin up a linux box and call it a day - done". Seriously? Is this really the whole understanding of the world of operations? I've been in ops for 12 years and I'd never want to go back to the old bad days of 2008, where most of us managed an handful of servers and services and, yet, datacenter ops were at one with deployment ops. That was the very world of manual and error prone ops. Orchestration has been a bliss but requires a unit of scheduling. A container, in this case. But it may be a VM as well.
So, the saddest thing here is that NanoVMs could be a perfect replacement for containers while retaining the benefits of orchestration (as in Kubevirt, for example), but the CEO prefers to throw away the dirty water along with the baby for marketing hype.
There is no way to go back to VM deployment and management. I used to be openstack wizard, proxmox ninja and kvm/vagrant contributor and I cannot really think about giving up my whole container orchestration (ci, deploy, update, introspection , monitoring, you name it) to use back VMs.. I'm 40, doing sysadmin since 2002 and computer scientist since 2006.