A Practical Introduction to Container Terminology
developers.redhat.com
developers.redhat.com
So I felt flat. Computing was diminished not because Java was inherently bad, but it just didn't appeal to me personally.
As it is with Docker and containers. I'm sorry to go against accepted wisdom but I just don't like this technology. To me it is a complex kludge. I feel that it is simpler software technology that should succeed current overly complex technology.
It's my personal subjective opinion but I just feel that docker and containers are a big complex mess and I'm saddened that this is the direction of the entire industry.
Probably you love docker and containers. I don't knock that. It's just not to my preference.
I think many people don't see that a lot of the popularity is less about technology, and more about politics. "Devops", in general, is basically an organization change to move sysadmin's duties into the development team's domain. It has predictable upsides, and downsides.
In the real world, distributed systems are a complication that is better left alone if they're not necessary for a problem space. You very likely don't need Kubernetes (watch out, this sentiment will get your HN account throttled; I've already been punished for it so it's safe for me to say).
The reason Google et al push these distributed systems is because it gets you to buy more cloud instances. Very predictable and straightforward behavior there.
Like all technology that proliferates, this is definitely more about politics than merit, but it doesn't really move the duties around practically; anyone who thinks that a group so undisciplined and lackluster as your average software developer will willingly take on new responsibilities is detached.
As a sysadmin, you just get developers delivering garbage to your doorstep and expecting you to deal with it, same as always. It's just packaged in a stupid way now, that makes the developers say "Well I sent you a container, so this should be up and running in like, 5 minutes, right?" In practice, most developers won't even write their own Dockerfiles; my experience has been that they develop locally as they always have, and then expect the sysadmins/"devops" (now a term for sysadmins) to do all that mumbo-jumbo.
But yes, we've been through this song and dance before. It's static v. dynamic linking. It's shared tenancy v. single tenancy. None of it is new, it just gets a different name and oscillates from position A to position B.
It's an embarrassment and a testament against our industry that we continue to thrash wildly and be so thoroughly dominated by fads.
There are going to be massive opportunities for good consultants in this muddle. I'm currently watching one train wreck, thankfully at a distance. Nothing novel - they moved ops under the engineering umbrella. The engineering manager not only has no clue what the differences in managing the two are, but seems unaware that there are any. So of course the good system people quickly left, the engineers are slowly discovering that containers don't magically erase the need for understanding how things work, and they're spending six figures a month on K8s consultants.
We seem to have a generational cycle on IT/sysadmin/ops hate. As you get older, it becomes amusing watching every new batch of engineers learn why they exist.
The old-school, master of the universe-style sysadmin who had to use advanced wizardry developed out of deep knowledge and intuition about how the computer worked at a fundamental level is still the ideal developer OR operator. But it's hard to find people who even understand this role, let alone people who are hiring for it.
People only seem to understand pigeonholes, despite occasional well-intended attempts to unify, that generally have disastrous results. You just need a special breed to get someone who can really function up and down the stack, and every company should seek out, train, and develop such highly-empowered "tech special forces", for lack of a better term. It's tragic that so many companies don't realize this.
Microservices are a more sensible idea in big companies because they can be used (provided you have a sane architect) to create fairly easy to use abstractions that code in any language can use. Where communication and coordination costs are very high but there's lots of resources and programmers (of varying flavors) available it makes sense.
Within individual teams and particularly in small companies, where communication and coordination costs are very low and there are not a lot of people or resources available, "microservicing" everything is a brain dead idea. Introducing a network endpoint does not necessarily decouple things any better than a function interface would and it imposes complications and costs that a function interface doesn't.
So yes, there is a way to do "microservices" right, provided your understanding of "micro" is sufficiently macro and it doesn't just become an excuse for every little self-selecting group of developers to go off in their own world and ignore everyone and everything they'd rather not worry about all day. But in my experience, it almost always does.
If you have a team disciplined and experienced enough to do microservices well, you probably have a team that is dubious of the hype and will just start building reasonable software without worrying about how micro-servicey it is.
He was utterly oblivious to the fact that:
* In a different environment with lower coordination costs it is net harmful
* "decoupling" and "microservices" are entirely orthogonal problems and if decoupling is not done while you microservice the shit out of everything your problems are brutally magnified. Tight coupling over network end points is the worst. My time wasted with this problem actually still makes me rage.
Before I worked in a big company I was a MAJOR microservices skeptic because my experience was pretty much just with seeing them grossly misapplied on small teams. Now I am working in a big company I find myself recommending the approach simply because I often see them as a working solution to often extreme people and resource coordination costs that in my previous lives in a small company, simply didn't exist.
I actually kind of shocked myself and had a mini identity crisis when I started doing that. I'm not the microservices guy! I'm the skeptic! Nonetheless, I had that problem and the solution made sense...
In a small company you tend to just have a never ending series of problems, people working at close to peak efficiency and your main problem isn't coordination but just not having enough time, people or resources to solve your problems. The introduction of unnecessary network endpoints everywhere is just going to slow you down and cause problems. If you also have an architect without a deep appreciation of loose coupling that's a recipe for a disastrous rube goldbergesque pile of steaming shit.
In a big company there's often tons of developers all over the place, mostly working very inefficiently and a lot of them will speak different programming languages. Your main problem isn't a lack of resources or time - both are plentiful, it's trying to prevent the rigid and slow moving bureaucracy from preventing you getting anything done and breaking down work into clean chunks so that individuals and teams have enough (but not too much) to work on and the problem gets solved. In that situation a few more network endpoints that are strictly necessary from a technical standpoint is a minor price to pay.
Strictly speaking this isn't 100% a big company / small company thing (my big company environment isn't necessarily like yours) but I'm convinced it's 100% about coordination costs.
Isn't this the way it should be, to some extent?
Developers make the "what" and "how" decisions for the application, sysadmins make them for the environment. Containers provide a clean separation of responsibilities. Decision-making and implementation are aligned with the feedback and results.
All those other container technologies are still there and available for use. from jails to zones to LXC/D...
Docker brought some useful additions to the ecosystem like Docker hub and also helped adoption of containers overall by making easy to use tools.
There's absolutely nothing to stop people picking and choosing what parts of the container ecosystem they use, which is one of the nice parts about it.
Just need to run a container image, use runc. Want a daemon to manage container lifecycle , use CRI-O or containerd. Want the additional features like network management and hub access, use Docker.
Are these docker terms? If so then perhaps my despair is illustrated.
Also, docker is not the only solution for Linux containers; rkt is also developing, and Intel had its own interesting approach.
Ultimately Docker is just a set of linux features bundled together in a nice package, so you can learn the underlying features (and their pros and cons) without having to actually use Docker, if you don't want to.
And that's where it gets kinda scary. In Zones/Jails, you started with as little shared as possible. A jail/zone is a fresh isolated environment. With Docker (and really all Linux container implementations) you have hack-and-slash isolation going on. It's a loosely coupled collection of namespacing mechanisms that relies on whatever spawns the container to coordinate all these mechanisms. By default, they all will happily put the process in a shared space.
A "container" is simply a process that has some extra annotations on it, which various systems understand to represent some type of "namespacing" or information about the domain in which the process is allowed to operate. In Docker, these are implemented via the Linux cgroups mechanisms, which offers these and various types of annotations, including annotations that the kernel understands to constrain resource usage, etc.
I don't know the details of the kernels well enough to discuss this next part in detail, but Linux containers implemented via cgroups are generally considered a "weak" security barrier, i.e., there are ways in which the isolation leaks out. illumos calls containers "zones" and is understood to have a stronger security/process isolation barrier, and I believe the same is true for FreeBSD jails, etc.
"Containers" are a very old idea. Docker is an attempt to dumb them down for the masses by pretending that they're actually virtual machines and making it easy to "push" or "pull" the zip files^W^WDocker images. This assumption has caused many terrible container-based deployments.
Most of the people running around tooting the horn of Docker and Kubernetes no doubt maintain these misconceptions and very superficial comprehensions of what's happening under the covers, and that's why they believe it's so magical and simple.
Engineers who are trying to solve actual problems in a consistent, repeatable, and reliable manner will avoid any unnecessary complications -- including Kubernetes and/or Docker -- unless they are truly warranted by the unique circumstances in play, rather than seeking them out because they need a new flashy bullet point on their resume.
That seems to be its Achilles' heel. Instead of having packages maintained by dedicated people (as with package managers like apt), you get a program that comes together with a collection of dependencies and libraries that aren't maintained by anyone.
The average developer maintains their own program but doesn't have the resources to backport every bug fix and security update to the specific version of every library they use, and neither does anyone else when every program is using a different version of every dependency.
Which is actually a fairly necessary thing, because, as is, many popular tech stacks simply don't have a built-in way to do this, and, even when they do, it's often not all that self-contained, and they all work differently. As a concrete example, I spent all of yesterday trying to get up and running with some sort of Spark/Scala notebook system running on my host machine, and it was an exercise in pure annoyance and frustration. This morning I gave up, rolled back that whole mess of packages and homebuilt binaries and environment variables and whatnot, and (re)installed Docker so I could give Zeppelin's Docker image a try. I've been working happily ever since.
So, I'm grudgingly accepting it. I also wish we had something less hacky. But tech in general is so duct taped together at this point that I'm not even sure that's a possibility. It's the solution we need, not the one we deserve.
But is it really the direction of the industry?
I think you'll agree that that the "industry" exists solely because software it is able to deliver "value" to individuals and companies, irregardless of tools used. Yes, we're on HN, and we're far more likely to be exposed to the latest and most shiny toys and buzzwords, but in the end, you just use the tools that you like, and can get shit done in an efficient way.
One last thing. When I was studying Physics, not a semester passed where a professor wouldn't mention Occam's Razor [1]. Maybe as CS matures with time, people will start thinking in a different way.
* increased complexity if you have a simple setup; for example if your deployment now consists of copying just 1 file or 1 folder and starting up an app/service, then using containers will include additional moving parts in your setup
* if you already have legacy tech, migrating won't be straighforward, especially for stateful applications (think JSF, ASP.NET Forms, etc.)
* as jeremyjh pointed out below, unless you have a specific strategy for this, you will have duplicated dependencies in containers, which need to be checked for vulnerabilities; you can't just apt update && apt upgrade on the host system from just the security repositories and call it a day (more or less)
* an extra layer that comes with its own bugs
Maybe I need to enable it in containers too? I have no idea how to use it on (containerized) Alpine Linux though.
Even without unattended upgrades, finding the list of `Dockerfile`s and `docker-compose.xml`s that might be affected by a new vulnerability sounds more complicated to me. Until now it hasn't been that difficult but I'm still a bit nervous if I somehow missed some vulnerable images.
That's not a container specific problem. That happens even when you don't use containers.
Making maintenance and updates harder is definitely a container specific problem.
Over-simplicity becoming complexity. You used to be able to write an app in C, pre-build it for a dozen platforms, then run it on a dozen platforms. Now you have to run everything on one platform (a virtualized containerized Linux environment) because anything else is literally impossible. You can still run the app anywhere - it just has five or six layers of virtualization, abstraction and tooling now.
More complexity. Containers are supposed to just be processes, but you can't run them without preparing the host OS first, and you can't really use them en-masse without a complex scheduler and tooling.
Nobody mentions it, but be careful your disks don't fill up just from deploying regularly. People think containers "cost almost nothing", and then all of their over-use of containers results in resource drain from the 15 million things necessary to configure and build and deploy and manage the things across distributed clusters. It's more resource-intense than people advertise.
The container is supposed to be this read-only isolated self-sufficient jail, so good luck getting anything out of it without a bunch of hoops. You seriously need to specify 50 different config values just to do simple things like networking, writing to a persistent disk, picking what to run or stop, interacting with other apps.
People say "a container is just another process" - except each one may be using an entirely different Linux distribution worth of software dependencies. And you still have to track what containers you're running, because you will depend on X release of Y container.
Functionally speaking, containers are almost like a VM inside a VM. It gets that obtuse.
I don't know that devs are actually saving time using containers vs a single dev platform. I think the problem all along was nobody knows how to maintain platforms, so they literally create and throw away whole platforms every time they build an app.
Most of these comments are about Docker, but not necessarily specific to it in practice.
Apparently it is not only unnecessary, it is an anti-pattern: https://www.socallinuxexpo.org/scale/15x/presentations/confi...
> You used to be able to write an app in C, pre-build it for a dozen platforms, then run it on a dozen platforms. Now you have to run everything on one platform (a virtualized containerized Linux environment) because anything else is literally impossible. You can still run the app anywhere - it just has five or six layers of virtualization, abstraction and tooling now.
I see more and more "here is a Docker image!" things pop up for tiny shell script utilities on github. It's like autotools with 100x the complexity and "portable" to exactly one platform. People have no clue what they are doing, they just want any excuse to use Docker.
> The container is supposed to be this read-only isolated self-sufficient jail, so good luck getting anything out of it without a bunch of hoops. You seriously need to specify 50 different config values just to do simple things like networking, writing to a persistent disk, picking what to run or stop, interacting with other apps.
This is my biggest pain point with containers, and where I feel Docker marketing is lying the most. Docker is pitched as making development easier, when in fact containers significantly impede debugging. You even need special tools (like https://github.com/emacs-pe/docker-tramp.el) to do the most basic tasks like look at files. The marketing hype is that all you need to solve production problems is a log server and a few line graphs of metrics (better hope you instrument your app code for the right metrics...). Add in microservices and now you need distributed tracing too. This extra infrastructure is a ton of work to set up for small apps.
If it's immutable containers we're talking about, then patching needs to be handled by your image build system and containers need to be restarted in order for the patches to be applied. Yes, you'll need an image build system and at least one staging environment.
Applications need to be designed with immutable containers in mind. This makes them a pretty niche technology.
LXD, which offers mutable containers, doesn't suffer from those last two points. Of course you don't get the benefits that the immutable design brings.
afaik windows containers only support a subset of the full win32 api, so it's exactly the same as running actual windows. that said, simple asp.net sites should work fine
Microsoft are obviously investing quite a lot of effort in making the windows container ecosystem more robust, as well as improving the experience of running Linux containers on Windows.
However, layers of abstraction should also bring benefits wherever they're used. Docker and Kubernetes go hand-in-hand with distributed software or "microservices." If you don't see any benefit to that architecture, you're likely going to introduce unnecessary complexity into your infrastructure with containers.
The more deployments of individual software applications you make, the more you can gain from containers. And the inverse is also true - if you rarely deploy new versions or updates to your software, using containers will require tackling complexity that doesn't benefit your use-case.
When you launch processes on Linux, you can choose for all the processes to share memory with each other (in which case we call them threads), or the processes can have separate memory. We call this capability "virtual memory", and it allows processes to pretend that they and the kernel are the only software running on the physical machine. (For example, every process might think that it can write to the memory address 0x08048000, but in fact that address is virtualized and mapped by the kernel to some different physical memory address.)
Though we take it for granted, this memory isolation between processes was the first form of kernel namespace. Modern Linux now has many of these namespaces, which can provide a virtualized view of the file system, network, IPC, uids, and so on. "Containers", then, are processes that employ many of these namespaces to provide isolation from others on the system. (To extend the analogy: just like regular processes think they have their own memory, containerized processes can have their own file system with a distinct /tmp, distinct process id namespace, or namespace of users that exist on the system. Every process can run as PID=1 if it wants, within its own PID namespace.)
These namespace primitives, from which containers are made, are present on modern Linux systems whether one chooses to employ them or not. The primary downsides to consider are efficiency of the system, and the complexity of managing it. If your system (CPU & OS kernel) implements virtual memory efficiently, then there isn’t much downside to using it. The same is probably true for most kernel name spaces, though file system and networking virtualization can have a performance hit if not configured with care.
Kernel namespaces are a core feature of Linux and can be taken advantage of without using technologies that are specifically "container-oriented". For example, consider systemd: it provides a comprehensive set of parameters for employing namespace features in processes that it launches. Parameters like PrivateTmp=yes or PrivateDevices= [1] will virtualize just one part of the system (/tmp or /dev and such, respectively). PrivateUsers= and PrivateNetwork= allow you to launch with a new user or network namespace. It's really that easy to employ namespaces: add a line to your application's unit file.
When you look at container technology as a set of system primitives, then it makes sense to consider employing them individually. They don't have to be used in an all-or-nothing way. You might give your daemon access to the regular network with a private file system, or you might use the regular file system too while virtualizing just /tmp and /dev. You can isolate processes in whatever way makes sense, and share whatever resources make sense. If a process doesn't need access to the network, and doesn't need a shared /tmp with the rest of the system, then there's little downside to giving the process its own namespace. Indeed, I expect it will increasingly be considered a best practice for services to isolate themselves according to the principle of least privilege. This reduces the chance that components will interact with each other in unexpected and undesired ways, and may have some security benefits.
(If the question is about using Docker or some other specific technology as part of a development and/or deployment process, then that's a different topic. Specific technology suites have tradeoffs beyond those of kernel namespaces.)
[1] https://www.freedesktop.org/software/systemd/man/systemd.exe...
This shouldn't be underestimated, though!
I agree that efficiency is a no-brainer for almost all purposes, especially compared to other forms of virtualization like KVM.
But: Managing the complexity can be tough, it sometimes feels like creating all low-level firewall rules by hand for a large network. Well, for firewalls you have stuff like firewall generators, in the context of containers these are projects like Docker or LXD. But they share the same downsides: You may have trouble to see what happens behind the scenes.
The largest risk here, however, is not the working time spent to learn and apply that stuff. It is a false sense of security where your complex system might, in the end, to have holes which you assumed should not exist.
For example, not many people are aware that if the user can start docker instances, they are effectively root.
(One might argue that this is more a failure of docker than of containers. But then, why do people use complex monsters like docker in the first place? Because managing plain containers is a pain and other complex building around containers have their downsides, too.)
[1]: Run within Kubernetes: https://github.com/kubernetes-incubator/cri-o [2]: Run Standalone: https://github.com/projectatomic/libpod/blob/master/docs/pod... [3]: Build: https://www.projectatomic.io/blog/2017/11/getting-started-wi... [4]: Move: https://rhelblog.redhat.com/2017/05/11/skopeo-copy-to-the-re...
Process memory has traditionally been an "implementation detail" of the process. It typically doesn't need to be shared, the addresses do not need to be fixed, etc. (Shared memory is an exception of course).
Resource like the file system or network, on the other hand are mostly shared. It does matter to the outside world through which port the program is reachable, it matters where it puts its files, etc. Again, there are exceptions of course, like temporary files.
Now the question is how can we pull out the configuration, like "what port must the program listen on?", and decide these tasks centrally. The container solution is (if I understand correctly) that the program will always listen on port 80, but the container configuration diverts that port to be accessible to the outside world as 8080, etc.
I predict that this is going to be a failure in the same sense that OOP has been a failure (if you agree with me on that point). It becomes a tangled mess of layers and indirections, and incredibly hard to debug.
Instead I think it is much cleaner to make the configurations like, "What port should I listen on?", "Where can I put my temporary files", easily accessible to the program. The program should not be required to define its own configuration for all the problems which containers want to deal with today. Instead it should import this configuration from a central location through a well-known interface.
There must be a simple convention where these decisions can be configured. The configuration must be easily accessible to A) the system, B) the program, C) the administrator. It's been customary to put configurations in text files for a long time, and that's mostly good. However what's been missing is that most programs have had their own configuration file written in a custom syntax and stored in a location that are not easily known by the rest of the system. The system is not really aware that the program is configured e.g. to listen on port 80. And the administrator needs to learn new configuration files for every program.
I'm not proposing a central registry where all the cruft of the system should accumulate. Instead, just for the concepts that are shared by many programs -- the resources which containers virtualize today -- should be configured through a system-wide API.
I'm looking forward to flatpak, and I currently use firejail + liberal use of `chromium --user-data-dir=~/.config/chromium-<website>`.
Because web-app based software and web-services I frequently interact with (slack, gitter, discord, twitter) should absolutely be separated from my general browser, and from each other. Most methods of web tracking and phishing/XSS attacks will be neutered in this setup.
And because my browser, and all of these separated services, should not have unfettered access to my system and personal files.
I don't think I will bother with containers for general software, but web services are an attack vector and should be treated as such, even at the expense of complexity over simplicity.
Frankly, the notion of simplicity was out the window the moment we started using the web as a development platform.
I was looking at containers for maintaining a sane and replicable environment for software development without interference from changes on my host machine, yet most docker guides do not go into that workflow.
Difficult to find crucial info, like can linux containers run on windows and vice versa, licencing requirements and so on at a quick glance.
Redhat is still refusing to admit that containers highlight systemd's weaknesses.
Containers benefit from simple init. The systemd argument has become so religious iside RH now they are blind to the fact that, for all systemd's benefits on a laptop linux install, it complicates a container.
Its affecting their position on containers badly, because they have no middle ground between "full os" and "1 bin per container". Everyone else can see that sshd in a container is not _ideal_ but a handy stop gap sometimes, as long as you dont then stuff the container with stuff you dont actually use. Redhat are blind to that and, as a customer, its horrible. They refuse to "support" simple RHEL OS containers, i.e. a container without systemd.
Its a full RHEL vm or Docker, or your warranty is void. This makes migration to containers hard for us. Its probably counter productive for Redhat because when we do go full "bin per container", RHEL is out of the picture entirely.
docker, however, is more about the abstraction of complexity stemming from the industry's overall drive toward scalable, distributed applications and development teams. it's more analogous a developer saying, "why do we even use virtual machines?!? bare metal is way faster!"