Why doesn’t anyone weep for Docker?
techrepublic.com
techrepublic.com
Yeah, Kubernetes initially learned a ton because of Borg and Google's deep investment into containers dating back a very long time. But, arguably, Kubernetes is successful because Google Let It Go. Its a true open source project, with governance by a wide number of industry advocates, underneath the Linux Foundation.
By comparison, Docker is a VC-backed profit-minded startup. Of course it was going to lose this race, for the same reason Windows isn't the dominant OS in the cloud.
Fundamentally: You can't build a hyperscale startup based on a technology. It doesn't appear to work anymore. The best case is the Docker/Kubernetes or Oracle/Postgres/MySQL case: someone else does it, maybe better, open sources it, community forms around it, you're toast. The worst case is the MongoDB/AWS or Elastic/AWS case; a cloud provider copies you, probably does it worse, but its cheaper and more integrated with the cloud, so they still win.
Docker was doomed; they could have been a very nice business, but the issue is taking on huge valuations and capital, scaling like mad, and then finding out you have no ground underneath your feet to support that valuation.
Which is really a shame.
All 'True' Unixes are just closed source versions of what originally were open source operating systems. By taking copyright seriously and having the misconception that there is intrinsic value in 'IP' they effectively sentenced their operating systems and investors to a long term grave.
But I doubt most of them feel bad about it. They got their millions and their nice fat retirements. It doesn't matter now if customers now view their once dominate systems as a sort of technical debt cancer.
If it wasn't for the destructive power that copyright has on technology and the demands of board members to monetize Unix.. we would all be using BSD right now. Unfortunately the tech people from 30 years ago didn't understand the power of 'letting go' and thus allowed them to destroy Unix.
As far as docker goes it has a lot of momentum as a daemon and it'll probably stay that way for a long time despite some technically superior solutions for running containers that have cropped up in the past few years. The newer container solutions just don't have the community backing them and that matters.
https://en.wikipedia.org/wiki/Single_UNIX_Specification#Curr...
Solaris’ first release was in June 1992, with first use of the name in marketing materials in September 1991 (https://en.wikipedia.org/wiki/Solaris_(operating_system)#His...)
Linus’ famous message was from the same time (September 17, 1991)
Calling Linux an open source version of Minix is more appropriate, but it still wouldn’t be a good example of this.
Minix isn’t dead. It moved to a BSD license, and is deployed in millions of hundreds of millions of Intel CPUs (https://en.wikipedia.org/wiki/Intel_Management_Engine#Hardwa...)
And at roughly the same time, IIRC, when Sun decided to migrate their Unix from a BSD flavor to a SysV flavor, which came to be called Solaris, they also used some GNU bits. Which might explain some similarities between the two.
However, the point I'd like to make is that it was clear back then just how much was contributed directly by the GNU project, and also how favored GNU's GPL was by developers who wanted to contribute their work.
GNU really did seem like it was everything that made the Linux kernel act and feel like a Unix system, though I had no proper appreciation of that at the time.
MINIX has been BSD licensed since 2000, BTW.
Oracle is very much okay still with superior technology, despite open source, and community, if we are talking about PostgreSQL.
Before Kubernetes was the dominant container tech, they were pushing Swarm but I remember being confused about where Docker “standalone” stopped and where Swarm began. Perhaps it would have been better as a separate tool with a more clear open core model?
Then there was Docker Hub, whose UI was never great and which always seemed light on features.
I don’t recall seeing any kind of container introspection tool from them for a while either, despite others coming out.
Meanwhile, they represented a threat to the cloud providers if you could truly run anything in a container on any cloud. But the cloud providers all neutralized that threat by the classic “commoditizing the complement” strategy where the Docker cluster and registry tech were all either open source or commoditized.
Once Kubernetes emerged as the winner and de-valued Swarm while the cloud providers all offered their own Kubernetes and Docker registry offerings, I’m not sure how much more profit there was for Docker to claim.
Honestly, startups are hard. Sometimes really hard. It’s hard to know if a different team would have gotten different results in this space.
Also .. the K was hardened at Google is BS. The ideas, maybe. But I am quite skeptical about the amount of prod internal google code went into early K (pls don't point at the Borg paper .. I'm talking about actual working code). I recall doing a deep comparison of swarm vs K circa 2015 and Swarm was clearly superior in both design and implementation. Today, K is better and has an ecosystem .. maybe the issue isn't that core Docker containers played nice with opensource .. rather .. swarm should have focused much more on playing well with others.
One point of contrast is Hashicorp .. they are in the workload orchestration and mgmt space and seem to be doing really well. Kudos to them!
Swarm was 2-3 people in the early days, without as much strong opinionation about what exactly they were building, which meant while it was a tighter, simpler system, it couldn’t evolve as easily.
I’m obviously biased - I was the first non googler to have commit on the repo. But it’s much easier to build something when you know upfront exactly what it looks like and you have a set of committed and experienced engineers with good leadership.
How well was your PR received?
I'm a huge fan of Docker, I've actively taken part since the early days, attending meetups and using it actively day to day.
Unfortunately, when I brought several issues to GitHub, or +1'd other people's issues that were affecting the usability within our company, the attitude was very much "f* you and your problems" because Docker want things to be one way and that's how it'll be.
There were issues raised 4+ years ago and are still open, for solutions to problems that would have mooted a need for us to use something like K8s (which doesn't work anyway for our requirements).
I believe Docker locking the community out of valuable features had also done harm and (possibly) failed to be the monitiser they'd hoped for.
After so long, I no longer go to Docker to solve problems that could be solved in Docker (secrets anyone? without the "hacks"), and just look towards the other tools solving the problems.
I'll continue to use Docker, but I don't consider it a friend.
Some of the issues are solved by the likes of K8s, but many of us don't want or need K8s for our use cases. Other issues are resolved by other told, and yet still others are only resolved with effort (you still can't expose/bind a port range, e.g. 10000 UDP ports in Docker without killing your server).
Are you talking about k8s or do you mean something like bitnamis sealed secrets?
I now maintain a daemon who's sole job is to undo Docker's meddling with resolv.conf until I can get the bandwidth to explore migrating to Kubernetes.
It wanted to ‘mv’ over resolv.conf but couldn’t so we had to run it outside the container because it was proprietary and we couldn’t readily modify it.
In docker its easy. Download Docker. Start container. Install container manager like platformio. Done.
But true to the article. Docker seems very hostile towards the community and towards getting revenue. If i think about Kubernetes and Revenue i hear IBM, Red Hat. And i am too cheap of a person and too small of a customer to ever need those guys. So i will still keep using Docker.
And because i know more about Docker i will probably try to use it at work as well. Easy as that.
But i am open to suggestions.
This will actually finally change soon'ish with Windows at least, I think as part of WSL2. Not sure if anything is going to change on MacOS.
Kubernetes is like an operating system, Docker is like a format for the executable files.
There is no problem with servers. We're running Kubernetes in production on a single Linux machine, essentially using Kubernetes as an alternative OS.
Mainly because the point is non-technical; while I'm not sure that Google is exactly a saintly organization, things like the Summer of Code and giving K8s away have procured some goodwill for them.
The whole article is over the head of Joe User (me) who is competent to use Docker but nowhere near deep enough to contribute code to it.
I can duplicate a service I'm running 1:1 with a new version for testing in 2 minutes, I can tear it down in 2 seconds. I can roll back changes in one command, I can wipe the server and reinstall everything from scratch in 25 minutes.
The environment is completely reproducible, and I can with a single command see every config that applies to a service.
All the usual deployment trouble is gone, no more weird setups and config situations. All the weirdness is nicely encapsulated.
For development, microk8s is actually very nice: https://microk8s.io
You'll never ever need either Docker or Kubernetes or even the latest and greatest javascript frameworks.
I started running a server before I knew anything and that server is still purring along happily.
But if you're ever targeting an enterprise, either as a freelancer or as an employee, those words are invaluable in your resume.
It of course helps if you actually know about those technologies!
And when you do get to learn them you'll wonder why everyone is coming full circle!
I agree with "no need for k8s and the latest JS frameworks", but strongly disagree on not needing Docker. It is extremely useful for setting up separate development instances for your projects - no matter if you're doing PHP development with N different versions of PHP (as some sites may still be stuck at 5.6 while others are already requiring 7.2 due to Composer dependencies) or, worse, nodejs and Java where each project will have its own requirements for node, Tomcat and whatnot.
I personally set up one mega-container for each project which runs all the services required - mysql/pgsql for the database, apache as frontend / mod-php, if needed Tomcat - and can simply shut them down when I'm done working on a project instead of having the databases and servers all consuming memory and resources all the time.
Still, I run it all on one server using Docker so there is a medium :-)
One to two hours per project multiplied by potentially one or two QA/UAT environments, and this is a tiny blip in the total time spent developing a solution. I think it would take me a few years before I paid back the debt of learning docker/kubernetes for it to actually start saving me time as a freelancer (or if multitenancy suddenly becomes out of the question).
Great thing about Docker is a commit is all that's needed to reproduce everything since I also use terraform to deploy the infrastructure.
We keep on using plain old VMs, while watching everyone rush into containers fashion.
VirtualBox feels like installing a regular computer. It takes a long time and is a lot of manual work. If you want to change something you login on the existing VM until you reach a point where you no longer remember all the changes you've made over the years. The machine is unclean.
I realize that there are solutions out there for automating VM deployments but Docker did a good job of catering, and perhaps marketing, to developers.
I have more experience with VMware and Hyper-V, as type 1 hypervisors, VirtualBox always felt a bit underpowered.
Unfair comparison. People running on VMs usually rely on configuration management to do the install+config part. Think of stuff like Ansible as a Dockerfile for VMs/bare-metal.
I use Ansible all day every day -- it's not comparable container tooling in the slightest.
Only if you are not actually cleanly managing containers. With containers you just pile the shit into them, close the doors and say "Oh look, we have a clean surface!" which is certainly fine in the dev.
If it is fine in production, then it does not matter if the shit is in containers, VMs, dedicated servers, etc. Container may as well be curl http://mylservice/containername.img followed by dd if=containername.img of=/dev/sdb ; set-boot-flag-sdb ; reboot
> No config management is all-encompassing and so without extreme diligence you will lose the state of your servers over time.
I'm not sure exactly what you mean by "all-encompassing". Surely all of the popular config management systems have 'escape hatches' to let you do anything you could do from a shell, which seems pretty "all-encompassing" to me. But if you mean the config management system don't completely enforce a specific 'state' for managed servers then of course you're correct. Containers don't do that either, beyond periodically replacing running containers with new ones created from a base image (and even that's almost certainly not perfect either).
Unless your container hosts are entirely managed by someone else (and probably even not then, in the fullness of time), you're always going to "lose the state of your servers over time". There's (almost always?) some state somewhere that has to be explicitly managed and thus requires, generally, "extreme diligence".
I do agree that containerization and config management are very different but, like everything, it's 'just' another set of tradeoffs to be made, hopefully depending on one's actual or expected needs and wants.
Most of developers that are suddenly becoming more productive with docker do equivalent of a full install of an distribution that they run in production, call it a container and get the claps from management because "We managed to do it faster". Never mind that the surface for brokenness is now even higher than the surface for brokenness on a VM that the container is going to be ran on had the VM got the full install of of the distribution.
> VirtualBox feels like installing a regular computer. It takes a long time and is a lot of manual work. If you want to change something you login on the existing VM until you reach a point where you no longer remember all the changes you've made over the years. The machine is unclean.
That's because in this approach no one bothered to do equivalent of what one does when creating a docker image - write chef/puppet/salt/ansible baseline configuration so the new VM is nothing other than "base-VM + special config for this function".
It's not like containers are new technology, even on Linux; we were using OpenVZ a decade ago. Now they're just integrated into the mainline.
And we can use native containers if we actually need them, so I see such tooling more as yet another consulting wave.
Using docker images means that we can take the thing that developer A built and that "runs on their machine" (without necessarily having that developer on hand) and have developer B easily launch it on their machine without having any dependency or isolation or reproducibility issues, and after making some trivial changes put it in a new, fresh production server without having to ask the original dev how it was/should be configured.
It's just a matter of preferences, where you want to invest your time to learn etc... Kubernetes being the fully fledged "state-of-the-art"
I would just recommend the work of Stefan Prodan who's repository and blog are full of open and well thought devops work, even if you are just interested in K8 or only Docker
Do you need kubernetes?
I know the hype cycle is mad for copying big tech, but if stackoverflow can operate on a couple of IIS instances I’d argue that you almost never need kubernetes.
For cloud services this is probably a good idea, even for users to a degree if the provider doesn't already give you a fitting box.
But otherwise it is not a must have in my opinion. Maybe that is a mistake and the apps I develop today are not going to work in 10 years. Well, worst case: I have to be paid again.
You can get 99% of the way using a stable distribution and a configuration management system (ansible, chef and the like). It's much much simpler than running an orchestration service. I feel most people don't need containers and orchestration, just config management running redundant system designs.
True, but you can do that with plain system containers such as with lxd, rather than having that bundled with the huge paradigm shift that Docker comes with.
The paradigm of lxd is pretty much exactly the same as the paradigm of a regular distribution installed on bare metal or inside a VM. If you can operate a regularly installed distribution, then you can operate inside a lxd container. The commands to create and destroy lxd containers are trivial ("lxc launch ubuntu:bionic" for example).
> Kernel minutiae is not and the technical skill requirements is much higher.
I'm not sure why you think you need to know kernel minutiae, cgroups or kernel namespaces. Operating lxd needs none of that.
> I am pretty familiar with my application and the distro of the container in a user perspective.
That's all you need.
Dockerfiles give you a fairly easy and consistent way to express, "The runtime environment needs to have Python 3.5 and these packages," in a format that doesn't introduce too many concepts over and above the basic command line junk you'd use to manage your environment without Docker. If your stack requires multiple services, docker-compose gives you another fairly easy way to describe what all goes into that. And then it gives you a _super_ easy interface for starting and stopping all those services, keeping track of what you have running, all of that.
(And it's all fairly disposable, which is nice, since, as devs, we tend to break things. TBH, if Docker has done nothing else for me, it's that it's turned nuking PostgreSQL to get back to a clean install a 10 second process instead of a 30 minute one.)
It's not really that simple, and I've spent my fair share of time screaming at Docker for being flaky and having confusing under-documented configuration. (And I don't think I'd use it at all if I were working on a platform that weren't so annoyingly susceptible to systemic dependency hell. But worse is better, so the unix philosophy won, so here we are.) But eventually you get over that hump, and it starts feeling fairly easy to understand.
I don't know Chef, but I've seen Ansible used in production, and it just doesn't seem nearly so attractive. It could just be how it's being used, but it felt like there was this infinite regress of complexity where everything was tied to something else and you have to have been the person who built it to understand it, kind of like the bad old days when people were trying to put too much smarts into the database itself so they'd just become this rat's nest of triggers and whatnot. I'm sure it's not that bad. . . but my initial impression was that Docker is great for scratching a developer's itches, but slightly sucks for ops, but maybe is still worthwhile there if you're dealing with microservices or elastic scaling or something like that and you can use Kubernetes to smooth over some of the flakier bits. Ansible is much more for ops, and does a great job there, but I don't see it scratching many dev itches at all.
It starts with a Dockerfile, which is a limited shell script, and it does not get any better beyond that. Shell scripts are simple, I'll give you that, but please don't sell them as some magic bullet. Dockerfiles are no configuration management system.
I've seen my fair share of hairy chef, puppet and ansible in the wild. Don't read into config management from those. I've also seen beautiful ansible installs, which deploy from dev setups all the way up to full infrastructure setup and deployment with blue-green deployment.
Y'know, we might violently agree. That is a much more concise statement than my rambling attempt to explain why I think Docker is so much more palatable for development workflows.
You're right, it is no magic bullet. And I misspoke when I said "configuration management"; I forgot that that's a term of art in operations. By "management" I really just meant "stick it all in one or two files so I can get my checklist down to one step, and manage shared packages in a way that's at least a little bit less kludgey than simply abusing environment variables." So I find that it save some yak shaving, and for that I can deal with it under certain circumstances.
I actually hate using it for deployment or production config management, because IMO it seems to do a crap job at it. And it does a crap job at it precisely because of the features (or lack of features) that make it convenient for development. Even using it to manage our integration tests' runtime dependencies is kind of a hot mess. But I'm willing to concede that, together with Kubernetes, it might be nice for cloud-native elastic scaling microservice-y stuff, insofar as it seems to be popular for that. I don't actually know firsthand; I'm allergic to complexity, so try to avoid building things that way.
This! I come from the embedded world with a bit of webui and having yocto for the embedded reproducibility and docker for the infrastructure, I am so happy. No more uncertainty when moving to another machine or upgrading my DEV machine. Nope Everything running right everywhere. Some scriptology and I had a full boot up from tftp for kernel and an ephemeral nfs from an ext4 master, and I could reliably make full system component tests all the way to web browser experience validation.I even had an autopilot simulator spawning ephemerally in a docker. Restarting the containers gave me a clean env again. Pure piece of mind and productivity. The initial investment was big though.
Regarding the investment it is an embedded industry problem. There is very little re-use. My experience is that we are an industry of wheel reinvention, where all of us are deep experts so we roll everything on our own. Maybe against myself I speak as I indeed developed this harnesses even though other solutions exist. My quip with what exists is that Intel absolutely owns, or used to own yocto and embedded tooling open source projects. This lead to very crappy code that was Intel specific being half merged into upstream. Mind you their featues are really not working,and were accepted because Intel is a gold yocto project sponsor. When you try to remove the broken code to a more sane one, you cannot because then the actual good rules of making small changes will bite you back and your changes will not be accepted. So you need to wait for Intel to cave in to remove their broken features. I digress sorry :). Even so Yocto project is a great step towards embedded projects productivity and knowledge re-use. https://www.reddit.com/r/embeddedlinux/comments/bk8a8k/yocto...
Builds use the dind images on gitlab's runners to build an image and push to their container registry.
For deployments I have a host with a personal CI runner instance on Linode's smallest instance type which can access a user on the "production" host when SSHing over a private network, and has the docker-compose command allowed in the sudoers file. Then it can run docker-compose up to deploy. The key for this is passed to the job via gitlab's secrets UI so someone getting read access to either of my hosts wouldn't be able to do anything.
While people will rightly point out that this does mean the CI builder effectively has root on the "prod" host, for a side project it's enough for me. I might investigate podman/buildah some weekend when I have time as apparently that allows for rootless container launches.
Compose and Swarm can take you pretty far, but TBH, it felt like Docker gave up on them years ago, even before k8s "won" the container orchestration war. A real shame :(
Ten years ago you started with PHP + memcache + MySQL running on just a physical box running linux. No Docker or Kubernetes. No virtual machine for what it is worth. Then you split it in multiple PHP frontends with a load balancer or MySQL master and slaves as the traffic demanded it.
I think a lot of systems these days are over-engineered and over-paid from day one. It may cost you 10-20 times what you would if you keep the "old day" approach.
That said Docker and Kubernetes are nice. They give you a lot of flexibility. But the most important part, I think, has been that they consolidated the shift from "pet" to "cattle" servers that simplifies a lot the sysOp and sysAdm work.
I’m not attacking docker, however, I can see why you would want containers. We still haven’t found an efficient usage for them at my place, but we never need to spin up more than one instance of our software. I think docker can sometimes be a way to cheat unsafe software around operations, but that’s more of an anti-pattern than an issue with docker.
Then again, I’m probably old and grumpy, but that sometimes has the advantage of not adopting techs before they are easy to use.
From my perspective k8s is really good for a company who has a team of experienced SRE who can manage k8s well and the company has a marketing team driving new development that needs to get to market fast
It is not cheaper than simple autoscale groups, it is not easier than rebuilding packages with jenkins ( rpm spec files or debian src rebuilds )
It is however the current popular framework so if you want to ride the wave learn it. Also learn how to migrate away from it as that will be a future role as several early adopters are pulling back out of it.
The larger issue is the simple fact automating cloud infrastructure is not trivial so abstract layers lets more people implement things without understanding the lower levels giving folks the appearance of having a complete framework.
It works great until it doesn't. Then it is interesting watching people try to figure out how to fix it.
Learn the lower levels of Linux and k8s will come to you organically
> the company has a marketing team driving new development that needs to get to market fast
At what point in the growth curve? Are your SRE shipping the new features, or running like a red queen to keep the product developers able to keep shipping?
What development process or cycle has overheated with friction that requires k8s? Where is that friction which k8s relieves?
Successful utilization of k8s, interesting metric, what is your criteria ?
>At what point in the growth curve?
I have seen properly staffed startups (in silicon valley) leverage the platform to pivot direction fast but overall I would say you need a large enterprise to support it. Ironically the large enterprise which could benefit the most wants to lay ITIL on top of k8s, call it agile and the methodologies conflict creating more issues than they had.
> Are your SRE shipping the new features, or running like a red queen to keep the product developers able to keep shipping?
Yes to both
>What development process or cycle has overheated with friction that requires k8s?
Upgrading k8s :)
>Where is that friction which k8s relieves?
What k8s relieves is once you get a pipeline getting product to production fast, the approval process from legacy ITIL groups being the problem ( I have yet to see an enterprise company use k8s properly though I have been to presentations where some claim to have done this )
If someone attempts to use ITIL and k8s you are going to have problems. Training people to NOT do this is the #1 issue in my experience.
> Successful utilization of k8s, interesting metric, what is your criteria ?
Sorry, what pain motivated using k8s, and using k8s relieved that pain.
> I have seen properly staffed startups (in silicon valley)
What is 'proper' for staffing?
>> Are your SRE shipping the new features, or running like a red queen to keep the product developers able to keep shipping? > Yes to both
Is it the appropriate use of a Reliability engineer's skills to develop or change arbitrary features? Or are you saying they are at least Engineers so they should be able to pitch in everywhere... CSS accessibility features or k8s config.
>> What development process or cycle has overheated with friction that requires k8s? > Upgrading k8s :)
This tautology keeps me away from k8s. Before k8s what pain in the process required using k8s to solve?
> a pipeline getting product to production fast
Does k8s make it fast? Does k8s make it possible? I think I'm deploying "fast", without k8s... But because you keep mentioning ITIL I suppose it's more about the infrastructure changes? Or are you talking about ITIL tension because k8s requires a more liberal policy than ITIL allows?
There is so much assumed context when talking about k8s, the only comprehensible part of these discussions is k8s is a rabbit hole to end all rabbit holes.
CTO / CIO / $SOME_C reads a blog and decides they want k8s is how it usually is introduced
> What is 'proper' for staffing?
Experienced C Developers who can do operations ( like google level SRE )
Summary of k8s : if you use the proper methodology to implement k8s and you are in a cloud framework you likely dont need k8s :)
What shines about k8s is it leverages the current container fad to ship code faster because devs like it. The current container fad pushes the burden of supporting buggy code to operations teams. IF the operations teams have issues they had better be highly skilled to figure them out
I have yet to see a company let requirements drive the choice to use k8s.
Let others run to Kubernetes, it doesn't sound like the right tool for you. That's ok!
For availability. If you have promised 99.9% or more availability to your customers you probably need some sort of redundancy or hot failover. Kubernetes is a good option to get that.
If you don't need 99.9%, simpler is better.
> Too cheap
You can get a single node Kubernetes cluster from Digital Ocean for $10 per month.
That's silly because a single node cluster doesn't have redundancy which is the main reason for Kubernetes, but it lets you get started. A minimum redundant cluster would have two nodes and a $15 load balancer, so $35 per month at DO.
DevOps is two things:
1. Applying the methods of modern software development (version control, automation, DSLs...) to operations (provisioning, config, deployment, monitoring, backups...).
2. Reducing silo barriers between devs and ops groups so that everyone is working together as a team, rather than blaming each other for poor communication and the resulting messes.
Then there are all the DevOps hijacking attempts, such as equating it to Agile or Scrum or XP, or insisting that it's a way to stop paying for expensive operations experts by making devs do it, or a way to stop paying for expensive devs by making ops do it, or a way to stop paying for expensive hardware by paying Amazon/Google/$CLOUD to do it.
No matter what your software-as-a-service company actually does, it will need to execute certain things:
- have computers to run software
- have computers to develop software
- have computers to run infrastructure support
You can outsource various aspects of these things to different degrees. Anywhere you need computers, you have a choice of buying computers (and figuring out where to put them and how to run them and maintain them), or leasing computers (just a financing distinction), or renting existing computers (dedicated machines at a datacenter) or renting time on someone else's infrastructure. If you rent time, you can do so via virtual machines (which pretend to be whole servers) or containers (which pretend to be application deployments) or "serverless", which is actually a small auto-scaled container.
Docker is a management scheme for containers. VMWare provides management schemes for virtual machines. Kubernetes is an extensive management scheme for virtual machines or containers.
A continuous integration tool is, essentially, a program that notes that you have committed changes to your version control system and tries to build the resulting program. A continuous deployment system takes the CI's program and tries to put it into production (or, if you're sensible, into a QA deployment first).
When you boil the Cloud, DevOps, CloudOps, SecOps, *Ops, CI, CD, Containers, VMs, and all the other technologies we've devised over the past ten years, you always end up at the basic building blocks.
You eventually come to the conclusion that all we're really doing with all these new tools is adding software layers on top of those building blocks in an attempt to make them easier and faster to consume.
And how have we done overall?
Not bad, if you ask me. Some solutions are overkill for most people (K8s is an example of over kill for a start up and even an SME.) But Terraform, Ansible and GitLab (CI) are something I'm currently developing a highly opinionated video training course on because I believe they strike the right balance of improving on prior experiences without taking the absolute piss.
I did a write-up on how I used it on my blog: [link redacted]
The workflow worked really well, provisioning Vagrant servers in staging and Digital Ocean droplets in production.
I moved away from Vagrant in favour of Terraform, but I agree Vagrant still holds its own and is a great choice (HashiCorp really nailed it, eh?)
Once your VM or container hits a complexity point above trivial, ansible is very much a useful tool for provisioning container states, and specifically for patching container images to, eg. include security updates.
...beyond that, as in, the intended use case of dynamically updating multiple live machines in parallel... dunno, I don’t use ansible for that... but it beats the hell of out having a single monolithic batch script to setup a container. I use it for that purpose all of the time.
For host based security patches (if I'm in an environment where the servers aren't managed), adding an item to the crontab in user data usually handles that, and again any fleet-wide changes would usually be propagated by updating the user data, pushing out the change and having automation rotate the fleet.
I believe in one tool to do one job really well.
Terraform is excellent at provisioning and managing infrastructure due in part to its DAG and HCL. On the other hand Ansible has been tuned over the years for managing configuration and the state of anything and everything from the OS upwards.
I also believe in using building blocks to get to where you're going, and these two bad boys click together quite well.
- DevOps is a peer with Agile and Lean. Scrum and XP are Agile implementations. Scrum doesn't prescribe ways to code, XP does.
- 90% of what people develop or run today should be in containers, and not because containers are great, but because of the DevOps patterns of IaC, immutability, reproducibility, homogeneous environments. Whether you run them on your laptop, a VPC, AWS Fargate, a K8s cluster, etc is dependent on your business needs.
- Continuous Integration and Continuous Delivery aren't so much tools as a practice, and they're more complicated to implement at scale than just using a tool. There are some great books on the subject.
If it helps, my core point is:
DevOps is the name we give to two philosophical ideas. The first idea is that the tools and methods of software development can be used to improve our ability to do operations work. The second idea is that siloing people with operational skill away from people with development skill is a terrible practice.
Along the way, I specifically denounced the idea that DevOps is a single methodology, or that some tools are more DevOps than others, or that DevOps makes prescriptions about what you should do. Those are all things that you immediately advocated.
Look at it this way: The Toyota Production System isn't about cars. It was developed specifically to produce cars as well as they could be, but it doesn't address "car problems"; it addresses business problems, production problems, workflow problems. It applies methods as practices in ways that are specific to the production of cars, but you can apply the principles of TPS to things other than building cars (as we do with Lean).
DevOps is comparable to TPS (well, Lean), but for software instead of cars, and it borrows from other systems, and it has a few of its own ideas specific to software.
> Along the way, I specifically denounced the idea that DevOps is a single methodology, or that some tools are more DevOps than others, or that DevOps makes prescriptions about what you should do. Those are all things that you immediately advocated.
I advocated using containers because they help reinforce DevOps principles better than alternatives. You don't have to use them, but that doesn't make them un-applicable to DevOps. There are different levels to DevOps, and one of them is "practices": particular ways of doing things that DevOps encourages, such as Infrastructure as Code, Immutable Infrastructure, Heterogeneous Environments, Continuous Integration & Delivery, etc. Things that containers are more useful at accomplishing than, for example, VMs.
You don't have to use Kanban to run a car production line. But it's more TPS than the alternatives.
Sorry but no. Container is __a__ way of achieving a small part of what you are talking about but not the only way.
Break it down:
- IaC: how do you containerise a load balancer? Terraform gives you infrastructure as code without containers.
- immutability: VMs, AMIs are immutable just like containers are (discounting the entropy that happens in every OS)
- reproducibility: Same, VMs, AMI, Terraform, Ansible all give your that
- homogeneous environments: Not sure what you mean by that, your Cisco or Juniper firewalls are not running in Docker so I am pretty sure you already have "heterogenous" environment if you meant that by what you wrote
I absolutely disagree this approach that we need containers for the reasons you just mentioned.
1) Terraform should be run in a container so that it will actually behave the way you expect, 2) containers are application environments built based on a Dockerfile, which makes it IaC.
> - immutability: VMs, AMIs are immutable just like containers are (discounting the entropy that happens in every OS)
True. But containers are easier and more portable, which is important to supporting the other aspects involved. Containers thus are a better general solution.
> - reproducibility: Same, VMs, AMI, Terraform, Ansible all give your that
Containers and VMs just... work. They're just collections of files. Very reproducible. Not 100% - you may need different guest drivers/kernels, different arguments to run your container in your particular system. But they're conceptually and operationally simple.
Terraform and Ansible are garbage fires of reproducibility and immutability. I could write a book on all the different ways these tools fail (most of it stemming from people trying to use them as interpreted programming languages, but also their designs are crap). There are whole frameworks built around Terraform and Ansible just to make sure they work right. They are overcomplicated, fragile bash scripts, and I'm quite frankly sick of using them. I think their entire existence is evidence of a huge gap in understanding how we should be operating systems today. [/rant]
> - homogeneous environments: Not sure what you mean by that [..] I am pretty sure you already have "heterogenous" environment
Those are opposites; homogeneous means "of uniform structure or composition throughout", heterogeneous means "consisting of dissimilar or diverse ingredients or constituents".
A homogeneous environment in a DevOps sense is when all environments have the same components and are operated the same way, and thus provide the closest results possible. This is incredibly important to prevent the classic "Well, it worked on my machine!" dev->production breakdown.
Homogeneous environments apply to lots of different things, but in the context of containers, they ensure that the environment the dev used to build the app is the same as what is in production. They also ensure that any scripts, tools, etc will use the same environment, if they are run in containers. I've wasted so much time in my career "correcting" heterogeneous environments in a bunch of different ways, whereas with containers the equivalent fix is "Please run the correct container version. Thanks"
The more systems you have, the more important this gets. At a certain point, the best choice is just to use baked VMs or containers for everything, everywhere, and containers are just so much easier, almost exclusively because Docker shoved so much extra useful functionality in. (I'll add that I do not necessarily like containers, but I do find them to be the most useful solution, because they solve the most problems in the most convenient ways)
Try focusing on "what do I want?", get a superficial understanding on how the tool works, then try to apply that knowledge to your search engine query.
For example: Say you know that docker has images and containers. That means it is somehow going to install an operating system into your operating system and make an image. Then you will copy your program into that image. Then you will start a container (an instance) based on that image. And this is basically all you need to know about docker to start searching for how to do things. Like "how to build a docker image?" or "how to start a docker container?".
Another example. You know that Integration Testing means running your servers and running tests against them as if they are in actual production and continuous integration is a service that runs your integration tests everytime someone merges a branch to a monitored branch in a version control system. From here on, you are able to look up how to set a monitored branch, how to create a build machine and how to scale it.
"I call it 'modern software development practices!'"
These tools for DevOps are no different than the tools and tutorials for developers. It's fine to copy-paste a tutorial that launches an entire webapp from scratch just to get your project off the ground, but if you don't eventually learn what that 10-minute autorun.sh script is doing behind the scenes, you'll fall behind.
Would you be willing to have a quick chat? I'd pay for your time, of course. I need to gather feed back from people looking to develop their skills into the CloudOps space and understand what it is they're looking for.
The immediate need for me is to package up a C#/.NET web app and its components (DB, etc.) into a container so that I can deploy it on any big-name cloud provider (Azure, AWS, Google Cloud). Now after reading through the comments in this HN post, I am not sure if I should choose Docker or something else. If you have any suggestion, I would love to learn. I'm more than happy to provide you with feedback and such (even for free) if I can learn from your tutorials. Thank you.
Pick one cloud provider first and write some setup scripts, i.e. scripts to build an initial (minimal) environment, 'from scratch', for your app and its components, e.g. create a new EC2 instance for the web app, upload your build package to it, etc.. Write the scripts so that any 'secrets', e.g. your AWS API key, are provided as either environment variables or regular command line arguments.
Then, still for the first cloud provider, write some update scripts, i.e. scripts to update your web app and its components.
Assuming your app, or its deployed 'instances', are relatively small and intended to serve a modest load, I'd suggest starting out treating the cloud servers or services as 'pets', i.e. entities you distinguish by name and for which you would be 'sad' if they 'died' (crashed or shutdown). At larger scales, it's often worth treating servers as 'cattle', i.e. a mass of nameless entities, but you probably won't need that at this point or anytime soon. (You'll know better tho.)
As for "containerization and other fancy, related tools that I've been reading on HN in the past couple of years", they're just like any other software – tools that can be used but aren't ever strictly necessary.
Containers are, basically, virtualized OSes, and they can be (very) useful. In my opinion, they're most useful as a way to bundle components of an app with an OS (and other OS components). That you can run those containers in different environments and be reasonably assured they're (mostly) identical can be a big benefit. But there are associated costs too (as with anything)! But if your app and all of its components can comfortably fit on a single (virtual) server, the cost of any changes you need to make to your app to run inside them are probably not worth the (currently) modest benefits.
And all of the other "fancy ... tools" are generally even more a matter of tradeoffs you'll need to make. Once your app, or (production) instances of it, are distributed over several, or many, servers, and you start adding things like load balancers, caching, separate search services, etc., then the benefits of the other 'fancy' tools will start to make more sense.
But, like with many things, it's good experience to directly run into some of the issues that containers and the other fancy tools aim to solve, and really try to solve them yourself with DIY solutions, before committing to use yet another program or tool to do it for you.
Of course, if you just want to learn those tools, and you'd like to use your own app as a 'motivate example', that's perfectly reasonable and valid too. I would recommend tho not to lean on those tools for your own immediate needs unless you really need them.
If you deploy a container to a Cloud provider, you'll need to first setup and understand the container engine as well. Not a bad thing to learn, for sure, but again start small and from the bottom upwards.
My tutorials won't cover containerisation because frankly I believe they're overkill for most situations. Sure they're fast and so on, but a slower golden image and a simple EC2 Instance in an AutoScaling Group is easier to understand, manage, and can be just as easily orchestrated.
They have admins that maintain their pool of servers.
They have developers that are fluent in the stack of their application.
They decide they need to have some of that cloud, containers, CI/CD stuff.
Turns out they need people who can write code that builds their programs, tests their programs, packages their programs, provisions cloud infrastructure, sets up that infrastructure, deploys the packaged program on their infrastructure and finally monitors its health and performance.
Most of their admins say they are not programmers, so it's not their job.
Most of their programmers say they are there to write Java/C#/Python/JS, so it's not their job either.
They find some people who don't mind learning all these things and call them their DevOps team.
In a not perfect, but generally just world this team disseminates their knowledge across both programmers and admins, making both aware of each other. Programmers now think about the infrastructure they need, admins now think about the workloads their infrastructure runs.
In an unjust world, you end up with three silos. Programmers say their code compiles on their machine, admins say they have installed the new server, devops frantically try to build some pipeline that deploys that code on that server.
You mean the companies we regularly fail every aspect of engineering?
dsr_ summed up pretty well how Amazon and a bunch of companies think abut DevOps. Coincidentally these companies produce the highest grade of software, tools, services etc.
>> Most of their admins say they are not programmers, so it's not their job.
What you are describing is the 90s approach to IT. These companies disappear really fast. IT is changing just like agriculture was changing long time ago. Toffler talks about this in The Third Wave.
Old approach: lets do everything by hand New approach: automate most of the things you can
>> In an unjust world, you end up with three silos. Programmers say their code compiles on their machine, admins say they have installed the new server, devops frantically try to build some pipeline that deploys that code on that server.
I migrated countless companies from 90s approach to CI/CD world, they never looked back. You just think that because there are late adopters this world is going to exist indefinitely. I do not think so.
https://devops.stackexchange.com/questions/653/what-is-the-d...
If you want to build a mini-VM using containers, LXC is a great choice. If you want to deploy software, easily, with CI/CD and [auto-]scaling, then containers are what you want.
The trouble with all these schedulers is they can't go from just one node (where scheduling and processes run on the same node .. and minikube is a hack; not a production system) to 100. You can't just setup a small k8s, and then add a node, and another node, and scale up. You go from a single docker system, to a big managed k8s system.
There needs to be more competition. It's the same deal with the dominance of systemd as the only system layer. Only the small startups seem to be using more lightweight stuff like Nomad, k3s, RancherOS (Rancher is mostly going the managed k8s solution anyway; even though they have their own k3s implementation).
A running k8s system can be okay, but there is a lot of room for improvement (in terms of making it simpler). Both DCOS and k8s seem to waste a lot of resources. Docker could have competed in this space, but everyone complained about all the bugs in Swarm and it never really went anywhere.
I did a writeup on container orchestration systems late last year:
https://penguindreams.org/blog/my-love-hate-relationship-wit...
Docker (the company) certainly helped to raise the profile of containerisation but they invented very little of it and did a poor job of implementing what they did do. Good riddance to them.
Sure, if you trust every developer in your company with the root password anyway, why not? That might be true at Docker (the company), I don’t know. Certainly wasn’t true at one company I worked at with 30,000 devs...
By the way, this problem does not exist with competing container tools like Podman/Buildah.
Surely you'd have proper release management where Ops teams would review deployment artifacts before deploying them?
In any case the notion of "the root password" seems weird, root passwords should be unique (even for VMs), randomly generated, and mostly not used; in most situations you'd use publickey authentication instead of passwords.
You can still mount filesystems as root from a container, if you have Docker command rights. In Docker's security model access to run docker commands on a given host == root, that's a design choice AFAIK, not an oversight.
It's perfectly possible to mitigate that issue, by restricting who can run containers and also ensuring that all containers specify and use a non-root user account (or enable user namespaces at the Docker daemon level)
Also, many early stage technologies don't prioritise security . For example, for several early releases of Kubernetes all you needed was remote access to a single port (10250/TCP) and you could get root access to the underlying host without any authentication...
Otherwise you can hard code a UID when creating the user in the Dockerfile but that means your containers aren't generally portable.
In the end, the path of least resistance is to run as root within the container and simply accept the security implications if using volumes.
That's what I am doing for local development setups with Docker.
See https://github.com/a2way-com/template-docker-laravel/blob/ma... and its README.
Much worse: Docker was never built with security in mind but the company kept pushing it as production ready.
They acted like you were bothering them and in fairness you probably were. Even today they wouldn't pick up.
When you start believing the hype reality becomes distorted.
> Fire and Motion. You move towards the enemy while firing your weapon. The firing forces him to keep his head down so he can’t fire at you. ... The companies who stumble are the ones who spend too much time reading tea leaves to figure out the future direction of Google. People get worried about kubernetes and decide to rewrite their whole architecture for kubernetes because they think they have to. Google is shooting at you, and it’s just cover fire so that they can move forward and you can’t
Docker played its role and introduced the majority developers to containerization. This is a major success for the industry.
The came up with this amazing tool, that lot of companies started using, but they did not have a business strategy on how to make money in a long term. They tried to keep up (Docker Swarm, Docker Hub Premium, Tutum, Moby, Docker Community vs Docker Enterprise etc). But at the end they just seem like they don't really know how to approach it.
Then I started tinkering with Docker Compose and for a while things were great. But after a while I started running into issues. Compose felt artificially crippled. No secrets? No health checks? Pushing me towards Docker Swarm?
Eventually I just sucked it up and switched to Kubernetes even though I think it's overkill for my applications.
As far as I can tell, everybody and his grandmother is using Docker. Why should we weep about it?
We actually used Google Cloud Platform's Docker Image hosting service, and that was expensive.
Yay!?
What else can we pay them for? All the stuff we need are available from them for free, except private storage. If they had a container hosting solution, we'd pay for it.
It actually seems quite cheap...we have something like 2TB of tags up there, and they don't charge for network I/O. I did feel slightly bad when a hidden crash looping pod set to always download lay undiscovered for a month...that's a LOT of I/O.
Not at all - thanks goodness.
Microsoft have done a load of work on getting containers running well on Windows servers and that work relies on Docker EE as the container runtime engine (you get a free Docker EE license to run on Windows servers AFAIK)
If Docker get bought up (by someone other than Microsoft), then that would seem to possibly place Microsoft's container efforts at risk...
K8s can run on any CRI-compatible runtime, and IBM/RedHat don't even want you to install Docker on RHEL8.
None of this applies to Docker itself. Docker is “just” a virtualization technology. Sure, Docker Swarm exists, but at this point it’s mostly used as a shimming UI for connecting the Docker client and daemon to the abstractions mentioned above, not a clustering solution in its own right. Swarm lost in the DCOS market. And the market for pure virtualization solutions isn’t anywhere near the market for DCOSes.
I still use Docker Swarm for small scale stuff, and am pretty happy with it - it's simple, easy to use and doesn't eat resources. But it very much feels like Docker have given up on it.
I'm particularly interested to know if there is anything simpler than k8s that's competitive?
Is this true? Isn't kubernetes "based" on work done at google but also a complete rewrite.
Can't the same be said for Git? Linux? Python? (That they didn't make the creator billions, and the creator is fine with that)
https://www.google.com/url?sa=i&source=web&cd=&ved=0ahUKEwiS...
..to:
https://www.techrepublic.com/article/why-doesnt-anyone-weep-...
EDIT: The article itself is quite interesting, on the rise of Kubernetes, its adoption by VMWare, and the reason why Docker failed to capture market value as much as it could have.