Give Me Bare-Metal
subbu.org
subbu.org
For example, I witnessed a co-worker spend days messing around with docker to host incredibly simple PHP/MySQL and Ruby apps. Instead of renting a few VM's on say Digital Ocean or Amazon and spending probably $50 a month, my co-worker spent at least 50-100 hours of time to put everything on Docker and Puppet.
Let's say the net cost of that orchestration was around $10,000 in terms of billable time. You could rent a separate VM for every project and still come out ahead without trying very hard and still spend less time doing the project.
The total cost on that was way out of whack AND docker continued to be a source of bugs and ongoing overhead costs in terms of time.
My point is that these tools are great when they save you time, energy, and money, but a lot of people are overinvesting in tools like docker too early and they will never get a good ROI on their investment.
I don't know at what scale these tools start to make a lot of sense, but I imagine it is somewhere over 20 or 30 machines or projects before containerization shows itself to be a good investment. I'd be curious to see someone run the numbers on that.
Doing the project? Yes. Running it - no!
These tools make sense at the scale of a single project with two hosts in my opinion. Here's what happens in my experience when developer with little ops experience takes care of a few VMs with small apps: (so a majority of services out there)
- where are the logs and what's the available space for them? depends on which VM, some don't log at all, some log debug because they had some work done weeks ago
- which hosts are running which app version and what's the upgrade procedure? likely every one has different manual steps
- what's your system/libraries/application upgrade story? likely missing / different for each host
- what's your DR story? likely missing
- what's the state of monitoring and are all hosts covered to the same extent? unlikely
- is the development and test environment at all similar to production? unlikely
For each of those items you're likely to go back and introduce something that docker/puppet would've either made easier or provided from the beginning. Knowing that you can consistently build the same host/app every time and that it's got the same options, utilities, tools, etc. available is a huge gain even on 2 VMs.
Yes, the huge benefits start at 1X hosts, but I've seen the benefits and I'm never deploying even one production machine without automation anymore. (salt in my case though) Not doing so is actually more work in a long run. And as soon as you deployed one app that way, you have your library of scripts - another app will likely look very similar and anything around it (monitoring, logging, backups, database deployments, ...) will likely be exactly the same.
So I think that you either pay that 10k for orchestration from the start, or you end up paying it bit by bit, every time something needs to be added or goes wrong.
What the OP was saying is that spending all that time to make docker work was a bit pointless. Especially as a VM makes no real difference to the end user at this stage.
Your points about DR and the like are valid, however if you are going to use puppet, the difference between deploying n docker instances (and keeping the state in the correct way) and VM instances are negligible. (yes, yes, docker is a chroot, so inherits the host's state, but you'll still need to control the docker environment.)
I use cgroups everyday, to avoid memory exhaustion, (25k CPU render farm) and for that its grand. Running it on a public host? madness I say. For things that require high IO(even then its questionable, AUFS...), or low security perhaps.
I disagree. Given that you operate a 25k CPU render farm, perhaps you are no longer aware of how poorly devops tends to be managed in small shops. I'd argue Docker buys you more in those small environments, because in large environments you can build tools that works for your environment. In small environment, the traditional "solution" has been to brush these concerns under the carpet.
Just getting repeatable builds of an environment so that you can ensure the dev, test, staging and live environments continue to bear a resemblance to each other is a massive improvement for many of this type of setups. That is, assuming that there are separate dev, test, staging and live environments - often you'll find all of it on a single server. Yes, really.
Basically, people that think like OP terrify me, because too often we've taken over operations of applications from people who think like that, to find that the operational cost is staggering. We have one customer, for example, that have over the last few years easily paid several times what it would cost to properly re-engineer their application from scratch in time spent for us on operational issues because they've refused to accept that it's necessary. (EDIT: Said customer pays for this time, so on paper it works out well for us; in practice it's bad for us as it leads to lots of tension with the customer over all the issues, and we'd much rather have spent that time making their app better, and help them grow their revenue from it - long term that'd be far better for our customer relationship). This is a "simple PHP and MySQL" application.
VMs aren't really a solution by themselves. It's not app-level isolation that "makes" Docker. It's the entire structure that encourages repeatable builds, a repository, and the light weight nature that encourages breaking things into smaller components. I agree that just Docker as isolation in a multi-tenant system is still scary.
I'm currently working on deploying an application in VMs. With Docker in each VM. The VMs gives us the isolation we need in a multi-tenant system. Docker lets us ensure that dependencies between each component is well documented (or the components won't have access...), and that we have a decent image repository, and that we have ways of managing the entire lifecycle of each component. The two are complementary technologies.
In a large enough environment you could solve the issues that Docker solves with process. In a small environment that process is usually non-existent or weak.
> AUFS
AUFS is not necessary. Btrfs or device mapper backends have been supported for a long time. And even then, for any data volums that needs high throughput you'd use a bind mount from the host.
(especially stuff like logging, backups, monitoring will usually affect your docker build to some degree and can be reused in other apps)
You just won't notice, because it will be 10 minutes here, 20 minutes there, and/or "emergencies" where the time spent will rarely be attributed back to the same original project, so most people don't see how much it costs them to have skimped on the original investment.
To me, these tools make sense from 1 machine, and you start seeing serious benefits once you have two (all the investment you did for one machine? You've now mostly halved the investment per machine, since you have a set of containers you know how works ready to go; you have some additional complexities as your cluster grows, such as orchestration across machine boundaries, but amortised per machine the cost seems to keep dropping)
If anything, what we are seeing is that these tools get people to do things that were easy to ignore before, causing people to think they were done setting up their environment because they had a server running the production code.
Never mind how often these setups have been unreproducible, with poor isolation, no failover/ha setup. They're cheap until they're not. And most often they're not - just that people don't really notice how expensive they get because the time is never fully attributed back the right places.
Having started with devops only recently, I find the concept of CoreOS + Docker Containers/Images/Dockerfiles (that are close enough to Bash scripting) easier to get into than learning custom UI/API of custom company to deploy to their cloud.
That's not really a concern if you're deploying simple RoR or php + mysql apps, which is point of the post.
I guess I have skipped that sentence. It makes all the difference for scaling.
Failover is still a concern though.
Most developers probably shouldn't be messing with Docker. If they aren't directly creating value for someone, then their efforts are probably best directed elsewhere. There is value to yourself for learning and scratching your own itch, but that effort needs to be treated like a side project. It doesn't cross over into your real work.
Docker is low level plumbing on which more "user friendly" workflows are being built. These workflows are what developers should be using. Docker is still young, so there is still a lot more attention on the plumbing than there is in any tools which have been built on top of it. That will change as the ecosystem develops.
But this is a common developer problem. Sometimes developers will spend time doing anything other than what they need to be doing to actually ship things. We don't get paid to write code, we get paid to deliver on things with the greatest impact on that bottom line.
But there's a big recruiting and retention issue also... kinda hard to find and retain good IT/devops people these days if you do everything manually :-)
More seriously, we tend to have an issue of over engineering. I have to pull myself up all the times to just focus on getting the job done and not on building the perfect architecture.
Nowadays it is fashionable to deploy applications consisting of hundreds of files in a complex structure, but this is fashion and not technology and that's easy to demonstrate: in the old days, a C application consisted of hundreds of files in a complex structure, that would compile into one self-contained executable. There's no technical reason you couldn't take a RoR app (for example) and "compile" it and manage it just like any other process. Start it, stop it, move it around, one binary executable and one plain text file for configuration, that's really all you need for any application. Really.
I look at things like Docker and think, you are solving a completely artificial problem, and as soon as fashion swings the other way (10 years at most) people will wonder why people ever did this.
And that's what Docker is built on: processes + namespaces to restrict what each process group can access.
> I look at things like Docker and think, you are solving a completely artificial problem, and as soon as fashion swings the other way (10 years at most) people will wonder why people ever did this.
Things like Docker will still have value with single-executable containers. Problems that people have had for years. Even with single-executables.
You still need to manage privileges. You still need service discovery. You still need to manage the network. You still need to manage data volumes. You still need to manage the environment.
Docker provides mechanisms that either solves all of that, or provides information for tools to hook into (e.g. ability to query what ports the currently running containers expose).
I agree with you that there is distinct benefit in self-contained executables, and I think you'll find a lot of people working with Docker thinks the same. Partly this is part of what Docker gives us: We get to treat a bigger hairball of an environment pretty much like a single entity, even if we're not able to slim it down.
But partly, note that a lot of the adoption of Go within devops tools appears to at least partially be motivated by the simplicity of distribution and installation from getting static binaries (without having to use C). I prefer Ruby, and find Go to be a ugly, too low level, language, but I definitively understand the appeal Go has when it comes to simplicity.
Yet despite having those devops tools as single executables, people want to put them in containers to manage them properly.
If anything, I hope and think that things will move even further down this route. For my part, I'm containerising my desktop environment part by part. There's no reason we should not move to far more locked down process environments.
1. Docker appears to only allow a single entry point per container. What do I do if I want to run a container consisting of say postfix and dovecot together? If this is possible can someone please point me at it.
2. Go doesn't allow you to bundle resources with the executables (as say Java/C#) so you have to rely on 3rd party libraries to do this that aren't very good. I can't easily ship my templates with my web server for example as part of the "go build" process. Again if I'm wrong, please point me at a solution.
But if you absolutely want to put them in the same container, the solution is to run a process manager as the entrypoint. E.g. base it on something like baseimage-docker [1]
Re: 2. I don't use Go, so I can't answer that part.
Been trying to build a canned mail server for a few years so this is probably a nice solution if I can get my head around it.
Docker configuration files are standardised. Application configuration files generally aren't.
I still have no idea how TFA is using it.
I think docker is nothing more than a hype from a new generation who wants everything hip, simple, fun, fantastic and awesuummm and the older and more experienced sysadmins like me have seen this coming years ago. It was inevitable, and now it's just a matter of waiting until the lazy generation finds out what we've been seeing all along: Docker is an abstraction of other technologies and you don't need Docker at all.
Hand-waving this away as a problem that "never existed before" is nonsense. And increase in individual machine power and the demands of applications means that we now have—quite validly—more complex applications, and more of them running on each machine. Administration of this is objectively difficult, and Docker is one of the tools that will help to make it easier.
I am saying that this is convenient for developers, but for us sysadmins, who have to implement Docker apps in already existent systems, it is just not all that convenient, so now we have a surplus of happy developers.... and the sysadmin "should just stop being so grumpy".
Sysadmins are expected to cheer in the same happy way, but in my opinion, many of us already knew how to build the kick ass systems /without/ containerization that the new developers ("devops"... yuch) are so cheery about.
I don't think this is the case.
My goal as a developer is to be able to deploy applications quickly and easily, to be able to scale them, and to avoid worrying about dependencies. That's all good practice.
I still need a sysadmin to handle that infrastructure, so Docker et. al. are not a tool to replace sysadmins.
That being the case, why do you think there is pressure from developers to adopt these tools? If sysadmins are generally capable of building 'kick ass systems' that offer the same features but without using Docker, then why aren't developers happy with the situation?
Docker is a tool that allows sysadmins to provide these useful features without having to arse about building them by hand. It's a tool for sysadmins, and dismissing it out-of-hand in the way you do is totally stupid.
An example: I can't easily rename containers, so if I want to install a new container with a set name for programmatic use, I have to destroy the old container, so there's no quick rollback. Instead I have to have a tag system for the docker images, and track those instead, keeping aware of which current unique name is 'old prod'.
Another example: docker doesn't really support having one repo support two clients with their own config, not without a lot of faffing about. You get one dockerfile, it is always at the root of its tree, and it only builds one artifact.
Another example: until you set up your own base images and repos, you're taking an app that used to be a few megabytes, and turning it into an image that is hundreds of megabytes in size. We have an nginx image that uses a stock docker base, and it's over 400MB! For nginx! That takes a non-trivial amount of time to download on a base image update, a new dev machine, a newly provisioned machine... if the docker hub happens to be up and not slow (we had a couple of issues there recently). And if you do have your own base images, you now have an extra operating system image that you need to keep up-to-date with security patches.
Docker is nice and will 'get there' eventually, but in its current state it benefits devs at the cost of sysadmins. I would counter that dismissing sysadmin concerns out-of-hand in the way you are doing is totally stupid.
One other thing that is annoying is networking. IPv6 is once again being ignored, and for the rest they offer either a simple NAT bridge (and who wants to NAT anyway?), or you're on your own for the stuff you cannot proxy with for example nginx.
I'm hoping that this will really improve because tools do exist already. Openvswitch is one of them.
(It's also ironic to witness in this discussion how matthewmacleod even helps prove my point - developers are usually not really aware of what sysadmins do, although the other way around, we know what devs are doing, that's for sure.)
Would it really be worth the effort, especially if you are devops and app dev in one person, always having to weigh benefits?
Or could Docker in the end prove to be good enough?
Keep waiting. It'll make it way easier to replace you with a small shell script.
So if you just run on a beefy bare metal server yourself, you can probably serve 50-100x as much traffic as you could on a typical VM. I guess the issue is that your software is probably not written to make use of the full machine. You might have to start 50 copies of a process, which might be OK. Ruby, node.js, and Python will probably neat that treatment.
> If I’m a front-end app developer, it’s much more convenient for me to use a container-based cluster management solution on bare-metal without dealing with VMs.
If you're spinning up containers on bare metal, you need to manage bare metal systems, or pay someone to do it for you. Or, you could put your containers in VMs, where you manage the VMs or pay somebody to do it for you.
Bare metal gives you some advantages (like tighter control over the components and the ability to customize the hardware to suit up-front needs) and some disadvantages (harder to dynamically scale, spin up/down times are longer), and VMs have their own advantages and disadvantages.
VMs are here to stay, just like dedicated servers are here to stay. Containers change what we do inside of our systems.