Hacker News Reader

I have used Mesos/Marathon from nearly the beginning and K8s. We currently use K8s. In your third paragraph about the 11 person team with custom scripts is why K8s has won out. It wasn't really the marketing by Google, it was that K8s provided by default what pretty much everyone needs (Discovery and storage management). Mesos took the approach that big shops like where everything can be extended and call APIs but if you are not very huge scale you really do not want to be writing scripts or doing any of that. You want an opinionated full stack.

Scaling from one node to 100s is not trivial in almost every stack. I just think that the requirement that something like that is trivial does not fit the use cases of almost everyone. There's always the "Can this be web scale?" question. We are able to add nodes on the fly but we would certainly face other non-k8s scaling issues with databases that are more complex.

And as for moving k8s to 100 nodes with the touch of a button pretty easily that is coming/is here. That is the power of a full stack that many people are on with. It will only get easier. Writing ad-hoc scripts and combining solutions will not.

So I'm heading a shop with some serious legacy deployment woes. I've got plans for everything but scaling. Now, we're likely to hire out Ops eventually, but for now I'd like to learn devops (backend eng currently) and cut my teeth by getting the ball rolling on a slightly more robust setup.

With that said, what do you recommend? We're a small shop, and our auto scaling needs a very very minimal. We're non-public facing, so our user base is pretty predictable. Yet, I'd like to migrate us to a more modern and mature system. Naturally I was looking towards K8s because of the overwhelming support I see for K8s on HN/etc, but your comment makes me question that.

Honestly, it's the vast options that really screw with my head. There are just so many tools. And where they lack is not immediately evident from the outside. It's difficult to pick a direction. At the moment I'm reviewing Nomad, as it sounds interesting to work with our existing by-hand infrastructure.

I'd suggest going in baby steps. Start with the absolute basics - standalone docker containers + docker-compose. Work in those containers, deploy those same containers. Build your deployment pipeline. That'll get you familiar with the workings of containers in the docker context, and you'll be forced to iron out issues you may find with this way of working.

Then slowly start to spread further - you could start with docker swarm, which is a smaller next-step for devs. Again it's to help you figure out what kind of deployment pipelines you would create and what constraints you need to introduce. The reason I call it a smaller next-step is because the docker compose file used for development is the same one you can use for deployment to standalone or swarm.

By the time you start hitting the limitations of swarm, you should be ready to move to k8s and look at the various options available to you.

I like this. This is good advice.

Containers are still fairly portable and I've worked at a shop where we moved from CoreOS+Consul to Marathon/DCOS and there were almost no app changes.

That is one of the big benefits of Docker. We had some stuff just running on nodes with Docker/docker-compose and even that stuff slowly got migrated over to the big platform.

Agreed, this method sounds nice. The only thing I'd like to find on top of it is some software to manage apps across container resources and physical resources. Ie, I've got some non-cloud old windows boxes that I want to deploy to, so ideally I need a mechanism further abstracts that. I'm currently looking at Nomad, though i'm very early in research.

Currently on this path almost exactly. Problem description and steps all the way up to limitations of swarm. Our swarm currently handles 1M/requests per day we haven't really hit any limitations yet. Good advice though, really

Have you considered hiring an ops/devops consultant for a few hours to help provide direction? Even at low four figures, it would be money well spent based on where you're currently at.

What you are going through is why “Just use Heroku” is still great advice from wise people.

There are so many options out there that until you actually hit the point where Heroku is too expensive or no longer makes sense...Heroku is your best option (if AWS is an option).

I'm hoping to keep it cheap though, and cloud deployment services scare me a bit on price. We're a ~15y old company (of which I'm recent) and in a domain where margins are fairly thin. I'd rather spend the sweat learning this (or hiring for it) than paying Heroku, fwiw.

Plus lock-in scares me.. a lot.

Oh, Heroku doesn’t lock you in. It’s just running your code on containers. There’s even an open source library called Dokku that uses their open source build packs to give run the same thing in Docker containers.

The minute you price out the time and training investment to factor into your deployment solution, Heroku starts winning in a landslide.

Existing scale certainly changes things, but the fact that it forces you to go ahead and build for 12 factor makes moving elsewhere pretty simple.

Good to know, thank you!

On that note, we have some odd ball legacy machines that are dictated/required by our customers. Things like WinXP. So I'm surveying Nomad now as a potential to support non-traditional OSs (though I wouldn't be surprised in Nomad doesn't support WinXP). Do you recommend anything for odd deployment needs like this?

I imagine we'll write custom plugins to handle the odd ones, since who the hell releases builds for WinXP or Win95 these days lol. Nevertheless, I figured I'd ask. Especially since broader solutions like Heroku are not going to help our WindowsX deployment woes. (note, they're physical machines currently, though virtualization might be possible)

For Windows XP, I have no idea. Windows deployments are something I’ve never had to worry about (both by chance and on purpose).

Heroku locks you in in the sense that Heroku has this huge ecosystem of "add-ons" that often can't be replicated 1:1 outside of the Heroku environment, or if they can, require 100x the configuration management that Heroku does.

Much of the task of moving a project from Heroku to e.g. AWS is taking all those "add-ons" and going through the whole process of converting the Heroku-bound account (where everything is managed through your Heroku dashboard) to a separate, full-fledged account with the same service provider. If that add-on is a data storage provider, that also often entails a data migration.

I suppose that is true if using the add on store. I always went direct to the services anyway so it wasn’t ever really an issue.

Don't forget Dokku - https://github.com/dokku/dokku

Dokku is extremely flexible Heroku alternative, and a good introduction to dev-ops. You can start-out treating it like a "dumb" Heroku host, but install plugins and customise to your heart's content. You can easily migrate from apps/buildpacks to Dockerfile's one app at a time etc.

Whilst it's great for getting your apps up and running, and learning basic dev-ops at your own pace. It's not really designed for scale/reliability. So if your small business/app/service starts to take off, you'll quite likely need to migrate to K8s, Docker Swarm, Amazon ECS, if not for scale, for redundancy/reliability.

I rented a 10 EUR VPS from Hetzner and put Dokku on it. It currently runs ~10 side-projects and I couldn't be happier.

This is something I've been hearing. I have a small dev team that deploys many small (but important) apps internally.

We feel we need to hire more ops folks to help us with maintenance and troubleshooting, but people keep telling me that instead of increasing headcount, to just outsource my ops to PaaS like Heroku. After reading the comments here, I may look into this more.

As somebody who was used to running my own servers when I was first forced to use Heroku on a project...I'll go ahead and tell you there is an adjustment period.

But after using then going back to the alternatives, there's a lot more that I miss about it than the other way around. You take for granted how much stuff you no longer have to deal with, until you have to again. There will be a moment when you'll be irritated about some very low-level knob that you want to turn that you can't get access to but when that happens, don't take for granted all the other ones you're not dealing with.

And it simplifies life for developers tremendously.

You almost certainly don't need containers. Just use a PAAS (platform as a service). Heroku is the "go to" default, and will be worth much more than they charge. If you're worried about scaling, I've run off the shelve PAAS providers past 5000+ TPS without any issues.

If for some reason you can't use Heroku, you could try a hosted Cloudfoundry (https://run.pivotal.io). If you want something that hybridises PAAS-like behaviour with the big IAAS services, Amazon's Elastic Beanstalk isn't entirely awful. All these setups largely honour the PAAS haiku of "Here is my source code / Run it on the cloud for me / I do not care how" (https://twitter.com/sramji/status/599255828772716544?lang=en).

Don't waste your time on getting container orchestration going, or even containerising your app yet. For internal, low scaling, app-focused scenarios, you'd be mad to go down this road _unless_ your code has been broken into many discrete deployable artefacts _and_ you need to run them all on the same machine (either locally or in production).

You have an app. You want to run it. PAASes do _exactly_ this.

Edit: I see you're worried about costs. PAAS are pricier than the equivalent AWS instance when you consider raw compute, sure. However, factor in your time in creating the same level of redundancy, deployability and monitoring, I'd be shocked if Heroku wasn't an order of magnitude cheaper. Note that PWS Cloudfoundry will host you any 12 factor app for $21.60/GbRAM/month with no bandwidth limit you'll ever hit.

I should have made it more clear, but one need of mine (as mentioned in another comment) is a deployment system that supports the ability to manage very old, archaic, wonky systems. WinXP, Win95, etc. Do you recommend any tooling to support this?

PAASs aren't going to help me much in managing our resources like physical legacy machines. Thoughts?

Ah, a bit out of my experience alas. Sorry! However, I'm still not sure how container orchestration would help with this.

I can still endorse PAAS for any "cloud" applications you may want to deploy.

Yea it containers definitely won't. Though something like Nomad _(as I'm researching now)_ that tries to manage both physical machines and container fleets might fit nicely.

K8S won't support them either. K8S doesn't support Windows at all at the moment. And even when it does, it likely won't support WinXP.

To add to that, the container story for Windows is pretty dismal right now, even on Azure. I was at Microsoft Build this year, and most of the PMs gave me the impression that Windows containers were an afterthought on Azure due to lack of use -- Linux support were priority. I also did some PoCs on Windows containers both on-prem and on Azure and they were unwieldy and buggy. The microsoft/windowsservercore Docker image is also a very heavy image at 10+ GB. The microsoft/nanoserver image is smaller, but has certain limitations.

tldr; if you want to use containers, go with Linux.

If you really need XP support, VMs are your only option. Tools like Hashicorp Packer [1] may be able to help with managing machine images.

[1] https://www.packer.io/intro/index.html

i think kubevirt is being rolled in soon. It adds a VM type to the set of things k8s can manage. i'm not sure what's required inside the vm to play nice. I haven't used kubevirt at all. But it might be worth checking out.

Depends on what you want to do with these machines, but for multi-OS setup you need VMs. I guess you could check out Vagrant, maybe with some AutoIT to help you avtomate tasks inside Windows. It won't be pretty, but it will work.

To be honest it sounds like you probably shouldn't. There is a risk to researching "the best" technology, it can be a full time job and you still wouldn't get any closer to your goal. You are likely to have a lot of low hanging fruit in a legacy systems shop. Start with those and move yourself forward from there.

Without any specific knowledge of your situation it's hard to be anything but generic advice. But first you need eyes and ears. Under no circumstances should customers know of problems before you do. Are you monitoring the right things?

If there is a product, take a long hard look at the build process. When those mostly work without that special someone that always does them because no one else can, start mapping out all the infrastructure, document it and place it under version control. Implement a configuration management system. No exceptions.

With repeatable builds and repeatable infrastructure you are in a much better place to see what your real requirements are.

I've met people at smaller startups who've gone the Nomad route instead, which seems a lot more sane in many ways, but still requires pretty careful planning and setup for large deploys.

As an admitted Hashicorp fanboy who has used Consul, Nomad, and Vault why do you say that?

The beauty of Nomad for me is that it can orchestrate anything - shell scripts, executables, Docker containers, etc.

Once you have a Consul cluster set up as a server, it’s dead simple. Both the Consul apps and the Nomad apps are small self contained executables that can run in server mode or client mode.

You set your Consul server cluster up first (crazy simple) and then your Nomad cluster automatically configures and registers itself using Consul.

Adding more Nomad nodes is a simple matter of copying the Consul app and the Nomad app and running them both as clients.

These days we do everything on AWS. It’s just as easy for us to spin up a bunch of small EC2 instances using autoscaling and we haven’t needed to use Docker for anything except for custom build environments for CodeBuild so I don’t have anything to compare it to.

GitLab's "Auto DevOps" with GCP Kubernetes Engine is pretty good at the 1 to 100 story. I clicked several buttons and got a google-managed k8s cluster that will pretend to be Heroku for me.

I've since outgrown the basic setup, but their docs showed me how to copy their "magic" configuration into my repo, and it was easy enough to edit to get what I wanted.

I kinda figured I'd be bypassing Kubernetes for this job, because I didn't want to spend a week setting it up, but they made it into an afternoon.

Thanks for the feedback, it's great to hear that our product is that easy to use.

Unless you are using DC/OS enterprise, DC/OS is incredibly insecure by default. Any container running in the cluster has access to the full marathon API (which has NO auth). So any compromised container will be able to: stop and delete everything, see all environment variables set for other containers (often contain secrets for databases or other external apis) and even start arbitrary new containers on your cluster.

So think about it, any compromise in anything you run on the cluster has the potential to access all your data.

This alone should be reason enough to never use the free DC/OS, and the enterprise offering has no public pricing information.

> None of them can go from 1 node running all your apps to 100, in an easy straight forward manner.

Redhat's Openshift platform can do that with one click.

Once you get it to install at least.

Seriously. Installing open shift has been the most frustrating exercise in my life.

I ended up finding rancher, and even though their 2.0 version is pretty much beta at the moment the installation is so much more painless than open shift.