Hacker News Reader

Comment by AndrewHampton | Hacker News Reader

I would be very interested in hearing about anyone's experience managing mesosphere dcos compared to kubernetes. My company is exploring our options in the container orchestration space and have been testing out kubernetes on aws in the past week. Getting started with kubernetes was pretty easy, but there seems to be so much magic behind kube-up.sh and very little documentation about what exactly it does. We have concerns about troubleshooting any issues that come up in the future since we don't really understand what all is going on behind the kube-up curtain.

View on HN

I can't speak for Kubernetes, as I haven't tried it, but we've just started the migration to using Mess/Marathon for infrastructure management.

Following this course will give you a solid foundation for debugging issues, and as has been mentioned, the logs are surprisingly useful for that task. Once we got the packages installed on the cluster, we tested various ways we could imagine it failing (application failing, docker container failing, image restarting, image dying, etc.) and found it pretty intuitive to fix most issues. Sometimes, the fix is necessarily outside of dcos - you'll need to set up an autoscaling group to ensure you always have the proper number of nodes running. You'll need to set up VPCs to ensure your public/private slaves are actually public and private. You'll need to tag those instances as they're launching so Marathon will know they're public/private respectively.

Assuming your team is pretty ops-savvy, I'd say dcos is surprisingly simple to manage and debug, and this advanced course does a great job of walking you through the entire stack of technologies used.

I'm (probably) the primary maintainer of kube-up on AWS. kube-up does a lot of tedious AWS configuration that I/we haven't documented. But there's really no magic involved; it is standard AWS, except for the fact that we use VPC alternative routing to assign each node a private /24 CIDR. There's a lot of tagging of resources as well, primarily so that we can find them later to delete them as part of kube-down, but also to support multiple kubernetes clusters in a single VPC.

Now that we are post-v1 I'm certainly hoping to replace kube-up with something more readable and maintainable (or at least start replacing it). Hopefully we'll find something which better promotes reuse between the different clouds, and hopefully we can document it a bit better as well. The split between kube-up and Salt is also not exactly elegant.

If there are any particular questions you have about how the magic happens, I'd be happy to try to answer them (or feel free to file issues and tag me on github); this will help me make the docs better.

Thanks for the response justin. I sent you an email with a couple comments to the address in your profile. Assuming my team decides to go with k8s, I'd be interested in helping create some of that documentation.

I find mesos fairly simple to administer. `service mesos-slave restart` / `service mesos-master restart` is often enough, though it has the standard complexities of running ZK. That said, when your org gets to a certain size, you almost always end up with a couple of ZK clusters hanging around for some service or whatnot.

The logs are _very_ good, and most unintuitive behaviour will often trigger a decent explanation in the logs.

That said, kube is a lot more integrated; you prob will spend more time setting up stuff like Mesos DNS or Bamboo (that's my plug).

It's worth noting that Kubernetes can also run on top of Mesos (https://github.com/GoogleCloudPlatform/kubernetes/tree/maste...) and DCOS (https://mesosphere.com/learn/).

So if you invest in Kubernetes and later want to take advantage of Mesos you can lift up k8s and put Mesos underneath.

Mesos is definitely more mature, and you can use Marathon for container orchestration, which is a little more mature than Kubernetes, but it's also a bit simpler. Marathon doesn't have the service, secrets, or pod abstractions, for example.

DCOS adds some additional benefits on top of Mesos, like the dcos-ui and dcos-cli. One of the more compelling features is one-step cluster package installation (ex: dcos package install kubernetes). Currently, the community edition is available on AWS (https://mesosphere.com/amazon/).

I do agree that the kube-up scripts in k8s are a bit of a mess. There's really hard to read and reverse engineer, and each provider has their own divergent deployment methods. That's one of the things DCOS is trying to standardize, to have a consistent deployment pattern for Mesos frameworks that deploys their core components inside Marathon, giving them a more battle-tested platform to live within, and granting automatic resurrection in case of failure.

As an alternative you might like to play with Lattice[0], which is extracted from Cloud Foundry to enable easy experimentation.

Disclaimer: I work for the company which donates the majority of engineering effort to Cloud Foundry.

[0] http://lattice.cf

Check out giantswarm.io. I'm the evangelist and we're doing some interesting things around the container management layer. We run a private shared cluster you can kick the tires on without having to install a bunch of stuff to see how it works. Ping me on Twitter if you'd like to try it out after plugging your email into our site! I can push the application through for you.

Reply on news.ycombinator.com