Kubernetes at Box: Microservices at Maximum Velocity

Have you checked out DC/OS? It's built on Mesos/Marathon/etc. and has some pretty cool service discovery/routing built in, i.e. Minuteman which allows you to define virtual IPs and ports that map to specific services.

Could you explain a bit more why things get harder if service X is using service Y? I'm sort of the opposite of you in that I haven't used much of the Mesos stack yet but am familiar with Kubernetes. Services depending on each other is pretty trivial in Kubernetes so I'm wondering what makes it so hard in mesos.

Thanks for the feedback - may I ask if you've looked at Kubernetes at all? It appears to do exactly what you describe, and I'd love to hear if you evaluated it but found something missing. You can see some details of how this would work in a recent blog post by CoreOS (https://coreos.com/kubernetes/docs/latest/services.html).

Disclosure: I work at Google on Kubernetes.

I'm using Mesos/Marathon stack, and even that part I don't have trouble with. Mesosphere has a service called mesos-dns that will take care of service discovery, and Marathon has built in support for dependencies (don't launch X until Y is launched).

We have built some tools for port discovery (talking to mesos to figure out what port service Y is built on), but even with all our tools, we recently did a complete cloud migration (to GCP), and it was easy as backing up and deploying Zookeeper on the new nodes. Once the slaves were up, everything was running as if nothing changed in under an hour.

Are you running on physical hardware?

I don't work at Box. It has also been 6 months since I touched K8S, so a lot of details I have about K8S in working memory is gone. I'm also interested in the answers to the question you raised.

Off the top of my head:

Have you thought about putting flanneld on the machines hosting the non-K8s services? Probably impractical, but it's something to consider.

The other is to treat the services inside the cluster as if it is in a different datacenter and explicitly expose nodeports that the other services need. If you're using HTTP as the transport, maybe use an http proxy running inside the cluster and proxying them to the services within the cluster. That's how I did it with getting AWS ELB to talk to the services within the cluster I set up.

The K8s slack channel is pretty good for things like this.

You can either bind the container to a host port and register the ip of the node (or use the k8s dns or api to find the ips). Otherwise register a service with a nodeport and all the nodes will accept traffic and load balance internally.

You can get a list of ips from the DNS (instead of just the service ip), and I think that interacts appropriately with host ports.

We ran into the same ksoftirqd issue in our own bare-metal deployment. Turns out there's a performance regression in the linux kernel that manifests when we configured the system with more receive queues than we had physical cores in a single socket.

We dropped the receive queues down to 12, from 48, and hit line rate. More info here:

https://github.com/coreos/bugs/issues/1275

I work at Box on this project.

Our network setup is constantly evolving due to a number of internal networking limitations related to nearly static ip-addressing and network acls. I'll describe our current setup and then describe where we'd like to go. The big piece of context is that we already have a number of services already being managed via puppet and a smaller number of new and transitioned services in Kubernetes so we need to allow interop though a number of different mechanisms.

We are currently using Flannel for ip-per-pod addressability within our cluster. No services are communicating inside the cluster so they aren't using kube-proxy yet. For services outside the cluster talking into the cluster we are using a heavily modified (https://github.com/kubernetes/contrib/tree/master/service-lo...) which we have contributed back yet. It supports SNI and virtual hosts. And we get HA and throughput for the individual loadbalancers by using anycast.

We have a number of internal services outside the cluster slowly moving to SmartStack. So I assume we will be figuring out interop with that and running it as a sidecar at some point. We would like to move to calico as we have some fairly high throughput services running outside of the cluster which we need to avoid bottlenecking on a loadbalancer for. We have separate project running internally to move our network acls from network routers to every host via Calico.

Hope that is more helpful than confusing.

We are using a product called calico which integrates with kube, openstack, baremetal to setup iptables rules which simulate network ACLs on all of the receiving hosts so that services are only able to reach network endpoints which are whitelisted for that service.

Disclaimer I work for Box on Kube

Shame you didn't read any further, it was a good article. Though not necessarily a criteria for success, some businesses have requirements that make many services make sense. We may not need that many services, but its always interesting to learn how folks solve their engineering problems.

You can easily end up with dozens of services if you split up your application aggressively enough. Think about all the parts in your application that are handled by workers like Sidekiq/Celery, these can all be applications and not part of the monolithic.

For example for us at Gogobot, every time a user submits a recommendation we detect the language. This can be a service instead of a worker and the code can live separately.

If you do this often enough and aggressively enough, you end up with dozens of services. Once you have an environment that allows easily testing/launching these it's much more efficient to launch a service than replicating your monolithic and assign worker for things.

I have talked to a number of people at large companies who have dozens to hundreds of services. Each team typically has a few services, but when you have hundreds to thousands of engineers, having dozens to hundreds of services seems totally reasonable to me.

s/kubernetes/box no?