As Kubernetes grows, a startup ecosystem develops in its wake

What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. We're currently on Google Kubernetes Engine, and I'm disappointed in the lack of integration throughout Cloud Platform. GCP is great in many areas, but the Kubernetes offering is clearly an afterthought. As an example, if you create an ingress, this creates a bunch of load balancer objects (forwarding rules, URL maps, backends). But these are all given obscure names (k8s-um-production-zippediboop-f8da5e6c92f38300), and none of the Kubernetes labels transfer over. Same with disks; GCE disks don't even have labels, so they put a bunch of JSON in the "description", which of course cannot be filtered or queried. Similar things happen with VMs and other things; the "mapping" of Kubernetes to GCP concepts is basically either lossy or non-existent. Many other similar weaknesses exist: Cloud SQL (if you want to use Cloud SQL, you have to manually run the Cloud SQL Proxy as a sidecar), GCP service accounts (different from Kubernetes service accounts!), etc. GKE is solid, but everything seems like it's been put together without any over-arching design; GKE is just another thing buried in a somewhat ramshackle jungle of different services.

There's an opportunity for a vendor to come in and offer a platform where you start with Kubernetes. In particular, this means that the platform should be manageable from Kubernetes, through the use of CRDs and pluggable controllers. For example, I should be able to define a persistent disk using a Kubernetes manifest, instead of having to go through some web UI or CLI ("gcloud compute disk create" or whatever).

That said, it's hard to compete with GCP at this point. Whoever wants to compete in this space have to compete not just with GKE, but with the technical awesomeness of GCP.

It's kind of interesting because Google and Microsoft have opened a new front against Amazon to counter their strategy of locking you in with services. If software from CNCF is good enough to be used in place of these services on each cloud provider then you can nullify AWS's advantage there and Google can lure you in with ML and Microsoft with their enterprise experience.

But I'm not sure these projects can make progress faster than AWS can release and update their fantastic array of services. Kubernetes being the obvious exception (when is EKS GA?!?!)

Just want to remind anyone who hasn't tried K8. You can run MiniKube locally right on your laptop to get a taste of its power (and complexity):

https://kubernetes.io/docs/getting-started-guides/minikube/

For example, a 3-node Redis Enterprise cluster, all run locally:

https://redislabs.com/blog/local-kubernetes-development-usin...

Currently using GCloud and Stackdriver monitoring, but a few of the tools I am excited about include:

Prometheus / Granafa / KSonnet

KubeFlow ML

https://www.youtube.com/watch?v=I6iMznIYwM8

Istio, for programmatic routing

https://istio.io/

And, Agones, for game hosting

https://agones.dev/

Recently, I spun up a simple pod-to-pod communication example but I found it pretty difficult. If you look up cluster networking in Kubernetes (https://kubernetes.io/docs/concepts/cluster-administration/n...) you'll find a whole fire hose of different options from ingress to calico to fabric and on and on.

This was what it took for me to try and rubber ducky my way to getting networking to work on Kubernetes, and in the end I had to get help from a friend at work (https://stackoverflow.com/questions/50195896/how-do-i-get-on...). It may be better than what came before, but it's not great.

After a few days of playing I set up lets encrypt with load balancing, a running app (Rails). A remaining issue are persistent volumes and how truly persistent these are. I haven't found out yet which solution I should pick for this. Longhorn is a Rancher product, which is probably what I will read more about now, but I cannot be sure. There are so many concepts and terminology that you need to figure out. Having Rancher in between is not helping me get a hang of kubernetes itself more purely of course.

A while back I was playing with Docker Swarm and I must say that I like Docker Swarm better in the sense that it feels closer to the source and because it is built into Docker. I get a feeling however that Kubernetes is where the future is so learning more about Docker Swarm is probably less worthwhile.

For those of us not backed by venture capital and not charging SV ex-googler rates to our clients, we need something to say "we'll host you on this git push containerization thing, it'll be cheap and easy, and we'll expand when the time comes, and that'll be cheap and easy too".

I did a (admittedly short) search for kubernetes and kernel bypass, and the only thing that seem remotely relevant was [0], however it didn't indicate whether they work together.

For background, I work for a Dark Pool Alternative Trading System, and we currently utilize kernel bypass for all of our networking using Solarflare NICs & openonload [1].

In the same vein, curious how containers work with CPU shielding and pinning threads to specific cores. Is it possible, and how do multiple containers on the same box interact in that regard. Do they need to be quasi-aware of each other so as to not pin a thread to the same core?

I'd greatly appreciate if anyone with experience with containers can answer these questions. I'm genuinely curious, but it's not worth researching further if there's no solution that can handle these strict requirements (e.g. it's a non-starter if containers increase latency).

[0] https://thenewstack.io/life-post-container-world/ [1] http://openonload.org/

At Datica (where I work), we started in 2014 with a bespoke container orchestration layer. This powered our HITRUST certified Platform. Think Heroku for regulated industries (like Healthcare). After years of hardship trying to keep up with the market we finally decided to go all in on K8s.

Kubernetes gives us the flexibility and community to focus on the compliance and security layer, while not having to build a Platform in house. Until now, no other open source solution had given us this type of flexibility. We're still working toward a GA release, but the speed at which we've been able to move has been incredible.

Now that there is a usable solution for complex orchestration, many newcomers will consider it the goto solution, regardless of scale. You need nginx and some python scripts? Kubernetes! You have 100 daily visitors? Kubernetes!

It’s not really their fault, it’s just a sad consequence of a convenient cloud solution.

I found that only bitbucket deployments is a good one in terms of simplicity of managing deployments (just like Heroku). Most vendors forces me to use their crappy CI solution for doing CD. Why they want me to migrate to your (very limited) build system? I can pay you for a fancy dashboard, but please, allow me to use whatever i need to build my software. There are many options on the market for CI and you just can't build them for everybody. But good CD is a way to go and easier to manage. Just give me a hook for registering for a new build (say docker image versions) and help me manage this stuff. I have tried Spinnakle, but it is too fragile for me - there are no simple way to install it (k8s even easier!) and UI is too bad for a small project.

Good CD is still missing in k8s ecosystem.

One great k8s tool I like is kompose. It allows our devs a very similar interface around secrets, networks, volumes, etc.