Comment by quaunaut | Hacker News Reader

Comment by quaunaut | Hacker News Reader

I'm beginning to think that what Kubernetes needs by far, is a better development environment story.

1. Something that makes it easy to run the application(s) you're working on locally... 2. ...but have it interact with services in a local kubernetes cluster easily. 3. Something that makes your entire development environment effectively immutable, read-from-files the same way as the immutable infrastructure we're talking so much about. 4. Something that gives you a good start to running in production, so the first day of bringing your service up isn't a mess of trying to find/remember what environment variables/config options to set. 5. Something that's one tool, instead of a mixture of `minikube`, `kubectl`, `docker`, `draft`, and `telepresence`. 6. Something that's opinionated, and follows "convention over configuration", with conventions that make for the best future designs.

Basically, we need something with a Rails-like impact but in the Kubernetes ecosystem.

Docker ships with swarm, and swarm + a compose file does everything you just described.

Swarm is such a pleasant and easy alternative that comes out of the box with Docker, does 90% of what Kubernetes does, and does 100% of what anyone with less than 50 servers needs. The learning curve is also an order of magnitude easier.

Docker started supporting local k8s as a swarm alternative, I think from last month's release.

I'm reading that it's only for Docker EE or Docker Desktop (Mac/Windows).

https://forums.docker.com/t/is-there-a-built-in-kubernetes-i...

https://blog.docker.com/2018/07/kubernetes-is-now-available-...

I just installed Docker via package manager so I guess I never really learned about the organization's offerings in total and didn't know which product I had (CE for Linux). Is there a more up-to-date/better practice for using Docker CE with k8s than this guide dated March[0] using Minikube[1]?

[0]: https://blog.sourcerer.io/a-kubernetes-quick-start-for-peopl... [1]: https://github.com/kubernetes/minikube

Fully agree. K8s is mostly beneficial for larger companies, because of namespaces support, and devops authentication. For smaller teams Swarm is just way more practical and versatile.

Totally agree. Docker Compose is excellent for local development, and Docker Swarm mode uses the same file and is almost the same thing. Some minor additions in extra docker-compose files and it's ready for production in a cluster. I don't get why Swarm is so underrated. For the same $5 bucks in a full Linux VPS with Linode or DigitalOcean (or even the free tier in Google Cloud) you can get a full cluster, starting with a single node, including automatic HTTPS certificates, etc. Here's my quick guide, with a Linux from scratch to a prod server in about 20 min. Including full-stack app: https://github.com/tiangolo/full-stack/blob/master/docker-sw...

Docker (at least on my mac) now comes with kubernetes built in, so you can just click a button and instantly have a kubernetes cluster running.

As for an app deployment tool I've had good success with Helm. It allows you to define all the services, app, variables, etc. that your app needs. Though the templates do seem to be a bit convoluted and hard to read at times if you aren't careful.

+1 for docker for Mac, -1 for Helm. Helms templating is just too simple for the actual complexity of k8s, but it can make a decent starter too.

Helm + Kustomize [0] to add in whatever wasn't in the helm template works pretty well IMO. Kustomize is pretty new and not the most user friendly though.

0: https://github.com/kubernetes-sigs/kustomize

I fully agree. I always found/find Helm underwhelming. Maybe I'm just using it wrong.

What I want is a Maven for Kubernetes. With transitive dependencies, strong conventions, injection of parameters etc.

The templating stuff feels very...unstructured to me. Every Chart is slightly different.

I share the same sentiment. Still looking for the holy grail but the best lead so far is using the combination of Bazel (excellent dependencies support), its rule for wrapping around good templating solutions (jsonnet, ksonnet, etc.) ; and some backend that integrates with infrastructure tooling to generate configurations to fetch into bazel.

1. This is something I've found a bit painful, too -- this looks helpful: https://github.com/GoogleContainerTools/skaffold

2. Haven't tried this, but I'm aware of it - does this address your point here ? https://www.telepresence.io/

3. Something like this? https://github.com/rycus86/docker-pycharm I agree that this could be very cool if further developed.

I agree that we'll start to see deeper integration across the development environment though, as more dev-hours are spent on building up the toolchain.

I migrated all of my services to k8s in the last ~6 months. The biggest hurdle was the development environment (testing and deployment pipelines). I ended up with a homebrewn strategy which happens to work really well.

# Local development / testing

I use "minikube" for developing and testing each service locally. I use a micro service architecture in which each service can be tested in isolation. If all tests pass, I create a Helm Chart with a new version for the service and push it to a private Helm Repo. This allows for fast dev/test cycles.

These are the tasks that I run from a "build" script:

* install: Install the service in your minikube cluster

* delete: Delete the service from your minikube cluster

* build: Build all artifacts, docker images, helm charts, etc.

* test: Restart pods in minikube cluster and run tests.

* deploy: Push Helm Chart / Docker images to private registry.

This fits in a 200 LOC Python script. The script relies on a library though, which contains most of the code that does the heavy lifting. I use that lib for for all micro-services which I deploy to k8s.

# Testing in a dev cluster

If local testing succeeds, I proceed testing the service in a dev cluster. The dev cluster is a (temporary) clone of the production cluster, running services with a domain-prefix (e.g. dev123.foo.com, dev-peter.foo.com). You can clone data from the production cluster via volume snapshots if you need. If you have multiple people in your org, each person could spawn their own dev clusters e.g. dev-peter.foo.com, dev-sarah.foo.com.

I install the new version of the micro-service in the dev-cluster via `helm install` and start testing.

These are the steps that need automation for cloning the prod cluster:

* Register nodes and spawn clean k8s cluster.

* Create prefixed subdomains and link them to the k8s master.

* Create new storage volumes or clone those from the production cluster or somewhere else.

* Update the domains and the volume IDs and run all cluster configs.

I haven't automated all of these steps, since I don't need to spawn new dev clusters too often. It takes about 20 minutes to clone an entire cluster, including 10 minutes of waiting for the nodes to come up. I'm going to automate most of this soon.

# Deploy in prod cluster

If the above tests pass I run `helm upgrade` for the service in the production cluster.

This works really well.

thanks for the details, and sorry for the perhaps distracting question: how do you handle DNS for the servers on your foo.com? Did you have to provide nameservers to your registrar so K8s manages the DNS? This is something I don't see addressed so often in k8s tutorials which usually assume minikube or GCK.

> thanks for the details, and sorry for the perhaps distracting question: how do you handle DNS for the servers on your foo.com?

No worries :) I use digitalocean for the nodes, storage volumes, etc. and namecheap to register the domains.

At namecheap I simply use the "Custom DNS" option for each domain to apply the digitalocean nameservers. I only have to do this once for every domain.

After that is done, I can use the digitalocean API, the "doctl" tool or the web-interface to handle domain records, subdomains, etc.

In order to connect a domain/subdomain to a k8s cluster, I point the domain/subdomain to the master node of a cluster. In each k8s cluster I use an "ingress" (nginx) [0] to handle the incoming traffic (directing each request to the right service/pod. You can think of it as a loadbalancer that runs within your cluster).

This strategy would work on other cloud providers (Azure, gcloud, etc) as well, I guess.

0: https://github.com/kubernetes/ingress-nginx

That was really helpful, thanks!

Yes, this! Not having a clearly recommended approach to development against k8 clusters is a huge barrier to entry!

I was really hyped about Azure Dev Spaces (https://docs.microsoft.com/en-us/azure/dev-spaces/), but it's Azure only, even though it sounds like something that should work on any k8 cluster..

Check out BOSH from pivotal labs.

Vagrant seems to be dead stupid simple, you download / git clone a folder and then you run vagrant up and follow up by vagrant ssh and look you're not on a Linux box where you can do whatever you gotta do. Using VirtualBox / Vagrant works perfectly for me so far, I'd love to give containers a shot, but if it's too much a hassle I try to avoid it.

Vagrant doesn't have to compete with containers. In fact, my best experience with Vagrant has been to standardize a development environment that then uses containers inside.

One vagrant VM allows you to make assumptions about what software is installed on a dev desktop which is especially useful if you're writing documentation. Getting into the Vagrant VM from various OSes is the main problem but once you're in... Then you can install container software and teach others how to work in docker given one assumed OS.

Since Vagrant and containers encourage some form of infrastructure/documentation as code, then more experienced developers can read through what goes into the Vagrant set up and replicate on their native OS if they prefer. As someone writing documentation, I can forget about those more experienced devs and just write to one OS.

Once you have documentation that you can guarantee works when using container software, much of the headache can be swept under the rug for more junior engineers which I think is the krux of the issue for others when using k8.

Skaffold

You've hit the on the head here. Kubernetes is like Java of cloud infrastructure if interfacing directly with every vendor could be seen as C.

I'd very much like a Ruby/Python/Rust of cloud infrastructure to come along and clean up this mess.

Isn't that a PaaS such-as Google App Engine?

That's more like Excel. I'm thinking something more like Heroku, but standardized/OSS.

Have you seen Dokku? https://github.com/dokku/dokku

Might be something like what you're after.

Dokku is great, I have been using it for years. The only downside is a lack of scaling/clustering.

Docker swarm could be the step after dokku for when you'd ever need scaling and the like rather than just a bigger VPS for when the first one starts to feel small.

I think you are confusing App Engine with Apps Script. Doesn't help the names are so similar... App Engine is like Heroku.

(I work for GCP)

Openshift is this. https://www.okd.io/

Reply on news.ycombinator.com