Hacker News Reader

Comment by yongjik | Hacker News Reader

Echoing others, you really don't want Kubernetes here.

I only have limited experience, but Kubernetes works very hard to abstract away the actual machines, and it works best that way: you just say "Deploy 20 instances of job X", and k8s will somehow find the machines to deploy them. You don't care where they are running - k8s handles that.

Once you start to care about actual machines and which jobs are running where, k8s starts to make less sense. You're paying for huge complexity (required to abstract away the machines), but you're getting none of the benefits - it just becomes a glorified wrapper around Docker daemon.

View on HN

Docker in production is a potential disaster, it runs everything as root from the daemon, we are heavy k8s users, and we are transitioning to rootless cri-o containers specificaly because of this. Investigate podman which is a docker cli clone that builds cri-o containers from docker files and allows you to run them under a designated user id, the only restriction is that in rootless mode you cant map ports that are less than 1024, for that you need to run them under sudo which defeats tbe purpose. Howver we use an ingress controller to map inbound requests to containers so it can do the port mapping. We used to use traefik, but now we use nginx as ibgress, becuase we can run the excellent lua-waf module.

We run on multiple clouds, so vendor apis and k8s implementations are not usefull to us.

I struggle to understand when Docker is supposed to be introduced to a stack.

If you have a 100 user SaaS platform, do you run Linux containers on top of Linux just for isolation?

If you need strong guarantees of isolation and if every user has their own dedicated installation. If you are a general service where isolation is enforced at the application level then just deploy straight away.

Docker is useful when you don't want to care about the underlying operating system or you might be deploying on different operating systems etc. Also if you want to quickly switch providers (like going from Ubuntu 18.04 on digital ocean to Windows Server on Azure) since it's a self contained image.

This self contained image is useful during development too. I can cleanly separate dependencies between applications and dev/test/prod. For example I develop a node app based on node 12 + pg 11 but I tinker with ghost too which requires node 10 + mysql. Having this in Docker keeps cruft and complexity off my macOS installation at the cost of space. Having multiple versions of a runtime/database etc can quickly become a nightmare to manage on local operating systems.

Operationally if you can control deployment and you don't need many servers, deploying to the base operating system makes sense. Docker/Kubernetes shine when you have scale + need to provision many servers across different platform providers.

> Operationally if you can control deployment and you don't need many servers, deploying to the base operating system makes sense. Docker/Kubernetes shine when you have scale + need to provision many servers across different platform providers.

On that same note... when do Terraform / Ansible start to make sense?

I have found that Ansible is necessary if you have more than one server and useful even for less than one (eg managing a service account on an old legacy shared machine owned by another team).

I use terraform mainly for provisioning infrastructure (how many VMs on which platform, how many databases etc) while Ansible for configuring the software side of the deployment - like the dependencies, which version of code to pull, configuring nginx, cache etc

Containers are fantastic at isolating one software application from another. All their dependencies - runtime, library, packages - live in separate chrooted filesystems.

User-to-user isolation is hard, and extremely hard if any of the users could be malicious. Docker is broadly neutral here - adds a few tools, might add a few vulnerabilities.

It can be introduced at many points.

in dev: You can setup dependencies without worrying about underlying os. So it is easier for building a new dev machine, adding a new team member, etc.

in integration test: you can setup multiple integration test environment as long as you have enough resource. Running multiple integration tests without interfering other team members is the major benefit here.

in prod: as you may know, for isolation and horizontal scaling. Easier roll back and update?

Reply on news.ycombinator.com