Comment by atleta | Hacker News Reader

Comment by atleta | Hacker News Reader

Yeah, you probably don't. And not only that, but it probably makes your life harder. I've interviewed for a tech lead position at a company working with freelancers and I'm pretty sure the reason they ended up rejecting me was that I mentioned the technical interviewer that I think containers, container infrastructures (like Kubernetes) and even cloud infrastructure is being overused/used without giving too much thought about it as if it came free (in the sense of setup and operating complexity). Too bad the interviewer started rambling about how he was into Kubernetes those days :). (Actually, this was the most technical part of the interview.)

I'm mostly working with startups and small companies creating MVPs and that was their client base too. Now most of the time these are just building CRUD apps, most of the time these apps don't see heavy usage for years (maybe never). Developers love technology, love to play with new(ish) things so quite a few of us will prefer using whatever is new and hip for the next project. Now it's containers and microservices. And it feels safe, because done right, these will give you scalability. And once you convince the client/boss that you need it it's unlikely that anyone will come back in a year and say: hey, it seems that we'll never need this thing that made the development $X more expensive. (Partly because they won't know the amount.) So actually politically it is the safe choice. But professionally and cost wise it's usually worse. It's a lot better to have to transition after seeing the need (preferably from the projected growth numbers). At least you minimize the expected value of the costs (bacause YAGNI).

I once got an interview from a company in the container space because one of their exec read an article I published talking about the trouble with container systems[1]. (Really good talk/interview, but I ended up not moving forward because I didn't want to move back to the west coast).

I've been in smaller shops that wasted a lot of time on K8s stuff and fell behind on their timeline. If you want to run k8s, DC/OS, etc. you need a lot of ramp up time and at least 4 ~ 8 dedicate staff members. I've talked to other startups that preferred running Nomad instead due to setup complexity.

I doubt k8s will go the way of Open Stack since it does actually work, but I do think we'll see it limited to big-end enterprise systems and smaller startups will push forward with other, easier to build up clustering technologies.

[1]: https://penguindreams.org/blog/my-love-hate-relationship-wit...

4-8 dedicate staff members to run k8s? seriously, How did you come up with that number?

1) You can run k8s hosted on Google, DigitalOcean with zero effort.

2) I built a k8s cluster in 3 days with zero experience after spending a week playing with minikube, reading the docs are kuberentes.io

> I built a k8s cluster in 3 days with zero experience after spending a week playing with minikube, reading the docs are kuberentes.io

I'm pretty sure I can do it in an afternoon from scripts on github. But if something goes wrong all bets are off. Just getting something setup is not building a competence around it.

How well would you, with zero experience managing and deploying a stack, do the same in an orthodox LAMP setting?

Honestly, the answer is only relevant if you actually have zero experience deploying any stack, or orthodox LAMP stack.

If you have no experience with (stack Z), then you will have to go out and get some experience before opting to use (stack Z). The problem is, many people hear this and stop there.

While there are some barriers to experience and production-readiness, they are not insurmountable, and there may be a pot of gold at the end of the rainbow. There is a cost for everything. Sometimes it's an opportunity cost. (Sometimes the cost can also come from not acting.)

And you feel like you understand the warts and pitfalls, suitable to keeping a mission-critical service running once the circuit breakers start to trip, based on 3-8 days of experimentation?

well, not the op, but i googled 'kindly pls to fix a kubernetes software on the RHEL servers' and I got this in a forum post:

    $ sudo apt-get install fix-muh-k8s && sudo fix-muh-k8s || reboot

so I think i'm good to go, thankyouverymuch.

Firing up minikube to toy with Kubernetes and switching your entire software stack to run on top of a production-grade Kubernetes cluster are two very different things.

built != run.

not sure if I agree with OP on precise figures, but, if you're dealing with:

- actually federated k8 infrastructure (distributed etcd, etc)

- enough system load that k8 is actually worthwhile and not 'cool'

- user / dev requests

- whatever background IT projects are going on (updates / new nodes, existing upgrades, testing new configs, etc)

and you want:

- 24x7 operations (or even stringent 8x5)

- people to be able to use the restroom and take vacations and still have 1-2 sets of hands available

1 person doesn't quite cut it, because (s)he will quit eventually, and you will have nooone.

dev-centric mentalities of 'it compiled, ship it, i made a new futuristic fancy feature aren't i amazeballs' often don't take into account real operational concerns, which i think is precisely the point in critiquing k8s here

I haven't been following OpenStack closely enough to understand the way you feel that it's gone or what I assume is an implication that it failed to work (in which sense?).

Having installed Openstack many times before, often having issues with it running:

> t k8s will go the way of Open Stack since it does actually work,

_ohh shit__!

My friend is trying his luck with his own start up, they have yet to launch a 1.0 of product but the CTO implemented K8s citing scalability. I legit laughed at that statement.

My friend has a startup and a single person got a k8s cluster running on aws with kops+GPUs in a couple weeks. He loves it. The people who are running it successfully don't come on here to complain.

> a single person got a k8s cluster running on aws with kops+GPUs in a couple weeks

In the context of your GP's comment:

1. How long would it have taken that one person (or the startup) to get started with AWS+GPUs without k8s?

2. How much effort would it take that person to debug an issue with if/when one springs up?

3. What happens when the single person goes on leave?

Probably a lot longer. Do you know of any other orchestrator that supported GPU scheduling natively 6 months ago?

The point is they have been coding away for more than a year without having getting out of beta. The CTO spent 3 weeks learning k8s and setting it up, those 3 weeks which could have been spent on features or ironing out bugs in the product and releasing an actual 1.0 of the product. You can always work on k8s once you have paying customers and have actual demand. Are you telling me you need k8s for not even 10 customers in beta?

For some use cases (GPU) it's considerably less expensive to spend the time learning kubernetes.

Yes you are right, but this is a glorified CRUD app.

Same with me, never had any interviewer actually give me an actual real rebuttal, other than "but it scales!!".

I love software, but I really get tired of the blind cargo-cult culture of most of the industry.

I'm not an interviewer, but:

* Version controlled deployments of your application via K8s state files * Once an application is live, you can easily deploy a new version and have traffic drained over for you * Recovery of applications (Pods) that fall over * Automatic replacement of your application when a piece of hardware fails (K8s will see the Pod has died and bring it back up elsewhere) * You can use labels and other mechanisms to "slice up" your infrastructure for specific work loads * The underlaying hardware can be virtually anything: a high-end physical server, an EC2 instance or a RaspberryPi -- you just define the resources available and what your application needs, that it's placed on the right hardware for you * With a few Pods setup, you can have all the hosts and Pods on the cluster report their CPU, memory, disk, network, and other metrics, straight into a Prometheus instance and have Grafana dashboards up nearly instantly, for free (in virtually in senses of the word) * New Pods are automatically added to the above setup, too * Integration with Cloud providers (so for example creating a service endpoint and saying you want it to be public results in K8s provisioning a public facing load balancer and IP for you and managing that...) * Much more..

This is all instantly available straight out of the box within 15 minutes with Kops on AWS.

I'm not arguing that K8s is automatically the correct answer, but to assume it's not an answer at all based on the feedback from interviews is a poor reason to dismiss it.

It does have a place and it doesn't have to be hard.

Hopefully I've been constructive here.

You've summarized it really well, been using k8s for the past year on a large microservice infra and it's been a pleasant experience. Most problems were flaws in the way our cluster was setup or our misunderstanding of the config options k8s provides.

> I'm pretty sure the reason they ended up rejecting me was that I mentioned the technical interviewer that I think containers, container infrastructures (like Kubernetes) and even cloud infrastructure is being overused/used without giving too much thought about it as if it came free (in the sense of setup and operating complexity).

I'd like to understand your thoughts more on why you believe cloud infrastructure is being "overused/used without giving too much thought ..." and more specifically, what the other options are.

I've come from a background of racking physical servers, plugging them into a network and having a PXE process install the OS for the client. It took a day to provision a single server in an enterprise hosting environment. It was mostly automated.

Speaking strictly "in the sense of setup and operating complexity", I'd love to know your thoughts on how dedicated physical servers in a local DC can outperform cloud based infrastructure in terms of (vast) availability and per-second billing. I don't think you could and I'd even be willing to pay for us to do an experiment: you call your local DC and have 30 high-end servers provisioned faster than me using the command-line.

You also put "... containers, container infrastructures (like Kubernetes) ..." under the same banner. I'd like to address this also, but to be fair and honest, I agree that Kubernetes is heavily overused and so I won't address it here. I'm mainly interested in how you consider containers to be overused given their simple as a concept and equally easy to get in place.

Put another way: in what way have you seen containers being abused? I want to avoid doing that my self and would love your thoughts on the matter.

To continue, if we take a rack full of high-end physical, dedicated servers and we want to deploy a Ruby Rails application (a very powerful, common software stack), how would you sell me a bare mental, direct-to-OS deployment of a Rails application versus using containers to deploy the same application?

Two of the biggest benefits to containers that make me put the effort into deploying them is portability and security. It's one "box" you have to logistically ship to a system and one command to open it and have its services supplied to the network. If the box is hacked due to an exploit, the hacker is trapped in the box and isn't roaming around the host server's file system looking for credit card details.

There's a good reason Discourse, for example, only support Docker has a means of deploying their application: it makes it easier.

> Now it's containers and microservices.

Yeah, microservices have been blown way out of proportion in our industry. They're amazing and great when you're Netflix, Facebook, Google, or Amazon, but there are only four companies that are that big and I just named them.

I may have been sloppy to use the cloud infrastructure expression. It can mean different things to different people. Sure thing, you want to use at least VPS-s, from the very beginning. Maybe even blob storage as/if needed. But probably that's it at the beginning.

> Put another way: in what way have you seen containers being abused?

Well, I wouldn't say abused, but they aren't really necessary a lot of time. E.g. if we talk about a small company with a single product (which, the developers being sane, is a 'monolith' as opposed to being broken into microservices) then containerizing the thing really adds nothing. The server will just have this one component anyway. And even if not, you can install multiple apps on the same server easily. You're unlikely to have dependency conflicts even if you have multiple apps as we're talking about a small org, small software and of course whatever platform you use (JVM, Node, Ruby, Python) they all come with solutions for this (jars, npm, gems, virtualenv). The only problem you may have if apps (probably through their dependencies) depend on conflicting OS level packages. Then you need (or better off with) containers.

> To continue, if we take a rack full of high-end physical, dedicated servers and we want to deploy a Ruby Rails application (a very powerful, common software stack), how would you sell me a bare mental, direct-to-OS deployment of a Rails application versus using containers to deploy the same application?

So we use VPSs. Which kind of solves the issue in itself. Also, the context I'm talking about is the small company/startup hosting their own SaaS solution. Yes, if you want to install 3rd party apps, containers are pretty convenient. I remember wrestling quite a lot with installing GitLab, RedMine and Sentry locally 4-5 years ago. (Not sure about the problems with the last one.) Wasn't fun.

On security: I'm far from being an expert but I do know that LXC (the containerization solution docker uses) is not really meant for security purposes. A kernel exploit is enough to break out of them. See e.g.: https://security.stackexchange.com/questions/169642/what-mak... . So if your container is compromised, then your VPS/server is compromised. Which means that you probably don't want to deploy these 3rd party apps on the same VPSs as your SaaS. But that kind of goes without saying.

> ... if we talk about a small company with a single product (which, the developers being sane, is a 'monolith' as opposed to being broken into microservices) then containerizing the thing really adds nothing.

I totally agree with the monolith thing. Did that my self recently too. I don't agree about it not being helpful to containerise them, though.

I've containerised a recent Go application I wrote. All it contains is the binary, an assets/ directory for CSS, and a templates/ directory for the HTML server-side templates. Deploying a new version consists of 'make docker-push' followed by 'make docker-run' on the remote server(s). All the Makefile is wrapping is Docker build, push, pull, rm, and run commands. Hardly taxing work.

It's also much easier to version control components when they're all placed into a neat and tidy "box" (image.) And because changes are just diffs of the filesystem, it's absolutely fine having 5,000 versions of your application because like branching in Git, it's cheap.

An example of this is wanting to change a library your application is using and trying it out. Super easy: change the lib', package it into an image with a specific version for that "product", deploy it, it fails (let's say), and you redeploy the previous image and you're back up.

Not to sound too arrogant or obnoxious, but anything other than the above is creating more work for oneself for no reason at all.

> So we use VPSs. Which kind of solves the issue in itself.

Do you pay per VPS? Sounds expensive. Would it not be cheaper and easier to have a "cluster" of VPS and run Docker on them, containerising your application? No need for K8s or any kind of orchestration, but using VMs as a "container" (like using AMIs for example) seems like a heavy, expensive option? I could be wrong.

I think you're solving the problems containers do using a heavier, weightier solution that's also slower and more expensive. Again could be wrong (and happy to be proven so)

> A kernel exploit is enough to break out of them

It would have to be an exploit that gets you a shell or the ability to execute code on the host. They're quite rare, I believe.

When you deploy new versions of your app, is there a short downtime between the moment you stop the old version and the moment you start the new one?

At present, yes. It could be easily overcome with a simple A/B deployment mechanism.

I used to do that before switching to Docker swarm mode (docker stack deploy automates rolling deploys with zero downtime). In my experience, it's not that simple to do it correctly. You have to start the new containers, wait for them to be ready, then switch traffic to the new containers, drain connections to the old containers, and eventually stop the old containers. You need some kind of reverse proxy to do this, or at least iptable rules.

I'm curious: how would you do it? Is your app directly exposed on the Internet, or is it already behind a reverse proxy?

You're not wrong. It's not simple, but it's also not that hard, really.

Yeah I use a DigitalOcean load balancer in front of the servers. At this point in time the application serves my family and friends. It's not critical. And even when or if it does become public, even then I would just use maintenance windows.

Not sure what the benefit of zero downtime deployments are, to be honest. They make things complicated for a few seconds of downtime to switch traffic over. You can even make a maintenance page that's all fancy and checks for the backend coming back up and then redirects traffic... not that hard at all.

I might consider Swarm. Probably not though. I do agree those technologies can complicate matters but they can also make life easier. I guess it's a balancing act, isn't it?

Thanks for following up!

> Yeah I use a DigitalOcean load balancer in front of the servers.

Why the load balancer if the app only serves family and friends, and maintenance windows are okay? It should be able to run on a single server.

> They make things complicated for a few seconds of downtime

In some cases, it's more than "a few seconds of downtime". In one of the apps I maintain, some requests accept file uploads, and some stream long .csv files that are generated on demand. These requests can take several tens of seconds. If the system is unable to start a new version of the app, and switch traffic to it, while keeping existing connections to the old version, then we're talking about at least 1 minute of downtime.

We've thought about simplifying things using a "maintenance page that automatically checks for the backend coming back", but 1 minute is just too long for our paying users.

Some day, we deploy new versions multiple times. Our paying users would be unhappy about the multiple interruptions :-)

That said, with the advent of single-page applications, maybe it should be okay to transparently retry requests on the client side while the app is restarting on the server side (returning 503). No maintenance page. No complex zero downtime logic.

> I might consider Swarm.

We were already using Docker and docker-compose. We manage zero downtime deployments using docker-compose and nginx as a reverse proxy, scripted with Ansible but it was a bit too hacky for my taste. It's when I started to consider Docker swarm mode. And it's really great, even on a single-node, especially when you're already using docker-compose. I'm oversimplifying a bit, but it's basically running `docker swarm init` and replacing docker-compose with `docker stack deploy`. I recommend it.

> Why the load balancer if the app only serves family and friends, and maintenance windows are okay? It should be able to run on a single server.

I use DO to manage the DNS for the domain. I use their LB to manage the TLS cert for the domain. By doing this I don't have to manage TLS connections in my application nor deal with certification creation and rotations. DO are doing it for me for $10/month. That's a bargain given my time is worth $100/hour.

I can also easily swap out the backend server, or add more, as I see fit. This falls in line with your zero-downtime argument. The LB is there and ready to go should I want to attempt such things :)

> In some cases, it's more than "a few seconds of downtime". In one of the apps I maintain, some requests accept file uploads, and some stream long .csv files that are generated on demand. These requests can take several tens of seconds. If the system is unable to start a new version of the app, and switch traffic to it, while keeping existing connections to the old version, then we're talking about at least 1 minute of downtime.

But this is the key point of it all: your application and its users demand zero-downtime. My mother (in-law) does not :)

> I'm oversimplifying a bit, but it's basically running `docker swarm init` and replacing docker-compose with `docker stack deploy`. I recommend it.

Looking into it right now. Thanks a lot for the recommendation.

Right, DigitalOcean load balancer is really handy!

I'm curious to know what you think of swarm after you tried it :-)

Hi.

It's very good. I will likely deploy my application to it and eventually, move an open source hosting platform I'm building into it too. It's simple and gets to the point.

Thanks for the recommendation :)

Hi. Glad you liked it!

I like Docker swarm lightness and simplicity, but I regret there is no official and built-in solution to automatically attach/detach volumes associated to a service (for example backed by Amazon Elastic Block Storage, Google Persistent Disks, or DigitalOcean Block Storage volumes).

I also miss the concept of pod (the ability to colocate multiple containers on the same machine and let them share the same networking stack), and the ability to launch one-off jobs (for example to run database migrations).

On a single-node cluster, these problems don't exist, which paradoxically makes Docker swarm a great solution for single-node clusters :-)

> I've containerised a recent Go application I wrote. All it contains is the binary, an assets/ directory for CSS ...

So basically you're using docker in place of tar here really. Not saying it's not convenient but it doesn't give you anything above the tools we already had before. You create an installation package locally (it could be a deb, a tar file or even just a local directory) and then transfer it to the deployment host. Yes, that's how you do deploy :). You still need the same amount of commands to copy the files into the target file system (in this case docker, but again could be a tar or a local dir). Then you use a command to copy it to the server. Instead of docker push you could use scp (for the tar), or rsync (with the local dir). You can actually just deploy the diff with rsync: it can work with 2 source directories. One would be a remote dir (the current version of the deployment) and the other the local (new version) and it would create a copy of the local/new version based on these transferring just the diff. Whether you want to try a new version of a lib or a new version of your app.

It's similar to how I deploy the projects. I use fabric (for copying files and executing remote commands, e.g. for db migration) in place of a makefile, but the idea is the same. Sure, I had to create it once but it's mostly reusable between the projects.

> Do you pay per VPS? Sounds expensive. Would it not be cheaper and easier to have a "cluster" of VPS and run Docker on them

Again, let's remember the context/use case I'm talking about: a company running their (monolithic) SaaS product. You don't have dozens of low traffic services that you want to host isolated. You have the same app all over your infrastructure with maybe a few services for ops purposes (monitoring, logging, error logging, etc). But you could just buy hosted plans for these. If not, you can stash them onto a few VPSs. Docker or not doesn't matter, whatever the most convenient way to install them. These aren't public facing.

So no, it doesn't result in unused/wasted capacity. What you suggest is workable of course, but you may end up with something that is more complex/expensive to operate than going with an existing orchestration solution. (Unless you suggest deploying something simpler than K8s, but still a third party/open source solution.) But even then, it's more complicated than not dealing with containers.

> So basically you're using docker in place of tar here really.

Not even in the slightest. They don't even compare... at all.

By using a TAR archive of you deployment files, you're completely side stepping all the benefits of isolating an application into a container, whether it's a monolith or 56 microservices.

First up: portability and allowing anyone, on any hardware, and any OS to run the application with a single command. All they need to do is install Docker and "docker run" will do the rest. With a TAR file based deployment, they need TAR, they need to know how-to deploy the application and manage the files; they may need some runtime on their system like Ruby, Python or NodeJS; and they cannot restrict the resources the application is using...

On the back of the above: for your developers to fire up a new version of the application that a collogue may have written requires muddling around in Git and using branches -- this interrupts work flows and can lead to lose of work if "git stash" isn't used right. With a container: they just run it.

Secondly you can restrict the resources the application is using inside of a container much easier than you can with one that is not.

I'll make one final point here: Docker Compose. You're missing out on a lot of free stuff here. It's so easy to write a Docker Compose file that brings up everything needed by the application - databases, caches, mocked APIs - allowing anyone in your organisation to bring up the application on their laptop and play with it, including the CEO, DBAs, QA, the receptionist.

You're ignoring all of this because... well I can't put my finger on it, to be honest. Maybe you enjoy hurting your self? lol

Good luck. I don't feel it's worth responding to future comments on this thread. You're clearly unwilling to listen or try new things.

> It's similar to how I deploy the projects. I use fabric (for copying files and executing remote commands, e.g. for db migration) in place of a makefile, but the idea is the same.

Haha! No, they're not the same idea. One is idempotent and the other is not. That's a big differences. And why Fabric? Why not Ansible or Salt Stack?

Man what a way to make your life more difficult for no reason.

Reply on news.ycombinator.com