Hacker News Reader

Comment by lewisjoe | Hacker News Reader

Here's a tweet from Solomon Hykes noting, if WASM existed back in 2008, he wouldn't have created Docker! https://twitter.com/solomonstre/status/1111004913222324225?s...

My biggest concern with K8s is how the entire ecosystem is a costly affair. Can I run a service with multiple nodes for 10-50$ per month?

This is why I'd love WASM to win. Right now your browser can run 100s of tabs, with inter-tab communications and all the security sophistications on a single machine. For me, this symbolises a future where WASM can bring cheaper computing to the cloud as well. I don't want to pay a fortune as AWS/Azure bills just because k8s is a costly ecosystem.

View on HN

When I’m reading about Kubernetes, it makes me tremble. Like you need three servers with 16 GB RAM each just for some management stuff.

I have one server with 1 GB RAM. Can I start with that? And expand on more servers later if need arises.

Yes with 1GB you can run k3s.

Yes you can, that's the point really. You can just start off on your own machine with something like Minikube [1] and deploy to GCP on a bunch of much larger nodes to see how it behaves. The overhead of Kubernetes itself isn't much, it's just a bunch of API daemons.

For small scale development, plain docker/podman is better in my opinion.

[1]: https://minikube.sigs.k8s.io/docs/start/

Seconding K3s being a good option, supports most distros and is easy to set up in minutes: https://k3s.io/

If you'd like to use a management UI instead of just the CLI, Portainer is also a wonderfully lightweight option: https://www.portainer.io/ (and also supports Docker Swarm, if you'd like even more lightweight container orchestration)

That said, in my eyes the biggest problem with Kubernetes is that some distros by default reserve resources and thus disallow overcommit, which is fine from a stability perspective, but doesn't really work for overcommit scenarios.

For example, with Docker Swarm, it's exceedingly easy to say: "Okay, this PostgreSQL instance can use anywhere from 0 to 256 MB of RAM, but no more than that" and with 1 GB of RAM i could easily run 8 of those instances if the average usage was half or less of this maximum limit - a risk that i'd sometimes like to take, when i know that the limits in place are more along the lines of controls so that the whole OS doesn't run into OOM errors.

Of course, if you mess around with the YAML, i guess that's doable on some level as well.

As a sidenote: most of the software that i run (Java, .NET, Ruby, Python, Node, Go, PHP) all is memory constrained most of the times, so when looking for server specs, i usually prioritized the memory the highest, CPU, storage and network speed all usually being a secondary concern. Sadly, in the current day and age, i'm not sure how far you could actually get with 1 GB or why even use container orchestration at that point. The smallest VPSes that i have come with 2 GB of RAM and even those are for smaller pages/proxies and such, whereas the majority have at least 4 GB (since GitLab and Nexus won't launch and work with much less) and my homelab servers have 16 GB, whereas my workstation has 24 GB at this point, because that's all that i can afford.

Wirth's law be damned, i doubt that even OSes will work well with that much in the future, especially because of desktop software being infected with Electron and server software having JVM eat as much memory as you'll give it (though that also happens with MySQL or MariaDB). I don't care about "Unused RAM is wasted RAM" as much as others do, i just want to run more stuff on my servers and to have software that doesn't NEED the memory not take it.

I'd suggest trying out https://k3s.io/ then. Runs on pretty much anything and supports multiple nodes that can be added later. They have ripped out a lot of things you might not need (cloud drivers, etc). Despite that, it was easy to install things like storage drivers or Cilium for networking when I wanted that.

I used k3sup to setup my cluster, but that was just for convenience.

Yes, you can start with that. There is a k8s distro called k3s that runs fine even on 1GB Raspberry Pi 3.

I have been running it on those for a while, but also on other SBCs. You can grow the single master by attaching one or more workers to it. The "pain" is mainly that you can't expand the single-master to a multi-master setup, so once you want to go beyond a single master, you have to re-install it all. On the other hand, with k8s you have all deployments, etc. in yaml files and you simply redeploy these on the new cluster.

My remaining pain-point is persistent storage - it's simple to run complex setups with stateless containers or that can retrieve there data at initialization (i.e. a secondary name server that populates itself from the master at startup), but for classic databases I currently still have a manual step to import the DB dump once the service got (re)started/migrated.

Sources: - https://k3s.io/ - https://www.jeffgeerling.com/blog/2020/installing-k3s-kubern...

Where on earth are you getting that notion?

K8s operates just fine on 1GB nodes. It's very low overhead.

You can operate it on a single node cluster, but that's generally not recommended, not for performance reasons, but for redundancy.

The reason larger nodes are recommended is that k8s is for fleets of applications. If you are only talking about a single service, then a k8s cluster is overkill.

K8s is the cheapest way to run a fleet of microservices.

I believe you can get away with much less, but this is something that concerns me as well. We deal with customers that want Kubernetes, but the management plane uses more servers/VMs than is required to run their code.

There is mini-Kubernetes implementation, but I don't know if they're any good for production.

There seems to be a limit to how small you can go with Kubernetes in my mind, and most people never need to cross the line where they're large enough that Kubernetes makes sense. It's honestly a niche product, but right now it's being viewed as a catch all solution.

I think k3d would be a good choice for you as others have mentioned. It's a lightweight wrapper on top of k3s - https://k3d.io/v5.2.2/

More and more it appears that k8s is designed to extract maximum value from the tenants of the cloud companies.

You need to connect your cluster to public Internet? No problem that's $x for each load balancer (whether its needed or not).

Ohh you are feeling the whole thing feels like a blackbox and difficult to debug or observe? No problem that is $x for data dog.

Gcp is the worst when it comes to setting up k8s with preemptive nodes. Thought you could get away with preemptive nodes? Not so fast, we restart all the nodes together at end of 24 hours period so that your multi node cluster will have zero availability for 5 minutes everyday. Or jump through hoops to killing your own nodes periodically to keep them all restarting at the same time.

Those aren't features of Kubernetes.

If you like you can install a free ingress such as NginX and route traffic to it.

If you like you can just have logs on your physical nodes and go look one by one, just as you would have to before things like k8s came along. Datadog is a value add. It's not essential.

GCP preemptible VMs are nothing to do with k8s - they are literally designed to be short-term (up to 24 hours) VMs to do things with. Yes, GKE can use them, but not as persistent resources. That's not what they're for.

Here's what they're for:

> Preemptible VMs are Compute Engine VM instances that last a maximum of 24 hours, and provide no availability guarantees.

I.e. don't try and "get away with" them. There are plenty of options for cheap K8s.

Yes, you can setup your own kubernetes if you want, using just ec2, but nobody wants to do it because it's easier to just pay.

I think k8s actually saves you money, because it allows you to use your resources better.

Yep - you can also just use a VPS or even some Raspberry Pis in your home.

The ingress can be free with Nginx but not the inbound firewall rule that passes the traffic to Nginx. And 'coincidentally' it costs the same $ as using their load balancer which automatically has this rule applied.

You're talking about a cloud provider's fees, not kubernetes.

It sounds like it's just setting up the load balancer for you behind the scenes? Kubernetes isn't a load balancing engine, to my knowledge. It just sets up containers and stuff.

If there is anything nefarious in the design and rollout of k8s it's just to make it uncool to run something small.

I've worked with it for a bunch of years now and run a cluster at home, but I do hate that it scales down so poorly. Is so much work to run a bare minimum, if not for SME-purposes but for local development. It's just starting to get acceptable.

My only real gripe with GCP is that every project is set up to use their premium networking tier by default.

> if WASM existed back in 2008, he wouldn't have created Docker!

doubt it as there was JVM back then.

>This is why I'd love WASM to win. Right now your browser can run 100s of tabs, with inter-tab communications and all the security sophistications on a single machine.

Now just add K8s on top to orchestrate it :) Somebody will just develop new WASM based container ...

Ah, but WASM is Java for the haters, so it sells much better at the bytecode shop.

I wonder why the Java haters chose WASM over Kotlin

Kotlin for WASM is in the works[1], so once that arrives they won't have to choose between them.

[1] https://www.youtube.com/watch?v=-pqz9sKXatw

"I wonder why the C++ haters chose ARM over Go" is a comparable sentence. I hope the error is clear.

I don't think so. The parent said:

>[...] WASM is Java for the haters, so it sells much better at the bytecode shop.

Which means WASM was advertised as a new bytecode VM to replace Java.

What I'm saying is that, for these supposed Java haters, why did they choose to completely reinvent the bytecode VM when they could have just chosen Kotlin and not have to reinvent the wheel? It would fix most if not all of their concerns about Java.

Is the JVM made to be isolated like WASM is?

You just use the Java standard library to do everything, and the Java runtime handles keeping them namespaced. At some point (right around the dot-com crash) everyone switched away from Java (because Python was free) and that resulted in using POSIX to do everything. No more namespacing. Then people wanted to scale, and, viola! Docker.

Docker was not the only container based system back then and we did have other projects for running arbitrary code in the browser. Were they as good as WASM? No but the browser wasn't as good either. Point is it's not just WASM that makes him feel that way today. It's pretty damn hard to roll back your mind 12 years and then apply 12 years of innovation from the side and make accurate predictions of what you would have done differently.

Just like "if we had internet in the 19th centry", it implies so much more.

Ironically the only WASM that is as good as Flash tooling in 2011, is Flash itself recompiled to WASM.

But why using a browser in the first place ? It's not like your OS is incapable of running 100s of processes (with inter-process communications and all the security sophistications), without the additional weird abstraction of tabs and the browser quirks...

IDK either. It's like we have meta-platforms, or platform adapters. Platforms inside platforms.

Right... Chromium uses user namespaces as part of its own security model but I'm not sure how that extends to v8 or WASM runtimes. Where's the line between implementing a security model in a userspace VM and just providing an abstracting layer on top of OS features?

Why can't you run multiple Docker containers with inter-container communications and all the security sophistications on a single machine? You can do exactly that. I suspect you have different expectations from Docker and WASM, such as expecting to run heavy things in Docker and light things in WASM.

Anyway, to make WASM useful as a server platform it will end up re-implementing the entire Linux kernel API.

Doesn't Docker re-implement about 30% of the Linux kernel API?

> My biggest concern with K8s is how the entire ecosystem is a costly affair. Can I run a service with multiple nodes for 10-50$ per month?

If your service can run on few nodes, do you really need a scheduler like K8s? Why not go serverless like Fargate or CloudRun?

I wonder if it makes sense to build up a WASM environment on a union filesystem (layers) as it mostly seems to in Docker.

this is hyperbole, wasm is just code, it has nothing for data, networking, instrument, observability. How do you deploy your stack to wasm? You compile all the services to it, then what? Write more code to wire them together?

Reply on news.ycombinator.com