Show HN: Simplenetes – I replaced Kubernetes with 17k lines of shell script
github.com
github.com
Seems like my click-baity title brought out some strong feelings :)
I do actually like Kubernetes. I recommend it to clients, I assess candidates who are to work with it, but I do really think it is a Beast. Because it is.
Kubernetes is like C++, extremely useful but it just keeps growing and nobody really knows all of it.
While Simplenetes is like... Lua, batteries not included, your boss will not approve, it doesnt't hit the buzzspot, but some people will secretely use it and be happy about it.
I was thinking why are people talking about virtio so much.
There are many factors in complexity, but a big one for me is what I have to hold in my head – global state. From any given point, what do I need to understand to know what's actually going on.
For Go, this is actually comparatively little. Between a fairly straightforward language, static binaries, and the focus on ease of distribution, there's not a whole lot outside of a codebase itself to think about.
However for a shell like Bash there's a relatively large amount to consider. You're very exposed to the underlying OS, to libraries and system packages, to various tools, to configurations users may have specified, to things being redefined underneath you. There's a lot of global state to consider for a Bash script.
I suspect it is simpler than Kubernetes, but I don't think it's a clear cut case.
I'm not sure you can with shell scripting because of all the global system state.
edit: wc reports 47023 lines in *.sh[1], out of which 23836 are unique[2].
[1]: find . -name '*.sh' | xargs cat | wc -l
[2]: find . -name '*.sh' | xargs cat | sort | uniq | wc -l
It can distinguish between actual lines of code, comments and empty lines for a lot of different languages.
It's the same as cloc, but ludicrously fast.
One of the key benefits of kubernetes are API server and custom resource definitions. If you don't like how kubernetes does stuff. You can change the default behavior and orchestrate the way you want.
Either way, setting aside the laudable amount of work put into this, talking about the issues of complexity and different forms it takes seems quite on topic. (I don't have a cow in this fight, not using K8s or anything of the sort.)
> 17k lines of shell script
those two phrases don't belong together... to think about this, "17k lines" and "shell script" should not be used together either.
Also, you are a bit light on tests.. there only seems to be a single trivial one, which seems way too low for a codebase this size.
About the tests, can't disagree with you, they will come.
We just moved Space.sh from GitLab to GitHub (hurts in my heart, but yeah community is on GitHub).
Also we are lousy at promoting our stuff, maybe April 1st is not the best day to do this either...
I'm surprised when people say this. I thought communities were only around repos.
I assumed it was tongue-in-cheek?
portability across all the different unices with their weird c compilers and differing libcs that actually worked out of the box every time was a gargantuan task.
Bulk of complexity in opensource distro of kubernetes comes from the fact that everything is interface driven and has to work in an generic way. For example, you don't really have to implement cri-o like thing if you are opinionated platform and only supports docker. Similarly, you don't have to build CSI, if you only support ceph or aws' storage. I love the contrarian thought, but matter of the fact is modern orchestration platforms that want to be a general purpose platform is going to be big.
It is a really nice exercise however build something like this when you want to learn about networking, containers, orchestration, storage etc. and how to tied everything together.
However, if you are just dealing with a three node cluster (Yes I know many of you are running tri-cycles with k8s stickers on it) then maybe it can be fine for many applications to not have a scheduler.
Simplenetes does however support multiple replicas of pods so if one node fails it can be OK.
Why do I need 17k lines shellscript to run my tri-cycles? I would rather recommend using docker swarm or just podman itself.
you are adding more complexity without to a very simple problem.
Perfect, complete and too complex.
For big and great applications, spring if the obvious choice. It makes incredible complex and difficult things easy.
But, unfortunately, makes simple and quick applications incredible complex and difficult.
As a short-list sampling, a Micronaut SQL guide[0] lists the following:
JDBC
Hibernate
JAsync SQL
jOOQ
Jdbi
Reactive MySQL Client
Reactive Postgres Client
I myself chose JDBI for more recent projects using Java/Kotlin.[0] https://micronaut-projects.github.io/micronaut-sql/latest/gu...
what would you use these days?
Edit: I just did a google search for "Java Framework" and get a lot of useless top-N lists with old content retitled "in 2020" with hits like: Hibernate (not a framework), JSF (JavaServer Faces), GWT (Google Web Toolkit), Struts (The Later Version), DropWizard (seems to have stagnated and docs are all over the place. JDBI was extracted and has a life of its own).
If you really want something exotic in Scala that is impossible to express in Java, Go, Ruby, Elixir etc., then take a look at the Typelevel (cats, cats-effect, http4s, doobie) ecosystem or ZIO. If purely functional programming is your cup of tea.
BTW, I did work at a startup for years building microservices with Spring which is where I ran into it's limitations. I basically worked around the EntityManager, JPQL, and query template caching where ever it was necessary, which was frequent. I'm not against query builders or data mappers, having worked on making one myself[0]. I'm just more performance/resource conscious than most.
[1] - https://quarkus.io/ [2] - https://micronaut.io/
While it's true that tech choices are signals that enterprises do use to determine quality, if you are relying on a particular choice of programming language to sell into major enterprises that's simply a mistake. (And enterprises have invested vast sums in technology written in c# or PHP, so clearly it's not all "lol" out there.)
Rust's and Go's capabilities of compiling into a single binary, without requiring a JVM that needs to be fed and cared for, make them a particularly compelling choice for enterprise deployments. Kubernetes itself is written in Go.
persistence: jooq + postgres
search: postgres full text search
cache: ehcache
* for the umpteenth time i daresay that hn needs to grow a sense of humor sometimes...
this is a cool idea.
next up, somebody will modernize daemontools for the container/cluster era...
classic rookie mistake to just call a program that doesn't exit a daemon...
proper daemons can stay up for years.
Running "in the background", handling of stdin/out/err, logging, detaching from tty, creating a new process group/session, closing fds, and much more is handled these days mostly by systemd. No need to put all this into your program at all.
I'll give you that signal handling still needs to be done in the target process, if the defaults don't satisfy you.
But you really don't need anything special anymore to make any old script into a daemon.
if someone else runs it on a system without systemd it won't work correctly?
Without systemd you can use daemonize from shell, for example. I either run these kinds of service scripts via systemd in production, or via my special program that takes care of starting and monitoring, and restarting when the code of the script changes.
Why should I stuff daemonization machinery into every single script I might want to run as a daemon, especially when it's cumbersome in many languages where I want to do it, like bash or PHP, and in general less flexible than using some external tool?
it is true that it's rare to see a daemonize flag these days...
If it's something like sshd, it better not prevent any filesystems from umounting unless necessary.
Btw. the cluster consists out of three servers on Hetzner [2] and runs our web analytics platform Pirsch [3].
[0] https://www.youtube.com/channel/UC-AdvAxaagE9W2f0webyNUQ
[1] https://learn.hashicorp.com/consul (for Consul, the pages for Nomad and Vault are similar)
Part of what something like k3s enables is this: https://www.acc.af.mil/News/Article-Display/Article/2557413/...
They don't mention that this was accomplished via k3s, but I happen to know it was because it was colleagues of mine that did this. The point of being able to do something like this is you can deploy a completely identical stack of admin/controller level software to systems running in data centers and systems running in airplanes. As long as they can run kubernetes, you're good to go, and with k3s, just about any piece of hardware can run kubernetes. This has huge implications for defense because it means you can develop and test software that isn't directly responsible for specialized hardware control without having to build emulators or lab equipment identical to what is installed in the actual weapons platforms. You just need to ensure they can both run kubernetes.
It doesn't have to be kubernetes, obviously, but you get the advantage of a fairly rich ecosystem that includes things like k3s, which doesn't even require bash or a shell at all to work.
2. Why do people write "Simplenetes has a 100x less code than Kubernetes" instead of saying it has 1/100th the code?
By contrast, Kubernetes has at least 1000x more developers working on it
It’s way too complex already.
For comparison, IIRC WebKit had like 500 contributors as of five or six years ago.
The easy stuff:
- Non-exported variables are by convention lower_case, while exported variables are UPPER_CASE.
- Function names are by convention lower_case.
- There's a "quite" signature string and variable; should it be "quiet"?
- The lines like `local hosts=` can be simplified as `local hosts`. You can also join such lines, as with `export`, into `local foo bar baz …`.
- `[[` is generally considered more reliable than `[` (there are a bunch of posts about this on Stack Overflow and Unix Stack Exchange).
- Naming scripts .bash rather than .sh lets `shellcheck` inspect them without telling it which shell to verify against. You do check these files using `shellcheck`, right?
- Variable names like `list` and `tuple` are unhelpful; what do they actually contain? Also, they should probably be actual arrays rather than a space-separated string to avoid relying on word splitting, and to allow entries with whitespace in them.
The scary stuff:
- All the string packing and unpacking also makes me nervous. That's not simpler than using JSON/YAML, it's just a different kind of complexity.
- Reusing the same variable a bunch of times in the same function makes it really easy to end up with the wrong value at some point. Better to split stuff into more functions or rename the variables to make them single use.
- `set -o errexit -o pipefail` would be good to have for safety.
- `kill -9` anything[1]
- Why are some of the variables inside functions not local? They effectively end up being globals, which makes for really difficult debugging.
The good stuff:
- Using local variables everywhere.
- Descriptive error messages.
- Returning early in case of errors.
All in all, Bash is not a good language for any system of this size, no matter how diligently you program. The error handling isn't comparable to most mainstream languages, the data types are incredibly limited, and there's no native support of recursive languages like JSON. You've done a great job, but I'm afraid a system like this is doomed to failure from these facts alone.
[1] https://mywiki.wooledge.org/ProcessManagement#I.27m_trying_to_kill_-9_my_job_but_blah_blah_blah...A lot of good points.
Coding in shell sure is a challenge, I try to avoid bashims where I can, that's why most scripts are .sh not .bash, since they run also with dash/ash (that's why preferring `[` over `[[`).
The only Bash requirement we have is the `podc` (pod compiler) since it parses YAML docs, I couldn't pull that one off without the more powerful Bash.
The code base as a whole absolutely needs testing and shellchecks all over the place to be labelled as mature in any sense.
Our other large shell project is Space.sh [1] where go overboard on testing, each module is tested under a bunch of distributions. [2] [3] Next step would be to do the same for Simplenetes.
Given that in shell that the only way to protect a variable from functions down the call stack accessing it is to redeclare/shadow it as "local" brings some murky waters.
BUT, isn't it amazing what is actually possible to do with Shell??
For me Simplenetes in the long run is not about it being written in Shell, I'm perfectly happy rewriting it in Rust if we get traction on it.
Simplenetes is primarily about having a simpler architecture of what a container cluster is and is not. I think many projects just get too complex because they want to fill every single use case out there, while the interesting part is saying no to things.
[1] https://github.com/space-sh/space [2] https://github.com/space-sh/space/tree/master/test [3] https://github.com/space-sh/space/blob/master/.github/workfl...
Yeah, this represent a few months of full time work.
How can you afford this?
The real use case is super simple clusters, which could be just 1 node, no etcd, no API (SSH managed), no iptables.
Simplenetes is not a living breathing thing which auto heals it self, it's a deterministic approach where you can see your cluster with all hosts in the git repo which represents the cluster.
Also, it's focused on making local development easy, so you can develop in a micro services architecture straight on your laptop with very quick iterations.
Despite whatever the k8s crowd misuses the term to be, 1 node is not a cluster. It might be a single node managed using cluster-capable technologies, but it's still not a cluster.
Main upside is that I can put Kubernetes on my resume.
I'm in love! <3 :-D
Related: Bocker, Docker rewritten in 100 lines of Bash https://github.com/p8952/bocker
Quite an engineering feat, though, I am sure! Is this to have fun, to learn, to actually use in production?
But thank you for bringing the world another tiny step closer to abandoning k8s and k8s (over-engineering) mentality.
You’re doing so much for the world and for all the developers. Thank you!
While Kubernetes is hopefully Very Good, it is complicated, and I have often wondered what other approaches would yield (e.g. more imperative ones, because writing admission controllers is not exactly easy). So I'm glad to see people playing around, though I don't know if this amount of bash is healthy or if 17,000 lines is simple.
Impressive
Now I’m trying to figure out if I just got too old to appreciate them, or if the good ones have all been used up.
- No magic involved
says the one with a single file 13k LoC shell script.The source files involved are smaller [1].
[1] https://github.com/simplenetes-io/simplenetes/tree/master/in...
So that would seem to say the title is also counting that compiled version, when in fact it is much simpler. But then also, why is the compiled version longer than the sum of the files in includes?
The line count is a rough (overestimate) summoning the source of the three projects involved, simplenetes, simplenetesd and podc, and it also includes comments and blank lines.
The compiled output pulls in some dependency modules which is reusable code (STRING, etc). So it's not clear where to draw the line :)
Reason the compiled output gets bigger is because the compilation process `(make.sh)` includes compilations of actions, which I use when connecting over SSH agentless to manage the nodes. This tough part is handled by Space.sh [1]
-q Set to be quite.
I doubt it's intentional, but I kinda love it.Lighten up folks.
Also namespaces are a killer feature of K8s imo
Nice job
And some hapless person will be stuck maintaining and debugging it. It could be me, or you, your friend. You don’t want this to happen.
I think the 17kloc would discourage that kind of thinking.
Waste of time... Well... Let's not think too much about that. I'm at least a very happy user of this :)
That's shade?
It will be a useful learning experience.
The same goes about taking up k8s with no thinking, by the way.
The title seems both true (for the author's situation) and falsifiable. I don't think it's clickbait.