Acorn: A simple application deployment framework for Kubernetes
acorn.io
acorn.io
But the real problem is Kubernetes. What do I mean by that? I mean the setup, care and feeding of K8s.
It's nice to have a tool that you can quickly push stuff to K8s. There are other tools out there that do that too.
The thing is this, the majority of people who use K8s, don't need to use K8s. Those people could easily get by with something like Dokku or even just start containers by hand to deploy.
Acorn is just perpetuation the cycle of recommending the over-engineering of K8s for small workloads and small companies that really and truly don't need it.
I'll say it again people. Spin up a few giant servers and call it a day. Save thousands of dollars in hosting costs and tens of thousands if not hundreds of thousands of dollars in engineer hiring costs. Bonus side effect? 10x performance than a passel of wimpy VPS's.
P.S. I'm coming from using K8s in both Google Cloud and on Bare Metal.
You're fair to argue then we should have a less complicated tool that's being standardized on but Kubernetes is where the industry is congregating now and that less complicated tool, that satisfies the most use cases and a high level of standardization, has yet to emerge, there have been multiple that have tried but never gained traction or outright proved to actually be more complex to manage once you go beyond small businesses.
Personally as a maturing profession we should stop reinventing the wheel for much of our tooling, adopt common tooling across as much of the industry as we can and then draft and produce standards engineers can work by so we stop having so much slop across the industry.
moving between cloud providers is rare. being able to transfer knowledge easily is a bigger benefit of k8s.
Does the loadbalancer know how the healthceck is implemented?
How do you determine it's time to scale?
How do you implement always-on-process? service unit, initd, cron?
How do you export the logs?
How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config?
What about secrets?
Service discovery? Is unbound/bind9?
And on and on and on and on.
These items are best done in a standard way. That's a major selling point in k8s. You define and implement these in a consistent way that also integrates seamlessly with other tools. Doubly so on cloud. Less reinventing the wheel means less money. The more teams, the faster this becomes exponentially expensive. Take this into consideration before your next interview.
At least when US east is down, half the internet is down.
And you learn from the situation, understand what are your new tolerance for data loss, failures, etc., then start new plans accordingly.
I think if you’re a small org with just a few devs and one or a handful of services, sure, K8s is overkill. But for a decent sized dev org, it’s probably way better and significantly simpler than what you’d rollout without it.
> Does the loadbalancer know how the healthceck is implemented?
Write a ping/sping endpoint like people did pre-Kubernetes? LBs provided by Cloud provider all have a trivial way of setting a healthcheck endpoint. If you are not in the cloud, nginx and HAProxy make it equally simple to configure a healthcheck. There is no k8s secret sauce behind healtchecks.
> How do you determine it's time to scale?
Monitor relevant metrics (CPU/Memory/DB), have alerts and add new vms or whatever when they hit a specified threshold. Or use your cloud provider's "elastic scaling".
> How do you implement always-on-process? service unit, initd, cron?
Yes. And the problem with that is? Processmanagers have been around forever, are simple to use and a well understood technology.
> How do you export the logs?
Cloud providers generally have a built in mechanism for pumping logs out, if you are not a cloud provider log export daemons exist that much simpler to work than running k8s deployment.
> How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config?
Do whatever your organizations feels comfortable with... config files, json, .properties files, env vars whatever, it doesn't really matter, since the code needs to be aware of the configuration mechanism anyways it doesn't make a difference. And again not worth the tradeoff of running full blown k8s deployment with its significant complexities
> What about secrets?
Cloud providers have convenient ways of solving this problem without involving k8s. If you are not on the cloud one can use cloud secret solutions (AWS Secret Manager, Google Secret Manager, Cloud Vault). All of which are much simpler to use than k8s secrets
> Service discovery? Is unbound/bind9?
DNS works well enough for most projects. Static configuration which lists domain names of whatever services you depend on (which are behind a load balancer) likely works for 90% of all applications.
I am not trying to be argumentative just making a point that "standardizing" all these things you mentioned "the Kubernetes way" are not worth the significant complexity of running and managing k8s for most organizations. And certainly, non-k8s ways of doing these things are much simpler to reason about for developers. A team of 2-3 developers can handle of these things and be able to keep the entire state of their system in there head, whereas with k8s it is suicide to have a k8s cluster without having a dedicated devops team.
And I disagree that there needs to be an industry wide standard of doing these things in the first place (and certainly not a standard like k8s that adds mountains of complexity). The standardization if needed, needs to happen at a team or organization level. Individual organizations should be free to pick and choose how they solve these problems rather than be pressured into the complexity of k8s.
There is a way to get all of those necessary things without having the massive complexity of k8s - use Nomad, a vastly simpler orchestrator that does many things better than k8s - e.g. integrated templating instead of go templates over yaml with Helm or other similar abominations; not using yaml for configuration; self-contained configuration and handling of updates (e.g. if your ConfigMap or Secret in k8s change, you have to manually restart/reload your pods; with Nomad you can configure it to handle that directly). It doesn't have nearly the same ecosystem, but there's enough of it through CSI, CNI, various load balancer and autoscaling integrations that you get very far with a production environment.
Any good tutorials with Noman + Containers would be nice for CI/CD/Executor setup.
That doesn't mean Nomad can't orchestrate containers (in fact that's the main use case for it).
Most use of Vagrant I saw was in demoing how to set up and deploy a cluster. So deploying 3 Nomad nodes within 3 VMs created in Vagrant.
https://news.ycombinator.com/item?id=32588931
That makes be a bit wary to use it because I would like my cloud storage to be reliable.
(I work for HashiCorp on Nomad)
The cloud providers generally aren't actually doing any significant amount of work on your behalf on the etcd front.
Etcd v3.5.[0-2] is not recommended for production => https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQ...
https://github.com/etcd-io/etcd/issues/13766#issuecomment-10...
Also, a lot of these products are missing a lot of documentation. I often find the tutorials are inflexible and focus on only 2 use cases out of the 4 most common. For example, focus on cloud when lots of engineers would rather deploy on bare metal. I would love for there so be one simple Nomad ansible role that would help setup nomad on bare metal with wireguard networking. This would probably be easy if consul was not required and it did not need so much babysitting.
Since Nomad 1.3, Consul is no longer required for Service Discovery, and since 1.4 Vault will no longer be needed for KV secrets.
And for what it's worth, i used to run Nomad+Consul in production on bare metal and it was easy to setup (2x 3 servers with a binary to start with some basic conf) and maintain (an update every month or so, which consisted of upgrading the binary on each node and restarting the service after 5 mins of reading the release notes).
Career wise all major platforms (Azure, GCP, and AWS) offer K8s, and therefore it is much better to learn the principles of K8s instead of focussing on the relatively obscure Nomad platform.
This part is still really hard for a lot of workloads. We have some applications that have a really long startup time (5-10minutes) and also have on the whole predicable traffic patterns. It'd be great to be able to tell k8s to scale up/down based on time of day in addition to the usual metrics without having to resort to a hacky cronjob to adjust the min replicas.
Sure it won't help with a unexpected traffic spike, and it would be better if the applications could come online faster (vendor app out of our control) but most days this feature would allow us to right size the workload way easier.
There's been a few issues like https://github.com/kubernetes/kubernetes/issues/49931 but it doesn't seem to be going anywhere right now
[0]: https://keda.sh/
k8s is not a "standard" in any way. It is clunky, prototype-level code that has never been engineered for elegance or ease of use. It's pretty much the same problem systemd has, except to an even greater extent (because clustering is inherently a harder problem than tweaking a single system to enable the use cases systemd devs care about).
How did you come to this conclusion? And what did you compare to? I'm not a devops so I don't have experience with Kubernetes.
For example, even something as basic as Docker Swarm will see you a lot of the way through.
> How do you implement healthcheck?
Supported by Docker: https://docs.docker.com/engine/reference/builder/#healthchec...
> Does the loadbalancer know how the healthceck is implemented?
When the health checks pass in accordance with the above config, the container state will change from "starting" to "healthy" and traffic will be able to be routed to it. Until then you can have a web server or whatever show a different page/implement circuit breaking or whatever.
> How do you determine it's time to scale?
Docker Swarm doesn't have an abstraction for autoscaling, though there are a few community projects. One can feasibly even write something like that themselves in an evening: https://github.com/jcwimer/docker-swarm-autoscaler
That said, I mostly ignore this concern because I'm yet to see a workload that needs to dynamically scale in any number of private or government projects that I've worked on. Most of the time people want predictable infrastructure and being able to deal with backpressure (e.g. message queue), though that's different with startups.
> How do you implement always-on-process? service unit, initd, cron?
The service abstraction comes out of the box: https://docs.docker.com/engine/swarm/how-swarm-mode-works/se...
You might also want to decide how to best schedule it: wherever available, on a particular node (hostname/tag/...) or on all nodes, which is actually what Portainer agent does! Example: https://docs.portainer.io/start/install/server/swarm/linux
> How do you export the logs?
Docker supports multiple logging drivers: https://docs.docker.com/config/containers/logging/configure/
> How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config?
Docker and Compose/Swarm support environment variables: https://docs.docker.com/compose/compose-file/#environment
If you need config files, you can also use bind mounts: https://docs.docker.com/storage/bind-mounts/
> What about secrets?
Docker supports secrets out of the box: https://docs.docker.com/engine/swarm/secrets/
> Service discovery? Is unbound/bind9?
Docker Swarm supports built in DNS, even allows for multiple separate networks: https://docs.docker.com/engine/swarm/networking/
> These items are best done in a standard way.
Agreed! Though I'd say that the only two options being "running everything on *nix directly" and "running everything in Kubernetes" is a false narrative! The former can work but can also lead to non-standard and error-prone environments with a horrible waste of human resources, whereas the latter can work but can also lead to overcomplicated and hard to debug environments with a horrible waste of human resources.
The best path for many folk probably lies somewhere in the middle, with Nomad/Swarm/Compose/Docker, regardless of what others might claim. The best path for folks interested in a DevOps career is probably running on cloud managed Kubernetes clusters and just using their APIs to lots of great results, not caring about how expensive that is or how easy it would be to self-host on-prem.
I am yet to see this warning when running any workload on Swarm. Feature complete is not necessarily deprecated, Swarm simply won't get any fancy cloud platform integrations due to hype lying elsewhere instead, yet there has been no official word of an EOL, like: https://www.centos.org/centos-linux-eol/
Of course, one can also argue that with the Mirantis acquisition the writing is on the wall and if you care about maximum hireability, then you should grok Kubernetes as well.
None of that prevents me from also running Swarm in setups where it makes sense (e.g. a step up from Docker Compose, limited resources, not lots of time for DevOps related stuff etc.).
> I bet Nomad has no future and will surrender in the coming years.
I'm not so sure about this, Nomad integrates nicely in the rest of the HashiStack and it seems that there are plenty of large orgs investing in it, even when they don't need something like Consul or other offerings, due to its simplicity and more active development.
Currently this claim could use more of a factual basis, otherwise it ends up sounding like ".NET will kill Java" or "Linux on the desktop will kill Windows" or something of that variety.
> Using anything but k8s in 2022 is not wise.
If you live exclusively in the cloud and use only managed offerings, that may as well be true.
Otherwise it seems like a clear clash between the arguments of "pick the right tool for the job" and "pick whatever is popular and widely supported". In the deployments I've worked with, Docker Swarm is better for the first claim (limited complexity, low resource usage, easy to use) which matters more than the latter (on-prem). Someone else who runs stuff primarily in the cloud might have the opposite takeaway, due to how widely Kubernetes is supported, even despite the technical complexity or resource usage.
That said, personally I think that the K3s Kubernetes distribution is one of the better options for many of the deployments out there: https://k3s.io/
In addition, tools like Kompose can make moving from Swarm to Kubernetes something that would take less than a day: https://kompose.io/ (or you can just use Helm), but until the day where Swarm will get a clear deprecation notice or I'll have a good reason to put a particular project on Kubernetes (as some already are), I'll just stick with Swarm/Nomad.
Of course, if a team/employer tells you to just use technology X, do that. Or better yet, make running your OCI compatible containers someone else's problem.
And it enables easier distribution of software. Layer one is source code. You need to build it. Layer two is OCI image. Smart people, often original developers did the build and Linux dependency management, and you just need some runtime to run it. But you still need to manage external service dependencies, which ports to expose, which HTTP paths to expose. K8s yamls with some generator like Helm or Kustomize complete that picture, so you can deploy some service along its dependencies with one command.
Docker-compose could be a better alternative. I like it more. But it’s not feature-complete (e.g. no ingresses, security is a complete afterthought) and at this point will not be a viable competitor.
> How do you implement healthcheck?
Most applications implement a simple endpoint which returns a known answer (whether it's an HTTP 200 code, a banner, etc.) You can just hit this endpoint using your favorite tool.
> Does the loadbalancer know how the healthceck is implemented?
Well, a lot of tooling supports native (periodic) healthchecks, e.g. NGINX or NGINX Plus (if you want to pay something else than managing K8S).
> How do you determine it's time to scale?
It is not that expensive to run performance monitoring such as netdata, sometimes, it is just a call to your distribution package manager, put it in your intranet, then you're done.
> How do you implement always-on-process? service unit, initd, cron?
Depending on what you're using in Linux distributions, why bother with initd and cron if you're running on a systemd-based distribution? Just use unit, timers, service, scope, slice and so on.
> How do you export the logs?
Again, if you're using a systemd-based distribution: `man systemd-journald-remote`.
Otherwise, Loki, rsyslog, many solutions do exist for these scenarios and do not require that much fancy configuration.
For mono-nodes, I would argue that the null logger exporter, called: "rsync your logs somewhere", is quite efficient.
> How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config?
I do not even know why people worry about that. If you're using a systemd-based distribution, your configuration should live more or less in the service unit using Environment, EnvironmentFile and LoadCredential(Encrypted), if your app want some files, sure, put it in /etc/xxx or /var/lib/xxx/config.
As long as you do separate configuration from state (and use StateDirectory, etc.) and you do backup them, it is okay.
> What about secrets?
As said earlier, LoadCredential do the job, if you want to be fancy, you can run a UNIX socket server to distribute the secrets and some secret engine such as HashiCorp Vault.
Uncertain that K8S has a better story than that.
> Service discovery? Is unbound/bind9?
Uncertain you actually need service discovery in early stages. Assuming you need, you could run envoy if you have advanced needs. Or you could run PowerDNS/nsd/unbound/BIND9 as you said with a thin wrapper to register your services.
> And on and on and on and on.
Well, I do not agree, those are items you choose. Not everyone has these requirements, and not everyone needs to solve them the k8s way.
> These items are best done in a standard way. That's a major selling point in k8s. You define and implement these in a consistent way that also integrates seamlessly with other tools. Doubly so on cloud. Less reinventing the wheel means less money. The more teams, the faster this becomes exponentially expensive. Take this into consideration before your next interview.
That is a fallacy, k8s is not a standard way, they push for their standard way and it is in no case what everyone is doing.
Defining and implementing thing in a "consistent way" is something we have been doing for a long time before k8s, through standardization process with IETF and W3C for example. k8s creates its own world, and it is indeed compatible with the external world sometimes, but I do not believe that a pile of complexity on the top of YAML files and YAML manipulation tooling is a "consistent way" to define and implement things, no offense here.
> Doubly so on cloud.
Yeah, because the "cloud ecosystem" is pushing for this, but it has not been a standard.
> Less reinventing the wheel means less money.
Using a "simple feature" is not reinventing the wheel, but if we go this way, I have a challenger: NixOS. Just reuse their expert modules database and do your classical YAML stuff, but this time with a simple functional programming language.
> The more teams, the faster this becomes exponentially expensive. Take this into consideration before your next interview.
I think GP was discussing of an early stage and I do not think that all teams require to push at a very fast rate, there is definitely time for packaging and shipping it. BTW, in these organizations where I have worked, they adopted k8s and these "teams" could not use it because it was too complicated and wondered where their packaging team has gone.
There is no silver bullet and choosing a deployment platform stack is not a figured out science, but there is a lot of value of choosing composable and simple tools before moving to more integrated ecosystems.
> Take this into consideration before your next interview.
That is an amazing level of arrogance, I only hope you may accept there are alternatives. :)
Maybe, but the pro-k8s camp always assumes everyone is building microservices or web services that need to scale.
K8s really shines when what you want to do is to optimize usage of your resources (I'd argue especially when running on bare metal)
K8s might be too hard to use right now. Too expensive for simple websites. That’s a problem we should solve.
But that's the obvious runtime the market is interested in right now. So what we are focusing on is let the infrastructure/ops team do their k8s thing, but insulate the devs/people that just want to run apps from it.
If real demand for it appears, with that abstraction we could write an alternate backend that talks to say Nomad or a-fancy-shell-script-that-runs-docker-cli instead of k8s. And then your app's Acornfile could now run on a totally different runtime, with no (or maybe minimal) changes.
https://thenewstack.io/acorn-a-lightweight-portable-paas-for...
Sure, deploying a single app config is much more awkward than it should be. (Build config: Helm, Jsonnet, Cue, kustomize,...? How to deploy: Gitops/CI, manually , Octopus, Tekton, Argo, ... ?)
But the real difficulty is in connecting different parts of your architecture.
The Getting Started example is extremely flawed: for local development you want a local Postgres and Redis, but in production those will either be provided by the environment (RDS etc) or self-managed,but as a separate deployment because the authorization/lifetime/scaling/upgrade requirements are very different.
Local, staging and production are so diverse that you will always need extensive customization. And there is no way around that, because the needs are fundamentally different.
There definitely is an opportunity to make this easier. It would have to either be done with tooling that has extensive integrations with cloud providers, or with a distinct end to end platform.
But I don't see how Acorn helps with that all.
I said: until you give me two consistent applications I can’t give you consistent patterns!
Literally everything is unique and special and has distinct requirements for each environment.
The orgs that can get the economies of scale are doing some combination of:
- developing everything in house for the same platform in the same langue with the same tooling.
- deploying thousands of apps, which means even rare configurations will have dozens of identical examples.
- never deploying weird proprietary apps with quirky licensing or crazy pricing models for non-production.
- don’t care about the cost of non-production and are happy to overpay for hardware to ensure consistency with production.
Even at large government departments none of the above is true.
This results in many unique deployments and no cost savings from templating or automation.
In one company they were spending 400k in aws for non production envs (of a 1.2M budget we committed to spend to get better rates).
Tons of microservices, all in node (with three exceptions), a mix of managed databases.
Things were consistent across envs (only the number of replicas were different, same containers) but we could have used less envs.
Disclosure: acorn creator.
It's not clear what problem Acorn is solving.
Forgive me if this is obvious, but isn't Acorn open source[1]? How is it proprietary?
(Though also, they’re technically correct — “proprietary” just means “controlled by one party.” Open-source things can still be proprietary: namely, if you’re free to fork, but the original upstream can only be merged to by employees of a corporation. I believe e.g. most of Apple’s “open source” is proprietary in this sense.)
Awesome, thanks for this explanation -- this is why I asked. I had always equated proprietary to closed source. Good to have my definition expanded. Thank you!
> Caution: Consuming this project can expose you to chemicals, which are known to the State of California to cause cancer and birth defects or other reproductive harm.
https://github.com/acorn-io/acorn/blob/8a474b9e593956c2ad56b...
[1] https://www.nytimes.com/wirecutter/blog/what-is-prop-65/amp/
[0]: https://www.brightbox.com/blog/2020/02/12/rubynetes-kubernet...
Use a language rather than a tool if you need the flexibility.
The problem with a fully fledged language is the flexibility. There is enough of it to make a mess unless you and your teammates are highly skilled and disciplined. Messy code sucks to deal with, debug, modify; let alone when it's infrastructure code where a wrong logical loop can delete your database or persistent volume.
Infrastructure as code, not with code. DSLs, with their limitations, ensure there's a lot less you can shoot yourself in the foot with, and that everyone vaguely familiar with that DSL can read your code and be absolutely sure what it does, worst case scenario with a Google search or two. No such thing is possible with code in a language as complex and as flexible as Ruby, Python or JavaScript.
Disclaimer: i work at Hashicorp that happens to have one of the most popular IaC DSLs out there, but i have believed that tools like Pulimi are little more than a disaster waiting to happen long before joining.
That's kind of why personally I'd just go for Helm or something of that variety - popular enough to have plenty of tutorials and answers about it, simple enough to just be viewed as a bunch of templating, going from values.yaml to your filled out template files that can be applied against a Kubernetes cluster.
It even generates a lot of what you might need (deployment, service etc.) as a part of the base set of templates, thus you don't have to waste your time trying to write manifests from scratch and have plenty of workable examples for setting env values etc.
It looks like Acorn is not related to Rancher at all beyond the creators being former Rancher founders.
Yaml doesnt spark joy for me, but it's ok & there's lots of good tools & libraries to support it. Json & toml are basically the only other modestly well supported options out there.
This looks like a Yet Another Next Generation (YANG) variant. At least just use YANG. Even though it's vastly less supported.
For real, the amount of griping about YAML is unbelievable. It's not great but so what. It does the job. What people really are griping about usually is how much configuration/state is required. Where the spaces go, whether there's parenthesis or not is so far down the list of important factors. Metric miles behind "are there good tools & libraries?," for one.
It would seem however from the responses I've been getting, that though YAML can be difficult or messy it's still better than another custom format.
JSON if you hate comments
So if acorn is also a different use case or level of abstraction, it can't really reuse those formats. Their set of 8 top level structures[1] seems reasonable at first glance to me.
[1] args, profiles, containers, volumes, jobs, acorns, secrets, localData
Why not just use yaml as a syntax? At the top of the document they already say it’s very similar anyway, I don’t see what this does that can’t be done with yaml?
Ex. ports: [ "80/http", "8080/http", ]
While a stylistic choice, it also makes it a PITA for others to write parsers for their DSL, and generally isn’t as readable as say the vanilla Kubernetes port model.
Next let’s take a look at the level of abstraction provided. Typically with a PaaS you are at a higher level of abstraction than an IaaS platform.
args: { // defines arguments the consumer can provide } profiles: { // defines a set of default arguments for different deployment types } containers: { // defines the containers to run the application } volumes: { // defines persistent storage volumes for the containers to consume } jobs: { // defines tasks to run on changes or via cron } acorns: { // other Acorn applications that need to be deployed with your app (databases, etc.) } secrets: { // defines secret bits of data that are automatically generated or passed by the user } localData: { // default data and configuration variables }
Looking at the list above of what we need (or can) provide for a deployment, it doesn’t seem we are at any higher a level of abstraction than we are when just using Kubernetes.
It’s incredibly easy to have the same artifact deployed in multiple environments using Kubernetes with no additional tooling.
So all this leaves me with the question, why?
These are CLI shorthands. There is a canonical expanded-out object that you would use if you were scripting something or talking to the API.
> Looking at the list above of what we need (or can) provide for a deployment, it doesn’t seem we are at any higher a level of abstraction than we are when just using Kubernetes.
Of course the same basic concepts of what to run, where to store output and how to take input are needed to describe computing. But there's two ways to run things ("containers" stay running, "jobs" finish) instead of 8 to choose from (bare pod, deployment, replicationcontroller, replicaset, daemonset, statefulset, cronjob, job) with all slightly varying syntaxes.
The args/volumes/secrets aren't just things that will be created, but also documentation and first-class objects that can be configured/overridden by the consumer of the Acorn. Without you adding dozens of lines of values.yaml and golang template soup to a helm chart to allow every possible configuration a user might want (and never quite the same way in two different helm charts from 2 different people).
You can create an image with reasonable defaults to work for dev/toying around on a laptop; run a local database, generate the secret for it automatically, etc. With the tools needed to ship and run that same artifact in production but pointed at a real DB in RDS. Packaged up as one OCI image so there's one artifact to manage from one source, instead of a helm chart + n images from n containers from n registries.
(Acorn cofounder)
1. Ugh, yet another DSL. Why not just use Lua?
2. The config file still looks complicated. What makes this better than writing your own Helm chart or Docker Compose?