Harbormaster: Anti-Kubernetes for your personal server
gitlab.com
gitlab.com
This also worked very well for work, where we have some simple services and scripts that run constantly on a micro AWS server. It's made deployments completely automated and works really well, and now people can deploy their own services just by adding a line to a config instead of having to learn a whole complicated system or SSH in and make changes manually.
I thought I'd share this with you, in case it was useful to you too.
Not saying this would at all replace Harbormaster, but with DOCKER_HOST or `docker context` one can easily run docker and docker-compose commands without "ever logging in to the machine". Well, it does use SSH under the hood but this here seems more of a UX issue so there you go.
Discovering the DOCKER_HOST env var (changes the daemon socket) has made my usage of docker stuff much more powerful. Think "spawn a container on the machine with bad data" à la Bryan Cantrill at Joyent.
Agreed on it being a bit too heavy-handed, and the tooling isn't very helpful for dealing with it unless you're neck-deep into the ecosystem already.
You could put your SSH server configuration in a repo. You could put your SSH authorization key in a repo. You could even put your private key in a repo if you really wanted.
The nicety here on harbormaster seems to be that there are some ways to use the same code as a template in which specific differences are dynamically inserted by harbormaster. I'm not aware of how you could use docker-compose (without swarm) to accomplish this, unless you start doing a lot of bash stuff.
I also appreciate that harbormaster offers opinions on secrets management.
While I haven't used it personally, there is [0] Watchtower which aims to automate updating docker containers.
You run what's supposed to run the same way you would anything else. It's the same for the environment variables.
How would you track what's supposed to run and what's not for Docker? Using the `DOCKER_HOST` environment variable to connect over SSH is the exact same way.
I have never used Chef. This is babble to me.
In an imprecise nutshell: You specify what needs to exist on the target system using Chef's DSL and Chef client will converge the state of the target to the desired one.
To "converge" is to run something until it is stable. This terminology, I think, comes from the early configuration managemnt system CFEngine, where you write your configuration in declarative(-ish) "make it so this is true" steps, instead of imperative "perform this change" steps the way that a shell script would do. See e.g. https://www.usenix.org/legacy/publications/library/proceedin...
chef-solo is a command that executes Chef's client - the thing that actually makes configuration changes, that is to say, "converges cookbooks" - in a way that does not require a server component. The normal way of deploying Chef is that a server runs things on clients, the machines being configured, but chef-solo is appropriate for the case where there is no such distinction and there's just one machine where you wish to run Chef.
He has a lot to say about zones and jails and chroot predating docker, and why docker and co. "won" so to speak.
"Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017" https://youtu.be/30jNsCVLpAE
Ed: oh, here we go I think?
> Running Aground: Debugging Docker in Production Bryan Cantrill19,102 views16 Jan 2018 Talk originally given at DockerCon '15, which (despite being a popular presentation and still broadly current) Docker Inc. has elected to delist.
Single machine deployments are generally easy, you can do it DIY. The complexity arises the moment you have another machine in the setup, scheduling workloading, networking, setup to name a few, starts becoming complicated.
From my perspective, kubernetes was designed for multiple team, working on multiple services and jobs, making operation kind of self serviced. So I can understand the anti-kubernetes sentiment.
There is gap in the market between VM oriented simple deployments and kubernetes based setup.
if you decide to grow past 1 node, it's a little more complex, but not by a lot, like k8s.
[1] https://juju.is/
At this point, I think Juju is most likely used in place of other metal or VM provisioning tools (like chef or Ansible) so that you can automatically provision and scale a system as you bring new machines online.
The single-file YAML config (so it's easy to discover exactly what's running on the server), the separated data/cache/archive directories, the easy updates, the fact that it doesn't need built images but builds them on-the-fly, those are the big advantages, rather than the actual `docker-compose up`.
Anecdotal, but anyone I know running home lab setups that aren’t software guys are doing vSphere or Proxmox or whatever equivalent for their home usecases. But I know a lot of old school sysadmin guys, so YMMV.
You cannot avoid learning k8s, you will end up encountering it everywhere, whether you like it or not. It is the tech-buzz word for past few years followed by cloud native and devops.
I really thinking if you wish to be great engineer and truly respect new general tools in generally, you have to go through the route setting up proxmox cluster, loading images, building those VM templates etc. Jumping directly on containers and cloud you kind of skip steps. It is not bad, you do miss our on few foundational concepts, around networking, operating systems etc.
The way I would put it is - A chef who is also farming their own vegetables a.k.a setting up your own clusters and deploying your apps VS a chef who goes to high-end wholeseller to buy premium vegetables does not care how it is grown aka. developers using kubernetes and container orchestration, PaaS.
It took a bit to figure out setting up an https cert provider but then it was pretty much off to the races
What's wrong with Ansible? You can deploy docker containers using a very similar configuration to docker-compose.
Right now I have a deployment hook that can propagate an app to more machines also running Piku after the deployment finishes correctly on the first one, but stuff like green/blue and database migrations is a major pain and requires more logic.
In my experience, there are actually two platforms that do this pretty well.
First, there's Docker Swarm ( https://docs.docker.com/engine/swarm/ ) - it comes preinstalled with Docker, can handle either single machine deployments or clusters, even multi-master deployments. Furthermore, it just adds a few values to Docker Compose YAML format ( https://docs.docker.com/compose/compose-file/compose-file-v3... ) , so it's incredibly easy to launch containers with it. And there are lovely web interfaces, such as Portainer ( https://www.portainer.io/ ) or Swarmpit ( https://swarmpit.io/ ) for simpler management.
Secondly, there's also Hashicorp Nomad ( https://www.nomadproject.io/ ) - it's a single executable package, which allows similar setups to Docker Swarm, integrates nicely with service meshes like Consul ( https://www.consul.io/ ), and also allows non-containerized deployments to be managed, such as Java applications and others ( https://www.nomadproject.io/docs/drivers ). The only serious downsides is having to use the HCL DSL ( https://github.com/hashicorp/hcl ) and their web UI being read only in the last versions that i checked.
There are also some other tools, like CapRover ( https://caprover.com/ ) available, but many of those use Docker Swarm under the hood and i personally haven't used them. Of course, if you still want Kubernetes but implemented in a slightly simpler way, then there's also the Rancher K3s project ( https://k3s.io/ ) which packages the core of Kubernetes into a smaller executable and uses SQLite by default for storage, if i recall correctly. I've used it briefly and the resource usage was indeed far more reasonable than that of full Kubernetes clusters (like RKE).
> The only serious downsides is having to use the HCL DSL ( https://github.com/hashicorp/hcl ) and their web UI being read only in the last versions that i checked.
1. IIRC you can run jobs directly from UI now, but IMO this is kinda useless. Running a job is simple as 'nomad run jobspec.nomad'. You can also run a great alternative UI ( https://github.com/jippi/hashi-ui ).
2. IMO HCL > YAML for job definitions. I've used both extensively and HCL always felt much more human friendly. The way K8s uses YAML looks to me like stretching it to it's limits and barely readable at times with templates.
One thing that makes nomad a go-to for me is that it is able to run workloads pretty much anywhere. Linux, Windows, FreeBSD, OpenBSD, Illumos and ofc Mac.
I know HCL gets a lot of hate around here but I find just dandy and I'd rather write HCL over YAML any day of the week.
Also to just put another thumbs up for Nomad because it's been absolutely fantastic for us and has been so much easier to manage and deploy. We've begun testing edge deployments with it as well and so far looks very promising.
Consul and Vault are also two Hasicorp products I can't live without at this point either. Vault is probably one of the first things I deploy now.
Anyway rambling, I think a lot of people just immediately jump to Kubernetes without really ever giving Nomad a look first.
I agree that the format Kubernetes uses is probably a bit overcomplicated, which is why I've also sometimes used the aforementioned Compose format with tools like Kompose to save some of my time.
In that regard, it's perhaps not the fault of YAML itself (even though the format has its own complexity problems), but rather of the tool using it.
What personally keeps me away from HCL for the most part is the fact that a lot of the time you can find Compose files for various pieces of software online and only have to change some parameters for the most part to get stuff running vs having to rewrite all of it in a different format.
Are there any Compose to HCL tools out there yet?
And yeah, there is not much examples of HCL files for different software, but most of the time there is not much to adapt. Usually when i need to run something, i just take some random HCL file i have and substitute the docker image, edit the amount of resources and add storage and config via templates. And done. It might be confusing/intimidating at first, but like with anything after some time you should be ok.
When migrating from a non-containerized deployment process to a containerized one, there are a lot of new skills the employees have to learn. We've had 40+ employees, all who are basically full of work, and the mandate comes down to containerize, and all of these old school RPM/DEB folks suddenly need to start doing docker. No big deal, right? Except...half the stuff does not dockerize easily requires some slightly-more-than-beginner docker skills. People will struggle and be frustrated. Folks start with running one container manually, and quickly outgrow that to use compose. They almost always eventually use compose to run stuff in prod at some point, which works but eventually that one server is full. This the is the value of swarm - letting people expand to multi-server and get a taste of orchestration, without needing them to install new tools or learn new languages. Swarm adds just one or two small new concepts (stack and service) on top of everything they have already learned. It's a god send to tell a team they can just run swarm init, use their existing yaml files, and add a worker to the cluster. Most folks start to learn about placement constraints, deployment strategies, dynamic infrastructure like reverse proxy or service mesh, etc. After a bit of comfort and growth, a switch to k8s is manageable and the team is excited about learning it instead of overwhelmed. A lot (?all?) of the concepts in swarm are readily present in k8s, so the transition is much simpler
We generally avoid mounting volumes at all costs. The challenge of mapping host uid:gid to container uid:gid (and keeping that mapping from breaking) proved painful and not worth the effort
I created a shell script to easily set this up: https://github.com/badsyntax/docker-box
There's different kind of workloads, I use Docker containers the most, but jobs can also run on a system-level, there's also different types of operating modes, some jobs can be scheduled like cron, where other jobs just exposes a port and wants to be registered in Consuls service-mesh.
A job can also consist of multiple subtasks, an example could be nginx + django/rails subtasks that will be deployed together.
You can see an example of a Docker job here: https://www.nomadproject.io/docs/job-specification#example
With a few modifications you can easily allow for blue/green-deployments.
I'm at: gordon dot stewart 333 at gmail dot com
The only things I'd change are switching to Caddy instead of Traefik (because Traefik 2.x config is just so bewilderingly complex!), and I'm not convinced Portainer is really adding any value.
Appreciate you sharing your setup script too.
My problem is in the two to eight server space, but networking is already externally managed and I have a loadbalancer. It’s in this space I feel that we’re lacking good solution. The size is to small to justify taking out nodes for a control plane, but big enough that Ansible feels weird.
(You can use docker-compose with it as well, but as a deployment step — I might bake in something nicer if there is enough interest)
Thanks for that, Rui!
It has also been deployed on all top 5 cloud providers via could-init (and I’m going back to AWS plain non-Ubuntu AMIs whenever I can figure out the right packages).
My original use case was _exactly_ that (MQTT services).
systemd has "slice units" that are implemented very similarly to Docker containers, and it's basically the default on every Linux system from the last few years. It's underdocumented but you can read a little about it here: https://opensource.com/article/20/10/cgroups
Sure, you could publish your app as a .tar.gz of the filesystem root and then users could extract it and have a script to bind mount everything and chroot into it. You could then set up a systemd service(s) for that script and set up a slice. But then you're just reinventing Docker from scratch for absolutely no reason. You could also use systemd-nspawn, but then you're losing out on a bunch of really useful features that the devs omitted because of philosophical reasons (as usual with systemd, they think they know better than you).
That said, the project looks pretty good! I'll have a tinker and maybe I'll be converted
With Harbormaster, I just copy one YAML file and the `data/` directory and I'm done. It's extremely convenient.
Isn't that exactly what watchtower does?
https://github.com/containrrr/watchtower
It works great on my mediacenter server running deluge, plex, sonarr, radarr, jackett and OpenVPN in docker.
Then again, Harbormaster doesn't do that either unless the upstream git repo changes.
My server was much more stable after it didn’t try to update all the time any more.
I wonder if I can set a minimum timeout.
I want to "read all available updates" at my convenience, not get alerts reminding me to update my server.
Maybe I need to write some sort of plugin to DUIN that appends to a text file or web page or SQLite db... Hm.
Personally, since I’m a big fan of RSS, I’d set up email in Diun and send it to an email generated by https://kill-the-newsletter.com/
Currently, if you want to scale Dokku horizontally and aren’t ready to take the kubernetes plunge, you have to put a load balancer in front of your multiple VMs running Dokku and that comes with it’s own headaches.
My biggest complaint would be the downtime when the docker script runs after each deployment.
It's kind of abandonware because it was the developer's PhD project and he graduated, but it is rather unfortunately widely used in one of the largest GEOINT programs in the US government right now because it was the only thing that offered this capability 5 years ago. Raytheon developers have been begging to fork it for a long time so they can update and make bug fixes, but Raytheon legal won't let them fork a GPL-licensed project.
"Someone forked it so now our fixes can get merged! :D"
I don't think that is the reason. When Raytheon or other contractors perform software work under a DOD contract (i.e., they charge the labor to a contract) the government generally gets certain exclusive rights to the software created. Raytheon is technically still the copyright holder, but effectively is required to grant the US government an irrevocable license to do whatever they want with the source in support of government missions if the code is delivered to the government. Depending on the contract, such code may also fall under blanket non-disclosure agreements. I believe both of these are incompatible with the GPL, and the latter with having a public fork at all.
The company could work this out with the government, but it would be an expensive and time-consuming process because government program offices are slow, bureaucratic, and hate dealing with small exceptions on large contracts. They might even still refuse to make the contract mods required at the end simply because they don't understand it or they are too risk averse. Legal is likely of the opinion that it isn't worth trying, and the Raytheon program office likely won't push them unless they can show a significant benefit for the company.
Especially the backup and Let's encrypt elements are great. And it handles docker networks, which makes it very flexible.
Will definitely check it out.
I made a similar thing recently as well, although with the goal to handle ingress and monitoring out the box as well, whilst still able to run comfortably on a small box.
I took a fairly similar approach, leveraging docker-compose files, and using a single data directory for ease of backup (although it's on my to-do list to split out conf/data).
If there was a way to get a truly slim and easy to setup k8s compatible environment I'd probably prefer that, but I couldn't find anything that wouldn't eat most of my small servers ram
https://github.com/mnahkies/shoe-string-server if you're interested
I'm ok with this level of overhead for the benefits of containerization - several years ago I was running a bunch of PHP sites on Apache, one was compromised and this took down the rest as well. In addition to the isolation it also simplifies my deployment story significantly.
If K3s has a similar level of overhead then I should probably move to this, something for me to look into, however my understanding was that components like etcd require a fairly significant amount of resource even for tiny one node "clusters".
I'll try to rework the README to hopefully make it more understandable, but looking at your project's README I get as overwhelmed as I imagine you get looking at mine. It's a lot of stuff to explain in a short page.
Instead of deploying changes as git commits, you deploy them as container image updates. I'm not going to call it a good solution, exactly, but it meant I could just use one kind of thing to solve my problem, which is a real treat if you've spent much time in the dockerverse.
It was using GitHub so just needed a read-only key and could be bootstrapped by connecting to the server directly and running the playbook once
In addition, it didn't need any special privileges or permissions. The playbook setup remote logging (shipping to CloudWatch Logs since we used AWS heavily) along with some basic metrics so the whole thing could be monitored. Plus, you can get a cron email as basic monitoring to know if it failed
Imo it was a pretty clever way to do continuous deploy/updates without complicated orchestrators, management servers, etc
What I couldn't immediately see from skimming the repo is:
How hard would it be to use a docker-based automatic https proxy such as this [1] with all projects?
I've had a handfull of docker-based services running for many years and love the convenience. What I'm doing now is simply wrap the images in a bash script that stops the containers, snapshots the ZFS volume, pulls newer versions and re-launches everything. That's then run via cron once a day. Zero issues across at least five years.
Sounds like a very good ingress solution, I'll try it for myself too, thanks! I use Caddy now but configuration is a bit too manual.
One thing to note is that you'll need to make sure that all the compose bundles are on the same network.
I.e. add this to all of them:
networks:
default:
external:
name: nginx-proxyI already added a config for Plex in the Harbormaster repo, but obviously it's better if the upstream app itself has it:
https://gitlab.com/stavros/harbormaster/-/blob/master/apps/p...
Traefik can be a bit hairy in some ways, but for anything you'd run Harbormaster for it should be a good fit.
Right now I have some Frankenstein situation with all of Traefik, Nginx, HAProxy, Envoy (though this is inherited from Consul Connect) at different points... I keep thinking about replacing Traefik with Envoy, but the docs and complexity are a bit daunting.
If is a startup, use some buzzwords like cloud native, devops etc. Check their sentiments towards kubernetes.
On a serious note, You might have to jump on the kubernetes bandwagon whether you like it or not as many of the companies are serious investing their resources. Having spoken to various companies from series A to Enterprise. I do see the kubernetes adoption is actually not as much as I would have imagined based on the hype.
P.S discussion of kubernetes or not kubernetes was recently accelerated by a post from Ably [1]
Simple one-time setup and then everything is a container?
If that interesting to OP then I might look into that one weekend soon.
I could also see this being great for a personal lab/playground server. Or for learning/workshops/hackathons. Super easy to get people running from 0.
If I ever run a class or workshop that has some server-side aspect to it, I'll keep this in mind for sure.
i just recently decided to graduate from just `docker-compose up` running inside tmux to a more fully fledged system myself...
since i know Chef quite well i just decided to use Chef in local mode with the docker community cookbook
i also get the nice tooling around testing changes to the infrastructure in test kitchen
if this would have existed before i made that switch, i may have considered it, nice work!
It sounds like a good option too, I don't want all the complexity of Kubernetes at home. If I worked for the cloud team in work I might use it at home but I don't.
K8s seems way overused in spheres without a real need for it. That would explain the « k8s is over complicated » I keep reading everywhere. It’s not over complicated, you just don’t need it.
https://wickedmoocode.blogspot.com/2020/09/simple-way-to-dep...
inb4: Flatpak is not as powerful as snap. E.g., you can't use a different level with Flatpack.
If anything, Python in Docker is more of a pain than bare-metal since the distro (usually alpine or debian) ships its own Python version and packages that are almost entirely disconnected from the one the image provides.
I kind of punted on the decision of how to run the top layer (ie have Harbormaster be a daemon that auto-pulls its config), but it's very simple to add a cronjob to `git pull; harbormaster` (and is more composable) so I didn't do any more work in that direction.
also kind of want RDS for my machine -- backup/restore + upgrade for databases hosted locally