Scaling Mastodon with systemd template units | Hacker News Reader

Scaling Mastodon with systemd template units

eigenmagic.com

69 pointsghuntley24 comments

    Environment="NODE_ENV=production"
    Environment="PORT=4000"
    Environment="BIND=::"
    Environment="STREAMING_CLUSTER_NUM=2"

You probably want to use EnvironmentFile= and shove that file with the rest of app's configs

And, uh, shove the queue size parameter there instead of

    sudo systemctl enable mastodon-sidekiq-default@25.service
    sudo systemctl enable mastodon-sidekiq-mailers@2.service
    sudo systemctl enable mastodon-sidekiq-scheduler@2.service

using feature designed to run multiple instances as a poor man's way to configure stuff. Let's just say nothing will stop you from also doing

    systemctl enable mastodon-sidekiq-default@15.service

and systemd will just try to start the sidekiq twice

The article's proposed way of using templates is TERRIBLE. I had assumed from the title it will show off how to run multiple instances using same template units (say to vary a run dir or whatever), but no, it is using them as glorified config file for number of threads/db conns. Just shove that into env file, you can use env parameters in rest of the file too...

You sound very knowledgable on systemd, could ypu share some suggestions for learning resources? I would like to push a better use of it in my org, but besides some unit file messing about and using all the cool anaylysis tools, I don't know as much as I'd like.

Well, I've written a blogpost on dependencies [1] but really reading all of the man pages (the most useful ones [2][3][4]) will get you a good idea on what it can do, I had many "oh, we can use THAT" moments with it. They are written pretty well.

The directives we got most use in org are probably Restart* family and RequiresMountsFor, althought the last one is more and more common in distro packages so it is not even needed that often. The usecase being "don't start DB if mount fails" usually.

Overrides are probably the most used feature overall, we often find that the standard distro unit just needs a little bit of tweaking for our purpose (like adding restart), so we just create file /etc/systemd/system/servicename.service.d/01-restart.conf with

    [Service]
    Restart=always
    RestartSec=30

and voila, service now auto-restarts and that behaviour won't break when package updates, as we're not changing any package file. The few other modifications of existing ones included adding new read-write dirs or adding env variables for proxy.

Technically you can even go as far as making services into little containers with all the options for limiting permissions but I haven't got that far. Most apps don't exactly need it either, have readonly/readwrite dir + some limits and it is pretty safe out of the box, I've also noticed many distro packages going that way.

* [1] https://devrandom.eu/blog/post/2022-02-27_increasing_system_...

* [2] https://www.freedesktop.org/software/systemd/man/systemd.exe...

* [3] https://www.freedesktop.org/software/systemd/man/systemd.uni...

* [4] https://www.freedesktop.org/software/systemd/man/systemd.ser...

systemd, in my opinion (one that doesn't seem to be widely shared), really does make administering a Linux server easier. No longer do I have to wrangle with shell scripts for the process control of daemons, or worry about the dependencies between them, instead I can organise that logic in a nice and neat .service file for systemd to worry about. The templating feature as this article demonstrates is also incredibly handy for DRYing up some .service files you may write.

Another feature of systemd I really like are the timers [1][2] it introduces. It offers a much more sane approach to handling scheduled jobs, and allows for an easy overview of which jobs are running and when the next job will fire via systemctl --list-timers.

[1] - https://www.freedesktop.org/software/systemd/man/systemd.tim...

[2] - https://wiki.archlinux.org/title/systemd/Timers

I think it's more widely shared than one might believe, because of the people who don't like systemd there's a contingent that really, REALLY hate it.

I despise the amount of random stuff that's tied to it and journalctl data format is still an abomination (and my bug with it trying to search thru entirety of log dir to show single app status is still open)

...but we did remove few thousands lines of fixed init scripts (anyone that tells you sysv scripts are easy and simple is lying to you, multiple projects fail there) *and* an bunch of Monit instances thanks to the features the base service management has. And simplified a bunch of other cases.

So yeah, overall even with issues it is a huge benefit.

The process control in systemd I think is unambigously good, but it would be nice if it wasn't as tightly coupled with journalctl and all of the other random stuff it's gobbling up.

Well, to be fair you can't exactly ask typical syslog demon "give me log files of an app" but the way journalctl implemented it is just fucking terrible.

It takes 4 whole seconds [1] for systemctl/journald to tell me it has no logs for the daemon on my NAS and it opens 985 files while it is doing it

If it just used sqlite as a backend it might've been useful for analysis (on top of way faster...) but Lennart wanted to have a go at implementing binary DB format badly so we're stuck with it.

If it at least kept a pointer to last file the app's logs were written to the lack of proper indexing also wouldn't be a problem.

But nope, it's just entirely worse than text format. And I do mean that in entirety, acking thru text files is faster than systemctl trying to find log files for the app...

* [1] https://github.com/systemd/systemd/issues/2460

My favorite experience with journalctl from many years ago was when an application kept spamming the logs till there was no space left. Once that point was reached, the binary logs got corrupted so couldn't be opened to investigate the cause. Don't remember how many turns it took me.

systemd is a significant improvement over script-based inits (and upstart, which I always found buggy). Nobody sings its praises because they don't have to: it "won" a long time ago.

> Requires=network.target tells systemd not to start this mastodon.target if the network.target hasn’t been reached successfully.

That’s not true, Wants=/Requires= do not specify any order between units, only that without them the unit cannot be marked as succesful. This way they can still be started in parallel.

If you really want it to happen after network is up, you should add a dependency of After= under the Requires=.

Also network-online.target might be a better choice but YMMV.

Blog says "reached" and not "finished" starting, the example here also includes After= for network target...

there is also a gist with some notes from Gargron himself: https://gist.github.com/Gargron/aa9341a49dc91d5a721019d9e0c9...

It's odd to me that systemd didn't implement a "run X number of processes" feature. I've also found it difficult to find a well-written tiny application written in C/Rust/Go whose only job it is to start and supervise X number of processes. supervisord is the well-known workhorse here but it's really old and quite buggy.

Have you seen overmind? https://github.com/DarthSim/overmind

Unfortunately Procfile doesn't support declaratively setting the number of processes, Heroku designed it so you have to specify that with their CLI.

From my experience there are very little use cases where you want to start same app exactly multiple times, usually you always need to vary at least one parameter (usually config file or workdir), and that's pretty app-specific.

...that being said

make template unit `myapp@` then just

    systemctl start myapp@1
    systemctl start myapp@2
    systemctl start myapp@3

can also tie them to start/stop together via dependencies

or make wrapper script that starts all and use man systemd.kill modes to send signal to all of them.

Queue workers is a classic example. It's not necessarily about wanting an exact multiple of processes, but more about wanting to scale them up and down with a declarative configuration format.

In the past I did actually use the templated solution, e.g.

    systemctl enable myapp-worker@{1..4}
    systemctl start/restart/reload/stop myapp-worker@*

However, in order to scale the number of processes down, you have to know how many are running already. Not that difficult to write a wrapper script around of course, but why isn't it just built-in?

Okay, so we add an option to specify number of processed spawned within service

You increase it, it spawns more, you decrease it, what next ?

How systemd knows which process to kill ? You want to kill the idle one, not the one currently doing the work.

The working process MIGHT BE IDLE coz it is waiting on network. Systemd have no way of knowing. There is no way to find out which process should be killed without processes itself cooperating.

And frankly because it's rare use case. Majority of apps that use have workers follow pattern of having a process that accepts requests and spawns or sends them to the workers and just accept number of workers as a parameter. Precisely because of the above, you do want to have logic on stopping the worker. And it's cheaper on memory to spawn threads instead of fully fledged processes anyway.

Systemd doesn't know when you need more processes so you have to script it anyway even if it had a feature to modify number of spawned daemons within single service. If you have that already, might as well write the the worker spawner there.

Personally the feature I miss is ability to have watchdog. Auto-restart is nice but if there was an option to "run this command, if it fails then restart main ExecStart app" it would've been perfect, easy way to add healthcheck that checks whether app actually works vs being unresponsive.

There is a watchdog feature where (if I remember correctly) systemd listens on a socket and the service must write to that socket periodically or it gets automatically killed and restarted. Unfortunately, the feature is rather obscure and not well documented.

I found that the sd-notify[1] package on NPM works well. If your app happens to use NodeJS then it's pretty simple to set up. If not, looking at the way that package is implemented is probably a good way to figure out how the feature works so that you can do the same thing in your own application. Maybe the documentation for the feature has improved, however, the last time I checked (only a few months ago) everything I found was bad. I did stop looking once I found that the NPM package works well.

[1] https://www.npmjs.com/package/sd-notify

Bit OT: Are dynamically scalable process workers on a single machine the right choice?

It seems a bit niche to be in a situation where you need the flexibility of dynamic scheduling whilst only having a single machine. Once you have more than 1 machine, systemd isn’t designed for it anyway, so you’d be looking at a cluster scheduler like Nomad or Kubernetes anyway.

In the single machine case, would it not be better to either:

a) Just spin up a static number n of workers (where n is the number of machine cores) and let them idle if not in use.

b) Use a thread pool (or green threads) inside a single process with shared memory (if e.g. idle memory use by workers is a concern).

I can’t immediately see the benefit of being able to have dynamic control over a number of workers AND keep them as separate processes that need to be adjusted manually.

I guess there may not be a choice if the software is not designed to be run inside threads AND memory use is a concern.

    We have the following files in /lib/systemd/system/ to define our sidekiq
    unit templates, which are all just copies of the same content as above.
    
    [email protected]
    [email protected]
    [email protected]
    [email protected]
    [email protected]
    [email protected]

Wat

Cloudflare weirdness I suppose. I see them all, but perhaps they think you're likely a bot or something and don't allow you to see it? (That or you have JS off?)

Ah yeah, if I enable JS for the page then I get the right content.

sigh

Reply on news.ycombinator.com