Hacker News Reader

Comment by traverseda | Hacker News Reader

https://htmx.org/essays/locality-of-behaviour/

Locality of behavior is important. Note how this systemd-thingy is split into two separate files? It's like that a lot in systemd, whoever is doing the architecture doesn't seem to understand why locality of behavior would be desirable, instead taking the IDE approach of writing a bunch of tools to make managing the increased complexity more manageable. Don't glance at a crontab to see when things will next execute, run some other tool that will inspect everything for you. Make things more complex and difficult to read, and then write tools to make it easier to read again.

They could have just added "periodic" service type that accepts timer options, and at least then it wouldn't have to be two files.

Most of the advantages really don't seem like advantages to me. Templated unit files seems especially silly since it's equivalent to just copy and pasting a line in your crontab and changing an argument.

Seems to work for a lot of people, it's just not to my tastes I guess. Still, I find the whole thing pretty mystifying.

View on HN

>>Locality of behavior is important.

Even your own link talks about the Subjectivity of that opinion, and how LoB is often in conflict with DRY and SoC, both of which I support far more than LoB.

In this context systemd favors Separation of Concerns, the the service is what is being run, and the timer is when is it being run...

it is not that the devs do not "seem to understand why locality of behavior would be desirable" it is they disagree that is more desirable than having a logical separation of concerns...

To me I prefer greater separation of concern than locality

I think that's probably a trade off that large enterprises often make. It tends to work well for very large teams, and especially if you're trying to make your tech team fungible. Still, there's such a thing as taking it too far.

I think there's a middle ground. Sure, a single crontab makes it hard for packages to schedule things and makes it hard to assign ownership to individual packages/people/whatever. I can recognize that, and how it would be a problem for a large enterprise or even just for software developers who want to bundle some kind of cronjob with their package.

What you could do instead of a crontab is have one folder with a bunch of services/timers in it, where each file represents a single service that needs to be run. It would mean that you can still understand all of what's going on timer wise by looking carefully at one source, but you can still assign different services/timers to different people/packages/organizational-units/whatever.

Splitting each individual cronjobs into 2 different files is just crazy from where I am. Like you already have a bunch of different service types, at least give us the option to use a "periodic-oneshot" service type instead of this craziness. There's no need to have different packages/people responsible for a service and its timer, and if there is some edge case where it is you can still have ssperate timer services just they can live in a separate periodic-oneshot service file and use the original service as a dependency.

There's separation of concerns and then there's this. Take any principle too far and you get some craziness, and systemd has most of the infrastructure needed to solve this in a much more elegant way. Like at some point it has to just be bad design and not just different priorities, right?

>>Sure, a single crontab makes it hard for packages to schedule things and makes it hard to assign ownership to individual packages/people/whatever.

and makes is hard for admins to schedule things, and make is hard to figure out what is scheduled when, and makes it hard to test if the process will work under the schedule.

I absolutely hate cron's format, and pretty much everything else about it. systemd timers is a WONDERFUL and absolutely needed change to the world of linux administration.

cron was written in a time where verbosity had actual costs to the system, we are not in those times and the need for nostalgia to be cryptic and only use minimum chars is over and I for one am glad for it.

Both in programming and administration people need to be MORE VERBOSE not less

Well I'm not complaining about systemd's verbosity here, I generally agree on that point. Nor am I saying cron is particularly good, I think there's definitely room for improvement.

What I am saying is that splitting a basic timer between two different files is really bad for readability and understandability.

You are not splitting a timer between 2 files

you are splitting a service and a timer, not all services need a timer. I think that is point being glossed over

services may be called by a target, a timer, or manually. or all 3 when the need arises.

So for example if I have a process that I need to run on startup, every Friday, and sometimes adhoc I can create a single service file and do all 3, and know it is executed in the same way every time

could not do that with cron or even older init systems

Right but you basically never run the service ran by timer manually, so on average it's just adding noise to equation.

Having option for separate timer is all and well (after all you might want to use it to periodically start service, for example start some sync service after midnight) but for cron-like behaviour it should really have all in one option, perhaps by just adding [Timer] block in .service or vice versa

You might never run the service outside of a timer, but you may have multiple timers reference the same service.

I'm constantly running the service ran by timer manually, mainly for debugging.

You do when testing it!

With cron this is troublesome, because cron uses an environment different from what you get from the commandline. So things can work from the shell and not from cron.

So to really check that cron works you have to add a line to be called in 5 minutes, then check your mail to see what happened.

With systemd though, you can launch the service at any time and see what it does without all that fiddling.

I can't count the times I've had to debug cron scripts by copying the cron command line and running it through sudo because the script ran into an edge case, only to find out I didn't have the exact cron environment so manual runs didn't trigger the bug.

I much prefer the systemd solution. As an added bonus, systemd services allow for easy configuration of things like sandboxes and resource limits that would need to be hacked into a giant concatenated command line in cron, presumably without comments even.

And I said "add option to have it in one file" (instead of "only have it in single file") for exactly that reason

Why isn't the timer stuff just an optional [Timer] in a unit file? Then it would be separate (separate section), even if it's in the same file.

that would be a convention or configuration debate

I prefer convention for systemd. The justification for have a separate timer file instead it being in the unit file would be making it easier for someone to quickly look at what services are running on timers vs running by targets.

if timers where in the services file you would have to open each one to find out what had a timer, with a separate file you just look at which service as a timer provided your naming is standardized)

In practice, I run `systemctl list-timers` to see all the timers that exist. That could be implemented with timers in a section of the unit file as well. But by being a separate file, you can have package that installs the unit and another that installs the timer. You can use symlinks to enable the timer and delete the symlink to remove the timer.

In general convention doesn't work so well if you're not really into the system. I.e the casual users.

Convention means that behaviour depends on not just what's present but also what's not present. So it is of course missing some of the explicitness. For the directories, you need to understand which directories to look into and in which order they are taken into account (for systemd this is hard enough that's it's best to be tool assisted).

Because a service may not always be triggered by a timer. Sometimes it could be triggered manually. Sometimes by a volume being mounted or unmounted. Sometimes by a request from a socket. By decoupling them they become reusable components. Actually, much like Unix tools which people love going on about.

Binary log files is another example of something that could have advantages in theory but never seems to work well in practice. What used to be "tail /var/log/nginx.log" is now

man journalctl journalctl -u nginx <G to skip to latest, ctrl+C because it's taking too long> journalctl | grep nginx <wait...> go back to man page to look for more options to search

etc

One of my earliest experiences with systemd's binary log files was on an embedded system I had ~~chrooted~~ systemd-nspawned into, and every time I tried to get the log output it would segfault (some issue with qemu-static probably). Very frustrating experience.

Really feels like "Embrace, extend, extinguish" but the thing they're extinguishing is unix-as-a-programming-environemnt.

Or a Redhat plan to generate more consulting/support income. Ever wonder why Oracle products were so hard to install/maintain?

It's the IBM way.

Everything be in 4-digit or 5-digit error code.

Need to RTFM for such codes. Or worse, StackOverflow them.

Yes, a for-profit corporation has the nefarious plan to... checks notes ...try to earn more revenues. We certainly can't stand for that!

Heh, well making your product worse for more revenue seems pretty warped, and increases the risk of lost market share. Even more warped is having 2 of your engineers join the Debian technical committee and use the bylaws to force a vote before anyone figures out exactly what SystemD is.

Were they coerced to vote for it? Why did they vote for it if they didn't know what it is? Honestly it's hilarious how systemd haters see conspiracies everywhere.

"tail /var/log/nginx.log" is now "journalctl -fu nginx" or "jouranlctl -eu nginx". Not that much of a difference, it really depends what you grow up with. We could argue "tail" is ugly because you never know if it's "tail /var/log/nginx.log" or "tail /var/log/nginx/nginx.log".

A nice feature is that journald gives you flags like "--boot" to see logs only emitted during a specific boot, or "--since '5m ago'" which is not that straightforward to do with the approach you favor. (if there is a way to do it easily, please let me know!)

I personally find journalctl -fu $service very cathartic, because I'm almost always typing this command when $service isn't working.

I really need to alias `journalctl -fu` to something because it's always so tedious to type all that out over and over when I'm trying to debug something. Also, is there any autocomplete for systemd unit names? Typing `journalctl -fu amazon-cloudwatch-agent` over and over and over is so much more tedious--ideally it would be something closer to `jctl amaz<tab>`.

I do get autocomplete for even that with bash (on nixos). Tried to have a look at my config, but I forgot what enables that, but it is a game changer.

There is autocompletion. It works by default on Pop OS and Debian. (You may need to install bash-completion and/or dbus if your install is very minimal.)

Knowing the path to the log file isn't more complex than knowing the name of the systemd unit, but not needing to know the right set of flags to pass because the sane behavior is the default is _really_ nice. That said, I didn't know about `--boot` or `--since`, so maybe I'll hate journalctl a little less (also, I've only ever used systemd-based systems, so I don't think "it's what you grew up on" applies, at least in my case).

> We could argue "tail" is ugly because you never know if it's "tail /var/log/nginx.log" or "tail /var/log/nginx/nginx.log".

The difference is that you can use ls to find out, the same ls that you use when working with everything else, rather than needing to know some journald-specific thing that will only ever work for journald and will no doubt change again in another 5 years.

And you can just issue “systemctl” to list every service. Do you blame excavation machines because you can’t drive them when you only know how to drive a car?

If someone switched my car for an excavation machine when the car was working fine I'd definitely complain.

I'd recommend running strace on that journalctl command, you might be surprised what happens. It's even worse if you actually store journalctl logs on disk (most distros defer to storing it in ram). I have it set to store on disk and keep at most 1GB of logs

On my NAS

    # strace journalctl -u nginx 2>&1 |grep open |wc -l
    827

and curiously

    # strace journalctl -u baked-potato 2>&1 |grep open |wc -l
    827

(no i do not have baked-potato service on that machine)

It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful. If the binary database was a SQLite file with columns containing app name/log level/etc. it would actually be useful but current one is just shit.

Bonus round:

    #:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
    #:/var/log/journal time journalctl -u nginx >/dev/null 2>&1

    real 0m3,232s
    user 0m0,036s
    sys 0m0,292s
    #:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
    #:/var/log/journal time ag -R nginx . >/dev/null 2>&1

    real 0m2,531s
    user 0m0,004s
    sys 0m0,290s

slower than actually searching thru files directly...

Why would it care about stracing? It was never too slow for me. A "bad" performance might be caused by ex. consistency checking. Benchmarks like these don't tell much - you can't reduce software performance to one number.

> It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful.

That sounds interesting, can you please elaborate on the internal structure of journald files or link to further documentation? And why would I care if journald handles it for me?

I don't have a negative experience with journald file format. That said, the format is documented here:

https://systemd.io/JOURNAL_FILE_FORMAT/

On the other hand I agree that journald seems to call open syscall more often than I would originally expect, which can be a problem for some edge cases[1], but I don't consider this to be a real problem.

[1] https://blog.marbu.eu/posts/2022-11-20-ebpf-journald/

The problem is:

If cache is hot, you have hundreds of MBs of systemd logs littering your buffer cache, while it could be used for something more useful

If cache is cold simple operations like systemctl status take up to tens of seconds, especially on loaded servers. Example:

    # time systemctl status influxdb >/dev/null

    real 0m11.913s
    user 0m0.013s
    sys 0m0.531s

It doesn't even keep index of "last file where app wrote logs" which causes above.

But this buffer cache usage is caused by readers/clients checking logs for some reason, not logging itself, right? I see no significant difference compared to syslog here: if I grep all logs for something, it will also place these logs into buffer cache.

Your example with dropping cache and running systemctl status is indeed interesting, 11s looks like too much. But it's a number without a context, and I wonder how big a problem this actually is. I haven't noticed it myself before.

> It doesn't even keep index of "last file where app wrote logs" which causes above.

While I definitely see some room for optimization, I'm not sure what you mean here: journald uses one active journald database file, there should not be a problem with figuring out where the file is, should it?

> Why would it care about stracing? It was never too slow for me. A "bad" performance might be caused by ex. consistency checking. Benchmarks like these don't tell much - you can't reduce software performance to one number.

Well, I was routinely getting multi-second wait for operations like systemctl status servicename which is not something rare, because it didn't even kept the pointer to file that has last few lines of app's logs. It's not just benchmarks

It also trashes cache with hundreds of MBs of logs instead of stuff apps running on server actually needs. An equivalent of "tail -f" can go thru hundreds of megabytes of binary logs.

No, having a special way to tail your special log file isn't a feature, tail is a general utility used system wide for this for decades; breaking this is idiotic no matter what you grew up with.

I'm happy we have special-purpose tools for dealing with logfiles. I don't want to craft one-liners for "give me logs around timestamp" or "give me logs about my service's first start after the boot" every other day.

"I don't want to craft one-liners"

I do. I specifically want that as a more valuable and powerful feature. Undifferentiated, open-ended, general purpose, sharp tools are infinitely more useful.

> Not that much of a difference

It's a big difference, not because one command, but because tail is part of the standard unix toolset that works with all files. Except now you have this one log file that is special and needs a different unique tool.

Can still do that?

Your gripes all seem to boil down to your unfamiliarity, rather than anything specific that stands on its own as a problem. Unfamiliarity is understandable, but if that were enough to avoid progress, we wouldn't never have progress.

> ctrl+C because it's taking too long

Try --since=today. Then you end up in less, and that should be more familiar, including for search. This is then the equivalent of traditional daily rotating log files, with no other knowledge needed apart from --since and --unit (the latter is noted in "systemctl status" output as a reminder).

Then every further feature is an optional bonus.

As other commenters have mentioned, the unfamiliarity is a symptom of the time I spend using less,grep,sed,sort.. etc on a daily basis. If I could use those journalctl flags for everything I do relating to files on the system then I would be very familiar with them.

It's the same rule I use when purchasing kitchen equipment - often it's better to have a single multi-purpose tool than a bunch of specific implements. Otherwise you end up with a drawer full of strawberry-stem removers, pizza slicers, corn-cob peelers, avocado-slicers, etc.

This is to say nothing of the binary format issues that others have discussed.

Maybe nginx logs at usable, but my experience with reading logs in the old way was often "grep servicename syslog" and prayer that the tool didn't have its own magical log file somewhere else.

The backing store may be controversial, but the journalctl command line is much more pleasant. "I want to see log messages from the end of the last boot when the system crashed" is no longer a Rube Goldberg command line involving less/sed/grep/cut/awk and manual scrolling, it's supported out of the box with no configuration necessary.

Then again original cron is also kinda shit.

It gets the "one file to define the job" right but that's the only thing that it gets right.

There is no holistic view on the jobs so you can't easily say "show me last jobs that failed" and shove that into alert.

There is email on failure but only on failure so you can't also do simple "if this mail returns fail, mark check as failed, if it returns okay, mark it as okay". You can get that info from logs but you can't get the content of error message of the app from those logs...

Environmental options history is.... interesting, better to just make script that sets everything up beforehand instead of even trying to do it via cron.

From ops perspective of managing multiple servers its utter garbage. From single user perspective it kinda works but is annoying

Systemd's timers should just be [Timer] block inside the .service

> Systemd's timers should just be [Timer] block inside the .service

systemd was inspired by launchd (http://0pointer.de/blog/projects/systemd.html), and that does that (https://developer.apple.com/library/archive/documentation/Ma...), so they must have explicitly made that choice.

The wrong choice in my opinion

The thing about cron though is this: I learned how to run crontab easily once and I've retained the knowledge of how to use it ever since. It's a piece of cake, I don't think I've ever had a cron job fail on me except for the path issue but I've always caught that on creation since I test them right after I make it.

With systemd timers, I have to look up how to write a unitfile every. single. time. and it's not a trivial 30-second lookup like every other command, it's a 5-10 minute read through every time. And even then there's weird variations on the timer unit files I don't quite understand, and most of the time it takes more than one try to get right. It fails in some bizarre ways on certain tasks due to internal systemd inconsistencies. (Like using different escape characters in different contexts, wtf?!) And I can't remember how it's set-up so I have to look it up again when I want to administer it or change something and on the whole it's just a huge PITA.

At this point, I've used systemd timers several times MORE than I have cron and I still have no idea how to write a systemd timer unit file without looking it up, and I'd have to start from scratch to write a new one.

About the only positive is that when it fails systemd reports it decently so I can follow up on it, but man, it's not worth it. I've spent a lot more time dicking with the unit files than I have debugging failed cron jobs.

Agreed. It reminds me how I recently tried to figure out why cert rotation failed, only to realize that the cron job is sending it's error messages via email to root on a system where there is no local mail server ...

I think timers handle this fine -- one part sets up the work (service), the other sets up the scheduling of the job (timer)

Speaking of behavior, let's look at cron.

Consider every time you've written one. There's a good chance that it failed for some time because you forgot PATH is make believe there

Or logging. How many times has rotation been rewritten (and broken)?

Without digging through logs that may or may not exist, can you tell me when it last ran successfully?

I can with timers, 'for free'

I'm definitely not saying someone couldn't write a better cron, or that it doesn't have warts. Still, put 1/10th of the budget towards cron that went towards this and I think you'd get something a lot more usable.

Some people just seem to love reinventing square wheels. I suspect this "spiral of complexity" is largely the result of people trying to justify their salary, because it lends itself to generating a lot of otherwise unnecessary work.

Starting a service is way different than the service itself, what if I want it to be started when a request hits a given port? Also, the service files themselves are usually packaged up and are immutable, and every user will want different behavior on when it is started.

To me it sounds like proper architecture and your solution would be needless close coupling.

Sure, then let me put a command in my files instead of requiring a separate service.

>what if I want it to be started when a request hits a given port?

I really don't think you should be turning your init system into a general programming environment that can do that.

> I really don't think you should be turning your init system into a general programming environment that can do that.

We have :) https://www.freedesktop.org/software/systemd/man/systemd.soc...

No need for multiple anything, the units can be parameterized -- user@my.service, for example

Oh I know it's possible, I just don't think you should be doing that in an init system.

Understood, fair enough. I'm not too bothered by it, it's been useful here and there

Socket activation is declarative. My last point is the dynamic one, but that is just calling out to systemd. The point is, all three can share the same service description.

This split between service file (what is going to happen exactly) and a timer file (when it's going to happen) is ok for complex use cases, but it really feels a bit silly when a single cron line would do the trick instead.

You can still use cron if you want...

there is cmdline shortcut for one-off ones at the very least

    systemd-run --user --unit=timer-name --on-active=60 /exec/path

will run it in 60 seconds

Nice, thanks for sharing this.

Also the split allows easier debugging: one can define and run the service, check that it works, and then define a timer and test that separately.

Well, except for /etc/crontab, /etc/corn.daily|weekly|whatever, /var/spool/crontabs, etc.

/etc/cron.{hourly,daily,weekly,monthly} are just some convenience shortcuts. No one is forcing you to use them.

When a bunch of important system functionality is done using them, I’m required to know they exist. /etc/crontab is no different.

It’s not that much different than systemd in that sense.

Especially since the /etc/cron.whatever special rules like files with dots in them get ignored (ugh)

Another fun one I've gotten bitten by a lot is that the /etc/cron.* directories are run sequentially, so if someone who doesn't know this put some long running process in it, ideally with a name that sorts first, it blocks everything else in that directory. (I personally know this, obviously, since I'm saying it now, but I've been bitten by the other people who do not realize this and do not think about how putting a process that may run for three hours at the beginning of cron.hourly means that the other processes may not run for that long.)

NixOS fortunately solves this problem. Here's a daily rsync with some names changed to protect any innocent hosts on my network. Note that the contents of "sync.sh" could easily go in there as well, but I was previously running it via cron, so didn't bother to move that over.

  systemd.services.somehost-sync = {
    startAt = "daily";
    path = [ pkgs.coreutils pkgs.rsync pkgs.openssh ];
    environment = {
      HOME = "/home/someuser";
    };
    script = ''/external/media/sync.sh'';
    serviceConfig.User = "someuser";
  };

Not everything in NixOS has been a "win" but systemd timers are so much more ergonomic under nix

Counter argument: Try implementing support for Alpine, Debian and other distros whilst using a cronjob.

It's impossible. All that opinionated /etc/crontab and /etc/cron.daily shit can go to hell for all I care. Every opinionated distro uses whatever kind of file format that the next best sh guru came up with, which id also prone for exploitation btw.

With systemd there's a failsafe way for maintainers to offer a cross-distro way to integrate their software.

And that's what systemd is about: automation of integration through convention.

I'm not sure what you mean? Alpine doesn't have systemd at all.

Failsafe?

I'm still experiencing file corruption regularly with journald on various distros.

Cannot recall corruption nor unreadability with syslog.

Then you have some corrupted hardware.

Also, syslog wasn’t capable of logging during the initial stages of boot, which is pretty important.

That's what dmesg is for.

Then there aught to be some kind of fsck (file checking) for journal files.

To forever complain until user intervention is downright silly.

Five separate but disparate hardware does not make for a probable hardware error.

> Don't glance at a crontab to see when things will next execute, run some other tool that will inspect everything for you. Make things more complex and difficult to read, and then write tools to make it easier to read again.

systemctl list-timers is vastly more informative in a practical sense than just looking at the crontab. It just straight up tells you exactly when it ran last, how long ago that was, when it will run next, and how much time is left until that happens.

More importantly, you don't have to worry about fat-fingering an entry in your crontab and having everything from that point onward silently fail.

'Make everything simple as it can be but no simpler'. To me your ask violates this.

Timers augment the variety of existing service types, so your idea just doesnt work straight up, unless we severely limit the flexibility of what timers can be via the 'periodic' type. Or we make 'periodic-oneshot' 'periodic-serice' et cetera, which feels absurd.

Perhaps we could leave Service= alone & just shoehorn each systemd.timer option into system.service. But then we need some way to view the timer statuses & logs, which we can easily review and handle by having a separate unit. 'list-timers' is ok & might still work but we cant filter and log as well if everything is jammed together. We also cant just disable a timer temporarily, & go manage the service manually for a bit, if somethings going sideways & we need to step in; the two are now one in your world.

And what if you want multiple timers, with different configurations between them (maybe one long running one with WakeSystem= and a shorter one without?). We eliniate a lot of creative ability by glomming stuff together.

I dont like this idea. I think it's a misservice to jam everything together. Systemd has similar patterns of representing different pars in different places that I think has served it well. It's an even more trivial case that is more supportable by this idea you've floated, but there is a systemd.mount and systemd.automoumt. It's just good, as an operator, having clear boundaries between these units, and has always made it clearer to me what aspect Im operating on, and enabled healthier patterns of system growth.

>And what if you want multiple timers, with different configurations between them (maybe one long running one with WakeSystem= and a shorter one without?).

Well, that's where you can write a regular timer or just add a dependency to your timer-service.

But also you're sort of getting to the territory of inventing your own scripting language on top of systemd at that point, if your needs are that complex just write a script.

This feels quite dismissive to me. I dont see the appeal of any of these three options, given how well systemd.timers solve all of this today & how nicely they compose.

The whole point is to not need to keep running into a bunch of artisinal handcrafted did-it-myself shell scripts every time come to a system. Being able to compose timers that trigger units enables this nicely with a lot of flexibility. Managing them separately is powerful & makes sense.

> And what if you want multiple timers, with different configurations between them (maybe one long running one with WakeSystem= and a shorter one without?). We eliniate a lot of creative ability by glomming stuff together.

It's not either-or situation.

The .service could just have [Timer] block

Then you could just put all in one file, or still have option of putting timer somewhere else.

Then it'd be annoying if you have to stop a timer temporarily

It’s odd, but actually you can not restart an active unit with a timer

>We also cant just disable a timer temporarily

Sure you can, comment that line out of the service file. It is ultimately just a text file that you can edit.

Again, inventing arbitrary hacks totally on your own that none of the builtin tools will know about or help you do. This seems obviously worse than having a seperate services actuated by separate timers.

I guess preferences will mostly depend on how the system is being used. If this is your desktop machine sure, do crontab -e and add a line. If it is a system sitting in a colo or cloud provider, it's likely that it's been configured through automation anyways so "locality" doesn't matter. Your scripts know where stuff is located.

My scripts an also do things like assemble a crontab, since it doesn't matter.

Reply on news.ycombinator.com