Hacker News Reader

Comment by hoodoof | Hacker News Reader

Can you post your 10 lines for us to see?

Yes, please - genuinely interested.

Sure.

    [Unit]
    Description=Blah app

    [Service]
    ExecStart=/var/www/blah/bin/www
    Restart=always
    User=nobody
    Group=nobody
    Environment=PATH=/usr/bin:/usr/local/bin
    Environment=NODE_ENV=production
    WorkingDirectory=/var/www/blah

    [Install]
    WantedBy=multi-user.target

Starting as nobody is probably not a good idea if you have more than one service which does that. They'll be able to interact.

Could you please explain in more details the issues here and the right way to do it?

With two applications running under the same user, if one is compromised, it has ready access to the other.

Generally it's fairly easy to make a system user for an application if no special accesses are required - just make a custom system user for the application, and ensure it can access the various things it needs to touch (eg: logging).

... and if you want to learn more than has been already said, start with the famous design of qmail as a collection of mutually untrusting daemons that run under individual unprivileged user accounts, precisely to insulate each from the others using the normal operating system mechanisms that insulate one user's processes, files, directories, and so forth from another's.

A good question to ask yourself: "Presume that my daemon is compromised by an attacker ...

* ... what processes can it trace, debug, read/write the memory spaces and environments of, or send signals to?"

* ... what files, devices, and directories can it read and write?"

* ... what files, devices, and directories does it own, and can thus grant access to?"

Then ask: "How does the answer change if instead of all of my daemons running as the same (even if unprivileged) user they each run as different users, some (where appropriate) having no write access to anything and owning no files nor directories at all?"

Beautiful!

It's not even the number of lines that matters -- because start-stop-daemon could probably accomplish all of that in "1" gigantic line.

It's the fact that it's incredibly readable. Someone with only a passing knowledge of UNIX and no knowledge of systemd could understand almost everything in this file at a glance.

I can't say I agree. Does Restart=always mean the service is restarted if it is killed? Or something else? Why is there a block named [Install]? Why would my service file need to be installed? What is multi-user.wants? How do I know that this sorta-kinda-but-not-really corresponds to Runlevel 5?

However, the reverse view raises similar questions. How do you know what runlevel 5 is? (Hint: Someone with "a passing knowledge of Unix" knows that there's no coherent definition of run level 5. It's "power the system off" on some Unices, for example. Someone without a passing knowledge of Unix is probably at a complete loss as to what "level 5" could be.)

In actual fact, the multi-user boot versus single-user boot notion goes back to pretty much the beginning (https://news.ycombinator.com/item?id=10207674), and the name "multi-user" is not exactly difficult to cope with if one has a Unix background. It's not novel or a systemd-ism. It's been a Solaris SMF milestone name for over a decade. It was a concept used in the 4.1 Berkeley Distribution manual for init(8) in 1981.

> ...the name "multi-user" is not exactly difficult to cope with if one has a Unix background. It's not novel or a systemd-ism.

Sure, but I had to do a bit of reading to understand that

  [Install]
  WantedBy=multi-user.target

meant: "This service should start in the runlevel called 'multi-user'".

It also seems... backwards to put runlevel membership in a service definition file. Runlevels are an administratively controlled thing. Does systemd have two or more places a given service's runlevel membership can be specified, or is the only way to change a service's runlevel in its service definition file?

Yes, I know that the notion of "runlevels" is probably "legacy". However, assume that -on my OpenRC system- I have two particular runlevels, one for regular multi-user operation, and one for operation when I connect to hostile networks. Transitioning to this second runlevel shuts down all services that listen on anything other than localhost. Transitioning back brings those services back up.

Although this situation is a little contrived, this sort of flexibility is valuable.

The WantedBy value in the system definition file isn't an absolute (nor are the similar values Wants/Requres/RequiredBy/etc). Instead they are defaults.

What systemd actually does at startup, or when starting services individually, is look for symlinks. If I have a foo.service and a bar.service, and a symlink called foo.service.d/bar.service -> ../bar.service, then systemd will start bar.service automatically whenever I ask it to start foo.service. If it also sees a multi-user.wants.d/foo.service -> ../foo.service, then it will start foo (and also bar) whenever it enters the multi-user "runlevel". (This is almost exactly how sysv runlevels work, except that they're named rather than numbered, and they can depend on each other.)

I can make that symlink manually, if I want, or I can call "systemctl enable foo.service" in which case systemd will read the dependency information from the .service files and make the appropriate symlinks.

If I don't like how the distro or packager has set up the service file, I can override it. The service files live in /etc/systemd/system by default, but systemd also looks in /var/systemd/system for overrides. This is the real reason why systemd uses key=value stores for units; merging them is trivial while merging shell scripts is decidedly not.

Putting the overrides in a separate directory is a pretty nice decision; it's always obvious what changes have been made, and upgrades never overwrite them. I can even put those overrides into version control and track changes to them over time.

> The WantedBy value in the system definition file isn't an absolute (nor are the similar values Wants/Requr[i]es/RequiredBy/etc).

Is this true of the identically-named keys in the Unit section?

> If I don't like how the distro or packager has set up the service file, I can override it. ... This is the real reason why systemd uses key=value stores for units; merging them is trivial while merging shell scripts is decidedly not.

You should look at how OpenRC handles service dependency specification. It's at least as flexible as the system you describe in systemd. :)

> Putting the overrides in a separate directory is a pretty nice decision...

Yep. At dependency graph generation time, OpenRC examines its master config file at /etc/rc.conf as well as config files in /etc/conf.d/ for -among other things- additions to the need, use, before, &etc declarations in the depends section of a given service's init script. To augment -say- the "use" list in the openvpn service, either add the line

  rc_use="my_custom_service"

to /etc/conf.d/openvpn , or add the line

  openvpn_rc_use="my_custom_service"

to the OpenRC master config file at /etc/rc.conf . If I -say- wanted to override OpenVPN's default dependency on the "net" virtual service, I would add

  rc_need="!net"

to /etc/conf.d/openvpn (or perform a similar transformation as before and store in /etc/rc.conf).

> I can make that symlink manually, if I want, or I can call...

To add the openvpn service to the "custom_init" and "default" runlevels, I would simply do

  rc-update add openvpn custom_init default

If I omit the runlevel, the service is added to the current runlevel. I could screw around with symlinks if I cared to, but why would I? :)

The OpenRC configuration processing system is substantially more powerful than I describe, handles the processing of OpenRC-standard config files [0] automatically, and lets you shift the complexity of a configurable init script that should be in a config file into a config file.

Because my overrides and augmentations to the distro-supplied init scripts are in text files stored in standard places, I -too- can easily put them in version control and track their changes over time.

[0] Which -shockingly enough- look exactly like what any Linux programmer would expect a non-XML non-JSON text config file to look like. ;)

Assuming you want all configuration in the init script, [0] here's the OpenRC equivalent. The USER/ENV/WKDIR variables are there for legibility.

  #!/sbin/runscript
  
  USER="--user nobody --group nobody"
  ENV="--env PATH=/usr/bin:/usr/local/bin --env NODE_ENV=production"
  WKDIR="--chdir /var/www/blah"
  
  description="Blah app"
  command="/var/www/blah/bin/www"
  start_stop_daemon_args="$USER $ENV $WKDIR"

Assuming you had a "multi-user" runlevel on your system and the service was named "blah", you would do the following to add that service to it:

  rc-update add blah multi-user

But, the "default" runlevel is probably the equivalent to systemd's multi-user runlevel. So:

  rc-update add blah default

Describe works as expected:

  $ /etc/init.d/blah describe
   * Blah app

[0] Real init scripts make use of OpenRC's machinery for processing config files in /etc/conf.d

Looks very similar to supervisor.conf

I'm not the same author, but I run several Webapps in systemd units, with the below service for all of them:

Note the @ in the name, that means I reuse it as webapp@api.domain.name webapp@frontend.domain.name and so on.

----8<---- webapp@.service ---->8---- [Unit] Description=%i webapp

    [Service]
    User=%i
    Group=%i

    # Note that the first argument of the command line (i.e. the program to execute) 
    # may not include % specifiers.
    ExecStart=/srv/webapp %i
    WorkingDirectory=/srv/%i/

    PrivateTmp=true
    Type=simple
    Restart=always
    RestartSec=2s

    [Install]
    WantedBy=multi-user.target

---- 8< ----

/srv/webapp is a simple script that will do "exec $1/webapp" which is the actual executable.

> /srv/webapp is a simple script that will do "exec $1/webapp"

I'm not trying to start a fight.

Why do you call out to a script, rather than having systemd make the call on its own?

It's for the same reason that java has to be invoked as

   /usr/bin/env $JAVA_HOME/bin/java

thus raising questions such as http://unix.stackexchange.com/questions/229523/ .

systemd does not permit variable expansions or unit parameters in the first word of ExecStart.

Assuming that your explanation applies to Spidler's situation, why not do

  ExecStart=/bin/sh -c "%i/webapp"

rather than using a one-off wrapper? Does that wrapper ship with systemd?

No assumption is necessary.

Roger.

Why do the thing Spidler described in his comment, rather than the thing I described in my comment?

Reply on news.ycombinator.com