The Surreal Horror of PAM (2021) | Hacker News Reader

The Surreal Horror of PAM (2021)

xeiaso.net

81 pointspopcalc49 comments

> /etc/passwd is world-readable by design for some arcane reason

The reason is actually very straightforward: if you can’t read /etc/password, you don’t know who anyone is! Reading this file is (or was, prior to the introduction of nsswitch et al) the way you mapped a UID to a login name, so if you couldn’t read it then anything that needed to show a user (ps, who, ls -l, etc) would just show numbers instead.

Putting the actual password hash in it was, in hindsight, a mistake[1]; this was corrected pretty quickly with /etc/shadow but the file had to stick around with the same name because of all the things that were reading it already. (This is the same reason the “GECOS” field stuck around long after Bell Labs stopped using a GE computer for their print spooler.)

[1] I’m sure it made sense at the time—the fact that the password wasn’t just stored in plaintext was a novelty in its own right and nobody was going to spend that much CPU time trying to crack hashes when there were better things to do with the computer.

Also, one of the "cons" has been noted as it "works" on a single machine. You can "fix" that by enabling LDAP on the hosts and authenticating everything via that thing called an LDAP server.

We're doing this for a very long time on 500+ machines, and it just works. If you want to sync configs, you can do this with another "femtoservice" running somewhere.

I understand the concerns, but just badmouthing things because they're evolving for decades instead of built from top to bottom every three years is bad to put it mildly.

The problem of cross-machine passwd was solved before LDAP by something called NIS, also from Sun Microsystems. NIS was called Yellow Pages originally, so it has program names like ypbind.

I remember the name ypbind, but didn't know what it was. Thanks for the info, greatly appreciated.

> this was corrected pretty quickly with /etc/shadow

Not sure about "pretty quickly". If we peg the start of Unix at 1975, we then have Julie Hugh writing the Shadow Password Suite in 1987, thirteen years into the game, and the adoption of shadow didn't take place overnight.

Building on the point that PAM is simple, I wrote a blog post helping you write your very first PAM module!

https://rkeene.org/projects/info/wiki/222

Everytime I delve into PAM I can't help but think it's carrying a ton of legacy tech debt. It seems like it tries to support a lot of things out of the box that may not be relevant these days, like mail and limits. The modules themselves are littered with arcane options, and the entire thing is just this commented mess of inheritance and options that most users will just use to compare a hash in a file. On a basic debian install, there's around 455 lines of configs in /etc/pam.d!

Maybe systemd will streamline it someday.

> Everytime I delve into PAM I can't help but think it's carrying a ton of legacy tech debt. It seems like it tries to support a lot of things out of the box that may not be relevant these days, like mail and limits.

Beyond that, PAM has been saddled with a whole slew of responsibilities which go beyond user authentication, like supporting anonymous FTP (pam_ftp), maintaining /var/log/lastlog (pam_lastlog), or telling shell users that they have new mail when they log in (pam_mail). A lot of it is pretty specific to supporting traditional shell login workflows on shared machines, and is rarely used in most installs.

I do like the module that automatically creates a home directory on first login. It works well when you use ldap or similar mechanism for authentication.

And if you don't like that, you can disable it. It's both modular and flexible.

> rarely used in most installs.

I was going to object but you are absolutely right. Most Linux installs are instances of something that are created and destroyed without a human ever touching them. However, I wouldn't go so far as to characterize these features as technical debt. A fully functional Linux system should be able to tell users if they have mail, for instance.

> Most Linux installs are instances of something that are created and destroyed without a human ever touching them.

I'm not just talking about containers or managed servers, though. Most Linux desktop systems rarely or never have a user perform a PAM login in a situation where the mail message would be seen. (Opening a terminal emulator doesn't go through PAM; it starts the shell directly, and logging into X / Wayland might use PAM, but doesn't display login messages.) And, of course, it isn't relevant at all on embedded systems; a lot of those probably aren't even running a getty.

> A fully functional Linux system should be able to tell users if they have mail, for instance.

Hot take: storing and processing email shouldn't be a required function of the OS. It should be an optional feature, just like serving web pages over HTTP or having a graphical desktop.

That’s a good take though. I still hang on to the fantasy of having my mail delivered to my machine like it was big iron, but in the 25 years I have been using Linux it hasn’t happened.

Maybe systemd will streamline it someday.

Oh dear god I hope not. Systemd has added immense complication, has sinplified nothing, and is massive bloat.

When you understand how a PAM config file works, it's all pretty simple and readable. The biggest problem is missing documents detailing how it works.

And yes, systemd shouldn't touch to that.

I disagree that "it's all pretty simple and readable". The PAM configuration syntax is horrific, IMNSHO. It's a ruleset specification without explicit rule syntax, it has zero syntax checking and there are no validation/testing tools. The configuration files are line-based rather than having a separate scope for each rule; even ini-style headers would have been an improvement. The config syntax allows conditional rule evaluation by specifying how many of the following rules to skip (hello GOTO!). It even allows rule inheritance by using @include, but the including rule cannot modify/evaluate the results: if the rules you're inheriting have hard fails or hard accepts (such as root_ok), the rest of your own rules might not even get evaluated.

What does PAM actually do? It allows you to specify different authentication flows depending on the agent (authorizing application) and the intended action/flow. The resulting ruleset is effectively a determination tree, but there is no way to either input your ruleset as a tree structure, nor is there a way to get PAM to dump its abstract syntax tree for debugging.

Tangentially related: when I run Gitea on a LAN server under a Tailscale hostname, and i have Tailscale SSH enabled, do I have to use Gitea's built-in SSH server with a custom port? When I try using port 22, even with PAM on (as requested by Gitea), sshing to gitea@hostname asks for my tailscale login then gives me a shell session (causing git push to fail). https://tailscale.com/kb/1193/tailscale-ssh/#advertise-ssh-o... says tailscale ssh "configures tailscaled to intercept all traffic from your tailnet that is routed to port 22 on the Tailscale IP address".

I suppose I could set an ACL to only make Tailscale SSH intercept logins to nyanpasu64, and not other users like root or gitea (I haven't tested if Tailscale will reject other users or passthrough to gitea). I'm not sure if I should only approve my account, all nonroot, or "all but root and gitea" (unsure if possible with current syntax).

If I recall Gitea can run its owns libssh instance on a different port, and requires manually importing keys for each user. If you are using the https interface by default, than one may need to explicitly use the ssh:// uri.

Gitea should not be exposed to the web, and a port knock sequence is highly recommended to keep private projects private. ;)

I'm running Gitea in my LAN only, and currently hiding behind Tailscale (only listening to the Tailscale IP for HTTP, forgot to set that up for SSH over a custom port but that's not really a vulnerability) rather than exposed to other LAN devices. I may open it up to my full LAN in the future, I haven't decided.

Another issue with Gitea we found was memory requirements are somewhat unclear. So if you have to version large binary images in a source tree, the daemon can develop a number of hard to diagnose failure modes. It may sound ridiculous, but make sure you have at least 4GiB of free ram. Most of the time it will never be used, but it is far better than chasing ghost issue for that edge case where it does. ;)

For those who just don't get PAM, I can recommend PAM Mastery by Michael W Lucas https://mwl.io/nonfiction/tools#pam. It is readable and has a very specific sysadmin kind of humor I like. It certainly helped me to understand at least something. (His other books are also quite helpful, SSH Mastery, ZFS Mastery and others to name a few.)

Thank you, bookmarked. It's been a while since I last tried to understand what PAM actually is - or does, before giving up, knowing less than I did before, with the answer seemingly being "everything".

Discussed at the time:

The Surreal Horror of Pam - https://news.ycombinator.com/item?id=29167560 - Nov 2021 (137 comments)

That is the furriest furry blog I have seen in some time.

> how do you debug PAM?

> It ain’t easy.

VMWare workstation has a built-in gdb stub that can debug guest kernel, or any Linux guest userspace process quite easily. There is also support for windows guests, and even a visual studio plugin for that. I am surprised this did not come up as a very simple option to do this.

I go into more detail about this in my followup talk: https://xeiaso.net/talks/rustconf-2022-sheer-terror-pam#:~:t....

It is not easy in the slightest. PAM has aged like bottled milk in a desert.

Can confirm from being the maintainer of a PAM module in a past job. The only thing that was stable was debug builds that enabled awful fprintfs to a file in /tmp. It felt like how a freshman in college debugs their code.

So what's a proper way to solve the problem that PAM solves? Instead of an in-process library, have it be a daemon that responds to DBUS requests?

Aside from the config language, this sounds mostly like "it's a pre-login modular C library, and that's alien territory".

It would probably need kernel support for this to happen, but I'd like to see a daemon in the system that runs authentication logic and then arbitrarily curses the requesting process with the UID/GID/whatever it needs to get the job done.

How would that work with, for example a single deamon supporting multiple (different) remote users - for example ftpd, or sshd?

(I imagine you could say that sshd forks for each user, but some of the legacy systems like ftpd/ircd wouldn't have that as an option..)

I think Android has a more saner system, but I’m a layperson when it comes to these — android runs every user process under a different user that has very restricted access (restricted with SELinux as well), and they do extensive IPC. There is a daemon that handles elevated requests through IPC, so that’s where dynamic access control happens.

forget debugging, just configuring PAM is a freaking nightmare. oh and make a mistake? can't log in to fix it lmao. i got really good about keeping a root shell open after i made that mistake a few times.

It's about to get more surreal & horrible as several vendors are pushing privilege access management systems, and despite being in the auth[z|n] industry (they should know better) and having ~17k non-conflicting TLAs to choose from, are gleefully re-purposing PAM.

I recently wrote my first PAM module, using pam-rs[1] so i could do it in rust.

It was a learning experience, a rather enjoyable one though. While going down the rabbit hole, I feel even deeper when my module kept halting in openssh, due to a bug[2] from 2018.

I didn't really feel the pain of debugging the author had, most likely just because my application was simple enough to be tested via log statements though.

[1] https://github.com/anowell/pam-rs [2] https://bugzilla.mindrot.org/show_bug.cgi?id=2876

> password, the things that check passwords or do other kinds of validation like that (if you want to use Google Authenticator TOTP codes, you’d do that here)

password is for workflows that change passwords. It is not used for TOTP, that is an `auth` module. From the Google Authenticator PAM module docs:

> Add this line to your PAM configuration file: `auth required pam_google_authenticator.so no_increment_hotp`

Identifying a (prospective) user by their network origin? This must be that "zero trust" I've been hearing about, yes?

Meh, it's a VPN and they're apparently enforcing good authentication at that level. I still favour treating the network as public and doing real TLS (because your services want to do that anyway), but single sign on is definitely what you want, and if it were Kerberos or something in place of Tailscale then the PAM side of things would be more or less the same (other than not having to reinvent the wheel, because PAM for Kerberos has already been implemented).

The overall idea isn't that bad because tailscale basically already authes you on the network level, and in the kinds of infras this is for, this means you are already behind like 5 VPNs. I agree that automatically authing without user input is a bit much (eg worms) but tailscale has support for recieving and handling a "are you sure you want to conect to x" confirmation on the authentication source, it's just a matter of actually using it.

I think PAM was very cute in the office. Maybe I just wasn't aware of all the details.

It's a ripe opportunity for a rewrite in Rust

This doesn't sound so bad, I'm not really clear on what their major complaint about PAM is. Sounds about the same amount of hassle as developing anything deeply embedded in the operating system. I've seen worse.

Author of the post here. You are making a factual error by calling me "his". Can you please edit your comment to use the word "her" or "their"? Thanks!

Unfortunately the site isn't letting me edit the comment. Sorry about that, I heard a male voice on the video and assumed he was the author.

I've edited it to say 'their' now. Sorry for getting to this late.

Hacker News only allows editing for a few minutes then you are stuck with what you wrote typos mistakes and all. Because of this it makes little sense to ask anyone to edit anything as by the time you've noticed and they've noticed your reply that ship has sailed.

You can always ask us to edit it later. Emailing hn@ycombinator.com is the best way to ask us things.

That's nice but do you think the benefits of not allowing editing make sense on the overall and makes sense in the current configuration.

Of the top of my head it promotes a continuous coherent narrative not full of annoying deletions and comments that don't differ wildly from original posts. No posting something incredibly inflammatory, waiting for the conflagration, and then replacing original post with something milder.

On the other hand I have definitely posted something from my phone and cursed when I realized I couldn't fix a typo or decided something could have been communicated better.

It feels like the edit window could at least be a little longer.

The editing window is at least an hour not a few minutes and people who misuse editing tend to get set straight pretty quickly - it's one of the more 'users hate this and do something about it' HN sins.

> Sounds about the same amount of hassle as developing anything deeply embedded in the operating system.

Sure. Needing an external debug rig instead of just hitting f5 in your ide does tend to scare some people. No safety blanket.

Reply on news.ycombinator.com