The Surreal Horror of Pam | Hacker News Reader

The Surreal Horror of Pam

christine.website

278 pointstodsacerdoti137 comments

So the fact that PAM was adopted by Linux is mostly my fault. I was the tech lead for Kerberos at MIT, and in 1995 I was visiting Sun to discuss how kerberos might be included in Solaris. One of the technical discussions that we had was that each time people wanted to add support for various new large scale distributed infrastructures, whether it was Yellow Pages, or Hesiod (YP's rough equivalent developed at MIT Project Athena), or OpenLDAP, or Kerberos, or when we wanted to automatically mount the user's home directory, either using NFS as in Sun, or creating a temporary home directory, or creating a symlink into AFS and then getting AFS tokens, etc. we had to keep on modifying the sources for /bin/login. And this was a positive drag. Basically each large scale site was editing the source code for /bin/login and there was a private customized copy of login at MIT Project Athena, CMU Project Andrew, various US National Labs, yadda, yadda, yadda.

This was how things were done before PAM, and it presumed that you could twist the arms of the proprietary Unix vendors hard enough that they would give you to the source to /bin/login, which of course back then was encumbered by the AT&T Unix license, which means you needed to pay AT&T to get a Source License before you went back to twisting the arms of the proprietary Unix vendor. And if you had a multi-vendor deployment, you might need to separately customize the /bin/login for OSF/1, Solaris, AIX, HP/UX, AUX, and Irix for your large scale deployment --- since all of the proprietary Unix vendors had added their own, incompatible, "value adds" to the OS. Yelch!

The Solaris developers told me about this cool library called Pluggable Authentication Modules which they had been working on, and it was clear to me that this was the answer we were looking for. No longer would each site need to hand edit C code to customize what was supposed to happen vis-a-vis using the user's password to get Kerberos tickets, or get AFS tokens, and what might be needed for session startup such as site-specific ways of attaching the user's home directory.

So I took the idea back from the Bay Area, and I started talking to folks in the Linux community and said, this is the answer to allow us to be able to distribute advanced systems such as Yellow Pages, Kerberos, OpenAFS, etc., and when the user installs the right packages we can automatically make /bin/login do the right thing. Huzzah! Michael K. Johnson at Red Hat and I managed to recruit Andrew Morgan to be the maintainer of Linux-PAM, and it started shipping in Linux distributions in 1996. (Red Hat Linux 3.0.4, shipped in August 1996 had PAM support --- note, this is RHL 3.0.4 which is not RHEL 3. Red Hat Linux predated Red Hat Enterprise Linux.)

Although we did a clean-room reimplementation of the PAM spec, using only the man pages which the Solaris developers had provided to me, it started shipping in Linux distributions before Sun managed to ship their implementation of PAM in Solaris in 1997, even though they developed the spec and had a prototype before we had even started coding.

So the history in Christine's talk isn't quite right. PAM was not developed because of the existence of SSH. It's true that SSH was first written in 1995 as well, and the fact that it made it easier to integrate SSH was a bonus, but the primary concern that I (as a Linux developer and Kerberos Tech Lead) and Sun was most interested in was how to avoid custom, site-specific customized versions of /bin/login so we could more easily promote adoption of new technologies like Kerberos without requiring the site administrator be able to modify /bin/login, or having a combinatorial explosion of different /bin/logins for all of the different distributed computing systems which needed changes to /bin/login.

Oh, and Java was released as a Beta in 1995. The reason why PAM modules wasn't written in Java was because (a) it would have been insane to use something the size of a JVM for a critical system program like /bin/login, and (b) Sun Microsystems' marketing arm hadn't yet started promoting Java like crazy promising "write once, run^H^H^H^H debug everywhere", and claiming that Java was the right answer no matter what the question was. Also, (c) I believe that the Solaris engineers, for whom I have immense respect, had way more good taste than that. :-)

Can you email me@christine.website? I'd love to talk and poke the brain of an expert. Maybe even do an interview or something.

And it was a good call. PAM has plenty of shortcomings, but compared to the alternative options, it was a vast improvement.

You make HN interesting. Reminds me of slashdot 20 years ago.

So I'm sitting in a corner of the world and suddenly I come in touch with this nugget of computer history told in the first person.

There is this feeling of an expanse, of seeing something way larger than oneself, that one experiences when this occurs.

Thank you. And thank you HN.

Was thinking the exact same thing. It always amazes me (in a really good way) when there are these "key-people/nerd-celebs" that casually comment and partake on HN ! Like this post or when John Carmack comments on some people's post.

Whenever I come across some code and think "who could possibly have thought this was a good idea?" I realize there is usually a story behind it.

I feel like this comment belongs right along side the PAM source, if only to remind others that we weren't all that nutty coding it this way.

Thanks for this writeup!

Brilliant! I've run into several instances of Chesterton's Fence lately (trying to talk people out of destroying something they didn't see the use of), and this is a great example.

https://fs.blog/chestertons-fence/

(Though counterpoint: destroying something is often the fastest way of finding out what its use was. Needs a certain willingness to be run over by a charging bull though.)

What is/was Yellow Pages? Never heard the term used in this context

You can think of it as a precursor to LDAP.

It was a way to provide centralized access to User/Groups/Password and hostnames information over the network. YP was a staple of most UNIX deployments for some time, before being replaced by LDAP and DNS.

You'll find some historical context here [1].

[1] https://en.wikipedia.org/wiki/Network_Information_Service

Thanks!

Sibling comment links to the relevant Wikipedia page, but most people who've dealt with YP probably know it as NIS (turns out using a trademarked term wasn't ideal). https://lwn.net/Articles/874174/ is a discussion of the current state of the world - Fedora is considering removing support for it in a future release, but it's otherwise still hanging around in Linux distributions.

> https://lwn.net/Articles/874174/ is a discussion of the current state of the world - Fedora is considering removing support for it in a future release

Subscriber link: https://lwn.net/SubscriberLink/874174/693a5853d33f4b45/

OK, suddenly I feel old. When I knew people using YP, we all knew that it was the rightful name, and NIS was the name slapped on it later due to trademark concerns...

If I'm not mistaken, Sun has made that mistake more than once. Bryan Cantrill has often pointed out that the original name of Solaris zones was Kevlar.

And don't forget Happy Meal Ethernet!

Hah! And another interesting tidbit, the guy who invented PAM worked for me. The problem we had was that I had designed this new "secure" name server, that a number of folks were working together to make real and the existing SunOS code could only talk to either Yellowpages (nee YP, nee NIS) or files. I was adding this new thing (NIS+[1]) and we has crossed the threshold that now we had three things you might want to use for authentication (and things like host lookups too!) His name was Andrew, he was Australian and was depressed because he felt like none of the work he was doing would live very long. I tried to explain to him that this part of the problem was very important and it would last a long time but he didn't believe me. It out lasted NIS+ to be sure!

Oh, and I was also in the Java group after I moved from the Systems Group and the first implementation of PAM was in 1991 if you can believe it. It definitely pre-dated Java and when I joined "Firstperson" (which was the soooper seekrit project that was developing 'oak' which was to become Java) in 1992 the target was embedded systems like TVs or set top boxes not the web.

[1] Which was called the Zeus Name Service or ZNS at the time.

Edit, I was misremembering his name, it was Thomas.

Was Hesiod called Hesiod because it documented the names of Athena, Heracles, Achilles, Castor, Thanatos, Charon, etc.?

TIL. For some reason, I thought PAM was an original invention by RH.

awesome! thanks for the story! i'm kinda loving how pervasive this website is getting and how often these "oh yeah, i built that. here's how it went" comments are appearing!

Almost begs for a curated list of such HN posts. The story of X as told on HN series …

PAM is quite something. Normally you ignore it and it quietly gets on with the job. Then one day you find yourself deploying something like this on your laptop:

  %PAM-1.0

  auth  required                   pam_faillock.so      preauth
  auth  [success=3 default=ignore] pam_unix.so          try_first_pass nullok
  auth  [success=2 default=ignore] pam_winbind.so
  -auth [success=1 default=ignore] pam_systemd_home.so
  auth  [default=die]              pam_faillock.so      authfail
  auth  required                   pam_permit.so
  auth  required                   pam_env.so
  auth  required                   pam_faillock.so      authsucc

  -account [success=2 default=ignore]  pam_systemd_home.so
  account  [success=1 default=ignore]  pam_winbind.so
  account  required                    pam_unix.so
  account  optional                    pam_permit.so
  account  required                    pam_time.so

  -password [success=2 default=ignore] pam_systemd_home.so
  password  [success=1 default=ignore] pam_winbind.so
  password  required                   pam_unix.so sha512 shadow
  password  optional                   pam_permit.so

  session required  pam_limits.so
  session required  pam_unix.so
  session required  pam_winbind.so
  session optional  pam_permit.so

The syntax is quite terse and like nothing else. No idea why it insists on wasting characters though. auth could be au, account: ac, password: pa and session: se. Why on earth put pam_ on the front of the modules and .so on the end. Tut!

The winbind stuff is configured elsewhere in several places - /etc/samba/smb.conf and /etc/security/pam_winbind.conf (obviously).

In general it is good enough to keep a spare virtual console logged in as root whilst you play with PAM. Otherwise get some experience with say Gentoo and Arch and the like. Then you will have rather good system rescue skills. Keep a System Rescue CD/USB to hand ... boot, mount /, /boot, /dev, /sys etc, chroot ... fix whatever you broke and then reboot. Rinse, repeat. Or spin up a VM and snapshot/checkpoint it before experimenting.

> Why on earth put pam_ on the front of the modules and .so on the end

Because it uses dynamic linking of the shared libraries used for authentication.

I was being facetious.

The config file doesn't need to worry about the nuts and bolts. All of the PAM modules are named pam_xxxxx.so so the configuration file only needs to specify the name, ie the xxxxx. Also some parts of the configuration are abbreviated, eg: auth and other bits are not, eg: password.

Those "[success = n" things mean "skip this number of lines if this invocation succeeds. It's a pretty mad "if f(x) == true goto n" statement.

Begrudgingly I have to say that despite the oddness of it all, it is reasonably easy to follow what is going on when you understand the rules.

It is because when we wrote the parser for it our goal was to make it braindead simple.

Who is "we", here?

"We"

I suspect you have awoken the Dark Ones (errr the PAM devs)

Honestly if this is how you get in contact with the PAM devs I should have done this a lot sooner.

The Dark Lords are always here.

PAM is pretty odd but not that bad!

> All of the PAM modules are named pam_xxxxx.so so the configuration file only needs to specify the name, ie the xxxxx.

I would rather type a few extra characters so that it is blindingly obvious that "pam_foo.so" in the config file will load code from pam_foo.so in the filesystem.

Typing only "foo" would remove a little clutter, but it would also increase my cognitive load a little.

>> All of the PAM modules are named pam_xxxxx.so so the configuration file only needs to specify the name, ie the xxxxx.

> I would rather type a few extra characters so that it is blindingly obvious that "pam_foo.so" in the config file will load code from pam_foo.so in the filesystem.

> Typing only "foo" would remove a little clutter, but it would also increase my cognitive load a little.

Exactly. Sometimes magic like what the GP wants is evil. It sub-optimally trades a steeper learning curve for some too-small convenience. It's like buying a shiny new penny for a dollar.

> It's like buying a shiny new penny for a dollar.

That's bonk. PAM significantly simplified the ecosystem - at the time there were dozens specialized hack upon hack upon hack login suits such as

* login-otp

* login-shadow

* login-skey

* des-login

* login-rsa-otp

that implemented some of the functionality for some of the manipulations, sometimes on the same system. Some of them already used shared libraries and some of those shared libraries already used the names. Specifying loading paths fully was considered a bad idea because of how subsystems were installed on the deployed servers. The decisions about name collisions ( pam_ prefix ), following Sun's original specs ( some of the names of stages ), etc were made from those constraints. It needed to be deployable on the top of the existing login systems where the users would not be locked out because of some sort of a conflict.

PAM significantly simplified that messy ecosystem while being extremely system, which is why it won and remains the integral part of the ecosystem for twenty seven years.

But hey, this is open source -- don't like it? Rewrite it in Rust! Just remember that first you would need to make sure that Rust exists on all the platforms PAM has been used on.[0]

[0] Pam-static was created as during the first few releases Linux/Alpha did not have support for shared libraries. So there it was, Pluggable Authentication Modules which on Alpha did not have dynamic loaders.

>>>> All of the PAM modules are named pam_xxxxx.so so the configuration file only needs to specify the name, ie the xxxxx.

>> It's like buying a shiny new penny for a dollar.

> That's bonk. PAM significantly simplified the ecosystem...

That's not what I was talking about at all. I wasn't talking about PAM itself, but the commenter's impulse to use an abbreviated name of the module in the config rather than the module's actual name. PAM did that right by using the full name in my opinion.

I misunderstood. My apologies.

udev also suffers from a similar config not-quite-programming-language syntax.

My theory is that the authors of both didn't know about parser generators.

Considering that some of the people behind PAM had their hands in the writing/patching those parsers, I doubt it

A further serious answer for those unfamiliar, the .so files are all found in a PAM module search path. You can replace a search-path-relative path (pam_blah.so) with an absolute path to a .so file if you wish.

    auth required /etc/whatever/my_pam_module.so

And the best part is that it supports conditional logic with goto's! [success=3] mean's jump three rules ahead. Which means you can't really modify this file without considering the whole thing.

This led Ubuntu to create /usr/share/pam-configs which have files like

    Name: SSS authentication
    Default: yes
    Priority: 128

    Auth-Type: Primary
    Auth:
            [success=end default=ignore]    pam_sss.so use_first_pass
    Auth-Initial:
            [success=end default=ignore]    pam_sss.so forward_pass
    Account-Type: Additional
    Account:
            sufficient                      pam_localuser.so
            [default=bad success=ok user_unknown=ignore]    pam_sss.so
    Session-Type: Additional
    Session-Interactive-Only: yes
    Session:
            optional                        pam_sss.so
    Password-Type: Primary
    Password:
            sufficient                      pam_sss.so use_authtok
    Password-Initial:
            sufficient                      pam_sss.so

to try and make something declarative. RHEL/CentOS went the authconfig route but both ended up saying to never ever touch anything in /etc/pam.d

Wait, is this why my arbitrary changes to /etc/pam.d on Ubuntu have resulted in failure?

Maybe! Rewriting PAM is usually only done when messing with packages but can be called manually if you drop files in there with your config management.

“Wow this could be really useful” you might think. “Where “is it documented?”

https://wiki.ubuntu.com/PAMConfigFrameworkSpec

“But where is the reference, how do I know what directives are even available?”

…

This is the part that really gets me about PAM. There really did need to be something where it sits, and the plugin approach is perfectly sound. (Aside: I hope the expressed horror at "plugins in C" is one of the satirical parts, because plenty of people really do have such a poor grasp of computing history.) On the other hand, the implementation of those basic ideas has always been plagued by inscrutable APIs/formats, poor documentation, and unnecessarily catastrophic failure modes - as shown in the talk itself, and some of the comments above. Actually working with PAM is almost as unpleasant as working with SELinux, and neither has to be so bad.

You can always take control back. I have a fair few Ubuntus lying around whilst I use Arch on my laptop and workstations.

Ubuntu has /etc/pam.d/common-auth which looks like the main module to define your intent. Strip out the comments and see what you have left - read the comments first! Copy the whole lot somewhere and login to another console with root rights. Now you can play.

What would you like to do?

> both ended up saying to never ever touch anything in /etc/pam.d

As I've learned the hard way, at least keep a root shell open somewhere if you do...

It's important to be aware that this goto nonsense is not PAM. It's not in Solaris PAM nor in OpenPAM (the BSD's PAM, as used in macOS, FreeBSD, NetBSD, and DragonFly BSD.)

Inventing bizarre and unusual turing complete configuration languages is a GNU/Linux hobby - just look at UDev reals.

Characters are cheap nowadays. I'd much rather it be readable and understandable than have a bunch of cryptic abbreviations.

He's making a joke exactly about that.

It's not readable and understandable, though.

I have more experience with PAM than most Linux admins, I think, and yet I still have to refer to man pages or Google every time I need to do something with it. The keywords and options are already basically nonsense.

Not to mention that linux-pam's `[success=2 default=ignore]` isn't even standard, I think OpenPAM only has the five control flags required/requisite/sufficient/binding/optional. (One may reasonably despair as to the difference between "required" and "requisite".)

It's almost as if PAM were developed before the invention of the if-then-else construct in the 1950s.[0]

[0]: https://github.com/e-n-f/if-then-else/blob/master/if-then-el...

Originally pam.conf was envisioned to be quite simple. Say, about the number of lines in /etc/inetd.conf. It was only later that people started doing these really complex things, and most of that was because Linux distributions were shipping far more distributed computing components than was originally were envisioned in 1995.

Since ordering was critical in PAM config files, and it was presumed that typical sysadmins weren't going to be editing PAM config files, it had to ship with mentions of every single package that might require a PAM config file. And this is what caused it to get super complex --- and not very well documented, since the presumption was that only distro-engineers needed to understand it, and tech writers have always been underappreciated and underpaid for as long as I can remember in the field (and probably longer).

Just to say, if I'm reading a config file and see "pam_winbind.so", I have a much better idea of what's going on than if the same file says "winbind" instead.

You already know that it's PAM-related, since it's a PAM config file. What extra info does the `pam_` give you?

.so does give you some info (namely, that it's an external shared object), but I'm not entirely certain how that alone gives you any better idea what's going on: that shared object could do literally anything. You need to know how PAM uses the shared object to know what's going on.

> You already know that it's PAM-related, since it's a PAM config file. What extra info does the `pam_` give you?

The filename, obviously.

If you want to play with PAM, the best thing to do is to create a test program which links to the PAM library and which uses a different PAM service name so you can play with a config file which is different from the ones which ssh, su, sudo, etc., rely upon. That way you don't have to worry about screwing up your system and not having a way to recover.

An example of such a test program can be found at http://pamtester.sourceforge.net. I haven't tried using it myself, but I have taken a quick look at the source code, and it appears legit, if a bit over-engineered. Creating a test driver program which links against libpam really isn't all that complex, especially if you use the "configure the program by editing /tmp/foo.c and recompiling" methodology. :-)

I started playing around with PAM when I set up this new laptop.

I make use of the pam_u2f.so module to require a FIDO2 fob AND password for my logins.

I even configued it to have backup fobs allowed in case I lose my primary, secondary, etc. (which I store in exceedingly secure spaces).

It's great that somebody would need both my password & my FIDO2 to access my laptop and there's no clear indication at login that the FIDO2 is required first if it isn't plugged in.

> because /etc/passwd is world-readable by design for some arcane reason that I can’t find on Google

Because it maps user IDs to names so that you can see names in `ls -l` and `who` and such.

Also things like "finger" looking up their home directory and showing you their .plan file. Not saying it was a good choice then, let alone now, but the number of Chesterton's Fence violations in this presentation is pretty high.

> the number of Chesterton's Fence violations in this presentation is pretty high

The article's tagged as satire, I'm not sure you should be expecting good logic and sources

Is there anything besides the tag to indicate that it's satire, in a sense that any teacher or student of writing would recognize? People have used "just kidding" as a way to duck criticism since time immemorial, but that doesn't mean we have to accept it unthinkingly.

> This is a partially satirical talk. It is intended to be mostly factually accurate, however some of the details are stretched for comedic effect.

I'd think this would cover all the bases. But what do I know, I just wrote the talk.

Yes, you did, and it was a good talk. I actually did enjoy the snarky tone. PAM is a mess. On the other hand, I don't think that disclaimer really does cover all the bases. You yourself say it's partially satirical, implying that other parts are sincerely meant. Why shouldn't those be subject to the same discussion, correction, or even criticism as any other sincerely meant opinion in any other thing linked here? I'm pretty sure you can handle that back-and-forth better than Sir Lexicality gives you credit for.

And if anyone's still offended: my comments are partially satirical too. There, I'm immunized.

This is the perfect cover to do whatever you want at all times.

I made an incorrect statement? Don't worry, I _meant_ to get that part wrong.

This part here that was insightful and you enjoyed? I meant that part 100%.

On the other hand, if it hadn't been playful and glib about the decisions that led up to PAM as we know it, we might have had fewer of these lovely greybeards pop up to teach us some PAM and Unix history in the discussion :)

maybe Xe can address their comments in a followup, perhaps even after consulting with them a bit on the project itself

And the sensitive parts were split off into the root-only /etc/shadow a long time ago.

I mean anything that happened in the 90's seems to be considered "yesterday" in computing history terms.

That would be more like E.G. the 1950s or 60s in Aviation terms, or perhaps the 1920s in electricity terms.

But that doesn't really answer the question. Nothing was really stopping a setuid program from granting access to the non-sensitive bits of the passwd/group files.

Efficiency (spawning setuid program for every line printed by ls -alv is a bit too much), and you don't want to make too privileged programs too (suid ls?)

I mean the hypothetical program would have just dumped the whole file minus the sensitive bits.

Exposing a setuid program that prints the file would be more dangerous though. The result is the same (you can read /etc/passwd) except now you have a vector for gaining root access through an exploit to this setuid program.

see getpwnam, which was already public for years and linked by everything. it queries one record at a time.

getpwnam() gets its data by parsing /etc/{passwd,shadow}[0]. It isn't a magical system call that elevates the privileges of the calling program.

[0] assuming your nsswitch says so

yeah. agree, was merely pointing out the public interface for querying passwd records for the purposes of highlighting why said interface might be another reason to reconsider the proposed idea of a setuid shim that reads a read protected /etc/passwd that contains password hashes and returns only the public components.

And then you've reinvented /etc/shadow but worse. :)

...and nobody thought of dictionary attacks when it was designed.

So it was world readable.

Sounds like maybe "passwd" wasn't the best name for this file, then

Not anymore, but it is way too late to change it now, especially since it has been user readable since it was created.

It's really just "accountdetails" now, with the real password living in a different file.

Of course I remember the old "yp" system that would blast the whole file with passwords over the network to anyone who asked. The old Unix guys had a lot of faith in their hash algorithms.

Also because there wasn't anything deemed secret there, as the passwords were salted and encrypted. Knowledge about password crackers and crypto became widespread only much later. The pw encryption algorithm was also pretty feeble, a version of some ww2 era rotor machine IIRC.

I'm the author of the talk if you have any questions. This was a fun talk to write, easily one of my favorites.

In SSH you can limit the login to a specific user or user with less permissions than root.

Is that something your PAM module(s) would need to handle?

e.g. Now that you're authenticated via the IP, how do we authorize the login?

That is something that I have yet to really figure out. I would love to have the ability to send an authz message to the machine connecting to that other machine, identd style. I also figure that this could also result in a GUI prompt to confirm it. Maybe root access over tailpam would need you to be a tailnet admin? I'm not totally sure. Still very much thinking this through.

Either way this needs a lot of work to be super secure and useful, right now it's mostly a proof of concept that has taken FOREVER because debugging PAM is so painful. There's more to come in the future, but I fully expect this to be a bit of an uphill sell to people. Writing stuff that mucks with the authentication stack is rightfully kinda scary, and there will need to be a lot of security and subject matter expert review. I expect this to be a slow burn, but good god I would love to see it happen.

Well with ssh usually you need to configure which keys are allowed per user so maybe `~/.tailpam_authorized` with either user, group, or ip (similar to tailscale config) could control access per user.

Alternatively, now that it's authenticated securely enabling password for authz could work maybe?

> I fully expect this to be a bit of an uphill sell to people. Writing stuff that mucks with the authentication stack is rightfully kinda scary, and there will need to be a lot of security and subject matter expert review. I expect this to be a slow burn, but good god I would love to see it happen.

Actually using it would be a long ways off, but I don't think conceptually it's a hard sell. I'm interested in making authentication easier for my developers lately, and also in dodging some key management in favor of accounts people already have at work. Right now, I'm evaluating Tailscale for VPN access, since I find myself supporting developers without (yet) any remote access to them or any endpoint management crap on their machines, and walking them through configuring WireGuard has proven painful (more painful than I expected, which maybe sounds dumb). (Incidentally, ‘Tailscale’ is a name I remembered as a possible VPN solution because I think your blog is good.)

One of the things that's occurred to me during my testing is that if it could be as stunningly easy to configure as Tailscale, it'd be great to use some kind of SSH gateway that plugged into everyone's SSO instead of managing a bunch of SSH keys. Plus, being able to have admins confirm privilege escalation requests might make it easier for me to convince some higher-ups to give the developers I support sudo rights.

There are products that support things like this, but I think the use case is a natural extension of what people are likely already using Tailscale for, and it would be a killer feature to integrate.

Would you mind dropping a link to the code... somewhere... close to the talk? I clicked through the youtube link on the site and couldn't find it, it doesn't look like the code repo is in the top several on your GitHub, and I would rather not pull out my phone to scan the QR code (though that would be a good way to send the link visually).

Sorry about that! https://github.com/tailscale/pam. I'll edit the talk markdown when I get back home.

Hey - thanks for the help! I've been thinking about getting into writing PAM modules for myself for a couple weeks - maybe this will get me started!

PS - great talk. Security/auth is tricky, and it's never clear when I'm doing the right thing or something _supremely dumb_. Thanks for putting yourself out there.

I wrote pam_ldap and did the initial port to Darwin. It is probably showing its age - you probably want better integration with GSS-API and also something like CredUI [1] for GUI support. But it's a great deal better than having to modify /bin/login each time you add an authentication mechanism! And although each implementation has its quirks, its age and thus relative simplicity means that it was quite widely implemented.

[1] https://github.com/PADL/CredUI

In this vein, I wrote a blog post on writing your first PAM module here [0].

[0] https://rkeene.org/projects/info/wiki/222

> each time you add an authentication mechanism

I don't do that. No one ever does that. Normal people don't keep a picturebook of authentication mechanisms that they need to cycle through. All the non-/etc/password PAM modules that some clueless IT ever forced on me (for as long as it took me to root and purge them from existence) to hook up with some network service just end up breaking your user account. The highlight is when they don't even authenticate with the network because they instead cache credentials.

Oh it's this awesome blog again.

If I understand Tailscale correctly a PAM module that knows which machine it is running on and the username logging in will allow them to decide on a per-machine level if the current client is allowed to go there (to this account), so it could bring their authentication game to a whole other level.

Not sure if they should do that (or if anyone should do that) but it's an interesting thing to think about. Even if it's just to hide away some of the pain that are the PAMs/LDAP Groups of this world.

I wrote a bunch of words here: https://news.ycombinator.com/item?id=29169348

i like this post, both style and content. only thing missing would perhaps be a small reference to nis, nis+ and netinfo which were early attempts to build distributed and centrally controlled databases for /etc stuff and ye old faithful which i always preferred for production systems which was to rsync flatfiles out for maximum reliability.

not sure if i totally agree with the immutability argument for /etc, but i suppose that's more about protecting against errant devs abusing sudo privs to do weird things rather than securing things against malicious actors.

> ye old faithful which i always preferred for production systems which was to rsync flatfiles out for maximum reliability.

This is the best way. The key is making sure that somewhere you're managing user ids, cause cleaning up after assigning the same uid to different users on different systems gets icky.

Some of us like to keep /etc files in git, and git clone them, and check out a branch.

also forgot, it's important not to forget the spiritual successor to the client side of nis which is sssd. i think it might be a redhat or canonical thing. either way, if you are insisting on doing live network queries for passwd entries, you generally want it, as it will cache in the event that the network goes down.

one of the reasons to prefer flat files is if getpwnam and friends start to hang or return inconsistent results, the system will become unresponsive or misbehave in all sorts of amusing ways.

Given that the link here is stylized as "Pam" and not "PAM", I was really hoping this was about the cooking spray (https://www.pamcookingspray.com).

I can't view the site due to a cloudwall and thought it was about Hurricane Pam until I read the comments here.

Maybe I haven't watched enough presentations but this is the first time I've heard /etc pronounced like "Etsy". All these years I've been wasting syllables saying E-T-C

How do you pronounce SCSI? :)

I think the craft-marketplace Etsy is actually named after that pronunciation of "etc"

("scuzzy")

Fun fact: in the very early days, people actually suggested that it should be pronounced "sexy" but its true nature asserted itself and "scuzzy" won out.

And despite it having left the everyday vernacular of your average computer user, its solidly embedded in many other protocols (and frequently tends to be the "working" part) because like TCP, AT command sets, etc tends to actually be pretty good following 20+ years of "bug" fixes. Even the glorious NVMe (which initially just looked like the merging of a hba+scsi) just embraced SES in its entirely.

So, when you say USB, I reach for my SCSI manuals.. :)

Yep, and other protocols that borrowed the SCSI command set are still busy reimplementing inferior versions of tagged command queuing. The semantics that SCSI already had in 1994 when I started working with them weren't perfect, but that only makes it even more amazing that protocols have mostly gone backwards in the 27 years since. Many people seem to enjoy standing on the toes of giants.

FWIW I’ve heard it pronounced that way since the early 90’s. Not sure how/when it started though.

etc stands for et cetera ("and others"), or "et c."

The latin word "et" ("and"), and the single abbreviated letter C.

I actually write "etc." (in prose) as "et c." for this reason.

I like to pronounce it [e`ts] when I think about it internally, but I live in Norway so I think about it that way because of the Norwegian word “etse”. When I talk about /etc to someone else I usually spell it out e-t-c though.

It's odd that I almost never encounter people pronouncing it like the English abbreviation, "et cetera". I suppose it's more syllables -- "ee tee cee" versus "et set err uh" or "et see" -- but not that many. I've never met anyone who pronounced /home "aich oh em ee" or /dev "dee ee vee".

> I suppose it's more syllables -- "ee tee cee" versus "et set err uh" or "et see"

> I've never met anyone who pronounced /home "aich oh em ee" or /dev "dee ee vee"

I think fewer syllables is _exactly_ why people don't say those other pronunciations for /home and /dev. If there was a reasonable way to pronounce /etc as a single syllable, I'm almost 100% certain that would be how people pronounce it.

I've always said "etcetera" rather than like goatse, but I guess as with all languages, there will be accents in unix as well.

(I "grew up", as it were, pronouncing fsck as "fossick" which seems appropriately onomatopoeic.)

Like Polkit, sometimes I wonder if the point of these super complex authentication systems is to assure that its impossible for a mere mortal to audit the system.

I tried to demystify it once: https://venam.nixers.net/blog/unix/2020/07/06/dbus-polkit.ht... Maybe it can help someone else understand it too.

Not the main point -- and I don't know why a filesystem needs to be in kernel space -- but the swipe at network filesystems is wrong. Of the ones I know, at least OrangeFS can use public keys, if that's what you want (though I don't know if they encrypt traffic for use outside something VPN-ish). I don't know what the current story is on AFS encryption (which was weak), but NFS can use Kerberos 5, not that you use /etc/shadow with Kerberos anyway. I'll defer to Ted Ts'o on whether it's inappropriate these days, but I've used SSH for ages with Kerberos.

If you expect that "domain users" will always do their first authn to the machine remotely via SSH, then you can skip making a custom PAM module, and instead set up the combination of domain SSH authentication, with dynamic provisioning of local POSIX accounts. It's honestly much less of a headache.

1. dynamic provisioning: in /etc/pam.d/sshd, add "pam_mkhomedir.so" as a session module. Easy as that.

2. domain SSH auth: in /etc/ssh/sshd_config, add an "AuthorizedKeysCommand". This is a program you'd write, that should accept a username on the command line (the username passed by the SSH client), and should emit a list of valid SSH pubkeys for that user. This slots in place of the default SSHd behavior of reading $USER_HOME/.ssh/authorized_keys — `cat ~/.ssh/authorized_keys` is essentially the "built-in" AuthorizedKeysCommand.

The key insight with step 2 is that you can make this authentication step serve double-duty as an authorization step. If you don't want a particular domain user on this system, you can just have your AuthorizedKeysCommand emit no acceptable SSH keys for that user, leaving them unable to authenticate.

If and when the AuthorizedKeysCommand authenticates+authorizes you (by whatever means), the pam_mkhomedir module will turn around and make you a local POSIX user and home directory to correspond with your domain account (with whatever username the client provided, so make sure to validate that), and you'll end up logged into said local POSIX account.

(You won't have a password configured / any way to convince PAM to re-authenticate you while logged in; but that's fine for most things — you can still fork out new processes, start new sessions of screen/tmux, etc. The only consideration is sudo access: just ensure that the sudo group has the NOPASSWD attribute set in /etc/sudoers.)

This is essentially how GCP's "IAM-based" OS Login (https://cloud.google.com/compute/docs/instances/managing-ins...) works: there's an AuthorizedKeysCommand (/usr/bin/google_authorized_keys) that fetches the SSH keys recorded in IAM for the SSH username [= escaped GCP IAM ID] you're claiming to be logging in as, and emits them if-and-only-if that GCP IAM ID has the OS-Login IAM role; and then PAM's dynamic provisioning kicks in and handles the rest.

Also, for a maybe-clearer example, I wrote my own little toy AuthorizedKeysCommand, that uses Github as an authentication domain (https://github.com/tsutsu/github-auth3). You tell it a Github org, and then anyone in that Github org can log in by with their Github username + an SSH key registered to that Github user. (Again, it checks the username for org membership, and only if the user is in the org, prints the SSH pubkeys registered to the user.)

=====

Of course, compared to doing everything in PAM, this approach isn't as elegant: every "domain user" that logs into the system ends up being able to leave junk there—at the very least, they end up with a UID permanently allocated to them in /etc/passwd.

You can ameliorate this slightly by NFS-mounting /home; or even individual homedirs as-needed using autofs.

(Intriguingly, macOS ships by default with autofs preconfigured to mount homedirs for OpenDirectory users, despite Server.app not exposing any such OpenDirectory capability — presumably there's some Apple-internal setup for their workstations that seeks parity with Windows' Roaming User Profiles.)

> However the peak of network filesystem security on Linux is “don’t get your network hacked lol”.

I think this is incorrect. If you want encryption that's built into NFS (instead of a kernel VPN like IPsec/Wireguard or tunneling) then there is krb5p.

https://whyistheinternetbroken.wordpress.com/2017/07/24/onta...

IPsec also has transport mode if you don’t need a whole VPN.

When you want to 'bookmark' the discussion on your hacker news thread history, but only the discussion...

If you rely purely on Tailscale networking, and trust any process on the client, why do you need PAM (assuming OpenSSH sshd configuration if you can't simply listen on that address)?

Misguided. This is IP authentication misusing PAM.

Not misguided. The IP is from a wireguard network setup by tailscale which only allows packets signed by the private key of the device assigned that IP. It's as secure as an SSH key pair would be.

The only question left for me really is the authorization of which users you're allowed to login as.

Bingo. I expect that to be the more complicated issue, which is much more interesting and will require collaboration to accomplish. Either way I just wanna see where this rabbit hole goes.

is it a wise to grant root to your servers to any process running on your machine?

I probably wouldn't just rely on SSH host key pairs either outside something like a compute cluster.

Why do so many tech blogs load up their stuff with stupid memes?

Author of the talk here. I am sorry if you are disappointed. If you like I can process a full monetary refund.

Because then it's cute and accessible and if people disagree with the content or point out any mistakes it was all for fun and whimsy anyways. I wasn't meant to be taken seriously. Oops!

> because /etc/passwd is world-readable by design for some arcane reason that I can’t find on Google

It is pretty comical when people who do not know basics of how the system permissions work wax about authentication and authorization used by the same systems.

Especially when it is not an arcane reason at all.

There was a time when it was considered ok to have the salted and MD5-hashed password available to all users.

Then people learned to bruteforce the password and so the hash itself was moved to shadow file but passwd was left because a lot of platform software and scripts depended on it.

This may shock people nowadays but back then this was fine. I remember having to explain to the users (and sometimes loosing) the need to have a password at all.

Eh, no, it is world readable because even though programs that manipulate/read hashes are privileged, programs that map <uid>:<gid>:<name> are not. This is still the same. If you want to be able to use "ls" as a non-privileged user and see names, you need to be able to access /etc/passwd.

So... what did I write? To quote:

"but passwd was left because a lot of platform software and scripts depended on it."

because it is affecting libc and glibc, not the platform software or scripts depending on it.

A different world.

The confusion rests on the name of the file I think. If it were named "accounts" (closer to its use) people wouldn't be too surprised it was readable. Then shadow could have been named "passwords."

I've ... had that argument, and only very barely succeeded in prevailing, within single-digit years.

With a user who's lost multiple accounts to weak passwords.

Or should I say, the same weak password. Which they still use in some contexts....

I'm of the opinion that allowing users free choice in passwords is no longer viable. At a bare minimum reuse of any known-compromised password must be rejected.

RMS famously campaigned against passwords in his early pre-GNU days at MIT (though I'd say his perspective was specific to the computing environment then; it was more of a matter of how a small group of hackers should cooperate to use an expensive machine, he wanted everyone to have access to everything).

Pam is one of my least favorite characters from the office. She isn't funny, she just causes drama and cries all the time, and on top of that is pretty manipulative, awkward and unpleasant. Once the show starts doubling down on Pam & Jim, around season 5, the show takes a huge dive.

Seasons 2-4 are pure gold, but season 5 and on is dog water.

Reply on news.ycombinator.com