Hacker News Reader

Comment by crabbone | Hacker News Reader

Here are some:

* Doesn't scale. Having passwords in a plain text file is not a scalable solution for users directory. Can probably go up to a hundred users, but not much more.

* In computer clusters you want user identity to "stick" to the user when they use multiple machines, containers etc. That's why you have LDAP... but it doesn't help all that much because user id is encoded into the file system (huge mistake...) which makes it very difficult to contain users to things they should control. If your only mechanism was the /etc/passwd, it would mean you'd have to constantly synchronize this file across all those machines and containers you have.

View on HN

It may be old and not particularly appealing, but LDAP has been serving that role effectively at many companies.[0]

Using a terminal remains standard practice for sysadmins and devops.[1]

I believe there's some confusion in both the article and the comment between authentication and authorization. LDAP is fully equipped to handle both tasks.

[0]https://access.redhat.com/documentation/en-us/red_hat_enterp...

[1]https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-co...

To anyone reading this and thinking "yeah dummy, of course it doesn't scale because you're not supposed to store passwords in plain text in the first place" I'll direct you to Chapter 7ish of The Linux Programming Interface.

If you look in your /etc/passwd right now, you'll almost certainly see a single "x" where the (EDIT: no, it was still encrypted!) password originally was - nowadays that single "x" is an instruction to go look in /etc/shadow instead, for the salted hash of the password you're trying to check.

I think this minimizes the number of users who need read permissions to /etc/shadow, and the amount of time they need it for.

This has been your seemingly useless bit of Linux trivia for today. :)

plaintext vs plain text

unencrypted vs unstructured

Of course, unstructured is also incorrect; the passwd and shadow files have structured records, one per line.

…and being structured, the passwd file content should be accessed with the getpwent family of functions.

Which unfortunately are not thread safe.

"unencrypted" is normally written as "cleartext". "plaintext" means "(readable / intended to be read) without a special viewer". Your.ssh /id_rsa is plaintext but not cleartext.

/etc/shadow was born not because /etc/passwd had a plain text password but because the hashes became crackable and /etc/passwd is a public read file. Linux has never had them. Here's the man page indicating encrypted passwords for Unix v7 /etc/passwd release in 1979: https://man.cat-v.org/unix_7th/5/passwd

Whoops! My bad, this is an even better bit of trivia.

My mistaken memory really sells the underlying point that everything old is new again.

I have a vague recollection of my 20 floppy of Slackware already having /etc/shadow. That would have been fall of 92 or winter 93, based on where I was living at the time.

I have a vague recollection of being given the choice on a 90s vintage distribution with some warning about security and password length if I did not use shadow passwords. At some point in the early 2000s we started authenticating regular users against AD but the shadow file was still there for root.

We had a couple of labs of sparcstations that just went away a couple of times a year because something bad would happen with all of the NFS mounted partitions and they'd have to turn the cluster on one box at a time to prevent thundering herd issues with NFS.

I think they may have been mounting parts of /etc as well. People get the idea that managing accounts for a cluster of boxes should be centralized. It's all fun and games until the network mount disappears.

Shadow file was definitely from quite a while ago.

Can't remember exact date, but might have been around time of SVR4 intro.

I know because I remember going "ugh", but without investigating the reason why it (shadow) was introduced :) - which was of course wrong on my part.

That was not a "plaintext password," it was a DES hash (from 7th edition onwards).

This is the same format used by the classic htpasswd utility.

https://en.wikipedia.org/wiki/Crypt_(C)#Traditional_DES-base...

My school had 30k students, including grad and doctoral students. When they gave students shell accounts, they tried to put them all onto a single Sequent box. They got my incoming class, the following, and anyone previous who asked for one onto that box. I’m pretty sure it had an /etc/password file, and would have had about 8-10k people on it.

After that they gave up. Even with aggressive ulimits it was too hard, and each new class was apportioned to a separate Sparc. Which was a shame because we learned an awful lot about Unix administration from users pranking each other, figuring out what they did and how they did it, protecting yourself and retaliating (some people would prank others without first protecting themselves).

Made it a lot harder chatting with people from other classes as well. For electives and humanities you weren’t always in classes with people your exact age. They could be ahead of you or behind.

> Doesn't scale. Having passwords in a plain text file is not a scalable solution for users directory. Can probably go up to a hundred users, but not much more.

This simply cannot be true. A users directory that is frequently read will be in cache. The file is also fairly simple to parse. Even on old hardware you should be able to parse thousands of users.

Exactly, even large textfile based DNS servers have capability to "compile" the textfile to a db file for faster access.

So what if they do?

This file is a public interface exposed by Linux to other programs. So what if Linux caches its contents when eg. Python interpreter on launch will read this file. And it's not coming from some "fake" filesystem like procfs or sysfs. It's an actual physical file most of the time.

the python interpreter reads /etc/passwd on launch? i guess it's looking for home directory

It's more of a problem of the number of nodes your users will be accessing than the number of users. It's a PITA to make mechanisms for password changes, for adding and removing users, for unlocking accounts, etc. when you have a distinct file on each server that needs to be changed. As well as adding new nodes to the environment and bringing them up to date on the list of users and etc.

Fair point, networking always makes things more complicated. I'm not sure it's really that much more complicated though. Like any concurrency problem with shared state, a single-writer and multiple readers keeps things simple, eg. a single host is authoritative and any changes are made there, and any file changes trigger automated scripts that distribute the changed file over NFS, FTP, etc.

As long as new nodes start with the same base configuration/image this seems manageable. Simple to do with inotify, but even prior to that some simple C programs to broadcast that a change is available and listen for a change broadcast and pull the changed file is totally within the realm of a classic sysadmin's skillset.

You would have to lock the file, or guarantee consistency in some other way. Right now, I don't believe Linux does anything about consistency of reads / writes to that file... which is bad, but we pretend not to notice.

So... the system is kind of broken to begin with, and it's kind of pointless to try to assess its performance.

Also, it would obviously make a lot of difference if you had a hundred of users with only a handful being active users, or if you had a hundred of active users. I meant active users. Running programs all the time.

NB. You might have heard about this language called Python. Upon starting the interpreter it reads /etc/passwd (because it needs to populate some "static" data in os module). Bet a bunch of similar tools do the same thing. If you have a bunch of users all running Python scripts while there are some changes to the user directory... things are going to get interesting.

> You would have to lock the file, or guarantee consistency in some other way.

I think the standard approach to atomicity is to copy, change the copy, then move that copy overwriting the original (edit: file moves are sorta atomic). Not perfect but generally works.

I agree that this approach is not good for a users directory, I'm just disagreeing that the reason it's not good is performance-related.

Moves are atomic. During the move, at no time is it possible to get the contents of file 1 and file 2 confused when reading from the file descriptors. (Confusion by the human operating things is eminently possible.)

Most systems come with "vipw" which does the atomic-rename dance to avoid problems with /etc/password. In practice this works fine. Things get more complicated when you have alternate PAM arrangements.

A whole bunch of standard functions like getpwents() are defined to read /etc/password, so that can't be changed.

`getpwents()` is not defined to only read `/etc/passwd`. There is only a requirement that there is some abstract "user database" or "password database" (depending on if you're reading the linux man pages or the Single Unix Specification man pages).

In practice, `getpwent` on linux uses the nsswitch mechanism to provide return values from `getpwent`. One can probably disable using `/etc/passwd` entirely when using glibc if: all users do use `getpwent`, and you remove `files` from the `passwd` entry in `/etc/nsswitch.conf`.

I had >30k users on a bunch of systems in 2001 (I inherited that approach, mind, I'm not -recommending- it).

We moved to LDAP a couple years later because it was a much nicer architecture to deal with overall, but performance and consistency weren't problems in practice.

> user id is encoded into the file system

This is kind of unavoidable, but you do have 32 bits to play with. Windows did it slightly better with the SID: https://learn.microsoft.com/en-us/windows-server/identity/ad...

> which makes it very difficult to contain users to things they should control

It's not the file system that's the problem here, it's that "everything is a file" is not true for a whole bunch of important stuff that you might want to apply access control to on a UNIX system. Such as the right to initiate TCP connections. This sort of thing is why containers are so popular.

NIS and LDAP do let you have a large number of users. Heck, we managed a few thousand users in /etc/password back when I was running https://www.srcf.net/ .. in 2000.

> it's not the file system that's the problem here, it's that "everything is a file" is not true for a whole bunch of important stuff that you might want to apply access control to on a UNIX system

I wonder if there has ever been an attempt to really lean into, and push the limits of sticking with the "everything is a file" philosophy in this realm.

I.e. how far could you get with having special files for fine grained permissions like "right to initiate a TCP connection", and making access control management be, essentially, managing which groups a user belonged to?

I think that was Plan 9.

Plan 9 probably took this the furthest. Sad it didn't take off. https://en.m.wikipedia.org/wiki/Plan_9_from_Bell_Labs

I think Hurd and Plan 9 take the EIAF further.

Plan9 tried to "remedy this".

But in reality a file is not a good abstraction for an internet socket. The ACLs would in essence spell out firewall rules. Because the bigger question is where can it connect to than "user" that is connecting.

That's why this is done on the level of kernel networking, where kernel knows what process is trying to open a socket and can firewall it.

This sounds like a completely unrelated thing and you are not constrained by the plain text password/shadow file for scale. NIS existed for many decades. You can even use Active Directory (or samba) for authentication and user management.

But the article is not about this at all.

This shows you either didn't read what you replied to or don't understand the subject.

This file is the public interface of the Linux system to everyone who wants to get information about users on the system. It doesn't matter that alternative tools exist: they were already mentioned in the post you replied to. It's not the point...

If I didn't understand I'd be grateful if you could explain it.

As far as I know that file does not get referenced for the users in an external directory server. That's how the systems scale without needing to put the users in the file. Aren't we talking about a high number of users (and their authorization levels) when talking about scalibility in this case?

> This file is the public interface of the Linux system to everyone who wants to get information about users on the system

No it isn't. PAM is. The password file is only one of the places where users might be defined.

I've wondered why we don't have a passwd.d folder, the way we do with other things in the UNIX filesystem, with individual user accounts represented by individual files. Could even retain the same line-oriented format, just stored separately.

I think a directory of 10000 files is worse than a file of 10000 lines.

Maybe, but account synchronization across machines might be a lot simpler and easier.

> * Doesn't scale. Having passwords in a plain text file is not a scalable solution for users directory. Can probably go up to a hundred users, but not much more.

Why not? A file 100,000 line file will only take a moment to scan.

That's the first understandable explanation I've ever heard of what exactly LDAP is and what it's for!

If you have services with even a few users you should set up LDAP (openldap is fine) and then experiment with it.

LDAP is pretty old and very well supported overall. At one point I configured ldap for postfix virtual hosting. I chose LDAP rather than a database backed solution because of widespread first class support everywhere. The same directory later enabled me to use a ton of other things including SSO. You're always finding new ways to use it once you have it.

It's a great skill to have and is nowhere near as complicated as people make it out to be, including live replication.

Reply on news.ycombinator.com