In Unix Everything Is a File | Hacker News Reader

In Unix Everything Is a File

ph7spot.com

70 pointsmutor56 comments

Open articleView on HN

Any UNIX old timer knows that actually:

In UNIX everything is a file, except when it is not.

> In Plan 9, everything is a file

Fixed that.

> > In Plan 9, everything is a file > Fixed that.

It was a short snarky comment. Also the article mentions that too at the bottom.

Yeah they "fixed it" but by that time not enough people cared. The abstraction purity was nice and developers and OS fans loved it, but it just didn't matter for most application. Another way to put it is that Linux was already fairly decent and it wasn't worth switching to a whole new OS just to have everything be a file.

Another obstacle was that Plan 9 wasn't marketed or pushed enough. Imagine if Google backed Plan 9 and pushed it as the Android OS, put it on Chromebooks, updated it regularly etc. That would be enough for people to switch to it.

As we discussed the other day, there was still some issues with stuff like graphical performance with such a concept, suitable for gaming and such.

In any case, Inferno with Limbo would be much better, and to pick on your example not much different from Android's architecture.

I understand why it didn't take off, but I don't think it's true that Plan 9 was just a tiny incremental improvement over UNIX's architecture. Modern networking and distributed systems would be a LOT nicer today if Plan 9 had taken off in the 90s. I think the economic benefit over today's state of affairs would have been massive, not just something that appealed to computer geeks' sense of elegance.

> Modern networking and distributed systems would be a LOT nicer today if Plan 9 had taken off in the 90s.

But it didn't take off. That's already decided. It wasn't enough. One can say well if it happened 10 years earlier maybe, it would have taken off. Or if Linux never happened. But it's speculative.

Heck, Plan 9 source is there. If so much economic benefit can be derived, why hasn't it become a dominant player. It had been around enough time, and there are enough developers around to have flocked to it and make it overtake Linux by now.

> I think the economic benefit over today's state of affairs would have been massive,

I think we already reaping enormous benefits open source licensing model in general. We live in a world where you can download production grade operating systems, databases, compilers, libraries for free and also inspect their source. That is often taken for granted and is incredible compared to who it used to be.

I can't really disagree with most of your post.

The reason Plan 9 or its ideas can't be resurrected very easily now is because every machine on the internet has to pretend to be UNIX and speak a thousand ad-hoc protocols. UNIX's many flaws have been papered over, one by one, by incredible brute force (instead of in one fell swoop). I'm only lamenting the fact that the 90s were a crucial period in the development of the commercial internet, and in another timeline it could have developed in a very different direction.

>if Plan 9 had taken off in the 90s

Plan9 is a joke, file abstraction is a ridiculous idea. File abstraction is terrible for anything beyond storing information.

It sucks for multimedia, IPC, input. Unix showed that, escaping from file abstraction in any domain (except networking, maybe). That's why we use d-bus instead of pipes, opengl instead of writing buffer etc. People need disposable, modular and introspectable APIs, like in mirageos, lisp machines or smalltalk OSs, not a ridiculous file crap.

Well my only examples were networking-related, but something like D-Bus could easily have a filesystem-based API instead. I'm willing to accept that files are a bad abstraction for graphics. I don't really know anything about that domain, but UNIX didn't get graphics right either.

> UNIX didn't get graphics right either.

That is a bit more debatable, as X11 is not the only way to do graphics on UNIX.

I would argue that among UNIX variants, NeWS, NeXTSTEP and OS X did got graphics right.

I think X-Windows did not get graphics right

Bravo! I wholeheartedly agree that X-Windows didn't get graphics right (and that you're right to refer to it by the name "X-Windows" to annoy the X-Windows Fanatics, who take it upon themselves to correct you every time you call it X-Windows)!

https://medium.com/@donhopkins/the-x-windows-disaster-128d39...

>X Graphics: Square Peg in a Round Hole: Programming X-Windows is like trying to find the square root of pi using roman numerals. - Unknown

As I stated before (which ernst_klim chose to ignore, but apparently strongly agrees with), Eric S Raymond was full of shit when he wrote that "in Unix everything is a file" in his terrible book "The Art of UNIX Programming", just as he was also full of shit when he wrote "given enough eyeballs, all bugs are shallow" and falsely attributed it as Linus's Law, and ESR was full of racist hatred and bullshit when he more recently wrote "The average IQ of the Haitian population is 67... Haiti is, quite literally, a country full of violent idiots" (although ernst_klim may not agree with my opinion about that ESR quote).

http://esr.ibiblio.org/?p=7011#comment-1685142

The extensible linguistic approach that NeWS took towards a graphics API is also a great approach to a file system API, which I summarized in the recent discussion of "The everything-is-a-file principle - Linus Torvalds":

https://news.ycombinator.com/item?id=17532320

>I prefer the "everything is a computer" API. (Which happens to be Alan Kay's model of object oriented programming as message passing.)

https://softwareengineering.stackexchange.com/questions/4659...

>"I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages (so messaging came at the very beginning -- it took a while to see how to do messaging in a programming language efficiently enough to be useful)." -Alan Kay

>NeFS -- aka NFS 3.0 -- used a PostScript interpreter as the file system API.

https://news.ycombinator.com/item?id=17061967

>Even 16 or so years earlier in 1990, Sun played with the idea of putting a PostScript interpreter in the SunOS kernel.

>Like NeWS was the Network extensible Window System, so NeFS was the Network extensible File System, or NFS 3.0.

>It was actually a great idea, just a wee bit before its time, and very poorly named and positioned!

>For example: If you want to make a copy of a file on the server, you can send a PostScript program that runs in the kernel and copies the file locally on the server in the kernel with ZERO context switches, instead of sending it over the net to the client, then back from the client to the server. Even if you rsh'ed the user command "cp" on the server, it would still incur context switching, but if your copy loop was running in the kernel then it didn't need to switch in and out and in and out for every block it copied.

https://news.ycombinator.com/item?id=17482780

>I disagree that history proves them wrong about Telescript. They just had the client/server relationship backwards (and asymmetrical).

>Andy Hertzfeld: "You didn't really need to do that. Just the remote procedure paradigm was good enough. Which is really what the web is based on. You don't inject code into the web to do you work for you and come back to you. You just ask a server with an http request."

>It doesn't take Yakov Smirnoff to observe that now the web injects code into you.

>The remote procedure paradigm isn't good enough. That just leads to the X-Windows Disaster.

NT's everything is an object is a much more rational design. OS X is based off similar concepts except in the kernel thanks to NextStep.

And what's an object, a somewhat opaque blob with some attributes? It all depends on where in the sand you decide to draw the line.

It has just a bit more structure than a file. An object has some data and some verbs or actions that can be performed on it.

The problem with files is that signaling has to happen in band or with hacks like ioctls. Or like in plan 9 through seperate files but with fewer guarantees on synchronization.

"Everything is a file" is a nice idea not taken far enough. The execution is full of frustrating omissions and holes at the edges, e.g.:

- unless you are using Plan 9 or gpm, reading mouse events is almost certainly going to be a lot more involved than opening /dev/mouse

- you can't portably use select() to wait on pthread conditions

- lots of useful things (like POSIX shared memory objects) are file descriptors, but aren't filesystem namespaced

Sometimes I wonder what the world would look like if Plan 9 had won.

That is easy, microservices über all.

I never had such appreciation for Plan 9, as I consider Inferno the best design of what they were trying to achieve.

The only caveat was that they never came around to add an ahead of time compilation support to DisVM.

Which was the last Unix release/flavor where you could still read(2) a directory? All manpages that I've found say that you get EISDIR if you try it.

I certainly remember being surprised when I accidentally cat'd a directory on AIX and it worked. I can't remember exactly what you get back; I think it's a NUL-separated list of the files in it.

That depends on the file system (and, possibly, the OS, but I think the OS should just forward the call to the relevant driver and return its result)

FreeBSD’s man page says (https://www.freebsd.org/cgi/man.cgi?sektion=2&query=read):

  [EISDIR] The file descriptor is associated with a directory
           residing on a file system that does not allow 
           regular read operations on directories (e.g. NFS)

NetBSD says something similar, adding ”The readdir() function should be used instead.”

I also am not convinced that, for example, the Linux man page (http://man7.org/linux/man-pages/man2/read.2.html) guarantees you get EISDIR when reading from a directory. It says that if you get EISDIR, fd refers to a directory, but isn’t explicit about the converse.

You are right. The default implementation in Linux returns -EISDIR, but filesystems are free to change it to something else:

https://github.com/torvalds/linux/blob/2a56bb596b2c1fb612f99...

https://github.com/torvalds/linux/search?p=1&q=generic_read_...

FreeBSD would allow this as of a few years ago (2007 at the earliest).

Still does. I'm running 12-CURRENT and it looks like UFS allows it. I occasionally cat or more a directory by accident and get a screen full of junk.

Imagine if unix was invented today and you would navigate a folder by reading it, and having a JSON encoded file of its content returned. Slow but so elegant and pretty.

If unix was invented today, you would have 3rd party files everywhere, executing randomly and typing the newest offers to your prompt automatically.

I know that SunOS 5.10 could read(2) a directory (that's what lore.cs.purdue.edu was running).

Wordpress strikes again.

Page is from 2014. Archive link: https://web.archive.org/web/20091119184152/https://ph7spot.c...

Edit: archive of 6 years ago with some styling: http://archive.is/B9He (The page actually loaded for me on the second try, and the design looks a lot more modern now.)

Your link says the page is from 2009, not 2014 (2009-11-19 to be exact).

Oh right, my bad. I just saw a first archive.org index at 2014 but that is of course not necessarily the time of writing/publication.

I like the Linus Torvalds' interpretation: that (almost) everything is actually a file descriptor. The few things that are not, like fork() returns, are subject to criticism (DJB wrote about how fork() should return an fd, I just couldn't find the URL right now.)

This? http://cr.yp.to/docs/selfpipe.html

> Richard Stevens's 1992 book ``Advanced programming in the UNIX environment'' says that you can't safely mix select() or poll() with SIGCHLD (or other signals). The SIGCHLD might go off while select() is starting, too early to interrupt it, too late to change its timeout.

Solution: the self-pipe trick. Maintain a pipe and select for readability on the pipe input. Inside the SIGCHLD handler, write a byte (non-blocking, just in case) to the pipe output. Done.

Of course, the Right Thing would be to have fork() return a file descriptor, not a process ID.

I have a faint recall of a DJB text with more in-depth discussion around the topic, thanks anyway for providing a true reference!

You're welcome! I found the link here: http://www.greenend.org.uk/rjk/tech/fork.html

Is the sendfile system call still the fastest way to transfer data locally on Linux systems?

For communicating between processes running locally on system. Or, for example, a server running locally on an android device. Unix domain sockets typically beat TCP connections for most large file transfers. But is there anything faster?

And what about transfers over the public net from machine to machine? I usually just use netcat, ssh, or scp.

Consider trying rsync instead of scp - I've found it to be faster for multiple small files.

Piping tar works much faster too. It also allows preserving metadata.

I love rsync, but being exceptionally fast is most certainly not one of it's properties.

Shared memory is faster than sockets

But unsafe.

That depends on what you're doing. Sendfile, splice and copy_file_range are the fastest if you want to shovel things back and forth between file descriptors. The latter supports reflink copy if the the underlying filesystem does.

And I think there's some special casing in vmsplice that lets you transfer memory pages from process to process over a pipe. At least I've seen some patches for that, not sure if they made it into the kernel. Alternatively there's memfd.

But if you want to coordinate processes you'll use a shared memory and futexes.

If you just want to hand over an entire file to a different process you can send the file descriptor itself over a unix socket.

>>> you can send the file descriptor itself over a unix socket

Oh, that is genius ;)

Described in detail here, from Stevens Unix Network Programming

http://www.masterraghu.com/subjects/np/introduction/unix_net...

And it looks like the server may have run out of files (well, descriptors) :-)

google cache link: https://webcache.googleusercontent.com/search?q=cache:L6aopS...

And if you use XML, all you need to do, is to add another forward slash after the filename. And now, the network, the host and the data have become one: Everything is a node (or an atomic value).

But hey, that's stupid. Let's not do that...!

By way of XPath. I don't disagree but--that's stupid because of... using XML at all? Complications of altering/documenting ad hoc taxonomies? Mixing paradigms under a unified syntax? Something else?

Would you mind giving me the one or other example? Or references to some documents about this (in regard to XML), that I could reach?

I think the author might have confused Plan9 and UNIX.

He also confused "The Art of UNIX Programming" with a "fantastic book".

What's wrong with the book ? I have read it and I think it provides a great introduction to Unix.

One obvious problem with the book which this article demonstrates is that it leaves the reader with the mistaken impression that "In Unix Everything Is a File".

I assumed this was common knowledge, but apparently not, so here goes: Another reason not to buy the book is that the author, Eric S Raymond, is a racist sexist Islamophobic conspiracy theory spreading global warming denying gun nut, who threatened to kill Bruce Perens.

In summary, and just because you asked, Eric S Raymond is a mediocre programmer who has made a career of trying to pose as a hacker, define what a hacker is, tell people how to become a hacker, all while viciously, personally, professionally, and unfairly attacking actual successful influential hackers like Richard Stallman, including his philosophy and his life's work, by hijacking and distorting the definition of Free Software and the very definitions of words in the hacker's dictionary, to reflect his own extremist narcissistic racist political ideology, which most certainly does not align with the non-bigoted ideology in the Hacker's Manifesto: "We exist without skin color, without nationality, without religious bias."

https://en.wikiquote.org/wiki/Eric_S._Raymond

"In the U.S., blacks are 12% of the population but commit 50% of violent crimes; can anyone honestly think this is unconnected to the fact that they average 15 points of IQ lower than the general population? That stupid people are more violent is a fact independent of skin color." -Eric S Raymond

"A clash of civilizations driven by the failure of Islamic/Arab culture (though I would stress the problem of the Islamic commandment to jihad more than he does). I think he [Steven den Beste] is also right to say that our long-term objective must be to break, crush and eventually destroy this culture, because we can't live on the same planet with people who both carry those memes and have access to weapons of mass destruction. They will hate us and seek to destroy us not for what we've done but for what we are." -Eric S Raymond

"And for any agents or proxy of the regime interested in asking me questions face to face, I’ve got some bullets slathered in pork fat to make you feel extra special welcome." -Eric S Raymond

"When I hear the words "social responsibility", I want to reach for my gun." -Eric S Raymond

As much as he thinks he and everyone else should own lots guns, Eric S Raymond has recently stated (2017) that he believes blacks are too unintelligent to own or to be trained to use guns:

https://twitter.com/tqbf/status/816449724127608833

>Unfortunately, this doesn't cover the BLM crowd, which would have an average IQ of about 85 if it's statistically representative of American blacks as a whole. I've never tried to train anyone that dim and wouldn't want to.

https://rationalwiki.org/wiki/Eric_S._Raymond

Eric S. Raymond (born 1957) is a computer programmer, advocate of open source software, and author or editor of several written works, including The Cathedral and the Bazaar and the Jargon File. Coincidentally, "Yonder Racism" is an anagram of his name.

Unfortunately, his work and expertise in computer technology[1][2] has been all but overshadowed by his batshit insane wingnut tendencies in the wake of 9/11, his misogynistic claims,[3] and his increasingly wanky ego-gazing,[4][5] as he acknowledges that people aren't so crazy about him anymore.

https://en.wikipedia.org/wiki/Talk:Russ_Nelson#Blacks_are_la...

Eric S Raymond threw down the gauntlet and defended Russ Nelson's infamous "Blacks are Lazy" article by calling everyone who wanted him to resign his position as president of OSI "fools and thugs", and demanded OSI spend their precious time and resources fighting critics of racism and defending Russ Nelson's racist beliefs which ESR shares, instead of promoting open source software.

http://www.eweek.com/servers/new-osi-president-steps-down

>“Abetting somebody elses witch hunt is no less disgusting than starting your own.” “Personally, I wanted to fight this on principle,” Raymond said. “Russ resigned the presidency rather than get OSI into that fight, and the board quite properly respected his wishes in the matter. That sacrifice makes me angrier at the fools and thugs who pulled him down.”

This would be funny if it weren't exactly like Trump's recent "'nt" flip-flop on Putin:

>Nelson subsequently sought to explain himself in an essay titled "Blacks are Not Lazy."

https://encyclopediadramatica.rs/Eric_S._Raymond

In 1999, Debian developer Bruce Perens published an "email threat" that he allegedly received from Raymond. Raymond then "clarified" that he only meant to defame Perens. From this we can assume that he is batshit insane and will fucking kill and or write to anyone that says anything about him or his software. If you are lucky you might get an O'Rielly book about you.

https://lists.debian.org/debian-user/1999/04/msg00623.html

    To: debian-user@lists.debian.org
    Subject: email threat
    From: bruce@perens.com
    Date: 5 Apr 1999 22:48:42 -0000
    Message-id: <19990405224842.2386.qmail@perens.com>

    Today I received the following threat in e-mail from Eric Raymond. The message
    was copied to the Silicon Valley Linux User's Group officers, who you may
    consult regarding its authenticity. The police have been notified.

    Because I know that Eric is a firearms enthusiast, for my own protection,
    I feel the best strategy is for me to publicize the threat widely.

            Thanks

            Bruce Perens

    > Damn straight I took it personally.  And if you ever again behave like
    > that kind of disruptive asshole in public, insult me, and jeopardize
    > the interests of our entire tribe, I'll take it just as personally --
    > and I will find a way to make you regret it.  Watch your step.

The irony of Eric S Raymond threatening someone else for behaving "like that kind of disruptive asshole in public" is rich -- very rich.

https://geekz.co.uk/lovesraymond/cat/bruce-perens/page/10

https://geekz.co.uk/lovesraymond/archive/terrorismistic

Your reasons are totally ad hominem and dictated by your political views and have nothing to do with the topic.

Sure they do. He asked about what was wrong with the book, which I answered. Besides the obvious factual error I pointed out (which you chose to ignore, but is literally the topic of this discussion, therefore relevant), there is the well known fact that Eric S Raymond is a racist with a long history of making ad hominem attacks against entire races and religions, as well as his personal death threat against Bruce Perens and long career of attacking Richard Stallman and the Free Software community.

I'm sorry that a single quote is just not capable of capturing the vast depth and breadth of his racism and hatred of Perens and Stallman.

Is there something you find inappropriate or annoying about me expressing my (apparently controversial) political view that racism is wrong, and quoting Eric Raymond's own words to prove he is a racist? Why is that more annoying to you than ESR's racism, threats and ad-hominem attacks themselves?

In the same vein (or bloody car seat), whenever anyone mentions the Reiser File System, the fact that he murdered his wife often comes up. Do you also find it inappropriate to mention that fact on Hacker News or Wikipedia too?

https://en.wikipedia.org/w/index.php?title=Comparison_of_fil...

https://en.wikipedia.org/wiki/ReiserFS

I don't see what Islam or irrational wiki or racism has to do with unix programming and the book. Your political beliefs contradicting with ones of the author does not make a programming book bad.

I was directly responding to a question from somebody who didn't know about ESR. Were you previously aware of the fact that Eric Raymond is a batshit crazy racist, or did you just learn something new from me that you didn't previously know? If so, then mission accomplished! I'm glad to have opened your eyes even a little bit.

Do my political beliefs about racism being bad and obligatory to call out contradict your own beliefs, and annoy you so much that you're compelled to argue and continue to drag this out? Or do you know some actual facts you can prove about ESR that contradict the proof I posted with links and quotes? Then by all means share your proof I'm wrong, instead of stifling the discussion! Why are you so irritated and motivated to suppress those widely known facts from being mentioned on Hacker News?

If simply stating the facts and quoting ESR's own words sounds to you like an ad hominem attack, then that's ESR's problem, and maybe he should have kept his mouth shut instead of publicly saying things like "The average IQ of the Haitian population is 67... Haiti is, quite literally, a country full of violent idiots", and he exercises enough personal responsibility that he doesn't need you to bravely step up and defend his racism.

http://esr.ibiblio.org/?p=7011

Or is your real issue with the valid criticism I raised about the book (and that you awkwardly ignored and haven't countered) about "In Unix Everything Is A File" being wrong, which is LITERALLY the topic of this conversation, a point many other people are discussing, and a major mistake of the book and the article?

If you really believe that my criticism of the book in response to Lordarminius's direct question has "nothing to do with the topic", then why aren't you criticizing Lordarminius for asking me "What's wrong with the book ?"

And why aren't you arguing with all the other people here who have pointed out that the API for reading directories is different than the API for reading files, that ioctl is a kitchen sink of non-file-like operations, that many objects in the Unix kernel are simply not files, etc?

Or is it just my second criticism of ESR's racism what ticks you off so much that you totally ignored and failed to address my first criticism of his book's factual errors, and incorrectly claimed my reasons for disliking the book are "totally ad hominem"? Go re-read the very first sentence of my reply.

I don't see what ESR recasting the Jargon file to reflect his own political beliefs has to do with hacking, but that's what he did, and it's also valid to raise that point when discussing the flaws of other books he wrote.

https://news.slashdot.org/story/03/06/08/1534249/esr-recasts...

http://www.ntk.net/2003/06/06/

Given that they also reference Plan 9 in the article, I doubt it.

Are there Unixes where TCP / UDP ports are files? setcap on Linux gets really old when you just want to give a user access to a porf.

I can't say for sure, but i do know ucspi-tcp is a great tool that wraps the noise for you into FDs. But it's only useful if you can tank the cost of invoking a process per tcp connection.

https://cr.yp.to/ucspi-tcp.html

this needs a 2009 tag.

it should actually be removed as the article is wrong. even in 2009, how could someone get this wrong? pretty sure 2009 was finally going to be the year of the linux desktop! IOW unix had reached the masses by then.

Reply on news.ycombinator.com