Comment by 1_player | Hacker News Reader

Comment by 1_player | Hacker News Reader

Parent Full thread

Not sure I agree with you there: should it not fork a process to run a heavy task? Should it not fork a process for each end of a pipe, especially considering we're deep into the multicore era of processors?

The fact that the basic shell could use more builtins instead of delegating everything to external processes (like calculating a simple expression) is totally unrelated to the Unix Philosophy being bankrupt.

If fork and processes weren't so expensive, I still think multiple processes are the best way to parallelise on multicore CPUs. In fact, there's a resurgence in programming environments based on multiple communicating, yet isolated, processes, see Erlang/Elixir.

Though I definitely would like to see someone seriously exploring a different paradigm than UNIX.

EDIT: Unix philosophy also is "everything is a file", and what is a file exactly? A bag of bytes. Everything we deal with is a bag of bytes if you think about it, so I find it perfectly fitting to model our world that way.

But then again, the current construct to represent this idea such as labels organised in directories, might be a bit long in the tooth and we're due for some paradigm shift on this front as well, but I digress.

"Everything is a file". Except ioctl. Except mmap. Except the shell makes it a huge pain in the ass to hook up arbitrary graphs of streams between processes.

EDIT: I'm too old for this. Think what you want about mmap and ioctl.

My point is that ioctl and mmap disprove the "Unix Philosophy" idea that files are one-dimensional streams of characters that are all you need to plug programs together.

Do they though?

Everything as a file is an abstraction, an enormously useful abstraction. The point isn't that it has to behave like a file in every way. At the C level, the fact that open(), read(), write(), or e/poll() behave the similarly is extremely helpful.

I can have a whole suite of functions that do not care what type of file descriptor they hold, but just that those functions behave in a predictable way. Maybe only 10% of my code has to care about the differences between them, instead of 70%.

Except sockets, Sys V IPC, GPGPUs, better then?

BSD Sockets were never part of "UNIX Philosophy", they are an ugly quick'n'dirty port/rewrite of code from totally alien system (TOPS-20) because DoD had to deal with vendor abandonment

And in the 35 years since, despite multiple new kernels and userlands and languages, a more unix-file-like socket API hasn't become popular. I'm not sure what does that tells us, but it's not nothing.

What new kernels? The three major OSes are two UNIX based, and one doing its own thing.

I don't think it's a great investment in time redesigning the whole socket API since you need to keep the old one around, unless you want to lose 35 years of source code compatibility.

The BSD socket API can definitely and easily be improved and redesigned, if only there was some new players in the field that wanted to drop POSIX and C compatibility and try something new.

> What new kernels? The three major OSes are two UNIX based, and one doing its own thing.

I meant that many new and forked UNIX-like kernels have been written over the past 35 years. Just the successful ones (for their time) include at least three BSDs, OSX, several commercial UNIXes (AIX, IRIX, Solaris, UnixWare...), and many others I'm unfamiliar with (e.g. embedded ones).

It's common to add new and better APIs while keeping old ones for compatibility. Sockets are a userland API, so if everyone had adopted a different one 20 years ago, the original API could probably be implemented in a userland shim with only a small performance hit.

However, a new file-based paradigm from sockets would probably work better with kernel involvement; that's why I mentioned kernels. We've seen many experiments in pretty much every other important API straddling kernel and userland. IPC, device management, filesystems, async/concurrent IO, you name it. Some succeeded, others failed. Why are there no widely successful, file-based alternatives to BSD sockets? The only one I know firsthand is /dev/tcp and that's a Bash internal.

Interestingly enough, Go's network functions, by virtue of being native to Plan9, actually are based around Plan9's file-based network API. It works pretty nicely, though "everything is a file stream" has its issues.

Government, Politics, resistance to change, NIH syndrome, "us vs them" and a bunch of other issues all conspired to keep BSD Sockets alive.

The first is the origin of BSD Sockets at all - UCB got paid by the government to port TCP/IP to Unix and, AFAIK, provide it royalty-free to everyone, because DoD needed a replacement for TOPS-20 as fast as possible and widely available, and there were tons of new Unix machines on anything that could get paging up (and some that couldn't).

Then you have the part where TLI/XTI ends up in Unix Wars, associated with slow implementations (STREAMS), despite being superior interface. NIH syndrome also struck in IETF which left us with the issues in IPv6 development and defensiveness against better interfaces than BSD Sockets because those tended to be associated with the "evil enemy OSI" (a polish joke gets lost there, due to "osi" easily being turned into "axis").

Finally, you have a slapped together port of some features from XTI that forms "getaddrinfo", but doesn't get much use for years after introduction (1998) so when you learn programming BSD Sockets years later you still need to manually handle IPv4/IPv6 because no one is passing knowledge around.

Except the part that BSD is one of the surviving UNIXes.

Which should drive home the point that "everything is a file" does not accurately describe the reality of the Unix "philosophy".

If none of the Unix shells are remotely compliant with the "Unix Philosophy", then what is the "Unix Philosophy" other than inaccurate after the fact rationalization and bullshit?

Name one shell that's "Unix Philosophy Compliant". To the extent that they do comply to the "Unix Philosophy" by omitting crucial features like floating point and practical string manipulation, they actually SUFFER and INTRODUCE unnecessary complexity and enormous overhead.

> "If none of the Unix shells are remotely compliant with the "Unix Philosophy", then what is the "Unix Philosophy" other than inaccurate after the fact rationalization and bullshit?"

Precisely that! It's post-hoc rationalization and bullshit, that's the point I was driving at.

Nothing is ever "pure". Linux isn't pure, ultimate Unix philosophy.

ioctl, mmap, sockets could be represented as files, why not?

`echo 1 > /root/some-file/%lock` instead of fcntl(F_SETLK) -- I just made up this syntax

`echo SIGTERM > /proc/15683/signal` instead of kill

Not sure about mmap, since its very purpose it to map a file to memory.

>ioctl, mmap, sockets could be represented as files, why not?

The fact that they're not may be a clue to answering that question.

So you propose making a "floating point multiply server", even though the CPU can do that in one instruction? Talk about microservices!

Au contraire, I'm saying the shell could turn your multiplication command into a single instruction, no need to call an external process.

But at some point you need to delegate to other processes. You can't have /bin/bash embed your Chrome browser in it.

But that's not even Unix philosophy. That's how DOS used to work as well.

I'm talking about the shell that actually exists, not some hypothetical shell that doesn't exist and never will. You can't defend the "Unix Philosophy" by proposing a non-standard MONOLITHIC shell be universally adopted that totally goes against what the Unix Philosophy explicitly states: "Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".

At least Apple figured out that floating point was useful early enough to replace INTEGER BASIC with APPLESOFT.

Forking "bc" from "bash" is the canonical way to multiply two floating point numbers in Unix, according to the "Unix Philosophy".

https://stackoverflow.com/questions/12722095/how-do-i-use-fl...

>You can't. bash only does integers; you must delegate to a tool such as bc.

The shell is meant to pipe programs together, not perform computations itself. Computations are meant to be made by programs, not shell scripts.

Use the right tool for the job, do not complain that you screwdriver sucks because you cannot cut wood with it.

You too are blinded by the limitations of your tools. Of course the shell performs all kinds of computations. Just not a useful enough set of them that you don't still need "bc" and "awk" and "sed" and "grep" and many other ridiculous "Unix Philosophy Compliant" tools.

I complain my screwdriver sucks when it's made of plastic, and it's only good for driving plastic screws into styrofoam.

The idea is that the tools are meant to be limited to avoid complexity (this is why bash, like most GNU tools, isn't really a good example of unix philosophy - it is too complex) and instead rely on composition.

As if the bash syntax for simple string manipulation and piping and quoting and escaping and even determining if a string is empty wasn't unnecessarily complex.

As I said, the "Unix Philosophy" of plugging together artificially limited tools by pipes is intellectually bankrupt (and terribly inefficient). That was my whole point.

It is inefficient, but that doesn't matter much for interactive use. After a shell script reaches a dozen lines, it's time to port it to Python. If the job needs absolute performance you rewrite in a performant language instead.

I'm not sure why you are upset to be honest? What design would you prefer? This one seems to hit all use cases.

In another thread I mention that fish has math and string manipulation builtins.

If you browse through GNU documents, you sometimes come across strange ancient passages like

> "Avoid arbitrary limits on the length or number of any data structure, including file names, lines, files, and symbols, by allocating all data structures dynamically. In most Unix utilities, “long lines are silently truncated”. This is not acceptable in a GNU utility. "

What do you suppose the ancients meant by this? Some have taken it as evidence that in ancient times before GNU, Unix tools were not in fact "doing one thing and doing it well" as the Unix priests so often claim. This of course is blasphemy and the guilty are punished accordingly... but are they wrong?

> shell ... Just not a useful enough set

    chsh - change login shell

    DESCRIPTION
      The chsh command changes the user login shell.

If you don't like the features of your shell, use a different shell.

> you don't still need ... "awk" and "sed" and "grep"

I don't need those programs - there are many other ways to solve most problems. I want to use them because they are both fast to start and execute, and make development time extremely fast.

> You too are blinded by the limitations of your tools.

You appear to be trying to use the shell for tasks that that should be done in a different language. You also don't seem to understand that some of use use the shell because discovered it was a much faster and easier to use tool for some problems than the other methods we've tried. For some tasks, complaining about the overhead to do some things like floating point or trivialities like the <100ms it takes to fork and run bc is a premature optimization.

If you don't find shell to be a useful tool for the problems you need to solve, that's fine; use whatever tool works best for you. We might be trying to solve different problems. Just because you don't like a particular tool doesn't mean it's "ridiculous".

What all those "ridiculous" tools allow, among other things, is:

- Cheap parallelism. Things as easy as `grep | cut | sort` will utilize multiple cores despite shell and all the tools it spawns being single-threaded. I can't imagine threading primitives in some supercharged version of bash without some ugly syntax and a number of additional questions about deadlocks on StackOverflow. That would not be bash, but some ugly Python parody instead. (Also note that even PowerShell seems to limit itself to job control primitives despite being more capable .NET-backed language: [1].) That usually saves me much more than ability of zsh to multiply floating point numbers without forking.

- (Inter)changeability. Want to multiply complex numbers? Use another calc. Want ridiculous regexes? pcregrep. Want to scan large hard disk efficiently? ffcnt [2]. Want to do something else, and to do it efficiently? Write a program without spending a time to learn your shell's C API [3], just read stdin or parse argv. Yeah, I too wish to not spend much time serializing and deserializing data in each and every pipe, but that would require not a new shell (we already have PowerShell) but, I guess, an entirely new OS as well as ecosystem designed just for that OS, which nobody would spend much time adapting to at this time.

[1] https://stackoverflow.com/questions/3325911/how-does-threadi...

[2] https://github.com/the8472/ffcnt

[3] Really, who got time to spend on something like https://docs.python.org/2/c-api/index.html (I know, Cython exists, but there's no Cython for every language out there.)

(edit: formatting; replaced platter-walk (which is a library) with ffcnt (which uses said library))

Plenty of Unix shells support floating point. For example, the conveniently-named ksh93 was released in 1993, and was / is available on a wide range of systems. I’d guess the hipster-compliant fish shell can do these things too, but can’t be bothered to RTFM.

I think the takeaway is that floating point shell use cases are so vanishingly rare that no one bothers to switch shells over it.

https://unix.stackexchange.com/questions/40786/how-to-do-int...

Reply on news.ycombinator.com