Hacker News Reader

Comment by wlib | Hacker News Reader

Why did we keep the language of the shell and the OS separate? It seems like a needless abstraction which creates more harm than good (read a shell script vs any other language). While I'm at it, why is the filesystem and syscall api not just part of a standard userland language? For example, the filesystem could be exposed like an object tree rather than some syscall ritual. The syscalls could just be invisible, where the language compiler deals with it instead of the programmer. I think that the old LISP machines got this right while we are stuck in a usless POSIX compatibility trap. The only reason I think they didn't design unix this way was because C was too low level, but we could write the OS in a "higher level" functional language.

View on HN

(re: Bash-vs-OS integration)

bash is a programming language like any other, and you could use anything with a REPL as your shell. Python should do. In fact, I'll try it right now..

Yes, it works. Just sudo chsh -s /usr/bin/python <username> and off you go.

Once you start doing this for a bit, you'll notice that the Python REPL is an incredibly poor UI for repeated execution of subprocesses. It is very elaborate. Having to constantly wrap your strings in quotes, calling subprocess modules, exec, painstaking handing of streams, etc.

Then you start looking for a language that has better syntax for calling external programs.. hmm...

Bash. Or zsh, or ksh, etc. These languages excel at it. But that's all they are: programming languages that happen to be super easy to use when it comes to starting external programs.

This is why it makes little sense to bind them to the OS. As far as the OS is concerned: there is no Bash. Just like there is no Python. There is just syscalls.

You may be interested in xonsh, a shell that supports both python and bash-like expressions: https://xon.sh/

is this like fish?

It's pretty much like fish, but with Python goodness.

Thank you for this

> Python REPL is an incredibly poor UI for repeated execution of subprocesses

Python REPL, even with recent additions of TAB completion, is a poor REPL, period.

IPython, on the other hand, offers a much better programming environment than shell while still allowing easy access to most of the things you mention. Example:

    In [1]: from pathlib import Path
    In [2]: file = Path("~/.sbclrc").expanduser().read_text()
    In [3]: !echo "$file" | awk '!/^;;/'   # EDIT: to clarify, this shows passing
                                           # Python variable to a shell pipeline.

    #-quicklisp
    (let ((quicklisp-init (merge-pathnames quicklisp/setup.lisp
                                           (user-homedir-pathname))))
      (when (probe-file quicklisp-init)
        (load quicklisp-init)))

All things considered, between !, %, %% and /, IPython is a decent shell. I was using it for a few years as my main shell, actually - its QT Console implementation specifically. I was working on Windows back then, PowerShell didn't exist yet, and `cmd.exe` was... well, exactly the same as today.

TLDR: a shell is just a REPL with a few conveniences for specific tasks. Re-implement these and suddenly any REPL becomes a workable shell.

> In [3]: !echo "$file" | awk '!/^;;/'

That's Bash (or at least sh). Which excels at piping the output of programs into other programs, and pay that cost for everything else.

Yep. IPython lets you execute bash.

Yeah, that's what I mean by "easy access" - you're working with Python, but if you want to do something that Python is not well-suited for[1] - like easily piping something through many programs - you can do it simply by adding `!` at the beginning of your pipeline, right there in the REPL.

It's actually the same philosophy that shells use: in bash you frequently invoke sed, awk, grep, etc. because it's easier than writing the same directly in the shell. In IPython, you invoke bash to do piping, because it's easier than connecting a host of subprocesses' stdins/stdouts in Python.

So again, there are some things you expect from your shell, and as long as you get those things (no matter how), you can use any REPL as a shell without problems.

[1] because of syntax or lack of library or something like that

Piping program output is just function composition. The downside is that it works over expensive system processes, and data must be serialized. It's a fragile system where instead of using built-in functions, we use "standard" executables.

I think, rather, function composition is just piping program output.

That was kind of the idea of Unix. C was just a nicer from Assembly. You might write a performance-sensitive or low-level routine in C, much like you might drop to C when writing a Python library. But the high-level language of the system was the shell. The `dc` executable wasn't just meant to be a user-facing calculator program, it was also meant to be the system's "bignum" library. That was big-picture Unix. The syscalls and C were little-picture Unix.

This 2013 thread shaped a lot of the way I think about Unix: https://news.ycombinator.com/item?id=6530180

Good insight, thank you

On the same line, let me recommend you a book: "The Design of the Unix Operating System," by Maurice J. Bach. If I am not mistaken, Mr. Bach was a member at Bell Labs.

My personal take on the book is that the kernel was meant to be a portable virtual machine, extensible through processes, and that those would be the building blocks of user applications for which the shell would act as glue.

In other words, the shell and the OS are separate because most of what we commonly call "the OS" may be interpreted as mere encapsulation of less versatile hardware architectures.

This makes me think of PowerShell. Individual Cmdlets can be written in PowerShell itself or in any language that compiles to .Net IL, plus one has dire t access to the entire .Net framework.

PowerShell takes it a bit further: There is no "magic" built-in commands needed for e.g. manipulating current directory, environment variables etc. In PowerShell the cmdlets execute in-process and may have access to the session and it's state. Indeed all of the built-in cmdlets are just cmdlets defined not by the shell, but by the core modules

If you're doing it right, you are solving very different problems with shell vs any other language. Shell is best used as a tool for orchestrating other programs, you should not be implementing your programs in shell.

Syscalls, in general, are used in lieu of objects or other abstractions because they more accurate mirror what the underlying hardware is doing. This isn't always the case, some syscalls are maintained for POSIX-compatibility and add a lot of complexity to emulate behavior that is no longer reflective of the hardware.

At the end of the day, you'll find that it's very difficult to maintain the highest levels of performance while also presenting an API that has a high level of abstraction. Things like dynamically-resizable lists, using hash tables for everything, runtime metaprogramming, and other such niceties of modern HLLs aren't free from a performance perspective.

If you really want to know more, I would suggest reading one of McKusick books on operating system design (the most recent being The Design and Implementation of the FreeBSD Operating System 2/e, but even the older ones are still largely relevant).

Maintaining this "useless POSIX compatibility trap" has a certain amount of utility; I for one like not having to re-write all of my programs every few years. I imagine others feel the same.

In closing, some projects that are pushing the boundaries of OS design which you may want to check out include:

* Redox OS (https://www.redox-os.org/) - a UNIX-like OS done from scratch in Rust

* OpenBSD (https://www.openbsd.org/) - one of the old-school Unices, written in plain old C, but with some modern security tricks up it's sleeves

* Helen OS (http://www.helenos.org/) - a new microkernel OS written from scratch in C++, Helen OS is not UNIX-like

* DragonFlyBSD (https://www.dragonflybsd.org/) - a FreeBSD fork focused on file systems research

* Haiku (https://www.haiku-os.org/) - binary and source compatible with BeOS, mostly written in C++, but also has a POSIX compatibility layer

It's odd not including Plan 9 here. But I guess that is the fate of Plan 9.

Redox reuses a lot of concepts from Plan 9

It has already pushed the boundaries, and now seems not to be developed or used that actively.

Yeah, I did not include anything without at least some recent development. AROS and plan9 both got cut for that reason.

I was on the fence about including ReactOS but wound up not including that either.

I would consider 9front quite active from their Hg repo.

https://code.9front.org/hg/plan9front

When I first learned one could write programs in Bash, I got very excited. It didn't take long for that to where off :) My Bash got a lot better though!

I think everyone who become proficient in shell tries to implement something large in it, gets burned, and realizes that it’s a bad idea. I certainly had that experience.

> you should not be implementing your programs in shell.

What should one be using instead in the current scenario?

Python, Rust, Golang?

I think parent means (generally speaking) not to implement larger programs in shell.

If the tasks performed by your program can be expressed maintainably and performantly in shell, go for it.

For programs one size up, something like Perl or Ruby or any other language that allows easy execution of shell commands can be a great solution.

Of course, this all depends on any number of variables. Team competencies, target environments, etc. And with some effort even larger bash scripts can be kept maintainable-ish, and in some target environments (can't always count on other languages being available!) shell scripts might be the only available programming environment.

It depends. I don’t think there is a language which is categorically the best. The solution to a given problem may be harder or easier to express in different languages. Some problems are better solved in C, some in Python, ..., and some in Shell.

It's not quite the same thing, but I've been enjoying using eshell (https://www.gnu.org/software/emacs/manual/html_mono/eshell.h...). It has the usual shell interaction, except you can also interleave it with arbitrary elisp. It's not quite the OS, but it allows interaction with any parts of Emacs, which is at least OS-like (and can itself interact with the OS in various ways).

I think you might like TempleOS.

You might want to look at the oilshell project [1]. It's an attempt to bring some sanity to the shell while keeping the ability to run existing scripts.

[1] http://www.oilshell.org/blog/2018/01/28.html

> stuck in a usless POSIX compatibility trap

Exactly. Unfortunately.

Unless we have a solution that is significantly better than POSIX/UNIX, switching to anything else incurs a significant cost that no one is willing to pay.

Short of that, we probably need a technology breakthrough that brings in a complete architectural change.

> Unless we have a solution that is significantly better than POSIX/UNIX, switching to anything else incurs a significant cost that no one is willing to pay.

The problem is that it doesn't even suffice for the new thing to be "significantly better". Because of the huge sunk cost, the new thing needs to be able to do desirable things that a POSIX-compatible OS strictly cannot do. Otherwise, it will always be easier and faster to just glue another subsystem onto the Linux kernel and continue using that.

If you think of the browser as an OS, which it almost is these days, JavaScript is that language.

> The only reason I think they didn't design unix this way was because C was too low level, but we could write the OS in a "higher level" functional language.

You've already lost my interest. Bash/Shell is incredibly powerful and doesn't need a higher-level abstraction. That is what programming languages and CLI tools are for.

That misses my poorly written point. Do you use a separate, awkward tool to call functions you wrote in python? Or do you call the function in python itself? The shell is a useless abstraction that was only necessary in unix because the alternative was c. Now that we have better languages, why not use them to write the OS bottom-up?

Because writing new OSes is hard and writing new OSes where existing software that people want to use keeps working is even harder.

And thus the tech industry piled abstraction upon abstraction, decade after decade, until finally all software collapsed into a singularity and destroyed the earth. Which, frankly, came as somewhat of a relief to all the people forced to use it.

It's strange to me to imagine that there would be one language that was the best choice for OS implementation and day to day user interaction/automation. I'm curious what language do you think would be good for both use cases?

A more syntactically friendly Haskell-like language

How would you deal with mutable state like, say, the current directory?

When all you have is a Haskell everything starts to look like a monad

I think that if we use purish FP as our system language it doesn't make sense to just emulate imperative/mutable system.

But mm... why you would want to mutable state like current directory in first place? Or even mutable filesystem? Why aren't those just immutable parameters of your program?

> But mm... why you would want to mutable state like current directory in first place? Or even mutable filesystem? Why aren't those just immutable parameters of your program?

Because that's how just about any OS, application, library, filesystem, device driver, database, nearly everything related to computing at all, works. Reinventing the universe has never, ever lead to success. If you want even the slightest hope for non-negligible adoption of your operating system, you need to be able to interface with the rest of the world. And that's why you don't want the filesystem to be an "immuatble parameter of your program".

You can still build compatibility layer on top of immutable FS. Like being mutable from program(s) perspective while being able to control how these changes are capsulated from rest of the system. You could, for example, see what your

> any OS, application, library, filesystem, device driver, database, nearly everything related to computing at all

-is trying to actually do to your files and operation like reverting changes is trivial. Sure it's not performance optimal and effects what kind of software is ideal to work within such a system. It's just a different approach with different trade-offs.

Anyway if I one day want to build new OS it's for trying different interesting approaches and not yet another 'successful any-OS'.

RIP Terry Davis

Because one of the scenarios we were talking about is interactive use - i.e. command prompt. Current directory (and other implicit state like that) is convenient there, and removing it would make it that much harder to use.

If your workflow is like: 1. Manipulate current directory 2. Run some program 3. Manipulate current directory again 4. Run yet another program...

You can actually consider current directory as immutable already.

It became "badly mutable" for example if you manipulate the current directory and this change is observable from a program point of view which is already running. Would that be really useful?

To clarify I am not trying to say that absolutely everything should be immutable but it's fun thought experiment how far you can go.

I don't see how. The suggestion was to use a Haskell-like scripting language for interactive use on the command prompt. Note, we're not talking about apps that run from that prompt; we're talking about the shell itself. And all shells today have mutable global state in form of current directory (and pushd/popd stack, usually).

Shell is kind of REPL (read–eval–print loop). In OOP you use looping with mutation and in FP you use recursion. This may not sound very informative but still: "Just fold over inputs, what's the problem?".

Being immutable doesn't mean that everything is static but that things aren't changeable in place. So instead of having a global mutable state, you have this local state which you can alter by creating new instances and passing them around. In the end, the difference is that on what scope changes are observable and how that constraint effects on your design. In some sense: mutable = changes are uncontrollable, immutable = changes are controllable.

So a ml like language like sml, ocaml or f#?

That’s great for developers who are competent in that paradigm but what about hobbyists or sysadmins who have little interest in software development?

I’m not saying POSIX shells are without fault but they weren’t just created because C is too low level; they were created because people used a terminal who weren’t always techies so Bell Labs created a language which anyone could easily write simple programs with. Granted that need has dissipated a little but your solution of having one language to rule them all creates more problems than it solves (really the only problem it solves is OCD for a few LISP enthusiasts).

The reason there are so many different programming languages isn’t just an element of NIH; some languages do genuinely handle some use cases better than some other languages. Plus there is always going to be an element of user preference. So your idea of a perfect system would be hell for a great many other people - myself included (and I do generally enjoy functional programming).

Yeah I never understood why many people, at least among programmers, seem to be strongly opposed towards shell and shell scripts.

At some point when I wasn't proficient enough in Bash yet, I was writing scripts for automation using Perl and Ruby. The 'logic part' is definitely much easier in these languages. But simple file/directory stuff is far more complex actually. A lot of this has to do with error handling and different expectations how that is supposed to work. In a default shell script, errors are handled very forgivingly which is very much how most people work when performing tasks manually - not every step is super important.

On the other hand Shell scripting documentation is crap. Most of it is from 80s/90s, full of irrelevant details for the practical person. A bit like 90s/00s JS documentation before MDN.

Haskell's lazy-by-default evaluation makes it more difficult to reason about memory usage, because it is all too easy to accidentally build up large expression trees at runtime [1]. This makes it a rather bad choice for a kernel or other low-level stuff.

Compare Rust, which tries to keep allocations (and the size/asymptotic complexity of allocations) explicit and visible.

[1] The optimizer catches some cases, but not all of them.

The choice of python as your shell would surely be as arbitrary as bash?

There is a long history of alternative shells csh, tcsh, zsh, fish, etc. All with various level of abstraction and various amounts of "programming language type constructs" (for lack of a better term).

At the end of the day, long arguments have been had over which is the better shell. It's all just personal preference. Hence it can be set in /etc/passwd per user.

You prefer python? chsh is there to change it.

My point wasn't about python, just an example of how a shell around python, made only for python function calls, would be a useless abstraction.

Something like IPython, amirite? XD

I have a different theory; perhaps line printers (which terminals try to emulate) are to blame.

Space on a line is limited and one directional, so thr tools naturally evolved towards strange single line incantations instead of full fledged programs.

Commodore Pet did it right by introducing a navigable terminal where you could move between lines and don't need to open an editor to write multiline programs.

I don't have time to Google now but I think that mainframe terminals worked like that. 3270 should be the model to search for.

This was already a solved problem in the 70s. IBM's 3270 series offered full buffered access to a character matrix display, and DEC's VT52 and later terminals added bi-directional scrolling.

Modern CLIs can usually emulate at least a few of the old scrolling character matrix displays, but only a few bash commands (e.g. top) use the extra features.

> Do you use a separate, awkward tool to call functions you wrote in python?

I use a separate tool to call a lot of functions written in C. That tool is called “Python”. Or “Ruby”.

Sometimes I use a separate tool to call functions written in Python (or JavaScript, or...) and that tool is called PostgreSQL.

Sometimes I use Ruby to call C to call Postgres to call JS.

> Now that we have better languages, why not use them to write the OS bottom-up?

What better languages for implementing an OS? Lisp? We had that before C. Rust? Either way—and I like both languages—I don't want to use either for a shell. Red? Maybe, but I don't think we're to the point of a Red OS yet, and while it's probably eventually usable for that purpose, I don't see it as necessarily ideal for OS implementation though it might be tolerable as a shell language.

Reply on news.ycombinator.com