Hacker News Reader

Comment by snazz | Hacker News Reader

This is exceedingly cool. It could be a modern-day Lisp Machine that takes advantage of the Linux kernel's great hardware support. Since Python is such a dynamic language, I'd be interested in how they plan to allow the Python processes to communicate. This would be a great opportunity to avoid the classic Unix mistake of using text files as the lowest common denominator everywhere; a structured data format like S-expressions could be passed around instead.

Edit: Other languages I'm thinking of for this idea:

* Lua or LuaJIT (even more dynamic and really great performance; smaller and somewhat fragmented ecosystem)

* Scheme (potentially also fast and good S-expression support, also very dynamic; small and very fragmented ecosystem)

View on HN

> This would be a great opportunity to avoid the classic Unix mistake of using text files as the lowest common denominator everywhere

There are a great many who do not consider this to be a mistake. The fact that after half a century, Unix-like operating systems and programming environments are de rigueur is plain enough evidence of this. I can't think of much else in the whole field of computing that has had this kind of staying power over such a long period of time, excepting perhaps the idea of binary digital computers themselves.

Fun history note: Part of the reason the Unix userspace and development chain lean so heavily on text processing is because its inventors were tasked with designing a system for more powerful tools for the writing and production of technical documentation. But they were dyed-in-the-wool systems and programming nerds so stand-alone documentation tools were not very interesting. So their solution was to invent a new OS, borrowing the features they liked from other OSes they had been exposed to, under the guise of delivering a documentation production system. If it turned out that this Unix thing they came up with happened to be an excellent programming environment as well, that was just a nice bonus as far as management was to be informed.

One of the greatest hacks of computing history, if you ask me.

Edit to add: As others have pointed out, you are hardly restricted to piping around text. And if you do, the power of text is that you can impose arbitrary structure on top of it, e.g. tabular data, shell scripts, XML, YAML, JSON, etc. Saying it's too bad Unix abstracts everything as text files is like saying it's too bad microprocessors abstract everything down to either a one or a zero when far more interesting number exist.

> The fact that after half a century, Unix-like operating systems and programming environments are de rigueur is plain enough evidence of this.

Unfortunately Unix blew a 28-3 lead against Netscape Navigator and now 99% of people spend their time in web browsers instead.

I think this speaks to the weakness of the experience using Unix, not its strength.

> Unfortunately Unix blew a 28-3 lead against Netscape Navigator and now 99% of people spend their time in web browsers instead.

There might be exceptions, but people generally don't develop software in their browser.

Thanks for the serious response to a somewhat snarky comment.

I see software development as only one among the many things a computing system should enable. It's definitely an important one. But computers should enable creativity and power for all kinds of tasks, not just programming.

It's sad to me that people in all these different fields besides ours get stuck with webapps. I understand why they use them, they're very convenient. But it would have been great if we could have provided them an even better OS/desktop environment so that they never would have had to switch to such an unempowering tool.

Additionally, programmers being one of the last holdouts of actually engaging with Unix-as-a-way-of-doing-things is a precarious place to be. Microsoft/GitHub is on full "embrace" mode again, and it looks like they're going to make another run at slurping everyone into their ecosystem. Same with tools like Repl.it. It seems to me that the proportion of people who are engaged with the Unix way of doing things (piping text around, etc.) is shrinking, not growing.

EDIT: To get myself back on topic, for these reasons I see people experimenting with new (and crucially local) environments as really exciting. Traditional Unix environments don't need to be the last word, we can do better.

Apple went down a very interesting path during the interregnum years from 1985 through 1996. It was a beacon of creativity where some of its engineers worked on projects that offered new visions of personal computing that empowered the user without requiring them to be full-fledged programmers. HyperCard and AppleScript are examples of such projects, and the Knowledge Navigator, while not an actual product, was one of Apple's visions. I also find OpenDoc to be quite intriguing; OpenDoc would have introduced component-based GUI software to the mass public. Users could perform tasks by mixing-and-matching components and applying them to a document.

It would have been really cool to have seen Copland and Gershwin in its completed forms, but sadly as creative Apple was in the 1990s, its management was a mess until Steve Jobs returned, and so Copland never saw the light of day other than a few developer releases.

I am grateful for the work that Steve Jobs did at NeXT and for his amazing work returning to Apple, saving it from possible bankruptcy, and making it the juggernaut that it is today. I've long been a fan of Mac OS X. However, parts of me miss the Apple of the 1980s and 1990s, where the engineers and researchers there explored ideas of how to make personal computing better. I feel that the personal computing experience for desktops have been stuck in a rut for nearly 20 years, with some aspects actually degrading rather than improving.

But projects such as Snakeware give me hope that we'll see desktop computing move forward. I'm actually working on a side project where I'm creating a Common Lisp-based desktop environment inspired by Lisp machines and OpenDoc that can run on Linux and the BSDs, but I'm in the design phase and I'm far from writing any code (I've been studying up on graphics programming since part of this project involves writing a new GUI toolkit). There is still a lot of room for innovation on the desktop, and as you said, Unix isn't the last word, even for programming.

> However, parts of me miss the Apple of the 1980s and 1990s, where the engineers and researchers there explored ideas of how to make personal computing better. I feel that the personal computing experience for desktops have been stuck in a rut for nearly 20 years, with some aspects actually degrading rather than improving.

I totally agree with this, and think it's an incredibly important issue.

If you post anything public about your Lisp-based DE please feel free to let me know (email in profile).

And given the hegemony of Windows desktops and game consoles, plenty also don't develop software on UNIX.

Piping data from one command to the next is one of those things that I love showing to new users and it gets them all excited, only to then have to disappoint them when they think of a cool idea because they'd need to learn regex, sed and awk first to do anything non-trivial.

It's a shame Powershell is a disaster, because having a structured-by-default pipeline would've really bridged the gap between power users and programmers.

Disaster? It feel quite good over here.

> It's a shame Powershell is a disaster

Yeah, lets get back to bash forever :S

Bash is a good shell with a fairly unimpressive pipeline. Powershell is bad shell with a good pipeline.

For me, as a programmer, the shell part of a shell is more important, as I can do the rest in a more powerful language. Powershell is too verbose for shell usage (see New-Item), brings nothing new to the table in terms of scripting and is ultimately not worth using just for the better pipeline.

(another commenter pointed me to Nushell, which seems to be a very good middle ground, and I've heard of a few more similar efforts, so we hopefully won't be stuck with just bash for too long)

Oh, here we go again with verbosity like aliases are non existent. For realz, New-Item is ni for example or mkdir or whatever...

> as I can do the rest in a more powerful language

Powershell is like repl for dotNet. You can't really get more powerfull.

> brings nothing new to the table in terms of scripting

Are you serious ? I guess somebody has a lot of learning to do.

> (another commenter pointed me to Nushell, which seems to be a very good middle ground, and I've heard of a few more similar efforts, so we hopefully won't be stuck with just bash for too long)

We were on that road number of times before...

> Oh, here we go again with verbosity like aliases are non existent

Aliases would have to be standardised (and I don't mean the crap they pulled with aliasing wget) and regardless, only provide an illusion of simplicity that breaks down the moment you try to do anything more advanced.

> Powershell is like repl for dotNet.

The only place dotNet actually matters on Windows. Everywhere else it's just unnecessary bloat.

> You can't really get more powerfull.

I mean, yeah, technically anything Turing-complete is as powerful as it gets. But things like generators, coroutines, easy multi{thread,process}ing, good support for object-oriented and functional patterns... are what I'm looking for in a powerful language.

> Are you serious ? I guess somebody has a lot of learning to do.

Besides structured data, which was the entire point of this discussion, I don't know of anything Powershell can do that other scripting languages can't. I am, of course, not counting the many system and application hooks, as those are usually 1) not part of Powershell itself, 2) Windows-only and 3) were already possible on Linux (unlike on Windows, basically everything has CLI support here).

In all of this, btw, I was not considering Python a scripting language, which I easily could have (there are even a few neat shells for it). Once you have Python, it's barely even a contest.

> Aliases would have to be standardised (and I don't mean the crap they pulled with aliasing wget) and regardless, only provide an illusion of simplicity that breaks down the moment you try to do anything more advanced.

OK, wget/curl was bad idea, but lets not overreact here about one bad judgement. Illusion of simplicity ? That breaks ? Cmon... never had a single problem with aliases until pwsh, and I am using Powershell 24x7 almost exclusivelly in aliased form. Also, powershell does standardize aliases, once they are in, they never change (except when it went x-plat, but everything changed, so that is moot point).

> The only place dotNet actually matters on Windows. Everywhere else it's just unnecessary bloat.

Let me just say here that entire sentence is crap :) I do not work for MS but life became good when it was x-platformed. MS reputation was bad in linux world but its a social construct that now starts to fade away. I am into technology, not into that social mumbo jubmo bs.

> I mean, yeah, technically anything Turing-complete is as powerful as it gets.

You cant really compare PowerShell with mainstream langauges, since its made for shell context. Regardless, you can do all those fancy stuff in C# and compile it inline. Native powershell has all the things you mention tho, seems you need to take a good look into it (except coroutines but who cares for that in shell ?)

> I don't know of anything Powershell can do that other scripting languages can't.

Let me cite another well known person to you - I mean, yeah, technically anything Turing-complete is as powerful as it gets. The point is to do it in easy to understand and concise manner, without reinventing the entire universe in the process.

> In all of this, btw, I was not considering Python a scripting language, which I easily could have (there are even a few neat shells for it). Once you have Python, it's barely even a contest.

I was waiting for this: lets get python (or ruby or <insert-non-shell-language-here>. Lets be serious about this.

> aliases, etc.

Aliases only hide verbosity in base commands. If the parameters and switches are overly verbose too, they can't help you there.

> dotNet

I'm not saying it's not good and certainly not just because it's from Microsoft. I've done Windows and Linux development and while it's the way to go on Windows, I've never missed it on Linux.

> You cant really compare PowerShell with mainstream langauges

I never intended to, but you called it as powerful as it gets, which really isn't true. I was just giving you examples of why it isn't.

> reinventing the universe

See, to me, the reason I can't switch to powershell is because it seems to reinvent too much - both from a shell and a scripting perspective.

> Python

The whole reason why I brought it up was to continue my original point of "unimpressive for scripting". While I appreciate the convenience of using the same language for shell and scripting, I have no problem using a non-shell language to do more complex things.

Either way, you have successfully convinced me to give Powershell a try one more time once I get some free time to experiment.

> If the parameters and switches are overly verbose too, they can't help you there.

There are parameter aliases too, proxy vars and you can shorten their name to unique subset which gets you as far as you can get anywhere

> Either way, you have successfully convinced me to give Powershell a try one more time once I get some free time to experiment.

Thats good to know :) Do it right, and you wont regret it, I can promise that.

> It's a shame Powershell is a disaster

What's your take on nushell?

I looked it up and it looks like basically exactly what I want. Thanks for the suggestion. I'll definitely give it a try in the following weeks.

"ex" stands for extraterrestrial.

Powershell is a disaster? Did no one tell Microsoft or the hundreds of vendors supporting Powershell as an interface to their product?

Wasn't UNIX originally used to format patent applications at Bell Labs/AT&T? That was purportedly the selling point of UNIX to the business people.

When someone suggests using something other than "text" (usually it's some binary format, I consider S-Expressions to be text), I always wonder, why is this person writing their suggestion using text?

Then we have the suggestions that "everything is a file" is also a "mistake". That's great but if a "file" is not a good enough abstraction then what is?

Text is a human language.

Computers do not really run on text. You can't talk to Unix like it's the bridge computer in Star Trek. No, Unix runs on bytes that sort of look like text because we have encoded and rendered it that way.

What computers need most for structured processing are data types. At instruction set level we had a period of experimentation with byte and word lengths which was still in effect when Unix was born. At that time we also had experimentation with text encodings such as EBCDIC and the various ASCII codepages. Since then we have pretty much settled on standards based on multiples of an 8-bit byte, and on a common text encoding - UTF-8. At the time of Unix's inception the byte stream was a relatively bold move because it enforced variable-length encodings of bytes as the building block of interoperability, versus some kind of static record. Now it is mundane, and the world has moved on to always parsing the bytes for some other purpose.

We desire standards that encode data containers, strings and numeric primitives, so that we do less parsing. XML and S-expressions offer two methods of encoding hierarchies of strings. More recently, JSON has become a very common encoding. It encodes strings, numeric values, and two types of hierarchical containers(key-value and array). We also have SQL databases as an older example, which encode various primitives and a variety of tabular data relationships. With each data language we get one or more associated query languages to express how we select data.

Unix's handling of files is incomplete because of the ambiguity of queries on files. We rely heavily on relationships between directories and on glob syntax to perform selection in a Unix filesystem. Unix paths and glob syntax are not really a byte stream or a file, but they are most certainly part of Unix, and you would be suffering in short order without those concepts. Yet Unix does not respect the existence of its own query language, and never declares a basic system representation for queries and their results akin to the results of a SQL query. Instead we parse the query as bytes and output the results of queries to bytes and parse those bytes, which means there is no guarantee of support at any of the boundaries where your program touches paths. As long as you only deal with one file at a time you don't have an issue, but at scale, the way we work with our applications needs more complex selections on files; the workaround is to "bottle up" the data into one file that has some other structure, which then necessitates special tools to move around the data.

And if you look at what people complain about when working with the shell, the behavior of file selection is a major complaint, and adds a lot of complexity.

> When someone suggests using something other than "text" (usually it's some binary format, I consider S-Expressions to be text), I always wonder, why is this person writing their suggestion using text?

Of course Unix actually operations on streams of bytes, which often represent text, but not always.

Linux is the best operating system for a project like this. The system call binary interface is stable at the processor architecture level:

https://www.kernel.org/doc/Documentation/ABI/stable/syscalls

http://man7.org/linux/man-pages/man2/syscalls.2.html

http://man7.org/linux/man-pages/man2/syscall.2.html

Any Linux process can perform Linux system calls, no matter what language the software is written in. Instead of relying on C libraries to do everything, a compiler could emit system call code when necessary. The entire Linux user space could be rewritten in a higher level language that way. Rust would be the likely candidate but there's no reason a Lisp or Scheme couldn't do it. The language would need full support for features of binary interfaces such as function calling conventions, pointers, structures with padding and alignment, etc.

Do you know of any projects exploring this space?

The Go runtime apparently uses system calls directly. The implementation of every other programming language I know uses the C standard library. Rust has a freestanding mode which is great. I even made a crate for Linux system calls but that needs inline assembly which isn't stable yet.

As for user space projects, I know this one:

https://github.com/arsv/minibase

I meant to start my own but replacing what already exists is a lifetime of work. Haven't had much free time these days nor the willpower to make it happen. I created a liblinux but it's incomplete:

https://github.com/matheusmoreira/liblinux

I also found out the Linux kernel itself has an excellent system call stubs header which they apparently use for their own tools:

https://github.com/torvalds/linux/blob/master/tools/include/...

Go does syscalls directly on Linux. Libc is only needed if you use a thing that only exists in libc (e.g. NSS).

In that vein, Zig also uses syscalls directly on Linux in its standard library so as to avoid linking with C unless desired. This is not possible on the various BSDs.

In matter of scheme you have the guile project. https://www.gnu.org/software/guile/guile.html

so maybe guixsd ?

> This would be a great opportunity to avoid the classic Unix mistake of using text files as the lowest common denominator everywhere

Text files are just a binary format that happens to be easily readable by humans, thus making the hurdle of parsing or generating something to use that format much lower. It usually is this hurdle that is most important to longevity and wide spread success than close to metal sympathies- cf JSON and Protocol Buffers

Text files are easily readable on a Teletype or a VT100. That was a solid design decision back in the day when you had to say "screen editor" to distinguish them from the ordinary line editors.

We didn't have interactive debuggers and inspectors and loggers, so we did everything we could to make systems naturally debuggable. It's like how the Model T engine cover just lifted off with a big door handle. When a technology is new, people using it are expected to be mechanics, too.

But today, even amateur wrote-it-over-the-weekend programs have realtime visual editing of rich data. There are countless free libraries for serialization of every format, and GUI (and TUI) libraries for every toolkit/language/style. Developers enjoy using JSON because it lowers the "hurdle of parsing or generating". It's not 1970 any more.

It's been done using Tcl [0] in the nineties, in the EtLinux [1] project for embedded systems. One of the goals was to shave off memory usage by implementing tools under a single scripting interpreter (instead of each of them having their own binary). Kind of overlaps with BusyBox [2] philosophy...

[0] https://www.tcl.tk/

[1] http://www.etlinux.org/

[2] https://busybox.net/FAQ.html#goals

Thinking about structured data at the OS level makes me want to see a revival of Classic Mac OS's resource fork.

Structured metadata in every file is a dream of mine.

The resource fork idea was OK, but it suffered from a implementation optimized for a floppy disk. Writes were deferred as long as possible, and if anything went wrong, the whole data structure was corrupted.

"Text Edit is not a text editor, and the resource fork is not a database" - early Apple publication. Both could have been. Programs tend to need little databases, for preferences and such, and the resource fork was way ahead of doing everything with a text file. But the implementation was never toughened up to database standards after the Macintosh got a hard drive and more memory. And Text Edit was limited to 32K for a long time.

Meanwhile, Unix struggled along with text files for everything. And lock files for the text files. And daemons to remove dead lock files. And a lack of file locking. And a lack of interprocess communication. Eventually the Unix/Linux world got that all fixed, but way too late.

This could be handled by convention of treating directories as though they were a single structured "file" under certain circumstances. There is precedent in the form of Application Directories from NeXT, RiscOS, and ROX Desktop. Modern MS Office files are also basically just a zipped up directory of files and metadata.

I have a similar idea. I've been working on a design for a Common Lisp-based desktop environment that can run on top of Linux and the BSDs. It's inspired by the Lisp machines of old and OpenDoc, an API for building component-based software that was championed by Apple and IBM back in the 1990s before Steve Jobs returned. I haven't started writing any code yet, partly because I've been learning X programming and other parts of the graphics stack like Cairo and Pango.

I have some more ideas at http://mmcthrow-musings.blogspot.com/2020/04/a-proposal-for-..., though I wrote this before I committed to using Common Lisp (I talk about using GNUstep near the end of the document, but I've since ruled it out due to the difficulty of finding modern Objective-C bindings for SBCL that work with GNUstep).

There is Mezzano, a Common Lisp operating system. Currently designed to be run within Virtualbox. https://github.com/froggey/Mezzano

> a structured data format like S-expressions could be passed around instead.

Why would a structured format be a better lowest common denominator than an unstructured one like text files? Or is it files that are the issue?

Unix pipes are great, but having to parse plain-text streams is inefficient and error-prone. Imagine how much more powerful it would be if commands returned data in structures that could be used by other applications without parsing an ad-hoc text format.

The filesystem can be used to approximate this, like it is in Plan 9, but I think that using text as the lowest common denominator leaves a lot of potential power on the table.

Powershell has been growing on me lately. It uses objects to pass data around and it's quite nice.

Using powershell for a month or so and then going back to doing things with text-parsing UNIX commands really highlights how much better things could be if only we moved past doing things the same way they were done in the 70s.

Have you checked out NuShell?

I'm really sad I missed the boat so hard on Powershell.

I grabbed it at the first public release (beta? alpha?), and the thing was absurdly slow. Trying to `ls` took around 10 seconds. I nope'd out and didn't touch it again.

Now I have a coworker who uses it religiously (on linux, who would have thought!) and when I'm watching over his shoulder I'm astounded by how great it is.

How do people use environment variables with powershell? I often want to do `FOO=bar ./program`, but this doesn't seem to be supported. I googled briefly, but there weren't many satisfactory answers.

there isn't a directly supported way. you write a function to backup, run and restore the old variable to get the same behaviour. Not very ideal.

    $Env:FOO='bar'; ./program

It's equivalent of `export FOO='bar'; ./program` , which is not same as `FOO='bar' ./program`, which, in turn, is equivalent to `OLD_FOO="$FOO"; export FOO='bar'; ./program; export FOO="$OLD_FOO";` .

It may be nice for small amount of data on pipeline, but is very slow compared to unix pipes. See their own documentation how they tell you to not do that: https://docs.microsoft.com/en-us/windows-server/administrati...

Interesting idea, might be important to brush up on the implementation rationale of Unix.

To that end there is The Design of The Unix Operating System, Bach [1]. Or A Fast File System for Unix [2]. Or The Design and Implementation of a Log-Structured File System [3]. It seems to me that files solve for the problem of how to deliver an economic memory management system, and they do it by paging cheap disk space so that it's an effective proxy for fast-but-expensive memory. Also files, with their file pointers, can certainly be used as data structures, but this 'understanding' of what each section means is clearly variable, so is best left to specific applications and it's not good to build into the system as a default. Different applications will want different structures for optimal performance...

[1] https://www.amazon.com/Design-UNIX-Operating-System/dp/01320...

[2] https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf

[3] https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf

Unix-like pipes are not restricted to sending text, they may send general bytestreams. Modern data serialization formats (flatbuffers, capnproto, msgpack, etc.) are highly efficient, and using serialization makes it easier to deploy applications in a distributed setting.

The value is in having everything speak a common standard. In unix, that 'standard' is lines of text, which is inherently linear and lacks the ability to idiomatically represent hierarchy.

Conceivably, one could use modern serialization formats or we could do something a bit more interesting, like supporting some notion of "objects" directly in the operating system or runtime (in the case of powershell, the "runtime" is the .net vm; in unix it's the OS).

Format:

* Field type: S - 16 bytes, D - 32, L - 64, Q - 128, B - 256, P - 512, T - 1024.

* Field ID - 3 bytes ASCII Letters, numbers, '_';

* Field-value separator - 2 bytes: ':' and field value terminator, e.g. ': ' or ':"' or ':[' or ':{' or ':(' ;

* Field value 9 bytes ASCII for S, 27 for D, and so on, padded with ' ';

* Field separator: 1 byte '\n';

  SOBJ: X23456789\n
  SFN_:"Length"  \n
  SFV_: 23       \n
  SEND:          \n

Such text can be parsed by both human and computer, because computer can just jump from field to field without parsing everything in between, like humans do.

Sure, but the standard GNU or BSD userland doesn't do that, does it? If you're going to the trouble of implementing your own userspace, then you might as well use a consistent, proper data serialization format (either text or binary, but text is more accessible).

You don't need to send text (but you should by default for 99%+ cases when the performance is good enough and use the advantages of a human-readable format)

The simplicity of a pipe is a feature.

Ultimately, these "text" formats are simply a matter of convention that makes some binary-formatted streams easier to document for humans' benefit. There's nothing wrong with a binary format if it's properly documented in some way, and modern serialization formats can be augmented with schemas and other features that help provide this.

Being able to easily read and edit them matters too (pipes are just files so one might choose that file to be a persistent file on disk).

Having a none-text format would mean all your file editing tools would have to then understand this new binary format. So you’d have to rewrite every tool to manipulate that data as if it’s text. By which point you’ve effectively just reinvented ASCII.

"This would be a great opportunity to avoid the classic Unix mistake of using text files"

Don't get it. Where is the mistake? I think it was its biggest strength. In the successor, Plan 9, everything is a file...

I'd love to see something like RPYC used for interprocess-communication.

Reply on news.ycombinator.com