Reaching the Unix philosophy's logical extreme with WebAssembly
xeiaso.net
xeiaso.net
Imagine a utility function that dumps an embedded string to an unlinked temporary file, sets the +x permission, and returns a /proc/self/fd/N filename so you can exec() a subprocess. It's somewhat difficult because of write^execute limitations in Linux.
Running WASM in process seems like a much saner idea.
what? what's the point?
for me it's the most annoying thing when the simple stuff is hard because why would it be?
In other words, it's just fun :)
The problem with using Rust in Go is that you entirely miss all the parts you don't use Rust; you get the VM overhead of WASM so it kills Rust perf; you most likely introduce problems at the boundaries of Rust/Go.
Again it's a neat idea but why in all things sane would anyone intentionally do this outside of puzzle-solving satisfaction?
WASM allows to distribute cross-platform binary code written in memory-safe Rust, running on pretty compact VM. Think about a plugin architecture that could be built on top of that.
IIRC Inferno OS is like Java in this regard.
How will WASM change what Java has already done?
Why would I compile Rust to WASM when I can compile Rust natively to any number of platforms? And use FFI?
I think WASM, while nice, doesn't bring much new to the table. It's been hyped for years, and I still only see it used here and there very sparingly.
The installed size of OpenJDK 17 JRE on my machine is 186MB, according to the package manager.
I suspect that the WASM VM embedded in the program demonstrated in the blog post is 1.5 to 2 orders of magnitude smaller.
Looking a the JRE size is a bit misleading bc it's been sort of deprecated. You're not really supposed to make Uberjars to run on a JRE anymore but are expected to bundle with the JVM modules you need. It can make very small bundles..
But naturally an Uberjars would be smaller. I think small executables are possible but are also just a nongoal now in the JVM world. Meanwhile they're obviously still very relevant in the webspace and hence WASM
you're not really gunna send JVM bundles dynamically over the wire.
I do sort of agree with the parent that while the goals are slightly different it feels like WASM reinvented the JVM without really bringing any huge improvement (while you loose several decades of libraries)
I don't want to be the guy who explains the joke, but sometimes Xe creates elaborate shitposts that aren't entirely shitposts but contain a very fun element to them rather than being for fully practical sake. Shitposting being an anagram for top insights after all.
That's the lens I used when interpretating this talk.
In terms of performance numbers though, I have quite intentionally NOT included performance benchmarks in this talk because getting stable performance information is nontrivial. I plan to write something in the future about WebAssembly vs native code as a subprocess (the differences with windows may surprise you!), but that is not a thing for today.
Ah, gotchya.
I guess WASM as a target and embeddable VM really helps with security in those cases. Couldn't we also do the same though with any number of arch/vm pairings?
I guess what WASM brings to the table is a compile target friendly enough for things like C and C++, i.e. low level code, and a reasonable VM implementation. It just has to be accepted by everything (both as an output format, and virtual machine impl in the language choice) to work, I think...
Edit: Like why arent we using https://stackoverflow.com/questions/4221605/compiling-c-for-...
There is nothing inherent in the JVM that would make it less secure, we just realized in the meanwhile that blacklisting is not the way ahead, but whitelisting is.
https://wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_and_...
how is running anything through a giant virtual machine (web browser) anywhere close to that? the browser is the monolith people. using a web browser to deliver an application is, always has been, and always will be the slowest, and most bloated way to do that. the benefit of course is, that the result is user friendly and cross platform. but lets not kid ourselves, this is as far from the unix ideals as you can get.
WASI is great but there was an opportunity to standardize on something much more powerful.
Anyway I think the logical extreme of Unix philosophy started with Unix, moved on to Plan 9, and continued to improve from there. It's just that those further advancements are less popular and well known.
An old-fashioned file is not the logical extreme of anything. Not that it isn't useful or interesting.
I stead we don't even have a good way to do any kind of networking.
One can take this further. There was a forgotten system called ChiOs which started with "A bit is a file. An ordered collection of files is a file." and went downhill from there.
"Everything is a file" has its limits.
I rather liked QNX. QNX is a microkernel with microservices. Everything is an inter-process function call. Call, wait for response or timeout. Works both locally and remotely. Very fast locally. There's a POSIX library, but when you call "read", some glue code makes a call to the file system service.
As a base abstraction, remote procedure calls work better than files. Files implemented via RPC are simple. APIs via RPC work are very similar to local function calls. RPC via a file interface is complicated. You have to put a protocol on top of stream/file oriented sockets to get a message interface.
The point that the author seems to be making is that WASI offers a standard API. It's one that crosses a memory protection boundary, like an inter-process function call. This is a reasonable way to do things. For historical reasons, neither Unix nor Windows supports that approach well.
Wasi Preview 2, which we are aiming to launch by the end of the year, rebases Wasi on the Component Model proposal, which enables composition of Wasm programs, including those which are written in different languages, and which do not trust each other. Wasi is now specified in the Wit IDL, which has a strong type system for representing records, variants, lists, strings, and best of all, external resources, including sugar for constructors, methods, and destructors.
Instead of basing everything on the filesystem abstraction, the core Wasi primitives are the `input-stream`, `output-stream`, and `pollable` resource types, for readable and writable bytestreams, and a pseudo-future: you can `poll-oneoff` on a `list<pollable>` and it will block until one is ready, and return a `list<bool>` indicating the set which are ready. `wasi:filesystem/types.{descriptor}` is the resource for files, but if you need to read, write, or append to a file, you can do so by calling a method on `descriptor` that returns a `input-stream` or `output-stream`.
Preview 2 is also adding networking: wasi-sockets for platforms which support sockets, and wasi-http for those which don't, like the Web.
We are closing in on shipping Wasi Preview 2 but its not quite fully baked yet - changes related to resources are slated to land in the net few weeks. The spec definitions are on github: https://github.com/WebAssembly/wasi-io/blob/main/wit/streams... https://github.com/WebAssembly/wasi-filesystem/blob/main/wit... . Stay tuned for much more approachable documentation, tutorials, and so on, once we are confident it is a stable target ready for users.
COM was pretty reliable. Yes, it was needlessly annoying and gave CS folk the screaming ab-dabs, but it worked and was predictable.
I can see COM-in-WASM being really useful. Especially if we can dynamically load components. And not only for browser coding.
Look at the latest Microsoft thing about embedding Python in Excel. They're going the wrong direction. What everybody wants is to be able to drive Excel from Python aka an API that people could hook into.
Even WASM is kind of ... weak ... because it has to deal with the lowest common denominator--a web page with a single thread of execution and no access to anything.
And, COM wasn't terrible--it's just that the languages attempting to support it were very underpowered at the time. COM with VB6 created a huge ecosystem that probably still isn't really matched today.
I just think it’s funny since it was so heavily derided by people. It took awhile for me to appreciate its reasons to exist, and I still don’t like some of the vocabulary they used.
https://en.wikipedia.org/wiki/Object-capability_model
https://monte.readthedocs.io/en/latest/taste.html#cooperatio...
I have found quite the opposite to be true in the real world, but Xe gets a special exemption for being a consistently high-quality writer and thinker.
Caveat: you'd need to use Wasmer forks of, e.g., rustc and Tokio, but I imagine there are at least some people who would be okay with that.
It is super nice with the local first approach, and I wonder what it would take to be able to `mount https://example.org` from `test.com`, if both websites are running the "wasm web OS" your linked to.
Cheers!
Unix is an object-oriented OS.
(The nomenclature is quite backwards. It would be like building a database of employees, companies, etc. then saying "everything is an employee", companies are just a special kind of employees.)
In your "companies are just a special kind of employee" example well if that is true within your data model then that is quite interesting and contributes to the listeners understanding of how things work. The alternative "companies are a just a special kind of entity" is less useful.
Or maybe at least Typescript-to-WASM compiler to skip JS output?
Also Typescript -> WASM exists in some form through assemblyscript
I guess after all these years of waiting for WebAssembly to actually be useful (as in: I can write apps as easily as I can on desktop, with similar performance, but with the safety guarantees of a web sandbox), I finally realized what I wanted: web pages that are backed by virtual machines. Every page is just a view into a VM's framebuffer. User events are delivered just like X11 or win32 events. The page works like a desktop app. Any code that can run in a VM, can run in the window.
After all, the browser is just an inner platform and you should be able to run an virtual machine in your inner platform.
When I say VM, I mean, "VMWare-style VMs" not "VMs that run a language the same way on multiple platforms".
The magic trick happens with #SheBang because that tells you who should interpret and execute the file-contents. That is "magic" because in a sense you must read (part of the content) to know how the contents which is just 0s and 1s should be interpreted.
A file is a collection of data, which usually has an associated identifier (filename and/or path). A USB mouse drive is not a file, but it can be represented as such.
The word "file" is ambiguous, and was used ambiguously in Unix documentation and papers. It of course meant an object with a name in the filesystem. But it also can refer to an object (whether or not existing on or even referenced by the filesystem) which is accessed using read/write, independent of how you acquired the reference (i.e. file descriptor) to that object. If you look at the whole of the Unix system, read/write over file descriptors is the backbone of the environment; process inheritance, shell redirection... much of the time a program has no idea how a resource reference was acquired, just that it can be accessed using read/write.
But something you can read and write to is not called a file, it is called a stream. It is no accident that Linux differentiates between “files” that you can write to at arbitrary places, or ones that are append-only.
> [...]
> Files are uniformly regarded as consisting of a stream of bytes; the system makes no assumptions as to their contents.
> [...]
> There is not distinction between "random" and sequential I/O. The read and write calls are sequential in that, for example, if you read 100 bytes from a file, the next read call will return bytes starting just after the last one read. It is however possible to move the read pointer around (by means of a "seek" call) so as to read the file in any order.
Source: https://www.bell-labs.com/usr/dmr/www/notes.html
Though the abstraction is incomplete, the core concept in Unix is that "files" are opaque streams of bytes. Ancillary functions (e.g. seek, ioctl) are then layered atop the basic API to deal with files of different types as required.
Consider that before Unix most file APIs were record or block oriented. Unix unified the I/O model behind a single "file" abstraction--a stream of bytes. Unix stretched the term "file" to encompass a broader range of I/O tasks, but in turn also changed the treatment of traditional disk files in a way that made them look more like non-disk file I/O. This abstraction couldn't completely hide how I/O was serviced on the other side, but one can't criticize "everything is a file" without understanding the context of the time; nor can one full appreciate the value-add.
Also, something I hadn't notice before is how DMR emphasizes the synchronous nature of the API. Apparently many I/O APIs back then were asynchronous. Unix made the I/O model synchronous, but made it easy to create and juggle multiple processes so that you could implement various asynchronous models if you wanted. IOW, they flipped the default case. With the contemporary concern with intra-process I/O concurrency, that's an evolution we seem to be recapitulating.
Is there also a reverse operation, turning multiple small files into a single bigger file?
A) Tasks should done by combinations of simple tools combined into an elegant pipe.
B) Each tools should do three things, one of them at least so-so well:
Those three are:
1) Hackily parse the output of the previous tool by flaky assumptions like that a certain delimiter is always present, that spaces will not occur in such and such an item, or that such and such a field is from this column to that one and never overflows.
2) Do the actual processing correctly and efficiently --- provided nothing in the data, exceeds a 1023 character limit, or overflows an addition of two ints or doubles.
3) Produce output in some way that is hard to parse correctly for the next tool, like columns that may be empty, contain items with spaces, or overflow so that column widths are not reliable.
1) This is not done by every tool. Instead, there are tools built for specific parsing purposes, and each tool is only in charge of interpreting its command-line arguments or, optionally, stdin in some suitable way. Tools are agnostic of the output of previous tools, and only care about processing data that makes sense to them. It's the task of the user to ensure this data is structured correctly.
2) Where are you getting these limits? Shell limits can be controlled by ulimit(1p), but the standard file descriptors function as unlimited streams of data.
3) Again, every process is free to choose the best way to output data. Some have flags that make processing the output easier by another tool, otherwise they default to what makes sense for the user.
You seem to have a bone to pick with the fact that the data shared between processes is unstructured, and that the user must handle this on their own, but this is what enables building independent tools that "do one thing well", yet are still able to work together. Sure, in a tightly controlled environment, tools can share structured data (e.g. objects in PowerShell, JSON in NuShell, Murex, etc.), but this comes at the expense of added complexity, since each tool needs to handle this specific format, encoding, etc. This is difficult to coordinate and scale, and arguably the rich Unix ecosystem wouldn't exist if some arcane format was chosen 40+ years ago, or if a new one needs to be supported whenever something "better" than JSON comes along. Leaving the data unstructured and up to each process and user to handle, ensures both past and future compatibility.
Mime type: text/arcane
Whatever it were, it would need updating, which means all tools would need to be updated to support the changes, while maintaining backwards compatibility. This is a mess in practice, and probably only acceptable to a single project or organization that maintains all tools, but it's not something that allows an open ecosystem to grow.
It's naive to think that a modern solution can "fix" this apparent problem. New shells and environments can be created that try to address it, but their future is uncertain. Meanwhile, the fact that Unix still exists today in many variations is a testament that those early decisions were largely correct.
And the fallacy of alive=right is also worse than "naivety" since it prolongs the pain for a few more decades longer than necessary (it's a big part of the reason why all those much better tools face uncertain future)
I’m going to poke around and see how much it would take to stretch this into a basic Turing machine emulator.
I don't know how I feel about the message "This is optimal!" displaying when you use one NAND gate to build an inverter at level 2. Level 1 forces you to build an NAND gate out of (1) an AND gate, plus (2) an inverter. It feels like it'd be more optimal to just reuse that inverter.
(And then level 3 is an AND gate, the other primitive† you started with...)
† Technically, you start with two primitives, implementing f(a, b) = a ∧ b and g(a,b) = ¬(b ⟶ a). You also get the ability to provide 0 and 1 as fixed inputs, thus ¬a ≡ g(a,1). What you lose after implementing the NAND gate is the ability to provide a fixed 1 input - instead, you are informed that you assume all gates automatically draw from this. Worst of all possible worlds.
and my implementation of the projects: https://github.com/porridgewithraisins/nand2tetris
alias NAND="AND | NEG"
Sorry? Take a second to think about what you're saying.
In what conceivable sense is "doing a logical and" one thing while "doing a logical nand" isn't?
You can equally claim that
x & y
(x ↑ y) ↑ (x ↑ y)What did you think I meant by this question?
>> In what conceivable sense is "doing a logical and" one thing while "doing a logical nand" isn't?
NAND is more complex than AND, in the sense that it is more expressive than AND (having functional completeness which AND does not).
Similarly, it can be built from other less complex operators (AND and NAND).
If you're taking "One thing" to the extreme, in terms of the granularity or complexity of that "one thing", NAND is not as granular or simple as AND - and therefore isn't taking it to as far "to the extreme".
+---+---+-------+
| a | b | a ↑ b |
+---+---+-------+
| 0 | 0 | 1 |
+---+---+-------+
| 0 | 1 | 1 |
+---+---+-------+
| 1 | 0 | 1 |
+---+---+-------+
| 1 | 1 | 0 |
+---+---+-------+
+---+---+-------+
| a | b | a & b |
+---+---+-------+
| 0 | 0 | 0 |
+---+---+-------+
| 0 | 1 | 0 |
+---+---+-------+
| 1 | 0 | 0 |
+---+---+-------+
| 1 | 1 | 1 |
+---+---+-------+
> It seems naively obvious to me that a(b(x)) is more complex than b(x).
This is just obvious gibberish; if you define b(x, y) = x & y and a(x) = ~x, then you can say "I think a(b(x, y)) looks more complex than b(x, y)", but how do you respond to "when c(x, y) = x ↑ y, I think c(c(x,y), c(x,y)) looks more complex than c(x,y)"? The two claims can't both be true!
Everything, no matter how simple, can be described as the end of an arbitrarily long chain of functions. So what?
It's "Everything is a File"[0]. That one has really stood the test of time. And it's often misunderstood to mean that everything implements {Read,Write,Seek,Truncate, etc.}. Then when a TCP socket or character device or whatever shows up, the whole abstraction leaks, and it seems like not everything is a file.
The Good Parts of Everything is a File:
1. Everything exists in a Filesystem, meaning a directory tree. There are directories, they list things, either resources, or more directories. A process gets access to all additional resources through the filesystem. We've really screwed up by putting networking in it's own place, environment variables, etc.
2. File descriptors. If you squint, these are capabilities[1], which is probably the best way to manage resource permissions. They are fine grained, they handle delegation, there is a tree of legitimacy back to the root. We were so close, but we got this one wrong too, with a single global root, instead of a chroot per process by default.
My takeaway from the transcript was that the author was chaining together webassembly programs like a pipeline in a Unix shell. Is the video about something else entirely?
The talk is fantastic; Xe is a prolific hacker, low-level OS engineer, and engaging speaker. If anyone prefers YT, here's the direct link: https://www.youtube.com/watch?v=QNDvfez6QL0
However, even if not all of {Read,Write,Seek,Truncate,etc} are implemented for all objects, some of them will be implemented, e.g. a TCP socket or character device can use read/write (or some might be read-only or write-only) but is not seekable. This is useful to use programs expecting other files and they will still work. For example, once I had a USB with a exfat file system, which I could not mount, but I knew it contained a ZIP file, so I tried that it it was able to extract the ZIP archive even though it could not be mounted; that is it could treat the USB device itself as a file. And, commonly, it is useful to write stuff that could be written to files, to pipes; for example in Heirloom-mailx you can write attachments to pipes instead of files, and I find this very helpful.
I think that the file system directory tree is not the best way. File descriptors are like capabilities, and that is good (and I agree that it is probably the best way to manage resource permissions), but I think that it could be done better as better capabilities. I think that a single global root and chroot per process are both not the best way; I have a (what I think, at least) better way.
My own design has a file system but does not have directory structures nor file names, but it is a hypertext file system, and files can have multiple forks, and the data streams can contain links (similar to UNIX hard links) to other files. Links can optionally be to a fixed version, or to a changeable version; copy on write can be used if you have both kinds of links to the same file. There is also journaling, and can have locks and transactions that can consist of multiple objects at once (this is necessary in the core system, so that you do not make a mess trying to do such things in user code like SQLite (and any other SQL database engine) does).
My own design also uses "proxy capabilities". Messages can be passed using capabilities, and these messages can contain sequences of bytes and/or capabilities. A program might also create its own capabilities, which can be proxies of others; you can implement fine grained security and also allow fault simulation and many other purposes. All I/O and system calls (except Yield and Quit) must use these capabilities (for full security). A program will receive an initial message when it starts, so it will start up with some capabilities that were passed in that message (which can be whatever capabilities the caller decided to give it). Also, the multiple objects transaction/locking mentioned above is actually a general feature of capabilities and is not specific to disk files; you can make a transaction or lock of any set of objects (if supported; some combinations might not be possible). Proxy capabilities are actually very useful and many of the high-level features of the system are implemented in terms of proxy capabilities, so the kernel does not need to know all of the possible uses.
In this way, you can easily emulate a "chroot per process", although it does not actually work like that. Such an initial message could as easily be used to emulate environment variables or whatever else you might want, too; a POSIX compatibility layer can be possible in user code if needed, although the low-level and high-level design of this system are not designed to be POSIX but rather something different, which does (what I am considering) working better in many ways.
(My own operating system design does not currently have a name.)
I laughed out loud during a work meeting when I read this. Bravo, Xe!
HR can kiss my fuzzy butt lmao
Good luck with the future job hunt.