WebAssembly (WASM) arch support for the Linux kernel | Hacker News Reader

WebAssembly (WASM) arch support for the Linux kernel

github.com

299 pointsmarcodiego65 comments

Demos at: https://joelseverin.github.io/linux-wasm/

That’s fast. Buggy, but fast. I’m totally impressed! Especially because I researched the necessary steps to do the same thing 10 years ago based on [0]. The patches required for this hack touch LLVM, libc, Linux kernel, BusyBox, ... and total approximately 15,000 lines of code.

I ran a small performance test with 'bc -lq' and compared with [0]:

  scale=1000
  4*a(1)

This WASM architecture compilation completely blows away my old emulation setup, which only managed around 200 MIPS. Maybe this approach can be generalized. Running a full Linux distribution at near-native speed right in the browser would be awesome.

[0] https://github.com/s-macke/jor1k

Your project was also really nice to play around with. I think it was one of the few which actually had an interesting idea including (blink), (copy.sh)

I generally preferred copy.sh more to be really honest. I have actually used it sometimes as a poor man's qemu. If I may ask, what are your thoughts on copy.sh as I found that its performance on busybox or (tinycore linux with gui) was so brilliant (the only downside was that the internet speed was abysmally slow, like for me really really slow.)

copy.sh has the advantage of being x86-compatible and can run many different Linux distributions. However, this CPU choice also makes it quite complex and relatively slow (not sure, if this is still correct).

My own OpenRISC CPU emulation fits into just 1,500 lines of code, and I optimized every single line. To make it work, I had to compile my own Linux distribution completely from scratch. I stopped working on it about eight years ago, but I’ve completed a dozen other successful projects since then.

I’m still very proud that nearly every browser-based Linux emulator, including JSLinux and copy.sh, uses my 9p-virtio- filesystem approach. It makes running complex Linux distributions in the browser much simpler.

Overall, my thoughts about copy.sh’s work are entirely positive.

nice benchmark. comparing to fabrice bellard's jslinux (https://bellard.org/jslinux/) it's roughly 20x faster (if arm on arm) and 64x faster (if x86 on arm)

What results did your benchmark get?

By a factor of about 170. But this is more of a micro benchmark that gives you a rough idea. It's not a definitive figure.

Tried running:

  ~ # du -h 
  (...)
  [Runner sh (2390656)]: Wasm crash: RuntimeError: operation does not support unaligned accesses
  [Main]: Stopping CPU 0
  [Main]: Stopping CPU 1
  [Main]: Stopping CPU 2
  Kernel panic - not syncing: Aiee, killing interrupt handler!
  [Runner sh (2390656)]: Kernel panic: Aiee, killing interrupt handle

> Due to a bug in LLVM's build system, building LLVM a second time fails when building runtimes (complaining that clang fails to build a simple test program). A workaround is to build it yet again (it works each other time, i.e. the 1st, 3rd, 5th etc. time).

I'm incredibly curious what this bug might be!

I just yesterday read about obfuscated Trojan quines in compilers, and a good test being building the compiler twice and it being the same, and now I hear this.

Spooked me for a sec xD

Unrelated to this issue but I've had a race condition with Automake which while run oin 2-4 threads occured exactly every 2nd run. With -j48 it was obvious it's a race condition. No idea how cache invalidation works in the automake stack, but that must have caused it to fail exactly 50% of the time.

This is cool because it avoids emulation. However I think it has many shortcomings today which could all be solved by emulating a real CPU architecture (e.g memory protection support, ecosystem with tooling and Linux distributions).

By the way I have developed a similar project, WebCM, a RISC-V emulator capable of running full Alpine Linux that can be embedded in the Web browser and can reach up to 500 MIPS for some users, which I think is pretty fast despite the emulation, you can try at https://edubart.github.io/webcm/. Booting is also fast, it always boots from scratch when you open the page, so you can boot fast even with emulation.

That indeed feel fast, awesome stuff!

That is excellent!

How does this compare to the c2w container2wasm approach?

container2wasm/container2wasm: https://github.com/container2wasm/container2wasm :

> container2wasm is a container-to-wasm image converter that enables to run the container on WASM.

> Converts a container to WASM with emulation by Bochs (for x86_64 containers), TinyEMU (for riscv64 containers) and QEMU.

> Runs on WASI runtimes (e.g. wasmtime, wamr, wasmer, wasmedge, wazero)

> Runs on browser

> x86_64, riscv64 or AArch64 containers are recommended.

/? container2wasm: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

ktock/vscode-container-wasm https://github.com/ktock/vscode-container-wasm :

> Containers on VSCode for the Web [ https://vscode.dev ]

ktock/vscode-container-wasm-gcc-example: https://github.com/ktock/vscode-container-wasm-gcc-example

JupyterLite works without install on Chromebooks.

JupyterLite still lacks a Terminal e.g. with BusyBox Ash in WASM, with a file system integrated with the Jupyter-xeus kernel file system.

This appears to load much more quickly than other Linux and I think even just bash in WASM demos I've seen.

That requires an ISA emulation layer, this new implementation doesn't - here, every binary is compiled as wasm, and every child process runs as a new Wasm WebWorker, and the Kernel ABI is exposed as Wasm export functions.

Removing the ISA translation layer has the potential to be massively faster for full-system environments. At the expense of maybe some new bugs.

The performance should ultimately be similar to compiling your userspace application directly as Wasm, but you now get to take advantage of the full kernel ABI instead of just the minimal shims that Emscripten give you / whatever DOM glue you create yourself.

One less layer of translation!

Shouldn't browser tabs and/or origins get their own SELinux contexts like all Android apps since Android 4.4, like container-selinux and openshift's k8s? https://news.ycombinator.com/item?id=45418918#45421242

uutils/coreutils, findutils, diffutils, and Toybox are written in Rust which IIRC has a cleaner compile to WASM: https://news.ycombinator.com/item?id=45495100

RustPython may for may not also have a faster loading time than CPython compiled to WASM, though there are already some patches to CPython for WASM.

Where are the tests for the post-patch bugs this finds? Are they're expected behaviors that are not yet in tests which specify?

Really impressive work. Would love to see it progress.

Some ways I can see it could improve:

  1. setjmp/longjmp could implemented via Wasm Exceptions (this is how we do it on WASIX) - no need to wait on stack switching proposal
  2. fork could work easily with asyncify (start/resume), per binary compiled
  3. JIT could work via dlopen/dlsym (compiling the Wasm and linking it), even with runtime patching (using memory spaces on tables and updating them as you go to newly compiled code).

In general, I recommend taking an inspiration from WASIX [1] for those things, as we have spend quite a bit of time to make things work as much as possible!

[1] https://wasix.org/

This is crazy cool. 8,000 CPUs. I wonder if any types of programs would ever make 10k tasks in their normal runtime behavior.

"One important difference is that there is no way to suspend execution of a task. There is a way around this though: Linux supports up to 8k CPUs (or possibly more...). We can just spin up a new CPU dedicated to each user task (process/thread) and never preempt it. Each task is backed by a Web Worker, which is in practice backed by a thread in the host OS (through the WebAssembly implementation). "

Very cool! I'm curious as to how it compares with WASIX in terms of both compatibility and performance.

Also tangentially related: I'd love to see a performant build of Node.js compatible with this runtime (or really any flavor of WASM), but I think you'd run into the same issues that I have with WASIX. Namely build headaches, JIT, and wasm(-in-wasm) support. I'd explore it myself but I've already sunk way more time than is reasonable on that endeavor.

The Gary Bernhardt prophecy is still alive.

(https://www.destroyallsoftware.com/talks/the-birth-and-death...)

Could this work with https://github.com/webassembly/wasi-libc ?

It seems like OP put together their own musl-based libc which is awesome, but being able to compile against WASI would open up a lot of possibilities.

This also reminds me of the recent thread on user-mode linux -- how easy it would be to compile to WASM was definitely on my mind.

   ~ # ping 8.8.8.8
   PING 8.8.8.8 (8.8.8.8): 56 data bytes
   ping: can't create raw socket: Function not implemented
   [Runner sh (18823808)]: Wasm crash: RuntimeError: memory access out of bounds

darn

Afaik wasm cannot open network sockets.

The segfault is unfortunate though

You can write a network device driver, which exports the network packages into JavaScript. The author already wrote a console device. So, not much of a deal.

https://github.com/joelseverin/linux-wasm/blob/master/patche...

Doable for http and https, but if you're running it in a browser environment, you'll eventually run into issues with CORS and other protocols. To get around this you need a proxy server running elsewhere that exposes the lower layers of the network stack.

This is exactly what [0] does. Try it out. If you know the IP you can even log in to another open browser window via telnet.

[0] https://github.com/s-macke/jor1k

Aha! Now I see I'm talking to the expert on the topic ;) Thanks for the link. I'll check this out.

This is such a brilliant thing.

I am not a person involved in building anything like this but I am a person who frequently used copy.sh/v86 and was actually building a tinycore fork which added jujutsu so that people can run jujutsu to try it out in their browser. It was a project which made me understand so much more about building linux from scratch, what isos are, and everything.

One surprising thing was that I was able to make the iso actually have jujutsu and it could run on something like qemu but not on tinycore due to some minor issue

Basically I believe that some really interesting quick-tries of some software can happen if we can have customized linux with our apps directly in the browser

This is such an interesting project really and it feels very snappy to me.

I have a quick suggestion as someone trying to make a custom iso linux / binary apps just work on linux in browser: Can you please, please, create a docker image where you can give static applications as an input in the docker and it would automatically generate the html page with wasm or the wasm output containing that static application in the /usr/bin

You mention something like this this

docker run -it -name full-linux-wasm linux-wasm-contained:dev /linux-wasm/linux-wasm.sh all

I haven't read the project more but I do find it incredibly minimalist and I genuinely hope you can add the ability to add static/hopefully some day glibc as well but I am mentioning it because there are a lot of golang tools which are statically linked and it would be so interesting to running them in browser if possible via their binaries

Yes I know that they could probably run via compiling into wasm itself but that just felt so much messy to me on how I could provide other tools like busybox with it or how it would actually feel like a linux environment to test it out as an example

I have to admit, your project is really really cool. Good luck on this project! Starred, and have a nice day.

Considering that WASM has a Lispy text syntax (known as WAT), I guess this is could be seen as a revival of Lisp Machine...somehow?

https://en.wikipedia.org/wiki/Lisp_machine

There is also elfconv (by my colleague): an AOT binary translator that directly converts Linux ELF to Wasm. https://yomaytk.github.io/elfconv-demo/

The Wasm port of the Linux kernel sounds quite interesting as it may potentially help improving the syscall compatibility of elfconv.

I think lightweight wasm OS runtimes can have real use cases for instance in "cloud terminals" (such as in Rancher or at hyperscalers) which currently just connect a web terminal to some k8s pod. Or in more snappy Jupyter scientific notebook ecosystems. But people have to meet a sweep spot of downloading <1MB runtime, realizing networking and enough tools within the VM.

How does it compare to https://xrsh.isvery.ninja (ignoring the XR aspect)?

Hopefully this will make WASM more popular. I tried to get into it but lack of documentation was already one reason to not invest too much; speed concerns mentioned by other bloggers also amplified this issue recently. For some reason WebAssembly is not really "breaking through" right now. Perhaps it is inertia, perhaps another reason.

Wasm is used in a lot of nooks and crannies, apart from games, Figma already uses it in the core and Wasm-GC has just started to become viable so we will se a lot of server-side languages get better web support.

Using Wasm as an end-all system was never the main intention even if we're heading that way now thanks to all the work people has put in.

I'd say that it's probably used where it's made sense so far.

I hope so too. Websites that load runtimes for various programming languages are too slim; they need to load the entire operating system, otherwise why do we need all these powerful home computers?.

There's already a problem of downloading 10s of megabytes of web assembly. We don't need to download gigabytes for a single page.

Yes, but then how else could I run a docker container in the browser?

So next step they run a browser in this Linux in WASM in the browser. And then run Doom in that browser using WASM.

I hope the situation gets better for Firefox.

What situation exactly? Tried the demo (https://joelseverin.github.io/linux-wasm/), seems to run fine. There isn't any benchmarking programs/scripts available inside of it, so can't really give out any numbers, but it doesn't seem to work worse than any other "Linux-in-a-browser-tab" I've tried earlier. Using a 5950x with Firefox on Linux 6.17.6-2 FWIW.

Sorry, I should have been more clear in my comment. I was referring to the statement from the project about debug capabilities:

> I recommend Chromium-based browsers over Firefox, as the latter does not work very well when debugging Wasm projects of this size.

Very cool.

If anyone's curious to see what's packed in here at a glance: https://modsurfer.dylibso.com/module?hash=3fa6b28252b0d72c82...

This is great, I started working on this during the early days of WASM, however left coz of busy schedule... Happy to see someone making progress ;)

killed by the fork bomb

    :(){ :|:& };:

How did that look on the host system CPU/memory wise?

Mine jumped up to ~3gb and then the vm crashed. It happened fast enough that I didn't really see the CPU spike too much. Firefox FTR

~ # wget

[Runner sh (18815616)]: Wasm crash: RuntimeError: abort

Illegal instruction

when docker ;)

What's the point ?! It's insane. Who on earth would use that?

These questions are the number two most important questions to ask, in software. The sanity/insanity part is not so relevant, but it is necessary to point out that, pretty much a huge percentage of software any of us uses on a daily basis, started off with someone having a random insanity, answering those two questions with a working binary, and thus setting the idea towards becoming normal and thus sane.

Soon enough, WASM may just well be the #1 platform upon which to run a Linux on a Desktop ..

I can think of a few uses quite easily:

- Testing a distro or specific software without downloading it

- Educational use (teaching Linux basics on Chromebooks etc)

- Bypassing restrictions on installing certain software

Ah, thank you!

I think this is really cool but I was struggling to think of a way it could be useful. Your last two suggestions seem especially pertinent.

Because someone can... While I don't see a practical use myself, beyond educational or experimental, that doesn't mean nobody else could, should or would.

In the end, it's kinda cool.

I don’t think it’s that big a stretch of the imagination to see how this could be used in smaller pieces than the entirety of Linux.

Yeah. Basically any code you can run on Linux you can now run on a browser with a lot less work.

Realistically, with quite a bit more work (compared to e.g. v86), but at much higher performance.

Like Lynx?

Yo dog

To elicit surprised cries of delight from the HN readers, of course!

$ rm -rf /

and it's gone

Not quite right. Try the following.

  echo *
  cd /proc
  echo *
  while read line; do echo $line; done < /proc/cpuinfo

The last line should work and print the entire file, but it seems there's a bug.

Well, it should not surprise you that the virtual file systems of the kernel remain.

Only on 127.0.0.1.

Not necessarily -- with tricks like https://github.com/Manav1011/webrtc-vpn you could import/exfiltrate data easily.

Thank you, Captain Obvious.

Reply on news.ycombinator.com