Isolates, microVMs, and WebAssembly | Hacker News Reader

Isolates, microVMs, and WebAssembly

notes.crmarsh.com

123 pointscharliermarsh55 comments

I'm a little fuzzy on the multitenant security promise of WebAssembly. I haven't dug deeply into it. It seems though that it can be asymptotically as secure as the host system wrapper you build around it: that is, the bugs won't be in the WebAssembly but in the bridge between WebAssembly and the host OS. This is approximately the same situation as with v8 isolates, except that we have reasons to believe that WASM has a more trustworthy and coherent design than v8 isolates, so we're not worried about things like memory corruption breaking boundaries between cotenant WASM programs running in the same process.

At the end of the day, if your runtime relies on a whole shared OS kernel, you have to be concerned about the bugs in the kernel. That's true of VMs as well, but to a much more limited extent: KVM is (by definition) much smaller than the whole kernel, and KVM bugs are rare.

I'm writing this mostly as a provocation; I don't have a clear enough understanding of backend WASM multitenant security to have strongly-held opinions about it.

While I mostly agree with you, let me play the devil a bit (just for fun!)

Let's agree with the base that (probably) most of the bugs will happen within the host defined imports (aka bugs in the systemcalls/kernel) vs bugs on the Wasm runtime per se. In that front, Wasm host system call implementations will offer not that much advantage compared to other virtualization approaches such as Firecracker.

However, those advantages are only perceived when you run software that's already fully containerized (in a OCI-like container). But here's the nit: there's many more software that is not containerized (for example: 3rd party libraries).

Running this libraries in Wasm can actually be more secure than running the native ones (since we are bringing Firecracker-like security to the systemcalls the library may use). Specially, when running this software locally.

So, in my view, Wasm forces most of the software to be secure, regardless of its kind (library or application/OCI-container).

It's a small thing that might make a big difference!

> This is approximately the same situation as with v8 isolates, except that we have reasons to believe that WASM has a more trustworthy and coherent design than v8 isolates, so we're not worried about things like memory corruption breaking boundaries between cotenant WASM programs running in the same process

My understanding is that WASM's spec isn't just more coherent than JavaScript's, but also orders of magnitude smaller, which means less surface-area to check for vulnerabilities

Something that appeals to me about the multitenant security story in WebAssembly is how easy it is to provide alternative implementations for system calls. In most wasm implementations, you begin with a "raw" wasm runtime, and explicitly provide host functions ("system calls") that the wasm code can call. Nowadays the wasm implementation is likely to provide a wasi implementation out of the box, but it's simple to replace the implementation of one or more of the wasi system calls with your own (or define your own syscall interface entirely!). In this way, you can put in extra protections, monitoring, alternative implementations, etc where you see fit.

It's kind of like a built-in mechanism to adopt gVisor's approach to container security. Implementing gVisor is a gargantuan task that few companies would embark on; comparatively, doing the same in wasm is absolutely trivial.

The stuff running inside your WASM sandbox is also an easier target than software running on a native host, since WASM lacks a lot of the security mitigations we would now take for granted, like ASLR. Corrupting function pointers is also very easy to leverage for attacks in wasm because function pointers and functions in general are sequentially numbered starting from 0 instead of being at semi-random offsets in memory, an invoke will either work or produce a signature mismatch error.

So in practice it is probably easier to actually break the stuff inside the sandbox, and then you get to have fun trying to compromise the host system wrapper.

Sure, but there are big protections like that only the start of functions (with compatible signatures) can be pointed to, removing the majority of ROP gadgets, and the return addresses on the stack can not be overridden, so you can't create a ROP chain.

Absolutely, but the flipside is that eval, 'new Function' or 'new WebAssembly.Module' are one import away :( At least you can use content security policy to disable those and modern linters strongly discourage their use from JS, but if you're using many modern frameworks they're often tucked away in the implementation to solve interop problems.

So far in my experience WebAssembly has been great for when I want to import a *little* bit of native code into my browser. I've yet to see it work well as a full on container based solution. I could certainly see a future where that is true, but it'd need the following:

- A way way better tooling story. Emscripten is an incredibly fiddly tool that requires a lot of flags and config. wasm-pack is decent for Rust but seems to be focused on the browser.

- Better interoperability. WASM works great if you write everything in Rust, or everything in C or everything in Zig. Not so much with multiple languages.

- Memory64 and WASI need to land. We need more than 4 gigs and we need proper syscalls.

- Near native needs to be, well, near native. I find it weird that people keep on saying WebAssembly is fast while in the same text decrying "a low-double-digits percentage hit" in performance. I'm fairly certain WebAssembly's "near native" is using near to mean "same order of magnitude", not "within 10%".

I'm also just confused as to what WebAssembly will look like for something that's more than a lambda conceptually. Like if we're running a classic back-end server with a database, are we going to run the database in wasm? Are we running it in a different wasm process? How do you share memory? Is wasm going to reinvent the operating system?

I like WebAssembly as an idea but I'm very unclear on how it'll get to the "replace Docker" level.

> - A way way better tooling story.

I think this is a real paint point of WebAssembly but it's getting better. You can now compile C/C++ code to WASM using clang as it became a first class citizen in v8 (or later?). Of course, clang does not provide all the runtime goodies that emscripten does but if you limit yourself to use WASM for some part of your code that require performance then all you need to provide is a few Javascript glue code. For prototyping purposes I've bundle an npm package that bring clang to your frontend project without the need to install it through a package [0].

Chrome debugger now support WASM [1] but you still need to install an extension. I've started to work on a way to convert the DWARF symbols from the WASM binary into sourcemap on the fly [2][3] but it still rough around the edge and need some work but it can definitely be done. This would allow WASM debugging natively in any browser supporting sourcemaps.

You also have all the WASM binary tools that you would find for any other kind of executable format in WABT [4].

[0]: https://www.npmjs.com/package/@jdmichaud/wasm-toolkit

[1]: https://developer.chrome.com/blog/wasm-debugging-2020/

[2]: https://github.com/jdmichaud/dwarf-2-sourcemap

[3]: http://site.novidee.com/blog/blog-entry.html?article=2022082...

[4]: https://github.com/WebAssembly/wabt

> Near native needs to be, well, near native.

You'll see similar performance variations of the same C code across different compilers, different CPU architectures, on the same CPU when the OS decides to switch between performance- or efficiency-cores, or even just slightly different compiler optimization options (e.g.: real-world performance of native code already varies for many reasons, and WASM is roughly in that same ballpark).

FWIW, in my home computer emulators written in vanilla C (e.g. no language extensions like SIMD intrinsics) the performance difference on an M1 Mac between the native version and WASM version running in Chrome (e.g. https://floooh.github.io/tiny8bit/c64.html) is indeed within 10%.

This is mostly straightforward integer bit-twiddling code, other types of code may behave differently (one should expect up to 2x slower as worst case).

> Better interoperability

AFAIK, the examples you give all target a basic C ABI [0] or can be made to target the same ABI. In Rust, it means targeting wasm32-unknown-emscripten

The Rust team is also working on a "WASM ABI"[1] which would be useful in taking advantage of stuff like multi-value returns, and other compilers could just choose to target that. More likely, the C ABI on WASM will be updated to account for missing features, and that'll be the standard for interoperability in the WASM ecosystem.

[0]: https://github.com/WebAssembly/tool-conventions/blob/main/Ba...

[1]: https://github.com/rust-lang/lang-team/blob/master/design-me...

The article says whether a language runtime like v8 is a stronger security boundary than a hypervisor is a matter of debate, however v8 has boatloads of CVEs and the further isolation of browser sandboxes has been a driving factor in OS development for nearly two decades. Meanwhile, multiple Fortune 500 companies bet their multi-billion dollar cloud businesses on the security of their hypervisors.

So when the author links to a confused HN thread about whether the v8 runtime is a security boundary and says "looks like there's debate", it makes the article look like a joke.

Well, V8 is a thing that writes and then executes memory (JIT), no wonder it had issues.

This feels a bit harsh. The question isn't whether V8 is a stronger security boundary than a hypervisor, it's whether Cloudflare's use of V8 demands process isolation. And the "confused HN thread" is Kenton Varda debating it with Kurt Mackey! But I'll try to make it clearer in the post.

The question is pretty definitively answered unless or until something changes in Google’s assessment of V8: if they don’t consider it safe for process multitenancy, which they have every incentive to do, it’s not safe.

There are use cases I can imagine (and have imagined, I have no bandwidth to pursue them) which would probably be fine. But a multi tenant server isn’t one of them, and I’m still shocked, even at Cloudflare scale, that it’s a consideration.

Is Google using V8 on their own hardware paid for with their own money? I thought they're not using it on GCP for anything. What's the incentive for them to move to process multitenancy? They can simply (ab)use that computers are getting faster, the end users are not going to complain as visibly as losing some money on a cloud platform would be.

> if they don’t consider it safe for process multitenancy, which they have every incentive to do, it’s not safe.

It's more nuanced than that. Unfortunately security is not a boolean thing, where it's either secure or it's not secure. V8 is designed to be a secure sandbox on its own. But security is actually about risk management. V8 is complicated, and it has bugs. It tends to have more bugs than a typical hypervisor does. So it's more risky to rely on it as a secure sandbox, unless you can find risk mitigations to layer on top of it. After a decade of relying on V8 alone to isolate frames from each other (within a tab), Chrome chose to add strict process isolation as a second security layer. But Chrome definitely doesn't think of that second layer as being the "secure" one -- security comes from having both together, so an attack has to bypass both at the same time.

Google still pays big bug bounties for V8 breakouts.

Yes, cloud computing is important, but how hard are security professionals trying to break it? Maybe there are more CVE's for V8 because it's a higher-profile target?

It is so tragically funny whatching everyone re-inventing mainframe language environments, JVM application servers and the CLR.

> > One of the exciting things in Visual Studio .NET is its language agnosticism. If a vendor has written a .NET-compliant language, you can use it in Visual Studio .NET. It'll work just as well as C# or C++ or Visual Basic. This isn't just a future feature-in-planning. There are already nearly two dozen languages being developed for Visual Studio .NET: Visual Basic, C#, C++, JScript, APL, Cobol, Eiffel, Fortran, Pascal, Perl, Python, RPG, Smalltalk, Oberon, Component Pascal, Haskell/Mondrian, Scheme, Mercury, Alice, and even the Java language.

-- February 2002 issue of MSDN Magazine

https://learn.microsoft.com/en-us/archive/msdn-magazine/2002...

It is as if a newer generation never bothered to learn about what came before.

> It is as if a newer generation never bothered to learn about what came before.

Why the pessimistic take? Maybe the newer generations improved upon what came before!

Sure, but then don't talk about the "new" stuff as if it was the best invention since sliced bread, without any clue that Egypitans, Babylons, Romans and ancient Greeks were already having it.

All too often they didn’t even know about these systems, at least in my (limited) experience. Not necessarily true here, but it’s not like NIH/reinventing the wheel isn’t common in certain parts of the industry.

AFAIK the most important difference between WASM and other VMs on the level of CLR or JVM is that WASM has a simple linear memory model, which allows languages which expect a linear heap (like C, C++, Rust, ...) to be supported by just adding new backends to existing compilers (and most importantly without requiring changes to the language).

The CLR and JVM might be able to simulate a linear heap through a byte array (just like asm.js did), but it's questionable whether this allows the same performance as WASM when the whole VM instruction set is built around high level object references.

CLR doesn't need to simulate anything, its bytecode is expressive enough for languages like C and C++.

> It'll work just as well as C# or C++ or Visual Basic.

That C++ in there is weird in a 2002 article. Did Microsoft have a C++ compiler in 2002 which supported .NET bytecode output? Up until very recently they had their own incompatible C++ dialect (don't remember what it was called) with new CLR-compatible pointer types. I'm pretty sure that it wasn't possible to just take a vanilla C or C++ library and compile it to the CLR or JVM at least around 2010 when Emscripten was first released (and from there it still was a long way to go to WASM).

Yes, it was called Managed C++, and was later replaced by C++/CLI on .NET 2.0.

Why people always complain about Microsoft language extensions to C and C++, and yet are perferctly fine with GCC C, GCC C++, clang C, clang C++ ?!?

And I am pretty sure to have wrapped a cross-platform C++ RPC framework for .NET in 2002 while using Managed C++, somehow it even ended up in the market for a contact center CRM, go figure.

Maybe one should learn about Microsoft technologies properly instead of just bashing and writing M$.

Not only does MSIL support all the features required for a language like C or C++, mixed mode Assemblies allow to embedded native code, at the expense of being marked unsafe for the verifier. You can force pure MISL, with the side effect that some UB patterns will be considered a compile error instead.

> Maybe one should learn about Microsoft technologies properly instead of just bashing and writing M$.

Well, if it so easy to build vanilla C/C++ code for .NET then Microsoft is definitely doing a very poor job promoting this feature ;)

Does the cross-platform dotnet toolchain support building C/C++? I never saw this mentioned when reading about dotnet (it was always about C#).

Managed C++ and C++/CLI are Windows only.

Here is an article about Managed C++ from July 2002!

https://www.codeproject.com/Articles/1479/Introduction-to-Ma...

There are others when searching from 2002 articles.

Mono did try to port them as well,

https://www.mono-project.com/docs/about-mono/languages/cplus...

So you see, the information is out there.

Well nice, really - but as you yourself point out, it's Windows only, and being locked to Windows is not an option. Perhaps if dotnet was open source and multi platform from the beginning, it would've been used instead of developing Wasm etc - but we're way past that.

Actually, .NET has been an ECMA standard since the early days.

That is how Mono started.

Additionally here is some historical refreshing about two other attempts,

https://www.gnu.org/software/dotgnu

https://www.codeguru.com/dotnet/net-nuts-bolts-the-joy-of-ro...

A surviving clone of the repo is available here,

https://github.com/SSCLI/sscli_20021101

Finally, this doesn't change the fact that the CLR was only yet another version of this idea, there are plenty of others since UNCOL in 1958.

That's cool, but you can't run standards. I tried Mono since its early days and kept trying until .NET Core came, but it was nowhere near production-ready, it wasn't enough for my school assignments, much less production.

I'm not saying it changes anything. Wasm is just a more modern attempt, like .NET itself. I don't see any reason to prefer one or the other only because of its age.

Those school assignments would have been sorted out by using Windows in first place.

It is not like all those beautiful WASM implementations support the same set of features, from the paper standard,

https://webassembly.org/roadmap/

Well, I wanted to use Linux, not Windows. And my servers ran (and still run) Linux, not Windows. The point wasn't that I couldn't do my school assignments (I was forced to use Windows VM) but that not even this simple task could be accomplished - so running it in production was straight up impossible.

Wasm is much newer than Mono was at the time. Let's wait.

Technically you can always write you own C compiler like this project https://github.com/ForNeVeR/Cesium. Obviosuly C++ is much more complicated journey and require real investment. But at this point this is not CoreCLR limitations mostly.

> It is as if a newer generation never bothered to learn about what came before.

I'd be very surprised if you could find a significant number of people who both 1. care about wasm and 2. had never heard of previous VMs that could run multiple languages.

WASM is cool. That doesn't mean that earlier takes on the same idea weren't also cool. It also doesn't mean that nobody is allowed to build a new take on the idea, and you can be 100% sure that the people involved in implementing WASM knew about the CLR and the JVM, and probably even things like the UCSD p-System.

Software engineers are far too keen to disparage new implementations of old ideas, or even new ideas - "it's not novel - it can't do anything a turing machine doesn't do".

The original people involved in the standard, certainly.

The dozen of blog posts praising WASM, or the dozen startups selling WASM with re-packed ideas, not so certain.

This article says of Fly: "but they don’t scale down to zero".

This isn't entirely true. A little known aspect of Fly Machines is that you can stop them and they will be automatically restarted by the Fly router (with a very short cold-start pause) when the next request arrives. So you can scale them to zero if you do a bit of extra work.

https://fly.io/docs/reference/machines/ hints at this:

> Machines are also the spawning ground for new platform features like wake-on-request (also known as scale-to-zero). You can stop a running machine to save on compute costs. It then may be started automatically when a request arrives at the Fly proxy.

I'm already experimenting moving workloads that I run on Workers to Fly Machines (because HTTP-only nature of Workers is limiting, and TCP Workers are no where in sight), and I like what I see.

That said:

- Fly is missing HTTP/3 (QUIC) support (and it doesn't look like its going to be easy for them to simply bolt it on). Cloudflare is quick to deploy newer standards (even when experimental) like they did with HTTP/3 and ECH.

- Sometimes Fly Machines start-up, but don't get sent any request from the Fly router.

- Workers have free cache (up to 512M per obj), which essentially functions like a free blob store. On Fly, 10G disks spin up alongside Machines, but those are wiped every start/stop cycle. Whenever Fly Machines could be paused/resumed (with state intact)-- likely in near future-- things will start getting really interesting.

- Plenty other rough (but livable) edges on Fly with Machines (that's okay, given it is still in preview). With Workers, Cloudflare has a very stable and highly optimized offering, I doubt established cloud providers can match that in the short-term, let alone upstarts. While Fly Machines (incredibly) boots my NodeJS app up in 200ms to 500ms, it is no where close to the 0ms to 5ms start-time on Workers.

That said, for an unrelated workload where I run OLTP queries on-demand on DuckDB from parquet files stored in R2, I am definitely moving over to Fly Machines from AWS AppRunner (and this workload I can never run on Workers because just the DuckDB binary for WASM goes beyond Workers' 1MB limit).

To me, Fly Machines are more of an AWS Fargate / AWS Lambda alternative than Workers, at this point in time.

I suspect that WebAssembly is actually the wrong abstraction here, and you might want something a little closer to BPF for non-process-isolated serverless compute, but also not BPF because BPF can do arbitrary memory reads. WebAssembly appears to be too permissive for this case, and BPF too restrictive to be useful.

Ideally, if you could have a language-level VM that guarantees that you only ever read or write data inside memory allocated to you and calls a limited set of syscalls (which you intercept/rewrite at compile time), you probably don't need any other protection.

Sounds like what Singularity and later Midori tried to achieve:

http://joeduffyblog.com/2015/11/03/blogging-about-midori/

You still need a way to protect the data inside the VM's memory from getting trampled, since the data could contain the spiritual equivalent of function pointers and those get passed out to the syscalls. Or at a simpler level, if gmail is running in a wasm sandbox, I still don't want an attacker corrupting gmail's heap and causing it to forward my entire inbox to the attacker even if my OS is safe.

There are ways to have a verifier check to make sure that this kind of code modification doesn't happen. It's a very hard problem, though, and maybe not suitable for things like Javascript.

What I'm referring to is not code modification, but heap corruption. The simplest example would be that if the to: and bcc: list for your email live in a wasm heap, any bug in that wasm application could be used to redirect your email. The same is true for JavaScript, C#, etc but the key difference is that those are memory-safe languages under normal circumstances, while WASM is only "memory safe" in that the memory accesses are contained to its sandbox (and part of the stack is outside of the heap and protected from stack overflows, like the return address). Everything else is still wide open for corruption and is easier to manipulate due to predictable function pointers.

The wasm shadow stack is a great idea for mitigating various attacks against C but since you can only put ints and floats on that stack and you can't pass values by reference, the vast majority of applications have a bunch of critical stack data stored in the heap at a fixed address next to non-critical data and a stray memcpy/strcpy can trample a function pointer. Function #37 will always be the same function and its arguments are probably being loaded from nearby locals on the stack.

If support for putting more complex types in arguments and locals ever ships in wasm, it would make it possible to mitigate a lot of this. I'm not sure whether that's in the cards, though.

On Micro-VM's specifically, shout out to Wyrcan[1], which is open source software similar-ish to Fly.io in taking a container image & booting into it. There's no platform here, but it's an advanced & secure bootloader for a container images that looks like the core critical capability furthest under the hood.

Back to isolates, I'd really love to see v8 isolates gain some more whatever it takes for people to be less critical about it's multitenancy. I havent really understood what the criticism is but it's fairly active. Since the process has to do it's owm scheduling of work, just having something like cgroups & resource priority seems like a fairly obvious absense: make sure everyone gets a turn. This scheduling seems semi obvious. But I think the security-minded folk are paniced over a lot more, and likely with legitacy, but the name isolates is afaik somewhat reasonably truthful, that data is fairly secure across isolates in the same process.

Throwing in a bonus ask, it'd be sweet if isolates could migrated. Perhaps snapshotting can already maybe do this? Being able to load manage is important! Even if v8 doesnt want to have vast multi-tenant scheduling capabilities my trivial dumb feel is that just moving aggro processes elsewhere would be a great start to handle aggressive tenant sub-proceeses.

I wish I had links on hand, but one of the things that most opened my horizons on wasm & it's role was considerations that browsers should be able to have multiple instances of a module. Like, if someone depends on a module, does everyone always get the one singleton instance? Can the browser start creating multiple instances if there's a lot of consumers? Very clear browserside question that really opened the floodgates that, oh, there's a lot of ways we could go forward with this!

[1] https://gitlab.com/wyrcan/wyrcan

What is it with these companies who have untold resources, and scores of the smartest engineers, yet what they do is repurpose some existing tech that kinda works instead of building stuff properly? Like selling:

- The browser scripting engine as a PaaS engine

- The VM language/runtime meant for running binaries in the browser as a PaaS engine

- The tooling meant to isolate Linux processes as a VM platform

Can't they build tech properly, for their own purposes?

Not to mention, the whole exercise seems like monkey-patching over the failures of modern OS design. A proper operating system should be able to run processes with zero trust and resource sandboxing.

One thing I learned from being on Microsoft's Windows accessibility team at its peak headcount is that it doesn't matter how big you are; there are always limited resources and competing priorities. By settling for "good enough" on a purely technical level you can focus more resources on things that are more important in the big (i.e. non-technical) picture (edit: like getting the thing shipped at all).

Towards the end of the article it sounds like the author was looking for this: https://github.com/deislabs/wagi

Microsoft Deis Labs folks are the ones who are running the show at Fermyon, which is repeatedly spoken about in the article.

Yup, but they have a lot of different awesome projects on the fire, it can be hard to know which one to use:

- https://github.com/deislabs/runwasi

- https://github.com/deislabs/hippo

- https://github.com/deislabs/wagi

- https://github.com/deislabs/krustlet

The description at the end of the article sounds like CGI, on WASM, which is what WAGI is.

> Most other companies (like Wasmer) seem to be focused on runtimes

Stay tuned. Exciting things coming on the Wasmer horizon :)

The JS / wasm ecosystem is slowly becoming Java / JVM. Wasm has a virtual machine that can run wasm code, which different languages can target with a specialised compiler. It also abstracts away things like memory, storage and networking, just like EJB did.

We can have a better js vm like quickjs or Hermes (from react native).

Does anyone know what Firebase uses for their functions service?

I presume this is the same as Google Cloud functions which use gVisor as mentioned in the article.

The future is here, people just haven’t noticed yet

Reply on news.ycombinator.com