1. Hardware design flaws are not irremediable and will be fixed in time.
2. It's still very useful to run software that is trusted to not be malicious and/but not trusted to be void of memory bugs.
There's no particular reason to believe this. Neither Intel nor AMD have committed to any form of side-effect-free speculative execution or similar. That'd required a rather large chunk of transistors and die space, and if nobody is footing the bill for it it's not going to happen.
Preventing side-effects from leaking between processes (and ring levels) entirely at the hardware level is definitely going to happen. Within a process, though? That's not going to happen without multiple major players demanding it. Since nearly all the major CPU consumers are currently happy with process boundaries being the security enforcement zones, there's no particular reason to believe that in-process sandboxing will ever have hardware fixes to prevent spectre attacks.
I do not think there is the degree of distinction between "processes" and "threads" that you think there is. If tools exist to isolate speculation state between processes, those tools can probably also isolate threads in a process.
The amount of untrusted code running in browser is only increasing over time, and the CPU vendors have absolutely noticed this. That's why ARMv8.3A added an instruction that literally has JavaScript in it's name (FJCVTZS) for instance.
Once you have mitigations for colocating trusted and untrusted code in ring 3, doing the same in ring 0 almost certainly isn't a big deal.
In-process sandboxing by all appearances is simply dead.
As for "doing the same in ring 0 almost certainly isn't a big deal" no, very extremely no. If you let code run in ring 0 it has everything ring 0 can do, period. You cannot put restrictions on it, that's what spectre proved. Give code access to a process and it has entire access to that process. Similarly give something ring 0, and it has the entirety of ring 0.
Untrusted code goes in ring 3 in an isolated process. That's the security model of x86, and it's the only model that CPU vendors have any pressure to fix.
Chip vendors have & will fix process boundaries. But nobody is talking about any sort of protection of any kind that would let in-process sandboxing work again. It's just not even on the table at this point.
The first attack isn't relevant to designs that don't use hardware isolation, but the second one absolutely is. If your virtual bytecode (wasm, JVM, Lua, whatever) is allowed access to a portion of memory, and inside the same hardware address space is other memory it shouldn't read (e.g., because there are two software-isolated processes in the same hardware address space), and a supervisor or JIT is guarding its memory accesses with branches, the second attack will let the software-isolated process execute cache timing attacks against the data on the wrong side of the branch.
(I believe the names are more-or-less that Meltdown is the first bug class and Spectre is the second, but the Spectre versions are rather different in characteristics - in particular I believe that Spectre v1 affects software-isolation-only systems and Spectre v2 less so. But the names confuse me.)
Now I'm wondering how browsers presumably already cope with this for JS, or how CloudFlare workers cope with it, or .. etc.
CPU speculates past the bounds check and loads values you should not read into cache and you try to find out which value was loaded into cache using precise timing but the timer is imprecise so you don't know what the value was which you were not supposed to be able to read.
Browsers just resort to process sandboxing entirely. They assume JS can escape its sandbox, but since it can only read contents that were produced from its origin anyway it doesn't really matter.
Xerox Parc workstations, IBM and Unisys mainframes, UCSD Pascal, Oberon, Inferno, Java, .NET, Flash.
Or for more actual examples, Garmin Apps, watchOS bitcode, DEX on Android.
System/360 had (and used) hardware privilege levels.
Seems like few of your examples involve not using hardware for process isolation. Java: no, except a few research OSes. Flash: no.
AS/400 though does use hardware heavily in it's isolation model, going so far as to have a custom PowerPC variant currently that adds tagged memory.
And yes I call them mainframes, because it is as I always heard people referring to them during my Summer job back in the day, so the name stuck with me even it isn't correct.
IBM System/38, evolved into AS/400 and IBM i, is one of architectures described in this book:
https://homes.cs.washington.edu/~levy/capabook/
Far as language-based security, the first mainframe for businesses used a high-level language combined with a CPU that dynamically checked the programs. Still sold by Unisys but I doubt hardware checks still exist.
http://www.smecc.org/The%20Architecture%20%20of%20the%20Burr...
The Flex Machine implemented capabilities and trusted procedures in the microcode:
https://en.wikipedia.org/wiki/Flex_machine
ASOS supported a mix of methods where each app was Ada for its safety features but a MLS kernel modeled in Gypsy separated various security levels:
https://pdfs.semanticscholar.org/42a4/e6f812538c18461277c58a...
SAFE explored tagging at CPU level which got commercialized as CoreGuard or Inherently Secure Processor:
http://www.crash-safe.org/papers.html
https://www.draper.com/explore-solutions/inherently-secure-p...
In embedded, there's Java processors that run bytecode natively with some support for separation. They blur the line between VM's and native apps:
http://ajile.com/index.php?option=com_content&view=article&i...
Going through your list:
Xerox PARC worstations: these didn't really run smalltalk in microcode, but ran an interpreter written Data General Nova asm, with the microcode "emulator task" dispatching and executing Nova machine code. There were a few new instructions added for smalltalk, but that was stuff like bit blit instructions. All that being said, even if the smalltalk VM interpreter was pushed down into microcode, it'd still be (a component of) a VM.
The big iron JITing environments are quintessential hardware/software codesigned VMs.
Oberon the language you might have a point, but the secure loader/verifier/compiler in it's only implementation is absolutely a VM.
Java with ART is absolutely a VM, unless you're going to make the argument that HotSpot isn't a VM.
.Net UWP is absolutely a VM too. Yes, it's partially compiled before it reaches end users, but also includes the entirety of .Net core linked in for cases where you're dynamiclly adding new code to your're running process.
Essientially, I think you've come up with some weird definition of VM that doesn't match industry or academia, and then berating people who don't follow that non standard definition.
Dalvik on Android is not a software isolation mechanism. Each app has its own Dalvik VM using traditional UNIX processes and user accounts for isolation, which are in turn powered by hardware isolation. (I'm betting they did this because they knew that JVM software isolation had been a disaster in practice, although in part this also means apps can link native code libraries without changing the security model at all.)
ART on Android 5 and 6 is as much runtime as any other programming language with a AOT compilation model.
And as of Android 7, there are multiple execution modes. An hand optimized interpreter written in Assembly, a JIT compiler with PGO feedback, and an AOT compiler that takes the JIT PGO data to generate a proper executable on when the device is charging and idle.
Traditional native code is also pretty much clamped down in recent versions of Android via SELinux, seccomp and white list of shared objects.
Google doesn't want you to do more than just implementing native methods, high performance 3D graphics, real time audio or importing "legacy" libraries.
I don't consider WASM that sound because it still allows for internal data corruption of modules written in unsafe languages, instead of supporting proper memory tagging like SPARC and the upcoming ARM architecture.
I think the parent comment's point about Android application isolation being enforced by hardware rather than software still stands.
The promise of the JVM failed not because of design, but because of buggy implementations. The drive for features and performance brought risk. And WASM is heading down the same path. WASM can add GC with an unimpeachable proof of design safety, but it would be irrelevant.
The issue isn't design or architecture. The issue is that the motivation for adding GC and other features to WASM is principally for performance. Version 1.0 of WASM will be the last specification that made security paramount. Everything after that will be a managed retreat as more and more features and code are placed outside the sandbox in the endless pursuit of performance.
To reiterate: Java applets weren't insecure because the JVM was insecure. Java applets were insecure because the vast majority of the implementation of the environment--and particularly the most complex aspects--existed outside the confines of the sandbox. The more feature rich and performant the sandbox environment then necessarily the more code and complexity must exist outside the sandbox.
ActiveX, Silverlight, and (worst of all) Flash were far more insecure, both in design and implementation. But that's a distinction without a difference; their relative inferiority didn't make Java applets more viable.
WASM will be insecure for all the same reasons: JITs and GCs are tremendously complex beasts, with implementations least amenable to verification methods relative to most other software projects. Rust's borrow checker is worthless for ensuring memory ordering and life cycle invariants of machine objects. DOM implementations have similarly become tremendously complex, yet the focus is on how to expose these implementations and their interfaces in entirely novel and brittle ways, prioritizing performance above all else. The most safe alternative and likely sufficient for 80% of use cases would be a message passing interface, afterall. Instead, design proposals are focused on finding the thinnest possible abstraction over direct addressing of DOM object references from within the VM. Thus the prioritization of GC.
Well said. That happened everywhere from desktop to server to embedded. However, safety-critical side of embedded further supported your point by building real-time, safe implementations of JVM (or subset) designed for certification. There were also companies using Ada to get systematic protections against errors with one, Praxis, doing semi-automated proofs with their SPARK language. A JVM implemented in Ada, SPARK, and (where necessary) C/C++ might have been much safer.
http://www.aonix.com/pdf/pico-web.pdf
https://www.adacore.com/books/safe-and-secure-software
"JITs and GCs are tremendously complex beasts, with implementations least amenable to verification methods relative to most other software projects. "
There's actually verified JIT's and GC's they can draw on. I doubt they will, though. History shows they'll go with a non-verified design followed by penetrate and patch.
"Rust's borrow checker is worthless for ensuring memory ordering and life cycle invariants of machine objects."
I've long pushed Abstract, State Machines (or languages based on them) to do this kind of stuff better. The work on memory models can be ported to something like Asmeta. The algorithms can be checked against them by solvers. Then, equivalent software and/or hardware comes out the code generator with more analysis/tests in case it messed up. I got excited seeing Galois was using ASM's for hardware/software verification recently. They'd be great for checking security of interpreters against software and hardware level issues.
"The most safe alternative and likely sufficient for 80% of use cases would be a message passing interface, afterall."
I haven't updated myself yet on advances in typing for that stuff. Pony's method for type checking might help here since it uses a capability-secure, actor model. Wallaroo also uses it for a high-performance database. So, it's not a slouch either.
I do not think that "JITs and GCs are tremendously complex beasts" is a meaningful argument against wasm, because plain old JavaScript on the web has JITs and GCs, and is overall a much more secure environment to run than Java. (For instance, approximately every enterprise disables Java for security reasons except for people with a business need for it, and approximately no enterprise disables JavaScript for security reasons. I'm not claiming JS is foolproof, just that it's been much much more secure in practice.) You might note that Sing#, where this subthread started, had a GC. So does Inferno's virtual machine. If you think GCs make things too complex for software fault isolation to work, do you believe Singularity and Inferno are insecure?
I don't claim to be an expert in this, but I think there's a coherent reason why Java's design failed: Java attempted to do isolation at the language level (and inside a language-specific bytecode), which is a richer interface. It should be entirely possible to develop a high-performance interface for software fault isolation as long as it's a small enough interface to successfully secure, and my sense is that that's where wasm is going.
One important advantage of wasm over Java, Flash, and Silverlight is that it can learn from their failures.
Message-passing doesn't need to be slow either - just write your messages in a high-performance but securely parseable format like Cap'n Proto. We know how to do such things now; we didn't 20 years ago. The state of the world keeps advancing.