You might be interested in this paper: "Unix, Plan 9 and the Lurking Smalltalk" (https://www.humprog.org/~stephen/research/papers/kell19unix-...)

Cuis Smalltalk and related implementations are rather self-contained systems to the point they seemed walled off from the rest of the system, making it difficult to develop Smalltalk programs using external tools.

However, there's something compelling about the idea of a Smalltalk (or Lisp) OS running on bare hardware, where everything runs in a single address space. I've been thinking about this for a few years, but I haven't had time to pursue these ideas. Some ideas from the 1994 paper "Sharing and Protection in a Single-Address-Space Operating System" (https://homes.cs.washington.edu/~levy/opal.pdf) could be applicable to add some security to a Smalltalk OS.