I am on a team that runs database services (mostly Postgres and DuckDB) on an internally maintained illumos branch that runs in VMs on the client's Oxide racks.

Dtrace, Zones, and an "untainted branch" of ZFS are the main reasons given when I asked why illumos and not Linux. I did later see the light (heh) with the Dtrace part for sure.

> Are there any workloads (other than as a VM host) that run on SunOS derived OSes?

Pretty much any workload that runs on Linux or BSD. The exceptions that are notable are Ceph and "big network" applications like XDP/VPP/DPDK centric stuff like edge router or DDoS protection.

Zones provide full security isolation. A downstream user can have root in an illumos Zone and there isn't anything to worry about other than CPU side-channel flaws (which are or are not a problem depending on use case). FreeBSD's Jails, as shown by a 39C3 talk given this winter showed that the FreeBSD kernel is highly vulnerable to processes running as root within a Jail. Security isolation that can be relied on for untrusted workloads in Linux, in the form of containers at least, never really materialized.