Hacker News Reader

Comment by QuantumNomad_ | Hacker News Reader

I remember several years ago, SmartOS was being mentioned many times on HN.

Joyent, the company behind SmartOS, was since acquired, and I don’t usually see anyone talking about SmartOS nowadays.

Is anyone on HN using SmartOS these days?

Running a number of production services on-premise on a big machine using native zones, a few using LX zones (the built-in Linux compatibilty layer), and a single bhyve zone. Actually, years ago this machine was the very first server we set up when our company was just getting started and for the first few years it ran pretty much everything. Zones were ideal for that, also to allow us to pack more services on less hardware, while having decent separation and everything snapshotting/backupping using ZFS. Nowadays we have a bunch more servers, with varying *nix operating systems (SmartOS, Debian, FreeBSD), as well as macOS and even Windows for some specific CI functions. (:

The global zone works great as a hypervisor if you prefer working over SSH in a real shell, and being able to run a lot of services natively just makes things like memory allocation to VM's and having a birds eye view of performance easier. Being able to CoW cp/mv files between zones because it's actually the same filesystem makes certain operations much easier than with actual VM's. Bhyve works well for the things that need an actual Linux kernel or other OS, at the cost of losing some of the zone benefits mentioned earlier.

Highlighting a few things we today run on SmartOS, grouped by their technology stacks: C (haproxy, nginx, PostgreSQL, MariaDB), PHP (various web apps), Java (Keycloak), Elixir/Phoenix (Plausible, fork of Firezone), Rust (rathole, some internal glue services), Go (Grafana, Consul, Prometheus). Most of those are readily available in the package manager, and a few offer native Solaris binaries which run fine on illumos. Others we do local builds in a utility zone before copying the binary package to the where it actually runs.

On LX zones we also run a number of services without problems, usually because they have Debian packaging available but are not in pkgsrc (for example Consul/Nomad, Fabio, some internal things that was already Linux-specific and we haven't bothered to port yet).

And at home a LX zone also runs Jellyfin just fine. (:

I used it for smaller scale (low 10s of physical servers) back in the day also. But my problems with that started when i needed a lot more, the sysadmin/devops/whatever story doesn't scale.

Yes, ansible exists but it's actually quite hard to run ansible on a few hundred machines -- you need lots of RAM just to run the playbook and your first hundred or so separate deployments, you do need to reach for something like Kubernetes.

As for LX, why emulate linux when it's .... right there? The linux kernel is not a lot of overhead vs having to justify emulating the linux ABI on an OS the industry has largely abandoned.

I use it for a home server. Zones provides a secure way to have (on one physical machine with one physical network interface) some stuff you can only get to on the local network and some things you can get to over the public internet and some things via internet if you have the right ssh key. Each contained natively from each other. Crossbow firewall provides a nice way to contain traffic securely as well. ZFS let me set up two big external usb drives as a raid array, the resulting zvol can (iirc) have multiple filesystems for use by multiple zones although I only use it from one right now for the lan only zone. That zone shares via SMB to my network so I can use it for backups and media streaming.

I’ve been able to do almost everything in native zones. I had a bhyve zone set up to run a photo related GitHub code base that really needed Linux.

SMF is a joy to use for services and package management with pkgsrc is great. The whole thing just feels very thoughtfully put together.

You can probably achieve all this on Linux with docker and the right iptables (or whatever succeeded it) config I imagine? But on smartos I am using facilities that are integrated deeply into the os going back like 20 years now. I also just prefer the old sun stuff.

I use it. It's not the most practical I'll admit but I like the simplicity (I'm not super experienced with linux server administration so both of them feel similarly foreign, but SmartOS is pretty minimal and has been pretty straightforward to manage), and I won't lie it's a fun gimmick to be running a descendant of SunOS for random household services.

Home stuff was the last holdout for me, but even that has been replaced by Proxmox these days. I used SmartOS for a solid 7-8 years, though, and like it for most of that time.

I couldn't point to any one single major reason that prompted the switch - just lots of small annoyances stemming from the world expecting you to be running Linux instead of Solaris, and once you move away from zones, you lose one of the most compelling reasons for being on SmartOS

Certainly Oxide computer company has some use of illumos still, which is strongly related to SmartOS

But isn't the end goal with Oxide to run primarily Linux(/Windows?) VMs on an Illumos host?

Are there any workloads (other than as a VM host) that run on SunOS derived OSes?

The whole cloud orchidtration platform and all you need for that.

But that is the same for most server images nowdays.

What in portend is that Oxide upstreams all their work so 'traditional' users should get benefit from it too.

I am on a team that runs database services (mostly Postgres and DuckDB) on an internally maintained illumos branch that runs in VMs on the client's Oxide racks.

Dtrace, Zones, and an "untainted branch" of ZFS are the main reasons given when I asked why illumos and not Linux. I did later see the light (heh) with the Dtrace part for sure.

> Are there any workloads (other than as a VM host) that run on SunOS derived OSes?

Pretty much any workload that runs on Linux or BSD. The exceptions that are notable are Ceph and "big network" applications like XDP/VPP/DPDK centric stuff like edge router or DDoS protection.

Zones provide full security isolation. A downstream user can have root in an illumos Zone and there isn't anything to worry about other than CPU side-channel flaws (which are or are not a problem depending on use case). FreeBSD's Jails, as shown by a 39C3 talk given this winter showed that the FreeBSD kernel is highly vulnerable to processes running as root within a Jail. Security isolation that can be relied on for untrusted workloads in Linux, in the form of containers at least, never really materialized.

SmartOS was Joyent's distribution of illumos, like how RedHat Linux is a distribution of Linux. Oxide's rack-scale compute is powered by Oxide's illumos build named Helios.

  https://rfd.shared.oxide.computer/rfd/0026

  https://github.com/oxidecomputer/helios

I think the big hit on Solaris and FreeBSD popularity since then was OpenZFS adopting Linux as their main reference operating system.

I have a personal box I keep updated running some utility zones and a couple VMs. I enjoy the tooling very much but it's so niche that I'm wary of using it for Customer engagements.

I never used Solaris in my real life but I can understand the appeal for people who did.

It was acquired by Samsung, which is notoriously bad at open source. But the reason why it quietly faded into the background wasn't that. It was that Joyent's ex Sun people had an annoying elitism that made them not care about working with the community.

It was acquired again in 2022 by MNX [1].

[1] https://www.tritondatacenter.com/blog/a-new-chapter-begins-f...

Thanks, I missed that one.

Yeah, but at that point it had lost all the momentum it had, don't you think?

(citation needed)

Many/Some of them are now working at https://oxide.computer

https://oxide.computer/blog/engineering-culture

I mean when you're that far ahead of the class is it any wonder?

In 2014 and 2015 I was able to make some great things happen because of the people (mostly former Sun people) in the #smartos and #illumos IRC channels on Freenode. They were very helpful. Maybe this is because I put actual thought and effort into my questions and didn't waste their time with stupid questions (ones already answered in documentation or found in mailing lists).

  https://www.catb.org/esr/faqs/smart-questions.html

Reply on news.ycombinator.com