Hacker News Reader

Comment by AceJohnny2 | Hacker News Reader

How does this work?

Fundamentally, containers are about namespace/isolation of a bunch of OS interfaces, so file system functions, network functions, memory management, process functions, etc, can all pretend like they're the only game in town, but crucially without having to virtualize out the kernel.

Does XNU have such namespacing functionality across all its interfaces?

Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.

The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out: they actually run a Linux virtual machine (using Hypervisor.framework/hvf), and have that just provide the container environment.

Is that what this project is doing? But then, how could it run a macOS container?

[1] based on the foundation that Linux, unlike BSDs, has a stable syscall interface!

View on HN

This introduces a long-standing problem that is really the main issue I have with Docker/Podman/containers on anything other than Linux - you have a dedicated VM that needs memory to be set aside for it exclusively.

This is fine if you have a 32/64GB machine, but less so on an 8GB non-upgradeable laptop.

I get it - memory is relatively cheap these days - and manufacturers that are building memory-limited devices are really only doing it to fleece you on obscene upgrade fees at the time of purchase - but it would be nice if there was a more elegant solution to this on Windows and macOS.

WSL 1 had a solution to this that clearly took a lot of work to put together, wherein they'd have a Linux kernel running side-by-side as a Windows sub-process so that the memory pool was shared. Unfortunately it might have been too much work as they scrapped it entirely for WSL 2 and just used essentially the same VM route.

If anyone knows of any projects trying to work around that problem I'd love to hear about it. If Apple really wanted to bring the development community back on board, focusing on these kind of use cases would be great, sadly it seems someone over there has taken the view that scrapping butterfly keys and the touchbar is "enough".

Say what you will about Microsoft, but they've focused really hard on developer use cases for decades, and it shows.

VM dynamic allocation of ram has been a standard feature of KVM, hyper-v and VMware for the better part of 10 years now.

However WSL for Docker works differently depending on the containers configuration.

When Windows containers are being used, it is just there to keep the docker daemon happy.

> If anyone knows of any projects trying to work around that problem I'd love to hear about it.

Containers are namespaced processes. These processes exec against the corresponding kernel they require. There is no workaround: if you have an ELF binary calling Linux syscalls it can only run on a Linux kernel†, so to run that you need a VM††. It's not as bad as it appears thanks to memory ballooning†††.

Conversely if you want to exec a Windows binary in a container, the Windows kernel needs to provide process namespacing features (which it does). And if you want to exec a Darwin binary in a container, then the Darwin kernel needs to provide process namespacing features (which it doesn't).

† WSL1 was implementing the Linux syscall API on the Windows kernel, which proved to be much more complex than it appears to be.

†† Or colinux (https://en.wikipedia.org/wiki/Cooperative_Linux), or user-mode Linux (https://en.wikipedia.org/wiki/User-mode_Linux).

††† https://en.wikipedia.org/wiki/Memory_ballooning

I suppose running a windows binary under wine in a Linux container also belong somewhere on this excellent summary.

> running a x86 windows binary - looking for 32bit stuff under SysWOW64 - under x86_64 wine in a x86_64 Linux container AOT translated to aarch64 by Rosetta 2 for Linux that switched the ARM M1 memory model to Total Store Ordering on a aarch64 Linux kernel under Virtualisation.framework on arm64 darwin.

FTFY ;) (and thanks!)

The challenge of running OCI containers in every OS is a bit similar to the problem of running the same binary in every OS.

I think the only true solutions are (a) OS vendors develop their own native container platforms with UX similar or better than Docker (b) OS vendors agree on some common ABI standard

> WSL1 was implementing the Linux syscall API on the Windows kernel, which proved to be much more complex than it appears to be.

I've long wondered, and again now that Chat GPT is proving so adept at coding, if this translation layer could be automated. Do humans actually have to hand-code each syscall? Or are there just enough edge conditions that can't be automated?

The hard part isn't so much writing the code as it is testing the code. The WINE server is decades old, porting API calls to another kernel isn't an entirely new or novel process. Getting all the software to work right is an ongoing process though, and it requires careful deliberation over what works and what doesn't. WINE code is still being refined to this day in response to new titles and old bugfixes.

So, I won't preclude the idea of AI helping, but I think human effort is still the bottleneck for projects like this. Even if AI could write perfect code 100% of the time, testing and troubleshooting would probably still be the larger timesink.

WSL 2 now allows for memory reclaim from the VM, essentially fixing this

    autoMemoryReclaim – Makes the WSL VM shrink in memory
    as you use it by reclaiming cached memory

https://devblogs.microsoft.com/commandline/windows-subsystem...

I think someone told me that the relies on a feature of the CPU itself and MS is the only OS vendor to actually implement it

This is supposed to be possible on macOS, but it's basically completely broken (just doesn't work) due to a bug. Apple is aware of it but unfortunately hasn't said much more.

Obviously that limits the options, but I'll still be taking one last shot at using creative workarounds to tackle the memory problem in OrbStack (another containers-on-macOS product).

WSL1 didn't run Linux kernel anywhere.

One of my issues too. The tools/ecosystem are Linux driven and doing anything on Windows requires a Linux 'stub' of some sort (VM, WSL, etc). I am comfortable with both OS's but all of my coworkers are Microsoft OS tied. 0% Linux experience. Cannot really introduce a managed kub/docker/containers into the picture without ending up being 'support' - not happening! The cloud is an option but that still leaves the developer experience (local laptop) short changed.

Edit: It has been a while since I last looked at this. Looks like containerd is, perhaps, a native option

Recent versions of Docker Desktop have a "Resource saver"[0] mode. It will de-allocate memory and CPU when containers aren't being used. If there's a leaf icon next to your Docker icon then it's in resource saver mode.

I imagine over time it will get smarter too. Right now it waits for no containers to be running for 30 seconds and enables resource saving mode but who knows what could happen in the future. Maybe it can internally profile and estimate load based on evaluating runtime stats of your contains and dynamically change the VM's resources on the fly and then expose a +% over provision threshold option or a way to turn off dynamic resource saver mode.

[0]: https://www.docker.com/blog/docker-desktop-4-22

WSL2 doesn't really use VMs in the traditional sense. msft have invested heavily in lightweight virtualisation (eg better memory management) for certain security features, which allowed WSL to perform well enough without the maintenance/support overhead of WSL1

The transitively linked docs have some answers:

  rund is an experimental containerd shim for running macOS containers on macOS.

  rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API.

  What rund provides:
    - Filesystem isolation via chroot(2)
    - Cleanup of container processes using process group
    - OCI Runtime Specification compatibility (to the extent it is possible on macOS)
    - Host-network mode only
    - bind mounts

So essentially a chroot with a bit of make-up and a lot of marketing?

Except for bind mounts (not even overlayfs...) there isn't much interesting.

> - Host-network mode only

Yeah expect a lot of things to break in subtle ways... most containers are developed kinda expecting you have your own network namespace (and that no one else is using ports)

overlayfs and bind mounts are orthogonal:

- bind mounting solves the exposition of filesystem within the root pivot.

- overlayfs solves the persistence efficiency issue using a layered union fs.

> most containers are developed

Most Linux (and Windows) containers. Since these are macOS containers there are no containers developed yet so by definition there is nothing to break.

Original author here.

> So essentially a chroot with a bit of make-up

Well.

1. It is not trivial to properly set up a chroot on macOS. If you try to find a working guide/tool that works with modern macOS, I doubt you'll find anything (at least, I failed, even though tried very hard) 2. I believe that ability to package stuff into a Docker image distributable via already existing infrastructure and compatible with already existing tools maybe "a bit of make-up", but it is an important makeup. 3. Kubernetes recently got HostProcesses for Windows: https://kubernetes.io/blog/2022/12/13/windows-host-process-c.... They are even less isolated from host than chroot and still, people find them useful for certain scenarios.

> and a lot of marketing?

Thanks for "a lot of marketing", that made me chuckle. My own submission got buried yesterday with humble 8 points: https://news.ycombinator.com/item?id=37640688

Great effort. I get why you call it container - but sounds more like jail or cheroot would give more appropriate expectations; like "tooling to build and run Darwin containers in a macOS chroot"?

I didn't want to use "jail" term because it is mostly unheard of outside of FreeBSD.

Container definition is very stretched nowadays. Look at Windows HostProcesses in Kubernetes [1]. They don't have neither process, network nor device isolation from the host.

I also plan to try macOS sandbox-exec tool, which should offer additional isolation from the host.

[1]: https://kubernetes.io/blog/2022/12/13/windows-host-process-c...

If the parent process of the container here changes its bootstrap port to itself or disinherits it then it could also create an isolated mach namespace, restricting access to mach/XPC services.

> bind mounts via MacFuse

> How does this work?

Essentially, like this:

https://ericchiang.github.io/post/containers-from-scratch/

https://earthly.dev/blog/chroot/

> The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out

It's not, it's a requirement for running Linux containers: https://news.ycombinator.com/item?id=37656401

DfM is more like running the CLI locally to a remote Linux machine, and all it does is conveniently expose /Users in the same place through the VM folder share so that you have the convenient illusion that it happens locally.

If Darwin had process namespacing features it would not make it magically able to run Linux processes.

> Does XNU have such namespacing functionality across all its interfaces?

I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK

> Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.

This project is about running macOS containers on macOS. It's not about running Linux containers.

> > Does XNU have such namespacing functionality across all its interfaces?

> I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK

Theoretically, probably, for coarse-grained yes/no things? I don't think it's able to go much further than "you can use the local network and/or internet" and "you can read/write to the filesystem location corresponding to you bundle identifier `com.foo.bar`" but not "hey let me present you with a namespaced view of loopback or process list".

Also not sure if it can be dynamically set by a parent process for a child? Seems like it's very bundle oriented (except maybe for Apple processes) so not very practical.

> Also not sure if it can be dynamically set by a parent process for a child?

Yes, it can. See sandbox-exec tool. And I actually plan to use it: https://github.com/macOScontainers/rund/issues/15

Oh my I totally forgot about sandbox-exec!

I played with it some time ago, can't recall the context but it was about build systems / packaging (maybe nix?), doing the configure/make/make install with reduced privileges.

> Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.

There is more to the container ecosystem than Linux containers; Windows native containers function much the same way (well, in two ways, with VM-backing or the traditional kernel syscall interface, but with Windows syscalls).

I am also interested in the API boundary. It seems that there are too options:

1. Rely on system call stability. This is like Linux containers but unlike Linux macOS doesn't provide a stable system call API. So this would break when the system updates with a change that modifies the system call API.

2. Install the host libraries into the container at runtime. This should provide as much stability as macOS apps usually have. It may also be beneficial as you wouldn't be embedding these into every container.

It seems like 2 would be preferable. However it may be a bit weird when building as the libraries you build against would be updated without the container being aware, but this is unlikely to break anything unless they are copying them to new paths which seems unlikely.

> macOS doesn't provide a stable system call API

I'm really wondering, do you have any links about macOS syscall stability over versions?

I do not. I'm not really a macOS user but have heard this mentioned many times. I remember one recent example is Golang switched from doing raw syscalls to using the macOS libc due to this issue. That was probably ~5 years ago now so you may be able to dig up examples there.

Generally speaking macOS does not guarantee syscall stability, and does not generally guarantee compatibility for any binaries not linked to `libSystem.dylib` (that is the supported ABI boundary)[1]. This has a number of implications, including (but not limited to):

* The most obvious is the commonly mentioned fact that syscalls may change. Here is an example where golang program broke because they were directly using the `gettimeofday()` syscalls[2].

* The interface between the kernel and the dynamic linker (which is required since ABI stability for statically linked executables is not guaranteed) is private and may change between versions. That means if your chroot contains a `dyld` from an OS version that is not the same as the host kernel it may not work.

* The format of the dyld shared cache changes most releases, which means you can't just use the old dyld that matches the host kernel in your chroot because it may not work with the dyld shared cache for the OS you are trying to run in the chroot.

* The system maintains a number of security policies around platform binaries, and those binaries are enumerated as part of the static trust cache[3]. Depending on what you are doing and what permissions it needs you may not be able to even run the system binaries from another release of macOS.

In practice you can often get away with a slight skew (~1 year), but you can rarely get away with skews of more than 2-3 years.

[1]: https://developer.apple.com/library/archive/qa/qa1118/_index...

[2]: https://github.com/golang/go/issues/16606

[3]: https://support.apple.com/guide/security/trust-caches-sec7d3...

https://github.com/macOScontainers/rund/blob/main/README.ado...

Reply on news.ycombinator.com