Hacker News Reader

Containers can and do remove the "it works on my machine" problem when you're on the same platform, and preferably the same OS exactly. The big deal here at least in my experience is the ability to completely replicate your environment, however you still need the underlying OS, because docker is not a VM.

It sounds to me like you're running a container on a massively different architecture and I think it's still pretty cool that it runs at all. Obviously when trying to run a game, which needs access to hardware and system calls beyond the use case of a simple CLI tool, running the container on a different architecture becomes more of a long shot.

I think given the circumstances it's actually kind of amazing you got this far, and if you're able to go all the way with your game with emulation, that'd be pretty cool.

Docker is a VM on macOS

And on M1/M2 Macs that means an arm64 Linux VM. So whatever issue came up here probably would come up attempting to run amd64 containers on regular arm64 linux (assuming the right compatibility tooling is installed).

> I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles.

Containers aren't VMs, they're Linux namespaces. If you want to run a container you need Linux. If you want to run Linux on MacOS you need virtualization. There's no "lightweight" VM about it - it's still a VM.

Containers are virtualization without the machine part.

When running Linux containers on Mac, the containers run within a full virtual machine.

Docker Desktop on Linux works this way, too.

While the machine part may not matter, the OS certainly does. Linux containers depend on the Linux kernel which is external to the container.

No, they are not. Containers are namespaced process groups (CGroups) with a sandboxed filesystem and (optionally) sandboxed network layer via bridge interfaces and virtual devices.

What is virtualization to you?

https://en.wikipedia.org/wiki/Docker_(software)

> Docker is a set of platform as a service products that use OS-level virtualization

https://www.netapp.com/devops-solutions/what-are-containers

> Containers are a form of operating system virtualization.

On Windows and macOS Docker runs Linux in a VM and then all the containers attach to that VM. I think this has changed to WSL2 on Windows of late but it's still the case on macOS AFAIK.

WSL2 is a Linux VM (albeit with some advanced features such as dynamic memory allocations, GPU & file sharing, etc.) on Windows hypervisor, and Docker for Windows just uses their own WSL2 VM.

It also 'magically' exposes Docker socket to your own WSL2 instances/VMs.

Windows actually has some kind of native containerization but I believe it's based on virtualization and not an equivalent of namespaces on Linux. I'm not even sure if such a feature would make sense on Windows in the same way it does on Linux.

But that's my point. It seems from this thread people don't understand what docker is except that they can kinda use it like a VM. But it's fundamentally different.

It’s worth noting there are different kinds of containers out there, and Docker is just one “ecosystem”. Docker on Windows and macOS runs in a VM, which is kind of a bummer because it means you can’t really allocate all your RAM for containers. By default Docker likes to max out at half the system RAM, which of course you can change post-install, but I feel like there’s a trade off or disadvantage in such a scenario vs Docker (or LXC or podman or something else) running on Linux.

It might be that the RAM is reserved up-front on Windows and macOS but not on Linux? I’m not sure.

Isn't Docker for Mac just using VMs in the background?

It is, and Docker for Mac uses QEMU for `--platform`, just like Podman AFAIK. The issue here is that it's QEMU user-space emulation, which does not support some syscalls, like `ioctl`.

The fix here would be to spin an amd64 VM and run the binary or install Docker there, instead of relying on the more convenient - but less compatible - user-space emulation of Docker. Not very related to this very interesting presentation on virtualizing ARM macOS guests on QEMU hosts..

> Docker for Mac uses QEMU for `--platform`

Only on older setups and on the default config. Updated MacOS (13+) and updated docker provides an option to use Rosetta and the new virtualization system. That, I believe, replaces QEMU for x86 emulation with the Apple emulator.

I've had substantial success with that method; not only does it seem a bit faster, but some containerized apps that very frequently failed to start (e.g. Apache Pulsar) work consistently in that mode. I suspect this difference in experience speaks to bugs in QEMU that trigger in the presence of certain application behavior (in Pulsar's case, something weird happening with the JVM on startup).

QEMU's still an incredible feat of engineering and something I frequently use on Linux, though. It just seems that the Apple emulation on Macs is better (which makes sense, given that it's built by the people who made the semi-proprietary silicon it's emulating from).

Can I ask a question? It seems like your post is combining two separate things. As I understand it and based on earlier comments in the thread:

1. To run a Linux container you need to be running Linux 2. This is why Docker for Max has always run a VM under the covers. It uses qemu. 3. This was true even before Apple Silicon when everything was x86_64 and nothing was emulated

Now you are bringing up Rosetta and x86 emulation. While I understand sometimes it’s necessary to run a container that hasn’t been compiled for aarch64 (or whatever they call Apple) so I can see why the actual cpu emulation capabilities of qemu or Rosetta would be necessary there, this seems to be beside the point.

if you have no Linux system involved because you have no VM, how would that work? What would the base system “under” the container be and what would respond to Linux’s OS-level calls?

This is the core of my challenges:

> sometimes it’s necessary to run a container that hasn’t been compiled for aarch64

I have several containers in that boat, for which no ARM versions exist at all. Those containers were previously using QEMU to emulate x86 on ARM linux (in Docker's VM on an ARM Mac). That emulation encountered the failures I described, and was slow.

Switching to use Rosetta 2 in Docker solved the problems; the process for enabling it is described here: https://levelup.gitconnected.com/docker-on-apple-silicon-mac...

Something I don't know and am curious about is how Docker-for-Mac is actually using Rosetta 2. Is it running an additional Linux VM containing an x86 Linux OS, and running that VM through a Rosetta-2-enabled hypervisor? Or is Rosetta 2 distributed as a Linux program that is being invoked inside Docker's pre-existing aarch64 Linux VM instead of (or inside of?) QEMU?

Edit: as for your question:

> if you have no Linux system involved because you have no VM, how would that work?

That's not my situation, so I'm not sure.

Yes, Rosetta 2 is being used by Docker’s pre-existing aarch64 Linux VM. So it’s only translating userspace x86 code to Arm, then running it on the ARM Linux VM. Syscalls, etc. are still handled by the (aarch64) Linux VM.

> I thought it ironic recalling how many people had sold containers as lighter weight VMs

That's your problem there, containers are by design not VMs. Anyone who sold them as such aren't understanding the tech and you shouldn't have blindly trusted them.

This should work, since it spins up a full VM to run your container.

Why not just use native Amd64 Linux machine for development? Using emulators and syscall debugger is not very productive experience!

That’s kinda the conclusion I came to. Being apart of the bleeding edge is fun in some senses but I need to get moving quicker.

That’s why I just said I’d spin up an EC2 instance that will 100% run on amd64 cpu arch.

I’ll be exploring vagrant but if it too fails I have workarounds.

That said I’m not really planning on hauling two computers with me just in case I want to test something on an intel processor.

Even when the arch matched, macOS was a bad platform for running containers. I'm not even sure you can call them "edge cases" because it was basic things like volumes not working right.

macOS is engineered for users and desktops, not servers and development. Apple's long history of blocking virtualizing macOS is a testament to this.

MacOS has been virtualizable in x86 since mac-on-x86 was released.

It also is engineered for development. Maybe not your particular type of development. Apple silicon isn't technically development-unfriendly, it's just apple silicon makes it less easy to do x86 development. But so does mips, sparc, ppc, and other architectures.

Do you understand the limits apple put on virtualization? Are you aware of how it's handled for cloud providers?

I didn't say it was technically impossible. I said apple has a history of blocking it.

maximum of two virtual machines running MacOS on a single Apple device, no matter how it is configured.

this is in the EULA and is enforced by Virtualization.Framework.

cloud providers get no exception to this.

MacOS can be simultaneously "engineered for development" and "a worse container host than a chromebook". They aren't mutually exclusive.