LXD containers on macOS at near-native speeds
beringresearch.github.io
beringresearch.github.io
""" The goal of this project is to enable MacOS users to:
Easily spin up and manage lightweight Alpine Linux environments. Use tiny VMs to take advantage of containerisation technologies, including LXD and Docker. Build and test software on x86_64 and aarch64 systems """
You get all the downsides of a VM (persistent wasted RAM/disk space, awful disk IO performance, etc) AND all the downsides of containers (difficult to introspect when your minimal image has no shell, janky networking, etc).
On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
What on Earth are you talking about? arm64 Docker images predate the emergence of M1, and their number has proliferated in the last two years like there was no tomorrow. Here is an excerpt from my own tiny, non-representative collection of Docker images I have had to deal with lately:
$ (for i in `docker images | tail -n +2 | awk '{ printf "%s:%s\n", $1, $2 }'`; do docker inspect $i | jq '.[].Architecture'; done) | grep -c amd64 $ (for i in `docker images | tail -n +2 | awk '{ printf "%s:%s\n", $1, $2 }'`; do docker inspect $i | jq '.[].Architecture'; done) | grep -c arm64Most of the 23 arm64 Docker images is pretty obscure stuff, with one or two manually rebuilt for the arm64 architecture. Yet, I have downloaded all of my obscure images directly from the Docker Hub – not traded them for a pinch of spice from a shady Docker image dealer at a drench inducing corner of the dark web. All of the arm64 images run at the native speed, locally and in the cloud.
The only reason I store x86 (i.e. «amd64») images locally is my laptop being a transient transit point between their source and the destination. I would have rebuilt them as well but the build process is either broken for each of them or is too convoluted to justify spending a time trying to fix it.
Nope, the whole cloud infrastructure does not need to switch.
Since containers are fully self-contained, self-sufficient and isolated from each other, they can coexist peacefully and run alongside each other in a mixed setup in AWS ECS/Fargate clusters.
And they do. AWS Lambda also supports aarch64, and the degree of isolation for the lambdas is even higher and they can also run out of a x86 or an aarch64 Docker image (if required), and 30-40% cheaper with performance loss incurred for the workloads I am interested in.
Should read: «… with no performance loss incurred for the workloads I am interested in». A typo crept in.
But it's hard to not see this as evidence for my point when you were only able to rebuild "one or two" of the 8-9 amd64 images you're running without significant time investment.
By your own stats roughly one third of the images you use don't support ARM. And if you want to run one of those on ARM - you're likely screwed!
Even some _official_ images don't support ARM, random example: https://hub.docker.com/_/percona
Docker ARM support is improving, but acting like it's ready for prime time is laughable. Try doing your day job with no amd64 containers for any reason for a month and come back to me.
> On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
The statement is false. x86 containers do not use the translation, hence they can't run on M1. People have been successfully running aarch64 (ARM) containers, including on AWS Graviton (re:Invent 2018) and on Graviton2/3 (more recently), for a few years now. Oracle cloud have been offering Ampere cloud compute resources for… a couple of years now?
But you have conveniently dodged it, for it did not suit your agenda and presented it as «exaggerating slightly». It is not slight and it is not exaggeration, it is a blantantly distorted and also a likely ignorant statement.
> But it's hard to not see this as evidence for my point when you were only able to rebuild "one or two" of the 8-9 amd64 images you're running without significant time investment.
> By your own stats roughly one third of the images you use don't support ARM. And if you want to run one of those on ARM - you're likely screwed!
Whether one is screwed or not is a matter of perspective (I am certainly not the one), just as 23 aarch64 images are hardly «were only able to rebuild "one or two"». I have revised my 9 locally stored x86 images, thanks to you, and purged one stray one, with the remaining ones being, in fact, an images itself, and an Docker image tag slapped on it. Therefore, I only have 4x remaining x86 Docker images, but of course it is not going to convince you and you will resort to a distorted narrative again.
> Docker ARM support is improving, but acting like it's ready for prime time is laughable.
«Docker for ARM» means «Docker running on Linux/ARM». Docker has supported Linux, and Linux has support aarch64 for a long time now. Solutions I design for my clients have been running as/in aarch64 containers for a few years now, and my clients indeed now find it laughable how they could not switch to ARM sooner due to 30-40% lesser cloud compute bills. Most solutions do not require neither containers nor servers, though, it is only the platform related stuff.
> Try doing your day job with no amd64 containers for any reason for a month and come back to me.
If you don't understand the difference between the aarch64, and that the Apple instantiation of the aarch64 architecture is not the only one, then you it is unlikely that you understand what Docker is and what Docker is not. Therefore, it will be difficult for you to self-assess and quantify the magnitude of your own uproarious laughter. We are at a point now when the CPU has become a configuration setting at the infrastructure provisioning time, and is not an insurmountable barrier.
All of the containers I need to run for my current needs, run, and they do it perfectly well in mix and match Fargate container clusters. Most containers are Graviton2 containers, with a few select ones (the ones that are cumbersome to rebuild), run in AWS x86 containers, and both types coexist peacefully. Last but not least, I do not even run most containers locally, I occasionally help developers with build related problems or to cobble a POC together to verify an idea, evolve it into a solution or a product, and scrap the container setup. Everything else, including production workloads, runs in the cloud, and it just works 24x7.
Sorry, let me admit I actually don't own an M1/M2 Mac and have only used one by proxy helping my coworkers. To be completely honest, I just assumed it was possible to run amd64 containers on them through some form of translation (QEMU/Rosetta).
That actually makes my whole point stronger though - if the container you want doesn't support amd64, you either fix it yourself (often a time consuming and difficult process) or you just can't use it at all.
The rest of your post is just chasing ghosts. We're talking about M1 Macs, remember, not cloud servers? Thanks for letting me know about Graviton, but I am literally employed by AWS and build aarch64 containers as part of my day job, I'm well aware.
In the end, it'll get better... I'm happier with WSL than Mac, even if I personally prefer straight Linux.
Your information is pretty out of date.
I'm getting cranky about arguments against VMs.
1) I have had more RAM than I ever use for about 25 years. I upgrade about every 4 to 6 years. Is everyone else writing code on Amigas?
2) I keep getting told by everyone (IT, devops, DBA, my boss, his boss) that disk space is cheap.
3) VirtualBox has been consistently ok for about 10 years and I almost always can outrun scripts in a Linux VM on my Windows machine compared to our production AWS instance running K8s. Of course my machine would never keep up with a high user count but this is about disk performance, and locally I can prove both my monolith and my VM are faster.
I don't see how arguments like these are good anymore with the modern hardware of a local dev system. That makes containers an over complication. The only reason to do them is to save devops the headache of having to create a docker file. I mean the guys read books and watch YouTube while I'm writing code, so...they can write the damn docker file.
I'm not a huge fan of VM development, generally speaking... that said, my current work setup is Windows + WSL + Rancher Desktop + VS Code (remoting extensions) for nearly everything. My windows terminal default is set to WSL, so it's pretty close to in the box... I can launch windows apps from the wsl environment... the integration is pretty seamless. In terms of performance, if you're stuck on an older Windows version (I am), then you can limit the max memory that the WSL environment gets, similar to a VM... I usually reserve 8-12gb for my host environment and give the rest to my dev environment.
My personal desktop is Linux (Ubuntu-Budgie). My previous job the laptop was an M1 Max, which had faster ram/disk than the windows laptop for the current job, or my personal desktop (which is impressive to me).
In the end, for most things, it's not significant enough to worry about... The aarch macs and i/o have been more problematic than wsl and other x86 environments... I think as rosetta support improves with qemu and docker, it will change and get significantly better.
That said, with VS Code's remoting extensions, I can be working on a remote system nearly as effectively as local anyway... which is how I typically use my laptop. Wireguard to home, vs code w/ ssh to my personal desktop.
Yes you are.
https://github.com/moby/hyperkit
(Or maybe you mean a non-Docker For Mac VM)
The aarch64 -> x86_64 virtualization performance is pretty rough.
Like, to the point where compiling a small hobbyist Rust project that needs to be deployed on Linux x86_64 in the cloud (cheap VPS)... you might as well just build it in the cloud and not locally on Mac host because it takes 10 minutes instead of 10 seconds
I’d love to containerize all of my local development efforts (scripts and rails apps) but the slow ass filesystem always ruined it in the past.
I think the new virtualization framework just came out of beta though so now it’s just VirtuoFS that needs to be enabled.
They all use the underlying OS hypervisor so I’m not sure you can get perceivable performance out of one solution over the other.
What do you mean? Multipass only runs Ubuntu server cloud images.
I believe multipass uses SSHFS (https://discourse.ubuntu.com/t/how-to-improve-mounts-perform...) to mount filesystems between the host and the VM. Performance has been excellent.
Lately I’ve been trying https://devenv.sh/ and it works great!
I haven’t tried it for ruby, though I have used vanilla nix shell for ruby projects before and it worked quite well after I over-rode GEM_PATH and GEM_HOME to the correct values.
nix file helps you build OCI image and then you run it not locally/somewhere else?
can that be compared apples to apples to docker where you can build with Dockerfile syntax, and then run locally the image?
(Aside: you can build oci images using nix, but that’s probably what you would use for building deployable artifacts for production, not for development).
I would not say it is comparable with a Dockerfile - a Dockerfile is closer to a shell script, with caching between each step. Credit to Dockerfile syntax for being very easy to grok, but it is inherently not reproducible.
The perceived reliability of docker for development probably comes from the fact that you can share a prebuilt image, but rebuilding might not always work.
Can devenv environment put itself in a chroot / sandbox where it cannot access my files, and is firewalled by default?
Like Mitchell Hashimoto (Hashicorp), I no longer do any development on macOS directly; I have a Linux VM where all my projects live and have their environments managed with a shell.nix file. Highly highly recommend it.
shell.nix - this brings the packages I need to develop into the $PATH when I run `nix-shell` in the repo
{pkgs ? import <nixpkgs> {}}:
with pkgs;
mkShell {
buildInputs = [
meilisearch
openssl
overmind
pkg-config
postgresql
rustup
tmux
];
shellHook = ''
export OVERMIND_PROCFILE=Procfile.dev
export PGDATA="data.pg"
export MEILI_DB_PATH="data.ms"
'';
}
postgres: postgres -K /tmp
meilisearch: meilisearch
That's pretty much it - with all the background services running you can load your monolith or microservices with cargo/npm/etc. It's all very nice and clean!
[1]: https://lazamar.co.uk/nix-versions/ - I actually just used this in a new shell.nix for my personal website to pin a version of Hugo from 2019 (when I last touched it) so that it would build again
Make sure that you're running a recent version of Lima, and use a 9p mount.
Edit: by editing one of these yamls: https://github.com/lima-vm/lima/tree/master/examples
If you use VS Code the remoting extensions pack includes SSH, for my personal laptop, I'm usually using code to my linux desktop, and editing remotely... it's pretty slick. Code will run a "code server" in the remote environment, and is effectively just displaying the text view to you, the actual work is happening in the remote envornment. You can do similar with containers, or in windows+wsl.
That said, there now seems to be a huge number of options for running various combinations of VMs and containers on Mac and Lenix now. I just use whatever seems popular and is easy to set up, but I have no sense of the pros and cons among them. I would love a coherent document on this topic.
So in other words, qemu runs at near-native speeds?
EDIT: This is from experience. When I started using a M1 mac, our docker builds on x86 images took up to 5x or 10x longer vs running on an arm64 image. Had slightly faster, but similar results running my own docker host on Lima, or podman, or building without docker inside a UTM host which also uses qemu.
https://developer.apple.com/documentation/virtualization/run...
From there you can register Rosetta as a runtime and use it to run x86-64 Linux apps on ARM64 if you want. (https://developer.apple.com/documentation/virtualization/run...)
The point of running stuff in qemu is to virtualize / emulate the hardware. You still need a “normal” OS inside qemu.
it's still surprising to me that they did this as i always got the sense that developer word of mouth helped drive a lot of macbook adoption in the 2000s.
I use ARM containers locally to dev and test locally, check in the code and the build server creates an x86 image for deployment to the server.
Worst case you have to build the ARM container yourself locally and tag it with the "remote" name and docker will just use it instead of pulling the remote version.
My current setup is to have both data and workload run on a remote machine and I connect to it via SSH. I can either run neovim inside or use the remote development plugin from VSCode. But as mentioned, the input lag can be very annoying. I’m wondering if there’s another setup where I can still retain some of the upsides of running the workloads remotely and still having a decent user experience (reduced lag)
- Local and remote copy of files - Two way sync
I've been using this for years, editing locally with Sublime, while running everything on a remote Linux vm. Because it's two way sync, anything created/modified on the server is also sync'd back to your local filesystem. I end up syncing most of the remote $HOME so I can easily fire up a new VM, sync, and go. It really is fantastic.
Or are you just talking about when using neovim?
https://docs.rackspace.com/blog/speeding-up-ssh-session-crea...
For reducing the latency of an interactive ssh session, there's mosh: https://mosh.org/
If you've ever tried to spin up a whole bunch of Docker containers in WSL2 and watched `vmmem` memory and CPU usage explode, you know that 'near-native speed' in VMs comes with lots of asterisks.
Does macOS have usable native macOS containers yet?
I come from the development background and the number one use case of containers on macOS is development enviroments, as on Windows too. For this use case, file system IO has always been bottleneck, not CPU. I do not know if there is some silver bullet in the horizon that could make this faster.
EDIT: Looks like this was indeed just added in the last couple of weeks https://github.com/lima-vm/lima/commit/c18ae239b69a47db77436...
I am personally slowly moving back to Docker after having used limactl for a long time. It's just a pain in the ass.
A Linux laptop is better for all intents and purposes, except the shitty battery life...
Macpine: https://github.com/beringresearch/macpine/blob/71788e9c3c09c...
colima: https://github.com/abiosoft/colima/blob/7ebcf14a69158afa43b2...
So it seems that it has same performance as colima project as well.
As for IO performance, see this colima issue https://github.com/abiosoft/colima/issues/146#issuecomment-1...
E.g. here is page 2 of one tutorial: https://access.redhat.com/documentation/en-us/red_hat_enterp...
All these rules made no sense to me, and while I suppose they become clear at some point, I like my tutorials to be clear from the start.
3 minutes search reveals an easy to follow tutorial.
I'm guessing bocker would be more useful for that: https://news.ycombinator.com/item?id=33218094
Lxc is somewhat to lxd as jails are to ezjails...
https://access.redhat.com/documentation/en-us/red_hat_enterp...
That said I don't remember having to learn all that much about cgroups when I last looked at lxc...
Well, I started looking into it because there were all these mounted filesystems that I couldn't unmount, even as root. I guess I will have to look into namespaces also.
You'll be pleased that at least with cgroups v2 there's only a single /sys/fs/cgroup instead of half a dozen filesystems mounted underneath that path. :)
Wow, I'm dying to see how it works!
> brew install qemu
:|