Hacker News Reader

Comment by yokem55 | Hacker News Reader

LXD (Canonical's daemon/API front end to lxc containers) is great -- as long as you aren't using the god awful snap package they insist on. The snap is probably fine for single dev machines, but it has zero place in anything production. This is because canonical insists on auto-updating and refreshing the snap at random intervals, even when you pin to a specific version channel. Three times I had to manually recover a cluster of lxd systems that broke during a snap refresh because the cluster couldn't cope with the snaps all refreshing at once.

Going forward we built and installed lxd from source.

View on HN

I got so annoyed with snapd that I finally patched the auto-update functionality to provide control via environment variable. It's ridiculous that this is what I have to personally go through in order to maintain control of when updates are applied on my own systems.

If enough people were to ever decide to get together and properly fork snapd and maintain the patched version I'd totally dedicate time to helping out.

https://gist.github.com/alyandon/97813f577fe906497495439c37d...

We blocked the annoying snapd autoupdate behavior by setting a http proxy to a nonexistent server. Whenever we had a maintenance window we would unset the proxy, allow the update, then set the set the nonexistent proxy server again.

Very annoying.

this feels both clever and stupid at the same time - not you but the software games you have to play.

theres a lot of wasted effort in the games we play to make software work in prod.

From being a production engineer for many years I can only agree: some of the brightest people of our generation spend most of their time compensating for the poor design choices of others. Not sure if this is better or worse than advertising.

That certainly works too but with my approach you can run "snap refresh" manually whenever you feel like updating.

> If enough people were to ever decide to get together and properly fork snapd and maintain the patched version I'd totally dedicate time to helping out.

Is that the gist of flatpak?

Agreed, it's the stupidest thing around, especially because Linux is about having control of your own systems, then they force updates down your throat, it goes against a lot of the things I use Linux for.

I just avoid snap and Ubuntu wherever possible now.

Makes you wonder whether Canonical has any idea about operating servers. Auto-updating packages is the last thing you want. Doing that for a container engine, without building in some jitter to avoid the scenario you described is absolutely insane.

Who even uses snap in production? If I squint my eyes I can see the use for desktops, but why insist on it for server technologies as well?

Canonical would gladly hand back full control of updates if you pay them for an "enterprise edition" snap store. https://ubuntu.com/core/docs/store-overview#:~:text=Brand%20....

And even then, controlling the package versions is only one of the problems. The bigger problem which isn’t solved with this (as far as I can tell) is not having the machines automatically update due to how the snap software works.

I had a huge argument in 2015 with a guy that wanted to move our every custom .deb package (100+) to Snap, because they had talked with Canonical and it would be the future, Docker would be obsolete. Main argument was to make distribution easier to worker/headless/server machines. Not that Docker is a direct replacement, but Snap is an abomination. They are mostly out of date, most of them requires system privilleges, unstable and the way they mount compressed rootfs is making starts very slow, even on a good machine.

That all being said, LXD is great way to run non-ephemeral containers that behave more like a VM. Also checkout multipass, by Canonical that also makes spinning up Ubuntu VMs as easy as Docker.

pro tip: you can use lxc/lxd to run VMs under the same infrastructure.

I belive it is something as easy as

    lxc launch -vm ubuntu/20.04 myvm

What VM software does this make use of? KVM?

Yes. LXD uses Qemu/KVM for VMs.

Wow, cannot wait to try it! Thanks for the tip.

On my Ubuntu 20 server, I tried setting up microk8s with juju using LXD and my god the experience was horrendous. One bug after another after another after another after another. Then I upgraded my memory and somehow snap/LXD got perma stuck in an invalid state. The only solution was to wipe and purge everything related to snap/LXD.

After that I setup minikube with a Docker backend. It all worked instantly, perfectly aligned with my mental model, zero bugs, zero hassle. Canonical builds a great OS, but their Snap/VM org is... not competitive.

Note that Canonical has special tools for installing and configuring that stuff (conjureup or something)

That's what I was using, and is largely how Juju is used

That's strange. I've had no issues using microk8s. Super easy to set up on Ubuntu 20.04.

Not even kidding, a huge part of what made me move to Arch was that it's one of the few distros that packages LXD. Apparently it's a pain, but I'm forever grateful!

Alpine is another distro that packages LXD. I have Arch on my workstation, but I'm not confident about securing Arch on a server. Alpine, on the other hand, is very much suited to be an LXD host. It's tiny, runs entirely from RAM and can tolerate disk access failures. Modifications to host filesystem won't persist after reboot, unless the admin 'commits' them. The modifications can be reviewed before a commit - so it's easy to notice any malicious modifications. I also heard a rumor that they are integrating ostree.

The only gripe I have with Alpine is its installation experience. Like Arch, Alpine has a DIY type installation (but a completely different style). But unlike Arch, it isn't easy to properly install Alpine without a lot of trial and error. Alpine documentation felt like it neglects a lot of important edge cases that trip you up during installation. Arch wiki is excellent on that aspect - they are likely to cover every misstep or unexpected problem you may encounter during installation.

openSUSE packages lxd

I did not know there were so few distros packaging LXD [0]. It does not come as a surprise that Canonical would not endorse non-snapded LXD though. And it is sad.

[0] https://repology.org/project/lxd/versions

> This is because canonical insists on auto-updating and refreshing the snap at random intervals, even when you pin to a specific version channel.

You can control snap updates to match your maintenance windows, or just defer them. Documentation here: https://snapcraft.io/docs/keeping-snaps-up-to-date#heading--...

What you cannot do without patching is defer an update for more than 90 days. [Edit: well, you sort of can, by bypassing the store and "sideloading" instead: https://forum.snapcraft.io/t/disabling-automatic-refresh-for...]

Why would any sysadmin that maintains a fleet of servers with varying maintenance windows (that may or may not be dependent on the packages installed) want to feed snapd a list of dates and times to basically ask it pretty please not to auto-update snaps?

It is much easier to give sysadmins back the power they've always had to perform updates when it is reasonable to do so on their own schedule without having to inform snapd of what that schedule is.

I really don't appreciate being told by Canonical that I have to jump through additional hoops and spend additional time satisfying snapd to maintain the systems under my control.

My initial reaction was that having schedules upgrades like that would be great, but then 5 seconds later I realized it's much better suited to Cron, SystemD, Ansible (etc.).

I think the reason for auto-updates like this is because selling the control back to us is the business plan. It's the same thing Microsoft does.

it looks like there has been considerable progress packaging in debian bookworm: https://blog.calenhad.com/posts/2022/01/lxd-packaging-report...

Just curious--how do you use LXD in production? It always struck me as something very neat/useful for dev machines, but I had trouble imagining how it would improve production workloads.

Maybe two years ago I wanted to use LXD on a fresh Ubuntu server (after testing it locally).

First they had just moved it to Snap which was not a great install experience compared to good old apt-get, and then all my containers had no IPv4 because of systemd for a reason I can't remember.

After two or three tries I just gave up, installed CapRover (still in use today) and have not tried again since.

The install experience should be the same. Using apt install lxd should give you snap. I've installed LXD quite a few times with different storage and network setups without any issues, so I do find it strange that it didn't work for you.

I was bitten by LXD auto-updating as well. Server was down and I couldn't understand why since I hadn't changed anything.

> The snap is probably fine for single dev machines

It is not good even on single dev machines.

I normally use podman for my dev machine. Unlike LXD, it can run rootless.

I am simple man with simple need, I am perfectly happy with a distro as long I have my editor my terminal and my browser.

I could not bear the snaps on ubuntu always coming back and hard to disable on every update, I gave up and just switched to arch and happy to have control on my system again.

I had a lot of crash running on Ubuntu when running huge rust based test suite doing a lot of IO (on btrfs), never had that issue on arch. not sure why, not sure how I can even debug it (full freeze, nothing in systemd logs) so I guess I just gave up.....

Canonical always backs the wrong horse. Unity, Snap, Mir, Upstart, etc. etc.

Yeah, it's truly terrible. I've had downtime from this as well.

Reply on news.ycombinator.com