RaspberryPi 4 home lab with LXD cluster
discuss.linuxcontainers.org
discuss.linuxcontainers.org
I was really impressed that I got line speed Ethernet between the Pis, 112MB/s doing a wget of a 1G file from one to another. Things have improved a lot since the original Raspberry Pi from 2012.
I need to figure out how to get the 8th Pi working though, since I'm using one of the PoE ports to power a Mikrotik that I'm using as a wireless bridge onto my existing broadcast domain. Technically I don't need the bridge, since each Pi can connect to my wireless AP over 5GHz 802.11ac just fine, but it feels nicer to funnel upstream data through one point. More importantly, it reduces WiFi collisions since there wouldn't be 8 devices broadcasting simultaneously and interfereing with each other (especially at those close distances) if they all needed to download something from the Internet at the same time... like a Docker image for a Kubernetes deployment :)
Of course that's only the theory...
Here are my answers for both:
Pi cluster: meshing won't save any additional wires here since there is already a wireless bridge set up (between the black Mikrotik device pictured, and an off-camera Mikrotik wireless router.) I chose to wire the cluster simply to guarantee a reliable connection within the cluster. You can absolutely use a Raspberry Pi over WiFi without an Ethernet cable, it would just be a less reliable connection (jitter, congestion), and then of course it would have to be powered conventionally rather than via PoE.
General 802.11s meshing: If all you want is fewer network cabling in your home specifically between switches and access points, then it could certainly help. However, the best choice network-wise is almost always a proper deployment with a central router and several APs (broadcasting the same SSID) wired back to it.
I had been thinking of building a PIs cluster for a while now and I was looking with interest at the turingpi board project [1] they are taking preorders for the second batch if anyone is interested. But I would like nodes with more than 1GB of RAM so I guess I'll have to wait for that.
Go can be a bit annoying to package in general and LXD was made a bit harder by also having a stack of C libraries for some bits, though with 4.6 we kicked out the need for a custom sqlite and for a coroutine library, so on the C side, outside of the C library, it's down to liblxc, libraft and libdqlite making it a bit easier to package.
For distributions that have a policy of splitting every single Go package into their own source packages (as is the case with Debian), our recommendation is to stick to the LTS release of LXD which only gets released every 2 years and where the bugfix releases don't normally alter dependencies.
The normal feature releases come out every month and those just aren't a good fit when it may take you more than a month to get any new dependency packaged independently first...
Anyway, as an upstream, we are very happy to work with packagers to get native packages in as many distribution as possible.
We do maintain the snap package ourselves and certainly do enjoy it as an upstream since it gives us very large distribution and release coverage with a single package.
Most common criticism of the snap package can be addressed with one of the mechanisms we outlined here: https://discuss.linuxcontainers.org/t/managing-the-lxd-snap/...
That being said, at the end of the day, all we care about is that our users get to run an up to date, secure, LXD. How they get is doesn't really matter to us as an upstream :)
And it's maybe interesting to point out that the majority of our userbase these days are on Chromebooks, effectively using the Gentoo ebuild package of LXD!
Like it or not, it is becoming an important Linux "distro".
The main issue left is that WSL2 doesn't start your normal init system and so makes it harder to run a daemon... You end up having to manually start it every time which is a bit annoying.
But my understanding is that Microsoft is actively working on this and that we should have the normal init system run in some way in the near future.
On your own forums the best answer people can find to this situation is basically "we have moved to snaps, this is not our problem".
Better things to do in life and I got what I wanted from podman with --rootfs.
I noticed the entire cluster was down because my Plex server ran on it and I couldn't watch a movie when I tried later that evening. Luckily I was able to ssh in and determine the issue quickly. Force removing the failed node from the cluster got things going again.
Lxd was quickly replaced by K3S which is unfortunate because the lxd cluster better fit my needs, except reliability, and was much easier to configure/manage.
Even for small platform edge K8S clusters I actual find microk8s much easier than K3S but that would put me back in snap upgrade hell.
Having said that, did you try to pin LXD version? For example
sudo snap install lxd --channel=4.1/stableFor reference my only interaction with it was for privilege escalation on HackTheBox: https://www.hackingarticles.in/lxd-privilege-escalation/
* Giving co-workers a container they can use to host small utilities on a static IP to share, instead of them hosting it on their own desktop. Think compiler-explorer ect.
* Giving co-workers access to our 64 core ThreadRipper for heavy workloads, sandboxed from one another.
But a mandatory "F* Snap" needs to be said.
Which means you can now use the same interface for, say, a Windows VM alongside your lightweight linux containers.
[paraphrased from a reddit post, I was wondering something similar a few weeks ago but haven't done it myself]
This. They can use it as they please, like a second desktop. I can limit their resources, migrate or archive the containers as needed.
One guy was training some ML, one was messing with compiler versions, one with Debian packaging tools ect.
So LXD is much better than docker except docker in spite of being an inferior solution became popular with marketing money spend on it due to hype. LXD stayed with people who believe in pragmatic simplicity. Docker is plagued by privilege escalation for a very long time. Check the details in general Docker has more vulnerabilities than LXD. [2] [3]
[1] https://lwn.net/Articles/687354/
[2] https://www.cvedetails.com/vulnerability-list/vendor_id-1353...
[3] https://www.cvedetails.com/vulnerability-list/vendor_id-1313...
This is kind of like saying a tractor is safer than a car. LXD does not have the same feature set as docker or k8s.
If all you want is a more secure docker then podman got you covered.
If you want more secure k8s then you will have to wait a bit I guess for https://github.com/rootless-containers/usernetes but the cri-o runtime for k8s does have a rootless mode.
LXD is not a replacement for docker or for k8s as it offers a different feature set from both of those.
Last I wanted what one of the features it offered, which is a persistent whole OS container, I tried to install LXD on fedora, and after trying to get lxc running, failing to do so and seeing it's horror show of a systemd setup while debugging [1], I looked elsewhere and instead settled for rootless podman with --rootfs.
[1] https://github.com/lxc/lxc/blob/master/config/init/systemd/l... does something which ubuntu is very fond off and is the main reason why I stopped using it. They call sysv-style init scripts like https://github.com/lxc/lxc/blob/master/config/init/sysvinit/... from systemd and in my experience this obsecures errors as somethign ends up failing but via systemd it still looks like it is running.
For over 90% of startups Docker with k8s is just not necessary and tie them to managed version of specific vendor or cloud provider with higher costs and a lot of overheads. It’s pretty hard for a startup to manage self hosted k8s, given large number of moving parts and management of k8s infrastructure is as big as a task of managing the startup product itself.
LXD is decent enough to build a good high available horizontally scalable cluster on cloud of your choice or bare metal and can be managed by startup teams. Obviously once the startup begins to reach millions of customers and users and have enough revenue than k8s might be viable.
In majority of the projects LXD provides much better infrastructure.
Tractors and cars have a lot of overlap also. They are still different things and people who want one normally do not consider the other as an alternative.
> For over 90% of startups Docker with k8s is just not necessary and tie them to managed version of specific vendor or cloud provider with higher costs and a lot of overheads.
This has not been my experience but at least there are managed k8s offerings from multiple vendors with some intersection of functionality. Sure different vendors have different extensions but there is only one LXD vendor and the support for LXD is kind of flaky in my experience whereas I have ran 3 different k8s distros on my fedora laptop without really breaking a sweat.
> In majority of the projects LXD provides much better infrastructure.
If you are happy with what LXD provides, and if you can actually use it, then great use it. I have yet to be in a situation where I could use it or where I really wanted what it offers. I tried installing it via snap, I tried installing it from copr, I tried manually installing it, then I concluded that I don't actually want anything from it that I cannot get elsewhere.
I think Mark Shuttleworth explained LXD best:
LXD is not an alternative to Docker or K8S, it is something different which offers different features.
And if we are just talking docker, and not k8s, then all the security you can ever want can be found in podman which by default operates rootless and daemonless and works on stock standard OCI containers.
If we are talking k8s there are already runtimes which support rootless operation like cri-o and there are k8s distros that support rootless operation https://github.com/rootless-containers/usernetes - these maybe are not as widely used as they should be and work is ongoing but you will soon see more of them I think.
Also rootless container has been a feature of LXC since 1.0 in 2013-14, which could not be incorporated in Docker as they tried to re-invent the wheels by writing their own libcontainer which eventually resulted in many vulnerabilities which even impacted k8s even in 2019.
Still today unless one use a managed version of k8s or use managed service by major cloud provider the infrastructure will be insecure with k8s given most of the Docker images still not tested as rootless containers. Also for a small team it’s pretty hard to have secure self-hosted k8s infrastructure given sheer complexity and moving parts.
Kubernetes is good for google size companies, for startups it’s an additional overhead and tie their products to vendors specific kubernetes’ distributions given the complexity and moving parts.
Try the above setup and than see the simplicity. Also give a try to kubernetes and see which one is easier and better able to help. In my view for installing database and blob storage infrastructure as part of the application VM’s are still better than containers.
In what way is LXD vendor neutral? Are there any vendors except canonical that make significant contributions and investments to it?
Funny enough indeed Debian, RedHat, Suse and Canonical contribute to LXC project as well.
If you had issues with LXD try asking your questions on LXD forum and community will support you with your issues. It will not need Canonical’s blessings to support you. It’s unlike if you had issues with anthos k8s from google, you need to be a paid subscriber to get support.
Ubuntu 20.04 on WSL2 doesn't use systemd which means no snapd which means no LXD.
This happens due to a series of really bad decisions by canonical and Microsoft. The biggest one being "apt install lxd" attempts to install the snap version but fails silently when snapd is missing.
So you would get everything appearing fine but when you dig down you find something is dying but because of their bad engineering choices it is incredibly difficult to figure out why.
Edit: Test results can be found in the link.