I have what I feel is an irrational desire to build an RPi k8s cluster. I feel like I would be better served by a small x86 box, but something about having a real cluster really appeals to me.
Sold all those, recovered most of the money and bought a dismissed laptop from my employer of the time: 3rd gen quad-core i7, 16gb ram (later upgraded to 32), 480gb ssd + 750gb hdd for waaaay less (just the residual price)
Installed proxmox, created small VMS and played with k8s and a lot more stuff in a cheaper and more performant way (full gigabit ethernet when RPI has usb-shared ethernet, ssd-grade I/O performance, core i7 compute performance).
If you want to experiment the raspberry pi is the dumbest thing that you can buy.
Go for old hardware and virtualization, you'll also learn more (containers are here to stay, but VMs aren't going away either).
Power usage was also surely higher than a single RPI, but negligible anyway: laptop Intel processors with speedstep technology that lower down power absorption, and can go down to like 15-20 watts/hour.
Pulled the trigger on that three times now.
My first cluster was Raspis and ODroids. The two issues I had - not everything runs on Arm (yet) and memory constraints were tough to work with.
My second cluster - 4 Atomic Pis. The (Intel Atom) quad cores were fine but the 2GB of memory was a real problem when I started playing with more interesting workloads.
The current "production" home cluster is 4 old laptops. They're earlier gen i7s and I've dropped 16GB of memory and 1TB spinning disks in each one. I've never been happier with a cluster. I've been free to spin whatever I want up and it just works. Using RKE to setup Kubernetes takes about 5 minutes to build a cluster and I'm using Longhorn for replicated container native storage. I've run experiments for work and my own experience. Whether its Redis / Cassandra cluster or a Kubernetes Operator I wrote, it can always handle it.
Also using RKE, it looks like the best cluster 'distro' around.
I used longhorn but found it a little slow, putting it on SSD was a bit of a waste. I tried ceph, it was much faster but i just do not understand how it works, etc. Well enoguh to fix anything if the cluster goes wrong.
I just had to switch to a SFF desktop PC with Ubuntu on it and never looked back. For toying around rPis can be fun but in real life applications, they were getting extremely hot. Had to add extra fans etc.
I still use my rPi4s, one of them connected to my 3D printer with Octoprint installed on it, but that's it.
Now I switched to x86, I’m even sharing my Plex library with a few more people (looking at the Tautulli stats, it’s been transcoding a lot) without any problems.
If it becomes an issue, I’d add a GPU but for now a 6 core AMD is doing a very good job.