s3rg4fts

Post 2 of my Kubernetes on Raspberry series. Preparing (bootstrapping) the Raspberry Pis to become Kubernetes nodes.

Post 1 of my Kubernetes on Raspberry series. Explaining the hardware used for the nodes, the cluster's node configuration and costs.

If you're templatizing manifests, kubespray does this pretty well I think. At least for the basics, it's pretty helpful so far. But indeed, I'm looking into deploying more things with Helm if possible.

Most services I've been using so far offer official Helm charts. But I get your point, it can be cumbersome and if there isn't an official one, then they can be pretty undocumented / hard to work around.

I haven't used CDK, but the concept is definitely similar. I think Pulumi most likely has wider support, since it's based on Terraform and even if you don't have a provider available on Pulumi you can "port it" (although never tried it, not sure if it works well). I like how it stores the state for you and secrets as well, saves quite a bit of trouble.

It most definitely is not, you are right! I think the main difference IMO is that by paying that price with k8s I at least have something that resembles an application platform and I can easily ship containers, deploy my app and not deal with hardware so much as I'd have to do in the past deploying to servers.

But, as mentioned, I only see it as a developer. My end product with k8s resembles something that's closer to my development tools than what I'd have maintaining a series of servers and using other tools to deploy apps onto.

I do dread that exact scenario you're mentioning. I know it is a possibility but I'm hoping that should this day arrive I'll be able to put it to rest and hopefully have learned stuff along the way.

Good luck! This is pretty useful for benchmarks https://github.com/longhorn/kbench?tab=readme-ov-file

I think the 3/4s have a lot of limitations indeed. The 5s are a bit more powerful, so I'm expecting your experience would be better.

Wasn't aware of NixOS, looks pretty interesting but I'm not sure about how easy / reliable it'd be to run it on a Pi 5 (https://wiki.nixos.org/wiki/NixOS_on_ARM/Raspberry_Pi_5). I'll be keeping an eye on it though!

As far as Helm vs Ansible, I'm using Ansible to deploy the basics (bootstrap control plane & worker nodes, networks plugin) and then everything is deployed with IaC (Pulumi) which installs Helm releases.

Wasn't aware of HomelabOS, looks pretty interesting!

How is your Longhorn performing? I tried setting it up on my nodes but with Gigabit networking and possibly the Pis pretty average CPUs I would get pretty awful performance both in distributed volumes (with replicas etc) but also on strict-local ones (for reasons I haven't yet figured out).

I am now considering using something like https://github.com/rancher/local-path-provisioner, since I mainly intend to use Longhorn for DBs that handle fault tolerance / backups etc on their own.

I did it on Procreate on an iPad. You can export layers as GIF (or an MP4 video). But I guess you could do the same with Photoshop on a desktop.

Sadly, in this specific case we update almost 1/10th of the table daily and we have also based our entire searching functionality on indexes in this table.

Bad news: our product relies a lot on having so many access patterns so it's hard to limit the indexes at this point

Good news: most of the overhead (as you can see from the table with the timings) seems to be caused by the GIN index. We'll most likely move full text search to ElasticSearch and drop this so things will most likely get better.

Once the move is done, we'll try dropping indexes and using ElasticSearch more for searches. I've seen this work tremendously well in the past so I have high hopes!

Hey, author here! The next thing I'm aiming to do is VACUUM more aggressively, like you propose. However, given that we update ~1m rows daily, I don't think that VACUUM alone would do the trick. Except if I went the opposite way and increased the pending list size a lot and VACUUM aggressively. But then I'm afraid that SELECTs could suffer because of the big (unordered) pending list.

Thank you for all of this information! I'll dive into it.

PS: your talk looks amazing. DTrace looks like an exceptional tool and the example you're making gives me a ton of ideas on how I can use it in the future. If I knew you could debug like this maybe I wouldn't have needed to ask in DBA stacked change.

A journey debugging slow write performance on PostgreSQL. Assumptions, validation, community support and lessons learned.