They laugh at me for constructing as much as I can from the infra as plain kubernetes objects.
Keep laughing, I can switch clouds in less time than you take for figuring out cloud costs.
Keep laughing, I can switch clouds in less time than you take for figuring out cloud costs.
For me the sweet spot is to have all compute in Kubernetes and stick to open source or "standard" (e.g S3) services for your auxiliary needs, but outsource it to the cloud provider. Fairly easy to move somewhere else, while still keeping the operational burden down.
But agree that having e.g all your databases in a proprietary solution from the cloud vendor seems sketchy.
AWS has very nice one-to-one mapping of K8s serviceAccount with IAM roles.
We used some cryptography-centric GitOps patterns to eliminate any human hands beyond a dev environment which also helps IAM be easier (but of no less reasonable granularity and quality).
> the jump to some much cheaper bare metal hosting is not that far.
Heh, at a consulting firm I was at not too long ago, all the K8s nodes were sole-tenant on the cloud providers these K8s nodes were on. Intra-cluster Cilium-based Pod-to-Pod networking across cloud-based datacenter sites has been super smooth, but I have to admit I'm probably tainted/biased by that team's uncommon access to talent.
I think for that particular firm, "K8s tax" measured in fiat currency is negligible and anything on the human side their people upon hearing "K8s tax" would respond with some ridicule along the lines of "someone doesn't know how to computer".
To be fair, most of the commenters on HackerNews should use something like Heroku.