My question specifically being: if I configure a k8s cluster to have all my slaves as preemptible nodes...would GCP automatically add new nodes as my old nodes are deleted (from what I understand preemptible nodes are assigned to you for a max of 24 hrs)?
Considering the pricing of preemptible nodes + the discounts that GCP assigns to you for sustained use, it makes cloud insanely cheap for an early stage startup.
Go for it as long as you understand the downside. It's possible that all instances get preempted at once (especially at the 24hr mark), that there isn't capacity to spin up new preemptible nodes in the selected zone once the old instance is deleted, etc. New VMs also take time to boot and join the cluster.
If you are just doing dev/test stuff, I'd recommend using a namespace in your production cluster or spinning up and down test clusters on demand (which can be preemptible).
If you have long running tasks (like a database) or are serving production traffic, using 100% preemptible nodes is not a good idea.
Preemptible can be great for burst traffic and batch jobs, or you can do a mix of preemptible and standard to get the right mix of stability and cost.
Not me or OP, but same team :)
Would those types of mitigations work similarly with Google's premetable VM's?
There are a few interesting projects out there that do the kind of automation you are speaking of like these:
https://github.com/binary-com/gce-manager https://github.com/skelterjohn/prevmtable
Spreading multiple smaller machines over a multi-zone k8s deployment might help mitigate, but it will never solve all the issues.