Honest question, if you're running at 10%, why have you gone with 12-core, 512 GB RAM servers? Couldn't you start with a more reasonable 4-core, 64 GB RAM until some threshold? What value do you get from such an early overallocation of resources?
Symmetry between servers has a value.
The main board, raid controller, network, and usually the storage is going to be planned out meticulously ahead of time based on the maximum load the server is going to see during it's lifetime. Often, Processor and Memory come down to "If we needed X Feature and we didn't have it" or "If we took the server down for 1 hour to upgrade it, how expensive is that?".
I initially pictured someone picking the highest spec option they can afford when setting up a new service, rather than choosing based on actual demand of each node.
> "If we took the server down for 1 hour to upgrade it, how expensive is that?"
Putting my CI / DevOps hat on for a sec: who takes production servers down for upgrades without some level of HA to avoid downtime? ;)
Every service gets run in two environments: test and prod. Both are co-located on the same Kube cluster in different namespaces. We also don't put any datastores in Kube. That stuff still lives in OpenStack for now. Ceph can make it possible but for fast disk I/O, it's tough to beat the local SAS bus.
Personal anecdote - we have everything on-prem using a pretty standard vSphere setup, I've got a couple PostgreSQL databases that aren't even that heavily used (I mean, they top out around 20-40 tx/sec) backed by a hybrid Tegile array. Randomly throughout the day my IOWAIT starts spiking because even over 8Gb fiber channel our storage latency starts spiking because everything else is on the same storage (well over 200 full VM's), or sometimes I need to run a table scan over a 40GB table for a one-off query and the storage bottlenecks the crap out of us because only our small active set is cached on the SSD's.
You can run databases on remote storage, people obviously do it, but there's a reason why even on AWS the best practice is to use your instance disks instead of EBS volumes if you care about performance. Noisy neighbors suck.
GCE seems poorly engineered if the profit margins are indeed low (normal, that is).
Interestingly, the "3 cores for 1 user acing one" would explain the pricing of "preemptible" instances - which can cost as little as 1/3rd of full instance. And the main difference is that they are fully rescheduled every 24h or more often, and there's no live-migration for maintenance...
… I am now wondering if GCE runs fault-tolerant VMs. Because if it does, holy moly
A GCE VM spins up in less than a minute. I've seen them come up in 20 seconds.