But alas, nobody will ever go for it and I'm left to dream.
I use Ansible at home as a replacement for the 1990s style of sysadmin where you manually managed your servers (which I did until the mid 2010s). Yet, I’m aware that Ansible is not declarative. It’s looks declarative, if you take your glasses off and don’t think too hard about it, but the key to understanding Ansible is really that the configuration is idempotent (not declarative).
Servers are cattle either way, it’s just that with Ansible, your servers will keep changes that you made months or years ago that have been deleted from your playbooks. That, to me, is what makes it old-fashioned.
I don’t think old-fashioned is bad. I’ve gotten into arguments where people have told me to manage my personal server(s) using K8s and I, politely as I could manage, told them to fuck off.
What about the specialized services from AWS provides leverage outside of the “cattle mindset”?
In my experience, AWS’s primary value and leverage is based on intermittent burst compute. It’s why you rarely get a hard answer on the Hz of each vCPU and have credits to the overuse or underuse of said instances. The cpu is then used to arbitrage value in the dedicated services like lambda and others - so your mindset and infra needs to match AWS’s incentives to get the same value, correct?
We use ~50 fairly high performance (RAM, CPU, disc) servers pretty much continuously, so the savings compared to AWS etc are considerable. Every couple of years there's a software upgrade required, and it would be convenient to have another 50 servers to use during the transition, just for a few days.
Compared to AWS prices, we can easily justify keeping some old servers around to help with this sort of thing. A managed server company (Hetzner etc) would be somewhere in-between; presumably able to rent us 50 extra machines for the duration, but (last time we checked) still more expensive than managing the servers ourselves.
This could be only the case is t* instances? I learned the hard way that CPU misses can add even 30% of request latency, especially when you often do I/O eg to external services like db/Redis/etc
I didn't notice such behaviors on AWS. but when thinking about it, I read somewhere that one reason they have dedicated CPUs is for partitioning CPU cache so it's not shared between VMs. so maybe with some tricks they could get some free compute without side effects?
Will you have redundant hardware running to failover to? What about disaster recovery, are you going to colo in more than 1 physical data center?
Who's going to be on-call: the same people who are on call now. HW failures don't need special on-call people.
Who is going to build/manage your monitoring/alerting stack. Same people as now. Your servers are monitored, right? Adding RAID into the mix is hardly difficult.
Will you hire a network engineer? No, colos will often offer to manage the network for you up to the switch and at that point it's just a matter of plugging the server in. You don't need any expertise to do that and remote hands can do it for you.
Redundant hardware? Only if you need it currently. HW is reliable these days and you'd going to have spare capacity anyway, so this really doesn't change much.
Disaster recovery? Take whatever answer you currently have in the cloud for multi-AZ.
Performance. You'd also get way faster hardware too.
Enterprise-grade SSDs is what I meant by direct-attach storage - I was comparing against network storage which is what all cloud providers use (your EBS volume is accessed over the network internally, which incurs some latency, ultimately limits IOPS and destroys random access performance which can't be cached or read-ahead by the underlying hypervisor).
“multiple nodes only give you reliability, not performance” - this, actually, is not true. Even mysql can offer ways to tune and gain performance improvements.
Presumably they would also lack the durability guarantees of the normal block storage offered by the provider.
It sounds like you should make a business case. 90% savings is a big deal, unless it's the difference between $1k and $100.