Hacker News Reader

Comment by 0xbadcafebee | Hacker News Reader

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work.

Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database per tenant that's a little bit less flexible, ..... or we could reinvent everything they do on our own software platform and expect the same results. All it'll cost us is extra expertise, extra staff, extra time, extra money, extra planning, and extra operations. But surely it will improve our product dramatically.

View on HN

AWS RDS is 10x slower than BareMetal MySQL (both reads and writes). Slowness is mainly due to the reason that Storage is over network for RDS.

Not bad to invest some extra time to get better performance.

You are falling to “Appeal to antiquity” fallacy if you think something old is better.

It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help.

If you're gonna spend effort in running Postgres manually, do it on bare-metal and at least get some reward out of it (performance and reduced cost).

> It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help.

On GCP, at least, you can provision a GKE node-pool where the nodes have direct-attached NVMe storage; deploy a privileged container that formats and RAID0s up the drives; and then make use of the resulting scratch filesystem via host-mounts.

> It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help.

What?? We run replicated Patroni on local NVMEs and it's incredibly fast.

If you have a K8S cluster running on bare-metal with directly attached disks, sure, it'll work great. I'm just pointing out that K8S by itself will not give you any performance boost if the underlying K8S nodes are the same, slow VMs with terrible IO performance.

Sure, of course. But that's a far cry from saying "on K8S (which is itself going to run on underpowered VMs with networked storage)"

What you describe is still a fallacy because it's assuming that just because you can get better performance with BareMetal, that somehow this is a cheaper or better option. In fact it will be either more error-prone, or more expensive, or both, because you are trying to reproduce from scratch what the whole RDS team has been doing for 10 years.

I don't think anyone's arguing RDS doesn't have useful features. The problem is that it's stupid expensive for the performance you get. RDS makes a lot of sense when prototyping, and if you want a failover database with checkpoint backups, but having it be your primary database of record only makes sense if you're not developing a data product, otherwise your profit margin becomes Amazon's.

RDS is to 2022 what Oracle was in 2002. It’s a safe choice.

I'm not so sure. All you have is another layer of abstraction between you and the problem that you are facing. And that level of abstraction may violate your SLAs unless you pitch $15k for the enterprise support option. And that may not even be fruitful because it relies on an uncertain network of folk at the other end who may or may not even be able to interpret and/or solve your problem. Also you are at the whim of their changes which may or may not break your shit.

Source: AWS user on very very large scale stuff for about 10 years now. It's not magic or perfection. It's just someone else's pile of problems that are lurking. The only consolation is they appear to try slightly harder than the datacentres that we replaced.

And when it all goes bottoms up it will be much more difficult to resolve.

Depends. A lot of postgres usage is often "things that might as well be redis", like session tokens (but the library we imported came configured for postgres) so if the postgres goes down, as long as it can be restarted it won't be the end of the world even if all the data were wiped.

Probably there is also an 80/20 for most users where it's not awful if you can restore from a cold storage, say 12h, backup.

Fortunately Postgres doesn’t do that often by itself. It usually needs some creative developer’s assistance.

I think you’re triggering the worst case a lot more often when it comes to running Postgres on k8s: the storage can be removed independently from the workload and the pod can be evicted much easier than it would be in traditional database hosting methods.

No need for developers to do anything strange at all.

Reply on news.ycombinator.com