What do you all use for your pg backups? Is Barman ( https://pgbarman.org/) still the way many do it? (I haven't deployed a new pg instance for a while, but thinking about it for a new project).
So while I partly agree with you, a lot of companies don't really need HA, read replicas, or even PITR (though I would argue the last one is so trivial and cheap to enable that why not), but they click the expensive check box, and I would argue that companies who do need these features should consider hiring at least a couple of DBAs and get more flexibility instead of the current status quo of everyone being scared of the database and everyone just hoping cloud support will come to their rescue if ever needed
Every place I’ve ever worked at that had DBAs had the complete opposite of more flexibility. You have to do things the DBA’s way, and if their way doesn’t work for your service, you need to fight for their time and priority.
Meanwhile every place I worked at where every team completely owned their databases + did periodic data recovery drills had much more flexibility and no data loss.
My personal experience has been the opposite of yours: lots of data inconsistency issues, data loss only resolved by the database team spinning up backups, etc. And somehow, even after yet another incident, there’s never been appetite to properly fix things.
Managed databases don’t lose customer data. They would be sued and lose, and if word got out, they would lose their customers. They are optimized for the machines they run on. The proprietary tooling from AWS for managing HA clusters, doing blue-green upgrades and monitoring for slow queries is all miles ahead of the open source tooling available for self-managed databases.
There’s nothing left for the DBA to do.
So the same thing I said we do, a periodic validation of recovery, but with an acceptably short period.
SWEs can do that too. They can even write software to do most of it automatically.
They went through systems design interviews to ensure they know how to scale without picking technology choices that require a dedicated human caretaker.
It’s the opposite. Needing a DBA to operate a completely managed database is a sign of not enough maturity.
And yet here we still are
Obviously "let RDS manage your database" doesn't require egregious read replicas. The decision to use read replicas or not is completely orthogonal to whether you use RDS to manage them.
Of course not, but an easy checkbox, a best practice AWS or terraform guide and someone doing AWS certified X associate makes it easier to happen without anyone ever really discussing it.
> The decision to use read replicas or not is completely orthogonal to whether you use RDS to manage them.
Assuming you're talking about letting RDS manage anything, then sure - apart from it being more likely to slip through the net if nobody has to configure them. Database is just an expensive cost nobody necessarily drills into.
However if you mean the decision to let $cloud manage the replicas (and keep the primary managed), that totally depends on the cloud and the options. For example have you ever tried having a primary in GCP Cloud SQL but the replica not in cloud SQL?
I'm defending your employer or their mindset, I'm just disagreeing with your claim that this follows from the parent's cost analysis claims. It _seems_ like your organization's problems are precisely because they _weren't_ doing the kind of cost analysis that the parent advocated. In other words, nothing in the parent's comment advocated for blindly following some Terraform guide. It feels unfair to the parent to suggest that their mindset caused your organization problems when it seems like your organization's problems were caused by _not having_ the parent's mindset.
Like I have no doubt that they’re all greedy bastards, but they aren’t doing much to lock people in. The people who complain about lock in are usually talking about “cloud providers making their services so much easier to use than bespoke platforms on bare Linux hosts such that no one will want to go back to the latter”.
A few years ago we were trying to do a live migration from on-prem into GCP (mostly to have GCP as a failover option but move everything there if on-prem went down). We configured an async GCP slave from the on-prem cluster and once running it was easy. Multi-TB migration straight into CloudSQL with seconds of downtime.
Before we signed this off, we also tested the reverse - getting back from CloudSQL into on-prem. Turns out you couldn't (at the time, I'm not sure about currently) use CloudSQL as master for an external slave, only a slave to an external master. You could migrate the data in with no downtime, to get it back out was a multi-TB transfer. The door only opens one way.
So no, I am talking about lock in.
If you're a SaaS, 1h of downtime costs a lot of money. In reputation, in lost business, in support time, in on-call response, in making a public postmortem and sharing it to customers and dealing with responses... For saving a few thousand bucks, it's often indeed a very bad deal.
Of course this isn't true for some companies and the cost of downtime far exceeds the cost of HA. Most companies think they are that company, most aren't. And even if they are, "AWS had an outage" does a lot of lifting.
Every time I get a new client they need constant point in time backups, failover, HA everything. Usually when the costs are explained they change their mind and an hour old snapshot restore is actually fine
What does the equivalent RDS setup cost you?
Also, if you are nickel and diming over expenses in the 50€ range, you’re probably at the wrong startup.
We’re not talking about maintaining your own k8s cluster here mind you, but a basic Postgres setup. If you can’t handle that comfortably in an afternoon, you probably shouldn’t be entrusted with customer data.
Intelligence is being able to set up your own postgres database with backups, PITR, retention policies, HA/replicas, and everything else needed to prevent catastrophic data loss. Wisdom is realizing that's a solved problem completely unrelated to your business, so you don't have to.
1. Don't use a third-party auth provider, if you can't build out an OAuth provider in an afternoon you shouldn't be entrusted with login credentials
2. Don't use Github, if you can't build out an internal Gitlab instance in an afternoon you shouldn't be entrusted with securing source code
3. Don't use Github Actions or CircleCI, if you can't self-host a Jenkins server in an afternoon you shouldn't be entrusted with secure deployments
4. Don't use a lawyer or outsourced HR, if you can't draft legally binding employment agreement in an afternoon you shouldn't be entrusted with hiring people
Other than that, yes! Setting up a Keycloak instance as opposed to handing over login credentials to Octa should be a consideration. Keeping your source code in GitLab over tying your entire development workflow to Microsoft vibe coders with an awful track record is a sensible stance.
> Don't use a lawyer or outsourced HR, if you can't draft legally binding employment agreement in an afternoon you shouldn't be entrusted with hiring people
That's a little ridiculous; there's an overlap between someone setting up a database for a startup and someone programmatically using a database at a startup. The overlap between that person and someone with a legal background is infinitesimally smaller.
And RDS is doing a lot more for you than what you can set up in an afternoon w.r.t observability and features that are just a click away. I actually implemented some of those features when I worked there :)
Offers point-in-time recovery which is an improvement over a custom solution we used to have which gave us nightly backups.
We have it backing up to Backblaze B2 (S3 like). Was relatively easy to setup and no problems really.
I didn't understand that, and after 5 months of usage caught my backblaze to be using 40TB, and nightly restores taking forever for other reasons. So: not ideal, and be careful to check!
Obviously past a certain point carting around full backups becomes time/dollar prohibitive, but this can take you very far.
Having done both, I'd recommend just starting with pgbackrest.
I just don't understand why people seem so drawn to the bad solution just because it ships with the database.
If you are on reasonable storage, I think it is arguable that the main reason you would have to recover is due to a software failure leading to corruption of the PostgreSQL backing state files. Otherwise, you would simply be restarting PostgreSQL on your durable and available storage. So, if the content has been corrupted by bugs, can you trust the recent WAL log in this scenario? Or can a plain dump be a more reliable "known-good" database recovery point?
And what is the business cost to rolling back to a less frequent dump such as nightly? Not every DB is some kind of multi-party OLTP ledger. Sometimes, a day of lost "new data" may be just a day of labor to repeat some data entry or other repeatable work...
A really robust contingency plan probably has both of these kinds of backup, and scenarios where one recovery versus the other is attempted. But if you are starting at a very small scale, hosted on some reasonable cloud storage like EBS, a cron-based dump that goes into a normal hierarchical file backup system may be totally sufficient for the extreme disaster scenario where you cannot simply restart your DB server on top of the surviving and available storage volume?
I'd much rather just do it right though.
Direct attached storage (or whatever storage is fastest / lowest latency for the environment you have available).
Setup pgbackreset or barman, use WAL archiving and setup block incremental backups with a new full backup every so often.
Now you have point in time recovery with very low RPO/RTO, and your database infrastructure can be treated a bit more like cattle instead of a pet, as you should be testing restores often, and this will become part of your yearly major postgres upgrade strategy.
You also get the ability to have a replica for almost free, as replicas can be created from the backup repo very cheaply, and get caught up to the primary without requiring the primary keep around the WAL for a long time. It just pulls it from the repo until caught up.
Any case where you think EBS was the right choice is better handled by having one (or more) replicas and a failover strategy.
Just do the right thing from the start and you save a lot of headaches. People have run production systems already, and have hit the pain points. Why keep hitting the same ones?
I'm all for treating DB as cattle, but your cattle needs to be able to survive long enough.
EBS is expensive but it works for PG. AWS uses EBS for RDS databases. Calling that a 'mistake' is a tall order.
We used EBS snapshots on AWS for multi-TB MongoDB to get incremental backups that are fast to create and fast to restore (with some performance degradation after restore).
It doesn't support point-in-time recovery, but since it's fast you can create frequent snapshots (e.g. hourly). I'd consider adding this as a secondary backup strategy, even if you use a higher-level postgres-specific backup tool.
This guide is only satisfactory if the database is managed, otherwise there are a whole bunch of things going on.
Here's a complete walkthrough on how I backup and restore: https://nickjanetakis.com/blog/how-to-back-up-postgresql-in-...
It covers using Plakar too (optionally) if you want deduplication and encryption.