Hacker News Reader

Comment by simonw | Hacker News Reader

Genuine question: what could go wrong?

I'm a big fan of "pinning" mechanisms where any time a user performs a write to your database you set a cookie that lasts for a few seconds and causes any subsequent requests from that user to go to the lesser, not a replica.

This solves the stale-read-after-write problem and ensures users will see any changes that they have personally made.

This does assume a mostly-reads, occasional-write application (which defines most of the applications I've worked on in my career).

So given that kind of application, what are the problems I'm not considering that I should look out for with a read-replica on a different continent?

View on HN

Of the many things that can go wrong, the one I would be most worried about would be replication lag causing the disk on the primary to fill with WAL logs bringing down the primary, or if you have limited the WAL logs, the replica not being able to keep up and detaching from the primary causing stale reads.

Also long distances will make replication latency increase so unless _all_ requests are sent to the primary region you will likely have a read after write state data situation unless you go out of your way to ensure consistency

Is the WAL log situation you describe here a likely function of both distance (hence latency) and volume of writes? So if I have a high volume of writes AND I'm trying to replicate them half way around the globe I'm more at risk of running out of WAL log space on the leader?

Yeah, if you are doing global replication you are much more likely to fall behind so you should have aggressive monitoring on the replica to remove it if replica lag gets to high, that will prevent the primary from retaining WAL files for too long.

It also means you will need to be able to resync the replica from a backup once write volume calms down.

Stale replicas are a "stop serving requests from a particular region" problem. We pretty aggressively monitor these and consider things broken if replication is lagged more than about 10s.

One thing about running thousands of DB clusters (which we did at Compose.com before this) is, these problems exist even without geo replication. And they exist at such a scale that you have to be good at handling them. Geo replicas are _more_ prone to these kinds of issues but we already had to be good at handling them.

That is good and should prevent against bringing down the primary. There is still the issue with being able to get that region back online once replication is severed, which isn't always possible even with co-located systems.

Reply on news.ycombinator.com