Hacker News Reader

Comment by arrty88 | Hacker News Reader

What about local read only replicas of the db in each region, and one primary in your primary region? Or a write through cache in each region.

View on HN

This design is really similar to the approach taken by litefs[1] and turso[2]. Writes are forwarded to the primary (like you would with a write-through cache) and a local copy of the data is held in each region. These local copies are only guaranteed to be up-to-date with the most recent write forwarded by that local copy.

[1] https://fly.io/docs/litefs/proxy/ [2] https://docs.turso.tech/reference/data-consistency#on-the-re...

From what I understand, that should be possible with Fly.io, using the fly-replay header to denote requests that need to be run against the primary database. I'm not sure if that's still their recommended approach, though.

EDIT: https://fly.io/blog/globally-distributed-postgres/ talks about how this can work. The approach it suggests for implementing this is to try running all database queries locally; if there's an error from the local database being a read-only replica, add the fly-replay header pointing to the primary region and return an HTTP 409, and Fly's proxy will rerun the request in the specified region. There's also some commentary on handling consistency issues to at least get read-your-own-writes.

Yes, it is. It's how we think about multi-region replication in our own tooling. It's not the only way to handle it (Cockroach would be an example of another approach; so would rqlite). We're not evangelists for multi-reader single-writer --- it's just the most common approach big Postgres shops tend to use, so we worked on making it effective. But the other approaches should work fine on Fly.io too.

What we especially like about multi-reader single-writer clusters is that you can mostly keep all the pieces in your head at the same time. If you understand Postgres or SQLite, and can get your head around the `fly-replay` header (which is just "replay this request over there please"), you've got the whole system.

Thanks for confirming this!

We do new stuff all the time but `fly-replay` is unlikely to go away, since I think it's Kurt's favorite thing we've ever come up with. :)

rqlite[1] creator here, happy to answer any questions. rqlite also supports read-only nodes, which can also help with reads at the "edge". It probably wouldn't scale to 100s of nodes, but it is an option.

"rqlite supports adding read-only nodes. You can use this feature to add read scalability to the cluster if you need a high volume of reads, or want to distribute copies of the data nearer to clients – but don’t want those nodes counted towards the quorum. These types of nodes are also known as non-voting nodes."

[1] https://rqlite.io/

[2] https://rqlite.io/docs/clustering/read-only-nodes/

This is relatively easy to orchestrate with MongoDB replica sets (and I'm sure with other databases, too). In a typical MongoDB deployment you want at least 3 nodes (1 primary, 2 or more secondaries).

You can put one of the secondaries on the other side of the world.

By default, all reads are sent to the primary. But you can easily specify "nearest" as your "read preference" on a query-by-query basis which would allow the app to read from the nreplica with the lowest latency (whether it be the primary or one of the secondaries).

The ability to specify read preference at the query level lets you have 2 categories of queries, 1) queries that must always be served with most recent writes, sent to primary at the expense , and 2) queries that are allowed to occasionally send stale data but benefit from latency improvements by reading from the nearest node.

PolyScale [1] focuses on many of these issues. It provides a globally distributed database cache at the edge. Writes pass through to the database and reads are cached locally to the app tier. The Smart Invalidation feature inspects updates/deletes/inserts and invalidates just the changed data from the cache, globally.

1. https://www.polyscale.ai/

> Or a write through cache in each region.

This is something I want to play around with using Cloudflare. I wonder if there’s enough of a guarantee for a user to “stick” to a node that you could use Workers KV as an eventually consistent write through cache for user unique data and a Durable Object as a strongly consistent write through cache.

I feel like that would give a pretty solid balance, wouldn’t it?

Cloudflare has Durable Objects/workers for consistent write-through... You can combine them. You can also consider D1, which might be the better option all around for many use cases.

Or just having a completely separate database for each region, much like the average MMO does it.

Reply on news.ycombinator.com