Hacker News Reader

Comment by rmbyrro | Hacker News Reader

> Now my functions just downloaded the file from S3, if it was newer than the local copy

if you have strong consistency requirements, this doesn't work. synchronizing clocks reliably between different servers is surprisingly hard. you might end up working with stale data. might work for use cases that can accept eventual consistency.

View on HN

This shouldn't depend on clocks, just tracking Etag is more consistency proof.

If you have strong consistency requirements, then it doesn't work by the very nature of making multiple copies of the database. Even if the clocks are perfect. (Though the clocks are probably close enough that it doesn't matter.)

One of the announcements from AWS this year at Re:invent is that they now can guarantee that the instances clocks are synced within microseconds of each other. Close enough that you can rely on it for distributed timekeeping.

I don't really know if that matters for this use case. Just by the very nature of source_data -> processing -> dest_data taking nonzero time anything consuming dest_data must already be tolerant of some amount of lag. And how it's coded guarantees you can never observe dest_data going new -> old -> new.

Wouldnt e-tag version numbers also work? Or just havkng .jsom with version metadata next to the db blob? No need to sync clocks. Just GET the small db-ver.json and compare version details?

I versioned the databases and my use case didn’t require strong consistency.

Reply on news.ycombinator.com