Interesting, I would have thought that the unidirectional, read-only nature of the publisher-subscriber relationship would have made this simple for a traditional SQL database with read replicas and a very basic partitioning scheme. You assign workers to monitor feeds for updates, they update the DB, and.. done.

Looks like they may have added some complexity with their feed parser implementation, what they refer to as "supernoders". Looks like they don't lock ownership of feeds during parsing, thus allowing concurrent supernoders to get into race conditions while parsing the same feed.

And so it turns into another NoSQL example of employing conflict resolution to fix things.

I wonder if they could just use a simple locking scheme to prevent more than 1 parser from parsing the same feed at the same time. This sounds simpler than conflict resolution, to me.