Hacker News Reader

Comment by ypcx | Hacker News Reader

Wondering how much of that 2TB dataset is necessary for the common daily functionality of reddit, probably less than 1%, and the rest is historical data, accessed by almost no one, except perhaps by the submission-dupe- checking algorithms, and similar?

View on HN

Are you suggesting moving that down the ladder and not having that data everywhere? I.e. if someone want to see an old post, there would be one extra step required to load the data (so cdn, cassandra, now subset of postgres with not so old data, "full" postgres). I think facebook does something similar, but they really have to, considering their size.

In terms of status updates (ie, stories which may mention check-ins or photos or similar, but not Facebook Messages), before Facebook's Timeline launch, there were multiple stores of data depending on age. With Timeline, all the different versions of data over all ages were put back together into a single (logical) store. More about that process at:

    https://www.facebook.com/notes/facebook-engineering/building-timeline-scaling-up-to-hold-your-life-story/10150468255628920

Reply on news.ycombinator.com