500 million tweets per day. 140 bytes (140 characters * 8-bit ASCII) per tweet.
140 bytes * 500 million = 70GB
Thats 70GB per day before metadata. Use this social network for a month and we've exceeded the 1TB mark, twice.
Remember this isn't just 70GB per day on one server, this is 70GB per day on every users PC.
the idea is to only replicate who you care about+a couple of their friends.
The average person has 208 twitter followers. So lets say you have 208 'friends' + a couple additional 'friends' for each of your original friends. That's 624 people total.
There are 100 million active twitter users each day and 500 million tweets per day, that's 5 tweets per person.
5 * 624 = 3120
That's 3120 posts you'll be processing per day. Multiply this by 140 bytes per post and you have 436800 bytes per day or 159.5 MB per year.
That's manageable.