Hacker News Reader

Comment by brooksbp | Hacker News Reader

"This means the open source community will almost certainly be more clever about the solution that one tiny closed source company because there will be lots of heads focused on an easy to understand problem."

To be honest, I have to completely disagree. Lots of heads is not a solution to a pure computer science problem. Most companies are formed around a specific solution to a computer science problem. They then make everything proprietary and profit.

That's like saying, only smart computer scientists exist in the open source world. Not true. In fact, you simply cannot claim that there is a strong trend in one or the other. Many smart brains in open source... Many smart ones in industry as well...

View on HN

Twitter isn't a "computer science problem". It's a nice GUI for subscribing to RSS feeds; RSS feeds that can only contain 140-character posts. Oh, and it can email new entries to your phone.

actually the scaling is absolutely an interesting problem. Its the reason they keep going down. because messages are not point to point it is a much harder problem than instant messaging. I have spent a fair amount of time considering the issue in detail because a friend of mine developed a caching system targeted at microblogging. It is non-trivial, and cannot be done with RSS feeds.

If you think it's a hard problem, you aren't thinking about it correctly.

When you say "@foo Hi, foo!" You're sending a message to some person named foo. That's a push. If you start thinking of it as a pull, then you get in to the sort of trouble you're mired in.

In a distributed system the process goes like this:

1) Sender queries name server: WHOIS foo? 2) Nameserver responds foo is http://www.foo.com/mytwitterfeed. 3) Sender pings recipient http//www.foo.com/mytwitterfeed?ping=messageid&origin=bar 4) Recipient queries name server: WHOIS bar? 5) Name server responds http://bar.com/mytwitterfeed 6) Recipient requests message text, http://bar.com/mytwitterfeed?id=messageid and verifies it actually is addressed to him.

From then on, you have to worry about spam but that's a solved problem (killfiles, Bayesian filtering, etc.) No caching is involved. Recipients store messages addressed to them, just like any other messaging system. This isn't at all a hard problem.

This appears to be a system that allows strangers to coerce my client into contacting their server.

What's your point? You might as well say HTTP is a method that allows strangers to get files from my server! While technically true, it doesn't say much.

Yes, people can send you false pings, which is why you need to go back through the name server and do a reverse lookup, and once you fetch the message you need to parse it and make sure it within bounds (less than 160 characters or whatever) and addressed to you (has @foo in the message text.) and is not spam (the name isn't in your killfile, is in your whitelist, akismet says it isn't spam, etc.) Rate limiting helps too, real people aren't going to send you 100 messages a second.

IMO: They are dealing with a tiny amount of bandwidth say there are 30million 1kb text messages a day that's 900 gb / month which is well easy for a single core CPU to handle.

Personally I would go with:

User Adam sends message to Bob: 1) Adam sends message to Twitter service. "Bob>Hi." 2) Twitter check to see if Bob knows Adam. 3) Twitter sends "Adam>Hi." to Bob.

(With optimal storage so Bob and Adam can see their old conversations)

Your system let's random spammers people find out people's address which IMO is bad.

PS: Or Adam could send a message to "All>I like this soup." and Twitter then sends the message to everyone that cares about Adam including Bob.

Edit: It looks like Twitter is sending around 2million Tweets a day.

You're thinking too qualitatively; imagine the spikes they get... Also most apps are I/O restricted, not CPU...

Also, all these proposed messaging architectures are somewhat flawed. Twitter isn't really "sending" messages to other people. It's more like a person's message history is bound to their account, and appropriate privileges are applied. Then, when people try to "read," privileges are obeyed and information is produced...

Am I the only one who believes the solution to Twitter does not involve a massive distributed system? Twitter is inherently centralized... people don't see that.

The most popular messaging systems are all centralized, too.

Ok I can't edit the old post but they are basically just reinventing mailing lists. Anyway, poling is fine if it's initiated by a user action or it's a vary limited number of systems but if you have a few million users you can't uses it per user. As to reading old log's that's a trivially parallizable problem see database replication for some hints. So you separate the active messaging that automatically adds new Tweets to a user’s cell phone from the way users read old messages with the webpage and your home free.

PS: Text is cheep EX: Slashdot is running off of ~4 computers. And bandwidth is not a scaling issue until you need a single system to handle more than ~1GB of bandwidth per second. (Saturating an OC-3 line costs a lot of money but you don't need to change your architecture to do so.)

Aren't the scaling issues more in the send-to-all cateogry than in the person-to-person?

Aka - a scoble broadcast out to 20,000 different endpoints causes challenges - whether it's pushed or pulled..

> "Many smart brains in open source... Many smart ones in industry as well..."

And many of them are the same brains, showing up in different places at different times.

Twitter could have been disintermediated any time in the past year; the interesting question is why this hasn't happened already.

Reply on news.ycombinator.com