How long will it take for Google to index this?
google.com
google.com
Of course, it may have been implemented in other ways.
Ultra-fast indexing depends on posting just before the crawler hits the index page. A lot of luck is involved to catch the crawler like that.
Each notice of indexing should alter the query text based upon the comments at the time.
In their excerpt, they note that it was posted 21 minutes ago. So that seems to be their score.
Like many of us, I've already tracked the time it takes for Google to index the changes to my own sites - about 14 hours. I work on sites that the Google robots hit only once every 2 weeks - less content, and fewer updates, though some of them earn 10 times what my more 'frequent' sites earn.
Good O, Hacker News! Which means sweet F all to me, all of my customers, and pretty much everybody else on the planet.
By the way, how long IS a piece off string?
The purpose was to measure a phenomenon I'd previously observed -- that it's getting harder to find answers on Google, because people post their questions on well-trafficked sites, and within a few minutes, those sites rocket to the top for a well-crafted search query trying to answer the question.
"By the way, how long IS a piece off string?"
I don't know, but strlen(piece) probably does.
In that sense, I think we can cut Twitter some slack...
Google manages to route large messages between GMail users instantly, without breaking a sweat, and with 100% uptime (so does Microsoft, Yahoo, and every other major email provider). They have no trouble dealing with users who send or receive thousands of emails a day, while twitter claims this is some sort of pathological, borderline-impossible case to deal with.
And GMail is relatively trivial compared to actually indexing the internet. The last decent estimate I saw (2005 - http://www.cs.uiowa.edu/~asignori/web-size/) suggested the indexable WWW is ~ 12 Billion pages - and god knows how much that has increased in the last four years.
Seriously, make sure you're in the right ballpark - hell, in the right galaxy - before talking out off your ass. Twitter's problem should be easy today - it was solved decades ago. The fact that Twitter can't scale is nothing more than a testament to their incompetence.
But I have also thought about mail, perhaps it would be easiest to replace Twitter with "micromailing", and use existing software solutions. Difference is of course the "only receive messages from people I subscribe to" thing, but that should only make it easier.
Edit: also, google mail doesn't do polling, so maybe it is very different after all. It's a completely different architecture. Maybe Twitter would work better with an inbox-per-user model, but I wouldn't bet on it.
Another difference: most emails don't go to dozens of recipients at the same time. Tweets often have several hundred recipients.
If your blog supports providing a dynamic sitemap, and notifying google that the sitemap has been updated, it basically does.
Check out: http://wordpress.org/extend/plugins/google-sitemap-generator...
On your other points, I'd be pretty surprised if you couldn't provide a twitter-type service, at twitter-type message volumes, using e-mail as the hidden back-end. I don't think hundreds of local recipients would be that big of an issue, especially at twitter message rates/volumes. I'd be willing to bet it could be done pretty easily with a small number of decent servers. Then again, it could be that the Twitter folks know something I don't. Or they're just too emotionally invested in their existing back-end. No idea.
And yes, it is true that SMTP doesn't use polling. But that is because polling for messages doesn't scale very well - a smart 15 year old could tell you that. I don't how twitter's architecture works, but if it polls all a user's subscriptions each time that user logs in then I say that was a bad architecture decision.
It would be pretty straightforward to emulate subscriptions with a 'mailing list' type solution. The simple fact of the matter is that Twitter chose a poor architecture. If they simply used an inbox/mailing list type solution all their problems would be non-existent and twitter would scale, near linearly, as you increased the number of servers.
Yes, it would be grossly inefficient HD space wise - but who cares? Normalization (seems to be) what got them in this mess. And disk space is cheap enough that it doesn't really matter.
Assume Twitter had a million active users (they don't), and that they each sent 5 Tweets/day on average (users don't), and that each user was subscribed by 5 Users on average (they aren't). That means about 5GB of Tweets are made a day. Using an inbox type scheme (with absolutely no normalization - which is unlikely) that would only require about 25GB of storage a day. Given reasonable cluster costs (and assuming data is 4x redundantly stored on the cluster), that would require well under a $100K/year in storage expansion (to store all tweets forever). Using semi-intelligent caching and partial normalization I think it would be pretty easy to cut those storage requirements in half w/o breaking a sweat.
I say again Twitter's problems are trivial - they just seem to have (initially) made a lot of poor design choices, and are now living with the consequences.
Just a note: email does polling, too. Most email clients poll the server on a regular basis. But that is usually every 5 minutes or less? Twitter has just limited polling to 30 times per minute for external clients, if I remember correctly.
I haven't thought too deeply about it, but I wonder if a mixture between polling and pushing could work. Either it could be decided on the nature of the client if it should push or poll (ie how many subscribers does it have, how active is it), or clients could connect and say "please push for the next 20 minutes" or just send keep-alives. If client alive, push, otherwise let him poll? I doubt that polling could be eliminated completely. If an email server is offline for an extended period of time, the message never arrives. Mabye that is acceptable, though.