Hacker News Reader

Except don't they have a special deal with Google to use the firehose?

IMO, that's exactly the reason. Before search engines could scrape the data and load the content for free. Now they'll need to reach firehose data agreements.

Take a look at the robots file without the "www" subdomain. Its likely to prevent content duplication/push for only the non "www" url to appear on search engines.

https://twitter.com/robots.txt

Uh... Excellent point. Seems the entire speculation is/was unfounded.

So only firehose agreements get data? What's the impact to general SEO or 3rd parties? Doesn't sound good.

Actually i believe its only from the "www" subdomain. Take a look at the robots without the "www"

https://www.twitter.com/robots.txt https://twitter.com/robots.txt

This is likely just to prevent content duplication/nudge users to visit without the "www".

yep makes a huge difference... as explained here as well http://webmarketingschool.com/no-twitter-did-not-just-de-ind...

You should never use Google's estimate as a real estimate, especially once it gets past 100. There are better ways to move content, like 301 redirects or rel=canonical.