Hacker News Reader

Comment by me_bx | Hacker News Reader

A few years ago, the free search API could retrieve tweets since the beginning of twitter (although twitter did not guarantee completeness of the results): https://web.archive.org/web/20150311065138/https://dev.twitt...

Twitter then limited this API endpoint to data from the past seven days only, and now just introduces a paid version for only last 30 days data.

I find it fair to charge for elevated API access. On the other hand, the 30 days limit for the paid version is pretty ridiculous. Makes it more worth to just scrape the free online advanced search...

View on HN

That's categorically not the case - the index for the search API has always been around 7 days since Twitter acquired Summize (which is where the search API came from). There was never an artificial limitation such as the one you're describing. From the page you're linking to from the archive:

"The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets."

Scraping the Twitter website is against the terms of service and may result in permanent suspension of access.

> Scraping the Twitter website is against the terms of service and may result in permanent suspension of access.

Scraping is always against the ToS, so no surprise there. I doubt it's really ever stopped anyone.

i would imagine it stops researchers.. who usually try do things legit

It does. We contacted them seeking academic discounts, and there wasn't. I was in a little university in Spain. No one is going to drop 20k like that.

My bad, my memory failed me, I thought remembering being able to perform search on the historical tweets via the search endpoint.

So is using it to incite violence. The Twitter tos isn't worth the paper it's written on.

Scraping is how the web works. After all, retweets are essentially manual scrapes

> Scraping the Twitter website is against the terms of service and may result in permanent suspension of access.

And what is Twitter gonna do about it? I never sign the TOS and scrape as much as I want.

> may result in permanent suspension of access

A.k.a.: they will try their best to IP ban you.

And then if you circumvent that, perhaps bring charges under CFAA.

Would work only if you are based in US thou

Yeah screw people accessing public information...

Well, that’s kinda the whole point. They don’t feel like it is truly public information. You gotta give something in return.

I wonder if this has changed in light of the recent LinkedIn decision?

Reply on news.ycombinator.com