Hacker News Reader

Comment by quotemstr | Hacker News Reader

You know, back before APIs were all the rage, people wrote clients for web services by scraping. Twitter really wouldn't be able to do anything about a Twitter client that pretended to be IE9.

View on HN

I've had a bot blocked by Myspace when I resorted to that. They started redirecting all my requests to " http://google.com ". And I was just working on an old Thinkpad running Ubuntu server in my dorm room, so I have to think blocking me was an automated response.

I'm sure Twitter knows how to block crawlers, especially ones that they're actually aware of.

I'm sure Twitter would block bots doing unuserlike things. But isn't the point of a Twitter client to do userlike things?

I suppose, and I suppose if the client used a user's IP they couldn't block them the same way. But wouldn't a client that scrapes the full HTML to get the data you want be horribly inefficient, and hard to maintain? API's are all the rage because they give you just what you want. Think about bandwidth.

Having written scrapers in the past, I can tell you that changes are usually pretty easy to deal with. And if you write your scraper nicely, many small changes won't even matter.

Since the client runs on the user's machine, performance isn't really an issue, either. (Trying to do it for all users on a server is a different story.)

APIs are indeed "all the rage" because they're easier and don't change as much. But when given no other option... You do what you have to.

I think the obvious solution is to have the client first check for updated scraping rules from a server operated by the client's creator. The client could check for these scraping rule updates every time the client starts up or on some interval (or maybe after certain errors even) and only pull down the whole thing when the cache is invalidated. Then, as you suggest, the client could do the actual scraping which would make it harder for twitter to block.

Not as straightforward as an API to be sure, but not impossible by a long shot.

To what end? You're going to base a business on scraping twitter.com? That sounds like a much worse idea than trying to work within the API.

> You're going to base a business on scraping twitter.com?

A business? Probably too high risk. A nice FOSS client, though? Why not?

Why not? Because of liability to law suits for breach of the terms of service or copyright violation. Third parties have been sued for scraping in the past.

You seem to have gotten downvoted without comment, so let me chime in. He's proposing a FOSS client. There's no third party here, the client is using this software to breach their own ToS (supposing there's a clause in there by which you're only allowed to access twitter.com from a browser).

This would only work for the public timeline. Most people would want to view their own timeline and that would be hard to since you'll have to create a browser instance for every user you have.

But a client would just run on their machine. Save cookies, and you're fine.

Considering how much javascript is involved in current twitter and how often can subtle things in markup change, I'd say it's just too brittle to rely on.

Javascript-filled sites are often cleaner to scrape. Usually, at some point in the process, a nice JSON structure containing all the useful data exists.

Twitter would blacklist the IP(s).

Reply on news.ycombinator.com