Why Twitter’s decision on Scraperwiki is bad for data democracy | Hacker News Reader

Why Twitter’s decision on Scraperwiki is bad for data democracy

gravyanecdote.com

31 pointsfrabcus16 comments

This issue is just an instance of a big problem: there is not such thing as a data democracy, only a theory. It's not about Scraperwiki only, the SLA for the free Twitter API is very restrictive (i.e: number of calls per hour).

Twitter (and others) works like a cartel, imagine that you want to pay to increase the number of API calls per hour, you don't have a straightforward way to do it. Twitter has a lot to learn from what Google did almost a decade ago: make it easy to pay and work with your service instantly (AdWords).

A little off topic, but I'm curious where the HN crowd lies on the ethics of scraping.

I'm a bit on the copyleft side -- I say, if something is made public, it's fair game. If you want to protect your data from being scraped, you very easily indicate that by putting it behind a login.

But in my opinion, anything google is allowed to access and put in the index, I can access.

I think at the startup level you can "hack the system". The issue comes when you grow.

As a practical, or ethical issue?

IE, is it just as "wrong" in your opinion to scrape sites, regardless of size or profitability? Or are you espousing the practical argument that it's fine because you can get away with it when flying under the radar?

What, a centralized service is not good for democracy? Who would have guessed!

I don't get it. Is the author under the impression that Twitter data is some sort of essential common resource? That's ridiculous. Its private and and its for sale by a for-profit company.

This isn't about "data democracy". Thats a weasel word the author used to try and sound persuasive despite his weak argument. This is about the author wanting to have cheaper access to Twitter data and whining about how much it will cost him to get that data.

Hiya! Francis from ScraperWiki here. There are a couple of things going on:

Firstly, it is about inconsistency between how developers are treated and how people-who-can-use-Excel are treated.

Secondly, it is that vetting of "data use" is so hard, it means low end users just can't buy it at all (financial and red tape reasons). Even if they are happy with the per-Tweet cost.

Speaking for ScraperWiki, we want Twitter to make more money from their data. More details here: https://blog.scraperwiki.com/2014/08/the-story-of-getting-tw...

Twitter seems to have a strategy for monetizing their data that is at odds with your company's strategy. This is not about "data democracy", its just a business dispute. You're making it sound like its ideological but its not. You want something from them and they said no. That's all.

Or, you know, you could learn to code. That would open up a myriad of opportunities for you, and help prevent this sort of loss of independence in the future.

I really don't think that will help. It is not that Twitter is insisting that people stop using (unlimited, unmetered) scraping and move to (unlimited, unmetered) APIs. Rather, it is that Twitter has stopped offering unlimited, unmetered APIs and is now charging for use.

I agree that that is the real problem--that said, the article took the stance of "Why is Twitter being so much nicer to developers than non-coders?"

Any links to information about the decision? Googling the two company names didn't turn up much. Has this been publicized?

Here is a link from ScraperWiki from the article:

https://blog.scraperwiki.com/2014/08/the-story-of-getting-tw...

Thanks!

Couldn't Scraperwiki release a tool which allowed anyone to scrape Twitter and have it upload the data back to Scraperwiki? Would that get around the SLA?

Probably not.

See Twitter's response to this blog post of somebody trying to do something similar: http://mashe.hawksey.info/2011/03/export-twitter-followers/

Even if we could, the users would have to register a Twitter developer application, making the user experience much worse than just OAuth, for no good reason.

Reply on news.ycombinator.com