Web Scraping to Create Open Data
blog.scrapinghub.com
blog.scrapinghub.com
The Italian Senate offers a SPARQL endpoint [1], which unfortunately doesn't offer access to the texts of the amendments. So I had to roll my own and create a small spider for them using Scrapy [2].
[0]: https://github.com/jacquerie/senato.py/blob/master/analysis....
[2]: https://github.com/jacquerie/senato.py/blob/master/senato/sp...
1. the collection and assembly of pre-existing material, facts, or data;
2. the selection, coordination, or arrangement of those materials; and
3. the creation, by virtue of the particular selection, coordination, or arrangement of an original work of authorship.
However, if what you're scraping are not "facts", but some creative works, such as blog posts, product descriptions, etc, then it is likely to be copyright infringement.
Then on top of that, even if there is copyright infringement, other defenses such as a license to use the data, or fair use may apply.
I'm not so sure. It would definitely be illegal in the US for me to cherry pick data out of Google Maps and add it to OpenStreetMap (and OSM has policies addressing exactly this).
They scrape everything in the world they can get their hands on.
It's a sad state of affairs when i'm not even allowed to scrape data generated using taxpayers' money, like the (required by EU laws) noise maps for cities, which I'd like to use to augment real estate offers, for example.
I think it's only the UK that has copyrightable fact databases
I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem.
(For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business.)
> I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do.
It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it is a public service. On the same line, a private company providing a public service should already do so. See motives [1].
> I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem.
Same as CityBikes is doing. If we receive a cease and desist, we remove their service from our API. As for Foursquare, I do not see Foursquare as a public service. Your taxdollars at work, and all that.
I tried to keep the article balanced but maybe it wasn't clear. There are many transportation companies willing and happy to be scraped, or looking forward to provide their information for people to reuse [2].
[1]: https://blog.scrapinghub.com/2016/03/30/web-scraping-to-crea...
I do not understand that implication. How is providing bike share information creating a competitor? I can't run a bike sharing service.
If your offshoot were to misrepresent data, for example, then you would become a liability, even if you weren't making money.
If a website does not have an API (BuzzFeed), I take care to only collect data that I need. Not anything that would damage the business. (E.g entire articles). Consequently, I sanitize the data of such things if I decide to release the dataset.
Heh.
Ok, but seriously, I look forward to seeing their appearance on r/drama when twitter discovers this.
Here's what they have to say about it, and why there's very little chance they will change anything:
http://laurent.bachelier.name/2013/12/weboob-the-asshole-det...
(This comment is not directed at you directly)
Seems like a bad name then, no?
But with Application names like 'Handjoob', 'Boobsize', and 'Flatboob', I think the naming is unavoidably distracting and inappropriate.
------------------------------
Full text:
We often get complaints around Weboob‘s name, and the various application names.
There’s no denying they’re childish. What they are not, however, is sexist.
There is “boob” in the main name, and “boob” is a friendly name referring to (mostly female) breasts. We would, for example, avoid using “tits” or “cunt”, because they are often demeaning1. Though it is a happy accident (our earlier ideas like “woob” and “webob” were taken), we certainly like playing with that.
The idea is the same with application names; it’s all about friendly jokes (like wetboobs the weather tool, which manages to be related to weather and boobs).
If you’re offended, just ask yourself “how is it sexist?”.
As it appears, Weboob is a formidable tool to detect people that are part of the “be offended first, think later” crowd. Interestingly, the crusaders2 are to date all male, and often assert that women can’t like jokes about breasts or sex in general3. How fucked up is that?
They will always make a scene4 on how they’re never going to use Weboob because of names. Guys, here’s the thing: we don’t need you and we certainly don’t want you. I for one am glad we created an Asshole Detector, albeit by accident. We however are mostly not native English speakers. Mistakes can happen. [↩] This is not an euphemism. They act like they are fighting for a good cause, but it’s only pretend. [↩] And who the hell are they to talk in place of others? That is actual sexism. [↩] So that it is abundantly clear, this is purely about making a scene; I do not care about their opinions or how many penises they may have. All our contributors do not and do not have to like the branding. [↩]
Edit: it directs me properly in an incognito window.
When I click, I'm redirected to a simple image. When I request it in incognito mode, I'm redirected to a "crash safari" url.
I'm not offended by the name, I get the joke, I just think it's a stupid joke.
I think it probably /is/ an asshole detector, but not quite in the way that the authors of that webpage /think/ it is.
from Wikipedia:
Psychological projection is a theory in psychology in which humans defend themselves against their own unpleasant impulses by denying their existence while attributing them to others.
That in itself is a very good reason why the naming scheme needs a long, hard rethink. It's distracting.
Put another way, I wouldn't blame everyone else if every time my hypothetical company "Natzie" was mentioned the conversation devolved into something unrelated to the reason it was mentioned. If you want to succeed, you need to make decisions based on the real world, not how the world should be (not that you can't push limits, but baby steps are often needed).
Only problem with using pybikes natively are some systems we call asynchronous. These are bike share websites that require actually clicking on a station to get the status information. This means that for getting accurate information on the entire the feed (let's say 500 stations), you would have to go through each of them. On these cases, it's way easier to just use the API.
[1]: https://gist.github.com/eskerda/bbd65539048a53eadfccc5d535ad...