ScraperWiki - an online tool to make scraping simpler and more collaborative
scraperwiki.com
scraperwiki.com
The front page is a bit split-personality: you have the idea of data mining; and the idea of sharing data sets. I think you should pick one to be the dominant focus, with other one present as an aside. It might make sense to focus on producers at the beginning ("scrape data and share it"), and once you've built up datasets, switch the emphasis to consumers ("wikipedia of datasets"). Maybe it's worthwhile to check how similar producer-consumer websites did it at the beginning: wikipedia, youtube, flickr.
BTW: What do you think of an interface like excel has for web-scraping: it displays the webpage, and you select the bit you want visually (no coding, not even html)?
It looks like they have thought things through pretty well, but I looked around and didn't find interesting data or useful code.
screen scraping is a bot that pulls the visual data from the screen and analyzes it. A bunch of tutorials talk about web scraping and call it screen scraping.
# Built in source control
really???
Or, if python is your bag, there's a scraping lib called scrapy that was opensourced last year that's ok.
On ScraperWiki, they need source control as the environment is actually run on their servers, I'd guess a bespin type implementation, so you put all your code on the wiki, and it can get augmented etc. hence needs SCM. At least, for the moment, it seems they've got plans of releasing their api engine or at least calls to it in which case you'd end up doing the code locally, and can use your own SCM.
Their confusing naming could be explained that their not pitching this at coders, but rather journalists to try and get them to use the vast data resources out on the net.
A more useful definition would be "extracting structured data from human interfaces"
Thanks for the link :)
Something like that -- where the community could then develop cross-platform tools to implement it -- would truly be ground-breaking.
(I don't think this is related to the nature of the html/regex debate, but I'd be happy to be educated)
Too many providers are taking data that the user knows intuitively that they own and sticking it behind a wall. Anything that helps get that data back out is awesome in my book.
There use to be a website for sharing "parselet" scripts.
We (tectonic and I) also wrote an in-browser IDE, and a Javascript-driven web crawler that runs on headless Firefoxes on EC2. We wrote something similar to scraperwiki, that integrated a simpler version of the IDE.
We got a couple consulting deals, building smart web crawlers for clients, and about 15 passionate open-source users. I think the primary problem is that unless you're scraping many sites, it's easier to write 50 lines in Ruby/Python/$language_you_already_know, than to learn a new langauge and cut it to 5 lines.
If there's interest in digging this project up, please contact at kyle@kylemaxwell.com.
Not sure I'd want to pursue it right now, but shit, it'd be a game changer on all kinds of levels. Especially if it allowed two-way communication.
EDIT: In fact, I know just how I'd deploy it -- as a shell mod in windows. You have a drive Y which is really a NortonBackup, no reason why you can have drives and folders that represent any sort of online service you have that stores your stuff -- FB updates, Twitter, etc. Let the O/S worry about synchronization and all of that. Why read 27 ads for FB games when you can just go to X:\Facebook\FriendsStatus and read all of the unadorned status updates, which is why you visit anyway?
Using my Markham's Estimating Tool (double the number and go to next higher units) that comes out at 6-8 weeks.
Windows shell extensions are all COM nonsense. You have to know the magic numbers and know where to put them. The rest is pretty straightforward code.
This might make your app robust against changing data formats, but would also leave it vulnerable to vandalism, which could potentially be far more damaging for this site than for something like Wikipedia.
But, unlike wiki pages, scrapers don't need to improve incrementally; they either work or they don't. So a community verification step before a new scraper goes into "production" might be a feasible way to deal with vandalism.
Anyway, the IDE approach looks very very nice. I have done similar thing before, but using Selenium/jQuery: http://github.com/tszming/Selenium-Google-Scrapper. I still believe my jQuery approach is more flexible for screen scrapping :)
We're actually talking to them about hosting one in South London in the next month or two.
Not sure what their plans in the US are, but it could be worth dropping the idea to some fellow journo's and seeing if there's an oganisation willing to act as host.
EDIT: forgot to mention, if you're interested in data you might also want to take a look at the open data initiative (data.gov.uk) and the guardian's data store http://www.guardian.co.uk/data-store apologies for the UK focus.
http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciat...
http://blog.ianbicking.org/2008/03/30/python-html-parser-per...
I recently switched from BeautifulSoup to lxml. It's tons faster, and I like it quite a bit more than BeautifulSoup. The documentation isn't quite as good, and it's a bit of a pain to install, but if you have to do a bunch of scraping, I'd check it out.
It supports CSS selectors, jquery-like DOM traversal and set manipulation, etc.
(Edit: Which makes me sad, because I loved their documentation writing style)