Scraping Hacker News on a Schedule with TaskPipes | Hacker News Reader

Scraping Hacker News on a Schedule with TaskPipes

blog.taskpipes.com

39 pointsf29212 comments

What Open Source projects are similar? It seems like data transformation tasks like this are a universal problem. I'd just do map/reduce, but that's the advantage that comes with being a developer. Great tools for non-developers to do limited programming tasks are incredibly useful.

Check out Diffbot @ http://www.diffbot.com/ -- you can setup repeat crawls and extract data into .csv/Excel file format or JSON with Crawlbot API, extract data automatically with automatic APIs, or Custom API Toolkit. You can get a free trial account at https://www.diffbot.com/plans/trial to try it out, and you can get paid plans at http://www.diffbot.com/pricing/.

This general category of tool seems very useful. Apparently there are many Yahoo Pipes followers. http://www.makeuseof.com/tag/12-best-yahoo-pipes-alternative...

Support for email and HTML tables seem to be some distinguishing features of this one.

I thought I remember reading that Yahoo was discontinuing the pipes tool?

I've seen something similar more than once on HN but my google-fu is failing. What other similar services are out there that'll let you smartly scrape a page into a spreadsheet.

In the past, I've tried Google spreadsheets with "ImportXML" option but got frustrated after a bit and resorted to python.

Kimonolabs.com

Allows you to scrape then import that data into a google spreadsheet. But its not the only way to consume the data.

Thanks, that's the one I had in mind. Also found : import.io, scrapinghub.com & 80legs.com

Looks pretty but I can't find pricing information.

Chatted with one of the guys

We're starting to move TaskPipes out of beta at the moment, but our pricing will start at $50/month for the service

I was looking for prices as well, or any other indicator for that matter that the tool will be available for a reasonable time at least.

I love these tools but before I invest more of my time leveraging this in my workflow I'd want to know that it'll be around for another few months at least...

I tried NewsBlur as an alternative to Yahoo Pipes. Like it because it has an Android app as well.

Reply on news.ycombinator.com