Ask HN: Has anyone used web-scraping-as-a-service that didn't suck? | Hacker News Reader

Ask HN: Has anyone used web-scraping-as-a-service that didn't suck?

6 pointspeteretep11 comments

I'm trying to be a little more commercially minded, and avoid writing my own scraping infrastructure and maintaining a bunch of scraping integrations for specific sites I want data from.

There seems to be a large number of web-scraping-as-a-service products out there, and very little to recommend between them. Has anyone used one which didn't suck, or am I likely to need to write my own scraping tools?

View on HN

I heard good things about https://import.io/

nah! it's not there yet.

Everyone I spoke with about Import.io said it was pretty inconsistent.

Give Feedity - https://feedity.com a try. Drop us a message https://feedity.com/contact.aspx if you have any questions or concerns.

I got great experiences with http://scrapinghub.com/, you can hire them as consultants also and short time contract is enough to get in the right direction. really smart people

Seconding this question... I've wanted a way to get HN top posts each hour, aggregated (and deduped) by day/week, so that I dont have to feel the need to check as often. Thanks for this links, hopefully one of these will get me there!

You don't have to scrape HN, it has an API. It's linked to at the bottom of each page.

I've used http://www.mozenda.com in the past. Worked on some fairly complex Web directories with multiple levels.

My colleagues at https://www.archive-it.org do a pretty darn good job

work with a group that use Mozenda - I haven't worked much with it directly. They made it through a couple years of scraping with Mozenda. They recently decided to spend more and installed Kapow. More mature platform - lots of features but also a lot more expensive if I understand correctly.

it depends, tools like Kimono can gather data on a topical layer, but cant really interact with the page(via js). I wanted to scrape some data on a site I was interested in, but I need to interact with the site to get the data, so I ended up writing my own crawler

Reply on news.ycombinator.com