jakubbalada

> A growing catalog of up-to-date scrapers for popular websites would put of lot of freelancers out of work. I would invest in this.

Check out Apify store (https://apify.com/store). It's built exactly for that purpose.

(Disclaimer: I'm a co-founder of Apify)

You can use services like Anti-captcha [1]

We have a public API on Apify for that [2]

[1] https://anti-captcha.com/mainpage

[2] https://www.apify.com/petr_cermak/anti-captcha-recaptcha

Typically depends on what you do with scraped data. There is an interesting recent US court ruling [1]

[1] http://www.reuters.com/article/us-microsoft-linkedin-ruling/...

(co-founder here) We have hundreds of customers, half of them on recurring subscriptions, half just one-time customers paying for crawler configurations.

You might try Apifier for that, we've recently scraped more than 150k reviews for 27k restaurants in London.

Here's a community crawler you can use: https://www.apifier.com/community/crawlers/Yonny/bcYqH-api-u...

It's also hard to get direct access to the data.

But you're right it's a hard sell to enterprises although we have some (e.g. real estate developer creating pricing maps)

Both - developers on a free plan using own RSS for sites without one and business people (mainly startups) building their products on top of Apifier.

Typical use is an aggregator that needs common API for all partners who are not able to provide it. So they have running API on Apifier in an hour. It might break once in a while - than you have to update your crawler (not that often if you use internal AJAX calls).

We see a lot of users who needs data from the web or APIs for sites which doesn't have one. Just not all of them can code and we have to scale custom development.

Disclaimer: I'm a co-founder of Apifier [1].

It's not an open source, but free up to 10k pages per month. And it can handle modern JS web applications (your code runs in a context of crawled page). You can for example scrape API key at first and then use internal AJAX calls.

There's also a community page [2] where you can find and use crawlers made by other users.

[1] https://www.apifier.com [2] https://www.apifier.com/community/crawlers

If you want to scrape many websites on a daily basis, have a look at https://www.apifier.com as an alternative.

Disclaimer: I'm a cofounder there

At least in our batch we felt we should really only focus on our product. That's why we've got the grant - to be able to work on our startup full-time.

(Apifier co-founder here) It will be less beneficial now than it was in the first YCF batch. Other fellows really motivated us during the group office hours and other events. And still - main thing is to focus on a product which you can do anywhere.

You're right, price per request would be easier for estimation. But as you can use JavaScript, you can scrape whole website with just one page request (see the SFO flights example). In other words, our costs doesn't correlate with page requests, but with data transfer.

Flat fee is also possible, but we think that it's fair that users pay based on their consumption.

Of course not, API will be available soon - in a week or two. If you have some other feature requests, please let us know, we need to help with prioritization.

Yes, by default we respect robots.txt. There is a switch to disable it - on your own responsibility. We don't fully respect Crawl-delay, but minimum delay between requests for all our crawlers is set to 2000ms. We don't publish our IP ranges yet.