After Kimono got shut down, I think a self-hosted open source version would be extremely popular. I want to build my own solution, but the API functionality and pagination / AJAX loaded data would be too difficult.
Otherwise, I'd recommend you check out Portia (open source). We're in the middle of releasing the beta 2.0 version.
Is this even a realistic business model? Seems like this is what Scrapy is doing and what Import.io is doing. Make the tool free in order to get free marketing and then charge people willing to pay money to extract data.
Meanwhile I see Mozenda charging like 5 cents for each page extracted, do you think this is a fair model or does it not matter?
Charges come with large scale crawls (above certain limits on our platform), additional products like Crawlera (our smart downloader that routes requests from a crawl through a pool of IP addresses to avoid bans), datasets, and for us to handle complex crawls for companies outsourcing to us.
Our model is that there is something for everyone whether you are looking to dip your toes into web scraping (free), use it occasionally (usually journalists) or dependent on web crawling for your business.
Right. I had first come across Scrapy, while browsing the web for Python software tools, some years ago, on the site of a company in Uruguay called Insophia. It was in the list of products developed by them, and that they worked on. Scrapinghub came later.
Horrible industry imho when you have to give away things for free just to be competitive. I just don't get why people would expect software to be free.
We provide the best Platform (as a Service) to run Scrapy or Portia spiders, and will soon be supporting most standard web scraping technologies. This is free for light users, but we charge for people who need extra or dedicated computing or network resources.
We also provide help to startups or enterprise orgs looking to get help in building a web data harvesting system (more than just parsing pages!), either by building it ourselves or by helping our partners train their engineers in using our technologies.
This has worked so far, and we're very healthy from a revenue perspective – more than doubling every year for a few years now, and good enough to grow to become the largest fully distributed company outside of the US.
We're pretty happy with being a brand that gives to the community, it tends to get repaid 10x in the long run.