Ask HN: Where do companies like outpan and semantics3 get their data from?

Outpan here,

We use a combination of polite massive-scale web crawling and user contribution. We had an in-house web crawler for years until recently when we released it as a stand-alone service.