Whereas this may be a common reason for failure, we failed because we were all engineers. Nobody with management/sales experience on the team!
26 karma · joined March 18, 2019
Common Crawl (non-profit): Stores regular, broad, monthly crawls as WARC files. Provides a separate index that can be used to look data up (no a fulltext index though). Used mostly in academia.
Mixnode (for-profit): Regularly crawls the web and lets users write SQL queries against the data. Not sure who the primary users are since it's in private beta.
There are some search engine APIs, but I don't think the conflict of interest would allow for cost-effective large-scale access and pricing...