But that is irrational. Selling data is worst possible thing a company can do, because anyone can buy the data including Facebook.
7,364 karma · joined October 3, 2010
But it does amaze me how often HN seems to fall for this kind of sensationalist story.
But they still seem to be on good terms with the client. If I understand correctly the problem is that they insist on getting paid through upwork:
"When things were not making sense, I still followed the rules. I could’ve easily gotten paid from Robin outside of Upwork. Heck, he was physically in front of me. "
If I search for a news event it's a news site.
If I search an error message, I know the result is going to likely be stackoverflow, github issues or the forum of the library.
etc.
I don't think this strategy will get you all the way there, but it could be combined with other ways of curating sites to crawl.
Do you have a cost estimate? Also could you be more selective in indexing, e.g. by having users requests sites to be crawled.