Ok, so in general I agree with your statements, but...
> how they would build a reliable business around arbitrary data structures which they have no control over
Google Search? The entire web could be described exactly like that.
> how they would build a reliable business around arbitrary data structures which they have no control over
Google Search? The entire web could be described exactly like that.
Another point here is that Google Search isn't an authoritative source of information, it is up to the end user to inspect the returned links and decide if they can trust that site. This is something that I would not try to automate to the point that I could ask users for money in exchange, and if it can't be automated it doesn't seem like a great fit for Google.
†: Parses unreliably at best, Turing-complete at worst. (aka javascript if you didn't catch that)
Nah, the user's mood may be ruined if the search results are junk, but ruining the the model you're trying to build has vastly more costly consequences. I don't think the tech is there (just yet) to have some code simply ingest whatever comes its way, chew it up and use it well; required xkcd (today's!): https://xkcd.com/1838/