As a content creator, what's my incentive to not block any bot from my site? Personally I think business models need to evolve along with the paradigm shift. I'll happily provide an interface for any bot to come and extract content from my site. I'll even add a bunch of context to help the lil' guy out. But I want to be paid for it. But it's not realisitc for google to have relationships with every single website. I think we need an open protocol to help build a market. Especially since some data is worth more than others (my pokemon database is not worth as much as your stock prediction database).

Know some people will downvote me for saying this, but I think crypto might be the best way to automate this. Would suck if Google or Microsoft or some startup ended up centralizing that market.