maybe it's time to centralize crawlers and search data, making it publicly available to anyone willing to process it.

Not only would that reduce crawler traffic on websites to a single crawler entity it would make page data available for any indexer.

The idea that one company owns this data is kind of silly, it should be a coalition or a group of companies working together...