96% of the web already is not indexed. Clearly there is something wrong with the algorithm they use to determine what should be indexed and what shouldn't. All that AI doesn't seem to help them much
https://www.sciencedirect.com/science/article/abs/pii/S03064...
The abstract from that article has a good introductory summary.