We are increasingly becoming blind. To me it looks as if this is done on purpose actually.
That's a travesty, considering that a huge chunk of science is public-funded; the public is being denied the benefits of what they're paying for, essentially.
Indefinitely? Probably not.
What about when a regime wants to make the science disappear?
Becase it costs money to serve them the content.
Is the answer regulate AI? Yes.
Because when you build it you aren't, presumably, polling their servers every fifteen minutes for the entire corpus. AI scrapers are currently incredibly impolite.
It's far from perfect but it does achieve its stated goal: of resurfacing real people on the internet.
It recently got some NLNet funding and I hope to see it flourish - to my knowledge there aren't any other projects trying to claw back control of the internet towards the commons.