Because google owns the index, if they don’t allow google to index they get no traffic. Google is a monopoly, there is no choice here, no market driving competition.
The behavior of publishers in countries where they won this battle is telling: a when laws were passed in Belgium and Spain requiring aggregators to license even small excerpts, Google stopped, and the publishers didn't take very long to offer free licenses.
[0] Here's one slightly dated example: https://www.sistrix.com/blog/new-data-is-google-or-facebook-...