You’ve almost nailed it!

“It survived because it was something that only academics cared about”

The critical change has been in the number of academics, not in their level of care. It was always a small exclusive group. It can only work if it’s a small exclusive group. We can’t make everyone an academic and expect quality output anymore than we can make everyone a coder and expect great software. Academia (higher ed) simply cannot scale and still be what it’s been. Great wine must be produced in small batches, perhaps the elite should be too.