It comes down to one thing: Money.

The older internet was not a network of for-profit users trying to rent-seek on every digital interface & every page view. Standards were created by academics, internet committees, and concerned & involved users, trying to figure out what would work best as a resilient distributed system.

Now what goes on online is defined by the rent-seekers, the data pimps, and the predators. Their actions have real-world, legal consequences which ruin it for everybody, and their size & clout overwhelms the decisions of average people.