1998 - 2003 was one of the most difficult times to find what you were looking for, even on Google. Many searches for basic information would return results buried in spam pages, pornography, and scams.
Deleting old content to manage "crawl budget" is a myth and does not work or help your SEO.
The real problems are that Google is directing the bulk of traffic to certain brand name websites. Another real problem is that Google set a simplistic AI with a goal of increasing clickthrough from search results and decreasing bounce rates. This leads to developers building all those top 10 lists where you have to click through each item (harder to bounce that way), and some of the pages that disable the back button in various nefarious ways.
I also agree Google should be showing smaller websites more frequently - perhaps optimize for a different goal than the one listed above. More weight on keyword matching perhaps or maybe following only a few "authoritative" users CTR & bounce rate habits.