Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.
Back before I found Kagi I used to use it everytime Google failed me.
So, yes, given he is the only one I know who manages this it isn't trivial.
But it clearly isn't impossible or that expensive either to run an index of the most useful and interesting parts of internet.
Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc.
Even though there's more than a few queries they leave something to desire, the breadth of queries they can answer is very difficult to approach.
But for me, if I want to look up local restaurants, I go straight to Maps/Yelp/FourSquare(RIP). If I want to look up releases of a band, I go straight to musicbrainz. Info about Metal Band, straight to the Encyclopeadia Metallum. History/Facts, straight to Wikipedia. Recipes, straight to yummly. And so on. I rarely start my search with a general search engine.
And now with GPT, I doubt I even perform a single search on a general search engine (google, bind, DDG) even once a day.
The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence.
You can get a long way by driving he frequently updating index via RSS feeds and social media firehoses to provide singnals for when to fetch new URLs.
This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.
1: https://www.iana.org/assignments/well-known-uris/well-known-...
Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.