But how come no other company with all their engineering prowess was able to compete with Google?
Altavista (supposedly) famously ran on a single "pizza box" Alpha server as a form of stunt marketing. There were other competitors that were invested in Oracle licenses and similar "big iron" approaches.
There was also the small matter of monetization. Google more or less refused to incorporate ads until they came up with a non-user-hostile capability. Their various competitors had advertiser-driven paid ranking, or "punch the monkey" banners, or other forms of shit sandwich users were supposed to accept. In general, the harder a search engine tried to monetize, the slower it's growth, unless they funneled their own revenue into marketing and advertising. When Google finally came up with their contextual search ads, users actually liked them and it was as if Google turned on a magical faucet that gave them money AND more users, and were able to scale to meet demand with more inventory.
Software is a business where “a few people in their garage” are fully capable of putting a dent in the universe.
Good tech is good. Good luck is better. But betting that one’s cache is permanent is a great way to end up on the “who were they again?” list.