Search has changed a lot in the past decade or so. Now most results pages are full of instant answers, e.g., knowledge graph responses (which we do completely ourselves) and local results (for which we have our own index + Apple Maps and local providers like TripAdvisor). In other words, it isn’t just one index, but about 20 that really matter in terms of adoption.
Additionally, people engage with the first thing on the page about twice as much as the second thing, and that second thing about twice as much as third thing, and so on. That means if instant answers are near the top (increasingly the case) it means that those sub-indexes are increasingly more important as they are engaged with relatively more than the traditional “10 blue links."
A lot of that traditional web index is also used for just navigational purposes, which we can easily do ourselves. We are already crawling for our tracker blocking data set, encryption data set, and several other things.
In other words, we do make use of our own crawlers, and also have the ability to change some things around for indexes we use, which we do routinely. However, you need all of these indexes to make search work well, and have them all come up at the right times (a lot of what we do in search ourselves).
In this context, I think people elevate the long-tail “web index” too much. Should we have satellites in space to make good maps? It is also necessary for search. Should we be collecting sports and stocks data ourselves? Also necessary for search, but no one really calls on us to do that.
Our approach has been to partner with the best data providers in each vertical so we can provide the best alternative to Google search. And when nothing exists or it isn’t good enough, we do it ourselves or add layers on top. To that end, it isn’t binary — there is a lot in between and in fact that’s the way it works today.