My understanding is that .com (commerce) prevailed and people who have websites want to be fully public, visible and searchable.
On the other hand deep web and dark web exist but most of the people who use it and operate there do it because of shady business.
I don't entirely understand your search engine idea but if internet would start all over again I think the right model for search engine would be to have website owners submit their websites to the search engines along with metadata so searching experience becomes rich.
Metadata would be similar to something like Resource Description Framework https://www.dlib.org/dlib/may98/miller/05miller.html