If I were to work on building a search engine from scratch, I would probably approach this from these directions:

(1) Investigate if running a DNS server will help me get a more robust picture of what websites exist.

(2) Investigate if supplying a custom browser would help me to leverage client PCs to do the crawling / processing for me.

(3) Investigate if there is any point in building a search engine with the data gathered in a non-proffit way... Non-proffits are not as sustainable as for proffit corporations.