I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
Where do you get stats on this info? Not refuting your claims but it's interesting data. I would've thought a wider gap in all biz to active websites since some countries run nearly all their biz on whatsapp/telegram/wechat.
I did count the number of ICANN domains on their zone files on czds and there are a few services selling pre compiled domain lists that do crawling for marketers (it is forbidden by ICANN to use zone files for marketing) I don't remember how I got the active website count of 100 to 150 million, maybe I interpolated by checking some ICANN domains or there's a domain list service that does it. Domain list services usually say how many domains are active regardless of whether they have a website or not
Of course parked domains add up to the active website count even though they have no relevant content
What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.
Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence.
Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing.
It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later.