Nowadays the bulk of linking goes to ecommerce sites (amazon) from content farms (reddit) and all those sites are submitted directly to Google. I don't think crawlable internet exists anymore.
You can start with seeds like common crawl, and go from there. You can also get DNS records from various providers. Then there's SSL cert logs that you can crawl. Plenty of sources, if you have funding (search by itself without ads sponsoring it might be a net loss, except some niche uses like kagi?)
You can ask Icann [0] access to gTld domain list files (if you have a legitimate reason to do so). Once access you are granted access to a gTld, you can download a compressed csv file with a line per couple <domain, nameserver>.