There is a subset of personal websites that I have rather not have poked and prodded by everyone, yet still be able to casually access them from random places.
You cannot have a website on the public internet and hope to hide it only by virtue of having no well-known CA cert. Your deployment, should it be truly public, has to be secure even in the face of its location (FQDN) being known. Else, it’d just be a matter of time until breaching happens.
You might be interested in the ACME DNS challenge as well.
I prefer defense in depth, and it starts with not having all kinds of stuff publicly visible on Shodan with zero effort required for an attacker.
The way they're obtained is, if you operate a huge public DNS service, you collect the set of answers you saw, with timestamps but no PII, and you just sell that. If you take this data from a handful of the biggest DNS services you've got maybe 95% coverage of everything out there.
Sure 99% of users of hr.big-bank.example will be Big Bank employeees at work with local DNS, but 1% is people who are at home, or off-site or whatever and maybe it doesn't even work without VPN, but someone will click it by accident and so it gets collected.
You can't hide that example.com exists, but you can hide all subdomains for that domain.