Why not use robots.txt instead of littering your html with googlebot instructions?
Combined with the fact that Google might list your site [based only on third-party links][1], robots.txt isn't an effective way to remove your site from Google's results.
Sorry, could have been clearer.
[1]: https://developers.google.com/search/docs/advanced/robots/in...
Seems clear enough to me
“ For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:”
The first option is the meta tag. It does mention an alternative directive for robots.txt, however.
Google crawlers IPs https://www.lifewire.com/what-is-the-ip-address-of-google-81...
> If other pages point to your page with descriptive text, Google could still index the URL without visiting the page
"If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex. "
> You can prevent a page from appearing in Google Search by including a noindex meta tag in the page's HTML code, or by returning a noindex header in the HTTP response.
Source: https://developers.google.com/search/docs/advanced/crawling/...
they won't show your site's content, but that doesn't mean they won't use your site's content.
I'd review the other usage on a case by case basis; e.g. determining ratings of other sites seems fair use to me. I'd guess you're allowing others to use your site's content when you're making your site public (TINLA).
You want no index robots tags on all your pages and let google see those.
You can use GSC (Google search console) to remove a site / page from the index
You can even tell which bots are allowed to index and not.
I also happen to ban Google ranges on multiple less-public sites specially since they completely ignore robots.txt and crawl-delay.