>> either ignore robots.txt or skip crawling your site.
That’s not really how it works now. Google pretty much ignores robots.txt now and crawls your site anyway.
That’s not really how it works now. Google pretty much ignores robots.txt now and crawls your site anyway.
They can and will crawl even if you say no, they've officially said so as well in their docs.
> A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.
https://developers.google.com/search/docs/advanced/robots/in...
They won't crawl it, but it they already did crawl it or if there's a link to it, it will still be part of their index (thus search results).
They even go further on their noindex page, by saying that it need to be crawlable (thus not part of the robots.txt) so that they can see the no index directive on the page...
> Important: For the noindex directive to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex directive, and the page can still appear in search results, for example if other pages link to it.
https://developers.google.com/search/docs/advanced/crawling/...