Hacker News Reader

Comment by alpb | Hacker News Reader

Surprisingly this person didn't even bother adding a noindex/robots.txt measure to the website.

So "deleted forever" is only as good as someone not saving the page to the Web Archive or archive.is etc. Then it's cached forever in archives, almost as good as notarized proof. http://web.archive.org/web/20210208070536/https://bruh-are-y... To the author: please don’t make promises you can’t keep or meet.

View on HN

Really useful comment that I had to think twice not to downvote because of its snarkfulness.

After reading your comment, I realized that I had to do this on my own site as well. Harsh but valuable comment (to me).

Mine got nuked somewhere along the way too.

Archivers aren’t going to crawl it unless someone actually provides the URL. At which point it’s the same as screenshooting and putting somewhere public, no?

Hey, thanks for letting me know this. This was definitely an oversight by me and I'll add this in.

Can’t you just ignore robots.txt?

Well behaved crawlers don't.

Products like Google Search and Bing have a self-interest in supporting robots.txt because the alternative is annoyed webmasters suing them for not providing an opt-out due to which their unsecured credentials are a quick search away.

In fact, I believe the robots.txt protocol was devised in consultation with search engines.

A lot of webmasters are amateurs who expose their credentials even when robot rules are respected. Google dorking is a way to find that credentials and other sensitive data and information.

Unless I’m mistaken, google don’t actually respect robots.txt any more. They recommend the use of some meta tag instead, from memory.

Their docs say they do: https://developers.google.com/search/reference/robots_txt

They do say that they ignore robots.txt for user-initiated actions eg. Google Translate, which makes sense in my opinion.

Ah yes you’re right, my apologies. They don’t support the noindex directive inside a robots.txt any more.

Reply on news.ycombinator.com