Hacker News Reader

Comment by matsemann | Hacker News Reader

Isn't the quick fix that you arrive at some page, and there have to press a button or load some JS to do some action? AFAIK most email providers (like Gmail) will also visit links. Therefore you shouldn't do actions directly on the GET request. For instance if you have an unsubscribe link and all you have to do is visit that address, most of your subscribers will be accidentally unsubscribed.

Same if you paste a link in Slack/FB/Discord/Twitter whatever, they will visit the page to create a preview. GET requests shouldn't have side effects.

View on HN

> AFAIK most email providers (like Gmail) will also visit links.

I keep hearing this, but our newsletter system has been using GET unsubscribe links since at least 2007 (but probably longer), and we never found a wave of Gmail users unsubscribing, we still have a lot of them. I wonder if this is simply an urban legend, if Gmail tries to recognize unsubscribe links, or if there is something else going on.

your newsletter system probably ignore bots or some IPs

We do not :) It was never an issue there. Bots get ignored for stats, Google IPs get blocked for Ads (Google seems to think every ad link has to be visited by a ton of bots, our customers actually started complaining about the traffic)

Yup this is true - I was just being lazy. But what surprised me was that Bing actually indexed them. (even though my robots.txt said not to)

Can you prove that by linking to a Bing search where one of your pages show up ?

yeh...

https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...

The URL in the search is this: https://shoprocket.io/email-confirmation/34b35b1...

I don't see that in the robots.txt https://shoprocket.io/robots.txt

User-agent: *

Disallow: /cdn-cgi/l/email-protection

Disallow: /login

Disallow: /register

Disallow: /404

Am I missing something?

You may be seeing a stale version, try this: https://shoprocket.io/robots.txt?bypass=1

(I made a lot of changes today when testing all, including "visit as Bingbot" from their webmaster tools with and without the URL blocked by robots.txt)

> was that Bing actually indexed them. (even though my robots.txt said not to)

Never mind indexing them (ie publishing them at Bing.com), if URLs are disallowed in robots.txt then Bing shouldn't even be retrieving them, even if only to scan the content for malware!

This is a common misconception about robots.txt. It tells bots what they should do while directly crawling your site. But if a search engine gets to a URL some other way—for example if it follows a link from somewhere outside your site—it will still index that page.

Robots.txt is not a reliable way to exclude pages from search engine indexes. That is not what it is for. It is for controlling crawler behavior.

The only reliable way to exclude a URL from a search engine index is to serve “noindex” on that URL, either with a metatag or an HTTP header, or both.

This is very useful information. You’d really hope that private emails would be excluded by default…

> It tells bots what they should do while directly crawling your site. But if a search engine gets to a URL some other way—for example if it follows a link from somewhere outside your site—it will still index that page.

I must confess I've been sceptial of robots.txt for a very long time (if I want to stop bots I serve them HTTP 403 Forbidden using .htaccess or similar).

Be that as it may, it appears I'm also confused about what robots.txt does and doesn't do.

Assuming you're correct: let's say I run EvilBot which scrapes sites and want to scrape your site example.com, but your robots.txt only allows Googlebot and disallows everyone else. Am I really OK to:

1. scrape the SERPs from google.com which mention your site ("site:example.com") then 2. using that list of URIs, use my EvilBot to scrape your site, without needing to touch or respect your robots.txt, since I got the list of URIs on your site from Google, not by scraping example.com directly?

Your step 1 is enough for URLs to be indexed. Even a well-behaved search engine does not need to visit your site to index a URL, including whatever anchor text pointed at it.

If the crawler does then visit your site, it will see your robots.txt and (if well-behaved) obey it and not crawl the contents of the page at that URL. But this does not mean it will remove the URL itself from its index.

Again: robots.txt is intended to control crawler behavior, not search index visibility.

Google's page is a pretty good overview of this distinction:

https://developers.google.com/search/docs/advanced/robots/in...

> Again: robots.txt is intended to control crawler behavior, not search index visibility.

I'm obviously not asking the question clearly, I'm wanting to stop bots from crawling (it's scraping that annoys me), not search engines from listing URIs.

If I want to completely stop a bot from crawling my site (in the sense of "retrieving my content"), won't robots.txt prevent that? Even in the case of the bot having obtained a valid list of my URIs but not the pages contents from a 3rd party source?

Lets say I email you a list of URIs on my site. My robots.txt forbids all crawlers. Are you allowed to give the list of URIs to your bot and retrieve the content?

You are correct: a bot that is well-behaved (follows robots.txt directions) will not crawl your site if your robots.txt forbids crawling.

yes but for login tokens the bing bot would be able to login none the less. Probably the login url would be in some logs at microsoft or antivirus vendor. It sounds paranoid but basically the url is a cleartext password laying around.

Yeh I think you're right, an extra JS step on the page that the email link leads to would help a lot.

Reply on news.ycombinator.com