Hacker News Reader

Comment by K0nserv | Hacker News Reader

> 1. they are crawling potentially sensitive information granted by links with tokens

Don't put Facebook tracking on sensitive pages. Actually as a service to your users don't put it anywhere where it doesn't add value.

> 2. they are triggering potentially harmful and/or confusing actions in your website by repeating links

They only perform idempotent[0]* requests which should not have any negative effect if performed multiple times

0: http://restcookbook.com/HTTP%20Methods/idempotency/

* They probably only actually perform GET in reality

View on HN

> Actually as a service to your users don't put it anywhere where it doesn't add value.

So don't put it anywhere.

I believe the tracking stuff comes with Like buttons and other Facebook widgets those are what I am referring to when I say "add value". But it could be argued that the tracking alone never adds value to the user.

It could also be argued that not even the like buttons add value

It could also be argued the exact opposite.

GET is only idempotent in theory. Way too many people abuse GET when creating websites.

As an example of the problem, see the "Issues and criticism" of the related topic "Link prefetching" in Wikipedia - https://en.wikipedia.org/wiki/Link_prefetching#Issues_and_cr... .

Almost all of those problems only apply to in-browser prefetchers, which reuse the user's session and connection.

Point taken. Thanks for the correction.

I believe that is the point that @K0nserv is making. The 'abuse' part I mean.

Those people should suffer the consequences.

I'm not a fan of facebook in the slightest, but they are crawling websites they were essentially invited to.

>websites they were essentially invited to

Using an analytics pixel is _not_ an invitation to crawl a website.

No, merely posting it on a public server was the invitation

It is not. Leaving your door unlocked is not inviting everybody in to take your stuff. You might make it easier for them to break in, but it still is a break in.

Making something available to the public is not the same as going to the google webmaster tools and telling them to index your page.

Someone viewing a webpage you put online is not at all like someone stealing something you own.

You can argue it's abuse or illegal or fraud or whatever you want, but here's the thing: how are you going to stop them? Sure, maybe you stop Facebook with a lawsuit... but everyone else is still doing it, even people outside of your legal jurisdiction. They're still going to do it, so it's up to you to stop them with your design. If someone breaks into your website and destroys a user's data or steals their credit card, that user is not going to want to hear "but what they did was against our ToS!"

This isn't your house where there are police patrolling and ready to respond at a moment's notice when they're called. This is the Internet, accessible by almost literally everyone on the planet, and they don't give a shit about your policy. That's why best practices and application security was invented. So use it.

"Hello, I am a HTTP client, can I have /some/super/secret/page?" "200 OK, here it is"

That's your server complying with the request. Whether by intent or by oversight, doesn't matter: the client comes and asks, and your server can refuse. If it complies, well, you told it to. Whether you have merely exposed the page to the public or also shouted its URL from the rooftops, that's completely irrelevant. If it's not supposed to be public, don't make it public.

"Hello, I am a HTTP client, can I have /some/super/secret/page?" "Oh, but you are ^User-agent$=.acebook ? Nope, 403 Forbidden, no data for you." (Or, more generally, "And who are you? 401 Authorize!" - or any other sort of mandatory access control)

thats like saying "having a public website is an invitation to DOS-attacks"

there are conventions and reasonable expectations, until now I did not expect that a tracking-pixel would be the basis for crawling, so far most crawlers tend to crawl whats publicly linked, not whats potentially publicly reachable if one knows every url there is

Posting a file to a public web server is an implicit invitation for clients (human or automated) to download that file. That's why "secret urls" are universally considered to provide very little security.

There are common conventions (not always followed) around robots.txt and what files to crawl, but I'm not aware of any rules or conventions or standards around URL discovery. Plenty of crawlers attempt to crawl every registered domain name, for example.

"DOS Attack" is sort of a loaded term since it implies malice. Clearly running a web server doesn't mean you invite malicious attacks (though perhaps you should expect them). Some people consider Googlebot to be a DOS attack since it can easily bring poorly designed sites to their knees.

I watch my site's Google index and I can tell you 100% I never gave Google explicit permission to crawl 90% of the pages that show up there.

Even according to that document, idempotent methods still can update resources as long as the representation doesn't change:

> Again, this only applies to the result, not the resource itself. This still can be manipulated (like an update-timestamp, provided this information is not shared in the (current) resource representation.

This means that tracking is still could potentially affect some stuff, but honestly not by much.

I fixed it:

Don't put tracking on sensitive pages.

Reply on news.ycombinator.com