Hacker News Reader

Comment by latchkey | Hacker News Reader

There is some fun stuff in there, like this?!

        Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-0
	Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-1

View on HN

Supposedly a honeypot: https://news.ycombinator.com/item?id=6573338

Also appears they block archive.is outright (which ignores robots.txt).

Can anyone explain how that honeypot works? i.e. how would anyone try to go to this page?

robots.txt is a publicly accessible list of pages that bots aren't allowed to go to. It's like publishing a list of home addresses you don't want robbed because you have something good there. So, some bad crawlers might go to the robots.txt since it tells you where the good stuff is hidden.

(In reality, it's likely nobody would put actual secret URLs in the robots.txt, but some developers might without thinking. This is likely just one of many ways Yelp attempts to catch bad webcrawlers.)

I think it's more likely that they have hidden links and references to these URLs scattered across their "normal" pages. A human would never click on them, but any bot that's just blindly following links would stumble across them, which alerts Yelp that it's likely a bot accessing the site.

To catch and block scripts scraping Yelp in ways they don't like.

If you fetch that url, you are most likely a bot that isn't following the rules and can be flagged or blocked outright without impacting any real users or search engine rankings no matter what your user agent claims to be.

I see. Thanks, but then why so many?

To make it hard for people to write code to get around it.

Well, wouldn't that only be very effective if Yelp update this file every so often? But this line was added as early as 1425 days ago (https://news.ycombinator.com/item?id=6573338).

I assume that it's supposed to be regularly updated, yes.

I wonder if any of their pages has a hidden link to one of those entries in robots.txt

Archive.is is designed that way, though. It performs requests on behalf of a user, acting as a proxy and setting X-Forwarded-For with the user's IP address.

So I can load that url on a high-traffic page and IP block all visitors to my page from yelp?

Be careful w/ these urls - I visited one and it appears my IP is now blocked from accessing yelp.com.

That's really interesting. They might be trying to blacklist scrapers that don't properly respect robots.txt files.

I assume the GP was using their browser to access these links, though they still be blacklisted by Yelp?

Scrapers can be pretty shady, so they're isn't a good way to ensure that web traffic is coming from a legitimate human using a browser. To the server it's all just bits on the wire.

Why would a scraper who doesn't respect robots.txt be accessing that file?

If I was building an unruly scraper (which, in case our new overlords are listening, I would never do), I would read robots.txt so that I had a clue where the secret information that the company did not want me to read was located.

I'm not allowed to look in /documents/source/? Perfect. Let's start there.

They might be using it as a way to find specific pages which have content Yelp doesn't want you to scrape. The "evil" scenario.

They might also not be looking at the file, and just appending random words to the end of Yelp's biz URLs to scrape every business. Which, at some point, it might hit that URL by accident, since they are all words you'd find in other business URLs. Though this seems less likely.

What do you suppose would happen if we all visited these URLs from every public access point we use? (edit: and shared VPNs, Tor, corporate networks, etc.)

I just visited all of them: the IP was not blacklisted

Worked on my laptop, but when I tried the same thing on my phone, I was not blocked. (even when I request the desktop version of the site)

Because they whitelist the exit IP for the phone network, else one person could block access for millions.

And what a great loss that is :-)

They assume the human who created the robot is built to Asimov's law.

Reply on news.ycombinator.com