How Did Yandex Discover a Private URL? | Hacker News Reader

How Did Yandex Discover a Private URL?

7 pointsfoxylad7 comments

I recently set up a VPS server to provide a private service for an Appengine app. To test exception handling, I set up a test URL /boo.hoo/test, and visited it a couple of times with Chrome browser on Ubuntu to make sure things worked the way I expected. This URL has never been used by the app.

I forgot to remove the test code, and this morning (three weeks later) received an error message, which shows my exception handling works... except I hadn't visited the URL and I can't understand how anyone else knew it existed.

The request log shows (long sequences of base64 replaced with "..."):

    46.42.171.81 - - [31/May/2018:06:40:12 +1200] "GET /boo.hoo/test HTTP/1.1" 401 3817 "http://yandex.ru/clck/jsredir?from=yandex.ru%3Bsearch%3Bweb%3B%3B&text=&etext=1804.R2Y-...&state=...&sign=...&keyno=0&cst=...&ref=...&l10n=ru&cts=1527705451111&mc=4.949890056" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 YaBrowser/18.2.1.174 Yowser/2.5 Safari/537.36"

So how did Yandex discover this URL?

We're just investigating a similar issue on our site. We have an internal URL only available to admins (and would return a 403 otherwise, not linked to from any public page), that suddenly got crawled by Yandex. We checked the IPs of the indexing itself and they are legit.

We have only a handful of admins with access to this resource. It's possible one of them had a malicious extension I suppose, but even if they did, how did this leak to Yandex?

My colleague also found this discussion[0] recently, which reports a similar problem.

[0]https://www.webmasterworld.com/webmaster/4829663.htm

What IP address did your request come from? Someone made the point that my request didn't come from Yandex itself, just appeared to be referred from them.

77.88.47.81 (which has a reverse PTR of 77-88-47-81.spider.yandex.com.).

We also saw that this page was indexed by Yandex when we searched on their website. This page wasn't/isn't indexed by Google, Bing, DuckDuckGo etc by the way...

Is /boo.hoo by itself (without the /test subpath) public? Adding /test to anything seems a pretty obvious heuristic for a search indexing crawler to try.

Also since Chrome is a Google product, you should expect they are logging what you do. So maybe Yandex has breached them? Yikes.

Other possibilities are somewhere in the network between you and the server, or somewhere in the Ubuntu networking stack(!). Maybe you thought of all these, but I'm mentioning them since you didn't.

No, boo.hoo on it's own returns a 404, and there is no other instance of boo.hoo in the app's routes. I hate to sound so paranoid, but I'm guessing this is evidence of Russia's inflitration of the internet.

Did you have any extensions enabled?

No extensions - I thought of that. I'm reasonably security-aware, use Ubuntu and Duck Duck Go for search.

Reply on news.ycombinator.com