Hacker News Reader

Comment by rsbadger | Hacker News Reader

Scanning something for malware and publishing it in search results seem like 2 completely different things to me...?

But there is nothing to indicate either in the post or in the referenced SO thread that the URLs are published to the search results. They are visited by bingbot, that much seems confirmed, but there’s no example where one of these results shows up in the public search results.

They were indexed in Bing results, I’ve shared the URL of that in this thread

Holy shit thats bad. Do unlisted youtube and gdrive share links get indexed through this?

You’d assume those have proper robots.txt configuration?

I have a disallow all robots.txt for a production system. Have had from the beginning.

Bing indexes it. This is my first major security incident and I have no idea how to fix this without making everything totally shitty for the users.

Some services ignore global disallow, but will respect rules explicitly targeted at them.

I've put in a hard block for all crawlers on all pages. Works for my scenario I think. Hopefully they don't lie in their user agent. Then it's going to be really bad.

Yes, but there is no indication they are publishing it in the search results.

The original post is just complaining that the malware scanning is visiting the links.

They come to the following conclusion

>This effectively makes all one-time use links like login/pass-reset/etc useless.

Which we all know is not true because sites like onetimesecret.com allow for entering a separate password to prevent this sort of thing when it does happen.

It would be an interesting discussion to talk about what Microsoft's whitelisting process looks like, but the original article doesn't seem to understand what is going on well enough to drive the conversation in that direction.

They are publishing them - it has bitten us (e.g. expired one click links for customers ending up on Bing from their emails)

I think this is where we use meta tags.

All pages with one click links should have no index follow or no index no follow. Your seo consultant (if you have one) should have advised you on this.

I am not saying this excuses the privacy violation but just suggesting there are things we can do...

Bing appears to be ignoring those headers for links crawled from emails

Worse would be links that are private to the people who posses the url. Like a private video on YouTube or a private document in google docs. The security depends on the URL being secret. This would silently publish secret information.

If those pages have no proper meta tags or robots.txt, there’s absolutely nothing wrong with this. Security by obscurity was never a good approach; from Proxies to security scanners, there has always been software that crawls unassuming URLs and published the results somewhere, if only a report to the admin.

robots.txt disallow is ignored for my production site at least. This is super bad.

Same for us - we have robots.txt disallow etc. and the relevant headers for personal customer links and Bing is ignoring and publishing all the same

If you can say for certain that the links being published are coming from the malware scanning, and not being taken from users' browser sessions that are using Microsoft Edge you should elaborate on this.

I would be pretty mortified if browsers were using user browser sessions to scan content and pass it to bingbot…? What about if you’re browsing something local? Or your bank account?

I would be too.

The point I was making is that someone should research this instead of relying on wild speculation as the basis for the conversation.

That would be even worse.

Nobody is saying it isn't.

It's about trying to get to the core of the issue, not just the random speculation going on in the article and in this comment thread.

Reply on news.ycombinator.com