Ask HN: Microsoft crawls private links – how can this be legal? | Hacker News Reader

Ask HN: Microsoft crawls private links – how can this be legal?

6 pointsnkmnz15 comments

I've built a tiny private website to share wedding photos. In order to protect the privacy of my guests, I've implemented authentication via supabase's magic link – guests need to enter their mail address and get an access link with a short-lived token sent to their mailbox. Unfortunately, this didn't work for users of mailboxes hosted with Microsoft, because Microsoft clicks all links in emails for "security" purposes. I've run some experiments and they seem to download all assets from the website linked to, including images, and try to crawl all other pages that can be reached through the first link in the mail, rendering not only the magiclink useless, but also downloading massive amounts of password protected images - a clear violation of GDPR. Why hasn't this been stopped?

View on HN

This might be Microsoft looking for pictures of naked children ?

https://www.microsoft.com/en-us/research/publication/metadat...

edit: In which case its part of the TOS and has government support. I would look for the press release statement for the change of terms of services but i dont want to search for "Child Sexual Abuse Material". Its apparently called photodna at microsoft.

If i recall apple changed theirs a few months ago.

edit: If you think i am being overly paranoid, last time i looked for the topic google put some pedophile self help sites into my results.

> Microsoft clicks all links in emails for "security" purposes.

SOP. many email providers do this. the magic link shouldn't directly take you to the thing, but rather be an interstitial, so that security vetting prefetch can be managed.

Further crawling seems quite bad. Are the pages actually public, protected only by URL obscurity? Not justifying MSFT behavior here, but you say "password protected" but the scheme you've described doesn't seem to be that. Can you set a session cookie after the magic link, and assuming MSFT crawler doesn't save cookies, "defeat" it that way? Or, identify it by UA and defeat it that way?

I think you need to provide more info about how they are crawling if you want a solution.

I don't see how this is a GDPR violation. Crawling the data and evaluting security or malware issues doesn't fall under GDPR. Saving the content would, and probably is additionally a violation on your part, but you haven't indicated they are doing that.

Also not clear why the magiclink doesn't work. Just because MSFT used it to crawl, why can't the user also use it later?

> > Microsoft clicks all links in emails for "security" purposes.

> SOP. many email providers do this

Heh, are they inadvertently unsubscribing folks from newsletters by making requests to the "Unsubscribe" links too?

Most Unsubscribe pages I've seen recently appear to be a two or more step process but I'm sure there's some folks out there making unsubscribe pages that do the thing on the initial GET request alone... (not justifying making a GET perform a side effect, just saying it's interesting.)

How would making a POST or PUT request to the endpoint logging the user in / unsubscribing from the newsletter change the problem at hand?

GET requests are supposed to be safe[0], if you have a unsubscribe link, you click that link, it makes a GET request, then unsubscribes the user, that GET request is no longer safe.

How I've seen developers approach this, is the initial GET request loads a form, and requires that the user clicks something once more, to then submit a POST request and confirm the unsubscribe action.

[0] https://developer.mozilla.org/en-US/docs/Glossary/Safe/HTTP

unsubscribe links are supposed to always be accompanied by a header with the same URL. X-Unsubscribe or smth like that. I guess a sneaky malware could make unsubscribe link a bad link, using the header to prevent clicking by the receiving server.

You're right, this is interesting, something I have wondered about before.

> the magic link shouldn't directly take you to the thing, but rather be an interstitial, so that security vetting prefetch can be managed

The magic link is taking you to a website that requires you to click to activate the link's token. MSFT is not only crawling all links on the page, but also clicking all elements with a click handler. It's necessary to add a recaptcha, but the problem stays the same: taking a secret that was not addressed to you doesn't allow you to access, process, and store the PII that you can find with that secret.

> I don't see how this is a GDPR violation. Crawling the data and evaluting security or malware issues doesn't fall under GDPR. Saving the content would...

‘processing’ means any operation (...) whether or not by automated means, such as (...) retrieval, storage, (...), use, (...) or destruction [https://gdpr-info.eu/art-4-gdpr/]

Microsoft retrieves the data, stores it (temporarily), uses it and deletes it (from memory). So all of these actions are subject to GDPR. The data cannot be deemed "publicly available", according to https://iapp.org/news/a/publicly-available-data-under-gdpr-m...

> Also not clear why the magiclink doesn't work. Just because MSFT used it to crawl, why can't the user also use it later?

The magic link contains a ~40 byte long token that is valid for a couple of minutes, just like an OTP.

May I suggest a honeypot?

I'm pretty sure that a robots.txt is only a suggestion, not legally binding in any way.

If enough people put honeypot links in their emails .....

Yes, robots.txt is only a suggestion. But stealing tokens - akin to password reset tokens - from emails is a completely different feat.

All major email cloud providers read your emails, unfortunately.

I think OP is more concerned about the fact that it's then crawling and presumably indexing what is meant to be a completely private site. That's different to "merely" reading the emails.

Both. At first, I was annoyed that outlook users complained about not being able to log in (there's no PII on the page, yet. Information about the venue only...). Now I am concerned about the massive amount of private data that microsoft probably scrapes this way illegally.

I know that Microsoft has been doing this for 10+ years...

Microsoft scans to check the website contains malware. IMHO the security blunder is a self-implemented magic link.

> massive amounts of password protected images

Not password protected if the password is part of the URL.

> a clear violation of GDPR

Unclear to me which PII is being stored or used.

> Microsoft scans to check the website contains malware. IMHO the security blunder is a self-implemented magic link.

It's not self-implemented, you can check it out here: https://github.com/supabase/gotrue

> Not password protected if the password is part of the URL.

It's a token that's valid for a couple of minutes – just like a password reset token. Indeed, in the given implementation, it's the very same as the password reset token. If you consider this implementation as "not password protected", any website with a password reset functionality is "not password protected".

Reply on news.ycombinator.com