“Magic links” can end up in Bing search results, rendering them useless

This... was impossible to do, because by long-pressing on iOS to get the Copy prompt, iOS also goes ahead and opens a preview of the link next to it

How about having a input field asking for the email again to double check?

I think that's a bit too much. Nothing in that suggests that they are breaking anything in the HTTP specification. You're right that GET requests has to be idempotent, but the exchange from the single-time use code you get in email with the API token, is most likely behind a non-GET request (like POST). The HTTP server responds to GET requests with the static assets (HTML/CSS/JS), but then the static assets has JavaScript that calls the POST endpoint for the exchange.

At least that's my guess. I agree it's a bug on their side, and they should fix it. But I think it's more of a UX issue than breaking the protocol.

Idempotent != side-effect free

The original post is just complaining that the malware scanning is visiting the links.

They come to the following conclusion

>This effectively makes all one-time use links like login/pass-reset/etc useless.

Which we all know is not true because sites like onetimesecret.com allow for entering a separate password to prevent this sort of thing when it does happen.

It would be an interesting discussion to talk about what Microsoft's whitelisting process looks like, but the original article doesn't seem to understand what is going on well enough to drive the conversation in that direction.

And also use the local client to scan unknown links? They probably dont want outsiders to have access to this code.

The password only makes this autentication less secure and it's not needed.

If you do so, in Outlook, there will be a pop that shows "Original URL: XXX". This allows users to make a determination for themselves whether the link is safe or not.

For a bit of added "fun", Google will do the same, but if you add a page to robots.txt and set noindex then they won't process the noindex parameter and external indexing sources might still generates search results: https://developers.google.com/search/docs/advanced/crawling/...

The recipient of the email, or their employer's IT department that is paying another company for mail services.

If you send me an email with a link then I do believe I have the right to send that link to a third party service that can validate that it's not malicious. If I decide to sign up for a mail service that promises to protect me from phishing emails, then I [0] expect said service to read the emails I receive and examine the links within them. I would be upset if the service used the info I share with them for purposes other than keeping me safe, though.

I readily admit that I have, at various points in my life, signed up for services without reading the entire TOS that I agreed to. I try to choose companies that I feel I can trust to not abuse me too much, and sometimes I avoid certain services because I don't trust the company behind them enough to respect my privacy.

[0] I acknowledge that not everyone is as knowledgeable as me, and many people might not realize that this is how the protection works. So if the argument is more education, I'm in favor.

That sounds like a narrow view of email. This has never ever been true. Corporate firewalls have always opened links, and many users use tracking blockers in their email provider that automatically opens incoming email and detect tracking cookies. I cannot stress strongly enough that you cannot rely on only one "user" clicking a link.

Email isn't WhatsApp, there are probably at least three or four parties who will scan the link in any way they like. If you control your side you can make sure there are only one or two parties scanning the email en route, but the number can never be guaranteed to be zero without workarounds.

Who gave the mail server permission to access the page? The person who set up email on the domain. If you don't trust the hostmaster, don't send email to that domain. What if it contains copyrighted materials? Well, you just shared a plaintext link with a whole bunch of people, depending on your local legislation you may be in trouble.

You can't even expect a link clicked once by a single user in a browser to only appear once on the server side. TCP connections get dropped and retried. This isn't some kind of philosophical interpretation of a mystical protocol spec, this happens in real life. If you use POST/PUT/whatever requests, the user agent will prompt the user if they really want to repeat a request; this protection has been built in for years. It's just how browsers work and how they've been working for decades.

If your recipient is behind a proxy, the link may be visited several times each hour for up to a month while the proxy refreshes its cache. This was a more prevalent problem back in the day, these days web proxies are mostly a thing of the past; however, proxies still exist, and if you don't pay attention to those things they will bite you in the ass.

In real life bugs happen. That's fine in these cases, web dev isn't exactly rocket science, bugs are tolerated and can be fixed. The bug here isn't the fact that links get visited twice, though: the bug here is that the developers who set up their magical links forgot about idempotency when they wrote their code, or they chose to ignore the problem because they never ran into it themselves. Either way, the responsibility to get it fixed isn't on anyone but the party violating the spec.

As a workaround, S/MIME or PGP should work around most of these problems as intermediate servers can't see what's going on. What the client's machine will do with the decrypted message is still up to interpretation, of course.

"In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.

Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them."

Personally, I think web crawlers like Bing shouldn't be executing javascript at all but front developers can't go without their client side rendering frameworks so search engines are more or less forced to.

As for a security mechanism, you want to emulate a browser as closely as possible to detect tricks like redirects from safe domains to attack domains and obfuscated URL crap. I'd expect any automated, non-interactive code to execute in a security analysis sandbox.

Is this breaking the standard? Who knows. What is a cloud antivirus but a web user agent running in a data center? The email protocol doesn't specify how the client should deal with links, the robots.txt only works for spiders, not for manually submitted URLs like those clicked in emails, and without a noindex tag you're going to see your page indexed by the mail provider company regardless of what your robots file says.

I think in theory your solution solves the spec breaking problem, but it doesn't solve the problem in practice because there are many other components for which there are no standards and defensive programming is required.

Never mind indexing them (ie publishing them at Bing.com), if URLs are disallowed in robots.txt then Bing shouldn't even be retrieving them, even if only to scan the content for malware!

> Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.

From https://developers.google.com/search/docs/advanced/robots/in...

Though that is optional and configurable.

Its a shame those links can't have an alttext to show the real link.

If I have a well-tested crawler sitting here, I don't see why I'd write a brand new one just to fetch previews in outlook...

Are there any cloud vendors that don't follow this approach?