Hacker News Reader

Comment by EvanAnderson | Hacker News Reader

Hijacking the thread: I do some "bulk" sending for a 501(c)3 I volunteer for. I include unsubscribe links that go to a form with a submit button (because I want the unsubscribe to be a POST request). Each link has a random opaque identifier in the query string. Something like:

hxxp://example.com/unsubscribe?id=abcd1234

A couple years ago I noticed that MSFT IPs hitting my unsubscribe links with invalid identifiers on the quest string. Anybody ever seen that?

View on HN

In a previous life, we prevented the GET url problem by having a javascript POST and forward to a secondary URL.

This allows everything to be "one click" (which honestly is a good thing) but prevents crawlers from accidentally triggering the unsubscribe.

Not sure this still works today and obviously this is not legal advice.

Not an answer to the question they asked.

---

Okay, HN. Go ahead and explain what's offensive here.

The question that was asked: "I noticed that MSFT IPs hitting my unsubscribe links with invalid identifiers on the [query] string. Anybody ever seen that?"

The question the parent commenter seems to have hallucinated: "Does anyone know how we can keep mail services from unsubscribing folks in error when these mail services scan our subscribers' emails, but also still offer our subscribers 1-click unsubscribe?"

You could imagine they started their post with "Yes, it's a pretty common problem here's what we did ..." and then it makes sense.

No, it doesn't. The original questioner is already aware of the extremely common phenomenon where mail providers scan links in emails. That's not what the question is about. The first comment contains a very specific question about something different. The response is derailing the discussion.

It seems like the more ideal solution would be to block the malicious IPs instead of lowering the accessibility of your site, no?

I think some crawlers run JS, because a lot of the web simply won't work without JS to initialise the page state these days.

You can use captcha or similar, one workaround I've seen has a submit that is hidden so never clicked by real people then a visible submit that sets a hidden input and clicks the other one which requires the hidden input... not foolproof but avoids some accidents.

A crawler that follows links found in emails and sends POST requests / submits forms will cause so much havoc. It could buy things, validate account sign-ups, delete data, etc. I have a hard time believing the answer isn't to ask users to switch to a normal email provider.

I know web search at least runs JS to get better results. Not sure about email pre-fetch but I assume they do. I don't think crawlers click buttons though unless they are malicious so it's probably fine for unique email links.

Yes, I know I've seen someone talk about this before, I think it's their link safety checking thing:

https://techcommunity.microsoft.com/t5/security-compliance-a...

It's odd that they're, essentially, fuzzing my app.

Agreed, it's curious! I wonder if they would still fuzz it if you changed the URL scheme to include the identifier as part of the URL path, rather than as a parameter? e.g., hxxp://example.com/unsubscribe/abcd1234

Please report back if you try it :-)

I could swear I've had that thing burn a one-time token for a password reset email before too, but it's hard to prove as a user. Doesn't feel great!

I might click an "unsubscribe" link in an email from an org that I know. But I'm not going to interact with the page that comes up.

If I get another email from that org, I click "report spam".

I thought it was part of CAN SPAM that you can’t require a second action and that was why the big email sending providers moved to that.

Probably true but how do you handle autodetonation of email links in that case? Too many emails servers will click links automatically to check for issues.

That was my understanding at least.

Unsubscribe link goes to a page that has a form that's automatically submitted via JavaScript. Disable that for the first 5 minutes of that link's life to get around automated things.

So many email security systems preemptively access every URL in messages. I found that I receive a GET for virtually every unsubscribe URL I send out.

I don't read clicking a "confirm" button as a second action. The attorney didn't either. He also said CAN SPAN doesn't apply to a 501(c)3. I still try to comply to be a good citizen.

You can require a second action such as clicking a button.

What you can't do is take them to a page that says "to unsubscribe, send a certified letter to our headquarters and wait 90 business days". The entire transaction must be completed at the page you link to.

Or, as I so often get, ask me to log in to "my account". Which is not my account, it's someone else's account but they don't know their own email address, so logging in is probably technically unauthorised access to computer systems and highly illegal.

Bonus points when contacting support requires me to log in to "my account" too.

It's not really common for clicking a link to immediately unsubscribe, almost everyone requires you to click a button after navigating to the unsubscribe link. Otherwise you have issues with link scanners unsubscribing your recipients without their knowledge. There are some more complex ways to approach this with JavaScript checks for "real browser" but IMO these are more likely to create frustrating friction to unsubscribing (by not working if the user has an adblocker for example) than having the user click a button.

I've seen this pattern of unsubscribe link, then click button approved as CAN-SPAM compliant more than once so I don't think there's a legal concern. The CAN-SPAM rule seems more targeted at the systems you used to see a lot that required the user to log into their account, type in their email address, or figure out a complicated "communications preferences" list to use the unsubscribe form.

check out https://www.ecfr.gov/current/title-16/chapter-I/subchapter-C...

It's a little fuzzy to me how exactly to interpret this but I think you could reasonably read it as allowing even unsubscribe pages that require you to type your email address in again (even though I detest these and don't think the problem they're intended to solve is a meaningful one).

> unsubscribe pages that require you to type your email address in again

These are fine for me if the email is prepopulated.

Google requires senders to use the `List-Unsubscribe-Post` and `List-Unsubscribe` headers, which use a POST request to avoid this problem.

Details: https://support.google.com/mail/answer/81126#zippy=%2Crequir...

(I work for Google, but on something totally unrelated, and don't speak for them or have any inside knowledge. I was just curious and looked it up.)

It'll likely be the URL rewriting feature in Microsoft 365 and Outlook.com. The URL will be scanned before it's rewritten.

I've never understood why some emails are ending with :J? I thought it was a meme I didn't understand. It turns out it's just Outlook doing its thing

Some versions of outlook autocorrected a smiley to a J in the wingdings font.

It's not just Outlook. Teams is messing that up just as well, and so did RiF on Android.

Turns out, dealing with Unicode beyond 2 bytes (i.e. anything above ASCII and the common Latin characters) is still a problem in 2023.

This is exactly what the list-unsubscribe-post header from RFC8058 provides: https://www.rfc-editor.org/rfc/rfc8058. The unsubscribe button that gmail, Apple Mail and others displays is driven by that; it's not a gmail feature.

Weirdly, if google thinks you're a dodgy sender they won't display the button, which seems counterproductive to me.

Some companies attempt to hit every link in order to cache the link and then embed their own link so that they can track those links, and also to examine for malware. I work for a marketing SaaS and what made us break away from the monolith structure was that our server was getting blasted to hell and back by the sheer volumes of tracking links that were automatically followed by email providers.

Several antivirus scanners and mail providers open links to check for malware. I believe they add some randomness to either bust through cache or to detect if the URL is encoded as an exact match (some exploit kits will redirect to google.com if you alter the URL in any way or after x requests to the same URL).

Yes have seen this is in a couple of my SaaS applications.

If it's in the querystring then they essentially fuzz it by changing some part(s) of the value. I noticed this because I use signed tokens and it raised an exception in Sentry when the signed token was invalid.

I ended up moving the signed token into the URL itself and the problem went away. eg. /unsubscribe/abcd1234/

If you are sending out HTML emails, cant you just make the unsubscribe button a submit button in the form?

I am sending text-only messages. (I hate HTML email, personally.)

You may not have to comply with CAN SPAM legally, but I absolutely hate when orgs do this.

Please try to make the world a better place instead of doing the legal minimum.

What else should I do? The list is double opt-in, every message includes a one-click unsubscribe link, full contact info for the organization is included, and I send text-only.

My reading is that in your comment they replied to, you said you require confirmation for unsubscription.

One click unsubscription is presumably what they want.

I require a button to be clicked to confirm. No entry, no JavaScript, nothing else. Just something to make a POST request because I receive GET requests for almost every URL I send out.

My experience is that every unsubscribe goes to a form w/ a submit button. Shitty ones make you type your email address. (Mine doesn't.)

I dont have a problem with that but it’s definitely a second click.

Just did a bit of unsubscribing and sydneytools.com.au, abc.net.au, squabblr.co, bundlehunt.com, and healingstreams.tv all one click unsubscribe.

I've seen a hybrid where you have a form with a button to confirm, but include JavaScript to auto-submit the form on load. For the crowd that has JS disabled, they can click the button, but otherwise it's one-click from the email.

No idea if this holds if/when the email crawler bots start executing JS on crawl.

Reply on news.ycombinator.com