Why does unsubscribing from a newsletter take “a few days”?
twitter.com
twitter.com
- have signed up and are receiving a verification email.
- are giving us money
- are using our site at least once a week during the first month, and even if they taper off, are at least using it every 180 days.
So, we're not sending email to people who we've never met, as it were. We have an unsubscribe link in all our emails, and an account email settings page that has about half the mailings we can send as initially subscribed (we like money), but which can be turned off. We even have links to our settings page in the emails and on all the site pages.
I like to think we're being fairly responsible, all in all.
Like I said, we send 2 million emails a day. That goes out on a lot of machines, and we have lots of automation going on nearly every minute, and lots of email queues being prepopulated to send out in volumes acceptable to gmail, outlook, yahoo, etc.
So, you've asked to unsubscribe to the email you received last week. We got you, fam. I analyzed logs and did some live testing using our VPN at offices across the world. Over the past 3 years, you've been removed from your chosen lists within 2 seconds of clicking the button. This obviously doesn't cover hardware/network/server problems.
So we're cool, right? Nope. You're unsubscribed all right. However, we already placed you in the queue for our most recent mailing about 4 hours ago, and you're going to get the last one from us anywhere within 2 seconds from now to maybe late tomorrow.
You really won't get anymore from us after that, though.
And they send (and inbox) millions of emails by having reasonable open rates, not spamming, and respecting opt-outs.
All modern email clients block all image and CSS requests by default, and email security filters examine any linked URLs automatically. So there is no reliable “opened but did not click” feedback mechanism at all in modern email.
Set up a seed list of 100 email addresses you control, send through email marketing tool of choice. Open exactly 1 of those emails from a default client. See what lies their “open rate” dashboard gives you.
Whatever open rate stats are shown to marketers are irrelevant to google's deliverability choices.
We also partner with a reputation monitor that makes sure we're behaving, and they also let us know if we're approaching thresholds of not email friendly behavior. The only time I've seen that happen was when we did some IP consolidation and certain mail handling companies automated systems found our behavior anomalous, and thus was flagged suspicious. Since we knew, and our monitor knew this would happen, it was handled, and everyone (us, monitor, mail domains, customers) was running back to normal behavior within about two weeks.
In the end, no blacklisting from this, and none while I've been here. We've had a fairly high inbox delivery rate (based on our rep monitor, about 3% higher than the average) over the years, and about the only thorn in my side is Comcast, and even our reputation monitor has "special" metrics to measure their customers in regards to Comcast. :)
They know how to send emails and, most importantly, how to collect complaints and bounces feedback from email providers.
Those numbers that Marketo and company show are either completely made up or at least have hidden “scaling algorithms” applied.
All modern email clients block all image and CSS requests by default, and email security filters examine any linked URLs automatically. So there is no reliable “opened but did not click” feedback mechanism at all in modern email.
Set up a seed list of 100 email addresses you control, send through email marketing tool of choice. Open exactly 1 of those emails from a default client. See what lies their “open rate” dashboard gives you.
Not letting people unsubscribe quickly is a conscious, commercial choice/tradeoff, not some kind of hard technical limit. Legislation can push this easily in a different direction.
Let's say 10million have unsubscribe, at 128bytes per email. That's roughly under 1.5 gigs of ram to store all those in memory.
Sending 2million emails daily comes out to an average of 23 a second. You can search 23 times through that list pretty quickly with one extra CPU core.
If n is the number of emails and k is the number of unsubs, it adds up to O(n) to check the table of unsubs before sending vs. O(k*n) to scan the queue after every unsub.
Or, to be less computer-sciency, a task that scans the queue of emails every time someone unsubs would be a pain to keep running. Elsewhere in this discussion the queue was described as a file. Who wants to scan the same file over and over all day?
The vast majority of bulk email is done via 3rd party providers like constant contact or mail chimp because delivery is important.
You can't just check an address--you have to check against the particular mailing list, email address and account which means you'd have to basically have to have the database. Otherwise, if I unsubscribe from "Megous Monthly Mailer" I'll stop getting the announcements from my kid's school with useful info like what the first day of school for my kid is.
So you need to know the account, the mailing list that was unsubscribed, the email address. Of course, the system is architected for high-volume, and scheduled email. Delivery volumes can be in the billions per minute. Anything too crazy and your service level is destroyed.
Queues are designed for high-volume queuing, not search, or retrieval. There's no "query API" for rabbitMQ, for example.
All of this is easy if we hand-wave away the requirements for high volume email delivery.
We built your customized email 4 hours ago. It's sitting in a directory somewhere waiting to be read and sent out to gmail. We're waiting because we already sent "X" emails to gmail addresses 3 minutes ago, and we need to wait "Z" more minutes. It might be in the list of emails to be sent as soon as the gates open up again. It might not be ready to go until another 50,000 emails before it have been sent.
So, let's say you've decided you don't want to get our emails anymore. Great. We've updated the appropriate tables. Next cron job will skip you. Now let's implement your criteria.
We've got to tell the MTA to figure out which is your email, and delete it from the queue. Hopefully we got to the MTA in time. We might not have. Remember the example I initially gave? You might be receiving the mailing 2 seconds after you unsubscribed. So we've spent some measurable time trying to prevent something that's already happened.
We don't have enough unsubscribers in general that this is going to cause a significant overhead. The thing is, I'm not even sure I can do that. Our routing application is a 3rd party commercial application that we don't have the source code to.
Not to mention, we're going to have to hop back and forth between machines as our databases are elsewhere. Not even sure they're in the same data center, now that I think about it. I really don't want to look up the NAT information to work this out for a simple internet discussion.
I'm gonna punt and say, "I dunno. I know how my part of the company works, and I'm still learning it. Maybe there's a better way to do it, and if you have information on commercial products that will do what you're stating is the desired goal, I'll do some research on them, and make recommendations to our company if they look viable."
I'm a C# developer. I'm still not quite sure how I ended up handling email. :D
So just...don't send it? I don't see why having done prior work to compose it means that you have to send it. You could argue that it's a waste of resources to have composed it and not sent it, but that's a sunk cost you can't get back, so why not just throw it away if you already know that's what I prefer?
So just heavily modify sendmail/qmail/whatever to maintain a list of email addresses for which it shouldn't send it?
Not associated with the grandparent, but the software likely composes and immediately "sends" the email, at which point it's going to be in MTA queue that's completely oblivious to the fact some email addresses have been unsubscribed since they entered the queue.
Overall, what you're suggesting sounds somewhat unreasonable. Once the email is "sent" (actually, added to MTA queue!), he might have no control whatsoever what the MTA does.
If you throw a stone, stopping it mid-air can be... hard. Perhaps it's doable. But is it a reasonable expectation?
I really don't think the possibility of receiving one more email rises to the level of needing new legislation and adding one more liability to help push people to moving all of the data and logic externally.
I’m fairly certain that’s something postfix will do for you too.
The reality: it's financially a waste of time for a company to create an A+ off-boarding experience. Getting 1-2 extra emails isn't the end of the world: it's good enough for the company, you get off the list, they don't have to think about that unprofitable problem any further.
Most companies don't put much effort into off-boarding. Some go out of their way to make it even less palatable. But for removing yourself for a mailing list, I don't consider this to be the worse. Things surrounding payments are much more important: if I cancel service, you better not continue charging me.
Most people don't consider the full lifecycle of anything, and I wish they did. We've created a society, for example, that consumes disposable plastics, as an ordinary daily activity, and that ends up in our waterways as microplastics. It's easy to make things. It's an order of magnitude more difficult to manage the full lifecycle of the thing.
This is true even if your company has in-house developers. At the small company I work for, we have barely enough developer bandwidth to cover most things related to our core products as it is. Improving internal support processes just isn’t even on the radar for us as a dev team. So those things either get contracted out (where we as a dev team may have limited or no evaluation input with regards to the quality of the solution), purchased (again with little or no input from us as devs) or are built ad-hoc by the less technical side of the business.
Companies have a lot of internal moving parts and resource limitations that together lead to these kinds of things.
If I receive any further e-mails after unsubscribing, I mark them as spam. I don't think I'm the only one who does. That should be some incentive against sloppy unsubscribing mechanisms.
Not solving nonproblems is how reasonable people operate.
If Amazon started sending me a ton of useless random packages every week, you bet that I'd expect them to have a team of ninjas chase the driver down and forcibly remove my package.
Spam and marketing mail should be - needs to be - opt-in and illegal.
I would love to have an ACL on my physical mailbox, where all senders I have not explicitly authorized get mail returned to sender at their own cost.
Might want to fix that with a filter on the queues.
In the UK if I haven't bought anything from you and I haven't asked you to email me and you email me then you may be breaking the law.
Well, at some point you took an architectural decision with an implication that if a customer requests to unsubsribe at time t, the request will be effective at t+N. This is a choice you made so the short of it is, it takes N becase it suits you.
A user unsubscribes. This creates an entry in one specific system. In order for that to be reflected across all systems, it needs to be copied somewhere central, then synced back out.
Ideally you'd have a webhook hit a Lambda function and call it a day.
But, again, largish company with a gotta-move-fast employee growth mindset, engineering doesn't want to work on it (or, if not a tech company, you don't have in-house engineering).
So you hire some consultants who convince you that your email marketing is a "big data" problem, and they contract out the work on some Enterprise Infrastructure Platform as a Service product (an expensive, slow Lambda). The resulting system is slow, and often breaks, and you run it every few days in one bulk load/unload.
Poor engineering is why it takes a few days.
If the newsletter appeared from nowhere, then yes, it's spam.
Go away Janice, I don't need an extended vehicle warranty, and no I didn't contact you for information.
Any of those have been demonstrated to be capable of spectre/meltdown/etc. Hope your passwords/ssh keys aren't stored in RAM.
Can't speak for Gmail specifically but we ran a biggish email operation a few years ago (sending real estate alerts) and were notified when some of the big providers users marked our emails as spam.
So unless things have changed, the newsletter providers are notified.
Or use something like Kafka / SQS / Pulsar / Google pubsub and process it as many times as necessary.
> A user unsubscribes. This creates an entry in one specific system. In order for that to be reflected across all systems, it needs to be copied somewhere central, then synced back out.
The idea that it takes any longer to unsubscribe just demonstrates the marketing companies being user-hostile. Let me demonstrate why what you're saying sounds ridiculous:
A user is subscribed to a marketing provider (probably against their knowledge or desire). Minutes later all three marketing providers have the user's information.
If the system is slow, the marketing company would solve that pretty quick. If it often breaks, the marketing company would solve that pretty quick. If you run bulk loads/unloads every few days... maybe I could see that. Maybe. But the marketing company is likely going to want to fix that too.
How many users do this database have? Thousands? Millions? Databases from twenty years ago handled that just fine and quite fast too. Are you trying to tell me that modern databases are slower? Or are there perhaps too many users in the database from, eg, a dragnet?
Is the Enterprise Infrastructure Platform as a Service product truly to blame? Are you trying to tell me that the service is incapable of handling the ingress of a data point (user subscribed, user unsubscribed) in a timely manner? If that's truly the case then I have to truly wonder what else that service doesn't handle well: user privacy and information security are two top points that I'd wonder about. Competence and susceptibility to spearphishing would be some others.
Oh, right. Marketing companies aren't "user-hostile". Users aren't their target audience; users aren't their "users". Businesses are.
If someone who joined today didn't receive something from the survey mailer, that was a-okay. For unsubscribes, people didn't complain. I suspect that was because (1) Our customers trusted us. (2) Most customers received no more emails. The number of people who got a second email from some auxiliary system was very small.
But we couldn't guarantee immediate, full unsubscribes.
Regarding competency, security, and privacy, those are rare resources in the tech industry. Most companies couldn't care less about security of user data or privacy. Most also can't achieve competence.
Oh, and in case you're wondering how they did (similar) things in the Good Old Days(TM), let me tell you a story from the late 70's/early 80's.
I subscribed to "Cycle" magazine in my youth, and due to mistakes on their part, for a couple of years I got two copies, and when I finally decided to unsubscribe, I got only one copy a month (but for another year).
My mother had written to them to unsubscribe me (it was a recurring birthday present).
My first year of college I wound up meeting a guy in the magazine subscription business, and told him my story, and told me how it worked.
Turns out that they got a LOT of mail at the (US) publisher. So they had a machine with a gripper that grabbed each letter, held it while a grinder ground off (!) three edges of the letter, and then put it on a conveyer belt. One person was tasked with folding the top page of the envelope back so the letter was revealed, and a few people (IIRC) took them and put them in boxes, taping the letter back together (!) when part had been ground off with the edge of the envelope.
These boxes of letters were then sent to Ireland (!) to be processed, where people entered what should be done on some kind of mainframe application, which then cut 9-track tapes that were sent back to the US for processing.
Told this to my Mom, with the delightful result of seeing her collapse in tears of laughter.
[1] https://www.ftc.gov/tips-advice/business-center/guidance/can...
Saying that unsubscribing takes a few days means that in the off-chance that this happens, the sender has some coverage against annoyed users who have one of these mails delivered after unsubscribing.
But this is just my guess.
I use a rule of thumb that every "spam" click costs me (as a bulk email sender) about $10. If somebody already unsubscribed -- it is quite likely that the next time they receive similar email -- they will click "Spam" button (which would cost me $10).
So I try to process "Unsubscribe" clicks quickly.
I just never use unsubscribe links period. If I don't want something (and it's basically never anything I actually signed up for like a news letter or mailing list) I just have it dumped to the trash or a spam folder. I never see them anymore and it no longer impacts me and as an added bonus it consumes resources for the companies who spam my inbox. Why bother with an unsubscribe attempt at all?
I got bombarded with 35+ emails every day few years back; & documented it at https://gitlab.com/davchana/gmail-indian-spam-domains/blob/m...
Unsubscribe link click marks your email as live/hot; & gets them a higher price everytime it is clicked by you & then sold by them as fresh hot.
Based on the responses I (rarely) get, I've helped boot a few spammers from their servers by providing evidence to the operator.
* This gives cover in case it takes a bit of time. Better to promise a few days and do it sooner than vice versa.
* Letting people go is, in general, bad for business so hasn't been optimized.
* Multiple systems are involved, increasing complexity.
Seeing the explanation go on and on and on without a clear indication of how close we are to the end enhanced the kafkaeske atmosphere.
This illustrates something that I think a lot of us in the "computer industry" often misunderstand. We see a mass email system (or anything happening at scale) and assume the whole thing is automated, because that's how we would do it.
Too often a system is cobbled together, only barely works, and is only semi-automated. Even something that is 99% automated but generates thousands of actions per day ends up creating a high workload for a human.
I've even seen where my email address was clearly hand-typed from a form (not even copy/paste), because I usually sign up for website accounts using Gmail's plus feature. I created an account at Website A with the address "myemail+websitea@gmail.com" and then received an email a day layer sent to "myemail+wesbitea@gmail.com" which Gmail still delivered because they ignore everything after the plus sign. The only way that error happens is if someone typed it by hand. [Plus emails are a great way to find out which companies sell your info to spammers. Most of the time nobody bothers to run a regex to fix "plus" addresses into the original address, so the evidence of data selling ends up right in the email header.] I feel sorry for the person that is eventually going to have RSI because they hand-type the entire list from their web form into their email software.
Presumably this refers to Nationwide Building Society, then.
The one to two weeks lead time always made a lot of sense to me.
(And yeah, you don't need to tell me how horrible everything about that process is... but it worked and no one's motivated to fix it. I wouldn't be surprised if that's the same process they're using today)
Why are big and medium companies usually such a mess?
Is it because the bigger the company the less people care?
Maybe it's that the complexity and discipline required is simply too much for the average human?
Maybe companies do not have or are unwilling to invest the needed resources? (which ironically creates more waste)
They are based in Swindon. I am quite surprised their in house tech is this bad because their online bank account is one of the better ones.
Mostly, it's just CYA language because of the way the various old and slow systems work, plus the CAN-SPAM act legally allows up to 10 days to process an unsub.
There are multiple checkpoints that prune lists as they get churned through the machine, so typically you'll be fully unsubbed within 24 hours (often much less), but they don't catch all cases at all times.
The abbreviated process for sending out a marketing campaign at a large ESP typically looks like:
- Marketing manager makes a request for a list of people that match x/y/z analytics criteria (e.g., purchased within last x months, typically opens email, geographic region, etc). Depending on how "sophisticated" the criteria are and how backed up the analyst department is, this may take a couple of days to get turned around.
- The list of addresses is created and then pruned down by known unsubscribes or other do-not-email constraints in the system at the time the query runs.
- The resulting list gets sent out for review and approval by the marketing manager and client (how many people are we going to mail, what is it going to cost, what sort of metrics do we expect, etc). Since this is a human-in-the-loop process, it may again take up to several days to turnaround.
- After approval, the list gets churned through the unsubscribe list again, dropping any new unsubs. This step should catch new unsubs within that "it may take up to few days" window mentioned in the title.
- The final list is then queued up for sending, which depending on the size and meter rate may go out over the course of several hours. If you've already been queued up your unsubscribe request is typically going to get missed for this run.
Now, add on the complexity of syncing up multiple databases between the ESP and the customer, which is typically a nightly batch job at best. So even though your unsubscribe hit some web server instantly, it may take a couple of days for it to fully filter through from the web server to the client's marketing databases into the ESP's database. It's similar to why banking and ACH is so terrible: it's just ancient design patterns and slow process and nobody wants to pay money to modernize it. And if they miss a few unsubs they are still well within the legal bounds so it's whatever.
tl;dr: A lot of email marketing still runs on chains of batch jobs which can introduce windows of unsynchronized lists getting sent out.