Gmail is killing independent email
igregious.com
igregious.com
Edit: We can at least keep using our servers for incoming mail, and just use those for sending.
For me, it wasn’t Gmail. It was Proofpoint and Microsoft. During a very email-critical time I couldn’t get emails out to certain Proofpoint customers or to Hotmail.com addresses (specifically). And it was important enough for me to switch rather than try to fix.
Microsoft was likely arguably fixable but Proofpoint wanted me to prove ownership and control over the entire IP allotment from which my mail was emanating in order to lift the restriction on my single IP. That wasn’t possible because I didn’t control the entire space my IP was in.
It’s become increasingly difficult to operate email independently without a lot more money and time at the least. This worries me.
We had almost no issues with Gmail, only with Outlook "Protection"
Google kills a lot of the bigger ecosystem.
(Not to mention lack of basic features. I've got a laptop at some point and wanted to synchronize read status between my devices. I ended up writing NNTP to Maildir converter so I can use my IMAP server for this. Not pretty.)
Unsolicited marketing email is a problem, and I don't think we send anything that a reasonable person could categorize as that. We send
-replies to user support requests
-new vehicle listing alerts that users have explicitly signed up for (eg by running a search for a car on autotempest and then entering their email address in the "get alerts for this search" field)
-monthly newsletter, again with explicit opt-in only
And I send a handful of personal B2B emails.
Additionally, the linked post explains that if you get flagged as spam or "sending to many emails" there's literally no way to fix it without sounding the alarm in a post like this.
Despite all the advancement in machine learning, spam filtering seems to keep getting worse. And both Google and Microsoft have absolutely laughable support for when your emails are getting blocked. Could we just dial things back a bit? Some spam is fine if the alternative is having an unknown amount of legitimate messages not delivered to you.
Even if you have SPF and DKIM working correctly, they will block your email to outlook, hotmail, and other services unless you fill out their form. If you fill out the form, then you still need to wait months before they decide whether to accept your email.
On my own servers, I warn users who try to use hotmail or outlook that my email to them may not work and they should use a different email address.
I assume the problem is because email is an afterthought at the big near-monopoly providers.
> And this is happening after SPF, DKIM and DMARC provided a solution to the spam problem.
is just wrong. Tons of spam comes from servers with SPF, DKIM and DMARC now. It stopped being a trustworthy signal of not-spam many years ago.
Yet Google and Microsoft don't do that. I can send replies on a personal domain and server to people I've talked with often in the past, and quite often they'll go to spam with Gmail. I can send emails to universities running Office365 from the domain of a prominent university, and sometimes they'll go to spam. I've seen Google and Microsoft email at universities have emails from mts-nature.nature.com, Nature's manuscript tracking system, something no academic would want sent to spam, get sent to spam. SPF and DKIM should allow these things to be rather simply avoidable. But they don't.
Yes. Great. Thing is that this is such a trivial barrier to entry that guess what? Spammers do it too! Email has become so utterly corrupted with spam that the reality is that an independent provider who has no existing reputation is, 99% of the time, going to be a spammer.
It would be wonderful if we could fix this - but so far noone's come up with a workable solution.
There are outlier Mail-wonks who maintain pay-to-send wouldn't "work" but they are somewhat in a minority: If you forced senders to pay even tokenistic per-mail sums, the attractiveness of mail would disappear.
People don't want to monetize Mail for complex reasons. The "won't work" aspect in spam pushback has always been debatable.
(there's a well known checkbox list of "your proposed anti-spam mechanism won't work because.. which is a huge antipattern to having a rational debate about it)
The problems are regulatory: who sets the price, who collects the price, whats done with the money, and what it does to the ecology of email internationally. But, it would within some definitions of the term "work"
I don't expect to convince you any more than I expect to be listened to really: I've held this view since the eighties, earlier list paradigms I used in the 70s met cost, and were policed by list administrators accordingly. This topic has been dominated by a very few loud voices who basically prevent rational discourse.
But more importantly: the list didn't give technical answers, it gave political tradeofs that we never debated. Meanwhile the Gmail team went ahead and did what they did, and now we are stuck with one particular system that people then complain about.
That isn’t “true decentralization”, but close enough: a global nonprofit organization with strict policies is hard for companies buy and governments to influence. And it’s not truly free either, but those who can’t afford $10/mo sadly have bigger issues than hosting a private e-mail server.
Also this is not a proposed solution to spam. It is a proposed solution to aggressive anti-spam.
The server is one thing, but the domain ownership should leave a trail for real enforcement.
To add to this, if the spammers are in the US, unsolicited spam is essentially legally protected there. I've tracked down the actual US offices of some companies sending me clearly unsolicited spam with database-harvested information (usually sketchy loan offers to email addresses they could have only gotten through inappropriate means), but they can say they have an unsubscribe link and thus comply with CAN SPAM, and even if they don't, the act doesn't actually provide any recourse to individuals or small companies being spammed.
I suspect that this is the actual reason; “School Interviews” seems to be a new thing, and anything new is viewed with suspicion by large e-mail providers.
Most of the problems of spam have to do with who's an approved sender and who's abusive. Regulate the mail, it's much harder to be abusive.
You're in this country and sending spam? We arrest you. You're in another country? We rate limit your mail, report you to foreign authorities, and flag anything that looks like your mail.
You can burn me at the stake now.
But I also think there should be a government regulated digital 'town square' for each level of government, with township / county being optional but the rest funded by taxes. Then have those groups determine what is allowed and what's not, through people they nominate in elections. This is like the 'public option' for Facebook.
But mostly I just think this because I want more tax dollars to go to open source projects trying to solve problems that are very real and very much caused by the incentives of private industry.
If I am a website owner, no way I am going to use regulated service that will bring police to my door if I make a bug in my script, unless I have to.
Further laws around spam might help, but you don’t need the government to actually run anything for this.
This website is operated by Virtual Industries Group (https://www.vig.co.nz/), which is according to their website operates three services: School Bookings/School Interviews (https://www.schoolinterviews.co.nz/, focused on school scheduling and the service discussed here), Care Bookings (https://www.carebookings.co.nz/, which is the same service as above but focuses on day-care and other similar functions), and MessageMyWay (https://www.messagemyway.com/, which according to their website "is the communications hub for your community. It is your emergency communications plan, your telephone tree, and your email broadcast system all in one").
From a cursory glance, these three services shares this set of outbound MX servers. While it is very unlikely that someone who uses School Interview and Care Bookings would mark their message as spam, if the messages relayed by MessageMyWay is sent on the same outbound MXes then I could immediately see the problem. A large part of MX operators knows this and separates "marketing" and "operational" messages into separate servers to prevent this exact thing from happening. While I understand this dev's frustration, maybe the messages relayed by MessageMyWay are the ones marked as spam by frustrated parents who are receiving irrelevant school marketing which are sent to same MXes as their important operational messages?
If I were the developer (and still insist on using on-prem email), I'll operate three groups of servers:
Set A: purely for company-initiated messages, never for the customers
Set B: "operational" customer-initiated messages: School Bookings, Care Bookings, and MessageMyWay mails which are marked "critical" by the users (which is apparently additional cost)
Set C: "marketing" customer-initiated messages: "normal" MessageMyWay mails
MessageMyWay is opt-in and has a robust opt-out mechanism - and is sadly moribund. It sends a maximum of a couple of hundred emails a day.
And we are registered with GMail spam feedback loop and Microsoft's SNDS, both of which tell us when someone marks a message as spam. This happens less than once a week, so this isn't the signal that triggers the rate-limiting.
The get sent to spam or (worse) silently have the links stripped out a decent percentage of the time
What I don't get is, what is Google trying to achieve with this? A rate limit that doesn't increase daily or check the spam rate or provide a support rep to lift it?
Gmail has been around a long time. Is this the first we're hearing about this?
We did all those technical bits like SPF and DKIM, put the one click unsubscribe link on the message and also on the header of the message so that clients like Gmail can put the unsubscribe link on their own UI [2], all the recommended practices. I remember using the tool Mail Tester [3] and the results where all green.
We doesn't have issues with being marked as SPAM by Gmail/Outlook, and have average open rates of 50%, which is a lot higher than the industry standard who is around 20% [4].
We have a good UX and an ethical way to treat our users, like all users have to opt-in to their desired newsletters when creating an account (or choose to receive newsletters without creating an account), one-click unsubscribe link in big text at the bottom of each newsletter, but also a one-click-no-need-to-be-logged-in link to opt-out of all the newsletters that the user was subscribed, and more stuff like that.
But one thing that I think give us a lot of reputation for the Gmail algorithm, it was that we designed a feature that if the user doesn't opened a newseltter for about 3 months, we started to send the newsletters with an alert at the top saying like "Seems that you aren't reading this newsletter anymore, you will be automatically unsubscribed in 30[n counter] days. Click here to disable the auto-unsuscribe." (the disable the auto-unsuscribe also was for people who have images disabled and we can't track the openings, but is small percentage). So with that feature, we get sure that our users where engaged with the newsletters, and we have a system to avoid sending messages to "dead" emails, maintaining a fresh and healthy database of emails, and seems that Gmail/Outlook knows and likes that.
[2] https://www.sendinblue.com/blog/list-unsubscribe-header/
[3] https://www.mail-tester.com/
[4] https://mailchimp.com/resources/email-marketing-benchmarks/
1. Your decision makers recognized that the important number to pursue is "engaged users", not "number of emails in the list".
2. You have a large enough subscriber base so that Gmail/Outlook realize you're a good player. This wouldn't happen if you only sent a few thousand mails per week.
3. You have the technical expertise to properly maintain the list and the surrounding (unsubscribe, etc.) infrastructure. Usually the budget is allocated to _create_ the list/service and then assumed that it will continue working forever with 0 investment.
IF you can get these things right too you'll (eventually) have a similar experience. If you fail on any of them... you're screwed like everyone else is saying. Of course, having to send large volumes of email to ensure that "the big guys" treat you fairly is why many people running small/personal email servers complains about them.
> We recommend choosing a word or phrase that means something to you, and then adding one or two numbers or symbols, like "!" or "+". So something like squash:club! makes an excellent password.
That is terrible advice.
Chose a nonsense phrase from some popucluture thing or whatever, and add in or replace some letters with numbers and symbols.
Something like kRyptoni4n!muGgl342r0nin is easy to remember, and not being brute forced anytime soon.
It's a guessable password + guessable ruleset. You've introduced far less entropy than you've imagined. If you want to have protection against attacks and use your technique, you have to come up with a ruleset you (1) don't see in the documentation (2) that's computationally unreasonable.
For instance, combine multiple languages. Strawberry Octopus Sundae, which is pretty memorable, can become FragolaOctopusEisbecher combining Italian, English and German. And with that we just went O(N^3) where N is dictionary size (actually more than that because you've got a wide choice of latin scripted language).
That gives you far more protection then say sTr4b33ry+0ct0pu5!5und43. You aren't fooling a GeForce RTX 4090 with those tricks.
That's an example I just came up with though. There's lots of things like that. But really run a cracker. Try it out.
That's ok, isn't it?
Using something like a 20 character password following that formula is going to be fine for most people - it isn't going to be brute forced before someone gets an alert that passwords are compromised and can change it.
It's a much better solution IMO than using something entirely random and writing it down, or using something like LastPass.
This response isn't for you par se, I just don't want people to go off and think they're protected with beyonceMadonna because they read it on hacker news.
Put your needles in the biggest haystacks you can find, not the rarefied air of celebrities.
Also don't think you're protected with "old" names like GreenDayEminem. Using older celebrity lists to preference forgotten or abandoned accounts is another common technique. Sometimes you're just a stepping stone and not the target.
Find a bigger crowd to get lost in. For instance, say an address + phone number of a place you remember that's not yours. 419FSt#201 is an old friend and 8184093100 is a friend I had in childhood. Those are two big crowds. Phone numbers have way more structure than you probably think but it's still a fairly big haystack.
That's the thing though, I'm talking about combining pop culture words in a way that they are not in any list.
beyonceMadonna is quite a bit different from `b3Y0nm-Ad0nn\ac3`
That's using two words from pop culture, merged together with numerous substitutions. It's easy to generate, easy to remember, and won't be in any lists, and shouldn't be bruteforced (well, I'd increase it to at least 20 from 16) in the time a breach is reported and a reasonable window for changing a password after being notified.
I do agree with everything you've said though.
Password guessing and cracking is part of my job. GP is right: the advice is absolutely terrible. Your example looks good but the secret sauce is the way by which you create it. Maybe it's just leetspeak for something a million fans know, in which case it would be in my dictionary by virtue of downloading Wikipedia and other cracked password lists and the leetspeakify rules will find it in a matter of hours or days for a typical leaked hash (this looks like a domain admin password we might find on a workstation they logged into).
Eyeballing mGgl3etc., it doesn't look like something you'd find in anyone else's password, which makes it unpredictable, which makes my job hard and your account safe. But you can't determine a password's strength from the generator's output (only if it's terrible) so idk.
Also memorize, if possible, a few other important things like your bank password, or rely on 2FA for that, e.g.: we have a system where you log in with a reader for your card itself, so your password being stolen and your chip+pin being stolen is a risk I accept. If someone is after me specifically enough to get physical things out of my pocket, I'm not confident about resisting a rubber hose for that password / they can also install a keylogger.
For generating a password, it just needs enough entropy if you calculate log(pool^elements)/log(2):
- pick 6 actually random words (like in the xkcd) from a 7776-word dictionary like diceware, or some other combination that gets you at least 72 bits of randomness (so here it is log(7776^6)/log(2)),
- or use a much shorter but slightly harder to remember set of random characters (a-z, A-Z, 0-9 would need 13 characters. If you don't want to bother with shift at all but include a few (10) symbols that don't need shift, add one character.
Depending on the password manager, it would strengthen with a KDF, but that depends on details and remembering just a handful of 72+ bits of randomness should be attainable for most people if they use it regularly (at least once or twice a month after the memorization phase).
I should really find a canonical source for this so I don't have to write this up every time. Not sure if Schneier has this somewhere but I looked at e.g. the Wikipedia on password strength (a source people would trust) and it's less explicit and also contained a pretty bad mistake for more than a year not so long ago...
OTOH, people in the old days used to memorize a lot of phone numbers, so remembering things is not as hard as you might think.
Sure, the example I gave was more complex, but I was specifically referring to the "We recommend choosing a word or phrase that means something to you, and then adding one or two numbers or symbols, like "!" or "+"." being terrible advice.
It's not great advice, but they are on the right track.
> Password guessing and cracking is part of my job
Same here.
> GP is right: the advice is absolutely terrible.
It was just worded badly.
> Maybe it's just leetspeak for something a million fans know, in which case it would be in my dictionary by virtue of downloading Wikipedia and other cracked password lists and the leetspeakify rules will find it in a matter of hours or days for a typical leaked hash (this looks like a domain admin password we might find on a workstation they logged into).
Sure, there are tons of domain specific dictionaries, but that's really more useful for people that are targeting an individual and know something about them, or are just going to go through every dictionary they can. Most attackers are not doing that.
And besides, it's not about ensuring the password can never be cracked, that's not a great goal considering the compromise it would take - the goal is to delay brute forcing enough so that you can change passwords after you have an alert that a site has been breached.
Crying boomer here ... forgot where I put my tissue again.
Look at the 3 words I modified..
kryptonian, muggle and the last was '47 ronin'.
Alternate case, make some substitutions for letters with numbers/symbols, add in additional numbers/symbols, and you'jj have something that isn't trivial to bruteforce.
Once you know the core words you are using, you only have to memorize the substitutions you have made, which is much easier.
Google doesn't have to care, who is going to ban Google SMTP servers? That would be suicide.
However, this is still hard to establish, because it depends on the destination domain server to decide whether you're sending good stuff or not. I would like to have a mechanism by which the user can decide.
I imagine something like this: each email provider, say Gmail, issues to its users a number of single use codes like "Sor7xeik". When the user wants to subscribe to a newsletter (say news@interesting.com) it gives its own email address and one of those codes. The first email from news@interesting.com contains some header like
Authorized-Sender: authorize Sor7xeik
Authorized-Sender: yes
With this mechanism bad practices like address harvesting and selling become much less useful (because an address alone is not that useful, if the sender is not authorized; and the authorization must be initiated by the user).
BTW, I am not saying that all emails should be sent with this authorized sender mechanism. I don't expect individual users to collect authorizations for each of their contacts. Email without the Authorized-Sender header would still be subject to the usual spam filtering, but agencies that often send legitimate mass emails can have a mechanism to prove that they're doing it with the user authorization.
I know people who've been running mx/smtp servers on the internet since 1994 who have now given up on running their own, not because they're technically incapable or unwilling to continue to do so, but because they've been forced into themselves using gsuite or office365 because of the monopolistic practices of the huge email-as-a-service providers.
Please don’t spread the FUD that it’s impossible to run your own e-mail anymore. It’s perfectly possible. It’s just hard: https://news.ycombinator.com/item?id=32716633
For the record I do not endorse bowing to the pressure of the gsuite/office365 monopolists.
One of the important parts that's hard to solve for a person who doesn't work at an ISP these days is that you absolutely don't want to be in a /24 netblock (and parent /22, /20, etc) that also contains $5 per month VPS/VM/hosting customers.
Because the inevitable result of having shared hosting IP space where anyone with a pulse and credit card can buy a VM for one month for 5 dollars is that your IP space will end up on some blacklists/spam lists/opaque and not publicly available spam-blockage lists run by gmail, because other peoples' server IPs in the same adjacent ranges announced into global BGP tables by the same ASN have historically been a source of spam/UCE within the past 12, 24, 36 months.
But other than that I have never had a big problem, it just works and I spend little time on maintaining the system.
Data point of one. It’s something I occasionally worry about actually but it has never happened to me. So I probably just shouldn’t worry.
source: i worked at fastmail and regularly got oncall alerts to go deal with some rbl issues
The OP describes carefully monitoring reputation and is their own self-hosted mailing provider (not a service-provider) so presumably isn't hosting spam...
The difference may be the human contacts? Or just that self-hosted mailing providers are so rare that it's hard to keep their patterns from looking like spam to the filters? Or other?
His complaint seems to be mostly about rate limiting an IP that doesn't normally send in bulk volume. It's a weird behaviour. I would recommend he not use fastmail or gmail, but an actual provider for bulk delivery.
"backing off" people submitting mail, mail resending when failed, these are all really good useful parts of the email system that help you recover when things go wrong, even though it feels like a slap in the face - it's much better than the alternative of whitelist only, or blocking people permanently.
It was frustrating for me sometimes.... free.fr was often on our rbl list, and if I recall it was mostly because a customer (different ones at different times) would send an email to bob@free.fr an that person was actually like bobby@free.fr, so free.fr thought you were sending crap, or spam or whatever. Usually it was just an undeliverable mail.
Another thing is - IPs yes - they have "warm up" as you start sending mail out of it. In fact, providers like fastmail have a bunch of IPs, that way if one was flagged, and they figured out what was going wrong (you wouldnt just flip to the new IP, because a outbound spammer would muck up your new IP instantly), you flip over to the other IP which wasn't on any RBL.
anyway, I'm probably botching the explaination, one time our anti-spam guy drew a giant whiteboard of how it all works and it gave me an instant hangover.
Just to say - yes it's more complex than 'open up the port and start sending mail' but certainly not approaching 'no independent providers are allowed'
From your further description, it definitely sounds like something hard to do without significant dedicated staff -- like I could believe it is not realistic to just "run your own mail server" anymore, when mail isn't meant to be your business. For better or worse, you need to use someone where mail is their dedicated business, with the scale to devote significant resources to keeping on top of it. (an "anti-spam guy"!) Which didn't used to be the case in olden days. But, as you say, that doesn't mean it needs to be one of the big three.
Is my conclusion at this point.
But a "bit" (lot) more is required than one interpretation of "just taking spam seriously and having prudent rules in place" might suggest!
Keep your domain, outsource your mail service.
Plus fastmail is just all around a cool company so, yeah I mean, who wouldn't want to support small local business that love open source and do 'the right thing' when it comes to their customers and really care about how email works, and how it's evolving.
ultimately i didn't make the move because email is unusable without custom aliases i can completely bounce email from. but i think i might attempt again, but again I recently found myself insanely paranoid about my interviewing situation and had to make the awkward transitioning to my gmail account and explain why to my interviewer
1. We are registered with Gmail, Microsoft and Yahoo to provide feedback when someone marks our messages as spam (we DO do everything right). We get reports less than once a week, so if I've got my maths right, less than 0.002% of recipients think we're spammers.
1. Our two servers send around 10,000 messages a day, spread fairly evenly over time. That works out at three or four messages per server per minute. If we were spammers, we'd be ashamed of ourselves.
Alternatively, some kind of class action?
(Asking any lawyers in the room)
Likely there's a reputation issue on those IPs. Consider filling out a form to investigate that: https://support.google.com/mail/contact/gmail_bulk_sender_es...
I have wondered if a pattern-matching algorithm is penalising us for having a similar IP address to a spammer. But given that practically all servers are now located at hosting companies, IP address is a completely valueless signal - and an arbitrarily damaging one. We have no control over other servers near us in IP space, and can't even ask our hosting company to deal with the spammer because we have no idea which IP address they are on.
If you have access to an internal directory, could I prevail on you to pass this concern on to the Gmail spam filter team? I think there is a genuine bug here, and if they fix it and let me know, I promise to post a followup article saying that fixing this shows GMail isn't killing independant email on purpose.
But I've been trying to slowly de-tangle myself, and for it's faults ProtonMail has been working out pretty OK for me as a compromise between usability and true digital freedom.
However, gmail has some anti abuse throttling, so you might need to do a one time painful manually rate limited slow sync to get them out.
The smaller providers generally don’t engage in such bullshit, in my experience.
One of the most useful block lists people upload is generated based on how many other servers have blocked it. So once you're there, even if the situation has changed, you may have to petition a dozen or so servers to remove a block on you before you come off the list everyone else is using.
Botspam is an increasing problem, servers can get blacklisted because too many spammers get on it faster than the admin addresses the issue. And there's really no automated tools for handling all of it yet.
OK I get the occasional spam through spamassasin, but I can live with archiving 5 spam emails/day.
I don’t think it’s public infrastructure.
And yes, I have direct personal experience here. :)
I suppose it would be an interesting exercise to slowly increase servers/IP addresses until rate limiting stops, but servers cost time and money.
Unfortunately regulatory capture is real.
More like murdered.
But we do it anyway - and we start with the identity layer. Email based on blockchain identity, free and open. I've been working on this for a while, still WIP but check it out: https://ubikom.cc