Stuff like this really hinders adoption.
Stuff like this really hinders adoption.
In other words, it's a Sybil-resistance mechanism, called Proof-of-IPv4. It works specifically because v4 addresses are scarce. v6 addresses are not nearly as such. Everything that makes IPv6 great for the Internet at large makes it terrible for mail providers. For example, because the original v6 design wanted to eat lower link layers, it reserves half the v6 address for an embedded MAC64. This never really panned out, but it's terrible for security, so every v6-capable OS nowadays will rotate addresses every few hours. The average machine will have hundreds of addresses. How do you assign a usable notion of per-IP reputation to that?
You could use v6 subnets for reputation, but there's still 64 subnet bits - enough to stick an entire IPv4 subnetwork inside of each IPv4 address. Some ISPs actually will assign a /64 per customer (because Comcast needs something to sell to Business customers), while others assign /56s or /48s. So there isn't even one granularity of subnetting that you can use for reputation tracking on v6.
Meanwhile, v4 pricing is getting worse and worse, which is great for mail providers. They don't necessarily need to turn a profit on incoming mail, but they do need to make it expensive for people who want to send lots of spam.
The problem of spam is actually solved, the problem is no one setups any of these security parameters correct, large and small companies alike all have bad SPF Records, bad or no DMARC, etc etc etc
If 99% of contacts you want to send mail to are on google/yahoo/microsoft you have to play by their rules. And those rules are effectively "send mail internally or gtfo".
I think maybe once in the last 3 years I ended up in someone’s spam box, total. In fact I just sent to a new gmail address and to a university I have never contacted before this week and both were delivered without issue.
Setting up DKIM/SPF/etc isn’t that hard and it’s fairly easy to verify with existing tools FYI.
Personally, I'm hesitant because I don't know if the end of all my effort will be constant blacklisting. If I could be confident that if I do it right I won't get blacklisted, I probably would.
Now most of them do not actually enforce it unless you become a problem, but most of them do put active measures on the network to stop SMTP Servers
For example here is a Exerpt from Comcast AUP prohibiting email and web hosting [1]
>>>use or run dedicated, stand-alone equipment or servers from the Premises that provide network content or any other services to anyone outside of your Premises local area network (“Premises LAN”), also commonly referred to as public services or servers. Examples of prohibited equipment and servers include, but are not limited to, email, web hosting, file sharing, and proxy services and servers;
[1] https://www.xfinity.com/corporate/customers/policies/highspe...
Why? Google Fiber is Available to less than 1% of US Households. Comcast is the largest Residential ISP last I looked in about 60-70% of US Markets...
ATT has the same policy, and I believe most of the other Cable Providers do as well. My guess would be over 90% of Residential Internet Plans today have a policy inline with Comcast not Google Fiber
Pointing to an outlier to the norm does not mean i need to update my worldview at all.
I looked for market share info on Google Fiber but was unable to find it. Mind sharing your source?
I was happy to move hosts to one that was considered trusted, but there was no way for me to know the IPv4 addresses the company had in the past, never mind if they'd been on a pertinent blacklist at some time.
Based on that I think it could work, but there are no guarantees that outside, historical characteristics won't screw things up for you.
The people that say "I Cant send anything" are likely trying to setup it up on a Residential or "Business" (which is really just a Residential with a slightly better SLA and less overselling) Internet Circuit... Not an Enterprise Internet Circuit
Hint: If they are trying to bundle TV Services with your Internet you are not on an Enterprise Line.
Even if you buy a dedicated IP from these services they still make is hard to impossible to send email on the circuit
Spam is an interesting problem. Assuming one self-hosts and makes their email address publicly available, then one can get a metric for how much spam is flying around. Eventually one will try to stop spam from coming to their inbox, and on doing so one might build a mental model for how the big mail providers combat spam and realize why one's emails are not being delivered. Then one might realize that one is sending mail that one would not willingly receive! And then take action to resolve.
In general though, there is some base effort to establish trust, and as long as you don't ruin it by sending spam, then you shouldn't end up on a blacklist. If you find that your IP was on a blacklist before it became yours, then work with the people that are blacklisting - but at that point it does become a bit of a job. I actually ran into an issue in my professional life where an AWS WAF rule started alerting on one of our own servers hosted in AWS because someone had previously used the IP for malware C&C.
Anyway - I will think on this and see if I can write something up. It's a good idea. My main concern is that there is a gap between the way I did things (sysadmin style) and the "new" way of doing things (containerized).
The issue people have trying to send mail is with the latter, where the email won't even show up in the spam filters, it will either be blocked by the mail system or silently ignored.
On my server, when I block a message due to trust, I reject the connection. When I block due to spam, the message is received but goes in the spam folder.
I get reports from Google/Microsoft/etc when other people try to send using my domains but their messages fail due to DKIM/SPF failures.
I just want to push back against the “it’s impossible to self host email” meme that seems comes around occasionally. Every time I’ve run into an issue there has been a solution.
Is that how Google/Microsoft/etc. do it too? If not, then your practice really doesn't matter. Most of my friends have an @gmail.com address, so if Google would pretend to accept mail from my hypothetical mail server, but instead just drop it on the floor, that's a non-starter for me hosting my own mail.
What I would like to get across is that self hosting is not impossible or unacceptably unreliable. If Google and Microsoft have policies that make it difficult to send messages to gmail or hotmail users, that isn’t a reason in and of itself that we should not self host. It’s a reason that we should work with Google/Microsoft to have better policies - but accepting things as they are and writing off self hosting as impossible is eventually accepting control of email by a limited handful of corporations, which I don’t think is a good thing.
Maybe thats why it works for you. Try making new one?
But...I've also been in the unfortunate position of leasing IPv4 addresses which were already blacklisted by various sources. It's not a terribly easy problem to solve if you need to contact customers NOW without using a 3rd party solution.
Unless you have a Commercial Line that has be specifically designated for hosting content then it is likely any IP you are issued is added to Google/Microsoft/etc Blacklist by the ISP. Most of them clearly spell out in their terms that running a Mail Server on the circuit is forbidden.
I recently made a presentation that has a full explanation of the techniques, why they exist and how they work, on Hetzner Cloud (from the original post):
https://nh2.me/recent/Running-your-own-mailserver.pdf
I find it very easy to configure with simple-nixos-mailserver (much easier than the manual setup on Ubuntu that I ran the years before):
https://gist.github.com/nh2/6814728dc3bea1508323e9bf2213c28d
Generating domains is fairly cheap though.
lsjfdlakj.com
There, I just generated a new one with a clean reputation. Just spend US$ 10 to register it and off we go.
It's the companies whom you rely on for email that are the worst abuser e.g. airlines need to inform you about delays and abuse this trust with holiday adverts incessantly.
Any company that claims to require your email for two factor auth should be given automatically generated fines for every email they ever send that is not auth related.
That would shake up Oracle sales dept. :)
The whole point of using a scarce identifier is to allow for a "neutral" reputation for new identifiers. If identifiers are less scarce, then known-bad actors can get free reputation (from bad to neutral) by just starting over with a new one. Which means that you have to distrust neutral reputation more. Without some level of scarcity of identification, introductions don't really work, because I have no idea if the new host I'm being talked to from today is just the one I banned yesterday wearing a different mask. This ultimately implies e-mail moving to some kind of federated whitelist system rather than the current system of federated blacklists.
e.g. when .com is on the list, and .somesite.com is not on the list, mail@somesite.com is from the same entity as mail@subdomain.somesite.com
From what I gathered from that, publicsuffix is a poorly-funded semi-volunteer org that shouldn't be relied upon for anything critical.
(Btw I’m pretty sure almost everyone is already using domain-based spam scoring.)
This also has a secondary problem for legitimate domain buyers. If the domain name they buy was previously used for spam that reputation will affect thier business for quite a while. There's actually a market where people buy domains with bad reputations, setup small legitimate businesses and get the reputation cleaned up, then sell the site domain and business for a substantial profit because a site with a good reputation history and established line of business will show up higher on internet searches.
would be very useful
(business opportunity here guys!)
What did you have in mind as far as a use case?
(for fraud detection it switches from block to identify)
for IPv4 this is generally the /32 (the single IPv4 address)
for IPv6 it's probably a /64, but may be a /56 or even a /48, and on some crappy providers even a /128
if the subnet is smaller than you think it is you risk banning an entire ISP (or country), whereas if if it's too large the abuse continues
it's quite a complicated problem as by design you can have subletting (subnetting!) within a block, e.g. a VPS provider gets a /48 from its ISP, and then they sublets out /64s to their customers (while not necessarily giving them all their own RIPE/ARIN records)
The other aspect is that a decent chunk of the IPv4 space at least is fairly dynamic. We've seen some blocks change owners every few weeks.
if i spent like a hundred bucks or something, i dont know... just asking. how would that work, does that "bring your own ip" that vps providers talk about mean this?
i
In pactice you cannot have less than a /24 because nobody will announce less than a /24
Edit: Seeing your use-case, this should probably be part of the whois records.
absolutely, assuming people subnetting to their customers delegate the space in the whois accordingly
(they do have an incentive to do that -- prevents all of their customers being banned if one misbehaves!)
Treating individual v6 addresses like individual v4 addresses is silly and nobody serious will take that approach.
You can use cloud providers, sure small ones do get blacklisted (which happens to also benefit Microsoft as they also are a cloud provider) but they can't really blacklist Googles or Amazons Cloud.
This is what is hindering adoption everywhere to be honest.
All of my forums, wikis, and game servers reject ipv6 purely for the same reason.
FWIW, I actually have residential service from Comcast and they'll assign you a /60 if you request it. I then use /64s for my subnets/VLANs.
You don't and that's the point.
Stop bloody tracking me.
What should you as a mail provider do with this new information?
Oh I dunno, innovate?
$ dig +short a ec2.amazonaws.com
52.46.140.46
$ dig +short aaaa ec2.amazonaws.com
(no response) api.ec2.us-east-2.aws
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Using...So yes, this is a long story.
But that was a long time ago. From what I hear, things are different now.
In case of icloud, I attribute it to the Proofpoint spam filtering system, which also sell service to ups.com.
And even gmail, but at least gmail accept the email, then just flagged it as spam.
Link: https://support.google.com/websearch/workflow/9308722?hl=en
The irony here is that much of the inter-service traffic on the internet could already be sent over IPv6 without anyone noticing. Getting end users onto IPv6 is always going to be a challenge as, well, ISPs, but when my mail server talks to your mail server there's no need for this to be IPv4.
I've completely given up to try to get my personal mail server delisted, as I can't even get Microsoft to tell me why they blacklisted it in the first place.
Instead I'm nowadays just rejecting all incoming emails originating from Microsoft with a message telling the sender to use another non-Microsoft email account.
It's just stupid. I never had problems with any other mail provider, but trouble with Microsoft as long as I can think of.
(For example Microsoft has blocked whole IPv4 ranges of cloud providers (i.e. Microsoft Azure competition) for E-Mail, supposedly because of abuse. But all cloud providers are used by people "producing bad mails" and somehow only small to mid-sized ones are blacklisted while e.g. Google or Amazon are not and to be clear that had not been cloud providers in some arbitrary small country but e.g. the EU).
(I don't really know what I'm talking about.)
$ host hotmail.com
hotmail.com has address 204.79.197.212
hotmail.com mail is handled by 2 hotmail-com.olc.protection.outlook.com.
$ host hotmail-com.olc.protection.outlook.com.
hotmail-com.olc.protection.outlook.com has address 104.47.57.161
hotmail-com.olc.protection.outlook.com has address 104.47.58.161
On the other side, if a host announces that they have an IPv6 address - do you think they do it mostly for spamers? $ host gmail.com | grep handled | head -n1
gmail.com mail is handled by 5 gmail-smtp-in.l.google.com.
$ host gmail-smtp-in.l.google.com.
gmail-smtp-in.l.google.com has address 173.194.73.27
gmail-smtp-in.l.google.com has IPv6 address 2a00:1450:4010:c1c::1a