Paul Vixie: Taking Back the DNS
circleid.com
circleid.com
I finally stopped running my own mail servers after 20 years because of the block lists. The RBL's continued false, silent bans caused too many mail delivery failures. The RBL's don't care about me, so I had to outsource to a mail provider so large the RBL's have to care.[1]
On censoring firewalls. There are businesses which cause their competitors to be blocked by http censorship lists. It is unlawful, but the censorship engines accept anonymous input to make their decisions and it isn't hard to hide the tortious interference.
Will DNS blocking have a similar effect of squelching the small and the new? Will it be abusable by nefarious people?
[1] For many years I didn't see RBLs as a problem, until I no longer controlled a large block of IPs that I had had from the early days of the internet. Now I have servers in mixed blocks. and most useless of all for email and web servers, reused IPs. (Blockers sometimes remember the sins of years ago and reinstate blocks based on a previous owner.)
The network infrastructure shouldn't be in the business of enforcing policy, period. This is exactly the same slippery slope as the net neutrality debate, and in just the same way, it favours the existing players and those with the deepest pockets. Do we, as startups, really need yet another disadvantage?
It's completely up to the DNS server administrator which policy servers they are going to configure or not, including using none at all.
In short - I don't see what all the negative fuss is over - corporate networks already filter based on subscribed lists via proxies, home-made things, etc - this is just bringing that ability into the DNS stack in an elegant way.
The threat here is that someone will identify a "bad website" and blacklist the domain via some publicly available policy server, which suddenly makes all services, not just HTTP, unavailable. For example, this means you might not be able to notify the admins of a domain via email if they host a blog (possibly yours) that gets exploited. Due to the fundamental nature of the service DNS provides, this could be an easy way to DOS an entire domain and all of its services.
I'm definitely interested in seeing what kind of community develops around this patch. If a good .edu focused anti-phishing feed shows up I'd strongly consider using it at work.
Coincidentally, just a few minutes ago -- just before loading up HN for the first time in a while -- I received an email from one of my clients asking me for advice. Their domain provider has been listed by SORBS, the domain provider is arguing that the listing was unwarranted and is refusing to pay any of the $50 "fine" that SORBS may request, if they request it. I have had to handle situations like this at least several times a year, and I'm not even an ISP. Another of my clients, a while back, briefly had their DSL IP listed in an RBL. They don't run a mail server from that IP, I vigilantly monitor their network for any signs of abuse so I'm fairly confident that there are no zombies on the network, and everyone's workstation is configured to send mail to an outbound mail server. Despite all this, the temporary listing in the RBL caused intermittent email disruption with misconfigured recipient servers that were incorrectly checking email headers for intermediate IPs listed in the RBL (mostly, ISPs using Barracuda Networks' latest bits of vomit). I contacted the RBL through the appropriate channels, and a few days later the block was removed without any other response or explanation from them.
This is absurdity. This is not how a healthy "internet" can work. This juvenile "my network, my rules" approach is no different from opposition to net neutrality. This isn't a small network of universities and hobbyists anymore; this is a global network of businesses that, for better or worse, rely on this system to handle important and time-sensitive information.
I would argue that, if you're an internet service provider or are otherwise accepting payment from customers for the purposes of handling some aspect of their electronic communications, then you are duty-bound to take every reasonable effort to ensure that all communications destined for them are able to reach them, and all of their communications are able to reach their destination. That means, for example, not subscribing to RBLs for the sake of lightening the load on your mail server and silently dropping email without notification to your customer.
There are better ways to handle spam now -- greylisting, user-configurable controls, etc. -- and issues in DNS are already being worked on by groups providing services to end users. I would really prefer not to see the adoption of another RBL-like system, since the first one managed to fail so spectacularly at all of its goals.
I can think of a couple of folks among our community who've raised a ruckus about RBLs, but they're folks that I know from experience raise a ruckus about everything (not to say you fall into that category). I can also think of another handful of folks who complained, who when I found more information found they actually are spammers, they just don't consider themselves spammers. It's strange how few spammers consider what they do wrong, and how indignant they become when things work against them. (Again, not to suggest that you fall into that category.)
Granted, RBLs don't really work all that well as a spam control mechanism. They aren't worthless in that regard, but they also aren't the most effective single tool. But, when used as a weighting factor, as in SpamAssassin, it's a perfectly sane addition to any mail server, and I wouldn't want to see them disappear.
In nearly every case of RBL listing I've ever seen in the years since RBLs came into existence, there actually was some reason for the listing: open relay, accidental spamming user with a virus of some sort, misconfigured DNS, etc.
Frankly, as someone that has maintained mail servers for all those years and for so many customers, RBLs are something I think about so rarely that I'm kinda stunned you have such a strong emotional feeling about them. I'm just not sure how my experience and yours can be so very different on the very same Internet.
In the end we gave up, email is a wonderful medium but if the reliability of the medium is left to people that are well willing to throw out the baby with the bathwater in a well meant but incompetent attempt at improving the situation then you're better off without it.
Blacklists are not the solution to any of these problems without a very rapid and transparent problem resolution process.
Spamhaus is one of the few such institutions that we've never had an issue with.
>a well meant but incompetent attempt at improving the situation
I do not mean to dismiss your frustration, but I wonder if you are comparing the present situation to an unobtainable ideal.
Yes, email worked great in the early 1990s. (E.g., most people who ran the net-news infrastructure demanded that there be a working email address in the From field of every message, and I cannot recall anyone back then disagreeing with that demand.)
But the internet is drastically different now. We cannot just go back to the way it was.
Blocking the way the RBL does it is counterproductive, you can't combat spam effectively that way.
Spamhaus has the right idea (block the spam from zombies), the RBL and it's cousins are out of control.
If the battle with spam will ever really be won however it will be by a combination of end user education and legislation that will make it more expensive to send spam than the money you will make on it.
The education part of it will be the larger component, the legislation to mop up the remaining die hards.
As for sending email out- sure that's a problem -but we keep ourselves off blacklists (Which you CAN do - you simply avoid email marketing altogether, and stop annoying your users with anything but password resets and things they ask for in realtime - the rest you do through the web).
My first collision with RBLs was actually back in '97 or thereabouts. I was just starting to really mess around on the internet, had visions of becoming a dial-up ISP, all that good stuff. Anyway, my ISP at the time ended up on the MAPS RBL due to the actions of a customer-of-a-customer. Rather than give in, they decided to fight them, and I got stuck in the middle.
For various reasons, I didn't have to deal with RBLs again for quite a while (I worked for other/larger companies, where it was someone else's problem; I left the computer industry entirely for a few years; so on), but eventually I got to be a tech-support/sysadmin monkey for a local ISP. They occasionally found themselves blackholed by one list or another for various reasons, whether it was a root'd box or a customer doing something dumb, usually a mailing list. Still, dealing with RBLs happened at least monthly.
Since starting my own business, I've had to deal with RBLs probably 3 or 4 times a year. Not super often, but I don't have a really massive list of clients yet either.
As yet another data point, nearlyfreespeech.net refuses to send out email notifications for planned outages. One of the reasons they give is that some percentage of their customers will lazily or mistakenly flag their message as spam and they'll end up on an RBL, which they don't want to deal with.
I'm actually not opposed to RBLs being used as a factor in products like SpamAssassin. For one thing, SA can be configured by the end-user, which means that if they miss out on anything, they only have themselves to blame; and secondly, so long as the RBL isn't the sole reason for dropping a message or connection, it doesn't bother me nearly as much.
At the ISP I worked for, "missing email" was probably the number 1 complaint, or close to it, and we spent an awful lot of time trying to fine-tune things so that legitimate messages wouldn't be lost. So, color me equally surprised that you've had no trouble with it at all. :-)
So, this is more than I come in contact with RBL issues by a pretty good amount. I think about it occasionally because users ask how to tune their SpamAssassin rules regarding them, but that's usually not in response to any problem.
But, what I find interesting is that something you deal with 3 or 4 times a year is considered a big deal, and worth throwing out the whole RBL system over (which, as I noted, I do consider a reasonably useful addition to a spam filtering system). I can think of a significant number of things I have to deal with a lot more often on a mail server, even if I did have RBL troubles that often: bounces, software updates, spam filter tuning, fixing clamav (honestly, dealing with clamav accounts for probably 75% of my mail server related work), troubleshooting why a large mailing is going slow, user stuff, whatever. All of that stuff happens all the time. RBL issues are once in a blue moon, and usually triggered by some other issue that would have had to be dealt with anyway...getting listed in an RBL just made me aware of the problem sooner.
I've had to request de-listing twice that I can recall: Once for a new server that happened to end up on an IP that had been used by spammers. Once because of a pseudo-open relay (a web application had a hole that allowed mail to be sent to arbitrary recipients). In both cases it was fixed within the same day. I've also advised people on fixing open relays, DNS issues, among other things, over the years. I can't, at the moment, think of any false positives that I've dealt with directly or indirectly.
To add to the argument that RBLs aren't more trouble than they're worth, I'll mention a relatively recent issue in SpamAssassin that did effect a lot of our users, and our own mail server. There was a rule, distributed in the default RHEL/CentOS SpamAssassin package, that considered mail with a date in 20x0 to be too far in the future, and thus very spammy. When we ticked over into 2010, suddenly thousands of users weren't getting any mail (except a few spams that cleverly used dates in the distant past)...that's pretty dramatic. Easy to fix, but it effected a lot of people, and led to a couple dozen tickets and support queries in our forums. That one bug has caused more support request related to email than RBLs ever have. (And ClamAV tickets/queries still far outnumber any other aspect of our mail stack, even after the SpamAssassin 20x0 debacle.) We obviously aren't throwing out SpamAssassin just because a few problems pop up now and then.
First, to the point that "3 or 4 times a year is not that much": At the moment I have in the neighborhood of a hundred active clients or so, many of them businesses. Having this particular issue come up with that small of an active client base is, to me, too much. Some of these businesses are research firms that are regularly sending data to outside contractors on tight deadlines and insurance companies trying to shuffle paperwork, contracts, and other agreements around.
Which brings me to my second point: we may simply have different ideas of what constitutes proper customer service. I believe, strongly, that it is totally unacceptable to have even one of my clients miss a deadline or experience even a relatively minor issue in their business because a mail server somewhere incorrectly flagged a message as spam, especially when that could easily have been avoided.
If the RBLs were managed more carefully, if there was accountability, if they had any incentive or desire whatsoever to go to great lengths to minimize the impacts on innocent users, then I would very likely have a different opinion of them. But, they do not. Vixie has been quite public in his very low opinion of spending any effort at all in minimizing the impact on innocent users. [1] We're talking about handing over massive amounts of network influence to a guy who would be willing to blackhole huge swaths of the internet just to decrease his personal spam volume by one or two messages per month. Is that really the internet you would like to be operating in? It's not the kind I want, for sure.
I don't see how the SpamAssassin issue you mentioned -- which I had to deal with also, albeit on a much much smaller scale -- makes RBLs any more or less trouble than they're worth. Either they're worth it, or they're not. I don't think they are, because I don't think they work well enough as a sole indicator of spammyness, and I think they have too many false positives. When they do have false positives, that translates to real, serious problems for real people.
Also, we're approaching this argument from two completely different environments. It's clear when you enumerate the list of problems that you have to spend time on that you're focusing mostly on things which bother a system administrator -- software updates and that sort of thing. However, I'm focusing on problems which bother users -- business owners, employees, individuals, professionals -- and while they don't tend to be impacted directly by things like software updates or slow mailing lists, they are directly impacted by things like all of their email to a contractor with whom they've had a longstanding relationship suddenly getting bounced by an erroneous RBL entry.
And, regardless of your experience with it, I'm telling you that it happens, and if my experience counts for anything at all, then it happens way too much.
[1]: For one quick example: http://www.mail-archive.com/nanog@merit.edu/msg23942.html
edit: For another example, http://www.mail-archive.com/nanog@merit.edu/msg04890.html and Vixie's reply ("zeal must become the norm.") at http://www.mail-archive.com/nanog@merit.edu/msg04904.html -- this is simply fundamentalism, and it will not lead to a healthy internet.
edit2: and more: http://www.networkworld.com/research/2001/0910feat.html ... ok, I'll quit now. Point is, there simply isn't grounds for saying, "it's not a problem."
edit3: http://www.infoworld.com/d/security-central/why-i-hate-rbls-...
This is an 8 year old post. And, I think, it's turned out to be an accurate prediction.
ISPs no longer run open relays, ever. In 2002, an awful lot of them had open relays, and did nothing to prevent spam from their users. Likewise hosting providers; if the server owner was paying the bills, they'd keep the lights on for them, even if they were spewing millions of messages a day into the world.
SMTP authentication has become standard and well-supported by all mail clients and mail servers, and is easy to configure.
DNS is now a fundamental part of the spam prevention regime, and DKIM makes it moreso.
Spamming costs more today, relatively speaking, than it did eight years ago. Not just because of RBLs, but it certainly doesn't hurt. Spammers have to keep moving, constantly, in order to keep sending out their spam. Making it cost more is the only thing that works. Asking nicely doesn't do it.
"It's clear when you enumerate the list of problems that you have to spend time on that you're focusing mostly on things which bother a system administrator -- software updates and that sort of thing. However, I'm focusing on problems which bother users -- business owners, employees, individuals, professionals"
System administrators are the ones who have to fix any problems in the mail server. They see all the problems, not just one email going rogue.
Users also care about spam. Users care a lot about spam. Try turning off your spam filters altogether, and see how much happier your customers are. I bet they'll be really excited to know they aren't missing any mail (especially the 823 messages from a Nigerian prince, and the 348 messages about "テクニックに自信ありますか"--I have no idea what that translates to, it's just a spam in my spam folder right now).
RBLs do reduce spam, and make it harder for spammers to operate. I'll continue to use them (in a limited, weighted, manner) until DKIM or some other authentication mechanism makes them completely unnecessary. Spam is a serious problem, and it takes dramatic solutions to fix it.
It seems to me that all the hoopla has long since past. The problems that RBL maintainers were fighting against are no longer standard practice, and when you run into them it generally means the system administrator has done something wrong. False positives are certainly a bad thing, but they happen in all spam fighting systems.
Anyway, I wouldn't use them as a binary switch and I don't know many people that suggest doing so these days, but I certainly like having them available in SpamAssassin.
I did. In 2002 or 2003, they made a change which caused my then-users to lose mail. I can't remember the details now (something to do with rewriting mails too aggressively, leading to empty messages, and some changed defaults) but I still remember the shame of it, seven or eight years ago.
It was in a sense my fault, because I didn't test a new version nearly thoroughly enough.
I kept SpamAssassin available for my users, but I haven't used it myself since then, and don't plan to ever turn it on by default ever again.
Trust is hard-gained and easily lost.
7+ years ago. And I still feel ashamed.
* 'my ISP at the time ended up on the MAPS RBL due to the actions of a customer-of-a-customer. (My ISP) decided to fight them'
* Then the place you worked at had: 'a root'd box (sic) or a customer doing something dumb, usually a mailing list'
That sounds similar to my own experience dpoing Unix work in the late 90s:
* Education provider has MS Exchange 5.0 box and is being used to send out gigs of porno spam (there is no relay control at all built into Exchange 5.0, MS made everyone pay to buy Exchange 5.5)
* Various boxes running Sendmail and incorrectly configured (fairly easy on sendmail).
In both my and your cases, the impression I got of RBL was positive. They Blacklist found real abuse, and the RBL's users/customers got what they wanted: less spam.
Blacklists have also proven quite effective in pressuring ISPs to clean up their act - they can still ignore bots/spammers/assorted morons, but they will soon find themselves unable to send e-mail, and hence, without legitimate customers. I admit there's some burden on the well-meaning, but it makes life a lot harder for spammers and spam-abiding ISPs.
The lists themselves don't bother me so much. The people behind them, though, scare the bejesus out of me.
There's a line in Nancy Kress' novel Beggars in Spain, where one of the main characters says, of another (a woman whose paranoia has driven her to do some fairly extreme things), that she would nuke the world if it would finally make her feel safe. Too often, that quote floats into my mind when reading things like the SPEWS FAQ.
Frankly black lists, grey lists and spammers should just go to hell.
Spammers first though. Otherwise things will get worse before they get better.
The benefit to the RBL system was that it creates economic pressure so that ISPs have an incentive to disconnect spammers.
The concept seems relevant here. Operators ought not block mail solely due to some RBL data. Users ought not report legitimate mail as spam and those reported "spams" ought not be fed into blacklists. (I gave up on Vipul's Razor as a bad job once I noticed that this happens.) System administrators ought to focus on providing useful mail service—blocking spam ought to be an aftereffect. And so on.
Not enough of these are. They sometimes are, of course, but it just takes a few system administrators with fucked up priorities to ruin it for the rest of us. Vixie's proposal won't help anybody.
This is something that administrators already do through less elegant and less scalable methods. It's trivial to put together a list of domains, or subscribe to a list, and have them resolve differnetly.
Some jackass who abuses that system will be systematically ignored.