How do we explain email to an “expert”?
sobersecurity.blogspot.com
sobersecurity.blogspot.com
2. None of that is actually solved by simply trusting someone else to run your email servers for you if you also can't judge whether they are doing it properly and, in particular, in your interest.
3. If you think that trusting someone else to run your email servers for you is an acceptable solution, that is functionally equivalent to trusting someone else to provide you with a software package to run on your email servers. Everything that someone else could do on an email server that they run for you, they could just as well package into, say, an install CD image, which you then could use to run your own functionally equivalent, hopefully well-configured and well-updated, mail server.
So, I would think, this is at best an argument against putting together your own mail server from distro packages, and configuring everything yourself, maybe.
Other than that, there is no fundamental reason why you would need to know email RFCs any more for running an email server than you need to know them for running an email client. In either case, knowing them can help you with figuring out some problems. But also, in either case, if the software is good, the programmers should have taken care of reading the RFCs for you, so you don't need to, for the most part.
Gmail won't give you their entire software stack (also good luck running it even if they did). Neither will any of the other big players.
The perfect spam filter Gmail has its probably based on data and ML analysis, not a simple software you can put on a CD and install on some little server; so even if someone who works there wanted to help you they probably can't do it.
It's anything but perfect. It's a black box, with no user-serviceable parts inside. Completely useless for any organization with needs that deviate from Joe User.
You can sort of turn off spam filtering on incoming emails but I don't most people or organizations would want that.
> Completely useless for any organization with needs that deviate from Joe User.
Maybe but being slightly over aggressive means a world in terms of user happiness. Think about email before Gmail. How much junk did you see in your inbox? There is a lot LESS today. Joe User would be very happy if they dried to think about it for a second.
What do you mean "implement"? You are aware that there exists free spam filtering software that you can install, like, say, spamassassin (which has been around longer than gmail)?
> Maybe but being slightly over aggressive means a world in terms of user happiness.
Dropping legitimate emails is good for user happiness? You have strange users. I imagine if the post office decided to throw away some letters "slighty over aggressive[ly]" ... there are people who would be happy if that happened?
> Think about email before Gmail. How much junk did you see in your inbox? There is a lot LESS today.
I see maybe one to three spam mails per day, with nearly zero false positives (maybe one per year). And that's without gmail. Obviously, I could reduce that to zero spam, at the cost of increased false positives, which is not a sensible option as far as I am concerned, deleting three mails per day isn't really all that much work.
What's the point of pointing out the one example among millions of email server setups that is most likely to not be published? Especially so, given that that would be about the most useless setup to replicate, as noone needs the scalability of gmail for their own setup, and thus the complexity that would come with it.
> The perfect spam filter Gmail has its probably based on data and ML analysis, not a simple software you can put on a CD and install on some little server; so even if someone who works there wanted to help you they probably can't do it.
If you think that gmail's spam filter is perfect, I have a very simple even more perfect spam filter: Just throw away all emails. Gmail has massive false positive rates, that's not a perfect filter, that's just a filter that throws away a lot of emails.
If anything, the control that google has over which emails it arbitrarily labels as spam is completely unacceptable, especially so given that they don't even accept liability for incorrectly filtered emails.
2. They are by themselves a major security problem because they necessarily have access to all your email. And what's worse, if everyone is using them, they have access to everyone's email. That's a huge amount of power waiting to be abused, if not directly by them, by someone bribing them, putting pressure on them, or simply by infiltrating them.
The first part of that may be true, but the second part is definitely false. It is very hard to write your own secure anything-connected-to-a-network.
No, it's not. Where did you get that idea?
But let me ask you: What are the big security problems that you see with a personal email server that you think are difficult to get right/to avoid?
Note that I'm not at all claiming this stuff is impossible or a total lost cause, just that I think it easily passes the bar for "hard", and is certainly tricky enough to get right that I personally don't think it's worth the effort to run network-connected services unless they're for a business that I'm hoping will pay for my time and effort somehow.
That's about as unspecific as you can be, isn't it? So, let me give the equally uninformative answer: All of those are either easy to prevent, or they lead to a DoS, which isn't a big concern for a personal email server.
> "The Art of Software Security Assessment" is 1200 pages long
Haven't read it, but by its TOC, it's mostly irrelevant to the given problem (except maybe that it teaches you to not write you personal mail server in C, who would have thought ...).
> "The Web Application Hacker's Handbook" is another 900 pages.
Didn't I say something about avoiding complexity? No, you should not build your mail server as a web application, in case that wasn't clear.
> Nobody has solved all the issues discussed in those (and many other) books in such a way that makes this no longer "hard".
Well, how about you give an example of one problem that you think is still hard enough to be a major obstacle to implementing you own secure personal mail server?
> [...] that I personally don't think it's worth the effort to run network-connected services unless they're for a business that I'm hoping will pay for my time and effort somehow.
How is it that you suddenly switched to "services"? I thought we were talking about "network software"? Am I supposed to believe that you are not running a web browser unless it's for a business that you're hoping will pay for your time and effort somehow? How about web applications? Or what is your justification for a distinction between "services" and "network software", as far as possible security impact is concerned?
As for the rest of it... there was a time when I would have bothered trying to delineate some specifics for you, thinking I was providing some educational service or something, or that maybe I'd learn something from you in the discussion, but I'm getting the sense that if I did that, you would just be dismissive of all those things as "not a big concern", "because of complexity", "because of C", etc., and I would be wasting my time.
There are plenty of places you can go read about this stuff if you want to, but I don't think you do, I think you're just looking for an argument. Good luck to you!
Well, a web browser is accessible across a network. An email client is accessible across a network. What fundamentally distinguishes a mail server from either of those that justifies rejecting to run one of those yourself outright, while presumably being fine with the others?
You seem to have some implicit assumption that a piece of software that opens a listening TCP socket is somehow much more vulnerable than one that doesn't. I don't think that that is justified, and if anything, gives you a false sense of security.
> but I'm getting the sense that if I did that, you would just be dismissive of all those things as "not a big concern", "because of complexity", "because of C", etc., and I would be wasting my time.
I am not dismissing anything, I am just pointing out why those aren't problems. If you disagree, feel free to disagree. And maybe make an actual more specific argument than "there are a lot of things one can do wrong".
I mean, how is it relevant that there are tons of ways to screw up buffer handling in C when the obvious solution is to just not write your own personal mail server in C? Yes, it is hard to write secure software in C. Very hard. Yes, there are cases where there are good reasons to write network software in C despite that. Is your own personal mail server one of them? No.
Or how is it relevant that the web is a hugely complex mess of inherently insecure APIs when the task at hand is writing an email server? Just don't write your email server as a web application and you don't have to deal with any of the myriads of security problems that plague the web.
> There are plenty of places you can go read about this stuff if you want to, but I don't think you do, I think you're just looking for an argument. Good luck to you!
Thanks, but I think I have found enough vulnerabilities to not need any introductory books to understand how people constantly screw up security in their code.
I honestly hope you're right that someday I'll work on a project and find myself thinking "wow, it wasn't even that hard to get the security story right on this", and I guess that day has already come for you, but it certainly hasn't for me.
Well, maybe. But I don't really think it takes all that much expertise. Maybe it does, in the sense that there are so many tools out there that you in principle could use to try and build an email server, and without any expertise, it might be hard to tell which tools to use, and, more importantly, which ones to avoid.
So, without expertise, you might end up stuck in the local minimum that is C, say, which in turn then requires a lot of expertise and effort to build something secure, because there are just to many pitfalls to be constantly aware of if you want to pull it off. But if you manage to avoid that local minimum and use a memory-safe language with garbage collection, suddenly, avoiding buffer overflow exploits is trivial. You don't even need to understand what a buffer overflow is and why/how it can be exploited. They simply don't happen, without any detailed expertise needed.
The thing is that you very well might end up writing your email server in a memory-safe language by accident. Writing a secure email server in C by accident, though? Extremely unlikely.
Also, many possible security problems that plague software simply don't apply to email servers. The web is a mess and difficult to build something secure for. But email servers don't have to deal with the web, so you cannot even accidentally stumble into those problems when you are writing an email server.
I think you may be living in a bubble. Building any working software takes a lot of expertise. Maybe you just mean "expertise in addition to that necessary to build any working software". In which case, maybe I don't disagree with you so much. But again, we're just debating the definitions of words like "hard" and "expertise"!
There are more security issues to be concerned about than buffer overflow. Just as an example, your email server needs authentication and authorization, which are also tricky to get right. There are plenty of logic-level security issues like that for which just choosing a memory-safe language is not a solution.
There's a lot that goes into making a secure system. And there is considerable economy of scale -- a large company can afford to have 24/7 monitoring and issue escalation processes. You cannot.
Which applies equally to gmail.
> If the government knocks on the door of your cloud hosting provider, will they cooperate freely?
Which applies equally to gmail.
> When your linux distro announces a security issue to a trusted list ahead of a public announcement, do you have someone on that list ready to respond to the issue before your server is hit by a 0-day?
Which can be automated.
> There's a lot that goes into making a secure system. And there is considerable economy of scale -- a large company can afford to have 24/7 monitoring and issue escalation processes. You cannot.
And you don't need to. If you don't need to do this with your email client, you also don't need to do this with your email server.
Also, there is also a major economy of scale to compromising google instead of your personal email server.
No, of course, there is no "perfect security" (whatever that is). And that's true for both gmail and self-hosting. But it is perfectly possible to write an email server that's very unlikely to be exploited surprisingly into remote code execution as a single developer.
If you are talking about easy to detect abuses, such as participation in a DDoS attack, say: (1) those aren't really that bad for you, and if you fix them within 24 hours, not even for the attacked, and (2) your upstream provider might just notice and shut off your machine until you are awake if it's really bad.
But with all that said: Neither of those is particularly likely to happen anyhow if you take care of installing updates in time. What gets owned are high-value targets, bad passwords, and machines with ancient software (as in: security update has been available for a few years), the latter two of which are trivial to avoid.
Everything that someone else could do on an email
server that they run for you, they could just as well
package into, say, an install CD image, which you then
could use to run your own functionally equivalent,
hopefully well-configured and well-updated, mail server.
> Then you'll understand there are no experts.
I disagree. There are, but experts have to keep learning in order to maintain their expertise.
> There are literally more topics than one could read in a lifetime.
Literally no part of that sentence is accurate.
If you're the type who understands why not to do it, but does it anyway "for fun", that's probably OK because you understand the risks and what types of bad things can happen (and probably have some idea of how to prevent and/or fix them).
If you don't and "just install OwnCloud" or whatever the flavor of the week is for "own your whole social stack by installing linux on that dusty tower sitting in your garage and STICK IT TO THE MAN!", you're likely setting yourself up to get your identity stolen at the very least, with having the FBI come and seize all of your electronic equipment not out of the picture, e.g. in the case of some hacking crew using your poorly-secured box as a proxy / jump host / C&C / worse. No thank you very much.
I know most of this stuff because I've been there, and the people like me who have done it for years and see the horrors are honestly trying to help those who might be tempted. I set up exim with SpamAssassin and clamav on Debian before I could drive with my own bind server hosting all the zones out of the basement, Server 2k3 running AD, Zimbra and slapd + freeradius on opensuse. I did it to learn, and after getting it all set up (which took weeks back then) and getting inbox deliver to gmail (again, in a time when doing so had a much lower bar), I thought I was some kind of Internet wizard. I did learn a ton before my first tech job doing this, and I also learned why it was a bad idea. I got a letter from my ISP a few weeks after getting this all online (of course I had remote access set up via pfsense with SSH on a non-standard port with root login disabled) informing me that there was a botnet C&C running at our home IP. Figuring this was just a mistake, as I was active on IRC at the time, I dug into the traffic and basically found that what the ISP said was true. I was able to set up a mitm on pfsense to get a log of the IRC activity in order to figure out exactly what was going on. Most of my set-up had been taken over without me even realizing it.
Quite a few years later, I worked in web hosting for a few years. The amount of terrible things which happened to unmanaged colos and dedicated servers was astounding. There were quite a few legal disputes which involved data on systems we physically controlled. There were companies which went out of business due to data loss. There were others which were blackmailed by hackers. Others simply went under. There wascredit card theft, databases dumped, unsalted passwords, you name it, and plenty more we probably didn't know about. The Internet was a very hostile place for a server years ago, keeping hackers busy looting and pillaging. I'm sure it's much worse now, with more profit to be made, tons more stuff to be pwnt, and tons more people launching poorly-configured stuff.
Leave the MTAs to the pros. I promise that somewhere out there, someone is getting paid a good amount to spend quite a bit of time keeping that mail server online and in good shape. They are very likely better at this than you, have way more clout with their domains than you (no one cares if your domain / NS / netblock / AS gets blacklisted, but if gmail / yahoo / aol / Comcast stops working, postmasters@ all over will jump to fix it), and are getting paid to do it, likely as a full-time job, likely for many systems. You may be able to set it up once, and make it work a few times, until it annoys someone who cuts it off, at which point you have very little recourse and very little chance of ever making it work as well again.
The point isn't that you can't do it, the point is that it's not worth the risk when there are so many other great options out there.
Granted it's still too difficult for Joe User but initiatives similar to Sandstorm may make it a reality one day.
So your saying google (and a few others) broke a fundamental part of the internet?
Yes, yes they did.
I run my own mailserver for personal email as well as for rsync.net and 0x.co. It's trivially easy to do from a technical standpoint and simple to lock down to prevent abuse.
The problem is that yes, in fact, google/gmail will drop email, by default, into gmail users' spam folder from a privately run email server.
Specifically, I mean that with squeaky clean IP addresses from your own squeaky clean private allocation, with a mail server properly configured with DKIM signing and a DMARC record, etc., etc., and 10/10 non-spamminess scores from mail-tester.com, you will be treated as spam by default by gmail.
It's total horseshit. It's "evil".
If the most simple, least common denominator, least technically challenging aspect of being a peer on the network is thwarted by google, how will they behave towards things that are actually interesting ? How will we run our own asterisk exchanges and create our own micro-cloud-hosts and host our own websites from a routable ipv6 address that we have in our home ?
That alone is the reason to not use gmail, if any of you think it hasn't happened to you, trust me it will.
Well, sure, not dangerous like underwater welding, or electrical powerline installation or something, but it's something that requires constant attention to do right.
Even people who do their homework and start out vigilant, can eventually develop entropy and then 3 years later you get a message from your host telling you your machine is being used for spam - like this guy who is also coincidentally on the front page:
https://news.ycombinator.com/item?id=12384880
Sure, it's not dangerous, but it's awfully time consuming if/when things go wrong.
Really, the likelyhood that something terrible happens to your own email setup isn't all that great. Not much greater than that something terrible happens because someone breaks into your web browser, say.
What it costs in a year is less than the time/effort it would cost me to maintain a fully patched and up to email date server.
Perhaps, but that can be said of most things. Model rocketry, glass blowing, hell even woodworking. That doesn't mean people shouldn't roll up their sleeves and get into it because 'some megaultracorp has perfected the process', for some incredibly generous definition of 'perfected'.
Running your own server does in fact take some time and a little bit of elbow-grease, but scare-articles like this does nothing to further an open internet. As the tech-elite (for the most part), we should be encouraging folks to host their own data, learning how 'those damn computers' work, and do our best to raise the next generation of engineers, sysadmins, devopsen, docker deployers, or even just computer dabblers 'right' so they at least have some concept of How The Internet Works(tm) beyond "oh, send this dict to an API...". If hosting your own email server is too scary for you at this time, suck it up, do it anyway, and figure out what needs to happen to make it better for the NEXT guy. Write down what breaks and when. Publish that data. Let us learn from your fail, as you learn from ours. Holler at your nearest Google or Comcast or Centurylink engineer, suit, or peon to un-fsck their infra. It's not glamorous work, but it is necessary.
As fun as it was, I now find it less hassle to just outsource that to a third-party that specialises in it.
I think this could be phrased better. Or at least expanded on what "dangerous" means here. Someone breaking in is just one tiny part of it. Getting incoming emails dropped due to misconfiguration is another. Then there's the whole issue of getting your mail accepted.
After such comment there's always a few people ready to say that it just worked for them and they don't have any issues. That's great. As long as they actually know that's the case (how often do you test for your mail being silently dropped? how often do you send copies to yourself on gmail, yahoo, etc and verify they don't go straight to spam), and as long as nothing around their network changes (two spammers in your /24 means you're suddenly being filtered, someone changes description of your range to dialup and you're being filtered, etc.).
Running your own email long term can be easy, hard or a full time job. And that can change day to day. "pretty dangerous"? No... Closer to: suddenly requiring a great deal of work while you're wondering if you dropped an important document.
I once looked at writing an immediate email forwarder in Go, a mail forwarder for servers that don't receive mail. It would, upon receiving a message via SMTP but before replying with a status, open a connection to forward the email, and return the status of the send to the input SMTP connection. No local message storage at all. Runs in a jail. It's not hard to do this as a minimum viable product in Go. But all the anti-spam stuff you need today makes it a far larger project.
It would be interesting to write both that, and a pure SMTP to IMAP server. No local mail access, just pure IMAP. Together, the two programs add up to a mail server much simpler than most today.
Both qmail and postfix have security models pretty similar to the ones you describe - the process that accepts the mail in either has almost no privileges. See here for how postfix does this: http://www.postfix.org/OVERVIEW.html
The problem is it then needs to be popular enough for widespread use to sand down the rough edges, which is chicken and egg.
A way out of that might be to stamp out the common refrain that you shouldn't run your own mail server.
I've since moved all my mail for the last year (for about 10 domains) via a cmail setup and so far have not had a major hickup. Sure, it may not be as full-featured as other mail servers, but thats not the mission statement ;) One of the main reasons I'll probably never go back (besides pride) is the ease of altering the runtime configuration via our admin tools (eg. adding a new domain to the mail server is one command in the simplest form).
Some of my peers who contributed code or bug reports also run instances and are very happy with the amount of configuration they need to support for cmail vs. the amount of things to do to run for example exim.
Code-wise, it's still C (because that's what I'm most comfortable with), but hopefully it doesn't read as ancient :) I've tried to keep the bulk of it understandable and modular. We extensively use static analysis (clang analyzer and Coverity Scan) as well as fuzzing with afl-fuzz as part of our testing.
IMAP is the only major protocol we do not yet support (though the groundwork is laid in another branch of the main repo), and that's mainly because the specification is a major PITA (at least compared with the relatively sanely specified SMTP and POP3). IMAP also implicitly requires multithreaded handling of connections, which so far cmail has managed to avoid, because in my experience multithreading is a) not necessary in most sane server designs and b) a major source of non-trivial bugs.
Edit: Oh, since you mention build systems - we use a simple makefile, so yeah, no m4 macros ;)
The OpenBSD project has OpenSMTPd, it's C of course, but it was initially released in 2013 so it's still fresh.
The codebase has been audited by Qualys recently, so its security has been verified (and detected issues fixed).
Of course, I'd still rather have something written in e.g. Go than C.
There just isn't much of an attack surface there.
The real problems are spam and convincing other mail servers to accept your email.
Since a server is a much juicier and visible target than any client, expect attacks.
Why do you think that?
Isn't a web browser that has access to your gmail inbox just as juicy as your self-hosted imap server?
I don't really see why a "server" would somehow be inherently a more interesting target. Google's servers, sure, they give you acess to billions of mailboxes, but your own private email server?
It's a nightmare. I postponed this for 15 years knowing how bad it will be. It's far worse than that. During the last two years while I've been trying to figure it all out, I seriously thought about ditching e-mail altogether. The last year was spent reading various documentation.
The DNS MX records aren't even in yet, and I already see port scans and hack attempts on port 25, probing for security flaws and configuration errors, already trying to exploit my MTA infrastructure.
And before I go live, I have to recompile OpenSSL, SASL, antivirus and the MTA at the minimum, to get all the security fixes rolled in. I'm not even done yet, and the antivirus engine is already complaining that it's outdated.
It's a nightmare.
So? How many webservers are constantly _trying_ to exploit your web browser infrastructure? It's just that you don't have a log file that's telling you about it, but that doesn't mean it's not happening. And in any case, it doesn't really matter. Just because someone is testing whether your web browser is some ancient internet explorer, doesnt mean there is an actual risk to you, unless you are running an ancient internet explorer.
> I'm not even done yet, and the antivirus engine is already complaining that it's outdated.
Just forget about antivirus, it's snakeoil anyhow?
Nat Meysenburg is a technologist at New America’s Open Technology Institute
Ahh yes, "New America", who runs that again? Oh right, long-time Clinton lackey Anne Marie-Slaughter who was "devasted" when she thought Hillary criticized her a little bit that one time [0], along with Google's Eric Schmidt who runs a stealth startup for Hillary [1] named "The Groundwork" complete with an Illuminati-esque logo [2]
[0] http://www.politico.com/story/2015/11/hillary-clinton-emails...
[1] http://qz.com/520652/groundwork-eric-schmidt-startup-working...
Not to mention that it's pretty normal to feel at least a bit bad when someone you were working for and admire says you're whining whenever you talk about your difficulties.
Plus I'm pretty sure Schmidt is beholden to no-one at this point. "Fuck You" money is an understatement....
Though I can get these people rationalising a defence due to emotions, I don't think there's a big web of conspiracy. Popular people get defended by a lot of people...
Kind of like how he was explaining how the author didn't outright defend Hillary, but subtley did defend Hillary.
However. I know setting it up is a PITA. And if I move to it I am stuck with it.
The various mailinabox solutions sound great, but I can't quite tell if they do what I want and email is too damn complex to vet myself. And they need an entire bloody box because they don't play nice with containers apparently (on last google) - all to securely parse some text files. Overkill or what.
Reminded me of this other gem article: http://www.clickhole.com/blogpost/if-black-lives-matter-isnt...
Basically trying to prove a point with very... umm... poor arguments :)