Court rules search warrant applies to US emails stored on foreign servers | Hacker News Reader

Court rules search warrant applies to US emails stored on foreign servers

casetext.com

161 pointsmarklyon125 comments

As an EU citizen, I wonder how this conflicts with EU legislation. As I understand it, if Google holds data in the EU, then EU law says that they may not move the data outside the EU without the data owner's permission. Now the USA insists that Google move data from (among other places) the EU to the USA based on a USA warrant.

This could make gmail pretty much a non-starter for many uses in the EU and may possibly even make it problematic for some email users to correspond with a gmail address.

I'm pretty sure it's all covered in here: https://www.state.gov/documents/organization/180815.pdf

I wonder how it will effect other vendors - ms is hosting email and storage for lots of UK.gov and health stuff, also amazon.

I don't know about amazon, but MSFT seems to have this issue covered with a completely separate non-public Azure Germany region. https://azure.microsoft.com/en-us/overview/clouds/germany/

> Each account holder resides in the United States, the crimes they are suspected of committing occurred solely in the United States, and the electronic data at issue was exchanged between persons located in the United States.

I don't feel like this is a bad call.

Wasn't there a judgement recently that non-US residents can't have their stuff searched just based on a US warrant?

Wasn't there a judgement recently that non-US residents can't have their stuff searched just based on a US warrant?

Ha! Somebody forgot to tell the Kim Dotcom cops!

Data "ownership" is a somewhat complex issue. When Google, or any other data broker offers some service in exchange for the data, they are obtaining their own data from the user. The idea of "ownership" in that case differs, as the content and 'meaning' of the data is now no longer owned by the original owner. In fact, the data itself has lost all or much of its value in "ownership" or "privacy" by the user, and this is reflected in an economic sense by the bulk pricing data brokers like Google, etc... offer through various services for adverts, etc.

In terms of "moving data" by means of "ownership", Google and the other companies, already state in their ToS that they can move the data between locales at their whim. The servers, hardware and networks they utilize are owned or rented by them. Paired with the ToS, and the simple fact that Google offers these services in exchange to exploit their users' data as a currency, Google is not concerned about the privacy aspects.

They are simply concerned about the value of the data they are gathering.

Legally speaking, in the EU, you can put whatever you want in your ToS but if they conflict with EU law they are void.

Practically speaking, I agree with you.

This seems reasonable, they have a warrant and the judge seems to have considered the case fully.

I don't get why so many people are advocating a world where the government can't do this. Why should your email hosting provider have the ability to make law enforcement decisions? Why should email be treated differently from something like my bank account or diary? This seems like the optimal outcome.

There's a lot of libertarian fantasyland stuff in tech forums.

In reality, there's a built in tension between what a prosecutor or regulator wants and the interest of the individual. 200 years of law in the US was built around your "papers", mostly held in your home or on your person.

Now you have this new world where your "papers" may be a detailed diary of every place you've been for most of your life. My wife's phone has immediate access to 60,000 pictures with date and geo-stamps dating back to 2003. Oh, and by the way the storage of this stuff may be on a phone or computer in your possession, or in the hands of a 3rd party like iCloud, Google, etc.

The law is behind in it's understanding, and the various authorities want to have the ability to have total information awareness.

It's a question of jurisdiction. A company with subsidiaries in multiple countries necessarily needs to ensure that the subsidiary in a country complies with all the laws of that country. However, that doesn't mean that, for instance, it should be possible to serve a warrant in one country to obtain data controlled by a subsidiary in another.

It's entirely possible to compartmentalize hosting and authorized access, such that one subsidiary has no access to the data stored by another. That's critical if, for instance, you need to meet regulatory requirements such as the "data protection" laws in the EU. Rulings like this one, if not successfully appealed, will make it nearly impossible for any company with subsidiaries in multiple countries to meet such regulatory requirements.

Let's say a Russian company opens up a chain of coffee shops in the US, has surveillance cameras for theft prevention, keeps various records on its customers etc.

Now the Russian government issues a warrant to demand data from the US subsidiary. Do you think that US jurisdiction should have no say in the matter?

If the Russians compelled their US subsidiary to hand over the data in violation of US law, how long do you think they'd be allowed to operate in the US?

The end result of this US overreach will be trade wars and the further rise of national enclaves of the Internet. I'd argue that that's not in anyone's long term interests, least alone the US's.

On the contrary, I'd argue that it's in most users' interests for there to be national enclaves of the Internet, for just this reason: Your data is within the jurisdiction of the laws of your country. If this is clearly defined, you do not have to worry about another country's laws, like Russia, affecting your data.

It isn't in certain corporations' interests though, because the whole cloud provider thing hinges on shoving everyone's data into a small number of datacenters globally. I think this is an example of the quite common situation where corporate interests and individual interests are not aligned.

For what it's worth, there are technical limitations to this idea. IP routing makes basically no guarantees about what countries your data will travel through on the way to where it's going. Crypto helps with this, but it can't solve everything. (Sometimes all you want to know is that I visited a certain website, for example, and it doesn't matter that you can't read my traffic.)

There are limitations caused by the software configuration of most AS's. Significant work is being done on novel approaches to IP routing with better security, scalability and control guarantees by ETH Zurich, as part of the SCION project. It's sponsored by the EU, Google, Swisscom, the NSF and a few other entities.

Link: http://www.scion-architecture.net/

This is already "live" in the sense that it is implemented and there are 14 router nodes active, and it is meant to be incrementally deployed. I haven't gotten around to hooking up a client myself, that will be an interesting weekend project someday. Any interested ISP can set up their own node and join this project.

Some cool things about SCION and its extensions:

* Isolation domains: can be used to enforce traffic to never leave a given routing sub-plane, for instance a country or a state

* SIBRA: volumetric DDoS mitigation and enablement of the creation of "dynamic interdomain leased lines"

* HORNET: high-speed onion routing

Are you suggesting that users should be banned from using internet services that are run from other countries?

No, but if I voluntarily use another countries' web service, I'd say I'm handing data over to that country's jurisdiction. We should be wary of companies who can't even guarantee our data is remaining within the boundaries of our own laws.

A lot of the "cloud" is an obfuscation. At the end of the day, our data lives on hard drives on servers that are physically somewhere. We should know where that is, and what laws apply to it. When I selected a web hosting provider, I picked the one that disclosed it's datacenter location, the physical security and environmental protection measures, etc. I know the physical place I am entrusting my data to.

>We should be wary of companies who can't even guarantee our data is remaining within the boundaries of our own laws.

Global companies cannot always guarantee that because they operate under a large number of sometimes contradictory laws that keep changing all the time.

Also, if you consider a web app that lets users communicate with each other then the jurisdictions involved might be all of the following:

1) The web site owner's jurisdiction.

2) The hosting firm's jurisdiction.

3) The jursidiction of every single user.

And all of these jurisdictions may have agreements governing data exchange and law enforcement.

This is bound to be extremely complicated.

When those datacenters are in your own jurisdiction, but the Global HQ of the company is in a different jurisdiction, then you have a problem. This is bigger than corporate vs individual interests.

What you're really saying in the first part of your comment is that there would need to be 'an independent google' for every jurisdiction. That isn't going to happen.

The current situation is a little different since:

"Each account holder resides in the United States, the crimes they are suspected of committing occurred solely in the United States, and the electronic data at issue was exchanged between persons located in the United States."

In this case it does seem reasonable that the company should comply.

Well I'd agree that an independent google for every jurisdiction won't happen, but I'd bet on independent googles in major legal blocs (e.g. the EU) for exactly this reason.

Whilst this case my seem straightforward, the precedent (that data held in other countries can be requested by a court in the US based on the fact that Google is a US corp) would lead them necessarily to establish legal entities to prevent that spreading.

> * ... would lead them necessarily to establish legal entities to prevent that spreading*

They already have multiple legal entities. And there are two different things being conflated here.

1. Info/data that was generated and pertains to people in one jurisdiction that just 'happens' to be shipped to a datacenter elsewhere, just because Google can. This seems to be the current story.

2. Info/data that was generated and pertains to people in one jurisdiction that a legal system in a different jurisdiction tries to claim rights to. This is also happening and has implications for service users, privacy law, etc.

This isn't an American court asking an American company for data held abroad about a foreigner. "Each account holder" charged "resides in the United States, the crimes they are suspected of committing occurred solely in the United States, and the electronic data at issue was exchanged between persons located in the United States."

The analogy is an American coffee shop opening a branch in Russia, filming Russians, storing that data in America and then refusing to turn it over to Russian authorities. Practically speaking, you surrender as collateral your property and people in a country's jurisdiction when you choose to do business there.

There are challenges with that approach though. If a UK corporation does business in the US as well as the UK and uses Google, should it be possible for a US court to compel production of their e-mail.

Should the reverse be true, i.e. should a UK court be able to compel google to produce data on from one of their US customers if that customer had done business in the UK?...

it's a very sticky area, personally I think it will lead down the path that Microsoft have started on which is that the cloud providers will end up with partially independent companies running in major legal jurisdictions and the data will be held in those countries.

It is because solving a non-problem here creates very real ones. And your analogy is backwards.

The reason why it is a non-problem is that there is an established procedure for this that works quite well. Take the warrant from a US court to a court in the other country, cite the appropriate international treaty we have with them, and then access the information in that other country in accord with local law.

This is how we have to go about getting access to foreign bank accounts. Or getting your personal papers searched if you lie abroad. The same standard should apply to emails.

But US courts wish to apply US law to emails and people who do not live in the USA. Why should US law apply to them? What are the limits of a government of one place applying its laws elsewhere? Do you wish France to be able to access your emails? How about Iran?

Furthermore complying with US decisions like this puts Google in conflict with foreign laws about data privacy. This issue is already costing them a lot of lost business. What penalties are appropriate for their violations of local law?

Reading the case at hand, there is a complication. The issue here is that the crime happened when all involved lived in the USA. The perpetrators have since moved abroad and therefore live outside of the USA. There is no question that at the time the crime happened, the warrant would have been reasonable and valid. The question is whether the warrant is reasonable and valid now.

Apparently Google has moved the data abroad so it is no longer on US servers. Google regularly does this - they try to place information close to where it gets requested for reasons of efficiency. On the other hand this fact is something that nobody outside of Google can verify. Are we sure that there are no copies left in the USA? Did you look in your backups? Are you sure?

So on a physical analogy, it is like a warrant for documents that were purportedly carried out of the country. If they really were, then you would appropriately have to deal with that other country. If they weren't, then Google should produce the documents.

The court doesn't want to debate Google on unverifiable facts. The crime took place in the USA. Google can easily produce the documents. We only have Google's word that this requires accessing data from abroad.

I can make arguments for this on both sides.

There are a few problems with your proposed solution, based on the facts as described in the opinion (and thus as alleged by Google's lawyers):

First, Google's architecture supposedly constantly moves data from place to place, so US law enforcement could spend considerable effort going to a court in France and convincing them they should hand over the data, only to hear, oops, while you were doing that we moved it to Canada. It also potentially splits data into many parts, requiring the US to go to several different courts. (Google also didn't even say which country or countries the data is stored in, though I suppose the US could probably subpoena that information.)

Second, Google says that the only system they have for giving data to law enforcement requires going through employees in the US, regardless of where the servers to be accessed are located. Makes sense, since their architecture is location-agnostic by default, and handling law enforcement requests is a specialized job. But a decision from a foreign court is not binding on those employees! US law enforcement would have to try to get the country hosting the data to go and physically seize Google's servers, and manually search through their hard drives. Admittedly, they'd probably only have to pull a stunt like that once before Google decided to get more cooperative, but it's still a messy situation.

FWIW, I agree with your concerns about foreign data privacy laws, but the solution can't be to effectively make certain data immune to court-ordered access of any kind. (Well, at any rate, such a solution would be contrary to political orthodoxy even in the EU, though cypherpunk types might be happy with it - a category I count myself in to some extent. :) Maybe the US needs to pass laws explicitly addressing this situation, but considering the current political situation, I wouldn't expect that to happen anytime soon…

> "...an individual who resided in the United States and was a target of an investigation pertaining to the theft of trade secrets from a corporation located in the United States..."

Google turns over data like this all the time. In fact, ever thing seems reasonable about the warrant, so why are they fighting this? It's right there.

I suspect somewhere within either the Google corporate structure or structures of one of their trusted partners are individuals that have done some thing that could potentially put Google in hot water. It's just a theory, but I think it's plausible.

The linked decision, if you read it, discusses this. Google has in the past responded to similar requests but after the Microsoft decision they responded differently.

The problem seems, to me, to be one of jurisdiction and international legal privacy differences.

US corps like Google want to be able to provide services to people all around the world. If they can be compelled by a judge in any country to provide data on any user, that could cause a lot of problems.

for example say a user in Germany makes a deal with a user in the USA which then goes bad and the case is heard in the US. In Germany the privacy laws are different and could prevent the data from being used in the case. Can a US judge require that data to be produced?

Next step in the chain, a russian user and a US user have a deal, which goes bad, and a russian court requires the data from the US user to be provided. Should it?

Now the obvious argument here is one of american exceptionalism, that google is a US company and therefore needs only concern itself with US law. Unfortunately that leads down the line of companies in other countries not making use of Google's services as they can't legally allow data they process to be subject to US laws...

So you are fine if Chinese government wants your data stored in USA by a Chinese company ?

The whole point of keeping data in a different country is to keep it inside the jurisdiction of that country. This makes American companies less competitive. US government already treats non-Americans like shit, they might want to access all emails of all non-American people using Gmail where the data is stored in EU or India.

I think American companies must lobby with foreign governments to put an end to this.

This is all very normal indeed and is well covered under MLAT. Have process MLAT requests before, pretty standard stuff. I'm pretty sure this is the relevant treaty: https://www.state.gov/documents/organization/180815.pdf

Lack of trust. You need secret warrents, national security letters, etc? I assume that with all the we can't tell you this and we can't tell you that, you are up to some nefarious thing.

Now if you have shown consistently that you had opportunities to cheat me but haven't, I might trust you on this one. The US government? Fuck no.

US government should not be able to make decision like that when it comes to data of foreing citizens on foreing soil.

tl;dr: Gmail(USA) required to provide data of persons in the USA, of communications between people in the USA orded by a USA judge.

Sounds fine with me, even when Google has it hosted in some other country, this seems to me a fine example of correct usage of the law and jurisdiction

Yea, it seems like if they found the other way, every sizable corporation, or really any sizable group, would host all their data in the Cayman Islandsor wherever, making them more or less immune from US subpoenas. Which would be bad for any attempt to hold them legally accountable (and, relevant to the HN crowd, bad for Tech workers in the US as storage providers move their facilities over-seas).

>bad for Tech workers in the US as storage providers move their facilities over-seas

Great for the rest of the world's rights though!

Why are they even storing this USA data outside of the USA? Seems weird to me, no?

I'd bet its cheaper.

So what MS, Apple and Google need to do is basically to set up a "hosting provider" company in a country with strong data protection laws (Switzerland? Iceland?) and delegate all their hosting operations to this company... that could work to really prevent this sort of espionage via court, imho.

Microsoft is already doing that: https://azure.microsoft.com/en-us/overview/clouds/germany/

They're using T-Systems (a subsidiary of Deutsche Telekom) as a trustee. It's about 25% more expensive than the normal Azure.

Personal opinion on the quote from the Azure page... they quote T-Systems as "well-respected IT provider".

Anyone who had ever to do business with them will respectfully disagree on this one. They were involved in Toll Collect and supplied the software and hardware infrastructure for Munich schools. The scandals surrounding Toll Collect speak for themselves, and a couple of years back when I still was at school I had enough fun hacking their cr.p to pieces to never ever trust anything with T-Systems involved. My personal favorite was "www-data ALL=NOPASSWD: ALL" in /etc/sudoers, and students were allowed to upload PHP scripts to their personal web space. You can guess what my entrypoint was, and I believe it wasn't fixed despite multiple warnings for over eight years (the reason was that they employed a PHP-written framework whose name I've long since forgotten, it was well known at the time and was a bit similar to Webmin...)

That said, the Azure Cloud seems to be quite stable, but I guess that all operating experience as well as the software stack comes from MS and T-Systems only has to keep the servers up and running.

My guess is more along the lines of subsidiaries or licensees being set up within the political boundaries in which their users exist. That means some users will have to receive lower quality or higher priced service depending on local infrastructure.

An effort like this would be interesting as it would be akin to the degree of effort put forth by large companies making financial tax arrangements. Though I am sceptical that customer privacy is as motivating as money, I would love to be surprised.

Or, those interested could just use a Swiss/Icelandic email provider.

I'm a fan of KolabNow (fully FOSS, green energy, no US presence, POWER8 servers).

I recently started using ProtonMail. It's Swiss and (asymmetrically) encrypted at rest.

"POWER8 servers"

Produced by NSA's original and longest running partner. :) Doesn't bother me as KolabNow is just a storage and transport to me. I layer GPG-encrypted files on top of it.

these articles push me more and more to drop using gmail in favor of "self hosted email". i really do not understand why we do not think and give it all up for free to google (and compromise our personal security in the mean time).

I think Google fought very hard for it's case.

"Self Hosted" is certainly not going to fare much easier in this same situation.

I do not think that the same result would be had if Google was refusing to deliver data on a non-US server for a non-US citizen.

I think that in this specific case Google was able to push back much harder than a typical individual would. Who can afford a team of world class lawyers to go up against the fed?

And they can still appeal potentially up to SCOTUS.

Self-hosted would absolutely fare better in this situation. It's not perfect, but at least you would know you're being investigated. This whole mess is predicated on the fact the government is allowed to request your data from Google without much fanfare because they are technically in possession of it. The DOJ was able to successfully argue that user emails are actually business documents owned by the email provider.

This breaks down when the person they are investigating is also the email provider.

You'd have to own the server too, at that point.

If you were a VPS or even a dedicated lease, or shared - I think the fed would be able to pull the same thing.

Doesn't matter if you encrypt your emails, since you have control of your server. Let the fed have the encrypted data.

... at which point you might get to rot in jail for contempt until you fork over the decryption key.

I don't have a convenient link to the xkcd comic right now that talks about the difference between theoretical and practical security. ;)

To quote Spivak higher up in the comment chain:

> It's not perfect, but at least you would know you're being investigated.

Am I gonna be jailed for refusing to decrypt my files?

IANAL, but my understanding of current American law is that if the material is deemed by a judge to be evidence, and you can decrypt it, and you won't decrypt it, you can be held in contempt of court.

So they have to prove the evidence is in my emails first, and then prove my emails are on my "self-hosted" server. And I will have full control of my own data.

That's not how contempt works.

If the prosecution asserts you have evidence material to the case that you would be legally required to render and won't render it, and the judge believes you probably do, that's it; they don't have to prove the evidence is in your emails to search for the evidence in your emails. Fail to render up the emails or render them up in an intentionally-obfuscated form, and they can hold you in contempt at pretty much the judge's discretion (your mileage may vary depending on severity of crime and state law, where applicable).

(Personal observation: people of a technical bent seem, for whatever reason, to underestimate the wide swath of power the legal process has in investigating a murder case).

Not necessarily. You can use POP3. If you pickup often they would find precisely nothing.

> It's not perfect, but at least you would know you're being investigated.

And this basically is the reason multi national companies self host email servers

If the feds want your "self hosted" emails they will break down your door, shoot your dog, and take every computer-resembling object on the premises.

It's a lot more effort than just asking google. The goal behind practical security is to make things expensive, not impossible.

You could "self-host" on a cloud server in, say, China, or Russia, or Iran (if they have any hosting services.)

I mean, the governments of those places will probably snoop your emails, but if their contents have nothing to do with them, they won't care. And they have no treaties with the US to force their hand to turn anything over.

Think of your server as Edward Snowden. What country should it hide in, so the US can't legally get to it?

Iceland.

"I mean, the governments of those places will probably snoop your emails, but if their contents have nothing to do with them, they won't care."

Can't you say the exact same thing about the US government?

> they have no treaties with the US to force their hand to turn anything over

Sure, but that doesn't mean they won't happily exchange that info as part of a deal with the US, assuming your data is valuable enough.

You're forgetting the possibility of rubber hose cryptanalysis applied on you. In fact just by hosting in such places, you're probably inviting more attention.

>the governments of those places will probably snoop your emails

Uhm, how? Gmail supports Transport Layer Security (TLS), and >80% of their emails to and from other providers do as well (https://www.google.com/transparencyreport/saferemail/). Reject non-TSL emails, give the server a public key and tell it to throw away the email plaintext, and the only remaining threat vectors seem like "get rubber hosed into disclosing your private key" and "server gets compromised, causing future emails (but not past ones) to get exfiltrated".

SMTP TLS doesn't and can't validate the certs. It is trivial to MITM it.

I'm amazed that

"Nothing to hide, nothing to fear"

Has become

"Autonomy is pointless, resistance is futile"

In about a month.

It hasn't.

You need to remember the fact that already Snowden's revelations have proven that the NSA and other government agencies all have specific budgets for astro turfing activities (manipulating the public opinion by massively participating in online discussions).

And a couple of days ago, there was a nice post on Reddit's front page summing up the situation on Reddit. Reddit is basically completely compromised by whoever has lots of money (government, big industries, etc). Any company can buy astro turfing services nowadays.

So no, you can't trust public online discussion anymore. Not on Reddit and not here. Unless for topics you are absolutely certain that no economic interest is part of the equation.

Ah, but what if this post is anti-Reddit astroturfing?

They can get everybodys gmail with a single request, how many doors are they going to break down, and dogs are they going to shoot?

Are we talking about bulk requests? The case we seem to be discussing here involves "data associated with three Google accounts held by an individual who resided in the United States."

Yes, rights aren't absolute. If the governments wants your data on a self hosted server they need a warrant. In comparison, you have basically zero privacy protections when your data is in the hands of a third party.

In the case under discussion the government has a warrant.

And will be pissed looking at a bunch of encrypted files.

Exactly! They have to have good reason to do so. With "the cloud", they can just use their dragnet storage.

Yup. But you'll have more due process.

At least you know what happened. With google and such, you wouldn't even know you're under threat.

* Gmail's search and spam-filtering are both very good, trained and tuned on datasets no self-hosted product could ever match (and harnessing parallel algorithms across large clusters that'd be quite costly on one machine)

* Google doesn't lose my email; I probably will lose my email, because an email server is backed by a database and doing database backups right is hard if that is not your day-job.

* You can get a good email-receiving experience, but email-sending is very difficult these days if you're a nobody, because a lot of first-stage network-level spam filtering has come down to reputation, and your server IP won't have any (or, if it's a cloud provider IP, will have very likely been used at least once to send spam in the past.) And residential ranges get dinged, too, from the heuristic (stereotype) that the most likely reason to get an SMTP connection from a residential IP is that it's a member of a botnet.

> Google doesn't lose my email; I probably will lose my email, because an email server is backed by a database and doing database backups right is hard if that is not your day-job.

I use IMAP email. My email is simultaneously stored on my server and on every client. If the server is nuked, I can set up a new IMAP server elsewhere and sync my email client to it; I'd want to do this from work where I have gigabit internet, or this would take a while, but it can re-upload all the data to the server.

That said, I'm using a managed account. I'm not communicating about anything that I care if the government subpoenas, and I have no plans to.

Unless we end up in a totalitarian state where constructive criticism of the government becomes an offense. But in that case my public posts would be more than enough to convict me without looking at my emails.

> Gmail's search and spam-filtering are both very good

Google/Microsoft's spam filtering makes it impossible to send e-mail from a self-hosted solution.

http://penguindreams.org/blog/how-google-and-microsoft-made-...

Unless you're sending out thousands of e-mails per day and build your reputation with their magic-goo trust filter algorithm, you cannot run your own e-mail server and run with the big players. They have made self-hosted e-mail totally unreliable.

I think what you meant by "very good" is "piss fucking terrible."

Not for my use-case. There's basically nobody self-hosting email that I want to receive emails from. It turns out the egalitarian "Everyone is an Internet admin" solution favored the spammers heavily over the technocrats or common users; letting Google build a system that defaults to trust-off for self-hosting proved to be valuable for a lot of people.

(Because if a tech-savvy user really wants to email me, they know how to make a throwaway email account and sign the correspondence with a verifiable PGP key).

I had this problem self hosting but was able to remediate it by making sure my server was doing all the smart modern things like dmarc etc... there are some good resources on HN from others who've set up all the right things.

Of course, this all happened after I got bitten during a job search and had most of my applications hit spam folders ಠ_ಠ

If you read the post I linked, I have the correct DMARC, SPF and DKIM records and signatures happening. If I send them to my old University (google) account, I see all that get verified and correct. It doesn't really help.

I suspect part of it might be that it's on a Linode and might be sharing a subnet with other spammy machines. That's probably why MailChimp owns a class C and refuses to sell any of it.

Interesting. I am on Linode as well, that sucks.

Do you host an https site on the same domain? Is your mail server responding to ipv6? (I hear this can be a problem)

Can you recommend a good resource for "how to set up your mail server like it's 2017" for those of us who would like to self-host but don't want to spend 6 months figuring it all out?

You may find this interesting:

https://mailinabox.email/

From recent memory this post covers all the stuff I did with my server: https://news.ycombinator.com/item?id=11946756

There are a lot of testing tools you can run mail through as well to see how well you score.

I haven't had any particular issues getting past spam filters, it certainly takes some time to build IP reputation but in general with nothing more than SPF and RDNS properly configured my mails get through. I really should get DKIM/DMARC working eventually, but my current email solution (GroupWise) doesn't support it natively so I'll have to do some nonsense for that..

> Gmail's search and spam-filtering are both very good, trained and tuned on datasets no self-hosted product could ever match (and harnessing parallel algorithms across large clusters that'd be quite costly on one machine)

As someone who self hosts, this is clearly not true. With gmail I was receiving a lot of spam from various email marketing companies like mailchimp, easymail, etc. There's a lot of these companies and they are mostly country specific, some less, some more shady.

With self hosting it is easy to block their servers en masse and forget about them. Some companies spam the DNS namespace with predictable, but extremely numerous domain names, which are easy to block using a few regular expressions. Try to make filters in gmail for that, if you don't know from which of the 100 domains the next email will come.

Email from hacked servers is also easy to block. It's mostly PHP servers and all you need to look for is mention of eval() in the headers as nobody sane hopefully evals PHP code to send email.

It just took me a month of spending a few minutes every other day analyzing headers of odd email or two which passes through some generic checks like checking if sending IP address has a domain name and figuring out how to block the sender entirely if possible.

Now I don't get any legitimately looking spam at all and what I get is easily filtered with bayes filter in thunderbird.

Anyway, with spam the hard job is checking the spam folder and that's annoying as hell with gmail, because it's always full of crap, and it's not easy to see occasional false positive. Now I only get 1 spam every two to three days and that's easy to check. Legitimate people who get blocked get bounce message immediately and have chance to re-send according to instructions in the bounce, instead of falling into spam folder and feeling ignored.

Much better experience overall.

Actually what is hardest to filter is bounces from gmail servers. I'm not really sure how spammers generate them. They are not in response to anything that I send. It seems like google ignores my SPF records, even though it indicates that it found that the sender forged the From header and sends me the bounce with attached spam that is targeted at me anyway. Quite annoying.

EDIT: I guess I can just reject the gmail bounce if it contains the "Received-SPF: fail (google.com:". Ah!

How do you deal with 'important mass notices' from your utility company or bank? Do they only use their own email servers?

I agree. I have been self-hosting my personal email since 1998, and there was a period when this was difficult due to technical issues related to encryption. But for the past decade or so those issues are gone. The benefits are great. For example, being able to block entire netblocks at the routing or firewall level is an amazing anti-spam tool that is completely free when you self-host.

I spent 2 years trying to get them to understand that alerts from my credit card company were not spam before finally giving up and moving off Gmail. I am very happy to be done with their spam-filtering.

I use filters in such cases – they have worked fine for me so far.

> ...because an email server is backed by a database and doing database backups right is hard if that is not your day-job.

I store my email on dovecot with Maildir storage. For a single or just a few accounts is perfectly fine and you can backup the emails with your favorite backup tool.

Especially when Gmail is not as reliable as we used to think. Gmail DOES loose emails.

http://uladzislau.com/gmail-emails-missing/

> Gmail DOES loose emails.

Sounds like a tagline for an erotic movie, that does.

I think you meant "lose".

To be fair, English is difficult.

Lose could rhyme with rose or close. But it doesn't. Instead it rhymes with cruise, clues, two's, moos, and choose (almost).

Lunatic conspiracy theories about what Google does with email abound. There are several likely and reasonable explanations for why you can't find the mails you expect. Two of the most likely being 1) the messages were never acknowledged with a 220 response from gmail's smtp protocol translators to begin with; 2) the messages were accepted but are still in flight for some reason. Gmail does not accept and then silently drop messages. If they are accepted, they will be delivered.

I recently switched from gmail to ProtonMail. Not self-hosted, but (theoretically) encrypted while at rest using a GPG key derived from my password. Definitely an improvement. If you really want SMTP, as I understand it they have an "SMTP bridge" software that you host yourself that uses ProtonMail as a backend. Seems like a good compromise. You don't have to worry about constant uptime or disk failure, but your data is still fairly well protected.

You probably don't need constant uptime. Sending server will try to deliver for a day or two, usually. Legitimate servers use sending queues.

I wonder if it would be legal/practical for the email hosting company, Google in this case, to provide the data satisfying the subpoena in the form of a word cloud. Once the law enforcement party has that information in hand they can search the word cloud for interesting words and potential phrases and make a secondary request for the email provider to surrender all occurrences of the suspicious words and phrases in order to establish the context of their use in the subpoenaed communications.

This would provide an enhanced degree of privacy to the user since there would not be a bulk surrender of every conversation they've had during the time period covered by the subpoena. I have a hard time seeing why it is important for the government to grab everything when their target is well known to them and the medium they are searching is likely to be text-based and therefore all targeted communications are easily separable at the email provider end.

This didn't work for lavabit who tried to print out the requested data on paper and it won't work for Google.

Personally, while it's sad this didn't work from a privacy perspective we really shouldn't be betting such important moral concepts such as privacy on minor legal technicalities.

That's a losing battle even before you start.

As long as the warrant is specific and limited I also don't really see this as a problem.

The bigger and real fight is with NSLs and FISA courts.

Ask any divorce attorney what happens if you refuse to turn over assets in a foreign account.

The judge may not have jurisdiction over property in another country, but she does have the power to toss you in jail for contempt.

Can someone explain why Microsoft doesn't have to but Google does?

I think I read briefly somewhere that Microsoft already stores foreign emails in foreign data-centers in order to comply with foreign laws better since they target enterprise customers that have those types of absolute requirements. I guess Google might be having their foreign emails pass through US servers at some point?

I'm sure someone else has more detailed/informed answer.

This decision is basically saying that the Microsoft decision was wrong, setting up the possibility of resolution by the Supreme Court.

So two different Districts, basically?

This article seems to include the explanation: https://www.techdirt.com/articles/20170204/06524836631/penns...

Essentially, Google doesn't claim to be able to definitively segregate what is or isn't in the US at any given time. Including the possibility that the location differs between when the request is made and when the retrieval is done.

Instead of self-hosting your email, why not just delete them after you're done reading them?

But if you never hosted them in the first place, how can you be sure they are deleted?

There is no reason a warrant couldn't request a search of the backups.

As I recall, google is required to keep certain records of emails for years (someone please correct me if I'm wrong here, this came up in conversation with some people working on gmail so I may mis-remember). So even if you delete your email, it might be sitting in a tape archive somewhere until google can (legally) get rid of it.

That means you're under active investigation and there's already a subpoena or warrant in place.

If you don't want somebody reading your steamy emails or social media postings in the future, you have options... don't create them, delete, or store on physical media in your home.

This sounds awfully close to "don't worry if you have nothing to hide". How about this; I'll create whatever steamy emails I feel like, and I'll use services that preserve my privacy while I do it.

No, it is more "manage risk".

If you are a cop, teacher, etc where your employer will demand to inspect your social media, you need to act appropriately.

If you are generally concerned about privacy -- US Mail.

> where your employer will demand to inspect your social media, you need to act appropriately.

"Acting appropriately" in that case involves telling your employer to mind their own damned business.

Hmm, time to move to Fastmail completely.

... Who are colo'd in New York City.

and at Switch Datacenters in the Netherlands. I am EU customer. And its aussie company.

Reply on news.ycombinator.com