Modern anti-spam and E2E crypto
moderncrypto.org
moderncrypto.org
All of my mails are newsletters containing 10-30 links, and more than once I've found the mere inclusion of a single link to a certain domain can get something into spam versus a version without that link, often with no clear reason why (domains that are particularly new are one marker, though). Or.. how about using a Unicode 'tick' symbol in a mail? That can get a reputable sender into Spam versus a version without the same single character (all double tested against a clean, new Gmail account) :-) Or how about if you have a link title that includes both ALL CAPS words and ! anywhere? Your risk goes up a good bit, but just go with one of them, you're fine..
I now have a playbook based around numerous findings like this, some based on gut feelings looking at the results and some truly proven, and even with my solid reputation as a sender, I'm having to negotiate a lot content-wise each week. But do I like it? Yeah, in a way, because it's also what stops everyone else being a success at it.. Gmail sets the bar high! :-)
(Oh, a bonus one.. include a graphic over a certain size? Your chance of ending up in the Promotions folder just leapt up. Remove it, you're good. It doesn't seem to be swayed much by actual content. So I've stopped using images where at all possible now and open rates stay up because of it.)
I realized I should add a note, however, that everything I've said only applies to bulk e-mail (and sent through systems with a reputation for such) and not transactional or manual e-mail which suffers from fewer oddities for obvious reasons.
I think there could be multiple, relatively easy methods to avoid encrypted spam.
Someone here already suggested the first email being a "poke". And only if you send a poke back, would that user be allowed to send you an e-mail.
The user could also have some description about him, from his profile, appear when you hover over his profile image or whatever. If you receive an e-mail say from a company you're expecting to receive email from, then you could poke back, so they can send you that email. I mean there should be ways to make it easy for people to know who's a total stranger that could be a spammer, or someone trying to reach out to them for good reasons.
Then you could also have the emails under different labels by default. All the trusted e-mails would come to the regular Inbox, while the rest will go under a different label.
As you said, the email provider could also see the user's reputation over time, and if he's a spammer or not.
And these are just some easy solutions we can come up with almost immediately. I'm sure there can be others with a little bit more thought put into it. I certainly don't see encrypted email as some kind of "doomsday scenario" like the author predicts in the post.
There are at least three major downsides:
1) You still leak lots of metadata and the full data of the poke including most obviously the subject line.
2) Do users understand that their spangly new "encrypted mail" actually fails to protect a lot of important data? What if they (gasp) came to rely on it? I'd want to see usability studies showing a clear understanding of what is protected and what isn't.
3) You break other features that rely on the server being able to see content, like search, and the ads that pay for all of this.
It seems to work reasonably well, although there are some interesting ways you can game it. One I learnt from the Internet marketing world was some list builders (using legit methods, but perhaps promoting things that often get caught by spam filters) hire people or implement techniques to encourage new list signups to reply to mails sent from the same address as the list by asking them questions, etc.
This wouldn't work, because a miner can easily pay himself any amount of bitcoins that he has saved up in fees, and include this transaction in his own block (not broadcasting it). Thus he can basically create these "deposits" for free, and sell them for a profit.
That's the thing: whatever you try as a counter-measure, you always come back to money: in the above scenario, money would replace "deposits" because "deposits" would just be sold on the open market for money. Proof-of-work becomes money: if something important requires proof-of-work, you can be sure that a web app would surface that performs proof-of-work in exchange for money.
It always comes back to money, because whatever restriction you put on something, whether it be "pay fee to Bitcoin miners", "Solve proof-of-work puzzle", or something else entirely, these things will always end up being sold for money in an efficient market, because of the increased efficiency of division of labor: why should I use my inefficient smartphone to calculate proof-of-work, when I can pay a service with custom ASICs to do the job for me at a fraction of the cost?
As far as I can see, the only alternative that can work besides money is something that cannot be sold for money. And I can't come up with anything that fits this requirement.
Of course spammers can buy their services but when the price for normal sending is effectively free for normal users you can jack up the price for spammers to make it too expensive for them.
A nice benefit is that it forces sites to use something other than email, such as RSS, since they can't afford to send newsletters anymore.
Email currently uses a Receiver Stores model. SMTP servers can relay messages, but in almost all cases the message is transmitted directly from the originator's network to the recipient's network. The storage of the message only effectively changes _ownership_ once, even if the message headers say it was forwarded many times.
That makes email a Receiver Stores model: the recipient's network is expected to accept the message at any time and then hold it until the recipient comes to look at it.
Some of the bitcoin messaging protocols propose a Sender Stores model. That is, the message may be transmitted any number of times but the recipient's network is not responsible for long-term storage. The sender's network must be able to provide the message at any time up to the point when the recipient actually looks at the message.
There are some obvious restrictions such as requiring that the message be encrypted with a Diffie-Helman key (negotiated when the message is first transmitted to the receiver's network) to reduce the feasibility of de-duplicating millions of messages. And in order to prevent revealing exactly when the recipient reads the message, the recipient's network doesn't ack the message for a while.
Ultimately all of this is just designed to make bulk email (slightly) more expensive. Spammers run on very, very thin margins. But it doesn't do anything to solve the problem of account termination or blacklisting.
See http://cr.yp.to/im2000.html and http://en.wikipedia.org/wiki/Internet_Mail_2000
End-to-end encryption is needed to increase the load on a spammer as much as possible. Even if the spammer tries to re-generate the message "at retrieval time" the receiver should request retrieval several times (to obfuscate when the message is actually read) and the message should use multiple iterations of a cipher (and possibly HMAC) after an initial DH negotiation, or any other means to increase the cost for a spammer _and_ tie a message to a unique sender for reputation-tracking purposes.
Since early recipients of the spam will likely report it, it's fairly likely that subsequent retrieval attempts will find a downed (or sanitized) host, no longer delivering spam.
This will reduce the amount of spam actually delivered, and the spammer's production / revenue margins.
It would provide protection against passive snooping (NSA/GCHQ) even if it wouldn't prevent active attacks.
Edit: Typo
The usual approaches are:
1) verification of they key fingerprint by some other channel, such as the PSTN, but this is obnoxious and feels like tradecraft; you are unlikely to get normal people to do this for normal communications.
2) certification of trust based on 3rd-party verification of government identity documents or control of some address.
3) the web of trust. Might work well for a bunch of security-conscious HN types, but unlikely to be a good solution for people such as our mothers who have neither the cryptographic background to make intelligent decisions about signing keys, nor the inclination to care.
One issue I could see though is the initial email would essentially devolve into a "poke". Nobody would bother writing anything in it, which would mean the spam filters would have nothing to filter on.
that is a good thing: if the first message contains something else that is not just "poke", it's spam.
The person I replied to said that "anything that isn't a poke would be spam". Do you really think spammers are that stupid?
I've been working on custom software to improve the spam filtering on my mail server for the last year (side project). It currently works by letting hosted users forward spam messages to a flytrap account, and then the daemon runs, reads the forwarded message, tracks down the original in the user's mail directory, does a whois on the origin in the mail headers, consults its logs, and then adds a temporary network-wide blackhole to iptables.
Originally it was intended to work alongside SpamAssassin and SQLGrey and all that, but last night I started considering replacing SpamAssassin altogether. I love SA, but the spammers are beating it regularly now. My TODO notes in the code actually say, "reputation tracking for embedded URLs, domains, ccTLDs and gTLDs, sender addresses, and content keywords." I wrote the first bits of code for reputation tracking this morning.
It's not much of a step for the software really, because it already uses embedded URLs in a message as part of the profile "fingerprint" for finding the original message from a forwarded version.
But I'm a bit chuffed to hear that I'm on the right track, considering how effective Gmail's tactics have been. :-)
Small service providers have it really tough right now. Users don't tolerate any spam at all. A few years ago, the state of the art for small independent services was SpamAssassin + SQLGrey (or other greylisting) plus a few other tricks; that's not sufficient anymore, and most of us smallfry lack the resources to come up with something much better.
After just 6 weeks in production, the software already has 20+million IPs blocked at any given time.
There are some challenges though. Probably the biggest one is that it's designed for a specific mail server setup: postfix + dovecot (configured for maildir) + fail2ban + php (the code is in php, because it was convenient) + mysql. I don't know yet how portable it will be.
If you'd like to try it anyway, let me know and I'll post what I've got to GitHub in the next few days.
edit: alternatively, I've been more seriously considering making the current network ban list available as an RBL. Since I already have DNS servers, it would be pretty trivial to do.
Having said that, if this were available as an RBL that would also be really great.
Beyond that also one of the things SA doesn't do well is actually rejecting hard on sensible blacklists like the CBL. We worked hard to make everything heuristic based but it wasn't always the right choice. Some things need to be black and white. There's some code on SA for short circuiting now but it's not really the best solution. In my own spam filtering I have a bunch of hard rejects and they work really well.
Anyway, check out Haraka or Qpsmtpd for solid anti spam mail serving solutions. They work really well.
If all messages are seen by my phone, just to be inspected and discarded, then my phone is consuming bandwidth and processing cycles on irrelevant data. And unless my phone has free bandwidth, and the processing cycles don't consume power (or it has infinite power), how can it ever not be a factor?
There's also botnets, whose resources hijackers are happy to exploit.
Interesting that SMS appears to have simply legislated the problem from existence (or so says the article), I suppose if anyone can connect to a telephony network for cheap the problem would persist there too.
The anti-spam system is basically what you said: you spend a small amount of bitcoin, and the server grants you some large number of "tokens". These tokens can be used to perform actions which use resources on the server, either storing messages, or getting messages. The exact price is configurable, and currently set low [1].
There is a second mechanism that I use, which is intended to lower costs for the average user, but still prevent spam. Essentially, in each bitcoin transaction, a small "transaction fee" is required, which goes to the miner. It is possible to prove, via a cryptographic signature, that you were the originator of a transaction, and therefore the person who spent that transaction fee. The bitnet system grants tokens to people who can prove they spent money on transaction fees [2]. The idea being, a typical bitcoin user will accumulate transaction fees anyways, but a spammer will have to go out of his way to send bulk messages.
[1] https://github.com/ortutay/bitnet/blob/master/bitnet.go#L50 [2] https://github.com/ortutay/bitnet/blob/master/bitnet.go#L38
[1] - https://en.wikipedia.org/wiki/Hashcash
[2] - http://hashcash.org/papers/pvp.pdf
[3] - https://moderncrypto.org/mail-archive/messaging/2014/000782....
The idea seems to be that you provably "burn" a small amount of Bitcoin to get an identity. An innocent person can then carry on using that identity forever without doing any more computation. Meanwhile a spammer will ruin that identity's reputation almost immediately and then have to pay again to get another one.
OpenBazzar a bit-torrent like p2p version of ebay uses proof of burn to make bad behavior expensive.
In a system like that, maybe you could send your encrypted message along with some encrypted keywords that you consider to be spammy to some centralised service. That would, at least, avoid some of the client-side-filtering-is-too-hard problem.
As far as reputation, this might be one of the rare times where a Web of Trust seems like a good idea. Generating lots of false positives and negatives would be a lot less powerful if the value of those reports was filtered by how much you trust the account that made them. With email you already have an implicit source of trust, in that anyone you mutually email with is unlikely to be a spammer.
Seems like a really interesting problem space to be involved in.
There is idea that this somehow blocks access to email for people who have a hard time paying for things on the internet (for whatever reason), but it is misguided: everybody who has access to the internet pays for it. ISPs could easily give every subscribed 10000 free emails every month.
Texting costs money and yet people do it.
What am I missing?
The specs and the software for a start.
Then there's the fact that it's not the spammers that will end up paying, but the people running the systems that are compromised and abused to send spam, be they shared hosting servers or home computers.
Then there's the network effect. I'm not going to feel good telling my friends and family that it will now cost them money to email me. Especially when they can just contact me using Facebook instead for free and without having to set anything new up. Especially when the email service they're already using probably wont even support this new fangled paid-email system.
It would be a massive task to add this functionality to email, and it wouldn't stop the spam, so it's not worth it.
Alice wants to receive mail from Bob. Alice generates a public/private key pair and gives the public half to Bob. When Bob wants to send mail to Alice, Bob uses the public key Alice gave him. If Alice receives spam, she marks the public key it was encrypted with as "fuck it, the spammers got it" and never receives mail with that key again. Then she notifies Bob that the key he had has been compromised and sends him a new one. Alice could then, after Bob has lost her key to spammers one too many times, simply decide not to talk to someone like him.
This would give mailing list operators a large incentive never to share your email with anyone, otherwise you could just block them forever.
On the flip side, if the mailing list is really important to you, the operator could reject your new key and tell you you'll either receive their spam or you won't be part of the mailing list. Though I don't see why someone would do that in favour of just including ads in the mails themselves.
Alice gives her key to Bob, in the expectation that Bob would not be sending her spam. Bob then sends both spam, as well as legit mail that alice did want. Assuming that alice does not want to stop receiving the legit mail, but want to stop the spam, how does she do it in this scenario?
If alice blacklist the key for bob, but sends a new one, the situation didn't improve. If she doesn't send a new one, she stops receiving legit mail (that she wants, and cannot go without).
I switched to gmail because my mail with every other provider and client was choked with phishing messages from major banks. So much work has been done on preventing origin spoofing in 2014 that accepting phony mail from chase.com is a sign of gross incompetence.
Why is the cost of end-to-end crypto never taken into account?
I just can't believe that we have reached a point where it is possible to cheaply mass mail the way spammers do if you need to encrypt each email for each recipient. That alone should be disuasive enough, at least that's always what I thought. If I'm right, all the discussion about the need for client to extract features from emails and send them to a necessarily trusted centralized third party is useless. But I may be missing something, where am I wrong?
But let's make the crazy assumptions that we are effectively in a world where end-to-end crypto is massively used, virtually by everyone.
I guess it would be stupid to assume that the message are encrypted but not signed.
This means that it would be easy to have a list of identities (i.e., keys) which are sending spams, for instance using a web of trust, without users having to disclose any other informations that "I trust these identities, not these ones" (which could be as easy to do as clicking "spam" or "not spam" for the users).
Now that means that the spammers not only have to encrypt every single email for every single recipient but also to generate new key-pairs for almost each encryption.
Of course it is also very easy to mark as spam any email signed with a key that is considered too small (i.e., too quick to generate).
Now if you tell be that still won't do it without a "centralized spam system with global knowledge", I have to seriously rethink a lot of my assumptions about the cost of some computations.
- Public key encryption: 2048bit RSA can achieve up to about 200k encryptions per second on a high end cpu [1]. ElGamal-like encryption schemes using elliptic curve cryptography can get to about 100k encryptions [2].
- Public key signatures: RSA is hopelessly slow for good keys. There is no reason to use strong keys for this case, though. If you reuse primes you can use a 1000 primes to generate a million RSA modules (you need two primes per public key), effectively eliminating the costly prime search. Using elliptic key cryptography the key generation step is dirt cheap: With ed25519 you get 200k key generations per second [3]
[1] http://bench.cr.yp.to/results-encrypt.html [2] http://bench.cr.yp.to/results-dh.html [3] http://bench.cr.yp.to/results-sign.html
Formula used for each of the above numbers: 3500x10^6/[median for Intel Xeon E3]x4
I guess there's still the solution to only accept message from trusted keys but that means that 1- we have to be able to detect signature-rings from spammers in the graph (that clearly should not be a big problem), and 2- that there is an easy and realistic way to have a legitimate key in the trusted network before it can be used to sends emails…
Anyway, all that is very interesting from a theoretical point of view, but we are far from being in a world where end-to-end crypto is massively used.
Theoretically, any spammer could research you enough to send to mail purporting to be from your friends, but realistically no one would bother.
The problem with Web-of-Trust is that once you try to roll it out mainstream, most people will just click "accept" or "I trust this key" without thinking about it. Instead of "signature-rings from spammers" you would have spammers duping less-aware real people into trusting their keys.
That said, it would seem to be a longer feedback loop to get a number of actual people to open emails and trust their keys.
What if they don't have to do that because it's transparently done for them when they use the "flag as spam", "not spam", and "reply" buttons for instance?
Could anyone comment how spammers make money actually?
So basically, I can't send email from home? This is… unfortunate. If we want freedom, we need decentralization, and this kills it.
One (open, centralized) system for global comms. Another (closed, decentralized) for secure comms. Maybe even more than one "another", if contexts require different audiences.
Email is supposed to go from the sender's machine to the receiver's machine. That's how it should work by default, that's how TLS connections makes the communication vaguely secure, and that's how it makes it difficult for powerful third parties to have a peek at everyone's communications.
As far as I know, Gmail accounts are only a subpoena away from the US government. But a sheeva plug (or R-Pi) hosted at my home? They need a warrant. Even for countries that don't need warrants, wire-tapping everyone is expensive: it must be done one home at a time.
Now maybe the botnet situation is so bad that it is worth sacrificing our ability to send e-mail. Still, this strikes me as the wrong solution. Blocking outgoing 25 by default is fine, but we need to be able to lift the restriction if we want.
And there's the case of looking for work. I'm somewhat proud of showing my personalized domain name, but if using something other than a huge webmail can cause it to fall into a spam folder… Fortunately that has yet to happen, but this is one of the reason I hesitate to switch from remote SSH to a physical server at home.
I know that's probably not why it's private, but it got me thinking.
Trying to game the system in web search is a huge thing, cause there's a big reward. But G fights it pretty well.
Not such a bad idea G publicly ranks mail senders.
How does that work? Don't SIM cards cost more than 10 cents?
(in mailing lists, and yes, I checked, it seemed to be a legitimate email, so I let the sender know, but never heard back)
The first incident was that Gmail flagged an important email from my landlord as spam because it contained a forwarded message written in Danish, which the filter deemed to be a language I don't normally correspond in (it is nice, to be fair, that the filter actually tells me why it flagged the message). True enough. But I do live in Denmark, and in fact a mail containing Danish is a very good signal, for me, that it should not be spam-filtered.
I've moved to hosting my own mail as a result, and it's been going well so far. I use a fairly conservative host-based filtering approach. Just blocking hosts whose DNS doesn't match their rDNS rejects >70% of spam attempts, and adding Spamhaus's DNSBL brings that up to >95%. As far as I can tell from perusing the logs, it's quite conservative, and they're all true positives. And at least it rejects (if it's going to) in the SMTP session, so the sender will get a bounce rather than get silently filed into a spam folder, like Gmail does.
I do still get some spam, almost all of it from legitimate free-mail hosts who I can't feasibly filter by host (mostly Yahoo and Gmail). But it's fairly infrequent.
And cranking up the proof of work isn't going to do anything to prevent it. The only thing that prevents Bitmessage from becoming a cesspool is its obscurity.
And the only reason there isn't tons of it is because Bitmessage doesn't have the kind of user base that it's profitable to spam to.
It is much eaiser to spam Bitmessage on an industrial scale and profit than it is to do so against Gmail's anti-spam systems today. If Bitmessage had a hundred million users, then it would be profitable to spam there and it absolutely would happen a lot. In my opinion spam would quickly become the dominant form of traffic on the network, and the broadcast message feature would go the way of VRFY and open relays in SMTP.
If you didn't obtain a fresh piece of this script, but instead either reused, or tried to guess the token, then your signup still succeeds, but is marked as bad. Then, in an undetermined amount of time, a wave of bans would occur on all accounts that got marked as bad.
This prevents signups via scripts, but does so via making signup scripts untrustworthy, and so no one would be willing to put money in for they cannot be sure that it actually works.
> So I think we need totally new approaches. The first idea people have is to make sending email cost money, but that sucks for several reasons; most obviously - free global communication is IMHO one of humanities greatest achievements, right up there with putting a man on the moon. Someone from rural China can send me a message within seconds, for free, and I can reply, for free! Think about that for a second.
Someone from rural China can send me a message within seconds,
for free, and I can reply, for free!
This offer is only available for you alone and in person. Are you interested? Because I'd really like to know how your pedantic definition of "free" works out for you when it's obvious that things that require infrastructure (such as online communications, or traveling to a different country)... require infrastructure.
(TLDR: If you don't find it amazing that with a very small indirect investment we can actually communicate with people anywhere in the world for free, you and I need to have a long chat about being spoiled.)
It costs money, yes, but if you're sending a one off message and it's not time citical, it's not unreasonable to say that sending the message via email is cheaper overall.
Edit: which isn't to say that communication via the Internet isn't really cool, just that it's not actually free, nor would it necessarily be cheaper than sending a message via other means, depending on the circumstances.
Are we seriously debating the merits of email over snail-mail on hacker news? I hope this is merely people enjoying being thick and provocative, because the alternative is that there's some massive bubble-blindness going on right now.
I also thought your £50 example was a bit silly. Of course infrastructure costs money. There's means of communicating and delivering things like money that use infrastructure that require a far lower initial investment for a user, and that might be a better deal for that user, circumstances dependent. And not all communications need to be near-instantaneous.
> Of course infrastructure costs money.
You do realize you're.. well, you're not contradicting yourself, but on the one hand you agree it's obvious and on the other hand you insist it must be pointed out.
My example about money was just that. I could use Western Union or a bank transfer. Those things also have non-nonexistent (sic) costs. If you start pointing out everywhere that "Oh but there's this cost you didn't think about!", you won't be done by next week. You know all those libraries that let you check out books for zero cost? Reading those books requires education, which in turn is no free (No matter how much you wish to contradict that statement, the road you've taken requires you to admit to an unending set of cost upon cost which you don't think about in your day to day life).
You live in a society which expects you to have access to certain things. In the case you can't, there usually is infrastructure in place to help you out at little to no cost.
So again, please, just take this in: You can communicate with people at the other end of the globe. For free. This is not a legal forum and we don't need asterisks everywhere.
Obviously, both of us need computers, email accounts, network access, etc but there's no per region metering or anything. The cost of sending an email to someone sitting 10 feet from me or 10,000 miles from me is exactly the same. Mike's right, this is revolutionary.