LavaBit's Architecture (2009) | Hacker News Reader

LavaBit's Architecture (2009)

possibility.com

140 pointsyankcrime57 comments

Just after the PRISM scandal exploded, I came up with the idea of encrypted-storage based email [1]. Unfortunately, within a little while of thinking of this, I read about lavabit and realized that they had the exact same security model that I was thinking of.

Now that lavabit's gone, I think there is a good opportunity for someone in a sane country with good data-protection laws.

[1]: http://firespotting.com/item?id=1632

It's very interesting. Something like this is really required. How ever, if this is a private project then people may not have absolute trust on it. If it is open source, independent verification is possible which will improve user trust. Also when malicious unnamed agents come after you through legal channels, you can drop the service like lavabit but some one else can setup an independent server and do a direct import of user data from your servers with user permission.

My plan was to keep it open source, and have free accounts, as far as possible. The monetization could have come from extra storage, like GMail, or custom domains.

Every time I see the phrase "open source", I mentally add "but not open binary"; how do I know that the service that is run uses binaries that are compiled from that open source alone? with trusted libraries? After all, anyone can employ a site-specific patch in their build process that adds additional "features" to their normally open source project.

Is there a reasonable way to have trusted individuals intermittently audit the service?

A no-trust-required system would consist of taking these sources and hosting your mail yourself. Which pretty much defeats the purpose, since what you want to avoid is hosting your own email (another alternative).

Honestly I lend more towards phasing out email completely. Maybe providing gateways to other protocols.

For a couple of weeks now, I've been brainstorming about how to create a system like this. I'd very much like to use this sort of setup for my own emails, but it simply doesn't seem to exist, especially not as open source (while lavabit claimed to store your emails encrypted, there was no way for anyone to actually vet their approach).

The most viable option I came up with so far is modding eCryptfs to allow "write only" mounts by supplying it directly with the derived public key for the passphrase instead of the passphrase itself. This would allow you to use vanilla components such as postfix & dovecot instead of having to write a custom application server.

Using the maildir format, incoming mails could be written to the write-only mount without having to read any files from the mount. On IMAP login, the mail daemon would use a passphrase derived from the login password to mount the maildir as read-write, using unshare to ensure that the mount is visible to that process alone, and automatically unmounting it when the IMAP user logs out.

Thoughts and ideas are welcome :)

Nice plan. I've also been brainstorming ideas - I'm now considering setting up something similar (though I have less knowledge in these matters):

* encrypt all incoming mail to a public key associated with the recipient's email address (new account creation would have to involve uploading a public key; on the other hand, if we are only talking about one user, then it becomes much simpler)

* this can be done while there is nothing written to disk (via exim, etc.)

* either write the resulting encrypted email to disk, or keep it in memory (depending on paranoia level)

* ideally, check incoming email if it already has a PGP header; if it does, do not encrypt on top of that (this would confuse certain mail clients)

* user logs in via IMAP, downloads new mail, mail client can be set up to transparently decrypt mail. User is encouraged to then delete mail from the server

This wouldn't be user friendly in the traditional sense, but my initial goal presumes that compromises involving friendliness vs end to end encryption suck, so screw it. I'd like to set up something like that for personal + friends' use.

Your idea of passphrase ~== private key is very nice. Mine is more ugly in the usability sense (at least for the median user), but at least at no point in the process does the server have to do anything with the private key. Still, of course, it would be possible to wiretap the whole thing at the "process incoming mail and encrypt it" point..

If you have to customize the client anyway, you might as well encrypt the metadata. It looks like you can use an unmodified IMAP server -- your MTA would encrypt each mail part individually and your mail client would decrypt.

The sticking point is server side search, which is available in IMAP4. You'd either have to skip that, or implement one of the search over encrypted data algorithms (which are rather slow).

> If you have to customize the client anyway, you might as well encrypt the metadata.

Yes, but then you'd need a custom extension, whereas the proposed setup would be covered by, e.g., Enigmail for Thunderbird. Depends on the userbase / use case - if one were to set up such a system for themselves, then yes, this would be nice.

> It looks like you can use an unmodified IMAP server -- your MTA would encrypt each mail part individually and your mail client would decrypt.

Yes, that's the plan :) No need to modify the mail server itself.

> The sticking point is server side search, which is available in IMAP4. You'd either have to skip that, or implement one of the search over encrypted data algorithms (which are rather slow).

That is true, and this would indeed involve writing custom code/extensions. Normally, email subjects are not encrypted / presumed 'public' (in a sense), so one way to do it would be to simply allow searching for email subject lines, which would not be stored encrypted on the server. Not sure how much customization server-side this would require.

edit another way to look at it would be, it shouldn't be possible to do any kind of search on the server. The client (which would presumably actually store all the mail archives (e.g. in an encrypted container which could be hooked up to a simple automatic offsite backup solution)) would do all the search. Again, however, depending on the setup, this might require some customization.

Well I guess it comes down to how important you think the metadata is. In reponse to your edit, if you are storing everything client side than POP works and is considerably simpler. Heck you could even just do local delivery.

The idea for IMAP with encrypted mail parts would be to allow multiple clients to access a remote mail store where the remote mail store could not read any of the stored email or its metadata even if it wanted to.

You'd also want your MTA to reject mail not sent over TLS. No matter what you do though if the remote server is compromised -- technically or legally -- your future emails are subject to capture.

> You'd also want your MTA to reject mail not sent over TLS. No matter what you do though if the remote server is compromised -- technically or legally -- your future emails are subject to capture.

agree & agree. Overall, the email protocol is fucked either way. :) As of now, I'm looking into, e.g. Pond [1]. Of course convincing people to just go ahead and move to something else is kind of quixotic / sisyphean; though it might be possible for a small circle of acquaintances where encryption is actually critical.

[1]: https://pond.imperialviolet.org/

That's funny. I was thinking of exactly the same thing. Another thing to do is limit how much email is kept, so old mail is automatically purged.

I suspect lots of people are considering these kinds of designs these days. :) There's been some discussion on various mailing lists, usually assinine, but once in a while clever folk join in.

> Another thing to do is limit how much email is kept, so old mail is automatically purged.

Makes sense.

I'm also not sure how easy it would be to provide an option for the user to choose, for example, whether to accept incoming mail over a plaintext channel. SMTP TLS handshake takes place immediately after the initial "hello". Ideally there could be two servers listening (the user would have to decide which server to use, each of which would imply a different email suffix), one with a forced TLS mode, and each user could choose which mode to use. Maybe I'm overthinking things over a problem that is unsolvable in the framework of the email protocol (which does suck in this regard).

I also considered the approach of transparently encrypting all incoming mail with the user's public key, but it has a couple of downsides, mainly:

* This only works if all of a user's clients support GPG. I'm looking for a solution that would work for any average Joe (on his desktop, phone and tablet) rather than a solution geared towards technical people.

* This process only encrypts the email's body, and not the metadata (from, to, subject, headers containing IP addresses, etc). As a bunch of people have shown after the PRISM fallout, analysis on metadata alone can reveal an incredible amount of private information.

* It would be impossible to support features such as server-side search, effectively making email search on your mobile devices impossible.

Of course any solution to this problem will be a compromise between security and usability, so let's hope a number of different solutions will see the light of day, so that everyone can choose the solution that works the best for them.

Nice, yes, thanks the reply, agree with everything.

> This process only encrypts the email's body, and not the metadata (from, to, subject, headers containing IP addresses, etc). As a bunch of people have shown after the PRISM fallout, analysis on metadata alone can reveal an incredible amount of private information.

I also had the idea to have encrypted containers something akin to what you are proposing, with PGP for the body underneath. It gets complicated..

Anyway,

> Of course any solution to this problem will be a compromise between security and usability, so let's hope a number of different solutions will see the light of day, so that everyone can choose the solution that works the best for them.

I very much agree. Here's to hoping to see multiple solutions soon :)

if you're doing this on your own hardware, why not just encrypt your entire disk? (or just a single partition, and give imap access to that alone through unix perms or, if you want to go further, apparmor etc).

Firstly because I'm looking to create a solution that can be hosted centrally: the average email user is simply never going to setup and host their own mailserver and setup the encryption.

Secondly: simply using a regular encrypted partition for both SMTP and IMAP access means that partition has to be mounted permanently, otherwise new incoming mails couldn't be delivered. This means if a hacker were to gain access to the system, or if a party would force the server operators to give them access to the servers, all emails would be instantly compromised.

I still think one of the sanest ways to do this is to set up a filter that takes an email message on standard input, and outputs the same message to a Maildir/mbox[1], with the body encrypted with the recipients public key (possibly signed by a server key, but that wouldn't add all that much in terms of security, as the server key would have to be always available).

1: Probably one filter that takes an email on standard input, and writes an encrypted "copy" on standard output, and another program that takes a message on standard input and delivers it to a Maildir/mbox:

    #.forward pseudocode:
    | gpg-encrypt-to-self | deliver

Optionally check if the mail is already encrypted:

    if is-gpg-message
    then
      deliver
    else
      encrypt-to-self|deliver
    fi

If you don't care about metadata then imho the approach presented in a blog post[1] linked in your firespotting thread is most reasonable. Most importantly the private key never touches the mail-server.

If you do care about metadata then you could do some lavabitesque tricks in addition to encrypting the message bodies.

[1] https://grepular.com/Automatically_Encrypting_all_Incoming_E...

Considering that STARTTLS is still optional and most MTAs don't implement it, what value is there behind storing the mail encrypted? Because governments are storing every email anyways, when they want to get to my data, why would they bother asking you to decrypt it for them (which you can't) instead of just looking at the SMTP communication between $OTHERSERVER and you.

With a very high likelyhood, that SMTP communication is unencrypted.

Of course $EVILGUY might have encrypted their message before sending it, but then you can't help the government anyways.

IMHO, as it stands currently, encrypted mail storage provides a false sense of security.

Is it really true that most MTAs don't implement it?

Looking through a bunch of random emails of mine (on gmail), I see version=TLSv1 somewhere in the headers, which I thought indicated that STARTTLS was being used. Am I wrong about this? Or is there some intermediate MTA that might not be using TLS in this case?

Could very well be. You can try and check all Received:-Headers, but there's no obligation that the MTA actually has to write whether TLS was used, nor can you be sure that the MTAs aren't simply adding some TLS string without actually having used TLS. Of the big email providers, only Gmail supports TLS.

The tech required for a truly safe email lives on a client, not a server: http://en.wikipedia.org/wiki/S/MIME

The problem is that only the body is encrypted. Who you are communicating with (and in term, who they are communicating with) is at least as sensitive as what you are writing.

But that's a different problem though, it exists outside of email. The meaning of "who you are" changes depending on how high/low you're on the OSI stack: sometimes it's 00:C0:C1:A4:C8:29 or 60.56.228.48 and sometimes it's anon2342foobar@yahoo.com

If they're wiretapping everything, there's no way for you to hide the fact of a communication taking place, but at least you can protect the data itself using client-side encryption.

> If they're wiretapping everything, there's no way for you to hide the fact of a communication taking place, but at least you can protect the data itself using client-side encryption

No, you misunderstand. With S/MIME, the email headers are not encrypted. This means someone can go to your email provider, get all your emails, and even though these are all S/MIME encrypted, build a nice graph of who you are communicating with, how often, correlate with significant dates, see the subject of each letter, file names of attached files, etc.

Here's an idea: public mail pools. The entire email message is encrypted and randomly routed to one of many addresses. Each customer downloads all the messages from all the addresses in the pool. You try to decrypt each message, and the ones that decrypt are (obviously) yours. The rest belong to someone else.

To make life tougher on spies, add fake email messages to the pool regularly.

That's exactly how BT Sync (http://labs.bittorrent.com/experiments/sync.html) works, but for data.

What about 1) requiring STARTTLS, 2) encrypting incoming mail that isn't already encrypted, using the customer's public key. The message then cannot be decrypted by anyone but the customer. (The customer experience is that all the messages they download are encrypted.)

Of course a system such as this will not provide you with foolproof protection against any adversary ever. If that is what you need, email is not what you're looking for, period.

But there is in fact an enormous difference between the proposed system and traditional email hosting. With traditional email hosting, all it takes is one FISA rubber stamp, one hacker gaining access to the mailserver or any of its backups, one rogue or compromised employee to access the entire history of every email you have ever received or sent.

In the proposed system, assuming it is properly implemented, the only way to snoop on your emails is tapping the line between the local and remote SMTP server, easily prevented if both parties use SSL. The system could even, based on the user preferences, warn about this ("Warning: this email was not delivered via SSL, it could thus have been intercepted while on the wire") or block incoming/outgoing SMTP entirely if the remote party doesn't support SSL.

It's not a solution against (government) snooping per se, especially not a solution against a party such as the US government specifically targetting you. But it is a defense against half of the world population's email history indefinitely being stored in your friendly neighborhood NSA facility.

You could always make it open-source and allow users to "download" and "upload" their mailbox. This way, when one service provider is shutdown, you can find a host and install the email system and upload your mailbox.

I had a similar idea, but I have no time to execute it.

> * How do you backup and restore your system?

> All critical servers use RAID 5.

Unfortunately RAID 5 (or any other flavor) is not a backup solution. It doesn't account for user error, RAID controller failure and numerous other scenarios.

It seems LavaBit had an interesting product, but this seems like a pretty critical design failure.

It's much harder to delete things from backups though. Lavabit promised that when you deleted a message, it was actually deleted. Redundancy against hardware failure but not against user error, software error, etc. was probably part of their security-convenience tradeoff.

Lavabit also flipped the bit and destroyed everyone's emails (without much complaint, at least that I have read). The goal was security, not availability. If security is the goal it's better to fail hard than fail [possibly] insecurely. POP/IMAP email also backs up itself--you retain a copy on your own machine.

That might be the best article/review I've ever read.

It is taken directly from the Lavabit "about us" pages* - I read them all a few weeks ago when I signed up and paid for pro! :)

* - the pages are gone now, which is why they needed to be mirrored somewhere.

Ladar was kind enough to answer the questions in an architecture template I have for High Scalability (http://highscalability.com/architecture-template-advice-need...). I guess they also used it for their about page, which makes sense given all the work that went into it. He's a very nice guy, a shame to see how things went down.

I'm still hopeful they'll all emigrate to Iceland (or somewhere) and start up again. Slightly hopeful, anyway.

It would be really awesome if the LavaBit team released all their trade secrets, including their C code for their custom daemons, and other substantial details, as an open source project. ...that is, if there's truly no hope, and they've shuttered the business for good, and never plan on re-opening their doors and returning to business.

Hardware and Service Provider expenses aside, this would represent an opportunity for many people to create their own systems and participate in a distributed, federated network of professionally vetted, (ostensibly) secure e-mail services, based on field tested architectural standards.

If the company is dead, why not turn lavabit.com into the homepage of a project spawned from the ashes of the original business.

I don't understand why the service provider needs to store or provide any keys. Let the customer generate a private and public key, store the private key anyway he/she chooses, and then broadcast an address/public-key combo to would-be contacts. You can communicate using a totally insecure file-sharing service -- e.g. usenet, anonymous FTP, or mailinator.

Unless you're shipping the customer a persistent component that doesn't automatically update itself every time they connect to the service, you might as well just store the keys, because (as LavaBit apparently discovered) if a court decides you need to cough up your users info, they'll probably just get you to install whatever code is needed to make that happen.

If I understand correctly, the decryption code was in the server, so the user was providing the password for the private key, and the code from the server was doing the encryption/decryption (by server-side or client-side execution, I don't know). This scheme could be backdoored by altering that server-side code. So, why not keep the code on the client-side (like signed, verified browser add-on or native application)? Then there will be no way to get the password of the private key from the user (except by installing a malware with some 0-day browser vuln).

Note that Chrome extensions can suffer from essentially the same problem, since they can autoupdate.

The reason nobody does this, though, is that end-users don't want to install software on their machines. Unfortunately, there is a fundamental conflict between desire to run software off of other people's machines rather than your own and desire for cryptographic security.

The client would have to be open source in order for that to work. Otherwise they can just force the provider to put a backdoor in the client code.

Probably open source with no downloadable binaries would be best. Maybe even better would be something like only pulling off a source control system so there's an easily accessible audit trail. The idea being not necessarily to prevent an evil adversary, but to convince a judge that there's no way to convincingly be an evil adversary.

I don't think you've addressed my point. I don't hand my private key to anyone, including the service operator, so there's nothing they can cough up to make anything happen.

If you go back to the original backpack encryption model -- I leave my padlocks out in public and tell people to send me messages at some dropoff spot (alt.wesley.crusher.die.die.die for example) and collect my encrypted (padlocked) messages whenever I see fit, which I then unlock with the key I have never given out.

This could obviously be dressed up like email if necessary. E.g. set up a mail server that bounces all unencrypted incoming messages to me with a "try again using this public key".

Yes, the authorities can track my attempts to access the server (but guess what, they already can). And because my inbox requires no password to access they can't tell me from any random person or bot trying to access my mail.

The authorities (and anyone else) can also deluge me with encrypted spam (including spam I can't decrypt :-)). It's not perfect.

That is not how the POP or IMAP protocols work. To make the system work with existing clients, the server needs to have the plain text when it is sending to the client. Surely this is also how they complied with valid warrants (wait until user logs in, then execute warrant).

Asymmetric encryption on the server means that an intruder can't read the content (except possibly for those accounts where the private key passphrase is still in memory) and nobody can search old mails until a user logs back in.

Um huh? What the heck are you talking about? Imap and pop can't tell if the body text of a message is plaintext or not beyond poop left by encoding /encrypting engines. Certainly my proposed system would allow plaintext messages disguised as encrypted messages to be sent to the end user, but so what?

I am not talking about how lavabit worked, I am asking why not design a system where the service provider need hold no private keys and thus have no way to comply with requests for keys. Yes, they can help the government track users and they can try to install malware on your machine, but fundamentally they don't know whether you're reading your email on a mac, pc, raspberry pie, or microwave oven.

Here's a pgp encrypted message. What can't I send by conventional email? (Or simply post on usenet, as my earlier post.) "They" can try to track every person who inadvertently downloads messages left for me.

-----BEGIN PGP MESSAGE----- <-- poop

Version: BCPG C# v1.6.1.0

hQEMAz/dtuqQ9lvGAQf/Rqb+/hNYGhdTli66144SlhBIDineb9uY0tc7p5kDOEm1 DmwqoQNoyX8LshRe1YlpCIiS7nW6Mmzhs86U65yA2/W4Rfs0gsfBx8R//01bBr54 qgRAMsoW426hIVc16XjlIVy+o7/FrynHkY3Vf0E7Ft7qbHL2OcKjIMxDtl0mK2dj W2c5/rvTiZeq6j1iKTn22DaD94PFjHVcE7H4IRGRKRnp5TxgZq0OAzGD00aSqWMM 4xZdiqFNr7J9o9Akoz8qYotSBjLXFoep+pDyD8EU9I6oA4Eqea3Ka2YXQ9m6/QwS 9VS6cPYccfqjms4X0V/E+fWRnkpyXomVETSamar2IMktO4BiRY6/qCjhpUywcag8 bJ+rOFrwVsSS+xy3XpXvRtlYRPGk8dA/BYH4b3Wz =D8kd

-----END PGP MESSAGE-----

the "persistent component that doesn't automatically update itself" that tptacek was talking about is the software used to generate keys, do encryption etc.

the safe alternative is that each user has to go find and install reliable third party software themselves. this is already possible with gpg et al and it is not used.

so instead someone needs to package the crypto code. and as soon as you do that, if there's any kind of update process, the code package can be forced (apparently) to modify the code to leak information.

so sure, you can do this securely. it's already possible, but it's not popular. and anything easy enough to be popular appears unreliable.

I believe that the LavaBit fiasco just proves the need for a decentralized system with no central points of failure. Although it's still in the beginning stages, bitmessage (https://bitmessage.org/wiki/Main_Page) offers a solution similar to the bitcoin protocol.

I would more people to start using solutions such as these instead of email. Although email has served its purposed for a long time, it was never designed around security and privacy in the first place.

> I believe that the LavaBit fiasco just proves the need for a decentralized system with no central points of failure

That's probably the only thing e-mail actually does really well. Centralization is definitely not the problem with email. Lack of end-to-end encryption and meta-data leakage are the real problems.

In theory, sure - but in practice... Most of my friends are @yahoo.com, @gmail.com, or their ISP.

Meaning, in practice they're centralized.

From a technical point of view, they are not centralized. Gmail is a massively decentralized system in itself, a feature partially enabled by emails flexibility. Even if gmail would go down, (other) emails would still continue flowing. And when gmail would come back online, most messages would find their destination due the store-and-forward nature of email.

The fact that even you mention several providers shows that federation does work.

Currently the major pain point in the architecture of email arises when people bind their online identities to singular email providers, even when email has nice facilities of not doing so by having a domain independent from the mail provider and setting appropriate MX records.

From a legal perspective, GMail is operated by one corporation which can have a single warrant issued, and be forced to reveal all of your emails, no matter how many servers your account resides upon.

Even if you do end-to-end encryption, they're still going to see who you're sending email to.

With a more decentralized system that for instance does onion routing it might be possible to radically improve privacy.

Old article, circa March 2009! Look at the monthly visits mentioned, and also this highscalability article- http://highscalability.com/blog/2009/3/30/lavabit-architectu...

I wonder how much of this architecture still exits and how much has changed!

Do they still host servers, use CentOS+Apache+MySQL+Memcache or was there an architectural change with more apt software components?

I'm guessing that the government wants Lavabit's private keys. Maybe they already sniffed the traffic and now need keys to decrypt it. What do you all think?

actually, I'm wondering if something like this could fit the bill for a HIPAA compliant medical communication system?

ok?

Reply on news.ycombinator.com