Secure your code: OpenVPN in the cloud | Hacker News Reader

Secure your code: OpenVPN in the cloud

code.mixpanel.com

66 pointstrefn40 comments

I'm not totally wild about OpenVPN. If you asked me which I'd rather expose, SSH or OpenVPN, I'd have to flip a coin. In either case, it's easy to realize the "right" configuration, with a single "bastion" host (hate that word) that provides authorized clients with transparent access to servers.

One thing you can say for SSH is that it gets a lot more audit attention than OpenVPN does.

In either case, the problem I think most startups should move to solve early is multi-factor authentication. I have friends building things with Yubikeys, and others building things with SMS-based auth; either are better than straight passwords, which are (among other things) a policy nightmare.

OpenVPN can be configured to be stealthy (to start a connection, the first packet must be signed with a shared-secret, otherwise it doesn't respond), while SSH doesn't. And OpenVPN has the performance advantage (tunneling is done over UDP instead of SSH's TCP - TCP over TCP brings about a couple of performance problems, especially when there's congestion).

Couldn't you just set up a port knocking for ssh (or anything else for that matter)?

The whole stealth thing (and "port knocking" along with it) is wildly overrated. If your security depends on people not knowing you're running SSH or OpenVPN, you have bigger problems.

If you want to get all cutesy, go hack SSH to add an extra round or two to OpenSSL's AES. That'll learn 'em.

it depends upon your threat model.

there's probably a bug in some code on your system that's exploitable. maybe openvpn or ssh or its configuration (c.f., debian crypto bug).

under this model, anything that increases the time til (inevitable, under this threat model) success, works.

running ssh or openvpn on a different port or whatever to "hide" it, will help against mass scans for common exploits. it increases the work to find the problem. of course it won't really help against a concerted, targeted attack. but i'm pretty sure unskilled, mass attacks are more common than targeted attacks.

The guy with the OpenVPN zero day probably isn't getting slowed down by your stealth configuration. At least, that's how my threat model works.

If you believe in stuff like this, then do it right: require the attacker to know a secret algorithm (like "AES with 3 extra rounds") to talk to you.

does the guy with zero day want to:

1. break into your machine?

2. break into some machine?

if (1) then i certainly agree that none of this really matters (i apologise if that wasn't clear from my previous comment).

on the other hand, if his objective is (2), and not everyone is using some of this obfuscation, then this separates you from the crowd for a little while (but my model is that as time goes on, you are inevitably exploited).

i realise after your company was exploited it might make you feel like (1) is always the case but for many (2) is the salient threat.

Generally speaking, the people running around with OpenVPN and SSH zero day are looking to break into your machine. The people looking to break into any machine are either targeting Windows clientsides, or weeks-to-months-old web vulnerabilities.

People with SSH zero-day are not, by and large, looking to burn those vulnerabilities by spraying them into every busybody's honey pot logs.

> The people looking to break into any machine are either targeting Windows clientsides, or weeks-to-months-old web vulnerabilities.

Actually the people looking to break into your machine are targeting windows clientsides and weeks-to-months-old web vulnerabilities.

There's a cost involved in developing 0day, droppers, remote access trojans, maintaining breach and exfil teams etc. If these guys can get into the developer laptops with an email, a wink and a PDF then why waste the 0day? If you're putting all your effort into a custom SSH daemon without expending equivalent effort on your connection sources (especially when connecting to the Internet) then you're doing it wrong.

Still, if OpenVPN drops packets lacking the HMAC packet authentication without processing them further, then either

1) the guy would also need the HMAC key, or 2) the zero-day is in the code that looks at the HMAC signature.

It's not that I only count on this for security, but it's a matter of reducing the attack surface. Likewise, I don't have passwordless guest accounts on all my servers, since that would make the attack surface even greater.

I think the parent comment is referring to OpenVPN's HMAC packet authentication of the TLS control channel, the --tls-auth option. From the man page:

The rationale for this feature is as follows. TLS requires a multi-packet exchange before it is able to authenticate a peer. During this time before authentication, OpenVPN is allocating resources (memory and CPU) to this potential peer. The potential peer is also exposing many parts of OpenVPN and the OpenSSL library to the packets it is sending. Most successful network attacks today seek to either exploit bugs in programs (such as buffer overflow attacks) or force a program to consume so many resources that it becomes unusable. Of course the first line of defense is always to produce clean, well-audited code. OpenVPN has been written with buffer overflow attack prevention as a top priority. But as history has shown, many of the most widely used network applications have, from time to time, fallen to buffer overflow attacks.

So as a second line of defense, OpenVPN offers this special layer of authentication on top of the TLS control channel so that every packet on the control channel is authenticated by an HMAC signature and a unique ID for replay protection. This signature will also help protect against DoS (Denial of Service) attacks. An important rule of thumb in reducing vulnerability to DoS attacks is to minimize the amount of resources a potential, but as yet unauthenticated, client is able to consume.

tls-auth does this by signing every TLS control channel packet with an HMAC signature, including packets which are sent before the TLS level has had a chance to authenticate the peer. The result is that packets without the correct signature can be dropped immediately upon reception, before they have a chance to consume additional system resources such as by initiating a TLS handshake. --tls-auth can be strengthened by adding the --replay-persist option which will keep OpenVPN’s replay protection state in a file so that it is not lost across restarts.

and I just realized who I'm responding to... :) I'd be interested in knowing if the above isn't a strong/decent rationale.

Well, wasn't SSH de facto broken for almost two years on Debian distros, due to the bad random number generation bug?

Everything crypto on Debian was tainted. OpenVPN also depends on OpenSSL.

Multi-factor is a big deal, especially if you ever want to deal with credit cards at all. We looked at WiKID (open source two factor solution) but building a LDAP backed VPN solution around it was going to be a fair bit of work, so we ended up using PhoneFactor, which is a pretty cheap commercial offering using your cell phone instead of physical token. Big fan.

If they only thing you're planning to run over your VPN is SSH, then yeah, it's a coin toss whether you should run a VPN or just expose OpenSSH to the internet. But if you're planning on running anything else over your VPN, then I'd wildly disagree that the two are equivalent.

Someone else already mentioned the performance advantage, but just the user convenience advantage alone would be a deciding factor for me.

I'm considering the security of the two programs, not the convenience --- although I think SSH is more convenient than you give it credit for.

You can tunnel other protocols over SSH.

Well aware of that, but it can be a bit of a pain if you're tunneling more than a couple protocol or if your tunneling to more than a couple of hosts. Also, VPN let's you use the same configurations as you would if you were local to the machines, you don't have to use localhost, etc.

And of course, there's also well known performance problems tunneling over SSH because you incur a double hit from the TCP backoff algorithm. Not to mention that you can't (easily) tunnel udp over SSH....

Come to think of it, there's no real comparison. SSH is a great remote shell protocol, but a shitty VPN protocol (not surprising since it's not * intended* to be a VPN protocol).

If you would rather use ssh than OpenVPN but still want the convenience of "direct" connections to computers on the other side of the gateway host, and you want to avoid the tcp-over-tcp slowness problem, try my sshuttle project: http://github.com/apenwarr/sshuttle

Yeah I'd go the ssh route myself. Nothing against OpenVPN, but the (or maybe another) big problem in this scenario is that people are using passwords to log into servers in the first place.

OpenVPN supports certificate based auth but key-based SSH auth is considerably easier to do (and depending on your experience may be easier to manage).

You can also restrict the commands that can be run so you can use things like Git or tunnel MySQL loopback connections without handing over the box.

We have been using OpenVPN to secure one of our Rackspace servers dedicated to development for the last couple of years. We also have ejabberd running on that box (just talking to the vpn interface). It works great for distributed development and the internal secured jabber IMs are great for transferring sensitive stuff like passwords between remote folks. We also have a persistent conference room for a shared "watercooler".

The main problem we have is key management. Does anyone know of a good admin app for generating, storing and revoking keys?

I've been using tinyca2, which has a really nice gnome interface for managing a certificate authority. I'm really happy with it and it takes a lot of the pain out of CA and certificate management.

Are you still generating keys by hand or have you found something that works a bit better?

I've been looking around for an admin app as well. It would be even better if I could tie it into an intranet site as well.

Interesting. We ran ejabberd over SSL for years without needing a VPN. Most DVCS' support SSH in some form or another. You might find SSH key management a bit easier.

tinc is really good little secret I have found if you want a true mesh P2P secure overlay network. It's open source, has a good community in freenode #tinc, and really easy to install and use, just read the man page http://linux.die.net/man/5/tinc.conf

http://tinc-vpn.org/

can you tell me what the actual difference between OpenVPN and tinc is? I know OpenVPN very well and really like it. Why would I prefer tinc over OpenVPN (or the other way around)?

In my opinion, they share a lot of the same features like running in userspace, easy configuration, ssl/certificate based authentication - just the same.

Front page of the project website:

"Automatic full mesh routing. Regardless of how you set up the tinc daemons to connect to each other, VPN traffic is always (if possible) sent directly to the destination, without going through intermediate hop"

OpenVPN doesn't do that, and it sounds very useful when multiple machines are involved

I strongly recommend not relying on the VPN written in C that nobody's heard of that uses a hand-rolled custom RSA authentication algorithm to authenticate connections for your perimeter security.

we are doing that in our production environment. We use different boxes (1 on US and 2 on EU), they are connected with openvpn to serv our different needs.

Our personal development boxes are also connected to each others using OPENVPN and firewalled. All the internal code made for our app is using the VPN ips, and every application is listening on these IP only (not localhost nor 0.0.0.0)

(applications are mysql/couchdb/git/apache)

That's pretty great !

I was just looking for something like this, literally about 30 seconds before you posted it. Thanks!

My problem with OpenVPN is that you cannot access it from an iPhone.

The diagram has connection lines between all nodes; does this imply N² point-to-point OpenVPN connections, with a subnet for each connection?

That diagram was drawn like that just for exemplifying purposes.

Just to illustrate what network interfaces you would use for communicating between server processes.

The lines between the nodes were actually meant to show that those nodes use the Rackspace internal network (ServiceNet) for communication - see the colored labels on the diagram.

Ohh, I see, blue = unencrypted. That makes sense, thanks. So there really isn't any gain over using SSH with private key authentication with this?

Yeah, there is. First, doing it with SSH+keys requires either multi-step (which is annoying, esp. for copying files), or exposing each servers SSH to the net (which might be unfeasible if you don't have public IPs for everything).

Second, getting a dump of a table from your MySQL DB with SSH+keys:

  ssh prod-db 
  mysqldump what-i-need > file.sql
  exit
  scp prod-db:file.sql .

With VPN (assuming the connection is running):

  mysqldump -h prod-db whatever > file.sql

(I'm not saying it's neccesarily a good idea to du dumps of your production db, but that was the example that came to mind.)

There's also stuff like instrumenting a running Tomcat with JVisualVM on a server in your remote environment (which, no, you can't easily do over SSH-forwarded ports.. also, portforwarding is annoying).

Finally, VPN allows you to run internal services. A jabber-server was mentioned in the article, and perhaps a webserver with some reports/dashboard thing, cacti/nagios stuff etc. without worrying about setting up authentication and SSL for each. Each VPN user gets his own IP, so you can just do crude IP allow/deny if you need, or you can put the customer people on one subnet and allow them to access the reports server, and the devs/ops on another that can reach everything.

While I think you're right in that VPN setups make a lot of things smoother, SSH actually compares pretty well with the first two of your examples.

"Yeah, there is. First, doing it with SSH+keys requires either multi-step (which is annoying, esp. for copying files), or exposing each servers SSH to the net (which might be unfeasible if you don't have public IPs for everything)."

You can work around much of the annoying stuff in multi-step setups by using SSH's ProxyCommand configuration variable. For example, you could have something like this in your local .ssh/config:

  Host prod-db
    ProxyCommand ssh bastion nc -w1 %h %p

After which 'ssh prod-db' and 'scp prod-db:...' work as if the bastion host wasn't between you and prod-db.

"Second, getting a dump of a table from your MySQL DB with SSH+keys:"

If you can access prod-db with SSH, you can dump the database with a single command:

  $ ssh prod-db mysqldump what-i-need > file.sql

  > ssh prod-db 
  > mysqldump what-i-need > file.sql
  > exit
  > scp prod-db:file.sql .

I remember running across a tutorial/blog post once about how to automatically setup back-channel on an ssh connection so that you could do:

  > ssh prod-db
  > mysqldump what-i-need > file.sql
  > scp file.sql mymachine:.

Edit: Here's a similar article (the one I was thinking about is older than this one) --

article: http://codysoyland.com/2010/jun/6/ssh-tip-automatic-reverse-...

hn discussion: http://news.ycombinator.com/item?id=1409263

  > ssh prod-db 
  > mysqldump what-i-need > file.sql
  > exit
  > scp prod-db:file.sql .

You can definitely do better than that with ssh:

  ssh prod-db "mysqldump what-i-need" > file.sql

If the dump is large and the network is the bottleneck, you can also do it like this:

  ssh prod-db "mysqldump what-i-need | p7zip" | p7zip -d > file.sql

Edit: Noste gave the right oneliner, sorry I didn't notice it at first. Consider mine only as an expansion of his second point.

Reply on news.ycombinator.com