Whitelist outbound access from your network. It's not perfect and it can be painful to deploy, particularly gems, but it stops several categories of attacks.