Securing PostgreSQL [pdf]

Virtually nobody writes their own ciphers. But "Don't roll your own crypto" is still important and still very common advice. Why? Because the stuff you do to actually apply "AES256" to your data is the stuff that actually causes crypto vulnerabilities.

In fact: if you do some reading on modern stream ciphers and design your own stream cipher, and then plug that cipher into libsodium, you are probably more secure than you would be if you tried to use AES256 in your own library without libsodium!

Or do you mean that re-implementing AES-256 in your own library instead of using a well-tested implementation is "bad"?

The problem here is that if your application requires the use of encryption keys, and the user which runs this application gets compromised, you have a problem anyway - since like most applications, yours probably also doesn't care about the key's security once it's in memory. If they get that far, the only thing you can do is replace/revoke those keys and take the hit.

In more secure non-cloud setups, encryption/decryption is being done by something like a HSM, a box with only one interface (usually PKCS#11) which can be used to encrypt, decrypt and sign stuff, and you never see the keys. You have software HSM's - and you could try to apply the same principal in the cloud, where you have an isolated box with only the very well protected and audited soft HSM running.

But not sure the cost of learning and maintaining such a system is worth it for most situations, where I would use an API service like Hashicorp's Vault. Most of the compromising of keys and secrets doesn't happen on your servers, but on your or some developer/user's work machine. How many crap is exchanged over email, dropbox links, slack, skype, ...? Keeping keys out of the hand of the user and eliminating the need for your users to have them at all is higher on my priority list.

Multiply the odds of the problem by the costs of the problem. If a DO outage would cost you millions of dollars, then it will be much, much cheaper to spend $250,000 a year to fix a problem than it will be to deal with the consequences.

The problem is that most software developers are gamblers, and most businesses encourage that behavior. Some even cultivate it. We take stupid risks all the time and when we don't get caught we think that means that none of our actions have consequences.

And mostly those people are right. Stock options pay off so infrequently and so inconsistently that they are a poor incentive for long term thinking. In fact the only time I ever got any money out of options was because of a pump and dump by the founders (aka an acquisition).

Even the worst developers I've worked with would take 3-5 years to drive a company into the ground, unless nobody else was paying any attention at all. And when the company folds, they've still got years of take-home pay and they just have to find a new job.

The problem is that younger devs/owners are much less likely to have ever experienced any significant outage, so it's a non-issue for them. They can't see why they should pay good money to prevent it.

We've had much better results with slightly older and more experienced people, who realize that our service is an absolute bargain compared to the cost and risk of attempting to run continuous backups, etc. themselves.

Don't use pgcrypto. Use libsodium to encrypt in your application.

You can get that fraction closer to 50% without a human in the loop by segregating your crypto code from the application in a virtual HSM, using something like a TLS client certificate to authenticate your application to the HSM, have humans in the loop for restarts/bringups of the HSM itself, and doing pretty aggressive monitoring on the request patterns between the app and the HSM.

The threat model here is that someone owns up your app server --- if that wasn't in your model, you wouldn't need application layer crypto. The idea is that owning up the app server will get an attacker enough access to make requests to the HSM, but not control of the HSM itself. If the HSM can detect abnormal volume of requests, you can use it as a circuit breaker to prevent bulk exfiltration of your secure data. You also get accountability, so that when your server is compromised you might know exactly what records were accessed.

Typically, the app server itself will handle all of the database operations, and the virtual HSM just provides a "seal" and "unseal" operation --- convert plaintext to ciphertext, convert ciphertext to plaintext.

† The 5% you're getting in the dumb online crypto scenario is that if you store the root crypto secrets in a file, an attacker can't necessarily recover it from an SQL injection attack --- but in reality the percentage is probably lower, since most of the time SQLI will equate to RCE.

You still have to bootstrap vault, but it's a pretty robust and easy to implement HA system.

If you're worried about a SQL dump exposing password hashes, segregate password validation into its own microservice. This comes with other benefits: for instance, you can ratchet up the work factor on your password hash, because the service will very easily scale horizontally.