Hacker News Reader

Comment by xhrpost | Hacker News Reader

Looking at some of these encryption-as-a-service providers, I'm a bit confused on one of the selling points. From my limited understanding: with a traditional system, you encrypt in the database but your encryption key likely exists on your main server, possibly in an environment variable. Attacker compromises main system and has access to both encrypted data and encryption keys. So, you instead use something like Vault to request an encryption key in real-time from a remote service and thus don't need to store it on your server. So, one of their selling points on their site is that Vault is better because two systems would have to be compromised by an attacker in order to decrypt sensitive data. The part I don't understand though is, if an attacker has compromised my server, could they not just initiate a request to Vault for a decryption key at that point? I feel like I'm missing something because this sounds like it remains a single point of failure.

View on HN

It's part of security in depth. Yes, if they own the server, they could keep requesting decryption keys. The main thing this protects against is someone getting a copy of the encrypted data, then breaking into a server and getting a key that is good forever. By using this system, it renders their copy of the encrypted data useless.

Then you have to add in defenses for the active attack, such as rate limits, anomaly detection on access patterns for decryption keys, and the usual host and network based intrusion detection.

It's one part of a complete security strategy.

> The main thing this protects against is someone getting a copy of the encrypted data, then breaking into a server and getting a key that is good forever. By using this system, it renders their copy of the encrypted data useless.

Does it? Is there any example on when encrypted data and a key in ones possession can be prevented from decryption?

(I haven't used Vault but I’ve used systems that solve related problems, like KMS.)

One approach is that the app server doesn’t possess the encryption key at all (or at least not the master key). Instead, it calls a remote service to decrypt each item as needed (or the item-level data key aka envelope encryption key).

In this way, an attacker can compromise the entire data set and the app server, but they still can't decrypt the data. They have to maintain ongoing access to a compromised app server in order to decrypt data items one by one, which (i) could take a long time for large data sets (ii) can be slowed down by rate-limiting (iii) runs the risk of being noticed (iv) if detected, attacker's access can be shut off immediately.

Additionally, you might manage and monitor your key management servers differently than your app servers; for example, you have the expectation that no one will ever log into key servers routinely, so any interactive access or unexpected running processes can generate alarms. The set of people who have access to key servers is different and much more limited than the set of people who have access to the app server. The key server can run in a different virtual network from the app server while providing extremely limited access to the app server (just e.g. TCP on the one port needed to provide this service).

This approach contrasts to schemes where the app server or data store has an encryption key. If the attacker compromises that, they can lift the entire data set and encryption key out of your systems and process it later -- and it's irretrievably gone. With the key server approach, stolen encrypted data provides no value on its own, and the attacker needs ongoing access to the key server to make sense of the data.

Yep.

Another aspect is separation of responsibility - don't forget insider-risk. Security is a process, not a product, as the cliche goes.

Where I work, we use Vault for (among other things) authentication between internal services. The developers do not have access to Vault and never see tokens/certs/passwords/etc, which are created by a different group. So if you want to run a rogue service, you need a conspiracy across departments, increasing your risk of detection.

Same principle you see in accounting, of course. You develop process, loci of responsibility and audit trails designed to enable the desired outcomes while at the same time making attempts to defraud the system impossible, obvious, or investigable after the fact, in descending order of preference.

"So if you want to run a rogue service, you need a conspiracy across departments"

Or a developer that plants a back door in the code. He then exploits it on the production server.

Maybe time is better spent on code reviews ?

You're right! That's one approach to exploiting this.

It's worth considering that multiple approaches, across multiple departments, can be used. A good environment might use Vault for authentication between services and require that code cannot go into production without a review and enforce it in code. Then you tightly control who has access to the administration of those restrictions and log all usage to somewhere else, such that even if someone does compromise the infrastructure to allow them to insert their back door it can be detected and the culprit identified.

Again, you're completely right. Code reviews can be a great way to spot malicious code! You're also right that the Vault usage pattern that parent pointed to definitely has vulnerabilities. It's perhaps worth considering that this approach could be used in a context where it might not be the only defense. Perhaps you could ask parent for a more detailed explanation of their org's information security practices?

As the other poster noted, this is not the only mechanism employed, which would be pretty silly. Among some of the others are, indeed, code reviews, static analysis and anomaly detection.

Another point worth noting about insider threats: overconfidence on the part of the attacker is frequently the defenders' friend. Metaphorically speaking, the crook knows about the camera over the door and will avoid showing their face to it, but didn't notice the other ones. And because they work there, they think they have a better handle on how things work than they actually do.

You're looking at a bad example. Good backup encryption generates a local AES key on a ramdisk, encrypts the backup with that, decrypts the AES key with a public key and stores the result. This makes it cryptographically hard to access the backups. The host never has access to the private key to decrypt the AES key to encrypt the backup into a usable form. There's no vault involved there because public keys aren't security relevant.

On the other hand, we're running a lot of mutual TLS authentication via CAs in vault. Vault has the private key of these CAs, and authenticated hosts can request signed client certificates for these CA chains. In this case, vault enforces certificate parameters, TTLs, CLR and other things. You could keep requesting certs, but once I revoke the vault authentication you stole, you're done once the certificate expires within 3 days.

And on top of that - yes, it's possible to escalate to vault using local credentials for vault. However, that's a very different threat level. Botnets won't abuse your vault instance. Heck, script kiddies wouldn't know how to abuse your vault instance after getting shell access.

If someone can RCE into a box, and find your vault integration, and abuse your vault integration, you're in for a ride and usually, vault won't be your problem at this point.

Vault provides an audit log. Since each process gets their own token it allows to correlate which machine/container got compromised. I think we will see more tooling being developed around that in the future to make the search more efficient.

The other aspect is that if secret rotation is well exercised, it means that in case of a breach, taking an instance out of commission and rotating all the secrets is not a risky task anymore. If developers copy-and-paste tokens on slack, the window of attack is limited to the TTL of the secret. It also prevents developers from using the same secret for another unrelated service.

You have N number of encryption keys for different parts of your data, then a single client server can only decrypt a small part of the data.

Since you need to request the key each time to decrypt things like key rotation become easier since you can do them centrally. Even if you can decrypt data X on a client node the key you got an hour ago for X doesn't work anymore, you must request a new one.

Then the server can do things like impose rate-limits. Maybe you can decrypt A-F, but only at a certain imposed rate, and alarms will be raised if the rate is exceeded.

You're basically right. However, it does mean that your secrets aren't stored on disk, so if your backups or SAN are compromised then you're still good.

It is also good for syncing secrets across multiple servers, and for keeping the secret up to date, if you have a password that needs to update nightly or monthly or whatever.

We use Vault in our environment and deployments... Vault has a concept of using tokens for authentication to pull secrets. You can assign max number of uses, max TTL and other parameters when you generate them.

For our deployments, we generate a single token with a max # of uses that matches the target server count for our deploy and also a very short TTL of 5 seconds. Our code gets pushed and the token is passed to each server (in an env var) during that time and a command is passed to refresh environment vars with secrets from Vault. So if a token gets compromised it's very likely to be used up and/or expired.

Very simply, it removes the possibility of keys showing up in config in source control or in server backups.

I like Ansible's approach to this better. We keep all our secrets in ansible-vault and have a password to unlock it for those "who need to know".

Sure, this means provisioning VM's requires human interaction to unlock the vault (once) during a kickstart. So what. I'd rather do that than have plaintext or even pre-hashed passwords lying around in source control.

As others have pointed out, Hashicorp Vault seems like a good solution to the problems of another Hashicorp product ... Terraform, which used to keep passwords in its state files (not sure if it still does).

You can also configure Ansible to lookup to Hashicorp Vault on the fly.

https://docs.ansible.com/ansible/2.5/plugins/lookup/hashi_va...

Unfortunately it wouldn't work for our network even if I wanted to use it, since our systems aren't allowed to connect outbound to the Internet (for good reason). I suppose I could set up a Vault server locally (?), but it's complete overkill for a problem we already solved in a simpler way.

Ansible is gradually adding modules for about everything. It's come along way since I started using it back in the 1.8 or 1.9 days.

Vault is usually hosted in-house.

Vault offers features such as short-term secrets which are unique to each client. Vault itself manages creation and destruction of the credentials on the server, allowing it to enforce credential lifetimes

Those are neat features, but our systems do not require them, and for our small team it would be unnecessary overhead.

And by in house, I presume you would mean within a cloud hosted environment, eg. AWS VDC, Google Cloud etc. in the most common case.

How do you handle autoscaling? Server restart? Autoscaling requires credentials as part of the VM image. For server restart the credentials must be stored on the machine.

Not everybody uses or needs autoscaling. In fact I would venture that the vast majority of companies doing business on the internet, even those that have sunk their money into the cloud computing gold rush, do not need autoscaling.

I would argue that everyone needs auto scaling, even if it's only to use it to scale manually, or change/upgrade instances easily. Being able to just tell your auto scaling group to do +3 then -3 to replace your 3 instances without downtime is quite nice.

We host internally on our own hardware and have other much simpler ways of achieving the same thing, should we need to.

You put the service account token on the filesystem and then have it removed from the filesystem after loading it. then it lives inside a process, where it is still recoverable by a determined adversary who attempts to obtain root and can inspect process memory, but you can further protect that by encrypting it in memory with an ephemeral key (which also should not be at a predictable location).

When you deploy a traditionally configured application to the server you will have a bazillion credentials to access different stuff like databases, external systems. Think how you are going to secure that key store you need for SSL? What is the point of having it encrypted when the password lies in a configuration file two folders away?

Vaults like this allow you to not have that cryptographic material on the server.

Now, this doesn't solve all problems. As you mentioned, when you have breached server security you have access to everything.

But, there are other problematic situations that this helps solve. Now you don't have to provide secrets when you are building images and you will not have secrets on the filesystem when somebody only gains partial access that lets him only read files.

This also lets you see who and when is accessing secrets and lets you manage secrets (think how you are going to replace database password?)

The real utility of these products is in generating dynamic short lived credentials. The longer a set of credentials exist the greater chance they end up written down or widely distributednor going home with a former employee. With something like Vault your credentials have a lifespan of days, hours, minutes, whatever the shortest timespan your comfortable with is. That limits your attack surface greatly. If your doing a break glass procedure you can be assured that credentials are revoked after a reasonable amount of time.

Vault provides telemetry. The secret request patterns for each service are available. Depending on complexity,anomaly detection can be build on top.

In addition to what others have said, having a secrets management system like this makes re-keying easier in the event of a compromise. If one app is compromised, you can easily update the DB keys without having to recompile/redeploy every app that uses the DB. You would still have to redeploy the compromised app to change its keys for accessing the secrets management service, though.

Reply on news.ycombinator.com