Comment by runeks | Hacker News Reader

Comment by runeks | Hacker News Reader

Parent Full thread

Is it me, or does it seem weird to encrypt your secrets by uploading the secret key to GCP (contained in the config .yaml file)? I assume the controller instances are operated by Google in this[1] example.

Moreover, is there any sensible way at all to encrypt secrets without baking the secret key into your image? I can’t think of any.

I want to deploy an app that makes use of one or more fairly important secrets, but I haven’t found a sensible way to make it auto-scale while keeping the secrets on-premise.

As far as I can see, the only sensible solution is to create in-cloud/off-premise secret keys that can only be accessed by images signed with an on-premise secret key.

So,

1. Create secret key on an offline, on-premise machine

2. Produce application image, transfer to offline machine, sign with on-premise secret key

3. Create off-premise (in-cloud) secret, which can only be accessed by images signed with the on-premise secret key

4. Upload app image and signature to the cloud, allowing only this image access to the in-cloud secret

[1] https://github.com/kelseyhightower/kubernetes-the-hard-way/b...

To do this sort of encryption at rest you need a root. There are essentially three common solutions:

1. Keep the secret key in memory. This can be annoying though as a system administrator needs to unlock every process as it comes up; automation can help but is rarely deployed. This is what Vault does for example. There is a long design doc that we all worked on with this as an eventual option[1] but it is unclear if there is a ton of utility for users to be unlocking API servers

2. Hide the secret key inside of hardware using something like an HSM or a TPM. This is really the state of the art however it doesn't really work well in many environments and requires good inventory management.

3. Give machines a stable cryptographic identity and exchange that identity for a secret key that is kept in memory. This solution sort of shifts the problem around but is generally a good tradeoff between automation and security. The idea is, similar to SSH host keys, on first boot a machine generates an identity and then a system administrator approves that identity and later gives that identity authorization to do tasks. Kubernetes has a lot of the groundwork in place for this and it is still underway[2].

[1]: https://github.com/destijl/community/blob/3418b4e4c6358f5dc7... [2]: https://github.com/kubernetes/features/issues/43

We are working on a solution to this problem in the Clevis project[0]. It is a basic FUSE filesystem that will transparently decrypt your secrets/configuration. It will evaluate your decryption policy on each open and log the attempt.

You can see the initial proof of concept[1]. It isn't secure yet, for a variety of reasons. But it is enough to play around with. Moving to a better encryption scheme will give us the ability to do locks and per-block validation.

[0]: https://github.com/latchset/clevis [1]: https://github.com/npmccallum/clevis/blob/fuse/src/clevis-fu...

In AWS the combination of IAM roles + Param store gives us a pretty decent version of this. We have the app talk to param store to get secrets then just keeps them in memory.

>Moreover, is there any sensible way at all to encrypt secrets without baking the secret key into your image? I can’t think of any.

Using PGP / GPG sounds like what you need

Have you looked into docker swarm secrets? They work only with swarm services but I think they do what you want.

Kelsey Hightower has a repo experimenting with using HashiCorp Vault as a Secrets backend.

Have you considered using AWS KMS?

Reply on news.ycombinator.com