Comment by yeukhon | Hacker News Reader

Comment by yeukhon | Hacker News Reader

Parent Full thread

> "The simplest hack is to use a configuration management tool"

> "You can also use a credential management suite for a more robust solution"

You have to start somewhere, this is why automated system with credentials isn't a simple task to solve.

Imagine this, you use consul to store your secrets, as you wrote up a script for Ansible to look things up, this include secrets. Great, but you need to authorize. So you need to keep the token somewhere. It's okay if you have a person typing in, but if you deploy through Jenkins on a regular basis automatically, you need to add that token as a password to Jenkins's data store.

But you provision and configure your Jenkins using your Ansible playbook, and your playbook needs to get to Consul to get to the secret so you can compare the key on Jenkins. Now this is getting tricker.

Okay, say you are on EC2, you can use instance profile to grant instance access to S3 / DynamoDB where you put your ultimate secrets. But if you are paranoid or for some reason you decided to do client-side encryption and keeping the master key to yourself (you generate them and you keep them, never given out to AWS), then you are back to square one. Where the heck do you keep your master key safe? KMS? No? In your own Consul? Okay where should we keep the token? Ugh.

What if you don't use AWS? Google Cloud?

You have to trust at least one place. If your repo can be trusted, let it be. There are three major factors in combating a system that has to put trust in password/cert/key:

* keep rotate the credentials

* everyone should pull from the same credentials management infrastructure

* least privilege

But #2 takes a while to complete, mainly because now you need to develop / rewrite your script on Jenkins to not use Jenkins's password datastore, and the password masking plugin. You have to develop your own masking plugin.

The most trusted place is where you will need to add extreme security measure. Assuming the encryption can't be broken in reasonable amount of time, the only way to keep a secret unrecoverable is if the secret holder is dead ("only one can live if two people know a secret).

I don't think it's quite as complicated as you make it out to be.

1. Write a script that will, when given a specific key, will decrypt a file, extract a specific credential that matches the key, and return it. (Credential Extractor-Writer)

2. Put the CEW file somewhere the devs don't have access. (Credential Store)

3. Make a script to connect to a specific host and port using PKI or public-key auth (HTTPS, SSH), and make it feed a key on stdout, and receive a credential on stdin. (Credential Retrieval Script)

4. Make a script to push new keys out to hosts/services regularly (Host-Service Key Pusher) and a script to change/rotate credentials on the CS (Credential Changer).

(all of that can be implemented with Bash, Openssh, GPG, and 7Zip/Rar/whatever)

For your example, whatever manages Jenkins' configuration can simply call the CRS and save the resulting credential however Jenkins wants it. This covers just about everything except that single point of failure, the CEW file.

To protect that, keep each part of the CEW encrypted by an individual key, so only services requesting their key will get access to the credential, and distribute new keys when a credential is changed. Put expirations on key/credential pairs. Add key signing to everything. You can go farther and plug in google-authenticator and 2FA. Implement a two-man rule. Go crazy.

I am not arguing you can't implement a solution. You literally just repeated what I said by specifying A specific implementation. My point earlier is that you will never be able to keep anything secret. There is always a point of trust you have to give up and hope your extreme defense system doesn't crack / expose.

Ah, I thought you were just saying it's too complicated. Of course nothing is totally secure. But there is a point where "extreme" mitigations are called for, and not considered extreme, and the scope of potential problems of trust can be limited.

>> "The simplest hack is to use a configuration management tool"

>> "You can also use a credential management suite for a more robust solution"

> Imagine this, you use consul to store your secrets, as you wrote up a script for Ansible to look things up, this include secrets. Great, but you need to authorize. So you need to keep the token somewhere.

You see, the problem springs from you mistaking Ansible for configuration management tool. It is not, it is somebody's deployment script. What you want is CFEngine, Puppet, or Chef, and you need it download configuration templates and fill them with secrets on the server that you want configured. (CFEngine can easily do that; Chef -- probably, since it's not a standalone tool, but a Ruby framework, so you have ERB at your disposal; Puppet? no idea if it can use ERB in masterless mode.)

Now you need to solve the problem how to get the secrets to the server being configured. This would be easy if you had a specialized database service that ships credentials, and only the ones that are relevant to the server are sent. It's a pity we don't have such a database (or at least I am not aware of any), but it's not exactly a famine problem to solve, it's just juggling with access lists and some simple log replication protocol.

When you said

> "You see, the problem springs from you mistaking Ansible for configuration management tool. It is not, it is somebody's deployment script."

You lost credibility from my PoV. Ansible is capable of doing what Chef can do. Ansible can use dynamic inventory which means Ansible can use dynamic secret infrastructure. That's not the default mode of Ansible out of box, but much like Chef and Puppet, Ansible is made up of somebody's deployment script as well. You and I didn't just write up Puppet out of no where, and there are community plugins available for used, and those plugins are just somebody else's deployment script/component.

> Ansible is capable of doing what Chef can do.

So is gawk, except nobody tries to shoehorn it to do so or claim that it's designed for this. Ansible's very architecture and operation model are causing troubles where should be none, like applying configuration changes to hosts that happened to be temporarily down.

> [...] much like Chef and Puppet, Ansible is made up of somebody's deployment script as well.

I don't know the history of Chef, but I assure you that Puppet was never somebody's deployment script that happened to be published and gain traction. It was designed ground up as a configuration management tool, after its author deemed cfengine 2.x too troublesome, which again was a configuration management since its early 2.0 days (and probably since 1.0 version).

Configuration management requires completely different operation mode than deployment; one is asynchronous and unattended, the other is a synchronous list of steps to take and often needs to stop on the first trouble to appear.

> you need it download configuration templates and fill them with secrets on the server that you want configured.

Ansible can also use configuration templates just fine. It can also support secrets management, depending on how you want to store the secrets. The first tool covered in the series itself was Credstash (https://tozny.com/blog/crypto-tools-for-devops-credstash/). Ansible supports retrieving secrets from Credstash natively, either for use in commands or when populating templates: http://docs.ansible.com/ansible/playbooks_lookups.html#the-c...

Git-crypt is but one solution for storing secrets _next to_ the code that uses them. Credstash is another for storing secrets in another, server-accessible location. Both can work with tools like Ansible (or Puppet or Chef or Salt).

From the tone of the above (and your other responses) you're not a heavy Ansible user. That's fine. Use the tool you're comfortable with. But don't try to turn people away from an equally useable alternative to something you _do_ use just because it's not your tool of choice.

Puppet can do templates just fine in masterless mode. Though nowadays, you should use EPP instead of ERB for robustness (strong typing helps a lot)

Your puppet master can be your credential "database" if you use eg. hiera+eyaml. You still need to secure your puppetmaster, obviously, but since it uses public key crypto, you don't even need to make it possible for developers to decrypt secrets. Master key leaks would still be a problem, though.

There are various caveats when working with sensitive data in Puppet (4.something added the "Sensitive" data type which prevents things from leaking into catalogs) but since puppet clients are authenticated with client certificates, you can control which server should have have access to what data.

Reply on news.ycombinator.com