Strongbox: a secret manager for AWS | Hacker News Reader

Strongbox: a secret manager for AWS

schibsted.github.io

164 pointssergimansilla55 comments

why not just use parameter store? https://aws.amazon.com/ec2/systems-manager/parameter-store/. Plus: don't use credentials directly, use the assume role feature.

The parameter store was not available at the time Strongbox was designed and developed. The Strongbox project page has a nice matrix comparing how Strongbox stacks up against Parameter Store.

In terms of not using credentials directly, relying on AWS KMS for encryption keys, and the use of IAM policies to control access to secrets, Strongbox and AWS Parameter Store share a similar design.

This. We used credstash for a while (same concept as this, at a glance) and now we're running for parameter store

AWS Parameter Store certainly has merit. As you can see from the comparison table there are certain functionality that is supported by Strongbox which AWS Parameter Store does not currently offer, like larger secrets, more secrets etc.

You can use the assume role functionality with Strongbox. The examples don't do this for simplicity.

Yup, Strongbox has a dependency on DynamoDB so one may as well research the "native AWS" approach: https://aws.amazon.com/blogs/compute/managing-secrets-for-am... The nice thing about doing it the way described in the linked article, is that it composes really well with IAM. VMs, ECS tasks, Lambda functions and so on.

I don't understand why so few people talk about ParameterStore. It's a fantastic tool, and AWS does such a terrible job of advertising it.. hidden under SSM, an offering non-enterprise users probably don't care about, and giving the wrong impression that it's only for EC2.

ParameterStore should be part of KMS instead I think.

You're right. I've never even heard of ParameterStore before.

Exactly. I was wishing for an AWS based cli accessible credstore service for a while and didn't even know this existed until just the other week.

One big problem with AWS and their incremental features / improvements is you never hear about much of it, or its buried in a post from Jeff's blog that you have to watch like a hawk to keep up with.

I had/have the same problem.

Since a while, I subscribed to „AWS this week“ on YouTube from a cloud Guru. This is just 5 minutes, and you have a good idea what‘s going on. (I have no affiliation)

I can recommend subscribing to Most recent announcements from AWS[1] for keeping up. You'll get weekly summary emails with headlines, HN style. They link to articles here[2].

[1] https://aws.amazon.com/subscribe-to-aws-announcements/

[2] https://aws.amazon.com/about-aws/whats-new/2017/

Is it possible to use Parameter Store without configuring System Manager for the instances where it's used?

I recently looked into SSM but was put off because the docs[1] suggested that you add the managed AmazonEC2RoleforSSM policy to instances, which among other things give them full read/write access to all S3 buckets. Edit: also discovered by others[2].

[1] http://docs.aws.amazon.com/systems-manager/latest/userguide/...

[2] http://www.daemonology.net/blog/2016-10-09-EC2s-most-dangero...

Works for me with instances having AmazonSSMReadOnlyAccess.

Yes using AllowReadingSSMParameters and eventually the KMS key to encrypt/decrypt the parameter

Yes, you can - it has a separate API with its own separate permissions. There's no need to use the full SSM to take advantage of Parameter Store.

Some of Amazon's IAM examples are very permissive. Perhaps it's because of the need for brevity but a link to additional examples that are more comprehensive and secure would be great.

Just revised a set of developer policies today. Most systems/users will only need a couple of permissions to leverage parameter store. And if you take the time to dig into the IAM permissions, it's quite impressive how granular you can get.

The policy I built today, for example, granted SSM:GetParameter* for parameters in the '/dev' or '/staging' path hierarchy. You won't find this fully documented at the moment, but you can separately manage encryption/decryption of secrets using conditions and kms:EncryptionContext, e.g.,

"Condition": { "StringLike: { "kms:EncryptionContext:PARAMETER_ARN": "arn:aws:ssm:<region>:<acct#>:parameter/dev*" } }

One point I will note in relation to other secret management schemes is that Parameter Store seems to use CMKs directly to encrypt parameters rather than relying on data keys and envelope encryption.

We wrote about our use of parameter store extensively. We have been very happy with it:

https://segment.com/blog/the-right-way-to-manage-secrets/

I really think it's just an AWS account with credstash under the hood, that some Amazon engineers work on full time. Seems like a no-brainer to use imo, since it's very polished.

The comparison with Vault is somewhat misleading.

- There are several free UIs available for Vault like https://github.com/djenriquez/vault-ui, they just aren't provided by Hashicorp

- I'm not sure what "file secrets" might refer to other than storing the contents of a file, for which Vault's CLI provides shortcuts.

- While Vault's generic k/v backend doesn't support purging expired values in order to avoid accidental data loss, there are plenty of other backends that integrate with AWS STS, Postgres, etc. to provide dynamic, short-lived credentials.

- I'm not sure how Vault doesn't support encrypted backups seeing as the data is already encrypted on disk. Take EBS snapshots or something.

- Versioning of secrets is a popular request and Hashicorp has some "vague plans" which is an improvement from "no plans." https://github.com/hashicorp/vault/issues/1364#issuecomment-...

Plus you get a full REST API, a ton of auth methods, granular access policies, etc.

It also has that snarky little footnote saying that a "TTL" isn't the same thing as an expiration time next to the entry claiming vault doesn't support secret expiration.

The footnote and the entry was a bit rushed and was not intended to be snarky.

To the best of my knowledge - please correct me if I'm wrong - you cannot say a secret should be valid in a time period in Vault. The TTL will tell the client that they should check back with Vault for the most up to date secret when the TTL expire. If the client credentials are still valid, and the secret has not been revoked or updated in some other fashion, the client will fetch the same value again. Hence the claim that it does not support secret value expiration.

Thank you for your feedback and clarifications.

Vault is a well regarded project that is designed to solve a broader set of problems than Strongbox (i.e. Vault has many different backends, more auth solutions etc.). While there are third party solutions and plans to get more convenience functionality into Vault, we decided to focus on what you would get out of the box today.

"file secrets" was intended to be that you can input a binary file directly either in the CLI or GUI without having to manually base64 encode it first. Please correct me if I'm wrong, but I think that is not currently possible with Vault? Strongbox used to require the user to base64 encode the binary file first as well.

Hi, this really needs a REST API.

Most secrets are injected in real time at the startup of of a container. For tools like this or plugins in k8 / docker to access your store you should support this. It's really bad practice to create a hard dependency on your secrets store by using the Java sdk directly in your application. If that's not the intent definitely provide a Golang sdk for proper plugin integration.

Can you explain how secrets are injected now?

Today people are using the SDK directly or injecting secrets via the CLI. Multi-language support and integration with e.g. Kubernetes is high on the wish list. The approach we are looking at for multi-language support is to drop the dependency on the AWS Encryption SDK. Creating a read-only SDK in other languages is then straight forward.

Sure, but what if the language I'm using is not high on your priority list? Isn't it easier to add a high level rest api?

That is certainly one approach we have looked at. The question then is where would you run the REST API? As a side car, as a separate service, e.g. using AWS API Gateway and Lambdas?

If you look at the work required to write a simplified read-only client using the AWS SDK for KMS and DynamoDB, while perhaps a bit more work than integrating with a REST API, it is not that far off, which is why we are considering that approach. That would of course not help if you are using a language that does not have an AWS SDK.

Any of the suggestions above would most likely work. Running an API sidecar or separate service would just be the cost of exposing the REST API.

I could definitely envision people deciding against using your key store when comparing to something like Vault based just on the fact that you limit extensibility with no API.

All AWS SDKs use the underlying REST(ey) API. The SDKs are literally machine generated. I can download the WSDLs or take these files https://github.com/aws/aws-sdk-go/tree/master/models/apis and generate one myself.

I'm the creator of Strongbox. Feel free to ask me technical questions about the project.

Thanks for sharing and gj, nice to see Schibsted contributing to open-source :)

Thanks, your project looks useful

Here are some questions:

Are you using Strongbox in production?

The backup is mentioned, but how exactly do you backup?

Thanks

The project has been used in production since the summer of 2016. It is being used by a number of teams, and has been contributed to by more than 20 developers (there is a CONTRIBUTORS file).

The backup will take the data stored in DynamoDB (or a file) and serialize to an encrypted file. It will not backup the KMS key, or IAM policies. Hence, it is useful to take snapshots of your state, but not a full disaster recovery.

You can take a backup by a click of a button in the GUI, or using the cli: `strongbox group backup --group team.project --dst-path my-file.backup`

Including the Spring-Cloud config server would have been interesting , because it has all the mentioned features, but is running on top of git. If you are on AWS, you can use one if the free CodeCommit repos.

The cool thing is obviously the spring integration, which pulls the secrets on app startup and overwrites or augments the values in the application.(properties|yml)

It can also run kind of serverless by embedding it in your app.

Yeah, we have the same features with the strongbox-spring-boot-starter, it injects all secrets as properties on application bootstrap: https://github.com/schibsted/strongbox/blob/master/spring-bo...

If you're all in on AWS, just use parameter store. This scheme is just CredStash, and so is parameter store under the hood.

Even if you're not all-in on AWS, you can use Parameter Store for a trivial sum. The latency might be a bit higher, but in most cases it shouldn't matter since secrets usually have a nonzero TTL associated with them.

If you're not using AWS, the use of Parameter Store (or Credstash) becomes a turtles problem--because you need to provision AWS credentials. At that point, whatever you provisioned AWS credentials into might as well store your other secrets, too.

If you are using AWS, EC2 itself is a trusted third party that grants the appropriate permissions for your executing system (container, instance, whatever).

While this is true, it's something you can work with in the future, and it defined a secrets process earlier rather than later

If you do nothing you end up stuck with secrets in git history and you essentially have to roll them all.

So, my advice is generally SSM with no security is better than no secrets Management solution.

If you do nothing, you end up with secrets manually provisioned by logging into a machine and splatting out a configuration file. Which is a more secure solution than a turtles-problem parameter store.

You're assuming readers are managing snowflake servers, and I'm assuming they are in an containerized/autoscaling/more modern environment.

SSM wins for mine. I think you're right that the file approach wins for yours.

I'm assuming readers are just starting out, where auto-scaling and bloggable container architectures make very little sense. Configuration management is a prerequisite for those. That said, having a CM in place also makes the please-invest-in-us container ecosystem much less attractive, given that you can easily fall back to singleton-container-on-an-instance patterns (which is a great one for deployment regardless of your approach).

To that end, yes, their servers are probably hand-rolled or provisioned with minimal scripting like Ansible. And so manual secret deployment is to be expected.

If somebody is attempting to autoscale without having this solved already, they are making mistakes.

“Bloggable” LOL

Charging $1 per secret is interesting. I wonder if that will affect how people design their application.

We've had a pretty splendid experience with Vault so far, so I'm not exactly in the market for another solution, but this looks interesting.

I assume you started development on this because Vault et al didn't exist yet at the time? Or was there another driver?

The charges are for the AWS services that Strongbox uses to implement its functionality. That's $1 per KMS key, which can be used to manage multiple secrets, plus the cost of the storage backend. In the most common setup, DynamoDB tables are being used for this.

Strongbox was designed prior to AWS Parameter Store, and took its inspiration from projects like Fugue CredStash and similar AWS KMS based projects. While there are certainly pros and cons with the different approaches, which you can see in the linked comparison table, I think one of Strongbox' strengths is that it is easy to set up, and you get a lot of convenience functionality related to secrets management. Not having to maintain running services, and natural compartmentalization using both AWS IAM and AWS accounts was two of the drivers.

$1 per group of secrets, e.g. a service. In many cases I think this is fine, but it is certainly not a strength.

Thanks for the additional explanation, and for clarifying RE: $1/service; that makes sense.

neat, so the invoice already leaks about the number of secrets stored?

There's also https://www.envkey.com/

Founder of EnvKey here--thanks for the mention. If anyone's interested, EnvKey takes more of a saas plug-and-play approach to managing secrets and configuration and isn't tied to AWS services (though it does run on AWS).

Tools like this one (Strongbox), Parameter Store, etc. are great when you know AWS well and prefer more of a DIY approach on the integration side. EnvKey is more of a "just works" solution for when you either don't have as much DevOps expertise or want to get a secure setup up and running asap so you can move on to the next thing.

EnvKey also helps on the configuration management side with juggling multiple environments (development, staging, production, etc.), inheritance between environments, and keeping developers in sync with each other.

Happy user of Envkey here. I was trying to get my head around AWS's secrets management, and even with useful blog posts from the likes of Segment.io etc., it was giving me a headache.

Envkey let me set up and deploy secrets management across 3 server groups (development/staging/production) for our web app in under 15 minutes. And I can add new developers and devops people to our app in seconds.

I normally don't like hijacking other people threads about their hard work like this, but I just thought this product should be mentioned as it was a real time saver and headache reducer for us. Very happy to see people like the developers of StrongBox trying to solve this issue and make secrets management easier. The more players in this space, the better, IMO.

If you're comfortable with using KMS already, I recommend checking out SOPS: https://github.com/mozilla/sops

It can be versioned if you use git, access controlled via the KMS access privileges, and easily plugs into other tools because it uses YAML. Simplicity is the killer feature.

How does SecretBox compare with Vault [0] or BlackBox [1]?

[0] https://github.com/hashicorp/vault

[1] https://github.com/stackexchange/blackbox

[1] Please follow the link to see a comparison table with Hashicorp Vault and AWS Parameter Store: https://schibsted.github.io/strongbox/ [2] One of the fundamental differences between BlackBox and Strongbox is that BlackBox is based on GPG, while Strongbox is based on AWS KMS for encryption. Also BlackBox stores secrets in VCS repositories, while Strongbox primarily use AWS DynamoDB (while files are also supported).

Current parameter store user - one feature not mentioned in the comparison is the fact that accessing / updating a secret in parameter store will show up in Cloudtrail. Is there a similar audit trail in strongbox?

For Strongbox CloudTrail logs are indirect: you need to look at the logs of the backends used by Strongbox, i.e. KMS and DynamoDB. Consistent naming of the resources (managed by Strongbox) should help you filter out the correct entries.

I prefer the RBAC and YAML policies Conjur provides... https://github.com/cyberark/conjur

Reply on news.ycombinator.com